Property-Based Testing: Trying the Inputs Examples Never Reach ── C# in Practice with FsCheck and xUnit

When you have AI write code and tests, the tests often take the form of a list of examples: “for this input, the result is this.” The examples were chosen by whoever wrote the tests, so an input that was overlooked while writing the implementation may also be missing from the test examples.

Property-based testing replaces the list of examples with a “property that should hold for any input,” and lets a tool generate large numbers of inputs. When the tool finds an input for which the property does not hold, it shrinks that input as far as it can and reports it.

This article explains how to write property-based tests in C# with FsCheck and xUnit, using a small function with a bug as the example. The code follows the documentation and API of FsCheck 3.6.0 and FsCheck.Xunit 3.6.0. The examples were run with the .NET 8 SDK and FsCheck.Xunit 3.6.0, and I confirmed that the tests that pass and the tests that fail behave as described here. Some of the syntax differs from the documentation and samples for FsCheck 2.x, so if you use 2.x, check against the documentation for that version.

This is the fourth article in the series “Improving the Quality of Programs Written by AI.” The weaknesses of tests written by AI are covered in “Can You Trust Tests Written by AI?”, finding code that never runs in “Finding Untested Code with Code Coverage”, and finding code whose results are never checked in “Testing Your Tests with Mutation Testing”.

What Example-Based Tests Check

An example-based test checks whether the expected result comes out for the inputs you chose. It says nothing about the inputs you did not choose. Property-based testing hands the job of choosing inputs to a tool, and the person writing the test writes the properties the result must satisfy.

flowchart TB
        accTitle: Example-based tests versus property-based tests
        accDescr: An example-based test checks inputs and expected values chosen by the author; a property-based test has a tool generate many inputs, checks whether the written property holds, and if it does not, shrinks the counterexample and reports it.
        subgraph example ["Example-based test"]
            E1["The author chooses inputs"] --> E2["Compare with expected values"]
        end
        subgraph property ["Property-based test"]
            P1["The tool generates many inputs"] --> P2["Check whether the property holds"]
            P2 -->|"Does not hold"| P3["Shrink and report the counterexample"]
        end
        example ~~~ property

Figure 1: An example-based test checks only the inputs you chose; a property-based test checks properties against the inputs it generates.

The 2000 paper on QuickCheck, a testing tool for Haskell, describes QuickCheck as a tool in which you write the properties a function must satisfy and the tool checks them automatically against random inputs¹. FsCheck is likewise a tool that checks whether the properties you write hold for many randomly generated inputs, and shows a minimal counterexample when one fails².

A Splitting Function with a Bug

As the example, we use a function that splits an array into pages of a fixed number of items. Assume the code was written by AI; it has a mistake in how it calculates the number of pages.

public static class Paging
{
    public static List<T[]> Chunk<T>(IReadOnlyList<T> items, int size)
    {
        var pages = new List<T[]>();
        int pageCount = items.Count / size + 1; // one page too many when the count divides evenly
        for (int p = 0; p < pageCount; p++)
        {
            pages.Add(items.Skip(p * size).Take(size).ToArray());
        }
        return pages;
    }
}

items.Count / size + 1 gives the right number of pages when the item count is not divisible by size. When it is divisible (including when the count is 0), you get one page too many, with an empty page at the end.

flowchart TB
        accTitle: The page-count mistake in the splitting function
        accDescr: When the item count is not divisible by the page size the page count is correct; when it is divisible, or the count is 0, there is one page too many and an empty page is appended.
        A["Page count = count ÷ size + 1"] --> B["Is the count divisible by size?"]
        B -->|"Not divisible (e.g. 10 items, 3 per page)"| C["4 pages, correct"]
        B -->|"Divisible (e.g. 6 items, 0 items)"| D["One page too many, with an empty page"]

Figure 2: Only when the item count is divisible by the page size does an extra empty page appear.

Under the same assumption, we also write an example-based test. Splitting 10 items into pages of 3 works correctly, so this test passes.

public class ChunkExampleTests
{
    [Fact]
    public void Splits_ten_items_into_pages_of_three()
    {
        var pages = Paging.Chunk(Enumerable.Range(1, 10).ToArray(), 3);

        Assert.Equal(new[] { 3, 3, 3, 1 }, pages.Select(p => p.Length).ToArray());
    }
}

The input this test chose happened to be a count that does not divide evenly. Adding a divisible count as an example would expose the mistake, but whether it gets added depends on whether the author thinks of it.

In the diagram a solid line marks a relation that always holds and a dashed line marks a conditional one (the conditions are given per relation on the detail page). The full list of relations (5 in total, with evidence and certainty) and the definitions of the main concepts are collected on the knowledge map detail page (in Japanese). Data: JSON-LD / Turtle

How to Write Properties

A property is “something that should hold for any input.” The following forms are easy to use with business code.

flowchart TB
        accTitle: The main forms of properties
        accDescr: Four main forms of properties are shown: a round trip that returns to the original after converting and converting back, an invariant the output satisfies, comparison with a simple alternative implementation, and idempotence, where applying twice gives the same result.
        P["The main forms of properties"] --> R["Round trip (converting back restores the original)"]
        P --> I["Invariants on the output"]
        P --> O["Comparison with a simple alternative implementation"]
        P --> D["Idempotence (same result twice)"]

Figure 3: The four commonly used forms of properties are round trip, invariant, comparison with another implementation, and idempotence.

Round Trip

This is a property such as “encoding and then decoding gives back the original” or “writing and then reading gives back the original.” For the splitting function you can write “joining the pages back together restores the original array.”

Invariants on the Output

These are conditions the output should always satisfy. For the splitting function you can write “every page has at least 1 and at most size items” and “the number of pages is the item count divided by size, rounded up.”

Comparison with a Simple Alternative Implementation

This property compares the result with a different implementation that should return the same result, one that may be slow but is reliable. A mechanism that decides whether a test result is correct is called a test oracle³. In property-based testing, the properties you write make this decision automatically¹. Using an independently built alternative implementation as the oracle has long been known as a pseudo-oracle³. For the splitting function, you can compare the result with Enumerable.Chunk⁴, which was added in .NET 6.

Idempotence

This is the property that applying an operation twice gives the same result as applying it once. It is useful for things like normalizing strings or removing duplicates.

Mistakes a Round Trip Alone Will Not Find

If the properties you write are weak, the tests pass even when there is a mistake. The mistake in the splitting function only adds one empty page, so joining the pages back together still restores the original array. The round-trip property passes, while the invariant (every page has at least one item) and the comparison with the alternative implementation fail.

flowchart TB
        accTitle: Properties that do and do not find the mistake in the splitting function
        accDescr: Against the mistake that adds one empty page, the round-trip property that joining the pages restores the original passes, while the invariant that every page has at least one item and the comparison with Enumerable.Chunk fail and find the mistake.
        B["Mistake: one extra empty page"] --> R["Round trip: joining restores the original"]
        B --> I["Invariant: every page has at least one item"]
        B --> O["Comparison with Enumerable.Chunk"]
        R --> R2["Passes (does not find it)"]
        I --> I2["Fails (finds it)"]
        O --> O2["Fails (finds it)"]

Figure 4: The round-trip property alone cannot find the empty page; the invariant and the comparison with another implementation find it.

Do not stop at a single property; write several of different forms and combine them.

Writing Them with FsCheck and xUnit

In an xUnit v2 test project, install the FsCheck.Xunit package. FsCheck.Xunit 3.6.0 depends on FsCheck 3.6.0 and on xUnit packages at version 2.4.1 or later but earlier than 3.0⁵. If you use xUnit v3, install FsCheck.Xunit.v3 instead. The namespace is the same, FsCheck.Xunit⁶.

dotnet add package FsCheck.Xunit --version 3.6.0

When you put the [Property] attribute on a test method, FsCheck generates values for the method’s parameters and calls the method repeatedly⁶. If any input makes the return value false, the test fails.

using FsCheck;
using FsCheck.Fluent;
using FsCheck.Xunit;

public class ChunkProperties
{
    private static int ToSize(int rawSize) => Math.Abs(rawSize % 10) + 1; // map to 1–10

    [Property]
    public bool Concatenating_pages_restores_items(int[] items, int rawSize)
    {
        var pages = Paging.Chunk(items, ToSize(rawSize));
        return pages.SelectMany(page => page).SequenceEqual(items);
    }

    [Property]
    public bool Every_page_has_1_to_size_items(int[] items, int rawSize)
    {
        int size = ToSize(rawSize);
        return Paging.Chunk(items, size).All(page => page.Length >= 1 && page.Length <= size);
    }

    [Property]
    public bool Matches_Enumerable_Chunk(int[] items, int rawSize)
    {
        int size = ToSize(rawSize);
        var expected = items.Chunk(size).ToList();
        var actual = Paging.Chunk(items, size);
        return actual.Count == expected.Count
            && actual.Zip(expected).All(pair => pair.First.SequenceEqual(pair.Second));
    }
}

The int[] and int values are generated by FsCheck’s built-in generators. Generators for reference types such as strings do not produce null by default⁷. The page size is mapped from rawSize into the range 1 to 10. This way you can use the built-in generators, and the shrinking procedure explained later, as they are.

Of the three, the first one, the round-trip property, passes. The second and third fail when an array whose length divides evenly (including the empty array) is generated. When I actually ran them, the second and third both failed on the very first input, the empty array.

flowchart TB
        accTitle: How a test with the Property attribute runs
        accDescr: FsCheck generates an input and calls the test method; if the property holds it moves to the next input and succeeds once the set number of inputs has been reached; if it finds an input for which the property does not hold, it shrinks that input and then reports a failure.
        G["Generate an input"] --> C["Call the test method"]
        C --> J["Does the property hold?"]
        J -->|"Holds"| N["Has the set number been reached?"]
        N -->|"Not yet"| G
        N -->|"Reached"| OK["Success"]
        J -->|"Does not hold"| S["Shrink the input"]
        S --> F["Report as a failure"]

Figure 5: FsCheck keeps generating and checking inputs; when it finds an input for which the property does not hold, it shrinks it and then reports it.

You can change the number of inputs to try with MaxTest⁸. When a property succeeds with the default setting, FsCheck reports that it passed 100 tests⁹.

Shrinking Counterexamples

When FsCheck finds an input for which a property does not hold, it shrinks that input and looks for the smallest counterexample that still fails⁹. This process is called shrinking.

Randomly generated inputs may happen to be long arrays or large numbers. Reported as they are, they make it hard to see where the mistake is. A shrunk counterexample is small, which makes it easier to trace where the mistake happens. The mistake in the splitting function also occurs when the count is 0, so shrinking can lead to a small counterexample such as the empty array.

flowchart TB
        accTitle: How a counterexample is shrunk
        accDescr: From the failing input, smaller candidates are generated and the property is checked again; if a candidate still fails, it becomes the new counterexample and shrinking continues; when no candidate fails any more, the current one is reported as the minimal counterexample.
        A["The failing input"] --> B["Generate smaller candidates"]
        B --> C["Does a candidate also fail?"]
        C -->|"It fails"| D["Make that candidate the new counterexample"]
        D --> B
        C -->|"Every candidate passes"| E["Report as the minimal counterexample"]

Figure 6: As long as it keeps failing, the counterexample is replaced with smaller candidates.

The FsCheck documentation shows the failure report in the following form⁹. The first line shows after how many tests it failed, how many times it shrank, and two numbers in parentheses. “Original” is the input found first, and “Shrunk” is the input after shrinking.

Falsifiable, after 3 tests (2 shrinks) (3469625241648231580,10457760963846162097).
Last step was invoked with size of 4 and seed of (18049846384052835765,8010492880672050043).
Replay directly at failing step with (18049846384052835765,8010492880672050043,4).
Original:
[0; 0; 0; 1]
Shrunk:
[0; 1]
with exception:
System.Exception: Expected true, got false.

This is the F# example from the documentation, not output from this article’s splitting function.

Arbitrary: Generating Inputs

When the built-in generators cannot produce business values well, you write your own generator. In FsCheck, a generator that produces values combined with a procedure that shrinks them is called an Arbitrary⁷.

public readonly record struct PageSize(int Value);

public class PagingArbitraries
{
    public static Arbitrary<PageSize> PageSizes() =>
        Gen.Choose(1, 10).Select(n => new PageSize(n)).ToArbitrary();
}

public class ChunkPropertiesWithArbitrary
{
    [Property(Arbitrary = new[] { typeof(PagingArbitraries) })]
    public bool Every_page_has_1_to_size_items(int[] items, PageSize size) =>
        Paging.Chunk(items, size.Value).All(page => page.Length >= 1 && page.Length <= size.Value);
}

Gen.Choose(1, 10) generates integers from 1 to 10 inclusive¹⁰. Select turns them into the business type, and ToArbitrary turns that into an Arbitrary. When you pass a type to the Arbitrary property of [Property], the methods on that type that return an Arbitrary are used only for that test method⁶.

flowchart TB
        accTitle: What an Arbitrary contains and how it relates to shrinking
        accDescr: An Arbitrary combines a generator that produces values with a procedure that shrinks them; if it is built from a generator alone without a shrinking procedure, values of that type are not shrunk.
        A["Arbitrary"] --> G["Generator (produces values)"]
        A --> S["Shrinking procedure"]
        S -.-> N["Not shrunk unless one is supplied"]

Figure 7: An Arbitrary is a pair of a generator and a shrinking procedure; without the shrinking procedure, its values are not shrunk.

An Arbitrary built from a generator alone has no shrinking procedure. The FsCheck documentation explains that if you do not specify a shrinking procedure, no shrinking takes place⁷. The PageSize above cannot be shrunk, so a counterexample’s PageSize is reported with the value it had when found. ToArbitrary also has a parameter for passing a shrinking procedure¹⁰. Deriving values from built-in types, as in the previous section, is sometimes simpler.

Properties with Conditions

When you want to put a precondition on the inputs, you can write the condition with When¹¹.

[Property]
public Property Every_page_has_1_to_size_items_when_size_is_positive(int[] items, int size) =>
    Paging.Chunk(items, Math.Max(size, 1)).All(page => page.Length >= 1 && page.Length <= size)
        .When(size > 0);

Inputs that do not satisfy the condition are discarded¹¹. If the condition rarely holds, the number of discarded inputs reaches its limit and the test ends before the set number of inputs has been tried. In that case FsCheck reports how many tests it could run with inputs that satisfied the condition, as in “Arguments exhausted after 97 tests.”¹¹. Writing a generator that produces values satisfying the condition from the start secures more tests than filtering with a condition.

Replay: Reproducing a Failure

Inputs in property-based testing are generated randomly, so running the same test again does not necessarily generate the same inputs. To check whether you fixed a failure, you need to have it generate the same input as when it failed.

If you pass the two numbers in parentheses on the first line of the failure report to the Replay property of [Property], the same input is generated again. According to the FsCheck.Xunit description, you can paste the printed pair as is, and you can also specify the initial size as a third value¹⁰. The three numbers on the “Replay directly at failing step with” line of the report are in this form. The numbers below are taken from the sample output in the documentation.

[Property(Replay = "3469625241648231580,10457760963846162097")]
public bool Every_page_has_1_to_size_items(int[] items, int rawSize)
{
    int size = Math.Abs(rawSize % 10) + 1;
    return Paging.Chunk(items, size).All(page => page.Length >= 1 && page.Length <= size);
}

flowchart TB
        accTitle: Reproducing a failure with Replay
        accDescr: Paste the two numbers from the failure report into Replay to reproduce the failure with the same input, fix the implementation and confirm that it passes, and finally remove Replay to go back to normal random inputs.
        A["The two numbers in the failure report"] --> B["Paste them into Replay"]
        B --> C["Reproduce the failure with the same input"]
        C --> D["Fix the implementation and confirm it passes"]
        D --> E["Remove Replay"]

Figure 8: Pass the numbers from the failure report to Replay to reproduce it, and remove Replay once it is fixed.

Once you have confirmed the fix, remove Replay. If you leave it in, the test tries only the same input every time. Keep the shrunk counterexample separately as an example-based test, so you notice immediately if the same mistake comes back.

The Limits of Property-Based Testing

Property-based testing checks only the properties you write. As we saw with the round-trip property, if you write only weak properties, the tests pass even when there is a mistake. Which properties will find mistakes is something you have to work out by reading the specification and the code. The difficulty of providing test oracles automatically is studied in testing research as “the test oracle problem”³.

It tries only as many inputs as it generates. Success means “the property held for the inputs that were tried”; it does not prove correctness for every input. Complex generators take time per input, so also weigh the number of inputs to try against the total test time.

flowchart TB
        accTitle: The roles of the three methods covered in the series
        accDescr: Coverage finds code that never runs, mutation testing finds code that runs but whose results are not checked, and property-based testing checks properties against inputs the author did not think of.
        Q["Looking for weak spots in tests"] --> C["Coverage"]
        Q --> M["Mutation testing"]
        Q --> P["Property-based testing"]
        C --> C2["Code that never runs"]
        M --> M2["Code whose results are not checked"]
        P --> P2["Inputs nobody thought of"]

Figure 9: Coverage, mutation testing and property-based testing look for different weak spots in tests.

Coverage finds code that never runs, and mutation testing finds code that runs but whose results are not checked. Property-based testing tries inputs the author did not think of. Combining these three before accepting code and tests written by AI lets you examine, from separate directions, how much your tests actually check.

Related Articles

• Can You Trust Tests Written by AI?
• Finding Untested Code with Coverage
• Testing Your Tests with Mutation Testing
• Where to Draw the Line Between Unit Tests and Integration Tests
• Safely Modifying a Legacy Business App That Has No Tests

References

Related Articles

Recent articles sharing the same tags. Deepen your understanding with closely related topics.

Related Topics

These are topic pages close to this theme. Starting from the article, you can go on to related services and other articles.

Services Connected to This Topic

This article leads to the following service pages. Start from the entry point closest to your situation.

Frequently Asked Questions

Questions commonly asked at the time of consultation about the topic of this article.

What is property-based testing?

It is a testing method in which, instead of listing pairs of inputs and expected values, you write a property that should hold for any input and let a tool generate large numbers of inputs. When the tool finds an input for which the property does not hold, it shrinks that input as far as it can and reports it.

What should I use to write property-based tests in C#?

This article uses FsCheck. Install the FsCheck.Xunit package for xUnit v2 or FsCheck.Xunit.v3 for xUnit v3, and you can write tests by putting the [Property] attribute on test methods. The code in this article follows the FsCheck 3.6.0 API.

If a property-based test passes, does that mean the implementation is correct?

No. It checks only the properties you wrote, and the inputs are a randomly generated subset. If you wrote only weak properties, the tests pass even when there is a bug. Combine it with coverage and mutation testing to examine what your tests check from other directions as well.

Author Profile

Profile page for the article author.

Go Komura

Representative of KomuraSoft LLC

Focused on Windows software development, technical consulting, and bug investigation, with particular strength in projects where existing assets remain and in investigating failures whose cause is hard to see.