Testing Your Tests with Mutation Testing ── Finding Tests That Check Nothing with Stryker.NET

When you have AI write code and tests together, the tests may all pass and coverage may be high, yet the tests check nothing. These are tests that receive a result but never compare it, or only check that it is greater than 0. The first article in this series, “Can You Trust Tests Written by AI?,” covered these types of tests.

Mutation testing is a way to find such tests mechanically. It deliberately changes the production code slightly (for example, >= to >) and runs the tests. If a test fails, the tests noticed the change. If they all pass, the tests are not checking the behavior of that part.

This article uses Stryker.NET, a tool for C# and .NET, to explain the ideas behind mutation testing, setup and running, reading reports, handling surviving mutants, and keeping run times down. The names of commands and settings were checked against the official Stryker.NET documentation at the time of writing (November 2026). The code examples in the article are small examples for explanation. The discount example was actually run with Stryker.NET 5.0.0, confirming the results described here (run with the .NET 10 SDK; the project under test and the test project target .NET 8).

The series consists of these four articles:

1. Can You Trust Tests Written by AI?
2. Finding untested code with code coverage
3. Testing Your Tests with Mutation Testing (this article)
4. Trying inputs AI would not think of with property-based testing

How mutation testing works

The Stryker documentation explains mutation testing as follows. Bugs (mutants) are automatically inserted into the production code, and the tests are run for each mutant. If a test fails, that mutant is killed; if the tests pass, the mutant survived¹.

flowchart TB
        accTitle: How mutation testing flows
        accDescr: Shows creating mutants by slightly changing the production code, running the tests for each mutant, and the mutant becoming killed if a test fails or survived if all pass.
        A["Production code"] --> B["Create slightly changed mutants"]
        B --> C["Run the tests for each mutant"]
        C --> D["Test results"]
        D -->|"Any test fails"| E["killed (the change was noticed)"]
        D -->|"All pass"| F["survived (the change was not noticed)"]

Figure 1: The tests run for each mutant; if one fails the mutant is killed, and if all pass it survived.

How mutants are made is determined by fixed rewriting rules (mutators). The Stryker.NET documentation lists rewrites such as the following².

Each rewrite resembles a small mistake that could happen in real code: mixing up >= and > at a boundary, or typing the wrong operator. If the tests do not fail on such changes, they may not notice a real mistake of the same kind either.

Mutant states

Stryker sorts the results into the following states³.

Survived and No coverage are both mutants the tests did not notice, but they mean different things. With No coverage, the tests do not execute that location, so the first step is to add tests. With Survived, the tests execute that location but do not notice the difference in the result. What this article is mainly concerned with is Survived.

flowchart TB
        accTitle: How mutants end up in each state
        accDescr: Shows that a mutant becomes No coverage if the tests do not execute its location, Killed if they execute it and a test fails, Timeout if the run times out, and Survived if all tests pass.
        M["Mutant"] --> Q1["Do the tests execute its location?"]
        Q1 -->|"Not executed"| NC["No coverage (add tests)"]
        Q1 -->|"Executed"| Q2["Test results"]
        Q2 -->|"Failure"| K["Killed"]
        Q2 -->|"Timed out"| T["Timeout"]
        Q2 -->|"All pass"| S["Survived (not checked)"]

Figure 2: Locations the tests do not execute become No coverage; locations they execute without noticing the change become Survived.

Mutation score

The results are summarized in a single percentage called the mutation score. In Stryker’s definition, the number detected (Killed plus Timeout) is divided by the number of valid mutants (detected plus Survived plus No coverage) and multiplied by 100. Mutants that caused compile errors or runtime errors are not counted as valid³.

Mutation score = detected ÷ valid × 100
  detected = Killed + Timeout
  valid    = detected + Survived + No coverage

There is also a “mutation score based on covered code,” calculated only over the locations the tests executed; it divides the number detected by the number covered (detected plus Survived)³. Because it removes No coverage from the denominator, it shows how well the tests notice changes in the places they do execute.

In the diagram a solid line marks a relation that always holds and a dashed line marks a conditional one (the conditions are given per relation on the detail page). The full list of relations (6 in total, with evidence and certainty) and the definitions of the main concepts are collected on the knowledge map detail page (in Japanese). Data: JSON-LD / Turtle

“Tests that check nothing,” which coverage cannot see

Code coverage shows which lines and branches were executed when the tests ran (covered in the second article in the series). The Stryker documentation also says that coverage does not tell you everything about how effective your tests are¹. A line that was executed is not necessarily a line whose correctness the tests checked.

flowchart TB
        accTitle: The questions coverage and mutation testing answer
        accDescr: Shows that code coverage answers whether the tests executed a line, and mutation testing answers whether the tests fail when that line is changed.
        L["One line of production code"] --> C1["Coverage asks: did the tests execute it?"]
        L --> M1["Mutation testing asks: do the tests fail if it changes?"]
        C1 -.-> C2["Green even if it was only executed"]
        M1 -.-> M2["Survives if the result is not checked"]

Figure 3: Coverage asks whether code was executed; mutation testing asks whether it was checked.

Let us look at a small example: a method that gives a 10% discount when the purchase amount is 10,000 yen or more.

public static class Discount
{
    // Give a 10% discount when the purchase amount is 10,000 yen or more
    public static int Apply(int amount)
    {
        if (amount >= 10_000)
        {
            return amount * 9 / 10;
        }

        return amount;
    }
}

Suppose it has the following test (written with xUnit).

public class DiscountTests
{
    [Fact]
    public void Apply_ReturnsPositiveValue()
    {
        var result = Discount.Apply(20_000);

        Assert.True(result > 0);
    }
}

This test passes and executes the discount line too. But all it checks is that “the result is greater than 0.” Consider the mutants the rewriting rules above would create, applied to this test.

• Changing >= to >: 20,000 yen is still discounted to 18,000 yen, which is greater than 0, so the test passes.
• Changing >= to <: 20,000 yen is not discounted and stays at 20,000 yen, but it is greater than 0, so the test passes.
• Changing * to /: 20000 / 9 / 10 gives 222, but it is greater than 0, so the test passes.
• Changing / to *: 20000 * 9 * 10 gives 1,800,000, but it is greater than 0, so the test passes.

This test misses every one of these changes. When actually run with Stryker.NET 5.0.0, in addition to these four, a mutant negating the whole condition (!(amount >= 10_000)) was also created, and all five Survived, for a score of 0%. A mutant that removes the entire body of the if was also created, but Stryker decided to exclude it from testing (Ignored). Because the discount line is executed, a coverage report shows this line as executed.

If you fix the test as follows, every one of the same mutants can be killed.

public class DiscountTests
{
    [Theory]
    [InlineData(9_999, 9_999)]   // One yen below the boundary: no discount
    [InlineData(10_000, 9_000)]  // Exactly at the boundary: discount
    [InlineData(20_000, 18_000)]
    public void Apply_DiscountsTenPercentFrom10000Yen(int amount, int expected)
    {
        Assert.Equal(expected, Discount.Apply(amount));
    }
}

• Changing >= to >: 10,000 yen is not discounted and stays at 10,000 yen, which does not match the expected 9,000 yen, so the test fails.
• Changing >= to <: 9,999 yen is discounted to 8,999 yen, which does not match the expected 9,999 yen, so the test fails.
• The mutants changing * to / and / to * fail because the result for 20,000 yen is not 18,000 yen.

Running with the fixed test, all five mutants, including the one negating the whole condition, were Killed, for a score of 100%. The differences are that the results are compared with expected values, and that inputs on both sides of the boundary are included. With these two, the test can notice boundary mix-ups and calculation errors.

flowchart TB
        accTitle: The difference between the weak test and the fixed test
        accDescr: Shows that with a test that only checks that the result is greater than 0, the mutants survive, while with a test that compares against expected values using inputs on both sides of the boundary, the mutants are killed.
        T1["Test that only checks the result is above 0"] --> R1["Four mutants survive"]
        T2["Test comparing expected values on both sides of the boundary"] --> R2["Four mutants are killed"]
        R1 -.-> N1["Looks executed in coverage"]

Figure 4: Comparing results with expected values and including both sides of the boundary kills the same mutants.

When AI writes tests, they can end up like the first example, with checks that hold for almost any input: “no exception is thrown,” “not null,” “greater than 0.” Mutation testing points to such tests as Survived mutants.

Setting up and running Stryker.NET

Installation

Following the official documentation, Stryker.NET is installed as a .NET tool. If your team should use the same version, creating a tool manifest in the repository and installing it there works well⁴.

dotnet new tool-manifest

dotnet tool install dotnet-stryker

Other developers can install the same tool with the following command after getting the repository.

dotnet tool restore

According to the official documentation at the time of writing, running Stryker.NET requires the .NET 10 runtime or later. The application under test does not need to use .NET 10; having the runtime installed is enough. The Stryker.NET 5.0.0 package is built only for .NET 10, and in an environment with only the .NET 8 SDK, dotnet tool install dotnet-stryker failed with an error saying DotnetToolSettings.xml could not be found. For .NET Framework projects, there are further requirements, such as NuGet being on the PATH⁴.

Running

Move to the unit test project’s folder and run the following command⁴.

dotnet stryker

Settings can be written in stryker-config.json placed in the folder where you run it. The root key is stryker-config⁵. For example, specifying the project under test, the files to mutate, and the reports to output looks like this.

{
  "stryker-config": {
    "project": "Shop.Core.csproj",
    "mutate": [
      "**/*.cs",
      "!**/*.Generated.cs"
    ],
    "reporters": ["progress", "html"]
  }
}

In project, write the file name of the production project that the test project references. The rule is to write a name, not a path. mutate takes file globs, and those prefixed with ! are excluded⁵.

flowchart TB
        accTitle: From setting up Stryker.NET to running it
        accDescr: Shows creating a tool manifest, installing dotnet-stryker, running dotnet stryker in the unit test project's folder, and reading the HTML report.
        A["Create a tool manifest"] --> B["Install dotnet-stryker"]
        B --> C["Move to the unit test project's folder"]
        C --> D["Run dotnet stryker"]
        D --> E["Read the HTML report"]
        F["stryker-config.json"] -.-> D

Figure 5: Install the tool, run it in the unit test project’s folder, and read the report.

Reading the report

The default reporters are html and progress. By default the html report is written under the name mutation-report⁵. If you run with --open-report, the report opens when the run finishes⁵.

The report lets you check the state of each mutant. Read it in this order:

1. Where there are No coverage mutants, the tests do not execute that logic. Start by considering whether to add tests.
2. For Survived mutants, read one by one how the code was rewritten and why the tests still passed. Use the criteria in the next section to decide whether to fix the tests or leave them as they are.

How to handle surviving mutants

Survived mutants come in two kinds. One kind is where the tests do not check behavior they should check. The other kind behaves the same as the original code and cannot be killed by any test. The latter are called equivalent mutants⁶.

In the following example, changing >= to > does not change the result.

public static int Max(int a, int b) => a >= b ? a : b;

When a and b are equal, the original code returns a and the rewritten code returns b. But the two are the same value, so the returned value does not change. No input can tell them apart, so this mutant is an equivalent mutant.

flowchart TB
        accTitle: Deciding what to do with a surviving mutant
        accDescr: Shows that for a surviving mutant, if there is an input whose result differs from the original code, you add or fix a test, and if no input changes the result, it is an equivalent mutant and you either rewrite the code or accept that the score falls short.
        S["Survived mutant"] --> Q["Is there an input that changes the result?"]
        Q -->|"Yes"| A["Add a test comparing expected values for that input"]
        Q -->|"No"| E["Equivalent mutant"]
        E --> E1["Rewrite the code"]
        E --> E2["Accept that the score falls short of 100%"]

Figure 6: If an input changes the result, add a test; if not, treat it as an equivalent mutant.

The criterion is “is there an input for which the original code and the rewritten code give different results?”

• If an input changes the result, add that input to the tests and compare against the expected value. The discount example in the previous section is this case. Think first about the inputs that AI-written tests tend to miss: boundary values, zero or empty inputs, negative values.
• If no input changes the result, it is an equivalent mutant. The Stryker documentation says that the only way to find equivalent mutants is to investigate manually, which takes time. As responses, it lists rewriting the code so that equivalent mutants are not produced, or accepting that the score will not reach 100%⁶.

Because equivalent mutants exist, making the score itself the goal means spending time on mutants that cannot be killed. Use the score as a prompt to read the surviving mutants, and make the judgment one mutant at a time.

Keeping run times down

Mutation testing runs the tests for each mutant, so it takes longer than running the tests normally. Rather than targeting the whole project every time, narrow the scope.

• With since, only code changed since a specified branch or commit is targeted. Only mutants in changed code get results in the report; the rest have no result. If you change a file in the test project, every mutant the tests in that file execute is treated as changed⁵. On the command line, specify it like --since:main; in the settings file, write "since": { "target": "main" }.
• Use mutate globs to target only files that contain logic and exclude generated code and the like⁵.
• concurrency changes how many runs happen at the same time. The default is half the number of logical processors⁵.

flowchart TB
        accTitle: Ways to narrow what Stryker.NET runs
        accDescr: Shows narrowing to changed code with since and to target files with mutate, and building it into PR checks by using the fact that falling below the break threshold ends the run with a nonzero exit code.
        P["Code changed in the PR"] --> S1["Target only changes with since"]
        S1 --> S2["Narrow the files with mutate"]
        S2 --> R["Check the score"]
        R -->|"Below break"| X["Ends with a nonzero exit code"]
        R -->|"Above break"| OK["Continue as is"]

Figure 7: Run on narrowed changes and files, and treat a score below break as a failure.

To build it into PR checks, use thresholds. The defaults are high 80, low 60, and break 0. The official documentation explains that when the score falls below break, Stryker exits with exit code 1⁵. When tried with Stryker.NET 5.0.0, the exit code was 2. Both are nonzero, so CI can treat the run as a failure. If you start with break set low and raise it after reading surviving mutants has become a habit, you can avoid builds being blocked over and over by equivalent mutants.

Limits of mutation testing

What mutation testing tells you is “whether the tests fail for the fixed set of rewrites.” It cannot tell you the following:

• Whether the tests would notice kinds of mistakes outside the rewriting rules (misreading the specification, missing features, errors in how modules combine).
• Whether the expected values themselves are correct. If the code and the expected values both disagree with the specification in the same way, the tests pass and the mutants are killed.

Also, because of equivalent mutants the score may not reach 100%, and runs take time. Use the tools in a division of labor: find code that is not executed with coverage, find code that is not checked with mutation testing, and have people check the expected values against the specification. Trying a wide range of inputs is covered in the fourth article in the series, “Trying inputs AI would not think of with property-based testing.”

Related Articles

• Can You Trust Tests Written by AI?
• Finding Untested Code with Coverage
• Property-Based Testing: Trying the Inputs Examples Never Reach
• Where to Draw the Line Between Unit Tests and Integration Tests
• Safely Modifying a Legacy Business App That Has No Tests

References

Related Articles

Recent articles sharing the same tags. Deepen your understanding with closely related topics.

Related Topics

These are topic pages close to this theme. Starting from the article, you can go on to related services and other articles.

Services Connected to This Topic

This article leads to the following service pages. Start from the entry point closest to your situation.

Frequently Asked Questions

Questions commonly asked at the time of consultation about the topic of this article.

What is mutation testing?

It is a technique that automatically creates slightly changed versions of the production code (mutants), runs the tests against each, and checks whether they fail. If a test fails, the mutant is "killed"; if all tests pass, it "survived." Surviving mutants point to places where the tests do not check that change.

If code coverage is high, do we still need mutation testing?

Coverage shows only whether the tests executed the code, not whether they checked the results. Even tests that do not check results raise coverage. Mutation testing finds out whether the tests check the code by seeing whether they fail when the code is changed.

How do I run Stryker.NET?

Install dotnet-stryker with dotnet tool install, then run dotnet stryker in the unit test project's folder. Settings can go in stryker-config.json. According to the official documentation at the time of writing, running it requires the .NET 10 runtime or later, but the target application does not need to use .NET 10.

Should we aim for a 100% mutation score?

You do not need to. Equivalent mutants, which behave the same as the original code, cannot be killed by any test, so the score may never reach 100%. Rather than raising the number, use it to read the surviving mutants one by one and judge whether any behavior that should be checked is missing.

Author Profile

Profile page for the article author.

Go Komura

Representative of KomuraSoft LLC

Focused on Windows software development, technical consulting, and bug investigation, with particular strength in projects where existing assets remain and in investigating failures whose cause is hard to see.