Can You Trust Tests Written by AI? ── How to Spot Tests That Pass Without Checking Anything

AI coding assistants will write tests along with the implementation. When every test passes, you may feel reassured, but a test checks only what its assertions say. If the assertions are weak, or the expected values were copied from the implementation, the tests pass even when the implementation is wrong.

This article is the first in the series “Improving the Quality of AI-Written Programs.” It sorts tests that pass without checking anything into five types and explains how to tell them apart in review. The same types appear in tests written by people, but when you hand the tests to AI as well, the implementation and the tests are produced by the same author on the same assumptions, so oversights can stack up. For each type, the article shows where to read and which tools can help.

The examples are written in C# and xUnit, but the ideas carry over to other languages and test frameworks. The code was written for this explanation and does not come from any particular project. The examples were run with the .NET 8 SDK and xUnit to confirm that the tests pass and fail exactly as described here. The article does not cover statistics such as how often AI writes bad tests.

The other three articles in the series are Finding untested code with code coverage, Checking your tests with mutation testing, and Trying inputs AI would not think of with property-based testing.

Assertions decide whether a test passes

Unit tests are commonly written in three parts: Arrange, Act, and Assert¹. What decides pass or fail is the last part, the assertion. However carefully the arrange and act steps are written, a test whose assertion checks nothing has no way to fail.

flowchart TB
        accTitle: The three parts of a unit test and how pass or fail is decided
        accDescr: Shows that after arrange and act, the assertion compares the result with the expected value to decide pass or fail, and that a test whose assertion checks nothing always passes.
        A["Arrange"] --> B["Act"]
        B --> C["Assert"]
        C --> D{"Does the result match the expected value?"}
        D -->|"It matches"| E["Pass"]
        D -->|"It does not match"| F["Fail"]
        C -.-> G["Always passes if nothing is checked"]

Figure 1: Pass or fail is decided by the assertions, and what the assertions do not check, the test does not check either.

An assertion compares a result with an expected value. In software testing research, this expected value, in other words the basis for judging “this is the correct result,” is called a test oracle. A test oracle is a procedure that distinguishes the correct behavior of the system under test from incorrect behavior². When you read a test, look at two things: what the assertion compares, and where the expected value it compares against came from.

Microsoft’s guidance on .NET testing also lists, among the properties of a good unit test, that it can determine pass or fail without human help (Self-Checking)¹. Since the assertions are what decide pass or fail automatically, the content of the assertions is the content of the test.

In the diagram a solid line marks a relation that always holds and a dashed line marks a conditional one (the conditions are given per relation on the detail page). The full list of relations (6 in total, with evidence and certainty) and the definitions of the main concepts are collected on the knowledge map detail page (in Japanese). Data: JSON-LD / Turtle

Five types of tests that pass without checking anything

From here on, we use the following small class as the example. It implements the rule “apply a 10% discount when the subtotal is 10,000 yen or more.”

public sealed class PriceCalculator
{
    public decimal Total(decimal subtotal)
    {
        if (subtotal >= 10000m)
        {
            return subtotal * 0.9m;
        }

        return subtotal;
    }
}

Against this class, here are five types of tests that pass without checking anything.

flowchart TB
        accTitle: Five types of tests that pass without checking anything
        accDescr: Shows four types that appear in the test code — weak assertions, expected values copied from the implementation, overused test doubles, and happy-path-only tests — and one type that appears in the change history, rewriting tests to match a bug.
        T["Tests that pass without checking anything"] --> G1["Types visible in the test code"]
        T --> G2["Types visible in the change history"]
        G1 --> A["Weak assertions"]
        G1 --> B["Expected values copied from the code"]
        G1 --> C["Overused test doubles"]
        G1 --> D["Happy path only"]
        G2 --> E["Rewritten to match a bug"]

Figure 2: The five types differ in where the weakness lies: how results are checked, where expected values come from, what was replaced, which inputs were tried, and how the test changed over time.

No assertions, or weak ones

The easiest type to recognize is a test with no assertion, or one whose assertion says almost nothing.

[Fact]
public void Total_LargeOrder_DoesNotThrow()
{
    var calculator = new PriceCalculator();

    calculator.Total(12000m);
}

[Fact]
public void Total_LargeOrder_ReturnsPositive()
{
    var calculator = new PriceCalculator();

    var actual = calculator.Total(12000m);

    Assert.True(actual > 0);
}

The first passes as long as no exception is thrown. The second passes whether the code forgets the discount and returns 12,000 yen or gets the rate wrong and returns 6,000 yen. Martin Fowler writes that because tests without assertions still execute the code, they can reach 100% coverage³. Being executed and working correctly are two different things.

In review, read the assertion lines before the test method names. If a test contains only conditions that a wrong result would also satisfy, such as Assert.NotNull, Assert.True(x > 0), or Assert.Equal(3, list.Count) comparing only a count, the test is not checking what its name claims.

flowchart TB
        accTitle: What a test with a weak assertion lets through
        accDescr: Shows that the correct implementation, an implementation that forgets the discount, and an implementation with the wrong discount rate all satisfy the weak assertion that the result is greater than zero, so all of them pass.
        W["Assert.True (result is greater than 0)"]
        R["Correct implementation: 10,800 yen"] --> W
        X["Forgot the discount: 12,000 yen"] --> W
        Y["Wrong discount rate: 6,000 yen"] --> W
        W --> P["All of them pass"]

Figure 3: A weak assertion cannot tell a correct implementation from a wrong one.

Expected values copied from the implementation

The next type has an assertion, but builds the expected value with the same calculation as the implementation.

[Fact]
public void Total_LargeOrder_AppliesDiscount()
{
    var calculator = new PriceCalculator();
    var subtotal = 10000m;
    var expected = subtotal >= 10000m ? subtotal * 0.9m : subtotal;

    var actual = calculator.Total(subtotal);

    Assert.Equal(expected, actual);
}

Suppose the specification actually said “discount when the subtotal exceeds 10,000 yen.” The >= in the implementation is a mistake, but the test builds its expected value with the same >=, so the two agree and the test passes. When the expected value is a copy of the implementation, the test carries the same mistake as the implementation.

Microsoft’s guidance recommends not writing logic such as if, for, or string building inside tests, because putting logic in a test greatly increases the chance of introducing a bug into the test itself¹. Write the expected value as the literal value the specification dictates for the input.

public static TheoryData<decimal, decimal> BoundaryCases => new()
{
    { 10000m, 10000m },
    { 10001m, 9000.9m },
};

[Theory]
[MemberData(nameof(BoundaryCases))]
public void Total_DiscountStartsAboveTenThousand(decimal subtotal, decimal expected)
{
    var calculator = new PriceCalculator();

    var actual = calculator.Total(subtotal);

    Assert.Equal(expected, actual);
}

Written this way, the mismatch between the specification’s “exceeds” and the implementation’s >= makes the 10,000-yen case fail. C# attribute arguments cannot hold decimal values, so the inputs and expected values are passed with TheoryData instead of InlineData.

flowchart TB
        accTitle: How the source of the expected value changes the outcome
        accDescr: Shows that an expected value built with the same calculation as the implementation makes the test share the implementation's mistake and pass, while an expected value written from the specification makes the test fail because of the implementation's mistake.
        S["Specification: discount above 10,000 yen"]
        I["Implementation: discount at 10,000 yen or more (wrong)"]
        I --> C1["Expected value built with the same calculation"]
        C1 --> P1["The mistakes agree, so it passes"]
        S --> C2["Expected value written from the specification"]
        C2 --> F1["Fails because of the implementation's mistake"]

Figure 4: Writing expected values from the specification lets the test catch mistakes in the implementation.

Overused test doubles

Tests sometimes replace dependencies such as databases and external services with stand-ins built for testing. These stand-ins are collectively called test doubles. They include stubs that return fixed values and mocks that record and verify how they were called⁴. With test doubles you can cut off slow or unstable dependencies and exercise only the part you want to check¹.

On the other hand, if you replace even the logic you meant to check with a stand-in, the test merely receives the value the stand-in returned.

public interface IPriceCalculator
{
    decimal Total(decimal subtotal);
}

public sealed class OrderService
{
    private readonly IPriceCalculator _calculator;

    public OrderService(IPriceCalculator calculator) => _calculator = calculator;

    public decimal Checkout(decimal subtotal) => _calculator.Total(subtotal);
}

public sealed class FixedPriceCalculator : IPriceCalculator
{
    private readonly decimal _result;

    public FixedPriceCalculator(decimal result) => _result = result;

    public decimal Total(decimal subtotal) => _result;
}

[Fact]
public void Checkout_LargeOrder_ReturnsDiscountedTotal()
{
    var service = new OrderService(new FixedPriceCalculator(10800m));

    var actual = service.Checkout(12000m);

    Assert.Equal(10800m, actual);
}

The test’s name says “returns the discounted total,” but the discount calculation never runs. 10,800 yen is the value the test handed to the stand-in, and OrderService simply returned it. This test confirms that OrderService returns the calculated result, but not that the discount is correct. If no other test checks the discount itself, the discount calculation is untested.

flowchart TB
        accTitle: What was replaced with a test double and what is checked
        accDescr: Shows that when the test replaces the discount calculation with a test double that returns a fixed value, it checks only that OrderService passes the value through, and the discount calculation never runs.
        T["Test"] --> O["OrderService (real)"]
        O --> D["Test double: just returns 10,800"]
        D -.-> N["The discount calculation never runs"]
        T --> A["Checks that 10,800 is returned"]

Figure 5: Logic replaced by a test double is not checked by that test.

In review, write down what was replaced with test doubles and what remains real. If the logic named in the test’s name sits on the test-double side, the test does not check that logic. Using test doubles is also covered in “Where to Draw the Line Between Unit Tests and Integration Tests.”

Only the happy path is tried

Even when the assertions and expected values are correct, the inputs tried may be only typical values. A test that tries only 12,000 yen against the PriceCalculator above tries none of 10,000 and 10,001 yen at the discount boundary, 0 yen, or negative values.

The mix-up between >= and > in the example above stays invisible unless you try the boundary values. Microsoft’s guidance says that well-named tests answer questions such as “what happens if I pass an empty string or null?”¹. Tests double as documentation of the specification, so if the handling of boundaries and invalid inputs is not written down in tests, nobody is checking that part of the specification.

flowchart TB
        accTitle: The branch a happy-path-only test never takes
        accDescr: Shows that a test trying only 12,000 yen takes only the discount branch, leaving the no-discount branch and the boundary values untried.
        IN["Input"] --> Q{"Is the subtotal 10,000 yen or more?"}
        Q -->|"Yes"| A["Apply the discount (passes at 12,000 yen)"]
        Q -->|"No"| B["No discount (not tried)"]
        Q -.-> E["The boundary 10,000 and 10,001 yen are not tried either"]

Figure 6: A test that uses only typical values does not try one of the branches or the boundary.

Branches that were never tried can be found with branch coverage. For example, in code with only two branches, a test that takes just one of them has 50% branch coverage⁵.

Rewritten to match a bug

The last type shows up in the change history. The implementation changed, a test failed, and someone rewrote the test’s expected value to match the new output so the test would pass.

If the specification changed and the expected value changed with it, rewriting the test is the right thing to do. But if the test failed because of a mistake in the implementation and the expected value was adjusted to match, the test now records the bug as correct behavior. When you ask an AI coding agent to “fix things until the tests pass,” fixing the implementation and fixing the expected value are, to the agent, both ways of making the tests pass.

sequenceDiagram
        accTitle: Rewriting a test to match a bug
        accDescr: Shows that a change to the implementation makes a test fail, the expected value is rewritten to the new output so the test passes, and as a result the bug is recorded as correct behavior.
        participant Dev as Implementation change
        participant Test as Test
        Dev->>Test: Run
        Test-->>Dev: Fails because the expected value differs
        Dev->>Test: Rewrite the expected value to the new output
        Test-->>Dev: Passes
        Note over Test: The wrong output stays as correct behavior

Figure 7: Matching a failing test’s expected value to the output records the bug in the test.

Tests that record the code’s current behavior as it is also have a legitimate use: characterization tests. They pin down the current behavior before you modify old code that has no tests (see “Safely Modifying a Legacy Business App That Has No Tests”). But that recording is done knowingly, on the understanding that “we do not know whether this is correct, but this is how it behaves now.” If, in a change to newly written code, expected values have been rewritten with no explanation, read those rewrites with suspicion.

In review, read the test diff together with the implementation diff. If an assertion’s expected value changed in the same change, check whether you can explain why the value changed as a change in the specification.

What to look at in review

The five types can be told apart by reading the test code. When reviewing tests written by AI, read in this order:

1. Read the assertion lines and check that they are not conditions a wrong result would also satisfy.
2. Check where the expected values came from. If they are computed inside the test, ask for them to be rewritten as concrete values dictated by the specification.
3. Write down what was replaced with test doubles and what remains real, and check that the logic named in the test’s name is on the real side.
4. Write down the inputs being tried, and check that boundaries, empty values, null, and invalid values are included.
5. Read the test diff together with the implementation diff, and if any expected values were rewritten, check the reason.

flowchart TB
        accTitle: The order for reviewing tests written by AI
        accDescr: Shows reading the assertions, the source of the expected values, the scope replaced with test doubles, the inputs being tried, and the test diff in that order, and asking for a rewrite if there is a problem.
        A["Read the assertion lines"] --> B["Check where the expected values come from"]
        B --> C["Write down what is a double and what is real"]
        C --> D["Write down the inputs being tried"]
        D --> E["Read the test diff"]
        E --> F{"Is there a problem?"}
        F -->|"Yes"| G["Ask for a rewrite"]
        F -->|"No"| H["Accept"]

Figure 8: Read from the assertions onward, and ask for a rewrite if there is a problem.

There are also things you can do on the side that asks the AI to do the work. The official Claude Code documentation notes that a report that “all tests passed” tells you almost nothing on its own, and recommends asking for the commands that were run and their output as evidence. When asking for a bug fix, it gives the example of first having the agent write a failing test that reproduces the bug, and only then having it fix the bug⁶. Seeing the test fail first lets you confirm that the test can detect the bug.

What tools can cover

Review relies on human eyes, so with many tests, things get missed. For four of the five types, tools give you mechanical clues for finding them.

Code coverage measures which lines and branches the tests executed⁵. It is well suited to finding logic that never runs because it was replaced with a test double, and branches that were never tried. But it only tells you whether code was executed, so it cannot show weak assertions. Microsoft’s guidance also says that a high coverage percentage is not an indicator of success and does not imply code quality¹. Fowler likewise writes that coverage is useful for finding untested parts of the code but of little use as a number that shows how good the tests are⁷.

Mutation testing deliberately breaks the code under test in small ways (for example, changing >= to >) and checks whether the tests fail. If the tests still pass after the code is broken, they are not checking that change⁸. On .NET you can use Stryker.NET⁹.

flowchart TB
        accTitle: What code coverage and mutation testing look at
        accDescr: Shows that code coverage looks at whether the tests executed the code, and mutation testing looks at whether, having executed it, the tests can detect a change.
        CODE["Code under test"] --> Q1{"Did the tests execute it?"}
        Q1 -->|"No"| U["Found by coverage"]
        Q1 -->|"Yes"| Q2{"Do the tests fail when it is broken?"}
        Q2 -->|"No"| W["Found by mutation testing"]
        Q2 -->|"Yes"| OK["That change is checked"]

Figure 9: Coverage finds code that is not executed; mutation testing finds code that is executed but not checked.

Property-based testing, instead of listing individual inputs and expected values, states a property that should hold for every input, and checks that property against many inputs generated by the tool. For .NET there is FsCheck, which shrinks a failing input down to a small counterexample¹⁰. A property such as “the discounted amount never exceeds the subtotal” can be written without copying the implementation’s calculation, so it counters the type that copies expected values from the implementation. A property is a pass/fail criterion that takes the place of an expected value, which is to say one form of test oracle².

A rewrite that matches a bug cannot be distinguished from a legitimate rewrite by looking at the test code alone. A tool cannot know whether the specification changed, so a person reads the test diff.

Related Articles

• Where to Draw the Line Between Unit Tests and Integration Tests
• Safely Modifying a Legacy Business App That Has No Tests — Characterization Testing and Refactoring in Practice
• Finding Untested Code with Coverage
• Testing Your Tests with Mutation Testing
• Property-Based Testing: Trying the Inputs Examples Never Reach

References

Related Articles

Recent articles sharing the same tags. Deepen your understanding with closely related topics.

Related Topics

These are topic pages close to this theme. Starting from the article, you can go on to related services and other articles.

Services Connected to This Topic

This article leads to the following service pages. Start from the entry point closest to your situation.

Frequently Asked Questions

Questions commonly asked at the time of consultation about the topic of this article.

If the AI says "all tests passed," can I just accept that?

That alone is not enough to decide. A test checks only what its assertions say, so tests with weak assertions, expected values copied from the implementation, or test doubles standing in for the real logic will pass even when the implementation is wrong. Ask for the commands that were run and their output, read the tests' assertions and where their expected values come from, and then decide.

Where should the expected values in a test come from?

From the specification or the business rules. If you compute the expected value inside the test with the same calculation as the implementation, the test inherits the implementation's mistake, the two agree, and the test passes. The basic approach is to write a concrete input and the output the specification dictates for it, as literal values.

If code coverage is high, are the tests good enough?

Not necessarily. Coverage measures which code the tests executed, not whether they checked the results. Even tests without assertions raise coverage. Use it as a tool for finding code that never runs, and investigate weak checks with techniques such as mutation testing.

Author Profile

Profile page for the article author.

Go Komura

Representative of KomuraSoft LLC

Focused on Windows software development, technical consulting, and bug investigation, with particular strength in projects where existing assets remain and in investigating failures whose cause is hard to see.