Mutation Testing, Explained: How to Know If Your Tests Can Actually Fail

Here is a test. It passes. The function it tests is wrong. Can you see why before you scroll?

def apply_discount(price, pct):
    return price * (1 - pct) / 100

def test_discount():
    price, pct = 200, 15
    expected = price * (1 - pct) / 100
    assert apply_discount(price, pct) == expected

The formula should be price * (1 - pct / 100). For 200 at 15% off, the right answer is 170. This function returns -28, and so does the test's expected value, because the test was written by copying the code's formula. Both sides are wrong together, so they agree, and the suite is green.

That test executes every line of apply_discount. Your coverage report counts it. It cannot fail for the bug it exists to catch.

This post is about that gap: coverage tells you code ran; mutation testing tells you whether a test would notice if the code were wrong. And you can do the second one by hand in fifteen minutes before you ever install a tool.

Why coverage can't see this

Line or branch coverage records one event: a test executed this code. It does not record whether any assertion depended on what the code did.

The best-known evidence is Inozemtseva and Holmes, Coverage Is Not Strongly Correlated with Test Suite Effectiveness (ICSE 2014). They generated 31,000 test suites for five Java systems of up to 724,000 lines and used seeded faults (mutants) to measure how many bugs each suite caught. Coverage did correlate with effectiveness, but much of that came from suite size. With the number of tests held constant, the correlation dropped to low-to-moderate.

Read that as: low coverage is a real warning, high coverage is not proof.

Coverage tells you the code ran. It can't tell you anything was watching.

Five tests that can't fail (with code)

1. The no-assert test

def test_generate_invoice():
    generate_invoice(order)   # passes as long as nothing throws

It lifts coverage and guards nothing. Every line it touches is "covered" and unprotected.

2. The mirror test

The opening example. The expected value is recomputed with the implementation's formula. Fix: assert a literal you worked out yourself.

def test_discount():
    assert apply_discount(200, 15) == 170

3. The mock echo

def test_get_active_user(mocker):
    repo = mocker.Mock()
    repo.find.return_value = User(id=1, name="Ada", active=True)
    assert UserService(repo).get_active_user(1).name == "Ada"

The service is supposed to hide deactivated users. This test never gives it one. Ask "what would make this go red?" and the only honest answer is "changing the mock."

4. The rubber-stamp snapshot

A snapshot records today's output and someone presses "update" without reading the diff. If today's output had a bug, the snapshot now defends it, and the eventual fix shows up as a failing test.

5. The shape check

assert result is not None
assert len(items) > 0
assert response.status_code == 200

Right shape, any values. A total that's off by a factor of ten passes all three.

Bonus: the async test that ends early. In Jest, this passes even when the name is wrong, because the test returns before the promise resolves:

test("loads the user", () => {
  fetchUser(1).then((user) => {
    expect(user.name).toBe("Ada");
  });
});

Return the promise or use async/await so the runner actually waits for the assertion.

The Sabotage Test: manual mutation testing in 15 minutes

Pick one function you changed this week. Break it on purpose, one change at a time, and run only the tests that cover it.

1. Flip a comparison: > to >=, == to !=.
2. Shift a boundary: 50 to 49 or 51, < n to <= n.
3. Delete a line: especially a side effect (save, emit, cache invalidation).
4. Return a default: 0, [], None, True.
5. Swallow an error: wrap the body in a try that discards the exception.

Red means a test noticed: revert and try the next break. Still green means a silent survivor: a bug your suite would ship.

Walkthrough. The spec says orders of 50 or more ship free:

def shipping_fee(order_total):
    if order_total > 50:
        return 0
    return 5

def test_shipping_fee():
    assert shipping_fee(80) == 0
    assert shipping_fee(20) == 5

100% line and branch coverage. Now apply break #1 and change > to >=. Both tests still pass. The survivor is the bug: the suite cannot tell the > program from the >= program, and only one matches the spec. One boundary assertion kills the survivor and catches the real defect:

    assert shipping_fee(50) == 0   # fails on the current code, as it should

When a break survives, your tests can't tell two programs apart. Only one of them is right.

Two habits make it stick:

• Revert-red for every bug fix. Write the regression test, revert the fix, watch it fail, restore the fix. If it passes on the old code, it isn't testing the bug.
• The literal rule. Expected values are literals from the spec or your own working, never recomputed with the code's logic.

Automating it: mutation testing tools

A mutation testing tool does the Sabotage Test for you. It generates small changes (mutants), runs the relevant tests against each, and reports the survivors.

Two costs to plan for: it's slow (the relevant tests run once per mutant), and it's noisy (equivalent mutants that don't change behaviour, plus mutants in code like log messages that nobody needs to test). So don't run it over the whole repo and chase a percentage. Most tools have an incremental or diff-only mode; check your tool's docs and run it on changed code in CI.

How Google and Meta run it at scale

Google. Practical Mutation Testing at Scale (Petrović, Ivanković, Fraser, Just, 2021) describes mutants surfaced as code-review findings, only on changed lines, at most one per line, with "arid" code such as logging suppressed. The median number of mutants per change went from 820 to 77 with the one-per-line cap, then to 7 with suppression. Across more than 24,000 developers, about 82% of mutants that received feedback were rated productive. The lesson is the scoping: small, diff-only, shown where the merge decision happens.

Meta. Mutation-Guided LLM-based Test Generation at Meta (2025) describes ACH: generate mutants no existing test catches, then have an LLM write a test that catches each. Engineers accepted 73% of the tests shown to them, and 277 of the 571 tests produced would have been discarded by a rule that only keeps tests adding line coverage. Nearly half the useful tests were invisible to coverage.

Getting honest tests out of an AI assistant

If you paste a function and ask for tests, the model's only source of truth is the implementation, so it tends to assert what the code does. Konstantinou, Degiovanni and Papadakis (2024) found GPT-3.5 prone to writing oracles that capture actual rather than expected behaviour across 24 Java projects; its accuracy at recognising correct assertions dropped by roughly 8 to 9 points on buggy code, and generic names cut accuracy by up to 16 points. One older model, one language: a direction, not a law. It's the same lesson as why agreement isn't proof with temperature 0: two things agreeing tells you nothing if they share a source.

A prompt that works with the mechanism instead of against it:

Here is the requirement (not the implementation):
<paste the spec or ticket>

Write unit tests that would FAIL if an implementation got this wrong.
- Use literal expected values worked out from the requirement.
- Include every boundary the requirement mentions, at, just below and just above.
- Above each test, name the specific bug it would catch.
Do not look at or infer from any existing implementation.

"Name the bug it would catch" is the Sabotage Test in advance: a test that can't name its bug probably can't catch one. Then run the real Sabotage Test anyway. The same discipline applies to the code itself; see how to review AI-generated code.

For coding agents it matters twice: the loop stops when tests go green, so a suite that can't fail approves every attempt. More on that in how to actually work with AI coding agents.

Key Takeaways

• Coverage measures execution, not verification. High coverage is not evidence a suite catches bugs.
• A test you've never seen fail is untested. Watch every new test go red once.
• Five shapes can't fail: no-assert, mirror, mock echo, rubber-stamp snapshot, shape check (plus the async test that ends early).
• The Sabotage Test (flip, shift, delete, default, swallow) finds silent survivors in 15 minutes without tools.
• Scope mutation tools to the diff and surface results in code review, as Google does.
• Give AI the requirement, not the code, and ask each test to name the bug it catches.

FAQ

What is mutation testing in simple terms?

Mutation testing checks your tests by making small deliberate bugs in your code (mutants), such as flipping > to >=, and running your tests against each one. If a test fails, the mutant is "killed." If every test still passes, the mutant "survived," which means your suite wouldn't notice that bug in real life.

Is 100% code coverage enough?

No. Coverage only records that a test executed a line, not that any assertion depended on it. A test with no assertions, or one that computes its expected value with the same logic as the code, gives full coverage and catches nothing. Research by Inozemtseva and Holmes (ICSE 2014) found only a low-to-moderate link between coverage and bug-catching once suite size is controlled.

What is a good mutation score?

There's no universal target, and chasing 100% is usually a mistake because some mutants are equivalent (they don't change behaviour) and some sit in code not worth testing, like log messages. A more useful rule: on code you changed, every surviving mutant should either get a test or a one-line reason it doesn't matter.

Is mutation testing too slow for CI?

Over a whole codebase, often yes. Scoped to changed lines it's practical: Google's approach surfaced a median of about 7 mutants per code review by limiting mutants to changed lines, one per line, and suppressing low-value code. Most tools offer an incremental or diff-only mode.

Why do AI-generated tests miss bugs?

When a model writes tests from the implementation, it tends to assert what the code currently does, including its bugs. A 2024 study found GPT-3.5 prone to capturing actual rather than expected behaviour. Give the model the requirement instead of the code, ask for literal expected values and boundaries, and verify each test fails against a deliberately broken version.

How do I test my tests without installing a tool?

Run the Sabotage Test: pick one function, make one deliberate break (flip a comparison, shift a boundary, delete a line, return a default, or swallow an error), run its tests, and see whether anything turns red. Any break that stays green is a bug your suite would ship. Write the test that kills it.

Related reading

• How to Review AI-Generated Code: the same skepticism, applied to the code instead of the tests.
• How to Actually Work With AI Coding Agents: why your tests are now your agent's judgment.
• Which Developer Skills AI Replaces Next: Sort by the Cost of the Check: owning a trustworthy check is becoming the valuable part of the job.
• Temperature 0 Won't Fix Your AI. It Just Hides the Doubt.: why agreement from a shared source isn't evidence.

The bottom line

Green is a claim. Red is evidence. Coverage can tell you which code your tests visited, but only a failure on broken code tells you a test was watching. Break something on purpose this week and see who notices.

A test you have never seen fail is a test you have never tested.

Adam Jaber is a software engineer who writes Simply Explained: complex topics, made simple. No jargon, no hype.