The mutation test that proved nothing

When I write a test after the code — which happens, whatever the ideal says — I do not trust it until I have watched it fail. A test written against working code passes immediately, and a passing test that has never failed has demonstrated nothing. It might be asserting something trivially true. It might be pointed at the wrong thing entirely.

So I break the code deliberately and check the test catches it. Twice now, that check itself has been worthless, in two different ways.

The first failure: the mutation broke compilation

I had written a test asserting that a legally required disclosure always renders next to a generated affiliate link. Displaying it is a condition of the program — omit it and you can lose the account and the earnings — so I wanted the test to be real, not decorative.

The test passed. To verify it meant something, I deleted the block of markup that renders the disclosure and re-ran.

Test Files  1 failed
     Tests  no tests

"No tests." Not a failing assertion — nothing ran at all. My crude deletion had left invalid markup, the file no longer compiled, and the runner could not load it.

That is a red result, and if I had been moving quickly I might have read "red, good, the test catches it" and moved on. But it proves nothing about the test. It proves the file was broken. A test that never executed cannot demonstrate that it detects anything.

The fix: mutate a value, not a structure

The second attempt changed the disclosure constant to an empty string:

- disclosure: "As an Amazon Associate I earn from qualifying purchases.",
+ disclosure: "",

That compiles cleanly. And the result was what I actually wanted to see:

× carries the exact disclosure the program requires
× shows the disclosure whenever it shows a link
  Tests  2 failed | 20 passed

Two failures, both specific, both naming the property I care about — and eighteen other tests still passing, which tells me the mutation was targeted rather than a blast radius.

The general rule: mutate values, not syntax. Changing a constant, flipping a comparison, swapping a return value — these keep the program valid and isolate exactly one behaviour. Deleting a block tests your build.

The second failure: the mutation landed somewhere else

The earlier version of this mistake was subtler and I like it more, because nothing about the output looked wrong.

I wanted to prove a test would catch a dangerous reordering — a delete happening before validation instead of after. I applied the mutation by searching for an anchor string and patching around it. The suite ran. Everything passed.

I nearly concluded the test was worthless. In fact the anchor string appeared many times across a seven-thousand-line file, and my patch had landed in a different function than the one under test. The mutation was real, it compiled, it ran — and it was in the wrong place. The test was fine. My verification was not.

The checklist I use now

1. Confirm the mutation landed. Diff the file. If you patched by string match, verify the match was unique — that one was expensive to learn.
2. Confirm the code still compiles and the suite still runs. "No tests" and "0 passed" are not failures in the sense you need.
3. Confirm the right tests failed, by name. Not "some things went red." The specific assertions covering the specific property.
4. Confirm unrelated tests still pass. If everything goes red, you broke something structural and learned nothing about the one behaviour.
5. Revert, and re-run to green before you commit anything.

Why bother

This is maybe two minutes of work per test that matters, and I do not do it for every test. I do it when a test is the only thing standing between me and a consequence I cannot undo — a legal disclosure vanishing from a page, a delete running against unvalidated data.

For those, the question is not "do I have coverage." It is "have I seen this test catch the thing it exists to catch." Those are very different claims, and only one of them survives contact with a real failure.