The tests passed. The feature was broken for every user.

I shipped a feature with five passing tests. It was broken for every single user, immediately, in a way that made the product worse than before I touched it. The tests were not wrong. They were answering a different question than the one that mattered.

The feature

My campaign tool has a research component that suggests products worth promoting. For each one it prints a commission estimate in bold green type and a trend statistic — something like "search volume up 180% year over year" — as flat, unqualified fact.

Both numbers are model output. The system does run live web research and gets back real sources, but nothing tied any specific number to any specific source. A figure the research fully supported and a figure the model invented rendered identically. Someone could spend money against either.

So I built provenance: ask the model to cite which retrieved sources back each claim, filter those citations against the URLs research actually returned, and mark anything unsupported as an unverified estimate. I wrote five tests. A citation that matches a retrieved source survives. One that does not is dropped. A product whose citations all fail is marked unsourced. Missing citations do not crash. All green. Shipped.

What actually happened

Later I ran the real endpoint against the real model, because a rule I try to hold is that a feature is not verified until it has been exercised against live data. Research retrieved five sources. All six products came back unsourced.

Every card in production now read "Unverified estimate." Not because the numbers were bad, but because the filter dropped everything. I had replaced an unlabelled figure with a figure labelled untrustworthy, across the board. That is worse. An unlabelled number invites you to check it. A number the system itself flags as unverified trains you to ignore the flag.

The bug was in the prompt

The grounding block sent to the model contained the research text — a synthesised summary of what the sources said. It did not contain the source URLs.

So the instruction was, in effect: cite verbatim from a list you cannot see. The model did the only reasonable thing and returned nothing, or returned plausible-looking URLs from memory that my filter correctly rejected. The filter worked perfectly. It was filtering against an empty intersection.

Why the tests could not have caught it

This is the part worth internalising. My tests all had this shape:

mockModel.returns({ sourceUrls: ["https://example.com/report"] })
mockResearch.returns({ sources: [{ url: "https://example.com/report" }] })
// assert the citation survives

Every test handed the endpoint citations and asserted what it did with them. The defect was that in production the model produced no usable citations, because of what the prompt did not contain. No amount of testing the filter can surface a bug in the input to the filter, when the test supplies the input.

The mock encoded my assumption — that the model would return matching URLs — and then confirmed it. That is the failure mode of mocking: a mock is a statement of what you believe the world does, dressed as a test of what your code does.

The test I added afterwards

The property I was missing was not about the filter at all. It was about the prompt:

it("shows the model the retrieved URLs, or it can never cite one", ...)
// assert every retrieved source URL appears in the text sent to the model

That test would have failed on the original implementation. It asserts the one precondition that makes everything downstream reachable. It is worth asking, of any LLM feature: what must be true of the prompt for this feature to be possible at all? — and testing that, separately from what you do with the response.

Two things I took from it

Mocked tests verify plumbing, not behaviour. For anything involving a model, they tell you the code handles a response correctly. They tell you nothing about whether that response will ever occur. Those are different claims and it is easy to feel covered by the first while needing the second.

Run it against the real thing before believing it. One live call took under a minute and found a defect that a full green suite had certified as working. If a feature depends on a model's behaviour, budget for exercising it against the model — not as a final smoke test, but as the thing that decides whether you shipped what you think you shipped.

There is a smaller detail I have thought about since. When the fix landed, five of six products cited correctly. The sixth cited flexoffers.com/... where the retrieved URL was www.flexoffers.com/... — a missing subdomain. My filter dropped it, correctly. A human reviewer would have glanced at that and waved it through. That is the case for matching citations mechanically rather than eyeballing them: the near-misses are exactly what a person's eye smooths over.