I build an instrument that measures whether AI coding agents stay inside their brief. This morning a user pointed it at a folder of working documents, and it told them this:
Specification fidelity 0 of 12 delivered ✗ finalise the contract and Ouaknine notes to paste-ready not found ✗ the 'why this role' screening answer for the service desk not found ✗ add real, attributed testimonials to the CV not found ✗ rebuild the service-desk CV proof-led, not claim-led not found …eight more
None of these is a software feature. Nobody had asked for them in that folder.
The reading is meaningless, and it is stated with total confidence.
How it got there
When nobody supplies a specification, the instrument derives a provisional one from whatever in the project states intent. It tries the strongest sources first: a requirements document, then the README, then package metadata. Last, and weakest, the subjects of recent feat: commits.
That last source carries its own warning in the code — a commit says what was done, which is only evidence of what was asked for. The warning was accurate and insufficient. This was a documents repository. Its commits were written feat: <errand>. Discovery harvested twelve errands and handed them over as a specification.
The analyser then did its job perfectly. It looked for a command or a route implementing “finalise the contract”, found none, and recorded not found — twelve times.
The defect is not where it looks
It is not in discovery, and it is not in the analyser. Each did exactly what it was built to do.
The defect is that a verdict derived from a guess was reported with the same confidence as a verdict derived from evidence.
Those two things look identical downstream and are not remotely equivalent. A specification the caller supplies is evidence: scored against it, zero delivered is a finding, and my study contains one — given a command-line specification under a verified-clean workspace, with a prompt explicitly forbidding pipeline output, an agent shipped a data pipeline. Zero overlap with the brief. That zero means something.
A specification the instrument guessed is not evidence. Zero delivered against it says far more about the guess than about the code. Every feature unmatched is the signature of having read intent off the wrong thing.
What makes this uncomfortable is that I had already written the rule down. My dissertation states it twice: a scope drift figure is only as good as the brief behind it. The evidence appendix withholds two readings for exactly this reason. The principle existed in prose and was never enforced in code.
The first fix was not a fix
I shipped a boolean. discovery_failed, set when a guessed brief matched nothing at all. Tests passed, the bad reading disappeared, and I wrote it up.
Reviewed properly an hour later, it does not hold up. It catches one failure mode. A guessed brief matching one feature in twelve is no more trustworthy than one matching none, and sailed straight through that gate — reported exactly as a supplied brief would be. The concept the tool was missing was still missing. It had been given a name in one special case and nowhere else.
A boolean about a single failure mode is a special case wearing a concept’s clothes.
What the tool needed was a grade
Every fidelity reading now carries what it rests on, computed where the reading is computed, consulted by every surface before anything renders.
| Grade | When | What may be shown |
|---|---|---|
| MEASURED | a human supplied the brief | Report it plainly. |
| PROVISIONAL | derived, and the code corroborated some of it | Report it, labelled, every time. |
| WITHHELD | derived, and it corroborated nothing | No verdict. Give the reason. |
Two properties are load-bearing.
A derived brief is never a measurement, however much of it matches. The thing in doubt is the brief, not the degree of fit. Ninety per cent matched is still ninety per cent of a guess, and PROVISIONAL carries a disclosure no surface can drop.
There is no threshold to tune. The only bright line is corroboration versus none — and that is not a tuned parameter. A brief that matches nothing has supplied the single piece of evidence bearing on its own validity.
Where a verdict used to be, the user now gets the reason and the remedy:
Intent unknown: Could not establish what this project was asked
to build. The provisional brief read from recent feat: commits
matched nothing in the code, which points at the brief rather
than at the code. Scope drift is not reported. Supply a
specification with --spec, and it will be.The test that matters is not the one proving the bad reading is gone. It is test_a_supplied_brief_still_reports_zero_delivered — which pins the behaviour the fix must not break, because that zero is my study’s central result. A fix that silences guesses must not silence findings.
What it cost, and what it bought
Reported by a user, not caught by my suite, on a folder I would never have thought to test. The instrument shipped a reading it had no basis for, and somebody outside the project saw it before anybody inside did.
It is the fourth defect found in this tool and written up publicly rather than quietly patched — and the first found by someone who is not me. I would rather that record exist than a clean one.
Every tool that reports a number can do two different things that look identical in the output: it can measure, and it can guess. Mine could do both and could not tell you which it had just done.
An instrument built to audit whether generated code stays inside its brief earns that right only by being honest about the readings it cannot support.