The test that shipped with the fix
An agent closed a ticket about duplicate sessions. It found the bug, fixed it, added a test, and reported: fixed the de-duplication bug, added a test for it, suite passes. All three sentences were true in the way that matters least.
test('de-duplicates the list', () => {
const out = dedupe([{ id: 'a' }, { id: 'a' }]);
assert.ok(Array.isArray(out));
});That test passes against the broken code. It passed before the fix, it passes after, and it will keep passing on the day someone deletes the fix. It is green, it sits in the diff next to the change, and it is evidence of nothing.
I have started calling this a green lie: a suite that is green for a reason unrelated to the thing it is supposed to be watching. It is not new — every codebase has some. What changed is the rate at which it is produced, and who is left to notice.
What other people have measured
These numbers are not mine. They agree with each other, which is what makes them worth repeating.
The paper on false success gives the reason the obvious remedy fails: judges read the confident closing tone rather than the state of the machine. "Done! All tests pass." is written in the same voice whether or not it is true.
The question a tone of voice cannot answer
The whole idea is old enough to be embarrassing.
- 01
Put the source back
Check out the commit the change was built on, in a throwaway worktree. The tests stay as they were written; only the source goes back.
- 02
Run the new test there
One test, on its own, by name. Read the exit code and nothing else.
- 03
It must fail
A test written for this change has to be red without it. If it passes, it did not test the change.
That is red-green-refactor. What is new is that nobody is watching the red any more, because the thing writing the test is also the thing reporting on it. The red used to be enforced by a human having to sit through it. A test that is green for the wrong reason is a cousin of one that is red for the wrong reason: both are a suite telling you something about itself rather than about the code.
So I wrote something that sits through it: alibi. No model is asked anything, there is no network call in the repository, and it gives the same answer every time.
Two decisions that were not obvious
Exit codes, never parsed output. Every runner prints results, and every one prints them differently, changes the format between versions, and changes it again when a reporter is configured. A parser that is 98% right produces a false accusation every fiftieth run, and one false accusation costs more trust than fifty correct ones earn. So each test is run on its own and only the exit code is read. It is slower, and it is the difference between "this test passed against the old source" and "this text looked like a pass".
A third verdict, because two would be a lie. A new test for a module that did not exist yet cannot import it: the runner exits non-zero without ever evaluating an assertion. Calling that an alibi would hand one to every test written against new code, which is most of them. Calling it a failure to prove would accuse a perfectly good test. So it gets its own name — provisional — and proves nothing on purpose.
What 200 agent pull requests said
A tool is a claim about the world, so I pointed it at the world. I collected 200 merged pull requests written by five coding agents — Devin, Copilot, Claude, Codegen, OpenHands — across 138 repositories and four languages, keeping only ones that changed both source and tests, and at most two per repository so no single project could dominate.
For each: check the repository out at the base commit, install what the suite needs, confirm the suite is green there, copy the pull request’s test files over the old source, and run each new test on its own.
The honest summary is narrower than the headline number: in 8 of 17 pull requests, at least one added test would have passed without the change it shipped with. That is the part I would defend.
The tool was wrong, and checking by hand is what found it
My first run gave 34.9%. One pull request accounted for 32 of the 67 findings — half of them — so before writing anything down I went and looked at it.
Every one of those tests imported a module that the base commit did not have. They could not have passed there honestly. They should have come back as provisional, and instead they came back as accusations.
The cause: pip install -e . writes an absolute path to the original working tree into site-packages. I had put the worktree first on PYTHONPATH, which reorders the import search — but reordering cannot stop it. A module the base commit never had is not in the worktree at all, so Python keeps walking and finds the new one through the editable install. The test then passed against code that was supposed to be reverted.
The fix is to ask instead of assume: import the changed package inside the worktree and print where it came from. A path outside the worktree means the revert did not take, and those tests are reported as unanswered, naming the module and the path it leaked to. One pull request went from 33 accusations to 39 honest non-answers, and the headline went from 38% to 25%.
I am telling you this because it is the part that decides whether the rest is worth anything. A number that survives its author trying to break it is worth more than a bigger number that did not.
Six ways to buy a green suite
The repository ships a museum of them. Each is a real repository that gets built and checked in the test suite, so an exhibit that stops being caught breaks the build.
- The new test passes against the old source — it tested nothing.
- The failing test was deleted — green because there is less of it.
- The failing test was switched off with one word.
- The test asserts nothing at all — it can only fail by crashing.
- The test mocks the thing that changed — a photo of a bridge holds no weight.
- The assertion was softened until it agreed — it checked contents, now it checks length.
A seventh that the tool does not catch is the most useful thing anyone could send me.
What it will not tell you
- It cannot prove a change is correct. It finds tests that are not evidence for one, which is a much smaller claim. An empty report means nothing was caught.
- It says nothing about tests the diff did not touch — whatever alibi they had, they had before.
- Rust unit tests live in the file they test, so reverting the source would revert the test with it. Those are reported as unexamined rather than guessed at.
- One process per test. On a suite with heavy global setup this is slow, and there is deliberately no fast mode that trusts parsed output.
- A characterization test — one written to lock in behaviour that already exists — has no alibi by definition. Mark it, and it stays in the report without counting against you. There is no way to silence a finding invisibly.
Running it
npx skills add BOTIROFF-D/alibi # your agent runs the check before it says done
npm install -g @botiroff/alibi # for your terminal and CIThe skill needs nothing installed: it walks the agent through the check with plain git and the project’s own test command, so it works in a repository that has never seen node. MIT, zero runtime dependencies, works with vitest, jest, mocha, node --test, pytest, go test, cargo test, rspec and phpunit — or any shell command you give it.
The corpus harness is in the repository under experiments/corpus, together with the collected pull requests and the raw results, so the numbers above can be checked without re-running anything.
Frequently asked questions
Is this just mutation testing?
It is adjacent. Mutation testing asks whether the suite would catch an injected bug, and costs hundreds of runs to answer. This asks whether it caught the bug that was actually there: your change is the mutant. One run per new test, no operators to argue about. A mutation pass over the changed lines is included for the second question, and it is off by default.
Can a test legitimately pass before the fix?
Yes — a characterization test locks in behaviour that already exists and has no alibi by definition. Mark it with a comment and it stays in the report, labelled, without counting against the verdict. The exception being visible is the reason it is safe to have.
Why not ask a model to review the tests instead?
Because it has been measured. No configuration across five judge models and five prompt strategies exceeded AUROC 0.65, and the same judges reached 0.54 on AppWorld. The judge reads the confident closing tone. An exit code does not have one.
Does 25% mean a quarter of agent tests are worthless?
No. It means that in the seventeen pull requests where this could be checked mechanically, a quarter of the added tests passed against the code from before the change. The median pull request had none, and one repository supplied 14 of the 35. Treat it as a floor from a narrow sample.
Is one process per test not unusably slow?
On a suite with heavy global setup, sometimes. The tests examined are the ones the diff touched, not all of them, and the whole-suite run can be switched off. There will not be a fast mode that trusts parsed output, because parsed output is exactly what the tool refuses to trust.
Sources
Every claim here can be checked: below are the primary sources, not a retelling.
- 01From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents — The 75.8% figure and the judge AUROC results
- 02Test Coverage Analysis of Agentic Pull Requests — 4,882 agent pull requests; coverage improved in 35.9% (Java) and 22.5% (Python)
- 03Stack Overflow Developer Survey 2025 — "Almost right, but not quite" — 45% of 49,000 respondents
- 04alibi — the tool and the corpus harness — MIT; experiments/corpus holds the collected pull requests and the raw results
This piece is signed by name, unlike the rest: there is no client behind it. The code it discusses is fully open and published under the same name — you can read it, run it and check the article’s claims instead of taking any of them on trust.