All tests pass. The feature is still broken.

My AI agents kept telling me “all tests pass”. And the feature was still broken.

That happened enough times that I stopped trusting the sentence. Not because the agents lie. The tests really pass. The problem is what a passing test actually tells you.

Green tests prove the test

A green test suite proves the code does what the test says. That’s it. It doesn’t prove the screen looks right. It doesn’t prove the API got the right payload. It doesn’t even prove the test is checking the thing you care about.

A terminal shows three test suites passing, while the phone screen next to it shows a cart with the total as $NaN. The tests check the math. Nobody checked the screen.

When I write the code myself, I usually open the app and look. An agent doesn’t do that unless you make it. So I made it.

No proof, no PR

Now my agents can’t open a pull request without proof inside it. One of three kinds:

1. A screenshot. Before and after, same screen, the base build vs the change. If the PR touches UI, this one is required. A table of numbers doesn’t count for a UI change.

A pull request called Fix cart total with two phone screenshots. Before: the total shows $NaN. After: the total shows $18.00.

2. The real output. The log, or the payload that actually went out, pasted in the PR as text. Not “I tested it locally”. Something anyone can read without asking me.

3. The test failing first. Break the thing on purpose, show the test going red, put it back, show it green. Both runs go in the PR.

Two runs of the same test. With the feature removed, testCartTotal fails: expected 18.00, got NaN. With the feature back, it passes.

The third one caught me

A test I never saw fail might be testing nothing. I know this, and it still got me.

I asked an agent to prove a new check worked. It “broke” the code, ran the test, and the test stayed green. The agent reported that as a problem with the test. It wasn’t. It had broken a line the feature didn’t even need. The code it removed was dead, so of course nothing failed.

So now there’s a second question after the red run: delete the biggest line in the code under test and see what stays green. Whatever still passes isn’t really covered.

Proof goes in the PR, not the repo

Screenshots in git just make the repo heavy. Git can’t compress images well, so every version of every screenshot stays in the history forever, and every clone pays for it.

So images get attached to the pull request, and text proof goes in the PR description as a code block. The reviewer is already there anyway.

The picture is what people look at

Honestly, this was the biggest surprise. I used to put measured tables in my PRs. Nobody read them. Everybody looks at the screenshot.

When I send a change to someone outside engineering, the before and after is the whole message. It answers “did it work?” in two seconds.

It’s enforced, not a rule

I tried writing this down as a rule first. Agents followed it most of the time, which means they didn’t follow it some of the time.

So it’s a check now. Before a PR opens, a hook reads the PR body and refuses it if there’s no image, no pasted output, and no red and green pair. If the diff touches UI files, only an image is accepted. There’s one way out: a line saying there’s no proof and why, so skipping it is always on purpose.

If you work with agents, try it for a week: no proof, no PR. How do you check what your agents built, besides the tests?