Green Isn't Done: Six Silent Failures and the Check That Caught Each
Every failure below came back looking like a success: exit code 0, a green CI run, a file in the dashboard, a "sent" status. None of them threw an error. I caught each one only because something read the real state of the world instead of trusting the report.
They come from my own projects, not from production agents. But they are exactly the failure modes an agent harness produces at scale: the tool says it's done, the transcript looks fine, and the job didn't happen.
Class A: Claimed Done Is Not World State
1. The deploy that was green and from July. CI for my habit tracker's site "deployed" successfully on every push. The deploy step's token was missing, and the CLI exited 0 anyway. Meanwhile the domain was attached to the wrong hosting site and had been serving a July build ever since. A ~90-line curl smoke test against the real domain turned the run red: 6 of 6 checks failed — a stale title, legal pages serving the app shell, the universal-link file served as HTML, a missing feature, a public source map.
The check: a postcondition read from the system users actually touch, not the tool's return value.
2. The upload that saved nothing. A storefront's uploader accepted a browser-driven upload of a product file, and the dashboard listed it. It had stored 0 bytes. Only a fresh page load plus actually downloading and unzipping the file exposed it.
The check: round-trip the artifact. Fetch it back and open it.
3. The sign-up form that dropped everyone. A hosted subscribe form cleared on submit and showed no error. The account had zero follower records, unconfirmed ones included: every submission was lost before it reached the server — from a normal browser, by a human, not just from automation.
The check: count records at the destination, not submissions at the source.
Class B: Right Shape, Wrong Content
4. The render that downgraded itself. A video editor's render job reported success. The output was 720p with a watermark, against a 1080p request.
The check: probe the actual file's properties against the request.
5. The narration that was silence. An audio render exited 0 and produced a perfectly valid file of silence.
The check: a PCM-level test that the signal has energy. Format validity says nothing about content.
Class C: Right Action, Wrong Population
6. The reminder push that woke a graveyard. The first scheduled push my app ever delivered worked perfectly — to the wrong people. About two dozen sends went out on night one when only four active pairs existed. Recipients included accounts dormant for 90+ days, only two sends reached a device, and 9 of 43 push calls died on stale keys. Three eligibility guards (active pair, recent owner activity, recent partner activity) would have cut the sends to two.
The check: assert the action set is a subset of the eligible set, and alert when its size is anomalous.
Applying This to an Agent
Take an agent asked to build a 200-account prospect list for a fuzzy target profile and write it into a CRM. Each class maps to an eval:
- A — postcondition. After the agent says "done," read the destination back: row count, IDs, and that the write landed in the right table. The gap between claimed completion and verified completion is the number I'd want on a dashboard. It is the silent-failure rate, and no transcript-based grader can see it.
- B — content probes. Sample rows and re-verify each enriched field against the source it cites. Flag fields with no provenance. A plausible headcount that matches no source is a fabrication that looks like enrichment.
- C — population guards. Write the target profile as executable predicates and measure the out-of-profile rate and the duplicate rate. Seed known negatives — accounts that must never appear — and treat any appearance as a hard fail.
- D — evaluate the evaluator. Before trusting a green eval, prove it can go red. Inject known failures (drop rows, swap a domain, slip in an excluded company) and require each to be caught. The curl test in case 1 earned its trust by failing 6 of 6 on a real broken domain. An eval that has never failed is a hypothesis.
The Asymmetry Underneath
An agent loop recovers from visible errors for free: it sees the traceback on the next turn and tries again. It builds confidently on silent ones. Every check above converts a silent error into a visible one, and in my experience that buys more reliability than any prompt change.
The limits are worth stating: these are small systems I built and broke myself, and the agent section is a design, not a measured result. But every one of these failures passed every check I had at the time — until I added one that looked at the world instead of the report.