A Five-Minute Test That Catches an AI Agent Skipping a Step

"Done" is a claim, not a checkpoint

Give an agent a five-step task and there's a failure mode worth planning for specifically: it reports success having genuinely completed steps one, two, and four, and skipped three -- not maliciously, just because the model's internal sense of "I've done what was asked" doesn't reliably track a literal checklist the way a human ticking boxes would.

The five-minute test to write

Pick the task's most commonly skipped step -- usually the unglamorous one: updating a changelog, running a specific test file, regenerating a lockfile, bumping a version string. Write one small, fast check whose only job is confirming that specific artifact changed: did the changelog file's mtime move, does the version string in the manifest match what was asked for, did the specific test file actually execute (not just "did the suite pass," which a skip can still satisfy).

Wire it to run after every claimed completion

The check doesn't belong in a document the agent might read and might not. It belongs as the actual last command run before you accept the task as done -- a script that exits non-zero if the artifact didn't change. That turns "I did all five steps" from a claim you have to take on faith into something that either passes or doesn't, in under five seconds.

Why this beats a longer checklist

A longer written checklist handed to the agent doesn't fix this -- the agent can skip a step in a five-item list exactly as easily as a one-item instruction, because the failure isn't a lack of instructions, it's execution drift. A checklist is something the agent follows. A test is something that runs regardless of what the agent believes it did, which is the only kind of check that catches the step it silently skipped.