09/02/2026
Four ways an AI agent was confidently wrong on a migration
Running an agent across a 150-page site migration produces a useful record of how this kind of work fails. Not hallucinated APIs or invented libraries — those are caught by the first build. The interesting failures are the ones that produce a plausible, well-reasoned, wrong answer that survives review.
Four from one project.
The four
1. The right words, the wrong subject
MySQL Workbench screenshots paired to an article about Doctrine try/catch, on three words that genuinely appear in both.
2. A server that had been right for forty-five hours
A route committed two hours after the process started. The process never had the code, and nothing failed loudly enough to say so.
3. A test that wrote down facts it did not own
Block index 8, a nine-block assertion and a search string, all about a page the test had no control over.
4. A confident diagnosis from simulated data
A 2,061ms render delay with a plausible cause. A real browser showed first and largest contentful paint at the same moment.
1. A match on the right words and the wrong subject
Thirty-four orphaned images needed pairing with the posts they belonged to. Matching filename words against post titles proposed seven pairs. Five were correct. Two were not: a pair of MySQL Workbench screenshots was matched to an article about using try/catch to detect a failed Doctrine insert.
The overlap was database, success and
failure — three words that genuinely appear in both. The article
never mentions MySQL Workbench, and the screenshots belong to a database
setup guide that was never migrated at all.
What caught it was checking each proposed match against the article's actual body text rather than its title. A title is a summary, and two things can summarize the same way while being about different subjects.
2. A server that had been right for forty-five hours
The site editor started returning 404 for one of its own API routes. The route existed, was correctly registered, and had tests.
The editor process had been running for 45 hours. The route had been committed 2 hours and 7 minutes after that process started. It had never had the code.
What made this expensive is that nothing failed. An unrecognised API path on that server redirects to the editor shell, so the browser received a 200 with an HTML body, JSON parsing quietly returned nothing, and the feature it powered simply never appeared. The client code handled the failure correctly — which is exactly why it stayed invisible for two days.
The fix was not a repair. It was a signal: a health endpoint reporting when the process started and when its source was last written, and a banner when the two disagree. A server too old to have that endpoint answers with the same shell redirect, so a non-JSON response is the answer.
3. A test that wrote down facts about something it did not own
A test suite had been failing for a day. It ran against a real site page rather than a fixture — deliberately, and for a good reason: the properties it defends are properties of real content, and a fixture would be written to pass.
But it had written three facts about that page into itself: an operation on block index 8, an assertion that the page had nine blocks, and a search for a literal comment string. The page lost a block, and the comment was rewritten.
Only the first announced itself, and it did so destructively — the operation threw, the helper returned null, and the null reached the YAML parser and ended the run, so every check after it never executed. The third failed in the other direction and said nothing: the string search returned -1 and it printed "comment line 0 -> 0" while asserting nothing at all.
Two separate lessons. Derive from the artifact rather than writing down facts about something you do not control. And a check that throws must not take the suite down with it, or one stale assumption hides every real result behind it.
4. A confident diagnosis from simulated data
A page was reporting a 2,061ms element render delay. The proposed cause was a
missing Content-Type header on a WebP image, which was a real fault
and was worth fixing. Fixing it moved the score by two points.
Measuring the same page in a real browser showed first contentful paint and largest contentful paint happening at the same moment. There was no render delay. The number came from a simulated network model, and the actual cost was a render-blocking stylesheet — 476ms, measured directly.
The diagnosis was not careless. It was drawn from a real number produced by a respected tool, and it was wrong because the number described a model rather than the page.
The common shape
None of these is a knowledge failure, and none would be fixed by a better model. Each is a case of evidence that looked direct and was one step removed: a title standing in for an article, a running process standing in for the code on disk, a hard-coded index standing in for a file, a simulation standing in for a browser.
The habit that catches all four is the same. Ask what the evidence actually is, and then go and look at the thing itself.