Fact-Checking AI-Drafted Technical Articles: The Numbers Were Right, the Explanations Weren't

Articles on this site are drafted with an AI agent, Claude, working from my git history and design notes, and every draft goes through a verification pass before it's scheduled. On 15 October, an unattended pipeline run drafted and scheduled seven articles. I then checked them against the commits they were based on, and six of the seven contained a factual error. The headline figures were right. Most of what was wrong was explanations.

What the pass caught and what it didn't

The pipeline's instructions at the time had a verification step: go through the draft claim by claim, every number, date, quantity and causal statement, and re-check each against its source. It reliably caught the kind of error it was described with: a rounded figure turned exact, a date attached to the wrong event.

The errors it let through were of a different kind. Each was a confident sentence about how something works, which the commit didn't say:

• An article about API Platform 4 dropping the hydra: prefix said the backend tests had kept passing against keys that no longer existed. Reading a missing key fails, and the follow-up commit was the one that fixed the failing tests.
• An article about a Symfony route collision said the routes registered in "filesystem order". Symfony's directory loader sorts the files by path; the order is alphabetical by design.
• An installer article said the /status endpoint failed open when the database was down. In the code, the fail-open endpoint is /system-check; /status answers "not installed".
• A Docker article said running composer install inside the container would be undone on restart. Through the bind mount, it writes to the host and persists.

The rest were counting errors of the same nature: "eight tables in one file plus two in another" where the migration had eleven in one file, "three host ports" where the commit changed two, and "three minutes" and "five minutes" for the same gap in one article. Each was fixed on the live site before the article went live.

Why explanations slip through

A number has an obvious source to compare it with. An explanation usually doesn't: the commit says what changed, and the explanation of why is something the draft adds. A pass that asks "does this match the source?" finds nothing to compare it with, and moves on. The sentences that went wrong were the ones that sounded most like expertise.

The rule that replaced it

The pipeline's instructions now treat explanations like numbers, in four categories:

• how a tool or framework behaves: find it in the dependency's own source code, the project's code, or official documentation
• what would have happened: reason it through against the real code before writing it
• which part did what: name what the code names
• counts and lists: recount from the diff, never from memory or the idea notes

It also asks for a reread for internal consistency, a recheck of anything that leans on a corrected claim, and a fallback: if an explanation can't be sourced, describe what happened and leave out why. The checked claims go into a small file per article, with a source for each line, so the pass can be audited afterwards.

Applying it by hand

Over the following day I wrote 25 articles interactively with the same rules and kept the claims files. 22 of the 25 needed at least one correction at the checking stage; the other three needed only style changes or none. A few of the catches:

• A commit message said search engines would find the sitemap through robots.txt. The site's robots.txt has no Sitemap: line. That article is on hold until the site is fixed.
• A draft said the Linux OOM killer kills the biggest process. Its choice depends on a score, and the source only recorded which process it killed.
• MySQL's documented rules for LEAST() suggested one comparison type, and a commit claimed another. A test against MySQL 8.0 settled it, and the article says both.
• A draft explained a quoting workaround for ddev exec from a code comment. Testing ddev showed the real behavior was more specific: it puts quotes around arguments that contain spaces and inserts the rest as they are.

None of these were caught by reading the draft for plausibility. They were caught by opening the source the sentence claimed to rest on.

What generalises

If you use an AI model to draft technical writing, the numbers are the easy part to verify and the least likely to be wrong. Check the explanations: every "because", every "would have", every description of how a tool works. Ask where each one came from, and accept "the code", "the docs" or "I ran it" as answers. A sentence that came from nowhere is the one most likely to be confidently wrong.