In August, a page from a small Danish website turned up on the "recently indexed" admin page of my search engine. That site's robots.txt is two lines long and couldn't be clearer:
User-agent: *Disallow: /
Everything is off limits, to every crawler. My crawler checked that file before fetching the page, concluded it was allowed, and indexed it. The cause was three invisible bytes at the start of the file.
Three bytes before "User-agent"
The site served its robots.txt with a UTF-8 byte order mark: the bytes EF BB BF at the very start of the file. It also used Windows line endings, CR LF. Neither shows up when you look at the file in a browser.
My parser handled each line the usual way: split on the first colon, take the field name, run it through strtolower(trim(...)), and compare it with 'user-agent'. The problem is that PHP's trim(), without a second argument, strips exactly six characters: space, tab, line feed, carriage return, NUL and vertical tab. A byte order mark isn't one of them. On the first line, the field name came out as "\xEF\xBB\xBFuser-agent", which is not equal to 'user-agent'. You can check it in one line of PHP: trim("\xEF\xBB\xBFUser-agent") === "User-agent" is false.
How a failed comparison became "allow everything"
A broken first line could have failed in a harmless way. It didn't, because of how the rest of the parser was built.
Rules in robots.txt belong to the user-agent lines above them. The parser kept a list of the user agents for the current group, and a rule that appeared before any user-agent line had no group to belong to, so it was dropped. With the first line unrecognised, the list was still empty when the parser reached Disallow: /. So that line was dropped too.
The result was an empty rule set. The function that answers "may I fetch this URL?" starts from "allowed" and lets matching rules change the answer. With no rules, nothing changed it. A file that forbids everything parsed exactly like an empty file. I reproduced it by running the parser on the site's actual bytes: an empty list of groups, exactly as diagnosed.
Not one odd site
The obvious question was whether this was one strange file. The three crawl servers cached the robots.txt files they fetched, so I scanned those caches: 5,357 files with a leading byte order mark out of about 676,000. Not all of those have a restrictive policy that the bug was defeating, but the site that started this proved that at least some did.
Two other parsers are worth comparing. RFC 9309, the robots.txt standard, says the file must be UTF-8 encoded and doesn't mention a byte order mark at all. Google's documentation is explicit: Google ignores invalid lines, including a byte order mark at the start of the file. So Googlebot reads that two-line file as intended, and mine read it as an open door.
The fix, and what it didn't fix
The fix is a small helper that removes the three bytes if the text starts with them, called at the top of the two functions that parse raw robots.txt text: the one that builds the rule groups and the one that extracts Sitemap: lines. Because the crawler cached only the raw bytes and parsed them fresh on every check, the fix applied to every cached file at once, with no cache purge. A regression test uses the real byte content of the file that exposed the bug.
The investigation also surfaced a gap that I noted and didn't fix. robots.txt only gated future fetches. Nothing in the system went back and removed pages that had been indexed while the effective answer was "allowed", whether because of this bug, a stale cache, or a robots.txt request that got a 4xx response, which the crawler also treated as "no restrictions". A page indexed by mistake stayed indexed.
If you write anything that reads robots.txt, strip the byte order mark before you parse. And if you rely on robots.txt to keep crawlers out, know that it only works if every crawler parses it correctly; it often doesn't block what you think it does even when they do.
Leave a comment
Comments are reviewed before they appear.