My search engine parsed every crawled page with PHP 8.4's new HTML5 parser, Dom\HTMLDocument::createFromString(). In July, someone pointed out that a Danish result showed its title as "Bes�g i hansestaden L�beck" instead of "Besøg i hansestaden Lübeck". Fixing that took two rounds a month apart, and the second round was a bug in the first fix that quietly made pages disappear from the index.
The parser only sees bytes
createFromString() takes a string of bytes. It has no idea what HTTP headers the page arrived with. Unless you tell it the encoding, all it can go on is the document itself: a <meta charset> tag if there is one, and a default if there isn't.
That's where the HTML standard's own order matters. Its encoding sniffing algorithm says that if the transport layer, meaning the HTTP Content-Type header, specifies an encoding the browser supports, that encoding wins outright. Only after that does the parser prescan the start of the document for a <meta> declaration. Browsers get the header for free. A crawler that hands the parser a string has to pass it on itself.
Two ways to get it wrong
I reproduced both failure cases with PHP 8.5's version of the same class:
- No declaration anywhere. Latin-1 bytes, no charset in the header, no meta tag. The parser falls back to UTF-8, the bytes for ø and ü aren't valid UTF-8, and they come out as replacement characters: "Bes�g i hansestaden L�beck". That was the Danish page in the report.
- A meta tag that lies. A page whose bytes are UTF-8 but whose template still says
<meta charset="iso-8859-1">, typical of a site migrated to UTF-8 without updating its template. The parser believes the tag, and "Besøg" becomes "Besøg".
The first fix: pass the encoding in
createFromString() has a third argument, $overrideEncoding. The crawler already stored each page's Content-Type header, so the indexer got a small function to choose an override:
- if the header names a charset, use it, matching the standard's priority
- otherwise, if the bytes are valid UTF-8, pass nothing and let the parser do its job
- otherwise, if the page declares a charset itself, also pass nothing: with no header to contradict it, there's no better evidence to override it with
- otherwise, use Windows-1252
The last rule follows the HTML standard too. Its table of suggested default encodings by locale, derived from what Windows, Chrome and Firefox do, gives windows-1252 for every locale it doesn't list separately, Danish included. And in the WHATWG Encoding Standard, the labels iso-8859-1 and latin1 are names for windows-1252: PHP's parser reports windows-1252 even when you ask it for ISO-8859-1. In my test, the Windows-1252 override turned the broken title back into "Besøg i hansestaden Lübeck", and a UTF-8 override fixed the lying meta tag.
The function deliberately leaves one case alone: a wrong meta tag with no header. There's no independent signal to correct it with, and guessing would break pages that are what they claim to be.
The second bug: the fix threw away pages
A month later, a different metric went wrong. The share of pages that were crawled cleanly but ended up with no extracted text climbed from 23% to about 79% in three days, around the time a change broadened the crawl to many more long-tail sites. The indexer's log had the answer:
Dom\HTMLDocument::createFromString(): Argument #3 ($overrideEncoding) must be a valid document encoding
The function checked that the header's charset looked like an encoding name, not that the parser recognised it. Real servers send all sorts of things in that slot: typos like iso-8559-1 and unf-8, placeholders like none and auto, and, surprisingly often among Danish sites, a MySQL collation name such as latin1_danish_ci where a charset should be. createFromString() throws a ValueError for any of them, and I confirmed that with both latin1_danish_ci and iso-8559-1. The exception wasn't caught, so the whole document's extraction was abandoned, and it stayed in the database with no text, never to be retried.
The fix catches the ValueError and parses the page again with no override, so the parser falls back to the document's own declaration. A bad header now costs, at worst, the accuracy the header could have added, instead of the whole page.
What generalises
If you parse HTML from a string, pass the encoding from the HTTP header, because the parser can't see it. Validate a charset by asking the thing that will use it, not by its shape. And treat any input that came from a remote server as hostile to your assumptions: the first fix was correct for every page that followed the rules, and the web is full of pages that don't.
Leave a comment
Comments are reviewed before they appear.