Expanding a content-cleanup list by language code should be one of the safer changes a search engine can make. A live check of what was actually in the data turned up three entries that looked exactly like real languages and weren't — one of them the exact language the entire system exists to prioritize.
What the list is for
A host-level eviction policy removes low-value content, confidently classified by majority language, from hosts that don't match the languages this search engine actually cares about. Denmark-focused, with English and Danish getting real crawl-priority treatment — everything else is a candidate for removal once a host's content is clearly, overwhelmingly, one of the excluded languages. The list itself had only ever covered a narrow starting set. Expanding it meant checking what languages actually show up across the corpus at meaningful volume, and adding every one that crossed a real threshold.
What a live distribution check actually surfaces
Running that check against the real, current data didn't just return clean two-letter language codes. It returned three entries with real host counts behind them — genuinely present in the data, genuinely crossing the volume threshold that would normally justify adding them straight to the denylist — that turned out not to be real language judgments at all.
The one that mattered most
The most consequential of the three had 407 hosts behind it. It isn't a valid ISO language code. It's almost certainly a mislabeled version of Danish — the exact language this entire eviction policy exists to protect, not remove. Folding it into a denylist meant to prioritize Danish content would have meant deleting Danish content, under a policy specifically designed to prevent exactly that outcome, because a mislabeled tag happened to cross the same volume threshold as a genuine foreign-language signal.
The other two, smaller but the same shape
A second tag, with 68 hosts, is also not a real language code — most likely a mislabeled version of Chinese, which was already separately covered under its correct code at low volume regardless. A third, with 91 hosts, is a real, valid code — but not a language judgment at all. It's the specific ISO designation for "no linguistic content," used for pages that are effectively images or PDFs with no meaningful text to classify in the first place. Applying a language-based eviction policy to a tag that explicitly means "there is no language here" would have quietly removed a category of content the policy was never actually designed to make a judgment about.
Why all three looked exactly like the data they were checking for
Nothing about a live distribution check flags a mislabeled or non-standard tag as different from a legitimate one — both show up as a plain code with a host count attached, sorted by the same volume ranking as everything else. A tag that clears the threshold looks, from the check's own output, identical whether it represents a real, correctly-tagged foreign language or a data artifact that happens to share the same shape. Telling the two apart required actually knowing what a real ISO language code looks like well enough to notice three entries that didn't quite fit — not something the volume check itself could surface on its own.
Excluded, and said so directly
All three were left out of the expanded list, and the reasoning for each is written into the configuration change itself rather than left as a silent omission — what each anomalous tag actually is, and specifically why it was excluded rather than folded in. A future person looking at this same list, seeing three plausible-looking language codes conspicuously missing, has an immediate answer for why, instead of a reason to wonder whether they were simply overlooked.
What this says about "the data agrees with me"
A volume check that confirms a language code is common enough to matter answers a narrower question than it looks like it answers. It confirms the tag exists at real scale in the data. It says nothing about whether the tag means what its two or three letters suggest it means. The actual safety check here wasn't the distribution query — it was someone looking at the query's own output closely enough to notice that a couple of entries didn't belong on a list of real languages at all, in time to keep a cleanup policy from becoming the exact kind of accidental deletion it was built to prevent.
Leave a comment
Comments are reviewed before they appear.