By the end of part 2 the search engine had a corpus that could grow without filling a disk, on hardware that could hold it. That raised a different question. A broad crawl finds the same page under many URLs, mirrors and copies, thin and spammy pages, and whole sites in languages nobody will search for here. This part is about deciding what to keep, which copy counts, and how to tell whether results got better.
Every weight has to earn its place
Part 1 introduced the golden set, a list of queries with known right answers. Every ranking signal since then has had to pass through it. When a page-quality score was switched on in early July, I swept its weight from 0 to 20 and ran the golden set at each step. Weights 0 to 3 tied exactly, and 5 to 12 cost one query its place in the top 10. It went live at 2. The commit message notes the limitation: the golden set was mostly canonical, high-authority pages, so it barely exercised a quality signal in the first place.
The same page under many URLs
URL parameters, session IDs and mirrors give the same content many addresses, and each copy costs a crawl slot on every recrawl plus a document in the index. Design 0009 added near-duplicate detection with SimHash, written without any external library:
- split the extracted text into word trigrams, so word order matters
- hash each trigram to 64 bits
- take a per-bit majority vote across all the hashes
- clear the top bit, so PHP's signed integers and the database agree on the value
The first version only caught exact matches, through a single indexed equality lookup. That sounds weak, but content extraction already strips navigation, headers and footers, so the extracted text of a URL variant is usually byte-identical even when the raw HTML differs. Fuzzy matching was deferred until there was a reason to build it.
Which copy wins
The exact-match version had a known gap: the first copy indexed always stayed canonical. If a low-quality mirror got crawled first, it won permanently. Design 0011 replaced that with three tiers. First, a rel=canonical declaration counts if it points at the other page in the pair. Otherwise, authority and quality decide, with a guard against comparing a page PageRank hasn't scored yet. Only if nothing differs does the existing canonical stay.
The same design added fuzzy matching with a simple trick. The 63-bit fingerprint is split into four bands. Two fingerprints at most 3 bits apart can't differ in all four bands, so at least one band matches exactly. Four indexed equality lookups find every candidate, and a real Hamming distance check confirms it.
Deleting on purpose
Part 2 covered why eviction became necessary. What it deletes is a separate decision. The first real run on the Denmark box removed 297,311 URLs and 293,774 OpenSearch documents, near-duplicates and pages that were gone, in 3 hours 40 minutes with zero errors. Afterward, none of the 74.68 million remaining rows were flagged near-duplicate or gone.
A third rule followed: delete pages whose quality is below 0.2 and whose authority is below 0.7. The thresholds are deliberately tight. Looser pairs caught real false positives in manual samples: photo galleries, directory pages and the homepage of a live news outlet.
The last rule evicts hosts by language. Testing it against the real corpus found a bug before it ran: a host's language came from the first <html lang> value seen, so news.bbc.co.uk was labelled Arabic because of one early Arabic page, and would have been deleted. Host language became a majority vote. The same list came close to deleting the wrong thing again in September, which has its own article, as does the first manual host eviction.
"Discussed on"
Not every feature was about removing things. A plain list of links looks like every other search engine. A line under a result saying that people have discussed it, with a short quote, is something a small engine can offer without beating anyone on relevance.
The constraint was no new data source: no Reddit API, no list of allowed forums. Design 0012 builds it entirely from data the crawl already had. Pages are already categorized at index time, including a community category for forums and comment threads, and every inbound link is already recorded. "Discussed on X" means a community page on X links to the result, and the link's anchor text becomes the quote. It went live on 18 July with at most three entries per result.
Pages that links don't reach
In August, I noticed that www.vordingborg.com, a real Danish site, had only 11 known URLs after being crawled. Its homepage didn't link to most of its own catalog. A search through the code found that sitemap.xml wasn't used anywhere, and the robots.txt parser ignored the Sitemap: line. A page that no other page links to was invisible to the crawler, even when the site would hand over a complete list.
Sitemap discovery shipped on 16 August, with caps on depth, URL count and bytes, and tests for XML entity attacks. It ran as a worker the same day and was crash-looping on memory the next. In September a single sitemap froze the whole queue for 60 hours.
When Danish broke the measurements
The wipe in part 2 reseeded the crawl from Danish sites, and that broke two things that had nothing to do with ranking.
The first was the golden set. Its queries were mostly English open-source and reference sites, and on 19 July it scored MRR 0.16, down from 0.6646. Checking all 20 queries by hand found 16 broken, because their target sites were barely in the new corpus. One apparent hit for gnu.org was really savannah.nongnu.org matching a wildcard. By 10 August MRR had fallen to 0.0932, and the top results showed why: English queries now returned unrelated Danish pages. I rebuilt the set around 12 verified Danish queries plus the three English ones that still worked. The new baseline was MRR 0.6329, with every target in the top 10.
The second was language routing. Every language has its own OpenSearch index with its own analyzer, and the code matched a page's lang attribute against bare codes like da. Real sites declare da-DK, which never matched, so those pages fell into a catch-all default index with generic text analysis. On 16 August that index held 3,085,213 documents, and 2,611,223 of them, 85%, belonged in a real language index. The fix was one normalization step. Moving the documents took a staged reindex, the smallest languages first and Danish last.
All of this assumed the authority score was right. Authority comes from PageRank, and PageRank is the part of this system that has never fitted comfortably anywhere. That's part 4.
Leave a comment
Comments are reviewed before they appear.