Part 1 ended with search running on OpenSearch and a benchmark to measure ranking against. This part is about the other half of the problem: finding pages at all, and storing what the crawler finds. Over about six weeks the corpus outgrew one server, then two, was thrown away twice, and ended up on a different database.
The acquisition wall
Design document 0002 from 14 June gives the starting point: about 1.5 million known URLs across 115,000 hosts, of which about 90,000 pages were indexed. Its diagnosis was that indexing and ranking were already ahead of what the crawl delivered, so crawl throughput was the thing to scale.
The first rule in that document was to instrument before optimizing. The crawler got a per-run log line with throughput and a breakdown by phase, and that line showed the bottleneck was network fetching, not choosing which URL to crawl next. Without it, I would have optimized the wrong thing.
The design that followed coordinates crawlers through the database instead of through a coordinator process. A crawler claims a whole host with an atomic SQL update, a host lease, which also enforces the per-host crawl delay. Expensive work such as ranking the frontier moved into background jobs, so picking the next URL stays cheap however big the corpus gets.
It worked. By 8 July the url table had about 63.5 million rows across roughly 3 million hosts, about forty times the June numbers.
One box gives up
Everything still ran on one VPS in Germany: 4 GB of RAM, 2 cores, and a MySQL buffer pool of 256 MB. Under sustained crawler load the buffer pool jammed in a state it didn't recover from even when idle, PHP's queries hung and nginx returned 504s. That happened three times on 4 July, and the box was swapping even with the workers switched off.
The way out was the decision from part 1. Search read only from OpenSearch, and OpenSearch documents already stored each result's URL and title, so the public search path could be made completely MySQL-free. Germany became the search frontend: OpenSearch, nginx, PHP and Caddy for TLS. A spare 8 GB, 4-core box in Denmark took MySQL, with a 2 GB buffer pool, plus the whole crawl, index and rank pipeline. The split was done the same day, and the 504s stopped.
30 GB a day
The next wall was disk. An investigation on 9 July measured the Denmark box's database growing by about 30 GB a day against a hard 160 GB disk. 99.45% of the new URLs came from hosts the crawler already knew, as known sites revealed more and more of their pages. Buying a bigger disk would only have bought days.
The crawlers were stopped while an eviction policy got designed: keep discovering, but continuously delete the lowest-value content so disk use stays flat. Moving the daily backups to spare disk on the German box took Denmark from 96% to 85% used. In mid-July a third box joined, carrying only crawler and indexer replicas for extra throughput.
Deleting rows doesn't give disk back
By the time eviction shipped, Denmark was at 98% disk with 3.8 GB free, and deleting rows turned out not to help. InnoDB keeps the pages of deleted rows for its own reuse; it doesn't return them to the operating system until the table is rebuilt with OPTIMIZE TABLE. A rebuild needs scratch space of one to one and a half times the table's size, and the tables were 56 GB for link and 38 GB for url, with less than 7 GB free all day. A real eviction pass deleted thousands of rows, and free space still went down, because the crawl kept inserting faster.
On 18 July I wiped the corpus instead. TRUNCATE drops and recreates a table's file immediately, which DELETE doesn't. The crawl was reseeded from Danish sites, dr.dk and da.wikipedia.org, instead of English Wikipedia, and it now grows under an admission throttle that slows crawling as the database approaches set disk limits, so it can't quietly run into the same wall again.
A detour through RocksDB
The link table was still the problem. By 22 July it had over 180 million rows (23.5 GB) and took about a million inserts an hour. It didn't fit in the 3 GB buffer pool next to the other hot tables, and the box had no RAM to spare. A write-heavy table bigger than memory is what LSM-tree storage is built for, so the plan was MyRocks: swap MySQL for Percona Server, then convert link to the RocksDB engine with pt-online-schema-change, which copies a live table in the background.
The Percona swap went fine. The conversion didn't. The copy stalled at about 8% for hours: high mysqld CPU, an almost idle disk, no lock waits. I never found the cause. The copy was aborted cleanly, the live link table was untouched, and the commit that reverted the memory settings says "root cause never fully diagnosed".
Cold turkey to PostgreSQL
Two days later the database changed completely. The migration guide gives the reason: MySQL wasn't suited to the graph work at the center of the system, PageRank and link traversal, and was causing memory pressure. A new dedicated database server with 24 GB of RAM and 8 cores took PostgreSQL and OpenSearch, and the other boxes reached it through SSH tunnels.
The strategy was deliberately blunt. Configuration tables such as crawl-ignore rules and crawl budgets were copied from MySQL. The corpus tables, URLs, links and hosts, were flushed, and the crawler was left to find everything again. That made it the second corpus reset in a week, and it was the cheaper choice: the configuration held decisions I had made by hand, while the corpus was something the crawler could always rebuild.
The commit log for 26 July is a list of dialect fixes: MySQL's integer booleans compared as IS TRUE, INSERT IGNORE rewritten for PostgreSQL, a DROP TEMPORARY TABLE that PostgreSQL doesn't accept.
Four boxes
On 9 August the database server also became a third place to run crawlers and indexers, which gave the fleet the shape it still has. The database server holds the system of record and a light pipeline. Denmark runs the full crawl, index and rank workers. The third box adds crawler and indexer capacity. Germany serves search. From 15 August OpenSearch became a two-node cluster across the database server and Germany, so search reads never cross between data centers.
By late August the link table in PostgreSQL was 77 GB, with 214 GB of the database server's 591 GB disk still free. The link table alone was bigger than it had been when Denmark's disk filled up, this time on hardware that could hold it.
A corpus this size raises a different question: of everything the crawler finds, what's worth keeping, and which version of a page should count? That's part 3.
Leave a comment
Comments are reviewed before they appear.