A search engine's core ranking signal sat at zero for every single page in the corpus, fleet-wide, for 29 days. Nothing crashed. No error fired. The number that determines how results get ordered was simply wrong, silently, for a month, until someone went looking for a different reason entirely and found it by accident.
For about a day in August, the Postgres server behind my search engine kept dying. Not slowing down, not timing out: the Linux OOM killer was terminating it, roughly every 20 to 90 minutes at the worst point, and each kill meant a crash-recovery cycle of about 90 seconds while Postgres replayed its write-ahead log.
A database upgrade tool refused to run, complaining about two disks that don't exist. The two volumes it was pointed at were sitting on the exact same physical disk. The error was correct anyway — from the tool's own point of view, they might as well have been on different planets.
Keeping thirty days of database backups sounds conservative. Nobody had actually multiplied the size of one backup by thirty and compared it against the size of the disk holding them, and by the time the gap surfaced, it wasn't backups that broke — it was the search index, silently unable to grow, on a machine that looked otherwise healthy.
By the end of part 2 the search engine had a corpus that could grow without filling a disk, on hardware that could hold it. That raised a different question. A broad crawl finds the same page under many URLs, mirrors and copies, thin and spammy pages, and whole sites in languages nobody will search for here.
My search engine has a small side feature: a gallery of people it has come across, grouped by what they're known for. Artists, musicians, politicians, journalists, and so on. On 2026-09-12 I noticed that the Artist bucket, a few entries away from Picasso, contained a well-known 20th-century dictator.
In September I started looking at replacing OpenSearch in my search engine with something closer to home, beginning with plain Postgres full-text search: tsvector columns, a GIN index, ts_rank_cd for scoring. The big question was latency at a realistic scale, so I built a benchmark on the Raspberry Pi I'm prototyping on.
A query rewrite that shipped an 8.1x speedup still had one unbounded edge case left in it. Finding the actual fix took trying two more reasonable-sounding optimizations first, measuring both, and watching both make things worse.
My search engine has a worker that discovers sitemaps: for each host that's due, it fetches the declared sitemap, walks any child sitemaps, and records what it found. For 60-odd hours in September, that worker made no progress at all. Every cycle it picked up the same host, crashed, and started over on the same host again. Every other host waiting for a sitemap check sat behind it.
Part 1 ended with search running on OpenSearch and a benchmark to measure ranking against. This part is about the other half of the problem: finding pages at all, and storing what the crawler finds.