Keeping thirty days of database backups sounds conservative. Nobody had actually multiplied the size of one backup by thirty and compared it against the size of the disk holding them, and by the time the gap surfaced, it wasn't backups that broke — it was the search index, silently unable to grow, on a machine that looked otherwise healthy.
By the end of part 2 the search engine had a corpus that could grow without filling a disk, on hardware that could hold it. That raised a different question. A broad crawl finds the same page under many URLs, mirrors and copies, thin and spammy pages, and whole sites in languages nobody will search for here.
My search engine has a small side feature: a gallery of people it has come across, grouped by what they're known for. Artists, musicians, politicians, journalists, and so on. On 2026-09-12 I noticed that the Artist bucket, a few entries away from Picasso, contained a well-known 20th-century dictator.
In September I started looking at replacing OpenSearch in my search engine with something closer to home, beginning with plain Postgres full-text search: tsvector columns, a GIN index, ts_rank_cd for scoring. The big question was latency at a realistic scale, so I built a benchmark on the Raspberry Pi I'm prototyping on.
A query rewrite that shipped an 8.1x speedup still had one unbounded edge case left in it. Finding the actual fix took trying two more reasonable-sounding optimizations first, measuring both, and watching both make things worse.
My search engine has a worker that discovers sitemaps: for each host that's due, it fetches the declared sitemap, walks any child sitemaps, and records what it found. For 60-odd hours in September, that worker made no progress at all. Every cycle it picked up the same host, crashed, and started over on the same host again. Every other host waiting for a sitemap check sat behind it.
Part 1 ended with search running on OpenSearch and a benchmark to measure ranking against. This part is about the other half of the problem: finding pages at all, and storing what the crawler finds.
Expanding a content-cleanup list by language code should be one of the safer changes a search engine can make. A live check of what was actually in the data turned up three entries that looked exactly like real languages and weren't — one of them the exact language the entire system exists to prioritize.
On 2026-09-12 I rescheduled 38 articles on this site with a single SQL statement. The JSON:API showed the new schedule straight away, and it kept showing it for 17 days. The Scheduler module, which is the thing that actually publishes them, never saw the change. It kept publishing on the old schedule the whole time.
A lot of the articles on this site over the last months have come out of one project: a web search engine I'm building from scratch.