A query rewrite that shipped an 8.1x speedup still had one unbounded edge case left in it. Finding the actual fix took trying two more reasonable-sounding optimizations first, measuring both, and watching both make things worse.
My search engine has a worker that discovers sitemaps: for each host that's due, it fetches the declared sitemap, walks any child sitemaps, and records what it found. For 60-odd hours in September, that worker made no progress at all. Every cycle it picked up the same host, crashed, and started over on the same host again. Every other host waiting for a sitemap check sat behind it.
Part 1 ended with search running on OpenSearch and a benchmark to measure ranking against. This part is about the other half of the problem: finding pages at all, and storing what the crawler finds.
Expanding a content-cleanup list by language code should be one of the safer changes a search engine can make. A live check of what was actually in the data turned up three entries that looked exactly like real languages and weren't — one of them the exact language the entire system exists to prioritize.
On 2026-09-12 I rescheduled 38 articles on this site with a single SQL statement. The JSON:API showed the new schedule straight away, and it kept showing it for 17 days. The Scheduler module, which is the thing that actually publishes them, never saw the change. It kept publishing on the old schedule the whole time.
A lot of the articles on this site over the last months have come out of one project: a web search engine I'm building from scratch.
My search engine has an admin page, /admin/host-space, where I can mark a whole host for eviction: every URL from that host gets dropped from the corpus on the next eviction cycle. I shipped it on 2026-09-13.
A query written to handle "a host's uncrawled pages" ran fine against every host in the corpus except one. That one host had six times as many uncrawled pages as the next largest, and a query shaped around the assumption of an ordinary-sized host turned a routine operation into a hundred-second stall.
My from-scratch Linux build boots into a KDE Plasma desktop now, and for a while it booted into one with a strange set of problems. Wi-Fi scans in NetworkManager didn't work. Power management didn't work. Login sessions weren't being registered properly. Three subsystems with nothing obvious in common, all broken at once.
A slow query got fixed with a rewrite — 8.1x faster, same result set, verified row for row. The more obvious fix, the one that would have addressed the actual physical reason the query was slow in the first place, got rejected. Not out of caution. Because it was measured first, and the measurement said no.