Parts 2 to 4 were about building the search engine: getting pages, deciding which ones count, computing authority. This last part is about what it takes to keep four servers running it, mostly as one person, and about the decision that brings the series back to where part 1 started: replacing OpenSearch, the one piece I didn't build.
The upgrade
PostgreSQL 15 had never been a choice. The image for Apache AGE, the graph extension from part 4, only existed for version 15. Once AGE was removed on 30 August, nothing held the database back, and the upgrade to 16 happened later the same day.
It was planned as a real maintenance operation, because the database server held a 211 GB corpus and was the only PostgreSQL instance for the whole fleet. Before anything was stopped there was a fresh dump, 18.8 GB and verified twice. Then every service that writes was frozen across three boxes, which turned up two workers that weren't on the stop list. pg_upgrade --link was the only mode that fitted in 124 GB of free disk, because it hard-links data files instead of copying them.
Two things broke along the way, both because initdb was run by hand instead of through the official image's entrypoint. The hard links failed because the old and new data directories sat on two separate mounts. The new cluster then came up healthy but only listened on localhost. The old data volume was kept for forensics, not as a rollback, because after a linked upgrade it can't be one. The real rollback was the dump.
Total fleet downtime was about 20 minutes. Afterward, every index was rebuilt with REINDEX DATABASE CONCURRENTLY as a precaution, after an earlier image swap had made an index quietly return wrong results. The rebuild reclaimed 35 GB: total index size went from 115 GB to 80 GB.
When the fix from part 4 stopped the crawl
On 8 September, a report that nothing had crawled for 30 minutes led to a look back at the past week. Crawling had been paused for roughly 100 to 120 of the last 190 hours.
The cause was the fix from part 4. Crawlers and indexers back off while a full PageRank anchor runs, and the anchor's import step had been getting slower every cycle. The documented baseline in late August was 64.5 minutes. By 8 September a single import had been running for more than 13.5 hours. The url and link tables were filling with dead rows from constant updates, and every cycle had more of them to wade through.
The fix the same day was a lower fillfactor, which leaves room on each page for updates, and more aggressive autovacuum settings on both tables. The day after, crawling was gated on an anchor that's actually running rather than on a cooldown window around it.
Small things that cost days
Most incidents in September were small changes with large effects, and several have their own articles. A new admin page for evicting whole hosts triggered repeated full scans of a 43-million-row table, because the query planner guessed 22,000 matching hosts when the real number was zero. A query rewrite that made one job 8.1 times faster was chosen over the more obvious fix because the obvious fix was measured first, and it still had one host it couldn't handle. Thirty days of backups turned out to be arithmetic nobody had done against the size of the disk holding them.
A backup that wasn't
The biggest finding of that period was a question asked on 31 August: do we have a backup of OpenSearch? Not really. Two snapshot repositories were registered. The only real snapshot was a one-off from 15 August, and the repository meant for ongoing backups had been broken since the day it was created.
That mattered more than it sounds, because of a rule from part 1: the database is the system of record, and OpenSearch is a derived index that can always be rebuilt. The indexer deletes the raw crawled content once a page is indexed, so a rebuild wouldn't come from local files. It would mean crawling all 7.76 million indexed pages again, on about 4.4 million hosts, which at the measured rate of 176,082 pages a day would take about 44 days. The index could be rebuilt in principle. In practice, losing it would have cost six weeks. A design to fix the snapshots was scoped the same day and hasn't been built.
Replacing the piece that wasn't mine
By 17 September the fleet was powered off for a pause in costs and maintenance, and the project moved to a Raspberry Pi for experiments. Instead of reinstalling OpenSearch there, I used the pause to look at replacing it. The design document gives two reasons. OpenSearch is the one major piece of this from-scratch search engine that isn't from scratch. It's also the most expensive part to operate: a dedicated two-node cluster, its own heap-tuning incidents, its own broken backup, and the only service excluded from every automated redeploy because it was too risky to touch casually.
Three options were considered: PostgreSQL's own full-text search, SQLite's FTS5 with one file per language, and a genuinely from-scratch inverted index with a compiled sidecar, like the Go engine that computes PageRank. The last one fits the project best and is by far the largest job: tokenizing and stemming per language, compressed postings, segment merging, a query planner and BM25F scoring. The design says that should be a deliberate choice, not a default.
The first version, on 18 September, used PostgreSQL full-text search. A benchmark on the Pi gave two wrong answers before a useful one, and the useful one was bad enough: about 2.9 seconds of pure CPU to rank 22,000 matching rows with every page in memory, because PostgreSQL fetches and scores each matching row separately. The same day it was rewritten onto pg_search, an extension that brings Tantivy's BM25 into PostgreSQL. That's an honest irony to keep: the replacement for an external search engine is, so far, another one, just running inside the database I already operate.
Where it stands
The new searcher returns results on the Pi, and the Pi is now the intended target for up to 10 million pages, not a stepping stone. It hasn't been measured against the golden set yet, and that is the next step, before any tuning. If PostgreSQL's limits turn out to be in scale or latency rather than ranking quality, the from-scratch index becomes the informed next step.
That's the same rule the project started with in June, when the first commit of a real benchmark measured MRR 0.041. Every big decision since has been made the same way: measure first, then decide. This series ends with that measurement still to be made.
Leave a comment
Comments are reviewed before they appear.