Part 3 ended on an assumption: that the authority score behind every ranking was right. Authority comes from PageRank, computed over the whole link graph, and it's the part of this search engine that has never fitted comfortably anywhere. It started inside the database, moved off the box, went silently dead for a month, and was the reason for two graph-database experiments that both ended up removed.
A signal from the first week
PageRank went into ranking on 12 June, the same week as BM25 and before the move to OpenSearch. The golden set from part 1 justified it quickly: canonical pages tend to have high PageRank and foreign-language stub pages have almost none, and raising its weight from 0.5 to 5 was one of the changes that took MRR from 0.041 to 0.147. It settled at a weight of 3 in late June, as a factor in every result's score.
In the database, holding the database
The first implementation ran power iteration entirely in SQL, to keep the graph out of PHP's memory. By 22 June it had a cost nobody had planned for. Each pass scanned the link table, then 72.6 million edges, and held locks long enough that the indexers' own inserts into link timed out. The indexer logs filled with "Deadlock indexing" warnings. They weren't deadlocks. MySQL's counters showed lock-wait timeouts climbing by about 62 per pass while deadlocks stayed at 2, and one PageRank statement was caught holding 52 million row locks 103 seconds in.
The fix was to stop reading the live tables. PageRank now copies the edges into its own snapshot table and reads that with a weaker isolation level, and the indexers stopped timing out.
Computing on the change
That made the daily pass contention-free, but every phase was still a full scan that grew with the corpus. At about 100 million edges, each of the 20 iterations scanned and sorted every edge, spilling to disk.
Design 0005, built on 3 July, changed the approach. The normal pass is now incremental: it takes only the pages indexed since the last pass, rebuilds their edges and pushes the change outward through the graph. A full pass, the anchor, still runs as a correctness backstop, at the latest every 30 days.
Off the box
The anchor didn't fit the 4 GB server in Germany. Its working set was 5 to 20 times the 256 MB buffer pool, it thrashed for hours, and the first attempt got mysqld killed for running out of memory. Design 0007 on 4 July moved the computation elsewhere. The database exports nodes and edges, a separate engine written in Go computes PageRank on another box, and the result is imported back. On a 25-million-edge graph it matched the database's own result to within 1e-9. Ten days later the engine was changed to stream edges from disk instead of holding them all in memory, and peak memory settled at 1.4 to 1.9 GB.
Part 2's server split happened the same day, so for a while PageRank was local again and the off-node path was only a fallback. On 12 July I found that its cron entry, documented in the deployment guide, had never been installed.
A month at zero
The PostgreSQL switch on 24 July broke this pipeline completely, and nothing said so. On 22 August an unrelated audit of the database tables found that the authority column was 0 for every one of more than 111 million URL rows, and that the off-node pipeline hadn't completed once since 24 July. For 29 days the ranking had multiplied a weight of 3 by zero for every query. Searches still returned results, so from the outside nothing looked wrong. That incident has its own article.
The causes were five ordinary gaps: a memory limit sized for a smaller graph, no retry for a failed export chunk, no SSH keepalive on the long-running compute step, no retry within a scheduled run, and no signal anywhere that showed how old the PageRank results were. All five were fixed and deployed by 23 August, the last one as a staleness signal in the stats service.
The network wasn't the problem
The first run after the fix showed the next problem. The export, designed as a cheap local read, took over an hour to write about 450 MB. The script still ran from the German box, which no longer had a database, so all of its roughly 2,320 chunked queries crossed the network. Moving the script to the database server on 23 August made no difference: about 8 MB a minute in both places. Network latency only added up to about 50 seconds in total. The real cost was query time on a PostgreSQL server that the whole fleet's crawlers and indexers were using at the same time.
The answer to that, implemented on 30 August, is for the crawlers and indexers to back off while an anchor export runs, instead of PageRank competing with them for the same database.
Two graph databases
The PostgreSQL migration guide gave graph work as its main reason: MySQL wasn't suited to PageRank and link traversal. Its plan was to install Apache AGE, a graph extension for PostgreSQL, and port PageRank to its query language, Cypher.
That was the second graph experiment. The first had been a standalone graph database on its own box, set up as a bounded prototype with abandonment criteria written down in advance. It was torn down without a single real node loaded into it. Both evaluations are compared here.
AGE got further. It was installed and verified against real data on 13 and 14 August. Loading the graph through Cypher wasn't an option at 75.6 million nodes and 388 million edges, for reasons covered in a separate article. Porting PageRank itself was shelved on 14 August for two reasons. The Cypher version was extrapolated at 20 to 52 hours for the computation alone, against a Go engine that finished in under two hours and a schedule that runs every six. And the memory problem that motivated the whole migration had already been solved by the off-node Go engine. AGE was removed completely on 30 August.
On 13 September I wrote down the conclusion for the next time someone suggests a graph database: nothing in this pipeline needs fast variable-length path traversal, which is the one thing a graph database does better than SQL. The link graph stays in PostgreSQL.
Where it stands
PageRank now runs through the off-node pipeline every six hours, with retries at each step and a visible signal when it goes stale. A design for TrustRank, which would use the same graph to measure how much of a page's authority comes from likely spam, has been scoped since 31 August but not started.
The common thread in this part is that the hard problems were rarely in the PageRank math. They were in where it ran, what it competed with, and whether anyone would notice when it stopped. Part 5 is about that side of the project in general: running it.
Leave a comment
Comments are reviewed before they appear.