A search engine's core ranking signal sat at zero for every single page in the corpus, fleet-wide, for 29 days. Nothing crashed. No error fired. The number that determines how results get ordered was simply wrong, silently, for a month, until someone went looking for a different reason entirely and found it by accident.
What the number was supposed to do
Every indexed page in this system carries an authority value — the output of a PageRank-style computation over the entire link graph, used directly in how search results get ranked against each other. A page with zero authority isn't a page that scored poorly; it's a page the ranking system has no signal about at all, indistinguishable from every other page also sitting at zero. When the whole corpus reads zero at once, the ranking signal isn't degraded. It doesn't exist.
Why nothing caught it for a month
The computation itself runs on a schedule — a periodic job that recalculates authority values and writes the results back into the corpus. A scheduled job that stops actually updating anything doesn't look like a failure from the outside: the application keeps serving search results, those results keep getting ranked by whatever value happens to be sitting in the database, and nothing about a stale value looks different from a correctly computed one unless someone specifically checks how recently it changed. The gap wasn't a bug in the computation itself — it was the complete absence of anything watching whether the computation had actually run recently at all.
What finally surfaced it
The discovery came out of an unrelated investigation into two other admin-panel problems: a statistics page that was slow enough to time out, and a "force refresh" option that quietly wasn't forcing anything. Neither of those two bugs was about ranking at all. But building the tooling to investigate them — closely inspecting what the admin panel actually reported about system health — surfaced the authority value's real staleness as a side effect of looking closely at numbers nobody had scrutinized in a while.
The fix that mattered most wasn't the original bug
Whatever specific reason had stopped the periodic job from succeeding got corrected, but the more consequential fix was structural: a new, explicit signal added to the admin health view, reporting exactly how many hours have passed since the last successful computation. That number turns "is the ranking data fresh" from a question nobody thinks to ask into a value visible on a page someone already checks. The actual failure that caused the 29-day gap was one specific bug, fixable and fixed. The 29 days of nobody noticing was a separate, more durable problem, and the one that actually got closed for good.
The two smaller bugs found in the same pass
The admin statistics page's slowness came from running an exact row-by-row count over tables tens of millions of rows deep, on every page load — replaced with a sampling-based estimate that reads the database's own internal statistics instead of counting every row directly, trading exact precision most of this feature never needed for a page that actually loads. The refresh flag turned out to be checked but never acted on: passing it changed nothing about whether a cached value got recomputed, so anyone who'd ever tried to force a fresh number had been silently served the same stale one regardless.
Why "nothing crashed" is the actual danger, not the absence of one
A system that fails loudly gets fixed quickly, because the failure itself creates the pressure to look. A system that fails by quietly not updating a number nobody has a reason to check regularly can run in that broken state indefinitely — 29 days here, potentially far longer without the unrelated investigation that happened to shine a light on it. The lesson isn't about this one specific computation. It's that any value a system depends on but never actively displays its own freshness for is a candidate for exactly this failure mode: correct-looking, silently stale, and invisible until someone happens to ask the one extra question — not "what does this say," but "when did this last actually change."
Leave a comment
Comments are reviewed before they appear.