A query written to handle "a host's uncrawled pages" ran fine against every host in the corpus except one. That one host had six times as many uncrawled pages as the next largest, and a query shaped around the assumption of an ordinary-sized host turned a routine operation into a hundred-second stall.
A slow query got fixed with a rewrite — 8.1x faster, same result set, verified row for row. The more obvious fix, the one that would have addressed the actual physical reason the query was slow in the first place, got rejected. Not out of caution. Because it was measured first, and the measurement said no.
Two different graph databases got evaluated for the same search engine, three weeks apart, by way of two completely independent efforts. Neither one made it into production. The second attempt's own closing writeup ends up explaining, cleanly, why the first one never had a chance either — even though nobody set out to prove that when either evaluation started.
A graph-database extension got installed, tested, proven correct, and kept running for two weeks on the explicit reasoning that it might be useful later. Then it got removed entirely — not because anything about it was broken, but because two weeks of not needing it turned out to be the actual answer to whether it was worth keeping.
Loading data into a graph database usually means writing it in the graph database's own query language. For one real workload — 388 million edges — that turned out to be the one approach guaranteed not to finish. The fix wasn't a tuning flag or a bigger machine. It was skipping the query language entirely.
A migration meant to clean up duplicate rows took six real, distinct failures to get right. None of the six were the same mistake repeated — each one revealed a different way "deduplicate this data" is harder than it sounds once the data is large and the duplicates aren't isolated from each other.
Fixing one database incident accidentally created a second, unrelated one — and the second one hid the evidence needed to see whether the first one was actually fixed. One misordered database setting, changed to avoid a minor inconvenience, ended with an entire application's schema rebuilt from scratch in the wrong place.
A search engine's Postgres database swapped one Docker image for another — the same Postgres version, same data, same disk. Nothing about that should have touched a single row. It changed how the database compares strings, and for days, an index that reported itself perfectly healthy was quietly giving wrong answers to live production queries.
Right after a routine Postgres container swap, a quick health check showed zero live rows in three of the database's most important tables. For a moment, that looked exactly like catastrophic data loss. It wasn't — and the difference between panicking and checking came down to running one more, cheaper query before believing the first one.
A cryptominer showed up on production twice in one morning. The interesting part wasn't the malware — it was how ordinary the way in turned out to be, and how many wrong turns it took to find it.