Backup Retention Math Nobody Had Actually Done

Keeping thirty days of database backups sounds conservative. Nobody had actually multiplied the size of one backup by thirty and compared it against the size of the disk holding them, and by the time the gap surfaced, it wasn't backups that broke — it was the search index, silently unable to grow, on a machine that looked otherwise healthy.

The setting nobody had revisited

A backup archive on one machine kept thirty daily database dumps at a time — a real, deliberate retention policy, set once and left alone. Thirty felt like a generous, safe buffer against needing to go back further than expected. Nothing about the number was wrong when it was chosen. What made it wrong later was a fact about the database itself that the number never accounted for: the database was growing, and every dump grows with it.

The growth that made the number stop working

Two real data points, eight days apart, told the story on their own: one dump measured 7.7GB; the next, eight days later, measured 10.4GB — and still climbing, tracking a corpus that was actively being crawled and indexed around the clock. Thirty dumps at a size that's actively increasing doesn't settle into some fixed total. It heads toward whatever thirty times the current size happens to be, and that total keeps moving upward every single day the retention count stays the same.

What thirty daily dumps actually added up to

Multiplying it out: thirty dumps at a growing multi-gigabyte size each put the total archive on a trajectory well past 270 gigabytes. The disk holding that archive had a total capacity of 196 gigabytes — smaller than the backup archive alone was heading toward, before accounting for anything else that machine needed to store.

What else lived on the same disk

The same disk wasn't dedicated to backups. It also held a search index's own data — a separate system with its own, independent growth, competing for the same finite space as a backup policy that had never been checked against real numbers. Two unrelated systems sharing one disk, one of them quietly consuming an ever-larger share because nobody had done the arithmetic on its retention setting since the day it was chosen.

How it actually surfaced

The problem wasn't caught by anyone reviewing disk usage proactively. It was found live, when disk usage crossed 86% full and the search index's own safety threshold — a setting that blocks it from allocating new replica shards once available space drops below a configured watermark — kicked in after what looked like an unrelated, routine restart. The restart didn't cause the shortage. The disk had been on this trajectory for a while; the restart was simply the first thing that happened to need the headroom the backups had already quietly consumed, and the first moment anything actually failed because of it.

Why the actual retention number came down to judgment, not a formula

The fix lowered retention from thirty dumps to seven. Seven isn't a value derived from a precise capacity calculation — it's a judgment call, informed by what this specific archive is actually for: this machine's own recent backups are only the deepest-history layer in the picture, since a separate, more frequent backup tier already exists elsewhere, and the corpus is also synced to an entirely independent location outside this fleet altogether. Seven days of the deep-history archive is enough headroom for its actual role, without carrying weeks of redundant growth that a different, faster-refreshing backup tier already covers.

The gap between "conservative" and "checked"

Thirty felt safe when it was picked, and it would have stayed safe indefinitely against a database that wasn't actively growing. The number itself was never audited against what actually determines whether it's safe — the real growth rate of what it's retaining, multiplied by how many copies it keeps, compared honestly against the real size of the disk holding it. A retention policy that sounded conservative and a retention policy that was actually safe turned out to be two different claims, and the gap between them stayed invisible until a disk almost ran out because of it.