A reindex that was scheduled purely as insurance, against a problem that almost certainly wasn't going to happen this time, finished with a real, measurable bonus nobody had gone looking for: nearly a third of the database's total index storage disappeared, cleanly, as a side effect of doing something that was never actually about storage in the first place.
Why the reindex ran at all, when the risk it guarded against wasn't really present
A previous database incident — swapping a Postgres container to a different base image and getting collation-related data corruption as a result — had already cost real, expensive debugging time earlier in this project. This upgrade's base images, checked directly beforehand, turned out to actually match on the underlying operating system family, meaningfully lowering the odds of hitting that exact problem again. The reindex ran anyway, as a deliberate policy rather than a response to a specific detected risk this time: having paid once for assuming an image swap was safe without checking, the plan treated a full reindex as the default move after any database engine change, not something to skip just because this particular case looked lower-risk on paper.
What actually happens during a full reindex
Rebuilding every index in a database from scratch means discarding the existing on-disk structure entirely and constructing each one again from the current table data, using whichever version of the database engine's own comparison and storage logic is running now. Run in the mode that avoids locking the tables against normal reads and writes, this is slower than an in-place operation, but it lets a live system keep serving traffic the entire time the rebuild is happening in the background.
Where the extra thirty percent had been hiding
Indexes accumulate a specific kind of waste over time that has nothing to do with corruption or misconfiguration: as rows get updated and deleted, an index's old entries don't always get reclaimed immediately or efficiently, and the space they once occupied can linger, unused but still allocated, inside the index's own on-disk structure. This is ordinary, expected behavior for a database under continuous write load over a long period — not a bug, just an accumulating cost that only ever gets reclaimed by an operation that rebuilds the structure from a clean slate. Total index size before this reindex: 115 gigabytes. After: 80 gigabytes. Roughly 30 percent of what had been allocated for indexes was pure accumulated bloat, sitting there the entire time, invisible to anything short of an actual full rebuild.
Why this wasn't the point of doing it, and that's exactly why it counts
Nobody scheduled this reindex to reclaim disk space. It ran to close off a specific correctness risk that, on reflection, wasn't especially likely to be present this particular time. The bloat reclamation was a completely separate, unplanned benefit that happened to come bundled with an operation done for an entirely different reason. That distinction matters: a genuine bonus discovered as a side effect of doing the right thing anyway is a different, more durable kind of evidence than a benefit chased directly, because it didn't require anyone to correctly predict in advance that the bloat was even there to reclaim.
The actual case for "run it anyway" as a policy
Treating a defensive reindex as a default after any database engine change, rather than a case-by-case judgment call about how risky this particular change looks, meant not needing to be right about the risk assessment to get real value out of doing it. The correctness risk this reindex was meant to guard against may well not have materialized even without running it — but the thirty percent of reclaimed storage would never have shown up either way if the reindex itself had been skipped as unnecessary insurance. A policy that pays off even when the specific risk it was written for turns out to be low is a policy worth keeping as a default, independent of how good anyone's risk assessment happens to be in any one particular case.
Leave a comment
Comments are reviewed before they appear.