A freshly upgraded database reported itself perfectly healthy, right up until every other service that needed to talk to it got a flat connection refusal. The database wasn't lying about being healthy. It genuinely was — to itself, and to nothing else.
Why the setup skipped the usual safety net
Postgres's official container image doesn't just start the database — its own startup script does real first-boot setup work behind the scenes: opening the database up to network connections beyond the container's own loopback interface, and adding the access rules that let other containers on the same network actually connect. This particular upgrade ran a lower-level tool directly, initializing a fresh Postgres data directory by hand rather than going through a full container startup, in order to run the upgrade tool itself against both the old and new versions side by side. That approach never touches the official image's own startup script — and everything that script would normally have configured automatically was left exactly at Postgres's own conservative, network-closed default.
What that actually looked like from the outside
A basic health check against the new database, run from inside its own container, passed cleanly — that check happens over a local socket connection, which never leaves the box and was never affected by anything network-related in the first place. Every other container in the fleet, trying to reach the database over the ordinary Docker network the way it always had, got an immediate connection refusal instead. Two different ways of checking "is the database up" gave two contradictory answers, both genuinely correct from where each one was standing: yes, if you're already inside the same container; no, from literally anywhere else.
The fix, and the part that got caught before it shipped
Fixing the network side meant two real changes: telling Postgres to listen for connections beyond its own loopback interface, and adding an access rule permitting connections from the network the other containers actually live on. The first draft of that access rule was written broadly — permissive enough to guarantee it would work without needing to know the exact network details first. That draft got flagged before it went anywhere, by a safety check built into how changes get reviewed in this project, specifically because a rule that wide echoes an earlier, real incident on this same project where a database ended up reachable more broadly than intended. The rule that actually shipped was scoped down to the specific, real subnet the Docker network in question actually uses — checked directly rather than assumed — tighter than the first draft, and just as functional.
Why "it reported healthy" wasn't actually reassuring
The instinct to treat a passing health check as a green light makes sense most of the time, because most health checks are designed to represent the thing that actually matters to whoever's asking. This one didn't, not because it was broken, but because it was answering a narrower question than the situation needed answered — is the database process running and responsive, not is the database actually reachable by the rest of the system that depends on it. Those two questions usually have the same answer. The one time they don't is exactly the kind of gap that shows up right after skipping the normal setup path that would have otherwise kept them in sync automatically.
What actually closed the gap
Nothing about diagnosing this required guessing. A health check succeeding from inside the same container while every other container gets refused is a specific, recognizable signature — not "the database is broken," but "the database is fine and reachable exactly one way." Once framed that way, the fix followed directly: find what the official startup path would normally have configured, apply it by hand, and verify against the actual network the other containers use rather than opening things wider than necessary just to make the symptom go away.
Leave a comment
Comments are reviewed before they appear.