A database upgrade tool refused to run, complaining about two disks that don't exist. The two volumes it was pointed at were sitting on the exact same physical disk. The error was correct anyway — from the tool's own point of view, they might as well have been on different planets.
Why the upgrade needed hard links in the first place
A major-version Postgres upgrade on a database this size normally means running the upgrade tool in a mode that copies every data file from the old cluster into a new one — safe, straightforward, and requiring roughly as much free disk space as the database itself already occupies. This particular box didn't have that much room to spare: not enough free space to hold a second full copy of a 211-gigabyte database alongside the original. The tool's alternative mode avoids that requirement by hard-linking files instead of copying them — creating a second reference to the same underlying data on disk, instant and free of extra space, rather than duplicating the bytes.
What a hard link actually requires
A hard link isn't a copy and isn't a shortcut — it's a second name pointing at the exact same data on the exact same physical filesystem. That last part is a hard requirement, not a preference: a hard link genuinely cannot span two different filesystems, even ones sitting on the same physical disk, because the underlying mechanism depends on both names referring to the same filesystem's internal bookkeeping of where the data actually lives.
Where the setup went wrong without anyone touching the disk itself
The old and new Postgres data lived in two separate Docker-managed volumes. Docker volumes for a given host are typically stored under one common location on the physical disk — meaning both of these volumes really were sitting on the same disk, no different from any other file on the same drive. But the upgrade ran inside a container, and each volume was mounted into that container as its own separate mount point. From inside the container's own filesystem view, two separate mount points look like two separate filesystems, regardless of what's actually true one level down on the real disk underneath them. The upgrade tool did exactly what a hard-link operation is supposed to do when asked to cross what looks, from where it's standing, like a real filesystem boundary: it refused, with a plain, accurate error naming exactly what it detected — Invalid cross-device link.
The fix: give it one view of what was already one disk
The actual disk layout never needed to change. What needed to change was how the container saw it. Rather than mounting the two volumes as separate targets, the fix bind-mounted their shared parent directory as a single mount point instead — the same real files, on the same real disk, now presented to the container as one filesystem rather than two. With both the old and new data visible under one mount, the hard-link operation had exactly what it needed: two paths that the container's own filesystem view now agreed were the same filesystem, because they actually were.
Why the error wasn't wrong, even though the assumption behind it felt wrong
It would be easy to read "cross-device link" and think the tool got confused about the disk layout. It didn't. The tool answered the question it was actually asked correctly: are these two mount points the same filesystem, as far as the container running right now can tell? They weren't, because two separate -v mounts genuinely do present as two separate filesystems inside a container, regardless of what's true about the physical disk one level below Docker's own volume management. The mistake wasn't in the tool's detection. It was in setting up the mounts in a way that didn't match what the physical layout actually was — a distinction that only matters the moment an operation, like a hard link, depends on the container's view and the disk's reality actually agreeing.
Leave a comment
Comments are reviewed before they appear.