A method's own comment already explained exactly what its SQL needed to do, and why. The actual SQL did the opposite of what the comment described. Nobody wrote a contradiction on purpose — the comment was right, the implementation had drifted away from it, and the gap sat unnoticed while it quietly locked over ninety-eight percent of one entire class of database row into a state it could never recover from on its own.
What the comment said, and what the code did
The method's own docblock stated the requirement plainly: when a host gets queued again while its previous queue entry is still marked as claimed by a worker, the new insert needs to reopen that entry rather than silently losing the update — an explicit description of exactly why a specific conflict-handling behavior mattered. The actual SQL used a conflict clause that does nothing at all when a row already exists, rather than the update behavior the comment describes. The documented intent and the shipped behavior weren't subtly different. They were opposites.
Why "do nothing on conflict" isn't obviously wrong at a glance
Choosing to do nothing when a row already exists is a completely reasonable default in plenty of situations — it's often exactly the right call when a duplicate insert should simply be ignored. Nothing about the syntax itself looks like a mistake; it's valid, common, and unremarkable on its own. The bug only exists in the gap between this specific choice and what this specific method's own stated purpose required, and that gap is invisible unless someone reads the comment and the query together and actually checks that they agree.
How a claimed row was supposed to un-stick itself
A host's queue entry gets marked "claimed" when a worker picks it up to process. Under normal operation, that claim gets released once the worker finishes — a completely ordinary lifecycle. The comment's described behavior existed specifically for the abnormal case: a worker process that gets killed hard, mid-operation, skips its own cleanup step entirely, leaving a claim in place with nothing left running to ever release it. The intended fix for that specific situation was exactly the behavior the comment described — a later re-queue of the same host should be able to reopen a stuck claim, self-healing the situation without needing any separate cleanup process to notice and intervene.
What actually happened instead, at scale
With the conflict clause silently doing nothing instead, a claim that leaked this way could never be reopened by anything. Every subsequent attempt to re-queue that same host hit the same do-nothing conflict behavior and changed nothing, forever. Checked directly against the live data: 13,585 out of 13,819 rows of this type — over 98 percent — sat permanently stuck in a claimed state, the result of 28 distinct instances of the underlying hard-kill scenario accumulating over roughly three weeks, at a rate of two to three per day.
The test that had already caught this, unheeded
A specific automated test covering exactly this behavior had been failing for a while before the fix landed — and had been carried, explicitly, on a documented list of "known, pre-existing failures" not to worry about, on the assumption that it was an outdated assertion nobody had gotten around to updating. It wasn't outdated. It was correctly, continuously reporting a real, actively recurring production bug the entire time it sat on that list, un-investigated because it looked like exactly the kind of stale test failure every codebase accumulates a few of.
Why the fix matters less than the near-miss around it
The actual code change here is small — one conflict clause, corrected to match a comment that had been right all along. The more important fact is how close this came to staying unnoticed indefinitely: a self-inflicted data problem, silently compounding at two to three incidents a day, sitting behind a test result that had been deliberately reclassified as noise. The gap between "this test is failing" and "this test is telling us something true" isn't always obvious from the outside, and treating every long-standing failure as presumptively stale is exactly the assumption that let a real, quietly worsening bug keep accumulating stuck rows for three weeks before anyone looked closely enough to tell the two apart.
Leave a comment
Comments are reviewed before they appear.