One morning in August, a worker container in my search engine had been using about 86% of a CPU core for nine hours and producing no output at all. docker top showed only one busy process: the shell script that runs the worker, not the PHP program it starts. The script was stuck in a loop around wait. The fix that went in at the time worked, but the commit admitted it couldn't reproduce the trigger. Writing this up, I found it, and it's a common shell mistake: treating an exit status above 128 as "interrupted".
The loop
Every worker runs a small POSIX shell script, worker.sh, that starts the PHP command in the background and waits for it. The waiting function looked like this:
wait "$_pid"_status=$?while [ "$_status" -gt 128 ]; dowait "$_pid"_status=$?done
The intent was sound. The script traps TERM and INT so it can finish the current pass before stopping, and POSIX says that when the shell is waiting with wait and a signal arrives for which a trap is set, wait returns immediately with a status greater than 128. So "above 128" can mean "the wait was interrupted, the child is still running, wait again".
What else is above 128
It can also mean something that never changes. POSIX says that if the process being waited for was terminated by a signal, the status is greater than 128. And a process can simply exit with a high status of its own: PHP exits with 255 after a fatal error, including running out of its memory limit, which I checked with a deliberately starved php -d memory_limit=2M.
Then the second part: once the shell knows a child's status, POSIX has wait return that status. Ask again, and you get the same answer. I tested it in Debian's dash, the /bin/sh the worker containers use:
- a child that exits with 255:
waitreturns 255, and again 255 on the second call - a child killed with
kill -9: 137, 137, 137 on three calls in a row - in both cases
kill -0on the PID reports the process is gone
Bash behaves the same way. So a loop that waits again "while the status is above 128" never ends if the child died from a signal or exited with 255. It doesn't block, either: each wait returns at once, which is what turns it into a busy loop using most of a core.
The project's notes record that the worker had hit an out-of-memory error in PHP, reported at a line in Doctrine's Result.php, right before it started spinning. A PHP fatal error means an exit status of 255, which fits exactly. The commit that fixed it says its isolated tests couldn't reproduce a status above 128; they delivered signals during a blocking wait, the case the loop was written for, not a child that had already died.
The fix, and why it works
The fix that went in caps the loop at ten retries, sleeps one second between them, and checks whether the child still exists before waiting again:
while [ "$_status" -gt 128 ] && [ "$_tries" -lt 10 ]; dokill -0 "$_pid" 2>/dev/null || breaksleep 1...
The kill -0 line is the one that matters for this trigger. kill -0 sends no signal; it only checks that the process exists. After a crash, the process is gone, so the loop exits on its first check and returns the real status, 255, to the caller, which already logs a failed pass and retries after its normal interval. The cap and the sleep are a backstop for anything else that might keep the status high. The commit says the fix was tested in isolation first: normal success and failure paths behaved exactly as before, and the worst case was now about ten seconds instead of forever.
One deployment detail: the script is the container's long-running entrypoint, so the fix only took effect when a container restarted. Only the affected worker was restarted straight away.
What generalises
A status above 128 from wait has several meanings: interrupted by a trapped signal, child killed by a signal, or child exited with a status like 255. Only the first one goes away if you wait again. If you retry a wait, first check that the child still exists with kill -0, and never write a retry loop without a cap or a delay. The day the loop's condition stays true, a loop without either becomes a process that burns a CPU core and says nothing.
It's a close relative of another wait surprise I hit in a different script: in both cases, the number wait returned meant something the code didn't expect.
Leave a comment
Comments are reviewed before they appear.