A Shell wait Loop That Spun for Nine Hours: Exit Status Above 128 Doesn't Mean Interrupted

By Lars Nielsen ·

One morning in August, a worker container in my search engine had been using about 86% of a CPU core for nine hours and producing no output at all. docker top showed only one busy process: the shell script that runs the worker, not the PHP program it starts. The script was stuck in a loop around wait. The fix that went in at the time worked, but the commit admitted it couldn't reproduce the trigger. Writing this up, I found it, and it's a common shell mistake: treating an exit status above 128 as "interrupted".

The loop

Every worker runs a small POSIX shell script, worker.sh, that starts the PHP command in the background and waits for it. The waiting function looked like this:

  • wait "$_pid"
  • _status=$?
  • while [ "$_status" -gt 128 ]; do
  • wait "$_pid"
  • _status=$?
  • done

The intent was sound. The script traps TERM and INT so it can finish the current pass before stopping, and POSIX says that when the shell is waiting with wait and a signal arrives for which a trap is set, wait returns immediately with a status greater than 128. So "above 128" can mean "the wait was interrupted, the child is still running, wait again".

What else is above 128

It can also mean something that never changes. POSIX says that if the process being waited for was terminated by a signal, the status is greater than 128. And a process can simply exit with a high status of its own: PHP exits with 255 after a fatal error, including running out of its memory limit, which I checked with a deliberately starved php -d memory_limit=2M.

Then the second part: once the shell knows a child's status, POSIX has wait return that status. Ask again, and you get the same answer. I tested it in Debian's dash, the /bin/sh the worker containers use:

  • a child that exits with 255: wait returns 255, and again 255 on the second call
  • a child killed with kill -9: 137, 137, 137 on three calls in a row
  • in both cases kill -0 on the PID reports the process is gone

Bash behaves the same way. So a loop that waits again "while the status is above 128" never ends if the child died from a signal or exited with 255. It doesn't block, either: each wait returns at once, which is what turns it into a busy loop using most of a core.

The project's notes record that the worker had hit an out-of-memory error in PHP, reported at a line in Doctrine's Result.php, right before it started spinning. A PHP fatal error means an exit status of 255, which fits exactly. The commit that fixed it says its isolated tests couldn't reproduce a status above 128; they delivered signals during a blocking wait, the case the loop was written for, not a child that had already died.

The fix, and why it works

The fix that went in caps the loop at ten retries, sleeps one second between them, and checks whether the child still exists before waiting again:

  • while [ "$_status" -gt 128 ] && [ "$_tries" -lt 10 ]; do
  • kill -0 "$_pid" 2>/dev/null || break
  • sleep 1
  • ...

The kill -0 line is the one that matters for this trigger. kill -0 sends no signal; it only checks that the process exists. After a crash, the process is gone, so the loop exits on its first check and returns the real status, 255, to the caller, which already logs a failed pass and retries after its normal interval. The cap and the sleep are a backstop for anything else that might keep the status high. The commit says the fix was tested in isolation first: normal success and failure paths behaved exactly as before, and the worst case was now about ten seconds instead of forever.

One deployment detail: the script is the container's long-running entrypoint, so the fix only took effect when a container restarted. Only the affected worker was restarted straight away.

What generalises

A status above 128 from wait has several meanings: interrupted by a trapped signal, child killed by a signal, or child exited with a status like 255. Only the first one goes away if you wait again. If you retry a wait, first check that the child still exists with kill -0, and never write a retry loop without a cap or a delay. The day the loop's condition stays true, a loop without either becomes a process that burns a CPU core and says nothing.

It's a close relative of another wait surprise I hit in a different script: in both cases, the number wait returned meant something the code didn't expect.

Filed under

Leave a comment

Comments are reviewed before they appear.

Basic HTML allowed: links, emphasis, code and lists. Web addresses become links automatically.