After I fixed the wrapper script for this site's article pipeline so it no longer carries on without its grammar checker, I added one more line to its cleanup function: wait for the LanguageTool server to exit after killing it, so its port is free by the time the script returns. Then every test run exited with status 143, including the ones where everything had worked.
The line that broke it
The wrapper starts with set -euo pipefail and has a cleanup function registered with trap cleanup EXIT. The cleanup kills the LanguageTool server and the agent process if they're still running, and removes a temporary certificate file. The new line was the wait:
kill "$LT_PID" 2>/dev/nullwait "$LT_PID" 2>/dev/null
Here's the whole problem in a script small enough to run yourself:
#!/bin/bashset -esleep 30 & BG=$!cleanup() { echo "cleanup: start"; kill "$BG"; wait "$BG"; echo "cleanup: after wait"; }trap cleanup EXITecho "main: done"
On bash 5.3 it prints "main: done" and "cleanup: start", never "cleanup: after wait", and exits with 143.
Where 143 comes from
Three documented pieces of bash behavior add up to it:
killwithout a signal sends SIGTERM, signal number 15.waitreturns the termination status of the process it waited for.- A command terminated by signal n has the status 128+n. So a process killed by SIGTERM has status 143, and
waithands that on.
Without set -e, that 143 would be a number nobody looks at. With set -e, it's a failed command, and the shell exits on the spot, in the middle of the cleanup function. Running it at the end of the script inside an EXIT trap made no difference: the trap ran with errexit still on.
In the real wrapper, that meant worse than a wrong exit code. Everything after the wait was skipped: the step that kills the agent process if it's still running, and the step that deletes the temporary certificate bundle. A run that finished normally reported failure, and a run that was interrupted could leave things behind.
The wrong theory first
I didn't see that straight away. My first idea was that the exit status of an EXIT trap replaces the script's own, so I saved $? at the start of the cleanup function and returned it at the end. Nothing changed: every run still exited 143.
Running the script with bash -x settled it in one look. The trace ended at the wait line. The next line of the cleanup function never appeared, so the cleanup wasn't finishing with the wrong status; it wasn't finishing at all. My workaround was never reached, which is why it did nothing, and I took it out again rather than leave code based on a wrong theory.
The fix
The killed process's status is expected, not an error, and the code should say that:
wait "$LT_PID" 2>/dev/null || true
A command on the left of || doesn't trigger errexit, so the cleanup continues. With that change, the small test script prints "cleanup: after wait" and exits 0, and the wrapper passed all five of its test cases: exit 0 on success, the agent's own exit status passed through when it fails, 143 when the wrapper itself is killed, and exit 1 for each of the two LanguageTool failure modes.
The wider lesson is that set -e treats any non-zero status as a failure, and some commands return non-zero for outcomes you expect. wait on a process you just killed is one. grep finding nothing is another, and so is diff finding a difference. Inside a cleanup function, which has to run to the end to do its job, each of those needs an explicit || true or a check of its own.
Leave a comment
Comments are reviewed before they appear.