LanguageTool Server Failed With "Address Already in Use", and My Script Carried On Anyway

By Lars Nielsen ·

The script that drafts and schedules articles for this site starts a self-hosted LanguageTool server for each run, so every draft gets a grammar check before it's published. On 15 October, on the LFS laptop it now runs on, I found that LanguageTool hadn't started at all, and nothing in the script's output said so.

"Address already in use", in a log nobody reads

The wrapper script started LanguageTool's HTTP server on port 8081, with its output going to a log file in /tmp. That log ended in a Java stack trace:

  • java.lang.RuntimeException: Could not start LanguageTool HTTP server on localhost, port 8081
  • Caused by: java.net.BindException: Adressen er allerede i brug

The second line is Danish, because the laptop's locale is: "address already in use". Something else was already listening on 8081. ss -ltnp showed what:

  • LISTEN 0 512 127.0.0.1:8081 0.0.0.0:* users:(("llama-server",pid=1700790,fd=10))

A local llama.cpp server had taken the port. LanguageTool exited, and the port it was supposed to own kept answering requests, just not as LanguageTool.

Why the script carried on

The wrapper did wait for LanguageTool to come up. Its readiness loop asked the server for its list of languages once a second, for up to 20 seconds:

  • curl -s -o /dev/null -w "%{http_code}" "http://localhost:$LT_PORT/v2/languages" | grep -q 200

llama.cpp's server answers that path with a 404, so the check never passed. But when the 20 seconds ran out, the loop simply ended, and the script went on to start the agent that writes the articles. Nothing checked whether the loop had ended because the server was ready or because it had given up.

The agent is told a LanguageTool server is running on 8081, so it would have sent each draft there and got llama.cpp's errors back instead of grammar matches. I stopped that run before it had drafted anything, so no article from it went out without the check.

The fix: fail loudly, and recognise the right server

There were three problems, and the fix addresses each:

  • The port. LanguageTool moved to 8091. The prompt that tells the agent where the server is now gets the port filled in from the same variable, so it's set in exactly one place.
  • A busy port. Before starting anything, the script asks ss -ltnH "sport = :$LT_PORT" whether something already listens there. If so, it prints which process and exits with status 1.
  • Giving up silently. The readiness loop now sets a flag only when it gets a real answer, and stops early if the Java process dies. If the flag isn't set after 20 seconds, the script prints the end of LanguageTool's log and exits with status 1, before the agent ever starts.

The readiness check changed too. "Any HTTP 200" isn't proof that LanguageTool answered, so it now requires the language list to contain "longCode":"en-US", which a LanguageTool server includes and an unrelated service won't. I checked what the real response looks like before writing that check: LanguageTool answers in about a second with a JSON list of languages, and llama.cpp answers the same path with a 404 and an error object.

Testing the failure paths

I tested the wrapper with stand-ins for the agent and for ddev, so nothing touched the live site:

  • a normal run: exit 0, and the prompt pointed at the right port
  • port 8091 held by a placeholder Python web server: exit 1, with the process named
  • LanguageTool crashing on startup: exit 1 after about 2 seconds, with the log tail
  • the agent failing with status 7: the script exited 7
  • the wrapper killed partway through: exit 143, and the port was free afterwards

That last line hides a second bug I introduced while fixing the first, with its own lesson about set -e and cleanup traps. It's a story for another article.

The general point is older than this script: a readiness check that times out and then proceeds isn't a check, it's a delay. This is the same kind of lesson as the earlier kill-cascade bug in the same wrapper: the script did what its code said, and the code assumed the happy path.

Filed under

Leave a comment

Comments are reviewed before they appear.

Basic HTML allowed: links, emphasis, code and lists. Web addresses become links automatically.