A cryptominer showed up on production twice in one morning. The interesting part wasn't the malware — it was how ordinary the way in turned out to be, and how many wrong turns it took to find it.
It started with a load average. One of the boxes in my search engine's little fleet — a general-purpose server I use for git hosting and a graph-database experiment on the side — was pinned at forty, on two cores. That's not "busy," that's "something is wrong."
It was a cryptominer. Disguised as a security tool, of all things — the binary was named thg and shipped the exact command-line syntax of TruffleHog, a real, legitimate secret-scanner. Alongside it: a bandwidth-monetization daemon that quietly sells your spare upload to strangers, and a handful of directories in /tmp with names like .perf.c and .xdiag — close enough to real system paths that a bored sysadmin skims right past them.
The part where I made it worse
My first instinct was to cut the box off from the network entirely — allow the SSH session I already had open, drop everything new. Standard incident response. I did it in one shell command over one SSH connection.
Which meant the next command — the one that would've verified the isolation actually worked — opened a fresh connection. That connection was, by the rule I'd just written, exactly the kind of thing the rule was built to block.
I'd locked myself out of the box I was supposed to be defending.
The fix was almost funny: I still had a browser tab open with an active terminal session from before the lockdown. Established connections were exempt from my own rule — that's what "allow existing sessions" means, and I'd forgotten I was one of them. One line into that surviving session reopened the door I'd shut on myself, and the actual cleanup could start.
Small thing, but it's the kind of small thing that decides how an incident goes: not the sophistication of your response, but whether you've thought through what happens when your own tools work exactly as instructed.
Chasing ghosts
With the box isolated (for real this time), I went hunting for every trace of the thing. This is where it got genuinely confusing, twice.
First: I searched every container for processes matching the malware's name, using the process list's own search tool. It found nine matches in the monitoring container. Alarming — until I noticed the search command itself, sitting right there in the process list, matched its own search string. I was watching my own search find itself.
Second, and more interesting: monitoring agents like the one on this box typically get a read-only view into the host's process table, so they can report accurate metrics for every container running alongside them. That's completely standard and, for what it's worth, still the right design. But it means a process check run inside that monitoring container doesn't show you the monitoring container's processes — it shows you everything on the entire machine, with no way to tell which container anything actually belongs to.
What actually worked: Linux tags every process with a cgroup path that names its real container, even when the process list itself is host-wide and undifferentiated. Cross-referencing that path against docker ps gives you ground truth no amount of guessing from inside a container will:
$ cat /proc/<pid>/cgroup0::/system.slice/docker-4836d4e02848....scope$ docker ps --no-trunc --format '{{.ID}} {{.Names}}' | grep ^48364836d4e02848... php_fpm
Not the monitoring agent at all. The actual web application. That reframed the whole investigation — and turned out to matter a lot more than it seemed to at the time.
Rebuild, rotate, repeat
Standard playbook from there: full container rebuild from a clean image, not a restart (you can't trust a process tree that's already been compromised once). Every credential the box had ever held, rotated — including one I found sitting in the git history in plain text, a private key that happened to grant root access to the database server. That one had been there for a while. Embarrassing, in retrospect, and a good reminder that "we'll get to auditing the repo" is a task that ages badly.
I also went looking at the network topology while I was in there, and found the obvious-in-hindsight problem: every container on that box — the web app, a git server, an unrelated graph-database experiment — shared one flat network. A compromised web app could reach the git server directly, for no reason connected to its actual job. Splitting that apart cost about twenty minutes and closes an entire category of "well, now it can spread" scenarios for free.
Job done, I thought. Root cause unknown, but contained, hardened, moving on.
It happened again
A couple of hours later, mid-conversation about something else entirely: the exact same malware, on a completely different box — the actual database server this time, not the peripheral one.
Same binary name. Same fake TruffleHog command line. Same hidden directories. This wasn't a fluke or a coincidence — something structural was being exploited, twice, and I still didn't know what it was.
That's the moment an incident stops being "clean up and move on" and becomes "find the actual door, or this keeps happening."
The boring answer
Using the same cgroup trick, the second infection also traced straight back to the PHP application container. Both times. Never the crawler, never the git server, never anything else running on either box. That consistency was the real clue.
PHP applications commonly run behind a protocol called FastCGI, which a web server (nginx, in my case) speaks to on a specific port — traditionally 9000 — to hand off requests for the PHP interpreter to execute. It's not meant to face the internet directly. It has no authentication of its own; it assumes a web server is standing in front of it, deciding what's allowed through.
The actual root cause: that port was published to every network interface, with nothing blocking it. No default firewall policy, no restriction. On the box that ran this as a standing service, it had been reachable from the entire internet, continuously, for as long as it had been running.
A raw, unauthenticated FastCGI port, directly reachable — that's not a clever exploit. It's a documented, well-known technique with no need for any bug in the application at all. Boring, mechanical, and completely sufficient.
What confirmed it beyond reasonable doubt: my third, public-facing box had this exact same port publish in its base configuration too — but a previous, unrelated tightening pass had already reset its published ports to none at all, for reasons that had nothing to do with this. That box was never touched. Every other box, still wide open, got hit. One config line was the entire difference between "compromised" and "fine."
Before — every box, published everywhere:
ports:- "9000:9000"
After — loopback only; nginx still reaches it internally by container name regardless:
ports:- "127.0.0.1:9000:9000"
Don't panic, verify
With the actual door found and closed, one more habit paid for itself. A routine dependency audit turned up two serious, high-severity vulnerabilities in libraries the application used — the kind of finding that reasonably makes you want to drop everything and patch immediately.
Before doing that, I traced both. One lived entirely inside a debug tool that nothing in the deployment ever actually started — a real vulnerability, with genuinely zero path for anyone to reach it. The other required a specific security-configuration pattern the application had deliberately never used, for reasons written into the code's own comments, predating the vulnerability's public disclosure by a wide margin.
Both real bugs. Neither reachable. A severity score describes what a library could do in some configuration — not what your specific, actual deployment exposes. Worth ten minutes of tracing before a rushed dependency upgrade on live infrastructure, which carries its own real risk of breaking something that was working fine.
The bug hiding in the cleanup
One last thing, because it's a good closing joke on the whole day: while double-checking that the nightly database backups still worked after all the credential rotation, I found they'd been silently re-running every few minutes instead of once a day — for a boring, unrelated reason. The scheduling script computed "hours until the next run" using plain shell arithmetic on a zero-padded hour string, and a leading zero in shell arithmetic means "read this as octal." The digits 8 and 9 don't exist in octal. The script had been quietly crashing into an immediate retry, every single day, during two specific hours, for who knows how long — and silently computing the wrong answer the rest of the time, without erroring at all.
No harm done — the backups themselves were fine, and old ones were still being pruned correctly. But it's the same lesson as the FastCGI port, in miniature: the boring, structural bug is usually the real one, and it's usually hiding in the part of the system nobody's looked at recently because it's "just" scheduling, or "just" a default port setting nobody thought to question.
None of this was exotic. Not the way in, not the mistake mid-response, not the bug found on the way out. That's sort of the point.
Written the same day it happened, while the coffee was still working.