The monitoring stack for my search engine ran Prometheus and Loki in Docker on one of the project's servers, with their ports published: 9090 for Prometheus and 3100 for Loki. The compose file still said "LAN only" next to both ports, from when the stack lived on a local network. On a public VPS, both were reachable from the whole internet, and neither has any authentication on the endpoints that receive data: anyone could write metrics and logs into them, or read them out. A host firewall that only allows web traffic doesn't change that, because Docker's published ports don't go through it.
Why ufw doesn't see published ports
Docker's documentation says it plainly: Docker and ufw use firewall rules in ways that make them incompatible. Docker routes container traffic in the nat table, so packets for a published port are diverted before they reach the INPUT and OUTPUT chains that ufw uses. The packets are routed before the firewall rules can be applied, which, in the documentation's words, effectively ignores your firewall configuration. A server where ufw status says only 80 and 443 are open will still answer on any port a container publishes.
The project's deploy notes list this as the first of "two things that bite", with the safest answer for the production setup: don't publish internal ports at all. The main application's production overlay only publishes the web server's ports; the database and search index are reachable only on Docker's internal network. The monitoring stack was the exception, because the project's other servers had to push their metrics and logs to it from outside.
DOCKER-USER: the supported place to filter
For ports that must stay published, Docker provides a chain for exactly this. Its documentation describes DOCKER-USER as a placeholder for user-defined rules, processed before Docker's own forwarding rules. The fix was a small script that adds rules there for ports 9090 and 3100:
- allow localhost
- allow Docker's internal range 172.16.0.0/12, because the stack's own collector reached the host's public address from inside a container, and those packets arrive with a 172.x source
- allow the server's own public address, in case that traffic is masqueraded to it
- allow the other servers that push their metrics and logs (one at first; a second was added to the list later)
- drop everything else to those two ports
Grafana talks to Prometheus container-to-container on the stack's own bridge network, never through the published port, so the rules don't affect dashboards.
One subtlety: which port --dport sees
Docker's documentation has a warning that matters for rules like these: by the time packets reach DOCKER-USER, they've already been through destination NAT. So --dport and -d match the container's internal port and address, not the ones the client connected to. To match the original destination, the documentation points to the conntrack extension, --ctorigdstport.
Here it didn't matter, because both ports were published one-to-one, 9090:9090 and 3100:3100, so the container port and the published port are the same number. With a mapping like 8080:80, a --dport 8080 rule in DOCKER-USER would match nothing.
A script you can review before it runs
The script defaults to a dry run that prints the rules it would add. apply inserts them, status lists them, and remove deletes every rule carrying the script's comment tag, which makes rolling back a single command. The insertion order is spelled out in the code: each port's DROP is inserted first and the allow rules on top of it, so the final order is the allows, then the drop, then the chain's default.
Its header also lists what to verify after applying, including that the push from the other server keeps arriving, and that a request to port 9090 from an untrusted host times out. It also says what it doesn't do: iptables rules don't survive a reboot until they're saved, for example with netfilter-persistent.
The rules were in place by mid-August, when a later fix extended the same allowlist to Grafana's port 3000 as defense in depth.
What generalises
On a Docker host, check what's reachable from outside, not what ufw says is allowed. The best fix is not publishing a port at all, or publishing it only to 127.0.0.1. Docker's documentation adds a warning there too: in releases older than 28.0.0, hosts on the same local network segment could still reach ports published to localhost. When a port has to be public, filter it in DOCKER-USER, and remember that the rules there see the container's side of the address translation.
Leave a comment
Comments are reviewed before they appear.