On the database server of my search engine project, /etc/ssh/sshd_config said PasswordAuthentication no. The server accepted password logins anyway, and a bot on the internet was guessing the root password around the clock. Every attempt failed, but only because the password held, not because the setting did.
I found it in August during a security sweep of the project's four servers, by asking sshd what it was really running with instead of reading its config file.
The file said no, the server said yes
sshd -T is the command for that. The manual calls it extended test mode: it checks the configuration and prints the effective configuration, every setting with the value sshd will use, after all includes have been processed. On the database server, it printed:
passwordauthentication yes
The main file wasn't wrong. Something read before it was.
Why the drop-in won
Two rules in the sshd_config manual explain it, and together they're the opposite of what most config systems teach you to expect.
The first: "for each keyword, the first obtained value will be used." Not the last. If you're used to formats where a later line overrides an earlier one, you put your overrides at the bottom. sshd keeps the first value it sees for a keyword and ignores the rest.
The second is the Include directive, which pulls in other files and processes wildcard matches in lexical order. On these servers, the main sshd_config has its Include for /etc/ssh/sshd_config.d/*.conf near the top, so everything in that directory is processed before the rest of the main file.
One file in that directory wasn't mine: /etc/ssh/sshd_config.d/50-cloud-init.conf, a cloud-init drop-in that turned password authentication on. It was read first, so its value won, and my PasswordAuthentication no further down the main file was a line sshd read and discarded.
Nothing in the main file hints at this. If you open sshd_config, see the setting, and move on, you've checked the wrong thing.
The fix uses the same rule
The fix was a one-line file of my own in the same directory:
/etc/ssh/sshd_config.d/00-harden-password-auth.conf:PasswordAuthentication no
The name matters. Includes are processed in lexical order, so 00- sorts before 50-cloud-init.conf and before any other NN- file a provider might add later. The first-value rule that let the provider's file override mine now makes mine override theirs.
Then validate and reload, in that order, so a typo can't lock you out of a remote machine:
sshd -t && systemctl reload ssh
sshd -t is test mode: it checks the configuration and exits, and the reload only happens if that check passed.
Every server, checked the same way
One server with the problem meant the others could have it too. A second server had the same gap. The two remaining ones were checked the same way, by their effective configuration rather than their files, and fixed the same day.
Fixing it by hand once isn't enough on a fleet that gets reprovisioned, so the same change went into the project's Ansible setup as a small role. On the two servers Ansible reaches as root, it writes the file, then runs the validate-and-reload command whenever the file changed. On the two where Ansible connects as an unprivileged user without passwordless sudo, it can't write to /etc/ssh/sshd_config.d/ at all, so it prints the exact commands to run by hand instead of applying half a change. Running the role against the two servers already fixed by hand reported nothing changed, which is what you want from an idempotent role.
What to take from it
If you harden SSH on a cloud server, check the effective configuration, not the file:
sshd -T | grep -i passwordauthentication
And if you need a setting to stick, put it in a drop-in that sorts first, not at the bottom of the main file. In sshd, the first value wins.
Leave a comment
Comments are reviewed before they appear.