mysql -p Put My Database Password in ps aux, and docker run Kept It There

In July, while debugging a data export script for my search engine, I ran ps aux on the server as part of the diagnosis. The output contained the application's MySQL password, in plain text, because the script passed it on the command line. The output went into a session transcript, so the password had to be treated as leaked: I rotated it the same evening and redeployed the new one to all three servers that used it, before fixing the script itself.

How the password got into ps

The export script ran the MySQL client in a throwaway container, connected through a tunnel, like this:

• docker run --rm -i --network "container:..." mysql:8.0.43 \
• mysql -h 127.0.0.1 -P 3306 -u "$USER" -p"$PASSWORD" "$DB" "$@"

On Linux, a process's command line is readable through /proc/<pid>/cmdline, and by default every user can read every process's /proc directory. The hidepid mount option restricts that, but its default is the traditional behavior: everybody may access all of them. That's what ps reads.

MySQL's own documentation is direct about -p on the command line: convenient but insecure, because on some systems the password becomes visible to programs like ps. It also says MySQL clients overwrite the command-line password with zeros during startup, leaving only a brief interval in which it's visible.

That last sentence doesn't help when the client runs inside docker run. The mysql process inside the container can scrub its own arguments, but the docker run process on the host has the same password in its own command line, and it stays alive for as long as the container runs. I checked that with a harmless stand-in: a docker run with -pFAKESECRET123 among its arguments showed up in ps, secret included, for as long as the container was running. A long export means a long window.

The fix, and its caveat

The fix stopped putting the password on any command line. The script now writes it to a temporary file that only its owner can read, and hands Docker the file's path:

• ENV_FILE=$(mktemp)
• chmod 600 "$ENV_FILE"
• printf 'MYSQL_PWD=%s\n' "$PASSWORD" > "$ENV_FILE"
• docker run ... --env-file "$ENV_FILE" mysql:8.0.43 mysql -h 127.0.0.1 -P 3306 -u "$USER" "$DB" "$@"

The file is removed as soon as the client exits. The MySQL client reads the MYSQL_PWD environment variable by itself, so no -p is needed, and the only thing on docker run's command line is a file path. A process's environment is also better protected than its command line: access to /proc/<pid>/environ is governed by a ptrace access-mode check, the kernel's rules for which processes may inspect another, rather than being open to every user.

There's a caveat I should have looked up at the time. MySQL's documentation lists MYSQL_PWD as deprecated as of MySQL 8.0, to be removed in a future version, and warns that on some systems it's exposed to ps as well. It works with the 8.0 client this script pins, and it's much better than -p on the command line, but it isn't the end state. The documented alternatives are an option file with the password in a [client] section, passed with --defaults-extra-file and readable only by its owner, or a login path stored with mysql_config_editor. The temporary-file pattern above carries over to an option file almost unchanged.

What to check in your own scripts

The general version of this is easy to search for. Look for credentials passed as arguments, especially -p followed directly by a value, --password=, and tokens in URLs given to curl. Every one of them is readable by any user on the machine for as long as the process runs, and wrapping the command in docker run, sudo or ssh doesn't hide it; it adds one more process whose arguments show the same thing.

And when a secret does leak, rotate it first and fix the cause second. The rotation commit says the fix to the script was still open when the password had already been changed, which is the right order. A leaked password stays leaked no matter how quickly the code that leaked it is fixed.