CaRMen, my CRM, has a backup script that's 22 lines long. It runs every night from cron and before every deploy, and the deploy script restores from it automatically when a release changes the schema and then fails its health check. A script that a rollback depends on has to be right in the boring details. Three of them are worth spelling out.
1. A failed dump must not look like a backup
The core of the script is one pipeline:
docker compose ... exec -T db pg_dump -U carmen carmen | gzip > "$file.tmp"mv "$file.tmp" "$file"
The danger in a pipeline like this is that the shell's status for it is, by default, the status of the last command, here gzip. If pg_dump dies halfway, gzip still compresses what it received and exits successfully. The result is a perfectly valid .sql.gz file containing half a database.
I checked that: a producer that prints one line and exits with status 1, piped into gzip, without pipefail. The script carried on, exited 0, and left behind a gzip file that decompresses cleanly to that one line. Nothing about the file says it's incomplete.
The script avoids that in three layers:
set -euo pipefailat the top: withpipefail, the pipeline fails if any command in it fails, and-estops the script there.- The dump is written to
$file.tmp, and only renamed to its real name after the pipeline succeeded. On the same filesystem,mvis a rename, and the rename system call replaces the target atomically, so there's no moment where a reader finds a half-written file under the real name. - A trap removes
$file.tmpon exit, so a failed run doesn't leave debris that a later cleanup or restore might mistake for something.
With all three, the same failing-producer test exits with status 1 and leaves no file at all. Either a complete backup exists under its final name, or nothing new does.
2. Two kinds of backup, two retention counts
The script takes a label, nightly or predeploy, which goes into the file name. Retention is per label: the newest 14 nightly dumps and the newest 10 pre-deploy dumps per instance, pruned with
ls -1t "$dir/$instance-$label-"*.sql.gz | tail -n +$((keep + 1)) | xargs -r rm --
The earlier version had one label and pruned every .sql.gz in the directory to the newest 14. Once deploys started making their own dumps, that rule would have counted them against the same 14: a busy afternoon of deploys could have pushed the nightly backups out entirely, leaving two weeks of history as two hours. Splitting by label keeps the long tail of nightly dumps no matter how often you deploy.
Two small details in that line: ls -1t sorts newest first so tail -n +15 is everything after the fourteenth, and xargs -r doesn't run rm at all when there's nothing to delete.
3. Say where the backup is
The script's last line prints the path of the dump it just made. That's for the deploy script, which records it and, if the release fails after changing the schema, restores exactly that file. A backup that the rollback has to find by guessing, for example "the newest file in the directory", can pick up the wrong one when two things write backups close together.
The directory is also created with mode 700. Database dumps contain everything in the database, and they shouldn't be readable by every account on the server.
The general version
For any backup script: use set -o pipefail when the dump goes through a pipe, write to a temporary name and rename on success, clean up the temporary file on exit, keep separate retention for backups made for different reasons, and print what you made. None of it is clever, and all of it is the difference between a backup and a file that looks like one. I've written before about retention math for a different project; this is the part before the math, making sure each file is worth keeping.
Leave a comment
Comments are reviewed before they appear.