A Backup Script Where a Failed Dump Can't Look Like a Backup: pipefail, Temp Files and Per-Label Retention

By Lars Nielsen ·

CaRMen, my CRM, has a backup script that's 22 lines long. It runs every night from cron and before every deploy, and the deploy script restores from it automatically when a release changes the schema and then fails its health check. A script that a rollback depends on has to be right in the boring details. Three of them are worth spelling out.

1. A failed dump must not look like a backup

The core of the script is one pipeline:

  • docker compose ... exec -T db pg_dump -U carmen carmen | gzip > "$file.tmp"
  • mv "$file.tmp" "$file"

The danger in a pipeline like this is that the shell's status for it is, by default, the status of the last command, here gzip. If pg_dump dies halfway, gzip still compresses what it received and exits successfully. The result is a perfectly valid .sql.gz file containing half a database.

I checked that: a producer that prints one line and exits with status 1, piped into gzip, without pipefail. The script carried on, exited 0, and left behind a gzip file that decompresses cleanly to that one line. Nothing about the file says it's incomplete.

The script avoids that in three layers:

  • set -euo pipefail at the top: with pipefail, the pipeline fails if any command in it fails, and -e stops the script there.
  • The dump is written to $file.tmp, and only renamed to its real name after the pipeline succeeded. On the same filesystem, mv is a rename, and the rename system call replaces the target atomically, so there's no moment where a reader finds a half-written file under the real name.
  • A trap removes $file.tmp on exit, so a failed run doesn't leave debris that a later cleanup or restore might mistake for something.

With all three, the same failing-producer test exits with status 1 and leaves no file at all. Either a complete backup exists under its final name, or nothing new does.

2. Two kinds of backup, two retention counts

The script takes a label, nightly or predeploy, which goes into the file name. Retention is per label: the newest 14 nightly dumps and the newest 10 pre-deploy dumps per instance, pruned with

  • ls -1t "$dir/$instance-$label-"*.sql.gz | tail -n +$((keep + 1)) | xargs -r rm --

The earlier version had one label and pruned every .sql.gz in the directory to the newest 14. Once deploys started making their own dumps, that rule would have counted them against the same 14: a busy afternoon of deploys could have pushed the nightly backups out entirely, leaving two weeks of history as two hours. Splitting by label keeps the long tail of nightly dumps no matter how often you deploy.

Two small details in that line: ls -1t sorts newest first so tail -n +15 is everything after the fourteenth, and xargs -r doesn't run rm at all when there's nothing to delete.

3. Say where the backup is

The script's last line prints the path of the dump it just made. That's for the deploy script, which records it and, if the release fails after changing the schema, restores exactly that file. A backup that the rollback has to find by guessing, for example "the newest file in the directory", can pick up the wrong one when two things write backups close together.

The directory is also created with mode 700. Database dumps contain everything in the database, and they shouldn't be readable by every account on the server.

The general version

For any backup script: use set -o pipefail when the dump goes through a pipe, write to a temporary name and rename on success, clean up the temporary file on exit, keep separate retention for backups made for different reasons, and print what you made. None of it is clever, and all of it is the difference between a backup and a file that looks like one. I've written before about retention math for a different project; this is the part before the math, making sure each file is worth keeping.

Filed under

Leave a comment

Comments are reviewed before they appear.

Basic HTML allowed: links, emphasis, code and lists. Web addresses become links automatically.