CaRMen, my CRM, runs on a single VPS as two instances: the real one and a public demo. In October, I gave it a deploy script that can undo itself when a new version doesn't come up, including, when it has to, the database. The database is the interesting part.
The shape of a deploy
CI now builds the Docker images once, after the tests pass on the main branch, and tags them with the commit SHA. On the server, one script does the rest:
deploy.sh deploy <instance> <sha>: switch an instance to the images built for that commitdeploy.sh rollback <instance>: switch back to the version that ran beforedeploy.sh status: show each instance's health, current and previous version, and the last few deploys
A deploy pulls the images, takes a backup of the database (for the real instance; the demo resets itself anyway), switches the containers to the new tag, and runs a health check. The script remembers the current and previous tag for each instance in small state files and appends every attempt to a deploy log. A lock file with flock makes sure only one deploy runs at a time.
What "healthy" means
The health check polls every three seconds, for up to two minutes, until two things are true: the front page answers, and the installer's status endpoint reports "installed":true. The second condition is there because a broken database connection doesn't stop the front page from loading. That endpoint answers "not installed" when the database can't be reached, so it catches the failure that matters: an app that runs but can't see its data. The one exception is a brand-new instance, which has no users yet; for that, any answer from the status endpoint counts.
Why rolling back the code isn't enough
If the new version doesn't become healthy, the script switches back to the previous tag automatically. That part is simple. The hard part is the database schema.
CaRMen's production container runs doctrine:migrations:migrate every time it starts. So by the time a new version fails its health check, it may already have changed the schema. The previous version's code expects the previous schema. Switching the containers back without touching the database would replace one broken state with another: old code against a new schema.
The script handles that by recording the newest migration version (the highest entry in Doctrine's doctrine_migration_versions table) before the switch. After a failed health check it reads it again. If it changed, and there's a pre-deploy backup, it stops the app, drops and recreates the public schema, restores the backup, and only then starts the previous version. If the schema didn't change, the backup is left alone and only the containers switch back.
The rule is deliberately narrow. A restore throws away anything written to the database since the backup, so it only happens when the failed version has provably changed the schema, never as a precaution.
A manual rollback is different
deploy.sh rollback exists for the case where a version passed its health check and turned out to be wrong later. That rollback does not restore the database, because by then real data may have been written that a restore would lose. Instead, it prints the path of the newest pre-deploy backup and says that if the rolled-back version ran migrations, that's the file to look at. Restoring a production database is a decision for a person who knows what was written since, not for a script.
The backups have their own small rules: nightly and pre-deploy dumps are kept with separate retention, the script prints the path of the dump it made, and a failed dump doesn't leave a half-written file behind for a later restore to trust.
Tested by breaking it on purpose
Before relying on it, I tested three scenarios locally: a fresh deploy, a manual rollback, and a deliberately broken release that changed the schema and then crashed. In the broken case the script did what it's for: the health check failed, it saw the migration version had moved, restored the backup, rolled back to the previous version, and the data was intact.
Every failure path ends in a line in the deploy log, including the worst one, where the rollback itself fails and the instance needs manual attention. Nothing about it is clever. It's under two hundred lines of Bash, and the important part asks "did the schema change?" before deciding whether undoing a deploy means undoing the code, or the code and the data.
Leave a comment
Comments are reviewed before they appear.