GitHub Actions Can't SSH to My Server: Retrying on Exit 255 and a Server-Side Deploy Fallback

CaRMen's deploys run from GitHub Actions: once the tests pass on the main branch, a job builds and publishes the Docker images, and a deploy job connects to the server over SSH and tells it to switch to the new version. Within a day of setting that up, the deploy job started failing some of the time with SSH errors, while the images it was meant to deploy had been published just fine.

The cause wasn't in the workflow. My hosting provider drops SSH connections from some of the IP addresses GitHub's runners use, so the same job could connect on one run and fail on the next.

Retrying only what can be retried

The first part of the fix is a retry around the SSH call: three attempts, a 20-second connect timeout, and 20 seconds between attempts. The tricky part of retrying a deploy is not retrying the wrong thing. If the deploy itself ran and failed, for example because the new version failed its health check and was rolled back, running it again would be wrong.

ssh makes this distinguishable. Its manual says it exits with the exit status of the remote command, or with 255 if an error occurred in ssh itself. So the loop retries only on 255. Any other status, success or a failed deploy, ends the loop immediately and becomes the job's result:

• ssh ... -o ConnectTimeout=20 "$host" "$cmd" || rc=$?
• [ "$rc" -ne 255 ] && exit "$rc"

There's a small edge to that: a remote command that itself exits with 255 would look like a connection failure. The deploy script's own exit statuses are 1 and 2, so in practice it doesn't come up here, but it's worth knowing before copying the pattern.

When retrying isn't enough

If all three attempts fail, the job still fails, but it doesn't just fail. It writes a short section to the run's summary page, $GITHUB_STEP_SUMMARY, saying that the images are published and giving the exact command to run on the server to deploy the same commit. Whoever looks at the failed run gets the way around it on the same page.

Deploying from the server instead

For that fallback to be one command, the deploy script on the server had to learn to work out what to deploy by itself. deploy.sh deploy <instance> now takes the commit as optional:

• Without a commit, it fetches main and walks back through the last 20 commits until it finds one whose images CI has published. If the newest commit on main isn't published yet, it says so and uses the newest one that is.
• With a commit, it checks three things before doing anything: the commit exists, it's on main (git merge-base --is-ancestor, which exits 0 only if the first commit is an ancestor of the second), and its images are published.

"Published" is checked with docker manifest inspect against the registry for each of the three images. That asks the registry for the image's manifest without pulling the image, and the images are public, so it needs no login.

Then comes the step that matters most: the script checks out the chosen commit, detached, and re-runs itself from that checkout with exec. The compose file and the helper scripts in the repository have to match the images being deployed, and a server checkout that's a few commits behind or ahead would otherwise deploy new images with an old compose file. An environment variable marks the second run so it doesn't loop, and from there the deploy proceeds exactly as the GitHub-triggered one would, with the same backup, health check and automatic rollback.

deploy.sh status also got a line showing the newest published version, so it's visible at a glance when the server is behind what CI has built.

Normal path and fallback

The repository's deploy README now says it plainly: GitHub stays the normal path, and the server-side command is the fallback. Both end in the same script doing the same steps, which is what makes the fallback safe to use under pressure. There's no second deploy procedure to remember, only a second way to start the same one.

The general lesson is about where to put the logic. Because the deploy decisions live in a script on the server rather than in the workflow, losing the connection from CI costs one copy-pasted command, not a manual deploy.