Skip to main content

Deploying by hand

The pipeline (ci.md) deploys on every merge to main. This page is for when you're driving it yourself: the pipeline isn't wired up yet, a secret is wrong, GitHub is down, or you're bringing up a fresh VPS and want to watch each step. The runner is the same script CI calls, so nothing here is a parallel code path that can rot.

Everything below runs on the VPS, in /srv/motorph, as the operating user.


The short version

cd /srv/motorph
git pull --ff-only origin main # compose files, Caddyfile and deploy.sh come from git
./deploy/deploy.sh prod preflight # read-only: does this deploy stand a chance?
./deploy/deploy.sh prod deploy # no tag = this checkout's HEAD

That's it. deploy.sh takes a pre-deploy pg_dump, brings up the edge, pulls the images, gates on the backend healthcheck, and rolls back automatically if the new tag never turns healthy.

Always run preflight first

deploy/preflight.sh (also deploy.sh <env> preflight) touches nothing — no containers, no files, no firewall. It answers "will this deploy fail, and why", in about ten seconds, instead of finding out three minutes in with a stack half-recreated. Every check in it corresponds to a real failure from the first production bring-up:

CheckThe failure it catches
Compose-required varsCompose names only the first empty ${VAR:?}, so a half-filled .env takes several attempts to fix. Preflight lists them all at once.
DOMAIN formatIt's a Caddy site address; the Caddyfile builds stage.{$DOMAIN} and CORS_ALLOWED_ORIGINS=https://${DOMAIN} from it. https://www.example.com produces stage.https://www.example.com and an edge that won't start.
Container egressAn unqualified -j DROP in DOCKER-USER (vps-guide.md §2) drops containers' outbound traffic too. Certificates never issue, SMTP times out, docker build fails on Maven Central — while the host's own connectivity is perfect. This one is invisible without a deliberate test.
iptables persistenceThose rules don't survive a reboot unless saved. A reboot silently restores the broken state.
Ports 80/443Anything else holding them (an old Traefik, a host nginx) fails ensure_edge after the deploy has begun.
Container namesThe compose files pin container_name; a dev stack run by hand from a home directory owns the same names and blocks compose up outright.
Image in GHCRA tag typo or a stale docker login should fail here, not after the DB backup.
Dirty working treeCI's git pull --ff-only origin main refuses to update a tree someone hand-edited.

Preflight exits non-zero only on hard failures; warnings are printed and don't block.

Choosing a tag

Images are tagged sha-<12 hex> by CI, matching the commit they were built from. deploy.sh <env> deploy with no tag uses this checkout's HEAD — correct whenever you've just pulled and want to ship what you pulled.

Pass one explicitly to ship something else:

./deploy/deploy.sh prod deploy sha-cb67bb9c4285

The tag must already exist in GHCR — deploying does not build. If CI hasn't built that commit yet, there is nothing to deploy; preflight says so.

After the deploy

curl -o /dev/null -w '%{http_code}\n' https://$(grep -m1 '^DOMAIN=' .env | cut -d= -f2)/
docker exec motorph_caddy ls /data/caddy/certificates/acme-v02.api.letsencrypt.org-directory/
docker compose --project-directory . --env-file .env -f deploy/docker-compose.prod.yml logs --tail 50 motorph_payroll_backend

Expect 200, and a certificate directory per hostname Caddy has a site block for. A hostname with no site block has no certificate, which Cloudflare reports as 525 — that is a missing route, not a broken origin.

When it goes wrong

deploy.sh rolls production back to the previous tag by itself if the health gate fails, and still exits non-zero — a red deploy always means "look at this", even when the site is fine. To move between tags deliberately:

./deploy/deploy.sh prod rollback # the previously deployed tag
./deploy/deploy.sh prod rollback sha-abc123 # a specific one
./deploy/deploy.sh prod backup # pg_dump only, any time

Rollback flips images only — Flyway migrations are never reverted. The old jar boots against the newer schema. When the schema itself is the problem, the answer is restoring a dump: vps-guide.md §10.

The .env files are disposable

/srv/motorph/.env and .env.stage are rewritten from the PROD_ENV_FILE and STAGE_ENV_FILE GitHub Secrets on every pipeline deploy. Editing them on the VPS is fine for an immediate manual deploy, and lasts exactly until CI next runs.

Anything you fix on the VPS must be fixed in the Secret too, or the next pipeline deploy silently reverts it. That includes DOMAIN, all MAIL_* values, and every credential.

IMAGE_TAG is the exception: deploy.sh maintains it in the file so a plain docker compose ps resolves, and the deploy always pins it explicitly anyway. Leave it out of the Secret.

Bringing up a VPS that has never deployed

Order matters, and preflight will tell you which step you skipped:

  1. VPS hardening and the Cloudflare-only firewall rules — vps-guide.md §2. Read the -i $EXT_IF warning there; getting it wrong disables all container networking in a way that looks like anything but a firewall.
  2. docker login ghcr.io with a read:packages PAT — private packages won't pull otherwise.
  3. Fill .env from deploy/.env.prod.example, chmod 600.
  4. DNS: the apex and every hostname the Caddyfile serves (stage., www., any client subdomains) pointing at the box.
  5. ./deploy/deploy.sh prod preflight until it's clean.
  6. ./deploy/deploy.sh prod deploy.

Cloudflare's SSL/TLS mode should be Full (strict) — Caddy holds real Let's Encrypt certificates, so there's no reason to accept less. If you see 521, nothing is listening on 443; if you see 525, something is, but it has no certificate for that hostname.