Production VPS guide

A from-scratch runbook for the hosted product: one Ubuntu VPS running two environments of the shared multi-tenant stack — production at <domain>, staging at stage.<domain>, each with its own Postgres — plus a documentation site, all behind one Caddy edge (automatic HTTPS via Let's Encrypt) fronted by a CDN in Full (strict) TLS mode. Deployment is automated: GitHub Actions builds images once per merge, deploys staging, gates on health, and promotes the same tag to production in the same run.

It is written to be followed on a new box — including the case where the old one is gone or expiring. If you are migrating, start at §0 while the old host is still alive; everything else assumes nothing exists yet.

Compose-file and Dockerfile internals are in docker.md; the per-workflow reference is ci.md; driving deploys by hand is manual-deploy.md. Related decisions: ADR-0013, ADR-0014, ADR-0015, ADR-0017.

Internet
   │  HTTPS (CDN edge certificate)
   ▼
CDN (proxied DNS, Full-strict TLS to origin)
   │  HTTPS :443 ── firewalled so only the CDN's ranges get through (§4)
   ▼
VPS ── Caddy :80/:443 (edge project; the ONLY published ports)
   │
   ├─ <domain>          PRODUCTION  ─ /api,/ws → backend :8080 → its postgres
   │                                 └ else    → frontend (static SPA)
   ├─ www.<domain>      permanent redirect to the apex
   ├─ stage.<domain>    STAGING     ─ same shape, its own postgres + Mailpit
   ├─ docs.<domain>     Documentation Central (static, stateless)
   ├─ grafana.<domain>  Grafana, behind a basic-auth gate (optional overlay)
   └─ acme.<domain>     legacy per-client stack, if you run one

Only Caddy's ports are exposed to the host. Both databases, both backends, both frontends and the docs site are reachable only on the internal Docker network, and that fact is load-bearing (§9).

Honesty box — read before trusting this setup

• One stack serves every tenant. Isolation is rows, not containers: a Hibernate tenant filter plus Postgres row-level security (ADR-0013). RLS only binds if the app connects as the right database role — §10 and §11 verify it, because getting it wrong fails silently, not loudly.
• Deploys have downtime — seconds to a couple of minutes while compose recreates containers and the backend health-checks. A deliberate trade against blue/green complexity at this scale.
• Exactly one backend instance. Rate-limit buckets, ALTCHA replay protection and signup verification codes are all in-memory. Scaling horizontally silently weakens all three; see scaling.md.
• Deploys are test-gated, shallowly. deploy.yml calls tests.yml first, so backend and frontend unit tests must pass before an image is built. Frontend lint is continue-on-error, and the end-to-end suite is not in this path — only a smoke spec gates PRs.
• Rollback flips image tags, never the schema. Flyway has no down migrations. §12 explains exactly what rollback does and does not undo.
• Two stacks on one box need RAM. Two JVMs, two Postgres, Caddy, Mailpit and the docs site want a 4 GB VPS minimum (8 GB is comfortable).

---

0. Before you lose the old box

Skip this section if you are building the very first host. If you are replacing one, do it now, while the old box still answers SSH — most of what follows cannot be recovered afterwards.

What exists only on that machine:

Thing	Where	Recoverable elsewhere?
Production and staging data	Docker volumes	No
.env / .env.stage	/srv/motorph/	Only from your password manager. GitHub Secrets are write-only — you cannot read PROD_ENV_FILE back out, ever
Which image tag is live	.deploy-state*	Rediscoverable from the Actions history, tediously
Grafana dashboards, Prometheus/Loki history	Docker volumes	No (and usually not worth moving)
TLS certificates	motorph_caddy_data volume	Don't bother — the new host issues its own

The env files are the sharp edge. A database dump without them restores to a stack nobody can log into: the JWT signing key, the ALTCHA key and the app_runtime password all live there, and the dump's password hashes are meaningless without the secret that signs sessions.

One command takes all of it:

cd /srv/motorph
deploy/backup.sh                        # databases + env files + deploy state
deploy/backup.sh --with-observability   # ...and the Grafana/Prometheus/Loki/Tempo volumes

It writes a single timestamped archive to ~/motorph-backups/ and refuses to write inside the repo, so a bundle of production secrets can never end up in a commit. Then get it off the machine — a backup on the box you are about to lose is not a backup:

# from your laptop
scp deploy@<OLD_VPS_IP>:~/motorph-backups/motorph-<stamp>.tar.gz .

Verify the copy before you cancel anything:

tar -tzf motorph-<stamp>.tar.gz          # env.prod, env.stage, prod.sql.gz, stage.sql.gz, MANIFEST
tar -xzOf motorph-<stamp>.tar.gz --wildcards '*/prod.sql.gz' | gunzip | grep -c '^CREATE TABLE'

A non-zero table count and both env files present means you can rebuild. §11 puts the archive back.

1. What you are building

Four compose projects, all committed to the repo, pulled by git, never hand-written on the VPS:

Project	File	Containers	Published ports
edge	deploy/docker-compose.edge.yml	motorph_caddy	80, 443, 443/udp — the only public ones
prod	deploy/docker-compose.prod.yml	motorph_payroll_db / _backend / _frontend	none
stage	deploy/docker-compose.stage.yml	motorph_stage_db / _backend / _frontend / _mailpit	Mailpit UI on 127.0.0.1:8025 only
docs	deploy/docker-compose.docs.yml	motorph_docs	none

All four join one shared external Docker network, motorph_payroll_network, which is how a single Caddy routes to all of them by container name. Two consequences that bite people in this order:

1. deploy.sh creates that network; the docs deploy refuses to. A docs deploy on a box where the network does not exist dies on purpose, because an empty network created there would be the wrong one and the site would be unreachable in a way that looks fine. So the docs stack can never be the first thing you deploy.
2. Even a staging deploy reads production's .env. The edge is brought up with --env-file .env whatever you are deploying, because DOMAIN and the Grafana gate credentials live there. .env must exist before anything deploys, including a stage-only deploy.

The observability overlay (§13) is optional and prod-only.

2. Provision the server

Ubuntu LTS, 4 GB RAM minimum. Everything in §2–§5 is what deploy/bootstrap.sh automates:

# as root on the new box
git clone <repo-url> /tmp/motorph-bootstrap
/tmp/motorph-bootstrap/deploy/bootstrap.sh --ssh-key ~/your-key.pub

It is idempotent, and --dry-run prints every command without touching anything. If you prefer to do it by hand, or want to know what it did, the rest of §2–§5 is the same work spelled out.

Create the operating user:

adduser deploy
usermod -aG sudo deploy

From your local machine, install your key and confirm it works before locking sshd down:

ssh-copy-id deploy@<VPS_IP>
ssh deploy@<VPS_IP>

Then disable root and password logins:

sudo sed -i 's/^#\?PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
sudo sed -i 's/^#\?PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
sudo systemctl restart ssh

Brute-force protection and automatic security patches:

sudo apt update
sudo apt install -y fail2ban unattended-upgrades
sudo systemctl enable --now fail2ban
sudo dpkg-reconfigure -plow unattended-upgrades

3. Install Docker

curl -fsSL https://get.docker.com | sudo sh
sudo usermod -aG docker deploy

Log out and back in for the group change, then confirm with docker ps. Membership in the docker group is effectively root on this machine — anyone in it can mount the host filesystem into a container. Fine for a single-operator box; don't add anyone you wouldn't give root.

4. Host firewall

Host services first:

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow OpenSSH
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow 443/udp    # HTTP/3
sudo ufw enable

Docker vs UFW, the honest version. Docker publishes container ports by writing its own iptables rules, which sit in front of UFW — so once Caddy is up, its ports are reachable regardless of UFW's opinion. UFW fully protects host services (SSH); restricting 80/443 to the CDN has to happen in the DOCKER-USER chain instead.

Put the egress guard in first. This is the most expensive mistake this deployment has made.

EXT_IF=$(ip route show default | awk '{print $5; exit}')   # e.g. eth0

# 1. Anything not arriving on the external interface returns immediately.
sudo iptables -I DOCKER-USER 1 ! -i "$EXT_IF" -j RETURN

# 2. Now the inbound restriction is safe to add.
for p in 80 443; do sudo iptables -A DOCKER-USER -i "$EXT_IF" -p tcp --dport "$p" -j DROP; done
sudo iptables -A DOCKER-USER -i "$EXT_IF" -p udp --dport 443 -j DROP
for ip in $(curl -s https://www.cloudflare.com/ips-v4); do
  for p in 80 443; do
    sudo iptables -I DOCKER-USER 2 -i "$EXT_IF" -p tcp --dport "$p" -s "$ip" -j ACCEPT
  done
  sudo iptables -I DOCKER-USER 2 -i "$EXT_IF" -p udp --dport 443 -s "$ip" -j ACCEPT
done

Why the guard, and why -i $EXT_IF is not optional. DOCKER-USER is traversed by every forwarded packet, not just inbound ones. An unqualified --dport 443 -j DROP therefore also kills each container's outbound HTTPS — its source is a 172.x address, which no CDN range matches. The failure looks nothing like a firewall: certificate issuance hangs on the ACME API, docker build dies with "Network is unreachable" from Maven Central, SMTP and billing calls time out — while the host has perfect connectivity and every port scan looks right. The port-80 copy is worse: it drops Caddy → frontend, so the SPA breaks even after a valid certificate exists.

Diagnosis is sudo iptables -S DOCKER-USER — look for DROP rules with no -i. The rescue on a live box is the guard itself: sudo iptables -I DOCKER-USER 1 ! -i "$EXT_IF" -j RETURN.

If the VPS has public IPv6, repeat with ip6tables and the CDN's v6 list. These rules do not survive a reboot on their own:

sudo apt install iptables-persistent
sudo netfilter-persistent save

This is not optional hardening. §9's client-IP scheme trusts the forwarded-IP header precisely because only the CDN can reach the origin. Skip the allowlist and that header is attacker-settable on direct hits.

Side effect: a direct curl https://<VPS_IP> now times out. Test through the domain, not the IP.

5. DNS and the CDN

Point six records at the box. The first three are the product; the rest are separate site blocks in the Caddyfile, and each needs its own record or Caddy cannot get a certificate for it.

Type	Name	Content	Proxy	What it serves
A	@	<VPS_IP>	Proxied	production
A	www	<VPS_IP>	Proxied	redirect to the apex
A	stage	<VPS_IP>	Proxied	staging
A	docs	<VPS_IP>	Proxied	Documentation Central
A	grafana	<VPS_IP>	Proxied	Grafana, if you run the overlay (§13)
A	acme	<VPS_IP>	Proxied	legacy per-client stack, only if you run one

www exists so the TLS handshake has a certificate to present; without the record and the site block, a visitor typing www. gets a CDN 525 rather than a redirect. The frontend calls the API with relative /api URLs on the same origin, so there is no api. subdomain and no CORS in play.

In SSL/TLS → Overview, set the mode to Full (strict). Caddy obtains a real Let's Encrypt certificate (§8), which is what "strict" verifies. Flexible is wrong here in a way that looks like a bug: the CDN speaks HTTP to an origin that redirects HTTP to HTTPS, and the browser gets a redirect loop.

Three bootstrap traps:

• Leave "Always Use HTTPS" off until the first certificate has issued. Caddy 301s HTTP→HTTPS itself; what it needs port 80 for is the ACME HTTP-01 challenge, and a CDN-side redirect of that challenge to an origin with no certificate yet is a chicken-and-egg 526.
• 521/522 right after first boot usually means Caddy has not obtained its certificate yet — check docker logs motorph_caddy. As a last resort switch the record to DNS-only (grey cloud), let the certificate issue, then re-proxy. Note that once §4's firewall is in place a grey-cloud record is unreachable for you too, so open the DOCKER-USER chain if you go this route.
• A wildcard * record makes every subdomain resolve, including ones no site block serves. The symptom is then 525 (TLS handshake failed) rather than NXDOMAIN, which reads like a broken certificate when in fact the edge has never heard of that hostname. If a brand-new subdomain 525s, check that Caddy actually loaded a site block for it (§10) before touching TLS settings.

6. Get the code onto the box

The workflows all cd /srv/motorph, so use exactly that path:

sudo mkdir -p /srv/motorph
sudo chown deploy: /srv/motorph
git clone <repo-url> /srv/motorph
cd /srv/motorph

What lives where — the split matters for understanding deploys:

Path	What	Comes from
/srv/motorph/deploy/	compose files, Caddyfile, deploy.sh, docs-deploy.sh, backup.sh, restore.sh	git — git pull on every deploy
/srv/motorph/.env	production secrets + IMAGE_TAG; also DOMAIN/ACME_EMAIL/Grafana gate for the edge	the PROD_ENV_FILE secret, rewritten by every pipeline run (§7)
/srv/motorph/.env.stage	staging's own secrets + its IMAGE_TAG	the STAGE_ENV_FILE secret, same rule
/srv/motorph/.deploy-state*	CURRENT_TAG/PREVIOUS_TAG per environment	deploy.sh / docs-deploy.sh
/srv/motorph/backups/	pre-deploy pg_dumps	deploy.sh
GHCR ghcr.io/<owner>/motorph-payroll-{backend,frontend,docs}	the images	GitHub Actions — never built on the VPS

If the GHCR packages are private, log in once as deploy with a classic PAT that has read:packages:

docker login ghcr.io -u <github-username>   # paste the PAT as the password

For a public repo, making the packages public and skipping registry auth on the VPS is simpler.

7. Configuration

Configuration lives in GitHub Secrets, not on the VPS: two secrets, PROD_ENV_FILE and STAGE_ENV_FILE, each holding the full dotenv content for its environment. Every pipeline run writes them to /srv/motorph/.env and .env.stage (mode 600) before deploying. Two consequences worth stating plainly:

• To change any value: edit the secret, then re-run Deploy. Hand-editing the files on the VPS works until the next pipeline run silently overwrites them.
• GitHub Secrets are write-only. Keep the finished payloads in a password manager. That copy — not GitHub, not the VPS — is your master, and §0 exists because of it.

Migrating from another host? Skip the generation below and put the old files back instead:

deploy/restore.sh motorph-<stamp>.tar.gz --env-only

Otherwise build them from .env.example, generating each secret rather than inventing one:

cp .env.example /tmp/prod.env
cp .env.example /tmp/stage.env

# each file gets its OWN secrets — staging never reuses production's
for f in /tmp/prod.env /tmp/stage.env; do
  sed -i "s|^POSTGRES_PASSWORD=.*|POSTGRES_PASSWORD=$(openssl rand -hex 32)|" "$f"
  sed -i "s|^APP_DB_PASSWORD=.*|APP_DB_PASSWORD=$(openssl rand -hex 32)|"     "$f"
  sed -i "s|^JWT_SECRET=.*|JWT_SECRET=$(openssl rand -hex 64)|"               "$f"
  sed -i "s|^ALTCHA_HMAC_KEY=.*|ALTCHA_HMAC_KEY=$(openssl rand -hex 32)|"     "$f"
done

Then fill in the rest by hand: DOMAIN and ACME_EMAIL, the MAIL_* relay block (§10's checklist covers picking a relay), the Grafana gate (§13), and billing in the production file. Staging needs neither a relay (its mail lands in Mailpit) nor real billing (sandbox). Save both payloads in your password manager, paste each into its secret, delete the /tmp copies.

Required variables. deploy/preflight.sh discovers them by grepping the compose files for the :? form, so this list stays honest automatically: DOMAIN, ACME_EMAIL, GRAFANA_BASIC_AUTH_USER, GRAFANA_BASIC_AUTH_HASH (edge); POSTGRES_PASSWORD, APP_DB_PASSWORD, JWT_SECRET, ALTCHA_HMAC_KEY, BILLING_PROVIDER (both stacks); MAIL_HOST, MAIL_USERNAME, MAIL_PASSWORD, MAIL_FROM (production only — staging defaults to Mailpit). Compose refuses to start with a descriptive error rather than booting a broken stack.

Single-quote any value containing $. Compose expands env-file values, so a JWT_SECRET of sec$RETvalue123 reaches the container as sec — and the app then signs and verifies tokens perfectly with a three-character key, with nothing anywhere reporting a problem. deploy.sh hard-fails on unquoted $ values before it deploys, which is the only reason this is a footnote and not an outage. Double quotes do not help.

DOMAIN must be a bare apex — no scheme, no path, no port, and not www.. The compose files derive stage.${DOMAIN}, https://${DOMAIN} and the CORS origins from it, so https://www.example.com/ produces site addresses Caddy refuses and a stack that crash-loops. deploy.sh checks this before every deploy; it took production down three times in one night before it did.

APP_DB_PASSWORD must be in the secret before the first run — not after. Migration V101 runs once and bakes this password into the app_runtime role the app connects as. Set it later and the role's password is whatever was there at first boot. Rotating afterwards is manual:

docker exec -it motorph_payroll_db psql -U motorph -d motorph \
  -c "ALTER ROLE app_runtime LOGIN PASSWORD 'the-new-password'"

Why it exists: the app must connect as a non-owner role for row-level security to bind. Connect as the owner and every tenant-isolation policy is a silent no-op — a new tenant's dashboard shows another tenant's payroll, as data, with no error anywhere. The rule applies per database; staging has its own V101 run.

PLATFORM_ADMIN_PASSWORD is a one-shot. Blank → no platform operator account (the backend logs a WARN and nobody can provision tenants). Set → the account is created once, forced to change its password at first login, and the variable is ignored forever after. There is no password-recovery flow anywhere in the product.

8. Wire GitHub

Generate a dedicated CI key — never reuse your personal one:

ssh-keygen -t ed25519 -f motorph-deploy-ci -C github-actions -N ""
ssh-copy-id -i motorph-deploy-ci.pub deploy@<VPS_IP>

Repo → Settings → Secrets and variables → Actions:

Secret	Value
VPS_HOST	the VPS IP — not the domain, which resolves to the CDN, and CDNs don't proxy SSH
VPS_USER	deploy
VPS_SSH_KEY	contents of the private key file motorph-deploy-ci
VPS_PORT	only if sshd isn't on 22 (every workflow defaults to 22)
PROD_ENV_FILE	full production dotenv payload (§7)
STAGE_ENV_FILE	full staging dotenv payload (§7)

Five are required; VPS_PORT is optional. Image pushes need no registry secret — the workflows authenticate to GHCR with the automatic GITHUB_TOKEN.

Check the scope you put them in. deploy.yml and docs.yml declare environment: Production, so secrets scoped to that environment resolve for them. promote.yml and rollback.yml declare no environment — if the secrets are environment-scoped, those two workflows cannot read them and fail with a generic "missing server host". If you want the Rollback button to work, either add the environment to those workflows or keep the secrets at repository scope.

One security consequence to accept knowingly: anyone who can push to main or edit workflows can deploy to production and can exfiltrate the secrets (a workflow edit can print them). That is inherent to config-in-CI; the mitigations are a tight collaborator list and branch protection on main.

9. How traffic reaches the app

deploy/Caddyfile holds six site blocks — the apex, www, stage, docs, grafana and acme — each deriving its hostname from {$DOMAIN}. Caddy substitutes those from the edge container's environment and obtains one certificate per block automatically. HSTS is emitted here and only here; the backends and the frontend nginx see plain HTTP and deliberately never set it. Caddy resolves upstreams per request, so a stack that is down 502s on its own hostname while the others keep working.

Prometheus and Alertmanager are deliberately not routed. Neither has any authentication; they are reachable over an SSH tunnel and that is the point.

Why /api bypasses the frontend nginx

This is the one piece of routing that looks redundant and absolutely isn't.

The backend keys rate limiting, account lockout and audit logging on the client IP, which it takes from the last entry of X-Forwarded-For — last, not first, because the first entry is whatever the client claims. "Last entry" is sound with exactly one trusted proxy hop. Chain a second appending hop — CDN → Caddy → frontend nginx → backend, each adding what it saw — and the last entry becomes a Docker-network container address. Every user on the site then shares one rate-limit bucket: ten failed logins per minute, total, across all tenants, locks everyone out. Nothing errors; login just starts flaking.

Hence two rules, both already in the committed files:

1. Caddy talks to the backend directly for /api/* and /ws. The frontend nginx still contains an /api proxy block for dev and the legacy shape — here it simply never receives traffic.
2. Caddy overwrites the forwarded-IP header with the CDN's authenticated statement of the real client address. That is trustworthy only because §4's firewall means nothing but the CDN can reach 80/443.

§10 includes a check that this survived contact with reality.

10. First launch

Order matters, and only the first step is unusual — everything after it is what every deploy does forever.

cd /srv/motorph
deploy/preflight.sh prod          # env file, DOMAIN shape, ports, egress, images

Preflight exits 0 when it is safe to deploy and prints every problem it found rather than stopping at the first. Then:

deploy/deploy.sh prod deploy <tag>     # creates the network, brings up the edge, deploys

<tag> is sha-<12 hex> from a successful build — take one from the Actions run or from GHCR. The script creates the shared network, brings up the edge, pulls images first (auth failures and typos die before the stack is touched), takes a pg_dump if there is a database to dump, swaps IMAGE_TAG, runs compose up, and polls the health endpoint. On a first deploy the health timeout is raised to 600 seconds automatically (first boot runs the full migration set) and there is no previous tag to auto-roll-back to, so a failure leaves the stack up and failing rather than reverting.

Then staging, then the docs site:

deploy/deploy.sh stage deploy <tag>
deploy/docs-deploy.sh <tag>            # requires the network to exist already

After that, the pipeline takes over: push to main and deploy.yml builds, deploys staging, gates on health, and promotes the same tag to production.

Verify before telling anyone the URL

Read the boot log first:

docker logs motorph_payroll_backend 2>&1 \
  | grep -iE "row-level security|platform administrator|Successfully applied|Schema .* is up to date"

Three lines matter:

1. Row-level security is in force (connected as 'app_runtime') — the single most important line. The WARN variant means the app connected as the owner and tenant isolation is silently off. It does not fail the boot.
2. Flyway applying the full migration set on first boot, Schema "public" is up to date afterwards.
3. Created platform administrator — or the WARN saying PLATFORM_ADMIN_PASSWORD is unset.

Then from outside:

for h in "" www. stage. docs.; do
  printf '%-28s %s\n' "$h<domain>" "$(curl -s -o /dev/null -w '%{http_code}' "https://$h<domain>/")"
done
# expect 200, 301, 200, 200

To separate an origin problem from a CDN problem, test the origin directly — this is the check that distinguishes "the edge has no site block" from "TLS is misconfigured":

curl -sv --resolve "docs.<domain>:443:127.0.0.1" https://docs.<domain>/ -o /dev/null 2>&1 \
  | grep -E 'subject:|issuer:|HTTP/|alert'

A TLS alert here with a working CDN elsewhere means Caddy has no certificate for that hostname, which almost always means it has no site block for it — see §14.

Confirm Swagger is off and the health endpoint is internal-only:

curl -s -o /dev/null -w '%{http_code}\n' https://<domain>/v3/api-docs   # expect 404
docker exec motorph_payroll_backend wget -qO- http://127.0.0.1:8080/actuator/health

The client-IP check — do not skip it. Fail a login on purpose, then look at the audit table:

docker exec motorph_payroll_db psql -U motorph -d motorph -c \
  "SELECT action, ip_address, log_date_time FROM user_log ORDER BY log_id DESC LIMIT 5"

ip_address must be your real public IP. A 172.x/10.x address means the forwarding chain is broken (§9) and every user shares one rate-limit bucket.

First-boot checklist

1. Rotate or disable every seeded account. The migrations seed demo accounts into every database, production included, and their passwords are in the repository. They all live in tenant 1, which cannot be suspended. Log in, create your real admin account, verify it, then rotate or deactivate all of them — on staging too. stage.<domain> is just as public, and a compromised staging account still sends mail from your IP on your domain.
2. Do the platform admin's first login. Every API call except the change-password flow answers 403 until the forced change is done. Store the new password in a password manager — no recovery flow exists.
3. Verify signup end to end with a real mailbox. A 503 on step one means mail is off or the relay is down; signup hard-requires working SMTP. Rate limits while testing: 6 signup requests/hour/IP, 10 verify attempts/min/IP, 8 codes/day per address. A backend restart clears all of it.
4. Set up the mail relay properly. Most VPS hosts block outbound port 25; the compose file assumes a submission relay on 587 with auth and STARTTLS. Configure SPF and DKIM for the MAIL_FROM domain, or verification codes land in spam and signup "doesn't work" in the least debuggable way.
5. Don't link the public careers pages yet. /careers and /api/public/job-openings are not tenant-scoped: on a shared deployment they list every tenant's postings in one page.
6. Register the billing webhook when billing goes live, and put the endpoint's signing secret — not the endpoint URL — in POLAR_WEBHOOK_SECRET. Never run a promoted deployment with BILLING_PROVIDER=stub; the stub exposes an unsigned webhook anyone can call.

11. Restore data onto the new box

The counterpart to §0. Ordering is enforced by the script, but understanding it matters: a dump restores a database, not a cluster. The app_runtime role is cluster-level and is not in the dump, so a dump loaded into a fresh volume with no such role fails every GRANT it contains — and the app then falls back to the owner role, silently disabling row-level security.

So the sequence is: configuration, then a normal deploy (which creates the containers, the volumes and the role), then the data.

cd /srv/motorph
deploy/restore.sh motorph-<stamp>.tar.gz --env-only   # 1. .env, .env.stage, deploy state
deploy/preflight.sh prod
deploy/deploy.sh prod deploy <tag>                    # 2. stack up, V101 creates app_runtime
deploy/restore.sh motorph-<stamp>.tar.gz              # 3. load both databases

Step 3 verifies the archive's checksums, creates or aligns the app_runtime role from the restored env file, stops each backend, drops and reloads its schema, and starts it again. It refuses to overwrite a database that already has tables unless you pass --force, so pointing it at a live box by accident does nothing.

Verify exactly as §10 does — the row-level-security line first, then the client-IP check. Then:

docker exec motorph_payroll_db psql -U motorph -d motorph \
  -c "SELECT count(*) FROM flyway_schema_history"

The dump contains flyway_schema_history, so the restored database knows which migrations it has and the next deploy continues cleanly.

Certificates re-issue themselves on the new host. Don't copy the old Caddy volume, and don't loop-recreate it either: Let's Encrypt rate-limits issuance per registered domain, and a rebuild that keeps wiping motorph_caddy_data can lock you out of new certificates for a week.

12. Backups from here on

deploy.sh takes a dump before every deploy and every rollback, into /srv/motorph/backups/. That covers "the migration broke it". It does not cover losing the host, because it does not capture the env files — that is what deploy/backup.sh is for (§0).

Schedule both:

0 3 * * * /srv/motorph/deploy/deploy.sh prod backup >> /home/deploy/backup.log 2>&1
0 4 * * 0 /srv/motorph/deploy/backup.sh >> /home/deploy/backup.log 2>&1

Staging deliberately has no backup cron — its data is recreated by the next merge.

Two properties the strategy relies on:

• There is no media directory. Resumes and filing receipts are database blobs, so pg_dump covers everything the app knows. No file/DB drift.
• The dotenv master copies live in your password manager, because secrets cannot be read back out of GitHub and the VPS files are disposable pipeline artifacts.

Get them off the machine, and prune — nothing in the repo prunes backups/, it grows unbounded, and preflight only warns once free space drops below 2 GB:

rsync -az deploy@<VPS_IP>:/srv/motorph/backups/ ./motorph-backups/
ssh deploy@<VPS_IP> 'find /srv/motorph/backups -name "*.sql.gz" -mtime +30 -delete'

Rehearse the §11 restore once on a scratch machine before you need it under pressure. A backup you have never restored is a hypothesis.

13. Operating it

Promote and roll back. Merging to main deploys staging and promotes to production automatically. The Promote to Production workflow re-ships an explicit tag; Rollback returns an environment to its previous tag (or one you name) and applies the same health gate. Rollback flips image tags only — Flyway has no down migrations, and a compose overlay tells the older jar to tolerate applied migrations it doesn't know about. If the migration did the damage, rollback is the wrong tool: restore the pre-deploy dump (§11).

Monitoring is an opt-in, production-only overlay:

WITH_MONITORING=1 deploy/deploy.sh prod deploy <tag>

Once on, leave it on — omitting the flag later does not remove the containers, it just stops managing them. Before the first run, put these in PROD_ENV_FILE:

GRAFANA_ADMIN_PASSWORD=<generate one>
MONITORING_BIND=127.0.0.1
PROMETHEUS_RETENTION_SIZE=8GB

MONITORING_BIND matters: Docker's port publishing bypasses UFW (§4), so anything other than loopback puts an unauthenticated Prometheus on the public internet. Reach the UIs over a tunnel:

ssh -L 3000:localhost:3000 -L 9090:localhost:9090 deploy@<VPS_IP>

Grafana is also published at grafana.<domain> behind a basic-auth gate in the Caddyfile, which is why the edge requires GRAFANA_BASIC_AUTH_USER and GRAFANA_BASIC_AUTH_HASH — both, or the edge refuses to start. Generate the hash and store only the hash:

docker run --rm caddy:2.10-alpine caddy hash-password --plaintext 'your-password'

Put the result in PROD_ENV_FILE in single quotes — bcrypt hashes contain $, and an unquoted value is truncated by compose into a hash that rejects every password. deploy.sh checks the full 60-character shape before deploying, because a prefix-only check is what let a broken hash reach production once.

Full reference: monitoring.md.

14. Troubleshooting

525 from the CDN on a hostname that should work. The origin could not complete TLS, which nearly always means Caddy has no certificate because it has no site block for that name. Check what the edge actually loaded:

docker exec motorph_caddy wget -qO- http://127.0.0.1:2019/config/ \
  | jq -r '[.. | objects | .host? // empty] | flatten | unique[]'

If the hostname is missing while deploy/Caddyfile on disk has it, the running container is reading a stale file: a single-file bind mount resolves to an inode when the container starts, and git pull replaces the file rather than rewriting it, so caddy reload --config /etc/caddy/Caddyfile re-applies the old config and reports success. Compare them:

docker exec motorph_caddy grep -c 'docs\.' /etc/caddy/Caddyfile   # container's view
grep -c 'docs\.' /srv/motorph/deploy/Caddyfile                    # what git has

The deploy scripts copy the file in before reloading for exactly this reason. To fix a running edge by hand, without downtime:

docker cp /srv/motorph/deploy/Caddyfile motorph_caddy:/tmp/Caddyfile
docker exec motorph_caddy caddy reload --config /tmp/Caddyfile --adapter caddyfile

"application not healthy after 60s" while the container serves fine. Check what the healthcheck actually probes. In Alpine images localhost resolves to ::1 first, and a server listening on IPv4 only refuses every probe while answering perfectly on 127.0.0.1:

docker exec <container> wget -q -O /dev/null http://localhost/  ; echo "localhost: $?"
docker exec <container> wget -q -O /dev/null http://127.0.0.1/  ; echo "127.0.0.1: $?"

A secret works locally but not in the container. Compose expanded it: any env-file value containing $ and not wrapped in single quotes is truncated at the $. deploy.sh refuses to deploy such a file; if you hand-edited /srv/motorph/.env, that is the check you bypassed.

Caddy logs tls.obtain errors mentioning 127.0.0.53:53. The container inherited the host's systemd-resolved stub, unreachable from a container namespace. The edge compose file sets dns: [1.1.1.1, 8.8.8.8] for this.

ACME challenge fails / certificate never issues. "Always Use HTTPS" is on before the first certificate (§5), or the CDN allowlist in DOCKER-USER is stale — providers add ranges; re-run the loop in §4.

Everything container-side loses the network at once — ACME hangs, image builds fail to reach Maven Central, SMTP times out — while the host is fine. A DOCKER-USER DROP without -i <ext-if>. See §4; the rescue is the RETURN guard.

failed to bind host port 0.0.0.0:80. Something on the host already listens there — often the distro's own nginx/caddy/apache package. sudo ss -tlnp | grep -E ':80|:443', then disable it.

Backend loops with Detected applied migration not resolved locally. An older jar met a newer schema without the rollback overlay: someone deployed an old tag by hand instead of through deploy.sh <env> rollback.

Backend (health: starting) for a few minutes on first boot is normal — the full migration set runs once. Worry only if it flips to unhealthy. The designed fail-fasts are a blank/short JWT_SECRET or ALTCHA_HMAC_KEY, and MAIL_ENABLED=true with a blank MAIL_HOST; each says so in the exception.

Boot log WARN Row-level security is NOT in force. The app connected as the owner role — APP_DB_USER/APP_DB_PASSWORD missing or not matching the role's actual password (§7). The app runs fine; that is the danger.

Audit log full of 172.x addresses. The forwarding chain broke — someone pointed /api back through the frontend nginx or removed the header_up line. Re-read §9; every user is sharing one rate-limit bucket.

Container name collisions. The dev compose file uses the same container names as production (motorph_payroll_*). Never run the dev stack on the VPS; preflight refuses to deploy when a container of the right name belongs to a different project directory.

GHCR pull fails denied/unauthorized. Packages are private and the VPS isn't logged in, or the PAT expired (§6).

Deploy workflow fails at the SSH step. Wrong secret values, sshd moved ports, or fail2ban banned the runner IPs after earlier failures (sudo fail2ban-client status sshd). VPS_HOST must be the IP, not the domain. If only Promote or Rollback fails this way, check the environment scoping in §8.

git pull --ff-only refuses on the VPS. Someone hand-edited committed files there. .env*, .deploy-state* and backups/ are gitignored and never conflict.

A config change "didn't take". You edited /srv/motorph/.env* instead of the secret, or edited the secret without re-running Deploy. The secret plus a pipeline run is the only path that sticks.

Frontend serves a stale UI after a deploy. Images are immutable by tag, so it is a cache — purge the CDN and hard-reload. If docker ps shows the old tag, the deploy itself failed.

15. Security checklist

• All six DNS records proxied, TLS mode Full (strict).
• Root SSH login and password auth disabled; UFW deny-incoming; fail2ban and unattended-upgrades running.
• DOCKER-USER restricts 80/443 to the CDN, the ! -i <ext-if> -j RETURN guard is rule 1, and the rules are persisted.
• Only Caddy publishes public ports; no app stack has a ports: entry.
• Both boot logs show Row-level security is in force.
• Platform admin created, first-login change done, password in a password manager.
• Every seeded demo account rotated or disabled — production and staging.
• https://<domain>/v3/api-docs returns 404; the ALTCHA widget appears on the login page.
• BILLING_PROVIDER is not stub; the webhook secret is the signing secret, not the URL.
• Failed-login smoke test shows a real public IP, not 172.x.
• CI deploys via a dedicated key; env payloads differ per environment; master copies in a password manager; VPS env files never hand-edited.
• Branch protection on main and a tight collaborator list.
• deploy/backup.sh scheduled, archives copied off the box, and the restore rehearsed at least once.