20 — The big picture: what happens when you push

Read this first: this is the first chapter of Track B, where the course stops building toys and starts reading the real system. It traces one commit from git push to a browser loading the live site, naming every moving part. It is a map, not a runbook — the operational detail lives in DevOps, and this page exists so that when you go there, you know where you are.

Time: about 35 minutes of reading. Assumes lessons 08, 09 and 15.

The one paragraph

Push to main. GitHub runs the tests. If they pass, it builds two images — backend and frontend — tags both with the commit sha, and pushes them to a registry. It then SSHes into a single Ubuntu server, writes the configuration files from GitHub Secrets, and runs a deploy script that pulls those images and restarts staging. It waits for the application to report healthy. Only if it does does the same script promote the identical images to production, wait for health again, and finish. If either health gate fails, the script rolls back to the previous images and the workflow goes red.

Everything else is detail. Read that paragraph again — it is the whole system.

The picture

Build once, deploy twice

The single most important design decision here: production runs the exact bytes that staging proved. Not a rebuild from the same commit — the same image, identified by the same tag.

The tag is computed in the build job:

      - id: tag
        run: echo "tag=sha-${GITHUB_SHA:0:12}" >> "$GITHUB_OUTPUT"

${GITHUB_SHA:0:12} is bash substring expansion: the first 12 characters of the commit sha. Writing to $GITHUB_OUTPUT is how a step publishes a value that later jobs can read, via needs.build-backend.outputs.tag.

A rebuild would defeat the point. Two builds of the same source can differ — a dependency resolves to a new patch version, a base image is updated. If production rebuilds, staging tested something else.

The four stacks on one server

The production host runs four independent compose projects sharing one Docker network:

Project	File	Contains
motorph-edge	deploy/docker-compose.edge.yml	Caddy — the only container publishing ports
motorph	deploy/docker-compose.prod.yml	Production database, backend, frontend
motorph-stage	deploy/docker-compose.stage.yml	Staging database, backend, frontend, Mailpit
motorph-docs	deploy/docker-compose.docs.yml	This documentation site

They are separate projects so they can be deployed independently — a documentation typo must not queue behind a payroll release, and the docs deploy has its own lock for exactly that reason.

They share one network so the edge can reach all of them by container name — the DNS behaviour from lesson 08, at production scale. And only the edge publishes ports. Everything else is reachable only from inside that network, which is why docker compose ps on that box shows port bindings on one container and nothing on the rest.

How a request reaches the app

Caddy owns 80 and 443 and routes by hostname. Six site blocks, all derived from one {$DOMAIN} variable: the app, stage., docs., grafana. (behind basic auth), a legacy client host, and a www. redirect.

Within the main site block, one matcher decides everything:

	@api path /api/* /ws /ws/*
	handle @api {
		reverse_proxy motorph_payroll_backend:8080 {
			header_up X-Forwarded-For {header.CF-Connecting-IP}
		}
	}

So /api/* goes straight to the backend, and everything else goes to the frontend's nginx, which serves the built React files.

The header_up line rewards a close look. It overwrites the X-Forwarded-For header rather than appending to it. That is normally the wrong thing to do — but the backend takes the client IP from the last entry, and that is only sound with exactly one trusted hop. Add another proxy in the chain and the last entry becomes a Docker-internal address, which would put every visitor on the site into a single rate-limit bucket. The overwrite is trustworthy only because ports 80 and 443 are firewalled to the CDN's address ranges, so nothing else can reach Caddy to forge it.

That is a good example of the shape of a real system: a line that looks wrong, is right, and is only right because of something enforced somewhere else entirely.

What the deploy script does on the box

deploy/deploy.sh is 439 lines, and its job is to make failure cheap. The order of operations is the safety:

1. Take a lock. flock on a file descriptor, so two deploys can never interleave — GitHub's concurrency setting is not enough, because a cancelled workflow can leave its SSH-spawned script still running on the server.
2. Validate the configuration, including the $-in-env-file check you met in lesson 09.
3. Pull the images first. A wrong tag or an authentication failure dies here, with the running stack untouched.
4. Back up the database, before changing anything.
5. Start the new containers with the tag pinned explicitly for that one command.
6. Wait for healthy, polling the container's health status against a deadline.
7. On success, record the new tag and rotate the previous one. On failure, roll back to the previous tag and exit non-zero regardless — the workflow must go red either way, because a deploy that had to roll back did not succeed.

Step 7 has a subtlety worth stealing: re-deploying the tag that is already live must not rotate the state, or a routine re-run would set PREVIOUS = CURRENT and destroy the only useful rollback target.

Where the configuration comes from

The server holds no secrets of its own. The production and staging env files are stored as GitHub Secrets and written to the box by the workflow on every run:

        env:
          PROD_ENV: ${{ secrets.PROD_ENV_FILE }}
        with:
          envs: PROD_ENV,STAGE_ENV
          script: |
            set -euo pipefail
            cd /srv/motorph
            git pull --ff-only origin main
            umask 077
            printf '%s\n' "$PROD_ENV"  > .env

Three details, each deliberate:

• The file content travels as an environment variable (envs:), never interpolated into the script text with ${{ }}. Interpolation would paste a multi-line secret containing arbitrary characters directly into a shell script — a quoting disaster and an injection risk at once.
• umask 077 before the redirect is why the file lands readable only by its owner. That is lesson 02's permission bits, doing real work.
• The env files on the server are therefore disposable. Editing one by hand is pointless; the next deploy overwrites it. Change the Secret instead.

The thing that surprises everyone

The staging deploy job declares environment: Production.

That looks like a bug and is not. In GitHub Actions, environment: is also a scope selector for secrets: secrets attached to an environment are only readable by a job that declares it. The staging job needs the same server credentials, so it declares the same environment. Getting this wrong once produced a deploy failure reporting a missing server host — and then, after a partial fix, every secret silently reading as an empty string.

The workflow carries a long comment explaining this, and it is the single most valuable paragraph in the repository for interview purposes. "Why does your staging job declare a Production environment?" is a question with a real answer.

What you can read now

You have the vocabulary for the whole system. A reasonable order:

Read	For
ci.md	What each workflow gates, and what is deliberately not gated
docker.md	Every compose file and Dockerfile, in reference form
vps-guide.md	Standing the whole thing up from a bare server
troubleshooting.md	The symptom index, for when it is 2am

Those pages assume the models this course just built. That was the point.

Recap

• Build once, deploy twice: production runs the identical image staging proved, identified by sha-<first 12 of the commit>.
• Four compose projects, one network, one edge — and only the edge publishes ports.
• The deploy script's ordering is its safety: lock, validate, pull, back up, start, health-gate, and roll back on failure while still going red.
• The server stores no secrets. They are written from GitHub Secrets on every deploy, which makes the files on disk disposable.

Chapters 21 to 26 go through the real workflows, deploy.sh, the compose stacks and the edge line by line. They are being written; until then, the reference docs above are the next stop.