13 — Firewalls, and the rule that killed the containers

Read this first: this lesson teaches the one firewall fact that costs people real outages — Docker's rules are evaluated before UFW's, so UFW does not protect a published container port — and then has you reproduce, in a box you can throw away, the exact mistake that took this project's production down on 30 July 2026.

Time: about 50 minutes. Runs on the lab box from lesson 10. ./reset.sh is your escape hatch and you will want it.

Two layers, and only one of them is obvious

UFW is the friendly front end: ufw allow 22, ufw deny 8080, done. It is what every tutorial teaches.

Underneath, both UFW and Docker write rules into the kernel's netfilter tables. Docker inserts its rules ahead of UFW's. So when Docker publishes a container port, that traffic is accepted before UFW's rules are ever consulted.

Break it on purpose: UFW does not do what you think

sudo ufw --force enable
sudo ufw default deny incoming
docker run -d --name web -p 9000:80 nginx:alpine
sudo ufw deny 9000/tcp

Predict: you have explicitly denied port 9000. Is the container reachable?

curl -s -o /dev/null -w "%{http_code}\n" localhost:9000

It answers. UFW denied it and the container is still reachable, because Docker's rule matched first.

This is not a Docker bug and it is not a UFW bug — they are two tools writing to the same kernel tables with different assumptions. But it means that on any host running Docker, "I set up UFW" is not the same sentence as "the ports are closed".

The correct hook is a chain Docker provides for exactly this purpose: DOCKER-USER. It is consulted before Docker's own accept rules, so it is the only place a rule can win.

The trap: DOCKER-USER sees traffic in both directions

Here is where it goes wrong, and it goes wrong in a way that looks fine.

DOCKER-USER is in the FORWARD path. It sees every packet Docker forwards — packets arriving from the internet heading into a container, and packets from a container heading out to the internet. A rule that does not say which direction it means will match both.

Set up the mistake. The goal is reasonable: block inbound HTTPS so only the CDN can reach us.

sudo iptables -I DOCKER-USER 1 -p tcp --dport 443 -j DROP
sudo iptables -S DOCKER-USER

-N DOCKER-USER
-A DOCKER-USER -p tcp -m tcp --dport 443 -j DROP
-A DOCKER-USER -j RETURN

Predict, before running anything:

1. Can the host itself still reach the internet?
2. Can a container reach the internet?
3. Does DNS from inside a container still work?

Now measure all three.

curl -sS -m 5 -o /dev/null https://example.com && echo "HOST: fine"

HOST: fine

sudo docker run --rm alpine:3 sh -c "wget -q -T8 -O /dev/null https://example.com && echo OK || echo CONTAINER-EGRESS-DEAD"

CONTAINER-EGRESS-DEAD

sudo docker run --rm alpine:3 sh -c "nslookup example.com >/dev/null 2>&1 && echo 'DNS: still resolves'"

DNS: still resolves

Read those three results together, because that combination is what makes this so expensive:

• The host is perfectly healthy. Every check you would instinctively run passes.
• DNS from inside a container still works — the red herring. Docker's embedded resolver forwards the query to the daemon, which resolves it on the host, where nothing is blocked. So name resolution succeeds while no container can open a connection to anything.
• Only outbound TCP from containers is dead.

What that looks like in production: certificate renewal hangs, image pulls hang, outbound email stops, and every dashboard says the server is fine. That is the 30 July outage.

Fix it

Qualify the rule with an interface, and put a RETURN for everything that did not arrive from outside, first:

EXT_IF=$(ip route show default | awk '{print $5; exit}')
echo "default interface: $EXT_IF"
sudo iptables -I DOCKER-USER 1 ! -i "$EXT_IF" -j RETURN
sudo iptables -S DOCKER-USER

default interface: eth0
-N DOCKER-USER
-A DOCKER-USER ! -i eth0 -j RETURN
-A DOCKER-USER -p tcp -m tcp --dport 443 -j DROP
-A DOCKER-USER -j RETURN

sudo docker run --rm alpine:3 sh -c "wget -q -T8 -O /dev/null https://example.com && echo EGRESS-RESTORED"

EGRESS-RESTORED

Read the fix carefully: ! -i eth0 -j RETURN means "if this packet did not arrive on the external interface, stop processing this chain and let it through." Container-to-internet traffic arrives on the Docker bridge, not eth0, so it returns immediately and never reaches the DROP. Genuine inbound traffic does arrive on eth0, skips the RETURN, and gets filtered as intended.

It has to be first. Inserted after the DROP it would never be reached.

This is exactly what deploy/bootstrap.sh does, and it carries the comment explaining why:

  if ! iptables -C DOCKER-USER ! -i "$EXT_IF" -j RETURN 2>/dev/null; then
    run iptables -I DOCKER-USER 1 ! -i "$EXT_IF" -j RETURN

Note the iptables -C ... || shape: -C checks whether a rule exists and the insert only happens if it does not. That makes the script idempotent — you can run it twenty times and get one rule, which matters because bootstrap scripts get re-run.

Rules do not survive a reboot

sudo netfilter-persistent save

iptables rules live in memory. Without saving them, a reboot silently removes your firewall and the box comes back wide open — with everything appearing to work perfectly, which is the worst possible symptom.

bootstrap.sh installs iptables-persistent for this, and it preseeds the debconf answers so that an unattended run does not hang forever on a dialog box nobody is watching.

Verify from the right place

The mistake in verification mirrors the mistake in the rule: checking from the wrong place. deploy/preflight.sh tests container egress from inside a container, over TCP:

docker run --rm --network ... curlimages/curl:latest -sS -m 15 -o /dev/null https://acme-v02.api.letsencrypt.org/directory

and its comment says explicitly not to use DNS as the test, for the reason you measured above. A check that passes when the thing it checks is broken is worse than no check.

Clean up

cd learn-devops/lab-vps
./reset.sh

If iptables state is still strange afterwards, ./reset.sh --cold.

On a real server there is no reset button. That asymmetry is the point of doing this here. Keep a second terminal already logged in whenever you edit a firewall on a machine that matters — if the new rule locks you out, the existing session usually survives long enough to undo it.

A note on addresses

Every firewall example in this course uses <VPS_IP>, $EXT_IF, or the documentation ranges reserved for exactly this purpose (192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24). That is not only good manners: the documentation site's CI fails the build if a published page shows an iptables or ufw command next to a routable address, because a rule naming a real address is somebody's real allowlist.

Recap

• Docker's rules are evaluated before UFW's, so UFW does not protect a published container port. DOCKER-USER is the only chain that wins.
• DOCKER-USER sees forwarded traffic in both directions. A rule without an interface match also kills container-to-internet egress.
• The symptom is brutal: host healthy, DNS working, all container egress dead.
• Fix with ! -i <external-interface> -j RETURN inserted first, and netfilter-persistent save so it survives a reboot.

Next: 14 — YAML without tears.