Skip to main content

Troubleshooting — MotorPH Enterprise Payroll System

Self-service reference for Docker Compose and runtime issues. Written so a new dev can get unstuck without asking anyone: start with the triage flow, then jump to your symptom in the index.

Start here: triage in order​

When anything is wrong with the stack, run these three commands before guessing — between them they diagnose nearly every issue on this page:

# 1. What state is everything in? Backend must show "(healthy)".
docker compose ps

# 2. If a container is restarting/unhealthy/exited — read its logs. The
#    backend logs the exact reason it refuses to boot (JWT, Flyway, ...).
docker compose logs motorph_payroll_backend --tail=50

# 3. What config is the stack ACTUALLY running with? (.env + any local
#    override file merged) — catches stale env values and override surprises.
docker compose config | less

Then find your symptom below.

Symptom index​

SymptomJump to
permission denied ... docker.sock or Cannot connect to the Docker daemonDocker daemon unreachable
bind: address already in use on 5173/8081/5434Port already in use
network <hash> not foundStale network
Backend container keeps restartingBackend keeps restarting
JWT_SECRET is still set to the insecure placeholderJWT error
Detected applied migration not resolved locallyFlyway history mismatch
Found more than one migration with version NDuplicate migration version
Blank page / API errors right after startupFrontend before backend
Healthcheck never turns healthyHealthcheck failing
no space left on device during buildDisk full
Edited frontend code, browser shows old versionStatic build
CORS errors in the browser consoleCORS
429 / account locked while testing loginsRate limit & lockout
Can't log in at all — which accounts exist?reference/demo-users.md
App "sent" an email but nothing arrivedEmails go to Mailpit
E2E fails with HTTP 428 at loginALTCHA
Playwright: Executable doesn't existPlaywright browsers
E2E failures that make no senseConcurrent runs
Flyway ignores/skips migrations mysteriouslyLocal override file
Monitoring containers survive docker compose downMonitoring overlay
Grafana login rejected / dashboards missingMonitoring overlay
Nav items missing for a role after an RBAC migrationRBAC sync

Getting the stack up​

Docker daemon unreachable or permission denied​

permission denied while trying to connect to the Docker daemon socket
Cannot connect to the Docker daemon at unix:///var/run/docker.sock

Cause: either the Docker daemon isn't running, or your user isn't allowed to talk to it.

Fix:

# Daemon not running (Linux):
sudo systemctl start docker

# Permission denied — add yourself to the docker group, then log out/in
# (or run `newgrp docker` for the current shell):
sudo usermod -aG docker $USER

On Docker Desktop (macOS/Windows), just make sure the Desktop app is running.

Port already in use​

Error ... bind: address already in use

Cause: something on your machine already listens on 5173, 8081, 5434, 8025, or (with the monitoring overlay) 3000/9090/9093.

Fix: find the squatter, or move ours — all host ports come from .env:

# who owns the port?
ss -ltnp | grep 5173

# or move the stack's ports in .env:
FRONTEND_PORT=5174
BACKEND_PORT=8082
POSTGRES_PORT=5435

If you change FRONTEND_PORT, add the new origin to CORS_ALLOWED_ORIGINS in .env too, then docker compose up -d.

network <hash> not found​

Error response from daemon: failed to set up container networking:
network 74f42806d... not found

Cause: Docker stored a network by its auto-generated hash. After a daemon restart or unclean shutdown the hash is gone but containers still reference it.

Fix:

docker compose down --remove-orphans
docker compose up -d --build

If the error persists, prune dangling networks:

docker network prune
docker compose up -d --build

Backend keeps restarting (restarting state)​

Check the logs first:

docker compose logs motorph_payroll_backend --tail=50

Then consult the specific error below.

Backend crash-loops right after up with a JWT error​

IllegalStateException: JWT_SECRET is still set to the insecure placeholder default

Cause: .env still has JWT_SECRET=change-me-in-production. The backend refuses to start with the placeholder, a blank value, or anything under 32 characters (JwtProperties.validateSecret).

Fix:

sed -i "s/^JWT_SECRET=.*/JWT_SECRET=$(openssl rand -hex 64)/" .env
docker compose up -d

Flyway: Detected applied migration not resolved locally: 20​

Validate failed: Migrations have failed validation
Detected applied migration not resolved locally: 20.
If you removed this migration intentionally, run repair to mark the migration as deleted.

Cause: The database has a record of migration V20 in flyway_schema_history, but there is no V20 file in the backend's migration folder (e.g., it was renamed to V25). Flyway refuses to start until the history matches the local files.

Fix A — Remove the stale history row (preserves data):

docker compose exec motorph_payroll_db psql -U motorph -d motorph -c \
  "DELETE FROM flyway_schema_history WHERE version = '20';"

docker compose restart motorph_payroll_backend

Fix B — Full clean start (wipes all data):

docker compose down --volumes
docker compose up -d --build

Use Fix B when you don't need the existing data (fresh dev environment).

Flyway: Found more than one migration with version 20​

Found more than one migration with version 20
Offenders:
→ /demo-seed/V20__rich_seed_employees.sql
→ .../db/migration/V20__message_cursor_index.sql

Cause: Running the demo compose (docker-compose.demo.yml) mounts extra seed files that conflict with a migration of the same version in the backend.

Fix: Check the version numbers of all demo seed files (demo/seed/V*.sql) and ensure no application migration shares the same version number. Rename the conflicting application migration to the next available version.

Frontend starts before backend is ready​

Symptom: Blank page or API errors on first load right after docker compose up.

Cause: The old compose didn't wait for Spring Boot to finish starting before nginx launched.

Status: Fixed in docker-compose.yml. The frontend now uses depends_on: condition: service_healthy and waits for the backend healthcheck to pass (~90 seconds on first boot due to Flyway migrations).

If you still see this, check backend health:

docker compose ps
# motorph_payroll_backend should show "(healthy)" before frontend starts

Backend healthcheck always failing​

docker inspect motorph_payroll_backend | grep -A5 Health

If the health status is unhealthy, check if the container can reach its own port:

docker compose exec motorph_payroll_backend \
  wget -qS -O /dev/null http://localhost:8080/ 2>&1 | head -3

Expected output: a line beginning with HTTP/1.1 (any status code is fine — even 401 means the server is up).

no space left on device during docker build​

Cause: repeated --build runs accumulate old images, build cache, and dangling layers; Docker's disk fills up long before yours does.

Fix:

docker system df          # see what's eating the space
docker system prune       # removes stopped containers, dangling images, build cache
docker builder prune      # bigger hammer for build cache alone

docker system prune --volumes also deletes unused volumes — your running stack's data volume is in use and survives, but stopped stacks' data does not. Run it only if you don't need any stopped stack's data.

Runtime issues​

Frontend change doesn't show up in the browser​

Cause: The frontend image is a static build — nginx serves the dist/ bundle produced at docker build time. There is no hot reload in the Docker stack, and VITE_API_URL is baked in at build time too.

Fix: rebuild the image (docker compose up -d --build motorph_payroll_frontend), or use the Vite dev-server loop for iteration (onboarding.md §5).

CORS errors in the browser console (host-dev only)​

Cause: You're running the Vite dev server (or changed FRONTEND_PORT), so the browser origin is no longer in the backend's CORS_ALLOWED_ORIGINS. The dockerized nginx setup never hits CORS because everything is same-origin.

Fix: add your dev origin to CORS_ALLOWED_ORIGINS in .env and docker compose up -d motorph_payroll_backend.

429 Too Many Requests / account locked (423) while testing logins​

Cause: The login rate limiter (Bucket4j, 10 requests/min per IP) or the exponential account lockout (5 failures → 15 min, doubling). Both are deliberate (security/authentication.md).

Fix: wait it out, or — since both are in-memory — restart the backend: docker compose restart motorph_payroll_backend. The e2e helpers already cache logins per worker to stay under the limit.

Emails never arrive​

Symptom: the app says it sent a signup code / welcome mail / payslip notification, but no inbox ever receives it.

Cause: that's by design in dev — the stack sends all mail to the local Mailpit catcher, never to the internet.

Fix: open http://localhost:8025 — every email is there. If Mailpit shows nothing either, check that MAIL_ENABLED=true in .env and that the backend log doesn't show a send error. Details: backend/email.md.

Flyway behaves differently than the migration folder suggests​

Cause: A local, gitignored docker-compose.override.yml can inject Flyway settings (e.g. SPRING_FLYWAY_IGNORE_MIGRATION_PATTERNS) without anything in the repo showing it — Docker Compose merges it automatically.

Fix: run docker compose config | grep -i flyway to see what the backend actually receives; delete or adjust the override file.

Test-suite issues​

E2E suite fails at login with HTTP 428​

Cause: ALTCHA is enabled; the Playwright helpers post credentials directly to /api/auth/login without a proof-of-work payload.

Fix: keep ALTCHA_ENABLED=false (the .env.example default) on any stack you run the e2e suite against. See e2e/README.md.

Playwright: Executable doesn't exist​

browserType.launch: Executable doesn't exist at ~/.cache/ms-playwright/...

Cause: the Playwright npm package is installed but the browser binaries aren't — they're a separate download.

Fix (from the repo root):

npx playwright install --with-deps chromium

E2E failures that make no sense​

Symptom: tests that pass in isolation fail in a full run — wrong row counts, entities that "shouldn't exist", flaky assertions.

Likely causes, in order:

  1. Someone else's run is hitting the same stack. The suite runs against a shared live database; two concurrent runs race each other. Check ps aux | grep playwright before starting a full run.
  2. Seed data has drifted. The specs assume the seed baseline documented in e2e/README.md; a stack where you've been creating entities by hand will fail assertions. Reset with docker compose down --volumes && docker compose up -d --build.
  3. Your code actually broke something. Only conclude this after ruling out 1 and 2.

Monitoring overlay issues​

The observability stack (deployment/monitoring.md) is an overlay — most confusion comes from forgetting that it's a second compose file layered on the first.

  • Monitoring containers survive docker compose down: any command that should affect the overlay needs both -f flags —

    docker compose -f docker-compose.yml -f docker-compose.monitoring.yml down

    A single-file down considers the monitoring containers orphans and leaves them running (add --remove-orphans to sweep them up).

  • Grafana login rejected: the default is admin/admin, overridden by GRAFANA_ADMIN_USER/GRAFANA_ADMIN_PASSWORD in .env. The admin password is set on first boot of the volume — changing .env later doesn't change an existing Grafana's password (reset by removing the grafana_data volume, or change it inside Grafana).

  • Grafana/Prometheus unreachable from another machine: intentional. All monitoring UIs bind to 127.0.0.1 because Docker's port publishing bypasses UFW. Use an SSH tunnel (ssh -L 3000:localhost:3000 user@host) rather than changing MONITORING_BIND.

  • Dashboard edits disappear: dashboards are provisioned read-only from infra/monitoring/grafana/dashboards/ and re-read every 30 s; UI edits are refused because the next reload would discard them. Edit the JSON on disk, or use Save As in Grafana for scratch work.

Useful Commands​

# Bring everything up (normal workflow)
docker compose up -d --build

# View logs for all services
docker compose logs -f

# View logs for one service
docker compose logs -f motorph_payroll_backend

# Check container health and status
docker compose ps

# Restart only the backend (e.g., after config change)
docker compose restart motorph_payroll_backend

# Stop all containers (keeps volumes/data)
docker compose down

# Stop and remove orphan containers from old compose runs
docker compose down --remove-orphans

# Full reset — removes containers AND all data volumes
docker compose down --volumes

# See the effective merged config (env + overrides) the stack runs with
docker compose config

# Connect to the database directly
docker compose exec motorph_payroll_db psql -U motorph -d motorph

# View Flyway migration history in the DB
docker compose exec motorph_payroll_db psql -U motorph -d motorph \
  -c "SELECT version, description, success, installed_on FROM flyway_schema_history ORDER BY installed_rank;"

Symptom (original case, 2026-06-27): Logged in as Warehouse Manager, the Inventory sidebar does not show Transfers, Serial Numbers, or Lot Numbers, even though the backend migrations (V21__serial_lot_rbac.sql, V24__warehouse_transfer_rbac.sql) correctly insert the permissions and assign them to the role.

Root cause: The frontend resolves permissions entirely from the static ROLE_PERMISSIONS map in frontend/src/constants/role-permissions.ts — it does not fetch them from the API or JWT at runtime. When INVENTORY_SERIALS_VIEW, INVENTORY_SERIALS_MANAGE, INVENTORY_TRANSFERS_VIEW, and INVENTORY_TRANSFERS_MANAGE were introduced in later migrations, they were never added to the 'Warehouse Manager' (and 'System Administrator') entries in that file. As a result, hasPermission() returned false and the nav items were filtered out.

Fix: Added the four missing constants to both roles in frontend/src/constants/role-permissions.ts:

// added after INVENTORY_RECEIVING_MANAGE for both 'Warehouse Manager' and 'System Administrator'
PermissionConstants.INVENTORY_SERIALS_VIEW,
PermissionConstants.INVENTORY_SERIALS_MANAGE,
PermissionConstants.INVENTORY_TRANSFERS_VIEW,
PermissionConstants.INVENTORY_TRANSFERS_MANAGE,

Prevention: Whenever a new Flyway RBAC migration adds permissions to a role, the matching constants must also be added to frontend/src/constants/role-permissions.ts. The two sources must stay in sync manually — there is no runtime sync between them.

Where the architecture details went​

The request-routing and startup-order diagrams that used to live here moved to deployment/docker.md, next to the rest of the compose documentation. Short version: the browser only ever talks to nginx on :5173, which proxies /api/ and /ws to the backend over the internal Docker network; startup order is db → backend (healthy after ~90s on first boot) → frontend, gated by healthchecks.