Skip to main content

Monitoring & Observability

Metrics, logs, traces, dashboards and alerting for the whole stack, in one docker compose command. Every component is free and open source — Prometheus and OpenTelemetry are CNCF projects, the rest is Grafana Labs' OSS tier. There is no paid tier anywhere in this document and nothing phones home.

docker compose -f docker-compose.yml -f docker-compose.monitoring.yml up -d
UIURLLogin
Grafanahttp://127.0.0.1:3000admin / ${GRAFANA_ADMIN_PASSWORD} (default admin)
Prometheushttp://127.0.0.1:9090none
Alertmanagerhttp://127.0.0.1:9093none

Everything binds to 127.0.0.1 by default. That is not paranoia: Docker publishes ports by writing its own iptables rules, which bypass UFW entirely — a 0.0.0.0 binding puts an unauthenticated Prometheus on the public internet no matter what the firewall says. Reach the UIs over SSH instead:

ssh -L 3000:localhost:3000 -L 9090:localhost:9090 deploy@your-vps

To stop just the monitoring half, use the same two -f flags with stop; a plain docker compose down (single -f) leaves the containers orphaned.

What is in the stack​

                    ┌──────────────────────────── Grafana ───────────────────────────┐
                    │  dashboards · Explore · alert view · trace↔log↔metric links    │
                    └───────┬──────────────────┬───────────────────┬─────────────────┘
                            │                  │                   │
                     Prometheus              Loki               Tempo
                      (metrics)             (logs)             (traces)
                            ▲                  ▲                   ▲
        ┌───────────────────┼──────────┐       │                   │
        │      │      │     │     │    │    promtail      OpenTelemetry Collector
     backend  node  cAdvisor pg  nginx blackbox  ▲                   ▲
    /actuator/  exporter    exporter exporter    │                   │
     prometheus                                  └── docker API ─────┤
                            │                                        │
                     Alertmanager ◄── alert rules            Spring Boot (OTLP)
                            │
                  Slack · Discord · email
ContainerImageJob
motorph_prometheusprom/prometheus:v2.53.0scrapes metrics, evaluates 44 alert + 12 recording rules
motorph_alertmanagerprom/alertmanager:v0.27.0groups, deduplicates and routes alerts
motorph_grafanagrafana/grafana:11.1.09 provisioned dashboards, 4 provisioned datasources
motorph_lokigrafana/loki:3.1.1log storage
motorph_promtailgrafana/promtail:3.1.1ships every container's stdout to Loki
motorph_tempografana/tempo:2.5.0trace storage + span-derived RED metrics
motorph_otel_collectorotel/opentelemetry-collector-contrib:0.104.0the single OTLP endpoint the app talks to
motorph_node_exporterprom/node-exporter:v1.8.2host CPU / RAM / disk / network
motorph_cadvisorgcr.io/cadvisor/cadvisor:v0.49.1per-container CPU / RAM / restarts
motorph_postgres_exporterprometheuscommunity/postgres-exporter:v0.15.0connections, locks, cache ratio, table sizes
motorph_blackbox_exporterprom/blackbox-exporter:v0.25.0synthetic uptime + TLS expiry
motorph_nginx_exporternginx/nginx-prometheus-exporter:1.3.0nginx connections and requests

Dashboards​

Provisioned from infra/monitoring/grafana/dashboards/, one folder per subdirectory, re-read every 30 s. Edit a JSON on disk and Grafana picks it up — there is no import step, and no dashboard that exists only in one person's browser. UI edits are refused (allowUiUpdates: false) because the next reload would silently discard them; use Save As for scratch work.

FolderDashboardAnswers
InfrastructureServer HealthIs the box healthy? CPU by mode, memory, disk, load, network
InfrastructureDocker ContainersWhich container is eating the box? Per-container CPU/RAM/net, restart counts
ApplicationSpring Boot APITraffic, error rate, p50/p95/p99, slowest and busiest endpoints, 24 h availability
ApplicationJVM & Connection PoolHeap, GC, threads, Tomcat, HikariCP
DatabasePostgreSQLConnections, TPS, cache hit ratio, locks, deadlocks, largest tables, dead rows
BusinessPayroll KPIsPayroll runs, payslips generated, failed runs, generation time, leave/overtime/attendance
SecurityAuth SecurityLogin outcomes, lockouts, rate-limit rejections, ALTCHA, signup codes, mail
LogsLog ExplorerEvery container's logs, filterable by level and free text
UptimeUptime & SLOProbe success, availability, error budget, certificate expiry

The three signals, and how they connect​

The point of running all three is not having three tools. It is that an error on a dashboard is two clicks from the log line that produced it and the trace that shows which call was slow. That only works if one identifier survives the whole journey, and it does:

Micrometer Tracing writes traceId/spanId into the MDC
  → logback-spring.xml puts them in the JSON log line
    → promtail lifts them into Loki structured metadata
      → the Loki datasource's derivedFields links them to Tempo
        → Tempo's tracesToLogsV2 links back the other way

Metrics — Prometheus​

The backend exposes /actuator/prometheus; Prometheus scrapes it every 15 s. Two things were added to make it useful beyond counters:

  • Histogram buckets for http.server.requests and motorph.payroll.payslip.generation, which is what makes p95/p99 possible at all. Deliberately not enabled globally — every histogram is roughly 15 extra series per tag combination, and http.server.requests already carries uri × method × status.
  • A common env tag on every metric, from METRICS_ENV. Dashboards and alerts group by it. It comes from the application rather than from a Prometheus target label because Tempo's span metrics arrive by remote write and never pass through a scrape config where a label could be attached.

Scrape targets live in infra/monitoring/prometheus/targets/*.yml, not in prometheus.yml. Prometheus re-reads those files on a timer, so adding an environment is an edit to a data file — no restart, no container recreate.

Scrape the backend by its hyphenated alias, never its container name. Tomcat validates the Host header against RFC 9110, underscores are not legal in a hostname, and motorph_payroll_backend:8080 therefore returns a bare Tomcat 400 before Spring sees the request. The compose files give the backend the network alias motorph-payroll-backend for exactly this. The application never noticed the problem because nginx and Caddy both forward the browser's Host rather than the upstream's.

Logs — Loki + Promtail​

Promtail discovers containers through the Docker API, so a new service is collected by existing — no per-container configuration. Backend logs are shipped as JSON (LOG_FORMAT=json, the default in every containerised environment) and parsed into:

  • labels — container, service, stack, stream, level, and status_class for nginx. All bounded sets.
  • structured metadata — trace_id, span_id, logger, thread, and nginx's method/path/status. Stored but not indexed.

That split is the whole discipline of running Loki. Labels are an index; a label with unbounded values (a trace id, a user, a tenant) creates a stream per value and is the single most common way to make Loki unusable.

Set LOG_FORMAT=console when you would rather read docker logs directly — ./dev.sh host mode already does, since those logs are read by a person.

Traces — Tempo + OpenTelemetry Collector​

Off by default. Turn it on with one line in .env:

TRACING_ENABLED=true

The application sends spans to the collector and knows nothing else about where traces go, which is the point of the collector being there: sampling harder, dropping a noisy span, or replacing Tempo becomes a change to infra/monitoring/otel/config.yml rather than a redeploy.

The collector drops /actuator/* spans before storage — Prometheus alone hits that path four times a minute forever, and keeping it would fill Tempo's retention window with the monitoring watching itself.

Tempo's metrics_generator derives RED metrics and a service graph from spans and remote-writes them to Prometheus as traces_spanmetrics_* and traces_service_graph_*. This is why Prometheus runs with --web.enable-remote-write-receiver.

Two traps worth knowing, both of which fail silently:

  • The Boot 3 property management.otlp.tracing.endpoint still appears in Boot 4's configuration metadata but no longer binds. Boot 4 wants management.opentelemetry.tracing.export.otlp.endpoint. Set the old one and the exporter quietly falls back to localhost:4318, which inside a container is the container itself.
  • micrometer-tracing-bridge-otel alone is not enough on Boot 4. Boot 4 split autoconfiguration into per-technology modules, so the bridge puts the libraries on the classpath and configures nothing. The dependency that actually wires it up is spring-boot-starter-opentelemetry (with micrometer-registry-otlp excluded, since metrics here are pulled, not pushed).

Alerting​

44 rules in infra/monitoring/prometheus/rules/alerts.yml, across availability, host, containers, application, database, security, probes, business, and the monitoring stack watching itself. Two severities, and the distinction is the whole point:

  • critical — someone is paged now; the product is down or about to be.
  • warning — goes to the chat channel; look at it during working hours.

Every for: duration is deliberate. An alert with no delay fires on a single bad scrape, and an alert that fires on noise gets muted — which is worse than not having it.

Routing lives in alertmanager.yml; the receivers are generated at container start by entrypoint.sh from environment variables, because Alertmanager does not interpolate env vars into its config and a Slack webhook URL is a bearer token that has no business in git.

ALERT_SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...
ALERT_DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/...   # plain URL; /slack is appended for you
ALERT_EMAIL_TO=[email protected]

With none of them set, alerts still fire, still group, and are still visible in the Alertmanager UI and in Grafana → Alerting. Nothing breaks; you just have to go and look.

Staging is routed to chat only and never pages. inhibit_rules suppress the downstream noise — a BackendDown mutes every other alert for that environment, because a dead backend explains all of them.

Business alerts​

The ones a non-engineer would recognise, and the reason business metrics exist:

AlertFires when
PayrollGenerationFailedany payslip generation run threw, in the last 15 min
PayrollGenerationSlowp95 generation time above 2 minutes
MailDeliveryFailingmore than 20 % of outbound email failing

Application metrics​

Three registries, all under backend/src/main/java/com/motorph/payroll/metrics/.

AuthMetrics — login outcomes, lockouts, rate-limit rejections, ALTCHA verifications, signup codes, password rehashes, portal auth. These are the only durable record of anonymous rejections: rate-limit and ALTCHA refusals happen before any user is resolved, and user_log.user_id is a NOT NULL FK to users, so they cannot be audited there.

BusinessMetrics — what the product did, instrumented at the service layer rather than the controller so the numbers reflect work that was actually committed:

Prometheus metricTags
motorph_payroll_runs_totalaction: created | payslips_generated | approved | rejected | processed | cancelled
motorph_payroll_payslip_generation_seconds_*outcome: success | failure — timer with buckets
motorph_payroll_payslips_generated_total—
motorph_leave_requests_totalaction: created | approved | rejected | cancelled
motorph_overtime_requests_totalaction: created | cancelled
motorph_timesheet_events_totalaction: clock_in | clock_out | submitted | status_changed

BuildInfoMetrics — motorph_build_info, a constant-1 gauge whose labels carry the deployed version and commit. It answers the question that comes up in every incident and that no other metric can: which build is actually running? Overlay it on a latency graph and "it got slow at 14:20" becomes "it got slow when we shipped abc123".

Cardinality doctrine, which applies to anything added later: every tag value comes from a small fixed set defined in the source. Never tag with an employee number, a payroll id, or a tenant slug. One series per employee is how a Prometheus dies, and per-tenant series would put customer shape into a store with no meaningful access control.

All counters are in-memory and reset when the backend restarts — dashboards and alerts read them through rate() / increase(), never as running totals.

Running it in production​

Set one line in the PROD_ENV_FILE secret:

MONITORING_ENABLED=true

Every production deploy from then on brings the stack up and keeps it managed — including the automated ones. That is deliberately a value in the env file rather than a flag on the command line: the pipeline invokes deploy.sh itself and passes no shell variables, so an invocation-only switch would mean the stack came up once, by hand, and silently stopped being managed on the next deploy from CI.

For a one-off trial without committing to it, the override still exists:

WITH_MONITORING=1 deploy/deploy.sh prod deploy <tag>

Turning it back off stops the overlay being managed but does not remove the containers — compose leaves unreferenced services running rather than guessing. Stop them explicitly if that is what you want.

Production only — deploy.sh refuses WITH_MONITORING=1 for staging. The overlay's containers have fixed names (motorph_prometheus, motorph_grafana, …), so a staging stack bringing them up would not get its own copy: compose would adopt production's containers and repoint them at staging's config and volumes. Production's monitoring quietly becoming staging's is a worse outcome than having none, so the refusal is a hard failure rather than a warning.

Staging is still observed. Production's Prometheus reaches it across the shared network — uncomment the motorph-stage-backend entry in infra/monitoring/prometheus/targets/apps.yml. One stack, both environments, and staging runs nothing extra.

Staging's backend therefore keeps METRICS_ENV=stage (so its series are distinguishable once scraped) but logs plain text rather than JSON, and has no tracing variables at all — there is no Loki or collector on that side to receive either.

Configuration goes in the GitHub Secrets, not on the box​

deploy.sh rewrites /srv/motorph/.env and /srv/motorph/.env.stage from the PROD_ENV_FILE / STAGE_ENV_FILE GitHub Secrets on every run. A hand-edit on the VPS survives exactly until the next deploy. Add these to the secret:

GRAFANA_ADMIN_PASSWORD=<a real password>   # set BEFORE first start
MONITORING_BIND=127.0.0.1
PROMETHEUS_RETENTION=30d
PROMETHEUS_RETENTION_SIZE=8GB
LOKI_RETENTION=720h
TEMPO_RETENTION=168h
TRACING_ENABLED=false
TRACING_SAMPLE_RATE=0.1
ALERT_SLACK_WEBHOOK_URL=

GRAFANA_ADMIN_PASSWORD applies only when Grafana initialises its database for the first time. On every later start it is ignored — the password lives in grafana.db in the grafana_data volume from then on. Changing the variable and redeploying does not rotate it. To actually reset it, see the credential-reset runbook in docs/deployment/monitoring.md (operator-only, not published).

Disk​

This stack writes continuously, and the retention settings above are the only thing standing between it and a full disk. Check df -h before raising them. Rough shape on a small deployment: Prometheus is the largest and the most useful historically; Loki is next; Tempo is the most disposable — a two-week-old trace answers no question the metrics and logs cannot.

Public probes​

infra/monitoring/prometheus/targets/probes-http.yml and probes-tls.yml ship with the public entries commented out, since local dev has no public hostname. Uncomment them on the VPS with the real domain — the TLS probe is what powers CertificateExpiringSoon, and Caddy renews at two thirds of a certificate's life, so that alert firing means renewal is broken (usually port 80 no longer reaching Caddy for the ACME challenge).

Troubleshooting​

A dashboard is empty. Check Prometheus → Status → Targets first, or the "Scrape targets" panel on Uptime & SLO. A missing exporter makes its dashboard lie by omission rather than break.

The backend target is down with a 400. It is being scraped by its underscored container name. Use motorph-payroll-backend:8080 — see the box under Metrics above.

Logs arrive as unparsed text. LOG_FORMAT is not json for that container.

Tempo is empty. TRACING_ENABLED is false, or the OTLP endpoint property is the Boot 3 spelling — see Traces above. Confirm spans are moving with otelcol_receiver_accepted_spans and otelcol_exporter_send_failed_spans in Prometheus.

Alerts fire but nobody is told. AlertmanagerNotificationsFailing covers the case where a webhook is configured and broken. If no ALERT_* variable is set at all, that is by design — check the Alertmanager UI.

Promtail logs "timestamp too old" on first start. It is replaying the history of long-running containers and Loki rejects entries older than reject_old_samples_max_age (7 days). It stops once promtail catches up.

A deploy fails with "contain '$' and are NOT single-quoted". That is the generic guard, and it applies to every secret, not just Grafana's. Docker compose expands env-file values, so a JWT_SECRET of sec$RETvalue123 reaches the container as sec — the app then signs and verifies tokens perfectly with a three-character key, and nothing anywhere reports a problem. Wrap the value in single quotes in the PROD_ENV_FILE / STAGE_ENV_FILE secret; double quotes do not help. If a credential has already been deployed truncated, rotate it.

Reloading config without a restart. Prometheus runs with --web.enable-lifecycle, so curl -X POST http://127.0.0.1:9090/-/reload applies rule and scrape-config edits in place. Target files and Grafana dashboards need nothing at all — both are re-read on a timer.