Scaling and deployment limits

What this system is built for today, what it is not built for, and exactly what would have to change to grow. Everything below is derived from the code — each limit names the file that causes it, so you can check the claim rather than trust it.

These are architectural limits, not measured ones. No load testing has been done on this codebase. The user counts below are the honest bounds of the shape of the deployment — where a design stops working — not throughput figures from a benchmark. Treat them as "this is the tier where you will hit a wall", not as a capacity guarantee. If you need real numbers, run a load test against your own data volume and hardware.

The shape today

One VPS runs everything: a Caddy edge terminating TLS, one backend container, one frontend container, and PostgreSQL — plus, optionally, a second full application stack for staging and an observability stack. See README.md for the deployment shapes and docker.md for what each container is.

The important word is one. The application is designed and deployed as a single backend instance, and several parts of it assume that. That assumption is fine — it is the right trade for the size this runs at — but it is load-bearing, and it is the thing that decides when you can and cannot add a second server.

What each tier requires

Tier	Users	Shape	Ready today?
1	~1–50	Single VPS, one instance of each service	Yes. This is what the repo deploys.
2	~50–5,000	Bigger VPS: more CPU/RAM, tuned Postgres, still one backend instance	Mostly. Vertical scaling needs no code change. Tune the connection pool and Postgres first (below).
3	~5,000–50,000	Several backend instances behind a load balancer	No. Five things break; see the next section.
4	50,000–1,000,000+ daily	Distributed: horizontal scaling, split services, shared cache, replicated database	No. Tier 3's work plus a genuine architecture programme.

Tier 1 → Tier 2 is buy a bigger machine. Tier 2 → Tier 3 is the real boundary, because it is where "one backend instance" stops being true.

What breaks with a second backend instance

Each of these is a correctness problem, not a performance one. They do not surface as errors — they surface as duplicated work, weakened security, or features that quietly stop working for half your users.

#	What	Why it breaks	Consequence
1	Login rate limiting	Token buckets live in a ConcurrentHashMap in process memory (LoginRateLimitFilter.java)	Each instance enforces the limit independently, so N instances allow roughly N× the intended attempts. A brute-force guard that silently weakens as you scale.
2	WebSocket delivery	registry.enableSimpleBroker("/topic", "/queue") (WebSocketConfig.java) is an in-memory broker	A message published on instance A never reaches a client connected to instance B. Notifications and live updates work for some users and not others, with no error anywhere.
3	Scheduled jobs	Four @Scheduled jobs and no distributed lock (no ShedLock, no leader election, no advisory lock)	Every instance runs every job. TrialExpiryScheduler and HolidayGenerationScheduler mutate business state on a cron — duplicated transitions and duplicate rows unless every job is provably idempotent.
4	CAPTCHA replay protection	Accepted ALTCHA signatures are remembered in a per-process map (AltchaService.java)	A solution already spent on instance A is unknown to instance B, so it can be replayed there.
5	Database connections	Hikari has no explicit maximum-pool-size (application.yml), so each instance takes the default of 10	Connection count multiplies by instance count. Postgres has a fixed max_connections; the ceiling arrives sooner than expected, and the failure mode is refused connections under load.

None of these are hard to fix, and none require redesigning the application. They are all the same fix in different clothes: move in-process state somewhere shared. In practice that means a Redis (or equivalent) for items 1 and 4, a STOMP broker relay for item 2, a distributed lock or a single designated scheduler instance for item 3, and pool arithmetic plus possibly PgBouncer for item 5.

What already works with multiple instances

Worth knowing, because it is the expensive half and it is already done:

• Authentication is stateless. SessionCreationPolicy.STATELESS (SecurityConfiguration.java) — JWT only, no server-side session, so no sticky sessions are needed for ordinary HTTP traffic.
• Account lockout is in the database, not in memory: failedLoginAttempts and lockedUntil are columns on the user record (User.java). Lockout stays correct across instances even though the rate limiter above does not.
• Nothing is written to the container filesystem. Payslip and certificate PDFs are generated and streamed rather than written to local disk, so there is no shared-storage problem to solve and no data lost on redeploy.
• Tenant isolation survives connection pooling. The row-level security scope is re-applied on every connection checkout and the connection is closed if that fails (TenantAwareDataSource.java) — so a pooled connection can never carry the previous borrower's tenant. See ADR-0013.
• Migrations tolerate simultaneous boots. Flyway takes a lock on its schema history table, so instances starting together serialize rather than race.

Two caveats on startup: the seeders (DemoDataSeeder, DefaultHolidaySeeder, PlatformAdminSeeder) are ApplicationRunners and therefore run on every instance boot, so they must stay idempotent; and long migrations still block startup for everyone.

Before you scale up: things to do first

Cheaper than adding servers, and they buy real headroom at Tier 2:

1. Set an explicit connection pool size rather than relying on the default, and size Postgres max_connections against it deliberately.
2. Load-test with realistic data volume. Payroll workloads are bursty — a period-end run is nothing like the average minute. The limits that matter are most likely in payroll generation and reporting, not in request throughput.
3. Watch the database first. The observability stack (monitoring.md) already ships Postgres metrics; slow queries and connection saturation will almost certainly bite before CPU does.
4. Check index coverage against your own data. Query plans that are fine on seeded demo data can change shape at 50,000 rows.

Deploying it yourself

The repository deploys to one VPS with a specific domain and registry, and a few values are wired to this project rather than parameterized — the image names in the compose files, the paths in the deploy scripts, and the hostnames in the Caddyfile. Standing up your own instance means editing those; see README.md, docker.md and ci.md for what each one does.

Two deployment models exist in the repository and it is worth knowing which you want: isolated stacks per client (ADR-0007) and shared-database multi-tenancy (ADR-0013), the latter superseding the former for the hosted offering. They imply very different operational work.

Honest summary

This is a well-built single-instance application with the expensive parts of multi-instance readiness — statelessness, tenant isolation, no local file state — already handled, and five specific pieces of in-process state that are not. It will serve a company comfortably on one machine. It will not survive being scaled horizontally today, and the failures if you tried would be quiet rather than loud, which is the dangerous kind.

Everything in the "what breaks" table is a known, bounded piece of work rather than a redesign. But none of it has been done, and the documentation should not pretend otherwise.