Skip to main content

14 — Schedulers, sockets, and mail: the work between requests

Read this first: everything you have read so far started with an HTTP request. This lesson is the rest: the code that runs when nobody is calling — scheduled jobs, WebSocket frames, outbound email, and the metrics that watch all of it. The recurring question, fresh from lessons 12 and 13, is the same every time: there is no request here, so whose tenant is this? By the end you can enumerate every piece of background machinery and answer that question for each one.

Time: about 35 minutes. Assumes lesson 13.

The complete inventory​

Five pieces of background work exist. Four are @Scheduled; the fifth deliberately is not.

Job                                          When                        Runs as
-------------------------------------------  --------------------------  ----------------------
HolidayGenerationScheduler.seedUpcomingYear  Dec 1, 03:00 Asia/Manila    each tenant, in a loop
TrialExpiryScheduler.expireFinishedTrials    daily, 03:30 Asia/Manila    global
AltchaService.sweepExpiredSignatures         every 5 min (fixedDelay)    no tenant (in-memory)
SignupVerificationService.sweepExpiredCodes  every 5 min (fixedDelay)    no tenant (in-memory)
LoginRateLimitFilter.sweepIdleEntries        opportunistic, not a job    no tenant (in-memory)

Only the first two touch the database, and they answer the tenant question in opposite ways. That contrast is the heart of this lesson.

The holiday job: one cron, N tenants​

HolidayGenerationScheduler states its own reason for existing:

/**
 * Yearly safety net for servers that run across a year boundary without a restart:
 * every December 1 the upcoming year's default holidays are seeded so January payroll
 * never starts against an empty calendar. Delegates to the same idempotent service
 * method as {@link DefaultHolidaySeeder} and the manual generate endpoint.
 */

Predict: the cron fires at 03:00 on December 1. Which tenant's holiday table does it write?

There is no request, so there is no X-Tenant header, no JWT, nothing to inherit a tenant from — and after lesson 13 you know the database would reject an unscoped write anyway. The method comment resolves it:

/**
 * Each tenant keeps its own holiday calendar -- the defaults are national, but companies edit
 * them and add their own -- so this generates per tenant rather than once. A tenant whose
 * generation fails is logged and skipped so the rest still get a populated January.
 */
@Scheduled(cron = "0 0 3 1 12 *", zone = "Asia/Manila")

It writes every active tenant's table, one at a time, by handing the work to TenantScopedExecutor.forEachActiveTenant. That class's javadoc is the payoff line for the whole tenancy arc:

Scheduled jobs and startup seeders used to be able to ignore the question entirely — there was one company, so "generate next year's holidays" meant one thing. With many, the same job has to be a loop, and each iteration has to declare whose data it is touching.

Inside the loop, each iteration runs under TenantContext.runAs(tenant.getTenantId(), ...) — the exact mechanism a request filter uses in lesson 12, just driven by a for loop instead of an HTTP header. A failing tenant is caught, logged with the job's name ("used only for logging, so a failure names the job rather than a stack frame"), and skipped — one company's bad configuration must not cost every other company its January calendar.

The trial sweep: a job that is not the enforcement​

TrialExpiryScheduler closes trials whose window has passed — and its javadoc immediately tells you what it is not:

Nightly rather than continuous, because this is not what stops an expired trial from being used: Tenant#isActive() reads the clock on every request, so a lapsed trial is already locked out before this runs. The sweep exists so the registry the operator looks at agrees with that, rather than showing a row as TRIAL for another few hours.

Enforcement is the clock; the sweep is bookkeeping. Even its schedule is a documented decision:

Runs half an hour after the holiday scheduler's slot so two per-tenant jobs never contend for the connection pool on the one night of the year they coincide.

And unlike the holiday job, this one runs global — with a comment explaining both the scope and the placement of the scope:

/**
 * The scope has to be established around the <em>call</em>, not inside the service method:
 * Hibernate fixes a session's tenant when the session opens, and the transaction starts as the
 * method is entered. Global is right here -- the tenant registry belongs to no tenant.
 */
@Scheduled(cron = "0 30 3 * * *", zone = "Asia/Manila")
public void expireFinishedTrials() {
    int expired = TenantContext.runAsGlobal(tenantAdminService::expireFinishedTrials);

Two nightly jobs, thirty minutes apart, one per-tenant and one global — each with the reasoning written where you will find it when you need it.

Two sweeps that only forget​

The remaining @Scheduled methods are five-minute fixedDelay housekeeping over in-memory maps, so the tenant question does not arise — no database is touched. AltchaService.sweepExpiredSignatures drops solved-captcha signatures from the replay registry once their timestamps lapse. And SignupVerificationService.sweepExpiredCodes says exactly how much it matters:

/** Same shape as the ALTCHA signature sweep: entries expire on their own, this just forgets them. */

Expiry is checked at use in both cases; the sweep only reclaims memory.

The counter-example: the sweep that cannot be @Scheduled​

The login rate limiter (lesson 10) also holds an in-memory map — a Bucket4j bucket per client IP and path — and it also needs eviction. But:

Opportunistic eviction of idle buckets. The filter isn't a Spring bean (it's constructed inline in SecurityConfiguration), so @Scheduled isn't available — instead the first throttled request after each sweep interval pays the cleanup cost.

@Scheduled only works on beans the container manages. So sweepIdleEntries() runs at the top of doFilterInternal, at most once per ten minutes. And eviction here has a subtlety the other two sweeps do not, because dropping this state is not neutral:

Dropping a bucket hands back a full one, so an entry may only be evicted once it would have refilled anyway. That is why the retention below is the longest window any path configures rather than the sweep interval: the signup bucket refills over an hour, and evicting it after ten minutes of quiet would hand out a fresh bucket six times an hour — multiplying the hourly limit by six.

A sweep that forgets a captcha signature loses nothing. A sweep that forgets a rate-limit bucket resets a limit. Same shape, different stakes.

WebSockets: authentication happens once, tenancy happens every frame​

WebSocketConfig is 36 lines: a /ws SockJS endpoint, a simple in-memory broker serving /topic and /queue, /app as the application prefix, /user for per-user queues. The interesting file is the interceptor it registers. WebSocketAuthChannelInterceptor validates a Bearer token from the Authorization header on the CONNECT frame only, resolves the user (under a global scope, "since that is what is being looked up"), and stores the principal on the session. Every later frame reuses it:

// Every other frame runs on a messaging thread with no request behind it, so the tenant
// has to be re-established from the principal captured at CONNECT. Cleared in
// afterSendCompletion -- these threads are pooled just like request threads.

That is lesson 12's discipline transplanted to a second thread pool: set the context on the way in, clear it on the way out, because the thread will be reused by someone else.

Predict: an authenticated client from tenant 7 sends SUBSCRIBE /topic/audit-logs.3. The JWT is valid. What stops it?

Nothing in the broker — a simple broker will happily fan out to any subscriber. What stops it is rejectForeignTenantSubscription, which runs on every SUBSCRIBE frame and compares the requested suffix to the tenant just re-established from the principal:

Publishing per tenant only keeps data apart if subscribing does too — otherwise anyone who can guess a tenant id can listen to another company's live audit trail, which needs no query and no permission, just a destination string.

Tenant 7 asking for .3 gets a MessageDeliveryException; only a global principal may cross.

The three producers​

Three services push into that broker. AuditServiceImpl broadcasts each audit entry to a tenant-suffixed topic, and its comment explains why the suffix is not optional:

// Per tenant, not global. The live audit tail is a stream of "who did what to which
// record", so a single shared topic would push one company's activity to every other
// company's administrators -- the one place in the system where a leak needs no query
// at all, just a subscription.

The topic name comes from one public method, auditTopicFor(tenantId), "so the WebSocket authorization rules and the frontend subscription agree on one definition rather than two string literals that can drift apart." NotificationServiceImpl.createAndPush saves a notification and sends it to /queue/notifications for each user account behind the employee — a per-user queue, so the broker's /user machinery does the isolating. ConversationServiceImpl does the same for chat: /queue/messages and /queue/conversations, addressed to a username, never broadcast.

Mail: the send that must not throw​

The MailService port is small enough to quote whole:

/**
 * Sends transactional email.
 *
 * <p><strong>Implementations must not throw.</strong> Every current caller is finishing a business
 * action that has already committed -- a provisioned tenant, a created account -- and an exception
 * escaping the send would turn a completed action into a 5xx. In the provisioning case that would
 * also destroy a one-time password permanently, since it is never stored in plaintext. A failed
 * send is therefore logged and counted (see {@link com.motorph.payroll.metrics.MailMetrics}), not
 * propagated. That makes dropped mail invisible to the caller by design, which is why the counter
 * exists.
 */

SmtpMailService implements it with one injection trick and one startup refusal. The trick: the JavaMailSender arrives as an ObjectProvider, because Boot only autoconfigures a sender when spring.mail.host is set, and "a hard dependency would stop the application from starting on any deployment that has not configured a relay — and motorph.mail.enabled=false could not prevent it, because injection happens before any flag is read. Boot 4 ships a failure analyzer for exactly this mistake." The refusal: MAIL_ENABLED=true with an empty MAIL_HOST still satisfies Boot's property condition, producing a sender whose every send fails at connect time — so @PostConstruct validate() rejects that combination at boot, because "mail configured to go nowhere should be a boot failure naming the variable, not a silent leak of every credential this product mails out."

Rendering: two bodies, no engine​

MailTemplateRenderer substitutes {{token}} values into template pairs, and defends both decisions in its javadoc. Why no Thymeleaf:

Substitution is a deliberate {{token}} replace rather than a template engine. The emails this product sends are short and few, and adding Thymeleaf would bring a resolver, a dialect and a caching story for the sake of string interpolation — the same reasoning that kept ALTCHA and the payment provider hand-rolled behind a port.

Why every mail is two files:

Every mail has two bodies: mail/<name>.html and mail/<name>.txt. Both are required. A plain-text alternative is not decoration — it is what a text-only client, a screen reader and most spam filters actually read, and writing it by hand keeps the two saying the same thing.

And the two rules that make hand-rolled substitution safe: values are HTML-escaped in the HTML body and left alone in the text body, "because company and person names arrive from user input with no character restrictions. And a token the model does not supply is an error, not an empty string: it fails here, in a unit test, instead of mailing a customer a literal {{username}}."

Coda: the meters watching all of it​

The metrics/ package holds four classes — AuthMetrics, BusinessMetrics, MailMetrics, BuildInfoMetrics — all feeding /actuator/prometheus. You have already met MailMetrics (the counter that makes swallowed sends visible) and AuthMetrics (the 429s from lesson 10's rate limiter). BusinessMetrics counts committed work at the service layer, and carries the one rule that governs every tag in the package:

Never tag with an employee number, a payroll id, or a tenant slug — one series per employee is how a Prometheus dies, and per-tenant series would put customer shape into a store that has no access control worth the name.

Tags come from small fixed sets defined in the file; identity never becomes a time series.

Where this shows up in MotorPH​

Recap​

  • Background work must declare its tenant. Per-tenant jobs loop with TenantScopedExecutor; registry work runs runAsGlobal — around the call, because Hibernate fixes the session's tenant when the session opens.
  • A sweep is not always the enforcement. Trial lockout is Tenant#isActive() reading the clock per request; captcha and signup codes expire at use. The sweeps just make state agree.
  • Eviction can be a grant. Dropping a rate-limit bucket hands back a full one, so retention must be at least the longest refill window — and a non-bean cannot use @Scheduled at all.
  • WebSocket frames are requests without a request. Re-establish the tenant per frame, clear it after, and gate SUBSCRIBE — publishing per tenant only isolates if subscribing does too.
  • Mail never throws and metrics never carry identity. A committed action must not become a 5xx over SMTP, and a per-employee tag is how a Prometheus dies.

Next: 15 — The payroll spine: runs, payslips, and a ledger that never computes.