Monitoring nodes, environments and changing the contract

Once nodes are live, the operator's job is to see trouble early, keep the federation healthy through outages and restores, move releases safely through the environments, and make sure contract changes never break a peer. This lesson covers all four.

The Ministries overview and node status

The hub's back-office home page is the Ministries overview: one card per ministry back office with its status in words, its active services, the applications it received today and its outstanding incidents.

The hub collects this with the status.summary operation (scope status:read). The job fed-status-pull asks every active node, five at a time, about twice a minute, and keeps the last signed answer per node. Only aggregate counts and the node's readiness travel: never a name, a case number or anything about a person. An administrator can press Check now, at most every 30 seconds.

Status	Meaning
Green: "Running"	A summary received within 60 seconds that says the node is ready
Red: "Not ready"	A fresh summary, but a readiness check failed (database, migrations or storage); the card says which
Red: "Not reachable"	No answer, or a refused or invalid one
Red: "Not checked recently"	The hub's job has not run
Red: "Suspended" / "Not registered"	The registry says so
Unknown: "Not reporting"	The node answers a signed ping but not the summary (an older build, or a node without the `status:read` scope)

Catalogue sync and alerts

Hub: GET /api/federation/catalogue/status shows, per node, the lag between the node signing a version and the
hub applying it, and how many versions arrived only through the 15-minute pull (viaPull). A high viaPull count means pushes are failing.
Node: the same endpoint shows lagSeconds, oldestUndeliveredAt and lastDeliveredAt for its outbox.
Pushes are retried with back-off from 30 seconds up to an hour. A refusal, a push given up, and the third failed try
are audited federation.catalogue.push_failed, and the alert rule federation-catalogue-push-failing notifies administrators. On the hub, repeated pull failures raise federation-catalogue-pull-failing.

Other places to look:

What	Where
Queued applications per node	`GET /api/federation/submissions-queue`
Dead status events on a node	Redeliver with `POST /api/federation/status-events/:id/redeliver`
One message across both systems	Search both audit trails for its `X-GSB-Federation-Id`
Registry and keys	`GET /api/federation/status` (role, NODE_KEY, hub, active and published key ids, mTLS)

Outages

Failure	What happens
Node down	Its services stay listed, marked stale with the time last confirmed. New applications are queued (or refused if the fee is quoted live). Status shows the last known value. Payments for that node are unavailable with a clear message; nothing is charged. The node's card is red.
Hub down	Node staff and POS keep working. Events queue on the node and are retried for about 22 hours, then kept in a dead-letter list that can be replayed.
Network partition	Both sides queue. Idempotency keys make replay safe.

When the node returns, the hub delivers queued applications in order and repairs any gaps in status events from the node's log. No operator action is normally needed.

Backup and disaster recovery per node

Each node is backed up and restored on its own: its database, its bucket and its secret-store path.

After restoring a node, the hub may hold events or submissions the node no longer has. Replay them from the hub's
queues: the idempotency keys make replays safe.
The NODE_KEY never changes. The signing key and certificate are restored from the secret store or re-issued.
A disaster-recovery instance uses the same NODE_KEY, signing key and federation host name, so peers need no registry
change.

Environments and promotion

Environment	Purpose	Rules
Development	A developer's own machine, with a local PostgreSQL and PostGIS	Never point local tools at a remote database
Test	A shared test deployment for demos and product-owner review	Deploy only when the owner asks; never real citizen data; provider test tenants or stubs
Staging	Government servers, the same infrastructure as production	Helm overlays per cluster; images promoted by digest
Production	Government servers	Reached only by promotion from staging, with approvals; no direct deploys

Configuration differs between environments only through environment variables and Helm values, never code branches. Production refuses development keys, such as a local signing key or the development CA.

A production run of the deploy scripts:

refuses --tag and deploys exactly the image digests that staging recorded in deploy/estate/releases/stg.json;
refuses when the chart, the estate or the generator changed since the commit that staging deployed;
needs a clean checkout, the cluster identity, --yes and a typed confirmation;
stops at the first failure.

Rollback uses Helm (a failed upgrade rolls back by itself) or Argo CD with the earlier digests. Migrations are expand-only, so the previous image runs on the newer schema and a rollback never runs a down-migration.

Changing the federation contract

A change to a federation message is a contract change. The contract lives in zcm-backend/docs/federation/ (OpenAPI, JSON Schemas and the events catalogue) and is published as the Federation API v1 reference.

Update the schema and openapi.json. A test fails when the code and the OpenAPI document disagree.
Stay backward compatible, or ship hub and nodes together. Additive, optional fields are the normal way. Every
object refuses unknown fields, so an older hub refuses a field it does not know: upgrade the hub before its nodes. New operations are advertised (for example as ping capabilities) so the hub asks only nodes that offer them, and an older node's 404 or 501 is read as "not offered".
A breaking change becomes /fed/v2, run side by side with v1. When v1 is to be retired, FED_V1_SUNSET adds
Deprecation and Sunset headers to every v1 answer.
Add two-instance tests (*TwoInstances.test.js): a real hub and a real node, two processes, two databases.
Run the end-to-end journeys in federated topology: E2E_TOPOLOGY=federated npm run e2e boots a hub and a
node, registers the node under maker-checker, and runs the cross-instance journeys.

Check your understanding

What information travels in a node's status summary, and what never does?
What does a red "Not ready" card tell you that "Not reachable" does not?
Which catalogue figure suggests that pushes are failing even though the portal looks current?
What happens to a citizen's application if the node is down when they submit?
How does a production deployment know which image to run?
In which order do you upgrade the hub and its nodes for an additive contract change, and why?