Federation: hub and nodes

The platform can run as one installation, or as a federation: each ministry, department or agency (MDA) runs its own back office, API and database (a node), and one national public portal (the hub) serves every e-Service of every MDA. This guide explains the shape, the trust model, the data rules and the operator's tasks. The contract itself is in the platform repository under zcm-backend/docs/federation/ and in the Federation API v1 reference.

Three roles, one codebase

The same image runs in one of three roles, set by PLATFORM_ROLE:

Role	What it is
`combined`	Hub and node in one installation (the default). Federation calls run through an in-process loopback adapter: the same signature, replay, scope, idempotency and schema checks, without a network hop.
`hub`	The national portal and aggregation. It keeps the node registry, signs citizens in with ZamPass, hosts the forms of node services, relays to nodes and keeps an index.
`node`	One MDA's back office and API. It owns its services, cases, payments and documents.

Each instance has a permanent NODE_KEY (for example hub, or a short MDA key) and its own ES256 signing key. Two public discovery documents describe it:

GET /.well-known/gsb-federation: contract version, role, NODE_KEY, where the keys and the /fed/v1 API are, and
(on a node) its hub. Nothing secret.
GET /.well-known/jwks.json: the public federation signing keys, including any key published for a rotation.

What lives where

Services are owned and designed on the MDA's own back office. The hub receives the catalogue by federation sync and never becomes the place where a node's service is designed: on the hub, editing a node's service answers 409 SERVICE_MANAGED_BY_NODE.

Data	Hub	Node
Citizen account, ZamPass link, contacts, preferences	Yes	A "shadow applicant": pairwise id and the claims released per assertion
Service catalogue	A signed mirror	The source
Form answers, documents, decisions, case history, payments, issued documents	Only while relaying (staged, deleted once the node acknowledges)	Yes
Application index: numbers, status words, dates, amounts due, document labels	Yes	The source
Staff accounts and the audit of case work	Hub staff only	Yes
Module data	No module for a ministry that runs its own node	Yes

The hub keeps only the index data it needs. Events carry index data only (numbers, status words, amounts, dates, the pairwise subject) and their schemas refuse anything else.

Signed, versioned messages

Every /fed/v1 request carries a detached compact JWS in the header X-GSB-Fed-Signature, ES256 over the canonical JSON of the body (keys sorted, no white space). The protected header binds the message to its context:

Member	Checked
`alg`	only `ES256`
`kid`	in the signer's JWKS or pinned keys
`typ`	`gsb-fed+jws` for requests, `gsb-fed-response+jws` for answers
`v`	contract version `1`
`iss`, `aud`	sender and receiver NODE_KEY; the sender must be an active peer
`iat`	at most 300 seconds old (`FED_MESSAGE_TTL_S`), at most 60 seconds ahead
`jti`	single use: a second delivery is `FED_REPLAY`
`htm`, `htu`	the HTTP method and path
`idk`, `fid`	the Idempotency-Key and the `X-GSB-Federation-Id` correlation id

Answers are signed too, and the caller refuses an unsigned or mismatched answer (FED_RESPONSE_UNVERIFIED). Transport is mTLS with one certificate per instance from the federation CA; with FED_MTLS_REQUIRED=1 the certificate's CN must be the peer's NODE_KEY.

Every write needs an Idempotency-Key. A retry is a new signed message with the same key, and gets the first answer again. The same key with a different body is 422 IDEMPOTENCY_KEY_REUSED. Bodies are validated against the JSON Schemas in docs/federation/schemas/, and every object refuses unknown fields (400 FED_INVALID).

The contract is versioned in the path (/fed/v1) and in the signature (v). A breaking change becomes /fed/v2, run side by side with v1.

What travels

Flow	How
Catalogue	The node pushes a signed service manifest (`PUT /fed/v1/catalogue/services/{id}`) after every published change; the hub checks the owner, content hash and manifest signature. A pull every 15 minutes is the safety net. Drafts never leave the node.
Citizen identity	ZamPass sign-in happens only at the hub. The hub issues a short-lived citizen assertion (ES256, 120 s, single use, a pairwise subject per node, only the claims the node's `sharedClaims` allow). A node session from an assertion is always clamped to an applicant.
Applications	The citizen fills the mirrored form on the hub; the hub relays it with `submissions.create`. If the node is down, the application is queued and delivered in order when it is back (unless the fee is quoted live).
Status	The node sends ordered CloudEvents (`case.`, `payment.`, `document.issued`, `action.required`); the hub applies them strictly in order and repairs gaps from the node's event log.
Payments	The node raises the invoice on its own ZamPay merchant and finalises it through its own single payment path. The hub never holds a node's ZamPay keys and never marks a node invoice paid.
Documents and verification	Documents stay on the node; the hub relays a short-lived link or proxies the bytes. Codes and receipts carry the node key, so the hub's verification pages route them to the right node.

Changing the contract

A change to a federation message is a contract change. Before you merge one:

Update the schema in zcm-backend/docs/federation/schemas/ and openapi.json. test/federation.test.js fails
when the code and the OpenAPI document disagree.
Keep it backward compatible, or ship hub and nodes together. Additive, optional fields are the normal path.
Because every object is closed, an older hub refuses a field it does not know: the documented rule is to upgrade the hub before its nodes. The citizen assertion is a JWT, so it grows additively and a node ignores claims it does not know.
Add two-instance tests (test/integration/*TwoInstances.test.js): a real hub and a real node, two processes and
two databases. Existing examples cover the catalogue, submissions, status, payments, permits, registers, geo search, status summary and staff single sign-on.
Run the end-to-end journeys in federated topology: E2E_TOPOLOGY=federated npm run e2e in e2e/ boots a hub
and a node and runs the fed-* journeys and the module journeys whose service can live on a node.
Ask for every feature: does this work when the service lives on another node? Check applying, paying,
notifications, documents and verification in all three roles.

Onboarding a new MDA node

Use this checklist with docs/deploy/per-mda-backoffices.md and the runbook deploy/runbooks/federation-node.md.

People and paperwork

Data-sharing agreement signed, record of processing (RoPA) entry made, federation DPIA update approved.
A NODE_KEY agreed: lower-case and permanent. A retired NODE_KEY is never reused.
The MDA's ZamPay merchant registered, with callbacks to the node's own API. Until then the node refuses to raise
invoices.

Platform

The MDA added to the estate manifest with its key, codes, modules and sizing; generated files rendered and
committed.
Namespace, database, bucket, secrets, DNS and federation certificate created by the platform team.
A signing key generated where it will live (never e-mailed) and stored in the secret store.
Deployed with the deploy scripts: test, then stage, then production by promotion only.

Federation

/.well-known/gsb-federation on the node shows role: node and its NODE_KEY; /.well-known/jwks.json shows
its key id.
Hub administrator A proposes the node under System administration → Federation (NODE_KEY, name,
ministries, base URL, scopes, shared claims); hub administrator B approves. The proposer cannot approve.
A signed ping succeeds both ways, and both audits show the same federation id.
The node's first administrators are set up with MFA; its modules are checked.
Later, the catalogue:write and events:write scopes are proposed and approved as the flows go live.

Operating the federation

Node status. The hub's Ministries overview asks every node for an aggregate status summary every 30 seconds
(status.summary). A node is green ("Running") when a fresh summary says it is ready.
Catalogue sync. GET /api/federation/catalogue/status shows the lag per node and what arrived only by the pull.
Failed pushes and pulls raise alerts to administrators.
Suspending a node is a registry change under maker-checker; it takes effect on the next message.
Key rotation publishes the new key first, then switches, then removes the old key after an overlap.
Outages. If a node is down, its services stay listed (marked stale), applications are queued, and its payments
are unavailable with a clear message. If the hub is down, node staff keep working and events queue on the node.

Take the Federation operator course to learn these tasks step by step.