Skip to main content
Clustering is an appliance-only capability, configured by the appliance owner under Admin → Cluster / High Availability. It is not available on the hosted (SaaS) service, and it requires the HA clustering license capability (included with the Enterprise Advanced tier, or added per-deal).

​
What clustering gives you

A standalone appliance runs a single database. Clustering joins 2 or more appliances into one logical system so that a node failure does not take the service down or lose data:
  • Single writable primary + synchronous standbys — one node is the primary; every other node holds a full, continuously-streamed replica. Writes are acknowledged only once a synchronous standby has them, so an intra-datacenter failover loses zero committed data (RPO 0).
  • Automatic failover — if the primary fails, a healthy standby is promoted automatically (typically within ~10 seconds) and the ingress address follows it. No manual intervention.
  • Stable ingress — users always reach one address (a floating VIP, your load balancer, or a DNS name) that survives every failover.
  • Optional shared session/job tier (P2) — clustered Redis + RabbitMQ so user sessions and in-flight jobs also survive a node loss, not just committed data.
This is single-primary replication (writes route to the current primary, reads are served everywhere) — not multi-master. This is a deliberate design choice: it keeps the tamper-evident audit hash-chain and access-control model strictly consistent, which multi-master cannot guarantee.

​
Prerequisites

Work through this checklist before you start. A missing prerequisite is the most common cause of a failed join.
The primary’s license must include the HA clustering capability. It is a default of the Enterprise Advanced tier and can be added to other tiers per-deal. Without it, Create cluster returns a “not licensed” error. Check under Admin → Licensing; contact your account team to add it.
Every node must run the same appliance version (build the OVA once, deploy it to every node — do not mix versions), and each should have the same resource tier (CPU/RAM/disk). A cluster node needs at least the standard production sizing; the eval/8 GB tier is for single-node testing only.
Every node must be NTP-synchronized to a common source before joining. Clustering coordination and the sealed join token are time-sensitive; a skewed clock breaks membership and rejects the token. The appliance ships with the hypervisor’s guest time-sync disabled on purpose — use NTP as the single clock discipline. Create cluster / Join will refuse to run on an unsynchronized clock. (A lab-only override exists in the wizard for disconnected test benches — never use it in production.)
Each node needs a unique machine hostname — it becomes that node’s cluster member identity and is fixed once the node joins. Every OVA boots as enclave, so you must rename each node before it joins (for example enclave-01, enclave-02, enclave-03). Set it on the console with cluster node-name enclave-02, or via the hostname control on the first-run screen.
  • All nodes must reach each other over the network. For the built-in VIP ingress mode, all nodes must be on the same subnet (Layer 2); for cross-subnet or cross-site, use a load balancer or DNS ingress instead.
  • Reserve a spare IP on the subnet for the VIP (the floating address users hit) — it must not be assigned to any host.
  • Open these ports between nodes (the appliance firewall opens them on cluster activation, but any upstream network ACLs must allow them too):
Take a fresh backup of the node that will become the primary before you start (Admin → Backup & Restore). Converting a standalone appliance to a primary preserves its data and takes an automatic safety snapshot first, but a known-good backup off the box is always the right precaution.
The joining nodes are wiped. When a node joins, its local database is replaced by a full copy streamed from the primary. Only join fresh appliances (or nodes whose local data you are willing to discard). The primary keeps all its data.

​
Choose an ingress mode

Ingress is the stable front-door address users always hit. Pick one up front — you set it when you create the cluster.

​
Step-by-step: build the cluster

You can do everything from the admin GUI (recommended). The console is the break-glass equivalent for an operator without browser access. The GUI stages each action and runs it on the host; the console runs the same verbs directly.
1

On the FIRST node — create the cluster

This promotes your standalone appliance to the cluster primary. Its existing data is adopted in place (a safety backup is taken first).
  1. Go to Admin → Cluster / High Availability.
  2. Click Create cluster.
  3. Choose the ingress mode and, for VIP, enter the reserved VIP address and the full list of node IPs.
  4. Confirm. The board turns green with this node shown as Leader and the status Healthy — single node (no standby yet).
2

On the primary — mint a join token for the new node

A join token is a single-use, time-limited sealed bundle carrying the cluster’s shared secrets. It is shown once — copy it now.
  1. On the primary’s Cluster page, click Add node.
  2. Copy the sealed join bundle and its one-time passphrase. Send the bundle and the passphrase to the new node over separate channels.
The token expires in 15 minutes and works only once. Mint a fresh one if it lapses. The bundle never contains the cluster CA private key — a leaked bundle alone cannot mint a trusted member.
3

On the NEW node — join the cluster

Do this on the fresh appliance you are adding. Make sure its hostname is unique first.
  1. On the new node, go to Admin → Cluster / High Availability.
  2. Click Join existing cluster.
  3. Paste the join bundle and the passphrase, and enter the full list of node IPs.
  4. Submit. A live progress panel shows each phase: Enrolling → Awaiting operator approval → Adopting → Registering → Starting → Streaming → Verifying.
4

On the primary — approve the new node

The joining node stops at Awaiting operator approval and displays a CSR fingerprint on its own screen/console. This human check is what stops a rogue node from joining.
  1. On the primary, a Nodes awaiting approval card appears with the node’s CSR fingerprint.
  2. Verify that fingerprint matches exactly the one shown on the joining node, then click Approve.
By default, a valid one-time token auto-approves the node (the token is the authorization). To require an operator to verify the fingerprint by hand for every node, set CLUSTER_ENROLL_AUTO_APPROVE=false on the primary before minting tokens — the node then waits at Awaiting approval until you click Approve.
5

Verify the node is streaming

After approval the node adopts the primary’s data and starts streaming.
The board now lists two rows: enclave-01 — Leader — Up and enclave-02 — Sync Standby — Up — Lag 0. Status shows Healthy · Quorum 2/2 OK.
6

Repeat for each additional node

Mint a fresh token per node and repeat steps 2–5 for node 3 (and beyond). Aim for 3 nodes for production HA.
After a node finishes joining, it adopts the cluster-wide session keys, so your admin session on that node is rotated — sign in again (with your cluster admin credentials) to keep managing it. This is expected and only affects the node that just joined.

​
Optional: enable the shared session/job tier (P2)

Phase 1 (above) makes your database highly available. P2 additionally makes Redis (sessions, one-time codes, rate limits) and RabbitMQ (in-flight background jobs) cluster-wide, so a failover no longer logs users out or drops running jobs.
1

Enable on the leader first, then every other node

On the Cluster page, click Enable P2 — first on the Leader node, then on each other node. It is idempotent; the peer list and primary are detected automatically.
2

Verify

P2 is per-node — a node you skip keeps its own private Redis/RabbitMQ and is not part of the shared tier. Run Enable P2 on every node. Redis and RabbitMQ failover then run independently of the database primary.

​
Test a failover

Once you have 3 healthy nodes, prove the cluster does its job:
  1. Power off the primary node.
  2. Within ~10 seconds a standby is promoted to primary and the VIP moves to it. Users on the ingress address stay connected (or reconnect immediately).
  3. Confirm on any surviving node: cluster status shows a new Leader on a higher timeline, and the remaining standby re-pairs to it.
  4. Power the old primary back on — it rejoins automatically as a standby and catches up.

​
Day-2 operations

cluster rollback is the safe way to take a node out of a cluster — it restores standalone operation and keeps the node serving. Use it before you re-cluster a node, or to wind a cluster down.

​
Removing and decommissioning a node

Remove takes a node out of the cluster permanently (hardware retirement, replacement, or recovering from an unhealthy member). It is a three-part operation, and the last part is yours:
1

Remove it from the cluster

On the Cluster board, node ⋮ → Remove (you cannot remove the current primary — switch over first). The cluster deregisters the node, revokes its membership certificate so it can never silently re-join, and fences its database access: replication and application connections from that node are rejected immediately, so no new data reaches it even while it is still powered on.
2

Power the node off

A removed node that is still running shows on the board as “Fenced — power off this node to complete removal” (it re-announces itself until it is shut down — this is expected). Power the machine off to complete the removal; the board then returns to the surviving members only.
3

Wipe or destroy the machine

Until its disk is wiped, the removed node still holds a full replica of your data as of the moment it was fenced. Treat it like any retired database server: securely wipe the virtual disk or destroy the VM as your data-handling policy requires.
Re-using the same machine later is fully supported: return it to standalone (cluster rollback), or reinstall it, and join it again with a fresh token — a legitimate re-join under the same name automatically lifts the fence.

​
Security model

  • Mutual TLS everywhere — every inter-node channel (database replication, etcd, the Patroni control API) is authenticated with certificates from an internal cluster CA. An on-network stranger without a CA-signed certificate cannot read the coordination store or register as a member.
  • CA private key stays on the host — it is never mounted into a container, so a compromised application container cannot mint cluster certificates. The key lives only on the node where the cluster was created; it is never copied to other members or into join bundles. You can run Add node and approvals from any node’s page — the certificate signing itself is always performed by the cluster-creation node automatically. Protect that node’s backups: if it is permanently lost, new nodes can no longer be enrolled into this cluster’s PKI (existing members keep working).
  • Sealed, single-use join tokens — the join bundle is encrypted and authenticated under a one-time passphrase (sent over a separate channel), carries an authenticated 15-minute expiry, and does not contain the CA key.
  • Post-quantum ready — node enrollment is authorized with a hybrid (classical + post-quantum, Ed25519 + ML-DSA-65) attestation, and inter-node transport negotiates a hybrid key exchange where the components support it.
  • Every action is audited — create/join/approve/reject/switchover and failovers are written to the tamper-evident audit log, and the audit hash-chain is re-anchored on every failover so integrity survives a primary change.

​
Troubleshooting

The primary’s license does not include the HA clustering capability. Add it under Admin → Licensing (Enterprise Advanced includes it, or request a per-deal grant), then retry.
Bring the node’s clock into NTP sync with the rest of the cluster and retry. Do not use the lab override outside a disconnected test bench.
Tokens are single-use with a 15-minute lifetime. Mint a fresh one on the primary (Add node / cluster join-token) and join again.
Every node needs a unique hostname before joining. Set it with cluster node-name enclave-02, then re-run the join.
A failed join safely rolls the node back to standalone and keeps it serving — just fix the underlying issue (clock, network, hostname) and retry. Detailed logs are on the node under /var/log/enclave/cluster/.
A node that just joined rotated its session keys. Sign out and back in on that node (with your cluster admin credentials).
On releases before 0.11.6, minting a join token and signing the approval had to run on the cluster-creation node (the only holder of the cluster CA key) — after a failover, requests could land on another member and fail with this message. Workaround on those releases: open the cluster-creation node’s own address directly, switch Node-join approval to manual, mint and approve from there. From 0.11.6 onward this works from any node: token minting no longer needs the CA key, and approvals are routed to the CA-holding node automatically.
Expected until it is powered off — a removed-but-running node keeps announcing itself. Its database access is already fenced (no data flows to it); the board marks it “Fenced — power off this node to complete removal”. Shut the machine down and the entry disappears.

​
Day-2: certificates, key recovery & revocation

The cluster runs its own internal certificate authority (CA) for the encrypted channels between nodes. The CA private key lives only on the node you created the cluster on. These day-2 operations keep that trust healthy over the appliance’s life.

Automatic certificate renewal

Per-node certificates auto-renew well before expiry — the CA-holding node renews its own, and replicas renew through the CA holder automatically on a background schedule (no admin page needs to be open). No action needed unless a node warns it could not renew (then check that the CA-holding node is up).

CA-key escrow & recovery

Escrow the CA key under a K-of-N operator quorum so it survives the loss of the CA-holding node — no more re-building the whole cluster. Set this up once, soon after creating the cluster, and rehearse the recovery drill.

Revoke a node's certificate

Decommissioned or compromised a node? Revoke its certificate so it is rejected by the cluster. Revocation is distributed to every node automatically; strict enforcement at the encrypted channels (etcd rejects a revoked node) is turned on per node with cluster-crl-enforce on (it validates the CRL first, then applies on the next stack restart) — see the certificate-expiry runbook.

Verified switchover & self-healing rolls

Planned leader switchover now confirms leadership actually moved before reporting success, and a stalled rolling upgrade self-heals (retrying every source) and shows each node’s own reason on the board.
Set up CA-key escrow before you need it. Without escrow, losing the cluster-creation node means rebuilding the cluster from scratch. The escrow bundle is stored off-box and is useless without a quorum of custodian shares.

​
Set up CA-key escrow (once, soon after creating the cluster)

1

Choose custodians and a quorum

Pick N custodians and a threshold K (e.g. 3-of-5). Each custodian generates an age key pair on their own machine and shares only their public recipient (age1…) — the private key stays with them, off-box.
2

Create the escrow bundle on the CA-holding node

From the console on the cluster-creation node, run cluster-ca-escrow -k K -n N -o <bundle> <recipient…>. It writes an off-box bundle (the CA key encrypted under a random secret, plus one sealed share per custodian). The plaintext CA key never leaves the node and is never seen by the application.
3

Distribute and store

Move the bundle to secure off-box storage and give each custodian their own share-<i>.age. On the Cluster admin page, confirm the badge reads CA-key escrow: Configured (K-of-N).
4

Rehearse the recovery drill

Prove it works before you depend on it: gather K decrypted shares and run cluster-ca-recover --dry-run — it reconstructs and verifies the key against the CA without installing anything.

​
Recover the CA key onto a replacement node

1

Confirm the loss

The CA-holding node is permanently gone and no other node holds the key (new node certs, renewals and enrollments start failing). You have the off-box bundle and can reach at least K custodians.
2

Collect a quorum of shares

Each of ≥K custodians opens their share off-box (age -d -i <identity> share-<i>.age) and returns the decrypted value. Assemble them, one per line, into a shares file.
3

Rebuild the key on the new holder

On the replacement node, run cluster-ca-recover -b <bundle> -s <shares>. It verifies the shares against the bundle and installs the CA key only on a passing check. That node is now the CA holder; enrollments and renewals resume automatically.
The Cluster admin page shows CA-key escrow state at a glance — Configured, Not configured (the CA key is a single point of loss — set up escrow), or Unknown. Escrow setup and recovery are deliberately console-only ceremonies: the CA key never touches the web tier, and recovery works even when the cluster is down.

​
Remove a node and re-join it later

Removing and re-adding a node is a console procedure run on a healthy member — the appliance cannot stop a remote node’s stack, so a removed box is powered off (or rolled back to standalone) and then re-joined with a fresh certificate.
1

Remove the node (on a healthy member)

cluster-remove <node> deregisters it from etcd + Patroni, revokes its enrollment and certificate, and fences its database access. This works whether the node is still live or already gone (revoked/evicted/powered off) — in the already-gone case it also cleans up the stale etcd membership for you, so there is never any manual step.
2

Return the removed box to standalone (on that box)

If the box is still running and you want to re-use it, roll it back with cluster-rollback. It becomes a standalone appliance on its own IP again.
3

Re-join it (Add node)

Mint a join bundle with cluster-join-token (or the Cluster → Add node button), run cluster-join <bundle> <all-node-ips> on the box, then approve its CSR fingerprint on the Cluster board. It receives a fresh, non-revoked certificate — accepted even with strict CRL enforcement on — and streams back into the cluster.
Revoking a node does not instantly kick it out. Revocation (and the CRL) reject a node at the next connection attempt, so a revoked node can never re-join — but a node that is already connected keeps running until it is restarted or removed. To evict a live or compromised node, use cluster-remove (or power it off). And always revoke first, then turn on cluster-crl-enforce and restart each survivor — etcd reads the revocation list at start-up, so enabling enforcement before the revoke leaves a survivor accepting the old certificate until its next restart.
These are operator (console) procedures. See the operator runbooks for exact, copy-paste step-by-step instructions: CA recovery, certificate expiry & revocation, node removal & re-join, and rolling-upgrade recovery.
Diagnostics. support-bundle produces a secret-scrubbed tarball that now includes per-container application logs and full cluster PKI state (etcd membership, the CRL, and the certificate/enrollment ledger) — everything support needs to diagnose a cluster or app-tier fault off-box. For a deeper session, raise verbosity with log-level debug (recreates just the backend) and set it back to info afterwards.