> ## Documentation Index
> Fetch the complete documentation index at: https://docs.kvelden.com/llms.txt
> Use this file to discover all available pages before exploring further.

# High availability & clustering

> Join two or more Enclave appliances into a single high-availability cluster with automatic failover, synchronous replication (RPO 0), and a stable ingress address — configurable entirely from the admin GUI or the console.

<Info>
  Clustering is an **appliance-only** capability, configured by the appliance
  **owner** under **Admin → Cluster / High Availability**. It is not available on
  the hosted (SaaS) service, and it requires the **HA clustering** license
  capability (included with the Enterprise Advanced tier, or added per-deal).
</Info>

## What clustering gives you

A standalone appliance runs a single database. Clustering joins **2 or more
appliances** into one logical system so that a node failure does not take the
service down or lose data:

* **Single writable primary + synchronous standbys** — one node is the primary;
  every other node holds a full, continuously-streamed replica. Writes are
  acknowledged only once a synchronous standby has them, so an intra-datacenter
  failover loses **zero committed data (RPO 0)**.
* **Automatic failover** — if the primary fails, a healthy standby is promoted
  automatically (typically within \~10 seconds) and the ingress address follows
  it. No manual intervention.
* **Stable ingress** — users always reach one address (a floating VIP, your load
  balancer, or a DNS name) that survives every failover.
* **Optional shared session/job tier (P2)** — clustered Redis + RabbitMQ so user
  sessions and in-flight jobs also survive a node loss, not just committed data.

<Note>
  This is **single-primary** replication (writes route to the current primary,
  reads are served everywhere) — **not** multi-master. This is a deliberate design
  choice: it keeps the tamper-evident audit hash-chain and access-control model
  strictly consistent, which multi-master cannot guarantee.
</Note>

## Prerequisites

Work through this checklist **before** you start. A missing prerequisite is the
most common cause of a failed join.

<AccordionGroup>
  <Accordion title="License — HA clustering enabled">
    The primary's license must include the **HA clustering** capability. It is a
    default of the **Enterprise Advanced** tier and can be added to other tiers
    per-deal. Without it, **Create cluster** returns a "not licensed" error.
    Check under **Admin → Licensing**; contact your account team to add it.
  </Accordion>

  <Accordion title="Node count — 3 recommended, odd numbers">
    * **1** = standalone (no HA).
    * **2** = supported, but a two-node cluster cannot form a quorum on its own —
      run **3** for automatic failover without split-brain risk.
    * **3 (recommended)** = one primary + two standbys; tolerates the loss of any
      single node with automatic failover.

    Use an **odd** number of nodes so the coordination layer (etcd) always has a
    clear majority.
  </Accordion>

  <Accordion title="Identical build + resources">
    Every node must run the **same appliance version** (build the OVA once, deploy
    it to every node — do not mix versions), and each should have the **same
    resource tier** (CPU/RAM/disk). A cluster node needs at least the standard
    production sizing; the eval/8 GB tier is for single-node testing only.
  </Accordion>

  <Accordion title="Time sync (NTP) — a hard requirement">
    Every node **must** be NTP-synchronized to a common source before joining.
    Clustering coordination and the sealed join token are time-sensitive; a skewed
    clock breaks membership and rejects the token. The appliance ships with the
    hypervisor's guest time-sync **disabled** on purpose — use NTP as the single
    clock discipline. `Create cluster` / `Join` will refuse to run on an
    unsynchronized clock. (A lab-only override exists in the wizard for
    disconnected test benches — never use it in production.)
  </Accordion>

  <Accordion title="Unique hostname per node">
    Each node needs a **unique machine hostname** — it becomes that node's cluster
    member identity and is fixed once the node joins. Every OVA boots as `enclave`,
    so you must rename each node **before** it joins (for example `enclave-01`,
    `enclave-02`, `enclave-03`). Set it on the console with
    `cluster node-name enclave-02`, or via the hostname control on the first-run
    screen.
  </Accordion>

  <Accordion title="Network — reachability, ports & the VIP">
    * All nodes must reach each other over the network. For the built-in **VIP**
      ingress mode, all nodes must be on the **same subnet (Layer 2)**; for
      cross-subnet or cross-site, use a **load balancer** or **DNS** ingress
      instead.
    * Reserve a spare IP on the subnet for the **VIP** (the floating address users
      hit) — it must not be assigned to any host.
    * Open these ports **between nodes** (the appliance firewall opens them on
      cluster activation, but any upstream network ACLs must allow them too):

      | Purpose                          | Ports                   |
      | -------------------------------- | ----------------------- |
      | Database replication (Patroni)   | `5432`, `8008`          |
      | Cluster coordination (etcd)      | `2379`, `2380`          |
      | VIP failover (keepalived / VRRP) | protocol **112 (VRRP)** |
      | Shared Redis (P2, optional)      | `6379`, `26379`         |
      | Shared RabbitMQ (P2, optional)   | `5672`, `4369`, `25672` |
  </Accordion>

  <Accordion title="A recent backup">
    Take a fresh backup of the node that will become the **primary** before you
    start (**Admin → Backup & Restore**). Converting a standalone appliance to a
    primary preserves its data and takes an automatic safety snapshot first, but a
    known-good backup off the box is always the right precaution.
  </Accordion>
</AccordionGroup>

<Warning>
  **The joining nodes are wiped.** When a node joins, its local database is
  **replaced** by a full copy streamed from the primary. Only join **fresh
  appliances** (or nodes whose local data you are willing to discard). The
  **primary keeps** all its data.
</Warning>

## Choose an ingress mode

Ingress is the stable front-door address users always hit. Pick one up front —
you set it when you create the cluster.

| Mode               | Use when                            | Notes                                                                                                    |
| ------------------ | ----------------------------------- | -------------------------------------------------------------------------------------------------------- |
| **VIP** (built-in) | All nodes on one subnet (single DC) | Floating IP via keepalived/VRRP — HA with no external dependency. Recommended for a single-site cluster. |
| **Load balancer**  | You already run an L4/L7 LB         | Point the LB at all node IPs; it health-checks the primary.                                              |
| **DNS / GTM**      | Cross-site DR                       | A DNS/global-traffic-manager record you repoint on failover.                                             |
| **Standalone**     | Single node only                    | The node's own IP (no HA).                                                                               |

## Step-by-step: build the cluster

You can do everything from the **admin GUI** (recommended). The **console** is the
break-glass equivalent for an operator without browser access. The GUI stages
each action and runs it on the host; the console runs the same verbs directly.

<Steps>
  <Step title="On the FIRST node — create the cluster">
    This promotes your standalone appliance to the cluster **primary**. Its
    existing data is adopted in place (a safety backup is taken first).

    <Tabs>
      <Tab title="Admin GUI">
        1. Go to **Admin → Cluster / High Availability**.
        2. Click **Create cluster**.
        3. Choose the **ingress mode** and, for VIP, enter the reserved **VIP
           address** and the full list of node IPs.
        4. Confirm. The board turns green with this node shown as **Leader** and
           the status **Healthy — single node (no standby yet)**.
      </Tab>

      <Tab title="Console">
        At the `enclave>` prompt on the primary:

        ```text theme={null}
        cluster node-name enclave-01          # set a unique hostname (if not already)
        cluster init vip 192.168.0.240        # ingress mode + VIP; use 'lb' or 'dns' instead of vip as needed
        cluster status                        # verify: enclave-01 | Leader | running
        ```
      </Tab>
    </Tabs>
  </Step>

  <Step title="On the primary — mint a join token for the new node">
    A join token is a **single-use, time-limited sealed bundle** carrying the
    cluster's shared secrets. It is shown **once** — copy it now.

    <Tabs>
      <Tab title="Admin GUI">
        1. On the primary's Cluster page, click **Add node**.
        2. Copy the **sealed join bundle** and its **one-time passphrase**. Send
           the bundle and the passphrase to the new node over **separate**
           channels.
      </Tab>

      <Tab title="Console">
        ```text theme={null}
        cluster join-token
        ```

        Copy the printed **bundle** and the **passphrase** it shows separately.
      </Tab>
    </Tabs>

    <Warning>
      The token expires in **15 minutes** and works only once. Mint a fresh one if
      it lapses. The bundle never contains the cluster CA private key — a leaked
      bundle alone cannot mint a trusted member.
    </Warning>
  </Step>

  <Step title="On the NEW node — join the cluster">
    Do this on the fresh appliance you are adding. Make sure its hostname is unique
    first.

    <Tabs>
      <Tab title="Admin GUI">
        1. On the **new** node, go to **Admin → Cluster / High Availability**.
        2. Click **Join existing cluster**.
        3. Paste the **join bundle** and the **passphrase**, and enter the full
           list of node IPs.
        4. Submit. A live progress panel shows each phase: *Enrolling → Awaiting
           operator approval → Adopting → Registering → Starting → Streaming →
           Verifying*.
      </Tab>

      <Tab title="Console">
        ```text theme={null}
        cluster node-name enclave-02                         # unique hostname, BEFORE joining
        cluster join <sealed-bundle> 192.168.0.11,192.168.0.12,192.168.0.13
        # you'll be prompted for the one-time passphrase
        ```
      </Tab>
    </Tabs>
  </Step>

  <Step title="On the primary — approve the new node">
    The joining node stops at **Awaiting operator approval** and displays a **CSR
    fingerprint** on its own screen/console. This human check is what stops a rogue
    node from joining.

    <Tabs>
      <Tab title="Admin GUI">
        1. On the primary, a **Nodes awaiting approval** card appears with the
           node's **CSR fingerprint**.
        2. Verify that fingerprint **matches exactly** the one shown on the joining
           node, then click **Approve**.
      </Tab>

      <Tab title="Console">
        Approval is normally automatic (the single-use sealed token is the
        authorization). If you have enabled manual approval, approve from the
        primary's Cluster page after verifying the fingerprint.
      </Tab>
    </Tabs>

    <Note>
      By default, a valid one-time token **auto-approves** the node (the token is
      the authorization). To require an operator to verify the fingerprint by hand
      for every node, set `CLUSTER_ENROLL_AUTO_APPROVE=false` on the primary before
      minting tokens — the node then waits at *Awaiting approval* until you click
      **Approve**.
    </Note>
  </Step>

  <Step title="Verify the node is streaming">
    After approval the node adopts the primary's data and starts streaming.

    <Tabs>
      <Tab title="Admin GUI">
        The board now lists **two** rows: `enclave-01 — Leader — Up` and
        `enclave-02 — Sync Standby — Up — Lag 0`. Status shows **Healthy · Quorum
        2/2 OK**.
      </Tab>

      <Tab title="Console">
        ```text theme={null}
        cluster status
        # expect: enclave-01 | Leader | running
        #         enclave-02 | Sync Standby | streaming | Lag 0
        ```
      </Tab>
    </Tabs>
  </Step>

  <Step title="Repeat for each additional node">
    Mint a fresh token per node and repeat steps 2–5 for node 3 (and beyond). Aim
    for **3 nodes** for production HA.
  </Step>
</Steps>

<Warning>
  After a node finishes joining, it adopts the cluster-wide session keys, so your
  admin session **on that node** is rotated — sign in again (with your cluster
  admin credentials) to keep managing it. This is expected and only affects the
  node that just joined.
</Warning>

## Optional: enable the shared session/job tier (P2)

Phase 1 (above) makes your **database** highly available. **P2** additionally makes
**Redis** (sessions, one-time codes, rate limits) and **RabbitMQ** (in-flight
background jobs) cluster-wide, so a failover no longer logs users out or drops
running jobs.

<Steps>
  <Step title="Enable on the leader first, then every other node">
    <Tabs>
      <Tab title="Admin GUI">
        On the Cluster page, click **Enable P2** — first on the **Leader** node,
        then on each other node. It is idempotent; the peer list and primary are
        detected automatically.
      </Tab>

      <Tab title="Console">
        ```text theme={null}
        cluster p2-enable      # run on every node, leader first
        ```
      </Tab>
    </Tabs>
  </Step>

  <Step title="Verify">
    ```text theme={null}
    # RabbitMQ cluster formed (all nodes, one partition):
    rabbitmqctl cluster_status
    # Redis Sentinel sees the master + replicas:
    redis-cli -p 26379 sentinel master enclave-redis
    ```
  </Step>
</Steps>

<Note>
  P2 is **per-node** — a node you skip keeps its own private Redis/RabbitMQ and is
  not part of the shared tier. Run **Enable P2 on every node**. Redis and RabbitMQ
  failover then run independently of the database primary.
</Note>

## Test a failover

Once you have 3 healthy nodes, prove the cluster does its job:

1. Power off the **primary** node.
2. Within \~10 seconds a standby is **promoted** to primary and the VIP moves to
   it. Users on the ingress address stay connected (or reconnect immediately).
3. Confirm on any surviving node: `cluster status` shows a new **Leader** on a
   higher timeline, and the remaining standby re-pairs to it.
4. Power the old primary back on — it **rejoins automatically** as a standby and
   catches up.

## Day-2 operations

| Task                              | GUI                            | Console                                                    |
| --------------------------------- | ------------------------------ | ---------------------------------------------------------- |
| Check health / topology           | Cluster board (auto-refreshes) | `cluster status`                                           |
| Planned role change (maintenance) | Node ⋮ → **Make primary**      | `cluster switchover <node>`                                |
| Remove a node                     | Node ⋮ → **Remove**            | `cluster remove <node>`                                    |
| Return a node to standalone       | **Leave cluster**              | `cluster rollback`                                         |
| Promote the DR site               | **Promote DR**                 | `cluster promote-dr`                                       |
| Change ingress                    | **Ingress settings**           | `cluster ingress <mode> [vip]`                             |
| Inspect logs                      | —                              | `cluster logs patroni\|etcd\|haproxy\|redis\|rabbitmq [N]` |

<Note>
  `cluster rollback` is the safe way to take a node **out** of a cluster — it
  restores standalone operation and keeps the node serving. Use it before you
  re-cluster a node, or to wind a cluster down.
</Note>

## Removing and decommissioning a node

**Remove** takes a node out of the cluster permanently (hardware retirement,
replacement, or recovering from an unhealthy member). It is a three-part
operation, and the last part is yours:

<Steps>
  <Step title="Remove it from the cluster">
    On the Cluster board, node ⋮ → **Remove** (you cannot remove the current
    primary — switch over first). The cluster deregisters the node, revokes its
    membership certificate so it can never silently re-join, and **fences its
    database access**: replication and application connections from that node
    are rejected immediately, so no new data reaches it even while it is still
    powered on.
  </Step>

  <Step title="Power the node off">
    A removed node that is still running shows on the board as
    **"Fenced — power off this node to complete removal"** (it re-announces
    itself until it is shut down — this is expected). Power the machine off to
    complete the removal; the board then returns to the surviving members only.
  </Step>

  <Step title="Wipe or destroy the machine">
    Until its disk is wiped, the removed node **still holds a full replica of
    your data as of the moment it was fenced**. Treat it like any retired
    database server: securely wipe the virtual disk or destroy the VM as your
    data-handling policy requires.
  </Step>
</Steps>

<Note>
  Re-using the same machine later is fully supported: return it to standalone
  (`cluster rollback`), or reinstall it, and join it again with a fresh token —
  a legitimate re-join under the same name automatically lifts the fence.
</Note>

## Security model

* **Mutual TLS everywhere** — every inter-node channel (database replication, etcd,
  the Patroni control API) is authenticated with certificates from an internal
  cluster CA. An on-network stranger without a CA-signed certificate cannot read
  the coordination store or register as a member.
* **CA private key stays on the host** — it is never mounted into a container, so a
  compromised application container cannot mint cluster certificates. The key
  lives **only on the node where the cluster was created**; it is never copied
  to other members or into join bundles. You can run **Add node** and approvals
  from any node's page — the certificate signing itself is always performed by
  the cluster-creation node automatically. Protect that node's backups: if it
  is permanently lost, new nodes can no longer be enrolled into this cluster's
  PKI (existing members keep working).
* **Sealed, single-use join tokens** — the join bundle is encrypted and
  authenticated under a one-time passphrase (sent over a separate channel), carries
  an authenticated 15-minute expiry, and does **not** contain the CA key.
* **Post-quantum ready** — node enrollment is authorized with a **hybrid**
  (classical + post-quantum, Ed25519 + ML-DSA-65) attestation, and inter-node
  transport negotiates a hybrid key exchange where the components support it.
* **Every action is audited** — create/join/approve/reject/switchover and failovers
  are written to the tamper-evident audit log, and the audit hash-chain is
  **re-anchored on every failover** so integrity survives a primary change.

## Troubleshooting

<AccordionGroup>
  <Accordion title="Create cluster says the feature isn't licensed">
    The primary's license does not include the HA clustering capability. Add it
    under **Admin → Licensing** (Enterprise Advanced includes it, or request a
    per-deal grant), then retry.
  </Accordion>

  <Accordion title="Join is refused for an unsynchronized clock">
    Bring the node's clock into NTP sync with the rest of the cluster and retry.
    Do not use the lab override outside a disconnected test bench.
  </Accordion>

  <Accordion title="The join token expired / won't work twice">
    Tokens are single-use with a 15-minute lifetime. Mint a fresh one on the
    primary (**Add node** / `cluster join-token`) and join again.
  </Accordion>

  <Accordion title="Hostname already a member / duplicate name">
    Every node needs a unique hostname before joining. Set it with
    `cluster node-name enclave-02`, then re-run the join.
  </Accordion>

  <Accordion title="A join failed partway">
    A failed join safely rolls the node back to standalone and keeps it serving —
    just fix the underlying issue (clock, network, hostname) and retry. Detailed
    logs are on the node under `/var/log/enclave/cluster/`.
  </Accordion>

  <Accordion title="The joined node's cluster page shows an error / won't load">
    A node that just joined rotated its session keys. **Sign out and back in** on
    that node (with your cluster admin credentials).
  </Accordion>

  <Accordion title="Add node fails with 'cluster PKI missing' (older releases)">
    On releases before 0.11.6, minting a join token and signing the approval had
    to run on the **cluster-creation node** (the only holder of the cluster CA
    key) — after a failover, requests could land on another member and fail with
    this message. Workaround on those releases: open the cluster-creation node's
    own address directly, switch **Node-join approval** to manual, mint and
    approve from there. From 0.11.6 onward this works from any node: token
    minting no longer needs the CA key, and approvals are routed to the
    CA-holding node automatically.
  </Accordion>

  <Accordion title="A removed node still appears on the board">
    Expected until it is powered off — a removed-but-running node keeps
    announcing itself. Its database access is already fenced (no data flows to
    it); the board marks it **"Fenced — power off this node to complete
    removal"**. Shut the machine down and the entry disappears.
  </Accordion>
</AccordionGroup>

## Day-2: certificates, key recovery & revocation

The cluster runs its own internal certificate authority (CA) for the encrypted
channels between nodes. The CA private key lives only on the node you created the
cluster on. These day-2 operations keep that trust healthy over the appliance's life.

<CardGroup cols={2}>
  <Card title="Automatic certificate renewal" icon="rotate">
    Per-node certificates auto-renew well before expiry — the CA-holding node renews
    its own, and replicas renew through the CA holder automatically on a background
    schedule (no admin page needs to be open). No action needed unless a node warns it
    could not renew (then check that the CA-holding node is up).
  </Card>

  <Card title="CA-key escrow & recovery" icon="key">
    Escrow the CA key under a **K-of-N operator quorum** so it survives the loss of the
    CA-holding node — no more re-building the whole cluster. Set this up once, soon
    after creating the cluster, and rehearse the recovery drill.
  </Card>

  <Card title="Revoke a node's certificate" icon="ban">
    Decommissioned or compromised a node? Revoke its certificate so it is rejected by
    the cluster. Revocation is distributed to every node automatically; strict
    **enforcement** at the encrypted channels (etcd rejects a revoked node) is turned on
    per node with `cluster-crl-enforce on` (it validates the CRL first, then applies on the
    next stack restart) — see the certificate-expiry runbook.
  </Card>

  <Card title="Verified switchover & self-healing rolls" icon="shield-check">
    Planned leader switchover now confirms leadership actually moved before reporting
    success, and a stalled rolling upgrade self-heals (retrying every source) and shows
    each node's own reason on the board.
  </Card>
</CardGroup>

<Warning>
  Set up CA-key escrow **before** you need it. Without escrow, losing the
  cluster-creation node means rebuilding the cluster from scratch. The escrow bundle is
  stored off-box and is useless without a quorum of custodian shares.
</Warning>

### Set up CA-key escrow (once, soon after creating the cluster)

<Steps>
  <Step title="Choose custodians and a quorum">
    Pick **N** custodians and a threshold **K** (e.g. 3-of-5). Each custodian generates an
    `age` key pair on their own machine and shares only their **public** recipient
    (`age1…`) — the private key stays with them, off-box.
  </Step>

  <Step title="Create the escrow bundle on the CA-holding node">
    From the console on the cluster-creation node, run
    `cluster-ca-escrow -k K -n N -o <bundle> <recipient…>`. It writes an off-box bundle
    (the CA key encrypted under a random secret, plus one sealed share per custodian). The
    **plaintext CA key never leaves the node** and is never seen by the application.
  </Step>

  <Step title="Distribute and store">
    Move the bundle to secure off-box storage and give each custodian their own
    `share-<i>.age`. On the **Cluster** admin page, confirm the badge reads
    **CA-key escrow: Configured (K-of-N)**.
  </Step>

  <Step title="Rehearse the recovery drill">
    Prove it works before you depend on it: gather **K** decrypted shares and run
    `cluster-ca-recover --dry-run` — it reconstructs and verifies the key against the CA
    **without installing anything**.
  </Step>
</Steps>

### Recover the CA key onto a replacement node

<Steps>
  <Step title="Confirm the loss">
    The CA-holding node is permanently gone and no other node holds the key (new node
    certs, renewals and enrollments start failing). You have the off-box bundle and can
    reach at least **K** custodians.
  </Step>

  <Step title="Collect a quorum of shares">
    Each of ≥K custodians opens their share off-box (`age -d -i <identity> share-<i>.age`)
    and returns the decrypted value. Assemble them, one per line, into a shares file.
  </Step>

  <Step title="Rebuild the key on the new holder">
    On the replacement node, run `cluster-ca-recover -b <bundle> -s <shares>`. It verifies
    the shares against the bundle and installs the CA key **only** on a passing check. That
    node is now the CA holder; enrollments and renewals resume automatically.
  </Step>
</Steps>

<Note>
  The **Cluster** admin page shows CA-key escrow state at a glance — **Configured**,
  **Not configured** (the CA key is a single point of loss — set up escrow), or
  **Unknown**. Escrow setup and recovery are deliberately **console-only** ceremonies:
  the CA key never touches the web tier, and recovery works even when the cluster is down.
</Note>

### Remove a node and re-join it later

Removing and re-adding a node is a **console** procedure run on a healthy member — the
appliance cannot stop a remote node's stack, so a removed box is powered off (or rolled
back to standalone) and then re-joined with a fresh certificate.

<Steps>
  <Step title="Remove the node (on a healthy member)">
    `cluster-remove <node>` deregisters it from etcd + Patroni, revokes its enrollment and
    certificate, and fences its database access. This works whether the node is still live
    **or** already gone (revoked/evicted/powered off) — in the already-gone case it also
    cleans up the stale etcd membership for you, so there is never any manual step.
  </Step>

  <Step title="Return the removed box to standalone (on that box)">
    If the box is still running and you want to re-use it, roll it back with
    `cluster-rollback`. It becomes a standalone appliance on its own IP again.
  </Step>

  <Step title="Re-join it (Add node)">
    Mint a join bundle with `cluster-join-token` (or the **Cluster → Add node** button),
    run `cluster-join <bundle> <all-node-ips>` on the box, then **approve** its CSR
    fingerprint on the Cluster board. It receives a **fresh, non-revoked** certificate —
    accepted even with strict CRL enforcement on — and streams back into the cluster.
  </Step>
</Steps>

<Warning>
  **Revoking a node does not instantly kick it out.** Revocation (and the CRL) reject a
  node at the next connection attempt, so a revoked node can never *re-join* — but a node
  that is already connected keeps running until it is restarted or removed. To evict a live
  or compromised node, use `cluster-remove` (or power it off). And always **revoke first,
  then turn on `cluster-crl-enforce`** and restart each survivor — etcd reads the revocation
  list at start-up, so enabling enforcement before the revoke leaves a survivor accepting the
  old certificate until its next restart.
</Warning>

These are operator (console) procedures. See the operator runbooks for exact, copy-paste
step-by-step instructions: CA recovery, certificate expiry & revocation, node removal &
re-join, and rolling-upgrade recovery.

<Note>
  **Diagnostics.** `support-bundle` produces a secret-scrubbed tarball that now includes
  per-container application logs and full cluster PKI state (etcd membership, the CRL, and the
  certificate/enrollment ledger) — everything support needs to diagnose a cluster or app-tier
  fault off-box. For a deeper session, raise verbosity with `log-level debug` (recreates just
  the backend) and set it back to `info` afterwards.
</Note>