> ## Documentation Index
> Fetch the complete documentation index at: https://docs.kvelden.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Cluster in-place upgrade (GUI)

> Step-by-step guide to upgrading a Kvelden Enclave cluster to a new release from the admin GUI with zero downtime — one node at a time, standbys first and the primary last, with automatic bundle distribution, per-node readiness checks, and automatic rollback of any node that fails.

<Info>
  This page covers a **clustered** deployment (two or more appliances joined for
  high availability). The rolling upgrade takes each node through the new version
  **one at a time — standbys first, the primary last — so the service stays
  available throughout (zero user-visible downtime).** For a single appliance,
  use the [Standalone upgrade](/deployment/appliance/upgrade/standalone) instead.
</Info>

<Note>
  New to how this works? Read [System upgrade](/deployment/appliance/upgrade/overview)
  first — it explains the signed bundle, the A/B slots, and the automatic safety
  net that each node uses during its own step.
</Note>

## How the rolling upgrade works

You upload the new bundle **once**. The cluster then upgrades itself node by node:

<Steps>
  <Step title="Standbys first, primary last">
    Standby nodes upgrade before the primary. When it's the primary's turn, the
    cluster **automatically fails over** to an already-upgraded standby first, so
    a writable node is always serving.
  </Step>

  <Step title="One node at a time">
    Only one node is ever offline (rebooting into the new version) at a time. The
    remaining nodes keep a quorum and keep serving through the stable ingress
    address (VIP / load balancer / DNS).
  </Step>

  <Step title="Automatic distribution">
    Each node pulls the bundle from the node you uploaded to over the cluster
    network — you don't copy files to each appliance.
  </Step>

  <Step title="Per-node safety net">
    Each node applies the update with the same verified, self-healing engine as a
    standalone upgrade (verify → snapshot → install to the standby slot → reboot →
    health-gate → commit or auto-rollback). A node that fails **halts the rollout
    with the other nodes still serving**.
  </Step>
</Steps>

## Before you start

<AccordionGroup>
  <Accordion title="HA clustering license">
    The rolling upgrade requires the **HA clustering** capability (the same one
    used to form the cluster). Check **Admin → Licensing**.
  </Accordion>

  <Accordion title="Sign in as the owner, via the cluster address">
    Open the cluster's ingress address (the VIP, load balancer, or DNS name) and
    sign in as the **owner**. Manage the upgrade from **Admin → Cluster / High
    Availability**.
  </Accordion>

  <Accordion title="Cluster is healthy">
    On the Cluster board, confirm **Healthy**, **Quorum N/N OK**, every node
    **Up / streaming**, and **Lag 0**. Resolve any degraded node before
    upgrading — the preflight will refuse an unhealthy cluster.
  </Accordion>

  <Accordion title="Free disk space on every node">
    Each node needs working room for the pulled bundle plus the new images. The
    per-node readiness check verifies this and marks any short node red before
    you can start.
  </Accordion>

  <Accordion title="Clock synchronized on every node">
    All nodes must have a correct, synchronized clock (required for leader
    election, TLS, and signature validation). Confirm under **System →
    Diagnostics** on each node.
  </Accordion>

  <Accordion title="A backup key on every node">
    Each node takes its own safety snapshot during its step, which requires a
    backup key. Set one on every node under **Admin → Backup & Restore** if
    not already configured.
  </Accordion>

  <Accordion title="Same bundle, compatible version">
    Use one `enclave-update-<version>.tar.gz` for the whole cluster. Only an
    adjacent version step is allowed (the current version → the next); the
    preflight enforces this.
  </Accordion>
</AccordionGroup>

## Upgrade steps

<Steps>
  <Step title="Upload the bundle (once)">
    Go to **System → Software Update** and upload
    `enclave-update-<version>.tar.gz` with **Choose bundle…**. You only need to
    do this on the node you're signed in to — the cluster distributes it to the
    other nodes automatically.
  </Step>

  <Step title="Open the Rolling upgrade panel">
    Go to **Admin → Cluster / High Availability** and scroll to **Rolling
    upgrade**. Select the bundle you uploaded from the **Update bundle**
    drop-down.
  </Step>

  <Step title="Run Preflight">
    Select **Preflight**. The cluster checks it is safe to roll:

    * The bundle is **signed, valid, and a permitted version step**.
    * The cluster has **quorum**, every node is **running/streaming**, and
      **replication lag is 0**.

    A green confirmation (`cluster preflight OK … <from> → <to>`) means the
    cluster is ready.
  </Step>

  <Step title="Review the per-node readiness table">
    After preflight passes, a **per-node readiness** table appears with a row per
    node and **disk / version / health** indicators plus reachability. Every node
    must be **ready** (all green). A red row shows the reason — see
    [Troubleshooting](#troubleshooting). **Start** stays disabled until all nodes
    are ready.
  </Step>

  <Step title="Start the rolling upgrade">
    Select **Start rolling upgrade** and confirm your password (step-up). The
    rollout begins.
  </Step>

  <Step title="Watch it roll, node by node">
    Each node, standbys first, goes through: **distribute** (pull the bundle) →
    **install to the standby slot** → **reboot & activate** → **health-gate** →
    **rejoin** (streaming, Lag 0). The panel shows the current node and stage.

    When it's the **primary's** turn, the cluster **switches over** to an
    upgraded standby first (the ingress address follows), then upgrades the old
    primary. Throughout, the service stays reachable on the cluster address.
  </Step>

  <Step title="Verify the cluster is fully up-level">
    When the rollout completes, confirm on the Cluster board:

    * **All nodes show the new version.**
    * **Healthy**, **Quorum N/N OK**, all nodes **streaming**, **Lag 0**.
    * The primary may have changed (from the switchover) — that's expected.
    * Your data is intact via the cluster address and from each node.
  </Step>
</Steps>

<Check>
  Keep the cluster address open in a browser tab during the rollout — it should
  stay reachable the whole time. A single node being briefly offline while it
  reboots is normal and does not interrupt the service.
</Check>

## If a node fails

The rollout is **fail-safe**:

* The node that failed its health gate **automatically rolls back** to its
  previous version (OS + application together) and rejoins on the old version.
* The **rollout halts** at that point — every other node keeps serving, so you
  are never left mid-upgrade with the service down.
* The panel shows which node failed and why.

Resolve the cause on that node (usually disk space or a transient health issue),
then run **Preflight** and **Start rolling upgrade** again — nodes already on the
new version are skipped, and the rollout continues from where it stopped.

## Troubleshooting

<AccordionGroup>
  <Accordion title="Preflight failed — cluster not ready">
    The cluster isn't in a safe state to roll: a node is down, replication lag is
    non-zero, or quorum is not met. Open the Cluster board, bring the cluster
    back to **Healthy · Quorum N/N · Lag 0**, then re-run Preflight.
  </Accordion>

  <Accordion title="Preflight failed — bundle rejected">
    The bundle's signature/format didn't validate or it isn't a permitted version
    step (only current → next is allowed). Re-download the correct release bundle
    from Kvelden.
  </Accordion>

  <Accordion title="A node row shows disk red">
    That node is short on free space for the bundle + new images. Free space on
    it (remove old uploaded bundles under **Software Update**), then re-run
    Preflight.
  </Accordion>

  <Accordion title="A node row shows version red">
    That node's version is too far from the target — only an adjacent step is
    allowed. Make sure every node is on the same current version before rolling.
  </Accordion>

  <Accordion title="A node row shows unreachable">
    The initiating node couldn't reach that peer over the cluster network.
    Confirm the node is **Up** on the Cluster board and that the nodes can reach
    each other on the cluster network, then re-run Preflight.
  </Accordion>

  <Accordion title="The rollout halted partway">
    A node failed and rolled itself back; the rest kept serving. Review the
    reason in the panel, fix it on that node, and re-run **Preflight → Start** —
    already-upgraded nodes are skipped.
  </Accordion>

  <Accordion title="A node is stuck 'distributing' / 'waiting for the bundle'">
    Each node now shows its **own** reason on the board (pulling, waiting on a
    peer, or "self-healing"). Distribution self-heals — it retries every source
    every few seconds and fails fast on an unresponsive peer, so a single slow node
    no longer strands the roll. If a node stays stuck, it is usually a node-local
    issue (disk space, or a wedged stack): free space, or restart that node's stack
    (leadership fails over automatically), then the roll resumes. A distribution
    that genuinely cannot reach every node halts safely with an actionable message
    rather than hanging — nothing has been applied at that point.
  </Accordion>
</AccordionGroup>

<Info>
  The rollout **always fails safe**: it never auto-rolls-back a partially-upgraded
  cluster. A halt holds in a supported mixed state (some nodes new, some old) until
  you resume or abort — the old primary keeps serving throughout, so there is no
  downtime while you investigate.
</Info>

<Card title="Related" icon="layer-group" href="/deployment/appliance/clustering">
  See [High availability & clustering](/deployment/appliance/clustering) for
  how to form and operate the cluster itself.
</Card>