# Proxmox VE Cluster Design That Survives Failure

> Quorum, storage, and networking decisions that are expensive to reverse

Canonical: https://everettkildare.com/blog/proxmox-ve-cluster-design-that-survives-failure
Author: Everett Kildare
Published: 2026-07-20
Updated: 2026-11-01

Proxmox VE has moved from homelab curiosity to a genuine vSphere alternative, and Broadcom's licensing changes accelerated that considerably. But a Proxmox cluster designed like a vSphere cluster will disappoint you in specific, predictable ways.

This covers the design decisions that are expensive to reverse: quorum, storage, and networking.

## Quorum: get this right or nothing else matters

Proxmox uses Corosync for cluster membership. Corosync needs a majority to operate, and when it loses quorum the cluster stops making changes — you cannot start VMs, migrate, or edit configuration. Nodes keep running whatever they were already running, which is the correct behaviour but surprises people the first time.

The arithmetic is unforgiving:

| Nodes | Votes needed | Failures tolerated |
|---|---|---|
| 2 | 2 | 0 |
| 3 | 2 | 1 |
| 4 | 3 | 1 |
| 5 | 3 | 2 |
| 6 | 4 | 2 |

A two-node cluster tolerates zero failures. A four-node cluster tolerates exactly as many as a three-node cluster. **Odd node counts are not a stylistic preference — even counts buy you nothing.**

If you must run two nodes, add a QDevice. It is a lightweight arbitrator that runs on any Debian-ish host — a NAS, a management VM elsewhere, a Raspberry Pi:

```bash
# On the external arbitrator
apt install corosync-qnetd

# On each cluster node
apt install corosync-qdevice

# From one cluster node
pvecm qdevice setup 10.20.0.9 -f
pvecm status
```

```text
Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate Qdevice
```

> [!CAUTION]
> Put Corosync on its own physical network, or at minimum its own VLAN with strict priority queuing. Corosync is latency-sensitive — Proxmox requires latencies under 5 ms between nodes (LAN performance), and it will fence nodes if a backup job saturates the link it shares. This is the single most common cause of "my cluster randomly reboots nodes".

Configure a second Corosync ring for redundancy. Two rings on separate physical paths, and the cluster survives a switch failure without a fencing event:

```ini
# /etc/pve/corosync.conf  — bump config_version on every edit
nodelist {
  node {
    name: pve01
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 10.20.0.11
    ring1_addr: 10.30.0.11
  }
}

totem {
  version: 2
  cluster_name: prod-cluster
  transport: knet
  link_mode: passive
}
```

## Storage: pick the model before you buy hardware

Proxmox supports many storage backends. In practice you are choosing between three architectures, and they have very different hardware implications.

### Local ZFS with replication

Each node has its own ZFS pool. Proxmox replicates VM disks between nodes on a schedule — as often as every minute.

This is the highest-performance and cheapest option. The trade-off is stated plainly in the name: replication is asynchronous, so failover loses everything written since the last replication run.

```bash
# Replicate VM 101 to pve02 every 5 minutes
pvesr create-local-job 101-0 pve02 --schedule "*/5" --rate 100

pvesr status
```

Right for: performance-sensitive workloads that tolerate minutes of data loss on node failure, and budgets that do not stretch to a shared storage fabric.

### Ceph

Proxmox ships Ceph integration, and it is genuinely good. Synchronous replication, no data loss on node failure, live migration without moving disks.

Ceph is also demanding, and undersized Ceph clusters perform terribly:

- **Three nodes is the documented minimum; I recommend five for production.** Three works and is widely deployed, but with `size=3, min_size=2` losing one node in a three-node cluster leaves zero redundancy during recovery. Five is my recommendation for headroom, not a Proxmox requirement.
- **10 GbE minimum, 25 GbE preferred**, on a dedicated network. Ceph replication traffic is roughly 2× your write traffic.
- **Enterprise NVMe with power-loss protection.** Consumer SSDs collapse under Ceph's O_DSYNC write pattern — I have measured 90% throughput drops on drives that benchmark beautifully otherwise.
- **Separate public and cluster networks** once you exceed about five nodes.

```bash
pveceph init --network 10.40.0.0/24 --cluster-network 10.50.0.0/24
pveceph mon create
pveceph osd create /dev/nvme1n1
pveceph pool create vm-storage --size 3 --min_size 2 --pg_autoscale_mode on
```

Sizing rule of thumb: usable capacity is raw ÷ 3, and you should not plan past 70% full. 12 × 3.84 TB NVMe across three nodes gives ~46 TB raw, ~15 TB usable, ~10 TB you should actually plan around.

### Shared SAN over iSCSI or Fibre Channel

If you already own a SAN, use it. Thick LVM over a shared LUN gives live migration without disk movement, and the array handles redundancy.

The caveat that catches vSphere migrants: **there is no VMFS equivalent.** Before Proxmox VE 9, LVM on a shared LUN had no snapshots at all. PVE 9 adds snapshots as a volume chain: set `snapshot-as-volume-chain` on the storage and new VM disks are created as qcow2 layered on separate LVM volumes, on any block storage including iSCSI and Fibre Channel SANs. It is still a technology preview, it only applies to disks created after the flag is set, and each snapshot volume is thick-provisioned, so efficient use depends on the array supporting thin provisioning and discard. If you rely on snapshot-before-patch workflows in production today, test it hard first — or use a storage array that presents per-VM volumes, or qcow2 on NFS.

| Backend | Snapshots | Live migration | HA failover data loss |
|---|---|---|---|
| ZFS local + replication | Yes | Yes (with replication) | Up to the replication interval |
| Ceph RBD | Yes | Yes | None |
| LVM (thick) on shared LUN | PVE 9+, volume chain (tech preview) | Yes | None |
| NFS with qcow2 | Yes | Yes | None |

## Networking

Use Open vSwitch or Linux bridges with VLAN awareness — not a bridge per VLAN. VLAN-aware bridges let you tag at the VM's NIC, which is how vSphere port groups behave and what your team expects.

```ini
# /etc/network/interfaces
auto bond0
iface bond0 inet manual
    bond-slaves ens3f0 ens3f1
    bond-mode 802.3ad
    bond-xmit-hash-policy layer3+4
    bond-miimon 100

auto vmbr0
iface vmbr0 inet static
    address 10.10.0.11/24
    gateway 10.10.0.1
    bridge-ports bond0
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094
```

`layer3+4` hashing matters. The default `layer2` hash pins each MAC to a single physical link, so a busy VM never exceeds one NIC's bandwidth regardless of how many you bond.

Plan for at least four logical networks, and physically separate the first two:

1. **Corosync** — its own NICs, low latency, redundant ring
2. **Storage** — Ceph or iSCSI, 10 GbE+, jumbo frames
3. **VM traffic** — VLAN-tagged, bonded
4. **Management** — the web UI and SSH

## High availability

HA in Proxmox is opt-in per VM. Nothing fails over unless you say so. Placement used to be set with HA groups; since Proxmox VE 9.0 groups are deprecated and migrated to HA node affinity rules, so `ha-manager groupadd` is the old way.

```bash
ha-manager add vm:101 --max_restart 2 --max_relocate 2
ha-manager rules add node-affinity prod-vm101 \
    --resources vm:101 --nodes "pve01:2,pve02:1,pve03:1"
ha-manager status
```

The node priorities set preference, not restriction: VM 101 prefers pve01, falls back to the others. Add `--strict 1` to the rule only if the VM must never run anywhere else — a strict rule stops the resource when none of its listed nodes is available.

> [!WARNING]
> HA requires working fencing, and Proxmox fences via watchdog self-fencing. A node that loses quorum resets itself after ~60 seconds. This is correct and necessary — but it means a Corosync network problem reboots production VMs. Which brings us back to the first section: give Corosync its own network.

## Backup

`vzdump` is built in and works. Proxmox Backup Server is the reason to care: deduplication, incremental backups, and verification.

```bash
# Backup jobs live in /etc/pve/jobs.cfg
vzdump 101 --storage pbs --mode snapshot --compress zstd --notes-template "{{guestname}}"
```

The `snapshot` mode with the QEMU guest agent installed gives filesystem-consistent backups. Without the agent you get crash-consistent backups — usually fine, occasionally not, and always worse than the alternative. Install the agent.

Set up verification jobs on PBS. An unverified backup is a hypothesis.

## A concrete starting point

For a mid-size production cluster — say 60 VMs, 400 GB RAM working set:

- **5 nodes**, dual-socket, 512 GB RAM each (comfortable N+1)
- **Ceph** with 4 × 3.84 TB enterprise NVMe per node, 25 GbE dedicated storage network
- **Corosync** on dedicated 1 GbE with a second ring over the management NICs
- **2 × 25 GbE LACP** for VM traffic, VLAN-aware bridge
- **Proxmox Backup Server** on separate hardware, ideally a separate site, with weekly verification

The temptation is to start with three nodes and grow. That works, and the migration from three to five is straightforward. What is not straightforward is retrofitting a dedicated Corosync network or replacing consumer SSDs across a live Ceph cluster. Spend the money on those two up front.

## Sources

- [Cluster Manager - Proxmox VE](https://pve.proxmox.com/wiki/Cluster_Manager) — quorum behaviour, QDevice setup, redundant Corosync links, `config_version`, sub-5 ms latency requirement.
- [High Availability - Proxmox VE](https://pve.proxmox.com/wiki/High_Availability) — watchdog self-fencing after 60 seconds, `max_restart`/`max_relocate`, HA groups deprecated and migrated to node affinity rules in 9.0, `ha-manager rules add node-affinity` syntax and `--strict`.
- [Storage - Proxmox VE](https://pve.proxmox.com/wiki/Storage) — snapshot and sharing capabilities per storage backend.
- [Storage: LVM](https://pve.proxmox.com/wiki/Storage:_LVM) — `snapshot-as-volume-chain`: qcow2 backing chains on LVM, iSCSI/FC SANs included, technology preview, new disks only, thin provisioning and discard for efficiency.
- [Storage: LVM Thin](https://pve.proxmox.com/wiki/Storage:_LVM_Thin) — LVM thin pools are local only and cannot be shared.
- [Storage Replication - Proxmox VE](https://pve.proxmox.com/wiki/Storage_Replication) — scheduled replication, one-minute minimum interval, `pvesr` syntax.
- [Deploy Hyper-Converged Ceph Cluster](https://pve.proxmox.com/wiki/Deploy_Hyper-Converged_Ceph_Cluster) — three-node minimum, 10/25 Gbps dedicated Ceph network, public and cluster networks.
- [Linux Ethernet Bonding Driver HOWTO](https://www.kernel.org/doc/html/latest/networking/bonding.html) — `layer2` as the default transmit hash and `layer3+4` behaviour.
- [Broadcom halves subscription price for VMware's flagship hybrid cloud suite](https://www.theregister.com/software/2023/12/12/broadcom-halves-sub-price-for-vmwares-flagship-hybrid-cloud/550264) — Broadcom ending VMware perpetual licences.