Proxmox VE Cluster Design That Survives Failure

Proxmox VE has moved from homelab curiosity to a genuine vSphere alternative, and Broadcom's licensing changes accelerated that considerably. But a Proxmox cluster designed like a vSphere cluster will disappoint you in specific, predictable ways.

This covers the design decisions that are expensive to reverse: quorum, storage, and networking.

Quorum: get this right or nothing else matters#

Proxmox uses Corosync for cluster membership. Corosync needs a majority to operate, and when it loses quorum the cluster stops making changes — you cannot start VMs, migrate, or edit configuration. Nodes keep running whatever they were already running, which is the correct behaviour but surprises people the first time.

The arithmetic is unforgiving:

A two-node cluster tolerates zero failures. A four-node cluster tolerates exactly as many as a three-node cluster. Odd node counts are not a stylistic preference — even counts buy you nothing.

If you must run two nodes, add a QDevice. It is a lightweight arbitrator that runs on any Debian-ish host — a NAS, a management VM elsewhere, a Raspberry Pi:

# On the external arbitrator
apt install corosync-qnetd

# On each cluster node
apt install corosync-qdevice

# From one cluster node
pvecm qdevice setup 10.20.0.9 -f
pvecm status

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate Qdevice

Put Corosync on its own physical network, or at minimum its own VLAN with strict priority queuing. Corosync is latency-sensitive — it wants sub-millisecond round trips and it will fence nodes if a backup job saturates the link it shares. This is the single most common cause of "my cluster randomly reboots nodes".

Configure a second Corosync ring for redundancy. Two rings on separate physical paths, and the cluster survives a switch failure without a fencing event:

# /etc/pve/corosync.conf  — bump config_version on every edit
nodelist {
  node {
    name: pve01
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 10.20.0.11
    ring1_addr: 10.30.0.11
  }
}

totem {
  version: 2
  cluster_name: prod-cluster
  transport: knet
  link_mode: passive
}

Storage: pick the model before you buy hardware#

Proxmox supports many storage backends. In practice you are choosing between three architectures, and they have very different hardware implications.

Local ZFS with replication#

Each node has its own ZFS pool. Proxmox replicates VM disks between nodes on a schedule — as often as every minute.

This is the highest-performance and cheapest option. The trade-off is stated plainly in the name: replication is asynchronous, so failover loses everything written since the last replication run.

# Replicate VM 101 to pve02 every 5 minutes
pvesr create-local-job 101-0 pve02 --schedule "*/5" --rate 100

pvesr status

Right for: performance-sensitive workloads that tolerate minutes of data loss on node failure, and budgets that do not stretch to a shared storage fabric.

Ceph#

Proxmox ships Ceph integration, and it is genuinely good. Synchronous replication, no data loss on node failure, live migration without moving disks.

Ceph is also demanding, and undersized Ceph clusters perform terribly:

• Five nodes minimum for production. Three works and is widely deployed, but with size=3, min_size=2 losing one node in a three-node cluster leaves zero redundancy during recovery.
• 10 GbE minimum, 25 GbE preferred, on a dedicated network. Ceph replication traffic is roughly 2× your write traffic.
• Enterprise NVMe with power-loss protection. Consumer SSDs collapse under Ceph's O_DSYNC write pattern — I have measured 90% throughput drops on drives that benchmark beautifully otherwise.
• Separate public and cluster networks once you exceed about five nodes.

pveceph init --network 10.40.0.0/24 --cluster-network 10.50.0.0/24
pveceph mon create
pveceph osd create /dev/nvme1n1
pveceph pool create vm-storage --size 3 --min_size 2 --pg_autoscale_mode on

Sizing rule of thumb: usable capacity is raw ÷ 3, and you should not plan past 70% full. 12 × 3.84 TB NVMe across three nodes gives ~46 TB raw, ~15 TB usable, ~10 TB you should actually plan around.

Shared SAN over iSCSI or Fibre Channel#

If you already own a SAN, use it. LVM-thin over a shared LUN gives live migration without disk movement, and the array handles redundancy.

The caveat that catches vSphere migrants: there is no VMFS equivalent. LVM on a shared LUN does not support snapshots on raw volumes. If you rely on snapshot-before-patch workflows, either use a storage array that presents per-VM volumes, or accept qcow2 on NFS instead.

Networking#

Use Open vSwitch or Linux bridges with VLAN awareness — not a bridge per VLAN. VLAN-aware bridges let you tag at the VM's NIC, which is how vSphere port groups behave and what your team expects.

# /etc/network/interfaces
auto bond0
iface bond0 inet manual
    bond-slaves ens3f0 ens3f1
    bond-mode 802.3ad
    bond-xmit-hash-policy layer3+4
    bond-miimon 100

auto vmbr0
iface vmbr0 inet static
    address 10.10.0.11/24
    gateway 10.10.0.1
    bridge-ports bond0
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094

layer3+4 hashing matters. The default layer2 hash pins each MAC to a single physical link, so a busy VM never exceeds one NIC's bandwidth regardless of how many you bond.

Plan for at least four logical networks, and physically separate the first two:

1. Corosync — its own NICs, low latency, redundant ring
2. Storage — Ceph or iSCSI, 10 GbE+, jumbo frames
3. VM traffic — VLAN-tagged, bonded
4. Management — the web UI and SSH

High availability#

HA in Proxmox is opt-in per VM through HA groups. Nothing fails over unless you say so.

ha-manager groupadd prod --nodes "pve01:2,pve02:1,pve03:1"
ha-manager add vm:101 --group prod --max_restart 2 --max_relocate 2
ha-manager status

The node priorities set preference, not restriction: VM 101 prefers pve01, falls back to the others.

HA requires working fencing, and Proxmox fences via watchdog self-fencing. A node that loses quorum resets itself after ~60 seconds. This is correct and necessary — but it means a Corosync network problem reboots production VMs. Which brings us back to the first section: give Corosync its own network.

Backup#

vzdump is built in and works. Proxmox Backup Server is the reason to care: deduplication, incremental backups, and verification.

# Backup jobs live in /etc/pve/jobs.cfg
vzdump 101 --storage pbs --mode snapshot --compress zstd --notes-template "{{guestname}}"

The snapshot mode with the QEMU guest agent installed gives filesystem-consistent backups. Without the agent you get crash-consistent backups — usually fine, occasionally not, and always worse than the alternative. Install the agent.

Set up verification jobs on PBS. An unverified backup is a hypothesis.

A concrete starting point#

For a mid-size production cluster — say 60 VMs, 400 GB RAM working set:

• 5 nodes, dual-socket, 512 GB RAM each (comfortable N+1)
• Ceph with 4 × 3.84 TB enterprise NVMe per node, 25 GbE dedicated storage network
• Corosync on dedicated 1 GbE with a second ring over the management NICs
• 2 × 25 GbE LACP for VM traffic, VLAN-aware bridge
• Proxmox Backup Server on separate hardware, ideally a separate site, with weekly verification

The temptation is to start with three nodes and grow. That works, and the migration from three to five is straightforward. What is not straightforward is retrofitting a dedicated Corosync network or replacing consumer SSDs across a live Ceph cluster. Spend the money on those two up front.