ZFS has a reputation for being complicated. It is not — it has a small number of decisions that are permanent, and a larger number that are trivially changeable. Knowing which is which is most of the skill.

The permanent decisions

Three choices cannot be undone without destroying and rebuilding the pool. Get these right.

vdev layout

A pool is a stripe across vdevs. Losing any vdev loses the entire pool. Redundancy lives inside each vdev, never between them.

Layout Usable (12 × 8 TB) Fault tolerance Random IOPS
6 × 2-way mirror 48 TB 1 per vdev ~6× single disk
2 × 6-disk RAIDZ2 64 TB 2 per vdev ~2× single disk
1 × 12-disk RAIDZ2 80 TB 2 total ~1× single disk
3 × 4-disk RAIDZ1 72 TB 1 per vdev ~3× single disk

The rule that matters: a vdev delivers roughly the random IOPS of one member disk. Ten disks in one RAIDZ2 vdev give you the random performance of a single drive. Ten disks in five mirrors give you five times that.

For virtual machines and databases, use mirrors. The capacity loss is real and the performance difference is not subtle. For backup targets, archives, and media, RAIDZ2 is correct — sequential throughput scales fine, and capacity is the point.

Avoid RAIDZ1 on drives above about 4 TB. Rebuild times on large drives run to many hours, and the whole vdev is one failure from destruction throughout. The risk is not theoretical: rebuild is when drives are stressed hardest, and a second failure during resilver is the classic way pools die.

ashift

ashift sets the sector size and is fixed at vdev creation. Set it to 12 (4 KiB) always:

Bash
zpool create -o ashift=12 tank mirror /dev/disk/by-id/ata-... /dev/disk/by-id/ata-...

Modern drives are 4K native but many report 512-byte sectors for compatibility. If ZFS believes them it sets ashift=9, and every write becomes a read-modify-write. The performance penalty is severe and the only fix is rebuilding the pool.

Always reference disks by /dev/disk/by-id/, never /dev/sda. Device names reorder across reboots; by-id paths do not.

recordsize per dataset

Not strictly permanent — changing it affects new writes only — but existing data keeps the old size until rewritten, so it is awkward to change later.

Workload recordsize Reasoning
General files 128K (default) Good all-round balance
PostgreSQL 16K Matches its 8K page with compression headroom
MySQL InnoDB 16K Matches the InnoDB page size
VM images (zvol) 16K–64K Matches typical guest I/O
Video, backups 1M Fewer, larger records; better compression

Mismatched recordsize causes write amplification. A database writing 8K pages into a 128K record forces ZFS to read the whole record, modify it, and write it back — sixteen times the I/O the workload asked for.

Bash
zfs create -o recordsize=16K -o logbias=throughput tank/postgres
zfs create -o recordsize=1M tank/backups

The settings you should change immediately

Compression: on, always

Bash
zfs set compression=lz4 tank

LZ4 is fast enough that it usually increases throughput — compressing 128K down to 60K means writing less than half the data, and LZ4 compresses faster than a disk can write. There is essentially no workload where leaving compression off is correct.

For archival data where CPU is spare, zstd-3 compresses substantially better at moderate cost:

Bash
zfs set compression=zstd-3 tank/archive
zfs get compressratio tank/archive

atime: off

Bash
zfs set atime=off tank

Access-time updates turn every read into a write. Almost nothing uses atime. Turn it off, or use relatime if some tool genuinely needs it.

ARC sizing

ZFS caches in ARC, and by default it will take up to half of RAM on Linux. On a dedicated storage server, give it more; on a hypervisor sharing RAM with VMs, cap it:

Bash
# /etc/modprobe.d/zfs.conf — 32 GiB max, 8 GiB min
options zfs zfs_arc_max=34359738368
options zfs zfs_arc_min=8589934592

Check the hit rate — below about 90% on a read-heavy workload means more RAM will help more than any other change:

Bash
arc_summary | head -40

SLOG and L2ARC: usually not what you need

These two get added constantly, and usually to no effect.

SLOG accelerates synchronous writes only. If your workload is asynchronous — file serving, most backup traffic — a SLOG does nothing whatsoever. Check before buying:

Bash
zpool iostat -r tank 5     # look at the sync write columns

NFS exports, databases, and iSCSI with write-through do use sync writes, and a SLOG helps them substantially. It must be a device with power-loss protection — an Optane or an enterprise NVMe with capacitors. A consumer SSD as SLOG is worse than none, because it will acknowledge a flush it has not completed and you will lose data on power loss while believing you cannot.

L2ARC extends the read cache to SSD. It consumes RAM to index itself — roughly 70 bytes per record — so a large L2ARC on a RAM-constrained system reduces effective cache. Add RAM first. Every time.

Snapshots and replication

This is where ZFS pays for itself.

Bash
# Snapshots are instant and cost nothing until data diverges
zfs snapshot tank/vms@$(date +%Y%m%d-%H%M)

# Automated with a retention policy
zfs-auto-snapshot --quiet --syslog --label=hourly --keep=24 //

Replication sends only changed blocks:

Bash
# First run — full send
zfs send -c tank/vms@base | ssh backup zfs recv -F backup/vms

# Subsequent — incremental, only the delta
zfs send -c -i tank/vms@base tank/vms@latest | ssh backup zfs recv backup/vms

-c sends already-compressed blocks without decompressing, which cuts both CPU and network substantially.

Snapshots are not backups. They live in the same pool, so a pool failure takes them with it. Replication to a separate system is the backup. Test restoring from it — a replication stream you have never received from is a hypothesis, not a recovery plan.

Scrubs and monitoring

Scrub monthly. It reads every block, verifies checksums, and repairs from redundancy:

Bash
zpool scrub tank
zpool status -v tank
Text
  pool: tank
 state: ONLINE
  scan: scrub repaired 0B in 04:22:17 with 0 errors on Sun Jul 12 04:22:18 2026
config:
        NAME                        STATE     READ WRITE CKSUM
        tank                        ONLINE       0     0     0
          mirror-0                  ONLINE       0     0     0
            ata-WDC_WD80EFBX-1      ONLINE       0     0     0
            ata-WDC_WD80EFBX-2      ONLINE       0     0     0

Non-zero CKSUM is the number to watch. It means ZFS detected and corrected corruption that a traditional RAID controller would have passed to your application silently. A drive accumulating checksum errors is failing, regardless of what SMART reports.

Alert on pool state, not just disk health:

Bash
# Simple check for cron / monitoring agent
zpool status -x | grep -q "all pools are healthy" || echo "ZFS POOL DEGRADED" | mail -s "ZFS alert $(hostname)" ops@example.com

Capacity planning

Keep pools under 80% full. Past that, ZFS switches from best-fit to first-fit allocation and fragmentation accelerates sharply — write performance can halve. Past 90% it degrades severely.

Because compression and snapshots make df misleading, check properly:

Bash
zfs list -o name,used,avail,refer,compressratio,usedbysnapshots

usedbysnapshots is the column that surprises people. A dataset with heavy churn and 90 days of snapshots can hold several times its live size in retained blocks. When a pool fills unexpectedly, this is almost always why.