# ZFS for Production Storage > The three decisions you cannot undo, and the settings to change on day one Canonical: https://everettkildare.com/blog/zfs-for-production-storage Author: Everett Kildare Published: 2026-06-06 Updated: 2026-10-31 ZFS has a reputation for being complicated. It is not — it has a small number of decisions that are permanent, and a larger number that are trivially changeable. Knowing which is which is most of the skill. ## The permanent decisions Three choices cannot be undone without destroying and rebuilding the pool. Get these right. ### vdev layout A pool is a stripe across vdevs. **Losing any vdev loses the entire pool.** Redundancy lives inside each vdev, never between them. | Layout | Usable (12 × 8 TB) | Fault tolerance | Random IOPS | |---|---|---|---| | 6 × 2-way mirror | 48 TB | 1 per vdev | ~6× single disk | | 2 × 6-disk RAIDZ2 | 64 TB | 2 per vdev | ~2× single disk | | 1 × 12-disk RAIDZ2 | 80 TB | 2 total | ~1× single disk | | 3 × 4-disk RAIDZ1 | 72 TB | 1 per vdev | ~3× single disk | The rule that matters: **a vdev delivers roughly the random IOPS of one member disk.** Ten disks in one RAIDZ2 vdev give you the random performance of a single drive. Ten disks in five mirrors give you five times that. For virtual machines and databases, use mirrors. The capacity loss is real and the performance difference is not subtle. For backup targets, archives, and media, RAIDZ2 is correct — sequential throughput scales fine, and capacity is the point. > [!WARNING] > Avoid RAIDZ1 on drives above about 4 TB. Rebuild times on large drives run to many hours, and the whole vdev is one failure from destruction throughout. The risk is not theoretical: rebuild is when drives are stressed hardest, and a second failure during resilver is the classic way pools die. ### ashift `ashift` sets the sector size and is fixed at vdev creation. Set it to 12 (4 KiB) always: ```bash zpool create -o ashift=12 tank mirror /dev/disk/by-id/ata-... /dev/disk/by-id/ata-... ``` Modern drives are 4K native but many report 512-byte sectors for compatibility. If ZFS believes them it sets `ashift=9`, and every write becomes a read-modify-write. The performance penalty is severe and the only fix is rebuilding the pool. Always reference disks by `/dev/disk/by-id/`, never `/dev/sda`. Device names reorder across reboots; by-id paths do not. ### recordsize per dataset Not strictly permanent — changing it affects new writes only — but existing data keeps the old size until rewritten, so it is awkward to change later. | Workload | recordsize | Reasoning | |---|---|---| | General files | 128K (default) | Good all-round balance | | PostgreSQL | 16K–32K | OpenZFS suggests 32K with LZ4; 16K trades some compression for less read-modify-write | | MySQL InnoDB | 16K | Matches the InnoDB page size | | Video, backups | 1M | Fewer, larger records; better compression | Mismatched recordsize causes write amplification. A database writing 8K pages into a 128K record forces ZFS to read the whole record, modify it, and write it back — sixteen times the I/O the workload asked for. ```bash zfs create -o recordsize=16K -o logbias=throughput tank/postgres zfs create -o recordsize=1M tank/backups ``` VM images on zvols are the exception: a zvol has no `recordsize`. It uses `volblocksize`, which is fixed once the volume has been written — set it at creation (`zfs create -V 100G -o volblocksize=16K tank/vm-101`). The default has been 16K since OpenZFS 2.2; smaller suits random guest I/O at the cost of more metadata, larger suits sequential work and compresses better. ## The settings you should change immediately ### Compression: on, always ```bash zfs set compression=lz4 tank ``` LZ4 is fast enough that it usually *increases* throughput — compressing 128K down to 60K means writing less than half the data, and LZ4 compresses faster than a disk can write. There is essentially no workload where leaving compression off is correct. For archival data where CPU is spare, `zstd-3` compresses substantially better at moderate cost: ```bash zfs set compression=zstd-3 tank/archive zfs get compressratio tank/archive ``` ### atime: off ```bash zfs set atime=off tank ``` Access-time updates turn every read into a write. Almost nothing uses atime. Turn it off, or use `relatime` if some tool genuinely needs it. ### ARC sizing ZFS caches in ARC, and by default it will take up to half of RAM on Linux (OpenZFS 2.2 and earlier) and most of it from 2.3 onwards. On 2.2 and earlier, a dedicated storage server should raise the limit itself; on a hypervisor sharing RAM with VMs, cap it: ```bash # /etc/modprobe.d/zfs.conf — 32 GiB max, 8 GiB min options zfs zfs_arc_max=34359738368 options zfs zfs_arc_min=8589934592 ``` Check the hit rate — below about 90% on a read-heavy workload means more RAM will help more than any other change: ```bash arc_summary | head -40 ``` ## SLOG and L2ARC: usually not what you need These two get added constantly, and usually to no effect. **SLOG** accelerates *synchronous* writes only. If your workload is asynchronous — file serving, most backup traffic — a SLOG does nothing whatsoever. Check before buying: ```bash zpool iostat -r tank 5 # look at the sync write columns ``` NFS exports, databases, and iSCSI with write-through do use sync writes, and a SLOG helps them substantially. It **must** be a device with power-loss protection — an Optane or an enterprise NVMe with capacitors. A consumer SSD as SLOG is worse than none, because it will acknowledge a flush it has not completed and you will lose data on power loss while believing you cannot. **L2ARC** extends the read cache to SSD. It consumes RAM to index itself — roughly 70 bytes per record — so a large L2ARC on a RAM-constrained system *reduces* effective cache. Add RAM first. Every time. ## Snapshots and replication This is where ZFS pays for itself. ```bash # Snapshots are instant and cost nothing until data diverges zfs snapshot tank/vms@$(date +%Y%m%d-%H%M) # Automated with a retention policy zfs-auto-snapshot --quiet --syslog --label=hourly --keep=24 // ``` Replication sends only changed blocks: ```bash # First run — full send zfs send -c tank/vms@base | ssh backup zfs recv -F backup/vms # Subsequent — incremental, only the delta zfs send -c -i tank/vms@base tank/vms@latest | ssh backup zfs recv backup/vms ``` `-c` sends already-compressed blocks without decompressing, which cuts both CPU and network substantially. > [!IMPORTANT] > Snapshots are not backups. They live in the same pool, so a pool failure takes them with it. Replication to a separate system is the backup. Test restoring from it — a replication stream you have never received from is a hypothesis, not a recovery plan. ## Scrubs and monitoring Scrub monthly. It reads every block, verifies checksums, and repairs from redundancy: ```bash zpool scrub tank zpool status -v tank ``` ```text pool: tank state: ONLINE scan: scrub repaired 0B in 04:22:17 with 0 errors on Sun Jul 12 04:22:18 2026 config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ata-WDC_WD80EFBX-1 ONLINE 0 0 0 ata-WDC_WD80EFBX-2 ONLINE 0 0 0 ``` Non-zero CKSUM is the number to watch. It means ZFS detected and corrected corruption that a traditional RAID controller would have passed to your application silently. A drive accumulating checksum errors is failing, regardless of what SMART reports. Alert on pool state, not just disk health: ```bash # Simple check for cron / monitoring agent zpool status -x | grep -q "all pools are healthy" || echo "ZFS POOL DEGRADED" | mail -s "ZFS alert $(hostname)" ops@example.com ``` ## Capacity planning Keep pools under 80% full. Past that, more and more metaslabs fall below the 4% free-space threshold at which ZFS switches from first-fit to best-fit allocation, and fragmentation accelerates sharply — write performance can halve. Past 90% it degrades severely. Because compression and snapshots make `df` misleading, check properly: ```bash zfs list -o name,used,avail,refer,compressratio,usedbysnapshots ``` `usedbysnapshots` is the column that surprises people. A dataset with heavy churn and 90 days of snapshots can hold several times its live size in retained blocks. When a pool fills unexpectedly, this is almost always why. ## Sources - [Workload Tuning — OpenZFS documentation](https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Workload%20Tuning.html) — mirror versus RAIDZ IOPS, ashift, recordsize for databases (32K for PostgreSQL with LZ4), zvol `volblocksize` (16K default since 2.2), LZ4, atime, `logbias`, free-space and allocation thresholds. - [zfsprops.7 — OpenZFS documentation](https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html) — `volblocksize` defaults to 16 KiB and cannot be changed once the volume has been written. - [zfs.4 — OpenZFS documentation](https://openzfs.github.io/openzfs-docs/man/v2.3/4/zfs.4.html) — `zfs_arc_max` default from OpenZFS 2.3 and `metaslab_df_free_pct` (4%). - [zfs.4 — OpenZFS documentation](https://openzfs.github.io/openzfs-docs/man/v2.2/4/zfs.4.html) — the 2.2 manual: the earlier half-of-RAM ARC default on Linux. - [Hardware — OpenZFS documentation](https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Hardware.html) — drives misreporting 512-byte sectors, correct ashift, flushed-data protection and SLOG device choice. - [Compensating for RAM Constraints with L2ARC on ZFS](https://klarasystems.com/articles/compensating-for-ram-constraints-with-l2arc-on-arc/) — roughly 70–80 bytes of ARC header per L2ARC block.