diff --git a/truenas.md b/truenas.md index 58a8f3f..05a5c1c 100644 --- a/truenas.md +++ b/truenas.md @@ -3,7 +3,7 @@ ## Access - **Hostname**: nas.dohertylan.com -- **Version**: TrueNAS 25.04.2.6 (Electric Eel / Linux-based) +- **Version**: TrueNAS `25.10.16.12.33-production+truenas` (per `/etc/version`, checked 2026-09-17; this doc previously said 25.04.2.6 — older 25.04.x boot environments are still present on `boot-pool`) - **SSH**: `ssh truenas_admin@10.100.0.25` (passwordless) - **Web UI**: https://10.100.0.25 (ports 80/443 listening) - **Domain**: dohertylan.com @@ -26,13 +26,84 @@ ## ZFS Pools -| Pool | Size | Used | Free | Health | -|------|------|------|------|--------| -| mypool | 175T | 83.5T | 91.2T | **DEGRADED** | -| SSD_Pool | 2.91T | 3.01G | 2.90T | ONLINE | -| boot-pool | 15G | 11.6G | 3.40G | ONLINE | +| Pool | Size | Alloc | Free | Cap | Frag | Health | +|------|------|-------|------|-----|------|--------| +| mypool | 175T | 96.4T | 78.3T | 55% | 2% | ONLINE | +| SSD_Pool | 2.91T | 3.01G | 2.90T | — | — | ONLINE | +| boot-pool | 15G | 11.6G | 3.40G | — | — | ONLINE | -**Note**: `mypool` is in DEGRADED state. +> `mypool` figures verified 2026-09-17. SSD_Pool / boot-pool rows are carried over from an earlier pass and were not +> re-checked. + +**`mypool` is ONLINE, not DEGRADED** (the earlier note in this doc was stale). `errors: No known data errors`, DEDUP +1.00x. `zpool status` does still print a latched `status:` banner — *"One or more devices has experienced an +unrecoverable error... Applications are unaffected"* — which is a historical flag, not a current fault; clear it with +`zpool clear mypool` once you are satisfied the drive is healthy. + +- **Layout**: 6 x raidz1 vdevs, each 4 x 7.28T +- **Hot spare**: `scsi-35000c500d2a29937`, AVAIL +- **Last resilver**: 705G in 01:24:15 with 0 errors, Wed 2026-09-16 01:07:30 + +## Memory / ARC + +VM is configured with **128 GB** (`MemTotal` 125.8 GiB). `zfs_arc_max` is **pinned by a TrueNAS tunable**, not derived +from RAM — this is the single most important fact on this page if the VM is ever resized. + +| Item | Value | +|---|---| +| `zfs_arc_max` (tunable id 8) | `103079215104` = 96 GiB | +| `zfs_dirty_data_max` (tunable id 15) | `17179869184` = 16 GiB | +| ARC size / target | 95.6 GiB / 96 GiB — saturated | +| Dedup | **none** (`no DDT entries`, DEDUP 1.00x) | +| L2ARC / special vdev | **none** (`l2_hdr_size` 0) | +| Apps / VMs on the NAS | **none** (0 / 0) | +| Sum of all process RSS | 1.78 GiB | + +### Sizing analysis (2026-09-17, 1d 10h uptime) + +The ARC is roughly 4x larger than the working set actually needs: + +| ARC region | Size | Hits served | +|---|---:|---:| +| MFU (frequently re-read) | 20.7 GiB | 66.5M — **94%** | +| MRU (touched once) | 74.7 GiB | 4.3M — 6% | +| Metadata | 1.3 GiB | 99.92% hit rate | + +Overall hit rate 98.90% (70,488,374 hits / 781,848 misses); demand data 96.72%. + +**The decisive metric is the ghost lists** — blocks evicted from ARC and then requested again, i.e. the direct measure +of "would a larger ARC help?": + +``` +mru_ghost_hits 4 +mfu_ghost_hits 0 +``` + +Four, against 70 million hits. Nothing being evicted is being re-read, so the cache is far past the point of +diminishing returns. The real working set is **~22 GiB** (MFU + metadata); the other ~75 GiB is streaming media and +Veeam backup data read once and never again. + +**Conclusion: 64 GB is safe** — but only if `zfs_arc_max` comes down with it. Left at 96 GiB on a 64 GiB VM, ZFS +targets an ARC larger than the whole machine: it grows until the kernel shrinker forces it back, giving continuous +reclaim pressure, erratic latency and real OOM risk for `smbd` / `nfsd` / `middlewared` under a write burst. + +Suggested values for a 64 GB config (leaves ~16 GiB for OS, services and headroom): + +| Tunable | Current | 64 GB target | +|---|---:|---:| +| `zfs_arc_max` | 96 GiB | **40 GiB** (`42949672960`) | +| `zfs_dirty_data_max` | 16 GiB | **8 GiB** (`8589934592`) | + +Apply with `midclt call tunable.update '{"value": ""}'`. Both parameters are runtime-writable +(`/sys/module/zfs/parameters/`), so no reboot is needed to change them — but the VM must be powered off to change its +RAM, which takes NFS and SMB down for Veeam, Plex and the docker host. + +**Caveat on the sample:** stats were taken over a 1d 10h window that included a Veeam *incremental* (2026-09-17 03:01) +but not a weekly *Full* (last 2026-09-12). Fulls are write-heavy on the NAS side and writes go through the dirty-data +path rather than ARC, so the conclusion should hold — re-check `mfu_ghost_hits` after a Full if you want certainty. + +**Validation trick:** lower `zfs_arc_max` to 40 GiB *while still on 128 GB RAM* and watch the ghost counters. If they +stay near zero, 64 GB is proven safe before the VM is ever touched. ### Key Datasets