diff --git a/truenas.md b/truenas.md index 05a5c1c..c592363 100644 --- a/truenas.md +++ b/truenas.md @@ -35,10 +35,13 @@ > `mypool` figures verified 2026-09-17. SSD_Pool / boot-pool rows are carried over from an earlier pass and were not > re-checked. -**`mypool` is ONLINE, not DEGRADED** (the earlier note in this doc was stale). `errors: No known data errors`, DEDUP -1.00x. `zpool status` does still print a latched `status:` banner — *"One or more devices has experienced an -unrecoverable error... Applications are unaffected"* — which is a historical flag, not a current fault; clear it with -`zpool clear mypool` once you are satisfied the drive is healthy. +**`mypool` is ONLINE and healthy**, not DEGRADED — the earlier note in this doc was stale. `errors: No known data +errors`, DEDUP 1.00x. + +> Until the 2026-09-17 reboot, `zpool status` printed a latched banner — *"One or more devices has experienced an +> unrecoverable error... Applications are unaffected"* — and `pool.query` reported `healthy: False`. That was a +> historical flag from the 2026-09-16 resilver, not a live fault; it cleared on reimport and the pool now reports +> `healthy: True`. If it ever reappears without a fresh resilver, treat it as real. - **Layout**: 6 x raidz1 vdevs, each 4 x 7.28T - **Hot spare**: `scsi-35000c500d2a29937`, AVAIL @@ -46,14 +49,16 @@ unrecoverable error... Applications are unaffected"* — which is a historical f ## Memory / ARC -VM is configured with **128 GB** (`MemTotal` 125.8 GiB). `zfs_arc_max` is **pinned by a TrueNAS tunable**, not derived -from RAM — this is the single most important fact on this page if the VM is ever resized. +> **Reduced 128 GB → 64 GB on 2026-09-17.** The analysis and the resize procedure are both below. + +VM is configured with **64 GB** (`MemTotal` 62.8 GiB). `zfs_arc_max` is **pinned by a TrueNAS tunable**, not derived +from RAM — this is the single most important fact on this page if the VM is ever resized again. | Item | Value | |---|---| -| `zfs_arc_max` (tunable id 8) | `103079215104` = 96 GiB | -| `zfs_dirty_data_max` (tunable id 15) | `17179869184` = 16 GiB | -| ARC size / target | 95.6 GiB / 96 GiB — saturated | +| `zfs_arc_max` (tunable id 8) | `42949672960` = 40 GiB | +| `zfs_dirty_data_max` (tunable id 15) | `8589934592` = 8 GiB | +| ARC size / target | warming after the 2026-09-17 reboot; cap 40 GiB | | Dedup | **none** (`no DDT entries`, DEDUP 1.00x) | | L2ARC / special vdev | **none** (`l2_hdr_size` 0) | | Apps / VMs on the NAS | **none** (0 / 0) | @@ -87,23 +92,41 @@ Veeam backup data read once and never again. targets an ARC larger than the whole machine: it grows until the kernel shrinker forces it back, giving continuous reclaim pressure, erratic latency and real OOM risk for `smbd` / `nfsd` / `middlewared` under a write burst. -Suggested values for a 64 GB config (leaves ~16 GiB for OS, services and headroom): +### Resize executed 2026-09-17 -| Tunable | Current | 64 GB target | +Values applied (both persisted as tunables, both runtime-writable so no reboot was needed *for the tunables*): + +| Tunable | Was | Now | |---|---:|---:| -| `zfs_arc_max` | 96 GiB | **40 GiB** (`42949672960`) | -| `zfs_dirty_data_max` | 16 GiB | **8 GiB** (`8589934592`) | +| `zfs_arc_max` (id 8) | 96 GiB | **40 GiB** (`42949672960`) | +| `zfs_dirty_data_max` (id 15) | 16 GiB | **8 GiB** (`8589934592`) | -Apply with `midclt call tunable.update '{"value": ""}'`. Both parameters are runtime-writable -(`/sys/module/zfs/parameters/`), so no reboot is needed to change them — but the VM must be powered off to change its -RAM, which takes NFS and SMB down for Veeam, Plex and the docker host. +**The VM has PCI passthrough** (`pcipassthrough-13000`, the HBA), which forces +`memoryReservationLockedToMax: true` — so all of its RAM is *reserved* on the ESXi host whether ARC uses it or not. +That is why this change moved the host from **83.9% → 58.8%** memory usage. The reservation followed the resize +automatically; it did not need setting by hand. -**Caveat on the sample:** stats were taken over a 1d 10h window that included a Veeam *incremental* (2026-09-17 03:01) -but not a weekly *Full* (last 2026-09-12). Fulls are write-heavy on the NAS side and writes go through the dirty-data -path rather than ARC, so the conclusion should hold — re-check `mfu_ghost_hits` after a Full if you want certainty. +Procedure that worked: -**Validation trick:** lower `zfs_arc_max` to 40 GiB *while still on 128 GB RAM* and watch the ghost counters. If they -stay near zero, 64 GB is proven safe before the VM is ever touched. +```bash +# pre-flight: no scrub/resilver running, no middleware jobs, no SMB *write* handles, no Plex sessions +govc vm.power -s TrueNAS # graceful, via VMware Tools; took ~10s +govc vm.change -vm TrueNAS -m 65536 +govc vm.power -on TrueNAS # ~3 min to SSH (HBA init + pool import) +``` + +**Client impact was nil.** Both docker-host NFS mounts (`/mnt/share`, `/mnt/other`) are `hard`, so containers stall in +uninterruptible sleep and resume when the server returns — `plex`, `sonarr` and `radarr` never restarted, and the only +D-state process during the outage was the `10.50.0.25-manager` NFS kernel thread. Check for SMB **write** handles rather +than sessions before scheduling: `midclt call smb.status LOCKS` — idle Explorer windows show as directory handles +(`filename: "."`, `WRITE_DATA: false`) and are safe to drop. + +> **Side effect:** the latched `zpool status` error banner cleared on reboot — `mypool` now reports `healthy: True`. +> The `zpool clear` suggested above is no longer needed. + +**Still to confirm:** ARC was cold after the reboot, so the post-change ghost-hit measurement has not been retaken. The +real test is the next Veeam **Full** — re-check `mfu_ghost_hits` after one. Note it had already moved 0 → 129 when ARC +was first capped at 40 GiB on the 128 GB VM, which is still negligible (0.0065% of hits) but no longer structurally zero. ### Key Datasets