dohertj2/network
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Reduce TrueNAS VM from 128 GB to 64 GB

Browse Source 
Executes the sizing analysis committed in db35cff. The VM has HBA PCI
passthrough, so memoryReservationLockedToMax is true and all of its RAM
is reserved on the host whether ARC uses it or not - this moved the ESXi
host from 83.9% to 58.8% memory usage. The reservation followed the
resize automatically.

  govc vm.power -s TrueNAS        # graceful via Tools, ~10s
  govc vm.change -vm TrueNAS -m 65536
  govc vm.power -on TrueNAS       # ~3 min to SSH (HBA init + pool import)

Tunables now zfs_arc_max 40 GiB and zfs_dirty_data_max 8 GiB, both
persisted and verified live. The second one matters more at 64 GB: left
at 16 GiB it would have been 25% of RAM and stacked with ARC during the
nightly Veeam window.

Client impact was nil. Both docker-host NFS mounts are hard, so plex,
sonarr and radarr stalled in uninterruptible sleep and resumed without
restarting; the only D-state process during the outage was the NFS
kernel thread. Pre-flight checked SMB write handles rather than sessions
- the two live sessions were idle Explorer directory handles with
WRITE_DATA false, safe to drop. Post-boot: pool imported, all three NFS
exports restored including the Veeam repo, SMB and NFS running, Plex
serving all three libraries.

Two things resolved themselves: mypool's latched "unrecoverable error"
banner cleared on reimport and it now reports healthy true, so the
zpool clear advice is dropped.

Records the procedure and the pre-flight checks, since the pinned-tunable
trap applies to any future resize of this VM.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
dohertj2
2026-09-17 13:06:04 -04:00
parent db35cffd8d
commit 1f7b1ac487
1 changed files with 44 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
truenas.md
+44 -21
View File
@@ -35,10 +35,13 @@
> `mypool` figures verified 2026-09-17. SSD_Pool / boot-pool rows are carried over from an earlier pass and were not
> re-checked.
**`mypool` is ONLINE, not DEGRADED** (the earlier note in this doc was stale). `errors: No known data errors`, DEDUP
1.00x. `zpool status` does still print a latched `status:` banner — *"One or more devices has experienced an
unrecoverable error... Applications are unaffected"* — which is a historical flag, not a current fault; clear it with
`zpool clear mypool` once you are satisfied the drive is healthy.
**`mypool` is ONLINE and healthy**, not DEGRADED — the earlier note in this doc was stale. `errors: No known data
errors`, DEDUP 1.00x.
> Until the 2026-09-17 reboot, `zpool status` printed a latched banner — *"One or more devices has experienced an
> unrecoverable error... Applications are unaffected"* — and `pool.query` reported `healthy: False`. That was a
> historical flag from the 2026-09-16 resilver, not a live fault; it cleared on reimport and the pool now reports
> `healthy: True`. If it ever reappears without a fresh resilver, treat it as real.
- **Layout**: 6 x raidz1 vdevs, each 4 x 7.28T
- **Hot spare**: `scsi-35000c500d2a29937`, AVAIL
@@ -46,14 +49,16 @@ unrecoverable error... Applications are unaffected"* — which is a historical f
## Memory / ARC
VM is configured with **128 GB** (`MemTotal` 125.8 GiB). `zfs_arc_max` is **pinned by a TrueNAS tunable**, not derived
from RAM — this is the single most important fact on this page if the VM is ever resized.
> **Reduced 128 GB → 64 GB on 2026-09-17.** The analysis and the resize procedure are both below.
VM is configured with **64 GB** (`MemTotal` 62.8 GiB). `zfs_arc_max` is **pinned by a TrueNAS tunable**, not derived
from RAM — this is the single most important fact on this page if the VM is ever resized again.
| Item | Value |
|---|---|
| `zfs_arc_max` (tunable id 8) | `103079215104` = 96 GiB |
| `zfs_dirty_data_max` (tunable id 15) | `17179869184` = 16 GiB |
| ARC size / target | 95.6 GiB / 96 GiB — saturated |
| `zfs_arc_max` (tunable id 8) | `42949672960` = 40 GiB |
| `zfs_dirty_data_max` (tunable id 15) | `8589934592` = 8 GiB |
| ARC size / target | warming after the 2026-09-17 reboot; cap 40 GiB |
| Dedup | **none** (`no DDT entries`, DEDUP 1.00x) |
| L2ARC / special vdev | **none** (`l2_hdr_size` 0) |
| Apps / VMs on the NAS | **none** (0 / 0) |
@@ -87,23 +92,41 @@ Veeam backup data read once and never again.
targets an ARC larger than the whole machine: it grows until the kernel shrinker forces it back, giving continuous
reclaim pressure, erratic latency and real OOM risk for `smbd` / `nfsd` / `middlewared` under a write burst.
Suggested values for a 64 GB config (leaves ~16 GiB for OS, services and headroom):
### Resize executed 2026-09-17
| Tunable | Current | 64 GB target |
Values applied (both persisted as tunables, both runtime-writable so no reboot was needed *for the tunables*):
| Tunable | Was | Now |
|---|---:|---:|
| `zfs_arc_max` | 96 GiB | **40 GiB** (`42949672960`) |
| `zfs_dirty_data_max` | 16 GiB | **8 GiB** (`8589934592`) |
| `zfs_arc_max` (id 8) | 96 GiB | **40 GiB** (`42949672960`) |
| `zfs_dirty_data_max` (id 15) | 16 GiB | **8 GiB** (`8589934592`) |
Apply with `midclt call tunable.update <id> '{"value": "<bytes>"}'`. Both parameters are runtime-writable
(`/sys/module/zfs/parameters/`), so no reboot is needed to change them — but the VM must be powered off to change its
RAM, which takes NFS and SMB down for Veeam, Plex and the docker host.
**The VM has PCI passthrough** (`pcipassthrough-13000`, the HBA), which forces
`memoryReservationLockedToMax: true` — so all of its RAM is *reserved* on the ESXi host whether ARC uses it or not.
That is why this change moved the host from **83.9% → 58.8%** memory usage. The reservation followed the resize
automatically; it did not need setting by hand.
**Caveat on the sample:** stats were taken over a 1d 10h window that included a Veeam *incremental* (2026-09-17 03:01)
but not a weekly *Full* (last 2026-09-12). Fulls are write-heavy on the NAS side and writes go through the dirty-data
path rather than ARC, so the conclusion should hold — re-check `mfu_ghost_hits` after a Full if you want certainty.
Procedure that worked:
**Validation trick:** lower `zfs_arc_max` to 40 GiB *while still on 128 GB RAM* and watch the ghost counters. If they
stay near zero, 64 GB is proven safe before the VM is ever touched.
```bash
# pre-flight: no scrub/resilver running, no middleware jobs, no SMB *write* handles, no Plex sessions
govc vm.power -s TrueNAS # graceful, via VMware Tools; took ~10s
govc vm.change -vm TrueNAS -m 65536
govc vm.power -on TrueNAS # ~3 min to SSH (HBA init + pool import)
```
**Client impact was nil.** Both docker-host NFS mounts (`/mnt/share`, `/mnt/other`) are `hard`, so containers stall in
uninterruptible sleep and resume when the server returns — `plex`, `sonarr` and `radarr` never restarted, and the only
D-state process during the outage was the `10.50.0.25-manager` NFS kernel thread. Check for SMB **write** handles rather
than sessions before scheduling: `midclt call smb.status LOCKS` — idle Explorer windows show as directory handles
(`filename: "."`, `WRITE_DATA: false`) and are safe to drop.
> **Side effect:** the latched `zpool status` error banner cleared on reboot — `mypool` now reports `healthy: True`.
> The `zpool clear` suggested above is no longer needed.
**Still to confirm:** ARC was cold after the reboot, so the post-change ghost-hit measurement has not been retaken. The
real test is the next Veeam **Full** — re-check `mfu_ghost_hits` after one. Note it had already moved 0 → 129 when ARC
was first capped at 40 GiB on the 128 GB VM, which is still negligible (0.0065% of hits) but no longer structurally zero.
### Key Datasets