dohertj2/network
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document TrueNAS ARC sizing; correct stale pool and version facts

Browse Source 
Assessed whether the TrueNAS VM can drop from 128 GB to 64 GB of RAM.
It can, but zfs_arc_max is pinned by an explicit tunable (id 8, 96 GiB)
rather than derived from RAM, so it does not follow the VM down. Left
alone on a 64 GiB VM, ZFS would target an ARC larger than the whole
machine - ARC grows until the kernel shrinker forces it back, giving
continuous reclaim pressure and real OOM risk for smbd/nfsd/middlewared
under a write burst. zfs_dirty_data_max (id 15) is pinned at 16 GiB and
has the same problem at 25% of a 64 GiB box.

The sizing evidence is the ARC ghost lists, which count blocks evicted
and then re-requested - the direct measure of whether a larger cache
would help. mru_ghost_hits was 4 and mfu_ghost_hits 0 against 70M hits,
so nothing being evicted is being re-read. The working set is ~22 GiB
(MFU 20.7 GiB serving 94% of hits, plus 1.3 GiB metadata); the other
~75 GiB of ARC is MRU holding streaming media and Veeam backup data read
once. No dedup, no L2ARC, no apps or VMs, 1.78 GiB total process RSS.

Also corrects two stale facts found while looking: mypool is ONLINE with
no known data errors at 96.4T/55%, not DEGRADED at 83.5T - the scary
zpool status banner is a latched historical flag clearable with
zpool clear - and /etc/version reports 25.10.16.12.33, not 25.04.2.6.
SSD_Pool and boot-pool rows were not re-verified and are marked as such.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
dohertj2
2026-09-17 12:45:36 -04:00
parent 14cc4faa28
commit db35cffd8d
1 changed files with 78 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
truenas.md
+78 -7
View File
@@ -3,7 +3,7 @@
## Access
- **Hostname**: nas.dohertylan.com
- **Version**: TrueNAS 25.04.2.6 (Electric Eel / Linux-based)
- **Version**: TrueNAS `25.10.16.12.33-production+truenas` (per `/etc/version`, checked 2026-09-17; this doc previously said 25.04.2.6 — older 25.04.x boot environments are still present on `boot-pool`)
- **SSH**: `ssh truenas_admin@10.100.0.25` (passwordless)
- **Web UI**: https://10.100.0.25 (ports 80/443 listening)
- **Domain**: dohertylan.com
@@ -26,13 +26,84 @@
## ZFS Pools
| Pool | Size | Used | Free | Health |
|------|------|------|------|--------|
| mypool | 175T | 83.5T | 91.2T | **DEGRADED** |
| SSD_Pool | 2.91T | 3.01G | 2.90T | ONLINE |
| boot-pool | 15G | 11.6G | 3.40G | ONLINE |
| Pool | Size | Alloc | Free | Cap | Frag | Health |
|------|------|-------|------|-----|------|--------|
| mypool | 175T | 96.4T | 78.3T | 55% | 2% | ONLINE |
| SSD_Pool | 2.91T | 3.01G | 2.90T | — | — | ONLINE |
| boot-pool | 15G | 11.6G | 3.40G | — | — | ONLINE |
**Note**: `mypool` is in DEGRADED state.
> `mypool` figures verified 2026-09-17. SSD_Pool / boot-pool rows are carried over from an earlier pass and were not
> re-checked.
**`mypool` is ONLINE, not DEGRADED** (the earlier note in this doc was stale). `errors: No known data errors`, DEDUP
1.00x. `zpool status` does still print a latched `status:` banner — *"One or more devices has experienced an
unrecoverable error... Applications are unaffected"* — which is a historical flag, not a current fault; clear it with
`zpool clear mypool` once you are satisfied the drive is healthy.
- **Layout**: 6 x raidz1 vdevs, each 4 x 7.28T
- **Hot spare**: `scsi-35000c500d2a29937`, AVAIL
- **Last resilver**: 705G in 01:24:15 with 0 errors, Wed 2026-09-16 01:07:30
## Memory / ARC
VM is configured with **128 GB** (`MemTotal` 125.8 GiB). `zfs_arc_max` is **pinned by a TrueNAS tunable**, not derived
from RAM — this is the single most important fact on this page if the VM is ever resized.
| Item | Value |
|---|---|
| `zfs_arc_max` (tunable id 8) | `103079215104` = 96 GiB |
| `zfs_dirty_data_max` (tunable id 15) | `17179869184` = 16 GiB |
| ARC size / target | 95.6 GiB / 96 GiB — saturated |
| Dedup | **none** (`no DDT entries`, DEDUP 1.00x) |
| L2ARC / special vdev | **none** (`l2_hdr_size` 0) |
| Apps / VMs on the NAS | **none** (0 / 0) |
| Sum of all process RSS | 1.78 GiB |
### Sizing analysis (2026-09-17, 1d 10h uptime)
The ARC is roughly 4x larger than the working set actually needs:
| ARC region | Size | Hits served |
|---|---:|---:|
| MFU (frequently re-read) | 20.7 GiB | 66.5M — **94%** |
| MRU (touched once) | 74.7 GiB | 4.3M — 6% |
| Metadata | 1.3 GiB | 99.92% hit rate |
Overall hit rate 98.90% (70,488,374 hits / 781,848 misses); demand data 96.72%.
**The decisive metric is the ghost lists** — blocks evicted from ARC and then requested again, i.e. the direct measure
of "would a larger ARC help?":
```
mru_ghost_hits 4
mfu_ghost_hits 0
```
Four, against 70 million hits. Nothing being evicted is being re-read, so the cache is far past the point of
diminishing returns. The real working set is **~22 GiB** (MFU + metadata); the other ~75 GiB is streaming media and
Veeam backup data read once and never again.
**Conclusion: 64 GB is safe** — but only if `zfs_arc_max` comes down with it. Left at 96 GiB on a 64 GiB VM, ZFS
targets an ARC larger than the whole machine: it grows until the kernel shrinker forces it back, giving continuous
reclaim pressure, erratic latency and real OOM risk for `smbd` / `nfsd` / `middlewared` under a write burst.
Suggested values for a 64 GB config (leaves ~16 GiB for OS, services and headroom):
| Tunable | Current | 64 GB target |
|---|---:|---:|
| `zfs_arc_max` | 96 GiB | **40 GiB** (`42949672960`) |
| `zfs_dirty_data_max` | 16 GiB | **8 GiB** (`8589934592`) |
Apply with `midclt call tunable.update <id> '{"value": "<bytes>"}'`. Both parameters are runtime-writable
(`/sys/module/zfs/parameters/`), so no reboot is needed to change them — but the VM must be powered off to change its
RAM, which takes NFS and SMB down for Veeam, Plex and the docker host.
**Caveat on the sample:** stats were taken over a 1d 10h window that included a Veeam *incremental* (2026-09-17 03:01)
but not a weekly *Full* (last 2026-09-12). Fulls are write-heavy on the NAS side and writes go through the dirty-data
path rather than ARC, so the conclusion should hold — re-check `mfu_ghost_hits` after a Full if you want certainty.
**Validation trick:** lower `zfs_arc_max` to 40 GiB *while still on 128 GB RAM* and watch the ghost counters. If they
stay near zero, 64 GB is proven safe before the VM is ever touched.
### Key Datasets