dohertj2
db35cffd8d
Assessed whether the TrueNAS VM can drop from 128 GB to 64 GB of RAM. It can, but zfs_arc_max is pinned by an explicit tunable (id 8, 96 GiB) rather than derived from RAM, so it does not follow the VM down. Left alone on a 64 GiB VM, ZFS would target an ARC larger than the whole machine - ARC grows until the kernel shrinker forces it back, giving continuous reclaim pressure and real OOM risk for smbd/nfsd/middlewared under a write burst. zfs_dirty_data_max (id 15) is pinned at 16 GiB and has the same problem at 25% of a 64 GiB box. The sizing evidence is the ARC ghost lists, which count blocks evicted and then re-requested - the direct measure of whether a larger cache would help. mru_ghost_hits was 4 and mfu_ghost_hits 0 against 70M hits, so nothing being evicted is being re-read. The working set is ~22 GiB (MFU 20.7 GiB serving 94% of hits, plus 1.3 GiB metadata); the other ~75 GiB of ARC is MRU holding streaming media and Veeam backup data read once. No dedup, no L2ARC, no apps or VMs, 1.78 GiB total process RSS. Also corrects two stale facts found while looking: mypool is ONLINE with no known data errors at 96.4T/55%, not DEGRADED at 83.5T - the scary zpool status banner is a latched historical flag clearable with zpool clear - and /etc/version reports 25.10.16.12.33, not 25.04.2.6. SSD_Pool and boot-pool rows were not re-verified and are marked as such. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
6.7 KiB
TrueNAS Server
Access
- Hostname: nas.dohertylan.com
- Version: TrueNAS
25.10.16.12.33-production+truenas(per/etc/version, checked 2026-09-17; this doc previously said 25.04.2.6 — older 25.04.x boot environments are still present onboot-pool) - SSH:
ssh truenas_admin@10.100.0.25(passwordless) - Web UI: https://10.100.0.25 (ports 80/443 listening)
- Domain: dohertylan.com
- DNS: 10.100.0.1
Network Interfaces
ens224 — Management / General (10.100.0.0/24)
- MTU: 1500
- IPs: 10.100.0.25, .26, .27
- Default gateway: 10.100.0.1
- Services: SMB (445/139), WS-Discovery (5357), SSH (22), HTTP/S (80/443)
- No NFS on this interface — NFS (2049) does not bind to 10.100.0.x addresses
ens256 — Storage / High-speed (10.50.0.0/24)
- MTU: 9000 (jumbo frames)
- IPs: 10.50.0.25, .26, .27, .28
- Services: SMB (445/139), NFS (2049)
- No default route — this is a dedicated storage network
ZFS Pools
| Pool | Size | Alloc | Free | Cap | Frag | Health |
|---|---|---|---|---|---|---|
| mypool | 175T | 96.4T | 78.3T | 55% | 2% | ONLINE |
| SSD_Pool | 2.91T | 3.01G | 2.90T | — | — | ONLINE |
| boot-pool | 15G | 11.6G | 3.40G | — | — | ONLINE |
mypoolfigures verified 2026-09-17. SSD_Pool / boot-pool rows are carried over from an earlier pass and were not re-checked.
mypool is ONLINE, not DEGRADED (the earlier note in this doc was stale). errors: No known data errors, DEDUP
1.00x. zpool status does still print a latched status: banner — "One or more devices has experienced an
unrecoverable error... Applications are unaffected" — which is a historical flag, not a current fault; clear it with
zpool clear mypool once you are satisfied the drive is healthy.
- Layout: 6 x raidz1 vdevs, each 4 x 7.28T
- Hot spare:
scsi-35000c500d2a29937, AVAIL - Last resilver: 705G in 01:24:15 with 0 errors, Wed 2026-09-16 01:07:30
Memory / ARC
VM is configured with 128 GB (MemTotal 125.8 GiB). zfs_arc_max is pinned by a TrueNAS tunable, not derived
from RAM — this is the single most important fact on this page if the VM is ever resized.
| Item | Value |
|---|---|
zfs_arc_max (tunable id 8) |
103079215104 = 96 GiB |
zfs_dirty_data_max (tunable id 15) |
17179869184 = 16 GiB |
| ARC size / target | 95.6 GiB / 96 GiB — saturated |
| Dedup | none (no DDT entries, DEDUP 1.00x) |
| L2ARC / special vdev | none (l2_hdr_size 0) |
| Apps / VMs on the NAS | none (0 / 0) |
| Sum of all process RSS | 1.78 GiB |
Sizing analysis (2026-09-17, 1d 10h uptime)
The ARC is roughly 4x larger than the working set actually needs:
| ARC region | Size | Hits served |
|---|---|---|
| MFU (frequently re-read) | 20.7 GiB | 66.5M — 94% |
| MRU (touched once) | 74.7 GiB | 4.3M — 6% |
| Metadata | 1.3 GiB | 99.92% hit rate |
Overall hit rate 98.90% (70,488,374 hits / 781,848 misses); demand data 96.72%.
The decisive metric is the ghost lists — blocks evicted from ARC and then requested again, i.e. the direct measure of "would a larger ARC help?":
mru_ghost_hits 4
mfu_ghost_hits 0
Four, against 70 million hits. Nothing being evicted is being re-read, so the cache is far past the point of diminishing returns. The real working set is ~22 GiB (MFU + metadata); the other ~75 GiB is streaming media and Veeam backup data read once and never again.
Conclusion: 64 GB is safe — but only if zfs_arc_max comes down with it. Left at 96 GiB on a 64 GiB VM, ZFS
targets an ARC larger than the whole machine: it grows until the kernel shrinker forces it back, giving continuous
reclaim pressure, erratic latency and real OOM risk for smbd / nfsd / middlewared under a write burst.
Suggested values for a 64 GB config (leaves ~16 GiB for OS, services and headroom):
| Tunable | Current | 64 GB target |
|---|---|---|
zfs_arc_max |
96 GiB | 40 GiB (42949672960) |
zfs_dirty_data_max |
16 GiB | 8 GiB (8589934592) |
Apply with midclt call tunable.update <id> '{"value": "<bytes>"}'. Both parameters are runtime-writable
(/sys/module/zfs/parameters/), so no reboot is needed to change them — but the VM must be powered off to change its
RAM, which takes NFS and SMB down for Veeam, Plex and the docker host.
Caveat on the sample: stats were taken over a 1d 10h window that included a Veeam incremental (2026-09-17 03:01)
but not a weekly Full (last 2026-09-12). Fulls are write-heavy on the NAS side and writes go through the dirty-data
path rather than ARC, so the conclusion should hold — re-check mfu_ghost_hits after a Full if you want certainty.
Validation trick: lower zfs_arc_max to 40 GiB while still on 128 GB RAM and watch the ghost counters. If they
stay near zero, 64 GB is proven safe before the VM is ever touched.
Key Datasets
| Dataset | Mountpoint | Used |
|---|---|---|
| mypool | /mnt/mypool | 60.7T (33.5T direct) |
| mypool/veeam | /mnt/mypool/veeam | 26.5T |
| SSD_Pool/benchmark | /mnt/SSD_Pool/benchmark | 3.00G |
/mnt/mypool/share and /mnt/mypool/Other are directories within the mypool dataset (not child datasets).
SMB Shares
- Workgroup: DOHERTYLAN
- NetBIOS name: NAS
- Multichannel: enabled
- Bind interfaces: all IPs on both subnets + loopback
- NTLMv1: disabled (
ntlm auth = False) - Guest: disabled on all shares (
guest ok = False)
| Share | Path | Notes |
|---|---|---|
| share | /mnt/mypool/share | Oplocks disabled, performance-tuned (aio 16K, max_xmit 64K) |
| Other | /mnt/mypool/Other | "Other files", POSIX locking off |
| benchmark | /mnt/SSD_Pool/benchmark | On SSD pool, io_uring sqpoll enabled, aio 16K |
SMB Users
| User | UID | Full Name |
|---|---|---|
| dohertj2 | 1000 | Joseph Doherty |
| homero | 3001 | Home RO |
NFS Exports
| Export | Allowed Clients | Squash | anonuid/gid |
|---|---|---|---|
| /mnt/mypool/share | * (all) | all_squash | 1000/0 |
| /mnt/mypool/Other | * (all) | all_squash | 1000/0 |
| /mnt/SSD_Pool/benchmark | * (all) | all_squash | 1000/1000 |
| /mnt/mypool/veeam | 10.50.0.0/24 only | root (anonuid=0) | 0/— |
All exports use sec=sys (AUTH_SYS, no Kerberos).
Important: NFS (port 2049) only listens on 10.50.0.x addresses. NFS is not reachable via the 10.100.0.0/24 network.
Filesystem ACLs
| Path | Owner | Group | Permissions |
|---|---|---|---|
| /mnt/mypool/share | dohertj2 | root | 755 |
| /mnt/mypool/Other | dohertj2 | dohertj2 | 755 |
| /mnt/SSD_Pool/benchmark | dohertj2 | dohertj2 | 777 |
| /mnt/mypool/veeam | root | root | 755 |
Firewall
No firewall rules — iptables INPUT/OUTPUT/FORWARD all ACCEPT. No nftables rules.