network/truenas.md

TrueNAS Server

Access

• Hostname: nas.dohertylan.com
• Version: TrueNAS 25.10.16.12.33-production+truenas (per /etc/version, checked 2026-09-17; this doc previously said 25.04.2.6 — older 25.04.x boot environments are still present on boot-pool)
• SSH: ssh truenas_admin@10.100.0.25 (passwordless)
• Web UI: https://10.100.0.25 (ports 80/443 listening)
• Domain: dohertylan.com
• DNS: 10.100.0.1

Network Interfaces

ens224 — Management / General (10.100.0.0/24)

• MTU: 1500
• IPs: 10.100.0.25, .26, .27
• Default gateway: 10.100.0.1
• Services: SMB (445/139), WS-Discovery (5357), SSH (22), HTTP/S (80/443)
• No NFS on this interface — NFS (2049) does not bind to 10.100.0.x addresses

ens256 — Storage / High-speed (10.50.0.0/24)

• MTU: 9000 (jumbo frames)
• IPs: 10.50.0.25, .26, .27, .28
• Services: SMB (445/139), NFS (2049)
• No default route — this is a dedicated storage network

ZFS Pools

Pool	Size	Alloc	Free	Cap	Frag	Health
mypool	175T	96.4T	78.3T	55%	2%	ONLINE
SSD_Pool	2.91T	3.01G	2.90T	—	—	ONLINE
boot-pool	15G	11.6G	3.40G	—	—	ONLINE

mypool figures verified 2026-09-17. SSD_Pool / boot-pool rows are carried over from an earlier pass and were not re-checked.

mypool is ONLINE and healthy, not DEGRADED — the earlier note in this doc was stale. errors: No known data errors, DEDUP 1.00x.

Until the 2026-09-17 reboot, zpool status printed a latched banner — "One or more devices has experienced an unrecoverable error... Applications are unaffected" — and pool.query reported healthy: False. That was a historical flag from the 2026-09-16 resilver, not a live fault; it cleared on reimport and the pool now reports healthy: True. If it ever reappears without a fresh resilver, treat it as real.

• Layout: 6 x raidz1 vdevs, each 4 x 7.28T
• Hot spare: scsi-35000c500d2a29937, AVAIL
• Last resilver: 705G in 01:24:15 with 0 errors, Wed 2026-09-16 01:07:30

Memory / ARC

Reduced 128 GB → 64 GB on 2026-09-17. The analysis and the resize procedure are both below.

VM is configured with 64 GB (MemTotal 62.8 GiB). zfs_arc_max is pinned by a TrueNAS tunable, not derived from RAM — this is the single most important fact on this page if the VM is ever resized again.

Item	Value
zfs_arc_max (tunable id 8)	42949672960 = 40 GiB
zfs_dirty_data_max (tunable id 15)	8589934592 = 8 GiB
ARC size / target	warming after the 2026-09-17 reboot; cap 40 GiB
Dedup	none (no DDT entries, DEDUP 1.00x)
L2ARC / special vdev	none (l2_hdr_size 0)
Apps / VMs on the NAS	none (0 / 0)
Sum of all process RSS	1.78 GiB

Sizing analysis (2026-09-17, 1d 10h uptime)

The ARC is roughly 4x larger than the working set actually needs:

ARC region	Size	Hits served
MFU (frequently re-read)	20.7 GiB	66.5M — 94%
MRU (touched once)	74.7 GiB	4.3M — 6%
Metadata	1.3 GiB	99.92% hit rate

Overall hit rate 98.90% (70,488,374 hits / 781,848 misses); demand data 96.72%.

The decisive metric is the ghost lists — blocks evicted from ARC and then requested again, i.e. the direct measure of "would a larger ARC help?":

mru_ghost_hits  4
mfu_ghost_hits  0

Four, against 70 million hits. Nothing being evicted is being re-read, so the cache is far past the point of diminishing returns. The real working set is ~22 GiB (MFU + metadata); the other ~75 GiB is streaming media and Veeam backup data read once and never again.

Conclusion: 64 GB is safe — but only if zfs_arc_max comes down with it. Left at 96 GiB on a 64 GiB VM, ZFS targets an ARC larger than the whole machine: it grows until the kernel shrinker forces it back, giving continuous reclaim pressure, erratic latency and real OOM risk for smbd / nfsd / middlewared under a write burst.

Resize executed 2026-09-17

Values applied (both persisted as tunables, both runtime-writable so no reboot was needed for the tunables):

Tunable	Was	Now
zfs_arc_max (id 8)	96 GiB	40 GiB (42949672960)
zfs_dirty_data_max (id 15)	16 GiB	8 GiB (8589934592)

The VM has PCI passthrough (pcipassthrough-13000, the HBA), which forces memoryReservationLockedToMax: true — so all of its RAM is reserved on the ESXi host whether ARC uses it or not. That is why this change moved the host from 83.9% → 58.8% memory usage. The reservation followed the resize automatically; it did not need setting by hand.

Procedure that worked:

# pre-flight: no scrub/resilver running, no middleware jobs, no SMB *write* handles, no Plex sessions
govc vm.power -s TrueNAS          # graceful, via VMware Tools; took ~10s
govc vm.change -vm TrueNAS -m 65536
govc vm.power -on TrueNAS         # ~3 min to SSH (HBA init + pool import)

Client impact was nil. Both docker-host NFS mounts (/mnt/share, /mnt/other) are hard, so containers stall in uninterruptible sleep and resume when the server returns — plex, sonarr and radarr never restarted, and the only D-state process during the outage was the 10.50.0.25-manager NFS kernel thread. Check for SMB write handles rather than sessions before scheduling: midclt call smb.status LOCKS — idle Explorer windows show as directory handles (filename: ".", WRITE_DATA: false) and are safe to drop.

Side effect: the latched zpool status error banner cleared on reboot — mypool now reports healthy: True. The zpool clear suggested above is no longer needed.

Still to confirm: ARC was cold after the reboot, so the post-change ghost-hit measurement has not been retaken. The real test is the next Veeam Full — re-check mfu_ghost_hits after one. Note it had already moved 0 → 129 when ARC was first capped at 40 GiB on the 128 GB VM, which is still negligible (0.0065% of hits) but no longer structurally zero.

Key Datasets

Dataset	Mountpoint	Used
mypool	/mnt/mypool	60.7T (33.5T direct)
mypool/veeam	/mnt/mypool/veeam	26.5T
SSD_Pool/benchmark	/mnt/SSD_Pool/benchmark	3.00G

/mnt/mypool/share and /mnt/mypool/Other are directories within the mypool dataset (not child datasets).

SMB Shares

• Workgroup: DOHERTYLAN
• NetBIOS name: NAS
• Multichannel: enabled
• Bind interfaces: all IPs on both subnets + loopback
• NTLMv1: disabled (ntlm auth = False)
• Guest: disabled on all shares (guest ok = False)

Share	Path	Notes
share	/mnt/mypool/share	Oplocks disabled, performance-tuned (aio 16K, max_xmit 64K)
Other	/mnt/mypool/Other	"Other files", POSIX locking off
benchmark	/mnt/SSD_Pool/benchmark	On SSD pool, io_uring sqpoll enabled, aio 16K

SMB Users

User	UID	Full Name
dohertj2	1000	Joseph Doherty
homero	3001	Home RO

NFS Exports

Export	Allowed Clients	Squash	anonuid/gid
/mnt/mypool/share	* (all)	all_squash	1000/0
/mnt/mypool/Other	* (all)	all_squash	1000/0
/mnt/SSD_Pool/benchmark	* (all)	all_squash	1000/1000
/mnt/mypool/veeam	10.50.0.0/24 only	root (anonuid=0)	0/—

All exports use sec=sys (AUTH_SYS, no Kerberos).

Important: NFS (port 2049) only listens on 10.50.0.x addresses. NFS is not reachable via the 10.100.0.0/24 network.

Filesystem ACLs

Path	Owner	Group	Permissions
/mnt/mypool/share	dohertj2	root	755
/mnt/mypool/Other	dohertj2	dohertj2	755
/mnt/SSD_Pool/benchmark	dohertj2	dohertj2	777
/mnt/mypool/veeam	root	root	755

Firewall

No firewall rules — iptables INPUT/OUTPUT/FORWARD all ACCEPT. No nftables rules.