network/truenas.md

# TrueNAS Server

## Access

- **Hostname**: nas.dohertylan.com
- **Version**: TrueNAS `25.10.16.12.33-production+truenas` (per `/etc/version`, checked 2026-09-17; this doc previously said 25.04.2.6 — older 25.04.x boot environments are still present on `boot-pool`)
- **SSH**: `ssh truenas_admin@10.100.0.25` (passwordless)
- **Web UI**: https://10.100.0.25 (ports 80/443 listening)
- **Domain**: dohertylan.com
- **DNS**: 10.100.0.1

## Network Interfaces

### ens224 — Management / General (10.100.0.0/24)
- **MTU**: 1500
- **IPs**: 10.100.0.25, .26, .27
- **Default gateway**: 10.100.0.1
- **Services**: SMB (445/139), WS-Discovery (5357), SSH (22), HTTP/S (80/443)
- **No NFS on this interface** — NFS (2049) does not bind to 10.100.0.x addresses

### ens256 — Storage / High-speed (10.50.0.0/24)
- **MTU**: 9000 (jumbo frames)
- **IPs**: 10.50.0.25, .26, .27, .28
- **Services**: SMB (445/139), NFS (2049)
- **No default route** — this is a dedicated storage network

## ZFS Pools

| Pool | Size | Alloc | Free | Cap | Frag | Health |
|------|------|-------|------|-----|------|--------|
| mypool | 175T | 96.4T | 78.3T | 55% | 2% | ONLINE |
| SSD_Pool | 2.91T | 3.01G | 2.90T | — | — | ONLINE |
| boot-pool | 15G | 11.6G | 3.40G | — | — | ONLINE |

> `mypool` figures verified 2026-09-17. SSD_Pool / boot-pool rows are carried over from an earlier pass and were not
> re-checked.

**`mypool` is ONLINE and healthy**, not DEGRADED — the earlier note in this doc was stale. `errors: No known data
errors`, DEDUP 1.00x.

> Until the 2026-09-17 reboot, `zpool status` printed a latched banner — *"One or more devices has experienced an
> unrecoverable error... Applications are unaffected"* — and `pool.query` reported `healthy: False`. That was a
> historical flag from the 2026-09-16 resilver, not a live fault; it cleared on reimport and the pool now reports
> `healthy: True`. If it ever reappears without a fresh resilver, treat it as real.

- **Layout**: 6 x raidz1 vdevs, each 4 x 7.28T
- **Hot spare**: `scsi-35000c500d2a29937`, AVAIL
- **Last resilver**: 705G in 01:24:15 with 0 errors, Wed 2026-09-16 01:07:30

## Memory / ARC

> **Reduced 128 GB → 64 GB on 2026-09-17.** The analysis and the resize procedure are both below.

VM is configured with **64 GB** (`MemTotal` 62.8 GiB). `zfs_arc_max` is **pinned by a TrueNAS tunable**, not derived
from RAM — this is the single most important fact on this page if the VM is ever resized again.

| Item | Value |
|---|---|
| `zfs_arc_max` (tunable id 8) | `42949672960` = 40 GiB |
| `zfs_dirty_data_max` (tunable id 15) | `8589934592` = 8 GiB |
| ARC size / target | warming after the 2026-09-17 reboot; cap 40 GiB |
| Dedup | **none** (`no DDT entries`, DEDUP 1.00x) |
| L2ARC / special vdev | **none** (`l2_hdr_size` 0) |
| Apps / VMs on the NAS | **none** (0 / 0) |
| Sum of all process RSS | 1.78 GiB |

### Sizing analysis (2026-09-17, 1d 10h uptime)

The ARC is roughly 4x larger than the working set actually needs:

| ARC region | Size | Hits served |
|---|---:|---:|
| MFU (frequently re-read) | 20.7 GiB | 66.5M — **94%** |
| MRU (touched once) | 74.7 GiB | 4.3M — 6% |
| Metadata | 1.3 GiB | 99.92% hit rate |

Overall hit rate 98.90% (70,488,374 hits / 781,848 misses); demand data 96.72%.

**The decisive metric is the ghost lists** — blocks evicted from ARC and then requested again, i.e. the direct measure
of "would a larger ARC help?":

```
mru_ghost_hits  4
mfu_ghost_hits  0
```

Four, against 70 million hits. Nothing being evicted is being re-read, so the cache is far past the point of
diminishing returns. The real working set is **~22 GiB** (MFU + metadata); the other ~75 GiB is streaming media and
Veeam backup data read once and never again.

**Conclusion: 64 GB is safe** — but only if `zfs_arc_max` comes down with it. Left at 96 GiB on a 64 GiB VM, ZFS
targets an ARC larger than the whole machine: it grows until the kernel shrinker forces it back, giving continuous
reclaim pressure, erratic latency and real OOM risk for `smbd` / `nfsd` / `middlewared` under a write burst.

### Resize executed 2026-09-17

Values applied (both persisted as tunables, both runtime-writable so no reboot was needed *for the tunables*):

| Tunable | Was | Now |
|---|---:|---:|
| `zfs_arc_max` (id 8) | 96 GiB | **40 GiB** (`42949672960`) |
| `zfs_dirty_data_max` (id 15) | 16 GiB | **8 GiB** (`8589934592`) |

**The VM has PCI passthrough** (`pcipassthrough-13000`, the HBA), which forces
`memoryReservationLockedToMax: true` — so all of its RAM is *reserved* on the ESXi host whether ARC uses it or not.
That is why this change moved the host from **83.9% → 58.8%** memory usage. The reservation followed the resize
automatically; it did not need setting by hand.

Procedure that worked:

```bash
# pre-flight: no scrub/resilver running, no middleware jobs, no SMB *write* handles, no Plex sessions
govc vm.power -s TrueNAS          # graceful, via VMware Tools; took ~10s
govc vm.change -vm TrueNAS -m 65536
govc vm.power -on TrueNAS         # ~3 min to SSH (HBA init + pool import)
```

**Client impact was nil.** Both docker-host NFS mounts (`/mnt/share`, `/mnt/other`) are `hard`, so containers stall in
uninterruptible sleep and resume when the server returns — `plex`, `sonarr` and `radarr` never restarted, and the only
D-state process during the outage was the `10.50.0.25-manager` NFS kernel thread. Check for SMB **write** handles rather
than sessions before scheduling: `midclt call smb.status LOCKS` — idle Explorer windows show as directory handles
(`filename: "."`, `WRITE_DATA: false`) and are safe to drop.

> **Side effect:** the latched `zpool status` error banner cleared on reboot — `mypool` now reports `healthy: True`.
> The `zpool clear` suggested above is no longer needed.

**Still to confirm:** ARC was cold after the reboot, so the post-change ghost-hit measurement has not been retaken. The
real test is the next Veeam **Full** — re-check `mfu_ghost_hits` after one. Note it had already moved 0 → 129 when ARC
was first capped at 40 GiB on the 128 GB VM, which is still negligible (0.0065% of hits) but no longer structurally zero.

### Key Datasets

| Dataset | Mountpoint | Used |
|---------|------------|------|
| mypool | /mnt/mypool | 60.7T (33.5T direct) |
| mypool/veeam | /mnt/mypool/veeam | 26.5T |
| SSD_Pool/benchmark | /mnt/SSD_Pool/benchmark | 3.00G |

`/mnt/mypool/share` and `/mnt/mypool/Other` are directories within the `mypool` dataset (not child datasets).

## SMB Shares

- **Workgroup**: DOHERTYLAN
- **NetBIOS name**: NAS
- **Multichannel**: enabled
- **Bind interfaces**: all IPs on both subnets + loopback
- **NTLMv1**: disabled (`ntlm auth = False`)
- **Guest**: disabled on all shares (`guest ok = False`)

| Share | Path | Notes |
|-------|------|-------|
| share | /mnt/mypool/share | Oplocks disabled, performance-tuned (aio 16K, max_xmit 64K) |
| Other | /mnt/mypool/Other | "Other files", POSIX locking off |
| benchmark | /mnt/SSD_Pool/benchmark | On SSD pool, io_uring sqpoll enabled, aio 16K |

### SMB Users

| User | UID | Full Name |
|------|-----|-----------|
| dohertj2 | 1000 | Joseph Doherty |
| homero | 3001 | Home RO |

## NFS Exports

| Export | Allowed Clients | Squash | anonuid/gid |
|--------|----------------|--------|-------------|
| /mnt/mypool/share | * (all) | all_squash | 1000/0 |
| /mnt/mypool/Other | * (all) | all_squash | 1000/0 |
| /mnt/SSD_Pool/benchmark | * (all) | all_squash | 1000/1000 |
| /mnt/mypool/veeam | 10.50.0.0/24 only | root (anonuid=0) | 0/— |

All exports use `sec=sys` (AUTH_SYS, no Kerberos).

**Important**: NFS (port 2049) only listens on 10.50.0.x addresses. NFS is not reachable via the 10.100.0.0/24 network.

## Filesystem ACLs

| Path | Owner | Group | Permissions |
|------|-------|-------|-------------|
| /mnt/mypool/share | dohertj2 | root | 755 |
| /mnt/mypool/Other | dohertj2 | dohertj2 | 755 |
| /mnt/SSD_Pool/benchmark | dohertj2 | dohertj2 | 777 |
| /mnt/mypool/veeam | root | root | 755 |

## Firewall

No firewall rules — iptables INPUT/OUTPUT/FORWARD all ACCEPT. No nftables rules.