dohertj2
db35cffd8d
Assessed whether the TrueNAS VM can drop from 128 GB to 64 GB of RAM. It can, but zfs_arc_max is pinned by an explicit tunable (id 8, 96 GiB) rather than derived from RAM, so it does not follow the VM down. Left alone on a 64 GiB VM, ZFS would target an ARC larger than the whole machine - ARC grows until the kernel shrinker forces it back, giving continuous reclaim pressure and real OOM risk for smbd/nfsd/middlewared under a write burst. zfs_dirty_data_max (id 15) is pinned at 16 GiB and has the same problem at 25% of a 64 GiB box. The sizing evidence is the ARC ghost lists, which count blocks evicted and then re-requested - the direct measure of whether a larger cache would help. mru_ghost_hits was 4 and mfu_ghost_hits 0 against 70M hits, so nothing being evicted is being re-read. The working set is ~22 GiB (MFU 20.7 GiB serving 94% of hits, plus 1.3 GiB metadata); the other ~75 GiB of ARC is MRU holding streaming media and Veeam backup data read once. No dedup, no L2ARC, no apps or VMs, 1.78 GiB total process RSS. Also corrects two stale facts found while looking: mypool is ONLINE with no known data errors at 96.4T/55%, not DEGRADED at 83.5T - the scary zpool status banner is a latched historical flag clearable with zpool clear - and /etc/version reports 25.10.16.12.33, not 25.04.2.6. SSD_Pool and boot-pool rows were not re-verified and are marked as such. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
165 lines
6.7 KiB
Markdown
165 lines
6.7 KiB
Markdown
# TrueNAS Server
|
|
|
|
## Access
|
|
|
|
- **Hostname**: nas.dohertylan.com
|
|
- **Version**: TrueNAS `25.10.16.12.33-production+truenas` (per `/etc/version`, checked 2026-09-17; this doc previously said 25.04.2.6 — older 25.04.x boot environments are still present on `boot-pool`)
|
|
- **SSH**: `ssh truenas_admin@10.100.0.25` (passwordless)
|
|
- **Web UI**: https://10.100.0.25 (ports 80/443 listening)
|
|
- **Domain**: dohertylan.com
|
|
- **DNS**: 10.100.0.1
|
|
|
|
## Network Interfaces
|
|
|
|
### ens224 — Management / General (10.100.0.0/24)
|
|
- **MTU**: 1500
|
|
- **IPs**: 10.100.0.25, .26, .27
|
|
- **Default gateway**: 10.100.0.1
|
|
- **Services**: SMB (445/139), WS-Discovery (5357), SSH (22), HTTP/S (80/443)
|
|
- **No NFS on this interface** — NFS (2049) does not bind to 10.100.0.x addresses
|
|
|
|
### ens256 — Storage / High-speed (10.50.0.0/24)
|
|
- **MTU**: 9000 (jumbo frames)
|
|
- **IPs**: 10.50.0.25, .26, .27, .28
|
|
- **Services**: SMB (445/139), NFS (2049)
|
|
- **No default route** — this is a dedicated storage network
|
|
|
|
## ZFS Pools
|
|
|
|
| Pool | Size | Alloc | Free | Cap | Frag | Health |
|
|
|------|------|-------|------|-----|------|--------|
|
|
| mypool | 175T | 96.4T | 78.3T | 55% | 2% | ONLINE |
|
|
| SSD_Pool | 2.91T | 3.01G | 2.90T | — | — | ONLINE |
|
|
| boot-pool | 15G | 11.6G | 3.40G | — | — | ONLINE |
|
|
|
|
> `mypool` figures verified 2026-09-17. SSD_Pool / boot-pool rows are carried over from an earlier pass and were not
|
|
> re-checked.
|
|
|
|
**`mypool` is ONLINE, not DEGRADED** (the earlier note in this doc was stale). `errors: No known data errors`, DEDUP
|
|
1.00x. `zpool status` does still print a latched `status:` banner — *"One or more devices has experienced an
|
|
unrecoverable error... Applications are unaffected"* — which is a historical flag, not a current fault; clear it with
|
|
`zpool clear mypool` once you are satisfied the drive is healthy.
|
|
|
|
- **Layout**: 6 x raidz1 vdevs, each 4 x 7.28T
|
|
- **Hot spare**: `scsi-35000c500d2a29937`, AVAIL
|
|
- **Last resilver**: 705G in 01:24:15 with 0 errors, Wed 2026-09-16 01:07:30
|
|
|
|
## Memory / ARC
|
|
|
|
VM is configured with **128 GB** (`MemTotal` 125.8 GiB). `zfs_arc_max` is **pinned by a TrueNAS tunable**, not derived
|
|
from RAM — this is the single most important fact on this page if the VM is ever resized.
|
|
|
|
| Item | Value |
|
|
|---|---|
|
|
| `zfs_arc_max` (tunable id 8) | `103079215104` = 96 GiB |
|
|
| `zfs_dirty_data_max` (tunable id 15) | `17179869184` = 16 GiB |
|
|
| ARC size / target | 95.6 GiB / 96 GiB — saturated |
|
|
| Dedup | **none** (`no DDT entries`, DEDUP 1.00x) |
|
|
| L2ARC / special vdev | **none** (`l2_hdr_size` 0) |
|
|
| Apps / VMs on the NAS | **none** (0 / 0) |
|
|
| Sum of all process RSS | 1.78 GiB |
|
|
|
|
### Sizing analysis (2026-09-17, 1d 10h uptime)
|
|
|
|
The ARC is roughly 4x larger than the working set actually needs:
|
|
|
|
| ARC region | Size | Hits served |
|
|
|---|---:|---:|
|
|
| MFU (frequently re-read) | 20.7 GiB | 66.5M — **94%** |
|
|
| MRU (touched once) | 74.7 GiB | 4.3M — 6% |
|
|
| Metadata | 1.3 GiB | 99.92% hit rate |
|
|
|
|
Overall hit rate 98.90% (70,488,374 hits / 781,848 misses); demand data 96.72%.
|
|
|
|
**The decisive metric is the ghost lists** — blocks evicted from ARC and then requested again, i.e. the direct measure
|
|
of "would a larger ARC help?":
|
|
|
|
```
|
|
mru_ghost_hits 4
|
|
mfu_ghost_hits 0
|
|
```
|
|
|
|
Four, against 70 million hits. Nothing being evicted is being re-read, so the cache is far past the point of
|
|
diminishing returns. The real working set is **~22 GiB** (MFU + metadata); the other ~75 GiB is streaming media and
|
|
Veeam backup data read once and never again.
|
|
|
|
**Conclusion: 64 GB is safe** — but only if `zfs_arc_max` comes down with it. Left at 96 GiB on a 64 GiB VM, ZFS
|
|
targets an ARC larger than the whole machine: it grows until the kernel shrinker forces it back, giving continuous
|
|
reclaim pressure, erratic latency and real OOM risk for `smbd` / `nfsd` / `middlewared` under a write burst.
|
|
|
|
Suggested values for a 64 GB config (leaves ~16 GiB for OS, services and headroom):
|
|
|
|
| Tunable | Current | 64 GB target |
|
|
|---|---:|---:|
|
|
| `zfs_arc_max` | 96 GiB | **40 GiB** (`42949672960`) |
|
|
| `zfs_dirty_data_max` | 16 GiB | **8 GiB** (`8589934592`) |
|
|
|
|
Apply with `midclt call tunable.update <id> '{"value": "<bytes>"}'`. Both parameters are runtime-writable
|
|
(`/sys/module/zfs/parameters/`), so no reboot is needed to change them — but the VM must be powered off to change its
|
|
RAM, which takes NFS and SMB down for Veeam, Plex and the docker host.
|
|
|
|
**Caveat on the sample:** stats were taken over a 1d 10h window that included a Veeam *incremental* (2026-09-17 03:01)
|
|
but not a weekly *Full* (last 2026-09-12). Fulls are write-heavy on the NAS side and writes go through the dirty-data
|
|
path rather than ARC, so the conclusion should hold — re-check `mfu_ghost_hits` after a Full if you want certainty.
|
|
|
|
**Validation trick:** lower `zfs_arc_max` to 40 GiB *while still on 128 GB RAM* and watch the ghost counters. If they
|
|
stay near zero, 64 GB is proven safe before the VM is ever touched.
|
|
|
|
### Key Datasets
|
|
|
|
| Dataset | Mountpoint | Used |
|
|
|---------|------------|------|
|
|
| mypool | /mnt/mypool | 60.7T (33.5T direct) |
|
|
| mypool/veeam | /mnt/mypool/veeam | 26.5T |
|
|
| SSD_Pool/benchmark | /mnt/SSD_Pool/benchmark | 3.00G |
|
|
|
|
`/mnt/mypool/share` and `/mnt/mypool/Other` are directories within the `mypool` dataset (not child datasets).
|
|
|
|
## SMB Shares
|
|
|
|
- **Workgroup**: DOHERTYLAN
|
|
- **NetBIOS name**: NAS
|
|
- **Multichannel**: enabled
|
|
- **Bind interfaces**: all IPs on both subnets + loopback
|
|
- **NTLMv1**: disabled (`ntlm auth = False`)
|
|
- **Guest**: disabled on all shares (`guest ok = False`)
|
|
|
|
| Share | Path | Notes |
|
|
|-------|------|-------|
|
|
| share | /mnt/mypool/share | Oplocks disabled, performance-tuned (aio 16K, max_xmit 64K) |
|
|
| Other | /mnt/mypool/Other | "Other files", POSIX locking off |
|
|
| benchmark | /mnt/SSD_Pool/benchmark | On SSD pool, io_uring sqpoll enabled, aio 16K |
|
|
|
|
### SMB Users
|
|
|
|
| User | UID | Full Name |
|
|
|------|-----|-----------|
|
|
| dohertj2 | 1000 | Joseph Doherty |
|
|
| homero | 3001 | Home RO |
|
|
|
|
## NFS Exports
|
|
|
|
| Export | Allowed Clients | Squash | anonuid/gid |
|
|
|--------|----------------|--------|-------------|
|
|
| /mnt/mypool/share | * (all) | all_squash | 1000/0 |
|
|
| /mnt/mypool/Other | * (all) | all_squash | 1000/0 |
|
|
| /mnt/SSD_Pool/benchmark | * (all) | all_squash | 1000/1000 |
|
|
| /mnt/mypool/veeam | 10.50.0.0/24 only | root (anonuid=0) | 0/— |
|
|
|
|
All exports use `sec=sys` (AUTH_SYS, no Kerberos).
|
|
|
|
**Important**: NFS (port 2049) only listens on 10.50.0.x addresses. NFS is not reachable via the 10.100.0.0/24 network.
|
|
|
|
## Filesystem ACLs
|
|
|
|
| Path | Owner | Group | Permissions |
|
|
|------|-------|-------|-------------|
|
|
| /mnt/mypool/share | dohertj2 | root | 755 |
|
|
| /mnt/mypool/Other | dohertj2 | dohertj2 | 755 |
|
|
| /mnt/SSD_Pool/benchmark | dohertj2 | dohertj2 | 777 |
|
|
| /mnt/mypool/veeam | root | root | 755 |
|
|
|
|
## Firewall
|
|
|
|
No firewall rules — iptables INPUT/OUTPUT/FORWARD all ACCEPT. No nftables rules.
|