Grid now uses PowerMonitorMain (Aeotec ZWA046) per leg since clamp 1 is
reversed; water uses the Droplet integral sensor with flow rate. Refresh
HA version and entity counts.
Executes the sizing analysis committed in db35cff. The VM has HBA PCI
passthrough, so memoryReservationLockedToMax is true and all of its RAM
is reserved on the host whether ARC uses it or not - this moved the ESXi
host from 83.9% to 58.8% memory usage. The reservation followed the
resize automatically.
govc vm.power -s TrueNAS # graceful via Tools, ~10s
govc vm.change -vm TrueNAS -m 65536
govc vm.power -on TrueNAS # ~3 min to SSH (HBA init + pool import)
Tunables now zfs_arc_max 40 GiB and zfs_dirty_data_max 8 GiB, both
persisted and verified live. The second one matters more at 64 GB: left
at 16 GiB it would have been 25% of RAM and stacked with ARC during the
nightly Veeam window.
Client impact was nil. Both docker-host NFS mounts are hard, so plex,
sonarr and radarr stalled in uninterruptible sleep and resumed without
restarting; the only D-state process during the outage was the NFS
kernel thread. Pre-flight checked SMB write handles rather than sessions
- the two live sessions were idle Explorer directory handles with
WRITE_DATA false, safe to drop. Post-boot: pool imported, all three NFS
exports restored including the Veeam repo, SMB and NFS running, Plex
serving all three libraries.
Two things resolved themselves: mypool's latched "unrecoverable error"
banner cleared on reimport and it now reports healthy true, so the
zpool clear advice is dropped.
Records the procedure and the pre-flight checks, since the pinned-tunable
trap applies to any future resize of this VM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Assessed whether the TrueNAS VM can drop from 128 GB to 64 GB of RAM.
It can, but zfs_arc_max is pinned by an explicit tunable (id 8, 96 GiB)
rather than derived from RAM, so it does not follow the VM down. Left
alone on a 64 GiB VM, ZFS would target an ARC larger than the whole
machine - ARC grows until the kernel shrinker forces it back, giving
continuous reclaim pressure and real OOM risk for smbd/nfsd/middlewared
under a write burst. zfs_dirty_data_max (id 15) is pinned at 16 GiB and
has the same problem at 25% of a 64 GiB box.
The sizing evidence is the ARC ghost lists, which count blocks evicted
and then re-requested - the direct measure of whether a larger cache
would help. mru_ghost_hits was 4 and mfu_ghost_hits 0 against 70M hits,
so nothing being evicted is being re-read. The working set is ~22 GiB
(MFU 20.7 GiB serving 94% of hits, plus 1.3 GiB metadata); the other
~75 GiB of ARC is MRU holding streaming media and Veeam backup data read
once. No dedup, no L2ARC, no apps or VMs, 1.78 GiB total process RSS.
Also corrects two stale facts found while looking: mypool is ONLINE with
no known data errors at 96.4T/55%, not DEGRADED at 83.5T - the scary
zpool status banner is a latched historical flag clearable with
zpool clear - and /etc/version reports 25.10.16.12.33, not 25.04.2.6.
SSD_Pool and boot-pool rows were not re-verified and are marked as such.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Completes section 9 step 3. govc vm.destroy Plex removed the inventory
entry and the [NVME_1] Plex/ folder; datastore free space went
1,619.8 -> 1,756.1 GB (+136.3 GB). 10.100.0.24 / 10.50.0.31 and MACs
00:0c:29:4a:ef:3d / 00:0c:29:4a:ef:47 are now free for reuse.
Done on 2026-09-17 rather than the planned ~2026-10-15. Checking the
plan's own Veeam precondition inverted its assumption: the Backup VMs
job has EnableDeletedVmDataRetention set to False with Simple retention
of 7 restore points, so a VM removed from the job has its points pruned
as the remaining VMs roll the chain - Plex's are due to age out around
2026-09-24, about three weeks before the disks were scheduled for
deletion. Deleting while a good restore point still existed (2026-09-17
03:01, consistent) was safer than deleting later with none.
Recovery position is now: Veeam image restore until ~2026-09-24, then
only the pre-migration library DB at
/mnt/share/backups/plex-migration-2026-09-16/, which rebuilds a Plex
install but not the VM.
veeam.md gains the retention gotcha, since it applies to any future
decommission and the 30-day DeletedVmsDataRetentionPeriodDays value is
misleading while the toggle is off.
Also flags in esxi.md that its VM tables are incomplete: the host has 10
VMs, WW_DEV_VM / Rockwell_Win10 / TCBSD are absent, and DevVM is powered
off rather than on. Left the tables themselves alone as out of scope.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plex has run as a container on the Docker host since the 2026-09-16
migration. The old Debian 11 VM (10.100.0.24) is now retired per
plex-docker-migration.md section 9, steps 1, 2 and 4-7:
- removed from the Veeam "Backup VMs" job
- guest shut down gracefully, pre-migration snapshot removed and
consolidated back to Plex.vmdk
- Guacamole "Plex SSH" connection deleted
- .24 / 10.50.0.31 DHCP leases released on the UDM
The VMDKs are deliberately kept until ~2026-10-15 so the VM remains a
one-command rollback; step 3 stays open until then.
Before powering the VM off, every integration was re-verified as
pointing at 10.100.0.35: Sonarr, Radarr, Tautulli, seerr, Uptime Kuma,
Homepage, Traefik and the UDM PLEX port-forward.
Section 9 gains a decommission log recording two behaviours that cost
time to work out: Veeam turns a removed job include into an explicit
Exclude tombstone rather than deleting it, and a plain sqlite3
integrity_check on a Plex library DB fails with "unknown tokenizer:
collating" because the stock binary lacks Plex's FTS collation - that
is not corruption.
ESXi port-group counts corrected (LAN_100 7->6, DATA_50 4->3), verified
by counting NICs on powered-on VMs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plex ran on a dedicated Debian 11 VM at 10.100.0.24 until that OS reached
LTS EOL on 2026-08-31 and apt security updates started 404ing. It now runs
as a host-networked container on the Docker host at 10.100.0.35.
Adds plex-docker-migration.md: the decision record and the executed
procedure. The two choices worth knowing about are that media is bind
mounted at the paths the library DB already stores (/mnt/nfs/*) so the
migration rewrote none of its 32,262 absolute paths, and that the
container runs on host networking so Plex still sees real client source
IPs and does not throttle LAN playback as remote.
Preferences.xml was carried across, so the server kept machine identifier
7aa65063599874eac3d9a2f62df74c1816400c88 and Tautulli, seerr and existing
clients continued working without reconfiguration. Verified after cutover:
libraries 1959/421/146 and 11,481 history entries, all matching the
pre-migration baseline exactly.
Rewrites plex.md for the container, adds plex to docker.md, and updates
the component list in CLAUDE.md. The old VM is retained powered on with
plexmediaserver stopped and disabled as rollback, so docs for ESXi and
Veeam are deliberately left describing it until decommission.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reverses the recent Infisical-pointer convention. Each <service>.md
holds its credentials inline under the Access section again. The
Infisical service itself still runs as a Docker stack on the docker
host — it just isn't the source of truth for these docs anymore.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
dohertj2