docs(deploy): vd03 + QA vq03/vq04 deployed to c9640f5c (#901 #902 #903 #834 #872 #904)

DEV vd03 6c4d2990 -> c9640f5c (~101 s), QA pair fea7de0a -> c9640f5c (~91 s pair
outage). #901 settings record written once per node, swaps gated on -Verify,
all gates pass (SL 250 / 250-240, failover no failback, LocalDb sync, canary,
no historian writes). #903 edit-and-revert leaves drift: RowVersion is in the
revision hash.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

docs/deploy/2026-11-13-next-vd03-qa-to-c9640f5c.md

@@ -1,6 +1,10 @@
 # Next deploy: DEV vd03 `6c4d2990` and QA vq03/vq04 `fea7de0a` → master `c9640f5c`
 
-> **Status: PREPARED, NOT EXECUTED (2026-11-13).** This pack covers both tiers, plus the read-only ConfigDb preflight,
+> **EXECUTED 2026-11-13** (DEV vd03 07:33Z, QA pair 07:47Z): record [`2026-11-13-vd03-qa-to-c9640f5c.md`](2026-11-13-vd03-qa-to-c9640f5c.md).
+> Deviations there: longer downtime from the in-swap `-Verify` coverage walk; the §5 #903 revert cannot return the chip to
+> "In sync" (`RowVersion` is in the revision hash); QA Secondary swapped before the Primary stopped.
+>
+> **Status when written: PREPARED, NOT EXECUTED (2026-11-13).** This pack covers both tiers, plus the read-only ConfigDb preflight,
 > which was run over direct SQL from the Mac on 2026-11-13. Nothing was built, staged or run on any wonder box. **No
 > blocker found; no issue filed.**
 >

docs/deploy/2026-11-13-vd03-qa-to-c9640f5c.md

@@ -0,0 +1,138 @@
+# DEV vd03 (`6c4d2990`) and QA vq03/vq04 (`fea7de0a`) upgrade → master `c9640f5c` (2026-11-13)
+
+> **Status: EXECUTED 2026-11-13.** DEV vd03 07:33–07:44Z; QA pair 07:44–07:54Z. All three nodes run **`c9640f5c`**
+> (`c9640f5ced383e2a722702409adbc62298731994`, #904). The owner approved the deploy on 2026-11-13, with WinRM for the
+> on-box steps. Runbook: [`2026-11-13-next-vd03-qa-to-c9640f5c.md`](2026-11-13-next-vd03-qa-to-c9640f5c.md) (the pack).
+> Deployed fixes: #834, #872, #900 (inert until HG ≥ `a220ec0`), #901, #902, #903, #904 (QA also gets the `fea7de0a..6c4d2990`
+> range vd03 received earlier the same day).
+
+Channel: WinRM + SMB from the Mac (the `winrm` skill), used only for the pack's on-box steps (prepare/backup, the one-time
+#901 settings record, stop/swap/start, `-Verify`, the read-only cleanup query for the #903 gate). Health, OPC UA, AdminUI,
+HG metrics, Overview and ScadaBridge checks ran from the Mac over HTTP / opc.tcp; logs and the live DLLs were read over SMB.
+No Galaxy writes, no historian writes or tag creation, no other service touched.
+
+## Build
+
+- `dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj -c Release -r win-x64 --self-contained true`
+  on the Mac (SDK 10.0.403), in a clean worktree at `c9640f5c`: 0 warnings, **1398 files** (1396 + `Grpc.StatusProto.dll` +
+  `Google.Api.CommonProtos.dll`, the #900 package), `OtOpcUa.Host.dll` ProductVersion `1.0.0+c9640f5ced383e2a…`,
+  `script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe` present.
+- `deps.json`: Telemetry(.Serilog) 0.3.1, LocalDb(.Replication) 0.3.7, MxGateway.Client 0.4.0, HistorianGateway.Client 1.0.0,
+  Grpc.StatusProto 2.76.0, Auth 0.4.1, Secrets 0.6.2, Health 0.3.0, Theme 0.5.0.
+- Zip `otopcua-c9640f5c.zip` (139,402,794 B), SHA-256 `12C46E60F596F7F5107274731FC2E189D2118F52882C48A7C1712130ACEDC878`,
+  checked on every box before expanding. `OtOpcUa.Host.dll` SHA-256 prefix `aff960bd2d36`.
+- Install scripts staged from `c9640f5c`; their SHA-256 matched the pack's table on every box (`7d4078cb…`, `ae9d84fd…`, `9498fd80…`).
+
+## DEV: wonder-app-vd03 (`6c4d2990` → `c9640f5c`)
+
+Stage `E:\ApiInstall\_stage-otopcua-c9640f5c\` (`prepare.ps1`, `iso.ps1`, `swap.ps1`, `rollback.ps1`, `envdigest.ps1`,
+`driftq.ps1`, zip + `.sha256`, `install\`).
+
+| Step | Result |
+|---|---|
+| Prepare (service up) | Backup folder `E:\ApiInstall\OtOpcUa-backups\pre-20261113-otopcua-c9640f5c\` created with a protected DACL (SYSTEM + Administrators); `OtOpcUaHost.reg` exported into it. Zip hash ok, 1398 files expanded to `E:\ApiInstall\OtOpcUa.new-c9640f5c`, ProductVersion checked. `BACKUP DATABASE … COPY_ONLY, CHECKSUM` → `OtOpcUaConfig-pre-20261113-otopcua-c9640f5c.bak` (instance default dir, 1913 pages). ConfigDb 9 migrations, last `20261022102459_EnableSnapshotIsolation`, snapshot ON. E: 178 GB free. |
+| Env (names + SHA-256 prefixes only) | **28 entries**, block SHA (ordered) `05db5d058fee`, (sorted) `e3f876a407a8`. `OtlpEndpoint=http://127.0.0.1:4317` (scheme present). `ServerHistorian__ProvisionTags`, `…EnableNativeTimeWeighted`, `Scripting__Runner` absent. |
+| One-time #901 record | `reg query …\OtOpcUaScriptWorker\Parameters /v DenyPaths` = exactly the 5 expected paths. `-WhatIf` (13 operations, no `HostServiceEnv`), then the full-argument install (`-InstallRoot E:\ApiInstall\OtOpcUa -HostServiceAccount LocalSystem -DenyPaths <the 5>`, no `-ConfigureHostService`): **`SettingsRecord` changed, the other 12 steps unchanged.** `-Verify`: **VERIFY OK 13/13, exit 0.** DenyPaths after: the same 5. |
+| Swap (`swap.ps1`) | Stop issued 07:33:50.6Z, stopped in 6 s. `OtOpcUa-data` robocopied into the backup folder. `Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -Candidate …\OtOpcUa.new-c9640f5c -PreviousInstall …\OtOpcUa.bak.20261113-pre-c9640f5c` (default carry): carried 5 `appsettings*.json`, `pki`, `logs`; `-Reapply` **changed `WorkerFolderAcl`, `InstallRootDeny`, `SensitiveDirDeny` (`pki`)**, the other 10 unchanged; `-Verify` **OK 13/13**; started. `/health/ready` 200 at 07:35:32Z. |
+| Downtime | **About 101 s** (stop → ready). Longer than the pack's 20–40 s estimate: `-Reapply` and `-Verify` each walk every denied folder for the protected-DACL coverage scan, and on vd03 that includes the whole of `E:\ApiInstall` (48 WARN items), all while the service is stopped. |
+| Post-start `-Verify` | **VERIFY OK 13/13, exit 0.** |
+| Env after | 28 entries, block SHA `05db5d058fee` / `e3f876a407a8`: **unchanged**. |
+
+### vd03 gates
+
+| Gate | Result |
+|---|---|
+| Version | Live `E:\ApiInstall\OtOpcUa\OtOpcUa.Host.dll` (SMB copy) ProductVersion `1.0.0+c9640f5ced383e2a…`; SHA-256 prefix `aff960bd2d36` = the published build. |
+| Health | `/healthz`, `/health/ready` (opcua-endpoint, script-runner, configdb, config-publish, akka Up, 1 member) and `/health/active` all 200. |
+| ServiceLevel | **250** on three fresh-session reads. |
+| FOCAS | 22 raw variables: **21 Good**, `Program/BlockCount` `BadNoData` (as before). 2 UNS variables. |
+| Boot log | `roles=[admin,driver]`; `Historian gateway probe: the gateway is reachable`; 12 aggregates advertised (Average, TimeAverage, Total, …); `applied plan (PureAdd, added=36 …)`; `raw subtree materialised (… tags=22 … failed=0)`; **`Credential-reference preflight (lmxopcua#815/#834): every stored and deployed reference is namespaced.`**; no FTL, no Telemetry endpoint error, no provisioning line. ERR lines only the pre-existing certificate-domain noise (from the gate's own CLI reads by IP). |
+| HG `/metrics` (`ea9f21c`) | 07:33:49Z → 07:37:43Z: `HistorianStatus/Probe` 589 → 594; `GetTagMetadata` 8, `ReadRaw` 283, `ReadAggregate` 6 unchanged. **No `EnsureTags`, `AddTagExtendedProperties`, `WriteLiveValues`, `AddHistoricalValues`, `SendEvent(s)`.** |
+| **#903 drift chip** | Prerendered `/deployments` HTML (curl) and the live page both show **"Checking for changes…"** first, then **"In sync"** (rev `fa9a2e28…`, deployment `595cb909`). Draft change on `/raw`: `parts-required` (`TAG-82e8944deef9`) WriteIdempotent off → on, saved, **no deploy**. Reload `/deployments`: "Checking for changes…" → **"Configuration drift"** with no interaction. **PASS.** See the deviation below for the revert. |
+| **#902 Monaco** | `/scripts/new` opened (editor loaded, 02:41:29 local), tab closed without saving. Log from 02:41:20 to 02:44:44 local (over 3 min, past circuit retention): **0 `MonacoEditor` / `Unhandled exception in circuit` lines, 0 `[ERR]`**. Control: the `6c4d2990` build logged that `[ERR]` for the same action earlier the same day. **PASS.** |
+| Overview `:5320` | 6 registered, 6 Up; OtOpcUa Up, 1 instance. |
+| ScadaBridge | `:8085/health/ready` 200 (no OtOpcUa link). |
+
+**Deviation: the #903 revert leaves the chip on "Configuration drift".** WriteIdempotent was switched back off and saved;
+the `dbo.Tag` row is back to its sealed content (`WriteIdempotent=0`, `PollGroupId` null, TagConfig
+`{"deviceHostAddress":"10.201.31.5:8193","address":"MACRO:3902","dataType":"Float64"}`, compared with the sealed artifact
+by a read-only on-box query). The chip still shows drift because `ConfigComposer` serialises each row's **`RowVersion`**
+into the artifact, and `RevisionHash` = SHA-256 of that blob, so any saved edit, even one reverted to identical content,
+changes the draft hash. The pack's "chip returns to In sync, revision still `fa9a2e28…`" cannot hold for an edit-and-revert.
+The **running configuration is unchanged** (nodes serve sealed `595cb909` / `fa9a2e28…`). No deployment was made to clear
+the chip. A "Deploy current configuration" would clear it with content-identical rows, but it would also create a new revision.
+That is left to the owner.
+
+## QA: wonder-app-vq03 + vq04 (`fea7de0a` → `c9640f5c`)
+
+Stage `E:\psrem-stage\otopcua-c9640f5c\` on each node (`prepare.ps1`, `iso.ps1`, `swap.ps1`, `start.ps1`, `rollback.ps1`,
+`envdigest.ps1`, zip + `.sha256`, `install\`); backups in `…\backup\`, created with a protected DACL.
+
+**Before the window:** ServiceLevel vq03 **240**, vq04 **250** (Primary vq04, as since the 2026-11-12 OTel restarts); both
+canary values Good and advancing.
+
+| Step | vq03 | vq04 |
+|---|---|---|
+| Prepare (services up) | 1398 files, hash + ProductVersion ok; `OtOpcUaHost-pre-c9640f5c.reg` exported; **DB:** `BACKUP DATABASE OtOpcUaConfig … COPY_ONLY, CHECKSUM, COMPRESSION` → `OtOpcUaConfig_pre-c9640f5c_20261113.bak` (wonder-sql-vq03 default dir, 1929 pages), **`RESTORE VERIFYONLY` valid**; 9 migrations, snapshot ON; latest sealed `5905003e` rev `82dd0598…`. E: 215 GB free. | same (no DB step); E: 230 GB free. |
+| Env | **48 entries**, ordered `692ee505f911` / sorted `78e6acfab539`. | **47 entries**, `41abda6b7dec` / `5df6eacf7152`. |
+| Env cross-node | Same `Security__ApiKeys__Pepper` (entry SHA prefix `ea9ddc525fc0`) and `LocalDb__Replication__ApiKey` (`1ce08c4554b0`) on both. Differences only in node-specific entries (`Cluster__PublicHostname`, `Cluster__SeedNodes__*` order, `OpcUa__ApplicationUri`, `OpcUa__PublicHostname`, vq03's `LocalDb__Replication__PeerAddress`). `Scripting__Runner=OutOfProcess`, seams Calculation + VirtualTag, `Scripting__Worker__*` present; `OtlpEndpoint=http://127.0.0.1:4317`. | |
+| One-time #901 record | `reg query` DenyPaths = exactly the **12** expected (OtelCollector included). `-WhatIf` 21 operations, then the full-argument install with `-ConfigureHostService` and the recorded 12: **`SettingsRecord` changed + `HostServiceEnv` changed (reorder only)**, 19 unchanged. Env 48 → 48, ordered SHA `692ee505f911` → `a04ca3f1fecb`, **sorted SHA `78e6acfab539` unchanged** (same entries, same values). `-Verify` **OK 21/21**. DenyPaths after: the same 12. | same: 2 changed (`SettingsRecord`, `HostServiceEnv` reorder), 47 → 47, ordered `41abda6b7dec` → `c6adaf68ef48`, sorted `5df6eacf7152` unchanged; **VERIFY OK 21/21**; 12 paths. |
+| Stop + backup + swap (`-NoStart`) | Stop 07:47:46Z (6 s). `data\`, `pki\` → `backup\{data,pki}-pre-c9640f5c`. Swap: carried 5 `appsettings*.json` + `logs`; `-Reapply` **changed `WorkerFolderAcl`, `InstallRootDeny`** (19 unchanged); **`-Verify` OK 21/21**; left stopped 07:48:40Z. | Stop 07:48:51Z (6 s), same backups, same carry; `-Reapply` same 2 changed; **`-Verify` OK 21/21**; left stopped 07:49:57Z. |
+| Start | 07:50:13Z → ready 07:50:22Z (9 s). | 07:50:05Z (first, the old Primary) → ready 07:50:22Z (16 s). |
+| Post-start `-Verify` | **OK 21/21, exit 0.** Env 48, `a04ca3f1fecb` / `78e6acfab539`. | **OK 21/21, exit 0.** Env 47, `c6adaf68ef48` / `5df6eacf7152`. |
+
+- **Pair outage: about 91 s** (vq04 stop 07:48:51Z → both ready 07:50:22Z). vq03 was down 07:47:46–07:50:22Z while vq04 still served.
+- **Order deviation (deliberate):** the pack stops both and then swaps both. Here vq03 (Secondary) was stopped, backed up
+  and swapped (`-NoStart`, gated on `-Verify`) while vq04 kept serving, and only then was vq04 stopped and swapped. No node
+  ran the new build until both were swapped and verified, and a `-Verify` failure on vq03 would have left the old Primary
+  serving. Each node's swap took about 50–60 s of `-Reapply`/`-Verify`, so this kept that time out of the pair outage.
+- **Primary after the start: vq03**, although vq04 was started 8 s earlier. Both came Up together, and vq03 ended up the
+  oldest member. After the failover gate below, Primary is **vq04 again**, the same as before the window.
+
+### QA gates
+
+| Gate | Result |
+|---|---|
+| Isolation | `-Verify` OK 21/21 (exit 0) after each swap and after each start, both nodes. |
+| Version | Live `E:\ApiInstall\OtOpcUa\app\OtOpcUa.Host.dll` on both (SMB copy): SHA-256 prefix `aff960bd2d36` = the published build; ProductVersion `1.0.0+c9640f5ced383e2a…`. |
+| Health | `/health/ready` 200 on both, 5/5 components: opcua-endpoint, **`Out-of-process script runner ready.`**, configdb, config-publish, akka Up. `/health/active` 200 on the Primary, 503 on the Secondary. |
+| ServiceLevel | After the start: vq03 **250** ×3, vq04 **240** ×3 (fresh sessions). |
+| **Failover** | `Stop-Service` on the Primary vq03 at 07:51:43Z (stopped in 1 s). vq04: **250, `/health/active` 200 from the first sample** (07:51:46Z) and on all 4 samples. vq03 restarted 07:52:11Z → ready 07:52:19Z (8 s), `-Verify` OK 21/21. Then 6 samples over 40 s: **vq03 240 / 503, vq04 250 / 200**: never above 240, **no failback**. |
+| LocalDb sync | `localdb-replication`: **connected, oplog backlog 0** on both, before and after the failover. |
+| Canary | `ns=2;s=script-runner-canary/Device1/canary-calc-seconds` and `ns=3;s=EQ-ec3979550d90/canary-vt-second`: **Good and advancing about once a second on both endpoints**, right after the start and again after the failover. `otopcua_script_worker_up{shard="0"}` = 1 and `{shard="1"}` = 1 on both (`{shard="iso"}` = 0, idle); `otopcua_script_runner_state` 0. The optional #780 runaway drill was not run. |
+| Boot log (both, both starts on vq03) | `roles=[admin,driver]`; `Historian gateway probe: the gateway is reachable`; 12 aggregates advertised; `raw subtree materialised (containers=2, tags=1, … failed=0)`; **`Credential-reference preflight (lmxopcua#815/#834): every stored and deployed reference is namespaced.`**; no FTL, no Telemetry endpoint error. ERR lines only: `/health/*` 503 request logs, the standby `cluster-primary` check, and vq04's bootstrap `akka … Removed` before the join (the same pattern as the 2026-11-12 record). 0 other ERR. |
+| HG `/metrics` (vq03 `:5220`, `fa9cf31`) | 07:47:44Z → 07:54:19Z: `HistorianStatus/Probe` 1194 → 1214, nothing else. **No historian write or tag-creation RPC.** |
+| Overview (vq03 + vq04 `:5320`) | 12 registered, 12 Up; OtOpcUa **Up**, 2 instances, **2 members / 0 unreachable**. |
+| ScadaBridge | `:8085/health/ready` 200 on both (no OtOpcUa link). |
+| #902 / #903 on QA | Optional in the pack; not run (they need the temporary QA AdminUI role-mapping row). Verified on vd03. |
+| `serilog-selflog.txt` (vq04) | Two `RollingFileSink … disposed` lines, 2026-11-12 21:54Z and 2026-11-13 02:48Z: shutdown-time emits from earlier stops, not from this window. |
+
+## Rollback material (left on the boxes)
+
+**DEV vd03 → `6c4d2990`:** `E:\ApiInstall\_stage-otopcua-c9640f5c\rollback.ps1` (pack §6: `Invoke-OtOpcUaInstallSwap.ps1
+-Candidate E:\ApiInstall\OtOpcUa.bak.20261113-pre-c9640f5c -PreviousInstall E:\ApiInstall\OtOpcUa.failed-c9640f5c -NoCarry
+-NoStart`, mirror `OtOpcUa-data` back, start, `-Verify`).
+- Previous install: `E:\ApiInstall\OtOpcUa.bak.20261113-pre-c9640f5c`.
+- Backups: `E:\ApiInstall\OtOpcUa-backups\pre-20261113-otopcua-c9640f5c\{OtOpcUaHost.reg, OtOpcUa-data\}` (protected).
+- ConfigDb: `OtOpcUaConfig-pre-20261113-otopcua-c9640f5c.bak` (vd03 instance default backup dir).
+- The registry did not change, so no import is needed.
+
+**QA → `fea7de0a`:** stop both, then run `E:\psrem-stage\otopcua-c9640f5c\rollback.ps1` on each node (the swap back with
+`-NoCarry -NoStart` and `-Reapply`/`-Verify` inside, then a mirror of `data\` from the backup). Start both after that, the
+old Primary first.
+- Previous install: `E:\ApiInstall\OtOpcUa\app.bak.20261113-pre-c9640f5c` on each node.
+- Backups: `E:\psrem-stage\otopcua-c9640f5c\backup\{OtOpcUaHost-pre-c9640f5c.reg, data-pre-c9640f5c\, pki-pre-c9640f5c\}`
+  (protected). The `.reg` import is not needed, because the env changed only in order.
+- ConfigDb: `OtOpcUaConfig_pre-c9640f5c_20261113.bak` on wonder-sql-vq03 (VERIFYONLY ok).
+- The #901 settings record and the denies stay valid for `fea7de0a`.
+
+## Deviations from the pack (summary)
+
+1. vd03 downtime about 101 s against an estimate of 20–40 s, because the coverage walk runs inside `-Reapply`/`-Verify`
+   while the service is stopped. QA swaps took about 50–60 s per node for the same reason.
+2. #903: the edit-and-revert cannot return the chip to "In sync", because `RowVersion` is part of the revision hash (see
+   above). Draft content is restored; the chip shows drift until the next deploy.
+3. QA order: the Secondary was stopped and swapped while the Primary still served, then the Primary. Both started
+   together. Primary roles: vq04 → vq03 after the start → vq04 after the failover gate.
+4. vd03 #903 draft change: there is no description field on the tag form, so the harmless change was the WriteIdempotent
+   toggle on a read-only FOCAS tag.