Files
lmxopcua/docs/deploy/2026-11-13-vd03-qa-to-c9640f5c.md
T
Joseph Doherty 0c19f26c81
v2-ci / unit-tests (push) Has been cancelled
fix(install): cut the swap's stopped window to the swapped-in folder (#905)
Root cause: the protected-DACL coverage scan, not the ACL writes. Both -Reapply and
-Verify walked every denied folder (all of E:\ApiInstall on vd03, nested roots again)
with Get-ChildItem -Recurse + Get-Acl per item + a pipeline per item for -Skip, about
3.4 ms an item on PS 5.1, all while the service was stopped. An unchanged deny is read,
not rewritten; the re-applied install-folder deny propagates over ~1,400 files in 0.5 s.

- Find-OtOpcUaDenyCoverageGap: .NET enumeration, -Skip folders pruned, DACL-only
  GetAccessControl(Access) (about 0.15 ms an item); reparse points not followed.
- Get-OtOpcUaCoverageWalkRoots: each denied folder walked once (outermost only).
- Install-ScriptWorkerIsolation.ps1 -Scope All|InstallRoot|Outside (with -Reapply,
  -Verify, -PlanOnly). InstallRoot = the folder ACEs a swap drops; Outside = the rest,
  including the firewall rule (its query costs 3-5 s; its exe path does not change).
  The two partition the plan. Prints the scan time.
- Invoke-OtOpcUaInstallSwap.ps1: -Reapply/-Verify -Scope Outside before the stop (a
  failure stops before anything moves), -Scope InstallRoot inside the window; preflight
  checks the candidate carries the worker exe; prints the stopped window and its parts.
  -NoStart, -NoCarry and the pre-#901 refusal unchanged.

windev, vd03-like tree (17,851 items): stopped window 153.1 s -> 6.0 s (PS 5.1),
101.6 s -> 6.2 s (PS 7); the Outside part takes ~23 s with the old build serving.
Dry-run harness 101 -> 123 passed; Windows test 72 passed on PS 5.1 and PS 7.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 03:51:42 -05:00

141 lines
16 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# DEV vd03 (`6c4d2990`) and QA vq03/vq04 (`fea7de0a`) upgrade → master `c9640f5c` (2026-11-13)
> **Status: EXECUTED 2026-11-13.** DEV vd03 07:33–07:44Z; QA pair 07:44–07:54Z. All three nodes run **`c9640f5c`**
> (`c9640f5ced383e2a722702409adbc62298731994`, #904). The owner approved the deploy on 2026-11-13, with WinRM for the
> on-box steps. Runbook: [`2026-11-13-next-vd03-qa-to-c9640f5c.md`](2026-11-13-next-vd03-qa-to-c9640f5c.md) (the pack).
> Deployed fixes: #834, #872, #900 (inert until HG ≥ `a220ec0`), #901, #902, #903, #904 (QA also gets the `fea7de0a..6c4d2990`
> range vd03 received earlier the same day).
Channel: WinRM + SMB from the Mac (the `winrm` skill), used only for the pack's on-box steps (prepare/backup, the one-time
#901 settings record, stop/swap/start, `-Verify`, the read-only cleanup query for the #903 gate). Health, OPC UA, AdminUI,
HG metrics, Overview and ScadaBridge checks ran from the Mac over HTTP / opc.tcp; logs and the live DLLs were read over SMB.
No Galaxy writes, no historian writes or tag creation, no other service touched.
## Build
- `dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj -c Release -r win-x64 --self-contained true`
on the Mac (SDK 10.0.403), in a clean worktree at `c9640f5c`: 0 warnings, **1398 files** (1396 + `Grpc.StatusProto.dll` +
`Google.Api.CommonProtos.dll`, the #900 package), `OtOpcUa.Host.dll` ProductVersion `1.0.0+c9640f5ced383e2a…`,
`script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe` present.
- `deps.json`: Telemetry(.Serilog) 0.3.1, LocalDb(.Replication) 0.3.7, MxGateway.Client 0.4.0, HistorianGateway.Client 1.0.0,
Grpc.StatusProto 2.76.0, Auth 0.4.1, Secrets 0.6.2, Health 0.3.0, Theme 0.5.0.
- Zip `otopcua-c9640f5c.zip` (139,402,794 B), SHA-256 `12C46E60F596F7F5107274731FC2E189D2118F52882C48A7C1712130ACEDC878`,
checked on every box before expanding. `OtOpcUa.Host.dll` SHA-256 prefix `aff960bd2d36`.
- Install scripts staged from `c9640f5c`; their SHA-256 matched the pack's table on every box (`7d4078cb…`, `ae9d84fd…`, `9498fd80…`).
## DEV: wonder-app-vd03 (`6c4d2990` → `c9640f5c`)
Stage `E:\ApiInstall\_stage-otopcua-c9640f5c\` (`prepare.ps1`, `iso.ps1`, `swap.ps1`, `rollback.ps1`, `envdigest.ps1`,
`driftq.ps1`, zip + `.sha256`, `install\`).
| Step | Result |
|---|---|
| Prepare (service up) | Backup folder `E:\ApiInstall\OtOpcUa-backups\pre-20261113-otopcua-c9640f5c\` created with a protected DACL (SYSTEM + Administrators); `OtOpcUaHost.reg` exported into it. Zip hash ok, 1398 files expanded to `E:\ApiInstall\OtOpcUa.new-c9640f5c`, ProductVersion checked. `BACKUP DATABASE … COPY_ONLY, CHECKSUM` → `OtOpcUaConfig-pre-20261113-otopcua-c9640f5c.bak` (instance default dir, 1913 pages). ConfigDb 9 migrations, last `20261022102459_EnableSnapshotIsolation`, snapshot ON. E: 178 GB free. |
| Env (names + SHA-256 prefixes only) | **28 entries**, block SHA (ordered) `05db5d058fee`, (sorted) `e3f876a407a8`. `OtlpEndpoint=http://127.0.0.1:4317` (scheme present). `ServerHistorian__ProvisionTags`, `…EnableNativeTimeWeighted`, `Scripting__Runner` absent. |
| One-time #901 record | `reg query …\OtOpcUaScriptWorker\Parameters /v DenyPaths` = exactly the 5 expected paths. `-WhatIf` (13 operations, no `HostServiceEnv`), then the full-argument install (`-InstallRoot E:\ApiInstall\OtOpcUa -HostServiceAccount LocalSystem -DenyPaths <the 5>`, no `-ConfigureHostService`): **`SettingsRecord` changed, the other 12 steps unchanged.** `-Verify`: **VERIFY OK 13/13, exit 0.** DenyPaths after: the same 5. |
| Swap (`swap.ps1`) | Stop issued 07:33:50.6Z, stopped in 6 s. `OtOpcUa-data` robocopied into the backup folder. `Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -Candidate …\OtOpcUa.new-c9640f5c -PreviousInstall …\OtOpcUa.bak.20261113-pre-c9640f5c` (default carry): carried 5 `appsettings*.json`, `pki`, `logs`; `-Reapply` **changed `WorkerFolderAcl`, `InstallRootDeny`, `SensitiveDirDeny` (`pki`)**, the other 10 unchanged; `-Verify` **OK 13/13**; started. `/health/ready` 200 at 07:35:32Z. |
| Downtime | **About 101 s** (stop → ready). Longer than the pack's 20–40 s estimate: `-Reapply` and `-Verify` each walk every denied folder for the protected-DACL coverage scan, and on vd03 that includes the whole of `E:\ApiInstall` (48 WARN items), all while the service is stopped. |
| Post-start `-Verify` | **VERIFY OK 13/13, exit 0.** |
| Env after | 28 entries, block SHA `05db5d058fee` / `e3f876a407a8`: **unchanged**. |
### vd03 gates
| Gate | Result |
|---|---|
| Version | Live `E:\ApiInstall\OtOpcUa\OtOpcUa.Host.dll` (SMB copy) ProductVersion `1.0.0+c9640f5ced383e2a…`; SHA-256 prefix `aff960bd2d36` = the published build. |
| Health | `/healthz`, `/health/ready` (opcua-endpoint, script-runner, configdb, config-publish, akka Up, 1 member) and `/health/active` all 200. |
| ServiceLevel | **250** on three fresh-session reads. |
| FOCAS | 22 raw variables: **21 Good**, `Program/BlockCount` `BadNoData` (as before). 2 UNS variables. |
| Boot log | `roles=[admin,driver]`; `Historian gateway probe: the gateway is reachable`; 12 aggregates advertised (Average, TimeAverage, Total, …); `applied plan (PureAdd, added=36 …)`; `raw subtree materialised (… tags=22 … failed=0)`; **`Credential-reference preflight (lmxopcua#815/#834): every stored and deployed reference is namespaced.`**; no FTL, no Telemetry endpoint error, no provisioning line. ERR lines only the pre-existing certificate-domain noise (from the gate's own CLI reads by IP). |
| HG `/metrics` (`ea9f21c`) | 07:33:49Z → 07:37:43Z: `HistorianStatus/Probe` 589 → 594; `GetTagMetadata` 8, `ReadRaw` 283, `ReadAggregate` 6 unchanged. **No `EnsureTags`, `AddTagExtendedProperties`, `WriteLiveValues`, `AddHistoricalValues`, `SendEvent(s)`.** |
| **#903 drift chip** | Prerendered `/deployments` HTML (curl) and the live page both show **"Checking for changes…"** first, then **"In sync"** (rev `fa9a2e28…`, deployment `595cb909`). Draft change on `/raw`: `parts-required` (`TAG-82e8944deef9`) WriteIdempotent off → on, saved, **no deploy**. Reload `/deployments`: "Checking for changes…" → **"Configuration drift"** with no interaction. **PASS.** See the deviation below for the revert. |
| **#902 Monaco** | `/scripts/new` opened (editor loaded, 02:41:29 local), tab closed without saving. Log from 02:41:20 to 02:44:44 local (over 3 min, past circuit retention): **0 `MonacoEditor` / `Unhandled exception in circuit` lines, 0 `[ERR]`**. Control: the `6c4d2990` build logged that `[ERR]` for the same action earlier the same day. **PASS.** |
| Overview `:5320` | 6 registered, 6 Up; OtOpcUa Up, 1 instance. |
| ScadaBridge | `:8085/health/ready` 200 (no OtOpcUa link). |
**Deviation: the #903 revert leaves the chip on "Configuration drift".** WriteIdempotent was switched back off and saved;
the `dbo.Tag` row is back to its sealed content (`WriteIdempotent=0`, `PollGroupId` null, TagConfig
`{"deviceHostAddress":"10.201.31.5:8193","address":"MACRO:3902","dataType":"Float64"}`, compared with the sealed artifact
by a read-only on-box query). The chip still shows drift because `ConfigComposer` serialises each row's **`RowVersion`**
into the artifact, and `RevisionHash` = SHA-256 of that blob, so any saved edit, even one reverted to identical content,
changes the draft hash. The pack's "chip returns to In sync, revision still `fa9a2e28…`" cannot hold for an edit-and-revert.
The **running configuration is unchanged** (nodes serve sealed `595cb909` / `fa9a2e28…`). No deployment was made to clear
the chip. A "Deploy current configuration" would clear it with content-identical rows, but it would also create a new revision.
That is left to the owner.
## QA: wonder-app-vq03 + vq04 (`fea7de0a` → `c9640f5c`)
Stage `E:\psrem-stage\otopcua-c9640f5c\` on each node (`prepare.ps1`, `iso.ps1`, `swap.ps1`, `start.ps1`, `rollback.ps1`,
`envdigest.ps1`, zip + `.sha256`, `install\`); backups in `…\backup\`, created with a protected DACL.
**Before the window:** ServiceLevel vq03 **240**, vq04 **250** (Primary vq04, as since the 2026-11-12 OTel restarts); both
canary values Good and advancing.
| Step | vq03 | vq04 |
|---|---|---|
| Prepare (services up) | 1398 files, hash + ProductVersion ok; `OtOpcUaHost-pre-c9640f5c.reg` exported; **DB:** `BACKUP DATABASE OtOpcUaConfig … COPY_ONLY, CHECKSUM, COMPRESSION` → `OtOpcUaConfig_pre-c9640f5c_20261113.bak` (wonder-sql-vq03 default dir, 1929 pages), **`RESTORE VERIFYONLY` valid**; 9 migrations, snapshot ON; latest sealed `5905003e` rev `82dd0598…`. E: 215 GB free. | same (no DB step); E: 230 GB free. |
| Env | **48 entries**, ordered `692ee505f911` / sorted `78e6acfab539`. | **47 entries**, `41abda6b7dec` / `5df6eacf7152`. |
| Env cross-node | Same `Security__ApiKeys__Pepper` (entry SHA prefix `ea9ddc525fc0`) and `LocalDb__Replication__ApiKey` (`1ce08c4554b0`) on both. Differences only in node-specific entries (`Cluster__PublicHostname`, `Cluster__SeedNodes__*` order, `OpcUa__ApplicationUri`, `OpcUa__PublicHostname`, vq03's `LocalDb__Replication__PeerAddress`). `Scripting__Runner=OutOfProcess`, seams Calculation + VirtualTag, `Scripting__Worker__*` present; `OtlpEndpoint=http://127.0.0.1:4317`. | |
| One-time #901 record | `reg query` DenyPaths = exactly the **12** expected (OtelCollector included). `-WhatIf` 21 operations, then the full-argument install with `-ConfigureHostService` and the recorded 12: **`SettingsRecord` changed + `HostServiceEnv` changed (reorder only)**, 19 unchanged. Env 48 → 48, ordered SHA `692ee505f911` → `a04ca3f1fecb`, **sorted SHA `78e6acfab539` unchanged** (same entries, same values). `-Verify` **OK 21/21**. DenyPaths after: the same 12. | same: 2 changed (`SettingsRecord`, `HostServiceEnv` reorder), 47 → 47, ordered `41abda6b7dec` → `c6adaf68ef48`, sorted `5df6eacf7152` unchanged; **VERIFY OK 21/21**; 12 paths. |
| Stop + backup + swap (`-NoStart`) | Stop 07:47:46Z (6 s). `data\`, `pki\` → `backup\{data,pki}-pre-c9640f5c`. Swap: carried 5 `appsettings*.json` + `logs`; `-Reapply` **changed `WorkerFolderAcl`, `InstallRootDeny`** (19 unchanged); **`-Verify` OK 21/21**; left stopped 07:48:40Z. | Stop 07:48:51Z (6 s), same backups, same carry; `-Reapply` same 2 changed; **`-Verify` OK 21/21**; left stopped 07:49:57Z. |
| Start | 07:50:13Z → ready 07:50:22Z (9 s). | 07:50:05Z (first, the old Primary) → ready 07:50:22Z (16 s). |
| Post-start `-Verify` | **OK 21/21, exit 0.** Env 48, `a04ca3f1fecb` / `78e6acfab539`. | **OK 21/21, exit 0.** Env 47, `c6adaf68ef48` / `5df6eacf7152`. |
- **Pair outage: about 91 s** (vq04 stop 07:48:51Z → both ready 07:50:22Z). vq03 was down 07:47:46–07:50:22Z while vq04 still served.
- **Order deviation (deliberate):** the pack stops both and then swaps both. Here vq03 (Secondary) was stopped, backed up
and swapped (`-NoStart`, gated on `-Verify`) while vq04 kept serving, and only then was vq04 stopped and swapped. No node
ran the new build until both were swapped and verified, and a `-Verify` failure on vq03 would have left the old Primary
serving. Each node's swap took about 50–60 s of `-Reapply`/`-Verify`, so this kept that time out of the pair outage.
- **Primary after the start: vq03**, although vq04 was started 8 s earlier. Both came Up together, and vq03 ended up the
oldest member. After the failover gate below, Primary is **vq04 again**, the same as before the window.
### QA gates
| Gate | Result |
|---|---|
| Isolation | `-Verify` OK 21/21 (exit 0) after each swap and after each start, both nodes. |
| Version | Live `E:\ApiInstall\OtOpcUa\app\OtOpcUa.Host.dll` on both (SMB copy): SHA-256 prefix `aff960bd2d36` = the published build; ProductVersion `1.0.0+c9640f5ced383e2a…`. |
| Health | `/health/ready` 200 on both, 5/5 components: opcua-endpoint, **`Out-of-process script runner ready.`**, configdb, config-publish, akka Up. `/health/active` 200 on the Primary, 503 on the Secondary. |
| ServiceLevel | After the start: vq03 **250** ×3, vq04 **240** ×3 (fresh sessions). |
| **Failover** | `Stop-Service` on the Primary vq03 at 07:51:43Z (stopped in 1 s). vq04: **250, `/health/active` 200 from the first sample** (07:51:46Z) and on all 4 samples. vq03 restarted 07:52:11Z → ready 07:52:19Z (8 s), `-Verify` OK 21/21. Then 6 samples over 40 s: **vq03 240 / 503, vq04 250 / 200**: never above 240, **no failback**. |
| LocalDb sync | `localdb-replication`: **connected, oplog backlog 0** on both, before and after the failover. |
| Canary | `ns=2;s=script-runner-canary/Device1/canary-calc-seconds` and `ns=3;s=EQ-ec3979550d90/canary-vt-second`: **Good and advancing about once a second on both endpoints**, right after the start and again after the failover. `otopcua_script_worker_up{shard="0"}` = 1 and `{shard="1"}` = 1 on both (`{shard="iso"}` = 0, idle); `otopcua_script_runner_state` 0. The optional #780 runaway drill was not run. |
| Boot log (both, both starts on vq03) | `roles=[admin,driver]`; `Historian gateway probe: the gateway is reachable`; 12 aggregates advertised; `raw subtree materialised (containers=2, tags=1, … failed=0)`; **`Credential-reference preflight (lmxopcua#815/#834): every stored and deployed reference is namespaced.`**; no FTL, no Telemetry endpoint error. ERR lines only: `/health/*` 503 request logs, the standby `cluster-primary` check, and vq04's bootstrap `akka … Removed` before the join (the same pattern as the 2026-11-12 record). 0 other ERR. |
| HG `/metrics` (vq03 `:5220`, `fa9cf31`) | 07:47:44Z → 07:54:19Z: `HistorianStatus/Probe` 1194 → 1214, nothing else. **No historian write or tag-creation RPC.** |
| Overview (vq03 + vq04 `:5320`) | 12 registered, 12 Up; OtOpcUa **Up**, 2 instances, **2 members / 0 unreachable**. |
| ScadaBridge | `:8085/health/ready` 200 on both (no OtOpcUa link). |
| #902 / #903 on QA | Optional in the pack; not run (they need the temporary QA AdminUI role-mapping row). Verified on vd03. |
| `serilog-selflog.txt` (vq04) | Two `RollingFileSink … disposed` lines, 2026-11-12 21:54Z and 2026-11-13 02:48Z: shutdown-time emits from earlier stops, not from this window. |
## Rollback material (left on the boxes)
**DEV vd03 → `6c4d2990`:** `E:\ApiInstall\_stage-otopcua-c9640f5c\rollback.ps1` (pack §6: `Invoke-OtOpcUaInstallSwap.ps1
-Candidate E:\ApiInstall\OtOpcUa.bak.20261113-pre-c9640f5c -PreviousInstall E:\ApiInstall\OtOpcUa.failed-c9640f5c -NoCarry
-NoStart`, mirror `OtOpcUa-data` back, start, `-Verify`).
- Previous install: `E:\ApiInstall\OtOpcUa.bak.20261113-pre-c9640f5c`.
- Backups: `E:\ApiInstall\OtOpcUa-backups\pre-20261113-otopcua-c9640f5c\{OtOpcUaHost.reg, OtOpcUa-data\}` (protected).
- ConfigDb: `OtOpcUaConfig-pre-20261113-otopcua-c9640f5c.bak` (vd03 instance default backup dir).
- The registry did not change, so no import is needed.
**QA → `fea7de0a`:** stop both, then run `E:\psrem-stage\otopcua-c9640f5c\rollback.ps1` on each node (the swap back with
`-NoCarry -NoStart` and `-Reapply`/`-Verify` inside, then a mirror of `data\` from the backup). Start both after that, the
old Primary first.
- Previous install: `E:\ApiInstall\OtOpcUa\app.bak.20261113-pre-c9640f5c` on each node.
- Backups: `E:\psrem-stage\otopcua-c9640f5c\backup\{OtOpcUaHost-pre-c9640f5c.reg, data-pre-c9640f5c\, pki-pre-c9640f5c\}`
(protected). The `.reg` import is not needed, because the env changed only in order.
- ConfigDb: `OtOpcUaConfig_pre-c9640f5c_20261113.bak` on wonder-sql-vq03 (VERIFYONLY ok).
- The #901 settings record and the denies stay valid for `fea7de0a`.
## Deviations from the pack (summary)
1. vd03 downtime about 101 s against an estimate of 20–40 s, because the coverage walk runs inside `-Reapply`/`-Verify`
while the service is stopped. QA swaps took about 50–60 s per node for the same reason.
→ lmxopcua#905: fixed (scan about 20× faster per item, walked once, and only the swapped-in folder inside the
window; windev 17.9k items: stopped window 153 s → 6 s on PS 5.1, 102 s → 6 s on PS 7).
2. #903: the edit-and-revert cannot return the chip to "In sync", because `RowVersion` is part of the revision hash (see
above). Draft content is restored; the chip shows drift until the next deploy.
3. QA order: the Secondary was stopped and swapped while the Primary still served, then the Primary. Both started
together. Primary roles: vq04 → vq03 after the start → vq04 after the failover gate.
4. vd03 #903 draft change: there is no description field on the tag form, so the harmless change was the WriteIdempotent
toggle on a read-only FOCAS tag.