DEV vd036c4d2990->c9640f5c(~101 s), QA pairfea7de0a->c9640f5c(~91 s pair outage). #901 settings record written once per node, swaps gated on -Verify, all gates pass (SL 250 / 250-240, failover no failback, LocalDb sync, canary, no historian writes). #903 edit-and-revert leaves drift: RowVersion is in the revision hash. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
16 KiB
DEV vd03 (6c4d2990) and QA vq03/vq04 (fea7de0a) upgrade → master c9640f5c (2026-11-13)
Status: EXECUTED 2026-11-13. DEV vd03 07:33–07:44Z; QA pair 07:44–07:54Z. All three nodes run
c9640f5c(c9640f5ced383e2a722702409adbc62298731994, #904). The owner approved the deploy on 2026-11-13, with WinRM for the on-box steps. Runbook:2026-11-13-next-vd03-qa-to-c9640f5c.md(the pack). Deployed fixes: #834, #872, #900 (inert until HG ≥a220ec0), #901, #902, #903, #904 (QA also gets thefea7de0a..6c4d2990range vd03 received earlier the same day).
Channel: WinRM + SMB from the Mac (the winrm skill), used only for the pack's on-box steps (prepare/backup, the one-time
#901 settings record, stop/swap/start, -Verify, the read-only cleanup query for the #903 gate). Health, OPC UA, AdminUI,
HG metrics, Overview and ScadaBridge checks ran from the Mac over HTTP / opc.tcp; logs and the live DLLs were read over SMB.
No Galaxy writes, no historian writes or tag creation, no other service touched.
Build
dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj -c Release -r win-x64 --self-contained trueon the Mac (SDK 10.0.403), in a clean worktree atc9640f5c: 0 warnings, 1398 files (1396 +Grpc.StatusProto.dll+Google.Api.CommonProtos.dll, the #900 package),OtOpcUa.Host.dllProductVersion1.0.0+c9640f5ced383e2a…,script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exepresent.deps.json: Telemetry(.Serilog) 0.3.1, LocalDb(.Replication) 0.3.7, MxGateway.Client 0.4.0, HistorianGateway.Client 1.0.0, Grpc.StatusProto 2.76.0, Auth 0.4.1, Secrets 0.6.2, Health 0.3.0, Theme 0.5.0.- Zip
otopcua-c9640f5c.zip(139,402,794 B), SHA-25612C46E60F596F7F5107274731FC2E189D2118F52882C48A7C1712130ACEDC878, checked on every box before expanding.OtOpcUa.Host.dllSHA-256 prefixaff960bd2d36. - Install scripts staged from
c9640f5c; their SHA-256 matched the pack's table on every box (7d4078cb…,ae9d84fd…,9498fd80…).
DEV: wonder-app-vd03 (6c4d2990 → c9640f5c)
Stage E:\ApiInstall\_stage-otopcua-c9640f5c\ (prepare.ps1, iso.ps1, swap.ps1, rollback.ps1, envdigest.ps1,
driftq.ps1, zip + .sha256, install\).
| Step | Result |
|---|---|
| Prepare (service up) | Backup folder E:\ApiInstall\OtOpcUa-backups\pre-20261113-otopcua-c9640f5c\ created with a protected DACL (SYSTEM + Administrators); OtOpcUaHost.reg exported into it. Zip hash ok, 1398 files expanded to E:\ApiInstall\OtOpcUa.new-c9640f5c, ProductVersion checked. BACKUP DATABASE … COPY_ONLY, CHECKSUM → OtOpcUaConfig-pre-20261113-otopcua-c9640f5c.bak (instance default dir, 1913 pages). ConfigDb 9 migrations, last 20261022102459_EnableSnapshotIsolation, snapshot ON. E: 178 GB free. |
| Env (names + SHA-256 prefixes only) | 28 entries, block SHA (ordered) 05db5d058fee, (sorted) e3f876a407a8. OtlpEndpoint=http://127.0.0.1:4317 (scheme present). ServerHistorian__ProvisionTags, …EnableNativeTimeWeighted, Scripting__Runner absent. |
| One-time #901 record | reg query …\OtOpcUaScriptWorker\Parameters /v DenyPaths = exactly the 5 expected paths. -WhatIf (13 operations, no HostServiceEnv), then the full-argument install (-InstallRoot E:\ApiInstall\OtOpcUa -HostServiceAccount LocalSystem -DenyPaths <the 5>, no -ConfigureHostService): SettingsRecord changed, the other 12 steps unchanged. -Verify: VERIFY OK 13/13, exit 0. DenyPaths after: the same 5. |
Swap (swap.ps1) |
Stop issued 07:33:50.6Z, stopped in 6 s. OtOpcUa-data robocopied into the backup folder. Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -Candidate …\OtOpcUa.new-c9640f5c -PreviousInstall …\OtOpcUa.bak.20261113-pre-c9640f5c (default carry): carried 5 appsettings*.json, pki, logs; -Reapply changed WorkerFolderAcl, InstallRootDeny, SensitiveDirDeny (pki), the other 10 unchanged; -Verify OK 13/13; started. /health/ready 200 at 07:35:32Z. |
| Downtime | About 101 s (stop → ready). Longer than the pack's 20–40 s estimate: -Reapply and -Verify each walk every denied folder for the protected-DACL coverage scan, and on vd03 that includes the whole of E:\ApiInstall (48 WARN items), all while the service is stopped. |
Post-start -Verify |
VERIFY OK 13/13, exit 0. |
| Env after | 28 entries, block SHA 05db5d058fee / e3f876a407a8: unchanged. |
vd03 gates
| Gate | Result |
|---|---|
| Version | Live E:\ApiInstall\OtOpcUa\OtOpcUa.Host.dll (SMB copy) ProductVersion 1.0.0+c9640f5ced383e2a…; SHA-256 prefix aff960bd2d36 = the published build. |
| Health | /healthz, /health/ready (opcua-endpoint, script-runner, configdb, config-publish, akka Up, 1 member) and /health/active all 200. |
| ServiceLevel | 250 on three fresh-session reads. |
| FOCAS | 22 raw variables: 21 Good, Program/BlockCount BadNoData (as before). 2 UNS variables. |
| Boot log | roles=[admin,driver]; Historian gateway probe: the gateway is reachable; 12 aggregates advertised (Average, TimeAverage, Total, …); applied plan (PureAdd, added=36 …); raw subtree materialised (… tags=22 … failed=0); Credential-reference preflight (lmxopcua#815/#834): every stored and deployed reference is namespaced.; no FTL, no Telemetry endpoint error, no provisioning line. ERR lines only the pre-existing certificate-domain noise (from the gate's own CLI reads by IP). |
HG /metrics (ea9f21c) |
07:33:49Z → 07:37:43Z: HistorianStatus/Probe 589 → 594; GetTagMetadata 8, ReadRaw 283, ReadAggregate 6 unchanged. No EnsureTags, AddTagExtendedProperties, WriteLiveValues, AddHistoricalValues, SendEvent(s). |
| #903 drift chip | Prerendered /deployments HTML (curl) and the live page both show "Checking for changes…" first, then "In sync" (rev fa9a2e28…, deployment 595cb909). Draft change on /raw: parts-required (TAG-82e8944deef9) WriteIdempotent off → on, saved, no deploy. Reload /deployments: "Checking for changes…" → "Configuration drift" with no interaction. PASS. See the deviation below for the revert. |
| #902 Monaco | /scripts/new opened (editor loaded, 02:41:29 local), tab closed without saving. Log from 02:41:20 to 02:44:44 local (over 3 min, past circuit retention): 0 MonacoEditor / Unhandled exception in circuit lines, 0 [ERR]. Control: the 6c4d2990 build logged that [ERR] for the same action earlier the same day. PASS. |
Overview :5320 |
6 registered, 6 Up; OtOpcUa Up, 1 instance. |
| ScadaBridge | :8085/health/ready 200 (no OtOpcUa link). |
Deviation: the #903 revert leaves the chip on "Configuration drift". WriteIdempotent was switched back off and saved;
the dbo.Tag row is back to its sealed content (WriteIdempotent=0, PollGroupId null, TagConfig
{"deviceHostAddress":"10.201.31.5:8193","address":"MACRO:3902","dataType":"Float64"}, compared with the sealed artifact
by a read-only on-box query). The chip still shows drift because ConfigComposer serialises each row's RowVersion
into the artifact, and RevisionHash = SHA-256 of that blob, so any saved edit, even one reverted to identical content,
changes the draft hash. The pack's "chip returns to In sync, revision still fa9a2e28…" cannot hold for an edit-and-revert.
The running configuration is unchanged (nodes serve sealed 595cb909 / fa9a2e28…). No deployment was made to clear
the chip. A "Deploy current configuration" would clear it with content-identical rows, but it would also create a new revision.
That is left to the owner.
QA: wonder-app-vq03 + vq04 (fea7de0a → c9640f5c)
Stage E:\psrem-stage\otopcua-c9640f5c\ on each node (prepare.ps1, iso.ps1, swap.ps1, start.ps1, rollback.ps1,
envdigest.ps1, zip + .sha256, install\); backups in …\backup\, created with a protected DACL.
Before the window: ServiceLevel vq03 240, vq04 250 (Primary vq04, as since the 2026-11-12 OTel restarts); both canary values Good and advancing.
| Step | vq03 | vq04 |
|---|---|---|
| Prepare (services up) | 1398 files, hash + ProductVersion ok; OtOpcUaHost-pre-c9640f5c.reg exported; DB: BACKUP DATABASE OtOpcUaConfig … COPY_ONLY, CHECKSUM, COMPRESSION → OtOpcUaConfig_pre-c9640f5c_20261113.bak (wonder-sql-vq03 default dir, 1929 pages), RESTORE VERIFYONLY valid; 9 migrations, snapshot ON; latest sealed 5905003e rev 82dd0598…. E: 215 GB free. |
same (no DB step); E: 230 GB free. |
| Env | 48 entries, ordered 692ee505f911 / sorted 78e6acfab539. |
47 entries, 41abda6b7dec / 5df6eacf7152. |
| Env cross-node | Same Security__ApiKeys__Pepper (entry SHA prefix ea9ddc525fc0) and LocalDb__Replication__ApiKey (1ce08c4554b0) on both. Differences only in node-specific entries (Cluster__PublicHostname, Cluster__SeedNodes__* order, OpcUa__ApplicationUri, OpcUa__PublicHostname, vq03's LocalDb__Replication__PeerAddress). Scripting__Runner=OutOfProcess, seams Calculation + VirtualTag, Scripting__Worker__* present; OtlpEndpoint=http://127.0.0.1:4317. |
|
| One-time #901 record | reg query DenyPaths = exactly the 12 expected (OtelCollector included). -WhatIf 21 operations, then the full-argument install with -ConfigureHostService and the recorded 12: SettingsRecord changed + HostServiceEnv changed (reorder only), 19 unchanged. Env 48 → 48, ordered SHA 692ee505f911 → a04ca3f1fecb, sorted SHA 78e6acfab539 unchanged (same entries, same values). -Verify OK 21/21. DenyPaths after: the same 12. |
same: 2 changed (SettingsRecord, HostServiceEnv reorder), 47 → 47, ordered 41abda6b7dec → c6adaf68ef48, sorted 5df6eacf7152 unchanged; VERIFY OK 21/21; 12 paths. |
Stop + backup + swap (-NoStart) |
Stop 07:47:46Z (6 s). data\, pki\ → backup\{data,pki}-pre-c9640f5c. Swap: carried 5 appsettings*.json + logs; -Reapply changed WorkerFolderAcl, InstallRootDeny (19 unchanged); -Verify OK 21/21; left stopped 07:48:40Z. |
Stop 07:48:51Z (6 s), same backups, same carry; -Reapply same 2 changed; -Verify OK 21/21; left stopped 07:49:57Z. |
| Start | 07:50:13Z → ready 07:50:22Z (9 s). | 07:50:05Z (first, the old Primary) → ready 07:50:22Z (16 s). |
Post-start -Verify |
OK 21/21, exit 0. Env 48, a04ca3f1fecb / 78e6acfab539. |
OK 21/21, exit 0. Env 47, c6adaf68ef48 / 5df6eacf7152. |
- Pair outage: about 91 s (vq04 stop 07:48:51Z → both ready 07:50:22Z). vq03 was down 07:47:46–07:50:22Z while vq04 still served.
- Order deviation (deliberate): the pack stops both and then swaps both. Here vq03 (Secondary) was stopped, backed up
and swapped (
-NoStart, gated on-Verify) while vq04 kept serving, and only then was vq04 stopped and swapped. No node ran the new build until both were swapped and verified, and a-Verifyfailure on vq03 would have left the old Primary serving. Each node's swap took about 50–60 s of-Reapply/-Verify, so this kept that time out of the pair outage. - Primary after the start: vq03, although vq04 was started 8 s earlier. Both came Up together, and vq03 ended up the oldest member. After the failover gate below, Primary is vq04 again, the same as before the window.
QA gates
| Gate | Result |
|---|---|
| Isolation | -Verify OK 21/21 (exit 0) after each swap and after each start, both nodes. |
| Version | Live E:\ApiInstall\OtOpcUa\app\OtOpcUa.Host.dll on both (SMB copy): SHA-256 prefix aff960bd2d36 = the published build; ProductVersion 1.0.0+c9640f5ced383e2a…. |
| Health | /health/ready 200 on both, 5/5 components: opcua-endpoint, Out-of-process script runner ready., configdb, config-publish, akka Up. /health/active 200 on the Primary, 503 on the Secondary. |
| ServiceLevel | After the start: vq03 250 ×3, vq04 240 ×3 (fresh sessions). |
| Failover | Stop-Service on the Primary vq03 at 07:51:43Z (stopped in 1 s). vq04: 250, /health/active 200 from the first sample (07:51:46Z) and on all 4 samples. vq03 restarted 07:52:11Z → ready 07:52:19Z (8 s), -Verify OK 21/21. Then 6 samples over 40 s: vq03 240 / 503, vq04 250 / 200: never above 240, no failback. |
| LocalDb sync | localdb-replication: connected, oplog backlog 0 on both, before and after the failover. |
| Canary | ns=2;s=script-runner-canary/Device1/canary-calc-seconds and ns=3;s=EQ-ec3979550d90/canary-vt-second: Good and advancing about once a second on both endpoints, right after the start and again after the failover. otopcua_script_worker_up{shard="0"} = 1 and {shard="1"} = 1 on both ({shard="iso"} = 0, idle); otopcua_script_runner_state 0. The optional #780 runaway drill was not run. |
| Boot log (both, both starts on vq03) | roles=[admin,driver]; Historian gateway probe: the gateway is reachable; 12 aggregates advertised; raw subtree materialised (containers=2, tags=1, … failed=0); Credential-reference preflight (lmxopcua#815/#834): every stored and deployed reference is namespaced.; no FTL, no Telemetry endpoint error. ERR lines only: /health/* 503 request logs, the standby cluster-primary check, and vq04's bootstrap akka … Removed before the join (the same pattern as the 2026-11-12 record). 0 other ERR. |
HG /metrics (vq03 :5220, fa9cf31) |
07:47:44Z → 07:54:19Z: HistorianStatus/Probe 1194 → 1214, nothing else. No historian write or tag-creation RPC. |
Overview (vq03 + vq04 :5320) |
12 registered, 12 Up; OtOpcUa Up, 2 instances, 2 members / 0 unreachable. |
| ScadaBridge | :8085/health/ready 200 on both (no OtOpcUa link). |
| #902 / #903 on QA | Optional in the pack; not run (they need the temporary QA AdminUI role-mapping row). Verified on vd03. |
serilog-selflog.txt (vq04) |
Two RollingFileSink … disposed lines, 2026-11-12 21:54Z and 2026-11-13 02:48Z: shutdown-time emits from earlier stops, not from this window. |
Rollback material (left on the boxes)
DEV vd03 → 6c4d2990: E:\ApiInstall\_stage-otopcua-c9640f5c\rollback.ps1 (pack §6: Invoke-OtOpcUaInstallSwap.ps1 -Candidate E:\ApiInstall\OtOpcUa.bak.20261113-pre-c9640f5c -PreviousInstall E:\ApiInstall\OtOpcUa.failed-c9640f5c -NoCarry -NoStart, mirror OtOpcUa-data back, start, -Verify).
- Previous install:
E:\ApiInstall\OtOpcUa.bak.20261113-pre-c9640f5c. - Backups:
E:\ApiInstall\OtOpcUa-backups\pre-20261113-otopcua-c9640f5c\{OtOpcUaHost.reg, OtOpcUa-data\}(protected). - ConfigDb:
OtOpcUaConfig-pre-20261113-otopcua-c9640f5c.bak(vd03 instance default backup dir). - The registry did not change, so no import is needed.
QA → fea7de0a: stop both, then run E:\psrem-stage\otopcua-c9640f5c\rollback.ps1 on each node (the swap back with
-NoCarry -NoStart and -Reapply/-Verify inside, then a mirror of data\ from the backup). Start both after that, the
old Primary first.
- Previous install:
E:\ApiInstall\OtOpcUa\app.bak.20261113-pre-c9640f5con each node. - Backups:
E:\psrem-stage\otopcua-c9640f5c\backup\{OtOpcUaHost-pre-c9640f5c.reg, data-pre-c9640f5c\, pki-pre-c9640f5c\}(protected). The.regimport is not needed, because the env changed only in order. - ConfigDb:
OtOpcUaConfig_pre-c9640f5c_20261113.bakon wonder-sql-vq03 (VERIFYONLY ok). - The #901 settings record and the denies stay valid for
fea7de0a.
Deviations from the pack (summary)
- vd03 downtime about 101 s against an estimate of 20–40 s, because the coverage walk runs inside
-Reapply/-Verifywhile the service is stopped. QA swaps took about 50–60 s per node for the same reason. - #903: the edit-and-revert cannot return the chip to "In sync", because
RowVersionis part of the revision hash (see above). Draft content is restored; the chip shows drift until the next deploy. - QA order: the Secondary was stopped and swapped while the Primary still served, then the Primary. Both started together. Primary roles: vq04 → vq03 after the start → vq04 after the failover gate.
- vd03 #903 draft change: there is no description field on the tag form, so the harmless change was the WriteIdempotent toggle on a read-only FOCAS tag.