Files
lmxopcua/docs/deploy/2026-11-13-vd03-qa-to-76e73e7d.md
T
Joseph Doherty a157b27628
v2-ci / unit-tests (push) Has been cancelled
fix(install): swap waits for the host process to exit before moving the folder (#908)
On QA (76e73e7d deploy) Invoke-OtOpcUaInstallSwap.ps1's first Move-Item of
app\ failed "file in use" right after the service reported Stopped: SCM
reports Stopped while the host is still exiting (e.g. a log file under
InstallRoot still open).

- After the stop, wait (bounded, -ProcessExitTimeoutSeconds, default 60)
  until the service's host process (PID + start time captured before the
  stop) and every process imaged under InstallRoot have exited; otherwise
  throw naming them, with nothing moved.
- The renames use [IO.Directory]::Move instead of Move-Item. On windev,
  PowerShell 7's Move-Item falls back to an item-by-item move when the
  rename fails because a file under the folder is open, and leaves a partial
  PreviousInstall tree behind (a retry then nests into it).
- The first rename is retried for at most -MoveRetrySeconds (default 10)
  while it fails because something under InstallRoot is open (a backstop
  for a holder whose image is outside InstallRoot).
- -Verify/-Reapply/-Scope semantics unchanged.

Tests: dry-run harness gains fake-process-list tests for the wait and the
retry plus AST order checks (145 passed); the Windows harness gains three
real cases (holder past the budget, holder released 2 s into the wait,
holder outside InstallRoot) - 77/0 on pwsh 7 and Windows PowerShell 5.1 on
windev; with the wait removed the first two fail.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 08:46:27 -05:00

20 KiB
Raw Blame History

DEV vd03 and QA vq03/vq04 c9640f5c → master 76e73e7d (2026-11-13)

Status: EXECUTED 2026-11-13.

  • DEV vd03: 12:27–12:32Z.
  • QA pair: 12:34–12:48Z.

All three nodes run 76e73e7d (76e73e7dbf1160e74d2c89036c4235eefdc566dd). The owner approved the deploy on 2026-11-13, with WinRM for the on-box deploy steps only.

Runbook: 2026-11-13-next-vd03-qa-to-47f0b98c.md, the pack. It was written for 47f0b98c. The target moved to the tip 76e73e7d (#900 follow-up: HG Client 1.1.0, typed error reason, behaviour unchanged); see the pack's header note.

Deployed:

  • #906: graceful cluster leave before Kestrel drains, plus LocalDb 0.3.8.
  • #905: Outside-scope isolation before the stop.
  • #717: Warm redundancy, inert under Mode Unset.
  • #900: HG Client 1.1.0.

Channel: WinRM + SMB from the Mac (the winrm skill), used only for the pack's on-box steps:

  • prepare (backups, expand, env digest, pre-window -Verify, swap -WhatIf);
  • swap, start, -Verify;
  • the two failover-leg stops and restarts.

Everything else ran from the Mac:

  • health, OPC UA, /metrics, HG /metrics, Overview and ScadaBridge checks, over HTTP and opc.tcp;
  • logs and the live DLLs, read over SMB.

Not done: no Galaxy writes, no historian writes or tag creation, no configuration deploy. No other service was touched (ScadaBridge, mxgw, HistorianGateway, Overview, OTel collector). OpcUa:Redundancy:Mode is unset on every node, in env and appsettings, checked by name.

Build

  • dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj -c Release -r win-x64 --self-contained true on the Mac (SDK 10.0.403), in a clean worktree at 76e73e7d:
    • 0 warnings, 1398 files;
    • OtOpcUa.Host.dll ProductVersion 1.0.0+76e73e7dbf1160e7…, SHA-256 prefix 6a8ec583a328;
    • script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe present.
  • deps.json:
    • LocalDb(.Replication, .Contracts) 0.3.8 and HistorianGateway.Client/Contracts 1.1.0: the two deltas from c9640f5c;
    • unchanged: Telemetry 0.3.1, MxGateway.Client 0.4.0, Grpc.StatusProto 2.76.0 (now transitive), Auth 0.4.1, Secrets 0.6.2, Health 0.3.0, Theme 0.5.0.
    • ZB.MOM.WW.LocalDb.dll is 0.3.8.0, SHA-256 prefix 62e85cc0cedb.
  • Zip otopcua-76e73e7d.zip (139,455,907 B), SHA-256 22A26D19FEF364AA6FDA5B72CD693B6301B0C05DAEFB52AB850724A2804F4A0B. Its hash was checked on every box before expanding.
  • Install scripts staged from 76e73e7d. They are byte-identical to the pack's table (bb31d275…, c75a6e3e…, 09a0bb68…), and the hashes were checked on every box.

DEV: wonder-app-vd03 (c9640f5c → 76e73e7d)

Stage E:\ApiInstall\_stage-otopcua-76e73e7d\ holds:

  • prepare.ps1, swap.ps1, rollback.ps1, envdigest.ps1;
  • the zip and its .sha256;
  • install\.
Step Result
Prepare (service up) Backup folder E:\ApiInstall\OtOpcUa-backups\pre-20261113-otopcua-76e73e7d\ created with a protected DACL (SYSTEM + Administrators), OtOpcUaHost.reg exported into it. 1398 files expanded to E:\ApiInstall\OtOpcUa.new-76e73e7d; ProductVersion and deps.json (LocalDb 0.3.8, HG Client 1.1.0) checked. BACKUP DATABASE … COPY_ONLY, CHECKSUM → OtOpcUaConfig-pre-20261113-otopcua-76e73e7d.bak (1905 pages). ConfigDb: 9 migrations, last 20261022102459_EnableSnapshotIsolation, snapshot ON, latest sealed 595cb909 / fa9a2e28…. Pre-window -Verify with the new installer: VERIFY OK 14/14. Swap -WhatIf: preflight ok. E: 176 GB free.
Env (names + SHA-256 prefixes only) 34 entries, ordered 69a493345099 / sorted caa047aaaf42 (= the pack). Runner OutOfProcess + 3 seams + 2 Scripting__Worker__*; OTLP endpoint has a scheme. OpcUa__Redundancy* / OpcUa__PeerApplicationUris*: 0 entries; appsettings*.json Redundancy/PeerApplicationUris: 0 hits.
Swap (-NoStart, started with the service up) Invoked 12:28:02.2Z. -Reapply -Scope Outside: 11/11 unchanged (incl. HostServiceEnv, SettingsRecord). -Verify -Scope Outside: OK 11/11. That pre-stop isolation took 28.0 s with the old build serving. Stop issued 12:28:30.58Z. Moved; carried 5 appsettings*.json, pki, logs. -Reapply -Scope InstallRoot changed WorkerFolderAcl, InstallRootDeny, SensitiveDirDeny (pki); -Verify -Scope InstallRoot OK 3/3. stop -> swapped and verified (left stopped): 5.7 s [stop 1.5 s, move 0.0 s, carry 0.8 s, isolation 3.4 s].
Backup + start OtOpcUa-data robocopied into the backup folder (stopped, consistent). Start 12:28:36.46Z → /health/ready 200 at 12:28:44.56Z (8.1 s).
Downtime 14.0 s (stop issued → ready, on-box). From the Mac, /health/ready was unreachable from 12:28:30.5Z to 12:28:44.2Z (13.7 s). c9640f5c window: ~101 s.
Post-start full -Verify VERIFY OK 14/14, exit 0. Env after: 34, 69a493345099 / caa047aaaf42: unchanged.

vd03 gates

Gate Result
Version Live E:\ApiInstall\OtOpcUa\OtOpcUa.Host.dll (SMB copy): SHA-256 prefix 6a8ec583a328 = the published build; ProductVersion 1.0.0+76e73e7d…. Live ZB.MOM.WW.LocalDb.dll 0.3.8.0 (62e85cc0cedb).
Health /healthz, /health/ready and /health/active all 200. /health/ready is 5/5: opcua-endpoint; Out-of-process script runner ready. (state Ready, peCache true, shards not started, vd03 has no scripts); configdb; config-publish; akka Up with 1 member.
ServiceLevel 250 on three fresh-session reads.
Redundancy under Unset redundancy CLI reports Mode None, no server URIs. i=3709 = 0 Good; i=11314 Good String[]; i=2254 Good. All match the pack's §3 baseline. The boot log has OPC UA redundancy: Mode="Unset", effective ApplicationUri=urn:OtOpcUa, peers=[]. /metrics: otopcua_redundancy_warm_role = -1 ("not Warm" by the gauge's own definition); no warm_transition / snapshot_rejected / mode_mismatch / opcua_standby_* series.
FOCAS 22 raw variables: 21 Good, Program/BlockCount BadNoData (as before). 2 UNS variables. All 115 ns≥1 variables were read: 114 Good.
Boot log roles=[admin,driver]; Historian gateway probe: the gateway is reachable.; applied plan (PureAdd, added=36 …); raw subtree materialised (containers=10, tags=22, … failed=0); Credential-reference preflight (lmxopcua#815/#834): every stored and deployed reference is namespaced.; 0 FTL, no options-validation failure. The 60 ERR lines are all The domain 'opc.tcp://10.220.157.247:4840/OtOpcUa' is not listed in the server certificate (the gate's own reads by IP).
Historian path (added gate, #900 / HG Client 1.1.0) HG :5220/metrics (6365f34), historian_gateway_grpc_requests_total{service="HistorianStatus",method="Probe",outcome="ok"}: 494 (12:27:17Z) → 504 (12:31:52Z) → 536 (12:47:45Z). OtOpcUa on the new client probes on its cycle, and the boot log has Historian gateway probe: the gateway is reachable. at 07:28:40 local. Unchanged: BrowseTagNames 1, ReadRaw 1, ReadAggregate ok 1 / error 2, Probe error 1. No EnsureTags, AddTagExtendedProperties, WriteLiveValues, AddHistoricalValues, SendEvent(s) series at all. PASS.
#903 drift chip Not touched: no configuration deploy was made. The nodes still serve sealed 595cb909 / fa9a2e28… (preflight SQL). The pack says the chip shows "Configuration drift" and that this is not a rollback trigger.
Overview / ScadaBridge Overview :5320 200. ScadaBridge :8085/health/ready 200 (untouched; 8e32f353).

QA: wonder-app-vq03 + vq04 (c9640f5c → 76e73e7d)

Stage E:\psrem-stage\otopcua-76e73e7d\ on each node holds:

  • prepare.ps1, swap.ps1, swapretry.ps1, swap2.ps1 (vq03), start.ps1, legstop.ps1, rollback.ps1, envdigest.ps1;
  • the zip and its .sha256;
  • install\.

Backups are in …\backup\, which has a protected DACL.

Before the window (12:25–12:35Z):

  • ServiceLevel vq03 250 (Primary), vq04 240, as the pack read at 11:58Z.
  • Both canaries Good on both endpoints.
  • redundancy CLI on vq03: Mode None.
Step vq03 vq04
Prepare (services up) 1398 files, hash + ProductVersion + deps.json ok; OtOpcUaHost-pre-76e73e7d.reg exported. DB: BACKUP DATABASE OtOpcUaConfig … COPY_ONLY, CHECKSUM, COMPRESSION → OtOpcUaConfig_pre-76e73e7d_20261113.bak (wonder-sql-vq03, 1969 pages), RESTORE VERIFYONLY valid. 9 migrations, snapshot ON, latest sealed a8e078f0 / 82dd0598…. Pre-window -Verify OK 21/21; swap -WhatIf ok. E: 213 GB free. Same (no DB step). Pre-window -Verify OK 21/21. E: 228 GB free.
Env 49 entries, 5750bbeb4734 / 4263c3ac5a46 (= the pack). 48 entries, ef146b2681b4 / d19e968212f3 (= the pack).
Env cross-node Same Security__ApiKeys__Pepper (entry prefix ea9ddc525fc0) and LocalDb__Replication__ApiKey (1ce08c4554b0) on both. Seams Calculation / VirtualTag / ScriptedAlarm on both. No OpcUa__Redundancy* / OpcUa__PeerApplicationUris* entries and no appsettings hits on either node.
Swap (-NoStart, service up) Second. Started 12:38:39Z; -Scope Outside 19/19 unchanged, OK. Stop issued 12:38:58.22Z, 1.3 s. The first Move-Item app → app.bak… failed file in use (lmxopcua#908). The in-call retry (swap2.ps1) found no OtOpcUa.Host / ScriptWorker process 0.5 s later, waited 2 s and re-ran the swap with the service stopped: Outside 19/19 unchanged + OK again, moved, carried 5 appsettings*.json + logs, InstallRoot changed WorkerFolderAcl, InstallRootDeny, OK 2/2. Window line of the retry: 4.6 s [stop 0.0 s, move 0.0 s, carry 0.8 s, isolation 3.8 s]. Left stopped 12:39:25.1Z; data\ + pki\ → backup\{data,pki}-pre-76e73e7d. First (the Secondary, while vq03 served). -Scope Outside 19/19 unchanged, -Verify OK 19/19 (17.9 s pre-stop). Stop issued 12:35:26.96Z, Stopped at 12:35:57.08Z: 30.1 s. This is the #906 "before" figure, on the old build. The first Move-Item then failed file in use: nothing moved, the service was Stopped, app\ was intact. swapretry.ps1 at 12:36:46Z found 0 processes with an image or module under app\ and re-ran the swap (service stopped): Outside 19/19 unchanged + OK, moved, carried, InstallRoot changed 2 + OK 2/2, window 4.8 s. Left stopped 12:37:53Z; data\ + pki\ backed up.
Start (vq03 first, vq04 ~1 s later) 12:39:26.71Z → ready 12:39:35.29Z (8.6 s). Full -Verify OK 21/21, exit 0. Env 49, 5750bbeb4734 / 4263c3ac5a46: unchanged. 12:39:28.00Z → ready 12:39:38.06Z (10.1 s). OK 21/21, exit 0. Env 48, ef146b2681b4 / d19e968212f3: unchanged.
  • Pair outage: 36.5 s. No node was ready from 12:38:58.36Z to 12:39:34.85Z (0.5 s Mac poll of both /health/ready); both were ready at 12:39:37.8Z. The c9640f5c window was ~91 s; the pack estimated 30–40 s. About 20 s of it is the vq03 retry re-running the Outside scope (lmxopcua#908); without the move failure the outage would have been about 15 s.
  • vq04's 30 s old-build stop and its retry were outside the outage, because vq03 served throughout. The poll shows vq03 200/200 from 12:35 to 12:38:58Z.
  • Primary after the start: vq03 (250), vq04 240, ×3 fresh-session reads each.

QA gates

Gate Result
Swap output Outside and InstallRoot -Verify OK on both nodes (after the retries). Window lines recorded above.
Isolation Full -Verify OK 21/21, exit 0 after every start: both nodes after the window, vq03 after leg 1, vq04 after leg 2.
Version Live E:\ApiInstall\OtOpcUa\app\OtOpcUa.Host.dll on both (SMB copy): 6a8ec583a328 = the published build, ProductVersion 1.0.0+76e73e7d…. ZB.MOM.WW.LocalDb.dll 0.3.8.0 (62e85cc0cedb) on both.
Health /health/ready 200 on both, 5/5 components: opcua-endpoint, Out-of-process script runner ready. (shards 0 and 1 up, quarantined 0), configdb, config-publish (Ready on vq03 / NotHosted on vq04), akka Up with 2 members / 0 unreachable. /health/active 200 on vq03, 503 on vq04.
Redundancy under Unset On both nodes, redundancy CLI Mode None, no server URIs; i=3709 = 0 Good; i=11314 / i=2254 Good String[]. Boot logs (every start): OPC UA redundancy: Mode="Unset", effective ApplicationUri=urn:wonder-app-vq0N:OtOpcUa, peers=[]. warm_role -1 on both; no warm_transition / snapshot_rejected / mode_mismatch / opcua_standby_*. 0 Warm redundancy / mode_mismatch log lines. The Secondary (vq04, 240) serves Good canary data.
LocalDb sync localdb-replication (on /health/active): "LocalDb replication is connected; oplog backlog 0." on both nodes, after the start, after leg 1 and after leg 2. localdb_oplog_depth 0, meter scope version 0.3.8.
Canary + runner canary-calc-seconds and canary-vt-second are Good and advancing about 1/s on both endpoints: after the start, after leg 1 and at the end. otopcua_script_worker_up{shard="0"} = 1 and {shard="1"} = 1, runner_state 0, quarantined 0 on both.
ScriptedAlarm seam Env carries Scripting__Runner__Seams__2=ScriptedAlarm (digest unchanged) and the runner is Ready on both nodes. There is no scripted alarm in the canary-only config, so kind="alarm" does not count. The live proof remains the 11:44–11:53Z drill on c9640f5c. The optional drill was not re-run.
Boot log (both nodes, every start) roles=[admin,driver]; Historian gateway probe: the gateway is reachable.; raw subtree materialised (containers=2, tags=1, … failed=0); credential-reference preflight clean; 0 FTL. ERR lines: the standby /health/active 503 request logs and cluster-primary Unhealthy on whichever node was the Secondary, plus one TCPSERVERCHANNEL … BadConnectionClosed per node from the gate's own CLI subscription being killed. Nothing else.
HG /metrics (vq03 :5220) 12:34:54Z → 12:47:35Z: HistorianStatus/Probe ok 988 → 1028, nothing else changed. No historian write or tag-creation RPC.
Overview / ScadaBridge :5320 200 and :8085/health/ready 200 on both nodes (untouched).

Failover + #906 timing

Method:

  • a background subscribe -n i=2267 -i 250 on each endpoint, each update stamped with its Mac arrival time;
  • a 0.5 s poll of /health/ready + /health/active on both nodes;
  • legstop.ps1, which wraps a stopwatch around Stop-Service and then checks the service stays Stopped for 15 s.
Leg 1: stop vq03 (Primary, sync client) Leg 2: stop vq04 (Primary, sync server): the #906 proof
Stop-Service issued 12:42:04.372Z 12:44:07.472Z
Peer → 250 (subscription arrival) vq04 250 at 12:42:04.728Z (0.36 s) vq03 250 at 12:44:07.944Z (0.47 s)
Peer /health/active 200 on the first sample (12:42:04.938Z) 200 on the first sample (12:44:07.600Z)
Stop-Service duration 6.10 s 5.70 s (old build, same node, 1 h earlier: 30.1 s)
Stopping node's log Application is shutting down 07:42:04.391 local → Host stopping: leaving the Akka cluster before the web server drains its open requests (lmxopcua#906). .394 → Leaving .409 → Akka CoordinatedShutdown completed in 647 ms Application is shutting down 07:44:07.479 → LocalDbSync/Sync responded 200 in 86702 ms .480 (the stream ended 1 ms after the stop) → … (lmxopcua#906). .483 → Leaving .500 → Akka CoordinatedShutdown completed in 236 ms (.718). No Sync responded 200 in ~30000+ ms wait.
Stays stopped Stopped after 15 s, no host process (no SCM recovery restart) same
Restart 12:42:38.6Z → ready 12:42:46.9Z (8.3 s), -Verify OK 21/21. vq03 240 (first value 12:42:49Z, unchanged after), /health/active 503 on all samples: no failback 12:46:03.3Z → ready 12:46:12.6Z (9.3 s), -Verify OK 21/21. vq04 240, 503: no failback

#906 before / after (same node vq04, the sync server, same day):

Before (c9640f5c, the window's vq04 stop, 12:35Z) After (76e73e7d, leg 2, 12:44Z)
Application is shutting down → Leaving 30.0 s (07:35:26.975 → Sync responded 200 07:35:56.989 → Leaving 07:35:57.020) 21 ms (07:44:07.479 → 07:44:07.500)
Stop-Service 30.1 s 5.7 s
Peer at 250 (vq03 was already Primary) — #780 record: ~31 s 0.47 s

Observation (not a #906 regression): on both legs the host logged nothing for ~5 s after CoordinatedShutdown completed. The gap ended with SERVER - Service Fault Occurred. Reason=BadTimeout: vq03 07:42:10.134, vq04 07:44:12.778.

The likely cause is the OPC UA stack closing the gate's own open subscribe session at server stop. That would put Stop-Service at ~6 s rather than the pack's 1–3 s target. The c9640f5c record's vq03 stop (no client session open) took 1 s.

The hand-over is not affected, because the cluster leave comes first. The peer took over within 0.5 s on both legs.

End state: vq03 250 (Primary), vq04 240, the same as before the window. Canary Good on both, sync connected, backlog 0.

Rollback material (left on the boxes)

DEV vd03 → c9640f5c:

  • Script: E:\ApiInstall\_stage-otopcua-76e73e7d\rollback.ps1 (pack §6). It runs Invoke-OtOpcUaInstallSwap.ps1 -Candidate E:\ApiInstall\OtOpcUa.bak.20261113-pre-76e73e7d -PreviousInstall E:\ApiInstall\OtOpcUa.failed-76e73e7d -NoCarry -NoStart, mirrors OtOpcUa-data back, starts and runs a full -Verify.
  • Previous install: E:\ApiInstall\OtOpcUa.bak.20261113-pre-76e73e7d (= c9640f5c).
  • Backups: E:\ApiInstall\OtOpcUa-backups\pre-20261113-otopcua-76e73e7d\{OtOpcUaHost.reg, OtOpcUa-data\} (protected).
  • ConfigDb: OtOpcUaConfig-pre-20261113-otopcua-76e73e7d.bak (vd03 instance default backup dir). No migration, so this is a configuration fallback only.
  • No registry import is needed: the env was unchanged.

QA → c9640f5c:

  1. Stop both nodes, Secondary first.
  2. Run E:\psrem-stage\otopcua-76e73e7d\rollback.ps1 on each node. It swaps back with -NoCarry -NoStart (Reapply/Verify inside) and mirrors data\ from the backup.
  3. Start both, vq03 first.
  4. Run start.ps1 on each node (it includes -Verify).

Do not run a mixed pair: RedundancyStateChanged changed shape in this range. After a rollback, the #906 ~30 s stop of vq04 is back.

  • Previous install: E:\ApiInstall\OtOpcUa\app.bak.20261113-pre-76e73e7d on each node.
  • Backups: E:\psrem-stage\otopcua-76e73e7d\backup\{OtOpcUaHost-pre-76e73e7d.reg, data-pre-76e73e7d\, pki-pre-76e73e7d\} (protected). No .reg import is needed: the env was unchanged.
  • ConfigDb: OtOpcUaConfig_pre-76e73e7d_20261113.bak on wonder-sql-vq03 (VERIFYONLY ok).

Deviations from the pack

  1. Target 76e73e7d instead of 47f0b98c. The tip adds only the #900 follow-up (HG Client 1.1.0); see the pack's header. One gate was added for it: vd03 HG Probe growing, with no write RPCs. PASS.
  2. QA: the first Move-Item of app\ failed "file in use" on both nodes, right after the stop. Nothing moved, the service was Stopped, and the retry with the service stopped went through on both (vq04: a separate retry; vq03: an in-call retry in swap2.ps1).
    • The likely cause is the host process still exiting after SCM reported Stopped. The pre-#905 wrappers waited for the process to exit; Invoke-OtOpcUaInstallSwap.ps1 does not.
    • It cost about 20 s of the vq03 pair outage (the retry re-runs -Scope Outside).
    • Filed as lmxopcua#908. vd03 did not hit it.
    • Fixed (lmxopcua#908, closed): Invoke-OtOpcUaInstallSwap.ps1 now waits, after Stopped, for the host process and every process imaged under InstallRoot to exit (-ProcessExitTimeoutSeconds, default 60) and retries the first move on a sharing violation (-MoveRetrySeconds, default 10). The next deploy needs no hand retry.
  3. Stop-Service on the new build took 5.7–6.1 s instead of the 1–3 s target, because of the ~5 s OPC UA session close after the cluster leave (see "Observation"). The hand-over, the actual #906 criterion, was 0.36 s and 0.47 s.
  4. Logs: the QA host log for the window rolled to otopcua-20261113_001.log on vq04 (size limit). Read it from there.