On QA (76e73e7d deploy) Invoke-OtOpcUaInstallSwap.ps1's first Move-Item of
app\ failed "file in use" right after the service reported Stopped: SCM
reports Stopped while the host is still exiting (e.g. a log file under
InstallRoot still open).
- After the stop, wait (bounded, -ProcessExitTimeoutSeconds, default 60)
until the service's host process (PID + start time captured before the
stop) and every process imaged under InstallRoot have exited; otherwise
throw naming them, with nothing moved.
- The renames use [IO.Directory]::Move instead of Move-Item. On windev,
PowerShell 7's Move-Item falls back to an item-by-item move when the
rename fails because a file under the folder is open, and leaves a partial
PreviousInstall tree behind (a retry then nests into it).
- The first rename is retried for at most -MoveRetrySeconds (default 10)
while it fails because something under InstallRoot is open (a backstop
for a holder whose image is outside InstallRoot).
- -Verify/-Reapply/-Scope semantics unchanged.
Tests: dry-run harness gains fake-process-list tests for the wait and the
retry plus AST order checks (145 passed); the Windows harness gains three
real cases (holder past the budget, holder released 2 s into the wait,
holder outside InstallRoot) - 77/0 on pwsh 7 and Windows PowerShell 5.1 on
windev; with the wait removed the first two fail.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
20 KiB
DEV vd03 and QA vq03/vq04 c9640f5c → master 76e73e7d (2026-11-13)
Status: EXECUTED 2026-11-13.
- DEV vd03: 12:27–12:32Z.
- QA pair: 12:34–12:48Z.
All three nodes run
76e73e7d(76e73e7dbf1160e74d2c89036c4235eefdc566dd). The owner approved the deploy on 2026-11-13, with WinRM for the on-box deploy steps only.Runbook:
2026-11-13-next-vd03-qa-to-47f0b98c.md, the pack. It was written for47f0b98c. The target moved to the tip76e73e7d(#900 follow-up: HG Client 1.1.0, typed error reason, behaviour unchanged); see the pack's header note.Deployed:
- #906: graceful cluster leave before Kestrel drains, plus LocalDb 0.3.8.
- #905: Outside-scope isolation before the stop.
- #717: Warm redundancy, inert under
ModeUnset.- #900: HG Client 1.1.0.
Channel: WinRM + SMB from the Mac (the winrm skill), used only for the pack's on-box steps:
- prepare (backups, expand, env digest, pre-window
-Verify, swap-WhatIf); - swap, start,
-Verify; - the two failover-leg stops and restarts.
Everything else ran from the Mac:
- health, OPC UA,
/metrics, HG/metrics, Overview and ScadaBridge checks, over HTTP and opc.tcp; - logs and the live DLLs, read over SMB.
Not done: no Galaxy writes, no historian writes or tag creation, no configuration deploy. No other service was touched (ScadaBridge, mxgw,
HistorianGateway, Overview, OTel collector). OpcUa:Redundancy:Mode is unset on every node, in env and appsettings, checked by name.
Build
dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj -c Release -r win-x64 --self-contained trueon the Mac (SDK 10.0.403), in a clean worktree at76e73e7d:- 0 warnings, 1398 files;
OtOpcUa.Host.dllProductVersion1.0.0+76e73e7dbf1160e7…, SHA-256 prefix6a8ec583a328;script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exepresent.
deps.json:- LocalDb(.Replication, .Contracts) 0.3.8 and HistorianGateway.Client/Contracts 1.1.0: the two deltas from
c9640f5c; - unchanged: Telemetry 0.3.1, MxGateway.Client 0.4.0, Grpc.StatusProto 2.76.0 (now transitive), Auth 0.4.1, Secrets 0.6.2, Health 0.3.0, Theme 0.5.0.
ZB.MOM.WW.LocalDb.dllis 0.3.8.0, SHA-256 prefix62e85cc0cedb.
- LocalDb(.Replication, .Contracts) 0.3.8 and HistorianGateway.Client/Contracts 1.1.0: the two deltas from
- Zip
otopcua-76e73e7d.zip(139,455,907 B), SHA-25622A26D19FEF364AA6FDA5B72CD693B6301B0C05DAEFB52AB850724A2804F4A0B. Its hash was checked on every box before expanding. - Install scripts staged from
76e73e7d. They are byte-identical to the pack's table (bb31d275…,c75a6e3e…,09a0bb68…), and the hashes were checked on every box.
DEV: wonder-app-vd03 (c9640f5c → 76e73e7d)
Stage E:\ApiInstall\_stage-otopcua-76e73e7d\ holds:
prepare.ps1,swap.ps1,rollback.ps1,envdigest.ps1;- the zip and its
.sha256; install\.
| Step | Result |
|---|---|
| Prepare (service up) | Backup folder E:\ApiInstall\OtOpcUa-backups\pre-20261113-otopcua-76e73e7d\ created with a protected DACL (SYSTEM + Administrators), OtOpcUaHost.reg exported into it. 1398 files expanded to E:\ApiInstall\OtOpcUa.new-76e73e7d; ProductVersion and deps.json (LocalDb 0.3.8, HG Client 1.1.0) checked. BACKUP DATABASE … COPY_ONLY, CHECKSUM → OtOpcUaConfig-pre-20261113-otopcua-76e73e7d.bak (1905 pages). ConfigDb: 9 migrations, last 20261022102459_EnableSnapshotIsolation, snapshot ON, latest sealed 595cb909 / fa9a2e28…. Pre-window -Verify with the new installer: VERIFY OK 14/14. Swap -WhatIf: preflight ok. E: 176 GB free. |
| Env (names + SHA-256 prefixes only) | 34 entries, ordered 69a493345099 / sorted caa047aaaf42 (= the pack). Runner OutOfProcess + 3 seams + 2 Scripting__Worker__*; OTLP endpoint has a scheme. OpcUa__Redundancy* / OpcUa__PeerApplicationUris*: 0 entries; appsettings*.json Redundancy/PeerApplicationUris: 0 hits. |
Swap (-NoStart, started with the service up) |
Invoked 12:28:02.2Z. -Reapply -Scope Outside: 11/11 unchanged (incl. HostServiceEnv, SettingsRecord). -Verify -Scope Outside: OK 11/11. That pre-stop isolation took 28.0 s with the old build serving. Stop issued 12:28:30.58Z. Moved; carried 5 appsettings*.json, pki, logs. -Reapply -Scope InstallRoot changed WorkerFolderAcl, InstallRootDeny, SensitiveDirDeny (pki); -Verify -Scope InstallRoot OK 3/3. stop -> swapped and verified (left stopped): 5.7 s [stop 1.5 s, move 0.0 s, carry 0.8 s, isolation 3.4 s]. |
| Backup + start | OtOpcUa-data robocopied into the backup folder (stopped, consistent). Start 12:28:36.46Z → /health/ready 200 at 12:28:44.56Z (8.1 s). |
| Downtime | 14.0 s (stop issued → ready, on-box). From the Mac, /health/ready was unreachable from 12:28:30.5Z to 12:28:44.2Z (13.7 s). c9640f5c window: ~101 s. |
Post-start full -Verify |
VERIFY OK 14/14, exit 0. Env after: 34, 69a493345099 / caa047aaaf42: unchanged. |
vd03 gates
| Gate | Result |
|---|---|
| Version | Live E:\ApiInstall\OtOpcUa\OtOpcUa.Host.dll (SMB copy): SHA-256 prefix 6a8ec583a328 = the published build; ProductVersion 1.0.0+76e73e7d…. Live ZB.MOM.WW.LocalDb.dll 0.3.8.0 (62e85cc0cedb). |
| Health | /healthz, /health/ready and /health/active all 200. /health/ready is 5/5: opcua-endpoint; Out-of-process script runner ready. (state Ready, peCache true, shards not started, vd03 has no scripts); configdb; config-publish; akka Up with 1 member. |
| ServiceLevel | 250 on three fresh-session reads. |
| Redundancy under Unset | redundancy CLI reports Mode None, no server URIs. i=3709 = 0 Good; i=11314 Good String[]; i=2254 Good. All match the pack's §3 baseline. The boot log has OPC UA redundancy: Mode="Unset", effective ApplicationUri=urn:OtOpcUa, peers=[]. /metrics: otopcua_redundancy_warm_role = -1 ("not Warm" by the gauge's own definition); no warm_transition / snapshot_rejected / mode_mismatch / opcua_standby_* series. |
| FOCAS | 22 raw variables: 21 Good, Program/BlockCount BadNoData (as before). 2 UNS variables. All 115 ns≥1 variables were read: 114 Good. |
| Boot log | roles=[admin,driver]; Historian gateway probe: the gateway is reachable.; applied plan (PureAdd, added=36 …); raw subtree materialised (containers=10, tags=22, … failed=0); Credential-reference preflight (lmxopcua#815/#834): every stored and deployed reference is namespaced.; 0 FTL, no options-validation failure. The 60 ERR lines are all The domain 'opc.tcp://10.220.157.247:4840/OtOpcUa' is not listed in the server certificate (the gate's own reads by IP). |
| Historian path (added gate, #900 / HG Client 1.1.0) | HG :5220/metrics (6365f34), historian_gateway_grpc_requests_total{service="HistorianStatus",method="Probe",outcome="ok"}: 494 (12:27:17Z) → 504 (12:31:52Z) → 536 (12:47:45Z). OtOpcUa on the new client probes on its cycle, and the boot log has Historian gateway probe: the gateway is reachable. at 07:28:40 local. Unchanged: BrowseTagNames 1, ReadRaw 1, ReadAggregate ok 1 / error 2, Probe error 1. No EnsureTags, AddTagExtendedProperties, WriteLiveValues, AddHistoricalValues, SendEvent(s) series at all. PASS. |
| #903 drift chip | Not touched: no configuration deploy was made. The nodes still serve sealed 595cb909 / fa9a2e28… (preflight SQL). The pack says the chip shows "Configuration drift" and that this is not a rollback trigger. |
| Overview / ScadaBridge | Overview :5320 200. ScadaBridge :8085/health/ready 200 (untouched; 8e32f353). |
QA: wonder-app-vq03 + vq04 (c9640f5c → 76e73e7d)
Stage E:\psrem-stage\otopcua-76e73e7d\ on each node holds:
prepare.ps1,swap.ps1,swapretry.ps1,swap2.ps1(vq03),start.ps1,legstop.ps1,rollback.ps1,envdigest.ps1;- the zip and its
.sha256; install\.
Backups are in …\backup\, which has a protected DACL.
Before the window (12:25–12:35Z):
- ServiceLevel vq03 250 (Primary), vq04 240, as the pack read at 11:58Z.
- Both canaries Good on both endpoints.
redundancyCLI on vq03: ModeNone.
| Step | vq03 | vq04 |
|---|---|---|
| Prepare (services up) | 1398 files, hash + ProductVersion + deps.json ok; OtOpcUaHost-pre-76e73e7d.reg exported. DB: BACKUP DATABASE OtOpcUaConfig … COPY_ONLY, CHECKSUM, COMPRESSION → OtOpcUaConfig_pre-76e73e7d_20261113.bak (wonder-sql-vq03, 1969 pages), RESTORE VERIFYONLY valid. 9 migrations, snapshot ON, latest sealed a8e078f0 / 82dd0598…. Pre-window -Verify OK 21/21; swap -WhatIf ok. E: 213 GB free. |
Same (no DB step). Pre-window -Verify OK 21/21. E: 228 GB free. |
| Env | 49 entries, 5750bbeb4734 / 4263c3ac5a46 (= the pack). |
48 entries, ef146b2681b4 / d19e968212f3 (= the pack). |
| Env cross-node | Same Security__ApiKeys__Pepper (entry prefix ea9ddc525fc0) and LocalDb__Replication__ApiKey (1ce08c4554b0) on both. Seams Calculation / VirtualTag / ScriptedAlarm on both. No OpcUa__Redundancy* / OpcUa__PeerApplicationUris* entries and no appsettings hits on either node. |
|
Swap (-NoStart, service up) |
Second. Started 12:38:39Z; -Scope Outside 19/19 unchanged, OK. Stop issued 12:38:58.22Z, 1.3 s. The first Move-Item app → app.bak… failed file in use (lmxopcua#908). The in-call retry (swap2.ps1) found no OtOpcUa.Host / ScriptWorker process 0.5 s later, waited 2 s and re-ran the swap with the service stopped: Outside 19/19 unchanged + OK again, moved, carried 5 appsettings*.json + logs, InstallRoot changed WorkerFolderAcl, InstallRootDeny, OK 2/2. Window line of the retry: 4.6 s [stop 0.0 s, move 0.0 s, carry 0.8 s, isolation 3.8 s]. Left stopped 12:39:25.1Z; data\ + pki\ → backup\{data,pki}-pre-76e73e7d. |
First (the Secondary, while vq03 served). -Scope Outside 19/19 unchanged, -Verify OK 19/19 (17.9 s pre-stop). Stop issued 12:35:26.96Z, Stopped at 12:35:57.08Z: 30.1 s. This is the #906 "before" figure, on the old build. The first Move-Item then failed file in use: nothing moved, the service was Stopped, app\ was intact. swapretry.ps1 at 12:36:46Z found 0 processes with an image or module under app\ and re-ran the swap (service stopped): Outside 19/19 unchanged + OK, moved, carried, InstallRoot changed 2 + OK 2/2, window 4.8 s. Left stopped 12:37:53Z; data\ + pki\ backed up. |
| Start (vq03 first, vq04 ~1 s later) | 12:39:26.71Z → ready 12:39:35.29Z (8.6 s). Full -Verify OK 21/21, exit 0. Env 49, 5750bbeb4734 / 4263c3ac5a46: unchanged. |
12:39:28.00Z → ready 12:39:38.06Z (10.1 s). OK 21/21, exit 0. Env 48, ef146b2681b4 / d19e968212f3: unchanged. |
- Pair outage: 36.5 s. No node was ready from 12:38:58.36Z to 12:39:34.85Z (0.5 s Mac poll of both
/health/ready); both were ready at 12:39:37.8Z. Thec9640f5cwindow was ~91 s; the pack estimated 30–40 s. About 20 s of it is the vq03 retry re-running the Outside scope (lmxopcua#908); without the move failure the outage would have been about 15 s. - vq04's 30 s old-build stop and its retry were outside the outage, because vq03 served throughout. The poll shows vq03 200/200 from 12:35 to 12:38:58Z.
- Primary after the start: vq03 (250), vq04 240, ×3 fresh-session reads each.
QA gates
| Gate | Result |
|---|---|
| Swap output | Outside and InstallRoot -Verify OK on both nodes (after the retries). Window lines recorded above. |
| Isolation | Full -Verify OK 21/21, exit 0 after every start: both nodes after the window, vq03 after leg 1, vq04 after leg 2. |
| Version | Live E:\ApiInstall\OtOpcUa\app\OtOpcUa.Host.dll on both (SMB copy): 6a8ec583a328 = the published build, ProductVersion 1.0.0+76e73e7d…. ZB.MOM.WW.LocalDb.dll 0.3.8.0 (62e85cc0cedb) on both. |
| Health | /health/ready 200 on both, 5/5 components: opcua-endpoint, Out-of-process script runner ready. (shards 0 and 1 up, quarantined 0), configdb, config-publish (Ready on vq03 / NotHosted on vq04), akka Up with 2 members / 0 unreachable. /health/active 200 on vq03, 503 on vq04. |
| Redundancy under Unset | On both nodes, redundancy CLI Mode None, no server URIs; i=3709 = 0 Good; i=11314 / i=2254 Good String[]. Boot logs (every start): OPC UA redundancy: Mode="Unset", effective ApplicationUri=urn:wonder-app-vq0N:OtOpcUa, peers=[]. warm_role -1 on both; no warm_transition / snapshot_rejected / mode_mismatch / opcua_standby_*. 0 Warm redundancy / mode_mismatch log lines. The Secondary (vq04, 240) serves Good canary data. |
| LocalDb sync | localdb-replication (on /health/active): "LocalDb replication is connected; oplog backlog 0." on both nodes, after the start, after leg 1 and after leg 2. localdb_oplog_depth 0, meter scope version 0.3.8. |
| Canary + runner | canary-calc-seconds and canary-vt-second are Good and advancing about 1/s on both endpoints: after the start, after leg 1 and at the end. otopcua_script_worker_up{shard="0"} = 1 and {shard="1"} = 1, runner_state 0, quarantined 0 on both. |
| ScriptedAlarm seam | Env carries Scripting__Runner__Seams__2=ScriptedAlarm (digest unchanged) and the runner is Ready on both nodes. There is no scripted alarm in the canary-only config, so kind="alarm" does not count. The live proof remains the 11:44–11:53Z drill on c9640f5c. The optional drill was not re-run. |
| Boot log (both nodes, every start) | roles=[admin,driver]; Historian gateway probe: the gateway is reachable.; raw subtree materialised (containers=2, tags=1, … failed=0); credential-reference preflight clean; 0 FTL. ERR lines: the standby /health/active 503 request logs and cluster-primary Unhealthy on whichever node was the Secondary, plus one TCPSERVERCHANNEL … BadConnectionClosed per node from the gate's own CLI subscription being killed. Nothing else. |
HG /metrics (vq03 :5220) |
12:34:54Z → 12:47:35Z: HistorianStatus/Probe ok 988 → 1028, nothing else changed. No historian write or tag-creation RPC. |
| Overview / ScadaBridge | :5320 200 and :8085/health/ready 200 on both nodes (untouched). |
Failover + #906 timing
Method:
- a background
subscribe -n i=2267 -i 250on each endpoint, each update stamped with its Mac arrival time; - a 0.5 s poll of
/health/ready+/health/activeon both nodes; legstop.ps1, which wraps a stopwatch aroundStop-Serviceand then checks the service stays Stopped for 15 s.
| Leg 1: stop vq03 (Primary, sync client) | Leg 2: stop vq04 (Primary, sync server): the #906 proof | |
|---|---|---|
Stop-Service issued |
12:42:04.372Z | 12:44:07.472Z |
| Peer → 250 (subscription arrival) | vq04 250 at 12:42:04.728Z (0.36 s) | vq03 250 at 12:44:07.944Z (0.47 s) |
Peer /health/active |
200 on the first sample (12:42:04.938Z) | 200 on the first sample (12:44:07.600Z) |
Stop-Service duration |
6.10 s | 5.70 s (old build, same node, 1 h earlier: 30.1 s) |
| Stopping node's log | Application is shutting down 07:42:04.391 local → Host stopping: leaving the Akka cluster before the web server drains its open requests (lmxopcua#906). .394 → Leaving .409 → Akka CoordinatedShutdown completed in 647 ms |
Application is shutting down 07:44:07.479 → LocalDbSync/Sync responded 200 in 86702 ms .480 (the stream ended 1 ms after the stop) → … (lmxopcua#906). .483 → Leaving .500 → Akka CoordinatedShutdown completed in 236 ms (.718). No Sync responded 200 in ~30000+ ms wait. |
| Stays stopped | Stopped after 15 s, no host process (no SCM recovery restart) | same |
| Restart | 12:42:38.6Z → ready 12:42:46.9Z (8.3 s), -Verify OK 21/21. vq03 240 (first value 12:42:49Z, unchanged after), /health/active 503 on all samples: no failback |
12:46:03.3Z → ready 12:46:12.6Z (9.3 s), -Verify OK 21/21. vq04 240, 503: no failback |
#906 before / after (same node vq04, the sync server, same day):
Before (c9640f5c, the window's vq04 stop, 12:35Z) |
After (76e73e7d, leg 2, 12:44Z) |
|
|---|---|---|
Application is shutting down → Leaving |
30.0 s (07:35:26.975 → Sync responded 200 07:35:56.989 → Leaving 07:35:57.020) |
21 ms (07:44:07.479 → 07:44:07.500) |
Stop-Service |
30.1 s | 5.7 s |
| Peer at 250 | (vq03 was already Primary) — #780 record: ~31 s | 0.47 s |
Observation (not a #906 regression): on both legs the host logged nothing for ~5 s after CoordinatedShutdown completed. The gap ended with SERVER - Service Fault Occurred. Reason=BadTimeout: vq03 07:42:10.134, vq04 07:44:12.778.
The likely cause is the OPC UA stack closing the gate's own open subscribe session at server stop. That would put Stop-Service at ~6 s rather than the pack's 1–3 s target. The c9640f5c record's vq03 stop (no client session open) took 1 s.
The hand-over is not affected, because the cluster leave comes first. The peer took over within 0.5 s on both legs.
End state: vq03 250 (Primary), vq04 240, the same as before the window. Canary Good on both, sync connected, backlog 0.
Rollback material (left on the boxes)
DEV vd03 → c9640f5c:
- Script:
E:\ApiInstall\_stage-otopcua-76e73e7d\rollback.ps1(pack §6). It runsInvoke-OtOpcUaInstallSwap.ps1 -Candidate E:\ApiInstall\OtOpcUa.bak.20261113-pre-76e73e7d -PreviousInstall E:\ApiInstall\OtOpcUa.failed-76e73e7d -NoCarry -NoStart, mirrorsOtOpcUa-databack, starts and runs a full-Verify. - Previous install:
E:\ApiInstall\OtOpcUa.bak.20261113-pre-76e73e7d(=c9640f5c). - Backups:
E:\ApiInstall\OtOpcUa-backups\pre-20261113-otopcua-76e73e7d\{OtOpcUaHost.reg, OtOpcUa-data\}(protected). - ConfigDb:
OtOpcUaConfig-pre-20261113-otopcua-76e73e7d.bak(vd03 instance default backup dir). No migration, so this is a configuration fallback only. - No registry import is needed: the env was unchanged.
QA → c9640f5c:
- Stop both nodes, Secondary first.
- Run
E:\psrem-stage\otopcua-76e73e7d\rollback.ps1on each node. It swaps back with-NoCarry -NoStart(Reapply/Verify inside) and mirrorsdata\from the backup. - Start both, vq03 first.
- Run
start.ps1on each node (it includes-Verify).
Do not run a mixed pair: RedundancyStateChanged changed shape in this range. After a rollback, the #906 ~30 s stop of vq04 is back.
- Previous install:
E:\ApiInstall\OtOpcUa\app.bak.20261113-pre-76e73e7don each node. - Backups:
E:\psrem-stage\otopcua-76e73e7d\backup\{OtOpcUaHost-pre-76e73e7d.reg, data-pre-76e73e7d\, pki-pre-76e73e7d\}(protected). No.regimport is needed: the env was unchanged. - ConfigDb:
OtOpcUaConfig_pre-76e73e7d_20261113.bakon wonder-sql-vq03 (VERIFYONLY ok).
Deviations from the pack
- Target
76e73e7dinstead of47f0b98c. The tip adds only the #900 follow-up (HG Client 1.1.0); see the pack's header. One gate was added for it: vd03 HG Probe growing, with no write RPCs. PASS. - QA: the first
Move-Itemofapp\failed "file in use" on both nodes, right after the stop. Nothing moved, the service was Stopped, and the retry with the service stopped went through on both (vq04: a separate retry; vq03: an in-call retry inswap2.ps1).- The likely cause is the host process still exiting after SCM reported Stopped. The pre-#905 wrappers waited for the
process to exit;
Invoke-OtOpcUaInstallSwap.ps1does not. - It cost about 20 s of the vq03 pair outage (the retry re-runs
-Scope Outside). - Filed as lmxopcua#908. vd03 did not hit it.
- Fixed (lmxopcua#908, closed):
Invoke-OtOpcUaInstallSwap.ps1now waits, afterStopped, for the host process and every process imaged underInstallRootto exit (-ProcessExitTimeoutSeconds, default 60) and retries the first move on a sharing violation (-MoveRetrySeconds, default 10). The next deploy needs no hand retry.
- The likely cause is the host process still exiting after SCM reported Stopped. The pre-#905 wrappers waited for the
process to exit;
Stop-Serviceon the new build took 5.7–6.1 s instead of the 1–3 s target, because of the ~5 s OPC UA session close after the cluster leave (see "Observation"). The hand-over, the actual #906 criterion, was 0.36 s and 0.47 s.- Logs: the QA host log for the window rolled to
otopcua-20261113_001.logon vq04 (size limit). Read it from there.