Files
lmxopcua/docs/deploy/2026-11-13-next-vd03-qa-to-47f0b98c.md
T
Joseph Doherty 5baf05f634
v2-ci / unit-tests (push) Has been cancelled
docs(deploy): vd03 + QA deployed c9640f5c -> 76e73e7d (#905 #906 #900 #717, LocalDb 0.3.8)
Executed 2026-11-13 12:27-12:48Z; target moved from the pack's 47f0b98c to the tip 76e73e7d
(HG Client 1.1.0, behaviour unchanged; note added to the pack).

- vd03: ~14 s downtime (was ~101 s; #905 window 5.7 s), SL 250, FOCAS 21/22, Mode Unset,
  HG Probe 494 -> 536 with no write RPCs (added gate for #900).
- QA: pair outage 36.5 s (was ~91 s), 250/240, both failover legs, no failback, sync
  connected, canary Good. #906: vq04 (sync server) shutdown -> Leaving 30.0 s on c9640f5c,
  21 ms on 76e73e7d; peer at 250 in 0.47 s (leg 1: 0.36 s).
- Deviation: the first Move-Item of app\ failed "file in use" right after the stop on both
  QA nodes; a retry with the service stopped succeeded (lmxopcua#908).

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 07:51:01 -05:00

29 KiB
Raw Blame History

Next deploy: DEV vd03 and QA vq03/vq04 c9640f5c → master 47f0b98c

Executed 2026-11-13 12:27–12:48Z, target moved to 76e73e7d. Master gained one commit after this pack, 76e73e7d (#900 follow-up: HistorianGateway Client/Contracts 1.0.0 → 1.1.0; GatewayErrorReasons reads the typed HistorianGatewayException reason instead of parsing the gRPC trailer; behaviour unchanged). DEV and QA HG already run the 1.1.0 server (6365f34). git log 47f0b98c..76e73e7d showed nothing else. The install scripts are byte-identical to this pack's table, there are no migrations, and the only new package delta is HG Client 1.1.0. So the owner deployed the tip rather than pinning 47f0b98c. One gate was added: on vd03 the historian path still answers (HG /metrics Probe grows, no write RPCs). Record: 2026-11-13-vd03-qa-to-76e73e7d.md.

Status: PREPARED, NOT EXECUTED (2026-11-13). The owner approved the deploy itself, to run after the LocalDb 0.3.8 pin landed (it did: 47f0b98c, 11:55Z). This pack covers both tiers. The read-only preflight below (ConfigDb SELECTs, SMB reads of the box config, OPC UA / HTTP reads) ran from the Mac on 2026-11-13 11:55–12:00Z. Nothing was built, staged, stopped or run on any wonder box. No blocker found; no issue filed.

Target: origin/master 47f0b98c (47f0b98c6e0eeb663ee14d4e6fa063623c74408d, scadaproj#57). If master moves before the window, either pin to 47f0b98c or re-run §1–§3 for the new range.

Templates: the previous pack 2026-11-13-next-vd03-qa-to-c9640f5c.md and its record 2026-11-13-vd03-qa-to-c9640f5c.md (layout, staging, backups, gates), and the #780 record 2026-11-13-780-scriptedalarm-qa-and-vd03-flip.md (the env both tiers run now).

0. State since the c9640f5c deploy (all three nodes run c9640f5c)

  • vd03 runner flip (#780 Part 2, 2688bd5c): Scripting__Runner=OutOfProcess with seams Calculation, VirtualTag, ScriptedAlarm; the #901 record now has ConfigureHostService=1 and the host env has the two Scripting__Worker__* entries. Env 34 entries, ordered 69a493345099 / sorted caa047aaaf42. vd03 has no scripts, so the shards are not started. The installer on vd03 now also manages HostServiceEnv (it is in the Outside scope, below).
  • QA ScriptedAlarm seam (#780 Part 1): Scripting__Runner__Seams__2=ScriptedAlarm on both nodes. Env vq03 49 (5750bbeb4734 / 4263c3ac5a46), vq04 48 (ef146b2681b4 / d19e968212f3).
  • QA drill (003c7f89): PASS, finished 11:53Z. Config is back on the canary-only revision: a8e078f0, rev 82dd0598c37f41f8… (identical to 5905003e). 2 scripts, 1 VT, 0 scripted alarms, 0 role-mapping rows (the temporary row was deleted). Primary is now vq03 (250), vq04 240 (read 11:58Z, below).
  • vd03 #903 drift: the draft still differs from the sealed 595cb909 / fa9a2e28… because of the edit-and-revert in the c9640f5c gate (RowVersion is in the revision hash). The running configuration is the sealed one.

1. Range c9640f5c..47f0b98c (16 commits, same on both tiers)

Commit Change Effect on DEV / QA
dcff5ae9 #906 Host: GracefulClusterLeaveService runs Akka CoordinatedShutdown in StoppingAsync, before any hosted service stops, so the node leaves the cluster before Kestrel drains. ShutdownStreamCancellationInterceptor (innermost on the gRPC pipeline) ends open streaming calls on ApplicationStopping and refuses new ones Unavailable. QA: a planned stop of the node that serves the LocalDb sync stream (vq04; vq03 is the client, it has LocalDb__Replication__PeerAddress) no longer waits ~30 s. The #780 record measured vq04 stop → vq03 250 in ~31 s; the target is ~1 s. vd03 (single node): stop leaves a 1-member cluster first; faster or equal. New stop log lines (§5).
47f0b98c LocalDb 0.3.8 (scadaproj#57) Library side of #906: LocalDbSyncService ends open sync streams on ApplicationStopping and refuses new ones Unavailable. Wire-compatible with 0.3.7; no schema change. The host interceptor stays (it also covers the mesh streams). Both tiers: package bump only. QA: the pair runs the same version after stop-both/start-both.
0c19f26c #905 Invoke-OtOpcUaInstallSwap.ps1: -Reapply/-Verify -Scope Outside (full coverage scan, firewall, host env, records) before the stop, only -Scope InstallRoot inside the window; preflight checks the candidate carries the worker exe; prints stopped window (…): N s [stop, move, carry, isolation]. Coverage scan ~20× faster per item. Changes the deploy procedure (§4). windev vd03-like tree: stopped window 153 s → 6 s. No runtime change.
7829b937, f817c77e, 687256d6, c569f806 #717 Warm non-transparent redundancy, default-off (OpcUa:Redundancy:Mode Unset). New: options validation of Mode/PeerApplicationUris (Hot/unknown fail start; Warm needs peers; None needs none), a startup guard (Warm without the driver role fails start), revisioned RedundancyStateChanged snapshots, standby projection hooks in the node manager / SDK server, new metrics otopcua.redundancy.{warm_role,warm_transition,snapshot_rejected,mode_mismatch}, otopcua.opcua.standby.*. c569f806 = T13 fixes (Warm crash survivor stays Active; verdict reads its own member events). Inert under Unset on every tier: no metadata written, no snapshot ordering, no member-event subscription, no deploy-time mode cross-check (CheckRedundancyMode returns on Unset). Evidence: T13 L9 + the Unset re-check after c569f806 (docs/plans/evidence/2026-11-13-717-t13/RESULTS-unset-recheck.md: 250/240, Good on both, writes + alarms on the Secondary, graceful 259 ms hand-over, crash path as documented, no failback; A/B identical to the pre-fix build). RedundancyStateChanged gained an optional Revision: do not run a mixed c9640f5c/47f0b98c pair (QA stops both anyway).
f826af8e, 36eb6d55, a42c9d7f, 2238da65, 8667ce58 #717 docs, spec/plan, T0/T13 evidence (incl. a docker-dev Warm overlay) —
6e67250e, c0ba450d, 2688bd5c, 003c7f89 deploy docs (c9640f5c pack + record, #780 records) —

What a ScadaBridge consumer would notice

Nothing on DEV/QA (ScadaBridge there has no OtOpcUa data connection). For an OtOpcUa-bound client: a planned stop of the Primary hands over in about 1 s instead of up to ~30 s, and the stopping node's sessions end sooner. Under Unset RedundancySupport stays None (0), ServerUriArray (i=11314) stays the SDK orphan (Good [] by NodeId), the Secondary keeps serving Good at 240. No proto, NodeId or ServiceLevel-semantics change.

2. Migrations and settings

  • ConfigDb migrations: none. Nothing under src/Core/ZB.MOM.WW.OtOpcUa.Configuration changed in the range. The repo head stays 20261022102459_EnableSnapshotIsolation (9 migrations); both DBs are there (§3).
  • Deployment artifact format: unchanged. DeploymentArtifact.ReadClusterRedundancyMode is read-only; the composer did not change, so the revision hash of an unchanged draft does not move.
  • New optional settings (leave all unset):
    • OpcUa:Redundancy:Mode = Unset (default) | None | Warm. Hot or any other value fails host start. Warm without the driver role fails start. No tier sets it until ScadaBridge#138 ships (owner decision).
    • OpcUa:Redundancy:PromoteWhilePrimaryUnreachable (default true; Warm only).
    • OpcUa:PeerApplicationUris: unchanged meaning; validated only under None (must be empty) / Warm (≥1, distinct, not self). Neither tier sets it (§3).
  • Latent, for the later Warm flip (not this deploy): ServerCluster.RedundancyMode is Warm (1) on QA and None (0) on DEV, and the sealed artifacts carry it. Under Unset the cross-check is skipped. Under a future local Mode=Warm QA matches; a local Mode=None on QA would log the mode_mismatch Error. Nothing to do now.
  • Shutdown: no new key. The host's ShutdownTimeout (30 s default) still bounds the CoordinatedShutdown wait; on timeout it logs a Warning and continues.
  • Required env: no new variable. Existing ones unchanged (pepper, JWT key, QA LocalDb__Replication__ApiKey byte-identical on both nodes).
  • Packages: ZB.MOM.WW.LocalDb(+.Replication) 0.3.7 → 0.3.8. Everything else as c9640f5c (Telemetry 0.3.1, MxGateway.Client 0.4.0, HistorianGateway.Client 1.0.0, Grpc.StatusProto 2.76.0, Auth 0.4.1, Secrets 0.6.2, Health 0.3.0, Theme 0.5.0).
  • Peer services: none to upgrade first. HG is now 6365f34 on vd03 and QA (≥ a220ec0, so #900 is live; no historized tags on either tier). mxgw 76a71bc8 unaffected (no Galaxy driver).

3. Read-only preflight (run 2026-11-13 11:55–12:00Z from the Mac)

SQL: wwadmin, sqlcmd -C -I, password in SQLCMDPASSWORD, DB OtOpcUaConfig on wonder-app-vd03.zmr.zimmer.com,1433 (DEV) and wonder-sql-vq03.zmr.zimmer.com,1433 (QA). SELECTs only: scripts/preflight/vd03-preupgrade-b8b4c18f-configdb-checks.sql unchanged, plus ad-hoc SELECTs on __EFMigrationsHistory, sys.databases, ServerCluster, ClusterNode, Deployment (incl. DECOMPRESS of the latest Sealed artifact), NodeDeploymentState, LdapGroupRoleMapping and inventory counts.

Check DEV vd03 QA (shared ConfigDb on wonder-sql-vq03)
__EFMigrationsHistory 9 rows, head 20261022102459_EnableSnapshotIsolation = repo head at 47f0b98c. Snapshot ON, RCSI off, compat 160. Same: 9 rows, same head, snapshot ON, compat 160.
ServerCluster DEV/DEV, NodeCount 1, RedundancyMode None DEV/QA, NodeCount 2, RedundancyMode Warm (see §2 "Latent")
ClusterNode wonder-app-vd03.zmr.zimmer.com:4053, ApplicationUri urn:OtOpcUa:wonder-app-vd03 (the live endpoint reports urn:OtOpcUa: no env override, as before) 10.220.158.2:4053 urn:wonder-app-vq03:OtOpcUa, 10.220.158.3:4053 urn:wonder-app-vq04:OtOpcUa, both Enabled, MaintenanceMode 0
Latest Sealed deployment 595cb909, rev fa9a2e28… (06:34Z), applied (Status 1) on the node; artifact RedundancyMode":0 a8e078f0, rev 82dd0598… (11:53Z, the post-drill restore), applied on both nodes; artifact RedundancyMode":1
Inventory 1 driver (FOCAS), 22 tags, 0 VT, 0 scripts, 0 scripted alarms 1 driver (Calculation canary), 1 tag, 1 VT, 2 scripts, 0 scripted alarms
LdapGroupRoleMapping 0 rows 0 rows (the #780 drill row is gone)
Preflight script checks 1–12 all ok all ok
Check 0 "Environment" attention: stale expectation (written for b8b4c18f); not an issue same
Check 13 FOCAS renames 9 info rows (live since 2026-11-12) ok

OpcUa:Redundancy:Mode is set nowhere (nor OpcUa:PeerApplicationUris, nor any Redundancy key):

  • appsettings (SMB read): appsettings.json, .admin.json, .driver.json, .admin-driver.json, .Development.json in E:\ApiInstall\OtOpcUa\ (vd03) and E:\ApiInstall\OtOpcUa\app\ (vq03, vq04): no Redundancy / PeerApplicationUris key. No appsettings.Production.json on any box.
  • Service env (SMB read of the latest .reg backups, names only, values never decoded to output): vd03 OtOpcUa-backups\pre-20261113-780-runner-flip\OtOpcUaHost.reg (28 names, pre-flip) and QA E:\psrem-stage\otopcua-780-scriptedalarm\backup\OtOpcUaHost-pre-780-seam2.reg (48 / 47 names, pre-seam): no OpcUa__Redundancy__*, no OpcUa__PeerApplicationUris__*. The #780 changes after those backups added only Scripting__* entries (the record's env diffs). Re-confirm on-box in §4 step 1 (names only), because the live env is newer than the backups.
  • ConfigDb: the ServerCluster.RedundancyMode column is not the host setting (see §2).

Live baseline (Mac, 11:58Z):

vd03 vq03 vq04
/health/ready / /health/active 200 / 200 200 / 200 200 / 503
ServiceLevel (i=2267) 250 250 (Primary) 240
RedundancySupport (i=3709) 0 (None) 0 0
ServerUriArray (i=11314) / ServerArray (i=2254) Good String[] / Good String[] same same
Runner (vq03 /metrics) — runner_state 0, worker_up shard 0/1 = 1, iso 0; script-runner Ready, quarantined 0 —

redundancy CLI (src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI): Mode None, no server URIs, on all three. These are the "unchanged under Unset" reference values for §5.

Verdict: no blocker on either tier; no issue filed. Re-run the SQL (it takes a minute) just before the window if the configuration has changed.

4. Build, stage, swap

Build (Mac, isolated worktree at 47f0b98c; not done here)

git worktree add ../OtOpcUa-wt-47f0b98c 47f0b98c && cd ../OtOpcUa-wt-47f0b98c
dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj -c Release -r win-x64 --self-contained true -o ../publish-47f0b98c
# expect ~1398 files (as c9640f5c), OtOpcUa.Host.dll ProductVersion 1.0.0+47f0b98c…, script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe present;
# deps.json: ZB.MOM.WW.LocalDb + .Replication 0.3.8 (the only package change), Telemetry 0.3.1, HistorianGateway.Client 1.0.0
cd ../publish-47f0b98c && zip -qr ../otopcua-47f0b98c.zip . && shasum -a 256 ../otopcua-47f0b98c.zip > ../otopcua-47f0b98c.zip.sha256

Copy the zip + .sha256 over SMB; check the hash on the box before expanding. Stage the three install scripts from 47f0b98c (the swap invokes the installer next to it, so the #905 -Scope logic must be the staged one):

File SHA-256 at 47f0b98c
scripts/install/Install-ScriptWorkerIsolation.ps1 bb31d2759d0de29292c123d74860226ed37af554f2c8fe6871312c7e81f7c05a
scripts/install/ScriptWorkerIsolation.psm1 c75a6e3e2c0bfbb58a360e92249efeb02ac025b50035ea73b95c32304c7dbff3
scripts/install/Invoke-OtOpcUaInstallSwap.ps1 09a0bb6874af23a726ec349bd7bd86615832d08c58d0e70f864642afda3bc866

On-box steps below are deploy steps (winrm skill, deploy scope), in an elevated Windows PowerShell from the staged install\ folder. Never run Install-Services.ps1, Uninstall-Services.ps1 or Refresh-Services.ps1.

The #901 settings records already exist on all three nodes (written in the c9640f5c window; vd03's re-written by the #780 flip with ConfigureHostService=1). No one-time full-argument install this time. Do not re-run the installer with arguments; the swap reads the record.

Data backup inside the window, Outside scope outside it. The swap stops the service itself. To keep the #905 gain, start the swap with the service running and -NoStart: it runs the Outside scope while the old build serves, stops, swaps, runs the InstallRoot scope and leaves the service stopped. Then back up data (consistent SQLite triplets) and start. Stopping the service by hand first would move the ~20+ s Outside scan back into the window.

DEV vd03 (E:\ApiInstall\OtOpcUa, data E:\ApiInstall\OtOpcUa-data, runner OutOfProcess, single node)

Staging E:\ApiInstall\_stage-otopcua-47f0b98c\ (zip, .sha256, install\). Backups E:\ApiInstall\OtOpcUa-backups\pre-20261113-otopcua-47f0b98c\, created with a protected DACL (SYSTEM + Administrators, #891).

  1. Prepare (service up).
    • reg export HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaHost into the backup folder.
    • Hash check, expand to E:\ApiInstall\OtOpcUa.new-47f0b98c, check ProductVersion and deps.json LocalDb 0.3.8.
    • BACKUP DATABASE [OtOpcUaConfig] TO DISK = N'OtOpcUaConfig-pre-20261113-otopcua-47f0b98c.bak' WITH COPY_ONLY, CHECKSUM (configuration fallback only; no migration).
    • Env digest (names + SHA-256 prefixes only): expect 34, 69a493345099 / caa047aaaf42; no OpcUa__Redundancy__*, no OpcUa__PeerApplicationUris__*; Scripting__Runner=OutOfProcess + 3 seams + 2 Scripting__Worker__*.
    • Pre-window isolation check with the new installer, service up: .\Install-ScriptWorkerIsolation.ps1 -Verify → VERIFY OK (14/14 last time), exit 0. Proves the 47f0b98c installer reads the existing record.
    • .\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -Candidate E:\ApiInstall\OtOpcUa.new-47f0b98c -PreviousInstall E:\ApiInstall\OtOpcUa.bak.20261113-pre-47f0b98c -WhatIf → preflight only (record found, candidate has the worker exe, same volume, PreviousInstall free).
    • Snapshot HG /metrics (http://wonder-app-vd03.zmr.zimmer.com:5220/metrics, historian_gateway_grpc_requests_total).
  2. Swap (service running when started):
    .\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -Candidate E:\ApiInstall\OtOpcUa.new-47f0b98c `
        -PreviousInstall E:\ApiInstall\OtOpcUa.bak.20261113-pre-47f0b98c -NoStart
    
    • Default -Carry (appsettings*.json, pki, logs) is right for vd03.
    • Expect pre-stop: -Reapply -Scope Outside all unchanged (incl. HostServiceEnv, SettingsRecord), -Verify -Scope Outside OK. In the window: -Reapply -Scope InstallRoot changes WorkerFolderAcl, InstallRootDeny, the pki SensitiveDirDeny; -Verify -Scope InstallRoot OK.
    • Record the stop -> swapped and verified (left stopped): N s [stop …, move …, carry …, isolation …] line.
    • Gate: no throw. A pre-stop failure leaves the old build serving (nothing moved). A post-swap failure leaves it stopped: fix, run -Reapply/-Verify (no -Scope), then start.
  3. Backup + start: robocopy E:\ApiInstall\OtOpcUa-data <backup>\OtOpcUa-data /E /COPY:DAT (≈0.5 MB), then Start-Service OtOpcUaHost. Note the start time; poll /health/ready.
  4. After ready: .\Install-ScriptWorkerIsolation.ps1 -Verify (full) → OK, exit 0. Env digest again: 34, unchanged.

QA vq03 + vq04 (stop both, swap both, start both)

Layout E:\ApiInstall\OtOpcUa\{app,data,pki}, service OtOpcUaHost (LocalSystem), runner OutOfProcess (3 seams). Staging E:\psrem-stage\otopcua-47f0b98c\ (zip, .sha256, install\), backups in …\backup\ with a protected DACL.

  1. Prepare (both nodes, services up).
    • Expand to E:\ApiInstall\OtOpcUa\app.new-47f0b98c (hash, ProductVersion, deps.json LocalDb 0.3.8).
    • reg export …\OtOpcUaHost backup\OtOpcUaHost-pre-47f0b98c.reg.
    • Env digest, names only: expect vq03 49 5750bbeb4734 / 4263c3ac5a46, vq04 48 ef146b2681b4 / d19e968212f3; Seams__0..2 = Calculation, VirtualTag, ScriptedAlarm; no OpcUa__Redundancy__* or OpcUa__PeerApplicationUris__*; pepper and LocalDb__Replication__ApiKey entry prefixes equal on both nodes.
    • Once: BACKUP DATABASE OtOpcUaConfig … COPY_ONLY, CHECKSUM, COMPRESSION on wonder-sql-vq03 → OtOpcUaConfig_pre-47f0b98c_20261113.bak, then RESTORE VERIFYONLY.
    • Pre-window .\Install-ScriptWorkerIsolation.ps1 -Verify with the new installer on each node → OK (21/21 last time).
    • The swap with -WhatIf on each node (preflight only).
    • ServiceLevel on both: expect vq03 250 / vq04 240 (as at 11:58Z). Snapshot vq03 HG /metrics (:5220).
  2. Swap the Secondary (vq04) first, while vq03 serves (service running when started):
    .\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -Candidate E:\ApiInstall\OtOpcUa\app.new-47f0b98c `
        -PreviousInstall E:\ApiInstall\OtOpcUa\app.bak.20261113-pre-47f0b98c -Carry 'appsettings*.json','logs' -NoStart
    
    • Expect the stop part of this window to be ~30 s: vq04 is still c9640f5c and it serves vq03's open LocalDb sync stream (the #906 bug, last seen on this build). It is outside the pair outage (vq03 serves) and inside the default -StopTimeoutSeconds 120. Record it as the "before" control for the #906 gate.
    • Expect -Reapply -Scope Outside all unchanged; -Scope InstallRoot changes WorkerFolderAcl, InstallRootDeny; both -Verify OK. Then robocopy data backup\data-pre-47f0b98c /E /COPY:DAT and the same for pki.
  3. Swap the Primary (vq03), same command. The pair outage starts at its stop. Its stop should take a few seconds (the peer is down, no inbound stream). Then the data + pki robocopy.
    • Gate: both swaps returned without throwing, both left stopped. If -Verify failed on either, start neither: fix, -Reapply/-Verify, then go on. If vq04's swap failed pre-stop, vq04 is still serving the old build: stop and re-plan rather than starting a mixed pair.
  4. Start both: vq03 first (the old Primary), then vq04 at once. Record each start → /health/ready.
  5. On each node after ready: .\Install-ScriptWorkerIsolation.ps1 -Verify (full) → OK, exit 0. Env digests unchanged.

5. Post-deploy gates

From the Mac over HTTP / opc.tcp (the Client CLI: dotnet ZB.MOM.WW.OtOpcUa.Client.CLI.dll read|redundancy|subscribe -u …), logs over SMB (logs\otopcua-YYYYMMDD.log; an open log's listed size reads 0, read the contents).

Gate DEV vd03 QA vq03 + vq04
Swap output -Verify -Scope Outside + -Scope InstallRoot OK; the window line recorded same, both nodes
Isolation full -Verify exit 0 after the start same, both nodes
Version live OtOpcUa.Host.dll (SMB copy) ProductVersion 1.0.0+47f0b98c…, SHA-256 = the published build; live ZB.MOM.WW.LocalDb.dll file version 0.3.8 same, both nodes
Health /healthz, /health/ready (5/5, script-runner: Out-of-process script runner ready., shards not started), /health/active 200 /health/ready 200, 5/5 incl. runner ready, on both; /health/active 200 Primary / 503 Secondary
ServiceLevel 250 ×3 fresh sessions 250 / 240 ×3, stable
Redundancy metadata unchanged under Unset redundancy CLI: Mode None, no server URIs; i=3709 = 0 Good; i=11314 Good String[] (empty); i=2254 Good — all as the §3 baseline. Log: no Warm redundancy / mode_mismatch lines; /metrics: no otopcua_redundancy_warm_transition/snapshot_rejected/mode_mismatch counts, otopcua_opcua_standby_* absent or 0 same on both nodes; the Secondary keeps serving Good data at 240 (canary reads on vq04)
Failover + #906 timing (QA) — Run a background subscribe -n i=2267 -i 250 on each endpoint and a 0.5 s /health/active poll on both. Leg 1: stop the Primary (vq03, the sync client) with a stopwatch around Stop-Service: vq04 → 250 and /health/active 200 on its first samples. Restart vq03 → 240, never above, no failback. Leg 2 (the #906 proof): stop vq04 (now Primary, the sync server): target Stop-Service ≈ 1–3 s and vq03 at 250 within ~1–2 s (before: ~30 s stop, vq03 250 after ~31 s — #780 record; and step 2's control). vq04's log must show Host stopping: leaving the Akka cluster before the web server drains its open requests (lmxopcua#906). then Akka CoordinatedShutdown completed in N ms, and no …/LocalDbSync/Sync responded 200 in ~30000+ ms. The service must stay Stopped (no SCM recovery restart). Restart vq04 → 240, no failback. End state: vq03 250, vq04 240. Record both legs' stop durations and hand-over times.
LocalDb sync (QA) — localdb-replication connected, oplog backlog 0 on both, after the start and after each failover leg
Canary + runner (QA) — ns=2;s=script-runner-canary/Device1/canary-calc-seconds and ns=3;s=EQ-ec3979550d90/canary-vt-second Good and advancing ~1/s on both endpoints, after the start and after each leg. otopcua_script_worker_up{shard="0"} = 1 and {shard="1"} = 1 on both; runner_state 0; quarantined 0.
ScriptedAlarm seam (QA) (vd03: env carries Seams__2=ScriptedAlarm; no scripts, so nothing runs) Seam ready = env digest unchanged with Scripting__Runner__Seams__2=ScriptedAlarm + runner Ready on both nodes. There is no scripted alarm in the canary-only config, so otopcua_script_ipc_roundtrip{kind="alarm"} will not count; the live alarm proof is the 11:44–11:53Z drill on c9640f5c. Re-running the drill's normal alarm (canary-alarm-cycle) is optional and needs the owner's AdminUI sign-in + the temporary role row, then the cleanup back to rev 82dd0598….
Driver data (vd03) FOCAS raw: 22 variables, 21 Good, Program/BlockCount BadNoData (as before); 2 UNS variables —
Boot log roles=[admin,driver]; credential-reference preflight "every stored and deployed reference is namespaced."; Historian gateway probe: the gateway is reachable; raw subtree materialised (… failed=0); no FTL, no options-validation failure (OpcUa:Redundancy…); ERR only the known certificate-domain noise from reads by IP same, both nodes, plus the known standby /health/* 503 and bootstrap akka … Removed lines
Historian writes HG :5220/metrics before/after: only HistorianStatus/Probe grows. No EnsureTags, AddTagExtendedProperties, WriteLiveValues, AddHistoricalValues, SendEvent(s). same on vq03 HG :5220
#903 drift chip (vd03) /deployments still shows "Configuration drift" (the c9640f5c edit-and-revert; see Rollback) and the nodes serve 595cb909 / fa9a2e28…. Do not "Deploy current configuration" in the window: it would seal a new revision; that is the owner's call. Optional: /deployments should read "In sync" (draft = a8e078f0); needs the AdminUI sign-in
Overview / ScadaBridge :5320 OtOpcUa Up; ScadaBridge :8085/health/ready 200 :5320 on both: OtOpcUa Up, 2 members / 0 unreachable; ScadaBridge 200 on both

6. Rollback

No schema undo (no migration). The ConfigDb .bak is a configuration fallback only. The env changes since c9640f5c was installed (#780) live in the registry and already ran on c9640f5c, so they stay valid. The isolation record and denies are build-independent; never run -Uninstall as part of a rollback.

DEV vd03 → c9640f5c:

.\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -Candidate E:\ApiInstall\OtOpcUa.bak.20261113-pre-47f0b98c `
    -PreviousInstall E:\ApiInstall\OtOpcUa.failed-47f0b98c -NoCarry -NoStart

Then mirror <backup>\OtOpcUa-data back over E:\ApiInstall\OtOpcUa-data, Start-Service OtOpcUaHost, full -Verify, health, ServiceLevel 250, FOCAS 21/22. The registry is not changed by this deploy, so no import. The older E:\ApiInstall\OtOpcUa.bak.20261113-pre-c9640f5c (= 6c4d2990) is one generation further back; do not use it here.

vd03 #903 drift note. The /deployments chip shows "Configuration drift" before this deploy, after it and after a rollback: the draft's re-saved dbo.Tag row has a new RowVersion, and RowVersion is serialised into the artifact whose SHA-256 is the revision hash. The content is identical to the sealed fa9a2e28…. A rollback does not clear it, and restoring OtOpcUaConfig-pre-20261113-otopcua-47f0b98c.bak restores the same drifted draft (the …-c9640f5c.bak predates the edit and would clear it, but would also roll back anything else saved since). It is not a rollback trigger. Only a new deployment clears it (owner's call).

QA → c9640f5c: stop both (Secondary first), then on each node:

.\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -Candidate E:\ApiInstall\OtOpcUa\app.bak.20261113-pre-47f0b98c `
    -PreviousInstall E:\ApiInstall\OtOpcUa\app.failed-47f0b98c -NoCarry -NoStart

Mirror backup\data-pre-47f0b98c back over data\ (and pki if it changed), full -Verify on each (exit 0), start both, vq03 first. No .reg import (the env is not touched). DB fallback OtOpcUaConfig_pre-47f0b98c_20261113.bak. Do not run a mixed pair: RedundancyStateChanged changed shape in this range. After a rollback the #906 ~30 s stop of vq04 is back; expect it.

7. Downtime estimates (#905 numbers)

Tier Previous (c9640f5c window) Estimate for this pack
vd03 ~101 s stop → ready: the full coverage walk of E:\ApiInstall ran twice inside the window ~20–25 s: stop of the old build (2–6 s) + move/carry (<1 s) + -Scope InstallRoot reapply/verify (~6 s on windev's vd03-like tree: 153 s → 6 s on PS 5.1) + OtOpcUa-data robocopy (<1 s) + start → ready (~9 s, the #780 flip restart). The Outside part (~23 s on windev) runs before the stop.
QA pair outage ~91 s (Primary stop → both ready), with 50–60 s of -Reapply/-Verify per node ~30–40 s: vq03's stop (a few s; no inbound stream once vq04 is down) + ~6–10 s InstallRoot scope + backup (<1 s) + operator gap + start → ready (9–16 s; the first node waits for its peer). vq04's ~30 s old-build stop is outside the outage.
QA failover legs Primary stop → peer 250: ~3 s when vq03 stopped (c9640f5c record); ~31 s when vq04 stopped (#780 record) ~1–2 s on both legs (#906; GracefulStopWithOpenSyncStreamTests stop 895 ms)