Executed 2026-11-13 12:27-12:48Z; target moved from the pack's47f0b98cto the tip76e73e7d(HG Client 1.1.0, behaviour unchanged; note added to the pack). - vd03: ~14 s downtime (was ~101 s; #905 window 5.7 s), SL 250, FOCAS 21/22, Mode Unset, HG Probe 494 -> 536 with no write RPCs (added gate for #900). - QA: pair outage 36.5 s (was ~91 s), 250/240, both failover legs, no failback, sync connected, canary Good. #906: vq04 (sync server) shutdown -> Leaving 30.0 s onc9640f5c, 21 ms on 76e73e7d; peer at 250 in 0.47 s (leg 1: 0.36 s). - Deviation: the first Move-Item of app\ failed "file in use" right after the stop on both QA nodes; a retry with the service stopped succeeded (lmxopcua#908). Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
29 KiB
Next deploy: DEV vd03 and QA vq03/vq04 c9640f5c → master 47f0b98c
Executed 2026-11-13 12:27–12:48Z, target moved to
76e73e7d. Master gained one commit after this pack,76e73e7d(#900 follow-up: HistorianGateway Client/Contracts 1.0.0 → 1.1.0;GatewayErrorReasonsreads the typedHistorianGatewayExceptionreason instead of parsing the gRPC trailer; behaviour unchanged). DEV and QA HG already run the 1.1.0 server (6365f34).git log 47f0b98c..76e73e7dshowed nothing else. The install scripts are byte-identical to this pack's table, there are no migrations, and the only new package delta is HG Client 1.1.0. So the owner deployed the tip rather than pinning47f0b98c. One gate was added: on vd03 the historian path still answers (HG/metricsProbe grows, no write RPCs). Record:2026-11-13-vd03-qa-to-76e73e7d.md.
Status: PREPARED, NOT EXECUTED (2026-11-13). The owner approved the deploy itself, to run after the LocalDb 0.3.8 pin landed (it did:
47f0b98c, 11:55Z). This pack covers both tiers. The read-only preflight below (ConfigDb SELECTs, SMB reads of the box config, OPC UA / HTTP reads) ran from the Mac on 2026-11-13 11:55–12:00Z. Nothing was built, staged, stopped or run on any wonder box. No blocker found; no issue filed.Target:
origin/master47f0b98c(47f0b98c6e0eeb663ee14d4e6fa063623c74408d, scadaproj#57). If master moves before the window, either pin to47f0b98cor re-run §1–§3 for the new range.Templates: the previous pack
2026-11-13-next-vd03-qa-to-c9640f5c.mdand its record2026-11-13-vd03-qa-to-c9640f5c.md(layout, staging, backups, gates), and the #780 record2026-11-13-780-scriptedalarm-qa-and-vd03-flip.md(the env both tiers run now).
0. State since the c9640f5c deploy (all three nodes run c9640f5c)
- vd03 runner flip (#780 Part 2,
2688bd5c):Scripting__Runner=OutOfProcesswith seams Calculation, VirtualTag, ScriptedAlarm; the #901 record now has ConfigureHostService=1 and the host env has the twoScripting__Worker__*entries. Env 34 entries, ordered69a493345099/ sortedcaa047aaaf42. vd03 has no scripts, so the shards arenot started. The installer on vd03 now also managesHostServiceEnv(it is in the Outside scope, below). - QA ScriptedAlarm seam (#780 Part 1):
Scripting__Runner__Seams__2=ScriptedAlarmon both nodes. Env vq03 49 (5750bbeb4734/4263c3ac5a46), vq04 48 (ef146b2681b4/d19e968212f3). - QA drill (
003c7f89): PASS, finished 11:53Z. Config is back on the canary-only revision:a8e078f0, rev82dd0598c37f41f8…(identical to5905003e). 2 scripts, 1 VT, 0 scripted alarms, 0 role-mapping rows (the temporary row was deleted). Primary is now vq03 (250), vq04 240 (read 11:58Z, below). - vd03 #903 drift: the draft still differs from the sealed
595cb909/fa9a2e28…because of the edit-and-revert in thec9640f5cgate (RowVersionis in the revision hash). The running configuration is the sealed one.
1. Range c9640f5c..47f0b98c (16 commits, same on both tiers)
| Commit | Change | Effect on DEV / QA |
|---|---|---|
dcff5ae9 #906 |
Host: GracefulClusterLeaveService runs Akka CoordinatedShutdown in StoppingAsync, before any hosted service stops, so the node leaves the cluster before Kestrel drains. ShutdownStreamCancellationInterceptor (innermost on the gRPC pipeline) ends open streaming calls on ApplicationStopping and refuses new ones Unavailable. |
QA: a planned stop of the node that serves the LocalDb sync stream (vq04; vq03 is the client, it has LocalDb__Replication__PeerAddress) no longer waits ~30 s. The #780 record measured vq04 stop → vq03 250 in ~31 s; the target is ~1 s. vd03 (single node): stop leaves a 1-member cluster first; faster or equal. New stop log lines (§5). |
47f0b98c LocalDb 0.3.8 (scadaproj#57) |
Library side of #906: LocalDbSyncService ends open sync streams on ApplicationStopping and refuses new ones Unavailable. Wire-compatible with 0.3.7; no schema change. The host interceptor stays (it also covers the mesh streams). |
Both tiers: package bump only. QA: the pair runs the same version after stop-both/start-both. |
0c19f26c #905 |
Invoke-OtOpcUaInstallSwap.ps1: -Reapply/-Verify -Scope Outside (full coverage scan, firewall, host env, records) before the stop, only -Scope InstallRoot inside the window; preflight checks the candidate carries the worker exe; prints stopped window (…): N s [stop, move, carry, isolation]. Coverage scan ~20× faster per item. |
Changes the deploy procedure (§4). windev vd03-like tree: stopped window 153 s → 6 s. No runtime change. |
7829b937, f817c77e, 687256d6, c569f806 #717 |
Warm non-transparent redundancy, default-off (OpcUa:Redundancy:Mode Unset). New: options validation of Mode/PeerApplicationUris (Hot/unknown fail start; Warm needs peers; None needs none), a startup guard (Warm without the driver role fails start), revisioned RedundancyStateChanged snapshots, standby projection hooks in the node manager / SDK server, new metrics otopcua.redundancy.{warm_role,warm_transition,snapshot_rejected,mode_mismatch}, otopcua.opcua.standby.*. c569f806 = T13 fixes (Warm crash survivor stays Active; verdict reads its own member events). |
Inert under Unset on every tier: no metadata written, no snapshot ordering, no member-event subscription, no deploy-time mode cross-check (CheckRedundancyMode returns on Unset). Evidence: T13 L9 + the Unset re-check after c569f806 (docs/plans/evidence/2026-11-13-717-t13/RESULTS-unset-recheck.md: 250/240, Good on both, writes + alarms on the Secondary, graceful 259 ms hand-over, crash path as documented, no failback; A/B identical to the pre-fix build). RedundancyStateChanged gained an optional Revision: do not run a mixed c9640f5c/47f0b98c pair (QA stops both anyway). |
f826af8e, 36eb6d55, a42c9d7f, 2238da65, 8667ce58 |
#717 docs, spec/plan, T0/T13 evidence (incl. a docker-dev Warm overlay) | — |
6e67250e, c0ba450d, 2688bd5c, 003c7f89 |
deploy docs (c9640f5c pack + record, #780 records) |
— |
What a ScadaBridge consumer would notice
Nothing on DEV/QA (ScadaBridge there has no OtOpcUa data connection). For an OtOpcUa-bound client: a planned stop of the
Primary hands over in about 1 s instead of up to ~30 s, and the stopping node's sessions end sooner. Under Unset
RedundancySupport stays None (0), ServerUriArray (i=11314) stays the SDK orphan (Good [] by NodeId), the
Secondary keeps serving Good at 240. No proto, NodeId or ServiceLevel-semantics change.
2. Migrations and settings
- ConfigDb migrations: none. Nothing under
src/Core/ZB.MOM.WW.OtOpcUa.Configurationchanged in the range. The repo head stays20261022102459_EnableSnapshotIsolation(9 migrations); both DBs are there (§3). - Deployment artifact format: unchanged.
DeploymentArtifact.ReadClusterRedundancyModeis read-only; the composer did not change, so the revision hash of an unchanged draft does not move. - New optional settings (leave all unset):
OpcUa:Redundancy:Mode=Unset(default) |None|Warm.Hotor any other value fails host start.Warmwithout thedriverrole fails start. No tier sets it until ScadaBridge#138 ships (owner decision).OpcUa:Redundancy:PromoteWhilePrimaryUnreachable(defaulttrue; Warm only).OpcUa:PeerApplicationUris: unchanged meaning; validated only underNone(must be empty) /Warm(≥1, distinct, not self). Neither tier sets it (§3).
- Latent, for the later Warm flip (not this deploy):
ServerCluster.RedundancyModeis Warm (1) on QA and None (0) on DEV, and the sealed artifacts carry it. Under Unset the cross-check is skipped. Under a future localMode=WarmQA matches; a localMode=Noneon QA would log themode_mismatchError. Nothing to do now. - Shutdown: no new key. The host's
ShutdownTimeout(30 s default) still bounds the CoordinatedShutdown wait; on timeout it logs a Warning and continues. - Required env: no new variable. Existing ones unchanged (pepper, JWT key, QA
LocalDb__Replication__ApiKeybyte-identical on both nodes). - Packages:
ZB.MOM.WW.LocalDb(+.Replication) 0.3.7 → 0.3.8. Everything else asc9640f5c(Telemetry 0.3.1, MxGateway.Client 0.4.0, HistorianGateway.Client 1.0.0, Grpc.StatusProto 2.76.0, Auth 0.4.1, Secrets 0.6.2, Health 0.3.0, Theme 0.5.0). - Peer services: none to upgrade first. HG is now
6365f34on vd03 and QA (≥a220ec0, so #900 is live; no historized tags on either tier). mxgw76a71bc8unaffected (no Galaxy driver).
3. Read-only preflight (run 2026-11-13 11:55–12:00Z from the Mac)
SQL: wwadmin, sqlcmd -C -I, password in SQLCMDPASSWORD, DB OtOpcUaConfig on
wonder-app-vd03.zmr.zimmer.com,1433 (DEV) and wonder-sql-vq03.zmr.zimmer.com,1433 (QA). SELECTs only:
scripts/preflight/vd03-preupgrade-b8b4c18f-configdb-checks.sql unchanged, plus ad-hoc SELECTs on __EFMigrationsHistory,
sys.databases, ServerCluster, ClusterNode, Deployment (incl. DECOMPRESS of the latest Sealed artifact),
NodeDeploymentState, LdapGroupRoleMapping and inventory counts.
| Check | DEV vd03 | QA (shared ConfigDb on wonder-sql-vq03) |
|---|---|---|
__EFMigrationsHistory |
9 rows, head 20261022102459_EnableSnapshotIsolation = repo head at 47f0b98c. Snapshot ON, RCSI off, compat 160. |
Same: 9 rows, same head, snapshot ON, compat 160. |
ServerCluster |
DEV/DEV, NodeCount 1, RedundancyMode None |
DEV/QA, NodeCount 2, RedundancyMode Warm (see §2 "Latent") |
ClusterNode |
wonder-app-vd03.zmr.zimmer.com:4053, ApplicationUri urn:OtOpcUa:wonder-app-vd03 (the live endpoint reports urn:OtOpcUa: no env override, as before) |
10.220.158.2:4053 urn:wonder-app-vq03:OtOpcUa, 10.220.158.3:4053 urn:wonder-app-vq04:OtOpcUa, both Enabled, MaintenanceMode 0 |
| Latest Sealed deployment | 595cb909, rev fa9a2e28… (06:34Z), applied (Status 1) on the node; artifact RedundancyMode":0 |
a8e078f0, rev 82dd0598… (11:53Z, the post-drill restore), applied on both nodes; artifact RedundancyMode":1 |
| Inventory | 1 driver (FOCAS), 22 tags, 0 VT, 0 scripts, 0 scripted alarms | 1 driver (Calculation canary), 1 tag, 1 VT, 2 scripts, 0 scripted alarms |
LdapGroupRoleMapping |
0 rows | 0 rows (the #780 drill row is gone) |
| Preflight script checks 1–12 | all ok | all ok |
| Check 0 "Environment" | attention: stale expectation (written for b8b4c18f); not an issue |
same |
| Check 13 FOCAS renames | 9 info rows (live since 2026-11-12) | ok |
OpcUa:Redundancy:Mode is set nowhere (nor OpcUa:PeerApplicationUris, nor any Redundancy key):
- appsettings (SMB read):
appsettings.json,.admin.json,.driver.json,.admin-driver.json,.Development.jsoninE:\ApiInstall\OtOpcUa\(vd03) andE:\ApiInstall\OtOpcUa\app\(vq03, vq04): noRedundancy/PeerApplicationUriskey. Noappsettings.Production.jsonon any box. - Service env (SMB read of the latest
.regbackups, names only, values never decoded to output): vd03OtOpcUa-backups\pre-20261113-780-runner-flip\OtOpcUaHost.reg(28 names, pre-flip) and QAE:\psrem-stage\otopcua-780-scriptedalarm\backup\OtOpcUaHost-pre-780-seam2.reg(48 / 47 names, pre-seam): noOpcUa__Redundancy__*, noOpcUa__PeerApplicationUris__*. The #780 changes after those backups added onlyScripting__*entries (the record's env diffs). Re-confirm on-box in §4 step 1 (names only), because the live env is newer than the backups. - ConfigDb: the
ServerCluster.RedundancyModecolumn is not the host setting (see §2).
Live baseline (Mac, 11:58Z):
| vd03 | vq03 | vq04 | |
|---|---|---|---|
/health/ready / /health/active |
200 / 200 | 200 / 200 | 200 / 503 |
ServiceLevel (i=2267) |
250 | 250 (Primary) | 240 |
RedundancySupport (i=3709) |
0 (None) | 0 | 0 |
ServerUriArray (i=11314) / ServerArray (i=2254) |
Good String[] / Good String[] |
same | same |
Runner (vq03 /metrics) |
— | runner_state 0, worker_up shard 0/1 = 1, iso 0; script-runner Ready, quarantined 0 |
— |
redundancy CLI (src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI): Mode None, no server URIs, on all three. These are the
"unchanged under Unset" reference values for §5.
Verdict: no blocker on either tier; no issue filed. Re-run the SQL (it takes a minute) just before the window if the configuration has changed.
4. Build, stage, swap
Build (Mac, isolated worktree at 47f0b98c; not done here)
git worktree add ../OtOpcUa-wt-47f0b98c 47f0b98c && cd ../OtOpcUa-wt-47f0b98c
dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj -c Release -r win-x64 --self-contained true -o ../publish-47f0b98c
# expect ~1398 files (as c9640f5c), OtOpcUa.Host.dll ProductVersion 1.0.0+47f0b98c…, script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe present;
# deps.json: ZB.MOM.WW.LocalDb + .Replication 0.3.8 (the only package change), Telemetry 0.3.1, HistorianGateway.Client 1.0.0
cd ../publish-47f0b98c && zip -qr ../otopcua-47f0b98c.zip . && shasum -a 256 ../otopcua-47f0b98c.zip > ../otopcua-47f0b98c.zip.sha256
Copy the zip + .sha256 over SMB; check the hash on the box before expanding. Stage the three install scripts from
47f0b98c (the swap invokes the installer next to it, so the #905 -Scope logic must be the staged one):
| File | SHA-256 at 47f0b98c |
|---|---|
scripts/install/Install-ScriptWorkerIsolation.ps1 |
bb31d2759d0de29292c123d74860226ed37af554f2c8fe6871312c7e81f7c05a |
scripts/install/ScriptWorkerIsolation.psm1 |
c75a6e3e2c0bfbb58a360e92249efeb02ac025b50035ea73b95c32304c7dbff3 |
scripts/install/Invoke-OtOpcUaInstallSwap.ps1 |
09a0bb6874af23a726ec349bd7bd86615832d08c58d0e70f864642afda3bc866 |
On-box steps below are deploy steps (winrm skill, deploy scope), in an elevated Windows PowerShell from the staged
install\ folder. Never run Install-Services.ps1, Uninstall-Services.ps1 or Refresh-Services.ps1.
The #901 settings records already exist on all three nodes (written in the c9640f5c window; vd03's re-written by the
#780 flip with ConfigureHostService=1). No one-time full-argument install this time. Do not re-run the installer with
arguments; the swap reads the record.
Data backup inside the window, Outside scope outside it. The swap stops the service itself. To keep the #905 gain,
start the swap with the service running and -NoStart: it runs the Outside scope while the old build serves, stops,
swaps, runs the InstallRoot scope and leaves the service stopped. Then back up data (consistent SQLite triplets) and
start. Stopping the service by hand first would move the ~20+ s Outside scan back into the window.
DEV vd03 (E:\ApiInstall\OtOpcUa, data E:\ApiInstall\OtOpcUa-data, runner OutOfProcess, single node)
Staging E:\ApiInstall\_stage-otopcua-47f0b98c\ (zip, .sha256, install\). Backups
E:\ApiInstall\OtOpcUa-backups\pre-20261113-otopcua-47f0b98c\, created with a protected DACL (SYSTEM + Administrators, #891).
- Prepare (service up).
reg export HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaHostinto the backup folder.- Hash check, expand to
E:\ApiInstall\OtOpcUa.new-47f0b98c, check ProductVersion anddeps.jsonLocalDb 0.3.8. BACKUP DATABASE [OtOpcUaConfig] TO DISK = N'OtOpcUaConfig-pre-20261113-otopcua-47f0b98c.bak' WITH COPY_ONLY, CHECKSUM(configuration fallback only; no migration).- Env digest (names + SHA-256 prefixes only): expect 34,
69a493345099/caa047aaaf42; noOpcUa__Redundancy__*, noOpcUa__PeerApplicationUris__*;Scripting__Runner=OutOfProcess+ 3 seams + 2Scripting__Worker__*. - Pre-window isolation check with the new installer, service up:
.\Install-ScriptWorkerIsolation.ps1 -Verify→VERIFY OK(14/14 last time), exit 0. Proves the47f0b98cinstaller reads the existing record. .\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -Candidate E:\ApiInstall\OtOpcUa.new-47f0b98c -PreviousInstall E:\ApiInstall\OtOpcUa.bak.20261113-pre-47f0b98c -WhatIf→ preflight only (record found, candidate has the worker exe, same volume, PreviousInstall free).- Snapshot HG
/metrics(http://wonder-app-vd03.zmr.zimmer.com:5220/metrics,historian_gateway_grpc_requests_total).
- Swap (service running when started):
.\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -Candidate E:\ApiInstall\OtOpcUa.new-47f0b98c ` -PreviousInstall E:\ApiInstall\OtOpcUa.bak.20261113-pre-47f0b98c -NoStart- Default
-Carry(appsettings*.json,pki,logs) is right for vd03. - Expect pre-stop:
-Reapply -Scope Outsideall unchanged (incl.HostServiceEnv,SettingsRecord),-Verify -Scope OutsideOK. In the window:-Reapply -Scope InstallRootchangesWorkerFolderAcl,InstallRootDeny, thepkiSensitiveDirDeny;-Verify -Scope InstallRootOK. - Record the
stop -> swapped and verified (left stopped): N s [stop …, move …, carry …, isolation …]line. - Gate: no throw. A pre-stop failure leaves the old build serving (nothing moved). A post-swap failure leaves it stopped:
fix, run
-Reapply/-Verify(no-Scope), then start.
- Default
- Backup + start:
robocopy E:\ApiInstall\OtOpcUa-data <backup>\OtOpcUa-data /E /COPY:DAT(≈0.5 MB), thenStart-Service OtOpcUaHost. Note the start time; poll/health/ready. - After ready:
.\Install-ScriptWorkerIsolation.ps1 -Verify(full) → OK, exit 0. Env digest again: 34, unchanged.
QA vq03 + vq04 (stop both, swap both, start both)
Layout E:\ApiInstall\OtOpcUa\{app,data,pki}, service OtOpcUaHost (LocalSystem), runner OutOfProcess (3 seams). Staging
E:\psrem-stage\otopcua-47f0b98c\ (zip, .sha256, install\), backups in …\backup\ with a protected DACL.
- Prepare (both nodes, services up).
- Expand to
E:\ApiInstall\OtOpcUa\app.new-47f0b98c(hash, ProductVersion,deps.jsonLocalDb 0.3.8). reg export …\OtOpcUaHost backup\OtOpcUaHost-pre-47f0b98c.reg.- Env digest, names only: expect vq03 49
5750bbeb4734/4263c3ac5a46, vq04 48ef146b2681b4/d19e968212f3;Seams__0..2= Calculation, VirtualTag, ScriptedAlarm; noOpcUa__Redundancy__*orOpcUa__PeerApplicationUris__*; pepper andLocalDb__Replication__ApiKeyentry prefixes equal on both nodes. - Once:
BACKUP DATABASE OtOpcUaConfig … COPY_ONLY, CHECKSUM, COMPRESSIONon wonder-sql-vq03 →OtOpcUaConfig_pre-47f0b98c_20261113.bak, thenRESTORE VERIFYONLY. - Pre-window
.\Install-ScriptWorkerIsolation.ps1 -Verifywith the new installer on each node → OK (21/21 last time). - The swap with
-WhatIfon each node (preflight only). - ServiceLevel on both: expect vq03 250 / vq04 240 (as at 11:58Z). Snapshot vq03 HG
/metrics(:5220).
- Expand to
- Swap the Secondary (vq04) first, while vq03 serves (service running when started):
.\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -Candidate E:\ApiInstall\OtOpcUa\app.new-47f0b98c ` -PreviousInstall E:\ApiInstall\OtOpcUa\app.bak.20261113-pre-47f0b98c -Carry 'appsettings*.json','logs' -NoStart- Expect the stop part of this window to be ~30 s: vq04 is still
c9640f5cand it serves vq03's open LocalDb sync stream (the #906 bug, last seen on this build). It is outside the pair outage (vq03 serves) and inside the default-StopTimeoutSeconds 120. Record it as the "before" control for the #906 gate. - Expect
-Reapply -Scope Outsideall unchanged;-Scope InstallRootchangesWorkerFolderAcl,InstallRootDeny; both-VerifyOK. Thenrobocopy data backup\data-pre-47f0b98c /E /COPY:DATand the same forpki.
- Expect the stop part of this window to be ~30 s: vq04 is still
- Swap the Primary (vq03), same command. The pair outage starts at its stop. Its stop should take a few seconds (the
peer is down, no inbound stream). Then the
data+pkirobocopy.- Gate: both swaps returned without throwing, both left stopped. If
-Verifyfailed on either, start neither: fix,-Reapply/-Verify, then go on. If vq04's swap failed pre-stop, vq04 is still serving the old build: stop and re-plan rather than starting a mixed pair.
- Gate: both swaps returned without throwing, both left stopped. If
- Start both: vq03 first (the old Primary), then vq04 at once. Record each start →
/health/ready. - On each node after ready:
.\Install-ScriptWorkerIsolation.ps1 -Verify(full) → OK, exit 0. Env digests unchanged.
5. Post-deploy gates
From the Mac over HTTP / opc.tcp (the Client CLI: dotnet ZB.MOM.WW.OtOpcUa.Client.CLI.dll read|redundancy|subscribe -u …),
logs over SMB (logs\otopcua-YYYYMMDD.log; an open log's listed size reads 0, read the contents).
| Gate | DEV vd03 | QA vq03 + vq04 |
|---|---|---|
| Swap output | -Verify -Scope Outside + -Scope InstallRoot OK; the window line recorded |
same, both nodes |
| Isolation | full -Verify exit 0 after the start |
same, both nodes |
| Version | live OtOpcUa.Host.dll (SMB copy) ProductVersion 1.0.0+47f0b98c…, SHA-256 = the published build; live ZB.MOM.WW.LocalDb.dll file version 0.3.8 |
same, both nodes |
| Health | /healthz, /health/ready (5/5, script-runner: Out-of-process script runner ready., shards not started), /health/active 200 |
/health/ready 200, 5/5 incl. runner ready, on both; /health/active 200 Primary / 503 Secondary |
| ServiceLevel | 250 ×3 fresh sessions | 250 / 240 ×3, stable |
| Redundancy metadata unchanged under Unset | redundancy CLI: Mode None, no server URIs; i=3709 = 0 Good; i=11314 Good String[] (empty); i=2254 Good — all as the §3 baseline. Log: no Warm redundancy / mode_mismatch lines; /metrics: no otopcua_redundancy_warm_transition/snapshot_rejected/mode_mismatch counts, otopcua_opcua_standby_* absent or 0 |
same on both nodes; the Secondary keeps serving Good data at 240 (canary reads on vq04) |
| Failover + #906 timing (QA) | — | Run a background subscribe -n i=2267 -i 250 on each endpoint and a 0.5 s /health/active poll on both. Leg 1: stop the Primary (vq03, the sync client) with a stopwatch around Stop-Service: vq04 → 250 and /health/active 200 on its first samples. Restart vq03 → 240, never above, no failback. Leg 2 (the #906 proof): stop vq04 (now Primary, the sync server): target Stop-Service ≈ 1–3 s and vq03 at 250 within ~1–2 s (before: ~30 s stop, vq03 250 after ~31 s — #780 record; and step 2's control). vq04's log must show Host stopping: leaving the Akka cluster before the web server drains its open requests (lmxopcua#906). then Akka CoordinatedShutdown completed in N ms, and no …/LocalDbSync/Sync responded 200 in ~30000+ ms. The service must stay Stopped (no SCM recovery restart). Restart vq04 → 240, no failback. End state: vq03 250, vq04 240. Record both legs' stop durations and hand-over times. |
| LocalDb sync (QA) | — | localdb-replication connected, oplog backlog 0 on both, after the start and after each failover leg |
| Canary + runner (QA) | — | ns=2;s=script-runner-canary/Device1/canary-calc-seconds and ns=3;s=EQ-ec3979550d90/canary-vt-second Good and advancing ~1/s on both endpoints, after the start and after each leg. otopcua_script_worker_up{shard="0"} = 1 and {shard="1"} = 1 on both; runner_state 0; quarantined 0. |
| ScriptedAlarm seam (QA) | (vd03: env carries Seams__2=ScriptedAlarm; no scripts, so nothing runs) |
Seam ready = env digest unchanged with Scripting__Runner__Seams__2=ScriptedAlarm + runner Ready on both nodes. There is no scripted alarm in the canary-only config, so otopcua_script_ipc_roundtrip{kind="alarm"} will not count; the live alarm proof is the 11:44–11:53Z drill on c9640f5c. Re-running the drill's normal alarm (canary-alarm-cycle) is optional and needs the owner's AdminUI sign-in + the temporary role row, then the cleanup back to rev 82dd0598…. |
| Driver data (vd03) | FOCAS raw: 22 variables, 21 Good, Program/BlockCount BadNoData (as before); 2 UNS variables |
— |
| Boot log | roles=[admin,driver]; credential-reference preflight "every stored and deployed reference is namespaced."; Historian gateway probe: the gateway is reachable; raw subtree materialised (… failed=0); no FTL, no options-validation failure (OpcUa:Redundancy…); ERR only the known certificate-domain noise from reads by IP |
same, both nodes, plus the known standby /health/* 503 and bootstrap akka … Removed lines |
| Historian writes | HG :5220/metrics before/after: only HistorianStatus/Probe grows. No EnsureTags, AddTagExtendedProperties, WriteLiveValues, AddHistoricalValues, SendEvent(s). |
same on vq03 HG :5220 |
| #903 drift chip (vd03) | /deployments still shows "Configuration drift" (the c9640f5c edit-and-revert; see Rollback) and the nodes serve 595cb909 / fa9a2e28…. Do not "Deploy current configuration" in the window: it would seal a new revision; that is the owner's call. |
Optional: /deployments should read "In sync" (draft = a8e078f0); needs the AdminUI sign-in |
| Overview / ScadaBridge | :5320 OtOpcUa Up; ScadaBridge :8085/health/ready 200 |
:5320 on both: OtOpcUa Up, 2 members / 0 unreachable; ScadaBridge 200 on both |
6. Rollback
No schema undo (no migration). The ConfigDb .bak is a configuration fallback only. The env changes since c9640f5c was
installed (#780) live in the registry and already ran on c9640f5c, so they stay valid. The isolation record and denies
are build-independent; never run -Uninstall as part of a rollback.
DEV vd03 → c9640f5c:
.\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -Candidate E:\ApiInstall\OtOpcUa.bak.20261113-pre-47f0b98c `
-PreviousInstall E:\ApiInstall\OtOpcUa.failed-47f0b98c -NoCarry -NoStart
Then mirror <backup>\OtOpcUa-data back over E:\ApiInstall\OtOpcUa-data, Start-Service OtOpcUaHost, full -Verify,
health, ServiceLevel 250, FOCAS 21/22. The registry is not changed by this deploy, so no import. The older
E:\ApiInstall\OtOpcUa.bak.20261113-pre-c9640f5c (= 6c4d2990) is one generation further back; do not use it here.
vd03 #903 drift note. The /deployments chip shows "Configuration drift" before this deploy, after it and after a
rollback: the draft's re-saved dbo.Tag row has a new RowVersion, and RowVersion is serialised into the artifact
whose SHA-256 is the revision hash. The content is identical to the sealed fa9a2e28…. A rollback does not clear it, and
restoring OtOpcUaConfig-pre-20261113-otopcua-47f0b98c.bak restores the same drifted draft (the
…-c9640f5c.bak predates the edit and would clear it, but would also roll back anything else saved since). It is not a
rollback trigger. Only a new deployment clears it (owner's call).
QA → c9640f5c: stop both (Secondary first), then on each node:
.\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -Candidate E:\ApiInstall\OtOpcUa\app.bak.20261113-pre-47f0b98c `
-PreviousInstall E:\ApiInstall\OtOpcUa\app.failed-47f0b98c -NoCarry -NoStart
Mirror backup\data-pre-47f0b98c back over data\ (and pki if it changed), full -Verify on each (exit 0), start
both, vq03 first. No .reg import (the env is not touched). DB fallback OtOpcUaConfig_pre-47f0b98c_20261113.bak. Do not
run a mixed pair: RedundancyStateChanged changed shape in this range. After a rollback the #906 ~30 s stop of vq04 is
back; expect it.
7. Downtime estimates (#905 numbers)
| Tier | Previous (c9640f5c window) |
Estimate for this pack |
|---|---|---|
| vd03 | ~101 s stop → ready: the full coverage walk of E:\ApiInstall ran twice inside the window |
~20–25 s: stop of the old build (2–6 s) + move/carry (<1 s) + -Scope InstallRoot reapply/verify (~6 s on windev's vd03-like tree: 153 s → 6 s on PS 5.1) + OtOpcUa-data robocopy (<1 s) + start → ready (~9 s, the #780 flip restart). The Outside part (~23 s on windev) runs before the stop. |
| QA pair outage | ~91 s (Primary stop → both ready), with 50–60 s of -Reapply/-Verify per node |
~30–40 s: vq03's stop (a few s; no inbound stream once vq04 is down) + ~6–10 s InstallRoot scope + backup (<1 s) + operator gap + start → ready (9–16 s; the first node waits for its peer). vq04's ~30 s old-build stop is outside the outage. |
| QA failover legs | Primary stop → peer 250: ~3 s when vq03 stopped (c9640f5c record); ~31 s when vq04 stopped (#780 record) |
~1–2 s on both legs (#906; GracefulStopWithOpenSyncStreamTests stop 895 ms) |