From 97d513f7dc5e2026c7636138aa47e3b6bda523dd Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 13 Nov 2026 07:02:20 -0500 Subject: [PATCH] docs(deploy): next vd03 + QA deploy pack c9640f5c -> 47f0b98c (#905 #906 #717 LocalDb 0.3.8), read-only preflight Range, settings (no migration; OpcUa:Redundancy:Mode unset everywhere), read-only ConfigDb/SMB/OPC UA preflight on both tiers, #905 scoped swap with -NoStart on both tiers, #906 failover timing gate (stop the sync-server node as Primary), Unset metadata gate, rollback with the vd03 #903 drift note. Not executed. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi --- .../2026-11-13-next-vd03-qa-to-47f0b98c.md | 283 ++++++++++++++++++ 1 file changed, 283 insertions(+) create mode 100644 docs/deploy/2026-11-13-next-vd03-qa-to-47f0b98c.md diff --git a/docs/deploy/2026-11-13-next-vd03-qa-to-47f0b98c.md b/docs/deploy/2026-11-13-next-vd03-qa-to-47f0b98c.md new file mode 100644 index 000000000..b114a6400 --- /dev/null +++ b/docs/deploy/2026-11-13-next-vd03-qa-to-47f0b98c.md @@ -0,0 +1,283 @@ +# Next deploy: DEV vd03 and QA vq03/vq04 `c9640f5c` → master `47f0b98c` + +> **Status: PREPARED, NOT EXECUTED (2026-11-13).** The owner approved the deploy itself, to run after the LocalDb 0.3.8 +> pin landed (it did: `47f0b98c`, 11:55Z). This pack covers both tiers. The read-only preflight below (ConfigDb SELECTs, +> SMB reads of the box config, OPC UA / HTTP reads) ran from the Mac on 2026-11-13 11:55–12:00Z. Nothing was built, +> staged, stopped or run on any wonder box. **No blocker found; no issue filed.** +> +> **Target:** `origin/master` **`47f0b98c`** (`47f0b98c6e0eeb663ee14d4e6fa063623c74408d`, scadaproj#57). If master moves +> before the window, either pin to `47f0b98c` or re-run §1–§3 for the new range. +> +> Templates: the previous pack [`2026-11-13-next-vd03-qa-to-c9640f5c.md`](2026-11-13-next-vd03-qa-to-c9640f5c.md) and its +> record [`2026-11-13-vd03-qa-to-c9640f5c.md`](2026-11-13-vd03-qa-to-c9640f5c.md) (layout, staging, backups, gates), and +> the #780 record [`2026-11-13-780-scriptedalarm-qa-and-vd03-flip.md`](2026-11-13-780-scriptedalarm-qa-and-vd03-flip.md) +> (the env both tiers run now). + +## 0. State since the c9640f5c deploy (all three nodes run `c9640f5c`) + +- **vd03 runner flip (#780 Part 2, `2688bd5c`):** `Scripting__Runner=OutOfProcess` with seams Calculation, VirtualTag, + ScriptedAlarm; the #901 record now has **ConfigureHostService=1** and the host env has the two `Scripting__Worker__*` + entries. Env **34** entries, ordered `69a493345099` / sorted `caa047aaaf42`. vd03 has no scripts, so the shards are + `not started`. The installer on vd03 now also manages `HostServiceEnv` (it is in the Outside scope, below). +- **QA ScriptedAlarm seam (#780 Part 1):** `Scripting__Runner__Seams__2=ScriptedAlarm` on both nodes. Env vq03 **49** + (`5750bbeb4734` / `4263c3ac5a46`), vq04 **48** (`ef146b2681b4` / `d19e968212f3`). +- **QA drill (`003c7f89`): PASS, finished 11:53Z.** Config is back on the canary-only revision: `a8e078f0`, rev + `82dd0598c37f41f8…` (identical to `5905003e`). 2 scripts, 1 VT, 0 scripted alarms, 0 role-mapping rows (the temporary + row was deleted). **Primary is now vq03 (250), vq04 240** (read 11:58Z, below). +- **vd03 #903 drift:** the draft still differs from the sealed `595cb909` / `fa9a2e28…` because of the edit-and-revert + in the c9640f5c gate (`RowVersion` is in the revision hash). The running configuration is the sealed one. + +## 1. Range `c9640f5c..47f0b98c` (16 commits, same on both tiers) + +| Commit | Change | Effect on DEV / QA | +|---|---|---| +| `dcff5ae9` **#906** | Host: `GracefulClusterLeaveService` runs Akka CoordinatedShutdown in `StoppingAsync`, before any hosted service stops, so the node leaves the cluster **before** Kestrel drains. `ShutdownStreamCancellationInterceptor` (innermost on the gRPC pipeline) ends open streaming calls on `ApplicationStopping` and refuses new ones `Unavailable`. | **QA: a planned stop of the node that serves the LocalDb sync stream (vq04; vq03 is the client, it has `LocalDb__Replication__PeerAddress`) no longer waits ~30 s.** The #780 record measured vq04 stop → vq03 250 in ~31 s; the target is ~1 s. vd03 (single node): stop leaves a 1-member cluster first; faster or equal. New stop log lines (§5). | +| `47f0b98c` LocalDb **0.3.8** (scadaproj#57) | Library side of #906: `LocalDbSyncService` ends open sync streams on `ApplicationStopping` and refuses new ones `Unavailable`. Wire-compatible with 0.3.7; no schema change. The host interceptor stays (it also covers the mesh streams). | Both tiers: package bump only. QA: the pair runs the same version after stop-both/start-both. | +| `0c19f26c` **#905** | `Invoke-OtOpcUaInstallSwap.ps1`: `-Reapply`/`-Verify -Scope Outside` (full coverage scan, firewall, host env, records) **before the stop**, only `-Scope InstallRoot` inside the window; preflight checks the candidate carries the worker exe; prints `stopped window (…): N s [stop, move, carry, isolation]`. Coverage scan ~20× faster per item. | Changes the deploy procedure (§4). windev vd03-like tree: stopped window 153 s → 6 s. No runtime change. | +| `7829b937`, `f817c77e`, `687256d6`, `c569f806` **#717** | Warm non-transparent redundancy, **default-off** (`OpcUa:Redundancy:Mode` Unset). New: options validation of `Mode`/`PeerApplicationUris` (Hot/unknown fail start; Warm needs peers; None needs none), a startup guard (Warm without the driver role fails start), revisioned `RedundancyStateChanged` snapshots, standby projection hooks in the node manager / SDK server, new metrics `otopcua.redundancy.{warm_role,warm_transition,snapshot_rejected,mode_mismatch}`, `otopcua.opcua.standby.*`. `c569f806` = T13 fixes (Warm crash survivor stays Active; verdict reads its own member events). | **Inert under Unset** on every tier: no metadata written, no snapshot ordering, no member-event subscription, no deploy-time mode cross-check (`CheckRedundancyMode` returns on Unset). Evidence: T13 L9 + the Unset re-check after `c569f806` (`docs/plans/evidence/2026-11-13-717-t13/RESULTS-unset-recheck.md`: 250/240, Good on both, writes + alarms on the Secondary, graceful 259 ms hand-over, crash path as documented, no failback; A/B identical to the pre-fix build). `RedundancyStateChanged` gained an optional `Revision`: **do not run a mixed c9640f5c/47f0b98c pair** (QA stops both anyway). | +| `f826af8e`, `36eb6d55`, `a42c9d7f`, `2238da65`, `8667ce58` | #717 docs, spec/plan, T0/T13 evidence (incl. a docker-dev Warm overlay) | — | +| `6e67250e`, `c0ba450d`, `2688bd5c`, `003c7f89` | deploy docs (c9640f5c pack + record, #780 records) | — | + +### What a ScadaBridge consumer would notice + +Nothing on DEV/QA (ScadaBridge there has no OtOpcUa data connection). For an OtOpcUa-bound client: a planned stop of the +Primary hands over in about 1 s instead of up to ~30 s, and the stopping node's sessions end sooner. Under Unset +`RedundancySupport` stays `None` (0), `ServerUriArray` (`i=11314`) stays the SDK orphan (Good `[]` by NodeId), the +Secondary keeps serving Good at 240. No proto, NodeId or ServiceLevel-semantics change. + +## 2. Migrations and settings + +- **ConfigDb migrations: none.** Nothing under `src/Core/ZB.MOM.WW.OtOpcUa.Configuration` changed in the range. The repo + head stays `20261022102459_EnableSnapshotIsolation` (9 migrations); both DBs are there (§3). +- **Deployment artifact format: unchanged.** `DeploymentArtifact.ReadClusterRedundancyMode` is read-only; the composer did + not change, so the revision hash of an unchanged draft does not move. +- **New optional settings (leave all unset):** + - `OpcUa:Redundancy:Mode` = `Unset` (default) | `None` | `Warm`. `Hot` or any other value **fails host start**. + `Warm` without the `driver` role fails start. **No tier sets it until ScadaBridge#138 ships** (owner decision). + - `OpcUa:Redundancy:PromoteWhilePrimaryUnreachable` (default `true`; Warm only). + - `OpcUa:PeerApplicationUris`: unchanged meaning; validated only under `None` (must be empty) / `Warm` (≥1, distinct, + not self). Neither tier sets it (§3). +- **Latent, for the later Warm flip (not this deploy):** `ServerCluster.RedundancyMode` is **Warm (1) on QA** and **None (0) + on DEV**, and the sealed artifacts carry it. Under Unset the cross-check is skipped. Under a future local `Mode=Warm` QA + matches; a local `Mode=None` on QA would log the `mode_mismatch` Error. Nothing to do now. +- **Shutdown:** no new key. The host's `ShutdownTimeout` (30 s default) still bounds the CoordinatedShutdown wait; on + timeout it logs a Warning and continues. +- **Required env:** no new variable. Existing ones unchanged (pepper, JWT key, QA `LocalDb__Replication__ApiKey` + byte-identical on both nodes). +- **Packages:** `ZB.MOM.WW.LocalDb`(+`.Replication`) 0.3.7 → **0.3.8**. Everything else as `c9640f5c` (Telemetry 0.3.1, + MxGateway.Client 0.4.0, HistorianGateway.Client 1.0.0, Grpc.StatusProto 2.76.0, Auth 0.4.1, Secrets 0.6.2, Health 0.3.0, + Theme 0.5.0). +- **Peer services:** none to upgrade first. HG is now `6365f34` on vd03 and QA (≥ `a220ec0`, so #900 is live; no + historized tags on either tier). mxgw `76a71bc8` unaffected (no Galaxy driver). + +## 3. Read-only preflight (run 2026-11-13 11:55–12:00Z from the Mac) + +**SQL:** `wwadmin`, `sqlcmd -C -I`, password in `SQLCMDPASSWORD`, DB `OtOpcUaConfig` on +`wonder-app-vd03.zmr.zimmer.com,1433` (DEV) and `wonder-sql-vq03.zmr.zimmer.com,1433` (QA). SELECTs only: +`scripts/preflight/vd03-preupgrade-b8b4c18f-configdb-checks.sql` unchanged, plus ad-hoc SELECTs on `__EFMigrationsHistory`, +`sys.databases`, `ServerCluster`, `ClusterNode`, `Deployment` (incl. `DECOMPRESS` of the latest Sealed artifact), +`NodeDeploymentState`, `LdapGroupRoleMapping` and inventory counts. + +| Check | DEV vd03 | QA (shared ConfigDb on wonder-sql-vq03) | +|---|---|---| +| `__EFMigrationsHistory` | **9 rows, head `20261022102459_EnableSnapshotIsolation`** = repo head at `47f0b98c`. Snapshot ON, RCSI off, compat 160. | **Same: 9 rows, same head**, snapshot ON, compat 160. | +| `ServerCluster` | `DEV`/`DEV`, NodeCount 1, **RedundancyMode None** | `DEV`/`QA`, NodeCount 2, **RedundancyMode Warm** (see §2 "Latent") | +| `ClusterNode` | `wonder-app-vd03.zmr.zimmer.com:4053`, ApplicationUri `urn:OtOpcUa:wonder-app-vd03` (the live endpoint reports `urn:OtOpcUa`: no env override, as before) | `10.220.158.2:4053` `urn:wonder-app-vq03:OtOpcUa`, `10.220.158.3:4053` `urn:wonder-app-vq04:OtOpcUa`, both Enabled, MaintenanceMode 0 | +| Latest Sealed deployment | `595cb909`, rev `fa9a2e28…` (06:34Z), applied (Status 1) on the node; artifact `RedundancyMode":0` | **`a8e078f0`, rev `82dd0598…` (11:53Z, the post-drill restore)**, applied on both nodes; artifact `RedundancyMode":1` | +| Inventory | 1 driver (FOCAS), 22 tags, 0 VT, 0 scripts, 0 scripted alarms | 1 driver (Calculation canary), 1 tag, 1 VT, 2 scripts, **0 scripted alarms** | +| `LdapGroupRoleMapping` | 0 rows | **0 rows** (the #780 drill row is gone) | +| Preflight script checks 1–12 | all **ok** | all **ok** | +| Check 0 "Environment" | `attention`: stale expectation (written for `b8b4c18f`); not an issue | same | +| Check 13 FOCAS renames | 9 info rows (live since 2026-11-12) | ok | + +**`OpcUa:Redundancy:Mode` is set nowhere** (nor `OpcUa:PeerApplicationUris`, nor any `Redundancy` key): + +- **appsettings (SMB read):** `appsettings.json`, `.admin.json`, `.driver.json`, `.admin-driver.json`, `.Development.json` in + `E:\ApiInstall\OtOpcUa\` (vd03) and `E:\ApiInstall\OtOpcUa\app\` (vq03, vq04): no `Redundancy` / `PeerApplicationUris` + key. No `appsettings.Production.json` on any box. +- **Service env (SMB read of the latest `.reg` backups, names only, values never decoded to output):** vd03 + `OtOpcUa-backups\pre-20261113-780-runner-flip\OtOpcUaHost.reg` (28 names, pre-flip) and QA + `E:\psrem-stage\otopcua-780-scriptedalarm\backup\OtOpcUaHost-pre-780-seam2.reg` (48 / 47 names, pre-seam): no + `OpcUa__Redundancy__*`, no `OpcUa__PeerApplicationUris__*`. The #780 changes after those backups added only + `Scripting__*` entries (the record's env diffs). **Re-confirm on-box in §4 step 1** (names only), because the live env is + newer than the backups. +- **ConfigDb:** the `ServerCluster.RedundancyMode` column is not the host setting (see §2). + +**Live baseline (Mac, 11:58Z):** + +| | vd03 | vq03 | vq04 | +|---|---|---|---| +| `/health/ready` / `/health/active` | 200 / 200 | 200 / 200 | 200 / 503 | +| ServiceLevel (`i=2267`) | 250 | **250** (Primary) | 240 | +| `RedundancySupport` (`i=3709`) | 0 (None) | 0 | 0 | +| `ServerUriArray` (`i=11314`) / `ServerArray` (`i=2254`) | Good `String[]` / Good `String[]` | same | same | +| Runner (vq03 `/metrics`) | — | `runner_state` 0, `worker_up` shard 0/1 = 1, iso 0; `script-runner` Ready, quarantined 0 | — | + +`redundancy` CLI (`src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI`): Mode `None`, no server URIs, on all three. These are the +"unchanged under Unset" reference values for §5. + +**Verdict: no blocker on either tier; no issue filed.** Re-run the SQL (it takes a minute) just before the window if the +configuration has changed. + +## 4. Build, stage, swap + +### Build (Mac, isolated worktree at `47f0b98c`; not done here) + +```bash +git worktree add ../OtOpcUa-wt-47f0b98c 47f0b98c && cd ../OtOpcUa-wt-47f0b98c +dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj -c Release -r win-x64 --self-contained true -o ../publish-47f0b98c +# expect ~1398 files (as c9640f5c), OtOpcUa.Host.dll ProductVersion 1.0.0+47f0b98c…, script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe present; +# deps.json: ZB.MOM.WW.LocalDb + .Replication 0.3.8 (the only package change), Telemetry 0.3.1, HistorianGateway.Client 1.0.0 +cd ../publish-47f0b98c && zip -qr ../otopcua-47f0b98c.zip . && shasum -a 256 ../otopcua-47f0b98c.zip > ../otopcua-47f0b98c.zip.sha256 +``` + +Copy the zip + `.sha256` over SMB; check the hash on the box before expanding. **Stage the three install scripts from +`47f0b98c`** (the swap invokes the installer next to it, so the #905 `-Scope` logic must be the staged one): + +| File | SHA-256 at `47f0b98c` | +|---|---| +| `scripts/install/Install-ScriptWorkerIsolation.ps1` | `bb31d2759d0de29292c123d74860226ed37af554f2c8fe6871312c7e81f7c05a` | +| `scripts/install/ScriptWorkerIsolation.psm1` | `c75a6e3e2c0bfbb58a360e92249efeb02ac025b50035ea73b95c32304c7dbff3` | +| `scripts/install/Invoke-OtOpcUaInstallSwap.ps1` | `09a0bb6874af23a726ec349bd7bd86615832d08c58d0e70f864642afda3bc866` | + +On-box steps below are deploy steps (`winrm` skill, deploy scope), in an elevated Windows PowerShell from the staged +`install\` folder. **Never run `Install-Services.ps1`, `Uninstall-Services.ps1` or `Refresh-Services.ps1`.** + +**The #901 settings records already exist on all three nodes** (written in the c9640f5c window; vd03's re-written by the +#780 flip with ConfigureHostService=1). **No one-time full-argument install this time.** Do not re-run the installer with +arguments; the swap reads the record. + +**Data backup inside the window, Outside scope outside it.** The swap stops the service itself. To keep the #905 gain, +start the swap **with the service running** and `-NoStart`: it runs the Outside scope while the old build serves, stops, +swaps, runs the InstallRoot scope and leaves the service stopped. Then back up `data` (consistent SQLite triplets) and +start. Stopping the service by hand first would move the ~20+ s Outside scan back into the window. + +### DEV vd03 (`E:\ApiInstall\OtOpcUa`, data `E:\ApiInstall\OtOpcUa-data`, runner OutOfProcess, single node) + +Staging `E:\ApiInstall\_stage-otopcua-47f0b98c\` (zip, `.sha256`, `install\`). Backups +`E:\ApiInstall\OtOpcUa-backups\pre-20261113-otopcua-47f0b98c\`, created with a protected DACL (SYSTEM + Administrators, #891). + +1. **Prepare (service up).** + - `reg export HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaHost` into the backup folder. + - Hash check, expand to `E:\ApiInstall\OtOpcUa.new-47f0b98c`, check ProductVersion and `deps.json` LocalDb 0.3.8. + - `BACKUP DATABASE [OtOpcUaConfig] TO DISK = N'OtOpcUaConfig-pre-20261113-otopcua-47f0b98c.bak' WITH COPY_ONLY, CHECKSUM` + (configuration fallback only; no migration). + - Env digest (names + SHA-256 prefixes only): expect **34**, `69a493345099` / `caa047aaaf42`; **no `OpcUa__Redundancy__*`, + no `OpcUa__PeerApplicationUris__*`**; `Scripting__Runner=OutOfProcess` + 3 seams + 2 `Scripting__Worker__*`. + - Pre-window isolation check with the **new** installer, service up: `.\Install-ScriptWorkerIsolation.ps1 -Verify` → + `VERIFY OK` (14/14 last time), exit 0. Proves the 47f0b98c installer reads the existing record. + - `.\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -Candidate E:\ApiInstall\OtOpcUa.new-47f0b98c -PreviousInstall E:\ApiInstall\OtOpcUa.bak.20261113-pre-47f0b98c -WhatIf` + → preflight only (record found, candidate has the worker exe, same volume, PreviousInstall free). + - Snapshot HG `/metrics` (`http://wonder-app-vd03.zmr.zimmer.com:5220/metrics`, `historian_gateway_grpc_requests_total`). +2. **Swap (service running when started):** + ```powershell + .\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -Candidate E:\ApiInstall\OtOpcUa.new-47f0b98c ` + -PreviousInstall E:\ApiInstall\OtOpcUa.bak.20261113-pre-47f0b98c -NoStart + ``` + - Default `-Carry` (`appsettings*.json`, `pki`, `logs`) is right for vd03. + - Expect pre-stop: `-Reapply -Scope Outside` **all unchanged** (incl. `HostServiceEnv`, `SettingsRecord`), `-Verify -Scope + Outside` OK. In the window: `-Reapply -Scope InstallRoot` changes `WorkerFolderAcl`, `InstallRootDeny`, the `pki` + `SensitiveDirDeny`; `-Verify -Scope InstallRoot` OK. + - **Record the `stop -> swapped and verified (left stopped): N s [stop …, move …, carry …, isolation …]` line.** + - Gate: no throw. A pre-stop failure leaves the old build serving (nothing moved). A post-swap failure leaves it stopped: + fix, run `-Reapply`/`-Verify` (no `-Scope`), then start. +3. **Backup + start:** `robocopy E:\ApiInstall\OtOpcUa-data \OtOpcUa-data /E /COPY:DAT` (≈0.5 MB), then + `Start-Service OtOpcUaHost`. Note the start time; poll `/health/ready`. +4. After ready: `.\Install-ScriptWorkerIsolation.ps1 -Verify` (full) → OK, exit 0. Env digest again: 34, unchanged. + +### QA vq03 + vq04 (stop both, swap both, start both) + +Layout `E:\ApiInstall\OtOpcUa\{app,data,pki}`, service `OtOpcUaHost` (LocalSystem), runner OutOfProcess (3 seams). Staging +`E:\psrem-stage\otopcua-47f0b98c\` (zip, `.sha256`, `install\`), backups in `…\backup\` with a protected DACL. + +1. **Prepare (both nodes, services up).** + - Expand to `E:\ApiInstall\OtOpcUa\app.new-47f0b98c` (hash, ProductVersion, `deps.json` LocalDb 0.3.8). + - `reg export …\OtOpcUaHost backup\OtOpcUaHost-pre-47f0b98c.reg`. + - Env digest, names only: expect vq03 **49** `5750bbeb4734` / `4263c3ac5a46`, vq04 **48** `ef146b2681b4` / + `d19e968212f3`; `Seams__0..2` = Calculation, VirtualTag, ScriptedAlarm; **no `OpcUa__Redundancy__*` or + `OpcUa__PeerApplicationUris__*`**; pepper and `LocalDb__Replication__ApiKey` entry prefixes equal on both nodes. + - Once: `BACKUP DATABASE OtOpcUaConfig … COPY_ONLY, CHECKSUM, COMPRESSION` on wonder-sql-vq03 → + `OtOpcUaConfig_pre-47f0b98c_20261113.bak`, then `RESTORE VERIFYONLY`. + - Pre-window `.\Install-ScriptWorkerIsolation.ps1 -Verify` with the new installer on each node → OK (21/21 last time). + - The swap with `-WhatIf` on each node (preflight only). + - ServiceLevel on both: expect **vq03 250 / vq04 240** (as at 11:58Z). Snapshot vq03 HG `/metrics` (`:5220`). +2. **Swap the Secondary (vq04) first, while vq03 serves** (service running when started): + ```powershell + .\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -Candidate E:\ApiInstall\OtOpcUa\app.new-47f0b98c ` + -PreviousInstall E:\ApiInstall\OtOpcUa\app.bak.20261113-pre-47f0b98c -Carry 'appsettings*.json','logs' -NoStart + ``` + - **Expect the stop part of this window to be ~30 s**: vq04 is still `c9640f5c` and it serves vq03's open LocalDb sync + stream (the #906 bug, last seen on this build). It is outside the pair outage (vq03 serves) and inside the default + `-StopTimeoutSeconds 120`. **Record it as the "before" control** for the #906 gate. + - Expect `-Reapply -Scope Outside` all unchanged; `-Scope InstallRoot` changes `WorkerFolderAcl`, `InstallRootDeny`; + both `-Verify` OK. Then `robocopy data backup\data-pre-47f0b98c /E /COPY:DAT` and the same for `pki`. +3. **Swap the Primary (vq03)**, same command. The pair outage starts at its stop. Its stop should take a few seconds (the + peer is down, no inbound stream). Then the `data` + `pki` robocopy. + - **Gate: both swaps returned without throwing**, both left stopped. If `-Verify` failed on either, start neither: fix, + `-Reapply`/`-Verify`, then go on. If vq04's swap failed pre-stop, vq04 is still serving the old build: stop and + re-plan rather than starting a mixed pair. +4. **Start both:** vq03 first (the old Primary), then vq04 at once. Record each start → `/health/ready`. +5. On each node after ready: `.\Install-ScriptWorkerIsolation.ps1 -Verify` (full) → OK, exit 0. Env digests unchanged. + +## 5. Post-deploy gates + +From the Mac over HTTP / opc.tcp (the Client CLI: `dotnet ZB.MOM.WW.OtOpcUa.Client.CLI.dll read|redundancy|subscribe -u …`), +logs over SMB (`logs\otopcua-YYYYMMDD.log`; an open log's listed size reads 0, read the contents). + +| Gate | DEV vd03 | QA vq03 + vq04 | +|---|---|---| +| Swap output | `-Verify -Scope Outside` + `-Scope InstallRoot` OK; the window line recorded | same, both nodes | +| Isolation | full `-Verify` exit 0 after the start | same, both nodes | +| Version | live `OtOpcUa.Host.dll` (SMB copy) ProductVersion `1.0.0+47f0b98c…`, SHA-256 = the published build; live `ZB.MOM.WW.LocalDb.dll` file version 0.3.8 | same, both nodes | +| Health | `/healthz`, `/health/ready` (5/5, `script-runner`: **`Out-of-process script runner ready.`**, shards `not started`), `/health/active` 200 | `/health/ready` 200, 5/5 incl. runner ready, on both; `/health/active` 200 Primary / 503 Secondary | +| ServiceLevel | **250** ×3 fresh sessions | **250 / 240** ×3, stable | +| **Redundancy metadata unchanged under Unset** | `redundancy` CLI: Mode `None`, no server URIs; `i=3709` = 0 Good; `i=11314` Good `String[]` (empty); `i=2254` Good — all as the §3 baseline. Log: no `Warm redundancy` / `mode_mismatch` lines; `/metrics`: no `otopcua_redundancy_warm_transition`/`snapshot_rejected`/`mode_mismatch` counts, `otopcua_opcua_standby_*` absent or 0 | same on both nodes; the Secondary keeps serving **Good** data at 240 (canary reads on vq04) | +| **Failover + #906 timing (QA)** | — | Run a background `subscribe -n i=2267 -i 250` on each endpoint and a 0.5 s `/health/active` poll on both. **Leg 1:** stop the Primary (vq03, the sync client) with a stopwatch around `Stop-Service`: vq04 → **250** and `/health/active` 200 on its first samples. Restart vq03 → **240**, never above, no failback. **Leg 2 (the #906 proof):** stop vq04 (now Primary, the sync server): **target `Stop-Service` ≈ 1–3 s and vq03 at 250 within ~1–2 s** (before: ~30 s stop, vq03 250 after ~31 s — #780 record; and step 2's control). vq04's log must show `Host stopping: leaving the Akka cluster before the web server drains its open requests (lmxopcua#906).` then `Akka CoordinatedShutdown completed in N ms`, and **no** `…/LocalDbSync/Sync responded 200 in ~30000+ ms`. The service must stay Stopped (no SCM recovery restart). Restart vq04 → 240, no failback. End state: vq03 250, vq04 240. Record both legs' stop durations and hand-over times. | +| LocalDb sync (QA) | — | `localdb-replication` connected, oplog backlog 0 on both, after the start and after each failover leg | +| Canary + runner (QA) | — | `ns=2;s=script-runner-canary/Device1/canary-calc-seconds` and `ns=3;s=EQ-ec3979550d90/canary-vt-second` **Good and advancing ~1/s** on both endpoints, after the start and after each leg. `otopcua_script_worker_up{shard="0"}` = 1 and `{shard="1"}` = 1 on both; `runner_state` 0; quarantined 0. | +| ScriptedAlarm seam (QA) | (vd03: env carries `Seams__2=ScriptedAlarm`; no scripts, so nothing runs) | **Seam ready** = env digest unchanged with `Scripting__Runner__Seams__2=ScriptedAlarm` + runner Ready on both nodes. There is no scripted alarm in the canary-only config, so `otopcua_script_ipc_roundtrip{kind="alarm"}` will not count; the live alarm proof is the 11:44–11:53Z drill on `c9640f5c`. Re-running the drill's normal alarm (`canary-alarm-cycle`) is **optional** and needs the owner's AdminUI sign-in + the temporary role row, then the cleanup back to rev `82dd0598…`. | +| Driver data (vd03) | FOCAS raw: 22 variables, **21 Good**, `Program/BlockCount` `BadNoData` (as before); 2 UNS variables | — | +| Boot log | `roles=[admin,driver]`; credential-reference preflight "every stored and deployed reference is namespaced."; `Historian gateway probe: the gateway is reachable`; `raw subtree materialised (… failed=0)`; no FTL, no options-validation failure (`OpcUa:Redundancy…`); ERR only the known certificate-domain noise from reads by IP | same, both nodes, plus the known standby `/health/*` 503 and bootstrap `akka … Removed` lines | +| **Historian writes** | HG `:5220/metrics` before/after: only `HistorianStatus/Probe` grows. **No `EnsureTags`, `AddTagExtendedProperties`, `WriteLiveValues`, `AddHistoricalValues`, `SendEvent(s)`.** | same on vq03 HG `:5220` | +| **#903 drift chip (vd03)** | `/deployments` still shows **"Configuration drift"** (the c9640f5c edit-and-revert; see Rollback) and the nodes serve `595cb909` / `fa9a2e28…`. **Do not "Deploy current configuration"** in the window: it would seal a new revision; that is the owner's call. | Optional: `/deployments` should read "In sync" (draft = `a8e078f0`); needs the AdminUI sign-in | +| Overview / ScadaBridge | `:5320` OtOpcUa Up; ScadaBridge `:8085/health/ready` 200 | `:5320` on both: OtOpcUa Up, 2 members / 0 unreachable; ScadaBridge 200 on both | + +## 6. Rollback + +No schema undo (no migration). The ConfigDb `.bak` is a configuration fallback only. The env changes since `c9640f5c` was +installed (#780) live in the registry and already ran on `c9640f5c`, so they stay valid. The isolation record and denies +are build-independent; never run `-Uninstall` as part of a rollback. + +**DEV vd03 → `c9640f5c`:** +```powershell +.\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -Candidate E:\ApiInstall\OtOpcUa.bak.20261113-pre-47f0b98c ` + -PreviousInstall E:\ApiInstall\OtOpcUa.failed-47f0b98c -NoCarry -NoStart +``` +Then mirror `\OtOpcUa-data` back over `E:\ApiInstall\OtOpcUa-data`, `Start-Service OtOpcUaHost`, full `-Verify`, +health, ServiceLevel 250, FOCAS 21/22. The registry is not changed by this deploy, so no import. The older +`E:\ApiInstall\OtOpcUa.bak.20261113-pre-c9640f5c` (= `6c4d2990`) is one generation further back; do not use it here. + +**vd03 #903 drift note.** The `/deployments` chip shows "Configuration drift" before this deploy, after it and after a +rollback: the draft's re-saved `dbo.Tag` row has a new `RowVersion`, and `RowVersion` is serialised into the artifact +whose SHA-256 is the revision hash. The content is identical to the sealed `fa9a2e28…`. A rollback does not clear it, and +restoring `OtOpcUaConfig-pre-20261113-otopcua-47f0b98c.bak` restores the same drifted draft (the +`…-c9640f5c.bak` predates the edit and would clear it, but would also roll back anything else saved since). It is not a +rollback trigger. Only a new deployment clears it (owner's call). + +**QA → `c9640f5c`:** stop both (Secondary first), then on each node: +```powershell +.\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -Candidate E:\ApiInstall\OtOpcUa\app.bak.20261113-pre-47f0b98c ` + -PreviousInstall E:\ApiInstall\OtOpcUa\app.failed-47f0b98c -NoCarry -NoStart +``` +Mirror `backup\data-pre-47f0b98c` back over `data\` (and `pki` if it changed), full `-Verify` on each (exit 0), start +both, vq03 first. No `.reg` import (the env is not touched). DB fallback `OtOpcUaConfig_pre-47f0b98c_20261113.bak`. Do not +run a mixed pair: `RedundancyStateChanged` changed shape in this range. After a rollback the #906 ~30 s stop of vq04 is +back; expect it. + +## 7. Downtime estimates (#905 numbers) + +| Tier | Previous (c9640f5c window) | Estimate for this pack | +|---|---|---| +| vd03 | **~101 s** stop → ready: the full coverage walk of `E:\ApiInstall` ran twice inside the window | **~20–25 s**: stop of the old build (2–6 s) + move/carry (<1 s) + `-Scope InstallRoot` reapply/verify (~6 s on windev's vd03-like tree: 153 s → 6 s on PS 5.1) + `OtOpcUa-data` robocopy (<1 s) + start → ready (~9 s, the #780 flip restart). The Outside part (~23 s on windev) runs before the stop. | +| QA pair outage | **~91 s** (Primary stop → both ready), with 50–60 s of `-Reapply`/`-Verify` per node | **~30–40 s**: vq03's stop (a few s; no inbound stream once vq04 is down) + ~6–10 s InstallRoot scope + backup (<1 s) + operator gap + start → ready (9–16 s; the first node waits for its peer). vq04's ~30 s old-build stop is outside the outage. | +| QA failover legs | Primary stop → peer 250: ~3 s when vq03 stopped (c9640f5c record); **~31 s** when vq04 stopped (#780 record) | **~1–2 s** on both legs (#906; `GracefulStopWithOpenSyncStreamTests` stop 895 ms) |