dohertj2/lmxopcua
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(deploy): vd03 + QA deployed c9640f5c -> 76e73e7d (#905 #906 #900 #717, LocalDb 0.3.8)
v2-ci / unit-tests (push) Has been cancelled
Details

Browse Source 
Executed 2026-11-13 12:27-12:48Z; target moved from the pack's 47f0b98c to the tip 76e73e7d
(HG Client 1.1.0, behaviour unchanged; note added to the pack).

- vd03: ~14 s downtime (was ~101 s; #905 window 5.7 s), SL 250, FOCAS 21/22, Mode Unset,
  HG Probe 494 -> 536 with no write RPCs (added gate for #900).
- QA: pair outage 36.5 s (was ~91 s), 250/240, both failover legs, no failback, sync
  connected, canary Good. #906: vq04 (sync server) shutdown -> Leaving 30.0 s on c9640f5c,
  21 ms on 76e73e7d; peer at 250 in 0.47 s (leg 1: 0.36 s).
- Deviation: the first Move-Item of app\ failed "file in use" right after the stop on both
  QA nodes; a retry with the service stopped succeeded (lmxopcua#908).

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-13 07:51:00 -05:00
parent c9bc358dc9
commit 5baf05f634
2 changed files with 197 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/deploy/2026-11-13-next-vd03-qa-to-47f0b98c.md
+8
View File
@@ -1,5 +1,13 @@
# Next deploy: DEV vd03 and QA vq03/vq04 `c9640f5c` → master `47f0b98c`
> **Executed 2026-11-13 12:27–12:48Z, target moved to `76e73e7d`.** Master gained one commit after this pack,
> `76e73e7d` (#900 follow-up: HistorianGateway Client/Contracts 1.0.0 → 1.1.0; `GatewayErrorReasons` reads the typed
> `HistorianGatewayException` reason instead of parsing the gRPC trailer; behaviour unchanged). DEV and QA HG already run the
> 1.1.0 server (`6365f34`). `git log 47f0b98c..76e73e7d` showed nothing else. The install scripts are byte-identical to this
> pack's table, there are no migrations, and the only new package delta is HG Client 1.1.0. So the owner deployed the tip
> rather than pinning `47f0b98c`. One gate was added: on vd03 the historian path still answers (HG `/metrics` Probe grows,
> no write RPCs). Record: [`2026-11-13-vd03-qa-to-76e73e7d.md`](2026-11-13-vd03-qa-to-76e73e7d.md).
> **Status: PREPARED, NOT EXECUTED (2026-11-13).** The owner approved the deploy itself, to run after the LocalDb 0.3.8
> pin landed (it did: `47f0b98c`, 11:55Z). This pack covers both tiers. The read-only preflight below (ConfigDb SELECTs,
> SMB reads of the box config, OPC UA / HTTP reads) ran from the Mac on 2026-11-13 11:55–12:00Z. Nothing was built,
docs/deploy/2026-11-13-vd03-qa-to-76e73e7d.md
+189
View File
@@ -0,0 +1,189 @@
# DEV vd03 and QA vq03/vq04 `c9640f5c` → master `76e73e7d` (2026-11-13)
> **Status: EXECUTED 2026-11-13.**
> - DEV vd03: 12:27–12:32Z.
> - QA pair: 12:34–12:48Z.
>
> All three nodes run **`76e73e7d`** (`76e73e7dbf1160e74d2c89036c4235eefdc566dd`). The owner approved the deploy on
> 2026-11-13, with WinRM for the on-box deploy steps only.
>
> **Runbook:** [`2026-11-13-next-vd03-qa-to-47f0b98c.md`](2026-11-13-next-vd03-qa-to-47f0b98c.md), the pack. It was
> written for `47f0b98c`. The target moved to the tip `76e73e7d` (#900 follow-up: HG Client 1.1.0, typed error reason,
> behaviour unchanged); see the pack's header note.
>
> **Deployed:**
> - #906: graceful cluster leave before Kestrel drains, plus LocalDb **0.3.8**.
> - #905: Outside-scope isolation before the stop.
> - #717: Warm redundancy, inert under `Mode` Unset.
> - #900: HG Client 1.1.0.
Channel: WinRM + SMB from the Mac (the `winrm` skill), used only for the pack's on-box steps:
- prepare (backups, expand, env digest, pre-window `-Verify`, swap `-WhatIf`);
- swap, start, `-Verify`;
- the two failover-leg stops and restarts.
Everything else ran from the Mac:
- health, OPC UA, `/metrics`, HG `/metrics`, Overview and ScadaBridge checks, over HTTP and opc.tcp;
- logs and the live DLLs, read over SMB.
Not done: no Galaxy writes, no historian writes or tag creation, no configuration deploy. No other service was touched (ScadaBridge, mxgw,
HistorianGateway, Overview, OTel collector). `OpcUa:Redundancy:Mode` is unset on every node, in env and appsettings, checked by name.
## Build
- `dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj -c Release -r win-x64 --self-contained true`
on the Mac (SDK 10.0.403), in a clean worktree at `76e73e7d`:
- 0 warnings, **1398 files**;
- `OtOpcUa.Host.dll` ProductVersion `1.0.0+76e73e7dbf1160e7…`, SHA-256 prefix `6a8ec583a328`;
- `script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe` present.
- `deps.json`:
- **LocalDb(.Replication, .Contracts) 0.3.8** and **HistorianGateway.Client/Contracts 1.1.0**: the two deltas from `c9640f5c`;
- unchanged: Telemetry 0.3.1, MxGateway.Client 0.4.0, Grpc.StatusProto 2.76.0 (now transitive), Auth 0.4.1, Secrets 0.6.2, Health 0.3.0, Theme 0.5.0.
- `ZB.MOM.WW.LocalDb.dll` is 0.3.8.0, SHA-256 prefix `62e85cc0cedb`.
- Zip `otopcua-76e73e7d.zip` (139,455,907 B), SHA-256 `22A26D19FEF364AA6FDA5B72CD693B6301B0C05DAEFB52AB850724A2804F4A0B`.
Its hash was checked on every box before expanding.
- Install scripts staged from `76e73e7d`. They are byte-identical to the pack's table (`bb31d275…`, `c75a6e3e…`,
`09a0bb68…`), and the hashes were checked on every box.
## DEV: wonder-app-vd03 (`c9640f5c` → `76e73e7d`)
Stage `E:\ApiInstall\_stage-otopcua-76e73e7d\` holds:
- `prepare.ps1`, `swap.ps1`, `rollback.ps1`, `envdigest.ps1`;
- the zip and its `.sha256`;
- `install\`.
| Step | Result |
|---|---|
| Prepare (service up) | Backup folder `E:\ApiInstall\OtOpcUa-backups\pre-20261113-otopcua-76e73e7d\` created with a protected DACL (SYSTEM + Administrators), `OtOpcUaHost.reg` exported into it. 1398 files expanded to `E:\ApiInstall\OtOpcUa.new-76e73e7d`; ProductVersion and `deps.json` (LocalDb 0.3.8, HG Client 1.1.0) checked. `BACKUP DATABASE … COPY_ONLY, CHECKSUM` → `OtOpcUaConfig-pre-20261113-otopcua-76e73e7d.bak` (1905 pages). ConfigDb: 9 migrations, last `20261022102459_EnableSnapshotIsolation`, snapshot ON, latest sealed `595cb909` / `fa9a2e28…`. **Pre-window `-Verify` with the new installer: VERIFY OK 14/14.** Swap `-WhatIf`: preflight ok. E: 176 GB free. |
| Env (names + SHA-256 prefixes only) | **34 entries**, ordered `69a493345099` / sorted `caa047aaaf42` (= the pack). Runner OutOfProcess + 3 seams + 2 `Scripting__Worker__*`; OTLP endpoint has a scheme. **`OpcUa__Redundancy*` / `OpcUa__PeerApplicationUris*`: 0 entries; `appsettings*.json` `Redundancy`/`PeerApplicationUris`: 0 hits.** |
| Swap (`-NoStart`, started with the service up) | Invoked 12:28:02.2Z. `-Reapply -Scope Outside`: **11/11 unchanged** (incl. `HostServiceEnv`, `SettingsRecord`). `-Verify -Scope Outside`: **OK 11/11**. That pre-stop isolation took 28.0 s with the old build serving. **Stop issued 12:28:30.58Z.** Moved; carried 5 `appsettings*.json`, `pki`, `logs`. `-Reapply -Scope InstallRoot` **changed `WorkerFolderAcl`, `InstallRootDeny`, `SensitiveDirDeny` (`pki`)**; `-Verify -Scope InstallRoot` **OK 3/3**. **`stop -> swapped and verified (left stopped): 5.7 s [stop 1.5 s, move 0.0 s, carry 0.8 s, isolation 3.4 s]`.** |
| Backup + start | `OtOpcUa-data` robocopied into the backup folder (stopped, consistent). Start 12:28:36.46Z → `/health/ready` 200 at 12:28:44.56Z (8.1 s). |
| **Downtime** | **14.0 s** (stop issued → ready, on-box). From the Mac, `/health/ready` was unreachable from 12:28:30.5Z to 12:28:44.2Z (13.7 s). **c9640f5c window: ~101 s.** |
| Post-start full `-Verify` | **VERIFY OK 14/14, exit 0.** Env after: 34, `69a493345099` / `caa047aaaf42`: **unchanged**. |
### vd03 gates
| Gate | Result |
|---|---|
| Version | Live `E:\ApiInstall\OtOpcUa\OtOpcUa.Host.dll` (SMB copy): SHA-256 prefix `6a8ec583a328` = the published build; ProductVersion `1.0.0+76e73e7d…`. Live `ZB.MOM.WW.LocalDb.dll` 0.3.8.0 (`62e85cc0cedb`). |
| Health | `/healthz`, `/health/ready` and `/health/active` all 200. `/health/ready` is 5/5: opcua-endpoint; **`Out-of-process script runner ready.`** (state Ready, peCache true, shards `not started`, vd03 has no scripts); configdb; config-publish; akka Up with 1 member. |
| ServiceLevel | **250** on three fresh-session reads. |
| Redundancy under Unset | `redundancy` CLI reports Mode `None`, no server URIs. `i=3709` = 0 Good; `i=11314` Good `String[]`; `i=2254` Good. All match the pack's §3 baseline. The boot log has **`OPC UA redundancy: Mode="Unset", effective ApplicationUri=urn:OtOpcUa, peers=[]`**. `/metrics`: `otopcua_redundancy_warm_role` = **-1** ("not Warm" by the gauge's own definition); no `warm_transition` / `snapshot_rejected` / `mode_mismatch` / `opcua_standby_*` series. |
| FOCAS | 22 raw variables: **21 Good**, `Program/BlockCount` `BadNoData` (as before). 2 UNS variables. All 115 `ns≥1` variables were read: 114 Good. |
| Boot log | `roles=[admin,driver]`; `Historian gateway probe: the gateway is reachable.`; `applied plan (PureAdd, added=36 …)`; `raw subtree materialised (containers=10, tags=22, … failed=0)`; **`Credential-reference preflight (lmxopcua#815/#834): every stored and deployed reference is namespaced.`**; 0 FTL, no options-validation failure. The 60 ERR lines are all `The domain 'opc.tcp://10.220.157.247:4840/OtOpcUa' is not listed in the server certificate` (the gate's own reads by IP). |
| **Historian path (added gate, #900 / HG Client 1.1.0)** | HG `:5220/metrics` (`6365f34`), `historian_gateway_grpc_requests_total{service="HistorianStatus",method="Probe",outcome="ok"}`: **494 (12:27:17Z) → 504 (12:31:52Z) → 536 (12:47:45Z)**. OtOpcUa on the new client probes on its cycle, and the boot log has `Historian gateway probe: the gateway is reachable.` at 07:28:40 local. Unchanged: `BrowseTagNames` 1, `ReadRaw` 1, `ReadAggregate` ok 1 / error 2, `Probe` error 1. **No `EnsureTags`, `AddTagExtendedProperties`, `WriteLiveValues`, `AddHistoricalValues`, `SendEvent(s)` series at all.** **PASS.** |
| #903 drift chip | Not touched: no configuration deploy was made. The nodes still serve sealed `595cb909` / `fa9a2e28…` (preflight SQL). The pack says the chip shows "Configuration drift" and that this is not a rollback trigger. |
| Overview / ScadaBridge | Overview `:5320` 200. ScadaBridge `:8085/health/ready` 200 (untouched; `8e32f353`). |
## QA: wonder-app-vq03 + vq04 (`c9640f5c` → `76e73e7d`)
Stage `E:\psrem-stage\otopcua-76e73e7d\` on each node holds:
- `prepare.ps1`, `swap.ps1`, `swapretry.ps1`, `swap2.ps1` (vq03), `start.ps1`, `legstop.ps1`, `rollback.ps1`, `envdigest.ps1`;
- the zip and its `.sha256`;
- `install\`.
Backups are in `…\backup\`, which has a protected DACL.
**Before the window (12:25–12:35Z):**
- ServiceLevel vq03 **250** (Primary), vq04 **240**, as the pack read at 11:58Z.
- Both canaries Good on both endpoints.
- `redundancy` CLI on vq03: Mode `None`.
| Step | vq03 | vq04 |
|---|---|---|
| Prepare (services up) | 1398 files, hash + ProductVersion + `deps.json` ok; `OtOpcUaHost-pre-76e73e7d.reg` exported. **DB:** `BACKUP DATABASE OtOpcUaConfig … COPY_ONLY, CHECKSUM, COMPRESSION` → `OtOpcUaConfig_pre-76e73e7d_20261113.bak` (wonder-sql-vq03, 1969 pages), **`RESTORE VERIFYONLY` valid**. 9 migrations, snapshot ON, latest sealed `a8e078f0` / `82dd0598…`. Pre-window `-Verify` **OK 21/21**; swap `-WhatIf` ok. E: 213 GB free. | Same (no DB step). Pre-window `-Verify` **OK 21/21**. E: 228 GB free. |
| Env | **49 entries**, `5750bbeb4734` / `4263c3ac5a46` (= the pack). | **48 entries**, `ef146b2681b4` / `d19e968212f3` (= the pack). |
| Env cross-node | Same `Security__ApiKeys__Pepper` (entry prefix `ea9ddc525fc0`) and `LocalDb__Replication__ApiKey` (`1ce08c4554b0`) on both. Seams Calculation / VirtualTag / ScriptedAlarm on both. **No `OpcUa__Redundancy*` / `OpcUa__PeerApplicationUris*` entries and no appsettings hits on either node.** | |
| Swap (`-NoStart`, service up) | **Second.** Started 12:38:39Z; `-Scope Outside` 19/19 unchanged, OK. **Stop issued 12:38:58.22Z, 1.3 s.** **The first `Move-Item app → app.bak…` failed `file in use` (lmxopcua#908)**. The in-call retry (`swap2.ps1`) found no `OtOpcUa.Host` / ScriptWorker process 0.5 s later, waited 2 s and re-ran the swap with the service stopped: Outside 19/19 unchanged + OK again, moved, carried 5 `appsettings*.json` + `logs`, InstallRoot **changed `WorkerFolderAcl`, `InstallRootDeny`**, **OK 2/2**. Window line of the retry: `4.6 s [stop 0.0 s, move 0.0 s, carry 0.8 s, isolation 3.8 s]`. Left stopped 12:39:25.1Z; `data\` + `pki\` → `backup\{data,pki}-pre-76e73e7d`. | **First** (the Secondary, while vq03 served). `-Scope Outside` 19/19 unchanged, `-Verify` OK 19/19 (17.9 s pre-stop). **Stop issued 12:35:26.96Z, `Stopped` at 12:35:57.08Z: 30.1 s. This is the #906 "before" figure, on the old build.** **The first `Move-Item` then failed `file in use`**: nothing moved, the service was Stopped, `app\` was intact. `swapretry.ps1` at 12:36:46Z found 0 processes with an image or module under `app\` and re-ran the swap (service stopped): Outside 19/19 unchanged + OK, moved, carried, InstallRoot changed 2 + OK 2/2, window `4.8 s`. Left stopped 12:37:53Z; `data\` + `pki\` backed up. |
| Start (vq03 first, vq04 ~1 s later) | 12:39:26.71Z → ready 12:39:35.29Z (8.6 s). Full `-Verify` **OK 21/21, exit 0**. Env 49, `5750bbeb4734` / `4263c3ac5a46`: **unchanged**. | 12:39:28.00Z → ready 12:39:38.06Z (10.1 s). **OK 21/21, exit 0.** Env 48, `ef146b2681b4` / `d19e968212f3`: **unchanged**. |
- **Pair outage: 36.5 s.** No node was ready from 12:38:58.36Z to 12:39:34.85Z (0.5 s Mac poll of both `/health/ready`); both were ready at
12:39:37.8Z. The c9640f5c window was ~91 s; the pack estimated 30–40 s. About 20 s of it is the vq03 retry re-running the
Outside scope (lmxopcua#908); without the move failure the outage would have been about 15 s.
- vq04's 30 s old-build stop and its retry were outside the outage, because vq03 served throughout. The poll shows
vq03 200/200 from 12:35 to 12:38:58Z.
- **Primary after the start: vq03 (250), vq04 240**, ×3 fresh-session reads each.
### QA gates
| Gate | Result |
|---|---|
| Swap output | Outside and InstallRoot `-Verify` OK on both nodes (after the retries). Window lines recorded above. |
| Isolation | Full `-Verify` **OK 21/21, exit 0** after every start: both nodes after the window, vq03 after leg 1, vq04 after leg 2. |
| Version | Live `E:\ApiInstall\OtOpcUa\app\OtOpcUa.Host.dll` on both (SMB copy): **`6a8ec583a328`** = the published build, ProductVersion `1.0.0+76e73e7d…`. `ZB.MOM.WW.LocalDb.dll` 0.3.8.0 (`62e85cc0cedb`) on both. |
| Health | `/health/ready` 200 on both, 5/5 components: opcua-endpoint, **`Out-of-process script runner ready.`** (shards 0 and 1 up, quarantined 0), configdb, config-publish (Ready on vq03 / NotHosted on vq04), akka Up with **2 members / 0 unreachable**. `/health/active` 200 on vq03, 503 on vq04. |
| Redundancy under Unset | On both nodes, `redundancy` CLI Mode `None`, no server URIs; `i=3709` = 0 Good; `i=11314` / `i=2254` Good `String[]`. Boot logs (every start): **`OPC UA redundancy: Mode="Unset", effective ApplicationUri=urn:wonder-app-vq0N:OtOpcUa, peers=[]`**. `warm_role` -1 on both; no `warm_transition` / `snapshot_rejected` / `mode_mismatch` / `opcua_standby_*`. 0 `Warm redundancy` / `mode_mismatch` log lines. The Secondary (vq04, 240) serves **Good** canary data. |
| LocalDb sync | `localdb-replication` (on `/health/active`): **"LocalDb replication is connected; oplog backlog 0."** on both nodes, after the start, after leg 1 and after leg 2. `localdb_oplog_depth` 0, meter scope version 0.3.8. |
| Canary + runner | `canary-calc-seconds` and `canary-vt-second` are **Good and advancing about 1/s on both endpoints**: after the start, after leg 1 and at the end. `otopcua_script_worker_up{shard="0"}` = 1 and `{shard="1"}` = 1, `runner_state` 0, quarantined 0 on both. |
| ScriptedAlarm seam | Env carries `Scripting__Runner__Seams__2=ScriptedAlarm` (digest unchanged) and the runner is Ready on both nodes. There is no scripted alarm in the canary-only config, so `kind="alarm"` does not count. The live proof remains the 11:44–11:53Z drill on `c9640f5c`. The optional drill was not re-run. |
| Boot log (both nodes, every start) | `roles=[admin,driver]`; `Historian gateway probe: the gateway is reachable.`; `raw subtree materialised (containers=2, tags=1, … failed=0)`; credential-reference preflight clean; **0 FTL**. ERR lines: the standby `/health/active` 503 request logs and `cluster-primary` Unhealthy on whichever node was the Secondary, plus one `TCPSERVERCHANNEL … BadConnectionClosed` per node from the gate's own CLI subscription being killed. Nothing else. |
| HG `/metrics` (vq03 `:5220`) | 12:34:54Z → 12:47:35Z: `HistorianStatus/Probe` ok **988 → 1028**, nothing else changed. **No historian write or tag-creation RPC.** |
| Overview / ScadaBridge | `:5320` 200 and `:8085/health/ready` 200 on both nodes (untouched). |
### Failover + #906 timing
Method:
- a background `subscribe -n i=2267 -i 250` on each endpoint, each update stamped with its Mac arrival time;
- a 0.5 s poll of `/health/ready` + `/health/active` on both nodes;
- `legstop.ps1`, which wraps a stopwatch around `Stop-Service` and then checks the service stays Stopped for 15 s.
| | Leg 1: stop vq03 (Primary, sync **client**) | Leg 2: stop vq04 (Primary, sync **server**): **the #906 proof** |
|---|---|---|
| `Stop-Service` issued | 12:42:04.372Z | 12:44:07.472Z |
| Peer → 250 (subscription arrival) | **vq04 250 at 12:42:04.728Z (0.36 s)** | **vq03 250 at 12:44:07.944Z (0.47 s)** |
| Peer `/health/active` | 200 on the first sample (12:42:04.938Z) | 200 on the first sample (12:44:07.600Z) |
| `Stop-Service` duration | 6.10 s | **5.70 s** (old build, same node, 1 h earlier: **30.1 s**) |
| Stopping node's log | `Application is shutting down` 07:42:04.391 local → **`Host stopping: leaving the Akka cluster before the web server drains its open requests (lmxopcua#906).`** .394 → `Leaving` .409 → **`Akka CoordinatedShutdown completed in 647 ms`** | `Application is shutting down` 07:44:07.479 → `LocalDbSync/Sync responded 200 in 86702 ms` **.480** (the stream ended 1 ms after the stop) → **`… (lmxopcua#906).`** .483 → `Leaving` .500 → **`Akka CoordinatedShutdown completed in 236 ms`** (.718). **No `Sync responded 200 in ~30000+ ms` wait.** |
| Stays stopped | Stopped after 15 s, no host process (no SCM recovery restart) | same |
| Restart | 12:42:38.6Z → ready 12:42:46.9Z (8.3 s), `-Verify` OK 21/21. **vq03 240** (first value 12:42:49Z, unchanged after), `/health/active` 503 on all samples: **no failback** | 12:46:03.3Z → ready 12:46:12.6Z (9.3 s), `-Verify` OK 21/21. **vq04 240**, 503: **no failback** |
**#906 before / after (same node vq04, the sync server, same day):**
| | Before (`c9640f5c`, the window's vq04 stop, 12:35Z) | After (`76e73e7d`, leg 2, 12:44Z) |
|---|---|---|
| `Application is shutting down` → `Leaving` | **30.0 s** (07:35:26.975 → `Sync responded 200` 07:35:56.989 → `Leaving` 07:35:57.020) | **21 ms** (07:44:07.479 → 07:44:07.500) |
| `Stop-Service` | **30.1 s** | **5.7 s** |
| Peer at 250 | (vq03 was already Primary) — #780 record: **~31 s** | **0.47 s** |
**Observation (not a #906 regression):** on both legs the host logged nothing for ~5 s after `CoordinatedShutdown completed`. The gap ended with `SERVER - Service Fault Occurred. Reason=BadTimeout`: vq03 07:42:10.134, vq04 07:44:12.778.
The likely cause is the OPC UA stack closing the gate's own open `subscribe` session at server stop. That would put `Stop-Service` at ~6 s rather than the pack's 1–3 s target. The c9640f5c record's vq03 stop (no client session open) took 1 s.
The hand-over is not affected, because the cluster leave comes first. The peer took over within 0.5 s on both legs.
**End state: vq03 250 (Primary), vq04 240**, the same as before the window. Canary Good on both, sync connected, backlog 0.
## Rollback material (left on the boxes)
**DEV vd03 → `c9640f5c`:**
- Script: `E:\ApiInstall\_stage-otopcua-76e73e7d\rollback.ps1` (pack §6). It runs `Invoke-OtOpcUaInstallSwap.ps1 -Candidate E:\ApiInstall\OtOpcUa.bak.20261113-pre-76e73e7d -PreviousInstall E:\ApiInstall\OtOpcUa.failed-76e73e7d -NoCarry -NoStart`, mirrors `OtOpcUa-data` back, starts and runs a full `-Verify`.
- Previous install: `E:\ApiInstall\OtOpcUa.bak.20261113-pre-76e73e7d` (= `c9640f5c`).
- Backups: `E:\ApiInstall\OtOpcUa-backups\pre-20261113-otopcua-76e73e7d\{OtOpcUaHost.reg, OtOpcUa-data\}` (protected).
- ConfigDb: `OtOpcUaConfig-pre-20261113-otopcua-76e73e7d.bak` (vd03 instance default backup dir). No migration, so this is a configuration fallback only.
- No registry import is needed: the env was unchanged.
**QA → `c9640f5c`:**
1. Stop both nodes, Secondary first.
2. Run `E:\psrem-stage\otopcua-76e73e7d\rollback.ps1` on each node. It swaps back with `-NoCarry -NoStart` (Reapply/Verify inside) and mirrors `data\` from the backup.
3. Start both, vq03 first.
4. Run `start.ps1` on each node (it includes `-Verify`).
Do not run a mixed pair: `RedundancyStateChanged` changed shape in this range. After a rollback, the #906 ~30 s stop of vq04 is back.
- Previous install: `E:\ApiInstall\OtOpcUa\app.bak.20261113-pre-76e73e7d` on each node.
- Backups: `E:\psrem-stage\otopcua-76e73e7d\backup\{OtOpcUaHost-pre-76e73e7d.reg, data-pre-76e73e7d\, pki-pre-76e73e7d\}` (protected). No `.reg` import is needed: the env was unchanged.
- ConfigDb: `OtOpcUaConfig_pre-76e73e7d_20261113.bak` on wonder-sql-vq03 (VERIFYONLY ok).
## Deviations from the pack
1. **Target `76e73e7d` instead of `47f0b98c`.** The tip adds only the #900 follow-up (HG Client 1.1.0); see the pack's header.
One gate was added for it: vd03 HG Probe growing, with no write RPCs. **PASS.**
2. **QA: the first `Move-Item` of `app\` failed "file in use" on both nodes, right after the stop.** Nothing moved, the
service was Stopped, and the retry with the service stopped went through on both (vq04: a separate retry; vq03: an in-call
retry in `swap2.ps1`).
- The likely cause is the host process still exiting after SCM reported Stopped. The pre-#905 wrappers waited for the
process to exit; `Invoke-OtOpcUaInstallSwap.ps1` does not.
- It cost about 20 s of the vq03 pair outage (the retry re-runs `-Scope Outside`).
- Filed as **lmxopcua#908**. vd03 did not hit it.
3. `Stop-Service` on the new build took 5.7–6.1 s instead of the 1–3 s target, because of the ~5 s OPC UA session close
after the cluster leave (see "Observation"). The hand-over, the actual #906 criterion, was 0.36 s and 0.47 s.
4. Logs: the QA host log for the window rolled to `otopcua-20261113_001.log` on vq04 (size limit). Read it from there.