Files
lmxopcua/docs/deploy/2026-11-13-vd03-qa-to-76e73e7d.md
T
Joseph Doherty a157b27628
v2-ci / unit-tests (push) Has been cancelled
fix(install): swap waits for the host process to exit before moving the folder (#908)
On QA (76e73e7d deploy) Invoke-OtOpcUaInstallSwap.ps1's first Move-Item of
app\ failed "file in use" right after the service reported Stopped: SCM
reports Stopped while the host is still exiting (e.g. a log file under
InstallRoot still open).

- After the stop, wait (bounded, -ProcessExitTimeoutSeconds, default 60)
  until the service's host process (PID + start time captured before the
  stop) and every process imaged under InstallRoot have exited; otherwise
  throw naming them, with nothing moved.
- The renames use [IO.Directory]::Move instead of Move-Item. On windev,
  PowerShell 7's Move-Item falls back to an item-by-item move when the
  rename fails because a file under the folder is open, and leaves a partial
  PreviousInstall tree behind (a retry then nests into it).
- The first rename is retried for at most -MoveRetrySeconds (default 10)
  while it fails because something under InstallRoot is open (a backstop
  for a holder whose image is outside InstallRoot).
- -Verify/-Reapply/-Scope semantics unchanged.

Tests: dry-run harness gains fake-process-list tests for the wait and the
retry plus AST order checks (145 passed); the Windows harness gains three
real cases (holder past the budget, holder released 2 s into the wait,
holder outside InstallRoot) - 77/0 on pwsh 7 and Windows PowerShell 5.1 on
windev; with the wait removed the first two fail.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 08:46:27 -05:00

193 lines
20 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# DEV vd03 and QA vq03/vq04 `c9640f5c` → master `76e73e7d` (2026-11-13)
> **Status: EXECUTED 2026-11-13.**
> - DEV vd03: 12:27–12:32Z.
> - QA pair: 12:34–12:48Z.
>
> All three nodes run **`76e73e7d`** (`76e73e7dbf1160e74d2c89036c4235eefdc566dd`). The owner approved the deploy on
> 2026-11-13, with WinRM for the on-box deploy steps only.
>
> **Runbook:** [`2026-11-13-next-vd03-qa-to-47f0b98c.md`](2026-11-13-next-vd03-qa-to-47f0b98c.md), the pack. It was
> written for `47f0b98c`. The target moved to the tip `76e73e7d` (#900 follow-up: HG Client 1.1.0, typed error reason,
> behaviour unchanged); see the pack's header note.
>
> **Deployed:**
> - #906: graceful cluster leave before Kestrel drains, plus LocalDb **0.3.8**.
> - #905: Outside-scope isolation before the stop.
> - #717: Warm redundancy, inert under `Mode` Unset.
> - #900: HG Client 1.1.0.
Channel: WinRM + SMB from the Mac (the `winrm` skill), used only for the pack's on-box steps:
- prepare (backups, expand, env digest, pre-window `-Verify`, swap `-WhatIf`);
- swap, start, `-Verify`;
- the two failover-leg stops and restarts.
Everything else ran from the Mac:
- health, OPC UA, `/metrics`, HG `/metrics`, Overview and ScadaBridge checks, over HTTP and opc.tcp;
- logs and the live DLLs, read over SMB.
Not done: no Galaxy writes, no historian writes or tag creation, no configuration deploy. No other service was touched (ScadaBridge, mxgw,
HistorianGateway, Overview, OTel collector). `OpcUa:Redundancy:Mode` is unset on every node, in env and appsettings, checked by name.
## Build
- `dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj -c Release -r win-x64 --self-contained true`
on the Mac (SDK 10.0.403), in a clean worktree at `76e73e7d`:
- 0 warnings, **1398 files**;
- `OtOpcUa.Host.dll` ProductVersion `1.0.0+76e73e7dbf1160e7…`, SHA-256 prefix `6a8ec583a328`;
- `script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe` present.
- `deps.json`:
- **LocalDb(.Replication, .Contracts) 0.3.8** and **HistorianGateway.Client/Contracts 1.1.0**: the two deltas from `c9640f5c`;
- unchanged: Telemetry 0.3.1, MxGateway.Client 0.4.0, Grpc.StatusProto 2.76.0 (now transitive), Auth 0.4.1, Secrets 0.6.2, Health 0.3.0, Theme 0.5.0.
- `ZB.MOM.WW.LocalDb.dll` is 0.3.8.0, SHA-256 prefix `62e85cc0cedb`.
- Zip `otopcua-76e73e7d.zip` (139,455,907 B), SHA-256 `22A26D19FEF364AA6FDA5B72CD693B6301B0C05DAEFB52AB850724A2804F4A0B`.
Its hash was checked on every box before expanding.
- Install scripts staged from `76e73e7d`. They are byte-identical to the pack's table (`bb31d275…`, `c75a6e3e…`,
`09a0bb68…`), and the hashes were checked on every box.
## DEV: wonder-app-vd03 (`c9640f5c` → `76e73e7d`)
Stage `E:\ApiInstall\_stage-otopcua-76e73e7d\` holds:
- `prepare.ps1`, `swap.ps1`, `rollback.ps1`, `envdigest.ps1`;
- the zip and its `.sha256`;
- `install\`.
| Step | Result |
|---|---|
| Prepare (service up) | Backup folder `E:\ApiInstall\OtOpcUa-backups\pre-20261113-otopcua-76e73e7d\` created with a protected DACL (SYSTEM + Administrators), `OtOpcUaHost.reg` exported into it. 1398 files expanded to `E:\ApiInstall\OtOpcUa.new-76e73e7d`; ProductVersion and `deps.json` (LocalDb 0.3.8, HG Client 1.1.0) checked. `BACKUP DATABASE … COPY_ONLY, CHECKSUM` → `OtOpcUaConfig-pre-20261113-otopcua-76e73e7d.bak` (1905 pages). ConfigDb: 9 migrations, last `20261022102459_EnableSnapshotIsolation`, snapshot ON, latest sealed `595cb909` / `fa9a2e28…`. **Pre-window `-Verify` with the new installer: VERIFY OK 14/14.** Swap `-WhatIf`: preflight ok. E: 176 GB free. |
| Env (names + SHA-256 prefixes only) | **34 entries**, ordered `69a493345099` / sorted `caa047aaaf42` (= the pack). Runner OutOfProcess + 3 seams + 2 `Scripting__Worker__*`; OTLP endpoint has a scheme. **`OpcUa__Redundancy*` / `OpcUa__PeerApplicationUris*`: 0 entries; `appsettings*.json` `Redundancy`/`PeerApplicationUris`: 0 hits.** |
| Swap (`-NoStart`, started with the service up) | Invoked 12:28:02.2Z. `-Reapply -Scope Outside`: **11/11 unchanged** (incl. `HostServiceEnv`, `SettingsRecord`). `-Verify -Scope Outside`: **OK 11/11**. That pre-stop isolation took 28.0 s with the old build serving. **Stop issued 12:28:30.58Z.** Moved; carried 5 `appsettings*.json`, `pki`, `logs`. `-Reapply -Scope InstallRoot` **changed `WorkerFolderAcl`, `InstallRootDeny`, `SensitiveDirDeny` (`pki`)**; `-Verify -Scope InstallRoot` **OK 3/3**. **`stop -> swapped and verified (left stopped): 5.7 s [stop 1.5 s, move 0.0 s, carry 0.8 s, isolation 3.4 s]`.** |
| Backup + start | `OtOpcUa-data` robocopied into the backup folder (stopped, consistent). Start 12:28:36.46Z → `/health/ready` 200 at 12:28:44.56Z (8.1 s). |
| **Downtime** | **14.0 s** (stop issued → ready, on-box). From the Mac, `/health/ready` was unreachable from 12:28:30.5Z to 12:28:44.2Z (13.7 s). **c9640f5c window: ~101 s.** |
| Post-start full `-Verify` | **VERIFY OK 14/14, exit 0.** Env after: 34, `69a493345099` / `caa047aaaf42`: **unchanged**. |
### vd03 gates
| Gate | Result |
|---|---|
| Version | Live `E:\ApiInstall\OtOpcUa\OtOpcUa.Host.dll` (SMB copy): SHA-256 prefix `6a8ec583a328` = the published build; ProductVersion `1.0.0+76e73e7d…`. Live `ZB.MOM.WW.LocalDb.dll` 0.3.8.0 (`62e85cc0cedb`). |
| Health | `/healthz`, `/health/ready` and `/health/active` all 200. `/health/ready` is 5/5: opcua-endpoint; **`Out-of-process script runner ready.`** (state Ready, peCache true, shards `not started`, vd03 has no scripts); configdb; config-publish; akka Up with 1 member. |
| ServiceLevel | **250** on three fresh-session reads. |
| Redundancy under Unset | `redundancy` CLI reports Mode `None`, no server URIs. `i=3709` = 0 Good; `i=11314` Good `String[]`; `i=2254` Good. All match the pack's §3 baseline. The boot log has **`OPC UA redundancy: Mode="Unset", effective ApplicationUri=urn:OtOpcUa, peers=[]`**. `/metrics`: `otopcua_redundancy_warm_role` = **-1** ("not Warm" by the gauge's own definition); no `warm_transition` / `snapshot_rejected` / `mode_mismatch` / `opcua_standby_*` series. |
| FOCAS | 22 raw variables: **21 Good**, `Program/BlockCount` `BadNoData` (as before). 2 UNS variables. All 115 `ns≥1` variables were read: 114 Good. |
| Boot log | `roles=[admin,driver]`; `Historian gateway probe: the gateway is reachable.`; `applied plan (PureAdd, added=36 …)`; `raw subtree materialised (containers=10, tags=22, … failed=0)`; **`Credential-reference preflight (lmxopcua#815/#834): every stored and deployed reference is namespaced.`**; 0 FTL, no options-validation failure. The 60 ERR lines are all `The domain 'opc.tcp://10.220.157.247:4840/OtOpcUa' is not listed in the server certificate` (the gate's own reads by IP). |
| **Historian path (added gate, #900 / HG Client 1.1.0)** | HG `:5220/metrics` (`6365f34`), `historian_gateway_grpc_requests_total{service="HistorianStatus",method="Probe",outcome="ok"}`: **494 (12:27:17Z) → 504 (12:31:52Z) → 536 (12:47:45Z)**. OtOpcUa on the new client probes on its cycle, and the boot log has `Historian gateway probe: the gateway is reachable.` at 07:28:40 local. Unchanged: `BrowseTagNames` 1, `ReadRaw` 1, `ReadAggregate` ok 1 / error 2, `Probe` error 1. **No `EnsureTags`, `AddTagExtendedProperties`, `WriteLiveValues`, `AddHistoricalValues`, `SendEvent(s)` series at all.** **PASS.** |
| #903 drift chip | Not touched: no configuration deploy was made. The nodes still serve sealed `595cb909` / `fa9a2e28…` (preflight SQL). The pack says the chip shows "Configuration drift" and that this is not a rollback trigger. |
| Overview / ScadaBridge | Overview `:5320` 200. ScadaBridge `:8085/health/ready` 200 (untouched; `8e32f353`). |
## QA: wonder-app-vq03 + vq04 (`c9640f5c` → `76e73e7d`)
Stage `E:\psrem-stage\otopcua-76e73e7d\` on each node holds:
- `prepare.ps1`, `swap.ps1`, `swapretry.ps1`, `swap2.ps1` (vq03), `start.ps1`, `legstop.ps1`, `rollback.ps1`, `envdigest.ps1`;
- the zip and its `.sha256`;
- `install\`.
Backups are in `…\backup\`, which has a protected DACL.
**Before the window (12:25–12:35Z):**
- ServiceLevel vq03 **250** (Primary), vq04 **240**, as the pack read at 11:58Z.
- Both canaries Good on both endpoints.
- `redundancy` CLI on vq03: Mode `None`.
| Step | vq03 | vq04 |
|---|---|---|
| Prepare (services up) | 1398 files, hash + ProductVersion + `deps.json` ok; `OtOpcUaHost-pre-76e73e7d.reg` exported. **DB:** `BACKUP DATABASE OtOpcUaConfig … COPY_ONLY, CHECKSUM, COMPRESSION` → `OtOpcUaConfig_pre-76e73e7d_20261113.bak` (wonder-sql-vq03, 1969 pages), **`RESTORE VERIFYONLY` valid**. 9 migrations, snapshot ON, latest sealed `a8e078f0` / `82dd0598…`. Pre-window `-Verify` **OK 21/21**; swap `-WhatIf` ok. E: 213 GB free. | Same (no DB step). Pre-window `-Verify` **OK 21/21**. E: 228 GB free. |
| Env | **49 entries**, `5750bbeb4734` / `4263c3ac5a46` (= the pack). | **48 entries**, `ef146b2681b4` / `d19e968212f3` (= the pack). |
| Env cross-node | Same `Security__ApiKeys__Pepper` (entry prefix `ea9ddc525fc0`) and `LocalDb__Replication__ApiKey` (`1ce08c4554b0`) on both. Seams Calculation / VirtualTag / ScriptedAlarm on both. **No `OpcUa__Redundancy*` / `OpcUa__PeerApplicationUris*` entries and no appsettings hits on either node.** | |
| Swap (`-NoStart`, service up) | **Second.** Started 12:38:39Z; `-Scope Outside` 19/19 unchanged, OK. **Stop issued 12:38:58.22Z, 1.3 s.** **The first `Move-Item app → app.bak…` failed `file in use` (lmxopcua#908)**. The in-call retry (`swap2.ps1`) found no `OtOpcUa.Host` / ScriptWorker process 0.5 s later, waited 2 s and re-ran the swap with the service stopped: Outside 19/19 unchanged + OK again, moved, carried 5 `appsettings*.json` + `logs`, InstallRoot **changed `WorkerFolderAcl`, `InstallRootDeny`**, **OK 2/2**. Window line of the retry: `4.6 s [stop 0.0 s, move 0.0 s, carry 0.8 s, isolation 3.8 s]`. Left stopped 12:39:25.1Z; `data\` + `pki\` → `backup\{data,pki}-pre-76e73e7d`. | **First** (the Secondary, while vq03 served). `-Scope Outside` 19/19 unchanged, `-Verify` OK 19/19 (17.9 s pre-stop). **Stop issued 12:35:26.96Z, `Stopped` at 12:35:57.08Z: 30.1 s. This is the #906 "before" figure, on the old build.** **The first `Move-Item` then failed `file in use`**: nothing moved, the service was Stopped, `app\` was intact. `swapretry.ps1` at 12:36:46Z found 0 processes with an image or module under `app\` and re-ran the swap (service stopped): Outside 19/19 unchanged + OK, moved, carried, InstallRoot changed 2 + OK 2/2, window `4.8 s`. Left stopped 12:37:53Z; `data\` + `pki\` backed up. |
| Start (vq03 first, vq04 ~1 s later) | 12:39:26.71Z → ready 12:39:35.29Z (8.6 s). Full `-Verify` **OK 21/21, exit 0**. Env 49, `5750bbeb4734` / `4263c3ac5a46`: **unchanged**. | 12:39:28.00Z → ready 12:39:38.06Z (10.1 s). **OK 21/21, exit 0.** Env 48, `ef146b2681b4` / `d19e968212f3`: **unchanged**. |
- **Pair outage: 36.5 s.** No node was ready from 12:38:58.36Z to 12:39:34.85Z (0.5 s Mac poll of both `/health/ready`); both were ready at
12:39:37.8Z. The c9640f5c window was ~91 s; the pack estimated 30–40 s. About 20 s of it is the vq03 retry re-running the
Outside scope (lmxopcua#908); without the move failure the outage would have been about 15 s.
- vq04's 30 s old-build stop and its retry were outside the outage, because vq03 served throughout. The poll shows
vq03 200/200 from 12:35 to 12:38:58Z.
- **Primary after the start: vq03 (250), vq04 240**, ×3 fresh-session reads each.
### QA gates
| Gate | Result |
|---|---|
| Swap output | Outside and InstallRoot `-Verify` OK on both nodes (after the retries). Window lines recorded above. |
| Isolation | Full `-Verify` **OK 21/21, exit 0** after every start: both nodes after the window, vq03 after leg 1, vq04 after leg 2. |
| Version | Live `E:\ApiInstall\OtOpcUa\app\OtOpcUa.Host.dll` on both (SMB copy): **`6a8ec583a328`** = the published build, ProductVersion `1.0.0+76e73e7d…`. `ZB.MOM.WW.LocalDb.dll` 0.3.8.0 (`62e85cc0cedb`) on both. |
| Health | `/health/ready` 200 on both, 5/5 components: opcua-endpoint, **`Out-of-process script runner ready.`** (shards 0 and 1 up, quarantined 0), configdb, config-publish (Ready on vq03 / NotHosted on vq04), akka Up with **2 members / 0 unreachable**. `/health/active` 200 on vq03, 503 on vq04. |
| Redundancy under Unset | On both nodes, `redundancy` CLI Mode `None`, no server URIs; `i=3709` = 0 Good; `i=11314` / `i=2254` Good `String[]`. Boot logs (every start): **`OPC UA redundancy: Mode="Unset", effective ApplicationUri=urn:wonder-app-vq0N:OtOpcUa, peers=[]`**. `warm_role` -1 on both; no `warm_transition` / `snapshot_rejected` / `mode_mismatch` / `opcua_standby_*`. 0 `Warm redundancy` / `mode_mismatch` log lines. The Secondary (vq04, 240) serves **Good** canary data. |
| LocalDb sync | `localdb-replication` (on `/health/active`): **"LocalDb replication is connected; oplog backlog 0."** on both nodes, after the start, after leg 1 and after leg 2. `localdb_oplog_depth` 0, meter scope version 0.3.8. |
| Canary + runner | `canary-calc-seconds` and `canary-vt-second` are **Good and advancing about 1/s on both endpoints**: after the start, after leg 1 and at the end. `otopcua_script_worker_up{shard="0"}` = 1 and `{shard="1"}` = 1, `runner_state` 0, quarantined 0 on both. |
| ScriptedAlarm seam | Env carries `Scripting__Runner__Seams__2=ScriptedAlarm` (digest unchanged) and the runner is Ready on both nodes. There is no scripted alarm in the canary-only config, so `kind="alarm"` does not count. The live proof remains the 11:44–11:53Z drill on `c9640f5c`. The optional drill was not re-run. |
| Boot log (both nodes, every start) | `roles=[admin,driver]`; `Historian gateway probe: the gateway is reachable.`; `raw subtree materialised (containers=2, tags=1, … failed=0)`; credential-reference preflight clean; **0 FTL**. ERR lines: the standby `/health/active` 503 request logs and `cluster-primary` Unhealthy on whichever node was the Secondary, plus one `TCPSERVERCHANNEL … BadConnectionClosed` per node from the gate's own CLI subscription being killed. Nothing else. |
| HG `/metrics` (vq03 `:5220`) | 12:34:54Z → 12:47:35Z: `HistorianStatus/Probe` ok **988 → 1028**, nothing else changed. **No historian write or tag-creation RPC.** |
| Overview / ScadaBridge | `:5320` 200 and `:8085/health/ready` 200 on both nodes (untouched). |
### Failover + #906 timing
Method:
- a background `subscribe -n i=2267 -i 250` on each endpoint, each update stamped with its Mac arrival time;
- a 0.5 s poll of `/health/ready` + `/health/active` on both nodes;
- `legstop.ps1`, which wraps a stopwatch around `Stop-Service` and then checks the service stays Stopped for 15 s.
| | Leg 1: stop vq03 (Primary, sync **client**) | Leg 2: stop vq04 (Primary, sync **server**): **the #906 proof** |
|---|---|---|
| `Stop-Service` issued | 12:42:04.372Z | 12:44:07.472Z |
| Peer → 250 (subscription arrival) | **vq04 250 at 12:42:04.728Z (0.36 s)** | **vq03 250 at 12:44:07.944Z (0.47 s)** |
| Peer `/health/active` | 200 on the first sample (12:42:04.938Z) | 200 on the first sample (12:44:07.600Z) |
| `Stop-Service` duration | 6.10 s | **5.70 s** (old build, same node, 1 h earlier: **30.1 s**) |
| Stopping node's log | `Application is shutting down` 07:42:04.391 local → **`Host stopping: leaving the Akka cluster before the web server drains its open requests (lmxopcua#906).`** .394 → `Leaving` .409 → **`Akka CoordinatedShutdown completed in 647 ms`** | `Application is shutting down` 07:44:07.479 → `LocalDbSync/Sync responded 200 in 86702 ms` **.480** (the stream ended 1 ms after the stop) → **`… (lmxopcua#906).`** .483 → `Leaving` .500 → **`Akka CoordinatedShutdown completed in 236 ms`** (.718). **No `Sync responded 200 in ~30000+ ms` wait.** |
| Stays stopped | Stopped after 15 s, no host process (no SCM recovery restart) | same |
| Restart | 12:42:38.6Z → ready 12:42:46.9Z (8.3 s), `-Verify` OK 21/21. **vq03 240** (first value 12:42:49Z, unchanged after), `/health/active` 503 on all samples: **no failback** | 12:46:03.3Z → ready 12:46:12.6Z (9.3 s), `-Verify` OK 21/21. **vq04 240**, 503: **no failback** |
**#906 before / after (same node vq04, the sync server, same day):**
| | Before (`c9640f5c`, the window's vq04 stop, 12:35Z) | After (`76e73e7d`, leg 2, 12:44Z) |
|---|---|---|
| `Application is shutting down` → `Leaving` | **30.0 s** (07:35:26.975 → `Sync responded 200` 07:35:56.989 → `Leaving` 07:35:57.020) | **21 ms** (07:44:07.479 → 07:44:07.500) |
| `Stop-Service` | **30.1 s** | **5.7 s** |
| Peer at 250 | (vq03 was already Primary) — #780 record: **~31 s** | **0.47 s** |
**Observation (not a #906 regression):** on both legs the host logged nothing for ~5 s after `CoordinatedShutdown completed`. The gap ended with `SERVER - Service Fault Occurred. Reason=BadTimeout`: vq03 07:42:10.134, vq04 07:44:12.778.
The likely cause is the OPC UA stack closing the gate's own open `subscribe` session at server stop. That would put `Stop-Service` at ~6 s rather than the pack's 1–3 s target. The c9640f5c record's vq03 stop (no client session open) took 1 s.
The hand-over is not affected, because the cluster leave comes first. The peer took over within 0.5 s on both legs.
**End state: vq03 250 (Primary), vq04 240**, the same as before the window. Canary Good on both, sync connected, backlog 0.
## Rollback material (left on the boxes)
**DEV vd03 → `c9640f5c`:**
- Script: `E:\ApiInstall\_stage-otopcua-76e73e7d\rollback.ps1` (pack §6). It runs `Invoke-OtOpcUaInstallSwap.ps1 -Candidate E:\ApiInstall\OtOpcUa.bak.20261113-pre-76e73e7d -PreviousInstall E:\ApiInstall\OtOpcUa.failed-76e73e7d -NoCarry -NoStart`, mirrors `OtOpcUa-data` back, starts and runs a full `-Verify`.
- Previous install: `E:\ApiInstall\OtOpcUa.bak.20261113-pre-76e73e7d` (= `c9640f5c`).
- Backups: `E:\ApiInstall\OtOpcUa-backups\pre-20261113-otopcua-76e73e7d\{OtOpcUaHost.reg, OtOpcUa-data\}` (protected).
- ConfigDb: `OtOpcUaConfig-pre-20261113-otopcua-76e73e7d.bak` (vd03 instance default backup dir). No migration, so this is a configuration fallback only.
- No registry import is needed: the env was unchanged.
**QA → `c9640f5c`:**
1. Stop both nodes, Secondary first.
2. Run `E:\psrem-stage\otopcua-76e73e7d\rollback.ps1` on each node. It swaps back with `-NoCarry -NoStart` (Reapply/Verify inside) and mirrors `data\` from the backup.
3. Start both, vq03 first.
4. Run `start.ps1` on each node (it includes `-Verify`).
Do not run a mixed pair: `RedundancyStateChanged` changed shape in this range. After a rollback, the #906 ~30 s stop of vq04 is back.
- Previous install: `E:\ApiInstall\OtOpcUa\app.bak.20261113-pre-76e73e7d` on each node.
- Backups: `E:\psrem-stage\otopcua-76e73e7d\backup\{OtOpcUaHost-pre-76e73e7d.reg, data-pre-76e73e7d\, pki-pre-76e73e7d\}` (protected). No `.reg` import is needed: the env was unchanged.
- ConfigDb: `OtOpcUaConfig_pre-76e73e7d_20261113.bak` on wonder-sql-vq03 (VERIFYONLY ok).
## Deviations from the pack
1. **Target `76e73e7d` instead of `47f0b98c`.** The tip adds only the #900 follow-up (HG Client 1.1.0); see the pack's header.
One gate was added for it: vd03 HG Probe growing, with no write RPCs. **PASS.**
2. **QA: the first `Move-Item` of `app\` failed "file in use" on both nodes, right after the stop.** Nothing moved, the
service was Stopped, and the retry with the service stopped went through on both (vq04: a separate retry; vq03: an in-call
retry in `swap2.ps1`).
- The likely cause is the host process still exiting after SCM reported Stopped. The pre-#905 wrappers waited for the
process to exit; `Invoke-OtOpcUaInstallSwap.ps1` does not.
- It cost about 20 s of the vq03 pair outage (the retry re-runs `-Scope Outside`).
- Filed as **lmxopcua#908**. vd03 did not hit it.
- **Fixed (lmxopcua#908, closed):** `Invoke-OtOpcUaInstallSwap.ps1` now waits, after `Stopped`, for the host process
and every process imaged under `InstallRoot` to exit (`-ProcessExitTimeoutSeconds`, default 60) and retries the first
move on a sharing violation (`-MoveRetrySeconds`, default 10). The next deploy needs no hand retry.
3. `Stop-Service` on the new build took 5.7–6.1 s instead of the 1–3 s target, because of the ~5 s OPC UA session close
after the cluster leave (see "Observation"). The hand-over, the actual #906 criterion, was 0.36 s and 0.47 s.
4. Logs: the QA host log for the window rolled to `otopcua-20261113_001.log` on vq04 (size limit). Read it from there.