docs(deploy): QA vq03/vq04 switched to OpcUa:Redundancy:Mode=Warm; #138 live gate PASS (#907, #717)

Two service-env entries per node (Mode=Warm, PeerApplicationUris__0), pair stopped and
started together. Gates: 250/90, RedundancySupport Warm, standby BadOutOfService,
LocalDb sync, canary Good on the Primary; standby Write could only show BadNotWritable
(QA tags are read-only). ScadaBridge WarmCanary moved off the Standby by ServiceLevel in
0.1 s (connect-time preference); graceful Primary stop still took the 66 s connectivity
path (ScadaBridge#141). End state: Warm, vq04 Primary.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

docs/deploy/2026-11-13-qa-warm-717.md

@@ -0,0 +1,209 @@
+# QA vq03/vq04: `OpcUa:Redundancy:Mode=Warm` switch-over and #138 live gate (2026-11-13)
+
+> **Status: EXECUTED 2026-11-13, 13:41–13:58Z. QA is on Warm.**
+> - #907 (tier flip) and ScadaBridge#138 (ServiceLevel-aware failover) were gated together on the QA pair.
+> - The owner approved the change, with WinRM for the on-box config, stop/start and verify steps only.
+> - **End state:** both nodes run `Mode=Warm` on build `76e73e7d`. **vq04 is Primary at 250 and vq03 is Standby at 90.**
+>   The Primary moved during the gate, and Warm does not fail back.
+> - ScadaBridge's `WarmCanary` connection stays in place as a monitor.
+
+Nothing was rebuilt: both nodes keep master `76e73e7d` from `2026-11-13-vd03-qa-to-76e73e7d.md`. The only change is two
+service-env entries per node.
+
+Not done or touched:
+- no configuration deploy, no Galaxy or historian writes, no tag creation;
+- no other service on the boxes (mxgw, HistorianGateway, Overview, collector; ScadaBridge only through its management API);
+- DEV vd03 stays on `Mode` Unset.
+
+## Change
+
+| Node | Entries added to the `OtOpcUaHost` service env | Env after (count, ordered / sorted digest) |
+|---|---|---|
+| vq03 | `OpcUa__Redundancy__Mode=Warm` (`68b419d808d1`), `OpcUa__PeerApplicationUris__0=urn:wonder-app-vq04:OtOpcUa` (`f15799748844`) | 49 → **51**, `734bf356306e` / `b5c93e4fce6b` |
+| vq04 | `OpcUa__Redundancy__Mode=Warm` (`68b419d808d1`), `OpcUa__PeerApplicationUris__0=urn:wonder-app-vq03:OtOpcUa` (`7fe38ae93fd6`) | 48 → **50**, `55054c72a487` / `f0a64b1a385f` |
+
+- Digests are SHA-256 prefixes of `name=value`. Values were checked against locally computed hashes and never printed.
+  The existing `OpcUa__ApplicationUri` entries hash to `urn:wonder-app-vq03:OtOpcUa` (`e1486d21fdbc`) and
+  `urn:wonder-app-vq04:OtOpcUa` (`ee29df45d241`), so each node's peer is the other node's effective URI.
+- `PromoteWhilePrimaryUnreachable` is not set, so it keeps its default (`true`).
+- **Mechanism:** the same registry `Environment` MultiString edit that the #901 `HostServiceEnv` step makes. Other entries
+  and their order are kept. `warm-env.ps1 -Action Set|Unset` in `E:\psrem-stage\otopcua-76e73e7d\` is idempotent.
+  It was tested first as a no-op `Unset` on both nodes ("env unchanged").
+- `Install-ScriptWorkerIsolation.ps1 -Verify` ran after every start: **OK 21/21** on both nodes. The isolation record does
+  not own these two names, so `-Reapply` neither needs them nor touches them.
+- `appsettings*.json` were not changed (0 `Redundancy`/`PeerApplicationUris` hits before and after).
+- ConfigDb `ServerCluster.RedundancyMode` is already Warm (1) (#717 deploy note). The deploy-time cross-check therefore
+  agrees: there is no `mode_mismatch` series and no mismatch log line.
+- **Backups:** `reg export` of the service key before the edit, in
+  `E:\psrem-stage\otopcua-76e73e7d\backup\OtOpcUaHost-pre-warm-20261113T13474{7,9}Z.reg` (vq03, vq04).
+
+On-box scripts (stage `E:\psrem-stage\otopcua-76e73e7d\`):
+- `warm-pre.ps1`: preflight, env names and digests, and the reg export;
+- `warm-env.ps1`: the env edit;
+- `pairstop.ps1`: stops the service, then waits until no `OtOpcUa.Host` / ScriptWorker process is left (the #908 guard);
+- `legstop.ps1` and `start.ps1`, from the 76e73e7d deploy.
+
+## Steps and timings (UTC)
+
+Monitoring from the Mac:
+- `i=2267` subscriptions on both nodes (250 ms);
+- a 0.5 s `/health/ready` + `/health/active` poll;
+- ScadaBridge `debug stream` of the `WarmCanary` instance;
+- ScadaBridge site and OtOpcUa logs, read over SMB.
+
+### 1. ScadaBridge test connection (Mode Unset)
+
+ScadaBridge connection `WarmCanary` (id 2, QA site `site-a`), set up through the management API:
+- primary `opc.tcp://wonder-app-vq03.zmr.zimmer.com:4840/OtOpcUa`, backup `…vq04…`;
+- security None, anonymous;
+- `serviceLevelFailover` left at its defaults (enabled, 10 s, 60 s).
+
+Template `WarmCanary` (id 27) with two attributes:
+- `CalcSeconds` → `ns=2;s=script-runner-canary/Device1/canary-calc-seconds`;
+- `VtSecond` → `ns=3;s=EQ-ec3979550d90/canary-vt-second`.
+
+Instance `WarmCanary` (id 16) is bound to the connection and deployed. QA OtOpcUa allows anonymous
+(`OpcUa:AllowAnonymous` default), so no `otua_` key was minted. Connected to the Primary at 13:42:16.5Z, and both values
+were Good and advancing at 1/s from 13:42:23Z.
+
+### 2. Baseline under Unset: stop the Primary vq03
+
+| Event | Time | Δ from stop |
+|---|---|---|
+| `Stop-Service` issued (vq03, Primary 250) | 13:44:13.008 | 0 |
+| vq04 → 250 (subscription) | 13:44:13.581 | 0.57 s |
+| vq03 Stopped | 13:44:19.109 | 6.1 s |
+| ScadaBridge: `AdapterDisconnected` → Reconnecting, Bad | 13:44:23.575 | 10.6 s |
+| Connect to vq03 failed ×2 (`Could not connect`), then `Failing over from "Primary" to "Backup"` | 13:44:40.7 / 57.9 / 13:45:15.05 | |
+| `Reconnected successfully on "Backup"`, first Good | 13:45:20.10 / .125 | **67.1 s** (Bad 56.5 s) |
+| vq03 start → ready, `-Verify` OK 21/21 | 13:46:00.5 → 13:46:09.0 | |
+
+After the restart, vq03 sat at 240 and vq04 at 250, with no failback. ScadaBridge stayed on vq04 with no Bad sample.
+
+### 3. Warm switch-over (pair window)
+
+| Event | Time |
+|---|---|
+| Env edit on both nodes (services up; no effect until start) | 13:48:0x–13:48:2x |
+| Stop vq03 (Secondary): Stopped 6.11 s, no process left 6.40 s | 13:48:47.822 |
+| Stop vq04 (Primary): Stopped 6.40 s, no process left 6.69 s | 13:48:56.027 |
+| Start vq03: ready 10.8 s; boot `Mode="Warm" … peers=[urn:wonder-app-vq04:OtOpcUa]`; **`Warm redundancy promotion to "Active" (reason=agreed, ServiceLevel=250)`** at 13:49:20.298 | 13:49:09.300 |
+| Start vq04 (after vq03 was ready): ready 9.2 s; boot `Mode="Warm"`; levels 50 then 90 | 13:49:21.660 |
+
+- **Pair outage: 22.8 s.** No node was ready from 13:48:57.2Z to 13:49:19.9Z (Mac poll).
+- **lmxopcua#908 did not occur.** `pairstop.ps1` waited for the processes to exit, and nothing was moved.
+
+**#138 during the window (unplanned bonus):**
+- ScadaBridge (Bad from 13:49:07.139) first reached the **booting Standby** vq04: `Reconnected successfully on "Backup"`
+  at 13:49:30.841.
+- At once it logged `"Backup" endpoint reports ServiceLevel 50 (Degraded band) — will check the other endpoint now
+  (connect-time preference)`.
+- Then `Failing over from "Backup" to "Primary" (ServiceLevel 50 Degraded for 4s; "Primary" reports 250)` at
+  13:49:35.338.
+- Reconnected on the Primary at 13:49:39.419; Good at 13:49:39.441.
+
+One connect attempt to vq04 during its boot failed with `Endpoint does not support the user identity type provided`
+(13:49:25.4). That is transient: the endpoint was coming up.
+
+### Step 3 gate
+
+| Gate | Result |
+|---|---|
+| Primary 250, Standby in the Warm band | **PASS.** vq03 **250**; vq04 **90** (after 50 during boot). `redundancy` CLI on both: Mode **Warm**, Server URIs self-first (`[vq03, vq04]` on vq03, `[vq04, vq03]` on vq04) |
+| RedundancySupport = Warm | **PASS** (`redundancy` CLI, i=3709) |
+| Standby data reads | **PASS.** On vq04, `canary-calc-seconds` and `canary-vt-second` read **`BadOutOfService`**, value null, ×2. On vq03 both are Good and advancing |
+| Standby write refused `BadInvalidState` | **Not exercisable on QA, see deviation 1.** A typed anonymous Write of both canaries on vq04 returned **`BadNotWritable`**, and so did the same Write on the Primary. QA's only tags are the read-only Calculation tag and the VirtualTag, so the SDK's AccessLevel check answers before the Warm refusal can. The Warm refusal is proven by #717 T13 L10 |
+| LocalDb sync | **PASS.** `localdb-replication`: "connected; oplog backlog 0" on both; `localdb_oplog_depth` 0 |
+| Canary on the Primary | **PASS.** Good, about 1/s; script runner Ready, shards 0/1 up, quarantined 0 |
+| Metrics | vq03 `warm_role` **1**, `warm_transition{promote,agreed}` 1. vq04 `warm_role` **0**, `service_level_change{50}`/`{90}`. No `mode_mismatch` or `snapshot_rejected` series |
+| Health | `/health/ready` 200 on both, akka 2 members / 0 unreachable; `/health/active` 200 vq03 / 503 vq04 |
+| Logs | Since the switch, vq03 has no ERR/FTL. vq04 has only the standby `/health/active` 503 / `cluster-primary` lines, plus Warnings already seen before the switch (singleton-proxy at boot, the DEV mesh-PSK note). Also one `Client connects with an endpointUrl … which does not match Server hostnames` Warning: ScadaBridge uses the FQDN, and it is cosmetic |
+
+### 4. #138 live gate under Warm
+
+**4a: stop the Warm Primary vq03.**
+
+| Event | Time | Δ |
+|---|---|---|
+| `Stop-Service` issued (vq03, Primary 250) | 13:52:56.661 | 0 |
+| vq03: `Host stopping …(lmxopcua#906)` → **`demotion to "Standby" (reason=membership, ServiceLevel=0)`** | 13:52:56.675 / .689 | 0.03 s |
+| vq04: **`promotion to "Active" (reason=agreed, ServiceLevel=250)`**; subscription 250 | 13:52:56.842 / 13:52:57.314 | 0.18 s / 0.65 s |
+| ScadaBridge: the last vq03 sample delivered `BadOutOfService` (stream: Bad) | 13:52:57.510 | 0.85 s |
+| DCL: **`"Primary" endpoint reports ServiceLevel 0 (Degraded band, < 200) — will check the other endpoint if it persists for 10s`** | 13:52:57.435 | 0.77 s |
+| vq03 Stopped (6.10 s); the next poll found the level unreadable: `ServiceLevel is unknown — degraded watch cleared` | 13:53:03.432 | 6.8 s |
+| `AdapterDisconnected` → Reconnecting | 13:53:06.439 | 9.8 s |
+| Connect to vq03 failed ×2, then `Failing over from "Primary" to "Backup"` (the connectivity trigger) | 13:53:23.6 / 40.7 / 57.855 | |
+| `Reconnected successfully on "Backup"` (vq04, 250); first Good | 13:54:02.886 / 13:54:02.913 | **66.3 s** |
+
+**Cause recorded:** the connectivity failover, not ServiceLevel. The leaving node reported 0 for about 6 s. That is
+shorter than the 10 s degraded window, so the streak was cleared when the read went unknown, and the switch then waited
+for the existing 3 × (connect timeout + 5 s) retry cycle. Time to Good matched the Unset baseline (66.3 s vs 67.1 s).
+This is filed as follow-up ScadaBridge#141.
+
+**4b: restart the old Primary.**
+- Start at 13:54:33.5; ready in 7.9 s; `-Verify` OK 21/21.
+- Boot: `Mode="Warm"`, levels 50 then **90** (13:54:44). `warm_role` 0, `/health/active` 503.
+- **No failback:** vq04 stayed at 250, `warm_role` 1, with no new transition.
+- ScadaBridge stayed on vq04 with no Bad sample and no DCL line for 110 s (13:54:33 → 13:56:45). It never started a
+  degraded streak, because its endpoint was at 250.
+- The 60 s dwell could not be exercised live (see deviation 2).
+
+**4c: ScadaBridge on the Standby over a healthy session (the case only #138 handles).**
+- The connection's subscription display name was changed and site artifacts were deployed (13:56:48.087). This is an
+  in-place reconfigure, which re-arms the connection on its **primary**, vq03, now the Standby at 90.
+- 13:56:52.401: `Reconnected successfully on "Primary"`.
+- 13:56:52.413: **`"Primary" endpoint reports ServiceLevel 90 (Degraded band, < 200) — will check the other endpoint now (connect-time preference)`**.
+- 13:56:52.436: **`Failing over from "Primary" to "Backup" (ServiceLevel 90 Degraded for 0s; "Backup" reports 250)`** → `Entering Reconnecting state (ServiceLevel failover)`.
+- 13:56:52.475: `Reconnected successfully on "Backup"`. First Good at 13:56:52.504.
+- **The move took 0.1 s from the connect to the Standby**, with no disconnect involved. It took 4.4 s from the
+  reconfigure, most of which is the reconnect to vq03.
+
+The site health report then showed `WarmCanary: Connected to backup`, 2 Good / 0 Bad. `MxGateway` stayed Connected
+with 2196/2196 Good throughout, and 10 instances are deployed (9 + `WarmCanary`).
+
+## End state (13:58Z)
+
+| | vq03 | vq04 |
+|---|---|---|
+| Build | `76e73e7d` | `76e73e7d` |
+| Mode | Warm | Warm |
+| Role / ServiceLevel | Standby **90** | **Primary 250** |
+| Canary | `BadOutOfService` | Good, about 1/s |
+| LocalDb sync | connected, backlog 0 | connected, backlog 0 |
+| `-Verify` | OK 21/21 | OK 21/21 |
+
+- **ScadaBridge `WarmCanary`:** on the backup endpoint (vq04), Good. It is kept as the Warm monitor.
+- **HistorianGateway `/metrics` (vq03 `:5220`):** only `Probe`, plus the counters from before the window
+  (`BrowseTagNames` 1, `ReadRaw` 1, `ReadAggregate` 1 ok / 2 error). No write or tag-creation RPC.
+
+## Rollback (back to Unset)
+
+On **each** node:
+1. `& "E:\psrem-stage\otopcua-76e73e7d\warm-env.ps1" -Action Unset`. This removes exactly the two entries.
+   Expected env digests: vq03 49 / `5750bbeb4734`, vq04 48 / `ef146b2681b4`.
+2. Stop **both** nodes (`pairstop.ps1`, Standby first).
+3. Start both, Primary-to-be first (`start.ps1` waits for ready and runs `-Verify`).
+4. Check 250/240, Mode `None` in the `redundancy` CLI, Good canary on both, and the boot banner `Mode="Unset"`.
+
+The `.reg` exports are a fallback only. Mode is startup-static, so never run a mixed pair (one Warm, one Unset).
+
+The ScadaBridge `WarmCanary` connection works under Unset too (250/240 never starts a streak). Remove it only if the
+monitor is no longer wanted: `instance delete --id 16`, `template delete --id 27`, `data-connection delete --id 2`, then
+`deploy artifacts --site-id 2`.
+
+## Deviations
+
+1. **Write refusal on the Standby.** It could not be shown as `BadInvalidState` on QA. Both QA tags are read-only, so
+   every Write is refused `BadNotWritable` before the Warm check, on either node. Part 9 method refusal could not be
+   shown either: QA has no alarm condition.
+   - What QA shows instead: a Write on the Standby changes nothing, and no tag is writable on either node.
+   - The `BadInvalidState` path itself is proven by T13 L10 (`f826af8e`).
+   - To prove it on QA would need a writable tag, which means a config deploy with a writable driver. Out of scope here.
+2. **60 s dwell.** No live sequence produced two ServiceLevel switches inside 60 s. An in-place reconfigure resets the
+   dwell by design, and a role flip with a healthy session needs the Primary to leave, which drops the session. During
+   4b ScadaBridge never left vq04 (no flap). The dwell is covered by ScadaBridge's actor tests.
+3. **"ServiceLevel poll active" evidence.** At Information level a Healthy sample logs nothing (by design), so step 1
+   has no poll line. The poll is evidenced by the 50/90/0 `Degraded band` Warnings in steps 3, 4a and 4c, each logged
+   within 15–35 ms of the connect or demotion.
+4. **Primary placement.** The Primary moved from vq03 to vq04 in 4a, and Warm does not fail back. QA now runs with vq04
+   as Primary. This is expected; it is not restored.