diff --git a/docs/deploy/2026-11-13-qa-warm-717.md b/docs/deploy/2026-11-13-qa-warm-717.md new file mode 100644 index 000000000..e6d9a2bf2 --- /dev/null +++ b/docs/deploy/2026-11-13-qa-warm-717.md @@ -0,0 +1,209 @@ +# QA vq03/vq04: `OpcUa:Redundancy:Mode=Warm` switch-over and #138 live gate (2026-11-13) + +> **Status: EXECUTED 2026-11-13, 13:41–13:58Z. QA is on Warm.** +> - #907 (tier flip) and ScadaBridge#138 (ServiceLevel-aware failover) were gated together on the QA pair. +> - The owner approved the change, with WinRM for the on-box config, stop/start and verify steps only. +> - **End state:** both nodes run `Mode=Warm` on build `76e73e7d`. **vq04 is Primary at 250 and vq03 is Standby at 90.** +> The Primary moved during the gate, and Warm does not fail back. +> - ScadaBridge's `WarmCanary` connection stays in place as a monitor. + +Nothing was rebuilt: both nodes keep master `76e73e7d` from `2026-11-13-vd03-qa-to-76e73e7d.md`. The only change is two +service-env entries per node. + +Not done or touched: +- no configuration deploy, no Galaxy or historian writes, no tag creation; +- no other service on the boxes (mxgw, HistorianGateway, Overview, collector; ScadaBridge only through its management API); +- DEV vd03 stays on `Mode` Unset. + +## Change + +| Node | Entries added to the `OtOpcUaHost` service env | Env after (count, ordered / sorted digest) | +|---|---|---| +| vq03 | `OpcUa__Redundancy__Mode=Warm` (`68b419d808d1`), `OpcUa__PeerApplicationUris__0=urn:wonder-app-vq04:OtOpcUa` (`f15799748844`) | 49 → **51**, `734bf356306e` / `b5c93e4fce6b` | +| vq04 | `OpcUa__Redundancy__Mode=Warm` (`68b419d808d1`), `OpcUa__PeerApplicationUris__0=urn:wonder-app-vq03:OtOpcUa` (`7fe38ae93fd6`) | 48 → **50**, `55054c72a487` / `f0a64b1a385f` | + +- Digests are SHA-256 prefixes of `name=value`. Values were checked against locally computed hashes and never printed. + The existing `OpcUa__ApplicationUri` entries hash to `urn:wonder-app-vq03:OtOpcUa` (`e1486d21fdbc`) and + `urn:wonder-app-vq04:OtOpcUa` (`ee29df45d241`), so each node's peer is the other node's effective URI. +- `PromoteWhilePrimaryUnreachable` is not set, so it keeps its default (`true`). +- **Mechanism:** the same registry `Environment` MultiString edit that the #901 `HostServiceEnv` step makes. Other entries + and their order are kept. `warm-env.ps1 -Action Set|Unset` in `E:\psrem-stage\otopcua-76e73e7d\` is idempotent. + It was tested first as a no-op `Unset` on both nodes ("env unchanged"). +- `Install-ScriptWorkerIsolation.ps1 -Verify` ran after every start: **OK 21/21** on both nodes. The isolation record does + not own these two names, so `-Reapply` neither needs them nor touches them. +- `appsettings*.json` were not changed (0 `Redundancy`/`PeerApplicationUris` hits before and after). +- ConfigDb `ServerCluster.RedundancyMode` is already Warm (1) (#717 deploy note). The deploy-time cross-check therefore + agrees: there is no `mode_mismatch` series and no mismatch log line. +- **Backups:** `reg export` of the service key before the edit, in + `E:\psrem-stage\otopcua-76e73e7d\backup\OtOpcUaHost-pre-warm-20261113T13474{7,9}Z.reg` (vq03, vq04). + +On-box scripts (stage `E:\psrem-stage\otopcua-76e73e7d\`): +- `warm-pre.ps1`: preflight, env names and digests, and the reg export; +- `warm-env.ps1`: the env edit; +- `pairstop.ps1`: stops the service, then waits until no `OtOpcUa.Host` / ScriptWorker process is left (the #908 guard); +- `legstop.ps1` and `start.ps1`, from the 76e73e7d deploy. + +## Steps and timings (UTC) + +Monitoring from the Mac: +- `i=2267` subscriptions on both nodes (250 ms); +- a 0.5 s `/health/ready` + `/health/active` poll; +- ScadaBridge `debug stream` of the `WarmCanary` instance; +- ScadaBridge site and OtOpcUa logs, read over SMB. + +### 1. ScadaBridge test connection (Mode Unset) + +ScadaBridge connection `WarmCanary` (id 2, QA site `site-a`), set up through the management API: +- primary `opc.tcp://wonder-app-vq03.zmr.zimmer.com:4840/OtOpcUa`, backup `…vq04…`; +- security None, anonymous; +- `serviceLevelFailover` left at its defaults (enabled, 10 s, 60 s). + +Template `WarmCanary` (id 27) with two attributes: +- `CalcSeconds` → `ns=2;s=script-runner-canary/Device1/canary-calc-seconds`; +- `VtSecond` → `ns=3;s=EQ-ec3979550d90/canary-vt-second`. + +Instance `WarmCanary` (id 16) is bound to the connection and deployed. QA OtOpcUa allows anonymous +(`OpcUa:AllowAnonymous` default), so no `otua_` key was minted. Connected to the Primary at 13:42:16.5Z, and both values +were Good and advancing at 1/s from 13:42:23Z. + +### 2. Baseline under Unset: stop the Primary vq03 + +| Event | Time | Δ from stop | +|---|---|---| +| `Stop-Service` issued (vq03, Primary 250) | 13:44:13.008 | 0 | +| vq04 → 250 (subscription) | 13:44:13.581 | 0.57 s | +| vq03 Stopped | 13:44:19.109 | 6.1 s | +| ScadaBridge: `AdapterDisconnected` → Reconnecting, Bad | 13:44:23.575 | 10.6 s | +| Connect to vq03 failed ×2 (`Could not connect`), then `Failing over from "Primary" to "Backup"` | 13:44:40.7 / 57.9 / 13:45:15.05 | | +| `Reconnected successfully on "Backup"`, first Good | 13:45:20.10 / .125 | **67.1 s** (Bad 56.5 s) | +| vq03 start → ready, `-Verify` OK 21/21 | 13:46:00.5 → 13:46:09.0 | | + +After the restart, vq03 sat at 240 and vq04 at 250, with no failback. ScadaBridge stayed on vq04 with no Bad sample. + +### 3. Warm switch-over (pair window) + +| Event | Time | +|---|---| +| Env edit on both nodes (services up; no effect until start) | 13:48:0x–13:48:2x | +| Stop vq03 (Secondary): Stopped 6.11 s, no process left 6.40 s | 13:48:47.822 | +| Stop vq04 (Primary): Stopped 6.40 s, no process left 6.69 s | 13:48:56.027 | +| Start vq03: ready 10.8 s; boot `Mode="Warm" … peers=[urn:wonder-app-vq04:OtOpcUa]`; **`Warm redundancy promotion to "Active" (reason=agreed, ServiceLevel=250)`** at 13:49:20.298 | 13:49:09.300 | +| Start vq04 (after vq03 was ready): ready 9.2 s; boot `Mode="Warm"`; levels 50 then 90 | 13:49:21.660 | + +- **Pair outage: 22.8 s.** No node was ready from 13:48:57.2Z to 13:49:19.9Z (Mac poll). +- **lmxopcua#908 did not occur.** `pairstop.ps1` waited for the processes to exit, and nothing was moved. + +**#138 during the window (unplanned bonus):** +- ScadaBridge (Bad from 13:49:07.139) first reached the **booting Standby** vq04: `Reconnected successfully on "Backup"` + at 13:49:30.841. +- At once it logged `"Backup" endpoint reports ServiceLevel 50 (Degraded band) — will check the other endpoint now + (connect-time preference)`. +- Then `Failing over from "Backup" to "Primary" (ServiceLevel 50 Degraded for 4s; "Primary" reports 250)` at + 13:49:35.338. +- Reconnected on the Primary at 13:49:39.419; Good at 13:49:39.441. + +One connect attempt to vq04 during its boot failed with `Endpoint does not support the user identity type provided` +(13:49:25.4). That is transient: the endpoint was coming up. + +### Step 3 gate + +| Gate | Result | +|---|---| +| Primary 250, Standby in the Warm band | **PASS.** vq03 **250**; vq04 **90** (after 50 during boot). `redundancy` CLI on both: Mode **Warm**, Server URIs self-first (`[vq03, vq04]` on vq03, `[vq04, vq03]` on vq04) | +| RedundancySupport = Warm | **PASS** (`redundancy` CLI, i=3709) | +| Standby data reads | **PASS.** On vq04, `canary-calc-seconds` and `canary-vt-second` read **`BadOutOfService`**, value null, ×2. On vq03 both are Good and advancing | +| Standby write refused `BadInvalidState` | **Not exercisable on QA, see deviation 1.** A typed anonymous Write of both canaries on vq04 returned **`BadNotWritable`**, and so did the same Write on the Primary. QA's only tags are the read-only Calculation tag and the VirtualTag, so the SDK's AccessLevel check answers before the Warm refusal can. The Warm refusal is proven by #717 T13 L10 | +| LocalDb sync | **PASS.** `localdb-replication`: "connected; oplog backlog 0" on both; `localdb_oplog_depth` 0 | +| Canary on the Primary | **PASS.** Good, about 1/s; script runner Ready, shards 0/1 up, quarantined 0 | +| Metrics | vq03 `warm_role` **1**, `warm_transition{promote,agreed}` 1. vq04 `warm_role` **0**, `service_level_change{50}`/`{90}`. No `mode_mismatch` or `snapshot_rejected` series | +| Health | `/health/ready` 200 on both, akka 2 members / 0 unreachable; `/health/active` 200 vq03 / 503 vq04 | +| Logs | Since the switch, vq03 has no ERR/FTL. vq04 has only the standby `/health/active` 503 / `cluster-primary` lines, plus Warnings already seen before the switch (singleton-proxy at boot, the DEV mesh-PSK note). Also one `Client connects with an endpointUrl … which does not match Server hostnames` Warning: ScadaBridge uses the FQDN, and it is cosmetic | + +### 4. #138 live gate under Warm + +**4a: stop the Warm Primary vq03.** + +| Event | Time | Δ | +|---|---|---| +| `Stop-Service` issued (vq03, Primary 250) | 13:52:56.661 | 0 | +| vq03: `Host stopping …(lmxopcua#906)` → **`demotion to "Standby" (reason=membership, ServiceLevel=0)`** | 13:52:56.675 / .689 | 0.03 s | +| vq04: **`promotion to "Active" (reason=agreed, ServiceLevel=250)`**; subscription 250 | 13:52:56.842 / 13:52:57.314 | 0.18 s / 0.65 s | +| ScadaBridge: the last vq03 sample delivered `BadOutOfService` (stream: Bad) | 13:52:57.510 | 0.85 s | +| DCL: **`"Primary" endpoint reports ServiceLevel 0 (Degraded band, < 200) — will check the other endpoint if it persists for 10s`** | 13:52:57.435 | 0.77 s | +| vq03 Stopped (6.10 s); the next poll found the level unreadable: `ServiceLevel is unknown — degraded watch cleared` | 13:53:03.432 | 6.8 s | +| `AdapterDisconnected` → Reconnecting | 13:53:06.439 | 9.8 s | +| Connect to vq03 failed ×2, then `Failing over from "Primary" to "Backup"` (the connectivity trigger) | 13:53:23.6 / 40.7 / 57.855 | | +| `Reconnected successfully on "Backup"` (vq04, 250); first Good | 13:54:02.886 / 13:54:02.913 | **66.3 s** | + +**Cause recorded:** the connectivity failover, not ServiceLevel. The leaving node reported 0 for about 6 s. That is +shorter than the 10 s degraded window, so the streak was cleared when the read went unknown, and the switch then waited +for the existing 3 × (connect timeout + 5 s) retry cycle. Time to Good matched the Unset baseline (66.3 s vs 67.1 s). +This is filed as follow-up ScadaBridge#141. + +**4b: restart the old Primary.** +- Start at 13:54:33.5; ready in 7.9 s; `-Verify` OK 21/21. +- Boot: `Mode="Warm"`, levels 50 then **90** (13:54:44). `warm_role` 0, `/health/active` 503. +- **No failback:** vq04 stayed at 250, `warm_role` 1, with no new transition. +- ScadaBridge stayed on vq04 with no Bad sample and no DCL line for 110 s (13:54:33 → 13:56:45). It never started a + degraded streak, because its endpoint was at 250. +- The 60 s dwell could not be exercised live (see deviation 2). + +**4c: ScadaBridge on the Standby over a healthy session (the case only #138 handles).** +- The connection's subscription display name was changed and site artifacts were deployed (13:56:48.087). This is an + in-place reconfigure, which re-arms the connection on its **primary**, vq03, now the Standby at 90. +- 13:56:52.401: `Reconnected successfully on "Primary"`. +- 13:56:52.413: **`"Primary" endpoint reports ServiceLevel 90 (Degraded band, < 200) — will check the other endpoint now (connect-time preference)`**. +- 13:56:52.436: **`Failing over from "Primary" to "Backup" (ServiceLevel 90 Degraded for 0s; "Backup" reports 250)`** → `Entering Reconnecting state (ServiceLevel failover)`. +- 13:56:52.475: `Reconnected successfully on "Backup"`. First Good at 13:56:52.504. +- **The move took 0.1 s from the connect to the Standby**, with no disconnect involved. It took 4.4 s from the + reconfigure, most of which is the reconnect to vq03. + +The site health report then showed `WarmCanary: Connected to backup`, 2 Good / 0 Bad. `MxGateway` stayed Connected +with 2196/2196 Good throughout, and 10 instances are deployed (9 + `WarmCanary`). + +## End state (13:58Z) + +| | vq03 | vq04 | +|---|---|---| +| Build | `76e73e7d` | `76e73e7d` | +| Mode | Warm | Warm | +| Role / ServiceLevel | Standby **90** | **Primary 250** | +| Canary | `BadOutOfService` | Good, about 1/s | +| LocalDb sync | connected, backlog 0 | connected, backlog 0 | +| `-Verify` | OK 21/21 | OK 21/21 | + +- **ScadaBridge `WarmCanary`:** on the backup endpoint (vq04), Good. It is kept as the Warm monitor. +- **HistorianGateway `/metrics` (vq03 `:5220`):** only `Probe`, plus the counters from before the window + (`BrowseTagNames` 1, `ReadRaw` 1, `ReadAggregate` 1 ok / 2 error). No write or tag-creation RPC. + +## Rollback (back to Unset) + +On **each** node: +1. `& "E:\psrem-stage\otopcua-76e73e7d\warm-env.ps1" -Action Unset`. This removes exactly the two entries. + Expected env digests: vq03 49 / `5750bbeb4734`, vq04 48 / `ef146b2681b4`. +2. Stop **both** nodes (`pairstop.ps1`, Standby first). +3. Start both, Primary-to-be first (`start.ps1` waits for ready and runs `-Verify`). +4. Check 250/240, Mode `None` in the `redundancy` CLI, Good canary on both, and the boot banner `Mode="Unset"`. + +The `.reg` exports are a fallback only. Mode is startup-static, so never run a mixed pair (one Warm, one Unset). + +The ScadaBridge `WarmCanary` connection works under Unset too (250/240 never starts a streak). Remove it only if the +monitor is no longer wanted: `instance delete --id 16`, `template delete --id 27`, `data-connection delete --id 2`, then +`deploy artifacts --site-id 2`. + +## Deviations + +1. **Write refusal on the Standby.** It could not be shown as `BadInvalidState` on QA. Both QA tags are read-only, so + every Write is refused `BadNotWritable` before the Warm check, on either node. Part 9 method refusal could not be + shown either: QA has no alarm condition. + - What QA shows instead: a Write on the Standby changes nothing, and no tag is writable on either node. + - The `BadInvalidState` path itself is proven by T13 L10 (`f826af8e`). + - To prove it on QA would need a writable tag, which means a config deploy with a writable driver. Out of scope here. +2. **60 s dwell.** No live sequence produced two ServiceLevel switches inside 60 s. An in-place reconfigure resets the + dwell by design, and a role flip with a healthy session needs the Primary to leave, which drops the session. During + 4b ScadaBridge never left vq04 (no flap). The dwell is covered by ScadaBridge's actor tests. +3. **"ServiceLevel poll active" evidence.** At Information level a Healthy sample logs nothing (by design), so step 1 + has no poll line. The poll is evidenced by the 50/90/0 `Degraded band` Warnings in steps 3, 4a and 4c, each logged + within 15–35 ms of the connect or demotion. +4. **Primary placement.** The Primary moved from vq03 to vq04 in 4a, and Warm does not fail back. QA now runs with vq04 + as Primary. This is expected; it is not restored.