Files
lmxopcua/docs/deploy/2026-11-13-qa-warm-717.md
T
Joseph Doherty 1862fab671
v2-ci / unit-tests (push) Failing after 14m56s
docs(deploy): QA vq03/vq04 switched to OpcUa:Redundancy:Mode=Warm; #138 live gate PASS (#907, #717)
Two service-env entries per node (Mode=Warm, PeerApplicationUris__0), pair stopped and
started together. Gates: 250/90, RedundancySupport Warm, standby BadOutOfService,
LocalDb sync, canary Good on the Primary; standby Write could only show BadNotWritable
(QA tags are read-only). ScadaBridge WarmCanary moved off the Standby by ServiceLevel in
0.1 s (connect-time preference); graceful Primary stop still took the 66 s connectivity
path (ScadaBridge#141). End state: Warm, vq04 Primary.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 09:00:35 -05:00

210 lines
14 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# QA vq03/vq04: `OpcUa:Redundancy:Mode=Warm` switch-over and #138 live gate (2026-11-13)
> **Status: EXECUTED 2026-11-13, 13:41–13:58Z. QA is on Warm.**
> - #907 (tier flip) and ScadaBridge#138 (ServiceLevel-aware failover) were gated together on the QA pair.
> - The owner approved the change, with WinRM for the on-box config, stop/start and verify steps only.
> - **End state:** both nodes run `Mode=Warm` on build `76e73e7d`. **vq04 is Primary at 250 and vq03 is Standby at 90.**
> The Primary moved during the gate, and Warm does not fail back.
> - ScadaBridge's `WarmCanary` connection stays in place as a monitor.
Nothing was rebuilt: both nodes keep master `76e73e7d` from `2026-11-13-vd03-qa-to-76e73e7d.md`. The only change is two
service-env entries per node.
Not done or touched:
- no configuration deploy, no Galaxy or historian writes, no tag creation;
- no other service on the boxes (mxgw, HistorianGateway, Overview, collector; ScadaBridge only through its management API);
- DEV vd03 stays on `Mode` Unset.
## Change
| Node | Entries added to the `OtOpcUaHost` service env | Env after (count, ordered / sorted digest) |
|---|---|---|
| vq03 | `OpcUa__Redundancy__Mode=Warm` (`68b419d808d1`), `OpcUa__PeerApplicationUris__0=urn:wonder-app-vq04:OtOpcUa` (`f15799748844`) | 49 → **51**, `734bf356306e` / `b5c93e4fce6b` |
| vq04 | `OpcUa__Redundancy__Mode=Warm` (`68b419d808d1`), `OpcUa__PeerApplicationUris__0=urn:wonder-app-vq03:OtOpcUa` (`7fe38ae93fd6`) | 48 → **50**, `55054c72a487` / `f0a64b1a385f` |
- Digests are SHA-256 prefixes of `name=value`. Values were checked against locally computed hashes and never printed.
The existing `OpcUa__ApplicationUri` entries hash to `urn:wonder-app-vq03:OtOpcUa` (`e1486d21fdbc`) and
`urn:wonder-app-vq04:OtOpcUa` (`ee29df45d241`), so each node's peer is the other node's effective URI.
- `PromoteWhilePrimaryUnreachable` is not set, so it keeps its default (`true`).
- **Mechanism:** the same registry `Environment` MultiString edit that the #901 `HostServiceEnv` step makes. Other entries
and their order are kept. `warm-env.ps1 -Action Set|Unset` in `E:\psrem-stage\otopcua-76e73e7d\` is idempotent.
It was tested first as a no-op `Unset` on both nodes ("env unchanged").
- `Install-ScriptWorkerIsolation.ps1 -Verify` ran after every start: **OK 21/21** on both nodes. The isolation record does
not own these two names, so `-Reapply` neither needs them nor touches them.
- `appsettings*.json` were not changed (0 `Redundancy`/`PeerApplicationUris` hits before and after).
- ConfigDb `ServerCluster.RedundancyMode` is already Warm (1) (#717 deploy note). The deploy-time cross-check therefore
agrees: there is no `mode_mismatch` series and no mismatch log line.
- **Backups:** `reg export` of the service key before the edit, in
`E:\psrem-stage\otopcua-76e73e7d\backup\OtOpcUaHost-pre-warm-20261113T13474{7,9}Z.reg` (vq03, vq04).
On-box scripts (stage `E:\psrem-stage\otopcua-76e73e7d\`):
- `warm-pre.ps1`: preflight, env names and digests, and the reg export;
- `warm-env.ps1`: the env edit;
- `pairstop.ps1`: stops the service, then waits until no `OtOpcUa.Host` / ScriptWorker process is left (the #908 guard);
- `legstop.ps1` and `start.ps1`, from the 76e73e7d deploy.
## Steps and timings (UTC)
Monitoring from the Mac:
- `i=2267` subscriptions on both nodes (250 ms);
- a 0.5 s `/health/ready` + `/health/active` poll;
- ScadaBridge `debug stream` of the `WarmCanary` instance;
- ScadaBridge site and OtOpcUa logs, read over SMB.
### 1. ScadaBridge test connection (Mode Unset)
ScadaBridge connection `WarmCanary` (id 2, QA site `site-a`), set up through the management API:
- primary `opc.tcp://wonder-app-vq03.zmr.zimmer.com:4840/OtOpcUa`, backup `…vq04…`;
- security None, anonymous;
- `serviceLevelFailover` left at its defaults (enabled, 10 s, 60 s).
Template `WarmCanary` (id 27) with two attributes:
- `CalcSeconds` → `ns=2;s=script-runner-canary/Device1/canary-calc-seconds`;
- `VtSecond` → `ns=3;s=EQ-ec3979550d90/canary-vt-second`.
Instance `WarmCanary` (id 16) is bound to the connection and deployed. QA OtOpcUa allows anonymous
(`OpcUa:AllowAnonymous` default), so no `otua_` key was minted. Connected to the Primary at 13:42:16.5Z, and both values
were Good and advancing at 1/s from 13:42:23Z.
### 2. Baseline under Unset: stop the Primary vq03
| Event | Time | Δ from stop |
|---|---|---|
| `Stop-Service` issued (vq03, Primary 250) | 13:44:13.008 | 0 |
| vq04 → 250 (subscription) | 13:44:13.581 | 0.57 s |
| vq03 Stopped | 13:44:19.109 | 6.1 s |
| ScadaBridge: `AdapterDisconnected` → Reconnecting, Bad | 13:44:23.575 | 10.6 s |
| Connect to vq03 failed ×2 (`Could not connect`), then `Failing over from "Primary" to "Backup"` | 13:44:40.7 / 57.9 / 13:45:15.05 | |
| `Reconnected successfully on "Backup"`, first Good | 13:45:20.10 / .125 | **67.1 s** (Bad 56.5 s) |
| vq03 start → ready, `-Verify` OK 21/21 | 13:46:00.5 → 13:46:09.0 | |
After the restart, vq03 sat at 240 and vq04 at 250, with no failback. ScadaBridge stayed on vq04 with no Bad sample.
### 3. Warm switch-over (pair window)
| Event | Time |
|---|---|
| Env edit on both nodes (services up; no effect until start) | 13:48:0x–13:48:2x |
| Stop vq03 (Secondary): Stopped 6.11 s, no process left 6.40 s | 13:48:47.822 |
| Stop vq04 (Primary): Stopped 6.40 s, no process left 6.69 s | 13:48:56.027 |
| Start vq03: ready 10.8 s; boot `Mode="Warm" … peers=[urn:wonder-app-vq04:OtOpcUa]`; **`Warm redundancy promotion to "Active" (reason=agreed, ServiceLevel=250)`** at 13:49:20.298 | 13:49:09.300 |
| Start vq04 (after vq03 was ready): ready 9.2 s; boot `Mode="Warm"`; levels 50 then 90 | 13:49:21.660 |
- **Pair outage: 22.8 s.** No node was ready from 13:48:57.2Z to 13:49:19.9Z (Mac poll).
- **lmxopcua#908 did not occur.** `pairstop.ps1` waited for the processes to exit, and nothing was moved.
**#138 during the window (unplanned bonus):**
- ScadaBridge (Bad from 13:49:07.139) first reached the **booting Standby** vq04: `Reconnected successfully on "Backup"`
at 13:49:30.841.
- At once it logged `"Backup" endpoint reports ServiceLevel 50 (Degraded band) — will check the other endpoint now
(connect-time preference)`.
- Then `Failing over from "Backup" to "Primary" (ServiceLevel 50 Degraded for 4s; "Primary" reports 250)` at
13:49:35.338.
- Reconnected on the Primary at 13:49:39.419; Good at 13:49:39.441.
One connect attempt to vq04 during its boot failed with `Endpoint does not support the user identity type provided`
(13:49:25.4). That is transient: the endpoint was coming up.
### Step 3 gate
| Gate | Result |
|---|---|
| Primary 250, Standby in the Warm band | **PASS.** vq03 **250**; vq04 **90** (after 50 during boot). `redundancy` CLI on both: Mode **Warm**, Server URIs self-first (`[vq03, vq04]` on vq03, `[vq04, vq03]` on vq04) |
| RedundancySupport = Warm | **PASS** (`redundancy` CLI, i=3709) |
| Standby data reads | **PASS.** On vq04, `canary-calc-seconds` and `canary-vt-second` read **`BadOutOfService`**, value null, ×2. On vq03 both are Good and advancing |
| Standby write refused `BadInvalidState` | **Not exercisable on QA, see deviation 1.** A typed anonymous Write of both canaries on vq04 returned **`BadNotWritable`**, and so did the same Write on the Primary. QA's only tags are the read-only Calculation tag and the VirtualTag, so the SDK's AccessLevel check answers before the Warm refusal can. The Warm refusal is proven by #717 T13 L10 |
| LocalDb sync | **PASS.** `localdb-replication`: "connected; oplog backlog 0" on both; `localdb_oplog_depth` 0 |
| Canary on the Primary | **PASS.** Good, about 1/s; script runner Ready, shards 0/1 up, quarantined 0 |
| Metrics | vq03 `warm_role` **1**, `warm_transition{promote,agreed}` 1. vq04 `warm_role` **0**, `service_level_change{50}`/`{90}`. No `mode_mismatch` or `snapshot_rejected` series |
| Health | `/health/ready` 200 on both, akka 2 members / 0 unreachable; `/health/active` 200 vq03 / 503 vq04 |
| Logs | Since the switch, vq03 has no ERR/FTL. vq04 has only the standby `/health/active` 503 / `cluster-primary` lines, plus Warnings already seen before the switch (singleton-proxy at boot, the DEV mesh-PSK note). Also one `Client connects with an endpointUrl … which does not match Server hostnames` Warning: ScadaBridge uses the FQDN, and it is cosmetic |
### 4. #138 live gate under Warm
**4a: stop the Warm Primary vq03.**
| Event | Time | Δ |
|---|---|---|
| `Stop-Service` issued (vq03, Primary 250) | 13:52:56.661 | 0 |
| vq03: `Host stopping …(lmxopcua#906)` → **`demotion to "Standby" (reason=membership, ServiceLevel=0)`** | 13:52:56.675 / .689 | 0.03 s |
| vq04: **`promotion to "Active" (reason=agreed, ServiceLevel=250)`**; subscription 250 | 13:52:56.842 / 13:52:57.314 | 0.18 s / 0.65 s |
| ScadaBridge: the last vq03 sample delivered `BadOutOfService` (stream: Bad) | 13:52:57.510 | 0.85 s |
| DCL: **`"Primary" endpoint reports ServiceLevel 0 (Degraded band, < 200) — will check the other endpoint if it persists for 10s`** | 13:52:57.435 | 0.77 s |
| vq03 Stopped (6.10 s); the next poll found the level unreadable: `ServiceLevel is unknown — degraded watch cleared` | 13:53:03.432 | 6.8 s |
| `AdapterDisconnected` → Reconnecting | 13:53:06.439 | 9.8 s |
| Connect to vq03 failed ×2, then `Failing over from "Primary" to "Backup"` (the connectivity trigger) | 13:53:23.6 / 40.7 / 57.855 | |
| `Reconnected successfully on "Backup"` (vq04, 250); first Good | 13:54:02.886 / 13:54:02.913 | **66.3 s** |
**Cause recorded:** the connectivity failover, not ServiceLevel. The leaving node reported 0 for about 6 s. That is
shorter than the 10 s degraded window, so the streak was cleared when the read went unknown, and the switch then waited
for the existing 3 × (connect timeout + 5 s) retry cycle. Time to Good matched the Unset baseline (66.3 s vs 67.1 s).
This is filed as follow-up ScadaBridge#141.
**4b: restart the old Primary.**
- Start at 13:54:33.5; ready in 7.9 s; `-Verify` OK 21/21.
- Boot: `Mode="Warm"`, levels 50 then **90** (13:54:44). `warm_role` 0, `/health/active` 503.
- **No failback:** vq04 stayed at 250, `warm_role` 1, with no new transition.
- ScadaBridge stayed on vq04 with no Bad sample and no DCL line for 110 s (13:54:33 → 13:56:45). It never started a
degraded streak, because its endpoint was at 250.
- The 60 s dwell could not be exercised live (see deviation 2).
**4c: ScadaBridge on the Standby over a healthy session (the case only #138 handles).**
- The connection's subscription display name was changed and site artifacts were deployed (13:56:48.087). This is an
in-place reconfigure, which re-arms the connection on its **primary**, vq03, now the Standby at 90.
- 13:56:52.401: `Reconnected successfully on "Primary"`.
- 13:56:52.413: **`"Primary" endpoint reports ServiceLevel 90 (Degraded band, < 200) — will check the other endpoint now (connect-time preference)`**.
- 13:56:52.436: **`Failing over from "Primary" to "Backup" (ServiceLevel 90 Degraded for 0s; "Backup" reports 250)`** → `Entering Reconnecting state (ServiceLevel failover)`.
- 13:56:52.475: `Reconnected successfully on "Backup"`. First Good at 13:56:52.504.
- **The move took 0.1 s from the connect to the Standby**, with no disconnect involved. It took 4.4 s from the
reconfigure, most of which is the reconnect to vq03.
The site health report then showed `WarmCanary: Connected to backup`, 2 Good / 0 Bad. `MxGateway` stayed Connected
with 2196/2196 Good throughout, and 10 instances are deployed (9 + `WarmCanary`).
## End state (13:58Z)
| | vq03 | vq04 |
|---|---|---|
| Build | `76e73e7d` | `76e73e7d` |
| Mode | Warm | Warm |
| Role / ServiceLevel | Standby **90** | **Primary 250** |
| Canary | `BadOutOfService` | Good, about 1/s |
| LocalDb sync | connected, backlog 0 | connected, backlog 0 |
| `-Verify` | OK 21/21 | OK 21/21 |
- **ScadaBridge `WarmCanary`:** on the backup endpoint (vq04), Good. It is kept as the Warm monitor.
- **HistorianGateway `/metrics` (vq03 `:5220`):** only `Probe`, plus the counters from before the window
(`BrowseTagNames` 1, `ReadRaw` 1, `ReadAggregate` 1 ok / 2 error). No write or tag-creation RPC.
## Rollback (back to Unset)
On **each** node:
1. `& "E:\psrem-stage\otopcua-76e73e7d\warm-env.ps1" -Action Unset`. This removes exactly the two entries.
Expected env digests: vq03 49 / `5750bbeb4734`, vq04 48 / `ef146b2681b4`.
2. Stop **both** nodes (`pairstop.ps1`, Standby first).
3. Start both, Primary-to-be first (`start.ps1` waits for ready and runs `-Verify`).
4. Check 250/240, Mode `None` in the `redundancy` CLI, Good canary on both, and the boot banner `Mode="Unset"`.
The `.reg` exports are a fallback only. Mode is startup-static, so never run a mixed pair (one Warm, one Unset).
The ScadaBridge `WarmCanary` connection works under Unset too (250/240 never starts a streak). Remove it only if the
monitor is no longer wanted: `instance delete --id 16`, `template delete --id 27`, `data-connection delete --id 2`, then
`deploy artifacts --site-id 2`.
## Deviations
1. **Write refusal on the Standby.** It could not be shown as `BadInvalidState` on QA. Both QA tags are read-only, so
every Write is refused `BadNotWritable` before the Warm check, on either node. Part 9 method refusal could not be
shown either: QA has no alarm condition.
- What QA shows instead: a Write on the Standby changes nothing, and no tag is writable on either node.
- The `BadInvalidState` path itself is proven by T13 L10 (`f826af8e`).
- To prove it on QA would need a writable tag, which means a config deploy with a writable driver. Out of scope here.
2. **60 s dwell.** No live sequence produced two ServiceLevel switches inside 60 s. An in-place reconfigure resets the
dwell by design, and a role flip with a healthy session needs the Primary to leave, which drops the session. During
4b ScadaBridge never left vq04 (no flap). The dwell is covered by ScadaBridge's actor tests.
3. **"ServiceLevel poll active" evidence.** At Information level a Healthy sample logs nothing (by design), so step 1
has no poll line. The poll is evidenced by the 50/90/0 `Degraded band` Warnings in steps 3, 4a and 4c, each logged
within 15–35 ms of the connect or demotion.
4. **Primary placement.** The Primary moved from vq03 to vq04 in 4a, and Warm does not fail back. QA now runs with vq04
as Primary. This is expected; it is not restored.