lmxopcua/docs/deploy/2026-11-13-qa-warm-717.md

QA vq03/vq04: OpcUa:Redundancy:Mode=Warm switch-over and #138 live gate (2026-11-13)

Status: EXECUTED 2026-11-13, 13:41–13:58Z. QA is on Warm.

• #907 (tier flip) and ScadaBridge#138 (ServiceLevel-aware failover) were gated together on the QA pair.
• The owner approved the change, with WinRM for the on-box config, stop/start and verify steps only.
• End state: both nodes run Mode=Warm on build 76e73e7d. vq04 is Primary at 250 and vq03 is Standby at 90. The Primary moved during the gate, and Warm does not fail back.
• ScadaBridge's WarmCanary connection stays in place as a monitor.

Nothing was rebuilt: both nodes keep master 76e73e7d from 2026-11-13-vd03-qa-to-76e73e7d.md. The only change is two service-env entries per node.

Not done or touched:

• no configuration deploy, no Galaxy or historian writes, no tag creation;
• no other service on the boxes (mxgw, HistorianGateway, Overview, collector; ScadaBridge only through its management API);
• DEV vd03 stays on Mode Unset.

Change

Node	Entries added to the OtOpcUaHost service env	Env after (count, ordered / sorted digest)
vq03	OpcUa__Redundancy__Mode=Warm (68b419d808d1), OpcUa__PeerApplicationUris__0=urn:wonder-app-vq04:OtOpcUa (f15799748844)	49 → 51, 734bf356306e / b5c93e4fce6b
vq04	OpcUa__Redundancy__Mode=Warm (68b419d808d1), OpcUa__PeerApplicationUris__0=urn:wonder-app-vq03:OtOpcUa (7fe38ae93fd6)	48 → 50, 55054c72a487 / f0a64b1a385f

• Digests are SHA-256 prefixes of name=value. Values were checked against locally computed hashes and never printed. The existing OpcUa__ApplicationUri entries hash to urn:wonder-app-vq03:OtOpcUa (e1486d21fdbc) and urn:wonder-app-vq04:OtOpcUa (ee29df45d241), so each node's peer is the other node's effective URI.
• PromoteWhilePrimaryUnreachable is not set, so it keeps its default (true).
• Mechanism: the same registry Environment MultiString edit that the #901 HostServiceEnv step makes. Other entries and their order are kept. warm-env.ps1 -Action Set|Unset in E:\psrem-stage\otopcua-76e73e7d\ is idempotent. It was tested first as a no-op Unset on both nodes ("env unchanged").
• Install-ScriptWorkerIsolation.ps1 -Verify ran after every start: OK 21/21 on both nodes. The isolation record does not own these two names, so -Reapply neither needs them nor touches them.
• appsettings*.json were not changed (0 Redundancy/PeerApplicationUris hits before and after).
• ConfigDb ServerCluster.RedundancyMode is already Warm (1) (#717 deploy note). The deploy-time cross-check therefore agrees: there is no mode_mismatch series and no mismatch log line.
• Backups: reg export of the service key before the edit, in E:\psrem-stage\otopcua-76e73e7d\backup\OtOpcUaHost-pre-warm-20261113T13474{7,9}Z.reg (vq03, vq04).

On-box scripts (stage E:\psrem-stage\otopcua-76e73e7d\):

• warm-pre.ps1: preflight, env names and digests, and the reg export;
• warm-env.ps1: the env edit;
• pairstop.ps1: stops the service, then waits until no OtOpcUa.Host / ScriptWorker process is left (the #908 guard);
• legstop.ps1 and start.ps1, from the 76e73e7d deploy.

Steps and timings (UTC)

Monitoring from the Mac:

• i=2267 subscriptions on both nodes (250 ms);
• a 0.5 s /health/ready + /health/active poll;
• ScadaBridge debug stream of the WarmCanary instance;
• ScadaBridge site and OtOpcUa logs, read over SMB.

1. ScadaBridge test connection (Mode Unset)

ScadaBridge connection WarmCanary (id 2, QA site site-a), set up through the management API:

• primary opc.tcp://wonder-app-vq03.zmr.zimmer.com:4840/OtOpcUa, backup …vq04…;
• security None, anonymous;
• serviceLevelFailover left at its defaults (enabled, 10 s, 60 s).

Template WarmCanary (id 27) with two attributes:

• CalcSeconds → ns=2;s=script-runner-canary/Device1/canary-calc-seconds;
• VtSecond → ns=3;s=EQ-ec3979550d90/canary-vt-second.

Instance WarmCanary (id 16) is bound to the connection and deployed. QA OtOpcUa allows anonymous (OpcUa:AllowAnonymous default), so no otua_ key was minted. Connected to the Primary at 13:42:16.5Z, and both values were Good and advancing at 1/s from 13:42:23Z.

2. Baseline under Unset: stop the Primary vq03

Event	Time	Δ from stop
Stop-Service issued (vq03, Primary 250)	13:44:13.008	0
vq04 → 250 (subscription)	13:44:13.581	0.57 s
vq03 Stopped	13:44:19.109	6.1 s
ScadaBridge: AdapterDisconnected → Reconnecting, Bad	13:44:23.575	10.6 s
Connect to vq03 failed ×2 (Could not connect), then Failing over from "Primary" to "Backup"	13:44:40.7 / 57.9 / 13:45:15.05
Reconnected successfully on "Backup", first Good	13:45:20.10 / .125	67.1 s (Bad 56.5 s)
vq03 start → ready, -Verify OK 21/21	13:46:00.5 → 13:46:09.0

After the restart, vq03 sat at 240 and vq04 at 250, with no failback. ScadaBridge stayed on vq04 with no Bad sample.

3. Warm switch-over (pair window)

Event	Time
Env edit on both nodes (services up; no effect until start)	13:48:0x–13:48:2x
Stop vq03 (Secondary): Stopped 6.11 s, no process left 6.40 s	13:48:47.822
Stop vq04 (Primary): Stopped 6.40 s, no process left 6.69 s	13:48:56.027
Start vq03: ready 10.8 s; boot Mode="Warm" … peers=[urn:wonder-app-vq04:OtOpcUa]; Warm redundancy promotion to "Active" (reason=agreed, ServiceLevel=250) at 13:49:20.298	13:49:09.300
Start vq04 (after vq03 was ready): ready 9.2 s; boot Mode="Warm"; levels 50 then 90	13:49:21.660

• Pair outage: 22.8 s. No node was ready from 13:48:57.2Z to 13:49:19.9Z (Mac poll).
• lmxopcua#908 did not occur. pairstop.ps1 waited for the processes to exit, and nothing was moved.

#138 during the window (unplanned bonus):

• ScadaBridge (Bad from 13:49:07.139) first reached the booting Standby vq04: Reconnected successfully on "Backup" at 13:49:30.841.
• At once it logged "Backup" endpoint reports ServiceLevel 50 (Degraded band) — will check the other endpoint now (connect-time preference).
• Then Failing over from "Backup" to "Primary" (ServiceLevel 50 Degraded for 4s; "Primary" reports 250) at 13:49:35.338.
• Reconnected on the Primary at 13:49:39.419; Good at 13:49:39.441.

One connect attempt to vq04 during its boot failed with Endpoint does not support the user identity type provided (13:49:25.4). That is transient: the endpoint was coming up.

Step 3 gate

Gate	Result
Primary 250, Standby in the Warm band	PASS. vq03 250; vq04 90 (after 50 during boot). redundancy CLI on both: Mode Warm, Server URIs self-first ([vq03, vq04] on vq03, [vq04, vq03] on vq04)
RedundancySupport = Warm	PASS (redundancy CLI, i=3709)
Standby data reads	PASS. On vq04, canary-calc-seconds and canary-vt-second read BadOutOfService, value null, ×2. On vq03 both are Good and advancing
Standby write refused BadInvalidState	Not exercisable on QA, see deviation 1. A typed anonymous Write of both canaries on vq04 returned BadNotWritable, and so did the same Write on the Primary. QA's only tags are the read-only Calculation tag and the VirtualTag, so the SDK's AccessLevel check answers before the Warm refusal can. The Warm refusal is proven by #717 T13 L10
LocalDb sync	PASS. localdb-replication: "connected; oplog backlog 0" on both; localdb_oplog_depth 0
Canary on the Primary	PASS. Good, about 1/s; script runner Ready, shards 0/1 up, quarantined 0
Metrics	vq03 warm_role 1, warm_transition{promote,agreed} 1. vq04 warm_role 0, service_level_change{50}/{90}. No mode_mismatch or snapshot_rejected series
Health	/health/ready 200 on both, akka 2 members / 0 unreachable; /health/active 200 vq03 / 503 vq04
Logs	Since the switch, vq03 has no ERR/FTL. vq04 has only the standby /health/active 503 / cluster-primary lines, plus Warnings already seen before the switch (singleton-proxy at boot, the DEV mesh-PSK note). Also one Client connects with an endpointUrl … which does not match Server hostnames Warning: ScadaBridge uses the FQDN, and it is cosmetic

4. #138 live gate under Warm

4a: stop the Warm Primary vq03.

Event	Time	Δ
Stop-Service issued (vq03, Primary 250)	13:52:56.661	0
vq03: Host stopping …(lmxopcua#906) → demotion to "Standby" (reason=membership, ServiceLevel=0)	13:52:56.675 / .689	0.03 s
vq04: promotion to "Active" (reason=agreed, ServiceLevel=250); subscription 250	13:52:56.842 / 13:52:57.314	0.18 s / 0.65 s
ScadaBridge: the last vq03 sample delivered BadOutOfService (stream: Bad)	13:52:57.510	0.85 s
DCL: "Primary" endpoint reports ServiceLevel 0 (Degraded band, < 200) — will check the other endpoint if it persists for 10s	13:52:57.435	0.77 s
vq03 Stopped (6.10 s); the next poll found the level unreadable: ServiceLevel is unknown — degraded watch cleared	13:53:03.432	6.8 s
AdapterDisconnected → Reconnecting	13:53:06.439	9.8 s
Connect to vq03 failed ×2, then Failing over from "Primary" to "Backup" (the connectivity trigger)	13:53:23.6 / 40.7 / 57.855
Reconnected successfully on "Backup" (vq04, 250); first Good	13:54:02.886 / 13:54:02.913	66.3 s

Cause recorded: the connectivity failover, not ServiceLevel. The leaving node reported 0 for about 6 s. That is shorter than the 10 s degraded window, so the streak was cleared when the read went unknown, and the switch then waited for the existing 3 × (connect timeout + 5 s) retry cycle. Time to Good matched the Unset baseline (66.3 s vs 67.1 s). This is filed as follow-up ScadaBridge#141.

4b: restart the old Primary.

• Start at 13:54:33.5; ready in 7.9 s; -Verify OK 21/21.
• Boot: Mode="Warm", levels 50 then 90 (13:54:44). warm_role 0, /health/active 503.
• No failback: vq04 stayed at 250, warm_role 1, with no new transition.
• ScadaBridge stayed on vq04 with no Bad sample and no DCL line for 110 s (13:54:33 → 13:56:45). It never started a degraded streak, because its endpoint was at 250.
• The 60 s dwell could not be exercised live (see deviation 2).

4c: ScadaBridge on the Standby over a healthy session (the case only #138 handles).

• The connection's subscription display name was changed and site artifacts were deployed (13:56:48.087). This is an in-place reconfigure, which re-arms the connection on its primary, vq03, now the Standby at 90.
• 13:56:52.401: Reconnected successfully on "Primary".
• 13:56:52.413: "Primary" endpoint reports ServiceLevel 90 (Degraded band, < 200) — will check the other endpoint now (connect-time preference).
• 13:56:52.436: Failing over from "Primary" to "Backup" (ServiceLevel 90 Degraded for 0s; "Backup" reports 250) → Entering Reconnecting state (ServiceLevel failover).
• 13:56:52.475: Reconnected successfully on "Backup". First Good at 13:56:52.504.
• The move took 0.1 s from the connect to the Standby, with no disconnect involved. It took 4.4 s from the reconfigure, most of which is the reconnect to vq03.

The site health report then showed WarmCanary: Connected to backup, 2 Good / 0 Bad. MxGateway stayed Connected with 2196/2196 Good throughout, and 10 instances are deployed (9 + WarmCanary).

End state (13:58Z)

	vq03	vq04
Build	76e73e7d	76e73e7d
Mode	Warm	Warm
Role / ServiceLevel	Standby 90	Primary 250
Canary	BadOutOfService	Good, about 1/s
LocalDb sync	connected, backlog 0	connected, backlog 0
-Verify	OK 21/21	OK 21/21

• ScadaBridge WarmCanary: on the backup endpoint (vq04), Good. It is kept as the Warm monitor.
• HistorianGateway /metrics (vq03 :5220): only Probe, plus the counters from before the window (BrowseTagNames 1, ReadRaw 1, ReadAggregate 1 ok / 2 error). No write or tag-creation RPC.

Rollback (back to Unset)

On each node:

1. & "E:\psrem-stage\otopcua-76e73e7d\warm-env.ps1" -Action Unset. This removes exactly the two entries. Expected env digests: vq03 49 / 5750bbeb4734, vq04 48 / ef146b2681b4.
2. Stop both nodes (pairstop.ps1, Standby first).
3. Start both, Primary-to-be first (start.ps1 waits for ready and runs -Verify).
4. Check 250/240, Mode None in the redundancy CLI, Good canary on both, and the boot banner Mode="Unset".

The .reg exports are a fallback only. Mode is startup-static, so never run a mixed pair (one Warm, one Unset).

The ScadaBridge WarmCanary connection works under Unset too (250/240 never starts a streak). Remove it only if the monitor is no longer wanted: instance delete --id 16, template delete --id 27, data-connection delete --id 2, then deploy artifacts --site-id 2.

Deviations

1. Write refusal on the Standby. It could not be shown as BadInvalidState on QA. Both QA tags are read-only, so every Write is refused BadNotWritable before the Warm check, on either node. Part 9 method refusal could not be shown either: QA has no alarm condition.
   • What QA shows instead: a Write on the Standby changes nothing, and no tag is writable on either node.
   • The BadInvalidState path itself is proven by T13 L10 (f826af8e).
   • To prove it on QA would need a writable tag, which means a config deploy with a writable driver. Out of scope here.
2. 60 s dwell. No live sequence produced two ServiceLevel switches inside 60 s. An in-place reconfigure resets the dwell by design, and a role flip with a healthy session needs the Primary to leave, which drops the session. During 4b ScadaBridge never left vq04 (no flap). The dwell is covered by ScadaBridge's actor tests.
3. "ServiceLevel poll active" evidence. At Information level a Healthy sample logs nothing (by design), so step 1 has no poll line. The poll is evidenced by the 50/90/0 Degraded band Warnings in steps 3, 4a and 4c, each logged within 15–35 ms of the connect or demotion.
4. Primary placement. The Primary moved from vq03 to vq04 in 4a, and Warm does not fail back. QA now runs with vq04 as Primary. This is expected; it is not restored.