lmxopcua/docs/plans/phase-6-3-redundancy-interop-plan.md

Phase 6.3 Redundancy — Client Interop Matrix and Cutover Validation Plan

Revised for the current redundancy model (lmxopcua#716, source audit; not a live client pass). The original Phase 6.3 text of this plan described the v1 model (ClusterNode.RedundancyRole, PeerHttpProbeLoop / PeerUaProbeLoop, RecoveryStateManager, ApplyLeaseRegistry, the 255/230/200/180/100/80 service-level bands). Those types and bands no longer exist. Primary/secondary is now derived from Akka cluster membership (oldest Up driver member of each application Cluster's own 2-node mesh) and ServiceLevel comes from ServiceLevelCalculator. This document is the executable checklist for GA gate G9 (see docs/plans/v2-ga-lab-gates-plan.md, tracked by #615); it still requires two live nodes and a real third-party client, and no lab-client result is claimed here.

Source of truth: docs/Redundancy.md (election, tiers, downing, manual failover, ServerArray), src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Redundancy/ServiceLevelCalculator.cs and the tests named below. docs/v2/redundancy-interop-playbook.md and scripts/compliance/phase-6-3-compliance.ps1 are historical (they still name the deleted types and old bands) and must not be used as pass criteria.

Standard variables used by this plan

Identifiers verified against the OPC UA SDK constants shipped with the pinned OPCFoundation.NetStandard.Opc.Ua.Core (VariableIds):

Variable	NodeId	Notes
Server.ServiceLevel	i=2267	Byte; published by SdkServiceLevelPublisher.
Server.ServerArray	i=2254	String[]; the peer-discovery surface OtOpcUa actually populates (self + OpcUa:PeerApplicationUris).
Server.ServerRedundancy.RedundancySupport	i=3709	Nothing in src/ writes this; it reads the SDK default. See A4.
Server.ServerRedundancy.ServerUriArray	i=11314	The standard non-transparent redundancy variable. Not populated (the SDK's default ServerRedundancy object type has no such child).
Server.ServerCapabilities.LocaleIdArray	i=2271	Not a redundancy variable. Earlier revisions of this plan read i=2271 as ServerUriArray; that was wrong.

Current service-level bands

From docs/Redundancy.md (ServiceLevelCalculator, the live publish path; recomputed on every ~5 s HealthTick and on each RedundancyStateChanged):

Byte	Meaning
250	Healthy and the driver Primary (oldest Up driver member of the cluster's mesh).
240	Healthy follower (DB reachable, peer OPC UA probe ok, not stale, not Primary).
200	Stale data, ConfigDb reachable.
100	ConfigDb unreachable and data stale (DB-backed nodes only; a driver-only node feeds DbReachable=true and stays 240/250).
0	Member not Up/Joining, node has no driver role, or this node's OPC UA endpoint failed to start (EndpointListening=false).

There is no recovery-dwell band, no mid-apply band and no "isolated primary" band any more. A Primary whose peer is stopped stays 250: OpcUaProbeOk is a peer probing this node, absent/stale results default to true, and a departed peer never demotes it.

What is already tested (no live cluster needed)

Ordinary dotnet test coverage of the pieces this runbook relies on:

Area	Test class(es)	What it asserts
ServiceLevel tiers	ServiceLevelCalculatorTests (tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests, tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests)	Truth table above, including endpoint-failed = 0
Primary election	RedundancyPrimaryElectionTests (ControlPlane.Tests)	Oldest Up driver member wins, even when it holds the higher address
Manual failover	ManualFailoverServiceTests (ControlPlane.Tests)	Leave of the oldest driver; parity with the election
Endpoint-failure step-down	OpcUaEndpointStepDownActorTests, OpcUaPublishActorEndpointStateTests (Runtime.Tests)	Failed Primary hands over only when the peer is listening; ServiceLevel 0
Downing strategy	SplitBrainResolverActivationTests (Cluster.Tests)	auto-down installed through the production bootstrap
ServerArray peer URIs	DualEndpointTests (OpcUaServer.IntegrationTests)	Real OPCFoundation client reads both URIs from i=2254 on loopback
Topology validity	DraftValidatorTests (Configuration.Tests)	NodeCount/RedundancyMode pairs

Run with:

dotnet test ZB.MOM.WW.OtOpcUa.slnx --filter "FullyQualifiedName~Redundancy|FullyQualifiedName~ServiceLevelCalculator|FullyQualifiedName~ManualFailover|FullyQualifiedName~DualEndpoint"

What cannot be automated — requires two live instances

• UaExpert, Kepware KEPServerEX and AVEVA OI Gateway are closed-source Windows GUI binaries with no headless interface for the subscribe/browse flows.
• The AVEVA MXAccess failover leg involves the Galaxy runtime's own client-redundancy policy and COM-layer session model, both outside this repo.
• Even the scriptable subset (our otopcua-cli as the client) needs two distinct listening endpoints, i.e. two live processes, which is out of scope for dotnet test fixtures.

Test matrix

Prerequisites

1. Two OtOpcUa.Host processes forming one application Cluster's own 2-node Akka mesh (see docs/Redundancy.md "Per-cluster meshes (Phase 6)"): each node carries the roles driver and cluster-{ClusterId} (via Cluster:Roles; OTOPCUA_ROLES is only the fallback), and each node's Cluster:SeedNodes lists itself first, then its partner. The cluster row has NodeCount=2, RedundancyMode=Warm or Hot (DraftValidator rejects other pairs).
2. Each node has a distinct ApplicationUri (unique per ClusterNode), and each node's OpcUa:PeerApplicationUris lists its partner's ApplicationUri so Server.ServerArray returns both.
3. There is no operator-written role and no ServiceLevelBase input: ClusterNode.ServiceLevelBase still exists in the schema but does not drive the runtime value. The Primary is whichever node is the oldest Up driver member, so start Node A first and let it come Up before starting Node B if Node A is meant to be Primary.
4. At least one DriverInstance with a reachable simulator or PLC (e.g. the Modbus sim on the fixture host).
5. Client machine with UaExpert >= 1.7. Optional: Kepware KEPServerEX 6.x QuickClient, AVEVA OI Gateway 2020R2+.

Block A — OPC UA protocol signals

ID	Scenario	Procedure	Pass criterion	Automatable?
A1	ServiceLevel on Primary	Read Server/ServerStatus/ServiceLevel (i=2267) on the Primary.	250	Yes (Client CLI)
A2	ServiceLevel on follower	Read the same node on the other node.	240	Yes
A3	Peer stop does not demote the Primary	Stop the follower. Watch ServiceLevel on the Primary for >= 40 s.	Stays 250 (the benefit-of-the-doubt rule). A change here is a defect or an unrelated health input.	Yes
A4	RedundancySupport	Read i=3709 on either node (otopcua-cli redundancy).	Known unmet: no code sets it, so it reports the SDK default, not the cluster's Warm/Hot. Record the observed value; do not mark PASS.	Yes
A5	ServerArray peer discovery	Read i=2254 on either node.	String[] contains both ApplicationUri values (self and peer). This is the discovery surface in use today.	Yes
A5b	ServerUriArray conformance	Read i=11314 on either node.	Known unmet standard-metadata gate. The node is absent, so the read fails or is empty; the client CLI tolerates this. ServerArray (A5) is not a substitute and must not turn A5b into a PASS.	Yes
A6	Endpoint-failure step-down	Occupy port 4840 on the Primary host before it starts (fused admin+driver node), or follow lmxopcua#761.	Failed node publishes 0; the peer becomes Primary and publishes 250 (see docs/Redundancy.md "Endpoint start failure: Primary step-down").	No
A7	Client CLI reads ServiceLevel	dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://<node-a>:4840 -n "i=2267"	Prints the expected byte.	Yes
A8	otopcua-cli failover reconnect	... -- connect -u opc.tcp://<node-a>:4840 -F opc.tcp://<node-b>:4840, then stop Node A.	CLI reconnects to Node B within the session keep-alive timeout.	Yes

Block B — Third-party client failover

ID	Scenario	Procedure	Pass criterion
B1	UaExpert picks Primary by ServiceLevel	Configure a Redundancy Group in UaExpert with both endpoint URLs.	Client connects to the 250 node.
B2	UaExpert cuts over on Primary kill	Stop the Primary's service (supervised restart is expected, see below).	Client reconnects to the surviving node within UaExpert's reconnect timeout; data-change monitored items resume. The survivor becomes oldest Up after Akka downs the dead node (auto-down, default auto-down-unreachable-after = 15s), then publishes 250. Expect the cutover to take at least that long plus the 5 s HealthTick.
B3	Returning node settles as follower	Restart the stopped node.	It rejoins as the youngest member and settles at 240; it does not take Primary back (oldest-Up election). Whether the client switches is client-policy-dependent; both outcomes are accepted.
B4	Kepware QuickClient failover	Repeat B1–B3 with Kepware.	Same criteria.
B5	AVEVA OI Gateway	Configure the client object against both endpoints; stop the Primary.	Data quality recovers within ReconnectInterval (default 20 s); no permanent data-loss alert.

Block C — Galaxy MXAccess failover

Requires a running Galaxy and $MxAccessClient object (AVEVA System Platform installed, Galaxy deployed).

ID	Scenario	Procedure	Pass criterion
C1	Galaxy binds to Primary on first connect	Start a Galaxy $MxAccessClient with both node URLs.	QUALITY = Good; values stream from the Primary.
C2	Galaxy redirects on Primary drop	Stop the Primary.	QUALITY briefly Uncertain, then Good, streaming from the survivor within ReconnectInterval (default 20 s).
C3	Galaxy tolerates a deployment	Trigger a deployment (AdminUI Deployments).	Galaxy stays bound; no quality interruption. (There is no mid-apply ServiceLevel dip in the current model.)

A negative result on C1–C3 does not necessarily indicate an OtOpcUa defect; confirm Blocks A/B first.

Step-by-step cutover-validation runbook

Minimum procedure for the v2 GA exit criterion "Non-transparent redundancy cutover validated with at least one production client (Ignition 8.3 recommended — see decision #85)". Standard metadata gaps remain (A4, A5b); those rows stay unmet and the production-client cutover gate stays open until a real client pass is recorded.

Step 1 — Provision the cluster

Create/verify the ServerCluster (NodeCount=2, RedundancyMode=Warm) with two enabled ClusterNode rows with distinct ApplicationUri values, via the AdminUI. Set per-node Cluster:Roles, self-first Cluster:SeedNodes and OpcUa:PeerApplicationUris in each node's configuration.

Step 2 — Start both nodes, oldest first

Start Node A, wait until it is Up (/health/ready), then start Node B. Wait ~40 s for the first HealthTick, snapshot and peer-probe cycles (probe interval 10 s, connect timeout 2 s, freshness 30 s).

Step 3 — Verify baseline with the Client CLI

dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://<node-a-host>:4840 -n "i=2267"   # expect 250
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://<node-b-host>:4840 -n "i=2267"   # expect 240
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- redundancy -u opc.tcp://<node-a-host>:4840

Step 4 — Verify peer discovery and the unmet metadata

dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://<node-a-host>:4840 -n "i=2254"    # ServerArray: expect both URIs
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://<node-a-host>:4840 -n "i=11314"   # ServerUriArray: expect absent/empty (unmet gate A5b)

Record ServerArray as A5 PASS/FAIL and ServerUriArray separately as A5b; an absent ServerUriArray is recorded as unmet, never as a pass via ServerArray.

Step 5 — Primary kill and failover (B2)

1. Connect the third-party client's redundancy group to both endpoints; subscribe to a variable.
2. Stop the Primary's service (sc stop OtOpcUaHost on a Windows node; production service recovery restarts it, so stop the supervisor action or accept the restart).
3. Observe the survivor's ServiceLevel go 240 -> 250 once Akka downs the dead node (about 15 s plus up to a 5 s tick), and the client resume data-change events on it.
4. Record time from stop to reconnect and any data gap.

Planned alternative: use the AdminUI Trigger failover button on /clusters/{id}/redundancy (graceful Leave, Administrator only); the survivor becomes oldest and advertises 250 without the downing delay.

Step 6 — Recovery (B3)

1. Restart the stopped node.
2. Observe it rejoin as the youngest member and settle at 240 while the survivor stays 250. There is no recovery dwell and no automatic return of the Primary role.
3. Record whether the client switches back.

Step 7 — Deployment under load (C3)

Trigger a trivial deployment from the AdminUI while watching both nodes' ServiceLevel; expect no change.

Step 8 — Record results

Run date: YYYY-MM-DD
Release SHA: <git sha>
Cluster: <cluster-id>  Node A: <host>  Node B: <host>

A1: [PASS/FAIL]  evidence: <CLI output>
A2: [PASS/FAIL]
A3: [PASS/FAIL]
A4: [UNMET - observed value: <value>]
A5: [PASS/FAIL]  ServerArray contents: <..>
A5b: [UNMET - ServerUriArray absent]   (must not be PASS while the node is absent)
A6: [PASS/FAIL/SKIP]
A7: [PASS/FAIL]
A8: [PASS/FAIL]
B1..B5: [PASS/FAIL/SKIP]  reconnect time: <N>s
C1..C3: [PASS/FAIL/SKIP - Galaxy not available]

One pass of every non-SKIP row, with A4/A5b either genuinely passing after the metadata work lands or being explicitly recorded as unmet, is required before G9 can be called satisfied.

Known limitations

A4 / A5b — standard redundancy metadata not exposed

OtOpcUa does not set RedundancySupport and does not expose Server.ServerRedundancy.ServerUriArray (i=11314): the SDK's default ServerRedundancy object is the base type with no such child. Peer discovery works through Server.ServerArray (i=2254) only. Clients that require the standard NonTransparentRedundancyType metadata will not recognise the pair. The metadata gap is tracked separately from this runbook.

Failover timing is bounded by downing, not by probes

A crashed Primary is replaced only after Akka marks it unreachable and auto-down removes it (15 s by default), plus the 5 s ServiceLevel tick. A genuine partition leaves both sides active (accepted availability-over-partition-safety trade; see docs/Redundancy.md "Split-brain / downing").

The returning node does not reclaim Primary

Election is oldest-Up, so a restarted node is the youngest and stays a follower (240) until the other node leaves the mesh.

Dependency on existing tests

The runbook validates the end-to-end wire path; the math and edge cases are locked by the tests enumerated above. A failing step that contradicts a passing unit test indicates a deployment configuration error (roles, seed order, PeerApplicationUris) or an SDK version mismatch. Check the PeerOpcUaProbeActor / OpcUaPublishActor logs first.