Joseph Doherty
239 lines
16 KiB
Markdown
239 lines
16 KiB
Markdown
# Phase 6.3 Redundancy — Client Interop Matrix and Cutover Validation Plan
|
||
|
||
> **Revised for the current redundancy model (lmxopcua#716, source audit; not a live client pass).**
|
||
> The original Phase 6.3 text of this plan described the v1 model (`ClusterNode.RedundancyRole`,
|
||
> `PeerHttpProbeLoop` / `PeerUaProbeLoop`, `RecoveryStateManager`, `ApplyLeaseRegistry`, the
|
||
> 255/230/200/180/100/80 service-level bands). Those types and bands no longer exist. Primary/secondary is
|
||
> now derived from Akka cluster membership (oldest Up `driver` member of each application `Cluster`'s own
|
||
> 2-node mesh) and ServiceLevel comes from `ServiceLevelCalculator`. This document is the executable
|
||
> checklist for GA gate **G9** (see `docs/plans/v2-ga-lab-gates-plan.md`, tracked by #615); it still
|
||
> requires two live nodes and a real third-party client, and **no lab-client result is claimed here**.
|
||
>
|
||
> **Source of truth**: `docs/Redundancy.md` (election, tiers, downing, manual failover, `ServerArray`),
|
||
> `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Redundancy/ServiceLevelCalculator.cs` and the tests named below.
|
||
> `docs/v2/redundancy-interop-playbook.md` and `scripts/compliance/phase-6-3-compliance.ps1` are
|
||
> **historical** (they still name the deleted types and old bands) and must not be used as pass criteria.
|
||
|
||
## Standard variables used by this plan
|
||
|
||
Identifiers verified against the OPC UA SDK constants shipped with the pinned
|
||
`OPCFoundation.NetStandard.Opc.Ua.Core` (`VariableIds`):
|
||
|
||
| Variable | NodeId | Notes |
|
||
|---|---|---|
|
||
| `Server.ServiceLevel` | `i=2267` | Byte; published by `SdkServiceLevelPublisher`. |
|
||
| `Server.ServerArray` | `i=2254` | String[]; the **peer-discovery surface OtOpcUa actually populates** (self + `OpcUa:PeerApplicationUris`). |
|
||
| `Server.ServerRedundancy.RedundancySupport` | `i=3709` | Nothing in `src/` writes this; it reads the SDK default. See A4. |
|
||
| `Server.ServerRedundancy.ServerUriArray` | `i=11314` | The standard non-transparent redundancy variable. **Not populated** (the SDK's default `ServerRedundancy` object type has no such child). |
|
||
| `Server.ServerCapabilities.LocaleIdArray` | `i=2271` | **Not** a redundancy variable. Earlier revisions of this plan read `i=2271` as `ServerUriArray`; that was wrong. |
|
||
|
||
## Current service-level bands
|
||
|
||
From `docs/Redundancy.md` (`ServiceLevelCalculator`, the live publish path; recomputed on every ~5 s
|
||
`HealthTick` and on each `RedundancyStateChanged`):
|
||
|
||
| Byte | Meaning |
|
||
|---|---|
|
||
| 250 | Healthy and the driver Primary (oldest Up `driver` member of the cluster's mesh). |
|
||
| 240 | Healthy follower (DB reachable, peer OPC UA probe ok, not stale, not Primary). |
|
||
| 200 | Stale data, ConfigDb reachable. |
|
||
| 100 | ConfigDb unreachable and data stale (DB-backed nodes only; a driver-only node feeds `DbReachable=true` and stays 240/250). |
|
||
| 0 | Member not `Up`/`Joining`, node has no `driver` role, or this node's OPC UA endpoint failed to start (`EndpointListening=false`). |
|
||
|
||
There is no recovery-dwell band, no mid-apply band and no "isolated primary" band any more. A Primary whose
|
||
peer is stopped **stays 250**: `OpcUaProbeOk` is a peer probing *this* node, absent/stale results default to
|
||
`true`, and a departed peer never demotes it.
|
||
|
||
## What is already tested (no live cluster needed)
|
||
|
||
Ordinary `dotnet test` coverage of the pieces this runbook relies on:
|
||
|
||
| Area | Test class(es) | What it asserts |
|
||
|---|---|---|
|
||
| ServiceLevel tiers | `ServiceLevelCalculatorTests` (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests`, `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests`) | Truth table above, including endpoint-failed = 0 |
|
||
| Primary election | `RedundancyPrimaryElectionTests` (ControlPlane.Tests) | Oldest Up `driver` member wins, even when it holds the higher address |
|
||
| Manual failover | `ManualFailoverServiceTests` (ControlPlane.Tests) | Leave of the oldest driver; parity with the election |
|
||
| Endpoint-failure step-down | `OpcUaEndpointStepDownActorTests`, `OpcUaPublishActorEndpointStateTests` (Runtime.Tests) | Failed Primary hands over only when the peer is listening; ServiceLevel 0 |
|
||
| Downing strategy | `SplitBrainResolverActivationTests` (Cluster.Tests) | `auto-down` installed through the production bootstrap |
|
||
| `ServerArray` peer URIs | `DualEndpointTests` (`OpcUaServer.IntegrationTests`) | Real OPCFoundation client reads both URIs from `i=2254` on loopback |
|
||
| Topology validity | `DraftValidatorTests` (Configuration.Tests) | `NodeCount`/`RedundancyMode` pairs |
|
||
|
||
Run with:
|
||
|
||
```bash
|
||
dotnet test ZB.MOM.WW.OtOpcUa.slnx --filter "FullyQualifiedName~Redundancy|FullyQualifiedName~ServiceLevelCalculator|FullyQualifiedName~ManualFailover|FullyQualifiedName~DualEndpoint"
|
||
```
|
||
|
||
## What cannot be automated — requires two live instances
|
||
|
||
- UaExpert, Kepware KEPServerEX and AVEVA OI Gateway are closed-source Windows GUI binaries with no
|
||
headless interface for the subscribe/browse flows.
|
||
- The AVEVA MXAccess failover leg involves the Galaxy runtime's own client-redundancy policy and COM-layer
|
||
session model, both outside this repo.
|
||
- Even the scriptable subset (our `otopcua-cli` as the client) needs two distinct listening endpoints, i.e.
|
||
two live processes, which is out of scope for `dotnet test` fixtures.
|
||
|
||
## Test matrix
|
||
|
||
### Prerequisites
|
||
|
||
1. Two `OtOpcUa.Host` processes forming **one application `Cluster`'s own 2-node Akka mesh** (see
|
||
`docs/Redundancy.md` "Per-cluster meshes (Phase 6)"): each node carries the roles `driver` and
|
||
`cluster-{ClusterId}` (via `Cluster:Roles`; `OTOPCUA_ROLES` is only the fallback), and each node's
|
||
`Cluster:SeedNodes` lists **itself first**, then its partner. The cluster row has `NodeCount=2`,
|
||
`RedundancyMode=Warm` or `Hot` (`DraftValidator` rejects other pairs).
|
||
2. Each node has a distinct `ApplicationUri` (unique per `ClusterNode`), and each node's
|
||
`OpcUa:PeerApplicationUris` lists its partner's `ApplicationUri` so `Server.ServerArray` returns both.
|
||
3. There is no operator-written role and no `ServiceLevelBase` input: `ClusterNode.ServiceLevelBase` still
|
||
exists in the schema but does not drive the runtime value. The Primary is whichever node is the oldest Up
|
||
`driver` member, so **start Node A first and let it come Up before starting Node B** if Node A is meant to
|
||
be Primary.
|
||
4. At least one `DriverInstance` with a reachable simulator or PLC (e.g. the Modbus sim on the fixture host).
|
||
5. Client machine with UaExpert >= 1.7. Optional: Kepware KEPServerEX 6.x QuickClient, AVEVA OI Gateway 2020R2+.
|
||
|
||
### Block A — OPC UA protocol signals
|
||
|
||
| ID | Scenario | Procedure | Pass criterion | Automatable? |
|
||
|----|----------|-----------|----------------|--------------|
|
||
| A1 | ServiceLevel on Primary | Read `Server/ServerStatus/ServiceLevel` (`i=2267`) on the Primary. | 250 | Yes (Client CLI) |
|
||
| A2 | ServiceLevel on follower | Read the same node on the other node. | 240 | Yes |
|
||
| A3 | Peer stop does not demote the Primary | Stop the follower. Watch `ServiceLevel` on the Primary for >= 40 s. | Stays 250 (the benefit-of-the-doubt rule). A change here is a defect or an unrelated health input. | Yes |
|
||
| A4 | RedundancySupport | Read `i=3709` on either node (`otopcua-cli redundancy`). | **Known unmet:** no code sets it, so it reports the SDK default, not the cluster's `Warm`/`Hot`. Record the observed value; do not mark PASS. | Yes |
|
||
| A5 | `ServerArray` peer discovery | Read `i=2254` on either node. | String[] contains both `ApplicationUri` values (self and peer). This is the discovery surface in use today. | Yes |
|
||
| A5b | `ServerUriArray` conformance | Read `i=11314` on either node. | **Known unmet standard-metadata gate.** The node is absent, so the read fails or is empty; the client CLI tolerates this. `ServerArray` (A5) is **not** a substitute and must not turn A5b into a PASS. | Yes |
|
||
| A6 | Endpoint-failure step-down | Occupy port 4840 on the Primary host before it starts (fused admin+driver node), or follow lmxopcua#761. | Failed node publishes 0; the peer becomes Primary and publishes 250 (see `docs/Redundancy.md` "Endpoint start failure: Primary step-down"). | No |
|
||
| A7 | Client CLI reads ServiceLevel | `dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://<node-a>:4840 -n "i=2267"` | Prints the expected byte. | Yes |
|
||
| A8 | otopcua-cli failover reconnect | `... -- connect -u opc.tcp://<node-a>:4840 -F opc.tcp://<node-b>:4840`, then stop Node A. | CLI reconnects to Node B within the session keep-alive timeout. | Yes |
|
||
|
||
### Block B — Third-party client failover
|
||
|
||
| ID | Scenario | Procedure | Pass criterion |
|
||
|----|----------|-----------|----------------|
|
||
| B1 | UaExpert picks Primary by ServiceLevel | Configure a Redundancy Group in UaExpert with both endpoint URLs. | Client connects to the 250 node. |
|
||
| B2 | UaExpert cuts over on Primary kill | Stop the Primary's service (supervised restart is expected, see below). | Client reconnects to the surviving node within UaExpert's reconnect timeout; data-change monitored items resume. The survivor becomes oldest Up after Akka downs the dead node (`auto-down`, default `auto-down-unreachable-after = 15s`), then publishes 250. Expect the cutover to take at least that long plus the 5 s `HealthTick`. |
|
||
| B3 | Returning node settles as follower | Restart the stopped node. | It rejoins as the youngest member and settles at 240; **it does not take Primary back** (oldest-Up election). Whether the client switches is client-policy-dependent; both outcomes are accepted. |
|
||
| B4 | Kepware QuickClient failover | Repeat B1–B3 with Kepware. | Same criteria. |
|
||
| B5 | AVEVA OI Gateway | Configure the client object against both endpoints; stop the Primary. | Data quality recovers within `ReconnectInterval` (default 20 s); no permanent data-loss alert. |
|
||
|
||
### Block C — Galaxy MXAccess failover
|
||
|
||
Requires a running Galaxy and `$MxAccessClient` object (AVEVA System Platform installed, Galaxy deployed).
|
||
|
||
| ID | Scenario | Procedure | Pass criterion |
|
||
|----|----------|-----------|----------------|
|
||
| C1 | Galaxy binds to Primary on first connect | Start a Galaxy `$MxAccessClient` with both node URLs. | `QUALITY = Good`; values stream from the Primary. |
|
||
| C2 | Galaxy redirects on Primary drop | Stop the Primary. | `QUALITY` briefly `Uncertain`, then `Good`, streaming from the survivor within `ReconnectInterval` (default 20 s). |
|
||
| C3 | Galaxy tolerates a deployment | Trigger a deployment (AdminUI **Deployments**). | Galaxy stays bound; no quality interruption. (There is no mid-apply ServiceLevel dip in the current model.) |
|
||
|
||
A negative result on C1–C3 does not necessarily indicate an OtOpcUa defect; confirm Blocks A/B first.
|
||
|
||
## Step-by-step cutover-validation runbook
|
||
|
||
Minimum procedure for the v2 GA exit criterion "Non-transparent redundancy cutover validated with at least
|
||
one production client (Ignition 8.3 recommended — see decision #85)". **Standard metadata gaps remain
|
||
(A4, A5b); those rows stay unmet and the production-client cutover gate stays open until a real client
|
||
pass is recorded.**
|
||
|
||
### Step 1 — Provision the cluster
|
||
|
||
Create/verify the `ServerCluster` (`NodeCount=2`, `RedundancyMode=Warm`) with two enabled `ClusterNode` rows
|
||
with distinct `ApplicationUri` values, via the AdminUI. Set per-node `Cluster:Roles`, self-first
|
||
`Cluster:SeedNodes` and `OpcUa:PeerApplicationUris` in each node's configuration.
|
||
|
||
### Step 2 — Start both nodes, oldest first
|
||
|
||
Start Node A, wait until it is Up (`/health/ready`), then start Node B. Wait ~40 s for the first
|
||
`HealthTick`, snapshot and peer-probe cycles (probe interval 10 s, connect timeout 2 s, freshness 30 s).
|
||
|
||
### Step 3 — Verify baseline with the Client CLI
|
||
|
||
```bash
|
||
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://<node-a-host>:4840 -n "i=2267" # expect 250
|
||
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://<node-b-host>:4840 -n "i=2267" # expect 240
|
||
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- redundancy -u opc.tcp://<node-a-host>:4840
|
||
```
|
||
|
||
### Step 4 — Verify peer discovery and the unmet metadata
|
||
|
||
```bash
|
||
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://<node-a-host>:4840 -n "i=2254" # ServerArray: expect both URIs
|
||
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://<node-a-host>:4840 -n "i=11314" # ServerUriArray: expect absent/empty (unmet gate A5b)
|
||
```
|
||
|
||
Record `ServerArray` as A5 PASS/FAIL and `ServerUriArray` separately as A5b; an absent `ServerUriArray` is
|
||
recorded as **unmet**, never as a pass via `ServerArray`.
|
||
|
||
### Step 5 — Primary kill and failover (B2)
|
||
|
||
1. Connect the third-party client's redundancy group to both endpoints; subscribe to a variable.
|
||
2. Stop the Primary's service (`sc stop OtOpcUaHost` on a Windows node; production service recovery restarts
|
||
it, so stop the supervisor action or accept the restart).
|
||
3. Observe the survivor's `ServiceLevel` go 240 -> 250 once Akka downs the dead node (about 15 s plus up to a
|
||
5 s tick), and the client resume data-change events on it.
|
||
4. Record time from stop to reconnect and any data gap.
|
||
|
||
Planned alternative: use the AdminUI **Trigger failover** button on `/clusters/{id}/redundancy` (graceful
|
||
`Leave`, Administrator only); the survivor becomes oldest and advertises 250 without the downing delay.
|
||
|
||
### Step 6 — Recovery (B3)
|
||
|
||
1. Restart the stopped node.
|
||
2. Observe it rejoin as the youngest member and settle at 240 while the survivor stays 250. There is no
|
||
recovery dwell and no automatic return of the Primary role.
|
||
3. Record whether the client switches back.
|
||
|
||
### Step 7 — Deployment under load (C3)
|
||
|
||
Trigger a trivial deployment from the AdminUI while watching both nodes' `ServiceLevel`; expect no change.
|
||
|
||
### Step 8 — Record results
|
||
|
||
```
|
||
Run date: YYYY-MM-DD
|
||
Release SHA: <git sha>
|
||
Cluster: <cluster-id> Node A: <host> Node B: <host>
|
||
|
||
A1: [PASS/FAIL] evidence: <CLI output>
|
||
A2: [PASS/FAIL]
|
||
A3: [PASS/FAIL]
|
||
A4: [UNMET - observed value: <value>]
|
||
A5: [PASS/FAIL] ServerArray contents: <..>
|
||
A5b: [UNMET - ServerUriArray absent] (must not be PASS while the node is absent)
|
||
A6: [PASS/FAIL/SKIP]
|
||
A7: [PASS/FAIL]
|
||
A8: [PASS/FAIL]
|
||
B1..B5: [PASS/FAIL/SKIP] reconnect time: <N>s
|
||
C1..C3: [PASS/FAIL/SKIP - Galaxy not available]
|
||
```
|
||
|
||
One pass of every non-SKIP row, with A4/A5b either genuinely passing after the metadata work lands or being
|
||
explicitly recorded as unmet, is required before G9 can be called satisfied.
|
||
|
||
## Known limitations
|
||
|
||
### A4 / A5b — standard redundancy metadata not exposed
|
||
|
||
OtOpcUa does not set `RedundancySupport` and does not expose `Server.ServerRedundancy.ServerUriArray`
|
||
(`i=11314`): the SDK's default `ServerRedundancy` object is the base type with no such child. Peer discovery
|
||
works through `Server.ServerArray` (`i=2254`) only. Clients that require the standard
|
||
`NonTransparentRedundancyType` metadata will not recognise the pair. The metadata gap is tracked separately
|
||
from this runbook.
|
||
|
||
### Failover timing is bounded by downing, not by probes
|
||
|
||
A crashed Primary is replaced only after Akka marks it unreachable and `auto-down` removes it (15 s by
|
||
default), plus the 5 s ServiceLevel tick. A genuine partition leaves both sides active (accepted
|
||
availability-over-partition-safety trade; see `docs/Redundancy.md` "Split-brain / downing").
|
||
|
||
### The returning node does not reclaim Primary
|
||
|
||
Election is oldest-Up, so a restarted node is the youngest and stays a follower (240) until the other node
|
||
leaves the mesh.
|
||
|
||
## Dependency on existing tests
|
||
|
||
The runbook validates the end-to-end wire path; the math and edge cases are locked by the tests enumerated
|
||
above. A failing step that contradicts a passing unit test indicates a deployment configuration error
|
||
(roles, seed order, `PeerApplicationUris`) or an SDK version mismatch. Check the `PeerOpcUaProbeActor` /
|
||
`OpcUaPublishActor` logs first.
|