dohertj2/lmxopcua
1
0
Fork 0
Code Issues 32 Pull Requests Actions 1 Packages Projects Releases Wiki Activity

docs(redundancy): rewrite phase 6.3 interop runbook for the current model and correct standard NodeIds (lmxopcua#716)

Browse Source 
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 12:30:03 -05:00
parent 5594bc7bb9
commit e31888c9fd
1 changed files with 162 additions and 202 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plans/phase-6-3-redundancy-interop-plan.md
+162 -202
View File
@@ -1,278 +1,238 @@
# Phase 6.3 Redundancy — Client Interop Matrix and Cutover Validation Plan
> **Scope**: Phase 6.3 redundancy runtime core shipped (PRs #89-90, #98-99,
> #24-peerprobe, Stream C node wiring, Stream D lease wrap). What remains is
> Stream F (task #150): validating that third-party OPC UA clients honour
> our `ServiceLevel` / `ServerUriArray` / `RedundancySupport` signals and
> fail over correctly when the Primary drops. This document defines what is
> automatable as integration tests, what requires two live instances plus a
> real client, and a step-by-step cutover-validation runbook.
> **Revised for the current redundancy model (lmxopcua#716, source audit; not a live client pass).**
> The original Phase 6.3 text of this plan described the v1 model (`ClusterNode.RedundancyRole`,
> `PeerHttpProbeLoop` / `PeerUaProbeLoop`, `RecoveryStateManager`, `ApplyLeaseRegistry`, the
> 255/230/200/180/100/80 service-level bands). Those types and bands no longer exist. Primary/secondary is
> now derived from Akka cluster membership (oldest Up `driver` member of each application `Cluster`'s own
> 2-node mesh) and ServiceLevel comes from `ServiceLevelCalculator`. This document is the executable
> checklist for GA gate **G9** (see `docs/plans/v2-ga-lab-gates-plan.md`, tracked by #615); it still
> requires two live nodes and a real third-party client, and **no lab-client result is claimed here**.
>
> **Source of truth**: `docs/Redundancy.md`, `docs/v2/redundancy-interop-playbook.md`,
> `docs/v2/implementation/phase-6-3-redundancy-runtime.md`,
> `scripts/compliance/phase-6-3-compliance.ps1`.
> **Source of truth**: `docs/Redundancy.md` (election, tiers, downing, manual failover, `ServerArray`),
> `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Redundancy/ServiceLevelCalculator.cs` and the tests named below.
> `docs/v2/redundancy-interop-playbook.md` and `scripts/compliance/phase-6-3-compliance.ps1` are
> **historical** (they still name the deleted types and old bands) and must not be used as pass criteria.
## Standard variables used by this plan
Identifiers verified against the OPC UA SDK constants shipped with the pinned
`OPCFoundation.NetStandard.Opc.Ua.Core` (`VariableIds`):
| Variable | NodeId | Notes |
|---|---|---|
| `Server.ServiceLevel` | `i=2267` | Byte; published by `SdkServiceLevelPublisher`. |
| `Server.ServerArray` | `i=2254` | String[]; the **peer-discovery surface OtOpcUa actually populates** (self + `OpcUa:PeerApplicationUris`). |
| `Server.ServerRedundancy.RedundancySupport` | `i=3709` | Nothing in `src/` writes this; it reads the SDK default. See A4. |
| `Server.ServerRedundancy.ServerUriArray` | `i=11314` | The standard non-transparent redundancy variable. **Not populated** (the SDK's default `ServerRedundancy` object type has no such child). |
| `Server.ServerCapabilities.LocaleIdArray` | `i=2271` | **Not** a redundancy variable. Earlier revisions of this plan read `i=2271` as `ServerUriArray`; that was wrong. |
## Current service-level bands
From `docs/Redundancy.md` (`ServiceLevelCalculator`, the live publish path; recomputed on every ~5 s
`HealthTick` and on each `RedundancyStateChanged`):
| Byte | Meaning |
|---|---|
| 250 | Healthy and the driver Primary (oldest Up `driver` member of the cluster's mesh). |
| 240 | Healthy follower (DB reachable, peer OPC UA probe ok, not stale, not Primary). |
| 200 | Stale data, ConfigDb reachable. |
| 100 | ConfigDb unreachable and data stale (DB-backed nodes only; a driver-only node feeds `DbReachable=true` and stays 240/250). |
| 0 | Member not `Up`/`Joining`, node has no `driver` role, or this node's OPC UA endpoint failed to start (`EndpointListening=false`). |
There is no recovery-dwell band, no mid-apply band and no "isolated primary" band any more. A Primary whose
peer is stopped **stays 250**: `OpcUaProbeOk` is a peer probing *this* node, absent/stale results default to
`true`, and a departed peer never demotes it.
## What is already tested (no live cluster needed)
The following are covered by existing automated tests that run in ordinary
`dotnet test`:
Ordinary `dotnet test` coverage of the pieces this runbook relies on:
| Area | Test class(es) | What it asserts |
|---|---|---|
| `ServiceLevelCalculator` — 8-state matrix | `ServiceLevelCalculatorTests` | All 10 band values; role × self-health × peer-http × peer-ua × apply × recovery × topology combinations |
| `RecoveryStateManager` — dwell + witness | `RecoveryStateManagerTests` | 60 s dwell default; premature-exit rejection; witness-required gate |
| `ApplyLeaseRegistry` — lease lifecycle | `ApplyLeaseRegistryTests` | Disposal on success / exception / cancellation; watchdog force-close at 10 min |
| `ServerRedundancyNodeWriter` — OPC UA variable binding | `ServerRedundancyNodeWriterTests` | `ServiceLevel` byte push; `RedundancySupport` enum; `ServerUriArray` skip-log when node absent |
| `RedundancyStatePublisher` — orchestration | `RedundancyStatePublisherTests` | Edge-triggered `OnStateChanged`; idempotent dedup |
| `ClusterTopologyLoader` | `ClusterTopologyLoaderTests` | Two-node seed; one-node degenerate; duplicate-URI rejection |
| `DraftValidator.ValidateClusterTopology` | `DraftValidatorTests` (8 cases) | NodeCount/mode pairs; Enabled-count vs NodeCount; multiple-Primary rejection |
| ServiceLevel tiers | `ServiceLevelCalculatorTests` (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests`, `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests`) | Truth table above, including endpoint-failed = 0 |
| Primary election | `RedundancyPrimaryElectionTests` (ControlPlane.Tests) | Oldest Up `driver` member wins, even when it holds the higher address |
| Manual failover | `ManualFailoverServiceTests` (ControlPlane.Tests) | Leave of the oldest driver; parity with the election |
| Endpoint-failure step-down | `OpcUaEndpointStepDownActorTests`, `OpcUaPublishActorEndpointStateTests` (Runtime.Tests) | Failed Primary hands over only when the peer is listening; ServiceLevel 0 |
| Downing strategy | `SplitBrainResolverActivationTests` (Cluster.Tests) | `auto-down` installed through the production bootstrap |
| `ServerArray` peer URIs | `DualEndpointTests` (`OpcUaServer.IntegrationTests`) | Real OPCFoundation client reads both URIs from `i=2254` on loopback |
| Topology validity | `DraftValidatorTests` (Configuration.Tests) | `NodeCount`/`RedundancyMode` pairs |
Run with:
```powershell
dotnet test ZB.MOM.WW.OtOpcUa.slnx --filter "FullyQualifiedName~Redundancy"
```bash
dotnet test ZB.MOM.WW.OtOpcUa.slnx --filter "FullyQualifiedName~Redundancy|FullyQualifiedName~ServiceLevelCalculator|FullyQualifiedName~ManualFailover|FullyQualifiedName~DualEndpoint"
```
Compliance gate (every Phase 6.3 static check):
```powershell
pwsh ./scripts/compliance/phase-6-3-compliance.ps1
```
Pass criteria: exit 0; all `[PASS]` lines green; `[DEFERRED]` lines are
known-deferred surfaces, not failures.
## What cannot be automated — requires two live instances
The scenarios below require two running `OtOpcUa.Server` processes in the
same `ServerCluster`, a real SQL Server Config DB, and at least one driver
instance with a reachable endpoint (simulator or real PLC).
### Why it cannot be unit/integration-tested in-process
- UaExpert, Kepware KEPServerEX, and AVEVA OI Gateway are closed-source
Windows GUI binaries with no headless CLI interface for the
subscribe/browse flows.
- The AVEVA MXAccess failover leg (`IAlarmSource` reconnect, `$MxAccessClient`
quality transition) involves the Galaxy runtime's own client-redundancy
policy and the COM-layer session model — both live outside this repo.
- Even the automatable sub-set (our own `otopcua-cli` as the client) needs
two distinct listening TCP endpoints; that requires two live processes,
which is out of scope for `dotnet test` integration fixtures.
- UaExpert, Kepware KEPServerEX and AVEVA OI Gateway are closed-source Windows GUI binaries with no
headless interface for the subscribe/browse flows.
- The AVEVA MXAccess failover leg involves the Galaxy runtime's own client-redundancy policy and COM-layer
session model, both outside this repo.
- Even the scriptable subset (our `otopcua-cli` as the client) needs two distinct listening endpoints, i.e.
two live processes, which is out of scope for `dotnet test` fixtures.
## Test matrix
### Prerequisites
1. Two `OtOpcUa.Server` processes on separate Windows hosts (or separate
ports on the same host for dev) sharing one Config DB (`ServerCluster`
with `NodeCount=2`, `RedundancyMode=Warm` or `Hot`).
2. Each node registered in `ClusterNode`:
- Node A: `RedundancyRole=Primary`, `ServiceLevelBase=255`,
`ApplicationUri=urn:node-a:OtOpcUa`
- Node B: `RedundancyRole=Secondary`, `ServiceLevelBase=100`,
`ApplicationUri=urn:node-b:OtOpcUa`
3. `PeerHttpProbeLoop` and `PeerUaProbeLoop` HostedServices running on both
nodes (registered via `AddHostedService<PeerHttpProbeLoop>` +
`AddHostedService<PeerUaProbeLoop>` in `Program.cs`).
4. At least one `DriverInstance` in the cluster with a reachable PLC or
simulator (e.g. Modbus sim at `10.100.0.35:5020`).
5. Client machine with UaExpert >= 1.7 installed.
6. Optional second client: Kepware KEPServerEX 6.x QuickClient or AVEVA
OI Gateway 2020R2+.
1. Two `OtOpcUa.Host` processes forming **one application `Cluster`'s own 2-node Akka mesh** (see
`docs/Redundancy.md` "Per-cluster meshes (Phase 6)"): each node carries the roles `driver` and
`cluster-{ClusterId}` (via `Cluster:Roles`; `OTOPCUA_ROLES` is only the fallback), and each node's
`Cluster:SeedNodes` lists **itself first**, then its partner. The cluster row has `NodeCount=2`,
`RedundancyMode=Warm` or `Hot` (`DraftValidator` rejects other pairs).
2. Each node has a distinct `ApplicationUri` (unique per `ClusterNode`), and each node's
`OpcUa:PeerApplicationUris` lists its partner's `ApplicationUri` so `Server.ServerArray` returns both.
3. There is no operator-written role and no `ServiceLevelBase` input: `ClusterNode.ServiceLevelBase` still
exists in the schema but does not drive the runtime value. The Primary is whichever node is the oldest Up
`driver` member, so **start Node A first and let it come Up before starting Node B** if Node A is meant to
be Primary.
4. At least one `DriverInstance` with a reachable simulator or PLC (e.g. the Modbus sim on the fixture host).
5. Client machine with UaExpert >= 1.7. Optional: Kepware KEPServerEX 6.x QuickClient, AVEVA OI Gateway 2020R2+.
### Block A — OPC UA protocol signals (UaExpert, no failover yet)
### Block A — OPC UA protocol signals
| ID | Scenario | Procedure | Pass criterion | Automatable? |
|----|----------|-----------|----------------|--------------|
| A1 | ServiceLevel published on Primary | Connect UaExpert to Node A. Browse `Server/ServerStatus/ServiceLevel`. | Value = 255 (`AuthoritativePrimary`) | No — requires UaExpert GUI |
| A2 | ServiceLevel published on Backup | Connect UaExpert to Node B. Read same node. | Value = 100 (`AuthoritativeBackup`) | No |
| A3 | ServiceLevel updates when peer drops | Node A connected. Stop Node B (`sc stop OtOpcUa`). Watch `ServiceLevel` on Node A. | Transitions 255 → 230 (`IsolatedPrimary`) within ~6 s (3 × 2 s HTTP probe interval) | No |
| A4 | RedundancySupport | Browse `Server/ServerRedundancy/RedundancySupport` on either node. | Value = `Warm` or `Hot` matching the cluster `RedundancyMode` | No |
| A5 | ServerUriArray | Browse `Server/ServerRedundancy/ServerUriArray` on either node. | Array contains both `ApplicationUri` values; self listed first. Note: requires non-transparent redundancy-type upgrade (currently logs-and-skips — see known limitation A5 below). | No |
| A6 | Mid-apply ServiceLevel dip | Trigger a `sp_PublishGeneration` apply (via Admin UI draft → publish) while watching Node A `ServiceLevel`. | Drops to 200 (`PrimaryMidApply`) for the apply duration; returns to 255 after `RefreshAsync`. | No |
| A7 | Client.CLI reads correct ServiceLevel | `dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://<node-a>:4840 -n "i=2267"` | Prints current byte value matching expected band. | **Yes** — scriptable with the Client CLI |
| A8 | otopcua-cli failover reconnect | `dotnet run ... -- connect -u opc.tcp://<node-a>:4840 -F opc.tcp://<node-b>:4840` — then kill Node A. | CLI session reconnects to Node B within the session keep-alive timeout. | **Yes** — scriptable with the Client CLI |
| A1 | ServiceLevel on Primary | Read `Server/ServerStatus/ServiceLevel` (`i=2267`) on the Primary. | 250 | Yes (Client CLI) |
| A2 | ServiceLevel on follower | Read the same node on the other node. | 240 | Yes |
| A3 | Peer stop does not demote the Primary | Stop the follower. Watch `ServiceLevel` on the Primary for >= 40 s. | Stays 250 (the benefit-of-the-doubt rule). A change here is a defect or an unrelated health input. | Yes |
| A4 | RedundancySupport | Read `i=3709` on either node (`otopcua-cli redundancy`). | **Known unmet:** no code sets it, so it reports the SDK default, not the cluster's `Warm`/`Hot`. Record the observed value; do not mark PASS. | Yes |
| A5 | `ServerArray` peer discovery | Read `i=2254` on either node. | String[] contains both `ApplicationUri` values (self and peer). This is the discovery surface in use today. | Yes |
| A5b | `ServerUriArray` conformance | Read `i=11314` on either node. | **Known unmet standard-metadata gate.** The node is absent, so the read fails or is empty; the client CLI tolerates this. `ServerArray` (A5) is **not** a substitute and must not turn A5b into a PASS. | Yes |
| A6 | Endpoint-failure step-down | Occupy port 4840 on the Primary host before it starts (fused admin+driver node), or follow lmxopcua#761. | Failed node publishes 0; the peer becomes Primary and publishes 250 (see `docs/Redundancy.md` "Endpoint start failure: Primary step-down"). | No |
| A7 | Client CLI reads ServiceLevel | `dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://<node-a>:4840 -n "i=2267"` | Prints the expected byte. | Yes |
| A8 | otopcua-cli failover reconnect | `... -- connect -u opc.tcp://<node-a>:4840 -F opc.tcp://<node-b>:4840`, then stop Node A. | CLI reconnects to Node B within the session keep-alive timeout. | Yes |
### Block B — Third-party client failover
| ID | Scenario | Procedure | Pass criterion |
|----|----------|-----------|----------------|
| B1 | UaExpert picks Primary by ServiceLevel | Configure a Redundancy Group in UaExpert with both endpoint URLs. | Client connects to Node A (higher ServiceLevel) |
| B2 | UaExpert cuts over on Primary kill | Kill Node A `OtOpcUa` service. | Client session reconnects to Node B within UaExpert's reconnect timeout (default 5 s). Data-change monitored items resume. |
| B3 | UaExpert returns when Primary restores | Start Node A. Wait >= 60 s recovery dwell. | `ServiceLevel` on Node A progresses: 180 (`RecoveringPrimary`) → 255 (`AuthoritativePrimary`). UaExpert may or may not switch back (client-policy-dependent; both outcomes accepted). |
| B4 | Kepware QuickClient failover | Repeat B1–B3 with Kepware configured for the same two endpoints. | Same pass criteria; establishes no UaExpert-specific behaviour. |
| B5 | AVEVA OI Gateway | Configure OI Gateway OPC DA/UA client object against the cluster. Kill Primary. | OI Gateway data quality recovers within `ReconnectInterval` (default 20 s); no permanent data-loss alert. |
| B1 | UaExpert picks Primary by ServiceLevel | Configure a Redundancy Group in UaExpert with both endpoint URLs. | Client connects to the 250 node. |
| B2 | UaExpert cuts over on Primary kill | Stop the Primary's service (supervised restart is expected, see below). | Client reconnects to the surviving node within UaExpert's reconnect timeout; data-change monitored items resume. The survivor becomes oldest Up after Akka downs the dead node (`auto-down`, default `auto-down-unreachable-after = 15s`), then publishes 250. Expect the cutover to take at least that long plus the 5 s `HealthTick`. |
| B3 | Returning node settles as follower | Restart the stopped node. | It rejoins as the youngest member and settles at 240; **it does not take Primary back** (oldest-Up election). Whether the client switches is client-policy-dependent; both outcomes are accepted. |
| B4 | Kepware QuickClient failover | Repeat B1–B3 with Kepware. | Same criteria. |
| B5 | AVEVA OI Gateway | Configure the client object against both endpoints; stop the Primary. | Data quality recovers within `ReconnectInterval` (default 20 s); no permanent data-loss alert. |
### Block C — Galaxy MXAccess failover
This block requires a running Galaxy and `$MxAccessClient` object (AVEVA
System Platform installed, Galaxy deployed on dev box — see project memory
`project_aveva_platform_installed.md`).
Requires a running Galaxy and `$MxAccessClient` object (AVEVA System Platform installed, Galaxy deployed).
| ID | Scenario | Procedure | Pass criterion |
|----|----------|-----------|----------------|
| C1 | Galaxy binds to Primary on first connect | Bring cluster up. Start a Galaxy `$MxAccessClient` with both node URLs configured. | Galaxy reports `QUALITY = Good`; initial values stream from Node A. |
| C2 | Galaxy redirects on Primary drop | Stop Node A. | Galaxy `QUALITY` briefly goes `Uncertain`, then returns to `Good`; values continue streaming from Node B within MXAccess's `ReconnectInterval` (default 20 s). |
| C3 | Galaxy tolerates mid-apply dip | Trigger generation apply on Node A. | Galaxy remains bound — mid-apply dip (200) is advisory, not a session drop. No quality interruption. |
| C1 | Galaxy binds to Primary on first connect | Start a Galaxy `$MxAccessClient` with both node URLs. | `QUALITY = Good`; values stream from the Primary. |
| C2 | Galaxy redirects on Primary drop | Stop the Primary. | `QUALITY` briefly `Uncertain`, then `Good`, streaming from the survivor within `ReconnectInterval` (default 20 s). |
| C3 | Galaxy tolerates a deployment | Trigger a deployment (AdminUI **Deployments**). | Galaxy stays bound; no quality interruption. (There is no mid-apply ServiceLevel dip in the current model.) |
Note: A negative result on C1–C3 does not necessarily indicate an OtOpcUa
defect. Cross-check with Block A / B first to confirm our `ServiceLevel`
signal is correct before debugging the MXAccess client layer.
A negative result on C1–C3 does not necessarily indicate an OtOpcUa defect; confirm Blocks A/B first.
## Step-by-step cutover-validation runbook
This is the minimum procedure to satisfy the v2 GA exit criterion:
"Non-transparent redundancy cutover validated with at least one production
client (Ignition 8.3 recommended — see decision #85)."
Minimum procedure for the v2 GA exit criterion "Non-transparent redundancy cutover validated with at least
one production client (Ignition 8.3 recommended — see decision #85)". **Standard metadata gaps remain
(A4, A5b); those rows stay unmet and the production-client cutover gate stays open until a real client
pass is recorded.**
### Step 1 — Provision the cluster
```powershell
# On the Config DB host, seed or verify cluster rows:
# ServerCluster: Id=<id>, Name="test-cluster", NodeCount=2, RedundancyMode=Warm
# ClusterNode A: NodeId="node-a", ClusterId=<id>, RedundancyRole=Primary,
# ServiceLevelBase=255, ApplicationUri="urn:node-a:OtOpcUa"
# ClusterNode B: NodeId="node-b", ClusterId=<id>, RedundancyRole=Secondary,
# ServiceLevelBase=100, ApplicationUri="urn:node-b:OtOpcUa"
Create/verify the `ServerCluster` (`NodeCount=2`, `RedundancyMode=Warm`) with two enabled `ClusterNode` rows
with distinct `ApplicationUri` values, via the AdminUI. Set per-node `Cluster:Roles`, self-first
`Cluster:SeedNodes` and `OpcUa:PeerApplicationUris` in each node's configuration.
### Step 2 — Start both nodes, oldest first
Start Node A, wait until it is Up (`/health/ready`), then start Node B. Wait ~40 s for the first
`HealthTick`, snapshot and peer-probe cycles (probe interval 10 s, connect timeout 2 s, freshness 30 s).
### Step 3 — Verify baseline with the Client CLI
```bash
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://<node-a-host>:4840 -n "i=2267" # expect 250
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://<node-b-host>:4840 -n "i=2267" # expect 240
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- redundancy -u opc.tcp://<node-a-host>:4840
```
Verify uniqueness constraint: no two `ClusterNode` rows share the same
`ApplicationUri` (unique index on `ApplicationUri`).
### Step 4 — Verify peer discovery and the unmet metadata
### Step 2 — Start both server instances
On Node A host:
```powershell
# appsettings.json: Node:NodeId = "node-a"
sc start OtOpcUa
```bash
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://<node-a-host>:4840 -n "i=2254" # ServerArray: expect both URIs
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://<node-a-host>:4840 -n "i=11314" # ServerUriArray: expect absent/empty (unmet gate A5b)
```
On Node B host:
Record `ServerArray` as A5 PASS/FAIL and `ServerUriArray` separately as A5b; an absent `ServerUriArray` is
recorded as **unmet**, never as a pass via `ServerArray`.
```powershell
# appsettings.json: Node:NodeId = "node-b"
sc start OtOpcUa
```
### Step 5 — Primary kill and failover (B2)
Wait 10 s for HostedServices to complete first probe cycle.
1. Connect the third-party client's redundancy group to both endpoints; subscribe to a variable.
2. Stop the Primary's service (`sc stop OtOpcUaHost` on a Windows node; production service recovery restarts
it, so stop the supervisor action or accept the restart).
3. Observe the survivor's `ServiceLevel` go 240 -> 250 once Akka downs the dead node (about 15 s plus up to a
5 s tick), and the client resume data-change events on it.
4. Record time from stop to reconnect and any data gap.
### Step 3 — Verify baseline ServiceLevel via Client CLI
Planned alternative: use the AdminUI **Trigger failover** button on `/clusters/{id}/redundancy` (graceful
`Leave`, Administrator only); the survivor becomes oldest and advertises 250 without the downing delay.
```powershell
# Node A should report 255
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read `
-u opc.tcp://<node-a-host>:4840 -n "i=2267"
### Step 6 — Recovery (B3)
# Node B should report 100
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read `
-u opc.tcp://<node-b-host>:4840 -n "i=2267"
```
1. Restart the stopped node.
2. Observe it rejoin as the youngest member and settle at 240 while the survivor stays 250. There is no
recovery dwell and no automatic return of the Primary role.
3. Record whether the client switches back.
Pass: Node A = 255, Node B = 100.
### Step 7 — Deployment under load (C3)
### Step 4 — Verify ServerUriArray
```powershell
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read `
-u opc.tcp://<node-a-host>:4840 -n "i=2271"
```
Pass: array returned contains both `ApplicationUri` strings. If
`ServerUriArray` node returns empty or an error, the non-transparent
redundancy-type upgrade follow-up is still pending (known limitation —
`ServerRedundancyNodeWriter.ApplyServerUriArray` logs-and-skips on the
base `ServerRedundancyState` object type).
### Step 5 — Execute Primary kill + failover (B2 scenario)
1. Connect UaExpert (or Kepware) Redundancy Group to both endpoints.
2. Confirm client is subscribed to at least one variable node.
3. Kill Node A: `sc stop OtOpcUa` on Node A host.
4. Observe:
- Node B `ServiceLevel` should transition: 100 (`AuthoritativeBackup`)
→ 80 (`IsolatedBackup`) within ~6 s.
- Client should reconnect to Node B and resume data-change events.
5. Record: time from kill to client reconnect; whether data gaps occurred.
### Step 6 — Verify Primary recovery (B3 scenario)
1. Restart Node A: `sc start OtOpcUa` on Node A host.
2. Observe Node A `ServiceLevel` progression:
- ~0 s: 1 (`NoData`) briefly while HostedServices start.
- Startup: 180 (`RecoveringPrimary`) — recovery dwell gate active.
- After >= 60 s dwell + one positive publish witness: 255 (`AuthoritativePrimary`).
3. Observe Node B:
- Returns to 100 (`AuthoritativeBackup`) once it sees Node A peer probe succeed.
4. Record dwell duration and whether the client (UaExpert/Kepware) switches back.
### Step 7 — Execute mid-apply dip (A6 scenario)
1. Via Admin UI, create a trivial draft change and publish.
2. Watch Node A `ServiceLevel` during apply.
3. Expected: drops to 200 (`PrimaryMidApply`) for the apply duration
(typically seconds); returns to 255 when `GenerationRefreshHostedService`
releases the lease.
Trigger a trivial deployment from the AdminUI while watching both nodes' `ServiceLevel`; expect no change.
### Step 8 — Record results
Copy the following block into a tracking doc:
```
Run date: YYYY-MM-DD
Release SHA: <git sha>
Cluster: <cluster-id> Primary: node-a Backup: node-b
Config DB: 10.100.0.35,14330
Cluster: <cluster-id> Node A: <host> Node B: <host>
A1: [PASS/FAIL] evidence: <screenshot or CLI output>
A1: [PASS/FAIL] evidence: <CLI output>
A2: [PASS/FAIL]
A3: [PASS/FAIL] time-to-IsolatedPrimary: <N>s
A4: [PASS/FAIL]
A5: [PASS/FAIL/DEFERRED - ServerUriArray upgrade pending]
A6: [PASS/FAIL] mid-apply duration: <N>s
A7: [PASS/FAIL] CLI output attached
A8: [PASS/FAIL] CLI reconnect observed
B1: [PASS/FAIL]
B2: [PASS/FAIL] reconnect time: <N>s
B3: [PASS/FAIL] dwell observed: <N>s
B4: [PASS/FAIL] (Kepware)
B5: [PASS/FAIL] (OI Gateway — if available)
C1: [PASS/FAIL/SKIP - Galaxy not available]
C2: [PASS/FAIL/SKIP]
C3: [PASS/FAIL/SKIP]
A3: [PASS/FAIL]
A4: [UNMET - observed value: <value>]
A5: [PASS/FAIL] ServerArray contents: <..>
A5b: [UNMET - ServerUriArray absent] (must not be PASS while the node is absent)
A6: [PASS/FAIL/SKIP]
A7: [PASS/FAIL]
A8: [PASS/FAIL]
B1..B5: [PASS/FAIL/SKIP] reconnect time: <N>s
C1..C3: [PASS/FAIL/SKIP - Galaxy not available]
```
One pass of every non-SKIP row is the v2 GA acceptance criterion.
One pass of every non-SKIP row, with A4/A5b either genuinely passing after the metadata work lands or being
explicitly recorded as unmet, is required before G9 can be called satisfied.
## Known limitations
### A5 — ServerUriArray node not yet writable
### A4 / A5b — standard redundancy metadata not exposed
The OPC UA .NET Standard SDK's default `Server.ServerRedundancy` object is the
base `ServerRedundancyState`, which has no `ServerUriArray` child node.
`ServerRedundancyNodeWriter.ApplyServerUriArray` currently logs a warning and
skips. The operator obtains `ServerUriArray` by reading `ClusterNode` rows
directly until the non-transparent redundancy-type upgrade follow-up ships.
OtOpcUa does not set `RedundancySupport` and does not expose `Server.ServerRedundancy.ServerUriArray`
(`i=11314`): the SDK's default `ServerRedundancy` object is the base type with no such child. Peer discovery
works through `Server.ServerArray` (`i=2254`) only. Clients that require the standard
`NonTransparentRedundancyType` metadata will not recognise the pair. The metadata gap is tracked separately
from this runbook.
### Recovery dwell is 60 s by default
### Failover timing is bounded by downing, not by probes
`RecoveryStateManager.DwellTime` defaults to `TimeSpan.FromSeconds(60)` in
`Program.cs`. Step 6 of the runbook will block for at least 60 s waiting for
Node A to return to `AuthoritativePrimary`. This is intentional per
decision #154 (thrash prevention) — do not lower it for the test run.
A crashed Primary is replaced only after Akka marks it unreachable and `auto-down` removes it (15 s by
default), plus the 5 s ServiceLevel tick. A genuine partition leaves both sides active (accepted
availability-over-partition-safety trade; see `docs/Redundancy.md` "Split-brain / downing").
### IsolatedBackup (80) does not auto-promote
### The returning node does not reclaim Primary
Per decision #154, the Backup at band 80 does not self-elevate. If the operator
needs authoritative service from Node B while Node A is down, they must write
`RedundancyRole=Primary` on the `ClusterNode` row for Node B and publish a
draft generation. The Admin UI `RedundancyTab` exposes this flow.
Election is oldest-Up, so a restarted node is the youngest and stays a follower (240) until the other node
leaves the mesh.
## Dependency on existing tests
The cutover runbook validates the end-to-end wire path. The math and edge cases
are already locked by the unit/integration tests enumerated in the first section.
A failing runbook step that contradicts a passing unit test indicates a
deployment configuration error or an SDK version mismatch — not a logic bug.
Check `PeerHttpProbeLoop` logs first (look for `PeerProbe` Serilog events).
The runbook validates the end-to-end wire path; the math and edge cases are locked by the tests enumerated
above. A failing step that contradicts a passing unit test indicates a deployment configuration error
(roles, seed order, `PeerApplicationUris`) or an SDK version mismatch. Check the `PeerOpcUaProbeActor` /
`OpcUaPublishActor` logs first.