From e31888c9fd3f2156a58a8f775a5e1ddacfa0ca88 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 9 Nov 2026 12:30:03 -0500 Subject: [PATCH] docs(redundancy): rewrite phase 6.3 interop runbook for the current model and correct standard NodeIds (lmxopcua#716) Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV --- .../phase-6-3-redundancy-interop-plan.md | 364 ++++++++---------- 1 file changed, 162 insertions(+), 202 deletions(-) diff --git a/docs/plans/phase-6-3-redundancy-interop-plan.md b/docs/plans/phase-6-3-redundancy-interop-plan.md index ae6464fc..f8792504 100644 --- a/docs/plans/phase-6-3-redundancy-interop-plan.md +++ b/docs/plans/phase-6-3-redundancy-interop-plan.md @@ -1,278 +1,238 @@ # Phase 6.3 Redundancy — Client Interop Matrix and Cutover Validation Plan -> **Scope**: Phase 6.3 redundancy runtime core shipped (PRs #89-90, #98-99, -> #24-peerprobe, Stream C node wiring, Stream D lease wrap). What remains is -> Stream F (task #150): validating that third-party OPC UA clients honour -> our `ServiceLevel` / `ServerUriArray` / `RedundancySupport` signals and -> fail over correctly when the Primary drops. This document defines what is -> automatable as integration tests, what requires two live instances plus a -> real client, and a step-by-step cutover-validation runbook. +> **Revised for the current redundancy model (lmxopcua#716, source audit; not a live client pass).** +> The original Phase 6.3 text of this plan described the v1 model (`ClusterNode.RedundancyRole`, +> `PeerHttpProbeLoop` / `PeerUaProbeLoop`, `RecoveryStateManager`, `ApplyLeaseRegistry`, the +> 255/230/200/180/100/80 service-level bands). Those types and bands no longer exist. Primary/secondary is +> now derived from Akka cluster membership (oldest Up `driver` member of each application `Cluster`'s own +> 2-node mesh) and ServiceLevel comes from `ServiceLevelCalculator`. This document is the executable +> checklist for GA gate **G9** (see `docs/plans/v2-ga-lab-gates-plan.md`, tracked by #615); it still +> requires two live nodes and a real third-party client, and **no lab-client result is claimed here**. > -> **Source of truth**: `docs/Redundancy.md`, `docs/v2/redundancy-interop-playbook.md`, -> `docs/v2/implementation/phase-6-3-redundancy-runtime.md`, -> `scripts/compliance/phase-6-3-compliance.ps1`. +> **Source of truth**: `docs/Redundancy.md` (election, tiers, downing, manual failover, `ServerArray`), +> `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Redundancy/ServiceLevelCalculator.cs` and the tests named below. +> `docs/v2/redundancy-interop-playbook.md` and `scripts/compliance/phase-6-3-compliance.ps1` are +> **historical** (they still name the deleted types and old bands) and must not be used as pass criteria. + +## Standard variables used by this plan + +Identifiers verified against the OPC UA SDK constants shipped with the pinned +`OPCFoundation.NetStandard.Opc.Ua.Core` (`VariableIds`): + +| Variable | NodeId | Notes | +|---|---|---| +| `Server.ServiceLevel` | `i=2267` | Byte; published by `SdkServiceLevelPublisher`. | +| `Server.ServerArray` | `i=2254` | String[]; the **peer-discovery surface OtOpcUa actually populates** (self + `OpcUa:PeerApplicationUris`). | +| `Server.ServerRedundancy.RedundancySupport` | `i=3709` | Nothing in `src/` writes this; it reads the SDK default. See A4. | +| `Server.ServerRedundancy.ServerUriArray` | `i=11314` | The standard non-transparent redundancy variable. **Not populated** (the SDK's default `ServerRedundancy` object type has no such child). | +| `Server.ServerCapabilities.LocaleIdArray` | `i=2271` | **Not** a redundancy variable. Earlier revisions of this plan read `i=2271` as `ServerUriArray`; that was wrong. | + +## Current service-level bands + +From `docs/Redundancy.md` (`ServiceLevelCalculator`, the live publish path; recomputed on every ~5 s +`HealthTick` and on each `RedundancyStateChanged`): + +| Byte | Meaning | +|---|---| +| 250 | Healthy and the driver Primary (oldest Up `driver` member of the cluster's mesh). | +| 240 | Healthy follower (DB reachable, peer OPC UA probe ok, not stale, not Primary). | +| 200 | Stale data, ConfigDb reachable. | +| 100 | ConfigDb unreachable and data stale (DB-backed nodes only; a driver-only node feeds `DbReachable=true` and stays 240/250). | +| 0 | Member not `Up`/`Joining`, node has no `driver` role, or this node's OPC UA endpoint failed to start (`EndpointListening=false`). | + +There is no recovery-dwell band, no mid-apply band and no "isolated primary" band any more. A Primary whose +peer is stopped **stays 250**: `OpcUaProbeOk` is a peer probing *this* node, absent/stale results default to +`true`, and a departed peer never demotes it. ## What is already tested (no live cluster needed) -The following are covered by existing automated tests that run in ordinary -`dotnet test`: +Ordinary `dotnet test` coverage of the pieces this runbook relies on: | Area | Test class(es) | What it asserts | |---|---|---| -| `ServiceLevelCalculator` — 8-state matrix | `ServiceLevelCalculatorTests` | All 10 band values; role × self-health × peer-http × peer-ua × apply × recovery × topology combinations | -| `RecoveryStateManager` — dwell + witness | `RecoveryStateManagerTests` | 60 s dwell default; premature-exit rejection; witness-required gate | -| `ApplyLeaseRegistry` — lease lifecycle | `ApplyLeaseRegistryTests` | Disposal on success / exception / cancellation; watchdog force-close at 10 min | -| `ServerRedundancyNodeWriter` — OPC UA variable binding | `ServerRedundancyNodeWriterTests` | `ServiceLevel` byte push; `RedundancySupport` enum; `ServerUriArray` skip-log when node absent | -| `RedundancyStatePublisher` — orchestration | `RedundancyStatePublisherTests` | Edge-triggered `OnStateChanged`; idempotent dedup | -| `ClusterTopologyLoader` | `ClusterTopologyLoaderTests` | Two-node seed; one-node degenerate; duplicate-URI rejection | -| `DraftValidator.ValidateClusterTopology` | `DraftValidatorTests` (8 cases) | NodeCount/mode pairs; Enabled-count vs NodeCount; multiple-Primary rejection | +| ServiceLevel tiers | `ServiceLevelCalculatorTests` (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests`, `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests`) | Truth table above, including endpoint-failed = 0 | +| Primary election | `RedundancyPrimaryElectionTests` (ControlPlane.Tests) | Oldest Up `driver` member wins, even when it holds the higher address | +| Manual failover | `ManualFailoverServiceTests` (ControlPlane.Tests) | Leave of the oldest driver; parity with the election | +| Endpoint-failure step-down | `OpcUaEndpointStepDownActorTests`, `OpcUaPublishActorEndpointStateTests` (Runtime.Tests) | Failed Primary hands over only when the peer is listening; ServiceLevel 0 | +| Downing strategy | `SplitBrainResolverActivationTests` (Cluster.Tests) | `auto-down` installed through the production bootstrap | +| `ServerArray` peer URIs | `DualEndpointTests` (`OpcUaServer.IntegrationTests`) | Real OPCFoundation client reads both URIs from `i=2254` on loopback | +| Topology validity | `DraftValidatorTests` (Configuration.Tests) | `NodeCount`/`RedundancyMode` pairs | Run with: -```powershell -dotnet test ZB.MOM.WW.OtOpcUa.slnx --filter "FullyQualifiedName~Redundancy" +```bash +dotnet test ZB.MOM.WW.OtOpcUa.slnx --filter "FullyQualifiedName~Redundancy|FullyQualifiedName~ServiceLevelCalculator|FullyQualifiedName~ManualFailover|FullyQualifiedName~DualEndpoint" ``` -Compliance gate (every Phase 6.3 static check): - -```powershell -pwsh ./scripts/compliance/phase-6-3-compliance.ps1 -``` - -Pass criteria: exit 0; all `[PASS]` lines green; `[DEFERRED]` lines are -known-deferred surfaces, not failures. - ## What cannot be automated — requires two live instances -The scenarios below require two running `OtOpcUa.Server` processes in the -same `ServerCluster`, a real SQL Server Config DB, and at least one driver -instance with a reachable endpoint (simulator or real PLC). - -### Why it cannot be unit/integration-tested in-process - -- UaExpert, Kepware KEPServerEX, and AVEVA OI Gateway are closed-source - Windows GUI binaries with no headless CLI interface for the - subscribe/browse flows. -- The AVEVA MXAccess failover leg (`IAlarmSource` reconnect, `$MxAccessClient` - quality transition) involves the Galaxy runtime's own client-redundancy - policy and the COM-layer session model — both live outside this repo. -- Even the automatable sub-set (our own `otopcua-cli` as the client) needs - two distinct listening TCP endpoints; that requires two live processes, - which is out of scope for `dotnet test` integration fixtures. +- UaExpert, Kepware KEPServerEX and AVEVA OI Gateway are closed-source Windows GUI binaries with no + headless interface for the subscribe/browse flows. +- The AVEVA MXAccess failover leg involves the Galaxy runtime's own client-redundancy policy and COM-layer + session model, both outside this repo. +- Even the scriptable subset (our `otopcua-cli` as the client) needs two distinct listening endpoints, i.e. + two live processes, which is out of scope for `dotnet test` fixtures. ## Test matrix ### Prerequisites -1. Two `OtOpcUa.Server` processes on separate Windows hosts (or separate - ports on the same host for dev) sharing one Config DB (`ServerCluster` - with `NodeCount=2`, `RedundancyMode=Warm` or `Hot`). -2. Each node registered in `ClusterNode`: - - Node A: `RedundancyRole=Primary`, `ServiceLevelBase=255`, - `ApplicationUri=urn:node-a:OtOpcUa` - - Node B: `RedundancyRole=Secondary`, `ServiceLevelBase=100`, - `ApplicationUri=urn:node-b:OtOpcUa` -3. `PeerHttpProbeLoop` and `PeerUaProbeLoop` HostedServices running on both - nodes (registered via `AddHostedService` + - `AddHostedService` in `Program.cs`). -4. At least one `DriverInstance` in the cluster with a reachable PLC or - simulator (e.g. Modbus sim at `10.100.0.35:5020`). -5. Client machine with UaExpert >= 1.7 installed. -6. Optional second client: Kepware KEPServerEX 6.x QuickClient or AVEVA - OI Gateway 2020R2+. +1. Two `OtOpcUa.Host` processes forming **one application `Cluster`'s own 2-node Akka mesh** (see + `docs/Redundancy.md` "Per-cluster meshes (Phase 6)"): each node carries the roles `driver` and + `cluster-{ClusterId}` (via `Cluster:Roles`; `OTOPCUA_ROLES` is only the fallback), and each node's + `Cluster:SeedNodes` lists **itself first**, then its partner. The cluster row has `NodeCount=2`, + `RedundancyMode=Warm` or `Hot` (`DraftValidator` rejects other pairs). +2. Each node has a distinct `ApplicationUri` (unique per `ClusterNode`), and each node's + `OpcUa:PeerApplicationUris` lists its partner's `ApplicationUri` so `Server.ServerArray` returns both. +3. There is no operator-written role and no `ServiceLevelBase` input: `ClusterNode.ServiceLevelBase` still + exists in the schema but does not drive the runtime value. The Primary is whichever node is the oldest Up + `driver` member, so **start Node A first and let it come Up before starting Node B** if Node A is meant to + be Primary. +4. At least one `DriverInstance` with a reachable simulator or PLC (e.g. the Modbus sim on the fixture host). +5. Client machine with UaExpert >= 1.7. Optional: Kepware KEPServerEX 6.x QuickClient, AVEVA OI Gateway 2020R2+. -### Block A — OPC UA protocol signals (UaExpert, no failover yet) +### Block A — OPC UA protocol signals | ID | Scenario | Procedure | Pass criterion | Automatable? | |----|----------|-----------|----------------|--------------| -| A1 | ServiceLevel published on Primary | Connect UaExpert to Node A. Browse `Server/ServerStatus/ServiceLevel`. | Value = 255 (`AuthoritativePrimary`) | No — requires UaExpert GUI | -| A2 | ServiceLevel published on Backup | Connect UaExpert to Node B. Read same node. | Value = 100 (`AuthoritativeBackup`) | No | -| A3 | ServiceLevel updates when peer drops | Node A connected. Stop Node B (`sc stop OtOpcUa`). Watch `ServiceLevel` on Node A. | Transitions 255 → 230 (`IsolatedPrimary`) within ~6 s (3 × 2 s HTTP probe interval) | No | -| A4 | RedundancySupport | Browse `Server/ServerRedundancy/RedundancySupport` on either node. | Value = `Warm` or `Hot` matching the cluster `RedundancyMode` | No | -| A5 | ServerUriArray | Browse `Server/ServerRedundancy/ServerUriArray` on either node. | Array contains both `ApplicationUri` values; self listed first. Note: requires non-transparent redundancy-type upgrade (currently logs-and-skips — see known limitation A5 below). | No | -| A6 | Mid-apply ServiceLevel dip | Trigger a `sp_PublishGeneration` apply (via Admin UI draft → publish) while watching Node A `ServiceLevel`. | Drops to 200 (`PrimaryMidApply`) for the apply duration; returns to 255 after `RefreshAsync`. | No | -| A7 | Client.CLI reads correct ServiceLevel | `dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://:4840 -n "i=2267"` | Prints current byte value matching expected band. | **Yes** — scriptable with the Client CLI | -| A8 | otopcua-cli failover reconnect | `dotnet run ... -- connect -u opc.tcp://:4840 -F opc.tcp://:4840` — then kill Node A. | CLI session reconnects to Node B within the session keep-alive timeout. | **Yes** — scriptable with the Client CLI | +| A1 | ServiceLevel on Primary | Read `Server/ServerStatus/ServiceLevel` (`i=2267`) on the Primary. | 250 | Yes (Client CLI) | +| A2 | ServiceLevel on follower | Read the same node on the other node. | 240 | Yes | +| A3 | Peer stop does not demote the Primary | Stop the follower. Watch `ServiceLevel` on the Primary for >= 40 s. | Stays 250 (the benefit-of-the-doubt rule). A change here is a defect or an unrelated health input. | Yes | +| A4 | RedundancySupport | Read `i=3709` on either node (`otopcua-cli redundancy`). | **Known unmet:** no code sets it, so it reports the SDK default, not the cluster's `Warm`/`Hot`. Record the observed value; do not mark PASS. | Yes | +| A5 | `ServerArray` peer discovery | Read `i=2254` on either node. | String[] contains both `ApplicationUri` values (self and peer). This is the discovery surface in use today. | Yes | +| A5b | `ServerUriArray` conformance | Read `i=11314` on either node. | **Known unmet standard-metadata gate.** The node is absent, so the read fails or is empty; the client CLI tolerates this. `ServerArray` (A5) is **not** a substitute and must not turn A5b into a PASS. | Yes | +| A6 | Endpoint-failure step-down | Occupy port 4840 on the Primary host before it starts (fused admin+driver node), or follow lmxopcua#761. | Failed node publishes 0; the peer becomes Primary and publishes 250 (see `docs/Redundancy.md` "Endpoint start failure: Primary step-down"). | No | +| A7 | Client CLI reads ServiceLevel | `dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://:4840 -n "i=2267"` | Prints the expected byte. | Yes | +| A8 | otopcua-cli failover reconnect | `... -- connect -u opc.tcp://:4840 -F opc.tcp://:4840`, then stop Node A. | CLI reconnects to Node B within the session keep-alive timeout. | Yes | ### Block B — Third-party client failover | ID | Scenario | Procedure | Pass criterion | |----|----------|-----------|----------------| -| B1 | UaExpert picks Primary by ServiceLevel | Configure a Redundancy Group in UaExpert with both endpoint URLs. | Client connects to Node A (higher ServiceLevel) | -| B2 | UaExpert cuts over on Primary kill | Kill Node A `OtOpcUa` service. | Client session reconnects to Node B within UaExpert's reconnect timeout (default 5 s). Data-change monitored items resume. | -| B3 | UaExpert returns when Primary restores | Start Node A. Wait >= 60 s recovery dwell. | `ServiceLevel` on Node A progresses: 180 (`RecoveringPrimary`) → 255 (`AuthoritativePrimary`). UaExpert may or may not switch back (client-policy-dependent; both outcomes accepted). | -| B4 | Kepware QuickClient failover | Repeat B1–B3 with Kepware configured for the same two endpoints. | Same pass criteria; establishes no UaExpert-specific behaviour. | -| B5 | AVEVA OI Gateway | Configure OI Gateway OPC DA/UA client object against the cluster. Kill Primary. | OI Gateway data quality recovers within `ReconnectInterval` (default 20 s); no permanent data-loss alert. | +| B1 | UaExpert picks Primary by ServiceLevel | Configure a Redundancy Group in UaExpert with both endpoint URLs. | Client connects to the 250 node. | +| B2 | UaExpert cuts over on Primary kill | Stop the Primary's service (supervised restart is expected, see below). | Client reconnects to the surviving node within UaExpert's reconnect timeout; data-change monitored items resume. The survivor becomes oldest Up after Akka downs the dead node (`auto-down`, default `auto-down-unreachable-after = 15s`), then publishes 250. Expect the cutover to take at least that long plus the 5 s `HealthTick`. | +| B3 | Returning node settles as follower | Restart the stopped node. | It rejoins as the youngest member and settles at 240; **it does not take Primary back** (oldest-Up election). Whether the client switches is client-policy-dependent; both outcomes are accepted. | +| B4 | Kepware QuickClient failover | Repeat B1–B3 with Kepware. | Same criteria. | +| B5 | AVEVA OI Gateway | Configure the client object against both endpoints; stop the Primary. | Data quality recovers within `ReconnectInterval` (default 20 s); no permanent data-loss alert. | ### Block C — Galaxy MXAccess failover -This block requires a running Galaxy and `$MxAccessClient` object (AVEVA -System Platform installed, Galaxy deployed on dev box — see project memory -`project_aveva_platform_installed.md`). +Requires a running Galaxy and `$MxAccessClient` object (AVEVA System Platform installed, Galaxy deployed). | ID | Scenario | Procedure | Pass criterion | |----|----------|-----------|----------------| -| C1 | Galaxy binds to Primary on first connect | Bring cluster up. Start a Galaxy `$MxAccessClient` with both node URLs configured. | Galaxy reports `QUALITY = Good`; initial values stream from Node A. | -| C2 | Galaxy redirects on Primary drop | Stop Node A. | Galaxy `QUALITY` briefly goes `Uncertain`, then returns to `Good`; values continue streaming from Node B within MXAccess's `ReconnectInterval` (default 20 s). | -| C3 | Galaxy tolerates mid-apply dip | Trigger generation apply on Node A. | Galaxy remains bound — mid-apply dip (200) is advisory, not a session drop. No quality interruption. | +| C1 | Galaxy binds to Primary on first connect | Start a Galaxy `$MxAccessClient` with both node URLs. | `QUALITY = Good`; values stream from the Primary. | +| C2 | Galaxy redirects on Primary drop | Stop the Primary. | `QUALITY` briefly `Uncertain`, then `Good`, streaming from the survivor within `ReconnectInterval` (default 20 s). | +| C3 | Galaxy tolerates a deployment | Trigger a deployment (AdminUI **Deployments**). | Galaxy stays bound; no quality interruption. (There is no mid-apply ServiceLevel dip in the current model.) | -Note: A negative result on C1–C3 does not necessarily indicate an OtOpcUa -defect. Cross-check with Block A / B first to confirm our `ServiceLevel` -signal is correct before debugging the MXAccess client layer. +A negative result on C1–C3 does not necessarily indicate an OtOpcUa defect; confirm Blocks A/B first. ## Step-by-step cutover-validation runbook -This is the minimum procedure to satisfy the v2 GA exit criterion: -"Non-transparent redundancy cutover validated with at least one production -client (Ignition 8.3 recommended — see decision #85)." +Minimum procedure for the v2 GA exit criterion "Non-transparent redundancy cutover validated with at least +one production client (Ignition 8.3 recommended — see decision #85)". **Standard metadata gaps remain +(A4, A5b); those rows stay unmet and the production-client cutover gate stays open until a real client +pass is recorded.** ### Step 1 — Provision the cluster -```powershell -# On the Config DB host, seed or verify cluster rows: -# ServerCluster: Id=, Name="test-cluster", NodeCount=2, RedundancyMode=Warm -# ClusterNode A: NodeId="node-a", ClusterId=, RedundancyRole=Primary, -# ServiceLevelBase=255, ApplicationUri="urn:node-a:OtOpcUa" -# ClusterNode B: NodeId="node-b", ClusterId=, RedundancyRole=Secondary, -# ServiceLevelBase=100, ApplicationUri="urn:node-b:OtOpcUa" +Create/verify the `ServerCluster` (`NodeCount=2`, `RedundancyMode=Warm`) with two enabled `ClusterNode` rows +with distinct `ApplicationUri` values, via the AdminUI. Set per-node `Cluster:Roles`, self-first +`Cluster:SeedNodes` and `OpcUa:PeerApplicationUris` in each node's configuration. + +### Step 2 — Start both nodes, oldest first + +Start Node A, wait until it is Up (`/health/ready`), then start Node B. Wait ~40 s for the first +`HealthTick`, snapshot and peer-probe cycles (probe interval 10 s, connect timeout 2 s, freshness 30 s). + +### Step 3 — Verify baseline with the Client CLI + +```bash +dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://:4840 -n "i=2267" # expect 250 +dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://:4840 -n "i=2267" # expect 240 +dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- redundancy -u opc.tcp://:4840 ``` -Verify uniqueness constraint: no two `ClusterNode` rows share the same -`ApplicationUri` (unique index on `ApplicationUri`). +### Step 4 — Verify peer discovery and the unmet metadata -### Step 2 — Start both server instances - -On Node A host: - -```powershell -# appsettings.json: Node:NodeId = "node-a" -sc start OtOpcUa +```bash +dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://:4840 -n "i=2254" # ServerArray: expect both URIs +dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://:4840 -n "i=11314" # ServerUriArray: expect absent/empty (unmet gate A5b) ``` -On Node B host: +Record `ServerArray` as A5 PASS/FAIL and `ServerUriArray` separately as A5b; an absent `ServerUriArray` is +recorded as **unmet**, never as a pass via `ServerArray`. -```powershell -# appsettings.json: Node:NodeId = "node-b" -sc start OtOpcUa -``` +### Step 5 — Primary kill and failover (B2) -Wait 10 s for HostedServices to complete first probe cycle. +1. Connect the third-party client's redundancy group to both endpoints; subscribe to a variable. +2. Stop the Primary's service (`sc stop OtOpcUaHost` on a Windows node; production service recovery restarts + it, so stop the supervisor action or accept the restart). +3. Observe the survivor's `ServiceLevel` go 240 -> 250 once Akka downs the dead node (about 15 s plus up to a + 5 s tick), and the client resume data-change events on it. +4. Record time from stop to reconnect and any data gap. -### Step 3 — Verify baseline ServiceLevel via Client CLI +Planned alternative: use the AdminUI **Trigger failover** button on `/clusters/{id}/redundancy` (graceful +`Leave`, Administrator only); the survivor becomes oldest and advertises 250 without the downing delay. -```powershell -# Node A should report 255 -dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read ` - -u opc.tcp://:4840 -n "i=2267" +### Step 6 — Recovery (B3) -# Node B should report 100 -dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read ` - -u opc.tcp://:4840 -n "i=2267" -``` +1. Restart the stopped node. +2. Observe it rejoin as the youngest member and settle at 240 while the survivor stays 250. There is no + recovery dwell and no automatic return of the Primary role. +3. Record whether the client switches back. -Pass: Node A = 255, Node B = 100. +### Step 7 — Deployment under load (C3) -### Step 4 — Verify ServerUriArray - -```powershell -dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read ` - -u opc.tcp://:4840 -n "i=2271" -``` - -Pass: array returned contains both `ApplicationUri` strings. If -`ServerUriArray` node returns empty or an error, the non-transparent -redundancy-type upgrade follow-up is still pending (known limitation — -`ServerRedundancyNodeWriter.ApplyServerUriArray` logs-and-skips on the -base `ServerRedundancyState` object type). - -### Step 5 — Execute Primary kill + failover (B2 scenario) - -1. Connect UaExpert (or Kepware) Redundancy Group to both endpoints. -2. Confirm client is subscribed to at least one variable node. -3. Kill Node A: `sc stop OtOpcUa` on Node A host. -4. Observe: - - Node B `ServiceLevel` should transition: 100 (`AuthoritativeBackup`) - → 80 (`IsolatedBackup`) within ~6 s. - - Client should reconnect to Node B and resume data-change events. -5. Record: time from kill to client reconnect; whether data gaps occurred. - -### Step 6 — Verify Primary recovery (B3 scenario) - -1. Restart Node A: `sc start OtOpcUa` on Node A host. -2. Observe Node A `ServiceLevel` progression: - - ~0 s: 1 (`NoData`) briefly while HostedServices start. - - Startup: 180 (`RecoveringPrimary`) — recovery dwell gate active. - - After >= 60 s dwell + one positive publish witness: 255 (`AuthoritativePrimary`). -3. Observe Node B: - - Returns to 100 (`AuthoritativeBackup`) once it sees Node A peer probe succeed. -4. Record dwell duration and whether the client (UaExpert/Kepware) switches back. - -### Step 7 — Execute mid-apply dip (A6 scenario) - -1. Via Admin UI, create a trivial draft change and publish. -2. Watch Node A `ServiceLevel` during apply. -3. Expected: drops to 200 (`PrimaryMidApply`) for the apply duration - (typically seconds); returns to 255 when `GenerationRefreshHostedService` - releases the lease. +Trigger a trivial deployment from the AdminUI while watching both nodes' `ServiceLevel`; expect no change. ### Step 8 — Record results -Copy the following block into a tracking doc: - ``` Run date: YYYY-MM-DD Release SHA: -Cluster: Primary: node-a Backup: node-b -Config DB: 10.100.0.35,14330 +Cluster: Node A: Node B: -A1: [PASS/FAIL] evidence: +A1: [PASS/FAIL] evidence: A2: [PASS/FAIL] -A3: [PASS/FAIL] time-to-IsolatedPrimary: s -A4: [PASS/FAIL] -A5: [PASS/FAIL/DEFERRED - ServerUriArray upgrade pending] -A6: [PASS/FAIL] mid-apply duration: s -A7: [PASS/FAIL] CLI output attached -A8: [PASS/FAIL] CLI reconnect observed -B1: [PASS/FAIL] -B2: [PASS/FAIL] reconnect time: s -B3: [PASS/FAIL] dwell observed: s -B4: [PASS/FAIL] (Kepware) -B5: [PASS/FAIL] (OI Gateway — if available) -C1: [PASS/FAIL/SKIP - Galaxy not available] -C2: [PASS/FAIL/SKIP] -C3: [PASS/FAIL/SKIP] +A3: [PASS/FAIL] +A4: [UNMET - observed value: ] +A5: [PASS/FAIL] ServerArray contents: <..> +A5b: [UNMET - ServerUriArray absent] (must not be PASS while the node is absent) +A6: [PASS/FAIL/SKIP] +A7: [PASS/FAIL] +A8: [PASS/FAIL] +B1..B5: [PASS/FAIL/SKIP] reconnect time: s +C1..C3: [PASS/FAIL/SKIP - Galaxy not available] ``` -One pass of every non-SKIP row is the v2 GA acceptance criterion. +One pass of every non-SKIP row, with A4/A5b either genuinely passing after the metadata work lands or being +explicitly recorded as unmet, is required before G9 can be called satisfied. ## Known limitations -### A5 — ServerUriArray node not yet writable +### A4 / A5b — standard redundancy metadata not exposed -The OPC UA .NET Standard SDK's default `Server.ServerRedundancy` object is the -base `ServerRedundancyState`, which has no `ServerUriArray` child node. -`ServerRedundancyNodeWriter.ApplyServerUriArray` currently logs a warning and -skips. The operator obtains `ServerUriArray` by reading `ClusterNode` rows -directly until the non-transparent redundancy-type upgrade follow-up ships. +OtOpcUa does not set `RedundancySupport` and does not expose `Server.ServerRedundancy.ServerUriArray` +(`i=11314`): the SDK's default `ServerRedundancy` object is the base type with no such child. Peer discovery +works through `Server.ServerArray` (`i=2254`) only. Clients that require the standard +`NonTransparentRedundancyType` metadata will not recognise the pair. The metadata gap is tracked separately +from this runbook. -### Recovery dwell is 60 s by default +### Failover timing is bounded by downing, not by probes -`RecoveryStateManager.DwellTime` defaults to `TimeSpan.FromSeconds(60)` in -`Program.cs`. Step 6 of the runbook will block for at least 60 s waiting for -Node A to return to `AuthoritativePrimary`. This is intentional per -decision #154 (thrash prevention) — do not lower it for the test run. +A crashed Primary is replaced only after Akka marks it unreachable and `auto-down` removes it (15 s by +default), plus the 5 s ServiceLevel tick. A genuine partition leaves both sides active (accepted +availability-over-partition-safety trade; see `docs/Redundancy.md` "Split-brain / downing"). -### IsolatedBackup (80) does not auto-promote +### The returning node does not reclaim Primary -Per decision #154, the Backup at band 80 does not self-elevate. If the operator -needs authoritative service from Node B while Node A is down, they must write -`RedundancyRole=Primary` on the `ClusterNode` row for Node B and publish a -draft generation. The Admin UI `RedundancyTab` exposes this flow. +Election is oldest-Up, so a restarted node is the youngest and stays a follower (240) until the other node +leaves the mesh. ## Dependency on existing tests -The cutover runbook validates the end-to-end wire path. The math and edge cases -are already locked by the unit/integration tests enumerated in the first section. -A failing runbook step that contradicts a passing unit test indicates a -deployment configuration error or an SDK version mismatch — not a logic bug. -Check `PeerHttpProbeLoop` logs first (look for `PeerProbe` Serilog events). +The runbook validates the end-to-end wire path; the math and edge cases are locked by the tests enumerated +above. A failing step that contradicts a passing unit test indicates a deployment configuration error +(roles, seed order, `PeerApplicationUris`) or an SDK version mismatch. Check the `PeerOpcUaProbeActor` / +`OpcUaPublishActor` logs first.