fix(redundancy): a failed OPC UA endpoint drops ServiceLevel to 0 and steps the Primary down (lmxopcua#762)
v2-ci / unit-tests (push) Has been cancelled

IOpcUaEndpointState (review-remediation lane C3) recorded whether the SDK started, but only the
health check read it: a fused node whose endpoint failed (port 4840 taken, ...) still advertised
250/240 and stayed driver Primary, so the Primary-gated data plane ran on a node no client could
reach while the reachable peer refused writes as Secondary.

- Move IOpcUaEndpointState/OpcUaEndpointState/OpcUaEndpointStatus from Host to Commons.OpcUa so
  the Runtime can consume it.
- ServiceLevelCalculator: new NodeHealthInputs.EndpointListening (default true). false -> 0,
  ahead of every other input including the +10 Primary bonus.
- OpcUaPublishActor: takes the endpoint state, feeds EndpointListening (Failed only; NotStarted /
  Stopped are boot/shutdown windows) on the DB-backed and DB-less paths, and the legacy role-only
  seam returns 0 for a failed endpoint too.
- New OpcUaEndpointStepDownActor (one per driver node): a Primary (ClusterActiveNode.SelfIsActive,
  the election's own primitive) with a Failed endpoint gracefully Leaves its mesh - the manual
  failover mechanism - so the peer becomes the oldest Up driver member, hosts the singletons and is
  Primary. The election itself is not bent. Guarded on a peer having recently reported its own
  endpoint listening (EndpointReport on the new opcua-endpoint-state DPS topic), so a pair with
  both endpoints failed does not bounce the role and restart nodes forever, and a lone node never
  shuts itself down.
- Runtime now references ZB.MOM.WW.Health.Akka (ClusterActiveNode).
- Docs: Redundancy.md tier table + new "Endpoint start failure: Primary step-down" section;
  cluster-topology rule; HealthEndpoints / OpcUaEndpointHealthCheck comments (F-C1 landed).

Tests: ServiceLevelCalculatorTests (+5), OpcUaPublishActorEndpointStateTests (7: DB-less,
DB-backed, legacy seam, NotStarted/Stopped do not demote, a later failure demotes on the next
tick), OpcUaEndpointStepDownActorTests (3, real two-node mesh: holds while the peer is not
listening then leaves and the peer becomes Primary; failed Secondary and healthy Primary never
leave). Each was mutation-checked: removing the calculator/legacy guard, the Leave, the
peer-listening guard or the Primary guard fails the corresponding tests.
This commit is contained in:
Joseph Doherty
2026-11-09 12:15:59 -05:00
parent 8ef5adc63a
commit acfb4fbaca
13 changed files with 668 additions and 30 deletions
@@ -44,6 +44,7 @@ public static class ServiceCollectionExtensions
public const string DependencyMuxActorName = "dependency-mux";
public const string OpcUaPublishActorName = "opcua-publish";
public const string PeerProbeSupervisorName = "peer-probe-supervisor";
public const string OpcUaEndpointStepDownActorName = "opcua-endpoint-step-down";
public const string ContinuousHistorizationRecorderActorName = "continuous-historization-recorder";
/// <summary>
@@ -555,6 +556,10 @@ public static class ServiceCollectionExtensions
// a node with no driver role never registers it and test harnesses build this graph without it;
// the actor treats null as "nothing consumes authorization inputs" and skips the decode.
var dataPlaneAuthority = resolver.GetService<IDataPlaneAuthoritySink>();
// lmxopcua#762: the "did the OPC UA SDK start?" fact the Host's OtOpcUaServerHostedService maintains.
// Optional: a harness that builds this graph without the hosted service registers none, and both
// consumers then behave as if the endpoint were listening (the pre-#762 behaviour).
var endpointState = resolver.GetService<IOpcUaEndpointState>();
var publishActor = system.ActorOf(
OpcUaPublishActor.Props(
sink: addressSpaceSink,
@@ -564,10 +569,19 @@ public static class ServiceCollectionExtensions
applier: applier,
dbHealthProbe: dbHealth,
dbLess: dbFactory is null,
dataPlaneAuthority: dataPlaneAuthority),
dataPlaneAuthority: dataPlaneAuthority,
endpointState: endpointState),
OpcUaPublishActorName);
registry.Register<OpcUaPublishActorKey>(publishActor);
// lmxopcua#762: a driver Primary whose OPC UA endpoint failed to start leaves its mesh once a peer
// reports a listening endpoint, so the peer becomes Primary. Every driver node runs one — it is also
// how this node tells its peer whether IT can take over.
if (endpointState is not null)
{
system.ActorOf(OpcUaEndpointStepDownActor.Props(endpointState), OpcUaEndpointStepDownActorName);
}
// Per-node peer-probe supervisor — keeps one OPC UA TCP probe per OTHER non-Detached
// driver node, so every node is continuously probed by all its peers. OpcUaPublishActor
// consumes the resulting probe verdicts to learn this node's own reachability.