fix(redundancy): a failed OPC UA endpoint drops ServiceLevel to 0 and steps the Primary down (lmxopcua#762)
v2-ci / unit-tests (push) Has been cancelled

IOpcUaEndpointState (review-remediation lane C3) recorded whether the SDK started, but only the
health check read it: a fused node whose endpoint failed (port 4840 taken, ...) still advertised
250/240 and stayed driver Primary, so the Primary-gated data plane ran on a node no client could
reach while the reachable peer refused writes as Secondary.

- Move IOpcUaEndpointState/OpcUaEndpointState/OpcUaEndpointStatus from Host to Commons.OpcUa so
  the Runtime can consume it.
- ServiceLevelCalculator: new NodeHealthInputs.EndpointListening (default true). false -> 0,
  ahead of every other input including the +10 Primary bonus.
- OpcUaPublishActor: takes the endpoint state, feeds EndpointListening (Failed only; NotStarted /
  Stopped are boot/shutdown windows) on the DB-backed and DB-less paths, and the legacy role-only
  seam returns 0 for a failed endpoint too.
- New OpcUaEndpointStepDownActor (one per driver node): a Primary (ClusterActiveNode.SelfIsActive,
  the election's own primitive) with a Failed endpoint gracefully Leaves its mesh - the manual
  failover mechanism - so the peer becomes the oldest Up driver member, hosts the singletons and is
  Primary. The election itself is not bent. Guarded on a peer having recently reported its own
  endpoint listening (EndpointReport on the new opcua-endpoint-state DPS topic), so a pair with
  both endpoints failed does not bounce the role and restart nodes forever, and a lone node never
  shuts itself down.
- Runtime now references ZB.MOM.WW.Health.Akka (ClusterActiveNode).
- Docs: Redundancy.md tier table + new "Endpoint start failure: Primary step-down" section;
  cluster-topology rule; HealthEndpoints / OpcUaEndpointHealthCheck comments (F-C1 landed).

Tests: ServiceLevelCalculatorTests (+5), OpcUaPublishActorEndpointStateTests (7: DB-less,
DB-backed, legacy seam, NotStarted/Stopped do not demote, a later failure demotes on the next
tick), OpcUaEndpointStepDownActorTests (3, real two-node mesh: holds while the peer is not
listening then leaves and the peer becomes Primary; failed Secondary and healthy Primary never
leave). Each was mutation-checked: removing the calculator/legacy guard, the Leave, the
peer-listening guard or the Primary guard fails the corresponding tests.
This commit is contained in:
Joseph Doherty
2026-11-09 12:15:59 -05:00
parent 8ef5adc63a
commit acfb4fbaca
13 changed files with 668 additions and 30 deletions
@@ -11,7 +11,7 @@ using ZB.MOM.WW.LocalDb.Replication;
using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Configuration;
using ZB.MOM.WW.OtOpcUa.Host.Configuration;
using ZB.MOM.WW.OtOpcUa.Host.OpcUa;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
namespace ZB.MOM.WW.OtOpcUa.Host.Health;
@@ -45,11 +45,12 @@ public static class HealthEndpoints
// The ACTIVE tier depends on the roles (coordinator ruling, fix round 1 / I6):
// - driver-only: Active too — a site node's whole purpose is its OPC UA endpoint, so nothing may
// route to it;
// - fused admin+driver: NOT Active, until the ServiceLevel calculator consumes IOpcUaEndpointState
// (follow-up F-C1). Traefik routes the AdminUI on /health/active, so putting the probe there would
// take the operator's only console out of rotation precisely when they need it to diagnose the
// failed endpoint, while the OPC UA failover (ServiceLevel) still would not move. Once F-C1 lands
// the peer takes Primary and the active tier follows it on its own.
// - fused admin+driver: NOT Active. Traefik routes the AdminUI on /health/active, so putting the probe
// there would take the operator's only console out of rotation precisely when they need it to
// diagnose the failed endpoint. The OPC UA failover is handled by the Runtime instead (F-C1,
// lmxopcua#762): the failed node computes ServiceLevel 0 and, when it is the Primary and its peer
// reports a listening endpoint, OpcUaEndpointStepDownActor makes it leave its mesh — the peer becomes
// the oldest Up member, so the active tier follows it on its own.
// Factory-registered so the state singleton is resolved per probe.
if (hasDriver)
{
@@ -1,5 +1,5 @@
using Microsoft.Extensions.Diagnostics.HealthChecks;
using ZB.MOM.WW.OtOpcUa.Host.OpcUa;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
namespace ZB.MOM.WW.OtOpcUa.Host.Health;
@@ -12,8 +12,9 @@ namespace ZB.MOM.WW.OtOpcUa.Host.Health;
/// <remarks>
/// Registered on driver-role nodes only (<see cref="HealthEndpoints.AddOtOpcUaHealth"/>); an
/// admin-only node serves no OPC UA endpoint and must not be degraded by one it never starts. On a fused
/// admin+driver node it is Ready-only until ServiceLevel consumes the fact (F-C1), because the Active tier
/// is what Traefik routes the AdminUI on.
/// admin+driver node it is Ready-only, because the Active tier is what Traefik routes the AdminUI on; the
/// Primary moves through the Runtime instead — ServiceLevel 0 plus <c>OpcUaEndpointStepDownActor</c>
/// (lmxopcua#762).
/// </remarks>
public sealed class OpcUaEndpointHealthCheck : IHealthCheck
{
@@ -1,73 +0,0 @@
namespace ZB.MOM.WW.OtOpcUa.Host.OpcUa;
/// <summary>Lifecycle of this node's OPC UA endpoint as <see cref="OtOpcUaServerHostedService"/> drives it.</summary>
public enum OpcUaEndpointStatus
{
/// <summary>The hosted service has not run <c>StartAsync</c> yet.</summary>
NotStarted,
/// <summary>The SDK is listening and the address-space / ServiceLevel sinks are bound.</summary>
Started,
/// <summary>The SDK failed to start (port in use, PKI unwritable, bad <c>OpcUa:*</c> override, …).</summary>
Failed,
/// <summary>The hosted service stopped the SDK; the port is released.</summary>
Stopped,
}
/// <summary>
/// The one place the "did the OPC UA SDK actually start?" fact lives (R03:2.1). Before it, a failed
/// start was one Error line: <c>/health/ready</c> stayed 200, the node joined its mesh, could be
/// elected driver Primary and acked deploys as Applied through the Null sink — while clients got
/// connection-refused from the node the fleet called healthy.
/// </summary>
/// <remarks>
/// Consumed by <c>OpcUaEndpointHealthCheck</c> (Ready + Active tiers). The redundancy election and the
/// ServiceLevel calculator do not read it yet — they live outside the Host (<c>Cluster</c> /
/// <c>Runtime</c>); feeding them is the Wave 3 follow-up the lane report names. Until then a fused
/// node whose SDK failed is visible (503) but still Primary-eligible.
/// </remarks>
public interface IOpcUaEndpointState
{
/// <summary>The endpoint's current lifecycle status.</summary>
OpcUaEndpointStatus Status { get; }
/// <summary>Why the start failed, when <see cref="Status"/> is <see cref="OpcUaEndpointStatus.Failed"/>.</summary>
string? FailureReason { get; }
}
/// <summary>Mutable <see cref="IOpcUaEndpointState"/>; written by the hosted service, read by the health check.</summary>
public sealed class OpcUaEndpointState : IOpcUaEndpointState
{
private volatile OpcUaEndpointStatus _status = OpcUaEndpointStatus.NotStarted;
private volatile string? _failureReason;
/// <inheritdoc />
public OpcUaEndpointStatus Status => _status;
/// <inheritdoc />
public string? FailureReason => _failureReason;
/// <summary>Records a successful SDK start.</summary>
public void MarkStarted()
{
_failureReason = null;
_status = OpcUaEndpointStatus.Started;
}
/// <summary>Records a failed SDK start.</summary>
/// <param name="reason">The failure, as the health probe should report it.</param>
public void MarkFailed(string reason)
{
_failureReason = reason;
_status = OpcUaEndpointStatus.Failed;
}
/// <summary>Records that the hosted service stopped the SDK.</summary>
public void MarkStopped()
{
_failureReason = null;
_status = OpcUaEndpointStatus.Stopped;
}
}