fix(redundancy): a failed OPC UA endpoint drops ServiceLevel to 0 and steps the Primary down (lmxopcua#762)
v2-ci / unit-tests (push) Has been cancelled

IOpcUaEndpointState (review-remediation lane C3) recorded whether the SDK started, but only the
health check read it: a fused node whose endpoint failed (port 4840 taken, ...) still advertised
250/240 and stayed driver Primary, so the Primary-gated data plane ran on a node no client could
reach while the reachable peer refused writes as Secondary.

- Move IOpcUaEndpointState/OpcUaEndpointState/OpcUaEndpointStatus from Host to Commons.OpcUa so
  the Runtime can consume it.
- ServiceLevelCalculator: new NodeHealthInputs.EndpointListening (default true). false -> 0,
  ahead of every other input including the +10 Primary bonus.
- OpcUaPublishActor: takes the endpoint state, feeds EndpointListening (Failed only; NotStarted /
  Stopped are boot/shutdown windows) on the DB-backed and DB-less paths, and the legacy role-only
  seam returns 0 for a failed endpoint too.
- New OpcUaEndpointStepDownActor (one per driver node): a Primary (ClusterActiveNode.SelfIsActive,
  the election's own primitive) with a Failed endpoint gracefully Leaves its mesh - the manual
  failover mechanism - so the peer becomes the oldest Up driver member, hosts the singletons and is
  Primary. The election itself is not bent. Guarded on a peer having recently reported its own
  endpoint listening (EndpointReport on the new opcua-endpoint-state DPS topic), so a pair with
  both endpoints failed does not bounce the role and restart nodes forever, and a lone node never
  shuts itself down.
- Runtime now references ZB.MOM.WW.Health.Akka (ClusterActiveNode).
- Docs: Redundancy.md tier table + new "Endpoint start failure: Primary step-down" section;
  cluster-topology rule; HealthEndpoints / OpcUaEndpointHealthCheck comments (F-C1 landed).

Tests: ServiceLevelCalculatorTests (+5), OpcUaPublishActorEndpointStateTests (7: DB-less,
DB-backed, legacy seam, NotStarted/Stopped do not demote, a later failure demotes on the next
tick), OpcUaEndpointStepDownActorTests (3, real two-node mesh: holds while the peer is not
listening then leaves and the peer becomes Primary; failed Secondary and healthy Primary never
leave). Each was mutation-checked: removing the calculator/legacy guard, the Leave, the
peer-listening guard or the Primary guard fails the corresponding tests.
This commit is contained in:
Joseph Doherty
2026-11-09 12:15:59 -05:00
parent 8ef5adc63a
commit acfb4fbaca
13 changed files with 668 additions and 30 deletions
@@ -11,7 +11,7 @@ using ZB.MOM.WW.LocalDb.Replication;
using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Configuration;
using ZB.MOM.WW.OtOpcUa.Host.Configuration;
using ZB.MOM.WW.OtOpcUa.Host.OpcUa;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
namespace ZB.MOM.WW.OtOpcUa.Host.Health;
@@ -45,11 +45,12 @@ public static class HealthEndpoints
// The ACTIVE tier depends on the roles (coordinator ruling, fix round 1 / I6):
// - driver-only: Active too — a site node's whole purpose is its OPC UA endpoint, so nothing may
// route to it;
// - fused admin+driver: NOT Active, until the ServiceLevel calculator consumes IOpcUaEndpointState
// (follow-up F-C1). Traefik routes the AdminUI on /health/active, so putting the probe there would
// take the operator's only console out of rotation precisely when they need it to diagnose the
// failed endpoint, while the OPC UA failover (ServiceLevel) still would not move. Once F-C1 lands
// the peer takes Primary and the active tier follows it on its own.
// - fused admin+driver: NOT Active. Traefik routes the AdminUI on /health/active, so putting the probe
// there would take the operator's only console out of rotation precisely when they need it to
// diagnose the failed endpoint. The OPC UA failover is handled by the Runtime instead (F-C1,
// lmxopcua#762): the failed node computes ServiceLevel 0 and, when it is the Primary and its peer
// reports a listening endpoint, OpcUaEndpointStepDownActor makes it leave its mesh — the peer becomes
// the oldest Up member, so the active tier follows it on its own.
// Factory-registered so the state singleton is resolved per probe.
if (hasDriver)
{
@@ -1,5 +1,5 @@
using Microsoft.Extensions.Diagnostics.HealthChecks;
using ZB.MOM.WW.OtOpcUa.Host.OpcUa;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
namespace ZB.MOM.WW.OtOpcUa.Host.Health;
@@ -12,8 +12,9 @@ namespace ZB.MOM.WW.OtOpcUa.Host.Health;
/// <remarks>
/// Registered on driver-role nodes only (<see cref="HealthEndpoints.AddOtOpcUaHealth"/>); an
/// admin-only node serves no OPC UA endpoint and must not be degraded by one it never starts. On a fused
/// admin+driver node it is Ready-only until ServiceLevel consumes the fact (F-C1), because the Active tier
/// is what Traefik routes the AdminUI on.
/// admin+driver node it is Ready-only, because the Active tier is what Traefik routes the AdminUI on; the
/// Primary moves through the Runtime instead — ServiceLevel 0 plus <c>OpcUaEndpointStepDownActor</c>
/// (lmxopcua#762).
/// </remarks>
public sealed class OpcUaEndpointHealthCheck : IHealthCheck
{
@@ -1,73 +0,0 @@
namespace ZB.MOM.WW.OtOpcUa.Host.OpcUa;
/// <summary>Lifecycle of this node's OPC UA endpoint as <see cref="OtOpcUaServerHostedService"/> drives it.</summary>
public enum OpcUaEndpointStatus
{
/// <summary>The hosted service has not run <c>StartAsync</c> yet.</summary>
NotStarted,
/// <summary>The SDK is listening and the address-space / ServiceLevel sinks are bound.</summary>
Started,
/// <summary>The SDK failed to start (port in use, PKI unwritable, bad <c>OpcUa:*</c> override, …).</summary>
Failed,
/// <summary>The hosted service stopped the SDK; the port is released.</summary>
Stopped,
}
/// <summary>
/// The one place the "did the OPC UA SDK actually start?" fact lives (R03:2.1). Before it, a failed
/// start was one Error line: <c>/health/ready</c> stayed 200, the node joined its mesh, could be
/// elected driver Primary and acked deploys as Applied through the Null sink — while clients got
/// connection-refused from the node the fleet called healthy.
/// </summary>
/// <remarks>
/// Consumed by <c>OpcUaEndpointHealthCheck</c> (Ready + Active tiers). The redundancy election and the
/// ServiceLevel calculator do not read it yet — they live outside the Host (<c>Cluster</c> /
/// <c>Runtime</c>); feeding them is the Wave 3 follow-up the lane report names. Until then a fused
/// node whose SDK failed is visible (503) but still Primary-eligible.
/// </remarks>
public interface IOpcUaEndpointState
{
/// <summary>The endpoint's current lifecycle status.</summary>
OpcUaEndpointStatus Status { get; }
/// <summary>Why the start failed, when <see cref="Status"/> is <see cref="OpcUaEndpointStatus.Failed"/>.</summary>
string? FailureReason { get; }
}
/// <summary>Mutable <see cref="IOpcUaEndpointState"/>; written by the hosted service, read by the health check.</summary>
public sealed class OpcUaEndpointState : IOpcUaEndpointState
{
private volatile OpcUaEndpointStatus _status = OpcUaEndpointStatus.NotStarted;
private volatile string? _failureReason;
/// <inheritdoc />
public OpcUaEndpointStatus Status => _status;
/// <inheritdoc />
public string? FailureReason => _failureReason;
/// <summary>Records a successful SDK start.</summary>
public void MarkStarted()
{
_failureReason = null;
_status = OpcUaEndpointStatus.Started;
}
/// <summary>Records a failed SDK start.</summary>
/// <param name="reason">The failure, as the health probe should report it.</param>
public void MarkFailed(string reason)
{
_failureReason = reason;
_status = OpcUaEndpointStatus.Failed;
}
/// <summary>Records that the hosted service stopped the SDK.</summary>
public void MarkStopped()
{
_failureReason = null;
_status = OpcUaEndpointStatus.Stopped;
}
}
@@ -241,6 +241,8 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
/// <c>DbReachable</c> is fed constant <c>true</c> and only snapshot staleness can demote. See
/// <see cref="RecomputeServiceLevel"/>.</summary>
private readonly bool _dbLess;
/// <summary>This node's OPC UA endpoint lifecycle fact (lmxopcua#762). Null = no fact (harness) = listening.</summary>
private readonly IOpcUaEndpointState? _endpointState;
private readonly TimeSpan _staleWindow;
private readonly TimeSpan _probeFreshnessWindow;
private readonly TimeSpan _healthTickInterval;
@@ -306,6 +308,9 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
/// ServiceLevel DB-reachability input is retired (<c>DbReachable</c> treated constant-true, only
/// staleness demotes). No <paramref name="dbHealthProbe"/> is wired on such a node.</param>
/// <param name="dataPlaneAuthority">The optional sink that receives the API keys and ACL grants carried by each applied artifact.</param>
/// <param name="endpointState">This node's OPC UA endpoint lifecycle fact (lmxopcua#762). A
/// <see cref="OpcUaEndpointStatus.Failed"/> endpoint computes ServiceLevel 0. Null (harnesses, a node with no
/// hosted SDK) is treated as listening.</param>
/// <returns>The configured <see cref="Props"/> for creating this actor, pinned to the
/// <see cref="DispatcherId"/> dispatcher.</returns>
public static Props Props(
@@ -320,7 +325,8 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
TimeSpan? healthTickInterval = null,
bool dbLess = false,
IDataPlaneAuthoritySink? dataPlaneAuthority = null,
Func<DateTime>? utcNow = null) =>
Func<DateTime>? utcNow = null,
IOpcUaEndpointState? endpointState = null) =>
Akka.Actor.Props.Create(() => new OpcUaPublishActor(
sink ?? NullOpcUaAddressSpaceSink.Instance,
serviceLevel ?? NullServiceLevelPublisher.Instance,
@@ -334,7 +340,8 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
healthTickInterval,
dbLess,
dataPlaneAuthority,
utcNow)).WithDispatcher(DispatcherId);
utcNow,
endpointState)).WithDispatcher(DispatcherId);
/// <summary>Test-only Props that omits the pinned-dispatcher requirement and skips the
/// DPS subscribe so unit tests can spin up the actor on a vanilla TestKit cluster.</summary>
@@ -359,6 +366,9 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
/// ServiceLevel DB-reachability input is retired (<c>DbReachable</c> treated constant-true, only
/// staleness demotes). No <paramref name="dbHealthProbe"/> is wired on such a node.</param>
/// <param name="dataPlaneAuthority">The optional sink that receives the API keys and ACL grants carried by each applied artifact.</param>
/// <param name="endpointState">This node's OPC UA endpoint lifecycle fact (lmxopcua#762). A
/// <see cref="OpcUaEndpointStatus.Failed"/> endpoint computes ServiceLevel 0. Null (harnesses, a node with no
/// hosted SDK) is treated as listening.</param>
/// <returns>The configured <see cref="Props"/> for creating this actor in tests, without dispatcher
/// pinning or the DPS subscribe.</returns>
public static Props PropsForTests(
@@ -374,7 +384,8 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
TimeSpan? healthTickInterval = null,
bool dbLess = false,
IDataPlaneAuthoritySink? dataPlaneAuthority = null,
Func<DateTime>? utcNow = null) =>
Func<DateTime>? utcNow = null,
IOpcUaEndpointState? endpointState = null) =>
Akka.Actor.Props.Create(() => new OpcUaPublishActor(
sink ?? NullOpcUaAddressSpaceSink.Instance,
serviceLevel ?? NullServiceLevelPublisher.Instance,
@@ -388,7 +399,8 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
healthTickInterval,
dbLess,
dataPlaneAuthority,
utcNow));
utcNow,
endpointState));
/// <summary>Test-only Props exposing the deferred-write bound and the plan-step seams.</summary>
internal static Props PropsForTests(
@@ -402,7 +414,7 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
// seam-carrying constructor is internal so the test seams stay off the public API (review M3).
Akka.Actor.Props.CreateBy(new TestProducer(() => new OpcUaPublishActor(
sink, NullServiceLevelPublisher.Instance, false, localNode, null, applier, null, null, null, null,
false, dataPlaneAuthority, null, deferredWriteCapacity, hooks)));
false, dataPlaneAuthority, null, deferredWriteCapacity, hooks, endpointState: null)));
/// <summary>Produces the actor through its internal (seam-carrying) constructor, for <see cref="PropsForTests(IOpcUaAddressSpaceSink, AddressSpaceApplier?, TestHooks?, int, NodeId?, IDataPlaneAuthoritySink?)"/>.</summary>
private sealed class TestProducer(Func<OpcUaPublishActor> create) : IIndirectActorProducer
@@ -434,6 +446,7 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
/// ServiceLevel DB-reachability input is retired (<c>DbReachable</c> treated constant-true, only
/// staleness demotes). No <paramref name="dbHealthProbe"/> is wired on such a node.</param>
/// <param name="dataPlaneAuthority">The optional sink that receives the API keys and ACL grants carried by each applied artifact.</param>
/// <param name="endpointState">This node's OPC UA endpoint lifecycle fact (lmxopcua#762); null is treated as listening.</param>
public OpcUaPublishActor(
IOpcUaAddressSpaceSink sink,
IServiceLevelPublisher serviceLevel,
@@ -447,15 +460,18 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
TimeSpan? healthTickInterval = null,
bool dbLess = false,
IDataPlaneAuthoritySink? dataPlaneAuthority = null,
Func<DateTime>? utcNow = null)
Func<DateTime>? utcNow = null,
IOpcUaEndpointState? endpointState = null)
: this(sink, serviceLevel, subscribeRedundancyTopic, localNode, dbFactory, applier, dbHealthProbe, staleWindow,
probeFreshnessWindow, healthTickInterval, dbLess, dataPlaneAuthority, utcNow, DefaultDeferredWriteCapacity, hooks: null)
probeFreshnessWindow, healthTickInterval, dbLess, dataPlaneAuthority, utcNow, DefaultDeferredWriteCapacity, hooks: null,
endpointState)
{
}
/// <summary>The full constructor: also the deferred-write bound and the test seams (internal — tests only).</summary>
/// <param name="deferredWriteCapacity">The bound on distinct nodes whose writes are held for replay while a rebuild plan is in flight.</param>
/// <param name="hooks">Test seams for the off-thread plan step; null in production.</param>
/// <param name="endpointState">This node's OPC UA endpoint lifecycle fact (lmxopcua#762); null is treated as listening.</param>
internal OpcUaPublishActor(
IOpcUaAddressSpaceSink sink,
IServiceLevelPublisher serviceLevel,
@@ -471,7 +487,8 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
IDataPlaneAuthoritySink? dataPlaneAuthority,
Func<DateTime>? utcNow,
int deferredWriteCapacity,
TestHooks? hooks)
TestHooks? hooks,
IOpcUaEndpointState? endpointState)
{
_deferred = new DeferredWrites(Math.Max(1, deferredWriteCapacity));
_hooks = hooks;
@@ -486,6 +503,7 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
_applier = applier;
_dbHealthProbe = dbHealthProbe;
_dbLess = dbLess;
_endpointState = endpointState;
_staleWindow = staleWindow ?? TimeSpan.FromSeconds(30);
_probeFreshnessWindow = probeFreshnessWindow ?? TimeSpan.FromSeconds(30);
_healthTickInterval = healthTickInterval ?? TimeSpan.FromSeconds(5);
@@ -1536,6 +1554,15 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
return verdict.Ok;
}
/// <summary>The OPC UA endpoint input for the calculator (lmxopcua#762): <c>false</c> only when this node's own
/// SDK start is recorded as <see cref="OpcUaEndpointStatus.Failed"/>. <see cref="OpcUaEndpointStatus.NotStarted"/>
/// and <see cref="OpcUaEndpointStatus.Stopped"/> are boot and shutdown windows, not faults, and a node with no
/// endpoint fact at all (a harness) is treated as listening. Read on every recompute — the periodic health tick
/// and every redundancy snapshot (heartbeated every 10 s) — so a change is picked up without a notification.
/// It is a LOCAL fact on purpose: the peer TCP probe behind <see cref="OpcUaProbeOk"/> cannot see a failed start
/// when another process holds the port, because its connect then succeeds against that process.</summary>
private bool EndpointListening() => _endpointState?.Status != OpcUaEndpointStatus.Failed;
/// <summary>
/// Computes the local OPC UA ServiceLevel and routes it through <see cref="ServiceLevelChanged"/>
/// (the dedup/publish/metric handler). The full <see cref="ServiceLevelCalculator"/> path is
@@ -1575,7 +1602,8 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
DbReachable: true,
OpcUaProbeOk: OpcUaProbeOk(),
Stale: (utcNow - entry.AsOfUtc) > _staleWindow,
IsDriverPrimary: entry.IsDriverPrimary);
IsDriverPrimary: entry.IsDriverPrimary,
EndpointListening: EndpointListening());
Self.Tell(new ServiceLevelChanged(ServiceLevelCalculator.Compute(dbLessInputs)));
return;
}
@@ -1585,7 +1613,9 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
// is the bootstrap value until the first DbHealthStatus lands.
if (_dbHealthProbe is null || _lastDbHealth is null)
{
Self.Tell(new ServiceLevelChanged(LegacyRoleOnly(entry)));
// The role-only seam has no health inputs, but a failed endpoint still wins (lmxopcua#762): the
// legacy switch would otherwise hand a node that serves nothing 240 as Primary.
Self.Tell(new ServiceLevelChanged(EndpointListening() ? LegacyRoleOnly(entry) : (byte)0));
return;
}
@@ -1597,7 +1627,8 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
Stale: !_lastDbHealth.Reachable
|| (now - _lastDbHealth.AsOfUtc) > _staleWindow
|| (now - entry.AsOfUtc) > _staleWindow,
IsDriverPrimary: entry.IsDriverPrimary);
IsDriverPrimary: entry.IsDriverPrimary,
EndpointListening: EndpointListening());
Self.Tell(new ServiceLevelChanged(ServiceLevelCalculator.Compute(inputs)));
}
@@ -0,0 +1,226 @@
using Akka.Actor;
using Akka.Cluster;
using Akka.Cluster.Tools.PublishSubscribe;
using Akka.Event;
using ZB.MOM.WW.Health.Akka;
using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Redundancy;
/// <summary>
/// Steps this node down from driver Primary while its own OPC UA endpoint is not listening (lmxopcua#762).
/// </summary>
/// <remarks>
/// <para>
/// <b>Why.</b> A fused admin+driver node whose SDK failed to start (port 4840 taken, PKI unwritable, a bad
/// <c>OpcUa:*</c> override) keeps running so the operator still has the AdminUI. Before this actor it also
/// stayed Primary: the oldest Up driver member, so the Primary-gated data plane (writes, alarm acks, the
/// alerts emit, the alarm-history drain, HistoryUpdate) stayed enabled on a node no client can reach, while
/// the peer that clients CAN reach refused writes as Secondary.
/// </para>
/// <para>
/// <b>How.</b> The Primary is the oldest Up <c>driver</c> member and must stay exactly where
/// <c>ClusterSingletonManager</c> puts the singletons (see <c>RedundancyStateActor.SelectDriverPrimary</c>),
/// so the election is NOT bent to skip this node — that would split the Primary-gated data plane from the
/// singletons. Instead the node gracefully <b>leaves</b> its mesh, the same mechanism as an operator's manual
/// failover: it hands its singletons to the peer, <c>CoordinatedShutdown</c> runs,
/// <c>ActorSystemTerminationWatchdog</c> exits the process, the service supervisor restarts it, and it rejoins
/// as the youngest member — a Secondary whose ServiceLevel is 0 (the endpoint fact also feeds
/// <c>ServiceLevelCalculator</c>). The peer is now the oldest Up driver member, so it is Primary, hosts the
/// singletons, advertises 250, and the <c>/health/active</c> tier follows it.
/// </para>
/// <para>
/// <b>Guards</b> — every one must hold, checked on each tick:
/// </para>
/// <list type="number">
/// <item>this node's endpoint is <see cref="OpcUaEndpointStatus.Failed"/> (not merely
/// <see cref="OpcUaEndpointStatus.NotStarted"/> or <see cref="OpcUaEndpointStatus.Stopped"/>, which are boot
/// and shutdown windows);</item>
/// <item>this node IS the driver Primary, judged by <see cref="ClusterActiveNode.SelfIsActive"/> — the same
/// primitive the election delegates to, so the two cannot disagree;</item>
/// <item>another Up driver member has RECENTLY reported, on <see cref="Topic"/>, that its own endpoint is
/// listening. Without this a pair whose endpoints BOTH failed would bounce the Primary between them forever,
/// restarting each node in turn — and a single node (no peer) would shut itself down for nothing. A peer on
/// an older build never reports, so it never receives the role: that is the safe direction.</item>
/// </list>
/// <para>
/// Every node also publishes its own <see cref="EndpointReport"/> on each tick, which is how its peer learns
/// whether it can take over. The topic is separate from <c>redundancy-state</c> so the half-dozen actors that
/// subscribe there are not handed a message type they do not handle.
/// </para>
/// </remarks>
public sealed class OpcUaEndpointStepDownActor : ReceiveActor, IWithTimers
{
/// <summary>The DistributedPubSub topic every driver node publishes its <see cref="EndpointReport"/> on.</summary>
public const string Topic = "opcua-endpoint-state";
/// <summary>Default period between self-reports and step-down evaluations.</summary>
public static readonly TimeSpan DefaultCheckInterval = TimeSpan.FromSeconds(5);
/// <summary>Default age beyond which a peer's report is ignored (it went away or stopped reporting).</summary>
public static readonly TimeSpan DefaultReportFreshness = TimeSpan.FromSeconds(20);
/// <summary>One node's statement about its own OPC UA endpoint.</summary>
/// <param name="Node">The reporting node, as <c>host:port</c> of its Akka address.</param>
/// <param name="Listening">Whether the node's SDK is started and serving.</param>
public sealed record EndpointReport(NodeId Node, bool Listening);
/// <summary>Periodic self-tick.</summary>
public sealed class Tick
{
/// <summary>The singleton tick instance.</summary>
public static readonly Tick Instance = new();
private Tick() { }
}
private readonly IOpcUaEndpointState _state;
private readonly TimeSpan _interval;
private readonly TimeSpan _freshness;
private readonly Func<DateTime> _utcNow;
private readonly Akka.Cluster.Cluster _cluster = Akka.Cluster.Cluster.Get(Context.System);
private readonly ILoggingAdapter _log = Context.GetLogger();
private readonly Dictionary<NodeId, (bool Listening, DateTime At)> _peerReports = new();
private readonly NodeId _self;
private bool _leaving;
private string? _lastHoldReason;
/// <summary>Gets or sets the timer scheduler.</summary>
public ITimerScheduler Timers { get; set; } = null!;
/// <summary>Creates the actor's props.</summary>
/// <param name="state">This node's OPC UA endpoint lifecycle fact.</param>
/// <param name="checkInterval">Period between self-reports and evaluations; defaults to <see cref="DefaultCheckInterval"/>.</param>
/// <param name="reportFreshness">Peer-report age limit; defaults to <see cref="DefaultReportFreshness"/>.</param>
/// <param name="utcNow">Clock; defaults to <see cref="DateTime.UtcNow"/>.</param>
/// <returns>The props.</returns>
public static Props Props(
IOpcUaEndpointState state,
TimeSpan? checkInterval = null,
TimeSpan? reportFreshness = null,
Func<DateTime>? utcNow = null) =>
Akka.Actor.Props.Create(() => new OpcUaEndpointStepDownActor(state, checkInterval, reportFreshness, utcNow));
/// <summary>Initializes a new instance of the <see cref="OpcUaEndpointStepDownActor"/> class.</summary>
/// <param name="state">This node's OPC UA endpoint lifecycle fact.</param>
/// <param name="checkInterval">Period between self-reports and evaluations.</param>
/// <param name="reportFreshness">Peer-report age limit.</param>
/// <param name="utcNow">Clock.</param>
public OpcUaEndpointStepDownActor(
IOpcUaEndpointState state,
TimeSpan? checkInterval,
TimeSpan? reportFreshness,
Func<DateTime>? utcNow)
{
_state = state ?? throw new ArgumentNullException(nameof(state));
_interval = checkInterval ?? DefaultCheckInterval;
_freshness = reportFreshness ?? DefaultReportFreshness;
_utcNow = utcNow ?? (() => DateTime.UtcNow);
_self = ToNodeId(_cluster.SelfAddress);
Receive<Tick>(_ => OnTick());
Receive<EndpointReport>(OnReport);
Receive<SubscribeAck>(_ => { });
}
/// <summary>
/// The canonical <c>host:port</c> id of an Akka address — the format <c>RedundancyStateActor.ToNodeId</c> and
/// <c>ClusterRoleInfo</c> use. Derived from the member address on both sides (reporter and reader), so the two
/// always agree.
/// </summary>
/// <param name="address">The member address.</param>
/// <returns>The node id.</returns>
public static NodeId ToNodeId(Address address) =>
NodeId.Parse($"{address.Host ?? string.Empty}:{address.Port ?? 0}");
/// <inheritdoc />
protected override void PreStart()
{
DistributedPubSub.Get(Context.System).Mediator.Tell(new Subscribe(Topic, Self));
Self.Tell(Tick.Instance);
Timers.StartPeriodicTimer("tick", Tick.Instance, _interval);
}
private void OnTick()
{
var listening = _state.Status == OpcUaEndpointStatus.Started;
try
{
DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(Topic, new EndpointReport(_self, listening)));
}
catch (Exception ex)
{
_log.Debug(ex, "OpcUaEndpointStepDown: could not publish this node's endpoint report");
}
Evaluate();
}
private void OnReport(EndpointReport report)
{
if (report.Node == _self) return; // our own publish echoed back
_peerReports[report.Node] = (report.Listening, _utcNow());
Evaluate();
}
private void Evaluate()
{
if (_leaving) return;
if (_state.Status != OpcUaEndpointStatus.Failed)
{
_lastHoldReason = null;
return;
}
bool isPrimary;
try
{
isPrimary = ClusterActiveNode.SelfIsActive(_cluster, RoleParser.Driver);
}
catch (Exception ex)
{
_log.Debug(ex, "OpcUaEndpointStepDown: cluster state unavailable; not evaluating");
return;
}
if (!isPrimary)
{
HoldOnce("not the driver Primary — nothing to step down from");
return;
}
var now = _utcNow();
var self = _cluster.SelfMember.UniqueAddress;
// Ordinal Member.HasRole on purpose — the same match ClusterActiveNode (and so the election) applies, so a
// candidate here is a node the election would actually name Primary once this one leaves.
var takeover = _cluster.State.Members
.Where(m => m.Status == MemberStatus.Up
&& m.HasRole(RoleParser.Driver)
&& !m.UniqueAddress.Equals(self))
.Select(m => (NodeId?)ToNodeId(m.Address))
.FirstOrDefault(id => _peerReports.TryGetValue(id!.Value, out var r) && r.Listening && now - r.At <= _freshness);
if (takeover is not { } peer)
{
HoldOnce("no Up driver peer reports a listening OPC UA endpoint — staying Primary rather than handing the role to a node that cannot serve either");
return;
}
_leaving = true;
_log.Warning(
"OpcUaEndpointStepDown: this node is the driver Primary but its OPC UA endpoint failed to start ({Reason}); "
+ "peer {Peer} reports a listening endpoint. Leaving the cluster so {Peer} becomes Primary — this node hands "
+ "over its singletons, exits, and is restarted by its supervisor as a Secondary (lmxopcua#762).",
_state.FailureReason ?? "no reason recorded", peer, peer);
_cluster.Leave(_cluster.SelfAddress);
}
private void HoldOnce(string reason)
{
if (reason == _lastHoldReason) return;
_lastHoldReason = reason;
_log.Warning("OpcUaEndpointStepDown: OPC UA endpoint failed to start ({Failure}); {Reason}",
_state.FailureReason ?? "no reason recorded", reason);
}
}
@@ -44,6 +44,7 @@ public static class ServiceCollectionExtensions
public const string DependencyMuxActorName = "dependency-mux";
public const string OpcUaPublishActorName = "opcua-publish";
public const string PeerProbeSupervisorName = "peer-probe-supervisor";
public const string OpcUaEndpointStepDownActorName = "opcua-endpoint-step-down";
public const string ContinuousHistorizationRecorderActorName = "continuous-historization-recorder";
/// <summary>
@@ -555,6 +556,10 @@ public static class ServiceCollectionExtensions
// a node with no driver role never registers it and test harnesses build this graph without it;
// the actor treats null as "nothing consumes authorization inputs" and skips the decode.
var dataPlaneAuthority = resolver.GetService<IDataPlaneAuthoritySink>();
// lmxopcua#762: the "did the OPC UA SDK start?" fact the Host's OtOpcUaServerHostedService maintains.
// Optional: a harness that builds this graph without the hosted service registers none, and both
// consumers then behave as if the endpoint were listening (the pre-#762 behaviour).
var endpointState = resolver.GetService<IOpcUaEndpointState>();
var publishActor = system.ActorOf(
OpcUaPublishActor.Props(
sink: addressSpaceSink,
@@ -564,10 +569,19 @@ public static class ServiceCollectionExtensions
applier: applier,
dbHealthProbe: dbHealth,
dbLess: dbFactory is null,
dataPlaneAuthority: dataPlaneAuthority),
dataPlaneAuthority: dataPlaneAuthority,
endpointState: endpointState),
OpcUaPublishActorName);
registry.Register<OpcUaPublishActorKey>(publishActor);
// lmxopcua#762: a driver Primary whose OPC UA endpoint failed to start leaves its mesh once a peer
// reports a listening endpoint, so the peer becomes Primary. Every driver node runs one — it is also
// how this node tells its peer whether IT can take over.
if (endpointState is not null)
{
system.ActorOf(OpcUaEndpointStepDownActor.Props(endpointState), OpcUaEndpointStepDownActorName);
}
// Per-node peer-probe supervisor — keeps one OPC UA TCP probe per OTHER non-Detached
// driver node, so every node is continuously probed by all its peers. OpcUaPublishActor
// consumes the resulting probe verdicts to learn this node's own reachability.
@@ -23,6 +23,10 @@
<!-- Core LocalDb only (ILocalDb + the connection/transaction seam) — the deployment-artifact
cache lives here, but the sync engine and its gRPC endpoint are the Host's concern. -->
<PackageReference Include="ZB.MOM.WW.LocalDb" />
<!-- ClusterActiveNode — the same "oldest Up member of role X" primitive RedundancyStateActor's election
delegates to. OpcUaEndpointStepDownActor (lmxopcua#762) asks "am I the driver Primary?" through it so
the step-down and the election cannot disagree about which node is Primary. -->
<PackageReference Include="ZB.MOM.WW.Health.Akka" />
</ItemGroup>
<ItemGroup>