fix(redundancy): a failed OPC UA endpoint drops ServiceLevel to 0 and steps the Primary down (lmxopcua#762)
v2-ci / unit-tests (push) Has been cancelled

IOpcUaEndpointState (review-remediation lane C3) recorded whether the SDK started, but only the
health check read it: a fused node whose endpoint failed (port 4840 taken, ...) still advertised
250/240 and stayed driver Primary, so the Primary-gated data plane ran on a node no client could
reach while the reachable peer refused writes as Secondary.

- Move IOpcUaEndpointState/OpcUaEndpointState/OpcUaEndpointStatus from Host to Commons.OpcUa so
  the Runtime can consume it.
- ServiceLevelCalculator: new NodeHealthInputs.EndpointListening (default true). false -> 0,
  ahead of every other input including the +10 Primary bonus.
- OpcUaPublishActor: takes the endpoint state, feeds EndpointListening (Failed only; NotStarted /
  Stopped are boot/shutdown windows) on the DB-backed and DB-less paths, and the legacy role-only
  seam returns 0 for a failed endpoint too.
- New OpcUaEndpointStepDownActor (one per driver node): a Primary (ClusterActiveNode.SelfIsActive,
  the election's own primitive) with a Failed endpoint gracefully Leaves its mesh - the manual
  failover mechanism - so the peer becomes the oldest Up driver member, hosts the singletons and is
  Primary. The election itself is not bent. Guarded on a peer having recently reported its own
  endpoint listening (EndpointReport on the new opcua-endpoint-state DPS topic), so a pair with
  both endpoints failed does not bounce the role and restart nodes forever, and a lone node never
  shuts itself down.
- Runtime now references ZB.MOM.WW.Health.Akka (ClusterActiveNode).
- Docs: Redundancy.md tier table + new "Endpoint start failure: Primary step-down" section;
  cluster-topology rule; HealthEndpoints / OpcUaEndpointHealthCheck comments (F-C1 landed).

Tests: ServiceLevelCalculatorTests (+5), OpcUaPublishActorEndpointStateTests (7: DB-less,
DB-backed, legacy seam, NotStarted/Stopped do not demote, a later failure demotes on the next
tick), OpcUaEndpointStepDownActorTests (3, real two-node mesh: holds while the peer is not
listening then leaves and the peer becomes Primary; failed Secondary and healthy Primary never
leave). Each was mutation-checked: removing the calculator/legacy guard, the Leave, the
peer-listening guard or the Primary guard fails the corresponding tests.
This commit is contained in:
Joseph Doherty
2026-11-09 12:15:59 -05:00
parent 8ef5adc63a
commit acfb4fbaca
13 changed files with 668 additions and 30 deletions
@@ -0,0 +1,226 @@
using Akka.Actor;
using Akka.Cluster;
using Akka.Cluster.Tools.PublishSubscribe;
using Akka.Event;
using ZB.MOM.WW.Health.Akka;
using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Redundancy;
/// <summary>
/// Steps this node down from driver Primary while its own OPC UA endpoint is not listening (lmxopcua#762).
/// </summary>
/// <remarks>
/// <para>
/// <b>Why.</b> A fused admin+driver node whose SDK failed to start (port 4840 taken, PKI unwritable, a bad
/// <c>OpcUa:*</c> override) keeps running so the operator still has the AdminUI. Before this actor it also
/// stayed Primary: the oldest Up driver member, so the Primary-gated data plane (writes, alarm acks, the
/// alerts emit, the alarm-history drain, HistoryUpdate) stayed enabled on a node no client can reach, while
/// the peer that clients CAN reach refused writes as Secondary.
/// </para>
/// <para>
/// <b>How.</b> The Primary is the oldest Up <c>driver</c> member and must stay exactly where
/// <c>ClusterSingletonManager</c> puts the singletons (see <c>RedundancyStateActor.SelectDriverPrimary</c>),
/// so the election is NOT bent to skip this node — that would split the Primary-gated data plane from the
/// singletons. Instead the node gracefully <b>leaves</b> its mesh, the same mechanism as an operator's manual
/// failover: it hands its singletons to the peer, <c>CoordinatedShutdown</c> runs,
/// <c>ActorSystemTerminationWatchdog</c> exits the process, the service supervisor restarts it, and it rejoins
/// as the youngest member — a Secondary whose ServiceLevel is 0 (the endpoint fact also feeds
/// <c>ServiceLevelCalculator</c>). The peer is now the oldest Up driver member, so it is Primary, hosts the
/// singletons, advertises 250, and the <c>/health/active</c> tier follows it.
/// </para>
/// <para>
/// <b>Guards</b> — every one must hold, checked on each tick:
/// </para>
/// <list type="number">
/// <item>this node's endpoint is <see cref="OpcUaEndpointStatus.Failed"/> (not merely
/// <see cref="OpcUaEndpointStatus.NotStarted"/> or <see cref="OpcUaEndpointStatus.Stopped"/>, which are boot
/// and shutdown windows);</item>
/// <item>this node IS the driver Primary, judged by <see cref="ClusterActiveNode.SelfIsActive"/> — the same
/// primitive the election delegates to, so the two cannot disagree;</item>
/// <item>another Up driver member has RECENTLY reported, on <see cref="Topic"/>, that its own endpoint is
/// listening. Without this a pair whose endpoints BOTH failed would bounce the Primary between them forever,
/// restarting each node in turn — and a single node (no peer) would shut itself down for nothing. A peer on
/// an older build never reports, so it never receives the role: that is the safe direction.</item>
/// </list>
/// <para>
/// Every node also publishes its own <see cref="EndpointReport"/> on each tick, which is how its peer learns
/// whether it can take over. The topic is separate from <c>redundancy-state</c> so the half-dozen actors that
/// subscribe there are not handed a message type they do not handle.
/// </para>
/// </remarks>
public sealed class OpcUaEndpointStepDownActor : ReceiveActor, IWithTimers
{
/// <summary>The DistributedPubSub topic every driver node publishes its <see cref="EndpointReport"/> on.</summary>
public const string Topic = "opcua-endpoint-state";
/// <summary>Default period between self-reports and step-down evaluations.</summary>
public static readonly TimeSpan DefaultCheckInterval = TimeSpan.FromSeconds(5);
/// <summary>Default age beyond which a peer's report is ignored (it went away or stopped reporting).</summary>
public static readonly TimeSpan DefaultReportFreshness = TimeSpan.FromSeconds(20);
/// <summary>One node's statement about its own OPC UA endpoint.</summary>
/// <param name="Node">The reporting node, as <c>host:port</c> of its Akka address.</param>
/// <param name="Listening">Whether the node's SDK is started and serving.</param>
public sealed record EndpointReport(NodeId Node, bool Listening);
/// <summary>Periodic self-tick.</summary>
public sealed class Tick
{
/// <summary>The singleton tick instance.</summary>
public static readonly Tick Instance = new();
private Tick() { }
}
private readonly IOpcUaEndpointState _state;
private readonly TimeSpan _interval;
private readonly TimeSpan _freshness;
private readonly Func<DateTime> _utcNow;
private readonly Akka.Cluster.Cluster _cluster = Akka.Cluster.Cluster.Get(Context.System);
private readonly ILoggingAdapter _log = Context.GetLogger();
private readonly Dictionary<NodeId, (bool Listening, DateTime At)> _peerReports = new();
private readonly NodeId _self;
private bool _leaving;
private string? _lastHoldReason;
/// <summary>Gets or sets the timer scheduler.</summary>
public ITimerScheduler Timers { get; set; } = null!;
/// <summary>Creates the actor's props.</summary>
/// <param name="state">This node's OPC UA endpoint lifecycle fact.</param>
/// <param name="checkInterval">Period between self-reports and evaluations; defaults to <see cref="DefaultCheckInterval"/>.</param>
/// <param name="reportFreshness">Peer-report age limit; defaults to <see cref="DefaultReportFreshness"/>.</param>
/// <param name="utcNow">Clock; defaults to <see cref="DateTime.UtcNow"/>.</param>
/// <returns>The props.</returns>
public static Props Props(
IOpcUaEndpointState state,
TimeSpan? checkInterval = null,
TimeSpan? reportFreshness = null,
Func<DateTime>? utcNow = null) =>
Akka.Actor.Props.Create(() => new OpcUaEndpointStepDownActor(state, checkInterval, reportFreshness, utcNow));
/// <summary>Initializes a new instance of the <see cref="OpcUaEndpointStepDownActor"/> class.</summary>
/// <param name="state">This node's OPC UA endpoint lifecycle fact.</param>
/// <param name="checkInterval">Period between self-reports and evaluations.</param>
/// <param name="reportFreshness">Peer-report age limit.</param>
/// <param name="utcNow">Clock.</param>
public OpcUaEndpointStepDownActor(
IOpcUaEndpointState state,
TimeSpan? checkInterval,
TimeSpan? reportFreshness,
Func<DateTime>? utcNow)
{
_state = state ?? throw new ArgumentNullException(nameof(state));
_interval = checkInterval ?? DefaultCheckInterval;
_freshness = reportFreshness ?? DefaultReportFreshness;
_utcNow = utcNow ?? (() => DateTime.UtcNow);
_self = ToNodeId(_cluster.SelfAddress);
Receive<Tick>(_ => OnTick());
Receive<EndpointReport>(OnReport);
Receive<SubscribeAck>(_ => { });
}
/// <summary>
/// The canonical <c>host:port</c> id of an Akka address — the format <c>RedundancyStateActor.ToNodeId</c> and
/// <c>ClusterRoleInfo</c> use. Derived from the member address on both sides (reporter and reader), so the two
/// always agree.
/// </summary>
/// <param name="address">The member address.</param>
/// <returns>The node id.</returns>
public static NodeId ToNodeId(Address address) =>
NodeId.Parse($"{address.Host ?? string.Empty}:{address.Port ?? 0}");
/// <inheritdoc />
protected override void PreStart()
{
DistributedPubSub.Get(Context.System).Mediator.Tell(new Subscribe(Topic, Self));
Self.Tell(Tick.Instance);
Timers.StartPeriodicTimer("tick", Tick.Instance, _interval);
}
private void OnTick()
{
var listening = _state.Status == OpcUaEndpointStatus.Started;
try
{
DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(Topic, new EndpointReport(_self, listening)));
}
catch (Exception ex)
{
_log.Debug(ex, "OpcUaEndpointStepDown: could not publish this node's endpoint report");
}
Evaluate();
}
private void OnReport(EndpointReport report)
{
if (report.Node == _self) return; // our own publish echoed back
_peerReports[report.Node] = (report.Listening, _utcNow());
Evaluate();
}
private void Evaluate()
{
if (_leaving) return;
if (_state.Status != OpcUaEndpointStatus.Failed)
{
_lastHoldReason = null;
return;
}
bool isPrimary;
try
{
isPrimary = ClusterActiveNode.SelfIsActive(_cluster, RoleParser.Driver);
}
catch (Exception ex)
{
_log.Debug(ex, "OpcUaEndpointStepDown: cluster state unavailable; not evaluating");
return;
}
if (!isPrimary)
{
HoldOnce("not the driver Primary — nothing to step down from");
return;
}
var now = _utcNow();
var self = _cluster.SelfMember.UniqueAddress;
// Ordinal Member.HasRole on purpose — the same match ClusterActiveNode (and so the election) applies, so a
// candidate here is a node the election would actually name Primary once this one leaves.
var takeover = _cluster.State.Members
.Where(m => m.Status == MemberStatus.Up
&& m.HasRole(RoleParser.Driver)
&& !m.UniqueAddress.Equals(self))
.Select(m => (NodeId?)ToNodeId(m.Address))
.FirstOrDefault(id => _peerReports.TryGetValue(id!.Value, out var r) && r.Listening && now - r.At <= _freshness);
if (takeover is not { } peer)
{
HoldOnce("no Up driver peer reports a listening OPC UA endpoint — staying Primary rather than handing the role to a node that cannot serve either");
return;
}
_leaving = true;
_log.Warning(
"OpcUaEndpointStepDown: this node is the driver Primary but its OPC UA endpoint failed to start ({Reason}); "
+ "peer {Peer} reports a listening endpoint. Leaving the cluster so {Peer} becomes Primary — this node hands "
+ "over its singletons, exits, and is restarted by its supervisor as a Secondary (lmxopcua#762).",
_state.FailureReason ?? "no reason recorded", peer, peer);
_cluster.Leave(_cluster.SelfAddress);
}
private void HoldOnce(string reason)
{
if (reason == _lastHoldReason) return;
_lastHoldReason = reason;
_log.Warning("OpcUaEndpointStepDown: OPC UA endpoint failed to start ({Failure}); {Reason}",
_state.FailureReason ?? "no reason recorded", reason);
}
}