fix(redundancy): a failed OPC UA endpoint drops ServiceLevel to 0 and steps the Primary down (lmxopcua#762)
v2-ci / unit-tests (push) Has been cancelled
v2-ci / unit-tests (push) Has been cancelled
IOpcUaEndpointState (review-remediation lane C3) recorded whether the SDK started, but only the health check read it: a fused node whose endpoint failed (port 4840 taken, ...) still advertised 250/240 and stayed driver Primary, so the Primary-gated data plane ran on a node no client could reach while the reachable peer refused writes as Secondary. - Move IOpcUaEndpointState/OpcUaEndpointState/OpcUaEndpointStatus from Host to Commons.OpcUa so the Runtime can consume it. - ServiceLevelCalculator: new NodeHealthInputs.EndpointListening (default true). false -> 0, ahead of every other input including the +10 Primary bonus. - OpcUaPublishActor: takes the endpoint state, feeds EndpointListening (Failed only; NotStarted / Stopped are boot/shutdown windows) on the DB-backed and DB-less paths, and the legacy role-only seam returns 0 for a failed endpoint too. - New OpcUaEndpointStepDownActor (one per driver node): a Primary (ClusterActiveNode.SelfIsActive, the election's own primitive) with a Failed endpoint gracefully Leaves its mesh - the manual failover mechanism - so the peer becomes the oldest Up driver member, hosts the singletons and is Primary. The election itself is not bent. Guarded on a peer having recently reported its own endpoint listening (EndpointReport on the new opcua-endpoint-state DPS topic), so a pair with both endpoints failed does not bounce the role and restart nodes forever, and a lone node never shuts itself down. - Runtime now references ZB.MOM.WW.Health.Akka (ClusterActiveNode). - Docs: Redundancy.md tier table + new "Endpoint start failure: Primary step-down" section; cluster-topology rule; HealthEndpoints / OpcUaEndpointHealthCheck comments (F-C1 landed). Tests: ServiceLevelCalculatorTests (+5), OpcUaPublishActorEndpointStateTests (7: DB-less, DB-backed, legacy seam, NotStarted/Stopped do not demote, a later failure demotes on the next tick), OpcUaEndpointStepDownActorTests (3, real two-node mesh: holds while the peer is not listening then leaves and the peer becomes Primary; failed Secondary and healthy Primary never leave). Each was mutation-checked: removing the calculator/legacy guard, the Leave, the peer-listening guard or the Primary guard fails the corresponding tests.
This commit is contained in:
@@ -0,0 +1,226 @@
|
||||
using Akka.Actor;
|
||||
using Akka.Cluster;
|
||||
using Akka.Cluster.Tools.PublishSubscribe;
|
||||
using Akka.Event;
|
||||
using ZB.MOM.WW.Health.Akka;
|
||||
using ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Types;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Runtime.Redundancy;
|
||||
|
||||
/// <summary>
|
||||
/// Steps this node down from driver Primary while its own OPC UA endpoint is not listening (lmxopcua#762).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Why.</b> A fused admin+driver node whose SDK failed to start (port 4840 taken, PKI unwritable, a bad
|
||||
/// <c>OpcUa:*</c> override) keeps running so the operator still has the AdminUI. Before this actor it also
|
||||
/// stayed Primary: the oldest Up driver member, so the Primary-gated data plane (writes, alarm acks, the
|
||||
/// alerts emit, the alarm-history drain, HistoryUpdate) stayed enabled on a node no client can reach, while
|
||||
/// the peer that clients CAN reach refused writes as Secondary.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>How.</b> The Primary is the oldest Up <c>driver</c> member and must stay exactly where
|
||||
/// <c>ClusterSingletonManager</c> puts the singletons (see <c>RedundancyStateActor.SelectDriverPrimary</c>),
|
||||
/// so the election is NOT bent to skip this node — that would split the Primary-gated data plane from the
|
||||
/// singletons. Instead the node gracefully <b>leaves</b> its mesh, the same mechanism as an operator's manual
|
||||
/// failover: it hands its singletons to the peer, <c>CoordinatedShutdown</c> runs,
|
||||
/// <c>ActorSystemTerminationWatchdog</c> exits the process, the service supervisor restarts it, and it rejoins
|
||||
/// as the youngest member — a Secondary whose ServiceLevel is 0 (the endpoint fact also feeds
|
||||
/// <c>ServiceLevelCalculator</c>). The peer is now the oldest Up driver member, so it is Primary, hosts the
|
||||
/// singletons, advertises 250, and the <c>/health/active</c> tier follows it.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Guards</b> — every one must hold, checked on each tick:
|
||||
/// </para>
|
||||
/// <list type="number">
|
||||
/// <item>this node's endpoint is <see cref="OpcUaEndpointStatus.Failed"/> (not merely
|
||||
/// <see cref="OpcUaEndpointStatus.NotStarted"/> or <see cref="OpcUaEndpointStatus.Stopped"/>, which are boot
|
||||
/// and shutdown windows);</item>
|
||||
/// <item>this node IS the driver Primary, judged by <see cref="ClusterActiveNode.SelfIsActive"/> — the same
|
||||
/// primitive the election delegates to, so the two cannot disagree;</item>
|
||||
/// <item>another Up driver member has RECENTLY reported, on <see cref="Topic"/>, that its own endpoint is
|
||||
/// listening. Without this a pair whose endpoints BOTH failed would bounce the Primary between them forever,
|
||||
/// restarting each node in turn — and a single node (no peer) would shut itself down for nothing. A peer on
|
||||
/// an older build never reports, so it never receives the role: that is the safe direction.</item>
|
||||
/// </list>
|
||||
/// <para>
|
||||
/// Every node also publishes its own <see cref="EndpointReport"/> on each tick, which is how its peer learns
|
||||
/// whether it can take over. The topic is separate from <c>redundancy-state</c> so the half-dozen actors that
|
||||
/// subscribe there are not handed a message type they do not handle.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class OpcUaEndpointStepDownActor : ReceiveActor, IWithTimers
|
||||
{
|
||||
/// <summary>The DistributedPubSub topic every driver node publishes its <see cref="EndpointReport"/> on.</summary>
|
||||
public const string Topic = "opcua-endpoint-state";
|
||||
|
||||
/// <summary>Default period between self-reports and step-down evaluations.</summary>
|
||||
public static readonly TimeSpan DefaultCheckInterval = TimeSpan.FromSeconds(5);
|
||||
|
||||
/// <summary>Default age beyond which a peer's report is ignored (it went away or stopped reporting).</summary>
|
||||
public static readonly TimeSpan DefaultReportFreshness = TimeSpan.FromSeconds(20);
|
||||
|
||||
/// <summary>One node's statement about its own OPC UA endpoint.</summary>
|
||||
/// <param name="Node">The reporting node, as <c>host:port</c> of its Akka address.</param>
|
||||
/// <param name="Listening">Whether the node's SDK is started and serving.</param>
|
||||
public sealed record EndpointReport(NodeId Node, bool Listening);
|
||||
|
||||
/// <summary>Periodic self-tick.</summary>
|
||||
public sealed class Tick
|
||||
{
|
||||
/// <summary>The singleton tick instance.</summary>
|
||||
public static readonly Tick Instance = new();
|
||||
private Tick() { }
|
||||
}
|
||||
|
||||
private readonly IOpcUaEndpointState _state;
|
||||
private readonly TimeSpan _interval;
|
||||
private readonly TimeSpan _freshness;
|
||||
private readonly Func<DateTime> _utcNow;
|
||||
private readonly Akka.Cluster.Cluster _cluster = Akka.Cluster.Cluster.Get(Context.System);
|
||||
private readonly ILoggingAdapter _log = Context.GetLogger();
|
||||
private readonly Dictionary<NodeId, (bool Listening, DateTime At)> _peerReports = new();
|
||||
private readonly NodeId _self;
|
||||
private bool _leaving;
|
||||
private string? _lastHoldReason;
|
||||
|
||||
/// <summary>Gets or sets the timer scheduler.</summary>
|
||||
public ITimerScheduler Timers { get; set; } = null!;
|
||||
|
||||
/// <summary>Creates the actor's props.</summary>
|
||||
/// <param name="state">This node's OPC UA endpoint lifecycle fact.</param>
|
||||
/// <param name="checkInterval">Period between self-reports and evaluations; defaults to <see cref="DefaultCheckInterval"/>.</param>
|
||||
/// <param name="reportFreshness">Peer-report age limit; defaults to <see cref="DefaultReportFreshness"/>.</param>
|
||||
/// <param name="utcNow">Clock; defaults to <see cref="DateTime.UtcNow"/>.</param>
|
||||
/// <returns>The props.</returns>
|
||||
public static Props Props(
|
||||
IOpcUaEndpointState state,
|
||||
TimeSpan? checkInterval = null,
|
||||
TimeSpan? reportFreshness = null,
|
||||
Func<DateTime>? utcNow = null) =>
|
||||
Akka.Actor.Props.Create(() => new OpcUaEndpointStepDownActor(state, checkInterval, reportFreshness, utcNow));
|
||||
|
||||
/// <summary>Initializes a new instance of the <see cref="OpcUaEndpointStepDownActor"/> class.</summary>
|
||||
/// <param name="state">This node's OPC UA endpoint lifecycle fact.</param>
|
||||
/// <param name="checkInterval">Period between self-reports and evaluations.</param>
|
||||
/// <param name="reportFreshness">Peer-report age limit.</param>
|
||||
/// <param name="utcNow">Clock.</param>
|
||||
public OpcUaEndpointStepDownActor(
|
||||
IOpcUaEndpointState state,
|
||||
TimeSpan? checkInterval,
|
||||
TimeSpan? reportFreshness,
|
||||
Func<DateTime>? utcNow)
|
||||
{
|
||||
_state = state ?? throw new ArgumentNullException(nameof(state));
|
||||
_interval = checkInterval ?? DefaultCheckInterval;
|
||||
_freshness = reportFreshness ?? DefaultReportFreshness;
|
||||
_utcNow = utcNow ?? (() => DateTime.UtcNow);
|
||||
_self = ToNodeId(_cluster.SelfAddress);
|
||||
|
||||
Receive<Tick>(_ => OnTick());
|
||||
Receive<EndpointReport>(OnReport);
|
||||
Receive<SubscribeAck>(_ => { });
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The canonical <c>host:port</c> id of an Akka address — the format <c>RedundancyStateActor.ToNodeId</c> and
|
||||
/// <c>ClusterRoleInfo</c> use. Derived from the member address on both sides (reporter and reader), so the two
|
||||
/// always agree.
|
||||
/// </summary>
|
||||
/// <param name="address">The member address.</param>
|
||||
/// <returns>The node id.</returns>
|
||||
public static NodeId ToNodeId(Address address) =>
|
||||
NodeId.Parse($"{address.Host ?? string.Empty}:{address.Port ?? 0}");
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void PreStart()
|
||||
{
|
||||
DistributedPubSub.Get(Context.System).Mediator.Tell(new Subscribe(Topic, Self));
|
||||
Self.Tell(Tick.Instance);
|
||||
Timers.StartPeriodicTimer("tick", Tick.Instance, _interval);
|
||||
}
|
||||
|
||||
private void OnTick()
|
||||
{
|
||||
var listening = _state.Status == OpcUaEndpointStatus.Started;
|
||||
try
|
||||
{
|
||||
DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(Topic, new EndpointReport(_self, listening)));
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_log.Debug(ex, "OpcUaEndpointStepDown: could not publish this node's endpoint report");
|
||||
}
|
||||
|
||||
Evaluate();
|
||||
}
|
||||
|
||||
private void OnReport(EndpointReport report)
|
||||
{
|
||||
if (report.Node == _self) return; // our own publish echoed back
|
||||
_peerReports[report.Node] = (report.Listening, _utcNow());
|
||||
Evaluate();
|
||||
}
|
||||
|
||||
private void Evaluate()
|
||||
{
|
||||
if (_leaving) return;
|
||||
if (_state.Status != OpcUaEndpointStatus.Failed)
|
||||
{
|
||||
_lastHoldReason = null;
|
||||
return;
|
||||
}
|
||||
|
||||
bool isPrimary;
|
||||
try
|
||||
{
|
||||
isPrimary = ClusterActiveNode.SelfIsActive(_cluster, RoleParser.Driver);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_log.Debug(ex, "OpcUaEndpointStepDown: cluster state unavailable; not evaluating");
|
||||
return;
|
||||
}
|
||||
|
||||
if (!isPrimary)
|
||||
{
|
||||
HoldOnce("not the driver Primary — nothing to step down from");
|
||||
return;
|
||||
}
|
||||
|
||||
var now = _utcNow();
|
||||
var self = _cluster.SelfMember.UniqueAddress;
|
||||
// Ordinal Member.HasRole on purpose — the same match ClusterActiveNode (and so the election) applies, so a
|
||||
// candidate here is a node the election would actually name Primary once this one leaves.
|
||||
var takeover = _cluster.State.Members
|
||||
.Where(m => m.Status == MemberStatus.Up
|
||||
&& m.HasRole(RoleParser.Driver)
|
||||
&& !m.UniqueAddress.Equals(self))
|
||||
.Select(m => (NodeId?)ToNodeId(m.Address))
|
||||
.FirstOrDefault(id => _peerReports.TryGetValue(id!.Value, out var r) && r.Listening && now - r.At <= _freshness);
|
||||
|
||||
if (takeover is not { } peer)
|
||||
{
|
||||
HoldOnce("no Up driver peer reports a listening OPC UA endpoint — staying Primary rather than handing the role to a node that cannot serve either");
|
||||
return;
|
||||
}
|
||||
|
||||
_leaving = true;
|
||||
_log.Warning(
|
||||
"OpcUaEndpointStepDown: this node is the driver Primary but its OPC UA endpoint failed to start ({Reason}); "
|
||||
+ "peer {Peer} reports a listening endpoint. Leaving the cluster so {Peer} becomes Primary — this node hands "
|
||||
+ "over its singletons, exits, and is restarted by its supervisor as a Secondary (lmxopcua#762).",
|
||||
_state.FailureReason ?? "no reason recorded", peer, peer);
|
||||
_cluster.Leave(_cluster.SelfAddress);
|
||||
}
|
||||
|
||||
private void HoldOnce(string reason)
|
||||
{
|
||||
if (reason == _lastHoldReason) return;
|
||||
_lastHoldReason = reason;
|
||||
_log.Warning("OpcUaEndpointStepDown: OPC UA endpoint failed to start ({Failure}); {Reason}",
|
||||
_state.FailureReason ?? "no reason recorded", reason);
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user