fix(redundancy): a failed OPC UA endpoint drops ServiceLevel to 0 and steps the Primary down (lmxopcua#762)
v2-ci / unit-tests (push) Has been cancelled
v2-ci / unit-tests (push) Has been cancelled
IOpcUaEndpointState (review-remediation lane C3) recorded whether the SDK started, but only the health check read it: a fused node whose endpoint failed (port 4840 taken, ...) still advertised 250/240 and stayed driver Primary, so the Primary-gated data plane ran on a node no client could reach while the reachable peer refused writes as Secondary. - Move IOpcUaEndpointState/OpcUaEndpointState/OpcUaEndpointStatus from Host to Commons.OpcUa so the Runtime can consume it. - ServiceLevelCalculator: new NodeHealthInputs.EndpointListening (default true). false -> 0, ahead of every other input including the +10 Primary bonus. - OpcUaPublishActor: takes the endpoint state, feeds EndpointListening (Failed only; NotStarted / Stopped are boot/shutdown windows) on the DB-backed and DB-less paths, and the legacy role-only seam returns 0 for a failed endpoint too. - New OpcUaEndpointStepDownActor (one per driver node): a Primary (ClusterActiveNode.SelfIsActive, the election's own primitive) with a Failed endpoint gracefully Leaves its mesh - the manual failover mechanism - so the peer becomes the oldest Up driver member, hosts the singletons and is Primary. The election itself is not bent. Guarded on a peer having recently reported its own endpoint listening (EndpointReport on the new opcua-endpoint-state DPS topic), so a pair with both endpoints failed does not bounce the role and restart nodes forever, and a lone node never shuts itself down. - Runtime now references ZB.MOM.WW.Health.Akka (ClusterActiveNode). - Docs: Redundancy.md tier table + new "Endpoint start failure: Primary step-down" section; cluster-topology rule; HealthEndpoints / OpcUaEndpointHealthCheck comments (F-C1 landed). Tests: ServiceLevelCalculatorTests (+5), OpcUaPublishActorEndpointStateTests (7: DB-less, DB-backed, legacy seam, NotStarted/Stopped do not demote, a later failure demotes on the next tick), OpcUaEndpointStepDownActorTests (3, real two-node mesh: holds while the peer is not listening then leaves and the peer becomes Primary; failed Secondary and healthy Primary never leave). Each was mutation-checked: removing the calculator/legacy guard, the Leave, the peer-listening guard or the Primary guard fails the corresponding tests.
This commit is contained in:
@@ -11,7 +11,7 @@ using ZB.MOM.WW.LocalDb.Replication;
|
||||
using ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration;
|
||||
using ZB.MOM.WW.OtOpcUa.Host.Configuration;
|
||||
using ZB.MOM.WW.OtOpcUa.Host.OpcUa;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.Health;
|
||||
|
||||
@@ -45,11 +45,12 @@ public static class HealthEndpoints
|
||||
// The ACTIVE tier depends on the roles (coordinator ruling, fix round 1 / I6):
|
||||
// - driver-only: Active too — a site node's whole purpose is its OPC UA endpoint, so nothing may
|
||||
// route to it;
|
||||
// - fused admin+driver: NOT Active, until the ServiceLevel calculator consumes IOpcUaEndpointState
|
||||
// (follow-up F-C1). Traefik routes the AdminUI on /health/active, so putting the probe there would
|
||||
// take the operator's only console out of rotation precisely when they need it to diagnose the
|
||||
// failed endpoint, while the OPC UA failover (ServiceLevel) still would not move. Once F-C1 lands
|
||||
// the peer takes Primary and the active tier follows it on its own.
|
||||
// - fused admin+driver: NOT Active. Traefik routes the AdminUI on /health/active, so putting the probe
|
||||
// there would take the operator's only console out of rotation precisely when they need it to
|
||||
// diagnose the failed endpoint. The OPC UA failover is handled by the Runtime instead (F-C1,
|
||||
// lmxopcua#762): the failed node computes ServiceLevel 0 and, when it is the Primary and its peer
|
||||
// reports a listening endpoint, OpcUaEndpointStepDownActor makes it leave its mesh — the peer becomes
|
||||
// the oldest Up member, so the active tier follows it on its own.
|
||||
// Factory-registered so the state singleton is resolved per probe.
|
||||
if (hasDriver)
|
||||
{
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
using Microsoft.Extensions.Diagnostics.HealthChecks;
|
||||
using ZB.MOM.WW.OtOpcUa.Host.OpcUa;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.Health;
|
||||
|
||||
@@ -12,8 +12,9 @@ namespace ZB.MOM.WW.OtOpcUa.Host.Health;
|
||||
/// <remarks>
|
||||
/// Registered on driver-role nodes only (<see cref="HealthEndpoints.AddOtOpcUaHealth"/>); an
|
||||
/// admin-only node serves no OPC UA endpoint and must not be degraded by one it never starts. On a fused
|
||||
/// admin+driver node it is Ready-only until ServiceLevel consumes the fact (F-C1), because the Active tier
|
||||
/// is what Traefik routes the AdminUI on.
|
||||
/// admin+driver node it is Ready-only, because the Active tier is what Traefik routes the AdminUI on; the
|
||||
/// Primary moves through the Runtime instead — ServiceLevel 0 plus <c>OpcUaEndpointStepDownActor</c>
|
||||
/// (lmxopcua#762).
|
||||
/// </remarks>
|
||||
public sealed class OpcUaEndpointHealthCheck : IHealthCheck
|
||||
{
|
||||
|
||||
@@ -1,73 +0,0 @@
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.OpcUa;
|
||||
|
||||
/// <summary>Lifecycle of this node's OPC UA endpoint as <see cref="OtOpcUaServerHostedService"/> drives it.</summary>
|
||||
public enum OpcUaEndpointStatus
|
||||
{
|
||||
/// <summary>The hosted service has not run <c>StartAsync</c> yet.</summary>
|
||||
NotStarted,
|
||||
|
||||
/// <summary>The SDK is listening and the address-space / ServiceLevel sinks are bound.</summary>
|
||||
Started,
|
||||
|
||||
/// <summary>The SDK failed to start (port in use, PKI unwritable, bad <c>OpcUa:*</c> override, …).</summary>
|
||||
Failed,
|
||||
|
||||
/// <summary>The hosted service stopped the SDK; the port is released.</summary>
|
||||
Stopped,
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The one place the "did the OPC UA SDK actually start?" fact lives (R03:2.1). Before it, a failed
|
||||
/// start was one Error line: <c>/health/ready</c> stayed 200, the node joined its mesh, could be
|
||||
/// elected driver Primary and acked deploys as Applied through the Null sink — while clients got
|
||||
/// connection-refused from the node the fleet called healthy.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Consumed by <c>OpcUaEndpointHealthCheck</c> (Ready + Active tiers). The redundancy election and the
|
||||
/// ServiceLevel calculator do not read it yet — they live outside the Host (<c>Cluster</c> /
|
||||
/// <c>Runtime</c>); feeding them is the Wave 3 follow-up the lane report names. Until then a fused
|
||||
/// node whose SDK failed is visible (503) but still Primary-eligible.
|
||||
/// </remarks>
|
||||
public interface IOpcUaEndpointState
|
||||
{
|
||||
/// <summary>The endpoint's current lifecycle status.</summary>
|
||||
OpcUaEndpointStatus Status { get; }
|
||||
|
||||
/// <summary>Why the start failed, when <see cref="Status"/> is <see cref="OpcUaEndpointStatus.Failed"/>.</summary>
|
||||
string? FailureReason { get; }
|
||||
}
|
||||
|
||||
/// <summary>Mutable <see cref="IOpcUaEndpointState"/>; written by the hosted service, read by the health check.</summary>
|
||||
public sealed class OpcUaEndpointState : IOpcUaEndpointState
|
||||
{
|
||||
private volatile OpcUaEndpointStatus _status = OpcUaEndpointStatus.NotStarted;
|
||||
private volatile string? _failureReason;
|
||||
|
||||
/// <inheritdoc />
|
||||
public OpcUaEndpointStatus Status => _status;
|
||||
|
||||
/// <inheritdoc />
|
||||
public string? FailureReason => _failureReason;
|
||||
|
||||
/// <summary>Records a successful SDK start.</summary>
|
||||
public void MarkStarted()
|
||||
{
|
||||
_failureReason = null;
|
||||
_status = OpcUaEndpointStatus.Started;
|
||||
}
|
||||
|
||||
/// <summary>Records a failed SDK start.</summary>
|
||||
/// <param name="reason">The failure, as the health probe should report it.</param>
|
||||
public void MarkFailed(string reason)
|
||||
{
|
||||
_failureReason = reason;
|
||||
_status = OpcUaEndpointStatus.Failed;
|
||||
}
|
||||
|
||||
/// <summary>Records that the hosted service stopped the SDK.</summary>
|
||||
public void MarkStopped()
|
||||
{
|
||||
_failureReason = null;
|
||||
_status = OpcUaEndpointStatus.Stopped;
|
||||
}
|
||||
}
|
||||
@@ -241,6 +241,8 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
|
||||
/// <c>DbReachable</c> is fed constant <c>true</c> and only snapshot staleness can demote. See
|
||||
/// <see cref="RecomputeServiceLevel"/>.</summary>
|
||||
private readonly bool _dbLess;
|
||||
/// <summary>This node's OPC UA endpoint lifecycle fact (lmxopcua#762). Null = no fact (harness) = listening.</summary>
|
||||
private readonly IOpcUaEndpointState? _endpointState;
|
||||
private readonly TimeSpan _staleWindow;
|
||||
private readonly TimeSpan _probeFreshnessWindow;
|
||||
private readonly TimeSpan _healthTickInterval;
|
||||
@@ -306,6 +308,9 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
|
||||
/// ServiceLevel DB-reachability input is retired (<c>DbReachable</c> treated constant-true, only
|
||||
/// staleness demotes). No <paramref name="dbHealthProbe"/> is wired on such a node.</param>
|
||||
/// <param name="dataPlaneAuthority">The optional sink that receives the API keys and ACL grants carried by each applied artifact.</param>
|
||||
/// <param name="endpointState">This node's OPC UA endpoint lifecycle fact (lmxopcua#762). A
|
||||
/// <see cref="OpcUaEndpointStatus.Failed"/> endpoint computes ServiceLevel 0. Null (harnesses, a node with no
|
||||
/// hosted SDK) is treated as listening.</param>
|
||||
/// <returns>The configured <see cref="Props"/> for creating this actor, pinned to the
|
||||
/// <see cref="DispatcherId"/> dispatcher.</returns>
|
||||
public static Props Props(
|
||||
@@ -320,7 +325,8 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
|
||||
TimeSpan? healthTickInterval = null,
|
||||
bool dbLess = false,
|
||||
IDataPlaneAuthoritySink? dataPlaneAuthority = null,
|
||||
Func<DateTime>? utcNow = null) =>
|
||||
Func<DateTime>? utcNow = null,
|
||||
IOpcUaEndpointState? endpointState = null) =>
|
||||
Akka.Actor.Props.Create(() => new OpcUaPublishActor(
|
||||
sink ?? NullOpcUaAddressSpaceSink.Instance,
|
||||
serviceLevel ?? NullServiceLevelPublisher.Instance,
|
||||
@@ -334,7 +340,8 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
|
||||
healthTickInterval,
|
||||
dbLess,
|
||||
dataPlaneAuthority,
|
||||
utcNow)).WithDispatcher(DispatcherId);
|
||||
utcNow,
|
||||
endpointState)).WithDispatcher(DispatcherId);
|
||||
|
||||
/// <summary>Test-only Props that omits the pinned-dispatcher requirement and skips the
|
||||
/// DPS subscribe so unit tests can spin up the actor on a vanilla TestKit cluster.</summary>
|
||||
@@ -359,6 +366,9 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
|
||||
/// ServiceLevel DB-reachability input is retired (<c>DbReachable</c> treated constant-true, only
|
||||
/// staleness demotes). No <paramref name="dbHealthProbe"/> is wired on such a node.</param>
|
||||
/// <param name="dataPlaneAuthority">The optional sink that receives the API keys and ACL grants carried by each applied artifact.</param>
|
||||
/// <param name="endpointState">This node's OPC UA endpoint lifecycle fact (lmxopcua#762). A
|
||||
/// <see cref="OpcUaEndpointStatus.Failed"/> endpoint computes ServiceLevel 0. Null (harnesses, a node with no
|
||||
/// hosted SDK) is treated as listening.</param>
|
||||
/// <returns>The configured <see cref="Props"/> for creating this actor in tests, without dispatcher
|
||||
/// pinning or the DPS subscribe.</returns>
|
||||
public static Props PropsForTests(
|
||||
@@ -374,7 +384,8 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
|
||||
TimeSpan? healthTickInterval = null,
|
||||
bool dbLess = false,
|
||||
IDataPlaneAuthoritySink? dataPlaneAuthority = null,
|
||||
Func<DateTime>? utcNow = null) =>
|
||||
Func<DateTime>? utcNow = null,
|
||||
IOpcUaEndpointState? endpointState = null) =>
|
||||
Akka.Actor.Props.Create(() => new OpcUaPublishActor(
|
||||
sink ?? NullOpcUaAddressSpaceSink.Instance,
|
||||
serviceLevel ?? NullServiceLevelPublisher.Instance,
|
||||
@@ -388,7 +399,8 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
|
||||
healthTickInterval,
|
||||
dbLess,
|
||||
dataPlaneAuthority,
|
||||
utcNow));
|
||||
utcNow,
|
||||
endpointState));
|
||||
|
||||
/// <summary>Test-only Props exposing the deferred-write bound and the plan-step seams.</summary>
|
||||
internal static Props PropsForTests(
|
||||
@@ -402,7 +414,7 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
|
||||
// seam-carrying constructor is internal so the test seams stay off the public API (review M3).
|
||||
Akka.Actor.Props.CreateBy(new TestProducer(() => new OpcUaPublishActor(
|
||||
sink, NullServiceLevelPublisher.Instance, false, localNode, null, applier, null, null, null, null,
|
||||
false, dataPlaneAuthority, null, deferredWriteCapacity, hooks)));
|
||||
false, dataPlaneAuthority, null, deferredWriteCapacity, hooks, endpointState: null)));
|
||||
|
||||
/// <summary>Produces the actor through its internal (seam-carrying) constructor, for <see cref="PropsForTests(IOpcUaAddressSpaceSink, AddressSpaceApplier?, TestHooks?, int, NodeId?, IDataPlaneAuthoritySink?)"/>.</summary>
|
||||
private sealed class TestProducer(Func<OpcUaPublishActor> create) : IIndirectActorProducer
|
||||
@@ -434,6 +446,7 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
|
||||
/// ServiceLevel DB-reachability input is retired (<c>DbReachable</c> treated constant-true, only
|
||||
/// staleness demotes). No <paramref name="dbHealthProbe"/> is wired on such a node.</param>
|
||||
/// <param name="dataPlaneAuthority">The optional sink that receives the API keys and ACL grants carried by each applied artifact.</param>
|
||||
/// <param name="endpointState">This node's OPC UA endpoint lifecycle fact (lmxopcua#762); null is treated as listening.</param>
|
||||
public OpcUaPublishActor(
|
||||
IOpcUaAddressSpaceSink sink,
|
||||
IServiceLevelPublisher serviceLevel,
|
||||
@@ -447,15 +460,18 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
|
||||
TimeSpan? healthTickInterval = null,
|
||||
bool dbLess = false,
|
||||
IDataPlaneAuthoritySink? dataPlaneAuthority = null,
|
||||
Func<DateTime>? utcNow = null)
|
||||
Func<DateTime>? utcNow = null,
|
||||
IOpcUaEndpointState? endpointState = null)
|
||||
: this(sink, serviceLevel, subscribeRedundancyTopic, localNode, dbFactory, applier, dbHealthProbe, staleWindow,
|
||||
probeFreshnessWindow, healthTickInterval, dbLess, dataPlaneAuthority, utcNow, DefaultDeferredWriteCapacity, hooks: null)
|
||||
probeFreshnessWindow, healthTickInterval, dbLess, dataPlaneAuthority, utcNow, DefaultDeferredWriteCapacity, hooks: null,
|
||||
endpointState)
|
||||
{
|
||||
}
|
||||
|
||||
/// <summary>The full constructor: also the deferred-write bound and the test seams (internal — tests only).</summary>
|
||||
/// <param name="deferredWriteCapacity">The bound on distinct nodes whose writes are held for replay while a rebuild plan is in flight.</param>
|
||||
/// <param name="hooks">Test seams for the off-thread plan step; null in production.</param>
|
||||
/// <param name="endpointState">This node's OPC UA endpoint lifecycle fact (lmxopcua#762); null is treated as listening.</param>
|
||||
internal OpcUaPublishActor(
|
||||
IOpcUaAddressSpaceSink sink,
|
||||
IServiceLevelPublisher serviceLevel,
|
||||
@@ -471,7 +487,8 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
|
||||
IDataPlaneAuthoritySink? dataPlaneAuthority,
|
||||
Func<DateTime>? utcNow,
|
||||
int deferredWriteCapacity,
|
||||
TestHooks? hooks)
|
||||
TestHooks? hooks,
|
||||
IOpcUaEndpointState? endpointState)
|
||||
{
|
||||
_deferred = new DeferredWrites(Math.Max(1, deferredWriteCapacity));
|
||||
_hooks = hooks;
|
||||
@@ -486,6 +503,7 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
|
||||
_applier = applier;
|
||||
_dbHealthProbe = dbHealthProbe;
|
||||
_dbLess = dbLess;
|
||||
_endpointState = endpointState;
|
||||
_staleWindow = staleWindow ?? TimeSpan.FromSeconds(30);
|
||||
_probeFreshnessWindow = probeFreshnessWindow ?? TimeSpan.FromSeconds(30);
|
||||
_healthTickInterval = healthTickInterval ?? TimeSpan.FromSeconds(5);
|
||||
@@ -1536,6 +1554,15 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
|
||||
return verdict.Ok;
|
||||
}
|
||||
|
||||
/// <summary>The OPC UA endpoint input for the calculator (lmxopcua#762): <c>false</c> only when this node's own
|
||||
/// SDK start is recorded as <see cref="OpcUaEndpointStatus.Failed"/>. <see cref="OpcUaEndpointStatus.NotStarted"/>
|
||||
/// and <see cref="OpcUaEndpointStatus.Stopped"/> are boot and shutdown windows, not faults, and a node with no
|
||||
/// endpoint fact at all (a harness) is treated as listening. Read on every recompute — the periodic health tick
|
||||
/// and every redundancy snapshot (heartbeated every 10 s) — so a change is picked up without a notification.
|
||||
/// It is a LOCAL fact on purpose: the peer TCP probe behind <see cref="OpcUaProbeOk"/> cannot see a failed start
|
||||
/// when another process holds the port, because its connect then succeeds against that process.</summary>
|
||||
private bool EndpointListening() => _endpointState?.Status != OpcUaEndpointStatus.Failed;
|
||||
|
||||
/// <summary>
|
||||
/// Computes the local OPC UA ServiceLevel and routes it through <see cref="ServiceLevelChanged"/>
|
||||
/// (the dedup/publish/metric handler). The full <see cref="ServiceLevelCalculator"/> path is
|
||||
@@ -1575,7 +1602,8 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
|
||||
DbReachable: true,
|
||||
OpcUaProbeOk: OpcUaProbeOk(),
|
||||
Stale: (utcNow - entry.AsOfUtc) > _staleWindow,
|
||||
IsDriverPrimary: entry.IsDriverPrimary);
|
||||
IsDriverPrimary: entry.IsDriverPrimary,
|
||||
EndpointListening: EndpointListening());
|
||||
Self.Tell(new ServiceLevelChanged(ServiceLevelCalculator.Compute(dbLessInputs)));
|
||||
return;
|
||||
}
|
||||
@@ -1585,7 +1613,9 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
|
||||
// is the bootstrap value until the first DbHealthStatus lands.
|
||||
if (_dbHealthProbe is null || _lastDbHealth is null)
|
||||
{
|
||||
Self.Tell(new ServiceLevelChanged(LegacyRoleOnly(entry)));
|
||||
// The role-only seam has no health inputs, but a failed endpoint still wins (lmxopcua#762): the
|
||||
// legacy switch would otherwise hand a node that serves nothing 240 as Primary.
|
||||
Self.Tell(new ServiceLevelChanged(EndpointListening() ? LegacyRoleOnly(entry) : (byte)0));
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -1597,7 +1627,8 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
|
||||
Stale: !_lastDbHealth.Reachable
|
||||
|| (now - _lastDbHealth.AsOfUtc) > _staleWindow
|
||||
|| (now - entry.AsOfUtc) > _staleWindow,
|
||||
IsDriverPrimary: entry.IsDriverPrimary);
|
||||
IsDriverPrimary: entry.IsDriverPrimary,
|
||||
EndpointListening: EndpointListening());
|
||||
|
||||
Self.Tell(new ServiceLevelChanged(ServiceLevelCalculator.Compute(inputs)));
|
||||
}
|
||||
|
||||
@@ -0,0 +1,226 @@
|
||||
using Akka.Actor;
|
||||
using Akka.Cluster;
|
||||
using Akka.Cluster.Tools.PublishSubscribe;
|
||||
using Akka.Event;
|
||||
using ZB.MOM.WW.Health.Akka;
|
||||
using ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Types;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Runtime.Redundancy;
|
||||
|
||||
/// <summary>
|
||||
/// Steps this node down from driver Primary while its own OPC UA endpoint is not listening (lmxopcua#762).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Why.</b> A fused admin+driver node whose SDK failed to start (port 4840 taken, PKI unwritable, a bad
|
||||
/// <c>OpcUa:*</c> override) keeps running so the operator still has the AdminUI. Before this actor it also
|
||||
/// stayed Primary: the oldest Up driver member, so the Primary-gated data plane (writes, alarm acks, the
|
||||
/// alerts emit, the alarm-history drain, HistoryUpdate) stayed enabled on a node no client can reach, while
|
||||
/// the peer that clients CAN reach refused writes as Secondary.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>How.</b> The Primary is the oldest Up <c>driver</c> member and must stay exactly where
|
||||
/// <c>ClusterSingletonManager</c> puts the singletons (see <c>RedundancyStateActor.SelectDriverPrimary</c>),
|
||||
/// so the election is NOT bent to skip this node — that would split the Primary-gated data plane from the
|
||||
/// singletons. Instead the node gracefully <b>leaves</b> its mesh, the same mechanism as an operator's manual
|
||||
/// failover: it hands its singletons to the peer, <c>CoordinatedShutdown</c> runs,
|
||||
/// <c>ActorSystemTerminationWatchdog</c> exits the process, the service supervisor restarts it, and it rejoins
|
||||
/// as the youngest member — a Secondary whose ServiceLevel is 0 (the endpoint fact also feeds
|
||||
/// <c>ServiceLevelCalculator</c>). The peer is now the oldest Up driver member, so it is Primary, hosts the
|
||||
/// singletons, advertises 250, and the <c>/health/active</c> tier follows it.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Guards</b> — every one must hold, checked on each tick:
|
||||
/// </para>
|
||||
/// <list type="number">
|
||||
/// <item>this node's endpoint is <see cref="OpcUaEndpointStatus.Failed"/> (not merely
|
||||
/// <see cref="OpcUaEndpointStatus.NotStarted"/> or <see cref="OpcUaEndpointStatus.Stopped"/>, which are boot
|
||||
/// and shutdown windows);</item>
|
||||
/// <item>this node IS the driver Primary, judged by <see cref="ClusterActiveNode.SelfIsActive"/> — the same
|
||||
/// primitive the election delegates to, so the two cannot disagree;</item>
|
||||
/// <item>another Up driver member has RECENTLY reported, on <see cref="Topic"/>, that its own endpoint is
|
||||
/// listening. Without this a pair whose endpoints BOTH failed would bounce the Primary between them forever,
|
||||
/// restarting each node in turn — and a single node (no peer) would shut itself down for nothing. A peer on
|
||||
/// an older build never reports, so it never receives the role: that is the safe direction.</item>
|
||||
/// </list>
|
||||
/// <para>
|
||||
/// Every node also publishes its own <see cref="EndpointReport"/> on each tick, which is how its peer learns
|
||||
/// whether it can take over. The topic is separate from <c>redundancy-state</c> so the half-dozen actors that
|
||||
/// subscribe there are not handed a message type they do not handle.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class OpcUaEndpointStepDownActor : ReceiveActor, IWithTimers
|
||||
{
|
||||
/// <summary>The DistributedPubSub topic every driver node publishes its <see cref="EndpointReport"/> on.</summary>
|
||||
public const string Topic = "opcua-endpoint-state";
|
||||
|
||||
/// <summary>Default period between self-reports and step-down evaluations.</summary>
|
||||
public static readonly TimeSpan DefaultCheckInterval = TimeSpan.FromSeconds(5);
|
||||
|
||||
/// <summary>Default age beyond which a peer's report is ignored (it went away or stopped reporting).</summary>
|
||||
public static readonly TimeSpan DefaultReportFreshness = TimeSpan.FromSeconds(20);
|
||||
|
||||
/// <summary>One node's statement about its own OPC UA endpoint.</summary>
|
||||
/// <param name="Node">The reporting node, as <c>host:port</c> of its Akka address.</param>
|
||||
/// <param name="Listening">Whether the node's SDK is started and serving.</param>
|
||||
public sealed record EndpointReport(NodeId Node, bool Listening);
|
||||
|
||||
/// <summary>Periodic self-tick.</summary>
|
||||
public sealed class Tick
|
||||
{
|
||||
/// <summary>The singleton tick instance.</summary>
|
||||
public static readonly Tick Instance = new();
|
||||
private Tick() { }
|
||||
}
|
||||
|
||||
private readonly IOpcUaEndpointState _state;
|
||||
private readonly TimeSpan _interval;
|
||||
private readonly TimeSpan _freshness;
|
||||
private readonly Func<DateTime> _utcNow;
|
||||
private readonly Akka.Cluster.Cluster _cluster = Akka.Cluster.Cluster.Get(Context.System);
|
||||
private readonly ILoggingAdapter _log = Context.GetLogger();
|
||||
private readonly Dictionary<NodeId, (bool Listening, DateTime At)> _peerReports = new();
|
||||
private readonly NodeId _self;
|
||||
private bool _leaving;
|
||||
private string? _lastHoldReason;
|
||||
|
||||
/// <summary>Gets or sets the timer scheduler.</summary>
|
||||
public ITimerScheduler Timers { get; set; } = null!;
|
||||
|
||||
/// <summary>Creates the actor's props.</summary>
|
||||
/// <param name="state">This node's OPC UA endpoint lifecycle fact.</param>
|
||||
/// <param name="checkInterval">Period between self-reports and evaluations; defaults to <see cref="DefaultCheckInterval"/>.</param>
|
||||
/// <param name="reportFreshness">Peer-report age limit; defaults to <see cref="DefaultReportFreshness"/>.</param>
|
||||
/// <param name="utcNow">Clock; defaults to <see cref="DateTime.UtcNow"/>.</param>
|
||||
/// <returns>The props.</returns>
|
||||
public static Props Props(
|
||||
IOpcUaEndpointState state,
|
||||
TimeSpan? checkInterval = null,
|
||||
TimeSpan? reportFreshness = null,
|
||||
Func<DateTime>? utcNow = null) =>
|
||||
Akka.Actor.Props.Create(() => new OpcUaEndpointStepDownActor(state, checkInterval, reportFreshness, utcNow));
|
||||
|
||||
/// <summary>Initializes a new instance of the <see cref="OpcUaEndpointStepDownActor"/> class.</summary>
|
||||
/// <param name="state">This node's OPC UA endpoint lifecycle fact.</param>
|
||||
/// <param name="checkInterval">Period between self-reports and evaluations.</param>
|
||||
/// <param name="reportFreshness">Peer-report age limit.</param>
|
||||
/// <param name="utcNow">Clock.</param>
|
||||
public OpcUaEndpointStepDownActor(
|
||||
IOpcUaEndpointState state,
|
||||
TimeSpan? checkInterval,
|
||||
TimeSpan? reportFreshness,
|
||||
Func<DateTime>? utcNow)
|
||||
{
|
||||
_state = state ?? throw new ArgumentNullException(nameof(state));
|
||||
_interval = checkInterval ?? DefaultCheckInterval;
|
||||
_freshness = reportFreshness ?? DefaultReportFreshness;
|
||||
_utcNow = utcNow ?? (() => DateTime.UtcNow);
|
||||
_self = ToNodeId(_cluster.SelfAddress);
|
||||
|
||||
Receive<Tick>(_ => OnTick());
|
||||
Receive<EndpointReport>(OnReport);
|
||||
Receive<SubscribeAck>(_ => { });
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The canonical <c>host:port</c> id of an Akka address — the format <c>RedundancyStateActor.ToNodeId</c> and
|
||||
/// <c>ClusterRoleInfo</c> use. Derived from the member address on both sides (reporter and reader), so the two
|
||||
/// always agree.
|
||||
/// </summary>
|
||||
/// <param name="address">The member address.</param>
|
||||
/// <returns>The node id.</returns>
|
||||
public static NodeId ToNodeId(Address address) =>
|
||||
NodeId.Parse($"{address.Host ?? string.Empty}:{address.Port ?? 0}");
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void PreStart()
|
||||
{
|
||||
DistributedPubSub.Get(Context.System).Mediator.Tell(new Subscribe(Topic, Self));
|
||||
Self.Tell(Tick.Instance);
|
||||
Timers.StartPeriodicTimer("tick", Tick.Instance, _interval);
|
||||
}
|
||||
|
||||
private void OnTick()
|
||||
{
|
||||
var listening = _state.Status == OpcUaEndpointStatus.Started;
|
||||
try
|
||||
{
|
||||
DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(Topic, new EndpointReport(_self, listening)));
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_log.Debug(ex, "OpcUaEndpointStepDown: could not publish this node's endpoint report");
|
||||
}
|
||||
|
||||
Evaluate();
|
||||
}
|
||||
|
||||
private void OnReport(EndpointReport report)
|
||||
{
|
||||
if (report.Node == _self) return; // our own publish echoed back
|
||||
_peerReports[report.Node] = (report.Listening, _utcNow());
|
||||
Evaluate();
|
||||
}
|
||||
|
||||
private void Evaluate()
|
||||
{
|
||||
if (_leaving) return;
|
||||
if (_state.Status != OpcUaEndpointStatus.Failed)
|
||||
{
|
||||
_lastHoldReason = null;
|
||||
return;
|
||||
}
|
||||
|
||||
bool isPrimary;
|
||||
try
|
||||
{
|
||||
isPrimary = ClusterActiveNode.SelfIsActive(_cluster, RoleParser.Driver);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_log.Debug(ex, "OpcUaEndpointStepDown: cluster state unavailable; not evaluating");
|
||||
return;
|
||||
}
|
||||
|
||||
if (!isPrimary)
|
||||
{
|
||||
HoldOnce("not the driver Primary — nothing to step down from");
|
||||
return;
|
||||
}
|
||||
|
||||
var now = _utcNow();
|
||||
var self = _cluster.SelfMember.UniqueAddress;
|
||||
// Ordinal Member.HasRole on purpose — the same match ClusterActiveNode (and so the election) applies, so a
|
||||
// candidate here is a node the election would actually name Primary once this one leaves.
|
||||
var takeover = _cluster.State.Members
|
||||
.Where(m => m.Status == MemberStatus.Up
|
||||
&& m.HasRole(RoleParser.Driver)
|
||||
&& !m.UniqueAddress.Equals(self))
|
||||
.Select(m => (NodeId?)ToNodeId(m.Address))
|
||||
.FirstOrDefault(id => _peerReports.TryGetValue(id!.Value, out var r) && r.Listening && now - r.At <= _freshness);
|
||||
|
||||
if (takeover is not { } peer)
|
||||
{
|
||||
HoldOnce("no Up driver peer reports a listening OPC UA endpoint — staying Primary rather than handing the role to a node that cannot serve either");
|
||||
return;
|
||||
}
|
||||
|
||||
_leaving = true;
|
||||
_log.Warning(
|
||||
"OpcUaEndpointStepDown: this node is the driver Primary but its OPC UA endpoint failed to start ({Reason}); "
|
||||
+ "peer {Peer} reports a listening endpoint. Leaving the cluster so {Peer} becomes Primary — this node hands "
|
||||
+ "over its singletons, exits, and is restarted by its supervisor as a Secondary (lmxopcua#762).",
|
||||
_state.FailureReason ?? "no reason recorded", peer, peer);
|
||||
_cluster.Leave(_cluster.SelfAddress);
|
||||
}
|
||||
|
||||
private void HoldOnce(string reason)
|
||||
{
|
||||
if (reason == _lastHoldReason) return;
|
||||
_lastHoldReason = reason;
|
||||
_log.Warning("OpcUaEndpointStepDown: OPC UA endpoint failed to start ({Failure}); {Reason}",
|
||||
_state.FailureReason ?? "no reason recorded", reason);
|
||||
}
|
||||
}
|
||||
@@ -44,6 +44,7 @@ public static class ServiceCollectionExtensions
|
||||
public const string DependencyMuxActorName = "dependency-mux";
|
||||
public const string OpcUaPublishActorName = "opcua-publish";
|
||||
public const string PeerProbeSupervisorName = "peer-probe-supervisor";
|
||||
public const string OpcUaEndpointStepDownActorName = "opcua-endpoint-step-down";
|
||||
public const string ContinuousHistorizationRecorderActorName = "continuous-historization-recorder";
|
||||
|
||||
/// <summary>
|
||||
@@ -555,6 +556,10 @@ public static class ServiceCollectionExtensions
|
||||
// a node with no driver role never registers it and test harnesses build this graph without it;
|
||||
// the actor treats null as "nothing consumes authorization inputs" and skips the decode.
|
||||
var dataPlaneAuthority = resolver.GetService<IDataPlaneAuthoritySink>();
|
||||
// lmxopcua#762: the "did the OPC UA SDK start?" fact the Host's OtOpcUaServerHostedService maintains.
|
||||
// Optional: a harness that builds this graph without the hosted service registers none, and both
|
||||
// consumers then behave as if the endpoint were listening (the pre-#762 behaviour).
|
||||
var endpointState = resolver.GetService<IOpcUaEndpointState>();
|
||||
var publishActor = system.ActorOf(
|
||||
OpcUaPublishActor.Props(
|
||||
sink: addressSpaceSink,
|
||||
@@ -564,10 +569,19 @@ public static class ServiceCollectionExtensions
|
||||
applier: applier,
|
||||
dbHealthProbe: dbHealth,
|
||||
dbLess: dbFactory is null,
|
||||
dataPlaneAuthority: dataPlaneAuthority),
|
||||
dataPlaneAuthority: dataPlaneAuthority,
|
||||
endpointState: endpointState),
|
||||
OpcUaPublishActorName);
|
||||
registry.Register<OpcUaPublishActorKey>(publishActor);
|
||||
|
||||
// lmxopcua#762: a driver Primary whose OPC UA endpoint failed to start leaves its mesh once a peer
|
||||
// reports a listening endpoint, so the peer becomes Primary. Every driver node runs one — it is also
|
||||
// how this node tells its peer whether IT can take over.
|
||||
if (endpointState is not null)
|
||||
{
|
||||
system.ActorOf(OpcUaEndpointStepDownActor.Props(endpointState), OpcUaEndpointStepDownActorName);
|
||||
}
|
||||
|
||||
// Per-node peer-probe supervisor — keeps one OPC UA TCP probe per OTHER non-Detached
|
||||
// driver node, so every node is continuously probed by all its peers. OpcUaPublishActor
|
||||
// consumes the resulting probe verdicts to learn this node's own reachability.
|
||||
|
||||
@@ -23,6 +23,10 @@
|
||||
<!-- Core LocalDb only (ILocalDb + the connection/transaction seam) — the deployment-artifact
|
||||
cache lives here, but the sync engine and its gRPC endpoint are the Host's concern. -->
|
||||
<PackageReference Include="ZB.MOM.WW.LocalDb" />
|
||||
<!-- ClusterActiveNode — the same "oldest Up member of role X" primitive RedundancyStateActor's election
|
||||
delegates to. OpcUaEndpointStepDownActor (lmxopcua#762) asks "am I the driver Primary?" through it so
|
||||
the step-down and the election cannot disagree about which node is Primary. -->
|
||||
<PackageReference Include="ZB.MOM.WW.Health.Akka" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
|
||||
Reference in New Issue
Block a user