dohertj2/scadaproj
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(health)!: active node is the oldest Up member, not the leader (0.3.0)

Browse Source 
Promotes the two apps' private replacements for ActiveNodeHealthCheck into one
shared primitive, and retires the leader/RoleLeader selection they were written
to avoid.

Both Akka consumers had already hand-rolled a replacement rather than use this
package (ScadaBridge OldestNodeActiveHealthCheck, OtOpcUa
ClusterPrimaryHealthCheck), so the entire active-node surface here —
ActiveNodeHealthCheck, AkkaActiveNodeGate — had ZERO consumers. It was not
merely unused: it was avoided, twice, for the same reason.

Leadership is address-ordered (host, then port) and has no relationship to time;
singleton placement is age-ordered. The two agree on a freshly-formed cluster,
which is why single-node and happy-path tests never caught it. They diverge
permanently after any restart: the restarted node rejoins as the youngest but
keeps its address, so if it holds the lower address it becomes leader while the
singletons — and all the work they own — stay on the other node.

New ClusterActiveNode is the single implementation: oldest Up member, optional
role scope, plus role-preference resolution for a fused node that must answer
for the role its singletons are pinned to. ActiveNodeHealthCheck and
AkkaActiveNodeGate both delegate to it, so an endpoint gate and the
/health/active probe an orchestrator routes by cannot disagree.

BREAKING (behaviour, not signature):
- Selection is by age, not leadership.
- The role-filtered mode no longer reports Healthy for a node LACKING the role.
  That "not applicable => Healthy" made the tier answer 200 on every node and
  silently broke leader-pinning (lmxopcua#494); it is now Unhealthy by default,
  overridable via NoActiveRoleStatus. This case is reachable, not defensive —
  OtOpcUa's RoleParser admits dev-only and cluster-role-only nodes.
- Identity compares UniqueAddress, so a node restarted on the same host:port is
  correctly a different member during the overlap.

Results now carry activeRole/selfAddress/activeNode in the 0.2.0 per-entry data
object, so a standby reports WHO is active and a dashboard can render a pair
from either node's payload alone.

Tests: 45 Akka (was 39), 76 total. The age-vs-address divergence is pinned
against a real two-node cluster built so the oldest member is not the
lowest-addressed one, with a fixture sanity check so it cannot pass for the
wrong reason.
This commit is contained in:
Joseph Doherty
2026-07-24 13:07:56 -04:00
parent c81a6615ba
commit a61e041e58
11 changed files with 1332 additions and 263 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ZB.MOM.WW.Health/src/ZB.MOM.WW.Health.Akka/AkkaActiveNodeGate.cs
+36 -13
View File
@@ -6,27 +6,47 @@ namespace ZB.MOM.WW.Health.Akka;
/// <summary>
/// <see cref="IActiveNodeGate"/> implementation that computes <see cref="IsActiveNode"/> directly
/// from the Akka cluster state (self member <c>Up</c> and the local node is the cluster leader).
/// Register as a singleton.
/// from the Akka cluster state — the local node is the <b>oldest Up member</b>, optionally within a
/// role scope. Register as a singleton.
/// </summary>
/// <remarks>
/// The <see cref="ActorSystem"/> is resolved lazily from the service provider; if it is not yet
/// available — e.g. during startup before Akka is initialised — <see cref="IsActiveNode"/> returns
/// <c>false</c> (the safe default during startup). This gate reads the cluster state directly and
/// does not resolve <see cref="ActiveNodeHealthCheck"/> from DI.
/// <para>
/// Shares its rule with <see cref="ActiveNodeHealthCheck"/> via
/// <see cref="ClusterActiveNode"/>, so an endpoint gated by this type and the
/// <c>/health/active</c> probe an orchestrator routes by can never disagree about which node
/// is in charge.
/// </para>
/// <para>
/// <b>Behaviour change in 0.3.0:</b> through 0.2.1 this gate selected by
/// <c>ClusterState.Leader</c>, which is address-ordered and diverges from singleton placement
/// after any restart. See <see cref="ClusterActiveNode"/>.
/// </para>
/// <para>
/// The <see cref="ActorSystem"/> is resolved lazily from the service provider; if it is not
/// yet available — e.g. during startup before Akka is initialised —
/// <see cref="IsActiveNode"/> returns <c>false</c> (the safe default during startup, matching
/// the standby case). This gate reads the cluster state directly and does not resolve
/// <see cref="ActiveNodeHealthCheck"/> from DI.
/// </para>
/// </remarks>
public sealed class AkkaActiveNodeGate : IActiveNodeGate
{
private readonly IServiceProvider _serviceProvider;
private readonly string? _role;
/// <summary>Initializes a new <see cref="AkkaActiveNodeGate"/>.</summary>
/// <param name="serviceProvider">
/// The application service provider. The <see cref="ActorSystem"/> is resolved lazily; if it is
/// not yet available <see cref="IsActiveNode"/> returns <c>false</c>.
/// </param>
public AkkaActiveNodeGate(IServiceProvider serviceProvider)
/// <param name="role">
/// The role to scope the competition to, or <c>null</c> (the default) to consider every Up
/// member.
/// </param>
public AkkaActiveNodeGate(IServiceProvider serviceProvider, string? role = null)
{
_serviceProvider = serviceProvider ?? throw new ArgumentNullException(nameof(serviceProvider));
_role = role;
}
/// <inheritdoc />
@@ -38,13 +58,16 @@ public sealed class AkkaActiveNodeGate : IActiveNodeGate
if (system is null)
return false;
var cluster = Cluster.Get(system);
var self = cluster.SelfMember;
if (self.Status != MemberStatus.Up)
try
{
return ClusterActiveNode.SelfIsActive(Cluster.Get(system), _role);
}
catch (Exception)
{
// ActorSystem exists but the cluster is not initialised yet. Same startup-safety
// rule as the health check, expressed as the gate's safe default: not active.
return false;
var leader = cluster.State.Leader;
return leader is not null && leader == self.Address;
}
}
}
}