dohertj2/scadaproj
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(health)!: active node is the oldest Up member, not the leader (0.3.0)

Browse Source 
Promotes the two apps' private replacements for ActiveNodeHealthCheck into one
shared primitive, and retires the leader/RoleLeader selection they were written
to avoid.

Both Akka consumers had already hand-rolled a replacement rather than use this
package (ScadaBridge OldestNodeActiveHealthCheck, OtOpcUa
ClusterPrimaryHealthCheck), so the entire active-node surface here —
ActiveNodeHealthCheck, AkkaActiveNodeGate — had ZERO consumers. It was not
merely unused: it was avoided, twice, for the same reason.

Leadership is address-ordered (host, then port) and has no relationship to time;
singleton placement is age-ordered. The two agree on a freshly-formed cluster,
which is why single-node and happy-path tests never caught it. They diverge
permanently after any restart: the restarted node rejoins as the youngest but
keeps its address, so if it holds the lower address it becomes leader while the
singletons — and all the work they own — stay on the other node.

New ClusterActiveNode is the single implementation: oldest Up member, optional
role scope, plus role-preference resolution for a fused node that must answer
for the role its singletons are pinned to. ActiveNodeHealthCheck and
AkkaActiveNodeGate both delegate to it, so an endpoint gate and the
/health/active probe an orchestrator routes by cannot disagree.

BREAKING (behaviour, not signature):
- Selection is by age, not leadership.
- The role-filtered mode no longer reports Healthy for a node LACKING the role.
  That "not applicable => Healthy" made the tier answer 200 on every node and
  silently broke leader-pinning (lmxopcua#494); it is now Unhealthy by default,
  overridable via NoActiveRoleStatus. This case is reachable, not defensive —
  OtOpcUa's RoleParser admits dev-only and cluster-role-only nodes.
- Identity compares UniqueAddress, so a node restarted on the same host:port is
  correctly a different member during the overlap.

Results now carry activeRole/selfAddress/activeNode in the 0.2.0 per-entry data
object, so a standby reports WHO is active and a dashboard can render a pair
from either node's payload alone.

Tests: 45 Akka (was 39), 76 total. The age-vs-address divergence is pinned
against a real two-node cluster built so the oldest member is not the
lowest-addressed one, with a fixture sanity check so it cannot pass for the
wrong reason.
This commit is contained in:
Joseph Doherty
2026-07-24 13:07:56 -04:00
parent c81a6615ba
commit a61e041e58
11 changed files with 1332 additions and 263 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ZB.MOM.WW.Health/src/ZB.MOM.WW.Health.Akka/ActiveNodeHealthCheck.cs
+84 -110
View File
@@ -9,99 +9,80 @@ using Microsoft.Extensions.Diagnostics.HealthChecks;
namespace ZB.MOM.WW.Health.Akka;
/// <summary>
/// Pure decision function for the active / leader probe, factored out of
/// <see cref="ActiveNodeHealthCheck"/> so the role-less and role-filtered matrices are exhaustively
/// table-testable without forming a real cluster.
/// </summary>
internal static class ActiveNodeDecision
{
/// <summary>
/// Maps the resolved cluster facts to a <see cref="HealthStatus"/>.
/// </summary>
/// <param name="selfUp">Whether the local node's member status is <c>Up</c>.</param>
/// <param name="isLeader">
/// Whether the local node is the leader: the cluster leader in role-less mode, or the
/// role-singleton leader in role-filtered mode.
/// </param>
/// <param name="hasRole">
/// Whether the local node carries <paramref name="requiredRole"/>. Ignored when
/// <paramref name="requiredRole"/> is <c>null</c>.
/// </param>
/// <param name="requiredRole">
/// The role to scope the check to, or <c>null</c> for the role-less (whole-cluster-leader) mode.
/// </param>
/// <returns>
/// Role-less: Healthy iff the node is Up and the cluster leader, otherwise Unhealthy.
/// Role-filtered: Healthy when the node lacks the role (probe irrelevant) or carries the role and
/// is the role-singleton leader; <strong>Unhealthy</strong> when it carries the role but is not
/// the leader.
/// </returns>
/// <remarks>
/// The role-member-but-not-leader case returns Unhealthy, not Degraded, because the active tier's
/// whole purpose is to be a 503 that an orchestrator can act on — the shared spec states it as
/// "Fails (503) on a standby or role-member-but-not-leader node", and the same spec maps Degraded
/// to 200. Returning Degraded made the tier answer 200 on every node, so a load balancer pointed
/// at it (OtOpcUa's Traefik routes the admin UI this way) kept the standby in the pool and its
/// leader-pinning silently never worked. Found on a live rig; see lmxopcua#494.
/// </remarks>
public static HealthStatus Evaluate(bool selfUp, bool isLeader, bool hasRole, string? requiredRole)
{
if (requiredRole is null)
return selfUp && isLeader ? HealthStatus.Healthy : HealthStatus.Unhealthy;
if (!hasRole)
return HealthStatus.Healthy;
return isLeader ? HealthStatus.Healthy : HealthStatus.Unhealthy;
}
}
/// <summary>
/// Health check that reports whether this node is the designated active / leader node.
/// An optional role scopes the check to nodes carrying that role. Register to the
/// <see cref="ZbHealthTags.Active"/> tag.
/// Active-tier probe: Healthy (200) on the one node in charge, Unhealthy (503) on its standby.
/// Register to the <see cref="ZbHealthTags.Active"/> tag.
/// </summary>
/// <remarks>
/// The <see cref="ActorSystem"/> is resolved lazily from the service provider. If it is not yet
/// available — e.g. during startup before Akka is initialised — the check returns
/// <see cref="HealthStatus.Degraded"/> rather than throwing, so it is startup-safe.
/// <para>
/// "In charge" means the <b>oldest Up member</b> — see <see cref="ClusterActiveNode"/> for
/// why leadership is the wrong answer and what breaks when it is used. Scope the competition
/// to a role with <see cref="ActiveNodeHealthCheckOptions.RolePreference"/>; leave it empty
/// for a cluster whose nodes all compete for the same single active slot.
/// </para>
/// <para>
/// <b>Behaviour change in 0.3.0.</b> Through 0.2.1 this check selected by
/// <c>ClusterState.Leader</c> / <c>RoleLeader</c>, and its role-filtered mode reported
/// Healthy for any node <i>lacking</i> the role. Both were wrong, both were found on live
/// rigs, and both Akka consumers had already replaced it with a private copy rather than use
/// it. It now selects by age and reports <see cref="HealthStatus.Unhealthy"/> for a node that
/// owns no active work. A node whose active tier previously answered 200 unconditionally will
/// now answer 503 unless it really is the active node — which is the point.
/// </para>
/// <para>
/// Startup-safe: before the <see cref="ActorSystem"/> or the cluster is available the result
/// is <see cref="HealthStatus.Degraded"/> (a 200 under the tier mapping), so a node that is
/// merely still booting is not reported as a failed standby and pulled from rotation on the
/// way up.
/// </para>
/// </remarks>
public sealed class ActiveNodeHealthCheck : IHealthCheck
{
private readonly IServiceProvider _serviceProvider;
private readonly string? _role;
private readonly ActiveNodeHealthCheckOptions _options;
/// <summary>
/// Role-less constructor: Healthy when the node is <c>Up</c> and the cluster leader
/// (ScadaBridge ActiveNode pattern); Unhealthy otherwise. Degraded when the ActorSystem /
/// cluster is not yet ready.
/// Unscoped: the active node is the oldest Up member of the whole cluster.
/// </summary>
/// <param name="serviceProvider">
/// The application service provider. The <see cref="ActorSystem"/> is resolved lazily so the
/// check is startup-safe: if no <see cref="ActorSystem"/> is registered yet the result is Degraded.
/// check is startup-safe.
/// </param>
public ActiveNodeHealthCheck(IServiceProvider serviceProvider)
: this(serviceProvider, new ActiveNodeHealthCheckOptions())
{
_serviceProvider = serviceProvider ?? throw new ArgumentNullException(nameof(serviceProvider));
_role = null;
}
/// <summary>
/// Role-filtered constructor: Healthy when the node lacks <paramref name="role"/> or carries it
/// and is the role-singleton leader; Unhealthy (503) when it carries the role but is not the
/// leader (OtOpcUa AdminRoleLeader pattern). Degraded when the ActorSystem / cluster is not yet
/// ready.
/// Role-scoped: the active node is the oldest Up member carrying <paramref name="role"/>.
/// </summary>
/// <param name="serviceProvider">
/// The application service provider. The <see cref="ActorSystem"/> is resolved lazily so the
/// check is startup-safe: if no <see cref="ActorSystem"/> is registered yet the result is Degraded.
/// check is startup-safe.
/// </param>
/// <param name="role">The Akka cluster role to scope the check to.</param>
/// <param name="role">The cluster role to scope the competition to.</param>
public ActiveNodeHealthCheck(IServiceProvider serviceProvider, string role)
: this(serviceProvider, NewOptions(role))
{
}
/// <summary>
/// Configured with an explicit <see cref="ActiveNodeHealthCheckOptions"/>.
/// </summary>
/// <param name="serviceProvider">
/// The application service provider. The <see cref="ActorSystem"/> is resolved lazily so the
/// check is startup-safe.
/// </param>
/// <param name="options">The role-scoping options.</param>
public ActiveNodeHealthCheck(IServiceProvider serviceProvider, ActiveNodeHealthCheckOptions options)
{
_serviceProvider = serviceProvider ?? throw new ArgumentNullException(nameof(serviceProvider));
_options = options ?? throw new ArgumentNullException(nameof(options));
}
private static ActiveNodeHealthCheckOptions NewOptions(string role)
{
ArgumentException.ThrowIfNullOrWhiteSpace(role);
_role = role;
return new ActiveNodeHealthCheckOptions { RolePreference = new[] { role } };
}
/// <inheritdoc />
@@ -113,10 +94,8 @@ public sealed class ActiveNodeHealthCheck : IHealthCheck
if (system is null)
return Task.FromResult(HealthCheckResult.Degraded("ActorSystem not yet available."));
MemberStatus selfStatus;
bool selfUp;
bool hasRole;
bool isLeader;
Member self;
IEnumerable<Member> members;
try
{
// Reading cluster membership can throw while the ActorSystem exists but the cluster has
@@ -124,54 +103,49 @@ public sealed class ActiveNodeHealthCheck : IHealthCheck
// ConfigurationException). The spec's startup-safety rule maps this to Degraded rather
// than letting the exception escape (which the host would record as Unhealthy).
var cluster = Cluster.Get(system);
var self = cluster.SelfMember;
selfStatus = self.Status;
selfUp = selfStatus == MemberStatus.Up;
if (_role is null)
{
hasRole = false;
var leader = cluster.State.Leader;
isLeader = leader is not null && leader == self.Address;
}
else
{
hasRole = self.HasRole(_role);
var roleLeader = cluster.State.RoleLeader(_role);
isLeader = roleLeader is not null && roleLeader == self.Address;
}
self = cluster.SelfMember;
members = cluster.State.Members;
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
return Task.FromResult(HealthCheckResult.Degraded("Akka cluster state not yet accessible.", ex));
}
var health = ActiveNodeDecision.Evaluate(selfUp, isLeader, hasRole, _role);
var description = DescribeResult(health, selfStatus, selfUp, isLeader);
var result = health switch
{
HealthStatus.Healthy => HealthCheckResult.Healthy(description),
HealthStatus.Degraded => HealthCheckResult.Degraded(description),
_ => HealthCheckResult.Unhealthy(description),
};
return Task.FromResult(result);
}
var scoped = _options.RolePreference.Count > 0;
var role = ClusterActiveNode.ResolveActiveRole(self.Roles, _options.RolePreference);
private string DescribeResult(HealthStatus health, MemberStatus status, bool selfUp, bool isLeader)
{
if (_role is null)
var data = new Dictionary<string, object>(StringComparer.Ordinal)
{
if (health == HealthStatus.Healthy)
return "Active node (cluster leader).";
return selfUp && !isLeader
? "Standby: node is Up but not the cluster leader."
: $"Standby: node is not Up (status: {status}).";
["selfAddress"] = self.Address.ToString(),
};
if (role is not null) data["activeRole"] = role;
if (scoped && role is null)
{
// The node carries none of the preferred roles, so it owns no active work and cannot be
// the active node for any of them. Reported per NoActiveRoleStatus — Unhealthy by
// default, because the active tier's whole job is to keep such a node out of the pool.
var message = $"Node carries none of the active roles: {string.Join(", ", _options.RolePreference)}.";
return Task.FromResult(new HealthCheckResult(_options.NoActiveRoleStatus, message, exception: null, data));
}
return health switch
var active = ClusterActiveNode.OldestUpMember(members, role);
if (active is not null) data["activeNode"] = active.Address.ToString();
var scopeLabel = role is null ? "the cluster" : $"role '{role}'";
if (active is null)
{
HealthStatus.Healthy => $"Active for role '{_role}' (or not a role member).",
_ => $"Role '{_role}' member but not leader.",
};
// Nobody is Up in the scope — a cold start, or every holder is still Joining or already
// Leaving. Nobody is active, so nobody may claim to be.
return Task.FromResult(HealthCheckResult.Unhealthy(
$"No Up member currently holds the active slot for {scopeLabel}.", exception: null, data));
}
// Compared by UniqueAddress: a node that restarted on the same host and port is a different
// member, and during the overlap the cluster briefly holds both incarnations.
return Task.FromResult(active.UniqueAddress.Equals(self.UniqueAddress)
? HealthCheckResult.Healthy($"Active node for {scopeLabel} (oldest Up member).", data)
: HealthCheckResult.Unhealthy($"Standby: active node for {scopeLabel} is {active.Address}.", exception: null, data));
}
}
ZB.MOM.WW.Health/src/ZB.MOM.WW.Health.Akka/ActiveNodeHealthCheckOptions.cs
+37
View File
@@ -0,0 +1,37 @@
using Microsoft.Extensions.Diagnostics.HealthChecks;
namespace ZB.MOM.WW.Health.Akka;
/// <summary>
/// Configures how <see cref="ActiveNodeHealthCheck"/> decides which role's active node this node is
/// competing to be.
/// </summary>
public sealed class ActiveNodeHealthCheckOptions
{
/// <summary>
/// Candidate roles in descending precedence. The check answers for the first one the local node
/// carries. Empty (the default) means unscoped: the active node is the oldest Up member of the
/// whole cluster.
/// </summary>
/// <remarks>
/// A fused node should list the role its singletons are pinned to first — e.g.
/// <c>["admin", "driver"]</c> for a node that hosts the admin singletons when fused and competes
/// as a plain driver when it is a driver-only node.
/// </remarks>
public IReadOnlyList<string> RolePreference { get; init; } = Array.Empty<string>();
/// <summary>
/// The status to report when <see cref="RolePreference"/> is non-empty and the node carries none
/// of those roles — it owns no active work at all. Defaults to
/// <see cref="HealthStatus.Unhealthy"/>.
/// </summary>
/// <remarks>
/// Unhealthy is the safe default and is deliberately <b>not</b> the "not applicable ⇒ Healthy"
/// convention used by readiness probes. The active tier exists so a load balancer can pick
/// exactly one node; a node that owns no active work must not be in that pool. Answering Healthy
/// there is what made the previous role-filtered check report 200 on every node and silently
/// broke leader-pinning (lmxopcua#494). Override only when a node genuinely should stay in the
/// active pool without holding any of the preferred roles.
/// </remarks>
public HealthStatus NoActiveRoleStatus { get; init; } = HealthStatus.Unhealthy;
}
ZB.MOM.WW.Health/src/ZB.MOM.WW.Health.Akka/AkkaActiveNodeGate.cs
+36 -13
View File
@@ -6,27 +6,47 @@ namespace ZB.MOM.WW.Health.Akka;
/// <summary>
/// <see cref="IActiveNodeGate"/> implementation that computes <see cref="IsActiveNode"/> directly
/// from the Akka cluster state (self member <c>Up</c> and the local node is the cluster leader).
/// Register as a singleton.
/// from the Akka cluster state — the local node is the <b>oldest Up member</b>, optionally within a
/// role scope. Register as a singleton.
/// </summary>
/// <remarks>
/// The <see cref="ActorSystem"/> is resolved lazily from the service provider; if it is not yet
/// available — e.g. during startup before Akka is initialised — <see cref="IsActiveNode"/> returns
/// <c>false</c> (the safe default during startup). This gate reads the cluster state directly and
/// does not resolve <see cref="ActiveNodeHealthCheck"/> from DI.
/// <para>
/// Shares its rule with <see cref="ActiveNodeHealthCheck"/> via
/// <see cref="ClusterActiveNode"/>, so an endpoint gated by this type and the
/// <c>/health/active</c> probe an orchestrator routes by can never disagree about which node
/// is in charge.
/// </para>
/// <para>
/// <b>Behaviour change in 0.3.0:</b> through 0.2.1 this gate selected by
/// <c>ClusterState.Leader</c>, which is address-ordered and diverges from singleton placement
/// after any restart. See <see cref="ClusterActiveNode"/>.
/// </para>
/// <para>
/// The <see cref="ActorSystem"/> is resolved lazily from the service provider; if it is not
/// yet available — e.g. during startup before Akka is initialised —
/// <see cref="IsActiveNode"/> returns <c>false</c> (the safe default during startup, matching
/// the standby case). This gate reads the cluster state directly and does not resolve
/// <see cref="ActiveNodeHealthCheck"/> from DI.
/// </para>
/// </remarks>
public sealed class AkkaActiveNodeGate : IActiveNodeGate
{
private readonly IServiceProvider _serviceProvider;
private readonly string? _role;
/// <summary>Initializes a new <see cref="AkkaActiveNodeGate"/>.</summary>
/// <param name="serviceProvider">
/// The application service provider. The <see cref="ActorSystem"/> is resolved lazily; if it is
/// not yet available <see cref="IsActiveNode"/> returns <c>false</c>.
/// </param>
public AkkaActiveNodeGate(IServiceProvider serviceProvider)
/// <param name="role">
/// The role to scope the competition to, or <c>null</c> (the default) to consider every Up
/// member.
/// </param>
public AkkaActiveNodeGate(IServiceProvider serviceProvider, string? role = null)
{
_serviceProvider = serviceProvider ?? throw new ArgumentNullException(nameof(serviceProvider));
_role = role;
}
/// <inheritdoc />
@@ -38,13 +58,16 @@ public sealed class AkkaActiveNodeGate : IActiveNodeGate
if (system is null)
return false;
var cluster = Cluster.Get(system);
var self = cluster.SelfMember;
if (self.Status != MemberStatus.Up)
try
{
return ClusterActiveNode.SelfIsActive(Cluster.Get(system), _role);
}
catch (Exception)
{
// ActorSystem exists but the cluster is not initialised yet. Same startup-safety
// rule as the health check, expressed as the gate's safe default: not active.
return false;
var leader = cluster.State.Leader;
return leader is not null && leader == self.Address;
}
}
}
}
ZB.MOM.WW.Health/src/ZB.MOM.WW.Health.Akka/ClusterActiveNode.cs
+133
View File
@@ -0,0 +1,133 @@
using Akka.Cluster;
namespace ZB.MOM.WW.Health.Akka;
/// <summary>
/// THE family-wide definition of "the active node": the <b>oldest Up member</b> of the cluster,
/// optionally scoped to a role — the member <c>ClusterSingletonManager</c> places singletons on.
/// </summary>
/// <remarks>
/// <para>
/// Cluster <b>leadership</b> (<c>ClusterState.Leader</c> / <c>RoleLeader</c>) is the wrong
/// answer and must never be used for a product-level active/standby decision. Leadership is
/// <i>address</i>-ordered (host, then port) and has no relationship to time; singleton
/// placement is <i>age</i>-ordered. On a freshly-formed cluster the two agree, which is why
/// the discrepancy hides in unit tests and in a rig that has never been restarted. They
/// diverge permanently after any restart: the restarted node rejoins as the youngest but
/// keeps its address, so if it holds the lower address it becomes leader while the
/// singletons — and all the work they own — stay on the other node.
/// </para>
/// <para>
/// Both Akka consumers in the family independently hit this and each wrote its own
/// replacement rather than use the shared leader-based check
/// (ScadaBridge <c>OldestNodeActiveHealthCheck</c>, OtOpcUa <c>ClusterPrimaryHealthCheck</c>).
/// This type is those two implementations promoted to one, so the health tier, the inbound
/// API gate, the store-and-forward delivery gate and OtOpcUa's OPC UA <c>ServiceLevel</c>
/// cannot drift apart about which node is in charge.
/// </para>
/// <para>
/// Only <see cref="MemberStatus.Up"/> members are eligible, matching singleton placement: a
/// <c>Leaving</c> node is handing its singletons over and must not be named active.
/// </para>
/// </remarks>
public static class ClusterActiveNode
{
/// <summary>
/// The oldest <see cref="MemberStatus.Up"/> member, optionally scoped to <paramref name="role"/>.
/// </summary>
/// <param name="members">The cluster members, in any order.</param>
/// <param name="role">
/// The role to scope the selection to, or <c>null</c> to consider every Up member.
/// </param>
/// <returns>The oldest eligible member, or <c>null</c> when none is Up.</returns>
public static Member? OldestUpMember(IEnumerable<Member> members, string? role = null)
{
ArgumentNullException.ThrowIfNull(members);
Member? oldest = null;
foreach (var member in members)
{
if (member.Status != MemberStatus.Up) continue;
if (role is not null && !member.HasRole(role)) continue;
if (oldest is null || member.IsOlderThan(oldest)) oldest = member;
}
return oldest;
}
/// <summary>
/// The oldest <see cref="MemberStatus.Up"/> member of <paramref name="cluster"/>, optionally
/// scoped to <paramref name="role"/>.
/// </summary>
/// <param name="cluster">The Akka cluster to evaluate.</param>
/// <param name="role">
/// The role to scope the selection to, or <c>null</c> to consider every Up member.
/// </param>
/// <returns>The oldest eligible member, or <c>null</c> when none is Up.</returns>
public static Member? OldestUpMember(Cluster cluster, string? role = null)
{
ArgumentNullException.ThrowIfNull(cluster);
return OldestUpMember(cluster.State.Members, role);
}
/// <summary>
/// Whether the local node is the active node — Up, and the oldest Up member in the role scope.
/// </summary>
/// <param name="cluster">The Akka cluster to evaluate.</param>
/// <param name="role">
/// The role to scope the selection to, or <c>null</c> to consider every Up member.
/// </param>
/// <returns>
/// <c>true</c> when the local member is Up, carries <paramref name="role"/> when one is given,
/// and no other eligible member is older.
/// </returns>
/// <remarks>
/// Identity is compared by <see cref="Member.UniqueAddress"/>, not <c>Address</c>: a node that
/// restarted on the same host and port is a different member, and during the overlap the cluster
/// briefly holds both incarnations.
/// </remarks>
public static bool SelfIsActive(Cluster cluster, string? role = null)
{
ArgumentNullException.ThrowIfNull(cluster);
var self = cluster.SelfMember;
if (self.Status != MemberStatus.Up) return false;
if (role is not null && !self.HasRole(role)) return false;
var oldest = OldestUpMember(cluster.State.Members, role);
return oldest is not null && oldest.UniqueAddress.Equals(self.UniqueAddress);
}
/// <summary>
/// The first role in <paramref name="rolePreference"/> that <paramref name="selfRoles"/> carries
/// — the role whose oldest Up member owns this node's active work.
/// </summary>
/// <param name="selfRoles">The local member's cluster roles.</param>
/// <param name="rolePreference">
/// Candidate roles in descending precedence, e.g. <c>["admin", "driver"]</c> for a node that
/// answers for <c>admin</c> when it is fused and for <c>driver</c> when it is a driver-only node.
/// </param>
/// <returns>
/// The first carried role, or <c>null</c> when the node carries none of them.
/// </returns>
/// <remarks>
/// Precedence matters on a fused node: it must answer for the role its singletons — and the
/// traffic being routed to it — are pinned to. Answering for a lower-precedence role could name a
/// different node the moment a single-role member joins the mesh.
/// </remarks>
public static string? ResolveActiveRole(IEnumerable<string> selfRoles, IReadOnlyList<string> rolePreference)
{
ArgumentNullException.ThrowIfNull(selfRoles);
ArgumentNullException.ThrowIfNull(rolePreference);
if (rolePreference.Count == 0) return null;
var roles = selfRoles as IReadOnlyCollection<string> ?? selfRoles.ToList();
foreach (var candidate in rolePreference)
{
if (roles.Contains(candidate)) return candidate;
}
return null;
}
}