dohertj2/scadaproj
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(health)!: active node is the oldest Up member, not the leader (0.3.0)

Browse Source 
Promotes the two apps' private replacements for ActiveNodeHealthCheck into one
shared primitive, and retires the leader/RoleLeader selection they were written
to avoid.

Both Akka consumers had already hand-rolled a replacement rather than use this
package (ScadaBridge OldestNodeActiveHealthCheck, OtOpcUa
ClusterPrimaryHealthCheck), so the entire active-node surface here —
ActiveNodeHealthCheck, AkkaActiveNodeGate — had ZERO consumers. It was not
merely unused: it was avoided, twice, for the same reason.

Leadership is address-ordered (host, then port) and has no relationship to time;
singleton placement is age-ordered. The two agree on a freshly-formed cluster,
which is why single-node and happy-path tests never caught it. They diverge
permanently after any restart: the restarted node rejoins as the youngest but
keeps its address, so if it holds the lower address it becomes leader while the
singletons — and all the work they own — stay on the other node.

New ClusterActiveNode is the single implementation: oldest Up member, optional
role scope, plus role-preference resolution for a fused node that must answer
for the role its singletons are pinned to. ActiveNodeHealthCheck and
AkkaActiveNodeGate both delegate to it, so an endpoint gate and the
/health/active probe an orchestrator routes by cannot disagree.

BREAKING (behaviour, not signature):
- Selection is by age, not leadership.
- The role-filtered mode no longer reports Healthy for a node LACKING the role.
  That "not applicable => Healthy" made the tier answer 200 on every node and
  silently broke leader-pinning (lmxopcua#494); it is now Unhealthy by default,
  overridable via NoActiveRoleStatus. This case is reachable, not defensive —
  OtOpcUa's RoleParser admits dev-only and cluster-role-only nodes.
- Identity compares UniqueAddress, so a node restarted on the same host:port is
  correctly a different member during the overlap.

Results now carry activeRole/selfAddress/activeNode in the 0.2.0 per-entry data
object, so a standby reports WHO is active and a dashboard can render a pair
from either node's payload alone.

Tests: 45 Akka (was 39), 76 total. The age-vs-address divergence is pinned
against a real two-node cluster built so the oldest member is not the
lowest-addressed one, with a fixture sanity check so it cannot pass for the
wrong reason.
This commit is contained in:
Joseph Doherty
2026-07-24 13:07:56 -04:00
parent c81a6615ba
commit a61e041e58
11 changed files with 1332 additions and 263 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ZB.MOM.WW.Health/Directory.Build.props
+1 -1
View File
@@ -5,7 +5,7 @@
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<LangVersion>latest</LangVersion>
<Version>0.2.1</Version>
<Version>0.3.0</Version>
<ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
<!-- Emit XML docs so the public API summaries ship inside the packed nupkgs (IntelliSense for
consumers). CS1591 (missing doc on a public member) is suppressed so undocumented test /
ZB.MOM.WW.Health/src/ZB.MOM.WW.Health.Akka/ActiveNodeHealthCheck.cs
+84 -110
View File
@@ -9,99 +9,80 @@ using Microsoft.Extensions.Diagnostics.HealthChecks;
namespace ZB.MOM.WW.Health.Akka;
/// <summary>
/// Pure decision function for the active / leader probe, factored out of
/// <see cref="ActiveNodeHealthCheck"/> so the role-less and role-filtered matrices are exhaustively
/// table-testable without forming a real cluster.
/// </summary>
internal static class ActiveNodeDecision
{
/// <summary>
/// Maps the resolved cluster facts to a <see cref="HealthStatus"/>.
/// </summary>
/// <param name="selfUp">Whether the local node's member status is <c>Up</c>.</param>
/// <param name="isLeader">
/// Whether the local node is the leader: the cluster leader in role-less mode, or the
/// role-singleton leader in role-filtered mode.
/// </param>
/// <param name="hasRole">
/// Whether the local node carries <paramref name="requiredRole"/>. Ignored when
/// <paramref name="requiredRole"/> is <c>null</c>.
/// </param>
/// <param name="requiredRole">
/// The role to scope the check to, or <c>null</c> for the role-less (whole-cluster-leader) mode.
/// </param>
/// <returns>
/// Role-less: Healthy iff the node is Up and the cluster leader, otherwise Unhealthy.
/// Role-filtered: Healthy when the node lacks the role (probe irrelevant) or carries the role and
/// is the role-singleton leader; <strong>Unhealthy</strong> when it carries the role but is not
/// the leader.
/// </returns>
/// <remarks>
/// The role-member-but-not-leader case returns Unhealthy, not Degraded, because the active tier's
/// whole purpose is to be a 503 that an orchestrator can act on — the shared spec states it as
/// "Fails (503) on a standby or role-member-but-not-leader node", and the same spec maps Degraded
/// to 200. Returning Degraded made the tier answer 200 on every node, so a load balancer pointed
/// at it (OtOpcUa's Traefik routes the admin UI this way) kept the standby in the pool and its
/// leader-pinning silently never worked. Found on a live rig; see lmxopcua#494.
/// </remarks>
public static HealthStatus Evaluate(bool selfUp, bool isLeader, bool hasRole, string? requiredRole)
{
if (requiredRole is null)
return selfUp && isLeader ? HealthStatus.Healthy : HealthStatus.Unhealthy;
if (!hasRole)
return HealthStatus.Healthy;
return isLeader ? HealthStatus.Healthy : HealthStatus.Unhealthy;
}
}
/// <summary>
/// Health check that reports whether this node is the designated active / leader node.
/// An optional role scopes the check to nodes carrying that role. Register to the
/// <see cref="ZbHealthTags.Active"/> tag.
/// Active-tier probe: Healthy (200) on the one node in charge, Unhealthy (503) on its standby.
/// Register to the <see cref="ZbHealthTags.Active"/> tag.
/// </summary>
/// <remarks>
/// The <see cref="ActorSystem"/> is resolved lazily from the service provider. If it is not yet
/// available — e.g. during startup before Akka is initialised — the check returns
/// <see cref="HealthStatus.Degraded"/> rather than throwing, so it is startup-safe.
/// <para>
/// "In charge" means the <b>oldest Up member</b> — see <see cref="ClusterActiveNode"/> for
/// why leadership is the wrong answer and what breaks when it is used. Scope the competition
/// to a role with <see cref="ActiveNodeHealthCheckOptions.RolePreference"/>; leave it empty
/// for a cluster whose nodes all compete for the same single active slot.
/// </para>
/// <para>
/// <b>Behaviour change in 0.3.0.</b> Through 0.2.1 this check selected by
/// <c>ClusterState.Leader</c> / <c>RoleLeader</c>, and its role-filtered mode reported
/// Healthy for any node <i>lacking</i> the role. Both were wrong, both were found on live
/// rigs, and both Akka consumers had already replaced it with a private copy rather than use
/// it. It now selects by age and reports <see cref="HealthStatus.Unhealthy"/> for a node that
/// owns no active work. A node whose active tier previously answered 200 unconditionally will
/// now answer 503 unless it really is the active node — which is the point.
/// </para>
/// <para>
/// Startup-safe: before the <see cref="ActorSystem"/> or the cluster is available the result
/// is <see cref="HealthStatus.Degraded"/> (a 200 under the tier mapping), so a node that is
/// merely still booting is not reported as a failed standby and pulled from rotation on the
/// way up.
/// </para>
/// </remarks>
public sealed class ActiveNodeHealthCheck : IHealthCheck
{
private readonly IServiceProvider _serviceProvider;
private readonly string? _role;
private readonly ActiveNodeHealthCheckOptions _options;
/// <summary>
/// Role-less constructor: Healthy when the node is <c>Up</c> and the cluster leader
/// (ScadaBridge ActiveNode pattern); Unhealthy otherwise. Degraded when the ActorSystem /
/// cluster is not yet ready.
/// Unscoped: the active node is the oldest Up member of the whole cluster.
/// </summary>
/// <param name="serviceProvider">
/// The application service provider. The <see cref="ActorSystem"/> is resolved lazily so the
/// check is startup-safe: if no <see cref="ActorSystem"/> is registered yet the result is Degraded.
/// check is startup-safe.
/// </param>
public ActiveNodeHealthCheck(IServiceProvider serviceProvider)
: this(serviceProvider, new ActiveNodeHealthCheckOptions())
{
_serviceProvider = serviceProvider ?? throw new ArgumentNullException(nameof(serviceProvider));
_role = null;
}
/// <summary>
/// Role-filtered constructor: Healthy when the node lacks <paramref name="role"/> or carries it
/// and is the role-singleton leader; Unhealthy (503) when it carries the role but is not the
/// leader (OtOpcUa AdminRoleLeader pattern). Degraded when the ActorSystem / cluster is not yet
/// ready.
/// Role-scoped: the active node is the oldest Up member carrying <paramref name="role"/>.
/// </summary>
/// <param name="serviceProvider">
/// The application service provider. The <see cref="ActorSystem"/> is resolved lazily so the
/// check is startup-safe: if no <see cref="ActorSystem"/> is registered yet the result is Degraded.
/// check is startup-safe.
/// </param>
/// <param name="role">The Akka cluster role to scope the check to.</param>
/// <param name="role">The cluster role to scope the competition to.</param>
public ActiveNodeHealthCheck(IServiceProvider serviceProvider, string role)
: this(serviceProvider, NewOptions(role))
{
}
/// <summary>
/// Configured with an explicit <see cref="ActiveNodeHealthCheckOptions"/>.
/// </summary>
/// <param name="serviceProvider">
/// The application service provider. The <see cref="ActorSystem"/> is resolved lazily so the
/// check is startup-safe.
/// </param>
/// <param name="options">The role-scoping options.</param>
public ActiveNodeHealthCheck(IServiceProvider serviceProvider, ActiveNodeHealthCheckOptions options)
{
_serviceProvider = serviceProvider ?? throw new ArgumentNullException(nameof(serviceProvider));
_options = options ?? throw new ArgumentNullException(nameof(options));
}
private static ActiveNodeHealthCheckOptions NewOptions(string role)
{
ArgumentException.ThrowIfNullOrWhiteSpace(role);
_role = role;
return new ActiveNodeHealthCheckOptions { RolePreference = new[] { role } };
}
/// <inheritdoc />
@@ -113,10 +94,8 @@ public sealed class ActiveNodeHealthCheck : IHealthCheck
if (system is null)
return Task.FromResult(HealthCheckResult.Degraded("ActorSystem not yet available."));
MemberStatus selfStatus;
bool selfUp;
bool hasRole;
bool isLeader;
Member self;
IEnumerable<Member> members;
try
{
// Reading cluster membership can throw while the ActorSystem exists but the cluster has
@@ -124,54 +103,49 @@ public sealed class ActiveNodeHealthCheck : IHealthCheck
// ConfigurationException). The spec's startup-safety rule maps this to Degraded rather
// than letting the exception escape (which the host would record as Unhealthy).
var cluster = Cluster.Get(system);
var self = cluster.SelfMember;
selfStatus = self.Status;
selfUp = selfStatus == MemberStatus.Up;
if (_role is null)
{
hasRole = false;
var leader = cluster.State.Leader;
isLeader = leader is not null && leader == self.Address;
}
else
{
hasRole = self.HasRole(_role);
var roleLeader = cluster.State.RoleLeader(_role);
isLeader = roleLeader is not null && roleLeader == self.Address;
}
self = cluster.SelfMember;
members = cluster.State.Members;
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
return Task.FromResult(HealthCheckResult.Degraded("Akka cluster state not yet accessible.", ex));
}
var health = ActiveNodeDecision.Evaluate(selfUp, isLeader, hasRole, _role);
var description = DescribeResult(health, selfStatus, selfUp, isLeader);
var result = health switch
{
HealthStatus.Healthy => HealthCheckResult.Healthy(description),
HealthStatus.Degraded => HealthCheckResult.Degraded(description),
_ => HealthCheckResult.Unhealthy(description),
};
return Task.FromResult(result);
}
var scoped = _options.RolePreference.Count > 0;
var role = ClusterActiveNode.ResolveActiveRole(self.Roles, _options.RolePreference);
private string DescribeResult(HealthStatus health, MemberStatus status, bool selfUp, bool isLeader)
{
if (_role is null)
var data = new Dictionary<string, object>(StringComparer.Ordinal)
{
if (health == HealthStatus.Healthy)
return "Active node (cluster leader).";
return selfUp && !isLeader
? "Standby: node is Up but not the cluster leader."
: $"Standby: node is not Up (status: {status}).";
["selfAddress"] = self.Address.ToString(),
};
if (role is not null) data["activeRole"] = role;
if (scoped && role is null)
{
// The node carries none of the preferred roles, so it owns no active work and cannot be
// the active node for any of them. Reported per NoActiveRoleStatus — Unhealthy by
// default, because the active tier's whole job is to keep such a node out of the pool.
var message = $"Node carries none of the active roles: {string.Join(", ", _options.RolePreference)}.";
return Task.FromResult(new HealthCheckResult(_options.NoActiveRoleStatus, message, exception: null, data));
}
return health switch
var active = ClusterActiveNode.OldestUpMember(members, role);
if (active is not null) data["activeNode"] = active.Address.ToString();
var scopeLabel = role is null ? "the cluster" : $"role '{role}'";
if (active is null)
{
HealthStatus.Healthy => $"Active for role '{_role}' (or not a role member).",
_ => $"Role '{_role}' member but not leader.",
};
// Nobody is Up in the scope — a cold start, or every holder is still Joining or already
// Leaving. Nobody is active, so nobody may claim to be.
return Task.FromResult(HealthCheckResult.Unhealthy(
$"No Up member currently holds the active slot for {scopeLabel}.", exception: null, data));
}
// Compared by UniqueAddress: a node that restarted on the same host and port is a different
// member, and during the overlap the cluster briefly holds both incarnations.
return Task.FromResult(active.UniqueAddress.Equals(self.UniqueAddress)
? HealthCheckResult.Healthy($"Active node for {scopeLabel} (oldest Up member).", data)
: HealthCheckResult.Unhealthy($"Standby: active node for {scopeLabel} is {active.Address}.", exception: null, data));
}
}
ZB.MOM.WW.Health/src/ZB.MOM.WW.Health.Akka/ActiveNodeHealthCheckOptions.cs
+37
View File
@@ -0,0 +1,37 @@
using Microsoft.Extensions.Diagnostics.HealthChecks;
namespace ZB.MOM.WW.Health.Akka;
/// <summary>
/// Configures how <see cref="ActiveNodeHealthCheck"/> decides which role's active node this node is
/// competing to be.
/// </summary>
public sealed class ActiveNodeHealthCheckOptions
{
/// <summary>
/// Candidate roles in descending precedence. The check answers for the first one the local node
/// carries. Empty (the default) means unscoped: the active node is the oldest Up member of the
/// whole cluster.
/// </summary>
/// <remarks>
/// A fused node should list the role its singletons are pinned to first — e.g.
/// <c>["admin", "driver"]</c> for a node that hosts the admin singletons when fused and competes
/// as a plain driver when it is a driver-only node.
/// </remarks>
public IReadOnlyList<string> RolePreference { get; init; } = Array.Empty<string>();
/// <summary>
/// The status to report when <see cref="RolePreference"/> is non-empty and the node carries none
/// of those roles — it owns no active work at all. Defaults to
/// <see cref="HealthStatus.Unhealthy"/>.
/// </summary>
/// <remarks>
/// Unhealthy is the safe default and is deliberately <b>not</b> the "not applicable ⇒ Healthy"
/// convention used by readiness probes. The active tier exists so a load balancer can pick
/// exactly one node; a node that owns no active work must not be in that pool. Answering Healthy
/// there is what made the previous role-filtered check report 200 on every node and silently
/// broke leader-pinning (lmxopcua#494). Override only when a node genuinely should stay in the
/// active pool without holding any of the preferred roles.
/// </remarks>
public HealthStatus NoActiveRoleStatus { get; init; } = HealthStatus.Unhealthy;
}
ZB.MOM.WW.Health/src/ZB.MOM.WW.Health.Akka/AkkaActiveNodeGate.cs
+36 -13
View File
@@ -6,27 +6,47 @@ namespace ZB.MOM.WW.Health.Akka;
/// <summary>
/// <see cref="IActiveNodeGate"/> implementation that computes <see cref="IsActiveNode"/> directly
/// from the Akka cluster state (self member <c>Up</c> and the local node is the cluster leader).
/// Register as a singleton.
/// from the Akka cluster state — the local node is the <b>oldest Up member</b>, optionally within a
/// role scope. Register as a singleton.
/// </summary>
/// <remarks>
/// The <see cref="ActorSystem"/> is resolved lazily from the service provider; if it is not yet
/// available — e.g. during startup before Akka is initialised — <see cref="IsActiveNode"/> returns
/// <c>false</c> (the safe default during startup). This gate reads the cluster state directly and
/// does not resolve <see cref="ActiveNodeHealthCheck"/> from DI.
/// <para>
/// Shares its rule with <see cref="ActiveNodeHealthCheck"/> via
/// <see cref="ClusterActiveNode"/>, so an endpoint gated by this type and the
/// <c>/health/active</c> probe an orchestrator routes by can never disagree about which node
/// is in charge.
/// </para>
/// <para>
/// <b>Behaviour change in 0.3.0:</b> through 0.2.1 this gate selected by
/// <c>ClusterState.Leader</c>, which is address-ordered and diverges from singleton placement
/// after any restart. See <see cref="ClusterActiveNode"/>.
/// </para>
/// <para>
/// The <see cref="ActorSystem"/> is resolved lazily from the service provider; if it is not
/// yet available — e.g. during startup before Akka is initialised —
/// <see cref="IsActiveNode"/> returns <c>false</c> (the safe default during startup, matching
/// the standby case). This gate reads the cluster state directly and does not resolve
/// <see cref="ActiveNodeHealthCheck"/> from DI.
/// </para>
/// </remarks>
public sealed class AkkaActiveNodeGate : IActiveNodeGate
{
private readonly IServiceProvider _serviceProvider;
private readonly string? _role;
/// <summary>Initializes a new <see cref="AkkaActiveNodeGate"/>.</summary>
/// <param name="serviceProvider">
/// The application service provider. The <see cref="ActorSystem"/> is resolved lazily; if it is
/// not yet available <see cref="IsActiveNode"/> returns <c>false</c>.
/// </param>
public AkkaActiveNodeGate(IServiceProvider serviceProvider)
/// <param name="role">
/// The role to scope the competition to, or <c>null</c> (the default) to consider every Up
/// member.
/// </param>
public AkkaActiveNodeGate(IServiceProvider serviceProvider, string? role = null)
{
_serviceProvider = serviceProvider ?? throw new ArgumentNullException(nameof(serviceProvider));
_role = role;
}
/// <inheritdoc />
@@ -38,13 +58,16 @@ public sealed class AkkaActiveNodeGate : IActiveNodeGate
if (system is null)
return false;
var cluster = Cluster.Get(system);
var self = cluster.SelfMember;
if (self.Status != MemberStatus.Up)
try
{
return ClusterActiveNode.SelfIsActive(Cluster.Get(system), _role);
}
catch (Exception)
{
// ActorSystem exists but the cluster is not initialised yet. Same startup-safety
// rule as the health check, expressed as the gate's safe default: not active.
return false;
var leader = cluster.State.Leader;
return leader is not null && leader == self.Address;
}
}
}
}
ZB.MOM.WW.Health/src/ZB.MOM.WW.Health.Akka/ClusterActiveNode.cs
+133
View File
@@ -0,0 +1,133 @@
using Akka.Cluster;
namespace ZB.MOM.WW.Health.Akka;
/// <summary>
/// THE family-wide definition of "the active node": the <b>oldest Up member</b> of the cluster,
/// optionally scoped to a role — the member <c>ClusterSingletonManager</c> places singletons on.
/// </summary>
/// <remarks>
/// <para>
/// Cluster <b>leadership</b> (<c>ClusterState.Leader</c> / <c>RoleLeader</c>) is the wrong
/// answer and must never be used for a product-level active/standby decision. Leadership is
/// <i>address</i>-ordered (host, then port) and has no relationship to time; singleton
/// placement is <i>age</i>-ordered. On a freshly-formed cluster the two agree, which is why
/// the discrepancy hides in unit tests and in a rig that has never been restarted. They
/// diverge permanently after any restart: the restarted node rejoins as the youngest but
/// keeps its address, so if it holds the lower address it becomes leader while the
/// singletons — and all the work they own — stay on the other node.
/// </para>
/// <para>
/// Both Akka consumers in the family independently hit this and each wrote its own
/// replacement rather than use the shared leader-based check
/// (ScadaBridge <c>OldestNodeActiveHealthCheck</c>, OtOpcUa <c>ClusterPrimaryHealthCheck</c>).
/// This type is those two implementations promoted to one, so the health tier, the inbound
/// API gate, the store-and-forward delivery gate and OtOpcUa's OPC UA <c>ServiceLevel</c>
/// cannot drift apart about which node is in charge.
/// </para>
/// <para>
/// Only <see cref="MemberStatus.Up"/> members are eligible, matching singleton placement: a
/// <c>Leaving</c> node is handing its singletons over and must not be named active.
/// </para>
/// </remarks>
public static class ClusterActiveNode
{
/// <summary>
/// The oldest <see cref="MemberStatus.Up"/> member, optionally scoped to <paramref name="role"/>.
/// </summary>
/// <param name="members">The cluster members, in any order.</param>
/// <param name="role">
/// The role to scope the selection to, or <c>null</c> to consider every Up member.
/// </param>
/// <returns>The oldest eligible member, or <c>null</c> when none is Up.</returns>
public static Member? OldestUpMember(IEnumerable<Member> members, string? role = null)
{
ArgumentNullException.ThrowIfNull(members);
Member? oldest = null;
foreach (var member in members)
{
if (member.Status != MemberStatus.Up) continue;
if (role is not null && !member.HasRole(role)) continue;
if (oldest is null || member.IsOlderThan(oldest)) oldest = member;
}
return oldest;
}
/// <summary>
/// The oldest <see cref="MemberStatus.Up"/> member of <paramref name="cluster"/>, optionally
/// scoped to <paramref name="role"/>.
/// </summary>
/// <param name="cluster">The Akka cluster to evaluate.</param>
/// <param name="role">
/// The role to scope the selection to, or <c>null</c> to consider every Up member.
/// </param>
/// <returns>The oldest eligible member, or <c>null</c> when none is Up.</returns>
public static Member? OldestUpMember(Cluster cluster, string? role = null)
{
ArgumentNullException.ThrowIfNull(cluster);
return OldestUpMember(cluster.State.Members, role);
}
/// <summary>
/// Whether the local node is the active node — Up, and the oldest Up member in the role scope.
/// </summary>
/// <param name="cluster">The Akka cluster to evaluate.</param>
/// <param name="role">
/// The role to scope the selection to, or <c>null</c> to consider every Up member.
/// </param>
/// <returns>
/// <c>true</c> when the local member is Up, carries <paramref name="role"/> when one is given,
/// and no other eligible member is older.
/// </returns>
/// <remarks>
/// Identity is compared by <see cref="Member.UniqueAddress"/>, not <c>Address</c>: a node that
/// restarted on the same host and port is a different member, and during the overlap the cluster
/// briefly holds both incarnations.
/// </remarks>
public static bool SelfIsActive(Cluster cluster, string? role = null)
{
ArgumentNullException.ThrowIfNull(cluster);
var self = cluster.SelfMember;
if (self.Status != MemberStatus.Up) return false;
if (role is not null && !self.HasRole(role)) return false;
var oldest = OldestUpMember(cluster.State.Members, role);
return oldest is not null && oldest.UniqueAddress.Equals(self.UniqueAddress);
}
/// <summary>
/// The first role in <paramref name="rolePreference"/> that <paramref name="selfRoles"/> carries
/// — the role whose oldest Up member owns this node's active work.
/// </summary>
/// <param name="selfRoles">The local member's cluster roles.</param>
/// <param name="rolePreference">
/// Candidate roles in descending precedence, e.g. <c>["admin", "driver"]</c> for a node that
/// answers for <c>admin</c> when it is fused and for <c>driver</c> when it is a driver-only node.
/// </param>
/// <returns>
/// The first carried role, or <c>null</c> when the node carries none of them.
/// </returns>
/// <remarks>
/// Precedence matters on a fused node: it must answer for the role its singletons — and the
/// traffic being routed to it — are pinned to. Answering for a lower-precedence role could name a
/// different node the moment a single-role member joins the mesh.
/// </remarks>
public static string? ResolveActiveRole(IEnumerable<string> selfRoles, IReadOnlyList<string> rolePreference)
{
ArgumentNullException.ThrowIfNull(selfRoles);
ArgumentNullException.ThrowIfNull(rolePreference);
if (rolePreference.Count == 0) return null;
var roles = selfRoles as IReadOnlyCollection<string> ?? selfRoles.ToList();
foreach (var candidate in rolePreference)
{
if (roles.Contains(candidate)) return candidate;
}
return null;
}
}
ZB.MOM.WW.Health/tests/ZB.MOM.WW.Health.Akka.Tests/ActiveNodeDecisionTests.cs
-139
View File
@@ -1,139 +0,0 @@
using Akka.Actor;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Diagnostics.HealthChecks;
using ZB.MOM.WW.Health.Akka;
namespace ZB.MOM.WW.Health.Akka.Tests;
/// <summary>
/// Table-driven tests for the pure <see cref="ActiveNodeDecision.Evaluate"/> helper covering both
/// the role-less (ScadaBridge ActiveNode) and role-filtered (OtOpcUa AdminRoleLeader) matrices,
/// plus the startup-safety null-guards on <see cref="ActiveNodeHealthCheck"/> and
/// <see cref="AkkaActiveNodeGate"/> when no <see cref="ActorSystem"/> is registered.
/// </summary>
public sealed class ActiveNodeDecisionTests
{
// Role-less: requiredRole == null. hasRole is irrelevant. Healthy iff (selfUp && isLeader), else Unhealthy.
public static IEnumerable<object?[]> RoleLessCases() => new[]
{
new object?[] { true, true, false, (string?)null, HealthStatus.Healthy },
new object?[] { true, false, false, (string?)null, HealthStatus.Unhealthy },
new object?[] { false, true, false, (string?)null, HealthStatus.Unhealthy },
new object?[] { false, false, false, (string?)null, HealthStatus.Unhealthy },
};
[Theory]
[MemberData(nameof(RoleLessCases))]
public void Evaluate_RoleLess(bool selfUp, bool isLeader, bool hasRole, string? requiredRole, HealthStatus expected)
{
Assert.Equal(expected, ActiveNodeDecision.Evaluate(selfUp, isLeader, hasRole, requiredRole));
}
// Role-filtered: requiredRole != null.
// lacks role -> Healthy (probe irrelevant for this node)
// has role & is leader -> Healthy (selfUp is ignored — role-filtered mode only cares about leadership)
// has role & not leader -> Unhealthy (503) — the tier exists to be a 503 an LB can act on
public static IEnumerable<object[]> RoleFilteredCases() => new[]
{
// node lacks the role -> Healthy regardless of selfUp / isLeader
new object[] { true, true, false, "admin", HealthStatus.Healthy },
new object[] { true, false, false, "admin", HealthStatus.Healthy },
new object[] { false, false, false, "admin", HealthStatus.Healthy },
// node carries the role and is leader -> Healthy (selfUp=true)
new object[] { true, true, true, "admin", HealthStatus.Healthy },
// node carries the role and is leader -> Healthy (selfUp=false: role-filtered mode ignores selfUp)
new object[] { false, true, true, "admin", HealthStatus.Healthy },
// Node carries the role but is not leader -> Unhealthy, i.e. HTTP 503.
// Degraded would be a 200 (see the shared health SPEC), which made the active tier answer
// 200 on every node and silently broke OtOpcUa's Traefik leader-pinning. lmxopcua#494.
new object[] { true, false, true, "admin", HealthStatus.Unhealthy },
new object[] { false, false, true, "admin", HealthStatus.Unhealthy },
};
[Theory]
[MemberData(nameof(RoleFilteredCases))]
public void Evaluate_RoleFiltered(bool selfUp, bool isLeader, bool hasRole, string? requiredRole, HealthStatus expected)
{
Assert.Equal(expected, ActiveNodeDecision.Evaluate(selfUp, isLeader, hasRole, requiredRole));
}
[Fact]
public async Task HealthCheck_RoleLess_NoActorSystem_ReturnsDegraded()
{
var provider = new ServiceCollection().BuildServiceProvider();
var check = new ActiveNodeHealthCheck(provider);
var result = await check.CheckHealthAsync(NewContext(check));
Assert.Equal(HealthStatus.Degraded, result.Status);
}
[Fact]
public async Task HealthCheck_RoleFiltered_NoActorSystem_ReturnsDegraded()
{
var provider = new ServiceCollection().BuildServiceProvider();
var check = new ActiveNodeHealthCheck(provider, "admin");
var result = await check.CheckHealthAsync(NewContext(check));
Assert.Equal(HealthStatus.Degraded, result.Status);
}
[Fact]
public void Gate_NoActorSystem_IsActiveNodeFalse()
{
var provider = new ServiceCollection().BuildServiceProvider();
var gate = new AkkaActiveNodeGate(provider);
Assert.False(gate.IsActiveNode);
}
[Fact]
public async Task HealthCheck_RoleLess_ClusterInaccessible_ReturnsDegraded()
{
// ActorSystem present but Akka.Cluster not configured → Cluster.Get throws. The check must
// return Degraded (startup-safety rule), not let the exception escape (→ Unhealthy).
using var system = ActorSystem.Create("plain-no-cluster-roleless");
try
{
var provider = new ServiceCollection()
.AddSingleton(system)
.BuildServiceProvider();
var check = new ActiveNodeHealthCheck(provider);
var result = await check.CheckHealthAsync(NewContext(check));
Assert.Equal(HealthStatus.Degraded, result.Status);
}
finally
{
await system.Terminate();
}
}
[Fact]
public async Task HealthCheck_RoleFiltered_ClusterInaccessible_ReturnsDegraded()
{
using var system = ActorSystem.Create("plain-no-cluster-rolefiltered");
try
{
var provider = new ServiceCollection()
.AddSingleton(system)
.BuildServiceProvider();
var check = new ActiveNodeHealthCheck(provider, "admin");
var result = await check.CheckHealthAsync(NewContext(check));
Assert.Equal(HealthStatus.Degraded, result.Status);
}
finally
{
await system.Terminate();
}
}
private static HealthCheckContext NewContext(IHealthCheck check) => new()
{
Registration = new HealthCheckRegistration("active-node", check, HealthStatus.Unhealthy, tags: null),
};
}
ZB.MOM.WW.Health/tests/ZB.MOM.WW.Health.Akka.Tests/ActiveNodeStartupSafetyTests.cs
+113
View File
@@ -0,0 +1,113 @@
using Akka.Actor;
using Akka.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Diagnostics.HealthChecks;
using ZB.MOM.WW.Health.Akka;
namespace ZB.MOM.WW.Health.Akka.Tests;
/// <summary>
/// The states a node passes through on the way up, where the active tier must not report a booting
/// node as a failed standby — now that the tier is a real 503, an Unhealthy here would pull the node
/// out of an orchestrator's pool while it starts.
/// </summary>
public sealed class ActiveNodeStartupSafetyTests
{
[Fact]
public async Task NoActorSystem_IsDegraded()
{
var check = new ActiveNodeHealthCheck(new ServiceCollection().BuildServiceProvider());
var result = await check.CheckHealthAsync(NewContext(check));
Assert.Equal(HealthStatus.Degraded, result.Status);
// Description-only: an empty data dictionary is what makes the writer omit "data" entirely.
Assert.Empty(result.Data);
}
[Fact]
public async Task NoActorSystem_RoleScoped_IsDegraded()
{
var check = new ActiveNodeHealthCheck(new ServiceCollection().BuildServiceProvider(), "admin");
Assert.Equal(HealthStatus.Degraded, (await check.CheckHealthAsync(NewContext(check))).Status);
}
[Fact]
public void Gate_NoActorSystem_IsNotActive()
{
Assert.False(new AkkaActiveNodeGate(new ServiceCollection().BuildServiceProvider()).IsActiveNode);
}
[Fact]
public async Task ClusterNotConfigured_IsDegraded()
{
// ActorSystem present but Akka.Cluster not configured → Cluster.Get throws. The check must
// return Degraded (the spec's startup-safety rule), not let the exception escape (→ Unhealthy).
using var system = ActorSystem.Create("active-node-no-cluster");
try
{
var provider = new ServiceCollection().AddSingleton(system).BuildServiceProvider();
var check = new ActiveNodeHealthCheck(provider);
Assert.Equal(HealthStatus.Degraded, (await check.CheckHealthAsync(NewContext(check))).Status);
}
finally
{
await system.Terminate();
}
}
[Fact]
public async Task Gate_ClusterNotConfigured_IsNotActive()
{
using var system = ActorSystem.Create("active-node-gate-no-cluster");
try
{
var provider = new ServiceCollection().AddSingleton(system).BuildServiceProvider();
Assert.False(new AkkaActiveNodeGate(provider).IsActiveNode);
}
finally
{
await system.Terminate();
}
}
[Fact]
public async Task ClusteredButNeverJoined_IsUnhealthy_NobodyIsActive()
{
// A cold start: the cluster provider is up but no member has reached Up. Nobody is active, so
// nobody may claim to be — this is the one case that must NOT be Degraded, because a cluster
// that never forms would otherwise sit at 200 forever.
var config = ConfigurationFactory.ParseString("""
akka {
loglevel = "WARNING"
actor.provider = cluster
remote.dot-netty.tcp { hostname = "127.0.0.1", port = 0 }
cluster.seed-nodes = []
}
""");
var system = ActorSystem.Create("active-node-never-joined", config);
try
{
var provider = new ServiceCollection().AddSingleton(system).BuildServiceProvider();
var check = new ActiveNodeHealthCheck(provider);
var result = await check.CheckHealthAsync(NewContext(check));
Assert.Equal(HealthStatus.Unhealthy, result.Status);
Assert.Contains("No Up member", result.Description);
Assert.False(new AkkaActiveNodeGate(provider).IsActiveNode);
}
finally
{
await system.Terminate();
}
}
private static HealthCheckContext NewContext(IHealthCheck check) => new()
{
Registration = new HealthCheckRegistration("active-node", check, HealthStatus.Unhealthy, tags: null),
};
}
ZB.MOM.WW.Health/tests/ZB.MOM.WW.Health.Akka.Tests/ActiveRoleResolutionTests.cs
+56
View File
@@ -0,0 +1,56 @@
using ZB.MOM.WW.Health.Akka;
namespace ZB.MOM.WW.Health.Akka.Tests;
/// <summary>
/// Covers <see cref="ClusterActiveNode.ResolveActiveRole"/> — which role a node competes for. Pure,
/// so no cluster is needed; the age-ordering half of the rule is pinned in
/// <see cref="ClusterActiveNodeTests"/> against a real two-node cluster.
/// </summary>
public sealed class ActiveRoleResolutionTests
{
[Fact]
public void EmptyPreference_IsUnscoped()
{
// Unscoped: every Up member competes for one active slot (the ScadaBridge central pattern).
Assert.Null(ClusterActiveNode.ResolveActiveRole(["admin", "driver"], []));
}
[Fact]
public void FusedNode_AnswersForTheHighestPrecedenceRoleItCarries()
{
Assert.Equal(
"admin",
ClusterActiveNode.ResolveActiveRole(["admin", "driver", "cluster-MAIN"], ["admin", "driver"]));
}
[Fact]
public void SingleRoleNode_AnswersForThatRole()
{
// The regression the OtOpcUa fix existed for: under the old role-filtered check every
// driver-only node reported Healthy — "or not a role member" — so all of them called
// themselves active and no consumer could find the active node of a site cluster. lmxopcua#494.
Assert.Equal(
"driver",
ClusterActiveNode.ResolveActiveRole(["driver", "cluster-SITE-A"], ["admin", "driver"]));
}
[Fact]
public void PreferenceOrderWins_NotTheOrderTheNodeListsItsRoles()
{
// Self-role order is incidental (it comes from config parsing); precedence must come from the
// caller's list alone, or a fused node's answer would depend on how its roles were spelled.
Assert.Equal(
"admin",
ClusterActiveNode.ResolveActiveRole(["driver", "admin"], ["admin", "driver"]));
}
[Fact]
public void NodeCarryingNoneOfThePreferredRoles_ResolvesToNull()
{
// Reachable, not merely defensive: OtOpcUa's RoleParser admits a dev-only or cluster-role-only
// node. Such a node owns no active work — the health check maps this to Unhealthy so it stays
// out of the active pool.
Assert.Null(ClusterActiveNode.ResolveActiveRole(["dev", "cluster-MAIN"], ["admin", "driver"]));
}
}
ZB.MOM.WW.Health/tests/ZB.MOM.WW.Health.Akka.Tests/ClusterActiveNodeTests.cs
+237
View File
@@ -0,0 +1,237 @@
using Akka.Actor;
using Akka.Cluster;
using Akka.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Diagnostics.HealthChecks;
using ZB.MOM.WW.Health.Akka;
namespace ZB.MOM.WW.Health.Akka.Tests;
/// <summary>
/// Pins <b>which</b> node the active tier names, against a real two-node cluster deliberately built
/// so the oldest member is <i>not</i> the lowest-addressed one.
/// </summary>
/// <remarks>
/// <para>
/// The divergence is the whole point. Akka offers two different "first" members:
/// <c>ClusterState.Leader</c> / <c>RoleLeader</c> is the <b>lowest-addressed</b> Up member
/// (host, then port — nothing to do with time), while <b>oldest</b> is the member with the
/// lowest up-number, which is what <c>ClusterSingletonManager</c> uses to place singletons.
/// </para>
/// <para>
/// They coincide on a freshly-formed cluster, which is why every single-node and happy-path
/// test missed it and why the leader-based check survived to 0.2.1. They diverge the moment a
/// node restarts: the restarted node rejoins as the youngest while keeping its address, so if
/// it holds the lower address it becomes leader while the singletons — and every piece of work
/// they own — stay on the other node. This fixture reproduces that state directly instead of
/// waiting for a restart.
/// </para>
/// </remarks>
public sealed class ClusterActiveNodeTests : IAsyncLifetime
{
// Fixed, unusual ports: the test needs a deterministic address ordering, which port 0 cannot give.
// Distinct from OtOpcUa's equivalent fixture (19_530/19_531) so the two suites can run at once.
private const int OldestPort = 19_541; // joins FIRST -> oldest, but the HIGHER address
private const int YoungestPort = 19_540; // joins SECOND -> cluster leader, the LOWER address
private const string SystemName = "health-active-node";
private ActorSystem? _oldest;
private ActorSystem? _youngest;
private static Config NodeConfig(int port, string roles) => ConfigurationFactory.ParseString($$"""
akka {
loglevel = "WARNING"
actor.provider = "Akka.Cluster.ClusterActorRefProvider, Akka.Cluster"
remote.dot-netty.tcp {
hostname = "127.0.0.1"
port = {{port}}
}
cluster {
seed-nodes = ["akka.tcp://{{SystemName}}@127.0.0.1:{{OldestPort}}"]
roles = [{{roles}}]
min-nr-of-members = 1
run-coordinated-shutdown-when-down = off
downing-provider-class = ""
}
}
""");
public async Task InitializeAsync()
{
// Order matters: the seed forms the cluster and is therefore the oldest member. The oldest
// node is fused (admin + driver); the younger one is driver-only, so the role-scoped
// assertions below can distinguish "oldest of this role" from "oldest overall".
_oldest = ActorSystem.Create(SystemName, NodeConfig(OldestPort, "\"admin\", \"driver\""));
await WaitForUpAsync(_oldest, expectedMembers: 1);
_youngest = ActorSystem.Create(SystemName, NodeConfig(YoungestPort, "\"driver\""));
await WaitForUpAsync(_oldest, expectedMembers: 2);
await WaitForUpAsync(_youngest, expectedMembers: 2);
}
public async Task DisposeAsync()
{
if (_youngest is not null) await _youngest.Terminate();
if (_oldest is not null) await _oldest.Terminate();
}
private static async Task WaitForUpAsync(ActorSystem system, int expectedMembers)
{
var cluster = Cluster.Get(system);
var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(30);
while (DateTime.UtcNow < deadline)
{
if (cluster.State.Members.Count(m => m.Status == MemberStatus.Up) >= expectedMembers) return;
await Task.Delay(100);
}
throw new TimeoutException(
$"cluster on port {cluster.SelfAddress.Port} never saw {expectedMembers} Up members "
+ $"(saw {cluster.State.Members.Count(m => m.Status == MemberStatus.Up)})");
}
/// <summary>
/// Sanity check on the fixture itself: if the two orderings did not actually diverge, every
/// assertion below would pass for the wrong reason.
/// </summary>
[Fact]
public void Fixture_actually_produces_divergent_age_and_address_ordering()
{
var state = Cluster.Get(_oldest!).State;
Assert.Equal(YoungestPort, state.Leader!.Port);
Assert.Equal(YoungestPort, state.RoleLeader("driver")!.Port);
}
[Fact]
public void OldestUpMember_IsTheOldest_NotTheLeader()
{
var members = Cluster.Get(_oldest!).State.Members;
Assert.Equal(OldestPort, ClusterActiveNode.OldestUpMember(members)!.Address.Port);
Assert.Equal(OldestPort, ClusterActiveNode.OldestUpMember(members, "driver")!.Address.Port);
}
[Fact]
public void OldestUpMember_IsRoleScoped()
{
var members = Cluster.Get(_oldest!).State.Members;
// Only the fused node carries admin, so it owns the admin slot even though it is not the
// lowest-addressed member.
Assert.Equal(OldestPort, ClusterActiveNode.OldestUpMember(members, "admin")!.Address.Port);
// A role nobody carries has no owner.
Assert.Null(ClusterActiveNode.OldestUpMember(members, "no-such-role"));
}
[Fact]
public void SelfIsActive_IsTrueOnlyOnTheOldestNode()
{
Assert.True(ClusterActiveNode.SelfIsActive(Cluster.Get(_oldest!)));
Assert.False(ClusterActiveNode.SelfIsActive(Cluster.Get(_youngest!)));
}
[Fact]
public void SelfIsActive_RoleScoped_YoungestOwnsNoSlot()
{
// Driver-scoped: the fused node is still the oldest driver, so the driver-only node is standby.
Assert.True(ClusterActiveNode.SelfIsActive(Cluster.Get(_oldest!), "driver"));
Assert.False(ClusterActiveNode.SelfIsActive(Cluster.Get(_youngest!), "driver"));
// Admin-scoped: the driver-only node does not carry admin, so it can never be the admin active
// node — even though it is the cluster leader.
Assert.False(ClusterActiveNode.SelfIsActive(Cluster.Get(_youngest!), "admin"));
}
[Fact]
public async Task HealthCheck_IsHealthyOnTheOldestAndUnhealthyOnTheLeader()
{
var onOldest = await CheckAsync(_oldest!, new ActiveNodeHealthCheckOptions());
var onYoungest = await CheckAsync(_youngest!, new ActiveNodeHealthCheckOptions());
Assert.Equal(HealthStatus.Healthy, onOldest.Status);
// The node the old leader-based check would have called active. Exactly one 200 per cluster is
// the property an orchestrator's active-tier routing depends on.
Assert.Equal(HealthStatus.Unhealthy, onYoungest.Status);
}
[Fact]
public async Task HealthCheck_DataNamesTheActiveNode_EvenFromAStandby()
{
// A standby reporting *who* is active is what lets a dashboard render the pair from either
// node's payload alone (the 0.2.0 per-entry data object).
var result = await CheckAsync(_youngest!, new ActiveNodeHealthCheckOptions());
Assert.Equal($"akka.tcp://{SystemName}@127.0.0.1:{OldestPort}", result.Data["activeNode"]);
Assert.Equal($"akka.tcp://{SystemName}@127.0.0.1:{YoungestPort}", result.Data["selfAddress"]);
}
[Fact]
public async Task HealthCheck_RolePreference_FusedNodeAnswersForAdmin()
{
var options = new ActiveNodeHealthCheckOptions { RolePreference = ["admin", "driver"] };
var onOldest = await CheckAsync(_oldest!, options);
var onYoungest = await CheckAsync(_youngest!, options);
Assert.Equal(HealthStatus.Healthy, onOldest.Status);
Assert.Equal("admin", onOldest.Data["activeRole"]);
// Resolves to driver (it has no admin role) and loses to the older fused node.
Assert.Equal(HealthStatus.Unhealthy, onYoungest.Status);
Assert.Equal("driver", onYoungest.Data["activeRole"]);
}
[Fact]
public async Task HealthCheck_NodeOwningNoneOfThePreferredRoles_IsUnhealthyByDefault()
{
var options = new ActiveNodeHealthCheckOptions { RolePreference = ["no-such-role"] };
var result = await CheckAsync(_oldest!, options);
// Not "not applicable ⇒ Healthy": a node that owns no active work must stay out of the pool.
Assert.Equal(HealthStatus.Unhealthy, result.Status);
Assert.Contains("carries none of the active roles", result.Description);
Assert.DoesNotContain("activeRole", result.Data.Keys);
}
[Fact]
public async Task HealthCheck_NoActiveRoleStatus_IsOverridable()
{
var options = new ActiveNodeHealthCheckOptions
{
RolePreference = ["no-such-role"],
NoActiveRoleStatus = HealthStatus.Healthy,
};
Assert.Equal(HealthStatus.Healthy, (await CheckAsync(_oldest!, options)).Status);
}
[Fact]
public void Gate_IsActiveOnlyOnTheOldestNode()
{
Assert.True(new AkkaActiveNodeGate(ProviderFor(_oldest!)).IsActiveNode);
Assert.False(new AkkaActiveNodeGate(ProviderFor(_youngest!)).IsActiveNode);
// Role-scoped: the driver-only node cannot gate admin work.
Assert.False(new AkkaActiveNodeGate(ProviderFor(_youngest!), "admin").IsActiveNode);
}
private static IServiceProvider ProviderFor(ActorSystem system) =>
new ServiceCollection().AddSingleton(system).BuildServiceProvider();
private static async Task<HealthCheckResult> CheckAsync(
ActorSystem system,
ActiveNodeHealthCheckOptions options)
{
var check = new ActiveNodeHealthCheck(ProviderFor(system), options);
var context = new HealthCheckContext
{
Registration = new HealthCheckRegistration("active-node", check, HealthStatus.Unhealthy, tags: null),
};
return await check.CheckHealthAsync(context);
}
}