dohertj2/scadaproj
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(health)!: active node is the oldest Up member, not the leader (0.3.0)

Browse Source 
Promotes the two apps' private replacements for ActiveNodeHealthCheck into one
shared primitive, and retires the leader/RoleLeader selection they were written
to avoid.

Both Akka consumers had already hand-rolled a replacement rather than use this
package (ScadaBridge OldestNodeActiveHealthCheck, OtOpcUa
ClusterPrimaryHealthCheck), so the entire active-node surface here —
ActiveNodeHealthCheck, AkkaActiveNodeGate — had ZERO consumers. It was not
merely unused: it was avoided, twice, for the same reason.

Leadership is address-ordered (host, then port) and has no relationship to time;
singleton placement is age-ordered. The two agree on a freshly-formed cluster,
which is why single-node and happy-path tests never caught it. They diverge
permanently after any restart: the restarted node rejoins as the youngest but
keeps its address, so if it holds the lower address it becomes leader while the
singletons — and all the work they own — stay on the other node.

New ClusterActiveNode is the single implementation: oldest Up member, optional
role scope, plus role-preference resolution for a fused node that must answer
for the role its singletons are pinned to. ActiveNodeHealthCheck and
AkkaActiveNodeGate both delegate to it, so an endpoint gate and the
/health/active probe an orchestrator routes by cannot disagree.

BREAKING (behaviour, not signature):
- Selection is by age, not leadership.
- The role-filtered mode no longer reports Healthy for a node LACKING the role.
  That "not applicable => Healthy" made the tier answer 200 on every node and
  silently broke leader-pinning (lmxopcua#494); it is now Unhealthy by default,
  overridable via NoActiveRoleStatus. This case is reachable, not defensive —
  OtOpcUa's RoleParser admits dev-only and cluster-role-only nodes.
- Identity compares UniqueAddress, so a node restarted on the same host:port is
  correctly a different member during the overlap.

Results now carry activeRole/selfAddress/activeNode in the 0.2.0 per-entry data
object, so a standby reports WHO is active and a dashboard can render a pair
from either node's payload alone.

Tests: 45 Akka (was 39), 76 total. The age-vs-address divergence is pinned
against a real two-node cluster built so the oldest member is not the
lowest-addressed one, with a fixture sanity check so it cannot pass for the
wrong reason.
This commit is contained in:
Joseph Doherty
2026-07-24 13:07:56 -04:00
parent c81a6615ba
commit a61e041e58
11 changed files with 1332 additions and 263 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ZB.MOM.WW.Health/tests/ZB.MOM.WW.Health.Akka.Tests/ActiveNodeDecisionTests.cs
-139
View File
@@ -1,139 +0,0 @@
using Akka.Actor;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Diagnostics.HealthChecks;
using ZB.MOM.WW.Health.Akka;
namespace ZB.MOM.WW.Health.Akka.Tests;
/// <summary>
/// Table-driven tests for the pure <see cref="ActiveNodeDecision.Evaluate"/> helper covering both
/// the role-less (ScadaBridge ActiveNode) and role-filtered (OtOpcUa AdminRoleLeader) matrices,
/// plus the startup-safety null-guards on <see cref="ActiveNodeHealthCheck"/> and
/// <see cref="AkkaActiveNodeGate"/> when no <see cref="ActorSystem"/> is registered.
/// </summary>
public sealed class ActiveNodeDecisionTests
{
// Role-less: requiredRole == null. hasRole is irrelevant. Healthy iff (selfUp && isLeader), else Unhealthy.
public static IEnumerable<object?[]> RoleLessCases() => new[]
{
new object?[] { true, true, false, (string?)null, HealthStatus.Healthy },
new object?[] { true, false, false, (string?)null, HealthStatus.Unhealthy },
new object?[] { false, true, false, (string?)null, HealthStatus.Unhealthy },
new object?[] { false, false, false, (string?)null, HealthStatus.Unhealthy },
};
[Theory]
[MemberData(nameof(RoleLessCases))]
public void Evaluate_RoleLess(bool selfUp, bool isLeader, bool hasRole, string? requiredRole, HealthStatus expected)
{
Assert.Equal(expected, ActiveNodeDecision.Evaluate(selfUp, isLeader, hasRole, requiredRole));
}
// Role-filtered: requiredRole != null.
// lacks role -> Healthy (probe irrelevant for this node)
// has role & is leader -> Healthy (selfUp is ignored — role-filtered mode only cares about leadership)
// has role & not leader -> Unhealthy (503) — the tier exists to be a 503 an LB can act on
public static IEnumerable<object[]> RoleFilteredCases() => new[]
{
// node lacks the role -> Healthy regardless of selfUp / isLeader
new object[] { true, true, false, "admin", HealthStatus.Healthy },
new object[] { true, false, false, "admin", HealthStatus.Healthy },
new object[] { false, false, false, "admin", HealthStatus.Healthy },
// node carries the role and is leader -> Healthy (selfUp=true)
new object[] { true, true, true, "admin", HealthStatus.Healthy },
// node carries the role and is leader -> Healthy (selfUp=false: role-filtered mode ignores selfUp)
new object[] { false, true, true, "admin", HealthStatus.Healthy },
// Node carries the role but is not leader -> Unhealthy, i.e. HTTP 503.
// Degraded would be a 200 (see the shared health SPEC), which made the active tier answer
// 200 on every node and silently broke OtOpcUa's Traefik leader-pinning. lmxopcua#494.
new object[] { true, false, true, "admin", HealthStatus.Unhealthy },
new object[] { false, false, true, "admin", HealthStatus.Unhealthy },
};
[Theory]
[MemberData(nameof(RoleFilteredCases))]
public void Evaluate_RoleFiltered(bool selfUp, bool isLeader, bool hasRole, string? requiredRole, HealthStatus expected)
{
Assert.Equal(expected, ActiveNodeDecision.Evaluate(selfUp, isLeader, hasRole, requiredRole));
}
[Fact]
public async Task HealthCheck_RoleLess_NoActorSystem_ReturnsDegraded()
{
var provider = new ServiceCollection().BuildServiceProvider();
var check = new ActiveNodeHealthCheck(provider);
var result = await check.CheckHealthAsync(NewContext(check));
Assert.Equal(HealthStatus.Degraded, result.Status);
}
[Fact]
public async Task HealthCheck_RoleFiltered_NoActorSystem_ReturnsDegraded()
{
var provider = new ServiceCollection().BuildServiceProvider();
var check = new ActiveNodeHealthCheck(provider, "admin");
var result = await check.CheckHealthAsync(NewContext(check));
Assert.Equal(HealthStatus.Degraded, result.Status);
}
[Fact]
public void Gate_NoActorSystem_IsActiveNodeFalse()
{
var provider = new ServiceCollection().BuildServiceProvider();
var gate = new AkkaActiveNodeGate(provider);
Assert.False(gate.IsActiveNode);
}
[Fact]
public async Task HealthCheck_RoleLess_ClusterInaccessible_ReturnsDegraded()
{
// ActorSystem present but Akka.Cluster not configured → Cluster.Get throws. The check must
// return Degraded (startup-safety rule), not let the exception escape (→ Unhealthy).
using var system = ActorSystem.Create("plain-no-cluster-roleless");
try
{
var provider = new ServiceCollection()
.AddSingleton(system)
.BuildServiceProvider();
var check = new ActiveNodeHealthCheck(provider);
var result = await check.CheckHealthAsync(NewContext(check));
Assert.Equal(HealthStatus.Degraded, result.Status);
}
finally
{
await system.Terminate();
}
}
[Fact]
public async Task HealthCheck_RoleFiltered_ClusterInaccessible_ReturnsDegraded()
{
using var system = ActorSystem.Create("plain-no-cluster-rolefiltered");
try
{
var provider = new ServiceCollection()
.AddSingleton(system)
.BuildServiceProvider();
var check = new ActiveNodeHealthCheck(provider, "admin");
var result = await check.CheckHealthAsync(NewContext(check));
Assert.Equal(HealthStatus.Degraded, result.Status);
}
finally
{
await system.Terminate();
}
}
private static HealthCheckContext NewContext(IHealthCheck check) => new()
{
Registration = new HealthCheckRegistration("active-node", check, HealthStatus.Unhealthy, tags: null),
};
}
ZB.MOM.WW.Health/tests/ZB.MOM.WW.Health.Akka.Tests/ActiveNodeStartupSafetyTests.cs
+113
View File
@@ -0,0 +1,113 @@
using Akka.Actor;
using Akka.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Diagnostics.HealthChecks;
using ZB.MOM.WW.Health.Akka;
namespace ZB.MOM.WW.Health.Akka.Tests;
/// <summary>
/// The states a node passes through on the way up, where the active tier must not report a booting
/// node as a failed standby — now that the tier is a real 503, an Unhealthy here would pull the node
/// out of an orchestrator's pool while it starts.
/// </summary>
public sealed class ActiveNodeStartupSafetyTests
{
[Fact]
public async Task NoActorSystem_IsDegraded()
{
var check = new ActiveNodeHealthCheck(new ServiceCollection().BuildServiceProvider());
var result = await check.CheckHealthAsync(NewContext(check));
Assert.Equal(HealthStatus.Degraded, result.Status);
// Description-only: an empty data dictionary is what makes the writer omit "data" entirely.
Assert.Empty(result.Data);
}
[Fact]
public async Task NoActorSystem_RoleScoped_IsDegraded()
{
var check = new ActiveNodeHealthCheck(new ServiceCollection().BuildServiceProvider(), "admin");
Assert.Equal(HealthStatus.Degraded, (await check.CheckHealthAsync(NewContext(check))).Status);
}
[Fact]
public void Gate_NoActorSystem_IsNotActive()
{
Assert.False(new AkkaActiveNodeGate(new ServiceCollection().BuildServiceProvider()).IsActiveNode);
}
[Fact]
public async Task ClusterNotConfigured_IsDegraded()
{
// ActorSystem present but Akka.Cluster not configured → Cluster.Get throws. The check must
// return Degraded (the spec's startup-safety rule), not let the exception escape (→ Unhealthy).
using var system = ActorSystem.Create("active-node-no-cluster");
try
{
var provider = new ServiceCollection().AddSingleton(system).BuildServiceProvider();
var check = new ActiveNodeHealthCheck(provider);
Assert.Equal(HealthStatus.Degraded, (await check.CheckHealthAsync(NewContext(check))).Status);
}
finally
{
await system.Terminate();
}
}
[Fact]
public async Task Gate_ClusterNotConfigured_IsNotActive()
{
using var system = ActorSystem.Create("active-node-gate-no-cluster");
try
{
var provider = new ServiceCollection().AddSingleton(system).BuildServiceProvider();
Assert.False(new AkkaActiveNodeGate(provider).IsActiveNode);
}
finally
{
await system.Terminate();
}
}
[Fact]
public async Task ClusteredButNeverJoined_IsUnhealthy_NobodyIsActive()
{
// A cold start: the cluster provider is up but no member has reached Up. Nobody is active, so
// nobody may claim to be — this is the one case that must NOT be Degraded, because a cluster
// that never forms would otherwise sit at 200 forever.
var config = ConfigurationFactory.ParseString("""
akka {
loglevel = "WARNING"
actor.provider = cluster
remote.dot-netty.tcp { hostname = "127.0.0.1", port = 0 }
cluster.seed-nodes = []
}
""");
var system = ActorSystem.Create("active-node-never-joined", config);
try
{
var provider = new ServiceCollection().AddSingleton(system).BuildServiceProvider();
var check = new ActiveNodeHealthCheck(provider);
var result = await check.CheckHealthAsync(NewContext(check));
Assert.Equal(HealthStatus.Unhealthy, result.Status);
Assert.Contains("No Up member", result.Description);
Assert.False(new AkkaActiveNodeGate(provider).IsActiveNode);
}
finally
{
await system.Terminate();
}
}
private static HealthCheckContext NewContext(IHealthCheck check) => new()
{
Registration = new HealthCheckRegistration("active-node", check, HealthStatus.Unhealthy, tags: null),
};
}
ZB.MOM.WW.Health/tests/ZB.MOM.WW.Health.Akka.Tests/ActiveRoleResolutionTests.cs
+56
View File
@@ -0,0 +1,56 @@
using ZB.MOM.WW.Health.Akka;
namespace ZB.MOM.WW.Health.Akka.Tests;
/// <summary>
/// Covers <see cref="ClusterActiveNode.ResolveActiveRole"/> — which role a node competes for. Pure,
/// so no cluster is needed; the age-ordering half of the rule is pinned in
/// <see cref="ClusterActiveNodeTests"/> against a real two-node cluster.
/// </summary>
public sealed class ActiveRoleResolutionTests
{
[Fact]
public void EmptyPreference_IsUnscoped()
{
// Unscoped: every Up member competes for one active slot (the ScadaBridge central pattern).
Assert.Null(ClusterActiveNode.ResolveActiveRole(["admin", "driver"], []));
}
[Fact]
public void FusedNode_AnswersForTheHighestPrecedenceRoleItCarries()
{
Assert.Equal(
"admin",
ClusterActiveNode.ResolveActiveRole(["admin", "driver", "cluster-MAIN"], ["admin", "driver"]));
}
[Fact]
public void SingleRoleNode_AnswersForThatRole()
{
// The regression the OtOpcUa fix existed for: under the old role-filtered check every
// driver-only node reported Healthy — "or not a role member" — so all of them called
// themselves active and no consumer could find the active node of a site cluster. lmxopcua#494.
Assert.Equal(
"driver",
ClusterActiveNode.ResolveActiveRole(["driver", "cluster-SITE-A"], ["admin", "driver"]));
}
[Fact]
public void PreferenceOrderWins_NotTheOrderTheNodeListsItsRoles()
{
// Self-role order is incidental (it comes from config parsing); precedence must come from the
// caller's list alone, or a fused node's answer would depend on how its roles were spelled.
Assert.Equal(
"admin",
ClusterActiveNode.ResolveActiveRole(["driver", "admin"], ["admin", "driver"]));
}
[Fact]
public void NodeCarryingNoneOfThePreferredRoles_ResolvesToNull()
{
// Reachable, not merely defensive: OtOpcUa's RoleParser admits a dev-only or cluster-role-only
// node. Such a node owns no active work — the health check maps this to Unhealthy so it stays
// out of the active pool.
Assert.Null(ClusterActiveNode.ResolveActiveRole(["dev", "cluster-MAIN"], ["admin", "driver"]));
}
}
ZB.MOM.WW.Health/tests/ZB.MOM.WW.Health.Akka.Tests/ClusterActiveNodeTests.cs
+237
View File
@@ -0,0 +1,237 @@
using Akka.Actor;
using Akka.Cluster;
using Akka.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Diagnostics.HealthChecks;
using ZB.MOM.WW.Health.Akka;
namespace ZB.MOM.WW.Health.Akka.Tests;
/// <summary>
/// Pins <b>which</b> node the active tier names, against a real two-node cluster deliberately built
/// so the oldest member is <i>not</i> the lowest-addressed one.
/// </summary>
/// <remarks>
/// <para>
/// The divergence is the whole point. Akka offers two different "first" members:
/// <c>ClusterState.Leader</c> / <c>RoleLeader</c> is the <b>lowest-addressed</b> Up member
/// (host, then port — nothing to do with time), while <b>oldest</b> is the member with the
/// lowest up-number, which is what <c>ClusterSingletonManager</c> uses to place singletons.
/// </para>
/// <para>
/// They coincide on a freshly-formed cluster, which is why every single-node and happy-path
/// test missed it and why the leader-based check survived to 0.2.1. They diverge the moment a
/// node restarts: the restarted node rejoins as the youngest while keeping its address, so if
/// it holds the lower address it becomes leader while the singletons — and every piece of work
/// they own — stay on the other node. This fixture reproduces that state directly instead of
/// waiting for a restart.
/// </para>
/// </remarks>
public sealed class ClusterActiveNodeTests : IAsyncLifetime
{
// Fixed, unusual ports: the test needs a deterministic address ordering, which port 0 cannot give.
// Distinct from OtOpcUa's equivalent fixture (19_530/19_531) so the two suites can run at once.
private const int OldestPort = 19_541; // joins FIRST -> oldest, but the HIGHER address
private const int YoungestPort = 19_540; // joins SECOND -> cluster leader, the LOWER address
private const string SystemName = "health-active-node";
private ActorSystem? _oldest;
private ActorSystem? _youngest;
private static Config NodeConfig(int port, string roles) => ConfigurationFactory.ParseString($$"""
akka {
loglevel = "WARNING"
actor.provider = "Akka.Cluster.ClusterActorRefProvider, Akka.Cluster"
remote.dot-netty.tcp {
hostname = "127.0.0.1"
port = {{port}}
}
cluster {
seed-nodes = ["akka.tcp://{{SystemName}}@127.0.0.1:{{OldestPort}}"]
roles = [{{roles}}]
min-nr-of-members = 1
run-coordinated-shutdown-when-down = off
downing-provider-class = ""
}
}
""");
public async Task InitializeAsync()
{
// Order matters: the seed forms the cluster and is therefore the oldest member. The oldest
// node is fused (admin + driver); the younger one is driver-only, so the role-scoped
// assertions below can distinguish "oldest of this role" from "oldest overall".
_oldest = ActorSystem.Create(SystemName, NodeConfig(OldestPort, "\"admin\", \"driver\""));
await WaitForUpAsync(_oldest, expectedMembers: 1);
_youngest = ActorSystem.Create(SystemName, NodeConfig(YoungestPort, "\"driver\""));
await WaitForUpAsync(_oldest, expectedMembers: 2);
await WaitForUpAsync(_youngest, expectedMembers: 2);
}
public async Task DisposeAsync()
{
if (_youngest is not null) await _youngest.Terminate();
if (_oldest is not null) await _oldest.Terminate();
}
private static async Task WaitForUpAsync(ActorSystem system, int expectedMembers)
{
var cluster = Cluster.Get(system);
var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(30);
while (DateTime.UtcNow < deadline)
{
if (cluster.State.Members.Count(m => m.Status == MemberStatus.Up) >= expectedMembers) return;
await Task.Delay(100);
}
throw new TimeoutException(
$"cluster on port {cluster.SelfAddress.Port} never saw {expectedMembers} Up members "
+ $"(saw {cluster.State.Members.Count(m => m.Status == MemberStatus.Up)})");
}
/// <summary>
/// Sanity check on the fixture itself: if the two orderings did not actually diverge, every
/// assertion below would pass for the wrong reason.
/// </summary>
[Fact]
public void Fixture_actually_produces_divergent_age_and_address_ordering()
{
var state = Cluster.Get(_oldest!).State;
Assert.Equal(YoungestPort, state.Leader!.Port);
Assert.Equal(YoungestPort, state.RoleLeader("driver")!.Port);
}
[Fact]
public void OldestUpMember_IsTheOldest_NotTheLeader()
{
var members = Cluster.Get(_oldest!).State.Members;
Assert.Equal(OldestPort, ClusterActiveNode.OldestUpMember(members)!.Address.Port);
Assert.Equal(OldestPort, ClusterActiveNode.OldestUpMember(members, "driver")!.Address.Port);
}
[Fact]
public void OldestUpMember_IsRoleScoped()
{
var members = Cluster.Get(_oldest!).State.Members;
// Only the fused node carries admin, so it owns the admin slot even though it is not the
// lowest-addressed member.
Assert.Equal(OldestPort, ClusterActiveNode.OldestUpMember(members, "admin")!.Address.Port);
// A role nobody carries has no owner.
Assert.Null(ClusterActiveNode.OldestUpMember(members, "no-such-role"));
}
[Fact]
public void SelfIsActive_IsTrueOnlyOnTheOldestNode()
{
Assert.True(ClusterActiveNode.SelfIsActive(Cluster.Get(_oldest!)));
Assert.False(ClusterActiveNode.SelfIsActive(Cluster.Get(_youngest!)));
}
[Fact]
public void SelfIsActive_RoleScoped_YoungestOwnsNoSlot()
{
// Driver-scoped: the fused node is still the oldest driver, so the driver-only node is standby.
Assert.True(ClusterActiveNode.SelfIsActive(Cluster.Get(_oldest!), "driver"));
Assert.False(ClusterActiveNode.SelfIsActive(Cluster.Get(_youngest!), "driver"));
// Admin-scoped: the driver-only node does not carry admin, so it can never be the admin active
// node — even though it is the cluster leader.
Assert.False(ClusterActiveNode.SelfIsActive(Cluster.Get(_youngest!), "admin"));
}
[Fact]
public async Task HealthCheck_IsHealthyOnTheOldestAndUnhealthyOnTheLeader()
{
var onOldest = await CheckAsync(_oldest!, new ActiveNodeHealthCheckOptions());
var onYoungest = await CheckAsync(_youngest!, new ActiveNodeHealthCheckOptions());
Assert.Equal(HealthStatus.Healthy, onOldest.Status);
// The node the old leader-based check would have called active. Exactly one 200 per cluster is
// the property an orchestrator's active-tier routing depends on.
Assert.Equal(HealthStatus.Unhealthy, onYoungest.Status);
}
[Fact]
public async Task HealthCheck_DataNamesTheActiveNode_EvenFromAStandby()
{
// A standby reporting *who* is active is what lets a dashboard render the pair from either
// node's payload alone (the 0.2.0 per-entry data object).
var result = await CheckAsync(_youngest!, new ActiveNodeHealthCheckOptions());
Assert.Equal($"akka.tcp://{SystemName}@127.0.0.1:{OldestPort}", result.Data["activeNode"]);
Assert.Equal($"akka.tcp://{SystemName}@127.0.0.1:{YoungestPort}", result.Data["selfAddress"]);
}
[Fact]
public async Task HealthCheck_RolePreference_FusedNodeAnswersForAdmin()
{
var options = new ActiveNodeHealthCheckOptions { RolePreference = ["admin", "driver"] };
var onOldest = await CheckAsync(_oldest!, options);
var onYoungest = await CheckAsync(_youngest!, options);
Assert.Equal(HealthStatus.Healthy, onOldest.Status);
Assert.Equal("admin", onOldest.Data["activeRole"]);
// Resolves to driver (it has no admin role) and loses to the older fused node.
Assert.Equal(HealthStatus.Unhealthy, onYoungest.Status);
Assert.Equal("driver", onYoungest.Data["activeRole"]);
}
[Fact]
public async Task HealthCheck_NodeOwningNoneOfThePreferredRoles_IsUnhealthyByDefault()
{
var options = new ActiveNodeHealthCheckOptions { RolePreference = ["no-such-role"] };
var result = await CheckAsync(_oldest!, options);
// Not "not applicable ⇒ Healthy": a node that owns no active work must stay out of the pool.
Assert.Equal(HealthStatus.Unhealthy, result.Status);
Assert.Contains("carries none of the active roles", result.Description);
Assert.DoesNotContain("activeRole", result.Data.Keys);
}
[Fact]
public async Task HealthCheck_NoActiveRoleStatus_IsOverridable()
{
var options = new ActiveNodeHealthCheckOptions
{
RolePreference = ["no-such-role"],
NoActiveRoleStatus = HealthStatus.Healthy,
};
Assert.Equal(HealthStatus.Healthy, (await CheckAsync(_oldest!, options)).Status);
}
[Fact]
public void Gate_IsActiveOnlyOnTheOldestNode()
{
Assert.True(new AkkaActiveNodeGate(ProviderFor(_oldest!)).IsActiveNode);
Assert.False(new AkkaActiveNodeGate(ProviderFor(_youngest!)).IsActiveNode);
// Role-scoped: the driver-only node cannot gate admin work.
Assert.False(new AkkaActiveNodeGate(ProviderFor(_youngest!), "admin").IsActiveNode);
}
private static IServiceProvider ProviderFor(ActorSystem system) =>
new ServiceCollection().AddSingleton(system).BuildServiceProvider();
private static async Task<HealthCheckResult> CheckAsync(
ActorSystem system,
ActiveNodeHealthCheckOptions options)
{
var check = new ActiveNodeHealthCheck(ProviderFor(system), options);
var context = new HealthCheckContext
{
Registration = new HealthCheckRegistration("active-node", check, HealthStatus.Unhealthy, tags: null),
};
return await check.CheckHealthAsync(context);
}
}