Joseph Doherty
4550486144
Closes the remaining half of #494, and corrects the half fixed in8dd9da7d. The shared ActiveNodeHealthCheck(role: "admin") answered the wrong question twice over. It returns Healthy for any node LACKING the role, so all four driver-only site nodes called themselves active and no consumer could find the Primary of a site Cluster. And it selects by RoleLeader - the lowest-ADDRESSED member - which is not where Akka places singletons. Replaced with ClusterPrimaryHealthCheck ("cluster-primary"). One rule: this node is active iff it is the OLDEST Up member carrying its own active role, where the active role is admin when the node has it and driver otherwise. That serves both consumers correctly - a fused admin node answers for admin, so Traefik pins the AdminUI to the node hosting the singletons; a driver-only site node answers for driver, which is exactly SelectDriverPrimary, the same election behind IsDriverPrimary and the OPC UA ServiceLevel 250/240 split. Per-mesh scoping is free after Phase 6: ClusterState.Members already contains only this node's own Cluster. SelectDriverPrimary is generalised to SelectOldestUpMemberOfRole so the age-ordering rule has one implementation. Two copies of "oldest Up member of a role" would be two chances to silently disagree about who is in charge, and the tier and the redundancy snapshot must never disagree. THE ROLE-LEADER HALF MATTERED IN PRACTICE, not just in theory. On the rebuilt rig the two orderings diverge right now: akka leader (lowest address) = central-1 oldest admin member = central-2 <- hosts the singletons8dd9da7dmade the tier a real 503 but still selected by RoleLeader, so it would have pinned Traefik to central-1 - the node NOT hosting the work. With this change Traefik correctly routes to central-2. Live-verified, exactly one 200 per mesh: MAIN central-2 200 central-1 503 traefik: central-2 UP, central-1 DOWN SITE-A site-a-1 200 site-a-2 503 SITE-B site-b-1 200 site-b-2 503 Tests: role-selection matrix and the startup-safe Degraded path in Host.Tests (26 pass); parity between the tier's selector and the redundancy snapshot's, plus role scoping, added to RedundancyPrimaryElectionTests, which forms a real two-node cluster deliberately built so the oldest member is not the lowest-addressed one (5 pass).
66 lines
2.7 KiB
C#
66 lines
2.7 KiB
C#
using Microsoft.Extensions.DependencyInjection;
|
|
using Microsoft.Extensions.Diagnostics.HealthChecks;
|
|
using Shouldly;
|
|
using Xunit;
|
|
using ZB.MOM.WW.OtOpcUa.Host.Health;
|
|
|
|
namespace ZB.MOM.WW.OtOpcUa.Host.Tests.Health;
|
|
|
|
/// <summary>
|
|
/// Pins the rule the active tier now uses to decide which node is in charge, and the startup path
|
|
/// that must not report a booting node as a failed standby.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// The cluster-membership half of the rule — oldest Up member, never <c>RoleLeader</c> — is pinned
|
|
/// against a real two-node cluster in <c>RedundancyPrimaryElectionTests</c>, including parity with
|
|
/// the redundancy snapshot's own election. What is left to cover here is the role-selection rule and
|
|
/// the startup guard.
|
|
/// </remarks>
|
|
public sealed class ClusterPrimaryHealthCheckTests
|
|
{
|
|
[Fact]
|
|
public void Fused_node_answers_for_the_admin_role()
|
|
{
|
|
// Admin wins on a node carrying both, because the cluster singletons and the AdminUI that
|
|
// Traefik routes to are pinned to that role. Answering for 'driver' here could name a
|
|
// different node the moment an admin-only or driver-only member joins the mesh.
|
|
ClusterPrimaryHealthCheck.ActiveRoleFor(["admin", "driver", "cluster-MAIN"])
|
|
.ShouldBe("admin");
|
|
}
|
|
|
|
[Fact]
|
|
public void Driver_only_node_answers_for_the_driver_role()
|
|
{
|
|
// The regression this whole change exists for. Under the previous admin-scoped check every
|
|
// site node returned Healthy — "or not a role member" — so all four called themselves
|
|
// active and no consumer could find the Primary of a site Cluster. lmxopcua#494.
|
|
ClusterPrimaryHealthCheck.ActiveRoleFor(["driver", "cluster-SITE-A"])
|
|
.ShouldBe("driver");
|
|
}
|
|
|
|
[Fact]
|
|
public void Admin_only_node_answers_for_the_admin_role()
|
|
{
|
|
ClusterPrimaryHealthCheck.ActiveRoleFor(["admin"]).ShouldBe("admin");
|
|
}
|
|
|
|
[Fact]
|
|
public void Node_in_neither_role_has_no_active_role()
|
|
{
|
|
ClusterPrimaryHealthCheck.ActiveRoleFor(["cluster-MAIN"]).ShouldBeNull();
|
|
}
|
|
|
|
[Fact]
|
|
public async Task No_actor_system_is_degraded_not_unhealthy()
|
|
{
|
|
// A node that is still booting has not lost an election. Reporting Unhealthy here would make
|
|
// every start look like a failed standby, and — now that the tier is a real 503 — would pull
|
|
// the node out of Traefik's pool on the way up.
|
|
var check = new ClusterPrimaryHealthCheck(new ServiceCollection().BuildServiceProvider());
|
|
|
|
var result = await check.CheckHealthAsync(new HealthCheckContext(), CancellationToken.None);
|
|
|
|
result.Status.ShouldBe(HealthStatus.Degraded);
|
|
}
|
|
}
|