fix(health): active tier reports the cluster Primary, not the admin role leader

Closes the remaining half of #494, and corrects the half fixed in 8dd9da7d.

The shared ActiveNodeHealthCheck(role: "admin") answered the wrong question twice
over. It returns Healthy for any node LACKING the role, so all four driver-only
site nodes called themselves active and no consumer could find the Primary of a
site Cluster. And it selects by RoleLeader - the lowest-ADDRESSED member - which
is not where Akka places singletons.

Replaced with ClusterPrimaryHealthCheck ("cluster-primary"). One rule: this node
is active iff it is the OLDEST Up member carrying its own active role, where the
active role is admin when the node has it and driver otherwise. That serves both
consumers correctly - a fused admin node answers for admin, so Traefik pins the
AdminUI to the node hosting the singletons; a driver-only site node answers for
driver, which is exactly SelectDriverPrimary, the same election behind
IsDriverPrimary and the OPC UA ServiceLevel 250/240 split. Per-mesh scoping is
free after Phase 6: ClusterState.Members already contains only this node's own
Cluster.

SelectDriverPrimary is generalised to SelectOldestUpMemberOfRole so the
age-ordering rule has one implementation. Two copies of "oldest Up member of a
role" would be two chances to silently disagree about who is in charge, and the
tier and the redundancy snapshot must never disagree.

THE ROLE-LEADER HALF MATTERED IN PRACTICE, not just in theory. On the rebuilt rig
the two orderings diverge right now:

  akka leader (lowest address) = central-1
  oldest admin member          = central-2   <- hosts the singletons

8dd9da7d made the tier a real 503 but still selected by RoleLeader, so it would
have pinned Traefik to central-1 - the node NOT hosting the work. With this change
Traefik correctly routes to central-2.

Live-verified, exactly one 200 per mesh:

  MAIN     central-2 200   central-1 503     traefik: central-2 UP, central-1 DOWN
  SITE-A   site-a-1  200   site-a-2  503
  SITE-B   site-b-1  200   site-b-2  503

Tests: role-selection matrix and the startup-safe Degraded path in Host.Tests
(26 pass); parity between the tier's selector and the redundancy snapshot's, plus
role scoping, added to RedundancyPrimaryElectionTests, which forms a real two-node
cluster deliberately built so the oldest member is not the lowest-addressed one
(5 pass).
This commit is contained in:
Joseph Doherty
2026-07-24 12:43:27 -04:00
parent 8dd9da7d4d
commit 4550486144
5 changed files with 267 additions and 7 deletions
@@ -141,12 +141,30 @@ public sealed class RedundancyStateActor : ReceiveActor, IWithTimers
/// </remarks>
/// <param name="members">The current cluster members, in any order.</param>
/// <returns>The oldest Up driver member's address, or <c>null</c> when there is none.</returns>
public static Address? SelectDriverPrimary(IEnumerable<Member> members)
public static Address? SelectDriverPrimary(IEnumerable<Member> members) =>
SelectOldestUpMemberOfRole(members, DriverRole);
/// <summary>
/// Selects the oldest Up member carrying <paramref name="role"/> — the node that
/// <c>ClusterSingletonManager</c> would place a role-scoped singleton on.
/// </summary>
/// <param name="members">The current cluster members, in any order.</param>
/// <param name="role">The cluster role to scope the selection to.</param>
/// <returns>The oldest Up member's address for that role, or <c>null</c> when there is none.</returns>
/// <remarks>
/// The age-ordering rationale in <see cref="SelectDriverPrimary"/> applies verbatim to any role:
/// oldest, never <c>RoleLeader</c>. Factored out so the rule has exactly one implementation —
/// the health tier that reports which node is active needs the same answer the redundancy
/// snapshot does, and two copies of "oldest Up member of a role" would be two chances to
/// silently disagree about who is in charge.
/// </remarks>
public static Address? SelectOldestUpMemberOfRole(IEnumerable<Member> members, string role)
{
ArgumentNullException.ThrowIfNull(members);
ArgumentException.ThrowIfNullOrWhiteSpace(role);
return members
.Where(m => m.Status == MemberStatus.Up && m.Roles.Contains(DriverRole))
.Where(m => m.Status == MemberStatus.Up && m.Roles.Contains(role))
.OrderBy(m => m, Member.AgeOrdering)
.FirstOrDefault()
?.Address;