refactor(health): adopt the shared active-node check (Health 0.3.0)
ClusterPrimaryHealthCheck was written three days' worth of debugging ago because the shared ActiveNodeHealthCheck selected by RoleLeader and reported Healthy for any node lacking the role. Health 0.3.0 fixes the shared check — oldest Up member, role-preference scoping, Unhealthy when the node owns no active work — so the private copy is now duplication rather than a workaround, and it is deleted. RolePreference [admin, driver] reproduces exactly what it did: a fused central node answers for admin (where the singletons and the AdminUI are pinned), a driver-only site node answers for driver. SelectOldestUpMemberOfRole now delegates to the shared ClusterActiveNode instead of re-implementing the age ordering. This is the point of the change rather than a tidy-up: the redundancy snapshot drives the OPC UA ServiceLevel 250/240 split while the health tier drives Traefik's admin routing, so two copies of "oldest Up member of a role" would be two chances to advertise one node as authoritative while gating the data plane on another. ControlPlane takes a ZB.MOM.WW.Health.Akka reference for it; the layering trade-off is recorded at the PackageReference. Behaviour note: a node carrying neither admin nor driver now answers 503 on the active tier instead of 200. That case is reachable — RoleParser admits dev-only and cluster-role-only nodes — and 503 is correct, since such a node owns no active work and must not be in an active-tier pool. No docker-dev node is affected: every rig node is admin+driver or driver-only. Tests replaced in kind. The rule itself is now pinned in the library against a real two-node cluster; what stays OtOpcUa's own decision is the wiring, so ActiveTierRegistrationTests pins which check is on the active tag, that it is registered on driver-only nodes too, and that it is not on the ready tier. Verified: Host.Tests 25/25, ControlPlane redundancy 15/15.
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,59 @@
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Diagnostics.HealthChecks;
|
||||
using Microsoft.Extensions.Options;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.Health;
|
||||
using ZB.MOM.WW.Health.Akka;
|
||||
using ZB.MOM.WW.OtOpcUa.Host.Health;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.Tests.Health;
|
||||
|
||||
/// <summary>
|
||||
/// Pins OtOpcUa's active-tier wiring. The <i>rule</i> (oldest Up member of a role) now lives in the
|
||||
/// shared <c>ClusterActiveNode</c> and is tested there against a real two-node cluster; what stays
|
||||
/// OtOpcUa's own decision — and what regressed in <c>lmxopcua#494</c> — is <b>which check is
|
||||
/// registered on the active tag, and with which roles</b>.
|
||||
/// </summary>
|
||||
public sealed class ActiveTierRegistrationTests
|
||||
{
|
||||
private static HealthCheckRegistration ClusterPrimaryRegistration(bool hasAdmin) =>
|
||||
new ServiceCollection()
|
||||
.AddOtOpcUaHealth(hasAdmin)
|
||||
.BuildServiceProvider()
|
||||
.GetRequiredService<IOptions<HealthCheckServiceOptions>>()
|
||||
.Value.Registrations
|
||||
.Where(r => r.Name == "cluster-primary")
|
||||
.ShouldHaveSingleItem();
|
||||
|
||||
[Fact]
|
||||
public void Active_tier_uses_the_shared_ActiveNodeHealthCheck()
|
||||
{
|
||||
// Not a bespoke copy. Two apps in the family independently hand-rolled this check because the
|
||||
// shared one selected by RoleLeader; since Health 0.3.0 the shared one is the correct rule, so
|
||||
// a private reimplementation reappearing here would be the regression.
|
||||
var registration = ClusterPrimaryRegistration(hasAdmin: true);
|
||||
|
||||
registration.Factory(new ServiceCollection().BuildServiceProvider())
|
||||
.ShouldBeOfType<ActiveNodeHealthCheck>();
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(true)]
|
||||
[InlineData(false)]
|
||||
public void Active_tier_check_is_registered_on_every_node_role(bool hasAdmin)
|
||||
{
|
||||
// Registered on driver-only nodes too, unlike the admin-only configdb probe. A site pair needs
|
||||
// an active tier just as much as the central pair does — under the old admin-scoped check all
|
||||
// four driver-only site nodes answered 200 and no consumer could find a site's Primary.
|
||||
ClusterPrimaryRegistration(hasAdmin).Tags.ShouldContain(ZbHealthTags.Active);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Active_tier_check_is_not_on_the_ready_tier()
|
||||
{
|
||||
// Readiness must not depend on being the active node: a healthy standby is ready to serve, and
|
||||
// tagging it ready would take the whole pair out of rotation whenever one node is standby.
|
||||
ClusterPrimaryRegistration(hasAdmin: true).Tags.ShouldNotContain(ZbHealthTags.Ready);
|
||||
}
|
||||
}
|
||||
@@ -1,65 +0,0 @@
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Diagnostics.HealthChecks;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Host.Health;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.Tests.Health;
|
||||
|
||||
/// <summary>
|
||||
/// Pins the rule the active tier now uses to decide which node is in charge, and the startup path
|
||||
/// that must not report a booting node as a failed standby.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The cluster-membership half of the rule — oldest Up member, never <c>RoleLeader</c> — is pinned
|
||||
/// against a real two-node cluster in <c>RedundancyPrimaryElectionTests</c>, including parity with
|
||||
/// the redundancy snapshot's own election. What is left to cover here is the role-selection rule and
|
||||
/// the startup guard.
|
||||
/// </remarks>
|
||||
public sealed class ClusterPrimaryHealthCheckTests
|
||||
{
|
||||
[Fact]
|
||||
public void Fused_node_answers_for_the_admin_role()
|
||||
{
|
||||
// Admin wins on a node carrying both, because the cluster singletons and the AdminUI that
|
||||
// Traefik routes to are pinned to that role. Answering for 'driver' here could name a
|
||||
// different node the moment an admin-only or driver-only member joins the mesh.
|
||||
ClusterPrimaryHealthCheck.ActiveRoleFor(["admin", "driver", "cluster-MAIN"])
|
||||
.ShouldBe("admin");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Driver_only_node_answers_for_the_driver_role()
|
||||
{
|
||||
// The regression this whole change exists for. Under the previous admin-scoped check every
|
||||
// site node returned Healthy — "or not a role member" — so all four called themselves
|
||||
// active and no consumer could find the Primary of a site Cluster. lmxopcua#494.
|
||||
ClusterPrimaryHealthCheck.ActiveRoleFor(["driver", "cluster-SITE-A"])
|
||||
.ShouldBe("driver");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Admin_only_node_answers_for_the_admin_role()
|
||||
{
|
||||
ClusterPrimaryHealthCheck.ActiveRoleFor(["admin"]).ShouldBe("admin");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Node_in_neither_role_has_no_active_role()
|
||||
{
|
||||
ClusterPrimaryHealthCheck.ActiveRoleFor(["cluster-MAIN"]).ShouldBeNull();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task No_actor_system_is_degraded_not_unhealthy()
|
||||
{
|
||||
// A node that is still booting has not lost an election. Reporting Unhealthy here would make
|
||||
// every start look like a failed standby, and — now that the tier is a real 503 — would pull
|
||||
// the node out of Traefik's pool on the way up.
|
||||
var check = new ClusterPrimaryHealthCheck(new ServiceCollection().BuildServiceProvider());
|
||||
|
||||
var result = await check.CheckHealthAsync(new HealthCheckContext(), CancellationToken.None);
|
||||
|
||||
result.Status.ShouldBe(HealthStatus.Degraded);
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user