diff --git a/CLAUDE.md b/CLAUDE.md index 4b2aaa7..e60d6c9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -143,7 +143,7 @@ each project's **code-verified current state**, and the **gaps** between. See |---|---|---|---|---| | Auth (login / identity / authz) | Adopted (lib `0.1.4`; all 3 apps, merged to **local default** main/master + **pushed to origin** (gitea)) | Shared `ZB.MOM.WW.Auth` lib | [`components/auth/`](components/auth/) | [`ZB.MOM.WW.Auth/`](ZB.MOM.WW.Auth/) | | UI Theme (layout / tokens / components) | Adopted (feed `0.2.0`–`0.3.1`; **all 4 apps pinned `0.3.1`**, pushed to origin) | Shared `ZB.MOM.WW.Theme` RCL | [`components/ui-theme/`](components/ui-theme/) | [`ZB.MOM.WW.Theme/`](ZB.MOM.WW.Theme/) | -| Health (readiness / liveness / active-node) | **Adopted** (feed carries `0.1.0`/`0.2.0`/`0.2.1`; **all 4 apps** reference it + wire `MapZbHealth`). **`0.2.0` (2026-07-24)** added an optional per-entry `data` object to the canonical health JSON and made `AkkaClusterHealthCheck` publish its cluster view (`leader`/`selfAddress`/`selfRoles`/`memberCount`/`unreachableCount`) — additive, so pre-0.2.0 payloads still parse. **`0.2.1` (2026-07-24) is a behaviour fix with a live consumer impact:** the active tier now returns **Unhealthy (503)** on a role-member-but-not-leader node instead of `Degraded`, which the spec maps to 200. The old value made `/health/active` answer 200 on *every* node, so it could not distinguish active from standby over HTTP at all — and OtOpcUa's Traefik routes the admin UI by exactly this probe, so its leader-pinning had silently never worked ([`lmxopcua#494`](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/494)). **Only OtOpcUa is affected** — it is the sole user of the role-filtered mode; ScadaBridge deliberately uses its own `OldestNodeActiveHealthCheck` (leadership is address-ordered and diverges from the singleton host), and mxgw/HistorianGateway have no Akka cluster. | Shared `ZB.MOM.WW.Health` lib | [`components/health/`](components/health/) | [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) | +| Health (readiness / liveness / active-node) | **Adopted** (feed carries `0.1.0`/`0.2.0`/`0.2.1`; **all 4 apps** reference it + wire `MapZbHealth`). **`0.2.0` (2026-07-24)** added an optional per-entry `data` object to the canonical health JSON and made `AkkaClusterHealthCheck` publish its cluster view (`leader`/`selfAddress`/`selfRoles`/`memberCount`/`unreachableCount`) — additive, so pre-0.2.0 payloads still parse. **`0.2.1` (2026-07-24) is a behaviour fix with a live consumer impact:** the active tier now returns **Unhealthy (503)** on a role-member-but-not-leader node instead of `Degraded`, which the spec maps to 200. The old value made `/health/active` answer 200 on *every* node, so it could not distinguish active from standby over HTTP at all — and OtOpcUa's Traefik routes the admin UI by exactly this probe, so its leader-pinning had silently never worked ([`lmxopcua#494`](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/494)). **`0.3.0` (2026-07-24) finished that fix and ENDED a family-wide duplication.** 0.2.1 made the tier a real 503 but still selected by `RoleLeader`; on a rebuilt rig the two orderings diverge live (akka leader `central-1`, oldest admin `central-2` — the singleton host), so it would have pinned Traefik to the node *not* hosting the work. **Leadership is address-ordered (host, then port) and has no relationship to time; singleton placement is age-ordered.** They agree on a freshly-formed cluster — which is why single-node and happy-path tests never caught it — and diverge permanently after any restart. Both Akka apps had already discovered this independently and each written a private replacement rather than use the shared check (ScadaBridge `OldestNodeActiveHealthCheck`, OtOpcUa `ClusterPrimaryHealthCheck`), so the package's entire active-node surface had **zero consumers — not unused but avoided, twice, for the same reason.** 0.3.0 promotes those two into one `ClusterActiveNode` primitive (oldest `Up` member, optional role scope, plus an ordered role-preference so a fused node answers for the role its singletons are pinned to); `ActiveNodeHealthCheck` and `AkkaActiveNodeGate` both delegate to it, and results now carry `activeRole`/`selfAddress`/`activeNode` in the 0.2.0 `data` object so a **standby reports who IS active**. Two further behaviour changes: a node holding **none** of the preferred roles is now **Unhealthy**, not Healthy — the old "not applicable ⇒ Healthy" is exactly what made the tier answer 200 everywhere, and the case is reachable (OtOpcUa's `RoleParser` admits `dev`-only and `cluster-`only nodes); and identity compares `UniqueAddress`, so a node restarted on the same host:port is correctly a different member. **Both apps migrated and their private copies deleted** — OtOpcUa (`feat/health-0.3.0-active-node` @ `88adec04`) also routes `RedundancyStateActor.SelectOldestUpMemberOfRole` through the shared primitive, so the tier and the OPC UA `ServiceLevel` 250/240 split cannot drift; ScadaBridge (`feat/site-node-health` @ `5d4853a1`) routes `ActiveNodeEvaluator` through it, unifying the S&F delivery gate, the heartbeat `IsActive` stamp, the inbound-API gate and the health tier on one rule. ScadaBridge keeps `SitePairActiveNodeHealthCheck` deliberately — it is a thin adapter over the site's own role-scoped `IClusterNodeProvider` (itself now backed by the shared rule), not a second copy of it. mxgw/HistorianGateway have no Akka cluster and are unaffected. Both app branches are **committed but NOT pushed or merged**. | Shared `ZB.MOM.WW.Health` lib | [`components/health/`](components/health/) | [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) | | Observability (metrics / traces / logs) | **Adopted** (libs `0.1.0`, on the feed; **all 4 apps** call `AddZbTelemetry`; `AddZbSerilog` in all but ScadaBridge, which keeps its own factory by design) | Shared `ZB.MOM.WW.Telemetry` lib + `.Serilog` | [`components/observability/`](components/observability/) | [`ZB.MOM.WW.Telemetry/`](ZB.MOM.WW.Telemetry/) | | Config + validation (options / startup validation) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed — the "local only / not yet pushed" caveat was stale) | Shared `ZB.MOM.WW.Configuration` lib | [`components/configuration/`](components/configuration/) | [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/) | | Audit (event model + writer seam) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed to origin) | Shared `ZB.MOM.WW.Audit` lib | [`components/audit/`](components/audit/) | [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) | diff --git a/ZB.MOM.WW.Health/CLAUDE.md b/ZB.MOM.WW.Health/CLAUDE.md index f00befc..a9e1e08 100644 --- a/ZB.MOM.WW.Health/CLAUDE.md +++ b/ZB.MOM.WW.Health/CLAUDE.md @@ -4,7 +4,7 @@ Health-check libraries for the **ZB.MOM.WW SCADA family** (OtOpcUa, MxAccessGate The library normalizes the three-tier health endpoint convention (`/health/ready`, `/health/active`, `/healthz`) and provides reusable probe implementations so the three sister projects share a common surface without duplicating probe logic. -**Built at 0.2.0 and published to the Gitea NuGet feed. Adopted by all four apps** — adoption is tracked in `~/Desktop/scadaproj/components/health/GAPS.md`. +**Built at 0.3.0 and published to the Gitea NuGet feed. Adopted by all four apps** — adoption is tracked in `~/Desktop/scadaproj/components/health/GAPS.md`. --- @@ -13,7 +13,7 @@ The library normalizes the three-tier health endpoint convention (`/health/ready | Package | Responsibilities | Key Dependencies | |---|---|---| | `ZB.MOM.WW.Health` | Core tier convention, `MapZbHealth` extension, canonical JSON writer (`ZbHealthWriter`), `IActiveNodeGate` seam, `GrpcDependencyHealthCheck` reachability probe, tier-tag constants (`ZbHealthTags`). No Akka or EF dependency. | `Microsoft.AspNetCore.App` (framework ref), `Grpc.Net.Client` | -| `ZB.MOM.WW.Health.Akka` | `AkkaClusterHealthCheck` with a configurable `AkkaClusterStatusPolicy` (presets: `Default` / `OtOpcUaCompat`), `ActiveNodeHealthCheck` with an optional role filter, and `AkkaActiveNodeGate` that backs `IActiveNodeGate` from cluster member state. | `ZB.MOM.WW.Health`, `Akka.Cluster` | +| `ZB.MOM.WW.Health.Akka` | `AkkaClusterHealthCheck` with a configurable `AkkaClusterStatusPolicy` (presets: `Default` / `OtOpcUaCompat`); `ClusterActiveNode` — the family-wide **oldest-`Up`-member** rule, optionally role-scoped; `ActiveNodeHealthCheck` over it (role-preference list, `activeRole`/`activeNode` in the result `data`); and `AkkaActiveNodeGate`, which backs `IActiveNodeGate` from the same rule. | `ZB.MOM.WW.Health`, `Akka.Cluster` | | `ZB.MOM.WW.Health.EntityFrameworkCore` | `DatabaseHealthCheck` — resolves `IDbContextFactory` when registered, else a scoped `TContext`; pool-safe. Default probe: `CanConnectAsync`. Optional `ProbeQuery` delegate for query-based validation. | `ZB.MOM.WW.Health`, `Microsoft.EntityFrameworkCore` | --- @@ -46,14 +46,19 @@ dotnet test ZB.MOM.WW.Health.slnx dotnet pack ZB.MOM.WW.Health.slnx -c Release -o ./artifacts ``` -All three test assemblies run offline (the cluster-data tests form a single-node in-process Akka cluster on a loopback port): +All three test assemblies run offline. The Akka suite forms real in-process clusters on loopback: a +single-node one for the cluster-data payload, and a **two-node one built so the oldest member is not +the lowest-addressed one** — the only condition under which age- and leader-ordering disagree, and so +the only fixture that can catch a leader-based active-node regression. It binds fixed ports +19540/19541 (distinct from OtOpcUa's equivalent fixture) and carries a sanity check that the +divergence actually occurred, so the assertions cannot pass for the wrong reason: | Assembly | Tests | |---|---| | `ZB.MOM.WW.Health.Tests` | 25 | -| `ZB.MOM.WW.Health.Akka.Tests` | 39 | +| `ZB.MOM.WW.Health.Akka.Tests` | 45 | | `ZB.MOM.WW.Health.EntityFrameworkCore.Tests` | 6 | -| **Total** | **70** | +| **Total** | **76** | `GeneratePackageOnBuild` is off — pack explicitly with the command above. @@ -61,7 +66,7 @@ All three test assemblies run offline (the cluster-data tests form a single-node ## Status -Built at **0.2.0** and published to the Gitea NuGet feed. **Adopted** by all four apps — adoption is tracked in the component backlog: +Built at **0.3.0** and published to the Gitea NuGet feed. **Adopted** by all four apps — adoption is tracked in the component backlog: - `~/Desktop/scadaproj/components/health/GAPS.md` — adoption order, effort, and risk diff --git a/components/health/GAPS.md b/components/health/GAPS.md index df5590a..33c0d86 100644 --- a/components/health/GAPS.md +++ b/components/health/GAPS.md @@ -17,6 +17,28 @@ reach the shared `ZB.MOM.WW.Health` library. Status legend: ⛔ gap · 🟡 part > until someone re-walks them. They also predate HistorianGateway, which is absent from every > table below. +> **⚠️ Gaps L1/L2 were closed the WRONG WAY, then reopened and closed properly at `0.3.0` +> (2026-07-24).** Everything below about the active-node check describes the leader-based design and +> is **superseded** — read [`spec/SPEC.md`](spec/SPEC.md) §2.3 for the current rule. +> +> L1 shipped `ActiveNodeHealthCheck` selecting by `ClusterState.Leader` / `RoleLeader`, and L2 backed +> `AkkaActiveNodeGate` the same way. Leadership is **address**-ordered (host, then port) with no +> relationship to time, while cluster-singleton placement is **age**-ordered. The two agree on a +> freshly-formed cluster — so every unit test and happy-path rig passed — and diverge permanently +> after any restart: the restarted node rejoins youngest but keeps its address, so if it holds the +> lower address it becomes leader while the singletons stay on the other node. During a partition +> both sides also compute themselves leader. +> +> Both apps adopted L1/L2 as written, both hit the defect on live rigs, and both **reverted to +> private copies** (ScadaBridge `OldestNodeActiveHealthCheck`, OtOpcUa `ClusterPrimaryHealthCheck`). +> The result was that the package's whole active-node surface had **zero consumers** — avoided, not +> unused. `0.3.0` promoted those two copies into one `ClusterActiveNode` primitive, rebuilt the check +> and the gate on it, and both apps migrated back and deleted their copies. +> +> **The lesson worth keeping:** a shared component can be adopted, green, and still wrong, when the +> only fixture that distinguishes right from wrong is a cluster whose oldest member is not its +> lowest-addressed one. The library now pins exactly that fixture. + ## Divergence vs spec ### §1 Endpoint tiers diff --git a/components/health/shared-contract/ZB.MOM.WW.Health.md b/components/health/shared-contract/ZB.MOM.WW.Health.md index 7686d95..3ee6cd6 100644 --- a/components/health/shared-contract/ZB.MOM.WW.Health.md +++ b/components/health/shared-contract/ZB.MOM.WW.Health.md @@ -10,7 +10,7 @@ exist: OtOpcUa `Health/` (three-tier + probes), ScadaBridge `Health/` (inline pr ``` ZB.MOM.WW.Health # core: tier convention, response writer, IActiveNodeGate, GrpcDependencyHealthCheck -ZB.MOM.WW.Health.Akka # AkkaClusterHealthCheck, ActiveNodeHealthCheck, AkkaActiveNodeGate +ZB.MOM.WW.Health.Akka # AkkaClusterHealthCheck, ClusterActiveNode, ActiveNodeHealthCheck, AkkaActiveNodeGate ZB.MOM.WW.Health.EntityFrameworkCore # DatabaseHealthCheck ``` @@ -79,7 +79,7 @@ public static class ZbHealthWriter public static Task WriteJsonAsync(HttpContext context, HealthReport report); } -/// Single-property seam: is this node the active/leader node? +/// Single-property seam: is this node the active node (oldest Up member)? /// Attach to route groups via RequireActiveNode(). Implement with AkkaActiveNodeGate (Health.Akka) /// or a project-specific implementation for non-Akka nodes. public interface IActiveNodeGate @@ -162,49 +162,77 @@ public sealed class AkkaClusterStatusPolicy public static AkkaClusterStatusPolicy OtOpcUaCompat { get; } } -/// Checks whether this node is the designated leader / active node. -/// Optional role parameter scopes the check to nodes carrying that role. -/// Register to tag ZbHealthTags.Active. +/// THE family-wide definition of "the active node": the OLDEST Up member, optionally scoped to a +/// role — the member ClusterSingletonManager places singletons on. Never cluster leadership, which is +/// address-ordered and diverges from singleton placement permanently after any restart (0.3.0). +public static class ClusterActiveNode +{ + /// The oldest Up member, optionally scoped to a role. Null when none is Up. + public static Member? OldestUpMember(IEnumerable members, string? role = null); + public static Member? OldestUpMember(Cluster cluster, string? role = null); + + /// True when the local member is Up, carries the role (if given), and no eligible member is + /// older. Identity compares UniqueAddress, so a node restarted on the same host:port is a + /// different member during the overlap. + public static bool SelfIsActive(Cluster cluster, string? role = null); + + /// The first role in rolePreference that selfRoles carries, else null. Lets a fused node answer + /// for the role its singletons are pinned to (e.g. ["admin", "driver"]). + public static string? ResolveActiveRole(IEnumerable selfRoles, IReadOnlyList rolePreference); +} + +/// Configures which role's active node a node competes to be. +public sealed class ActiveNodeHealthCheckOptions +{ + /// Candidate roles in descending precedence. Empty (default) = unscoped: oldest Up member of the + /// whole cluster. + public IReadOnlyList RolePreference { get; init; } + + /// Status when RolePreference is non-empty and the node carries none of those roles — it owns no + /// active work. Defaults to Unhealthy: the active tier exists so a load balancer can pick exactly + /// one node, and "not applicable => Healthy" made the tier answer 200 on every node (lmxopcua#494). + public HealthStatus NoActiveRoleStatus { get; init; } +} + +/// Active-tier probe: Healthy on the one node in charge, Unhealthy (503) on its standby. +/// Register to tag ZbHealthTags.Active. Selects via ClusterActiveNode (oldest Up member). +/// Result data carries selfAddress, activeRole (when scoped) and activeNode, so a standby reports +/// WHO is active. /// -/// The ActorSystem is resolved lazily from the service provider. If the ActorSystem is not yet -/// available (e.g. during startup before Akka is initialised), the check returns Degraded rather -/// than throwing. This makes the check startup-safe. +/// The ActorSystem is resolved lazily from the service provider. If the ActorSystem or cluster is not +/// yet available (e.g. during startup before Akka is initialised), the check returns Degraded rather +/// than throwing. This makes the check startup-safe. A reachable cluster with no Up member in scope +/// is Unhealthy, not Degraded — a cluster that never forms must not sit at 200. /// public sealed class ActiveNodeHealthCheck : IHealthCheck { - /// Role-less constructor: Healthy = node is Up AND cluster leader (ScadaBridge ActiveNode pattern). - /// Returns Degraded when ActorSystem/cluster is not yet ready. - /// - /// The application service provider. ActorSystem is resolved lazily so the check is - /// startup-safe: if no ActorSystem is registered yet the result is Degraded. - /// + /// Unscoped: the active node is the oldest Up member of the whole cluster. public ActiveNodeHealthCheck(IServiceProvider serviceProvider); - /// Role-filtered constructor: Healthy = (node lacks the role) OR (node carries role AND is role-singleton leader). - /// Degraded = node carries role but is not the role-singleton leader (OtOpcUa AdminRoleLeader pattern). - /// Returns Degraded when ActorSystem/cluster is not yet ready. - /// - /// The application service provider. ActorSystem is resolved lazily so the check is - /// startup-safe: if no ActorSystem is registered yet the result is Degraded. - /// + /// Role-scoped: the active node is the oldest Up member carrying `role`. public ActiveNodeHealthCheck(IServiceProvider serviceProvider, string role); + /// Configured, including a multi-role preference order. + public ActiveNodeHealthCheck(IServiceProvider serviceProvider, ActiveNodeHealthCheckOptions options); + public Task CheckHealthAsync( HealthCheckContext context, CancellationToken cancellationToken = default); } -/// IActiveNodeGate implementation that computes IsActiveNode directly from the ActorSystem -/// (SelfMember Up + cluster leader), null-guarded for startup safety. +/// IActiveNodeGate implementation that computes IsActiveNode directly from the ActorSystem via +/// ClusterActiveNode (oldest Up member, optional role scope), null-guarded for startup safety. +/// Shares its rule with ActiveNodeHealthCheck, so a gated endpoint and /health/active agree. /// Register as a singleton. Does NOT resolve ActiveNodeHealthCheck from DI. public sealed class AkkaActiveNodeGate : IActiveNodeGate { /// /// The application service provider. ActorSystem is resolved lazily; if not yet available /// IsActiveNode returns false (safe default during startup). - /// The gate checks SelfMember.Status == Up AND cluster.State.Leader == self.Address directly. + /// The gate delegates to ClusterActiveNode.SelfIsActive(cluster, role) directly. /// - public AkkaActiveNodeGate(IServiceProvider serviceProvider); + /// Optional role scope; null (default) considers every Up member. + public AkkaActiveNodeGate(IServiceProvider serviceProvider, string? role = null); public bool IsActiveNode { get; } } diff --git a/components/health/spec/SPEC.md b/components/health/spec/SPEC.md index 6612904..9cc6b98 100644 --- a/components/health/spec/SPEC.md +++ b/components/health/spec/SPEC.md @@ -10,8 +10,10 @@ three code-verified current-state docs (`../current-state/`). Goal is *path to s `/healthz`) with canonical tags `ready` / `active` / `live` and their semantics; the canonical JSON response shape; the `IActiveNodeGate` request-gating seam; a configurable `AkkaClusterHealthCheck` with two named policy presets that reconcile the diverging Akka logic in -OtOpcUa and ScadaBridge; a role-filtered `ActiveNodeHealthCheck` that unifies OtOpcUa's -`AdminRoleLeaderHealthCheck` and ScadaBridge's `ActiveNodeHealthCheck`; a generic +OtOpcUa and ScadaBridge; the `ClusterActiveNode` oldest-`Up`-member rule and the +`ActiveNodeHealthCheck` over it, which unify OtOpcUa's `ClusterPrimaryHealthCheck` and ScadaBridge's +`OldestNodeActiveHealthCheck` — the two private replacements each app wrote after the original +leader-based check proved wrong on a live rig; a generic `DatabaseHealthCheck` that covers both apps' EF Core probe patterns; a `GrpcDependencyHealthCheck` for downstream gRPC reachability. @@ -29,7 +31,7 @@ Three tiers, always served in this order, each filtered to a named tag: | Tier | Endpoint | Tag | Semantics | Healthy→ | Degraded→ | Unhealthy→ | |---|---|---|---|---|---|---| | Ready | `/health/ready` | `ready` | Can this node serve its dependencies? Fails if a DB, gRPC dependency, or cluster membership check is unhealthy. Orchestrators use this to gate traffic. | 200 | 200 | 503 | -| Active | `/health/active` | `active` | Is this the leader / active node? Fails (503) on a standby or role-member-but-not-leader node. Used to route write traffic or admin requests to exactly one node. | 200 | 200 | 503 | +| Active | `/health/active` | `active` | Is this the active node — the **oldest `Up` member**, optionally within a role? Fails (503) on a standby, and on a node that owns no active work. Used to route write traffic or admin requests to exactly one node. | 200 | 200 | 503 | | Live | `/healthz` | `live` | Bare process liveness — is the process alive and not deadlocked? **No probes registered to this tag** (predicate `_ => false`). Always 200 as long as the process can handle HTTP. | 200 | 200 | 200 | Notes: @@ -77,21 +79,45 @@ rather than Unhealthy. Registered to the `ready` tag. > `OtOpcUaCompat` column refer to states that collapse into Degraded via the member-scan result, > not to an explicit policy match. -### 2.3 Active / leader probe — `ActiveNodeHealthCheck` +### 2.3 Active-node probe — `ActiveNodeHealthCheck` -Checks whether this node is the designated leader (active node). Accepts an optional Akka -cluster role name that scopes the check to nodes carrying that role. +Checks whether this node is the **active node**: the **oldest `Up` member**, optionally scoped to a +role. Registered to the `active` tag. The rule itself is `ClusterActiveNode`, which is also what the +`IActiveNodeGate` implementation and each app's own primary/standby logic call, so an endpoint gate +and the probe an orchestrator routes by cannot disagree. -**Two behaviors unify the existing divergence:** +**Oldest, never leader.** Cluster leadership (`ClusterState.Leader` / `RoleLeader`) is +*address*-ordered — host, then port, with no relationship to time — while singleton placement is +*age*-ordered. The two agree on a freshly-formed cluster and diverge permanently after any restart: +the restarted node rejoins as the youngest but keeps its address, so if it holds the lower address it +becomes leader while the singletons, and all the work they own, stay on the other node. A leader-based +active tier therefore routes traffic to a node that is not hosting the work. During a partition both +sides also compute themselves leader, so both answer 200. -| Mode | Role param | Origin | Healthy | Degraded | Unhealthy | -|---|---|---|---|---|---| -| Role-less | `null` | ScadaBridge `ActiveNodeHealthCheck` | Node is `Up` **and** cluster leader | — | Otherwise | -| Role-filtered | e.g. `"admin"` | OtOpcUa `AdminRoleLeaderHealthCheck` | Node does **not** carry the role (not a participant — ignore it) **or** node carries the role and is the role-singleton leader | Carries the role but is **not** the role-singleton leader (role member, not leader) | — | +| Mode | `RolePreference` | Healthy | Unhealthy | +|---|---|---|---| +| Unscoped | *(empty)* | Node is the oldest `Up` member of the cluster | Otherwise | +| Role-scoped | e.g. `["admin", "driver"]` | Node is the oldest `Up` member carrying the first listed role it holds | It is not — **or** it holds none of the listed roles (see below) | -The role-filtered variant maps "not a member of the role" to Healthy (transparent — the probe -is irrelevant for this node). This is the correct behavior for heterogeneous clusters where not -every node carries every role. Registered to the `active` tag. +`RolePreference` is an ordered list so a fused node answers for the role its singletons are pinned to +(`admin` before `driver`), while a single-role node answers for its own role. + +**A node holding none of the listed roles is `Unhealthy` by default** (`NoActiveRoleStatus`). This is +deliberately *not* the "not applicable ⇒ Healthy" convention used by readiness probes: the active tier +exists so a load balancer can pick exactly one node, and a node owning no active work must not be in +that pool. The earlier role-filtered mode mapped that case to Healthy, which made the tier answer 200 +on **every** node and silently broke leader-pinning ([lmxopcua#494]). + +Startup-safe: `Degraded` (a 200) while the `ActorSystem` or cluster is not yet available, so a booting +node is not reported as a failed standby and pulled from rotation on the way up. But once the cluster +is reachable and simply has no `Up` member in scope, the answer is `Unhealthy` — a cluster that never +forms must not sit at 200 forever. + +Results carry `selfAddress`, `activeRole` (when scoped) and `activeNode` in the per-entry `data` +object, so a standby reports **who** is active and a dashboard can render a pair from either node's +payload alone. + +[lmxopcua#494]: https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/494 ### 2.4 gRPC dependency probe — `GrpcDependencyHealthCheck` @@ -151,9 +177,11 @@ plug it into `MapHealthChecks` options and also call it from custom endpoints. `IActiveNodeGate` is a single-property interface (`bool IsActiveNode { get; }`) that expresses whether the current node should accept write / active-role requests. The default implementation, `AkkaActiveNodeGate`, reads cluster state **directly**: `IsActiveNode` returns `true` iff the -`ActorSystem` is available, `SelfMember.Status == Up`, and the node is the cluster leader. It is -null-guarded and returns `false` when the `ActorSystem` is not yet ready (safe default during -startup). It does **not** resolve `ActiveNodeHealthCheck` from DI. A `RequireActiveNode()` extension on +`ActorSystem` is available and the node is the **oldest `Up` member**, optionally within a role — the +same `ClusterActiveNode` rule §2.3 describes, so a gated endpoint and the `/health/active` probe an +orchestrator routes by cannot disagree. It is null-guarded and returns `false` when the `ActorSystem` +or cluster is not yet ready (safe default during startup, matching the standby case). It does **not** +resolve `ActiveNodeHealthCheck` from DI. A `RequireActiveNode()` extension on `IEndpointConventionBuilder` attaches a policy that short-circuits with `503 Service Unavailable` on standby nodes. @@ -180,6 +208,13 @@ predicates and response writer. ## 6. Migration notes +> **Historical.** This table records the pre-adoption state (2026-06) and is kept for provenance. +> Adoption is complete. Two rows carry guidance that was later proven wrong: "replace with shared +> `ActiveNodeHealthCheck(role: "admin")`" and "replace inline `ActiveNodeGate` with +> `AkkaActiveNodeGate`" both pointed at the leader-based selection. Both apps followed it, both hit +> the defect on a live rig, and both reverted to private copies until 0.3.0 rebuilt the shared check +> on oldest-`Up`-member semantics (§2.3) — which is what they now use. + | Project | Current state | Gap | What normalizes | |---|---|---|---| | **OtOpcUa** | All three tiers present (`/health/ready`, `/health/active`, `/healthz`); `DatabaseHealthCheck`, `AkkaClusterHealthCheck`, `AdminRoleLeaderHealthCheck` inline. | Inline probes diverge from the shared policy model; no `IActiveNodeGate`. | Replace inline `AkkaClusterHealthCheck` with shared + `OtOpcUaCompat` preset; replace `AdminRoleLeaderHealthCheck` with shared `ActiveNodeHealthCheck(role: "admin")`; replace inline `DatabaseHealthCheck` with shared generic; call `app.MapZbHealth()`. |