docs(health): oldest-Up-member is the active-node rule (0.3.0)

Rewrites the active-node sections of the spec and shared contract, which described the leader-based design that both consumers rejected. - SPEC §2.3 replaced: the rule, the ordered RolePreference, why a node holding none of the preferred roles is Unhealthy rather than Healthy, and the data payload. §4 (IActiveNodeGate) and the tier table follow. - SPEC §6 marked historical: it still tells adopters to use ActiveNodeHealthCheck(role: "admin") and AkkaActiveNodeGate, which is the advice that produced the defect. Kept for provenance, annotated so nobody follows it again. - GAPS: gaps L1/L2 recorded as closed the wrong way, reopened, then closed at 0.3.0 — including the part worth remembering, that both apps adopted the shared check as specified, hit it live, and reverted to private copies, so the surface reached zero consumers by being avoided rather than unused. - Index rows updated with the consumer-visible changes and both app branches.
2026-07-24 13:24:30 -04:00
parent a61e041e58
commit 6a4fa220de
5 changed files with 139 additions and 49 deletions
@@ -143,7 +143,7 @@ each project's **code-verified current state**, and the **gaps** between. See
 |---|---|---|---|---|
 | Auth (login / identity / authz) | Adopted (lib `0.1.4`; all 3 apps, merged to **local default** main/master + **pushed to origin** (gitea)) | Shared `ZB.MOM.WW.Auth` lib | [`components/auth/`](components/auth/) | [`ZB.MOM.WW.Auth/`](ZB.MOM.WW.Auth/) |
 | UI Theme (layout / tokens / components) | Adopted (feed `0.2.0`–`0.3.1`; **all 4 apps pinned `0.3.1`**, pushed to origin) | Shared `ZB.MOM.WW.Theme` RCL | [`components/ui-theme/`](components/ui-theme/) | [`ZB.MOM.WW.Theme/`](ZB.MOM.WW.Theme/) |
-| Health (readiness / liveness / active-node) | **Adopted** (feed carries `0.1.0`/`0.2.0`/`0.2.1`; **all 4 apps** reference it + wire `MapZbHealth`). **`0.2.0` (2026-07-24)** added an optional per-entry `data` object to the canonical health JSON and made `AkkaClusterHealthCheck` publish its cluster view (`leader`/`selfAddress`/`selfRoles`/`memberCount`/`unreachableCount`) — additive, so pre-0.2.0 payloads still parse. **`0.2.1` (2026-07-24) is a behaviour fix with a live consumer impact:** the active tier now returns **Unhealthy (503)** on a role-member-but-not-leader node instead of `Degraded`, which the spec maps to 200. The old value made `/health/active` answer 200 on *every* node, so it could not distinguish active from standby over HTTP at all — and OtOpcUa's Traefik routes the admin UI by exactly this probe, so its leader-pinning had silently never worked ([`lmxopcua#494`](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/494)). **Only OtOpcUa is affected** — it is the sole user of the role-filtered mode; ScadaBridge deliberately uses its own `OldestNodeActiveHealthCheck` (leadership is address-ordered and diverges from the singleton host), and mxgw/HistorianGateway have no Akka cluster. | Shared `ZB.MOM.WW.Health` lib | [`components/health/`](components/health/) | [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) |
+| Health (readiness / liveness / active-node) | **Adopted** (feed carries `0.1.0`/`0.2.0`/`0.2.1`; **all 4 apps** reference it + wire `MapZbHealth`). **`0.2.0` (2026-07-24)** added an optional per-entry `data` object to the canonical health JSON and made `AkkaClusterHealthCheck` publish its cluster view (`leader`/`selfAddress`/`selfRoles`/`memberCount`/`unreachableCount`) — additive, so pre-0.2.0 payloads still parse. **`0.2.1` (2026-07-24) is a behaviour fix with a live consumer impact:** the active tier now returns **Unhealthy (503)** on a role-member-but-not-leader node instead of `Degraded`, which the spec maps to 200. The old value made `/health/active` answer 200 on *every* node, so it could not distinguish active from standby over HTTP at all — and OtOpcUa's Traefik routes the admin UI by exactly this probe, so its leader-pinning had silently never worked ([`lmxopcua#494`](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/494)). **`0.3.0` (2026-07-24) finished that fix and ENDED a family-wide duplication.** 0.2.1 made the tier a real 503 but still selected by `RoleLeader`; on a rebuilt rig the two orderings diverge live (akka leader `central-1`, oldest admin `central-2` — the singleton host), so it would have pinned Traefik to the node *not* hosting the work. **Leadership is address-ordered (host, then port) and has no relationship to time; singleton placement is age-ordered.** They agree on a freshly-formed cluster — which is why single-node and happy-path tests never caught it — and diverge permanently after any restart. Both Akka apps had already discovered this independently and each written a private replacement rather than use the shared check (ScadaBridge `OldestNodeActiveHealthCheck`, OtOpcUa `ClusterPrimaryHealthCheck`), so the package's entire active-node surface had **zero consumers — not unused but avoided, twice, for the same reason.** 0.3.0 promotes those two into one `ClusterActiveNode` primitive (oldest `Up` member, optional role scope, plus an ordered role-preference so a fused node answers for the role its singletons are pinned to); `ActiveNodeHealthCheck` and `AkkaActiveNodeGate` both delegate to it, and results now carry `activeRole`/`selfAddress`/`activeNode` in the 0.2.0 `data` object so a **standby reports who IS active**. Two further behaviour changes: a node holding **none** of the preferred roles is now **Unhealthy**, not Healthy — the old "not applicable ⇒ Healthy" is exactly what made the tier answer 200 everywhere, and the case is reachable (OtOpcUa's `RoleParser` admits `dev`-only and `cluster-`only nodes); and identity compares `UniqueAddress`, so a node restarted on the same host:port is correctly a different member. **Both apps migrated and their private copies deleted** — OtOpcUa (`feat/health-0.3.0-active-node` @ `88adec04`) also routes `RedundancyStateActor.SelectOldestUpMemberOfRole` through the shared primitive, so the tier and the OPC UA `ServiceLevel` 250/240 split cannot drift; ScadaBridge (`feat/site-node-health` @ `5d4853a1`) routes `ActiveNodeEvaluator` through it, unifying the S&F delivery gate, the heartbeat `IsActive` stamp, the inbound-API gate and the health tier on one rule. ScadaBridge keeps `SitePairActiveNodeHealthCheck` deliberately — it is a thin adapter over the site's own role-scoped `IClusterNodeProvider` (itself now backed by the shared rule), not a second copy of it. mxgw/HistorianGateway have no Akka cluster and are unaffected. Both app branches are **committed but NOT pushed or merged**. | Shared `ZB.MOM.WW.Health` lib | [`components/health/`](components/health/) | [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) |
 | Observability (metrics / traces / logs) | **Adopted** (libs `0.1.0`, on the feed; **all 4 apps** call `AddZbTelemetry`; `AddZbSerilog` in all but ScadaBridge, which keeps its own factory by design) | Shared `ZB.MOM.WW.Telemetry` lib + `.Serilog` | [`components/observability/`](components/observability/) | [`ZB.MOM.WW.Telemetry/`](ZB.MOM.WW.Telemetry/) |
 | Config + validation (options / startup validation) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed — the "local only / not yet pushed" caveat was stale) | Shared `ZB.MOM.WW.Configuration` lib | [`components/configuration/`](components/configuration/) | [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/) |
 | Audit (event model + writer seam) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed to origin) | Shared `ZB.MOM.WW.Audit` lib | [`components/audit/`](components/audit/) | [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) |
@@ -4,7 +4,7 @@ Health-check libraries for the **ZB.MOM.WW SCADA family** (OtOpcUa, MxAccessGate

 The library normalizes the three-tier health endpoint convention (`/health/ready`, `/health/active`, `/healthz`) and provides reusable probe implementations so the three sister projects share a common surface without duplicating probe logic.

-**Built at 0.2.0 and published to the Gitea NuGet feed. Adopted by all four apps** — adoption is tracked in `~/Desktop/scadaproj/components/health/GAPS.md`.
+**Built at 0.3.0 and published to the Gitea NuGet feed. Adopted by all four apps** — adoption is tracked in `~/Desktop/scadaproj/components/health/GAPS.md`.

 ---

@@ -13,7 +13,7 @@ The library normalizes the three-tier health endpoint convention (`/health/ready
 | Package | Responsibilities | Key Dependencies |
 |---|---|---|
 | `ZB.MOM.WW.Health` | Core tier convention, `MapZbHealth` extension, canonical JSON writer (`ZbHealthWriter`), `IActiveNodeGate` seam, `GrpcDependencyHealthCheck` reachability probe, tier-tag constants (`ZbHealthTags`). No Akka or EF dependency. | `Microsoft.AspNetCore.App` (framework ref), `Grpc.Net.Client` |
-| `ZB.MOM.WW.Health.Akka` | `AkkaClusterHealthCheck` with a configurable `AkkaClusterStatusPolicy` (presets: `Default` / `OtOpcUaCompat`), `ActiveNodeHealthCheck` with an optional role filter, and `AkkaActiveNodeGate` that backs `IActiveNodeGate` from cluster member state. | `ZB.MOM.WW.Health`, `Akka.Cluster` |
+| `ZB.MOM.WW.Health.Akka` | `AkkaClusterHealthCheck` with a configurable `AkkaClusterStatusPolicy` (presets: `Default` / `OtOpcUaCompat`); `ClusterActiveNode` — the family-wide **oldest-`Up`-member** rule, optionally role-scoped; `ActiveNodeHealthCheck` over it (role-preference list, `activeRole`/`activeNode` in the result `data`); and `AkkaActiveNodeGate`, which backs `IActiveNodeGate` from the same rule. | `ZB.MOM.WW.Health`, `Akka.Cluster` |
 | `ZB.MOM.WW.Health.EntityFrameworkCore` | `DatabaseHealthCheck<TContext>` — resolves `IDbContextFactory<TContext>` when registered, else a scoped `TContext`; pool-safe. Default probe: `CanConnectAsync`. Optional `ProbeQuery` delegate for query-based validation. | `ZB.MOM.WW.Health`, `Microsoft.EntityFrameworkCore` |

 ---
@@ -46,14 +46,19 @@ dotnet test ZB.MOM.WW.Health.slnx
 dotnet pack ZB.MOM.WW.Health.slnx -c Release -o ./artifacts
 ```

-All three test assemblies run offline (the cluster-data tests form a single-node in-process Akka cluster on a loopback port):
+All three test assemblies run offline. The Akka suite forms real in-process clusters on loopback: a
+single-node one for the cluster-data payload, and a **two-node one built so the oldest member is not
+the lowest-addressed one** — the only condition under which age- and leader-ordering disagree, and so
+the only fixture that can catch a leader-based active-node regression. It binds fixed ports
+19540/19541 (distinct from OtOpcUa's equivalent fixture) and carries a sanity check that the
+divergence actually occurred, so the assertions cannot pass for the wrong reason:

 | Assembly | Tests |
 |---|---|
 | `ZB.MOM.WW.Health.Tests` | 25 |
-| `ZB.MOM.WW.Health.Akka.Tests` | 39 |
+| `ZB.MOM.WW.Health.Akka.Tests` | 45 |
 | `ZB.MOM.WW.Health.EntityFrameworkCore.Tests` | 6 |
-| **Total** | **70** |
+| **Total** | **76** |

 `GeneratePackageOnBuild` is off — pack explicitly with the command above.

@@ -61,7 +66,7 @@ All three test assemblies run offline (the cluster-data tests form a single-node

 ## Status

-Built at **0.2.0** and published to the Gitea NuGet feed. **Adopted** by all four apps — adoption is tracked in the component backlog:
+Built at **0.3.0** and published to the Gitea NuGet feed. **Adopted** by all four apps — adoption is tracked in the component backlog:

 - `~/Desktop/scadaproj/components/health/GAPS.md` — adoption order, effort, and risk

@@ -17,6 +17,28 @@ reach the shared `ZB.MOM.WW.Health` library. Status legend: ⛔ gap · 🟡 part
 > until someone re-walks them. They also predate HistorianGateway, which is absent from every
 > table below.

+> **⚠️ Gaps L1/L2 were closed the WRONG WAY, then reopened and closed properly at `0.3.0`
+> (2026-07-24).** Everything below about the active-node check describes the leader-based design and
+> is **superseded** — read [`spec/SPEC.md`](spec/SPEC.md) §2.3 for the current rule.
+>
+> L1 shipped `ActiveNodeHealthCheck` selecting by `ClusterState.Leader` / `RoleLeader`, and L2 backed
+> `AkkaActiveNodeGate` the same way. Leadership is **address**-ordered (host, then port) with no
+> relationship to time, while cluster-singleton placement is **age**-ordered. The two agree on a
+> freshly-formed cluster — so every unit test and happy-path rig passed — and diverge permanently
+> after any restart: the restarted node rejoins youngest but keeps its address, so if it holds the
+> lower address it becomes leader while the singletons stay on the other node. During a partition
+> both sides also compute themselves leader.
+>
+> Both apps adopted L1/L2 as written, both hit the defect on live rigs, and both **reverted to
+> private copies** (ScadaBridge `OldestNodeActiveHealthCheck`, OtOpcUa `ClusterPrimaryHealthCheck`).
+> The result was that the package's whole active-node surface had **zero consumers** — avoided, not
+> unused. `0.3.0` promoted those two copies into one `ClusterActiveNode` primitive, rebuilt the check
+> and the gate on it, and both apps migrated back and deleted their copies.
+>
+> **The lesson worth keeping:** a shared component can be adopted, green, and still wrong, when the
+> only fixture that distinguishes right from wrong is a cluster whose oldest member is not its
+> lowest-addressed one. The library now pins exactly that fixture.
+
 ## Divergence vs spec

 ### §1 Endpoint tiers
@@ -10,7 +10,7 @@ exist: OtOpcUa `Health/` (three-tier + probes), ScadaBridge `Health/` (inline pr

 ```
 ZB.MOM.WW.Health                        # core: tier convention, response writer, IActiveNodeGate, GrpcDependencyHealthCheck
-ZB.MOM.WW.Health.Akka                   # AkkaClusterHealthCheck, ActiveNodeHealthCheck, AkkaActiveNodeGate
+ZB.MOM.WW.Health.Akka                   # AkkaClusterHealthCheck, ClusterActiveNode, ActiveNodeHealthCheck, AkkaActiveNodeGate
 ZB.MOM.WW.Health.EntityFrameworkCore    # DatabaseHealthCheck<TContext>
 ```

@@ -79,7 +79,7 @@ public static class ZbHealthWriter
    public static Task WriteJsonAsync(HttpContext context, HealthReport report);
 }

-/// Single-property seam: is this node the active/leader node?
+/// Single-property seam: is this node the active node (oldest Up member)?
 /// Attach to route groups via RequireActiveNode(). Implement with AkkaActiveNodeGate (Health.Akka)
 /// or a project-specific implementation for non-Akka nodes.
 public interface IActiveNodeGate
@@ -162,49 +162,77 @@ public sealed class AkkaClusterStatusPolicy
    public static AkkaClusterStatusPolicy OtOpcUaCompat { get; }
 }

-/// Checks whether this node is the designated leader / active node.
-/// Optional role parameter scopes the check to nodes carrying that role.
-/// Register to tag ZbHealthTags.Active.
+/// THE family-wide definition of "the active node": the OLDEST Up member, optionally scoped to a
+/// role — the member ClusterSingletonManager places singletons on. Never cluster leadership, which is
+/// address-ordered and diverges from singleton placement permanently after any restart (0.3.0).
+public static class ClusterActiveNode
+{
+    /// The oldest Up member, optionally scoped to a role. Null when none is Up.
+    public static Member? OldestUpMember(IEnumerable<Member> members, string? role = null);
+    public static Member? OldestUpMember(Cluster cluster, string? role = null);
+
+    /// True when the local member is Up, carries the role (if given), and no eligible member is
+    /// older. Identity compares UniqueAddress, so a node restarted on the same host:port is a
+    /// different member during the overlap.
+    public static bool SelfIsActive(Cluster cluster, string? role = null);
+
+    /// The first role in rolePreference that selfRoles carries, else null. Lets a fused node answer
+    /// for the role its singletons are pinned to (e.g. ["admin", "driver"]).
+    public static string? ResolveActiveRole(IEnumerable<string> selfRoles, IReadOnlyList<string> rolePreference);
+}
+
+/// Configures which role's active node a node competes to be.
+public sealed class ActiveNodeHealthCheckOptions
+{
+    /// Candidate roles in descending precedence. Empty (default) = unscoped: oldest Up member of the
+    /// whole cluster.
+    public IReadOnlyList<string> RolePreference { get; init; }
+
+    /// Status when RolePreference is non-empty and the node carries none of those roles — it owns no
+    /// active work. Defaults to Unhealthy: the active tier exists so a load balancer can pick exactly
+    /// one node, and "not applicable => Healthy" made the tier answer 200 on every node (lmxopcua#494).
+    public HealthStatus NoActiveRoleStatus { get; init; }
+}
+
+/// Active-tier probe: Healthy on the one node in charge, Unhealthy (503) on its standby.
+/// Register to tag ZbHealthTags.Active. Selects via ClusterActiveNode (oldest Up member).
+/// Result data carries selfAddress, activeRole (when scoped) and activeNode, so a standby reports
+/// WHO is active.
 /// <remarks>
-/// The ActorSystem is resolved lazily from the service provider. If the ActorSystem is not yet
-/// available (e.g. during startup before Akka is initialised), the check returns Degraded rather
-/// than throwing. This makes the check startup-safe.
+/// The ActorSystem is resolved lazily from the service provider. If the ActorSystem or cluster is not
+/// yet available (e.g. during startup before Akka is initialised), the check returns Degraded rather
+/// than throwing. This makes the check startup-safe. A reachable cluster with no Up member in scope
+/// is Unhealthy, not Degraded — a cluster that never forms must not sit at 200.
 /// </remarks>
 public sealed class ActiveNodeHealthCheck : IHealthCheck
 {
-    /// Role-less constructor: Healthy = node is Up AND cluster leader (ScadaBridge ActiveNode pattern).
-    /// Returns Degraded when ActorSystem/cluster is not yet ready.
-    /// <param name="serviceProvider">
-    /// The application service provider. ActorSystem is resolved lazily so the check is
-    /// startup-safe: if no ActorSystem is registered yet the result is Degraded.
-    /// </param>
+    /// Unscoped: the active node is the oldest Up member of the whole cluster.
    public ActiveNodeHealthCheck(IServiceProvider serviceProvider);

-    /// Role-filtered constructor: Healthy = (node lacks the role) OR (node carries role AND is role-singleton leader).
-    /// Degraded = node carries role but is not the role-singleton leader (OtOpcUa AdminRoleLeader pattern).
-    /// Returns Degraded when ActorSystem/cluster is not yet ready.
-    /// <param name="serviceProvider">
-    /// The application service provider. ActorSystem is resolved lazily so the check is
-    /// startup-safe: if no ActorSystem is registered yet the result is Degraded.
-    /// </param>
+    /// Role-scoped: the active node is the oldest Up member carrying `role`.
    public ActiveNodeHealthCheck(IServiceProvider serviceProvider, string role);

+    /// Configured, including a multi-role preference order.
+    public ActiveNodeHealthCheck(IServiceProvider serviceProvider, ActiveNodeHealthCheckOptions options);
+
    public Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken  cancellationToken = default);
 }

-/// IActiveNodeGate implementation that computes IsActiveNode directly from the ActorSystem
-/// (SelfMember Up + cluster leader), null-guarded for startup safety.
+/// IActiveNodeGate implementation that computes IsActiveNode directly from the ActorSystem via
+/// ClusterActiveNode (oldest Up member, optional role scope), null-guarded for startup safety.
+/// Shares its rule with ActiveNodeHealthCheck, so a gated endpoint and /health/active agree.
 /// Register as a singleton. Does NOT resolve ActiveNodeHealthCheck from DI.
 public sealed class AkkaActiveNodeGate : IActiveNodeGate
 {
    /// <param name="serviceProvider">
    /// The application service provider. ActorSystem is resolved lazily; if not yet available
    /// IsActiveNode returns false (safe default during startup).
-    /// The gate checks SelfMember.Status == Up AND cluster.State.Leader == self.Address directly.
+    /// The gate delegates to ClusterActiveNode.SelfIsActive(cluster, role) directly.
    /// </param>
-    public AkkaActiveNodeGate(IServiceProvider serviceProvider);
+    /// <param name="role">Optional role scope; null (default) considers every Up member.</param>
+    public AkkaActiveNodeGate(IServiceProvider serviceProvider, string? role = null);

    public bool IsActiveNode { get; }
 }
@@ -10,8 +10,10 @@ three code-verified current-state docs (`../current-state/`). Goal is *path to s
 `/healthz`) with canonical tags `ready` / `active` / `live` and their semantics; the canonical
 JSON response shape; the `IActiveNodeGate` request-gating seam; a configurable
 `AkkaClusterHealthCheck` with two named policy presets that reconcile the diverging Akka logic in
-OtOpcUa and ScadaBridge; a role-filtered `ActiveNodeHealthCheck` that unifies OtOpcUa's
-`AdminRoleLeaderHealthCheck` and ScadaBridge's `ActiveNodeHealthCheck`; a generic
+OtOpcUa and ScadaBridge; the `ClusterActiveNode` oldest-`Up`-member rule and the
+`ActiveNodeHealthCheck` over it, which unify OtOpcUa's `ClusterPrimaryHealthCheck` and ScadaBridge's
+`OldestNodeActiveHealthCheck` — the two private replacements each app wrote after the original
+leader-based check proved wrong on a live rig; a generic
 `DatabaseHealthCheck<TContext>` that covers both apps' EF Core probe patterns; a
 `GrpcDependencyHealthCheck` for downstream gRPC reachability.

@@ -29,7 +31,7 @@ Three tiers, always served in this order, each filtered to a named tag:
 | Tier | Endpoint | Tag | Semantics | Healthy→ | Degraded→ | Unhealthy→ |
 |---|---|---|---|---|---|---|
 | Ready | `/health/ready` | `ready` | Can this node serve its dependencies? Fails if a DB, gRPC dependency, or cluster membership check is unhealthy. Orchestrators use this to gate traffic. | 200 | 200 | 503 |
-| Active | `/health/active` | `active` | Is this the leader / active node? Fails (503) on a standby or role-member-but-not-leader node. Used to route write traffic or admin requests to exactly one node. | 200 | 200 | 503 |
+| Active | `/health/active` | `active` | Is this the active node — the **oldest `Up` member**, optionally within a role? Fails (503) on a standby, and on a node that owns no active work. Used to route write traffic or admin requests to exactly one node. | 200 | 200 | 503 |
 | Live | `/healthz` | `live` | Bare process liveness — is the process alive and not deadlocked? **No probes registered to this tag** (predicate `_ => false`). Always 200 as long as the process can handle HTTP. | 200 | 200 | 200 |

 Notes:
@@ -77,21 +79,45 @@ rather than Unhealthy. Registered to the `ready` tag.
 > `OtOpcUaCompat` column refer to states that collapse into Degraded via the member-scan result,
 > not to an explicit policy match.

-### 2.3 Active / leader probe — `ActiveNodeHealthCheck`
+### 2.3 Active-node probe — `ActiveNodeHealthCheck`

-Checks whether this node is the designated leader (active node). Accepts an optional Akka
-cluster role name that scopes the check to nodes carrying that role.
+Checks whether this node is the **active node**: the **oldest `Up` member**, optionally scoped to a
+role. Registered to the `active` tag. The rule itself is `ClusterActiveNode`, which is also what the
+`IActiveNodeGate` implementation and each app's own primary/standby logic call, so an endpoint gate
+and the probe an orchestrator routes by cannot disagree.

-**Two behaviors unify the existing divergence:**
+**Oldest, never leader.** Cluster leadership (`ClusterState.Leader` / `RoleLeader`) is
+*address*-ordered — host, then port, with no relationship to time — while singleton placement is
+*age*-ordered. The two agree on a freshly-formed cluster and diverge permanently after any restart:
+the restarted node rejoins as the youngest but keeps its address, so if it holds the lower address it
+becomes leader while the singletons, and all the work they own, stay on the other node. A leader-based
+active tier therefore routes traffic to a node that is not hosting the work. During a partition both
+sides also compute themselves leader, so both answer 200.

-| Mode | Role param | Origin | Healthy | Degraded | Unhealthy |
-|---|---|---|---|---|---|
-| Role-less | `null` | ScadaBridge `ActiveNodeHealthCheck` | Node is `Up` **and** cluster leader | — | Otherwise |
-| Role-filtered | e.g. `"admin"` | OtOpcUa `AdminRoleLeaderHealthCheck` | Node does **not** carry the role (not a participant — ignore it) **or** node carries the role and is the role-singleton leader | Carries the role but is **not** the role-singleton leader (role member, not leader) | — |
+| Mode | `RolePreference` | Healthy | Unhealthy |
+|---|---|---|---|
+| Unscoped | *(empty)* | Node is the oldest `Up` member of the cluster | Otherwise |
+| Role-scoped | e.g. `["admin", "driver"]` | Node is the oldest `Up` member carrying the first listed role it holds | It is not — **or** it holds none of the listed roles (see below) |

-The role-filtered variant maps "not a member of the role" to Healthy (transparent — the probe
-is irrelevant for this node). This is the correct behavior for heterogeneous clusters where not
-every node carries every role. Registered to the `active` tag.
+`RolePreference` is an ordered list so a fused node answers for the role its singletons are pinned to
+(`admin` before `driver`), while a single-role node answers for its own role.
+
+**A node holding none of the listed roles is `Unhealthy` by default** (`NoActiveRoleStatus`). This is
+deliberately *not* the "not applicable ⇒ Healthy" convention used by readiness probes: the active tier
+exists so a load balancer can pick exactly one node, and a node owning no active work must not be in
+that pool. The earlier role-filtered mode mapped that case to Healthy, which made the tier answer 200
+on **every** node and silently broke leader-pinning ([lmxopcua#494]).
+
+Startup-safe: `Degraded` (a 200) while the `ActorSystem` or cluster is not yet available, so a booting
+node is not reported as a failed standby and pulled from rotation on the way up. But once the cluster
+is reachable and simply has no `Up` member in scope, the answer is `Unhealthy` — a cluster that never
+forms must not sit at 200 forever.
+
+Results carry `selfAddress`, `activeRole` (when scoped) and `activeNode` in the per-entry `data`
+object, so a standby reports **who** is active and a dashboard can render a pair from either node's
+payload alone.
+
+[lmxopcua#494]: https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/494

 ### 2.4 gRPC dependency probe — `GrpcDependencyHealthCheck`

@@ -151,9 +177,11 @@ plug it into `MapHealthChecks` options and also call it from custom endpoints.
 `IActiveNodeGate` is a single-property interface (`bool IsActiveNode { get; }`) that expresses
 whether the current node should accept write / active-role requests. The default implementation,
 `AkkaActiveNodeGate`, reads cluster state **directly**: `IsActiveNode` returns `true` iff the
-`ActorSystem` is available, `SelfMember.Status == Up`, and the node is the cluster leader. It is
-null-guarded and returns `false` when the `ActorSystem` is not yet ready (safe default during
-startup). It does **not** resolve `ActiveNodeHealthCheck` from DI. A `RequireActiveNode()` extension on
+`ActorSystem` is available and the node is the **oldest `Up` member**, optionally within a role — the
+same `ClusterActiveNode` rule §2.3 describes, so a gated endpoint and the `/health/active` probe an
+orchestrator routes by cannot disagree. It is null-guarded and returns `false` when the `ActorSystem`
+or cluster is not yet ready (safe default during startup, matching the standby case). It does **not**
+resolve `ActiveNodeHealthCheck` from DI. A `RequireActiveNode()` extension on
 `IEndpointConventionBuilder` attaches a policy that short-circuits with `503 Service Unavailable`
 on standby nodes.

@@ -180,6 +208,13 @@ predicates and response writer.

 ## 6. Migration notes

+> **Historical.** This table records the pre-adoption state (2026-06) and is kept for provenance.
+> Adoption is complete. Two rows carry guidance that was later proven wrong: "replace with shared
+> `ActiveNodeHealthCheck(role: "admin")`" and "replace inline `ActiveNodeGate` with
+> `AkkaActiveNodeGate`" both pointed at the leader-based selection. Both apps followed it, both hit
+> the defect on a live rig, and both reverted to private copies until 0.3.0 rebuilt the shared check
+> on oldest-`Up`-member semantics (§2.3) — which is what they now use.
+
 | Project | Current state | Gap | What normalizes |
 |---|---|---|---|
 | **OtOpcUa** | All three tiers present (`/health/ready`, `/health/active`, `/healthz`); `DatabaseHealthCheck`, `AkkaClusterHealthCheck`, `AdminRoleLeaderHealthCheck` inline. | Inline probes diverge from the shared policy model; no `IActiveNodeGate`. | Replace inline `AkkaClusterHealthCheck` with shared + `OtOpcUaCompat` preset; replace `AdminRoleLeaderHealthCheck` with shared `ActiveNodeHealthCheck(role: "admin")`; replace inline `DatabaseHealthCheck` with shared generic; call `app.MapZbHealth()`. |