dohertj2/scadaproj
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(health): oldest-Up-member is the active-node rule (0.3.0)

Browse Source 
Rewrites the active-node sections of the spec and shared contract, which
described the leader-based design that both consumers rejected.

- SPEC §2.3 replaced: the rule, the ordered RolePreference, why a node holding
  none of the preferred roles is Unhealthy rather than Healthy, and the data
  payload. §4 (IActiveNodeGate) and the tier table follow.
- SPEC §6 marked historical: it still tells adopters to use
  ActiveNodeHealthCheck(role: "admin") and AkkaActiveNodeGate, which is the
  advice that produced the defect. Kept for provenance, annotated so nobody
  follows it again.
- GAPS: gaps L1/L2 recorded as closed the wrong way, reopened, then closed at
  0.3.0 — including the part worth remembering, that both apps adopted the
  shared check as specified, hit it live, and reverted to private copies, so the
  surface reached zero consumers by being avoided rather than unused.
- Index rows updated with the consumer-visible changes and both app branches.
This commit is contained in:
Joseph Doherty
2026-07-24 13:24:30 -04:00
parent a61e041e58
commit 6a4fa220de
5 changed files with 139 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files
components/health/spec/SPEC.md
+52 -17
View File
@@ -10,8 +10,10 @@ three code-verified current-state docs (`../current-state/`). Goal is *path to s
`/healthz`) with canonical tags `ready` / `active` / `live` and their semantics; the canonical
JSON response shape; the `IActiveNodeGate` request-gating seam; a configurable
`AkkaClusterHealthCheck` with two named policy presets that reconcile the diverging Akka logic in
OtOpcUa and ScadaBridge; a role-filtered `ActiveNodeHealthCheck` that unifies OtOpcUa's
`AdminRoleLeaderHealthCheck` and ScadaBridge's `ActiveNodeHealthCheck`; a generic
OtOpcUa and ScadaBridge; the `ClusterActiveNode` oldest-`Up`-member rule and the
`ActiveNodeHealthCheck` over it, which unify OtOpcUa's `ClusterPrimaryHealthCheck` and ScadaBridge's
`OldestNodeActiveHealthCheck` — the two private replacements each app wrote after the original
leader-based check proved wrong on a live rig; a generic
`DatabaseHealthCheck<TContext>` that covers both apps' EF Core probe patterns; a
`GrpcDependencyHealthCheck` for downstream gRPC reachability.
@@ -29,7 +31,7 @@ Three tiers, always served in this order, each filtered to a named tag:
| Tier | Endpoint | Tag | Semantics | Healthy→ | Degraded→ | Unhealthy→ |
|---|---|---|---|---|---|---|
| Ready | `/health/ready` | `ready` | Can this node serve its dependencies? Fails if a DB, gRPC dependency, or cluster membership check is unhealthy. Orchestrators use this to gate traffic. | 200 | 200 | 503 |
| Active | `/health/active` | `active` | Is this the leader / active node? Fails (503) on a standby or role-member-but-not-leader node. Used to route write traffic or admin requests to exactly one node. | 200 | 200 | 503 |
| Active | `/health/active` | `active` | Is this the active node — the **oldest `Up` member**, optionally within a role? Fails (503) on a standby, and on a node that owns no active work. Used to route write traffic or admin requests to exactly one node. | 200 | 200 | 503 |
| Live | `/healthz` | `live` | Bare process liveness — is the process alive and not deadlocked? **No probes registered to this tag** (predicate `_ => false`). Always 200 as long as the process can handle HTTP. | 200 | 200 | 200 |
Notes:
@@ -77,21 +79,45 @@ rather than Unhealthy. Registered to the `ready` tag.
> `OtOpcUaCompat` column refer to states that collapse into Degraded via the member-scan result,
> not to an explicit policy match.
### 2.3 Active / leader probe — `ActiveNodeHealthCheck`
### 2.3 Active-node probe — `ActiveNodeHealthCheck`
Checks whether this node is the designated leader (active node). Accepts an optional Akka
cluster role name that scopes the check to nodes carrying that role.
Checks whether this node is the **active node**: the **oldest `Up` member**, optionally scoped to a
role. Registered to the `active` tag. The rule itself is `ClusterActiveNode`, which is also what the
`IActiveNodeGate` implementation and each app's own primary/standby logic call, so an endpoint gate
and the probe an orchestrator routes by cannot disagree.
**Two behaviors unify the existing divergence:**
**Oldest, never leader.** Cluster leadership (`ClusterState.Leader` / `RoleLeader`) is
*address*-ordered — host, then port, with no relationship to time — while singleton placement is
*age*-ordered. The two agree on a freshly-formed cluster and diverge permanently after any restart:
the restarted node rejoins as the youngest but keeps its address, so if it holds the lower address it
becomes leader while the singletons, and all the work they own, stay on the other node. A leader-based
active tier therefore routes traffic to a node that is not hosting the work. During a partition both
sides also compute themselves leader, so both answer 200.
| Mode | Role param | Origin | Healthy | Degraded | Unhealthy |
|---|---|---|---|---|---|
| Role-less | `null` | ScadaBridge `ActiveNodeHealthCheck` | Node is `Up` **and** cluster leader | — | Otherwise |
| Role-filtered | e.g. `"admin"` | OtOpcUa `AdminRoleLeaderHealthCheck` | Node does **not** carry the role (not a participant — ignore it) **or** node carries the role and is the role-singleton leader | Carries the role but is **not** the role-singleton leader (role member, not leader) | — |
| Mode | `RolePreference` | Healthy | Unhealthy |
|---|---|---|---|
| Unscoped | *(empty)* | Node is the oldest `Up` member of the cluster | Otherwise |
| Role-scoped | e.g. `["admin", "driver"]` | Node is the oldest `Up` member carrying the first listed role it holds | It is not — **or** it holds none of the listed roles (see below) |
The role-filtered variant maps "not a member of the role" to Healthy (transparent — the probe
is irrelevant for this node). This is the correct behavior for heterogeneous clusters where not
every node carries every role. Registered to the `active` tag.
`RolePreference` is an ordered list so a fused node answers for the role its singletons are pinned to
(`admin` before `driver`), while a single-role node answers for its own role.
**A node holding none of the listed roles is `Unhealthy` by default** (`NoActiveRoleStatus`). This is
deliberately *not* the "not applicable ⇒ Healthy" convention used by readiness probes: the active tier
exists so a load balancer can pick exactly one node, and a node owning no active work must not be in
that pool. The earlier role-filtered mode mapped that case to Healthy, which made the tier answer 200
on **every** node and silently broke leader-pinning ([lmxopcua#494]).
Startup-safe: `Degraded` (a 200) while the `ActorSystem` or cluster is not yet available, so a booting
node is not reported as a failed standby and pulled from rotation on the way up. But once the cluster
is reachable and simply has no `Up` member in scope, the answer is `Unhealthy` — a cluster that never
forms must not sit at 200 forever.
Results carry `selfAddress`, `activeRole` (when scoped) and `activeNode` in the per-entry `data`
object, so a standby reports **who** is active and a dashboard can render a pair from either node's
payload alone.
[lmxopcua#494]: https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/494
### 2.4 gRPC dependency probe — `GrpcDependencyHealthCheck`
@@ -151,9 +177,11 @@ plug it into `MapHealthChecks` options and also call it from custom endpoints.
`IActiveNodeGate` is a single-property interface (`bool IsActiveNode { get; }`) that expresses
whether the current node should accept write / active-role requests. The default implementation,
`AkkaActiveNodeGate`, reads cluster state **directly**: `IsActiveNode` returns `true` iff the
`ActorSystem` is available, `SelfMember.Status == Up`, and the node is the cluster leader. It is
null-guarded and returns `false` when the `ActorSystem` is not yet ready (safe default during
startup). It does **not** resolve `ActiveNodeHealthCheck` from DI. A `RequireActiveNode()` extension on
`ActorSystem` is available and the node is the **oldest `Up` member**, optionally within a role — the
same `ClusterActiveNode` rule §2.3 describes, so a gated endpoint and the `/health/active` probe an
orchestrator routes by cannot disagree. It is null-guarded and returns `false` when the `ActorSystem`
or cluster is not yet ready (safe default during startup, matching the standby case). It does **not**
resolve `ActiveNodeHealthCheck` from DI. A `RequireActiveNode()` extension on
`IEndpointConventionBuilder` attaches a policy that short-circuits with `503 Service Unavailable`
on standby nodes.
@@ -180,6 +208,13 @@ predicates and response writer.
## 6. Migration notes
> **Historical.** This table records the pre-adoption state (2026-06) and is kept for provenance.
> Adoption is complete. Two rows carry guidance that was later proven wrong: "replace with shared
> `ActiveNodeHealthCheck(role: "admin")`" and "replace inline `ActiveNodeGate` with
> `AkkaActiveNodeGate`" both pointed at the leader-based selection. Both apps followed it, both hit
> the defect on a live rig, and both reverted to private copies until 0.3.0 rebuilt the shared check
> on oldest-`Up`-member semantics (§2.3) — which is what they now use.
| Project | Current state | Gap | What normalizes |
|---|---|---|---|
| **OtOpcUa** | All three tiers present (`/health/ready`, `/health/active`, `/healthz`); `DatabaseHealthCheck`, `AkkaClusterHealthCheck`, `AdminRoleLeaderHealthCheck` inline. | Inline probes diverge from the shared policy model; no `IActiveNodeGate`. | Replace inline `AkkaClusterHealthCheck` with shared + `OtOpcUaCompat` preset; replace `AdminRoleLeaderHealthCheck` with shared `ActiveNodeHealthCheck(role: "admin")`; replace inline `DatabaseHealthCheck` with shared generic; call `app.MapZbHealth()`. |