scadaproj/components/health/spec/SPEC.md

# Health — normalized target spec

Status: **Draft**. The single design the sister projects converge on. Derived from the
three code-verified current-state docs (`../current-state/`). Goal is *path to shared code*
(`../shared-contract/ZB.MOM.WW.Health.md`), so each normalized section maps to a shared library seam.

## 0. Scope

**Normalized here:** the three-tier endpoint convention (`/health/ready`, `/health/active`,
`/healthz`) with canonical tags `ready` / `active` / `live` and their semantics; the canonical
JSON response shape; the `IActiveNodeGate` request-gating seam; a configurable
`AkkaClusterHealthCheck` with two named policy presets that reconcile the diverging Akka logic in
OtOpcUa and ScadaBridge; the `ClusterActiveNode` oldest-`Up`-member rule and the
`ActiveNodeHealthCheck` over it, which unify OtOpcUa's `ClusterPrimaryHealthCheck` and ScadaBridge's
`OldestNodeActiveHealthCheck` — the two private replacements each app wrote after the original
leader-based check proved wrong on a live rig; a generic
`DatabaseHealthCheck<TContext>` that covers both apps' EF Core probe patterns; a
`GrpcDependencyHealthCheck` for downstream gRPC reachability.

**Explicitly NOT normalized** (domain-specific — keep per project): which probes each app
registers and how it wires them to tags; orchestrator / Traefik routing rules and routing priorities;
ScadaBridge's `HealthMonitoring/` domain-aggregation pipeline — this is a distributed, actor-based
domain-health telemetry system (background services + Akka actors that aggregate site-cluster signals
into a central health picture) and is **not** an ASP.NET health-probe; it is an independent concern
that happens to share the word "health".

## 1. Tier convention

Three tiers, always served in this order, each filtered to a named tag:

| Tier | Endpoint | Tag | Semantics | Healthy→ | Degraded→ | Unhealthy→ |
|---|---|---|---|---|---|---|
| Ready | `/health/ready` | `ready` | Can this node serve its dependencies? Fails if a DB, gRPC dependency, or cluster membership check is unhealthy. Orchestrators use this to gate traffic. | 200 | 200 | 503 |
| Active | `/health/active` | `active` | Is this the active node — the **oldest `Up` member**, optionally within a role? Fails (503) on a standby, and on a node that owns no active work. Used to route write traffic or admin requests to exactly one node. | 200 | 200 | 503 |
| Live | `/healthz` | `live` | Bare process liveness — is the process alive and not deadlocked? **No probes registered to this tag** (predicate `_ => false`). Always 200 as long as the process can handle HTTP. | 200 | 200 | 200 |

Notes:

- The `live` tier intentionally carries no probes. Registering a probe to `live` is an error —
  a liveness failure that kills the pod should be reserved for total process hangs, not probe failures.
- `Degraded` maps to HTTP 200 (not 503) for the `ready` and `active` tiers. Orchestrators use 503
  to remove a node from load-balancing; Degraded means "still up but degraded" — remove the node
  only on hard failure.
- The tag names (`ready`, `active`, `live`) are declared as constants in `ZbHealthTags` and used
  consistently across all three apps. Per-project probe registrations must filter by these tags.

## 2. Probe catalog

### 2.1 Database probe — `DatabaseHealthCheck<TContext>`

Wraps an EF Core `DbContext` to verify database reachability. Default behavior calls
`context.Database.CanConnectAsync()` — matches ScadaBridge's pattern. An optional delegate
(`Func<TContext, CancellationToken, Task>`) overrides the default for more specific validation
(matches OtOpcUa's "query `Deployments`" pattern). Registered to the `ready` tag.

### 2.2 Akka cluster probe — `AkkaClusterHealthCheck`

Checks the local node's cluster membership status via Akka.Cluster. The status-to-health
mapping is **configurable** through `AkkaClusterStatusPolicy`.

**Two named policy presets reconcile the existing divergence:**

| Preset | Origin | `Up` / `Joining` | `Leaving` / `Exiting` | Other (`WeaklyUp`, `Down`, `Removed`, `Unknown`) |
|---|---|---|---|---|
| `AkkaClusterStatusPolicy.Default` | ScadaBridge `AkkaClusterHealthCheck.cs` | Healthy | Degraded | Unhealthy |
| `AkkaClusterStatusPolicy.OtOpcUaCompat` | OtOpcUa `AkkaClusterHealthCheck.cs` | Healthy (if self is `Up` among reachable members) | Degraded[^1] | Degraded |

[^1]: In the `OtOpcUaCompat` member-scan approach, `Leaving`/`Exiting` statuses also map to
Degraded because a member with those statuses will not appear with `Status == Up` in the reachable
member set — the scan finds self without `Up`, so the result is Degraded.

The `Default` preset is the convergence target. `OtOpcUaCompat` is provided for backward
compatibility during OtOpcUa's migration; it maps any non-`Up`-among-members state to Degraded
rather than Unhealthy. Registered to the `ready` tag.

> **Note on error/exception cases:** in both modes, if the ActorSystem is not yet ready or cluster
> state is inaccessible (e.g. during startup), the check returns Degraded (startup-safety rule).
> The status cells in the table above describe the normal-operation path only; the "—" cells in the
> `OtOpcUaCompat` column refer to states that collapse into Degraded via the member-scan result,
> not to an explicit policy match.

### 2.3 Active-node probe — `ActiveNodeHealthCheck`

Checks whether this node is the **active node**: the **oldest `Up` member**, optionally scoped to a
role. Registered to the `active` tag. The rule itself is `ClusterActiveNode`, which is also what the
`IActiveNodeGate` implementation and each app's own primary/standby logic call, so an endpoint gate
and the probe an orchestrator routes by cannot disagree.

**Oldest, never leader.** Cluster leadership (`ClusterState.Leader` / `RoleLeader`) is
*address*-ordered — host, then port, with no relationship to time — while singleton placement is
*age*-ordered. The two agree on a freshly-formed cluster and diverge permanently after any restart:
the restarted node rejoins as the youngest but keeps its address, so if it holds the lower address it
becomes leader while the singletons, and all the work they own, stay on the other node. A leader-based
active tier therefore routes traffic to a node that is not hosting the work. During a partition both
sides also compute themselves leader, so both answer 200.

| Mode | `RolePreference` | Healthy | Unhealthy |
|---|---|---|---|
| Unscoped | *(empty)* | Node is the oldest `Up` member of the cluster | Otherwise |
| Role-scoped | e.g. `["admin", "driver"]` | Node is the oldest `Up` member carrying the first listed role it holds | It is not — **or** it holds none of the listed roles (see below) |

`RolePreference` is an ordered list so a fused node answers for the role its singletons are pinned to
(`admin` before `driver`), while a single-role node answers for its own role.

**A node holding none of the listed roles is `Unhealthy` by default** (`NoActiveRoleStatus`). This is
deliberately *not* the "not applicable ⇒ Healthy" convention used by readiness probes: the active tier
exists so a load balancer can pick exactly one node, and a node owning no active work must not be in
that pool. The earlier role-filtered mode mapped that case to Healthy, which made the tier answer 200
on **every** node and silently broke leader-pinning ([lmxopcua#494]).

Startup-safe: `Degraded` (a 200) while the `ActorSystem` or cluster is not yet available, so a booting
node is not reported as a failed standby and pulled from rotation on the way up. But once the cluster
is reachable and simply has no `Up` member in scope, the answer is `Unhealthy` — a cluster that never
forms must not sit at 200 forever.

Results carry `selfAddress`, `activeRole` (when scoped) and `activeNode` in the per-entry `data`
object, so a standby reports **who** is active and a dashboard can render a pair from either node's
payload alone.

[lmxopcua#494]: https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/494

### 2.4 gRPC dependency probe — `GrpcDependencyHealthCheck`

Checks that a downstream gRPC channel is reachable by invoking a caller-supplied probe
delegate (`Func<GrpcChannel, CancellationToken, Task<bool>>`). The default probe calls
`GrpcChannel.ConnectAsync`. Used by:

- OtOpcUa — checks the MxAccessGateway gRPC channel.
- MxGateway — checks the x86 worker gRPC channel.

Registered to the `ready` tag.

## 3. Response-writer contract

All health endpoints share one canonical JSON serializer. The shape is lifted from ScadaBridge's
`HealthChecks.UI.Client` style and becomes the library default (replacing per-project divergence).

**Content-type:** `application/json`

**Shape:**

```json
{
  "status": "Healthy",
  "totalDurationMs": 12,
  "entries": {
    "database": {
      "status": "Healthy",
      "description": "SQL Server reachable",
      "durationMs": 12
    },
    "akka-cluster": {
      "status": "Healthy",
      "description": "Member status: Up",
      "durationMs": 0.1
    }
  }
}
```

**Field rules:**

| Field | Type | Notes |
|---|---|---|
| `status` | string | `"Healthy"` \| `"Degraded"` \| `"Unhealthy"` — the aggregate across all filtered checks |
| `totalDurationMs` | long | Total wall-clock time for all probes in this tier, milliseconds |
| `entries` | object | Keyed by check registration name |
| `entries.<name>.status` | string | Per-check status |
| `entries.<name>.description` | string? | Human-readable detail (may be null) |
| `entries.<name>.durationMs` | number | Per-check elapsed time, milliseconds |

The writer is exposed as a static `Task WriteJsonAsync(HttpContext, HealthReport)` so consumers can
plug it into `MapHealthChecks` options and also call it from custom endpoints.

## 4. Active-node gating seam — `IActiveNodeGate`

`IActiveNodeGate` is a single-property interface (`bool IsActiveNode { get; }`) that expresses
whether the current node should accept write / active-role requests. The default implementation,
`AkkaActiveNodeGate`, reads cluster state **directly**: `IsActiveNode` returns `true` iff the
`ActorSystem` is available and the node is the **oldest `Up` member**, optionally within a role — the
same `ClusterActiveNode` rule §2.3 describes, so a gated endpoint and the `/health/active` probe an
orchestrator routes by cannot disagree. It is null-guarded and returns `false` when the `ActorSystem`
or cluster is not yet ready (safe default during startup, matching the standby case). It does **not**
resolve `ActiveNodeHealthCheck` from DI. A `RequireActiveNode()` extension on
`IEndpointConventionBuilder` attaches a policy that short-circuits with `503 Service Unavailable`
on standby nodes.

This seam is generalized from ScadaBridge's `ActiveNodeGate.cs`. It is in the core `ZB.MOM.WW.Health`
package (not the Akka satellite) so MxGateway can implement it without an Akka dependency if needed.

## 5. Endpoint registration

`app.MapZbHealth()` maps all three tiers in one call:

```csharp
app.MapZbHealth();                        // all three tiers, defaults
app.MapZbHealth(o => {
    o.ReadyPath   = "/health/ready";      // override paths if needed
    o.ActivePath  = "/health/active";
    o.LivePath    = "/healthz";
    o.ResponseWriter = ZbHealthWriter.WriteJsonAsync;
});
```

The library does **not** call `services.AddHealthChecks()` — that is the app's responsibility, as
the probe set is per-project. `MapZbHealth` only maps the three endpoints with the correct tag
predicates and response writer.

## 6. Migration notes

> **Historical.** This table records the pre-adoption state (2026-06) and is kept for provenance.
> Adoption is complete. Two rows carry guidance that was later proven wrong: "replace with shared
> `ActiveNodeHealthCheck(role: "admin")`" and "replace inline `ActiveNodeGate` with
> `AkkaActiveNodeGate`" both pointed at the leader-based selection. Both apps followed it, both hit
> the defect on a live rig, and both reverted to private copies until 0.3.0 rebuilt the shared check
> on oldest-`Up`-member semantics (§2.3) — which is what they now use.

| Project | Current state | Gap | What normalizes |
|---|---|---|---|
| **OtOpcUa** | All three tiers present (`/health/ready`, `/health/active`, `/healthz`); `DatabaseHealthCheck`, `AkkaClusterHealthCheck`, `AdminRoleLeaderHealthCheck` inline. | Inline probes diverge from the shared policy model; no `IActiveNodeGate`. | Replace inline `AkkaClusterHealthCheck` with shared + `OtOpcUaCompat` preset; replace `AdminRoleLeaderHealthCheck` with shared `ActiveNodeHealthCheck(role: "admin")`; replace inline `DatabaseHealthCheck` with shared generic; call `app.MapZbHealth()`. |
| **ScadaBridge** | `/health/ready` + `/health/active` present; no `/healthz`; `DatabaseHealthCheck`, `AkkaClusterHealthCheck`, `ActiveNodeHealthCheck`, `ActiveNodeGate` inline. | Missing `/healthz` live tier; inline implementations. | Add `/healthz` via `MapZbHealth()`; replace inline probes with shared equivalents (Default policy); replace inline `ActiveNodeGate` with `AkkaActiveNodeGate`. |
| **MxGateway** | Only `/health/live` (custom `GatewayHealthReply`); `AddHealthChecks()` called but zero probes registered. | Missing `ready` and `active` tiers; no probes; not using standard health middleware. | Replace custom endpoint with `app.MapZbHealth()`; register `GrpcDependencyHealthCheck` for the x86 worker channel on the `ready` tag. |

## 7. Acceptance (what "converged" means)

A project is converged when: (a) it calls `app.MapZbHealth()` and exposes all three canonical
endpoints; (b) its Akka probes (if applicable) use the `AkkaClusterHealthCheck` + `ActiveNodeHealthCheck`
from `ZB.MOM.WW.Health.Akka` with the Default policy; (c) its DB probe uses `DatabaseHealthCheck<TContext>`
from `ZB.MOM.WW.Health.EntityFrameworkCore`; (d) its gRPC-dependency probe (if applicable) uses
`GrpcDependencyHealthCheck`; (e) its `IActiveNodeGate` implementation is `AkkaActiveNodeGate`
(or a project-specific implementation of the shared interface); (f) all health endpoints return the
canonical JSON shape defined in §3.