scadaproj/components/health/spec/SPEC.md

Health — normalized target spec

Status: Draft. The single design the sister projects converge on. Derived from the three code-verified current-state docs (../current-state/). Goal is path to shared code (../shared-contract/ZB.MOM.WW.Health.md), so each normalized section maps to a shared library seam.

0. Scope

Normalized here: the three-tier endpoint convention (/health/ready, /health/active, /healthz) with canonical tags ready / active / live and their semantics; the canonical JSON response shape; the IActiveNodeGate request-gating seam; a configurable AkkaClusterHealthCheck with two named policy presets that reconcile the diverging Akka logic in OtOpcUa and ScadaBridge; the ClusterActiveNode oldest-Up-member rule and the ActiveNodeHealthCheck over it, which unify OtOpcUa's ClusterPrimaryHealthCheck and ScadaBridge's OldestNodeActiveHealthCheck — the two private replacements each app wrote after the original leader-based check proved wrong on a live rig; a generic DatabaseHealthCheck<TContext> that covers both apps' EF Core probe patterns; a GrpcDependencyHealthCheck for downstream gRPC reachability.

Explicitly NOT normalized (domain-specific — keep per project): which probes each app registers and how it wires them to tags; orchestrator / Traefik routing rules and routing priorities; ScadaBridge's HealthMonitoring/ domain-aggregation pipeline — this is a distributed, actor-based domain-health telemetry system (background services + Akka actors that aggregate site-cluster signals into a central health picture) and is not an ASP.NET health-probe; it is an independent concern that happens to share the word "health".

1. Tier convention

Three tiers, always served in this order, each filtered to a named tag:

Tier	Endpoint	Tag	Semantics	Healthy→	Degraded→	Unhealthy→
Ready	/health/ready	ready	Can this node serve its dependencies? Fails if a DB, gRPC dependency, or cluster membership check is unhealthy. Orchestrators use this to gate traffic.	200	200	503
Active	/health/active	active	Is this the active node — the oldest Up member, optionally within a role? Fails (503) on a standby, and on a node that owns no active work. Used to route write traffic or admin requests to exactly one node.	200	200	503
Live	/healthz	live	Bare process liveness — is the process alive and not deadlocked? No probes registered to this tag (predicate _ => false). Always 200 as long as the process can handle HTTP.	200	200	200

Notes:

• The live tier intentionally carries no probes. Registering a probe to live is an error — a liveness failure that kills the pod should be reserved for total process hangs, not probe failures.
• Degraded maps to HTTP 200 (not 503) for the ready and active tiers. Orchestrators use 503 to remove a node from load-balancing; Degraded means "still up but degraded" — remove the node only on hard failure.
• The tag names (ready, active, live) are declared as constants in ZbHealthTags and used consistently across all three apps. Per-project probe registrations must filter by these tags.

2. Probe catalog

2.1 Database probe — DatabaseHealthCheck<TContext>

Wraps an EF Core DbContext to verify database reachability. Default behavior calls context.Database.CanConnectAsync() — matches ScadaBridge's pattern. An optional delegate (Func<TContext, CancellationToken, Task>) overrides the default for more specific validation (matches OtOpcUa's "query Deployments" pattern). Registered to the ready tag.

2.2 Akka cluster probe — AkkaClusterHealthCheck

Checks the local node's cluster membership status via Akka.Cluster. The status-to-health mapping is configurable through AkkaClusterStatusPolicy.

Two named policy presets reconcile the existing divergence:

Preset	Origin	Up / Joining	Leaving / Exiting	Other (WeaklyUp, Down, Removed, Unknown)
AkkaClusterStatusPolicy.Default	ScadaBridge AkkaClusterHealthCheck.cs	Healthy	Degraded	Unhealthy
AkkaClusterStatusPolicy.OtOpcUaCompat	OtOpcUa AkkaClusterHealthCheck.cs	Healthy (if self is Up among reachable members)	Degraded1	Degraded

The Default preset is the convergence target. OtOpcUaCompat is provided for backward compatibility during OtOpcUa's migration; it maps any non-Up-among-members state to Degraded rather than Unhealthy. Registered to the ready tag.

Note on error/exception cases: in both modes, if the ActorSystem is not yet ready or cluster state is inaccessible (e.g. during startup), the check returns Degraded (startup-safety rule). The status cells in the table above describe the normal-operation path only; the "—" cells in the OtOpcUaCompat column refer to states that collapse into Degraded via the member-scan result, not to an explicit policy match.

2.3 Active-node probe — ActiveNodeHealthCheck

Checks whether this node is the active node: the oldest Up member, optionally scoped to a role. Registered to the active tag. The rule itself is ClusterActiveNode, which is also what the IActiveNodeGate implementation and each app's own primary/standby logic call, so an endpoint gate and the probe an orchestrator routes by cannot disagree.

Oldest, never leader. Cluster leadership (ClusterState.Leader / RoleLeader) is address-ordered — host, then port, with no relationship to time — while singleton placement is age-ordered. The two agree on a freshly-formed cluster and diverge permanently after any restart: the restarted node rejoins as the youngest but keeps its address, so if it holds the lower address it becomes leader while the singletons, and all the work they own, stay on the other node. A leader-based active tier therefore routes traffic to a node that is not hosting the work. During a partition both sides also compute themselves leader, so both answer 200.

Mode	RolePreference	Healthy	Unhealthy
Unscoped	(empty)	Node is the oldest Up member of the cluster	Otherwise
Role-scoped	e.g. ["admin", "driver"]	Node is the oldest Up member carrying the first listed role it holds	It is not — or it holds none of the listed roles (see below)

RolePreference is an ordered list so a fused node answers for the role its singletons are pinned to (admin before driver), while a single-role node answers for its own role.

A node holding none of the listed roles is Unhealthy by default (NoActiveRoleStatus). This is deliberately not the "not applicable ⇒ Healthy" convention used by readiness probes: the active tier exists so a load balancer can pick exactly one node, and a node owning no active work must not be in that pool. The earlier role-filtered mode mapped that case to Healthy, which made the tier answer 200 on every node and silently broke leader-pinning (lmxopcua#494).

Startup-safe: Degraded (a 200) while the ActorSystem or cluster is not yet available, so a booting node is not reported as a failed standby and pulled from rotation on the way up. But once the cluster is reachable and simply has no Up member in scope, the answer is Unhealthy — a cluster that never forms must not sit at 200 forever.

Results carry selfAddress, activeRole (when scoped) and activeNode in the per-entry data object, so a standby reports who is active and a dashboard can render a pair from either node's payload alone.

2.4 gRPC dependency probe — GrpcDependencyHealthCheck

Checks that a downstream gRPC channel is reachable by invoking a caller-supplied probe delegate (Func<GrpcChannel, CancellationToken, Task<bool>>). The default probe calls GrpcChannel.ConnectAsync. Used by:

• OtOpcUa — checks the MxAccessGateway gRPC channel.
• MxGateway — checks the x86 worker gRPC channel.

Registered to the ready tag.

3. Response-writer contract

All health endpoints share one canonical JSON serializer. The shape is lifted from ScadaBridge's HealthChecks.UI.Client style and becomes the library default (replacing per-project divergence).

Content-type: application/json

Shape:

{
  "status": "Healthy",
  "totalDurationMs": 12,
  "entries": {
    "database": {
      "status": "Healthy",
      "description": "SQL Server reachable",
      "durationMs": 12
    },
    "akka-cluster": {
      "status": "Healthy",
      "description": "Member status: Up",
      "durationMs": 0.1
    }
  }
}

Field rules:

Field	Type	Notes
status	string	"Healthy" | "Degraded" | "Unhealthy" — the aggregate across all filtered checks
totalDurationMs	long	Total wall-clock time for all probes in this tier, milliseconds
entries	object	Keyed by check registration name
entries.<name>.status	string	Per-check status
entries.<name>.description	string?	Human-readable detail (may be null)
entries.<name>.durationMs	number	Per-check elapsed time, milliseconds

The writer is exposed as a static Task WriteJsonAsync(HttpContext, HealthReport) so consumers can plug it into MapHealthChecks options and also call it from custom endpoints.

4. Active-node gating seam — IActiveNodeGate

IActiveNodeGate is a single-property interface (bool IsActiveNode { get; }) that expresses whether the current node should accept write / active-role requests. The default implementation, AkkaActiveNodeGate, reads cluster state directly: IsActiveNode returns true iff the ActorSystem is available and the node is the oldest Up member, optionally within a role — the same ClusterActiveNode rule §2.3 describes, so a gated endpoint and the /health/active probe an orchestrator routes by cannot disagree. It is null-guarded and returns false when the ActorSystem or cluster is not yet ready (safe default during startup, matching the standby case). It does not resolve ActiveNodeHealthCheck from DI. A RequireActiveNode() extension on IEndpointConventionBuilder attaches a policy that short-circuits with 503 Service Unavailable on standby nodes.

This seam is generalized from ScadaBridge's ActiveNodeGate.cs. It is in the core ZB.MOM.WW.Health package (not the Akka satellite) so MxGateway can implement it without an Akka dependency if needed.

5. Endpoint registration

app.MapZbHealth() maps all three tiers in one call:

app.MapZbHealth();                        // all three tiers, defaults
app.MapZbHealth(o => {
    o.ReadyPath   = "/health/ready";      // override paths if needed
    o.ActivePath  = "/health/active";
    o.LivePath    = "/healthz";
    o.ResponseWriter = ZbHealthWriter.WriteJsonAsync;
});

The library does not call services.AddHealthChecks() — that is the app's responsibility, as the probe set is per-project. MapZbHealth only maps the three endpoints with the correct tag predicates and response writer.

6. Migration notes

Historical. This table records the pre-adoption state (2026-06) and is kept for provenance. Adoption is complete. Two rows carry guidance that was later proven wrong: "replace with shared ActiveNodeHealthCheck(role: "admin")" and "replace inline ActiveNodeGate with AkkaActiveNodeGate" both pointed at the leader-based selection. Both apps followed it, both hit the defect on a live rig, and both reverted to private copies until 0.3.0 rebuilt the shared check on oldest-Up-member semantics (§2.3) — which is what they now use.

Project	Current state	Gap	What normalizes
OtOpcUa	All three tiers present (/health/ready, /health/active, /healthz); DatabaseHealthCheck, AkkaClusterHealthCheck, AdminRoleLeaderHealthCheck inline.	Inline probes diverge from the shared policy model; no IActiveNodeGate.	Replace inline AkkaClusterHealthCheck with shared + OtOpcUaCompat preset; replace AdminRoleLeaderHealthCheck with shared ActiveNodeHealthCheck(role: "admin"); replace inline DatabaseHealthCheck with shared generic; call app.MapZbHealth().
ScadaBridge	/health/ready + /health/active present; no /healthz; DatabaseHealthCheck, AkkaClusterHealthCheck, ActiveNodeHealthCheck, ActiveNodeGate inline.	Missing /healthz live tier; inline implementations.	Add /healthz via MapZbHealth(); replace inline probes with shared equivalents (Default policy); replace inline ActiveNodeGate with AkkaActiveNodeGate.
MxGateway	Only /health/live (custom GatewayHealthReply); AddHealthChecks() called but zero probes registered.	Missing ready and active tiers; no probes; not using standard health middleware.	Replace custom endpoint with app.MapZbHealth(); register GrpcDependencyHealthCheck for the x86 worker channel on the ready tag.

7. Acceptance (what "converged" means)

A project is converged when: (a) it calls app.MapZbHealth() and exposes all three canonical endpoints; (b) its Akka probes (if applicable) use the AkkaClusterHealthCheck + ActiveNodeHealthCheck from ZB.MOM.WW.Health.Akka with the Default policy; (c) its DB probe uses DatabaseHealthCheck<TContext> from ZB.MOM.WW.Health.EntityFrameworkCore; (d) its gRPC-dependency probe (if applicable) uses GrpcDependencyHealthCheck; (e) its IActiveNodeGate implementation is AkkaActiveNodeGate (or a project-specific implementation of the shared interface); (f) all health endpoints return the canonical JSON shape defined in §3.

---

1. In the OtOpcUaCompat member-scan approach, Leaving/Exiting statuses also map to Degraded because a member with those statuses will not appear with Status == Up in the reachable member set — the scan finds self without Up, so the result is Degraded. ↩︎