# ZB.MOM.WW.Health Health-check libraries for the **ZB.MOM.WW SCADA family** (OtOpcUa, MxAccessGateway, ScadaBridge). These are **libraries, not a service** — each package is linked directly into the consuming application at build time. There is no central health process or network hop; probes run in-process alongside the application. The library normalizes the three-tier health endpoint convention (`/health/ready`, `/health/active`, `/healthz`) and provides reusable probe implementations so the three sister projects share a common surface without duplicating probe logic. **Built at 0.3.0 and published to the Gitea NuGet feed. Adopted by all four apps** — adoption is tracked in `~/Desktop/scadaproj/components/health/GAPS.md`. --- ## Packages | Package | Responsibilities | Key Dependencies | |---|---|---| | `ZB.MOM.WW.Health` | Core tier convention, `MapZbHealth` extension, canonical JSON writer (`ZbHealthWriter`), `IActiveNodeGate` seam, `GrpcDependencyHealthCheck` reachability probe, tier-tag constants (`ZbHealthTags`). No Akka or EF dependency. | `Microsoft.AspNetCore.App` (framework ref), `Grpc.Net.Client` | | `ZB.MOM.WW.Health.Akka` | `AkkaClusterHealthCheck` with a configurable `AkkaClusterStatusPolicy` (presets: `Default` / `OtOpcUaCompat`); `ClusterActiveNode` — the family-wide **oldest-`Up`-member** rule, optionally role-scoped; `ActiveNodeHealthCheck` over it (role-preference list, `activeRole`/`activeNode` in the result `data`); and `AkkaActiveNodeGate`, which backs `IActiveNodeGate` from the same rule. | `ZB.MOM.WW.Health`, `Akka.Cluster` | | `ZB.MOM.WW.Health.EntityFrameworkCore` | `DatabaseHealthCheck` — resolves `IDbContextFactory` when registered, else a scoped `TContext`; pool-safe. Default probe: `CanConnectAsync`. Optional `ProbeQuery` delegate for query-based validation. | `ZB.MOM.WW.Health`, `Microsoft.EntityFrameworkCore` | --- ## Consumer matrix | Consumer | `ZB.MOM.WW.Health` (core) | `ZB.MOM.WW.Health.Akka` | `ZB.MOM.WW.Health.EntityFrameworkCore` | |---|:---:|:---:|:---:| | **OtOpcUa** | yes | yes | yes | | **MxAccessGateway** | yes | — | — | | **ScadaBridge** | yes | yes | yes | MxAccessGateway consumes the core package only — it has no Akka cluster and no EF DbContext. OtOpcUa and ScadaBridge consume all three packages. --- ## Build, test, and pack commands ```bash # From ZB.MOM.WW.Health/ # Build dotnet build ZB.MOM.WW.Health.slnx dotnet build ZB.MOM.WW.Health.slnx -c Release # Test (no external dependencies — no running Akka cluster, no database) dotnet test ZB.MOM.WW.Health.slnx # Pack (three .nupkg files land in artifacts/) dotnet pack ZB.MOM.WW.Health.slnx -c Release -o ./artifacts ``` All three test assemblies run offline. The Akka suite forms real in-process clusters on loopback: a single-node one for the cluster-data payload, and a **two-node one built so the oldest member is not the lowest-addressed one** — the only condition under which age- and leader-ordering disagree, and so the only fixture that can catch a leader-based active-node regression. It binds fixed ports 19540/19541 (distinct from OtOpcUa's equivalent fixture) and carries a sanity check that the divergence actually occurred, so the assertions cannot pass for the wrong reason: | Assembly | Tests | |---|---| | `ZB.MOM.WW.Health.Tests` | 25 | | `ZB.MOM.WW.Health.Akka.Tests` | 45 | | `ZB.MOM.WW.Health.EntityFrameworkCore.Tests` | 6 | | **Total** | **76** | `GeneratePackageOnBuild` is off — pack explicitly with the command above. --- ## Status Built at **0.3.0** and published to the Gitea NuGet feed. **Adopted** by all four apps — adoption is tracked in the component backlog: - `~/Desktop/scadaproj/components/health/GAPS.md` — adoption order, effort, and risk Design documentation: - `~/Desktop/scadaproj/components/health/spec/SPEC.md` — normalized three-tier target - `~/Desktop/scadaproj/components/health/shared-contract/ZB.MOM.WW.Health.md` — proposed API (aligned to shipped code) - `~/Desktop/scadaproj/components/health/current-state/` — per-project current state (code-verified) ## Index history (migrated from the scadaproj root CLAUDE.md) > Moved verbatim out of the scadaproj root `CLAUDE.md` on 2026-09-03 so it loads only when needed. **This file is now the authoritative place for this history — append new entries here and keep the root index row short.** ### Component-table row as it read in the index | Component | Status | Goal | Design | Implementation | |---|---|---|---|---| | Health (readiness / liveness / active-node) | **Adopted** (feed carries `0.1.0`–`0.3.0`, current lib version **`0.3.0`**; **all 4 apps** reference it + wire `MapZbHealth`). **`0.2.0` (2026-07-24)** added an optional per-entry `data` object to the canonical health JSON and made `AkkaClusterHealthCheck` publish its cluster view (`leader`/`selfAddress`/`selfRoles`/`memberCount`/`unreachableCount`) — additive, so pre-0.2.0 payloads still parse. **`0.2.1` (2026-07-24) is a behaviour fix with a live consumer impact:** the active tier now returns **Unhealthy (503)** on a role-member-but-not-leader node instead of `Degraded`, which the spec maps to 200. The old value made `/health/active` answer 200 on *every* node, so it could not distinguish active from standby over HTTP at all — and OtOpcUa's Traefik routes the admin UI by exactly this probe, so its leader-pinning had silently never worked ([`lmxopcua#494`](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/494)). **`0.3.0` (2026-07-24) finished that fix and ENDED a family-wide duplication.** 0.2.1 made the tier a real 503 but still selected by `RoleLeader`; on a rebuilt rig the two orderings diverge live (akka leader `central-1`, oldest admin `central-2` — the singleton host), so it would have pinned Traefik to the node *not* hosting the work. **Leadership is address-ordered (host, then port) and has no relationship to time; singleton placement is age-ordered.** They agree on a freshly-formed cluster — which is why single-node and happy-path tests never caught it — and diverge permanently after any restart. Both Akka apps had already discovered this independently and each written a private replacement rather than use the shared check (ScadaBridge `OldestNodeActiveHealthCheck`, OtOpcUa `ClusterPrimaryHealthCheck`), so the package's entire active-node surface had **zero consumers — not unused but avoided, twice, for the same reason.** 0.3.0 promotes those two into one `ClusterActiveNode` primitive (oldest `Up` member, optional role scope, plus an ordered role-preference so a fused node answers for the role its singletons are pinned to); `ActiveNodeHealthCheck` and `AkkaActiveNodeGate` both delegate to it, and results now carry `activeRole`/`selfAddress`/`activeNode` in the 0.2.0 `data` object so a **standby reports who IS active**. Two further behaviour changes: a node holding **none** of the preferred roles is now **Unhealthy**, not Healthy — the old "not applicable ⇒ Healthy" is exactly what made the tier answer 200 everywhere, and the case is reachable (OtOpcUa's `RoleParser` admits `dev`-only and `cluster-`only nodes); and identity compares `UniqueAddress`, so a node restarted on the same host:port is correctly a different member. **Both apps migrated and their private copies deleted** — OtOpcUa (`feat/health-0.3.0-active-node` @ `88adec04`) also routes `RedundancyStateActor.SelectOldestUpMemberOfRole` through the shared primitive, so the tier and the OPC UA `ServiceLevel` 250/240 split cannot drift; ScadaBridge (`feat/site-node-health` @ `5d4853a1`) routes `ActiveNodeEvaluator` through it, unifying the S&F delivery gate, the heartbeat `IsActive` stamp, the inbound-API gate and the health tier on one rule. ScadaBridge keeps `SitePairActiveNodeHealthCheck` deliberately — it is a thin adapter over the site's own role-scoped `IClusterNodeProvider` (itself now backed by the shared rule), not a second copy of it. mxgw/HistorianGateway have no Akka cluster and are unaffected. **Both app branches are MERGED and PUSHED** (verified 2026-08-07): OtOpcUa `88adec04` is on `origin/master`; ScadaBridge merged via `b3dc17a5` (2026-07-24) on `origin/main`, private-copy deletions included. | Shared `ZB.MOM.WW.Health` lib | [`components/health/`](components/health/) | [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) | ### Narrative The health component is fully populated: a normalized [`spec`](components/health/spec/SPEC.md), a [`shared-contract`](components/health/shared-contract/ZB.MOM.WW.Health.md), three [`current-state`](components/health/current-state/) docs, and an adoption [`GAPS`](components/health/GAPS.md) backlog. Shared = three-tier endpoint convention (ready/active/healthz) + canonical JSON writer + `IActiveNodeGate` seam + `GrpcDependencyHealthCheck` + `AkkaClusterHealthCheck` + `ActiveNodeHealthCheck` + `DatabaseHealthCheck`; left per-project = which probes each app registers, orchestrator wiring, and ScadaBridge's distributed health-monitoring pipeline. The shared library is **built and lives in this repo** at [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) (.NET 10; 3 packages — `ZB.MOM.WW.Health`, `ZB.MOM.WW.Health.Akka`, `ZB.MOM.WW.Health.EntityFrameworkCore`; 64 tests (verified 2026-07-18); `dotnet pack` → 3 nupkgs @ 0.1.0). **Adopted by all four apps** — verified 2026-07-18 (each references the package and wires `MapZbHealth`); the previous "not yet adopted" claim was stale. Per-app probe coverage vs the spec has NOT been re-verified — see [`components/health/GAPS.md`](components/health/GAPS.md). Build/test from `ZB.MOM.WW.Health/`: `dotnet test`. Consumer matrix: MxAccessGateway → core only; OtOpcUa & ScadaBridge → all three packages.