From 9cbe65c8444038cec4c0e899914c82430b3fa39d Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 9 Nov 2026 14:00:44 -0500 Subject: [PATCH] docs(health): re-verify per-app probe coverage against the SPEC; refresh GAPS and current-state (scadaproj#11) Audited the registration and mapping code of each app at its origin default branch: OtOpcUa c904472a, MxAccessGateway de86f8b8, ScadaBridge c7d868c5, HistorianGateway 4d6a9ad (all on Health 0.3.0, all MapZbHealth with default paths: /healthz runs nothing, /health/ready and /health/active by tag). GAPS.md now carries the verified per-endpoint coverage matrix and a SPEC-by-app table; each current-state doc lists exactly what each endpoint runs, and HistorianGateway gets its first current-state doc. Recorded: mxaccessgw#206 (worker-executable + galaxy-sql readiness checks, on /health/ready) is merged; historiangw#87 (session-pool check latches 503 until restart) is still open at 4d6a9ad. Deferred P2 downstream probes are now per-app; ScadaBridge IActiveNodeGate unification stays deferred (shared rule already used, only the interface is local). Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV --- ZB.MOM.WW.Health/CLAUDE.md | 7 +- components/health/GAPS.md | 294 +++++++----------- components/health/README.md | 21 +- .../historiangateway/CURRENT-STATE.md | 72 +++++ .../current-state/mxaccessgw/CURRENT-STATE.md | 165 +++------- .../current-state/otopcua/CURRENT-STATE.md | 190 ++++------- .../scadabridge/CURRENT-STATE.md | 227 ++++---------- 7 files changed, 346 insertions(+), 630 deletions(-) create mode 100644 components/health/current-state/historiangateway/CURRENT-STATE.md diff --git a/ZB.MOM.WW.Health/CLAUDE.md b/ZB.MOM.WW.Health/CLAUDE.md index 884f702..83f56f7 100644 --- a/ZB.MOM.WW.Health/CLAUDE.md +++ b/ZB.MOM.WW.Health/CLAUDE.md @@ -2,7 +2,7 @@ Health-check libraries for the **ZB.MOM.WW SCADA family** (OtOpcUa, MxAccessGateway, ScadaBridge). These are **libraries, not a service** — each package is linked directly into the consuming application at build time. There is no central health process or network hop; probes run in-process alongside the application. -**Backlog / open work:** lives in Gitea issues on `dohertj2/scadaproj` (`tea issues list --repo dohertj2/scadaproj`) — currently #11 (re-verify per-app probe coverage vs the spec). The history below is a record, not a status tracker; file new follow-ups as issues. +**Backlog / open work:** lives in Gitea issues on `dohertj2/scadaproj` (`tea issues list --repo dohertj2/scadaproj`) — #11 (re-verify per-app probe coverage vs the spec) was done 2026-11-09; remaining per-app gaps live in the app repos' trackers (see `components/health/GAPS.md`). The history below is a record, not a status tracker; file new follow-ups as issues. The library normalizes the three-tier health endpoint convention (`/health/ready`, `/health/active`, `/healthz`) and provides reusable probe implementations so the three sister projects share a common surface without duplicating probe logic. @@ -27,8 +27,9 @@ The library normalizes the three-tier health endpoint convention (`/health/ready | **OtOpcUa** | yes | yes | yes | | **MxAccessGateway** | yes | — | — | | **ScadaBridge** | yes | yes | yes | +| **HistorianGateway** | yes | — | — | -MxAccessGateway consumes the core package only — it has no Akka cluster and no EF DbContext. OtOpcUa and ScadaBridge consume all three packages. +All four pin `0.3.0` (verified 2026-11-09). MxAccessGateway and HistorianGateway consume the core package only — no Akka cluster, no EF DbContext. `GrpcDependencyHealthCheck` currently has **no** consumer. OtOpcUa and ScadaBridge consume all three packages. --- @@ -101,6 +102,6 @@ orchestrator wiring, and ScadaBridge's distributed health-monitoring pipeline. The shared library is **built and lives in this repo** at [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) (.NET 10; 3 packages — `ZB.MOM.WW.Health`, `ZB.MOM.WW.Health.Akka`, `ZB.MOM.WW.Health.EntityFrameworkCore`; 64 tests (verified 2026-07-18); `dotnet pack` → 3 nupkgs @ 0.1.0). -**Adopted by all four apps** — verified 2026-07-18 (each references the package and wires `MapZbHealth`); the previous "not yet adopted" claim was stale. Per-app probe coverage vs the spec has NOT been re-verified (→ #11) — see [`components/health/GAPS.md`](components/health/GAPS.md). +**Adopted by all four apps** — verified 2026-07-18 (each references the package and wires `MapZbHealth`); the previous "not yet adopted" claim was stale. Per-app probe coverage vs the spec was **re-verified 2026-11-09** (scadaproj#11) against each app's origin default branch — the per-endpoint coverage matrix, SPEC deviations and remaining gaps are in [`components/health/GAPS.md`](components/health/GAPS.md), per-app detail in `components/health/current-state/`. Build/test from `ZB.MOM.WW.Health/`: `dotnet test`. Consumer matrix: MxAccessGateway → core only; OtOpcUa & ScadaBridge → all three packages. diff --git a/components/health/GAPS.md b/components/health/GAPS.md index cc538a0..31150d2 100644 --- a/components/health/GAPS.md +++ b/components/health/GAPS.md @@ -1,211 +1,125 @@ -# Health — gaps & adoption backlog +# Health — gaps vs spec -> **Status lives in Gitea.** Open items from this file are tracked as issues on -> [gitea.dohertylan.com/dohertj2/scadaproj](https://gitea.dohertylan.com/dohertj2/scadaproj/issues) -> (label `migrated`, 2026-11-06). This file is a historical/design record — do not track status -> here; file new work as an issue. +> **Status lives in Gitea.** Open work is tracked as issues on the owning repo's tracker +> (`gitea.dohertylan.com/dohertj2/`); this file is a design record of where each app stands against +> [`spec/SPEC.md`](spec/SPEC.md). Do not add status rows here — file an issue and reference it. -Divergence of each project from [`spec/SPEC.md`](spec/SPEC.md), and the ordered backlog to -reach the shared `ZB.MOM.WW.Health` library. Status legend: ⛔ gap · 🟡 partial · ✅ matches. +**Re-verified 2026-11-09 (scadaproj#11)** against each repo's origin default branch, by reading the +registration and mapping code (not just the `PackageReference`): -> **✅ ADOPTED — verified 2026-07-18.** This file previously carried no adoption banner and the -> root docs claimed the library was "**not yet adopted**". That was stale. Verified directly: -> `ZB.MOM.WW.Health` `0.1.0` is on the Gitea feed, and **all four apps reference it and wire -> `MapZbHealth`** — OtOpcUa (+ `.Akka` + `.EntityFrameworkCore`, uses `AkkaClusterHealthCheck`), -> MxAccessGateway (core only), ScadaBridge (all three packages; uses `AkkaClusterHealthCheck` + -> `IActiveNodeGate`), HistorianGateway (core; uses `GrpcDependencyHealthCheck`). -> -> **Caveat on the evidence:** this was verified by `PackageReference` + registration-call grep, -> which proves the library is wired in — *not* that every per-app probe below is registered or -> that the endpoint tiers match the spec. The divergence tables further down were written -> pre-adoption and have **not** been re-verified against current code (→ #11); treat them as historical -> until someone re-walks them. They also predate HistorianGateway, which is absent from every -> table below. - -> **⚠️ Gaps L1/L2 were closed the WRONG WAY, then reopened and closed properly at `0.3.0` -> (2026-07-24).** Everything below about the active-node check describes the leader-based design and -> is **superseded** — read [`spec/SPEC.md`](spec/SPEC.md) §2.3 for the current rule. -> -> L1 shipped `ActiveNodeHealthCheck` selecting by `ClusterState.Leader` / `RoleLeader`, and L2 backed -> `AkkaActiveNodeGate` the same way. Leadership is **address**-ordered (host, then port) with no -> relationship to time, while cluster-singleton placement is **age**-ordered. The two agree on a -> freshly-formed cluster — so every unit test and happy-path rig passed — and diverge permanently -> after any restart: the restarted node rejoins youngest but keeps its address, so if it holds the -> lower address it becomes leader while the singletons stay on the other node. During a partition -> both sides also compute themselves leader. -> -> Both apps adopted L1/L2 as written, both hit the defect on live rigs, and both **reverted to -> private copies** (ScadaBridge `OldestNodeActiveHealthCheck`, OtOpcUa `ClusterPrimaryHealthCheck`). -> The result was that the package's whole active-node surface had **zero consumers** — avoided, not -> unused. `0.3.0` promoted those two copies into one `ClusterActiveNode` primitive, rebuilt the check -> and the gate on it, and both apps migrated back and deleted their copies. -> -> **The lesson worth keeping:** a shared component can be adopted, green, and still wrong, when the -> only fixture that distinguishes right from wrong is a cluster whose oldest member is not its -> lowest-addressed one. The library now pins exactly that fixture. - -## Divergence vs spec - -### §1 Endpoint tiers - -| Spec tier | OtOpcUa | MxAccessGateway | ScadaBridge | +| App | Audited at | Health pin | Detail | |---|---|---|---| -| `/health/ready` (tag `ready`) | ✅ present | ⛔ absent | ✅ present (name-predicate) | -| `/health/active` (tag `active`) | ✅ present | ⛔ absent | ✅ present (name-predicate) | -| `/healthz` (bare process liveness) | ✅ present | ⛔ absent | ⛔ absent | -| `/health/live` (non-standard) | — | ⛔ present (hardcoded `"Healthy"`, bypasses health-check pipeline) | — | +| OtOpcUa | `origin/master` `c904472a` | 0.3.0 (core + `.Akka` + `.EntityFrameworkCore`) | [current-state/otopcua](current-state/otopcua/CURRENT-STATE.md) | +| MxAccessGateway | `origin/main` `de86f8b8` | 0.3.0 (core) | [current-state/mxaccessgw](current-state/mxaccessgw/CURRENT-STATE.md) | +| ScadaBridge | `origin/main` `c7d868c5` | 0.3.0 (core + `.Akka` + `.EntityFrameworkCore`) | [current-state/scadabridge](current-state/scadabridge/CURRENT-STATE.md) | +| HistorianGateway | `origin/main` `4d6a9ad` | 0.3.0 (core) | [current-state/historiangateway](current-state/historiangateway/CURRENT-STATE.md) | -→ **Gap T1 (P1):** MxAccessGateway has no standard health tiers. The existing `/health/live` - `MapGet` lambda must be replaced by `app.MapZbHealth()` + real probes. -→ **Gap T2:** ScadaBridge lacks `/healthz`. `MapZbHealth()` adds it automatically. -→ **Gap T3:** MxAccessGateway's `/health/live` uses a raw `MapGet` that bypasses the ASP.NET Core - health-check middleware — it does not participate in `IHealthCheckPublisher`, `HealthReport`, or - UI integration. Must be removed. +All four call `MapZbHealth()` with default paths and the canonical writer, so every app serves +`/healthz` (no checks), `/health/ready` (tag `ready`) and `/health/active` (tag `active`), anonymously. +No app tags anything `live`. -### §2 Probe coverage +## Coverage matrix (what each endpoint runs) -| Probe | OtOpcUa | MxAccessGateway | ScadaBridge | +| App / node | `/health/ready` | `/health/active` | Active-node rule | |---|---|---|---| -| Database connectivity | ✅ `DatabaseHealthCheck` (query probe) | ⛔ none | ✅ `DatabaseHealthCheck` (`CanConnectAsync`) | -| Akka cluster membership | ✅ `AkkaClusterHealthCheck` (2-way) | n/a (no Akka) | ✅ `AkkaClusterHealthCheck` (3-way) | -| Active / leader node | ✅ `AdminRoleLeaderHealthCheck` (role-filtered) | n/a | ✅ `ActiveNodeHealthCheck` (role-less) | -| Downstream gRPC dependency | ⛔ none | ⛔ none | ⛔ none | +| OtOpcUa admin-only | `configdb`, `akka` | `configdb`, `akka`, `cluster-primary`, `localdb-replication` | shared `ActiveNodeHealthCheck`, roles `[admin, driver]` | +| OtOpcUa fused admin+driver | `opcua-endpoint`, `configdb`, `akka` | `configdb`, `akka`, `cluster-primary`, `localdb-replication` | same | +| OtOpcUa driver-only | `opcua-endpoint`, `akka` | `opcua-endpoint`, `akka`, `cluster-primary`, `localdb-replication` | same | +| ScadaBridge central | `database`, `akka-cluster`, `required-singletons` | `database`, `active-node` | shared `ActiveNodeHealthCheck`, unscoped | +| ScadaBridge site | `akka-cluster`, `localdb` | `active-node` | `SitePairActiveNodeHealthCheck` → shared `ClusterActiveNode` (site role) | +| MxAccessGateway | `auth-store`, `worker-executable`, `galaxy-sql` | `mxaccess-sessions` | none (singleton, no cluster) | +| HistorianGateway | `historian-connection`, `galaxy-sql`, `store-forward-drain`*, `historian-session-pool`*, `historian-event-session-pool`* | *(empty — always 200)* | none (singleton, no cluster) | -→ **Gap P1 (P1):** MxAccessGateway has zero probes — `AddHealthChecks()` at - `GatewayApplication.cs:61` is dead code. Minimum viable: a `GrpcDependencyHealthCheck` - targeting the x86 worker IPC channel. -→ **Gap P2:** No project probes its downstream gRPC dependency. OtOpcUa should probe the - MxAccessGateway channel; MxAccessGateway should probe the worker IPC. -→ **Gap P3:** Dead `AddHealthChecks()` in MxAccessGateway (`GatewayApplication.cs:61`) should be - removed or replaced — it currently implies health checks are configured when they are not. +\* registered only when the feature is enabled; both session pools default to enabled. -### §3 Akka status-policy divergence +Checks that can only return **Degraded** (so never cause a 503): OtOpcUa `akka` (`OtOpcUaCompat`) and +`localdb-replication`; HistorianGateway `store-forward-drain`; the shared Akka checks during startup. -| Aspect | OtOpcUa | ScadaBridge | -|---|---|---| -| Probe implementation | Scans `State.Members` for self by address | Reads `SelfMember.Status` directly | -| Joining status | Degraded (not in Members as Up) | Healthy | -| Leaving/Exiting status | Degraded | Degraded | -| Other (Removed, Down…) | Degraded | Unhealthy | -| ActorSystem null guard | — (none; `ActorSystem` injected directly) | ✅ Degraded if null | +## Spec requirements vs apps -The two implementations diverge in how they classify `Joining` (ScadaBridge calls it Healthy; -OtOpcUa would see it as Degraded because `SelfMember` with status `Joining` would not appear as -`Up` in the member scan). They also diverge in the Removed/Down classification (ScadaBridge -Unhealthy, OtOpcUa Degraded). +✅ covered · 🟡 partial / deliberate deviation · ⛔ missing · n/a not applicable -The shared `ZB.MOM.WW.Health.Akka.AkkaClusterHealthCheck` ships two presets to preserve both -behaviors rather than forcing one onto the other: -- **Default** — ScadaBridge's three-way policy (`Up`/`Joining`=Healthy, `Leaving`/`Exiting`=Degraded, - else Unhealthy) -- **OtOpcUaCompat** — OtOpcUa's self-Up-among-members scan (found Up=Healthy, not found=Degraded) +| SPEC requirement | OtOpcUa | MxAccessGateway | ScadaBridge | HistorianGateway | +|---|---|---|---|---| +| §1/§5 three tiers via `MapZbHealth`, canonical tags | ✅ | ✅ | ✅ (central + site) | ✅ | +| §1 nothing tagged `live` | ✅ | ✅ (test-asserted) | ✅ | ✅ | +| §2.1 DB probe (`DatabaseHealthCheck` where EF exists) | ✅ `ProbeQuery` | n/a (custom SQLite + Galaxy SQL) | ✅ central; site custom LocalDb | n/a (custom `SqlConnectionHealthCheck`) | +| §2.2 / §7(b) Akka probe, **Default** policy | 🟡 `OtOpcUaCompat` | n/a | ✅ | n/a | +| §2.3 active tier = oldest `Up` member | ✅ | 🟡 tier used for session health | ✅ central; 🟡 site adapter | 🟡 tier empty | +| §2.3 startup-safe Degraded + `activeNode` data | ✅ | n/a | ✅ central; ⛔ site | n/a | +| §2.4 downstream dependency probe | ⛔ (see P2) | 🟡 presence + Galaxy SQL, not worker IPC | n/a | ✅ custom (Historian via pool) | +| §3 canonical JSON writer | ✅ | ✅ | ✅ | ✅ | +| §4 / §7(e) `IActiveNodeGate` | n/a (Traefik routes by probe) | n/a | 🟡 local interface, shared rule | n/a | -→ **Gap A1:** OtOpcUa adopts the `OtOpcUaCompat` preset; ScadaBridge adopts the `Default` preset. - Both preserve existing behavior without forcing convergence on a single policy. -→ **Gap A2:** OtOpcUa's `AkkaClusterHealthCheck` injects `ActorSystem` directly (no null guard). - The shared implementation injects via `AkkaHostedService` for startup safety. +## Closed -### §4 Database probe technique +- **T1/T2/T3, P1, P3, W1** (pre-adoption: mxgw had no tiers or probes and a bespoke `/health/live`; + ScadaBridge lacked `/healthz`; per-app writers) — closed by the 2026-06-01 adoption; confirmed on all + four tips above. No bespoke health endpoint remains anywhere. +- **L1/L2 active-node** — closed properly at Health `0.3.0`. They first shipped leader-based (address- + ordered) selection, which diverges from age-ordered singleton placement after any restart; both Akka + apps hit it live and reverted to private copies until `0.3.0` rebuilt `ActiveNodeHealthCheck` and + `AkkaActiveNodeGate` on `ClusterActiveNode` (oldest `Up` member). Both now use the shared rule; their + private copies are deleted. The library pins the only fixture that tells the two orderings apart (a + two-node cluster whose oldest member is not the lowest-addressed one). +- **D1/D2 DB probe technique** — shared `DatabaseHealthCheck` covers both (OtOpcUa `ProbeQuery`, + ScadaBridge `CanConnectAsync` via self-scoping; the `IDbContextFactory` switch was dropped as + unnecessary). +- **mxaccessgw#206** (closed 2026-11-09, `8c21678c`) — `worker-executable` and bounded `galaxy-sql` + readiness checks, both `ready` only; sessions stay on `active`. No LDAP check (out of scope by decision). -| Aspect | OtOpcUa | ScadaBridge | -|---|---|---| -| Probe method | `db.Deployments.AsNoTracking().Take(1).ToListAsync()` (query) | `_dbContext.Database.CanConnectAsync()` (connection only) | -| Injection style | `IDbContextFactory` (pooled, safe for concurrent probes) | `DbContext` directly (scoped, requires care in background use) | -| Schema verification | ✅ implies schema is applied | ⛔ connection only | +## Remaining gaps -→ **Gap D1:** `ZB.MOM.WW.Health.EntityFrameworkCore.DatabaseHealthCheck` uses - `CanConnectAsync` as the default (ScadaBridge behavior). An optional `ProbeQuery` delegate covers - OtOpcUa's stricter approach. Both apps retain their existing probe semantics; neither is forced - to change unless desired. -→ **Gap D2:** ScadaBridge injects `DbContext` directly; the shared probe should use - `IDbContextFactory` for safe reuse from a background-service health-check context. - ScadaBridge's DI registration will need updating on adoption. +- **A1 — OtOpcUa still on `AkkaClusterStatusPolicy.OtOpcUaCompat`.** Its `akka` check can only report + Degraded, so a node that has left or been downed never fails readiness on cluster grounds. SPEC §2.2 / + §7(b) name `Default` as the convergence target. Needs a lmxopcua issue if convergence is still wanted. +- **ScadaBridge site `active-node` adapter** (`SitePairActiveNodeHealthCheck`) returns Unhealthy while the + ActorSystem is still null (SPEC §2.3 says Degraded during startup) and emits no + `selfAddress`/`activeRole`/`activeNode` `data`, so the Overview dashboard cannot tell from a site standby + which node is active. Same rule as the shared check; only the contract around it differs. Needs a + ScadaBridge issue if the site tier should match central's (registering the shared + `ActiveNodeHealthCheck` with the site role would close both). +- **HistorianGateway session-pool readiness latch — historiangw#87 (open, High/P1).** The + `historian-session-pool` / `historian-event-session-pool` checks go Unhealthy after 3 consecutive factory + failures and reset only on a factory success, which needs an RPC, which a 503-ed pod never receives; + caller cancellation also counts. Unfixed on `4d6a9ad` and on by default. Detail: + [current-state/historiangateway](current-state/historiangateway/CURRENT-STATE.md). +- **SPEC text is stale on §2.4.** It names OtOpcUa and MxGateway as `GrpcDependencyHealthCheck` users; + **no app uses it** (HistorianGateway, once cited as a user, now probes the Historian through its + connection pool). The check stays in the library but has zero consumers; the SPEC/README wording should + be corrected when the spec is next revised. -### §5 Active-node / leader check +## Deferred items — status from code (2026-11-09) -| Aspect | OtOpcUa | ScadaBridge | -|---|---|---| -| Probe type | `AdminRoleLeaderHealthCheck` (role-filtered: `"admin"`) | `ActiveNodeHealthCheck` (role-less; Up + leader) | -| Non-role-bearing node | Healthy immediately | n/a (all central nodes have no role filter) | -| Leader status | Healthy | Healthy | -| Non-leader (standby) | Degraded | Unhealthy | -| `IActiveNodeGate` backing | Not present | `ActiveNodeGate` (separate type, duplicated logic) | +- **Downstream dependency probes (P2).** Re-scoped, not adopted as `GrpcDependencyHealthCheck` anywhere: + - *MxAccessGateway:* worker IPC is named pipes and workers exist only per session, so there is no + channel to probe at idle. mxaccessgw#206 closed the practical part (worker binary present, Galaxy SQL + reachable). A live-worker probe is not planned. + - *HistorianGateway:* covered by its own `historian-connection` and `galaxy-sql` checks. + - *OtOpcUa:* still **no** probe of mxaccessgw or any driver dependency — gateway channels are created + per driver from DB config, and OtOpcUa's availability guarantee (serve last-known-good when a source + is down) argues against failing readiness on a driver dependency. Treat as **by design**; if a + gateway-down signal is wanted, it belongs on a Degraded-only check or the driver-status surface, and + needs a lmxopcua issue. +- **ScadaBridge `IActiveNodeGate` unification (L2 follow-on).** Still deferred, low value: ScadaBridge + registers `ActiveNodeGate` as its own `ZB.MOM.WW.ScadaBridge.InboundAPI.IActiveNodeGate` + (`Program.cs:614`), consumed by `InboundApiEndpointFilter`, `InboundMethodWarmupService`, + `AuditWriteMiddleware` and `FlowDebugRunner`. The behaviour already converged: the gate calls + `ClusterActivityEvaluator.SelfIsOldest` → `ActiveNodeEvaluator.SelfIsOldestUp` → shared + `ClusterActiveNode.SelfIsActive`, the same rule as `AkkaActiveNodeGate` and the `active-node` probe, so + the gate and the probe cannot disagree. Only the interface type is local. Swapping to the shared + interface would touch the InboundAPI project (which would then reference `ZB.MOM.WW.Health`) for no + behaviour change. -→ **Gap L1:** `ZB.MOM.WW.Health.Akka.ActiveNodeHealthCheck` with an optional `RoleFilter` - parameter unifies both behaviors. OtOpcUa passes `RoleFilter = "admin"` (role-filtered); - ScadaBridge uses no role filter. -→ **Gap L2:** ScadaBridge's `ActiveNodeGate` duplicates `ActiveNodeHealthCheck` logic. The shared - `IActiveNodeGate` seam + a backing singleton eliminates the duplication. - -### §6 Response writer - -| | OtOpcUa | MxAccessGateway | ScadaBridge | -|---|---|---|---| -| Writer | Default (plain-text/JSON) | Bespoke `GatewayHealthReply` JSON | `UIResponseWriter.WriteHealthCheckUIResponse` | - -→ **Gap W1:** the shared `ZB.MOM.WW.Health` package ships a canonical JSON response writer - (lifting `HealthChecks.UI.Client` style to the default). All three projects adopt it on - `MapZbHealth()` call — no per-project writer wiring needed. - -### §7 Endpoint authentication - -Both OtOpcUa and ScadaBridge expose health endpoints without authentication (`AllowAnonymous` or -open by default). MxAccessGateway's `/health/live` has no authentication requirement. The spec -canonizes this: health tiers are `AllowAnonymous`; `MapZbHealth()` applies `AllowAnonymous` by -default. - -No gap — consistent across all three. `MapZbHealth()` should document and enforce this default. - -## Adoption backlog (ordered) - -| # | Item | Projects | Priority | Effort | Risk | Notes | -|---|---|---|---|---|---|---| -| 1 | MxAccessGateway: remove dead `/health/live` + `AddHealthChecks()`, add `GrpcDependencyHealthCheck` (worker IPC) + `MapZbHealth()` | MxGateway | P1 | S | Low | Gap T1, T3, P1, P3 — no probes/tiers today; highest delta | -| 2 | OtOpcUa: replace 3 bespoke checks with shared probes (`AkkaClusterHealthCheck` OtOpcUaCompat + `ActiveNodeHealthCheck` role-filtered + `DatabaseHealthCheck` ProbeQuery) | OtOpcUa | P2 | S | Low | Gap A1, D1, L1 | -| 3 | ScadaBridge: replace 3 bespoke checks with shared probes (Default policy + role-less Active + `CanConnectAsync`) + add `/healthz` + unify `ActiveNodeGate` with `IActiveNodeGate` seam | ScadaBridge | P2 | S | Low | Gap T2, A1, D2, L1, L2 | -| 4 | OtOpcUa + MxAccessGateway: add `GrpcDependencyHealthCheck` for downstream gRPC channel | OtOpcUa, MxGateway | P2 | S | Low | Gap P2 — closes the silent-gateway-down scenario | -| 5 | All: adopt canonical response writer (switch from per-project writers to `MapZbHealth` default) | all 3 | P3 | XS | Low | Gap W1 — mechanical; bundled with #1–3 | -| 6 | DB injection style: switch ScadaBridge from injected `DbContext` to `IDbContextFactory` | ScadaBridge | P3 | XS | Low | Gap D2 — background-service safety | - -**Note: adoption items #1–6 are all follow-on tasks.** They are tracked here as the backlog for -after `ZB.MOM.WW.Health` @ 0.1.0 is published. The library build itself (nupkgs, tests) is a -separate task. This is consistent with how `ZB.MOM.WW.Auth` and `ZB.MOM.WW.Theme` are structured: -the library is built first; adoption by the three apps is the next step. - -## Adoption status — 2026-06-01 (DONE) - -`ZB.MOM.WW.Health` 0.1.0 was published to the `dohertj2-gitea` NuGet feed and adopted across all -three apps on branch `feat/adopt-zb-health` in each repo (one branch per repo; commits below). -Plan + design: [`../../docs/plans/2026-06-01-health-library-adoption.md`](../../docs/plans/2026-06-01-health-library-adoption.md). - -| Repo | What shipped | Build / tests | -|---|---|---| -| **MxAccessGateway** | Removed the pipeline-bypassing `/health/live` lambda + dead `AddHealthChecks()`; added a custom `AuthStoreHealthCheck` (readiness probe over the SQLite auth store) tagged `Ready`; `MapZbHealth()` → ready/active/healthz + canonical writer. | Server builds; 568 pass / 3 **pre-existing** macOS failures unrelated to health (OrphanWorkerTerminator ×2, fake-worker timeout ×1). | -| **OtOpcUa** | Swapped all 3 bespoke checks → shared probes (`DatabaseHealthCheck` + `ProbeQuery`, `AkkaClusterHealthCheck` **OtOpcUaCompat**, `ActiveNodeHealthCheck(role:"admin")`); `MapZbHealth()`. | Host builds clean; health tests pass **including a real two-node-cluster integration test**. Independently code-reviewed: APPROVED, behaviour-preserving. | -| **ScadaBridge** | Swapped 3 bespoke checks → shared probes (`DatabaseHealthCheck` scoped fallback, `AkkaClusterHealthCheck` **Default**, role-less `ActiveNodeHealthCheck`); added transient `ActorSystem` DI bridge (Central + Site roots); added `/healthz`; canonical writer. Kept its own `ActiveNodeGate`. | Builds clean (TreatWarningsAsErrors); 212 Host tests pass. | - -### Deferred (verified ill-fitting on adoption — re-scoped from the original backlog) - -- **#4 downstream gRPC dependency probes — DROPPED for now.** (→ #11) Neither repo holds a host-level - `GrpcChannel` to probe: the MxGateway↔worker IPC is **named pipes** (not gRPC), and OtOpcUa's - gateway channel is created **per-driver** from DB config (no DI-level channel). MxGateway readiness - instead probes the SQLite auth store (`AuthStoreHealthCheck`). A real worker/downstream probe needs - a custom non-gRPC check — future work. -- **#3 `IActiveNodeGate` seam unification (ScadaBridge) — DEFERRED.** (→ #11) ScadaBridge's `IActiveNodeGate` - is `…ScadaBridge.InboundAPI.IActiveNodeGate`, wired into inbound-API endpoint gating — a different - interface from the shared `ZB.MOM.WW.Health.IActiveNodeGate`. Unification touches the InboundAPI - path; its existing `ActiveNodeGate` (logic identical to the shared `AkkaActiveNodeGate`) was kept. -- **#6 `IDbContextFactory` switch (ScadaBridge) — DROPPED as unnecessary.** The shared - `DatabaseHealthCheck` self-scopes (creates its own DI scope per probe) when no factory is - registered — that *is* the background-safety fix — and ScadaBridge's context is built with an - injected `IDataProtectionProvider`, which `AddDbContextFactory` does not accommodate cleanly. - -### Accepted behaviour change (one) — flag for ops - -ScadaBridge `/health/active` during the **startup window** (ActorSystem/cluster not yet ready) now -returns **Degraded → HTTP 200** instead of the prior **Unhealthy → HTTP 503**. This is the shared -`ActiveNodeHealthCheck`'s documented startup-safe behaviour and the normalized convergence target. -The steady-state standby case (node Up but not leader) is **unchanged** (Unhealthy → 503). If a -load-balancer (Traefik) keys strictly on a 503 from `/health/active` to fence standby nodes during -startup, that fail-safe is briefly relaxed until the cluster forms. +## Behaviour notes (kept for ops) +- **Extra probes on the active tier.** ScadaBridge central (`database`) and OtOpcUa admin nodes + (`configdb`) put their DB check on `/health/active` deliberately: a DB-dead active node leaves Traefik + rotation. Side effect: during a config-DB outage **both** nodes answer 503 on `/health/active`, so + Traefik has no backend for the routed UI until the DB returns. +- **Startup window.** The shared `ActiveNodeHealthCheck` returns Degraded (200) while the ActorSystem or + cluster is not yet available (accepted 2026-06-01 for ScadaBridge central). The ScadaBridge site + adapter returns 503 in the same window (see Remaining gaps). +- **Singleton apps answer 200 on `/health/active` on every node.** MxAccessGateway (unless all sessions + are faulted) and HistorianGateway (always). The QA pair runs both on vq03 and vq04 as independent + singletons, so this is correct, but `/health/active` cannot be used to pick one of them. diff --git a/components/health/README.md b/components/health/README.md index f076a61..944659c 100644 --- a/components/health/README.md +++ b/components/health/README.md @@ -23,14 +23,12 @@ semantics. ## Status by project -| Project | Endpoints today | Probes today | Response writer | `/healthz` | `IActiveNodeGate` | Adoption status | -|---|---|---|---|---|---|---| -| **OtOpcUa** | `/health/ready`, `/health/active`, `/healthz` | Database (query), AkkaCluster (2-way), AdminRoleLeader (role-filtered) | Default (plain-text/JSON) | ✅ present | — | Not started | -| **MxAccessGateway** | `/health/live` only (raw `MapGet`; hardcoded `"Healthy"`) | **None** (`AddHealthChecks()` called but unused) | Bespoke `GatewayHealthReply` JSON | ⛔ absent | — | Not started | -| **ScadaBridge** | `/health/ready`, `/health/active` | Database (`CanConnectAsync`), AkkaCluster (3-way), ActiveNode (role-less) | `HealthChecks.UI.Client` JSON | ⛔ absent | `ActiveNodeGate` (backs Inbound API 503 gate) | Not started | - -See each project's [`current-state//CURRENT-STATE.md`](current-state/) for the -code-verified detail and its adoption plan. +**Adopted by all four apps at `0.3.0`; probe coverage re-verified 2026-11-09 (scadaproj#11).** Every app +calls `MapZbHealth()` (`/healthz`, `/health/ready`, `/health/active`, canonical JSON). The per-endpoint +coverage matrix, SPEC deviations and remaining gaps are in [`GAPS.md`](GAPS.md); code-verified detail per +app (OtOpcUa, MxAccessGateway, ScadaBridge, HistorianGateway) is in +[`current-state//CURRENT-STATE.md`](current-state/). The "Why" section above and the bullets +below describe the original 2026-06 design rationale. ## Normalized vs. left per-project @@ -81,8 +79,5 @@ all three. ## Component status -**Status: Draft — library built at 0.1.0.** Spec and shared-contract written; current-state docs -verified; GAPS backlog populated. Library implemented and packed at -[`../../ZB.MOM.WW.Health/`](../../ZB.MOM.WW.Health/) (3 packages, 58 tests; -`ZB.MOM.WW.Health`, `ZB.MOM.WW.Health.Akka`, `ZB.MOM.WW.Health.EntityFrameworkCore`). -Adoption by the three apps is the next follow-on tracked in [`GAPS.md`](GAPS.md). +**Status: Adopted — library at 0.3.0** ([`../../ZB.MOM.WW.Health/`](../../ZB.MOM.WW.Health/), 3 packages), +pinned by all four apps; coverage re-verified 2026-11-09. Remaining gaps: [`GAPS.md`](GAPS.md). diff --git a/components/health/current-state/historiangateway/CURRENT-STATE.md b/components/health/current-state/historiangateway/CURRENT-STATE.md new file mode 100644 index 0000000..375da27 --- /dev/null +++ b/components/health/current-state/historiangateway/CURRENT-STATE.md @@ -0,0 +1,72 @@ +# Health — current state: HistorianGateway + +**Verified 2026-11-09 at `historiangw@4d6a9ad` (`origin/main`)** for scadaproj#11. Read-only `git grep` / +`git show` against the origin default branch; all paths relative to the repo root +(`~/Desktop/HistorianGateway`). First current-state doc for this app (it postdates the original +three-project survey). + +- **Package:** `ZB.MOM.WW.Health` **0.3.0**, core only + (`src/ZB.MOM.WW.HistorianGateway.Server/ZB.MOM.WW.HistorianGateway.Server.csproj:23`). +- **Registration:** inline block in `src/ZB.MOM.WW.HistorianGateway.Server/Program.cs:256-329`. +- **Mapping:** `app.MapZbHealth()` with default paths/writer (`Program.cs:417`). +- **Not** `GrpcDependencyHealthCheck`: earlier scadaproj docs said HistorianGateway used it. It does not + on `main` — the gRPC-only Historian client owns its own channel, so the Historian probe goes through the + connection pool instead (comment at `Program.cs:265-268`). + +## Per-endpoint contents + +| Endpoint | Checks run | Can return 503 when | +|---|---|---| +| `/healthz` (live) | none | never | +| `/health/ready` | `historian-connection`, `galaxy-sql`, + `store-forward-drain` (if `StoreForward:Enabled`), + `historian-session-pool` / `historian-event-session-pool` (if each pool is enabled — **both default `true`**) | Historian probe fails/> 3 s; Galaxy SQL fails/> `CommandTimeoutSeconds`; a session-pool factory has ≥ 3 consecutive failures | +| `/health/active` | **none** | never — an empty report is Healthy, so every node answers 200 | + +## Checks + +Files under `src/ZB.MOM.WW.HistorianGateway.Server/Health/`. All tagged `ready`, `failureStatus: null`. + +| Name | Class | Probes | Failure | +|---|---|---|---| +| `historian-connection` | `HistorianConnectionHealthCheck` | `IHistorianConnectionPool.AcquireAsync` + `client.ProbeAsync`, 3 s cap; reports a faulted client back to the pool | Unhealthy | +| `galaxy-sql` | `SqlConnectionHealthCheck` (raw `SqlConnection` + `SELECT 1`; no EF) | Galaxy Repository DB from the top-level `Galaxy` section, bounded by its command timeout | Unhealthy | +| `store-forward-drain` | `StoreForwardDrainHealthCheck` | `IHistorianStoreForward.GetStatusAsync`; backing up = `Storing \|\| ErrorOccurred` | **Degraded** only (never 503) | +| `historian-session-pool` | `SessionPoolHealthCheck("read", …)` | `HistorianSessionPool.Snapshot.ConsecutiveFactoryFailures` | 0 → Healthy; 1–2 → Degraded; ≥ 3 → **Unhealthy** | +| `historian-event-session-pool` | `SessionPoolHealthCheck("event", …)` | same over `HistorianEventSessionPool` | same | + +### Open finding: the session-pool check latches Unhealthy — historiangw#87 (open) + +historiangw#87 (Server-007, High/P1): `_consecutiveFactoryFailures` is reset only by a **successful +factory call**, and the factory only runs when a real RPC needs a new lease (the keepalive sweep never +opens sessions). After a Historian outage of ≥ 3 failed leases, `/health/ready` goes 503, a k8s probe +removes the pod, no RPC arrives, and the check **never recovers** until restart; three clients whose +short deadlines cancel a cold handshake also trip it (the `catch` counts caller cancellation). On the NSSM +wonder boxes it shows as a false Unhealthy on Overview. + +**Still present on `origin/main` `4d6a9ad`:** `HistorianSessionPool.cs:264-273` and +`HistorianEventSessionPool.cs:254-262` increment on any throw and reset only on success; nothing probes +or decays the streak; both pools default to `Enabled = true` +(`Configuration/HistorianSessionPoolOptions.cs:24`, `EventSessionPoolOptions.cs:24`), so the latching +check is on the ready tier in the default configuration. + +## Active-node gating + +None, by design: a single-process sidecar per host with no cluster. No `IActiveNodeGate`, no +`RequireActiveNode()`. `Redundancy` options concern the Historian servers it talks to, not node +election. + +## Other endpoints + +`MapZbMetrics()`, `MapZbGalaxyRepository()`, the dashboard (its `HealthPage` renders the same +ready-tier results), and the four Historian gRPC services. + +## vs SPEC + +| SPEC | Status | Note | +|---|---|---| +| §1 three tiers via `MapZbHealth`, canonical tags | ✅ | `/health/active` is empty (always 200) | +| §1 nothing on `live` | ✅ | | +| §2.1 DB probe | 🟡 custom | `SqlConnectionHealthCheck` (no EF context to wrap) | +| §2.4 downstream dependency probe | ✅ (custom) | Historian via the pool, not `GrpcDependencyHealthCheck` | +| Ready = "can this node serve its dependencies?" | 🟡 | correct while dependencies fail; **does not recover** after a session-pool outage (#87) | +| §3 canonical JSON writer | ✅ | | +| §4 `IActiveNodeGate` | n/a | | diff --git a/components/health/current-state/mxaccessgw/CURRENT-STATE.md b/components/health/current-state/mxaccessgw/CURRENT-STATE.md index c99310f..525e5df 100644 --- a/components/health/current-state/mxaccessgw/CURRENT-STATE.md +++ b/components/health/current-state/mxaccessgw/CURRENT-STATE.md @@ -1,133 +1,68 @@ # Health — current state: MxAccessGateway -Repo: `~/Desktop/MxAccessGateway`. Stack: .NET 10 gateway (x64) + .NET 4.8 worker (x86), gRPC; -solution `src/MxGateway.sln`. -Health code lives in `src/ZB.MOM.WW.MxGateway.Server/GatewayApplication.cs`. All paths relative -to repo root. -Verified 2026-06-01. +**Verified 2026-11-09 at `mxaccessgw@de86f8b8` (`origin/main`)** for scadaproj#11. Read-only `git grep` / +`git show` against the origin default branch; all paths relative to the repo root +(`~/Desktop/MxAccessGateway`). The pre-adoption state (verified 2026-06-01: a hardcoded `/health/live` +`MapGet`, a dead `AddHealthChecks()`, no probes) is gone; see git history of this file. -**Summary: bare liveness only.** MxAccessGateway has a single `/health/live` endpoint that returns -a hardcoded `GatewayHealthReply` JSON object. `AddHealthChecks()` is called at startup but is -entirely unused — no `IHealthCheck` implementations are registered, `MapHealthChecks` is never -called, and there is no readiness or active-node tier. The net48 x86 worker process has no HTTP -server and therefore no health endpoint of any kind. +- **Package:** `ZB.MOM.WW.Health` **0.3.0**, core only (`src/ZB.MOM.WW.MxGateway.Server/ZB.MOM.WW.MxGateway.Server.csproj:20`). + No `.Akka`, no `.EntityFrameworkCore` — the gateway has no Akka cluster and no EF context. +- **Registration:** `builder.Services.AddGatewayHealthChecks()` (`GatewayApplication.cs:118`) → + `src/ZB.MOM.WW.MxGateway.Server/Diagnostics/GatewayHealthCheckExtensions.cs:19-41`. +- **Mapping:** `endpoints.MapZbHealth()` with default paths/writer (`GatewayApplication.cs:346`), + anonymous unconditionally (the Overview dashboard polls `/health/active` without credentials). -## 1. Endpoint wiring +## Per-endpoint contents -`src/ZB.MOM.WW.MxGateway.Server/GatewayApplication.cs`: - -- `:61` — `builder.Services.AddHealthChecks()` is called in the DI registration block. **This call - is dead**: no `.AddCheck()` call follows it, no `MapHealthChecks` is ever called. The - framework registers the health-check infrastructure but nothing is wired through it. -- `:139–145` — `MapGatewayEndpoints` maps a raw `endpoints.MapGet("/health/live", ...)` (not - `MapHealthChecks`). The handler is an inline lambda that returns `Results.Ok(new GatewayHealthReply(...))` - with a hardcoded `Status: "Healthy"`: - - ```csharp - endpoints.MapGet( - "/health/live", - () => Results.Ok(new GatewayHealthReply( - Status: "Healthy", - DefaultBackend: GatewayContractInfo.DefaultBackendName, - WorkerProtocolVersion: GatewayContractInfo.WorkerProtocolVersion))) - .WithName("LiveHealth"); - ``` - -This endpoint always returns HTTP 200 `{"Status":"Healthy",...}` as long as the process is alive. -It carries no authentication requirement (no `[Authorize]` or `.RequireAuthorization()`). - -## 2. Response shape - -`GatewayHealthReply` is a record with three fields: -- `Status` — always `"Healthy"` (hardcoded string, not the ASP.NET Core `HealthStatus` enum) -- `DefaultBackend` — value of `GatewayContractInfo.DefaultBackendName` (the configured backend - name, useful for confirming which gateway instance a probe hit) -- `WorkerProtocolVersion` — value of `GatewayContractInfo.WorkerProtocolVersion` (the gRPC - protocol version the gateway expects from the worker, useful for version-skew detection) - -The response is not `HealthChecks.UI.Client` JSON and is not the standard ASP.NET Core health -response shape. It is a bespoke JSON record. - -## 3. Probes - -None. There is no `IHealthCheck` registered. The `/health/live` response does not reflect: - -- Whether the SQLite auth-store is reachable -- Whether any active MXAccess session is functional -- Whether the x86 worker named-pipe IPC is connected or the worker process is alive -- Whether the gRPC service is actually accepting calls - -The endpoint is purely a process liveness indicator. - -## 4. Tier coverage - -| Tier | Endpoint | Status | +| Endpoint | Checks run | Can return 503 when | |---|---|---| -| Process liveness | `/health/live` (raw `MapGet`) | ✅ present (but non-standard) | -| Readiness | `/health/ready` | ⛔ absent | -| Active node | `/health/active` | ⛔ absent (not Akka-based; not applicable as-is) | -| `healthz` convention | `/healthz` | ⛔ absent | +| `/healthz` (live) | none (`_ => false`) | never (process up) | +| `/health/ready` | `auth-store`, `worker-executable`, `galaxy-sql` | auth SQLite unreachable; worker exe missing; Galaxy SQL unreachable or > 5 s | +| `/health/active` | `mxaccess-sessions` | every open session is `Faulted` (zero sessions = Healthy) | -MxAccessGateway is not an Akka.NET application — it has no cluster, no leader election, and no -active-node concept. The "active" tier in the shared spec translates here to "is the worker process -connected and the gRPC service ready to accept calls?" rather than cluster leadership. +No check is tagged `live` (asserted by `GatewayHealthCheckRegistrationTests`). -## 5. x86 worker +## Checks -`ZB.MOM.WW.MxGateway.Worker` is a .NET 4.8 console application communicating with the gateway -over Windows named-pipe IPC. It has no HTTP server, no health endpoint, and no exposure to any -probe mechanism. Its liveness must be inferred indirectly — either via the gateway process -monitoring it (not currently implemented) or via the `GrpcDependencyHealthCheck` the gateway -could use to probe the IPC channel. +All registered with `AddTypeActivatedCheck`, `failureStatus: null` (the check's own status is used). +Files under `src/ZB.MOM.WW.MxGateway.Server/Diagnostics/`. -## 6. Notable gaps +| Name | Class | Tag | Probes | Failure | +|---|---|---|---|---| +| `auth-store` | `AuthStoreHealthCheck` | ready | Opens the API-key SQLite store via `AuthSqliteConnectionFactory`, runs `SELECT 1` | Unhealthy | +| `worker-executable` | `WorkerExecutableHealthCheck` | ready | `File.Exists(Path.GetFullPath(MxGateway:Worker:ExecutablePath))` — same resolution as `WorkerProcessLauncher.ResolveExecutablePath` | Unhealthy (missing or invalid path) | +| `galaxy-sql` | `GalaxySqlHealthCheck` | ready | `IGalaxyRepository.TestConnectionAsync` bounded by a 5 s `TimeProvider` timeout | Unhealthy (false, throw, or timeout) | +| `mxaccess-sessions` | `SessionHealthCheck` | active | Buckets `ISessionRegistry.Snapshot()` by state; `data` = total/ready/faulted/starting/closing | Healthy if 0 faulted (incl. 0 sessions); Degraded if some faulted; Unhealthy if all usable sessions faulted | -- `AddHealthChecks()` at `:61` is dead code. No `IHealthCheck` is ever registered via this call. -- `/health/live` uses `MapGet` (a raw minimal-API handler) rather than `MapHealthChecks`. It - bypasses the ASP.NET Core health-check middleware entirely, which means it does not participate - in the standard health-check pipeline (no `IHealthCheckPublisher`, no `HealthReport`, no UI - integration). -- The hardcoded `"Healthy"` status means the endpoint cannot reflect real probe results even if - probes were added later — the handler must be replaced, not just supplemented. -- No readiness gating: orchestrators (Kubernetes, Traefik) that rely on `/health/ready` returning - 503 until the process is actually ready will receive 200 (or 404) from MxAccessGateway today. +`worker-executable` and `galaxy-sql` are new on `main` `8c21678c` (mxaccessgw#206, closed 2026-11-09). +They check that the worker **binary exists** and that **Galaxy SQL answers**; neither checks a running +worker or the named-pipe IPC (workers are per-session and only exist while a client holds a session). +No LDAP check was added (#206 comment: deliberately out of scope). ---- +`galaxy-sql` is unconditional: `AddZbGalaxyRepository` is always registered (`GatewayApplication.cs:167`), +so a gateway whose Galaxy SQL is down drops out of readiness even though its MXAccess data path does not +use SQL. That is the #206 design ("must take the gateway out of rotation"), not an accident. -## Adoption plan → `ZB.MOM.WW.Health` +## Active-node gating -**Replace `/health/live` + wire the shared tiers:** +None. The gateway is a singleton per host (no cluster), registers no `IActiveNodeGate`, and uses no +`RequireActiveNode()`. The `active` tier is used for a **session-health** signal instead of an +active/standby signal, so every gateway answers 200 on `/health/active` unless all its sessions are +faulted. -The `AddHealthChecks()` call at `GatewayApplication.cs:61` is already present — it just needs -probes registered against it. The raw `MapGet("/health/live", ...)` handler at `:139–145` must be -removed and replaced with `app.MapZbHealth()` from `ZB.MOM.WW.Health`. +## Other endpoints -Steps: +`MapZbMetrics()` (`:347`, anonymous by default, see `MetricsOptions`), `MapZbGalaxyRepository()` (`:359`), +dashboard routes. No bespoke health endpoint remains. -1. **Remove** the inline `MapGet("/health/live", ...)` lambda (`:139–145`). The `GatewayHealthReply` - record and `DefaultBackend`/`WorkerProtocolVersion` metadata can be surfaced differently (e.g., a - `/info` endpoint or as custom data on the health response). -2. **Register a `GrpcDependencyHealthCheck`** (from `ZB.MOM.WW.Health`) that probes the - named-pipe IPC channel to the x86 worker. Tag `["ready"]`. This replaces the hardcoded - liveness-only response with a real probe that reflects whether the worker is reachable. -3. **Optionally add a `GrpcDependencyHealthCheck`** for any downstream gRPC dependency (e.g., the - Galaxy Repository connection) if the gateway is expected to be healthy only when its upstreams are - reachable. Tag `["ready"]`. -4. **Call `app.MapZbHealth()`** — this maps `/health/ready` (tag `ready`), `/health/active` (tag - `active`; initially empty — no active-node concept in MxGateway), and `/healthz` (bare liveness). - The `/healthz` endpoint replaces the semantic role that `/health/live` served today. -5. **Do not add `ZB.MOM.WW.Health.Akka`** — MxAccessGateway has no Akka dependency. The consumer - matrix in the design specifies MxGateway uses the core package only. +## vs SPEC -**Keep bespoke:** - -- The `WorkerProtocolVersion` / `DefaultBackend` metadata from `GatewayHealthReply` is - MxAccessGateway-specific; keep it as a separate `/info` endpoint or embed it as `Data` on a - custom probe rather than normalizing it into the shared contract. -- The x86 worker itself (net48 console, named-pipe IPC, no HTTP) remains outside the shared health - scheme. The `GrpcDependencyHealthCheck` observes the worker indirectly from the gateway side. -- Per-gateway auth and TLS concerns on who may call health endpoints remain per-project. - -**Adoption is a follow-on task** (tracked in `GAPS.md`), not part of the `ZB.MOM.WW.Health` -library build. MxGateway is the **highest-priority adopter** (P1 gap — no probes/tiers today) -and should be the first app wired up once the nupkg is available. +| SPEC | Status | Note | +|---|---|---| +| §1 three tiers via `MapZbHealth`, canonical tags | ✅ | default paths | +| §1 nothing on `live` | ✅ | test-asserted | +| §2.1 `DatabaseHealthCheck` | n/a | no EF; custom SQLite + Galaxy SQL checks | +| §2.3 active tier = oldest `Up` member | 🟡 deviation | no cluster; active tier carries `mxaccess-sessions` (deliberate, documented in `SessionHealthCheck`) | +| §2.4 `GrpcDependencyHealthCheck` for the worker | ⛔ not used | worker IPC is named pipes, not gRPC; `worker-executable` is a presence proxy (see GAPS P2) | +| §3 canonical JSON writer | ✅ | `MapZbHealth` default | +| §4 `IActiveNodeGate` | n/a | no active/standby | diff --git a/components/health/current-state/otopcua/CURRENT-STATE.md b/components/health/current-state/otopcua/CURRENT-STATE.md index 83f34f0..e2b038d 100644 --- a/components/health/current-state/otopcua/CURRENT-STATE.md +++ b/components/health/current-state/otopcua/CURRENT-STATE.md @@ -1,154 +1,68 @@ # Health — current state: OtOpcUa -Repo: `~/Desktop/OtOpcUa`. Stack: .NET 10, Akka.NET, OPC UA; solution `ZB.MOM.WW.OtOpcUa.slnx`. -Health code lives in `src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/`. All paths relative to repo root. -Verified 2026-06-01. +**Verified 2026-11-09 at `lmxopcua@c904472a` (`origin/master`)** for scadaproj#11. Read-only `git grep` / +`git show` against the origin default branch; all paths relative to the repo root (`~/Desktop/OtOpcUa`). +The pre-adoption state (verified 2026-06-01: three bespoke checks incl. `AdminRoleLeaderHealthCheck`, +default writer) is gone; see git history of this file. -Full three-tier pattern: `/health/ready`, `/health/active`, and `/healthz`. Three probes covering -the database, the Akka cluster, and the admin-role leader. All endpoints are `AllowAnonymous` to -permit Traefik and load-balancer probing without credentials. +- **Packages:** `ZB.MOM.WW.Health`, `.Akka`, `.EntityFrameworkCore` all **0.3.0** + (`Directory.Packages.props:211,219,220`; referenced by `src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj:37-39`; + `.Akka` also by `ControlPlane` and `Runtime`, which use `ClusterActiveNode` for the redundancy election). +- **Registration:** `builder.Services.AddOtOpcUaHealth(hasAdmin, hasDriver)` (`src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs:581`) + → `src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/HealthEndpoints.cs:40-130`. The check set depends on the node's roles. +- **Mapping:** `app.MapOtOpcUaHealth()` (`Program.cs:767`) → `app.MapZbHealth()` with defaults (`HealthEndpoints.cs:135`). -## 1. Endpoint wiring +## Per-endpoint contents (by node role) -`src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/HealthEndpoints.cs`: - -- `:13` — XML comment explicitly names this as "ScadaLink's three-tier pattern: `ready` = boot ok; - `active` = fully serving traffic; `healthz` = bare process liveness." -- `:17` — `AddOtOpcUaHealth(IServiceCollection)` calls `services.AddHealthChecks()` and registers - all three probes (lines 20–22): - - `DatabaseHealthCheck` name `"configdb"`, tags `["ready","active"]` - - `AkkaClusterHealthCheck` name `"akka"`, tags `["ready","active"]` - - `AdminRoleLeaderHealthCheck` name `"admin-leader"`, tags `["active"]` only -- `:28` — `MapOtOpcUaHealth(IEndpointRouteBuilder)` maps three endpoints (lines 33–44): - - `/health/ready` — predicate `c => c.Tags.Contains("ready")`, `.AllowAnonymous()` (lines 33–36) - - `/health/active` — predicate `c => c.Tags.Contains("active")`, `.AllowAnonymous()` (lines 37–40) - - `/healthz` — predicate `_ => false` (no probes run; bare process liveness only), `.AllowAnonymous()` (lines 41–44) - -`Program.cs`: -- `:137` — `builder.Services.AddOtOpcUaHealth()` -- `:159` — `app.MapOtOpcUaHealth()` - -Response writer: default ASP.NET Core plain-text/JSON (no `HealthChecks.UI.Client`). - -## 2. Probes - -### DatabaseHealthCheck -`src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/DatabaseHealthCheck.cs`: - -- `:9` — injects `IDbContextFactory` -- `:25–37` — opens a pooled context via `CreateDbContextAsync`, runs - `db.Deployments.AsNoTracking().Take(1).ToListAsync()`. If the query succeeds → - `HealthCheckResult.Healthy("ConfigDb reachable")` (`:31`). If it throws → - `HealthCheckResult.Unhealthy("ConfigDb unreachable", ex)` (`:35`). No `Degraded` path. - -The probe exercises a real query (not just `CanConnectAsync`) — it confirms the `Deployments` table -is readable, which implies the schema migration has run. This is **stricter** than ScadaBridge's -`CanConnectAsync` but more opaque about the failure reason. - -Tags on registration: `["ready","active"]` — the database must be reachable for both readiness and -active-node determination. - -### AkkaClusterHealthCheck -`src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/AkkaClusterHealthCheck.cs`: - -- `:9` — injects `ActorSystem` directly -- `:27–33` — calls `Cluster.Get(_system)`, scans `cluster.State.Members` for the member whose - `Address == cluster.SelfAddress` and `Status == MemberStatus.Up`: - - Found Up → `HealthCheckResult.Healthy($"Self Up; {cluster.State.Members.Count} member(s)")` (`:32`) - - Not found → `HealthCheckResult.Degraded("Self not yet Up in cluster")` (`:33`) - -No `Unhealthy` path — joining/leaving/removed nodes are all reported as `Degraded`. This differs from -ScadaBridge's more granular three-way policy (see GAPS). - -Tags on registration: `["ready","active"]`. - -### AdminRoleLeaderHealthCheck -`src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/AdminRoleLeaderHealthCheck.cs`: - -- `:14` — injects `IClusterRoleInfo` -- `:27–38` — three-path logic: - - Node does not carry the `"admin"` role → `Healthy("Node does not carry admin role")` (`:30`) — - non-admin nodes are immediately healthy, so this probe never gates a non-admin node. - - Admin role + node is the role leader → `Healthy($"Admin leader ({...})")` (`:36`) - - Admin role + not the leader → `Degraded($"Admin member but not leader (leader=...)")` (`:37`) - -Tags on registration: `["active"]` only — does not participate in `/health/ready`. The intent is -Traefik routing: the active node (admin-role leader) gets sticky admin-UI traffic; standby nodes -are reachable for data-plane OPC UA but report `Degraded` on `/health/active` so the load balancer -does not route control-plane traffic to them. - -Note: no `Unhealthy` path for the role-filter case. If the ActorSystem is not running, `IClusterRoleInfo` -presumably returns safe defaults (no role); this is not separately health-checked. - -## 3. Tag / tier summary - -| Probe | `/health/ready` | `/health/active` | `/healthz` | +| Endpoint | admin-only node | fused admin+driver | driver-only node | |---|---|---|---| -| `DatabaseHealthCheck` | ✅ | ✅ | — | -| `AkkaClusterHealthCheck` | ✅ | ✅ | — | -| `AdminRoleLeaderHealthCheck` | — | ✅ | — | -| (no probes) | — | — | ✅ (bare liveness) | +| `/healthz` | none | none | none | +| `/health/ready` | `configdb`, `akka` | `opcua-endpoint`, `configdb`, `akka` | `opcua-endpoint`, `akka` | +| `/health/active` | `configdb`, `akka`, `cluster-primary`, `localdb-replication` | `configdb`, `akka`, `cluster-primary`, `localdb-replication` | `opcua-endpoint`, `akka`, `cluster-primary`, `localdb-replication` | -`/healthz` runs zero probes — it is a pure process liveness sentinel (process reachable = healthy; -a crashed process = no response). Kubernetes liveness probes, Traefik TCP checks, and uptime -monitors use this tier. +`opcua-endpoint` is deliberately **not** on the fused node's active tier: Traefik routes the AdminUI by +`/health/active`, and a failed SDK start must not take the operator console out of rotation; OPC UA +failover is handled by the Runtime (ServiceLevel 0 + `OpcUaEndpointStepDownActor`, lmxopcua#762) +(`HealthEndpoints.cs:44-65`). -## 4. Downstream dependency coverage +## Checks -No probe for the upstream MxAccessGateway gRPC channel. If the gateway is unreachable, OtOpcUa -reports healthy here (the GalaxyDriver will surface errors in OPC UA diagnostics, but `/health/ready` -and `/health/active` will not reflect it). This is a gap that the shared `GrpcDependencyHealthCheck` -probe in `ZB.MOM.WW.Health` would close. +| Name | Class | Tags | Probes | Failure | +|---|---|---|---|---| +| `opcua-endpoint` (driver nodes) | `OpcUaEndpointHealthCheck` (host, `Health/OpcUaEndpointHealthCheck.cs`) | ready (+ active on driver-only) | `IOpcUaEndpointState.Status`: `Started` healthy | Unhealthy on `NotStarted` / `Stopped` / failed | +| `configdb` (admin nodes) | shared `DatabaseHealthCheck` | ready, active | `ProbeQuery` = `Deployments.AsNoTracking().Take(1)` (implies schema applied) | Unhealthy | +| `akka` | shared `AkkaClusterHealthCheck`, policy **`OtOpcUaCompat`** | ready, active | self `Up` among reachable members | **Degraded only** (never 503) | +| `cluster-primary` | shared `ActiveNodeHealthCheck`, `RolePreference = [admin, driver]` | active | oldest `Up` member of the first preferred role held | Unhealthy on standby and on a node holding neither role; Degraded while the ActorSystem/cluster is not yet available | +| `localdb-replication` | `LocalDbReplicationHealthCheck` (host) | active | `ISyncStatus` connected + oplog backlog < 100 000 | **Degraded only**; Healthy when LocalDb or replication is absent/off | -## 5. Notable design choices +Driver-only nodes register no `configdb` check: they hold no ConfigDb (FetchAndCache from central), and the +type-activated check would throw resolving the factory. -- **AllowAnonymous on all tiers** — see `HealthEndpoints.cs:30–32` comment: "Without it the - `AddOtOpcUaAuth` fallback policy 401s every probe and Traefik marks every backend unhealthy." -- **Query probe, not `CanConnectAsync`** — the `Deployments` query validates that the schema has - been applied. ScadaBridge uses `CanConnectAsync`; neither is wrong but they diverge. -- **`Degraded` semantics** — the Akka check uses `Degraded` (not `Unhealthy`) for a joining/pre-Up - node. ASP.NET Core maps `Degraded` to HTTP 200 by default; Traefik sees 200 and considers the - node ready. If `Unhealthy` (HTTP 503) is required to gate traffic, the `Degraded` path is - insufficient. -- **`IClusterRoleInfo` abstraction** — the admin-leader check depends on `IClusterRoleInfo`, an OtOpcUa - interface, not the raw `Akka.Cluster.Cluster` API. This is a testability-friendly layer absent in - ScadaBridge's direct Akka usage. +## Active-node gating ---- +- **Probe:** `cluster-primary` above — the shared `ClusterActiveNode` rule. Per-cluster meshes (Phase 6) + make "oldest `Up` member" already per-`Cluster`, so exactly one node per `Cluster` answers 200. + `RedundancyStateActor`'s Primary election routes through the same primitive, so the tier and the + OPC UA ServiceLevel 250/240 split cannot disagree. +- **Request gate:** none. OtOpcUa registers no `IActiveNodeGate` and uses no `RequireActiveNode()`; the + AdminUI is pinned to the active node by Traefik routing on `/health/active`. +- **Extra probes on the active tier:** `configdb` (Unhealthy) — a ConfigDb outage makes the admin pair + answer 503 on both nodes, so Traefik has no AdminUI backend until it recovers. `akka` and + `localdb-replication` can only Degrade, so they never change routing. -## Adoption plan → `ZB.MOM.WW.Health` +## Other endpoints -**Replace with shared probes:** +`MapOtOpcUaMetrics()`; mesh gRPC services (`MapOtOpcUaMeshGrpcServices`). No bespoke health endpoint. -- `AkkaClusterHealthCheck` → `ZB.MOM.WW.Health.Akka.AkkaClusterHealthCheck` using the - **`OtOpcUaCompat` preset** (self-Up-among-members scan → Healthy/Degraded). The preset keeps - OtOpcUa's existing two-way policy without forcing ScadaBridge's three-way policy onto it. -- `AdminRoleLeaderHealthCheck` → `ZB.MOM.WW.Health.Akka.ActiveNodeHealthCheck` with - `RoleFilter = "admin"`. The role-filter parameter produces identical behavior: non-admin nodes - immediately healthy, admin leader healthy, admin non-leader degraded. -- `DatabaseHealthCheck` → `ZB.MOM.WW.Health.EntityFrameworkCore.DatabaseHealthCheck` - with a `ProbeQuery` delegate of `db => db.Deployments.AsNoTracking().Take(1).ToListAsync()`. - The delegate preserves the stricter query probe rather than falling back to `CanConnectAsync`. -- Add `GrpcDependencyHealthCheck` targeting the MxAccessGateway channel (closes the downstream - dependency gap noted in §4). Tag `["ready","active"]`. -- Replace `AddOtOpcUaHealth` / `MapOtOpcUaHealth` with - `services.AddHealthChecks().AddCheck<...>()` (one call per probe, per spec §5) + - `app.MapZbHealth()`. The `/healthz` bare-liveness tier is part of `MapZbHealth` by default — - no separate wiring needed. +## vs SPEC -**Keep bespoke:** - -- `IClusterRoleInfo` and its Akka implementation — on adoption this testability seam is given up - for the health-check path. The shared `ActiveNodeHealthCheck` reads cluster role state from the - ActorSystem directly (resolving it lazily via `IServiceProvider`); it does not accept - `IClusterRoleInfo` as an injection point. This is an accepted trade-off: the shared implementation - is simpler and consistent across projects, while `IClusterRoleInfo` remains available elsewhere - in the OtOpcUa codebase where it is used outside health checks. -- The `AllowAnonymous` policy — this is an OtOpcUa auth concern; `MapZbHealth` must document that - callers are responsible for applying `AllowAnonymous` (or the shared helper applies it by default). -- Which probes are registered and their tag assignments — the shared library supplies the check - implementations; the wiring (which names, which tags, which options) remains per-project. - -**Adoption is a follow-on task** (tracked in `GAPS.md`), not part of the `ZB.MOM.WW.Health` -library build. The library build delivers the shared implementations; adoption lands in the -OtOpcUa repo as a separate commit once the nupkg is available. +| SPEC | Status | Note | +|---|---|---| +| §1 three tiers via `MapZbHealth`, canonical tags | ✅ | | +| §1 nothing on `live` | ✅ | | +| §2.1 `DatabaseHealthCheck` | ✅ | `ProbeQuery` variant | +| §2.2 / §7(b) Akka probe with **Default** policy | 🟡 | still `OtOpcUaCompat` (Degraded-only); see GAPS A1 | +| §2.3 / §7(b) `ActiveNodeHealthCheck` | ✅ | role-preference mode | +| §2.4 downstream (mxaccessgw) gRPC probe | ⛔ not used | gateway channels are per-driver, created from DB config; see GAPS P2 | +| §3 canonical JSON writer | ✅ | | +| §4 / §7(e) `IActiveNodeGate` | n/a | no gated endpoints; Traefik routes by the probe | diff --git a/components/health/current-state/scadabridge/CURRENT-STATE.md b/components/health/current-state/scadabridge/CURRENT-STATE.md index dea29f8..c702288 100644 --- a/components/health/current-state/scadabridge/CURRENT-STATE.md +++ b/components/health/current-state/scadabridge/CURRENT-STATE.md @@ -1,186 +1,71 @@ # Health — current state: ScadaBridge -Repo: `~/Desktop/ScadaBridge`. Stack: .NET 10, Akka.NET; solution `ZB.MOM.WW.ScadaBridge.slnx`. -Health code centers on `src/ZB.MOM.WW.ScadaBridge.Host/Health/` (ASP.NET probes) and the -separate `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/` project (domain aggregation pipeline). -All paths relative to repo root. -Verified 2026-06-01. +**Verified 2026-11-09 at `ScadaBridge@c7d868c5` (`origin/main`)** for scadaproj#11. Read-only `git grep` / +`git show` against the origin default branch; all paths relative to the repo root +(`~/Desktop/ScadaBridge`). The pre-adoption state (verified 2026-06-01: name-predicate endpoints, no +`/healthz`, `HealthChecks.UI.Client` writer) is gone; see git history of this file. -Two-tier pattern: `/health/ready` and `/health/active` — no `/healthz`. Three probes (database, -Akka cluster, active-node). ScadaBridge also has a bespoke distributed `HealthMonitoring/` -pipeline that is entirely separate from the ASP.NET health checks and is out of scope for the -shared library. +Out of scope, as before: the `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/` domain-aggregation pipeline +(site → central health reports over Akka) is not an ASP.NET health probe. -## 1. Endpoint wiring +- **Packages:** `ZB.MOM.WW.Health`, `.Akka`, `.EntityFrameworkCore` all **0.3.0** (`Directory.Packages.props:97-99`; + referenced by `src/ZB.MOM.WW.ScadaBridge.Host/ZB.MOM.WW.ScadaBridge.Host.csproj:32-34`; `.Akka` also by `Communication`). +- **Two composition roots**, both calling `app.MapZbHealth()` with defaults: + central (`src/ZB.MOM.WW.ScadaBridge.Host/Program.cs:544-574` register, `:741` map) and + site (`src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs:267-282` register, `Program.cs:861` map, on the + HTTP/1.1 metrics listener, default `:8084`). +- `CentralListenerIsolation.cs:14-15` admits `/healthz`, `/health/ready`, `/health/active` and `/metrics` + on a dedicated metrics port and 404s everything else there. -`src/ZB.MOM.WW.ScadaBridge.Host/Program.cs`: +## Per-endpoint contents -- `:114–117` — `builder.Services.AddHealthChecks()` followed by three `.AddCheck()` calls - (no tags, checked by name at the endpoint level): - - `.AddCheck("database")` - - `.AddCheck("akka-cluster")` - - `.AddCheck("active-node")` -- `:131` — `builder.Services.AddSingleton()` registers the - production `IActiveNodeGate` implementation (Inbound API gating, not a health-check probe). -- `:222–226` — `/health/ready` mapped with `Predicate = check => check.Name != "active-node"` and - `ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse` (from `HealthChecks.UI.Client`). - Excludes the active-node check so a healthy standby node reports ready. -- `:229–233` — `/health/active` mapped with `Predicate = check => check.Name == "active-node"` and - `ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse`. Active-node check only. +| Endpoint | Central node | Site node | +|---|---|---| +| `/healthz` | none | none | +| `/health/ready` | `database`, `akka-cluster`, `required-singletons` | `akka-cluster`, `localdb` | +| `/health/active` | `database`, `active-node` | `active-node` | -No `/healthz` endpoint. Both mapped endpoints use `HealthChecks.UI.Client` JSON (not the default -plain-text writer), which is a divergence from OtOpcUa. +## Checks -## 2. Probes +| Root | Name | Class | Tags | Probes | Failure | +|---|---|---|---|---|---| +| central | `database` | shared `DatabaseHealthCheck` (scoped context, no factory) | ready, active | `CanConnectAsync` | Unhealthy | +| central | `akka-cluster` | shared `AkkaClusterHealthCheck`, policy **`Default`** | ready | `SelfMember.Status` | Up/Joining Healthy; Leaving/Exiting Degraded; else Unhealthy; Degraded during startup | +| central | `required-singletons` | `RequiredSingletonsHealthCheck` (host) | ready | `Identify` (2 s, concurrent) on 5 singleton proxies (`notification-outbox`, `audit-log-ingest`, `site-call-audit`, `audit-log-purge`, `site-audit-reconciliation`) | Unhealthy if any unreachable or no ActorSystem | +| central | `active-node` | shared `ActiveNodeHealthCheck`, unscoped | active | oldest `Up` member of the central cluster | Unhealthy on standby; Degraded while cluster not yet available | +| site | `akka-cluster` | shared `AkkaClusterHealthCheck`, `Default` | ready | as above | as above | +| site | `localdb` | `SiteLocalDbHealthCheck` (host) | ready | `SELECT 1` on the site LocalDb; replication state in `data` only | Unhealthy if unreachable | +| site | `active-node` | `SitePairActiveNodeHealthCheck` (host) | active | `IClusterNodeProvider.SelfIsPrimary` → `ClusterActiveNode.SelfIsActive(cluster, siteRole)` | **Unhealthy** on standby **and while the ActorSystem is null** | -### DatabaseHealthCheck -`src/ZB.MOM.WW.ScadaBridge.Host/Health/DatabaseHealthCheck.cs`: +The `database` check on central's active tier is deliberate (review 01 [High]): a DB-dead active node drops +out of Traefik rotation (`Program.cs:534-537`). -- `:11` — injects `ScadaBridgeDbContext` directly (not a factory) -- `:33–43` — calls `_dbContext.Database.CanConnectAsync(cancellationToken)`: - - Returns `true` → `HealthCheckResult.Healthy("Database connection is available.")` (`:34–35`) - - Returns `false` → `HealthCheckResult.Unhealthy("Database connection failed.")` (`:36`) - - Throws → `HealthCheckResult.Unhealthy("Database connection failed.", ex)` (`:40`) +## Active-node gating -`CanConnectAsync` tests the connection layer only — it does not run any query or verify schema -state. This is less strict than OtOpcUa's `Deployments` query but more transparent about failure -cause (connection vs. schema). No `Degraded` path. +- **Probes:** central `active-node` (shared check) and site `active-node` (thin adapter over the site's + role-scoped `IClusterNodeProvider`). Both route through the shared `ClusterActiveNode` via + `Communication.ClusterState.ActiveNodeEvaluator.SelfIsOldestUp` — the same rule as the S&F delivery + gate and the heartbeat `IsActive` stamp. +- **Request gate:** `ActiveNodeGate` (`Host/Health/ActiveNodeGate.cs`) registered as + **`ZB.MOM.WW.ScadaBridge.InboundAPI.IActiveNodeGate`** (`Program.cs:614`), *not* the shared + `ZB.MOM.WW.Health.IActiveNodeGate`. Consumed by `InboundApiEndpointFilter` (503 on standby), + `InboundMethodWarmupService`, `AuditWriteMiddleware` and `FlowDebugRunner`. Its logic is + `ClusterActivityEvaluator.SelfIsOldest(cluster)` → shared `ClusterActiveNode` — identical in behaviour to + `AkkaActiveNodeGate`; only the interface type is local. Central only; sites register no gate. -### AkkaClusterHealthCheck -`src/ZB.MOM.WW.ScadaBridge.Host/Health/AkkaClusterHealthCheck.cs`: +## Other endpoints -- `:13` — injects `AkkaHostedService` (not `ActorSystem` directly) -- `:33–50` — gets `_akkaService.ActorSystem`, guards on null → `Degraded("ActorSystem not yet - available.")`, then reads `cluster.SelfMember.Status`: - - `Up` or `Joining` → `Healthy($"Akka cluster member status: {status}")` (`:43`) - - `Leaving` or `Exiting` → `Degraded($"Akka cluster member status: {status}")` (`:45`) - - anything else (Removed, Down, WeaklyUp…) → `Unhealthy($"Akka cluster member status: {status}")` (`:47`) +`MapZbMetrics()` on both roots; central gRPC control plane on its own h2c port. -Three-way status policy: Healthy / Degraded / Unhealthy. This is more granular than OtOpcUa's -two-way policy (self-Up-or-not → Healthy/Degraded with no Unhealthy path). +## vs SPEC -### ActiveNodeHealthCheck -`src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeHealthCheck.cs`: - -- `:13` — injects `AkkaHostedService` -- `:29–44` — three-path logic: - - `ActorSystem == null` → `Unhealthy("ActorSystem not yet available.")` (`:31`) - - `SelfMember.Status != Up` → `Unhealthy($"Node not Up (status: ...)")` (`:37`) - - `Up` AND `cluster.State.Leader == self.Address` → `Healthy("Active node (cluster leader).")` (`:41`) - - `Up` but not leader → `Unhealthy("Standby node (not cluster leader).")` (`:43`) - -No `Degraded` path — `ActiveNodeHealthCheck` uses `Unhealthy` for standby and non-Up states, -which causes `/health/active` to return HTTP 503 on a standby. This is the intended behavior for -Traefik active-node routing. - -## 3. Tag / tier summary - -ScadaBridge uses **name-based predicates** at the endpoint level rather than tags on the check -registration. Tags are absent from all three `.AddCheck()` calls. - -| Probe | `/health/ready` | `/health/active` | `/healthz` | -|---|---|---|---| -| `DatabaseHealthCheck` | ✅ | — (excluded by name) | ⛔ absent | -| `AkkaClusterHealthCheck` | ✅ | — (excluded by name) | ⛔ absent | -| `ActiveNodeHealthCheck` | — (excluded by name) | ✅ | ⛔ absent | - -`/healthz` is absent — there is no bare process liveness endpoint. Kubernetes or Traefik liveness -probes must either use `/health/ready` or tolerate its 503-until-ready behavior. - -## 4. IActiveNodeGate and Inbound API gating - -`src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeGate.cs`: - -- `:24` — `ActiveNodeGate` implements `IActiveNodeGate` (from the `InboundAPI` project) -- `:40` — `IsActiveNode` property: returns `true` only when `_akkaService.ActorSystem != null` - AND `cluster.SelfMember.Status == MemberStatus.Up` AND `cluster.State.Leader == self.Address`. - Defaults to `false` safely during startup (`:45–46`). -- `:131` in `Program.cs` — registered as a singleton. The `InboundApiEndpointFilter` consults this - gate on every `/api/*` request and returns HTTP 503 on a standby node. - -`ActiveNodeGate` mirrors the exact same logic as `ActiveNodeHealthCheck` — both check Up + leader. -They are separate types serving two different concerns (the health endpoint and the API gate) but -are not abstracted into a shared service; each reads cluster state independently. - -`IActiveNodeGate` is the generalized seam the `ZB.MOM.WW.Health` core package lifts to the shared -library. - -## 5. HealthMonitoring domain pipeline (out of scope for shared library) - -`src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/` is a separate project implementing a distributed -health aggregation pipeline. It is **not ASP.NET Core health checks** and is **not in scope** for -`ZB.MOM.WW.Health`. - -Key types: -- `SiteHealthCollector` — thread-safe singleton accumulating per-site error counters, connection - metrics, and tag-read metrics. Populated by actors in the DCL layer. -- `HealthReportSender` — a background service on site clusters that serializes `SiteHealthState` - and ships it to the central cluster via Akka remoting at a configurable interval. -- `CentralHealthReportLoop` — central-only background service that generates a synthetic - `SiteHealthReport` for the central cluster itself (siteId `"$central"`) and feeds it into the - central aggregator. -- `CentralHealthAggregator` — a `BackgroundService` on the central cluster tracking the latest - health report per site and detecting offline sites via heartbeat timeout. Exposes - `GetAggregatedHealth()` to the Central UI's `/monitoring/health` endpoint. - -This pipeline is domain-specific (multi-site ScadaBridge topology) and will remain per-project -regardless of shared-library adoption. - -## 6. Notable design choices - -- **Name-based predicates vs. tags** — ScadaBridge uses `check.Name == "active-node"` predicate - logic at the endpoint level. OtOpcUa uses tag membership (`c.Tags.Contains("ready")`). The tag - approach is more composable (a probe can participate in multiple tiers), the name approach is - more explicit. The shared `MapZbHealth` should use tags by default. -- **`HealthChecks.UI.Client` response writer** — ScadaBridge uses the richer JSON response writer - from the `AspNetCore.HealthChecks.UI.Client` package. OtOpcUa uses the default plain-text writer. - The shared library's canonical response writer standardizes this. -- **`ActiveNodeHealthCheck` returns `Unhealthy` for standby** — a standby is not *unhealthy* in the - system sense; it is a deliberate routing discriminator. Using `Unhealthy` here ensures `/health/active` - returns HTTP 503 (Traefik sees the node as down for active traffic). The naming is semantically - imprecise but operationally correct. -- **`IActiveNodeGate` + `ActiveNodeGate` duplication** — the gate and the health check implement the - same logic independently. The shared library's `IActiveNodeGate` seam + `ActiveNodeHealthCheck` - unify them: one backing service, two consumers. - ---- - -## Adoption plan → `ZB.MOM.WW.Health` - -**Replace with shared probes:** - -- `AkkaClusterHealthCheck` → `ZB.MOM.WW.Health.Akka.AkkaClusterHealthCheck` using the **Default - policy** (`Up`/`Joining`=Healthy, `Leaving`/`Exiting`=Degraded, else Unhealthy). ScadaBridge's - existing three-way policy is the Default — no preset selection needed. -- `ActiveNodeHealthCheck` → `ZB.MOM.WW.Health.Akka.ActiveNodeHealthCheck` with no role filter - (role-less default: Up && leader = Healthy, else Unhealthy). The shared implementation also backs - `IActiveNodeGate`, eliminating the duplicated leader-check logic between `ActiveNodeHealthCheck` - and `ActiveNodeGate`. -- `DatabaseHealthCheck` → `ZB.MOM.WW.Health.EntityFrameworkCore.DatabaseHealthCheck` - using the default `CanConnectAsync` probe (ScadaBridge's existing behavior). No `ProbeQuery` - delegate needed. -- Replace the name-based predicates with tag-based predicates by adding tags at registration time: - `"database"` and `"akka-cluster"` → `["ready"]`; `"active-node"` → `["active"]`. Then call - `app.MapZbHealth()` instead of the two manual `MapHealthChecks` calls. -- **Add `/healthz`** — `MapZbHealth()` maps the bare liveness tier automatically. ScadaBridge - currently lacks this endpoint. -- Switch `ResponseWriter` from `UIResponseWriter.WriteHealthCheckUIResponse` to the shared - canonical writer (a convergence item — `HealthChecks.UI.Client` style lifted to the default in - `ZB.MOM.WW.Health`). - -**Keep bespoke:** - -- `HealthMonitoring/` domain pipeline (`SiteHealthCollector`, `CentralHealthAggregator`, etc.) — - entirely per-project, no shared-library equivalent. -- `IActiveNodeGate` moves from the `InboundAPI` project to `ZB.MOM.WW.Health` (core package) on - adoption. `InboundApiEndpointFilter` references the shared interface; `AkkaActiveNodeGate` - (from `ZB.MOM.WW.Health.Akka`) becomes the singleton implementation registered in DI. The - interface definition is no longer owned by the `InboundAPI` project. -- The Central UI's `/monitoring/health` endpoint — powered by `CentralHealthAggregator`, not by - ASP.NET health checks. -- The comment at `Program.cs:217–221` explains the readiness design decision (standby nodes are - ready; leadership is a separate concern). This intent is preserved by the tag-based approach. - -**Adoption is a follow-on task** (tracked in `GAPS.md`), not part of the `ZB.MOM.WW.Health` -library build. The library build delivers the shared implementations; adoption lands in the -ScadaBridge repo as a separate commit once the nupkg is available. +| SPEC | Status | Note | +|---|---|---| +| §1 three tiers via `MapZbHealth`, canonical tags | ✅ | both roots | +| §1 nothing on `live` | ✅ | | +| §2.1 `DatabaseHealthCheck` | ✅ central | site uses a custom LocalDb check (no EF on sites) | +| §2.2 / §7(b) Akka probe, Default policy | ✅ | | +| §2.3 `ActiveNodeHealthCheck` | ✅ central; 🟡 site | site adapter is Unhealthy (not Degraded) during startup and carries no `selfAddress`/`activeNode` `data` | +| §2.4 downstream probe | n/a | DCL connection health is reported through HealthMonitoring, not readiness | +| §3 canonical JSON writer | ✅ | | +| §4 / §7(e) `IActiveNodeGate` | 🟡 | same rule, local interface type (GAPS L2 deferral) |