docs(health): re-verify per-app probe coverage against the SPEC; refresh GAPS and current-state (scadaproj#11)
Audited the registration and mapping code of each app at its origin default branch: OtOpcUa c904472a, MxAccessGateway de86f8b8, ScadaBridge c7d868c5, HistorianGateway 4d6a9ad (all on Health 0.3.0, all MapZbHealth with default paths: /healthz runs nothing, /health/ready and /health/active by tag). GAPS.md now carries the verified per-endpoint coverage matrix and a SPEC-by-app table; each current-state doc lists exactly what each endpoint runs, and HistorianGateway gets its first current-state doc. Recorded: mxaccessgw#206 (worker-executable + galaxy-sql readiness checks, on /health/ready) is merged; historiangw#87 (session-pool check latches 503 until restart) is still open at 4d6a9ad. Deferred P2 downstream probes are now per-app; ScadaBridge IActiveNodeGate unification stays deferred (shared rule already used, only the interface is local). Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
@@ -2,7 +2,7 @@
|
||||
|
||||
Health-check libraries for the **ZB.MOM.WW SCADA family** (OtOpcUa, MxAccessGateway, ScadaBridge). These are **libraries, not a service** — each package is linked directly into the consuming application at build time. There is no central health process or network hop; probes run in-process alongside the application.
|
||||
|
||||
**Backlog / open work:** lives in Gitea issues on `dohertj2/scadaproj` (`tea issues list --repo dohertj2/scadaproj`) — currently #11 (re-verify per-app probe coverage vs the spec). The history below is a record, not a status tracker; file new follow-ups as issues.
|
||||
**Backlog / open work:** lives in Gitea issues on `dohertj2/scadaproj` (`tea issues list --repo dohertj2/scadaproj`) — #11 (re-verify per-app probe coverage vs the spec) was done 2026-11-09; remaining per-app gaps live in the app repos' trackers (see `components/health/GAPS.md`). The history below is a record, not a status tracker; file new follow-ups as issues.
|
||||
|
||||
The library normalizes the three-tier health endpoint convention (`/health/ready`, `/health/active`, `/healthz`) and provides reusable probe implementations so the three sister projects share a common surface without duplicating probe logic.
|
||||
|
||||
@@ -27,8 +27,9 @@ The library normalizes the three-tier health endpoint convention (`/health/ready
|
||||
| **OtOpcUa** | yes | yes | yes |
|
||||
| **MxAccessGateway** | yes | — | — |
|
||||
| **ScadaBridge** | yes | yes | yes |
|
||||
| **HistorianGateway** | yes | — | — |
|
||||
|
||||
MxAccessGateway consumes the core package only — it has no Akka cluster and no EF DbContext. OtOpcUa and ScadaBridge consume all three packages.
|
||||
All four pin `0.3.0` (verified 2026-11-09). MxAccessGateway and HistorianGateway consume the core package only — no Akka cluster, no EF DbContext. `GrpcDependencyHealthCheck` currently has **no** consumer. OtOpcUa and ScadaBridge consume all three packages.
|
||||
|
||||
---
|
||||
|
||||
@@ -101,6 +102,6 @@ orchestrator wiring, and ScadaBridge's distributed health-monitoring pipeline.
|
||||
The shared library is **built and lives in this repo** at [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/)
|
||||
(.NET 10; 3 packages — `ZB.MOM.WW.Health`, `ZB.MOM.WW.Health.Akka`, `ZB.MOM.WW.Health.EntityFrameworkCore`;
|
||||
64 tests (verified 2026-07-18); `dotnet pack` → 3 nupkgs @ 0.1.0).
|
||||
**Adopted by all four apps** — verified 2026-07-18 (each references the package and wires `MapZbHealth`); the previous "not yet adopted" claim was stale. Per-app probe coverage vs the spec has NOT been re-verified (→ #11) — see [`components/health/GAPS.md`](components/health/GAPS.md).
|
||||
**Adopted by all four apps** — verified 2026-07-18 (each references the package and wires `MapZbHealth`); the previous "not yet adopted" claim was stale. Per-app probe coverage vs the spec was **re-verified 2026-11-09** (scadaproj#11) against each app's origin default branch — the per-endpoint coverage matrix, SPEC deviations and remaining gaps are in [`components/health/GAPS.md`](components/health/GAPS.md), per-app detail in `components/health/current-state/`.
|
||||
Build/test from `ZB.MOM.WW.Health/`: `dotnet test`. Consumer matrix: MxAccessGateway → core only;
|
||||
OtOpcUa & ScadaBridge → all three packages.
|
||||
|
||||
+104
-190
@@ -1,211 +1,125 @@
|
||||
# Health — gaps & adoption backlog
|
||||
# Health — gaps vs spec
|
||||
|
||||
> **Status lives in Gitea.** Open items from this file are tracked as issues on
|
||||
> [gitea.dohertylan.com/dohertj2/scadaproj](https://gitea.dohertylan.com/dohertj2/scadaproj/issues)
|
||||
> (label `migrated`, 2026-11-06). This file is a historical/design record — do not track status
|
||||
> here; file new work as an issue.
|
||||
> **Status lives in Gitea.** Open work is tracked as issues on the owning repo's tracker
|
||||
> (`gitea.dohertylan.com/dohertj2/<repo>`); this file is a design record of where each app stands against
|
||||
> [`spec/SPEC.md`](spec/SPEC.md). Do not add status rows here — file an issue and reference it.
|
||||
|
||||
Divergence of each project from [`spec/SPEC.md`](spec/SPEC.md), and the ordered backlog to
|
||||
reach the shared `ZB.MOM.WW.Health` library. Status legend: ⛔ gap · 🟡 partial · ✅ matches.
|
||||
**Re-verified 2026-11-09 (scadaproj#11)** against each repo's origin default branch, by reading the
|
||||
registration and mapping code (not just the `PackageReference`):
|
||||
|
||||
> **✅ ADOPTED — verified 2026-07-18.** This file previously carried no adoption banner and the
|
||||
> root docs claimed the library was "**not yet adopted**". That was stale. Verified directly:
|
||||
> `ZB.MOM.WW.Health` `0.1.0` is on the Gitea feed, and **all four apps reference it and wire
|
||||
> `MapZbHealth`** — OtOpcUa (+ `.Akka` + `.EntityFrameworkCore`, uses `AkkaClusterHealthCheck`),
|
||||
> MxAccessGateway (core only), ScadaBridge (all three packages; uses `AkkaClusterHealthCheck` +
|
||||
> `IActiveNodeGate`), HistorianGateway (core; uses `GrpcDependencyHealthCheck`).
|
||||
>
|
||||
> **Caveat on the evidence:** this was verified by `PackageReference` + registration-call grep,
|
||||
> which proves the library is wired in — *not* that every per-app probe below is registered or
|
||||
> that the endpoint tiers match the spec. The divergence tables further down were written
|
||||
> pre-adoption and have **not** been re-verified against current code (→ #11); treat them as historical
|
||||
> until someone re-walks them. They also predate HistorianGateway, which is absent from every
|
||||
> table below.
|
||||
|
||||
> **⚠️ Gaps L1/L2 were closed the WRONG WAY, then reopened and closed properly at `0.3.0`
|
||||
> (2026-07-24).** Everything below about the active-node check describes the leader-based design and
|
||||
> is **superseded** — read [`spec/SPEC.md`](spec/SPEC.md) §2.3 for the current rule.
|
||||
>
|
||||
> L1 shipped `ActiveNodeHealthCheck` selecting by `ClusterState.Leader` / `RoleLeader`, and L2 backed
|
||||
> `AkkaActiveNodeGate` the same way. Leadership is **address**-ordered (host, then port) with no
|
||||
> relationship to time, while cluster-singleton placement is **age**-ordered. The two agree on a
|
||||
> freshly-formed cluster — so every unit test and happy-path rig passed — and diverge permanently
|
||||
> after any restart: the restarted node rejoins youngest but keeps its address, so if it holds the
|
||||
> lower address it becomes leader while the singletons stay on the other node. During a partition
|
||||
> both sides also compute themselves leader.
|
||||
>
|
||||
> Both apps adopted L1/L2 as written, both hit the defect on live rigs, and both **reverted to
|
||||
> private copies** (ScadaBridge `OldestNodeActiveHealthCheck`, OtOpcUa `ClusterPrimaryHealthCheck`).
|
||||
> The result was that the package's whole active-node surface had **zero consumers** — avoided, not
|
||||
> unused. `0.3.0` promoted those two copies into one `ClusterActiveNode` primitive, rebuilt the check
|
||||
> and the gate on it, and both apps migrated back and deleted their copies.
|
||||
>
|
||||
> **The lesson worth keeping:** a shared component can be adopted, green, and still wrong, when the
|
||||
> only fixture that distinguishes right from wrong is a cluster whose oldest member is not its
|
||||
> lowest-addressed one. The library now pins exactly that fixture.
|
||||
|
||||
## Divergence vs spec
|
||||
|
||||
### §1 Endpoint tiers
|
||||
|
||||
| Spec tier | OtOpcUa | MxAccessGateway | ScadaBridge |
|
||||
| App | Audited at | Health pin | Detail |
|
||||
|---|---|---|---|
|
||||
| `/health/ready` (tag `ready`) | ✅ present | ⛔ absent | ✅ present (name-predicate) |
|
||||
| `/health/active` (tag `active`) | ✅ present | ⛔ absent | ✅ present (name-predicate) |
|
||||
| `/healthz` (bare process liveness) | ✅ present | ⛔ absent | ⛔ absent |
|
||||
| `/health/live` (non-standard) | — | ⛔ present (hardcoded `"Healthy"`, bypasses health-check pipeline) | — |
|
||||
| OtOpcUa | `origin/master` `c904472a` | 0.3.0 (core + `.Akka` + `.EntityFrameworkCore`) | [current-state/otopcua](current-state/otopcua/CURRENT-STATE.md) |
|
||||
| MxAccessGateway | `origin/main` `de86f8b8` | 0.3.0 (core) | [current-state/mxaccessgw](current-state/mxaccessgw/CURRENT-STATE.md) |
|
||||
| ScadaBridge | `origin/main` `c7d868c5` | 0.3.0 (core + `.Akka` + `.EntityFrameworkCore`) | [current-state/scadabridge](current-state/scadabridge/CURRENT-STATE.md) |
|
||||
| HistorianGateway | `origin/main` `4d6a9ad` | 0.3.0 (core) | [current-state/historiangateway](current-state/historiangateway/CURRENT-STATE.md) |
|
||||
|
||||
→ **Gap T1 (P1):** MxAccessGateway has no standard health tiers. The existing `/health/live`
|
||||
`MapGet` lambda must be replaced by `app.MapZbHealth()` + real probes.
|
||||
→ **Gap T2:** ScadaBridge lacks `/healthz`. `MapZbHealth()` adds it automatically.
|
||||
→ **Gap T3:** MxAccessGateway's `/health/live` uses a raw `MapGet` that bypasses the ASP.NET Core
|
||||
health-check middleware — it does not participate in `IHealthCheckPublisher`, `HealthReport`, or
|
||||
UI integration. Must be removed.
|
||||
All four call `MapZbHealth()` with default paths and the canonical writer, so every app serves
|
||||
`/healthz` (no checks), `/health/ready` (tag `ready`) and `/health/active` (tag `active`), anonymously.
|
||||
No app tags anything `live`.
|
||||
|
||||
### §2 Probe coverage
|
||||
## Coverage matrix (what each endpoint runs)
|
||||
|
||||
| Probe | OtOpcUa | MxAccessGateway | ScadaBridge |
|
||||
| App / node | `/health/ready` | `/health/active` | Active-node rule |
|
||||
|---|---|---|---|
|
||||
| Database connectivity | ✅ `DatabaseHealthCheck` (query probe) | ⛔ none | ✅ `DatabaseHealthCheck` (`CanConnectAsync`) |
|
||||
| Akka cluster membership | ✅ `AkkaClusterHealthCheck` (2-way) | n/a (no Akka) | ✅ `AkkaClusterHealthCheck` (3-way) |
|
||||
| Active / leader node | ✅ `AdminRoleLeaderHealthCheck` (role-filtered) | n/a | ✅ `ActiveNodeHealthCheck` (role-less) |
|
||||
| Downstream gRPC dependency | ⛔ none | ⛔ none | ⛔ none |
|
||||
| OtOpcUa admin-only | `configdb`, `akka` | `configdb`, `akka`, `cluster-primary`, `localdb-replication` | shared `ActiveNodeHealthCheck`, roles `[admin, driver]` |
|
||||
| OtOpcUa fused admin+driver | `opcua-endpoint`, `configdb`, `akka` | `configdb`, `akka`, `cluster-primary`, `localdb-replication` | same |
|
||||
| OtOpcUa driver-only | `opcua-endpoint`, `akka` | `opcua-endpoint`, `akka`, `cluster-primary`, `localdb-replication` | same |
|
||||
| ScadaBridge central | `database`, `akka-cluster`, `required-singletons` | `database`, `active-node` | shared `ActiveNodeHealthCheck`, unscoped |
|
||||
| ScadaBridge site | `akka-cluster`, `localdb` | `active-node` | `SitePairActiveNodeHealthCheck` → shared `ClusterActiveNode` (site role) |
|
||||
| MxAccessGateway | `auth-store`, `worker-executable`, `galaxy-sql` | `mxaccess-sessions` | none (singleton, no cluster) |
|
||||
| HistorianGateway | `historian-connection`, `galaxy-sql`, `store-forward-drain`*, `historian-session-pool`*, `historian-event-session-pool`* | *(empty — always 200)* | none (singleton, no cluster) |
|
||||
|
||||
→ **Gap P1 (P1):** MxAccessGateway has zero probes — `AddHealthChecks()` at
|
||||
`GatewayApplication.cs:61` is dead code. Minimum viable: a `GrpcDependencyHealthCheck`
|
||||
targeting the x86 worker IPC channel.
|
||||
→ **Gap P2:** No project probes its downstream gRPC dependency. OtOpcUa should probe the
|
||||
MxAccessGateway channel; MxAccessGateway should probe the worker IPC.
|
||||
→ **Gap P3:** Dead `AddHealthChecks()` in MxAccessGateway (`GatewayApplication.cs:61`) should be
|
||||
removed or replaced — it currently implies health checks are configured when they are not.
|
||||
\* registered only when the feature is enabled; both session pools default to enabled.
|
||||
|
||||
### §3 Akka status-policy divergence
|
||||
Checks that can only return **Degraded** (so never cause a 503): OtOpcUa `akka` (`OtOpcUaCompat`) and
|
||||
`localdb-replication`; HistorianGateway `store-forward-drain`; the shared Akka checks during startup.
|
||||
|
||||
| Aspect | OtOpcUa | ScadaBridge |
|
||||
|---|---|---|
|
||||
| Probe implementation | Scans `State.Members` for self by address | Reads `SelfMember.Status` directly |
|
||||
| Joining status | Degraded (not in Members as Up) | Healthy |
|
||||
| Leaving/Exiting status | Degraded | Degraded |
|
||||
| Other (Removed, Down…) | Degraded | Unhealthy |
|
||||
| ActorSystem null guard | — (none; `ActorSystem` injected directly) | ✅ Degraded if null |
|
||||
## Spec requirements vs apps
|
||||
|
||||
The two implementations diverge in how they classify `Joining` (ScadaBridge calls it Healthy;
|
||||
OtOpcUa would see it as Degraded because `SelfMember` with status `Joining` would not appear as
|
||||
`Up` in the member scan). They also diverge in the Removed/Down classification (ScadaBridge
|
||||
Unhealthy, OtOpcUa Degraded).
|
||||
✅ covered · 🟡 partial / deliberate deviation · ⛔ missing · n/a not applicable
|
||||
|
||||
The shared `ZB.MOM.WW.Health.Akka.AkkaClusterHealthCheck` ships two presets to preserve both
|
||||
behaviors rather than forcing one onto the other:
|
||||
- **Default** — ScadaBridge's three-way policy (`Up`/`Joining`=Healthy, `Leaving`/`Exiting`=Degraded,
|
||||
else Unhealthy)
|
||||
- **OtOpcUaCompat** — OtOpcUa's self-Up-among-members scan (found Up=Healthy, not found=Degraded)
|
||||
| SPEC requirement | OtOpcUa | MxAccessGateway | ScadaBridge | HistorianGateway |
|
||||
|---|---|---|---|---|
|
||||
| §1/§5 three tiers via `MapZbHealth`, canonical tags | ✅ | ✅ | ✅ (central + site) | ✅ |
|
||||
| §1 nothing tagged `live` | ✅ | ✅ (test-asserted) | ✅ | ✅ |
|
||||
| §2.1 DB probe (`DatabaseHealthCheck<T>` where EF exists) | ✅ `ProbeQuery` | n/a (custom SQLite + Galaxy SQL) | ✅ central; site custom LocalDb | n/a (custom `SqlConnectionHealthCheck`) |
|
||||
| §2.2 / §7(b) Akka probe, **Default** policy | 🟡 `OtOpcUaCompat` | n/a | ✅ | n/a |
|
||||
| §2.3 active tier = oldest `Up` member | ✅ | 🟡 tier used for session health | ✅ central; 🟡 site adapter | 🟡 tier empty |
|
||||
| §2.3 startup-safe Degraded + `activeNode` data | ✅ | n/a | ✅ central; ⛔ site | n/a |
|
||||
| §2.4 downstream dependency probe | ⛔ (see P2) | 🟡 presence + Galaxy SQL, not worker IPC | n/a | ✅ custom (Historian via pool) |
|
||||
| §3 canonical JSON writer | ✅ | ✅ | ✅ | ✅ |
|
||||
| §4 / §7(e) `IActiveNodeGate` | n/a (Traefik routes by probe) | n/a | 🟡 local interface, shared rule | n/a |
|
||||
|
||||
→ **Gap A1:** OtOpcUa adopts the `OtOpcUaCompat` preset; ScadaBridge adopts the `Default` preset.
|
||||
Both preserve existing behavior without forcing convergence on a single policy.
|
||||
→ **Gap A2:** OtOpcUa's `AkkaClusterHealthCheck` injects `ActorSystem` directly (no null guard).
|
||||
The shared implementation injects via `AkkaHostedService` for startup safety.
|
||||
## Closed
|
||||
|
||||
### §4 Database probe technique
|
||||
- **T1/T2/T3, P1, P3, W1** (pre-adoption: mxgw had no tiers or probes and a bespoke `/health/live`;
|
||||
ScadaBridge lacked `/healthz`; per-app writers) — closed by the 2026-06-01 adoption; confirmed on all
|
||||
four tips above. No bespoke health endpoint remains anywhere.
|
||||
- **L1/L2 active-node** — closed properly at Health `0.3.0`. They first shipped leader-based (address-
|
||||
ordered) selection, which diverges from age-ordered singleton placement after any restart; both Akka
|
||||
apps hit it live and reverted to private copies until `0.3.0` rebuilt `ActiveNodeHealthCheck` and
|
||||
`AkkaActiveNodeGate` on `ClusterActiveNode` (oldest `Up` member). Both now use the shared rule; their
|
||||
private copies are deleted. The library pins the only fixture that tells the two orderings apart (a
|
||||
two-node cluster whose oldest member is not the lowest-addressed one).
|
||||
- **D1/D2 DB probe technique** — shared `DatabaseHealthCheck<T>` covers both (OtOpcUa `ProbeQuery`,
|
||||
ScadaBridge `CanConnectAsync` via self-scoping; the `IDbContextFactory` switch was dropped as
|
||||
unnecessary).
|
||||
- **mxaccessgw#206** (closed 2026-11-09, `8c21678c`) — `worker-executable` and bounded `galaxy-sql`
|
||||
readiness checks, both `ready` only; sessions stay on `active`. No LDAP check (out of scope by decision).
|
||||
|
||||
| Aspect | OtOpcUa | ScadaBridge |
|
||||
|---|---|---|
|
||||
| Probe method | `db.Deployments.AsNoTracking().Take(1).ToListAsync()` (query) | `_dbContext.Database.CanConnectAsync()` (connection only) |
|
||||
| Injection style | `IDbContextFactory<T>` (pooled, safe for concurrent probes) | `DbContext` directly (scoped, requires care in background use) |
|
||||
| Schema verification | ✅ implies schema is applied | ⛔ connection only |
|
||||
## Remaining gaps
|
||||
|
||||
→ **Gap D1:** `ZB.MOM.WW.Health.EntityFrameworkCore.DatabaseHealthCheck<TContext>` uses
|
||||
`CanConnectAsync` as the default (ScadaBridge behavior). An optional `ProbeQuery` delegate covers
|
||||
OtOpcUa's stricter approach. Both apps retain their existing probe semantics; neither is forced
|
||||
to change unless desired.
|
||||
→ **Gap D2:** ScadaBridge injects `DbContext` directly; the shared probe should use
|
||||
`IDbContextFactory<TContext>` for safe reuse from a background-service health-check context.
|
||||
ScadaBridge's DI registration will need updating on adoption.
|
||||
- **A1 — OtOpcUa still on `AkkaClusterStatusPolicy.OtOpcUaCompat`.** Its `akka` check can only report
|
||||
Degraded, so a node that has left or been downed never fails readiness on cluster grounds. SPEC §2.2 /
|
||||
§7(b) name `Default` as the convergence target. Needs a lmxopcua issue if convergence is still wanted.
|
||||
- **ScadaBridge site `active-node` adapter** (`SitePairActiveNodeHealthCheck`) returns Unhealthy while the
|
||||
ActorSystem is still null (SPEC §2.3 says Degraded during startup) and emits no
|
||||
`selfAddress`/`activeRole`/`activeNode` `data`, so the Overview dashboard cannot tell from a site standby
|
||||
which node is active. Same rule as the shared check; only the contract around it differs. Needs a
|
||||
ScadaBridge issue if the site tier should match central's (registering the shared
|
||||
`ActiveNodeHealthCheck` with the site role would close both).
|
||||
- **HistorianGateway session-pool readiness latch — historiangw#87 (open, High/P1).** The
|
||||
`historian-session-pool` / `historian-event-session-pool` checks go Unhealthy after 3 consecutive factory
|
||||
failures and reset only on a factory success, which needs an RPC, which a 503-ed pod never receives;
|
||||
caller cancellation also counts. Unfixed on `4d6a9ad` and on by default. Detail:
|
||||
[current-state/historiangateway](current-state/historiangateway/CURRENT-STATE.md).
|
||||
- **SPEC text is stale on §2.4.** It names OtOpcUa and MxGateway as `GrpcDependencyHealthCheck` users;
|
||||
**no app uses it** (HistorianGateway, once cited as a user, now probes the Historian through its
|
||||
connection pool). The check stays in the library but has zero consumers; the SPEC/README wording should
|
||||
be corrected when the spec is next revised.
|
||||
|
||||
### §5 Active-node / leader check
|
||||
## Deferred items — status from code (2026-11-09)
|
||||
|
||||
| Aspect | OtOpcUa | ScadaBridge |
|
||||
|---|---|---|
|
||||
| Probe type | `AdminRoleLeaderHealthCheck` (role-filtered: `"admin"`) | `ActiveNodeHealthCheck` (role-less; Up + leader) |
|
||||
| Non-role-bearing node | Healthy immediately | n/a (all central nodes have no role filter) |
|
||||
| Leader status | Healthy | Healthy |
|
||||
| Non-leader (standby) | Degraded | Unhealthy |
|
||||
| `IActiveNodeGate` backing | Not present | `ActiveNodeGate` (separate type, duplicated logic) |
|
||||
- **Downstream dependency probes (P2).** Re-scoped, not adopted as `GrpcDependencyHealthCheck` anywhere:
|
||||
- *MxAccessGateway:* worker IPC is named pipes and workers exist only per session, so there is no
|
||||
channel to probe at idle. mxaccessgw#206 closed the practical part (worker binary present, Galaxy SQL
|
||||
reachable). A live-worker probe is not planned.
|
||||
- *HistorianGateway:* covered by its own `historian-connection` and `galaxy-sql` checks.
|
||||
- *OtOpcUa:* still **no** probe of mxaccessgw or any driver dependency — gateway channels are created
|
||||
per driver from DB config, and OtOpcUa's availability guarantee (serve last-known-good when a source
|
||||
is down) argues against failing readiness on a driver dependency. Treat as **by design**; if a
|
||||
gateway-down signal is wanted, it belongs on a Degraded-only check or the driver-status surface, and
|
||||
needs a lmxopcua issue.
|
||||
- **ScadaBridge `IActiveNodeGate` unification (L2 follow-on).** Still deferred, low value: ScadaBridge
|
||||
registers `ActiveNodeGate` as its own `ZB.MOM.WW.ScadaBridge.InboundAPI.IActiveNodeGate`
|
||||
(`Program.cs:614`), consumed by `InboundApiEndpointFilter`, `InboundMethodWarmupService`,
|
||||
`AuditWriteMiddleware` and `FlowDebugRunner`. The behaviour already converged: the gate calls
|
||||
`ClusterActivityEvaluator.SelfIsOldest` → `ActiveNodeEvaluator.SelfIsOldestUp` → shared
|
||||
`ClusterActiveNode.SelfIsActive`, the same rule as `AkkaActiveNodeGate` and the `active-node` probe, so
|
||||
the gate and the probe cannot disagree. Only the interface type is local. Swapping to the shared
|
||||
interface would touch the InboundAPI project (which would then reference `ZB.MOM.WW.Health`) for no
|
||||
behaviour change.
|
||||
|
||||
→ **Gap L1:** `ZB.MOM.WW.Health.Akka.ActiveNodeHealthCheck` with an optional `RoleFilter`
|
||||
parameter unifies both behaviors. OtOpcUa passes `RoleFilter = "admin"` (role-filtered);
|
||||
ScadaBridge uses no role filter.
|
||||
→ **Gap L2:** ScadaBridge's `ActiveNodeGate` duplicates `ActiveNodeHealthCheck` logic. The shared
|
||||
`IActiveNodeGate` seam + a backing singleton eliminates the duplication.
|
||||
|
||||
### §6 Response writer
|
||||
|
||||
| | OtOpcUa | MxAccessGateway | ScadaBridge |
|
||||
|---|---|---|---|
|
||||
| Writer | Default (plain-text/JSON) | Bespoke `GatewayHealthReply` JSON | `UIResponseWriter.WriteHealthCheckUIResponse` |
|
||||
|
||||
→ **Gap W1:** the shared `ZB.MOM.WW.Health` package ships a canonical JSON response writer
|
||||
(lifting `HealthChecks.UI.Client` style to the default). All three projects adopt it on
|
||||
`MapZbHealth()` call — no per-project writer wiring needed.
|
||||
|
||||
### §7 Endpoint authentication
|
||||
|
||||
Both OtOpcUa and ScadaBridge expose health endpoints without authentication (`AllowAnonymous` or
|
||||
open by default). MxAccessGateway's `/health/live` has no authentication requirement. The spec
|
||||
canonizes this: health tiers are `AllowAnonymous`; `MapZbHealth()` applies `AllowAnonymous` by
|
||||
default.
|
||||
|
||||
No gap — consistent across all three. `MapZbHealth()` should document and enforce this default.
|
||||
|
||||
## Adoption backlog (ordered)
|
||||
|
||||
| # | Item | Projects | Priority | Effort | Risk | Notes |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 1 | MxAccessGateway: remove dead `/health/live` + `AddHealthChecks()`, add `GrpcDependencyHealthCheck` (worker IPC) + `MapZbHealth()` | MxGateway | P1 | S | Low | Gap T1, T3, P1, P3 — no probes/tiers today; highest delta |
|
||||
| 2 | OtOpcUa: replace 3 bespoke checks with shared probes (`AkkaClusterHealthCheck` OtOpcUaCompat + `ActiveNodeHealthCheck` role-filtered + `DatabaseHealthCheck<T>` ProbeQuery) | OtOpcUa | P2 | S | Low | Gap A1, D1, L1 |
|
||||
| 3 | ScadaBridge: replace 3 bespoke checks with shared probes (Default policy + role-less Active + `CanConnectAsync`) + add `/healthz` + unify `ActiveNodeGate` with `IActiveNodeGate` seam | ScadaBridge | P2 | S | Low | Gap T2, A1, D2, L1, L2 |
|
||||
| 4 | OtOpcUa + MxAccessGateway: add `GrpcDependencyHealthCheck` for downstream gRPC channel | OtOpcUa, MxGateway | P2 | S | Low | Gap P2 — closes the silent-gateway-down scenario |
|
||||
| 5 | All: adopt canonical response writer (switch from per-project writers to `MapZbHealth` default) | all 3 | P3 | XS | Low | Gap W1 — mechanical; bundled with #1–3 |
|
||||
| 6 | DB injection style: switch ScadaBridge from injected `DbContext` to `IDbContextFactory<T>` | ScadaBridge | P3 | XS | Low | Gap D2 — background-service safety |
|
||||
|
||||
**Note: adoption items #1–6 are all follow-on tasks.** They are tracked here as the backlog for
|
||||
after `ZB.MOM.WW.Health` @ 0.1.0 is published. The library build itself (nupkgs, tests) is a
|
||||
separate task. This is consistent with how `ZB.MOM.WW.Auth` and `ZB.MOM.WW.Theme` are structured:
|
||||
the library is built first; adoption by the three apps is the next step.
|
||||
|
||||
## Adoption status — 2026-06-01 (DONE)
|
||||
|
||||
`ZB.MOM.WW.Health` 0.1.0 was published to the `dohertj2-gitea` NuGet feed and adopted across all
|
||||
three apps on branch `feat/adopt-zb-health` in each repo (one branch per repo; commits below).
|
||||
Plan + design: [`../../docs/plans/2026-06-01-health-library-adoption.md`](../../docs/plans/2026-06-01-health-library-adoption.md).
|
||||
|
||||
| Repo | What shipped | Build / tests |
|
||||
|---|---|---|
|
||||
| **MxAccessGateway** | Removed the pipeline-bypassing `/health/live` lambda + dead `AddHealthChecks()`; added a custom `AuthStoreHealthCheck` (readiness probe over the SQLite auth store) tagged `Ready`; `MapZbHealth()` → ready/active/healthz + canonical writer. | Server builds; 568 pass / 3 **pre-existing** macOS failures unrelated to health (OrphanWorkerTerminator ×2, fake-worker timeout ×1). |
|
||||
| **OtOpcUa** | Swapped all 3 bespoke checks → shared probes (`DatabaseHealthCheck<OtOpcUaConfigDbContext>` + `ProbeQuery`, `AkkaClusterHealthCheck` **OtOpcUaCompat**, `ActiveNodeHealthCheck(role:"admin")`); `MapZbHealth()`. | Host builds clean; health tests pass **including a real two-node-cluster integration test**. Independently code-reviewed: APPROVED, behaviour-preserving. |
|
||||
| **ScadaBridge** | Swapped 3 bespoke checks → shared probes (`DatabaseHealthCheck<ScadaBridgeDbContext>` scoped fallback, `AkkaClusterHealthCheck` **Default**, role-less `ActiveNodeHealthCheck`); added transient `ActorSystem` DI bridge (Central + Site roots); added `/healthz`; canonical writer. Kept its own `ActiveNodeGate`. | Builds clean (TreatWarningsAsErrors); 212 Host tests pass. |
|
||||
|
||||
### Deferred (verified ill-fitting on adoption — re-scoped from the original backlog)
|
||||
|
||||
- **#4 downstream gRPC dependency probes — DROPPED for now.** (→ #11) Neither repo holds a host-level
|
||||
`GrpcChannel` to probe: the MxGateway↔worker IPC is **named pipes** (not gRPC), and OtOpcUa's
|
||||
gateway channel is created **per-driver** from DB config (no DI-level channel). MxGateway readiness
|
||||
instead probes the SQLite auth store (`AuthStoreHealthCheck`). A real worker/downstream probe needs
|
||||
a custom non-gRPC check — future work.
|
||||
- **#3 `IActiveNodeGate` seam unification (ScadaBridge) — DEFERRED.** (→ #11) ScadaBridge's `IActiveNodeGate`
|
||||
is `…ScadaBridge.InboundAPI.IActiveNodeGate`, wired into inbound-API endpoint gating — a different
|
||||
interface from the shared `ZB.MOM.WW.Health.IActiveNodeGate`. Unification touches the InboundAPI
|
||||
path; its existing `ActiveNodeGate` (logic identical to the shared `AkkaActiveNodeGate`) was kept.
|
||||
- **#6 `IDbContextFactory<T>` switch (ScadaBridge) — DROPPED as unnecessary.** The shared
|
||||
`DatabaseHealthCheck<T>` self-scopes (creates its own DI scope per probe) when no factory is
|
||||
registered — that *is* the background-safety fix — and ScadaBridge's context is built with an
|
||||
injected `IDataProtectionProvider`, which `AddDbContextFactory` does not accommodate cleanly.
|
||||
|
||||
### Accepted behaviour change (one) — flag for ops
|
||||
|
||||
ScadaBridge `/health/active` during the **startup window** (ActorSystem/cluster not yet ready) now
|
||||
returns **Degraded → HTTP 200** instead of the prior **Unhealthy → HTTP 503**. This is the shared
|
||||
`ActiveNodeHealthCheck`'s documented startup-safe behaviour and the normalized convergence target.
|
||||
The steady-state standby case (node Up but not leader) is **unchanged** (Unhealthy → 503). If a
|
||||
load-balancer (Traefik) keys strictly on a 503 from `/health/active` to fence standby nodes during
|
||||
startup, that fail-safe is briefly relaxed until the cluster forms.
|
||||
## Behaviour notes (kept for ops)
|
||||
|
||||
- **Extra probes on the active tier.** ScadaBridge central (`database`) and OtOpcUa admin nodes
|
||||
(`configdb`) put their DB check on `/health/active` deliberately: a DB-dead active node leaves Traefik
|
||||
rotation. Side effect: during a config-DB outage **both** nodes answer 503 on `/health/active`, so
|
||||
Traefik has no backend for the routed UI until the DB returns.
|
||||
- **Startup window.** The shared `ActiveNodeHealthCheck` returns Degraded (200) while the ActorSystem or
|
||||
cluster is not yet available (accepted 2026-06-01 for ScadaBridge central). The ScadaBridge site
|
||||
adapter returns 503 in the same window (see Remaining gaps).
|
||||
- **Singleton apps answer 200 on `/health/active` on every node.** MxAccessGateway (unless all sessions
|
||||
are faulted) and HistorianGateway (always). The QA pair runs both on vq03 and vq04 as independent
|
||||
singletons, so this is correct, but `/health/active` cannot be used to pick one of them.
|
||||
|
||||
@@ -23,14 +23,12 @@ semantics.
|
||||
|
||||
## Status by project
|
||||
|
||||
| Project | Endpoints today | Probes today | Response writer | `/healthz` | `IActiveNodeGate` | Adoption status |
|
||||
|---|---|---|---|---|---|---|
|
||||
| **OtOpcUa** | `/health/ready`, `/health/active`, `/healthz` | Database (query), AkkaCluster (2-way), AdminRoleLeader (role-filtered) | Default (plain-text/JSON) | ✅ present | — | Not started |
|
||||
| **MxAccessGateway** | `/health/live` only (raw `MapGet`; hardcoded `"Healthy"`) | **None** (`AddHealthChecks()` called but unused) | Bespoke `GatewayHealthReply` JSON | ⛔ absent | — | Not started |
|
||||
| **ScadaBridge** | `/health/ready`, `/health/active` | Database (`CanConnectAsync`), AkkaCluster (3-way), ActiveNode (role-less) | `HealthChecks.UI.Client` JSON | ⛔ absent | `ActiveNodeGate` (backs Inbound API 503 gate) | Not started |
|
||||
|
||||
See each project's [`current-state/<project>/CURRENT-STATE.md`](current-state/) for the
|
||||
code-verified detail and its adoption plan.
|
||||
**Adopted by all four apps at `0.3.0`; probe coverage re-verified 2026-11-09 (scadaproj#11).** Every app
|
||||
calls `MapZbHealth()` (`/healthz`, `/health/ready`, `/health/active`, canonical JSON). The per-endpoint
|
||||
coverage matrix, SPEC deviations and remaining gaps are in [`GAPS.md`](GAPS.md); code-verified detail per
|
||||
app (OtOpcUa, MxAccessGateway, ScadaBridge, HistorianGateway) is in
|
||||
[`current-state/<project>/CURRENT-STATE.md`](current-state/). The "Why" section above and the bullets
|
||||
below describe the original 2026-06 design rationale.
|
||||
|
||||
## Normalized vs. left per-project
|
||||
|
||||
@@ -81,8 +79,5 @@ all three.
|
||||
|
||||
## Component status
|
||||
|
||||
**Status: Draft — library built at 0.1.0.** Spec and shared-contract written; current-state docs
|
||||
verified; GAPS backlog populated. Library implemented and packed at
|
||||
[`../../ZB.MOM.WW.Health/`](../../ZB.MOM.WW.Health/) (3 packages, 58 tests;
|
||||
`ZB.MOM.WW.Health`, `ZB.MOM.WW.Health.Akka`, `ZB.MOM.WW.Health.EntityFrameworkCore`).
|
||||
Adoption by the three apps is the next follow-on tracked in [`GAPS.md`](GAPS.md).
|
||||
**Status: Adopted — library at 0.3.0** ([`../../ZB.MOM.WW.Health/`](../../ZB.MOM.WW.Health/), 3 packages),
|
||||
pinned by all four apps; coverage re-verified 2026-11-09. Remaining gaps: [`GAPS.md`](GAPS.md).
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
# Health — current state: HistorianGateway
|
||||
|
||||
**Verified 2026-11-09 at `historiangw@4d6a9ad` (`origin/main`)** for scadaproj#11. Read-only `git grep` /
|
||||
`git show` against the origin default branch; all paths relative to the repo root
|
||||
(`~/Desktop/HistorianGateway`). First current-state doc for this app (it postdates the original
|
||||
three-project survey).
|
||||
|
||||
- **Package:** `ZB.MOM.WW.Health` **0.3.0**, core only
|
||||
(`src/ZB.MOM.WW.HistorianGateway.Server/ZB.MOM.WW.HistorianGateway.Server.csproj:23`).
|
||||
- **Registration:** inline block in `src/ZB.MOM.WW.HistorianGateway.Server/Program.cs:256-329`.
|
||||
- **Mapping:** `app.MapZbHealth()` with default paths/writer (`Program.cs:417`).
|
||||
- **Not** `GrpcDependencyHealthCheck`: earlier scadaproj docs said HistorianGateway used it. It does not
|
||||
on `main` — the gRPC-only Historian client owns its own channel, so the Historian probe goes through the
|
||||
connection pool instead (comment at `Program.cs:265-268`).
|
||||
|
||||
## Per-endpoint contents
|
||||
|
||||
| Endpoint | Checks run | Can return 503 when |
|
||||
|---|---|---|
|
||||
| `/healthz` (live) | none | never |
|
||||
| `/health/ready` | `historian-connection`, `galaxy-sql`, + `store-forward-drain` (if `StoreForward:Enabled`), + `historian-session-pool` / `historian-event-session-pool` (if each pool is enabled — **both default `true`**) | Historian probe fails/> 3 s; Galaxy SQL fails/> `CommandTimeoutSeconds`; a session-pool factory has ≥ 3 consecutive failures |
|
||||
| `/health/active` | **none** | never — an empty report is Healthy, so every node answers 200 |
|
||||
|
||||
## Checks
|
||||
|
||||
Files under `src/ZB.MOM.WW.HistorianGateway.Server/Health/`. All tagged `ready`, `failureStatus: null`.
|
||||
|
||||
| Name | Class | Probes | Failure |
|
||||
|---|---|---|---|
|
||||
| `historian-connection` | `HistorianConnectionHealthCheck` | `IHistorianConnectionPool.AcquireAsync` + `client.ProbeAsync`, 3 s cap; reports a faulted client back to the pool | Unhealthy |
|
||||
| `galaxy-sql` | `SqlConnectionHealthCheck` (raw `SqlConnection` + `SELECT 1`; no EF) | Galaxy Repository DB from the top-level `Galaxy` section, bounded by its command timeout | Unhealthy |
|
||||
| `store-forward-drain` | `StoreForwardDrainHealthCheck` | `IHistorianStoreForward.GetStatusAsync`; backing up = `Storing \|\| ErrorOccurred` | **Degraded** only (never 503) |
|
||||
| `historian-session-pool` | `SessionPoolHealthCheck("read", …)` | `HistorianSessionPool.Snapshot.ConsecutiveFactoryFailures` | 0 → Healthy; 1–2 → Degraded; ≥ 3 → **Unhealthy** |
|
||||
| `historian-event-session-pool` | `SessionPoolHealthCheck("event", …)` | same over `HistorianEventSessionPool` | same |
|
||||
|
||||
### Open finding: the session-pool check latches Unhealthy — historiangw#87 (open)
|
||||
|
||||
historiangw#87 (Server-007, High/P1): `_consecutiveFactoryFailures` is reset only by a **successful
|
||||
factory call**, and the factory only runs when a real RPC needs a new lease (the keepalive sweep never
|
||||
opens sessions). After a Historian outage of ≥ 3 failed leases, `/health/ready` goes 503, a k8s probe
|
||||
removes the pod, no RPC arrives, and the check **never recovers** until restart; three clients whose
|
||||
short deadlines cancel a cold handshake also trip it (the `catch` counts caller cancellation). On the NSSM
|
||||
wonder boxes it shows as a false Unhealthy on Overview.
|
||||
|
||||
**Still present on `origin/main` `4d6a9ad`:** `HistorianSessionPool.cs:264-273` and
|
||||
`HistorianEventSessionPool.cs:254-262` increment on any throw and reset only on success; nothing probes
|
||||
or decays the streak; both pools default to `Enabled = true`
|
||||
(`Configuration/HistorianSessionPoolOptions.cs:24`, `EventSessionPoolOptions.cs:24`), so the latching
|
||||
check is on the ready tier in the default configuration.
|
||||
|
||||
## Active-node gating
|
||||
|
||||
None, by design: a single-process sidecar per host with no cluster. No `IActiveNodeGate`, no
|
||||
`RequireActiveNode()`. `Redundancy` options concern the Historian servers it talks to, not node
|
||||
election.
|
||||
|
||||
## Other endpoints
|
||||
|
||||
`MapZbMetrics()`, `MapZbGalaxyRepository()`, the dashboard (its `HealthPage` renders the same
|
||||
ready-tier results), and the four Historian gRPC services.
|
||||
|
||||
## vs SPEC
|
||||
|
||||
| SPEC | Status | Note |
|
||||
|---|---|---|
|
||||
| §1 three tiers via `MapZbHealth`, canonical tags | ✅ | `/health/active` is empty (always 200) |
|
||||
| §1 nothing on `live` | ✅ | |
|
||||
| §2.1 DB probe | 🟡 custom | `SqlConnectionHealthCheck` (no EF context to wrap) |
|
||||
| §2.4 downstream dependency probe | ✅ (custom) | Historian via the pool, not `GrpcDependencyHealthCheck` |
|
||||
| Ready = "can this node serve its dependencies?" | 🟡 | correct while dependencies fail; **does not recover** after a session-pool outage (#87) |
|
||||
| §3 canonical JSON writer | ✅ | |
|
||||
| §4 `IActiveNodeGate` | n/a | |
|
||||
@@ -1,133 +1,68 @@
|
||||
# Health — current state: MxAccessGateway
|
||||
|
||||
Repo: `~/Desktop/MxAccessGateway`. Stack: .NET 10 gateway (x64) + .NET 4.8 worker (x86), gRPC;
|
||||
solution `src/MxGateway.sln`.
|
||||
Health code lives in `src/ZB.MOM.WW.MxGateway.Server/GatewayApplication.cs`. All paths relative
|
||||
to repo root.
|
||||
Verified 2026-06-01.
|
||||
**Verified 2026-11-09 at `mxaccessgw@de86f8b8` (`origin/main`)** for scadaproj#11. Read-only `git grep` /
|
||||
`git show` against the origin default branch; all paths relative to the repo root
|
||||
(`~/Desktop/MxAccessGateway`). The pre-adoption state (verified 2026-06-01: a hardcoded `/health/live`
|
||||
`MapGet`, a dead `AddHealthChecks()`, no probes) is gone; see git history of this file.
|
||||
|
||||
**Summary: bare liveness only.** MxAccessGateway has a single `/health/live` endpoint that returns
|
||||
a hardcoded `GatewayHealthReply` JSON object. `AddHealthChecks()` is called at startup but is
|
||||
entirely unused — no `IHealthCheck` implementations are registered, `MapHealthChecks` is never
|
||||
called, and there is no readiness or active-node tier. The net48 x86 worker process has no HTTP
|
||||
server and therefore no health endpoint of any kind.
|
||||
- **Package:** `ZB.MOM.WW.Health` **0.3.0**, core only (`src/ZB.MOM.WW.MxGateway.Server/ZB.MOM.WW.MxGateway.Server.csproj:20`).
|
||||
No `.Akka`, no `.EntityFrameworkCore` — the gateway has no Akka cluster and no EF context.
|
||||
- **Registration:** `builder.Services.AddGatewayHealthChecks()` (`GatewayApplication.cs:118`) →
|
||||
`src/ZB.MOM.WW.MxGateway.Server/Diagnostics/GatewayHealthCheckExtensions.cs:19-41`.
|
||||
- **Mapping:** `endpoints.MapZbHealth()` with default paths/writer (`GatewayApplication.cs:346`),
|
||||
anonymous unconditionally (the Overview dashboard polls `/health/active` without credentials).
|
||||
|
||||
## 1. Endpoint wiring
|
||||
## Per-endpoint contents
|
||||
|
||||
`src/ZB.MOM.WW.MxGateway.Server/GatewayApplication.cs`:
|
||||
|
||||
- `:61` — `builder.Services.AddHealthChecks()` is called in the DI registration block. **This call
|
||||
is dead**: no `.AddCheck<T>()` call follows it, no `MapHealthChecks` is ever called. The
|
||||
framework registers the health-check infrastructure but nothing is wired through it.
|
||||
- `:139–145` — `MapGatewayEndpoints` maps a raw `endpoints.MapGet("/health/live", ...)` (not
|
||||
`MapHealthChecks`). The handler is an inline lambda that returns `Results.Ok(new GatewayHealthReply(...))`
|
||||
with a hardcoded `Status: "Healthy"`:
|
||||
|
||||
```csharp
|
||||
endpoints.MapGet(
|
||||
"/health/live",
|
||||
() => Results.Ok(new GatewayHealthReply(
|
||||
Status: "Healthy",
|
||||
DefaultBackend: GatewayContractInfo.DefaultBackendName,
|
||||
WorkerProtocolVersion: GatewayContractInfo.WorkerProtocolVersion)))
|
||||
.WithName("LiveHealth");
|
||||
```
|
||||
|
||||
This endpoint always returns HTTP 200 `{"Status":"Healthy",...}` as long as the process is alive.
|
||||
It carries no authentication requirement (no `[Authorize]` or `.RequireAuthorization()`).
|
||||
|
||||
## 2. Response shape
|
||||
|
||||
`GatewayHealthReply` is a record with three fields:
|
||||
- `Status` — always `"Healthy"` (hardcoded string, not the ASP.NET Core `HealthStatus` enum)
|
||||
- `DefaultBackend` — value of `GatewayContractInfo.DefaultBackendName` (the configured backend
|
||||
name, useful for confirming which gateway instance a probe hit)
|
||||
- `WorkerProtocolVersion` — value of `GatewayContractInfo.WorkerProtocolVersion` (the gRPC
|
||||
protocol version the gateway expects from the worker, useful for version-skew detection)
|
||||
|
||||
The response is not `HealthChecks.UI.Client` JSON and is not the standard ASP.NET Core health
|
||||
response shape. It is a bespoke JSON record.
|
||||
|
||||
## 3. Probes
|
||||
|
||||
None. There is no `IHealthCheck` registered. The `/health/live` response does not reflect:
|
||||
|
||||
- Whether the SQLite auth-store is reachable
|
||||
- Whether any active MXAccess session is functional
|
||||
- Whether the x86 worker named-pipe IPC is connected or the worker process is alive
|
||||
- Whether the gRPC service is actually accepting calls
|
||||
|
||||
The endpoint is purely a process liveness indicator.
|
||||
|
||||
## 4. Tier coverage
|
||||
|
||||
| Tier | Endpoint | Status |
|
||||
| Endpoint | Checks run | Can return 503 when |
|
||||
|---|---|---|
|
||||
| Process liveness | `/health/live` (raw `MapGet`) | ✅ present (but non-standard) |
|
||||
| Readiness | `/health/ready` | ⛔ absent |
|
||||
| Active node | `/health/active` | ⛔ absent (not Akka-based; not applicable as-is) |
|
||||
| `healthz` convention | `/healthz` | ⛔ absent |
|
||||
| `/healthz` (live) | none (`_ => false`) | never (process up) |
|
||||
| `/health/ready` | `auth-store`, `worker-executable`, `galaxy-sql` | auth SQLite unreachable; worker exe missing; Galaxy SQL unreachable or > 5 s |
|
||||
| `/health/active` | `mxaccess-sessions` | every open session is `Faulted` (zero sessions = Healthy) |
|
||||
|
||||
MxAccessGateway is not an Akka.NET application — it has no cluster, no leader election, and no
|
||||
active-node concept. The "active" tier in the shared spec translates here to "is the worker process
|
||||
connected and the gRPC service ready to accept calls?" rather than cluster leadership.
|
||||
No check is tagged `live` (asserted by `GatewayHealthCheckRegistrationTests`).
|
||||
|
||||
## 5. x86 worker
|
||||
## Checks
|
||||
|
||||
`ZB.MOM.WW.MxGateway.Worker` is a .NET 4.8 console application communicating with the gateway
|
||||
over Windows named-pipe IPC. It has no HTTP server, no health endpoint, and no exposure to any
|
||||
probe mechanism. Its liveness must be inferred indirectly — either via the gateway process
|
||||
monitoring it (not currently implemented) or via the `GrpcDependencyHealthCheck` the gateway
|
||||
could use to probe the IPC channel.
|
||||
All registered with `AddTypeActivatedCheck`, `failureStatus: null` (the check's own status is used).
|
||||
Files under `src/ZB.MOM.WW.MxGateway.Server/Diagnostics/`.
|
||||
|
||||
## 6. Notable gaps
|
||||
| Name | Class | Tag | Probes | Failure |
|
||||
|---|---|---|---|---|
|
||||
| `auth-store` | `AuthStoreHealthCheck` | ready | Opens the API-key SQLite store via `AuthSqliteConnectionFactory`, runs `SELECT 1` | Unhealthy |
|
||||
| `worker-executable` | `WorkerExecutableHealthCheck` | ready | `File.Exists(Path.GetFullPath(MxGateway:Worker:ExecutablePath))` — same resolution as `WorkerProcessLauncher.ResolveExecutablePath` | Unhealthy (missing or invalid path) |
|
||||
| `galaxy-sql` | `GalaxySqlHealthCheck` | ready | `IGalaxyRepository.TestConnectionAsync` bounded by a 5 s `TimeProvider` timeout | Unhealthy (false, throw, or timeout) |
|
||||
| `mxaccess-sessions` | `SessionHealthCheck` | active | Buckets `ISessionRegistry.Snapshot()` by state; `data` = total/ready/faulted/starting/closing | Healthy if 0 faulted (incl. 0 sessions); Degraded if some faulted; Unhealthy if all usable sessions faulted |
|
||||
|
||||
- `AddHealthChecks()` at `:61` is dead code. No `IHealthCheck` is ever registered via this call.
|
||||
- `/health/live` uses `MapGet` (a raw minimal-API handler) rather than `MapHealthChecks`. It
|
||||
bypasses the ASP.NET Core health-check middleware entirely, which means it does not participate
|
||||
in the standard health-check pipeline (no `IHealthCheckPublisher`, no `HealthReport`, no UI
|
||||
integration).
|
||||
- The hardcoded `"Healthy"` status means the endpoint cannot reflect real probe results even if
|
||||
probes were added later — the handler must be replaced, not just supplemented.
|
||||
- No readiness gating: orchestrators (Kubernetes, Traefik) that rely on `/health/ready` returning
|
||||
503 until the process is actually ready will receive 200 (or 404) from MxAccessGateway today.
|
||||
`worker-executable` and `galaxy-sql` are new on `main` `8c21678c` (mxaccessgw#206, closed 2026-11-09).
|
||||
They check that the worker **binary exists** and that **Galaxy SQL answers**; neither checks a running
|
||||
worker or the named-pipe IPC (workers are per-session and only exist while a client holds a session).
|
||||
No LDAP check was added (#206 comment: deliberately out of scope).
|
||||
|
||||
---
|
||||
`galaxy-sql` is unconditional: `AddZbGalaxyRepository` is always registered (`GatewayApplication.cs:167`),
|
||||
so a gateway whose Galaxy SQL is down drops out of readiness even though its MXAccess data path does not
|
||||
use SQL. That is the #206 design ("must take the gateway out of rotation"), not an accident.
|
||||
|
||||
## Adoption plan → `ZB.MOM.WW.Health`
|
||||
## Active-node gating
|
||||
|
||||
**Replace `/health/live` + wire the shared tiers:**
|
||||
None. The gateway is a singleton per host (no cluster), registers no `IActiveNodeGate`, and uses no
|
||||
`RequireActiveNode()`. The `active` tier is used for a **session-health** signal instead of an
|
||||
active/standby signal, so every gateway answers 200 on `/health/active` unless all its sessions are
|
||||
faulted.
|
||||
|
||||
The `AddHealthChecks()` call at `GatewayApplication.cs:61` is already present — it just needs
|
||||
probes registered against it. The raw `MapGet("/health/live", ...)` handler at `:139–145` must be
|
||||
removed and replaced with `app.MapZbHealth()` from `ZB.MOM.WW.Health`.
|
||||
## Other endpoints
|
||||
|
||||
Steps:
|
||||
`MapZbMetrics()` (`:347`, anonymous by default, see `MetricsOptions`), `MapZbGalaxyRepository()` (`:359`),
|
||||
dashboard routes. No bespoke health endpoint remains.
|
||||
|
||||
1. **Remove** the inline `MapGet("/health/live", ...)` lambda (`:139–145`). The `GatewayHealthReply`
|
||||
record and `DefaultBackend`/`WorkerProtocolVersion` metadata can be surfaced differently (e.g., a
|
||||
`/info` endpoint or as custom data on the health response).
|
||||
2. **Register a `GrpcDependencyHealthCheck`** (from `ZB.MOM.WW.Health`) that probes the
|
||||
named-pipe IPC channel to the x86 worker. Tag `["ready"]`. This replaces the hardcoded
|
||||
liveness-only response with a real probe that reflects whether the worker is reachable.
|
||||
3. **Optionally add a `GrpcDependencyHealthCheck`** for any downstream gRPC dependency (e.g., the
|
||||
Galaxy Repository connection) if the gateway is expected to be healthy only when its upstreams are
|
||||
reachable. Tag `["ready"]`.
|
||||
4. **Call `app.MapZbHealth()`** — this maps `/health/ready` (tag `ready`), `/health/active` (tag
|
||||
`active`; initially empty — no active-node concept in MxGateway), and `/healthz` (bare liveness).
|
||||
The `/healthz` endpoint replaces the semantic role that `/health/live` served today.
|
||||
5. **Do not add `ZB.MOM.WW.Health.Akka`** — MxAccessGateway has no Akka dependency. The consumer
|
||||
matrix in the design specifies MxGateway uses the core package only.
|
||||
## vs SPEC
|
||||
|
||||
**Keep bespoke:**
|
||||
|
||||
- The `WorkerProtocolVersion` / `DefaultBackend` metadata from `GatewayHealthReply` is
|
||||
MxAccessGateway-specific; keep it as a separate `/info` endpoint or embed it as `Data` on a
|
||||
custom probe rather than normalizing it into the shared contract.
|
||||
- The x86 worker itself (net48 console, named-pipe IPC, no HTTP) remains outside the shared health
|
||||
scheme. The `GrpcDependencyHealthCheck` observes the worker indirectly from the gateway side.
|
||||
- Per-gateway auth and TLS concerns on who may call health endpoints remain per-project.
|
||||
|
||||
**Adoption is a follow-on task** (tracked in `GAPS.md`), not part of the `ZB.MOM.WW.Health`
|
||||
library build. MxGateway is the **highest-priority adopter** (P1 gap — no probes/tiers today)
|
||||
and should be the first app wired up once the nupkg is available.
|
||||
| SPEC | Status | Note |
|
||||
|---|---|---|
|
||||
| §1 three tiers via `MapZbHealth`, canonical tags | ✅ | default paths |
|
||||
| §1 nothing on `live` | ✅ | test-asserted |
|
||||
| §2.1 `DatabaseHealthCheck<T>` | n/a | no EF; custom SQLite + Galaxy SQL checks |
|
||||
| §2.3 active tier = oldest `Up` member | 🟡 deviation | no cluster; active tier carries `mxaccess-sessions` (deliberate, documented in `SessionHealthCheck`) |
|
||||
| §2.4 `GrpcDependencyHealthCheck` for the worker | ⛔ not used | worker IPC is named pipes, not gRPC; `worker-executable` is a presence proxy (see GAPS P2) |
|
||||
| §3 canonical JSON writer | ✅ | `MapZbHealth` default |
|
||||
| §4 `IActiveNodeGate` | n/a | no active/standby |
|
||||
|
||||
@@ -1,154 +1,68 @@
|
||||
# Health — current state: OtOpcUa
|
||||
|
||||
Repo: `~/Desktop/OtOpcUa`. Stack: .NET 10, Akka.NET, OPC UA; solution `ZB.MOM.WW.OtOpcUa.slnx`.
|
||||
Health code lives in `src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/`. All paths relative to repo root.
|
||||
Verified 2026-06-01.
|
||||
**Verified 2026-11-09 at `lmxopcua@c904472a` (`origin/master`)** for scadaproj#11. Read-only `git grep` /
|
||||
`git show` against the origin default branch; all paths relative to the repo root (`~/Desktop/OtOpcUa`).
|
||||
The pre-adoption state (verified 2026-06-01: three bespoke checks incl. `AdminRoleLeaderHealthCheck`,
|
||||
default writer) is gone; see git history of this file.
|
||||
|
||||
Full three-tier pattern: `/health/ready`, `/health/active`, and `/healthz`. Three probes covering
|
||||
the database, the Akka cluster, and the admin-role leader. All endpoints are `AllowAnonymous` to
|
||||
permit Traefik and load-balancer probing without credentials.
|
||||
- **Packages:** `ZB.MOM.WW.Health`, `.Akka`, `.EntityFrameworkCore` all **0.3.0**
|
||||
(`Directory.Packages.props:211,219,220`; referenced by `src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj:37-39`;
|
||||
`.Akka` also by `ControlPlane` and `Runtime`, which use `ClusterActiveNode` for the redundancy election).
|
||||
- **Registration:** `builder.Services.AddOtOpcUaHealth(hasAdmin, hasDriver)` (`src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs:581`)
|
||||
→ `src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/HealthEndpoints.cs:40-130`. The check set depends on the node's roles.
|
||||
- **Mapping:** `app.MapOtOpcUaHealth()` (`Program.cs:767`) → `app.MapZbHealth()` with defaults (`HealthEndpoints.cs:135`).
|
||||
|
||||
## 1. Endpoint wiring
|
||||
## Per-endpoint contents (by node role)
|
||||
|
||||
`src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/HealthEndpoints.cs`:
|
||||
|
||||
- `:13` — XML comment explicitly names this as "ScadaLink's three-tier pattern: `ready` = boot ok;
|
||||
`active` = fully serving traffic; `healthz` = bare process liveness."
|
||||
- `:17` — `AddOtOpcUaHealth(IServiceCollection)` calls `services.AddHealthChecks()` and registers
|
||||
all three probes (lines 20–22):
|
||||
- `DatabaseHealthCheck` name `"configdb"`, tags `["ready","active"]`
|
||||
- `AkkaClusterHealthCheck` name `"akka"`, tags `["ready","active"]`
|
||||
- `AdminRoleLeaderHealthCheck` name `"admin-leader"`, tags `["active"]` only
|
||||
- `:28` — `MapOtOpcUaHealth(IEndpointRouteBuilder)` maps three endpoints (lines 33–44):
|
||||
- `/health/ready` — predicate `c => c.Tags.Contains("ready")`, `.AllowAnonymous()` (lines 33–36)
|
||||
- `/health/active` — predicate `c => c.Tags.Contains("active")`, `.AllowAnonymous()` (lines 37–40)
|
||||
- `/healthz` — predicate `_ => false` (no probes run; bare process liveness only), `.AllowAnonymous()` (lines 41–44)
|
||||
|
||||
`Program.cs`:
|
||||
- `:137` — `builder.Services.AddOtOpcUaHealth()`
|
||||
- `:159` — `app.MapOtOpcUaHealth()`
|
||||
|
||||
Response writer: default ASP.NET Core plain-text/JSON (no `HealthChecks.UI.Client`).
|
||||
|
||||
## 2. Probes
|
||||
|
||||
### DatabaseHealthCheck
|
||||
`src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/DatabaseHealthCheck.cs`:
|
||||
|
||||
- `:9` — injects `IDbContextFactory<OtOpcUaConfigDbContext>`
|
||||
- `:25–37` — opens a pooled context via `CreateDbContextAsync`, runs
|
||||
`db.Deployments.AsNoTracking().Take(1).ToListAsync()`. If the query succeeds →
|
||||
`HealthCheckResult.Healthy("ConfigDb reachable")` (`:31`). If it throws →
|
||||
`HealthCheckResult.Unhealthy("ConfigDb unreachable", ex)` (`:35`). No `Degraded` path.
|
||||
|
||||
The probe exercises a real query (not just `CanConnectAsync`) — it confirms the `Deployments` table
|
||||
is readable, which implies the schema migration has run. This is **stricter** than ScadaBridge's
|
||||
`CanConnectAsync` but more opaque about the failure reason.
|
||||
|
||||
Tags on registration: `["ready","active"]` — the database must be reachable for both readiness and
|
||||
active-node determination.
|
||||
|
||||
### AkkaClusterHealthCheck
|
||||
`src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/AkkaClusterHealthCheck.cs`:
|
||||
|
||||
- `:9` — injects `ActorSystem` directly
|
||||
- `:27–33` — calls `Cluster.Get(_system)`, scans `cluster.State.Members` for the member whose
|
||||
`Address == cluster.SelfAddress` and `Status == MemberStatus.Up`:
|
||||
- Found Up → `HealthCheckResult.Healthy($"Self Up; {cluster.State.Members.Count} member(s)")` (`:32`)
|
||||
- Not found → `HealthCheckResult.Degraded("Self not yet Up in cluster")` (`:33`)
|
||||
|
||||
No `Unhealthy` path — joining/leaving/removed nodes are all reported as `Degraded`. This differs from
|
||||
ScadaBridge's more granular three-way policy (see GAPS).
|
||||
|
||||
Tags on registration: `["ready","active"]`.
|
||||
|
||||
### AdminRoleLeaderHealthCheck
|
||||
`src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/AdminRoleLeaderHealthCheck.cs`:
|
||||
|
||||
- `:14` — injects `IClusterRoleInfo`
|
||||
- `:27–38` — three-path logic:
|
||||
- Node does not carry the `"admin"` role → `Healthy("Node does not carry admin role")` (`:30`) —
|
||||
non-admin nodes are immediately healthy, so this probe never gates a non-admin node.
|
||||
- Admin role + node is the role leader → `Healthy($"Admin leader ({...})")` (`:36`)
|
||||
- Admin role + not the leader → `Degraded($"Admin member but not leader (leader=...)")` (`:37`)
|
||||
|
||||
Tags on registration: `["active"]` only — does not participate in `/health/ready`. The intent is
|
||||
Traefik routing: the active node (admin-role leader) gets sticky admin-UI traffic; standby nodes
|
||||
are reachable for data-plane OPC UA but report `Degraded` on `/health/active` so the load balancer
|
||||
does not route control-plane traffic to them.
|
||||
|
||||
Note: no `Unhealthy` path for the role-filter case. If the ActorSystem is not running, `IClusterRoleInfo`
|
||||
presumably returns safe defaults (no role); this is not separately health-checked.
|
||||
|
||||
## 3. Tag / tier summary
|
||||
|
||||
| Probe | `/health/ready` | `/health/active` | `/healthz` |
|
||||
| Endpoint | admin-only node | fused admin+driver | driver-only node |
|
||||
|---|---|---|---|
|
||||
| `DatabaseHealthCheck` | ✅ | ✅ | — |
|
||||
| `AkkaClusterHealthCheck` | ✅ | ✅ | — |
|
||||
| `AdminRoleLeaderHealthCheck` | — | ✅ | — |
|
||||
| (no probes) | — | — | ✅ (bare liveness) |
|
||||
| `/healthz` | none | none | none |
|
||||
| `/health/ready` | `configdb`, `akka` | `opcua-endpoint`, `configdb`, `akka` | `opcua-endpoint`, `akka` |
|
||||
| `/health/active` | `configdb`, `akka`, `cluster-primary`, `localdb-replication` | `configdb`, `akka`, `cluster-primary`, `localdb-replication` | `opcua-endpoint`, `akka`, `cluster-primary`, `localdb-replication` |
|
||||
|
||||
`/healthz` runs zero probes — it is a pure process liveness sentinel (process reachable = healthy;
|
||||
a crashed process = no response). Kubernetes liveness probes, Traefik TCP checks, and uptime
|
||||
monitors use this tier.
|
||||
`opcua-endpoint` is deliberately **not** on the fused node's active tier: Traefik routes the AdminUI by
|
||||
`/health/active`, and a failed SDK start must not take the operator console out of rotation; OPC UA
|
||||
failover is handled by the Runtime (ServiceLevel 0 + `OpcUaEndpointStepDownActor`, lmxopcua#762)
|
||||
(`HealthEndpoints.cs:44-65`).
|
||||
|
||||
## 4. Downstream dependency coverage
|
||||
## Checks
|
||||
|
||||
No probe for the upstream MxAccessGateway gRPC channel. If the gateway is unreachable, OtOpcUa
|
||||
reports healthy here (the GalaxyDriver will surface errors in OPC UA diagnostics, but `/health/ready`
|
||||
and `/health/active` will not reflect it). This is a gap that the shared `GrpcDependencyHealthCheck`
|
||||
probe in `ZB.MOM.WW.Health` would close.
|
||||
| Name | Class | Tags | Probes | Failure |
|
||||
|---|---|---|---|---|
|
||||
| `opcua-endpoint` (driver nodes) | `OpcUaEndpointHealthCheck` (host, `Health/OpcUaEndpointHealthCheck.cs`) | ready (+ active on driver-only) | `IOpcUaEndpointState.Status`: `Started` healthy | Unhealthy on `NotStarted` / `Stopped` / failed |
|
||||
| `configdb` (admin nodes) | shared `DatabaseHealthCheck<OtOpcUaConfigDbContext>` | ready, active | `ProbeQuery` = `Deployments.AsNoTracking().Take(1)` (implies schema applied) | Unhealthy |
|
||||
| `akka` | shared `AkkaClusterHealthCheck`, policy **`OtOpcUaCompat`** | ready, active | self `Up` among reachable members | **Degraded only** (never 503) |
|
||||
| `cluster-primary` | shared `ActiveNodeHealthCheck`, `RolePreference = [admin, driver]` | active | oldest `Up` member of the first preferred role held | Unhealthy on standby and on a node holding neither role; Degraded while the ActorSystem/cluster is not yet available |
|
||||
| `localdb-replication` | `LocalDbReplicationHealthCheck` (host) | active | `ISyncStatus` connected + oplog backlog < 100 000 | **Degraded only**; Healthy when LocalDb or replication is absent/off |
|
||||
|
||||
## 5. Notable design choices
|
||||
Driver-only nodes register no `configdb` check: they hold no ConfigDb (FetchAndCache from central), and the
|
||||
type-activated check would throw resolving the factory.
|
||||
|
||||
- **AllowAnonymous on all tiers** — see `HealthEndpoints.cs:30–32` comment: "Without it the
|
||||
`AddOtOpcUaAuth` fallback policy 401s every probe and Traefik marks every backend unhealthy."
|
||||
- **Query probe, not `CanConnectAsync`** — the `Deployments` query validates that the schema has
|
||||
been applied. ScadaBridge uses `CanConnectAsync`; neither is wrong but they diverge.
|
||||
- **`Degraded` semantics** — the Akka check uses `Degraded` (not `Unhealthy`) for a joining/pre-Up
|
||||
node. ASP.NET Core maps `Degraded` to HTTP 200 by default; Traefik sees 200 and considers the
|
||||
node ready. If `Unhealthy` (HTTP 503) is required to gate traffic, the `Degraded` path is
|
||||
insufficient.
|
||||
- **`IClusterRoleInfo` abstraction** — the admin-leader check depends on `IClusterRoleInfo`, an OtOpcUa
|
||||
interface, not the raw `Akka.Cluster.Cluster` API. This is a testability-friendly layer absent in
|
||||
ScadaBridge's direct Akka usage.
|
||||
## Active-node gating
|
||||
|
||||
---
|
||||
- **Probe:** `cluster-primary` above — the shared `ClusterActiveNode` rule. Per-cluster meshes (Phase 6)
|
||||
make "oldest `Up` member" already per-`Cluster`, so exactly one node per `Cluster` answers 200.
|
||||
`RedundancyStateActor`'s Primary election routes through the same primitive, so the tier and the
|
||||
OPC UA ServiceLevel 250/240 split cannot disagree.
|
||||
- **Request gate:** none. OtOpcUa registers no `IActiveNodeGate` and uses no `RequireActiveNode()`; the
|
||||
AdminUI is pinned to the active node by Traefik routing on `/health/active`.
|
||||
- **Extra probes on the active tier:** `configdb` (Unhealthy) — a ConfigDb outage makes the admin pair
|
||||
answer 503 on both nodes, so Traefik has no AdminUI backend until it recovers. `akka` and
|
||||
`localdb-replication` can only Degrade, so they never change routing.
|
||||
|
||||
## Adoption plan → `ZB.MOM.WW.Health`
|
||||
## Other endpoints
|
||||
|
||||
**Replace with shared probes:**
|
||||
`MapOtOpcUaMetrics()`; mesh gRPC services (`MapOtOpcUaMeshGrpcServices`). No bespoke health endpoint.
|
||||
|
||||
- `AkkaClusterHealthCheck` → `ZB.MOM.WW.Health.Akka.AkkaClusterHealthCheck` using the
|
||||
**`OtOpcUaCompat` preset** (self-Up-among-members scan → Healthy/Degraded). The preset keeps
|
||||
OtOpcUa's existing two-way policy without forcing ScadaBridge's three-way policy onto it.
|
||||
- `AdminRoleLeaderHealthCheck` → `ZB.MOM.WW.Health.Akka.ActiveNodeHealthCheck` with
|
||||
`RoleFilter = "admin"`. The role-filter parameter produces identical behavior: non-admin nodes
|
||||
immediately healthy, admin leader healthy, admin non-leader degraded.
|
||||
- `DatabaseHealthCheck` → `ZB.MOM.WW.Health.EntityFrameworkCore.DatabaseHealthCheck<OtOpcUaConfigDbContext>`
|
||||
with a `ProbeQuery` delegate of `db => db.Deployments.AsNoTracking().Take(1).ToListAsync()`.
|
||||
The delegate preserves the stricter query probe rather than falling back to `CanConnectAsync`.
|
||||
- Add `GrpcDependencyHealthCheck` targeting the MxAccessGateway channel (closes the downstream
|
||||
dependency gap noted in §4). Tag `["ready","active"]`.
|
||||
- Replace `AddOtOpcUaHealth` / `MapOtOpcUaHealth` with
|
||||
`services.AddHealthChecks().AddCheck<...>()` (one call per probe, per spec §5) +
|
||||
`app.MapZbHealth()`. The `/healthz` bare-liveness tier is part of `MapZbHealth` by default —
|
||||
no separate wiring needed.
|
||||
## vs SPEC
|
||||
|
||||
**Keep bespoke:**
|
||||
|
||||
- `IClusterRoleInfo` and its Akka implementation — on adoption this testability seam is given up
|
||||
for the health-check path. The shared `ActiveNodeHealthCheck` reads cluster role state from the
|
||||
ActorSystem directly (resolving it lazily via `IServiceProvider`); it does not accept
|
||||
`IClusterRoleInfo` as an injection point. This is an accepted trade-off: the shared implementation
|
||||
is simpler and consistent across projects, while `IClusterRoleInfo` remains available elsewhere
|
||||
in the OtOpcUa codebase where it is used outside health checks.
|
||||
- The `AllowAnonymous` policy — this is an OtOpcUa auth concern; `MapZbHealth` must document that
|
||||
callers are responsible for applying `AllowAnonymous` (or the shared helper applies it by default).
|
||||
- Which probes are registered and their tag assignments — the shared library supplies the check
|
||||
implementations; the wiring (which names, which tags, which options) remains per-project.
|
||||
|
||||
**Adoption is a follow-on task** (tracked in `GAPS.md`), not part of the `ZB.MOM.WW.Health`
|
||||
library build. The library build delivers the shared implementations; adoption lands in the
|
||||
OtOpcUa repo as a separate commit once the nupkg is available.
|
||||
| SPEC | Status | Note |
|
||||
|---|---|---|
|
||||
| §1 three tiers via `MapZbHealth`, canonical tags | ✅ | |
|
||||
| §1 nothing on `live` | ✅ | |
|
||||
| §2.1 `DatabaseHealthCheck<T>` | ✅ | `ProbeQuery` variant |
|
||||
| §2.2 / §7(b) Akka probe with **Default** policy | 🟡 | still `OtOpcUaCompat` (Degraded-only); see GAPS A1 |
|
||||
| §2.3 / §7(b) `ActiveNodeHealthCheck` | ✅ | role-preference mode |
|
||||
| §2.4 downstream (mxaccessgw) gRPC probe | ⛔ not used | gateway channels are per-driver, created from DB config; see GAPS P2 |
|
||||
| §3 canonical JSON writer | ✅ | |
|
||||
| §4 / §7(e) `IActiveNodeGate` | n/a | no gated endpoints; Traefik routes by the probe |
|
||||
|
||||
@@ -1,186 +1,71 @@
|
||||
# Health — current state: ScadaBridge
|
||||
|
||||
Repo: `~/Desktop/ScadaBridge`. Stack: .NET 10, Akka.NET; solution `ZB.MOM.WW.ScadaBridge.slnx`.
|
||||
Health code centers on `src/ZB.MOM.WW.ScadaBridge.Host/Health/` (ASP.NET probes) and the
|
||||
separate `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/` project (domain aggregation pipeline).
|
||||
All paths relative to repo root.
|
||||
Verified 2026-06-01.
|
||||
**Verified 2026-11-09 at `ScadaBridge@c7d868c5` (`origin/main`)** for scadaproj#11. Read-only `git grep` /
|
||||
`git show` against the origin default branch; all paths relative to the repo root
|
||||
(`~/Desktop/ScadaBridge`). The pre-adoption state (verified 2026-06-01: name-predicate endpoints, no
|
||||
`/healthz`, `HealthChecks.UI.Client` writer) is gone; see git history of this file.
|
||||
|
||||
Two-tier pattern: `/health/ready` and `/health/active` — no `/healthz`. Three probes (database,
|
||||
Akka cluster, active-node). ScadaBridge also has a bespoke distributed `HealthMonitoring/`
|
||||
pipeline that is entirely separate from the ASP.NET health checks and is out of scope for the
|
||||
shared library.
|
||||
Out of scope, as before: the `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/` domain-aggregation pipeline
|
||||
(site → central health reports over Akka) is not an ASP.NET health probe.
|
||||
|
||||
## 1. Endpoint wiring
|
||||
- **Packages:** `ZB.MOM.WW.Health`, `.Akka`, `.EntityFrameworkCore` all **0.3.0** (`Directory.Packages.props:97-99`;
|
||||
referenced by `src/ZB.MOM.WW.ScadaBridge.Host/ZB.MOM.WW.ScadaBridge.Host.csproj:32-34`; `.Akka` also by `Communication`).
|
||||
- **Two composition roots**, both calling `app.MapZbHealth()` with defaults:
|
||||
central (`src/ZB.MOM.WW.ScadaBridge.Host/Program.cs:544-574` register, `:741` map) and
|
||||
site (`src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs:267-282` register, `Program.cs:861` map, on the
|
||||
HTTP/1.1 metrics listener, default `:8084`).
|
||||
- `CentralListenerIsolation.cs:14-15` admits `/healthz`, `/health/ready`, `/health/active` and `/metrics`
|
||||
on a dedicated metrics port and 404s everything else there.
|
||||
|
||||
`src/ZB.MOM.WW.ScadaBridge.Host/Program.cs`:
|
||||
## Per-endpoint contents
|
||||
|
||||
- `:114–117` — `builder.Services.AddHealthChecks()` followed by three `.AddCheck<T>()` calls
|
||||
(no tags, checked by name at the endpoint level):
|
||||
- `.AddCheck<DatabaseHealthCheck>("database")`
|
||||
- `.AddCheck<AkkaClusterHealthCheck>("akka-cluster")`
|
||||
- `.AddCheck<ActiveNodeHealthCheck>("active-node")`
|
||||
- `:131` — `builder.Services.AddSingleton<IActiveNodeGate, ActiveNodeGate>()` registers the
|
||||
production `IActiveNodeGate` implementation (Inbound API gating, not a health-check probe).
|
||||
- `:222–226` — `/health/ready` mapped with `Predicate = check => check.Name != "active-node"` and
|
||||
`ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse` (from `HealthChecks.UI.Client`).
|
||||
Excludes the active-node check so a healthy standby node reports ready.
|
||||
- `:229–233` — `/health/active` mapped with `Predicate = check => check.Name == "active-node"` and
|
||||
`ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse`. Active-node check only.
|
||||
| Endpoint | Central node | Site node |
|
||||
|---|---|---|
|
||||
| `/healthz` | none | none |
|
||||
| `/health/ready` | `database`, `akka-cluster`, `required-singletons` | `akka-cluster`, `localdb` |
|
||||
| `/health/active` | `database`, `active-node` | `active-node` |
|
||||
|
||||
No `/healthz` endpoint. Both mapped endpoints use `HealthChecks.UI.Client` JSON (not the default
|
||||
plain-text writer), which is a divergence from OtOpcUa.
|
||||
## Checks
|
||||
|
||||
## 2. Probes
|
||||
| Root | Name | Class | Tags | Probes | Failure |
|
||||
|---|---|---|---|---|---|
|
||||
| central | `database` | shared `DatabaseHealthCheck<ScadaBridgeDbContext>` (scoped context, no factory) | ready, active | `CanConnectAsync` | Unhealthy |
|
||||
| central | `akka-cluster` | shared `AkkaClusterHealthCheck`, policy **`Default`** | ready | `SelfMember.Status` | Up/Joining Healthy; Leaving/Exiting Degraded; else Unhealthy; Degraded during startup |
|
||||
| central | `required-singletons` | `RequiredSingletonsHealthCheck` (host) | ready | `Identify` (2 s, concurrent) on 5 singleton proxies (`notification-outbox`, `audit-log-ingest`, `site-call-audit`, `audit-log-purge`, `site-audit-reconciliation`) | Unhealthy if any unreachable or no ActorSystem |
|
||||
| central | `active-node` | shared `ActiveNodeHealthCheck`, unscoped | active | oldest `Up` member of the central cluster | Unhealthy on standby; Degraded while cluster not yet available |
|
||||
| site | `akka-cluster` | shared `AkkaClusterHealthCheck`, `Default` | ready | as above | as above |
|
||||
| site | `localdb` | `SiteLocalDbHealthCheck` (host) | ready | `SELECT 1` on the site LocalDb; replication state in `data` only | Unhealthy if unreachable |
|
||||
| site | `active-node` | `SitePairActiveNodeHealthCheck` (host) | active | `IClusterNodeProvider.SelfIsPrimary` → `ClusterActiveNode.SelfIsActive(cluster, siteRole)` | **Unhealthy** on standby **and while the ActorSystem is null** |
|
||||
|
||||
### DatabaseHealthCheck
|
||||
`src/ZB.MOM.WW.ScadaBridge.Host/Health/DatabaseHealthCheck.cs`:
|
||||
The `database` check on central's active tier is deliberate (review 01 [High]): a DB-dead active node drops
|
||||
out of Traefik rotation (`Program.cs:534-537`).
|
||||
|
||||
- `:11` — injects `ScadaBridgeDbContext` directly (not a factory)
|
||||
- `:33–43` — calls `_dbContext.Database.CanConnectAsync(cancellationToken)`:
|
||||
- Returns `true` → `HealthCheckResult.Healthy("Database connection is available.")` (`:34–35`)
|
||||
- Returns `false` → `HealthCheckResult.Unhealthy("Database connection failed.")` (`:36`)
|
||||
- Throws → `HealthCheckResult.Unhealthy("Database connection failed.", ex)` (`:40`)
|
||||
## Active-node gating
|
||||
|
||||
`CanConnectAsync` tests the connection layer only — it does not run any query or verify schema
|
||||
state. This is less strict than OtOpcUa's `Deployments` query but more transparent about failure
|
||||
cause (connection vs. schema). No `Degraded` path.
|
||||
- **Probes:** central `active-node` (shared check) and site `active-node` (thin adapter over the site's
|
||||
role-scoped `IClusterNodeProvider`). Both route through the shared `ClusterActiveNode` via
|
||||
`Communication.ClusterState.ActiveNodeEvaluator.SelfIsOldestUp` — the same rule as the S&F delivery
|
||||
gate and the heartbeat `IsActive` stamp.
|
||||
- **Request gate:** `ActiveNodeGate` (`Host/Health/ActiveNodeGate.cs`) registered as
|
||||
**`ZB.MOM.WW.ScadaBridge.InboundAPI.IActiveNodeGate`** (`Program.cs:614`), *not* the shared
|
||||
`ZB.MOM.WW.Health.IActiveNodeGate`. Consumed by `InboundApiEndpointFilter` (503 on standby),
|
||||
`InboundMethodWarmupService`, `AuditWriteMiddleware` and `FlowDebugRunner`. Its logic is
|
||||
`ClusterActivityEvaluator.SelfIsOldest(cluster)` → shared `ClusterActiveNode` — identical in behaviour to
|
||||
`AkkaActiveNodeGate`; only the interface type is local. Central only; sites register no gate.
|
||||
|
||||
### AkkaClusterHealthCheck
|
||||
`src/ZB.MOM.WW.ScadaBridge.Host/Health/AkkaClusterHealthCheck.cs`:
|
||||
## Other endpoints
|
||||
|
||||
- `:13` — injects `AkkaHostedService` (not `ActorSystem` directly)
|
||||
- `:33–50` — gets `_akkaService.ActorSystem`, guards on null → `Degraded("ActorSystem not yet
|
||||
available.")`, then reads `cluster.SelfMember.Status`:
|
||||
- `Up` or `Joining` → `Healthy($"Akka cluster member status: {status}")` (`:43`)
|
||||
- `Leaving` or `Exiting` → `Degraded($"Akka cluster member status: {status}")` (`:45`)
|
||||
- anything else (Removed, Down, WeaklyUp…) → `Unhealthy($"Akka cluster member status: {status}")` (`:47`)
|
||||
`MapZbMetrics()` on both roots; central gRPC control plane on its own h2c port.
|
||||
|
||||
Three-way status policy: Healthy / Degraded / Unhealthy. This is more granular than OtOpcUa's
|
||||
two-way policy (self-Up-or-not → Healthy/Degraded with no Unhealthy path).
|
||||
## vs SPEC
|
||||
|
||||
### ActiveNodeHealthCheck
|
||||
`src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeHealthCheck.cs`:
|
||||
|
||||
- `:13` — injects `AkkaHostedService`
|
||||
- `:29–44` — three-path logic:
|
||||
- `ActorSystem == null` → `Unhealthy("ActorSystem not yet available.")` (`:31`)
|
||||
- `SelfMember.Status != Up` → `Unhealthy($"Node not Up (status: ...)")` (`:37`)
|
||||
- `Up` AND `cluster.State.Leader == self.Address` → `Healthy("Active node (cluster leader).")` (`:41`)
|
||||
- `Up` but not leader → `Unhealthy("Standby node (not cluster leader).")` (`:43`)
|
||||
|
||||
No `Degraded` path — `ActiveNodeHealthCheck` uses `Unhealthy` for standby and non-Up states,
|
||||
which causes `/health/active` to return HTTP 503 on a standby. This is the intended behavior for
|
||||
Traefik active-node routing.
|
||||
|
||||
## 3. Tag / tier summary
|
||||
|
||||
ScadaBridge uses **name-based predicates** at the endpoint level rather than tags on the check
|
||||
registration. Tags are absent from all three `.AddCheck<T>()` calls.
|
||||
|
||||
| Probe | `/health/ready` | `/health/active` | `/healthz` |
|
||||
|---|---|---|---|
|
||||
| `DatabaseHealthCheck` | ✅ | — (excluded by name) | ⛔ absent |
|
||||
| `AkkaClusterHealthCheck` | ✅ | — (excluded by name) | ⛔ absent |
|
||||
| `ActiveNodeHealthCheck` | — (excluded by name) | ✅ | ⛔ absent |
|
||||
|
||||
`/healthz` is absent — there is no bare process liveness endpoint. Kubernetes or Traefik liveness
|
||||
probes must either use `/health/ready` or tolerate its 503-until-ready behavior.
|
||||
|
||||
## 4. IActiveNodeGate and Inbound API gating
|
||||
|
||||
`src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeGate.cs`:
|
||||
|
||||
- `:24` — `ActiveNodeGate` implements `IActiveNodeGate` (from the `InboundAPI` project)
|
||||
- `:40` — `IsActiveNode` property: returns `true` only when `_akkaService.ActorSystem != null`
|
||||
AND `cluster.SelfMember.Status == MemberStatus.Up` AND `cluster.State.Leader == self.Address`.
|
||||
Defaults to `false` safely during startup (`:45–46`).
|
||||
- `:131` in `Program.cs` — registered as a singleton. The `InboundApiEndpointFilter` consults this
|
||||
gate on every `/api/*` request and returns HTTP 503 on a standby node.
|
||||
|
||||
`ActiveNodeGate` mirrors the exact same logic as `ActiveNodeHealthCheck` — both check Up + leader.
|
||||
They are separate types serving two different concerns (the health endpoint and the API gate) but
|
||||
are not abstracted into a shared service; each reads cluster state independently.
|
||||
|
||||
`IActiveNodeGate` is the generalized seam the `ZB.MOM.WW.Health` core package lifts to the shared
|
||||
library.
|
||||
|
||||
## 5. HealthMonitoring domain pipeline (out of scope for shared library)
|
||||
|
||||
`src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/` is a separate project implementing a distributed
|
||||
health aggregation pipeline. It is **not ASP.NET Core health checks** and is **not in scope** for
|
||||
`ZB.MOM.WW.Health`.
|
||||
|
||||
Key types:
|
||||
- `SiteHealthCollector` — thread-safe singleton accumulating per-site error counters, connection
|
||||
metrics, and tag-read metrics. Populated by actors in the DCL layer.
|
||||
- `HealthReportSender` — a background service on site clusters that serializes `SiteHealthState`
|
||||
and ships it to the central cluster via Akka remoting at a configurable interval.
|
||||
- `CentralHealthReportLoop` — central-only background service that generates a synthetic
|
||||
`SiteHealthReport` for the central cluster itself (siteId `"$central"`) and feeds it into the
|
||||
central aggregator.
|
||||
- `CentralHealthAggregator` — a `BackgroundService` on the central cluster tracking the latest
|
||||
health report per site and detecting offline sites via heartbeat timeout. Exposes
|
||||
`GetAggregatedHealth()` to the Central UI's `/monitoring/health` endpoint.
|
||||
|
||||
This pipeline is domain-specific (multi-site ScadaBridge topology) and will remain per-project
|
||||
regardless of shared-library adoption.
|
||||
|
||||
## 6. Notable design choices
|
||||
|
||||
- **Name-based predicates vs. tags** — ScadaBridge uses `check.Name == "active-node"` predicate
|
||||
logic at the endpoint level. OtOpcUa uses tag membership (`c.Tags.Contains("ready")`). The tag
|
||||
approach is more composable (a probe can participate in multiple tiers), the name approach is
|
||||
more explicit. The shared `MapZbHealth` should use tags by default.
|
||||
- **`HealthChecks.UI.Client` response writer** — ScadaBridge uses the richer JSON response writer
|
||||
from the `AspNetCore.HealthChecks.UI.Client` package. OtOpcUa uses the default plain-text writer.
|
||||
The shared library's canonical response writer standardizes this.
|
||||
- **`ActiveNodeHealthCheck` returns `Unhealthy` for standby** — a standby is not *unhealthy* in the
|
||||
system sense; it is a deliberate routing discriminator. Using `Unhealthy` here ensures `/health/active`
|
||||
returns HTTP 503 (Traefik sees the node as down for active traffic). The naming is semantically
|
||||
imprecise but operationally correct.
|
||||
- **`IActiveNodeGate` + `ActiveNodeGate` duplication** — the gate and the health check implement the
|
||||
same logic independently. The shared library's `IActiveNodeGate` seam + `ActiveNodeHealthCheck`
|
||||
unify them: one backing service, two consumers.
|
||||
|
||||
---
|
||||
|
||||
## Adoption plan → `ZB.MOM.WW.Health`
|
||||
|
||||
**Replace with shared probes:**
|
||||
|
||||
- `AkkaClusterHealthCheck` → `ZB.MOM.WW.Health.Akka.AkkaClusterHealthCheck` using the **Default
|
||||
policy** (`Up`/`Joining`=Healthy, `Leaving`/`Exiting`=Degraded, else Unhealthy). ScadaBridge's
|
||||
existing three-way policy is the Default — no preset selection needed.
|
||||
- `ActiveNodeHealthCheck` → `ZB.MOM.WW.Health.Akka.ActiveNodeHealthCheck` with no role filter
|
||||
(role-less default: Up && leader = Healthy, else Unhealthy). The shared implementation also backs
|
||||
`IActiveNodeGate`, eliminating the duplicated leader-check logic between `ActiveNodeHealthCheck`
|
||||
and `ActiveNodeGate`.
|
||||
- `DatabaseHealthCheck` → `ZB.MOM.WW.Health.EntityFrameworkCore.DatabaseHealthCheck<ScadaBridgeDbContext>`
|
||||
using the default `CanConnectAsync` probe (ScadaBridge's existing behavior). No `ProbeQuery`
|
||||
delegate needed.
|
||||
- Replace the name-based predicates with tag-based predicates by adding tags at registration time:
|
||||
`"database"` and `"akka-cluster"` → `["ready"]`; `"active-node"` → `["active"]`. Then call
|
||||
`app.MapZbHealth()` instead of the two manual `MapHealthChecks` calls.
|
||||
- **Add `/healthz`** — `MapZbHealth()` maps the bare liveness tier automatically. ScadaBridge
|
||||
currently lacks this endpoint.
|
||||
- Switch `ResponseWriter` from `UIResponseWriter.WriteHealthCheckUIResponse` to the shared
|
||||
canonical writer (a convergence item — `HealthChecks.UI.Client` style lifted to the default in
|
||||
`ZB.MOM.WW.Health`).
|
||||
|
||||
**Keep bespoke:**
|
||||
|
||||
- `HealthMonitoring/` domain pipeline (`SiteHealthCollector`, `CentralHealthAggregator`, etc.) —
|
||||
entirely per-project, no shared-library equivalent.
|
||||
- `IActiveNodeGate` moves from the `InboundAPI` project to `ZB.MOM.WW.Health` (core package) on
|
||||
adoption. `InboundApiEndpointFilter` references the shared interface; `AkkaActiveNodeGate`
|
||||
(from `ZB.MOM.WW.Health.Akka`) becomes the singleton implementation registered in DI. The
|
||||
interface definition is no longer owned by the `InboundAPI` project.
|
||||
- The Central UI's `/monitoring/health` endpoint — powered by `CentralHealthAggregator`, not by
|
||||
ASP.NET health checks.
|
||||
- The comment at `Program.cs:217–221` explains the readiness design decision (standby nodes are
|
||||
ready; leadership is a separate concern). This intent is preserved by the tag-based approach.
|
||||
|
||||
**Adoption is a follow-on task** (tracked in `GAPS.md`), not part of the `ZB.MOM.WW.Health`
|
||||
library build. The library build delivers the shared implementations; adoption lands in the
|
||||
ScadaBridge repo as a separate commit once the nupkg is available.
|
||||
| SPEC | Status | Note |
|
||||
|---|---|---|
|
||||
| §1 three tiers via `MapZbHealth`, canonical tags | ✅ | both roots |
|
||||
| §1 nothing on `live` | ✅ | |
|
||||
| §2.1 `DatabaseHealthCheck<T>` | ✅ central | site uses a custom LocalDb check (no EF on sites) |
|
||||
| §2.2 / §7(b) Akka probe, Default policy | ✅ | |
|
||||
| §2.3 `ActiveNodeHealthCheck` | ✅ central; 🟡 site | site adapter is Unhealthy (not Degraded) during startup and carries no `selfAddress`/`activeNode` `data` |
|
||||
| §2.4 downstream probe | n/a | DCL connection health is reported through HealthMonitoring, not readiness |
|
||||
| §3 canonical JSON writer | ✅ | |
|
||||
| §4 / §7(e) `IActiveNodeGate` | 🟡 | same rule, local interface type (GAPS L2 deferral) |
|
||||
|
||||
Reference in New Issue
Block a user