scadaproj/components/health/GAPS.md

# Health — gaps vs spec

> **Status lives in Gitea.** Open work is tracked as issues on the owning repo's tracker
> (`gitea.dohertylan.com/dohertj2/<repo>`); this file is a design record of where each app stands against
> [`spec/SPEC.md`](spec/SPEC.md). Do not add status rows here — file an issue and reference it.

**Re-verified 2026-11-09 (scadaproj#11)** against each repo's origin default branch, by reading the
registration and mapping code (not just the `PackageReference`):

| App | Audited at | Health pin | Detail |
|---|---|---|---|
| OtOpcUa | `origin/master` `c904472a` | 0.3.0 (core + `.Akka` + `.EntityFrameworkCore`) | [current-state/otopcua](current-state/otopcua/CURRENT-STATE.md) |
| MxAccessGateway | `origin/main` `de86f8b8` | 0.3.0 (core) | [current-state/mxaccessgw](current-state/mxaccessgw/CURRENT-STATE.md) |
| ScadaBridge | `origin/main` `c7d868c5` | 0.3.0 (core + `.Akka` + `.EntityFrameworkCore`) | [current-state/scadabridge](current-state/scadabridge/CURRENT-STATE.md) |
| HistorianGateway | `origin/main` `4d6a9ad` | 0.3.0 (core) | [current-state/historiangateway](current-state/historiangateway/CURRENT-STATE.md) |

All four call `MapZbHealth()` with default paths and the canonical writer, so every app serves
`/healthz` (no checks), `/health/ready` (tag `ready`) and `/health/active` (tag `active`), anonymously.
No app tags anything `live`.

## Coverage matrix (what each endpoint runs)

| App / node | `/health/ready` | `/health/active` | Active-node rule |
|---|---|---|---|
| OtOpcUa admin-only | `configdb`, `akka` | `configdb`, `akka`, `cluster-primary`, `localdb-replication` | shared `ActiveNodeHealthCheck`, roles `[admin, driver]` |
| OtOpcUa fused admin+driver | `opcua-endpoint`, `configdb`, `akka` | `configdb`, `akka`, `cluster-primary`, `localdb-replication` | same |
| OtOpcUa driver-only | `opcua-endpoint`, `akka` | `opcua-endpoint`, `akka`, `cluster-primary`, `localdb-replication` | same |
| ScadaBridge central | `database`, `akka-cluster`, `required-singletons` | `database`, `active-node` | shared `ActiveNodeHealthCheck`, unscoped |
| ScadaBridge site | `akka-cluster`, `localdb` | `active-node` | `SitePairActiveNodeHealthCheck` → shared `ClusterActiveNode` (site role) |
| MxAccessGateway | `auth-store`, `worker-executable`, `galaxy-sql` | `mxaccess-sessions` | none (singleton, no cluster) |
| HistorianGateway | `historian-connection`, `galaxy-sql`, `store-forward-drain`*, `historian-session-pool`*, `historian-event-session-pool`* | *(empty — always 200)* | none (singleton, no cluster) |

\* registered only when the feature is enabled; both session pools default to enabled.

Checks that can only return **Degraded** (so never cause a 503): OtOpcUa `akka` (`OtOpcUaCompat`) and
`localdb-replication`; HistorianGateway `store-forward-drain`; the shared Akka checks during startup.

## Spec requirements vs apps

✅ covered · 🟡 partial / deliberate deviation · ⛔ missing · n/a not applicable

| SPEC requirement | OtOpcUa | MxAccessGateway | ScadaBridge | HistorianGateway |
|---|---|---|---|---|
| §1/§5 three tiers via `MapZbHealth`, canonical tags | ✅ | ✅ | ✅ (central + site) | ✅ |
| §1 nothing tagged `live` | ✅ | ✅ (test-asserted) | ✅ | ✅ |
| §2.1 DB probe (`DatabaseHealthCheck<T>` where EF exists) | ✅ `ProbeQuery` | n/a (custom SQLite + Galaxy SQL) | ✅ central; site custom LocalDb | n/a (custom `SqlConnectionHealthCheck`) |
| §2.2 / §7(b) Akka probe, **Default** policy | 🟡 `OtOpcUaCompat` | n/a | ✅ | n/a |
| §2.3 active tier = oldest `Up` member | ✅ | 🟡 tier used for session health | ✅ central; 🟡 site adapter | 🟡 tier empty |
| §2.3 startup-safe Degraded + `activeNode` data | ✅ | n/a | ✅ central; ⛔ site | n/a |
| §2.4 downstream dependency probe | ⛔ (see P2) | 🟡 presence + Galaxy SQL, not worker IPC | n/a | ✅ custom (Historian via pool) |
| §3 canonical JSON writer | ✅ | ✅ | ✅ | ✅ |
| §4 / §7(e) `IActiveNodeGate` | n/a (Traefik routes by probe) | n/a | 🟡 local interface, shared rule | n/a |

## Closed

- **T1/T2/T3, P1, P3, W1** (pre-adoption: mxgw had no tiers or probes and a bespoke `/health/live`;
  ScadaBridge lacked `/healthz`; per-app writers) — closed by the 2026-06-01 adoption; confirmed on all
  four tips above. No bespoke health endpoint remains anywhere.
- **L1/L2 active-node** — closed properly at Health `0.3.0`. They first shipped leader-based (address-
  ordered) selection, which diverges from age-ordered singleton placement after any restart; both Akka
  apps hit it live and reverted to private copies until `0.3.0` rebuilt `ActiveNodeHealthCheck` and
  `AkkaActiveNodeGate` on `ClusterActiveNode` (oldest `Up` member). Both now use the shared rule; their
  private copies are deleted. The library pins the only fixture that tells the two orderings apart (a
  two-node cluster whose oldest member is not the lowest-addressed one).
- **D1/D2 DB probe technique** — shared `DatabaseHealthCheck<T>` covers both (OtOpcUa `ProbeQuery`,
  ScadaBridge `CanConnectAsync` via self-scoping; the `IDbContextFactory` switch was dropped as
  unnecessary).
- **mxaccessgw#206** (closed 2026-11-09, `8c21678c`) — `worker-executable` and bounded `galaxy-sql`
  readiness checks, both `ready` only; sessions stay on `active`. No LDAP check (out of scope by decision).

## Remaining gaps

- **A1 — OtOpcUa still on `AkkaClusterStatusPolicy.OtOpcUaCompat`.** Its `akka` check can only report
  Degraded, so a node that has left or been downed never fails readiness on cluster grounds. SPEC §2.2 /
  §7(b) name `Default` as the convergence target. Needs a lmxopcua issue if convergence is still wanted.
- **ScadaBridge site `active-node` adapter** (`SitePairActiveNodeHealthCheck`) returns Unhealthy while the
  ActorSystem is still null (SPEC §2.3 says Degraded during startup) and emits no
  `selfAddress`/`activeRole`/`activeNode` `data`, so the Overview dashboard cannot tell from a site standby
  which node is active. Same rule as the shared check; only the contract around it differs. Needs a
  ScadaBridge issue if the site tier should match central's (registering the shared
  `ActiveNodeHealthCheck` with the site role would close both).
- **HistorianGateway session-pool readiness latch — historiangw#87 (FIXED on main `5de3dd9`, 2026-11-09; deployed boxes still run `d8d890c`).** The
  `historian-session-pool` / `historian-event-session-pool` checks go Unhealthy after 3 consecutive factory
  failures and reset only on a factory success, which needs an RPC, which a 503-ed pod never receives;
  caller cancellation also counted. Fixed on main `5de3dd9`: caller cancellation no longer counts and each keepalive tick probes one session while the count is non-zero; still present on deployed builds until historiangw#78 redeploys. Detail:
  [current-state/historiangateway](current-state/historiangateway/CURRENT-STATE.md).
- **SPEC text was stale on §2.4 - corrected (scadaproj#28).** It named OtOpcUa and MxGateway as `GrpcDependencyHealthCheck` users;
  **no app uses it** (HistorianGateway, once cited as a user, now probes the Historian through its
  connection pool). Decision: **keep** the check (a breaking public-API removal for no gain; small, unit-tested,
  a ready building block if a gRPC dependency ever needs a readiness probe) but document it as having no current
  consumers. SPEC §2.4/§6/§7, both READMEs and the shared-contract doc now say so.

## Deferred items — status from code (2026-11-09)

- **Downstream dependency probes (P2).** Re-scoped, not adopted as `GrpcDependencyHealthCheck` anywhere:
  - *MxAccessGateway:* worker IPC is named pipes and workers exist only per session, so there is no
    channel to probe at idle. mxaccessgw#206 closed the practical part (worker binary present, Galaxy SQL
    reachable). A live-worker probe is not planned.
  - *HistorianGateway:* covered by its own `historian-connection` and `galaxy-sql` checks.
  - *OtOpcUa:* still **no** probe of mxaccessgw or any driver dependency — gateway channels are created
    per driver from DB config, and OtOpcUa's availability guarantee (serve last-known-good when a source
    is down) argues against failing readiness on a driver dependency. Treat as **by design**; if a
    gateway-down signal is wanted, it belongs on a Degraded-only check or the driver-status surface, and
    needs a lmxopcua issue.
- **ScadaBridge `IActiveNodeGate` unification (L2 follow-on).** Still deferred, low value: ScadaBridge
  registers `ActiveNodeGate` as its own `ZB.MOM.WW.ScadaBridge.InboundAPI.IActiveNodeGate`
  (`Program.cs:614`), consumed by `InboundApiEndpointFilter`, `InboundMethodWarmupService`,
  `AuditWriteMiddleware` and `FlowDebugRunner`. The behaviour already converged: the gate calls
  `ClusterActivityEvaluator.SelfIsOldest` → `ActiveNodeEvaluator.SelfIsOldestUp` → shared
  `ClusterActiveNode.SelfIsActive`, the same rule as `AkkaActiveNodeGate` and the `active-node` probe, so
  the gate and the probe cannot disagree. Only the interface type is local. Swapping to the shared
  interface would touch the InboundAPI project (which would then reference `ZB.MOM.WW.Health`) for no
  behaviour change.

## Behaviour notes (kept for ops)

- **Extra probes on the active tier.** ScadaBridge central (`database`) and OtOpcUa admin nodes
  (`configdb`) put their DB check on `/health/active` deliberately: a DB-dead active node leaves Traefik
  rotation. Side effect: during a config-DB outage **both** nodes answer 503 on `/health/active`, so
  Traefik has no backend for the routed UI until the DB returns.
- **Startup window.** The shared `ActiveNodeHealthCheck` returns Degraded (200) while the ActorSystem or
  cluster is not yet available (accepted 2026-06-01 for ScadaBridge central). The ScadaBridge site
  adapter returns 503 in the same window (see Remaining gaps).
- **Singleton apps answer 200 on `/health/active` on every node.** MxAccessGateway (unless all sessions
  are faulted) and HistorianGateway (always). The QA pair runs both on vq03 and vq04 as independent
  singletons, so this is correct, but `/health/active` cannot be used to pick one of them.