scadaproj/components/health/GAPS.md

Health — gaps vs spec

Status lives in Gitea. Open work is tracked as issues on the owning repo's tracker (gitea.dohertylan.com/dohertj2/<repo>); this file is a design record of where each app stands against spec/SPEC.md. Do not add status rows here — file an issue and reference it.

Re-verified 2026-11-09 (scadaproj#11) against each repo's origin default branch, by reading the registration and mapping code (not just the PackageReference):

App	Audited at	Health pin	Detail
OtOpcUa	origin/master c904472a	0.3.0 (core + .Akka + .EntityFrameworkCore)	current-state/otopcua
MxAccessGateway	origin/main de86f8b8	0.3.0 (core)	current-state/mxaccessgw
ScadaBridge	origin/main c7d868c5	0.3.0 (core + .Akka + .EntityFrameworkCore)	current-state/scadabridge
HistorianGateway	origin/main 4d6a9ad	0.3.0 (core)	current-state/historiangateway

All four call MapZbHealth() with default paths and the canonical writer, so every app serves /healthz (no checks), /health/ready (tag ready) and /health/active (tag active), anonymously. No app tags anything live.

Coverage matrix (what each endpoint runs)

App / node	/health/ready	/health/active	Active-node rule
OtOpcUa admin-only	configdb, akka	configdb, akka, cluster-primary, localdb-replication	shared ActiveNodeHealthCheck, roles [admin, driver]
OtOpcUa fused admin+driver	opcua-endpoint, configdb, akka	configdb, akka, cluster-primary, localdb-replication	same
OtOpcUa driver-only	opcua-endpoint, akka	opcua-endpoint, akka, cluster-primary, localdb-replication	same
ScadaBridge central	database, akka-cluster, required-singletons	database, active-node	shared ActiveNodeHealthCheck, unscoped
ScadaBridge site	akka-cluster, localdb	active-node	SitePairActiveNodeHealthCheck → shared ClusterActiveNode (site role)
MxAccessGateway	auth-store, worker-executable, galaxy-sql	mxaccess-sessions	none (singleton, no cluster)
HistorianGateway	historian-connection, galaxy-sql, store-forward-drain, historian-session-pool, historian-event-session-pool*	(empty — always 200)	none (singleton, no cluster)

* registered only when the feature is enabled; both session pools default to enabled.

Checks that can only return Degraded (so never cause a 503): OtOpcUa akka (OtOpcUaCompat) and localdb-replication; HistorianGateway store-forward-drain; the shared Akka checks during startup.

Spec requirements vs apps

✅ covered · 🟡 partial / deliberate deviation · ⛔ missing · n/a not applicable

SPEC requirement	OtOpcUa	MxAccessGateway	ScadaBridge	HistorianGateway
§1/§5 three tiers via MapZbHealth, canonical tags	✅	✅	✅ (central + site)	✅
§1 nothing tagged live	✅	✅ (test-asserted)	✅	✅
§2.1 DB probe (DatabaseHealthCheck<T> where EF exists)	✅ ProbeQuery	n/a (custom SQLite + Galaxy SQL)	✅ central; site custom LocalDb	n/a (custom SqlConnectionHealthCheck)
§2.2 / §7(b) Akka probe, Default policy	🟡 OtOpcUaCompat	n/a	✅	n/a
§2.3 active tier = oldest Up member	✅	🟡 tier used for session health	✅ central; 🟡 site adapter	🟡 tier empty
§2.3 startup-safe Degraded + activeNode data	✅	n/a	✅ central; ⛔ site	n/a
§2.4 downstream dependency probe	⛔ (see P2)	🟡 presence + Galaxy SQL, not worker IPC	n/a	✅ custom (Historian via pool)
§3 canonical JSON writer	✅	✅	✅	✅
§4 / §7(e) IActiveNodeGate	n/a (Traefik routes by probe)	n/a	🟡 local interface, shared rule	n/a

Closed

• T1/T2/T3, P1, P3, W1 (pre-adoption: mxgw had no tiers or probes and a bespoke /health/live; ScadaBridge lacked /healthz; per-app writers) — closed by the 2026-06-01 adoption; confirmed on all four tips above. No bespoke health endpoint remains anywhere.
• L1/L2 active-node — closed properly at Health 0.3.0. They first shipped leader-based (address- ordered) selection, which diverges from age-ordered singleton placement after any restart; both Akka apps hit it live and reverted to private copies until 0.3.0 rebuilt ActiveNodeHealthCheck and AkkaActiveNodeGate on ClusterActiveNode (oldest Up member). Both now use the shared rule; their private copies are deleted. The library pins the only fixture that tells the two orderings apart (a two-node cluster whose oldest member is not the lowest-addressed one).
• D1/D2 DB probe technique — shared DatabaseHealthCheck<T> covers both (OtOpcUa ProbeQuery, ScadaBridge CanConnectAsync via self-scoping; the IDbContextFactory switch was dropped as unnecessary).
• mxaccessgw#206 (closed 2026-11-09, 8c21678c) — worker-executable and bounded galaxy-sql readiness checks, both ready only; sessions stay on active. No LDAP check (out of scope by decision).

Remaining gaps

• A1 — OtOpcUa still on AkkaClusterStatusPolicy.OtOpcUaCompat. Its akka check can only report Degraded, so a node that has left or been downed never fails readiness on cluster grounds. SPEC §2.2 / §7(b) name Default as the convergence target. Needs a lmxopcua issue if convergence is still wanted.
• ScadaBridge site active-node adapter (SitePairActiveNodeHealthCheck) returns Unhealthy while the ActorSystem is still null (SPEC §2.3 says Degraded during startup) and emits no selfAddress/activeRole/activeNode data, so the Overview dashboard cannot tell from a site standby which node is active. Same rule as the shared check; only the contract around it differs. Needs a ScadaBridge issue if the site tier should match central's (registering the shared ActiveNodeHealthCheck with the site role would close both).
• HistorianGateway session-pool readiness latch — historiangw#87 (FIXED on main 5de3dd9, 2026-11-09; deployed boxes still run d8d890c). The historian-session-pool / historian-event-session-pool checks go Unhealthy after 3 consecutive factory failures and reset only on a factory success, which needs an RPC, which a 503-ed pod never receives; caller cancellation also counted. Fixed on main 5de3dd9: caller cancellation no longer counts and each keepalive tick probes one session while the count is non-zero; still present on deployed builds until historiangw#78 redeploys. Detail: current-state/historiangateway.
• SPEC text was stale on §2.4 - corrected (scadaproj#28). It named OtOpcUa and MxGateway as GrpcDependencyHealthCheck users; no app uses it (HistorianGateway, once cited as a user, now probes the Historian through its connection pool). Decision: keep the check (a breaking public-API removal for no gain; small, unit-tested, a ready building block if a gRPC dependency ever needs a readiness probe) but document it as having no current consumers. SPEC §2.4/§6/§7, both READMEs and the shared-contract doc now say so.

Deferred items — status from code (2026-11-09)

• Downstream dependency probes (P2). Re-scoped, not adopted as GrpcDependencyHealthCheck anywhere:
  • MxAccessGateway: worker IPC is named pipes and workers exist only per session, so there is no channel to probe at idle. mxaccessgw#206 closed the practical part (worker binary present, Galaxy SQL reachable). A live-worker probe is not planned.
  • HistorianGateway: covered by its own historian-connection and galaxy-sql checks.
  • OtOpcUa: still no probe of mxaccessgw or any driver dependency — gateway channels are created per driver from DB config, and OtOpcUa's availability guarantee (serve last-known-good when a source is down) argues against failing readiness on a driver dependency. Treat as by design; if a gateway-down signal is wanted, it belongs on a Degraded-only check or the driver-status surface, and needs a lmxopcua issue.
• ScadaBridge IActiveNodeGate unification (L2 follow-on). Still deferred, low value: ScadaBridge registers ActiveNodeGate as its own ZB.MOM.WW.ScadaBridge.InboundAPI.IActiveNodeGate (Program.cs:614), consumed by InboundApiEndpointFilter, InboundMethodWarmupService, AuditWriteMiddleware and FlowDebugRunner. The behaviour already converged: the gate calls ClusterActivityEvaluator.SelfIsOldest → ActiveNodeEvaluator.SelfIsOldestUp → shared ClusterActiveNode.SelfIsActive, the same rule as AkkaActiveNodeGate and the active-node probe, so the gate and the probe cannot disagree. Only the interface type is local. Swapping to the shared interface would touch the InboundAPI project (which would then reference ZB.MOM.WW.Health) for no behaviour change.

Behaviour notes (kept for ops)

• Extra probes on the active tier. ScadaBridge central (database) and OtOpcUa admin nodes (configdb) put their DB check on /health/active deliberately: a DB-dead active node leaves Traefik rotation. Side effect: during a config-DB outage both nodes answer 503 on /health/active, so Traefik has no backend for the routed UI until the DB returns.
• Startup window. The shared ActiveNodeHealthCheck returns Degraded (200) while the ActorSystem or cluster is not yet available (accepted 2026-06-01 for ScadaBridge central). The ScadaBridge site adapter returns 503 in the same window (see Remaining gaps).
• Singleton apps answer 200 on /health/active on every node. MxAccessGateway (unless all sessions are faulted) and HistorianGateway (always). The QA pair runs both on vq03 and vq04 as independent singletons, so this is correct, but /health/active cannot be used to pick one of them.