Files

T

Joseph Doherty 9cbe65c844 docs(health): re-verify per-app probe coverage against the SPEC; refresh GAPS and current-state (scadaproj#11)

Audited the registration and mapping code of each app at its origin default branch: OtOpcUa
c904472a, MxAccessGateway de86f8b8, ScadaBridge c7d868c5, HistorianGateway 4d6a9ad (all on Health
0.3.0, all MapZbHealth with default paths: /healthz runs nothing, /health/ready and /health/active
by tag). GAPS.md now carries the verified per-endpoint coverage matrix and a SPEC-by-app table;
each current-state doc lists exactly what each endpoint runs, and HistorianGateway gets its first
current-state doc.

Recorded: mxaccessgw#206 (worker-executable + galaxy-sql readiness checks, on /health/ready) is
merged; historiangw#87 (session-pool check latches 503 until restart) is still open at 4d6a9ad.
Deferred P2 downstream probes are now per-app; ScadaBridge IActiveNodeGate unification stays
deferred (shared rule already used, only the interface is local).

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV

2026-11-09 14:00:44 -05:00

4.8 KiB

Raw Blame History

Health — current state: ScadaBridge

Verified 2026-11-09 at ScadaBridge@c7d868c5 (origin/main) for scadaproj#11. Read-only git grep / git show against the origin default branch; all paths relative to the repo root (~/Desktop/ScadaBridge). The pre-adoption state (verified 2026-06-01: name-predicate endpoints, no /healthz, HealthChecks.UI.Client writer) is gone; see git history of this file.

Out of scope, as before: the src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ domain-aggregation pipeline (site → central health reports over Akka) is not an ASP.NET health probe.

Packages: ZB.MOM.WW.Health, .Akka, .EntityFrameworkCore all 0.3.0 (Directory.Packages.props:97-99; referenced by src/ZB.MOM.WW.ScadaBridge.Host/ZB.MOM.WW.ScadaBridge.Host.csproj:32-34; .Akka also by Communication).
Two composition roots, both calling app.MapZbHealth() with defaults: central (src/ZB.MOM.WW.ScadaBridge.Host/Program.cs:544-574 register, :741 map) and site (src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs:267-282 register, Program.cs:861 map, on the HTTP/1.1 metrics listener, default :8084).
CentralListenerIsolation.cs:14-15 admits /healthz, /health/ready, /health/active and /metrics on a dedicated metrics port and 404s everything else there.

Per-endpoint contents

Endpoint	Central node	Site node
`/healthz`	none	none
`/health/ready`	`database`, `akka-cluster`, `required-singletons`	`akka-cluster`, `localdb`
`/health/active`	`database`, `active-node`	`active-node`

Checks

Root	Name	Class	Tags	Probes	Failure
central	`database`	shared `DatabaseHealthCheck<ScadaBridgeDbContext>` (scoped context, no factory)	ready, active	`CanConnectAsync`	Unhealthy
central	`akka-cluster`	shared `AkkaClusterHealthCheck`, policy `Default`	ready	`SelfMember.Status`	Up/Joining Healthy; Leaving/Exiting Degraded; else Unhealthy; Degraded during startup
central	`required-singletons`	`RequiredSingletonsHealthCheck` (host)	ready	`Identify` (2 s, concurrent) on 5 singleton proxies (`notification-outbox`, `audit-log-ingest`, `site-call-audit`, `audit-log-purge`, `site-audit-reconciliation`)	Unhealthy if any unreachable or no ActorSystem
central	`active-node`	shared `ActiveNodeHealthCheck`, unscoped	active	oldest `Up` member of the central cluster	Unhealthy on standby; Degraded while cluster not yet available
site	`akka-cluster`	shared `AkkaClusterHealthCheck`, `Default`	ready	as above	as above
site	`localdb`	`SiteLocalDbHealthCheck` (host)	ready	`SELECT 1` on the site LocalDb; replication state in `data` only	Unhealthy if unreachable
site	`active-node`	`SitePairActiveNodeHealthCheck` (host)	active	`IClusterNodeProvider.SelfIsPrimary` → `ClusterActiveNode.SelfIsActive(cluster, siteRole)`	Unhealthy on standby and while the ActorSystem is null

The database check on central's active tier is deliberate (review 01 [High]): a DB-dead active node drops out of Traefik rotation (Program.cs:534-537).

Active-node gating

Probes: central active-node (shared check) and site active-node (thin adapter over the site's role-scoped IClusterNodeProvider). Both route through the shared ClusterActiveNode via Communication.ClusterState.ActiveNodeEvaluator.SelfIsOldestUp — the same rule as the S&F delivery gate and the heartbeat IsActive stamp.
Request gate: ActiveNodeGate (Host/Health/ActiveNodeGate.cs) registered as ZB.MOM.WW.ScadaBridge.InboundAPI.IActiveNodeGate (Program.cs:614), not the shared ZB.MOM.WW.Health.IActiveNodeGate. Consumed by InboundApiEndpointFilter (503 on standby), InboundMethodWarmupService, AuditWriteMiddleware and FlowDebugRunner. Its logic is ClusterActivityEvaluator.SelfIsOldest(cluster) → shared ClusterActiveNode — identical in behaviour to AkkaActiveNodeGate; only the interface type is local. Central only; sites register no gate.

Other endpoints

MapZbMetrics() on both roots; central gRPC control plane on its own h2c port.

vs SPEC

SPEC	Status	Note
§1 three tiers via `MapZbHealth`, canonical tags	✅	both roots
§1 nothing on `live`	✅
§2.1 `DatabaseHealthCheck<T>`	✅ central	site uses a custom LocalDb check (no EF on sites)
§2.2 / §7(b) Akka probe, Default policy	✅
§2.3 `ActiveNodeHealthCheck`	✅ central; 🟡 site	site adapter is Unhealthy (not Degraded) during startup and carries no `selfAddress`/`activeNode` `data`
§2.4 downstream probe	n/a	DCL connection health is reported through HealthMonitoring, not readiness
§3 canonical JSON writer	✅
§4 / §7(e) `IActiveNodeGate`	🟡	same rule, local interface type (GAPS L2 deferral)

4.8 KiB Raw Blame History