Files
scadaproj/components/health/current-state/historiangateway/CURRENT-STATE.md
T

4.9 KiB
Raw Blame History

Health — current state: HistorianGateway

Verified 2026-11-09 at historiangw@4d6a9ad (origin/main) for scadaproj#11. Read-only git grep / git show against the origin default branch; all paths relative to the repo root (~/Desktop/HistorianGateway). First current-state doc for this app (it postdates the original three-project survey).

  • Package: ZB.MOM.WW.Health 0.3.0, core only (src/ZB.MOM.WW.HistorianGateway.Server/ZB.MOM.WW.HistorianGateway.Server.csproj:23).
  • Registration: inline block in src/ZB.MOM.WW.HistorianGateway.Server/Program.cs:256-329.
  • Mapping: app.MapZbHealth() with default paths/writer (Program.cs:417).
  • Not GrpcDependencyHealthCheck: earlier scadaproj docs said HistorianGateway used it. It does not on main — the gRPC-only Historian client owns its own channel, so the Historian probe goes through the connection pool instead (comment at Program.cs:265-268).

Per-endpoint contents

Endpoint Checks run Can return 503 when
/healthz (live) none never
/health/ready historian-connection, galaxy-sql, + store-forward-drain (if StoreForward:Enabled), + historian-session-pool / historian-event-session-pool (if each pool is enabled — both default true) Historian probe fails/> 3 s; Galaxy SQL fails/> CommandTimeoutSeconds; a session-pool factory has ≥ 3 consecutive failures
/health/active none never — an empty report is Healthy, so every node answers 200

Checks

Files under src/ZB.MOM.WW.HistorianGateway.Server/Health/. All tagged ready, failureStatus: null.

Name Class Probes Failure
historian-connection HistorianConnectionHealthCheck IHistorianConnectionPool.AcquireAsync + client.ProbeAsync, 3 s cap; reports a faulted client back to the pool Unhealthy
galaxy-sql SqlConnectionHealthCheck (raw SqlConnection + SELECT 1; no EF) Galaxy Repository DB from the top-level Galaxy section, bounded by its command timeout Unhealthy
store-forward-drain StoreForwardDrainHealthCheck IHistorianStoreForward.GetStatusAsync; backing up = Storing || ErrorOccurred Degraded only (never 503)
historian-session-pool SessionPoolHealthCheck("read", …) HistorianSessionPool.Snapshot.ConsecutiveFactoryFailures 0 → Healthy; 1–2 → Degraded; ≥ 3 → Unhealthy
historian-event-session-pool SessionPoolHealthCheck("event", …) same over HistorianEventSessionPool same

Finding: the session-pool check latched Unhealthy — historiangw#87 (fixed on main 5de3dd9)

Fixed 2026-11-09 on origin/main 5de3dd9 (after this audit's 4d6a9ad): caller cancellation no longer counts, and while the failure count is non-zero each keepalive tick opens one probe session (success resets it). Deployed boxes still run d8d890c until historiangw#78. The analysis below describes the pre-fix code.

historiangw#87 (Server-007, High/P1): _consecutiveFactoryFailures is reset only by a successful factory call, and the factory only runs when a real RPC needs a new lease (the keepalive sweep never opens sessions). After a Historian outage of ≥ 3 failed leases, /health/ready goes 503, a k8s probe removes the pod, no RPC arrives, and the check never recovers until restart; three clients whose short deadlines cancel a cold handshake also trip it (the catch counts caller cancellation). On the NSSM wonder boxes it shows as a false Unhealthy on Overview.

Still present on origin/main 4d6a9ad: HistorianSessionPool.cs:264-273 and HistorianEventSessionPool.cs:254-262 increment on any throw and reset only on success; nothing probes or decays the streak; both pools default to Enabled = true (Configuration/HistorianSessionPoolOptions.cs:24, EventSessionPoolOptions.cs:24), so the latching check is on the ready tier in the default configuration.

Active-node gating

None, by design: a single-process sidecar per host with no cluster. No IActiveNodeGate, no RequireActiveNode(). Redundancy options concern the Historian servers it talks to, not node election.

Other endpoints

MapZbMetrics(), MapZbGalaxyRepository(), the dashboard (its HealthPage renders the same ready-tier results), and the four Historian gRPC services.

vs SPEC

SPEC Status Note
§1 three tiers via MapZbHealth, canonical tags ✅ /health/active is empty (always 200)
§1 nothing on live ✅
§2.1 DB probe 🟡 custom SqlConnectionHealthCheck (no EF context to wrap)
§2.4 downstream dependency probe ✅ (custom) Historian via the pool, not GrpcDependencyHealthCheck
Ready = "can this node serve its dependencies?" 🟡 correct while dependencies fail; did not recover after a session-pool outage (#87 — fixed on main 5de3dd9, not yet deployed)
§3 canonical JSON writer ✅
§4 IActiveNodeGate n/a