Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
4.9 KiB
Health — current state: HistorianGateway
Verified 2026-11-09 at historiangw@4d6a9ad (origin/main) for scadaproj#11. Read-only git grep /
git show against the origin default branch; all paths relative to the repo root
(~/Desktop/HistorianGateway). First current-state doc for this app (it postdates the original
three-project survey).
- Package:
ZB.MOM.WW.Health0.3.0, core only (src/ZB.MOM.WW.HistorianGateway.Server/ZB.MOM.WW.HistorianGateway.Server.csproj:23). - Registration: inline block in
src/ZB.MOM.WW.HistorianGateway.Server/Program.cs:256-329. - Mapping:
app.MapZbHealth()with default paths/writer (Program.cs:417). - Not
GrpcDependencyHealthCheck: earlier scadaproj docs said HistorianGateway used it. It does not onmain— the gRPC-only Historian client owns its own channel, so the Historian probe goes through the connection pool instead (comment atProgram.cs:265-268).
Per-endpoint contents
| Endpoint | Checks run | Can return 503 when |
|---|---|---|
/healthz (live) |
none | never |
/health/ready |
historian-connection, galaxy-sql, + store-forward-drain (if StoreForward:Enabled), + historian-session-pool / historian-event-session-pool (if each pool is enabled — both default true) |
Historian probe fails/> 3 s; Galaxy SQL fails/> CommandTimeoutSeconds; a session-pool factory has ≥ 3 consecutive failures |
/health/active |
none | never — an empty report is Healthy, so every node answers 200 |
Checks
Files under src/ZB.MOM.WW.HistorianGateway.Server/Health/. All tagged ready, failureStatus: null.
| Name | Class | Probes | Failure |
|---|---|---|---|
historian-connection |
HistorianConnectionHealthCheck |
IHistorianConnectionPool.AcquireAsync + client.ProbeAsync, 3 s cap; reports a faulted client back to the pool |
Unhealthy |
galaxy-sql |
SqlConnectionHealthCheck (raw SqlConnection + SELECT 1; no EF) |
Galaxy Repository DB from the top-level Galaxy section, bounded by its command timeout |
Unhealthy |
store-forward-drain |
StoreForwardDrainHealthCheck |
IHistorianStoreForward.GetStatusAsync; backing up = Storing || ErrorOccurred |
Degraded only (never 503) |
historian-session-pool |
SessionPoolHealthCheck("read", …) |
HistorianSessionPool.Snapshot.ConsecutiveFactoryFailures |
0 → Healthy; 1–2 → Degraded; ≥ 3 → Unhealthy |
historian-event-session-pool |
SessionPoolHealthCheck("event", …) |
same over HistorianEventSessionPool |
same |
Finding: the session-pool check latched Unhealthy — historiangw#87 (fixed on main 5de3dd9)
Fixed 2026-11-09 on
origin/main5de3dd9(after this audit's4d6a9ad): caller cancellation no longer counts, and while the failure count is non-zero each keepalive tick opens one probe session (success resets it). Deployed boxes still rund8d890cuntil historiangw#78. The analysis below describes the pre-fix code.
historiangw#87 (Server-007, High/P1): _consecutiveFactoryFailures is reset only by a successful
factory call, and the factory only runs when a real RPC needs a new lease (the keepalive sweep never
opens sessions). After a Historian outage of ≥ 3 failed leases, /health/ready goes 503, a k8s probe
removes the pod, no RPC arrives, and the check never recovers until restart; three clients whose
short deadlines cancel a cold handshake also trip it (the catch counts caller cancellation). On the NSSM
wonder boxes it shows as a false Unhealthy on Overview.
Still present on origin/main 4d6a9ad: HistorianSessionPool.cs:264-273 and
HistorianEventSessionPool.cs:254-262 increment on any throw and reset only on success; nothing probes
or decays the streak; both pools default to Enabled = true
(Configuration/HistorianSessionPoolOptions.cs:24, EventSessionPoolOptions.cs:24), so the latching
check is on the ready tier in the default configuration.
Active-node gating
None, by design: a single-process sidecar per host with no cluster. No IActiveNodeGate, no
RequireActiveNode(). Redundancy options concern the Historian servers it talks to, not node
election.
Other endpoints
MapZbMetrics(), MapZbGalaxyRepository(), the dashboard (its HealthPage renders the same
ready-tier results), and the four Historian gRPC services.
vs SPEC
| SPEC | Status | Note |
|---|---|---|
§1 three tiers via MapZbHealth, canonical tags |
✅ | /health/active is empty (always 200) |
§1 nothing on live |
✅ | |
| §2.1 DB probe | 🟡 custom | SqlConnectionHealthCheck (no EF context to wrap) |
| §2.4 downstream dependency probe | ✅ (custom) | Historian via the pool, not GrpcDependencyHealthCheck |
| Ready = "can this node serve its dependencies?" | 🟡 | correct while dependencies fail; did not recover after a session-pool outage (#87 — fixed on main 5de3dd9, not yet deployed) |
| §3 canonical JSON writer | ✅ | |
§4 IActiveNodeGate |
n/a |