# Health — current state: HistorianGateway **Verified 2026-11-09 at `historiangw@4d6a9ad` (`origin/main`)** for scadaproj#11. Read-only `git grep` / `git show` against the origin default branch; all paths relative to the repo root (`~/Desktop/HistorianGateway`). First current-state doc for this app (it postdates the original three-project survey). - **Package:** `ZB.MOM.WW.Health` **0.3.0**, core only (`src/ZB.MOM.WW.HistorianGateway.Server/ZB.MOM.WW.HistorianGateway.Server.csproj:23`). - **Registration:** inline block in `src/ZB.MOM.WW.HistorianGateway.Server/Program.cs:256-329`. - **Mapping:** `app.MapZbHealth()` with default paths/writer (`Program.cs:417`). - **Not** `GrpcDependencyHealthCheck`: earlier scadaproj docs said HistorianGateway used it. It does not on `main` — the gRPC-only Historian client owns its own channel, so the Historian probe goes through the connection pool instead (comment at `Program.cs:265-268`). ## Per-endpoint contents | Endpoint | Checks run | Can return 503 when | |---|---|---| | `/healthz` (live) | none | never | | `/health/ready` | `historian-connection`, `galaxy-sql`, + `store-forward-drain` (if `StoreForward:Enabled`), + `historian-session-pool` / `historian-event-session-pool` (if each pool is enabled — **both default `true`**) | Historian probe fails/> 3 s; Galaxy SQL fails/> `CommandTimeoutSeconds`; a session-pool factory has ≥ 3 consecutive failures | | `/health/active` | **none** | never — an empty report is Healthy, so every node answers 200 | ## Checks Files under `src/ZB.MOM.WW.HistorianGateway.Server/Health/`. All tagged `ready`, `failureStatus: null`. | Name | Class | Probes | Failure | |---|---|---|---| | `historian-connection` | `HistorianConnectionHealthCheck` | `IHistorianConnectionPool.AcquireAsync` + `client.ProbeAsync`, 3 s cap; reports a faulted client back to the pool | Unhealthy | | `galaxy-sql` | `SqlConnectionHealthCheck` (raw `SqlConnection` + `SELECT 1`; no EF) | Galaxy Repository DB from the top-level `Galaxy` section, bounded by its command timeout | Unhealthy | | `store-forward-drain` | `StoreForwardDrainHealthCheck` | `IHistorianStoreForward.GetStatusAsync`; backing up = `Storing \|\| ErrorOccurred` | **Degraded** only (never 503) | | `historian-session-pool` | `SessionPoolHealthCheck("read", …)` | `HistorianSessionPool.Snapshot.ConsecutiveFactoryFailures` | 0 → Healthy; 1–2 → Degraded; ≥ 3 → **Unhealthy** | | `historian-event-session-pool` | `SessionPoolHealthCheck("event", …)` | same over `HistorianEventSessionPool` | same | ### Finding: the session-pool check latched Unhealthy — historiangw#87 (fixed on main `5de3dd9`) > **Fixed 2026-11-09 on `origin/main` `5de3dd9`** (after this audit's `4d6a9ad`): caller cancellation no longer counts, and while the failure count is non-zero each keepalive tick opens one probe session (success resets it). Deployed boxes still run `d8d890c` until historiangw#78. The analysis below describes the pre-fix code. historiangw#87 (Server-007, High/P1): `_consecutiveFactoryFailures` is reset only by a **successful factory call**, and the factory only runs when a real RPC needs a new lease (the keepalive sweep never opens sessions). After a Historian outage of ≥ 3 failed leases, `/health/ready` goes 503, a k8s probe removes the pod, no RPC arrives, and the check **never recovers** until restart; three clients whose short deadlines cancel a cold handshake also trip it (the `catch` counts caller cancellation). On the NSSM wonder boxes it shows as a false Unhealthy on Overview. **Still present on `origin/main` `4d6a9ad`:** `HistorianSessionPool.cs:264-273` and `HistorianEventSessionPool.cs:254-262` increment on any throw and reset only on success; nothing probes or decays the streak; both pools default to `Enabled = true` (`Configuration/HistorianSessionPoolOptions.cs:24`, `EventSessionPoolOptions.cs:24`), so the latching check is on the ready tier in the default configuration. ## Active-node gating None, by design: a single-process sidecar per host with no cluster. No `IActiveNodeGate`, no `RequireActiveNode()`. `Redundancy` options concern the Historian servers it talks to, not node election. ## Other endpoints `MapZbMetrics()`, `MapZbGalaxyRepository()`, the dashboard (its `HealthPage` renders the same ready-tier results), and the four Historian gRPC services. ## vs SPEC | SPEC | Status | Note | |---|---|---| | §1 three tiers via `MapZbHealth`, canonical tags | ✅ | `/health/active` is empty (always 200) | | §1 nothing on `live` | ✅ | | | §2.1 DB probe | 🟡 custom | `SqlConnectionHealthCheck` (no EF context to wrap) | | §2.4 downstream dependency probe | ✅ (custom) | Historian via the pool, not `GrpcDependencyHealthCheck` | | Ready = "can this node serve its dependencies?" | 🟡 | correct while dependencies fail; did not recover after a session-pool outage (#87 — fixed on main `5de3dd9`, not yet deployed) | | §3 canonical JSON writer | ✅ | | | §4 `IActiveNodeGate` | n/a | |