Files
scadaproj/components/health/current-state/historiangateway/CURRENT-STATE.md
T

75 lines
4.9 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Health — current state: HistorianGateway
**Verified 2026-11-09 at `historiangw@4d6a9ad` (`origin/main`)** for scadaproj#11. Read-only `git grep` /
`git show` against the origin default branch; all paths relative to the repo root
(`~/Desktop/HistorianGateway`). First current-state doc for this app (it postdates the original
three-project survey).
- **Package:** `ZB.MOM.WW.Health` **0.3.0**, core only
(`src/ZB.MOM.WW.HistorianGateway.Server/ZB.MOM.WW.HistorianGateway.Server.csproj:23`).
- **Registration:** inline block in `src/ZB.MOM.WW.HistorianGateway.Server/Program.cs:256-329`.
- **Mapping:** `app.MapZbHealth()` with default paths/writer (`Program.cs:417`).
- **Not** `GrpcDependencyHealthCheck`: earlier scadaproj docs said HistorianGateway used it. It does not
on `main` — the gRPC-only Historian client owns its own channel, so the Historian probe goes through the
connection pool instead (comment at `Program.cs:265-268`).
## Per-endpoint contents
| Endpoint | Checks run | Can return 503 when |
|---|---|---|
| `/healthz` (live) | none | never |
| `/health/ready` | `historian-connection`, `galaxy-sql`, + `store-forward-drain` (if `StoreForward:Enabled`), + `historian-session-pool` / `historian-event-session-pool` (if each pool is enabled — **both default `true`**) | Historian probe fails/> 3 s; Galaxy SQL fails/> `CommandTimeoutSeconds`; a session-pool factory has ≥ 3 consecutive failures |
| `/health/active` | **none** | never — an empty report is Healthy, so every node answers 200 |
## Checks
Files under `src/ZB.MOM.WW.HistorianGateway.Server/Health/`. All tagged `ready`, `failureStatus: null`.
| Name | Class | Probes | Failure |
|---|---|---|---|
| `historian-connection` | `HistorianConnectionHealthCheck` | `IHistorianConnectionPool.AcquireAsync` + `client.ProbeAsync`, 3 s cap; reports a faulted client back to the pool | Unhealthy |
| `galaxy-sql` | `SqlConnectionHealthCheck` (raw `SqlConnection` + `SELECT 1`; no EF) | Galaxy Repository DB from the top-level `Galaxy` section, bounded by its command timeout | Unhealthy |
| `store-forward-drain` | `StoreForwardDrainHealthCheck` | `IHistorianStoreForward.GetStatusAsync`; backing up = `Storing \|\| ErrorOccurred` | **Degraded** only (never 503) |
| `historian-session-pool` | `SessionPoolHealthCheck("read", …)` | `HistorianSessionPool.Snapshot.ConsecutiveFactoryFailures` | 0 → Healthy; 1–2 → Degraded; ≥ 3 → **Unhealthy** |
| `historian-event-session-pool` | `SessionPoolHealthCheck("event", …)` | same over `HistorianEventSessionPool` | same |
### Finding: the session-pool check latched Unhealthy — historiangw#87 (fixed on main `5de3dd9`)
> **Fixed 2026-11-09 on `origin/main` `5de3dd9`** (after this audit's `4d6a9ad`): caller cancellation no longer counts, and while the failure count is non-zero each keepalive tick opens one probe session (success resets it). Deployed boxes still run `d8d890c` until historiangw#78. The analysis below describes the pre-fix code.
historiangw#87 (Server-007, High/P1): `_consecutiveFactoryFailures` is reset only by a **successful
factory call**, and the factory only runs when a real RPC needs a new lease (the keepalive sweep never
opens sessions). After a Historian outage of ≥ 3 failed leases, `/health/ready` goes 503, a k8s probe
removes the pod, no RPC arrives, and the check **never recovers** until restart; three clients whose
short deadlines cancel a cold handshake also trip it (the `catch` counts caller cancellation). On the NSSM
wonder boxes it shows as a false Unhealthy on Overview.
**Still present on `origin/main` `4d6a9ad`:** `HistorianSessionPool.cs:264-273` and
`HistorianEventSessionPool.cs:254-262` increment on any throw and reset only on success; nothing probes
or decays the streak; both pools default to `Enabled = true`
(`Configuration/HistorianSessionPoolOptions.cs:24`, `EventSessionPoolOptions.cs:24`), so the latching
check is on the ready tier in the default configuration.
## Active-node gating
None, by design: a single-process sidecar per host with no cluster. No `IActiveNodeGate`, no
`RequireActiveNode()`. `Redundancy` options concern the Historian servers it talks to, not node
election.
## Other endpoints
`MapZbMetrics()`, `MapZbGalaxyRepository()`, the dashboard (its `HealthPage` renders the same
ready-tier results), and the four Historian gRPC services.
## vs SPEC
| SPEC | Status | Note |
|---|---|---|
| §1 three tiers via `MapZbHealth`, canonical tags | ✅ | `/health/active` is empty (always 200) |
| §1 nothing on `live` | ✅ | |
| §2.1 DB probe | 🟡 custom | `SqlConnectionHealthCheck` (no EF context to wrap) |
| §2.4 downstream dependency probe | ✅ (custom) | Historian via the pool, not `GrpcDependencyHealthCheck` |
| Ready = "can this node serve its dependencies?" | 🟡 | correct while dependencies fail; did not recover after a session-pool outage (#87 — fixed on main `5de3dd9`, not yet deployed) |
| §3 canonical JSON writer | ✅ | |
| §4 `IActiveNodeGate` | n/a | |