2fea619da3
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
75 lines
4.9 KiB
Markdown
75 lines
4.9 KiB
Markdown
# Health — current state: HistorianGateway
|
||
|
||
**Verified 2026-11-09 at `historiangw@4d6a9ad` (`origin/main`)** for scadaproj#11. Read-only `git grep` /
|
||
`git show` against the origin default branch; all paths relative to the repo root
|
||
(`~/Desktop/HistorianGateway`). First current-state doc for this app (it postdates the original
|
||
three-project survey).
|
||
|
||
- **Package:** `ZB.MOM.WW.Health` **0.3.0**, core only
|
||
(`src/ZB.MOM.WW.HistorianGateway.Server/ZB.MOM.WW.HistorianGateway.Server.csproj:23`).
|
||
- **Registration:** inline block in `src/ZB.MOM.WW.HistorianGateway.Server/Program.cs:256-329`.
|
||
- **Mapping:** `app.MapZbHealth()` with default paths/writer (`Program.cs:417`).
|
||
- **Not** `GrpcDependencyHealthCheck`: earlier scadaproj docs said HistorianGateway used it. It does not
|
||
on `main` — the gRPC-only Historian client owns its own channel, so the Historian probe goes through the
|
||
connection pool instead (comment at `Program.cs:265-268`).
|
||
|
||
## Per-endpoint contents
|
||
|
||
| Endpoint | Checks run | Can return 503 when |
|
||
|---|---|---|
|
||
| `/healthz` (live) | none | never |
|
||
| `/health/ready` | `historian-connection`, `galaxy-sql`, + `store-forward-drain` (if `StoreForward:Enabled`), + `historian-session-pool` / `historian-event-session-pool` (if each pool is enabled — **both default `true`**) | Historian probe fails/> 3 s; Galaxy SQL fails/> `CommandTimeoutSeconds`; a session-pool factory has ≥ 3 consecutive failures |
|
||
| `/health/active` | **none** | never — an empty report is Healthy, so every node answers 200 |
|
||
|
||
## Checks
|
||
|
||
Files under `src/ZB.MOM.WW.HistorianGateway.Server/Health/`. All tagged `ready`, `failureStatus: null`.
|
||
|
||
| Name | Class | Probes | Failure |
|
||
|---|---|---|---|
|
||
| `historian-connection` | `HistorianConnectionHealthCheck` | `IHistorianConnectionPool.AcquireAsync` + `client.ProbeAsync`, 3 s cap; reports a faulted client back to the pool | Unhealthy |
|
||
| `galaxy-sql` | `SqlConnectionHealthCheck` (raw `SqlConnection` + `SELECT 1`; no EF) | Galaxy Repository DB from the top-level `Galaxy` section, bounded by its command timeout | Unhealthy |
|
||
| `store-forward-drain` | `StoreForwardDrainHealthCheck` | `IHistorianStoreForward.GetStatusAsync`; backing up = `Storing \|\| ErrorOccurred` | **Degraded** only (never 503) |
|
||
| `historian-session-pool` | `SessionPoolHealthCheck("read", …)` | `HistorianSessionPool.Snapshot.ConsecutiveFactoryFailures` | 0 → Healthy; 1–2 → Degraded; ≥ 3 → **Unhealthy** |
|
||
| `historian-event-session-pool` | `SessionPoolHealthCheck("event", …)` | same over `HistorianEventSessionPool` | same |
|
||
|
||
### Finding: the session-pool check latched Unhealthy — historiangw#87 (fixed on main `5de3dd9`)
|
||
|
||
> **Fixed 2026-11-09 on `origin/main` `5de3dd9`** (after this audit's `4d6a9ad`): caller cancellation no longer counts, and while the failure count is non-zero each keepalive tick opens one probe session (success resets it). Deployed boxes still run `d8d890c` until historiangw#78. The analysis below describes the pre-fix code.
|
||
|
||
historiangw#87 (Server-007, High/P1): `_consecutiveFactoryFailures` is reset only by a **successful
|
||
factory call**, and the factory only runs when a real RPC needs a new lease (the keepalive sweep never
|
||
opens sessions). After a Historian outage of ≥ 3 failed leases, `/health/ready` goes 503, a k8s probe
|
||
removes the pod, no RPC arrives, and the check **never recovers** until restart; three clients whose
|
||
short deadlines cancel a cold handshake also trip it (the `catch` counts caller cancellation). On the NSSM
|
||
wonder boxes it shows as a false Unhealthy on Overview.
|
||
|
||
**Still present on `origin/main` `4d6a9ad`:** `HistorianSessionPool.cs:264-273` and
|
||
`HistorianEventSessionPool.cs:254-262` increment on any throw and reset only on success; nothing probes
|
||
or decays the streak; both pools default to `Enabled = true`
|
||
(`Configuration/HistorianSessionPoolOptions.cs:24`, `EventSessionPoolOptions.cs:24`), so the latching
|
||
check is on the ready tier in the default configuration.
|
||
|
||
## Active-node gating
|
||
|
||
None, by design: a single-process sidecar per host with no cluster. No `IActiveNodeGate`, no
|
||
`RequireActiveNode()`. `Redundancy` options concern the Historian servers it talks to, not node
|
||
election.
|
||
|
||
## Other endpoints
|
||
|
||
`MapZbMetrics()`, `MapZbGalaxyRepository()`, the dashboard (its `HealthPage` renders the same
|
||
ready-tier results), and the four Historian gRPC services.
|
||
|
||
## vs SPEC
|
||
|
||
| SPEC | Status | Note |
|
||
|---|---|---|
|
||
| §1 three tiers via `MapZbHealth`, canonical tags | ✅ | `/health/active` is empty (always 200) |
|
||
| §1 nothing on `live` | ✅ | |
|
||
| §2.1 DB probe | 🟡 custom | `SqlConnectionHealthCheck` (no EF context to wrap) |
|
||
| §2.4 downstream dependency probe | ✅ (custom) | Historian via the pool, not `GrpcDependencyHealthCheck` |
|
||
| Ready = "can this node serve its dependencies?" | 🟡 | correct while dependencies fail; did not recover after a session-pool outage (#87 — fixed on main `5de3dd9`, not yet deployed) |
|
||
| §3 canonical JSON writer | ✅ | |
|
||
| §4 `IActiveNodeGate` | n/a | |
|