Files

T

Joseph Doherty 9cbe65c844 docs(health): re-verify per-app probe coverage against the SPEC; refresh GAPS and current-state (scadaproj#11)

Audited the registration and mapping code of each app at its origin default branch: OtOpcUa
c904472a, MxAccessGateway de86f8b8, ScadaBridge c7d868c5, HistorianGateway 4d6a9ad (all on Health
0.3.0, all MapZbHealth with default paths: /healthz runs nothing, /health/ready and /health/active
by tag). GAPS.md now carries the verified per-endpoint coverage matrix and a SPEC-by-app table;
each current-state doc lists exactly what each endpoint runs, and HistorianGateway gets its first
current-state doc.

Recorded: mxaccessgw#206 (worker-executable + galaxy-sql readiness checks, on /health/ready) is
merged; historiangw#87 (session-pool check latches 503 until restart) is still open at 4d6a9ad.
Deferred P2 downstream probes are now per-app; ScadaBridge IActiveNodeGate unification stays
deferred (shared rule already used, only the interface is local).

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV

2026-11-09 14:00:44 -05:00

4.7 KiB

Raw Blame History

Health — current state: OtOpcUa

Verified 2026-11-09 at lmxopcua@c904472a (origin/master) for scadaproj#11. Read-only git grep / git show against the origin default branch; all paths relative to the repo root (~/Desktop/OtOpcUa). The pre-adoption state (verified 2026-06-01: three bespoke checks incl. AdminRoleLeaderHealthCheck, default writer) is gone; see git history of this file.

Packages: ZB.MOM.WW.Health, .Akka, .EntityFrameworkCore all 0.3.0 (Directory.Packages.props:211,219,220; referenced by src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj:37-39; .Akka also by ControlPlane and Runtime, which use ClusterActiveNode for the redundancy election).
Registration: builder.Services.AddOtOpcUaHealth(hasAdmin, hasDriver) (src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs:581) → src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/HealthEndpoints.cs:40-130. The check set depends on the node's roles.
Mapping: app.MapOtOpcUaHealth() (Program.cs:767) → app.MapZbHealth() with defaults (HealthEndpoints.cs:135).

Per-endpoint contents (by node role)

Endpoint	admin-only node	fused admin+driver	driver-only node
`/healthz`	none	none	none
`/health/ready`	`configdb`, `akka`	`opcua-endpoint`, `configdb`, `akka`	`opcua-endpoint`, `akka`
`/health/active`	`configdb`, `akka`, `cluster-primary`, `localdb-replication`	`configdb`, `akka`, `cluster-primary`, `localdb-replication`	`opcua-endpoint`, `akka`, `cluster-primary`, `localdb-replication`

opcua-endpoint is deliberately not on the fused node's active tier: Traefik routes the AdminUI by /health/active, and a failed SDK start must not take the operator console out of rotation; OPC UA failover is handled by the Runtime (ServiceLevel 0 + OpcUaEndpointStepDownActor, lmxopcua#762) (HealthEndpoints.cs:44-65).

Checks

Name	Class	Tags	Probes	Failure
`opcua-endpoint` (driver nodes)	`OpcUaEndpointHealthCheck` (host, `Health/OpcUaEndpointHealthCheck.cs`)	ready (+ active on driver-only)	`IOpcUaEndpointState.Status`: `Started` healthy	Unhealthy on `NotStarted` / `Stopped` / failed
`configdb` (admin nodes)	shared `DatabaseHealthCheck<OtOpcUaConfigDbContext>`	ready, active	`ProbeQuery` = `Deployments.AsNoTracking().Take(1)` (implies schema applied)	Unhealthy
`akka`	shared `AkkaClusterHealthCheck`, policy `OtOpcUaCompat`	ready, active	self `Up` among reachable members	Degraded only (never 503)
`cluster-primary`	shared `ActiveNodeHealthCheck`, `RolePreference = [admin, driver]`	active	oldest `Up` member of the first preferred role held	Unhealthy on standby and on a node holding neither role; Degraded while the ActorSystem/cluster is not yet available
`localdb-replication`	`LocalDbReplicationHealthCheck` (host)	active	`ISyncStatus` connected + oplog backlog < 100 000	Degraded only; Healthy when LocalDb or replication is absent/off

Driver-only nodes register no configdb check: they hold no ConfigDb (FetchAndCache from central), and the type-activated check would throw resolving the factory.

Active-node gating

Probe: cluster-primary above — the shared ClusterActiveNode rule. Per-cluster meshes (Phase 6) make "oldest Up member" already per-Cluster, so exactly one node per Cluster answers 200. RedundancyStateActor's Primary election routes through the same primitive, so the tier and the OPC UA ServiceLevel 250/240 split cannot disagree.
Request gate: none. OtOpcUa registers no IActiveNodeGate and uses no RequireActiveNode(); the AdminUI is pinned to the active node by Traefik routing on /health/active.
Extra probes on the active tier: configdb (Unhealthy) — a ConfigDb outage makes the admin pair answer 503 on both nodes, so Traefik has no AdminUI backend until it recovers. akka and localdb-replication can only Degrade, so they never change routing.

Other endpoints

MapOtOpcUaMetrics(); mesh gRPC services (MapOtOpcUaMeshGrpcServices). No bespoke health endpoint.

vs SPEC

SPEC	Status	Note
§1 three tiers via `MapZbHealth`, canonical tags	✅
§1 nothing on `live`	✅
§2.1 `DatabaseHealthCheck<T>`	✅	`ProbeQuery` variant
§2.2 / §7(b) Akka probe with Default policy	🟡	still `OtOpcUaCompat` (Degraded-only); see GAPS A1
§2.3 / §7(b) `ActiveNodeHealthCheck`	✅	role-preference mode
§2.4 downstream (mxaccessgw) gRPC probe	⛔ not used	gateway channels are per-driver, created from DB config; see GAPS P2
§3 canonical JSON writer	✅
§4 / §7(e) `IActiveNodeGate`	n/a	no gated endpoints; Traefik routes by the probe

4.7 KiB Raw Blame History