docs(health): current-state x3 + GAPS + README
Code-verified current-state docs for OtOpcUa (three-tier full), ScadaBridge (two-tier, no /healthz), and MxAccessGateway (bare liveness only / no probes). GAPS backlog with P1 for MxGateway and convergence items for Akka status policy, DB probe technique, and response writer. README with per-project status table.
This commit is contained in:
@@ -0,0 +1,133 @@
|
||||
# Health — current state: MxAccessGateway
|
||||
|
||||
Repo: `~/Desktop/MxAccessGateway`. Stack: .NET 10 gateway (x64) + .NET 4.8 worker (x86), gRPC;
|
||||
solution `src/MxGateway.sln`.
|
||||
Health code lives in `src/ZB.MOM.WW.MxGateway.Server/GatewayApplication.cs`. All paths relative
|
||||
to repo root.
|
||||
Verified 2026-06-01.
|
||||
|
||||
**Summary: bare liveness only.** MxAccessGateway has a single `/health/live` endpoint that returns
|
||||
a hardcoded `GatewayHealthReply` JSON object. `AddHealthChecks()` is called at startup but is
|
||||
entirely unused — no `IHealthCheck` implementations are registered, `MapHealthChecks` is never
|
||||
called, and there is no readiness or active-node tier. The net48 x86 worker process has no HTTP
|
||||
server and therefore no health endpoint of any kind.
|
||||
|
||||
## 1. Endpoint wiring
|
||||
|
||||
`src/ZB.MOM.WW.MxGateway.Server/GatewayApplication.cs`:
|
||||
|
||||
- `:61` — `builder.Services.AddHealthChecks()` is called in the DI registration block. **This call
|
||||
is dead**: no `.AddCheck<T>()` call follows it, no `MapHealthChecks` is ever called. The
|
||||
framework registers the health-check infrastructure but nothing is wired through it.
|
||||
- `:139–145` — `MapGatewayEndpoints` maps a raw `endpoints.MapGet("/health/live", ...)` (not
|
||||
`MapHealthChecks`). The handler is an inline lambda that returns `Results.Ok(new GatewayHealthReply(...))`
|
||||
with a hardcoded `Status: "Healthy"`:
|
||||
|
||||
```csharp
|
||||
endpoints.MapGet(
|
||||
"/health/live",
|
||||
() => Results.Ok(new GatewayHealthReply(
|
||||
Status: "Healthy",
|
||||
DefaultBackend: GatewayContractInfo.DefaultBackendName,
|
||||
WorkerProtocolVersion: GatewayContractInfo.WorkerProtocolVersion)))
|
||||
.WithName("LiveHealth");
|
||||
```
|
||||
|
||||
This endpoint always returns HTTP 200 `{"Status":"Healthy",...}` as long as the process is alive.
|
||||
It carries no authentication requirement (no `[Authorize]` or `.RequireAuthorization()`).
|
||||
|
||||
## 2. Response shape
|
||||
|
||||
`GatewayHealthReply` is a record with three fields:
|
||||
- `Status` — always `"Healthy"` (hardcoded string, not the ASP.NET Core `HealthStatus` enum)
|
||||
- `DefaultBackend` — value of `GatewayContractInfo.DefaultBackendName` (the configured backend
|
||||
name, useful for confirming which gateway instance a probe hit)
|
||||
- `WorkerProtocolVersion` — value of `GatewayContractInfo.WorkerProtocolVersion` (the gRPC
|
||||
protocol version the gateway expects from the worker, useful for version-skew detection)
|
||||
|
||||
The response is not `HealthChecks.UI.Client` JSON and is not the standard ASP.NET Core health
|
||||
response shape. It is a bespoke JSON record.
|
||||
|
||||
## 3. Probes
|
||||
|
||||
None. There is no `IHealthCheck` registered. The `/health/live` response does not reflect:
|
||||
|
||||
- Whether the SQLite auth-store is reachable
|
||||
- Whether any active MXAccess session is functional
|
||||
- Whether the x86 worker named-pipe IPC is connected or the worker process is alive
|
||||
- Whether the gRPC service is actually accepting calls
|
||||
|
||||
The endpoint is purely a process liveness indicator.
|
||||
|
||||
## 4. Tier coverage
|
||||
|
||||
| Tier | Endpoint | Status |
|
||||
|---|---|---|
|
||||
| Process liveness | `/health/live` (raw `MapGet`) | ✅ present (but non-standard) |
|
||||
| Readiness | `/health/ready` | ⛔ absent |
|
||||
| Active node | `/health/active` | ⛔ absent (not Akka-based; not applicable as-is) |
|
||||
| `healthz` convention | `/healthz` | ⛔ absent |
|
||||
|
||||
MxAccessGateway is not an Akka.NET application — it has no cluster, no leader election, and no
|
||||
active-node concept. The "active" tier in the shared spec translates here to "is the worker process
|
||||
connected and the gRPC service ready to accept calls?" rather than cluster leadership.
|
||||
|
||||
## 5. x86 worker
|
||||
|
||||
`ZB.MOM.WW.MxGateway.Worker` is a .NET 4.8 console application communicating with the gateway
|
||||
over Windows named-pipe IPC. It has no HTTP server, no health endpoint, and no exposure to any
|
||||
probe mechanism. Its liveness must be inferred indirectly — either via the gateway process
|
||||
monitoring it (not currently implemented) or via the `GrpcDependencyHealthCheck` the gateway
|
||||
could use to probe the IPC channel.
|
||||
|
||||
## 6. Notable gaps
|
||||
|
||||
- `AddHealthChecks()` at `:61` is dead code. No `IHealthCheck` is ever registered via this call.
|
||||
- `/health/live` uses `MapGet` (a raw minimal-API handler) rather than `MapHealthChecks`. It
|
||||
bypasses the ASP.NET Core health-check middleware entirely, which means it does not participate
|
||||
in the standard health-check pipeline (no `IHealthCheckPublisher`, no `HealthReport`, no UI
|
||||
integration).
|
||||
- The hardcoded `"Healthy"` status means the endpoint cannot reflect real probe results even if
|
||||
probes were added later — the handler must be replaced, not just supplemented.
|
||||
- No readiness gating: orchestrators (Kubernetes, Traefik) that rely on `/health/ready` returning
|
||||
503 until the process is actually ready will receive 200 (or 404) from MxAccessGateway today.
|
||||
|
||||
---
|
||||
|
||||
## Adoption plan → `ZB.MOM.WW.Health`
|
||||
|
||||
**Replace `/health/live` + wire the shared tiers:**
|
||||
|
||||
The `AddHealthChecks()` call at `GatewayApplication.cs:61` is already present — it just needs
|
||||
probes registered against it. The raw `MapGet("/health/live", ...)` handler at `:139–145` must be
|
||||
removed and replaced with `app.MapZbHealth()` from `ZB.MOM.WW.Health`.
|
||||
|
||||
Steps:
|
||||
|
||||
1. **Remove** the inline `MapGet("/health/live", ...)` lambda (`:139–145`). The `GatewayHealthReply`
|
||||
record and `DefaultBackend`/`WorkerProtocolVersion` metadata can be surfaced differently (e.g., a
|
||||
`/info` endpoint or as custom data on the health response).
|
||||
2. **Register a `GrpcDependencyHealthCheck`** (from `ZB.MOM.WW.Health`) that probes the
|
||||
named-pipe IPC channel to the x86 worker. Tag `["ready"]`. This replaces the hardcoded
|
||||
liveness-only response with a real probe that reflects whether the worker is reachable.
|
||||
3. **Optionally add a `GrpcDependencyHealthCheck`** for any downstream gRPC dependency (e.g., the
|
||||
Galaxy Repository connection) if the gateway is expected to be healthy only when its upstreams are
|
||||
reachable. Tag `["ready"]`.
|
||||
4. **Call `app.MapZbHealth()`** — this maps `/health/ready` (tag `ready`), `/health/active` (tag
|
||||
`active`; initially empty — no active-node concept in MxGateway), and `/healthz` (bare liveness).
|
||||
The `/healthz` endpoint replaces the semantic role that `/health/live` served today.
|
||||
5. **Do not add `ZB.MOM.WW.Health.Akka`** — MxAccessGateway has no Akka dependency. The consumer
|
||||
matrix in the design specifies MxGateway uses the core package only.
|
||||
|
||||
**Keep bespoke:**
|
||||
|
||||
- The `WorkerProtocolVersion` / `DefaultBackend` metadata from `GatewayHealthReply` is
|
||||
MxAccessGateway-specific; keep it as a separate `/info` endpoint or embed it as `Data` on a
|
||||
custom probe rather than normalizing it into the shared contract.
|
||||
- The x86 worker itself (net48 console, named-pipe IPC, no HTTP) remains outside the shared health
|
||||
scheme. The `GrpcDependencyHealthCheck` observes the worker indirectly from the gateway side.
|
||||
- Per-gateway auth and TLS concerns on who may call health endpoints remain per-project.
|
||||
|
||||
**Adoption is a follow-on task** (tracked in `GAPS.md`), not part of the `ZB.MOM.WW.Health`
|
||||
library build. MxGateway is the **highest-priority adopter** (P1 gap — no probes/tiers today)
|
||||
and should be the first app wired up once the nupkg is available.
|
||||
@@ -0,0 +1,150 @@
|
||||
# Health — current state: OtOpcUa
|
||||
|
||||
Repo: `~/Desktop/OtOpcUa`. Stack: .NET 10, Akka.NET, OPC UA; solution `ZB.MOM.WW.OtOpcUa.slnx`.
|
||||
Health code lives in `src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/`. All paths relative to repo root.
|
||||
Verified 2026-06-01.
|
||||
|
||||
Full three-tier pattern: `/health/ready`, `/health/active`, and `/healthz`. Three probes covering
|
||||
the database, the Akka cluster, and the admin-role leader. All endpoints are `AllowAnonymous` to
|
||||
permit Traefik and load-balancer probing without credentials.
|
||||
|
||||
## 1. Endpoint wiring
|
||||
|
||||
`src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/HealthEndpoints.cs`:
|
||||
|
||||
- `:13` — XML comment explicitly names this as "ScadaLink's three-tier pattern: `ready` = boot ok;
|
||||
`active` = fully serving traffic; `healthz` = bare process liveness."
|
||||
- `:17` — `AddOtOpcUaHealth(IServiceCollection)` calls `services.AddHealthChecks()` and registers
|
||||
all three probes (lines 20–22):
|
||||
- `DatabaseHealthCheck` name `"configdb"`, tags `["ready","active"]`
|
||||
- `AkkaClusterHealthCheck` name `"akka"`, tags `["ready","active"]`
|
||||
- `AdminRoleLeaderHealthCheck` name `"admin-leader"`, tags `["active"]` only
|
||||
- `:28` — `MapOtOpcUaHealth(IEndpointRouteBuilder)` maps three endpoints (lines 33–44):
|
||||
- `/health/ready` — predicate `c => c.Tags.Contains("ready")`, `.AllowAnonymous()` (lines 33–36)
|
||||
- `/health/active` — predicate `c => c.Tags.Contains("active")`, `.AllowAnonymous()` (lines 37–40)
|
||||
- `/healthz` — predicate `_ => false` (no probes run; bare process liveness only), `.AllowAnonymous()` (lines 41–44)
|
||||
|
||||
`Program.cs`:
|
||||
- `:137` — `builder.Services.AddOtOpcUaHealth()`
|
||||
- `:159` — `app.MapOtOpcUaHealth()`
|
||||
|
||||
Response writer: default ASP.NET Core plain-text/JSON (no `HealthChecks.UI.Client`).
|
||||
|
||||
## 2. Probes
|
||||
|
||||
### DatabaseHealthCheck
|
||||
`src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/DatabaseHealthCheck.cs`:
|
||||
|
||||
- `:9` — injects `IDbContextFactory<OtOpcUaConfigDbContext>`
|
||||
- `:25–37` — opens a pooled context via `CreateDbContextAsync`, runs
|
||||
`db.Deployments.AsNoTracking().Take(1).ToListAsync()`. If the query succeeds →
|
||||
`HealthCheckResult.Healthy("ConfigDb reachable")` (`:31`). If it throws →
|
||||
`HealthCheckResult.Unhealthy("ConfigDb unreachable", ex)` (`:35`). No `Degraded` path.
|
||||
|
||||
The probe exercises a real query (not just `CanConnectAsync`) — it confirms the `Deployments` table
|
||||
is readable, which implies the schema migration has run. This is **stricter** than ScadaBridge's
|
||||
`CanConnectAsync` but more opaque about the failure reason.
|
||||
|
||||
Tags on registration: `["ready","active"]` — the database must be reachable for both readiness and
|
||||
active-node determination.
|
||||
|
||||
### AkkaClusterHealthCheck
|
||||
`src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/AkkaClusterHealthCheck.cs`:
|
||||
|
||||
- `:9` — injects `ActorSystem` directly
|
||||
- `:27–33` — calls `Cluster.Get(_system)`, scans `cluster.State.Members` for the member whose
|
||||
`Address == cluster.SelfAddress` and `Status == MemberStatus.Up`:
|
||||
- Found Up → `HealthCheckResult.Healthy($"Self Up; {cluster.State.Members.Count} member(s)")` (`:32`)
|
||||
- Not found → `HealthCheckResult.Degraded("Self not yet Up in cluster")` (`:33`)
|
||||
|
||||
No `Unhealthy` path — joining/leaving/removed nodes are all reported as `Degraded`. This differs from
|
||||
ScadaBridge's more granular three-way policy (see GAPS).
|
||||
|
||||
Tags on registration: `["ready","active"]`.
|
||||
|
||||
### AdminRoleLeaderHealthCheck
|
||||
`src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/AdminRoleLeaderHealthCheck.cs`:
|
||||
|
||||
- `:14` — injects `IClusterRoleInfo`
|
||||
- `:27–38` — three-path logic:
|
||||
- Node does not carry the `"admin"` role → `Healthy("Node does not carry admin role")` (`:30`) —
|
||||
non-admin nodes are immediately healthy, so this probe never gates a non-admin node.
|
||||
- Admin role + node is the role leader → `Healthy($"Admin leader ({...})")` (`:36`)
|
||||
- Admin role + not the leader → `Degraded($"Admin member but not leader (leader=...)")` (`:37`)
|
||||
|
||||
Tags on registration: `["active"]` only — does not participate in `/health/ready`. The intent is
|
||||
Traefik routing: the active node (admin-role leader) gets sticky admin-UI traffic; standby nodes
|
||||
are reachable for data-plane OPC UA but report `Degraded` on `/health/active` so the load balancer
|
||||
does not route control-plane traffic to them.
|
||||
|
||||
Note: no `Unhealthy` path for the role-filter case. If the ActorSystem is not running, `IClusterRoleInfo`
|
||||
presumably returns safe defaults (no role); this is not separately health-checked.
|
||||
|
||||
## 3. Tag / tier summary
|
||||
|
||||
| Probe | `/health/ready` | `/health/active` | `/healthz` |
|
||||
|---|---|---|---|
|
||||
| `DatabaseHealthCheck` | ✅ | ✅ | — |
|
||||
| `AkkaClusterHealthCheck` | ✅ | ✅ | — |
|
||||
| `AdminRoleLeaderHealthCheck` | — | ✅ | — |
|
||||
| (no probes) | — | — | ✅ (bare liveness) |
|
||||
|
||||
`/healthz` runs zero probes — it is a pure process liveness sentinel (process reachable = healthy;
|
||||
a crashed process = no response). Kubernetes liveness probes, Traefik TCP checks, and uptime
|
||||
monitors use this tier.
|
||||
|
||||
## 4. Downstream dependency coverage
|
||||
|
||||
No probe for the upstream MxAccessGateway gRPC channel. If the gateway is unreachable, OtOpcUa
|
||||
reports healthy here (the GalaxyDriver will surface errors in OPC UA diagnostics, but `/health/ready`
|
||||
and `/health/active` will not reflect it). This is a gap that the shared `GrpcDependencyHealthCheck`
|
||||
probe in `ZB.MOM.WW.Health` would close.
|
||||
|
||||
## 5. Notable design choices
|
||||
|
||||
- **AllowAnonymous on all tiers** — see `HealthEndpoints.cs:30–32` comment: "Without it the
|
||||
`AddOtOpcUaAuth` fallback policy 401s every probe and Traefik marks every backend unhealthy."
|
||||
- **Query probe, not `CanConnectAsync`** — the `Deployments` query validates that the schema has
|
||||
been applied. ScadaBridge uses `CanConnectAsync`; neither is wrong but they diverge.
|
||||
- **`Degraded` semantics** — the Akka check uses `Degraded` (not `Unhealthy`) for a joining/pre-Up
|
||||
node. ASP.NET Core maps `Degraded` to HTTP 200 by default; Traefik sees 200 and considers the
|
||||
node ready. If `Unhealthy` (HTTP 503) is required to gate traffic, the `Degraded` path is
|
||||
insufficient.
|
||||
- **`IClusterRoleInfo` abstraction** — the admin-leader check depends on `IClusterRoleInfo`, an OtOpcUa
|
||||
interface, not the raw `Akka.Cluster.Cluster` API. This is a testability-friendly layer absent in
|
||||
ScadaBridge's direct Akka usage.
|
||||
|
||||
---
|
||||
|
||||
## Adoption plan → `ZB.MOM.WW.Health`
|
||||
|
||||
**Replace with shared probes:**
|
||||
|
||||
- `AkkaClusterHealthCheck` → `ZB.MOM.WW.Health.Akka.AkkaClusterHealthCheck` using the
|
||||
**`OtOpcUaCompat` preset** (self-Up-among-members scan → Healthy/Degraded). The preset keeps
|
||||
OtOpcUa's existing two-way policy without forcing ScadaBridge's three-way policy onto it.
|
||||
- `AdminRoleLeaderHealthCheck` → `ZB.MOM.WW.Health.Akka.ActiveNodeHealthCheck` with
|
||||
`RoleFilter = "admin"`. The role-filter parameter produces identical behavior: non-admin nodes
|
||||
immediately healthy, admin leader healthy, admin non-leader degraded.
|
||||
- `DatabaseHealthCheck` → `ZB.MOM.WW.Health.EntityFrameworkCore.DatabaseHealthCheck<OtOpcUaConfigDbContext>`
|
||||
with a `ProbeQuery` delegate of `db => db.Deployments.AsNoTracking().Take(1).ToListAsync()`.
|
||||
The delegate preserves the stricter query probe rather than falling back to `CanConnectAsync`.
|
||||
- Add `GrpcDependencyHealthCheck` targeting the MxAccessGateway channel (closes the downstream
|
||||
dependency gap noted in §4). Tag `["ready","active"]`.
|
||||
- Replace `AddOtOpcUaHealth` / `MapOtOpcUaHealth` with `services.AddZbHealthChecks()` +
|
||||
`app.MapZbHealth()`. The `/healthz` bare-liveness tier is part of `MapZbHealth` by default —
|
||||
no separate wiring needed.
|
||||
|
||||
**Keep bespoke:**
|
||||
|
||||
- `IClusterRoleInfo` and its Akka implementation — this is an OtOpcUa abstraction used for more
|
||||
than health checks; it should remain in the OtOpcUa codebase. The shared `ActiveNodeHealthCheck`
|
||||
will accept `IClusterRoleInfo` (or an equivalent cluster-info abstraction) as an injection point.
|
||||
- The `AllowAnonymous` policy — this is an OtOpcUa auth concern; `MapZbHealth` must document that
|
||||
callers are responsible for applying `AllowAnonymous` (or the shared helper applies it by default).
|
||||
- Which probes are registered and their tag assignments — the shared library supplies the check
|
||||
implementations; the wiring (which names, which tags, which options) remains per-project.
|
||||
|
||||
**Adoption is a follow-on task** (tracked in `GAPS.md`), not part of the `ZB.MOM.WW.Health`
|
||||
library build. The library build delivers the shared implementations; adoption lands in the
|
||||
OtOpcUa repo as a separate commit once the nupkg is available.
|
||||
@@ -0,0 +1,185 @@
|
||||
# Health — current state: ScadaBridge
|
||||
|
||||
Repo: `~/Desktop/ScadaBridge`. Stack: .NET 10, Akka.NET; solution `ZB.MOM.WW.ScadaBridge.slnx`.
|
||||
Health code centers on `src/ZB.MOM.WW.ScadaBridge.Host/Health/` (ASP.NET probes) and the
|
||||
separate `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/` project (domain aggregation pipeline).
|
||||
All paths relative to repo root.
|
||||
Verified 2026-06-01.
|
||||
|
||||
Two-tier pattern: `/health/ready` and `/health/active` — no `/healthz`. Three probes (database,
|
||||
Akka cluster, active-node). ScadaBridge also has a bespoke distributed `HealthMonitoring/`
|
||||
pipeline that is entirely separate from the ASP.NET health checks and is out of scope for the
|
||||
shared library.
|
||||
|
||||
## 1. Endpoint wiring
|
||||
|
||||
`src/ZB.MOM.WW.ScadaBridge.Host/Program.cs`:
|
||||
|
||||
- `:114–117` — `builder.Services.AddHealthChecks()` followed by three `.AddCheck<T>()` calls
|
||||
(no tags, checked by name at the endpoint level):
|
||||
- `.AddCheck<DatabaseHealthCheck>("database")`
|
||||
- `.AddCheck<AkkaClusterHealthCheck>("akka-cluster")`
|
||||
- `.AddCheck<ActiveNodeHealthCheck>("active-node")`
|
||||
- `:131` — `builder.Services.AddSingleton<IActiveNodeGate, ActiveNodeGate>()` registers the
|
||||
production `IActiveNodeGate` implementation (Inbound API gating, not a health-check probe).
|
||||
- `:222–226` — `/health/ready` mapped with `Predicate = check => check.Name != "active-node"` and
|
||||
`ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse` (from `HealthChecks.UI.Client`).
|
||||
Excludes the active-node check so a healthy standby node reports ready.
|
||||
- `:229–233` — `/health/active` mapped with `Predicate = check => check.Name == "active-node"` and
|
||||
`ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse`. Active-node check only.
|
||||
|
||||
No `/healthz` endpoint. Both mapped endpoints use `HealthChecks.UI.Client` JSON (not the default
|
||||
plain-text writer), which is a divergence from OtOpcUa.
|
||||
|
||||
## 2. Probes
|
||||
|
||||
### DatabaseHealthCheck
|
||||
`src/ZB.MOM.WW.ScadaBridge.Host/Health/DatabaseHealthCheck.cs`:
|
||||
|
||||
- `:11` — injects `ScadaBridgeDbContext` directly (not a factory)
|
||||
- `:33–43` — calls `_dbContext.Database.CanConnectAsync(cancellationToken)`:
|
||||
- Returns `true` → `HealthCheckResult.Healthy("Database connection is available.")` (`:34–35`)
|
||||
- Returns `false` → `HealthCheckResult.Unhealthy("Database connection failed.")` (`:36`)
|
||||
- Throws → `HealthCheckResult.Unhealthy("Database connection failed.", ex)` (`:40`)
|
||||
|
||||
`CanConnectAsync` tests the connection layer only — it does not run any query or verify schema
|
||||
state. This is less strict than OtOpcUa's `Deployments` query but more transparent about failure
|
||||
cause (connection vs. schema). No `Degraded` path.
|
||||
|
||||
### AkkaClusterHealthCheck
|
||||
`src/ZB.MOM.WW.ScadaBridge.Host/Health/AkkaClusterHealthCheck.cs`:
|
||||
|
||||
- `:13` — injects `AkkaHostedService` (not `ActorSystem` directly)
|
||||
- `:33–50` — gets `_akkaService.ActorSystem`, guards on null → `Degraded("ActorSystem not yet
|
||||
available.")`, then reads `cluster.SelfMember.Status`:
|
||||
- `Up` or `Joining` → `Healthy($"Akka cluster member status: {status}")` (`:43`)
|
||||
- `Leaving` or `Exiting` → `Degraded($"Akka cluster member status: {status}")` (`:45`)
|
||||
- anything else (Removed, Down, WeaklyUp…) → `Unhealthy($"Akka cluster member status: {status}")` (`:47`)
|
||||
|
||||
Three-way status policy: Healthy / Degraded / Unhealthy. This is more granular than OtOpcUa's
|
||||
two-way policy (self-Up-or-not → Healthy/Degraded with no Unhealthy path).
|
||||
|
||||
### ActiveNodeHealthCheck
|
||||
`src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeHealthCheck.cs`:
|
||||
|
||||
- `:13` — injects `AkkaHostedService`
|
||||
- `:29–44` — three-path logic:
|
||||
- `ActorSystem == null` → `Unhealthy("ActorSystem not yet available.")` (`:31`)
|
||||
- `SelfMember.Status != Up` → `Unhealthy($"Node not Up (status: ...)")` (`:37`)
|
||||
- `Up` AND `cluster.State.Leader == self.Address` → `Healthy("Active node (cluster leader).")` (`:41`)
|
||||
- `Up` but not leader → `Unhealthy("Standby node (not cluster leader).")` (`:43`)
|
||||
|
||||
No `Degraded` path — `ActiveNodeHealthCheck` uses `Unhealthy` for standby and non-Up states,
|
||||
which causes `/health/active` to return HTTP 503 on a standby. This is the intended behavior for
|
||||
Traefik active-node routing.
|
||||
|
||||
## 3. Tag / tier summary
|
||||
|
||||
ScadaBridge uses **name-based predicates** at the endpoint level rather than tags on the check
|
||||
registration. Tags are absent from all three `.AddCheck<T>()` calls.
|
||||
|
||||
| Probe | `/health/ready` | `/health/active` | `/healthz` |
|
||||
|---|---|---|---|
|
||||
| `DatabaseHealthCheck` | ✅ | — (excluded by name) | ⛔ absent |
|
||||
| `AkkaClusterHealthCheck` | ✅ | — (excluded by name) | ⛔ absent |
|
||||
| `ActiveNodeHealthCheck` | — (excluded by name) | ✅ | ⛔ absent |
|
||||
|
||||
`/healthz` is absent — there is no bare process liveness endpoint. Kubernetes or Traefik liveness
|
||||
probes must either use `/health/ready` or tolerate its 503-until-ready behavior.
|
||||
|
||||
## 4. IActiveNodeGate and Inbound API gating
|
||||
|
||||
`src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeGate.cs`:
|
||||
|
||||
- `:24` — `ActiveNodeGate` implements `IActiveNodeGate` (from the `InboundAPI` project)
|
||||
- `:40` — `IsActiveNode` property: returns `true` only when `_akkaService.ActorSystem != null`
|
||||
AND `cluster.SelfMember.Status == MemberStatus.Up` AND `cluster.State.Leader == self.Address`.
|
||||
Defaults to `false` safely during startup (`:45–46`).
|
||||
- `:131` in `Program.cs` — registered as a singleton. The `InboundApiEndpointFilter` consults this
|
||||
gate on every `/api/*` request and returns HTTP 503 on a standby node.
|
||||
|
||||
`ActiveNodeGate` mirrors the exact same logic as `ActiveNodeHealthCheck` — both check Up + leader.
|
||||
They are separate types serving two different concerns (the health endpoint and the API gate) but
|
||||
are not abstracted into a shared service; each reads cluster state independently.
|
||||
|
||||
`IActiveNodeGate` is the generalized seam the `ZB.MOM.WW.Health` core package lifts to the shared
|
||||
library.
|
||||
|
||||
## 5. HealthMonitoring domain pipeline (out of scope for shared library)
|
||||
|
||||
`src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/` is a separate project implementing a distributed
|
||||
health aggregation pipeline. It is **not ASP.NET Core health checks** and is **not in scope** for
|
||||
`ZB.MOM.WW.Health`.
|
||||
|
||||
Key types:
|
||||
- `SiteHealthCollector` — thread-safe singleton accumulating per-site error counters, connection
|
||||
metrics, and tag-read metrics. Populated by actors in the DCL layer.
|
||||
- `HealthReportSender` — a background service on site clusters that serializes `SiteHealthState`
|
||||
and ships it to the central cluster via Akka remoting at a configurable interval.
|
||||
- `CentralHealthReportLoop` — central-only background service that generates a synthetic
|
||||
`SiteHealthReport` for the central cluster itself (siteId `"$central"`) and feeds it into the
|
||||
central aggregator.
|
||||
- `CentralHealthAggregator` — a `BackgroundService` on the central cluster tracking the latest
|
||||
health report per site and detecting offline sites via heartbeat timeout. Exposes
|
||||
`GetAggregatedHealth()` to the Central UI's `/monitoring/health` endpoint.
|
||||
|
||||
This pipeline is domain-specific (multi-site ScadaBridge topology) and will remain per-project
|
||||
regardless of shared-library adoption.
|
||||
|
||||
## 6. Notable design choices
|
||||
|
||||
- **Name-based predicates vs. tags** — ScadaBridge uses `check.Name == "active-node"` predicate
|
||||
logic at the endpoint level. OtOpcUa uses tag membership (`c.Tags.Contains("ready")`). The tag
|
||||
approach is more composable (a probe can participate in multiple tiers), the name approach is
|
||||
more explicit. The shared `MapZbHealth` should use tags by default.
|
||||
- **`HealthChecks.UI.Client` response writer** — ScadaBridge uses the richer JSON response writer
|
||||
from the `AspNetCore.HealthChecks.UI.Client` package. OtOpcUa uses the default plain-text writer.
|
||||
The shared library's canonical response writer standardizes this.
|
||||
- **`ActiveNodeHealthCheck` returns `Unhealthy` for standby** — a standby is not *unhealthy* in the
|
||||
system sense; it is a deliberate routing discriminator. Using `Unhealthy` here ensures `/health/active`
|
||||
returns HTTP 503 (Traefik sees the node as down for active traffic). The naming is semantically
|
||||
imprecise but operationally correct.
|
||||
- **`IActiveNodeGate` + `ActiveNodeGate` duplication** — the gate and the health check implement the
|
||||
same logic independently. The shared library's `IActiveNodeGate` seam + `ActiveNodeHealthCheck`
|
||||
unify them: one backing service, two consumers.
|
||||
|
||||
---
|
||||
|
||||
## Adoption plan → `ZB.MOM.WW.Health`
|
||||
|
||||
**Replace with shared probes:**
|
||||
|
||||
- `AkkaClusterHealthCheck` → `ZB.MOM.WW.Health.Akka.AkkaClusterHealthCheck` using the **Default
|
||||
policy** (`Up`/`Joining`=Healthy, `Leaving`/`Exiting`=Degraded, else Unhealthy). ScadaBridge's
|
||||
existing three-way policy is the Default — no preset selection needed.
|
||||
- `ActiveNodeHealthCheck` → `ZB.MOM.WW.Health.Akka.ActiveNodeHealthCheck` with no role filter
|
||||
(role-less default: Up && leader = Healthy, else Unhealthy). The shared implementation also backs
|
||||
`IActiveNodeGate`, eliminating the duplicated leader-check logic between `ActiveNodeHealthCheck`
|
||||
and `ActiveNodeGate`.
|
||||
- `DatabaseHealthCheck` → `ZB.MOM.WW.Health.EntityFrameworkCore.DatabaseHealthCheck<ScadaBridgeDbContext>`
|
||||
using the default `CanConnectAsync` probe (ScadaBridge's existing behavior). No `ProbeQuery`
|
||||
delegate needed.
|
||||
- Replace the name-based predicates with tag-based predicates by adding tags at registration time:
|
||||
`"database"` and `"akka-cluster"` → `["ready"]`; `"active-node"` → `["active"]`. Then call
|
||||
`app.MapZbHealth()` instead of the two manual `MapHealthChecks` calls.
|
||||
- **Add `/healthz`** — `MapZbHealth()` maps the bare liveness tier automatically. ScadaBridge
|
||||
currently lacks this endpoint.
|
||||
- Switch `ResponseWriter` from `UIResponseWriter.WriteHealthCheckUIResponse` to the shared
|
||||
canonical writer (a convergence item — `HealthChecks.UI.Client` style lifted to the default in
|
||||
`ZB.MOM.WW.Health`).
|
||||
|
||||
**Keep bespoke:**
|
||||
|
||||
- `HealthMonitoring/` domain pipeline (`SiteHealthCollector`, `CentralHealthAggregator`, etc.) —
|
||||
entirely per-project, no shared-library equivalent.
|
||||
- `IActiveNodeGate` from the `InboundAPI` project is the contract the `InboundApiEndpointFilter`
|
||||
depends on; it can be implemented by the shared `ActiveNodeHealthCheck` backing service but the
|
||||
interface definition stays in the InboundAPI project (or moves to a shared abstractions package).
|
||||
- The Central UI's `/monitoring/health` endpoint — powered by `CentralHealthAggregator`, not by
|
||||
ASP.NET health checks.
|
||||
- The comment at `Program.cs:217–221` explains the readiness design decision (standby nodes are
|
||||
ready; leadership is a separate concern). This intent is preserved by the tag-based approach.
|
||||
|
||||
**Adoption is a follow-on task** (tracked in `GAPS.md`), not part of the `ZB.MOM.WW.Health`
|
||||
library build. The library build delivers the shared implementations; adoption lands in the
|
||||
ScadaBridge repo as a separate commit once the nupkg is available.
|
||||
Reference in New Issue
Block a user