scadaproj/components/health/current-state/scadabridge/CURRENT-STATE.md

Health — current state: ScadaBridge

Repo: ~/Desktop/ScadaBridge. Stack: .NET 10, Akka.NET; solution ZB.MOM.WW.ScadaBridge.slnx. Health code centers on src/ZB.MOM.WW.ScadaBridge.Host/Health/ (ASP.NET probes) and the separate src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ project (domain aggregation pipeline). All paths relative to repo root. Verified 2026-06-01.

Two-tier pattern: /health/ready and /health/active — no /healthz. Three probes (database, Akka cluster, active-node). ScadaBridge also has a bespoke distributed HealthMonitoring/ pipeline that is entirely separate from the ASP.NET health checks and is out of scope for the shared library.

1. Endpoint wiring

src/ZB.MOM.WW.ScadaBridge.Host/Program.cs:

• :114–117 — builder.Services.AddHealthChecks() followed by three .AddCheck<T>() calls (no tags, checked by name at the endpoint level):
  • .AddCheck<DatabaseHealthCheck>("database")
  • .AddCheck<AkkaClusterHealthCheck>("akka-cluster")
  • .AddCheck<ActiveNodeHealthCheck>("active-node")
• :131 — builder.Services.AddSingleton<IActiveNodeGate, ActiveNodeGate>() registers the production IActiveNodeGate implementation (Inbound API gating, not a health-check probe).
• :222–226 — /health/ready mapped with Predicate = check => check.Name != "active-node" and ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse (from HealthChecks.UI.Client). Excludes the active-node check so a healthy standby node reports ready.
• :229–233 — /health/active mapped with Predicate = check => check.Name == "active-node" and ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse. Active-node check only.

No /healthz endpoint. Both mapped endpoints use HealthChecks.UI.Client JSON (not the default plain-text writer), which is a divergence from OtOpcUa.

2. Probes

DatabaseHealthCheck

src/ZB.MOM.WW.ScadaBridge.Host/Health/DatabaseHealthCheck.cs:

• :11 — injects ScadaBridgeDbContext directly (not a factory)
• :33–43 — calls _dbContext.Database.CanConnectAsync(cancellationToken):
  • Returns true → HealthCheckResult.Healthy("Database connection is available.") (:34–35)
  • Returns false → HealthCheckResult.Unhealthy("Database connection failed.") (:36)
  • Throws → HealthCheckResult.Unhealthy("Database connection failed.", ex) (:40)

CanConnectAsync tests the connection layer only — it does not run any query or verify schema state. This is less strict than OtOpcUa's Deployments query but more transparent about failure cause (connection vs. schema). No Degraded path.

AkkaClusterHealthCheck

src/ZB.MOM.WW.ScadaBridge.Host/Health/AkkaClusterHealthCheck.cs:

• :13 — injects AkkaHostedService (not ActorSystem directly)
• :33–50 — gets _akkaService.ActorSystem, guards on null → Degraded("ActorSystem not yet available."), then reads cluster.SelfMember.Status:
  • Up or Joining → Healthy($"Akka cluster member status: {status}") (:43)
  • Leaving or Exiting → Degraded($"Akka cluster member status: {status}") (:45)
  • anything else (Removed, Down, WeaklyUp…) → Unhealthy($"Akka cluster member status: {status}") (:47)

Three-way status policy: Healthy / Degraded / Unhealthy. This is more granular than OtOpcUa's two-way policy (self-Up-or-not → Healthy/Degraded with no Unhealthy path).

ActiveNodeHealthCheck

src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeHealthCheck.cs:

• :13 — injects AkkaHostedService
• :29–44 — three-path logic:
  • ActorSystem == null → Unhealthy("ActorSystem not yet available.") (:31)
  • SelfMember.Status != Up → Unhealthy($"Node not Up (status: ...)") (:37)
  • Up AND cluster.State.Leader == self.Address → Healthy("Active node (cluster leader).") (:41)
  • Up but not leader → Unhealthy("Standby node (not cluster leader).") (:43)

No Degraded path — ActiveNodeHealthCheck uses Unhealthy for standby and non-Up states, which causes /health/active to return HTTP 503 on a standby. This is the intended behavior for Traefik active-node routing.

3. Tag / tier summary

ScadaBridge uses name-based predicates at the endpoint level rather than tags on the check registration. Tags are absent from all three .AddCheck<T>() calls.

Probe	/health/ready	/health/active	/healthz
DatabaseHealthCheck	✅	— (excluded by name)	⛔ absent
AkkaClusterHealthCheck	✅	— (excluded by name)	⛔ absent
ActiveNodeHealthCheck	— (excluded by name)	✅	⛔ absent

/healthz is absent — there is no bare process liveness endpoint. Kubernetes or Traefik liveness probes must either use /health/ready or tolerate its 503-until-ready behavior.

4. IActiveNodeGate and Inbound API gating

src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeGate.cs:

• :24 — ActiveNodeGate implements IActiveNodeGate (from the InboundAPI project)
• :40 — IsActiveNode property: returns true only when _akkaService.ActorSystem != null AND cluster.SelfMember.Status == MemberStatus.Up AND cluster.State.Leader == self.Address. Defaults to false safely during startup (:45–46).
• :131 in Program.cs — registered as a singleton. The InboundApiEndpointFilter consults this gate on every /api/* request and returns HTTP 503 on a standby node.

ActiveNodeGate mirrors the exact same logic as ActiveNodeHealthCheck — both check Up + leader. They are separate types serving two different concerns (the health endpoint and the API gate) but are not abstracted into a shared service; each reads cluster state independently.

IActiveNodeGate is the generalized seam the ZB.MOM.WW.Health core package lifts to the shared library.

5. HealthMonitoring domain pipeline (out of scope for shared library)

src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ is a separate project implementing a distributed health aggregation pipeline. It is not ASP.NET Core health checks and is not in scope for ZB.MOM.WW.Health.

Key types:

• SiteHealthCollector — thread-safe singleton accumulating per-site error counters, connection metrics, and tag-read metrics. Populated by actors in the DCL layer.
• HealthReportSender — a background service on site clusters that serializes SiteHealthState and ships it to the central cluster via Akka remoting at a configurable interval.
• CentralHealthReportLoop — central-only background service that generates a synthetic SiteHealthReport for the central cluster itself (siteId "$central") and feeds it into the central aggregator.
• CentralHealthAggregator — a BackgroundService on the central cluster tracking the latest health report per site and detecting offline sites via heartbeat timeout. Exposes GetAggregatedHealth() to the Central UI's /monitoring/health endpoint.

This pipeline is domain-specific (multi-site ScadaBridge topology) and will remain per-project regardless of shared-library adoption.

6. Notable design choices

• Name-based predicates vs. tags — ScadaBridge uses check.Name == "active-node" predicate logic at the endpoint level. OtOpcUa uses tag membership (c.Tags.Contains("ready")). The tag approach is more composable (a probe can participate in multiple tiers), the name approach is more explicit. The shared MapZbHealth should use tags by default.
• HealthChecks.UI.Client response writer — ScadaBridge uses the richer JSON response writer from the AspNetCore.HealthChecks.UI.Client package. OtOpcUa uses the default plain-text writer. The shared library's canonical response writer standardizes this.
• ActiveNodeHealthCheck returns Unhealthy for standby — a standby is not unhealthy in the system sense; it is a deliberate routing discriminator. Using Unhealthy here ensures /health/active returns HTTP 503 (Traefik sees the node as down for active traffic). The naming is semantically imprecise but operationally correct.
• IActiveNodeGate + ActiveNodeGate duplication — the gate and the health check implement the same logic independently. The shared library's IActiveNodeGate seam + ActiveNodeHealthCheck unify them: one backing service, two consumers.

---

Adoption plan → ZB.MOM.WW.Health

Replace with shared probes:

• AkkaClusterHealthCheck → ZB.MOM.WW.Health.Akka.AkkaClusterHealthCheck using the Default policy (Up/Joining=Healthy, Leaving/Exiting=Degraded, else Unhealthy). ScadaBridge's existing three-way policy is the Default — no preset selection needed.
• ActiveNodeHealthCheck → ZB.MOM.WW.Health.Akka.ActiveNodeHealthCheck with no role filter (role-less default: Up && leader = Healthy, else Unhealthy). The shared implementation also backs IActiveNodeGate, eliminating the duplicated leader-check logic between ActiveNodeHealthCheck and ActiveNodeGate.
• DatabaseHealthCheck → ZB.MOM.WW.Health.EntityFrameworkCore.DatabaseHealthCheck<ScadaBridgeDbContext> using the default CanConnectAsync probe (ScadaBridge's existing behavior). No ProbeQuery delegate needed.
• Replace the name-based predicates with tag-based predicates by adding tags at registration time: "database" and "akka-cluster" → ["ready"]; "active-node" → ["active"]. Then call app.MapZbHealth() instead of the two manual MapHealthChecks calls.
• Add /healthz — MapZbHealth() maps the bare liveness tier automatically. ScadaBridge currently lacks this endpoint.
• Switch ResponseWriter from UIResponseWriter.WriteHealthCheckUIResponse to the shared canonical writer (a convergence item — HealthChecks.UI.Client style lifted to the default in ZB.MOM.WW.Health).

Keep bespoke:

• HealthMonitoring/ domain pipeline (SiteHealthCollector, CentralHealthAggregator, etc.) — entirely per-project, no shared-library equivalent.
• IActiveNodeGate moves from the InboundAPI project to ZB.MOM.WW.Health (core package) on adoption. InboundApiEndpointFilter references the shared interface; AkkaActiveNodeGate (from ZB.MOM.WW.Health.Akka) becomes the singleton implementation registered in DI. The interface definition is no longer owned by the InboundAPI project.
• The Central UI's /monitoring/health endpoint — powered by CentralHealthAggregator, not by ASP.NET health checks.
• The comment at Program.cs:217–221 explains the readiness design decision (standby nodes are ready; leadership is a separate concern). This intent is preserved by the tag-based approach.

Adoption is a follow-on task (tracked in GAPS.md), not part of the ZB.MOM.WW.Health library build. The library build delivers the shared implementations; adoption lands in the ScadaBridge repo as a separate commit once the nupkg is available.