From dc7a613dae0b89405addb6d3bbbf3097736c5a42 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Wed, 8 Jul 2026 17:09:34 -0400 Subject: [PATCH] docs(cleanup): sync Traefik + Host docs to oldest-member active-node semantics + DB-gated /health/active; TLS roadmap note MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Renames stale ActiveNodeHealthCheck references to OldestNodeActiveHealthCheck (Task 7 rename) across the requirements + components doc sets, corrects leader→oldest-member wording and the embedded IsActiveNode snippet, and adds an explicit 'production TLS profile not yet implemented' roadmap note. deploy/wonder-app-vd03 NodeName overlay edits deferred: that production deploy artifact is not tracked in this repo (same as Tasks 16 & 20). --- docs/components/Host.md | 8 +++--- docs/components/TraefikProxy.md | 27 +++++++++------------ docs/requirements/Component-TraefikProxy.md | 22 +++++++++-------- 3 files changed, 28 insertions(+), 29 deletions(-) diff --git a/docs/components/Host.md b/docs/components/Host.md index 5658071a..c8c86447 100644 --- a/docs/components/Host.md +++ b/docs/components/Host.md @@ -51,13 +51,13 @@ The actor system name is always `scadabridge`. Site nodes carry two cluster role ### `/health/ready` — readiness gating -Central nodes register `DatabaseHealthCheck` (tagged `Ready`) and `AkkaClusterHealthCheck` (tagged `Ready`). The `/health/ready` endpoint returns 200 only when both pass. Readiness is explicitly not tied to cluster leadership: a fully operational standby central node still reports ready because `ActiveNodeHealthCheck` carries only the `Active` tag, not `Ready`. +Central nodes register `DatabaseHealthCheck` (tagged `Ready`) and `AkkaClusterHealthCheck` (tagged `Ready`). The `/health/ready` endpoint returns 200 only when both pass. Readiness is explicitly not tied to active-node (oldest-member) status: a fully operational standby central node still reports ready because `OldestNodeActiveHealthCheck` carries only the `Active` tag, not `Ready`. Load balancers and orchestrators should poll `/health/ready` to determine when a freshly started or failed-over node can receive traffic. ### `/health/active` — active-node routing for Traefik -`ActiveNodeHealthCheck` carries the `Active` tag and is served at `/health/active`. It returns 200 only on the cluster leader. Traefik polls this endpoint and routes inbound traffic — Central UI, Inbound API, Management API — exclusively to the node that answers 200. See [TraefikProxy](./TraefikProxy.md) for the upstream routing rules. +`OldestNodeActiveHealthCheck` carries the `Active` tag and is served at `/health/active`. It returns 200 only on the oldest Up cluster member (the singleton host) whose database check also passes — the oldest-member definition of "active" via `ClusterActivityEvaluator.SelfIsOldest`, not cluster leadership. Traefik polls this endpoint and routes inbound traffic — Central UI, Inbound API, Management API — exclusively to the node that answers 200. See [TraefikProxy](./TraefikProxy.md) for the upstream routing rules. The same leadership check backs `ActiveNodeGate`, the `IActiveNodeGate` implementation the Inbound API endpoint filter consults before executing a method script. A standby node therefore refuses inbound API calls even if traffic somehow reaches it directly. @@ -217,12 +217,12 @@ All other per-component configuration sections (`ScadaBridge:Communication`, `Sc - [Cluster Infrastructure (#13)](./ClusterInfrastructure.md) — the Host configures the underlying Akka.NET cluster (`AkkaHostedService.BuildHocon`); ClusterInfrastructure manages it at runtime. - [Configuration Database (#17)](./ConfigurationDatabase.md) — the Host registers `ScadaBridgeDbContext` and calls `AddConfigurationDatabase` (Central only); the `StartupRetry`-wrapped migration step runs before traffic is accepted. - [Central–Site Communication (#5)](./Communication.md) — the Host creates `CentralCommunicationActor` and `SiteCommunicationActor`, registers them with `ClusterClientReceptionist`, and wires the `ClusterClient` for site→central messaging; the gRPC server is mapped at `app.MapGrpcService()`. -- [Health Monitoring (#11)](./HealthMonitoring.md) — the Host registers health checks (`DatabaseHealthCheck`, `AkkaClusterHealthCheck`, `ActiveNodeHealthCheck`) and mounts them via `app.MapZbHealth()` on central; site nodes register `AddSiteHealthMonitoring` and `AkkaHealthReportTransport`. +- [Health Monitoring (#11)](./HealthMonitoring.md) — the Host registers health checks (`DatabaseHealthCheck`, `AkkaClusterHealthCheck`, `OldestNodeActiveHealthCheck`) and mounts them via `app.MapZbHealth()` on central; site nodes register `AddSiteHealthMonitoring` and `AkkaHealthReportTransport`. - [Audit Log (#23)](./AuditLog.md) — the Host calls `AddAuditLog` on both roles, `AddAuditLogCentralMaintenance` on central, and `AddAuditLogHealthMetricsBridge` on site; it creates the `AuditLogIngestActor` singleton and registers `SiteAuditTelemetryActor` on the dedicated dispatcher. - [Notification Outbox (#21)](./NotificationOutbox.md) — the Host creates the `NotificationOutboxActor` cluster singleton and hands its proxy to `CentralCommunicationActor`. - [Site Call Audit (#22)](./SiteCallAudit.md) — the Host creates the `SiteCallAuditActor` cluster singleton with a graceful-stop drain task registered in the `cluster-leave` coordinated-shutdown phase. - [Management Service (#18)](./ManagementService.md) — the Host creates `ManagementActor` and registers it with `ClusterClientReceptionist`; maps the Management and Audit HTTP APIs. -- [Traefik Proxy (#20)](./TraefikProxy.md) — Traefik polls `/health/active` to determine which central node to route traffic to; the Host implements the `ActiveNodeHealthCheck` and `ActiveNodeGate` that back this endpoint. +- [Traefik Proxy (#20)](./TraefikProxy.md) — Traefik polls `/health/active` to determine which central node to route traffic to; the Host implements the `OldestNodeActiveHealthCheck` and `ActiveNodeGate` that back this endpoint. - Design spec: [Component-Host.md](../requirements/Component-Host.md). ## Troubleshooting diff --git a/docs/components/TraefikProxy.md b/docs/components/TraefikProxy.md index d635a80e..ba78e45c 100644 --- a/docs/components/TraefikProxy.md +++ b/docs/components/TraefikProxy.md @@ -1,6 +1,6 @@ # Traefik Proxy -The Traefik Proxy is the reverse proxy and load balancer that fronts the central cluster's two web servers. It exposes a single stable entrypoint for all central traffic — Central UI, Management API, Inbound API — and routes exclusively to whichever central node is currently the Akka.NET cluster leader, using a health-check on each node's `/health/active` endpoint to make that determination. When the active node changes, Traefik detects the change on its next poll cycle and redirects traffic automatically, with no operator intervention. +The Traefik Proxy is the reverse proxy and load balancer that fronts the central cluster's two web servers. It exposes a single stable entrypoint for all central traffic — Central UI, Management API, Inbound API — and routes exclusively to whichever central node is currently the active node (the oldest Up cluster member, which hosts the cluster singletons), using a health-check on each node's `/health/active` endpoint to make that determination. When the active node changes, Traefik detects the change on its next poll cycle and redirects traffic automatically, with no operator intervention. ## Overview @@ -15,7 +15,7 @@ The proxy sits on the `scadabridge-net` Docker bridge network alongside both cen ### Active-node routing via `/health/active` -Traefik does not know which central node is the Akka.NET cluster leader — it discovers this by polling `/health/active` on both backends. The Host registers `ActiveNodeHealthCheck` under the `Active` health tag; `app.MapZbHealth()` serves it at `/health/active`. The check returns HTTP 200 on the leader and HTTP 503 on the standby (or when the actor system has not yet reached `MemberStatus.Up`): +Traefik does not know which central node is the active (oldest Up / singleton-host) node — it discovers this by polling `/health/active` on both backends. The Host registers `OldestNodeActiveHealthCheck` under the `Active` health tag; `app.MapZbHealth()` serves it at `/health/active`. The check returns HTTP 200 on the oldest Up member (and only when its database check also passes) and HTTP 503 on the standby (or when the actor system has not yet reached `MemberStatus.Up`): ```csharp public bool IsActiveNode @@ -26,22 +26,19 @@ public bool IsActiveNode if (system == null) return false; - var cluster = Cluster.Get(system); - var self = cluster.SelfMember; - if (self.Status != MemberStatus.Up) - return false; - - var leader = cluster.State.Leader; - return leader != null && leader == self.Address; + // Oldest Up member, NOT cluster leader: cluster singletons follow the + // oldest member, and leader/oldest can differ transiently during + // membership churn — routing must track the singleton host. + return ClusterActivityEvaluator.SelfIsOldest(Cluster.Get(system)); } } ``` -The identical leadership check backs `ActiveNodeGate` — the `IActiveNodeGate` implementation the Inbound API endpoint filter consults before executing method scripts. Both surfaces agree on which node is active because they share the same Akka cluster state. +The identical oldest-member check backs `ActiveNodeGate` — the `IActiveNodeGate` implementation the Inbound API endpoint filter consults before executing method scripts. Both surfaces agree on which node is active because they share the same `ClusterActivityEvaluator.SelfIsOldest` definition over Akka cluster state. ### Automatic failover -When the active central node goes down, the Akka cluster's keep-oldest split-brain resolver promotes the surviving node to leader (roughly 25 seconds: 10-second heartbeat threshold plus a 15-second stable-after period). Once the surviving node's `ActiveNodeHealthCheck` starts returning 200, Traefik's next poll cycle — within the 5-second interval — removes the failed backend from the pool and routes all subsequent requests to the new active node. No config change or restart is required on the Traefik side. +When the active central node goes down, the Akka cluster's keep-oldest split-brain resolver leaves the surviving node as the oldest Up member and the singletons migrate to it (roughly 25 seconds: 10-second heartbeat threshold plus a 15-second stable-after period). Once the surviving node's `OldestNodeActiveHealthCheck` starts returning 200, Traefik's next poll cycle — within the 5-second interval — removes the failed backend from the pool and routes all subsequent requests to the new active node. No config change or restart is required on the Traefik side. ## Architecture @@ -158,8 +155,8 @@ http: ## Dependencies & Interactions -- [Host (#15)](./Host.md) — implements and serves `/health/active` via `ActiveNodeHealthCheck` (tagged `Active`, mounted by `app.MapZbHealth()`). Also implements `ActiveNodeGate`, which enforces the same active-node contract at the Inbound API filter level, providing a defence-in-depth layer if traffic reaches the standby directly. -- [Cluster Infrastructure (#13)](./ClusterInfrastructure.md) — the underlying Akka.NET cluster determines which node is the leader. Traefik's routing decision is derived entirely from cluster leadership state via the health-check poll; Traefik has no Akka dependency of its own. +- [Host (#15)](./Host.md) — implements and serves `/health/active` via `OldestNodeActiveHealthCheck` (tagged `Active`, mounted by `app.MapZbHealth()`). Also implements `ActiveNodeGate`, which enforces the same active-node contract at the Inbound API filter level, providing a defence-in-depth layer if traffic reaches the standby directly. +- [Cluster Infrastructure (#13)](./ClusterInfrastructure.md) — the underlying Akka.NET cluster determines which node is the oldest Up member (the singleton host). Traefik's routing decision is derived entirely from that membership state via the health-check poll; Traefik has no Akka dependency of its own. - [Central UI (#9)](./CentralUI.md) — Blazor Server (SignalR/WebSocket circuits) is proxied through Traefik. Traefik proxies WebSocket connections natively with no additional config. On failover, active SignalR circuits on the failed node are lost; the browser's reconnection logic re-establishes the circuit on the new active node. Session continuity is preserved because authentication uses a cookie-embedded JWT with Data Protection keys shared across both central nodes. - [Inbound API (#14)](./InboundAPI.md) — external API consumers target `http://localhost:9000/api/{methodName}`. Traefik routes each request to the active node; if a request reaches the standby directly (bypassing Traefik), `ActiveNodeGate` responds with HTTP 503. - [CLI (#19)](./CLI.md) — the CLI connects to the Management API via `http://localhost:9000` (the Traefik entrypoint) by default, so it always reaches the active central node without needing to know which node is active. @@ -168,7 +165,7 @@ http: ### Both backends show unhealthy on the dashboard -If both `central-a` and `central-b` appear red on the Traefik dashboard, neither node's `ActiveNodeHealthCheck` is returning 200. Common causes: +If both `central-a` and `central-b` appear red on the Traefik dashboard, neither node's `OldestNodeActiveHealthCheck` is returning 200. Common causes: 1. **Akka cluster has not formed yet** — both nodes are still starting. Wait for the cluster to stabilise (typically 10–15 seconds after both containers are up). Check the central node logs for `Cluster is now ready`. 2. **Split-brain resolver has downed both nodes** — a network partition followed by a split-brain condition. Restart the cluster via `bash docker/deploy.sh`. @@ -180,7 +177,7 @@ If a client receives HTTP 503 with `X-ScadaBridge-Active: false`, the request re ### Health check succeeds but `/health/ready` returns degraded -`/health/active` and `/health/ready` are independent. A node can pass the active check (it is the leader) but fail the readiness check (database or Akka cluster health probe failed). Traefik only uses `/health/active`; readiness gating is for orchestration and monitoring. Check the node's structured logs for `database` or `akka-cluster` check failures. +`/health/active` and `/health/ready` overlap on the database check but are otherwise independent. `/health/active` is 200 only on the oldest Up member **and** requires its database check to pass; `/health/ready` gates on database + akka-cluster + required-singletons. Traefik only uses `/health/active`; readiness gating is for orchestration and monitoring. Check the node's structured logs for `database` or `akka-cluster` check failures. ## Related Documentation diff --git a/docs/requirements/Component-TraefikProxy.md b/docs/requirements/Component-TraefikProxy.md index a840ecc9..2ed1adba 100644 --- a/docs/requirements/Component-TraefikProxy.md +++ b/docs/requirements/Component-TraefikProxy.md @@ -13,7 +13,7 @@ Runs as a Docker container (`scadabridge-traefik`) in the cluster compose stack ## Responsibilities - Route all HTTP traffic (Central UI, Management API, Inbound API, health endpoints) to the active central node. -- Health-check both central nodes via `/health/active` to determine which is the active (cluster leader) node. +- Health-check both central nodes via `/health/active` to determine which is the active (oldest Up member / singleton host) node. - Automatically fail over to the standby node when the active node goes down. - Provide a dashboard for monitoring routing state and backend health. @@ -23,18 +23,18 @@ Runs as a Docker container (`scadabridge-traefik`) in the cluster compose stack Traefik polls `/health/active` on both central nodes every 5 seconds. This endpoint returns: -- **HTTP 200** on the active node (the Akka.NET cluster leader). -- **HTTP 503** on the standby node (or if the node is unreachable). +- **HTTP 200** on the active node (the oldest Up cluster member — the node that hosts the cluster singletons). +- **HTTP 503** on the standby node (or if the node is unreachable, or if its database is not reachable). -Only the node returning 200 receives traffic. The health check is implemented by `ActiveNodeHealthCheck` in the Host project, which checks `Cluster.Get(system).State.Leader == SelfMember.Address`. +Only the node returning 200 receives traffic. The health check is implemented by `OldestNodeActiveHealthCheck` in the Host project, which reports Healthy only when `ClusterActivityEvaluator.SelfIsOldest(cluster)` is true — the **oldest-member** definition of "active", not the cluster *leader* (leader and oldest can differ transiently during membership churn; singletons follow the oldest member, so routing must too). `/health/active` is additionally tagged with the database readiness check, so a node whose database is unreachable sheds traffic even while it is the oldest member. ### Failover Sequence 1. Active node fails (crash, network partition, or graceful shutdown). 2. Akka.NET cluster detects the failure (~10s heartbeat timeout). 3. Split-brain resolver acts after stable-after period (~15s). -4. Surviving node becomes cluster leader. -5. `ActiveNodeHealthCheck` on the surviving node starts returning 200. +4. Surviving node becomes the oldest Up member and the singletons migrate to it. +5. `OldestNodeActiveHealthCheck` on the surviving node starts returning 200 (its database check must also pass). 6. Traefik's next health poll (within 5s) detects the change. 7. Traffic routes to the new active node. @@ -112,13 +112,13 @@ The central nodes expose three health endpoints: | Endpoint | Purpose | Who Uses It | |----------|---------|-------------| -| `/health/ready` | Readiness gate — 200 when database + Akka cluster are healthy | Kubernetes probes, monitoring | -| `/health/active` | Active node — 200 only on cluster leader | **Traefik** (routing decisions) | +| `/health/ready` | Readiness gate — 200 when database + akka-cluster + required-singletons are healthy | Kubernetes probes, monitoring | +| `/health/active` | Active node — 200 only on the oldest Up member (singleton host) whose database is also reachable | **Traefik** (routing decisions) | ## Dependencies - **Central cluster nodes**: The two backends (`scadabridge-central-a`, `scadabridge-central-b`) on the `scadabridge-net` Docker network. -- **ActiveNodeHealthCheck**: Health check implementation in `src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeHealthCheck.cs` that determines cluster leader status. +- **OldestNodeActiveHealthCheck**: Health check implementation in `src/ZB.MOM.WW.ScadaBridge.Host/Health/OldestNodeActiveHealthCheck.cs` that reports the oldest-Up-member (singleton-host) status via `ClusterActivityEvaluator.SelfIsOldest`; `/health/active` also carries the database readiness check. - **Docker network**: All containers must be on the shared `scadabridge-net` bridge network. ## Interactions @@ -126,10 +126,12 @@ The central nodes expose three health endpoints: - **CLI**: Connects to `http://localhost:9000/management` — routed by Traefik to the active node. - **Browser (Central UI)**: Connects to `http://localhost:9000` — Blazor Server + SignalR routed to the active node. - **Inbound API consumers**: Connect to `http://localhost:9000/api/{methodName}` — routed to the active node. -- **Cluster Infrastructure**: The `ActiveNodeHealthCheck` relies on Akka.NET cluster gossip state to determine the leader. +- **Cluster Infrastructure**: The `OldestNodeActiveHealthCheck` relies on Akka.NET cluster membership state to determine the oldest Up member (the singleton host). ## Production Considerations +> **Production TLS profile: not yet implemented.** Traefik terminates plain HTTP, Akka remoting is unencrypted TCP, and site gRPC is `http://`. This is acceptable for the lab topology only; a production deployment requires TLS at all three layers (Traefik HTTPS entrypoint, Akka remoting TLS, gRPC over `https://`) and a secured Traefik dashboard. Tracked as an open roadmap item (arch review 01, underdeveloped area 3). + The current configuration is for development/testing. In production: - **TLS termination**: Add HTTPS entrypoint with certificates (Let's Encrypt via Traefik's ACME provider, or static certs).