docs(cleanup): sync Traefik + Host docs to oldest-member active-node semantics + DB-gated /health/active; TLS roadmap note
Renames stale ActiveNodeHealthCheck references to OldestNodeActiveHealthCheck (Task 7 rename) across the requirements + components doc sets, corrects leader→oldest-member wording and the embedded IsActiveNode snippet, and adds an explicit 'production TLS profile not yet implemented' roadmap note. deploy/wonder-app-vd03 NodeName overlay edits deferred: that production deploy artifact is not tracked in this repo (same as Tasks 16 & 20).
This commit is contained in:
@@ -13,7 +13,7 @@ Runs as a Docker container (`scadabridge-traefik`) in the cluster compose stack
|
||||
## Responsibilities
|
||||
|
||||
- Route all HTTP traffic (Central UI, Management API, Inbound API, health endpoints) to the active central node.
|
||||
- Health-check both central nodes via `/health/active` to determine which is the active (cluster leader) node.
|
||||
- Health-check both central nodes via `/health/active` to determine which is the active (oldest Up member / singleton host) node.
|
||||
- Automatically fail over to the standby node when the active node goes down.
|
||||
- Provide a dashboard for monitoring routing state and backend health.
|
||||
|
||||
@@ -23,18 +23,18 @@ Runs as a Docker container (`scadabridge-traefik`) in the cluster compose stack
|
||||
|
||||
Traefik polls `/health/active` on both central nodes every 5 seconds. This endpoint returns:
|
||||
|
||||
- **HTTP 200** on the active node (the Akka.NET cluster leader).
|
||||
- **HTTP 503** on the standby node (or if the node is unreachable).
|
||||
- **HTTP 200** on the active node (the oldest Up cluster member — the node that hosts the cluster singletons).
|
||||
- **HTTP 503** on the standby node (or if the node is unreachable, or if its database is not reachable).
|
||||
|
||||
Only the node returning 200 receives traffic. The health check is implemented by `ActiveNodeHealthCheck` in the Host project, which checks `Cluster.Get(system).State.Leader == SelfMember.Address`.
|
||||
Only the node returning 200 receives traffic. The health check is implemented by `OldestNodeActiveHealthCheck` in the Host project, which reports Healthy only when `ClusterActivityEvaluator.SelfIsOldest(cluster)` is true — the **oldest-member** definition of "active", not the cluster *leader* (leader and oldest can differ transiently during membership churn; singletons follow the oldest member, so routing must too). `/health/active` is additionally tagged with the database readiness check, so a node whose database is unreachable sheds traffic even while it is the oldest member.
|
||||
|
||||
### Failover Sequence
|
||||
|
||||
1. Active node fails (crash, network partition, or graceful shutdown).
|
||||
2. Akka.NET cluster detects the failure (~10s heartbeat timeout).
|
||||
3. Split-brain resolver acts after stable-after period (~15s).
|
||||
4. Surviving node becomes cluster leader.
|
||||
5. `ActiveNodeHealthCheck` on the surviving node starts returning 200.
|
||||
4. Surviving node becomes the oldest Up member and the singletons migrate to it.
|
||||
5. `OldestNodeActiveHealthCheck` on the surviving node starts returning 200 (its database check must also pass).
|
||||
6. Traefik's next health poll (within 5s) detects the change.
|
||||
7. Traffic routes to the new active node.
|
||||
|
||||
@@ -112,13 +112,13 @@ The central nodes expose three health endpoints:
|
||||
|
||||
| Endpoint | Purpose | Who Uses It |
|
||||
|----------|---------|-------------|
|
||||
| `/health/ready` | Readiness gate — 200 when database + Akka cluster are healthy | Kubernetes probes, monitoring |
|
||||
| `/health/active` | Active node — 200 only on cluster leader | **Traefik** (routing decisions) |
|
||||
| `/health/ready` | Readiness gate — 200 when database + akka-cluster + required-singletons are healthy | Kubernetes probes, monitoring |
|
||||
| `/health/active` | Active node — 200 only on the oldest Up member (singleton host) whose database is also reachable | **Traefik** (routing decisions) |
|
||||
|
||||
## Dependencies
|
||||
|
||||
- **Central cluster nodes**: The two backends (`scadabridge-central-a`, `scadabridge-central-b`) on the `scadabridge-net` Docker network.
|
||||
- **ActiveNodeHealthCheck**: Health check implementation in `src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeHealthCheck.cs` that determines cluster leader status.
|
||||
- **OldestNodeActiveHealthCheck**: Health check implementation in `src/ZB.MOM.WW.ScadaBridge.Host/Health/OldestNodeActiveHealthCheck.cs` that reports the oldest-Up-member (singleton-host) status via `ClusterActivityEvaluator.SelfIsOldest`; `/health/active` also carries the database readiness check.
|
||||
- **Docker network**: All containers must be on the shared `scadabridge-net` bridge network.
|
||||
|
||||
## Interactions
|
||||
@@ -126,10 +126,12 @@ The central nodes expose three health endpoints:
|
||||
- **CLI**: Connects to `http://localhost:9000/management` — routed by Traefik to the active node.
|
||||
- **Browser (Central UI)**: Connects to `http://localhost:9000` — Blazor Server + SignalR routed to the active node.
|
||||
- **Inbound API consumers**: Connect to `http://localhost:9000/api/{methodName}` — routed to the active node.
|
||||
- **Cluster Infrastructure**: The `ActiveNodeHealthCheck` relies on Akka.NET cluster gossip state to determine the leader.
|
||||
- **Cluster Infrastructure**: The `OldestNodeActiveHealthCheck` relies on Akka.NET cluster membership state to determine the oldest Up member (the singleton host).
|
||||
|
||||
## Production Considerations
|
||||
|
||||
> **Production TLS profile: not yet implemented.** Traefik terminates plain HTTP, Akka remoting is unencrypted TCP, and site gRPC is `http://`. This is acceptable for the lab topology only; a production deployment requires TLS at all three layers (Traefik HTTPS entrypoint, Akka remoting TLS, gRPC over `https://`) and a secured Traefik dashboard. Tracked as an open roadmap item (arch review 01, underdeveloped area 3).
|
||||
|
||||
The current configuration is for development/testing. In production:
|
||||
|
||||
- **TLS termination**: Add HTTPS entrypoint with certificates (Let's Encrypt via Traefik's ACME provider, or static certs).
|
||||
|
||||
Reference in New Issue
Block a user