docs(cleanup): sync Traefik + Host docs to oldest-member active-node semantics + DB-gated /health/active; TLS roadmap note
Renames stale ActiveNodeHealthCheck references to OldestNodeActiveHealthCheck (Task 7 rename) across the requirements + components doc sets, corrects leader→oldest-member wording and the embedded IsActiveNode snippet, and adds an explicit 'production TLS profile not yet implemented' roadmap note. deploy/wonder-app-vd03 NodeName overlay edits deferred: that production deploy artifact is not tracked in this repo (same as Tasks 16 & 20).
This commit is contained in:
Joseph Doherty
@@ -1,6 +1,6 @@
|
||||
# Traefik Proxy
|
||||
|
||||
The Traefik Proxy is the reverse proxy and load balancer that fronts the central cluster's two web servers. It exposes a single stable entrypoint for all central traffic — Central UI, Management API, Inbound API — and routes exclusively to whichever central node is currently the Akka.NET cluster leader, using a health-check on each node's `/health/active` endpoint to make that determination. When the active node changes, Traefik detects the change on its next poll cycle and redirects traffic automatically, with no operator intervention.
|
||||
The Traefik Proxy is the reverse proxy and load balancer that fronts the central cluster's two web servers. It exposes a single stable entrypoint for all central traffic — Central UI, Management API, Inbound API — and routes exclusively to whichever central node is currently the active node (the oldest Up cluster member, which hosts the cluster singletons), using a health-check on each node's `/health/active` endpoint to make that determination. When the active node changes, Traefik detects the change on its next poll cycle and redirects traffic automatically, with no operator intervention.
|
||||
|
||||
## Overview
|
||||
|
||||
@@ -15,7 +15,7 @@ The proxy sits on the `scadabridge-net` Docker bridge network alongside both cen
|
||||
|
||||
### Active-node routing via `/health/active`
|
||||
|
||||
Traefik does not know which central node is the Akka.NET cluster leader — it discovers this by polling `/health/active` on both backends. The Host registers `ActiveNodeHealthCheck` under the `Active` health tag; `app.MapZbHealth()` serves it at `/health/active`. The check returns HTTP 200 on the leader and HTTP 503 on the standby (or when the actor system has not yet reached `MemberStatus.Up`):
|
||||
Traefik does not know which central node is the active (oldest Up / singleton-host) node — it discovers this by polling `/health/active` on both backends. The Host registers `OldestNodeActiveHealthCheck` under the `Active` health tag; `app.MapZbHealth()` serves it at `/health/active`. The check returns HTTP 200 on the oldest Up member (and only when its database check also passes) and HTTP 503 on the standby (or when the actor system has not yet reached `MemberStatus.Up`):
|
||||
|
||||
```csharp
|
||||
public bool IsActiveNode
|
||||
@@ -26,22 +26,19 @@ public bool IsActiveNode
|
||||
if (system == null)
|
||||
return false;
|
||||
|
||||
var cluster = Cluster.Get(system);
|
||||
var self = cluster.SelfMember;
|
||||
if (self.Status != MemberStatus.Up)
|
||||
return false;
|
||||
|
||||
var leader = cluster.State.Leader;
|
||||
return leader != null && leader == self.Address;
|
||||
// Oldest Up member, NOT cluster leader: cluster singletons follow the
|
||||
// oldest member, and leader/oldest can differ transiently during
|
||||
// membership churn — routing must track the singleton host.
|
||||
return ClusterActivityEvaluator.SelfIsOldest(Cluster.Get(system));
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The identical leadership check backs `ActiveNodeGate` — the `IActiveNodeGate` implementation the Inbound API endpoint filter consults before executing method scripts. Both surfaces agree on which node is active because they share the same Akka cluster state.
|
||||
The identical oldest-member check backs `ActiveNodeGate` — the `IActiveNodeGate` implementation the Inbound API endpoint filter consults before executing method scripts. Both surfaces agree on which node is active because they share the same `ClusterActivityEvaluator.SelfIsOldest` definition over Akka cluster state.
|
||||
|
||||
### Automatic failover
|
||||
|
||||
When the active central node goes down, the Akka cluster's keep-oldest split-brain resolver promotes the surviving node to leader (roughly 25 seconds: 10-second heartbeat threshold plus a 15-second stable-after period). Once the surviving node's `ActiveNodeHealthCheck` starts returning 200, Traefik's next poll cycle — within the 5-second interval — removes the failed backend from the pool and routes all subsequent requests to the new active node. No config change or restart is required on the Traefik side.
|
||||
When the active central node goes down, the Akka cluster's keep-oldest split-brain resolver leaves the surviving node as the oldest Up member and the singletons migrate to it (roughly 25 seconds: 10-second heartbeat threshold plus a 15-second stable-after period). Once the surviving node's `OldestNodeActiveHealthCheck` starts returning 200, Traefik's next poll cycle — within the 5-second interval — removes the failed backend from the pool and routes all subsequent requests to the new active node. No config change or restart is required on the Traefik side.
|
||||
|
||||
## Architecture
|
||||
|
||||
@@ -158,8 +155,8 @@ http:
|
||||
|
||||
## Dependencies & Interactions
|
||||
|
||||
- [Host (#15)](./Host.md) — implements and serves `/health/active` via `ActiveNodeHealthCheck` (tagged `Active`, mounted by `app.MapZbHealth()`). Also implements `ActiveNodeGate`, which enforces the same active-node contract at the Inbound API filter level, providing a defence-in-depth layer if traffic reaches the standby directly.
|
||||
- [Cluster Infrastructure (#13)](./ClusterInfrastructure.md) — the underlying Akka.NET cluster determines which node is the leader. Traefik's routing decision is derived entirely from cluster leadership state via the health-check poll; Traefik has no Akka dependency of its own.
|
||||
- [Host (#15)](./Host.md) — implements and serves `/health/active` via `OldestNodeActiveHealthCheck` (tagged `Active`, mounted by `app.MapZbHealth()`). Also implements `ActiveNodeGate`, which enforces the same active-node contract at the Inbound API filter level, providing a defence-in-depth layer if traffic reaches the standby directly.
|
||||
- [Cluster Infrastructure (#13)](./ClusterInfrastructure.md) — the underlying Akka.NET cluster determines which node is the oldest Up member (the singleton host). Traefik's routing decision is derived entirely from that membership state via the health-check poll; Traefik has no Akka dependency of its own.
|
||||
- [Central UI (#9)](./CentralUI.md) — Blazor Server (SignalR/WebSocket circuits) is proxied through Traefik. Traefik proxies WebSocket connections natively with no additional config. On failover, active SignalR circuits on the failed node are lost; the browser's reconnection logic re-establishes the circuit on the new active node. Session continuity is preserved because authentication uses a cookie-embedded JWT with Data Protection keys shared across both central nodes.
|
||||
- [Inbound API (#14)](./InboundAPI.md) — external API consumers target `http://localhost:9000/api/{methodName}`. Traefik routes each request to the active node; if a request reaches the standby directly (bypassing Traefik), `ActiveNodeGate` responds with HTTP 503.
|
||||
- [CLI (#19)](./CLI.md) — the CLI connects to the Management API via `http://localhost:9000` (the Traefik entrypoint) by default, so it always reaches the active central node without needing to know which node is active.
|
||||
@@ -168,7 +165,7 @@ http:
|
||||
|
||||
### Both backends show unhealthy on the dashboard
|
||||
|
||||
If both `central-a` and `central-b` appear red on the Traefik dashboard, neither node's `ActiveNodeHealthCheck` is returning 200. Common causes:
|
||||
If both `central-a` and `central-b` appear red on the Traefik dashboard, neither node's `OldestNodeActiveHealthCheck` is returning 200. Common causes:
|
||||
|
||||
1. **Akka cluster has not formed yet** — both nodes are still starting. Wait for the cluster to stabilise (typically 10–15 seconds after both containers are up). Check the central node logs for `Cluster is now ready`.
|
||||
2. **Split-brain resolver has downed both nodes** — a network partition followed by a split-brain condition. Restart the cluster via `bash docker/deploy.sh`.
|
||||
@@ -180,7 +177,7 @@ If a client receives HTTP 503 with `X-ScadaBridge-Active: false`, the request re
|
||||
|
||||
### Health check succeeds but `/health/ready` returns degraded
|
||||
|
||||
`/health/active` and `/health/ready` are independent. A node can pass the active check (it is the leader) but fail the readiness check (database or Akka cluster health probe failed). Traefik only uses `/health/active`; readiness gating is for orchestration and monitoring. Check the node's structured logs for `database` or `akka-cluster` check failures.
|
||||
`/health/active` and `/health/ready` overlap on the database check but are otherwise independent. `/health/active` is 200 only on the oldest Up member **and** requires its database check to pass; `/health/ready` gates on database + akka-cluster + required-singletons. Traefik only uses `/health/active`; readiness gating is for orchestration and monitoring. Check the node's structured logs for `database` or `akka-cluster` check failures.
|
||||
|
||||
## Related Documentation
|
||||
|
||||
|
||||
Reference in New Issue
Block a user