dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(cluster): per-direction failover truth — first-seed bootstrap constraint + operator recovery actions; drop the undeliverable ~25s active-crash promise (plan R2-01 T2)

Browse Source
This commit is contained in:
Joseph Doherty
2026-07-13 10:33:56 -04:00
parent badc97afe8
commit d536450611
2 changed files with 18 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docker/README.md
+14 -8
View File
@@ -273,19 +273,25 @@ All test passwords are `password`. See `infra/glauth/config.toml` for the full l
### Automated failover drill (`failover-drill.sh`)
```bash
bash docker/failover-drill.sh
DRILL_MODE=standby bash docker/failover-drill.sh # default — survivable younger-node crash
DRILL_MODE=active bash docker/failover-drill.sh # oldest-node crash — measures the registered outage gap
```
The scripted drill is the repeatable version of the manual steps below. It:
The scripted drill (`docker kill` = SIGKILL, the hard-crash path — a `docker stop` would take the graceful `CoordinatedShutdown` path and would not prove crash recovery) has **two modes**, because under the unified oldest-member semantics the *active* node IS the oldest, i.e. the one crash two-node keep-oldest cannot survive:
1. Finds the **active** central node (via each node's `/health/active`).
2. **`docker kill`s it — SIGKILL, the hard-crash path.** This is the exact scenario arch-review 01 found had *no* automatic recovery before the SBR downing provider was enabled (`Akka.Cluster.SBR.SplitBrainResolverProvider` is now named explicitly in HOCON); a `docker stop` would take the graceful `CoordinatedShutdown` path instead and would not prove crash recovery.
3. Polls Traefik (`http://localhost:9000/health/active`, override with `TRAEFIK_URL`) until the survivor is routable, printing the observed failover time. Fails after `TIMEOUT_S` (default 90s).
4. Dumps the survivor's singleton/oldest/downing log evidence, then restarts the victim so it **rejoins as a fresh incarnation** (Task 20's recovery loop).
- **`DRILL_MODE=standby` (default) — kills the STANDBY (younger) central node.** The survivable direction: SBR downs the crashed member and the active node keeps its singletons. Expected result: **no routing outage at all** (the active node is never touched, so `/health/active` blips = 0) and member removal on the survivor within **~25s** (10s failure-detection threshold + 15s stable-after; the 2s heartbeat interval is not additive). PASS = the survivor logs the member removal within `TIMEOUT_S` (default 90s) while routing stays up.
- **`DRILL_MODE=active` — kills the ACTIVE (oldest) central node.** Expected result: a **total central outage** until the victim container is restarted — this is the registered deferred keep-oldest decision (master tracker 2026-07-08): keep-oldest downs the partition *without* the oldest, so the younger survivor downs itself, and it cannot re-form a cluster alone (see the seed-node constraint below). The drill confirms the dark window, then recovery within ~2 min of restarting the victim. The mode exists to make the registered gap *observable*, not to pretend it is covered.
Expected failover: **~25s** (2s heartbeat + 10s detection + 15s stable-after) **plus the Traefik ~5s health-check interval**. The drill exercises S1 (SBR downing on hard crash), S3 (single active node routed through Traefik), and the Task 20 restart/rejoin contract. Requires a running cluster (`bash docker/deploy.sh`) and `curl` + `docker` on the host.
The drill exercises S1 (SBR downing on hard crash), S3 (single active node routed through Traefik), and the Task 20 restart/rejoin contract. Requires a running cluster (`bash docker/deploy.sh`) and `curl` + `docker` on the host.
> **Observed failover time:** _run after next deploy and record here_ (the script was validated with `bash -n`; no live cluster was up when it was committed).
**Seed-node bootstrap constraint.** Only the FIRST seed in `Cluster:SeedNodes` may self-join to form a *new* cluster. Both central nodes list `scadabridge-central-a` first (`docker/central-node-a/appsettings.Central.json`, `docker/central-node-b/appsettings.Central.json`), so a lone restarted `central-b` (with `central-a` still down) loops on `InitJoin` forever — it never reaches `Up`, and `/health/active` never returns 200. Operator recovery actions: **(1)** restart the dead first-seed node (`central-a`) — preferred; or **(2)** restart the survivor with a self-first seed override (env `ScadaBridge__Cluster__SeedNodes__0=akka.tcp://scadabridge@<self-host>:8081`, `ScadaBridge__Cluster__SeedNodes__1=<peer>`). The repo deliberately does NOT ship self-first ordering per node: with *both* nodes self-first, a simultaneous cold start can let each self-join independently → two one-node clusters that never merge (the cold-start split-brain the identical-seed-order convention exists to prevent). The real remedy is the pending keep-oldest topology/strategy decision (deferred, owner: user).
> **Observed results** (recorded by plan R2-01 T3, cluster @ commit <sha>):
>
> | Direction (`DRILL_MODE`) | Outcome | Measured |
> |--------------------------|---------|----------|
> | `standby` (younger-node crash) | _pending T3_ | _pending T3_ |
> | `active` (oldest-node crash) | _pending T3_ | _pending T3_ |
### Central Failover