feat(cluster): self-first seed ordering closes the boot-alone outage gap

Every node now lists ITSELF as seed-nodes[0] and its partner second. Akka runs
FirstSeedNodeProcess -- the only bootstrap path that can form a NEW cluster when
no peer answers InitJoin -- exclusively for seed-nodes[0]; every other node runs
JoinSeedNodeProcess and retries InitJoin forever. That is why a lone cold-starting
central-b never came Up (the "registered outage gap"), and self-first ordering
closes it using Akka's own protocol.

- 6 node appsettings swapped (the *-node-b configs; the -a nodes were already
  self-first). All 14 shipped node configs now satisfy the invariant.
- StartupValidator enforces it at boot, comparing host AND port -- the invariant
  fails silently when broken, so it is enforced loudly. NOTE: the gitignored
  deploy/wonder-app-vd03/ overlay must be reordered before its next deploy or
  that node will refuse to boot.
- SelfFirstSeedBootstrapTests: real in-process clusters at production
  failure-detection timings, incl. a falsifiability control proving the OLD
  peer-first ordering never forms.

Rejected alternative (implemented, measured, discarded): an external self-form
timer calling Cluster.Join(SelfAddress) after a window. It sits outside Akka's
join handshake and so cannot tell "no seed answered" from "a seed answered and
the join is in flight". On a routine standby restart the peer is alive but the
join stalls behind removal of the node's own stale incarnation; a Join(self)
during TryingToJoin abandons the in-flight join and forms a second cluster at
the same address -- still split after 90s. Docs that claimed self-first ordering
was unsafe for simultaneous cold start are corrected: while mutually reachable
the InitJoin handshake converges them to one cluster (measured).
This commit is contained in:
Joseph Doherty
2026-07-22 06:32:00 -04:00
parent 69b3ccfc37
commit 4a6341d871
15 changed files with 316 additions and 24 deletions
+13 -1
View File
@@ -286,7 +286,19 @@ Both modes finish by restarting the victim and confirming it rejoins as a ready
**Partition trade (accepted).** Auto-down is availability-first: in a *real network partition* (both nodes alive, link cut) each side downs the other and both run active — dual-active until an operator restarts one side after the partition heals. This was an explicit owner decision (2026-07-21): site pairs have no shared lease infrastructure to arbitrate, and a stalled system is a bigger risk than a rare partition. See `docs/plans/2026-07-21-auto-down-availability-decision.md`.
**Seed-node bootstrap constraint (still applies to boot-alone).** Only the FIRST seed in `Cluster:SeedNodes` may self-join to form a *new* cluster. Both central nodes list `scadabridge-central-a` first (`docker/central-node-a/appsettings.Central.json`, `docker/central-node-b/appsettings.Central.json`), so a lone *restarted* `central-b` (with `central-a` still down) loops on `InitJoin` forever. Under auto-down this no longer causes the active-crash outage (the survivor keeps running — it never restarts), but it still bites when a node must boot alone (cold start of only the non-first-seed VM, or the survivor crashing while its peer is still dead). Operator recovery: **(1)** restart the first-seed node (`central-a`) — preferred; or **(2)** restart the survivor with a self-first seed override (env `ScadaBridge__Cluster__SeedNodes__0=akka.tcp://scadabridge@<self-host>:8081`, `ScadaBridge__Cluster__SeedNodes__1=<peer>`). The repo deliberately does NOT ship self-first ordering per node: with *both* nodes self-first, a simultaneous cold start can let each self-join independently → two one-node clusters that never merge.
**Seed-node ordering — every node lists ITSELF first (decision 2026-07-22).** Akka runs `FirstSeedNodeProcess` — the only bootstrap path that can form a *new* cluster when no peer answers `InitJoin` — exclusively when `seed-nodes[0]` is the node's own address; every other node runs `JoinSeedNodeProcess`, which retries `InitJoin` forever and can never form a cluster. Each shipped node config therefore lists itself first and its partner second (`docker/central-node-b/appsettings.Central.json` leads with `scadabridge-central-b`), and `StartupValidator` fails the boot if that ordering is ever broken. This closes the former **registered outage gap**, where a lone cold-starting `central-b` (with `central-a` down) never came `Up` and recovery was operator-driven.
Self-first ordering is safe, and the three interesting cases are covered by `SelfFirstSeedBootstrapTests` (real in-process clusters at production failure-detection timings):
| Scenario | Behavior |
|---|---|
| Lone cold-start, peer dead | Forms alone in ~5s (`seed-node-timeout`) — operational, unattended |
| Restart into a **live** peer | `InitJoinAck` answers, node rejoins; never islands |
| Both cold-start simultaneously (mutually reachable) | The `InitJoin` handshake resolves it *before* either self-joins → **one** 2-member cluster |
> An earlier revision of this README claimed the repo deliberately avoided self-first ordering because simultaneous cold start would produce "two one-node clusters that never merge". That is **not** what happens while the nodes are mutually reachable — the handshake converges them (measured, row 3 above). Only a genuine boot-time *partition* splits them, which is the same class `auto-down` already accepts.
> **Rejected alternative — an external self-form timer.** A watchdog that waits N seconds for membership and then calls `Cluster.Join(SelfAddress)` was implemented and discarded: it cannot see Akka's join handshake, so it cannot distinguish "no seed answered" from "a seed answered and the join is in flight". On a routine standby restart the peer is alive but the join stalls behind removal of the restarting node's own stale incarnation; a `Join(self)` issued during `TryingToJoin` abandons the in-flight join and forms a second cluster at the same address — a **permanent** split (measured: still split after 90s). Akka's own first-seed process has no such race because it *is* part of the handshake.
> **Observed results** (auto-down decision verification):
>