feat(cluster): self-first seed ordering closes the boot-alone outage gap

Every node now lists ITSELF as seed-nodes[0] and its partner second. Akka runs
FirstSeedNodeProcess -- the only bootstrap path that can form a NEW cluster when
no peer answers InitJoin -- exclusively for seed-nodes[0]; every other node runs
JoinSeedNodeProcess and retries InitJoin forever. That is why a lone cold-starting
central-b never came Up (the "registered outage gap"), and self-first ordering
closes it using Akka's own protocol.

- 6 node appsettings swapped (the *-node-b configs; the -a nodes were already
  self-first). All 14 shipped node configs now satisfy the invariant.
- StartupValidator enforces it at boot, comparing host AND port -- the invariant
  fails silently when broken, so it is enforced loudly. NOTE: the gitignored
  deploy/wonder-app-vd03/ overlay must be reordered before its next deploy or
  that node will refuse to boot.
- SelfFirstSeedBootstrapTests: real in-process clusters at production
  failure-detection timings, incl. a falsifiability control proving the OLD
  peer-first ordering never forms.

Rejected alternative (implemented, measured, discarded): an external self-form
timer calling Cluster.Join(SelfAddress) after a window. It sits outside Akka's
join handshake and so cannot tell "no seed answered" from "a seed answered and
the join is in flight". On a routine standby restart the peer is alive but the
join stalls behind removal of the node's own stale incarnation; a Join(self)
during TryingToJoin abandons the in-flight join and forms a second cluster at
the same address -- still split after 90s. Docs that claimed self-first ordering
was unsafe for simultaneous cold start are corrected: while mutually reachable
the InitJoin handshake converges them to one cluster (measured).
This commit is contained in:
Joseph Doherty
2026-07-22 06:32:00 -04:00
parent 69b3ccfc37
commit 4a6341d871
15 changed files with 316 additions and 24 deletions
@@ -114,9 +114,26 @@ When a node is downed (auto-downed by its peer after a partition heals, or a kee
1. Down ⇒ `CoordinatedShutdown``ActorSystem` termination.
2. The Host watches `ActorSystem.WhenTerminated`; a termination that is **not** the host's own `StopAsync` triggers `IHostApplicationLifetime.StopApplication()`, so **the process exits**.
3. The service supervisor restarts it — docker `restart: unless-stopped`, or Windows service recovery actions (`sc.exe failure … restart/…`).
4. The restarted process rejoins as a **fresh incarnation****but only while a peer still holding cluster state is reachable**. A lone restarted node that is *not* the first seed cannot re-form a cluster on its own (see the seed-node bootstrap constraint below); it waits for its peer.
4. The restarted process rejoins as a **fresh incarnation**, and can re-form the cluster on its own if no peer is reachable — see Seed Node Ordering below.
**Seed-node bootstrap constraint (still applies).** Only the FIRST seed listed in `Cluster:SeedNodes` may self-join to form a *new* cluster. All nodes list the same first seed (e.g. `scadabridge-central-a`), so a lone restarted non-first-seed node (with the first seed still down) loops on `InitJoin` forever — never `Up`, never routable. Under auto-down this constraint no longer causes the active-crash outage (the survivor keeps running and never restarts), but it still bites when a node must **boot alone** — e.g. a cold start of only the non-first-seed VM, or the survivor itself crashing while its peer is still dead. Recovery is operator-driven — either restart the first-seed node (preferred) or restart the survivor with a self-first seed override (`ScadaBridge__Cluster__SeedNodes__0` = self, `__1` = peer). The repo does not ship self-first ordering per node: with both nodes self-first a simultaneous cold start risks two independent one-node clusters that never merge.
### Seed Node Ordering
**Every node lists ITSELF as `seed-nodes[0]` and its partner second (decision 2026-07-22).**
Akka runs two different bootstrap processes depending on that first entry. When `seed-nodes[0]` is the node's own address it runs `FirstSeedNodeProcess`: it `InitJoin`s the *other* seeds and self-joins only after `seed-node-timeout` elapses with nobody answering. When it is not, the node runs `JoinSeedNodeProcess`, which can never form a new cluster — it retries `InitJoin` indefinitely.
Until 2026-07-22 every node listed the same first seed, so a node that had to **boot alone** — a cold start of only the non-first-seed VM, or the survivor crashing while its peer was still dead — never reached `Up` and was never routable. That was the **registered outage gap**, and recovery was operator-driven. Self-first ordering closes it using Akka's own protocol, and `StartupValidator` fails the boot if a node config ever breaks the ordering (the invariant is silent when violated, so it is enforced loudly).
Behavior, covered by `SelfFirstSeedBootstrapTests` (real in-process clusters built from `BuildHocon` at production failure-detection timings):
| Scenario | Behavior |
|---|---|
| Lone cold-start, peer dead | Self-joins after `seed-node-timeout` (~5s) — operational, unattended |
| Restart into a **live** peer | The peer answers `InitJoinAck`; the node joins the existing cluster and never islands |
| Both cold-start simultaneously, mutually reachable | The `InitJoin` handshake resolves it *before* either self-joins → **one** 2-member cluster |
| Both cold-start during a genuine boot-time **partition** | Each forms its own cluster — the same dual-active class `auto-down` already accepts, same recovery (restart one side) |
**Rejected alternative — an external self-form timer.** A watchdog that waited a configurable window for membership and then called `Cluster.Join(SelfAddress)` was implemented and discarded. Its success signal ("am I `Up` yet?") cannot distinguish *no seed answered* from *a seed answered and the join is in flight*, because it sits outside Akka's join handshake. On a routine standby restart the peer is alive but the join stalls behind removal of the restarting node's own stale incarnation (the peer must down it, then wait for the failure detector and a leader action); a `Join(self)` issued during `TryingToJoin` abandons the in-flight join and forms a second cluster at the same address. Measured: a permanent split that had not healed after 90s, converting a routine restart into an outage of the previously-healthy node. Akka's first-seed process has no such race because it is part of the handshake, which is why the ordering — not a timer — is the mechanism.
The docker failover drill (`docker/failover-drill.sh`) proves both directions: `standby` mode kills the younger node (active untouched, zero routing blips); `active` mode kills the active/oldest node and asserts the survivor **takes over while the victim is still down**.