feat(cluster): self-first seed ordering closes the boot-alone outage gap
Every node now lists ITSELF as seed-nodes[0] and its partner second. Akka runs FirstSeedNodeProcess -- the only bootstrap path that can form a NEW cluster when no peer answers InitJoin -- exclusively for seed-nodes[0]; every other node runs JoinSeedNodeProcess and retries InitJoin forever. That is why a lone cold-starting central-b never came Up (the "registered outage gap"), and self-first ordering closes it using Akka's own protocol. - 6 node appsettings swapped (the *-node-b configs; the -a nodes were already self-first). All 14 shipped node configs now satisfy the invariant. - StartupValidator enforces it at boot, comparing host AND port -- the invariant fails silently when broken, so it is enforced loudly. NOTE: the gitignored deploy/wonder-app-vd03/ overlay must be reordered before its next deploy or that node will refuse to boot. - SelfFirstSeedBootstrapTests: real in-process clusters at production failure-detection timings, incl. a falsifiability control proving the OLD peer-first ordering never forms. Rejected alternative (implemented, measured, discarded): an external self-form timer calling Cluster.Join(SelfAddress) after a window. It sits outside Akka's join handshake and so cannot tell "no seed answered" from "a seed answered and the join is in flight". On a routine standby restart the peer is alive but the join stalls behind removal of the node's own stale incarnation; a Join(self) during TryingToJoin abandons the in-flight join and forms a second cluster at the same address -- still split after 90s. Docs that claimed self-first ordering was unsafe for simultaneous cold start are corrected: while mutually reachable the InitJoin handshake converges them to one cluster (measured).
This commit is contained in:
Joseph Doherty
@@ -219,7 +219,7 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
|
||||
- CoordinatedShutdown for graceful singleton handover.
|
||||
- Automatic dual-node recovery from persistent storage.
|
||||
- **Active/standby is decided by `ActiveNodeEvaluator.SelfIsOldestUp`, never by cluster leadership** — see the Architecture note above. `/health/active` is **central-only** (site nodes map no `/health/*` at all) and backs both Traefik's active-node routing and `IActiveNodeGate`, so the proxy and the Inbound API always agree on which node is active. Central never needs to know which *site* node is active: ClusterClient contact rotation reaches either receptionist and the site-internal `ClusterSingletonProxy` lands the work on the active node for free. The **exception is gRPC**, which picks `GrpcNodeAAddress`/`GrpcNodeBAddress` explicitly and flips on error.
|
||||
- **Seed-node boot-alone constraint (still real under auto-down).** Only the *first* seed listed in `Cluster:SeedNodes` may self-join to form a new cluster, so a node that must BOOT alone while its peer is dead (cold start of only the non-first-seed VM, or the survivor crashing while the peer is still down) loops on `InitJoin` forever — never `Up`, never routable; recovery is operator-driven (restart the first seed, or a self-first seed override). The former keep-oldest active-crash total outage is CLOSED by the auto-down decision — the survivor keeps running and takes over; `failover-drill.sh DRILL_MODE=active` now asserts that takeover. See `docs/requirements/Component-ClusterInfrastructure.md` → Downing Strategy.
|
||||
- **Seed-node ordering: every node lists ITSELF first (decision 2026-07-22) — the boot-alone gap is CLOSED.** Only `seed-nodes[0]` may self-join to form a new cluster (Akka runs `FirstSeedNodeProcess` for it, `JoinSeedNodeProcess` — which can never form one — for everyone else). All 14 shipped node appsettings now lead with the node's own address, so any node can cold-start alone and become operational unattended (~5s, `seed-node-timeout`); `StartupValidator` fails the boot if the ordering is broken (compares host AND port; Akka does no DNS canonicalisation). Two nodes cold-starting together while mutually reachable converge on ONE cluster via the `InitJoin` handshake — they split only under a genuine boot-time partition, the same class auto-down accepts. **An external self-form timer (`Cluster.Join(SelfAddress)` after a window) was implemented and REJECTED:** it sits outside the join handshake, so on a routine standby restart — where the peer is alive but the join is stalled behind removal of the node's own stale incarnation — it fires mid-join and permanently splits the pair (measured: still split after 90s). Regression tests: `SelfFirstSeedBootstrapTests`. The keep-oldest active-crash total outage was separately closed by the auto-down decision. See `docs/requirements/Component-ClusterInfrastructure.md` → Seed Node Ordering.
|
||||
|
||||
### UI & Monitoring
|
||||
- Central UI: Blazor Server (ASP.NET Core + SignalR) with Bootstrap CSS. No third-party component frameworks (no Blazorise, MudBlazor, Radzen, etc.). Build custom Blazor components for tables, grids, forms, etc.
|
||||
|
||||
@@ -8,8 +8,8 @@
|
||||
},
|
||||
"Cluster": {
|
||||
"SeedNodes": [
|
||||
"akka.tcp://scadabridge@scadabridge-env2-central-a:8081",
|
||||
"akka.tcp://scadabridge@scadabridge-env2-central-b:8081"
|
||||
"akka.tcp://scadabridge@scadabridge-env2-central-b:8081",
|
||||
"akka.tcp://scadabridge@scadabridge-env2-central-a:8081"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
|
||||
@@ -10,8 +10,8 @@
|
||||
},
|
||||
"Cluster": {
|
||||
"SeedNodes": [
|
||||
"akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082",
|
||||
"akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082"
|
||||
"akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082",
|
||||
"akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
|
||||
+13
-1
@@ -286,7 +286,19 @@ Both modes finish by restarting the victim and confirming it rejoins as a ready
|
||||
|
||||
**Partition trade (accepted).** Auto-down is availability-first: in a *real network partition* (both nodes alive, link cut) each side downs the other and both run active — dual-active until an operator restarts one side after the partition heals. This was an explicit owner decision (2026-07-21): site pairs have no shared lease infrastructure to arbitrate, and a stalled system is a bigger risk than a rare partition. See `docs/plans/2026-07-21-auto-down-availability-decision.md`.
|
||||
|
||||
**Seed-node bootstrap constraint (still applies to boot-alone).** Only the FIRST seed in `Cluster:SeedNodes` may self-join to form a *new* cluster. Both central nodes list `scadabridge-central-a` first (`docker/central-node-a/appsettings.Central.json`, `docker/central-node-b/appsettings.Central.json`), so a lone *restarted* `central-b` (with `central-a` still down) loops on `InitJoin` forever. Under auto-down this no longer causes the active-crash outage (the survivor keeps running — it never restarts), but it still bites when a node must boot alone (cold start of only the non-first-seed VM, or the survivor crashing while its peer is still dead). Operator recovery: **(1)** restart the first-seed node (`central-a`) — preferred; or **(2)** restart the survivor with a self-first seed override (env `ScadaBridge__Cluster__SeedNodes__0=akka.tcp://scadabridge@<self-host>:8081`, `ScadaBridge__Cluster__SeedNodes__1=<peer>`). The repo deliberately does NOT ship self-first ordering per node: with *both* nodes self-first, a simultaneous cold start can let each self-join independently → two one-node clusters that never merge.
|
||||
**Seed-node ordering — every node lists ITSELF first (decision 2026-07-22).** Akka runs `FirstSeedNodeProcess` — the only bootstrap path that can form a *new* cluster when no peer answers `InitJoin` — exclusively when `seed-nodes[0]` is the node's own address; every other node runs `JoinSeedNodeProcess`, which retries `InitJoin` forever and can never form a cluster. Each shipped node config therefore lists itself first and its partner second (`docker/central-node-b/appsettings.Central.json` leads with `scadabridge-central-b`), and `StartupValidator` fails the boot if that ordering is ever broken. This closes the former **registered outage gap**, where a lone cold-starting `central-b` (with `central-a` down) never came `Up` and recovery was operator-driven.
|
||||
|
||||
Self-first ordering is safe, and the three interesting cases are covered by `SelfFirstSeedBootstrapTests` (real in-process clusters at production failure-detection timings):
|
||||
|
||||
| Scenario | Behavior |
|
||||
|---|---|
|
||||
| Lone cold-start, peer dead | Forms alone in ~5s (`seed-node-timeout`) — operational, unattended |
|
||||
| Restart into a **live** peer | `InitJoinAck` answers, node rejoins; never islands |
|
||||
| Both cold-start simultaneously (mutually reachable) | The `InitJoin` handshake resolves it *before* either self-joins → **one** 2-member cluster |
|
||||
|
||||
> An earlier revision of this README claimed the repo deliberately avoided self-first ordering because simultaneous cold start would produce "two one-node clusters that never merge". That is **not** what happens while the nodes are mutually reachable — the handshake converges them (measured, row 3 above). Only a genuine boot-time *partition* splits them, which is the same class `auto-down` already accepts.
|
||||
|
||||
> **Rejected alternative — an external self-form timer.** A watchdog that waits N seconds for membership and then calls `Cluster.Join(SelfAddress)` was implemented and discarded: it cannot see Akka's join handshake, so it cannot distinguish "no seed answered" from "a seed answered and the join is in flight". On a routine standby restart the peer is alive but the join stalls behind removal of the restarting node's own stale incarnation; a `Join(self)` issued during `TryingToJoin` abandons the in-flight join and forms a second cluster at the same address — a **permanent** split (measured: still split after 90s). Akka's own first-seed process has no such race because it *is* part of the handshake.
|
||||
|
||||
> **Observed results** (auto-down decision verification):
|
||||
>
|
||||
|
||||
@@ -8,8 +8,8 @@
|
||||
},
|
||||
"Cluster": {
|
||||
"SeedNodes": [
|
||||
"akka.tcp://scadabridge@scadabridge-central-a:8081",
|
||||
"akka.tcp://scadabridge@scadabridge-central-b:8081"
|
||||
"akka.tcp://scadabridge@scadabridge-central-b:8081",
|
||||
"akka.tcp://scadabridge@scadabridge-central-a:8081"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
|
||||
@@ -11,8 +11,8 @@
|
||||
},
|
||||
"Cluster": {
|
||||
"SeedNodes": [
|
||||
"akka.tcp://scadabridge@scadabridge-site-a-a:8082",
|
||||
"akka.tcp://scadabridge@scadabridge-site-a-b:8082"
|
||||
"akka.tcp://scadabridge@scadabridge-site-a-b:8082",
|
||||
"akka.tcp://scadabridge@scadabridge-site-a-a:8082"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
|
||||
@@ -11,8 +11,8 @@
|
||||
},
|
||||
"Cluster": {
|
||||
"SeedNodes": [
|
||||
"akka.tcp://scadabridge@scadabridge-site-b-a:8082",
|
||||
"akka.tcp://scadabridge@scadabridge-site-b-b:8082"
|
||||
"akka.tcp://scadabridge@scadabridge-site-b-b:8082",
|
||||
"akka.tcp://scadabridge@scadabridge-site-b-a:8082"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
|
||||
@@ -11,8 +11,8 @@
|
||||
},
|
||||
"Cluster": {
|
||||
"SeedNodes": [
|
||||
"akka.tcp://scadabridge@scadabridge-site-c-a:8082",
|
||||
"akka.tcp://scadabridge@scadabridge-site-c-b:8082"
|
||||
"akka.tcp://scadabridge@scadabridge-site-c-b:8082",
|
||||
"akka.tcp://scadabridge@scadabridge-site-c-a:8082"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
|
||||
@@ -113,7 +113,7 @@ Nothing in the emitted document enables remoting TLS or an Akka secure cookie
|
||||
|
||||
`run-coordinated-shutdown-when-down = on` means a downed node runs `CoordinatedShutdown` and terminates its own `ActorSystem`. The Host watches `ActorSystem.WhenTerminated`; a termination that is not the host's own `StopAsync` calls `IHostApplicationLifetime.StopApplication()` so the process exits and the service supervisor (docker `restart: unless-stopped`, Windows service recovery) restarts it as a fresh incarnation (`AkkaHostedService.cs:203-218`).
|
||||
|
||||
**Seed-node bootstrap constraint.** Only the *first* seed listed in `Cluster:SeedNodes` may self-join to form a new cluster, and all nodes list the same first seed. Under `auto-down` this no longer causes an active-crash outage — the survivor keeps running and never restarts — but it still bites when a node must **boot alone** (a cold start of only the non-first-seed VM, or the survivor crashing while its peer is still dead): that node loops on `InitJoin` until its peer returns. Recovery is operator-driven.
|
||||
**Seed-node ordering (decision 2026-07-22).** Only the *first* seed listed in `Cluster:SeedNodes` may self-join to form a new cluster — Akka runs `FirstSeedNodeProcess` for it and `JoinSeedNodeProcess` (which can never form one) for everyone else. Every node therefore lists **itself** first and its partner second, so any node can boot alone and become operational unattended; `StartupValidator` fails the boot if that ordering is broken. Until this change all nodes shared one first seed, and a node that had to boot alone looped on `InitJoin` until its peer returned — the registered outage gap. See `docs/requirements/Component-ClusterInfrastructure.md` → Seed Node Ordering for the scenario table and for why an external self-form timer was rejected.
|
||||
|
||||
### Failure detection and failover timeline
|
||||
|
||||
@@ -165,7 +165,7 @@ Each node carries one or more cluster roles set in the HOCON `roles` list, built
|
||||
|
||||
### Dual-node recovery
|
||||
|
||||
Because both nodes are configured as seed nodes, whichever node starts first after a simultaneous failure forms a new cluster; the second joins when it comes up. There is no pre-existing cluster to conflict with, so the "both starting fresh" case needs no downing decision at all. The one ordering dependency that does exist is the seed-node bootstrap constraint above: a node that must boot *alone* forms a cluster only if it is the first seed listed.
|
||||
Because both nodes are configured as seed nodes **and each lists itself first**, whichever node starts first after a simultaneous failure forms a new cluster; the second joins when it comes up. There is no pre-existing cluster to conflict with, so the "both starting fresh" case needs no downing decision at all. Since 2026-07-22 there is no remaining ordering dependency: a node that must boot *alone* forms a cluster regardless of which node it is. Two nodes cold-starting at the same moment converge on one cluster via the `InitJoin` handshake — they split only under a genuine boot-time partition, the same class `auto-down` already accepts.
|
||||
|
||||
### Cluster singletons hosted
|
||||
|
||||
@@ -300,7 +300,7 @@ This returns `false` while the actor system is warming up, and `SelfIsOldest` re
|
||||
|
||||
`ClusterOptionsValidator` rejects fewer than two seed nodes (without `AllowSingleNodeCluster`), a strategy outside `auto-down` / `keep-oldest`, `MinNrOfMembers != 1`, or `DownIfAlone = false` under `keep-oldest`, at startup with an `OptionsValidationException`. Check that both seed-node URIs reference the Akka remoting port, not the gRPC port (8083) or metrics port (8084) — on site nodes, `StartupValidator` explicitly rejects seed entries whose port matches `GrpcPort`.
|
||||
|
||||
A node that boots, logs no validation error, but never reaches `Up` is usually hitting the seed-node bootstrap constraint: it is not the first entry in `SeedNodes` and the first seed is down, so it loops on `InitJoin` waiting for a peer that can form the cluster.
|
||||
A node that boots, logs no validation error, but never reaches `Up` was — before 2026-07-22 — usually hitting the seed-node bootstrap constraint: it was not the first entry in `SeedNodes` and the first seed was down, so it looped on `InitJoin` waiting for a peer that could form the cluster. Self-first ordering plus the `StartupValidator` rule that enforces it should make this unreachable; if you still see it, check that `seed-nodes[0]` really resolves to this node's own `NodeHostname:RemotingPort` (the validator compares host *and* port, and Akka does no DNS canonicalisation — `node-a` and `node-a.example.com` are different seed identities).
|
||||
|
||||
### Singleton not starting after failover
|
||||
|
||||
|
||||
@@ -88,17 +88,19 @@ Both central nodes must be configured as seed nodes for each other:
|
||||
},
|
||||
"Cluster": {
|
||||
"SeedNodes": [
|
||||
"akka.tcp://scadabridge@central-01.example.com:8081",
|
||||
"akka.tcp://scadabridge@central-02.example.com:8081"
|
||||
"akka.tcp://scadabridge@central-02.example.com:8081",
|
||||
"akka.tcp://scadabridge@central-01.example.com:8081"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
> **Seed order is load-bearing — each node lists ITSELF first** (decision 2026-07-22). Note Node B's list is the reverse of Node A's. Akka only lets `seed-nodes[0]` form a *new* cluster, so a node listing its partner first can never boot while that partner is down. `StartupValidator` rejects the boot if the ordering is wrong, comparing host **and** port; use the same spelling of the hostname in `NodeHostname` and in the seed URI, since Akka does no DNS canonicalisation (`central-02` and `central-02.example.com` are different seed identities). See `docs/requirements/Component-ClusterInfrastructure.md` → Seed Node Ordering.
|
||||
|
||||
### Cluster Behavior
|
||||
|
||||
- **Split-brain resolver**: Keep-oldest with `down-if-alone = on`, 15-second stable-after.
|
||||
- **Split-brain resolver**: `auto-down` (`AutoDowning` provider, `auto-down-unreachable-after` = 15s) since the 2026-07-21 availability-over-partition-safety decision — the leader among the *reachable* members downs the unreachable peer, so a hard crash of **either** node fails over. Accepted trade: a real partition leaves both sides active until an operator restarts one. `keep-oldest` (with `down-if-alone = on`) remains a supported `SplitBrainResolverStrategy` value, but in a two-node cluster it cannot survive a crash of the oldest node. See `docs/plans/2026-07-21-auto-down-availability-decision.md`.
|
||||
- **Minimum members**: `min-nr-of-members = 1` — a single node can form a cluster.
|
||||
- **Failure detection**: 2-second heartbeat interval, 10-second threshold.
|
||||
- **Total failover time**: ~25 seconds from node failure to singleton migration.
|
||||
@@ -145,6 +147,8 @@ Each site has its own two-node cluster:
|
||||
}
|
||||
```
|
||||
|
||||
> **Site Node B reverses this list** — `site-01-b` first, `site-01-a` second — per the self-first seed rule above. It applies to site pairs exactly as it does to the central pair: without it, `site-01-b` cannot boot while `site-01-a` is down.
|
||||
|
||||
### Site Cluster Behavior
|
||||
|
||||
- Same split-brain resolver as central (keep-oldest).
|
||||
|
||||
@@ -114,9 +114,26 @@ When a node is downed (auto-downed by its peer after a partition heals, or a kee
|
||||
1. Down ⇒ `CoordinatedShutdown` ⇒ `ActorSystem` termination.
|
||||
2. The Host watches `ActorSystem.WhenTerminated`; a termination that is **not** the host's own `StopAsync` triggers `IHostApplicationLifetime.StopApplication()`, so **the process exits**.
|
||||
3. The service supervisor restarts it — docker `restart: unless-stopped`, or Windows service recovery actions (`sc.exe failure … restart/…`).
|
||||
4. The restarted process rejoins as a **fresh incarnation** — **but only while a peer still holding cluster state is reachable**. A lone restarted node that is *not* the first seed cannot re-form a cluster on its own (see the seed-node bootstrap constraint below); it waits for its peer.
|
||||
4. The restarted process rejoins as a **fresh incarnation**, and can re-form the cluster on its own if no peer is reachable — see Seed Node Ordering below.
|
||||
|
||||
**Seed-node bootstrap constraint (still applies).** Only the FIRST seed listed in `Cluster:SeedNodes` may self-join to form a *new* cluster. All nodes list the same first seed (e.g. `scadabridge-central-a`), so a lone restarted non-first-seed node (with the first seed still down) loops on `InitJoin` forever — never `Up`, never routable. Under auto-down this constraint no longer causes the active-crash outage (the survivor keeps running and never restarts), but it still bites when a node must **boot alone** — e.g. a cold start of only the non-first-seed VM, or the survivor itself crashing while its peer is still dead. Recovery is operator-driven — either restart the first-seed node (preferred) or restart the survivor with a self-first seed override (`ScadaBridge__Cluster__SeedNodes__0` = self, `__1` = peer). The repo does not ship self-first ordering per node: with both nodes self-first a simultaneous cold start risks two independent one-node clusters that never merge.
|
||||
### Seed Node Ordering
|
||||
|
||||
**Every node lists ITSELF as `seed-nodes[0]` and its partner second (decision 2026-07-22).**
|
||||
|
||||
Akka runs two different bootstrap processes depending on that first entry. When `seed-nodes[0]` is the node's own address it runs `FirstSeedNodeProcess`: it `InitJoin`s the *other* seeds and self-joins only after `seed-node-timeout` elapses with nobody answering. When it is not, the node runs `JoinSeedNodeProcess`, which can never form a new cluster — it retries `InitJoin` indefinitely.
|
||||
|
||||
Until 2026-07-22 every node listed the same first seed, so a node that had to **boot alone** — a cold start of only the non-first-seed VM, or the survivor crashing while its peer was still dead — never reached `Up` and was never routable. That was the **registered outage gap**, and recovery was operator-driven. Self-first ordering closes it using Akka's own protocol, and `StartupValidator` fails the boot if a node config ever breaks the ordering (the invariant is silent when violated, so it is enforced loudly).
|
||||
|
||||
Behavior, covered by `SelfFirstSeedBootstrapTests` (real in-process clusters built from `BuildHocon` at production failure-detection timings):
|
||||
|
||||
| Scenario | Behavior |
|
||||
|---|---|
|
||||
| Lone cold-start, peer dead | Self-joins after `seed-node-timeout` (~5s) — operational, unattended |
|
||||
| Restart into a **live** peer | The peer answers `InitJoinAck`; the node joins the existing cluster and never islands |
|
||||
| Both cold-start simultaneously, mutually reachable | The `InitJoin` handshake resolves it *before* either self-joins → **one** 2-member cluster |
|
||||
| Both cold-start during a genuine boot-time **partition** | Each forms its own cluster — the same dual-active class `auto-down` already accepts, same recovery (restart one side) |
|
||||
|
||||
**Rejected alternative — an external self-form timer.** A watchdog that waited a configurable window for membership and then called `Cluster.Join(SelfAddress)` was implemented and discarded. Its success signal ("am I `Up` yet?") cannot distinguish *no seed answered* from *a seed answered and the join is in flight*, because it sits outside Akka's join handshake. On a routine standby restart the peer is alive but the join stalls behind removal of the restarting node's own stale incarnation (the peer must down it, then wait for the failure detector and a leader action); a `Join(self)` issued during `TryingToJoin` abandons the in-flight join and forms a second cluster at the same address. Measured: a permanent split that had not healed after 90s, converting a routine restart into an outage of the previously-healthy node. Akka's first-seed process has no such race because it is part of the handshake, which is why the ordering — not a timer — is the mechanism.
|
||||
|
||||
The docker failover drill (`docker/failover-drill.sh`) proves both directions: `standby` mode kills the younger node (active untouched, zero routing blips); `active` mode kills the active/oldest node and asserts the survivor **takes over while the victim is still down**.
|
||||
|
||||
|
||||
@@ -31,8 +31,20 @@ public class ClusterOptions
|
||||
// when the binding sites can be updated in the same commit.
|
||||
|
||||
/// <summary>
|
||||
/// Akka.NET cluster seed nodes. Both nodes are seed nodes — each node lists
|
||||
/// itself and its partner — so either can start first and form the cluster.
|
||||
/// Akka.NET cluster seed nodes. Both nodes are seed nodes — each node lists itself and its
|
||||
/// partner.
|
||||
/// <para>
|
||||
/// <b>ORDER IS LOAD-BEARING (decision 2026-07-22): every node must list ITSELF first.</b>
|
||||
/// Akka runs <c>FirstSeedNodeProcess</c> — the only bootstrap path that can form a NEW
|
||||
/// cluster when no peer answers <c>InitJoin</c> — exclusively when <c>seed-nodes[0]</c> is
|
||||
/// this node's own address; any other node runs <c>JoinSeedNodeProcess</c> and retries
|
||||
/// <c>InitJoin</c> forever. So merely listing both nodes does NOT mean either can start
|
||||
/// first: a node that lists its partner first can never cold-start while that partner is
|
||||
/// down (the "registered outage gap", <c>docker/README.md</c>). Self-first ordering closes
|
||||
/// it using Akka's own protocol, which — unlike an external self-form timer — is part of
|
||||
/// the join handshake and so cannot mistake an in-flight join for an absent peer.
|
||||
/// Enforced at boot by <c>StartupValidator</c>.
|
||||
/// </para>
|
||||
/// Must contain at least one entry.
|
||||
/// </summary>
|
||||
public List<string> SeedNodes { get; set; } = new();
|
||||
|
||||
@@ -93,6 +93,22 @@ public static class StartupValidator
|
||||
.Require("ScadaBridge:Cluster:SeedNodes",
|
||||
_ => seedNodes != null && seedNodes.Count >= 2,
|
||||
"must have at least 2 entries")
|
||||
// Self-first seed ordering (decision 2026-07-22). Akka runs FirstSeedNodeProcess —
|
||||
// the ONLY bootstrap path that can form a new cluster when no peer answers InitJoin —
|
||||
// exclusively when seed-nodes[0] is this node's own address. Every other node runs
|
||||
// JoinSeedNodeProcess and retries InitJoin forever, so a node listing its partner
|
||||
// first cannot cold-start alone: that is the "registered outage gap"
|
||||
// (docker/README.md), and it is a silent failure at boot rather than a loud one.
|
||||
// Enforced here rather than in ClusterOptionsValidator because only this validator
|
||||
// sees both the node identity and the seed list.
|
||||
.Require("ScadaBridge:Cluster:SeedNodes",
|
||||
_ => seedNodes is not { Count: >= 1 }
|
||||
|| SeedNodeIsSelf(seedNodes[0], nodeSection["NodeHostname"], port),
|
||||
"must list this node itself first: seed-nodes[0] has to be this node's own "
|
||||
+ $"'akka.tcp://scadabridge@{nodeSection["NodeHostname"]}:{port}'. Akka only lets "
|
||||
+ "seed-nodes[0] form a new cluster, so with the partner listed first this node "
|
||||
+ "can never start while its peer is down (Component-ClusterInfrastructure.md → "
|
||||
+ "Seed Node Ordering)")
|
||||
// The big Site-only block: GrpcPort/MetricsPort validity + cross-field
|
||||
// collisions + seed-node-port loop, in the original order.
|
||||
.When(role == "Site", p =>
|
||||
@@ -155,4 +171,33 @@ public static class StartupValidator
|
||||
|
||||
return int.TryParse(seedNode[(lastColon + 1)..], out var port) ? port : -1;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Extracts the host from an Akka seed-node address of the form
|
||||
/// <c>akka.tcp://system@host:port</c>. Returns an empty string when no host can be parsed.
|
||||
/// </summary>
|
||||
private static string SeedNodeHost(string seedNode)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(seedNode))
|
||||
return string.Empty;
|
||||
|
||||
var at = seedNode.LastIndexOf('@');
|
||||
var lastColon = seedNode.LastIndexOf(':');
|
||||
if (at < 0 || lastColon <= at)
|
||||
return string.Empty;
|
||||
|
||||
return seedNode[(at + 1)..lastColon];
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// True when <paramref name="seedNode"/> addresses this node itself (host AND port).
|
||||
/// Host comparison is case-insensitive because DNS names are; it is otherwise exact —
|
||||
/// Akka does no DNS canonicalisation either, so <c>node-a</c> and
|
||||
/// <c>node-a.example.com</c> are genuinely different seed identities to the cluster.
|
||||
/// </summary>
|
||||
private static bool SeedNodeIsSelf(string seedNode, string? nodeHostname, int remotingPort)
|
||||
{
|
||||
return SeedNodePort(seedNode) == remotingPort
|
||||
&& string.Equals(SeedNodeHost(seedNode), nodeHostname, StringComparison.OrdinalIgnoreCase);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -124,6 +124,10 @@ public class StartupValidatorTests
|
||||
{
|
||||
var values = ValidCentralConfig();
|
||||
values["ScadaBridge:Node:RemotingPort"] = port;
|
||||
// The self-first seed rule (2026-07-22) compares host AND port, so this node's own
|
||||
// seed entry moves with its remoting port — otherwise this port-range test would be
|
||||
// asserting against a config that is inconsistent for an unrelated reason.
|
||||
values["ScadaBridge:Cluster:SeedNodes:0"] = $"akka.tcp://scadabridge@central-node1:{port}";
|
||||
var config = BuildConfig(values);
|
||||
|
||||
var ex = Record.Exception(() => StartupValidator.Validate(config));
|
||||
@@ -273,6 +277,52 @@ public class StartupValidatorTests
|
||||
Assert.Contains("SeedNodes must have at least 2 entries", ex.Message);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void PeerFirstSeedOrder_FailsValidation()
|
||||
{
|
||||
// Decision 2026-07-22: every node must list ITSELF as seed-nodes[0]. Akka only runs
|
||||
// FirstSeedNodeProcess (the process that can form a new cluster when no peer answers)
|
||||
// when seed-nodes[0] is this node's own address; with the peer first the node can
|
||||
// never cold-start alone — the "registered outage gap".
|
||||
var values = ValidCentralConfig();
|
||||
values["ScadaBridge:Cluster:SeedNodes:0"] = "akka.tcp://scadabridge@central-node2:8081";
|
||||
values["ScadaBridge:Cluster:SeedNodes:1"] = "akka.tcp://scadabridge@central-node1:8081";
|
||||
var config = BuildConfig(values);
|
||||
|
||||
var ex = Assert.Throws<InvalidOperationException>(() => StartupValidator.Validate(config));
|
||||
Assert.Contains("SeedNodes", ex.Message);
|
||||
Assert.Contains("must list this node itself first", ex.Message);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SelfFirstSeedOrder_OnASiteNode_PassesValidation()
|
||||
{
|
||||
// Positive control for the rule above: the shipped ordering must validate on a Site
|
||||
// node too (the rule is unconditional, not Central-only).
|
||||
var config = BuildConfig(ValidSiteConfig());
|
||||
|
||||
var ex = Record.Exception(() => StartupValidator.Validate(config));
|
||||
|
||||
Assert.Null(ex);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SelfFirstSeed_MatchedOnHostAndPort_NotJustHost()
|
||||
{
|
||||
// Both nodes of a pair can share a hostname when they differ by port (a two-node
|
||||
// dev/loopback install). The rule must compare host AND port, or such a node passes
|
||||
// while actually being the non-first seed.
|
||||
var values = ValidCentralConfig();
|
||||
values["ScadaBridge:Node:NodeHostname"] = "localhost";
|
||||
values["ScadaBridge:Node:RemotingPort"] = "8082";
|
||||
values["ScadaBridge:Cluster:SeedNodes:0"] = "akka.tcp://scadabridge@localhost:8081";
|
||||
values["ScadaBridge:Cluster:SeedNodes:1"] = "akka.tcp://scadabridge@localhost:8082";
|
||||
var config = BuildConfig(values);
|
||||
|
||||
var ex = Assert.Throws<InvalidOperationException>(() => StartupValidator.Validate(config));
|
||||
Assert.Contains("must list this node itself first", ex.Message);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("0")]
|
||||
[InlineData("-1")]
|
||||
|
||||
@@ -0,0 +1,152 @@
|
||||
using Akka.Actor;
|
||||
using Akka.Cluster;
|
||||
using Akka.Configuration;
|
||||
using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
|
||||
using ZB.MOM.WW.ScadaBridge.Host;
|
||||
using ZB.MOM.WW.ScadaBridge.Host.Actors;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;
|
||||
|
||||
/// <summary>
|
||||
/// Guards the self-first seed-ordering invariant (decision 2026-07-22): every node lists
|
||||
/// ITSELF as <c>seed-nodes[0]</c> and its partner second.
|
||||
///
|
||||
/// <para><b>Why the ordering is the mechanism.</b> Akka runs a different bootstrap process
|
||||
/// depending on whether <c>seed-nodes[0]</c> is this node's own address. When it is,
|
||||
/// <c>FirstSeedNodeProcess</c> runs: it InitJoins the OTHER seeds and self-joins only after
|
||||
/// <c>seed-node-timeout</c> passes with nobody answering. When it is not,
|
||||
/// <c>JoinSeedNodeProcess</c> runs, which can never form a new cluster — it retries InitJoin
|
||||
/// forever. That is the "registered outage gap" (docker/README.md): a lone cold-starting
|
||||
/// non-first seed never came Up. Listing self first closes it using Akka's own protocol.</para>
|
||||
///
|
||||
/// <para><b>Why not an external self-form watchdog.</b> A timer that waits N seconds for
|
||||
/// membership and then calls <c>Cluster.Join(SelfAddress)</c> cannot see Akka's join
|
||||
/// handshake, so it cannot distinguish "no seed answered" from "a seed answered and the join
|
||||
/// is in flight". <see cref="Restarting_node_rejoins_its_live_peer_instead_of_self_forming"/>
|
||||
/// is the case that killed that design: on a routine standby restart the peer is alive but
|
||||
/// the join is stalled behind removal of this node's own stale incarnation, and a Join(self)
|
||||
/// issued during <c>TryingToJoin</c> abandons the in-flight join and forms a SECOND cluster at
|
||||
/// the same address — a permanent split (measured: still split after 90s). Akka's own
|
||||
/// first-seed process has no such race because it is part of the handshake.</para>
|
||||
///
|
||||
/// These build REAL clusters from the production <c>BuildHocon</c> output, like
|
||||
/// <see cref="TwoNodeClusterFixture"/>, at production failure-detection timings.
|
||||
/// </summary>
|
||||
public sealed class SelfFirstSeedBootstrapTests : IAsyncLifetime
|
||||
{
|
||||
private readonly List<ActorSystem> _systems = new();
|
||||
|
||||
/// <summary>Starts a node configured the way every shipped node appsettings is: its own
|
||||
/// address first, the partner second. <paramref name="selfFirst"/> = false reproduces the
|
||||
/// pre-fix ordering, which is what makes the outage-gap assertions falsifiable.</summary>
|
||||
private ActorSystem StartNode(int selfPort, int peerPort, bool selfFirst = true)
|
||||
{
|
||||
var nodeOptions = new NodeOptions { Role = "Central", NodeHostname = "127.0.0.1", RemotingPort = selfPort };
|
||||
var self = $"akka.tcp://scadabridge@127.0.0.1:{selfPort}";
|
||||
var peer = $"akka.tcp://scadabridge@127.0.0.1:{peerPort}";
|
||||
var clusterOptions = new ClusterOptions
|
||||
{
|
||||
SeedNodes = selfFirst ? new List<string> { self, peer } : new List<string> { peer, self },
|
||||
// Production failure-detection envelope: this is what sets how long a restarting
|
||||
// node's join stays stalled behind its own stale incarnation.
|
||||
StableAfter = TimeSpan.FromSeconds(15),
|
||||
HeartbeatInterval = TimeSpan.FromSeconds(2),
|
||||
FailureDetectionThreshold = TimeSpan.FromSeconds(10),
|
||||
MinNrOfMembers = 1,
|
||||
};
|
||||
var hocon = AkkaHostedService.BuildHocon(
|
||||
nodeOptions, clusterOptions, new[] { "Central" },
|
||||
TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(3));
|
||||
// Crash simulation: Terminate() must NOT run CoordinatedShutdown (no Leave gossip),
|
||||
// matching TwoNodeClusterFixture — otherwise a "restart after crash" degenerates to
|
||||
// the graceful path where the peer has already removed the old member.
|
||||
var config = ConfigurationFactory
|
||||
.ParseString("akka.coordinated-shutdown.run-by-actor-system-terminate = off")
|
||||
.WithFallback(ConfigurationFactory.ParseString(hocon));
|
||||
var system = ActorSystem.Create("scadabridge", config);
|
||||
_systems.Add(system);
|
||||
return system;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Lone_cold_start_forms_a_cluster_when_the_peer_is_dead()
|
||||
{
|
||||
var selfPort = TwoNodeClusterFixture.GetFreeTcpPort();
|
||||
var deadPeerPort = TwoNodeClusterFixture.GetFreeTcpPort(); // nothing listening
|
||||
var node = StartNode(selfPort, deadPeerPort);
|
||||
|
||||
// Akka's FirstSeedNodeProcess self-joins after seed-node-timeout (~5s) once the dead
|
||||
// peer fails to answer InitJoin. This is the unattended cold-start-alone guarantee.
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(node, 1, TimeSpan.FromSeconds(30));
|
||||
Assert.Equal(MemberStatus.Up, Akka.Cluster.Cluster.Get(node).SelfMember.Status);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Peer_first_ordering_is_the_outage_gap_and_never_forms()
|
||||
{
|
||||
// FALSIFIABILITY CONTROL for the test above: with the OLD ordering (peer first) the
|
||||
// identical scenario never comes Up. If this test ever starts passing quickly, the
|
||||
// self-first ordering has stopped being the thing doing the work.
|
||||
var selfPort = TwoNodeClusterFixture.GetFreeTcpPort();
|
||||
var deadPeerPort = TwoNodeClusterFixture.GetFreeTcpPort();
|
||||
var node = StartNode(selfPort, deadPeerPort, selfFirst: false);
|
||||
|
||||
await Task.Delay(TimeSpan.FromSeconds(15)); // 3x the seed-node-timeout
|
||||
var cluster = Akka.Cluster.Cluster.Get(node);
|
||||
Assert.Empty(cluster.State.Members); // still InitJoin-looping — the registered outage gap
|
||||
|
||||
// Positive control: the node was formable all along; only the ordering blocked it.
|
||||
cluster.Join(cluster.SelfAddress);
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(node, 1, TimeSpan.FromSeconds(30));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Restarting_node_rejoins_its_live_peer_instead_of_self_forming()
|
||||
{
|
||||
// The case that rejected the self-form-watchdog design (code review 2026-07-22, C1):
|
||||
// a routine standby restart while the peer is alive must rejoin, never island.
|
||||
var portA = TwoNodeClusterFixture.GetFreeTcpPort();
|
||||
var portB = TwoNodeClusterFixture.GetFreeTcpPort();
|
||||
|
||||
var nodeA = StartNode(portA, portB);
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 1, TimeSpan.FromSeconds(30));
|
||||
var nodeB = StartNode(portB, portA);
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 2, TimeSpan.FromSeconds(30));
|
||||
|
||||
// Hard-crash B and immediately restart it at the SAME address, as a container
|
||||
// recreate / service restart does.
|
||||
await nodeB.Terminate().WaitAsync(TimeSpan.FromSeconds(10));
|
||||
var nodeB2 = StartNode(portB, portA);
|
||||
|
||||
// Both must converge on ONE 2-member cluster — never two 1-member clusters.
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(nodeB2, 2, TimeSpan.FromSeconds(90));
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 2, TimeSpan.FromSeconds(90));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Both_nodes_cold_starting_together_converge_on_one_cluster()
|
||||
{
|
||||
// Self-first on BOTH nodes raises the obvious question: does a simultaneous cold
|
||||
// start produce two clusters? While they are mutually reachable it does not — the
|
||||
// InitJoin handshake resolves it before either self-joins. (A genuine boot-time
|
||||
// PARTITION would still split, which is the same class auto-down already accepts.)
|
||||
var portA = TwoNodeClusterFixture.GetFreeTcpPort();
|
||||
var portB = TwoNodeClusterFixture.GetFreeTcpPort();
|
||||
|
||||
var nodeA = StartNode(portA, portB);
|
||||
var nodeB = StartNode(portB, portA);
|
||||
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 2, TimeSpan.FromSeconds(60));
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(nodeB, 2, TimeSpan.FromSeconds(60));
|
||||
}
|
||||
|
||||
public Task InitializeAsync() => Task.CompletedTask;
|
||||
|
||||
public async Task DisposeAsync()
|
||||
{
|
||||
foreach (var s in _systems)
|
||||
{
|
||||
try { await s.Terminate().WaitAsync(TimeSpan.FromSeconds(10)); } catch { /* teardown */ }
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user