fix(cluster): don't self-form while a seed peer is reachable — the fallback islanded a restarting node
The live gate for the manual-failover control caught the self-form fallback forming a SECOND cluster. A node bounced by a failover restarted, received InitJoinAck from its live peer — the join was in flight and healthy — but did not get the Welcome inside the 10s window, because the peer's ring still held the node's previous incarnation (Exiting -> Down -> Removed). The fallback fired on the timer and the node islanded itself until an operator restarted it. The original design assumed Cluster.Join(SelfAddress) would be ignored mid-handshake. It is not — it wins. And since manual failover deliberately produces that restart, every failover could island the node it bounced. Before self-forming, TCP-probe the other seed addresses; a reachable peer means wait another window instead. That is also what the fallback claims to detect: 'no seed answered InitJoin (peer down at boot)'. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
+18
-3
@@ -314,7 +314,21 @@ operational unattended.
|
||||
| `"00:00:10"` (**default**) | A node with no membership after 10 s self-forms. A live peer answers `InitJoin` in milliseconds, so on any normal boot the fallback never fires. |
|
||||
| `null` / `≤ 0` | Disabled — the pre-2026-07-22 behaviour: wait on `InitJoin` indefinitely. |
|
||||
|
||||
**The island guard is the load-bearing part.** The fallback fires **only when this node's own address
|
||||
**Two guards keep the fallback from creating the split it exists to avoid.**
|
||||
|
||||
**1 — Reachability.** An expired window is not evidence the peer is down, so before self-forming the
|
||||
fallback TCP-probes the other seed addresses; if any accepts a connection it waits another window instead.
|
||||
This was added after the live gate below caught the fallback islanding a node: a node bounced by a manual
|
||||
failover restarted, received `InitJoinAck` from its live peer — a join in flight and healthy — but did not
|
||||
get the Welcome inside the window, because the peer's ring still held the node's previous incarnation
|
||||
(`Exiting` → `Down` → `Removed`). The fallback fired on the timer and the node formed a **second cluster**.
|
||||
The original design assumed `Cluster.Join(SelfAddress)` would be ignored mid-handshake; **it is not — it
|
||||
wins.** Since manual failover deliberately produces exactly that restart, without this guard every manual
|
||||
failover could island the node it bounced. Pinned by
|
||||
`SelfFormBootstrapTests.Reachable_peer_suppresses_self_forming_until_it_goes_away`, whose "peer" is a bare
|
||||
`TcpListener` that speaks no Akka: reachable, never answering a join.
|
||||
|
||||
**2 — Seed membership.** The fallback fires **only when this node's own address
|
||||
appears in its own `SeedNodes`**. A node that is not one of its own seeds is never legitimately first, and
|
||||
if it self-formed, a later-booting real seed would form a second cluster the two could never merge. That is
|
||||
exactly the current docker-dev topology: the site-a/site-b driver nodes list only `central-1` as a seed, so
|
||||
@@ -330,8 +344,9 @@ strategy already accepts, with the same recovery (restart one side).
|
||||
|
||||
Pinned by `SelfFormBootstrapTests` (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/`), which starts real hosts
|
||||
through the production bootstrap: a lone non-first seed comes Up; a disabled window keeps waiting; a node
|
||||
absent from its own seed list never self-forms. The two negative cases each carry a positive control (an
|
||||
explicit self-join) so they cannot pass merely because the node was unformable.
|
||||
absent from its own seed list never self-forms; a node whose peer is merely *reachable* keeps waiting. Every
|
||||
negative case carries a positive control (an explicit self-join, or dropping the peer listener) so none can
|
||||
pass merely because the node was unformable.
|
||||
|
||||
> **Live gate: PASSED (docker-dev, 2026-07-22).** Drilled on the six-node rig, defect first and fix
|
||||
> second. On the pre-change image, `central-2` started alone (central-1 stopped) and logged **zero** join
|
||||
|
||||
Reference in New Issue
Block a user