fix(cluster): don't self-form while a seed peer is reachable — the fallback islanded a restarting node

The live gate for the manual-failover control caught the self-form fallback
forming a SECOND cluster. A node bounced by a failover restarted, received
InitJoinAck from its live peer — the join was in flight and healthy — but did
not get the Welcome inside the 10s window, because the peer's ring still held
the node's previous incarnation (Exiting -> Down -> Removed). The fallback
fired on the timer and the node islanded itself until an operator restarted it.

The original design assumed Cluster.Join(SelfAddress) would be ignored
mid-handshake. It is not — it wins. And since manual failover deliberately
produces that restart, every failover could island the node it bounced.

Before self-forming, TCP-probe the other seed addresses; a reachable peer means
wait another window instead. That is also what the fallback claims to detect:
'no seed answered InitJoin (peer down at boot)'.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-22 07:03:07 -04:00
parent d8a85c3d89
commit ea45ace1c3
3 changed files with 189 additions and 20 deletions
+18 -3
View File
@@ -314,7 +314,21 @@ operational unattended.
| `"00:00:10"` (**default**) | A node with no membership after 10 s self-forms. A live peer answers `InitJoin` in milliseconds, so on any normal boot the fallback never fires. |
| `null` / `≤ 0` | Disabled — the pre-2026-07-22 behaviour: wait on `InitJoin` indefinitely. |
**The island guard is the load-bearing part.** The fallback fires **only when this node's own address
**Two guards keep the fallback from creating the split it exists to avoid.**
**1 — Reachability.** An expired window is not evidence the peer is down, so before self-forming the
fallback TCP-probes the other seed addresses; if any accepts a connection it waits another window instead.
This was added after the live gate below caught the fallback islanding a node: a node bounced by a manual
failover restarted, received `InitJoinAck` from its live peer — a join in flight and healthy — but did not
get the Welcome inside the window, because the peer's ring still held the node's previous incarnation
(`Exiting``Down``Removed`). The fallback fired on the timer and the node formed a **second cluster**.
The original design assumed `Cluster.Join(SelfAddress)` would be ignored mid-handshake; **it is not — it
wins.** Since manual failover deliberately produces exactly that restart, without this guard every manual
failover could island the node it bounced. Pinned by
`SelfFormBootstrapTests.Reachable_peer_suppresses_self_forming_until_it_goes_away`, whose "peer" is a bare
`TcpListener` that speaks no Akka: reachable, never answering a join.
**2 — Seed membership.** The fallback fires **only when this node's own address
appears in its own `SeedNodes`**. A node that is not one of its own seeds is never legitimately first, and
if it self-formed, a later-booting real seed would form a second cluster the two could never merge. That is
exactly the current docker-dev topology: the site-a/site-b driver nodes list only `central-1` as a seed, so
@@ -330,8 +344,9 @@ strategy already accepts, with the same recovery (restart one side).
Pinned by `SelfFormBootstrapTests` (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/`), which starts real hosts
through the production bootstrap: a lone non-first seed comes Up; a disabled window keeps waiting; a node
absent from its own seed list never self-forms. The two negative cases each carry a positive control (an
explicit self-join) so they cannot pass merely because the node was unformable.
absent from its own seed list never self-forms; a node whose peer is merely *reachable* keeps waiting. Every
negative case carries a positive control (an explicit self-join, or dropping the peer listener) so none can
pass merely because the node was unformable.
> **Live gate: PASSED (docker-dev, 2026-07-22).** Drilled on the six-node rig, defect first and fix
> second. On the pre-change image, `central-2` started alone (central-1 stopped) and logged **zero** join