docs(cluster): self-first seed ordering replaces the self-form fallback
v2-ci / build (push) Successful in 3m44s
v2-ci / unit-tests (push) Failing after 10m7s

docs/Redundancy.md's bootstrap section is rewritten around the mechanism that is
actually in force: which of Akka's two bootstrap processes runs is decided by
whether seed-nodes[0] is this node's own address, so the ORDER of Cluster:SeedNodes
is the fix, not a timer. Adds the per-process table, the shipped self-first
example, why the validator is conditional (site nodes are not seeds) and why it
matches PublicHostname rather than the 0.0.0.0 bind address.

The retirement is documented rather than erased: a new subsection explains that a
watchdog outside the join handshake cannot tell "no seed answered" from "a seed
answered and the join is in flight", and the ea45ace1 live-gate finding is kept
verbatim as the evidence that produced that conclusion (InitJoinAck at 10:49:05,
old incarnation retired 10:49:06, watchdog fired 10:49:15, second cluster). The
watchdog's own earlier PASS is kept too — it did pass what it was gated on.

Also: a live gate for the NEW mechanism is registered as outstanding (not
re-drilled on the docker-dev rig since the swap), and the plan doc that shipped
the watchdog gets a superseded banner rather than an edit, so the execution record
and its Task 8 finding stay readable.

Ripple: CLAUDE.md cluster section, docs/Configuration.md + docs/v2/Cluster.md +
docs/ServiceHosting.md SeedNodes entries, the per-cluster-mesh design doc's two
"CLOSED by SelfFormAfter" notes, and mesh-program Phase 6 (which had this
convergence queued — now marked done early) + Phase 7's live-gate name.
This commit is contained in:
Joseph Doherty
2026-07-22 07:46:02 -04:00
parent 3f24d4d6bf
commit 9af935f237
9 changed files with 132 additions and 81 deletions
@@ -85,11 +85,13 @@ Their failover drill has a mode that exists *"to make the registered gap observa
it is covered."* **A two-node keep-oldest mesh has an acknowledged total-outage hole.** If OtOpcUa
adopts 2-node meshes, this must be decided deliberately, not inherited.
> **CLOSED 2026-07-22.** The `InitJoin` half of this gap is fixed in both repos by the self-form
> fallback: `Cluster:SelfFormAfter` (default 10 s) lets a node that is one of its own seeds form a
> cluster on itself when no seed answers. See `docs/Redundancy.md` §"Bootstrap: the self-form
> fallback" — including the island guard that keeps today's non-seed site nodes waiting, which the
> per-cluster mesh removes by making every pair node a seed of its own mesh.
> **CLOSED 2026-07-22.** The `InitJoin` half of this gap is fixed in both repos by **self-first seed
> ordering**: every node that is one of its own seeds lists ITSELF as `seed-nodes[0]`, which is the
> only ordering under which Akka runs `FirstSeedNodeProcess` and can form a cluster with no peer
> answering. (An earlier `Cluster:SelfFormAfter` watchdog was tried here and retired — it raced the
> join handshake.) See `docs/Redundancy.md` §"Bootstrap: self-first seed ordering". Today's
> driver-only site nodes are seeded by `central-1` only and are exempt; the per-cluster mesh removes
> even that exception by making every pair node a seed of its own mesh.
### Both inter-cluster transports are unauthenticated
@@ -261,11 +263,14 @@ with Phase 6/7, which makes every mesh exactly two nodes.
**The bootstrap half of the gap is also closed (2026-07-22).** Downing was only ever one of the two
outage modes: even with `auto-down`, a node cold-starting while its peer was dead looped on `InitJoin`
forever, because Akka lets only the first listed seed form a new cluster. `Cluster:SelfFormAfter`
(default 10 s, `ClusterBootstrapFallback`) now self-forms in that case, guarded so a node absent from
its own seed list never islands itself. Pinned by `SelfFormBootstrapTests`; see `docs/Redundancy.md`
§"Bootstrap: the self-form fallback". Under this design each pair node is a seed of its own two-node
mesh, so the fallback covers both sides and the current site-node exception disappears.
forever, because Akka lets only `seed-nodes[0]` form a new cluster. The fix is **self-first seed
ordering** — each seed node lists itself first, enforced at boot by `AkkaClusterOptionsValidator`.
The `Cluster:SelfFormAfter` watchdog first shipped for this is deleted: a timer outside Akka's join
handshake cannot tell "no seed answered" from "a seed answered and the join is in flight", and it
islanded a manually-failed-over node live. Pinned by `SelfFirstSeedBootstrapTests`; see
`docs/Redundancy.md` §"Bootstrap: self-first seed ordering". Under this design each pair node is a
seed of its own two-node mesh, so the ordering covers both sides and the current site-node exemption
disappears.
### 6.3 DECIDED — match ScadaBridge's auth posture for now