docs(cluster): self-first seed ordering replaces the self-form fallback
v2-ci / build (push) Successful in 3m44s
v2-ci / unit-tests (push) Failing after 10m7s

docs/Redundancy.md's bootstrap section is rewritten around the mechanism that is
actually in force: which of Akka's two bootstrap processes runs is decided by
whether seed-nodes[0] is this node's own address, so the ORDER of Cluster:SeedNodes
is the fix, not a timer. Adds the per-process table, the shipped self-first
example, why the validator is conditional (site nodes are not seeds) and why it
matches PublicHostname rather than the 0.0.0.0 bind address.

The retirement is documented rather than erased: a new subsection explains that a
watchdog outside the join handshake cannot tell "no seed answered" from "a seed
answered and the join is in flight", and the ea45ace1 live-gate finding is kept
verbatim as the evidence that produced that conclusion (InitJoinAck at 10:49:05,
old incarnation retired 10:49:06, watchdog fired 10:49:15, second cluster). The
watchdog's own earlier PASS is kept too — it did pass what it was gated on.

Also: a live gate for the NEW mechanism is registered as outstanding (not
re-drilled on the docker-dev rig since the swap), and the plan doc that shipped
the watchdog gets a superseded banner rather than an edit, so the execution record
and its Task 8 finding stay readable.

Ripple: CLAUDE.md cluster section, docs/Configuration.md + docs/v2/Cluster.md +
docs/ServiceHosting.md SeedNodes entries, the per-cluster-mesh design doc's two
"CLOSED by SelfFormAfter" notes, and mesh-program Phase 6 (which had this
convergence queued — now marked done early) + Phase 7's live-gate name.
This commit is contained in:
Joseph Doherty
2026-07-22 07:46:02 -04:00
parent 3f24d4d6bf
commit 9af935f237
9 changed files with 132 additions and 81 deletions
@@ -46,8 +46,9 @@ Two independent 2-node Akka clusters per site — they share hardware, never a m
| OtOpcUa LocalDb sync port | OtOpcUa driver | h2c LocalDb pair replication (default off/0 today — Phase 6 assigns a real per-site port) |
**Aligned HA posture (both products, per VM — already true or landing via the selfform plan):**
auto-down downing (15 s window), oldest-Up active/primary election, `SelfFormAfter` 10 s
self-form fallback, termination-watchdog → process exit → `sc.exe failure` restart recovery.
auto-down downing (15 s window), oldest-Up active/primary election, self-first seed ordering
(the `SelfFormAfter` self-form fallback that briefly stood in for it was retired 2026-07-22),
termination-watchdog → process exit → `sc.exe failure` restart recovery.
One failover story for operators regardless of product.
**Prerequisite ordering:** the fallback/manual-failover plan
@@ -143,12 +144,14 @@ observability).
kill-and-reconnect of the central dialer recovers every stream.
### Phase 6 — Mesh partition + co-location topology
**Scope:** per-cluster seed nodes — **adopt ScadaBridge's self-first ordering (each node lists
ITSELF as `seed-nodes[0]`, partner second) and RETIRE the `SelfFormAfter` watchdog + TCP
reachability guard** (2026-07-22 execution finding: the watchdog races Akka's join handshake —
`Join(self)` during an in-flight join islands the node, hit live here and fixed with the guard;
ScadaBridge proved self-first ordering is the race-free form of the same semantics, enforced by
a startup-validator rule — port that rule too); cluster-scoped roles `cluster-{ClusterId}` + singleton re-scoping,
**Scope:** per-cluster seed nodes — ~~adopt ScadaBridge's self-first ordering and RETIRE the
`SelfFormAfter` watchdog + TCP reachability guard~~ **DONE EARLY 2026-07-22** (docker-dev
`central-2` swapped to self-first, `ClusterBootstrapFallback`/`SelfFormAfter` deleted,
`AkkaClusterOptionsValidator` ports ScadaBridge's startup rule in its conditional form, and
`SelfFirstSeedBootstrapTests` replaces `SelfFormBootstrapTests`; see `docs/Redundancy.md`
§"Bootstrap: self-first seed ordering"). What remains for this phase is the per-pair
`Cluster__SeedNodes__*` matrix once the meshes actually split — every node in a pair is then a seed
of its own mesh, so the site-node exemption disappears; cluster-scoped roles `cluster-{ClusterId}` + singleton re-scoping,
central pair keeps the admin singletons; **docker-dev rig rewritten** to model the real topology —
including the co-location port table above (both products' compose files on shared per-site
networks, real LocalDb sync ports); remove the ClusterRedundancy page's mesh-scope caveat (the
@@ -163,12 +166,13 @@ cross separate meshes).
### Phase 7 — Failover drill + live gates
**Scope:** the drill ScadaBridge already has (`failover-drill.sh` analogue) run per pair, both
directions; **close the two outstanding live gates**: (a) auto-down 1-vs-1 crash-the-oldest
(deferred since Phase 0a — finally testable, every mesh is exactly two nodes), (b) `SelfFormAfter`
(deferred since Phase 0a — finally testable, every mesh is exactly two nodes), (b) self-first
lone-cold-start on the real per-pair topology; manual-failover button re-verified per pair;
operator runbook for the co-located site (one page covering both products' failover on the same
two VMs).
**Exit gate:** drill green on every pair type (central, site); runbook merged; design doc §7
table fully marked DONE.
table fully marked DONE. (Live gate (b) is now the **self-first cold-start-alone** drill, not
`SelfFormAfter` — the watchdog it named was retired 2026-07-22.)
---