docs(cluster): self-first seed ordering replaces the self-form fallback
docs/Redundancy.md's bootstrap section is rewritten around the mechanism that is
actually in force: which of Akka's two bootstrap processes runs is decided by
whether seed-nodes[0] is this node's own address, so the ORDER of Cluster:SeedNodes
is the fix, not a timer. Adds the per-process table, the shipped self-first
example, why the validator is conditional (site nodes are not seeds) and why it
matches PublicHostname rather than the 0.0.0.0 bind address.
The retirement is documented rather than erased: a new subsection explains that a
watchdog outside the join handshake cannot tell "no seed answered" from "a seed
answered and the join is in flight", and the ea45ace1 live-gate finding is kept
verbatim as the evidence that produced that conclusion (InitJoinAck at 10:49:05,
old incarnation retired 10:49:06, watchdog fired 10:49:15, second cluster). The
watchdog's own earlier PASS is kept too — it did pass what it was gated on.
Also: a live gate for the NEW mechanism is registered as outstanding (not
re-drilled on the docker-dev rig since the swap), and the plan doc that shipped
the watchdog gets a superseded banner rather than an edit, so the execution record
and its Task 8 finding stay readable.
Ripple: CLAUDE.md cluster section, docs/Configuration.md + docs/v2/Cluster.md +
docs/ServiceHosting.md SeedNodes entries, the per-cluster-mesh design doc's two
"CLOSED by SelfFormAfter" notes, and mesh-program Phase 6 (which had this
convergence queued — now marked done early) + Phase 7's live-gate name.
This commit is contained in:
@@ -46,8 +46,9 @@ Two independent 2-node Akka clusters per site — they share hardware, never a m
|
||||
| OtOpcUa LocalDb sync port | OtOpcUa driver | h2c LocalDb pair replication (default off/0 today — Phase 6 assigns a real per-site port) |
|
||||
|
||||
**Aligned HA posture (both products, per VM — already true or landing via the selfform plan):**
|
||||
auto-down downing (15 s window), oldest-Up active/primary election, `SelfFormAfter` 10 s
|
||||
self-form fallback, termination-watchdog → process exit → `sc.exe failure` restart recovery.
|
||||
auto-down downing (15 s window), oldest-Up active/primary election, self-first seed ordering
|
||||
(the `SelfFormAfter` self-form fallback that briefly stood in for it was retired 2026-07-22),
|
||||
termination-watchdog → process exit → `sc.exe failure` restart recovery.
|
||||
One failover story for operators regardless of product.
|
||||
|
||||
**Prerequisite ordering:** the fallback/manual-failover plan
|
||||
@@ -143,12 +144,14 @@ observability).
|
||||
kill-and-reconnect of the central dialer recovers every stream.
|
||||
|
||||
### Phase 6 — Mesh partition + co-location topology
|
||||
**Scope:** per-cluster seed nodes — **adopt ScadaBridge's self-first ordering (each node lists
|
||||
ITSELF as `seed-nodes[0]`, partner second) and RETIRE the `SelfFormAfter` watchdog + TCP
|
||||
reachability guard** (2026-07-22 execution finding: the watchdog races Akka's join handshake —
|
||||
`Join(self)` during an in-flight join islands the node, hit live here and fixed with the guard;
|
||||
ScadaBridge proved self-first ordering is the race-free form of the same semantics, enforced by
|
||||
a startup-validator rule — port that rule too); cluster-scoped roles `cluster-{ClusterId}` + singleton re-scoping,
|
||||
**Scope:** per-cluster seed nodes — ~~adopt ScadaBridge's self-first ordering and RETIRE the
|
||||
`SelfFormAfter` watchdog + TCP reachability guard~~ **DONE EARLY 2026-07-22** (docker-dev
|
||||
`central-2` swapped to self-first, `ClusterBootstrapFallback`/`SelfFormAfter` deleted,
|
||||
`AkkaClusterOptionsValidator` ports ScadaBridge's startup rule in its conditional form, and
|
||||
`SelfFirstSeedBootstrapTests` replaces `SelfFormBootstrapTests`; see `docs/Redundancy.md`
|
||||
§"Bootstrap: self-first seed ordering"). What remains for this phase is the per-pair
|
||||
`Cluster__SeedNodes__*` matrix once the meshes actually split — every node in a pair is then a seed
|
||||
of its own mesh, so the site-node exemption disappears; cluster-scoped roles `cluster-{ClusterId}` + singleton re-scoping,
|
||||
central pair keeps the admin singletons; **docker-dev rig rewritten** to model the real topology —
|
||||
including the co-location port table above (both products' compose files on shared per-site
|
||||
networks, real LocalDb sync ports); remove the ClusterRedundancy page's mesh-scope caveat (the
|
||||
@@ -163,12 +166,13 @@ cross separate meshes).
|
||||
### Phase 7 — Failover drill + live gates
|
||||
**Scope:** the drill ScadaBridge already has (`failover-drill.sh` analogue) run per pair, both
|
||||
directions; **close the two outstanding live gates**: (a) auto-down 1-vs-1 crash-the-oldest
|
||||
(deferred since Phase 0a — finally testable, every mesh is exactly two nodes), (b) `SelfFormAfter`
|
||||
(deferred since Phase 0a — finally testable, every mesh is exactly two nodes), (b) self-first
|
||||
lone-cold-start on the real per-pair topology; manual-failover button re-verified per pair;
|
||||
operator runbook for the co-located site (one page covering both products' failover on the same
|
||||
two VMs).
|
||||
**Exit gate:** drill green on every pair type (central, site); runbook merged; design doc §7
|
||||
table fully marked DONE.
|
||||
table fully marked DONE. (Live gate (b) is now the **self-first cold-start-alone** drill, not
|
||||
`SelfFormAfter` — the watchdog it named was retired 2026-07-22.)
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user