docs: correct the self-form watchdog record — rejected, not shipped
The InitJoin self-form watchdog (a timer calling Cluster.Join(SelfAddress) after a window) was implemented and REJECTED after live testing: it sits outside Akka's join handshake, so on a routine standby restart — peer alive, join stalled behind removal of the node's own stale incarnation — it fires mid-join, abandons the in-flight join and forms a permanent second cluster at the same address (measured: still split after 90s; OtOpcUa hit it live in its manual-failover drill). The safety argument 'a booting node only self-forms when no seed answers InitJoin' is true of Akka's NATIVE first-seed rule (FirstSeedNodeProcess, inside the handshake) and FALSE of the watchdog, which can fire while a seed is answering. That reasoning error is now called out explicitly so no future plan re-derives the pattern. - docs/plans/2026-07-22-initjoin-selfform-fallback.md: dated correction banner (history preserved) + the two surviving false claims struck in place (behavior-spec row and the Mechanism section's 'races are benign'); points at ScadaBridge Component-ClusterInfrastructure.md as the surviving design. - akka_failover.md 6.1: shipped design stated as self-first seed ordering plus the opt-in, default-OFF simultaneous-cold-start bootstrap guard; added a never-re-derive-the-watchdog note. 6.2: two items my 6.1 edit contradicted are marked closed (OtOpcUa auto-down live gate, mesh election scope — both done 2026-07-24). - CLAUDE.md: ScadaBridge issue #14 is CLOSED (2026-07-24), not open — v3 dual-namespace live gates all passed, ScadaBridge consumes OtOpcUa v3 end-to-end.
This commit is contained in:
+41
-7
@@ -449,7 +449,7 @@ is always "restart one side; it rejoins as youngest/standby."
|
||||
|
||||
## 6. Gaps vs the stated HA goal ("either node first, automatic failover")
|
||||
|
||||
### 6.1 The seed-node bootstrap constraint — CLOSED 2026-07-22 (differently per repo)
|
||||
### 6.1 The seed-node bootstrap constraint — CLOSED 2026-07-22 (self-form watchdog REJECTED; self-first ordering + opt-in bootstrap guard shipped in both repos)
|
||||
**Automatic failover: achieved.** Either node crashing is survivable, unattended, in
|
||||
~27–30 s. The remaining gap — a non-first seed cold-starting alone loops on `InitJoin`
|
||||
forever (the "registered outage gap") — was **closed in both repos on 2026-07-22**, but
|
||||
@@ -493,18 +493,52 @@ ordering test failed at 11 s against the watchdog — demonstrating the watchdog
|
||||
never usefully fire for the nodes that needed it.
|
||||
|
||||
**Result: both repos now share the identical mechanism** — self-first seed ordering,
|
||||
validator-enforced, no bootstrap runtime code. The mesh program's Phase 6 convergence
|
||||
item is already done; what remains there is only the per-pair seed topology itself.
|
||||
validator-enforced, with no self-form timer anywhere. OtOpcUa's mesh program has since
|
||||
completed (Phase 7, `c50ebcf7`, 2026-07-24), so the per-pair seed topology is in place too.
|
||||
|
||||
**Never re-derive the watchdog from the InitJoin argument (correction, 2026-08-01).** The
|
||||
recurring justification — "a booting node only self-forms when no seed answers InitJoin,
|
||||
so a self-form timer is safe" — describes **Akka's native first-seed rule**, which runs
|
||||
*inside* the join handshake and therefore knows whether a seed answered. A timer bolted
|
||||
alongside the handshake has only "am I `Up` yet?" to go on, cannot tell *no seed answered*
|
||||
from *a seed answered and the join is in flight*, and so **fires while a seed is
|
||||
answering**. That is the whole defect. OtOpcUa's interim TCP-reachability guard suppressed
|
||||
the observed shape but not the race, which is why the watchdog was deleted rather than
|
||||
patched. Any future plan proposing `Cluster.Join(SelfAddress)` on a timer should be
|
||||
rejected on sight.
|
||||
|
||||
**The residual cost of self-first-on-both — closed by an opt-in bootstrap guard.** Because
|
||||
every node is now its own `seed-nodes[0]`, a *truly simultaneous* cold start (shared power
|
||||
event, hypervisor host reboot) races `FirstSeedNodeProcess` on both nodes → two 1-node
|
||||
clusters that never merge. In-process loopback tests converge and hide it; real parallel VM
|
||||
starts do not — OtOpcUa reproduced it live on its rig. Both repos now ship the same fix as a
|
||||
**dark switch, default OFF** (guard-off behaviour is byte-identical to plain self-first):
|
||||
`ScadaBridge:Cluster:BootstrapGuard:Enabled` (Gitea #33) / OtOpcUa `Cluster:BootstrapGuard:Enabled`
|
||||
(`d1dac87f`, the original). When on, the node emits an **empty** seed list (Akka does not
|
||||
auto-join) and a coordinator issues **one** reachability-gated `JoinSeedNodes`: the
|
||||
lexicographically lower canonical `host:port` is the **founder** (self-first, forms
|
||||
immediately); the higher node TCP-probes the founder for `PartnerProbeSeconds` (25 s) and
|
||||
joins **peer-first** if reachable, **self-first** if not — so cold-start-alone survives for
|
||||
both nodes. Crucially the order is decided **before a single join** and the coordinator
|
||||
**never re-forms mid-handshake** — the exact property the rejected timer lacked. Accepted
|
||||
residual: if the founder dies inside the probe→join window the higher node hangs unjoined,
|
||||
logs a WARN, and a restart recovers it. Enable it on co-located VM pairs sharing a power
|
||||
domain. Reference: ScadaBridge `docs/requirements/Component-ClusterInfrastructure.md` →
|
||||
*Simultaneous cold start — the bootstrap guard*.
|
||||
|
||||
**Ops action (outstanding):** the gitignored `ScadaBridge/deploy/wonder-app-vd03/`
|
||||
overlay must have its `SeedNodes` reordered self-first before its next deploy — the new
|
||||
validator rule is a deliberate hard boot gate.
|
||||
|
||||
### 6.2 Other open items
|
||||
- **OtOpcUa auto-down live gate** still open (needs a real 1-vs-1 rig; deferred to
|
||||
per-cluster mesh Phase 6/7). ScadaBridge's identical mechanism *is* drill-proven.
|
||||
- **OtOpcUa election scope** is cluster-wide until per-cluster mesh Phases 1–7 land
|
||||
(§4.4) — only relevant while multiple pairs share one mesh.
|
||||
- ~~**OtOpcUa auto-down live gate** still open~~ — **CLOSED 2026-07-24** by per-cluster
|
||||
mesh Phase 7 drill D4 (`c50ebcf7`): the 1-vs-1 crash-the-oldest drill ran on the
|
||||
three-mesh rig and every survivor stayed Up without self-downing. ScadaBridge's
|
||||
identical mechanism was already drill-proven.
|
||||
- ~~**OtOpcUa election scope** is cluster-wide until per-cluster mesh Phases 1–7 land~~ —
|
||||
**CLOSED**: the mesh program completed 2026-07-24 (Phase 6 `2839deb5` partitioned the
|
||||
fleet into one 2-node mesh per application `Cluster`; the election is now scoped per
|
||||
`Cluster`, so there is one Primary per `Cluster` by design).
|
||||
- **OtOpcUa's 15 s downing window is a code constant**, not config — worth making
|
||||
tunable alongside any heartbeat-timeout reduction (ScadaBridge's already is:
|
||||
`Cluster:StableAfter`).
|
||||
|
||||
Reference in New Issue
Block a user