docs: correct the self-form watchdog record — rejected, not shipped

The InitJoin self-form watchdog (a timer calling Cluster.Join(SelfAddress) after a
window) was implemented and REJECTED after live testing: it sits outside Akka's join
handshake, so on a routine standby restart — peer alive, join stalled behind removal of
the node's own stale incarnation — it fires mid-join, abandons the in-flight join and
forms a permanent second cluster at the same address (measured: still split after 90s;
OtOpcUa hit it live in its manual-failover drill).

The safety argument 'a booting node only self-forms when no seed answers InitJoin' is
true of Akka's NATIVE first-seed rule (FirstSeedNodeProcess, inside the handshake) and
FALSE of the watchdog, which can fire while a seed is answering. That reasoning error is
now called out explicitly so no future plan re-derives the pattern.

- docs/plans/2026-07-22-initjoin-selfform-fallback.md: dated correction banner (history
  preserved) + the two surviving false claims struck in place (behavior-spec row and the
  Mechanism section's 'races are benign'); points at ScadaBridge
  Component-ClusterInfrastructure.md as the surviving design.
- akka_failover.md 6.1: shipped design stated as self-first seed ordering plus the
  opt-in, default-OFF simultaneous-cold-start bootstrap guard; added a
  never-re-derive-the-watchdog note. 6.2: two items my 6.1 edit contradicted are marked
  closed (OtOpcUa auto-down live gate, mesh election scope — both done 2026-07-24).
- CLAUDE.md: ScadaBridge issue #14 is CLOSED (2026-07-24), not open — v3 dual-namespace
  live gates all passed, ScadaBridge consumes OtOpcUa v3 end-to-end.
This commit is contained in:
Joseph Doherty
2026-08-01 11:09:45 -04:00
parent 7a6f997180
commit 6d222ea491
3 changed files with 89 additions and 10 deletions
+41 -7
View File
@@ -449,7 +449,7 @@ is always "restart one side; it rejoins as youngest/standby."
## 6. Gaps vs the stated HA goal ("either node first, automatic failover")
### 6.1 The seed-node bootstrap constraint — CLOSED 2026-07-22 (differently per repo)
### 6.1 The seed-node bootstrap constraint — CLOSED 2026-07-22 (self-form watchdog REJECTED; self-first ordering + opt-in bootstrap guard shipped in both repos)
**Automatic failover: achieved.** Either node crashing is survivable, unattended, in
~27–30 s. The remaining gap — a non-first seed cold-starting alone loops on `InitJoin`
forever (the "registered outage gap") — was **closed in both repos on 2026-07-22**, but
@@ -493,18 +493,52 @@ ordering test failed at 11 s against the watchdog — demonstrating the watchdog
never usefully fire for the nodes that needed it.
**Result: both repos now share the identical mechanism** — self-first seed ordering,
validator-enforced, no bootstrap runtime code. The mesh program's Phase 6 convergence
item is already done; what remains there is only the per-pair seed topology itself.
validator-enforced, with no self-form timer anywhere. OtOpcUa's mesh program has since
completed (Phase 7, `c50ebcf7`, 2026-07-24), so the per-pair seed topology is in place too.
**Never re-derive the watchdog from the InitJoin argument (correction, 2026-08-01).** The
recurring justification — "a booting node only self-forms when no seed answers InitJoin,
so a self-form timer is safe" — describes **Akka's native first-seed rule**, which runs
*inside* the join handshake and therefore knows whether a seed answered. A timer bolted
alongside the handshake has only "am I `Up` yet?" to go on, cannot tell *no seed answered*
from *a seed answered and the join is in flight*, and so **fires while a seed is
answering**. That is the whole defect. OtOpcUa's interim TCP-reachability guard suppressed
the observed shape but not the race, which is why the watchdog was deleted rather than
patched. Any future plan proposing `Cluster.Join(SelfAddress)` on a timer should be
rejected on sight.
**The residual cost of self-first-on-both — closed by an opt-in bootstrap guard.** Because
every node is now its own `seed-nodes[0]`, a *truly simultaneous* cold start (shared power
event, hypervisor host reboot) races `FirstSeedNodeProcess` on both nodes → two 1-node
clusters that never merge. In-process loopback tests converge and hide it; real parallel VM
starts do not — OtOpcUa reproduced it live on its rig. Both repos now ship the same fix as a
**dark switch, default OFF** (guard-off behaviour is byte-identical to plain self-first):
`ScadaBridge:Cluster:BootstrapGuard:Enabled` (Gitea #33) / OtOpcUa `Cluster:BootstrapGuard:Enabled`
(`d1dac87f`, the original). When on, the node emits an **empty** seed list (Akka does not
auto-join) and a coordinator issues **one** reachability-gated `JoinSeedNodes`: the
lexicographically lower canonical `host:port` is the **founder** (self-first, forms
immediately); the higher node TCP-probes the founder for `PartnerProbeSeconds` (25 s) and
joins **peer-first** if reachable, **self-first** if not — so cold-start-alone survives for
both nodes. Crucially the order is decided **before a single join** and the coordinator
**never re-forms mid-handshake** — the exact property the rejected timer lacked. Accepted
residual: if the founder dies inside the probe→join window the higher node hangs unjoined,
logs a WARN, and a restart recovers it. Enable it on co-located VM pairs sharing a power
domain. Reference: ScadaBridge `docs/requirements/Component-ClusterInfrastructure.md` →
*Simultaneous cold start — the bootstrap guard*.
**Ops action (outstanding):** the gitignored `ScadaBridge/deploy/wonder-app-vd03/`
overlay must have its `SeedNodes` reordered self-first before its next deploy — the new
validator rule is a deliberate hard boot gate.
### 6.2 Other open items
- **OtOpcUa auto-down live gate** still open (needs a real 1-vs-1 rig; deferred to
per-cluster mesh Phase 6/7). ScadaBridge's identical mechanism *is* drill-proven.
- **OtOpcUa election scope** is cluster-wide until per-cluster mesh Phases 1–7 land
(§4.4) — only relevant while multiple pairs share one mesh.
- ~~**OtOpcUa auto-down live gate** still open~~ — **CLOSED 2026-07-24** by per-cluster
mesh Phase 7 drill D4 (`c50ebcf7`): the 1-vs-1 crash-the-oldest drill ran on the
three-mesh rig and every survivor stayed Up without self-downing. ScadaBridge's
identical mechanism was already drill-proven.
- ~~**OtOpcUa election scope** is cluster-wide until per-cluster mesh Phases 1–7 land~~ —
**CLOSED**: the mesh program completed 2026-07-24 (Phase 6 `2839deb5` partitioned the
fleet into one 2-node mesh per application `Cluster`; the election is now scoped per
`Cluster`, so there is one Primary per `Cluster` by design).
- **OtOpcUa's 15 s downing window is a code constant**, not config — worth making
tunable alongside any heartbeat-timeout reduction (ScadaBridge's already is:
`Cluster:StableAfter`).