dohertj2/scadaproj
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: correct the self-form watchdog record — rejected, not shipped

Browse Source 
The InitJoin self-form watchdog (a timer calling Cluster.Join(SelfAddress) after a
window) was implemented and REJECTED after live testing: it sits outside Akka's join
handshake, so on a routine standby restart — peer alive, join stalled behind removal of
the node's own stale incarnation — it fires mid-join, abandons the in-flight join and
forms a permanent second cluster at the same address (measured: still split after 90s;
OtOpcUa hit it live in its manual-failover drill).

The safety argument 'a booting node only self-forms when no seed answers InitJoin' is
true of Akka's NATIVE first-seed rule (FirstSeedNodeProcess, inside the handshake) and
FALSE of the watchdog, which can fire while a seed is answering. That reasoning error is
now called out explicitly so no future plan re-derives the pattern.

- docs/plans/2026-07-22-initjoin-selfform-fallback.md: dated correction banner (history
  preserved) + the two surviving false claims struck in place (behavior-spec row and the
  Mechanism section's 'races are benign'); points at ScadaBridge
  Component-ClusterInfrastructure.md as the surviving design.
- akka_failover.md 6.1: shipped design stated as self-first seed ordering plus the
  opt-in, default-OFF simultaneous-cold-start bootstrap guard; added a
  never-re-derive-the-watchdog note. 6.2: two items my 6.1 edit contradicted are marked
  closed (OtOpcUa auto-down live gate, mesh election scope — both done 2026-07-24).
- CLAUDE.md: ScadaBridge issue #14 is CLOSED (2026-07-24), not open — v3 dual-namespace
  live gates all passed, ScadaBridge consumes OtOpcUa v3 end-to-end.
This commit is contained in:
Joseph Doherty
2026-08-01 11:09:45 -04:00
parent 7a6f997180
commit 6d222ea491
3 changed files with 89 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plans/2026-07-22-initjoin-selfform-fallback.md
+47 -2
View File
@@ -19,6 +19,45 @@
> - **Both repos now share the identical mechanism.** Everything merged + pushed 2026-07-22, including ScadaBridge Task 10 (site-pair failover relayed from the central UI).
> - The design sections below are retained as the decision record — read them through this banner. `akka_failover.md` §6.1 carries the corrected family-level account.
> ## 🛑 CORRECTION (2026-08-01) — the self-form watchdog is REJECTED, not shipped
>
> Re-read of this doc found the false safety claim still sitting **unmarked inside the design body**
> (behavior-spec row "Window expires while a join is mid-handshake" and the Mechanism section's
> "Races are benign") even though the 2026-07-22 banner above overturns it. Both are now struck in
> place. Stating it plainly, once, so no future reader reconstructs the pattern from the surviving prose:
>
> 1. **The watchdog pattern was implemented and REJECTED after live testing.** A timer that waits a
> window for membership and then calls `Cluster.Join(SelfAddress)` is **defective**. On a *routine
> standby restart* — peer alive and answering, join stalled behind removal of the restarting node's
> own stale incarnation (the peer must down it, wait out the failure detector, then take a leader
> action) — the timer fires **mid-join**, abandons the in-flight join, and forms a **second cluster
> at the same address**. Measured on ScadaBridge: **still split after 90 s** — a routine restart
> converted into an outage of the previously-healthy node. OtOpcUa hit the same race live during
> its manual-failover drill.
> 2. **The "only self-forms when no seed answers InitJoin" reasoning does NOT protect the watchdog.**
> That property is true of **Akka's native first-seed rule** (`FirstSeedNodeProcess`), which lives
> *inside* the join handshake and therefore knows whether a seed answered. The watchdog lives
> *outside* the handshake: its only signal is "am I `Up` yet?", which cannot distinguish *no seed
> answered* from *a seed answered and the join is in flight*. So the watchdog **can and does fire
> while a seed is answering**. Any doc, comment or plan that justifies a self-form timer with the
> InitJoin-answer argument is quoting the native rule's guarantee to excuse a mechanism that does
> not have it.
> 3. **The surviving design is documented in ScadaBridge, not here.** Authoritative source:
> `~/Desktop/ScadaBridge/docs/requirements/Component-ClusterInfrastructure.md` →
> **Seed Node Ordering** (self-first: every node lists ITSELF as `seed-nodes[0]`; only
> `seed-nodes[0]` may self-form, via Akka's own `FirstSeedNodeProcess`; `StartupValidator`
> hard-gates the ordering; `SelfFirstSeedBootstrapTests`) and **Simultaneous cold start — the
> bootstrap guard** (the separate, **opt-in, default-OFF** `ScadaBridge:Cluster:BootstrapGuard`,
> Gitea #33, which decides the join order from a TCP reachability probe **before** a single
> `JoinSeedNodes` and **never re-forms mid-handshake** — that last property is precisely what the
> watchdog lacked). Companion decision record:
> `~/Desktop/ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md`.
> 4. **Neither repo ships the watchdog.** ScadaBridge never merged it. OtOpcUa deleted
> `ClusterBootstrapFallback` + `SelfFormAfter` the same day; source re-verified 2026-08-01 —
> see the correction section in that repo's plan.
>
> Nothing below this banner is safe to execute as written. The Design sections are history.
**Decision provenance:** `scadaproj/akka_failover.md` §6.1 (decided 2026-07-22); design priority = availability over partition safety (memory `ha-availability-over-partition-safety`). Nodes share a datacenter/LAN → a live peer answers InitJoin in milliseconds, so 10 s is generous; operators tune per site.
---
@@ -39,7 +78,7 @@ Akka.NET only allows the **first-listed seed** to self-join and form a *new* clu
| Peer boots after survivor self-formed | Peer's `InitJoin` reaches the self-formed node (it's in the peer's seed list) → joins as youngest/standby. No island. |
| Both cold-start simultaneously, mutually unreachable (boot-time partition) | Both self-form → two islands (same dual-active class auto-down already accepts; same recovery: restart one side). Mitigable with a staggered service start. |
| `SelfFormAfter` = `null` or `<= 0` | Disabled — today's wait-forever behavior |
| Window expires while a join is mid-handshake | Benign race: Akka ignores `Join` once the node has already joined a cluster |
| ~~Window expires while a join is mid-handshake~~ | ~~Benign race: Akka ignores `Join` once the node has already joined a cluster~~ **FALSE — see the 2026-08-01 correction banner. `Cluster.Join(SelfAddress)` is NOT ignored mid-handshake: it abandons the in-flight join and forms a permanent second cluster at the same address (ScadaBridge: still split after 90 s; OtOpcUa: live island during the manual-failover drill). This row is the whole reason the watchdog was rejected.** |
### Config surface (same key both apps — family convention, like `SplitBrainResolverStrategy`)
@@ -56,7 +95,13 @@ Akka.NET only allows the **first-listed seed** to self-join and form a *new* clu
3. `RegisterOnMemberUp` completes a TCS; race it against `Task.Delay(window)`.
4. On window expiry (and system not terminated): prominent warn log + `Cluster.Join(cluster.SelfAddress)`.
Races are benign — Akka ignores `Join` once joined this incarnation.
~~Races are benign — Akka ignores `Join` once joined this incarnation.~~ **FALSE (correction
2026-08-01).** Step 3's success signal (`RegisterOnMemberUp`) is outside Akka's join handshake and
cannot tell *no seed answered* from *a seed answered and the join is in flight*; step 4's
`Cluster.Join(self)` fired during `TryingToJoin` wins and islands the node. **Do not implement this
mechanism.** The InitJoin-answer guarantee belongs to Akka's `FirstSeedNodeProcess`, not to a timer
bolted alongside it — use self-first seed ordering (ScadaBridge
`Component-ClusterInfrastructure.md` → Seed Node Ordering).
### Manual failover control (Part D design)