docs: correct the self-form watchdog record — rejected, not shipped

The InitJoin self-form watchdog (a timer calling Cluster.Join(SelfAddress) after a
window) was implemented and REJECTED after live testing: it sits outside Akka's join
handshake, so on a routine standby restart — peer alive, join stalled behind removal of
the node's own stale incarnation — it fires mid-join, abandons the in-flight join and
forms a permanent second cluster at the same address (measured: still split after 90s;
OtOpcUa hit it live in its manual-failover drill).

The safety argument 'a booting node only self-forms when no seed answers InitJoin' is
true of Akka's NATIVE first-seed rule (FirstSeedNodeProcess, inside the handshake) and
FALSE of the watchdog, which can fire while a seed is answering. That reasoning error is
now called out explicitly so no future plan re-derives the pattern.

- docs/plans/2026-07-22-initjoin-selfform-fallback.md: dated correction banner (history
  preserved) + the two surviving false claims struck in place (behavior-spec row and the
  Mechanism section's 'races are benign'); points at ScadaBridge
  Component-ClusterInfrastructure.md as the surviving design.
- akka_failover.md 6.1: shipped design stated as self-first seed ordering plus the
  opt-in, default-OFF simultaneous-cold-start bootstrap guard; added a
  never-re-derive-the-watchdog note. 6.2: two items my 6.1 edit contradicted are marked
  closed (OtOpcUa auto-down live gate, mesh election scope — both done 2026-07-24).
- CLAUDE.md: ScadaBridge issue #14 is CLOSED (2026-07-24), not open — v3 dual-namespace
  live gates all passed, ScadaBridge consumes OtOpcUa v3 end-to-end.

akka_failover.md

@@ -449,7 +449,7 @@ is always "restart one side; it rejoins as youngest/standby."
 
 ## 6. Gaps vs the stated HA goal ("either node first, automatic failover")
 
-### 6.1 The seed-node bootstrap constraint — CLOSED 2026-07-22 (differently per repo)
+### 6.1 The seed-node bootstrap constraint — CLOSED 2026-07-22 (self-form watchdog REJECTED; self-first ordering + opt-in bootstrap guard shipped in both repos)
 **Automatic failover: achieved.** Either node crashing is survivable, unattended, in
 ~27–30 s. The remaining gap — a non-first seed cold-starting alone loops on `InitJoin`
 forever (the "registered outage gap") — was **closed in both repos on 2026-07-22**, but
@@ -493,18 +493,52 @@ ordering test failed at 11 s against the watchdog — demonstrating the watchdog
 never usefully fire for the nodes that needed it.
 
 **Result: both repos now share the identical mechanism** — self-first seed ordering,
-validator-enforced, no bootstrap runtime code. The mesh program's Phase 6 convergence
-item is already done; what remains there is only the per-pair seed topology itself.
+validator-enforced, with no self-form timer anywhere. OtOpcUa's mesh program has since
+completed (Phase 7, `c50ebcf7`, 2026-07-24), so the per-pair seed topology is in place too.
+
+**Never re-derive the watchdog from the InitJoin argument (correction, 2026-08-01).** The
+recurring justification — "a booting node only self-forms when no seed answers InitJoin,
+so a self-form timer is safe" — describes **Akka's native first-seed rule**, which runs
+*inside* the join handshake and therefore knows whether a seed answered. A timer bolted
+alongside the handshake has only "am I `Up` yet?" to go on, cannot tell *no seed answered*
+from *a seed answered and the join is in flight*, and so **fires while a seed is
+answering**. That is the whole defect. OtOpcUa's interim TCP-reachability guard suppressed
+the observed shape but not the race, which is why the watchdog was deleted rather than
+patched. Any future plan proposing `Cluster.Join(SelfAddress)` on a timer should be
+rejected on sight.
+
+**The residual cost of self-first-on-both — closed by an opt-in bootstrap guard.** Because
+every node is now its own `seed-nodes[0]`, a *truly simultaneous* cold start (shared power
+event, hypervisor host reboot) races `FirstSeedNodeProcess` on both nodes → two 1-node
+clusters that never merge. In-process loopback tests converge and hide it; real parallel VM
+starts do not — OtOpcUa reproduced it live on its rig. Both repos now ship the same fix as a
+**dark switch, default OFF** (guard-off behaviour is byte-identical to plain self-first):
+`ScadaBridge:Cluster:BootstrapGuard:Enabled` (Gitea #33) / OtOpcUa `Cluster:BootstrapGuard:Enabled`
+(`d1dac87f`, the original). When on, the node emits an **empty** seed list (Akka does not
+auto-join) and a coordinator issues **one** reachability-gated `JoinSeedNodes`: the
+lexicographically lower canonical `host:port` is the **founder** (self-first, forms
+immediately); the higher node TCP-probes the founder for `PartnerProbeSeconds` (25 s) and
+joins **peer-first** if reachable, **self-first** if not — so cold-start-alone survives for
+both nodes. Crucially the order is decided **before a single join** and the coordinator
+**never re-forms mid-handshake** — the exact property the rejected timer lacked. Accepted
+residual: if the founder dies inside the probe→join window the higher node hangs unjoined,
+logs a WARN, and a restart recovers it. Enable it on co-located VM pairs sharing a power
+domain. Reference: ScadaBridge `docs/requirements/Component-ClusterInfrastructure.md` →
+*Simultaneous cold start — the bootstrap guard*.
 
 **Ops action (outstanding):** the gitignored `ScadaBridge/deploy/wonder-app-vd03/`
 overlay must have its `SeedNodes` reordered self-first before its next deploy — the new
 validator rule is a deliberate hard boot gate.
 
 ### 6.2 Other open items
-- **OtOpcUa auto-down live gate** still open (needs a real 1-vs-1 rig; deferred to
-  per-cluster mesh Phase 6/7). ScadaBridge's identical mechanism *is* drill-proven.
-- **OtOpcUa election scope** is cluster-wide until per-cluster mesh Phases 1–7 land
-  (§4.4) — only relevant while multiple pairs share one mesh.
+- ~~**OtOpcUa auto-down live gate** still open~~ — **CLOSED 2026-07-24** by per-cluster
+  mesh Phase 7 drill D4 (`c50ebcf7`): the 1-vs-1 crash-the-oldest drill ran on the
+  three-mesh rig and every survivor stayed Up without self-downing. ScadaBridge's
+  identical mechanism was already drill-proven.
+- ~~**OtOpcUa election scope** is cluster-wide until per-cluster mesh Phases 1–7 land~~ —
+  **CLOSED**: the mesh program completed 2026-07-24 (Phase 6 `2839deb5` partitioned the
+  fleet into one 2-node mesh per application `Cluster`; the election is now scoped per
+  `Cluster`, so there is one Primary per `Cluster` by design).
 - **OtOpcUa's 15 s downing window is a code constant**, not config — worth making
   tunable alongside any heartbeat-timeout reduction (ScadaBridge's already is:
   `Cluster:StableAfter`).

docs/plans/2026-07-22-initjoin-selfform-fallback.md

@@ -19,6 +19,45 @@
 > - **Both repos now share the identical mechanism.** Everything merged + pushed 2026-07-22, including ScadaBridge Task 10 (site-pair failover relayed from the central UI).
 > - The design sections below are retained as the decision record — read them through this banner. `akka_failover.md` §6.1 carries the corrected family-level account.
 
+> ## 🛑 CORRECTION (2026-08-01) — the self-form watchdog is REJECTED, not shipped
+>
+> Re-read of this doc found the false safety claim still sitting **unmarked inside the design body**
+> (behavior-spec row "Window expires while a join is mid-handshake" and the Mechanism section's
+> "Races are benign") even though the 2026-07-22 banner above overturns it. Both are now struck in
+> place. Stating it plainly, once, so no future reader reconstructs the pattern from the surviving prose:
+>
+> 1. **The watchdog pattern was implemented and REJECTED after live testing.** A timer that waits a
+>    window for membership and then calls `Cluster.Join(SelfAddress)` is **defective**. On a *routine
+>    standby restart* — peer alive and answering, join stalled behind removal of the restarting node's
+>    own stale incarnation (the peer must down it, wait out the failure detector, then take a leader
+>    action) — the timer fires **mid-join**, abandons the in-flight join, and forms a **second cluster
+>    at the same address**. Measured on ScadaBridge: **still split after 90 s** — a routine restart
+>    converted into an outage of the previously-healthy node. OtOpcUa hit the same race live during
+>    its manual-failover drill.
+> 2. **The "only self-forms when no seed answers InitJoin" reasoning does NOT protect the watchdog.**
+>    That property is true of **Akka's native first-seed rule** (`FirstSeedNodeProcess`), which lives
+>    *inside* the join handshake and therefore knows whether a seed answered. The watchdog lives
+>    *outside* the handshake: its only signal is "am I `Up` yet?", which cannot distinguish *no seed
+>    answered* from *a seed answered and the join is in flight*. So the watchdog **can and does fire
+>    while a seed is answering**. Any doc, comment or plan that justifies a self-form timer with the
+>    InitJoin-answer argument is quoting the native rule's guarantee to excuse a mechanism that does
+>    not have it.
+> 3. **The surviving design is documented in ScadaBridge, not here.** Authoritative source:
+>    `~/Desktop/ScadaBridge/docs/requirements/Component-ClusterInfrastructure.md` →
+>    **Seed Node Ordering** (self-first: every node lists ITSELF as `seed-nodes[0]`; only
+>    `seed-nodes[0]` may self-form, via Akka's own `FirstSeedNodeProcess`; `StartupValidator`
+>    hard-gates the ordering; `SelfFirstSeedBootstrapTests`) and **Simultaneous cold start — the
+>    bootstrap guard** (the separate, **opt-in, default-OFF** `ScadaBridge:Cluster:BootstrapGuard`,
+>    Gitea #33, which decides the join order from a TCP reachability probe **before** a single
+>    `JoinSeedNodes` and **never re-forms mid-handshake** — that last property is precisely what the
+>    watchdog lacked). Companion decision record:
+>    `~/Desktop/ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md`.
+> 4. **Neither repo ships the watchdog.** ScadaBridge never merged it. OtOpcUa deleted
+>    `ClusterBootstrapFallback` + `SelfFormAfter` the same day; source re-verified 2026-08-01 —
+>    see the correction section in that repo's plan.
+>
+> Nothing below this banner is safe to execute as written. The Design sections are history.
+
 **Decision provenance:** `scadaproj/akka_failover.md` §6.1 (decided 2026-07-22); design priority = availability over partition safety (memory `ha-availability-over-partition-safety`). Nodes share a datacenter/LAN → a live peer answers InitJoin in milliseconds, so 10 s is generous; operators tune per site.
 
 ---
@@ -39,7 +78,7 @@ Akka.NET only allows the **first-listed seed** to self-join and form a *new* clu
 | Peer boots after survivor self-formed | Peer's `InitJoin` reaches the self-formed node (it's in the peer's seed list) → joins as youngest/standby. No island. |
 | Both cold-start simultaneously, mutually unreachable (boot-time partition) | Both self-form → two islands (same dual-active class auto-down already accepts; same recovery: restart one side). Mitigable with a staggered service start. |
 | `SelfFormAfter` = `null` or `<= 0` | Disabled — today's wait-forever behavior |
-| Window expires while a join is mid-handshake | Benign race: Akka ignores `Join` once the node has already joined a cluster |
+| ~~Window expires while a join is mid-handshake~~ | ~~Benign race: Akka ignores `Join` once the node has already joined a cluster~~ **FALSE — see the 2026-08-01 correction banner. `Cluster.Join(SelfAddress)` is NOT ignored mid-handshake: it abandons the in-flight join and forms a permanent second cluster at the same address (ScadaBridge: still split after 90 s; OtOpcUa: live island during the manual-failover drill). This row is the whole reason the watchdog was rejected.** |
 
 ### Config surface (same key both apps — family convention, like `SplitBrainResolverStrategy`)
 
@@ -56,7 +95,13 @@ Akka.NET only allows the **first-listed seed** to self-join and form a *new* clu
 3. `RegisterOnMemberUp` completes a TCS; race it against `Task.Delay(window)`.
 4. On window expiry (and system not terminated): prominent warn log + `Cluster.Join(cluster.SelfAddress)`.
 
-Races are benign — Akka ignores `Join` once joined this incarnation.
+~~Races are benign — Akka ignores `Join` once joined this incarnation.~~ **FALSE (correction
+2026-08-01).** Step 3's success signal (`RegisterOnMemberUp`) is outside Akka's join handshake and
+cannot tell *no seed answered* from *a seed answered and the join is in flight*; step 4's
+`Cluster.Join(self)` fired during `TryingToJoin` wins and islands the node. **Do not implement this
+mechanism.** The InitJoin-answer guarantee belongs to Akka's `FirstSeedNodeProcess`, not to a timer
+bolted alongside it — use self-first seed ordering (ScadaBridge
+`Component-ClusterInfrastructure.md` → Seed Node Ordering).
 
 ### Manual failover control (Part D design)