diff --git a/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md b/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md index 36cd7170..04cc4473 100644 --- a/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md +++ b/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md @@ -17,6 +17,55 @@ > the execution record, including the Task 8 live-gate evidence, which is the whole reason the > watchdog was rejected. Do not execute the Task 1/2/3 code as written. +> ## 🛑 CORRECTION (2026-08-01) — watchdog REJECTED outright; the TCP guard was interim, not the fix +> +> Family-wide docs correction, applied to this plan and to its scadaproj half +> (`~/Desktop/scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md`). Two claims still +> readable in the task text below are wrong and are struck in place: +> +> 1. **The watchdog pattern was implemented and REJECTED after live testing — in both repos.** A +> timer that waits a window for membership and then calls `Cluster.Join(SelfAddress)` is +> **defective by construction**, not merely mis-tuned. On a *routine standby restart* — the peer +> is alive and answering, but the join stalls behind removal of the restarting node's own stale +> incarnation — the timer fires **mid-join**, abandons the in-flight join and forms a **permanent +> second cluster at the same address**. ScadaBridge measured a split that had **not healed after +> 90 s**; OtOpcUa hit the same shape live in the Task 8 manual-failover drill. +> 2. **"A booting node only self-forms when NO seed answers InitJoin" does NOT protect the +> watchdog** — that claim appears verbatim in the Task 2 doc comment below and is the reasoning +> error at the root of this design. The property is real, but it belongs to **Akka's native +> first-seed rule** (`FirstSeedNodeProcess`), which runs *inside* the join handshake. The watchdog +> runs *outside* it, seeing only "am I `Up` yet?", which cannot distinguish *no seed answered* +> from *a seed answered and the join is in flight*. The watchdog therefore **can fire while a seed +> is answering** — exactly what was observed. +> 3. **The Task 8 TCP reachability guard was an interim patch, not the resolution.** It suppressed +> the one observed shape; the race remained (a peer can answer between probe and join). The +> resolution was **deleting the watchdog**. +> +> **Shipped design (both repos, identical mechanism):** **self-first seed ordering** — every seed +> node lists ITSELF as `seed-nodes[0]`, so only Akka's own `FirstSeedNodeProcess` may form a new +> cluster, race-free inside the handshake. Here: `AkkaClusterOptionsValidator` (conditional — self +> must be `seed[0]` only IF self is a seed; identity on `PublicHostname`+port) and +> `SelfFirstSeedBootstrapTests`. Plus the separate **opt-in, default-OFF** simultaneous-cold-start +> **bootstrap guard** (`Cluster:BootstrapGuard:Enabled`, `ClusterBootstrapGuard` + +> `ClusterBootstrapCoordinator`, shipped here at `d1dac87f` and ported to ScadaBridge as Gitea #33), +> which decides the join order from a TCP reachability probe **before** a single `JoinSeedNodes` and +> **never re-forms mid-handshake** — the property the watchdog lacked. Authoritative family write-up: +> ScadaBridge `docs/requirements/Component-ClusterInfrastructure.md` → *Seed Node Ordering* and +> *Simultaneous cold start — the bootstrap guard*; local write-up `docs/Redundancy.md` → +> *Bootstrap: self-first seed ordering*. +> +> **Source re-verified 2026-08-01 — the defective watchdog is NOT present in this repo.** +> `ClusterBootstrapFallback.cs` does not exist under `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`; no +> `SelfFormAfter` option exists on `AkkaClusterOptions`; no `Cluster.Join(SelfAddress)` call exists +> anywhere in `src/` (the only `SelfAddress` uses are `AdminOperationsActor.cs:344/440/470`, reading +> `.Host` for an audit `SourceNode` stamp). `ServiceCollectionExtensions.cs:148-155` carries an +> explicit tombstone comment at the former arming site. What remains under +> `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/` is the **bootstrap guard** — `ClusterBootstrapGuard.cs` / +> `ClusterBootstrapCoordinator.cs`, which issue exactly one `JoinSeedNodes` +> (`ClusterBootstrapCoordinator.cs:116`) after a pre-join reachability decision and never re-join +> (`:139` — the watchdog loop there is read-only, it only logs an operator-visible WARN). **No code +> change is needed in this repo; this correction is documentation only.** + > **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. > > Shared cross-repo design: `~/Desktop/scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md` (design rationale, MNTR assessment, behavior spec). The ScadaBridge half lives in `~/Desktop/ScadaBridge/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`. This plan is self-contained for execution. @@ -47,7 +96,7 @@ | Peer boots after survivor self-formed | Its InitJoin is answered → joins as youngest. No island. | | Both pair nodes cold-start simultaneously, mutually unreachable | Both self-form → dual-active (same partition class auto-down accepts; restart one side) | | `SelfFormAfter` null/`≤0` | Disabled — today's wait-forever behavior | -| ~~Window expires mid-join-handshake~~ | ~~Benign: Akka ignores `Join` once joined~~ **FALSE — proven live (Task 8): `Join(self)` during an in-flight join abandons it and islands the node. Fixed by the TCP reachability guard: never self-form while any seed peer accepts a connection; wait a window and re-arm. ScadaBridge rejected the watchdog entirely for self-first seed ordering; OtOpcUa converges at mesh-program Phase 6.** | +| ~~Window expires mid-join-handshake~~ | ~~Benign: Akka ignores `Join` once joined~~ **FALSE — proven live (Task 8): `Join(self)` during an in-flight join abandons it and islands the node.** ~~Fixed by the TCP reachability guard: never self-form while any seed peer accepts a connection; wait a window and re-arm.~~ **Correction 2026-08-01: the TCP guard was an INTERIM patch that closed the one observed shape, not the fix — the race survived it. The watchdog was DELETED in both repos; the shipped mechanism is self-first seed ordering (+ the opt-in bootstrap guard for simultaneous cold start). The convergence did not wait for mesh-program Phase 6 — it landed the same day.** | **Manual failover rules:** graceful `Leave` of the **oldest Up `driver` member** (identical query to `RedundancyStateActor.SelectDriverPrimary` — `Member.AgeOrdering`, `Leaving` excluded via `Status == Up`), never `Down`. The leaving node hands over via the cluster-leave phases, `CoordinatedShutdown` runs, `ActorSystemTerminationWatchdog` exits the process, the supervisor restarts it, and it rejoins as youngest → the survivor is Primary (ServiceLevel 250 moves; OPC UA clients re-select). Admin-only; peer guard (<2 Up driver members → disabled); confirmation dialog; audited via the shared `ZB.MOM.WW.Audit.IAuditWriter` seam before the Leave. **Mesh-scope caveat:** until the per-cluster mesh lands, the election is mesh-wide — the button acts on THE Primary of the whole Akka cluster, not per-`ServerCluster` row; the UI must say so (mirrors the KNOWN LIMITATION in `docs/Redundancy.md:84-99`). @@ -126,9 +175,15 @@ namespace ZB.MOM.WW.OtOpcUa.Cluster; /// — a site node seeded only by central-1 must wait, /// not island. Sequential recovery is island-free: Akka's join protocol prefers an existing /// cluster (a booting node only self-forms when NO seed answers InitJoin). +/// <<< FALSE (correction 2026-08-01): that guarantee is Akka's FirstSeedNodeProcess's, +/// which lives INSIDE the join handshake. This timer lives outside it and fires while a seed IS +/// answering. >>> /// Benign race: Akka ignores Join once the node has joined a cluster this /// incarnation. Residual risk: both pair nodes cold-starting inside the window while mutually /// unreachable self-form separately — the same dual-active class auto-down accepts. +/// <<< FALSE (correction 2026-08-01): the race is NOT benign. Join(self) mid-handshake +/// abandons the in-flight join and forms a permanent second cluster at the same address. This +/// class was DELETED; do not re-create it. >>> /// public static class ClusterBootstrapFallback {