From 9303289d05c35067e39569f39ba2d10c89619940 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sat, 1 Aug 2026 11:09:57 -0400 Subject: [PATCH] =?UTF-8?q?docs(plan):=20correct=20the=20self-form=20watch?= =?UTF-8?q?dog=20record=20=E2=80=94=20rejected,=20TCP=20guard=20was=20inte?= =?UTF-8?q?rim?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Family-wide docs correction (paired with scadaproj 6d222ea4). This plan's superseded banner was right but its task body still read as if the watchdog were merely patched. - Dated 2026-08-01 correction section (history preserved): the timer calling Cluster.Join(SelfAddress) was implemented and REJECTED in both repos after live testing — outside Akka's join handshake, it fires mid-join on a routine standby restart (peer alive, join stalled behind stale-incarnation removal) and forms a permanent second cluster at the same address. ScadaBridge measured a split unhealed after 90s; Task 8 hit the same shape live here. - The 'a booting node only self-forms when NO seed answers InitJoin' claim in the Task 2 doc comment is marked FALSE for the watchdog: that guarantee belongs to Akka's native FirstSeedNodeProcess, which runs inside the handshake; the timer does not, and fires while a seed is answering. - Task 8's TCP reachability guard is recorded as an INTERIM patch, not the fix — the race survived it, which is why the watchdog was deleted rather than patched. - Shipped design restated: self-first seed ordering (AkkaClusterOptionsValidator + SelfFirstSeedBootstrapTests) plus the opt-in, default-OFF bootstrap guard (d1dac87f). Source re-verified 2026-08-01: the defective watchdog is NOT present in this repo — no ClusterBootstrapFallback.cs, no SelfFormAfter option, no Cluster.Join(SelfAddress) in src/. Documentation-only change; no code touched. --- ...2-selfform-fallback-and-manual-failover.md | 57 ++++++++++++++++++- 1 file changed, 56 insertions(+), 1 deletion(-) diff --git a/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md b/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md index 36cd7170..04cc4473 100644 --- a/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md +++ b/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md @@ -17,6 +17,55 @@ > the execution record, including the Task 8 live-gate evidence, which is the whole reason the > watchdog was rejected. Do not execute the Task 1/2/3 code as written. +> ## 🛑 CORRECTION (2026-08-01) — watchdog REJECTED outright; the TCP guard was interim, not the fix +> +> Family-wide docs correction, applied to this plan and to its scadaproj half +> (`~/Desktop/scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md`). Two claims still +> readable in the task text below are wrong and are struck in place: +> +> 1. **The watchdog pattern was implemented and REJECTED after live testing — in both repos.** A +> timer that waits a window for membership and then calls `Cluster.Join(SelfAddress)` is +> **defective by construction**, not merely mis-tuned. On a *routine standby restart* — the peer +> is alive and answering, but the join stalls behind removal of the restarting node's own stale +> incarnation — the timer fires **mid-join**, abandons the in-flight join and forms a **permanent +> second cluster at the same address**. ScadaBridge measured a split that had **not healed after +> 90 s**; OtOpcUa hit the same shape live in the Task 8 manual-failover drill. +> 2. **"A booting node only self-forms when NO seed answers InitJoin" does NOT protect the +> watchdog** — that claim appears verbatim in the Task 2 doc comment below and is the reasoning +> error at the root of this design. The property is real, but it belongs to **Akka's native +> first-seed rule** (`FirstSeedNodeProcess`), which runs *inside* the join handshake. The watchdog +> runs *outside* it, seeing only "am I `Up` yet?", which cannot distinguish *no seed answered* +> from *a seed answered and the join is in flight*. The watchdog therefore **can fire while a seed +> is answering** — exactly what was observed. +> 3. **The Task 8 TCP reachability guard was an interim patch, not the resolution.** It suppressed +> the one observed shape; the race remained (a peer can answer between probe and join). The +> resolution was **deleting the watchdog**. +> +> **Shipped design (both repos, identical mechanism):** **self-first seed ordering** — every seed +> node lists ITSELF as `seed-nodes[0]`, so only Akka's own `FirstSeedNodeProcess` may form a new +> cluster, race-free inside the handshake. Here: `AkkaClusterOptionsValidator` (conditional — self +> must be `seed[0]` only IF self is a seed; identity on `PublicHostname`+port) and +> `SelfFirstSeedBootstrapTests`. Plus the separate **opt-in, default-OFF** simultaneous-cold-start +> **bootstrap guard** (`Cluster:BootstrapGuard:Enabled`, `ClusterBootstrapGuard` + +> `ClusterBootstrapCoordinator`, shipped here at `d1dac87f` and ported to ScadaBridge as Gitea #33), +> which decides the join order from a TCP reachability probe **before** a single `JoinSeedNodes` and +> **never re-forms mid-handshake** — the property the watchdog lacked. Authoritative family write-up: +> ScadaBridge `docs/requirements/Component-ClusterInfrastructure.md` → *Seed Node Ordering* and +> *Simultaneous cold start — the bootstrap guard*; local write-up `docs/Redundancy.md` → +> *Bootstrap: self-first seed ordering*. +> +> **Source re-verified 2026-08-01 — the defective watchdog is NOT present in this repo.** +> `ClusterBootstrapFallback.cs` does not exist under `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`; no +> `SelfFormAfter` option exists on `AkkaClusterOptions`; no `Cluster.Join(SelfAddress)` call exists +> anywhere in `src/` (the only `SelfAddress` uses are `AdminOperationsActor.cs:344/440/470`, reading +> `.Host` for an audit `SourceNode` stamp). `ServiceCollectionExtensions.cs:148-155` carries an +> explicit tombstone comment at the former arming site. What remains under +> `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/` is the **bootstrap guard** — `ClusterBootstrapGuard.cs` / +> `ClusterBootstrapCoordinator.cs`, which issue exactly one `JoinSeedNodes` +> (`ClusterBootstrapCoordinator.cs:116`) after a pre-join reachability decision and never re-join +> (`:139` — the watchdog loop there is read-only, it only logs an operator-visible WARN). **No code +> change is needed in this repo; this correction is documentation only.** + > **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. > > Shared cross-repo design: `~/Desktop/scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md` (design rationale, MNTR assessment, behavior spec). The ScadaBridge half lives in `~/Desktop/ScadaBridge/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`. This plan is self-contained for execution. @@ -47,7 +96,7 @@ | Peer boots after survivor self-formed | Its InitJoin is answered → joins as youngest. No island. | | Both pair nodes cold-start simultaneously, mutually unreachable | Both self-form → dual-active (same partition class auto-down accepts; restart one side) | | `SelfFormAfter` null/`≤0` | Disabled — today's wait-forever behavior | -| ~~Window expires mid-join-handshake~~ | ~~Benign: Akka ignores `Join` once joined~~ **FALSE — proven live (Task 8): `Join(self)` during an in-flight join abandons it and islands the node. Fixed by the TCP reachability guard: never self-form while any seed peer accepts a connection; wait a window and re-arm. ScadaBridge rejected the watchdog entirely for self-first seed ordering; OtOpcUa converges at mesh-program Phase 6.** | +| ~~Window expires mid-join-handshake~~ | ~~Benign: Akka ignores `Join` once joined~~ **FALSE — proven live (Task 8): `Join(self)` during an in-flight join abandons it and islands the node.** ~~Fixed by the TCP reachability guard: never self-form while any seed peer accepts a connection; wait a window and re-arm.~~ **Correction 2026-08-01: the TCP guard was an INTERIM patch that closed the one observed shape, not the fix — the race survived it. The watchdog was DELETED in both repos; the shipped mechanism is self-first seed ordering (+ the opt-in bootstrap guard for simultaneous cold start). The convergence did not wait for mesh-program Phase 6 — it landed the same day.** | **Manual failover rules:** graceful `Leave` of the **oldest Up `driver` member** (identical query to `RedundancyStateActor.SelectDriverPrimary` — `Member.AgeOrdering`, `Leaving` excluded via `Status == Up`), never `Down`. The leaving node hands over via the cluster-leave phases, `CoordinatedShutdown` runs, `ActorSystemTerminationWatchdog` exits the process, the supervisor restarts it, and it rejoins as youngest → the survivor is Primary (ServiceLevel 250 moves; OPC UA clients re-select). Admin-only; peer guard (<2 Up driver members → disabled); confirmation dialog; audited via the shared `ZB.MOM.WW.Audit.IAuditWriter` seam before the Leave. **Mesh-scope caveat:** until the per-cluster mesh lands, the election is mesh-wide — the button acts on THE Primary of the whole Akka cluster, not per-`ServerCluster` row; the UI must say so (mirrors the KNOWN LIMITATION in `docs/Redundancy.md:84-99`). @@ -126,9 +175,15 @@ namespace ZB.MOM.WW.OtOpcUa.Cluster; /// — a site node seeded only by central-1 must wait, /// not island. Sequential recovery is island-free: Akka's join protocol prefers an existing /// cluster (a booting node only self-forms when NO seed answers InitJoin). +/// <<< FALSE (correction 2026-08-01): that guarantee is Akka's FirstSeedNodeProcess's, +/// which lives INSIDE the join handshake. This timer lives outside it and fires while a seed IS +/// answering. >>> /// Benign race: Akka ignores Join once the node has joined a cluster this /// incarnation. Residual risk: both pair nodes cold-starting inside the window while mutually /// unreachable self-form separately — the same dual-active class auto-down accepts. +/// <<< FALSE (correction 2026-08-01): the race is NOT benign. Join(self) mid-handshake +/// abandons the in-flight join and forms a permanent second cluster at the same address. This +/// class was DELETED; do not re-create it. >>> /// public static class ClusterBootstrapFallback {