dohertj2/lmxopcua
1
0
Fork 0
Code Issues 32 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(plan): correct the self-form watchdog record — rejected, TCP guard was interim

Browse Source 
Family-wide docs correction (paired with scadaproj 6d222ea4). This plan's superseded
banner was right but its task body still read as if the watchdog were merely patched.

- Dated 2026-08-01 correction section (history preserved): the timer calling
  Cluster.Join(SelfAddress) was implemented and REJECTED in both repos after live
  testing — outside Akka's join handshake, it fires mid-join on a routine standby
  restart (peer alive, join stalled behind stale-incarnation removal) and forms a
  permanent second cluster at the same address. ScadaBridge measured a split unhealed
  after 90s; Task 8 hit the same shape live here.
- The 'a booting node only self-forms when NO seed answers InitJoin' claim in the Task 2
  doc comment is marked FALSE for the watchdog: that guarantee belongs to Akka's native
  FirstSeedNodeProcess, which runs inside the handshake; the timer does not, and fires
  while a seed is answering.
- Task 8's TCP reachability guard is recorded as an INTERIM patch, not the fix — the
  race survived it, which is why the watchdog was deleted rather than patched.
- Shipped design restated: self-first seed ordering (AkkaClusterOptionsValidator +
  SelfFirstSeedBootstrapTests) plus the opt-in, default-OFF bootstrap guard (d1dac87f).

Source re-verified 2026-08-01: the defective watchdog is NOT present in this repo — no
ClusterBootstrapFallback.cs, no SelfFormAfter option, no Cluster.Join(SelfAddress) in
src/. Documentation-only change; no code touched.
This commit is contained in:
Joseph Doherty
2026-08-01 11:09:57 -04:00
parent a88cefd5ee
commit 9303289d05
1 changed files with 56 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md
+56 -1
View File
@@ -17,6 +17,55 @@
> the execution record, including the Task 8 live-gate evidence, which is the whole reason the
> watchdog was rejected. Do not execute the Task 1/2/3 code as written.
> ## 🛑 CORRECTION (2026-08-01) — watchdog REJECTED outright; the TCP guard was interim, not the fix
>
> Family-wide docs correction, applied to this plan and to its scadaproj half
> (`~/Desktop/scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md`). Two claims still
> readable in the task text below are wrong and are struck in place:
>
> 1. **The watchdog pattern was implemented and REJECTED after live testing — in both repos.** A
> timer that waits a window for membership and then calls `Cluster.Join(SelfAddress)` is
> **defective by construction**, not merely mis-tuned. On a *routine standby restart* — the peer
> is alive and answering, but the join stalls behind removal of the restarting node's own stale
> incarnation — the timer fires **mid-join**, abandons the in-flight join and forms a **permanent
> second cluster at the same address**. ScadaBridge measured a split that had **not healed after
> 90 s**; OtOpcUa hit the same shape live in the Task 8 manual-failover drill.
> 2. **"A booting node only self-forms when NO seed answers InitJoin" does NOT protect the
> watchdog** — that claim appears verbatim in the Task 2 doc comment below and is the reasoning
> error at the root of this design. The property is real, but it belongs to **Akka's native
> first-seed rule** (`FirstSeedNodeProcess`), which runs *inside* the join handshake. The watchdog
> runs *outside* it, seeing only "am I `Up` yet?", which cannot distinguish *no seed answered*
> from *a seed answered and the join is in flight*. The watchdog therefore **can fire while a seed
> is answering** — exactly what was observed.
> 3. **The Task 8 TCP reachability guard was an interim patch, not the resolution.** It suppressed
> the one observed shape; the race remained (a peer can answer between probe and join). The
> resolution was **deleting the watchdog**.
>
> **Shipped design (both repos, identical mechanism):** **self-first seed ordering** — every seed
> node lists ITSELF as `seed-nodes[0]`, so only Akka's own `FirstSeedNodeProcess` may form a new
> cluster, race-free inside the handshake. Here: `AkkaClusterOptionsValidator` (conditional — self
> must be `seed[0]` only IF self is a seed; identity on `PublicHostname`+port) and
> `SelfFirstSeedBootstrapTests`. Plus the separate **opt-in, default-OFF** simultaneous-cold-start
> **bootstrap guard** (`Cluster:BootstrapGuard:Enabled`, `ClusterBootstrapGuard` +
> `ClusterBootstrapCoordinator`, shipped here at `d1dac87f` and ported to ScadaBridge as Gitea #33),
> which decides the join order from a TCP reachability probe **before** a single `JoinSeedNodes` and
> **never re-forms mid-handshake** — the property the watchdog lacked. Authoritative family write-up:
> ScadaBridge `docs/requirements/Component-ClusterInfrastructure.md` → *Seed Node Ordering* and
> *Simultaneous cold start — the bootstrap guard*; local write-up `docs/Redundancy.md` →
> *Bootstrap: self-first seed ordering*.
>
> **Source re-verified 2026-08-01 — the defective watchdog is NOT present in this repo.**
> `ClusterBootstrapFallback.cs` does not exist under `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`; no
> `SelfFormAfter` option exists on `AkkaClusterOptions`; no `Cluster.Join(SelfAddress)` call exists
> anywhere in `src/` (the only `SelfAddress` uses are `AdminOperationsActor.cs:344/440/470`, reading
> `.Host` for an audit `SourceNode` stamp). `ServiceCollectionExtensions.cs:148-155` carries an
> explicit tombstone comment at the former arming site. What remains under
> `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/` is the **bootstrap guard** — `ClusterBootstrapGuard.cs` /
> `ClusterBootstrapCoordinator.cs`, which issue exactly one `JoinSeedNodes`
> (`ClusterBootstrapCoordinator.cs:116`) after a pre-join reachability decision and never re-join
> (`:139` — the watchdog loop there is read-only, it only logs an operator-visible WARN). **No code
> change is needed in this repo; this correction is documentation only.**
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
>
> Shared cross-repo design: `~/Desktop/scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md` (design rationale, MNTR assessment, behavior spec). The ScadaBridge half lives in `~/Desktop/ScadaBridge/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`. This plan is self-contained for execution.
@@ -47,7 +96,7 @@
| Peer boots after survivor self-formed | Its InitJoin is answered → joins as youngest. No island. |
| Both pair nodes cold-start simultaneously, mutually unreachable | Both self-form → dual-active (same partition class auto-down accepts; restart one side) |
| `SelfFormAfter` null/`≤0` | Disabled — today's wait-forever behavior |
| ~~Window expires mid-join-handshake~~ | ~~Benign: Akka ignores `Join` once joined~~ **FALSE — proven live (Task 8): `Join(self)` during an in-flight join abandons it and islands the node. Fixed by the TCP reachability guard: never self-form while any seed peer accepts a connection; wait a window and re-arm. ScadaBridge rejected the watchdog entirely for self-first seed ordering; OtOpcUa converges at mesh-program Phase 6.** |
| ~~Window expires mid-join-handshake~~ | ~~Benign: Akka ignores `Join` once joined~~ **FALSE — proven live (Task 8): `Join(self)` during an in-flight join abandons it and islands the node.** ~~Fixed by the TCP reachability guard: never self-form while any seed peer accepts a connection; wait a window and re-arm.~~ **Correction 2026-08-01: the TCP guard was an INTERIM patch that closed the one observed shape, not the fix — the race survived it. The watchdog was DELETED in both repos; the shipped mechanism is self-first seed ordering (+ the opt-in bootstrap guard for simultaneous cold start). The convergence did not wait for mesh-program Phase 6 — it landed the same day.** |
**Manual failover rules:** graceful `Leave` of the **oldest Up `driver` member** (identical query to `RedundancyStateActor.SelectDriverPrimary` — `Member.AgeOrdering`, `Leaving` excluded via `Status == Up`), never `Down`. The leaving node hands over via the cluster-leave phases, `CoordinatedShutdown` runs, `ActorSystemTerminationWatchdog` exits the process, the supervisor restarts it, and it rejoins as youngest → the survivor is Primary (ServiceLevel 250 moves; OPC UA clients re-select). Admin-only; peer guard (<2 Up driver members → disabled); confirmation dialog; audited via the shared `ZB.MOM.WW.Audit.IAuditWriter` seam before the Leave. **Mesh-scope caveat:** until the per-cluster mesh lands, the election is mesh-wide — the button acts on THE Primary of the whole Akka cluster, not per-`ServerCluster` row; the UI must say so (mirrors the KNOWN LIMITATION in `docs/Redundancy.md:84-99`).
@@ -126,9 +175,15 @@ namespace ZB.MOM.WW.OtOpcUa.Cluster;
/// <see cref="AkkaClusterOptions.SeedNodes"/> — a site node seeded only by central-1 must wait,
/// not island. Sequential recovery is island-free: Akka's join protocol prefers an existing
/// cluster (a booting node only self-forms when NO seed answers InitJoin).</para>
/// &lt;&lt;&lt; FALSE (correction 2026-08-01): that guarantee is Akka's FirstSeedNodeProcess's,
/// which lives INSIDE the join handshake. This timer lives outside it and fires while a seed IS
/// answering. &gt;&gt;&gt;
/// <para><b>Benign race:</b> Akka ignores <c>Join</c> once the node has joined a cluster this
/// incarnation. Residual risk: both pair nodes cold-starting inside the window while mutually
/// unreachable self-form separately — the same dual-active class auto-down accepts.</para>
/// &lt;&lt;&lt; FALSE (correction 2026-08-01): the race is NOT benign. Join(self) mid-handshake
/// abandons the in-flight join and forms a permanent second cluster at the same address. This
/// class was DELETED; do not re-create it. &gt;&gt;&gt;
/// </summary>
public static class ClusterBootstrapFallback
{