scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md

# InitJoin Self-Form Fallback (`Cluster:SelfFormAfter`) + Manual Failover Control — Shared Design & Index

> **For Claude:** This is the family-level DESIGN + INDEX document. The executable per-repo plans live in the repos (split 2026-07-22 — the OtOpcUa plan is expected to grow and has a reserved-additions section):
>
> - **ScadaBridge:** `~/Desktop/ScadaBridge/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md` (10 tasks: fallback 1–7, manual-failover UI 8–9, optional site-pair failover 10)
> - **OtOpcUa:** `~/Desktop/OtOpcUa/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md` (7 tasks + "Reserved for additions" section)
> - **OtOpcUa mesh-alignment PROGRAM (added 2026-07-22):** `~/Desktop/OtOpcUa/docs/plans/2026-07-22-per-cluster-mesh-program.md` — Phases 1–7 moving OtOpcUa to ScadaBridge's mesh shape (one 2-node Akka mesh per application Cluster, ClusterClient/gRPC/fetch transports, driver nodes off the ConfigDb), driven by the co-location decision: OtOpcUa pairs run on the SAME Windows VMs as the ScadaBridge nodes. The selfform plan above is its prerequisite. Design: `OtOpcUa/docs/plans/2026-07-21-per-cluster-mesh-design.md`.
>
> Execute each with superpowers-extended-cc:executing-plans **in that repo**. The two plans are fully independent and can run in parallel. Only Task C1 below executes from this file.

**Goal:** (1) Either node of a 2-node pair can cold-start alone and become operational, unattended — closing the seed-node bootstrap gap in both ScadaBridge and OtOpcUa via a configurable, fast (default 10 s) InitJoin timeout that falls back to self-forming a cluster. (2) An admin-only "Trigger failover" control on each app's cluster-status page for operator-initiated, graceful role swaps.

> ## ⚠️ EXECUTION OUTCOME (2026-07-22) — design partially overturned
>
> Both repo plans executed. **The behavior-spec row "Window expires while a join is mid-handshake → benign race: Akka ignores `Join` once joined" is FALSE** — the watchdog sits outside Akka's join handshake, and a `Join(self)` fired while a join is in flight (routine restart into a live peer, join stalled behind stale-incarnation removal) abandons the join and forms a **permanent second cluster at the same address**. Proven independently by a ScadaBridge test (split unhealed after 90 s) and by an OtOpcUa live island during the manual-failover drill.
>
> - **ScadaBridge** rejected the watchdog entirely and shipped **self-first seed ordering** (each node lists itself as `seed-nodes[0]`; Akka's `FirstSeedNodeProcess` provides the intended semantics race-free inside the handshake; `StartupValidator` hard-gates the ordering). No `SelfFormAfter` option exists there. See its plan's "Part 1 as executed".
> - **OtOpcUa** first kept `SelfFormAfter` behind a TCP reachability guard, then **converged the same day** (`lmxopcua @ 3f24d4d6`): `ClusterBootstrapFallback` + `SelfFormAfter` deleted, self-first ordering shipped with a conditional `AkkaClusterOptionsValidator` (self must be `seed[0]` only IF self is a seed — site nodes exempt; identity on `PublicHostname`+port, not the 0.0.0.0 bind).
> - **Both repos now share the identical mechanism.** Everything merged + pushed 2026-07-22, including ScadaBridge Task 10 (site-pair failover relayed from the central UI).
> - The design sections below are retained as the decision record — read them through this banner. `akka_failover.md` §6.1 carries the corrected family-level account.

> ## 🛑 CORRECTION (2026-08-01) — the self-form watchdog is REJECTED, not shipped
>
> Re-read of this doc found the false safety claim still sitting **unmarked inside the design body**
> (behavior-spec row "Window expires while a join is mid-handshake" and the Mechanism section's
> "Races are benign") even though the 2026-07-22 banner above overturns it. Both are now struck in
> place. Stating it plainly, once, so no future reader reconstructs the pattern from the surviving prose:
>
> 1. **The watchdog pattern was implemented and REJECTED after live testing.** A timer that waits a
>    window for membership and then calls `Cluster.Join(SelfAddress)` is **defective**. On a *routine
>    standby restart* — peer alive and answering, join stalled behind removal of the restarting node's
>    own stale incarnation (the peer must down it, wait out the failure detector, then take a leader
>    action) — the timer fires **mid-join**, abandons the in-flight join, and forms a **second cluster
>    at the same address**. Measured on ScadaBridge: **still split after 90 s** — a routine restart
>    converted into an outage of the previously-healthy node. OtOpcUa hit the same race live during
>    its manual-failover drill.
> 2. **The "only self-forms when no seed answers InitJoin" reasoning does NOT protect the watchdog.**
>    That property is true of **Akka's native first-seed rule** (`FirstSeedNodeProcess`), which lives
>    *inside* the join handshake and therefore knows whether a seed answered. The watchdog lives
>    *outside* the handshake: its only signal is "am I `Up` yet?", which cannot distinguish *no seed
>    answered* from *a seed answered and the join is in flight*. So the watchdog **can and does fire
>    while a seed is answering**. Any doc, comment or plan that justifies a self-form timer with the
>    InitJoin-answer argument is quoting the native rule's guarantee to excuse a mechanism that does
>    not have it.
> 3. **The surviving design is documented in ScadaBridge, not here.** Authoritative source:
>    `~/Desktop/ScadaBridge/docs/requirements/Component-ClusterInfrastructure.md` →
>    **Seed Node Ordering** (self-first: every node lists ITSELF as `seed-nodes[0]`; only
>    `seed-nodes[0]` may self-form, via Akka's own `FirstSeedNodeProcess`; `StartupValidator`
>    hard-gates the ordering; `SelfFirstSeedBootstrapTests`) and **Simultaneous cold start — the
>    bootstrap guard** (the separate, **opt-in, default-OFF** `ScadaBridge:Cluster:BootstrapGuard`,
>    Gitea #33, which decides the join order from a TCP reachability probe **before** a single
>    `JoinSeedNodes` and **never re-forms mid-handshake** — that last property is precisely what the
>    watchdog lacked). Companion decision record:
>    `~/Desktop/ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md`.
> 4. **Neither repo ships the watchdog.** ScadaBridge never merged it. OtOpcUa deleted
>    `ClusterBootstrapFallback` + `SelfFormAfter` the same day; source re-verified 2026-08-01 —
>    see the correction section in that repo's plan.
>
> Nothing below this banner is safe to execute as written. The Design sections are history.

**Decision provenance:** `scadaproj/akka_failover.md` §6.1 (decided 2026-07-22); design priority = availability over partition safety (memory `ha-availability-over-partition-safety`). Nodes share a datacenter/LAN → a live peer answers InitJoin in milliseconds, so 10 s is generous; operators tune per site.

---

## Design

### The defect being fixed

Akka.NET only allows the **first-listed seed** to self-join and form a *new* cluster. Every other node loops on `InitJoin` forever until some existing member answers. Both repos document this as the "registered outage gap": a non-first seed cold-starting while the first seed is down never becomes Up (recovery today = manual env-var override or reviving the first seed). ScadaBridge's `ClusterOptions.SeedNodes` doc comment even claims "either can start first and form the cluster" — which the deployed configs do **not** deliver (both nodes list node-a first). The plans make the claim true and fix the comment.

### Behavior specification

| Scenario | Behavior with fallback |
|---|---|
| Peer alive (any boot order) | Normal seed join within ms — fallback never fires (membership arrives before the window) |
| Lone cold-start, node IS in its own seed list | After `SelfFormAfter` (default 10 s): warn log + `Cluster.Join(SelfAddress)` → Up alone, singletons start (`min-nr-of-members=1`) |
| Lone cold-start, node NOT in its own seed list | Fallback inert (info log at arm time); node waits for a seed — self-forming would island it. **This is the OtOpcUa site-node topology** (seeded only by `central-1`). |
| Peer boots after survivor self-formed | Peer's `InitJoin` reaches the self-formed node (it's in the peer's seed list) → joins as youngest/standby. No island. |
| Both cold-start simultaneously, mutually unreachable (boot-time partition) | Both self-form → two islands (same dual-active class auto-down already accepts; same recovery: restart one side). Mitigable with a staggered service start. |
| `SelfFormAfter` = `null` or `<= 0` | Disabled — today's wait-forever behavior |
| ~~Window expires while a join is mid-handshake~~ | ~~Benign race: Akka ignores `Join` once the node has already joined a cluster~~ **FALSE — see the 2026-08-01 correction banner. `Cluster.Join(SelfAddress)` is NOT ignored mid-handshake: it abandons the in-flight join and forms a permanent second cluster at the same address (ScadaBridge: still split after 90 s; OtOpcUa: live island during the manual-failover drill). This row is the whole reason the watchdog was rejected.** |

### Config surface (same key both apps — family convention, like `SplitBrainResolverStrategy`)

- **ScadaBridge:** `ScadaBridge:Cluster:SelfFormAfter` → `ClusterOptions.SelfFormAfter` (`TimeSpan?`, default `00:00:10`), validated non-negative-or-null by `ClusterOptionsValidator`.
- **OtOpcUa:** `Cluster:SelfFormAfter` → `AkkaClusterOptions.SelfFormAfter` (`TimeSpan?`, default `00:00:10`); disable-on-`<=0` semantics live in the fallback itself.
- NOT a code constant (deliberate — see the `DowningStableAfter` lesson, `akka_failover.md` §6.2).

### Mechanism (both apps, deliberately duplicated like the termination watchdogs)

`ClusterBootstrapFallback.Arm(system, options, logger)`, called at ActorSystem creation (ScadaBridge: inside `AkkaHostedService.GetOrCreateActorSystem`; OtOpcUa: an Akka.Hosting `AddStartup` task inside `WithOtOpcUaClusterBootstrap`, so production and test hosts arm identically):

1. `SelfFormAfter` null/`≤0` → log info, return (disabled).
2. Self address not in own `SeedNodes` (parsed via `Address.Parse`, exact equality) → log info, return (island guard).
3. `RegisterOnMemberUp` completes a TCS; race it against `Task.Delay(window)`.
4. On window expiry (and system not terminated): prominent warn log + `Cluster.Join(cluster.SelfAddress)`.

~~Races are benign — Akka ignores `Join` once joined this incarnation.~~ **FALSE (correction
2026-08-01).** Step 3's success signal (`RegisterOnMemberUp`) is outside Akka's join handshake and
cannot tell *no seed answered* from *a seed answered and the join is in flight*; step 4's
`Cluster.Join(self)` fired during `TryingToJoin` wins and islands the node. **Do not implement this
mechanism.** The InitJoin-answer guarantee belongs to Akka's `FirstSeedNodeProcess`, not to a timer
bolted alongside it — use self-first seed ordering (ScadaBridge
`Component-ClusterInfrastructure.md` → Seed Node Ordering).

### Manual failover control (Part D design)

**Mechanism — graceful `Cluster.Leave`, never `Down`.** "Trigger failover" = issue `Cluster.Leave(<current active/primary member's address>)` from whichever node serves the UI (any member may initiate a leave for any member). The leaving node walks the graceful path: singleton hand-over via the cluster-leave phases (ScadaBridge additionally drains each singleton 10 s), its `CoordinatedShutdown` runs, the existing termination watchdog exits the process, the service supervisor restarts it, and it rejoins as the **youngest** member — a permanent role swap under the oldest-Up election, with no downing window and no dead interval beyond hand-over latency.

Shared rules:
- **Admin-only:** button renders inside the app's admin-policy `AuthorizeView` (ScadaBridge `AuthorizationPolicies.RequireAdmin` on `/monitoring/health`; OtOpcUa the mutating-pages policy from `AdminUiPolicies` on `/clusters/{id}/redundancy`).
- **Peer guard:** disabled (with tooltip) when fewer than 2 Up members carry the relevant role.
- **Confirmation dialog** stating exactly what happens.
- **Audited** through each app's existing audit seam before the Leave (`cluster.manual-failover`; ScadaBridge uses its CENTRAL audit writer, not the shared seam — dual-seam gotcha).
- **UX note (ScadaBridge):** Traefik routes the UI to the *active* node, so triggering central failover drops the Blazor circuit; the page reconnects against the new active. The dialog warns about this.
- **Scope note (OtOpcUa):** until the per-cluster mesh lands, the driver-Primary election is mesh-wide — the button acts on THE Primary of the whole Akka cluster; the panel says so (mirrors the KNOWN LIMITATION in `docs/Redundancy.md`).
- **Election parity:** each service's target query is intentionally identical to the app's active/primary election (`ActiveNodeEvaluator.SelfIsOldestUp` / `RedundancyStateActor.SelectDriverPrimary`), pinned by a parity test — the node acted on must be exactly the one hosting the singletons.
- **Self-form interplay:** none — the restarted node rejoins via its peer (alive by definition here), so `SelfFormAfter` never fires on this path.

### Multi-node testing assessment (getakka.net/articles/testing/multi-node-testing.html)

**Verdict: usable in principle, deliberately NOT adopted.** The MultiNode TestKit (`Akka.MultiNode.TestAdapter` + `Akka.Remote.TestKit` + `Akka.Cluster.TestKit`) runs each `RoleName` as a separate process under `dotnet test`, with `TestConductor` able to `Blackhole` links and `Exit` nodes, synchronized by `EnterBarrier`. Reasons not to use it here:

1. **Everything this feature needs is testable in-process** with real ActorSystems built from production bootstrap code (ScadaBridge `TwoNodeClusterFixture`; OtOpcUa `SplitBrainResolverActivationTests` pattern). Same fidelity, no new runner.
2. **Heavy project-level cost:** dedicated test project, parallelization disabled assembly-wide, single-public-constructor constraints, MNTR log plumbing.
3. **OtOpcUa xunit constraint:** the Akka TestKit family is xunit-v2-only; OtOpcUa's integration tree is xunit.v3.
4. **The one MNTR-only scenario** (true boot-time partition → dual self-form) validates an already-documented *accepted* risk; the family's preferred proof is the docker rig drill (`failover-drill.sh` precedent, or `docker network disconnect`).

**Follow-up:** revisit alongside OtOpcUa per-cluster-mesh Phase 7 (failover-drill phase) if a scripted dual-island drill is ever wanted.

### Out of scope (recorded follow-ups)

- Making OtOpcUa's 15 s `DowningStableAfter` configurable — now listed in the OtOpcUa plan's "Reserved for additions".
- Failure-detector tuning knobs for faster failover (both apps) — see `akka_failover.md` §2 tuning table; OtOpcUa side listed in its reserved-additions section.
- Staggered Windows service start delays (pure ops; document only).
- `deploy/wonder-app-vd03/` gitignored production overlay (ops applies `SelfFormAfter` on next deploy).
- ScadaBridge site-pair failover from the central UI — optional Task 10 in the ScadaBridge plan (new central→site message contract; confirm with the user first).

---

## Task C1: Family docs flip (execute AFTER both repo plans complete and merge)

**Classification:** trivial
**Estimated implement time:** ~2 min

**Files:**
- Modify: `~/Desktop/scadaproj/akka_failover.md` §6.1 — "Status: decided, not yet implemented" → implemented (merge commit refs from both repos); leave the §6.2 `DowningStableAfter` line unless the OtOpcUa reserved-addition shipped.
- Modify: `~/Desktop/scadaproj/CLAUDE.md` — brief addition to the ScadaBridge and OtOpcUa index rows.
- Update memory `ha-availability-over-partition-safety.md` — mark the decision implemented.