scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md

InitJoin Self-Form Fallback (Cluster:SelfFormAfter) + Manual Failover Control — Shared Design & Index

For Claude: This is the family-level DESIGN + INDEX document. The executable per-repo plans live in the repos (split 2026-07-22 — the OtOpcUa plan is expected to grow and has a reserved-additions section):

• ScadaBridge: ~/Desktop/ScadaBridge/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md (10 tasks: fallback 1–7, manual-failover UI 8–9, optional site-pair failover 10)
• OtOpcUa: ~/Desktop/OtOpcUa/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md (7 tasks + "Reserved for additions" section)
• OtOpcUa mesh-alignment PROGRAM (added 2026-07-22): ~/Desktop/OtOpcUa/docs/plans/2026-07-22-per-cluster-mesh-program.md — Phases 1–7 moving OtOpcUa to ScadaBridge's mesh shape (one 2-node Akka mesh per application Cluster, ClusterClient/gRPC/fetch transports, driver nodes off the ConfigDb), driven by the co-location decision: OtOpcUa pairs run on the SAME Windows VMs as the ScadaBridge nodes. The selfform plan above is its prerequisite. Design: OtOpcUa/docs/plans/2026-07-21-per-cluster-mesh-design.md.

Execute each with superpowers-extended-cc:executing-plans in that repo. The two plans are fully independent and can run in parallel. Only Task C1 below executes from this file.

Goal: (1) Either node of a 2-node pair can cold-start alone and become operational, unattended — closing the seed-node bootstrap gap in both ScadaBridge and OtOpcUa via a configurable, fast (default 10 s) InitJoin timeout that falls back to self-forming a cluster. (2) An admin-only "Trigger failover" control on each app's cluster-status page for operator-initiated, graceful role swaps.

⚠️ EXECUTION OUTCOME (2026-07-22) — design partially overturned

Both repo plans executed. The behavior-spec row "Window expires while a join is mid-handshake → benign race: Akka ignores Join once joined" is FALSE — the watchdog sits outside Akka's join handshake, and a Join(self) fired while a join is in flight (routine restart into a live peer, join stalled behind stale-incarnation removal) abandons the join and forms a permanent second cluster at the same address. Proven independently by a ScadaBridge test (split unhealed after 90 s) and by an OtOpcUa live island during the manual-failover drill.

• ScadaBridge rejected the watchdog entirely and shipped self-first seed ordering (each node lists itself as seed-nodes[0]; Akka's FirstSeedNodeProcess provides the intended semantics race-free inside the handshake; StartupValidator hard-gates the ordering). No SelfFormAfter option exists there. See its plan's "Part 1 as executed".
• OtOpcUa first kept SelfFormAfter behind a TCP reachability guard, then converged the same day (lmxopcua @ 3f24d4d6): ClusterBootstrapFallback + SelfFormAfter deleted, self-first ordering shipped with a conditional AkkaClusterOptionsValidator (self must be seed[0] only IF self is a seed — site nodes exempt; identity on PublicHostname+port, not the 0.0.0.0 bind).
• Both repos now share the identical mechanism. Everything merged + pushed 2026-07-22, including ScadaBridge Task 10 (site-pair failover relayed from the central UI).
• The design sections below are retained as the decision record — read them through this banner. akka_failover.md §6.1 carries the corrected family-level account.

🛑 CORRECTION (2026-08-01) — the self-form watchdog is REJECTED, not shipped

Re-read of this doc found the false safety claim still sitting unmarked inside the design body (behavior-spec row "Window expires while a join is mid-handshake" and the Mechanism section's "Races are benign") even though the 2026-07-22 banner above overturns it. Both are now struck in place. Stating it plainly, once, so no future reader reconstructs the pattern from the surviving prose:

1. The watchdog pattern was implemented and REJECTED after live testing. A timer that waits a window for membership and then calls Cluster.Join(SelfAddress) is defective. On a routine standby restart — peer alive and answering, join stalled behind removal of the restarting node's own stale incarnation (the peer must down it, wait out the failure detector, then take a leader action) — the timer fires mid-join, abandons the in-flight join, and forms a second cluster at the same address. Measured on ScadaBridge: still split after 90 s — a routine restart converted into an outage of the previously-healthy node. OtOpcUa hit the same race live during its manual-failover drill.
2. The "only self-forms when no seed answers InitJoin" reasoning does NOT protect the watchdog. That property is true of Akka's native first-seed rule (FirstSeedNodeProcess), which lives inside the join handshake and therefore knows whether a seed answered. The watchdog lives outside the handshake: its only signal is "am I Up yet?", which cannot distinguish no seed answered from a seed answered and the join is in flight. So the watchdog can and does fire while a seed is answering. Any doc, comment or plan that justifies a self-form timer with the InitJoin-answer argument is quoting the native rule's guarantee to excuse a mechanism that does not have it.
3. The surviving design is documented in ScadaBridge, not here. Authoritative source: ~/Desktop/ScadaBridge/docs/requirements/Component-ClusterInfrastructure.md → Seed Node Ordering (self-first: every node lists ITSELF as seed-nodes[0]; only seed-nodes[0] may self-form, via Akka's own FirstSeedNodeProcess; StartupValidator hard-gates the ordering; SelfFirstSeedBootstrapTests) and Simultaneous cold start — the bootstrap guard (the separate, opt-in, default-OFF ScadaBridge:Cluster:BootstrapGuard, Gitea #33, which decides the join order from a TCP reachability probe before a single JoinSeedNodes and never re-forms mid-handshake — that last property is precisely what the watchdog lacked). Companion decision record: ~/Desktop/ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md.
4. Neither repo ships the watchdog. ScadaBridge never merged it. OtOpcUa deleted ClusterBootstrapFallback + SelfFormAfter the same day; source re-verified 2026-08-01 — see the correction section in that repo's plan.

Nothing below this banner is safe to execute as written. The Design sections are history.

Decision provenance: scadaproj/akka_failover.md §6.1 (decided 2026-07-22); design priority = availability over partition safety (memory ha-availability-over-partition-safety). Nodes share a datacenter/LAN → a live peer answers InitJoin in milliseconds, so 10 s is generous; operators tune per site.

---

Design

The defect being fixed

Akka.NET only allows the first-listed seed to self-join and form a new cluster. Every other node loops on InitJoin forever until some existing member answers. Both repos document this as the "registered outage gap": a non-first seed cold-starting while the first seed is down never becomes Up (recovery today = manual env-var override or reviving the first seed). ScadaBridge's ClusterOptions.SeedNodes doc comment even claims "either can start first and form the cluster" — which the deployed configs do not deliver (both nodes list node-a first). The plans make the claim true and fix the comment.

Behavior specification

Scenario	Behavior with fallback
Peer alive (any boot order)	Normal seed join within ms — fallback never fires (membership arrives before the window)
Lone cold-start, node IS in its own seed list	After SelfFormAfter (default 10 s): warn log + Cluster.Join(SelfAddress) → Up alone, singletons start (min-nr-of-members=1)
Lone cold-start, node NOT in its own seed list	Fallback inert (info log at arm time); node waits for a seed — self-forming would island it. This is the OtOpcUa site-node topology (seeded only by central-1).
Peer boots after survivor self-formed	Peer's InitJoin reaches the self-formed node (it's in the peer's seed list) → joins as youngest/standby. No island.
Both cold-start simultaneously, mutually unreachable (boot-time partition)	Both self-form → two islands (same dual-active class auto-down already accepts; same recovery: restart one side). Mitigable with a staggered service start.
SelfFormAfter = null or <= 0	Disabled — today's wait-forever behavior
Window expires while a join is mid-handshake	Benign race: Akka ignores Join once the node has already joined a cluster FALSE — see the 2026-08-01 correction banner. Cluster.Join(SelfAddress) is NOT ignored mid-handshake: it abandons the in-flight join and forms a permanent second cluster at the same address (ScadaBridge: still split after 90 s; OtOpcUa: live island during the manual-failover drill). This row is the whole reason the watchdog was rejected.

Config surface (same key both apps — family convention, like SplitBrainResolverStrategy)

• ScadaBridge: ScadaBridge:Cluster:SelfFormAfter → ClusterOptions.SelfFormAfter (TimeSpan?, default 00:00:10), validated non-negative-or-null by ClusterOptionsValidator.
• OtOpcUa: Cluster:SelfFormAfter → AkkaClusterOptions.SelfFormAfter (TimeSpan?, default 00:00:10); disable-on-<=0 semantics live in the fallback itself.
• NOT a code constant (deliberate — see the DowningStableAfter lesson, akka_failover.md §6.2).

Mechanism (both apps, deliberately duplicated like the termination watchdogs)

ClusterBootstrapFallback.Arm(system, options, logger), called at ActorSystem creation (ScadaBridge: inside AkkaHostedService.GetOrCreateActorSystem; OtOpcUa: an Akka.Hosting AddStartup task inside WithOtOpcUaClusterBootstrap, so production and test hosts arm identically):

1. SelfFormAfter null/≤0 → log info, return (disabled).
2. Self address not in own SeedNodes (parsed via Address.Parse, exact equality) → log info, return (island guard).
3. RegisterOnMemberUp completes a TCS; race it against Task.Delay(window).
4. On window expiry (and system not terminated): prominent warn log + Cluster.Join(cluster.SelfAddress).

Races are benign — Akka ignores Join once joined this incarnation. FALSE (correction 2026-08-01). Step 3's success signal (RegisterOnMemberUp) is outside Akka's join handshake and cannot tell no seed answered from a seed answered and the join is in flight; step 4's Cluster.Join(self) fired during TryingToJoin wins and islands the node. Do not implement this mechanism. The InitJoin-answer guarantee belongs to Akka's FirstSeedNodeProcess, not to a timer bolted alongside it — use self-first seed ordering (ScadaBridge Component-ClusterInfrastructure.md → Seed Node Ordering).

Manual failover control (Part D design)

Mechanism — graceful Cluster.Leave, never Down. "Trigger failover" = issue Cluster.Leave(<current active/primary member's address>) from whichever node serves the UI (any member may initiate a leave for any member). The leaving node walks the graceful path: singleton hand-over via the cluster-leave phases (ScadaBridge additionally drains each singleton 10 s), its CoordinatedShutdown runs, the existing termination watchdog exits the process, the service supervisor restarts it, and it rejoins as the youngest member — a permanent role swap under the oldest-Up election, with no downing window and no dead interval beyond hand-over latency.

Shared rules:

• Admin-only: button renders inside the app's admin-policy AuthorizeView (ScadaBridge AuthorizationPolicies.RequireAdmin on /monitoring/health; OtOpcUa the mutating-pages policy from AdminUiPolicies on /clusters/{id}/redundancy).
• Peer guard: disabled (with tooltip) when fewer than 2 Up members carry the relevant role.
• Confirmation dialog stating exactly what happens.
• Audited through each app's existing audit seam before the Leave (cluster.manual-failover; ScadaBridge uses its CENTRAL audit writer, not the shared seam — dual-seam gotcha).
• UX note (ScadaBridge): Traefik routes the UI to the active node, so triggering central failover drops the Blazor circuit; the page reconnects against the new active. The dialog warns about this.
• Scope note (OtOpcUa): until the per-cluster mesh lands, the driver-Primary election is mesh-wide — the button acts on THE Primary of the whole Akka cluster; the panel says so (mirrors the KNOWN LIMITATION in docs/Redundancy.md).
• Election parity: each service's target query is intentionally identical to the app's active/primary election (ActiveNodeEvaluator.SelfIsOldestUp / RedundancyStateActor.SelectDriverPrimary), pinned by a parity test — the node acted on must be exactly the one hosting the singletons.
• Self-form interplay: none — the restarted node rejoins via its peer (alive by definition here), so SelfFormAfter never fires on this path.

Multi-node testing assessment (getakka.net/articles/testing/multi-node-testing.html)

Verdict: usable in principle, deliberately NOT adopted. The MultiNode TestKit (Akka.MultiNode.TestAdapter + Akka.Remote.TestKit + Akka.Cluster.TestKit) runs each RoleName as a separate process under dotnet test, with TestConductor able to Blackhole links and Exit nodes, synchronized by EnterBarrier. Reasons not to use it here:

1. Everything this feature needs is testable in-process with real ActorSystems built from production bootstrap code (ScadaBridge TwoNodeClusterFixture; OtOpcUa SplitBrainResolverActivationTests pattern). Same fidelity, no new runner.
2. Heavy project-level cost: dedicated test project, parallelization disabled assembly-wide, single-public-constructor constraints, MNTR log plumbing.
3. OtOpcUa xunit constraint: the Akka TestKit family is xunit-v2-only; OtOpcUa's integration tree is xunit.v3.
4. The one MNTR-only scenario (true boot-time partition → dual self-form) validates an already-documented accepted risk; the family's preferred proof is the docker rig drill (failover-drill.sh precedent, or docker network disconnect).

Follow-up: revisit alongside OtOpcUa per-cluster-mesh Phase 7 (failover-drill phase) if a scripted dual-island drill is ever wanted.

Out of scope (recorded follow-ups)

• Making OtOpcUa's 15 s DowningStableAfter configurable — now listed in the OtOpcUa plan's "Reserved for additions".
• Failure-detector tuning knobs for faster failover (both apps) — see akka_failover.md §2 tuning table; OtOpcUa side listed in its reserved-additions section.
• Staggered Windows service start delays (pure ops; document only).
• deploy/wonder-app-vd03/ gitignored production overlay (ops applies SelfFormAfter on next deploy).
• ScadaBridge site-pair failover from the central UI — optional Task 10 in the ScadaBridge plan (new central→site message contract; confirm with the user first).

---

Task C1: Family docs flip (execute AFTER both repo plans complete and merge)

Classification: trivial Estimated implement time: ~2 min

Files:

• Modify: ~/Desktop/scadaproj/akka_failover.md §6.1 — "Status: decided, not yet implemented" → implemented (merge commit refs from both repos); leave the §6.2 DowningStableAfter line unless the OtOpcUa reserved-addition shipped.
• Modify: ~/Desktop/scadaproj/CLAUDE.md — brief addition to the ScadaBridge and OtOpcUa index rows.
• Update memory ha-availability-over-partition-safety.md — mark the decision implemented.