lmxopcua/docs/plans/2026-07-24-mesh-phase7-failover-drills.md

Per-cluster mesh Phase 7 — failover drills + closing live gates

Final phase of the per-cluster mesh program. Run the failover drill per pair type, close the two outstanding live gates (auto-down 1-vs-1 crash-the-oldest; self-first cold-start-alone), re-verify the manual-failover button, and ship a one-page co-located operator runbook.

Runs against merged master on the docker-dev three-mesh rig (same rig as the Phase 6 gate).

Scope clarified by recon

• Failover mechanism is the same for every pair: the oldest Up driver member leaves/dies, the peer becomes oldest, takes the cluster-{ClusterId} redundancy singleton, and advertises ServiceLevel 250.
• Manual-failover button (IManualFailoverService) is MAIN-scoped by construction — it acts on the local node's Cluster.State (graceful Leave of the oldest Up driver), and the AdminUI runs only on the central (MAIN) pair. Site pairs are driver-only, no UI; they fail over via auto-down (kill) only. This is the documented pair-local caveat from Phase 6, not a gap to fix.
• Graceful (docker stop, SIGTERM → CoordinatedShutdown → cluster Leave) vs. ungraceful (docker kill, SIGKILL → peer auto-downs) are the two failover paths. Manual failover = graceful Leave; a crashed node = auto-down.

Drills

#	Drill	Status	Evidence
D1	MAIN graceful failover (manual-failover path) — Primary leaves, peer takes over	✅ PASS	AdminUI "Trigger failover" on /clusters/MAIN/redundancy: central-2 "Exiting completed" → central-1 became Primary (250) → central-2 restarted, "Welcome from central-1" (rejoined, no split). Roles swapped 250↔240
D2	SITE-A auto-down failover, BOTH directions (kill Primary, then kill new Primary)	✅ PASS	leg 8 killed site-a-1→site-a-2 Primary; D2 killed site-a-2→site-a-1 Primary (250), survived alone
D3	SITE-B auto-down failover (crash-the-oldest)	✅ PASS	killed site-b-1 (Primary/oldest) → site-b-2 "Member removed [site-b-1]", BecomingOldest→Oldest, 250
D4	Gate (a): auto-down 1-vs-1 crash-the-oldest — survivor stays Up, does not self-down	✅ PASS	every survivor across D2/D3/D5/leg8 stayed Up (e.g. site-a-1 "Up 49 minutes", site-b-2 "Up 56 minutes") — no self-down. Closes the gate deferred since Phase 0a
D5	Gate (b): self-first cold-start-alone — a node boots ALONE (partner down) and forms its mesh	✅ PASS	stopped both site-b, started only site-b-1: "JOINING itself … forming a new cluster" → leader → Up → serves 250 as sole Primary
D6	Recovery — a downed/restarted node rejoins its pair as Secondary (240)	✅ PASS	central-2, site-a-2, site-b-2 all restarted → "Welcome from " → rejoined 240

Phase 7 exit gate: MET. All 6 drills pass on every pair type. Final fleet: three healthy 2-node pairs, each one Primary (250) + one Secondary (240). Roles are wherever the last drill left them — each pair has exactly one Primary, which is what matters.

Findings / notes

• Manual-failover button is MAIN-only, by construction (acts on the local Cluster.State; AdminUI runs only on central). The UI now correctly states "this application Cluster's own Primary … each cluster's redundant pair runs its own independent 2-node Akka mesh" (Phase 6 text, verified live). Site pairs (driver-only, no UI) fail over via auto-down (crash) only — there is no manual-failover surface for them, and that is correct for the topology.
• Graceful failover restarts the node in-place and it rejoins (no split), because a single peer that is already Up answers the restarting node's InitJoin. The simultaneous cold-start split (Phase 6 finding) only bites when BOTH pair nodes start from nothing at once — see the runbook mitigation.
• Carried Phase-6 finding — simultaneous cold-start of both pair VMs — NOW CLOSED by a product guard (279d1d0f). On the real co-located VMs a site's two VMs can power-cycle together; two self-first seeds can then each form a 1-node cluster (split brain, two Primaries in one pair). Two mitigations now exist: (1) operational — stagger the two VMs' service-manager start / start the founder first (the docker-dev depends_on: service_healthy serialization, still used by site-b); (2) product — the opt-in Cluster:BootstrapGuard (default off), which makes the lower-address node the founder and the higher node probe-then-join, arbitrating the race inside the join decision. Live-gated on the site-a pair (guard on, serialization removed): simultaneous start → 250/240, no split; higher-node cold-start -alone → self-forms → 250. See docs/Redundancy.md §"Bootstrap guard". The guard does NOT need the compose depends_on that production hardware lacks — it is the production-faithful fix.

---

Operator runbook — co-located OtOpcUa + ScadaBridge site failover (one page)

Topology. Each site = 2 Windows VMs. Each VM runs one OtOpcUa driver node (driver,cluster-SITE-X, Akka :4053, OPC UA :4840) and one ScadaBridge site node — two independent 2-node Akka clusters sharing the hardware, never a shared mesh. Central = 2 VMs, each an OtOpcUa admin,driver,cluster-MAIN node (hosts the AdminUI) + a ScadaBridge central node.

Normal state. Each pair: one node Primary (OPC UA ServiceLevel 250), one Secondary (240). Check from any OPC UA client: otopcua-cli redundancy -u opc.tcp://<node>:4840.

Planned failover (move the Primary off a VM — e.g. for patching):

• Central (MAIN) pair: AdminUI → Clusters → MAIN → Redundancy → Trigger failover → Confirm. The Primary gracefully leaves, restarts, rejoins as Secondary; its peer becomes Primary (250) in a few seconds. OPC UA clients re-select the new Primary automatically.
• Site pairs (SITE-A/SITE-B): no UI (driver-only). Stop the OtOpcUa service on the Primary VM (graceful SIGTERM) — the peer auto-downs it and becomes Primary. Restart the service to rejoin.

Unplanned failover (a VM dies). The peer detects the loss (~10-15 s: failure detector + auto-down), removes the dead node, takes the redundancy singleton, and advertises 250. A 1-node pair keeps serving (auto-down 1-vs-1 survival — verified). Because the VM is shared, both products fail over together: confirm the ScadaBridge site node on the surviving VM also took over (its own runbook).

Recovery. Bring the dead VM back. The OtOpcUa node cold-starts, finds its peer Up, and rejoins as Secondary (240) — "Welcome from " in its log. No manual step.

⚠️ Both VMs down at once (site power event). Start the VMs staggered, or bring the designated founder VM up first and wait for its OtOpcUa node to reach ServiceLevel 250 before starting the second VM. Starting both OtOpcUa services simultaneously can split the pair into two 1-node clusters (two Primaries). If that happens: stop the OtOpcUa service on the non-founder VM, confirm the founder is the lone Primary (250), then restart the non-founder — it rejoins as Secondary.

Health checks.

• ServiceLevel per node: otopcua-cli redundancy -u opc.tcp://<node>:4840 (250 Primary / 240 Secondary).
• MAIN membership + Primary: AdminUI → Clusters → MAIN → Redundancy.
• A pair showing 250/250 = split brain — apply the "both VMs down" recovery above.
• A pair showing 240/240 or a lone 240 = no Primary elected — check the singleton host is Up.