# Per-cluster mesh Phase 7 — failover drills + closing live gates

> **Final phase of the per-cluster mesh program.** Run the failover drill per pair type, close the two
> outstanding live gates (auto-down 1-vs-1 crash-the-oldest; self-first cold-start-alone), re-verify the
> manual-failover button, and ship a one-page co-located operator runbook.
>
> Runs against merged master on the docker-dev three-mesh rig (same rig as the Phase 6 gate).

## Scope clarified by recon

- **Failover mechanism is the same for every pair**: the oldest Up `driver` member leaves/dies, the peer
  becomes oldest, takes the `cluster-{ClusterId}` redundancy singleton, and advertises ServiceLevel 250.
- **Manual-failover button (`IManualFailoverService`) is MAIN-scoped by construction** — it acts on the
  local node's `Cluster.State` (graceful `Leave` of the oldest Up driver), and the AdminUI runs only on
  the central (MAIN) pair. Site pairs are **driver-only, no UI**; they fail over via **auto-down** (kill)
  only. This is the documented pair-local caveat from Phase 6, not a gap to fix.
- **Graceful (`docker stop`, SIGTERM → CoordinatedShutdown → cluster Leave) vs. ungraceful (`docker kill`,
  SIGKILL → peer auto-downs)** are the two failover paths. Manual failover = graceful Leave; a crashed
  node = auto-down.

## Drills

| # | Drill | Status | Evidence |
|---|---|---|---|
| D1 | MAIN graceful failover (manual-failover path) — Primary leaves, peer takes over | ✅ PASS | AdminUI "Trigger failover" on `/clusters/MAIN/redundancy`: central-2 "Exiting completed" → central-1 became Primary (250) → central-2 restarted, "Welcome from central-1" (rejoined, no split). Roles swapped 250↔240 |
| D2 | SITE-A auto-down failover, BOTH directions (kill Primary, then kill new Primary) | ✅ PASS | leg 8 killed site-a-1→site-a-2 Primary; D2 killed site-a-2→site-a-1 Primary (250), survived alone |
| D3 | SITE-B auto-down failover (crash-the-oldest) | ✅ PASS | killed site-b-1 (Primary/oldest) → site-b-2 "Member removed [site-b-1]", BecomingOldest→Oldest, 250 |
| D4 | Gate (a): auto-down 1-vs-1 crash-the-oldest — survivor stays Up, does not self-down | ✅ PASS | every survivor across D2/D3/D5/leg8 stayed Up (e.g. site-a-1 "Up 49 minutes", site-b-2 "Up 56 minutes") — no self-down. Closes the gate deferred since Phase 0a |
| D5 | Gate (b): self-first cold-start-alone — a node boots ALONE (partner down) and forms its mesh | ✅ PASS | stopped both site-b, started only site-b-1: "JOINING itself … forming a new cluster" → leader → Up → serves 250 as sole Primary |
| D6 | Recovery — a downed/restarted node rejoins its pair as Secondary (240) | ✅ PASS | central-2, site-a-2, site-b-2 all restarted → "Welcome from <founder>" → rejoined 240 |

**Phase 7 exit gate: MET.** All 6 drills pass on every pair type. Final fleet: three healthy 2-node
pairs, each one Primary (250) + one Secondary (240). Roles are wherever the last drill left them — each
pair has exactly one Primary, which is what matters.

## Findings / notes

- **Manual-failover button is MAIN-only, by construction** (acts on the local `Cluster.State`; AdminUI
  runs only on central). The UI now correctly states "this application Cluster's own Primary … each
  cluster's redundant pair runs its own independent 2-node Akka mesh" (Phase 6 text, verified live).
  Site pairs (driver-only, no UI) fail over via auto-down (crash) only — there is no manual-failover
  surface for them, and that is correct for the topology.
- **Graceful failover restarts the node in-place and it rejoins** (no split), because a single peer that
  is already Up answers the restarting node's InitJoin. The **simultaneous cold-start** split (Phase 6
  finding) only bites when BOTH pair nodes start from nothing at once — see the runbook mitigation.
- **Carried Phase-6 finding — simultaneous cold-start of both pair VMs — NOW CLOSED by a product guard
  (`279d1d0f`).** On the real co-located VMs a site's two VMs can power-cycle together; two self-first
  seeds can then each form a 1-node cluster (split brain, two Primaries in one pair). Two mitigations now
  exist: (1) **operational** — stagger the two VMs' service-manager start / start the founder first (the
  docker-dev `depends_on: service_healthy` serialization, still used by site-b); (2) **product** — the
  opt-in `Cluster:BootstrapGuard` (default off), which makes the lower-address node the founder and the
  higher node probe-then-join, arbitrating the race inside the join decision. Live-gated on the site-a
  pair (guard on, serialization removed): simultaneous start → 250/240, no split; higher-node cold-start
  -alone → self-forms → 250. See `docs/Redundancy.md` §"Bootstrap guard". The guard does NOT need the
  compose `depends_on` that production hardware lacks — it is the production-faithful fix.

---

## Operator runbook — co-located OtOpcUa + ScadaBridge site failover (one page)

**Topology.** Each site = 2 Windows VMs. Each VM runs **one OtOpcUa driver node** (`driver,cluster-SITE-X`,
Akka `:4053`, OPC UA `:4840`) **and one ScadaBridge site node** — two independent 2-node Akka clusters
sharing the hardware, never a shared mesh. Central = 2 VMs, each an OtOpcUa `admin,driver,cluster-MAIN`
node (hosts the AdminUI) + a ScadaBridge central node.

**Normal state.** Each pair: one node Primary (OPC UA ServiceLevel **250**), one Secondary (**240**).
Check from any OPC UA client: `otopcua-cli redundancy -u opc.tcp://<node>:4840`.

**Planned failover (move the Primary off a VM — e.g. for patching):**
- *Central (MAIN) pair:* AdminUI → **Clusters → MAIN → Redundancy → Trigger failover → Confirm**. The
  Primary gracefully leaves, restarts, rejoins as Secondary; its peer becomes Primary (250) in a few
  seconds. OPC UA clients re-select the new Primary automatically.
- *Site pairs (SITE-A/SITE-B):* no UI (driver-only). Stop the OtOpcUa service on the Primary VM
  (graceful SIGTERM) — the peer auto-downs it and becomes Primary. Restart the service to rejoin.

**Unplanned failover (a VM dies).** The peer detects the loss (~10-15 s: failure detector + auto-down),
removes the dead node, takes the redundancy singleton, and advertises 250. **A 1-node pair keeps
serving** (auto-down 1-vs-1 survival — verified). Because the VM is shared, **both products fail over
together**: confirm the ScadaBridge site node on the surviving VM also took over (its own runbook).

**Recovery.** Bring the dead VM back. The OtOpcUa node cold-starts, finds its peer Up, and rejoins as
Secondary (240) — "Welcome from <peer>" in its log. No manual step.

**⚠️ Both VMs down at once (site power event).** Start the VMs **staggered**, or bring the **designated
founder VM up first** and wait for its OtOpcUa node to reach ServiceLevel 250 before starting the second
VM. Starting both OtOpcUa services simultaneously can split the pair into two 1-node clusters (two
Primaries). If that happens: stop the OtOpcUa service on the non-founder VM, confirm the founder is the
lone Primary (250), then restart the non-founder — it rejoins as Secondary.

**Health checks.**
- ServiceLevel per node: `otopcua-cli redundancy -u opc.tcp://<node>:4840` (250 Primary / 240 Secondary).
- MAIN membership + Primary: AdminUI → Clusters → MAIN → Redundancy.
- A pair showing **250/250** = split brain — apply the "both VMs down" recovery above.
- A pair showing **240/240 or a lone 240** = no Primary elected — check the singleton host is Up.