Joseph Doherty
c50ebcf7dc
All 6 failover drills pass on the docker-dev three-mesh rig: - D1 MAIN graceful manual-failover via the AdminUI "Trigger failover" button — Primary leaves, peer→250, restarts and rejoins with no split. - D2 SITE-A auto-down failover, both directions; D3 SITE-B crash-the-oldest. - D4 auto-down 1-vs-1 survival — every survivor stayed Up (closes the gate deferred since Phase 0a). - D5 self-first cold-start-alone — a node boots with its partner down and forms its own mesh (gate b). - D6 recovery — restarted nodes rejoin as Secondary (240). Ships a one-page co-located operator runbook (OtOpcUa + ScadaBridge on shared site VMs). Manual-failover button confirmed MAIN-only by construction; site pairs (driver-only, no UI) fail over via auto-down. Carried Phase-6 finding — simultaneous cold-start of both pair VMs can split — mitigated operationally in the runbook (staggered start); a product-level join guard is a candidate follow-up, not shipped. The per-cluster mesh program (Phases 0a–7) is COMPLETE. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
92 lines
7.0 KiB
Markdown
92 lines
7.0 KiB
Markdown
# Per-cluster mesh Phase 7 — failover drills + closing live gates
|
|
|
|
> **Final phase of the per-cluster mesh program.** Run the failover drill per pair type, close the two
|
|
> outstanding live gates (auto-down 1-vs-1 crash-the-oldest; self-first cold-start-alone), re-verify the
|
|
> manual-failover button, and ship a one-page co-located operator runbook.
|
|
>
|
|
> Runs against merged master on the docker-dev three-mesh rig (same rig as the Phase 6 gate).
|
|
|
|
## Scope clarified by recon
|
|
|
|
- **Failover mechanism is the same for every pair**: the oldest Up `driver` member leaves/dies, the peer
|
|
becomes oldest, takes the `cluster-{ClusterId}` redundancy singleton, and advertises ServiceLevel 250.
|
|
- **Manual-failover button (`IManualFailoverService`) is MAIN-scoped by construction** — it acts on the
|
|
local node's `Cluster.State` (graceful `Leave` of the oldest Up driver), and the AdminUI runs only on
|
|
the central (MAIN) pair. Site pairs are **driver-only, no UI**; they fail over via **auto-down** (kill)
|
|
only. This is the documented pair-local caveat from Phase 6, not a gap to fix.
|
|
- **Graceful (`docker stop`, SIGTERM → CoordinatedShutdown → cluster Leave) vs. ungraceful (`docker kill`,
|
|
SIGKILL → peer auto-downs)** are the two failover paths. Manual failover = graceful Leave; a crashed
|
|
node = auto-down.
|
|
|
|
## Drills
|
|
|
|
| # | Drill | Status | Evidence |
|
|
|---|---|---|---|
|
|
| D1 | MAIN graceful failover (manual-failover path) — Primary leaves, peer takes over | ✅ PASS | AdminUI "Trigger failover" on `/clusters/MAIN/redundancy`: central-2 "Exiting completed" → central-1 became Primary (250) → central-2 restarted, "Welcome from central-1" (rejoined, no split). Roles swapped 250↔240 |
|
|
| D2 | SITE-A auto-down failover, BOTH directions (kill Primary, then kill new Primary) | ✅ PASS | leg 8 killed site-a-1→site-a-2 Primary; D2 killed site-a-2→site-a-1 Primary (250), survived alone |
|
|
| D3 | SITE-B auto-down failover (crash-the-oldest) | ✅ PASS | killed site-b-1 (Primary/oldest) → site-b-2 "Member removed [site-b-1]", BecomingOldest→Oldest, 250 |
|
|
| D4 | Gate (a): auto-down 1-vs-1 crash-the-oldest — survivor stays Up, does not self-down | ✅ PASS | every survivor across D2/D3/D5/leg8 stayed Up (e.g. site-a-1 "Up 49 minutes", site-b-2 "Up 56 minutes") — no self-down. Closes the gate deferred since Phase 0a |
|
|
| D5 | Gate (b): self-first cold-start-alone — a node boots ALONE (partner down) and forms its mesh | ✅ PASS | stopped both site-b, started only site-b-1: "JOINING itself … forming a new cluster" → leader → Up → serves 250 as sole Primary |
|
|
| D6 | Recovery — a downed/restarted node rejoins its pair as Secondary (240) | ✅ PASS | central-2, site-a-2, site-b-2 all restarted → "Welcome from <founder>" → rejoined 240 |
|
|
|
|
**Phase 7 exit gate: MET.** All 6 drills pass on every pair type. Final fleet: three healthy 2-node
|
|
pairs, each one Primary (250) + one Secondary (240). Roles are wherever the last drill left them — each
|
|
pair has exactly one Primary, which is what matters.
|
|
|
|
## Findings / notes
|
|
|
|
- **Manual-failover button is MAIN-only, by construction** (acts on the local `Cluster.State`; AdminUI
|
|
runs only on central). The UI now correctly states "this application Cluster's own Primary … each
|
|
cluster's redundant pair runs its own independent 2-node Akka mesh" (Phase 6 text, verified live).
|
|
Site pairs (driver-only, no UI) fail over via auto-down (crash) only — there is no manual-failover
|
|
surface for them, and that is correct for the topology.
|
|
- **Graceful failover restarts the node in-place and it rejoins** (no split), because a single peer that
|
|
is already Up answers the restarting node's InitJoin. The **simultaneous cold-start** split (Phase 6
|
|
finding) only bites when BOTH pair nodes start from nothing at once — see the runbook mitigation.
|
|
- **Carried Phase-6 finding — simultaneous cold-start of both pair VMs.** On the real co-located VMs a
|
|
site's two VMs can power-cycle together; two self-first seeds can then each form a 1-node cluster
|
|
(split brain, two Primaries in one pair). The docker-dev rig serializes with a founder healthcheck +
|
|
`depends_on: service_healthy`. **Production mitigation (operational, not code):** stagger the two VMs'
|
|
service-manager start, or start the designated founder VM first. A product-level guard (e.g. a brief
|
|
reachability-gated join delay on the non-founder) is a candidate follow-up if operational staggering
|
|
proves fragile — tracked as a Phase 7 note, not shipped.
|
|
|
|
---
|
|
|
|
## Operator runbook — co-located OtOpcUa + ScadaBridge site failover (one page)
|
|
|
|
**Topology.** Each site = 2 Windows VMs. Each VM runs **one OtOpcUa driver node** (`driver,cluster-SITE-X`,
|
|
Akka `:4053`, OPC UA `:4840`) **and one ScadaBridge site node** — two independent 2-node Akka clusters
|
|
sharing the hardware, never a shared mesh. Central = 2 VMs, each an OtOpcUa `admin,driver,cluster-MAIN`
|
|
node (hosts the AdminUI) + a ScadaBridge central node.
|
|
|
|
**Normal state.** Each pair: one node Primary (OPC UA ServiceLevel **250**), one Secondary (**240**).
|
|
Check from any OPC UA client: `otopcua-cli redundancy -u opc.tcp://<node>:4840`.
|
|
|
|
**Planned failover (move the Primary off a VM — e.g. for patching):**
|
|
- *Central (MAIN) pair:* AdminUI → **Clusters → MAIN → Redundancy → Trigger failover → Confirm**. The
|
|
Primary gracefully leaves, restarts, rejoins as Secondary; its peer becomes Primary (250) in a few
|
|
seconds. OPC UA clients re-select the new Primary automatically.
|
|
- *Site pairs (SITE-A/SITE-B):* no UI (driver-only). Stop the OtOpcUa service on the Primary VM
|
|
(graceful SIGTERM) — the peer auto-downs it and becomes Primary. Restart the service to rejoin.
|
|
|
|
**Unplanned failover (a VM dies).** The peer detects the loss (~10-15 s: failure detector + auto-down),
|
|
removes the dead node, takes the redundancy singleton, and advertises 250. **A 1-node pair keeps
|
|
serving** (auto-down 1-vs-1 survival — verified). Because the VM is shared, **both products fail over
|
|
together**: confirm the ScadaBridge site node on the surviving VM also took over (its own runbook).
|
|
|
|
**Recovery.** Bring the dead VM back. The OtOpcUa node cold-starts, finds its peer Up, and rejoins as
|
|
Secondary (240) — "Welcome from <peer>" in its log. No manual step.
|
|
|
|
**⚠️ Both VMs down at once (site power event).** Start the VMs **staggered**, or bring the **designated
|
|
founder VM up first** and wait for its OtOpcUa node to reach ServiceLevel 250 before starting the second
|
|
VM. Starting both OtOpcUa services simultaneously can split the pair into two 1-node clusters (two
|
|
Primaries). If that happens: stop the OtOpcUa service on the non-founder VM, confirm the founder is the
|
|
lone Primary (250), then restart the non-founder — it rejoins as Secondary.
|
|
|
|
**Health checks.**
|
|
- ServiceLevel per node: `otopcua-cli redundancy -u opc.tcp://<node>:4840` (250 Primary / 240 Secondary).
|
|
- MAIN membership + Primary: AdminUI → Clusters → MAIN → Redundancy.
|
|
- A pair showing **250/250** = split brain — apply the "both VMs down" recovery above.
|
|
- A pair showing **240/240 or a lone 240** = no Primary elected — check the singleton host is Up.
|