Joseph Doherty
d1dac87f6f
The simultaneous-cold-start split-brain, carried as a Phase 7 "candidate follow-up,
not shipped", is now closed by the opt-in Cluster:BootstrapGuard (279d1d0f). Documents
the guard in CLAUDE.md's bootstrap section and marks the Phase 7 finding closed.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
95 lines
7.4 KiB
Markdown
95 lines
7.4 KiB
Markdown
# Per-cluster mesh Phase 7 — failover drills + closing live gates
|
|
|
|
> **Final phase of the per-cluster mesh program.** Run the failover drill per pair type, close the two
|
|
> outstanding live gates (auto-down 1-vs-1 crash-the-oldest; self-first cold-start-alone), re-verify the
|
|
> manual-failover button, and ship a one-page co-located operator runbook.
|
|
>
|
|
> Runs against merged master on the docker-dev three-mesh rig (same rig as the Phase 6 gate).
|
|
|
|
## Scope clarified by recon
|
|
|
|
- **Failover mechanism is the same for every pair**: the oldest Up `driver` member leaves/dies, the peer
|
|
becomes oldest, takes the `cluster-{ClusterId}` redundancy singleton, and advertises ServiceLevel 250.
|
|
- **Manual-failover button (`IManualFailoverService`) is MAIN-scoped by construction** — it acts on the
|
|
local node's `Cluster.State` (graceful `Leave` of the oldest Up driver), and the AdminUI runs only on
|
|
the central (MAIN) pair. Site pairs are **driver-only, no UI**; they fail over via **auto-down** (kill)
|
|
only. This is the documented pair-local caveat from Phase 6, not a gap to fix.
|
|
- **Graceful (`docker stop`, SIGTERM → CoordinatedShutdown → cluster Leave) vs. ungraceful (`docker kill`,
|
|
SIGKILL → peer auto-downs)** are the two failover paths. Manual failover = graceful Leave; a crashed
|
|
node = auto-down.
|
|
|
|
## Drills
|
|
|
|
| # | Drill | Status | Evidence |
|
|
|---|---|---|---|
|
|
| D1 | MAIN graceful failover (manual-failover path) — Primary leaves, peer takes over | ✅ PASS | AdminUI "Trigger failover" on `/clusters/MAIN/redundancy`: central-2 "Exiting completed" → central-1 became Primary (250) → central-2 restarted, "Welcome from central-1" (rejoined, no split). Roles swapped 250↔240 |
|
|
| D2 | SITE-A auto-down failover, BOTH directions (kill Primary, then kill new Primary) | ✅ PASS | leg 8 killed site-a-1→site-a-2 Primary; D2 killed site-a-2→site-a-1 Primary (250), survived alone |
|
|
| D3 | SITE-B auto-down failover (crash-the-oldest) | ✅ PASS | killed site-b-1 (Primary/oldest) → site-b-2 "Member removed [site-b-1]", BecomingOldest→Oldest, 250 |
|
|
| D4 | Gate (a): auto-down 1-vs-1 crash-the-oldest — survivor stays Up, does not self-down | ✅ PASS | every survivor across D2/D3/D5/leg8 stayed Up (e.g. site-a-1 "Up 49 minutes", site-b-2 "Up 56 minutes") — no self-down. Closes the gate deferred since Phase 0a |
|
|
| D5 | Gate (b): self-first cold-start-alone — a node boots ALONE (partner down) and forms its mesh | ✅ PASS | stopped both site-b, started only site-b-1: "JOINING itself … forming a new cluster" → leader → Up → serves 250 as sole Primary |
|
|
| D6 | Recovery — a downed/restarted node rejoins its pair as Secondary (240) | ✅ PASS | central-2, site-a-2, site-b-2 all restarted → "Welcome from <founder>" → rejoined 240 |
|
|
|
|
**Phase 7 exit gate: MET.** All 6 drills pass on every pair type. Final fleet: three healthy 2-node
|
|
pairs, each one Primary (250) + one Secondary (240). Roles are wherever the last drill left them — each
|
|
pair has exactly one Primary, which is what matters.
|
|
|
|
## Findings / notes
|
|
|
|
- **Manual-failover button is MAIN-only, by construction** (acts on the local `Cluster.State`; AdminUI
|
|
runs only on central). The UI now correctly states "this application Cluster's own Primary … each
|
|
cluster's redundant pair runs its own independent 2-node Akka mesh" (Phase 6 text, verified live).
|
|
Site pairs (driver-only, no UI) fail over via auto-down (crash) only — there is no manual-failover
|
|
surface for them, and that is correct for the topology.
|
|
- **Graceful failover restarts the node in-place and it rejoins** (no split), because a single peer that
|
|
is already Up answers the restarting node's InitJoin. The **simultaneous cold-start** split (Phase 6
|
|
finding) only bites when BOTH pair nodes start from nothing at once — see the runbook mitigation.
|
|
- **Carried Phase-6 finding — simultaneous cold-start of both pair VMs — NOW CLOSED by a product guard
|
|
(`279d1d0f`).** On the real co-located VMs a site's two VMs can power-cycle together; two self-first
|
|
seeds can then each form a 1-node cluster (split brain, two Primaries in one pair). Two mitigations now
|
|
exist: (1) **operational** — stagger the two VMs' service-manager start / start the founder first (the
|
|
docker-dev `depends_on: service_healthy` serialization, still used by site-b); (2) **product** — the
|
|
opt-in `Cluster:BootstrapGuard` (default off), which makes the lower-address node the founder and the
|
|
higher node probe-then-join, arbitrating the race inside the join decision. Live-gated on the site-a
|
|
pair (guard on, serialization removed): simultaneous start → 250/240, no split; higher-node cold-start
|
|
-alone → self-forms → 250. See `docs/Redundancy.md` §"Bootstrap guard". The guard does NOT need the
|
|
compose `depends_on` that production hardware lacks — it is the production-faithful fix.
|
|
|
|
---
|
|
|
|
## Operator runbook — co-located OtOpcUa + ScadaBridge site failover (one page)
|
|
|
|
**Topology.** Each site = 2 Windows VMs. Each VM runs **one OtOpcUa driver node** (`driver,cluster-SITE-X`,
|
|
Akka `:4053`, OPC UA `:4840`) **and one ScadaBridge site node** — two independent 2-node Akka clusters
|
|
sharing the hardware, never a shared mesh. Central = 2 VMs, each an OtOpcUa `admin,driver,cluster-MAIN`
|
|
node (hosts the AdminUI) + a ScadaBridge central node.
|
|
|
|
**Normal state.** Each pair: one node Primary (OPC UA ServiceLevel **250**), one Secondary (**240**).
|
|
Check from any OPC UA client: `otopcua-cli redundancy -u opc.tcp://<node>:4840`.
|
|
|
|
**Planned failover (move the Primary off a VM — e.g. for patching):**
|
|
- *Central (MAIN) pair:* AdminUI → **Clusters → MAIN → Redundancy → Trigger failover → Confirm**. The
|
|
Primary gracefully leaves, restarts, rejoins as Secondary; its peer becomes Primary (250) in a few
|
|
seconds. OPC UA clients re-select the new Primary automatically.
|
|
- *Site pairs (SITE-A/SITE-B):* no UI (driver-only). Stop the OtOpcUa service on the Primary VM
|
|
(graceful SIGTERM) — the peer auto-downs it and becomes Primary. Restart the service to rejoin.
|
|
|
|
**Unplanned failover (a VM dies).** The peer detects the loss (~10-15 s: failure detector + auto-down),
|
|
removes the dead node, takes the redundancy singleton, and advertises 250. **A 1-node pair keeps
|
|
serving** (auto-down 1-vs-1 survival — verified). Because the VM is shared, **both products fail over
|
|
together**: confirm the ScadaBridge site node on the surviving VM also took over (its own runbook).
|
|
|
|
**Recovery.** Bring the dead VM back. The OtOpcUa node cold-starts, finds its peer Up, and rejoins as
|
|
Secondary (240) — "Welcome from <peer>" in its log. No manual step.
|
|
|
|
**⚠️ Both VMs down at once (site power event).** Start the VMs **staggered**, or bring the **designated
|
|
founder VM up first** and wait for its OtOpcUa node to reach ServiceLevel 250 before starting the second
|
|
VM. Starting both OtOpcUa services simultaneously can split the pair into two 1-node clusters (two
|
|
Primaries). If that happens: stop the OtOpcUa service on the non-founder VM, confirm the founder is the
|
|
lone Primary (250), then restart the non-founder — it rejoins as Secondary.
|
|
|
|
**Health checks.**
|
|
- ServiceLevel per node: `otopcua-cli redundancy -u opc.tcp://<node>:4840` (250 Primary / 240 Secondary).
|
|
- MAIN membership + Primary: AdminUI → Clusters → MAIN → Redundancy.
|
|
- A pair showing **250/250** = split brain — apply the "both VMs down" recovery above.
|
|
- A pair showing **240/240 or a lone 240** = no Primary elected — check the singleton host is Up.
|