docs(mesh): Phase 7 DONE — failover drills PASSED, program complete
All 6 failover drills pass on the docker-dev three-mesh rig: - D1 MAIN graceful manual-failover via the AdminUI "Trigger failover" button — Primary leaves, peer→250, restarts and rejoins with no split. - D2 SITE-A auto-down failover, both directions; D3 SITE-B crash-the-oldest. - D4 auto-down 1-vs-1 survival — every survivor stayed Up (closes the gate deferred since Phase 0a). - D5 self-first cold-start-alone — a node boots with its partner down and forms its own mesh (gate b). - D6 recovery — restarted nodes rejoin as Secondary (240). Ships a one-page co-located operator runbook (OtOpcUa + ScadaBridge on shared site VMs). Manual-failover button confirmed MAIN-only by construction; site pairs (driver-only, no UI) fail over via auto-down. Carried Phase-6 finding — simultaneous cold-start of both pair VMs can split — mitigated operationally in the runbook (staggered start); a product-level join guard is a candidate follow-up, not shipped. The per-cluster mesh program (Phases 0a–7) is COMPLETE. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
@@ -339,4 +339,4 @@ resource sizing on the site VMs should be checked once both products run the ful
|
||||
| 4 cut driver ConfigDb | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase4` (Tasks 0–8 + 1b + 10–11; Task 9 table-drop deferred). ConfigDb admin-only; driver-only ⇒ FetchAndCache (validator); `DbHealthProbeActor` not spawned driver-only (client-visible ServiceLevel 240/250 with central SQL down — proven live); alarm condition state in replicated LocalDb `alarm_condition_state`; central persists acks; `OpcUaPublish` guard split fixes the driver-only address-space wipe; driver-only LDAP maps from appsettings only (6th consumer found mid-phase). Gate: deploy sealed green w/ 4 DB-less site nodes acking, ServiceLevel held 240 w/ SQL down, restart booted last-known-good from the LocalDb pointer. See `2026-07-23-mesh-phase4-cut-driver-configdb.md` + `2026-07-23-mesh-phase4-live-gate.md`. |
|
||||
| 5 gRPC telemetry | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase5`. 4-channel scope (`alerts`/`script-logs`/`driver-health`/`driver-resilience-status`), 3 deferred with rationale (`redundancy-state`, `fleet-status`, `deployment-acks`); dark switch `Telemetry:Mode`/`TelemetryDial:Mode` (Dps default); node hosts server / central dials; auth-from-day-one fail-closed bearer key, superseding design §6.3. Gate: full 12-stream mesh formed in Grpc mode (2 inbound per driver node, 6 outbound per central), AdminUI pill live (data path proven), kill-and-reconnect of a node recovered its stream in ~5s; Dps baseline dials nothing. Surfaced an upgrade gotcha — `ClusterNode.GrpcPort` must be populated on existing deployments (fresh installs seed it; nullable column doesn't backfill) — which live-validated the graceful null-`GrpcPort` skip. See `docs/Telemetry.md`, `2026-07-23-mesh-phase5-grpc-telemetry-stream.md` + `2026-07-23-mesh-phase5-live-gate.md`. |
|
||||
| 6 mesh partition + co-location | **DONE 2026-07-24** — three independent 2-node meshes (central/site-a/site-b), `cluster-{ClusterId}` roles, per-pair self-first seeds, cluster-scoped `RedundancyStateActor` singleton (two+ Primaries fleet-wide, by design), one `ClusterClient` per `Cluster`, `SplitTopologyTransportValidator`, pair-local secrets replication, compiled transport defaults intentionally kept `Dps`. See `2026-07-24-mesh-phase6-partition-and-colocation.md`. |
|
||||
| 7 drill + live gates | not started |
|
||||
| 7 drill + live gates | **DONE 2026-07-24 — all 6 drills PASSED** on the docker-dev three-mesh rig. D1 MAIN graceful manual-failover via the AdminUI button (Primary leaves, peer→250, restarts+rejoins no split); D2 SITE-A auto-down failover both directions; D3 SITE-B crash-the-oldest; D4 **auto-down 1-vs-1 survival (closes the gate deferred since Phase 0a)** — every survivor stayed Up; D5 **self-first cold-start-alone (gate b)** — a node boots alone and forms its mesh; D6 recovery/rejoin. Manual-failover button confirmed MAIN-only by construction (site pairs are driver-only, fail over via auto-down). One-page co-located operator runbook shipped. Carried Phase-6 finding: simultaneous cold-start of both pair VMs can split — operational mitigation (staggered start) in the runbook; product-level guard is a candidate follow-up, not shipped. See `2026-07-24-mesh-phase7-failover-drills.md`. **Per-cluster mesh program COMPLETE.** |
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
# Per-cluster mesh Phase 7 — failover drills + closing live gates
|
||||
|
||||
> **Final phase of the per-cluster mesh program.** Run the failover drill per pair type, close the two
|
||||
> outstanding live gates (auto-down 1-vs-1 crash-the-oldest; self-first cold-start-alone), re-verify the
|
||||
> manual-failover button, and ship a one-page co-located operator runbook.
|
||||
>
|
||||
> Runs against merged master on the docker-dev three-mesh rig (same rig as the Phase 6 gate).
|
||||
|
||||
## Scope clarified by recon
|
||||
|
||||
- **Failover mechanism is the same for every pair**: the oldest Up `driver` member leaves/dies, the peer
|
||||
becomes oldest, takes the `cluster-{ClusterId}` redundancy singleton, and advertises ServiceLevel 250.
|
||||
- **Manual-failover button (`IManualFailoverService`) is MAIN-scoped by construction** — it acts on the
|
||||
local node's `Cluster.State` (graceful `Leave` of the oldest Up driver), and the AdminUI runs only on
|
||||
the central (MAIN) pair. Site pairs are **driver-only, no UI**; they fail over via **auto-down** (kill)
|
||||
only. This is the documented pair-local caveat from Phase 6, not a gap to fix.
|
||||
- **Graceful (`docker stop`, SIGTERM → CoordinatedShutdown → cluster Leave) vs. ungraceful (`docker kill`,
|
||||
SIGKILL → peer auto-downs)** are the two failover paths. Manual failover = graceful Leave; a crashed
|
||||
node = auto-down.
|
||||
|
||||
## Drills
|
||||
|
||||
| # | Drill | Status | Evidence |
|
||||
|---|---|---|---|
|
||||
| D1 | MAIN graceful failover (manual-failover path) — Primary leaves, peer takes over | ✅ PASS | AdminUI "Trigger failover" on `/clusters/MAIN/redundancy`: central-2 "Exiting completed" → central-1 became Primary (250) → central-2 restarted, "Welcome from central-1" (rejoined, no split). Roles swapped 250↔240 |
|
||||
| D2 | SITE-A auto-down failover, BOTH directions (kill Primary, then kill new Primary) | ✅ PASS | leg 8 killed site-a-1→site-a-2 Primary; D2 killed site-a-2→site-a-1 Primary (250), survived alone |
|
||||
| D3 | SITE-B auto-down failover (crash-the-oldest) | ✅ PASS | killed site-b-1 (Primary/oldest) → site-b-2 "Member removed [site-b-1]", BecomingOldest→Oldest, 250 |
|
||||
| D4 | Gate (a): auto-down 1-vs-1 crash-the-oldest — survivor stays Up, does not self-down | ✅ PASS | every survivor across D2/D3/D5/leg8 stayed Up (e.g. site-a-1 "Up 49 minutes", site-b-2 "Up 56 minutes") — no self-down. Closes the gate deferred since Phase 0a |
|
||||
| D5 | Gate (b): self-first cold-start-alone — a node boots ALONE (partner down) and forms its mesh | ✅ PASS | stopped both site-b, started only site-b-1: "JOINING itself … forming a new cluster" → leader → Up → serves 250 as sole Primary |
|
||||
| D6 | Recovery — a downed/restarted node rejoins its pair as Secondary (240) | ✅ PASS | central-2, site-a-2, site-b-2 all restarted → "Welcome from <founder>" → rejoined 240 |
|
||||
|
||||
**Phase 7 exit gate: MET.** All 6 drills pass on every pair type. Final fleet: three healthy 2-node
|
||||
pairs, each one Primary (250) + one Secondary (240). Roles are wherever the last drill left them — each
|
||||
pair has exactly one Primary, which is what matters.
|
||||
|
||||
## Findings / notes
|
||||
|
||||
- **Manual-failover button is MAIN-only, by construction** (acts on the local `Cluster.State`; AdminUI
|
||||
runs only on central). The UI now correctly states "this application Cluster's own Primary … each
|
||||
cluster's redundant pair runs its own independent 2-node Akka mesh" (Phase 6 text, verified live).
|
||||
Site pairs (driver-only, no UI) fail over via auto-down (crash) only — there is no manual-failover
|
||||
surface for them, and that is correct for the topology.
|
||||
- **Graceful failover restarts the node in-place and it rejoins** (no split), because a single peer that
|
||||
is already Up answers the restarting node's InitJoin. The **simultaneous cold-start** split (Phase 6
|
||||
finding) only bites when BOTH pair nodes start from nothing at once — see the runbook mitigation.
|
||||
- **Carried Phase-6 finding — simultaneous cold-start of both pair VMs.** On the real co-located VMs a
|
||||
site's two VMs can power-cycle together; two self-first seeds can then each form a 1-node cluster
|
||||
(split brain, two Primaries in one pair). The docker-dev rig serializes with a founder healthcheck +
|
||||
`depends_on: service_healthy`. **Production mitigation (operational, not code):** stagger the two VMs'
|
||||
service-manager start, or start the designated founder VM first. A product-level guard (e.g. a brief
|
||||
reachability-gated join delay on the non-founder) is a candidate follow-up if operational staggering
|
||||
proves fragile — tracked as a Phase 7 note, not shipped.
|
||||
|
||||
---
|
||||
|
||||
## Operator runbook — co-located OtOpcUa + ScadaBridge site failover (one page)
|
||||
|
||||
**Topology.** Each site = 2 Windows VMs. Each VM runs **one OtOpcUa driver node** (`driver,cluster-SITE-X`,
|
||||
Akka `:4053`, OPC UA `:4840`) **and one ScadaBridge site node** — two independent 2-node Akka clusters
|
||||
sharing the hardware, never a shared mesh. Central = 2 VMs, each an OtOpcUa `admin,driver,cluster-MAIN`
|
||||
node (hosts the AdminUI) + a ScadaBridge central node.
|
||||
|
||||
**Normal state.** Each pair: one node Primary (OPC UA ServiceLevel **250**), one Secondary (**240**).
|
||||
Check from any OPC UA client: `otopcua-cli redundancy -u opc.tcp://<node>:4840`.
|
||||
|
||||
**Planned failover (move the Primary off a VM — e.g. for patching):**
|
||||
- *Central (MAIN) pair:* AdminUI → **Clusters → MAIN → Redundancy → Trigger failover → Confirm**. The
|
||||
Primary gracefully leaves, restarts, rejoins as Secondary; its peer becomes Primary (250) in a few
|
||||
seconds. OPC UA clients re-select the new Primary automatically.
|
||||
- *Site pairs (SITE-A/SITE-B):* no UI (driver-only). Stop the OtOpcUa service on the Primary VM
|
||||
(graceful SIGTERM) — the peer auto-downs it and becomes Primary. Restart the service to rejoin.
|
||||
|
||||
**Unplanned failover (a VM dies).** The peer detects the loss (~10-15 s: failure detector + auto-down),
|
||||
removes the dead node, takes the redundancy singleton, and advertises 250. **A 1-node pair keeps
|
||||
serving** (auto-down 1-vs-1 survival — verified). Because the VM is shared, **both products fail over
|
||||
together**: confirm the ScadaBridge site node on the surviving VM also took over (its own runbook).
|
||||
|
||||
**Recovery.** Bring the dead VM back. The OtOpcUa node cold-starts, finds its peer Up, and rejoins as
|
||||
Secondary (240) — "Welcome from <peer>" in its log. No manual step.
|
||||
|
||||
**⚠️ Both VMs down at once (site power event).** Start the VMs **staggered**, or bring the **designated
|
||||
founder VM up first** and wait for its OtOpcUa node to reach ServiceLevel 250 before starting the second
|
||||
VM. Starting both OtOpcUa services simultaneously can split the pair into two 1-node clusters (two
|
||||
Primaries). If that happens: stop the OtOpcUa service on the non-founder VM, confirm the founder is the
|
||||
lone Primary (250), then restart the non-founder — it rejoins as Secondary.
|
||||
|
||||
**Health checks.**
|
||||
- ServiceLevel per node: `otopcua-cli redundancy -u opc.tcp://<node>:4840` (250 Primary / 240 Secondary).
|
||||
- MAIN membership + Primary: AdminUI → Clusters → MAIN → Redundancy.
|
||||
- A pair showing **250/250** = split brain — apply the "both VMs down" recovery above.
|
||||
- A pair showing **240/240 or a lone 240** = no Primary elected — check the singleton host is Up.
|
||||
Reference in New Issue
Block a user