diff --git a/docs/plans/2026-07-22-per-cluster-mesh-program.md b/docs/plans/2026-07-22-per-cluster-mesh-program.md index 5bca2893..ab442a43 100644 --- a/docs/plans/2026-07-22-per-cluster-mesh-program.md +++ b/docs/plans/2026-07-22-per-cluster-mesh-program.md @@ -339,4 +339,4 @@ resource sizing on the site VMs should be checked once both products run the ful | 4 cut driver ConfigDb | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase4` (Tasks 0–8 + 1b + 10–11; Task 9 table-drop deferred). ConfigDb admin-only; driver-only ⇒ FetchAndCache (validator); `DbHealthProbeActor` not spawned driver-only (client-visible ServiceLevel 240/250 with central SQL down — proven live); alarm condition state in replicated LocalDb `alarm_condition_state`; central persists acks; `OpcUaPublish` guard split fixes the driver-only address-space wipe; driver-only LDAP maps from appsettings only (6th consumer found mid-phase). Gate: deploy sealed green w/ 4 DB-less site nodes acking, ServiceLevel held 240 w/ SQL down, restart booted last-known-good from the LocalDb pointer. See `2026-07-23-mesh-phase4-cut-driver-configdb.md` + `2026-07-23-mesh-phase4-live-gate.md`. | | 5 gRPC telemetry | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase5`. 4-channel scope (`alerts`/`script-logs`/`driver-health`/`driver-resilience-status`), 3 deferred with rationale (`redundancy-state`, `fleet-status`, `deployment-acks`); dark switch `Telemetry:Mode`/`TelemetryDial:Mode` (Dps default); node hosts server / central dials; auth-from-day-one fail-closed bearer key, superseding design §6.3. Gate: full 12-stream mesh formed in Grpc mode (2 inbound per driver node, 6 outbound per central), AdminUI pill live (data path proven), kill-and-reconnect of a node recovered its stream in ~5s; Dps baseline dials nothing. Surfaced an upgrade gotcha — `ClusterNode.GrpcPort` must be populated on existing deployments (fresh installs seed it; nullable column doesn't backfill) — which live-validated the graceful null-`GrpcPort` skip. See `docs/Telemetry.md`, `2026-07-23-mesh-phase5-grpc-telemetry-stream.md` + `2026-07-23-mesh-phase5-live-gate.md`. | | 6 mesh partition + co-location | **DONE 2026-07-24** — three independent 2-node meshes (central/site-a/site-b), `cluster-{ClusterId}` roles, per-pair self-first seeds, cluster-scoped `RedundancyStateActor` singleton (two+ Primaries fleet-wide, by design), one `ClusterClient` per `Cluster`, `SplitTopologyTransportValidator`, pair-local secrets replication, compiled transport defaults intentionally kept `Dps`. See `2026-07-24-mesh-phase6-partition-and-colocation.md`. | -| 7 drill + live gates | not started | +| 7 drill + live gates | **DONE 2026-07-24 — all 6 drills PASSED** on the docker-dev three-mesh rig. D1 MAIN graceful manual-failover via the AdminUI button (Primary leaves, peer→250, restarts+rejoins no split); D2 SITE-A auto-down failover both directions; D3 SITE-B crash-the-oldest; D4 **auto-down 1-vs-1 survival (closes the gate deferred since Phase 0a)** — every survivor stayed Up; D5 **self-first cold-start-alone (gate b)** — a node boots alone and forms its mesh; D6 recovery/rejoin. Manual-failover button confirmed MAIN-only by construction (site pairs are driver-only, fail over via auto-down). One-page co-located operator runbook shipped. Carried Phase-6 finding: simultaneous cold-start of both pair VMs can split — operational mitigation (staggered start) in the runbook; product-level guard is a candidate follow-up, not shipped. See `2026-07-24-mesh-phase7-failover-drills.md`. **Per-cluster mesh program COMPLETE.** | diff --git a/docs/plans/2026-07-24-mesh-phase7-failover-drills.md b/docs/plans/2026-07-24-mesh-phase7-failover-drills.md new file mode 100644 index 00000000..0f329862 --- /dev/null +++ b/docs/plans/2026-07-24-mesh-phase7-failover-drills.md @@ -0,0 +1,91 @@ +# Per-cluster mesh Phase 7 — failover drills + closing live gates + +> **Final phase of the per-cluster mesh program.** Run the failover drill per pair type, close the two +> outstanding live gates (auto-down 1-vs-1 crash-the-oldest; self-first cold-start-alone), re-verify the +> manual-failover button, and ship a one-page co-located operator runbook. +> +> Runs against merged master on the docker-dev three-mesh rig (same rig as the Phase 6 gate). + +## Scope clarified by recon + +- **Failover mechanism is the same for every pair**: the oldest Up `driver` member leaves/dies, the peer + becomes oldest, takes the `cluster-{ClusterId}` redundancy singleton, and advertises ServiceLevel 250. +- **Manual-failover button (`IManualFailoverService`) is MAIN-scoped by construction** — it acts on the + local node's `Cluster.State` (graceful `Leave` of the oldest Up driver), and the AdminUI runs only on + the central (MAIN) pair. Site pairs are **driver-only, no UI**; they fail over via **auto-down** (kill) + only. This is the documented pair-local caveat from Phase 6, not a gap to fix. +- **Graceful (`docker stop`, SIGTERM → CoordinatedShutdown → cluster Leave) vs. ungraceful (`docker kill`, + SIGKILL → peer auto-downs)** are the two failover paths. Manual failover = graceful Leave; a crashed + node = auto-down. + +## Drills + +| # | Drill | Status | Evidence | +|---|---|---|---| +| D1 | MAIN graceful failover (manual-failover path) — Primary leaves, peer takes over | ✅ PASS | AdminUI "Trigger failover" on `/clusters/MAIN/redundancy`: central-2 "Exiting completed" → central-1 became Primary (250) → central-2 restarted, "Welcome from central-1" (rejoined, no split). Roles swapped 250↔240 | +| D2 | SITE-A auto-down failover, BOTH directions (kill Primary, then kill new Primary) | ✅ PASS | leg 8 killed site-a-1→site-a-2 Primary; D2 killed site-a-2→site-a-1 Primary (250), survived alone | +| D3 | SITE-B auto-down failover (crash-the-oldest) | ✅ PASS | killed site-b-1 (Primary/oldest) → site-b-2 "Member removed [site-b-1]", BecomingOldest→Oldest, 250 | +| D4 | Gate (a): auto-down 1-vs-1 crash-the-oldest — survivor stays Up, does not self-down | ✅ PASS | every survivor across D2/D3/D5/leg8 stayed Up (e.g. site-a-1 "Up 49 minutes", site-b-2 "Up 56 minutes") — no self-down. Closes the gate deferred since Phase 0a | +| D5 | Gate (b): self-first cold-start-alone — a node boots ALONE (partner down) and forms its mesh | ✅ PASS | stopped both site-b, started only site-b-1: "JOINING itself … forming a new cluster" → leader → Up → serves 250 as sole Primary | +| D6 | Recovery — a downed/restarted node rejoins its pair as Secondary (240) | ✅ PASS | central-2, site-a-2, site-b-2 all restarted → "Welcome from " → rejoined 240 | + +**Phase 7 exit gate: MET.** All 6 drills pass on every pair type. Final fleet: three healthy 2-node +pairs, each one Primary (250) + one Secondary (240). Roles are wherever the last drill left them — each +pair has exactly one Primary, which is what matters. + +## Findings / notes + +- **Manual-failover button is MAIN-only, by construction** (acts on the local `Cluster.State`; AdminUI + runs only on central). The UI now correctly states "this application Cluster's own Primary … each + cluster's redundant pair runs its own independent 2-node Akka mesh" (Phase 6 text, verified live). + Site pairs (driver-only, no UI) fail over via auto-down (crash) only — there is no manual-failover + surface for them, and that is correct for the topology. +- **Graceful failover restarts the node in-place and it rejoins** (no split), because a single peer that + is already Up answers the restarting node's InitJoin. The **simultaneous cold-start** split (Phase 6 + finding) only bites when BOTH pair nodes start from nothing at once — see the runbook mitigation. +- **Carried Phase-6 finding — simultaneous cold-start of both pair VMs.** On the real co-located VMs a + site's two VMs can power-cycle together; two self-first seeds can then each form a 1-node cluster + (split brain, two Primaries in one pair). The docker-dev rig serializes with a founder healthcheck + + `depends_on: service_healthy`. **Production mitigation (operational, not code):** stagger the two VMs' + service-manager start, or start the designated founder VM first. A product-level guard (e.g. a brief + reachability-gated join delay on the non-founder) is a candidate follow-up if operational staggering + proves fragile — tracked as a Phase 7 note, not shipped. + +--- + +## Operator runbook — co-located OtOpcUa + ScadaBridge site failover (one page) + +**Topology.** Each site = 2 Windows VMs. Each VM runs **one OtOpcUa driver node** (`driver,cluster-SITE-X`, +Akka `:4053`, OPC UA `:4840`) **and one ScadaBridge site node** — two independent 2-node Akka clusters +sharing the hardware, never a shared mesh. Central = 2 VMs, each an OtOpcUa `admin,driver,cluster-MAIN` +node (hosts the AdminUI) + a ScadaBridge central node. + +**Normal state.** Each pair: one node Primary (OPC UA ServiceLevel **250**), one Secondary (**240**). +Check from any OPC UA client: `otopcua-cli redundancy -u opc.tcp://:4840`. + +**Planned failover (move the Primary off a VM — e.g. for patching):** +- *Central (MAIN) pair:* AdminUI → **Clusters → MAIN → Redundancy → Trigger failover → Confirm**. The + Primary gracefully leaves, restarts, rejoins as Secondary; its peer becomes Primary (250) in a few + seconds. OPC UA clients re-select the new Primary automatically. +- *Site pairs (SITE-A/SITE-B):* no UI (driver-only). Stop the OtOpcUa service on the Primary VM + (graceful SIGTERM) — the peer auto-downs it and becomes Primary. Restart the service to rejoin. + +**Unplanned failover (a VM dies).** The peer detects the loss (~10-15 s: failure detector + auto-down), +removes the dead node, takes the redundancy singleton, and advertises 250. **A 1-node pair keeps +serving** (auto-down 1-vs-1 survival — verified). Because the VM is shared, **both products fail over +together**: confirm the ScadaBridge site node on the surviving VM also took over (its own runbook). + +**Recovery.** Bring the dead VM back. The OtOpcUa node cold-starts, finds its peer Up, and rejoins as +Secondary (240) — "Welcome from " in its log. No manual step. + +**⚠️ Both VMs down at once (site power event).** Start the VMs **staggered**, or bring the **designated +founder VM up first** and wait for its OtOpcUa node to reach ServiceLevel 250 before starting the second +VM. Starting both OtOpcUa services simultaneously can split the pair into two 1-node clusters (two +Primaries). If that happens: stop the OtOpcUa service on the non-founder VM, confirm the founder is the +lone Primary (250), then restart the non-founder — it rejoins as Secondary. + +**Health checks.** +- ServiceLevel per node: `otopcua-cli redundancy -u opc.tcp://:4840` (250 Primary / 240 Secondary). +- MAIN membership + Primary: AdminUI → Clusters → MAIN → Redundancy. +- A pair showing **250/250** = split brain — apply the "both VMs down" recovery above. +- A pair showing **240/240 or a lone 240** = no Primary elected — check the singleton host is Up.