docs(mesh): Phase 7 DONE — failover drills PASSED, program complete
v2-ci / build (push) Successful in 3m48s
v2-ci / unit-tests (push) Failing after 14m8s

All 6 failover drills pass on the docker-dev three-mesh rig:
- D1 MAIN graceful manual-failover via the AdminUI "Trigger failover" button —
  Primary leaves, peer→250, restarts and rejoins with no split.
- D2 SITE-A auto-down failover, both directions; D3 SITE-B crash-the-oldest.
- D4 auto-down 1-vs-1 survival — every survivor stayed Up (closes the gate
  deferred since Phase 0a).
- D5 self-first cold-start-alone — a node boots with its partner down and forms
  its own mesh (gate b).
- D6 recovery — restarted nodes rejoin as Secondary (240).

Ships a one-page co-located operator runbook (OtOpcUa + ScadaBridge on shared
site VMs). Manual-failover button confirmed MAIN-only by construction; site
pairs (driver-only, no UI) fail over via auto-down. Carried Phase-6 finding —
simultaneous cold-start of both pair VMs can split — mitigated operationally in
the runbook (staggered start); a product-level join guard is a candidate
follow-up, not shipped.

The per-cluster mesh program (Phases 0a–7) is COMPLETE.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-24 08:01:26 -04:00
parent ecf5410433
commit c50ebcf7dc
2 changed files with 92 additions and 1 deletions
@@ -339,4 +339,4 @@ resource sizing on the site VMs should be checked once both products run the ful
| 4 cut driver ConfigDb | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase4` (Tasks 08 + 1b + 1011; Task 9 table-drop deferred). ConfigDb admin-only; driver-only ⇒ FetchAndCache (validator); `DbHealthProbeActor` not spawned driver-only (client-visible ServiceLevel 240/250 with central SQL down — proven live); alarm condition state in replicated LocalDb `alarm_condition_state`; central persists acks; `OpcUaPublish` guard split fixes the driver-only address-space wipe; driver-only LDAP maps from appsettings only (6th consumer found mid-phase). Gate: deploy sealed green w/ 4 DB-less site nodes acking, ServiceLevel held 240 w/ SQL down, restart booted last-known-good from the LocalDb pointer. See `2026-07-23-mesh-phase4-cut-driver-configdb.md` + `2026-07-23-mesh-phase4-live-gate.md`. |
| 5 gRPC telemetry | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase5`. 4-channel scope (`alerts`/`script-logs`/`driver-health`/`driver-resilience-status`), 3 deferred with rationale (`redundancy-state`, `fleet-status`, `deployment-acks`); dark switch `Telemetry:Mode`/`TelemetryDial:Mode` (Dps default); node hosts server / central dials; auth-from-day-one fail-closed bearer key, superseding design §6.3. Gate: full 12-stream mesh formed in Grpc mode (2 inbound per driver node, 6 outbound per central), AdminUI pill live (data path proven), kill-and-reconnect of a node recovered its stream in ~5s; Dps baseline dials nothing. Surfaced an upgrade gotcha — `ClusterNode.GrpcPort` must be populated on existing deployments (fresh installs seed it; nullable column doesn't backfill) — which live-validated the graceful null-`GrpcPort` skip. See `docs/Telemetry.md`, `2026-07-23-mesh-phase5-grpc-telemetry-stream.md` + `2026-07-23-mesh-phase5-live-gate.md`. |
| 6 mesh partition + co-location | **DONE 2026-07-24** — three independent 2-node meshes (central/site-a/site-b), `cluster-{ClusterId}` roles, per-pair self-first seeds, cluster-scoped `RedundancyStateActor` singleton (two+ Primaries fleet-wide, by design), one `ClusterClient` per `Cluster`, `SplitTopologyTransportValidator`, pair-local secrets replication, compiled transport defaults intentionally kept `Dps`. See `2026-07-24-mesh-phase6-partition-and-colocation.md`. |
| 7 drill + live gates | not started |
| 7 drill + live gates | **DONE 2026-07-24 — all 6 drills PASSED** on the docker-dev three-mesh rig. D1 MAIN graceful manual-failover via the AdminUI button (Primary leaves, peer→250, restarts+rejoins no split); D2 SITE-A auto-down failover both directions; D3 SITE-B crash-the-oldest; D4 **auto-down 1-vs-1 survival (closes the gate deferred since Phase 0a)** — every survivor stayed Up; D5 **self-first cold-start-alone (gate b)** — a node boots alone and forms its mesh; D6 recovery/rejoin. Manual-failover button confirmed MAIN-only by construction (site pairs are driver-only, fail over via auto-down). One-page co-located operator runbook shipped. Carried Phase-6 finding: simultaneous cold-start of both pair VMs can split — operational mitigation (staggered start) in the runbook; product-level guard is a candidate follow-up, not shipped. See `2026-07-24-mesh-phase7-failover-drills.md`. **Per-cluster mesh program COMPLETE.** |