docs(mesh): Phase 6 — mark done, remove single-mesh caveats, document pair-local meshes

Redundancy.md: KNOWN-LIMITATION (per-Akka-cluster election) marked RESOLVED;
new 'Per-cluster meshes (Phase 6)' section + pair-local secrets note; manual-
failover mesh-scope caveat removed; auto-down 1-vs-1 note updated (every mesh is
now two nodes, drill deferred to Phase 7). IManualFailoverService + ClusterRedundancy
razor caveats updated to pair-local reality. Configuration.md documents the
cluster-{ClusterId} role, per-pair self-first seeds, SplitTopologyTransportValidator,
and the intentional Dps default. Program + design status tables + CLAUDE.md marked
Phase 6 DONE.

(Task 8 subagent completed the edits but died on an API error before committing;
reviewed and committed by the controller.)

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-24 03:02:35 -04:00
parent 2a057a466f
commit 8a173bbaf6
7 changed files with 153 additions and 58 deletions
@@ -263,25 +263,45 @@ kill-and-reconnect of a site node's dialer recovers every stream (driver-health/
from the node hub's snapshot replay). Treat Phase 5 as **code-complete, not verified**, until this
lands — see `docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md` Task 12.
### Phase 6 — Mesh partition + co-location topology
### Phase 6 — Mesh partition + co-location topology — **DONE 2026-07-24**
**Scope:** per-cluster seed nodes — ~~adopt ScadaBridge's self-first ordering and RETIRE the
`SelfFormAfter` watchdog + TCP reachability guard~~ **DONE EARLY 2026-07-22** (docker-dev
`central-2` swapped to self-first, `ClusterBootstrapFallback`/`SelfFormAfter` deleted,
`AkkaClusterOptionsValidator` ports ScadaBridge's startup rule in its conditional form, and
`SelfFirstSeedBootstrapTests` replaces `SelfFormBootstrapTests`; see `docs/Redundancy.md`
§"Bootstrap: self-first seed ordering"). What remains for this phase is the per-pair
`Cluster__SeedNodes__*` matrix once the meshes actually split — every node in a pair is then a seed
of its own mesh, so the site-node exemption disappears; cluster-scoped roles `cluster-{ClusterId}` + singleton re-scoping,
central pair keeps the admin singletons; **docker-dev rig rewritten** to model the real topology —
including the co-location port table above (both products' compose files on shared per-site
networks, real LocalDb sync ports); remove the ClusterRedundancy page's mesh-scope caveat (the
election is pair-local now) and the fallback's site-node island-guard docs note (moot — every
node is a seed of its own mesh); `Cluster__SeedNodes__*` env matrix per pair.
**Exit gate:** rig up in the new shape; every existing live-gated behavior re-verified per pair
(deploy, redundancy 250/240 per pair — **two Primaries fleet-wide, one per pair, by design**);
secrets Akka replication re-verified or re-scoped (it rides DPS on the current single mesh — its
topology must be re-decided here, likely SQL-hub mode like ScadaBridge, since pub/sub cannot
cross separate meshes).
§"Bootstrap: self-first seed ordering"). The rest of the phase — actually splitting the single
fleet-wide mesh into three independent 2-node meshes (central `cluster-MAIN`, site-a
`cluster-SITE-A`, site-b `cluster-SITE-B`) — **shipped 2026-07-24**. See
`docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md` for the full plan and task list.
Delivered:
- Per-pair `Cluster:SeedNodes` (each node lists itself first, its pair partner second — no
cross-pair seeds) + `cluster-{ClusterId}` roles on every driver node + singleton re-scoping:
`RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton spawned on every driver
node (falls back to plain `driver` for legacy nodes), so **each pair elects its own Primary —
two or more Primaries fleet-wide, by design**; `ClusterNodeAddressReconciler` is own-cluster-scoped.
- `CentralCommunicationActor` now creates one `ClusterClient` per application `Cluster` and fans
commands across them (closes the `TODO(Phase 6)` left by Phase 2).
- New `SplitTopologyTransportValidator` fails host start on a `cluster-{ClusterId}`-role node left
on a DPS transport (`MeshTransport:Mode` must be `ClusterClient`; `Telemetry:Mode`/`TelemetryDial:Mode`
must be `Grpc`) — see `docs/Configuration.md` §"Per-cluster mesh split (Phase 6)".
- **Three decisions worth recording:**
1. **Secrets replication went pair-local, not SQL-hub.** The design flagged "likely SQL-hub mode
like ScadaBridge" as the fallback if DPS couldn't cross the split meshes — in the event, each
pair simply keeps replicating its own secrets within itself (no cross-cluster sync needed, and
no SQL-hub was built, consistent with Phase 4: site nodes have no SQL to hub through).
2. **The compiled transport-mode defaults (`MeshTransport:Mode`, `Telemetry:Mode`,
`TelemetryDial:Mode`) were deliberately left at `Dps`**, not flipped to the split-safe modes —
a blast-radius assessment found flipping the default breaks every integration test and every
appsettings profile that leaves the key unset. The guardrail is
`SplitTopologyTransportValidator` plus explicit per-deployment config, not a default flip.
3. **The docker-dev rig models three OtOpcUa-only meshes** (central/site-a/site-b), not the
ScadaBridge co-location topology sketched in the design's port table — co-location with
ScadaBridge's own site nodes stayed out of scope for this phase.
**Exit gate:** rig up in the new three-mesh shape; every existing live-gated behavior re-verified
per pair (deploy, redundancy 250/240 per pair, two-Primaries-fleet-wide confirmed); the
ClusterRedundancy page's mesh-scope caveat and `IManualFailoverService`'s doc-comment caveat
removed (the election is pair-local now); `docs/Redundancy.md`'s per-Akka-cluster KNOWN LIMITATION
marked resolved.
### Phase 7 — Failover drill + live gates
**Scope:** the drill ScadaBridge already has (`failover-drill.sh` analogue) run per pair, both
@@ -318,5 +338,5 @@ resource sizing on the site VMs should be checked once both products run the ful
| 3 fetch-and-cache | **DONE 2026-07-23**, merged `d01b0695`, live gate PASSED, pushed to origin (scadaproj umbrella index updated + pushed `b5e7bc8`). gRPC fetch RPC + shared node key; dark switch `ConfigSource:Mode` (Direct default). Rig flip: `OTOPCUA_CONFIG_MODE=FetchAndCache` on the site nodes. See `2026-07-22-mesh-phase3-config-fetch-and-cache.md`. |
| 4 cut driver ConfigDb | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase4` (Tasks 08 + 1b + 1011; Task 9 table-drop deferred). ConfigDb admin-only; driver-only ⇒ FetchAndCache (validator); `DbHealthProbeActor` not spawned driver-only (client-visible ServiceLevel 240/250 with central SQL down — proven live); alarm condition state in replicated LocalDb `alarm_condition_state`; central persists acks; `OpcUaPublish` guard split fixes the driver-only address-space wipe; driver-only LDAP maps from appsettings only (6th consumer found mid-phase). Gate: deploy sealed green w/ 4 DB-less site nodes acking, ServiceLevel held 240 w/ SQL down, restart booted last-known-good from the LocalDb pointer. See `2026-07-23-mesh-phase4-cut-driver-configdb.md` + `2026-07-23-mesh-phase4-live-gate.md`. |
| 5 gRPC telemetry | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase5`. 4-channel scope (`alerts`/`script-logs`/`driver-health`/`driver-resilience-status`), 3 deferred with rationale (`redundancy-state`, `fleet-status`, `deployment-acks`); dark switch `Telemetry:Mode`/`TelemetryDial:Mode` (Dps default); node hosts server / central dials; auth-from-day-one fail-closed bearer key, superseding design §6.3. Gate: full 12-stream mesh formed in Grpc mode (2 inbound per driver node, 6 outbound per central), AdminUI pill live (data path proven), kill-and-reconnect of a node recovered its stream in ~5s; Dps baseline dials nothing. Surfaced an upgrade gotcha — `ClusterNode.GrpcPort` must be populated on existing deployments (fresh installs seed it; nullable column doesn't backfill) — which live-validated the graceful null-`GrpcPort` skip. See `docs/Telemetry.md`, `2026-07-23-mesh-phase5-grpc-telemetry-stream.md` + `2026-07-23-mesh-phase5-live-gate.md`. |
| 6 mesh partition + co-location | not started |
| 6 mesh partition + co-location | **DONE 2026-07-24** — three independent 2-node meshes (central/site-a/site-b), `cluster-{ClusterId}` roles, per-pair self-first seeds, cluster-scoped `RedundancyStateActor` singleton (two+ Primaries fleet-wide, by design), one `ClusterClient` per `Cluster`, `SplitTopologyTransportValidator`, pair-local secrets replication, compiled transport defaults intentionally kept `Dps`. See `2026-07-24-mesh-phase6-partition-and-colocation.md`. |
| 7 drill + live gates | not started |