docs(mesh): Phase 6 — mark done, remove single-mesh caveats, document pair-local meshes
Redundancy.md: KNOWN-LIMITATION (per-Akka-cluster election) marked RESOLVED;
new 'Per-cluster meshes (Phase 6)' section + pair-local secrets note; manual-
failover mesh-scope caveat removed; auto-down 1-vs-1 note updated (every mesh is
now two nodes, drill deferred to Phase 7). IManualFailoverService + ClusterRedundancy
razor caveats updated to pair-local reality. Configuration.md documents the
cluster-{ClusterId} role, per-pair self-first seeds, SplitTopologyTransportValidator,
and the intentional Dps default. Program + design status tables + CLAUDE.md marked
Phase 6 DONE.
(Task 8 subagent completed the edits but died on an API error before committing;
reviewed and committed by the controller.)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
@@ -263,25 +263,45 @@ kill-and-reconnect of a site node's dialer recovers every stream (driver-health/
|
||||
from the node hub's snapshot replay). Treat Phase 5 as **code-complete, not verified**, until this
|
||||
lands — see `docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md` Task 12.
|
||||
|
||||
### Phase 6 — Mesh partition + co-location topology
|
||||
### Phase 6 — Mesh partition + co-location topology — **DONE 2026-07-24**
|
||||
**Scope:** per-cluster seed nodes — ~~adopt ScadaBridge's self-first ordering and RETIRE the
|
||||
`SelfFormAfter` watchdog + TCP reachability guard~~ **DONE EARLY 2026-07-22** (docker-dev
|
||||
`central-2` swapped to self-first, `ClusterBootstrapFallback`/`SelfFormAfter` deleted,
|
||||
`AkkaClusterOptionsValidator` ports ScadaBridge's startup rule in its conditional form, and
|
||||
`SelfFirstSeedBootstrapTests` replaces `SelfFormBootstrapTests`; see `docs/Redundancy.md`
|
||||
§"Bootstrap: self-first seed ordering"). What remains for this phase is the per-pair
|
||||
`Cluster__SeedNodes__*` matrix once the meshes actually split — every node in a pair is then a seed
|
||||
of its own mesh, so the site-node exemption disappears; cluster-scoped roles `cluster-{ClusterId}` + singleton re-scoping,
|
||||
central pair keeps the admin singletons; **docker-dev rig rewritten** to model the real topology —
|
||||
including the co-location port table above (both products' compose files on shared per-site
|
||||
networks, real LocalDb sync ports); remove the ClusterRedundancy page's mesh-scope caveat (the
|
||||
election is pair-local now) and the fallback's site-node island-guard docs note (moot — every
|
||||
node is a seed of its own mesh); `Cluster__SeedNodes__*` env matrix per pair.
|
||||
**Exit gate:** rig up in the new shape; every existing live-gated behavior re-verified per pair
|
||||
(deploy, redundancy 250/240 per pair — **two Primaries fleet-wide, one per pair, by design**);
|
||||
secrets Akka replication re-verified or re-scoped (it rides DPS on the current single mesh — its
|
||||
topology must be re-decided here, likely SQL-hub mode like ScadaBridge, since pub/sub cannot
|
||||
cross separate meshes).
|
||||
§"Bootstrap: self-first seed ordering"). The rest of the phase — actually splitting the single
|
||||
fleet-wide mesh into three independent 2-node meshes (central `cluster-MAIN`, site-a
|
||||
`cluster-SITE-A`, site-b `cluster-SITE-B`) — **shipped 2026-07-24**. See
|
||||
`docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md` for the full plan and task list.
|
||||
Delivered:
|
||||
- Per-pair `Cluster:SeedNodes` (each node lists itself first, its pair partner second — no
|
||||
cross-pair seeds) + `cluster-{ClusterId}` roles on every driver node + singleton re-scoping:
|
||||
`RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton spawned on every driver
|
||||
node (falls back to plain `driver` for legacy nodes), so **each pair elects its own Primary —
|
||||
two or more Primaries fleet-wide, by design**; `ClusterNodeAddressReconciler` is own-cluster-scoped.
|
||||
- `CentralCommunicationActor` now creates one `ClusterClient` per application `Cluster` and fans
|
||||
commands across them (closes the `TODO(Phase 6)` left by Phase 2).
|
||||
- New `SplitTopologyTransportValidator` fails host start on a `cluster-{ClusterId}`-role node left
|
||||
on a DPS transport (`MeshTransport:Mode` must be `ClusterClient`; `Telemetry:Mode`/`TelemetryDial:Mode`
|
||||
must be `Grpc`) — see `docs/Configuration.md` §"Per-cluster mesh split (Phase 6)".
|
||||
- **Three decisions worth recording:**
|
||||
1. **Secrets replication went pair-local, not SQL-hub.** The design flagged "likely SQL-hub mode
|
||||
like ScadaBridge" as the fallback if DPS couldn't cross the split meshes — in the event, each
|
||||
pair simply keeps replicating its own secrets within itself (no cross-cluster sync needed, and
|
||||
no SQL-hub was built, consistent with Phase 4: site nodes have no SQL to hub through).
|
||||
2. **The compiled transport-mode defaults (`MeshTransport:Mode`, `Telemetry:Mode`,
|
||||
`TelemetryDial:Mode`) were deliberately left at `Dps`**, not flipped to the split-safe modes —
|
||||
a blast-radius assessment found flipping the default breaks every integration test and every
|
||||
appsettings profile that leaves the key unset. The guardrail is
|
||||
`SplitTopologyTransportValidator` plus explicit per-deployment config, not a default flip.
|
||||
3. **The docker-dev rig models three OtOpcUa-only meshes** (central/site-a/site-b), not the
|
||||
ScadaBridge co-location topology sketched in the design's port table — co-location with
|
||||
ScadaBridge's own site nodes stayed out of scope for this phase.
|
||||
**Exit gate:** rig up in the new three-mesh shape; every existing live-gated behavior re-verified
|
||||
per pair (deploy, redundancy 250/240 per pair, two-Primaries-fleet-wide confirmed); the
|
||||
ClusterRedundancy page's mesh-scope caveat and `IManualFailoverService`'s doc-comment caveat
|
||||
removed (the election is pair-local now); `docs/Redundancy.md`'s per-Akka-cluster KNOWN LIMITATION
|
||||
marked resolved.
|
||||
|
||||
### Phase 7 — Failover drill + live gates
|
||||
**Scope:** the drill ScadaBridge already has (`failover-drill.sh` analogue) run per pair, both
|
||||
@@ -318,5 +338,5 @@ resource sizing on the site VMs should be checked once both products run the ful
|
||||
| 3 fetch-and-cache | **DONE 2026-07-23**, merged `d01b0695`, live gate PASSED, pushed to origin (scadaproj umbrella index updated + pushed `b5e7bc8`). gRPC fetch RPC + shared node key; dark switch `ConfigSource:Mode` (Direct default). Rig flip: `OTOPCUA_CONFIG_MODE=FetchAndCache` on the site nodes. See `2026-07-22-mesh-phase3-config-fetch-and-cache.md`. |
|
||||
| 4 cut driver ConfigDb | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase4` (Tasks 0–8 + 1b + 10–11; Task 9 table-drop deferred). ConfigDb admin-only; driver-only ⇒ FetchAndCache (validator); `DbHealthProbeActor` not spawned driver-only (client-visible ServiceLevel 240/250 with central SQL down — proven live); alarm condition state in replicated LocalDb `alarm_condition_state`; central persists acks; `OpcUaPublish` guard split fixes the driver-only address-space wipe; driver-only LDAP maps from appsettings only (6th consumer found mid-phase). Gate: deploy sealed green w/ 4 DB-less site nodes acking, ServiceLevel held 240 w/ SQL down, restart booted last-known-good from the LocalDb pointer. See `2026-07-23-mesh-phase4-cut-driver-configdb.md` + `2026-07-23-mesh-phase4-live-gate.md`. |
|
||||
| 5 gRPC telemetry | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase5`. 4-channel scope (`alerts`/`script-logs`/`driver-health`/`driver-resilience-status`), 3 deferred with rationale (`redundancy-state`, `fleet-status`, `deployment-acks`); dark switch `Telemetry:Mode`/`TelemetryDial:Mode` (Dps default); node hosts server / central dials; auth-from-day-one fail-closed bearer key, superseding design §6.3. Gate: full 12-stream mesh formed in Grpc mode (2 inbound per driver node, 6 outbound per central), AdminUI pill live (data path proven), kill-and-reconnect of a node recovered its stream in ~5s; Dps baseline dials nothing. Surfaced an upgrade gotcha — `ClusterNode.GrpcPort` must be populated on existing deployments (fresh installs seed it; nullable column doesn't backfill) — which live-validated the graceful null-`GrpcPort` skip. See `docs/Telemetry.md`, `2026-07-23-mesh-phase5-grpc-telemetry-stream.md` + `2026-07-23-mesh-phase5-live-gate.md`. |
|
||||
| 6 mesh partition + co-location | not started |
|
||||
| 6 mesh partition + co-location | **DONE 2026-07-24** — three independent 2-node meshes (central/site-a/site-b), `cluster-{ClusterId}` roles, per-pair self-first seeds, cluster-scoped `RedundancyStateActor` singleton (two+ Primaries fleet-wide, by design), one `ClusterClient` per `Cluster`, `SplitTopologyTransportValidator`, pair-local secrets replication, compiled transport defaults intentionally kept `Dps`. See `2026-07-24-mesh-phase6-partition-and-colocation.md`. |
|
||||
| 7 drill + live gates | not started |
|
||||
|
||||
Reference in New Issue
Block a user