diff --git a/docs/plans/2026-07-24-mesh-phase6-live-gate.md b/docs/plans/2026-07-24-mesh-phase6-live-gate.md index 7b8b85b1..299785b5 100644 --- a/docs/plans/2026-07-24-mesh-phase6-live-gate.md +++ b/docs/plans/2026-07-24-mesh-phase6-live-gate.md @@ -29,12 +29,18 @@ |---|---|---|---| | 1 | Three meshes form — each node sees ONLY its own pair (2 members) | ✅ PASS | each node handshakes ONLY its pair partner | | 2 | Two+ Primaries, one per pair; ServiceLevel 250/240 within each pair | ✅ PASS | MAIN central-2=250/central-1=240; SITE-A site-a-1=250/site-a-2=240; SITE-B site-b-1=250/site-b-2=240 | -| 3 | Deploy reaches all clusters over per-cluster ClusterClient; seals green | PENDING | | -| 4 | Telemetry per cluster over gRPC (:4056); pills live; kill/reconnect ~5s | PENDING | | -| 5 | Reconciler quiet — no `EnabledRowNotInCluster` for foreign rows | PENDING | | -| 6 | Secrets pair-local — no cross-mesh replication / no unreachable-peer errors | PENDING | | -| 7 | Split validator — a site node flipped to `MeshTransport__Mode=Dps` refuses to boot | PENDING | | -| 8 | Failover within a pair — kill a Primary, Secondary takes over (auto-down 1-vs-1) | PENDING | | +| 3 | Deploy reaches all clusters over per-cluster ClusterClient; seals green | ✅ PASS | "Deployment 019dbd99… sealed (acks=6)"; all 6 NodeDeploymentState rows acked; one ClusterClient per site cluster | +| 4 | Telemetry per cluster over gRPC (:4056); pills live; kill/reconnect ~5s | ✅ PASS | central-1 6 outbound :4056 dials, each site node 2 inbound (both admin dial in), 0 failures/30s; site-b restart failed→recovered the streams | +| 5 | Reconciler quiet — no `EnabledRowNotInCluster` for foreign rows | ✅ PASS | central-2 reconciler: "reconcile with cluster membership (2 driver members)" — own MAIN pair only; 0 foreign-row errors | +| 6 | Secrets pair-local — no cross-mesh replication / no unreachable-peer errors | ✅ PASS (by fallback criterion) | 0 secret errors fleet-wide; DPS topic rides each pair's isolated mesh (Leg 1) ⇒ structurally pair-local | +| 7 | Split validator — a site node flipped to `MeshTransport__Mode=Dps` refuses to boot | ✅ PASS | "Cluster:Roles declares a cluster-SITE-B role … MeshTransport:Mode is 'Dps' … set MeshTransport:Mode=ClusterClient" | +| 8 | Failover within a pair — kill a Primary, Secondary takes over (auto-down 1-vs-1) | ✅ PASS | killed site-a-1 (Primary); site-a-2 auto-downed it, took the singleton, 240→250, survived alone (auto-down 1-vs-1 = deferred Phase 0a gate, now proven) | + +**Phase 6 exit gate: MET.** All 8 legs pass. Three findings fixed forward on master (`3a4ed8dd` product +boot-crash, `792f28ec` rig LDAP + split-brain serialization). One test-methodology note: the Leg 7 +`compose run … site-b-2` throwaway shares the `site-b-2` network alias and briefly disrupted the SITE-B +Akka association — restart the affected pair after that probe (done), or run the validator check via a +container that does not join the compose network alias. --- diff --git a/docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md.tasks.json b/docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md.tasks.json index 2d381777..ac3df368 100644 --- a/docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md.tasks.json +++ b/docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md.tasks.json @@ -18,7 +18,7 @@ {"id": 6, "subject": "Task 6: compiled transport-mode defaults", "classification": "small", "status": "resolved-no-flip", "note": "DECIDED not to flip — blast-radius assessment: high-risk (breaks all integration tests + appsettings profiles + docker-dev site nodes), low-value (Task 5 validator already enforces split-correctness fail-closed). Documented in plan Task 6 + Configuration.md."}, {"id": 7, "subject": "Task 7: rewrite docker-dev rig into three 2-node meshes", "classification": "standard", "status": "completed", "commit": "741ab97b", "note": "3 meshes, per-pair self-first seeds, cluster roles, transports flipped (ClusterClient/Grpc), pair-local secrets note; docker compose config validates; seed SQL already correct (GrpcPort=4056)"}, {"id": 8, "subject": "Task 8: docs sweep — status tables, caveat removal, pair-local secrets note", "classification": "small", "status": "completed", "commit": "8a173bba", "note": "all 7 edits (Redundancy/Config/CLAUDE/2 plans/IManualFailoverService/ClusterRedundancy.razor); subagent died at commit step, edits reviewed+committed by controller; full-solution build clean (0 errors)"}, - {"id": 9, "subject": "Task 9: live gate — three meshes, per-pair redundancy, cross-mesh transports","classification": "high-risk", "status": "in-progress", "note": "local OrbStack docker up; full-solution build verified clean; building images"} + {"id": 9, "subject": "Task 9: live gate — three meshes, per-pair redundancy, cross-mesh transports","classification": "high-risk", "status": "completed", "note": "PASSED — all 8 legs green (docs/plans/2026-07-24-mesh-phase6-live-gate.md). Caught + fixed-forward 3 defects: (1) product boot-crash StackOverflow — cluster-redundancy singleton resolved IClusterRoleInfo eagerly inside the AddAkka lambda while the ActorSystem was building (fix 3a4ed8dd: derive scope from AkkaClusterOptions); (2) rig — site nodes crashed on LDAP options (no Security__Ldap__* block); (3) rig — site pairs split-brained on simultaneous start (both self-first seeds), fixed with a founder healthcheck + service_healthy gating (2+3 in 792f28ec). Legs: 3 isolated meshes, one Primary/pair 250/240, deploy sealed acks=6 over per-cluster ClusterClient, telemetry gRPC :4056 up + reconnect, reconciler own-cluster-scoped, secrets pair-local, split validator refuses Dps, failover+auto-down-1v1 survival proven. Master 792f28ec."} ], - "lastUpdated": "2026-07-24T00:00:00Z" + "lastUpdated": "2026-07-24T11:10:00Z" }