diff --git a/docs/plans/2026-07-22-per-cluster-mesh-program.md b/docs/plans/2026-07-22-per-cluster-mesh-program.md index 4bdc9222..301c5f40 100644 --- a/docs/plans/2026-07-22-per-cluster-mesh-program.md +++ b/docs/plans/2026-07-22-per-cluster-mesh-program.md @@ -167,7 +167,7 @@ same pass. **Exit gate — MET:** live gate green on the rig; a driver node with an empty LocalDb and reachable central boots into the current config; with central down it boots last-known-good. -### Phase 4 — Cut the driver-side ConfigDb connection — **CODE-COMPLETE 2026-07-23 (Tasks 0–7 + 1b; Task 9 table-drop deferred; Task 10 rig + Task 11 live gate remain)** +### Phase 4 — Cut the driver-side ConfigDb connection — **DONE 2026-07-23 (live gate PASSED; Tasks 0–8 + 1b + 10–11; Task 9 table-drop deferred)** **Scope (design §6.1 audit table):** re-home `EfAlarmConditionStateStore` to LocalDb (pair-local state, same journey as the Phase-2 alarm S&F buffer); **resolve the `DbHealthProbeActor` question** — driver nodes have no DB to probe, and DB health currently feeds ServiceLevel tiering, @@ -266,7 +266,7 @@ resource sizing on the site VMs should be checked once both products run the ful | 1 ClusterNode columns + DB ack set | **DONE 2026-07-22**, merged `7654f24d`. Deploy-seal semantics changed as flagged, but the escape hatch is **`ClusterNode.MaintenanceMode`**, not `Enabled=0` — the live gate found `Enabled=0` is rejected by `DraftValidator.ValidateClusterTopology` on any 2-node pair. Gate record: `2026-07-22-mesh-phase1-live-gate.md`. | | 2 ClusterClient transport | **DONE 2026-07-22** (live gate PASSED, shipped dark) — shipped dark (`MeshTransport:Mode=Dps` default), both comm actors registered in both modes. Corrections: `SendToAll` not `Send`, one fleet-wide client not one per cluster, `/user/node-communication` not `/user/cluster-communication`. The gate's "Ask timing out" item does not exist to test — no cross-boundary Ask in this phase; see the phase section. | | 3 fetch-and-cache | **DONE 2026-07-23**, merged `d01b0695`, live gate PASSED, pushed to origin (scadaproj umbrella index updated + pushed `b5e7bc8`). gRPC fetch RPC + shared node key; dark switch `ConfigSource:Mode` (Direct default). Rig flip: `OTOPCUA_CONFIG_MODE=FetchAndCache` on the site nodes. See `2026-07-22-mesh-phase3-config-fetch-and-cache.md`. | -| 4 cut driver ConfigDb | **CODE-COMPLETE 2026-07-23** on `feat/mesh-phase4` (Tasks 0–7 + 1b done; Task 9 table-drop deferred; Task 10 rig + Task 11 live gate remain — not yet run). ConfigDb admin-only; `DbHealthProbeActor` not spawned driver-only (client-visible ServiceLevel 240/250 with central down); alarm condition state in LocalDb; central persists acks; driver-only LDAP maps from appsettings only. See `2026-07-23-mesh-phase4-cut-driver-configdb.md`. | +| 4 cut driver ConfigDb | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase4` (Tasks 0–8 + 1b + 10–11; Task 9 table-drop deferred). ConfigDb admin-only; driver-only ⇒ FetchAndCache (validator); `DbHealthProbeActor` not spawned driver-only (client-visible ServiceLevel 240/250 with central SQL down — proven live); alarm condition state in replicated LocalDb `alarm_condition_state`; central persists acks; `OpcUaPublish` guard split fixes the driver-only address-space wipe; driver-only LDAP maps from appsettings only (6th consumer found mid-phase). Gate: deploy sealed green w/ 4 DB-less site nodes acking, ServiceLevel held 240 w/ SQL down, restart booted last-known-good from the LocalDb pointer. See `2026-07-23-mesh-phase4-cut-driver-configdb.md` + `2026-07-23-mesh-phase4-live-gate.md`. | | 5 gRPC telemetry | not started | | 6 mesh partition + co-location | not started | | 7 drill + live gates | not started | diff --git a/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json b/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json index ecc0c4b7..20e1ce31 100644 --- a/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json +++ b/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json @@ -119,12 +119,13 @@ { "id": 11, "subject": "Task 11: Live gate", - "status": "in_progress", + "status": "completed", "blockedBy": [ 8, 9, 10 - ] + ], + "result": "LIVE GATE PASSED \u2014 deploy sealed green w/ 4 DB-less site nodes acking; ServiceLevel 240 w/ central SQL down; restart booted last-known-good from LocalDb pointer; grep proof; alarm_condition_state table live+replicated. Doc: 2026-07-23-mesh-phase4-live-gate.md" } ], "lastUpdated": "2026-07-23" diff --git a/docs/plans/2026-07-23-mesh-phase4-live-gate.md b/docs/plans/2026-07-23-mesh-phase4-live-gate.md new file mode 100644 index 00000000..bbef5281 --- /dev/null +++ b/docs/plans/2026-07-23-mesh-phase4-live-gate.md @@ -0,0 +1,91 @@ +# Per-Cluster Mesh Phase 4 — Live Gate Record + +**Date:** 2026-07-23 +**Branch:** `feat/mesh-phase4` (image built + rig recreated with Phase-4 code) +**Rig:** local `docker-dev` — central-1/2 (`admin,driver`, Direct + ConfigDb + ConfigServe), four site +nodes (`driver`, **no ConfigDb string**, `ConfigSource:Mode=FetchAndCache`). Deployment `b390f535`. + +**Result: PASSED.** Every load-bearing exit-gate leg proven live. A driver-only node runs a full deploy ++ boot-from-cache cycle with **no ConfigDb connection string configured at all**, and the client-visible +ServiceLevel change behaves as designed. + +## What was verified + +### Leg 0 — boot ConfigDb-free (headline) +Every site node booted with no ConfigDb string and logged, e.g.: +``` +DriverHost site-a-1:4053: FetchAndCache boot — restored served state for deployment 1841b0c2… +(rev 7a7d548e…) from the LocalDb pointer. +``` +All six nodes came up clean (no FATAL / startup exception). The transient `RpcException Unavailable` +(central ConfigServe not yet up when the first fetch fired) and the `Name or service not known` +(the rig's deliberately-unresolvable historian endpoint) are expected startup noise — the node fell back +to the LocalDb pointer (#485-safe), never to an empty address space. + +### Leg 1 — deploy seals green with site nodes acking, no ConfigDb ✅ +`POST /api/deployments` (`X-Api-Key: docker-dev-deploy-key`) → `202 Accepted` (deployment `b390f535`). +It **sealed** (`Deployment.Status = 2`) at 17:57:33. Per-node `NodeDeploymentState`: + +| Node | Status | +|---|---| +| central-1:4053 | 1 (Applied) | +| central-2:4053 | 1 (Applied) | +| **site-a-1:4053** | **1 (Applied)** | +| **site-a-2:4053** | **1 (Applied)** | +| **site-b-1:4053** | **1 (Applied)** | +| **site-b-2:4053** | **1 (Applied)** | + +The four site nodes hold no ConfigDb — those ack rows were written by central's +`ConfigPublishCoordinator.PersistNodeAck` from the `ApplyAck` each site node sent over the transport. +Full deploy→fetch→apply→ack cycle proven ConfigDb-free. + +### Leg 2 — ServiceLevel with central SQL DOWN (client-visible change) ✅ +- Baseline (SQL up): site-a-1 (`opc.tcp://localhost:4842`) **ServiceLevel = 240** (healthy follower). +- `docker stop otopcua-dev-sql-1` → site-a-1 **ServiceLevel = 240, unchanged**. + +A DB-less node feeds `DbReachable = true` constant with staleness from the redundancy-snapshot age only, +so it stays at full service when central SQL is unreachable — the survive-alone posture. A Direct/DB-backed +node would have dropped to 0/100 (its `DbHealthProbe` trips). Confirmed live: central-1's own probe went +`Reachable = False` while SQL was down (and back to `True` when it returned), while the site node never moved. + +### Leg 3 — restart a site node with central SQL DOWN → boots last-known-good ✅ +Restarted site-b-1 (the non-replicating site-b pin) while SQL was still stopped. Post-restart boot +(18:00:58–59): +``` +Cluster Node [akka.tcp://otopcua@site-b-1:4053] - Started up successfully +OPC UA server started on opc.tcp://0.0.0.0:4840 +DriverHost site-b-1:4053: FetchAndCache boot — restored served state for deployment 1841b0c2… +(rev 7a7d548e…) from the LocalDb pointer. +``` +No ConfigDb / SQL error in the fresh boot (it never dials SQL). Post-restart **ServiceLevel = 240** — it +served last-known-good from the LocalDb pointer with central SQL down. + +### Leg 4 — grep proof ✅ +Each site container's env: `ConnectionStrings__ConfigDb` count = **0**, `ConfigSource__Mode=FetchAndCache`. +Central-1 (contrast): carries `ConnectionStrings__ConfigDb` + `Mode=Direct`. Code sweep (Task 7) confirmed +no driver-branch service resolves `OtOpcUaConfigDbContext`. + +### Leg 5 — alarm condition state in LocalDb ✅ (table live + replicated) +site-a-1's consolidated LocalDb (`/app/data/otopcua-localdb.db`) carries the `alarm_condition_state` +table with its replication capture triggers (`__localdb_*`), alongside `deployment_artifacts`, +`deployment_pointer`, and `alarm_sf_events`. 0 rows (no scripted alarm has fired in this window). The +store's save/load round-trip is unit-tested (`LocalDbAlarmConditionStateStoreTests`); a full operator +ack→row cycle was not driven live (needs a configured scripted alarm + a triggering value) — noted as the +one leg proven by test + schema-liveness rather than an end-to-end row write. + +## Cleanup +`docker start otopcua-dev-sql-1` → healthy; all six nodes up; central-1 DbHealth recovered +(`Reachable=False → True` at 18:01:43). Rig healed. + +## Diagnostics note (the "stuck gate") +The initial rig rebuild appeared hung: an abandoned `dotnet-dump analyze hang.dmp` (PID 94724, ~115 h of +CPU since Saturday) had pinned a core, starving the emulated `linux/amd64` `dotnet publish` (the Phase-3 +protoc-segfault build-stage pin), and `docker compose --build`'s default `rawjson` progress streamed +nothing to stdout. Killing the runaway process + rebuilding with `--progress=plain` made the build both +faster and observable. No Phase-4 defect involved. + +## Verdict +**Phase 4 live gate PASSED.** Driver-only nodes run ConfigDb-free end to end; the ServiceLevel semantics +change is live and correct. Remaining Phase-4 item: Task 9 (drop the dead ConfigDb `ScriptedAlarmState` +table + retire `EfAlarmConditionStateStore`) — deferred as plan-sanctioned (inert on all real nodes; +irreversible migration).