docs(mesh-phase4): live gate PASSED — driver nodes run ConfigDb-free

Deploy sealed green with 4 DB-less site nodes acking (central persists acks);
ServiceLevel held 240 on a site node with central SQL stopped (survive-alone);
restarted site node booted last-known-good from the LocalDb pointer, no ConfigDb
read; alarm_condition_state table live + replicated. Full record in
2026-07-23-mesh-phase4-live-gate.md. Task 9 (drop dead table) deferred.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-23 14:03:28 -04:00
parent 0170614961
commit 1281aebfa7
3 changed files with 96 additions and 4 deletions
@@ -167,7 +167,7 @@ same pass.
**Exit gate — MET:** live gate green on the rig; a driver node with an empty LocalDb and reachable
central boots into the current config; with central down it boots last-known-good.
### Phase 4 — Cut the driver-side ConfigDb connection — **CODE-COMPLETE 2026-07-23 (Tasks 07 + 1b; Task 9 table-drop deferred; Task 10 rig + Task 11 live gate remain)**
### Phase 4 — Cut the driver-side ConfigDb connection — **DONE 2026-07-23 (live gate PASSED; Tasks 08 + 1b + 1011; Task 9 table-drop deferred)**
**Scope (design §6.1 audit table):** re-home `EfAlarmConditionStateStore` to LocalDb (pair-local
state, same journey as the Phase-2 alarm S&F buffer); **resolve the `DbHealthProbeActor`
question** — driver nodes have no DB to probe, and DB health currently feeds ServiceLevel tiering,
@@ -266,7 +266,7 @@ resource sizing on the site VMs should be checked once both products run the ful
| 1 ClusterNode columns + DB ack set | **DONE 2026-07-22**, merged `7654f24d`. Deploy-seal semantics changed as flagged, but the escape hatch is **`ClusterNode.MaintenanceMode`**, not `Enabled=0` — the live gate found `Enabled=0` is rejected by `DraftValidator.ValidateClusterTopology` on any 2-node pair. Gate record: `2026-07-22-mesh-phase1-live-gate.md`. |
| 2 ClusterClient transport | **DONE 2026-07-22** (live gate PASSED, shipped dark) — shipped dark (`MeshTransport:Mode=Dps` default), both comm actors registered in both modes. Corrections: `SendToAll` not `Send`, one fleet-wide client not one per cluster, `/user/node-communication` not `/user/cluster-communication`. The gate's "Ask timing out" item does not exist to test — no cross-boundary Ask in this phase; see the phase section. |
| 3 fetch-and-cache | **DONE 2026-07-23**, merged `d01b0695`, live gate PASSED, pushed to origin (scadaproj umbrella index updated + pushed `b5e7bc8`). gRPC fetch RPC + shared node key; dark switch `ConfigSource:Mode` (Direct default). Rig flip: `OTOPCUA_CONFIG_MODE=FetchAndCache` on the site nodes. See `2026-07-22-mesh-phase3-config-fetch-and-cache.md`. |
| 4 cut driver ConfigDb | **CODE-COMPLETE 2026-07-23** on `feat/mesh-phase4` (Tasks 07 + 1b done; Task 9 table-drop deferred; Task 10 rig + Task 11 live gate remain — not yet run). ConfigDb admin-only; `DbHealthProbeActor` not spawned driver-only (client-visible ServiceLevel 240/250 with central down); alarm condition state in LocalDb; central persists acks; driver-only LDAP maps from appsettings only. See `2026-07-23-mesh-phase4-cut-driver-configdb.md`. |
| 4 cut driver ConfigDb | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase4` (Tasks 08 + 1b + 1011; Task 9 table-drop deferred). ConfigDb admin-only; driver-only ⇒ FetchAndCache (validator); `DbHealthProbeActor` not spawned driver-only (client-visible ServiceLevel 240/250 with central SQL down — proven live); alarm condition state in replicated LocalDb `alarm_condition_state`; central persists acks; `OpcUaPublish` guard split fixes the driver-only address-space wipe; driver-only LDAP maps from appsettings only (6th consumer found mid-phase). Gate: deploy sealed green w/ 4 DB-less site nodes acking, ServiceLevel held 240 w/ SQL down, restart booted last-known-good from the LocalDb pointer. See `2026-07-23-mesh-phase4-cut-driver-configdb.md` + `2026-07-23-mesh-phase4-live-gate.md`. |
| 5 gRPC telemetry | not started |
| 6 mesh partition + co-location | not started |
| 7 drill + live gates | not started |
@@ -119,12 +119,13 @@
{
"id": 11,
"subject": "Task 11: Live gate",
"status": "in_progress",
"status": "completed",
"blockedBy": [
8,
9,
10
]
],
"result": "LIVE GATE PASSED \u2014 deploy sealed green w/ 4 DB-less site nodes acking; ServiceLevel 240 w/ central SQL down; restart booted last-known-good from LocalDb pointer; grep proof; alarm_condition_state table live+replicated. Doc: 2026-07-23-mesh-phase4-live-gate.md"
}
],
"lastUpdated": "2026-07-23"
@@ -0,0 +1,91 @@
# Per-Cluster Mesh Phase 4 — Live Gate Record
**Date:** 2026-07-23
**Branch:** `feat/mesh-phase4` (image built + rig recreated with Phase-4 code)
**Rig:** local `docker-dev` — central-1/2 (`admin,driver`, Direct + ConfigDb + ConfigServe), four site
nodes (`driver`, **no ConfigDb string**, `ConfigSource:Mode=FetchAndCache`). Deployment `b390f535`.
**Result: PASSED.** Every load-bearing exit-gate leg proven live. A driver-only node runs a full deploy
+ boot-from-cache cycle with **no ConfigDb connection string configured at all**, and the client-visible
ServiceLevel change behaves as designed.
## What was verified
### Leg 0 — boot ConfigDb-free (headline)
Every site node booted with no ConfigDb string and logged, e.g.:
```
DriverHost site-a-1:4053: FetchAndCache boot — restored served state for deployment 1841b0c2…
(rev 7a7d548e…) from the LocalDb pointer.
```
All six nodes came up clean (no FATAL / startup exception). The transient `RpcException Unavailable`
(central ConfigServe not yet up when the first fetch fired) and the `Name or service not known`
(the rig's deliberately-unresolvable historian endpoint) are expected startup noise — the node fell back
to the LocalDb pointer (#485-safe), never to an empty address space.
### Leg 1 — deploy seals green with site nodes acking, no ConfigDb ✅
`POST /api/deployments` (`X-Api-Key: docker-dev-deploy-key`) → `202 Accepted` (deployment `b390f535`).
It **sealed** (`Deployment.Status = 2`) at 17:57:33. Per-node `NodeDeploymentState`:
| Node | Status |
|---|---|
| central-1:4053 | 1 (Applied) |
| central-2:4053 | 1 (Applied) |
| **site-a-1:4053** | **1 (Applied)** |
| **site-a-2:4053** | **1 (Applied)** |
| **site-b-1:4053** | **1 (Applied)** |
| **site-b-2:4053** | **1 (Applied)** |
The four site nodes hold no ConfigDb — those ack rows were written by central's
`ConfigPublishCoordinator.PersistNodeAck` from the `ApplyAck` each site node sent over the transport.
Full deploy→fetch→apply→ack cycle proven ConfigDb-free.
### Leg 2 — ServiceLevel with central SQL DOWN (client-visible change) ✅
- Baseline (SQL up): site-a-1 (`opc.tcp://localhost:4842`) **ServiceLevel = 240** (healthy follower).
- `docker stop otopcua-dev-sql-1` → site-a-1 **ServiceLevel = 240, unchanged**.
A DB-less node feeds `DbReachable = true` constant with staleness from the redundancy-snapshot age only,
so it stays at full service when central SQL is unreachable — the survive-alone posture. A Direct/DB-backed
node would have dropped to 0/100 (its `DbHealthProbe` trips). Confirmed live: central-1's own probe went
`Reachable = False` while SQL was down (and back to `True` when it returned), while the site node never moved.
### Leg 3 — restart a site node with central SQL DOWN → boots last-known-good ✅
Restarted site-b-1 (the non-replicating site-b pin) while SQL was still stopped. Post-restart boot
(18:00:5859):
```
Cluster Node [akka.tcp://otopcua@site-b-1:4053] - Started up successfully
OPC UA server started on opc.tcp://0.0.0.0:4840
DriverHost site-b-1:4053: FetchAndCache boot — restored served state for deployment 1841b0c2…
(rev 7a7d548e…) from the LocalDb pointer.
```
No ConfigDb / SQL error in the fresh boot (it never dials SQL). Post-restart **ServiceLevel = 240** — it
served last-known-good from the LocalDb pointer with central SQL down.
### Leg 4 — grep proof ✅
Each site container's env: `ConnectionStrings__ConfigDb` count = **0**, `ConfigSource__Mode=FetchAndCache`.
Central-1 (contrast): carries `ConnectionStrings__ConfigDb` + `Mode=Direct`. Code sweep (Task 7) confirmed
no driver-branch service resolves `OtOpcUaConfigDbContext`.
### Leg 5 — alarm condition state in LocalDb ✅ (table live + replicated)
site-a-1's consolidated LocalDb (`/app/data/otopcua-localdb.db`) carries the `alarm_condition_state`
table with its replication capture triggers (`__localdb_*`), alongside `deployment_artifacts`,
`deployment_pointer`, and `alarm_sf_events`. 0 rows (no scripted alarm has fired in this window). The
store's save/load round-trip is unit-tested (`LocalDbAlarmConditionStateStoreTests`); a full operator
ack→row cycle was not driven live (needs a configured scripted alarm + a triggering value) — noted as the
one leg proven by test + schema-liveness rather than an end-to-end row write.
## Cleanup
`docker start otopcua-dev-sql-1` → healthy; all six nodes up; central-1 DbHealth recovered
(`Reachable=False → True` at 18:01:43). Rig healed.
## Diagnostics note (the "stuck gate")
The initial rig rebuild appeared hung: an abandoned `dotnet-dump analyze hang.dmp` (PID 94724, ~115 h of
CPU since Saturday) had pinned a core, starving the emulated `linux/amd64` `dotnet publish` (the Phase-3
protoc-segfault build-stage pin), and `docker compose --build`'s default `rawjson` progress streamed
nothing to stdout. Killing the runaway process + rebuilding with `--progress=plain` made the build both
faster and observable. No Phase-4 defect involved.
## Verdict
**Phase 4 live gate PASSED.** Driver-only nodes run ConfigDb-free end to end; the ServiceLevel semantics
change is live and correct. Remaining Phase-4 item: Task 9 (drop the dead ConfigDb `ScriptedAlarmState`
table + retire `EfAlarmConditionStateStore`) — deferred as plan-sanctioned (inert on all real nodes;
irreversible migration).