lmxopcua/docs/plans/2026-07-23-mesh-phase4-live-gate.md

Per-Cluster Mesh Phase 4 — Live Gate Record

Date: 2026-07-23 Branch: feat/mesh-phase4 (image built + rig recreated with Phase-4 code) Rig: local docker-dev — central-1/2 (admin,driver, Direct + ConfigDb + ConfigServe), four site nodes (driver, no ConfigDb string, ConfigSource:Mode=FetchAndCache). Deployment b390f535.

Result: PASSED. Every load-bearing exit-gate leg proven live. A driver-only node runs a full deploy

• boot-from-cache cycle with no ConfigDb connection string configured at all, and the client-visible ServiceLevel change behaves as designed.

What was verified

Leg 0 — boot ConfigDb-free (headline)

Every site node booted with no ConfigDb string and logged, e.g.:

DriverHost site-a-1:4053: FetchAndCache boot — restored served state for deployment 1841b0c2…
(rev 7a7d548e…) from the LocalDb pointer.

All six nodes came up clean (no FATAL / startup exception). The transient RpcException Unavailable (central ConfigServe not yet up when the first fetch fired) and the Name or service not known (the rig's deliberately-unresolvable historian endpoint) are expected startup noise — the node fell back to the LocalDb pointer (#485-safe), never to an empty address space.

Leg 1 — deploy seals green with site nodes acking, no ConfigDb ✅

POST /api/deployments (X-Api-Key: docker-dev-deploy-key) → 202 Accepted (deployment b390f535). It sealed (Deployment.Status = 2) at 17:57:33. Per-node NodeDeploymentState:

Node	Status
central-1:4053	1 (Applied)
central-2:4053	1 (Applied)
site-a-1:4053	1 (Applied)
site-a-2:4053	1 (Applied)
site-b-1:4053	1 (Applied)
site-b-2:4053	1 (Applied)

The four site nodes hold no ConfigDb — those ack rows were written by central's ConfigPublishCoordinator.PersistNodeAck from the ApplyAck each site node sent over the transport. Full deploy→fetch→apply→ack cycle proven ConfigDb-free.

Leg 2 — ServiceLevel with central SQL DOWN (client-visible change) ✅

• Baseline (SQL up): site-a-1 (opc.tcp://localhost:4842) ServiceLevel = 240 (healthy follower).
• docker stop otopcua-dev-sql-1 → site-a-1 ServiceLevel = 240, unchanged.

A DB-less node feeds DbReachable = true constant with staleness from the redundancy-snapshot age only, so it stays at full service when central SQL is unreachable — the survive-alone posture. A Direct/DB-backed node would have dropped to 0/100 (its DbHealthProbe trips). Confirmed live: central-1's own probe went Reachable = False while SQL was down (and back to True when it returned), while the site node never moved.

Leg 3 — restart a site node with central SQL DOWN → boots last-known-good ✅

Restarted site-b-1 (the non-replicating site-b pin) while SQL was still stopped. Post-restart boot (18:00:58–59):

Cluster Node [akka.tcp://otopcua@site-b-1:4053] - Started up successfully
OPC UA server started on opc.tcp://0.0.0.0:4840
DriverHost site-b-1:4053: FetchAndCache boot — restored served state for deployment 1841b0c2…
(rev 7a7d548e…) from the LocalDb pointer.

No ConfigDb / SQL error in the fresh boot (it never dials SQL). Post-restart ServiceLevel = 240 — it served last-known-good from the LocalDb pointer with central SQL down.

Leg 4 — grep proof ✅

Each site container's env: ConnectionStrings__ConfigDb count = 0, ConfigSource__Mode=FetchAndCache. Central-1 (contrast): carries ConnectionStrings__ConfigDb + Mode=Direct. Code sweep (Task 7) confirmed no driver-branch service resolves OtOpcUaConfigDbContext.

Leg 5 — alarm condition state in LocalDb ✅ (table live + replicated)

site-a-1's consolidated LocalDb (/app/data/otopcua-localdb.db) carries the alarm_condition_state table with its replication capture triggers (__localdb_*), alongside deployment_artifacts, deployment_pointer, and alarm_sf_events. 0 rows (no scripted alarm has fired in this window). The store's save/load round-trip is unit-tested (LocalDbAlarmConditionStateStoreTests); a full operator ack→row cycle was not driven live (needs a configured scripted alarm + a triggering value) — noted as the one leg proven by test + schema-liveness rather than an end-to-end row write.

Cleanup

docker start otopcua-dev-sql-1 → healthy; all six nodes up; central-1 DbHealth recovered (Reachable=False → True at 18:01:43). Rig healed.

Diagnostics note (the "stuck gate")

The initial rig rebuild appeared hung: an abandoned dotnet-dump analyze hang.dmp (PID 94724, ~115 h of CPU since Saturday) had pinned a core, starving the emulated linux/amd64 dotnet publish (the Phase-3 protoc-segfault build-stage pin), and docker compose --build's default rawjson progress streamed nothing to stdout. Killing the runaway process + rebuilding with --progress=plain made the build both faster and observable. No Phase-4 defect involved.

Verdict

Phase 4 live gate PASSED. Driver-only nodes run ConfigDb-free end to end; the ServiceLevel semantics change is live and correct. Remaining Phase-4 item: Task 9 (drop the dead ConfigDb ScriptedAlarmState table + retire EfAlarmConditionStateStore) — deferred as plan-sanctioned (inert on all real nodes; irreversible migration).