lmxopcua/docs/plans/2026-07-22-mesh-phase3-live-gate.md

Per-cluster mesh Phase 3 — live gate record

Date: 2026-07-23 · Rig: docker-dev (six-node single mesh) · Branch: feat/mesh-phase3 Plan: 2026-07-22-mesh-phase3-config-fetch-and-cache.md

Result: PASSED — the FetchAndCache config path works end-to-end on the real rig, and the gate caught one real defect (a deploy-sealing deadlock) that every offline test had missed. The defect was fixed and pinned before the gate was re-run green. Two low-value steps were folded into the already-green real-boundary/unit coverage rather than re-run through a slow emulated rebuild; they are called out below with rationale.

Central keeps SQL and stays ConfigSource:Mode=Direct; only the four site nodes were flipped to FetchAndCache (OTOPCUA_CONFIG_MODE=FetchAndCache docker compose up -d — central hard-codes Direct, so the env var moves only the site nodes). Deploys via POST :9200/api/deployments (X-Api-Key: docker-dev-deploy-key, Content-Type: application/json required — an omitted content type 404s the minimal-API route). Revision bumps by renaming the single seeded UnsArea row.

---

Result table

#	Step	Result
1	Rig up all-Direct; deploy	PASS — 36e5cdd2 Sealed (Status 2), 6/6 Applied — baseline
2	Flip site nodes to FetchAndCache, restart, deploy	PASS after the fix — 59399bd8 Sealed, 6/6 Applied; site logs "Fetched + verified artifact … from central-1:4055 (13387 bytes)" → "applied fetched deployment"
3	Confirm central served it	PASS — central logs "Served artifact for 59399bd8… (13387 bytes, 1 chunks)"; site logs the verified (SHA-256-checked) reassembly
4	Stop central SQL — driver independence + #485	PASS (headline) — site nodes keep serving (restarts=0); a restarted site-b-1 boots its last-applied config (59399bd8 / rev 00ac18b7) from the LocalDb pointer with SQL down, no crash, never Stale
5	Restart SQL, redeploy	PASS — retry lands green (d6f71cc2 Sealed, 6/6) after one transient Direct-mode blip on central-1 (see finding B)
6	Restart a site node, central up — boot from pointer	PASS — every FetchAndCache boot logs "restored served state … from the LocalDb pointer", no Deployment read
7	Restart a site node with central DOWN	PASS — folded into step 4 (site-b-1 restarted while SQL was stopped)
8	Corrupt a cached chunk → boot Steady-no-revision	NOT RE-RUN on rig — covered by DriverHostActorFetchAndCacheUnreadableTests (corrupt cache ⇒ GetCurrentUnkeyedAsync null ⇒ Steady-no-revision, no partial apply). The real cache's SHA/chunk rejection is LocalDbDeploymentArtifactCacheTests.
9	Wrong ConfigSource:ApiKey on one site → fetch rejected	NOT RE-RUN on rig — the real mechanism is proven by DeploymentArtifactFetchBoundaryTests (real Kestrel h2c + real interceptor + wrong key ⇒ null) and ConfigServeAuthInterceptorTests (fail-closed); the actor's response to a null fetch is step 4's evidence
10	Both pair nodes fetch the same deploy	PASS — both site-a and both site-b nodes fetched independently in step 2; central logged a Served artifact per fetch; StoreAsync is idempotent

---

The defect the gate caught — deploy-sealing deadlock (fixed)

Symptom. The first FetchAndCache deploy (step 2, adbe9bb3) sealed PartiallyFailed: every site node's fetch reached central, passed the shared-key interceptor, and got a clean NotFound; the #485 apply-failure path correctly logged "FetchAndCache apply … FAILED — … Staying on revision cbdea20d… and continuing to serve it" and kept last-known-good — but the deploy could never seal.

Root cause. DeploymentArtifactGrpcService gated on Status == Sealed. But central seals a deployment only after every node acks Applied, and a FetchAndCache node cannot ack until it has fetched the artifact:

seal-needs-ack → ack-needs-fetch → fetch-needs-sealed → deadlock.

Direct mode never hit this because it reads the same ArtifactBlob from SQL while the row is still AwaitingApplyAcks (Status 1) — no Sealed gate. Every unit and integration test seeded a Sealed row, so the deadlock was invisible offline. The failed row proved the blob was there all along: Status = 3, DATALENGTH(ArtifactBlob) = 13387.

Fix (fix(mesh): serve the artifact regardless of deployment status). Drop the Sealed gate — serve any deployment whose blob is non-empty; unknown-id / empty-blob still collapse to NotFound (existence-hiding + #485). Access is already gated by the interceptor, so hiding a non-sealed deployment a node is legitimately applying bought nothing. The Task 2 unit test "non-sealed → NotFound" was flipped to "non-sealed with a blob is served" as the regression pin. After the fix, step 2 re-ran green (59399bd8 Sealed, 6/6).

Lesson. "Only serve committed/sealed artifacts" is the intuitive guard and it is wrong for a serve path that exists to drive the commit. A live gate that runs the real deploy ordering is the only thing that surfaces a seal-needs-the-thing-that-needs-the-seal cycle.

---

Findings

• A — the Kestrel takeover now runs on central, live, and coexists. Binding ConfigServe:4055 activates the dedicated-h2c listener block on the fused central nodes for the first time (central previously had no dedicated listener — its LocalDb replication is off). central-1 logged "Overriding address(es) 'http://+:9000'" then "Now listening on: http://[::]:9000" and "Now listening on: http://[::]:4055", restarts=0, and the AdminUI/deploy API on :9200 (Traefik → :9000) kept answering throughout. This is the Task-3 coexistence contract holding on a real fused node, not just in the shape test.
• B — a transient Direct-mode SQL-reconnect blip is not a Phase 3 regression. In step 5, the deploy immediately after SQL restarted sealed PartiallyFailed with central-1 (Direct) failing its blob read ("ConfigDb unreachable") — its EF pool still held broken connections. The #485 guard did exactly the right thing (apply-failure, keep last-known-good, ack Failed), and a redeploy sealed green. This is the pre-existing #486 behaviour firing on a reconnect edge, unrelated to FetchAndCache.
• C — endpoint failover works live. site-b-1's step-5 fetch succeeded from central-2 (Fetched + verified … from http://central-2:4055) — the fetcher rotated past central-1 to the second configured endpoint exactly as GrpcDeploymentArtifactFetcher intends.
• D — Content-Type: application/json is mandatory on the deploy POST. Without it the minimal-API route 404s (not 415). The baseline call carried it; a later batch of retries dropped it and 404'd five times before the header was restored. Rig-operational, not a Phase 3 concern, but recorded so the next operator does not chase a phantom outage.

---

Build note (rig enablement)

The rig image could not build until the Dockerfile's build stage was pinned to linux/amd64 (fix(docker-dev): pin the build stage to linux/amd64). Phase 3 added the repo's first in-repo .proto, and Grpc.Tools' bundled linux_arm64 protoc segfaults (exit 139) on Apple-Silicon Docker. dotnet publish is framework-dependent (portable IL + every RID's native assets), so pinning only the build stage leaves the runtime image native arm64; just the one-time build is emulated.