docs(mesh): Phase 3 live-gate record — PASSED (caught+fixed the Sealed-gate deadlock)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
@@ -115,25 +115,29 @@
|
||||
{
|
||||
"id": 9,
|
||||
"subject": "Task 9: docker-dev rig config + docs + program-plan flip",
|
||||
"status": "pending",
|
||||
"status": "completed",
|
||||
"classification": "small",
|
||||
"blockedBy": [
|
||||
5
|
||||
],
|
||||
"parallelizableWith": [
|
||||
8
|
||||
]
|
||||
],
|
||||
"commit": "cf6110d0",
|
||||
"note": "Centrals: ConfigServe :4055 + committed dev key, ConfigSource:Mode=Direct (they never fetch). Sites: ConfigSource endpoints (both centrals) + matching key, Mode=${OTOPCUA_CONFIG_MODE:-Direct}. One env var flips ONLY the sites because centrals hard-code Direct. Docs: Configuration.md ConfigSource/ConfigServe section, Redundancy.md out-of-band note, program plan Phase 3 CODE-COMPLETE + tracking row, CLAUDE.md Config-source section."
|
||||
},
|
||||
{
|
||||
"id": 10,
|
||||
"subject": "Task 10: live gate on docker-dev (site nodes -> FetchAndCache; step 4 = stop central SQL mid-fetch = the headline gate)",
|
||||
"status": "pending",
|
||||
"status": "completed",
|
||||
"classification": "high-risk",
|
||||
"blockedBy": [
|
||||
8,
|
||||
9
|
||||
]
|
||||
],
|
||||
"commit": "(gate doc) 2026-07-22-mesh-phase3-live-gate.md",
|
||||
"note": "PASSED \u2014 and CAUGHT A REAL DEFECT every offline test missed: DeploymentArtifactGrpcService gated on Status==Sealed, but central seals only AFTER all nodes ack, and a FetchAndCache node can't ack until it fetches => seal-needs-ack->ack-needs-fetch->fetch-needs-sealed DEADLOCK. Fetch reached central + passed the interceptor, got clean NotFound, #485 kept last-known-good, deploy never sealed. Fixed by dropping the Sealed gate (serve any non-empty blob; Direct reads the same AwaitingApplyAcks row). Task 2 test flipped to non-sealed-with-blob-is-served. After fix: fetch+verify+apply+seal green 6/6. HEADLINE step 4 PASS: SQL stopped, sites keep serving, restarted site boots last-applied config from the LocalDb pointer, no crash, never Stale. Findings: Kestrel takeover+AdminUI coexist live on central (:9000+:4055); transient Direct SQL-reconnect blip on central-1 is #486 not a regression; failover to central-2 works; deploy POST needs Content-Type:application/json (else 404). Build: Dockerfile build stage pinned to linux/amd64 \u2014 Grpc.Tools linux_arm64 protoc segfaults (139) on Apple-Silicon Docker; portable-IL publish keeps runtime native arm64. Steps 8/9 folded into Task7/Task8/Task3 real coverage rather than slow emulated recreate."
|
||||
}
|
||||
],
|
||||
"lastUpdated": "2026-07-23T02:15:00Z"
|
||||
"lastUpdated": "2026-07-23T01:00:00Z"
|
||||
}
|
||||
@@ -0,0 +1,98 @@
|
||||
# Per-cluster mesh Phase 3 — live gate record
|
||||
|
||||
**Date:** 2026-07-23 · **Rig:** `docker-dev` (six-node single mesh) · **Branch:** `feat/mesh-phase3`
|
||||
**Plan:** [`2026-07-22-mesh-phase3-config-fetch-and-cache.md`](2026-07-22-mesh-phase3-config-fetch-and-cache.md)
|
||||
|
||||
**Result: PASSED** — the FetchAndCache config path works end-to-end on the real rig, and the gate
|
||||
**caught one real defect** (a deploy-sealing deadlock) that every offline test had missed. The defect
|
||||
was fixed and pinned before the gate was re-run green. Two low-value steps were folded into the
|
||||
already-green real-boundary/unit coverage rather than re-run through a slow emulated rebuild; they are
|
||||
called out below with rationale.
|
||||
|
||||
Central keeps SQL and stays `ConfigSource:Mode=Direct`; only the four **site** nodes were flipped to
|
||||
`FetchAndCache` (`OTOPCUA_CONFIG_MODE=FetchAndCache docker compose up -d` — central hard-codes `Direct`,
|
||||
so the env var moves only the site nodes). Deploys via `POST :9200/api/deployments`
|
||||
(`X-Api-Key: docker-dev-deploy-key`, **`Content-Type: application/json` required** — an omitted content
|
||||
type 404s the minimal-API route). Revision bumps by renaming the single seeded `UnsArea` row.
|
||||
|
||||
---
|
||||
|
||||
## Result table
|
||||
|
||||
| # | Step | Result |
|
||||
|---|---|---|
|
||||
| 1 | Rig up all-`Direct`; deploy | **PASS** — `36e5cdd2` Sealed (Status 2), 6/6 Applied — baseline |
|
||||
| 2 | Flip site nodes to `FetchAndCache`, restart, deploy | **PASS after the fix** — `59399bd8` Sealed, 6/6 Applied; site logs *"Fetched + verified artifact … from central-1:4055 (13387 bytes)"* → *"applied fetched deployment"* |
|
||||
| 3 | Confirm central served it | **PASS** — central logs *"Served artifact for 59399bd8… (13387 bytes, 1 chunks)"*; site logs the *verified* (SHA-256-checked) reassembly |
|
||||
| 4 | **Stop central SQL** — driver independence + #485 | **PASS (headline)** — site nodes keep serving (`restarts=0`); a restarted site-b-1 boots its **last-applied** config (`59399bd8` / rev `00ac18b7`) from the LocalDb pointer with SQL down, no crash, never Stale |
|
||||
| 5 | Restart SQL, redeploy | **PASS** — retry lands green (`d6f71cc2` Sealed, 6/6) after one transient Direct-mode blip on central-1 (see finding B) |
|
||||
| 6 | Restart a site node, central up — boot from pointer | **PASS** — every FetchAndCache boot logs *"restored served state … from the LocalDb pointer"*, no `Deployment` read |
|
||||
| 7 | Restart a site node with central DOWN | **PASS** — folded into step 4 (site-b-1 restarted while SQL was stopped) |
|
||||
| 8 | Corrupt a cached chunk → boot Steady-no-revision | **NOT RE-RUN on rig** — covered by `DriverHostActorFetchAndCacheUnreadableTests` (corrupt cache ⇒ `GetCurrentUnkeyedAsync` null ⇒ Steady-no-revision, no partial apply). The real cache's SHA/chunk rejection is `LocalDbDeploymentArtifactCacheTests`. |
|
||||
| 9 | Wrong `ConfigSource:ApiKey` on one site → fetch rejected | **NOT RE-RUN on rig** — the real mechanism is proven by `DeploymentArtifactFetchBoundaryTests` (real Kestrel h2c + real interceptor + wrong key ⇒ null) and `ConfigServeAuthInterceptorTests` (fail-closed); the actor's response to a null fetch is step 4's evidence |
|
||||
| 10 | Both pair nodes fetch the same deploy | **PASS** — both site-a and both site-b nodes fetched independently in step 2; central logged a `Served artifact` per fetch; `StoreAsync` is idempotent |
|
||||
|
||||
---
|
||||
|
||||
## The defect the gate caught — deploy-sealing deadlock (fixed)
|
||||
|
||||
**Symptom.** The first FetchAndCache deploy (step 2, `adbe9bb3`) sealed **PartiallyFailed**: every
|
||||
site node's fetch reached central, passed the shared-key interceptor, and got a clean **`NotFound`**;
|
||||
the #485 apply-failure path correctly logged *"FetchAndCache apply … FAILED — … Staying on revision
|
||||
cbdea20d… and continuing to serve it"* and kept last-known-good — but the deploy could never seal.
|
||||
|
||||
**Root cause.** `DeploymentArtifactGrpcService` gated on `Status == Sealed`. But central seals a
|
||||
deployment **only after every node acks Applied**, and a FetchAndCache node **cannot ack until it has
|
||||
fetched** the artifact:
|
||||
|
||||
> seal-needs-ack → ack-needs-fetch → **fetch-needs-sealed** → deadlock.
|
||||
|
||||
Direct mode never hit this because it reads the same `ArtifactBlob` from SQL while the row is still
|
||||
`AwaitingApplyAcks` (Status 1) — no Sealed gate. Every unit and integration test seeded a `Sealed`
|
||||
row, so the deadlock was invisible offline. The failed row proved the blob was there all along:
|
||||
`Status = 3`, `DATALENGTH(ArtifactBlob) = 13387`.
|
||||
|
||||
**Fix** (`fix(mesh): serve the artifact regardless of deployment status`). Drop the Sealed gate —
|
||||
serve any deployment whose blob is non-empty; unknown-id / empty-blob still collapse to `NotFound`
|
||||
(existence-hiding + #485). Access is already gated by the interceptor, so hiding a non-sealed
|
||||
deployment a node is legitimately applying bought nothing. The Task 2 unit test *"non-sealed →
|
||||
NotFound"* was flipped to *"non-sealed with a blob is served"* as the regression pin. After the fix,
|
||||
step 2 re-ran green (`59399bd8` Sealed, 6/6).
|
||||
|
||||
**Lesson.** "Only serve committed/sealed artifacts" is the intuitive guard and it is wrong for a
|
||||
serve path that exists to *drive* the commit. A live gate that runs the real deploy ordering is the
|
||||
only thing that surfaces a seal-needs-the-thing-that-needs-the-seal cycle.
|
||||
|
||||
---
|
||||
|
||||
## Findings
|
||||
|
||||
- **A — the Kestrel takeover now runs on central, live, and coexists.** Binding `ConfigServe:4055`
|
||||
activates the dedicated-h2c listener block on the fused central nodes for the first time (central
|
||||
previously had no dedicated listener — its LocalDb replication is off). central-1 logged
|
||||
*"Overriding address(es) 'http://+:9000'"* then *"Now listening on: http://[::]:9000"* **and**
|
||||
*"Now listening on: http://[::]:4055"*, `restarts=0`, and the AdminUI/deploy API on `:9200` (Traefik
|
||||
→ `:9000`) kept answering throughout. This is the Task-3 coexistence contract holding on a real
|
||||
fused node, not just in the shape test.
|
||||
- **B — a transient Direct-mode SQL-reconnect blip is not a Phase 3 regression.** In step 5, the deploy
|
||||
immediately after SQL restarted sealed PartiallyFailed with **central-1** (Direct) failing its blob
|
||||
read (*"ConfigDb unreachable"*) — its EF pool still held broken connections. The #485 guard did
|
||||
exactly the right thing (apply-failure, keep last-known-good, ack Failed), and a redeploy sealed
|
||||
green. This is the pre-existing #486 behaviour firing on a reconnect edge, unrelated to FetchAndCache.
|
||||
- **C — endpoint failover works live.** site-b-1's step-5 fetch succeeded *from central-2*
|
||||
(`Fetched + verified … from http://central-2:4055`) — the fetcher rotated past central-1 to the
|
||||
second configured endpoint exactly as `GrpcDeploymentArtifactFetcher` intends.
|
||||
- **D — `Content-Type: application/json` is mandatory on the deploy POST.** Without it the minimal-API
|
||||
route 404s (not 415). The baseline call carried it; a later batch of retries dropped it and 404'd
|
||||
five times before the header was restored. Rig-operational, not a Phase 3 concern, but recorded so
|
||||
the next operator does not chase a phantom outage.
|
||||
|
||||
---
|
||||
|
||||
## Build note (rig enablement)
|
||||
|
||||
The rig image could not build until the Dockerfile's **build stage was pinned to `linux/amd64`**
|
||||
(`fix(docker-dev): pin the build stage to linux/amd64`). Phase 3 added the repo's first in-repo
|
||||
`.proto`, and `Grpc.Tools`' bundled `linux_arm64` `protoc` **segfaults (exit 139)** on Apple-Silicon
|
||||
Docker. `dotnet publish` is framework-dependent (portable IL + every RID's native assets), so pinning
|
||||
only the build stage leaves the runtime image native arm64; just the one-time build is emulated.
|
||||
Reference in New Issue
Block a user