docs(mesh): ConfigSource/ConfigServe config, rig flip switch, Phase 3 as shipped

Phase 3 Task 9. docker-dev rig: central-1/2 carry ConfigServe (:4055 h2c + committed
dev key) and stay Direct; the four site nodes carry ConfigSource (both central
endpoints + matching key) defaulting to Direct. Flip only the site nodes with
OTOPCUA_CONFIG_MODE=FetchAndCache at 'docker compose up' — central keeps SQL and models
the target topology. Binding :4055 activates the dedicated-h2c Kestrel takeover on
central (re-binds :9000 too). Docs: new docs/Configuration.md ConfigSource/ConfigServe
section; docs/Redundancy.md 'config bytes travel out-of-band' note; the program plan
Phase 3 section + tracking row flipped to CODE-COMPLETE; a CLAUDE.md 'Config source'
section.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW

CLAUDE.md

@@ -276,6 +276,45 @@ and `docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md`.
 - **There is no such thing as a cluster-scoped deployment.** `Deployment` has no `ClusterId`, `ConfigComposer` always snapshots the whole DB, and `DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The phase-1 plan asked for cluster-scope filtering of the ack set; it was dropped as describing a feature that does not exist.
 - `ClusterNode` gained `AkkaPort` (NOT NULL, 4053) + `GrpcPort` (nullable) as **central's dial targets** for Phases 2/5. They duplicate the node's own `Cluster:Port` / `Cluster:PublicHostname` with no schema-level enforcement, so `ClusterNodeAddressReconcilerActor` (admin singleton) logs an Error on drift. **Phase 2 must revisit it** — once the meshes split, an admin node cannot see site members and every site row would report `EnabledRowNotInCluster` forever.
 
+## Config source (`ConfigSource` / `ConfigServe`)
+
+Per-cluster mesh **Phase 3** (config fetch-and-cache) added a second way a **driver** node obtains its
+deployed configuration, selected by `ConfigSource:Mode`: `Direct` (read `Deployment.ArtifactBlob` from
+central SQL — **the default**, byte-for-byte today's behaviour) or `FetchAndCache` (fetch the artifact
+bytes from central over a dedicated-h2c **gRPC stream**, verify `SHA-256 == RevisionHash`, cache in
+LocalDb, and read **only** the cache — no central-SQL config read). Central (admin role) serves via
+`DeploymentArtifactService` behind `ConfigServeAuthInterceptor` on `ConfigServe:GrpcListenPort`, gated by
+a **shared node bearer key** (`ConfigServe:ApiKey == ConfigSource:ApiKey`, `FixedTimeEquals`,
+fail-closed). Things worth knowing before touching it:
+
+- **The serve side is wired in both modes** (mapped whenever `ConfigServe:GrpcListenPort > 0` on an
+  admin node), so flipping a node to `FetchAndCache` is a config change, not a redeploy — the same
+  dark-switch discipline as [MeshTransport](#mesh-command-transport-meshtransport).
+- **`RevisionHash == SHA-256(artifact bytes)` is load-bearing.** `ConfigComposer` computes the revision
+  hash as the lowercase-hex SHA-256 of the exact blob, so the node verifies a fetch against the
+  `RevisionHash` the dispatch already carries — **no hash rides in the proto, and `DispatchDeployment`
+  stays payload-free.** A future change to how the revision hash is derived silently breaks every fetch.
+- **A null fetch is "no answer," never an empty config (#485).** All endpoints down / `NotFound` / SHA
+  mismatch / zero-length ⇒ the fetcher returns null ⇒ the apply FAILS, the revision does not advance,
+  and the node keeps serving last-known-good. It never tears down to an empty address space. At boot a
+  `FetchAndCache` node restores served state from the LocalDb pointer with **no** central-SQL read; an
+  empty/unreadable cache lands Steady-with-no-revision (the first dispatch fetches), **never Stale**.
+- **Both pair nodes fetch independently** — no Primary gating of the fetch (same shape as both reading
+  SQL today; `StoreAsync` is idempotent and the cache replicates).
+- **The dedicated h2c listener coexists with the LocalDb-sync listener.** A fused admin+driver node with
+  both ports set re-binds its existing HTTP surface exactly **once** and adds both dedicated HTTP/2-only
+  ports — getting this wrong either double-binds (crash) or silently unbinds the AdminUI (the Kestrel
+  `Listen*`-discards-`ASPNETCORE_URLS` trap the LocalDb block already warned about).
+- **Phase boundary: config *reads* only.** A `FetchAndCache` node still writes its `NodeDeploymentState`
+  ack row to central SQL; `DbHealthProbe` and `EfAlarmConditionStateStore` still touch SQL. Cutting those
+  is Phase 4 — resist folding it in, or the "no SQL" scope becomes unprovable-in-parts.
+
+On the docker-dev rig the central pair carries `ConfigServe` (`:4055`, committed dev key) and stays
+`Direct`; the four site nodes carry `ConfigSource` (endpoints + matching key) defaulting to `Direct`.
+Flip only the site nodes with `OTOPCUA_CONFIG_MODE=FetchAndCache` at `docker compose up`. See
+`docs/Configuration.md` §`ConfigSource` / `ConfigServe`, `docs/Redundancy.md` §"Command transport", and
+`docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md`.
+
 ## LocalDb pair-local store (Phases 1 + 2)
 
 Every **driver-role** node keeps a consolidated `ZB.MOM.WW.LocalDb` SQLite database (the retired

docker-dev/docker-compose.yml

@@ -178,6 +178,14 @@ services:
       Cluster__Hostname: "0.0.0.0"
       Cluster__Port: "4053"
       Cluster__PublicHostname: "central-1"
+      # Per-cluster mesh Phase 3 (config fetch-and-cache). Central SERVES the deployed-configuration
+      # artifact to FetchAndCache site nodes over a dedicated h2c gRPC listener (:4055), gated by the
+      # shared node key. Central itself is ALWAYS Direct — it IS the SQL source and must never fetch.
+      # ConfigServe:ApiKey must be byte-identical to the site nodes' ConfigSource:ApiKey (fail-closed).
+      # Binding :4055 activates the dedicated-h2c Kestrel takeover on central (it re-binds :9000 too).
+      ConfigServe__GrpcListenPort: "4055"
+      ConfigServe__ApiKey: "configserve-docker-dev-key"
+      ConfigSource__Mode: "Direct"
       # Both redundancy peers are seeds (#459 / ScadaBridge parity) so a restarted node can
       # re-join the mesh via EITHER peer, not only central-1.
       #
@@ -269,6 +277,11 @@ services:
       Cluster__Hostname: "0.0.0.0"
       Cluster__Port: "4053"
       Cluster__PublicHostname: "central-2"
+      # Per-cluster mesh Phase 3 — serves the artifact to FetchAndCache site nodes (see central-1).
+      # Always Direct; ConfigServe:ApiKey must match the site nodes' ConfigSource:ApiKey.
+      ConfigServe__GrpcListenPort: "4055"
+      ConfigServe__ApiKey: "configserve-docker-dev-key"
+      ConfigSource__Mode: "Direct"
       # Both redundancy peers are seeds (#459 / ScadaBridge parity) — see central-1. SELF FIRST:
       # central-2 lists itself as seed-nodes[0], which is what lets it cold-start while central-1
       # is down (it previously listed central-1 first and simply never came Up in that case).
@@ -350,6 +363,15 @@ services:
       Cluster__Hostname: "0.0.0.0"
       Cluster__Port: "4053"
       Cluster__PublicHostname: "site-a-1"
+      # Per-cluster mesh Phase 3 dark switch. Default Direct (reads central SQL). Flip the site nodes to
+      # FetchAndCache with `OTOPCUA_CONFIG_MODE=FetchAndCache docker compose up`: they then fetch the
+      # deployed artifact from central over gRPC (endpoints below) and read ONLY the LocalDb cache — no
+      # central SQL config read. Central stays Direct, so this env var moves ONLY the four site nodes.
+      # ConfigSource:ApiKey must equal central's ConfigServe:ApiKey (the interceptor is fail-closed).
+      ConfigSource__Mode: "${OTOPCUA_CONFIG_MODE:-Direct}"
+      ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
+      ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
+      ConfigSource__ApiKey: "configserve-docker-dev-key"
       # Site nodes are deliberately NOT seeds — they join the central pair's mesh. The self-first
       # seed rule is conditional for exactly this reason (it binds only when a node's own address
       # is in its own seed list), so these configs are exempt rather than broken.
@@ -432,6 +454,12 @@ services:
       Cluster__Hostname: "0.0.0.0"
       Cluster__Port: "4053"
       Cluster__PublicHostname: "site-a-2"
+      # Per-cluster mesh Phase 3 dark switch (see site-a-1). Default Direct; OTOPCUA_CONFIG_MODE=FetchAndCache
+      # flips it to fetch the artifact from central over gRPC and read only the LocalDb cache.
+      ConfigSource__Mode: "${OTOPCUA_CONFIG_MODE:-Direct}"
+      ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
+      ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
+      ConfigSource__ApiKey: "configserve-docker-dev-key"
       Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
       Cluster__Roles__0: "driver"
       # Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
@@ -503,6 +531,12 @@ services:
       Cluster__Hostname: "0.0.0.0"
       Cluster__Port: "4053"
       Cluster__PublicHostname: "site-b-1"
+      # Per-cluster mesh Phase 3 dark switch (see site-a-1). Default Direct; OTOPCUA_CONFIG_MODE=FetchAndCache
+      # flips it to fetch the artifact from central over gRPC and read only the LocalDb cache.
+      ConfigSource__Mode: "${OTOPCUA_CONFIG_MODE:-Direct}"
+      ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
+      ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
+      ConfigSource__ApiKey: "configserve-docker-dev-key"
       Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
       Cluster__Roles__0: "driver"
       # Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
@@ -557,6 +591,12 @@ services:
       Cluster__Hostname: "0.0.0.0"
       Cluster__Port: "4053"
       Cluster__PublicHostname: "site-b-2"
+      # Per-cluster mesh Phase 3 dark switch (see site-a-1). Default Direct; OTOPCUA_CONFIG_MODE=FetchAndCache
+      # flips it to fetch the artifact from central over gRPC and read only the LocalDb cache.
+      ConfigSource__Mode: "${OTOPCUA_CONFIG_MODE:-Direct}"
+      ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
+      ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
+      ConfigSource__ApiKey: "configserve-docker-dev-key"
       Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
       Cluster__Roles__0: "driver"
       # Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",

docs/Configuration.md

@@ -133,6 +133,35 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only
 
 On the docker-dev rig every node carries these keys already; flip the whole rig with `OTOPCUA_MESH_MODE=ClusterClient` at `docker compose up`.
 
+### `ConfigSource` / `ConfigServe` (config fetch-and-cache)
+
+- **Purpose:** [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) **Phase 3**. `ConfigSource:Mode` selects where a **driver** node reads its deployed configuration: `Direct` (read `Deployment.ArtifactBlob` from central SQL, today's behaviour) or `FetchAndCache` (fetch the artifact bytes from central over a gRPC stream, verify SHA-256, cache in LocalDb, and read **only** the cache — no central-SQL config read). `ConfigServe` is the **central** (admin-role) serve side that a `FetchAndCache` node dials.
+- **Options classes:** `ConfigSourceOptions` (`SectionName = "ConfigSource"`) + `ConfigServeOptions` (`SectionName = "ConfigServe"`), both in `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`. Bound by `AddOtOpcUaCluster(config)`; `ConfigSourceOptionsValidator` runs at `ValidateOnStart`.
+
+**`ConfigSource` (driver-side fetch):**
+
+| Key | Type | Default | Meaning |
+|---|---|---|---|
+| `Mode` | string | `Direct` | `Direct` = read central SQL (unchanged). `FetchAndCache` = fetch from central over gRPC, read the LocalDb cache. Any other value **refuses to start**. |
+| `CentralFetchEndpoints` | string[] | `[]` | Central artifact-gRPC base addresses, e.g. `http://central-1:4055` (h2c ⇒ the `http` scheme). Tried in order for failover. **Required** under `FetchAndCache`. |
+| `ApiKey` | string | `""` | Shared bearer key; must equal central's `ConfigServe:ApiKey`. **Supply via env `ConfigSource__ApiKey` — never commit.** Required under `FetchAndCache`. |
+| `FetchTimeoutSeconds` | int | `30` | Per-fetch deadline. Must be positive under `FetchAndCache`. |
+
+**`ConfigServe` (central serve):**
+
+| Key | Type | Default | Meaning |
+|---|---|---|---|
+| `GrpcListenPort` | int | `0` | Dedicated **h2c-only** Kestrel port for the artifact gRPC service. `0` = disabled (nothing bound). Must differ from the main HTTP port and from `LocalDb:SyncListenPort` (h2c cannot share a cleartext HTTP/1 port). Mapped only on admin-role nodes. |
+| `ApiKey` | string | `""` | Shared bearer key the serve-side interceptor checks (constant-time, **fail-closed**: an unset key rejects every call). **Supply via env `ConfigServe__ApiKey` — never commit.** |
+
+**It is a dark switch; the serve side is always wired.** Central maps the `DeploymentArtifactService` behind the `ConfigServeAuthInterceptor` (path-scoped, `FixedTimeEquals`, fail-closed) whenever `ConfigServe:GrpcListenPort > 0`, in both modes — so flipping a node to `FetchAndCache` is a config change, not a redeploy. A `FetchAndCache` node that cannot fetch (all endpoints down, `NotFound`, SHA mismatch) **fails the apply and keeps last-known-good** — a null fetch is "no answer," never an empty configuration (the [#485](plans/2026-07-15-v3-batch4-address-space-plan.md) contract). At boot it restores served state from the LocalDb pointer with no SQL read; an empty or unreadable cache lands Steady-with-no-revision (the first dispatch fetches), never Stale.
+
+**The listener is dedicated h2c and coexists with the LocalDb-sync listener.** A fused admin+driver node with **both** `ConfigServe:GrpcListenPort` and `LocalDb:SyncListenPort` set re-binds its existing HTTP surface exactly once and adds both dedicated HTTP/2-only ports on top — see the same Kestrel-takeover caveat under [`LocalDb`](#localdb-pair-local-config-cache--driver-role-only).
+
+**`RevisionHash == SHA-256(artifact bytes)` is load-bearing.** `ConfigComposer` computes the deployment's revision hash as the lowercase-hex SHA-256 of the exact artifact bytes, so the node verifies a fetch against the `RevisionHash` the dispatch already carries — no hash rides in the proto. If a future change makes the revision hash anything else, every fetch fails verification.
+
+On the docker-dev rig the central pair carries `ConfigServe` (port `4055`, dev key) and stays `Direct`; the four site nodes carry `ConfigSource` (endpoints + matching key) defaulting to `Direct`. Flip only the site nodes with `OTOPCUA_CONFIG_MODE=FetchAndCache` at `docker compose up` — central keeps SQL and models the target topology.
+
 ### `ConnectionStrings` → `ConfigDb`
 
 - **Purpose:** the central Config DB connection string. **Required for every role** — `Program.cs` calls `AddOtOpcUaConfigDb` unconditionally.

docs/Redundancy.md

@@ -438,6 +438,19 @@ are untouched, and `redundancy-state` itself stays on DPS (it is bidirectional a
 `Cluster.State`, which makes it genuinely mesh-bound — the hardest remaining dependency before the
 meshes can split).
 
+**Config bytes travel out-of-band (Phase 3).** Commands stay on one of the two transports above, but
+the *configuration artifact itself* does not ride any Akka message — `DispatchDeployment` is
+payload-free (`DeploymentId` + `RevisionHash`). Under `ConfigSource:Mode = FetchAndCache` (see
+[Configuration.md § `ConfigSource`](Configuration.md#configsource--configserve-config-fetch-and-cache))
+a driver node fetches the bytes from central over a dedicated h2c **gRPC stream**, verifies
+`SHA-256 == RevisionHash`, and reads only its LocalDb cache — no central-SQL config read. **Both pair
+nodes fetch independently**, exactly as both read SQL today: there is no Primary gating of the fetch.
+That is safe because `StoreAsync` is idempotent and the cache replicates (pair replication carries the
+artifact to a node that missed its own fetch), and it keeps config delivery decoupled from redundancy
+timing. Central serves the bytes from its SQL row; a `FetchAndCache` node that cannot reach central
+fails the apply and keeps last-known-good (it never tears down to an empty address space), and it boots
+its served state from the LocalDb pointer with no central dependency at all.
+
 ## Pair-local store (LocalDb — Phases 1 + 2)
 
 Independently of the ServiceLevel machinery above, every **driver-role** node keeps a consolidated

docs/plans/2026-07-22-per-cluster-mesh-program.md

@@ -141,16 +141,30 @@ admin-change refresh), clusters know central from appsettings (static, restart t
 flow over ClusterClient (DPS paths deleted or dark-switched), including an Ask timing out cleanly
 against a stopped node.
 
-### Phase 3 — Config fetch-and-cache from central
-**Scope:** the deploy artifact is served **by central** (transport per its own phase plan — the
-ScadaBridge analogue is token-gated HTTP; decide HTTP vs a gRPC fetch RPC when planning) and
-cached in LocalDb; driver nodes read config exclusively from LocalDb (boot-from-cache becomes the
-normal path — design §6.1); chunking, SHA-256 verify, newest-2 retention, pair replication carry
-over unchanged. **This phase changes a running data path — it gets its own live gate** (design §7
-note): deploy lands on a driver pair with central SQL stopped mid-fetch → retry lands; #485
-last-known-good semantics re-proven on the new path.
-**Exit gate:** live gate green on the rig; a driver node with an empty LocalDb and reachable
-central boots into the current config; with central down it boots last-known-good.
+### Phase 3 — Config fetch-and-cache from central — **CODE-COMPLETE 2026-07-23 (Tasks 0–8; live gate Task 10 pending)**
+**Decisions taken:** the deploy artifact is served **by central over a gRPC fetch RPC**
+(`DeploymentArtifactService.Fetch`, streamed chunks — chosen over token-gated HTTP), authenticated by
+a **shared node bearer key** (`ConfigServe:ApiKey == ConfigSource:ApiKey`, `FixedTimeEquals`,
+fail-closed — chosen over a per-deployment token, so **no migration** and `DispatchDeployment` stays
+payload-free). Cutover is a **config-flagged dark switch** `ConfigSource:Mode` (`Direct` default |
+`FetchAndCache`), per node. Plan + task ledger:
+[`docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md`](2026-07-22-mesh-phase3-config-fetch-and-cache.md).
+**Shipped (all green, sabotage-checked):** `ConfigSource`/`ConfigServe` options + validator; the repo's
+first in-repo `.proto` + `Grpc.Tools` codegen; central `DeploymentArtifactGrpcService` (streams the
+sealed blob, NotFound-hides unknown/not-sealed/empty); `ConfigServeAuthInterceptor` + a dedicated h2c
+Kestrel listener merged with the LocalDb-sync listener (re-binds the existing surface once, adds both
+dedicated ports); node `GrpcDeploymentArtifactFetcher` (SHA-verified reassembly, endpoint failover,
+null-on-any-failure); the `FetchAndCache` apply path (`PipeTo(Self)`, fetch→cache→apply-from-bytes,
+null = apply-failure keeping last-known-good) and bootstrap-from-the-LocalDb-pointer (no driver-side
+config SQL read); #485 coverage; and a **real two-Host gRPC boundary test** (right-key reassembles,
+wrong-key control returns null, failover). Chunking, SHA-256 verify, newest-2 retention, pair
+replication carry over unchanged. **Phase boundary held:** config **reads** only — the
+`NodeDeploymentState` ack-row write, `DbHealthProbe`, `EfAlarmConditionStateStore` stay for Phase 4.
+**Remaining (Tasks 9–10):** rig config + docs (done) and the live gate — deploy on a site pair flipped
+to `FetchAndCache` with central SQL stopped mid-fetch → retry lands; #485 last-known-good re-proven on
+the new path.
+**Exit gate:** live gate green on the rig; a driver node with an empty LocalDb and reachable central
+boots into the current config; with central down it boots last-known-good.
 
 ### Phase 4 — Cut the driver-side ConfigDb connection
 **Scope (design §6.1 audit table):** re-home `EfAlarmConditionStateStore` to LocalDb (pair-local
@@ -229,7 +243,7 @@ resource sizing on the site VMs should be checked once both products run the ful
 | Prereq: selfform-fallback + manual-failover plan | **DONE 2026-07-22**, merged `a78425ea`. Shipped *not* as the planned `SelfFormAfter` watchdog — the live gate caught that islanding a manually-failed-over node — but as self-first seed ordering + `AkkaClusterOptionsValidator`. Phase 6's seed-ordering scope is therefore already discharged. |
 | 1 ClusterNode columns + DB ack set | **DONE 2026-07-22**, merged `7654f24d`. Deploy-seal semantics changed as flagged, but the escape hatch is **`ClusterNode.MaintenanceMode`**, not `Enabled=0` — the live gate found `Enabled=0` is rejected by `DraftValidator.ValidateClusterTopology` on any 2-node pair. Gate record: `2026-07-22-mesh-phase1-live-gate.md`. |
 | 2 ClusterClient transport | **DONE 2026-07-22** (live gate PASSED, shipped dark) — shipped dark (`MeshTransport:Mode=Dps` default), both comm actors registered in both modes. Corrections: `SendToAll` not `Send`, one fleet-wide client not one per cluster, `/user/node-communication` not `/user/cluster-communication`. The gate's "Ask timing out" item does not exist to test — no cross-boundary Ask in this phase; see the phase section. |
-| 3 fetch-and-cache | not started |
+| 3 fetch-and-cache | **CODE-COMPLETE 2026-07-23** (Tasks 0–8 merged to the phase branch, all green + sabotage-checked; live gate Task 10 pending). gRPC fetch RPC + shared node key; dark switch `ConfigSource:Mode` (Direct default). Rig flip: `OTOPCUA_CONFIG_MODE=FetchAndCache` on the site nodes. See `2026-07-22-mesh-phase3-config-fetch-and-cache.md`. |
 | 4 cut driver ConfigDb | not started |
 | 5 gRPC telemetry | not started |
 | 6 mesh partition + co-location | not started |