From cf6110d02c2dec1cecd0a93a98ac6302e82e367e Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Wed, 22 Jul 2026 20:06:23 -0400 Subject: [PATCH] docs(mesh): ConfigSource/ConfigServe config, rig flip switch, Phase 3 as shipped MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 3 Task 9. docker-dev rig: central-1/2 carry ConfigServe (:4055 h2c + committed dev key) and stay Direct; the four site nodes carry ConfigSource (both central endpoints + matching key) defaulting to Direct. Flip only the site nodes with OTOPCUA_CONFIG_MODE=FetchAndCache at 'docker compose up' — central keeps SQL and models the target topology. Binding :4055 activates the dedicated-h2c Kestrel takeover on central (re-binds :9000 too). Docs: new docs/Configuration.md ConfigSource/ConfigServe section; docs/Redundancy.md 'config bytes travel out-of-band' note; the program plan Phase 3 section + tracking row flipped to CODE-COMPLETE; a CLAUDE.md 'Config source' section. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- CLAUDE.md | 39 ++++++++++++++++++ docker-dev/docker-compose.yml | 40 +++++++++++++++++++ docs/Configuration.md | 29 ++++++++++++++ docs/Redundancy.md | 13 ++++++ .../2026-07-22-per-cluster-mesh-program.md | 36 ++++++++++++----- 5 files changed, 146 insertions(+), 11 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 160fbb52..307053f1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -276,6 +276,45 @@ and `docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md`. - **There is no such thing as a cluster-scoped deployment.** `Deployment` has no `ClusterId`, `ConfigComposer` always snapshots the whole DB, and `DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The phase-1 plan asked for cluster-scope filtering of the ack set; it was dropped as describing a feature that does not exist. - `ClusterNode` gained `AkkaPort` (NOT NULL, 4053) + `GrpcPort` (nullable) as **central's dial targets** for Phases 2/5. They duplicate the node's own `Cluster:Port` / `Cluster:PublicHostname` with no schema-level enforcement, so `ClusterNodeAddressReconcilerActor` (admin singleton) logs an Error on drift. **Phase 2 must revisit it** — once the meshes split, an admin node cannot see site members and every site row would report `EnabledRowNotInCluster` forever. +## Config source (`ConfigSource` / `ConfigServe`) + +Per-cluster mesh **Phase 3** (config fetch-and-cache) added a second way a **driver** node obtains its +deployed configuration, selected by `ConfigSource:Mode`: `Direct` (read `Deployment.ArtifactBlob` from +central SQL — **the default**, byte-for-byte today's behaviour) or `FetchAndCache` (fetch the artifact +bytes from central over a dedicated-h2c **gRPC stream**, verify `SHA-256 == RevisionHash`, cache in +LocalDb, and read **only** the cache — no central-SQL config read). Central (admin role) serves via +`DeploymentArtifactService` behind `ConfigServeAuthInterceptor` on `ConfigServe:GrpcListenPort`, gated by +a **shared node bearer key** (`ConfigServe:ApiKey == ConfigSource:ApiKey`, `FixedTimeEquals`, +fail-closed). Things worth knowing before touching it: + +- **The serve side is wired in both modes** (mapped whenever `ConfigServe:GrpcListenPort > 0` on an + admin node), so flipping a node to `FetchAndCache` is a config change, not a redeploy — the same + dark-switch discipline as [MeshTransport](#mesh-command-transport-meshtransport). +- **`RevisionHash == SHA-256(artifact bytes)` is load-bearing.** `ConfigComposer` computes the revision + hash as the lowercase-hex SHA-256 of the exact blob, so the node verifies a fetch against the + `RevisionHash` the dispatch already carries — **no hash rides in the proto, and `DispatchDeployment` + stays payload-free.** A future change to how the revision hash is derived silently breaks every fetch. +- **A null fetch is "no answer," never an empty config (#485).** All endpoints down / `NotFound` / SHA + mismatch / zero-length ⇒ the fetcher returns null ⇒ the apply FAILS, the revision does not advance, + and the node keeps serving last-known-good. It never tears down to an empty address space. At boot a + `FetchAndCache` node restores served state from the LocalDb pointer with **no** central-SQL read; an + empty/unreadable cache lands Steady-with-no-revision (the first dispatch fetches), **never Stale**. +- **Both pair nodes fetch independently** — no Primary gating of the fetch (same shape as both reading + SQL today; `StoreAsync` is idempotent and the cache replicates). +- **The dedicated h2c listener coexists with the LocalDb-sync listener.** A fused admin+driver node with + both ports set re-binds its existing HTTP surface exactly **once** and adds both dedicated HTTP/2-only + ports — getting this wrong either double-binds (crash) or silently unbinds the AdminUI (the Kestrel + `Listen*`-discards-`ASPNETCORE_URLS` trap the LocalDb block already warned about). +- **Phase boundary: config *reads* only.** A `FetchAndCache` node still writes its `NodeDeploymentState` + ack row to central SQL; `DbHealthProbe` and `EfAlarmConditionStateStore` still touch SQL. Cutting those + is Phase 4 — resist folding it in, or the "no SQL" scope becomes unprovable-in-parts. + +On the docker-dev rig the central pair carries `ConfigServe` (`:4055`, committed dev key) and stays +`Direct`; the four site nodes carry `ConfigSource` (endpoints + matching key) defaulting to `Direct`. +Flip only the site nodes with `OTOPCUA_CONFIG_MODE=FetchAndCache` at `docker compose up`. See +`docs/Configuration.md` §`ConfigSource` / `ConfigServe`, `docs/Redundancy.md` §"Command transport", and +`docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md`. + ## LocalDb pair-local store (Phases 1 + 2) Every **driver-role** node keeps a consolidated `ZB.MOM.WW.LocalDb` SQLite database (the retired diff --git a/docker-dev/docker-compose.yml b/docker-dev/docker-compose.yml index 71042130..18ac240d 100644 --- a/docker-dev/docker-compose.yml +++ b/docker-dev/docker-compose.yml @@ -178,6 +178,14 @@ services: Cluster__Hostname: "0.0.0.0" Cluster__Port: "4053" Cluster__PublicHostname: "central-1" + # Per-cluster mesh Phase 3 (config fetch-and-cache). Central SERVES the deployed-configuration + # artifact to FetchAndCache site nodes over a dedicated h2c gRPC listener (:4055), gated by the + # shared node key. Central itself is ALWAYS Direct — it IS the SQL source and must never fetch. + # ConfigServe:ApiKey must be byte-identical to the site nodes' ConfigSource:ApiKey (fail-closed). + # Binding :4055 activates the dedicated-h2c Kestrel takeover on central (it re-binds :9000 too). + ConfigServe__GrpcListenPort: "4055" + ConfigServe__ApiKey: "configserve-docker-dev-key" + ConfigSource__Mode: "Direct" # Both redundancy peers are seeds (#459 / ScadaBridge parity) so a restarted node can # re-join the mesh via EITHER peer, not only central-1. # @@ -269,6 +277,11 @@ services: Cluster__Hostname: "0.0.0.0" Cluster__Port: "4053" Cluster__PublicHostname: "central-2" + # Per-cluster mesh Phase 3 — serves the artifact to FetchAndCache site nodes (see central-1). + # Always Direct; ConfigServe:ApiKey must match the site nodes' ConfigSource:ApiKey. + ConfigServe__GrpcListenPort: "4055" + ConfigServe__ApiKey: "configserve-docker-dev-key" + ConfigSource__Mode: "Direct" # Both redundancy peers are seeds (#459 / ScadaBridge parity) — see central-1. SELF FIRST: # central-2 lists itself as seed-nodes[0], which is what lets it cold-start while central-1 # is down (it previously listed central-1 first and simply never came Up in that case). @@ -350,6 +363,15 @@ services: Cluster__Hostname: "0.0.0.0" Cluster__Port: "4053" Cluster__PublicHostname: "site-a-1" + # Per-cluster mesh Phase 3 dark switch. Default Direct (reads central SQL). Flip the site nodes to + # FetchAndCache with `OTOPCUA_CONFIG_MODE=FetchAndCache docker compose up`: they then fetch the + # deployed artifact from central over gRPC (endpoints below) and read ONLY the LocalDb cache — no + # central SQL config read. Central stays Direct, so this env var moves ONLY the four site nodes. + # ConfigSource:ApiKey must equal central's ConfigServe:ApiKey (the interceptor is fail-closed). + ConfigSource__Mode: "${OTOPCUA_CONFIG_MODE:-Direct}" + ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055" + ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055" + ConfigSource__ApiKey: "configserve-docker-dev-key" # Site nodes are deliberately NOT seeds — they join the central pair's mesh. The self-first # seed rule is conditional for exactly this reason (it binds only when a node's own address # is in its own seed list), so these configs are exempt rather than broken. @@ -432,6 +454,12 @@ services: Cluster__Hostname: "0.0.0.0" Cluster__Port: "4053" Cluster__PublicHostname: "site-a-2" + # Per-cluster mesh Phase 3 dark switch (see site-a-1). Default Direct; OTOPCUA_CONFIG_MODE=FetchAndCache + # flips it to fetch the artifact from central over gRPC and read only the LocalDb cache. + ConfigSource__Mode: "${OTOPCUA_CONFIG_MODE:-Direct}" + ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055" + ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055" + ConfigSource__ApiKey: "configserve-docker-dev-key" Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053" Cluster__Roles__0: "driver" # Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps", @@ -503,6 +531,12 @@ services: Cluster__Hostname: "0.0.0.0" Cluster__Port: "4053" Cluster__PublicHostname: "site-b-1" + # Per-cluster mesh Phase 3 dark switch (see site-a-1). Default Direct; OTOPCUA_CONFIG_MODE=FetchAndCache + # flips it to fetch the artifact from central over gRPC and read only the LocalDb cache. + ConfigSource__Mode: "${OTOPCUA_CONFIG_MODE:-Direct}" + ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055" + ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055" + ConfigSource__ApiKey: "configserve-docker-dev-key" Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053" Cluster__Roles__0: "driver" # Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps", @@ -557,6 +591,12 @@ services: Cluster__Hostname: "0.0.0.0" Cluster__Port: "4053" Cluster__PublicHostname: "site-b-2" + # Per-cluster mesh Phase 3 dark switch (see site-a-1). Default Direct; OTOPCUA_CONFIG_MODE=FetchAndCache + # flips it to fetch the artifact from central over gRPC and read only the LocalDb cache. + ConfigSource__Mode: "${OTOPCUA_CONFIG_MODE:-Direct}" + ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055" + ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055" + ConfigSource__ApiKey: "configserve-docker-dev-key" Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053" Cluster__Roles__0: "driver" # Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps", diff --git a/docs/Configuration.md b/docs/Configuration.md index b92ba78b..4fd14d89 100644 --- a/docs/Configuration.md +++ b/docs/Configuration.md @@ -133,6 +133,35 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only On the docker-dev rig every node carries these keys already; flip the whole rig with `OTOPCUA_MESH_MODE=ClusterClient` at `docker compose up`. +### `ConfigSource` / `ConfigServe` (config fetch-and-cache) + +- **Purpose:** [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) **Phase 3**. `ConfigSource:Mode` selects where a **driver** node reads its deployed configuration: `Direct` (read `Deployment.ArtifactBlob` from central SQL, today's behaviour) or `FetchAndCache` (fetch the artifact bytes from central over a gRPC stream, verify SHA-256, cache in LocalDb, and read **only** the cache — no central-SQL config read). `ConfigServe` is the **central** (admin-role) serve side that a `FetchAndCache` node dials. +- **Options classes:** `ConfigSourceOptions` (`SectionName = "ConfigSource"`) + `ConfigServeOptions` (`SectionName = "ConfigServe"`), both in `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`. Bound by `AddOtOpcUaCluster(config)`; `ConfigSourceOptionsValidator` runs at `ValidateOnStart`. + +**`ConfigSource` (driver-side fetch):** + +| Key | Type | Default | Meaning | +|---|---|---|---| +| `Mode` | string | `Direct` | `Direct` = read central SQL (unchanged). `FetchAndCache` = fetch from central over gRPC, read the LocalDb cache. Any other value **refuses to start**. | +| `CentralFetchEndpoints` | string[] | `[]` | Central artifact-gRPC base addresses, e.g. `http://central-1:4055` (h2c ⇒ the `http` scheme). Tried in order for failover. **Required** under `FetchAndCache`. | +| `ApiKey` | string | `""` | Shared bearer key; must equal central's `ConfigServe:ApiKey`. **Supply via env `ConfigSource__ApiKey` — never commit.** Required under `FetchAndCache`. | +| `FetchTimeoutSeconds` | int | `30` | Per-fetch deadline. Must be positive under `FetchAndCache`. | + +**`ConfigServe` (central serve):** + +| Key | Type | Default | Meaning | +|---|---|---|---| +| `GrpcListenPort` | int | `0` | Dedicated **h2c-only** Kestrel port for the artifact gRPC service. `0` = disabled (nothing bound). Must differ from the main HTTP port and from `LocalDb:SyncListenPort` (h2c cannot share a cleartext HTTP/1 port). Mapped only on admin-role nodes. | +| `ApiKey` | string | `""` | Shared bearer key the serve-side interceptor checks (constant-time, **fail-closed**: an unset key rejects every call). **Supply via env `ConfigServe__ApiKey` — never commit.** | + +**It is a dark switch; the serve side is always wired.** Central maps the `DeploymentArtifactService` behind the `ConfigServeAuthInterceptor` (path-scoped, `FixedTimeEquals`, fail-closed) whenever `ConfigServe:GrpcListenPort > 0`, in both modes — so flipping a node to `FetchAndCache` is a config change, not a redeploy. A `FetchAndCache` node that cannot fetch (all endpoints down, `NotFound`, SHA mismatch) **fails the apply and keeps last-known-good** — a null fetch is "no answer," never an empty configuration (the [#485](plans/2026-07-15-v3-batch4-address-space-plan.md) contract). At boot it restores served state from the LocalDb pointer with no SQL read; an empty or unreadable cache lands Steady-with-no-revision (the first dispatch fetches), never Stale. + +**The listener is dedicated h2c and coexists with the LocalDb-sync listener.** A fused admin+driver node with **both** `ConfigServe:GrpcListenPort` and `LocalDb:SyncListenPort` set re-binds its existing HTTP surface exactly once and adds both dedicated HTTP/2-only ports on top — see the same Kestrel-takeover caveat under [`LocalDb`](#localdb-pair-local-config-cache--driver-role-only). + +**`RevisionHash == SHA-256(artifact bytes)` is load-bearing.** `ConfigComposer` computes the deployment's revision hash as the lowercase-hex SHA-256 of the exact artifact bytes, so the node verifies a fetch against the `RevisionHash` the dispatch already carries — no hash rides in the proto. If a future change makes the revision hash anything else, every fetch fails verification. + +On the docker-dev rig the central pair carries `ConfigServe` (port `4055`, dev key) and stays `Direct`; the four site nodes carry `ConfigSource` (endpoints + matching key) defaulting to `Direct`. Flip only the site nodes with `OTOPCUA_CONFIG_MODE=FetchAndCache` at `docker compose up` — central keeps SQL and models the target topology. + ### `ConnectionStrings` → `ConfigDb` - **Purpose:** the central Config DB connection string. **Required for every role** — `Program.cs` calls `AddOtOpcUaConfigDb` unconditionally. diff --git a/docs/Redundancy.md b/docs/Redundancy.md index c08d35db..cacc4bd7 100644 --- a/docs/Redundancy.md +++ b/docs/Redundancy.md @@ -438,6 +438,19 @@ are untouched, and `redundancy-state` itself stays on DPS (it is bidirectional a `Cluster.State`, which makes it genuinely mesh-bound — the hardest remaining dependency before the meshes can split). +**Config bytes travel out-of-band (Phase 3).** Commands stay on one of the two transports above, but +the *configuration artifact itself* does not ride any Akka message — `DispatchDeployment` is +payload-free (`DeploymentId` + `RevisionHash`). Under `ConfigSource:Mode = FetchAndCache` (see +[Configuration.md § `ConfigSource`](Configuration.md#configsource--configserve-config-fetch-and-cache)) +a driver node fetches the bytes from central over a dedicated h2c **gRPC stream**, verifies +`SHA-256 == RevisionHash`, and reads only its LocalDb cache — no central-SQL config read. **Both pair +nodes fetch independently**, exactly as both read SQL today: there is no Primary gating of the fetch. +That is safe because `StoreAsync` is idempotent and the cache replicates (pair replication carries the +artifact to a node that missed its own fetch), and it keeps config delivery decoupled from redundancy +timing. Central serves the bytes from its SQL row; a `FetchAndCache` node that cannot reach central +fails the apply and keeps last-known-good (it never tears down to an empty address space), and it boots +its served state from the LocalDb pointer with no central dependency at all. + ## Pair-local store (LocalDb — Phases 1 + 2) Independently of the ServiceLevel machinery above, every **driver-role** node keeps a consolidated diff --git a/docs/plans/2026-07-22-per-cluster-mesh-program.md b/docs/plans/2026-07-22-per-cluster-mesh-program.md index 70218e14..ebc5e524 100644 --- a/docs/plans/2026-07-22-per-cluster-mesh-program.md +++ b/docs/plans/2026-07-22-per-cluster-mesh-program.md @@ -141,16 +141,30 @@ admin-change refresh), clusters know central from appsettings (static, restart t flow over ClusterClient (DPS paths deleted or dark-switched), including an Ask timing out cleanly against a stopped node. -### Phase 3 — Config fetch-and-cache from central -**Scope:** the deploy artifact is served **by central** (transport per its own phase plan — the -ScadaBridge analogue is token-gated HTTP; decide HTTP vs a gRPC fetch RPC when planning) and -cached in LocalDb; driver nodes read config exclusively from LocalDb (boot-from-cache becomes the -normal path — design §6.1); chunking, SHA-256 verify, newest-2 retention, pair replication carry -over unchanged. **This phase changes a running data path — it gets its own live gate** (design §7 -note): deploy lands on a driver pair with central SQL stopped mid-fetch → retry lands; #485 -last-known-good semantics re-proven on the new path. -**Exit gate:** live gate green on the rig; a driver node with an empty LocalDb and reachable -central boots into the current config; with central down it boots last-known-good. +### Phase 3 — Config fetch-and-cache from central — **CODE-COMPLETE 2026-07-23 (Tasks 0–8; live gate Task 10 pending)** +**Decisions taken:** the deploy artifact is served **by central over a gRPC fetch RPC** +(`DeploymentArtifactService.Fetch`, streamed chunks — chosen over token-gated HTTP), authenticated by +a **shared node bearer key** (`ConfigServe:ApiKey == ConfigSource:ApiKey`, `FixedTimeEquals`, +fail-closed — chosen over a per-deployment token, so **no migration** and `DispatchDeployment` stays +payload-free). Cutover is a **config-flagged dark switch** `ConfigSource:Mode` (`Direct` default | +`FetchAndCache`), per node. Plan + task ledger: +[`docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md`](2026-07-22-mesh-phase3-config-fetch-and-cache.md). +**Shipped (all green, sabotage-checked):** `ConfigSource`/`ConfigServe` options + validator; the repo's +first in-repo `.proto` + `Grpc.Tools` codegen; central `DeploymentArtifactGrpcService` (streams the +sealed blob, NotFound-hides unknown/not-sealed/empty); `ConfigServeAuthInterceptor` + a dedicated h2c +Kestrel listener merged with the LocalDb-sync listener (re-binds the existing surface once, adds both +dedicated ports); node `GrpcDeploymentArtifactFetcher` (SHA-verified reassembly, endpoint failover, +null-on-any-failure); the `FetchAndCache` apply path (`PipeTo(Self)`, fetch→cache→apply-from-bytes, +null = apply-failure keeping last-known-good) and bootstrap-from-the-LocalDb-pointer (no driver-side +config SQL read); #485 coverage; and a **real two-Host gRPC boundary test** (right-key reassembles, +wrong-key control returns null, failover). Chunking, SHA-256 verify, newest-2 retention, pair +replication carry over unchanged. **Phase boundary held:** config **reads** only — the +`NodeDeploymentState` ack-row write, `DbHealthProbe`, `EfAlarmConditionStateStore` stay for Phase 4. +**Remaining (Tasks 9–10):** rig config + docs (done) and the live gate — deploy on a site pair flipped +to `FetchAndCache` with central SQL stopped mid-fetch → retry lands; #485 last-known-good re-proven on +the new path. +**Exit gate:** live gate green on the rig; a driver node with an empty LocalDb and reachable central +boots into the current config; with central down it boots last-known-good. ### Phase 4 — Cut the driver-side ConfigDb connection **Scope (design §6.1 audit table):** re-home `EfAlarmConditionStateStore` to LocalDb (pair-local @@ -229,7 +243,7 @@ resource sizing on the site VMs should be checked once both products run the ful | Prereq: selfform-fallback + manual-failover plan | **DONE 2026-07-22**, merged `a78425ea`. Shipped *not* as the planned `SelfFormAfter` watchdog — the live gate caught that islanding a manually-failed-over node — but as self-first seed ordering + `AkkaClusterOptionsValidator`. Phase 6's seed-ordering scope is therefore already discharged. | | 1 ClusterNode columns + DB ack set | **DONE 2026-07-22**, merged `7654f24d`. Deploy-seal semantics changed as flagged, but the escape hatch is **`ClusterNode.MaintenanceMode`**, not `Enabled=0` — the live gate found `Enabled=0` is rejected by `DraftValidator.ValidateClusterTopology` on any 2-node pair. Gate record: `2026-07-22-mesh-phase1-live-gate.md`. | | 2 ClusterClient transport | **DONE 2026-07-22** (live gate PASSED, shipped dark) — shipped dark (`MeshTransport:Mode=Dps` default), both comm actors registered in both modes. Corrections: `SendToAll` not `Send`, one fleet-wide client not one per cluster, `/user/node-communication` not `/user/cluster-communication`. The gate's "Ask timing out" item does not exist to test — no cross-boundary Ask in this phase; see the phase section. | -| 3 fetch-and-cache | not started | +| 3 fetch-and-cache | **CODE-COMPLETE 2026-07-23** (Tasks 0–8 merged to the phase branch, all green + sabotage-checked; live gate Task 10 pending). gRPC fetch RPC + shared node key; dark switch `ConfigSource:Mode` (Direct default). Rig flip: `OTOPCUA_CONFIG_MODE=FetchAndCache` on the site nodes. See `2026-07-22-mesh-phase3-config-fetch-and-cache.md`. | | 4 cut driver ConfigDb | not started | | 5 gRPC telemetry | not started | | 6 mesh partition + co-location | not started |