dohertj2/lmxopcua
1
0
Fork 0
Code Issues 9 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs(mesh): ConfigSource/ConfigServe config, rig flip switch, Phase 3 as shipped

Browse Source 
Phase 3 Task 9. docker-dev rig: central-1/2 carry ConfigServe (:4055 h2c + committed
dev key) and stay Direct; the four site nodes carry ConfigSource (both central
endpoints + matching key) defaulting to Direct. Flip only the site nodes with
OTOPCUA_CONFIG_MODE=FetchAndCache at 'docker compose up' — central keeps SQL and models
the target topology. Binding :4055 activates the dedicated-h2c Kestrel takeover on
central (re-binds :9000 too). Docs: new docs/Configuration.md ConfigSource/ConfigServe
section; docs/Redundancy.md 'config bytes travel out-of-band' note; the program plan
Phase 3 section + tracking row flipped to CODE-COMPLETE; a CLAUDE.md 'Config source'
section.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-22 20:06:23 -04:00
parent b3210686e2
commit cf6110d02c
5 changed files with 146 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Configuration.md
+29
View File
@@ -133,6 +133,35 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only
On the docker-dev rig every node carries these keys already; flip the whole rig with `OTOPCUA_MESH_MODE=ClusterClient` at `docker compose up`.
### `ConfigSource` / `ConfigServe` (config fetch-and-cache)
- **Purpose:** [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) **Phase 3**. `ConfigSource:Mode` selects where a **driver** node reads its deployed configuration: `Direct` (read `Deployment.ArtifactBlob` from central SQL, today's behaviour) or `FetchAndCache` (fetch the artifact bytes from central over a gRPC stream, verify SHA-256, cache in LocalDb, and read **only** the cache — no central-SQL config read). `ConfigServe` is the **central** (admin-role) serve side that a `FetchAndCache` node dials.
- **Options classes:** `ConfigSourceOptions` (`SectionName = "ConfigSource"`) + `ConfigServeOptions` (`SectionName = "ConfigServe"`), both in `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`. Bound by `AddOtOpcUaCluster(config)`; `ConfigSourceOptionsValidator` runs at `ValidateOnStart`.
**`ConfigSource` (driver-side fetch):**
| Key | Type | Default | Meaning |
|---|---|---|---|
| `Mode` | string | `Direct` | `Direct` = read central SQL (unchanged). `FetchAndCache` = fetch from central over gRPC, read the LocalDb cache. Any other value **refuses to start**. |
| `CentralFetchEndpoints` | string[] | `[]` | Central artifact-gRPC base addresses, e.g. `http://central-1:4055` (h2c ⇒ the `http` scheme). Tried in order for failover. **Required** under `FetchAndCache`. |
| `ApiKey` | string | `""` | Shared bearer key; must equal central's `ConfigServe:ApiKey`. **Supply via env `ConfigSource__ApiKey` — never commit.** Required under `FetchAndCache`. |
| `FetchTimeoutSeconds` | int | `30` | Per-fetch deadline. Must be positive under `FetchAndCache`. |
**`ConfigServe` (central serve):**
| Key | Type | Default | Meaning |
|---|---|---|---|
| `GrpcListenPort` | int | `0` | Dedicated **h2c-only** Kestrel port for the artifact gRPC service. `0` = disabled (nothing bound). Must differ from the main HTTP port and from `LocalDb:SyncListenPort` (h2c cannot share a cleartext HTTP/1 port). Mapped only on admin-role nodes. |
| `ApiKey` | string | `""` | Shared bearer key the serve-side interceptor checks (constant-time, **fail-closed**: an unset key rejects every call). **Supply via env `ConfigServe__ApiKey` — never commit.** |
**It is a dark switch; the serve side is always wired.** Central maps the `DeploymentArtifactService` behind the `ConfigServeAuthInterceptor` (path-scoped, `FixedTimeEquals`, fail-closed) whenever `ConfigServe:GrpcListenPort > 0`, in both modes — so flipping a node to `FetchAndCache` is a config change, not a redeploy. A `FetchAndCache` node that cannot fetch (all endpoints down, `NotFound`, SHA mismatch) **fails the apply and keeps last-known-good** — a null fetch is "no answer," never an empty configuration (the [#485](plans/2026-07-15-v3-batch4-address-space-plan.md) contract). At boot it restores served state from the LocalDb pointer with no SQL read; an empty or unreadable cache lands Steady-with-no-revision (the first dispatch fetches), never Stale.
**The listener is dedicated h2c and coexists with the LocalDb-sync listener.** A fused admin+driver node with **both** `ConfigServe:GrpcListenPort` and `LocalDb:SyncListenPort` set re-binds its existing HTTP surface exactly once and adds both dedicated HTTP/2-only ports on top — see the same Kestrel-takeover caveat under [`LocalDb`](#localdb-pair-local-config-cache--driver-role-only).
**`RevisionHash == SHA-256(artifact bytes)` is load-bearing.** `ConfigComposer` computes the deployment's revision hash as the lowercase-hex SHA-256 of the exact artifact bytes, so the node verifies a fetch against the `RevisionHash` the dispatch already carries — no hash rides in the proto. If a future change makes the revision hash anything else, every fetch fails verification.
On the docker-dev rig the central pair carries `ConfigServe` (port `4055`, dev key) and stays `Direct`; the four site nodes carry `ConfigSource` (endpoints + matching key) defaulting to `Direct`. Flip only the site nodes with `OTOPCUA_CONFIG_MODE=FetchAndCache` at `docker compose up` — central keeps SQL and models the target topology.
### `ConnectionStrings` → `ConfigDb`
- **Purpose:** the central Config DB connection string. **Required for every role** — `Program.cs` calls `AddOtOpcUaConfigDb` unconditionally.
docs/Redundancy.md
+13
View File
@@ -438,6 +438,19 @@ are untouched, and `redundancy-state` itself stays on DPS (it is bidirectional a
`Cluster.State`, which makes it genuinely mesh-bound — the hardest remaining dependency before the
meshes can split).
**Config bytes travel out-of-band (Phase 3).** Commands stay on one of the two transports above, but
the *configuration artifact itself* does not ride any Akka message — `DispatchDeployment` is
payload-free (`DeploymentId` + `RevisionHash`). Under `ConfigSource:Mode = FetchAndCache` (see
[Configuration.md § `ConfigSource`](Configuration.md#configsource--configserve-config-fetch-and-cache))
a driver node fetches the bytes from central over a dedicated h2c **gRPC stream**, verifies
`SHA-256 == RevisionHash`, and reads only its LocalDb cache — no central-SQL config read. **Both pair
nodes fetch independently**, exactly as both read SQL today: there is no Primary gating of the fetch.
That is safe because `StoreAsync` is idempotent and the cache replicates (pair replication carries the
artifact to a node that missed its own fetch), and it keeps config delivery decoupled from redundancy
timing. Central serves the bytes from its SQL row; a `FetchAndCache` node that cannot reach central
fails the apply and keeps last-known-good (it never tears down to an empty address space), and it boots
its served state from the LocalDb pointer with no central dependency at all.
## Pair-local store (LocalDb — Phases 1 + 2)
Independently of the ServiceLevel machinery above, every **driver-role** node keeps a consolidated
docs/plans/2026-07-22-per-cluster-mesh-program.md
+25 -11
View File
@@ -141,16 +141,30 @@ admin-change refresh), clusters know central from appsettings (static, restart t
flow over ClusterClient (DPS paths deleted or dark-switched), including an Ask timing out cleanly
against a stopped node.
### Phase 3 — Config fetch-and-cache from central
**Scope:** the deploy artifact is served **by central** (transport per its own phase plan — the
ScadaBridge analogue is token-gated HTTP; decide HTTP vs a gRPC fetch RPC when planning) and
cached in LocalDb; driver nodes read config exclusively from LocalDb (boot-from-cache becomes the
normal path — design §6.1); chunking, SHA-256 verify, newest-2 retention, pair replication carry
over unchanged. **This phase changes a running data path — it gets its own live gate** (design §7
note): deploy lands on a driver pair with central SQL stopped mid-fetch → retry lands; #485
last-known-good semantics re-proven on the new path.
**Exit gate:** live gate green on the rig; a driver node with an empty LocalDb and reachable
central boots into the current config; with central down it boots last-known-good.
### Phase 3 — Config fetch-and-cache from central — **CODE-COMPLETE 2026-07-23 (Tasks 0–8; live gate Task 10 pending)**
**Decisions taken:** the deploy artifact is served **by central over a gRPC fetch RPC**
(`DeploymentArtifactService.Fetch`, streamed chunks — chosen over token-gated HTTP), authenticated by
a **shared node bearer key** (`ConfigServe:ApiKey == ConfigSource:ApiKey`, `FixedTimeEquals`,
fail-closed — chosen over a per-deployment token, so **no migration** and `DispatchDeployment` stays
payload-free). Cutover is a **config-flagged dark switch** `ConfigSource:Mode` (`Direct` default |
`FetchAndCache`), per node. Plan + task ledger:
[`docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md`](2026-07-22-mesh-phase3-config-fetch-and-cache.md).
**Shipped (all green, sabotage-checked):** `ConfigSource`/`ConfigServe` options + validator; the repo's
first in-repo `.proto` + `Grpc.Tools` codegen; central `DeploymentArtifactGrpcService` (streams the
sealed blob, NotFound-hides unknown/not-sealed/empty); `ConfigServeAuthInterceptor` + a dedicated h2c
Kestrel listener merged with the LocalDb-sync listener (re-binds the existing surface once, adds both
dedicated ports); node `GrpcDeploymentArtifactFetcher` (SHA-verified reassembly, endpoint failover,
null-on-any-failure); the `FetchAndCache` apply path (`PipeTo(Self)`, fetch→cache→apply-from-bytes,
null = apply-failure keeping last-known-good) and bootstrap-from-the-LocalDb-pointer (no driver-side
config SQL read); #485 coverage; and a **real two-Host gRPC boundary test** (right-key reassembles,
wrong-key control returns null, failover). Chunking, SHA-256 verify, newest-2 retention, pair
replication carry over unchanged. **Phase boundary held:** config **reads** only — the
`NodeDeploymentState` ack-row write, `DbHealthProbe`, `EfAlarmConditionStateStore` stay for Phase 4.
**Remaining (Tasks 9–10):** rig config + docs (done) and the live gate — deploy on a site pair flipped
to `FetchAndCache` with central SQL stopped mid-fetch → retry lands; #485 last-known-good re-proven on
the new path.
**Exit gate:** live gate green on the rig; a driver node with an empty LocalDb and reachable central
boots into the current config; with central down it boots last-known-good.
### Phase 4 — Cut the driver-side ConfigDb connection
**Scope (design §6.1 audit table):** re-home `EfAlarmConditionStateStore` to LocalDb (pair-local
@@ -229,7 +243,7 @@ resource sizing on the site VMs should be checked once both products run the ful
| Prereq: selfform-fallback + manual-failover plan | **DONE 2026-07-22**, merged `a78425ea`. Shipped *not* as the planned `SelfFormAfter` watchdog — the live gate caught that islanding a manually-failed-over node — but as self-first seed ordering + `AkkaClusterOptionsValidator`. Phase 6's seed-ordering scope is therefore already discharged. |
| 1 ClusterNode columns + DB ack set | **DONE 2026-07-22**, merged `7654f24d`. Deploy-seal semantics changed as flagged, but the escape hatch is **`ClusterNode.MaintenanceMode`**, not `Enabled=0` — the live gate found `Enabled=0` is rejected by `DraftValidator.ValidateClusterTopology` on any 2-node pair. Gate record: `2026-07-22-mesh-phase1-live-gate.md`. |
| 2 ClusterClient transport | **DONE 2026-07-22** (live gate PASSED, shipped dark) — shipped dark (`MeshTransport:Mode=Dps` default), both comm actors registered in both modes. Corrections: `SendToAll` not `Send`, one fleet-wide client not one per cluster, `/user/node-communication` not `/user/cluster-communication`. The gate's "Ask timing out" item does not exist to test — no cross-boundary Ask in this phase; see the phase section. |
| 3 fetch-and-cache | not started |
| 3 fetch-and-cache | **CODE-COMPLETE 2026-07-23** (Tasks 0–8 merged to the phase branch, all green + sabotage-checked; live gate Task 10 pending). gRPC fetch RPC + shared node key; dark switch `ConfigSource:Mode` (Direct default). Rig flip: `OTOPCUA_CONFIG_MODE=FetchAndCache` on the site nodes. See `2026-07-22-mesh-phase3-config-fetch-and-cache.md`. |
| 4 cut driver ConfigDb | not started |
| 5 gRPC telemetry | not started |
| 6 mesh partition + co-location | not started |