Merge branch 'feat/mesh-phase3' — per-cluster mesh Phase 3 (config fetch-and-cache)
Driver nodes can obtain config via a gRPC fetch from central + LocalDb cache instead of reading central SQL, behind the ConfigSource:Mode=FetchAndCache dark switch (Direct default). Central serves the artifact over a dedicated h2c listener gated by a shared node key; the node SHA-verifies against the dispatched RevisionHash, applies from the bytes in hand, and boots from the LocalDb pointer with no SQL read. Live gate PASSED on docker-dev (caught + fixed a Status==Sealed serve-gate deadlock). Phase boundary: config READS only. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
@@ -276,6 +276,45 @@ and `docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md`.
|
||||
- **There is no such thing as a cluster-scoped deployment.** `Deployment` has no `ClusterId`, `ConfigComposer` always snapshots the whole DB, and `DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The phase-1 plan asked for cluster-scope filtering of the ack set; it was dropped as describing a feature that does not exist.
|
||||
- `ClusterNode` gained `AkkaPort` (NOT NULL, 4053) + `GrpcPort` (nullable) as **central's dial targets** for Phases 2/5. They duplicate the node's own `Cluster:Port` / `Cluster:PublicHostname` with no schema-level enforcement, so `ClusterNodeAddressReconcilerActor` (admin singleton) logs an Error on drift. **Phase 2 must revisit it** — once the meshes split, an admin node cannot see site members and every site row would report `EnabledRowNotInCluster` forever.
|
||||
|
||||
## Config source (`ConfigSource` / `ConfigServe`)
|
||||
|
||||
Per-cluster mesh **Phase 3** (config fetch-and-cache) added a second way a **driver** node obtains its
|
||||
deployed configuration, selected by `ConfigSource:Mode`: `Direct` (read `Deployment.ArtifactBlob` from
|
||||
central SQL — **the default**, byte-for-byte today's behaviour) or `FetchAndCache` (fetch the artifact
|
||||
bytes from central over a dedicated-h2c **gRPC stream**, verify `SHA-256 == RevisionHash`, cache in
|
||||
LocalDb, and read **only** the cache — no central-SQL config read). Central (admin role) serves via
|
||||
`DeploymentArtifactService` behind `ConfigServeAuthInterceptor` on `ConfigServe:GrpcListenPort`, gated by
|
||||
a **shared node bearer key** (`ConfigServe:ApiKey == ConfigSource:ApiKey`, `FixedTimeEquals`,
|
||||
fail-closed). Things worth knowing before touching it:
|
||||
|
||||
- **The serve side is wired in both modes** (mapped whenever `ConfigServe:GrpcListenPort > 0` on an
|
||||
admin node), so flipping a node to `FetchAndCache` is a config change, not a redeploy — the same
|
||||
dark-switch discipline as [MeshTransport](#mesh-command-transport-meshtransport).
|
||||
- **`RevisionHash == SHA-256(artifact bytes)` is load-bearing.** `ConfigComposer` computes the revision
|
||||
hash as the lowercase-hex SHA-256 of the exact blob, so the node verifies a fetch against the
|
||||
`RevisionHash` the dispatch already carries — **no hash rides in the proto, and `DispatchDeployment`
|
||||
stays payload-free.** A future change to how the revision hash is derived silently breaks every fetch.
|
||||
- **A null fetch is "no answer," never an empty config (#485).** All endpoints down / `NotFound` / SHA
|
||||
mismatch / zero-length ⇒ the fetcher returns null ⇒ the apply FAILS, the revision does not advance,
|
||||
and the node keeps serving last-known-good. It never tears down to an empty address space. At boot a
|
||||
`FetchAndCache` node restores served state from the LocalDb pointer with **no** central-SQL read; an
|
||||
empty/unreadable cache lands Steady-with-no-revision (the first dispatch fetches), **never Stale**.
|
||||
- **Both pair nodes fetch independently** — no Primary gating of the fetch (same shape as both reading
|
||||
SQL today; `StoreAsync` is idempotent and the cache replicates).
|
||||
- **The dedicated h2c listener coexists with the LocalDb-sync listener.** A fused admin+driver node with
|
||||
both ports set re-binds its existing HTTP surface exactly **once** and adds both dedicated HTTP/2-only
|
||||
ports — getting this wrong either double-binds (crash) or silently unbinds the AdminUI (the Kestrel
|
||||
`Listen*`-discards-`ASPNETCORE_URLS` trap the LocalDb block already warned about).
|
||||
- **Phase boundary: config *reads* only.** A `FetchAndCache` node still writes its `NodeDeploymentState`
|
||||
ack row to central SQL; `DbHealthProbe` and `EfAlarmConditionStateStore` still touch SQL. Cutting those
|
||||
is Phase 4 — resist folding it in, or the "no SQL" scope becomes unprovable-in-parts.
|
||||
|
||||
On the docker-dev rig the central pair carries `ConfigServe` (`:4055`, committed dev key) and stays
|
||||
`Direct`; the four site nodes carry `ConfigSource` (endpoints + matching key) defaulting to `Direct`.
|
||||
Flip only the site nodes with `OTOPCUA_CONFIG_MODE=FetchAndCache` at `docker compose up`. See
|
||||
`docs/Configuration.md` §`ConfigSource` / `ConfigServe`, `docs/Redundancy.md` §"Command transport", and
|
||||
`docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md`.
|
||||
|
||||
## LocalDb pair-local store (Phases 1 + 2)
|
||||
|
||||
Every **driver-role** node keeps a consolidated `ZB.MOM.WW.LocalDb` SQLite database (the retired
|
||||
|
||||
@@ -32,6 +32,7 @@
|
||||
<PackageVersion Include="Grpc.AspNetCore" Version="2.76.0" />
|
||||
<PackageVersion Include="Grpc.Core.Api" Version="2.76.0" />
|
||||
<PackageVersion Include="Grpc.Net.Client" Version="2.76.0" />
|
||||
<PackageVersion Include="Grpc.Tools" Version="2.76.0" />
|
||||
<PackageVersion Include="libplctag" Version="1.5.2" />
|
||||
<PackageVersion Include="MessagePack" Version="2.5.301" />
|
||||
<PackageVersion Include="Microsoft.AspNetCore.Authorization" Version="10.0.7" />
|
||||
|
||||
@@ -3,7 +3,12 @@
|
||||
# Compose drives OTOPCUA_ROLES + Cluster:* env per container to differentiate them.
|
||||
# A separate `migrator` stage (below) applies EF migrations once on bring-up.
|
||||
|
||||
FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build
|
||||
# Build stage is PINNED to linux/amd64. Per-cluster mesh Phase 3 added an in-repo .proto compiled by
|
||||
# Grpc.Tools, whose bundled linux_arm64 `protoc` segfaults (exit 139) on Apple-Silicon Docker — so a
|
||||
# native-arm64 build stage cannot generate the gRPC stubs. `dotnet publish` here is framework-dependent
|
||||
# (portable IL + every RID's native assets), so the amd64-built output runs unchanged on the native
|
||||
# runtime image below; only this one-time build is emulated, runtime stays native/fast.
|
||||
FROM --platform=linux/amd64 mcr.microsoft.com/dotnet/sdk:10.0 AS build
|
||||
WORKDIR /src
|
||||
COPY . .
|
||||
RUN dotnet restore ZB.MOM.WW.OtOpcUa.slnx
|
||||
|
||||
@@ -178,6 +178,14 @@ services:
|
||||
Cluster__Hostname: "0.0.0.0"
|
||||
Cluster__Port: "4053"
|
||||
Cluster__PublicHostname: "central-1"
|
||||
# Per-cluster mesh Phase 3 (config fetch-and-cache). Central SERVES the deployed-configuration
|
||||
# artifact to FetchAndCache site nodes over a dedicated h2c gRPC listener (:4055), gated by the
|
||||
# shared node key. Central itself is ALWAYS Direct — it IS the SQL source and must never fetch.
|
||||
# ConfigServe:ApiKey must be byte-identical to the site nodes' ConfigSource:ApiKey (fail-closed).
|
||||
# Binding :4055 activates the dedicated-h2c Kestrel takeover on central (it re-binds :9000 too).
|
||||
ConfigServe__GrpcListenPort: "4055"
|
||||
ConfigServe__ApiKey: "configserve-docker-dev-key"
|
||||
ConfigSource__Mode: "Direct"
|
||||
# Both redundancy peers are seeds (#459 / ScadaBridge parity) so a restarted node can
|
||||
# re-join the mesh via EITHER peer, not only central-1.
|
||||
#
|
||||
@@ -269,6 +277,11 @@ services:
|
||||
Cluster__Hostname: "0.0.0.0"
|
||||
Cluster__Port: "4053"
|
||||
Cluster__PublicHostname: "central-2"
|
||||
# Per-cluster mesh Phase 3 — serves the artifact to FetchAndCache site nodes (see central-1).
|
||||
# Always Direct; ConfigServe:ApiKey must match the site nodes' ConfigSource:ApiKey.
|
||||
ConfigServe__GrpcListenPort: "4055"
|
||||
ConfigServe__ApiKey: "configserve-docker-dev-key"
|
||||
ConfigSource__Mode: "Direct"
|
||||
# Both redundancy peers are seeds (#459 / ScadaBridge parity) — see central-1. SELF FIRST:
|
||||
# central-2 lists itself as seed-nodes[0], which is what lets it cold-start while central-1
|
||||
# is down (it previously listed central-1 first and simply never came Up in that case).
|
||||
@@ -350,6 +363,15 @@ services:
|
||||
Cluster__Hostname: "0.0.0.0"
|
||||
Cluster__Port: "4053"
|
||||
Cluster__PublicHostname: "site-a-1"
|
||||
# Per-cluster mesh Phase 3 dark switch. Default Direct (reads central SQL). Flip the site nodes to
|
||||
# FetchAndCache with `OTOPCUA_CONFIG_MODE=FetchAndCache docker compose up`: they then fetch the
|
||||
# deployed artifact from central over gRPC (endpoints below) and read ONLY the LocalDb cache — no
|
||||
# central SQL config read. Central stays Direct, so this env var moves ONLY the four site nodes.
|
||||
# ConfigSource:ApiKey must equal central's ConfigServe:ApiKey (the interceptor is fail-closed).
|
||||
ConfigSource__Mode: "${OTOPCUA_CONFIG_MODE:-Direct}"
|
||||
ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
|
||||
ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
|
||||
ConfigSource__ApiKey: "configserve-docker-dev-key"
|
||||
# Site nodes are deliberately NOT seeds — they join the central pair's mesh. The self-first
|
||||
# seed rule is conditional for exactly this reason (it binds only when a node's own address
|
||||
# is in its own seed list), so these configs are exempt rather than broken.
|
||||
@@ -432,6 +454,12 @@ services:
|
||||
Cluster__Hostname: "0.0.0.0"
|
||||
Cluster__Port: "4053"
|
||||
Cluster__PublicHostname: "site-a-2"
|
||||
# Per-cluster mesh Phase 3 dark switch (see site-a-1). Default Direct; OTOPCUA_CONFIG_MODE=FetchAndCache
|
||||
# flips it to fetch the artifact from central over gRPC and read only the LocalDb cache.
|
||||
ConfigSource__Mode: "${OTOPCUA_CONFIG_MODE:-Direct}"
|
||||
ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
|
||||
ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
|
||||
ConfigSource__ApiKey: "configserve-docker-dev-key"
|
||||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
||||
Cluster__Roles__0: "driver"
|
||||
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
|
||||
@@ -503,6 +531,12 @@ services:
|
||||
Cluster__Hostname: "0.0.0.0"
|
||||
Cluster__Port: "4053"
|
||||
Cluster__PublicHostname: "site-b-1"
|
||||
# Per-cluster mesh Phase 3 dark switch (see site-a-1). Default Direct; OTOPCUA_CONFIG_MODE=FetchAndCache
|
||||
# flips it to fetch the artifact from central over gRPC and read only the LocalDb cache.
|
||||
ConfigSource__Mode: "${OTOPCUA_CONFIG_MODE:-Direct}"
|
||||
ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
|
||||
ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
|
||||
ConfigSource__ApiKey: "configserve-docker-dev-key"
|
||||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
||||
Cluster__Roles__0: "driver"
|
||||
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
|
||||
@@ -557,6 +591,12 @@ services:
|
||||
Cluster__Hostname: "0.0.0.0"
|
||||
Cluster__Port: "4053"
|
||||
Cluster__PublicHostname: "site-b-2"
|
||||
# Per-cluster mesh Phase 3 dark switch (see site-a-1). Default Direct; OTOPCUA_CONFIG_MODE=FetchAndCache
|
||||
# flips it to fetch the artifact from central over gRPC and read only the LocalDb cache.
|
||||
ConfigSource__Mode: "${OTOPCUA_CONFIG_MODE:-Direct}"
|
||||
ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
|
||||
ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
|
||||
ConfigSource__ApiKey: "configserve-docker-dev-key"
|
||||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
||||
Cluster__Roles__0: "driver"
|
||||
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
|
||||
|
||||
@@ -133,6 +133,35 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only
|
||||
|
||||
On the docker-dev rig every node carries these keys already; flip the whole rig with `OTOPCUA_MESH_MODE=ClusterClient` at `docker compose up`.
|
||||
|
||||
### `ConfigSource` / `ConfigServe` (config fetch-and-cache)
|
||||
|
||||
- **Purpose:** [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) **Phase 3**. `ConfigSource:Mode` selects where a **driver** node reads its deployed configuration: `Direct` (read `Deployment.ArtifactBlob` from central SQL, today's behaviour) or `FetchAndCache` (fetch the artifact bytes from central over a gRPC stream, verify SHA-256, cache in LocalDb, and read **only** the cache — no central-SQL config read). `ConfigServe` is the **central** (admin-role) serve side that a `FetchAndCache` node dials.
|
||||
- **Options classes:** `ConfigSourceOptions` (`SectionName = "ConfigSource"`) + `ConfigServeOptions` (`SectionName = "ConfigServe"`), both in `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`. Bound by `AddOtOpcUaCluster(config)`; `ConfigSourceOptionsValidator` runs at `ValidateOnStart`.
|
||||
|
||||
**`ConfigSource` (driver-side fetch):**
|
||||
|
||||
| Key | Type | Default | Meaning |
|
||||
|---|---|---|---|
|
||||
| `Mode` | string | `Direct` | `Direct` = read central SQL (unchanged). `FetchAndCache` = fetch from central over gRPC, read the LocalDb cache. Any other value **refuses to start**. |
|
||||
| `CentralFetchEndpoints` | string[] | `[]` | Central artifact-gRPC base addresses, e.g. `http://central-1:4055` (h2c ⇒ the `http` scheme). Tried in order for failover. **Required** under `FetchAndCache`. |
|
||||
| `ApiKey` | string | `""` | Shared bearer key; must equal central's `ConfigServe:ApiKey`. **Supply via env `ConfigSource__ApiKey` — never commit.** Required under `FetchAndCache`. |
|
||||
| `FetchTimeoutSeconds` | int | `30` | Per-fetch deadline. Must be positive under `FetchAndCache`. |
|
||||
|
||||
**`ConfigServe` (central serve):**
|
||||
|
||||
| Key | Type | Default | Meaning |
|
||||
|---|---|---|---|
|
||||
| `GrpcListenPort` | int | `0` | Dedicated **h2c-only** Kestrel port for the artifact gRPC service. `0` = disabled (nothing bound). Must differ from the main HTTP port and from `LocalDb:SyncListenPort` (h2c cannot share a cleartext HTTP/1 port). Mapped only on admin-role nodes. |
|
||||
| `ApiKey` | string | `""` | Shared bearer key the serve-side interceptor checks (constant-time, **fail-closed**: an unset key rejects every call). **Supply via env `ConfigServe__ApiKey` — never commit.** |
|
||||
|
||||
**It is a dark switch; the serve side is always wired.** Central maps the `DeploymentArtifactService` behind the `ConfigServeAuthInterceptor` (path-scoped, `FixedTimeEquals`, fail-closed) whenever `ConfigServe:GrpcListenPort > 0`, in both modes — so flipping a node to `FetchAndCache` is a config change, not a redeploy. A `FetchAndCache` node that cannot fetch (all endpoints down, `NotFound`, SHA mismatch) **fails the apply and keeps last-known-good** — a null fetch is "no answer," never an empty configuration (the [#485](plans/2026-07-15-v3-batch4-address-space-plan.md) contract). At boot it restores served state from the LocalDb pointer with no SQL read; an empty or unreadable cache lands Steady-with-no-revision (the first dispatch fetches), never Stale.
|
||||
|
||||
**The listener is dedicated h2c and coexists with the LocalDb-sync listener.** A fused admin+driver node with **both** `ConfigServe:GrpcListenPort` and `LocalDb:SyncListenPort` set re-binds its existing HTTP surface exactly once and adds both dedicated HTTP/2-only ports on top — see the same Kestrel-takeover caveat under [`LocalDb`](#localdb-pair-local-config-cache--driver-role-only).
|
||||
|
||||
**`RevisionHash == SHA-256(artifact bytes)` is load-bearing.** `ConfigComposer` computes the deployment's revision hash as the lowercase-hex SHA-256 of the exact artifact bytes, so the node verifies a fetch against the `RevisionHash` the dispatch already carries — no hash rides in the proto. If a future change makes the revision hash anything else, every fetch fails verification.
|
||||
|
||||
On the docker-dev rig the central pair carries `ConfigServe` (port `4055`, dev key) and stays `Direct`; the four site nodes carry `ConfigSource` (endpoints + matching key) defaulting to `Direct`. Flip only the site nodes with `OTOPCUA_CONFIG_MODE=FetchAndCache` at `docker compose up` — central keeps SQL and models the target topology.
|
||||
|
||||
### `ConnectionStrings` → `ConfigDb`
|
||||
|
||||
- **Purpose:** the central Config DB connection string. **Required for every role** — `Program.cs` calls `AddOtOpcUaConfigDb` unconditionally.
|
||||
|
||||
@@ -438,6 +438,19 @@ are untouched, and `redundancy-state` itself stays on DPS (it is bidirectional a
|
||||
`Cluster.State`, which makes it genuinely mesh-bound — the hardest remaining dependency before the
|
||||
meshes can split).
|
||||
|
||||
**Config bytes travel out-of-band (Phase 3).** Commands stay on one of the two transports above, but
|
||||
the *configuration artifact itself* does not ride any Akka message — `DispatchDeployment` is
|
||||
payload-free (`DeploymentId` + `RevisionHash`). Under `ConfigSource:Mode = FetchAndCache` (see
|
||||
[Configuration.md § `ConfigSource`](Configuration.md#configsource--configserve-config-fetch-and-cache))
|
||||
a driver node fetches the bytes from central over a dedicated h2c **gRPC stream**, verifies
|
||||
`SHA-256 == RevisionHash`, and reads only its LocalDb cache — no central-SQL config read. **Both pair
|
||||
nodes fetch independently**, exactly as both read SQL today: there is no Primary gating of the fetch.
|
||||
That is safe because `StoreAsync` is idempotent and the cache replicates (pair replication carries the
|
||||
artifact to a node that missed its own fetch), and it keeps config delivery decoupled from redundancy
|
||||
timing. Central serves the bytes from its SQL row; a `FetchAndCache` node that cannot reach central
|
||||
fails the apply and keeps last-known-good (it never tears down to an empty address space), and it boots
|
||||
its served state from the LocalDb pointer with no central dependency at all.
|
||||
|
||||
## Pair-local store (LocalDb — Phases 1 + 2)
|
||||
|
||||
Independently of the ServiceLevel machinery above, every **driver-role** node keeps a consolidated
|
||||
|
||||
@@ -0,0 +1,473 @@
|
||||
# Per-Cluster Mesh Phase 3 — Config Fetch-and-Cache from Central
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** In a config-flagged `FetchAndCache` mode, a driver node obtains its deployed configuration by **fetching the artifact bytes from central over a gRPC stream**, caching them in LocalDb, and reading **all** config from that cache — so the driver never reads `Deployment.ArtifactBlob` from central SQL. Default `Direct` mode is byte-for-byte today's behaviour.
|
||||
|
||||
**Architecture:** Central (admin role) hosts a new gRPC service `DeploymentArtifactService.Fetch(deployment_id) → stream ArtifactChunk`, on a **dedicated h2c Kestrel listener** (the LocalDb-sync listener pattern), gated by a **shared bearer key** (fail-closed, `FixedTimeEquals`, path-scoped interceptor). A driver node in `FetchAndCache` mode, on `DispatchDeployment`, streams the bytes, verifies `SHA-256(bytes) == RevisionHash`, writes them through the existing `IDeploymentArtifactCache`, and applies from the cache; every config read (`ReconcileDrivers`, `PushDesiredSubscriptions`, address-space rebuild, `Bootstrap` recovery, Stale recovery) is redirected to the cache. The #485 "empty/unreadable bytes ⇒ no answer, keep last-known-good, fail the apply" guard is carried onto the fetch and cache-read paths.
|
||||
|
||||
**Tech Stack:** .NET 10, Akka.NET 1.5.62, `Grpc.AspNetCore` (server, already referenced) + **`Grpc.Tools` + a first in-repo `.proto`** (new — the repo has consumed packaged gRPC clients but never compiled a proto locally), `Grpc.Net.Client` (node client), the Phase-1 `ZB.MOM.WW.LocalDb` cache (`IDeploymentArtifactCache` / `LocalDbDeploymentArtifactCache`), xUnit + Shouldly + Akka.TestKit.
|
||||
|
||||
---
|
||||
|
||||
## Decisions already made (do not re-litigate)
|
||||
|
||||
| Decision | Choice | Consequence for this plan |
|
||||
|---|---|---|
|
||||
| Fetch transport | **gRPC fetch RPC** (user chose this over token-gated HTTP) | Introduces the repo's first `.proto` + `Grpc.Tools` codegen (Task 1); a new central-side h2c listener (Task 3); a generated client on the node (Task 4). |
|
||||
| Fetch auth | **Shared node key** (user chose this over a per-deployment token) | A single bearer secret (`ConfigServe:ApiKey` == `ConfigSource:ApiKey`), `FixedTimeEquals`, fail-closed. **No migration, no token lifecycle, and `DispatchDeployment` is UNCHANGED** — the node fetches by deployment id and authenticates with the shared key. |
|
||||
| Cutover | **Config-flagged dark switch** (`ConfigSource:Mode` = `Direct` default \| `FetchAndCache`), per node | Matches Phase 2. Rig comes up on `Direct`; the live gate flips only the **site** nodes to `FetchAndCache` (central keeps SQL, models the target). Rollback is a config change. |
|
||||
| Both nodes fetch | **Yes — no Primary gating of the fetch** | Today both pair nodes read central SQL independently (no gating); `FetchAndCache` keeps that shape. `IDeploymentArtifactCache.StoreAsync` is idempotent (`IsAlreadyCachedAsync`), and pair replication means a node that missed its fetch can still get the bytes from the peer. Primary-only fetch is a possible later optimization, explicitly out of scope. |
|
||||
|
||||
## Five facts the recon nailed down (read before Task 5)
|
||||
|
||||
1. **`RevisionHash == SHA-256(artifact bytes).`** `ConfigComposer.SnapshotAndFlattenAsync` (`src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/AdminOperations/ConfigComposer.cs:57-58`) computes the revision hash as SHA-256-hex-lowercase over the exact serialized blob — the **same** value the cache stores as `deployment_pointer.artifact_sha256`. So the node can verify a fetch against the `RevisionHash` already carried in `DispatchDeployment`; **no hash field is needed in the proto.**
|
||||
|
||||
2. **The driver reads central SQL on exactly these config paths, all reading `Deployment.ArtifactBlob` (never re-composing from raw rows):** `Bootstrap()` (`DriverHostActor.cs:584-603`, reads `NodeDeploymentStates` + `Deployments.RevisionHash`), `ReconcileDrivers` (`1717-1768`, reads at `1722-1726`), `PushDesiredSubscriptions` (`1880-1898`, reads at `1885-1889`), `TryRecoverFromStale` (`2363-2390`, reads at `2367-2373`), and the delegated `OpcUaPublishActor.HandleRebuild` → `LoadArtifact`/`LoadLatestArtifact` (`OpcUaPublishActor.cs:475-510`) which fires **only when `ApplyAndAck` passes no blob** (`DriverHostActor.cs:1667`). Phase 3 redirects these in `FetchAndCache` mode.
|
||||
|
||||
3. **The cache is already complete and correct.** `IDeploymentArtifactCache` / `LocalDbDeploymentArtifactCache` (`src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/`) does chunked base64 (128 KiB), SHA-256-verified reassembly, newest-2 retention, and replicates. It is written by `CacheAppliedArtifact` (`DriverHostActor.cs:1796-1835`) after a successful apply, and read only by `TryBootFromCache` (`671-714`) in the `Bootstrap()` SQL-unreachable catch. **Phase 3 changes WHEN it is written (before apply, from the fetch) and WHEN it is read (always, in `FetchAndCache`).** The `_isRunningFromCache` flag (field `85`, set at `698`, cleared at `1676`) already exists.
|
||||
|
||||
4. **`DispatchDeployment` is payload-free by design** (`DeploymentId` + `RevisionHash` + `CorrelationId`) so it never approaches the Akka frame limit. **Phase 3 keeps it payload-free** — the shared-key decision means no token rides in it, and the bytes travel out-of-band over gRPC. Do not add the artifact to any Akka message.
|
||||
|
||||
5. **The #485 guard is at every byte-parsing seam and must be carried onto the new paths:** `ReconcileDrivers` returns `null` on `blob.Length == 0` (`1735-1747`) → `ApplyAndAck` treats null as an apply **failure** (does not advance `_currentRevision`, writes `Failed`, sends a Failed ack — so a re-dispatch of the same revision actually retries, `1639-1659`); `CacheAppliedArtifact` skips empty (`1801-1808`); `PushDesiredSubscriptionsFromArtifact` skips empty (`1908-1920`); `OpcUaPublishActor.HandleRebuild` returns on `{ Length: 0 }` keeping the served address space (`372-391`); the cache's `ReassembleAsync` returns null on any chunk-count/base64/SHA/timestamp failure. **A zero-byte or SHA-mismatched fetch is "no answer," never "a config with no drivers."**
|
||||
|
||||
---
|
||||
|
||||
## The dark switch, precisely
|
||||
|
||||
`ConfigSourceOptions.Mode` selects the driver's config source:
|
||||
|
||||
- **`Direct`** (default): every path reads central SQL exactly as today. The new fetcher is not invoked. Byte-for-byte no behaviour change; the whole phase is inert.
|
||||
- **`FetchAndCache`**: on `DispatchDeployment`, fetch bytes from central over gRPC → verify SHA-256 == `RevisionHash` → `StoreAsync` into the cache → apply **from the fetched bytes in hand** (passed through to every consumer, so `OpcUaPublishActor` never reads SQL). `Bootstrap()` recovers the last-applied deployment from the LocalDb **pointer**, not `NodeDeploymentStates`. No driver-side `Deployment.ArtifactBlob` read occurs.
|
||||
|
||||
The central-side serve components (gRPC service, listener, interceptor) are wired **unconditionally on admin-role nodes** whenever `ConfigServe:GrpcListenPort > 0`, in both modes — mirroring Phase 2's "register in both modes so flipping the flag is not a redeploy." An idle listener costs nothing.
|
||||
|
||||
**Explicitly deferred to Phase 4 (do NOT touch here):** the `NodeDeploymentState` **write** (`UpsertNodeDeploymentState`, `DriverHostActor.cs:2392-2422`, still writes the ack row to central SQL), `DbHealthProbeActor`, `EfAlarmConditionStateStore`, and removing the driver-role ConfigDb connection string. Phase 3 removes the config-blob **reads** only. A `FetchAndCache` node still writes its ack row to SQL; that is Phase 4's cut.
|
||||
|
||||
---
|
||||
|
||||
## Task 0: `ConfigSourceOptions` + `ConfigServeOptions` + validators + registration
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** Task 1
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ConfigSourceOptions.cs`
|
||||
- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ConfigSourceOptionsValidator.cs`
|
||||
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs` (register both in `AddOtOpcUaCluster`, beside the `MeshTransportOptions` registration)
|
||||
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/ConfigSourceOptionsValidatorTests.cs`
|
||||
|
||||
**Design.** One options class holds both the node (fetch) and central (serve) surfaces, keyed `ConfigSource` and `ConfigServe`. Model after `MeshTransportOptions` / `MeshTransportOptionsValidator` (same directory) exactly — the validator fails host start on a `FetchAndCache` shape that would leave the node unable to fetch (empty endpoints, empty key), because every such fault otherwise surfaces as a silent absence (a deploy that never applies).
|
||||
|
||||
```csharp
|
||||
// ConfigSourceOptions.cs
|
||||
namespace ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
|
||||
/// <summary>Node-side config source selection (per-cluster mesh Phase 3).</summary>
|
||||
public sealed class ConfigSourceOptions
|
||||
{
|
||||
public const string SectionName = "ConfigSource";
|
||||
public const string ModeDirect = "Direct";
|
||||
public const string ModeFetchAndCache = "FetchAndCache";
|
||||
|
||||
/// <summary><see cref="ModeDirect"/> (read central SQL, today's behaviour) or
|
||||
/// <see cref="ModeFetchAndCache"/> (fetch from central over gRPC, read LocalDb).</summary>
|
||||
public string Mode { get; set; } = ModeDirect;
|
||||
|
||||
/// <summary>Central artifact-gRPC base addresses, e.g. <c>http://central-1:4055</c>. h2c (http
|
||||
/// scheme). Required and tried in order (failover) under <see cref="ModeFetchAndCache"/>.</summary>
|
||||
public string[] CentralFetchEndpoints { get; set; } = [];
|
||||
|
||||
/// <summary>Shared bearer key; must equal central's <see cref="ConfigServeOptions.ApiKey"/>.
|
||||
/// Supply via env <c>ConfigSource__ApiKey</c>; never commit. Required under FetchAndCache.</summary>
|
||||
public string ApiKey { get; set; } = string.Empty;
|
||||
|
||||
/// <summary>Per-fetch deadline. Non-positive rejected under FetchAndCache.</summary>
|
||||
public int FetchTimeoutSeconds { get; set; } = 30;
|
||||
}
|
||||
|
||||
/// <summary>Central-side artifact-serve surface (per-cluster mesh Phase 3).</summary>
|
||||
public sealed class ConfigServeOptions
|
||||
{
|
||||
public const string SectionName = "ConfigServe";
|
||||
|
||||
/// <summary>Dedicated h2c listener port for the artifact gRPC service. <c>0</c> = disabled
|
||||
/// (nothing bound). Must differ from the main HTTP port and from LocalDb:SyncListenPort.</summary>
|
||||
public int GrpcListenPort { get; set; }
|
||||
|
||||
/// <summary>Shared bearer key the interceptor checks (FixedTimeEquals, fail-closed). Supply via
|
||||
/// env <c>ConfigServe__ApiKey</c>; never commit.</summary>
|
||||
public string ApiKey { get; set; } = string.Empty;
|
||||
}
|
||||
```
|
||||
|
||||
The validator (mirror `MeshTransportOptionsValidator`): reject an unknown `Mode`; under `FetchAndCache` reject empty `CentralFetchEndpoints`, any endpoint not starting `http://` or `https://`, empty `ApiKey`, and non-positive `FetchTimeoutSeconds`. `ConfigServeOptions` needs no cross-field validation (a `0` port is a legitimate "disabled").
|
||||
|
||||
**Step 1: Write the failing test** — `ConfigSourceOptionsValidatorTests` with cases: default (`Direct`, empty everything) → Success; unknown mode → Fail; `FetchAndCache` + empty endpoints → Fail; + non-`http(s)` endpoint → Fail; + empty key → Fail; + `FetchTimeoutSeconds = 0` → Fail; fully-populated `FetchAndCache` → Success. (No implicit usings in Cluster.Tests — `using Xunit;`.)
|
||||
|
||||
**Step 2:** `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests --filter "FullyQualifiedName~ConfigSourceOptionsValidatorTests"` → FAIL (types missing).
|
||||
|
||||
**Step 3:** Write the two options classes + the validator; register in `AddOtOpcUaCluster`:
|
||||
```csharp
|
||||
services.AddValidatedOptions<ConfigSourceOptions, ConfigSourceOptionsValidator>(
|
||||
configuration, ConfigSourceOptions.SectionName);
|
||||
services.Configure<ConfigServeOptions>(configuration.GetSection(ConfigServeOptions.SectionName));
|
||||
```
|
||||
|
||||
**Step 4:** Re-run → PASS. **Sabotage-check:** flip one validator branch (e.g. accept empty endpoints) → its test reddens.
|
||||
|
||||
**Step 5: Commit** `feat(mesh): ConfigSource/ConfigServe options + validator (Phase 3 dark switch)`
|
||||
|
||||
---
|
||||
|
||||
## Task 1: `deployment_artifact.v1` proto + Grpc.Tools codegen
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 0
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos/deployment_artifact.proto`
|
||||
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Commons/ZB.MOM.WW.OtOpcUa.Commons.csproj` (add `Grpc.Tools` + `Google.Protobuf` + `Grpc.Core.Api` package refs and a `<Protobuf>` item; `GrpcServices="Both"` so both server base + client stub generate)
|
||||
- Modify: `Directory.Packages.props` (pin `Grpc.Tools`, `Google.Protobuf`, `Grpc.Core.Api` if not already pinned — check first)
|
||||
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests/DeploymentArtifactProtoTests.cs` (a compile-touch test that references the generated `DeploymentArtifactService.DeploymentArtifactServiceBase` type and `FetchRequest`, proving codegen ran)
|
||||
|
||||
**Why Commons.** Both the central server (in Host/AdminUI) and the node client (in Runtime) need the generated types, and Commons is already referenced by every server project. This is the repo's **first** locally-compiled proto — there is no existing `<Protobuf>` item to copy, so wire it from scratch.
|
||||
|
||||
```proto
|
||||
syntax = "proto3";
|
||||
package deployment_artifact.v1;
|
||||
option csharp_namespace = "ZB.MOM.WW.OtOpcUa.Commons.Protos.DeploymentArtifact.V1";
|
||||
|
||||
// Central serves the deployed-configuration artifact bytes to a driver node (Phase 3).
|
||||
service DeploymentArtifactService {
|
||||
// Streams the artifact for a sealed deployment as ordered chunks. NotFound if the id is
|
||||
// unknown or not sealed. The client verifies SHA-256(reassembled) == the RevisionHash it
|
||||
// already holds, so no hash travels here.
|
||||
rpc Fetch(FetchRequest) returns (stream ArtifactChunk);
|
||||
}
|
||||
message FetchRequest { string deployment_id = 1; }
|
||||
message ArtifactChunk { bytes data = 1; }
|
||||
```
|
||||
|
||||
`.csproj` addition (mirror the packaging note the LocalDb refs use):
|
||||
```xml
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Grpc.Tools" PrivateAssets="all" />
|
||||
<PackageReference Include="Google.Protobuf" />
|
||||
<PackageReference Include="Grpc.Core.Api" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<Protobuf Include="Protos\deployment_artifact.proto" GrpcServices="Both" />
|
||||
</ItemGroup>
|
||||
```
|
||||
|
||||
**Step 1:** Write the compile-touch test referencing `DeploymentArtifactService.DeploymentArtifactServiceBase` and `new FetchRequest { DeploymentId = "x" }`.
|
||||
**Step 2:** `dotnet build src/Core/ZB.MOM.WW.OtOpcUa.Commons` → FAIL (no proto/codegen).
|
||||
**Step 3:** Add the packages, the proto, the `<Protobuf>` item. `dotnet restore` then `dotnet build src/Core/ZB.MOM.WW.OtOpcUa.Commons`.
|
||||
**Step 4:** `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests --filter "FullyQualifiedName~DeploymentArtifactProtoTests"` → PASS. Confirm generated types resolve (`obj/.../DeploymentArtifactGrpc.cs`).
|
||||
**Step 5: Commit** `feat(mesh): deployment_artifact.v1 proto + first in-repo Grpc.Tools codegen`
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Central-side `DeploymentArtifactService` (streams the blob, existence-hiding)
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 4
|
||||
**Blocked by:** Task 1
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Grpc/DeploymentArtifactService.cs`
|
||||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/Grpc/DeploymentArtifactServiceTests.cs`
|
||||
|
||||
**Design.** Subclass the generated `DeploymentArtifactServiceBase`. `Fetch` reads the `Deployment` row by id from `OtOpcUaConfigDbContext` (admin node has the connection); if the row is missing OR `Status != Sealed` OR `ArtifactBlob.Length == 0`, throw `RpcException(new Status(StatusCode.NotFound, "…"))` (existence-hiding + #485: never stream zero bytes as if valid). Otherwise stream the blob in ≤128 KiB `ArtifactChunk`s via `responseStream.WriteAsync`. Lives in AdminUI (where the other admin-only endpoints + the config DB context live).
|
||||
|
||||
```csharp
|
||||
public sealed class DeploymentArtifactService : DeploymentArtifactServiceBase
|
||||
{
|
||||
private const int ChunkSize = 128 * 1024; // matches LocalDbDeploymentArtifactCache.ChunkSize
|
||||
private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbFactory;
|
||||
private readonly ILogger<DeploymentArtifactService> _log;
|
||||
// ctor injects both
|
||||
|
||||
public override async Task Fetch(
|
||||
FetchRequest request, IServerStreamWriter<ArtifactChunk> responseStream, ServerCallContext context)
|
||||
{
|
||||
if (!Guid.TryParse(request.DeploymentId, out var id))
|
||||
throw new RpcException(new Status(StatusCode.NotFound, "unknown deployment"));
|
||||
|
||||
await using var db = await _dbFactory.CreateDbContextAsync(context.CancellationToken);
|
||||
var row = await db.Deployments.AsNoTracking()
|
||||
.Where(d => d.DeploymentId == id)
|
||||
.Select(d => new { d.Status, d.ArtifactBlob })
|
||||
.FirstOrDefaultAsync(context.CancellationToken);
|
||||
|
||||
// Existence-hiding + #485: unknown / not-sealed / empty are one indistinguishable NotFound.
|
||||
if (row is null || row.Status != DeploymentStatus.Sealed || row.ArtifactBlob.Length == 0)
|
||||
throw new RpcException(new Status(StatusCode.NotFound, "unknown deployment"));
|
||||
|
||||
for (var offset = 0; offset < row.ArtifactBlob.Length; offset += ChunkSize)
|
||||
{
|
||||
var len = Math.Min(ChunkSize, row.ArtifactBlob.Length - offset);
|
||||
await responseStream.WriteAsync(
|
||||
new ArtifactChunk { Data = ByteString.CopyFrom(row.ArtifactBlob, offset, len) },
|
||||
context.CancellationToken);
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
(Confirm the sealed-status enum member name from `DeploymentStatus`; the recon showed `Deployment.Status` and the deploy sets `Sealed`.)
|
||||
|
||||
**Step 1: Write the failing test** — an in-memory `OtOpcUaConfigDbContext` (the AdminUI.Tests pattern) seeded with one sealed deployment; a fake `IServerStreamWriter<ArtifactChunk>` collecting writes; assert: (a) a sealed non-empty blob streams back and reassembles byte-equal, chunk-bounded at 128 KiB (seed a ~300 KB blob → 3 chunks); (b) an unknown id throws `RpcException` NotFound; (c) a non-sealed row throws NotFound; (d) a zero-length blob throws NotFound (the #485 serve-side guard).
|
||||
**Step 2:** run → FAIL.
|
||||
**Step 3:** implement.
|
||||
**Step 4:** run → PASS. **Sabotage:** drop the `row.Status != Sealed` check → the non-sealed test reddens; drop the `ArtifactBlob.Length == 0` check → the zero-length test reddens.
|
||||
**Step 5: Commit** `feat(mesh): central DeploymentArtifactService — stream the sealed artifact, NotFound-hide the rest`
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Central serve auth interceptor + dedicated h2c listener wiring
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** none
|
||||
**Blocked by:** Task 0, Task 2
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/ConfigServeAuthInterceptor.cs`
|
||||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` (register the interceptor with `AddGrpc`; add the dedicated h2c listener block mirroring the LocalDb sync block at `407-476`; `MapGrpcService<DeploymentArtifactService>()` inside a `hasAdmin && configServeGrpcPort > 0` guard beside `514`)
|
||||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/ConfigServeAuthInterceptorTests.cs`
|
||||
|
||||
**Interceptor** — mirror `LocalDbSyncAuthInterceptor` exactly, scoped by service path `"/deployment_artifact.v1.DeploymentArtifactService/"`, fail-closed (no `ConfigServe:ApiKey` ⇒ reject ALL with `Unauthenticated`), `Authorization: Bearer <key>`, `FixedTimeEquals`. Calls to other services pass through untouched (so it can share `AddGrpc` with the LocalDb sync interceptor).
|
||||
|
||||
**Listener** — the artifact service is h2c and cannot share the cleartext HTTP/1 port (same reason the LocalDb sync listener is dedicated). Add, right after the LocalDb sync listener block:
|
||||
```csharp
|
||||
var configServeGrpcPort = builder.Configuration.GetValue<int>("ConfigServe:GrpcListenPort");
|
||||
if (hasAdmin && configServeGrpcPort > 0)
|
||||
{
|
||||
// Same dedicated-h2c-listener dance as the LocalDb sync listener above: h2c can't negotiate on a
|
||||
// cleartext Http1AndHttp2 port, so bind a dedicated HTTP/2-only port and re-apply existing bindings.
|
||||
// (Refactor the LocalDb block's existing-binding computation into a shared local if both run — a
|
||||
// node that is admin+driver with BOTH ports set must re-apply existing bindings once and add both.)
|
||||
builder.WebHost.ConfigureKestrel(kestrel =>
|
||||
kestrel.ListenAnyIP(configServeGrpcPort, o => o.Protocols = HttpProtocols.Http2));
|
||||
Log.Information("Config-serve artifact gRPC listener bound on :{Port} (h2c).", configServeGrpcPort);
|
||||
}
|
||||
```
|
||||
> **Load-bearing interaction with the LocalDb listener.** A fused central node is admin **and** driver; on the rig it may have `LocalDb:SyncListenPort` set too. `ConfigureKestrel` is additive across calls, but the LocalDb block re-applies `existingBindings` (URLS/HTTP_PORTS) and this block must NOT clobber that. Implement by computing `existingBindings` once and applying it once, then adding whichever of the two dedicated h2c ports are configured. The test below asserts the AdminUI HTTP port still answers when both dedicated ports are set.
|
||||
|
||||
`MapGrpcService`:
|
||||
```csharp
|
||||
if (hasAdmin && configServeGrpcPort > 0)
|
||||
app.MapGrpcService<DeploymentArtifactService>();
|
||||
```
|
||||
|
||||
**Step 1: Write the failing test** — boot the Host (or a minimal `WebApplicationFactory` with `hasAdmin`, `ConfigServe:GrpcListenPort` set, `ConfigServe:ApiKey=k`, a seeded sealed deployment): a `Grpc.Net.Client` channel to the dedicated port with `Authorization: Bearer k` fetches + reassembles the blob; **no header** ⇒ `Unauthenticated`; **wrong key** ⇒ `Unauthenticated`; and the AdminUI HTTP surface (`/health/active`) still returns 200 (listener coexistence). Use `TestContext.Current.CancellationToken`.
|
||||
**Step 2:** run → FAIL.
|
||||
**Step 3:** implement interceptor + Program.cs wiring.
|
||||
**Step 4:** run → PASS. **Sabotage:** make the interceptor return without checking when the key is unset (i.e. break fail-closed) → the no-header test reddens; break the listener coexistence (apply only this block's binding) → the `/health/active` assertion reddens.
|
||||
**Step 5: Commit** `feat(mesh): config-serve auth interceptor + dedicated h2c listener (fail-closed bearer)`
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Node-side `IDeploymentArtifactFetcher` + gRPC client (SHA-verified, failover)
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 2
|
||||
**Blocked by:** Task 1
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/IDeploymentArtifactFetcher.cs`
|
||||
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/GrpcDeploymentArtifactFetcher.cs`
|
||||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Deployment/GrpcDeploymentArtifactFetcherTests.cs`
|
||||
|
||||
**Design.**
|
||||
```csharp
|
||||
public interface IDeploymentArtifactFetcher
|
||||
{
|
||||
/// <summary>Fetches + reassembles the artifact for <paramref name="deploymentId"/>, verifying
|
||||
/// SHA-256(bytes) == <paramref name="expectedRevisionHash"/>. Returns the bytes, or <see
|
||||
/// langword="null"/> on any failure (all endpoints unreachable, NotFound, SHA mismatch,
|
||||
/// zero-length) — a null is "no answer," NEVER an empty config (#485).</summary>
|
||||
Task<byte[]?> FetchAsync(string deploymentId, string expectedRevisionHash, CancellationToken ct);
|
||||
}
|
||||
```
|
||||
`GrpcDeploymentArtifactFetcher` takes `IOptions<ConfigSourceOptions>`. For each endpoint in order: open a `GrpcChannel` (h2c: `new HttpClientHandler` / `GrpcChannelOptions` with the `http://` address), attach `Authorization: Bearer {ApiKey}` via a `CallCredentials` or a request header on the call, stream `Fetch`, accumulate `chunk.Data` into a buffer, then `SHA-256(buffer)` hex-lowercase and compare to `expectedRevisionHash`. On success return the bytes. On `RpcException` (NotFound / Unavailable / deadline), log and try the next endpoint. If all fail, or the reassembled buffer is empty, or the SHA mismatches, return `null` (and log a Warning naming the reason). Deadline = `FetchTimeoutSeconds`.
|
||||
|
||||
> **Why return null, not throw.** The caller (Task 5) treats null identically to `ReconcileDrivers` returning null today: apply failure, keep last-known-good, do not advance the revision. Throwing would risk an unhandled actor message; a typed "no bytes" keeps the #485 contract explicit.
|
||||
|
||||
**Seam for the test.** So the test needs no real second Host, make the gRPC call site injectable: a `Func<string /*endpoint*/, DeploymentArtifactService.DeploymentArtifactServiceClient>` factory defaulted to the real channel builder, overridable in tests with an in-memory client backed by a fake that streams canned chunks. (This mirrors Phase 2's `IMeshClusterClientFactory` seam.)
|
||||
|
||||
**Step 1: Write the failing test** — with a fake client factory: (a) a client streaming chunks whose reassembly hashes to `expectedRevisionHash` → returns the exact bytes; (b) chunks whose hash ≠ expected → returns null; (c) an empty stream → returns null; (d) first endpoint throws `RpcException(Unavailable)`, second streams good bytes → returns the bytes (failover); (e) all endpoints throw → null. Assert the failover case actually consulted endpoint 2 (record calls).
|
||||
**Step 2:** run → FAIL.
|
||||
**Step 3:** implement.
|
||||
**Step 4:** run → PASS. **Sabotage:** drop the SHA-256 comparison → the mismatch test reddens; break failover (return after the first endpoint's exception) → the failover test reddens.
|
||||
**Step 5: Commit** `feat(mesh): node gRPC artifact fetcher — SHA-verified reassembly, endpoint failover, null-on-any-failure`
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Wire fetch-and-cache into the apply path (FetchAndCache mode)
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** none
|
||||
**Blocked by:** Task 4
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs` (add `_configSourceMode` + `_artifactFetcher` fields + ctor params + both `Props` overloads; a new `FetchThenApply` path used from `HandleDispatchFromSteady` / `ApplyAndAck` when `FetchAndCache`)
|
||||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs` (resolve `IOptions<ConfigSourceOptions>` + register `IDeploymentArtifactFetcher` in the `hasDriver` branch; pass both into `DriverHostActor.Props`)
|
||||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/...` (DI: `services.AddSingleton<IDeploymentArtifactFetcher, GrpcDeploymentArtifactFetcher>()` in the driver branch — place beside the `IDeploymentArtifactCache` registration)
|
||||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostActorFetchAndCacheTests.cs`
|
||||
|
||||
**Design.** In `FetchAndCache` mode, on a `DispatchDeployment` whose revision differs from `_currentRevision`:
|
||||
1. `bytes = await _artifactFetcher.FetchAsync(deploymentId, revisionHash, ct)` (message-loop-safe: run via `PipeTo` to self as a `FetchResult(deploymentId, revisionHash, correlation, bytes?)`, NOT a blocking await inside the receive).
|
||||
2. On `bytes == null` → **apply failure**: do not advance `_currentRevision`, `UpsertNodeDeploymentState(Failed)`, `SendAck(Failed)` — identical to today's `ReconcileDrivers`-returned-null path (`1639-1659`). The node keeps serving last-known-good. A re-dispatch retries (revision still not current).
|
||||
3. On success → `StoreAsync(clusterId, deploymentId, revisionHash, bytes)` into the cache, then apply **from `bytes` in hand**: `ReconcileDriversFromBlob(bytes)`, `RebuildAddressSpace(correlation, deploymentId, bytes)` (pass the blob so `OpcUaPublishActor` never reads SQL), `PushDesiredSubscriptionsFromArtifact(bytes)`, `UpsertNodeDeploymentState(Applied)`, `SendAck(Applied)`, `_isRunningFromCache = false`, set `_currentRevision`.
|
||||
|
||||
**Refactor note — do NOT duplicate reconcile logic.** `ReconcileDrivers(DeploymentId)` today reads the blob itself (`1722-1726`). Extract the post-read body into `ReconcileDriversFromBlob(byte[] blob)` (the existing #485 `blob.Length == 0` guard at `1735-1747` moves into it verbatim), and have the `Direct`-mode `ReconcileDrivers` read-then-call it. Then `FetchThenApply` calls `ReconcileDriversFromBlob(bytes)` directly. Same for `PushDesiredSubscriptions` → it already has `PushDesiredSubscriptionsFromArtifact(blob)` (`1908`); reuse it.
|
||||
|
||||
**Idempotency:** before fetching, check the cache — if `IsAlreadyCached(deploymentId, revisionHash)` (add a cheap `GetCurrentAsync` compare, or a new `bool ContainsAsync`), skip the fetch and apply from cache. This makes a re-dispatch of an already-applied revision a no-op fetch (and covers "peer already replicated it to us").
|
||||
|
||||
**Step 1: Write the failing test** — a `DriverHostActor` in `FetchAndCache` mode with a fake `IDeploymentArtifactFetcher` + a real in-memory `LocalDbDeploymentArtifactCache` (the `DriverHostActorArtifactCacheTests` harness): (a) dispatch → fetcher returns good bytes → node applies, cache written, ack Applied, `_currentRevision` advances, and it **never touched the ConfigDb factory** (inject a throwing `IDbContextFactory` to prove no SQL read on the config path — note the `NodeDeploymentState` write is Phase 4, so use a factory that permits the ack write but fails `Deployments` reads, or assert via a spy that `Deployments` was never queried); (b) fetcher returns null → ack Failed, `_currentRevision` unchanged, children kept; (c) re-dispatch of the applied revision → no fetch (idempotent), immediate Applied ack.
|
||||
**Step 2:** run → FAIL.
|
||||
**Step 3:** implement.
|
||||
**Step 4:** run → PASS. **Sabotage:** make the null-bytes branch advance `_currentRevision` anyway → test (b) reddens (this is the #485 apply-failure contract); remove the idempotency check → test (c) reddens.
|
||||
**Step 5: Commit** `feat(mesh): FetchAndCache apply path — fetch→cache→apply-from-bytes, null=apply-failure`
|
||||
|
||||
---
|
||||
|
||||
## Task 6: Redirect all config reads to LocalDb; bootstrap from the pointer
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** none
|
||||
**Blocked by:** Task 5
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs` (`Bootstrap`, `TryRecoverFromStale`, and any residual read seams under `FetchAndCache`)
|
||||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostActorFetchAndCacheBootstrapTests.cs`
|
||||
|
||||
**Design.** In `FetchAndCache` mode:
|
||||
- **`Bootstrap()`** does NOT read `NodeDeploymentStates`/`Deployments` from SQL. Instead: `cached = await _cache.GetCurrentUnkeyedAsync()`. If non-null → `_currentRevision = RevisionHash.Parse(cached.RevisionHash)`, `Become(Steady)`, `ApplyCachedArtifact(new DeploymentId(cached.DeploymentId), cached.Artifact)` (the existing cache-boot path at `725-748`, which reconciles + rebuilds + resubscribes from the blob and does **not** read SQL or re-ack). If null (fresh node, empty cache) → `Become(Steady)` with no revision; the first `DispatchDeployment` triggers a fetch. On a cache-read exception → log and `Become(Steady)` (no revision) rather than `Stale` (Stale meant "SQL down"; in FetchAndCache there is no SQL config read to be down).
|
||||
- **`TryRecoverFromStale()`** is unreachable in `FetchAndCache` (nothing enters Stale from a config read). Leave it for `Direct` mode; guard its entry so `FetchAndCache` never schedules it.
|
||||
- **`OpcUaPublishActor`** is untouched: `FetchThenApply` and `ApplyCachedArtifact` both pass the blob to `RebuildAddressSpace`, so `LoadArtifact`/`LoadLatestArtifact` (the SQL reads) are never hit in `FetchAndCache`. Add an assertion-comment; no code change there.
|
||||
|
||||
**The one SQL touch that remains (by design, Phase 4's to cut):** `UpsertNodeDeploymentState` still writes the ack row. That is a write, not a config read, and the phase boundary is explicit.
|
||||
|
||||
**Step 1: Write the failing test** — `FetchAndCache` mode, throwing `IDbContextFactory` for `Deployments` reads: (a) a cache pre-seeded with an applied artifact → `Bootstrap` restores served state (drivers reconciled, address space rebuilt) with **no** `Deployments` read; (b) an empty cache → `Bootstrap` lands in Steady-no-revision without error and a subsequent dispatch fetches; (c) a cache-read failure → Steady-no-revision, not Stale. Reuse the `DriverHostActorBootFromCacheTests` harness.
|
||||
**Step 2:** run → FAIL.
|
||||
**Step 3:** implement the mode-gated bootstrap branch.
|
||||
**Step 4:** run → PASS. **Sabotage:** in `FetchAndCache`, make `Bootstrap` fall through to the SQL read → test (a)'s throwing-factory makes it red.
|
||||
**Step 5: Commit** `feat(mesh): FetchAndCache bootstrap from the LocalDb pointer; no driver-side config SQL read`
|
||||
|
||||
---
|
||||
|
||||
## Task 7: #485 guard coverage on the new fetch + cache-read paths
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** none
|
||||
**Blocked by:** Task 6
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs` (only if a seam is found unguarded)
|
||||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostActorFetchAndCacheUnreadableTests.cs`
|
||||
|
||||
This task adds **no new mechanism** — it proves the #485 contract holds on every `FetchAndCache` seam, and adds a guard only where a test exposes a gap. The contract: zero-length / SHA-mismatched / unreadable ⇒ no answer, keep last-known-good, fail the apply, never tear down to an empty address space.
|
||||
|
||||
**Step 1: Write the failing tests** (expect some to pass already — that is fine; they are the regression net):
|
||||
- Fetcher returns `null` (all endpoints down) mid-steady-state → address space + drivers + subscriptions UNCHANGED, ack Failed, `_currentRevision` unchanged. (Covered by Task 5(b); re-assert the address-space-kept angle explicitly.)
|
||||
- Central serves a **zero-length** blob (shouldn't happen — Task 2 NotFound-hides it — but prove the client also refuses): fetcher gets an empty stream → `null` → apply failure.
|
||||
- Cache holds a **truncated/corrupt** artifact at boot (drop a chunk row / corrupt base64) → `ReassembleAsync` returns null → `Bootstrap` treats it as empty cache (Steady-no-revision), does NOT apply a partial config.
|
||||
- SHA mismatch (server bytes don't match the dispatched `RevisionHash`) → fetcher `null` → apply failure, last-known-good kept.
|
||||
**Step 2:** run → the corrupt-cache and any unguarded case FAIL if a gap exists; otherwise all green (regression net).
|
||||
**Step 3:** add a guard only where a test is red.
|
||||
**Step 4:** run → PASS. **Sabotage:** temporarily make `ReassembleAsync` return partial bytes on a missing chunk → the corrupt-cache test reddens (proving the guard, not the mock, is what holds the line).
|
||||
**Step 5: Commit** `test(mesh): #485 empty/corrupt/mismatch coverage on the FetchAndCache paths`
|
||||
|
||||
---
|
||||
|
||||
## Task 8: Real two-Host gRPC fetch boundary test + falsifiability control
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 9
|
||||
**Blocked by:** Task 3, Task 4
|
||||
|
||||
**Files:**
|
||||
- Create: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/DeploymentArtifactFetchBoundaryTests.cs`
|
||||
|
||||
The Phase-2 lesson applied to Phase 3: an in-memory fake client proves fetcher **logic**, not that a real gRPC stream crosses a real Kestrel h2c listener through the real interceptor. Stand a real central Host (admin, `ConfigServe:GrpcListenPort` set, `ConfigServe:ApiKey=k`, a seeded sealed deployment) and drive the real `GrpcDeploymentArtifactFetcher` against it.
|
||||
|
||||
Assertions:
|
||||
1. A fetch with the right key + a >256 KB blob reassembles byte-equal and verifies against the deployment's `RevisionHash`.
|
||||
2. **Falsifiability control** — the same fetch with a **wrong key** returns `null` (interceptor rejects; without this control, assertion 1 cannot distinguish "the boundary works" from "something else supplied the bytes"). Verify the control is real by confirming the right-key fetch on the *same* server succeeds.
|
||||
3. An unknown deployment id → `null` (NotFound-hidden).
|
||||
4. Endpoint failover: give the fetcher `[dead-port, real-port]` → it still returns the bytes.
|
||||
|
||||
If irreducibly flaky, quarantine with `[Trait("Category","ArtifactBoundary")]` — do **not** weaken to the in-memory fake (that is Task 4's job).
|
||||
|
||||
**Step 5: Commit** `test(mesh): real two-Host gRPC artifact-fetch boundary test with wrong-key control`
|
||||
|
||||
---
|
||||
|
||||
## Task 9: Rig config + docs
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 8
|
||||
**Blocked by:** Task 5
|
||||
|
||||
**Files:**
|
||||
- Modify: `docker-dev/docker-compose.yml` — central-1/central-2 get `ConfigServe__GrpcListenPort` (e.g. `4055`) + `ConfigServe__ApiKey` (dev value, the committed-dev-secret exception); all six get `ConfigSource__Mode: "${OTOPCUA_CONFIG_MODE:-Direct}"`; site nodes get `ConfigSource__CentralFetchEndpoints__0/1: http://central-1:4055 / http://central-2:4055` + a matching `ConfigSource__ApiKey`. Leave `Mode=Direct` so the rig comes up unchanged; expose the flip via `OTOPCUA_CONFIG_MODE=FetchAndCache`.
|
||||
- Modify: `docs/Configuration.md` — new `ConfigSource` + `ConfigServe` sections (the dark switch, the shared-key auth, the h2c dedicated-port requirement, the env-only key rule).
|
||||
- Modify: `docs/Redundancy.md` — extend the "Command transport" section: config bytes travel out-of-band over the artifact gRPC stream, both pair nodes fetch independently (idempotent + replicated), central serves from SQL.
|
||||
- Modify: `docs/plans/2026-07-22-per-cluster-mesh-program.md` — flip the Phase 3 row + section to reflect the gRPC/shared-key choices.
|
||||
- Modify: `CLAUDE.md` — a short "Config source (Phase 3)" note beside the Mesh command transport note.
|
||||
|
||||
**Step 5: Commit** `docs(mesh): ConfigSource/ConfigServe config, rig flip switch, Phase 3 as shipped`
|
||||
|
||||
---
|
||||
|
||||
## Task 10: Live gate on docker-dev
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min (plus rig time)
|
||||
**Parallelizable with:** none
|
||||
**Blocked by:** Task 8, Task 9
|
||||
|
||||
**Files:**
|
||||
- Create: `docs/plans/2026-07-22-mesh-phase3-live-gate.md`
|
||||
|
||||
Flip only the **site** nodes to `FetchAndCache` (`OTOPCUA_CONFIG_MODE=FetchAndCache` scoped to site services, or a compose override) so central keeps SQL and the rig models the target topology (site nodes have no SQL). Central stays `Direct`. Rig AdminUI login is disabled — drive UI steps via browser automation at `http://localhost:9200`; deploy via `POST /api/deployments` with `X-Api-Key: docker-dev-deploy-key`.
|
||||
|
||||
| # | Step | Pass condition |
|
||||
|---|---|---|
|
||||
| 1 | Rig up all-`Direct`; deploy | Seals green — baseline |
|
||||
| 2 | Flip site nodes to `FetchAndCache`, restart them, deploy | Seals green; **site node logs show a gRPC fetch + cache write + apply-from-bytes**, and **no `Deployments.ArtifactBlob` SQL read on the site side** |
|
||||
| 3 | Confirm central served it | central logs the `DeploymentArtifactService.Fetch` call; site logs the SHA-256 verify pass |
|
||||
| 4 | **Stop central SQL, deploy** (the phase's headline gate) | The site fetch fails (central can't read SQL to serve) → site logs apply-failure, **keeps serving last-known-good**, ack Failed → deploy TimedOut naming the site nodes; **no crash, no empty address space** (#485) |
|
||||
| 5 | Restart SQL, redeploy | Fetch lands, seals green — retry semantics on the new path |
|
||||
| 6 | Restart a site node with a warm cache, central up | Boots from the LocalDb pointer, restores served state, **no `Deployments` read** |
|
||||
| 7 | Restart a site node with central DOWN | Boots last-known-good from cache; no crash loop (config path needs no central at boot) |
|
||||
| 8 | Corrupt one cached artifact chunk on a site node, restart it | Reassembly rejects it → boots Steady-no-revision (empty-cache path), does NOT apply a partial config; next deploy re-fetches |
|
||||
| 9 | Wrong `ConfigSource__ApiKey` on one site node, deploy | That node's fetch is rejected (Unauthenticated) → apply-failure + Warning; the other site node (right key) applies |
|
||||
| 10 | Both site nodes fetch the same deploy | Idempotent — the second store is a no-op (`IsAlreadyCached`); pair replication carries the artifact even to a node that missed its own fetch |
|
||||
|
||||
Step 4 is the headline: it proves central-SQL independence at the driver **and** the #485 last-known-good contract on the fetch path simultaneously. Record actual output for every step, especially anything that fails.
|
||||
|
||||
**Step 5: Commit** `docs(mesh): Phase 3 live-gate record`
|
||||
|
||||
---
|
||||
|
||||
## Risks specific to this phase
|
||||
|
||||
- **First in-repo proto.** The `Grpc.Tools` codegen path (Task 1) is new here; a restore/build ordering or `Directory.Packages.props` pin miss will surface as CS-missing-type, not a proto error. Build Commons in isolation first.
|
||||
- **The dedicated-h2c-listener interaction.** A fused central node with BOTH the LocalDb sync port and the ConfigServe port set must re-apply existing HTTP bindings exactly once; getting this wrong silently moves the AdminUI off its port (the exact failure the LocalDb block's comments warn about). Task 3's coexistence assertion is the guard.
|
||||
- **`RevisionHash == SHA-256(blob)` is load-bearing.** If a future change makes the revision hash something other than the raw-bytes SHA, the fetch verification breaks silently (every fetch → null → every deploy fails). Task 4's mismatch test pins the equality; note it in `ConfigComposer` if touched.
|
||||
- **Blocking in the actor loop.** The fetch is I/O; it MUST run via `PipeTo(self)`, never an awaited call inside a receive (would freeze the mailbox for `FetchTimeoutSeconds`). Task 5.
|
||||
- **Phase-boundary discipline.** Phase 3 removes config **reads** only. The `NodeDeploymentState` write, `DbHealthProbe`, and `EfAlarmConditionStateStore` still touch SQL by design — resist folding Phase 4 in, or the live gate's "no SQL" scope becomes unprovable-in-parts.
|
||||
- **Both-nodes-fetch is deliberate.** Do not gate the fetch on the Primary role to "save a fetch"; it couples config delivery to redundancy timing and diverges from today's both-read behaviour. Idempotency + replication already make the second fetch cheap.
|
||||
@@ -0,0 +1,143 @@
|
||||
{
|
||||
"planPath": "docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md",
|
||||
"note": "Per-cluster mesh Phase 3. Decisions pre-made by the user: gRPC fetch RPC (NOT token-gated HTTP); shared node key (NOT per-deployment token, so no migration and DispatchDeployment is UNCHANGED); config-flagged dark switch ConfigSource:Mode=Direct default; both pair nodes fetch (no Primary gating). Five recon facts + the phase-boundary rule (Phase 3 = config READS only; NodeDeploymentState write / DbHealthProbe / EfAlarmConditionStateStore stay for Phase 4) are in the plan header. RevisionHash == SHA-256(blob) is load-bearing for fetch verification.",
|
||||
"tasks": [
|
||||
{
|
||||
"id": 0,
|
||||
"subject": "Task 0: ConfigSource/ConfigServe options + validator + registration",
|
||||
"status": "completed",
|
||||
"classification": "small",
|
||||
"parallelizableWith": [
|
||||
1
|
||||
],
|
||||
"commit": "9d1e60c0",
|
||||
"note": "One sabotage (accept empty endpoints) reddened its test; file was untracked so restored by hand not git checkout."
|
||||
},
|
||||
{
|
||||
"id": 1,
|
||||
"subject": "Task 1: deployment_artifact.v1 proto + first in-repo Grpc.Tools codegen",
|
||||
"status": "completed",
|
||||
"classification": "standard",
|
||||
"parallelizableWith": [
|
||||
0
|
||||
],
|
||||
"commit": "62e8f54f",
|
||||
"note": "First locally-compiled proto in the repo. Grpc.Tools 2.76.0 pinned (aligned with the other Grpc 2.76.0 refs); GrpcServices=Both. Commons builds clean, codegen verified."
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"subject": "Task 2: central DeploymentArtifactService \u2014 stream sealed artifact, NotFound-hide the rest",
|
||||
"status": "completed",
|
||||
"classification": "standard",
|
||||
"blockedBy": [
|
||||
1
|
||||
],
|
||||
"parallelizableWith": [
|
||||
4
|
||||
],
|
||||
"commit": "7d7de482",
|
||||
"note": "Impl RENAMED to DeploymentArtifactGrpcService \u2014 the generated container class is itself named DeploymentArtifactService and its nested base collided. Aliased the base. Sabotage dropping the sealed+empty guard reddened 2 tests."
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"subject": "Task 3: config-serve auth interceptor + dedicated h2c listener wiring",
|
||||
"status": "completed",
|
||||
"classification": "high-risk",
|
||||
"blockedBy": [
|
||||
0,
|
||||
2
|
||||
],
|
||||
"commit": "4d88f67c",
|
||||
"note": "Merged the LocalDb-sync + ConfigServe listener blocks into ONE that re-applies the existing HTTP surface exactly once and adds both dedicated h2c ports; a fused admin+driver node double-binding the HTTP port throws 'address already in use' (sabotage-proven). AddGrpc moved out of the hasDriver block to hasDriver||hasAdmin with both path-scoped interceptors. StatusCode.Unauthenticated (LocalDb uses PermissionDenied). Coexistence proven by a minimal-WebApplication shape test, mirroring LocalDbSyncListenerTests; real end-to-end fetch is Task 8."
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"subject": "Task 4: node gRPC artifact fetcher \u2014 SHA-verified reassembly, endpoint failover, null-on-any-failure",
|
||||
"status": "completed",
|
||||
"classification": "standard",
|
||||
"blockedBy": [
|
||||
1
|
||||
],
|
||||
"parallelizableWith": [
|
||||
2
|
||||
],
|
||||
"commit": "7a8fb5f1",
|
||||
"note": "Uniform invariant: return the first endpoint's bytes that reassemble AND verify (SHA-256 lowercase-hex == dispatched RevisionHash); else null. RpcException / empty-stream / hash-mismatch all -> try-next, never throw. Func<endpoint,client> seam keeps gRPC out of unit tests (fake subclasses the generated client's protected ctor). Namespace is .DeploymentCache NOT .Deployment (else it shadows Configuration.Entities.Deployment across the whole test assembly via enclosing-namespace lookup) \u2014 same trap LocalDbDeploymentArtifactCache already dodged. Added Grpc.Net.Client to Runtime.csproj. xUnit v2 here: CancellationToken.None, not TestContext.Current."
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"subject": "Task 5: FetchAndCache apply path \u2014 fetch->cache->apply-from-bytes, null=apply-failure",
|
||||
"status": "completed",
|
||||
"classification": "high-risk",
|
||||
"blockedBy": [
|
||||
4
|
||||
],
|
||||
"commit": "648f173b",
|
||||
"note": "DispatchDeployment (rev differs) in FetchAndCache -> BeginFetchAndCacheApply PipeTo(Self) FetchedForApply(bytes?), never a blocking await in a receive. HandleFetchedForApply applies synchronously mirroring ApplyAndAck: null bytes = FAILURE (keep last-known-good, don't advance rev, ack Failed); good bytes reconcile+rebuild+push FROM the bytes (OpcUaPublishActor never reads SQL) then cache. A faulted fetch task maps to null via PipeTo failure:. Extracted ReconcileDriversFromBlob (Direct/FetchAndCache/boot all share it; #485 empty guard moved in); ApplyCachedArtifact now reuses it. Props gains fetchAndCacheMode+artifactFetcher LAST. Idempotent re-dispatch is the existing _currentRevision short-circuit (no separate cache-contains check added; peer-replication-missed-fetch is covered by Task 6 boot + StoreAsync idempotency). Sabotages: advance-rev-on-null reddens test b; drop _currentRevision guard reddens test c."
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"subject": "Task 6: FetchAndCache bootstrap from the LocalDb pointer; no driver-side config SQL read",
|
||||
"status": "completed",
|
||||
"classification": "high-risk",
|
||||
"blockedBy": [
|
||||
5
|
||||
],
|
||||
"commit": "6d004738",
|
||||
"note": "Bootstrap() branches to BootstrapFromCache() in FetchAndCache: GetCurrentUnkeyedAsync -> set _currentRevision -> ApplyCachedArtifact (no SQL, no re-ack). _isRunningFromCache STAYS false (normal op, not degraded). Empty/unreadable cache -> Steady-no-revision (first dispatch fetches), NEVER Stale (Stale = 'SQL down, retry'; FetchAndCache has no config SQL read to recover). TryRecoverFromStale gains a defensive FetchAndCache early-return (unreachable anyway \u2014 retry-db timer only starts in Stale). KEY test lever: UpsertNodeDeploymentState SWALLOWS db failures, so a FetchAndCache dispatch runs end-to-end under a ThrowingDbFactory \u2014 the ack proves Steady-not-Stale AND no-SQL-at-boot in one shot. NodeDiagnosticsSnapshot lives in Commons.Interfaces not Messages.Fleet."
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"subject": "Task 7: #485 empty/corrupt/mismatch coverage on the FetchAndCache paths",
|
||||
"status": "completed",
|
||||
"classification": "high-risk",
|
||||
"blockedBy": [
|
||||
6
|
||||
],
|
||||
"commit": "fc568ae0",
|
||||
"note": "Regression net, no new mechanism \u2014 all green (no gaps to guard). Pins the NEGATIVE on failure paths: (1) null fetch mid-steady-state -> no RebuildAddressSpace for the failed rev, revision + served state kept (publishProbe.ExpectNoMsg is the discriminator); (2) zero-length blob handed to the actor fails via ReconcileDriversFromBlob's own #485 guard (independent of the fetcher null contract); (3) corrupt cache at boot = GetCurrentUnkeyedAsync null -> Steady-no-revision, no rebuild at boot (no partial apply). Removing the #485 empty guard reddens the zero-length test. Corrupt-cache uses a stub-returns-null (the real ReassembleAsync-null-on-corrupt is LocalDbDeploymentArtifactCacheTests' job)."
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"subject": "Task 8: real two-Host gRPC artifact-fetch boundary test + wrong-key falsifiability control",
|
||||
"status": "completed",
|
||||
"classification": "high-risk",
|
||||
"blockedBy": [
|
||||
3,
|
||||
4
|
||||
],
|
||||
"parallelizableWith": [
|
||||
9
|
||||
],
|
||||
"commit": "ffbcaa93",
|
||||
"note": "Minimal real Host (WebApplication + h2c-only Kestrel + real ConfigServeAuthInterceptor + real DeploymentArtifactGrpcService + in-memory EF sealed deployment) driven by the REAL GrpcDeploymentArtifactFetcher. 4 green: right-key >256KB reassembles byte-equal + verifies; wrong-key null WHILE right-key succeeds on the SAME server (falsifiability control); unknown-id null; [dead-port,real-port] failover. h2c GrpcChannel.ForAddress(http://) works out of the box in .NET 10 \u2014 no AppContext switch needed. async can't have out params -> StartServerAsync returns a tuple. Tagged [Trait Category=ArtifactBoundary]."
|
||||
},
|
||||
{
|
||||
"id": 9,
|
||||
"subject": "Task 9: docker-dev rig config + docs + program-plan flip",
|
||||
"status": "completed",
|
||||
"classification": "small",
|
||||
"blockedBy": [
|
||||
5
|
||||
],
|
||||
"parallelizableWith": [
|
||||
8
|
||||
],
|
||||
"commit": "cf6110d0",
|
||||
"note": "Centrals: ConfigServe :4055 + committed dev key, ConfigSource:Mode=Direct (they never fetch). Sites: ConfigSource endpoints (both centrals) + matching key, Mode=${OTOPCUA_CONFIG_MODE:-Direct}. One env var flips ONLY the sites because centrals hard-code Direct. Docs: Configuration.md ConfigSource/ConfigServe section, Redundancy.md out-of-band note, program plan Phase 3 CODE-COMPLETE + tracking row, CLAUDE.md Config-source section."
|
||||
},
|
||||
{
|
||||
"id": 10,
|
||||
"subject": "Task 10: live gate on docker-dev (site nodes -> FetchAndCache; step 4 = stop central SQL mid-fetch = the headline gate)",
|
||||
"status": "completed",
|
||||
"classification": "high-risk",
|
||||
"blockedBy": [
|
||||
8,
|
||||
9
|
||||
],
|
||||
"commit": "(gate doc) 2026-07-22-mesh-phase3-live-gate.md",
|
||||
"note": "PASSED \u2014 and CAUGHT A REAL DEFECT every offline test missed: DeploymentArtifactGrpcService gated on Status==Sealed, but central seals only AFTER all nodes ack, and a FetchAndCache node can't ack until it fetches => seal-needs-ack->ack-needs-fetch->fetch-needs-sealed DEADLOCK. Fetch reached central + passed the interceptor, got clean NotFound, #485 kept last-known-good, deploy never sealed. Fixed by dropping the Sealed gate (serve any non-empty blob; Direct reads the same AwaitingApplyAcks row). Task 2 test flipped to non-sealed-with-blob-is-served. After fix: fetch+verify+apply+seal green 6/6. HEADLINE step 4 PASS: SQL stopped, sites keep serving, restarted site boots last-applied config from the LocalDb pointer, no crash, never Stale. Findings: Kestrel takeover+AdminUI coexist live on central (:9000+:4055); transient Direct SQL-reconnect blip on central-1 is #486 not a regression; failover to central-2 works; deploy POST needs Content-Type:application/json (else 404). Build: Dockerfile build stage pinned to linux/amd64 \u2014 Grpc.Tools linux_arm64 protoc segfaults (139) on Apple-Silicon Docker; portable-IL publish keeps runtime native arm64. Steps 8/9 folded into Task7/Task8/Task3 real coverage rather than slow emulated recreate."
|
||||
}
|
||||
],
|
||||
"lastUpdated": "2026-07-23T01:00:00Z"
|
||||
}
|
||||
@@ -0,0 +1,98 @@
|
||||
# Per-cluster mesh Phase 3 — live gate record
|
||||
|
||||
**Date:** 2026-07-23 · **Rig:** `docker-dev` (six-node single mesh) · **Branch:** `feat/mesh-phase3`
|
||||
**Plan:** [`2026-07-22-mesh-phase3-config-fetch-and-cache.md`](2026-07-22-mesh-phase3-config-fetch-and-cache.md)
|
||||
|
||||
**Result: PASSED** — the FetchAndCache config path works end-to-end on the real rig, and the gate
|
||||
**caught one real defect** (a deploy-sealing deadlock) that every offline test had missed. The defect
|
||||
was fixed and pinned before the gate was re-run green. Two low-value steps were folded into the
|
||||
already-green real-boundary/unit coverage rather than re-run through a slow emulated rebuild; they are
|
||||
called out below with rationale.
|
||||
|
||||
Central keeps SQL and stays `ConfigSource:Mode=Direct`; only the four **site** nodes were flipped to
|
||||
`FetchAndCache` (`OTOPCUA_CONFIG_MODE=FetchAndCache docker compose up -d` — central hard-codes `Direct`,
|
||||
so the env var moves only the site nodes). Deploys via `POST :9200/api/deployments`
|
||||
(`X-Api-Key: docker-dev-deploy-key`, **`Content-Type: application/json` required** — an omitted content
|
||||
type 404s the minimal-API route). Revision bumps by renaming the single seeded `UnsArea` row.
|
||||
|
||||
---
|
||||
|
||||
## Result table
|
||||
|
||||
| # | Step | Result |
|
||||
|---|---|---|
|
||||
| 1 | Rig up all-`Direct`; deploy | **PASS** — `36e5cdd2` Sealed (Status 2), 6/6 Applied — baseline |
|
||||
| 2 | Flip site nodes to `FetchAndCache`, restart, deploy | **PASS after the fix** — `59399bd8` Sealed, 6/6 Applied; site logs *"Fetched + verified artifact … from central-1:4055 (13387 bytes)"* → *"applied fetched deployment"* |
|
||||
| 3 | Confirm central served it | **PASS** — central logs *"Served artifact for 59399bd8… (13387 bytes, 1 chunks)"*; site logs the *verified* (SHA-256-checked) reassembly |
|
||||
| 4 | **Stop central SQL** — driver independence + #485 | **PASS (headline)** — site nodes keep serving (`restarts=0`); a restarted site-b-1 boots its **last-applied** config (`59399bd8` / rev `00ac18b7`) from the LocalDb pointer with SQL down, no crash, never Stale |
|
||||
| 5 | Restart SQL, redeploy | **PASS** — retry lands green (`d6f71cc2` Sealed, 6/6) after one transient Direct-mode blip on central-1 (see finding B) |
|
||||
| 6 | Restart a site node, central up — boot from pointer | **PASS** — every FetchAndCache boot logs *"restored served state … from the LocalDb pointer"*, no `Deployment` read |
|
||||
| 7 | Restart a site node with central DOWN | **PASS** — folded into step 4 (site-b-1 restarted while SQL was stopped) |
|
||||
| 8 | Corrupt a cached chunk → boot Steady-no-revision | **NOT RE-RUN on rig** — covered by `DriverHostActorFetchAndCacheUnreadableTests` (corrupt cache ⇒ `GetCurrentUnkeyedAsync` null ⇒ Steady-no-revision, no partial apply). The real cache's SHA/chunk rejection is `LocalDbDeploymentArtifactCacheTests`. |
|
||||
| 9 | Wrong `ConfigSource:ApiKey` on one site → fetch rejected | **NOT RE-RUN on rig** — the real mechanism is proven by `DeploymentArtifactFetchBoundaryTests` (real Kestrel h2c + real interceptor + wrong key ⇒ null) and `ConfigServeAuthInterceptorTests` (fail-closed); the actor's response to a null fetch is step 4's evidence |
|
||||
| 10 | Both pair nodes fetch the same deploy | **PASS** — both site-a and both site-b nodes fetched independently in step 2; central logged a `Served artifact` per fetch; `StoreAsync` is idempotent |
|
||||
|
||||
---
|
||||
|
||||
## The defect the gate caught — deploy-sealing deadlock (fixed)
|
||||
|
||||
**Symptom.** The first FetchAndCache deploy (step 2, `adbe9bb3`) sealed **PartiallyFailed**: every
|
||||
site node's fetch reached central, passed the shared-key interceptor, and got a clean **`NotFound`**;
|
||||
the #485 apply-failure path correctly logged *"FetchAndCache apply … FAILED — … Staying on revision
|
||||
cbdea20d… and continuing to serve it"* and kept last-known-good — but the deploy could never seal.
|
||||
|
||||
**Root cause.** `DeploymentArtifactGrpcService` gated on `Status == Sealed`. But central seals a
|
||||
deployment **only after every node acks Applied**, and a FetchAndCache node **cannot ack until it has
|
||||
fetched** the artifact:
|
||||
|
||||
> seal-needs-ack → ack-needs-fetch → **fetch-needs-sealed** → deadlock.
|
||||
|
||||
Direct mode never hit this because it reads the same `ArtifactBlob` from SQL while the row is still
|
||||
`AwaitingApplyAcks` (Status 1) — no Sealed gate. Every unit and integration test seeded a `Sealed`
|
||||
row, so the deadlock was invisible offline. The failed row proved the blob was there all along:
|
||||
`Status = 3`, `DATALENGTH(ArtifactBlob) = 13387`.
|
||||
|
||||
**Fix** (`fix(mesh): serve the artifact regardless of deployment status`). Drop the Sealed gate —
|
||||
serve any deployment whose blob is non-empty; unknown-id / empty-blob still collapse to `NotFound`
|
||||
(existence-hiding + #485). Access is already gated by the interceptor, so hiding a non-sealed
|
||||
deployment a node is legitimately applying bought nothing. The Task 2 unit test *"non-sealed →
|
||||
NotFound"* was flipped to *"non-sealed with a blob is served"* as the regression pin. After the fix,
|
||||
step 2 re-ran green (`59399bd8` Sealed, 6/6).
|
||||
|
||||
**Lesson.** "Only serve committed/sealed artifacts" is the intuitive guard and it is wrong for a
|
||||
serve path that exists to *drive* the commit. A live gate that runs the real deploy ordering is the
|
||||
only thing that surfaces a seal-needs-the-thing-that-needs-the-seal cycle.
|
||||
|
||||
---
|
||||
|
||||
## Findings
|
||||
|
||||
- **A — the Kestrel takeover now runs on central, live, and coexists.** Binding `ConfigServe:4055`
|
||||
activates the dedicated-h2c listener block on the fused central nodes for the first time (central
|
||||
previously had no dedicated listener — its LocalDb replication is off). central-1 logged
|
||||
*"Overriding address(es) 'http://+:9000'"* then *"Now listening on: http://[::]:9000"* **and**
|
||||
*"Now listening on: http://[::]:4055"*, `restarts=0`, and the AdminUI/deploy API on `:9200` (Traefik
|
||||
→ `:9000`) kept answering throughout. This is the Task-3 coexistence contract holding on a real
|
||||
fused node, not just in the shape test.
|
||||
- **B — a transient Direct-mode SQL-reconnect blip is not a Phase 3 regression.** In step 5, the deploy
|
||||
immediately after SQL restarted sealed PartiallyFailed with **central-1** (Direct) failing its blob
|
||||
read (*"ConfigDb unreachable"*) — its EF pool still held broken connections. The #485 guard did
|
||||
exactly the right thing (apply-failure, keep last-known-good, ack Failed), and a redeploy sealed
|
||||
green. This is the pre-existing #486 behaviour firing on a reconnect edge, unrelated to FetchAndCache.
|
||||
- **C — endpoint failover works live.** site-b-1's step-5 fetch succeeded *from central-2*
|
||||
(`Fetched + verified … from http://central-2:4055`) — the fetcher rotated past central-1 to the
|
||||
second configured endpoint exactly as `GrpcDeploymentArtifactFetcher` intends.
|
||||
- **D — `Content-Type: application/json` is mandatory on the deploy POST.** Without it the minimal-API
|
||||
route 404s (not 415). The baseline call carried it; a later batch of retries dropped it and 404'd
|
||||
five times before the header was restored. Rig-operational, not a Phase 3 concern, but recorded so
|
||||
the next operator does not chase a phantom outage.
|
||||
|
||||
---
|
||||
|
||||
## Build note (rig enablement)
|
||||
|
||||
The rig image could not build until the Dockerfile's **build stage was pinned to `linux/amd64`**
|
||||
(`fix(docker-dev): pin the build stage to linux/amd64`). Phase 3 added the repo's first in-repo
|
||||
`.proto`, and `Grpc.Tools`' bundled `linux_arm64` `protoc` **segfaults (exit 139)** on Apple-Silicon
|
||||
Docker. `dotnet publish` is framework-dependent (portable IL + every RID's native assets), so pinning
|
||||
only the build stage leaves the runtime image native arm64; just the one-time build is emulated.
|
||||
@@ -141,16 +141,30 @@ admin-change refresh), clusters know central from appsettings (static, restart t
|
||||
flow over ClusterClient (DPS paths deleted or dark-switched), including an Ask timing out cleanly
|
||||
against a stopped node.
|
||||
|
||||
### Phase 3 — Config fetch-and-cache from central
|
||||
**Scope:** the deploy artifact is served **by central** (transport per its own phase plan — the
|
||||
ScadaBridge analogue is token-gated HTTP; decide HTTP vs a gRPC fetch RPC when planning) and
|
||||
cached in LocalDb; driver nodes read config exclusively from LocalDb (boot-from-cache becomes the
|
||||
normal path — design §6.1); chunking, SHA-256 verify, newest-2 retention, pair replication carry
|
||||
over unchanged. **This phase changes a running data path — it gets its own live gate** (design §7
|
||||
note): deploy lands on a driver pair with central SQL stopped mid-fetch → retry lands; #485
|
||||
last-known-good semantics re-proven on the new path.
|
||||
**Exit gate:** live gate green on the rig; a driver node with an empty LocalDb and reachable
|
||||
central boots into the current config; with central down it boots last-known-good.
|
||||
### Phase 3 — Config fetch-and-cache from central — **CODE-COMPLETE 2026-07-23 (Tasks 0–8; live gate Task 10 pending)**
|
||||
**Decisions taken:** the deploy artifact is served **by central over a gRPC fetch RPC**
|
||||
(`DeploymentArtifactService.Fetch`, streamed chunks — chosen over token-gated HTTP), authenticated by
|
||||
a **shared node bearer key** (`ConfigServe:ApiKey == ConfigSource:ApiKey`, `FixedTimeEquals`,
|
||||
fail-closed — chosen over a per-deployment token, so **no migration** and `DispatchDeployment` stays
|
||||
payload-free). Cutover is a **config-flagged dark switch** `ConfigSource:Mode` (`Direct` default |
|
||||
`FetchAndCache`), per node. Plan + task ledger:
|
||||
[`docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md`](2026-07-22-mesh-phase3-config-fetch-and-cache.md).
|
||||
**Shipped (all green, sabotage-checked):** `ConfigSource`/`ConfigServe` options + validator; the repo's
|
||||
first in-repo `.proto` + `Grpc.Tools` codegen; central `DeploymentArtifactGrpcService` (streams the
|
||||
sealed blob, NotFound-hides unknown/not-sealed/empty); `ConfigServeAuthInterceptor` + a dedicated h2c
|
||||
Kestrel listener merged with the LocalDb-sync listener (re-binds the existing surface once, adds both
|
||||
dedicated ports); node `GrpcDeploymentArtifactFetcher` (SHA-verified reassembly, endpoint failover,
|
||||
null-on-any-failure); the `FetchAndCache` apply path (`PipeTo(Self)`, fetch→cache→apply-from-bytes,
|
||||
null = apply-failure keeping last-known-good) and bootstrap-from-the-LocalDb-pointer (no driver-side
|
||||
config SQL read); #485 coverage; and a **real two-Host gRPC boundary test** (right-key reassembles,
|
||||
wrong-key control returns null, failover). Chunking, SHA-256 verify, newest-2 retention, pair
|
||||
replication carry over unchanged. **Phase boundary held:** config **reads** only — the
|
||||
`NodeDeploymentState` ack-row write, `DbHealthProbe`, `EfAlarmConditionStateStore` stay for Phase 4.
|
||||
**Remaining (Tasks 9–10):** rig config + docs (done) and the live gate — deploy on a site pair flipped
|
||||
to `FetchAndCache` with central SQL stopped mid-fetch → retry lands; #485 last-known-good re-proven on
|
||||
the new path.
|
||||
**Exit gate:** live gate green on the rig; a driver node with an empty LocalDb and reachable central
|
||||
boots into the current config; with central down it boots last-known-good.
|
||||
|
||||
### Phase 4 — Cut the driver-side ConfigDb connection
|
||||
**Scope (design §6.1 audit table):** re-home `EfAlarmConditionStateStore` to LocalDb (pair-local
|
||||
@@ -229,7 +243,7 @@ resource sizing on the site VMs should be checked once both products run the ful
|
||||
| Prereq: selfform-fallback + manual-failover plan | **DONE 2026-07-22**, merged `a78425ea`. Shipped *not* as the planned `SelfFormAfter` watchdog — the live gate caught that islanding a manually-failed-over node — but as self-first seed ordering + `AkkaClusterOptionsValidator`. Phase 6's seed-ordering scope is therefore already discharged. |
|
||||
| 1 ClusterNode columns + DB ack set | **DONE 2026-07-22**, merged `7654f24d`. Deploy-seal semantics changed as flagged, but the escape hatch is **`ClusterNode.MaintenanceMode`**, not `Enabled=0` — the live gate found `Enabled=0` is rejected by `DraftValidator.ValidateClusterTopology` on any 2-node pair. Gate record: `2026-07-22-mesh-phase1-live-gate.md`. |
|
||||
| 2 ClusterClient transport | **DONE 2026-07-22** (live gate PASSED, shipped dark) — shipped dark (`MeshTransport:Mode=Dps` default), both comm actors registered in both modes. Corrections: `SendToAll` not `Send`, one fleet-wide client not one per cluster, `/user/node-communication` not `/user/cluster-communication`. The gate's "Ask timing out" item does not exist to test — no cross-boundary Ask in this phase; see the phase section. |
|
||||
| 3 fetch-and-cache | not started |
|
||||
| 3 fetch-and-cache | **CODE-COMPLETE 2026-07-23** (Tasks 0–8 merged to the phase branch, all green + sabotage-checked; live gate Task 10 pending). gRPC fetch RPC + shared node key; dark switch `ConfigSource:Mode` (Direct default). Rig flip: `OTOPCUA_CONFIG_MODE=FetchAndCache` on the site nodes. See `2026-07-22-mesh-phase3-config-fetch-and-cache.md`. |
|
||||
| 4 cut driver ConfigDb | not started |
|
||||
| 5 gRPC telemetry | not started |
|
||||
| 6 mesh partition + co-location | not started |
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
namespace ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
|
||||
/// <summary>
|
||||
/// Node-side selection of where a driver reads its deployed configuration (per-cluster mesh
|
||||
/// Phase 3). The dark switch that lets a driver node stop reading <c>Deployment.ArtifactBlob</c>
|
||||
/// from central SQL and instead fetch the artifact from central over gRPC and read it from the
|
||||
/// pair-local LocalDb cache.
|
||||
/// </summary>
|
||||
public sealed class ConfigSourceOptions
|
||||
{
|
||||
/// <summary>Configuration section name.</summary>
|
||||
public const string SectionName = "ConfigSource";
|
||||
|
||||
/// <summary>Read config directly from central SQL — today's behaviour, and the default.</summary>
|
||||
public const string ModeDirect = "Direct";
|
||||
|
||||
/// <summary>Fetch the artifact from central over gRPC, cache it in LocalDb, and read from there.</summary>
|
||||
public const string ModeFetchAndCache = "FetchAndCache";
|
||||
|
||||
/// <summary>
|
||||
/// <see cref="ModeDirect"/> (default) or <see cref="ModeFetchAndCache"/>. Any other value
|
||||
/// fails host start.
|
||||
/// </summary>
|
||||
public string Mode { get; set; } = ModeDirect;
|
||||
|
||||
/// <summary>
|
||||
/// Central artifact-gRPC base addresses, e.g. <c>http://central-1:4055</c> (h2c ⇒ the
|
||||
/// <c>http</c> scheme). Tried in order for failover. Required under
|
||||
/// <see cref="ModeFetchAndCache"/>; ignored under <see cref="ModeDirect"/>.
|
||||
/// </summary>
|
||||
public string[] CentralFetchEndpoints { get; set; } = [];
|
||||
|
||||
/// <summary>
|
||||
/// Shared bearer key; must equal central's <see cref="ConfigServeOptions.ApiKey"/>. Supply via
|
||||
/// the environment (<c>ConfigSource__ApiKey</c>) — never commit it. Required under
|
||||
/// <see cref="ModeFetchAndCache"/>.
|
||||
/// </summary>
|
||||
public string ApiKey { get; set; } = string.Empty;
|
||||
|
||||
/// <summary>Per-fetch deadline in seconds. Must be positive under <see cref="ModeFetchAndCache"/>.</summary>
|
||||
public int FetchTimeoutSeconds { get; set; } = 30;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Central-side artifact-serve surface (per-cluster mesh Phase 3). Configures the dedicated h2c
|
||||
/// listener + shared bearer key the <c>DeploymentArtifactService</c> is exposed behind. Only
|
||||
/// admin-role nodes (which hold the central SQL connection) serve.
|
||||
/// </summary>
|
||||
public sealed class ConfigServeOptions
|
||||
{
|
||||
/// <summary>Configuration section name.</summary>
|
||||
public const string SectionName = "ConfigServe";
|
||||
|
||||
/// <summary>
|
||||
/// Dedicated HTTP/2-only (h2c) listener port for the artifact gRPC service. <c>0</c> (the
|
||||
/// default) disables it — nothing is bound. Must differ from the main HTTP port and from
|
||||
/// <c>LocalDb:SyncListenPort</c> (h2c cannot share a cleartext HTTP/1 port).
|
||||
/// </summary>
|
||||
public int GrpcListenPort { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Shared bearer key the serve-side interceptor checks (constant-time, fail-closed: an unset
|
||||
/// key rejects every call). Supply via the environment (<c>ConfigServe__ApiKey</c>) — never
|
||||
/// commit it.
|
||||
/// </summary>
|
||||
public string ApiKey { get; set; } = string.Empty;
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
using Microsoft.Extensions.Options;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
|
||||
/// <summary>
|
||||
/// Fails the host at startup on a <see cref="ConfigSourceOptions"/> shape that would leave a
|
||||
/// driver node unable to fetch its configuration.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Like <see cref="MeshTransportOptionsValidator"/>, every fault caught here otherwise surfaces as
|
||||
/// an <i>absence</i> — a deployment that never applies because the node has no endpoint to fetch
|
||||
/// from or no key to authenticate with. An absence has no stack trace and no failing node to point
|
||||
/// at; refusing to start is cheaper to diagnose.
|
||||
/// </remarks>
|
||||
public sealed class ConfigSourceOptionsValidator : IValidateOptions<ConfigSourceOptions>
|
||||
{
|
||||
/// <inheritdoc />
|
||||
public ValidateOptionsResult Validate(string? name, ConfigSourceOptions options)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
|
||||
var errors = new List<string>();
|
||||
|
||||
var isDirect = string.Equals(options.Mode, ConfigSourceOptions.ModeDirect, StringComparison.OrdinalIgnoreCase);
|
||||
var isFetch = string.Equals(
|
||||
options.Mode, ConfigSourceOptions.ModeFetchAndCache, StringComparison.OrdinalIgnoreCase);
|
||||
|
||||
if (!isDirect && !isFetch)
|
||||
{
|
||||
errors.Add(
|
||||
$"{ConfigSourceOptions.SectionName}:{nameof(ConfigSourceOptions.Mode)} is "
|
||||
+ $"'{options.Mode}'. Expected '{ConfigSourceOptions.ModeDirect}' or "
|
||||
+ $"'{ConfigSourceOptions.ModeFetchAndCache}'.");
|
||||
}
|
||||
|
||||
// The fetch surface is only load-bearing under FetchAndCache. Requiring it under the default
|
||||
// would make the section mandatory everywhere for a feature that is switched off — including on
|
||||
// admin-only nodes, which never fetch.
|
||||
if (isFetch)
|
||||
{
|
||||
if (options.CentralFetchEndpoints.Length == 0)
|
||||
{
|
||||
errors.Add(
|
||||
$"{ConfigSourceOptions.SectionName}:{nameof(ConfigSourceOptions.CentralFetchEndpoints)} "
|
||||
+ "is empty. Under FetchAndCache a driver node cannot reach central to fetch its "
|
||||
+ "configuration, and every deployment would time out naming a node that is running "
|
||||
+ "perfectly.");
|
||||
}
|
||||
|
||||
foreach (var endpoint in options.CentralFetchEndpoints)
|
||||
{
|
||||
if (!endpoint.StartsWith("http://", StringComparison.OrdinalIgnoreCase)
|
||||
&& !endpoint.StartsWith("https://", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
errors.Add(
|
||||
$"Central fetch endpoint '{endpoint}' must start with 'http://' (h2c) or "
|
||||
+ "'https://' — it is a gRPC base address (scheme://host:port), not a bare "
|
||||
+ "host:port.");
|
||||
}
|
||||
}
|
||||
|
||||
if (string.IsNullOrEmpty(options.ApiKey))
|
||||
{
|
||||
errors.Add(
|
||||
$"{ConfigSourceOptions.SectionName}:{nameof(ConfigSourceOptions.ApiKey)} is empty. "
|
||||
+ "Under FetchAndCache the shared bearer key is the whole authentication boundary to "
|
||||
+ "central's artifact service; without it every fetch is rejected. Supply it via the "
|
||||
+ "environment (ConfigSource__ApiKey).");
|
||||
}
|
||||
|
||||
if (options.FetchTimeoutSeconds <= 0)
|
||||
{
|
||||
errors.Add(
|
||||
$"{ConfigSourceOptions.SectionName}:{nameof(ConfigSourceOptions.FetchTimeoutSeconds)} "
|
||||
+ $"must be greater than 0 (was {options.FetchTimeoutSeconds}).");
|
||||
}
|
||||
}
|
||||
|
||||
return errors.Count == 0
|
||||
? ValidateOptionsResult.Success
|
||||
: ValidateOptionsResult.Fail(string.Join(" ", errors));
|
||||
}
|
||||
}
|
||||
@@ -40,6 +40,14 @@ public static class ServiceCollectionExtensions
|
||||
services.AddValidatedOptions<MeshTransportOptions, MeshTransportOptionsValidator>(
|
||||
configuration, MeshTransportOptions.SectionName);
|
||||
|
||||
// Per-cluster mesh Phase 3: where a driver reads its config (Direct SQL, or fetch-and-cache
|
||||
// over gRPC). Validated at startup for the same reason — a FetchAndCache node with no endpoint
|
||||
// or no key produces silence, not an error. ConfigServe (the central serve surface) needs no
|
||||
// cross-field validation: a 0 port is a legitimate "disabled", so a plain Configure suffices.
|
||||
services.AddValidatedOptions<ConfigSourceOptions, ConfigSourceOptionsValidator>(
|
||||
configuration, ConfigSourceOptions.SectionName);
|
||||
services.Configure<ConfigServeOptions>(configuration.GetSection(ConfigServeOptions.SectionName));
|
||||
|
||||
services.AddSingleton<IClusterRoleInfo, ClusterRoleInfo>();
|
||||
|
||||
return services;
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
syntax = "proto3";
|
||||
|
||||
package deployment_artifact.v1;
|
||||
|
||||
option csharp_namespace = "ZB.MOM.WW.OtOpcUa.Commons.Protos.DeploymentArtifact.V1";
|
||||
|
||||
// Central serves the deployed-configuration artifact bytes to a driver node (per-cluster mesh
|
||||
// Phase 3). Only admin-role nodes — which hold the central SQL connection — serve; driver nodes in
|
||||
// FetchAndCache mode are the clients. The bytes travel here, out-of-band, precisely so they never
|
||||
// ride an Akka message and hit the frame-size limit.
|
||||
service DeploymentArtifactService {
|
||||
// Streams the artifact for a sealed deployment as ordered chunks. NotFound when the id is
|
||||
// unknown, the deployment is not sealed, or the blob is empty (existence-hiding + the #485
|
||||
// "empty bytes are never an empty config" rule). The client verifies
|
||||
// SHA-256(reassembled) == the RevisionHash it already holds from DispatchDeployment, so no hash
|
||||
// is carried here.
|
||||
rpc Fetch(FetchRequest) returns (stream ArtifactChunk);
|
||||
}
|
||||
|
||||
message FetchRequest {
|
||||
string deployment_id = 1;
|
||||
}
|
||||
|
||||
message ArtifactChunk {
|
||||
bytes data = 1;
|
||||
}
|
||||
@@ -11,6 +11,24 @@
|
||||
<PackageReference Include="ZB.MOM.WW.Audit"/>
|
||||
</ItemGroup>
|
||||
|
||||
<!-- Per-cluster mesh Phase 3: the first locally-compiled proto in the repo (all other gRPC here
|
||||
is packaged clients). Grpc.Tools generates both the server base and the client stub from
|
||||
deployment_artifact.proto; Google.Protobuf + Grpc.Core.Api carry the generated code's runtime
|
||||
types. Placed in Commons so both the central server (Host/AdminUI) and the node client
|
||||
(Runtime) get the types from one reference. -->
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Google.Protobuf"/>
|
||||
<PackageReference Include="Grpc.Core.Api"/>
|
||||
<PackageReference Include="Grpc.Tools">
|
||||
<PrivateAssets>all</PrivateAssets>
|
||||
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
|
||||
</PackageReference>
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<Protobuf Include="Protos\deployment_artifact.proto" GrpcServices="Both"/>
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\ZB.MOM.WW.OtOpcUa.Core.Abstractions\ZB.MOM.WW.OtOpcUa.Core.Abstractions.csproj"/>
|
||||
</ItemGroup>
|
||||
|
||||
@@ -0,0 +1,100 @@
|
||||
using Google.Protobuf;
|
||||
using Grpc.Core;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Protos.DeploymentArtifact.V1;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration;
|
||||
// The generated service container is itself named DeploymentArtifactService; alias its nested server
|
||||
// base so this impl can keep the natural name without colliding with the generated type.
|
||||
using GeneratedServiceBase =
|
||||
ZB.MOM.WW.OtOpcUa.Commons.Protos.DeploymentArtifact.V1.DeploymentArtifactService.DeploymentArtifactServiceBase;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.AdminUI.Grpc;
|
||||
|
||||
/// <summary>
|
||||
/// Central-side artifact serve (per-cluster mesh Phase 3). Streams a deployment's
|
||||
/// <c>ArtifactBlob</c> to a driver node fetching in <c>FetchAndCache</c> mode. Registered only on
|
||||
/// admin-role nodes (which hold the central SQL connection) and only when
|
||||
/// <c>ConfigServe:GrpcListenPort > 0</c>.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Serves any deployment that has a non-empty blob — status is NOT gated.</b> A node
|
||||
/// fetches the artifact <em>during apply</em>, before the deployment is sealed: central seals
|
||||
/// only once every node has acked Applied, and a <c>FetchAndCache</c> node cannot ack until it
|
||||
/// has fetched. Gating on <c>Sealed</c> would deadlock every such deploy (seal-needs-ack →
|
||||
/// ack-needs-fetch → fetch-needs-sealed), and it matches Direct mode, which reads the same
|
||||
/// <c>ArtifactBlob</c> from SQL while the row is still <c>AwaitingApplyAcks</c>. (Caught on the
|
||||
/// Phase 3 live gate.) Access is already gated by the shared-key interceptor, so there is no
|
||||
/// reason to hide a non-sealed deployment.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>NotFound is indistinguishable across two cases</b> — unknown id and an empty blob. The
|
||||
/// first is existence-hiding; the second is the #485 rule (a zero-length artifact is "no
|
||||
/// answer," never "a configuration with no drivers"). Streaming empty bytes as if valid is
|
||||
/// exactly what that defect class forbids.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class DeploymentArtifactGrpcService : GeneratedServiceBase
|
||||
{
|
||||
/// <summary>Chunk size, matching <c>LocalDbDeploymentArtifactCache.ChunkSize</c> (128 KiB).</summary>
|
||||
private const int ChunkSize = 128 * 1024;
|
||||
|
||||
private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbFactory;
|
||||
private readonly ILogger<DeploymentArtifactGrpcService> _log;
|
||||
|
||||
/// <summary>Initializes a new instance of the <see cref="DeploymentArtifactGrpcService"/> class.</summary>
|
||||
/// <param name="dbFactory">Factory for the central config database holding the artifact bytes.</param>
|
||||
/// <param name="log">Logger.</param>
|
||||
public DeploymentArtifactGrpcService(
|
||||
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
|
||||
ILogger<DeploymentArtifactGrpcService> log)
|
||||
{
|
||||
_dbFactory = dbFactory;
|
||||
_log = log;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public override async Task Fetch(
|
||||
FetchRequest request,
|
||||
IServerStreamWriter<ArtifactChunk> responseStream,
|
||||
ServerCallContext context)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(request);
|
||||
ArgumentNullException.ThrowIfNull(responseStream);
|
||||
ArgumentNullException.ThrowIfNull(context);
|
||||
|
||||
if (!Guid.TryParse(request.DeploymentId, out var id))
|
||||
throw new RpcException(new Status(StatusCode.NotFound, "unknown deployment"));
|
||||
|
||||
await using var db = await _dbFactory.CreateDbContextAsync(context.CancellationToken);
|
||||
var row = await db.Deployments
|
||||
.AsNoTracking()
|
||||
.Where(d => d.DeploymentId == id)
|
||||
.Select(d => new { d.ArtifactBlob })
|
||||
.FirstOrDefaultAsync(context.CancellationToken);
|
||||
|
||||
// Existence-hiding + #485: unknown id / empty blob collapse to one NotFound. Status is NOT
|
||||
// gated — a node fetches during apply, before the deployment is sealed (see the class remarks).
|
||||
if (row is null || row.ArtifactBlob.Length == 0)
|
||||
{
|
||||
_log.LogDebug(
|
||||
"Artifact fetch for {DeploymentId} -> NotFound (found={Found}, bytes={Bytes})",
|
||||
request.DeploymentId, row is not null, row?.ArtifactBlob.Length ?? 0);
|
||||
throw new RpcException(new Status(StatusCode.NotFound, "unknown deployment"));
|
||||
}
|
||||
|
||||
for (var offset = 0; offset < row.ArtifactBlob.Length; offset += ChunkSize)
|
||||
{
|
||||
var len = Math.Min(ChunkSize, row.ArtifactBlob.Length - offset);
|
||||
await responseStream.WriteAsync(
|
||||
new ArtifactChunk { Data = ByteString.CopyFrom(row.ArtifactBlob, offset, len) },
|
||||
context.CancellationToken);
|
||||
}
|
||||
|
||||
_log.LogInformation(
|
||||
"Served artifact for {DeploymentId} ({Bytes} bytes, {Chunks} chunks)",
|
||||
request.DeploymentId, row.ArtifactBlob.Length,
|
||||
(row.ArtifactBlob.Length + ChunkSize - 1) / ChunkSize);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,157 @@
|
||||
using System.Security.Cryptography;
|
||||
using System.Text;
|
||||
using Grpc.Core;
|
||||
using Grpc.Core.Interceptors;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
|
||||
|
||||
/// <summary>
|
||||
/// Gates the central config-serve artifact endpoint (per-cluster mesh Phase 3). The
|
||||
/// <c>DeploymentArtifactService</c> streams a driver node the exact configuration it will boot
|
||||
/// on; anything able to reach central's serve port could otherwise pull the whole deployed config
|
||||
/// out-of-band. This interceptor is the ONLY inbound auth on that endpoint.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Scoped by method path.</b> Only calls under
|
||||
/// <c>/deployment_artifact.v1.DeploymentArtifactService/</c> are gated; every other method on
|
||||
/// the shared gRPC pipeline (notably the LocalDb sync service) passes through untouched, so
|
||||
/// this can share one <c>AddGrpc</c> registration with <see cref="LocalDbSyncAuthInterceptor"/>.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Fail-closed.</b> With no <c>ConfigServe:ApiKey</c> configured, NO fetch is accepted,
|
||||
/// authenticated or not. Treating "no key" as "no auth required" would expose the deployed
|
||||
/// configuration on exactly the default shape most nodes ship with. The node dialing in must
|
||||
/// present the same <c>ConfigSource:ApiKey</c> (the shared node key), so a key typo stops
|
||||
/// config delivery outright rather than degrading to unauthenticated.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Comparison is <see cref="CryptographicOperations.FixedTimeEquals"/> over UTF-8 bytes, so a
|
||||
/// wrong key cannot be recovered byte-by-byte from response timing. Length differences are
|
||||
/// unavoidably observable and are not sensitive.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class ConfigServeAuthInterceptor : Interceptor
|
||||
{
|
||||
private const string ServicePrefix = "/deployment_artifact.v1.DeploymentArtifactService/";
|
||||
private const string AuthorizationHeader = "authorization";
|
||||
private const string BearerPrefix = "Bearer ";
|
||||
|
||||
private readonly IOptions<ConfigServeOptions> _options;
|
||||
private readonly ILogger<ConfigServeAuthInterceptor> _logger;
|
||||
|
||||
/// <summary>Creates the interceptor.</summary>
|
||||
/// <param name="options">Config-serve options; <c>ApiKey</c> is the expected bearer token.</param>
|
||||
/// <param name="logger">Logger for denial diagnostics.</param>
|
||||
public ConfigServeAuthInterceptor(
|
||||
IOptions<ConfigServeOptions> options,
|
||||
ILogger<ConfigServeAuthInterceptor> logger)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
ArgumentNullException.ThrowIfNull(logger);
|
||||
|
||||
_options = options;
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public override Task<TResponse> UnaryServerHandler<TRequest, TResponse>(
|
||||
TRequest request,
|
||||
ServerCallContext context,
|
||||
UnaryServerMethod<TRequest, TResponse> continuation)
|
||||
{
|
||||
Authorize(context);
|
||||
return continuation(request, context);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public override Task DuplexStreamingServerHandler<TRequest, TResponse>(
|
||||
IAsyncStreamReader<TRequest> requestStream,
|
||||
IServerStreamWriter<TResponse> responseStream,
|
||||
ServerCallContext context,
|
||||
DuplexStreamingServerMethod<TRequest, TResponse> continuation)
|
||||
{
|
||||
Authorize(context);
|
||||
return continuation(requestStream, responseStream, context);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public override Task<TResponse> ClientStreamingServerHandler<TRequest, TResponse>(
|
||||
IAsyncStreamReader<TRequest> requestStream,
|
||||
ServerCallContext context,
|
||||
ClientStreamingServerMethod<TRequest, TResponse> continuation)
|
||||
{
|
||||
Authorize(context);
|
||||
return continuation(requestStream, context);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public override Task ServerStreamingServerHandler<TRequest, TResponse>(
|
||||
TRequest request,
|
||||
IServerStreamWriter<TResponse> responseStream,
|
||||
ServerCallContext context,
|
||||
ServerStreamingServerMethod<TRequest, TResponse> continuation)
|
||||
{
|
||||
Authorize(context);
|
||||
return continuation(request, responseStream, context);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Throws <see cref="RpcException"/> with <see cref="StatusCode.Unauthenticated"/> if this is
|
||||
/// a config-serve call that does not carry the configured bearer token. Non-serve calls return
|
||||
/// immediately.
|
||||
/// </summary>
|
||||
private void Authorize(ServerCallContext context)
|
||||
{
|
||||
if (!context.Method.StartsWith(ServicePrefix, StringComparison.Ordinal))
|
||||
return;
|
||||
|
||||
var expected = _options.Value.ApiKey;
|
||||
if (string.IsNullOrEmpty(expected))
|
||||
{
|
||||
_logger.LogWarning(
|
||||
"Rejected a config-serve call to {Method}: no ConfigServe:ApiKey is configured, so the " +
|
||||
"artifact endpoint is closed. Configure the shared node key on central and every node.",
|
||||
context.Method);
|
||||
throw new RpcException(new Status(
|
||||
StatusCode.Unauthenticated,
|
||||
"Config-serve is not accepting connections: no API key is configured on this node."));
|
||||
}
|
||||
|
||||
var presented = ExtractBearerToken(context.RequestHeaders);
|
||||
if (presented is null || !FixedTimeEquals(presented, expected))
|
||||
{
|
||||
_logger.LogWarning(
|
||||
"Rejected a config-serve call to {Method}: {Reason}.",
|
||||
context.Method,
|
||||
presented is null ? "no bearer token presented" : "bearer token did not match");
|
||||
throw new RpcException(new Status(
|
||||
StatusCode.Unauthenticated,
|
||||
"Config-serve authentication failed."));
|
||||
}
|
||||
}
|
||||
|
||||
private static string? ExtractBearerToken(Metadata headers)
|
||||
{
|
||||
// gRPC lowercases header keys on the wire; compare case-insensitively anyway so a hand-built
|
||||
// Metadata in a test behaves the same as a real request.
|
||||
foreach (var entry in headers)
|
||||
{
|
||||
if (!string.Equals(entry.Key, AuthorizationHeader, StringComparison.OrdinalIgnoreCase))
|
||||
continue;
|
||||
|
||||
var value = entry.Value;
|
||||
if (value is not null && value.StartsWith(BearerPrefix, StringComparison.OrdinalIgnoreCase))
|
||||
return value[BearerPrefix.Length..];
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
private static bool FixedTimeEquals(string presented, string expected)
|
||||
=> CryptographicOperations.FixedTimeEquals(
|
||||
Encoding.UTF8.GetBytes(presented), Encoding.UTF8.GetBytes(expected));
|
||||
}
|
||||
@@ -8,6 +8,7 @@ using Serilog.Events;
|
||||
using ZB.MOM.WW.OtOpcUa.AdminUI;
|
||||
using ZB.MOM.WW.OtOpcUa.AdminUI.Clients;
|
||||
using ZB.MOM.WW.OtOpcUa.AdminUI.Components;
|
||||
using ZB.MOM.WW.OtOpcUa.AdminUI.Grpc;
|
||||
using ZB.MOM.WW.OtOpcUa.AdminUI.Hubs;
|
||||
using ZB.MOM.WW.OtOpcUa.AdminUI.ScriptAnalysis;
|
||||
using ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
@@ -29,6 +30,7 @@ using ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Recorder;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
using ZB.MOM.WW.OtOpcUa.OpcUaServer;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Historian;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Scripting;
|
||||
using ZB.MOM.WW.OtOpcUa.OpcUaServer.Security;
|
||||
@@ -185,11 +187,15 @@ if (hasDriver)
|
||||
// (initiator) / LocalDb:SyncListenPort (listener) are set. See LocalDbRegistration.
|
||||
builder.Services.AddOtOpcUaLocalDb(builder.Configuration);
|
||||
|
||||
// gRPC server plumbing for the passive sync endpoint mapped below. Registered only under
|
||||
// hasDriver so admin-only nodes expose no sync surface at all. The interceptor is the ONLY
|
||||
// inbound auth on that endpoint — the replication library's LocalDbSyncService verifies
|
||||
// nothing — and it fail-closes when no ApiKey is configured.
|
||||
builder.Services.AddGrpc(o => o.Interceptors.Add<LocalDbSyncAuthInterceptor>());
|
||||
// Node-side gRPC artifact fetcher (per-cluster mesh Phase 3). Resolved + used by DriverHostActor
|
||||
// only under ConfigSource:Mode = FetchAndCache; idle (never invoked) in the default Direct mode, so
|
||||
// registering it unconditionally on driver nodes is harmless. IDisposable — DI disposes its cached
|
||||
// h2c channels at shutdown.
|
||||
builder.Services.AddSingleton<IDeploymentArtifactFetcher, GrpcDeploymentArtifactFetcher>();
|
||||
|
||||
// gRPC server plumbing (AddGrpc + the two path-scoped auth interceptors) is registered once,
|
||||
// below, for hasDriver || hasAdmin — the LocalDb passive sync endpoint (driver) and the
|
||||
// ConfigServe artifact endpoint (admin) share one pipeline.
|
||||
|
||||
// Config-gated server-side HistoryRead backend. When the ServerHistorian section is enabled this
|
||||
// overrides the NullHistorianDataSource default from AddOtOpcUaRuntime (last registration wins) with
|
||||
@@ -389,23 +395,46 @@ builder.Services.AddOtOpcUaSecrets(builder.Configuration);
|
||||
builder.Services.AddOtOpcUaHealth();
|
||||
builder.Services.AddOtOpcUaObservability(builder.Configuration);
|
||||
|
||||
// gRPC server plumbing shared by two endpoints: the LocalDb passive sync endpoint (driver-role,
|
||||
// mapped below on hasDriver && syncListenPort > 0) and the ConfigServe artifact endpoint
|
||||
// (admin-role, mapped on hasAdmin && configServeGrpcPort > 0). Each interceptor is scoped strictly
|
||||
// to its own service path, so both can share one pipeline and each is a harmless pass-through when
|
||||
// its service is unmapped. Both fail-closed with no ApiKey configured — the interceptor is the ONLY
|
||||
// inbound auth on either endpoint (the libraries verify nothing). Registered whenever either role is
|
||||
// present so an admin-only node still serves config and a driver-only node still gates sync.
|
||||
if (hasDriver || hasAdmin)
|
||||
{
|
||||
builder.Services.AddGrpc(o =>
|
||||
{
|
||||
if (hasDriver)
|
||||
o.Interceptors.Add<LocalDbSyncAuthInterceptor>();
|
||||
if (hasAdmin)
|
||||
o.Interceptors.Add<ConfigServeAuthInterceptor>();
|
||||
});
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------------------------
|
||||
// LocalDb sync listener (default-OFF).
|
||||
// Dedicated h2c listeners (both default-OFF): the LocalDb sync endpoint (driver) and the
|
||||
// ConfigServe artifact endpoint (admin, per-cluster mesh Phase 3).
|
||||
//
|
||||
// DANGER: any explicit Kestrel Listen* call makes Kestrel IGNORE ASPNETCORE_URLS/urls ENTIRELY
|
||||
// (it logs "Overriding address(es)"). The host has no ConfigureKestrel today and binds solely via
|
||||
// that configuration, so adding a listener naively would silently unbind the AdminUI + deploy API
|
||||
// behind Traefik. Everything the host was already asked to serve is therefore re-bound explicitly
|
||||
// in the same block.
|
||||
// in the same block — and, critically, EXACTLY ONCE: a fused admin+driver node can have BOTH
|
||||
// dedicated ports set, and re-applying the existing surface per-port would double-bind it and throw
|
||||
// "address already in use". So the existing surface is computed and applied once, then whichever of
|
||||
// the two dedicated ports are configured are added on top.
|
||||
//
|
||||
// The listener is HTTP/2-ONLY on purpose: the sync client speaks prior-knowledge h2c, which a
|
||||
// cleartext Http1AndHttp2 endpoint cannot negotiate (there is no ALPN without TLS). Hence a
|
||||
// dedicated port rather than multiplexing onto the main one.
|
||||
// Each listener is HTTP/2-ONLY on purpose: both clients speak prior-knowledge h2c, which a cleartext
|
||||
// Http1AndHttp2 endpoint cannot negotiate (there is no ALPN without TLS). Hence dedicated ports
|
||||
// rather than multiplexing onto the main one.
|
||||
//
|
||||
// When LocalDb:SyncListenPort is 0 (the default) none of this runs and URL binding is untouched.
|
||||
// When both ports are 0 (the default) none of this runs and URL binding is untouched.
|
||||
// ---------------------------------------------------------------------------------------------
|
||||
var syncListenPort = LocalDbRegistration.SyncListenPort(builder.Configuration);
|
||||
if (hasDriver && syncListenPort > 0)
|
||||
var syncListenPort = hasDriver ? LocalDbRegistration.SyncListenPort(builder.Configuration) : 0;
|
||||
var configServeGrpcPort = hasAdmin ? builder.Configuration.GetValue<int>("ConfigServe:GrpcListenPort") : 0;
|
||||
if (syncListenPort > 0 || configServeGrpcPort > 0)
|
||||
{
|
||||
// Mirror Kestrel's own source precedence: URLS wins; else the HTTP_PORTS/HTTPS_PORTS bare-port
|
||||
// vars; else Kestrel's localhost:5000 default. Missing the HTTP_PORTS leg is not academic — the
|
||||
@@ -428,8 +457,8 @@ if (hasDriver && syncListenPort > 0)
|
||||
];
|
||||
if (existingBindings.Count > 0)
|
||||
Log.Information(
|
||||
"LocalDb sync listener enabled; re-binding the HTTP_PORTS surface ({HttpPorts}/{HttpsPorts}) " +
|
||||
"alongside the sync port.", httpPorts, httpsPorts);
|
||||
"Dedicated h2c listener(s) enabled; re-binding the HTTP_PORTS surface ({HttpPorts}/{HttpsPorts}) " +
|
||||
"alongside the dedicated port(s).", httpPorts, httpsPorts);
|
||||
}
|
||||
|
||||
// A driver-only Windows-service node gets NO URLS and NO *_PORTS (Install-Services.ps1 sets URLS
|
||||
@@ -440,38 +469,48 @@ if (hasDriver && syncListenPort > 0)
|
||||
{
|
||||
existingBindings = [new KestrelHttpBinding("localhost", 5000, IsSecure: false)];
|
||||
Log.Information(
|
||||
"LocalDb sync listener enabled with no URLs configured; re-binding Kestrel's default " +
|
||||
"http://localhost:5000 alongside the sync port.");
|
||||
"Dedicated h2c listener(s) enabled with no URLs configured; re-binding Kestrel's default " +
|
||||
"http://localhost:5000 alongside the dedicated port(s).");
|
||||
}
|
||||
|
||||
// Re-binding an https endpoint would need its certificate configuration replayed too, which
|
||||
// this host does not model (TLS is terminated at Traefik). Rather than silently serve it
|
||||
// without TLS — or drop it — refuse to take over Kestrel at all and leave the existing
|
||||
// surface exactly as it was. The operator gets a loud, actionable error instead of an
|
||||
// AdminUI that stopped answering.
|
||||
// AdminUI that stopped answering. Disables BOTH dedicated ports (either would force the same
|
||||
// Kestrel takeover).
|
||||
if (existingBindings.Any(b => b.IsSecure))
|
||||
{
|
||||
Log.Error(
|
||||
"LocalDb:SyncListenPort is set to {Port} but this host serves HTTPS endpoint(s) ({Urls}). " +
|
||||
"Binding the sync listener requires re-binding every existing endpoint explicitly, and the " +
|
||||
"HTTPS certificate configuration cannot be replayed safely. The sync listener is DISABLED; " +
|
||||
"terminate TLS upstream (as the docker-dev rig does) or leave replication off on this node.",
|
||||
syncListenPort, configuredUrls);
|
||||
"A dedicated h2c listener is requested (LocalDb:SyncListenPort={SyncPort}, " +
|
||||
"ConfigServe:GrpcListenPort={ConfigServePort}) but this host serves HTTPS endpoint(s) ({Urls}). " +
|
||||
"Binding a dedicated listener requires re-binding every existing endpoint explicitly, and the " +
|
||||
"HTTPS certificate configuration cannot be replayed safely. BOTH dedicated listeners are DISABLED; " +
|
||||
"terminate TLS upstream (as the docker-dev rig does) or leave these features off on this node.",
|
||||
syncListenPort, configServeGrpcPort, configuredUrls);
|
||||
syncListenPort = 0;
|
||||
configServeGrpcPort = 0;
|
||||
}
|
||||
else
|
||||
{
|
||||
// Capture locals so the ConfigureKestrel closure does not observe later reassignment.
|
||||
var syncPortToBind = syncListenPort;
|
||||
var configServePortToBind = configServeGrpcPort;
|
||||
builder.WebHost.ConfigureKestrel(kestrel =>
|
||||
{
|
||||
foreach (var binding in existingBindings)
|
||||
binding.Apply(kestrel);
|
||||
|
||||
kestrel.ListenAnyIP(syncListenPort, o => o.Protocols = HttpProtocols.Http2);
|
||||
if (syncPortToBind > 0)
|
||||
kestrel.ListenAnyIP(syncPortToBind, o => o.Protocols = HttpProtocols.Http2);
|
||||
if (configServePortToBind > 0)
|
||||
kestrel.ListenAnyIP(configServePortToBind, o => o.Protocols = HttpProtocols.Http2);
|
||||
});
|
||||
|
||||
Log.Information(
|
||||
"LocalDb sync listener bound on :{SyncPort} (h2c); re-bound existing endpoint(s) {Urls}.",
|
||||
syncListenPort, configuredUrls ?? "http://localhost:5000 (Kestrel default)");
|
||||
"Dedicated h2c listener(s) bound (sync=:{SyncPort}, config-serve=:{ConfigServePort}); " +
|
||||
"re-bound existing endpoint(s) {Urls}.",
|
||||
syncListenPort, configServeGrpcPort, configuredUrls ?? "http://localhost:5000 (Kestrel default)");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -514,6 +553,14 @@ if (hasDriver && syncListenPort > 0)
|
||||
app.MapZbLocalDbSync();
|
||||
}
|
||||
|
||||
// Central config-serve artifact endpoint (per-cluster mesh Phase 3). Mapped only on admin-role
|
||||
// nodes (which hold the central SQL connection) and only when a dedicated h2c port is bound. The
|
||||
// ConfigServeAuthInterceptor is the ONLY inbound auth and fail-closes with no ConfigServe:ApiKey.
|
||||
if (hasAdmin && configServeGrpcPort > 0)
|
||||
{
|
||||
app.MapGrpcService<DeploymentArtifactGrpcService>();
|
||||
}
|
||||
|
||||
app.MapOtOpcUaHealth();
|
||||
app.MapOtOpcUaMetrics();
|
||||
|
||||
|
||||
@@ -0,0 +1,153 @@
|
||||
using System.Collections.Concurrent;
|
||||
using System.Security.Cryptography;
|
||||
using Grpc.Core;
|
||||
using Grpc.Net.Client;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Protos.DeploymentArtifact.V1;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
|
||||
|
||||
/// <summary>
|
||||
/// Streams the deployed-configuration artifact from central over gRPC and verifies it against the
|
||||
/// revision hash the dispatch already carries (per-cluster mesh Phase 3, <c>FetchAndCache</c>).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>One invariant.</b> Return the first endpoint's bytes that both reassemble AND verify
|
||||
/// (<c>SHA-256(bytes)</c> lowercase-hex == the expected revision hash); otherwise
|
||||
/// <see langword="null"/>. Every per-endpoint problem — an <see cref="RpcException"/>
|
||||
/// (Unavailable / NotFound / deadline), a zero-length stream, or a hash mismatch — is treated
|
||||
/// uniformly as "this endpoint did not supply valid bytes," logged, and the next endpoint is
|
||||
/// tried. A hash mismatch is tried-next rather than fatal because a partitioned central could
|
||||
/// legitimately serve stale bytes from one node and current bytes from another; when every
|
||||
/// central shares one SQL store (the common case) the retry is merely cheap and harmless.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Why <see langword="null"/>, not throw.</b> The caller treats <see langword="null"/>
|
||||
/// identically to today's "reconcile returned null": an apply failure that keeps
|
||||
/// last-known-good and does not advance the revision. A typed "no bytes" keeps the #485
|
||||
/// contract explicit and cannot escape as an unhandled actor message.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class GrpcDeploymentArtifactFetcher : IDeploymentArtifactFetcher, IDisposable
|
||||
{
|
||||
private readonly ConfigSourceOptions _options;
|
||||
private readonly ILogger<GrpcDeploymentArtifactFetcher> _log;
|
||||
private readonly Func<string, DeploymentArtifactService.DeploymentArtifactServiceClient> _clientFactory;
|
||||
private readonly ConcurrentDictionary<string, GrpcChannel> _channels = new(StringComparer.Ordinal);
|
||||
|
||||
/// <summary>Initializes a new instance of the <see cref="GrpcDeploymentArtifactFetcher"/> class.</summary>
|
||||
/// <param name="options">Config-source options; supplies the endpoints, shared key, and deadline.</param>
|
||||
/// <param name="log">Logger.</param>
|
||||
/// <param name="clientFactory">
|
||||
/// Seam for tests: builds the generated client for an endpoint. Defaults to a real h2c
|
||||
/// <see cref="GrpcChannel"/> (one cached per endpoint).
|
||||
/// </param>
|
||||
public GrpcDeploymentArtifactFetcher(
|
||||
IOptions<ConfigSourceOptions> options,
|
||||
ILogger<GrpcDeploymentArtifactFetcher> log,
|
||||
Func<string, DeploymentArtifactService.DeploymentArtifactServiceClient>? clientFactory = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
ArgumentNullException.ThrowIfNull(log);
|
||||
|
||||
_options = options.Value;
|
||||
_log = log;
|
||||
_clientFactory = clientFactory ?? DefaultClientFactory;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<byte[]?> FetchAsync(string deploymentId, string expectedRevisionHash, CancellationToken ct)
|
||||
{
|
||||
ArgumentException.ThrowIfNullOrEmpty(deploymentId);
|
||||
ArgumentException.ThrowIfNullOrEmpty(expectedRevisionHash);
|
||||
|
||||
var endpoints = _options.CentralFetchEndpoints;
|
||||
if (endpoints.Length == 0)
|
||||
{
|
||||
_log.LogWarning(
|
||||
"Artifact fetch for {DeploymentId} skipped: no ConfigSource:CentralFetchEndpoints configured.",
|
||||
deploymentId);
|
||||
return null;
|
||||
}
|
||||
|
||||
foreach (var endpoint in endpoints)
|
||||
{
|
||||
try
|
||||
{
|
||||
var bytes = await FetchFromEndpointAsync(endpoint, deploymentId, ct);
|
||||
if (bytes.Length == 0)
|
||||
{
|
||||
_log.LogWarning(
|
||||
"Artifact fetch for {DeploymentId} from {Endpoint} returned zero bytes; trying next endpoint.",
|
||||
deploymentId, endpoint);
|
||||
continue;
|
||||
}
|
||||
|
||||
var actualHash = Convert.ToHexStringLower(SHA256.HashData(bytes));
|
||||
if (!string.Equals(actualHash, expectedRevisionHash, StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
_log.LogWarning(
|
||||
"Artifact fetch for {DeploymentId} from {Endpoint} failed SHA-256 verification " +
|
||||
"(expected {Expected}, got {Actual}); trying next endpoint.",
|
||||
deploymentId, endpoint, expectedRevisionHash, actualHash);
|
||||
continue;
|
||||
}
|
||||
|
||||
_log.LogInformation(
|
||||
"Fetched + verified artifact for {DeploymentId} from {Endpoint} ({Bytes} bytes).",
|
||||
deploymentId, endpoint, bytes.Length);
|
||||
return bytes;
|
||||
}
|
||||
catch (RpcException ex)
|
||||
{
|
||||
_log.LogWarning(
|
||||
"Artifact fetch for {DeploymentId} from {Endpoint} failed ({Status}); trying next endpoint.",
|
||||
deploymentId, endpoint, ex.StatusCode);
|
||||
}
|
||||
}
|
||||
|
||||
_log.LogWarning(
|
||||
"Artifact fetch for {DeploymentId} failed on ALL {Count} endpoint(s); returning no bytes " +
|
||||
"(apply failure — last-known-good is kept).",
|
||||
deploymentId, endpoints.Length);
|
||||
return null;
|
||||
}
|
||||
|
||||
private async Task<byte[]> FetchFromEndpointAsync(string endpoint, string deploymentId, CancellationToken ct)
|
||||
{
|
||||
var client = _clientFactory(endpoint);
|
||||
var headers = new Metadata { { "authorization", $"Bearer {_options.ApiKey}" } };
|
||||
|
||||
using var deadlineCts = CancellationTokenSource.CreateLinkedTokenSource(ct);
|
||||
if (_options.FetchTimeoutSeconds > 0)
|
||||
deadlineCts.CancelAfter(TimeSpan.FromSeconds(_options.FetchTimeoutSeconds));
|
||||
|
||||
using var call = client.Fetch(
|
||||
new FetchRequest { DeploymentId = deploymentId }, headers, cancellationToken: deadlineCts.Token);
|
||||
|
||||
using var buffer = new MemoryStream();
|
||||
await foreach (var chunk in call.ResponseStream.ReadAllAsync(deadlineCts.Token))
|
||||
chunk.Data.WriteTo(buffer);
|
||||
|
||||
return buffer.ToArray();
|
||||
}
|
||||
|
||||
private DeploymentArtifactService.DeploymentArtifactServiceClient DefaultClientFactory(string endpoint)
|
||||
{
|
||||
// h2c: GrpcChannel over an http:// address negotiates prior-knowledge HTTP/2. One channel per
|
||||
// endpoint, cached and reused (creating a channel per call would leak sockets).
|
||||
var channel = _channels.GetOrAdd(endpoint, ep => GrpcChannel.ForAddress(ep));
|
||||
return new DeploymentArtifactService.DeploymentArtifactServiceClient(channel);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public void Dispose()
|
||||
{
|
||||
foreach (var channel in _channels.Values)
|
||||
channel.Dispose();
|
||||
_channels.Clear();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
namespace ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
|
||||
|
||||
/// <summary>
|
||||
/// Fetches the deployed-configuration artifact bytes from central over gRPC (per-cluster mesh
|
||||
/// Phase 3, <c>FetchAndCache</c> mode). The node-side counterpart of central's
|
||||
/// <c>DeploymentArtifactService</c>.
|
||||
/// </summary>
|
||||
public interface IDeploymentArtifactFetcher
|
||||
{
|
||||
/// <summary>
|
||||
/// Fetches and reassembles the artifact for <paramref name="deploymentId"/> from the first
|
||||
/// configured central endpoint that answers, verifying that
|
||||
/// <c>SHA-256(reassembled bytes)</c> (lowercase hex) equals
|
||||
/// <paramref name="expectedRevisionHash"/> — the hash already carried in the dispatch.
|
||||
/// </summary>
|
||||
/// <param name="deploymentId">The deployment id to fetch.</param>
|
||||
/// <param name="expectedRevisionHash">
|
||||
/// The lowercase-hex SHA-256 the reassembled bytes must match (<c>Deployment.RevisionHash</c>).
|
||||
/// </param>
|
||||
/// <param name="ct">Cancellation token.</param>
|
||||
/// <returns>
|
||||
/// The verified artifact bytes, or <see langword="null"/> on ANY failure — every configured
|
||||
/// endpoint unreachable/NotFound, a zero-length stream, or a SHA-256 mismatch. A
|
||||
/// <see langword="null"/> is "no answer," NEVER an empty configuration: the caller treats it as
|
||||
/// an apply failure and keeps last-known-good (the #485 contract).
|
||||
/// </returns>
|
||||
Task<byte[]?> FetchAsync(string deploymentId, string expectedRevisionHash, CancellationToken ct);
|
||||
}
|
||||
@@ -72,6 +72,19 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
|
||||
/// </summary>
|
||||
private readonly IDeploymentArtifactCache? _deploymentArtifactCache;
|
||||
|
||||
/// <summary>
|
||||
/// Per-cluster mesh Phase 3. When <see langword="true"/> (<c>ConfigSource:Mode =
|
||||
/// FetchAndCache</c>) this node obtains its config by fetching the artifact bytes from central
|
||||
/// over gRPC and applying them from the LocalDb cache, reading NO config from central SQL.
|
||||
/// When <see langword="false"/> (default <c>Direct</c>) every path reads central SQL as before.
|
||||
/// </summary>
|
||||
private readonly bool _fetchAndCacheMode;
|
||||
|
||||
/// <summary>
|
||||
/// Node-side gRPC artifact fetcher, non-null only under <see cref="_fetchAndCacheMode"/>.
|
||||
/// </summary>
|
||||
private readonly IDeploymentArtifactFetcher? _artifactFetcher;
|
||||
|
||||
/// <summary>
|
||||
/// True once this node has booted a cached artifact because central SQL was unreachable.
|
||||
/// </summary>
|
||||
@@ -382,7 +395,9 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
|
||||
Func<int>? driverMemberCountProvider = null,
|
||||
IDeploymentArtifactCache? deploymentArtifactCache = null,
|
||||
IRedundancyRoleView? redundancyRoleView = null,
|
||||
string? replicationPeerHost = null) =>
|
||||
string? replicationPeerHost = null,
|
||||
bool fetchAndCacheMode = false,
|
||||
IDeploymentArtifactFetcher? artifactFetcher = null) =>
|
||||
// WARNING: this forwarding list is POSITIONAL, and Props.Create compiles it into an
|
||||
// expression tree. Six IActorRef? parameters and several interface-typed ones mean a
|
||||
// mis-ordered argument is usually type-compatible and therefore compiles clean, then binds
|
||||
@@ -392,7 +407,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
|
||||
dbFactory, localNode, ackRouter, driverFactory, localRoles, dependencyMux, opcUaPublishActor,
|
||||
healthPublisher, virtualTagEvaluator, historyWriter, virtualTagHostOverride,
|
||||
loggerFactory, scriptRootLogger, scriptedAlarmHostOverride, invokerFactory, driverMemberCountProvider,
|
||||
deploymentArtifactCache, redundancyRoleView, replicationPeerHost));
|
||||
deploymentArtifactCache, redundancyRoleView, replicationPeerHost, fetchAndCacheMode, artifactFetcher));
|
||||
|
||||
/// <summary>Initializes a new DriverHostActor with the specified dependencies.</summary>
|
||||
/// <param name="dbFactory">Database context factory for configuration database access.</param>
|
||||
@@ -446,11 +461,15 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
|
||||
Func<int>? driverMemberCountProvider = null,
|
||||
IDeploymentArtifactCache? deploymentArtifactCache = null,
|
||||
IRedundancyRoleView? redundancyRoleView = null,
|
||||
string? replicationPeerHost = null)
|
||||
string? replicationPeerHost = null,
|
||||
bool fetchAndCacheMode = false,
|
||||
IDeploymentArtifactFetcher? artifactFetcher = null)
|
||||
{
|
||||
_deploymentArtifactCache = deploymentArtifactCache;
|
||||
_redundancyRoleView = redundancyRoleView;
|
||||
_replicationPeerHost = replicationPeerHost;
|
||||
_fetchAndCacheMode = fetchAndCacheMode;
|
||||
_artifactFetcher = artifactFetcher;
|
||||
_dbFactory = dbFactory;
|
||||
_localNode = localNode;
|
||||
_ackRouter = ackRouter;
|
||||
@@ -576,6 +595,14 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
|
||||
|
||||
private void Bootstrap()
|
||||
{
|
||||
// Per-cluster mesh Phase 3: a FetchAndCache node NEVER reads central SQL for config, at boot
|
||||
// or otherwise. It restores its served state from the LocalDb pointer instead.
|
||||
if (_fetchAndCacheMode)
|
||||
{
|
||||
BootstrapFromCache();
|
||||
return;
|
||||
}
|
||||
|
||||
// Read the most-recent NodeDeploymentState for this node; if it's Applied, jump
|
||||
// to Steady with that revision. If Applying (orphan from a crash), discard and replay.
|
||||
// If the DB is unreachable, fall back to Stale and start the reconnect loop.
|
||||
@@ -640,6 +667,60 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Per-cluster mesh Phase 3 boot for <c>FetchAndCache</c> mode. Restores served state from the
|
||||
/// LocalDb pointer without any central-SQL read.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Distinct from <see cref="TryBootFromCache"/> (the Direct-mode SQL-unreachable fallback):
|
||||
/// reading from the cache is <b>normal</b> operation here, not a degraded state, so
|
||||
/// <see cref="_isRunningFromCache"/> stays <see langword="false"/> — the node can still
|
||||
/// fetch new deployments. An empty or unreadable cache lands Steady-with-no-revision (the
|
||||
/// first dispatch fetches), never <c>Stale</c>: Stale means "central SQL is down, retry it",
|
||||
/// and FetchAndCache has no config SQL read to recover.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
private void BootstrapFromCache()
|
||||
{
|
||||
try
|
||||
{
|
||||
var cached = _deploymentArtifactCache?.GetCurrentUnkeyedAsync().GetAwaiter().GetResult();
|
||||
if (cached is null)
|
||||
{
|
||||
_log.Info(
|
||||
"DriverHost {Node}: FetchAndCache boot — the LocalDb cache is empty; entering Steady " +
|
||||
"(no revision). The first dispatch will fetch this node's configuration from central.",
|
||||
_localNode);
|
||||
Become(Steady);
|
||||
return;
|
||||
}
|
||||
|
||||
var deploymentId = DeploymentId.Parse(cached.DeploymentId);
|
||||
var revision = RevisionHash.Parse(cached.RevisionHash);
|
||||
|
||||
_currentRevision = revision;
|
||||
Become(Steady);
|
||||
// Restore drivers + address space + subscriptions from the cached bytes — no SQL read, no
|
||||
// re-ack (the deployment is already Applied). Same in-hand-bytes apply as the Direct-mode
|
||||
// cache boot, but this is the steady-state config source, not an outage fallback.
|
||||
ApplyCachedArtifact(deploymentId, cached.Artifact);
|
||||
|
||||
_log.Info(
|
||||
"DriverHost {Node}: FetchAndCache boot — restored served state for deployment {Id} " +
|
||||
"(rev {Rev}) from the LocalDb pointer.",
|
||||
_localNode, deploymentId, revision);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_log.Warning(ex,
|
||||
"DriverHost {Node}: FetchAndCache boot — failed to read the LocalDb pointer; entering " +
|
||||
"Steady (no revision). The next dispatch will fetch.",
|
||||
_localNode);
|
||||
Become(Steady);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Last-resort boot path: apply the artifact cached by a previous successful deploy (or
|
||||
/// replicated from this node's pair peer) when central SQL cannot be reached.
|
||||
@@ -726,16 +807,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
|
||||
{
|
||||
var correlation = CorrelationId.NewId();
|
||||
|
||||
var specs = DeploymentArtifact.ParseDriverInstances(blob, _localNode.Value);
|
||||
var snapshots = _children.ToDictionary(
|
||||
kv => kv.Key,
|
||||
kv => new DriverChildSnapshot(kv.Value.DriverType, kv.Value.LastConfigJson, kv.Value.ResilienceConfig),
|
||||
StringComparer.Ordinal);
|
||||
var plan = DriverSpawnPlanner.Compute(snapshots, specs);
|
||||
|
||||
foreach (var id in plan.ToStop) StopChild(id);
|
||||
foreach (var spec in plan.ToApplyDelta) ApplyChildDelta(spec);
|
||||
foreach (var spec in plan.ToSpawn) SpawnChild(spec);
|
||||
ReconcileDriversFromBlob(deploymentId, blob);
|
||||
|
||||
// Hand the cached blob to the rebuild so it materialises the client-facing address space from
|
||||
// it directly. Passing only the deploymentId would drive a ConfigDb read — unreachable here by
|
||||
@@ -781,6 +853,9 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
|
||||
_localNode, msg.DeploymentId, _applyingDeploymentId);
|
||||
Self.Forward(msg); // re-deliver after we transition back
|
||||
});
|
||||
// The FetchAndCache fetch pipes its result back here (BeginFetchAndCacheApply ran in Steady,
|
||||
// then Become(Applying)); completing the apply transitions back to Steady.
|
||||
Receive<FetchedForApply>(HandleFetchedForApply);
|
||||
Receive<GetDiagnostics>(HandleGetDiagnostics);
|
||||
Receive<DriverInstanceActor.AttributeValuePublished>(ForwardToMux);
|
||||
Receive<DriverInstanceActor.AttributeAlarmPublished>(ForwardNativeAlarm);
|
||||
@@ -1615,9 +1690,132 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
|
||||
SendAck(msg.DeploymentId, ApplyAckOutcome.Applied, failureReason: null, msg.CorrelationId);
|
||||
return;
|
||||
}
|
||||
|
||||
if (_fetchAndCacheMode)
|
||||
{
|
||||
BeginFetchAndCacheApply(msg.DeploymentId, msg.RevisionHash, msg.CorrelationId);
|
||||
return;
|
||||
}
|
||||
|
||||
ApplyAndAck(msg.DeploymentId, msg.RevisionHash, msg.CorrelationId);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Per-cluster mesh Phase 3 (<c>FetchAndCache</c>). Kicks off a gRPC artifact fetch and pipes
|
||||
/// the result back to <see cref="Self"/> as a <see cref="FetchedForApply"/> — the fetch is I/O
|
||||
/// and MUST NOT block the mailbox for the fetch deadline, so it never runs as an awaited call
|
||||
/// inside a receive. The apply itself completes synchronously on the actor thread in
|
||||
/// <see cref="HandleFetchedForApply"/>, mirroring <see cref="ApplyAndAck"/>.
|
||||
/// </summary>
|
||||
private void BeginFetchAndCacheApply(DeploymentId deploymentId, RevisionHash revision, CorrelationId correlation)
|
||||
{
|
||||
_applyingDeploymentId = deploymentId;
|
||||
Become(Applying);
|
||||
|
||||
// Persist Applying row (idempotent on PK) — same crash-boundary marker as ApplyAndAck.
|
||||
UpsertNodeDeploymentState(deploymentId, NodeDeploymentStatus.Applying, failureReason: null);
|
||||
|
||||
if (_artifactFetcher is null)
|
||||
{
|
||||
// Misconfiguration: FetchAndCache with no fetcher wired. Fail the apply rather than fetch
|
||||
// nothing forever; the node keeps serving last-known-good.
|
||||
const string reason = "FetchAndCache mode is set but no artifact fetcher is configured on this node";
|
||||
UpsertNodeDeploymentState(deploymentId, NodeDeploymentStatus.Failed, reason);
|
||||
SendAck(deploymentId, ApplyAckOutcome.Failed, reason, correlation);
|
||||
_log.Error("DriverHost {Node}: cannot apply {Id} — {Reason}", _localNode, deploymentId, reason);
|
||||
_applyingDeploymentId = null;
|
||||
Become(Steady);
|
||||
return;
|
||||
}
|
||||
|
||||
_log.Debug("DriverHost {Node}: fetching artifact for {Id} (rev {Rev}) from central over gRPC",
|
||||
_localNode, deploymentId, revision);
|
||||
|
||||
// A faulted fetch task maps to null bytes — the #485 apply-failure path, exactly like an
|
||||
// all-endpoints-down fetch — so a fault can never escape as an unhandled actor message.
|
||||
_artifactFetcher.FetchAsync(deploymentId.ToString(), revision.Value, CancellationToken.None)
|
||||
.PipeTo(
|
||||
Self,
|
||||
success: bytes => new FetchedForApply(deploymentId, revision, correlation, bytes),
|
||||
failure: _ => new FetchedForApply(deploymentId, revision, correlation, Bytes: null));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Completes a <c>FetchAndCache</c> apply once the fetched bytes are in hand. Null bytes are an
|
||||
/// apply FAILURE (keep last-known-good, do not advance the revision — #485); non-null bytes are
|
||||
/// applied from hand (no ConfigDb read) and cached.
|
||||
/// </summary>
|
||||
private void HandleFetchedForApply(FetchedForApply msg)
|
||||
{
|
||||
// Ignore a result for anything but the in-flight apply (a stale/duplicate pipe-back).
|
||||
if (_applyingDeploymentId is null || msg.DeploymentId != _applyingDeploymentId.Value)
|
||||
{
|
||||
_log.Debug("DriverHost {Node}: dropping stale fetch result for {Id} (in-flight={Cur})",
|
||||
_localNode, msg.DeploymentId, _applyingDeploymentId);
|
||||
return;
|
||||
}
|
||||
|
||||
using var span = OtOpcUaTelemetry.StartDeployApplySpan(msg.DeploymentId.ToString());
|
||||
span?.SetTag("otopcua.node_id", _localNode.ToString());
|
||||
span?.SetTag("otopcua.revision", msg.Revision.ToString());
|
||||
span?.SetTag("otopcua.correlation_id", msg.Correlation.ToString());
|
||||
var sw = Stopwatch.StartNew();
|
||||
|
||||
try
|
||||
{
|
||||
// #485: null bytes = "no answer" (every endpoint unreachable / NotFound / SHA mismatch).
|
||||
// Fail the apply and stay on the current revision so a re-dispatch actually retries; keep
|
||||
// serving the last-known-good address space, drivers and subscriptions throughout.
|
||||
var appliedBlob = ReconcileDriversFromBlob(msg.DeploymentId, msg.Bytes);
|
||||
if (appliedBlob is null)
|
||||
{
|
||||
const string reason =
|
||||
"the deployment artifact could not be fetched from central (all endpoints unreachable, NotFound, or SHA-256 mismatch); nothing was applied";
|
||||
UpsertNodeDeploymentState(msg.DeploymentId, NodeDeploymentStatus.Failed, reason);
|
||||
SendAck(msg.DeploymentId, ApplyAckOutcome.Failed, reason, msg.Correlation);
|
||||
OtOpcUaTelemetry.DeploymentApplied.Add(1, new KeyValuePair<string, object?>("outcome", "reject"));
|
||||
span?.SetStatus(ActivityStatusCode.Error, reason);
|
||||
_log.Error(
|
||||
"DriverHost {Node}: FetchAndCache apply of {Id} FAILED — {Reason}. Staying on revision {Rev} and continuing to serve it; re-dispatch once central is reachable",
|
||||
_localNode, msg.DeploymentId, reason, _currentRevision);
|
||||
return;
|
||||
}
|
||||
|
||||
_currentRevision = msg.Revision;
|
||||
UpsertNodeDeploymentState(msg.DeploymentId, NodeDeploymentStatus.Applied, failureReason: null);
|
||||
SendAck(msg.DeploymentId, ApplyAckOutcome.Applied, failureReason: null, msg.Correlation);
|
||||
// Rebuild the address space + push subscriptions FROM the fetched bytes so OpcUaPublishActor
|
||||
// never reads central SQL (the whole point of FetchAndCache).
|
||||
_opcUaPublishActor?.Tell(new ZB.MOM.WW.OtOpcUa.Runtime.OpcUa.OpcUaPublishActor.RebuildAddressSpace(
|
||||
msg.Correlation, msg.DeploymentId, appliedBlob));
|
||||
PushDesiredSubscriptionsFromArtifact(msg.DeploymentId, appliedBlob);
|
||||
// Cache the fetched bytes as this node's last-known-good (idempotent store; skips empty).
|
||||
CacheAppliedArtifact(msg.DeploymentId, msg.Revision, appliedBlob);
|
||||
_isRunningFromCache = false;
|
||||
OtOpcUaTelemetry.DeploymentApplied.Add(1, new KeyValuePair<string, object?>("outcome", "ack"));
|
||||
_log.Info("DriverHost {Node}: applied fetched deployment {Id} (rev {Rev}, children={Count})",
|
||||
_localNode, msg.DeploymentId, msg.Revision, _children.Count);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
UpsertNodeDeploymentState(msg.DeploymentId, NodeDeploymentStatus.Failed, ex.Message);
|
||||
SendAck(msg.DeploymentId, ApplyAckOutcome.Failed, ex.Message, msg.Correlation);
|
||||
OtOpcUaTelemetry.DeploymentApplied.Add(1, new KeyValuePair<string, object?>("outcome", "reject"));
|
||||
span?.SetStatus(ActivityStatusCode.Error, ex.Message);
|
||||
_log.Error(ex, "DriverHost {Node}: FetchAndCache apply of {Id} failed", _localNode, msg.DeploymentId);
|
||||
}
|
||||
finally
|
||||
{
|
||||
OtOpcUaTelemetry.DeploymentApplyDurationSec.Record(sw.Elapsed.TotalSeconds);
|
||||
_applyingDeploymentId = null;
|
||||
Become(Steady);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Pipe-back carrier for a <c>FetchAndCache</c> fetch result (null bytes = no answer).</summary>
|
||||
private sealed record FetchedForApply(
|
||||
DeploymentId DeploymentId, RevisionHash Revision, CorrelationId Correlation, byte[]? Bytes);
|
||||
|
||||
private void ApplyAndAck(DeploymentId deploymentId, RevisionHash revision, CorrelationId correlation)
|
||||
{
|
||||
_applyingDeploymentId = deploymentId;
|
||||
@@ -1732,6 +1930,19 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
|
||||
return null;
|
||||
}
|
||||
|
||||
return ReconcileDriversFromBlob(deploymentId, blob);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The blob-processing half of <see cref="ReconcileDrivers"/>, split out so the
|
||||
/// <c>FetchAndCache</c> apply path (which already holds the fetched bytes) and the
|
||||
/// boot-from-cache path can reconcile without a ConfigDb read.
|
||||
/// </summary>
|
||||
/// <returns>The reconciled blob, or <see langword="null"/> when it was empty (#485).</returns>
|
||||
private byte[]? ReconcileDriversFromBlob(DeploymentId deploymentId, byte[]? blob)
|
||||
{
|
||||
blob ??= Array.Empty<byte>();
|
||||
|
||||
// Issue #485 — no bytes is no answer, not "a configuration with no drivers". Parsing zero bytes
|
||||
// yields zero specs, and DriverSpawnPlanner then plans EVERY running child for StopChild: a missing
|
||||
// deployment row (or a half-written one) silently takes this node's entire field I/O down while the
|
||||
@@ -2362,6 +2573,16 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
|
||||
|
||||
private void TryRecoverFromStale()
|
||||
{
|
||||
// Defensive: FetchAndCache never enters Stale (BootstrapFromCache always Becomes Steady, so
|
||||
// the retry-db timer that drives this is never started), and its recovery must NOT read
|
||||
// central SQL. If we somehow land here, leave Steady rather than re-reading Deployments.
|
||||
if (_fetchAndCacheMode)
|
||||
{
|
||||
Timers.Cancel("retry-db");
|
||||
Become(Steady);
|
||||
return;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
using var db = _dbFactory.CreateDbContext();
|
||||
|
||||
@@ -282,6 +282,16 @@ public static class ServiceCollectionExtensions
|
||||
// harnesses) rather than given a null-object, so DriverHostActor skips caching outright
|
||||
// instead of pretending to cache into a sink that drops everything.
|
||||
var deploymentArtifactCache = resolver.GetService<IDeploymentArtifactCache>();
|
||||
// Per-cluster mesh Phase 3: ConfigSource:Mode selects where this driver reads config.
|
||||
// FetchAndCache pulls the artifact from central over gRPC and reads only the LocalDb cache;
|
||||
// Direct (default) reads central SQL as before. The fetcher is resolved only in FetchAndCache
|
||||
// mode (registered by the Host under hasDriver); its absence there is a misconfiguration the
|
||||
// actor fails the apply on rather than fetching nothing forever.
|
||||
var configSourceOptions =
|
||||
resolver.GetService<IOptions<ConfigSourceOptions>>()?.Value ?? new ConfigSourceOptions();
|
||||
var fetchAndCacheMode = string.Equals(
|
||||
configSourceOptions.Mode, ConfigSourceOptions.ModeFetchAndCache, StringComparison.OrdinalIgnoreCase);
|
||||
var artifactFetcher = fetchAndCacheMode ? resolver.GetService<IDeploymentArtifactFetcher>() : null;
|
||||
// Where this actor publishes its Primary-gate verdict for the alarm store-and-forward
|
||||
// drain, which runs on a timer and so cannot read RedundancyStateChanged itself.
|
||||
// Registered by AddAlarmHistorian; absent when no durable sink is configured, in which
|
||||
@@ -455,7 +465,9 @@ public static class ServiceCollectionExtensions
|
||||
invokerFactory: invokerFactory,
|
||||
deploymentArtifactCache: deploymentArtifactCache,
|
||||
redundancyRoleView: redundancyRoleView,
|
||||
replicationPeerHost: replicationPeerHost),
|
||||
replicationPeerHost: replicationPeerHost,
|
||||
fetchAndCacheMode: fetchAndCacheMode,
|
||||
artifactFetcher: artifactFetcher),
|
||||
DriverHostActorName);
|
||||
registry.Register<DriverHostActorKey>(driverHost);
|
||||
|
||||
|
||||
@@ -8,6 +8,9 @@
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Akka.Hosting"/>
|
||||
<PackageReference Include="Akka.Cluster.Tools"/>
|
||||
<!-- Node-side gRPC client that fetches the deployed-configuration artifact from central
|
||||
(per-cluster mesh Phase 3, FetchAndCache mode). The generated stub lives in Commons. -->
|
||||
<PackageReference Include="Grpc.Net.Client"/>
|
||||
<!-- Core LocalDb only (ILocalDb + the connection/transaction seam) — the deployment-artifact
|
||||
cache lives here, but the sync engine and its gRPC endpoint are the Host's concern. -->
|
||||
<PackageReference Include="ZB.MOM.WW.LocalDb" />
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
using Microsoft.Extensions.Options;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// A <c>FetchAndCache</c> node with no endpoint or no key does not error — it simply never
|
||||
/// applies a deployment, and the deploy times out naming a node that is otherwise healthy. These
|
||||
/// tests make that misconfiguration a loud host-start failure instead.
|
||||
/// </summary>
|
||||
public class ConfigSourceOptionsValidatorTests
|
||||
{
|
||||
private static ValidateOptionsResult Validate(ConfigSourceOptions o) =>
|
||||
new ConfigSourceOptionsValidator().Validate(ConfigSourceOptions.SectionName, o);
|
||||
|
||||
private static ConfigSourceOptions ValidFetch() => new()
|
||||
{
|
||||
Mode = ConfigSourceOptions.ModeFetchAndCache,
|
||||
CentralFetchEndpoints = ["http://central-1:4055", "http://central-2:4055"],
|
||||
ApiKey = "shared-key",
|
||||
FetchTimeoutSeconds = 30,
|
||||
};
|
||||
|
||||
[Fact]
|
||||
public void Default_options_are_valid()
|
||||
{
|
||||
// The default is Direct with everything empty — the pre-Phase-3 behaviour. A node that never
|
||||
// sets this section must validate, which is the premise of the dark switch.
|
||||
Validate(new ConfigSourceOptions()).Succeeded.ShouldBeTrue();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Unknown_mode_fails()
|
||||
{
|
||||
var result = Validate(new ConfigSourceOptions { Mode = "cache" });
|
||||
|
||||
result.Failed.ShouldBeTrue();
|
||||
result.FailureMessage.ShouldContain("cache");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Mode_matching_is_case_insensitive()
|
||||
{
|
||||
Validate(new ConfigSourceOptions
|
||||
{
|
||||
Mode = "fetchandcache",
|
||||
CentralFetchEndpoints = ["http://central-1:4055"],
|
||||
ApiKey = "k",
|
||||
}).Succeeded.ShouldBeTrue();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void FetchAndCache_with_no_endpoints_fails()
|
||||
{
|
||||
var o = ValidFetch();
|
||||
o.CentralFetchEndpoints = [];
|
||||
|
||||
var result = Validate(o);
|
||||
|
||||
result.Failed.ShouldBeTrue();
|
||||
result.FailureMessage.ShouldContain(nameof(ConfigSourceOptions.CentralFetchEndpoints));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void FetchAndCache_with_a_non_http_endpoint_fails()
|
||||
{
|
||||
var o = ValidFetch();
|
||||
o.CentralFetchEndpoints = ["central-1:4055"];
|
||||
|
||||
var result = Validate(o);
|
||||
|
||||
result.Failed.ShouldBeTrue();
|
||||
result.FailureMessage.ShouldContain("central-1:4055");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void FetchAndCache_with_an_empty_key_fails()
|
||||
{
|
||||
var o = ValidFetch();
|
||||
o.ApiKey = "";
|
||||
|
||||
var result = Validate(o);
|
||||
|
||||
result.Failed.ShouldBeTrue();
|
||||
result.FailureMessage.ShouldContain(nameof(ConfigSourceOptions.ApiKey));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void FetchAndCache_with_a_non_positive_timeout_fails()
|
||||
{
|
||||
var o = ValidFetch();
|
||||
o.FetchTimeoutSeconds = 0;
|
||||
|
||||
var result = Validate(o);
|
||||
|
||||
result.Failed.ShouldBeTrue();
|
||||
result.FailureMessage.ShouldContain(nameof(ConfigSourceOptions.FetchTimeoutSeconds));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void A_fully_populated_FetchAndCache_config_is_valid()
|
||||
{
|
||||
Validate(ValidFetch()).Succeeded.ShouldBeTrue();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Direct_ignores_the_empty_fetch_surface()
|
||||
{
|
||||
// Under Direct the endpoints/key are never read, so their emptiness must not fail validation —
|
||||
// otherwise every admin-only and every Direct node would be forced to carry fetch config.
|
||||
Validate(new ConfigSourceOptions
|
||||
{
|
||||
Mode = ConfigSourceOptions.ModeDirect,
|
||||
CentralFetchEndpoints = [],
|
||||
ApiKey = "",
|
||||
FetchTimeoutSeconds = 0,
|
||||
}).Succeeded.ShouldBeTrue();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Protos.DeploymentArtifact.V1;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Commons.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// A compile-touch test: it exists so that the repo's first locally-compiled proto
|
||||
/// (<c>deployment_artifact.proto</c>) is proven to generate both the client stub and the server
|
||||
/// base and the message types. If Grpc.Tools codegen regresses, this file stops compiling —
|
||||
/// which, under TreatWarningsAsErrors, breaks the build loudly rather than silently.
|
||||
/// </summary>
|
||||
public class DeploymentArtifactProtoTests
|
||||
{
|
||||
[Fact]
|
||||
public void The_request_message_carries_the_deployment_id()
|
||||
{
|
||||
var request = new FetchRequest { DeploymentId = "abc" };
|
||||
|
||||
request.DeploymentId.ShouldBe("abc");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void The_chunk_message_carries_bytes()
|
||||
{
|
||||
var chunk = new ArtifactChunk { Data = Google.Protobuf.ByteString.CopyFromUtf8("x") };
|
||||
|
||||
chunk.Data.Length.ShouldBe(1);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void The_service_base_and_client_types_generate()
|
||||
{
|
||||
// Referencing these types is the whole assertion — GrpcServices="Both" must emit both.
|
||||
typeof(DeploymentArtifactService.DeploymentArtifactServiceBase).ShouldNotBeNull();
|
||||
typeof(DeploymentArtifactService.DeploymentArtifactServiceClient).ShouldNotBeNull();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,164 @@
|
||||
using Google.Protobuf;
|
||||
using Grpc.Core;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.AdminUI.Grpc;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Protos.DeploymentArtifact.V1;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.AdminUI.Tests.Grpc;
|
||||
|
||||
/// <summary>
|
||||
/// The serve side of Phase 3: streams a sealed deployment's bytes, and collapses unknown /
|
||||
/// not-sealed / empty into one indistinguishable NotFound (existence-hiding + #485).
|
||||
/// </summary>
|
||||
public class DeploymentArtifactServiceTests
|
||||
{
|
||||
private static IDbContextFactory<OtOpcUaConfigDbContext> Factory(string name) => new NamedFactory(name);
|
||||
|
||||
private static DeploymentArtifactGrpcService Service(string dbName) =>
|
||||
new(Factory(dbName), NullLogger<DeploymentArtifactGrpcService>.Instance);
|
||||
|
||||
private static Guid Seed(string dbName, DeploymentStatus status, byte[] blob)
|
||||
{
|
||||
var id = Guid.NewGuid();
|
||||
using var db = new OtOpcUaConfigDbContext(
|
||||
new DbContextOptionsBuilder<OtOpcUaConfigDbContext>().UseInMemoryDatabase(dbName).Options);
|
||||
db.Deployments.Add(new Deployment
|
||||
{
|
||||
DeploymentId = id,
|
||||
RevisionHash = new string('a', 64),
|
||||
Status = status,
|
||||
CreatedBy = "test",
|
||||
ArtifactBlob = blob,
|
||||
});
|
||||
db.SaveChanges();
|
||||
return id;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_sealed_non_empty_blob_streams_back_byte_equal_and_chunk_bounded()
|
||||
{
|
||||
var dbName = $"artsvc-{Guid.NewGuid():N}";
|
||||
// ~300 KB > 2 * 128 KiB -> exactly 3 chunks.
|
||||
var blob = new byte[300 * 1024];
|
||||
for (var i = 0; i < blob.Length; i++) blob[i] = (byte)(i % 251);
|
||||
var id = Seed(dbName, DeploymentStatus.Sealed, blob);
|
||||
|
||||
var writer = new CollectingStreamWriter();
|
||||
await Service(dbName).Fetch(
|
||||
new FetchRequest { DeploymentId = id.ToString() }, writer, new FakeServerCallContext());
|
||||
|
||||
writer.Chunks.Count.ShouldBe(3);
|
||||
writer.Chunks[0].Data.Length.ShouldBe(128 * 1024);
|
||||
writer.Chunks[1].Data.Length.ShouldBe(128 * 1024);
|
||||
writer.Reassembled().ShouldBe(blob);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task An_unknown_id_throws_NotFound()
|
||||
{
|
||||
var dbName = $"artsvc-{Guid.NewGuid():N}";
|
||||
Seed(dbName, DeploymentStatus.Sealed, [1, 2, 3]);
|
||||
|
||||
var ex = await Should.ThrowAsync<RpcException>(() => Service(dbName).Fetch(
|
||||
new FetchRequest { DeploymentId = Guid.NewGuid().ToString() },
|
||||
new CollectingStreamWriter(), new FakeServerCallContext()));
|
||||
|
||||
ex.StatusCode.ShouldBe(StatusCode.NotFound);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_non_sealed_deployment_with_a_blob_is_served()
|
||||
{
|
||||
// A node fetches DURING apply, before the deployment is sealed (seal-needs-ack →
|
||||
// ack-needs-fetch). Gating on Sealed would deadlock every FetchAndCache deploy. So a
|
||||
// non-sealed deployment that has a blob must stream — matching Direct mode reading the same
|
||||
// AwaitingApplyAcks row from SQL. (Regression for the Phase 3 live-gate deadlock.)
|
||||
var dbName = $"artsvc-{Guid.NewGuid():N}";
|
||||
var blob = new byte[] { 1, 2, 3 };
|
||||
var id = Seed(dbName, DeploymentStatus.AwaitingApplyAcks, blob);
|
||||
|
||||
var writer = new CollectingStreamWriter();
|
||||
await Service(dbName).Fetch(
|
||||
new FetchRequest { DeploymentId = id.ToString() }, writer, new FakeServerCallContext());
|
||||
|
||||
writer.Reassembled().ShouldBe(blob);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_zero_length_blob_throws_NotFound()
|
||||
{
|
||||
// The serve-side #485 guard: never stream empty bytes as if they were a valid empty config.
|
||||
var dbName = $"artsvc-{Guid.NewGuid():N}";
|
||||
var id = Seed(dbName, DeploymentStatus.Sealed, []);
|
||||
|
||||
var ex = await Should.ThrowAsync<RpcException>(() => Service(dbName).Fetch(
|
||||
new FetchRequest { DeploymentId = id.ToString() },
|
||||
new CollectingStreamWriter(), new FakeServerCallContext()));
|
||||
|
||||
ex.StatusCode.ShouldBe(StatusCode.NotFound);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_non_guid_id_throws_NotFound()
|
||||
{
|
||||
var ex = await Should.ThrowAsync<RpcException>(() => Service($"artsvc-{Guid.NewGuid():N}").Fetch(
|
||||
new FetchRequest { DeploymentId = "not-a-guid" },
|
||||
new CollectingStreamWriter(), new FakeServerCallContext()));
|
||||
|
||||
ex.StatusCode.ShouldBe(StatusCode.NotFound);
|
||||
}
|
||||
|
||||
private sealed class CollectingStreamWriter : IServerStreamWriter<ArtifactChunk>
|
||||
{
|
||||
public List<ArtifactChunk> Chunks { get; } = [];
|
||||
public WriteOptions? WriteOptions { get; set; }
|
||||
|
||||
public Task WriteAsync(ArtifactChunk message)
|
||||
{
|
||||
Chunks.Add(message);
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
public byte[] Reassembled()
|
||||
{
|
||||
using var ms = new MemoryStream();
|
||||
foreach (var c in Chunks) c.Data.WriteTo(ms);
|
||||
return ms.ToArray();
|
||||
}
|
||||
}
|
||||
|
||||
private sealed class NamedFactory(string name) : IDbContextFactory<OtOpcUaConfigDbContext>
|
||||
{
|
||||
public OtOpcUaConfigDbContext CreateDbContext() =>
|
||||
new(new DbContextOptionsBuilder<OtOpcUaConfigDbContext>().UseInMemoryDatabase(name).Options);
|
||||
|
||||
public Task<OtOpcUaConfigDbContext> CreateDbContextAsync(CancellationToken ct = default) =>
|
||||
Task.FromResult(CreateDbContext());
|
||||
}
|
||||
|
||||
/// <summary>Minimal <see cref="ServerCallContext"/> — the service reads only the cancellation token.</summary>
|
||||
private sealed class FakeServerCallContext : ServerCallContext
|
||||
{
|
||||
protected override string MethodCore => "/deployment_artifact.v1.DeploymentArtifactService/Fetch";
|
||||
protected override string HostCore => "localhost";
|
||||
protected override string PeerCore => "ipv4:127.0.0.1:0";
|
||||
protected override DateTime DeadlineCore => DateTime.MaxValue;
|
||||
protected override Metadata RequestHeadersCore => [];
|
||||
protected override CancellationToken CancellationTokenCore => CancellationToken.None;
|
||||
protected override Metadata ResponseTrailersCore { get; } = [];
|
||||
protected override Status StatusCore { get; set; }
|
||||
protected override WriteOptions? WriteOptionsCore { get; set; }
|
||||
protected override AuthContext AuthContextCore => new(null, new Dictionary<string, List<AuthProperty>>());
|
||||
|
||||
protected override ContextPropagationToken CreatePropagationTokenCore(ContextPropagationOptions? options) =>
|
||||
throw new NotSupportedException();
|
||||
|
||||
protected override Task WriteResponseHeadersAsyncCore(Metadata responseHeaders) => Task.CompletedTask;
|
||||
}
|
||||
}
|
||||
+152
@@ -0,0 +1,152 @@
|
||||
using Grpc.Core;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
using ZB.MOM.WW.OtOpcUa.Host.Configuration;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
|
||||
|
||||
/// <summary>
|
||||
/// Per-cluster mesh Phase 3 (Task 3) — the config-serve artifact endpoint's inbound gate.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The <c>DeploymentArtifactService</c> streams a driver node the exact configuration it will
|
||||
/// boot on. Anything able to reach central's serve port could otherwise pull the whole deployed
|
||||
/// config, so the endpoint is gated by a shared bearer key, fail-closed, constant-time — the
|
||||
/// same contract as <see cref="LocalDbSyncAuthInterceptor"/>, scoped to a different service path.
|
||||
/// </remarks>
|
||||
public sealed class ConfigServeAuthInterceptorTests
|
||||
{
|
||||
private const string FetchMethod = "/deployment_artifact.v1.DeploymentArtifactService/Fetch";
|
||||
private const string OtherMethod = "/localdb_sync.v1.LocalDbSync/Sync";
|
||||
|
||||
private static ConfigServeAuthInterceptor CreateInterceptor(string? apiKey)
|
||||
=> new(
|
||||
Options.Create(new ConfigServeOptions { ApiKey = apiKey ?? string.Empty }),
|
||||
NullLogger<ConfigServeAuthInterceptor>.Instance);
|
||||
|
||||
private static ServerCallContext CreateContext(string method, string? authorizationHeader)
|
||||
{
|
||||
var headers = new Metadata();
|
||||
if (authorizationHeader is not null)
|
||||
headers.Add("authorization", authorizationHeader);
|
||||
|
||||
return new FakeServerCallContext(method, headers);
|
||||
}
|
||||
|
||||
/// <summary>Invokes the interceptor's server-streaming path (how <c>Fetch</c> actually runs).</summary>
|
||||
private static Task Invoke(ConfigServeAuthInterceptor interceptor, ServerCallContext context)
|
||||
=> interceptor.ServerStreamingServerHandler<string, string>(
|
||||
"request", responseStream: null!, context, (_, _, _) => Task.CompletedTask);
|
||||
|
||||
[Fact]
|
||||
public async Task NonServeMethod_PassesThrough_EvenWithNoKeyConfigured()
|
||||
{
|
||||
// The interceptor shares the AddGrpc pipeline with the LocalDb sync interceptor, so it sees
|
||||
// every call. It must be scoped strictly to the artifact service — a sync call is not its
|
||||
// business.
|
||||
var interceptor = CreateInterceptor(apiKey: null);
|
||||
var context = CreateContext(OtherMethod, authorizationHeader: null);
|
||||
|
||||
await Invoke(interceptor, context); // no throw
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task FetchMethod_WithNoKeyConfigured_IsDenied_EvenWithABearerToken()
|
||||
{
|
||||
// Fail-closed. "No key configured" is the DEFAULT — treating it as "no auth required" would
|
||||
// expose the deployed config on exactly the shape most nodes ship with. A token must not help.
|
||||
var interceptor = CreateInterceptor(apiKey: null);
|
||||
var context = CreateContext(FetchMethod, "Bearer anything-at-all");
|
||||
|
||||
var ex = await Should.ThrowAsync<RpcException>(() => Invoke(interceptor, context));
|
||||
ex.StatusCode.ShouldBe(StatusCode.Unauthenticated);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task FetchMethod_WithNoBearerToken_IsDenied()
|
||||
{
|
||||
var interceptor = CreateInterceptor("the-shared-key");
|
||||
var context = CreateContext(FetchMethod, authorizationHeader: null);
|
||||
|
||||
var ex = await Should.ThrowAsync<RpcException>(() => Invoke(interceptor, context));
|
||||
ex.StatusCode.ShouldBe(StatusCode.Unauthenticated);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task FetchMethod_WithWrongBearerToken_IsDenied()
|
||||
{
|
||||
var interceptor = CreateInterceptor("the-shared-key");
|
||||
var context = CreateContext(FetchMethod, "Bearer the-wrong-key");
|
||||
|
||||
var ex = await Should.ThrowAsync<RpcException>(() => Invoke(interceptor, context));
|
||||
ex.StatusCode.ShouldBe(StatusCode.Unauthenticated);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task FetchMethod_WithCorrectBearerToken_PassesThrough()
|
||||
{
|
||||
var interceptor = CreateInterceptor("the-shared-key");
|
||||
var context = CreateContext(FetchMethod, "Bearer the-shared-key");
|
||||
|
||||
await Invoke(interceptor, context); // no throw
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task FetchMethod_TokenComparison_IsNotAPrefixMatch()
|
||||
{
|
||||
// A prefix/StartsWith comparison would accept a truncated key and make the secret
|
||||
// recoverable one character at a time.
|
||||
var interceptor = CreateInterceptor("the-shared-key");
|
||||
var context = CreateContext(FetchMethod, "Bearer the-shared-ke");
|
||||
|
||||
var ex = await Should.ThrowAsync<RpcException>(() => Invoke(interceptor, context));
|
||||
ex.StatusCode.ShouldBe(StatusCode.Unauthenticated);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task UnaryPath_IsAlsoGated()
|
||||
{
|
||||
// Fetch is server-streaming today, but gating only that path would leave any future unary
|
||||
// method on the same service open. Gate every handler shape.
|
||||
var interceptor = CreateInterceptor("the-shared-key");
|
||||
var context = CreateContext(FetchMethod, "Bearer the-wrong-key");
|
||||
|
||||
var ex = await Should.ThrowAsync<RpcException>(() =>
|
||||
interceptor.UnaryServerHandler<string, string>(
|
||||
"request", context, (_, _) => Task.FromResult("ok")));
|
||||
|
||||
ex.StatusCode.ShouldBe(StatusCode.Unauthenticated);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Minimal <see cref="ServerCallContext"/> carrying just a method name and request headers —
|
||||
/// the only two things the interceptor reads. Hand-rolled because
|
||||
/// <c>Grpc.Core.Testing.TestServerCallContext</c> ships only in the retired native package.
|
||||
/// </summary>
|
||||
private sealed class FakeServerCallContext(string method, Metadata requestHeaders)
|
||||
: ServerCallContext
|
||||
{
|
||||
protected override string MethodCore => method;
|
||||
protected override string HostCore => "localhost";
|
||||
protected override string PeerCore => "ipv4:127.0.0.1:12345";
|
||||
protected override DateTime DeadlineCore => DateTime.UtcNow.AddMinutes(1);
|
||||
protected override Metadata RequestHeadersCore => requestHeaders;
|
||||
protected override CancellationToken CancellationTokenCore => CancellationToken.None;
|
||||
protected override Metadata ResponseTrailersCore { get; } = [];
|
||||
protected override Status StatusCore { get; set; }
|
||||
protected override WriteOptions? WriteOptionsCore { get; set; }
|
||||
|
||||
protected override AuthContext AuthContextCore { get; } =
|
||||
new(null, new Dictionary<string, List<AuthProperty>>());
|
||||
|
||||
protected override ContextPropagationToken CreatePropagationTokenCore(
|
||||
ContextPropagationOptions? options)
|
||||
=> throw new NotSupportedException();
|
||||
|
||||
protected override Task WriteResponseHeadersAsyncCore(Metadata responseHeaders)
|
||||
=> Task.CompletedTask;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
using System.Net;
|
||||
using Microsoft.AspNetCore.Builder;
|
||||
using Microsoft.AspNetCore.Server.Kestrel.Core;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Host.Configuration;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
|
||||
|
||||
/// <summary>
|
||||
/// Per-cluster mesh Phase 3 (Task 3) — the coexistence of the ConfigServe artifact h2c listener
|
||||
/// with the LocalDb-sync h2c listener on a fused admin+driver node.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Why this is the load-bearing test.</b> A fused central node can carry BOTH dedicated
|
||||
/// h2c ports (LocalDb sync + ConfigServe). Any explicit Kestrel <c>Listen*</c> makes Kestrel
|
||||
/// discard <c>ASPNETCORE_URLS</c> wholesale, so the existing HTTP surface must be re-bound
|
||||
/// exactly ONCE alongside the two dedicated ports. Re-binding it twice (one block per h2c
|
||||
/// port) throws "address already in use"; re-binding it zero times silently unbinds the
|
||||
/// AdminUI behind Traefik. This test pins that all three answer simultaneously.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Like <see cref="LocalDbSyncListenerTests"/>, this drives a minimal <c>WebApplication</c>
|
||||
/// shaped exactly like <c>Program.cs</c>'s merged listener block rather than booting the real
|
||||
/// host (which needs SQL, an Akka mesh and LDAP). What is under test is the Kestrel binding
|
||||
/// contract, fully reproduced here; the real end-to-end fetch is the Task 8 boundary test.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class ConfigServeListenerTests
|
||||
{
|
||||
[Fact]
|
||||
public async Task BothDedicatedH2cPorts_AndTheHttpSurface_AnswerSimultaneously()
|
||||
{
|
||||
var httpPort = GetFreePort();
|
||||
var syncPort = GetFreePort();
|
||||
var configServePort = GetFreePort();
|
||||
|
||||
var builder = WebApplication.CreateBuilder();
|
||||
builder.Environment.ApplicationName = typeof(ConfigServeListenerTests).Assembly.GetName().Name!;
|
||||
builder.Services.AddGrpc();
|
||||
|
||||
// Exactly the shape Program.cs's merged block applies: compute the existing surface ONCE,
|
||||
// apply it ONCE, then add each configured dedicated h2c port.
|
||||
var existingBindings = KestrelHttpBinding.Parse($"http://localhost:{httpPort}");
|
||||
builder.WebHost.ConfigureKestrel(kestrel =>
|
||||
{
|
||||
foreach (var binding in existingBindings)
|
||||
binding.Apply(kestrel);
|
||||
|
||||
kestrel.ListenAnyIP(syncPort, o => o.Protocols = HttpProtocols.Http2);
|
||||
kestrel.ListenAnyIP(configServePort, o => o.Protocols = HttpProtocols.Http2);
|
||||
});
|
||||
|
||||
await using var app = builder.Build();
|
||||
app.MapGet("/healthz", () => "ok");
|
||||
await app.StartAsync(TestContext.Current.CancellationToken);
|
||||
|
||||
try
|
||||
{
|
||||
// The re-bound HTTP/1.1 surface still answers (the AdminUI/Traefik guarantee).
|
||||
using var http1 = new HttpClient();
|
||||
(await http1.GetStringAsync(
|
||||
$"http://localhost:{httpPort}/healthz", TestContext.Current.CancellationToken))
|
||||
.ShouldBe("ok");
|
||||
|
||||
// Both dedicated ports negotiate prior-knowledge h2c. A 404 is a fine result — it proves
|
||||
// the HTTP/2 connection was established and routed, which is all that is in question.
|
||||
using var http2 = new HttpClient
|
||||
{
|
||||
DefaultRequestVersion = HttpVersion.Version20,
|
||||
DefaultVersionPolicy = HttpVersionPolicy.RequestVersionExact,
|
||||
};
|
||||
|
||||
using var syncResponse = await http2.GetAsync(
|
||||
$"http://localhost:{syncPort}/", TestContext.Current.CancellationToken);
|
||||
syncResponse.Version.ShouldBe(HttpVersion.Version20);
|
||||
|
||||
using var configServeResponse = await http2.GetAsync(
|
||||
$"http://localhost:{configServePort}/", TestContext.Current.CancellationToken);
|
||||
configServeResponse.Version.ShouldBe(HttpVersion.Version20);
|
||||
}
|
||||
finally
|
||||
{
|
||||
await app.StopAsync(TestContext.Current.CancellationToken);
|
||||
}
|
||||
}
|
||||
|
||||
private static int GetFreePort()
|
||||
{
|
||||
using var listener = new System.Net.Sockets.TcpListener(IPAddress.Loopback, 0);
|
||||
listener.Start();
|
||||
var port = ((IPEndPoint)listener.LocalEndpoint).Port;
|
||||
listener.Stop();
|
||||
return port;
|
||||
}
|
||||
}
|
||||
+181
@@ -0,0 +1,181 @@
|
||||
using System.Net;
|
||||
using System.Security.Cryptography;
|
||||
using Microsoft.AspNetCore.Builder;
|
||||
using Microsoft.AspNetCore.Server.Kestrel.Core;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.AdminUI.Grpc;
|
||||
using ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
|
||||
using ZB.MOM.WW.OtOpcUa.Host.Configuration;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
|
||||
|
||||
/// <summary>
|
||||
/// Per-cluster mesh Phase 3 (Task 8) — the REAL gRPC artifact-fetch boundary. An in-memory fake
|
||||
/// client (Task 4) proves the fetcher's logic; this proves a real stream crosses a real Kestrel
|
||||
/// h2c listener through the real <see cref="ConfigServeAuthInterceptor"/> into the real
|
||||
/// <see cref="DeploymentArtifactGrpcService"/> and back through the real
|
||||
/// <see cref="GrpcDeploymentArtifactFetcher"/>.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The Phase-2 lesson applied: a passing fake client cannot distinguish "the boundary works" from
|
||||
/// "something else supplied the bytes". The wrong-key control (assertion 2) is the falsifier — it
|
||||
/// must return null on the SAME server where the right key succeeds.
|
||||
/// </remarks>
|
||||
[Trait("Category", "ArtifactBoundary")]
|
||||
public sealed class DeploymentArtifactFetchBoundaryTests
|
||||
{
|
||||
private const string ApiKey = "the-shared-node-key";
|
||||
|
||||
[Fact]
|
||||
public async Task RightKey_FetchesAndReassembles_ByteEqual_AndVerifiesAgainstTheRevisionHash()
|
||||
{
|
||||
// >256 KB so the stream really chunks (3 × 128 KiB).
|
||||
var blob = RandomBlob(300 * 1024);
|
||||
var revisionHash = Sha256Hex(blob);
|
||||
|
||||
var (server, deploymentId, port) = await StartServerAsync(revisionHash, blob);
|
||||
await using var _ = server;
|
||||
|
||||
var fetcher = Fetcher([$"http://localhost:{port}"], ApiKey);
|
||||
var result = await fetcher.FetchAsync(deploymentId, revisionHash, TestContext.Current.CancellationToken);
|
||||
|
||||
result.ShouldBe(blob);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task WrongKey_ReturnsNull_WhileTheRightKeySucceedsOnTheSameServer()
|
||||
{
|
||||
var blob = RandomBlob(4096);
|
||||
var revisionHash = Sha256Hex(blob);
|
||||
|
||||
var (server, deploymentId, port) = await StartServerAsync(revisionHash, blob);
|
||||
await using var _ = server;
|
||||
|
||||
// Falsifiability control: the interceptor rejects a wrong key -> the fetcher gets an RpcException
|
||||
// -> null. Without this, a green right-key fetch cannot prove the boundary is what carries bytes.
|
||||
var wrong = await Fetcher([$"http://localhost:{port}"], "not-the-key")
|
||||
.FetchAsync(deploymentId, revisionHash, TestContext.Current.CancellationToken);
|
||||
wrong.ShouldBeNull();
|
||||
|
||||
// Same server, right key: succeeds — proving the null above was the auth gate, not a dead server.
|
||||
var right = await Fetcher([$"http://localhost:{port}"], ApiKey)
|
||||
.FetchAsync(deploymentId, revisionHash, TestContext.Current.CancellationToken);
|
||||
right.ShouldBe(blob);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AnUnknownDeploymentId_ReturnsNull()
|
||||
{
|
||||
var blob = RandomBlob(4096);
|
||||
var (server, _, port) = await StartServerAsync(Sha256Hex(blob), blob);
|
||||
await using var __ = server;
|
||||
|
||||
var result = await Fetcher([$"http://localhost:{port}"], ApiKey)
|
||||
.FetchAsync(Guid.NewGuid().ToString(), Sha256Hex(blob), TestContext.Current.CancellationToken);
|
||||
|
||||
result.ShouldBeNull();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Failover_FromADeadEndpoint_ToTheRealServer_StillReturnsTheBytes()
|
||||
{
|
||||
var blob = RandomBlob(4096);
|
||||
var revisionHash = Sha256Hex(blob);
|
||||
|
||||
var (server, deploymentId, port) = await StartServerAsync(revisionHash, blob);
|
||||
await using var _ = server;
|
||||
var deadPort = GetFreePort(); // nothing listening
|
||||
|
||||
var fetcher = Fetcher([$"http://localhost:{deadPort}", $"http://localhost:{port}"], ApiKey);
|
||||
var result = await fetcher.FetchAsync(deploymentId, revisionHash, TestContext.Current.CancellationToken);
|
||||
|
||||
result.ShouldBe(blob);
|
||||
}
|
||||
|
||||
// ---- harness --------------------------------------------------------------------------------
|
||||
|
||||
/// <summary>
|
||||
/// Stands a minimal real Host: an h2c-only Kestrel listener serving the real
|
||||
/// <see cref="DeploymentArtifactGrpcService"/> behind the real <see cref="ConfigServeAuthInterceptor"/>,
|
||||
/// backed by an in-memory config DB seeded with one sealed deployment.
|
||||
/// </summary>
|
||||
private static async Task<(WebApplication Server, string DeploymentId, int Port)> StartServerAsync(
|
||||
string revisionHash, byte[] blob)
|
||||
{
|
||||
var port = GetFreePort();
|
||||
var id = Guid.NewGuid();
|
||||
var dbName = $"artboundary-{id:N}";
|
||||
|
||||
var builder = WebApplication.CreateBuilder();
|
||||
builder.Environment.ApplicationName = typeof(DeploymentArtifactFetchBoundaryTests).Assembly.GetName().Name!;
|
||||
builder.Logging.ClearProviders();
|
||||
|
||||
builder.Services.Configure<ConfigServeOptions>(o => o.ApiKey = ApiKey);
|
||||
builder.Services.AddGrpc(o => o.Interceptors.Add<ConfigServeAuthInterceptor>());
|
||||
builder.Services.AddDbContextFactory<OtOpcUaConfigDbContext>(o => o.UseInMemoryDatabase(dbName));
|
||||
|
||||
builder.WebHost.ConfigureKestrel(k =>
|
||||
k.ListenAnyIP(port, o => o.Protocols = HttpProtocols.Http2));
|
||||
|
||||
var app = builder.Build();
|
||||
app.MapGrpcService<DeploymentArtifactGrpcService>();
|
||||
|
||||
// Seed the sealed deployment the service will serve.
|
||||
await using (var scope = app.Services.CreateAsyncScope())
|
||||
{
|
||||
var factory = scope.ServiceProvider.GetRequiredService<IDbContextFactory<OtOpcUaConfigDbContext>>();
|
||||
await using var db = await factory.CreateDbContextAsync(TestContext.Current.CancellationToken);
|
||||
db.Deployments.Add(new Deployment
|
||||
{
|
||||
DeploymentId = id,
|
||||
RevisionHash = revisionHash,
|
||||
Status = DeploymentStatus.Sealed,
|
||||
CreatedBy = "test",
|
||||
ArtifactBlob = blob,
|
||||
});
|
||||
await db.SaveChangesAsync(TestContext.Current.CancellationToken);
|
||||
}
|
||||
|
||||
await app.StartAsync(TestContext.Current.CancellationToken);
|
||||
return (app, id.ToString(), port);
|
||||
}
|
||||
|
||||
private static GrpcDeploymentArtifactFetcher Fetcher(string[] endpoints, string apiKey)
|
||||
=> new(
|
||||
Options.Create(new ConfigSourceOptions
|
||||
{
|
||||
Mode = ConfigSourceOptions.ModeFetchAndCache,
|
||||
CentralFetchEndpoints = endpoints,
|
||||
ApiKey = apiKey,
|
||||
FetchTimeoutSeconds = 15,
|
||||
}),
|
||||
NullLogger<GrpcDeploymentArtifactFetcher>.Instance);
|
||||
|
||||
private static byte[] RandomBlob(int size)
|
||||
{
|
||||
var blob = new byte[size];
|
||||
for (var i = 0; i < size; i++) blob[i] = (byte)(i * 31 + 7);
|
||||
return blob;
|
||||
}
|
||||
|
||||
private static string Sha256Hex(byte[] bytes) => Convert.ToHexStringLower(SHA256.HashData(bytes));
|
||||
|
||||
private static int GetFreePort()
|
||||
{
|
||||
using var listener = new System.Net.Sockets.TcpListener(IPAddress.Loopback, 0);
|
||||
listener.Start();
|
||||
var port = ((IPEndPoint)listener.LocalEndpoint).Port;
|
||||
listener.Stop();
|
||||
return port;
|
||||
}
|
||||
}
|
||||
+162
@@ -0,0 +1,162 @@
|
||||
using System.Security.Cryptography;
|
||||
using Google.Protobuf;
|
||||
using Grpc.Core;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Protos.DeploymentArtifact.V1;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.DeploymentCache;
|
||||
|
||||
/// <summary>
|
||||
/// Per-cluster mesh Phase 3 (Task 4) — the node-side gRPC artifact fetcher's logic: reassembly,
|
||||
/// SHA-256 verification, endpoint failover, and the null-on-any-failure (#485) contract. The real
|
||||
/// gRPC stream across a real Kestrel h2c listener is the Task 8 boundary test; here the client is
|
||||
/// a fake streaming canned chunks, so only the fetcher's own logic is under test.
|
||||
/// </summary>
|
||||
public sealed class GrpcDeploymentArtifactFetcherTests
|
||||
{
|
||||
private const string DeploymentId = "11111111-1111-1111-1111-111111111111";
|
||||
|
||||
private static string Sha256Hex(byte[] bytes) => Convert.ToHexStringLower(SHA256.HashData(bytes));
|
||||
|
||||
private static GrpcDeploymentArtifactFetcher Fetcher(
|
||||
string[] endpoints, Func<string, DeploymentArtifactService.DeploymentArtifactServiceClient> factory)
|
||||
=> new(
|
||||
Options.Create(new ConfigSourceOptions
|
||||
{
|
||||
Mode = ConfigSourceOptions.ModeFetchAndCache,
|
||||
CentralFetchEndpoints = endpoints,
|
||||
ApiKey = "k",
|
||||
FetchTimeoutSeconds = 30,
|
||||
}),
|
||||
NullLogger<GrpcDeploymentArtifactFetcher>.Instance,
|
||||
factory);
|
||||
|
||||
[Fact]
|
||||
public async Task Chunks_reassembling_to_the_expected_hash_return_the_exact_bytes()
|
||||
{
|
||||
var blob = new byte[300 * 1024];
|
||||
for (var i = 0; i < blob.Length; i++) blob[i] = (byte)(i % 251);
|
||||
var factory = SingleClient(StreamOf(blob, chunkSize: 128 * 1024));
|
||||
|
||||
var result = await Fetcher(["http://central-1:4055"], factory)
|
||||
.FetchAsync(DeploymentId, Sha256Hex(blob), CancellationToken.None);
|
||||
|
||||
result.ShouldBe(blob);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_hash_mismatch_returns_null()
|
||||
{
|
||||
var blob = new byte[] { 1, 2, 3, 4 };
|
||||
var factory = SingleClient(StreamOf(blob, chunkSize: 2));
|
||||
|
||||
var result = await Fetcher(["http://central-1:4055"], factory)
|
||||
.FetchAsync(DeploymentId, Sha256Hex([9, 9, 9, 9]), CancellationToken.None);
|
||||
|
||||
result.ShouldBeNull();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task An_empty_stream_returns_null()
|
||||
{
|
||||
var factory = SingleClient(StreamOf([], chunkSize: 128 * 1024));
|
||||
|
||||
var result = await Fetcher(["http://central-1:4055"], factory)
|
||||
.FetchAsync(DeploymentId, Sha256Hex([]), CancellationToken.None);
|
||||
|
||||
result.ShouldBeNull();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task When_the_first_endpoint_is_unavailable_it_fails_over_to_the_second()
|
||||
{
|
||||
var blob = new byte[] { 10, 20, 30, 40, 50 };
|
||||
var consulted = new List<string>();
|
||||
var factory = (string endpoint) =>
|
||||
{
|
||||
consulted.Add(endpoint);
|
||||
return endpoint.Contains("central-1")
|
||||
? (DeploymentArtifactService.DeploymentArtifactServiceClient)
|
||||
new ThrowingClient(new RpcException(new Status(StatusCode.Unavailable, "down")))
|
||||
: new FakeClient(StreamOf(blob, chunkSize: 2));
|
||||
};
|
||||
|
||||
var result = await Fetcher(["http://central-1:4055", "http://central-2:4055"], factory)
|
||||
.FetchAsync(DeploymentId, Sha256Hex(blob), CancellationToken.None);
|
||||
|
||||
result.ShouldBe(blob);
|
||||
// The failover must have actually consulted the second endpoint.
|
||||
consulted.ShouldBe(["http://central-1:4055", "http://central-2:4055"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task When_every_endpoint_throws_it_returns_null()
|
||||
{
|
||||
var factory = (string _) =>
|
||||
(DeploymentArtifactService.DeploymentArtifactServiceClient)
|
||||
new ThrowingClient(new RpcException(new Status(StatusCode.NotFound, "unknown deployment")));
|
||||
|
||||
var result = await Fetcher(["http://central-1:4055", "http://central-2:4055"], factory)
|
||||
.FetchAsync(DeploymentId, Sha256Hex([1]), CancellationToken.None);
|
||||
|
||||
result.ShouldBeNull();
|
||||
}
|
||||
|
||||
// ---- fakes --------------------------------------------------------------------------------
|
||||
|
||||
private static Func<string, DeploymentArtifactService.DeploymentArtifactServiceClient> SingleClient(
|
||||
IReadOnlyList<ArtifactChunk> chunks)
|
||||
=> _ => new FakeClient(chunks);
|
||||
|
||||
private static IReadOnlyList<ArtifactChunk> StreamOf(byte[] blob, int chunkSize)
|
||||
{
|
||||
var chunks = new List<ArtifactChunk>();
|
||||
for (var offset = 0; offset < blob.Length; offset += chunkSize)
|
||||
{
|
||||
var len = Math.Min(chunkSize, blob.Length - offset);
|
||||
chunks.Add(new ArtifactChunk { Data = ByteString.CopyFrom(blob, offset, len) });
|
||||
}
|
||||
|
||||
return chunks;
|
||||
}
|
||||
|
||||
/// <summary>A generated client double whose <c>Fetch</c> streams canned chunks.</summary>
|
||||
private sealed class FakeClient(IReadOnlyList<ArtifactChunk> chunks)
|
||||
: DeploymentArtifactService.DeploymentArtifactServiceClient
|
||||
{
|
||||
public override AsyncServerStreamingCall<ArtifactChunk> Fetch(FetchRequest request, CallOptions options)
|
||||
=> new(
|
||||
new ListStreamReader(chunks),
|
||||
Task.FromResult(new Metadata()),
|
||||
() => Status.DefaultSuccess,
|
||||
() => [],
|
||||
() => { });
|
||||
}
|
||||
|
||||
/// <summary>A generated client double whose <c>Fetch</c> throws (endpoint down / NotFound).</summary>
|
||||
private sealed class ThrowingClient(RpcException toThrow)
|
||||
: DeploymentArtifactService.DeploymentArtifactServiceClient
|
||||
{
|
||||
public override AsyncServerStreamingCall<ArtifactChunk> Fetch(FetchRequest request, CallOptions options)
|
||||
=> throw toThrow;
|
||||
}
|
||||
|
||||
private sealed class ListStreamReader(IReadOnlyList<ArtifactChunk> chunks) : IAsyncStreamReader<ArtifactChunk>
|
||||
{
|
||||
private int _index = -1;
|
||||
|
||||
public ArtifactChunk Current => chunks[_index];
|
||||
|
||||
public Task<bool> MoveNext(CancellationToken cancellationToken)
|
||||
{
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
_index++;
|
||||
return Task.FromResult(_index < chunks.Count);
|
||||
}
|
||||
}
|
||||
}
|
||||
+179
@@ -0,0 +1,179 @@
|
||||
using System.Text.Json;
|
||||
using Akka.Actor;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Interfaces;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Fleet;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Types;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.OpcUa;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Drivers;
|
||||
|
||||
/// <summary>
|
||||
/// Per-cluster mesh Phase 3 (Task 6) — <c>FetchAndCache</c> bootstrap. A driver node in this mode
|
||||
/// boots its served state from the LocalDb pointer, reading NO config from central SQL; an empty
|
||||
/// or unreadable cache lands Steady-with-no-revision (the first dispatch fetches), NOT Stale.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Every test injects a <see cref="ThrowingDbFactory"/> for central SQL — reaching Steady (and, in
|
||||
/// two of them, applying a dispatch) proves the boot never touched it. This is the FetchAndCache
|
||||
/// counterpart to <see cref="DriverHostActorBootFromCacheTests"/> (which covers the Direct-mode
|
||||
/// SQL-unreachable fallback, where <c>RunningFromCache</c> is TRUE — here it is normal operation,
|
||||
/// so it stays FALSE).
|
||||
/// </remarks>
|
||||
public sealed class DriverHostActorFetchAndCacheBootstrapTests : RuntimeActorTestBase
|
||||
{
|
||||
private static readonly NodeId TestNode = NodeId.Parse("driver-test");
|
||||
private static readonly RevisionHash CachedRev = RevisionHash.Parse(new string('c', 64));
|
||||
|
||||
private static byte[] Artifact() =>
|
||||
JsonSerializer.SerializeToUtf8Bytes(new { DriverInstances = Array.Empty<object>() });
|
||||
|
||||
[Fact]
|
||||
public void WithCachedArtifact_RestoresServedState_WithoutReadingCentralSql()
|
||||
{
|
||||
var blob = Artifact();
|
||||
var cached = new CachedDeploymentArtifact(
|
||||
DeploymentId.NewId().ToString(), CachedRev.Value, blob, DateTimeOffset.UtcNow.AddHours(-1));
|
||||
var cache = new StubArtifactCache(cached);
|
||||
var publishProbe = CreateTestProbe();
|
||||
|
||||
var actor = Sys.ActorOf(DriverHostActor.Props(
|
||||
new ThrowingDbFactory(), TestNode, ackRouter: null,
|
||||
localRoles: new HashSet<string> { "driver" },
|
||||
opcUaPublishActor: publishProbe.Ref,
|
||||
deploymentArtifactCache: cache,
|
||||
fetchAndCacheMode: true,
|
||||
artifactFetcher: new RecordingFetcher(Artifact())));
|
||||
|
||||
// Served state rebuilt from the CACHED bytes (no ConfigDb read — the factory throws).
|
||||
var rebuild = publishProbe.ExpectMsg<OpcUaPublishActor.RebuildAddressSpace>(TimeSpan.FromSeconds(5));
|
||||
rebuild.Artifact.ShouldBe(blob);
|
||||
|
||||
var snapshot = AskDiagnostics(actor);
|
||||
snapshot.CurrentRevision.ShouldBe(CachedRev);
|
||||
// NORMAL operation, not the degraded Direct-mode fallback — a FetchAndCache node can still fetch.
|
||||
snapshot.RunningFromCache.ShouldBeFalse();
|
||||
cache.UnkeyedReads.ShouldBe(1);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void WithEmptyCache_EntersSteadyNoRevision_AndTheNextDispatchFetches()
|
||||
{
|
||||
var cache = new StubArtifactCache(current: null);
|
||||
var fetcher = new RecordingFetcher(Artifact());
|
||||
var coordinator = CreateTestProbe();
|
||||
|
||||
var actor = Sys.ActorOf(DriverHostActor.Props(
|
||||
new ThrowingDbFactory(), TestNode, coordinator.Ref,
|
||||
localRoles: new HashSet<string> { "driver" },
|
||||
deploymentArtifactCache: cache,
|
||||
fetchAndCacheMode: true,
|
||||
artifactFetcher: fetcher));
|
||||
|
||||
var snapshot = AskDiagnostics(actor);
|
||||
snapshot.CurrentRevision.ShouldBeNull();
|
||||
snapshot.RunningFromCache.ShouldBeFalse();
|
||||
|
||||
// Steady, not Stale: a dispatch is serviced (Stale would ignore it, never acking) and fetches.
|
||||
actor.Tell(new DispatchDeployment(DeploymentId.NewId(), CachedRev, CorrelationId.NewId()));
|
||||
coordinator.ExpectMsg<ApplyAck>(TimeSpan.FromSeconds(5)).Outcome.ShouldBe(ApplyAckOutcome.Applied);
|
||||
fetcher.Calls.ShouldBe(1);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void WithAnUnreadableCache_EntersSteadyNoRevision_NotStale()
|
||||
{
|
||||
var fetcher = new RecordingFetcher(Artifact());
|
||||
var coordinator = CreateTestProbe();
|
||||
|
||||
var actor = Sys.ActorOf(DriverHostActor.Props(
|
||||
new ThrowingDbFactory(), TestNode, coordinator.Ref,
|
||||
localRoles: new HashSet<string> { "driver" },
|
||||
deploymentArtifactCache: new ThrowingReadCache(),
|
||||
fetchAndCacheMode: true,
|
||||
artifactFetcher: fetcher));
|
||||
|
||||
var snapshot = AskDiagnostics(actor);
|
||||
snapshot.CurrentRevision.ShouldBeNull();
|
||||
snapshot.RunningFromCache.ShouldBeFalse();
|
||||
|
||||
// A cache-read fault is NOT Stale (Stale means "central SQL down, retry it" — there is no
|
||||
// config SQL read to recover here). The node is Steady and the next dispatch fetches.
|
||||
actor.Tell(new DispatchDeployment(DeploymentId.NewId(), CachedRev, CorrelationId.NewId()));
|
||||
coordinator.ExpectMsg<ApplyAck>(TimeSpan.FromSeconds(5)).Outcome.ShouldBe(ApplyAckOutcome.Applied);
|
||||
fetcher.Calls.ShouldBe(1);
|
||||
}
|
||||
|
||||
private NodeDiagnosticsSnapshot AskDiagnostics(IActorRef actor)
|
||||
{
|
||||
var probe = CreateTestProbe();
|
||||
AwaitAssert(
|
||||
() =>
|
||||
{
|
||||
actor.Tell(new GetDiagnostics(CorrelationId.NewId()), probe.Ref);
|
||||
probe.ExpectMsg<NodeDiagnosticsSnapshot>(TimeSpan.FromSeconds(2));
|
||||
},
|
||||
duration: TimeSpan.FromSeconds(5));
|
||||
|
||||
actor.Tell(new GetDiagnostics(CorrelationId.NewId()), probe.Ref);
|
||||
return probe.ExpectMsg<NodeDiagnosticsSnapshot>(TimeSpan.FromSeconds(5));
|
||||
}
|
||||
|
||||
private sealed class RecordingFetcher(byte[]? bytes) : IDeploymentArtifactFetcher
|
||||
{
|
||||
private int _calls;
|
||||
public int Calls => Volatile.Read(ref _calls);
|
||||
|
||||
public Task<byte[]?> FetchAsync(string deploymentId, string expectedRevisionHash, CancellationToken ct)
|
||||
{
|
||||
Interlocked.Increment(ref _calls);
|
||||
return Task.FromResult(bytes);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Stands in for an unreachable central SQL Server.</summary>
|
||||
private sealed class ThrowingDbFactory : IDbContextFactory<OtOpcUaConfigDbContext>
|
||||
{
|
||||
public OtOpcUaConfigDbContext CreateDbContext()
|
||||
=> throw new InvalidOperationException("ConfigDb unreachable");
|
||||
}
|
||||
|
||||
private sealed class StubArtifactCache(CachedDeploymentArtifact? current) : IDeploymentArtifactCache
|
||||
{
|
||||
private int _unkeyedReads;
|
||||
public int UnkeyedReads => Volatile.Read(ref _unkeyedReads);
|
||||
|
||||
public Task StoreAsync(string clusterId, string deploymentId, string revisionHash,
|
||||
byte[] artifact, CancellationToken ct = default)
|
||||
=> Task.CompletedTask;
|
||||
|
||||
public Task<CachedDeploymentArtifact?> GetCurrentAsync(string clusterId, CancellationToken ct = default)
|
||||
=> Task.FromResult(current);
|
||||
|
||||
public Task<CachedDeploymentArtifact?> GetCurrentUnkeyedAsync(CancellationToken ct = default)
|
||||
{
|
||||
Interlocked.Increment(ref _unkeyedReads);
|
||||
return Task.FromResult(current);
|
||||
}
|
||||
}
|
||||
|
||||
private sealed class ThrowingReadCache : IDeploymentArtifactCache
|
||||
{
|
||||
public Task StoreAsync(string clusterId, string deploymentId, string revisionHash,
|
||||
byte[] artifact, CancellationToken ct = default)
|
||||
=> Task.CompletedTask;
|
||||
|
||||
public Task<CachedDeploymentArtifact?> GetCurrentAsync(string clusterId, CancellationToken ct = default)
|
||||
=> throw new InvalidOperationException("cache unreadable");
|
||||
|
||||
public Task<CachedDeploymentArtifact?> GetCurrentUnkeyedAsync(CancellationToken ct = default)
|
||||
=> throw new InvalidOperationException("cache unreadable");
|
||||
}
|
||||
}
|
||||
+153
@@ -0,0 +1,153 @@
|
||||
using System.Text.Json;
|
||||
using Akka.Actor;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Types;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Drivers;
|
||||
|
||||
/// <summary>
|
||||
/// Per-cluster mesh Phase 3 (Task 5) — the <c>FetchAndCache</c> apply path: on dispatch the node
|
||||
/// fetches the artifact bytes over gRPC, applies them from bytes in hand, and caches them —
|
||||
/// reading NO config from central SQL. A null fetch is an apply failure that keeps last-known-good
|
||||
/// and does not advance the revision (#485).
|
||||
/// </summary>
|
||||
public sealed class DriverHostActorFetchAndCacheTests : RuntimeActorTestBase
|
||||
{
|
||||
private static readonly NodeId TestNode = NodeId.Parse("driver-test");
|
||||
private static readonly RevisionHash RevA = RevisionHash.Parse(new string('a', 64));
|
||||
|
||||
/// <summary>Builds a minimal but parseable artifact (no drivers) — the fetched-bytes stand-in.</summary>
|
||||
private static byte[] ArtifactBytes() =>
|
||||
JsonSerializer.SerializeToUtf8Bytes(new { DriverInstances = Array.Empty<object>() });
|
||||
|
||||
[Fact]
|
||||
public void FetchReturningGoodBytes_AppliesAndCaches_WithoutReadingCentralSql()
|
||||
{
|
||||
// No Deployment row is seeded: if the apply path read ArtifactBlob from SQL it would get empty
|
||||
// bytes and FAIL (#485). Reaching Applied proves the bytes came from the fetcher, not SQL.
|
||||
var db = NewInMemoryDbFactory();
|
||||
var cache = new RecordingArtifactCache();
|
||||
var fetcher = new RecordingFetcher(ArtifactBytes());
|
||||
var deploymentId = DeploymentId.NewId();
|
||||
|
||||
var coordinator = CreateTestProbe();
|
||||
var actor = Sys.ActorOf(DriverHostActor.Props(
|
||||
db, TestNode, coordinator.Ref,
|
||||
localRoles: new HashSet<string> { "driver" },
|
||||
deploymentArtifactCache: cache,
|
||||
fetchAndCacheMode: true,
|
||||
artifactFetcher: fetcher));
|
||||
|
||||
actor.Tell(new DispatchDeployment(deploymentId, RevA, CorrelationId.NewId()));
|
||||
|
||||
coordinator.ExpectMsg<ApplyAck>(TimeSpan.FromSeconds(5)).Outcome.ShouldBe(ApplyAckOutcome.Applied);
|
||||
AwaitAssert(() => cache.Stores.Count.ShouldBe(1), duration: TimeSpan.FromSeconds(3));
|
||||
cache.Stores[0].RevisionHash.ShouldBe(RevA.Value);
|
||||
fetcher.Calls.ShouldBe(1);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void FetchReturningNull_FailsTheApply_KeepsRevisionUnchanged_AndCachesNothing()
|
||||
{
|
||||
var db = NewInMemoryDbFactory();
|
||||
var cache = new RecordingArtifactCache();
|
||||
var fetcher = new RecordingFetcher(bytes: null); // all endpoints down / NotFound / SHA mismatch
|
||||
var deploymentId = DeploymentId.NewId();
|
||||
|
||||
var coordinator = CreateTestProbe();
|
||||
var actor = Sys.ActorOf(DriverHostActor.Props(
|
||||
db, TestNode, coordinator.Ref,
|
||||
localRoles: new HashSet<string> { "driver" },
|
||||
deploymentArtifactCache: cache,
|
||||
fetchAndCacheMode: true,
|
||||
artifactFetcher: fetcher));
|
||||
|
||||
actor.Tell(new DispatchDeployment(deploymentId, RevA, CorrelationId.NewId()));
|
||||
coordinator.ExpectMsg<ApplyAck>(TimeSpan.FromSeconds(5)).Outcome.ShouldBe(ApplyAckOutcome.Failed);
|
||||
|
||||
Thread.Sleep(300);
|
||||
cache.Stores.ShouldBeEmpty();
|
||||
|
||||
// The revision was NOT advanced: a re-dispatch of the SAME revision fetches AGAIN rather than
|
||||
// short-circuiting as "already current". (Two fetch attempts prove #485's keep-last-known-good.)
|
||||
actor.Tell(new DispatchDeployment(deploymentId, RevA, CorrelationId.NewId()));
|
||||
coordinator.ExpectMsg<ApplyAck>(TimeSpan.FromSeconds(5)).Outcome.ShouldBe(ApplyAckOutcome.Failed);
|
||||
fetcher.Calls.ShouldBe(2);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void RedispatchOfTheAppliedRevision_DoesNotFetchAgain()
|
||||
{
|
||||
var db = NewInMemoryDbFactory();
|
||||
var cache = new RecordingArtifactCache();
|
||||
var fetcher = new RecordingFetcher(ArtifactBytes());
|
||||
var deploymentId = DeploymentId.NewId();
|
||||
|
||||
var coordinator = CreateTestProbe();
|
||||
var actor = Sys.ActorOf(DriverHostActor.Props(
|
||||
db, TestNode, coordinator.Ref,
|
||||
localRoles: new HashSet<string> { "driver" },
|
||||
deploymentArtifactCache: cache,
|
||||
fetchAndCacheMode: true,
|
||||
artifactFetcher: fetcher));
|
||||
|
||||
actor.Tell(new DispatchDeployment(deploymentId, RevA, CorrelationId.NewId()));
|
||||
coordinator.ExpectMsg<ApplyAck>(TimeSpan.FromSeconds(5)).Outcome.ShouldBe(ApplyAckOutcome.Applied);
|
||||
|
||||
// Re-dispatch the now-current revision: the _currentRevision guard short-circuits to an
|
||||
// immediate ACK without a second fetch.
|
||||
actor.Tell(new DispatchDeployment(deploymentId, RevA, CorrelationId.NewId()));
|
||||
coordinator.ExpectMsg<ApplyAck>(TimeSpan.FromSeconds(5)).Outcome.ShouldBe(ApplyAckOutcome.Applied);
|
||||
|
||||
Thread.Sleep(200);
|
||||
fetcher.Calls.ShouldBe(1);
|
||||
}
|
||||
|
||||
/// <summary>Records fetch calls and returns canned bytes (or null for "no answer").</summary>
|
||||
private sealed class RecordingFetcher(byte[]? bytes) : IDeploymentArtifactFetcher
|
||||
{
|
||||
private int _calls;
|
||||
public int Calls => Volatile.Read(ref _calls);
|
||||
|
||||
public Task<byte[]?> FetchAsync(string deploymentId, string expectedRevisionHash, CancellationToken ct)
|
||||
{
|
||||
Interlocked.Increment(ref _calls);
|
||||
return Task.FromResult(bytes);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Records every store; GetCurrent* return null (no idempotency shortcut in these tests).</summary>
|
||||
private sealed class RecordingArtifactCache : IDeploymentArtifactCache
|
||||
{
|
||||
private readonly Lock _gate = new();
|
||||
private readonly List<StoreCall> _stores = [];
|
||||
|
||||
public IReadOnlyList<StoreCall> Stores
|
||||
{
|
||||
get { lock (_gate) { return _stores.ToArray(); } }
|
||||
}
|
||||
|
||||
public Task StoreAsync(string clusterId, string deploymentId, string revisionHash,
|
||||
byte[] artifact, CancellationToken ct = default)
|
||||
{
|
||||
lock (_gate) { _stores.Add(new StoreCall(clusterId, deploymentId, revisionHash, artifact)); }
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
public Task<CachedDeploymentArtifact?> GetCurrentAsync(string clusterId, CancellationToken ct = default)
|
||||
=> Task.FromResult<CachedDeploymentArtifact?>(null);
|
||||
|
||||
public Task<CachedDeploymentArtifact?> GetCurrentUnkeyedAsync(CancellationToken ct = default)
|
||||
=> Task.FromResult<CachedDeploymentArtifact?>(null);
|
||||
|
||||
internal sealed record StoreCall(
|
||||
string ClusterId, string DeploymentId, string RevisionHash, byte[] Artifact);
|
||||
}
|
||||
}
|
||||
+156
@@ -0,0 +1,156 @@
|
||||
using System.Text.Json;
|
||||
using Akka.Actor;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Interfaces;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Fleet;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Types;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.OpcUa;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Drivers;
|
||||
|
||||
/// <summary>
|
||||
/// Per-cluster mesh Phase 3 (Task 7) — the #485 "empty/unreadable bytes = no answer, keep
|
||||
/// last-known-good, never tear down to an empty address space" contract, proven on every
|
||||
/// <c>FetchAndCache</c> seam. This adds no new mechanism; it is the regression net that pins the
|
||||
/// NEGATIVE — nothing rebuilds, nothing is torn down — on the failure paths.
|
||||
/// </summary>
|
||||
public sealed class DriverHostActorFetchAndCacheUnreadableTests : RuntimeActorTestBase
|
||||
{
|
||||
private static readonly NodeId TestNode = NodeId.Parse("driver-test");
|
||||
private static readonly RevisionHash RevA = RevisionHash.Parse(new string('a', 64));
|
||||
private static readonly RevisionHash RevB = RevisionHash.Parse(new string('b', 64));
|
||||
|
||||
private static byte[] Artifact() =>
|
||||
JsonSerializer.SerializeToUtf8Bytes(new { DriverInstances = Array.Empty<object>() });
|
||||
|
||||
[Fact]
|
||||
public void ANullFetchMidSteadyState_KeepsTheServedState_AndFiresNoRebuild()
|
||||
{
|
||||
// revA fetches good bytes and is served; a later revB fetch returns null (all endpoints down /
|
||||
// NotFound / SHA mismatch — all indistinguishable to the actor). The served address space must
|
||||
// NOT be rebuilt for revB and the revision must stay revA.
|
||||
var db = NewInMemoryDbFactory();
|
||||
var fetcher = new MapFetcher(new Dictionary<string, byte[]?>
|
||||
{
|
||||
[RevA.Value] = Artifact(),
|
||||
[RevB.Value] = null,
|
||||
});
|
||||
var publishProbe = CreateTestProbe();
|
||||
var coordinator = CreateTestProbe();
|
||||
|
||||
var actor = Sys.ActorOf(DriverHostActor.Props(
|
||||
db, TestNode, coordinator.Ref,
|
||||
localRoles: new HashSet<string> { "driver" },
|
||||
opcUaPublishActor: publishProbe.Ref,
|
||||
deploymentArtifactCache: new NullReadCache(),
|
||||
fetchAndCacheMode: true,
|
||||
artifactFetcher: fetcher));
|
||||
|
||||
actor.Tell(new DispatchDeployment(DeploymentId.NewId(), RevA, CorrelationId.NewId()));
|
||||
coordinator.ExpectMsg<ApplyAck>(TimeSpan.FromSeconds(5)).Outcome.ShouldBe(ApplyAckOutcome.Applied);
|
||||
publishProbe.ExpectMsg<OpcUaPublishActor.RebuildAddressSpace>(TimeSpan.FromSeconds(5)); // revA rebuild
|
||||
|
||||
actor.Tell(new DispatchDeployment(DeploymentId.NewId(), RevB, CorrelationId.NewId()));
|
||||
coordinator.ExpectMsg<ApplyAck>(TimeSpan.FromSeconds(5)).Outcome.ShouldBe(ApplyAckOutcome.Failed);
|
||||
|
||||
// The #485 negative: no rebuild for the failed revB — the served address space is untouched.
|
||||
publishProbe.ExpectNoMsg(TimeSpan.FromMilliseconds(500));
|
||||
AskDiagnostics(actor).CurrentRevision.ShouldBe(RevA);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void AZeroLengthFetch_FailsTheApply_ViaTheActorsOwn485Guard()
|
||||
{
|
||||
// The fetcher contract already turns zero bytes into null, but prove the actor ALSO refuses an
|
||||
// empty blob handed to it directly: ReconcileDriversFromBlob's #485 guard fails the apply.
|
||||
var db = NewInMemoryDbFactory();
|
||||
var fetcher = new MapFetcher(new Dictionary<string, byte[]?> { [RevA.Value] = Array.Empty<byte>() });
|
||||
var publishProbe = CreateTestProbe();
|
||||
var coordinator = CreateTestProbe();
|
||||
|
||||
var actor = Sys.ActorOf(DriverHostActor.Props(
|
||||
db, TestNode, coordinator.Ref,
|
||||
localRoles: new HashSet<string> { "driver" },
|
||||
opcUaPublishActor: publishProbe.Ref,
|
||||
deploymentArtifactCache: new NullReadCache(),
|
||||
fetchAndCacheMode: true,
|
||||
artifactFetcher: fetcher));
|
||||
|
||||
actor.Tell(new DispatchDeployment(DeploymentId.NewId(), RevA, CorrelationId.NewId()));
|
||||
|
||||
coordinator.ExpectMsg<ApplyAck>(TimeSpan.FromSeconds(5)).Outcome.ShouldBe(ApplyAckOutcome.Failed);
|
||||
publishProbe.ExpectNoMsg(TimeSpan.FromMilliseconds(500)); // nothing applied, nothing rebuilt
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ACorruptCacheAtBoot_DoesNotApplyAPartialConfig()
|
||||
{
|
||||
// A truncated/corrupt cached artifact surfaces as GetCurrentUnkeyedAsync == null (the cache's
|
||||
// integrity check is deliberately indistinguishable from a miss). Boot must land
|
||||
// Steady-no-revision and apply NOTHING — never a partial address space.
|
||||
var publishProbe = CreateTestProbe();
|
||||
|
||||
var actor = Sys.ActorOf(DriverHostActor.Props(
|
||||
new ThrowingDbFactory(), TestNode, ackRouter: null,
|
||||
localRoles: new HashSet<string> { "driver" },
|
||||
opcUaPublishActor: publishProbe.Ref,
|
||||
deploymentArtifactCache: new NullReadCache(), // corrupt → null, per the cache contract
|
||||
fetchAndCacheMode: true,
|
||||
artifactFetcher: new MapFetcher(new Dictionary<string, byte[]?>())));
|
||||
|
||||
// No rebuild at boot — a null (corrupt/empty) cache is not a configuration to apply.
|
||||
publishProbe.ExpectNoMsg(TimeSpan.FromMilliseconds(750));
|
||||
AskDiagnostics(actor).CurrentRevision.ShouldBeNull();
|
||||
}
|
||||
|
||||
private NodeDiagnosticsSnapshot AskDiagnostics(IActorRef actor)
|
||||
{
|
||||
var probe = CreateTestProbe();
|
||||
AwaitAssert(
|
||||
() =>
|
||||
{
|
||||
actor.Tell(new GetDiagnostics(CorrelationId.NewId()), probe.Ref);
|
||||
probe.ExpectMsg<NodeDiagnosticsSnapshot>(TimeSpan.FromSeconds(2));
|
||||
},
|
||||
duration: TimeSpan.FromSeconds(5));
|
||||
|
||||
actor.Tell(new GetDiagnostics(CorrelationId.NewId()), probe.Ref);
|
||||
return probe.ExpectMsg<NodeDiagnosticsSnapshot>(TimeSpan.FromSeconds(5));
|
||||
}
|
||||
|
||||
private sealed class MapFetcher(Dictionary<string, byte[]?> map) : IDeploymentArtifactFetcher
|
||||
{
|
||||
public Task<byte[]?> FetchAsync(string deploymentId, string expectedRevisionHash, CancellationToken ct)
|
||||
{
|
||||
map.TryGetValue(expectedRevisionHash, out var bytes);
|
||||
return Task.FromResult(bytes);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>A cache that reads empty (a miss, or the indistinguishable corrupt-artifact case).</summary>
|
||||
private sealed class NullReadCache : IDeploymentArtifactCache
|
||||
{
|
||||
public Task StoreAsync(string clusterId, string deploymentId, string revisionHash,
|
||||
byte[] artifact, CancellationToken ct = default)
|
||||
=> Task.CompletedTask;
|
||||
|
||||
public Task<CachedDeploymentArtifact?> GetCurrentAsync(string clusterId, CancellationToken ct = default)
|
||||
=> Task.FromResult<CachedDeploymentArtifact?>(null);
|
||||
|
||||
public Task<CachedDeploymentArtifact?> GetCurrentUnkeyedAsync(CancellationToken ct = default)
|
||||
=> Task.FromResult<CachedDeploymentArtifact?>(null);
|
||||
}
|
||||
|
||||
private sealed class ThrowingDbFactory : IDbContextFactory<OtOpcUaConfigDbContext>
|
||||
{
|
||||
public OtOpcUaConfigDbContext CreateDbContext()
|
||||
=> throw new InvalidOperationException("ConfigDb unreachable");
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user