docs(mesh): Phase 3 plan — config fetch-and-cache from central (gRPC + shared key)
Eleven tasks. User decisions baked in: gRPC fetch RPC (not token-gated HTTP), shared node bearer key (not a per-deployment token, so no migration and DispatchDeployment is unchanged), a ConfigSource:Mode dark switch (Direct default), both pair nodes fetch (no Primary gating). Header carries the five recon facts and the Phase 3/4 boundary (config READS only; the NodeDeploymentState write, DbHealthProbe and EfAlarmConditionStateStore stay for Phase 4). Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,473 @@
|
||||
# Per-Cluster Mesh Phase 3 — Config Fetch-and-Cache from Central
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** In a config-flagged `FetchAndCache` mode, a driver node obtains its deployed configuration by **fetching the artifact bytes from central over a gRPC stream**, caching them in LocalDb, and reading **all** config from that cache — so the driver never reads `Deployment.ArtifactBlob` from central SQL. Default `Direct` mode is byte-for-byte today's behaviour.
|
||||
|
||||
**Architecture:** Central (admin role) hosts a new gRPC service `DeploymentArtifactService.Fetch(deployment_id) → stream ArtifactChunk`, on a **dedicated h2c Kestrel listener** (the LocalDb-sync listener pattern), gated by a **shared bearer key** (fail-closed, `FixedTimeEquals`, path-scoped interceptor). A driver node in `FetchAndCache` mode, on `DispatchDeployment`, streams the bytes, verifies `SHA-256(bytes) == RevisionHash`, writes them through the existing `IDeploymentArtifactCache`, and applies from the cache; every config read (`ReconcileDrivers`, `PushDesiredSubscriptions`, address-space rebuild, `Bootstrap` recovery, Stale recovery) is redirected to the cache. The #485 "empty/unreadable bytes ⇒ no answer, keep last-known-good, fail the apply" guard is carried onto the fetch and cache-read paths.
|
||||
|
||||
**Tech Stack:** .NET 10, Akka.NET 1.5.62, `Grpc.AspNetCore` (server, already referenced) + **`Grpc.Tools` + a first in-repo `.proto`** (new — the repo has consumed packaged gRPC clients but never compiled a proto locally), `Grpc.Net.Client` (node client), the Phase-1 `ZB.MOM.WW.LocalDb` cache (`IDeploymentArtifactCache` / `LocalDbDeploymentArtifactCache`), xUnit + Shouldly + Akka.TestKit.
|
||||
|
||||
---
|
||||
|
||||
## Decisions already made (do not re-litigate)
|
||||
|
||||
| Decision | Choice | Consequence for this plan |
|
||||
|---|---|---|
|
||||
| Fetch transport | **gRPC fetch RPC** (user chose this over token-gated HTTP) | Introduces the repo's first `.proto` + `Grpc.Tools` codegen (Task 1); a new central-side h2c listener (Task 3); a generated client on the node (Task 4). |
|
||||
| Fetch auth | **Shared node key** (user chose this over a per-deployment token) | A single bearer secret (`ConfigServe:ApiKey` == `ConfigSource:ApiKey`), `FixedTimeEquals`, fail-closed. **No migration, no token lifecycle, and `DispatchDeployment` is UNCHANGED** — the node fetches by deployment id and authenticates with the shared key. |
|
||||
| Cutover | **Config-flagged dark switch** (`ConfigSource:Mode` = `Direct` default \| `FetchAndCache`), per node | Matches Phase 2. Rig comes up on `Direct`; the live gate flips only the **site** nodes to `FetchAndCache` (central keeps SQL, models the target). Rollback is a config change. |
|
||||
| Both nodes fetch | **Yes — no Primary gating of the fetch** | Today both pair nodes read central SQL independently (no gating); `FetchAndCache` keeps that shape. `IDeploymentArtifactCache.StoreAsync` is idempotent (`IsAlreadyCachedAsync`), and pair replication means a node that missed its fetch can still get the bytes from the peer. Primary-only fetch is a possible later optimization, explicitly out of scope. |
|
||||
|
||||
## Five facts the recon nailed down (read before Task 5)
|
||||
|
||||
1. **`RevisionHash == SHA-256(artifact bytes).`** `ConfigComposer.SnapshotAndFlattenAsync` (`src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/AdminOperations/ConfigComposer.cs:57-58`) computes the revision hash as SHA-256-hex-lowercase over the exact serialized blob — the **same** value the cache stores as `deployment_pointer.artifact_sha256`. So the node can verify a fetch against the `RevisionHash` already carried in `DispatchDeployment`; **no hash field is needed in the proto.**
|
||||
|
||||
2. **The driver reads central SQL on exactly these config paths, all reading `Deployment.ArtifactBlob` (never re-composing from raw rows):** `Bootstrap()` (`DriverHostActor.cs:584-603`, reads `NodeDeploymentStates` + `Deployments.RevisionHash`), `ReconcileDrivers` (`1717-1768`, reads at `1722-1726`), `PushDesiredSubscriptions` (`1880-1898`, reads at `1885-1889`), `TryRecoverFromStale` (`2363-2390`, reads at `2367-2373`), and the delegated `OpcUaPublishActor.HandleRebuild` → `LoadArtifact`/`LoadLatestArtifact` (`OpcUaPublishActor.cs:475-510`) which fires **only when `ApplyAndAck` passes no blob** (`DriverHostActor.cs:1667`). Phase 3 redirects these in `FetchAndCache` mode.
|
||||
|
||||
3. **The cache is already complete and correct.** `IDeploymentArtifactCache` / `LocalDbDeploymentArtifactCache` (`src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/`) does chunked base64 (128 KiB), SHA-256-verified reassembly, newest-2 retention, and replicates. It is written by `CacheAppliedArtifact` (`DriverHostActor.cs:1796-1835`) after a successful apply, and read only by `TryBootFromCache` (`671-714`) in the `Bootstrap()` SQL-unreachable catch. **Phase 3 changes WHEN it is written (before apply, from the fetch) and WHEN it is read (always, in `FetchAndCache`).** The `_isRunningFromCache` flag (field `85`, set at `698`, cleared at `1676`) already exists.
|
||||
|
||||
4. **`DispatchDeployment` is payload-free by design** (`DeploymentId` + `RevisionHash` + `CorrelationId`) so it never approaches the Akka frame limit. **Phase 3 keeps it payload-free** — the shared-key decision means no token rides in it, and the bytes travel out-of-band over gRPC. Do not add the artifact to any Akka message.
|
||||
|
||||
5. **The #485 guard is at every byte-parsing seam and must be carried onto the new paths:** `ReconcileDrivers` returns `null` on `blob.Length == 0` (`1735-1747`) → `ApplyAndAck` treats null as an apply **failure** (does not advance `_currentRevision`, writes `Failed`, sends a Failed ack — so a re-dispatch of the same revision actually retries, `1639-1659`); `CacheAppliedArtifact` skips empty (`1801-1808`); `PushDesiredSubscriptionsFromArtifact` skips empty (`1908-1920`); `OpcUaPublishActor.HandleRebuild` returns on `{ Length: 0 }` keeping the served address space (`372-391`); the cache's `ReassembleAsync` returns null on any chunk-count/base64/SHA/timestamp failure. **A zero-byte or SHA-mismatched fetch is "no answer," never "a config with no drivers."**
|
||||
|
||||
---
|
||||
|
||||
## The dark switch, precisely
|
||||
|
||||
`ConfigSourceOptions.Mode` selects the driver's config source:
|
||||
|
||||
- **`Direct`** (default): every path reads central SQL exactly as today. The new fetcher is not invoked. Byte-for-byte no behaviour change; the whole phase is inert.
|
||||
- **`FetchAndCache`**: on `DispatchDeployment`, fetch bytes from central over gRPC → verify SHA-256 == `RevisionHash` → `StoreAsync` into the cache → apply **from the fetched bytes in hand** (passed through to every consumer, so `OpcUaPublishActor` never reads SQL). `Bootstrap()` recovers the last-applied deployment from the LocalDb **pointer**, not `NodeDeploymentStates`. No driver-side `Deployment.ArtifactBlob` read occurs.
|
||||
|
||||
The central-side serve components (gRPC service, listener, interceptor) are wired **unconditionally on admin-role nodes** whenever `ConfigServe:GrpcListenPort > 0`, in both modes — mirroring Phase 2's "register in both modes so flipping the flag is not a redeploy." An idle listener costs nothing.
|
||||
|
||||
**Explicitly deferred to Phase 4 (do NOT touch here):** the `NodeDeploymentState` **write** (`UpsertNodeDeploymentState`, `DriverHostActor.cs:2392-2422`, still writes the ack row to central SQL), `DbHealthProbeActor`, `EfAlarmConditionStateStore`, and removing the driver-role ConfigDb connection string. Phase 3 removes the config-blob **reads** only. A `FetchAndCache` node still writes its ack row to SQL; that is Phase 4's cut.
|
||||
|
||||
---
|
||||
|
||||
## Task 0: `ConfigSourceOptions` + `ConfigServeOptions` + validators + registration
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** Task 1
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ConfigSourceOptions.cs`
|
||||
- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ConfigSourceOptionsValidator.cs`
|
||||
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs` (register both in `AddOtOpcUaCluster`, beside the `MeshTransportOptions` registration)
|
||||
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/ConfigSourceOptionsValidatorTests.cs`
|
||||
|
||||
**Design.** One options class holds both the node (fetch) and central (serve) surfaces, keyed `ConfigSource` and `ConfigServe`. Model after `MeshTransportOptions` / `MeshTransportOptionsValidator` (same directory) exactly — the validator fails host start on a `FetchAndCache` shape that would leave the node unable to fetch (empty endpoints, empty key), because every such fault otherwise surfaces as a silent absence (a deploy that never applies).
|
||||
|
||||
```csharp
|
||||
// ConfigSourceOptions.cs
|
||||
namespace ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
|
||||
/// <summary>Node-side config source selection (per-cluster mesh Phase 3).</summary>
|
||||
public sealed class ConfigSourceOptions
|
||||
{
|
||||
public const string SectionName = "ConfigSource";
|
||||
public const string ModeDirect = "Direct";
|
||||
public const string ModeFetchAndCache = "FetchAndCache";
|
||||
|
||||
/// <summary><see cref="ModeDirect"/> (read central SQL, today's behaviour) or
|
||||
/// <see cref="ModeFetchAndCache"/> (fetch from central over gRPC, read LocalDb).</summary>
|
||||
public string Mode { get; set; } = ModeDirect;
|
||||
|
||||
/// <summary>Central artifact-gRPC base addresses, e.g. <c>http://central-1:4055</c>. h2c (http
|
||||
/// scheme). Required and tried in order (failover) under <see cref="ModeFetchAndCache"/>.</summary>
|
||||
public string[] CentralFetchEndpoints { get; set; } = [];
|
||||
|
||||
/// <summary>Shared bearer key; must equal central's <see cref="ConfigServeOptions.ApiKey"/>.
|
||||
/// Supply via env <c>ConfigSource__ApiKey</c>; never commit. Required under FetchAndCache.</summary>
|
||||
public string ApiKey { get; set; } = string.Empty;
|
||||
|
||||
/// <summary>Per-fetch deadline. Non-positive rejected under FetchAndCache.</summary>
|
||||
public int FetchTimeoutSeconds { get; set; } = 30;
|
||||
}
|
||||
|
||||
/// <summary>Central-side artifact-serve surface (per-cluster mesh Phase 3).</summary>
|
||||
public sealed class ConfigServeOptions
|
||||
{
|
||||
public const string SectionName = "ConfigServe";
|
||||
|
||||
/// <summary>Dedicated h2c listener port for the artifact gRPC service. <c>0</c> = disabled
|
||||
/// (nothing bound). Must differ from the main HTTP port and from LocalDb:SyncListenPort.</summary>
|
||||
public int GrpcListenPort { get; set; }
|
||||
|
||||
/// <summary>Shared bearer key the interceptor checks (FixedTimeEquals, fail-closed). Supply via
|
||||
/// env <c>ConfigServe__ApiKey</c>; never commit.</summary>
|
||||
public string ApiKey { get; set; } = string.Empty;
|
||||
}
|
||||
```
|
||||
|
||||
The validator (mirror `MeshTransportOptionsValidator`): reject an unknown `Mode`; under `FetchAndCache` reject empty `CentralFetchEndpoints`, any endpoint not starting `http://` or `https://`, empty `ApiKey`, and non-positive `FetchTimeoutSeconds`. `ConfigServeOptions` needs no cross-field validation (a `0` port is a legitimate "disabled").
|
||||
|
||||
**Step 1: Write the failing test** — `ConfigSourceOptionsValidatorTests` with cases: default (`Direct`, empty everything) → Success; unknown mode → Fail; `FetchAndCache` + empty endpoints → Fail; + non-`http(s)` endpoint → Fail; + empty key → Fail; + `FetchTimeoutSeconds = 0` → Fail; fully-populated `FetchAndCache` → Success. (No implicit usings in Cluster.Tests — `using Xunit;`.)
|
||||
|
||||
**Step 2:** `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests --filter "FullyQualifiedName~ConfigSourceOptionsValidatorTests"` → FAIL (types missing).
|
||||
|
||||
**Step 3:** Write the two options classes + the validator; register in `AddOtOpcUaCluster`:
|
||||
```csharp
|
||||
services.AddValidatedOptions<ConfigSourceOptions, ConfigSourceOptionsValidator>(
|
||||
configuration, ConfigSourceOptions.SectionName);
|
||||
services.Configure<ConfigServeOptions>(configuration.GetSection(ConfigServeOptions.SectionName));
|
||||
```
|
||||
|
||||
**Step 4:** Re-run → PASS. **Sabotage-check:** flip one validator branch (e.g. accept empty endpoints) → its test reddens.
|
||||
|
||||
**Step 5: Commit** `feat(mesh): ConfigSource/ConfigServe options + validator (Phase 3 dark switch)`
|
||||
|
||||
---
|
||||
|
||||
## Task 1: `deployment_artifact.v1` proto + Grpc.Tools codegen
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 0
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos/deployment_artifact.proto`
|
||||
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Commons/ZB.MOM.WW.OtOpcUa.Commons.csproj` (add `Grpc.Tools` + `Google.Protobuf` + `Grpc.Core.Api` package refs and a `<Protobuf>` item; `GrpcServices="Both"` so both server base + client stub generate)
|
||||
- Modify: `Directory.Packages.props` (pin `Grpc.Tools`, `Google.Protobuf`, `Grpc.Core.Api` if not already pinned — check first)
|
||||
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests/DeploymentArtifactProtoTests.cs` (a compile-touch test that references the generated `DeploymentArtifactService.DeploymentArtifactServiceBase` type and `FetchRequest`, proving codegen ran)
|
||||
|
||||
**Why Commons.** Both the central server (in Host/AdminUI) and the node client (in Runtime) need the generated types, and Commons is already referenced by every server project. This is the repo's **first** locally-compiled proto — there is no existing `<Protobuf>` item to copy, so wire it from scratch.
|
||||
|
||||
```proto
|
||||
syntax = "proto3";
|
||||
package deployment_artifact.v1;
|
||||
option csharp_namespace = "ZB.MOM.WW.OtOpcUa.Commons.Protos.DeploymentArtifact.V1";
|
||||
|
||||
// Central serves the deployed-configuration artifact bytes to a driver node (Phase 3).
|
||||
service DeploymentArtifactService {
|
||||
// Streams the artifact for a sealed deployment as ordered chunks. NotFound if the id is
|
||||
// unknown or not sealed. The client verifies SHA-256(reassembled) == the RevisionHash it
|
||||
// already holds, so no hash travels here.
|
||||
rpc Fetch(FetchRequest) returns (stream ArtifactChunk);
|
||||
}
|
||||
message FetchRequest { string deployment_id = 1; }
|
||||
message ArtifactChunk { bytes data = 1; }
|
||||
```
|
||||
|
||||
`.csproj` addition (mirror the packaging note the LocalDb refs use):
|
||||
```xml
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Grpc.Tools" PrivateAssets="all" />
|
||||
<PackageReference Include="Google.Protobuf" />
|
||||
<PackageReference Include="Grpc.Core.Api" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<Protobuf Include="Protos\deployment_artifact.proto" GrpcServices="Both" />
|
||||
</ItemGroup>
|
||||
```
|
||||
|
||||
**Step 1:** Write the compile-touch test referencing `DeploymentArtifactService.DeploymentArtifactServiceBase` and `new FetchRequest { DeploymentId = "x" }`.
|
||||
**Step 2:** `dotnet build src/Core/ZB.MOM.WW.OtOpcUa.Commons` → FAIL (no proto/codegen).
|
||||
**Step 3:** Add the packages, the proto, the `<Protobuf>` item. `dotnet restore` then `dotnet build src/Core/ZB.MOM.WW.OtOpcUa.Commons`.
|
||||
**Step 4:** `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests --filter "FullyQualifiedName~DeploymentArtifactProtoTests"` → PASS. Confirm generated types resolve (`obj/.../DeploymentArtifactGrpc.cs`).
|
||||
**Step 5: Commit** `feat(mesh): deployment_artifact.v1 proto + first in-repo Grpc.Tools codegen`
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Central-side `DeploymentArtifactService` (streams the blob, existence-hiding)
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 4
|
||||
**Blocked by:** Task 1
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Grpc/DeploymentArtifactService.cs`
|
||||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/Grpc/DeploymentArtifactServiceTests.cs`
|
||||
|
||||
**Design.** Subclass the generated `DeploymentArtifactServiceBase`. `Fetch` reads the `Deployment` row by id from `OtOpcUaConfigDbContext` (admin node has the connection); if the row is missing OR `Status != Sealed` OR `ArtifactBlob.Length == 0`, throw `RpcException(new Status(StatusCode.NotFound, "…"))` (existence-hiding + #485: never stream zero bytes as if valid). Otherwise stream the blob in ≤128 KiB `ArtifactChunk`s via `responseStream.WriteAsync`. Lives in AdminUI (where the other admin-only endpoints + the config DB context live).
|
||||
|
||||
```csharp
|
||||
public sealed class DeploymentArtifactService : DeploymentArtifactServiceBase
|
||||
{
|
||||
private const int ChunkSize = 128 * 1024; // matches LocalDbDeploymentArtifactCache.ChunkSize
|
||||
private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbFactory;
|
||||
private readonly ILogger<DeploymentArtifactService> _log;
|
||||
// ctor injects both
|
||||
|
||||
public override async Task Fetch(
|
||||
FetchRequest request, IServerStreamWriter<ArtifactChunk> responseStream, ServerCallContext context)
|
||||
{
|
||||
if (!Guid.TryParse(request.DeploymentId, out var id))
|
||||
throw new RpcException(new Status(StatusCode.NotFound, "unknown deployment"));
|
||||
|
||||
await using var db = await _dbFactory.CreateDbContextAsync(context.CancellationToken);
|
||||
var row = await db.Deployments.AsNoTracking()
|
||||
.Where(d => d.DeploymentId == id)
|
||||
.Select(d => new { d.Status, d.ArtifactBlob })
|
||||
.FirstOrDefaultAsync(context.CancellationToken);
|
||||
|
||||
// Existence-hiding + #485: unknown / not-sealed / empty are one indistinguishable NotFound.
|
||||
if (row is null || row.Status != DeploymentStatus.Sealed || row.ArtifactBlob.Length == 0)
|
||||
throw new RpcException(new Status(StatusCode.NotFound, "unknown deployment"));
|
||||
|
||||
for (var offset = 0; offset < row.ArtifactBlob.Length; offset += ChunkSize)
|
||||
{
|
||||
var len = Math.Min(ChunkSize, row.ArtifactBlob.Length - offset);
|
||||
await responseStream.WriteAsync(
|
||||
new ArtifactChunk { Data = ByteString.CopyFrom(row.ArtifactBlob, offset, len) },
|
||||
context.CancellationToken);
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
(Confirm the sealed-status enum member name from `DeploymentStatus`; the recon showed `Deployment.Status` and the deploy sets `Sealed`.)
|
||||
|
||||
**Step 1: Write the failing test** — an in-memory `OtOpcUaConfigDbContext` (the AdminUI.Tests pattern) seeded with one sealed deployment; a fake `IServerStreamWriter<ArtifactChunk>` collecting writes; assert: (a) a sealed non-empty blob streams back and reassembles byte-equal, chunk-bounded at 128 KiB (seed a ~300 KB blob → 3 chunks); (b) an unknown id throws `RpcException` NotFound; (c) a non-sealed row throws NotFound; (d) a zero-length blob throws NotFound (the #485 serve-side guard).
|
||||
**Step 2:** run → FAIL.
|
||||
**Step 3:** implement.
|
||||
**Step 4:** run → PASS. **Sabotage:** drop the `row.Status != Sealed` check → the non-sealed test reddens; drop the `ArtifactBlob.Length == 0` check → the zero-length test reddens.
|
||||
**Step 5: Commit** `feat(mesh): central DeploymentArtifactService — stream the sealed artifact, NotFound-hide the rest`
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Central serve auth interceptor + dedicated h2c listener wiring
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** none
|
||||
**Blocked by:** Task 0, Task 2
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/ConfigServeAuthInterceptor.cs`
|
||||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` (register the interceptor with `AddGrpc`; add the dedicated h2c listener block mirroring the LocalDb sync block at `407-476`; `MapGrpcService<DeploymentArtifactService>()` inside a `hasAdmin && configServeGrpcPort > 0` guard beside `514`)
|
||||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/ConfigServeAuthInterceptorTests.cs`
|
||||
|
||||
**Interceptor** — mirror `LocalDbSyncAuthInterceptor` exactly, scoped by service path `"/deployment_artifact.v1.DeploymentArtifactService/"`, fail-closed (no `ConfigServe:ApiKey` ⇒ reject ALL with `Unauthenticated`), `Authorization: Bearer <key>`, `FixedTimeEquals`. Calls to other services pass through untouched (so it can share `AddGrpc` with the LocalDb sync interceptor).
|
||||
|
||||
**Listener** — the artifact service is h2c and cannot share the cleartext HTTP/1 port (same reason the LocalDb sync listener is dedicated). Add, right after the LocalDb sync listener block:
|
||||
```csharp
|
||||
var configServeGrpcPort = builder.Configuration.GetValue<int>("ConfigServe:GrpcListenPort");
|
||||
if (hasAdmin && configServeGrpcPort > 0)
|
||||
{
|
||||
// Same dedicated-h2c-listener dance as the LocalDb sync listener above: h2c can't negotiate on a
|
||||
// cleartext Http1AndHttp2 port, so bind a dedicated HTTP/2-only port and re-apply existing bindings.
|
||||
// (Refactor the LocalDb block's existing-binding computation into a shared local if both run — a
|
||||
// node that is admin+driver with BOTH ports set must re-apply existing bindings once and add both.)
|
||||
builder.WebHost.ConfigureKestrel(kestrel =>
|
||||
kestrel.ListenAnyIP(configServeGrpcPort, o => o.Protocols = HttpProtocols.Http2));
|
||||
Log.Information("Config-serve artifact gRPC listener bound on :{Port} (h2c).", configServeGrpcPort);
|
||||
}
|
||||
```
|
||||
> **Load-bearing interaction with the LocalDb listener.** A fused central node is admin **and** driver; on the rig it may have `LocalDb:SyncListenPort` set too. `ConfigureKestrel` is additive across calls, but the LocalDb block re-applies `existingBindings` (URLS/HTTP_PORTS) and this block must NOT clobber that. Implement by computing `existingBindings` once and applying it once, then adding whichever of the two dedicated h2c ports are configured. The test below asserts the AdminUI HTTP port still answers when both dedicated ports are set.
|
||||
|
||||
`MapGrpcService`:
|
||||
```csharp
|
||||
if (hasAdmin && configServeGrpcPort > 0)
|
||||
app.MapGrpcService<DeploymentArtifactService>();
|
||||
```
|
||||
|
||||
**Step 1: Write the failing test** — boot the Host (or a minimal `WebApplicationFactory` with `hasAdmin`, `ConfigServe:GrpcListenPort` set, `ConfigServe:ApiKey=k`, a seeded sealed deployment): a `Grpc.Net.Client` channel to the dedicated port with `Authorization: Bearer k` fetches + reassembles the blob; **no header** ⇒ `Unauthenticated`; **wrong key** ⇒ `Unauthenticated`; and the AdminUI HTTP surface (`/health/active`) still returns 200 (listener coexistence). Use `TestContext.Current.CancellationToken`.
|
||||
**Step 2:** run → FAIL.
|
||||
**Step 3:** implement interceptor + Program.cs wiring.
|
||||
**Step 4:** run → PASS. **Sabotage:** make the interceptor return without checking when the key is unset (i.e. break fail-closed) → the no-header test reddens; break the listener coexistence (apply only this block's binding) → the `/health/active` assertion reddens.
|
||||
**Step 5: Commit** `feat(mesh): config-serve auth interceptor + dedicated h2c listener (fail-closed bearer)`
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Node-side `IDeploymentArtifactFetcher` + gRPC client (SHA-verified, failover)
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 2
|
||||
**Blocked by:** Task 1
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/IDeploymentArtifactFetcher.cs`
|
||||
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/GrpcDeploymentArtifactFetcher.cs`
|
||||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Deployment/GrpcDeploymentArtifactFetcherTests.cs`
|
||||
|
||||
**Design.**
|
||||
```csharp
|
||||
public interface IDeploymentArtifactFetcher
|
||||
{
|
||||
/// <summary>Fetches + reassembles the artifact for <paramref name="deploymentId"/>, verifying
|
||||
/// SHA-256(bytes) == <paramref name="expectedRevisionHash"/>. Returns the bytes, or <see
|
||||
/// langword="null"/> on any failure (all endpoints unreachable, NotFound, SHA mismatch,
|
||||
/// zero-length) — a null is "no answer," NEVER an empty config (#485).</summary>
|
||||
Task<byte[]?> FetchAsync(string deploymentId, string expectedRevisionHash, CancellationToken ct);
|
||||
}
|
||||
```
|
||||
`GrpcDeploymentArtifactFetcher` takes `IOptions<ConfigSourceOptions>`. For each endpoint in order: open a `GrpcChannel` (h2c: `new HttpClientHandler` / `GrpcChannelOptions` with the `http://` address), attach `Authorization: Bearer {ApiKey}` via a `CallCredentials` or a request header on the call, stream `Fetch`, accumulate `chunk.Data` into a buffer, then `SHA-256(buffer)` hex-lowercase and compare to `expectedRevisionHash`. On success return the bytes. On `RpcException` (NotFound / Unavailable / deadline), log and try the next endpoint. If all fail, or the reassembled buffer is empty, or the SHA mismatches, return `null` (and log a Warning naming the reason). Deadline = `FetchTimeoutSeconds`.
|
||||
|
||||
> **Why return null, not throw.** The caller (Task 5) treats null identically to `ReconcileDrivers` returning null today: apply failure, keep last-known-good, do not advance the revision. Throwing would risk an unhandled actor message; a typed "no bytes" keeps the #485 contract explicit.
|
||||
|
||||
**Seam for the test.** So the test needs no real second Host, make the gRPC call site injectable: a `Func<string /*endpoint*/, DeploymentArtifactService.DeploymentArtifactServiceClient>` factory defaulted to the real channel builder, overridable in tests with an in-memory client backed by a fake that streams canned chunks. (This mirrors Phase 2's `IMeshClusterClientFactory` seam.)
|
||||
|
||||
**Step 1: Write the failing test** — with a fake client factory: (a) a client streaming chunks whose reassembly hashes to `expectedRevisionHash` → returns the exact bytes; (b) chunks whose hash ≠ expected → returns null; (c) an empty stream → returns null; (d) first endpoint throws `RpcException(Unavailable)`, second streams good bytes → returns the bytes (failover); (e) all endpoints throw → null. Assert the failover case actually consulted endpoint 2 (record calls).
|
||||
**Step 2:** run → FAIL.
|
||||
**Step 3:** implement.
|
||||
**Step 4:** run → PASS. **Sabotage:** drop the SHA-256 comparison → the mismatch test reddens; break failover (return after the first endpoint's exception) → the failover test reddens.
|
||||
**Step 5: Commit** `feat(mesh): node gRPC artifact fetcher — SHA-verified reassembly, endpoint failover, null-on-any-failure`
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Wire fetch-and-cache into the apply path (FetchAndCache mode)
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** none
|
||||
**Blocked by:** Task 4
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs` (add `_configSourceMode` + `_artifactFetcher` fields + ctor params + both `Props` overloads; a new `FetchThenApply` path used from `HandleDispatchFromSteady` / `ApplyAndAck` when `FetchAndCache`)
|
||||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs` (resolve `IOptions<ConfigSourceOptions>` + register `IDeploymentArtifactFetcher` in the `hasDriver` branch; pass both into `DriverHostActor.Props`)
|
||||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/...` (DI: `services.AddSingleton<IDeploymentArtifactFetcher, GrpcDeploymentArtifactFetcher>()` in the driver branch — place beside the `IDeploymentArtifactCache` registration)
|
||||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostActorFetchAndCacheTests.cs`
|
||||
|
||||
**Design.** In `FetchAndCache` mode, on a `DispatchDeployment` whose revision differs from `_currentRevision`:
|
||||
1. `bytes = await _artifactFetcher.FetchAsync(deploymentId, revisionHash, ct)` (message-loop-safe: run via `PipeTo` to self as a `FetchResult(deploymentId, revisionHash, correlation, bytes?)`, NOT a blocking await inside the receive).
|
||||
2. On `bytes == null` → **apply failure**: do not advance `_currentRevision`, `UpsertNodeDeploymentState(Failed)`, `SendAck(Failed)` — identical to today's `ReconcileDrivers`-returned-null path (`1639-1659`). The node keeps serving last-known-good. A re-dispatch retries (revision still not current).
|
||||
3. On success → `StoreAsync(clusterId, deploymentId, revisionHash, bytes)` into the cache, then apply **from `bytes` in hand**: `ReconcileDriversFromBlob(bytes)`, `RebuildAddressSpace(correlation, deploymentId, bytes)` (pass the blob so `OpcUaPublishActor` never reads SQL), `PushDesiredSubscriptionsFromArtifact(bytes)`, `UpsertNodeDeploymentState(Applied)`, `SendAck(Applied)`, `_isRunningFromCache = false`, set `_currentRevision`.
|
||||
|
||||
**Refactor note — do NOT duplicate reconcile logic.** `ReconcileDrivers(DeploymentId)` today reads the blob itself (`1722-1726`). Extract the post-read body into `ReconcileDriversFromBlob(byte[] blob)` (the existing #485 `blob.Length == 0` guard at `1735-1747` moves into it verbatim), and have the `Direct`-mode `ReconcileDrivers` read-then-call it. Then `FetchThenApply` calls `ReconcileDriversFromBlob(bytes)` directly. Same for `PushDesiredSubscriptions` → it already has `PushDesiredSubscriptionsFromArtifact(blob)` (`1908`); reuse it.
|
||||
|
||||
**Idempotency:** before fetching, check the cache — if `IsAlreadyCached(deploymentId, revisionHash)` (add a cheap `GetCurrentAsync` compare, or a new `bool ContainsAsync`), skip the fetch and apply from cache. This makes a re-dispatch of an already-applied revision a no-op fetch (and covers "peer already replicated it to us").
|
||||
|
||||
**Step 1: Write the failing test** — a `DriverHostActor` in `FetchAndCache` mode with a fake `IDeploymentArtifactFetcher` + a real in-memory `LocalDbDeploymentArtifactCache` (the `DriverHostActorArtifactCacheTests` harness): (a) dispatch → fetcher returns good bytes → node applies, cache written, ack Applied, `_currentRevision` advances, and it **never touched the ConfigDb factory** (inject a throwing `IDbContextFactory` to prove no SQL read on the config path — note the `NodeDeploymentState` write is Phase 4, so use a factory that permits the ack write but fails `Deployments` reads, or assert via a spy that `Deployments` was never queried); (b) fetcher returns null → ack Failed, `_currentRevision` unchanged, children kept; (c) re-dispatch of the applied revision → no fetch (idempotent), immediate Applied ack.
|
||||
**Step 2:** run → FAIL.
|
||||
**Step 3:** implement.
|
||||
**Step 4:** run → PASS. **Sabotage:** make the null-bytes branch advance `_currentRevision` anyway → test (b) reddens (this is the #485 apply-failure contract); remove the idempotency check → test (c) reddens.
|
||||
**Step 5: Commit** `feat(mesh): FetchAndCache apply path — fetch→cache→apply-from-bytes, null=apply-failure`
|
||||
|
||||
---
|
||||
|
||||
## Task 6: Redirect all config reads to LocalDb; bootstrap from the pointer
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** none
|
||||
**Blocked by:** Task 5
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs` (`Bootstrap`, `TryRecoverFromStale`, and any residual read seams under `FetchAndCache`)
|
||||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostActorFetchAndCacheBootstrapTests.cs`
|
||||
|
||||
**Design.** In `FetchAndCache` mode:
|
||||
- **`Bootstrap()`** does NOT read `NodeDeploymentStates`/`Deployments` from SQL. Instead: `cached = await _cache.GetCurrentUnkeyedAsync()`. If non-null → `_currentRevision = RevisionHash.Parse(cached.RevisionHash)`, `Become(Steady)`, `ApplyCachedArtifact(new DeploymentId(cached.DeploymentId), cached.Artifact)` (the existing cache-boot path at `725-748`, which reconciles + rebuilds + resubscribes from the blob and does **not** read SQL or re-ack). If null (fresh node, empty cache) → `Become(Steady)` with no revision; the first `DispatchDeployment` triggers a fetch. On a cache-read exception → log and `Become(Steady)` (no revision) rather than `Stale` (Stale meant "SQL down"; in FetchAndCache there is no SQL config read to be down).
|
||||
- **`TryRecoverFromStale()`** is unreachable in `FetchAndCache` (nothing enters Stale from a config read). Leave it for `Direct` mode; guard its entry so `FetchAndCache` never schedules it.
|
||||
- **`OpcUaPublishActor`** is untouched: `FetchThenApply` and `ApplyCachedArtifact` both pass the blob to `RebuildAddressSpace`, so `LoadArtifact`/`LoadLatestArtifact` (the SQL reads) are never hit in `FetchAndCache`. Add an assertion-comment; no code change there.
|
||||
|
||||
**The one SQL touch that remains (by design, Phase 4's to cut):** `UpsertNodeDeploymentState` still writes the ack row. That is a write, not a config read, and the phase boundary is explicit.
|
||||
|
||||
**Step 1: Write the failing test** — `FetchAndCache` mode, throwing `IDbContextFactory` for `Deployments` reads: (a) a cache pre-seeded with an applied artifact → `Bootstrap` restores served state (drivers reconciled, address space rebuilt) with **no** `Deployments` read; (b) an empty cache → `Bootstrap` lands in Steady-no-revision without error and a subsequent dispatch fetches; (c) a cache-read failure → Steady-no-revision, not Stale. Reuse the `DriverHostActorBootFromCacheTests` harness.
|
||||
**Step 2:** run → FAIL.
|
||||
**Step 3:** implement the mode-gated bootstrap branch.
|
||||
**Step 4:** run → PASS. **Sabotage:** in `FetchAndCache`, make `Bootstrap` fall through to the SQL read → test (a)'s throwing-factory makes it red.
|
||||
**Step 5: Commit** `feat(mesh): FetchAndCache bootstrap from the LocalDb pointer; no driver-side config SQL read`
|
||||
|
||||
---
|
||||
|
||||
## Task 7: #485 guard coverage on the new fetch + cache-read paths
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** none
|
||||
**Blocked by:** Task 6
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs` (only if a seam is found unguarded)
|
||||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostActorFetchAndCacheUnreadableTests.cs`
|
||||
|
||||
This task adds **no new mechanism** — it proves the #485 contract holds on every `FetchAndCache` seam, and adds a guard only where a test exposes a gap. The contract: zero-length / SHA-mismatched / unreadable ⇒ no answer, keep last-known-good, fail the apply, never tear down to an empty address space.
|
||||
|
||||
**Step 1: Write the failing tests** (expect some to pass already — that is fine; they are the regression net):
|
||||
- Fetcher returns `null` (all endpoints down) mid-steady-state → address space + drivers + subscriptions UNCHANGED, ack Failed, `_currentRevision` unchanged. (Covered by Task 5(b); re-assert the address-space-kept angle explicitly.)
|
||||
- Central serves a **zero-length** blob (shouldn't happen — Task 2 NotFound-hides it — but prove the client also refuses): fetcher gets an empty stream → `null` → apply failure.
|
||||
- Cache holds a **truncated/corrupt** artifact at boot (drop a chunk row / corrupt base64) → `ReassembleAsync` returns null → `Bootstrap` treats it as empty cache (Steady-no-revision), does NOT apply a partial config.
|
||||
- SHA mismatch (server bytes don't match the dispatched `RevisionHash`) → fetcher `null` → apply failure, last-known-good kept.
|
||||
**Step 2:** run → the corrupt-cache and any unguarded case FAIL if a gap exists; otherwise all green (regression net).
|
||||
**Step 3:** add a guard only where a test is red.
|
||||
**Step 4:** run → PASS. **Sabotage:** temporarily make `ReassembleAsync` return partial bytes on a missing chunk → the corrupt-cache test reddens (proving the guard, not the mock, is what holds the line).
|
||||
**Step 5: Commit** `test(mesh): #485 empty/corrupt/mismatch coverage on the FetchAndCache paths`
|
||||
|
||||
---
|
||||
|
||||
## Task 8: Real two-Host gRPC fetch boundary test + falsifiability control
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 9
|
||||
**Blocked by:** Task 3, Task 4
|
||||
|
||||
**Files:**
|
||||
- Create: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/DeploymentArtifactFetchBoundaryTests.cs`
|
||||
|
||||
The Phase-2 lesson applied to Phase 3: an in-memory fake client proves fetcher **logic**, not that a real gRPC stream crosses a real Kestrel h2c listener through the real interceptor. Stand a real central Host (admin, `ConfigServe:GrpcListenPort` set, `ConfigServe:ApiKey=k`, a seeded sealed deployment) and drive the real `GrpcDeploymentArtifactFetcher` against it.
|
||||
|
||||
Assertions:
|
||||
1. A fetch with the right key + a >256 KB blob reassembles byte-equal and verifies against the deployment's `RevisionHash`.
|
||||
2. **Falsifiability control** — the same fetch with a **wrong key** returns `null` (interceptor rejects; without this control, assertion 1 cannot distinguish "the boundary works" from "something else supplied the bytes"). Verify the control is real by confirming the right-key fetch on the *same* server succeeds.
|
||||
3. An unknown deployment id → `null` (NotFound-hidden).
|
||||
4. Endpoint failover: give the fetcher `[dead-port, real-port]` → it still returns the bytes.
|
||||
|
||||
If irreducibly flaky, quarantine with `[Trait("Category","ArtifactBoundary")]` — do **not** weaken to the in-memory fake (that is Task 4's job).
|
||||
|
||||
**Step 5: Commit** `test(mesh): real two-Host gRPC artifact-fetch boundary test with wrong-key control`
|
||||
|
||||
---
|
||||
|
||||
## Task 9: Rig config + docs
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 8
|
||||
**Blocked by:** Task 5
|
||||
|
||||
**Files:**
|
||||
- Modify: `docker-dev/docker-compose.yml` — central-1/central-2 get `ConfigServe__GrpcListenPort` (e.g. `4055`) + `ConfigServe__ApiKey` (dev value, the committed-dev-secret exception); all six get `ConfigSource__Mode: "${OTOPCUA_CONFIG_MODE:-Direct}"`; site nodes get `ConfigSource__CentralFetchEndpoints__0/1: http://central-1:4055 / http://central-2:4055` + a matching `ConfigSource__ApiKey`. Leave `Mode=Direct` so the rig comes up unchanged; expose the flip via `OTOPCUA_CONFIG_MODE=FetchAndCache`.
|
||||
- Modify: `docs/Configuration.md` — new `ConfigSource` + `ConfigServe` sections (the dark switch, the shared-key auth, the h2c dedicated-port requirement, the env-only key rule).
|
||||
- Modify: `docs/Redundancy.md` — extend the "Command transport" section: config bytes travel out-of-band over the artifact gRPC stream, both pair nodes fetch independently (idempotent + replicated), central serves from SQL.
|
||||
- Modify: `docs/plans/2026-07-22-per-cluster-mesh-program.md` — flip the Phase 3 row + section to reflect the gRPC/shared-key choices.
|
||||
- Modify: `CLAUDE.md` — a short "Config source (Phase 3)" note beside the Mesh command transport note.
|
||||
|
||||
**Step 5: Commit** `docs(mesh): ConfigSource/ConfigServe config, rig flip switch, Phase 3 as shipped`
|
||||
|
||||
---
|
||||
|
||||
## Task 10: Live gate on docker-dev
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min (plus rig time)
|
||||
**Parallelizable with:** none
|
||||
**Blocked by:** Task 8, Task 9
|
||||
|
||||
**Files:**
|
||||
- Create: `docs/plans/2026-07-22-mesh-phase3-live-gate.md`
|
||||
|
||||
Flip only the **site** nodes to `FetchAndCache` (`OTOPCUA_CONFIG_MODE=FetchAndCache` scoped to site services, or a compose override) so central keeps SQL and the rig models the target topology (site nodes have no SQL). Central stays `Direct`. Rig AdminUI login is disabled — drive UI steps via browser automation at `http://localhost:9200`; deploy via `POST /api/deployments` with `X-Api-Key: docker-dev-deploy-key`.
|
||||
|
||||
| # | Step | Pass condition |
|
||||
|---|---|---|
|
||||
| 1 | Rig up all-`Direct`; deploy | Seals green — baseline |
|
||||
| 2 | Flip site nodes to `FetchAndCache`, restart them, deploy | Seals green; **site node logs show a gRPC fetch + cache write + apply-from-bytes**, and **no `Deployments.ArtifactBlob` SQL read on the site side** |
|
||||
| 3 | Confirm central served it | central logs the `DeploymentArtifactService.Fetch` call; site logs the SHA-256 verify pass |
|
||||
| 4 | **Stop central SQL, deploy** (the phase's headline gate) | The site fetch fails (central can't read SQL to serve) → site logs apply-failure, **keeps serving last-known-good**, ack Failed → deploy TimedOut naming the site nodes; **no crash, no empty address space** (#485) |
|
||||
| 5 | Restart SQL, redeploy | Fetch lands, seals green — retry semantics on the new path |
|
||||
| 6 | Restart a site node with a warm cache, central up | Boots from the LocalDb pointer, restores served state, **no `Deployments` read** |
|
||||
| 7 | Restart a site node with central DOWN | Boots last-known-good from cache; no crash loop (config path needs no central at boot) |
|
||||
| 8 | Corrupt one cached artifact chunk on a site node, restart it | Reassembly rejects it → boots Steady-no-revision (empty-cache path), does NOT apply a partial config; next deploy re-fetches |
|
||||
| 9 | Wrong `ConfigSource__ApiKey` on one site node, deploy | That node's fetch is rejected (Unauthenticated) → apply-failure + Warning; the other site node (right key) applies |
|
||||
| 10 | Both site nodes fetch the same deploy | Idempotent — the second store is a no-op (`IsAlreadyCached`); pair replication carries the artifact even to a node that missed its own fetch |
|
||||
|
||||
Step 4 is the headline: it proves central-SQL independence at the driver **and** the #485 last-known-good contract on the fetch path simultaneously. Record actual output for every step, especially anything that fails.
|
||||
|
||||
**Step 5: Commit** `docs(mesh): Phase 3 live-gate record`
|
||||
|
||||
---
|
||||
|
||||
## Risks specific to this phase
|
||||
|
||||
- **First in-repo proto.** The `Grpc.Tools` codegen path (Task 1) is new here; a restore/build ordering or `Directory.Packages.props` pin miss will surface as CS-missing-type, not a proto error. Build Commons in isolation first.
|
||||
- **The dedicated-h2c-listener interaction.** A fused central node with BOTH the LocalDb sync port and the ConfigServe port set must re-apply existing HTTP bindings exactly once; getting this wrong silently moves the AdminUI off its port (the exact failure the LocalDb block's comments warn about). Task 3's coexistence assertion is the guard.
|
||||
- **`RevisionHash == SHA-256(blob)` is load-bearing.** If a future change makes the revision hash something other than the raw-bytes SHA, the fetch verification breaks silently (every fetch → null → every deploy fails). Task 4's mismatch test pins the equality; note it in `ConfigComposer` if touched.
|
||||
- **Blocking in the actor loop.** The fetch is I/O; it MUST run via `PipeTo(self)`, never an awaited call inside a receive (would freeze the mailbox for `FetchTimeoutSeconds`). Task 5.
|
||||
- **Phase-boundary discipline.** Phase 3 removes config **reads** only. The `NodeDeploymentState` write, `DbHealthProbe`, and `EfAlarmConditionStateStore` still touch SQL by design — resist folding Phase 4 in, or the live gate's "no SQL" scope becomes unprovable-in-parts.
|
||||
- **Both-nodes-fetch is deliberate.** Do not gate the fetch on the Primary role to "save a fetch"; it couples config delivery to redundancy timing and diverges from today's both-read behaviour. Idempotency + replication already make the second fetch cheap.
|
||||
@@ -0,0 +1,88 @@
|
||||
{
|
||||
"planPath": "docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md",
|
||||
"note": "Per-cluster mesh Phase 3. Decisions pre-made by the user: gRPC fetch RPC (NOT token-gated HTTP); shared node key (NOT per-deployment token, so no migration and DispatchDeployment is UNCHANGED); config-flagged dark switch ConfigSource:Mode=Direct default; both pair nodes fetch (no Primary gating). Five recon facts + the phase-boundary rule (Phase 3 = config READS only; NodeDeploymentState write / DbHealthProbe / EfAlarmConditionStateStore stay for Phase 4) are in the plan header. RevisionHash == SHA-256(blob) is load-bearing for fetch verification.",
|
||||
"tasks": [
|
||||
{
|
||||
"id": 0,
|
||||
"subject": "Task 0: ConfigSource/ConfigServe options + validator + registration",
|
||||
"status": "pending",
|
||||
"classification": "small",
|
||||
"parallelizableWith": [1]
|
||||
},
|
||||
{
|
||||
"id": 1,
|
||||
"subject": "Task 1: deployment_artifact.v1 proto + first in-repo Grpc.Tools codegen",
|
||||
"status": "pending",
|
||||
"classification": "standard",
|
||||
"parallelizableWith": [0]
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"subject": "Task 2: central DeploymentArtifactService — stream sealed artifact, NotFound-hide the rest",
|
||||
"status": "pending",
|
||||
"classification": "standard",
|
||||
"blockedBy": [1],
|
||||
"parallelizableWith": [4]
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"subject": "Task 3: config-serve auth interceptor + dedicated h2c listener wiring",
|
||||
"status": "pending",
|
||||
"classification": "high-risk",
|
||||
"blockedBy": [0, 2]
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"subject": "Task 4: node gRPC artifact fetcher — SHA-verified reassembly, endpoint failover, null-on-any-failure",
|
||||
"status": "pending",
|
||||
"classification": "standard",
|
||||
"blockedBy": [1],
|
||||
"parallelizableWith": [2]
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"subject": "Task 5: FetchAndCache apply path — fetch->cache->apply-from-bytes, null=apply-failure",
|
||||
"status": "pending",
|
||||
"classification": "high-risk",
|
||||
"blockedBy": [4]
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"subject": "Task 6: FetchAndCache bootstrap from the LocalDb pointer; no driver-side config SQL read",
|
||||
"status": "pending",
|
||||
"classification": "high-risk",
|
||||
"blockedBy": [5]
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"subject": "Task 7: #485 empty/corrupt/mismatch coverage on the FetchAndCache paths",
|
||||
"status": "pending",
|
||||
"classification": "high-risk",
|
||||
"blockedBy": [6]
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"subject": "Task 8: real two-Host gRPC artifact-fetch boundary test + wrong-key falsifiability control",
|
||||
"status": "pending",
|
||||
"classification": "high-risk",
|
||||
"blockedBy": [3, 4],
|
||||
"parallelizableWith": [9]
|
||||
},
|
||||
{
|
||||
"id": 9,
|
||||
"subject": "Task 9: docker-dev rig config + docs + program-plan flip",
|
||||
"status": "pending",
|
||||
"classification": "small",
|
||||
"blockedBy": [5],
|
||||
"parallelizableWith": [8]
|
||||
},
|
||||
{
|
||||
"id": 10,
|
||||
"subject": "Task 10: live gate on docker-dev (site nodes -> FetchAndCache; step 4 = stop central SQL mid-fetch = the headline gate)",
|
||||
"status": "pending",
|
||||
"classification": "high-risk",
|
||||
"blockedBy": [8, 9]
|
||||
}
|
||||
],
|
||||
"lastUpdated": "2026-07-22T23:30:00Z"
|
||||
}
|
||||
Reference in New Issue
Block a user