lmxopcua/docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md

Per-Cluster Mesh Phase 3 — Config Fetch-and-Cache from Central

For Claude: REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.

Goal: In a config-flagged FetchAndCache mode, a driver node obtains its deployed configuration by fetching the artifact bytes from central over a gRPC stream, caching them in LocalDb, and reading all config from that cache — so the driver never reads Deployment.ArtifactBlob from central SQL. Default Direct mode is byte-for-byte today's behaviour.

Architecture: Central (admin role) hosts a new gRPC service DeploymentArtifactService.Fetch(deployment_id) → stream ArtifactChunk, on a dedicated h2c Kestrel listener (the LocalDb-sync listener pattern), gated by a shared bearer key (fail-closed, FixedTimeEquals, path-scoped interceptor). A driver node in FetchAndCache mode, on DispatchDeployment, streams the bytes, verifies SHA-256(bytes) == RevisionHash, writes them through the existing IDeploymentArtifactCache, and applies from the cache; every config read (ReconcileDrivers, PushDesiredSubscriptions, address-space rebuild, Bootstrap recovery, Stale recovery) is redirected to the cache. The #485 "empty/unreadable bytes ⇒ no answer, keep last-known-good, fail the apply" guard is carried onto the fetch and cache-read paths.

Tech Stack: .NET 10, Akka.NET 1.5.62, Grpc.AspNetCore (server, already referenced) + Grpc.Tools + a first in-repo .proto (new — the repo has consumed packaged gRPC clients but never compiled a proto locally), Grpc.Net.Client (node client), the Phase-1 ZB.MOM.WW.LocalDb cache (IDeploymentArtifactCache / LocalDbDeploymentArtifactCache), xUnit + Shouldly + Akka.TestKit.

---

Decisions already made (do not re-litigate)

Decision	Choice	Consequence for this plan
Fetch transport	gRPC fetch RPC (user chose this over token-gated HTTP)	Introduces the repo's first .proto + Grpc.Tools codegen (Task 1); a new central-side h2c listener (Task 3); a generated client on the node (Task 4).
Fetch auth	Shared node key (user chose this over a per-deployment token)	A single bearer secret (ConfigServe:ApiKey == ConfigSource:ApiKey), FixedTimeEquals, fail-closed. No migration, no token lifecycle, and DispatchDeployment is UNCHANGED — the node fetches by deployment id and authenticates with the shared key.
Cutover	Config-flagged dark switch (ConfigSource:Mode = Direct default | FetchAndCache), per node	Matches Phase 2. Rig comes up on Direct; the live gate flips only the site nodes to FetchAndCache (central keeps SQL, models the target). Rollback is a config change.
Both nodes fetch	Yes — no Primary gating of the fetch	Today both pair nodes read central SQL independently (no gating); FetchAndCache keeps that shape. IDeploymentArtifactCache.StoreAsync is idempotent (IsAlreadyCachedAsync), and pair replication means a node that missed its fetch can still get the bytes from the peer. Primary-only fetch is a possible later optimization, explicitly out of scope.

Five facts the recon nailed down (read before Task 5)

1. RevisionHash == SHA-256(artifact bytes). ConfigComposer.SnapshotAndFlattenAsync (src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/AdminOperations/ConfigComposer.cs:57-58) computes the revision hash as SHA-256-hex-lowercase over the exact serialized blob — the same value the cache stores as deployment_pointer.artifact_sha256. So the node can verify a fetch against the RevisionHash already carried in DispatchDeployment; no hash field is needed in the proto.

2. The driver reads central SQL on exactly these config paths, all reading Deployment.ArtifactBlob (never re-composing from raw rows): Bootstrap() (DriverHostActor.cs:584-603, reads NodeDeploymentStates + Deployments.RevisionHash), ReconcileDrivers (1717-1768, reads at 1722-1726), PushDesiredSubscriptions (1880-1898, reads at 1885-1889), TryRecoverFromStale (2363-2390, reads at 2367-2373), and the delegated OpcUaPublishActor.HandleRebuild → LoadArtifact/LoadLatestArtifact (OpcUaPublishActor.cs:475-510) which fires only when ApplyAndAck passes no blob (DriverHostActor.cs:1667). Phase 3 redirects these in FetchAndCache mode.

3. The cache is already complete and correct. IDeploymentArtifactCache / LocalDbDeploymentArtifactCache (src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/) does chunked base64 (128 KiB), SHA-256-verified reassembly, newest-2 retention, and replicates. It is written by CacheAppliedArtifact (DriverHostActor.cs:1796-1835) after a successful apply, and read only by TryBootFromCache (671-714) in the Bootstrap() SQL-unreachable catch. Phase 3 changes WHEN it is written (before apply, from the fetch) and WHEN it is read (always, in FetchAndCache). The _isRunningFromCache flag (field 85, set at 698, cleared at 1676) already exists.

4. DispatchDeployment is payload-free by design (DeploymentId + RevisionHash + CorrelationId) so it never approaches the Akka frame limit. Phase 3 keeps it payload-free — the shared-key decision means no token rides in it, and the bytes travel out-of-band over gRPC. Do not add the artifact to any Akka message.

5. The #485 guard is at every byte-parsing seam and must be carried onto the new paths: ReconcileDrivers returns null on blob.Length == 0 (1735-1747) → ApplyAndAck treats null as an apply failure (does not advance _currentRevision, writes Failed, sends a Failed ack — so a re-dispatch of the same revision actually retries, 1639-1659); CacheAppliedArtifact skips empty (1801-1808); PushDesiredSubscriptionsFromArtifact skips empty (1908-1920); OpcUaPublishActor.HandleRebuild returns on { Length: 0 } keeping the served address space (372-391); the cache's ReassembleAsync returns null on any chunk-count/base64/SHA/timestamp failure. A zero-byte or SHA-mismatched fetch is "no answer," never "a config with no drivers."

---

The dark switch, precisely

ConfigSourceOptions.Mode selects the driver's config source:

• Direct (default): every path reads central SQL exactly as today. The new fetcher is not invoked. Byte-for-byte no behaviour change; the whole phase is inert.
• FetchAndCache: on DispatchDeployment, fetch bytes from central over gRPC → verify SHA-256 == RevisionHash → StoreAsync into the cache → apply from the fetched bytes in hand (passed through to every consumer, so OpcUaPublishActor never reads SQL). Bootstrap() recovers the last-applied deployment from the LocalDb pointer, not NodeDeploymentStates. No driver-side Deployment.ArtifactBlob read occurs.

The central-side serve components (gRPC service, listener, interceptor) are wired unconditionally on admin-role nodes whenever ConfigServe:GrpcListenPort > 0, in both modes — mirroring Phase 2's "register in both modes so flipping the flag is not a redeploy." An idle listener costs nothing.

Explicitly deferred to Phase 4 (do NOT touch here): the NodeDeploymentState write (UpsertNodeDeploymentState, DriverHostActor.cs:2392-2422, still writes the ack row to central SQL), DbHealthProbeActor, EfAlarmConditionStateStore, and removing the driver-role ConfigDb connection string. Phase 3 removes the config-blob reads only. A FetchAndCache node still writes its ack row to SQL; that is Phase 4's cut.

---

Task 0: ConfigSourceOptions + ConfigServeOptions + validators + registration

Classification: small Estimated implement time: ~4 min Parallelizable with: Task 1

Files:

• Create: src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ConfigSourceOptions.cs
• Create: src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ConfigSourceOptionsValidator.cs
• Modify: src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs (register both in AddOtOpcUaCluster, beside the MeshTransportOptions registration)
• Test: tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/ConfigSourceOptionsValidatorTests.cs

Design. One options class holds both the node (fetch) and central (serve) surfaces, keyed ConfigSource and ConfigServe. Model after MeshTransportOptions / MeshTransportOptionsValidator (same directory) exactly — the validator fails host start on a FetchAndCache shape that would leave the node unable to fetch (empty endpoints, empty key), because every such fault otherwise surfaces as a silent absence (a deploy that never applies).

// ConfigSourceOptions.cs
namespace ZB.MOM.WW.OtOpcUa.Cluster;

/// <summary>Node-side config source selection (per-cluster mesh Phase 3).</summary>
public sealed class ConfigSourceOptions
{
    public const string SectionName = "ConfigSource";
    public const string ModeDirect = "Direct";
    public const string ModeFetchAndCache = "FetchAndCache";

    /// <summary><see cref="ModeDirect"/> (read central SQL, today's behaviour) or
    /// <see cref="ModeFetchAndCache"/> (fetch from central over gRPC, read LocalDb).</summary>
    public string Mode { get; set; } = ModeDirect;

    /// <summary>Central artifact-gRPC base addresses, e.g. <c>http://central-1:4055</c>. h2c (http
    /// scheme). Required and tried in order (failover) under <see cref="ModeFetchAndCache"/>.</summary>
    public string[] CentralFetchEndpoints { get; set; } = [];

    /// <summary>Shared bearer key; must equal central's <see cref="ConfigServeOptions.ApiKey"/>.
    /// Supply via env <c>ConfigSource__ApiKey</c>; never commit. Required under FetchAndCache.</summary>
    public string ApiKey { get; set; } = string.Empty;

    /// <summary>Per-fetch deadline. Non-positive rejected under FetchAndCache.</summary>
    public int FetchTimeoutSeconds { get; set; } = 30;
}

/// <summary>Central-side artifact-serve surface (per-cluster mesh Phase 3).</summary>
public sealed class ConfigServeOptions
{
    public const string SectionName = "ConfigServe";

    /// <summary>Dedicated h2c listener port for the artifact gRPC service. <c>0</c> = disabled
    /// (nothing bound). Must differ from the main HTTP port and from LocalDb:SyncListenPort.</summary>
    public int GrpcListenPort { get; set; }

    /// <summary>Shared bearer key the interceptor checks (FixedTimeEquals, fail-closed). Supply via
    /// env <c>ConfigServe__ApiKey</c>; never commit.</summary>
    public string ApiKey { get; set; } = string.Empty;
}

The validator (mirror MeshTransportOptionsValidator): reject an unknown Mode; under FetchAndCache reject empty CentralFetchEndpoints, any endpoint not starting http:// or https://, empty ApiKey, and non-positive FetchTimeoutSeconds. ConfigServeOptions needs no cross-field validation (a 0 port is a legitimate "disabled").

Step 1: Write the failing test — ConfigSourceOptionsValidatorTests with cases: default (Direct, empty everything) → Success; unknown mode → Fail; FetchAndCache + empty endpoints → Fail; + non-http(s) endpoint → Fail; + empty key → Fail; + FetchTimeoutSeconds = 0 → Fail; fully-populated FetchAndCache → Success. (No implicit usings in Cluster.Tests — using Xunit;.)

Step 2: dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests --filter "FullyQualifiedName~ConfigSourceOptionsValidatorTests" → FAIL (types missing).

Step 3: Write the two options classes + the validator; register in AddOtOpcUaCluster:

services.AddValidatedOptions<ConfigSourceOptions, ConfigSourceOptionsValidator>(
    configuration, ConfigSourceOptions.SectionName);
services.Configure<ConfigServeOptions>(configuration.GetSection(ConfigServeOptions.SectionName));

Step 4: Re-run → PASS. Sabotage-check: flip one validator branch (e.g. accept empty endpoints) → its test reddens.

Step 5: Commit feat(mesh): ConfigSource/ConfigServe options + validator (Phase 3 dark switch)

---

Task 1: deployment_artifact.v1 proto + Grpc.Tools codegen

Classification: standard Estimated implement time: ~5 min Parallelizable with: Task 0

Files:

• Create: src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos/deployment_artifact.proto
• Modify: src/Core/ZB.MOM.WW.OtOpcUa.Commons/ZB.MOM.WW.OtOpcUa.Commons.csproj (add Grpc.Tools + Google.Protobuf + Grpc.Core.Api package refs and a <Protobuf> item; GrpcServices="Both" so both server base + client stub generate)
• Modify: Directory.Packages.props (pin Grpc.Tools, Google.Protobuf, Grpc.Core.Api if not already pinned — check first)
• Test: tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests/DeploymentArtifactProtoTests.cs (a compile-touch test that references the generated DeploymentArtifactService.DeploymentArtifactServiceBase type and FetchRequest, proving codegen ran)

Why Commons. Both the central server (in Host/AdminUI) and the node client (in Runtime) need the generated types, and Commons is already referenced by every server project. This is the repo's first locally-compiled proto — there is no existing <Protobuf> item to copy, so wire it from scratch.

syntax = "proto3";
package deployment_artifact.v1;
option csharp_namespace = "ZB.MOM.WW.OtOpcUa.Commons.Protos.DeploymentArtifact.V1";

// Central serves the deployed-configuration artifact bytes to a driver node (Phase 3).
service DeploymentArtifactService {
  // Streams the artifact for a sealed deployment as ordered chunks. NotFound if the id is
  // unknown or not sealed. The client verifies SHA-256(reassembled) == the RevisionHash it
  // already holds, so no hash travels here.
  rpc Fetch(FetchRequest) returns (stream ArtifactChunk);
}
message FetchRequest { string deployment_id = 1; }
message ArtifactChunk { bytes data = 1; }

.csproj addition (mirror the packaging note the LocalDb refs use):

<ItemGroup>
  <PackageReference Include="Grpc.Tools" PrivateAssets="all" />
  <PackageReference Include="Google.Protobuf" />
  <PackageReference Include="Grpc.Core.Api" />
</ItemGroup>
<ItemGroup>
  <Protobuf Include="Protos\deployment_artifact.proto" GrpcServices="Both" />
</ItemGroup>

Step 1: Write the compile-touch test referencing DeploymentArtifactService.DeploymentArtifactServiceBase and new FetchRequest { DeploymentId = "x" }. Step 2: dotnet build src/Core/ZB.MOM.WW.OtOpcUa.Commons → FAIL (no proto/codegen). Step 3: Add the packages, the proto, the <Protobuf> item. dotnet restore then dotnet build src/Core/ZB.MOM.WW.OtOpcUa.Commons. Step 4: dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests --filter "FullyQualifiedName~DeploymentArtifactProtoTests" → PASS. Confirm generated types resolve (obj/.../DeploymentArtifactGrpc.cs). Step 5: Commit feat(mesh): deployment_artifact.v1 proto + first in-repo Grpc.Tools codegen

---

Task 2: Central-side DeploymentArtifactService (streams the blob, existence-hiding)

Classification: standard Estimated implement time: ~5 min Parallelizable with: Task 4 Blocked by: Task 1

Files:

• Create: src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Grpc/DeploymentArtifactService.cs
• Test: tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/Grpc/DeploymentArtifactServiceTests.cs

Design. Subclass the generated DeploymentArtifactServiceBase. Fetch reads the Deployment row by id from OtOpcUaConfigDbContext (admin node has the connection); if the row is missing OR Status != Sealed OR ArtifactBlob.Length == 0, throw RpcException(new Status(StatusCode.NotFound, "…")) (existence-hiding + #485: never stream zero bytes as if valid). Otherwise stream the blob in ≤128 KiB ArtifactChunks via responseStream.WriteAsync. Lives in AdminUI (where the other admin-only endpoints + the config DB context live).

public sealed class DeploymentArtifactService : DeploymentArtifactServiceBase
{
    private const int ChunkSize = 128 * 1024; // matches LocalDbDeploymentArtifactCache.ChunkSize
    private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbFactory;
    private readonly ILogger<DeploymentArtifactService> _log;
    // ctor injects both

    public override async Task Fetch(
        FetchRequest request, IServerStreamWriter<ArtifactChunk> responseStream, ServerCallContext context)
    {
        if (!Guid.TryParse(request.DeploymentId, out var id))
            throw new RpcException(new Status(StatusCode.NotFound, "unknown deployment"));

        await using var db = await _dbFactory.CreateDbContextAsync(context.CancellationToken);
        var row = await db.Deployments.AsNoTracking()
            .Where(d => d.DeploymentId == id)
            .Select(d => new { d.Status, d.ArtifactBlob })
            .FirstOrDefaultAsync(context.CancellationToken);

        // Existence-hiding + #485: unknown / not-sealed / empty are one indistinguishable NotFound.
        if (row is null || row.Status != DeploymentStatus.Sealed || row.ArtifactBlob.Length == 0)
            throw new RpcException(new Status(StatusCode.NotFound, "unknown deployment"));

        for (var offset = 0; offset < row.ArtifactBlob.Length; offset += ChunkSize)
        {
            var len = Math.Min(ChunkSize, row.ArtifactBlob.Length - offset);
            await responseStream.WriteAsync(
                new ArtifactChunk { Data = ByteString.CopyFrom(row.ArtifactBlob, offset, len) },
                context.CancellationToken);
        }
    }
}

(Confirm the sealed-status enum member name from DeploymentStatus; the recon showed Deployment.Status and the deploy sets Sealed.)

Step 1: Write the failing test — an in-memory OtOpcUaConfigDbContext (the AdminUI.Tests pattern) seeded with one sealed deployment; a fake IServerStreamWriter<ArtifactChunk> collecting writes; assert: (a) a sealed non-empty blob streams back and reassembles byte-equal, chunk-bounded at 128 KiB (seed a ~300 KB blob → 3 chunks); (b) an unknown id throws RpcException NotFound; (c) a non-sealed row throws NotFound; (d) a zero-length blob throws NotFound (the #485 serve-side guard). Step 2: run → FAIL. Step 3: implement. Step 4: run → PASS. Sabotage: drop the row.Status != Sealed check → the non-sealed test reddens; drop the ArtifactBlob.Length == 0 check → the zero-length test reddens. Step 5: Commit feat(mesh): central DeploymentArtifactService — stream the sealed artifact, NotFound-hide the rest

---

Task 3: Central serve auth interceptor + dedicated h2c listener wiring

Classification: high-risk Estimated implement time: ~5 min Parallelizable with: none Blocked by: Task 0, Task 2

Files:

• Create: src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/ConfigServeAuthInterceptor.cs
• Modify: src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs (register the interceptor with AddGrpc; add the dedicated h2c listener block mirroring the LocalDb sync block at 407-476; MapGrpcService<DeploymentArtifactService>() inside a hasAdmin && configServeGrpcPort > 0 guard beside 514)
• Test: tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/ConfigServeAuthInterceptorTests.cs

Interceptor — mirror LocalDbSyncAuthInterceptor exactly, scoped by service path "/deployment_artifact.v1.DeploymentArtifactService/", fail-closed (no ConfigServe:ApiKey ⇒ reject ALL with Unauthenticated), Authorization: Bearer <key>, FixedTimeEquals. Calls to other services pass through untouched (so it can share AddGrpc with the LocalDb sync interceptor).

Listener — the artifact service is h2c and cannot share the cleartext HTTP/1 port (same reason the LocalDb sync listener is dedicated). Add, right after the LocalDb sync listener block:

var configServeGrpcPort = builder.Configuration.GetValue<int>("ConfigServe:GrpcListenPort");
if (hasAdmin && configServeGrpcPort > 0)
{
    // Same dedicated-h2c-listener dance as the LocalDb sync listener above: h2c can't negotiate on a
    // cleartext Http1AndHttp2 port, so bind a dedicated HTTP/2-only port and re-apply existing bindings.
    // (Refactor the LocalDb block's existing-binding computation into a shared local if both run — a
    // node that is admin+driver with BOTH ports set must re-apply existing bindings once and add both.)
    builder.WebHost.ConfigureKestrel(kestrel =>
        kestrel.ListenAnyIP(configServeGrpcPort, o => o.Protocols = HttpProtocols.Http2));
    Log.Information("Config-serve artifact gRPC listener bound on :{Port} (h2c).", configServeGrpcPort);
}

Load-bearing interaction with the LocalDb listener. A fused central node is admin and driver; on the rig it may have LocalDb:SyncListenPort set too. ConfigureKestrel is additive across calls, but the LocalDb block re-applies existingBindings (URLS/HTTP_PORTS) and this block must NOT clobber that. Implement by computing existingBindings once and applying it once, then adding whichever of the two dedicated h2c ports are configured. The test below asserts the AdminUI HTTP port still answers when both dedicated ports are set.

MapGrpcService:

if (hasAdmin && configServeGrpcPort > 0)
    app.MapGrpcService<DeploymentArtifactService>();

Step 1: Write the failing test — boot the Host (or a minimal WebApplicationFactory with hasAdmin, ConfigServe:GrpcListenPort set, ConfigServe:ApiKey=k, a seeded sealed deployment): a Grpc.Net.Client channel to the dedicated port with Authorization: Bearer k fetches + reassembles the blob; no header ⇒ Unauthenticated; wrong key ⇒ Unauthenticated; and the AdminUI HTTP surface (/health/active) still returns 200 (listener coexistence). Use TestContext.Current.CancellationToken. Step 2: run → FAIL. Step 3: implement interceptor + Program.cs wiring. Step 4: run → PASS. Sabotage: make the interceptor return without checking when the key is unset (i.e. break fail-closed) → the no-header test reddens; break the listener coexistence (apply only this block's binding) → the /health/active assertion reddens. Step 5: Commit feat(mesh): config-serve auth interceptor + dedicated h2c listener (fail-closed bearer)

---

Task 4: Node-side IDeploymentArtifactFetcher + gRPC client (SHA-verified, failover)

Classification: standard Estimated implement time: ~5 min Parallelizable with: Task 2 Blocked by: Task 1

Files:

• Create: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/IDeploymentArtifactFetcher.cs
• Create: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/GrpcDeploymentArtifactFetcher.cs
• Test: tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Deployment/GrpcDeploymentArtifactFetcherTests.cs

Design.

public interface IDeploymentArtifactFetcher
{
    /// <summary>Fetches + reassembles the artifact for <paramref name="deploymentId"/>, verifying
    /// SHA-256(bytes) == <paramref name="expectedRevisionHash"/>. Returns the bytes, or <see
    /// langword="null"/> on any failure (all endpoints unreachable, NotFound, SHA mismatch,
    /// zero-length) — a null is "no answer," NEVER an empty config (#485).</summary>
    Task<byte[]?> FetchAsync(string deploymentId, string expectedRevisionHash, CancellationToken ct);
}

GrpcDeploymentArtifactFetcher takes IOptions<ConfigSourceOptions>. For each endpoint in order: open a GrpcChannel (h2c: new HttpClientHandler / GrpcChannelOptions with the http:// address), attach Authorization: Bearer {ApiKey} via a CallCredentials or a request header on the call, stream Fetch, accumulate chunk.Data into a buffer, then SHA-256(buffer) hex-lowercase and compare to expectedRevisionHash. On success return the bytes. On RpcException (NotFound / Unavailable / deadline), log and try the next endpoint. If all fail, or the reassembled buffer is empty, or the SHA mismatches, return null (and log a Warning naming the reason). Deadline = FetchTimeoutSeconds.

Why return null, not throw. The caller (Task 5) treats null identically to ReconcileDrivers returning null today: apply failure, keep last-known-good, do not advance the revision. Throwing would risk an unhandled actor message; a typed "no bytes" keeps the #485 contract explicit.

Seam for the test. So the test needs no real second Host, make the gRPC call site injectable: a Func<string /*endpoint*/, DeploymentArtifactService.DeploymentArtifactServiceClient> factory defaulted to the real channel builder, overridable in tests with an in-memory client backed by a fake that streams canned chunks. (This mirrors Phase 2's IMeshClusterClientFactory seam.)

Step 1: Write the failing test — with a fake client factory: (a) a client streaming chunks whose reassembly hashes to expectedRevisionHash → returns the exact bytes; (b) chunks whose hash ≠ expected → returns null; (c) an empty stream → returns null; (d) first endpoint throws RpcException(Unavailable), second streams good bytes → returns the bytes (failover); (e) all endpoints throw → null. Assert the failover case actually consulted endpoint 2 (record calls). Step 2: run → FAIL. Step 3: implement. Step 4: run → PASS. Sabotage: drop the SHA-256 comparison → the mismatch test reddens; break failover (return after the first endpoint's exception) → the failover test reddens. Step 5: Commit feat(mesh): node gRPC artifact fetcher — SHA-verified reassembly, endpoint failover, null-on-any-failure

---

Task 5: Wire fetch-and-cache into the apply path (FetchAndCache mode)

Classification: high-risk Estimated implement time: ~5 min Parallelizable with: none Blocked by: Task 4

Files:

• Modify: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs (add _configSourceMode + _artifactFetcher fields + ctor params + both Props overloads; a new FetchThenApply path used from HandleDispatchFromSteady / ApplyAndAck when FetchAndCache)
• Modify: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs (resolve IOptions<ConfigSourceOptions> + register IDeploymentArtifactFetcher in the hasDriver branch; pass both into DriverHostActor.Props)
• Modify: src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/... (DI: services.AddSingleton<IDeploymentArtifactFetcher, GrpcDeploymentArtifactFetcher>() in the driver branch — place beside the IDeploymentArtifactCache registration)
• Test: tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostActorFetchAndCacheTests.cs

Design. In FetchAndCache mode, on a DispatchDeployment whose revision differs from _currentRevision:

1. bytes = await _artifactFetcher.FetchAsync(deploymentId, revisionHash, ct) (message-loop-safe: run via PipeTo to self as a FetchResult(deploymentId, revisionHash, correlation, bytes?), NOT a blocking await inside the receive).
2. On bytes == null → apply failure: do not advance _currentRevision, UpsertNodeDeploymentState(Failed), SendAck(Failed) — identical to today's ReconcileDrivers-returned-null path (1639-1659). The node keeps serving last-known-good. A re-dispatch retries (revision still not current).
3. On success → StoreAsync(clusterId, deploymentId, revisionHash, bytes) into the cache, then apply from bytes in hand: ReconcileDriversFromBlob(bytes), RebuildAddressSpace(correlation, deploymentId, bytes) (pass the blob so OpcUaPublishActor never reads SQL), PushDesiredSubscriptionsFromArtifact(bytes), UpsertNodeDeploymentState(Applied), SendAck(Applied), _isRunningFromCache = false, set _currentRevision.

Refactor note — do NOT duplicate reconcile logic. ReconcileDrivers(DeploymentId) today reads the blob itself (1722-1726). Extract the post-read body into ReconcileDriversFromBlob(byte[] blob) (the existing #485 blob.Length == 0 guard at 1735-1747 moves into it verbatim), and have the Direct-mode ReconcileDrivers read-then-call it. Then FetchThenApply calls ReconcileDriversFromBlob(bytes) directly. Same for PushDesiredSubscriptions → it already has PushDesiredSubscriptionsFromArtifact(blob) (1908); reuse it.

Idempotency: before fetching, check the cache — if IsAlreadyCached(deploymentId, revisionHash) (add a cheap GetCurrentAsync compare, or a new bool ContainsAsync), skip the fetch and apply from cache. This makes a re-dispatch of an already-applied revision a no-op fetch (and covers "peer already replicated it to us").

Step 1: Write the failing test — a DriverHostActor in FetchAndCache mode with a fake IDeploymentArtifactFetcher + a real in-memory LocalDbDeploymentArtifactCache (the DriverHostActorArtifactCacheTests harness): (a) dispatch → fetcher returns good bytes → node applies, cache written, ack Applied, _currentRevision advances, and it never touched the ConfigDb factory (inject a throwing IDbContextFactory to prove no SQL read on the config path — note the NodeDeploymentState write is Phase 4, so use a factory that permits the ack write but fails Deployments reads, or assert via a spy that Deployments was never queried); (b) fetcher returns null → ack Failed, _currentRevision unchanged, children kept; (c) re-dispatch of the applied revision → no fetch (idempotent), immediate Applied ack. Step 2: run → FAIL. Step 3: implement. Step 4: run → PASS. Sabotage: make the null-bytes branch advance _currentRevision anyway → test (b) reddens (this is the #485 apply-failure contract); remove the idempotency check → test (c) reddens. Step 5: Commit feat(mesh): FetchAndCache apply path — fetch→cache→apply-from-bytes, null=apply-failure

---

Task 6: Redirect all config reads to LocalDb; bootstrap from the pointer

Classification: high-risk Estimated implement time: ~5 min Parallelizable with: none Blocked by: Task 5

Files:

• Modify: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs (Bootstrap, TryRecoverFromStale, and any residual read seams under FetchAndCache)
• Test: tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostActorFetchAndCacheBootstrapTests.cs

Design. In FetchAndCache mode:

• Bootstrap() does NOT read NodeDeploymentStates/Deployments from SQL. Instead: cached = await _cache.GetCurrentUnkeyedAsync(). If non-null → _currentRevision = RevisionHash.Parse(cached.RevisionHash), Become(Steady), ApplyCachedArtifact(new DeploymentId(cached.DeploymentId), cached.Artifact) (the existing cache-boot path at 725-748, which reconciles + rebuilds + resubscribes from the blob and does not read SQL or re-ack). If null (fresh node, empty cache) → Become(Steady) with no revision; the first DispatchDeployment triggers a fetch. On a cache-read exception → log and Become(Steady) (no revision) rather than Stale (Stale meant "SQL down"; in FetchAndCache there is no SQL config read to be down).
• TryRecoverFromStale() is unreachable in FetchAndCache (nothing enters Stale from a config read). Leave it for Direct mode; guard its entry so FetchAndCache never schedules it.
• OpcUaPublishActor is untouched: FetchThenApply and ApplyCachedArtifact both pass the blob to RebuildAddressSpace, so LoadArtifact/LoadLatestArtifact (the SQL reads) are never hit in FetchAndCache. Add an assertion-comment; no code change there.

The one SQL touch that remains (by design, Phase 4's to cut): UpsertNodeDeploymentState still writes the ack row. That is a write, not a config read, and the phase boundary is explicit.

Step 1: Write the failing test — FetchAndCache mode, throwing IDbContextFactory for Deployments reads: (a) a cache pre-seeded with an applied artifact → Bootstrap restores served state (drivers reconciled, address space rebuilt) with no Deployments read; (b) an empty cache → Bootstrap lands in Steady-no-revision without error and a subsequent dispatch fetches; (c) a cache-read failure → Steady-no-revision, not Stale. Reuse the DriverHostActorBootFromCacheTests harness. Step 2: run → FAIL. Step 3: implement the mode-gated bootstrap branch. Step 4: run → PASS. Sabotage: in FetchAndCache, make Bootstrap fall through to the SQL read → test (a)'s throwing-factory makes it red. Step 5: Commit feat(mesh): FetchAndCache bootstrap from the LocalDb pointer; no driver-side config SQL read

---

Task 7: #485 guard coverage on the new fetch + cache-read paths

Classification: high-risk Estimated implement time: ~4 min Parallelizable with: none Blocked by: Task 6

Files:

• Modify: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs (only if a seam is found unguarded)
• Test: tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostActorFetchAndCacheUnreadableTests.cs

This task adds no new mechanism — it proves the #485 contract holds on every FetchAndCache seam, and adds a guard only where a test exposes a gap. The contract: zero-length / SHA-mismatched / unreadable ⇒ no answer, keep last-known-good, fail the apply, never tear down to an empty address space.

Step 1: Write the failing tests (expect some to pass already — that is fine; they are the regression net):

• Fetcher returns null (all endpoints down) mid-steady-state → address space + drivers + subscriptions UNCHANGED, ack Failed, _currentRevision unchanged. (Covered by Task 5(b); re-assert the address-space-kept angle explicitly.)
• Central serves a zero-length blob (shouldn't happen — Task 2 NotFound-hides it — but prove the client also refuses): fetcher gets an empty stream → null → apply failure.
• Cache holds a truncated/corrupt artifact at boot (drop a chunk row / corrupt base64) → ReassembleAsync returns null → Bootstrap treats it as empty cache (Steady-no-revision), does NOT apply a partial config.
• SHA mismatch (server bytes don't match the dispatched RevisionHash) → fetcher null → apply failure, last-known-good kept. Step 2: run → the corrupt-cache and any unguarded case FAIL if a gap exists; otherwise all green (regression net). Step 3: add a guard only where a test is red. Step 4: run → PASS. Sabotage: temporarily make ReassembleAsync return partial bytes on a missing chunk → the corrupt-cache test reddens (proving the guard, not the mock, is what holds the line). Step 5: Commit test(mesh): #485 empty/corrupt/mismatch coverage on the FetchAndCache paths

---

Task 8: Real two-Host gRPC fetch boundary test + falsifiability control

Classification: high-risk Estimated implement time: ~5 min Parallelizable with: Task 9 Blocked by: Task 3, Task 4

Files:

• Create: tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/DeploymentArtifactFetchBoundaryTests.cs

The Phase-2 lesson applied to Phase 3: an in-memory fake client proves fetcher logic, not that a real gRPC stream crosses a real Kestrel h2c listener through the real interceptor. Stand a real central Host (admin, ConfigServe:GrpcListenPort set, ConfigServe:ApiKey=k, a seeded sealed deployment) and drive the real GrpcDeploymentArtifactFetcher against it.

Assertions:

1. A fetch with the right key + a >256 KB blob reassembles byte-equal and verifies against the deployment's RevisionHash.
2. Falsifiability control — the same fetch with a wrong key returns null (interceptor rejects; without this control, assertion 1 cannot distinguish "the boundary works" from "something else supplied the bytes"). Verify the control is real by confirming the right-key fetch on the same server succeeds.
3. An unknown deployment id → null (NotFound-hidden).
4. Endpoint failover: give the fetcher [dead-port, real-port] → it still returns the bytes.

If irreducibly flaky, quarantine with [Trait("Category","ArtifactBoundary")] — do not weaken to the in-memory fake (that is Task 4's job).

Step 5: Commit test(mesh): real two-Host gRPC artifact-fetch boundary test with wrong-key control

---

Task 9: Rig config + docs

Classification: small Estimated implement time: ~5 min Parallelizable with: Task 8 Blocked by: Task 5

Files:

• Modify: docker-dev/docker-compose.yml — central-1/central-2 get ConfigServe__GrpcListenPort (e.g. 4055) + ConfigServe__ApiKey (dev value, the committed-dev-secret exception); all six get ConfigSource__Mode: "${OTOPCUA_CONFIG_MODE:-Direct}"; site nodes get ConfigSource__CentralFetchEndpoints__0/1: http://central-1:4055 / http://central-2:4055 + a matching ConfigSource__ApiKey. Leave Mode=Direct so the rig comes up unchanged; expose the flip via OTOPCUA_CONFIG_MODE=FetchAndCache.
• Modify: docs/Configuration.md — new ConfigSource + ConfigServe sections (the dark switch, the shared-key auth, the h2c dedicated-port requirement, the env-only key rule).
• Modify: docs/Redundancy.md — extend the "Command transport" section: config bytes travel out-of-band over the artifact gRPC stream, both pair nodes fetch independently (idempotent + replicated), central serves from SQL.
• Modify: docs/plans/2026-07-22-per-cluster-mesh-program.md — flip the Phase 3 row + section to reflect the gRPC/shared-key choices.
• Modify: CLAUDE.md — a short "Config source (Phase 3)" note beside the Mesh command transport note.

Step 5: Commit docs(mesh): ConfigSource/ConfigServe config, rig flip switch, Phase 3 as shipped

---

Task 10: Live gate on docker-dev

Classification: high-risk Estimated implement time: ~5 min (plus rig time) Parallelizable with: none Blocked by: Task 8, Task 9

Files:

• Create: docs/plans/2026-07-22-mesh-phase3-live-gate.md

Flip only the site nodes to FetchAndCache (OTOPCUA_CONFIG_MODE=FetchAndCache scoped to site services, or a compose override) so central keeps SQL and the rig models the target topology (site nodes have no SQL). Central stays Direct. Rig AdminUI login is disabled — drive UI steps via browser automation at http://localhost:9200; deploy via POST /api/deployments with X-Api-Key: docker-dev-deploy-key.

#	Step	Pass condition
1	Rig up all-Direct; deploy	Seals green — baseline
2	Flip site nodes to FetchAndCache, restart them, deploy	Seals green; site node logs show a gRPC fetch + cache write + apply-from-bytes, and no Deployments.ArtifactBlob SQL read on the site side
3	Confirm central served it	central logs the DeploymentArtifactService.Fetch call; site logs the SHA-256 verify pass
4	Stop central SQL, deploy (the phase's headline gate)	The site fetch fails (central can't read SQL to serve) → site logs apply-failure, keeps serving last-known-good, ack Failed → deploy TimedOut naming the site nodes; no crash, no empty address space (#485)
5	Restart SQL, redeploy	Fetch lands, seals green — retry semantics on the new path
6	Restart a site node with a warm cache, central up	Boots from the LocalDb pointer, restores served state, no Deployments read
7	Restart a site node with central DOWN	Boots last-known-good from cache; no crash loop (config path needs no central at boot)
8	Corrupt one cached artifact chunk on a site node, restart it	Reassembly rejects it → boots Steady-no-revision (empty-cache path), does NOT apply a partial config; next deploy re-fetches
9	Wrong ConfigSource__ApiKey on one site node, deploy	That node's fetch is rejected (Unauthenticated) → apply-failure + Warning; the other site node (right key) applies
10	Both site nodes fetch the same deploy	Idempotent — the second store is a no-op (IsAlreadyCached); pair replication carries the artifact even to a node that missed its own fetch

Step 4 is the headline: it proves central-SQL independence at the driver and the #485 last-known-good contract on the fetch path simultaneously. Record actual output for every step, especially anything that fails.

Step 5: Commit docs(mesh): Phase 3 live-gate record

---

Risks specific to this phase

• First in-repo proto. The Grpc.Tools codegen path (Task 1) is new here; a restore/build ordering or Directory.Packages.props pin miss will surface as CS-missing-type, not a proto error. Build Commons in isolation first.
• The dedicated-h2c-listener interaction. A fused central node with BOTH the LocalDb sync port and the ConfigServe port set must re-apply existing HTTP bindings exactly once; getting this wrong silently moves the AdminUI off its port (the exact failure the LocalDb block's comments warn about). Task 3's coexistence assertion is the guard.
• RevisionHash == SHA-256(blob) is load-bearing. If a future change makes the revision hash something other than the raw-bytes SHA, the fetch verification breaks silently (every fetch → null → every deploy fails). Task 4's mismatch test pins the equality; note it in ConfigComposer if touched.
• Blocking in the actor loop. The fetch is I/O; it MUST run via PipeTo(self), never an awaited call inside a receive (would freeze the mailbox for FetchTimeoutSeconds). Task 5.
• Phase-boundary discipline. Phase 3 removes config reads only. The NodeDeploymentState write, DbHealthProbe, and EfAlarmConditionStateStore still touch SQL by design — resist folding Phase 4 in, or the live gate's "no SQL" scope becomes unprovable-in-parts.
• Both-nodes-fetch is deliberate. Do not gate the fetch on the Primary role to "save a fetch"; it couples config delivery to redundancy timing and diverges from today's both-read behaviour. Idempotency + replication already make the second fetch cheap.