lmxopcua/docs/plans/2026-08-06-clusterclient-to-grpc.md

# ClusterClient → gRPC-only cross-mesh transport — implementation plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement
> this plan task-by-task. Central tracking doc:
> `docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md` — **update it at every phase
> boundary and defect.** Live gate results go in
> `docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md`.

**Goal:** all central↔site mesh traffic rides gRPC with per-cluster PSK auth from
`ZB.MOM.WW.Secrets`; `ClusterClient`/`ClusterClientReceptionist` are deleted; Akka remoting
never crosses the mesh boundary. **Design:** `2026-08-06-clusterclient-to-grpc-design.md`
(read it first — §4 holds the plan-vs-code corrections this plan builds on).

**Architecture:** one new proto (`mesh_transport.v1`) with two unary services;
`NodeCommandService` hangs off the existing Telemetry listener on driver nodes,
`CentralCommandService` off the existing ConfigServe listener on central. The gRPC path is a
third `MeshTransport:Mode` branch inside the two existing choke-point actors
(`CentralCommunicationActor` / `NodeCommunicationActor`) — no new transport interface, no new
ports, no new DB columns.

**Tech Stack:** .NET 10, Grpc.Tools 2.76 build-time codegen, Akka.NET 1.5.62,
`ZB.MOM.WW.Configuration` validators, `ZB.MOM.WW.Secrets`, xunit + Akka.TestKit.Xunit2.

Everything below was code-verified 2026-08-06. If a cited line has drifted, re-locate by the
quoted identifier, never by line number.

---

## How to execute this plan (read before starting)

- **Phase order:** 0 → (1A ∥ 1B) → 2 → 3 → 4. Phases marked ∥ are parallel-safe **only in
  separate git worktrees** (`git worktree add ../OtOpcUa-1B feat/grpc-central-ack`) — never run
  two agents against one working tree (destructive git races are a known family incident).
  Designated merge order when tracks meet: **1A lands first, 1B rebases** (expected conflicts
  are confined to `Host/Program.cs` service-map blocks, `docker-dev/docker-compose.yml` env
  blocks, and `docs/Configuration.md` — resolve as union).
- **Subagent model:** dispatch each task to an **Opus subagent** (`model: "opus"`). Within one
  phase/worktree, run tasks **sequentially** (they share files and git state). Across
  worktrees, run the 1A and 1B task streams **concurrently** — one Opus agent per worktree.
  Tasks whose `Parallelizable with:` names a task in the *same* worktree may be batched into a
  single agent dispatch, never into two agents on one tree. Review each task's diff before the
  next dispatch (superpowers-extended-cc:subagent-driven-development).
- **Branches:** `feat/grpc-phase0-contract-auth`, `feat/grpc-node-command` (1A),
  `feat/grpc-central-ack` (1B), `feat/grpc-phase3-deletion`. One PR per phase to `master` with
  its DoD met; record PR number + merge SHA in the tracking doc. **Stop at each phase DoD** —
  PR + merge is user-owned unless told otherwise.
- **Build/test:** `dotnet build ZB.MOM.WW.OtOpcUa.slnx` (0 warnings — TreatWarningsAsErrors),
  `dotnet test ZB.MOM.WW.OtOpcUa.slnx`. TestKit tests inherit the existing harnesses
  (`ControlPlaneTestHarness`, `RuntimeActorTestBase`) — never hand-roll
  `ActorSystem.Create`+join.
- **Proto codegen is build-time** (`Grpc.Tools`, `<Protobuf>` items in
  `src/Core/ZB.MOM.WW.OtOpcUa.Commons/ZB.MOM.WW.OtOpcUa.Commons.csproj`) — no checked-in
  generated code. The docker image build is the cross-arch verification; if protoc ever
  misbehaves in the image, fall back to ScadaBridge's checked-in recipe and record it.
- **Rig:** `docker compose -f docker-dev/docker-compose.yml up -d --build` (6 host nodes:
  central-1/-2 = `admin,driver,cluster-MAIN`; site-a-1/-2, site-b-1/-2 = driver + cluster
  role; plus sql/traefik/migrator/cluster-seed). AdminUI `http://localhost:9200` (login
  disabled), headless deploy `POST /api/deployments` with `X-Api-Key: docker-dev-deploy-key`.
  gRPC ports are container-internal: ConfigServe `:4055` (central), Telemetry `:4056` (all
  driver nodes). Mesh mode env: `MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-ClusterClient}"`
  on all six nodes.
- **Coexistence rule:** every migrated path sits behind `MeshTransport:Mode` with
  `ClusterClient` remaining the rig default until Phase 2 and the compiled default staying
  `Dps` throughout (it is the single-mesh/in-process default, not this program's concern).
  Rollback at any point before Phase 3 = flip the mode back. DoD-test flag flips on the rig
  are **rig-only edits, reverted, never committed** until Phase 2.

## The two choke points (where all transport code changes)

- **Central→node:** `CentralCommunicationActor`
  (`src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/CentralCommunicationActor.cs`) —
  `HandleMeshCommand` (`:140-168`) branches on mode; the gRPC branch fans unary `Execute`
  calls instead of `ClusterClient.SendToAll`.
- **Node→central:** `NodeCommunicationActor`
  (`src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Communication/NodeCommunicationActor.cs`) —
  `HandleApplyAck` (`:78-92`) branches; the gRPC branch calls `ReportApply` instead of
  `ClusterClient.Send`. **Publishers and subscribers above these actors are untouched**
  (`ConfigPublishCoordinator`, `AdminOperationsActor`, `DriverHostActor`,
  `ScriptedAlarmHostActor`, the EventStream leg, the coordinator `Forward`).

---

## Phase 0 — Contract + auth + config groundwork (standalone; no behavior change)

Branch `feat/grpc-phase0-contract-auth`. Everything here is additive and dark.

### Task 0.1: `mesh_transport.proto` + contract lock

**Classification:** standard
**Estimated implement time:** ~20 min
**Parallelizable with:** Task 0.3 (same worktree — batch to one agent, do not split)

**Files:**
- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos/mesh_transport.proto`
- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos/MeshTransportProtoContract.cs`
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Commons/ZB.MOM.WW.OtOpcUa.Commons.csproj` (third `<Protobuf Include="Protos\mesh_transport.proto" GrpcServices="Both"/>`)
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests/MeshTransportProtoContractTests.cs`

Package `mesh_transport.v1`, `csharp_namespace "ZB.MOM.WW.OtOpcUa.Commons.Protos.MeshTransport.V1"`
(NOT `MeshCommand` — collides with `Commons.Messages.Mesh.MeshCommand`). Services + envelope per
design §3.2: `NodeCommandService.Execute(CommandEnvelope) returns (ExecuteAck)`;
`CentralCommandService.ReportApply(ApplyAckMessage) returns (ReportAck)`. `CommandEnvelope` has
`correlation_id` + `oneof command` of 4 messages mirroring `DispatchDeployment`
(`Commons/Messages/Deploy/DispatchDeployment.cs`), `RestartDriver`, `ReconnectDriver`
(`Commons/Messages/Admin/`), `AlarmCommand` (`Commons/OpcUa/AlarmCommand.cs`).
`ApplyAckMessage` mirrors `ApplyAck` (`DeploymentId, NodeId, Outcome, FailureReason,
CorrelationId`). **Every enum reserves 0 as `*_UNSPECIFIED`** (`AlarmCommand` verb,
`ApplyAck.Outcome`) — proto3 zero must never decode to a meaningful value.
`MeshTransportProtoContract.HandledCases` mirrors `TelemetryProtoContract.cs:18-24`; the test
mirrors `TelemetryProtoContractTests` (fails when a oneof case is added but unhandled). TDD:
write the contract test red (proto absent), add proto + helper, green, commit
`feat(mesh): mesh_transport.v1 proto contract`.

### Task 0.2: DTO mapper + round-trip goldens

**Classification:** high-risk (data contract — this is where ScadaBridge found five silent-drop bugs)
**Estimated implement time:** ~30 min
**Parallelizable with:** none (depends on 0.1)

**Files:**
- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Messages/Mesh/MeshTransportDtoMapper.cs`
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests/MeshTransportDtoMapperTests.cs`

Static mapper: `ToProto`/`FromProto` for all five messages + the envelope. Before writing it,
**enumerate every member of the five records in the test first** (golden round-trips: construct
each record with every property set to a non-default value, map to proto and back, assert
record equality; add a null-variant golden for every nullable member — `FailureReason` at
minimum). Guard against the known trap classes: init-only props (use object initializers in
`FromProto`), nullable strings where `""` is not null (use proto3 `optional`), enum zero
(decode unknown → a `Unknown` CLR value or throw, never a meaningful default). Unknown oneof
case in `FromProto` ⇒ throw `InvalidOperationException` naming the case — the contract-lock
test ensures it's unreachable. Commit `feat(mesh): DTO mapper with round-trip goldens`.

### Task 0.3: `MeshTransportOptions` + validators + SplitTopology relaxation

**Classification:** standard
**Estimated implement time:** ~25 min
**Parallelizable with:** Task 0.1 (same worktree — batch)

**Files:**
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptions.cs`
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptionsValidator.cs`
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/SplitTopologyTransportValidator.cs`
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/MeshTransportOptionsValidatorTests.cs`, `.../SplitTopologyTransportValidatorTests.cs`

Add `ModeGrpc = "Grpc"`; new members `string[] CentralGrpcEndpoints`, `string? GrpcPsk`,
`int DispatchTimeoutSeconds = 10`, `int AckTimeoutSeconds = 10`, and a config-map override
`Dictionary<string,string> ClusterPsks` (central-side rig/test override; production leaves it
empty and the runtime provider falls back to Secrets — mirrors ScadaBridge's
`SitePsks__site-*` rig pattern). Validator, following the house rules (roles via
`RoleParser.ResolveEffectiveRoles` from `IConfiguration`, keys required only where
load-bearing, messages name the exact key + env form `MeshTransport__GrpcPsk`, never echo
values): under `Grpc` + driver role ⇒ `GrpcPsk` non-empty, `CentralGrpcEndpoints` non-empty
and each `http(s)://`-prefixed, both timeouts > 0. `SplitTopologyTransportValidator:101-109`:
accept `ClusterClient` **or** `Grpc` (tighten to `Grpc`-only in Phase 3 — without this
relaxation a rig node flipped to `Grpc` refuses to boot). Also correct the three stale
"Phase 6 deletes the DPS branch" comments (`MeshTransportOptions.cs:11-13`,
`ConfigPublishCoordinator.cs:70-73`, `AdminOperationsActor.cs:90-92`) to match
`docs/Configuration.md:150-156`. TDD red-first on both validators. Commit
`feat(mesh): Grpc mode value + validation`.

### Task 0.4: node-side `MeshCommandAuthInterceptor`

**Classification:** high-risk (security)
**Estimated implement time:** ~20 min
**Parallelizable with:** Task 0.5 (different files, same worktree — batch)

**Files:**
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/MeshCommandAuthInterceptor.cs`
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs:550-562` (add to the `hasDriver` interceptor block)
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Configuration/MeshCommandAuthInterceptorTests.cs`

Clone `TelemetryStreamAuthInterceptor.cs` byte-for-byte in structure: ServicePrefix
`/mesh_transport.v1.NodeCommandService/`, all four `*ServerHandler` overrides → one
`Authorize`, case-insensitive `authorization` metadata walk, `Bearer ` prefix,
`CryptographicOperations.FixedTimeEquals` over UTF-8, expected token =
`IOptions<MeshTransportOptions>.GrpcPsk`, **fail-closed when unset** (reject all with
actionable Warning), deny `PermissionDenied`, non-matching paths pass through. **Exactly ONE
public constructor** + a `HasExactlyOnePublicConstructor` reflection test (the
`Grpc.AspNetCore` multi-ctor silent-skip trap, `TelemetryStreamAuthInterceptor.cs:38-42`; cost
ScadaBridge a live-gate day). No `x-otopcua-cluster` requirement here — header-on-node-side is
theatre (design §3.4). Tests: no-token / wrong-token / correct-token / unset-key /
non-matching-path, mirroring `TelemetryStreamAuthInterceptorTests`. Commit
`feat(mesh): node-side PSK interceptor`.

### Task 0.5: `IMeshPskProvider` + central-side `CentralCommandAuthInterceptor`

**Classification:** high-risk (security + the one new stateful component)
**Estimated implement time:** ~35 min
**Parallelizable with:** Task 0.4 (batch)

**Files:**
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/IMeshPskProvider.cs` (+ `MeshPskProvider`)
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/CentralCommandAuthInterceptor.cs`
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` (DI registration; add interceptor to the `hasAdmin` block)
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication/MeshPskProviderTests.cs`, `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Configuration/CentralCommandAuthInterceptorTests.cs`

`IMeshPskProvider.TryGetPsk(string clusterId, out string psk)`: check
`MeshTransportOptions.ClusterPsks` config map first, else resolve secret
`OT-MESH-PSK-{clusterId}` via `ISecretResolver` (the fail-closed lazy pattern the Galaxy
driver uses for `secret:` refs), **cache per cluster, `Invalidate(clusterId)` /
`InvalidateAll()` hooks** (wired to contact refresh in 1A). Fail-closed: unresolvable ⇒
`false`, caller denies/skips — never proceeds unauthenticated. Adversarial tests, not just
happy-path: unknown cluster, resolver throw, cache serves stale until invalidated, empty
secret value = failure. `CentralCommandAuthInterceptor`: ServicePrefix
`/mesh_transport.v1.CentralCommandService/`; reads `x-otopcua-cluster`, missing/unknown ⇒
`PermissionDenied`; verifies bearer against that cluster's PSK via the provider,
`FixedTimeEquals`; one public ctor + reflection test. Commit
`feat(mesh): per-cluster PSK provider + central interceptor`.

### Task 0.6: rig config, issue filing, docs stub

**Classification:** small
**Estimated implement time:** ~15 min
**Parallelizable with:** none (last in phase; touches compose + docs)

**Files:**
- Modify: `docker-dev/docker-compose.yml` (all six host nodes: `MeshTransport__GrpcPsk` dev literals `dev-mesh-psk-MAIN` / `-SITE-A` / `-SITE-B`; `MeshTransport__CentralGrpcEndpoints__0/1` = `http://central-1:4055`, `http://central-2:4055`; central pair additionally `MeshTransport__ClusterPsks__MAIN` / `SITE-A` / `SITE-B` config-map literals. **Mode stays `${OTOPCUA_MESH_MODE:-ClusterClient}`.**)
- Modify: `docs/Configuration.md` (MeshTransport section: document `Grpc` mode + new keys as "shipped dark, cutover pending")
- Modify: `docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md` (Phase 0 row)

Also: file the Gitea issue for `FleetStatusBroadcaster`'s central-pair-blind fleet view
(design §3.8 — pre-existing, transport-independent, NOT fixed here; record the number in the
tracking doc). Full-suite run; rig rebuild; confirm all six nodes boot with the new env
present and mode unchanged (validators must not fire — keys are only required under `Grpc`).
Commit `chore(mesh): rig PSK config + docs stub`.

**Phase 0 DoD:** suite green, 0 warnings; rig boots all six nodes on `ClusterClient` with the
dark config present; both interceptors + provider + mapper fully unit-tested; FleetStatus
issue filed; tracking doc updated. Live-gate doc Phase 0 section recorded (small gate: boot +
validator-firing checks, including one deliberate misconfig — `Mode=Grpc` with no `GrpcPsk` on
a scratch container refuses boot with the exact-key message). **STOP — PR + merge are
user-owned.**

---

## Phase 1A — Central→node command plane (worktree A, branch `feat/grpc-node-command`)

### Task 1A.1: `NodeCommandGrpcService`

**Classification:** standard
**Estimated implement time:** ~25 min
**Parallelizable with:** Task 1B.1 (OTHER worktree only)

**Files:**
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Grpc/NodeCommandGrpcService.cs`
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs:707-725` (map under `hasDriver && telemetryListenPort > 0`, beside `TelemetryStreamGrpcService`)
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Grpc/NodeCommandGrpcServiceTests.cs`

Follow the alias convention (`using GeneratedServiceBase = …NodeCommandService.NodeCommandServiceBase;`,
class suffixed `GrpcService`). `Execute`: decode via `MeshTransportDtoMapper.FromEnvelope`, Tell
the registered `NodeCommunicationActor` (resolve via `NodeCommunicationActorKey` registry,
`Runtime/ServiceCollectionExtensions.cs:583`), return `ExecuteAck`. Actor not yet registered
(boot window) ⇒ `RpcException(Unavailable)` — the readiness convention. Decode failure ⇒
`InvalidArgument`. Node side re-emits on EventStream exactly as under ClusterClient — the
existing `NodeCommunicationActor` receive path handles the four types already; verify
`EventStreamCommandDeliveryTests` pass **unmodified** (if they need edits, the seam is wrong).
TDD with TestKit probe as the registered actor. Commit `feat(mesh): NodeCommandService on the
telemetry listener`.

### Task 1A.2: `MeshGrpcCommandDialer` + the third branch in `CentralCommunicationActor`

**Classification:** high-risk (actor + concurrency)
**Estimated implement time:** ~40 min
**Parallelizable with:** Task 1B.2 (OTHER worktree only)

**Files:**
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/MeshGrpcCommandDialer.cs`
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/CentralCommunicationActor.cs`
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication/CentralCommunicationActorTests.cs` (new facts), `.../MeshGrpcCommandDialerTests.cs`

Dialer (plain class, injectable fake): `DispatchAsync(nodeEndpoint, CommandEnvelope, ct)` —
`GrpcChannel` cached per endpoint (`ConcurrentDictionary`, `IDisposable`, the
`GrpcDeploymentArtifactFetcher.cs:138-152` shape), bearer from `IMeshPskProvider` for the
target node's cluster + `x-otopcua-cluster` header, deadline via linked CTS
(`DispatchTimeoutSeconds`). Actor changes: extend the contact-load query to also select
`GrpcPort` (rows already loaded — `LoadContactsFromDb`, `CentralCommunicationActor.cs:207-245`);
under `Mode=Grpc`, `HandleMeshCommand` maps the message via `ToProto` and fans
`DispatchAsync` to every enabled non-maintenance node across all clusters (the `SendToAll`
recipient set), `PipeTo`-ing per-node results back for logging only — **a failure is logged
with node id + status and dropped; no retry** (design §3.3). Skip-with-warn a node whose
`GrpcPort` is NULL (mirror `TelemetryNodeSource:49-60`). Wire `IMeshPskProvider.Invalidate`
to the contact-refresh tick. New facts mirror the existing `SendToAll` facts
(`CentralCommunicationActorTests:142,191`) with a fake dialer: fan set, per-node failure
isolation (one dead node doesn't stop the fan), NULL-port skip, PSK-miss skip (fail-closed).
Commit `feat(mesh): gRPC command fan-out behind Mode=Grpc`.

### Task 1A.3: 1A live check (rig-only, nothing committed but the gate record)

**Classification:** small
**Estimated implement time:** ~20 min (rig wall-time)
**Parallelizable with:** none

Rebuild the rig from the branch. Flip **central-1/-2 only** to `MeshTransport__Mode: "Grpc"`
(rig-only edit, reverted after). Sites stay `ClusterClient`. Run live-gate Phase 1A checks
(deploy dispatch reaches all four site nodes over gRPC while acks return over ClusterClient —
coexistence proven; RestartDriver/AlarmCommand from AdminUI; unauthenticated + wrong-key
`grpcurl` probes against a site node's `:4056` rejected). Record in the live-gate doc; revert
the flip. **Phase 1A DoD:** suite green; gate section PASS; tracking updated. **STOP.**

---

## Phase 1B — Node→central ack plane (worktree B, branch `feat/grpc-central-ack`)

### Task 1B.1: `CentralCommandGrpcService` + admin validator rule

**Classification:** standard
**Estimated implement time:** ~25 min
**Parallelizable with:** Task 1A.1 (OTHER worktree only)

**Files:**
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Grpc/CentralCommandGrpcService.cs` (beside `DeploymentArtifactService.cs`)
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs:707-725` (map under `hasAdmin && configServeGrpcPort > 0`)
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptionsValidator.cs` (admin role + `Mode=Grpc` ⇒ `ConfigServe:GrpcListenPort > 0`, cross-read from `IConfiguration` — the SplitTopology pattern)
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/Grpc/CentralCommandGrpcServiceTests.cs`, validator tests

`ReportApply`: decode `ApplyAckMessage` → Tell `CentralCommunicationActor` (resolve via
`CentralCommunicationActorKey` registry, `ControlPlane/ServiceCollectionExtensions.cs:246`) →
its existing `HandleApplyAck` `Forward`s to the coordinator proxy untouched. Not-ready ⇒
`Unavailable`; decode failure ⇒ `InvalidArgument`. Commit `feat(mesh): CentralCommandService
on the ConfigServe listener`.

### Task 1B.2: `GrpcApplyAckSender` + node-side wiring

**Classification:** high-risk (actor wiring; the `Send`-not-`SendToAll` single-count semantic)
**Estimated implement time:** ~35 min
**Parallelizable with:** Task 1A.2 (OTHER worktree only)

**Files:**
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Communication/GrpcApplyAckSender.cs`
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Communication/NodeCommunicationActor.cs`
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs:389-416,536`
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Communication/NodeCommunicationActorTests.cs` (new facts), `.../GrpcApplyAckSenderTests.cs`

Sender: try `CentralGrpcEndpoints` **in order, stop at first success** (one ack per node —
the `Send` semantic; the endpoint loop + per-endpoint channel cache of
`GrpcDeploymentArtifactFetcher:67-116`), bearer = own `GrpcPsk` + `x-otopcua-cluster` = own
clusterId (from `RoleParser` cluster role), deadline `AckTimeoutSeconds`; all endpoints fail ⇒
log Warning + drop (today's no-client semantic, now visible). `NodeCommunicationActor` gains
an **optional** `ackSender` ctor param (null under `Dps`/`ClusterClient` — existing tests pass
with ZERO edits; if they need edits, the seam is wrong); under `Mode=Grpc` the registration
(`ServiceCollectionExtensions.cs:389-416`) creates no ClusterClient and passes the sender;
`_ackRouter` wiring (`:536`) is unchanged (still the actor). New facts: gRPC branch sends
exactly once on first-endpoint success, fails over to second, drops after both. Commit
`feat(mesh): gRPC ApplyAck path behind Mode=Grpc`.

### Task 1B.3: 1B live check (rig-only)

**Classification:** small
**Estimated implement time:** ~20 min (rig wall-time)
**Parallelizable with:** none

Rebuild from branch (after rebasing on merged 1A if it landed — expected union conflicts:
`Program.cs` map block, compose, `Configuration.md`). Flip **site-a-1/-2 only** to `Grpc`.
Deploy e2e: site-a acks arrive over gRPC (`PersistNodeAck` rows land, deployment seals),
site-b still acks over ClusterClient — coexistence proven. Kill central-1 mid-deploy: site-a
ack fails over to central-2's endpoint. Record; revert. **Phase 1B DoD:** suite green; gate
PASS; tracking updated. **STOP.**

---

## Phase 2 — Full-rig cutover + soak (sequential, after 1A+1B merged)

> **Targeted-test policy (phases 2–4):** the full solution suite is large (~3,000+ tests,
> 40+ min wall). Per-task verification runs are TARGETED — run only the projects/filters the
> task touches (named per task below). ONE full-suite run happens per phase, at the phase DoD
> before the merge/STOP — never per task. The mesh-relevant targeted set, for reference:
>
> ```bash
> # mesh transport core (actor, dialer, provider, starter order)
> dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/ --filter "FullyQualifiedName~Communication"
> # node-side actor + ack sender + registration wiring
> dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/ --filter "FullyQualifiedName~Communication|FullyQualifiedName~ServiceCollectionExtensions"
> # interceptors + gRPC services (both sides)
> dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/
> # options/validators
> dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/ --filter "FullyQualifiedName~MeshTransport|FullyQualifiedName~SplitTopology"
> # proto contract + mapper goldens
> dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests/ --filter "FullyQualifiedName~MeshTransport"
> # central ack ingress
> dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/ --filter "FullyQualifiedName~CentralCommandGrpcService"
> ```

### Task 2.0 (MUST FIX, opening task — before the flip and the soak): honest PreStart-race logging

**Classification:** small
**Estimated implement time:** ~15 min
**Parallelizable with:** none (first in phase)

**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/CentralCommunicationActor.cs`
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication/ControlPlaneStarterOrderTests.cs`, `CentralCommunicationActorTests.cs` (string/severity pins only)

Both live gates proved the `PreStart` resolve race fires on essentially EVERY admin boot
(1A: 2/3; 1B: 4/4) and logs two FALSE Errors — `Resolve`'s "continuing without it" (reads
permanent; the memoize heals it at first use) and the I-1 guard's "known-cluster gate will
NEVER be published" (it is published moments later — the 1B gate accepted acks and refused
`SITE-Z` on every such boot). A healthy central currently fails any error-grep, including
this phase's soak.

Required behavior after this task: a PreStart-time resolve failure logs at
**Debug/Info** ("not yet resolvable — starter ordering; will retry on first use"); the
**Error** fires only when a FIRST-USE retry also fails (the genuinely-permanent absence —
`Resolve` is already re-invoked by the memoizing properties, so this is a severity/wording
split by call site, not new machinery). The I-1 null-guard Errors in `PreStart` move to the
same first-use semantics or gain "if this persists past the first contact refresh" wording.
Net: **a healthy admin boot produces ZERO Warning/Error lines from this path** (pinned by a
test asserting the boot-time severity), while a truly-absent dependency still produces a
loud Error at first use (existing pins keep this). **Targeted tests only:**
`ControlPlaneStarterOrderTests` + `CentralCommunicationActorTests` filters. Small review
chain (code-reviewer). Commit `fix(mesh): PreStart resolve race logs honestly — quiet on
heal, loud on permanent absence`. Lands on `master` directly (post-1B-merge, pre-flip).

### The cutover

No feature branch; one commit flipping the rig default:
`docker-dev/docker-compose.yml` `MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}"`.

**Before flipping, run the BASELINE on the pre-change build** (live-gate doc has the exact
commands): reproduce the two gaps this program closes — (a) a site container holds a live Akka
association to central's `:4053` (the unauthenticated cross-WAN surface), (b) stop the site-b
pair, dispatch a deployment, observe the silent drop (no per-node error at dispatch; only the
2-min deadline marks it). Reproduced, not assumed.

Then: full rebuild all-Grpc; run the Phase 2 check table (deploy e2e all six nodes;
Restart/Reconnect/AlarmCommand matrix; kill-central-1 ack failover; kill-site-node
mid-dispatch = logged per-node failure + deadline backstop, no retry storm; PSK negatives
against both services on running containers; **zero ClusterClient log lines under `Grpc`**;
soak ≥ 2 h with 0 `PermissionDenied` noise — PSK drift detector; the soak error-grep must be
clean, which Task 2.0 makes possible on a healthy rig). Per-check verification is on the RIG;
unit runs during this phase are targeted (see policy above). **Phase 2 DoD:** every check
PASS in the live-gate doc; ONE full-suite run green on the cutover SHA; rig default
committed; tracking updated. **STOP.**

---

## Phase 3 — Deletion + tightening + docs (branch `feat/grpc-phase3-deletion`)

### Task 3.1: delete ClusterClient (code + HOCON + config)

**Classification:** high-risk (deletion ordering)
**Estimated implement time:** ~40 min
**Parallelizable with:** none

**Files:** `Runtime/ServiceCollectionExtensions.cs`, `ControlPlane/ServiceCollectionExtensions.cs`,
`ControlPlane/Communication/{CentralCommunicationActor.cs,IMeshClusterClientFactory.cs}`,
`Runtime/Communication/NodeCommunicationActor.cs`, `Cluster/Resources/akka.conf`,
`Cluster/MeshTransportOptions.cs` + validator, `docker-dev/docker-compose.yml`, affected tests.

In order: (1) delete `IMeshClusterClientFactory`/`DefaultMeshClusterClientFactory` + central
per-cluster client cache/`RebuildClient` (**keep `LoadContactsFromDb` — it feeds the gRPC
dialer**); (2) delete node client creation + `centralClient` param; (3) delete both
receptionist registrations; (4) `akka.conf`: remove `ClusterClientReceptionistExtensionProvider`
from `akka.extensions` + the whole `akka.cluster.client` block (**keep DPS extension +
`maximum-frame-size` canary — pair-local channels live**); (5) delete
`CentralContactPoints` + validator rules + all six compose entries; (6) delete the
`ClusterClient` mode value; `SplitTopologyTransportValidator` requires `Grpc`; (7) retire/
replace tests: `MeshClusterClientBoundaryTests` → `MeshGrpcBoundaryTests` (two clusters, gRPC
between them, **assert no cross-mesh Akka association forms** — the program's point, kept as a
test), `MeshClusterClientFactoryTests` deleted, `MeshTransportHoconTests` drops the
client-settings asserts (keeps frame-size + DPS pins), `CentralCommunicationActorTests` /
`NodeCommunicationActorTests` / `MeshRouterDispatchTests` reshaped to Dps+Grpc. Grep-gates:
`rg -i "clusterclient|receptionist" src tests docker-dev docs` → only historical
plans/CHANGELOG; `rg "CentralContactPoints"` → empty. Commit sequence per step.
**Targeted tests per step, NOT the full suite:** steps 1–3 → the ControlPlane + Runtime
`Communication`/`ServiceCollectionExtensions` filters; step 4 → `MeshTransportHoconTests`;
steps 5–6 → the Cluster.Tests `MeshTransport|SplitTopology` filter + Host.Tests; step 7 →
the replaced/reshaped fixtures by name. The FULL suite runs once, at the Phase 3 DoD.

### Task 3.2: docs sweep

**Classification:** small
**Estimated implement time:** ~20 min
**Parallelizable with:** none

`docs/Configuration.md` (MeshTransport = Dps|Grpc, keys, PSK seeding), `docs/Redundancy.md`
transport table, `docs/Telemetry.md` cross-refs, `OtOpcUa/CLAUDE.md` transport decisions,
`deferment.md` if applicable — and the **scadaproj index entry** (`~/Desktop/scadaproj/CLAUDE.md`,
committed on scadaproj `main` and pushed, per its index-edit rule). Update tracking doc.
Task 3.2 itself is docs-only — no test runs beyond a build.
**Phase 3 DoD:** ONE full-suite run green on the branch tip (the phase's only full run);
grep-gates clean; docs merged. **STOP.**

---

## Phase 4 — Terminal live gate (rig; sequential; every check PASS required)

Run on `master` AFTER Phase 3 merges, on the **post-deletion image** (verify image build
timestamp > merge time — the ScadaBridge provenance guard). Eight checks, recorded in the
live-gate doc: (1) PSK negatives both services incl. unset-key fail-closed; (2) command
matrix all six nodes; (3) ack matrix + seal; (4) central-kill ack failover + site-node-kill
dispatch honesty; (5) mid-deploy central kill → deadline/replay backstops hold, no dupes;
(6) **boundary check: no Akka association between any site container and central** (was the
baseline gap); (7) restart discipline: full-rig restart clean, **zero
receptionist/ClusterClient log lines anywhere**; (8) `MeshGrpcBoundaryTests` (targeted) +
the program's TERMINAL full-suite run, green on the merge SHA — this is Phase 4's one full
run and the program's last; everything before it in this phase is rig checks + targeted
fixtures. Close any carry-forwards named by earlier gates (incl. the I-4 cold-cache
measurement if a store-backed rig exists by then — else re-carry with rationale). Tracking
doc final row; program CLOSED.

---

## Gotchas for the executor (will bite; read twice)

1. **One public constructor per interceptor** — `Grpc.AspNetCore` silently skips multi-ctor
   interceptors; correct/wrong/no key then fail identically (`Unknown`), which is
   indistinguishable from "working" in unit tests that register via DI. Reflection-pin it.
2. **Enum zero is `UNSPECIFIED`, never a meaningful value** — proto3 decodes absent as 0.
3. **Do NOT touch the Kestrel block** (`Program.cs:564-671`) — no new listeners exist in this
   plan; any deviation that adds one re-imports the explicit-endpoints-suppresses-URLS trap.
4. **Fire-and-forget stays fire-and-forget** — no retry on `DeadlineExceeded` or anything
   else; the deploy deadline + boot replay are the only recovery. Adding a retry turns one
   `RestartDriver` into two.
5. **`ConfigPublishCoordinator`'s `deployment-acks` DPS subscription stays** — it is the
   `Dps`-mode ingress, not dead code.
6. **`SendToAll`/`Send` semantics:** fan = every enabled non-maintenance `ClusterNode`; ack =
   exactly once, first healthy endpoint. The coordinator counts acks — a double-send
   double-counts.
7. **Existing tests are the seam oracle:** `EventStreamCommandDeliveryTests` and the
   pre-existing `NodeCommunicationActorTests`/`CentralCommunicationActorTests` construction
   paths must pass **unmodified** through 1A/1B (nullable ctor params, never signature
   breaks). If they need edits before Phase 3, the seam is wrong.
8. **Rig flips before Phase 2 are never committed**; a redeploy from `master` must reset to
   all-ClusterClient until the Phase 2 commit.
9. **`x-otopcua-cluster` is a central-side routing hint** — do not add it to the node-side
   interceptor's requirements (theatre), and do not prove isolation with it (wrong-key
   rejection is the proof).
10. **Secrets on driver-only nodes work** (node-local SQLite + env KEK) — do not add a
    ConfigDb dependency anywhere on the node path; that constraint is the whole point of
    mesh Phases 3/4.

## Effort & parallelization

| Track | Est. | Parallel with |
|---|---|---|
| Phase 0 | 2–3 d | — (single worktree; tasks 0.1+0.3 and 0.4+0.5 batchable) |
| 1A | 3–4 d | 1B (separate worktrees; 1A merges first, 1B rebases) |
| 1B | 2–3 d | 1A |
| 2 | 1–2 d | — |
| 3 | 1–2 d | — |
| 4 | 1 d | — |

Critical path ≈ 0 → 1A → 2 → 3 → 4: **~1.5–2.5 weeks.**