lmxopcua/docs/plans/2026-08-06-clusterclient-to-grpc.md

ClusterClient → gRPC-only cross-mesh transport — implementation plan

For Claude: REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. Central tracking doc: docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md — update it at every phase boundary and defect. Live gate results go in docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md.

Goal: all central↔site mesh traffic rides gRPC with per-cluster PSK auth from ZB.MOM.WW.Secrets; ClusterClient/ClusterClientReceptionist are deleted; Akka remoting never crosses the mesh boundary. Design: 2026-08-06-clusterclient-to-grpc-design.md (read it first — §4 holds the plan-vs-code corrections this plan builds on).

Architecture: one new proto (mesh_transport.v1) with two unary services; NodeCommandService hangs off the existing Telemetry listener on driver nodes, CentralCommandService off the existing ConfigServe listener on central. The gRPC path is a third MeshTransport:Mode branch inside the two existing choke-point actors (CentralCommunicationActor / NodeCommunicationActor) — no new transport interface, no new ports, no new DB columns.

Tech Stack: .NET 10, Grpc.Tools 2.76 build-time codegen, Akka.NET 1.5.62, ZB.MOM.WW.Configuration validators, ZB.MOM.WW.Secrets, xunit + Akka.TestKit.Xunit2.

Everything below was code-verified 2026-08-06. If a cited line has drifted, re-locate by the quoted identifier, never by line number.

---

How to execute this plan (read before starting)

• Phase order: 0 → (1A ∥ 1B) → 2 → 3 → 4. Phases marked ∥ are parallel-safe only in separate git worktrees (git worktree add ../OtOpcUa-1B feat/grpc-central-ack) — never run two agents against one working tree (destructive git races are a known family incident). Designated merge order when tracks meet: 1A lands first, 1B rebases (expected conflicts are confined to Host/Program.cs service-map blocks, docker-dev/docker-compose.yml env blocks, and docs/Configuration.md — resolve as union).
• Subagent model: dispatch each task to an Opus subagent (model: "opus"). Within one phase/worktree, run tasks sequentially (they share files and git state). Across worktrees, run the 1A and 1B task streams concurrently — one Opus agent per worktree. Tasks whose Parallelizable with: names a task in the same worktree may be batched into a single agent dispatch, never into two agents on one tree. Review each task's diff before the next dispatch (superpowers-extended-cc:subagent-driven-development).
• Branches: feat/grpc-phase0-contract-auth, feat/grpc-node-command (1A), feat/grpc-central-ack (1B), feat/grpc-phase3-deletion. One PR per phase to master with its DoD met; record PR number + merge SHA in the tracking doc. Stop at each phase DoD — PR + merge is user-owned unless told otherwise.
• Build/test: dotnet build ZB.MOM.WW.OtOpcUa.slnx (0 warnings — TreatWarningsAsErrors), dotnet test ZB.MOM.WW.OtOpcUa.slnx. TestKit tests inherit the existing harnesses (ControlPlaneTestHarness, RuntimeActorTestBase) — never hand-roll ActorSystem.Create+join.
• Proto codegen is build-time (Grpc.Tools, <Protobuf> items in src/Core/ZB.MOM.WW.OtOpcUa.Commons/ZB.MOM.WW.OtOpcUa.Commons.csproj) — no checked-in generated code. The docker image build is the cross-arch verification; if protoc ever misbehaves in the image, fall back to ScadaBridge's checked-in recipe and record it.
• Rig: docker compose -f docker-dev/docker-compose.yml up -d --build (6 host nodes: central-1/-2 = admin,driver,cluster-MAIN; site-a-1/-2, site-b-1/-2 = driver + cluster role; plus sql/traefik/migrator/cluster-seed). AdminUI http://localhost:9200 (login disabled), headless deploy POST /api/deployments with X-Api-Key: docker-dev-deploy-key. gRPC ports are container-internal: ConfigServe :4055 (central), Telemetry :4056 (all driver nodes). Mesh mode env: MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-ClusterClient}" on all six nodes.
• Coexistence rule: every migrated path sits behind MeshTransport:Mode with ClusterClient remaining the rig default until Phase 2 and the compiled default staying Dps throughout (it is the single-mesh/in-process default, not this program's concern). Rollback at any point before Phase 3 = flip the mode back. DoD-test flag flips on the rig are rig-only edits, reverted, never committed until Phase 2.

The two choke points (where all transport code changes)

• Central→node: CentralCommunicationActor (src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/CentralCommunicationActor.cs) — HandleMeshCommand (:140-168) branches on mode; the gRPC branch fans unary Execute calls instead of ClusterClient.SendToAll.
• Node→central: NodeCommunicationActor (src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Communication/NodeCommunicationActor.cs) — HandleApplyAck (:78-92) branches; the gRPC branch calls ReportApply instead of ClusterClient.Send. Publishers and subscribers above these actors are untouched (ConfigPublishCoordinator, AdminOperationsActor, DriverHostActor, ScriptedAlarmHostActor, the EventStream leg, the coordinator Forward).

---

Phase 0 — Contract + auth + config groundwork (standalone; no behavior change)

Branch feat/grpc-phase0-contract-auth. Everything here is additive and dark.

Task 0.1: mesh_transport.proto + contract lock

Classification: standard Estimated implement time: ~20 min Parallelizable with: Task 0.3 (same worktree — batch to one agent, do not split)

Files:

• Create: src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos/mesh_transport.proto
• Create: src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos/MeshTransportProtoContract.cs
• Modify: src/Core/ZB.MOM.WW.OtOpcUa.Commons/ZB.MOM.WW.OtOpcUa.Commons.csproj (third <Protobuf Include="Protos\mesh_transport.proto" GrpcServices="Both"/>)
• Test: tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests/MeshTransportProtoContractTests.cs

Package mesh_transport.v1, csharp_namespace "ZB.MOM.WW.OtOpcUa.Commons.Protos.MeshTransport.V1" (NOT MeshCommand — collides with Commons.Messages.Mesh.MeshCommand). Services + envelope per design §3.2: NodeCommandService.Execute(CommandEnvelope) returns (ExecuteAck); CentralCommandService.ReportApply(ApplyAckMessage) returns (ReportAck). CommandEnvelope has correlation_id + oneof command of 4 messages mirroring DispatchDeployment (Commons/Messages/Deploy/DispatchDeployment.cs), RestartDriver, ReconnectDriver (Commons/Messages/Admin/), AlarmCommand (Commons/OpcUa/AlarmCommand.cs). ApplyAckMessage mirrors ApplyAck (DeploymentId, NodeId, Outcome, FailureReason, CorrelationId). Every enum reserves 0 as *_UNSPECIFIED (AlarmCommand verb, ApplyAck.Outcome) — proto3 zero must never decode to a meaningful value. MeshTransportProtoContract.HandledCases mirrors TelemetryProtoContract.cs:18-24; the test mirrors TelemetryProtoContractTests (fails when a oneof case is added but unhandled). TDD: write the contract test red (proto absent), add proto + helper, green, commit feat(mesh): mesh_transport.v1 proto contract.

Task 0.2: DTO mapper + round-trip goldens

Classification: high-risk (data contract — this is where ScadaBridge found five silent-drop bugs) Estimated implement time: ~30 min Parallelizable with: none (depends on 0.1)

Files:

• Create: src/Core/ZB.MOM.WW.OtOpcUa.Commons/Messages/Mesh/MeshTransportDtoMapper.cs
• Test: tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests/MeshTransportDtoMapperTests.cs

Static mapper: ToProto/FromProto for all five messages + the envelope. Before writing it, enumerate every member of the five records in the test first (golden round-trips: construct each record with every property set to a non-default value, map to proto and back, assert record equality; add a null-variant golden for every nullable member — FailureReason at minimum). Guard against the known trap classes: init-only props (use object initializers in FromProto), nullable strings where "" is not null (use proto3 optional), enum zero (decode unknown → a Unknown CLR value or throw, never a meaningful default). Unknown oneof case in FromProto ⇒ throw InvalidOperationException naming the case — the contract-lock test ensures it's unreachable. Commit feat(mesh): DTO mapper with round-trip goldens.

Task 0.3: MeshTransportOptions + validators + SplitTopology relaxation

Classification: standard Estimated implement time: ~25 min Parallelizable with: Task 0.1 (same worktree — batch)

Files:

• Modify: src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptions.cs
• Modify: src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptionsValidator.cs
• Modify: src/Core/ZB.MOM.WW.OtOpcUa.Cluster/SplitTopologyTransportValidator.cs
• Test: tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/MeshTransportOptionsValidatorTests.cs, .../SplitTopologyTransportValidatorTests.cs

Add ModeGrpc = "Grpc"; new members string[] CentralGrpcEndpoints, string? GrpcPsk, int DispatchTimeoutSeconds = 10, int AckTimeoutSeconds = 10, and a config-map override Dictionary<string,string> ClusterPsks (central-side rig/test override; production leaves it empty and the runtime provider falls back to Secrets — mirrors ScadaBridge's SitePsks__site-* rig pattern). Validator, following the house rules (roles via RoleParser.ResolveEffectiveRoles from IConfiguration, keys required only where load-bearing, messages name the exact key + env form MeshTransport__GrpcPsk, never echo values): under Grpc + driver role ⇒ GrpcPsk non-empty, CentralGrpcEndpoints non-empty and each http(s)://-prefixed, both timeouts > 0. SplitTopologyTransportValidator:101-109: accept ClusterClient or Grpc (tighten to Grpc-only in Phase 3 — without this relaxation a rig node flipped to Grpc refuses to boot). Also correct the three stale "Phase 6 deletes the DPS branch" comments (MeshTransportOptions.cs:11-13, ConfigPublishCoordinator.cs:70-73, AdminOperationsActor.cs:90-92) to match docs/Configuration.md:150-156. TDD red-first on both validators. Commit feat(mesh): Grpc mode value + validation.

Task 0.4: node-side MeshCommandAuthInterceptor

Classification: high-risk (security) Estimated implement time: ~20 min Parallelizable with: Task 0.5 (different files, same worktree — batch)

Files:

• Create: src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/MeshCommandAuthInterceptor.cs
• Modify: src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs:550-562 (add to the hasDriver interceptor block)
• Test: tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Configuration/MeshCommandAuthInterceptorTests.cs

Clone TelemetryStreamAuthInterceptor.cs byte-for-byte in structure: ServicePrefix /mesh_transport.v1.NodeCommandService/, all four *ServerHandler overrides → one Authorize, case-insensitive authorization metadata walk, Bearer prefix, CryptographicOperations.FixedTimeEquals over UTF-8, expected token = IOptions<MeshTransportOptions>.GrpcPsk, fail-closed when unset (reject all with actionable Warning), deny PermissionDenied, non-matching paths pass through. Exactly ONE public constructor + a HasExactlyOnePublicConstructor reflection test (the Grpc.AspNetCore multi-ctor silent-skip trap, TelemetryStreamAuthInterceptor.cs:38-42; cost ScadaBridge a live-gate day). No x-otopcua-cluster requirement here — header-on-node-side is theatre (design §3.4). Tests: no-token / wrong-token / correct-token / unset-key / non-matching-path, mirroring TelemetryStreamAuthInterceptorTests. Commit feat(mesh): node-side PSK interceptor.

Task 0.5: IMeshPskProvider + central-side CentralCommandAuthInterceptor

Classification: high-risk (security + the one new stateful component) Estimated implement time: ~35 min Parallelizable with: Task 0.4 (batch)

Files:

• Create: src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/IMeshPskProvider.cs (+ MeshPskProvider)
• Create: src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/CentralCommandAuthInterceptor.cs
• Modify: src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs (DI registration; add interceptor to the hasAdmin block)
• Test: tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication/MeshPskProviderTests.cs, tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Configuration/CentralCommandAuthInterceptorTests.cs

IMeshPskProvider.TryGetPsk(string clusterId, out string psk): check MeshTransportOptions.ClusterPsks config map first, else resolve secret OT-MESH-PSK-{clusterId} via ISecretResolver (the fail-closed lazy pattern the Galaxy driver uses for secret: refs), cache per cluster, Invalidate(clusterId) / InvalidateAll() hooks (wired to contact refresh in 1A). Fail-closed: unresolvable ⇒ false, caller denies/skips — never proceeds unauthenticated. Adversarial tests, not just happy-path: unknown cluster, resolver throw, cache serves stale until invalidated, empty secret value = failure. CentralCommandAuthInterceptor: ServicePrefix /mesh_transport.v1.CentralCommandService/; reads x-otopcua-cluster, missing/unknown ⇒ PermissionDenied; verifies bearer against that cluster's PSK via the provider, FixedTimeEquals; one public ctor + reflection test. Commit feat(mesh): per-cluster PSK provider + central interceptor.

Task 0.6: rig config, issue filing, docs stub

Classification: small Estimated implement time: ~15 min Parallelizable with: none (last in phase; touches compose + docs)

Files:

• Modify: docker-dev/docker-compose.yml (all six host nodes: MeshTransport__GrpcPsk dev literals dev-mesh-psk-MAIN / -SITE-A / -SITE-B; MeshTransport__CentralGrpcEndpoints__0/1 = http://central-1:4055, http://central-2:4055; central pair additionally MeshTransport__ClusterPsks__MAIN / SITE-A / SITE-B config-map literals. Mode stays ${OTOPCUA_MESH_MODE:-ClusterClient}.)
• Modify: docs/Configuration.md (MeshTransport section: document Grpc mode + new keys as "shipped dark, cutover pending")
• Modify: docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md (Phase 0 row)

Also: file the Gitea issue for FleetStatusBroadcaster's central-pair-blind fleet view (design §3.8 — pre-existing, transport-independent, NOT fixed here; record the number in the tracking doc). Full-suite run; rig rebuild; confirm all six nodes boot with the new env present and mode unchanged (validators must not fire — keys are only required under Grpc). Commit chore(mesh): rig PSK config + docs stub.

Phase 0 DoD: suite green, 0 warnings; rig boots all six nodes on ClusterClient with the dark config present; both interceptors + provider + mapper fully unit-tested; FleetStatus issue filed; tracking doc updated. Live-gate doc Phase 0 section recorded (small gate: boot + validator-firing checks, including one deliberate misconfig — Mode=Grpc with no GrpcPsk on a scratch container refuses boot with the exact-key message). STOP — PR + merge are user-owned.

---

Phase 1A — Central→node command plane (worktree A, branch feat/grpc-node-command)

Task 1A.1: NodeCommandGrpcService

Classification: standard Estimated implement time: ~25 min Parallelizable with: Task 1B.1 (OTHER worktree only)

Files:

• Create: src/Server/ZB.MOM.WW.OtOpcUa.Host/Grpc/NodeCommandGrpcService.cs
• Modify: src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs:707-725 (map under hasDriver && telemetryListenPort > 0, beside TelemetryStreamGrpcService)
• Test: tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Grpc/NodeCommandGrpcServiceTests.cs

Follow the alias convention (using GeneratedServiceBase = …NodeCommandService.NodeCommandServiceBase;, class suffixed GrpcService). Execute: decode via MeshTransportDtoMapper.FromEnvelope, Tell the registered NodeCommunicationActor (resolve via NodeCommunicationActorKey registry, Runtime/ServiceCollectionExtensions.cs:583), return ExecuteAck. Actor not yet registered (boot window) ⇒ RpcException(Unavailable) — the readiness convention. Decode failure ⇒ InvalidArgument. Node side re-emits on EventStream exactly as under ClusterClient — the existing NodeCommunicationActor receive path handles the four types already; verify EventStreamCommandDeliveryTests pass unmodified (if they need edits, the seam is wrong). TDD with TestKit probe as the registered actor. Commit feat(mesh): NodeCommandService on the telemetry listener.

Task 1A.2: MeshGrpcCommandDialer + the third branch in CentralCommunicationActor

Classification: high-risk (actor + concurrency) Estimated implement time: ~40 min Parallelizable with: Task 1B.2 (OTHER worktree only)

Files:

• Create: src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/MeshGrpcCommandDialer.cs
• Modify: src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/CentralCommunicationActor.cs
• Test: tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication/CentralCommunicationActorTests.cs (new facts), .../MeshGrpcCommandDialerTests.cs

Dialer (plain class, injectable fake): DispatchAsync(nodeEndpoint, CommandEnvelope, ct) — GrpcChannel cached per endpoint (ConcurrentDictionary, IDisposable, the GrpcDeploymentArtifactFetcher.cs:138-152 shape), bearer from IMeshPskProvider for the target node's cluster + x-otopcua-cluster header, deadline via linked CTS (DispatchTimeoutSeconds). Actor changes: extend the contact-load query to also select GrpcPort (rows already loaded — LoadContactsFromDb, CentralCommunicationActor.cs:207-245); under Mode=Grpc, HandleMeshCommand maps the message via ToProto and fans DispatchAsync to every enabled non-maintenance node across all clusters (the SendToAll recipient set), PipeTo-ing per-node results back for logging only — a failure is logged with node id + status and dropped; no retry (design §3.3). Skip-with-warn a node whose GrpcPort is NULL (mirror TelemetryNodeSource:49-60). Wire IMeshPskProvider.Invalidate to the contact-refresh tick. New facts mirror the existing SendToAll facts (CentralCommunicationActorTests:142,191) with a fake dialer: fan set, per-node failure isolation (one dead node doesn't stop the fan), NULL-port skip, PSK-miss skip (fail-closed). Commit feat(mesh): gRPC command fan-out behind Mode=Grpc.

Task 1A.3: 1A live check (rig-only, nothing committed but the gate record)

Classification: small Estimated implement time: ~20 min (rig wall-time) Parallelizable with: none

Rebuild the rig from the branch. Flip central-1/-2 only to MeshTransport__Mode: "Grpc" (rig-only edit, reverted after). Sites stay ClusterClient. Run live-gate Phase 1A checks (deploy dispatch reaches all four site nodes over gRPC while acks return over ClusterClient — coexistence proven; RestartDriver/AlarmCommand from AdminUI; unauthenticated + wrong-key grpcurl probes against a site node's :4056 rejected). Record in the live-gate doc; revert the flip. Phase 1A DoD: suite green; gate section PASS; tracking updated. STOP.

---

Phase 1B — Node→central ack plane (worktree B, branch feat/grpc-central-ack)

Task 1B.1: CentralCommandGrpcService + admin validator rule

Classification: standard Estimated implement time: ~25 min Parallelizable with: Task 1A.1 (OTHER worktree only)

Files:

• Create: src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Grpc/CentralCommandGrpcService.cs (beside DeploymentArtifactService.cs)
• Modify: src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs:707-725 (map under hasAdmin && configServeGrpcPort > 0)
• Modify: src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptionsValidator.cs (admin role + Mode=Grpc ⇒ ConfigServe:GrpcListenPort > 0, cross-read from IConfiguration — the SplitTopology pattern)
• Test: tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/Grpc/CentralCommandGrpcServiceTests.cs, validator tests

ReportApply: decode ApplyAckMessage → Tell CentralCommunicationActor (resolve via CentralCommunicationActorKey registry, ControlPlane/ServiceCollectionExtensions.cs:246) → its existing HandleApplyAck Forwards to the coordinator proxy untouched. Not-ready ⇒ Unavailable; decode failure ⇒ InvalidArgument. Commit feat(mesh): CentralCommandService on the ConfigServe listener.

Task 1B.2: GrpcApplyAckSender + node-side wiring

Classification: high-risk (actor wiring; the Send-not-SendToAll single-count semantic) Estimated implement time: ~35 min Parallelizable with: Task 1A.2 (OTHER worktree only)

Files:

• Create: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Communication/GrpcApplyAckSender.cs
• Modify: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Communication/NodeCommunicationActor.cs
• Modify: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs:389-416,536
• Test: tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Communication/NodeCommunicationActorTests.cs (new facts), .../GrpcApplyAckSenderTests.cs

Sender: try CentralGrpcEndpoints in order, stop at first success (one ack per node — the Send semantic; the endpoint loop + per-endpoint channel cache of GrpcDeploymentArtifactFetcher:67-116), bearer = own GrpcPsk + x-otopcua-cluster = own clusterId (from RoleParser cluster role), deadline AckTimeoutSeconds; all endpoints fail ⇒ log Warning + drop (today's no-client semantic, now visible). NodeCommunicationActor gains an optional ackSender ctor param (null under Dps/ClusterClient — existing tests pass with ZERO edits; if they need edits, the seam is wrong); under Mode=Grpc the registration (ServiceCollectionExtensions.cs:389-416) creates no ClusterClient and passes the sender; _ackRouter wiring (:536) is unchanged (still the actor). New facts: gRPC branch sends exactly once on first-endpoint success, fails over to second, drops after both. Commit feat(mesh): gRPC ApplyAck path behind Mode=Grpc.

Task 1B.3: 1B live check (rig-only)

Classification: small Estimated implement time: ~20 min (rig wall-time) Parallelizable with: none

Rebuild from branch (after rebasing on merged 1A if it landed — expected union conflicts: Program.cs map block, compose, Configuration.md). Flip site-a-1/-2 only to Grpc. Deploy e2e: site-a acks arrive over gRPC (PersistNodeAck rows land, deployment seals), site-b still acks over ClusterClient — coexistence proven. Kill central-1 mid-deploy: site-a ack fails over to central-2's endpoint. Record; revert. Phase 1B DoD: suite green; gate PASS; tracking updated. STOP.

---

Phase 2 — Full-rig cutover + soak (sequential, after 1A+1B merged)

Targeted-test policy (phases 2–4): the full solution suite is large (~3,000+ tests, 40+ min wall). Per-task verification runs are TARGETED — run only the projects/filters the task touches (named per task below). ONE full-suite run happens per phase, at the phase DoD before the merge/STOP — never per task. The mesh-relevant targeted set, for reference:

# mesh transport core (actor, dialer, provider, starter order)
dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/ --filter "FullyQualifiedName~Communication"
# node-side actor + ack sender + registration wiring
dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/ --filter "FullyQualifiedName~Communication|FullyQualifiedName~ServiceCollectionExtensions"
# interceptors + gRPC services (both sides)
dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/
# options/validators
dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/ --filter "FullyQualifiedName~MeshTransport|FullyQualifiedName~SplitTopology"
# proto contract + mapper goldens
dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests/ --filter "FullyQualifiedName~MeshTransport"
# central ack ingress
dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/ --filter "FullyQualifiedName~CentralCommandGrpcService"

Task 2.0 (MUST FIX, opening task — before the flip and the soak): honest PreStart-race logging

Classification: small Estimated implement time: ~15 min Parallelizable with: none (first in phase)

Files:

• Modify: src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/CentralCommunicationActor.cs
• Test: tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication/ControlPlaneStarterOrderTests.cs, CentralCommunicationActorTests.cs (string/severity pins only)

Both live gates proved the PreStart resolve race fires on essentially EVERY admin boot (1A: 2/3; 1B: 4/4) and logs two FALSE Errors — Resolve's "continuing without it" (reads permanent; the memoize heals it at first use) and the I-1 guard's "known-cluster gate will NEVER be published" (it is published moments later — the 1B gate accepted acks and refused SITE-Z on every such boot). A healthy central currently fails any error-grep, including this phase's soak.

Required behavior after this task: a PreStart-time resolve failure logs at Debug/Info ("not yet resolvable — starter ordering; will retry on first use"); the Error fires only when a FIRST-USE retry also fails (the genuinely-permanent absence — Resolve is already re-invoked by the memoizing properties, so this is a severity/wording split by call site, not new machinery). The I-1 null-guard Errors in PreStart move to the same first-use semantics or gain "if this persists past the first contact refresh" wording. Net: a healthy admin boot produces ZERO Warning/Error lines from this path (pinned by a test asserting the boot-time severity), while a truly-absent dependency still produces a loud Error at first use (existing pins keep this). Targeted tests only: ControlPlaneStarterOrderTests + CentralCommunicationActorTests filters. Small review chain (code-reviewer). Commit fix(mesh): PreStart resolve race logs honestly — quiet on heal, loud on permanent absence. Lands on master directly (post-1B-merge, pre-flip).

The cutover

No feature branch; one commit flipping the rig default: docker-dev/docker-compose.yml MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}".

Before flipping, run the BASELINE on the pre-change build (live-gate doc has the exact commands): reproduce the two gaps this program closes — (a) a site container holds a live Akka association to central's :4053 (the unauthenticated cross-WAN surface), (b) stop the site-b pair, dispatch a deployment, observe the silent drop (no per-node error at dispatch; only the 2-min deadline marks it). Reproduced, not assumed.

Then: full rebuild all-Grpc; run the Phase 2 check table (deploy e2e all six nodes; Restart/Reconnect/AlarmCommand matrix; kill-central-1 ack failover; kill-site-node mid-dispatch = logged per-node failure + deadline backstop, no retry storm; PSK negatives against both services on running containers; zero ClusterClient log lines under Grpc; soak ≥ 2 h with 0 PermissionDenied noise — PSK drift detector; the soak error-grep must be clean, which Task 2.0 makes possible on a healthy rig). Per-check verification is on the RIG; unit runs during this phase are targeted (see policy above). Phase 2 DoD: every check PASS in the live-gate doc; ONE full-suite run green on the cutover SHA; rig default committed; tracking updated. STOP.

---

Phase 3 — Deletion + tightening + docs (branch feat/grpc-phase3-deletion)

Task 3.1: delete ClusterClient (code + HOCON + config)

Classification: high-risk (deletion ordering) Estimated implement time: ~40 min Parallelizable with: none

Files: Runtime/ServiceCollectionExtensions.cs, ControlPlane/ServiceCollectionExtensions.cs, ControlPlane/Communication/{CentralCommunicationActor.cs,IMeshClusterClientFactory.cs}, Runtime/Communication/NodeCommunicationActor.cs, Cluster/Resources/akka.conf, Cluster/MeshTransportOptions.cs + validator, docker-dev/docker-compose.yml, affected tests.

In order: (1) delete IMeshClusterClientFactory/DefaultMeshClusterClientFactory + central per-cluster client cache/RebuildClient (keep LoadContactsFromDb — it feeds the gRPC dialer); (2) delete node client creation + centralClient param; (3) delete both receptionist registrations; (4) akka.conf: remove ClusterClientReceptionistExtensionProvider from akka.extensions + the whole akka.cluster.client block (keep DPS extension + maximum-frame-size canary — pair-local channels live); (5) delete CentralContactPoints + validator rules + all six compose entries; (6) delete the ClusterClient mode value; SplitTopologyTransportValidator requires Grpc; (7) retire/ replace tests: MeshClusterClientBoundaryTests → MeshGrpcBoundaryTests (two clusters, gRPC between them, assert no cross-mesh Akka association forms — the program's point, kept as a test), MeshClusterClientFactoryTests deleted, MeshTransportHoconTests drops the client-settings asserts (keeps frame-size + DPS pins), CentralCommunicationActorTests / NodeCommunicationActorTests / MeshRouterDispatchTests reshaped to Dps+Grpc. Grep-gates: rg -i "clusterclient|receptionist" src tests docker-dev docs → only historical plans/CHANGELOG; rg "CentralContactPoints" → empty. Commit sequence per step. Targeted tests per step, NOT the full suite: steps 1–3 → the ControlPlane + Runtime Communication/ServiceCollectionExtensions filters; step 4 → MeshTransportHoconTests; steps 5–6 → the Cluster.Tests MeshTransport|SplitTopology filter + Host.Tests; step 7 → the replaced/reshaped fixtures by name. The FULL suite runs once, at the Phase 3 DoD.

Task 3.2: docs sweep

Classification: small Estimated implement time: ~20 min Parallelizable with: none

docs/Configuration.md (MeshTransport = Dps|Grpc, keys, PSK seeding), docs/Redundancy.md transport table, docs/Telemetry.md cross-refs, OtOpcUa/CLAUDE.md transport decisions, deferment.md if applicable — and the scadaproj index entry (~/Desktop/scadaproj/CLAUDE.md, committed on scadaproj main and pushed, per its index-edit rule). Update tracking doc. Task 3.2 itself is docs-only — no test runs beyond a build. Phase 3 DoD: ONE full-suite run green on the branch tip (the phase's only full run); grep-gates clean; docs merged. STOP.

---

Phase 4 — Terminal live gate (rig; sequential; every check PASS required)

Run on master AFTER Phase 3 merges, on the post-deletion image (verify image build timestamp > merge time — the ScadaBridge provenance guard). Eight checks, recorded in the live-gate doc: (1) PSK negatives both services incl. unset-key fail-closed; (2) command matrix all six nodes; (3) ack matrix + seal; (4) central-kill ack failover + site-node-kill dispatch honesty; (5) mid-deploy central kill → deadline/replay backstops hold, no dupes; (6) boundary check: no Akka association between any site container and central (was the baseline gap); (7) restart discipline: full-rig restart clean, zero receptionist/ClusterClient log lines anywhere; (8) MeshGrpcBoundaryTests (targeted) + the program's TERMINAL full-suite run, green on the merge SHA — this is Phase 4's one full run and the program's last; everything before it in this phase is rig checks + targeted fixtures. Close any carry-forwards named by earlier gates (incl. the I-4 cold-cache measurement if a store-backed rig exists by then — else re-carry with rationale). Tracking doc final row; program CLOSED.

---

Gotchas for the executor (will bite; read twice)

1. One public constructor per interceptor — Grpc.AspNetCore silently skips multi-ctor interceptors; correct/wrong/no key then fail identically (Unknown), which is indistinguishable from "working" in unit tests that register via DI. Reflection-pin it.
2. Enum zero is UNSPECIFIED, never a meaningful value — proto3 decodes absent as 0.
3. Do NOT touch the Kestrel block (Program.cs:564-671) — no new listeners exist in this plan; any deviation that adds one re-imports the explicit-endpoints-suppresses-URLS trap.
4. Fire-and-forget stays fire-and-forget — no retry on DeadlineExceeded or anything else; the deploy deadline + boot replay are the only recovery. Adding a retry turns one RestartDriver into two.
5. ConfigPublishCoordinator's deployment-acks DPS subscription stays — it is the Dps-mode ingress, not dead code.
6. SendToAll/Send semantics: fan = every enabled non-maintenance ClusterNode; ack = exactly once, first healthy endpoint. The coordinator counts acks — a double-send double-counts.
7. Existing tests are the seam oracle: EventStreamCommandDeliveryTests and the pre-existing NodeCommunicationActorTests/CentralCommunicationActorTests construction paths must pass unmodified through 1A/1B (nullable ctor params, never signature breaks). If they need edits before Phase 3, the seam is wrong.
8. Rig flips before Phase 2 are never committed; a redeploy from master must reset to all-ClusterClient until the Phase 2 commit.
9. x-otopcua-cluster is a central-side routing hint — do not add it to the node-side interceptor's requirements (theatre), and do not prove isolation with it (wrong-key rejection is the proof).
10. Secrets on driver-only nodes work (node-local SQLite + env KEK) — do not add a ConfigDb dependency anywhere on the node path; that constraint is the whole point of mesh Phases 3/4.

Effort & parallelization

Track	Est.	Parallel with
Phase 0	2–3 d	— (single worktree; tasks 0.1+0.3 and 0.4+0.5 batchable)
1A	3–4 d	1B (separate worktrees; 1A merges first, 1B rebases)
1B	2–3 d	1A
2	1–2 d	—
3	1–2 d	—
4	1 d	—

Critical path ≈ 0 → 1A → 2 → 3 → 4: ~1.5–2.5 weeks.