Joseph Doherty
29 KiB
ClusterClient → gRPC-only cross-mesh transport — implementation plan
For Claude: REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. Central tracking doc:
docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md— update it at every phase boundary and defect. Live gate results go indocs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md.
Goal: all central↔site mesh traffic rides gRPC with per-cluster PSK auth from
ZB.MOM.WW.Secrets; ClusterClient/ClusterClientReceptionist are deleted; Akka remoting
never crosses the mesh boundary. Design: 2026-08-06-clusterclient-to-grpc-design.md
(read it first — §4 holds the plan-vs-code corrections this plan builds on).
Architecture: one new proto (mesh_transport.v1) with two unary services;
NodeCommandService hangs off the existing Telemetry listener on driver nodes,
CentralCommandService off the existing ConfigServe listener on central. The gRPC path is a
third MeshTransport:Mode branch inside the two existing choke-point actors
(CentralCommunicationActor / NodeCommunicationActor) — no new transport interface, no new
ports, no new DB columns.
Tech Stack: .NET 10, Grpc.Tools 2.76 build-time codegen, Akka.NET 1.5.62,
ZB.MOM.WW.Configuration validators, ZB.MOM.WW.Secrets, xunit + Akka.TestKit.Xunit2.
Everything below was code-verified 2026-08-06. If a cited line has drifted, re-locate by the quoted identifier, never by line number.
How to execute this plan (read before starting)
- Phase order: 0 → (1A ∥ 1B) → 2 → 3 → 4. Phases marked ∥ are parallel-safe only in
separate git worktrees (
git worktree add ../OtOpcUa-1B feat/grpc-central-ack) — never run two agents against one working tree (destructive git races are a known family incident). Designated merge order when tracks meet: 1A lands first, 1B rebases (expected conflicts are confined toHost/Program.csservice-map blocks,docker-dev/docker-compose.ymlenv blocks, anddocs/Configuration.md— resolve as union). - Subagent model: dispatch each task to an Opus subagent (
model: "opus"). Within one phase/worktree, run tasks sequentially (they share files and git state). Across worktrees, run the 1A and 1B task streams concurrently — one Opus agent per worktree. Tasks whoseParallelizable with:names a task in the same worktree may be batched into a single agent dispatch, never into two agents on one tree. Review each task's diff before the next dispatch (superpowers-extended-cc:subagent-driven-development). - Branches:
feat/grpc-phase0-contract-auth,feat/grpc-node-command(1A),feat/grpc-central-ack(1B),feat/grpc-phase3-deletion. One PR per phase tomasterwith its DoD met; record PR number + merge SHA in the tracking doc. Stop at each phase DoD — PR + merge is user-owned unless told otherwise. - Build/test:
dotnet build ZB.MOM.WW.OtOpcUa.slnx(0 warnings — TreatWarningsAsErrors),dotnet test ZB.MOM.WW.OtOpcUa.slnx. TestKit tests inherit the existing harnesses (ControlPlaneTestHarness,RuntimeActorTestBase) — never hand-rollActorSystem.Create+join. - Proto codegen is build-time (
Grpc.Tools,<Protobuf>items insrc/Core/ZB.MOM.WW.OtOpcUa.Commons/ZB.MOM.WW.OtOpcUa.Commons.csproj) — no checked-in generated code. The docker image build is the cross-arch verification; if protoc ever misbehaves in the image, fall back to ScadaBridge's checked-in recipe and record it. - Rig:
docker compose -f docker-dev/docker-compose.yml up -d --build(6 host nodes: central-1/-2 =admin,driver,cluster-MAIN; site-a-1/-2, site-b-1/-2 = driver + cluster role; plus sql/traefik/migrator/cluster-seed). AdminUIhttp://localhost:9200(login disabled), headless deployPOST /api/deploymentswithX-Api-Key: docker-dev-deploy-key. gRPC ports are container-internal: ConfigServe:4055(central), Telemetry:4056(all driver nodes). Mesh mode env:MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-ClusterClient}"on all six nodes. - Coexistence rule: every migrated path sits behind
MeshTransport:ModewithClusterClientremaining the rig default until Phase 2 and the compiled default stayingDpsthroughout (it is the single-mesh/in-process default, not this program's concern). Rollback at any point before Phase 3 = flip the mode back. DoD-test flag flips on the rig are rig-only edits, reverted, never committed until Phase 2.
The two choke points (where all transport code changes)
- Central→node:
CentralCommunicationActor(src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/CentralCommunicationActor.cs) —HandleMeshCommand(:140-168) branches on mode; the gRPC branch fans unaryExecutecalls instead ofClusterClient.SendToAll. - Node→central:
NodeCommunicationActor(src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Communication/NodeCommunicationActor.cs) —HandleApplyAck(:78-92) branches; the gRPC branch callsReportApplyinstead ofClusterClient.Send. Publishers and subscribers above these actors are untouched (ConfigPublishCoordinator,AdminOperationsActor,DriverHostActor,ScriptedAlarmHostActor, the EventStream leg, the coordinatorForward).
Phase 0 — Contract + auth + config groundwork (standalone; no behavior change)
Branch feat/grpc-phase0-contract-auth. Everything here is additive and dark.
Task 0.1: mesh_transport.proto + contract lock
Classification: standard Estimated implement time: ~20 min Parallelizable with: Task 0.3 (same worktree — batch to one agent, do not split)
Files:
- Create:
src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos/mesh_transport.proto - Create:
src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos/MeshTransportProtoContract.cs - Modify:
src/Core/ZB.MOM.WW.OtOpcUa.Commons/ZB.MOM.WW.OtOpcUa.Commons.csproj(third<Protobuf Include="Protos\mesh_transport.proto" GrpcServices="Both"/>) - Test:
tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests/MeshTransportProtoContractTests.cs
Package mesh_transport.v1, csharp_namespace "ZB.MOM.WW.OtOpcUa.Commons.Protos.MeshTransport.V1"
(NOT MeshCommand — collides with Commons.Messages.Mesh.MeshCommand). Services + envelope per
design §3.2: NodeCommandService.Execute(CommandEnvelope) returns (ExecuteAck);
CentralCommandService.ReportApply(ApplyAckMessage) returns (ReportAck). CommandEnvelope has
correlation_id + oneof command of 4 messages mirroring DispatchDeployment
(Commons/Messages/Deploy/DispatchDeployment.cs), RestartDriver, ReconnectDriver
(Commons/Messages/Admin/), AlarmCommand (Commons/OpcUa/AlarmCommand.cs).
ApplyAckMessage mirrors ApplyAck (DeploymentId, NodeId, Outcome, FailureReason, CorrelationId). Every enum reserves 0 as *_UNSPECIFIED (AlarmCommand verb,
ApplyAck.Outcome) — proto3 zero must never decode to a meaningful value.
MeshTransportProtoContract.HandledCases mirrors TelemetryProtoContract.cs:18-24; the test
mirrors TelemetryProtoContractTests (fails when a oneof case is added but unhandled). TDD:
write the contract test red (proto absent), add proto + helper, green, commit
feat(mesh): mesh_transport.v1 proto contract.
Task 0.2: DTO mapper + round-trip goldens
Classification: high-risk (data contract — this is where ScadaBridge found five silent-drop bugs) Estimated implement time: ~30 min Parallelizable with: none (depends on 0.1)
Files:
- Create:
src/Core/ZB.MOM.WW.OtOpcUa.Commons/Messages/Mesh/MeshTransportDtoMapper.cs - Test:
tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests/MeshTransportDtoMapperTests.cs
Static mapper: ToProto/FromProto for all five messages + the envelope. Before writing it,
enumerate every member of the five records in the test first (golden round-trips: construct
each record with every property set to a non-default value, map to proto and back, assert
record equality; add a null-variant golden for every nullable member — FailureReason at
minimum). Guard against the known trap classes: init-only props (use object initializers in
FromProto), nullable strings where "" is not null (use proto3 optional), enum zero
(decode unknown → a Unknown CLR value or throw, never a meaningful default). Unknown oneof
case in FromProto ⇒ throw InvalidOperationException naming the case — the contract-lock
test ensures it's unreachable. Commit feat(mesh): DTO mapper with round-trip goldens.
Task 0.3: MeshTransportOptions + validators + SplitTopology relaxation
Classification: standard Estimated implement time: ~25 min Parallelizable with: Task 0.1 (same worktree — batch)
Files:
- Modify:
src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptions.cs - Modify:
src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptionsValidator.cs - Modify:
src/Core/ZB.MOM.WW.OtOpcUa.Cluster/SplitTopologyTransportValidator.cs - Test:
tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/MeshTransportOptionsValidatorTests.cs,.../SplitTopologyTransportValidatorTests.cs
Add ModeGrpc = "Grpc"; new members string[] CentralGrpcEndpoints, string? GrpcPsk,
int DispatchTimeoutSeconds = 10, int AckTimeoutSeconds = 10, and a config-map override
Dictionary<string,string> ClusterPsks (central-side rig/test override; production leaves it
empty and the runtime provider falls back to Secrets — mirrors ScadaBridge's
SitePsks__site-* rig pattern). Validator, following the house rules (roles via
RoleParser.ResolveEffectiveRoles from IConfiguration, keys required only where
load-bearing, messages name the exact key + env form MeshTransport__GrpcPsk, never echo
values): under Grpc + driver role ⇒ GrpcPsk non-empty, CentralGrpcEndpoints non-empty
and each http(s)://-prefixed, both timeouts > 0. SplitTopologyTransportValidator:101-109:
accept ClusterClient or Grpc (tighten to Grpc-only in Phase 3 — without this
relaxation a rig node flipped to Grpc refuses to boot). Also correct the three stale
"Phase 6 deletes the DPS branch" comments (MeshTransportOptions.cs:11-13,
ConfigPublishCoordinator.cs:70-73, AdminOperationsActor.cs:90-92) to match
docs/Configuration.md:150-156. TDD red-first on both validators. Commit
feat(mesh): Grpc mode value + validation.
Task 0.4: node-side MeshCommandAuthInterceptor
Classification: high-risk (security) Estimated implement time: ~20 min Parallelizable with: Task 0.5 (different files, same worktree — batch)
Files:
- Create:
src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/MeshCommandAuthInterceptor.cs - Modify:
src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs:550-562(add to thehasDriverinterceptor block) - Test:
tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Configuration/MeshCommandAuthInterceptorTests.cs
Clone TelemetryStreamAuthInterceptor.cs byte-for-byte in structure: ServicePrefix
/mesh_transport.v1.NodeCommandService/, all four *ServerHandler overrides → one
Authorize, case-insensitive authorization metadata walk, Bearer prefix,
CryptographicOperations.FixedTimeEquals over UTF-8, expected token =
IOptions<MeshTransportOptions>.GrpcPsk, fail-closed when unset (reject all with
actionable Warning), deny PermissionDenied, non-matching paths pass through. Exactly ONE
public constructor + a HasExactlyOnePublicConstructor reflection test (the
Grpc.AspNetCore multi-ctor silent-skip trap, TelemetryStreamAuthInterceptor.cs:38-42; cost
ScadaBridge a live-gate day). No x-otopcua-cluster requirement here — header-on-node-side is
theatre (design §3.4). Tests: no-token / wrong-token / correct-token / unset-key /
non-matching-path, mirroring TelemetryStreamAuthInterceptorTests. Commit
feat(mesh): node-side PSK interceptor.
Task 0.5: IMeshPskProvider + central-side CentralCommandAuthInterceptor
Classification: high-risk (security + the one new stateful component) Estimated implement time: ~35 min Parallelizable with: Task 0.4 (batch)
Files:
- Create:
src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/IMeshPskProvider.cs(+MeshPskProvider) - Create:
src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/CentralCommandAuthInterceptor.cs - Modify:
src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs(DI registration; add interceptor to thehasAdminblock) - Test:
tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication/MeshPskProviderTests.cs,tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Configuration/CentralCommandAuthInterceptorTests.cs
IMeshPskProvider.TryGetPsk(string clusterId, out string psk): check
MeshTransportOptions.ClusterPsks config map first, else resolve secret
OT-MESH-PSK-{clusterId} via ISecretResolver (the fail-closed lazy pattern the Galaxy
driver uses for secret: refs), cache per cluster, Invalidate(clusterId) /
InvalidateAll() hooks (wired to contact refresh in 1A). Fail-closed: unresolvable ⇒
false, caller denies/skips — never proceeds unauthenticated. Adversarial tests, not just
happy-path: unknown cluster, resolver throw, cache serves stale until invalidated, empty
secret value = failure. CentralCommandAuthInterceptor: ServicePrefix
/mesh_transport.v1.CentralCommandService/; reads x-otopcua-cluster, missing/unknown ⇒
PermissionDenied; verifies bearer against that cluster's PSK via the provider,
FixedTimeEquals; one public ctor + reflection test. Commit
feat(mesh): per-cluster PSK provider + central interceptor.
Task 0.6: rig config, issue filing, docs stub
Classification: small Estimated implement time: ~15 min Parallelizable with: none (last in phase; touches compose + docs)
Files:
- Modify:
docker-dev/docker-compose.yml(all six host nodes:MeshTransport__GrpcPskdev literalsdev-mesh-psk-MAIN/-SITE-A/-SITE-B;MeshTransport__CentralGrpcEndpoints__0/1=http://central-1:4055,http://central-2:4055; central pair additionallyMeshTransport__ClusterPsks__MAIN/SITE-A/SITE-Bconfig-map literals. Mode stays${OTOPCUA_MESH_MODE:-ClusterClient}.) - Modify:
docs/Configuration.md(MeshTransport section: documentGrpcmode + new keys as "shipped dark, cutover pending") - Modify:
docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md(Phase 0 row)
Also: file the Gitea issue for FleetStatusBroadcaster's central-pair-blind fleet view
(design §3.8 — pre-existing, transport-independent, NOT fixed here; record the number in the
tracking doc). Full-suite run; rig rebuild; confirm all six nodes boot with the new env
present and mode unchanged (validators must not fire — keys are only required under Grpc).
Commit chore(mesh): rig PSK config + docs stub.
Phase 0 DoD: suite green, 0 warnings; rig boots all six nodes on ClusterClient with the
dark config present; both interceptors + provider + mapper fully unit-tested; FleetStatus
issue filed; tracking doc updated. Live-gate doc Phase 0 section recorded (small gate: boot +
validator-firing checks, including one deliberate misconfig — Mode=Grpc with no GrpcPsk on
a scratch container refuses boot with the exact-key message). STOP — PR + merge are
user-owned.
Phase 1A — Central→node command plane (worktree A, branch feat/grpc-node-command)
Task 1A.1: NodeCommandGrpcService
Classification: standard Estimated implement time: ~25 min Parallelizable with: Task 1B.1 (OTHER worktree only)
Files:
- Create:
src/Server/ZB.MOM.WW.OtOpcUa.Host/Grpc/NodeCommandGrpcService.cs - Modify:
src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs:707-725(map underhasDriver && telemetryListenPort > 0, besideTelemetryStreamGrpcService) - Test:
tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Grpc/NodeCommandGrpcServiceTests.cs
Follow the alias convention (using GeneratedServiceBase = …NodeCommandService.NodeCommandServiceBase;,
class suffixed GrpcService). Execute: decode via MeshTransportDtoMapper.FromEnvelope, Tell
the registered NodeCommunicationActor (resolve via NodeCommunicationActorKey registry,
Runtime/ServiceCollectionExtensions.cs:583), return ExecuteAck. Actor not yet registered
(boot window) ⇒ RpcException(Unavailable) — the readiness convention. Decode failure ⇒
InvalidArgument. Node side re-emits on EventStream exactly as under ClusterClient — the
existing NodeCommunicationActor receive path handles the four types already; verify
EventStreamCommandDeliveryTests pass unmodified (if they need edits, the seam is wrong).
TDD with TestKit probe as the registered actor. Commit feat(mesh): NodeCommandService on the telemetry listener.
Task 1A.2: MeshGrpcCommandDialer + the third branch in CentralCommunicationActor
Classification: high-risk (actor + concurrency) Estimated implement time: ~40 min Parallelizable with: Task 1B.2 (OTHER worktree only)
Files:
- Create:
src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/MeshGrpcCommandDialer.cs - Modify:
src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/CentralCommunicationActor.cs - Test:
tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication/CentralCommunicationActorTests.cs(new facts),.../MeshGrpcCommandDialerTests.cs
Dialer (plain class, injectable fake): DispatchAsync(nodeEndpoint, CommandEnvelope, ct) —
GrpcChannel cached per endpoint (ConcurrentDictionary, IDisposable, the
GrpcDeploymentArtifactFetcher.cs:138-152 shape), bearer from IMeshPskProvider for the
target node's cluster + x-otopcua-cluster header, deadline via linked CTS
(DispatchTimeoutSeconds). Actor changes: extend the contact-load query to also select
GrpcPort (rows already loaded — LoadContactsFromDb, CentralCommunicationActor.cs:207-245);
under Mode=Grpc, HandleMeshCommand maps the message via ToProto and fans
DispatchAsync to every enabled non-maintenance node across all clusters (the SendToAll
recipient set), PipeTo-ing per-node results back for logging only — a failure is logged
with node id + status and dropped; no retry (design §3.3). Skip-with-warn a node whose
GrpcPort is NULL (mirror TelemetryNodeSource:49-60). Wire IMeshPskProvider.Invalidate
to the contact-refresh tick. New facts mirror the existing SendToAll facts
(CentralCommunicationActorTests:142,191) with a fake dialer: fan set, per-node failure
isolation (one dead node doesn't stop the fan), NULL-port skip, PSK-miss skip (fail-closed).
Commit feat(mesh): gRPC command fan-out behind Mode=Grpc.
Task 1A.3: 1A live check (rig-only, nothing committed but the gate record)
Classification: small Estimated implement time: ~20 min (rig wall-time) Parallelizable with: none
Rebuild the rig from the branch. Flip central-1/-2 only to MeshTransport__Mode: "Grpc"
(rig-only edit, reverted after). Sites stay ClusterClient. Run live-gate Phase 1A checks
(deploy dispatch reaches all four site nodes over gRPC while acks return over ClusterClient —
coexistence proven; RestartDriver/AlarmCommand from AdminUI; unauthenticated + wrong-key
grpcurl probes against a site node's :4056 rejected). Record in the live-gate doc; revert
the flip. Phase 1A DoD: suite green; gate section PASS; tracking updated. STOP.
Phase 1B — Node→central ack plane (worktree B, branch feat/grpc-central-ack)
Task 1B.1: CentralCommandGrpcService + admin validator rule
Classification: standard Estimated implement time: ~25 min Parallelizable with: Task 1A.1 (OTHER worktree only)
Files:
- Create:
src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Grpc/CentralCommandGrpcService.cs(besideDeploymentArtifactService.cs) - Modify:
src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs:707-725(map underhasAdmin && configServeGrpcPort > 0) - Modify:
src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptionsValidator.cs(admin role +Mode=Grpc⇒ConfigServe:GrpcListenPort > 0, cross-read fromIConfiguration— the SplitTopology pattern) - Test:
tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/Grpc/CentralCommandGrpcServiceTests.cs, validator tests
ReportApply: decode ApplyAckMessage → Tell CentralCommunicationActor (resolve via
CentralCommunicationActorKey registry, ControlPlane/ServiceCollectionExtensions.cs:246) →
its existing HandleApplyAck Forwards to the coordinator proxy untouched. Not-ready ⇒
Unavailable; decode failure ⇒ InvalidArgument. Commit feat(mesh): CentralCommandService on the ConfigServe listener.
Task 1B.2: GrpcApplyAckSender + node-side wiring
Classification: high-risk (actor wiring; the Send-not-SendToAll single-count semantic)
Estimated implement time: ~35 min
Parallelizable with: Task 1A.2 (OTHER worktree only)
Files:
- Create:
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Communication/GrpcApplyAckSender.cs - Modify:
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Communication/NodeCommunicationActor.cs - Modify:
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs:389-416,536 - Test:
tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Communication/NodeCommunicationActorTests.cs(new facts),.../GrpcApplyAckSenderTests.cs
Sender: try CentralGrpcEndpoints in order, stop at first success (one ack per node —
the Send semantic; the endpoint loop + per-endpoint channel cache of
GrpcDeploymentArtifactFetcher:67-116), bearer = own GrpcPsk + x-otopcua-cluster = own
clusterId (from RoleParser cluster role), deadline AckTimeoutSeconds; all endpoints fail ⇒
log Warning + drop (today's no-client semantic, now visible). NodeCommunicationActor gains
an optional ackSender ctor param (null under Dps/ClusterClient — existing tests pass
with ZERO edits; if they need edits, the seam is wrong); under Mode=Grpc the registration
(ServiceCollectionExtensions.cs:389-416) creates no ClusterClient and passes the sender;
_ackRouter wiring (:536) is unchanged (still the actor). New facts: gRPC branch sends
exactly once on first-endpoint success, fails over to second, drops after both. Commit
feat(mesh): gRPC ApplyAck path behind Mode=Grpc.
Task 1B.3: 1B live check (rig-only)
Classification: small Estimated implement time: ~20 min (rig wall-time) Parallelizable with: none
Rebuild from branch (after rebasing on merged 1A if it landed — expected union conflicts:
Program.cs map block, compose, Configuration.md). Flip site-a-1/-2 only to Grpc.
Deploy e2e: site-a acks arrive over gRPC (PersistNodeAck rows land, deployment seals),
site-b still acks over ClusterClient — coexistence proven. Kill central-1 mid-deploy: site-a
ack fails over to central-2's endpoint. Record; revert. Phase 1B DoD: suite green; gate
PASS; tracking updated. STOP.
Phase 2 — Full-rig cutover + soak (sequential, after 1A+1B merged)
No feature branch; one commit flipping the rig default:
docker-dev/docker-compose.yml MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}".
Before flipping, run the BASELINE on the pre-change build (live-gate doc has the exact
commands): reproduce the two gaps this program closes — (a) a site container holds a live Akka
association to central's :4053 (the unauthenticated cross-WAN surface), (b) stop the site-b
pair, dispatch a deployment, observe the silent drop (no per-node error at dispatch; only the
2-min deadline marks it). Reproduced, not assumed.
Then: full rebuild all-Grpc; run the Phase 2 check table (deploy e2e all six nodes;
Restart/Reconnect/AlarmCommand matrix; kill-central-1 ack failover; kill-site-node
mid-dispatch = logged per-node failure + deadline backstop, no retry storm; PSK negatives
against both services on running containers; zero ClusterClient log lines under Grpc;
soak ≥ 2 h with 0 PermissionDenied noise — PSK drift detector). Phase 2 DoD: every
check PASS in the live-gate doc; rig default committed; tracking updated. STOP.
Phase 3 — Deletion + tightening + docs (branch feat/grpc-phase3-deletion)
Task 3.1: delete ClusterClient (code + HOCON + config)
Classification: high-risk (deletion ordering) Estimated implement time: ~40 min Parallelizable with: none
Files: Runtime/ServiceCollectionExtensions.cs, ControlPlane/ServiceCollectionExtensions.cs,
ControlPlane/Communication/{CentralCommunicationActor.cs,IMeshClusterClientFactory.cs},
Runtime/Communication/NodeCommunicationActor.cs, Cluster/Resources/akka.conf,
Cluster/MeshTransportOptions.cs + validator, docker-dev/docker-compose.yml, affected tests.
In order: (1) delete IMeshClusterClientFactory/DefaultMeshClusterClientFactory + central
per-cluster client cache/RebuildClient (keep LoadContactsFromDb — it feeds the gRPC
dialer); (2) delete node client creation + centralClient param; (3) delete both
receptionist registrations; (4) akka.conf: remove ClusterClientReceptionistExtensionProvider
from akka.extensions + the whole akka.cluster.client block (keep DPS extension +
maximum-frame-size canary — pair-local channels live); (5) delete
CentralContactPoints + validator rules + all six compose entries; (6) delete the
ClusterClient mode value; SplitTopologyTransportValidator requires Grpc; (7) retire/
replace tests: MeshClusterClientBoundaryTests → MeshGrpcBoundaryTests (two clusters, gRPC
between them, assert no cross-mesh Akka association forms — the program's point, kept as a
test), MeshClusterClientFactoryTests deleted, MeshTransportHoconTests drops the
client-settings asserts (keeps frame-size + DPS pins), CentralCommunicationActorTests /
NodeCommunicationActorTests / MeshRouterDispatchTests reshaped to Dps+Grpc. Grep-gates:
rg -i "clusterclient|receptionist" src tests docker-dev docs → only historical
plans/CHANGELOG; rg "CentralContactPoints" → empty. Commit sequence per step.
Task 3.2: docs sweep
Classification: small Estimated implement time: ~20 min Parallelizable with: none
docs/Configuration.md (MeshTransport = Dps|Grpc, keys, PSK seeding), docs/Redundancy.md
transport table, docs/Telemetry.md cross-refs, OtOpcUa/CLAUDE.md transport decisions,
deferment.md if applicable — and the scadaproj index entry (~/Desktop/scadaproj/CLAUDE.md,
committed on scadaproj main and pushed, per its index-edit rule). Update tracking doc.
Phase 3 DoD: suite green; grep-gates clean; docs merged. STOP.
Phase 4 — Terminal live gate (rig; sequential; every check PASS required)
Run on master AFTER Phase 3 merges, on the post-deletion image (verify image build
timestamp > merge time — the ScadaBridge provenance guard). Eight checks, recorded in the
live-gate doc: (1) PSK negatives both services incl. unset-key fail-closed; (2) command
matrix all six nodes; (3) ack matrix + seal; (4) central-kill ack failover + site-node-kill
dispatch honesty; (5) mid-deploy central kill → deadline/replay backstops hold, no dupes;
(6) boundary check: no Akka association between any site container and central (was the
baseline gap); (7) restart discipline: full-rig restart clean, zero
receptionist/ClusterClient log lines anywhere; (8) MeshGrpcBoundaryTests + full suite
green on the merge SHA. Close any carry-forwards named by earlier gates. Tracking doc final
row; program CLOSED.
Gotchas for the executor (will bite; read twice)
- One public constructor per interceptor —
Grpc.AspNetCoresilently skips multi-ctor interceptors; correct/wrong/no key then fail identically (Unknown), which is indistinguishable from "working" in unit tests that register via DI. Reflection-pin it. - Enum zero is
UNSPECIFIED, never a meaningful value — proto3 decodes absent as 0. - Do NOT touch the Kestrel block (
Program.cs:564-671) — no new listeners exist in this plan; any deviation that adds one re-imports the explicit-endpoints-suppresses-URLS trap. - Fire-and-forget stays fire-and-forget — no retry on
DeadlineExceededor anything else; the deploy deadline + boot replay are the only recovery. Adding a retry turns oneRestartDriverinto two. ConfigPublishCoordinator'sdeployment-acksDPS subscription stays — it is theDps-mode ingress, not dead code.SendToAll/Sendsemantics: fan = every enabled non-maintenanceClusterNode; ack = exactly once, first healthy endpoint. The coordinator counts acks — a double-send double-counts.- Existing tests are the seam oracle:
EventStreamCommandDeliveryTestsand the pre-existingNodeCommunicationActorTests/CentralCommunicationActorTestsconstruction paths must pass unmodified through 1A/1B (nullable ctor params, never signature breaks). If they need edits before Phase 3, the seam is wrong. - Rig flips before Phase 2 are never committed; a redeploy from
mastermust reset to all-ClusterClient until the Phase 2 commit. x-otopcua-clusteris a central-side routing hint — do not add it to the node-side interceptor's requirements (theatre), and do not prove isolation with it (wrong-key rejection is the proof).- Secrets on driver-only nodes work (node-local SQLite + env KEK) — do not add a ConfigDb dependency anywhere on the node path; that constraint is the whole point of mesh Phases 3/4.
Effort & parallelization
| Track | Est. | Parallel with |
|---|---|---|
| Phase 0 | 2–3 d | — (single worktree; tasks 0.1+0.3 and 0.4+0.5 batchable) |
| 1A | 3–4 d | 1B (separate worktrees; 1A merges first, 1B rebases) |
| 1B | 2–3 d | 1A |
| 2 | 1–2 d | — |
| 3 | 1–2 d | — |
| 4 | 1 d | — |
Critical path ≈ 0 → 1A → 2 → 3 → 4: ~1.5–2.5 weeks.