lmxopcua/docs/plans/2026-08-06-clusterclient-to-grpc-design.md

# OtOpcUa: migrating cross-mesh ClusterClient traffic to gRPC-only

**Date:** 2026-08-06. Code-verified inventory of every ClusterClient touchpoint in
`~/Desktop/OtOpcUa`, then the migration assessment. **Reference implementation:** ScadaBridge
shipped the identical migration 2026-07-22→08-01 (design
`~/Desktop/scadaproj/scadabridge_clusterclient_to_grpc.md`, plan + live gate under
`~/Desktop/ScadaBridge/docs/plans/2026-07-22-clusterclient-to-grpc-*`) — live-gated end to end,
ClusterClient deleted. OtOpcUa runs on the same cluster boxes, so this migration converges both
apps on one WAN boundary posture: gRPC + PSK, no Akka remoting between site and central.

Companion docs: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (why the boundary is
per-cluster meshes), `2026-07-22-mesh-phase2-clusterclient-transport.md` (how ClusterClient got
here), `2026-07-24-mesh-phase6-partition-and-colocation.md` (the mesh split this completes).

Everything below was code-verified 2026-08-06. If a cited line has drifted, re-locate by the
quoted identifier, never by line number.

---

## 1. What ClusterClient actually carries today

Everything crossing the central↔site mesh boundary over Akka funnels through **two actors** —
`CentralCommunicationActor` (central) and `NodeCommunicationActor` (node) — so the migration
surface is even better-bounded than ScadaBridge's was.

**Receptionist registrations (2):**

| Role | Actor | Path | File:line | Note |
|---|---|---|---|---|
| admin | `CentralCommunicationActor` | `/user/central-communication` | `ControlPlane/ServiceCollectionExtensions.cs:85` | registered unconditionally in both modes (deliberate) |
| driver | `NodeCommunicationActor` | `/user/node-communication` | `Runtime/ServiceCollectionExtensions.cs:416` | same |

Path constants: `Commons/Messages/Mesh/MeshPaths.cs`. Receptionist + DPS extensions are
force-started via `Cluster/Resources/akka.conf:16-24`; receptionist role scoping deliberately
empty (`akka.conf:113-119`).

**Client creation (2 sites, asymmetric discovery — documented `Cluster/MeshTransportOptions.cs:16-21`):**

- **Central:** one `ClusterClient` **per application Cluster**, contacts loaded **from the
  database** (`ClusterNode` rows `WHERE Enabled AND NOT MaintenanceMode`, grouped by
  `ClusterId`, `akka.tcp://…/system/receptionist` built from `Host`+`AkkaPort`) —
  `CentralCommunicationActor.cs:207-245`, refreshed every `MeshTransport:ContactRefreshSeconds`
  (default 60). Factory seam `IMeshClusterClientFactory` /
  `DefaultMeshClusterClientFactory` (`ControlPlane/Communication/IMeshClusterClientFactory.cs`).
- **Node:** one `central-cluster-client`, contacts from **static config**
  `MeshTransport:CentralContactPoints[]` — `Runtime/ServiceCollectionExtensions.cs:394-407`,
  created only under `MeshTransport:Mode=ClusterClient`.

**Central → node (4 message types), all `ClusterClient.SendToAll("/user/node-communication", …)`
fanned across every per-cluster client** (`CentralCommunicationActor.cs:167-168`; `SendToAll`
not `Send` is load-bearing — `Send` would deploy to one node only). Node side re-emits on the
local `EventStream` (`NodeCommunicationActor.cs:70-74`); subscribers (`DriverHostActor`,
`ScriptedAlarmHostActor`) are transport-blind.

| Message | Purpose | Sender | Backstop layered on top |
|---|---|---|---|
| `DispatchDeployment` | "revision exists — fetch + apply" (payload-free: id + `RevisionHash` + correlation; artifact bytes ride gRPC config-serve) | `ConfigPublishCoordinator` (admin singleton), `:198` | 2-min apply deadline (`DefaultApplyDeadline`, `:60`, handling `:283-300`) + boot replay of orphan `Applying` rows in `DriverHostActor.Bootstrap()` |
| `RestartDriver` | AdminUI restart of one driver instance | `AdminOperationsActor.cs:431` | none — local `RestartDriverResult` answers the UI *before/independent of* node delivery |
| `ReconnectDriver` | AdminUI reconnect | `AdminOperationsActor.cs:461` | none |
| `AlarmCommand` | operator Ack/Shelve/Unshelve into the scripted-alarm engine | `AdminOperationsActor.cs:122,169` | none |

**Node → central (1 message type), `ClusterClient.Send("/user/central-communication", …)`**
(`Send` not `SendToAll` is load-bearing — `SendToAll` would double-count the node at the
coordinator, `NodeCommunicationActor.cs:89-92`):

| Message | Purpose | Path |
|---|---|---|
| `ApplyAck` (`DeploymentId, NodeId, Outcome, FailureReason, CorrelationId`) | per-node deploy acknowledgement | `DriverHostActor.SendAck` (`:2488-2507`) → `_ackRouter` = `NodeCommunicationActor` → central `HandleApplyAck` (`:171-187`) → `Forward` to the `ConfigPublishCoordinator` singleton proxy → `PersistNodeAck` (`:247-260`) |

**The invariant that sizes this whole migration: there is NO Ask across this boundary.** Every
message is a one-way Tell; no sender preservation, no reply futures, no Ask timeouts
(`NodeCommunicationActor.cs:30-35` states it; code confirms it). The per-cluster mesh design
doc's line about "sender preservation is the whole Ask idiom" describes ScadaBridge, not this
codebase — do not size off it.

**Transport guarantees today: none.** `akka.cluster.client.buffer-size = 0` (no buffering, no
replay), `reconnect-timeout = off` (`akka.conf:106,111`). No client for a cluster ⇒ the command
is logged and dropped (`CentralCommunicationActor.cs:146-158`); no central client ⇒ the ack is
dropped (`NodeCommunicationActor.cs:78-87`). Serialization is Akka's default Newtonsoft JSON —
**no `serialization-bindings` exist for any mesh message** — under a 128 KB
`maximum-frame-size` with drop-don't-teardown semantics (`akka.conf:35-47`). There is no proto
contract and no serializer discipline to inherit; the wire surface must be authored fresh.

**Not ClusterClient (stays untouched):** gRPC config-serve (Phase 3), gRPC telemetry dial
(Phase 5), LocalDb sync gRPC, Akka secrets replication (pair-local), the pair-local DPS
channels (`alarm-commands`' OPC-UA-originated leg published by `OtOpcUaServerHostedService`,
`redundancy-state`, `fleet-status`), and the node-local EventStream delivery leg.

---

## 2. Why gRPC-only is a natural completion, not a rewrite

1. **ClusterClient is the last Akka hop crossing the WAN.** Config artifacts (Phase 3),
   telemetry (Phase 5) and LocalDb sync already ride gRPC across the boundary. After this
   migration, Akka remoting is pair-internal only — the exact end state ScadaBridge reached.
2. **The surface is tiny.** 5 one-way message types, 2 choke-point actors, no Ask/reply
   semantics, no ordering or delivery guarantees to preserve beyond "fan to all driver nodes"
   and "one ack per node." ScadaBridge migrated 35 message types with Ask semantics in ~5
   weeks; this is a fraction of that.
3. **The playbook is proven on the same clusters.** ScadaBridge's design, PSK pattern,
   coexistence flags, deletion sequencing and live-gate format all shipped and survived their
   gates. OtOpcUa co-deploys on the same boxes; converging both apps on one boundary posture
   (gRPC + per-relationship PSK, no cross-WAN Akka) halves the firewall/audit story.
4. **Auth is a real gap, not hygiene.** Cross-mesh commands (`RestartDriver`, `AlarmCommand`,
   `DispatchDeployment`) ride unauthenticated Akka remoting today — anyone who can reach a
   node's `:4053` and speak the protocol can reach `/user/node-communication`. The gRPC path
   inherits the family's fail-closed bearer/PSK discipline that already gates the other three
   channels.
5. **Silent drop becomes visible failure.** `buffer-size = 0` means an undeliverable command
   vanishes with one central-side log line. A unary gRPC call fails *per node, with a status
   code* — the deploy path's 2-min deadline stops being the only witness.

---

## 3. What the migration takes

### 3.1 Topology — both directions already have a listener to hang off

This is the OtOpcUa-shaped economy (ScadaBridge had to build central's listener from scratch;
OtOpcUa does not):

- **`NodeCommandService`** (central→node commands) is mapped on the **existing Telemetry
  listener** (`Telemetry:GrpcListenPort`, h2c, rig `:4056`) that every driver node already
  hosts. On a split-topology node, `SplitTopologyTransportValidator` already forces
  `Telemetry:Mode=Grpc`, whose validator forces `GrpcListenPort > 0` — the listener is
  guaranteed up.
- **`CentralCommandService`** (node→central acks) is mapped on the **existing ConfigServe
  listener** (`ConfigServe:GrpcListenPort`, rig `:4055`) that admin nodes host. A new validator
  rule ties `MeshTransport:Mode=Grpc` + admin role to `ConfigServe:GrpcListenPort > 0`.

**Rejected: a dedicated mesh-command port.** It would need a new `ClusterNode` column + EF
migration + the seed-backfill `UPDATE` dance (`docker-dev/seed/seed-clusters.sql:126-128`
exists precisely because Phase 5's column landed NULL on persisted volumes) + compose churn on
six services — for no isolation gain, because the interceptors path-scope shared listeners
anyway (three services already coexist in one `AddGrpc` pipeline).

**Rejected: a node-held long-lived command stream** (node subscribes, central pushes). It
re-implements the connection/reconnect management the receptionist gave for free, inverts the
existing dial direction convention (central already dials nodes for telemetry), and buys
nothing for 4 command types. Same reasoning ScadaBridge recorded for the mirror-image option.

**Either-pair-node delivery is already safe.** Acks land on whichever central answers; the
receiving `CentralCommandService` Tells `CentralCommunicationActor`, which `Forward`s to the
`ConfigPublishCoordinator` **singleton proxy** — same intra-pair routing that makes today's
receptionist arrival-node-agnostic. Commands land on the *specific* node dialed, which is the
required semantic (each node must apply its own deployment).

### 3.2 Contract

One new proto: `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos/mesh_transport.proto`, package
`mesh_transport.v1`, csharp_namespace `ZB.MOM.WW.OtOpcUa.Commons.Protos.MeshTransport.V1`
(deliberately NOT `MeshCommand` — that C# name is taken by the existing envelope record
`Commons.Messages.Mesh.MeshCommand`).

```proto
service NodeCommandService {
  // Central → node. One-way semantics: the ack means "received and re-emitted on the
  // node's EventStream", nothing more. No retry is layered on a failure.
  rpc Execute (CommandEnvelope) returns (ExecuteAck);
}
service CentralCommandService {
  // Node → central. One-way semantics: the ack means "handed to the coordinator proxy".
  rpc ReportApply (ApplyAckMessage) returns (ReportAck);
}
message CommandEnvelope {
  string correlation_id = 1;
  oneof command {
    DispatchDeploymentCommand dispatch_deployment = 2;
    RestartDriverCommand      restart_driver      = 3;
    ReconnectDriverCommand    reconnect_driver    = 4;
    AlarmCommandMessage       alarm_command       = 5;
  }
}
```

Codegen is **build-time `Grpc.Tools`** exactly like the two existing protos
(`Commons.csproj:28-31` `<Protobuf … GrpcServices="Both"/>`; generated code is NOT checked in —
OtOpcUa does not have ScadaBridge's linux_arm64 protoc segfault; the rig image build is the
verification). Add a `MeshTransportProtoContract` contract-lock helper + tests mirroring
`TelemetryProtoContract.cs` / `TelemetryProtoContractTests.cs` so a new oneof case cannot ship
unhandled. Mapper: `MeshTransportDtoMapper` in Commons with **round-trip golden tests** for all
five messages (the ScadaBridge T1A.1/T1B.1 lesson: nullable members, init-only props and
enum-zero placement are where mappers silently drop data — reserve proto enum 0 as
`UNSPECIFIED`, never a meaningful value).

### 3.3 Semantics to preserve (all transport-agnostic, all carry over)

- **Fire-and-forget stays fire-and-forget.** No retries, no queues, no new delivery
  guarantees. A failed `Execute` is logged per node with its status code and dropped — an
  honest upgrade over the silent drop, not a semantic change. The deploy path's backstops
  (2-min deadline, orphan-`Applying` boot replay) remain the only recovery mechanisms.
- **`SendToAll` fan ⇒ dial every enabled, non-maintenance `ClusterNode`.** The recipient set is
  already DB-derived (`DiscoverDriverNodes` derives the expected-ack set from the same rows) —
  the gRPC dispatcher uses the same query shape, so dispatch set == expected-ack set by
  construction.
- **`Send` singleton-counting ⇒ one `ReportApply` per node per dispatch.** Unary gives this
  naturally; the node tries central endpoints in order and stops at the first success.
- **Node-side delivery stays the EventStream.** The gRPC service Tells
  `NodeCommunicationActor` the decoded message; the actor's existing re-emit (`:70-74`) is
  untouched, so `EventStreamCommandDeliveryTests` and every subscriber remain valid.
- **Central-side ack handling stays `HandleApplyAck`.** The gRPC service Tells
  `CentralCommunicationActor`; the `Forward`-to-proxy and `PersistNodeAck` paths are untouched.
- **Fused nodes work unchanged.** central-1/-2 are `admin,driver,cluster-MAIN`: they serve
  both services, dial their own pair with the MAIN key, and ack to themselves. Nothing special.

### 3.4 Work you must do anyway: auth — per-cluster PSK from the Secrets store (decided 2026-08-06)

The command plane is unauthenticated today. The gRPC migration adopts ScadaBridge's
live-gate-proven shape, adapted to OtOpcUa's conventions:

- **Key scope: one PSK per application Cluster** — `OT-MESH-PSK-<clusterId>` (e.g.
  `OT-MESH-PSK-MAIN`, `OT-MESH-PSK-SITE-A`), covering both directions of that cluster's
  traffic. **Steelmanned alternative — one fleet-wide shared key** (what the other three
  OtOpcUa channels do): simpler seeding, but a compromised site node could then issue
  `RestartDriver`/`AlarmCommand` to *every other site* and forge acks fleet-wide. ScadaBridge
  rejected the shared key on exactly this blast-radius argument; the command plane is where it
  bites hardest, so we do too. (The three existing shared-bearer channels are read/telemetry
  planes; retrofitting them is out of scope.)
- **Node side (one key):** new option `MeshTransport:GrpcPsk`, production value
  `${secret:OT-MESH-PSK-<clusterId>}`, resolved by the **existing pre-host
  `SecretReferenceExpander`** (`Host/Program.cs:93-112`) — zero new resolution code. Works on
  driver-only nodes (node-local SQLite secrets store + env KEK; no ConfigDb involved).
- **Central dial + serve sides (many keys, cluster set is DB-dynamic):** new
  `IMeshPskProvider` (ControlPlane) over the runtime `ISecretResolver`, resolving
  `OT-MESH-PSK-{clusterId}` at channel-build/verify time, **cached per cluster, invalidated on
  contact refresh, fail-closed** (unresolvable ⇒ deny/skip, never proceed unauthenticated).
  A `MeshTransport:ClusterPsks:{clusterId}` config map is checked before the Secrets store —
  the rig/test override, mirroring ScadaBridge's `SitePsks__site-*` env pattern; production
  leaves it empty.
- **Node-side server interceptor** `MeshCommandAuthInterceptor`, cloned from
  `TelemetryStreamAuthInterceptor`: path prefix `/mesh_transport.v1.NodeCommandService/`,
  `authorization: Bearer` extraction, `CryptographicOperations.FixedTimeEquals` over UTF-8,
  **fail-closed when `GrpcPsk` unset**, reject `PermissionDenied` (the 2-of-3 majority
  convention; `ConfigServeAuthInterceptor`'s `Unauthenticated` is the outlier — don't copy it).
  **Exactly ONE public constructor** — `Grpc.AspNetCore` silently stops invoking an
  interceptor with two (documented `TelemetryStreamAuthInterceptor.cs:38-42`, pinned by
  `HasExactlyOnePublicConstructor`; ScadaBridge lost a live-gate day to this).
- **Central-side server interceptor** `CentralCommandAuthInterceptor`: gates
  `/mesh_transport.v1.CentralCommandService/`; callers send `x-otopcua-cluster: <clusterId>`
  so central picks which key to verify; missing/unknown cluster ⇒ `PermissionDenied`
  (fail-closed). **The header is a central-side routing hint only** — per-cluster isolation is
  proven by wrong-key rejection, not by the header; do not add a header requirement to the
  node-side interceptor (ScadaBridge's gate recorded that as theatre).
- **Client attach:** raw `Metadata` `authorization: Bearer <psk>` per call, the exact pattern
  of `GrpcDeploymentArtifactFetcher.cs:122` / `TelemetryStreamClient.cs:134`.
- **Boot validation:** `MeshTransportOptionsValidator` extended per the house rules (roles from
  `IConfiguration` via `RoleParser.ResolveEffectiveRoles` — never `IClusterRoleInfo`; require
  keys only where the mode makes them load-bearing; error messages name the exact key and env
  form, never echo the value). Under `Mode=Grpc`: driver role ⇒ `GrpcPsk` non-empty +
  `CentralGrpcEndpoints` non-empty and `http(s)://`-prefixed.
- **Transport honesty:** bearer PSK over plaintext h2c is readable/replayable on-path — same
  accepted posture (network isolation) as the three existing channels, and strictly better
  than today's unauthenticated remoting. TLS on the internal listeners is follow-on hardening;
  the PSK design doesn't change if it lands.
- **Seeding/rotation:** the `secret` CLI's *Reference audit & seed* flow per store; rotation =
  set new value both sides, restart the pair (pairs restart together by discipline). Rig uses
  committed dev literals per its zero-operator-step convention: `dev-mesh-psk-MAIN`,
  `dev-mesh-psk-SITE-A`, `dev-mesh-psk-SITE-B`.

### 3.5 Endpoint discovery — both patterns already exist

- **Central → node:** `ClusterNode.Host` + `GrpcPort` ⇒ `http://{host}:{port}` — the exact
  `TelemetryNodeSource` pattern, same `Enabled && !MaintenanceMode` filter, refreshed on the
  existing `MeshTransport:ContactRefreshSeconds` timer that today refreshes Akka contacts. **No
  new column, no migration, no seed change.**
- **Node → central:** new `MeshTransport:CentralGrpcEndpoints[]` (e.g.
  `["http://central-1:4055", "http://central-2:4055"]`), tried in order per ack — the
  `GrpcDeploymentArtifactFetcher` endpoint-failover loop shape, channels cached per endpoint.
  It replaces `CentralContactPoints` (deleted in Phase 3) rather than reusing
  `ConfigSource:CentralFetchEndpoints`, because a `Direct`-mode fused node legitimately has no
  ConfigSource section but still acks deployments.
- **Deadlines are net-new** (no Ask timeouts existed): `MeshTransport:DispatchTimeoutSeconds`
  (default 10) on `Execute`, `MeshTransport:AckTimeoutSeconds` (default 10) on `ReportApply`,
  enforced client-side via linked CTS like the fetcher. Never retry `DeadlineExceeded` — with
  one-way semantics a late-delivered command is worse than a dropped one only if we retry it
  into a duplicate.

### 3.6 The mode switch — third value in, one value out

`MeshTransport:Mode` gains **`Grpc`**. End state: `Dps` (compiled default; in-mesh/single-mesh
topologies where admin and driver share one gossip ring) | `Grpc` (split topologies).
`ClusterClient` is **deleted**. The DPS branch is deliberately **retained**: the
"Phase 6 deletes the DistributedPubSub branch" comments
(`MeshTransportOptions.cs:11-13`, `ConfigPublishCoordinator.cs:70-73`,
`AdminOperationsActor.cs:90-92`) describe a deletion that never happened and contradict
`docs/Configuration.md:150-156`, which records the deliberate reversal — the comments get
corrected in Phase 0. `SplitTopologyTransportValidator` accepts `{ClusterClient, Grpc}` during
the migration (Phase 0 relaxation — without it, flipping any rig node to `Grpc` fails boot) and
requires `Grpc` from Phase 3.

The coexistence granularity falls out of where the switch is read: **central's mode governs the
command direction, each node's mode governs its ack direction** — so a single site pair can be
flipped to gRPC acks while everything else stays on ClusterClient, and vice versa. (Better than
ScadaBridge's central-wide `SiteTransport`, whose per-site coexistence gate had to be recorded
as unachievable.)

### 3.7 Deletions at the end

`DefaultMeshClusterClientFactory` + `IMeshClusterClientFactory`, both receptionist
registrations, node client creation (`Runtime/ServiceCollectionExtensions.cs:394-407`) and
central `RebuildClient`/per-cluster client cache (**keep `LoadContactsFromDb` — it now feeds
the gRPC dialer's node list**), `akka.conf`'s `ClusterClientReceptionistExtensionProvider`
line + the whole `akka.cluster.client` block (**keep the DPS extension** — pair-local channels
live), `MeshTransport:CentralContactPoints` + its validator rules + rig env, the
`ClusterClient` mode value, and the ClusterClient-shaped tests
(`MeshClusterClientBoundaryTests` is *replaced* by a gRPC-boundary equivalent — the "no Akka
association crosses the boundary" assertion is the point of the whole program and must survive
as a test). `Akka.Cluster.Tools` package refs **stay** (ClusterSingleton + DPS).

### 3.8 Deliberately out of scope

- **`FleetStatusBroadcaster` is silently central-pair-blind post-Phase-6** (derives "fleet"
  from central's own gossip ring). Real, pre-existing, transport-independent — filed as its own
  issue in Phase 0, not folded in.
- **Cross-mesh `redundancy-state` / `fleet-status` visibility** — deferred by Phase 5, stays
  deferred; this migration adds no new cross-mesh channels.
- **`alarm-commands`' pair-local DPS leg** (OPC UA Part 9 method calls published node-locally
  by `OtOpcUaServerHostedService:158-159`) — untouched; it never crosses meshes.
- **Delivery-guarantee upgrades** (queuing, retry, dedup) — explicitly not this program.
- **Retrofitting per-cluster PSKs onto the three existing shared-bearer channels** — follow-on.

---

## 4. Code-verified findings this design is sized on (plan-vs-code corrections)

Baked in at authoring time rather than discovered mid-implementation (the ScadaBridge §7
lesson, run as a three-track inventory before writing this doc):

1. **No Ask exists across the boundary** — the per-cluster mesh design doc's sender-preservation
   paragraph describes ScadaBridge. OtOpcUa's surface is 4+1 one-way Tells. (Also: its line
   listing "queries, subscribe handshakes" over ClusterClient — none exist.)
2. **"Phase 6 deletes the DPS branch" never happened** and the compiled default is still `Dps`;
   three source comments assert the opposite of `docs/Configuration.md:150-156`. This design
   keeps DPS for in-mesh topologies and deletes only `ClusterClient`; the comments get fixed.
3. **There is no `ICommandTransport` seam.** The seam is the `MeshCommand` envelope + the
   nullable `meshRouter`/`centralClient`/`ackRouter` ctor params. The gRPC branch lands inside
   the two choke-point actors (a third mode branch), NOT as a new polymorphic interface — the
   same "no interface extraction" economy ScadaBridge's §7.1 landed on.
4. **`ConfigPublishCoordinator` subscribes to the `deployment-acks` DPS topic unconditionally**
   (`:132`) in both modes — it is the `Dps`-mode ack ingress and must survive; do not "clean it
   up" while deleting ClusterClient.
5. **`FleetStatusBroadcaster` was left behind by Phase 6** — see §3.8; issue, not scope.
6. **Interceptor multi-constructor trap and enum-zero mapping traps** are pre-known from
   ScadaBridge's live gate and OtOpcUa's own `TelemetryStreamAuthInterceptor.cs:38-42` — both
   are encoded as tests in the plan, not left to be rediscovered.

---

## 5. The other knob: keep ClusterClient (the do-nothing steelman)

ClusterClient works today: Phase 7 ran 6/6 failover drills through it, `SendToAll` fan-out and
DB-driven contacts are debugged, and the 5-message surface is small enough that "it's a lot of
transport" is not an argument. Keeping it costs nothing *now*.

What it leaves on the table: (1) the command plane stays **unauthenticated** across the WAN —
the one boundary path with no auth story, in the app whose sister already closed the identical
gap; (2) two transport postures on one set of boxes — every firewall rule, packet capture and
incident runbook must reason about Akka remoting *and* gRPC crossing the same boundary;
(3) `buffer-size=0` silent drops stay silent; (4) Akka.NET upstream has signalled the
ClusterClient/receptionist pair as legacy — ScadaBridge's design recorded the same
deprecation-direction argument and it has only aged one way.

**Verdict:** the auth gap alone justifies Phase 0-style hardening, and once auth + contract
exist, the remaining migration is small. Do the whole thing.

---

## 6. Options

- **A — full migration (recommended):** contract + PSK + both directions behind
  `MeshTransport:Mode=Grpc`, coexistence-flagged, then delete ClusterClient. ~2 weeks of
  focused work given the 5-message surface and the existing listeners/patterns.
- **B — auth-only:** bolt a PSK story onto ClusterClient (there isn't a natural one — Akka
  remoting auth means mTLS on the transport, a different and larger program) or accept the gap.
  Rejected: the "auth" half of the work has no cheap ClusterClient-shaped form.
- **C — do nothing:** see §5.

**Recommendation: A.**

---

## 7. Phasing and effort

| Phase | Content | Est. | Parallel with |
|---|---|---|---|
| 0 | Contract (proto + mappers + goldens), auth interceptors + `IMeshPskProvider`, options/validators (+`Grpc` value accepted, SplitTopology relaxation), rig PSK seeding, stale-comment fixes, FleetStatus issue filed | 2–3 d | 1A/1B prep |
| 1A | Central→node command plane behind central's `Mode=Grpc` (worktree A) | 3–4 d | 1B |
| 1B | Node→central ack plane behind node's `Mode=Grpc` (worktree B) | 2–3 d | 1A |
| 2 | Full-rig flip + soak + kill drills (live gate) | 1–2 d | — |
| 3 | Deletion + validator tightening + docs sweep | 1–2 d | — |
| 4 | Terminal live gate on the post-deletion build | 1 d | — |

Critical path ≈ 0 → 1A → 2 → 3 → 4: **~1.5–2.5 weeks**, one-third of ScadaBridge's, which is
what a 5-message no-Ask surface with pre-built listeners should cost.

**Risks, honestly:** the mapper/enum-zero class of bug (mitigated by goldens + contract-lock
tests, but ScadaBridge still found five of these in implementation); the Kestrel
explicit-endpoints trap does NOT apply (no new listeners — but any future "just add a port"
deviation re-imports it, see `Program.cs:564-583` before touching anything there); PSK cache
invalidation on cluster add/remove is the one genuinely new stateful component
(`IMeshPskProvider`) — its fail-closed behavior needs adversarial tests, not just happy-path.

Implementation plan: `2026-08-06-clusterclient-to-grpc.md`. Live gate:
`2026-08-06-clusterclient-to-grpc-live-gate.md`. Central tracking:
`2026-08-06-clusterclient-to-grpc-tracking.md`.