# ScadaBridge: migrating cross-cluster ClusterClient traffic to gRPC-only

**Date:** 2026-07-22. Code-verified inventory of every ClusterClient touchpoint in
`~/Desktop/ScadaBridge`, then the migration assessment. Companion to
`akka_msquic_transport.md` §8/§8a (why chunking can't fix this path) and
`akka_multidc_clusters.md` (why the site↔central boundary stays two clusters).

## 1. What ClusterClient actually carries today

Everything crossing the site↔central boundary funnels through **two actors** —
`SiteCommunicationActor` (site) and `CentralCommunicationActor` (central) — which makes the
migration surface unusually well-bounded.

**Receptionist registrations (3, all in `AkkaHostedService.cs`):**

| Role | Actor | Path | Note |
|---|---|---|---|
| Central | `CentralCommunicationActor` | `/user/central-communication` (`:436`) | the real workhorse |
| Central | `ManagementActor` | `/user/management` (`:459`) | **vestigial** — intended for an out-of-cluster CLI (REQ-HOST-6a), but the in-repo CLI has no ClusterClient; nothing in the repo sends to it |
| Site | `SiteCommunicationActor` | `/user/site-communication` (`:935`) | central→site target |

**Site → central** (single `central-cluster-client`, contacts from
`CommunicationOptions.CentralContactPoints`, `AkkaHostedService.cs:942-953`) — 7 message types,
all Ask/reply except the heartbeat (`SiteCommunicationActor.cs`):

| Message | Purpose | Guarantee layered on top |
|---|---|---|
| `NotificationSubmit` → `NotificationSubmitAck` | S&F notification delivery (central owns SMTP) | not-accepted/timeout → rows stay buffered; active-node-gated retry sweep |
| `NotificationStatusQuery` → response | S&F status lookup | — |
| `IngestAuditEventsCommand` → reply | audit batch push | ack = accepted-id list; throw-on-failure keeps SQLite rows Pending |
| `IngestCachedTelemetryCommand` → reply | audit+SiteCall dual-write push | same contract |
| `ReconcileSiteRequest` → response | per-node startup self-heal | — |
| `SiteHealthReport` → ack | rich periodic health → `CentralHealthAggregator` | monotonic sequence numbers + dedup; failed-send counter restore |
| `HeartbeatMessage` (Tell) | liveness, stamps `IsActive` | fire-and-forget by design |

**Central → site** (one ClusterClient **per site**, contacts = NodeA/NodeB from the config DB,
`CentralCommunicationActor.cs:469-598`): every command wrapped in
`SiteEnvelope(siteId, message)` — **29 command types** produced by `CommunicationService`
(instance lifecycle, artifact deploy, OPC-UA browse/verify/tag read-write, cert trust,
event-log/parked-message queries, integration routing, `TriggerSiteFailover`).

**Serialization: the default Newtonsoft JSON fallback** — no `serialization-bindings` anywhere.
This is the amplifier in the known 128 KB frame bug
(`docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md`): oversized
ClusterClient messages are **silently dropped**, which is why deploy config already moved to
HTTP notify-and-fetch.

**Not ClusterClient (stays untouched):** `DistributedPubSub` replica publishes
(`SiteHeartbeatReplica`/`SiteHealthReportReplica`) are intra-central-cluster; all pair-internal
Akka (singletons, membership, S&F actors) is intra-cluster.

## 2. Why gRPC-only is a natural completion, not a rewrite

The repo has already walked 3 of the 5 steps:

1. **`docs/plans/grpc_streams.md`** already rules ClusterClient "not built for high-throughput
   streaming" and moved all streaming values to gRPC — its stated end-state ("ClusterClient keeps
   command/control") is a waypoint, not a law.
2. **Deploy config** already left ClusterClient for token-gated HTTP pull after the frame-size
   incident — with the rationale "fits the existing pull-based gRPC pattern".
3. **The audit path is half-migrated already:** `sitestream.proto` *already defines*
   `IngestAuditEvents`/`IngestCachedTelemetry` with proto DTOs and an
   `ISiteStreamAuditClient` seam whose ClusterClient implementation is swappable — the contract
   and the seam exist; only the transport binding is Akka.
4. Upstream direction agrees: JVM Akka deprecated ClusterClient in favor of gRPC for
   cross-cluster traffic; Akka.NET keeps it working but it is not where investment goes.
5. Structural wins on arrival: the **silent-drop-over-128 KB failure class disappears** (gRPC
   default 4 MB cap + an explicit `ResourceExhausted` error instead of silence), Ask timeouts
   become real per-call deadlines, and **Akka remoting never crosses the boundary** — the
   site↔central firewall surface reduces to gRPC/HTTP ports, and the two clusters' quarantine
   lifecycles fully decouple (the coupling `akka_msquic_transport.md` §8a warns about).

## 3. What the migration takes

### 3.1 Topology decision — who serves what

Today the **site** is the only gRPC server (`SiteStreamService` on `:8083` h2c) and central the
only client; but both sides already dial each other at the Akka level, and sites already dial
central over HTTP for deploy fetch. So the clean shape is symmetric, no new firewall direction:

- **New `CentralControlService`** (gRPC) hosted on central's existing Kestrel (same host as the
  deploy-fetch endpoint): the 7 site→central RPCs.
- **Extended site service** (grow `SiteStreamService` or a sibling `SiteCommandService` on the
  same `:8083` listener): the central→site command envelope.

(The alternative — a central-initiated duplex stream tunneling site→central pushes — avoids the
central listener but re-implements connection management the receptionist gave for free; not
worth it when sites already reach central over HTTP.)

### 3.2 Contract work

- **Site→central (small):** 7 RPCs. Audit/telemetry are a near-free move — proto DTOs exist and
  the swap is a new `ISiteStreamAuditClient` implementation (the `NoOp`/`ClusterClient`
  implementations prove the seam). Notifications, health report, reconcile, heartbeat need
  proto messages written (they're modest DTOs today).
- **Central→site (the bulk):** 29 `SiteEnvelope` commands. Don't write 30 RPCs and don't smuggle
  JSON in one generic RPC — group by domain into a few RPCs with **`oneof` request/response
  envelopes** (lifecycle, deploy, OPC-UA browse/tag, queries, failover). Mirrors `SiteEnvelope`
  semantics with wire-level typing; mapper + golden tests are the real cost here.
- **Transport seams:** the site side has precedent (`AkkaHealthReportTransport`,
  `ISiteStreamAuditClient`); `CommunicationService` (central) needs an `ISiteCommandTransport`
  seam so the 29 call sites don't change — the seam is introduced first, Akka behind it, then
  the gRPC implementation swaps in per domain group.

### 3.3 Semantics to preserve (all transport-agnostic, all carry over)

- S&F transient-failure contract: gRPC `Unavailable`/`DeadlineExceeded` → rows stay buffered —
  identical to today's not-accepted/timeout handling, with better error taxonomy.
- Health sequence numbers + dedup + counter-restore: unchanged (they live above the transport).
- Heartbeat stays fire-and-forget (unary, errors logged not thrown).
- Dual-central routing: the receptionist let a site reach "whichever central answers"; the gRPC
  client needs try-in-order failover across the central A/B endpoints (`CentralContactPoints`
  becomes `CentralGrpcEndpoints`). The intra-central replica pub/sub already reconciles which
  central saw the report — unchanged.
- Per-site channel management on central: the DB-driven contact cache refresh loop
  (`CentralCommunicationActor.cs:532-598`) carries over as a `GrpcChannel`-per-site factory —
  which `grpc_streams.md` (§167, §810) already models for the streaming path.

### 3.4 Work you must do anyway: auth — preshared key from the Secrets store (decided 2026-07-22)

`SiteStreamService` today has **no auth at all** — plaintext h2c, no interceptor (only the
LocalDb sync path is ApiKey-gated, and only for its own service). This gap exists today
regardless of migration — it just becomes indefensible after it. **Decision: the gRPC channels
authenticate with a preshared key stored in `ZB.MOM.WW.Secrets`** (which ScadaBridge already
adopted, `${secret:NAME}` resolution, fail-closed). Design:

- **Key scope: one PSK per site relationship** (e.g. secret name `SB-GRPC-PSK-SITE-A`), covering
  both directions of that pair's traffic (central→site commands AND that site's calls to
  `CentralControlService`). Central's store holds one secret per site; each site's local SQLite
  store holds only its own. Blast radius: a compromised site never yields another site's key.
  (A single fleet-wide PSK is simpler to seed but is rejected for exactly that reason.)
- **Client side:** resolve `${secret:SB-GRPC-PSK-<site>}` at startup **fail-closed** (the
  Layer-B pattern the family already uses for driver secrets — boot fails loudly on a missing/
  undecryptable ref, no silent unauthenticated fallback) and attach it per-call as a bearer
  metadata header via a `CallCredentials`/interceptor on the channel pair from §3.5.
- **Server side:** a fail-closed interceptor modeled on `LocalDbSyncAuthInterceptor` (the
  in-repo precedent) on **both** services — constant-time comparison, applied to the SiteStream/
  SiteCommand services on `:8083` and to `CentralControlService` on central. The LocalDb sync
  path keeps its own existing key; no sharing between the two mechanisms.
- **Seeding/ops:** the `secret` CLI's *Reference audit & seed* flow (interactive console 0.3.0)
  is the deployment-setup tool — it classifies every `${secret:}` ref and seeds the gaps per
  node. Rotation: set the new value on both sides' stores, then restart the pair (pairs restart
  together by established discipline); if zero-blip rotation is ever needed, the server
  interceptor can accept old+new during a configured window.
- **Transport honesty:** a bearer PSK over plaintext h2c is readable/replayable by anyone on
  path — today that's accepted via network isolation (same posture as the unauthenticated status
  quo, but now with authentication). TLS on these listeners is the follow-on hardening; the PSK
  design doesn't change if/when TLS lands.

### 3.5 Active-node resolution — does the gRPC client need to find the pair's active node?

**No — and this is load-bearing good news, verified in code.** Inbound commands on a site node
are not executed locally: `SiteCommunicationActor` `Forward`s essentially everything to the
**Deployment Manager cluster-singleton proxy** (`SiteCommunicationActor.cs:101-181`), and the
code comments are explicit — the singleton "always lands on the active site node"; DCL
connections/browse sessions "only exist on the singleton's node," so commands hop through the
proxy by design (`:153-178`). The same shape holds centralward: `CentralCommunicationActor` runs
on both central nodes, and heartbeat/health state is reconciled across the pair via the
intra-cluster `DistributedPubSub` replicas. In other words, **each cluster already solves
active-node routing internally with singleton proxies + pub/sub — the cross-cluster transport
never had to know, and ClusterClient never did** (the receptionist delivers to whichever node
answers).

A gRPC server on either pair node therefore keeps the exact same contract: accept the call, hand
it to the same actor, let the singleton proxy route. What the migration must provide instead is
cheap **either-node failover at the client**:

**Client failover/failback policy (per pair — applies to central's per-site clients AND the
site's central client alike):**

- **Two channels, one preference.** Hold a `GrpcChannel` per endpoint (NodeA, NodeB) rather than
  one channel with re-dial logic; mark one *preferred* (config order, or `/health/active` probe
  once site-node health lands — Overview plan Phase 1). All calls go to the current channel.
- **Failover:** on `Unavailable`/connect failure/readiness rejection
  (`SiteStreamGrpcServer` already returns `StatusCode.Unavailable` when not ready — the one
  state where the singleton proxy can't route), flip to the secondary **if available** and stay
  there (sticky-until-failure, so a flapping primary doesn't ping-pong every call).
- **Failback:** a background probe (30–60 s cadence) checks the preferred endpoint
  (`grpc.health` check or `/health/ready`); when it answers again, new calls return to it —
  in-flight calls and open streams finish where they are. Failback is an optimization, never a
  correctness requirement: either node accepts and singleton-routes (§ above).
- **Retry safety:** automatic cross-node retry ONLY for failures that provably never reached a
  server (connect refused, `Unavailable` before response headers). **Never blind-retry
  `DeadlineExceeded` on non-idempotent commands** (`WriteTag`, `DeployArtifacts`,
  `TriggerSiteFailover`) — ClusterClient's Ask had the same ambiguity and the layers above
  already tolerate it (S&F dedup, sequence numbers), but the migration must not *add* an eager
  retry that turns one write into two. Queries/browses are freely retryable.
- **Streams don't fail over transparently:** an open `SubscribeInstance`/`SubscribeSite` stream
  dies with its channel; the subscriber re-subscribes via the policy above (the
  `grpc_streams.md` design already owns re-subscribe semantics — reuse, don't reinvent).
- Cost of landing on the standby is one intra-pair remoting hop (as today).
- One verification item for Phase 2: confirm none of the 29 command handlers bypasses the proxy
  and assumes local state (the audit/parked-message query paths are the ones to eyeball).

### 3.6 Deletions at the end

`ClusterClient`/`ClusterClientReceptionist` creation + registrations, `DefaultSiteClientFactory`,
the `SiteEnvelope` Akka path, `CommunicationOptions.CentralContactPoints`. The
`/user/management` registration is **deletable today** independent of everything else (no sender
exists in the repo). `Akka.Cluster.Tools` stays for ClusterSingleton.

### 3.7 The management CLI is unaffected (deliberately out of scope)

`scadabridge.exe` never used ClusterClient: it speaks HTTP Basic to a Central's `/management`
endpoints, which ask `ManagementActor` **in-process** (`ManagementEndpoints.cs:117` via
`ManagementActorHolder`) — the `/user/management` receptionist registration was built for an
out-of-cluster CLI that was never wired (REQ-HOST-6a) and has no sender in the repo. So this
migration requires **zero CLI changes**; the CLI's transport is already boundary-independent.
Retargeting the CLI onto the new gRPC services later is a separate, optional decision — it would
buy one consistent management wire, but costs the CLI a gRPC stack and auth story for no current
functional gain, so it is explicitly **not** part of this migration's scope or estimate.

## 4. Phasing and effort

| Phase | Content | Effort |
|---|---|---|
| 0 | Delete vestigial `ManagementActor` registration; add the §3.4 PSK-from-Secrets auth interceptor to `SiteStreamService` (standalone hardening win — central client attaches the PSK it resolves from its store) | days |
| 1 | `CentralControlService` + the 7 site→central RPCs behind existing seams; audit swap; heartbeat/health/notifications/reconcile | ~1–2 wks |
| 2 | `ISiteCommandTransport` seam + `oneof` command envelopes; migrate the 29 commands by domain group (deploy+lifecycle first, queries last) | ~2–3 wks |
| 3 | Remove ClusterClient/receptionist + config cutover; live gate on the docker rig: kill central-a mid-S&F drain, site-node failover, deploy, `TriggerSiteFailover`, dual-central failover of site→central calls | ~1 wk |

Total ≈ **4–6 weeks** with live gating. Both transports can coexist per-path throughout (the
seams make each path a one-line flip), so there is no big-bang cutover; rollback per phase is a
config/DI revert.

**Risks, honestly:** the `oneof` envelope mappers are tedious and where bugs will hide (golden
tests against captured payloads); ordering assumptions are low-risk (everything is request/reply
or sequence-numbered, verify none of the 30 commands relies on ClusterClient's per-connection
ordering across *different* commands); Ask-sender-forwarding tricks disappear (fine — RPC replies
are direct, but check `SiteCallAuditActor`'s relay paths); the auth addition must not break the
rig's current unauthenticated compose setup (env-gated like LocalDb's).

## 5. The other knob: raising the 128 KB frame cap

Since the frame limit motivates part of this migration, the obvious question — why not just
raise `akka.remote.dot-netty.tcp.maximum-frame-size`? It is a legitimate knob, and if
ClusterClient stays (Option B) it is worth doing deliberately, but know exactly what it buys:

- **How:** set `maximum-frame-size` (default 128000b) in the generated HOCON
  (`AkkaHostedService.BuildHocon` — today it never overrides it) and raise
  `send-buffer-size`/`receive-buffer-size` (default 256000b) to at least the new frame size.
  **The limit is enforced by the receiver**, so it must be raised on **every node of both
  clusters and rolled out everywhere before any sender relies on it** — a half-rolled-out raise
  reproduces the silent-drop bug in one direction. The pair-restart discipline makes the rollout
  cheap; the coordination requirement is the trap.
- **What it buys:** headroom for today's fat-ish commands (browse results, query replies) with
  zero code change; the deploy-config class of payload would fit at, say, 2–4 MB.
- **What it does not fix:** (1) the **silent-drop failure class survives** — the threshold moves,
  it doesn't disappear; anything over the new cap still vanishes without an error. (2)
  **Head-of-line gets worse, not better:** all ClusterClient traffic, heartbeats, and
  failure-detector signals share one ordered dot-netty channel, so a single 4 MB frame stalls
  everything behind it — on a 2-node availability-first posture with tight failure-detector
  timeouts, big frames are exactly what you don't want ahead of heartbeats (this is why the
  deploy fix chose notify-and-fetch over a cap raise, and why `akka_msquic_transport.md` §2
  treats the default cap as a guardrail, not a bug). (3) The default Newtonsoft serializer's
  double-JSON-escaping roughly **halves the effective budget** for JSON-bearing payloads. (4) It
  is a dead-end investment: Artery (1.6) replaces the pipeline, and Option A removes the
  constraint entirely (gRPC's 4 MB default cap fails loudly with `ResourceExhausted`, and is
  itself configurable per-call-type).

Verdict: as a *stopgap under Option B*, a modest raise (e.g. 512 KB–1 MB, buffers to match,
cluster-wide) plus an explicit size check before `ClusterClient.Send` (log-and-reject instead of
silent drop) is defensible. As part of Option A it's unnecessary — don't spend the rollout.

## 6. Options

**A. Full migration, phased as §4 (recommended).** Best-practice fit: it completes a direction
the repo already chose twice under pressure (streaming, deploy), eliminates the silent-drop
failure class on the control path, closes the unauthenticated-control-channel gap as part of the
work, decouples the clusters' failure domains entirely, and lands on the transport the whole
family already operates (gRPC + Kestrel + interceptors). Aligns with upstream's own deprecation
of ClusterClient.

**B. Status quo per `grpc_streams.md` (ClusterClient keeps command/control).** Defensible if no
current pain: the frame-size hazard is mostly defused (big payloads already moved) and the
receptionist's discovery/failover is free; pair it with the deliberate cap raise + pre-send size
check from §5. But it leaves two cross-cluster transports to operate,
the silent-drop class alive for any future fat command, an unauthenticated gRPC service, and a
dependency on the least-invested-in corner of Akka.NET. If choosing B, still do Phase 0 (auth +
vestigial-registration deletion) — it stands alone.

**C. Wait for Akka.NET 1.6.** Not a reason to wait: Artery changes intra-cluster transport;
ClusterClient's cross-cluster story doesn't improve. Deferring buys nothing here.

**Recommendation: A** — with Phase 0 done immediately regardless of the rest.

## 7. Design-completion deep dive (2026-07-22) — corrections and refinements

A three-track code deep dive (full message inventory; hosting/config/Secrets/tests; existing
gRPC machinery) closed the remaining unknowns. The implementation plan lives at
`ScadaBridge/docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`; corrections that supersede
earlier sections:

1. **The seam is smaller than §3.2 assumed — one choke point per direction.** Every central→site
   producer (all of `CommunicationService` AND `SiteCallAuditActor`'s two relays) funnels
   `SiteEnvelope` through **`CentralCommunicationActor`**, which unwraps and routes via the
   per-site ClusterClient; every site→central path funnels through **`SiteCommunicationActor`**.
   So the transport swap happens *inside the two communication actors* (an injected
   `ISiteCommandTransport` / `ICentralTransport` behind a config flag) — `CommunicationService`
   needs **no interface extraction** and its ~20 consumers are untouched. (It has no interface
   today and is DI-registered concrete; extracting one is unnecessary.)
2. **Exact command count: 29, and one is dead.** 27 from `CommunicationService` + 2
   `SiteCallAuditActor` relays. **Finding: `IntegrationCallRequest` is unwired in production** —
   `RegisterLocalHandler(Integration, …)` exists only in tests, so the command always answers
   "Integration handler not available." It is excluded from the gRPC contract (28 commands
   migrate) and filed as its own issue rather than silently ported or fixed.
3. **§3.5 verification item RESOLVED.** Of the 29: 22 are singleton-/cluster-routed (incl.
   artifact deploy and event-log query, whose "local handler" slots are actually registered to
   singleton proxies — the event-log one deliberately, per wiring comments); `TriggerSiteFailover`
   resolves its target from cluster state, node-agnostic. The **only node-local target is the
   parked-message handler** (5 command types) — and it is standby-safe *by replication*, not
   routing: it reads/mutates the LocalDb-replicated S&F store, and the delivery sweep is
   active-gated. The gRPC design must preserve exactly that (dispatch on the receiving node, no
   forced active-node routing) — which either-node gRPC does.
4. **Central hosting is net-new work.** Central has **no** `AddGrpc`/`MapGrpcService` and no
   explicit Kestrel listener (env-driven `:5000`, Traefik in front on HTTP/1). The migration adds
   an explicit HTTP/2 (h2c) listener on central (port `8083`, symmetric with sites; rig publishes
   it), `AddGrpc` + the PSK interceptor, and `CentralControlService`. Sites also get a **new**
   standing config `ScadaBridge:Communication:CentralGrpcEndpoints` — today the only central URL
   a site sees arrives *inside* `RefreshDeploymentCommand` (`CentralFetchBaseUrl`), so gRPC
   endpoint discovery genuinely needs new config.
5. **PSK mechanics, concretized from the in-repo templates.** Server side copies
   `LocalDbSyncAuthInterceptor` exactly (service-prefix match, `authorization: Bearer`,
   `CryptographicOperations.FixedTimeEquals`, fail-closed on unset key, `PermissionDenied`);
   clients attach the key as call-level `Metadata` (the LocalDb sync client pattern). Key
   sourcing: **site nodes** read a static config key supplied as `${secret:SB-GRPC-PSK-<site>}`
   via the existing pre-host `SecretReferenceExpander` (`Program.cs:56-65`); **central** — whose
   site set is dynamic from the DB — resolves `SB-GRPC-PSK-{siteId}` at channel-build time via
   the runtime `ISecretResolver` (the MxGateway G-3 pattern), cached, fail-closed. Sites identify
   themselves to `CentralControlService` with an `x-scadabridge-site` metadata header so central
   verifies against that site's key. The rig mirrors the LocalDb dev pattern (same literal on
   both pair nodes; prod uses `${secret:}`).
6. **Reuse map for §3.5's client policy:** per-site `GrpcNodeAAddress`/`GrpcNodeBAddress` columns
   already exist on the `Site` entity (streaming path reads them); primary/fallback single-retry
   exists in the pull clients; sticky flip + stability-window retry-budget exists in
   `DebugStreamBridgeActor`/`SiteAlarmAggregatorActor`. The plan extracts this into one shared
   `SitePairChannelProvider` rather than a fourth hand-rolled copy. **Caution — doc-vs-code gap:**
   `grpc_streams.md` names `CentralCommunicationActor.LoadSiteAddressesFromDb()` and
   singleton-aware node selection; neither exists in code (selection is blind A-then-B flip).
7. **Deadlines are net-new.** No gRPC call in the repo sets a deadline or retry policy today; the
   new services introduce per-call deadlines mirroring the eight `CommunicationOptions` Ask
   timeouts (`DeploymentTimeout` 2 m … `HealthReportTimeout` 10 s), preserving the documented
   inner-before-outer timeout ordering (e.g. `SiteCallAuditOptions.RelayTimeout` 10 s <
   `QueryTimeout` 30 s).
8. **Proto codegen is checked-in, not build-time** (protoc segfaults in the linux_arm64 build
   image): new protos follow the sitestream recipe — commented-out `<Protobuf>` item, generate on
   macOS, commit the generated C#, `docker/regen-proto.sh`.
9. **Test assets to extend:** `Communication.Tests` already covers both communication actors with
   Akka TestKit (xunit **v2** — TestKit.Xunit2 compatible) + NSubstitute;
   `DirectActorSiteStreamAuditClient` is an explicitly-reusable wire-bypassing double for
   ingest-path integration tests.
