scadaproj/scadabridge_clusterclient_to_grpc.md

ScadaBridge: migrating cross-cluster ClusterClient traffic to gRPC-only

Date: 2026-07-22. Code-verified inventory of every ClusterClient touchpoint in ~/Desktop/ScadaBridge, then the migration assessment. Companion to akka_msquic_transport.md §8/§8a (why chunking can't fix this path) and akka_multidc_clusters.md (why the site↔central boundary stays two clusters).

1. What ClusterClient actually carries today

Everything crossing the site↔central boundary funnels through two actors — SiteCommunicationActor (site) and CentralCommunicationActor (central) — which makes the migration surface unusually well-bounded.

Receptionist registrations (3, all in AkkaHostedService.cs):

Role	Actor	Path	Note
Central	CentralCommunicationActor	/user/central-communication (:436)	the real workhorse
Central	ManagementActor	/user/management (:459)	vestigial — intended for an out-of-cluster CLI (REQ-HOST-6a), but the in-repo CLI has no ClusterClient; nothing in the repo sends to it
Site	SiteCommunicationActor	/user/site-communication (:935)	central→site target

Site → central (single central-cluster-client, contacts from CommunicationOptions.CentralContactPoints, AkkaHostedService.cs:942-953) — 7 message types, all Ask/reply except the heartbeat (SiteCommunicationActor.cs):

Message	Purpose	Guarantee layered on top
NotificationSubmit → NotificationSubmitAck	S&F notification delivery (central owns SMTP)	not-accepted/timeout → rows stay buffered; active-node-gated retry sweep
NotificationStatusQuery → response	S&F status lookup	—
IngestAuditEventsCommand → reply	audit batch push	ack = accepted-id list; throw-on-failure keeps SQLite rows Pending
IngestCachedTelemetryCommand → reply	audit+SiteCall dual-write push	same contract
ReconcileSiteRequest → response	per-node startup self-heal	—
SiteHealthReport → ack	rich periodic health → CentralHealthAggregator	monotonic sequence numbers + dedup; failed-send counter restore
HeartbeatMessage (Tell)	liveness, stamps IsActive	fire-and-forget by design

Central → site (one ClusterClient per site, contacts = NodeA/NodeB from the config DB, CentralCommunicationActor.cs:469-598): every command wrapped in SiteEnvelope(siteId, message) — 29 command types produced by CommunicationService (instance lifecycle, artifact deploy, OPC-UA browse/verify/tag read-write, cert trust, event-log/parked-message queries, integration routing, TriggerSiteFailover).

Serialization: the default Newtonsoft JSON fallback — no serialization-bindings anywhere. This is the amplifier in the known 128 KB frame bug (docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md): oversized ClusterClient messages are silently dropped, which is why deploy config already moved to HTTP notify-and-fetch.

Not ClusterClient (stays untouched): DistributedPubSub replica publishes (SiteHeartbeatReplica/SiteHealthReportReplica) are intra-central-cluster; all pair-internal Akka (singletons, membership, S&F actors) is intra-cluster.

2. Why gRPC-only is a natural completion, not a rewrite

The repo has already walked 3 of the 5 steps:

1. docs/plans/grpc_streams.md already rules ClusterClient "not built for high-throughput streaming" and moved all streaming values to gRPC — its stated end-state ("ClusterClient keeps command/control") is a waypoint, not a law.
2. Deploy config already left ClusterClient for token-gated HTTP pull after the frame-size incident — with the rationale "fits the existing pull-based gRPC pattern".
3. The audit path is half-migrated already: sitestream.proto already defines IngestAuditEvents/IngestCachedTelemetry with proto DTOs and an ISiteStreamAuditClient seam whose ClusterClient implementation is swappable — the contract and the seam exist; only the transport binding is Akka.
4. Upstream direction agrees: JVM Akka deprecated ClusterClient in favor of gRPC for cross-cluster traffic; Akka.NET keeps it working but it is not where investment goes.
5. Structural wins on arrival: the silent-drop-over-128 KB failure class disappears (gRPC default 4 MB cap + an explicit ResourceExhausted error instead of silence), Ask timeouts become real per-call deadlines, and Akka remoting never crosses the boundary — the site↔central firewall surface reduces to gRPC/HTTP ports, and the two clusters' quarantine lifecycles fully decouple (the coupling akka_msquic_transport.md §8a warns about).

3. What the migration takes

3.1 Topology decision — who serves what

Today the site is the only gRPC server (SiteStreamService on :8083 h2c) and central the only client; but both sides already dial each other at the Akka level, and sites already dial central over HTTP for deploy fetch. So the clean shape is symmetric, no new firewall direction:

• New CentralControlService (gRPC) hosted on central's existing Kestrel (same host as the deploy-fetch endpoint): the 7 site→central RPCs.
• Extended site service (grow SiteStreamService or a sibling SiteCommandService on the same :8083 listener): the central→site command envelope.

(The alternative — a central-initiated duplex stream tunneling site→central pushes — avoids the central listener but re-implements connection management the receptionist gave for free; not worth it when sites already reach central over HTTP.)

3.2 Contract work

• Site→central (small): 7 RPCs. Audit/telemetry are a near-free move — proto DTOs exist and the swap is a new ISiteStreamAuditClient implementation (the NoOp/ClusterClient implementations prove the seam). Notifications, health report, reconcile, heartbeat need proto messages written (they're modest DTOs today).
• Central→site (the bulk): 29 SiteEnvelope commands. Don't write 30 RPCs and don't smuggle JSON in one generic RPC — group by domain into a few RPCs with oneof request/response envelopes (lifecycle, deploy, OPC-UA browse/tag, queries, failover). Mirrors SiteEnvelope semantics with wire-level typing; mapper + golden tests are the real cost here.
• Transport seams: the site side has precedent (AkkaHealthReportTransport, ISiteStreamAuditClient); CommunicationService (central) needs an ISiteCommandTransport seam so the 29 call sites don't change — the seam is introduced first, Akka behind it, then the gRPC implementation swaps in per domain group.

3.3 Semantics to preserve (all transport-agnostic, all carry over)

• S&F transient-failure contract: gRPC Unavailable/DeadlineExceeded → rows stay buffered — identical to today's not-accepted/timeout handling, with better error taxonomy.
• Health sequence numbers + dedup + counter-restore: unchanged (they live above the transport).
• Heartbeat stays fire-and-forget (unary, errors logged not thrown).
• Dual-central routing: the receptionist let a site reach "whichever central answers"; the gRPC client needs try-in-order failover across the central A/B endpoints (CentralContactPoints becomes CentralGrpcEndpoints). The intra-central replica pub/sub already reconciles which central saw the report — unchanged.
• Per-site channel management on central: the DB-driven contact cache refresh loop (CentralCommunicationActor.cs:532-598) carries over as a GrpcChannel-per-site factory — which grpc_streams.md (§167, §810) already models for the streaming path.

3.4 Work you must do anyway: auth — preshared key from the Secrets store (decided 2026-07-22)

SiteStreamService today has no auth at all — plaintext h2c, no interceptor (only the LocalDb sync path is ApiKey-gated, and only for its own service). This gap exists today regardless of migration — it just becomes indefensible after it. Decision: the gRPC channels authenticate with a preshared key stored in ZB.MOM.WW.Secrets (which ScadaBridge already adopted, ${secret:NAME} resolution, fail-closed). Design:

• Key scope: one PSK per site relationship (e.g. secret name SB-GRPC-PSK-SITE-A), covering both directions of that pair's traffic (central→site commands AND that site's calls to CentralControlService). Central's store holds one secret per site; each site's local SQLite store holds only its own. Blast radius: a compromised site never yields another site's key. (A single fleet-wide PSK is simpler to seed but is rejected for exactly that reason.)
• Client side: resolve ${secret:SB-GRPC-PSK-<site>} at startup fail-closed (the Layer-B pattern the family already uses for driver secrets — boot fails loudly on a missing/ undecryptable ref, no silent unauthenticated fallback) and attach it per-call as a bearer metadata header via a CallCredentials/interceptor on the channel pair from §3.5.
• Server side: a fail-closed interceptor modeled on LocalDbSyncAuthInterceptor (the in-repo precedent) on both services — constant-time comparison, applied to the SiteStream/ SiteCommand services on :8083 and to CentralControlService on central. The LocalDb sync path keeps its own existing key; no sharing between the two mechanisms.
• Seeding/ops: the secret CLI's Reference audit & seed flow (interactive console 0.3.0) is the deployment-setup tool — it classifies every ${secret:} ref and seeds the gaps per node. Rotation: set the new value on both sides' stores, then restart the pair (pairs restart together by established discipline); if zero-blip rotation is ever needed, the server interceptor can accept old+new during a configured window.
• Transport honesty: a bearer PSK over plaintext h2c is readable/replayable by anyone on path — today that's accepted via network isolation (same posture as the unauthenticated status quo, but now with authentication). TLS on these listeners is the follow-on hardening; the PSK design doesn't change if/when TLS lands.

3.5 Active-node resolution — does the gRPC client need to find the pair's active node?

No — and this is load-bearing good news, verified in code. Inbound commands on a site node are not executed locally: SiteCommunicationActor Forwards essentially everything to the Deployment Manager cluster-singleton proxy (SiteCommunicationActor.cs:101-181), and the code comments are explicit — the singleton "always lands on the active site node"; DCL connections/browse sessions "only exist on the singleton's node," so commands hop through the proxy by design (:153-178). The same shape holds centralward: CentralCommunicationActor runs on both central nodes, and heartbeat/health state is reconciled across the pair via the intra-cluster DistributedPubSub replicas. In other words, each cluster already solves active-node routing internally with singleton proxies + pub/sub — the cross-cluster transport never had to know, and ClusterClient never did (the receptionist delivers to whichever node answers).

A gRPC server on either pair node therefore keeps the exact same contract: accept the call, hand it to the same actor, let the singleton proxy route. What the migration must provide instead is cheap either-node failover at the client:

Client failover/failback policy (per pair — applies to central's per-site clients AND the site's central client alike):

• Two channels, one preference. Hold a GrpcChannel per endpoint (NodeA, NodeB) rather than one channel with re-dial logic; mark one preferred (config order, or /health/active probe once site-node health lands — Overview plan Phase 1). All calls go to the current channel.
• Failover: on Unavailable/connect failure/readiness rejection (SiteStreamGrpcServer already returns StatusCode.Unavailable when not ready — the one state where the singleton proxy can't route), flip to the secondary if available and stay there (sticky-until-failure, so a flapping primary doesn't ping-pong every call).
• Failback: a background probe (30–60 s cadence) checks the preferred endpoint (grpc.health check or /health/ready); when it answers again, new calls return to it — in-flight calls and open streams finish where they are. Failback is an optimization, never a correctness requirement: either node accepts and singleton-routes (§ above).
• Retry safety: automatic cross-node retry ONLY for failures that provably never reached a server (connect refused, Unavailable before response headers). Never blind-retry DeadlineExceeded on non-idempotent commands (WriteTag, DeployArtifacts, TriggerSiteFailover) — ClusterClient's Ask had the same ambiguity and the layers above already tolerate it (S&F dedup, sequence numbers), but the migration must not add an eager retry that turns one write into two. Queries/browses are freely retryable.
• Streams don't fail over transparently: an open SubscribeInstance/SubscribeSite stream dies with its channel; the subscriber re-subscribes via the policy above (the grpc_streams.md design already owns re-subscribe semantics — reuse, don't reinvent).
• Cost of landing on the standby is one intra-pair remoting hop (as today).
• One verification item for Phase 2: confirm none of the 29 command handlers bypasses the proxy and assumes local state (the audit/parked-message query paths are the ones to eyeball).

3.6 Deletions at the end

ClusterClient/ClusterClientReceptionist creation + registrations, DefaultSiteClientFactory, the SiteEnvelope Akka path, CommunicationOptions.CentralContactPoints. The /user/management registration is deletable today independent of everything else (no sender exists in the repo). Akka.Cluster.Tools stays for ClusterSingleton.

3.7 The management CLI is unaffected (deliberately out of scope)

scadabridge.exe never used ClusterClient: it speaks HTTP Basic to a Central's /management endpoints, which ask ManagementActor in-process (ManagementEndpoints.cs:117 via ManagementActorHolder) — the /user/management receptionist registration was built for an out-of-cluster CLI that was never wired (REQ-HOST-6a) and has no sender in the repo. So this migration requires zero CLI changes; the CLI's transport is already boundary-independent. Retargeting the CLI onto the new gRPC services later is a separate, optional decision — it would buy one consistent management wire, but costs the CLI a gRPC stack and auth story for no current functional gain, so it is explicitly not part of this migration's scope or estimate.

4. Phasing and effort

Phase	Content	Effort
0	Delete vestigial ManagementActor registration; add the §3.4 PSK-from-Secrets auth interceptor to SiteStreamService (standalone hardening win — central client attaches the PSK it resolves from its store)	days
1	CentralControlService + the 7 site→central RPCs behind existing seams; audit swap; heartbeat/health/notifications/reconcile	~1–2 wks
2	ISiteCommandTransport seam + oneof command envelopes; migrate the 29 commands by domain group (deploy+lifecycle first, queries last)	~2–3 wks
3	Remove ClusterClient/receptionist + config cutover; live gate on the docker rig: kill central-a mid-S&F drain, site-node failover, deploy, TriggerSiteFailover, dual-central failover of site→central calls	~1 wk

Total ≈ 4–6 weeks with live gating. Both transports can coexist per-path throughout (the seams make each path a one-line flip), so there is no big-bang cutover; rollback per phase is a config/DI revert.

Risks, honestly: the oneof envelope mappers are tedious and where bugs will hide (golden tests against captured payloads); ordering assumptions are low-risk (everything is request/reply or sequence-numbered, verify none of the 30 commands relies on ClusterClient's per-connection ordering across different commands); Ask-sender-forwarding tricks disappear (fine — RPC replies are direct, but check SiteCallAuditActor's relay paths); the auth addition must not break the rig's current unauthenticated compose setup (env-gated like LocalDb's).

5. The other knob: raising the 128 KB frame cap

Since the frame limit motivates part of this migration, the obvious question — why not just raise akka.remote.dot-netty.tcp.maximum-frame-size? It is a legitimate knob, and if ClusterClient stays (Option B) it is worth doing deliberately, but know exactly what it buys:

• How: set maximum-frame-size (default 128000b) in the generated HOCON (AkkaHostedService.BuildHocon — today it never overrides it) and raise send-buffer-size/receive-buffer-size (default 256000b) to at least the new frame size. The limit is enforced by the receiver, so it must be raised on every node of both clusters and rolled out everywhere before any sender relies on it — a half-rolled-out raise reproduces the silent-drop bug in one direction. The pair-restart discipline makes the rollout cheap; the coordination requirement is the trap.
• What it buys: headroom for today's fat-ish commands (browse results, query replies) with zero code change; the deploy-config class of payload would fit at, say, 2–4 MB.
• What it does not fix: (1) the silent-drop failure class survives — the threshold moves, it doesn't disappear; anything over the new cap still vanishes without an error. (2) Head-of-line gets worse, not better: all ClusterClient traffic, heartbeats, and failure-detector signals share one ordered dot-netty channel, so a single 4 MB frame stalls everything behind it — on a 2-node availability-first posture with tight failure-detector timeouts, big frames are exactly what you don't want ahead of heartbeats (this is why the deploy fix chose notify-and-fetch over a cap raise, and why akka_msquic_transport.md §2 treats the default cap as a guardrail, not a bug). (3) The default Newtonsoft serializer's double-JSON-escaping roughly halves the effective budget for JSON-bearing payloads. (4) It is a dead-end investment: Artery (1.6) replaces the pipeline, and Option A removes the constraint entirely (gRPC's 4 MB default cap fails loudly with ResourceExhausted, and is itself configurable per-call-type).

Verdict: as a stopgap under Option B, a modest raise (e.g. 512 KB–1 MB, buffers to match, cluster-wide) plus an explicit size check before ClusterClient.Send (log-and-reject instead of silent drop) is defensible. As part of Option A it's unnecessary — don't spend the rollout.

6. Options

A. Full migration, phased as §4 (recommended). Best-practice fit: it completes a direction the repo already chose twice under pressure (streaming, deploy), eliminates the silent-drop failure class on the control path, closes the unauthenticated-control-channel gap as part of the work, decouples the clusters' failure domains entirely, and lands on the transport the whole family already operates (gRPC + Kestrel + interceptors). Aligns with upstream's own deprecation of ClusterClient.

B. Status quo per grpc_streams.md (ClusterClient keeps command/control). Defensible if no current pain: the frame-size hazard is mostly defused (big payloads already moved) and the receptionist's discovery/failover is free; pair it with the deliberate cap raise + pre-send size check from §5. But it leaves two cross-cluster transports to operate, the silent-drop class alive for any future fat command, an unauthenticated gRPC service, and a dependency on the least-invested-in corner of Akka.NET. If choosing B, still do Phase 0 (auth + vestigial-registration deletion) — it stands alone.

C. Wait for Akka.NET 1.6. Not a reason to wait: Artery changes intra-cluster transport; ClusterClient's cross-cluster story doesn't improve. Deferring buys nothing here.

Recommendation: A — with Phase 0 done immediately regardless of the rest.

7. Design-completion deep dive (2026-07-22) — corrections and refinements

A three-track code deep dive (full message inventory; hosting/config/Secrets/tests; existing gRPC machinery) closed the remaining unknowns. The implementation plan lives at ScadaBridge/docs/plans/2026-07-22-clusterclient-to-grpc-plan.md; corrections that supersede earlier sections:

1. The seam is smaller than §3.2 assumed — one choke point per direction. Every central→site producer (all of CommunicationService AND SiteCallAuditActor's two relays) funnels SiteEnvelope through CentralCommunicationActor, which unwraps and routes via the per-site ClusterClient; every site→central path funnels through SiteCommunicationActor. So the transport swap happens inside the two communication actors (an injected ISiteCommandTransport / ICentralTransport behind a config flag) — CommunicationService needs no interface extraction and its ~20 consumers are untouched. (It has no interface today and is DI-registered concrete; extracting one is unnecessary.)
2. Exact command count: 29, and one is dead. 27 from CommunicationService + 2 SiteCallAuditActor relays. Finding: IntegrationCallRequest is unwired in production — RegisterLocalHandler(Integration, …) exists only in tests, so the command always answers "Integration handler not available." It is excluded from the gRPC contract (28 commands migrate) and filed as its own issue rather than silently ported or fixed.
3. §3.5 verification item RESOLVED. Of the 29: 22 are singleton-/cluster-routed (incl. artifact deploy and event-log query, whose "local handler" slots are actually registered to singleton proxies — the event-log one deliberately, per wiring comments); TriggerSiteFailover resolves its target from cluster state, node-agnostic. The only node-local target is the parked-message handler (5 command types) — and it is standby-safe by replication, not routing: it reads/mutates the LocalDb-replicated S&F store, and the delivery sweep is active-gated. The gRPC design must preserve exactly that (dispatch on the receiving node, no forced active-node routing) — which either-node gRPC does.
4. Central hosting is net-new work. Central has no AddGrpc/MapGrpcService and no explicit Kestrel listener (env-driven :5000, Traefik in front on HTTP/1). The migration adds an explicit HTTP/2 (h2c) listener on central (port 8083, symmetric with sites; rig publishes it), AddGrpc + the PSK interceptor, and CentralControlService. Sites also get a new standing config ScadaBridge:Communication:CentralGrpcEndpoints — today the only central URL a site sees arrives inside RefreshDeploymentCommand (CentralFetchBaseUrl), so gRPC endpoint discovery genuinely needs new config.
5. PSK mechanics, concretized from the in-repo templates. Server side copies LocalDbSyncAuthInterceptor exactly (service-prefix match, authorization: Bearer, CryptographicOperations.FixedTimeEquals, fail-closed on unset key, PermissionDenied); clients attach the key as call-level Metadata (the LocalDb sync client pattern). Key sourcing: site nodes read a static config key supplied as ${secret:SB-GRPC-PSK-<site>} via the existing pre-host SecretReferenceExpander (Program.cs:56-65); central — whose site set is dynamic from the DB — resolves SB-GRPC-PSK-{siteId} at channel-build time via the runtime ISecretResolver (the MxGateway G-3 pattern), cached, fail-closed. Sites identify themselves to CentralControlService with an x-scadabridge-site metadata header so central verifies against that site's key. The rig mirrors the LocalDb dev pattern (same literal on both pair nodes; prod uses ${secret:}).
6. Reuse map for §3.5's client policy: per-site GrpcNodeAAddress/GrpcNodeBAddress columns already exist on the Site entity (streaming path reads them); primary/fallback single-retry exists in the pull clients; sticky flip + stability-window retry-budget exists in DebugStreamBridgeActor/SiteAlarmAggregatorActor. The plan extracts this into one shared SitePairChannelProvider rather than a fourth hand-rolled copy. Caution — doc-vs-code gap: grpc_streams.md names CentralCommunicationActor.LoadSiteAddressesFromDb() and singleton-aware node selection; neither exists in code (selection is blind A-then-B flip).
7. Deadlines are net-new. No gRPC call in the repo sets a deadline or retry policy today; the new services introduce per-call deadlines mirroring the eight CommunicationOptions Ask timeouts (DeploymentTimeout 2 m … HealthReportTimeout 10 s), preserving the documented inner-before-outer timeout ordering (e.g. SiteCallAuditOptions.RelayTimeout 10 s < QueryTimeout 30 s).
8. Proto codegen is checked-in, not build-time (protoc segfaults in the linux_arm64 build image): new protos follow the sitestream recipe — commented-out <Protobuf> item, generate on macOS, commit the generated C#, docker/regen-proto.sh.
9. Test assets to extend: Communication.Tests already covers both communication actors with Akka TestKit (xunit v2 — TestKit.Xunit2 compatible) + NSubstitute; DirectActorSiteStreamAuditClient is an explicitly-reusable wire-bypassing double for ingest-path integration tests.