docs(akka): ClusterClient→gRPC migration design + chunking addendum
akka_msquic_transport.md gains §8a (Akka.Cluster.Chunking / Akka.Delivery — the 1.5-era in-cluster large-message tool, why it can't cross the ClusterClient boundary) and the Artery-now-in-development §9 fix. scadabridge_clusterclient_to_grpc.md: full migration assessment — ClusterClient inventory (two-actor choke points, 29 commands incl. one dead), gRPC-only design (topology, oneof contracts, active-node analysis: singleton-routed so transport needs only either-node failover+failback), PSK-from-Secrets auth, frame-cap-raise tradeoffs, and §7 deep-dive corrections. Companion impl plan lives in ScadaBridge/docs/plans.
This commit is contained in:
Joseph Doherty
@@ -282,6 +282,52 @@ Mostly no — mapping the family's gRPC surfaces:
|
||||
The "big messages so we don't need side channels" hope is really an Artery feature (dedicated
|
||||
chunked large-message streams), not a QUIC-transport feature.
|
||||
|
||||
### 8a. The overlooked 1.5-era tool: Akka.Cluster.Chunking (added 2026-07-22)
|
||||
|
||||
One mechanism was missing from the original analysis — Petabridge's
|
||||
**[Akka.Cluster.Chunking](https://github.com/petabridge/Akka.Cluster.Chunking)** (Apache-2.0),
|
||||
which exists precisely to bridge the pre-Artery gap this report is about: sending large messages
|
||||
(5 MB+) over Akka.Remote **without raising `maximum-frame-size` and without head-of-line
|
||||
blocking**. It chunks at the application layer on top of `Akka.Delivery`'s reliable-delivery
|
||||
machinery (ordered, ack'd, resend-on-loss), so every chunk is an ordinary sub-frame-cap remoting
|
||||
message and heartbeats/system messages interleave between chunks — directly defusing the §2
|
||||
guardrail concern with zero transport work.
|
||||
|
||||
Usage is pleasantly transparent: `ChunkingManager.For(system)` then
|
||||
`chunker.DeliverChunked(message, recipient, sender)` (Akka.Hosting: `.AddChunkingManager()`);
|
||||
config via `akka.cluster.chunking.chunk-size` (default 64 kB), `outbound-queue-capacity`
|
||||
(default 20), `request-timeout` (default 5 s). Unlike raw `Akka.Delivery`, no per-flow
|
||||
`ProducerController`/`ConsumerController` pair has to be hand-wired — the manager runs the
|
||||
delivery channels per node pair.
|
||||
|
||||
The honest caveats for our context:
|
||||
|
||||
- **Maturity.** v0.4.0 (Aug 2025), and the README says plainly it is "currently an experiment and
|
||||
is available on an as-is basis." For an OT product that's a real adoption bar; budget for
|
||||
owning/forking it if Petabridge doesn't graduate it.
|
||||
- **Acceptance ≠ delivery.** `DeliverChunked`'s task completes when the message is *accepted* for
|
||||
delivery, not when the far side has it — callers needing end-to-end confirmation still build
|
||||
their own ack on top.
|
||||
- **Requires Akka.Cluster, in-cluster only.** It rides remoting between cluster members. It does
|
||||
not cross the ScadaBridge site↔central boundary — two separate Akka clusters bridged by
|
||||
ClusterClient/gRPC — which is exactly where the family's big payloads actually flow.
|
||||
**Nor does ClusterClient itself chunk** (the tunneled payload + receptionist envelope must fit
|
||||
one frame). It *could* be bolted on — ClusterClient replies carry real IActorRefs, so raw
|
||||
`Akka.Delivery` could then run point-to-point over a direct cross-cluster remoting association —
|
||||
but that reopens exactly the direct-remoting coupling (firewall surface, quarantine interplay)
|
||||
the seed-partitioned boundary exists to avoid; large cross-boundary payloads stay on the gRPC
|
||||
side-channels.
|
||||
- **Correctness tool, not throughput tool** — shallow queues and the reliable-delivery handshake
|
||||
make it slower than plain tells; it removes a ceiling, it doesn't add speed.
|
||||
|
||||
Re-mapping §8 with this in hand changes no conclusion: mxaccessgw/HistorianGateway peers aren't
|
||||
cluster members; LocalDb sync is independent of cluster state *by design*; the ScadaBridge
|
||||
bulk/artifact side channels are cross-cluster, out of reach. Where it **is** the right answer is
|
||||
any *future within-cluster* large-message need (e.g. a big payload between the members of one
|
||||
pair): reach for Akka.Cluster.Chunking (or raw `Akka.Delivery` with
|
||||
`ProducerController.Settings.ChunkLargeMessagesBytes`, the in-box primitive it wraps) instead of
|
||||
raising the frame cap — until 1.6's Artery large-message lanes make both workarounds unnecessary.
|
||||
|
||||
## 9. Options
|
||||
|
||||
**A. Don't build it; tune what we have; track 1.6 (recommended).** Best-practice fit:
|
||||
@@ -289,8 +335,9 @@ transport-layer plumbing is exactly the code an OT product wants to own least, a
|
||||
design explicitly targets the wishlist (large-message channels, quarantine-bypass control channel —
|
||||
the latter directly relevant to the 2-node availability-first failover posture). Concretely: bump
|
||||
1.5.62 → 1.5.70 (eight releases of remoting/streams fixes, including a cluster-wedging
|
||||
consistent-hash-router bug), and if specific pains exist now, raise `maximum-frame-size` and/or
|
||||
enable the new dot-netty mTLS. Subscribe to #7466; when real alphas ship, the pair-restart
|
||||
consistent-hash-router bug), and if specific pains exist now, prefer Akka.Cluster.Chunking /
|
||||
`Akka.Delivery` chunking (§8a) over raising `maximum-frame-size` for any within-cluster large
|
||||
message, and/or enable the new dot-netty mTLS. Subscribe to #7466; when real alphas ship, the pair-restart
|
||||
discipline makes adoption cheap.
|
||||
|
||||
**B. Build a classic-SPI QUIC transport now.** ~3–5 weeks to production quality per §3. Choose this
|
||||
@@ -306,7 +353,8 @@ all get exercised) without owning it in production. A reasonable middle path.
|
||||
**Recommendation: A**, with **C** if hands-on evidence is wanted — the honest summary is that for
|
||||
this workload, an MsQuic transport on Akka 1.5 buys security and handshake ergonomics that are
|
||||
mostly available from config today, while the features that would actually reduce the gRPC
|
||||
side-channels only arrive with the Artery pipeline rewrite, which upstream hasn't started coding.
|
||||
side-channels only arrive with the Artery pipeline rewrite — now in active development upstream
|
||||
(see the §1 status update) — with Akka.Cluster.Chunking (§8a) as the interim in-cluster answer.
|
||||
|
||||
## Sources
|
||||
|
||||
@@ -317,3 +365,5 @@ side-channels only arrive with the Artery pipeline rewrite, which upstream hasn'
|
||||
- [Akka.NET remoting transports documentation](https://getakka.net/articles/remoting/transports.html)
|
||||
- [Akka.NET PR #7851 — mutual TLS for DotNetty transport](https://github.com/akkadotnet/akka.net/pull/7851)
|
||||
- [QUIC support in .NET (System.Net.Quic)](https://learn.microsoft.com/en-us/dotnet/fundamentals/networking/quic/quic-overview)
|
||||
- [petabridge/Akka.Cluster.Chunking](https://github.com/petabridge/Akka.Cluster.Chunking) — v0.4.0, Apache-2.0, experimental
|
||||
- [Akka.NET reliable delivery (Akka.Delivery)](https://getakka.net/articles/actors/reliable-delivery.html)
|
||||
|
||||
@@ -0,0 +1,351 @@
|
||||
# ScadaBridge: migrating cross-cluster ClusterClient traffic to gRPC-only
|
||||
|
||||
**Date:** 2026-07-22. Code-verified inventory of every ClusterClient touchpoint in
|
||||
`~/Desktop/ScadaBridge`, then the migration assessment. Companion to
|
||||
`akka_msquic_transport.md` §8/§8a (why chunking can't fix this path) and
|
||||
`akka_multidc_clusters.md` (why the site↔central boundary stays two clusters).
|
||||
|
||||
## 1. What ClusterClient actually carries today
|
||||
|
||||
Everything crossing the site↔central boundary funnels through **two actors** —
|
||||
`SiteCommunicationActor` (site) and `CentralCommunicationActor` (central) — which makes the
|
||||
migration surface unusually well-bounded.
|
||||
|
||||
**Receptionist registrations (3, all in `AkkaHostedService.cs`):**
|
||||
|
||||
| Role | Actor | Path | Note |
|
||||
|---|---|---|---|
|
||||
| Central | `CentralCommunicationActor` | `/user/central-communication` (`:436`) | the real workhorse |
|
||||
| Central | `ManagementActor` | `/user/management` (`:459`) | **vestigial** — intended for an out-of-cluster CLI (REQ-HOST-6a), but the in-repo CLI has no ClusterClient; nothing in the repo sends to it |
|
||||
| Site | `SiteCommunicationActor` | `/user/site-communication` (`:935`) | central→site target |
|
||||
|
||||
**Site → central** (single `central-cluster-client`, contacts from
|
||||
`CommunicationOptions.CentralContactPoints`, `AkkaHostedService.cs:942-953`) — 7 message types,
|
||||
all Ask/reply except the heartbeat (`SiteCommunicationActor.cs`):
|
||||
|
||||
| Message | Purpose | Guarantee layered on top |
|
||||
|---|---|---|
|
||||
| `NotificationSubmit` → `NotificationSubmitAck` | S&F notification delivery (central owns SMTP) | not-accepted/timeout → rows stay buffered; active-node-gated retry sweep |
|
||||
| `NotificationStatusQuery` → response | S&F status lookup | — |
|
||||
| `IngestAuditEventsCommand` → reply | audit batch push | ack = accepted-id list; throw-on-failure keeps SQLite rows Pending |
|
||||
| `IngestCachedTelemetryCommand` → reply | audit+SiteCall dual-write push | same contract |
|
||||
| `ReconcileSiteRequest` → response | per-node startup self-heal | — |
|
||||
| `SiteHealthReport` → ack | rich periodic health → `CentralHealthAggregator` | monotonic sequence numbers + dedup; failed-send counter restore |
|
||||
| `HeartbeatMessage` (Tell) | liveness, stamps `IsActive` | fire-and-forget by design |
|
||||
|
||||
**Central → site** (one ClusterClient **per site**, contacts = NodeA/NodeB from the config DB,
|
||||
`CentralCommunicationActor.cs:469-598`): every command wrapped in
|
||||
`SiteEnvelope(siteId, message)` — **29 command types** produced by `CommunicationService`
|
||||
(instance lifecycle, artifact deploy, OPC-UA browse/verify/tag read-write, cert trust,
|
||||
event-log/parked-message queries, integration routing, `TriggerSiteFailover`).
|
||||
|
||||
**Serialization: the default Newtonsoft JSON fallback** — no `serialization-bindings` anywhere.
|
||||
This is the amplifier in the known 128 KB frame bug
|
||||
(`docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md`): oversized
|
||||
ClusterClient messages are **silently dropped**, which is why deploy config already moved to
|
||||
HTTP notify-and-fetch.
|
||||
|
||||
**Not ClusterClient (stays untouched):** `DistributedPubSub` replica publishes
|
||||
(`SiteHeartbeatReplica`/`SiteHealthReportReplica`) are intra-central-cluster; all pair-internal
|
||||
Akka (singletons, membership, S&F actors) is intra-cluster.
|
||||
|
||||
## 2. Why gRPC-only is a natural completion, not a rewrite
|
||||
|
||||
The repo has already walked 3 of the 5 steps:
|
||||
|
||||
1. **`docs/plans/grpc_streams.md`** already rules ClusterClient "not built for high-throughput
|
||||
streaming" and moved all streaming values to gRPC — its stated end-state ("ClusterClient keeps
|
||||
command/control") is a waypoint, not a law.
|
||||
2. **Deploy config** already left ClusterClient for token-gated HTTP pull after the frame-size
|
||||
incident — with the rationale "fits the existing pull-based gRPC pattern".
|
||||
3. **The audit path is half-migrated already:** `sitestream.proto` *already defines*
|
||||
`IngestAuditEvents`/`IngestCachedTelemetry` with proto DTOs and an
|
||||
`ISiteStreamAuditClient` seam whose ClusterClient implementation is swappable — the contract
|
||||
and the seam exist; only the transport binding is Akka.
|
||||
4. Upstream direction agrees: JVM Akka deprecated ClusterClient in favor of gRPC for
|
||||
cross-cluster traffic; Akka.NET keeps it working but it is not where investment goes.
|
||||
5. Structural wins on arrival: the **silent-drop-over-128 KB failure class disappears** (gRPC
|
||||
default 4 MB cap + an explicit `ResourceExhausted` error instead of silence), Ask timeouts
|
||||
become real per-call deadlines, and **Akka remoting never crosses the boundary** — the
|
||||
site↔central firewall surface reduces to gRPC/HTTP ports, and the two clusters' quarantine
|
||||
lifecycles fully decouple (the coupling `akka_msquic_transport.md` §8a warns about).
|
||||
|
||||
## 3. What the migration takes
|
||||
|
||||
### 3.1 Topology decision — who serves what
|
||||
|
||||
Today the **site** is the only gRPC server (`SiteStreamService` on `:8083` h2c) and central the
|
||||
only client; but both sides already dial each other at the Akka level, and sites already dial
|
||||
central over HTTP for deploy fetch. So the clean shape is symmetric, no new firewall direction:
|
||||
|
||||
- **New `CentralControlService`** (gRPC) hosted on central's existing Kestrel (same host as the
|
||||
deploy-fetch endpoint): the 7 site→central RPCs.
|
||||
- **Extended site service** (grow `SiteStreamService` or a sibling `SiteCommandService` on the
|
||||
same `:8083` listener): the central→site command envelope.
|
||||
|
||||
(The alternative — a central-initiated duplex stream tunneling site→central pushes — avoids the
|
||||
central listener but re-implements connection management the receptionist gave for free; not
|
||||
worth it when sites already reach central over HTTP.)
|
||||
|
||||
### 3.2 Contract work
|
||||
|
||||
- **Site→central (small):** 7 RPCs. Audit/telemetry are a near-free move — proto DTOs exist and
|
||||
the swap is a new `ISiteStreamAuditClient` implementation (the `NoOp`/`ClusterClient`
|
||||
implementations prove the seam). Notifications, health report, reconcile, heartbeat need
|
||||
proto messages written (they're modest DTOs today).
|
||||
- **Central→site (the bulk):** 29 `SiteEnvelope` commands. Don't write 30 RPCs and don't smuggle
|
||||
JSON in one generic RPC — group by domain into a few RPCs with **`oneof` request/response
|
||||
envelopes** (lifecycle, deploy, OPC-UA browse/tag, queries, failover). Mirrors `SiteEnvelope`
|
||||
semantics with wire-level typing; mapper + golden tests are the real cost here.
|
||||
- **Transport seams:** the site side has precedent (`AkkaHealthReportTransport`,
|
||||
`ISiteStreamAuditClient`); `CommunicationService` (central) needs an `ISiteCommandTransport`
|
||||
seam so the 29 call sites don't change — the seam is introduced first, Akka behind it, then
|
||||
the gRPC implementation swaps in per domain group.
|
||||
|
||||
### 3.3 Semantics to preserve (all transport-agnostic, all carry over)
|
||||
|
||||
- S&F transient-failure contract: gRPC `Unavailable`/`DeadlineExceeded` → rows stay buffered —
|
||||
identical to today's not-accepted/timeout handling, with better error taxonomy.
|
||||
- Health sequence numbers + dedup + counter-restore: unchanged (they live above the transport).
|
||||
- Heartbeat stays fire-and-forget (unary, errors logged not thrown).
|
||||
- Dual-central routing: the receptionist let a site reach "whichever central answers"; the gRPC
|
||||
client needs try-in-order failover across the central A/B endpoints (`CentralContactPoints`
|
||||
becomes `CentralGrpcEndpoints`). The intra-central replica pub/sub already reconciles which
|
||||
central saw the report — unchanged.
|
||||
- Per-site channel management on central: the DB-driven contact cache refresh loop
|
||||
(`CentralCommunicationActor.cs:532-598`) carries over as a `GrpcChannel`-per-site factory —
|
||||
which `grpc_streams.md` (§167, §810) already models for the streaming path.
|
||||
|
||||
### 3.4 Work you must do anyway: auth — preshared key from the Secrets store (decided 2026-07-22)
|
||||
|
||||
`SiteStreamService` today has **no auth at all** — plaintext h2c, no interceptor (only the
|
||||
LocalDb sync path is ApiKey-gated, and only for its own service). This gap exists today
|
||||
regardless of migration — it just becomes indefensible after it. **Decision: the gRPC channels
|
||||
authenticate with a preshared key stored in `ZB.MOM.WW.Secrets`** (which ScadaBridge already
|
||||
adopted, `${secret:NAME}` resolution, fail-closed). Design:
|
||||
|
||||
- **Key scope: one PSK per site relationship** (e.g. secret name `SB-GRPC-PSK-SITE-A`), covering
|
||||
both directions of that pair's traffic (central→site commands AND that site's calls to
|
||||
`CentralControlService`). Central's store holds one secret per site; each site's local SQLite
|
||||
store holds only its own. Blast radius: a compromised site never yields another site's key.
|
||||
(A single fleet-wide PSK is simpler to seed but is rejected for exactly that reason.)
|
||||
- **Client side:** resolve `${secret:SB-GRPC-PSK-<site>}` at startup **fail-closed** (the
|
||||
Layer-B pattern the family already uses for driver secrets — boot fails loudly on a missing/
|
||||
undecryptable ref, no silent unauthenticated fallback) and attach it per-call as a bearer
|
||||
metadata header via a `CallCredentials`/interceptor on the channel pair from §3.5.
|
||||
- **Server side:** a fail-closed interceptor modeled on `LocalDbSyncAuthInterceptor` (the
|
||||
in-repo precedent) on **both** services — constant-time comparison, applied to the SiteStream/
|
||||
SiteCommand services on `:8083` and to `CentralControlService` on central. The LocalDb sync
|
||||
path keeps its own existing key; no sharing between the two mechanisms.
|
||||
- **Seeding/ops:** the `secret` CLI's *Reference audit & seed* flow (interactive console 0.3.0)
|
||||
is the deployment-setup tool — it classifies every `${secret:}` ref and seeds the gaps per
|
||||
node. Rotation: set the new value on both sides' stores, then restart the pair (pairs restart
|
||||
together by established discipline); if zero-blip rotation is ever needed, the server
|
||||
interceptor can accept old+new during a configured window.
|
||||
- **Transport honesty:** a bearer PSK over plaintext h2c is readable/replayable by anyone on
|
||||
path — today that's accepted via network isolation (same posture as the unauthenticated status
|
||||
quo, but now with authentication). TLS on these listeners is the follow-on hardening; the PSK
|
||||
design doesn't change if/when TLS lands.
|
||||
|
||||
### 3.5 Active-node resolution — does the gRPC client need to find the pair's active node?
|
||||
|
||||
**No — and this is load-bearing good news, verified in code.** Inbound commands on a site node
|
||||
are not executed locally: `SiteCommunicationActor` `Forward`s essentially everything to the
|
||||
**Deployment Manager cluster-singleton proxy** (`SiteCommunicationActor.cs:101-181`), and the
|
||||
code comments are explicit — the singleton "always lands on the active site node"; DCL
|
||||
connections/browse sessions "only exist on the singleton's node," so commands hop through the
|
||||
proxy by design (`:153-178`). The same shape holds centralward: `CentralCommunicationActor` runs
|
||||
on both central nodes, and heartbeat/health state is reconciled across the pair via the
|
||||
intra-cluster `DistributedPubSub` replicas. In other words, **each cluster already solves
|
||||
active-node routing internally with singleton proxies + pub/sub — the cross-cluster transport
|
||||
never had to know, and ClusterClient never did** (the receptionist delivers to whichever node
|
||||
answers).
|
||||
|
||||
A gRPC server on either pair node therefore keeps the exact same contract: accept the call, hand
|
||||
it to the same actor, let the singleton proxy route. What the migration must provide instead is
|
||||
cheap **either-node failover at the client**:
|
||||
|
||||
**Client failover/failback policy (per pair — applies to central's per-site clients AND the
|
||||
site's central client alike):**
|
||||
|
||||
- **Two channels, one preference.** Hold a `GrpcChannel` per endpoint (NodeA, NodeB) rather than
|
||||
one channel with re-dial logic; mark one *preferred* (config order, or `/health/active` probe
|
||||
once site-node health lands — Overview plan Phase 1). All calls go to the current channel.
|
||||
- **Failover:** on `Unavailable`/connect failure/readiness rejection
|
||||
(`SiteStreamGrpcServer` already returns `StatusCode.Unavailable` when not ready — the one
|
||||
state where the singleton proxy can't route), flip to the secondary **if available** and stay
|
||||
there (sticky-until-failure, so a flapping primary doesn't ping-pong every call).
|
||||
- **Failback:** a background probe (30–60 s cadence) checks the preferred endpoint
|
||||
(`grpc.health` check or `/health/ready`); when it answers again, new calls return to it —
|
||||
in-flight calls and open streams finish where they are. Failback is an optimization, never a
|
||||
correctness requirement: either node accepts and singleton-routes (§ above).
|
||||
- **Retry safety:** automatic cross-node retry ONLY for failures that provably never reached a
|
||||
server (connect refused, `Unavailable` before response headers). **Never blind-retry
|
||||
`DeadlineExceeded` on non-idempotent commands** (`WriteTag`, `DeployArtifacts`,
|
||||
`TriggerSiteFailover`) — ClusterClient's Ask had the same ambiguity and the layers above
|
||||
already tolerate it (S&F dedup, sequence numbers), but the migration must not *add* an eager
|
||||
retry that turns one write into two. Queries/browses are freely retryable.
|
||||
- **Streams don't fail over transparently:** an open `SubscribeInstance`/`SubscribeSite` stream
|
||||
dies with its channel; the subscriber re-subscribes via the policy above (the
|
||||
`grpc_streams.md` design already owns re-subscribe semantics — reuse, don't reinvent).
|
||||
- Cost of landing on the standby is one intra-pair remoting hop (as today).
|
||||
- One verification item for Phase 2: confirm none of the 29 command handlers bypasses the proxy
|
||||
and assumes local state (the audit/parked-message query paths are the ones to eyeball).
|
||||
|
||||
### 3.6 Deletions at the end
|
||||
|
||||
`ClusterClient`/`ClusterClientReceptionist` creation + registrations, `DefaultSiteClientFactory`,
|
||||
the `SiteEnvelope` Akka path, `CommunicationOptions.CentralContactPoints`. The
|
||||
`/user/management` registration is **deletable today** independent of everything else (no sender
|
||||
exists in the repo). `Akka.Cluster.Tools` stays for ClusterSingleton.
|
||||
|
||||
### 3.7 The management CLI is unaffected (deliberately out of scope)
|
||||
|
||||
`scadabridge.exe` never used ClusterClient: it speaks HTTP Basic to a Central's `/management`
|
||||
endpoints, which ask `ManagementActor` **in-process** (`ManagementEndpoints.cs:117` via
|
||||
`ManagementActorHolder`) — the `/user/management` receptionist registration was built for an
|
||||
out-of-cluster CLI that was never wired (REQ-HOST-6a) and has no sender in the repo. So this
|
||||
migration requires **zero CLI changes**; the CLI's transport is already boundary-independent.
|
||||
Retargeting the CLI onto the new gRPC services later is a separate, optional decision — it would
|
||||
buy one consistent management wire, but costs the CLI a gRPC stack and auth story for no current
|
||||
functional gain, so it is explicitly **not** part of this migration's scope or estimate.
|
||||
|
||||
## 4. Phasing and effort
|
||||
|
||||
| Phase | Content | Effort |
|
||||
|---|---|---|
|
||||
| 0 | Delete vestigial `ManagementActor` registration; add the §3.4 PSK-from-Secrets auth interceptor to `SiteStreamService` (standalone hardening win — central client attaches the PSK it resolves from its store) | days |
|
||||
| 1 | `CentralControlService` + the 7 site→central RPCs behind existing seams; audit swap; heartbeat/health/notifications/reconcile | ~1–2 wks |
|
||||
| 2 | `ISiteCommandTransport` seam + `oneof` command envelopes; migrate the 29 commands by domain group (deploy+lifecycle first, queries last) | ~2–3 wks |
|
||||
| 3 | Remove ClusterClient/receptionist + config cutover; live gate on the docker rig: kill central-a mid-S&F drain, site-node failover, deploy, `TriggerSiteFailover`, dual-central failover of site→central calls | ~1 wk |
|
||||
|
||||
Total ≈ **4–6 weeks** with live gating. Both transports can coexist per-path throughout (the
|
||||
seams make each path a one-line flip), so there is no big-bang cutover; rollback per phase is a
|
||||
config/DI revert.
|
||||
|
||||
**Risks, honestly:** the `oneof` envelope mappers are tedious and where bugs will hide (golden
|
||||
tests against captured payloads); ordering assumptions are low-risk (everything is request/reply
|
||||
or sequence-numbered, verify none of the 30 commands relies on ClusterClient's per-connection
|
||||
ordering across *different* commands); Ask-sender-forwarding tricks disappear (fine — RPC replies
|
||||
are direct, but check `SiteCallAuditActor`'s relay paths); the auth addition must not break the
|
||||
rig's current unauthenticated compose setup (env-gated like LocalDb's).
|
||||
|
||||
## 5. The other knob: raising the 128 KB frame cap
|
||||
|
||||
Since the frame limit motivates part of this migration, the obvious question — why not just
|
||||
raise `akka.remote.dot-netty.tcp.maximum-frame-size`? It is a legitimate knob, and if
|
||||
ClusterClient stays (Option B) it is worth doing deliberately, but know exactly what it buys:
|
||||
|
||||
- **How:** set `maximum-frame-size` (default 128000b) in the generated HOCON
|
||||
(`AkkaHostedService.BuildHocon` — today it never overrides it) and raise
|
||||
`send-buffer-size`/`receive-buffer-size` (default 256000b) to at least the new frame size.
|
||||
**The limit is enforced by the receiver**, so it must be raised on **every node of both
|
||||
clusters and rolled out everywhere before any sender relies on it** — a half-rolled-out raise
|
||||
reproduces the silent-drop bug in one direction. The pair-restart discipline makes the rollout
|
||||
cheap; the coordination requirement is the trap.
|
||||
- **What it buys:** headroom for today's fat-ish commands (browse results, query replies) with
|
||||
zero code change; the deploy-config class of payload would fit at, say, 2–4 MB.
|
||||
- **What it does not fix:** (1) the **silent-drop failure class survives** — the threshold moves,
|
||||
it doesn't disappear; anything over the new cap still vanishes without an error. (2)
|
||||
**Head-of-line gets worse, not better:** all ClusterClient traffic, heartbeats, and
|
||||
failure-detector signals share one ordered dot-netty channel, so a single 4 MB frame stalls
|
||||
everything behind it — on a 2-node availability-first posture with tight failure-detector
|
||||
timeouts, big frames are exactly what you don't want ahead of heartbeats (this is why the
|
||||
deploy fix chose notify-and-fetch over a cap raise, and why `akka_msquic_transport.md` §2
|
||||
treats the default cap as a guardrail, not a bug). (3) The default Newtonsoft serializer's
|
||||
double-JSON-escaping roughly **halves the effective budget** for JSON-bearing payloads. (4) It
|
||||
is a dead-end investment: Artery (1.6) replaces the pipeline, and Option A removes the
|
||||
constraint entirely (gRPC's 4 MB default cap fails loudly with `ResourceExhausted`, and is
|
||||
itself configurable per-call-type).
|
||||
|
||||
Verdict: as a *stopgap under Option B*, a modest raise (e.g. 512 KB–1 MB, buffers to match,
|
||||
cluster-wide) plus an explicit size check before `ClusterClient.Send` (log-and-reject instead of
|
||||
silent drop) is defensible. As part of Option A it's unnecessary — don't spend the rollout.
|
||||
|
||||
## 6. Options
|
||||
|
||||
**A. Full migration, phased as §4 (recommended).** Best-practice fit: it completes a direction
|
||||
the repo already chose twice under pressure (streaming, deploy), eliminates the silent-drop
|
||||
failure class on the control path, closes the unauthenticated-control-channel gap as part of the
|
||||
work, decouples the clusters' failure domains entirely, and lands on the transport the whole
|
||||
family already operates (gRPC + Kestrel + interceptors). Aligns with upstream's own deprecation
|
||||
of ClusterClient.
|
||||
|
||||
**B. Status quo per `grpc_streams.md` (ClusterClient keeps command/control).** Defensible if no
|
||||
current pain: the frame-size hazard is mostly defused (big payloads already moved) and the
|
||||
receptionist's discovery/failover is free; pair it with the deliberate cap raise + pre-send size
|
||||
check from §5. But it leaves two cross-cluster transports to operate,
|
||||
the silent-drop class alive for any future fat command, an unauthenticated gRPC service, and a
|
||||
dependency on the least-invested-in corner of Akka.NET. If choosing B, still do Phase 0 (auth +
|
||||
vestigial-registration deletion) — it stands alone.
|
||||
|
||||
**C. Wait for Akka.NET 1.6.** Not a reason to wait: Artery changes intra-cluster transport;
|
||||
ClusterClient's cross-cluster story doesn't improve. Deferring buys nothing here.
|
||||
|
||||
**Recommendation: A** — with Phase 0 done immediately regardless of the rest.
|
||||
|
||||
## 7. Design-completion deep dive (2026-07-22) — corrections and refinements
|
||||
|
||||
A three-track code deep dive (full message inventory; hosting/config/Secrets/tests; existing
|
||||
gRPC machinery) closed the remaining unknowns. The implementation plan lives at
|
||||
`ScadaBridge/docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`; corrections that supersede
|
||||
earlier sections:
|
||||
|
||||
1. **The seam is smaller than §3.2 assumed — one choke point per direction.** Every central→site
|
||||
producer (all of `CommunicationService` AND `SiteCallAuditActor`'s two relays) funnels
|
||||
`SiteEnvelope` through **`CentralCommunicationActor`**, which unwraps and routes via the
|
||||
per-site ClusterClient; every site→central path funnels through **`SiteCommunicationActor`**.
|
||||
So the transport swap happens *inside the two communication actors* (an injected
|
||||
`ISiteCommandTransport` / `ICentralTransport` behind a config flag) — `CommunicationService`
|
||||
needs **no interface extraction** and its ~20 consumers are untouched. (It has no interface
|
||||
today and is DI-registered concrete; extracting one is unnecessary.)
|
||||
2. **Exact command count: 29, and one is dead.** 27 from `CommunicationService` + 2
|
||||
`SiteCallAuditActor` relays. **Finding: `IntegrationCallRequest` is unwired in production** —
|
||||
`RegisterLocalHandler(Integration, …)` exists only in tests, so the command always answers
|
||||
"Integration handler not available." It is excluded from the gRPC contract (28 commands
|
||||
migrate) and filed as its own issue rather than silently ported or fixed.
|
||||
3. **§3.5 verification item RESOLVED.** Of the 29: 22 are singleton-/cluster-routed (incl.
|
||||
artifact deploy and event-log query, whose "local handler" slots are actually registered to
|
||||
singleton proxies — the event-log one deliberately, per wiring comments); `TriggerSiteFailover`
|
||||
resolves its target from cluster state, node-agnostic. The **only node-local target is the
|
||||
parked-message handler** (5 command types) — and it is standby-safe *by replication*, not
|
||||
routing: it reads/mutates the LocalDb-replicated S&F store, and the delivery sweep is
|
||||
active-gated. The gRPC design must preserve exactly that (dispatch on the receiving node, no
|
||||
forced active-node routing) — which either-node gRPC does.
|
||||
4. **Central hosting is net-new work.** Central has **no** `AddGrpc`/`MapGrpcService` and no
|
||||
explicit Kestrel listener (env-driven `:5000`, Traefik in front on HTTP/1). The migration adds
|
||||
an explicit HTTP/2 (h2c) listener on central (port `8083`, symmetric with sites; rig publishes
|
||||
it), `AddGrpc` + the PSK interceptor, and `CentralControlService`. Sites also get a **new**
|
||||
standing config `ScadaBridge:Communication:CentralGrpcEndpoints` — today the only central URL
|
||||
a site sees arrives *inside* `RefreshDeploymentCommand` (`CentralFetchBaseUrl`), so gRPC
|
||||
endpoint discovery genuinely needs new config.
|
||||
5. **PSK mechanics, concretized from the in-repo templates.** Server side copies
|
||||
`LocalDbSyncAuthInterceptor` exactly (service-prefix match, `authorization: Bearer`,
|
||||
`CryptographicOperations.FixedTimeEquals`, fail-closed on unset key, `PermissionDenied`);
|
||||
clients attach the key as call-level `Metadata` (the LocalDb sync client pattern). Key
|
||||
sourcing: **site nodes** read a static config key supplied as `${secret:SB-GRPC-PSK-<site>}`
|
||||
via the existing pre-host `SecretReferenceExpander` (`Program.cs:56-65`); **central** — whose
|
||||
site set is dynamic from the DB — resolves `SB-GRPC-PSK-{siteId}` at channel-build time via
|
||||
the runtime `ISecretResolver` (the MxGateway G-3 pattern), cached, fail-closed. Sites identify
|
||||
themselves to `CentralControlService` with an `x-scadabridge-site` metadata header so central
|
||||
verifies against that site's key. The rig mirrors the LocalDb dev pattern (same literal on
|
||||
both pair nodes; prod uses `${secret:}`).
|
||||
6. **Reuse map for §3.5's client policy:** per-site `GrpcNodeAAddress`/`GrpcNodeBAddress` columns
|
||||
already exist on the `Site` entity (streaming path reads them); primary/fallback single-retry
|
||||
exists in the pull clients; sticky flip + stability-window retry-budget exists in
|
||||
`DebugStreamBridgeActor`/`SiteAlarmAggregatorActor`. The plan extracts this into one shared
|
||||
`SitePairChannelProvider` rather than a fourth hand-rolled copy. **Caution — doc-vs-code gap:**
|
||||
`grpc_streams.md` names `CentralCommunicationActor.LoadSiteAddressesFromDb()` and
|
||||
singleton-aware node selection; neither exists in code (selection is blind A-then-B flip).
|
||||
7. **Deadlines are net-new.** No gRPC call in the repo sets a deadline or retry policy today; the
|
||||
new services introduce per-call deadlines mirroring the eight `CommunicationOptions` Ask
|
||||
timeouts (`DeploymentTimeout` 2 m … `HealthReportTimeout` 10 s), preserving the documented
|
||||
inner-before-outer timeout ordering (e.g. `SiteCallAuditOptions.RelayTimeout` 10 s <
|
||||
`QueryTimeout` 30 s).
|
||||
8. **Proto codegen is checked-in, not build-time** (protoc segfaults in the linux_arm64 build
|
||||
image): new protos follow the sitestream recipe — commented-out `<Protobuf>` item, generate on
|
||||
macOS, commit the generated C#, `docker/regen-proto.sh`.
|
||||
9. **Test assets to extend:** `Communication.Tests` already covers both communication actors with
|
||||
Akka TestKit (xunit **v2** — TestKit.Xunit2 compatible) + NSubstitute;
|
||||
`DirectActorSiteStreamAuditClient` is an explicitly-reusable wire-bypassing double for
|
||||
ingest-path integration tests.
|
||||
Reference in New Issue
Block a user