From dbfd5e7a7e662d8532381367ea8211db6be39334 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Wed, 22 Jul 2026 06:41:36 -0400 Subject: [PATCH] =?UTF-8?q?docs:=20Akka.NET=20transport=20deep-dives=20?= =?UTF-8?q?=E2=80=94=20MsQuic/1.6=20Artery=20and=20multi-DC=20clusters?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two research reports (2026-07-22): - akka_msquic_transport.md: what a QUIC transport on Akka 1.5.62 would take, the 1.5-vs-1.6 pipeline delta that gates the benefits, .NET-native alternatives (managed sockets ranked first), and measured throughput evidence (PR #4594: transport swap is throughput-neutral). - akka_multidc_clusters.md: JVM multi-DC semantics mapped onto the central-pair + site-pairs topology, the stalled Akka.NET port (#3261 / PR #4111), per-system fit vs the OtOpcUa mesh program, and the before-Phase-2 decision point. Recommendation in both: don't build; track upstream. --- akka_msquic_transport.md | 302 +++++++++++++++++++++++++++++++++++++++ akka_multidc_clusters.md | 241 +++++++++++++++++++++++++++++++ 2 files changed, 543 insertions(+) create mode 100644 akka_msquic_transport.md create mode 100644 akka_multidc_clusters.md diff --git a/akka_msquic_transport.md b/akka_msquic_transport.md new file mode 100644 index 0000000..00c8ea8 --- /dev/null +++ b/akka_msquic_transport.md @@ -0,0 +1,302 @@ +# Deep dive: an MsQuic transport for Akka.NET 1.5.62 + +*Research report, 2026-07-22. Question: what would it take to implement the Akka.NET 1.6 MsQuic +transport (akka.net #7466) against the Akka.NET version currently used by ScadaBridge and OtOpcUa, +and would features like higher throughput and larger-message support reduce the family's reliance +on gRPC?* + +**TL;DR:** Building a QUIC transport against the current Akka.NET (1.5.62) is genuinely feasible — +the classic transport SPI is public, and Akka's own docs literally use a hypothetical QUIC transport +as the custom-transport example. Roughly 3–5 weeks to production quality. But there's a structural +catch: plugged in under the classic 1.5 remoting pipeline, QUIC is reduced to "TCP with TLS 1.3 and +cheaper handshakes." The two things actually wanted — throughput multiplexing and non-blocking large +messages — come from the *Artery pipeline rewrite above the transport*, which is the part of #7466 +that doesn't exist yet and can't be replicated through the 1.5 SPI. And most of the family's gRPC +wouldn't go away regardless, because most of it isn't Akka-adjacent. Details below, options and a +recommendation at the end. + +--- + +## 1. Where #7466 actually stands (checked 2026-07-22) + +- [Issue #7466](https://github.com/akkadotnet/akka.net/issues/7466) is the **v1.6 "Artery" epic**, + opened Jan 2025: MsQuic-based Remoting V2 replacing the never-built Aeron plan. Design goals: + per-stream multiplexing ("this is where all of the significant performance gains come from" per + the [Petabridge v1.6 roadmap](https://petabridge.com/blog/akkadotnet-v1.6-roadmap/)), mandatory + TLS 1.3, automatic chunking of large messages onto dedicated streams, a system-message channel + that works even while quarantined, Akka.Streams backpressure, actor-ref compression, and a + 2–3M msg/sec target. Requires .NET 9+. +- **Status: design-only.** No linked PRs, no branches, no assignee; the only substantive activity is + a January 2025 design discussion (to11mtm/Aaronontheweb) about channel counts, ordering + guarantees, and serialization. The issue itself says a protocol spec is needed before + implementation starts. +- **The timeline has slipped badly.** The roadmap targeted alphas end of Q1 2025 and completion + early Q1 2026. As of July 2026 the + [releases page](https://github.com/akkadotnet/akka.net/releases) shows **zero 1.6 previews** — + latest is 1.5.70 (July 3, 2026), and a 1.5.71 milestone is open. The + [1.6.0 milestone](https://github.com/akkadotnet/akka.net/milestones) reads 44% complete, but the + closed items are the AOT/trimming/serialization workstreams, not the transport. A + [PR search for "quic"](https://github.com/akkadotnet/akka.net/pulls?q=is%3Apr+quic) confirms no + transport code has ever landed. + +So "wait for 1.6" is not a near-term plan. QUIC benefits this year would mean building it in-house. + +## 2. What the projects run today + +Both ScadaBridge and OtOpcUa pin **Akka 1.5.62** and use the classic **`dot-netty.tcp`** transport, +plaintext, with all size defaults — meaning the **128 KB `maximum-frame-size` cap** applies +(`ScadaBridge/src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs:310`; +`OtOpcUa/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:25`). Neither config touches TLS or +frame size. The Akka traffic is cluster gossip, heartbeats, singleton/pub-sub coordination, and +(ScadaBridge) actor messaging in a hub-and-spoke — modest volume, nowhere near transport-bound. + +Two facts that matter for the "do we even need this" question: + +- **The frame size can be raised today.** `akka.remote.dot-netty.tcp.maximum-frame-size = 10MiB` + (plus matching send/receive buffer sizes) is a config change, not a transport change. The 128 KB + default is a *guardrail*, not a TCP limitation — large messages on the single ordered association + channel delay heartbeats behind them, which is exactly the false-failure-detection risk the + 2s/10s failure-detector tuning cares about. +- **TLS on the existing transport is already available.** Akka.Remote's DotNetty transport gained + **mutual TLS** in [PR #7851](https://github.com/akkadotnet/akka.net/pull/7851) (merged Oct 2025 — + included in the pinned 1.5.62). If encryption-in-flight is a driver, QUIC is not required for it. + +## 3. How a transport plugs into 1.5 — and what it would take + +Akka.Remote has a public custom-transport SPI +([docs](https://getakka.net/articles/remoting/transports.html)): implement +`Akka.Remote.Transport.Transport`, register it via HOCON, and everything above it (handshake, +endpoint management, quarantine, cluster) is unchanged: + +```hocon +akka.remote { + enabled-transports = ["akka.remote.zb-quic"] + zb-quic { + transport-class = "ZB.MOM.WW.Akka.Transport.Quic.QuicTransport, ZB.MOM.WW.Akka.Transport.Quic" + transport-protocol = quic # addresses become akka.quic://System@host:port + hostname = "0.0.0.0" + port = 4053 # UDP now, not TCP + } +} +``` + +The concrete build, as a shared lib in scadaproj (the usual pattern): + +**Core (~1–2 weeks):** +- `QuicTransport : Transport` — ctor `(ActorSystem, Config)` (instantiated reflectively); implement + `SchemeIdentifier` (`"quic"`), `MaximumPayloadBytes` (configurable — this is where "large + messages" becomes trivial), `Listen()` (a `QuicListener.ListenAsync` with a private ALPN like + `"akka-remote"`, surfacing `InboundAssociation` events), `Associate(Address)` + (`QuicConnection.ConnectAsync` + open one bidirectional `QuicStream`), `Shutdown()`. +- `QuicAssociationHandle : AssociationHandle` — `Write(ByteString)` must be **non-blocking with a + boolean backpressure signal** (return `false` when the send queue is full; Akka's `EndpointWriter` + buffers and retries — getting this semantic right is the subtle part), `Disassociate()` (map to + `CloseAsync` with an app error code). +- **Framing** — QUIC streams are byte streams like TCP, so the 4-byte length-prefix framing that + dot-netty does must be re-implemented on both sides. The Akka protocol handshake/PDU layer rides + on top automatically. + +**Security config (~0.5–1 week):** QUIC **mandates TLS 1.3** — there is no plaintext mode. Every +node needs a certificate, and a validation policy is required (mutual TLS with a pinned internal CA +is the sane one; the Secrets lib is a natural home for cert material). This is new operational +surface the rigs don't have today. + +**Hardening + tests (~1–2 weeks):** association teardown races, quarantine behavior, reconnect +storms, failure-detector behavior under load, big-frame tests, and soak on the docker-dev rig. +Historical note for calibration: the Helios→DotNetty transition took Akka.NET years of corner-case +fixes (dissociation races, quarantine storms). A custom transport is its own support island for +that class of bug. + +**Migration:** the address scheme changes (`akka.tcp://` → `akka.quic://`), so seed nodes and every +stored address change; mixed-transport rolling upgrade isn't practical for a pair — but *both* +transports can be listed in `enabled-transports` during transition, and the site pairs already +stop/start together by policy, so cutover is easy here. + +## 4. The structural catch + +The classic SPI models an association as **one ordered, opaque byte channel**. The remoting layer +above it (system-message resend, handshake, cluster failure detection) *depends* on per-association +ordering, and the transport never sees message recipients — just framed byte blobs. Consequences: + +- **No multiplexing win.** One association cannot be fanned across N QUIC streams without either + breaking ordering guarantees or parsing Akka's PDU envelope inside the transport to route by + recipient while preserving per-pair ordering — at which point you are rewriting Artery's + dispatcher against internals with no stability contract. +- **No dedicated large-message or control channel.** A 10 MB frame still queues ahead of heartbeats + on the same stream — identical head-of-line behavior to TCP today. The 1.6 design fixes this + *above* the transport; the 1.5 SPI can't express it. +- What you *do* get: TLS 1.3 built in, faster connection establishment (nice for + reconnect-after-failover, though failover latency is dominated by the 10–15s + failure-detector/downing windows, not handshakes), connection migration, and an easy big + `MaximumPayloadBytes`. + +Net: a 1.5-SPI QUIC transport ≈ **dot-netty.tcp + mTLS + a raised frame size** — both of which are +available today with config on the stock transport. + +## 5. What 1.6 actually changes — the layer split, made explicit + +1.6 doesn't just swap the transport; it rewrites the remoting pipeline *above* the transport +(Artery), and that pipeline is where every benefit §4 marked "not achievable on 1.5" actually lives. + +**In 1.5**, remoting is two layers: the pluggable `Transport` SPI at the bottom (a dumb ordered byte +pipe — `Associate()`, `Write(ByteString)`, `Disassociate()`) and above it the fixed pipeline: +`AkkaProtocolTransport` handshake, one `EndpointWriter`/`EndpointReader` actor pair per association, +system-message resend buffering, and quarantine. That pipeline bakes in three assumptions no custom +transport can change: + +1. **One association = one ordered channel** — the system-message resend protocol and cluster + failure detection depend on strict ordering over that single channel. +2. **Payloads are opaque to the transport** — recipient, message class, and size are invisible by + the time bytes reach `Write()`, so "big message → own lane" or "heartbeat → priority lane" + routing is impossible without parsing Akka's internal PDU format (i.e. reimplementing the layer + above against unstable internals). +3. **Everything shares the lane** — user messages, system messages, cluster heartbeats. The 128 KB + frame guardrail exists exactly because a big message parking in front of heartbeats trips the + failure detector. + +**In 1.6/Artery**, that middle layer is replaced with a recipient-aware, multi-lane, Akka.Streams- +based pipeline: it knows recipient and size *before* the wire, assigns messages to QUIC streams +accordingly (ordering preserved per sender→receiver pair rather than per node pair), chunks +oversized messages onto **dedicated large-message streams**, keeps **system/control messages on +their own stream** (including a quarantine-bypass channel), adds headers for version negotiation, +and compresses actor refs to integers. QUIC is almost incidental — it is chosen because it gives +cheap independent streams for that pipeline to map onto. + +| Benefit | 1.5 + custom QUIC transport | 1.6 Artery | Why the pipeline is the difference | +|---|---|---|---| +| Throughput / multiplexing | ✗ (single ordered lane) | ✓ (the 2–3M msg/s target) | Lane assignment needs recipient visibility, which only the pipeline has | +| Large messages without side channels | ~ (raise the frame cap; heartbeats still queue behind big frames) | ✓ (dedicated chunked streams; control lane unaffected) | Size-based routing happens above the transport | +| Honest failure detection under load | ✗ (heartbeats share the data lane) | ✓ (isolated control stream) | Same | +| Split-brain handling while quarantined | ✗ | ✓ (quarantine-bypass channel) | Quarantine lives in the pipeline, not the transport | +| TLS 1.3 | ✓ (but dot-netty mTLS gets there too) | ✓ | Transport-level — the one benefit 1.5 delivers fully | +| Fast handshake / connection migration | ✓ | ✓ | Transport-level | + +Consequences for this report's conclusions: + +- **The "don't build it on 1.5" recommendation strengthens** — a 1.5 transport captures only the + bottom two rows, and both have cheaper paths (dot-netty mTLS; config). +- **The gRPC-alleviation answer partially flips at 1.6, but only for Akka-adjacent channels.** + ScadaBridge's HTTP notify-and-fetch for deploy configs, and any "too big for the frame cap" + workaround, exist purely because of rows 2–3; under Artery those payloads could legitimately ride + remoting. The genuinely non-Akka gRPC (mxaccessgw, HistorianGateway, LocalDb's deliberate + cluster-independence) stays gRPC regardless — see the mapping table below. +- **New costs arrive with the benefits:** Artery is a new wire protocol (flag-day cutover, no + rolling mix with 1.5 remoting — tolerable given the stop-pairs-together discipline), .NET 9+ + only, and it drags in the platform/ops constraints below (libmsquic, UDP firewall rules, Windows + Server 2022+, macOS dev friction) — those attach to *any* QUIC adoption, 1.5 or 1.6. + +## 6. Platform and ops constraints ([System.Net.Quic](https://learn.microsoft.com/en-us/dotnet/fundamentals/networking/quic/quic-overview)) + +- APIs are stable since .NET 9; the apps are .NET 10 — fine. +- **Windows: requires Windows 11 / Server 2022+** (older versions lack the TLS 1.3 crypto APIs). + ⚠️ Check the `wonder-*` boxes — if any production VM is Server 2019 or older, QUIC remoting is a + hard no there. +- **Linux:** `libmsquic` must be installed in every image — the `aspnet:10.0`-based images and the + docker-dev rig would all need `apt-get install libmsquic` from packages.microsoft.com. +- **macOS (dev machine): only partially supported** — Homebrew `libmsquic` + a + `DYLD_FALLBACK_LIBRARY_PATH` export, outside Microsoft's test matrix. Offline test suites that + spin up real ActorSystems with remoting would grow a fragile native dependency on the Mac. +- **Firewalls:** remoting moves from TCP to **UDP** — every rule for 4053/etc. needs a UDP twin, + and OT network gear is notoriously unkind to UDP. + +## 7. .NET-native alternatives to MsQuic + +Under the 1.5 SPI every transport degenerates into the same ordered byte pipe (§5), so candidate +wire protocols differ on **dependencies, platform reach, and TLS posture** — not on features. By +that metric, ranked: + +**1. Managed sockets + `SslStream` + `System.IO.Pipelines` — the best fit if anything gets built.** +100% in-box managed .NET: zero native dependencies, runs everywhere the apps run today (macOS test +suites, any Windows Server version, stock `aspnet:10.0` images, TCP firewall rules unchanged). +TLS 1.3 via `SslStream` is *optional* — can match the current plaintext posture with an mTLS +upgrade path, unlike QUIC's hard TLS mandate. Arbitrary frame sizes. Strategically this is a +"modern DotNetty replacement": it sheds the effectively unmaintained DotNetty dependency, which is +one of 1.6's own goals. JVM precedent that this is respectable: JVM Artery ships TCP and TLS-TCP +modes alongside Aeron, and TCP Artery reached ~650k msg/s — most of Artery's gain is the pipeline, +not the wire. What it gives up vs QUIC: fast handshake and connection migration — neither of which +was a driver here. + +*Throughput expectations, from real data:* this exact idea was prototyped by to11mtm in +[PR #4594](https://github.com/akkadotnet/akka.net/pull/4594) (Oct 2020, draft, never merged) — an +Akka.Streams/managed-socket TCP transport under the classic pipeline. On the author's i7-8750H: +DotNetty baseline ~220–240k msg/s; the managed transport alone ~170–240k (parity at best); adding +aggressive zero-copy work + a hand-written protobuf deserializer reached ~240–300k (+10–25%), and +the residual bottlenecks identified were all *pipeline*, not wire: serialization, the PDU codec, +`EndpointWriter` dispatch, blocking `.Result` in serialization paths. I.e. the transport swap is +throughput-neutral; only attacking the layer above moves the number, and even extreme effort caps +near +25%. (Reference points: the official +[RemotePingPong docs](https://getakka.net/articles/remoting/performance.html) show 82k → 141k msg/s +from flush-batching alone on older hardware; modern hardware on 1.5.x lands roughly 300–600k per +association; the 1.6 Artery target is 2–3M.) Two mitigating notes: per-*association* ceilings apply +per site pair, so hub-and-spoke aggregate scales with site count; and for large frames (the raised +`maximum-frame-size` case) a managed transport saturates the wire in bytes/sec — the msg/s ceiling +is a small-message phenomenon. + +**2. HTTP/2 — fully managed multiplexing, but the SPI can't use it.** Kestrel (server) + +`SocketsHttpHandler` (client), or gRPC bidirectional streaming as the framing, are fully managed +h2 with real stream multiplexing and h2c (no cert mandate). But the 1.5 SPI locks the multiplexing +away for exactly the §5 reasons, h2 streams still share one TCP connection (packet-level +head-of-line blocking remains), and it adds protocol overhead for no captured benefit. Only +distinct win is riding existing Kestrel/gRPC infrastructure. Not preferred over raw sockets. + +**3. WebSockets (`System.Net.WebSockets`)** — managed, but a single ordered stream, i.e. TCP with +extra framing. Only interesting for exotic proxy/firewall traversal. Skip. + +**No managed QUIC exists.** `System.Net.Quic` *is* the msquic interop — there is no fully managed +QUIC implementation in .NET, and nothing else in the box provides independent streams. So "QUIC +without libmsquic" is not on the table, on any .NET version. + +Forward-looking note: if Akka.NET's 1.6 Artery ships QUIC-only, libmsquic becomes unavoidable *at +that point* — worth watching whether upstream adds a TCP mode (to11mtm explicitly advocated +transport pluggability for firewall-restricted environments in the #7466 design discussion, citing +the JVM's TCP Artery). + +## 8. Would it actually alleviate the gRPC? + +Mostly no — mapping the family's gRPC surfaces: + +| gRPC usage | Could Akka-over-QUIC replace it? | +|---|---| +| mxaccessgw ↔ clients | **No.** It exists for COM bitness isolation; the worker is .NET 4.8 x86, which can't even load a .NET 9+ QUIC stack, and callers aren't cluster members. | +| HistorianGateway API | **No.** A standalone service contract for multiple heterogeneous consumers — service RPC is the right shape. | +| LocalDb pair sync | **Technically closest, but deliberately no.** It runs between the same nodes as the Akka cluster, but its whole value is being independent of cluster state (it's what lets a node boot from cache when things are degraded), and it's consumable by non-Akka apps. Coupling it to remoting would weaken it. Its 4 MB cap is a gRPC config choice, not a law. | +| ScadaBridge site→central (S&F etc.) | Already Akka where Akka fits; the bulk/artifact paths that avoid remoting avoid it because of the single-channel head-of-line problem — which, per §4, a 1.5 QUIC transport does not fix. | + +The "big messages so we don't need side channels" hope is really an Artery feature (dedicated +chunked large-message streams), not a QUIC-transport feature. + +## 9. Options + +**A. Don't build it; tune what we have; track 1.6 (recommended).** Best-practice fit: +transport-layer plumbing is exactly the code an OT product wants to own least, and the upstream +design explicitly targets the wishlist (large-message channels, quarantine-bypass control channel — +the latter directly relevant to the 2-node availability-first failover posture). Concretely: bump +1.5.62 → 1.5.70 (eight releases of remoting/streams fixes, including a cluster-wedging +consistent-hash-router bug), and if specific pains exist now, raise `maximum-frame-size` and/or +enable the new dot-netty mTLS. Subscribe to #7466; when real alphas ship, the pair-restart +discipline makes adoption cheap. + +**B. Build a classic-SPI QUIC transport now.** ~3–5 weeks to production quality per §3. Choose this +only if the *specific* wins (TLS 1.3 with modern cert handling, UDP traversal, fast reconnect) +matter enough to own a transport — knowing it delivers no throughput/multiplexing gain and gets +obsoleted by 1.6. + +**C. Weekend-scale spike, not production.** Implement the minimal `Transport`/`AssociationHandle` +pair, run a two-node pair on the rig, benchmark against dot-netty. Genuinely worthwhile as +de-risking/learning for the eventual 1.6 migration (certs, libmsquic packaging, UDP firewall rules +all get exercised) without owning it in production. A reasonable middle path. + +**Recommendation: A**, with **C** if hands-on evidence is wanted — the honest summary is that for +this workload, an MsQuic transport on Akka 1.5 buys security and handshake ergonomics that are +mostly available from config today, while the features that would actually reduce the gRPC +side-channels only arrive with the Artery pipeline rewrite, which upstream hasn't started coding. + +## Sources + +- [akka.net #7466 — Epic: Akka.Remote Quic-based Transport (Artery)](https://github.com/akkadotnet/akka.net/issues/7466) +- [Petabridge — Akka.NET v1.6 Roadmap](https://petabridge.com/blog/akkadotnet-v1.6-roadmap/) +- [Akka.NET releases](https://github.com/akkadotnet/akka.net/releases) +- [Akka.NET milestones](https://github.com/akkadotnet/akka.net/milestones) +- [Akka.NET remoting transports documentation](https://getakka.net/articles/remoting/transports.html) +- [Akka.NET PR #7851 — mutual TLS for DotNetty transport](https://github.com/akkadotnet/akka.net/pull/7851) +- [QUIC support in .NET (System.Net.Quic)](https://learn.microsoft.com/en-us/dotnet/fundamentals/networking/quic/quic-overview) diff --git a/akka_multidc_clusters.md b/akka_multidc_clusters.md new file mode 100644 index 0000000..92a371d --- /dev/null +++ b/akka_multidc_clusters.md @@ -0,0 +1,241 @@ +# Deep dive: Akka Multi-DC cluster support, ported to Akka.NET, applied to our topology + +*Research report, 2026-07-22. Question: Akka (JVM) has multi-DC cluster support that Akka.NET does +not. Given our layout (central 2-node cluster + N site 2-node clusters), what would porting it take, +and how would it be used in our systems?* + +**TL;DR:** Multi-DC maps onto our topology almost perfectly — "one DC per pair" gives per-site +leadership, per-site singletons, per-site oldest-node primaries, and WAN-partition tolerance inside +ONE cluster, which is precisely the problem set the OtOpcUa per-cluster mesh program and +ScadaBridge's three bespoke bridge transports were built to solve. But the port is core-gossip +surgery that has already defeated the Akka.NET maintainers twice (PRs #3284 and #4111, the latter a +67-commit functionally-complete draft abandoned in 2019 over multi-node-test instability, now +milestoned to the far-future 2.0.0). And two of our hardest cross-cluster problems — the 128 KB +frame limit and bulk data movement — are *not* clustering problems and survive multi-DC untouched. +Recommendation: don't port; keep the mesh-program direction; treat #4111 as a candidate for an +upstream-contribution play only if the mesh program's transport phases (2/5) prove heavier than +expected. Details and options below. + +--- + +## 1. What Akka multi-DC actually is (JVM; GA since Akka 2.5.6, 2017) + +One correction to the framing: this isn't an upcoming JVM feature — it has been GA on the JVM side +since 2017. What's true is that it never made it into Akka.NET. Semantics (from the +[Akka 2.6 docs](https://doc.akka.io/docs/akka/2.6/typed/cluster-dc.html) — 2.6.20 is the last +Apache-2.0 version, which matters for porting; 2.7+ is BSL-licensed): + +- **DC membership is just a role.** `akka.cluster.multi-data-center.self-data-center = "site-a"` + adds role `dc-site-a`. One cluster, every member tagged. Default DC is `"default"` — which is why + untagged and tagged nodes can't mix (JVM rolling 2.4→2.5 upgrades hit exactly this, akka/akka + #25496). +- **One leader per DC.** Each DC's leader performs joining/up/leaving/removal transitions for *its + own* members only. Crucially: **members can join and leave while DCs are partitioned from each + other** — impossible in a plain cluster, where any unreachable member blocks convergence and + therefore blocks all leader actions fleet-wide. +- **Two failure detectors.** `akka.cluster.failure-detector` intra-DC (all-to-all within the DC, + as today) and `akka.cluster.multi-data-center.failure-detector` cross-DC, run only by the + `cross-data-center-connections` (default 5) oldest nodes per DC. Cross-DC unreachability raises + `UnreachableDataCenter` / `ReachableDataCenter` events — it is treated as "the WAN link is bad," + explicitly NOT fed into member downing: *"For network partitions between data centers the system + should typically not down the unreachable nodes, but instead wait until it heals or a decision is + made by a human or external monitoring system."* +- **Cluster Singleton is per-DC.** Start the manager on all nodes with 3 DCs defined → 3 live + singletons, one per DC. Proxies default to the local DC and take a `dataCenter` parameter to + target another DC's singleton. A "global" singleton = start its manager only in one designated DC + and accept unavailability when that DC is cut off. +- **Cluster Sharding is per-DC** (own coordinator/regions per DC; cross-DC proxies available). + Not relevant to us — neither app uses sharding. +- **Distributed Pub/Sub is NOT DC-scoped.** The mediator replicates its registry among *all* + cluster nodes (optionally restricted by role) — so topics span every DC by default. This is the + single most valuable property for OtOpcUa (§4). +- Singleton placement and per-DC leadership both key off **oldest member within the DC** — i.e. + the JVM feature's per-DC primary rule is literally our `ActiveNodeEvaluator.SelfIsOldestUp` / + OtOpcUa Phase-0b `IsDriverPrimary` rule, applied per pair. + +## 2. Where Akka.NET stands on it + +- [Issue #3261](https://github.com/akkadotnet/akka.net/issues/3261) (opened Jan 2018 by + Aaronontheweb) is the tracking issue. Current milestone: **2.0.0 — Akka.Typed**, i.e. no + committed timeline; it is not in the 1.6 plan (1.6 = Artery/QUIC, .NET Standard drop, new + Streams materializer — see the companion report `akka_msquic_transport.md`). +- It was **implemented once**: Horusiath's [PR #3284](https://github.com/akkadotnet/akka.net/pull/3284) + (2018, milestoned 1.4.0, closed), preserved by Aaron as draft + [PR #4111](https://github.com/akkadotnet/akka.net/pull/4111) (Dec 2019, still open). 67 commits + touching membership state, gossip, cross-DC heartbeats, singleton/sharding DC-awareness, failure + detection, cluster events, and serialization, plus multi-node specs for split-brain and + sunny-weather scenarios. It reportedly passed on .NET Framework but **died on Multi-Node TestKit + instability on .NET Core** ("151 tests failing" at one point; deadlocks in MembershipState lazy + init, convergence races, CoordinatedShutdown exit-hook problems). It was dropped from 1.4.0 for + review/verification bandwidth and never revived. Its base is **mid-2018 `dev`** — six years and + two major version lines behind today's 1.5.70. + +So "porting the support" concretely means: re-porting from the JVM 2.6.20 sources (Apache 2.0 — +do NOT copy from 2.7+/BSL) with #4111 as a .NET-idiom reference, onto Akka.NET 1.5.x, and carrying +it as a fork until upstream lands its own in 2.0.0. + +## 3. The mapping onto our topology + +The fit is genuinely striking. Model: **every 2-node pair is a DC** in one spanning cluster. + +| DC | Members | Today | +|---|---|---| +| `dc-central` | central-a, central-b | ScadaBridge central cluster / OtOpcUa admin pair | +| `dc-site-a` | site-a-1, site-a-2 | separate site cluster (SB) / same mesh (OtOpcUa) | +| `dc-site-b` … | … | … | + +What each multi-DC property buys us: + +- **Per-DC oldest = pair primary, by construction.** The exact rule both products converged on + (ScadaBridge `ActiveNodeEvaluator.SelfIsOldestUp`; OtOpcUa Phase 0b) becomes the framework's own + scoping — the mesh design's core motivation ("every Primary-gated decision asks a cluster-wide + election about a pair-local resource") dissolves. +- **Per-DC singletons.** OtOpcUa's `admin-operations` singleton stays effectively central by + starting its manager only on `dc-central` nodes; per-pair singletons (e.g. a per-site drain + coordinator) come free. +- **WAN-safe failure handling.** Intra-pair failure detection + our `auto-down`(15s) + availability-over-partition-safety posture stays *within the pair*, where we want it. A WAN blip + to a site raises `UnreachableDataCenter` — no downing, no quarantine churn, and (the big one) + **membership operations at other sites keep working during the outage**. Today's OtOpcUa single + mesh has neither property, which is a large part of why the mesh program exists. +- **Pub/sub spans DCs.** OtOpcUa's nine DPS topics + deploy notify channel keep working unchanged + across the whole fleet. The Secrets Akka replicator (DPS-based) would also span everything — + today it explicitly cannot cross ScadaBridge's separate clusters (why secrets use SQL-hub mode + there). +- **Direct actor messaging replaces ClusterClient.** In one spanning cluster, receptionist + contact-point management, per-site client caching, and DB-driven address refresh + (`CentralCommunicationActor`'s `_siteClients`, the 60 s `Site`-row polling) reduce to ordinary + cluster-aware messaging. + +### What multi-DC does NOT fix (important) + +- **The 128 KB frame limit and serialization posture are unchanged.** ScadaBridge's HTTP + notify-and-fetch (deploy configs) and gRPC streaming (`:8083`, high-volume attribute/alarm + events) exist because Akka messaging is the wrong channel for bulk payloads — that remains true + in a multi-DC cluster. Those channels stay. (Same conclusion as the MsQuic report: bulk channels + aren't clustering problems.) +- **Store-and-forward stays.** At-least-once site→central delivery with a durable buffer + (`StoreAndForwardService` + replicated `sf_messages`) is an application guarantee; multi-DC + changes the pipe, not the need for the buffer. +- **Security got harder, not easier.** Today an attacker on the WAN can reach ClusterClient/gRPC + seams; in a spanning cluster they can attempt to *join the cluster*. Akka remoting is + unauthenticated h2c-equivalent in both products (accepted risk, mesh design §6.3) — a spanning + cluster raises the value of revisiting that. +- **Persistence single-writer caveats** (the JVM docs' sharding-across-DCs warning) don't bite us — + no sharding, and pair-local stores are LocalDb-replicated with HLC/LWW already. + +## 4. Per-system assessment + +### OtOpcUa — the big beneficiary, but the timing is awkward + +OtOpcUa *today* is one mesh spanning everything — i.e. it is already shaped like a multi-DC +cluster, minus the DC semantics. The per-cluster mesh program +(`OtOpcUa/docs/plans/2026-07-22-per-cluster-mesh-program.md`, Phases 1–7) exists to fix exactly the +things DC-tagging would fix in place: + +| Mesh-program phase | Under multi-DC | +|---|---| +| 0a auto-down / 0b oldest-primary | Already done; both carry over unchanged (intra-DC) | +| 1 `ClusterNode` address columns, DB-sourced ack set | **Mostly unnecessary** — membership stays visible fleet-wide | +| 2 ClusterClient comm actors + receptionists | **Unnecessary** — one cluster, direct messaging | +| 3 config fetch-and-cache from central | **Still required** — driven by "sites have no SQL Server," not by clustering | +| 4 cut driver-side ConfigDb | **Still required** — same reason | +| 5 gRPC migration of 7 observability topics | **Unnecessary** — DPS keeps working | +| 6 mesh partition into pairs | Replaced by DC tagging + seed layout (much smaller) | +| 7 failover drills / live gates | Still required, plus new cross-DC drills | + +So multi-DC would delete the two transport-migration phases (2 and 5 — the heaviest pure-plumbing +work) and shrink 1 and 6, while leaving the genuinely deployment-driven phases (3, 4) intact. That +is a real prize. It is also exactly why the timing is awkward: the program was just sequenced on +the ScadaBridge-proven separate-cluster shape, and switching horses means betting those phases on a +forked cluster core instead of on patterns already running in production-adjacent code. + +### ScadaBridge — already solved this; migration value is low + +ScadaBridge built the separate-cluster bridges years-of-work deep and they are live: per-site +`ClusterClient`s with DB-driven contact refresh, per-node receptionist registration, gRPC +server-on-site/client-on-central streaming, token-gated HTTP deploy fetch, S&F with active-node +gating. Two details actually make it multi-DC-*ready* — every cluster already shares the one +ActorSystem name `"scadabridge"` (separation is purely seed-node partitioning), and the roles +already include `site-{SiteId}` — but what multi-DC would replace is only the ClusterClient control +channel and the address-book machinery. The gRPC and HTTP channels stay (frame limit), S&F stays, +and the CLI's ClusterClient entry point would need rework. Replacing a proven, tested seam with a +forked gossip layer for a cleaner control channel is a poor trade on its own. + +## 5. What the port itself takes + +Scope (from the JVM 2.6.20 implementation and the #4111 diff): + +1. **Membership/gossip core** — `Member` gains a `dataCenter` (derived from the `dc-` role, so the + gossip **wire format is nearly unchanged** — the DC rides in roles); `MembershipState`/gossip + convergence, leader election, and reachability calculations all become DC-filtered (convergence + ignores cross-DC observations; per-DC leader actions permitted during inter-DC splits). This is + the delicate 20% that is 80% of the risk. +2. **Heartbeating** — keep today's all-to-all within a DC; add the cross-DC heartbeater run by the + N oldest per DC with its own failure-detector config; new `UnreachableDataCenter` / + `ReachableDataCenter` events. +3. **Downing** — `AutoDowning`/SBR must become DC-scoped (JVM SBR is; our `auto-down` posture then + applies per pair, which is exactly the HA stance we want). A non-DC-aware downing provider in a + multi-DC cluster is actively dangerous (downs the far side on WAN blips). +4. **Cluster tools** — Singleton manager/proxy DC-awareness (small — it is role-filtering); + sharding proxies (skippable for us); DPS needs nothing. +5. **Serialization** — additive protobuf changes for the new heartbeat/gossip fields; whole-fleet + cutover required (mixed DC-aware/unaware nodes don't interoperate — the JVM had the same + constraint). Our stop/start-pairs-together discipline makes this tolerable, but it is a + flag-day, not rolling. +6. **Verification — the real cost.** This is what killed it upstream twice: MNTK specs for + split-brain, join-during-partition, cross-DC unreachability, singleton-per-DC handover. Our + substitute would be rig-based live gates (docker-dev with real WAN-partition injection via + `tc`/iptables), which we're good at — but the failure modes are convergence races that soak + tests catch and unit tests don't. + +**Effort honestly:** the code port is plausibly 4–8 weeks against the Apache-2.0 JVM reference with +#4111 as a map. The validation to *trust* it under an OT platform's availability requirements is +the same again or more, and then it's a **permanent fork of Akka.Cluster** — every 1.5.x bump +(there have been 8 since our pin) becomes a merge with the most intricate code in the framework, +with no upstream support for the modified layer, until upstream's own version lands in 2.0.0 and +strands the fork entirely. Precedent matters here: the framework's own authors, with the MNTK +infrastructure we lack, shelved it twice. + +## 6. Options + +**A. Don't port. Continue the current direction (recommended).** ScadaBridge keeps its proven +bridges; OtOpcUa executes the mesh program as sequenced (selfform prerequisite → Phases 1–7). Every +pattern in that program is already running in ScadaBridge production code; risk lives in +application code we own rather than framework internals we'd fork. Cost: OtOpcUa still pays for +Phases 2 and 5 (ClusterClient + gRPC plumbing), and cross-cluster pub/sub stays unavailable +(secrets replication keeps SQL-hub mode on ScadaBridge). + +**B. Port multi-DC (fork Akka.Cluster).** Scope per §5, ~2–4 months including validation, plus the +standing fork tax. Payoff concentrated in OtOpcUa (skip Phases 2/5, keep DPS, keep one mesh with +correct per-pair scoping). Only rational as an **upstream-contribution play** — rebase/re-port +against current `dev`, revive #4111 with Petabridge (possibly as sponsored work), so the fork has +an exit. As a private fork: not worth it for two products with a working alternative. + +**C. Multi-DC-lite on stock Akka.NET** (keep OtOpcUa's single mesh, emulate the scoping +app-side): role-scoped singletons and role-scoped DPS groups per pair (supported today), the +already-done oldest-Up-per-role primary rule, plus a **custom `IDowningProvider`** that only downs +same-site peers (pluggable, no fork). What cannot be emulated: convergence — one unreachable node +anywhere still blocks joins/leaves fleet-wide, and cross-site heartbeating stays all-to-all over +the WAN. Tolerable for a small, static fleet that rarely adds nodes mid-outage; it is a +known-sharp-edges compromise, not a fix. Worth considering only if the mesh program stalls. + +**Recommendation: A.** The mesh program should keep going — it's sequenced, its prerequisite plan +exists, and its shape is production-proven next door. Revisit B strictly as "fund/contribute the +upstream port" if Phases 2/5 blow up in practice, and note in that case the decision point comes +*before* Phase 2 starts, because B makes exactly those phases unnecessary. + +## Sources + +- [Akka 2.6 Multi-DC Cluster docs](https://doc.akka.io/docs/akka/2.6/typed/cluster-dc.html) (last Apache-2.0 line) +- [Akka classic Distributed Pub/Sub docs](https://doc.akka.io/docs/akka/2.6/distributed-pub-sub.html) (mediator scope: all nodes / role-filtered) +- [akka.net #3261 — Multi-Data Center Support in Akka.Cluster](https://github.com/akkadotnet/akka.net/issues/3261) (milestone 2.0.0) +- [akka.net PR #3284 — original multi-DC implementation (Horusiath)](https://github.com/akkadotnet/akka.net/pull/3284) +- [akka.net PR #4111 — preserved 67-commit draft (Aaronontheweb)](https://github.com/akkadotnet/akka.net/pull/4111) +- [akka/akka #25496 — 2.4→2.5 rolling-upgrade DC requirement failure](https://github.com/akka/akka/issues/25496) +- ScadaBridge cross-cluster seam map: `CentralCommunicationActor.cs` / `SiteCommunicationActor.cs` (ClusterClient), + `SiteStreamGrpcClient/Server.cs` + `sitestream.proto` (gRPC :8083), `HttpDeploymentConfigFetcher.cs` + + `DeploymentConfigEndpoints.cs` (token-gated HTTP), `StoreAndForwardService.cs`, `ActiveNodeEvaluator.cs`; + ActorSystem name shared at `AkkaHostedService.cs:191` +- OtOpcUa mesh design + program: `OtOpcUa/docs/plans/2026-07-21-per-cluster-mesh-design.md`, + `2026-07-22-per-cluster-mesh-program.md`