dohertj2/scadaproj
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: Akka.NET transport deep-dives — MsQuic/1.6 Artery and multi-DC clusters

Browse Source 
Two research reports (2026-07-22):
- akka_msquic_transport.md: what a QUIC transport on Akka 1.5.62 would take,
  the 1.5-vs-1.6 pipeline delta that gates the benefits, .NET-native
  alternatives (managed sockets ranked first), and measured throughput
  evidence (PR #4594: transport swap is throughput-neutral).
- akka_multidc_clusters.md: JVM multi-DC semantics mapped onto the
  central-pair + site-pairs topology, the stalled Akka.NET port (#3261 /
  PR #4111), per-system fit vs the OtOpcUa mesh program, and the
  before-Phase-2 decision point.
Recommendation in both: don't build; track upstream.
This commit is contained in:
Joseph Doherty
2026-07-22 06:41:36 -04:00
parent 5919e29969
commit dbfd5e7a7e
2 changed files with 543 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
akka_msquic_transport.md
+302
View File
@@ -0,0 +1,302 @@
# Deep dive: an MsQuic transport for Akka.NET 1.5.62
*Research report, 2026-07-22. Question: what would it take to implement the Akka.NET 1.6 MsQuic
transport (akka.net #7466) against the Akka.NET version currently used by ScadaBridge and OtOpcUa,
and would features like higher throughput and larger-message support reduce the family's reliance
on gRPC?*
**TL;DR:** Building a QUIC transport against the current Akka.NET (1.5.62) is genuinely feasible —
the classic transport SPI is public, and Akka's own docs literally use a hypothetical QUIC transport
as the custom-transport example. Roughly 3–5 weeks to production quality. But there's a structural
catch: plugged in under the classic 1.5 remoting pipeline, QUIC is reduced to "TCP with TLS 1.3 and
cheaper handshakes." The two things actually wanted — throughput multiplexing and non-blocking large
messages — come from the *Artery pipeline rewrite above the transport*, which is the part of #7466
that doesn't exist yet and can't be replicated through the 1.5 SPI. And most of the family's gRPC
wouldn't go away regardless, because most of it isn't Akka-adjacent. Details below, options and a
recommendation at the end.
---
## 1. Where #7466 actually stands (checked 2026-07-22)
- [Issue #7466](https://github.com/akkadotnet/akka.net/issues/7466) is the **v1.6 "Artery" epic**,
opened Jan 2025: MsQuic-based Remoting V2 replacing the never-built Aeron plan. Design goals:
per-stream multiplexing ("this is where all of the significant performance gains come from" per
the [Petabridge v1.6 roadmap](https://petabridge.com/blog/akkadotnet-v1.6-roadmap/)), mandatory
TLS 1.3, automatic chunking of large messages onto dedicated streams, a system-message channel
that works even while quarantined, Akka.Streams backpressure, actor-ref compression, and a
2–3M msg/sec target. Requires .NET 9+.
- **Status: design-only.** No linked PRs, no branches, no assignee; the only substantive activity is
a January 2025 design discussion (to11mtm/Aaronontheweb) about channel counts, ordering
guarantees, and serialization. The issue itself says a protocol spec is needed before
implementation starts.
- **The timeline has slipped badly.** The roadmap targeted alphas end of Q1 2025 and completion
early Q1 2026. As of July 2026 the
[releases page](https://github.com/akkadotnet/akka.net/releases) shows **zero 1.6 previews** —
latest is 1.5.70 (July 3, 2026), and a 1.5.71 milestone is open. The
[1.6.0 milestone](https://github.com/akkadotnet/akka.net/milestones) reads 44% complete, but the
closed items are the AOT/trimming/serialization workstreams, not the transport. A
[PR search for "quic"](https://github.com/akkadotnet/akka.net/pulls?q=is%3Apr+quic) confirms no
transport code has ever landed.
So "wait for 1.6" is not a near-term plan. QUIC benefits this year would mean building it in-house.
## 2. What the projects run today
Both ScadaBridge and OtOpcUa pin **Akka 1.5.62** and use the classic **`dot-netty.tcp`** transport,
plaintext, with all size defaults — meaning the **128 KB `maximum-frame-size` cap** applies
(`ScadaBridge/src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs:310`;
`OtOpcUa/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:25`). Neither config touches TLS or
frame size. The Akka traffic is cluster gossip, heartbeats, singleton/pub-sub coordination, and
(ScadaBridge) actor messaging in a hub-and-spoke — modest volume, nowhere near transport-bound.
Two facts that matter for the "do we even need this" question:
- **The frame size can be raised today.** `akka.remote.dot-netty.tcp.maximum-frame-size = 10MiB`
(plus matching send/receive buffer sizes) is a config change, not a transport change. The 128 KB
default is a *guardrail*, not a TCP limitation — large messages on the single ordered association
channel delay heartbeats behind them, which is exactly the false-failure-detection risk the
2s/10s failure-detector tuning cares about.
- **TLS on the existing transport is already available.** Akka.Remote's DotNetty transport gained
**mutual TLS** in [PR #7851](https://github.com/akkadotnet/akka.net/pull/7851) (merged Oct 2025 —
included in the pinned 1.5.62). If encryption-in-flight is a driver, QUIC is not required for it.
## 3. How a transport plugs into 1.5 — and what it would take
Akka.Remote has a public custom-transport SPI
([docs](https://getakka.net/articles/remoting/transports.html)): implement
`Akka.Remote.Transport.Transport`, register it via HOCON, and everything above it (handshake,
endpoint management, quarantine, cluster) is unchanged:
```hocon
akka.remote {
enabled-transports = ["akka.remote.zb-quic"]
zb-quic {
transport-class = "ZB.MOM.WW.Akka.Transport.Quic.QuicTransport, ZB.MOM.WW.Akka.Transport.Quic"
transport-protocol = quic # addresses become akka.quic://System@host:port
hostname = "0.0.0.0"
port = 4053 # UDP now, not TCP
}
}
```
The concrete build, as a shared lib in scadaproj (the usual pattern):
**Core (~1–2 weeks):**
- `QuicTransport : Transport` — ctor `(ActorSystem, Config)` (instantiated reflectively); implement
`SchemeIdentifier` (`"quic"`), `MaximumPayloadBytes` (configurable — this is where "large
messages" becomes trivial), `Listen()` (a `QuicListener.ListenAsync` with a private ALPN like
`"akka-remote"`, surfacing `InboundAssociation` events), `Associate(Address)`
(`QuicConnection.ConnectAsync` + open one bidirectional `QuicStream`), `Shutdown()`.
- `QuicAssociationHandle : AssociationHandle` — `Write(ByteString)` must be **non-blocking with a
boolean backpressure signal** (return `false` when the send queue is full; Akka's `EndpointWriter`
buffers and retries — getting this semantic right is the subtle part), `Disassociate()` (map to
`CloseAsync` with an app error code).
- **Framing** — QUIC streams are byte streams like TCP, so the 4-byte length-prefix framing that
dot-netty does must be re-implemented on both sides. The Akka protocol handshake/PDU layer rides
on top automatically.
**Security config (~0.5–1 week):** QUIC **mandates TLS 1.3** — there is no plaintext mode. Every
node needs a certificate, and a validation policy is required (mutual TLS with a pinned internal CA
is the sane one; the Secrets lib is a natural home for cert material). This is new operational
surface the rigs don't have today.
**Hardening + tests (~1–2 weeks):** association teardown races, quarantine behavior, reconnect
storms, failure-detector behavior under load, big-frame tests, and soak on the docker-dev rig.
Historical note for calibration: the Helios→DotNetty transition took Akka.NET years of corner-case
fixes (dissociation races, quarantine storms). A custom transport is its own support island for
that class of bug.
**Migration:** the address scheme changes (`akka.tcp://` → `akka.quic://`), so seed nodes and every
stored address change; mixed-transport rolling upgrade isn't practical for a pair — but *both*
transports can be listed in `enabled-transports` during transition, and the site pairs already
stop/start together by policy, so cutover is easy here.
## 4. The structural catch
The classic SPI models an association as **one ordered, opaque byte channel**. The remoting layer
above it (system-message resend, handshake, cluster failure detection) *depends* on per-association
ordering, and the transport never sees message recipients — just framed byte blobs. Consequences:
- **No multiplexing win.** One association cannot be fanned across N QUIC streams without either
breaking ordering guarantees or parsing Akka's PDU envelope inside the transport to route by
recipient while preserving per-pair ordering — at which point you are rewriting Artery's
dispatcher against internals with no stability contract.
- **No dedicated large-message or control channel.** A 10 MB frame still queues ahead of heartbeats
on the same stream — identical head-of-line behavior to TCP today. The 1.6 design fixes this
*above* the transport; the 1.5 SPI can't express it.
- What you *do* get: TLS 1.3 built in, faster connection establishment (nice for
reconnect-after-failover, though failover latency is dominated by the 10–15s
failure-detector/downing windows, not handshakes), connection migration, and an easy big
`MaximumPayloadBytes`.
Net: a 1.5-SPI QUIC transport ≈ **dot-netty.tcp + mTLS + a raised frame size** — both of which are
available today with config on the stock transport.
## 5. What 1.6 actually changes — the layer split, made explicit
1.6 doesn't just swap the transport; it rewrites the remoting pipeline *above* the transport
(Artery), and that pipeline is where every benefit §4 marked "not achievable on 1.5" actually lives.
**In 1.5**, remoting is two layers: the pluggable `Transport` SPI at the bottom (a dumb ordered byte
pipe — `Associate()`, `Write(ByteString)`, `Disassociate()`) and above it the fixed pipeline:
`AkkaProtocolTransport` handshake, one `EndpointWriter`/`EndpointReader` actor pair per association,
system-message resend buffering, and quarantine. That pipeline bakes in three assumptions no custom
transport can change:
1. **One association = one ordered channel** — the system-message resend protocol and cluster
failure detection depend on strict ordering over that single channel.
2. **Payloads are opaque to the transport** — recipient, message class, and size are invisible by
the time bytes reach `Write()`, so "big message → own lane" or "heartbeat → priority lane"
routing is impossible without parsing Akka's internal PDU format (i.e. reimplementing the layer
above against unstable internals).
3. **Everything shares the lane** — user messages, system messages, cluster heartbeats. The 128 KB
frame guardrail exists exactly because a big message parking in front of heartbeats trips the
failure detector.
**In 1.6/Artery**, that middle layer is replaced with a recipient-aware, multi-lane, Akka.Streams-
based pipeline: it knows recipient and size *before* the wire, assigns messages to QUIC streams
accordingly (ordering preserved per sender→receiver pair rather than per node pair), chunks
oversized messages onto **dedicated large-message streams**, keeps **system/control messages on
their own stream** (including a quarantine-bypass channel), adds headers for version negotiation,
and compresses actor refs to integers. QUIC is almost incidental — it is chosen because it gives
cheap independent streams for that pipeline to map onto.
| Benefit | 1.5 + custom QUIC transport | 1.6 Artery | Why the pipeline is the difference |
|---|---|---|---|
| Throughput / multiplexing | ✗ (single ordered lane) | ✓ (the 2–3M msg/s target) | Lane assignment needs recipient visibility, which only the pipeline has |
| Large messages without side channels | ~ (raise the frame cap; heartbeats still queue behind big frames) | ✓ (dedicated chunked streams; control lane unaffected) | Size-based routing happens above the transport |
| Honest failure detection under load | ✗ (heartbeats share the data lane) | ✓ (isolated control stream) | Same |
| Split-brain handling while quarantined | ✗ | ✓ (quarantine-bypass channel) | Quarantine lives in the pipeline, not the transport |
| TLS 1.3 | ✓ (but dot-netty mTLS gets there too) | ✓ | Transport-level — the one benefit 1.5 delivers fully |
| Fast handshake / connection migration | ✓ | ✓ | Transport-level |
Consequences for this report's conclusions:
- **The "don't build it on 1.5" recommendation strengthens** — a 1.5 transport captures only the
bottom two rows, and both have cheaper paths (dot-netty mTLS; config).
- **The gRPC-alleviation answer partially flips at 1.6, but only for Akka-adjacent channels.**
ScadaBridge's HTTP notify-and-fetch for deploy configs, and any "too big for the frame cap"
workaround, exist purely because of rows 2–3; under Artery those payloads could legitimately ride
remoting. The genuinely non-Akka gRPC (mxaccessgw, HistorianGateway, LocalDb's deliberate
cluster-independence) stays gRPC regardless — see the mapping table below.
- **New costs arrive with the benefits:** Artery is a new wire protocol (flag-day cutover, no
rolling mix with 1.5 remoting — tolerable given the stop-pairs-together discipline), .NET 9+
only, and it drags in the platform/ops constraints below (libmsquic, UDP firewall rules, Windows
Server 2022+, macOS dev friction) — those attach to *any* QUIC adoption, 1.5 or 1.6.
## 6. Platform and ops constraints ([System.Net.Quic](https://learn.microsoft.com/en-us/dotnet/fundamentals/networking/quic/quic-overview))
- APIs are stable since .NET 9; the apps are .NET 10 — fine.
- **Windows: requires Windows 11 / Server 2022+** (older versions lack the TLS 1.3 crypto APIs).
⚠️ Check the `wonder-*` boxes — if any production VM is Server 2019 or older, QUIC remoting is a
hard no there.
- **Linux:** `libmsquic` must be installed in every image — the `aspnet:10.0`-based images and the
docker-dev rig would all need `apt-get install libmsquic` from packages.microsoft.com.
- **macOS (dev machine): only partially supported** — Homebrew `libmsquic` + a
`DYLD_FALLBACK_LIBRARY_PATH` export, outside Microsoft's test matrix. Offline test suites that
spin up real ActorSystems with remoting would grow a fragile native dependency on the Mac.
- **Firewalls:** remoting moves from TCP to **UDP** — every rule for 4053/etc. needs a UDP twin,
and OT network gear is notoriously unkind to UDP.
## 7. .NET-native alternatives to MsQuic
Under the 1.5 SPI every transport degenerates into the same ordered byte pipe (§5), so candidate
wire protocols differ on **dependencies, platform reach, and TLS posture** — not on features. By
that metric, ranked:
**1. Managed sockets + `SslStream` + `System.IO.Pipelines` — the best fit if anything gets built.**
100% in-box managed .NET: zero native dependencies, runs everywhere the apps run today (macOS test
suites, any Windows Server version, stock `aspnet:10.0` images, TCP firewall rules unchanged).
TLS 1.3 via `SslStream` is *optional* — can match the current plaintext posture with an mTLS
upgrade path, unlike QUIC's hard TLS mandate. Arbitrary frame sizes. Strategically this is a
"modern DotNetty replacement": it sheds the effectively unmaintained DotNetty dependency, which is
one of 1.6's own goals. JVM precedent that this is respectable: JVM Artery ships TCP and TLS-TCP
modes alongside Aeron, and TCP Artery reached ~650k msg/s — most of Artery's gain is the pipeline,
not the wire. What it gives up vs QUIC: fast handshake and connection migration — neither of which
was a driver here.
*Throughput expectations, from real data:* this exact idea was prototyped by to11mtm in
[PR #4594](https://github.com/akkadotnet/akka.net/pull/4594) (Oct 2020, draft, never merged) — an
Akka.Streams/managed-socket TCP transport under the classic pipeline. On the author's i7-8750H:
DotNetty baseline ~220–240k msg/s; the managed transport alone ~170–240k (parity at best); adding
aggressive zero-copy work + a hand-written protobuf deserializer reached ~240–300k (+10–25%), and
the residual bottlenecks identified were all *pipeline*, not wire: serialization, the PDU codec,
`EndpointWriter` dispatch, blocking `.Result` in serialization paths. I.e. the transport swap is
throughput-neutral; only attacking the layer above moves the number, and even extreme effort caps
near +25%. (Reference points: the official
[RemotePingPong docs](https://getakka.net/articles/remoting/performance.html) show 82k → 141k msg/s
from flush-batching alone on older hardware; modern hardware on 1.5.x lands roughly 300–600k per
association; the 1.6 Artery target is 2–3M.) Two mitigating notes: per-*association* ceilings apply
per site pair, so hub-and-spoke aggregate scales with site count; and for large frames (the raised
`maximum-frame-size` case) a managed transport saturates the wire in bytes/sec — the msg/s ceiling
is a small-message phenomenon.
**2. HTTP/2 — fully managed multiplexing, but the SPI can't use it.** Kestrel (server) +
`SocketsHttpHandler` (client), or gRPC bidirectional streaming as the framing, are fully managed
h2 with real stream multiplexing and h2c (no cert mandate). But the 1.5 SPI locks the multiplexing
away for exactly the §5 reasons, h2 streams still share one TCP connection (packet-level
head-of-line blocking remains), and it adds protocol overhead for no captured benefit. Only
distinct win is riding existing Kestrel/gRPC infrastructure. Not preferred over raw sockets.
**3. WebSockets (`System.Net.WebSockets`)** — managed, but a single ordered stream, i.e. TCP with
extra framing. Only interesting for exotic proxy/firewall traversal. Skip.
**No managed QUIC exists.** `System.Net.Quic` *is* the msquic interop — there is no fully managed
QUIC implementation in .NET, and nothing else in the box provides independent streams. So "QUIC
without libmsquic" is not on the table, on any .NET version.
Forward-looking note: if Akka.NET's 1.6 Artery ships QUIC-only, libmsquic becomes unavoidable *at
that point* — worth watching whether upstream adds a TCP mode (to11mtm explicitly advocated
transport pluggability for firewall-restricted environments in the #7466 design discussion, citing
the JVM's TCP Artery).
## 8. Would it actually alleviate the gRPC?
Mostly no — mapping the family's gRPC surfaces:
| gRPC usage | Could Akka-over-QUIC replace it? |
|---|---|
| mxaccessgw ↔ clients | **No.** It exists for COM bitness isolation; the worker is .NET 4.8 x86, which can't even load a .NET 9+ QUIC stack, and callers aren't cluster members. |
| HistorianGateway API | **No.** A standalone service contract for multiple heterogeneous consumers — service RPC is the right shape. |
| LocalDb pair sync | **Technically closest, but deliberately no.** It runs between the same nodes as the Akka cluster, but its whole value is being independent of cluster state (it's what lets a node boot from cache when things are degraded), and it's consumable by non-Akka apps. Coupling it to remoting would weaken it. Its 4 MB cap is a gRPC config choice, not a law. |
| ScadaBridge site→central (S&F etc.) | Already Akka where Akka fits; the bulk/artifact paths that avoid remoting avoid it because of the single-channel head-of-line problem — which, per §4, a 1.5 QUIC transport does not fix. |
The "big messages so we don't need side channels" hope is really an Artery feature (dedicated
chunked large-message streams), not a QUIC-transport feature.
## 9. Options
**A. Don't build it; tune what we have; track 1.6 (recommended).** Best-practice fit:
transport-layer plumbing is exactly the code an OT product wants to own least, and the upstream
design explicitly targets the wishlist (large-message channels, quarantine-bypass control channel —
the latter directly relevant to the 2-node availability-first failover posture). Concretely: bump
1.5.62 → 1.5.70 (eight releases of remoting/streams fixes, including a cluster-wedging
consistent-hash-router bug), and if specific pains exist now, raise `maximum-frame-size` and/or
enable the new dot-netty mTLS. Subscribe to #7466; when real alphas ship, the pair-restart
discipline makes adoption cheap.
**B. Build a classic-SPI QUIC transport now.** ~3–5 weeks to production quality per §3. Choose this
only if the *specific* wins (TLS 1.3 with modern cert handling, UDP traversal, fast reconnect)
matter enough to own a transport — knowing it delivers no throughput/multiplexing gain and gets
obsoleted by 1.6.
**C. Weekend-scale spike, not production.** Implement the minimal `Transport`/`AssociationHandle`
pair, run a two-node pair on the rig, benchmark against dot-netty. Genuinely worthwhile as
de-risking/learning for the eventual 1.6 migration (certs, libmsquic packaging, UDP firewall rules
all get exercised) without owning it in production. A reasonable middle path.
**Recommendation: A**, with **C** if hands-on evidence is wanted — the honest summary is that for
this workload, an MsQuic transport on Akka 1.5 buys security and handshake ergonomics that are
mostly available from config today, while the features that would actually reduce the gRPC
side-channels only arrive with the Artery pipeline rewrite, which upstream hasn't started coding.
## Sources
- [akka.net #7466 — Epic: Akka.Remote Quic-based Transport (Artery)](https://github.com/akkadotnet/akka.net/issues/7466)
- [Petabridge — Akka.NET v1.6 Roadmap](https://petabridge.com/blog/akkadotnet-v1.6-roadmap/)
- [Akka.NET releases](https://github.com/akkadotnet/akka.net/releases)
- [Akka.NET milestones](https://github.com/akkadotnet/akka.net/milestones)
- [Akka.NET remoting transports documentation](https://getakka.net/articles/remoting/transports.html)
- [Akka.NET PR #7851 — mutual TLS for DotNetty transport](https://github.com/akkadotnet/akka.net/pull/7851)
- [QUIC support in .NET (System.Net.Quic)](https://learn.microsoft.com/en-us/dotnet/fundamentals/networking/quic/quic-overview)
akka_multidc_clusters.md
+241
View File
@@ -0,0 +1,241 @@
# Deep dive: Akka Multi-DC cluster support, ported to Akka.NET, applied to our topology
*Research report, 2026-07-22. Question: Akka (JVM) has multi-DC cluster support that Akka.NET does
not. Given our layout (central 2-node cluster + N site 2-node clusters), what would porting it take,
and how would it be used in our systems?*
**TL;DR:** Multi-DC maps onto our topology almost perfectly — "one DC per pair" gives per-site
leadership, per-site singletons, per-site oldest-node primaries, and WAN-partition tolerance inside
ONE cluster, which is precisely the problem set the OtOpcUa per-cluster mesh program and
ScadaBridge's three bespoke bridge transports were built to solve. But the port is core-gossip
surgery that has already defeated the Akka.NET maintainers twice (PRs #3284 and #4111, the latter a
67-commit functionally-complete draft abandoned in 2019 over multi-node-test instability, now
milestoned to the far-future 2.0.0). And two of our hardest cross-cluster problems — the 128 KB
frame limit and bulk data movement — are *not* clustering problems and survive multi-DC untouched.
Recommendation: don't port; keep the mesh-program direction; treat #4111 as a candidate for an
upstream-contribution play only if the mesh program's transport phases (2/5) prove heavier than
expected. Details and options below.
---
## 1. What Akka multi-DC actually is (JVM; GA since Akka 2.5.6, 2017)
One correction to the framing: this isn't an upcoming JVM feature — it has been GA on the JVM side
since 2017. What's true is that it never made it into Akka.NET. Semantics (from the
[Akka 2.6 docs](https://doc.akka.io/docs/akka/2.6/typed/cluster-dc.html) — 2.6.20 is the last
Apache-2.0 version, which matters for porting; 2.7+ is BSL-licensed):
- **DC membership is just a role.** `akka.cluster.multi-data-center.self-data-center = "site-a"`
adds role `dc-site-a`. One cluster, every member tagged. Default DC is `"default"` — which is why
untagged and tagged nodes can't mix (JVM rolling 2.4→2.5 upgrades hit exactly this, akka/akka
#25496).
- **One leader per DC.** Each DC's leader performs joining/up/leaving/removal transitions for *its
own* members only. Crucially: **members can join and leave while DCs are partitioned from each
other** — impossible in a plain cluster, where any unreachable member blocks convergence and
therefore blocks all leader actions fleet-wide.
- **Two failure detectors.** `akka.cluster.failure-detector` intra-DC (all-to-all within the DC,
as today) and `akka.cluster.multi-data-center.failure-detector` cross-DC, run only by the
`cross-data-center-connections` (default 5) oldest nodes per DC. Cross-DC unreachability raises
`UnreachableDataCenter` / `ReachableDataCenter` events — it is treated as "the WAN link is bad,"
explicitly NOT fed into member downing: *"For network partitions between data centers the system
should typically not down the unreachable nodes, but instead wait until it heals or a decision is
made by a human or external monitoring system."*
- **Cluster Singleton is per-DC.** Start the manager on all nodes with 3 DCs defined → 3 live
singletons, one per DC. Proxies default to the local DC and take a `dataCenter` parameter to
target another DC's singleton. A "global" singleton = start its manager only in one designated DC
and accept unavailability when that DC is cut off.
- **Cluster Sharding is per-DC** (own coordinator/regions per DC; cross-DC proxies available).
Not relevant to us — neither app uses sharding.
- **Distributed Pub/Sub is NOT DC-scoped.** The mediator replicates its registry among *all*
cluster nodes (optionally restricted by role) — so topics span every DC by default. This is the
single most valuable property for OtOpcUa (§4).
- Singleton placement and per-DC leadership both key off **oldest member within the DC** — i.e.
the JVM feature's per-DC primary rule is literally our `ActiveNodeEvaluator.SelfIsOldestUp` /
OtOpcUa Phase-0b `IsDriverPrimary` rule, applied per pair.
## 2. Where Akka.NET stands on it
- [Issue #3261](https://github.com/akkadotnet/akka.net/issues/3261) (opened Jan 2018 by
Aaronontheweb) is the tracking issue. Current milestone: **2.0.0 — Akka.Typed**, i.e. no
committed timeline; it is not in the 1.6 plan (1.6 = Artery/QUIC, .NET Standard drop, new
Streams materializer — see the companion report `akka_msquic_transport.md`).
- It was **implemented once**: Horusiath's [PR #3284](https://github.com/akkadotnet/akka.net/pull/3284)
(2018, milestoned 1.4.0, closed), preserved by Aaron as draft
[PR #4111](https://github.com/akkadotnet/akka.net/pull/4111) (Dec 2019, still open). 67 commits
touching membership state, gossip, cross-DC heartbeats, singleton/sharding DC-awareness, failure
detection, cluster events, and serialization, plus multi-node specs for split-brain and
sunny-weather scenarios. It reportedly passed on .NET Framework but **died on Multi-Node TestKit
instability on .NET Core** ("151 tests failing" at one point; deadlocks in MembershipState lazy
init, convergence races, CoordinatedShutdown exit-hook problems). It was dropped from 1.4.0 for
review/verification bandwidth and never revived. Its base is **mid-2018 `dev`** — six years and
two major version lines behind today's 1.5.70.
So "porting the support" concretely means: re-porting from the JVM 2.6.20 sources (Apache 2.0 —
do NOT copy from 2.7+/BSL) with #4111 as a .NET-idiom reference, onto Akka.NET 1.5.x, and carrying
it as a fork until upstream lands its own in 2.0.0.
## 3. The mapping onto our topology
The fit is genuinely striking. Model: **every 2-node pair is a DC** in one spanning cluster.
| DC | Members | Today |
|---|---|---|
| `dc-central` | central-a, central-b | ScadaBridge central cluster / OtOpcUa admin pair |
| `dc-site-a` | site-a-1, site-a-2 | separate site cluster (SB) / same mesh (OtOpcUa) |
| `dc-site-b` … | … | … |
What each multi-DC property buys us:
- **Per-DC oldest = pair primary, by construction.** The exact rule both products converged on
(ScadaBridge `ActiveNodeEvaluator.SelfIsOldestUp`; OtOpcUa Phase 0b) becomes the framework's own
scoping — the mesh design's core motivation ("every Primary-gated decision asks a cluster-wide
election about a pair-local resource") dissolves.
- **Per-DC singletons.** OtOpcUa's `admin-operations` singleton stays effectively central by
starting its manager only on `dc-central` nodes; per-pair singletons (e.g. a per-site drain
coordinator) come free.
- **WAN-safe failure handling.** Intra-pair failure detection + our `auto-down`(15s)
availability-over-partition-safety posture stays *within the pair*, where we want it. A WAN blip
to a site raises `UnreachableDataCenter` — no downing, no quarantine churn, and (the big one)
**membership operations at other sites keep working during the outage**. Today's OtOpcUa single
mesh has neither property, which is a large part of why the mesh program exists.
- **Pub/sub spans DCs.** OtOpcUa's nine DPS topics + deploy notify channel keep working unchanged
across the whole fleet. The Secrets Akka replicator (DPS-based) would also span everything —
today it explicitly cannot cross ScadaBridge's separate clusters (why secrets use SQL-hub mode
there).
- **Direct actor messaging replaces ClusterClient.** In one spanning cluster, receptionist
contact-point management, per-site client caching, and DB-driven address refresh
(`CentralCommunicationActor`'s `_siteClients`, the 60 s `Site`-row polling) reduce to ordinary
cluster-aware messaging.
### What multi-DC does NOT fix (important)
- **The 128 KB frame limit and serialization posture are unchanged.** ScadaBridge's HTTP
notify-and-fetch (deploy configs) and gRPC streaming (`:8083`, high-volume attribute/alarm
events) exist because Akka messaging is the wrong channel for bulk payloads — that remains true
in a multi-DC cluster. Those channels stay. (Same conclusion as the MsQuic report: bulk channels
aren't clustering problems.)
- **Store-and-forward stays.** At-least-once site→central delivery with a durable buffer
(`StoreAndForwardService` + replicated `sf_messages`) is an application guarantee; multi-DC
changes the pipe, not the need for the buffer.
- **Security got harder, not easier.** Today an attacker on the WAN can reach ClusterClient/gRPC
seams; in a spanning cluster they can attempt to *join the cluster*. Akka remoting is
unauthenticated h2c-equivalent in both products (accepted risk, mesh design §6.3) — a spanning
cluster raises the value of revisiting that.
- **Persistence single-writer caveats** (the JVM docs' sharding-across-DCs warning) don't bite us —
no sharding, and pair-local stores are LocalDb-replicated with HLC/LWW already.
## 4. Per-system assessment
### OtOpcUa — the big beneficiary, but the timing is awkward
OtOpcUa *today* is one mesh spanning everything — i.e. it is already shaped like a multi-DC
cluster, minus the DC semantics. The per-cluster mesh program
(`OtOpcUa/docs/plans/2026-07-22-per-cluster-mesh-program.md`, Phases 1–7) exists to fix exactly the
things DC-tagging would fix in place:
| Mesh-program phase | Under multi-DC |
|---|---|
| 0a auto-down / 0b oldest-primary | Already done; both carry over unchanged (intra-DC) |
| 1 `ClusterNode` address columns, DB-sourced ack set | **Mostly unnecessary** — membership stays visible fleet-wide |
| 2 ClusterClient comm actors + receptionists | **Unnecessary** — one cluster, direct messaging |
| 3 config fetch-and-cache from central | **Still required** — driven by "sites have no SQL Server," not by clustering |
| 4 cut driver-side ConfigDb | **Still required** — same reason |
| 5 gRPC migration of 7 observability topics | **Unnecessary** — DPS keeps working |
| 6 mesh partition into pairs | Replaced by DC tagging + seed layout (much smaller) |
| 7 failover drills / live gates | Still required, plus new cross-DC drills |
So multi-DC would delete the two transport-migration phases (2 and 5 — the heaviest pure-plumbing
work) and shrink 1 and 6, while leaving the genuinely deployment-driven phases (3, 4) intact. That
is a real prize. It is also exactly why the timing is awkward: the program was just sequenced on
the ScadaBridge-proven separate-cluster shape, and switching horses means betting those phases on a
forked cluster core instead of on patterns already running in production-adjacent code.
### ScadaBridge — already solved this; migration value is low
ScadaBridge built the separate-cluster bridges years-of-work deep and they are live: per-site
`ClusterClient`s with DB-driven contact refresh, per-node receptionist registration, gRPC
server-on-site/client-on-central streaming, token-gated HTTP deploy fetch, S&F with active-node
gating. Two details actually make it multi-DC-*ready* — every cluster already shares the one
ActorSystem name `"scadabridge"` (separation is purely seed-node partitioning), and the roles
already include `site-{SiteId}` — but what multi-DC would replace is only the ClusterClient control
channel and the address-book machinery. The gRPC and HTTP channels stay (frame limit), S&F stays,
and the CLI's ClusterClient entry point would need rework. Replacing a proven, tested seam with a
forked gossip layer for a cleaner control channel is a poor trade on its own.
## 5. What the port itself takes
Scope (from the JVM 2.6.20 implementation and the #4111 diff):
1. **Membership/gossip core** — `Member` gains a `dataCenter` (derived from the `dc-` role, so the
gossip **wire format is nearly unchanged** — the DC rides in roles); `MembershipState`/gossip
convergence, leader election, and reachability calculations all become DC-filtered (convergence
ignores cross-DC observations; per-DC leader actions permitted during inter-DC splits). This is
the delicate 20% that is 80% of the risk.
2. **Heartbeating** — keep today's all-to-all within a DC; add the cross-DC heartbeater run by the
N oldest per DC with its own failure-detector config; new `UnreachableDataCenter` /
`ReachableDataCenter` events.
3. **Downing** — `AutoDowning`/SBR must become DC-scoped (JVM SBR is; our `auto-down` posture then
applies per pair, which is exactly the HA stance we want). A non-DC-aware downing provider in a
multi-DC cluster is actively dangerous (downs the far side on WAN blips).
4. **Cluster tools** — Singleton manager/proxy DC-awareness (small — it is role-filtering);
sharding proxies (skippable for us); DPS needs nothing.
5. **Serialization** — additive protobuf changes for the new heartbeat/gossip fields; whole-fleet
cutover required (mixed DC-aware/unaware nodes don't interoperate — the JVM had the same
constraint). Our stop/start-pairs-together discipline makes this tolerable, but it is a
flag-day, not rolling.
6. **Verification — the real cost.** This is what killed it upstream twice: MNTK specs for
split-brain, join-during-partition, cross-DC unreachability, singleton-per-DC handover. Our
substitute would be rig-based live gates (docker-dev with real WAN-partition injection via
`tc`/iptables), which we're good at — but the failure modes are convergence races that soak
tests catch and unit tests don't.
**Effort honestly:** the code port is plausibly 4–8 weeks against the Apache-2.0 JVM reference with
#4111 as a map. The validation to *trust* it under an OT platform's availability requirements is
the same again or more, and then it's a **permanent fork of Akka.Cluster** — every 1.5.x bump
(there have been 8 since our pin) becomes a merge with the most intricate code in the framework,
with no upstream support for the modified layer, until upstream's own version lands in 2.0.0 and
strands the fork entirely. Precedent matters here: the framework's own authors, with the MNTK
infrastructure we lack, shelved it twice.
## 6. Options
**A. Don't port. Continue the current direction (recommended).** ScadaBridge keeps its proven
bridges; OtOpcUa executes the mesh program as sequenced (selfform prerequisite → Phases 1–7). Every
pattern in that program is already running in ScadaBridge production code; risk lives in
application code we own rather than framework internals we'd fork. Cost: OtOpcUa still pays for
Phases 2 and 5 (ClusterClient + gRPC plumbing), and cross-cluster pub/sub stays unavailable
(secrets replication keeps SQL-hub mode on ScadaBridge).
**B. Port multi-DC (fork Akka.Cluster).** Scope per §5, ~2–4 months including validation, plus the
standing fork tax. Payoff concentrated in OtOpcUa (skip Phases 2/5, keep DPS, keep one mesh with
correct per-pair scoping). Only rational as an **upstream-contribution play** — rebase/re-port
against current `dev`, revive #4111 with Petabridge (possibly as sponsored work), so the fork has
an exit. As a private fork: not worth it for two products with a working alternative.
**C. Multi-DC-lite on stock Akka.NET** (keep OtOpcUa's single mesh, emulate the scoping
app-side): role-scoped singletons and role-scoped DPS groups per pair (supported today), the
already-done oldest-Up-per-role primary rule, plus a **custom `IDowningProvider`** that only downs
same-site peers (pluggable, no fork). What cannot be emulated: convergence — one unreachable node
anywhere still blocks joins/leaves fleet-wide, and cross-site heartbeating stays all-to-all over
the WAN. Tolerable for a small, static fleet that rarely adds nodes mid-outage; it is a
known-sharp-edges compromise, not a fix. Worth considering only if the mesh program stalls.
**Recommendation: A.** The mesh program should keep going — it's sequenced, its prerequisite plan
exists, and its shape is production-proven next door. Revisit B strictly as "fund/contribute the
upstream port" if Phases 2/5 blow up in practice, and note in that case the decision point comes
*before* Phase 2 starts, because B makes exactly those phases unnecessary.
## Sources
- [Akka 2.6 Multi-DC Cluster docs](https://doc.akka.io/docs/akka/2.6/typed/cluster-dc.html) (last Apache-2.0 line)
- [Akka classic Distributed Pub/Sub docs](https://doc.akka.io/docs/akka/2.6/distributed-pub-sub.html) (mediator scope: all nodes / role-filtered)
- [akka.net #3261 — Multi-Data Center Support in Akka.Cluster](https://github.com/akkadotnet/akka.net/issues/3261) (milestone 2.0.0)
- [akka.net PR #3284 — original multi-DC implementation (Horusiath)](https://github.com/akkadotnet/akka.net/pull/3284)
- [akka.net PR #4111 — preserved 67-commit draft (Aaronontheweb)](https://github.com/akkadotnet/akka.net/pull/4111)
- [akka/akka #25496 — 2.4→2.5 rolling-upgrade DC requirement failure](https://github.com/akka/akka/issues/25496)
- ScadaBridge cross-cluster seam map: `CentralCommunicationActor.cs` / `SiteCommunicationActor.cs` (ClusterClient),
`SiteStreamGrpcClient/Server.cs` + `sitestream.proto` (gRPC :8083), `HttpDeploymentConfigFetcher.cs` +
`DeploymentConfigEndpoints.cs` (token-gated HTTP), `StoreAndForwardService.cs`, `ActiveNodeEvaluator.cs`;
ActorSystem name shared at `AkkaHostedService.cs:191`
- OtOpcUa mesh design + program: `OtOpcUa/docs/plans/2026-07-21-per-cluster-mesh-design.md`,
`2026-07-22-per-cluster-mesh-program.md`