docs: Akka.NET transport deep-dives — MsQuic/1.6 Artery and multi-DC clusters
Two research reports (2026-07-22): - akka_msquic_transport.md: what a QUIC transport on Akka 1.5.62 would take, the 1.5-vs-1.6 pipeline delta that gates the benefits, .NET-native alternatives (managed sockets ranked first), and measured throughput evidence (PR #4594: transport swap is throughput-neutral). - akka_multidc_clusters.md: JVM multi-DC semantics mapped onto the central-pair + site-pairs topology, the stalled Akka.NET port (#3261 / PR #4111), per-system fit vs the OtOpcUa mesh program, and the before-Phase-2 decision point. Recommendation in both: don't build; track upstream.
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,302 @@
|
||||
# Deep dive: an MsQuic transport for Akka.NET 1.5.62
|
||||
|
||||
*Research report, 2026-07-22. Question: what would it take to implement the Akka.NET 1.6 MsQuic
|
||||
transport (akka.net #7466) against the Akka.NET version currently used by ScadaBridge and OtOpcUa,
|
||||
and would features like higher throughput and larger-message support reduce the family's reliance
|
||||
on gRPC?*
|
||||
|
||||
**TL;DR:** Building a QUIC transport against the current Akka.NET (1.5.62) is genuinely feasible —
|
||||
the classic transport SPI is public, and Akka's own docs literally use a hypothetical QUIC transport
|
||||
as the custom-transport example. Roughly 3–5 weeks to production quality. But there's a structural
|
||||
catch: plugged in under the classic 1.5 remoting pipeline, QUIC is reduced to "TCP with TLS 1.3 and
|
||||
cheaper handshakes." The two things actually wanted — throughput multiplexing and non-blocking large
|
||||
messages — come from the *Artery pipeline rewrite above the transport*, which is the part of #7466
|
||||
that doesn't exist yet and can't be replicated through the 1.5 SPI. And most of the family's gRPC
|
||||
wouldn't go away regardless, because most of it isn't Akka-adjacent. Details below, options and a
|
||||
recommendation at the end.
|
||||
|
||||
---
|
||||
|
||||
## 1. Where #7466 actually stands (checked 2026-07-22)
|
||||
|
||||
- [Issue #7466](https://github.com/akkadotnet/akka.net/issues/7466) is the **v1.6 "Artery" epic**,
|
||||
opened Jan 2025: MsQuic-based Remoting V2 replacing the never-built Aeron plan. Design goals:
|
||||
per-stream multiplexing ("this is where all of the significant performance gains come from" per
|
||||
the [Petabridge v1.6 roadmap](https://petabridge.com/blog/akkadotnet-v1.6-roadmap/)), mandatory
|
||||
TLS 1.3, automatic chunking of large messages onto dedicated streams, a system-message channel
|
||||
that works even while quarantined, Akka.Streams backpressure, actor-ref compression, and a
|
||||
2–3M msg/sec target. Requires .NET 9+.
|
||||
- **Status: design-only.** No linked PRs, no branches, no assignee; the only substantive activity is
|
||||
a January 2025 design discussion (to11mtm/Aaronontheweb) about channel counts, ordering
|
||||
guarantees, and serialization. The issue itself says a protocol spec is needed before
|
||||
implementation starts.
|
||||
- **The timeline has slipped badly.** The roadmap targeted alphas end of Q1 2025 and completion
|
||||
early Q1 2026. As of July 2026 the
|
||||
[releases page](https://github.com/akkadotnet/akka.net/releases) shows **zero 1.6 previews** —
|
||||
latest is 1.5.70 (July 3, 2026), and a 1.5.71 milestone is open. The
|
||||
[1.6.0 milestone](https://github.com/akkadotnet/akka.net/milestones) reads 44% complete, but the
|
||||
closed items are the AOT/trimming/serialization workstreams, not the transport. A
|
||||
[PR search for "quic"](https://github.com/akkadotnet/akka.net/pulls?q=is%3Apr+quic) confirms no
|
||||
transport code has ever landed.
|
||||
|
||||
So "wait for 1.6" is not a near-term plan. QUIC benefits this year would mean building it in-house.
|
||||
|
||||
## 2. What the projects run today
|
||||
|
||||
Both ScadaBridge and OtOpcUa pin **Akka 1.5.62** and use the classic **`dot-netty.tcp`** transport,
|
||||
plaintext, with all size defaults — meaning the **128 KB `maximum-frame-size` cap** applies
|
||||
(`ScadaBridge/src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs:310`;
|
||||
`OtOpcUa/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:25`). Neither config touches TLS or
|
||||
frame size. The Akka traffic is cluster gossip, heartbeats, singleton/pub-sub coordination, and
|
||||
(ScadaBridge) actor messaging in a hub-and-spoke — modest volume, nowhere near transport-bound.
|
||||
|
||||
Two facts that matter for the "do we even need this" question:
|
||||
|
||||
- **The frame size can be raised today.** `akka.remote.dot-netty.tcp.maximum-frame-size = 10MiB`
|
||||
(plus matching send/receive buffer sizes) is a config change, not a transport change. The 128 KB
|
||||
default is a *guardrail*, not a TCP limitation — large messages on the single ordered association
|
||||
channel delay heartbeats behind them, which is exactly the false-failure-detection risk the
|
||||
2s/10s failure-detector tuning cares about.
|
||||
- **TLS on the existing transport is already available.** Akka.Remote's DotNetty transport gained
|
||||
**mutual TLS** in [PR #7851](https://github.com/akkadotnet/akka.net/pull/7851) (merged Oct 2025 —
|
||||
included in the pinned 1.5.62). If encryption-in-flight is a driver, QUIC is not required for it.
|
||||
|
||||
## 3. How a transport plugs into 1.5 — and what it would take
|
||||
|
||||
Akka.Remote has a public custom-transport SPI
|
||||
([docs](https://getakka.net/articles/remoting/transports.html)): implement
|
||||
`Akka.Remote.Transport.Transport`, register it via HOCON, and everything above it (handshake,
|
||||
endpoint management, quarantine, cluster) is unchanged:
|
||||
|
||||
```hocon
|
||||
akka.remote {
|
||||
enabled-transports = ["akka.remote.zb-quic"]
|
||||
zb-quic {
|
||||
transport-class = "ZB.MOM.WW.Akka.Transport.Quic.QuicTransport, ZB.MOM.WW.Akka.Transport.Quic"
|
||||
transport-protocol = quic # addresses become akka.quic://System@host:port
|
||||
hostname = "0.0.0.0"
|
||||
port = 4053 # UDP now, not TCP
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The concrete build, as a shared lib in scadaproj (the usual pattern):
|
||||
|
||||
**Core (~1–2 weeks):**
|
||||
- `QuicTransport : Transport` — ctor `(ActorSystem, Config)` (instantiated reflectively); implement
|
||||
`SchemeIdentifier` (`"quic"`), `MaximumPayloadBytes` (configurable — this is where "large
|
||||
messages" becomes trivial), `Listen()` (a `QuicListener.ListenAsync` with a private ALPN like
|
||||
`"akka-remote"`, surfacing `InboundAssociation` events), `Associate(Address)`
|
||||
(`QuicConnection.ConnectAsync` + open one bidirectional `QuicStream`), `Shutdown()`.
|
||||
- `QuicAssociationHandle : AssociationHandle` — `Write(ByteString)` must be **non-blocking with a
|
||||
boolean backpressure signal** (return `false` when the send queue is full; Akka's `EndpointWriter`
|
||||
buffers and retries — getting this semantic right is the subtle part), `Disassociate()` (map to
|
||||
`CloseAsync` with an app error code).
|
||||
- **Framing** — QUIC streams are byte streams like TCP, so the 4-byte length-prefix framing that
|
||||
dot-netty does must be re-implemented on both sides. The Akka protocol handshake/PDU layer rides
|
||||
on top automatically.
|
||||
|
||||
**Security config (~0.5–1 week):** QUIC **mandates TLS 1.3** — there is no plaintext mode. Every
|
||||
node needs a certificate, and a validation policy is required (mutual TLS with a pinned internal CA
|
||||
is the sane one; the Secrets lib is a natural home for cert material). This is new operational
|
||||
surface the rigs don't have today.
|
||||
|
||||
**Hardening + tests (~1–2 weeks):** association teardown races, quarantine behavior, reconnect
|
||||
storms, failure-detector behavior under load, big-frame tests, and soak on the docker-dev rig.
|
||||
Historical note for calibration: the Helios→DotNetty transition took Akka.NET years of corner-case
|
||||
fixes (dissociation races, quarantine storms). A custom transport is its own support island for
|
||||
that class of bug.
|
||||
|
||||
**Migration:** the address scheme changes (`akka.tcp://` → `akka.quic://`), so seed nodes and every
|
||||
stored address change; mixed-transport rolling upgrade isn't practical for a pair — but *both*
|
||||
transports can be listed in `enabled-transports` during transition, and the site pairs already
|
||||
stop/start together by policy, so cutover is easy here.
|
||||
|
||||
## 4. The structural catch
|
||||
|
||||
The classic SPI models an association as **one ordered, opaque byte channel**. The remoting layer
|
||||
above it (system-message resend, handshake, cluster failure detection) *depends* on per-association
|
||||
ordering, and the transport never sees message recipients — just framed byte blobs. Consequences:
|
||||
|
||||
- **No multiplexing win.** One association cannot be fanned across N QUIC streams without either
|
||||
breaking ordering guarantees or parsing Akka's PDU envelope inside the transport to route by
|
||||
recipient while preserving per-pair ordering — at which point you are rewriting Artery's
|
||||
dispatcher against internals with no stability contract.
|
||||
- **No dedicated large-message or control channel.** A 10 MB frame still queues ahead of heartbeats
|
||||
on the same stream — identical head-of-line behavior to TCP today. The 1.6 design fixes this
|
||||
*above* the transport; the 1.5 SPI can't express it.
|
||||
- What you *do* get: TLS 1.3 built in, faster connection establishment (nice for
|
||||
reconnect-after-failover, though failover latency is dominated by the 10–15s
|
||||
failure-detector/downing windows, not handshakes), connection migration, and an easy big
|
||||
`MaximumPayloadBytes`.
|
||||
|
||||
Net: a 1.5-SPI QUIC transport ≈ **dot-netty.tcp + mTLS + a raised frame size** — both of which are
|
||||
available today with config on the stock transport.
|
||||
|
||||
## 5. What 1.6 actually changes — the layer split, made explicit
|
||||
|
||||
1.6 doesn't just swap the transport; it rewrites the remoting pipeline *above* the transport
|
||||
(Artery), and that pipeline is where every benefit §4 marked "not achievable on 1.5" actually lives.
|
||||
|
||||
**In 1.5**, remoting is two layers: the pluggable `Transport` SPI at the bottom (a dumb ordered byte
|
||||
pipe — `Associate()`, `Write(ByteString)`, `Disassociate()`) and above it the fixed pipeline:
|
||||
`AkkaProtocolTransport` handshake, one `EndpointWriter`/`EndpointReader` actor pair per association,
|
||||
system-message resend buffering, and quarantine. That pipeline bakes in three assumptions no custom
|
||||
transport can change:
|
||||
|
||||
1. **One association = one ordered channel** — the system-message resend protocol and cluster
|
||||
failure detection depend on strict ordering over that single channel.
|
||||
2. **Payloads are opaque to the transport** — recipient, message class, and size are invisible by
|
||||
the time bytes reach `Write()`, so "big message → own lane" or "heartbeat → priority lane"
|
||||
routing is impossible without parsing Akka's internal PDU format (i.e. reimplementing the layer
|
||||
above against unstable internals).
|
||||
3. **Everything shares the lane** — user messages, system messages, cluster heartbeats. The 128 KB
|
||||
frame guardrail exists exactly because a big message parking in front of heartbeats trips the
|
||||
failure detector.
|
||||
|
||||
**In 1.6/Artery**, that middle layer is replaced with a recipient-aware, multi-lane, Akka.Streams-
|
||||
based pipeline: it knows recipient and size *before* the wire, assigns messages to QUIC streams
|
||||
accordingly (ordering preserved per sender→receiver pair rather than per node pair), chunks
|
||||
oversized messages onto **dedicated large-message streams**, keeps **system/control messages on
|
||||
their own stream** (including a quarantine-bypass channel), adds headers for version negotiation,
|
||||
and compresses actor refs to integers. QUIC is almost incidental — it is chosen because it gives
|
||||
cheap independent streams for that pipeline to map onto.
|
||||
|
||||
| Benefit | 1.5 + custom QUIC transport | 1.6 Artery | Why the pipeline is the difference |
|
||||
|---|---|---|---|
|
||||
| Throughput / multiplexing | ✗ (single ordered lane) | ✓ (the 2–3M msg/s target) | Lane assignment needs recipient visibility, which only the pipeline has |
|
||||
| Large messages without side channels | ~ (raise the frame cap; heartbeats still queue behind big frames) | ✓ (dedicated chunked streams; control lane unaffected) | Size-based routing happens above the transport |
|
||||
| Honest failure detection under load | ✗ (heartbeats share the data lane) | ✓ (isolated control stream) | Same |
|
||||
| Split-brain handling while quarantined | ✗ | ✓ (quarantine-bypass channel) | Quarantine lives in the pipeline, not the transport |
|
||||
| TLS 1.3 | ✓ (but dot-netty mTLS gets there too) | ✓ | Transport-level — the one benefit 1.5 delivers fully |
|
||||
| Fast handshake / connection migration | ✓ | ✓ | Transport-level |
|
||||
|
||||
Consequences for this report's conclusions:
|
||||
|
||||
- **The "don't build it on 1.5" recommendation strengthens** — a 1.5 transport captures only the
|
||||
bottom two rows, and both have cheaper paths (dot-netty mTLS; config).
|
||||
- **The gRPC-alleviation answer partially flips at 1.6, but only for Akka-adjacent channels.**
|
||||
ScadaBridge's HTTP notify-and-fetch for deploy configs, and any "too big for the frame cap"
|
||||
workaround, exist purely because of rows 2–3; under Artery those payloads could legitimately ride
|
||||
remoting. The genuinely non-Akka gRPC (mxaccessgw, HistorianGateway, LocalDb's deliberate
|
||||
cluster-independence) stays gRPC regardless — see the mapping table below.
|
||||
- **New costs arrive with the benefits:** Artery is a new wire protocol (flag-day cutover, no
|
||||
rolling mix with 1.5 remoting — tolerable given the stop-pairs-together discipline), .NET 9+
|
||||
only, and it drags in the platform/ops constraints below (libmsquic, UDP firewall rules, Windows
|
||||
Server 2022+, macOS dev friction) — those attach to *any* QUIC adoption, 1.5 or 1.6.
|
||||
|
||||
## 6. Platform and ops constraints ([System.Net.Quic](https://learn.microsoft.com/en-us/dotnet/fundamentals/networking/quic/quic-overview))
|
||||
|
||||
- APIs are stable since .NET 9; the apps are .NET 10 — fine.
|
||||
- **Windows: requires Windows 11 / Server 2022+** (older versions lack the TLS 1.3 crypto APIs).
|
||||
⚠️ Check the `wonder-*` boxes — if any production VM is Server 2019 or older, QUIC remoting is a
|
||||
hard no there.
|
||||
- **Linux:** `libmsquic` must be installed in every image — the `aspnet:10.0`-based images and the
|
||||
docker-dev rig would all need `apt-get install libmsquic` from packages.microsoft.com.
|
||||
- **macOS (dev machine): only partially supported** — Homebrew `libmsquic` + a
|
||||
`DYLD_FALLBACK_LIBRARY_PATH` export, outside Microsoft's test matrix. Offline test suites that
|
||||
spin up real ActorSystems with remoting would grow a fragile native dependency on the Mac.
|
||||
- **Firewalls:** remoting moves from TCP to **UDP** — every rule for 4053/etc. needs a UDP twin,
|
||||
and OT network gear is notoriously unkind to UDP.
|
||||
|
||||
## 7. .NET-native alternatives to MsQuic
|
||||
|
||||
Under the 1.5 SPI every transport degenerates into the same ordered byte pipe (§5), so candidate
|
||||
wire protocols differ on **dependencies, platform reach, and TLS posture** — not on features. By
|
||||
that metric, ranked:
|
||||
|
||||
**1. Managed sockets + `SslStream` + `System.IO.Pipelines` — the best fit if anything gets built.**
|
||||
100% in-box managed .NET: zero native dependencies, runs everywhere the apps run today (macOS test
|
||||
suites, any Windows Server version, stock `aspnet:10.0` images, TCP firewall rules unchanged).
|
||||
TLS 1.3 via `SslStream` is *optional* — can match the current plaintext posture with an mTLS
|
||||
upgrade path, unlike QUIC's hard TLS mandate. Arbitrary frame sizes. Strategically this is a
|
||||
"modern DotNetty replacement": it sheds the effectively unmaintained DotNetty dependency, which is
|
||||
one of 1.6's own goals. JVM precedent that this is respectable: JVM Artery ships TCP and TLS-TCP
|
||||
modes alongside Aeron, and TCP Artery reached ~650k msg/s — most of Artery's gain is the pipeline,
|
||||
not the wire. What it gives up vs QUIC: fast handshake and connection migration — neither of which
|
||||
was a driver here.
|
||||
|
||||
*Throughput expectations, from real data:* this exact idea was prototyped by to11mtm in
|
||||
[PR #4594](https://github.com/akkadotnet/akka.net/pull/4594) (Oct 2020, draft, never merged) — an
|
||||
Akka.Streams/managed-socket TCP transport under the classic pipeline. On the author's i7-8750H:
|
||||
DotNetty baseline ~220–240k msg/s; the managed transport alone ~170–240k (parity at best); adding
|
||||
aggressive zero-copy work + a hand-written protobuf deserializer reached ~240–300k (+10–25%), and
|
||||
the residual bottlenecks identified were all *pipeline*, not wire: serialization, the PDU codec,
|
||||
`EndpointWriter` dispatch, blocking `.Result` in serialization paths. I.e. the transport swap is
|
||||
throughput-neutral; only attacking the layer above moves the number, and even extreme effort caps
|
||||
near +25%. (Reference points: the official
|
||||
[RemotePingPong docs](https://getakka.net/articles/remoting/performance.html) show 82k → 141k msg/s
|
||||
from flush-batching alone on older hardware; modern hardware on 1.5.x lands roughly 300–600k per
|
||||
association; the 1.6 Artery target is 2–3M.) Two mitigating notes: per-*association* ceilings apply
|
||||
per site pair, so hub-and-spoke aggregate scales with site count; and for large frames (the raised
|
||||
`maximum-frame-size` case) a managed transport saturates the wire in bytes/sec — the msg/s ceiling
|
||||
is a small-message phenomenon.
|
||||
|
||||
**2. HTTP/2 — fully managed multiplexing, but the SPI can't use it.** Kestrel (server) +
|
||||
`SocketsHttpHandler` (client), or gRPC bidirectional streaming as the framing, are fully managed
|
||||
h2 with real stream multiplexing and h2c (no cert mandate). But the 1.5 SPI locks the multiplexing
|
||||
away for exactly the §5 reasons, h2 streams still share one TCP connection (packet-level
|
||||
head-of-line blocking remains), and it adds protocol overhead for no captured benefit. Only
|
||||
distinct win is riding existing Kestrel/gRPC infrastructure. Not preferred over raw sockets.
|
||||
|
||||
**3. WebSockets (`System.Net.WebSockets`)** — managed, but a single ordered stream, i.e. TCP with
|
||||
extra framing. Only interesting for exotic proxy/firewall traversal. Skip.
|
||||
|
||||
**No managed QUIC exists.** `System.Net.Quic` *is* the msquic interop — there is no fully managed
|
||||
QUIC implementation in .NET, and nothing else in the box provides independent streams. So "QUIC
|
||||
without libmsquic" is not on the table, on any .NET version.
|
||||
|
||||
Forward-looking note: if Akka.NET's 1.6 Artery ships QUIC-only, libmsquic becomes unavoidable *at
|
||||
that point* — worth watching whether upstream adds a TCP mode (to11mtm explicitly advocated
|
||||
transport pluggability for firewall-restricted environments in the #7466 design discussion, citing
|
||||
the JVM's TCP Artery).
|
||||
|
||||
## 8. Would it actually alleviate the gRPC?
|
||||
|
||||
Mostly no — mapping the family's gRPC surfaces:
|
||||
|
||||
| gRPC usage | Could Akka-over-QUIC replace it? |
|
||||
|---|---|
|
||||
| mxaccessgw ↔ clients | **No.** It exists for COM bitness isolation; the worker is .NET 4.8 x86, which can't even load a .NET 9+ QUIC stack, and callers aren't cluster members. |
|
||||
| HistorianGateway API | **No.** A standalone service contract for multiple heterogeneous consumers — service RPC is the right shape. |
|
||||
| LocalDb pair sync | **Technically closest, but deliberately no.** It runs between the same nodes as the Akka cluster, but its whole value is being independent of cluster state (it's what lets a node boot from cache when things are degraded), and it's consumable by non-Akka apps. Coupling it to remoting would weaken it. Its 4 MB cap is a gRPC config choice, not a law. |
|
||||
| ScadaBridge site→central (S&F etc.) | Already Akka where Akka fits; the bulk/artifact paths that avoid remoting avoid it because of the single-channel head-of-line problem — which, per §4, a 1.5 QUIC transport does not fix. |
|
||||
|
||||
The "big messages so we don't need side channels" hope is really an Artery feature (dedicated
|
||||
chunked large-message streams), not a QUIC-transport feature.
|
||||
|
||||
## 9. Options
|
||||
|
||||
**A. Don't build it; tune what we have; track 1.6 (recommended).** Best-practice fit:
|
||||
transport-layer plumbing is exactly the code an OT product wants to own least, and the upstream
|
||||
design explicitly targets the wishlist (large-message channels, quarantine-bypass control channel —
|
||||
the latter directly relevant to the 2-node availability-first failover posture). Concretely: bump
|
||||
1.5.62 → 1.5.70 (eight releases of remoting/streams fixes, including a cluster-wedging
|
||||
consistent-hash-router bug), and if specific pains exist now, raise `maximum-frame-size` and/or
|
||||
enable the new dot-netty mTLS. Subscribe to #7466; when real alphas ship, the pair-restart
|
||||
discipline makes adoption cheap.
|
||||
|
||||
**B. Build a classic-SPI QUIC transport now.** ~3–5 weeks to production quality per §3. Choose this
|
||||
only if the *specific* wins (TLS 1.3 with modern cert handling, UDP traversal, fast reconnect)
|
||||
matter enough to own a transport — knowing it delivers no throughput/multiplexing gain and gets
|
||||
obsoleted by 1.6.
|
||||
|
||||
**C. Weekend-scale spike, not production.** Implement the minimal `Transport`/`AssociationHandle`
|
||||
pair, run a two-node pair on the rig, benchmark against dot-netty. Genuinely worthwhile as
|
||||
de-risking/learning for the eventual 1.6 migration (certs, libmsquic packaging, UDP firewall rules
|
||||
all get exercised) without owning it in production. A reasonable middle path.
|
||||
|
||||
**Recommendation: A**, with **C** if hands-on evidence is wanted — the honest summary is that for
|
||||
this workload, an MsQuic transport on Akka 1.5 buys security and handshake ergonomics that are
|
||||
mostly available from config today, while the features that would actually reduce the gRPC
|
||||
side-channels only arrive with the Artery pipeline rewrite, which upstream hasn't started coding.
|
||||
|
||||
## Sources
|
||||
|
||||
- [akka.net #7466 — Epic: Akka.Remote Quic-based Transport (Artery)](https://github.com/akkadotnet/akka.net/issues/7466)
|
||||
- [Petabridge — Akka.NET v1.6 Roadmap](https://petabridge.com/blog/akkadotnet-v1.6-roadmap/)
|
||||
- [Akka.NET releases](https://github.com/akkadotnet/akka.net/releases)
|
||||
- [Akka.NET milestones](https://github.com/akkadotnet/akka.net/milestones)
|
||||
- [Akka.NET remoting transports documentation](https://getakka.net/articles/remoting/transports.html)
|
||||
- [Akka.NET PR #7851 — mutual TLS for DotNetty transport](https://github.com/akkadotnet/akka.net/pull/7851)
|
||||
- [QUIC support in .NET (System.Net.Quic)](https://learn.microsoft.com/en-us/dotnet/fundamentals/networking/quic/quic-overview)
|
||||
Reference in New Issue
Block a user