scadaproj/akka_msquic_transport.md

# Deep dive: an MsQuic transport for Akka.NET 1.5.62

*Research report, 2026-07-22. Question: what would it take to implement the Akka.NET 1.6 MsQuic
transport (akka.net #7466) against the Akka.NET version currently used by ScadaBridge and OtOpcUa,
and would features like higher throughput and larger-message support reduce the family's reliance
on gRPC?*

**TL;DR:** Building a QUIC transport against the current Akka.NET (1.5.62) is genuinely feasible —
the classic transport SPI is public, and Akka's own docs literally use a hypothetical QUIC transport
as the custom-transport example. Roughly 3–5 weeks to production quality. But there's a structural
catch: plugged in under the classic 1.5 remoting pipeline, QUIC is reduced to "TCP with TLS 1.3 and
cheaper handshakes." The two things actually wanted — throughput multiplexing and non-blocking large
messages — come from the *Artery pipeline rewrite above the transport*, which is the part of #7466
that doesn't exist yet and can't be replicated through the 1.5 SPI. And most of the family's gRPC
wouldn't go away regardless, because most of it isn't Akka-adjacent. Details below, options and a
recommendation at the end.

---

## 1. Where #7466 actually stands (checked 2026-07-22)

- [Issue #7466](https://github.com/akkadotnet/akka.net/issues/7466) is the **v1.6 "Artery" epic**,
  opened Jan 2025: MsQuic-based Remoting V2 replacing the never-built Aeron plan. Design goals:
  per-stream multiplexing ("this is where all of the significant performance gains come from" per
  the [Petabridge v1.6 roadmap](https://petabridge.com/blog/akkadotnet-v1.6-roadmap/)), mandatory
  TLS 1.3, automatic chunking of large messages onto dedicated streams, a system-message channel
  that works even while quarantined, Akka.Streams backpressure, actor-ref compression, and a
  2–3M msg/sec target. Requires .NET 9+.
- **Status: design-only.** No linked PRs, no branches, no assignee; the only substantive activity is
  a January 2025 design discussion (to11mtm/Aaronontheweb) about channel counts, ordering
  guarantees, and serialization. The issue itself says a protocol spec is needed before
  implementation starts.
- **The timeline has slipped badly.** The roadmap targeted alphas end of Q1 2025 and completion
  early Q1 2026. As of July 2026 the
  [releases page](https://github.com/akkadotnet/akka.net/releases) shows **zero 1.6 previews** —
  latest is 1.5.70 (July 3, 2026), and a 1.5.71 milestone is open. The
  [1.6.0 milestone](https://github.com/akkadotnet/akka.net/milestones) reads 44% complete, but the
  closed items are the AOT/trimming/serialization workstreams, not the transport. A
  [PR search for "quic"](https://github.com/akkadotnet/akka.net/pulls?q=is%3Apr+quic) confirms no
  transport code has ever landed.

So "wait for 1.6" is not a near-term plan. QUIC benefits this year would mean building it in-house.

## 2. What the projects run today

Both ScadaBridge and OtOpcUa pin **Akka 1.5.62** and use the classic **`dot-netty.tcp`** transport,
plaintext, with all size defaults — meaning the **128 KB `maximum-frame-size` cap** applies
(`ScadaBridge/src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs:310`;
`OtOpcUa/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:25`). Neither config touches TLS or
frame size. The Akka traffic is cluster gossip, heartbeats, singleton/pub-sub coordination, and
(ScadaBridge) actor messaging in a hub-and-spoke — modest volume, nowhere near transport-bound.

Two facts that matter for the "do we even need this" question:

- **The frame size can be raised today.** `akka.remote.dot-netty.tcp.maximum-frame-size = 10MiB`
  (plus matching send/receive buffer sizes) is a config change, not a transport change. The 128 KB
  default is a *guardrail*, not a TCP limitation — large messages on the single ordered association
  channel delay heartbeats behind them, which is exactly the false-failure-detection risk the
  2s/10s failure-detector tuning cares about.
- **TLS on the existing transport is already available.** Akka.Remote's DotNetty transport gained
  **mutual TLS** in [PR #7851](https://github.com/akkadotnet/akka.net/pull/7851) (merged Oct 2025 —
  included in the pinned 1.5.62). If encryption-in-flight is a driver, QUIC is not required for it.

## 3. How a transport plugs into 1.5 — and what it would take

Akka.Remote has a public custom-transport SPI
([docs](https://getakka.net/articles/remoting/transports.html)): implement
`Akka.Remote.Transport.Transport`, register it via HOCON, and everything above it (handshake,
endpoint management, quarantine, cluster) is unchanged:

```hocon
akka.remote {
  enabled-transports = ["akka.remote.zb-quic"]
  zb-quic {
    transport-class = "ZB.MOM.WW.Akka.Transport.Quic.QuicTransport, ZB.MOM.WW.Akka.Transport.Quic"
    transport-protocol = quic     # addresses become akka.quic://System@host:port
    hostname = "0.0.0.0"
    port = 4053                   # UDP now, not TCP
  }
}
```

The concrete build, as a shared lib in scadaproj (the usual pattern):

**Core (~1–2 weeks):**
- `QuicTransport : Transport` — ctor `(ActorSystem, Config)` (instantiated reflectively); implement
  `SchemeIdentifier` (`"quic"`), `MaximumPayloadBytes` (configurable — this is where "large
  messages" becomes trivial), `Listen()` (a `QuicListener.ListenAsync` with a private ALPN like
  `"akka-remote"`, surfacing `InboundAssociation` events), `Associate(Address)`
  (`QuicConnection.ConnectAsync` + open one bidirectional `QuicStream`), `Shutdown()`.
- `QuicAssociationHandle : AssociationHandle` — `Write(ByteString)` must be **non-blocking with a
  boolean backpressure signal** (return `false` when the send queue is full; Akka's `EndpointWriter`
  buffers and retries — getting this semantic right is the subtle part), `Disassociate()` (map to
  `CloseAsync` with an app error code).
- **Framing** — QUIC streams are byte streams like TCP, so the 4-byte length-prefix framing that
  dot-netty does must be re-implemented on both sides. The Akka protocol handshake/PDU layer rides
  on top automatically.

**Security config (~0.5–1 week):** QUIC **mandates TLS 1.3** — there is no plaintext mode. Every
node needs a certificate, and a validation policy is required (mutual TLS with a pinned internal CA
is the sane one; the Secrets lib is a natural home for cert material). This is new operational
surface the rigs don't have today.

**Hardening + tests (~1–2 weeks):** association teardown races, quarantine behavior, reconnect
storms, failure-detector behavior under load, big-frame tests, and soak on the docker-dev rig.
Historical note for calibration: the Helios→DotNetty transition took Akka.NET years of corner-case
fixes (dissociation races, quarantine storms). A custom transport is its own support island for
that class of bug.

**Migration:** the address scheme changes (`akka.tcp://` → `akka.quic://`), so seed nodes and every
stored address change; mixed-transport rolling upgrade isn't practical for a pair — but *both*
transports can be listed in `enabled-transports` during transition, and the site pairs already
stop/start together by policy, so cutover is easy here.

## 4. The structural catch

The classic SPI models an association as **one ordered, opaque byte channel**. The remoting layer
above it (system-message resend, handshake, cluster failure detection) *depends* on per-association
ordering, and the transport never sees message recipients — just framed byte blobs. Consequences:

- **No multiplexing win.** One association cannot be fanned across N QUIC streams without either
  breaking ordering guarantees or parsing Akka's PDU envelope inside the transport to route by
  recipient while preserving per-pair ordering — at which point you are rewriting Artery's
  dispatcher against internals with no stability contract.
- **No dedicated large-message or control channel.** A 10 MB frame still queues ahead of heartbeats
  on the same stream — identical head-of-line behavior to TCP today. The 1.6 design fixes this
  *above* the transport; the 1.5 SPI can't express it.
- What you *do* get: TLS 1.3 built in, faster connection establishment (nice for
  reconnect-after-failover, though failover latency is dominated by the 10–15s
  failure-detector/downing windows, not handshakes), connection migration, and an easy big
  `MaximumPayloadBytes`.

Net: a 1.5-SPI QUIC transport ≈ **dot-netty.tcp + mTLS + a raised frame size** — both of which are
available today with config on the stock transport.

## 5. What 1.6 actually changes — the layer split, made explicit

1.6 doesn't just swap the transport; it rewrites the remoting pipeline *above* the transport
(Artery), and that pipeline is where every benefit §4 marked "not achievable on 1.5" actually lives.

**In 1.5**, remoting is two layers: the pluggable `Transport` SPI at the bottom (a dumb ordered byte
pipe — `Associate()`, `Write(ByteString)`, `Disassociate()`) and above it the fixed pipeline:
`AkkaProtocolTransport` handshake, one `EndpointWriter`/`EndpointReader` actor pair per association,
system-message resend buffering, and quarantine. That pipeline bakes in three assumptions no custom
transport can change:

1. **One association = one ordered channel** — the system-message resend protocol and cluster
   failure detection depend on strict ordering over that single channel.
2. **Payloads are opaque to the transport** — recipient, message class, and size are invisible by
   the time bytes reach `Write()`, so "big message → own lane" or "heartbeat → priority lane"
   routing is impossible without parsing Akka's internal PDU format (i.e. reimplementing the layer
   above against unstable internals).
3. **Everything shares the lane** — user messages, system messages, cluster heartbeats. The 128 KB
   frame guardrail exists exactly because a big message parking in front of heartbeats trips the
   failure detector.

**In 1.6/Artery**, that middle layer is replaced with a recipient-aware, multi-lane, Akka.Streams-
based pipeline: it knows recipient and size *before* the wire, assigns messages to QUIC streams
accordingly (ordering preserved per sender→receiver pair rather than per node pair), chunks
oversized messages onto **dedicated large-message streams**, keeps **system/control messages on
their own stream** (including a quarantine-bypass channel), adds headers for version negotiation,
and compresses actor refs to integers. QUIC is almost incidental — it is chosen because it gives
cheap independent streams for that pipeline to map onto.

| Benefit | 1.5 + custom QUIC transport | 1.6 Artery | Why the pipeline is the difference |
|---|---|---|---|
| Throughput / multiplexing | ✗ (single ordered lane) | ✓ (the 2–3M msg/s target) | Lane assignment needs recipient visibility, which only the pipeline has |
| Large messages without side channels | ~ (raise the frame cap; heartbeats still queue behind big frames) | ✓ (dedicated chunked streams; control lane unaffected) | Size-based routing happens above the transport |
| Honest failure detection under load | ✗ (heartbeats share the data lane) | ✓ (isolated control stream) | Same |
| Split-brain handling while quarantined | ✗ | ✓ (quarantine-bypass channel) | Quarantine lives in the pipeline, not the transport |
| TLS 1.3 | ✓ (but dot-netty mTLS gets there too) | ✓ | Transport-level — the one benefit 1.5 delivers fully |
| Fast handshake / connection migration | ✓ | ✓ | Transport-level |

Consequences for this report's conclusions:

- **The "don't build it on 1.5" recommendation strengthens** — a 1.5 transport captures only the
  bottom two rows, and both have cheaper paths (dot-netty mTLS; config).
- **The gRPC-alleviation answer partially flips at 1.6, but only for Akka-adjacent channels.**
  ScadaBridge's HTTP notify-and-fetch for deploy configs, and any "too big for the frame cap"
  workaround, exist purely because of rows 2–3; under Artery those payloads could legitimately ride
  remoting. The genuinely non-Akka gRPC (mxaccessgw, HistorianGateway, LocalDb's deliberate
  cluster-independence) stays gRPC regardless — see the mapping table below.
- **New costs arrive with the benefits:** Artery is a new wire protocol (flag-day cutover, no
  rolling mix with 1.5 remoting — tolerable given the stop-pairs-together discipline), .NET 9+
  only, and it drags in the platform/ops constraints below (libmsquic, UDP firewall rules, Windows
  Server 2022+, macOS dev friction) — those attach to *any* QUIC adoption, 1.5 or 1.6.

## 6. Platform and ops constraints ([System.Net.Quic](https://learn.microsoft.com/en-us/dotnet/fundamentals/networking/quic/quic-overview))

- APIs are stable since .NET 9; the apps are .NET 10 — fine.
- **Windows: requires Windows 11 / Server 2022+** (older versions lack the TLS 1.3 crypto APIs).
  ⚠️ Check the `wonder-*` boxes — if any production VM is Server 2019 or older, QUIC remoting is a
  hard no there.
- **Linux:** `libmsquic` must be installed in every image — the `aspnet:10.0`-based images and the
  docker-dev rig would all need `apt-get install libmsquic` from packages.microsoft.com.
- **macOS (dev machine): only partially supported** — Homebrew `libmsquic` + a
  `DYLD_FALLBACK_LIBRARY_PATH` export, outside Microsoft's test matrix. Offline test suites that
  spin up real ActorSystems with remoting would grow a fragile native dependency on the Mac.
- **Firewalls:** remoting moves from TCP to **UDP** — every rule for 4053/etc. needs a UDP twin,
  and OT network gear is notoriously unkind to UDP.

## 7. .NET-native alternatives to MsQuic

Under the 1.5 SPI every transport degenerates into the same ordered byte pipe (§5), so candidate
wire protocols differ on **dependencies, platform reach, and TLS posture** — not on features. By
that metric, ranked:

**1. Managed sockets + `SslStream` + `System.IO.Pipelines` — the best fit if anything gets built.**
100% in-box managed .NET: zero native dependencies, runs everywhere the apps run today (macOS test
suites, any Windows Server version, stock `aspnet:10.0` images, TCP firewall rules unchanged).
TLS 1.3 via `SslStream` is *optional* — can match the current plaintext posture with an mTLS
upgrade path, unlike QUIC's hard TLS mandate. Arbitrary frame sizes. Strategically this is a
"modern DotNetty replacement": it sheds the effectively unmaintained DotNetty dependency, which is
one of 1.6's own goals. JVM precedent that this is respectable: JVM Artery ships TCP and TLS-TCP
modes alongside Aeron, and TCP Artery reached ~650k msg/s — most of Artery's gain is the pipeline,
not the wire. What it gives up vs QUIC: fast handshake and connection migration — neither of which
was a driver here.

*Throughput expectations, from real data:* this exact idea was prototyped by to11mtm in
[PR #4594](https://github.com/akkadotnet/akka.net/pull/4594) (Oct 2020, draft, never merged) — an
Akka.Streams/managed-socket TCP transport under the classic pipeline. On the author's i7-8750H:
DotNetty baseline ~220–240k msg/s; the managed transport alone ~170–240k (parity at best); adding
aggressive zero-copy work + a hand-written protobuf deserializer reached ~240–300k (+10–25%), and
the residual bottlenecks identified were all *pipeline*, not wire: serialization, the PDU codec,
`EndpointWriter` dispatch, blocking `.Result` in serialization paths. I.e. the transport swap is
throughput-neutral; only attacking the layer above moves the number, and even extreme effort caps
near +25%. (Reference points: the official
[RemotePingPong docs](https://getakka.net/articles/remoting/performance.html) show 82k → 141k msg/s
from flush-batching alone on older hardware; modern hardware on 1.5.x lands roughly 300–600k per
association; the 1.6 Artery target is 2–3M.) Two mitigating notes: per-*association* ceilings apply
per site pair, so hub-and-spoke aggregate scales with site count; and for large frames (the raised
`maximum-frame-size` case) a managed transport saturates the wire in bytes/sec — the msg/s ceiling
is a small-message phenomenon.

**2. HTTP/2 — fully managed multiplexing, but the SPI can't use it.** Kestrel (server) +
`SocketsHttpHandler` (client), or gRPC bidirectional streaming as the framing, are fully managed
h2 with real stream multiplexing and h2c (no cert mandate). But the 1.5 SPI locks the multiplexing
away for exactly the §5 reasons, h2 streams still share one TCP connection (packet-level
head-of-line blocking remains), and it adds protocol overhead for no captured benefit. Only
distinct win is riding existing Kestrel/gRPC infrastructure. Not preferred over raw sockets.

**3. WebSockets (`System.Net.WebSockets`)** — managed, but a single ordered stream, i.e. TCP with
extra framing. Only interesting for exotic proxy/firewall traversal. Skip.

**No managed QUIC exists.** `System.Net.Quic` *is* the msquic interop — there is no fully managed
QUIC implementation in .NET, and nothing else in the box provides independent streams. So "QUIC
without libmsquic" is not on the table, on any .NET version.

Forward-looking note: if Akka.NET's 1.6 Artery ships QUIC-only, libmsquic becomes unavoidable *at
that point* — worth watching whether upstream adds a TCP mode (to11mtm explicitly advocated
transport pluggability for firewall-restricted environments in the #7466 design discussion, citing
the JVM's TCP Artery).

## 8. Would it actually alleviate the gRPC?

Mostly no — mapping the family's gRPC surfaces:

| gRPC usage | Could Akka-over-QUIC replace it? |
|---|---|
| mxaccessgw ↔ clients | **No.** It exists for COM bitness isolation; the worker is .NET 4.8 x86, which can't even load a .NET 9+ QUIC stack, and callers aren't cluster members. |
| HistorianGateway API | **No.** A standalone service contract for multiple heterogeneous consumers — service RPC is the right shape. |
| LocalDb pair sync | **Technically closest, but deliberately no.** It runs between the same nodes as the Akka cluster, but its whole value is being independent of cluster state (it's what lets a node boot from cache when things are degraded), and it's consumable by non-Akka apps. Coupling it to remoting would weaken it. Its 4 MB cap is a gRPC config choice, not a law. |
| ScadaBridge site→central (S&F etc.) | Already Akka where Akka fits; the bulk/artifact paths that avoid remoting avoid it because of the single-channel head-of-line problem — which, per §4, a 1.5 QUIC transport does not fix. |

The "big messages so we don't need side channels" hope is really an Artery feature (dedicated
chunked large-message streams), not a QUIC-transport feature.

## 9. Options

**A. Don't build it; tune what we have; track 1.6 (recommended).** Best-practice fit:
transport-layer plumbing is exactly the code an OT product wants to own least, and the upstream
design explicitly targets the wishlist (large-message channels, quarantine-bypass control channel —
the latter directly relevant to the 2-node availability-first failover posture). Concretely: bump
1.5.62 → 1.5.70 (eight releases of remoting/streams fixes, including a cluster-wedging
consistent-hash-router bug), and if specific pains exist now, raise `maximum-frame-size` and/or
enable the new dot-netty mTLS. Subscribe to #7466; when real alphas ship, the pair-restart
discipline makes adoption cheap.

**B. Build a classic-SPI QUIC transport now.** ~3–5 weeks to production quality per §3. Choose this
only if the *specific* wins (TLS 1.3 with modern cert handling, UDP traversal, fast reconnect)
matter enough to own a transport — knowing it delivers no throughput/multiplexing gain and gets
obsoleted by 1.6.

**C. Weekend-scale spike, not production.** Implement the minimal `Transport`/`AssociationHandle`
pair, run a two-node pair on the rig, benchmark against dot-netty. Genuinely worthwhile as
de-risking/learning for the eventual 1.6 migration (certs, libmsquic packaging, UDP firewall rules
all get exercised) without owning it in production. A reasonable middle path.

**Recommendation: A**, with **C** if hands-on evidence is wanted — the honest summary is that for
this workload, an MsQuic transport on Akka 1.5 buys security and handshake ergonomics that are
mostly available from config today, while the features that would actually reduce the gRPC
side-channels only arrive with the Artery pipeline rewrite, which upstream hasn't started coding.

## Sources

- [akka.net #7466 — Epic: Akka.Remote Quic-based Transport (Artery)](https://github.com/akkadotnet/akka.net/issues/7466)
- [Petabridge — Akka.NET v1.6 Roadmap](https://petabridge.com/blog/akkadotnet-v1.6-roadmap/)
- [Akka.NET releases](https://github.com/akkadotnet/akka.net/releases)
- [Akka.NET milestones](https://github.com/akkadotnet/akka.net/milestones)
- [Akka.NET remoting transports documentation](https://getakka.net/articles/remoting/transports.html)
- [Akka.NET PR #7851 — mutual TLS for DotNetty transport](https://github.com/akkadotnet/akka.net/pull/7851)
- [QUIC support in .NET (System.Net.Quic)](https://learn.microsoft.com/en-us/dotnet/fundamentals/networking/quic/quic-overview)