scadaproj/akka_msquic_transport.md

Deep dive: an MsQuic transport for Akka.NET 1.5.62

Research report, 2026-07-22. Question: what would it take to implement the Akka.NET 1.6 MsQuic transport (akka.net #7466) against the Akka.NET version currently used by ScadaBridge and OtOpcUa, and would features like higher throughput and larger-message support reduce the family's reliance on gRPC?

TL;DR: Building a QUIC transport against the current Akka.NET (1.5.62) is genuinely feasible — the classic transport SPI is public, and Akka's own docs literally use a hypothetical QUIC transport as the custom-transport example. Roughly 3–5 weeks to production quality. But there's a structural catch: plugged in under the classic 1.5 remoting pipeline, QUIC is reduced to "TCP with TLS 1.3 and cheaper handshakes." The two things actually wanted — throughput multiplexing and non-blocking large messages — come from the Artery pipeline rewrite above the transport, which is the part of #7466 that doesn't exist yet and can't be replicated through the 1.5 SPI. And most of the family's gRPC wouldn't go away regardless, because most of it isn't Akka-adjacent. Details below, options and a recommendation at the end.

---

1. Where #7466 actually stands (checked 2026-07-22)

• Issue #7466 is the v1.6 "Artery" epic, opened Jan 2025: MsQuic-based Remoting V2 replacing the never-built Aeron plan. Design goals: per-stream multiplexing ("this is where all of the significant performance gains come from" per the Petabridge v1.6 roadmap), mandatory TLS 1.3, automatic chunking of large messages onto dedicated streams, a system-message channel that works even while quarantined, Akka.Streams backpressure, actor-ref compression, and a 2–3M msg/sec target. Requires .NET 9+.
• Status: design-only. No linked PRs, no branches, no assignee; the only substantive activity is a January 2025 design discussion (to11mtm/Aaronontheweb) about channel counts, ordering guarantees, and serialization. The issue itself says a protocol spec is needed before implementation starts.
• The timeline has slipped badly. The roadmap targeted alphas end of Q1 2025 and completion early Q1 2026. As of July 2026 the releases page shows zero 1.6 previews — latest is 1.5.70 (July 3, 2026), and a 1.5.71 milestone is open. The 1.6.0 milestone reads 44% complete, but the closed items are the AOT/trimming/serialization workstreams, not the transport. A PR search for "quic" confirms no transport code has ever landed.

So "wait for 1.6" is not a near-term plan. QUIC benefits this year would mean building it in-house.

STATUS UPDATE (2026-07-22, same day, deeper look): the two paragraphs above are correct about QUIC but stale about Artery — searching PRs for "quic" missed work named "artery". Artery implementation began 2026-07-03 and is landing rapidly on dev, as Artery-TCP (the JVM-parallel move): design spec via openspec (#8313, "artery-tcp-remoting"), transport-substrate validation gate G0 (#8315), TCP framing + envelope codec G1 (#8319), handshake/association G2 (#8320), parallel inbound/outbound lanes (#8356/#8357, Jul 11), dedicated large-message stream streamId 3 (#8352, Jul 10), quarantine port in flight (#8416), MNTR/CI integration for both transports (#8373/#8413). 192/192 Artery specs passing as of #8352. In parallel, Serialization.V2 (source-generated + MessagePack internals) is in heavy development — ~30 merged PRs with wire-format snapshot tests and an open cross-component integration proof suite (#8407). TCP-first also confirms §7's forward-looking note: the Artery benefits (lanes, large-message stream, control stream) are arriving transport-agnostic over TCP — no libmsquic required — with QUIC as a later substrate swap. No preview release or public benchmark numbers yet; see the 1.6-status section of the conversation record for the remaining-work list.

2. What the projects run today

Both ScadaBridge and OtOpcUa pin Akka 1.5.62 and use the classic dot-netty.tcp transport, plaintext, with all size defaults — meaning the 128 KB maximum-frame-size cap applies (ScadaBridge/src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs:310; OtOpcUa/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:25). Neither config touches TLS or frame size. The Akka traffic is cluster gossip, heartbeats, singleton/pub-sub coordination, and (ScadaBridge) actor messaging in a hub-and-spoke — modest volume, nowhere near transport-bound.

Two facts that matter for the "do we even need this" question:

• The frame size can be raised today. akka.remote.dot-netty.tcp.maximum-frame-size = 10MiB (plus matching send/receive buffer sizes) is a config change, not a transport change. The 128 KB default is a guardrail, not a TCP limitation — large messages on the single ordered association channel delay heartbeats behind them, which is exactly the false-failure-detection risk the 2s/10s failure-detector tuning cares about.
• TLS on the existing transport is already available. Akka.Remote's DotNetty transport gained mutual TLS in PR #7851 (merged Oct 2025 — included in the pinned 1.5.62). If encryption-in-flight is a driver, QUIC is not required for it.

3. How a transport plugs into 1.5 — and what it would take

Akka.Remote has a public custom-transport SPI (docs): implement Akka.Remote.Transport.Transport, register it via HOCON, and everything above it (handshake, endpoint management, quarantine, cluster) is unchanged:

akka.remote {
  enabled-transports = ["akka.remote.zb-quic"]
  zb-quic {
    transport-class = "ZB.MOM.WW.Akka.Transport.Quic.QuicTransport, ZB.MOM.WW.Akka.Transport.Quic"
    transport-protocol = quic     # addresses become akka.quic://System@host:port
    hostname = "0.0.0.0"
    port = 4053                   # UDP now, not TCP
  }
}

The concrete build, as a shared lib in scadaproj (the usual pattern):

Core (~1–2 weeks):

• QuicTransport : Transport — ctor (ActorSystem, Config) (instantiated reflectively); implement SchemeIdentifier ("quic"), MaximumPayloadBytes (configurable — this is where "large messages" becomes trivial), Listen() (a QuicListener.ListenAsync with a private ALPN like "akka-remote", surfacing InboundAssociation events), Associate(Address) (QuicConnection.ConnectAsync + open one bidirectional QuicStream), Shutdown().
• QuicAssociationHandle : AssociationHandle — Write(ByteString) must be non-blocking with a boolean backpressure signal (return false when the send queue is full; Akka's EndpointWriter buffers and retries — getting this semantic right is the subtle part), Disassociate() (map to CloseAsync with an app error code).
• Framing — QUIC streams are byte streams like TCP, so the 4-byte length-prefix framing that dot-netty does must be re-implemented on both sides. The Akka protocol handshake/PDU layer rides on top automatically.

Security config (~0.5–1 week): QUIC mandates TLS 1.3 — there is no plaintext mode. Every node needs a certificate, and a validation policy is required (mutual TLS with a pinned internal CA is the sane one; the Secrets lib is a natural home for cert material). This is new operational surface the rigs don't have today.

Hardening + tests (~1–2 weeks): association teardown races, quarantine behavior, reconnect storms, failure-detector behavior under load, big-frame tests, and soak on the docker-dev rig. Historical note for calibration: the Helios→DotNetty transition took Akka.NET years of corner-case fixes (dissociation races, quarantine storms). A custom transport is its own support island for that class of bug.

Migration: the address scheme changes (akka.tcp:// → akka.quic://), so seed nodes and every stored address change; mixed-transport rolling upgrade isn't practical for a pair — but both transports can be listed in enabled-transports during transition, and the site pairs already stop/start together by policy, so cutover is easy here.

4. The structural catch

The classic SPI models an association as one ordered, opaque byte channel. The remoting layer above it (system-message resend, handshake, cluster failure detection) depends on per-association ordering, and the transport never sees message recipients — just framed byte blobs. Consequences:

• No multiplexing win. One association cannot be fanned across N QUIC streams without either breaking ordering guarantees or parsing Akka's PDU envelope inside the transport to route by recipient while preserving per-pair ordering — at which point you are rewriting Artery's dispatcher against internals with no stability contract.
• No dedicated large-message or control channel. A 10 MB frame still queues ahead of heartbeats on the same stream — identical head-of-line behavior to TCP today. The 1.6 design fixes this above the transport; the 1.5 SPI can't express it.
• What you do get: TLS 1.3 built in, faster connection establishment (nice for reconnect-after-failover, though failover latency is dominated by the 10–15s failure-detector/downing windows, not handshakes), connection migration, and an easy big MaximumPayloadBytes.

Net: a 1.5-SPI QUIC transport ≈ dot-netty.tcp + mTLS + a raised frame size — both of which are available today with config on the stock transport.

5. What 1.6 actually changes — the layer split, made explicit

1.6 doesn't just swap the transport; it rewrites the remoting pipeline above the transport (Artery), and that pipeline is where every benefit §4 marked "not achievable on 1.5" actually lives.

In 1.5, remoting is two layers: the pluggable Transport SPI at the bottom (a dumb ordered byte pipe — Associate(), Write(ByteString), Disassociate()) and above it the fixed pipeline: AkkaProtocolTransport handshake, one EndpointWriter/EndpointReader actor pair per association, system-message resend buffering, and quarantine. That pipeline bakes in three assumptions no custom transport can change:

1. One association = one ordered channel — the system-message resend protocol and cluster failure detection depend on strict ordering over that single channel.
2. Payloads are opaque to the transport — recipient, message class, and size are invisible by the time bytes reach Write(), so "big message → own lane" or "heartbeat → priority lane" routing is impossible without parsing Akka's internal PDU format (i.e. reimplementing the layer above against unstable internals).
3. Everything shares the lane — user messages, system messages, cluster heartbeats. The 128 KB frame guardrail exists exactly because a big message parking in front of heartbeats trips the failure detector.

In 1.6/Artery, that middle layer is replaced with a recipient-aware, multi-lane, Akka.Streams- based pipeline: it knows recipient and size before the wire, assigns messages to QUIC streams accordingly (ordering preserved per sender→receiver pair rather than per node pair), chunks oversized messages onto dedicated large-message streams, keeps system/control messages on their own stream (including a quarantine-bypass channel), adds headers for version negotiation, and compresses actor refs to integers. QUIC is almost incidental — it is chosen because it gives cheap independent streams for that pipeline to map onto.

Benefit	1.5 + custom QUIC transport	1.6 Artery	Why the pipeline is the difference
Throughput / multiplexing	✗ (single ordered lane)	✓ (the 2–3M msg/s target)	Lane assignment needs recipient visibility, which only the pipeline has
Large messages without side channels	~ (raise the frame cap; heartbeats still queue behind big frames)	✓ (dedicated chunked streams; control lane unaffected)	Size-based routing happens above the transport
Honest failure detection under load	✗ (heartbeats share the data lane)	✓ (isolated control stream)	Same
Split-brain handling while quarantined	✗	✓ (quarantine-bypass channel)	Quarantine lives in the pipeline, not the transport
TLS 1.3	✓ (but dot-netty mTLS gets there too)	✓	Transport-level — the one benefit 1.5 delivers fully
Fast handshake / connection migration	✓	✓	Transport-level

Consequences for this report's conclusions:

• The "don't build it on 1.5" recommendation strengthens — a 1.5 transport captures only the bottom two rows, and both have cheaper paths (dot-netty mTLS; config).
• The gRPC-alleviation answer partially flips at 1.6, but only for Akka-adjacent channels. ScadaBridge's HTTP notify-and-fetch for deploy configs, and any "too big for the frame cap" workaround, exist purely because of rows 2–3; under Artery those payloads could legitimately ride remoting. The genuinely non-Akka gRPC (mxaccessgw, HistorianGateway, LocalDb's deliberate cluster-independence) stays gRPC regardless — see the mapping table below.
• New costs arrive with the benefits: Artery is a new wire protocol (flag-day cutover, no rolling mix with 1.5 remoting — tolerable given the stop-pairs-together discipline), .NET 9+ only, and it drags in the platform/ops constraints below (libmsquic, UDP firewall rules, Windows Server 2022+, macOS dev friction) — those attach to any QUIC adoption, 1.5 or 1.6.

6. Platform and ops constraints (System.Net.Quic)

• APIs are stable since .NET 9; the apps are .NET 10 — fine.
• Windows: requires Windows 11 / Server 2022+ (older versions lack the TLS 1.3 crypto APIs). ⚠️ Check the wonder-* boxes — if any production VM is Server 2019 or older, QUIC remoting is a hard no there.
• Linux: libmsquic must be installed in every image — the aspnet:10.0-based images and the docker-dev rig would all need apt-get install libmsquic from packages.microsoft.com.
• macOS (dev machine): only partially supported — Homebrew libmsquic + a DYLD_FALLBACK_LIBRARY_PATH export, outside Microsoft's test matrix. Offline test suites that spin up real ActorSystems with remoting would grow a fragile native dependency on the Mac.
• Firewalls: remoting moves from TCP to UDP — every rule for 4053/etc. needs a UDP twin, and OT network gear is notoriously unkind to UDP.

7. .NET-native alternatives to MsQuic

Under the 1.5 SPI every transport degenerates into the same ordered byte pipe (§5), so candidate wire protocols differ on dependencies, platform reach, and TLS posture — not on features. By that metric, ranked:

1. Managed sockets + SslStream + System.IO.Pipelines — the best fit if anything gets built. 100% in-box managed .NET: zero native dependencies, runs everywhere the apps run today (macOS test suites, any Windows Server version, stock aspnet:10.0 images, TCP firewall rules unchanged). TLS 1.3 via SslStream is optional — can match the current plaintext posture with an mTLS upgrade path, unlike QUIC's hard TLS mandate. Arbitrary frame sizes. Strategically this is a "modern DotNetty replacement": it sheds the effectively unmaintained DotNetty dependency, which is one of 1.6's own goals. JVM precedent that this is respectable: JVM Artery ships TCP and TLS-TCP modes alongside Aeron, and TCP Artery reached ~650k msg/s — most of Artery's gain is the pipeline, not the wire. What it gives up vs QUIC: fast handshake and connection migration — neither of which was a driver here.

Throughput expectations, from real data: this exact idea was prototyped by to11mtm in PR #4594 (Oct 2020, draft, never merged) — an Akka.Streams/managed-socket TCP transport under the classic pipeline. On the author's i7-8750H: DotNetty baseline ~220–240k msg/s; the managed transport alone ~170–240k (parity at best); adding aggressive zero-copy work + a hand-written protobuf deserializer reached ~240–300k (+10–25%), and the residual bottlenecks identified were all pipeline, not wire: serialization, the PDU codec, EndpointWriter dispatch, blocking .Result in serialization paths. I.e. the transport swap is throughput-neutral; only attacking the layer above moves the number, and even extreme effort caps near +25%. (Reference points: the official RemotePingPong docs show 82k → 141k msg/s from flush-batching alone on older hardware; modern hardware on 1.5.x lands roughly 300–600k per association; the 1.6 Artery target is 2–3M.) Two mitigating notes: per-association ceilings apply per site pair, so hub-and-spoke aggregate scales with site count; and for large frames (the raised maximum-frame-size case) a managed transport saturates the wire in bytes/sec — the msg/s ceiling is a small-message phenomenon.

2. HTTP/2 — fully managed multiplexing, but the SPI can't use it. Kestrel (server) + SocketsHttpHandler (client), or gRPC bidirectional streaming as the framing, are fully managed h2 with real stream multiplexing and h2c (no cert mandate). But the 1.5 SPI locks the multiplexing away for exactly the §5 reasons, h2 streams still share one TCP connection (packet-level head-of-line blocking remains), and it adds protocol overhead for no captured benefit. Only distinct win is riding existing Kestrel/gRPC infrastructure. Not preferred over raw sockets.

3. WebSockets (System.Net.WebSockets) — managed, but a single ordered stream, i.e. TCP with extra framing. Only interesting for exotic proxy/firewall traversal. Skip.

No managed QUIC exists. System.Net.Quic is the msquic interop — there is no fully managed QUIC implementation in .NET, and nothing else in the box provides independent streams. So "QUIC without libmsquic" is not on the table, on any .NET version.

Forward-looking note — resolved 2026-07-22: upstream did exactly this. The Artery port that began landing on dev in July 2026 is Artery-TCP (openspec #8313), so the pipeline benefits arrive without libmsquic, and QUIC becomes a later optional substrate — the outcome to11mtm advocated in the #7466 design discussion. This also removes the platform/ops constraints of §6 from the 1.6 adoption path (TCP, no native deps, no UDP firewall changes) unless/until a QUIC substrate is chosen deliberately.

8. Would it actually alleviate the gRPC?

Mostly no — mapping the family's gRPC surfaces:

gRPC usage	Could Akka-over-QUIC replace it?
mxaccessgw ↔ clients	No. It exists for COM bitness isolation; the worker is .NET 4.8 x86, which can't even load a .NET 9+ QUIC stack, and callers aren't cluster members.
HistorianGateway API	No. A standalone service contract for multiple heterogeneous consumers — service RPC is the right shape.
LocalDb pair sync	Technically closest, but deliberately no. It runs between the same nodes as the Akka cluster, but its whole value is being independent of cluster state (it's what lets a node boot from cache when things are degraded), and it's consumable by non-Akka apps. Coupling it to remoting would weaken it. Its 4 MB cap is a gRPC config choice, not a law.
ScadaBridge site→central (S&F etc.)	Already Akka where Akka fits; the bulk/artifact paths that avoid remoting avoid it because of the single-channel head-of-line problem — which, per §4, a 1.5 QUIC transport does not fix.

The "big messages so we don't need side channels" hope is really an Artery feature (dedicated chunked large-message streams), not a QUIC-transport feature.

8a. The overlooked 1.5-era tool: Akka.Cluster.Chunking (added 2026-07-22)

One mechanism was missing from the original analysis — Petabridge's Akka.Cluster.Chunking (Apache-2.0), which exists precisely to bridge the pre-Artery gap this report is about: sending large messages (5 MB+) over Akka.Remote without raising maximum-frame-size and without head-of-line blocking. It chunks at the application layer on top of Akka.Delivery's reliable-delivery machinery (ordered, ack'd, resend-on-loss), so every chunk is an ordinary sub-frame-cap remoting message and heartbeats/system messages interleave between chunks — directly defusing the §2 guardrail concern with zero transport work.

Usage is pleasantly transparent: ChunkingManager.For(system) then chunker.DeliverChunked(message, recipient, sender) (Akka.Hosting: .AddChunkingManager()); config via akka.cluster.chunking.chunk-size (default 64 kB), outbound-queue-capacity (default 20), request-timeout (default 5 s). Unlike raw Akka.Delivery, no per-flow ProducerController/ConsumerController pair has to be hand-wired — the manager runs the delivery channels per node pair.

The honest caveats for our context:

• Maturity. v0.4.0 (Aug 2025), and the README says plainly it is "currently an experiment and is available on an as-is basis." For an OT product that's a real adoption bar; budget for owning/forking it if Petabridge doesn't graduate it.
• Acceptance ≠ delivery. DeliverChunked's task completes when the message is accepted for delivery, not when the far side has it — callers needing end-to-end confirmation still build their own ack on top.
• Requires Akka.Cluster, in-cluster only. It rides remoting between cluster members. It does not cross the ScadaBridge site↔central boundary — two separate Akka clusters bridged by ClusterClient/gRPC — which is exactly where the family's big payloads actually flow. Nor does ClusterClient itself chunk (the tunneled payload + receptionist envelope must fit one frame). It could be bolted on — ClusterClient replies carry real IActorRefs, so raw Akka.Delivery could then run point-to-point over a direct cross-cluster remoting association — but that reopens exactly the direct-remoting coupling (firewall surface, quarantine interplay) the seed-partitioned boundary exists to avoid; large cross-boundary payloads stay on the gRPC side-channels.
• Correctness tool, not throughput tool — shallow queues and the reliable-delivery handshake make it slower than plain tells; it removes a ceiling, it doesn't add speed.

Re-mapping §8 with this in hand changes no conclusion: mxaccessgw/HistorianGateway peers aren't cluster members; LocalDb sync is independent of cluster state by design; the ScadaBridge bulk/artifact side channels are cross-cluster, out of reach. Where it is the right answer is any future within-cluster large-message need (e.g. a big payload between the members of one pair): reach for Akka.Cluster.Chunking (or raw Akka.Delivery with ProducerController.Settings.ChunkLargeMessagesBytes, the in-box primitive it wraps) instead of raising the frame cap — until 1.6's Artery large-message lanes make both workarounds unnecessary.

9. Options

A. Don't build it; tune what we have; track 1.6 (recommended). Best-practice fit: transport-layer plumbing is exactly the code an OT product wants to own least, and the upstream design explicitly targets the wishlist (large-message channels, quarantine-bypass control channel — the latter directly relevant to the 2-node availability-first failover posture). Concretely: bump 1.5.62 → 1.5.70 (eight releases of remoting/streams fixes, including a cluster-wedging consistent-hash-router bug), and if specific pains exist now, prefer Akka.Cluster.Chunking / Akka.Delivery chunking (§8a) over raising maximum-frame-size for any within-cluster large message, and/or enable the new dot-netty mTLS. Subscribe to #7466; when real alphas ship, the pair-restart discipline makes adoption cheap.

B. Build a classic-SPI QUIC transport now. ~3–5 weeks to production quality per §3. Choose this only if the specific wins (TLS 1.3 with modern cert handling, UDP traversal, fast reconnect) matter enough to own a transport — knowing it delivers no throughput/multiplexing gain and gets obsoleted by 1.6.

C. Weekend-scale spike, not production. Implement the minimal Transport/AssociationHandle pair, run a two-node pair on the rig, benchmark against dot-netty. Genuinely worthwhile as de-risking/learning for the eventual 1.6 migration (certs, libmsquic packaging, UDP firewall rules all get exercised) without owning it in production. A reasonable middle path.

Recommendation: A, with C if hands-on evidence is wanted — the honest summary is that for this workload, an MsQuic transport on Akka 1.5 buys security and handshake ergonomics that are mostly available from config today, while the features that would actually reduce the gRPC side-channels only arrive with the Artery pipeline rewrite — now in active development upstream (see the §1 status update) — with Akka.Cluster.Chunking (§8a) as the interim in-cluster answer.

Sources

• akka.net #7466 — Epic: Akka.Remote Quic-based Transport (Artery)
• Petabridge — Akka.NET v1.6 Roadmap
• Akka.NET releases
• Akka.NET milestones
• Akka.NET remoting transports documentation
• Akka.NET PR #7851 — mutual TLS for DotNetty transport
• QUIC support in .NET (System.Net.Quic)
• petabridge/Akka.Cluster.Chunking — v0.4.0, Apache-2.0, experimental
• Akka.NET reliable delivery (Akka.Delivery)