dohertj2/scadaproj
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: Akka.NET transport deep-dives — MsQuic/1.6 Artery and multi-DC clusters

Browse Source 
Two research reports (2026-07-22):
- akka_msquic_transport.md: what a QUIC transport on Akka 1.5.62 would take,
  the 1.5-vs-1.6 pipeline delta that gates the benefits, .NET-native
  alternatives (managed sockets ranked first), and measured throughput
  evidence (PR #4594: transport swap is throughput-neutral).
- akka_multidc_clusters.md: JVM multi-DC semantics mapped onto the
  central-pair + site-pairs topology, the stalled Akka.NET port (#3261 /
  PR #4111), per-system fit vs the OtOpcUa mesh program, and the
  before-Phase-2 decision point.
Recommendation in both: don't build; track upstream.
This commit is contained in:
Joseph Doherty
2026-07-22 06:41:36 -04:00
parent 5919e29969
commit dbfd5e7a7e
2 changed files with 543 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
akka_multidc_clusters.md
+241
View File
@@ -0,0 +1,241 @@
# Deep dive: Akka Multi-DC cluster support, ported to Akka.NET, applied to our topology
*Research report, 2026-07-22. Question: Akka (JVM) has multi-DC cluster support that Akka.NET does
not. Given our layout (central 2-node cluster + N site 2-node clusters), what would porting it take,
and how would it be used in our systems?*
**TL;DR:** Multi-DC maps onto our topology almost perfectly — "one DC per pair" gives per-site
leadership, per-site singletons, per-site oldest-node primaries, and WAN-partition tolerance inside
ONE cluster, which is precisely the problem set the OtOpcUa per-cluster mesh program and
ScadaBridge's three bespoke bridge transports were built to solve. But the port is core-gossip
surgery that has already defeated the Akka.NET maintainers twice (PRs #3284 and #4111, the latter a
67-commit functionally-complete draft abandoned in 2019 over multi-node-test instability, now
milestoned to the far-future 2.0.0). And two of our hardest cross-cluster problems — the 128 KB
frame limit and bulk data movement — are *not* clustering problems and survive multi-DC untouched.
Recommendation: don't port; keep the mesh-program direction; treat #4111 as a candidate for an
upstream-contribution play only if the mesh program's transport phases (2/5) prove heavier than
expected. Details and options below.
---
## 1. What Akka multi-DC actually is (JVM; GA since Akka 2.5.6, 2017)
One correction to the framing: this isn't an upcoming JVM feature — it has been GA on the JVM side
since 2017. What's true is that it never made it into Akka.NET. Semantics (from the
[Akka 2.6 docs](https://doc.akka.io/docs/akka/2.6/typed/cluster-dc.html) — 2.6.20 is the last
Apache-2.0 version, which matters for porting; 2.7+ is BSL-licensed):
- **DC membership is just a role.** `akka.cluster.multi-data-center.self-data-center = "site-a"`
adds role `dc-site-a`. One cluster, every member tagged. Default DC is `"default"` — which is why
untagged and tagged nodes can't mix (JVM rolling 2.4→2.5 upgrades hit exactly this, akka/akka
#25496).
- **One leader per DC.** Each DC's leader performs joining/up/leaving/removal transitions for *its
own* members only. Crucially: **members can join and leave while DCs are partitioned from each
other** — impossible in a plain cluster, where any unreachable member blocks convergence and
therefore blocks all leader actions fleet-wide.
- **Two failure detectors.** `akka.cluster.failure-detector` intra-DC (all-to-all within the DC,
as today) and `akka.cluster.multi-data-center.failure-detector` cross-DC, run only by the
`cross-data-center-connections` (default 5) oldest nodes per DC. Cross-DC unreachability raises
`UnreachableDataCenter` / `ReachableDataCenter` events — it is treated as "the WAN link is bad,"
explicitly NOT fed into member downing: *"For network partitions between data centers the system
should typically not down the unreachable nodes, but instead wait until it heals or a decision is
made by a human or external monitoring system."*
- **Cluster Singleton is per-DC.** Start the manager on all nodes with 3 DCs defined → 3 live
singletons, one per DC. Proxies default to the local DC and take a `dataCenter` parameter to
target another DC's singleton. A "global" singleton = start its manager only in one designated DC
and accept unavailability when that DC is cut off.
- **Cluster Sharding is per-DC** (own coordinator/regions per DC; cross-DC proxies available).
Not relevant to us — neither app uses sharding.
- **Distributed Pub/Sub is NOT DC-scoped.** The mediator replicates its registry among *all*
cluster nodes (optionally restricted by role) — so topics span every DC by default. This is the
single most valuable property for OtOpcUa (§4).
- Singleton placement and per-DC leadership both key off **oldest member within the DC** — i.e.
the JVM feature's per-DC primary rule is literally our `ActiveNodeEvaluator.SelfIsOldestUp` /
OtOpcUa Phase-0b `IsDriverPrimary` rule, applied per pair.
## 2. Where Akka.NET stands on it
- [Issue #3261](https://github.com/akkadotnet/akka.net/issues/3261) (opened Jan 2018 by
Aaronontheweb) is the tracking issue. Current milestone: **2.0.0 — Akka.Typed**, i.e. no
committed timeline; it is not in the 1.6 plan (1.6 = Artery/QUIC, .NET Standard drop, new
Streams materializer — see the companion report `akka_msquic_transport.md`).
- It was **implemented once**: Horusiath's [PR #3284](https://github.com/akkadotnet/akka.net/pull/3284)
(2018, milestoned 1.4.0, closed), preserved by Aaron as draft
[PR #4111](https://github.com/akkadotnet/akka.net/pull/4111) (Dec 2019, still open). 67 commits
touching membership state, gossip, cross-DC heartbeats, singleton/sharding DC-awareness, failure
detection, cluster events, and serialization, plus multi-node specs for split-brain and
sunny-weather scenarios. It reportedly passed on .NET Framework but **died on Multi-Node TestKit
instability on .NET Core** ("151 tests failing" at one point; deadlocks in MembershipState lazy
init, convergence races, CoordinatedShutdown exit-hook problems). It was dropped from 1.4.0 for
review/verification bandwidth and never revived. Its base is **mid-2018 `dev`** — six years and
two major version lines behind today's 1.5.70.
So "porting the support" concretely means: re-porting from the JVM 2.6.20 sources (Apache 2.0 —
do NOT copy from 2.7+/BSL) with #4111 as a .NET-idiom reference, onto Akka.NET 1.5.x, and carrying
it as a fork until upstream lands its own in 2.0.0.
## 3. The mapping onto our topology
The fit is genuinely striking. Model: **every 2-node pair is a DC** in one spanning cluster.
| DC | Members | Today |
|---|---|---|
| `dc-central` | central-a, central-b | ScadaBridge central cluster / OtOpcUa admin pair |
| `dc-site-a` | site-a-1, site-a-2 | separate site cluster (SB) / same mesh (OtOpcUa) |
| `dc-site-b` … | … | … |
What each multi-DC property buys us:
- **Per-DC oldest = pair primary, by construction.** The exact rule both products converged on
(ScadaBridge `ActiveNodeEvaluator.SelfIsOldestUp`; OtOpcUa Phase 0b) becomes the framework's own
scoping — the mesh design's core motivation ("every Primary-gated decision asks a cluster-wide
election about a pair-local resource") dissolves.
- **Per-DC singletons.** OtOpcUa's `admin-operations` singleton stays effectively central by
starting its manager only on `dc-central` nodes; per-pair singletons (e.g. a per-site drain
coordinator) come free.
- **WAN-safe failure handling.** Intra-pair failure detection + our `auto-down`(15s)
availability-over-partition-safety posture stays *within the pair*, where we want it. A WAN blip
to a site raises `UnreachableDataCenter` — no downing, no quarantine churn, and (the big one)
**membership operations at other sites keep working during the outage**. Today's OtOpcUa single
mesh has neither property, which is a large part of why the mesh program exists.
- **Pub/sub spans DCs.** OtOpcUa's nine DPS topics + deploy notify channel keep working unchanged
across the whole fleet. The Secrets Akka replicator (DPS-based) would also span everything —
today it explicitly cannot cross ScadaBridge's separate clusters (why secrets use SQL-hub mode
there).
- **Direct actor messaging replaces ClusterClient.** In one spanning cluster, receptionist
contact-point management, per-site client caching, and DB-driven address refresh
(`CentralCommunicationActor`'s `_siteClients`, the 60 s `Site`-row polling) reduce to ordinary
cluster-aware messaging.
### What multi-DC does NOT fix (important)
- **The 128 KB frame limit and serialization posture are unchanged.** ScadaBridge's HTTP
notify-and-fetch (deploy configs) and gRPC streaming (`:8083`, high-volume attribute/alarm
events) exist because Akka messaging is the wrong channel for bulk payloads — that remains true
in a multi-DC cluster. Those channels stay. (Same conclusion as the MsQuic report: bulk channels
aren't clustering problems.)
- **Store-and-forward stays.** At-least-once site→central delivery with a durable buffer
(`StoreAndForwardService` + replicated `sf_messages`) is an application guarantee; multi-DC
changes the pipe, not the need for the buffer.
- **Security got harder, not easier.** Today an attacker on the WAN can reach ClusterClient/gRPC
seams; in a spanning cluster they can attempt to *join the cluster*. Akka remoting is
unauthenticated h2c-equivalent in both products (accepted risk, mesh design §6.3) — a spanning
cluster raises the value of revisiting that.
- **Persistence single-writer caveats** (the JVM docs' sharding-across-DCs warning) don't bite us —
no sharding, and pair-local stores are LocalDb-replicated with HLC/LWW already.
## 4. Per-system assessment
### OtOpcUa — the big beneficiary, but the timing is awkward
OtOpcUa *today* is one mesh spanning everything — i.e. it is already shaped like a multi-DC
cluster, minus the DC semantics. The per-cluster mesh program
(`OtOpcUa/docs/plans/2026-07-22-per-cluster-mesh-program.md`, Phases 1–7) exists to fix exactly the
things DC-tagging would fix in place:
| Mesh-program phase | Under multi-DC |
|---|---|
| 0a auto-down / 0b oldest-primary | Already done; both carry over unchanged (intra-DC) |
| 1 `ClusterNode` address columns, DB-sourced ack set | **Mostly unnecessary** — membership stays visible fleet-wide |
| 2 ClusterClient comm actors + receptionists | **Unnecessary** — one cluster, direct messaging |
| 3 config fetch-and-cache from central | **Still required** — driven by "sites have no SQL Server," not by clustering |
| 4 cut driver-side ConfigDb | **Still required** — same reason |
| 5 gRPC migration of 7 observability topics | **Unnecessary** — DPS keeps working |
| 6 mesh partition into pairs | Replaced by DC tagging + seed layout (much smaller) |
| 7 failover drills / live gates | Still required, plus new cross-DC drills |
So multi-DC would delete the two transport-migration phases (2 and 5 — the heaviest pure-plumbing
work) and shrink 1 and 6, while leaving the genuinely deployment-driven phases (3, 4) intact. That
is a real prize. It is also exactly why the timing is awkward: the program was just sequenced on
the ScadaBridge-proven separate-cluster shape, and switching horses means betting those phases on a
forked cluster core instead of on patterns already running in production-adjacent code.
### ScadaBridge — already solved this; migration value is low
ScadaBridge built the separate-cluster bridges years-of-work deep and they are live: per-site
`ClusterClient`s with DB-driven contact refresh, per-node receptionist registration, gRPC
server-on-site/client-on-central streaming, token-gated HTTP deploy fetch, S&F with active-node
gating. Two details actually make it multi-DC-*ready* — every cluster already shares the one
ActorSystem name `"scadabridge"` (separation is purely seed-node partitioning), and the roles
already include `site-{SiteId}` — but what multi-DC would replace is only the ClusterClient control
channel and the address-book machinery. The gRPC and HTTP channels stay (frame limit), S&F stays,
and the CLI's ClusterClient entry point would need rework. Replacing a proven, tested seam with a
forked gossip layer for a cleaner control channel is a poor trade on its own.
## 5. What the port itself takes
Scope (from the JVM 2.6.20 implementation and the #4111 diff):
1. **Membership/gossip core** — `Member` gains a `dataCenter` (derived from the `dc-` role, so the
gossip **wire format is nearly unchanged** — the DC rides in roles); `MembershipState`/gossip
convergence, leader election, and reachability calculations all become DC-filtered (convergence
ignores cross-DC observations; per-DC leader actions permitted during inter-DC splits). This is
the delicate 20% that is 80% of the risk.
2. **Heartbeating** — keep today's all-to-all within a DC; add the cross-DC heartbeater run by the
N oldest per DC with its own failure-detector config; new `UnreachableDataCenter` /
`ReachableDataCenter` events.
3. **Downing** — `AutoDowning`/SBR must become DC-scoped (JVM SBR is; our `auto-down` posture then
applies per pair, which is exactly the HA stance we want). A non-DC-aware downing provider in a
multi-DC cluster is actively dangerous (downs the far side on WAN blips).
4. **Cluster tools** — Singleton manager/proxy DC-awareness (small — it is role-filtering);
sharding proxies (skippable for us); DPS needs nothing.
5. **Serialization** — additive protobuf changes for the new heartbeat/gossip fields; whole-fleet
cutover required (mixed DC-aware/unaware nodes don't interoperate — the JVM had the same
constraint). Our stop/start-pairs-together discipline makes this tolerable, but it is a
flag-day, not rolling.
6. **Verification — the real cost.** This is what killed it upstream twice: MNTK specs for
split-brain, join-during-partition, cross-DC unreachability, singleton-per-DC handover. Our
substitute would be rig-based live gates (docker-dev with real WAN-partition injection via
`tc`/iptables), which we're good at — but the failure modes are convergence races that soak
tests catch and unit tests don't.
**Effort honestly:** the code port is plausibly 4–8 weeks against the Apache-2.0 JVM reference with
#4111 as a map. The validation to *trust* it under an OT platform's availability requirements is
the same again or more, and then it's a **permanent fork of Akka.Cluster** — every 1.5.x bump
(there have been 8 since our pin) becomes a merge with the most intricate code in the framework,
with no upstream support for the modified layer, until upstream's own version lands in 2.0.0 and
strands the fork entirely. Precedent matters here: the framework's own authors, with the MNTK
infrastructure we lack, shelved it twice.
## 6. Options
**A. Don't port. Continue the current direction (recommended).** ScadaBridge keeps its proven
bridges; OtOpcUa executes the mesh program as sequenced (selfform prerequisite → Phases 1–7). Every
pattern in that program is already running in ScadaBridge production code; risk lives in
application code we own rather than framework internals we'd fork. Cost: OtOpcUa still pays for
Phases 2 and 5 (ClusterClient + gRPC plumbing), and cross-cluster pub/sub stays unavailable
(secrets replication keeps SQL-hub mode on ScadaBridge).
**B. Port multi-DC (fork Akka.Cluster).** Scope per §5, ~2–4 months including validation, plus the
standing fork tax. Payoff concentrated in OtOpcUa (skip Phases 2/5, keep DPS, keep one mesh with
correct per-pair scoping). Only rational as an **upstream-contribution play** — rebase/re-port
against current `dev`, revive #4111 with Petabridge (possibly as sponsored work), so the fork has
an exit. As a private fork: not worth it for two products with a working alternative.
**C. Multi-DC-lite on stock Akka.NET** (keep OtOpcUa's single mesh, emulate the scoping
app-side): role-scoped singletons and role-scoped DPS groups per pair (supported today), the
already-done oldest-Up-per-role primary rule, plus a **custom `IDowningProvider`** that only downs
same-site peers (pluggable, no fork). What cannot be emulated: convergence — one unreachable node
anywhere still blocks joins/leaves fleet-wide, and cross-site heartbeating stays all-to-all over
the WAN. Tolerable for a small, static fleet that rarely adds nodes mid-outage; it is a
known-sharp-edges compromise, not a fix. Worth considering only if the mesh program stalls.
**Recommendation: A.** The mesh program should keep going — it's sequenced, its prerequisite plan
exists, and its shape is production-proven next door. Revisit B strictly as "fund/contribute the
upstream port" if Phases 2/5 blow up in practice, and note in that case the decision point comes
*before* Phase 2 starts, because B makes exactly those phases unnecessary.
## Sources
- [Akka 2.6 Multi-DC Cluster docs](https://doc.akka.io/docs/akka/2.6/typed/cluster-dc.html) (last Apache-2.0 line)
- [Akka classic Distributed Pub/Sub docs](https://doc.akka.io/docs/akka/2.6/distributed-pub-sub.html) (mediator scope: all nodes / role-filtered)
- [akka.net #3261 — Multi-Data Center Support in Akka.Cluster](https://github.com/akkadotnet/akka.net/issues/3261) (milestone 2.0.0)
- [akka.net PR #3284 — original multi-DC implementation (Horusiath)](https://github.com/akkadotnet/akka.net/pull/3284)
- [akka.net PR #4111 — preserved 67-commit draft (Aaronontheweb)](https://github.com/akkadotnet/akka.net/pull/4111)
- [akka/akka #25496 — 2.4→2.5 rolling-upgrade DC requirement failure](https://github.com/akka/akka/issues/25496)
- ScadaBridge cross-cluster seam map: `CentralCommunicationActor.cs` / `SiteCommunicationActor.cs` (ClusterClient),
`SiteStreamGrpcClient/Server.cs` + `sitestream.proto` (gRPC :8083), `HttpDeploymentConfigFetcher.cs` +
`DeploymentConfigEndpoints.cs` (token-gated HTTP), `StoreAndForwardService.cs`, `ActiveNodeEvaluator.cs`;
ActorSystem name shared at `AkkaHostedService.cs:191`
- OtOpcUa mesh design + program: `OtOpcUa/docs/plans/2026-07-21-per-cluster-mesh-design.md`,
`2026-07-22-per-cluster-mesh-program.md`