scadaproj/akka_multidc_clusters.md

Deep dive: Akka Multi-DC cluster support, ported to Akka.NET, applied to our topology

Research report, 2026-07-22. Question: Akka (JVM) has multi-DC cluster support that Akka.NET does not. Given our layout (central 2-node cluster + N site 2-node clusters), what would porting it take, and how would it be used in our systems?

TL;DR: Multi-DC maps onto our topology almost perfectly — "one DC per pair" gives per-site leadership, per-site singletons, per-site oldest-node primaries, and WAN-partition tolerance inside ONE cluster, which is precisely the problem set the OtOpcUa per-cluster mesh program and ScadaBridge's three bespoke bridge transports were built to solve. But the port is core-gossip surgery that has already defeated the Akka.NET maintainers twice (PRs #3284 and #4111, the latter a 67-commit functionally-complete draft abandoned in 2019 over multi-node-test instability, now milestoned to the far-future 2.0.0). And two of our hardest cross-cluster problems — the 128 KB frame limit and bulk data movement — are not clustering problems and survive multi-DC untouched. Recommendation: don't port; keep the mesh-program direction; treat #4111 as a candidate for an upstream-contribution play only if the mesh program's transport phases (2/5) prove heavier than expected. Details and options below.

---

1. What Akka multi-DC actually is (JVM; GA since Akka 2.5.6, 2017)

One correction to the framing: this isn't an upcoming JVM feature — it has been GA on the JVM side since 2017. What's true is that it never made it into Akka.NET. Semantics (from the Akka 2.6 docs — 2.6.20 is the last Apache-2.0 version, which matters for porting; 2.7+ is BSL-licensed):

• DC membership is just a role. akka.cluster.multi-data-center.self-data-center = "site-a" adds role dc-site-a. One cluster, every member tagged. Default DC is "default" — which is why untagged and tagged nodes can't mix (JVM rolling 2.4→2.5 upgrades hit exactly this, akka/akka #25496).
• One leader per DC. Each DC's leader performs joining/up/leaving/removal transitions for its own members only. Crucially: members can join and leave while DCs are partitioned from each other — impossible in a plain cluster, where any unreachable member blocks convergence and therefore blocks all leader actions fleet-wide.
• Two failure detectors. akka.cluster.failure-detector intra-DC (all-to-all within the DC, as today) and akka.cluster.multi-data-center.failure-detector cross-DC, run only by the cross-data-center-connections (default 5) oldest nodes per DC. Cross-DC unreachability raises UnreachableDataCenter / ReachableDataCenter events — it is treated as "the WAN link is bad," explicitly NOT fed into member downing: "For network partitions between data centers the system should typically not down the unreachable nodes, but instead wait until it heals or a decision is made by a human or external monitoring system."
• Cluster Singleton is per-DC. Start the manager on all nodes with 3 DCs defined → 3 live singletons, one per DC. Proxies default to the local DC and take a dataCenter parameter to target another DC's singleton. A "global" singleton = start its manager only in one designated DC and accept unavailability when that DC is cut off.
• Cluster Sharding is per-DC (own coordinator/regions per DC; cross-DC proxies available). Not relevant to us — neither app uses sharding.
• Distributed Pub/Sub is NOT DC-scoped. The mediator replicates its registry among all cluster nodes (optionally restricted by role) — so topics span every DC by default. This is the single most valuable property for OtOpcUa (§4).
• Singleton placement and per-DC leadership both key off oldest member within the DC — i.e. the JVM feature's per-DC primary rule is literally our ActiveNodeEvaluator.SelfIsOldestUp / OtOpcUa Phase-0b IsDriverPrimary rule, applied per pair.

2. Where Akka.NET stands on it

• Issue #3261 (opened Jan 2018 by Aaronontheweb) is the tracking issue. Current milestone: 2.0.0 — Akka.Typed, i.e. no committed timeline; it is not in the 1.6 plan (1.6 = Artery/QUIC, .NET Standard drop, new Streams materializer — see the companion report akka_msquic_transport.md).
• It was implemented once: Horusiath's PR #3284 (2018, milestoned 1.4.0, closed), preserved by Aaron as draft PR #4111 (Dec 2019, still open). 67 commits touching membership state, gossip, cross-DC heartbeats, singleton/sharding DC-awareness, failure detection, cluster events, and serialization, plus multi-node specs for split-brain and sunny-weather scenarios. It reportedly passed on .NET Framework but died on Multi-Node TestKit instability on .NET Core ("151 tests failing" at one point; deadlocks in MembershipState lazy init, convergence races, CoordinatedShutdown exit-hook problems). It was dropped from 1.4.0 for review/verification bandwidth and never revived. Its base is mid-2018 dev — six years and two major version lines behind today's 1.5.70.

So "porting the support" concretely means: re-porting from the JVM 2.6.20 sources (Apache 2.0 — do NOT copy from 2.7+/BSL) with #4111 as a .NET-idiom reference, onto Akka.NET 1.5.x, and carrying it as a fork until upstream lands its own in 2.0.0.

3. The mapping onto our topology

The fit is genuinely striking. Model: every 2-node pair is a DC in one spanning cluster.

DC	Members	Today
dc-central	central-a, central-b	ScadaBridge central cluster / OtOpcUa admin pair
dc-site-a	site-a-1, site-a-2	separate site cluster (SB) / same mesh (OtOpcUa)
dc-site-b …	…	…

What each multi-DC property buys us:

• Per-DC oldest = pair primary, by construction. The exact rule both products converged on (ScadaBridge ActiveNodeEvaluator.SelfIsOldestUp; OtOpcUa Phase 0b) becomes the framework's own scoping — the mesh design's core motivation ("every Primary-gated decision asks a cluster-wide election about a pair-local resource") dissolves.
• Per-DC singletons. OtOpcUa's admin-operations singleton stays effectively central by starting its manager only on dc-central nodes; per-pair singletons (e.g. a per-site drain coordinator) come free.
• WAN-safe failure handling. Intra-pair failure detection + our auto-down(15s) availability-over-partition-safety posture stays within the pair, where we want it. A WAN blip to a site raises UnreachableDataCenter — no downing, no quarantine churn, and (the big one) membership operations at other sites keep working during the outage. Today's OtOpcUa single mesh has neither property, which is a large part of why the mesh program exists.
• Pub/sub spans DCs. OtOpcUa's nine DPS topics + deploy notify channel keep working unchanged across the whole fleet. The Secrets Akka replicator (DPS-based) would also span everything — today it explicitly cannot cross ScadaBridge's separate clusters (why secrets use SQL-hub mode there).
• Direct actor messaging replaces ClusterClient. In one spanning cluster, receptionist contact-point management, per-site client caching, and DB-driven address refresh (CentralCommunicationActor's _siteClients, the 60 s Site-row polling) reduce to ordinary cluster-aware messaging.

What multi-DC does NOT fix (important)

• The 128 KB frame limit and serialization posture are unchanged. ScadaBridge's HTTP notify-and-fetch (deploy configs) and gRPC streaming (:8083, high-volume attribute/alarm events) exist because Akka messaging is the wrong channel for bulk payloads — that remains true in a multi-DC cluster. Those channels stay. (Same conclusion as the MsQuic report: bulk channels aren't clustering problems.)
• Store-and-forward stays. At-least-once site→central delivery with a durable buffer (StoreAndForwardService + replicated sf_messages) is an application guarantee; multi-DC changes the pipe, not the need for the buffer.
• Security got harder, not easier. Today an attacker on the WAN can reach ClusterClient/gRPC seams; in a spanning cluster they can attempt to join the cluster. Akka remoting is unauthenticated h2c-equivalent in both products (accepted risk, mesh design §6.3) — a spanning cluster raises the value of revisiting that.
• Persistence single-writer caveats (the JVM docs' sharding-across-DCs warning) don't bite us — no sharding, and pair-local stores are LocalDb-replicated with HLC/LWW already.

4. Per-system assessment

OtOpcUa — the big beneficiary, but the timing is awkward

OtOpcUa today is one mesh spanning everything — i.e. it is already shaped like a multi-DC cluster, minus the DC semantics. The per-cluster mesh program (OtOpcUa/docs/plans/2026-07-22-per-cluster-mesh-program.md, Phases 1–7) exists to fix exactly the things DC-tagging would fix in place:

Mesh-program phase	Under multi-DC
0a auto-down / 0b oldest-primary	Already done; both carry over unchanged (intra-DC)
1 ClusterNode address columns, DB-sourced ack set	Mostly unnecessary — membership stays visible fleet-wide
2 ClusterClient comm actors + receptionists	Unnecessary — one cluster, direct messaging
3 config fetch-and-cache from central	Still required — driven by "sites have no SQL Server," not by clustering
4 cut driver-side ConfigDb	Still required — same reason
5 gRPC migration of 7 observability topics	Unnecessary — DPS keeps working
6 mesh partition into pairs	Replaced by DC tagging + seed layout (much smaller)
7 failover drills / live gates	Still required, plus new cross-DC drills

So multi-DC would delete the two transport-migration phases (2 and 5 — the heaviest pure-plumbing work) and shrink 1 and 6, while leaving the genuinely deployment-driven phases (3, 4) intact. That is a real prize. It is also exactly why the timing is awkward: the program was just sequenced on the ScadaBridge-proven separate-cluster shape, and switching horses means betting those phases on a forked cluster core instead of on patterns already running in production-adjacent code.

ScadaBridge — already solved this; migration value is low

ScadaBridge built the separate-cluster bridges years-of-work deep and they are live: per-site ClusterClients with DB-driven contact refresh, per-node receptionist registration, gRPC server-on-site/client-on-central streaming, token-gated HTTP deploy fetch, S&F with active-node gating. Two details actually make it multi-DC-ready — every cluster already shares the one ActorSystem name "scadabridge" (separation is purely seed-node partitioning), and the roles already include site-{SiteId} — but what multi-DC would replace is only the ClusterClient control channel and the address-book machinery. The gRPC and HTTP channels stay (frame limit), S&F stays, and the CLI's ClusterClient entry point would need rework. Replacing a proven, tested seam with a forked gossip layer for a cleaner control channel is a poor trade on its own.

5. What the port itself takes

Scope (from the JVM 2.6.20 implementation and the #4111 diff):

1. Membership/gossip core — Member gains a dataCenter (derived from the dc- role, so the gossip wire format is nearly unchanged — the DC rides in roles); MembershipState/gossip convergence, leader election, and reachability calculations all become DC-filtered (convergence ignores cross-DC observations; per-DC leader actions permitted during inter-DC splits). This is the delicate 20% that is 80% of the risk.
2. Heartbeating — keep today's all-to-all within a DC; add the cross-DC heartbeater run by the N oldest per DC with its own failure-detector config; new UnreachableDataCenter / ReachableDataCenter events.
3. Downing — AutoDowning/SBR must become DC-scoped (JVM SBR is; our auto-down posture then applies per pair, which is exactly the HA stance we want). A non-DC-aware downing provider in a multi-DC cluster is actively dangerous (downs the far side on WAN blips).
4. Cluster tools — Singleton manager/proxy DC-awareness (small — it is role-filtering); sharding proxies (skippable for us); DPS needs nothing.
5. Serialization — additive protobuf changes for the new heartbeat/gossip fields; whole-fleet cutover required (mixed DC-aware/unaware nodes don't interoperate — the JVM had the same constraint). Our stop/start-pairs-together discipline makes this tolerable, but it is a flag-day, not rolling.
6. Verification — the real cost. This is what killed it upstream twice: MNTK specs for split-brain, join-during-partition, cross-DC unreachability, singleton-per-DC handover. Our substitute would be rig-based live gates (docker-dev with real WAN-partition injection via tc/iptables), which we're good at — but the failure modes are convergence races that soak tests catch and unit tests don't.

Effort honestly: the code port is plausibly 4–8 weeks against the Apache-2.0 JVM reference with #4111 as a map. The validation to trust it under an OT platform's availability requirements is the same again or more, and then it's a permanent fork of Akka.Cluster — every 1.5.x bump (there have been 8 since our pin) becomes a merge with the most intricate code in the framework, with no upstream support for the modified layer, until upstream's own version lands in 2.0.0 and strands the fork entirely. Precedent matters here: the framework's own authors, with the MNTK infrastructure we lack, shelved it twice.

6. Options

A. Don't port. Continue the current direction (recommended). ScadaBridge keeps its proven bridges; OtOpcUa executes the mesh program as sequenced (selfform prerequisite → Phases 1–7). Every pattern in that program is already running in ScadaBridge production code; risk lives in application code we own rather than framework internals we'd fork. Cost: OtOpcUa still pays for Phases 2 and 5 (ClusterClient + gRPC plumbing), and cross-cluster pub/sub stays unavailable (secrets replication keeps SQL-hub mode on ScadaBridge).

B. Port multi-DC (fork Akka.Cluster). Scope per §5, ~2–4 months including validation, plus the standing fork tax. Payoff concentrated in OtOpcUa (skip Phases 2/5, keep DPS, keep one mesh with correct per-pair scoping). Only rational as an upstream-contribution play — rebase/re-port against current dev, revive #4111 with Petabridge (possibly as sponsored work), so the fork has an exit. As a private fork: not worth it for two products with a working alternative.

C. Multi-DC-lite on stock Akka.NET (keep OtOpcUa's single mesh, emulate the scoping app-side): role-scoped singletons and role-scoped DPS groups per pair (supported today), the already-done oldest-Up-per-role primary rule, plus a custom IDowningProvider that only downs same-site peers (pluggable, no fork). What cannot be emulated: convergence — one unreachable node anywhere still blocks joins/leaves fleet-wide, and cross-site heartbeating stays all-to-all over the WAN. Tolerable for a small, static fleet that rarely adds nodes mid-outage; it is a known-sharp-edges compromise, not a fix. Worth considering only if the mesh program stalls.

Recommendation: A. The mesh program should keep going — it's sequenced, its prerequisite plan exists, and its shape is production-proven next door. Revisit B strictly as "fund/contribute the upstream port" if Phases 2/5 blow up in practice, and note in that case the decision point comes before Phase 2 starts, because B makes exactly those phases unnecessary.

Sources

• Akka 2.6 Multi-DC Cluster docs (last Apache-2.0 line)
• Akka classic Distributed Pub/Sub docs (mediator scope: all nodes / role-filtered)
• akka.net #3261 — Multi-Data Center Support in Akka.Cluster (milestone 2.0.0)
• akka.net PR #3284 — original multi-DC implementation (Horusiath)
• akka.net PR #4111 — preserved 67-commit draft (Aaronontheweb)
• akka/akka #25496 — 2.4→2.5 rolling-upgrade DC requirement failure
• ScadaBridge cross-cluster seam map: CentralCommunicationActor.cs / SiteCommunicationActor.cs (ClusterClient), SiteStreamGrpcClient/Server.cs + sitestream.proto (gRPC :8083), HttpDeploymentConfigFetcher.cs + DeploymentConfigEndpoints.cs (token-gated HTTP), StoreAndForwardService.cs, ActiveNodeEvaluator.cs; ActorSystem name shared at AkkaHostedService.cs:191
• OtOpcUa mesh design + program: OtOpcUa/docs/plans/2026-07-21-per-cluster-mesh-design.md, 2026-07-22-per-cluster-mesh-program.md