# Deep dive: Akka Multi-DC cluster support, ported to Akka.NET, applied to our topology

*Research report, 2026-07-22. Question: Akka (JVM) has multi-DC cluster support that Akka.NET does
not. Given our layout (central 2-node cluster + N site 2-node clusters), what would porting it take,
and how would it be used in our systems?*

**TL;DR:** Multi-DC maps onto our topology almost perfectly — "one DC per pair" gives per-site
leadership, per-site singletons, per-site oldest-node primaries, and WAN-partition tolerance inside
ONE cluster, which is precisely the problem set the OtOpcUa per-cluster mesh program and
ScadaBridge's three bespoke bridge transports were built to solve. But the port is core-gossip
surgery that has already defeated the Akka.NET maintainers twice (PRs #3284 and #4111, the latter a
67-commit functionally-complete draft abandoned in 2019 over multi-node-test instability, now
milestoned to the far-future 2.0.0). And two of our hardest cross-cluster problems — the 128 KB
frame limit and bulk data movement — are *not* clustering problems and survive multi-DC untouched.
Recommendation: don't port; keep the mesh-program direction; treat #4111 as a candidate for an
upstream-contribution play only if the mesh program's transport phases (2/5) prove heavier than
expected. Details and options below.

---

## 1. What Akka multi-DC actually is (JVM; GA since Akka 2.5.6, 2017)

One correction to the framing: this isn't an upcoming JVM feature — it has been GA on the JVM side
since 2017. What's true is that it never made it into Akka.NET. Semantics (from the
[Akka 2.6 docs](https://doc.akka.io/docs/akka/2.6/typed/cluster-dc.html) — 2.6.20 is the last
Apache-2.0 version, which matters for porting; 2.7+ is BSL-licensed):

- **DC membership is just a role.** `akka.cluster.multi-data-center.self-data-center = "site-a"`
  adds role `dc-site-a`. One cluster, every member tagged. Default DC is `"default"` — which is why
  untagged and tagged nodes can't mix (JVM rolling 2.4→2.5 upgrades hit exactly this, akka/akka
  #25496).
- **One leader per DC.** Each DC's leader performs joining/up/leaving/removal transitions for *its
  own* members only. Crucially: **members can join and leave while DCs are partitioned from each
  other** — impossible in a plain cluster, where any unreachable member blocks convergence and
  therefore blocks all leader actions fleet-wide.
- **Two failure detectors.** `akka.cluster.failure-detector` intra-DC (all-to-all within the DC,
  as today) and `akka.cluster.multi-data-center.failure-detector` cross-DC, run only by the
  `cross-data-center-connections` (default 5) oldest nodes per DC. Cross-DC unreachability raises
  `UnreachableDataCenter` / `ReachableDataCenter` events — it is treated as "the WAN link is bad,"
  explicitly NOT fed into member downing: *"For network partitions between data centers the system
  should typically not down the unreachable nodes, but instead wait until it heals or a decision is
  made by a human or external monitoring system."*
- **Cluster Singleton is per-DC.** Start the manager on all nodes with 3 DCs defined → 3 live
  singletons, one per DC. Proxies default to the local DC and take a `dataCenter` parameter to
  target another DC's singleton. A "global" singleton = start its manager only in one designated DC
  and accept unavailability when that DC is cut off.
- **Cluster Sharding is per-DC** (own coordinator/regions per DC; cross-DC proxies available).
  Not relevant to us — neither app uses sharding.
- **Distributed Pub/Sub is NOT DC-scoped.** The mediator replicates its registry among *all*
  cluster nodes (optionally restricted by role) — so topics span every DC by default. This is the
  single most valuable property for OtOpcUa (§4).
- Singleton placement and per-DC leadership both key off **oldest member within the DC** — i.e.
  the JVM feature's per-DC primary rule is literally our `ActiveNodeEvaluator.SelfIsOldestUp` /
  OtOpcUa Phase-0b `IsDriverPrimary` rule, applied per pair.

## 2. Where Akka.NET stands on it

- [Issue #3261](https://github.com/akkadotnet/akka.net/issues/3261) (opened Jan 2018 by
  Aaronontheweb) is the tracking issue. Current milestone: **2.0.0 — Akka.Typed**, i.e. no
  committed timeline; it is not in the 1.6 plan (1.6 = Artery/QUIC, .NET Standard drop, new
  Streams materializer — see the companion report `akka_msquic_transport.md`).
- It was **implemented once**: Horusiath's [PR #3284](https://github.com/akkadotnet/akka.net/pull/3284)
  (2018, milestoned 1.4.0, closed), preserved by Aaron as draft
  [PR #4111](https://github.com/akkadotnet/akka.net/pull/4111) (Dec 2019, still open). 67 commits
  touching membership state, gossip, cross-DC heartbeats, singleton/sharding DC-awareness, failure
  detection, cluster events, and serialization, plus multi-node specs for split-brain and
  sunny-weather scenarios. It reportedly passed on .NET Framework but **died on Multi-Node TestKit
  instability on .NET Core** ("151 tests failing" at one point; deadlocks in MembershipState lazy
  init, convergence races, CoordinatedShutdown exit-hook problems). It was dropped from 1.4.0 for
  review/verification bandwidth and never revived. Its base is **mid-2018 `dev`** — six years and
  two major version lines behind today's 1.5.70.

So "porting the support" concretely means: re-porting from the JVM 2.6.20 sources (Apache 2.0 —
do NOT copy from 2.7+/BSL) with #4111 as a .NET-idiom reference, onto Akka.NET 1.5.x, and carrying
it as a fork until upstream lands its own in 2.0.0.

## 3. The mapping onto our topology

The fit is genuinely striking. Model: **every 2-node pair is a DC** in one spanning cluster.

| DC | Members | Today |
|---|---|---|
| `dc-central` | central-a, central-b | ScadaBridge central cluster / OtOpcUa admin pair |
| `dc-site-a` | site-a-1, site-a-2 | separate site cluster (SB) / same mesh (OtOpcUa) |
| `dc-site-b` … | … | … |

What each multi-DC property buys us:

- **Per-DC oldest = pair primary, by construction.** The exact rule both products converged on
  (ScadaBridge `ActiveNodeEvaluator.SelfIsOldestUp`; OtOpcUa Phase 0b) becomes the framework's own
  scoping — the mesh design's core motivation ("every Primary-gated decision asks a cluster-wide
  election about a pair-local resource") dissolves.
- **Per-DC singletons.** OtOpcUa's `admin-operations` singleton stays effectively central by
  starting its manager only on `dc-central` nodes; per-pair singletons (e.g. a per-site drain
  coordinator) come free.
- **WAN-safe failure handling.** Intra-pair failure detection + our `auto-down`(15s)
  availability-over-partition-safety posture stays *within the pair*, where we want it. A WAN blip
  to a site raises `UnreachableDataCenter` — no downing, no quarantine churn, and (the big one)
  **membership operations at other sites keep working during the outage**. Today's OtOpcUa single
  mesh has neither property, which is a large part of why the mesh program exists.
- **Pub/sub spans DCs.** OtOpcUa's nine DPS topics + deploy notify channel keep working unchanged
  across the whole fleet. The Secrets Akka replicator (DPS-based) would also span everything —
  today it explicitly cannot cross ScadaBridge's separate clusters (why secrets use SQL-hub mode
  there).
- **Direct actor messaging replaces ClusterClient.** In one spanning cluster, receptionist
  contact-point management, per-site client caching, and DB-driven address refresh
  (`CentralCommunicationActor`'s `_siteClients`, the 60 s `Site`-row polling) reduce to ordinary
  cluster-aware messaging.

### What multi-DC does NOT fix (important)

- **The 128 KB frame limit and serialization posture are unchanged.** ScadaBridge's HTTP
  notify-and-fetch (deploy configs) and gRPC streaming (`:8083`, high-volume attribute/alarm
  events) exist because Akka messaging is the wrong channel for bulk payloads — that remains true
  in a multi-DC cluster. Those channels stay. (Same conclusion as the MsQuic report: bulk channels
  aren't clustering problems.)
- **Store-and-forward stays.** At-least-once site→central delivery with a durable buffer
  (`StoreAndForwardService` + replicated `sf_messages`) is an application guarantee; multi-DC
  changes the pipe, not the need for the buffer.
- **Security got harder, not easier.** Today an attacker on the WAN can reach ClusterClient/gRPC
  seams; in a spanning cluster they can attempt to *join the cluster*. Akka remoting is
  unauthenticated h2c-equivalent in both products (accepted risk, mesh design §6.3) — a spanning
  cluster raises the value of revisiting that.
- **Persistence single-writer caveats** (the JVM docs' sharding-across-DCs warning) don't bite us —
  no sharding, and pair-local stores are LocalDb-replicated with HLC/LWW already.

## 4. Per-system assessment

### OtOpcUa — the big beneficiary, but the timing is awkward

OtOpcUa *today* is one mesh spanning everything — i.e. it is already shaped like a multi-DC
cluster, minus the DC semantics. The per-cluster mesh program
(`OtOpcUa/docs/plans/2026-07-22-per-cluster-mesh-program.md`, Phases 1–7) exists to fix exactly the
things DC-tagging would fix in place:

| Mesh-program phase | Under multi-DC |
|---|---|
| 0a auto-down / 0b oldest-primary | Already done; both carry over unchanged (intra-DC) |
| 1 `ClusterNode` address columns, DB-sourced ack set | **Mostly unnecessary** — membership stays visible fleet-wide |
| 2 ClusterClient comm actors + receptionists | **Unnecessary** — one cluster, direct messaging |
| 3 config fetch-and-cache from central | **Still required** — driven by "sites have no SQL Server," not by clustering |
| 4 cut driver-side ConfigDb | **Still required** — same reason |
| 5 gRPC migration of 7 observability topics | **Unnecessary** — DPS keeps working |
| 6 mesh partition into pairs | Replaced by DC tagging + seed layout (much smaller) |
| 7 failover drills / live gates | Still required, plus new cross-DC drills |

So multi-DC would delete the two transport-migration phases (2 and 5 — the heaviest pure-plumbing
work) and shrink 1 and 6, while leaving the genuinely deployment-driven phases (3, 4) intact. That
is a real prize. It is also exactly why the timing is awkward: the program was just sequenced on
the ScadaBridge-proven separate-cluster shape, and switching horses means betting those phases on a
forked cluster core instead of on patterns already running in production-adjacent code.

### ScadaBridge — already solved this; migration value is low

ScadaBridge built the separate-cluster bridges years-of-work deep and they are live: per-site
`ClusterClient`s with DB-driven contact refresh, per-node receptionist registration, gRPC
server-on-site/client-on-central streaming, token-gated HTTP deploy fetch, S&F with active-node
gating. Two details actually make it multi-DC-*ready* — every cluster already shares the one
ActorSystem name `"scadabridge"` (separation is purely seed-node partitioning), and the roles
already include `site-{SiteId}` — but what multi-DC would replace is only the ClusterClient control
channel and the address-book machinery. The gRPC and HTTP channels stay (frame limit), S&F stays,
and the CLI's ClusterClient entry point would need rework. Replacing a proven, tested seam with a
forked gossip layer for a cleaner control channel is a poor trade on its own.

## 5. What the port itself takes

Scope (from the JVM 2.6.20 implementation and the #4111 diff):

1. **Membership/gossip core** — `Member` gains a `dataCenter` (derived from the `dc-` role, so the
   gossip **wire format is nearly unchanged** — the DC rides in roles); `MembershipState`/gossip
   convergence, leader election, and reachability calculations all become DC-filtered (convergence
   ignores cross-DC observations; per-DC leader actions permitted during inter-DC splits). This is
   the delicate 20% that is 80% of the risk.
2. **Heartbeating** — keep today's all-to-all within a DC; add the cross-DC heartbeater run by the
   N oldest per DC with its own failure-detector config; new `UnreachableDataCenter` /
   `ReachableDataCenter` events.
3. **Downing** — `AutoDowning`/SBR must become DC-scoped (JVM SBR is; our `auto-down` posture then
   applies per pair, which is exactly the HA stance we want). A non-DC-aware downing provider in a
   multi-DC cluster is actively dangerous (downs the far side on WAN blips).
4. **Cluster tools** — Singleton manager/proxy DC-awareness (small — it is role-filtering);
   sharding proxies (skippable for us); DPS needs nothing.
5. **Serialization** — additive protobuf changes for the new heartbeat/gossip fields; whole-fleet
   cutover required (mixed DC-aware/unaware nodes don't interoperate — the JVM had the same
   constraint). Our stop/start-pairs-together discipline makes this tolerable, but it is a
   flag-day, not rolling.
6. **Verification — the real cost.** This is what killed it upstream twice: MNTK specs for
   split-brain, join-during-partition, cross-DC unreachability, singleton-per-DC handover. Our
   substitute would be rig-based live gates (docker-dev with real WAN-partition injection via
   `tc`/iptables), which we're good at — but the failure modes are convergence races that soak
   tests catch and unit tests don't.

**Effort honestly:** the code port is plausibly 4–8 weeks against the Apache-2.0 JVM reference with
#4111 as a map. The validation to *trust* it under an OT platform's availability requirements is
the same again or more, and then it's a **permanent fork of Akka.Cluster** — every 1.5.x bump
(there have been 8 since our pin) becomes a merge with the most intricate code in the framework,
with no upstream support for the modified layer, until upstream's own version lands in 2.0.0 and
strands the fork entirely. Precedent matters here: the framework's own authors, with the MNTK
infrastructure we lack, shelved it twice.

## 6. Options

**A. Don't port. Continue the current direction (recommended).** ScadaBridge keeps its proven
bridges; OtOpcUa executes the mesh program as sequenced (selfform prerequisite → Phases 1–7). Every
pattern in that program is already running in ScadaBridge production code; risk lives in
application code we own rather than framework internals we'd fork. Cost: OtOpcUa still pays for
Phases 2 and 5 (ClusterClient + gRPC plumbing), and cross-cluster pub/sub stays unavailable
(secrets replication keeps SQL-hub mode on ScadaBridge).

**B. Port multi-DC (fork Akka.Cluster).** Scope per §5, ~2–4 months including validation, plus the
standing fork tax. Payoff concentrated in OtOpcUa (skip Phases 2/5, keep DPS, keep one mesh with
correct per-pair scoping). Only rational as an **upstream-contribution play** — rebase/re-port
against current `dev`, revive #4111 with Petabridge (possibly as sponsored work), so the fork has
an exit. As a private fork: not worth it for two products with a working alternative.

**C. Multi-DC-lite on stock Akka.NET** (keep OtOpcUa's single mesh, emulate the scoping
app-side): role-scoped singletons and role-scoped DPS groups per pair (supported today), the
already-done oldest-Up-per-role primary rule, plus a **custom `IDowningProvider`** that only downs
same-site peers (pluggable, no fork). What cannot be emulated: convergence — one unreachable node
anywhere still blocks joins/leaves fleet-wide, and cross-site heartbeating stays all-to-all over
the WAN. Tolerable for a small, static fleet that rarely adds nodes mid-outage; it is a
known-sharp-edges compromise, not a fix. Worth considering only if the mesh program stalls.

**Recommendation: A.** The mesh program should keep going — it's sequenced, its prerequisite plan
exists, and its shape is production-proven next door. Revisit B strictly as "fund/contribute the
upstream port" if Phases 2/5 blow up in practice, and note in that case the decision point comes
*before* Phase 2 starts, because B makes exactly those phases unnecessary.

## Sources

- [Akka 2.6 Multi-DC Cluster docs](https://doc.akka.io/docs/akka/2.6/typed/cluster-dc.html) (last Apache-2.0 line)
- [Akka classic Distributed Pub/Sub docs](https://doc.akka.io/docs/akka/2.6/distributed-pub-sub.html) (mediator scope: all nodes / role-filtered)
- [akka.net #3261 — Multi-Data Center Support in Akka.Cluster](https://github.com/akkadotnet/akka.net/issues/3261) (milestone 2.0.0)
- [akka.net PR #3284 — original multi-DC implementation (Horusiath)](https://github.com/akkadotnet/akka.net/pull/3284)
- [akka.net PR #4111 — preserved 67-commit draft (Aaronontheweb)](https://github.com/akkadotnet/akka.net/pull/4111)
- [akka/akka #25496 — 2.4→2.5 rolling-upgrade DC requirement failure](https://github.com/akka/akka/issues/25496)
- ScadaBridge cross-cluster seam map: `CentralCommunicationActor.cs` / `SiteCommunicationActor.cs` (ClusterClient),
  `SiteStreamGrpcClient/Server.cs` + `sitestream.proto` (gRPC :8083), `HttpDeploymentConfigFetcher.cs` +
  `DeploymentConfigEndpoints.cs` (token-gated HTTP), `StoreAndForwardService.cs`, `ActiveNodeEvaluator.cs`;
  ActorSystem name shared at `AkkaHostedService.cs:191`
- OtOpcUa mesh design + program: `OtOpcUa/docs/plans/2026-07-21-per-cluster-mesh-design.md`,
  `2026-07-22-per-cluster-mesh-program.md`
