lmxopcua/docs/plans/2026-07-21-per-cluster-mesh-design.md

Per-Cluster Akka Mesh — design

Status: design for review. Phases 0a and 0b are DONE and merged (2026-07-21); Phases 1–7 are not started and each needs its own plan. Decisions settled 2026-07-21 — see §6: one central DB with driver nodes disconnected from it, the two-node keep-oldest gap closed upstream and now closed here too (§6.2), and their auth posture matched for now.

1. What this changes

Today OtOpcUa runs one Akka mesh containing every node of every application Cluster. That was a deliberate choice — docs/plans/2026-06-07-per-cluster-scoping.md, "Per-ClusterId Scoping (hub-and-spoke single mesh)" — so the deploy channel could stay in-mesh: AdminUI → admin-operations singleton → deployments topic → every node filters to its own ClusterId.

The proposal is to move to the sister project's shape: one Akka mesh per application Cluster, two nodes maximum, with central and clusters joined by explicit transports instead of cluster gossip.

The motivation is that redundancy becomes correct by construction. Every Primary-gated decision (inbound device writes, native alarm acks, fleet alerts, ServiceLevel, and — since Phase 2 — the alarm-history drain) currently asks "am I the driver Primary?" of a cluster-wide election, while every resource being gated is pair-local. Splitting the mesh collapses the two scopes into one and the question stops being ambiguous.

2. What ScadaBridge actually does

Researched directly from /Users/dohertj2/Desktop/ScadaBridge. Five things differ from the summary in its own CLAUDE.md, and they matter:

Three transports cross the boundary, not two.

Transport	Direction	Carries
Akka ClusterClient	central → site (+ replies)	command/control: deploy notify, lifecycle, queries, subscribe handshakes
gRPC server-streaming	central dials into the site	real-time attribute + alarm events
Plain HTTP, token-gated	site → central	the deployment config itself (notify-and-fetch)

The gRPC direction is inverted from the obvious guess. Data flows site→central, but each site node hosts the gRPC server (Kestrel h2c :8083) and central is the client. There is no gRPC server on central at all.

Active node = OLDEST Up member, explicitly not the cluster leader. ActiveNodeEvaluator.SelfIsOldestUp is documented as "THE single definition of 'active node'":

"Cluster LEADERSHIP (lowest address) is an Akka-internal concept that diverges from singleton placement permanently once the original first node restarts and rejoins; every product-level active/standby decision must use this evaluator, never cluster.State.Leader."

The equivalence oldest-Up == where ClusterSingletonManager places singletons is the design.

All clusters share one ActorSystem name ("scadabridge", hardcoded). They are separate clusters only by seed-node partitioning — required, because Akka.Remote address matching means ClusterClient could not reach a differently-named system.

Site nodes carry two roles: "Site" and "site-{SiteId}". Singletons scope to the site-specific role.

Other load-bearing details:

• Central discovers sites from the database (a Site entity with NodeAAddress/NodeBAddress
  • GrpcNodeAAddress/GrpcNodeBAddress), refreshed every 60 s and on admin change. Sites discover central from appsettings — static, restart required. The asymmetry is deliberate.
• Exactly one actor is exposed per side via ClusterClientReceptionist.RegisterService — /user/central-communication and /user/site-communication — registered per node, not as a singleton, so contact rotation reaches whichever node answers.
• No central buffering when a site is unreachable. The send is dropped with a warning and the caller's Ask times out. "It keeps the central coordinator stateless with respect to site availability." A ConnectionStateChanged mechanism was built for this and deleted as dead code.
• Every unhandled message replies with a typed failure rather than dropping, so a central Ask can never stall on a missing handler. CORRECTED 2026-07-22 (Phase 2 recon). This overstates what ScadaBridge does. Neither of their comm actors has a ReceiveAny or an Unhandled override: the typed-failure idiom is per message type, and it fires only when a registration is missing (no ClusterClient for the site yet, no handler registered for that type). A genuinely unknown message type still dead-letters exactly as it would anywhere else. Do not build a catch-all failure reply believing you are copying them. OtOpcUa's Phase 2 comm actors list each type explicitly and deliberately let an unknown type dead-letter loudly rather than republish it blind.
• Sender preservation is the whole Ask idiom: _client.Tell(new ClusterClient.Send(path, msg), Sender) routes the reply straight back to the waiting Ask, bypassing the comm actor.

The registered outage gap — read before copying

Component-ClusterInfrastructure.md:113: "Only the FIRST seed listed in Cluster:SeedNodes may self-join to form a new cluster… a lone restarted non-first-seed node (with the first seed still down) loops on InitJoin forever — never Up, never routable. This is why the two-node keep-oldest oldest/active-node crash is a total-outage gap: after the oldest dies the younger survivor self-downs, and it cannot re-bootstrap alone. Recovery is operator-driven."

Their failover drill has a mode that exists "to make the registered gap observable — not to pretend it is covered." A two-node keep-oldest mesh has an acknowledged total-outage hole. If OtOpcUa adopts 2-node meshes, this must be decided deliberately, not inherited.

CLOSED 2026-07-22. The InitJoin half of this gap is fixed in both repos by self-first seed ordering: every node that is one of its own seeds lists ITSELF as seed-nodes[0], which is the only ordering under which Akka runs FirstSeedNodeProcess and can form a cluster with no peer answering. (An earlier Cluster:SelfFormAfter watchdog was tried here and retired — it raced the join handshake.) See docs/Redundancy.md §"Bootstrap: self-first seed ordering". Today's driver-only site nodes are seeded by central-1 only and are exempt; the per-cluster mesh removes even that exception by making every pair node a seed of its own mesh.

Both inter-cluster transports are unauthenticated

Akka remoting has no TLS, no secure cookie, no trusted-selection-paths. The gRPC listener is h2c with no auth covering SiteStreamService — including two Pull RPCs that return audit rows. The only authenticated pieces are the HTTP fetch token and LocalDbSyncAuthInterceptor, which gates only /localdb_sync.v1.LocalDbSync/. Their boundary assumes a trusted network.

OtOpcUa already ships that same fail-closed interceptor (LocalDb Phase 1) — it is a ready-made template if we choose to authenticate more than they did.

3. What OtOpcUa has today

Nine cross-node DPS topics plus a singleton, all currently relying on one mesh:

Channel	Direction	Purpose
deployments / deployment-acks	central ↔ nodes	deploy dispatch + acks
driver-control	central → nodes	AdminUI Reconnect/Restart
redundancy-state	central → nodes	roles + peer probe results
alerts	nodes → central	fleet alarms → /alerts
driver-health, driver-resilience-status, fleet-status, script-logs	nodes → central	AdminUI live panels
admin-operations singleton	central	operator ack/shelve into engines

Three facts make the split cheaper than that table suggests:

1. The deploy path is already notify-and-fetch. DispatchDeployment carries only DeploymentId + RevisionHash + CorrelationId; the node fetches the artifact itself. We independently arrived at the pattern ScadaBridge had to retrofit, and we do not carry its 128 KB Akka frame exposure on this path.
2. Deploy state already has a DB substrate. ConfigPublishCoordinator persists per-node ACKs to NodeDeploymentState so a singleton failover recovers in-flight state from the DB.
3. ClusterNode rows already enumerate every node per application Cluster — the same table AdminOperationsActor groups by cluster. This replaces the coordinator's one genuinely mesh-bound dependency: it derives its expected-ack set from Akka.Cluster.State.Members filtered by role.

So the deploy channel needs only a small notify + ack transport. The nine live-telemetry channels are the real work, and they are exactly what ScadaBridge built ClusterClient + gRPC for.

4. A defect to fix regardless of this design

OtOpcUa currently uses two different node-selection rules at once:

• Split-brain resolution is keep-oldest (Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:49).
• ClusterSingletonManager places singletons on the oldest member.
• But RedundancyStateActor.BuildSnapshot elects Primary from RoleLeader("driver") — lowest address among driver members.

Those select the same node only by coincidence. After a restart-and-rejoin the role leader and the oldest member diverge permanently, which means the node the SBR protects and hosts every singleton on need not be the node the data-plane gates consider Primary. This is precisely the rule ScadaBridge forbids by name, and for the reason it gives: "both sides claim leadership during a partition" — the dual-primary shape archreview 03/S4 exists to prevent.

Switching the role derivation to oldest-Up-with-role is small, is correct under either topology, and should not wait for this design. It also happens to be exactly what the separate-mesh model needs, so it is not throwaway work.

FIXED 2026-07-21 (Phase 0b). RedundancyStateActor.BuildSnapshot selects the oldest Up member carrying the driver role, matching singleton placement; NodeRedundancyState.IsRoleLeaderForDriver was renamed IsDriverPrimary so the contract stops asserting the old derivation. Pinned by RedundancyPrimaryElectionTests, which builds a real two-node cluster in which the oldest node holds the higher address, so the two orderings genuinely disagree.

5. Target architecture

Mirror ScadaBridge's shape, adapted to OtOpcUa's existing substrate:

• One Akka mesh per application Cluster, two nodes max. Same ActorSystem name (otopcua) everywhere; separation by seed-node partitioning only.
• Roles per node: driver plus a cluster-specific cluster-{ClusterId}, with singletons scoped to the cluster-specific role.
• Active node = oldest Up member with role (§4), one definition, used by the data-plane gates, ServiceLevel, and the alarm drain alike.
• Central ↔ cluster command/control over ClusterClient, with exactly one receptionist-registered actor per side and central discovering cluster node addresses from ClusterNode rows (extended with Akka + gRPC addresses, mirroring their Site entity).
• Deploy stays notify-and-fetch, but the fetch changes source. The notify crosses via ClusterClient; the artifact then comes from central, not from the ConfigDb the driver node no longer connects to (§6.1), and is cached in LocalDb. The coordinator's expected-ack set comes from ClusterNode rows instead of cluster membership.
• Driver nodes hold no ConfigDb connection. LocalDb becomes their steady-state configuration store; five current DB consumers are re-homed or re-sourced per §6.1.
• Live telemetry over gRPC, central dialling into each cluster node, replacing the seven observability topics with one stream contract carrying a oneof event — additive-only field evolution, contract-locked by test.

Copied: their node-local configuration store fed by fetch-and-cache (§6.1), their oldest-Up active-node rule (§4), their two-node keep-oldest posture including its acknowledged outage gap (§6.2), and their unauthenticated inter-cluster transports (§6.3).

Deliberately not copied: their transient-only central alarm cache. OtOpcUa persists alarm history centrally through the historian, and Phase 2's replicated store-and-forward buffer already guarantees delivery across a failover — so there is no reason to adopt a design whose stated trade-off is that a new active node re-seeds alarm state from scratch.

6. Decisions (settled 2026-07-21)

6.1 DECIDED — one central DB, and driver nodes never connect to it

Verified in ScadaBridge before adopting. Program.cs calls AddConfigurationDatabase(configDbConnectionString) at line 264 — inside the Central branch (89–478). The Site branch begins at line 479. Site nodes never register the central configuration database at all; they receive config from central (notify → token-gated HTTP fetch) and persist it locally. There is still exactly one authoritative configuration database — it is simply central-only.

OtOpcUa adopts the same shape: the ConfigDb stays the single source of truth, and driver nodes stop connecting to it. They obtain their configuration from the central nodes and cache it locally in LocalDb.

This promotes LocalDb from an outage cache to the driver node's steady-state configuration store. That is a reframing of the Phase 1 work rather than a contradiction of it: boot-from-cache stops being the exceptional path and becomes the normal one, and "central SQL unreachable" stops being a degraded mode for driver nodes because they never held a connection to lose. The chunking, SHA-256 verification, newest-2 retention and pair replication all carry over unchanged.

Scope — five driver-side ConfigDb consumers must be re-homed or re-sourced. Audited:

Consumer	Current use	Disposition
DriverHostActor	artifact fetch + ack writes (5 CreateDbContext sites)	artifact via fetch-and-cache; acks via the ack transport
EfAlarmConditionStateStore	Part 9 alarm condition state	move to LocalDb — it is pair-local state, the same journey Phase 2 made for the alarm S&F buffer
DbHealthProbeActor	DB reachability feeding ServiceLevel tiering	semantics change — a driver node has no DB to probe; decide what health input replaces it
OpcUaPublishActor	(audit required)	TBD
ServiceCollectionExtensions	registration	follows the above

DbHealthProbeActor deserves particular care: DB health is currently a ServiceLevel input, and under this model it stops existing for driver nodes. That is a behaviour change on a client-visible value, not just an internal refactor.

6.2 CORRECTED — the gap was closed in ScadaBridge, and OtOpcUa still has it

An earlier revision of this section was wrong. It reported the two-node keep-oldest outage gap as still open, citing Component-ClusterInfrastructure.md:113 and Gitea PR #12 (which made the failover drill honest about the gap rather than closing it). That doc line was stale: ScadaBridge closed the gap in cf3bd52f, "feat(cluster): auto-down downing strategy — either-node crash now fails over (owner decision 2026-07-21: availability over partition-safety)".

Its commit message carries the analysis, live-proven on their rig:

"Two-node keep-oldest could NEVER survive a crash of the oldest/active node: Akka.NET 1.5.62 KeepOldest.OldestDecision only lets down-if-alone rescue a side with >= 2 members, so the 1-vs-1 survivor takes DownReachable and downs ITSELF — proven live on the rig ('SBR took decision … including myself') before this change. static-quorum(1) is worse (IsTooManyMembers -> DownAll); keep-majority just re-keys the fatal crash to the lowest address."

OtOpcUa is running that exact configuration today — Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:49-53: active-strategy = "keep-oldest", keep-oldest { down-if-alone = on }, stable-after = 15s, with BuildClusterOptions setting the matching typed KeepOldestOption { DownIfAlone = true }.

So a two-node OtOpcUa pair could not fail over when the oldest node crashed: the survivor downed itself. That was a total-outage defect in the redundancy feature itself, independent of this design.

FIXED 2026-07-21 (Phase 0a). Cluster:SplitBrainResolverStrategy now selects the downing provider and defaults to auto-down — Akka's AutoDowning with a 15s window — so the leader among the reachable members downs the unreachable peer and a crash of either node fails over. keep-oldest remains selectable for deployments that prefer partition-safety. Pinned by SplitBrainResolverActivationTests, which asserts the provider read back off a running ActorSystem rather than the configured one. See docs/Redundancy.md.

Its live gate is still outstanding, and cannot run on today's rig: the failure only appears in a 1-vs-1 split, and docker-dev is a single six-node mesh. The crash-the-oldest drill therefore lands with Phase 6/7, which makes every mesh exactly two nodes.

The bootstrap half of the gap is also closed (2026-07-22). Downing was only ever one of the two outage modes: even with auto-down, a node cold-starting while its peer was dead looped on InitJoin forever, because Akka lets only seed-nodes[0] form a new cluster. The fix is self-first seed ordering — each seed node lists itself first, enforced at boot by AkkaClusterOptionsValidator. The Cluster:SelfFormAfter watchdog first shipped for this is deleted: a timer outside Akka's join handshake cannot tell "no seed answered" from "a seed answered and the join is in flight", and it islanded a manually-failed-over node live. Pinned by SelfFirstSeedBootstrapTests; see docs/Redundancy.md §"Bootstrap: self-first seed ordering". Under this design each pair node is a seed of its own two-node mesh, so the ordering covers both sides and the current site-node exemption disappears.

6.3 DECIDED — match ScadaBridge's auth posture for now

Inter-cluster transports stay unauthenticated and unencrypted, as theirs are: the boundary assumes a trusted network. Recorded as an accepted risk rather than an oversight, with two notes for whenever it is revisited: LocalDbSyncAuthInterceptor is already in our tree as a fail-closed template, and the surface most worth gating first is any Pull-style RPC that returns historical rows.

7. Sequencing sketch

Deliberately not a task plan — per-phase plans follow, one at a time.

Phase	Content	Independent of the split?
0a	Downing strategy: keep-oldest cannot survive an oldest-node crash in a 2-node cluster (§6.2)	Yes — DONE 2026-07-21, live gate deferred to Phase 7
0b	Oldest-Up role derivation (§4)	Yes — DONE 2026-07-21
1	ClusterNode gains Akka + gRPC address columns; coordinator sources its expected-ack set from the DB	Yes — DONE 2026-07-22 (see below)
2	Comm actors + receptionist registration; ClusterClient transport; deploy notify + acks across the boundary	No
3	Config fetch-and-cache: artifact served by central, driver nodes read LocalDb (§6.1)	No
4	Cut the driver-side ConfigDb connection: re-home EfAlarmConditionStateStore, resolve the DbHealthProbeActor ServiceLevel input, audit OpcUaPublishActor	No
5	gRPC stream contract; migrate the seven observability topics	No
6	Mesh partition: per-cluster seed nodes, cluster-scoped roles + singletons; rig models the real topology	No
7	Failover drill (both directions, per §6.2); live gate	No

Phases 3 and 4 are the ones that change a running system's data path rather than its wiring, and each deserves its own live gate.

Phase 1 as shipped (2026-07-22)

Plan: 2026-07-21-per-cluster-mesh-phase1.md. Two deviations from the sketch above, both decided before implementing:

• No cluster-scope filtering of the expected-ack set. The plan called for it, but there is no cluster-scoped deployment to filter on: Deployment has no ClusterId, ConfigComposer.SnapshotAndFlattenAsync always snapshots the whole DB, and DeploymentArtifact.ResolveClusterScope is node-side self-scoping of a fleet-wide artifact. The expected set is every enabled ClusterNode row, which is what the membership rule produced too.
• No per-node role column. The membership rule filtered on the driver role and the DB has none. Rather than add one — a second declaration of node roles, free to drift from Cluster:Roles — every ClusterNode row is now defined to be a driver node. An admin-only node must not be given one.

Also shipped beyond the sketch: ClusterNodeAddressReconciler (Task 4), an admin singleton that catches AkkaPort drifting from the node's own Cluster:Port. Phase 2 must revisit it — once the meshes split, an admin node cannot see site members and every site row would report EnabledRowNotInCluster forever.

8. Risks

• LocalDb becomes load-bearing for configuration, not just resilient. Phase 1 built it as a fallback; §6.1 makes it the driver node's only config store. A cache defect that was previously a degraded-mode bug becomes a total-availability bug — the #485 unreadable-artifact class, with the blast radius raised.
• DbHealthProbeActor feeds a client-visible value. Removing the driver-side DB connection changes what ServiceLevel means on those nodes; that is an interop-visible change, not internal.
• The rig currently models the topology that would be abolished. Phase 6 rewrites docker-dev/docker-compose.yml substantially, and every live gate that depends on it.
• Losing gossip loses free fleet-wide observability. Seven AdminUI panels are fed by DPS today; each needs an explicit stream and a reconnect story.
• Akka frame size. We are clean on the deploy path, but anything new that carries payload over ClusterClient inherits the 128 KB default with log-frame-size-exceeding off — a silent single-message drop that leaves the association healthy. If we send payload at all, set both.
• Do not stack application-level last-write-wins on LocalDb's HLC. Their SiteStorageService carries this warning in code. Verified: our deployment_pointer upsert is unconditional and the alarm sink is idempotent on a payload hash, so we are currently clean — keep it that way.