# Per-Cluster Akka Mesh — design > **Status:** design for review. **Phases 0a and 0b are DONE and merged** (2026-07-21); Phases 1–7 > are not started and each needs its own plan. > **Decisions settled 2026-07-21** — see §6: one central DB with driver nodes disconnected from it, > the two-node keep-oldest gap closed upstream and now closed here too (§6.2), and their auth posture > matched for now. ## 1. What this changes Today OtOpcUa runs **one Akka mesh containing every node of every application Cluster**. That was a deliberate choice — `docs/plans/2026-06-07-per-cluster-scoping.md`, "Per-ClusterId Scoping (hub-and-spoke **single mesh**)" — so the deploy channel could stay in-mesh: AdminUI → `admin-operations` singleton → `deployments` topic → every node filters to its own `ClusterId`. The proposal is to move to the sister project's shape: **one Akka mesh per application Cluster, two nodes maximum**, with central and clusters joined by explicit transports instead of cluster gossip. The motivation is that redundancy becomes correct *by construction*. Every Primary-gated decision (inbound device writes, native alarm acks, fleet alerts, ServiceLevel, and — since Phase 2 — the alarm-history drain) currently asks "am I the driver Primary?" of a **cluster-wide** election, while every resource being gated is **pair-local**. Splitting the mesh collapses the two scopes into one and the question stops being ambiguous. ## 2. What ScadaBridge actually does Researched directly from `/Users/dohertj2/Desktop/ScadaBridge`. Five things differ from the summary in its own CLAUDE.md, and they matter: **Three transports cross the boundary, not two.** | Transport | Direction | Carries | |---|---|---| | Akka **ClusterClient** | central → site (+ replies) | command/control: deploy notify, lifecycle, queries, subscribe handshakes | | **gRPC** server-streaming | **central dials into the site** | real-time attribute + alarm events | | Plain **HTTP**, token-gated | site → central | the deployment config itself (notify-and-fetch) | **The gRPC direction is inverted from the obvious guess.** Data flows site→central, but *each site node hosts the gRPC server* (Kestrel h2c :8083) and *central is the client*. There is no gRPC server on central at all. **Active node = OLDEST Up member, explicitly not the cluster leader.** `ActiveNodeEvaluator.SelfIsOldestUp` is documented as *"THE single definition of 'active node'"*: > *"Cluster LEADERSHIP (lowest address) is an Akka-internal concept that diverges from singleton > placement permanently once the original first node restarts and rejoins; every product-level > active/standby decision must use this evaluator, never `cluster.State.Leader`."* The equivalence **oldest-Up == where `ClusterSingletonManager` places singletons** *is* the design. **All clusters share one ActorSystem name** (`"scadabridge"`, hardcoded). They are separate clusters only by seed-node partitioning — required, because Akka.Remote address matching means ClusterClient could not reach a differently-named system. **Site nodes carry two roles:** `"Site"` and `"site-{SiteId}"`. Singletons scope to the **site-specific** role. Other load-bearing details: - **Central discovers sites from the database** (a `Site` entity with `NodeAAddress`/`NodeBAddress` + `GrpcNodeAAddress`/`GrpcNodeBAddress`), refreshed every 60 s and on admin change. Sites discover central from **appsettings** — static, restart required. The asymmetry is deliberate. - **Exactly one actor is exposed per side** via `ClusterClientReceptionist.RegisterService` — `/user/central-communication` and `/user/site-communication` — registered **per node, not as a singleton**, so contact rotation reaches whichever node answers. - **No central buffering when a site is unreachable.** The send is dropped with a warning and the caller's Ask times out. *"It keeps the central coordinator stateless with respect to site availability."* A `ConnectionStateChanged` mechanism was built for this and **deleted as dead code**. - **Every unhandled message replies with a typed failure rather than dropping**, so a central Ask can never stall on a missing handler. - **Sender preservation is the whole Ask idiom**: `_client.Tell(new ClusterClient.Send(path, msg), Sender)` routes the reply straight back to the waiting Ask, bypassing the comm actor. ### The registered outage gap — read before copying > `Component-ClusterInfrastructure.md:113`: *"Only the FIRST seed listed in `Cluster:SeedNodes` may > self-join to form a *new* cluster… a lone restarted non-first-seed node (with the first seed still > down) loops on `InitJoin` forever — never `Up`, never routable. This is why the two-node > keep-oldest **oldest/active-node crash is a total-outage gap**: after the oldest dies the younger > survivor self-downs, and it cannot re-bootstrap alone. Recovery is operator-driven."* Their failover drill has a mode that exists *"to make the registered gap observable — not to pretend it is covered."* **A two-node keep-oldest mesh has an acknowledged total-outage hole.** If OtOpcUa adopts 2-node meshes, this must be decided deliberately, not inherited. ### Both inter-cluster transports are unauthenticated Akka remoting has no TLS, no secure cookie, no `trusted-selection-paths`. The gRPC listener is h2c with no auth covering `SiteStreamService` — including two Pull RPCs that return audit rows. The only authenticated pieces are the HTTP fetch token and `LocalDbSyncAuthInterceptor`, which gates *only* `/localdb_sync.v1.LocalDbSync/`. Their boundary assumes a trusted network. **OtOpcUa already ships that same fail-closed interceptor** (LocalDb Phase 1) — it is a ready-made template if we choose to authenticate more than they did. ## 3. What OtOpcUa has today **Nine cross-node DPS topics plus a singleton**, all currently relying on one mesh: | Channel | Direction | Purpose | |---|---|---| | `deployments` / `deployment-acks` | central ↔ nodes | deploy dispatch + acks | | `driver-control` | central → nodes | AdminUI Reconnect/Restart | | `redundancy-state` | central → nodes | roles + peer probe results | | `alerts` | nodes → central | fleet alarms → `/alerts` | | `driver-health`, `driver-resilience-status`, `fleet-status`, `script-logs` | nodes → central | AdminUI live panels | | `admin-operations` singleton | central | operator ack/shelve into engines | Three facts make the split cheaper than that table suggests: 1. **The deploy path is already notify-and-fetch.** `DispatchDeployment` carries only `DeploymentId` + `RevisionHash` + `CorrelationId`; the node fetches the artifact itself. We independently arrived at the pattern ScadaBridge had to retrofit, and we do **not** carry its 128 KB Akka frame exposure on this path. 2. **Deploy state already has a DB substrate.** `ConfigPublishCoordinator` persists per-node ACKs to `NodeDeploymentState` so a singleton failover recovers in-flight state from the DB. 3. **`ClusterNode` rows already enumerate every node per application Cluster** — the same table `AdminOperationsActor` groups by cluster. This replaces the coordinator's one genuinely mesh-bound dependency: it derives its expected-ack set from `Akka.Cluster.State.Members` filtered by role. So the deploy channel needs only a small notify + ack transport. **The nine live-telemetry channels are the real work**, and they are exactly what ScadaBridge built ClusterClient + gRPC for. ## 4. A defect to fix regardless of this design OtOpcUa currently uses **two different node-selection rules at once**: - Split-brain resolution is **keep-oldest** (`Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:49`). - `ClusterSingletonManager` places singletons on the **oldest** member. - But `RedundancyStateActor.BuildSnapshot` elects Primary from **`RoleLeader("driver")`** — lowest address among driver members. Those select the same node only by coincidence. After a restart-and-rejoin the role leader and the oldest member diverge permanently, which means the node the SBR protects and hosts every singleton on need not be the node the data-plane gates consider Primary. This is precisely the rule ScadaBridge forbids by name, and for the reason it gives: *"both sides claim leadership during a partition"* — the dual-primary shape archreview 03/S4 exists to prevent. **Switching the role derivation to oldest-Up-with-role is small, is correct under either topology, and should not wait for this design.** It also happens to be exactly what the separate-mesh model needs, so it is not throwaway work. **FIXED 2026-07-21 (Phase 0b).** `RedundancyStateActor.BuildSnapshot` selects the oldest Up member carrying the `driver` role, matching singleton placement; `NodeRedundancyState.IsRoleLeaderForDriver` was renamed `IsDriverPrimary` so the contract stops asserting the old derivation. Pinned by `RedundancyPrimaryElectionTests`, which builds a real two-node cluster in which the oldest node holds the *higher* address, so the two orderings genuinely disagree. ## 5. Target architecture Mirror ScadaBridge's shape, adapted to OtOpcUa's existing substrate: - **One Akka mesh per application `Cluster`, two nodes max.** Same ActorSystem name (`otopcua`) everywhere; separation by seed-node partitioning only. - **Roles per node:** `driver` plus a cluster-specific `cluster-{ClusterId}`, with singletons scoped to the cluster-specific role. - **Active node = oldest Up member with role** (§4), one definition, used by the data-plane gates, ServiceLevel, and the alarm drain alike. - **Central ↔ cluster command/control over ClusterClient**, with exactly one receptionist-registered actor per side and central discovering cluster node addresses from `ClusterNode` rows (extended with Akka + gRPC addresses, mirroring their `Site` entity). - **Deploy stays notify-and-fetch, but the fetch changes source.** The notify crosses via ClusterClient; the artifact then comes **from central**, not from the ConfigDb the driver node no longer connects to (§6.1), and is cached in LocalDb. The coordinator's expected-ack set comes from `ClusterNode` rows instead of cluster membership. - **Driver nodes hold no ConfigDb connection.** LocalDb becomes their steady-state configuration store; five current DB consumers are re-homed or re-sourced per §6.1. - **Live telemetry over gRPC**, central dialling into each cluster node, replacing the seven observability topics with one stream contract carrying a `oneof` event — additive-only field evolution, contract-locked by test. **Copied:** their node-local configuration store fed by fetch-and-cache (§6.1), their oldest-Up active-node rule (§4), their two-node keep-oldest posture including its acknowledged outage gap (§6.2), and their unauthenticated inter-cluster transports (§6.3). **Deliberately not copied:** their *transient-only* central alarm cache. OtOpcUa persists alarm history centrally through the historian, and Phase 2's replicated store-and-forward buffer already guarantees delivery across a failover — so there is no reason to adopt a design whose stated trade-off is that a new active node re-seeds alarm state from scratch. ## 6. Decisions (settled 2026-07-21) ### 6.1 DECIDED — one central DB, and driver nodes never connect to it **Verified in ScadaBridge before adopting.** `Program.cs` calls `AddConfigurationDatabase(configDbConnectionString)` at line 264 — inside the Central branch (89–478). The Site branch begins at line 479. **Site nodes never register the central configuration database at all**; they receive config from central (notify → token-gated HTTP fetch) and persist it locally. There is still exactly one authoritative configuration database — it is simply central-only. OtOpcUa adopts the same shape: **the ConfigDb stays the single source of truth, and driver nodes stop connecting to it.** They obtain their configuration from the central nodes and cache it locally in LocalDb. **This promotes LocalDb from an outage cache to the driver node's steady-state configuration store.** That is a reframing of the Phase 1 work rather than a contradiction of it: boot-from-cache stops being the exceptional path and becomes the normal one, and "central SQL unreachable" stops being a degraded mode for driver nodes because they never held a connection to lose. The chunking, SHA-256 verification, newest-2 retention and pair replication all carry over unchanged. **Scope — five driver-side ConfigDb consumers must be re-homed or re-sourced.** Audited: | Consumer | Current use | Disposition | |---|---|---| | `DriverHostActor` | artifact fetch + ack writes (5 `CreateDbContext` sites) | artifact via fetch-and-cache; acks via the ack transport | | `EfAlarmConditionStateStore` | Part 9 alarm condition state | **move to LocalDb** — it is pair-local state, the same journey Phase 2 made for the alarm S&F buffer | | `DbHealthProbeActor` | DB reachability feeding ServiceLevel tiering | **semantics change** — a driver node has no DB to probe; decide what health input replaces it | | `OpcUaPublishActor` | (audit required) | TBD | | `ServiceCollectionExtensions` | registration | follows the above | `DbHealthProbeActor` deserves particular care: DB health is currently a ServiceLevel input, and under this model it stops existing for driver nodes. That is a behaviour change on a client-visible value, not just an internal refactor. ### 6.2 CORRECTED — the gap was closed in ScadaBridge, and OtOpcUa still has it **An earlier revision of this section was wrong.** It reported the two-node keep-oldest outage gap as still open, citing `Component-ClusterInfrastructure.md:113` and Gitea PR [#12](https://gitea.dohertylan.com/dohertj2/ScadaBridge/pulls/12) (which made the failover drill *honest* about the gap rather than closing it). That doc line was stale: ScadaBridge closed the gap in `cf3bd52f`, *"feat(cluster): auto-down downing strategy — either-node crash now fails over (owner decision 2026-07-21: availability over partition-safety)"*. Its commit message carries the analysis, live-proven on their rig: > *"Two-node keep-oldest could NEVER survive a crash of the oldest/active node: Akka.NET 1.5.62 > `KeepOldest.OldestDecision` only lets `down-if-alone` rescue a side with >= 2 members, so the 1-vs-1 > survivor takes `DownReachable` and downs ITSELF — proven live on the rig ('SBR took decision … > including myself') before this change. `static-quorum(1)` is worse (`IsTooManyMembers` -> `DownAll`); > `keep-majority` just re-keys the fatal crash to the lowest address."* **OtOpcUa is running that exact configuration today** — `Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:49-53`: `active-strategy = "keep-oldest"`, `keep-oldest { down-if-alone = on }`, `stable-after = 15s`, with `BuildClusterOptions` setting the matching typed `KeepOldestOption { DownIfAlone = true }`. So a two-node OtOpcUa pair **could not fail over when the oldest node crashed**: the survivor downed itself. That was a total-outage defect in the redundancy feature itself, independent of this design. **FIXED 2026-07-21 (Phase 0a).** `Cluster:SplitBrainResolverStrategy` now selects the downing provider and defaults to `auto-down` — Akka's `AutoDowning` with a 15s window — so the leader among the reachable members downs the unreachable peer and a crash of either node fails over. `keep-oldest` remains selectable for deployments that prefer partition-safety. Pinned by `SplitBrainResolverActivationTests`, which asserts the provider read back off a running ActorSystem rather than the configured one. See `docs/Redundancy.md`. **Its live gate is still outstanding**, and cannot run on today's rig: the failure only appears in a 1-vs-1 split, and `docker-dev` is a single six-node mesh. The crash-the-oldest drill therefore lands with Phase 6/7, which makes every mesh exactly two nodes. ### 6.3 DECIDED — match ScadaBridge's auth posture for now Inter-cluster transports stay unauthenticated and unencrypted, as theirs are: the boundary assumes a trusted network. Recorded as an accepted risk rather than an oversight, with two notes for whenever it is revisited: `LocalDbSyncAuthInterceptor` is already in our tree as a fail-closed template, and the surface most worth gating first is any Pull-style RPC that returns historical rows. ## 7. Sequencing sketch Deliberately not a task plan — per-phase plans follow, one at a time. | Phase | Content | Independent of the split? | |---|---|---| | 0a | Downing strategy: keep-oldest cannot survive an oldest-node crash in a 2-node cluster (§6.2) | **Yes** — **DONE 2026-07-21**, live gate deferred to Phase 7 | | 0b | Oldest-Up role derivation (§4) | **Yes** — **DONE 2026-07-21** | | 1 | `ClusterNode` gains Akka + gRPC address columns; coordinator sources its expected-ack set from the DB | Yes | | 2 | Comm actors + receptionist registration; ClusterClient transport; deploy notify + acks across the boundary | No | | 3 | Config fetch-and-cache: artifact served by central, driver nodes read LocalDb (§6.1) | No | | 4 | Cut the driver-side ConfigDb connection: re-home `EfAlarmConditionStateStore`, resolve the `DbHealthProbeActor` ServiceLevel input, audit `OpcUaPublishActor` | No | | 5 | gRPC stream contract; migrate the seven observability topics | No | | 6 | Mesh partition: per-cluster seed nodes, cluster-scoped roles + singletons; rig models the real topology | No | | 7 | Failover drill (both directions, per §6.2); live gate | No | Phases 3 and 4 are the ones that change a running system's data path rather than its wiring, and each deserves its own live gate. ## 8. Risks - **LocalDb becomes load-bearing for configuration, not just resilient.** Phase 1 built it as a fallback; §6.1 makes it the driver node's only config store. A cache defect that was previously a degraded-mode bug becomes a total-availability bug — the #485 unreadable-artifact class, with the blast radius raised. - **`DbHealthProbeActor` feeds a client-visible value.** Removing the driver-side DB connection changes what ServiceLevel means on those nodes; that is an interop-visible change, not internal. - **The rig currently models the topology that would be abolished.** Phase 6 rewrites `docker-dev/docker-compose.yml` substantially, and every live gate that depends on it. - **Losing gossip loses free fleet-wide observability.** Seven AdminUI panels are fed by DPS today; each needs an explicit stream and a reconnect story. - **Akka frame size.** We are clean on the deploy path, but anything new that carries payload over ClusterClient inherits the 128 KB default with `log-frame-size-exceeding` off — a silent single-message drop that leaves the association healthy. If we send payload at all, set both. - **Do not stack application-level last-write-wins on LocalDb's HLC.** Their `SiteStorageService` carries this warning in code. Verified: our `deployment_pointer` upsert is unconditional and the alarm sink is idempotent on a payload hash, so we are currently clean — keep it that way.