lmxopcua/docs/plans/2026-07-21-per-cluster-mesh-design.md

# Per-Cluster Akka Mesh — design

> **Status:** design for review. No implementation planned yet.
> **Decisions settled 2026-07-21** — see §6: one central DB with driver nodes disconnected from it,
> the two-node keep-oldest gap accepted as ScadaBridge accepts it, and their auth posture matched for
> now.

## 1. What this changes

Today OtOpcUa runs **one Akka mesh containing every node of every application Cluster**. That was a
deliberate choice — `docs/plans/2026-06-07-per-cluster-scoping.md`, "Per-ClusterId Scoping
(hub-and-spoke **single mesh**)" — so the deploy channel could stay in-mesh: AdminUI →
`admin-operations` singleton → `deployments` topic → every node filters to its own `ClusterId`.

The proposal is to move to the sister project's shape: **one Akka mesh per application Cluster,
two nodes maximum**, with central and clusters joined by explicit transports instead of cluster
gossip.

The motivation is that redundancy becomes correct *by construction*. Every Primary-gated decision
(inbound device writes, native alarm acks, fleet alerts, ServiceLevel, and — since Phase 2 — the
alarm-history drain) currently asks "am I the driver Primary?" of a **cluster-wide** election, while
every resource being gated is **pair-local**. Splitting the mesh collapses the two scopes into one
and the question stops being ambiguous.

## 2. What ScadaBridge actually does

Researched directly from `/Users/dohertj2/Desktop/ScadaBridge`. Five things differ from the
summary in its own CLAUDE.md, and they matter:

**Three transports cross the boundary, not two.**

| Transport | Direction | Carries |
|---|---|---|
| Akka **ClusterClient** | central → site (+ replies) | command/control: deploy notify, lifecycle, queries, subscribe handshakes |
| **gRPC** server-streaming | **central dials into the site** | real-time attribute + alarm events |
| Plain **HTTP**, token-gated | site → central | the deployment config itself (notify-and-fetch) |

**The gRPC direction is inverted from the obvious guess.** Data flows site→central, but *each site
node hosts the gRPC server* (Kestrel h2c :8083) and *central is the client*. There is no gRPC server
on central at all.

**Active node = OLDEST Up member, explicitly not the cluster leader.**
`ActiveNodeEvaluator.SelfIsOldestUp` is documented as *"THE single definition of 'active node'"*:

> *"Cluster LEADERSHIP (lowest address) is an Akka-internal concept that diverges from singleton
> placement permanently once the original first node restarts and rejoins; every product-level
> active/standby decision must use this evaluator, never `cluster.State.Leader`."*

The equivalence **oldest-Up == where `ClusterSingletonManager` places singletons** *is* the design.

**All clusters share one ActorSystem name** (`"scadabridge"`, hardcoded). They are separate clusters
only by seed-node partitioning — required, because Akka.Remote address matching means ClusterClient
could not reach a differently-named system.

**Site nodes carry two roles:** `"Site"` and `"site-{SiteId}"`. Singletons scope to the
**site-specific** role.

Other load-bearing details:

- **Central discovers sites from the database** (a `Site` entity with `NodeAAddress`/`NodeBAddress`
  + `GrpcNodeAAddress`/`GrpcNodeBAddress`), refreshed every 60 s and on admin change. Sites discover
  central from **appsettings** — static, restart required. The asymmetry is deliberate.
- **Exactly one actor is exposed per side** via `ClusterClientReceptionist.RegisterService` —
  `/user/central-communication` and `/user/site-communication` — registered **per node, not as a
  singleton**, so contact rotation reaches whichever node answers.
- **No central buffering when a site is unreachable.** The send is dropped with a warning and the
  caller's Ask times out. *"It keeps the central coordinator stateless with respect to site
  availability."* A `ConnectionStateChanged` mechanism was built for this and **deleted as dead
  code**.
- **Every unhandled message replies with a typed failure rather than dropping**, so a central Ask
  can never stall on a missing handler.
- **Sender preservation is the whole Ask idiom**: `_client.Tell(new ClusterClient.Send(path, msg),
  Sender)` routes the reply straight back to the waiting Ask, bypassing the comm actor.

### The registered outage gap — read before copying

> `Component-ClusterInfrastructure.md:113`: *"Only the FIRST seed listed in `Cluster:SeedNodes` may
> self-join to form a *new* cluster… a lone restarted non-first-seed node (with the first seed still
> down) loops on `InitJoin` forever — never `Up`, never routable. This is why the two-node
> keep-oldest **oldest/active-node crash is a total-outage gap**: after the oldest dies the younger
> survivor self-downs, and it cannot re-bootstrap alone. Recovery is operator-driven."*

Their failover drill has a mode that exists *"to make the registered gap observable — not to pretend
it is covered."* **A two-node keep-oldest mesh has an acknowledged total-outage hole.** If OtOpcUa
adopts 2-node meshes, this must be decided deliberately, not inherited.

### Both inter-cluster transports are unauthenticated

Akka remoting has no TLS, no secure cookie, no `trusted-selection-paths`. The gRPC listener is h2c
with no auth covering `SiteStreamService` — including two Pull RPCs that return audit rows. The only
authenticated pieces are the HTTP fetch token and `LocalDbSyncAuthInterceptor`, which gates *only*
`/localdb_sync.v1.LocalDbSync/`. Their boundary assumes a trusted network.

**OtOpcUa already ships that same fail-closed interceptor** (LocalDb Phase 1) — it is a ready-made
template if we choose to authenticate more than they did.

## 3. What OtOpcUa has today

**Nine cross-node DPS topics plus a singleton**, all currently relying on one mesh:

| Channel | Direction | Purpose |
|---|---|---|
| `deployments` / `deployment-acks` | central ↔ nodes | deploy dispatch + acks |
| `driver-control` | central → nodes | AdminUI Reconnect/Restart |
| `redundancy-state` | central → nodes | roles + peer probe results |
| `alerts` | nodes → central | fleet alarms → `/alerts` |
| `driver-health`, `driver-resilience-status`, `fleet-status`, `script-logs` | nodes → central | AdminUI live panels |
| `admin-operations` singleton | central | operator ack/shelve into engines |

Three facts make the split cheaper than that table suggests:

1. **The deploy path is already notify-and-fetch.** `DispatchDeployment` carries only
   `DeploymentId` + `RevisionHash` + `CorrelationId`; the node fetches the artifact itself. We
   independently arrived at the pattern ScadaBridge had to retrofit, and we do **not** carry its
   128 KB Akka frame exposure on this path.
2. **Deploy state already has a DB substrate.** `ConfigPublishCoordinator` persists per-node ACKs to
   `NodeDeploymentState` so a singleton failover recovers in-flight state from the DB.
3. **`ClusterNode` rows already enumerate every node per application Cluster** — the same table
   `AdminOperationsActor` groups by cluster. This replaces the coordinator's one genuinely
   mesh-bound dependency: it derives its expected-ack set from `Akka.Cluster.State.Members` filtered
   by role.

So the deploy channel needs only a small notify + ack transport. **The nine live-telemetry channels
are the real work**, and they are exactly what ScadaBridge built ClusterClient + gRPC for.

## 4. A defect to fix regardless of this design

OtOpcUa currently uses **two different node-selection rules at once**:

- Split-brain resolution is **keep-oldest** (`Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:49`).
- `ClusterSingletonManager` places singletons on the **oldest** member.
- But `RedundancyStateActor.BuildSnapshot` elects Primary from **`RoleLeader("driver")`** — lowest
  address among driver members.

Those select the same node only by coincidence. After a restart-and-rejoin the role leader and the
oldest member diverge permanently, which means the node the SBR protects and hosts every singleton
on need not be the node the data-plane gates consider Primary. This is precisely the rule ScadaBridge
forbids by name, and for the reason it gives: *"both sides claim leadership during a partition"* —
the dual-primary shape archreview 03/S4 exists to prevent.

**Switching the role derivation to oldest-Up-with-role is small, is correct under either topology,
and should not wait for this design.** It also happens to be exactly what the separate-mesh model
needs, so it is not throwaway work.

## 5. Target architecture

Mirror ScadaBridge's shape, adapted to OtOpcUa's existing substrate:

- **One Akka mesh per application `Cluster`, two nodes max.** Same ActorSystem name (`otopcua`)
  everywhere; separation by seed-node partitioning only.
- **Roles per node:** `driver` plus a cluster-specific `cluster-{ClusterId}`, with singletons scoped
  to the cluster-specific role.
- **Active node = oldest Up member with role** (§4), one definition, used by the data-plane gates,
  ServiceLevel, and the alarm drain alike.
- **Central ↔ cluster command/control over ClusterClient**, with exactly one receptionist-registered
  actor per side and central discovering cluster node addresses from `ClusterNode` rows (extended
  with Akka + gRPC addresses, mirroring their `Site` entity).
- **Deploy stays notify-and-fetch, but the fetch changes source.** The notify crosses via
  ClusterClient; the artifact then comes **from central**, not from the ConfigDb the driver node no
  longer connects to (§6.1), and is cached in LocalDb. The coordinator's expected-ack set comes from
  `ClusterNode` rows instead of cluster membership.
- **Driver nodes hold no ConfigDb connection.** LocalDb becomes their steady-state configuration
  store; five current DB consumers are re-homed or re-sourced per §6.1.
- **Live telemetry over gRPC**, central dialling into each cluster node, replacing the seven
  observability topics with one stream contract carrying a `oneof` event — additive-only field
  evolution, contract-locked by test.

**Copied:** their node-local configuration store fed by fetch-and-cache (§6.1), their oldest-Up
active-node rule (§4), their two-node keep-oldest posture including its acknowledged outage gap
(§6.2), and their unauthenticated inter-cluster transports (§6.3).

**Deliberately not copied:** their *transient-only* central alarm cache. OtOpcUa persists alarm
history centrally through the historian, and Phase 2's replicated store-and-forward buffer already
guarantees delivery across a failover — so there is no reason to adopt a design whose stated
trade-off is that a new active node re-seeds alarm state from scratch.

## 6. Decisions (settled 2026-07-21)

### 6.1 DECIDED — one central DB, and driver nodes never connect to it

**Verified in ScadaBridge before adopting.** `Program.cs` calls
`AddConfigurationDatabase(configDbConnectionString)` at line 264 — inside the Central branch
(89–478). The Site branch begins at line 479. **Site nodes never register the central configuration
database at all**; they receive config from central (notify → token-gated HTTP fetch) and persist it
locally. There is still exactly one authoritative configuration database — it is simply
central-only.

OtOpcUa adopts the same shape: **the ConfigDb stays the single source of truth, and driver nodes stop
connecting to it.** They obtain their configuration from the central nodes and cache it locally in
LocalDb.

**This promotes LocalDb from an outage cache to the driver node's steady-state configuration store.**
That is a reframing of the Phase 1 work rather than a contradiction of it: boot-from-cache stops
being the exceptional path and becomes the normal one, and "central SQL unreachable" stops being a
degraded mode for driver nodes because they never held a connection to lose. The chunking,
SHA-256 verification, newest-2 retention and pair replication all carry over unchanged.

**Scope — five driver-side ConfigDb consumers must be re-homed or re-sourced.** Audited:

| Consumer | Current use | Disposition |
|---|---|---|
| `DriverHostActor` | artifact fetch + ack writes (5 `CreateDbContext` sites) | artifact via fetch-and-cache; acks via the ack transport |
| `EfAlarmConditionStateStore` | Part 9 alarm condition state | **move to LocalDb** — it is pair-local state, the same journey Phase 2 made for the alarm S&F buffer |
| `DbHealthProbeActor` | DB reachability feeding ServiceLevel tiering | **semantics change** — a driver node has no DB to probe; decide what health input replaces it |
| `OpcUaPublishActor` | (audit required) | TBD |
| `ServiceCollectionExtensions` | registration | follows the above |

`DbHealthProbeActor` deserves particular care: DB health is currently a ServiceLevel input, and
under this model it stops existing for driver nodes. That is a behaviour change on a client-visible
value, not just an internal refactor.

### 6.2 DECIDED — accept the two-node keep-oldest gap, as ScadaBridge does

**Status corrected.** What was resolved is the *documentation and the drill*, not the gap. Gitea
**PR #12** (closed) — <https://gitea.dohertylan.com/dohertj2/ScadaBridge/pulls/12> — "R2-01: Cluster,
Host & Failover round-2 fixes" carried:

- **T1** (`badc97af`) — the failover drill now kills the STANDBY by default, and its `active` mode
  *"measures the registered keep-oldest outage instead of pretending recovery"*.
- **T2** (`d5364506`) — per-direction failover docs, dropping what the commit message calls *"the
  undeliverable ~25s active-crash promise"*.

So the closed work made the gap **honest**, not absent. `Component-ClusterInfrastructure.md:113`
still reads: *"Removing the gap itself is the **registered deferred keep-oldest topology/strategy
decision** (master tracker 2026-07-08, owner: user)."* There is no open Gitea issue for it — it is
tracked on that master tracker, not in the issue list.

OtOpcUa inherits the same gap and, for now, the same posture. Worth recording so it is a known
accepted risk rather than a surprise: after the oldest node of a two-node mesh dies, the survivor
self-downs and cannot re-bootstrap alone, because only the first-listed seed may self-join. Recovery
is operator-driven — restart the dead first seed, or restart the survivor with a self-first seed
override.

### 6.3 DECIDED — match ScadaBridge's auth posture for now

Inter-cluster transports stay unauthenticated and unencrypted, as theirs are: the boundary assumes a
trusted network. Recorded as an accepted risk rather than an oversight, with two notes for whenever
it is revisited: `LocalDbSyncAuthInterceptor` is already in our tree as a fail-closed template, and
the surface most worth gating first is any Pull-style RPC that returns historical rows.

## 7. Sequencing sketch

Deliberately not a task plan — per-phase plans follow, one at a time.

| Phase | Content | Independent of the split? |
|---|---|---|
| 0 | Oldest-Up role derivation (§4) | **Yes** — ship first |
| 1 | `ClusterNode` gains Akka + gRPC address columns; coordinator sources its expected-ack set from the DB | Yes |
| 2 | Comm actors + receptionist registration; ClusterClient transport; deploy notify + acks across the boundary | No |
| 3 | Config fetch-and-cache: artifact served by central, driver nodes read LocalDb (§6.1) | No |
| 4 | Cut the driver-side ConfigDb connection: re-home `EfAlarmConditionStateStore`, resolve the `DbHealthProbeActor` ServiceLevel input, audit `OpcUaPublishActor` | No |
| 5 | gRPC stream contract; migrate the seven observability topics | No |
| 6 | Mesh partition: per-cluster seed nodes, cluster-scoped roles + singletons; rig models the real topology | No |
| 7 | Failover drill (both directions, per §6.2); live gate | No |

Phases 3 and 4 are the ones that change a running system's data path rather than its wiring, and each
deserves its own live gate.

## 8. Risks

- **LocalDb becomes load-bearing for configuration, not just resilient.** Phase 1 built it as a
  fallback; §6.1 makes it the driver node's only config store. A cache defect that was previously a
  degraded-mode bug becomes a total-availability bug — the #485 unreadable-artifact class, with the
  blast radius raised.
- **`DbHealthProbeActor` feeds a client-visible value.** Removing the driver-side DB connection
  changes what ServiceLevel means on those nodes; that is an interop-visible change, not internal.
- **The rig currently models the topology that would be abolished.** Phase 6 rewrites
  `docker-dev/docker-compose.yml` substantially, and every live gate that depends on it.
- **Losing gossip loses free fleet-wide observability.** Seven AdminUI panels are fed by DPS today;
  each needs an explicit stream and a reconnect story.
- **Akka frame size.** We are clean on the deploy path, but anything new that carries payload over
  ClusterClient inherits the 128 KB default with `log-frame-size-exceeding` off — a silent
  single-message drop that leaves the association healthy. If we send payload at all, set both.
- **Do not stack application-level last-write-wins on LocalDb's HLC.** Their `SiteStorageService`
  carries this warning in code. Verified: our `deployment_pointer` upsert is unconditional and the
  alarm sink is idempotent on a payload hash, so we are currently clean — keep it that way.