lmxopcua/docs/plans/2026-07-21-per-cluster-mesh-design.md

# Per-Cluster Akka Mesh — design

> **Status:** design for review. **Phases 0a and 0b are DONE and merged** (2026-07-21); Phases 1–7
> are not started and each needs its own plan.
> **Decisions settled 2026-07-21** — see §6: one central DB with driver nodes disconnected from it,
> the two-node keep-oldest gap closed upstream and now closed here too (§6.2), and their auth posture
> matched for now.

## 1. What this changes

Today OtOpcUa runs **one Akka mesh containing every node of every application Cluster**. That was a
deliberate choice — `docs/plans/2026-06-07-per-cluster-scoping.md`, "Per-ClusterId Scoping
(hub-and-spoke **single mesh**)" — so the deploy channel could stay in-mesh: AdminUI →
`admin-operations` singleton → `deployments` topic → every node filters to its own `ClusterId`.

The proposal is to move to the sister project's shape: **one Akka mesh per application Cluster,
two nodes maximum**, with central and clusters joined by explicit transports instead of cluster
gossip.

The motivation is that redundancy becomes correct *by construction*. Every Primary-gated decision
(inbound device writes, native alarm acks, fleet alerts, ServiceLevel, and — since Phase 2 — the
alarm-history drain) currently asks "am I the driver Primary?" of a **cluster-wide** election, while
every resource being gated is **pair-local**. Splitting the mesh collapses the two scopes into one
and the question stops being ambiguous.

## 2. What ScadaBridge actually does

Researched directly from `/Users/dohertj2/Desktop/ScadaBridge`. Five things differ from the
summary in its own CLAUDE.md, and they matter:

**Three transports cross the boundary, not two.**

| Transport | Direction | Carries |
|---|---|---|
| Akka **ClusterClient** | central → site (+ replies) | command/control: deploy notify, lifecycle, queries, subscribe handshakes |
| **gRPC** server-streaming | **central dials into the site** | real-time attribute + alarm events |
| Plain **HTTP**, token-gated | site → central | the deployment config itself (notify-and-fetch) |

**The gRPC direction is inverted from the obvious guess.** Data flows site→central, but *each site
node hosts the gRPC server* (Kestrel h2c :8083) and *central is the client*. There is no gRPC server
on central at all.

**Active node = OLDEST Up member, explicitly not the cluster leader.**
`ActiveNodeEvaluator.SelfIsOldestUp` is documented as *"THE single definition of 'active node'"*:

> *"Cluster LEADERSHIP (lowest address) is an Akka-internal concept that diverges from singleton
> placement permanently once the original first node restarts and rejoins; every product-level
> active/standby decision must use this evaluator, never `cluster.State.Leader`."*

The equivalence **oldest-Up == where `ClusterSingletonManager` places singletons** *is* the design.

**All clusters share one ActorSystem name** (`"scadabridge"`, hardcoded). They are separate clusters
only by seed-node partitioning — required, because Akka.Remote address matching means ClusterClient
could not reach a differently-named system.

**Site nodes carry two roles:** `"Site"` and `"site-{SiteId}"`. Singletons scope to the
**site-specific** role.

Other load-bearing details:

- **Central discovers sites from the database** (a `Site` entity with `NodeAAddress`/`NodeBAddress`
  + `GrpcNodeAAddress`/`GrpcNodeBAddress`), refreshed every 60 s and on admin change. Sites discover
  central from **appsettings** — static, restart required. The asymmetry is deliberate.
- **Exactly one actor is exposed per side** via `ClusterClientReceptionist.RegisterService` —
  `/user/central-communication` and `/user/site-communication` — registered **per node, not as a
  singleton**, so contact rotation reaches whichever node answers.
- **No central buffering when a site is unreachable.** The send is dropped with a warning and the
  caller's Ask times out. *"It keeps the central coordinator stateless with respect to site
  availability."* A `ConnectionStateChanged` mechanism was built for this and **deleted as dead
  code**.
- **Every unhandled message replies with a typed failure rather than dropping**, so a central Ask
  can never stall on a missing handler.
- **Sender preservation is the whole Ask idiom**: `_client.Tell(new ClusterClient.Send(path, msg),
  Sender)` routes the reply straight back to the waiting Ask, bypassing the comm actor.

### The registered outage gap — read before copying

> `Component-ClusterInfrastructure.md:113`: *"Only the FIRST seed listed in `Cluster:SeedNodes` may
> self-join to form a *new* cluster… a lone restarted non-first-seed node (with the first seed still
> down) loops on `InitJoin` forever — never `Up`, never routable. This is why the two-node
> keep-oldest **oldest/active-node crash is a total-outage gap**: after the oldest dies the younger
> survivor self-downs, and it cannot re-bootstrap alone. Recovery is operator-driven."*

Their failover drill has a mode that exists *"to make the registered gap observable — not to pretend
it is covered."* **A two-node keep-oldest mesh has an acknowledged total-outage hole.** If OtOpcUa
adopts 2-node meshes, this must be decided deliberately, not inherited.

> **CLOSED 2026-07-22.** The `InitJoin` half of this gap is fixed in both repos by **self-first seed
> ordering**: every node that is one of its own seeds lists ITSELF as `seed-nodes[0]`, which is the
> only ordering under which Akka runs `FirstSeedNodeProcess` and can form a cluster with no peer
> answering. (An earlier `Cluster:SelfFormAfter` watchdog was tried here and retired — it raced the
> join handshake.) See `docs/Redundancy.md` §"Bootstrap: self-first seed ordering". Today's
> driver-only site nodes are seeded by `central-1` only and are exempt; the per-cluster mesh removes
> even that exception by making every pair node a seed of its own mesh.

### Both inter-cluster transports are unauthenticated

Akka remoting has no TLS, no secure cookie, no `trusted-selection-paths`. The gRPC listener is h2c
with no auth covering `SiteStreamService` — including two Pull RPCs that return audit rows. The only
authenticated pieces are the HTTP fetch token and `LocalDbSyncAuthInterceptor`, which gates *only*
`/localdb_sync.v1.LocalDbSync/`. Their boundary assumes a trusted network.

**OtOpcUa already ships that same fail-closed interceptor** (LocalDb Phase 1) — it is a ready-made
template if we choose to authenticate more than they did.

## 3. What OtOpcUa has today

**Nine cross-node DPS topics plus a singleton**, all currently relying on one mesh:

| Channel | Direction | Purpose |
|---|---|---|
| `deployments` / `deployment-acks` | central ↔ nodes | deploy dispatch + acks |
| `driver-control` | central → nodes | AdminUI Reconnect/Restart |
| `redundancy-state` | central → nodes | roles + peer probe results |
| `alerts` | nodes → central | fleet alarms → `/alerts` |
| `driver-health`, `driver-resilience-status`, `fleet-status`, `script-logs` | nodes → central | AdminUI live panels |
| `admin-operations` singleton | central | operator ack/shelve into engines |

Three facts make the split cheaper than that table suggests:

1. **The deploy path is already notify-and-fetch.** `DispatchDeployment` carries only
   `DeploymentId` + `RevisionHash` + `CorrelationId`; the node fetches the artifact itself. We
   independently arrived at the pattern ScadaBridge had to retrofit, and we do **not** carry its
   128 KB Akka frame exposure on this path.
2. **Deploy state already has a DB substrate.** `ConfigPublishCoordinator` persists per-node ACKs to
   `NodeDeploymentState` so a singleton failover recovers in-flight state from the DB.
3. **`ClusterNode` rows already enumerate every node per application Cluster** — the same table
   `AdminOperationsActor` groups by cluster. This replaces the coordinator's one genuinely
   mesh-bound dependency: it derives its expected-ack set from `Akka.Cluster.State.Members` filtered
   by role.

So the deploy channel needs only a small notify + ack transport. **The nine live-telemetry channels
are the real work**, and they are exactly what ScadaBridge built ClusterClient + gRPC for.

## 4. A defect to fix regardless of this design

OtOpcUa currently uses **two different node-selection rules at once**:

- Split-brain resolution is **keep-oldest** (`Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:49`).
- `ClusterSingletonManager` places singletons on the **oldest** member.
- But `RedundancyStateActor.BuildSnapshot` elects Primary from **`RoleLeader("driver")`** — lowest
  address among driver members.

Those select the same node only by coincidence. After a restart-and-rejoin the role leader and the
oldest member diverge permanently, which means the node the SBR protects and hosts every singleton
on need not be the node the data-plane gates consider Primary. This is precisely the rule ScadaBridge
forbids by name, and for the reason it gives: *"both sides claim leadership during a partition"* —
the dual-primary shape archreview 03/S4 exists to prevent.

**Switching the role derivation to oldest-Up-with-role is small, is correct under either topology,
and should not wait for this design.** It also happens to be exactly what the separate-mesh model
needs, so it is not throwaway work.

**FIXED 2026-07-21 (Phase 0b).** `RedundancyStateActor.BuildSnapshot` selects the oldest Up member
carrying the `driver` role, matching singleton placement; `NodeRedundancyState.IsRoleLeaderForDriver`
was renamed `IsDriverPrimary` so the contract stops asserting the old derivation. Pinned by
`RedundancyPrimaryElectionTests`, which builds a real two-node cluster in which the oldest node holds
the *higher* address, so the two orderings genuinely disagree.

## 5. Target architecture

Mirror ScadaBridge's shape, adapted to OtOpcUa's existing substrate:

- **One Akka mesh per application `Cluster`, two nodes max.** Same ActorSystem name (`otopcua`)
  everywhere; separation by seed-node partitioning only.
- **Roles per node:** `driver` plus a cluster-specific `cluster-{ClusterId}`, with singletons scoped
  to the cluster-specific role.
- **Active node = oldest Up member with role** (§4), one definition, used by the data-plane gates,
  ServiceLevel, and the alarm drain alike.
- **Central ↔ cluster command/control over ClusterClient**, with exactly one receptionist-registered
  actor per side and central discovering cluster node addresses from `ClusterNode` rows (extended
  with Akka + gRPC addresses, mirroring their `Site` entity).
- **Deploy stays notify-and-fetch, but the fetch changes source.** The notify crosses via
  ClusterClient; the artifact then comes **from central**, not from the ConfigDb the driver node no
  longer connects to (§6.1), and is cached in LocalDb. The coordinator's expected-ack set comes from
  `ClusterNode` rows instead of cluster membership.
- **Driver nodes hold no ConfigDb connection.** LocalDb becomes their steady-state configuration
  store; five current DB consumers are re-homed or re-sourced per §6.1.
- **Live telemetry over gRPC**, central dialling into each cluster node, replacing the seven
  observability topics with one stream contract carrying a `oneof` event — additive-only field
  evolution, contract-locked by test.

**Copied:** their node-local configuration store fed by fetch-and-cache (§6.1), their oldest-Up
active-node rule (§4), their two-node keep-oldest posture including its acknowledged outage gap
(§6.2), and their unauthenticated inter-cluster transports (§6.3).

**Deliberately not copied:** their *transient-only* central alarm cache. OtOpcUa persists alarm
history centrally through the historian, and Phase 2's replicated store-and-forward buffer already
guarantees delivery across a failover — so there is no reason to adopt a design whose stated
trade-off is that a new active node re-seeds alarm state from scratch.

## 6. Decisions (settled 2026-07-21)

### 6.1 DECIDED — one central DB, and driver nodes never connect to it

**Verified in ScadaBridge before adopting.** `Program.cs` calls
`AddConfigurationDatabase(configDbConnectionString)` at line 264 — inside the Central branch
(89–478). The Site branch begins at line 479. **Site nodes never register the central configuration
database at all**; they receive config from central (notify → token-gated HTTP fetch) and persist it
locally. There is still exactly one authoritative configuration database — it is simply
central-only.

OtOpcUa adopts the same shape: **the ConfigDb stays the single source of truth, and driver nodes stop
connecting to it.** They obtain their configuration from the central nodes and cache it locally in
LocalDb.

**This promotes LocalDb from an outage cache to the driver node's steady-state configuration store.**
That is a reframing of the Phase 1 work rather than a contradiction of it: boot-from-cache stops
being the exceptional path and becomes the normal one, and "central SQL unreachable" stops being a
degraded mode for driver nodes because they never held a connection to lose. The chunking,
SHA-256 verification, newest-2 retention and pair replication all carry over unchanged.

**Scope — five driver-side ConfigDb consumers must be re-homed or re-sourced.** Audited:

| Consumer | Current use | Disposition |
|---|---|---|
| `DriverHostActor` | artifact fetch + ack writes (5 `CreateDbContext` sites) | artifact via fetch-and-cache; acks via the ack transport |
| `EfAlarmConditionStateStore` | Part 9 alarm condition state | **move to LocalDb** — it is pair-local state, the same journey Phase 2 made for the alarm S&F buffer |
| `DbHealthProbeActor` | DB reachability feeding ServiceLevel tiering | **semantics change** — a driver node has no DB to probe; decide what health input replaces it |
| `OpcUaPublishActor` | (audit required) | TBD |
| `ServiceCollectionExtensions` | registration | follows the above |

`DbHealthProbeActor` deserves particular care: DB health is currently a ServiceLevel input, and
under this model it stops existing for driver nodes. That is a behaviour change on a client-visible
value, not just an internal refactor.

### 6.2 CORRECTED — the gap was closed in ScadaBridge, and OtOpcUa still has it

**An earlier revision of this section was wrong.** It reported the two-node keep-oldest outage gap as
still open, citing `Component-ClusterInfrastructure.md:113` and Gitea PR
[#12](https://gitea.dohertylan.com/dohertj2/ScadaBridge/pulls/12) (which made the failover drill
*honest* about the gap rather than closing it). That doc line was stale: ScadaBridge closed the gap
in `cf3bd52f`, *"feat(cluster): auto-down downing strategy — either-node crash now fails over (owner
decision 2026-07-21: availability over partition-safety)"*.

Its commit message carries the analysis, live-proven on their rig:

> *"Two-node keep-oldest could NEVER survive a crash of the oldest/active node: Akka.NET 1.5.62
> `KeepOldest.OldestDecision` only lets `down-if-alone` rescue a side with >= 2 members, so the 1-vs-1
> survivor takes `DownReachable` and downs ITSELF — proven live on the rig ('SBR took decision …
> including myself') before this change. `static-quorum(1)` is worse (`IsTooManyMembers` -> `DownAll`);
> `keep-majority` just re-keys the fatal crash to the lowest address."*

**OtOpcUa is running that exact configuration today** —
`Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:49-53`: `active-strategy = "keep-oldest"`,
`keep-oldest { down-if-alone = on }`, `stable-after = 15s`, with
`BuildClusterOptions` setting the matching typed `KeepOldestOption { DownIfAlone = true }`.

So a two-node OtOpcUa pair **could not fail over when the oldest node crashed**: the survivor
downed itself. That was a total-outage defect in the redundancy feature itself, independent of this
design.

**FIXED 2026-07-21 (Phase 0a).** `Cluster:SplitBrainResolverStrategy` now selects the downing
provider and defaults to `auto-down` — Akka's `AutoDowning` with a 15s window — so the leader among
the reachable members downs the unreachable peer and a crash of either node fails over. `keep-oldest`
remains selectable for deployments that prefer partition-safety. Pinned by
`SplitBrainResolverActivationTests`, which asserts the provider read back off a running ActorSystem
rather than the configured one. See `docs/Redundancy.md`.

**Its live gate is still outstanding**, and cannot run on today's rig: the failure only appears in a
1-vs-1 split, and `docker-dev` is a single six-node mesh. The crash-the-oldest drill therefore lands
with Phase 6/7, which makes every mesh exactly two nodes.

**The bootstrap half of the gap is also closed (2026-07-22).** Downing was only ever one of the two
outage modes: even with `auto-down`, a node cold-starting while its peer was dead looped on `InitJoin`
forever, because Akka lets only `seed-nodes[0]` form a new cluster. The fix is **self-first seed
ordering** — each seed node lists itself first, enforced at boot by `AkkaClusterOptionsValidator`.
The `Cluster:SelfFormAfter` watchdog first shipped for this is deleted: a timer outside Akka's join
handshake cannot tell "no seed answered" from "a seed answered and the join is in flight", and it
islanded a manually-failed-over node live. Pinned by `SelfFirstSeedBootstrapTests`; see
`docs/Redundancy.md` §"Bootstrap: self-first seed ordering". Under this design each pair node is a
seed of its own two-node mesh, so the ordering covers both sides and the current site-node exemption
disappears.

### 6.3 DECIDED — match ScadaBridge's auth posture for now

Inter-cluster transports stay unauthenticated and unencrypted, as theirs are: the boundary assumes a
trusted network. Recorded as an accepted risk rather than an oversight, with two notes for whenever
it is revisited: `LocalDbSyncAuthInterceptor` is already in our tree as a fail-closed template, and
the surface most worth gating first is any Pull-style RPC that returns historical rows.

## 7. Sequencing sketch

Deliberately not a task plan — per-phase plans follow, one at a time.

| Phase | Content | Independent of the split? |
|---|---|---|
| 0a | Downing strategy: keep-oldest cannot survive an oldest-node crash in a 2-node cluster (§6.2) | **Yes** — **DONE 2026-07-21**, live gate deferred to Phase 7 |
| 0b | Oldest-Up role derivation (§4) | **Yes** — **DONE 2026-07-21** |
| 1 | `ClusterNode` gains Akka + gRPC address columns; coordinator sources its expected-ack set from the DB | **Yes** — **DONE 2026-07-22** (see below) |
| 2 | Comm actors + receptionist registration; ClusterClient transport; deploy notify + acks across the boundary | No |
| 3 | Config fetch-and-cache: artifact served by central, driver nodes read LocalDb (§6.1) | No |
| 4 | Cut the driver-side ConfigDb connection: re-home `EfAlarmConditionStateStore`, resolve the `DbHealthProbeActor` ServiceLevel input, audit `OpcUaPublishActor` | No |
| 5 | gRPC stream contract; migrate the seven observability topics | No |
| 6 | Mesh partition: per-cluster seed nodes, cluster-scoped roles + singletons; rig models the real topology | No |
| 7 | Failover drill (both directions, per §6.2); live gate | No |

Phases 3 and 4 are the ones that change a running system's data path rather than its wiring, and each
deserves its own live gate.

### Phase 1 as shipped (2026-07-22)

Plan: `2026-07-21-per-cluster-mesh-phase1.md`. Two deviations from the sketch above, both decided
before implementing:

- **No cluster-scope filtering of the expected-ack set.** The plan called for it, but there is no
  cluster-scoped deployment to filter on: `Deployment` has no `ClusterId`,
  `ConfigComposer.SnapshotAndFlattenAsync` always snapshots the whole DB, and
  `DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The
  expected set is every enabled `ClusterNode` row, which is what the membership rule produced too.
- **No per-node role column.** The membership rule filtered on the `driver` role and the DB has none.
  Rather than add one — a second declaration of node roles, free to drift from `Cluster:Roles` — every
  `ClusterNode` row is now *defined* to be a driver node. An admin-only node must not be given one.

Also shipped beyond the sketch: `ClusterNodeAddressReconciler` (Task 4), an admin singleton that
catches `AkkaPort` drifting from the node's own `Cluster:Port`. **Phase 2 must revisit it** — once the
meshes split, an admin node cannot see site members and every site row would report
`EnabledRowNotInCluster` forever.

## 8. Risks

- **LocalDb becomes load-bearing for configuration, not just resilient.** Phase 1 built it as a
  fallback; §6.1 makes it the driver node's only config store. A cache defect that was previously a
  degraded-mode bug becomes a total-availability bug — the #485 unreadable-artifact class, with the
  blast radius raised.
- **`DbHealthProbeActor` feeds a client-visible value.** Removing the driver-side DB connection
  changes what ServiceLevel means on those nodes; that is an interop-visible change, not internal.
- **The rig currently models the topology that would be abolished.** Phase 6 rewrites
  `docker-dev/docker-compose.yml` substantially, and every live gate that depends on it.
- **Losing gossip loses free fleet-wide observability.** Seven AdminUI panels are fed by DPS today;
  each needs an explicit stream and a reconnect story.
- **Akka frame size.** We are clean on the deploy path, but anything new that carries payload over
  ClusterClient inherits the 128 KB default with `log-frame-size-exceeding` off — a silent
  single-message drop that leaves the association healthy. If we send payload at all, set both.
- **Do not stack application-level last-write-wins on LocalDb's HLC.** Their `SiteStorageService`
  carries this warning in code. Verified: our `deployment_pointer` upsert is unconditional and the
  alarm sink is idempotent on a payload hash, so we are currently clean — keep it that way.