lmxopcua/docs/plans/2026-07-22-per-cluster-mesh-program.md

# Per-Cluster Mesh Program — Align OtOpcUa's Akka Topology with ScadaBridge

> **For Claude:** This is a PROGRAM plan (phase roadmap + gates), not a bite-sized task plan.
> The authoritative design is `docs/plans/2026-07-21-per-cluster-mesh-design.md` (decisions settled
> 2026-07-21; Phases 0a/0b DONE). Per that design's own instruction, **each phase gets its own
> detailed plan, written when the phase starts** (superpowers writing-plans → executing-plans),
> so plans are authored against current code, not against a forecast. This document sequences the
> phases, fixes the deployment topology (co-location with ScadaBridge — NEW constraint 2026-07-22),
> and defines each phase's entry/exit gates.

**Goal:** OtOpcUa runs the same Akka.NET mesh shape as ScadaBridge — **one Akka mesh per
application `Cluster`, two nodes max**, central ↔ cluster joined by explicit transports
(ClusterClient + gRPC + fetch), driver nodes holding **no ConfigDb connection** — so that every
Primary-gated decision and every gated resource share one pair-local scope, and both products
present one operational model on the shared site hardware.

**Why now (2026-07-22):** OtOpcUa cluster pairs will run **on the same two Windows VMs as the
ScadaBridge site nodes**. That deployment makes the current single-fleet-mesh design actively
wrong for sites (a site's OtOpcUa nodes would gossip across the WAN to central and elect ONE
Primary fleet-wide), and makes the ScadaBridge shape the obviously correct one: each site's two
VMs host two independent, identically-postured 2-node clusters (one per product), surviving alone
on the site LAN. It also means driver nodes must not depend on reaching central SQL — **sites
have no SQL Server**, so §6.1's "driver nodes never connect to the ConfigDb" stops being an
architectural preference and becomes a deployment requirement.

---

## Deployment topology (the co-location constraint, NEW)

Each **site**: 2 Windows VMs. Each VM runs one ScadaBridge site node AND one OtOpcUa driver node.
Two independent 2-node Akka clusters per site — they share hardware, never a mesh. **Central**:
2 Windows VMs, each running a ScadaBridge central node and an OtOpcUa central node
(`admin,driver`), again as separate pairs.

**Per-VM port allocation (no collisions — verify against actual deployment configs in Phase 6):**

| Port | Owner | Purpose |
|---|---|---|
| 8081 / 8082 | ScadaBridge | Akka remoting (central / site) |
| 8083 | ScadaBridge site | gRPC h2c (streams + LocalDb sync) |
| 8084 | ScadaBridge site | metrics |
| 5000 (+Traefik) | ScadaBridge central | UI + Inbound API |
| **4053** | OtOpcUa | Akka remoting |
| **4840** | OtOpcUa | OPC UA endpoint |
| OtOpcUa AdminUI port | OtOpcUa central (admin) | AdminUI (driver-only site nodes host no UI) |
| OtOpcUa LocalDb sync port | OtOpcUa driver | h2c LocalDb pair replication (default off/0 today — Phase 6 assigns a real per-site port) |

**Aligned HA posture (both products, per VM — already true or landing via the selfform plan):**
auto-down downing (15 s window), oldest-Up active/primary election, self-first seed ordering
(the `SelfFormAfter` self-form fallback that briefly stood in for it was retired 2026-07-22),
termination-watchdog → process exit → `sc.exe failure` restart recovery.
One failover story for operators regardless of product.

**Prerequisite ordering:** the fallback/manual-failover plan
(`docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`) executes **before** this
program — it is small, independent, and Phase 6 depends on its semantics (under per-pair meshes
every node lists itself + partner as seeds, so the fallback covers both nodes of every pair;
the site-node island guard then simply never triggers).

---

## What changes, in one table (from the design doc)

| Aspect | Today (single fleet mesh) | Target (ScadaBridge shape) |
|---|---|---|
| Mesh | one gossip ring, all nodes, seeded by central-1 | one 2-node mesh per application `Cluster`; same ActorSystem name; separation by seed-node partitioning |
| Roles | `admin` / `driver` fleet-wide | `driver` + cluster-specific `cluster-{ClusterId}`; singletons scoped to the cluster role |
| Primary election | mesh-wide oldest Up driver (Phase 0b) | pair-local by construction — same rule, correct scope |
| Command/control | 9 DPS topics + singleton over gossip | ClusterClient (one receptionist actor per side); central discovers nodes from `ClusterNode` rows |
| Deploy | notify (DPS) + node fetches from ConfigDb | notify via ClusterClient; artifact fetched **from central**, cached in LocalDb |
| Driver ConfigDb connection | direct EF connection to central SQL | **none** — LocalDb is the steady-state config store |
| Live telemetry | 7 observability DPS topics | one gRPC stream contract (`oneof` event), central dials each cluster node |
| Rig | six-node single mesh (`docker-dev`) | per-cluster meshes; rig models the real topology |

---

## Phases

Each phase below is one row of design-doc §7, expanded with entry/exit gates. **Execution recipe
per phase:** (1) invoke writing-plans in this repo to produce
`docs/plans/2026-07-2X-mesh-phase-N-<name>.md` from the scope notes here + the design doc §
references, exploring current code first; (2) execute it task-by-task; (3) run the phase's exit
gate; (4) update this file's status column and the design doc's §7 table.

### Phase 1 — `ClusterNode` address columns + DB-sourced ack set — **DONE 2026-07-22**
Shipped per `2026-07-21-per-cluster-mesh-phase1.md`; see that plan's Task 6 gate record and the
design doc's "Phase 1 as shipped" note for the two scope deviations (no cluster-scope filtering — no
such deployment exists; no per-node role column) and the one addition (`ClusterNodeAddressReconciler`).
**AdminUI node edit was deferred to Phase 2** — nothing reads the columns until then, and the
migration default plus the rig seed cover every node today.

**Scope:** `ClusterNode` gains Akka + gRPC address columns (mirroring ScadaBridge's `Site`
entity `NodeAAddress`/`GrpcNodeAAddress` pattern, but per-node rows); EF migration; AdminUI node
edit surfaces the fields; `ConfigPublishCoordinator` derives its expected-ack set from
`ClusterNode` rows instead of `Akka.Cluster.State.Members` filtered by role (design §3 fact 3 —
this removes the coordinator's one genuinely mesh-bound dependency).
**Independent of the split:** yes — safe on the current mesh.
**Exit gate:** deploy on the unchanged docker-dev rig completes with the coordinator's expected-ack
set proven DB-sourced (test: a `ClusterNode` row present but node down → deploy reports that node
missing; a node up but row absent → its ack is not expected).

### Phase 2 — Comm actors + ClusterClient transport
**Scope:** one receptionist-registered actor per side (`/user/central-communication`,
`/user/cluster-communication`), registered **per node, not as a singleton** (contact rotation);
ClusterClient central → cluster carrying deploy notify + acks + driver-control; the ScadaBridge
idioms copied verbatim: sender-preserving `Tell(new ClusterClient.Send(...), Sender)` Ask relay,
typed-failure reply for every unhandled message, no central buffering toward unreachable clusters
(drop + warn), central discovers contacts from Phase 1's `ClusterNode` rows (60 s refresh +
admin-change refresh), clusters know central from appsettings (static, restart to change).
**Frame-size guard:** anything carrying payload sets both the frame limit and
`log-frame-size-exceeding` (design §8) — the deploy path stays payload-free by design.
**Exit gate:** on the still-single-mesh rig, deploy notify + acks and AdminUI Reconnect/Restart
flow over ClusterClient (DPS paths deleted or dark-switched), including an Ask timing out cleanly
against a stopped node.

### Phase 3 — Config fetch-and-cache from central
**Scope:** the deploy artifact is served **by central** (transport per its own phase plan — the
ScadaBridge analogue is token-gated HTTP; decide HTTP vs a gRPC fetch RPC when planning) and
cached in LocalDb; driver nodes read config exclusively from LocalDb (boot-from-cache becomes the
normal path — design §6.1); chunking, SHA-256 verify, newest-2 retention, pair replication carry
over unchanged. **This phase changes a running data path — it gets its own live gate** (design §7
note): deploy lands on a driver pair with central SQL stopped mid-fetch → retry lands; #485
last-known-good semantics re-proven on the new path.
**Exit gate:** live gate green on the rig; a driver node with an empty LocalDb and reachable
central boots into the current config; with central down it boots last-known-good.

### Phase 4 — Cut the driver-side ConfigDb connection
**Scope (design §6.1 audit table):** re-home `EfAlarmConditionStateStore` to LocalDb (pair-local
state, same journey as the Phase-2 alarm S&F buffer); **resolve the `DbHealthProbeActor`
question** — driver nodes have no DB to probe, and DB health currently feeds ServiceLevel tiering,
so define the replacement health input (candidate: central-reachability via the Phase 2/3
transports) — this is a client-visible ServiceLevel semantics change and must be documented in
`docs/Redundancy.md` + the interop playbook; audit `OpcUaPublishActor`'s ConfigDb use (TBD in the
design) and re-source it; registration cleanup in `ServiceCollectionExtensions`; driver-role
`Program.cs` branch registers no EF ConfigDb context at all (mirror ScadaBridge's central-only
`AddConfigurationDatabase`).
**Exit gate:** its own live gate — a driver pair runs a full deploy + alarm + historian cycle with
**no ConfigDb connection string configured at all**; grep-level proof no driver-branch service can
resolve the ConfigDb context.

### Phase 5 — gRPC stream contract for live telemetry
**Scope:** one server-streaming contract carrying a `oneof` event, **cluster nodes host the gRPC
server, central dials in** (the inverted direction is the load-bearing ScadaBridge finding —
design §2); migrate the seven observability topics (`alerts`, `driver-health`,
`driver-resilience-status`, `fleet-status`, `script-logs`, plus redundancy-state distribution and
deployment-acks if Phase 2 left them on DPS); additive-only field evolution, contract locked by
test; per-panel reconnect story for the AdminUI (design §8 — losing gossip loses free fleet
observability).
**Exit gate:** all AdminUI live panels green against a pair with DPS telemetry topics deleted;
kill-and-reconnect of the central dialer recovers every stream.

### Phase 6 — Mesh partition + co-location topology
**Scope:** per-cluster seed nodes — ~~adopt ScadaBridge's self-first ordering and RETIRE the
`SelfFormAfter` watchdog + TCP reachability guard~~ **DONE EARLY 2026-07-22** (docker-dev
`central-2` swapped to self-first, `ClusterBootstrapFallback`/`SelfFormAfter` deleted,
`AkkaClusterOptionsValidator` ports ScadaBridge's startup rule in its conditional form, and
`SelfFirstSeedBootstrapTests` replaces `SelfFormBootstrapTests`; see `docs/Redundancy.md`
§"Bootstrap: self-first seed ordering"). What remains for this phase is the per-pair
`Cluster__SeedNodes__*` matrix once the meshes actually split — every node in a pair is then a seed
of its own mesh, so the site-node exemption disappears; cluster-scoped roles `cluster-{ClusterId}` + singleton re-scoping,
central pair keeps the admin singletons; **docker-dev rig rewritten** to model the real topology —
including the co-location port table above (both products' compose files on shared per-site
networks, real LocalDb sync ports); remove the ClusterRedundancy page's mesh-scope caveat (the
election is pair-local now) and the fallback's site-node island-guard docs note (moot — every
node is a seed of its own mesh); `Cluster__SeedNodes__*` env matrix per pair.
**Exit gate:** rig up in the new shape; every existing live-gated behavior re-verified per pair
(deploy, redundancy 250/240 per pair — **two Primaries fleet-wide, one per pair, by design**);
secrets Akka replication re-verified or re-scoped (it rides DPS on the current single mesh — its
topology must be re-decided here, likely SQL-hub mode like ScadaBridge, since pub/sub cannot
cross separate meshes).

### Phase 7 — Failover drill + live gates
**Scope:** the drill ScadaBridge already has (`failover-drill.sh` analogue) run per pair, both
directions; **close the two outstanding live gates**: (a) auto-down 1-vs-1 crash-the-oldest
(deferred since Phase 0a — finally testable, every mesh is exactly two nodes), (b) self-first
lone-cold-start on the real per-pair topology; manual-failover button re-verified per pair;
operator runbook for the co-located site (one page covering both products' failover on the same
two VMs).
**Exit gate:** drill green on every pair type (central, site); runbook merged; design doc §7
table fully marked DONE. (Live gate (b) is now the **self-first cold-start-alone** drill, not
`SelfFormAfter` — the watchdog it named was retired 2026-07-22.)

---

## Risks carried from the design (§8, unchanged — re-read before each phase plan)

LocalDb becomes load-bearing for config (blast radius of the #485 class rises);
`DbHealthProbeActor` feeds a client-visible value; the rig models the doomed topology until
Phase 6; losing gossip loses free observability (7 panels); 128 KB ClusterClient frame drop is
silent unless both knobs are set; never stack app-level LWW on LocalDb's HLC. **New (this
program):** co-located VMs mean a VM loss now takes out one node of BOTH products at once — the
drill in Phase 7 must include the shared-VM failure (both products fail over together), and
resource sizing on the site VMs should be checked once both products run the full stack.

## Tracking

| Phase | Status |
|---|---|
| 0a downing strategy | DONE 2026-07-21 (live gate → Phase 7) |
| 0b oldest-Up election | DONE 2026-07-21 |
| Prereq: selfform-fallback + manual-failover plan | **DONE 2026-07-22**, merged `a78425ea`. Shipped *not* as the planned `SelfFormAfter` watchdog — the live gate caught that islanding a manually-failed-over node — but as self-first seed ordering + `AkkaClusterOptionsValidator`. Phase 6's seed-ordering scope is therefore already discharged. |
| 1 ClusterNode columns + DB ack set | **DONE 2026-07-22**, merged `7654f24d`. Deploy-seal semantics changed as flagged, but the escape hatch is **`ClusterNode.MaintenanceMode`**, not `Enabled=0` — the live gate found `Enabled=0` is rejected by `DraftValidator.ValidateClusterTopology` on any 2-node pair. Gate record: `2026-07-22-mesh-phase1-live-gate.md`. |
| 2 ClusterClient transport | **IN PROGRESS** — plan being written 2026-07-22 |
| 3 fetch-and-cache | not started |
| 4 cut driver ConfigDb | not started |
| 5 gRPC telemetry | not started |
| 6 mesh partition + co-location | not started |
| 7 drill + live gates | not started |