lmxopcua/docs/plans/2026-07-22-per-cluster-mesh-program.md

# Per-Cluster Mesh Program — Align OtOpcUa's Akka Topology with ScadaBridge

> **For Claude:** This is a PROGRAM plan (phase roadmap + gates), not a bite-sized task plan.
> The authoritative design is `docs/plans/2026-07-21-per-cluster-mesh-design.md` (decisions settled
> 2026-07-21; Phases 0a/0b DONE). Per that design's own instruction, **each phase gets its own
> detailed plan, written when the phase starts** (superpowers writing-plans → executing-plans),
> so plans are authored against current code, not against a forecast. This document sequences the
> phases, fixes the deployment topology (co-location with ScadaBridge — NEW constraint 2026-07-22),
> and defines each phase's entry/exit gates.

**Goal:** OtOpcUa runs the same Akka.NET mesh shape as ScadaBridge — **one Akka mesh per
application `Cluster`, two nodes max**, central ↔ cluster joined by explicit transports
(ClusterClient + gRPC + fetch), driver nodes holding **no ConfigDb connection** — so that every
Primary-gated decision and every gated resource share one pair-local scope, and both products
present one operational model on the shared site hardware.

**Why now (2026-07-22):** OtOpcUa cluster pairs will run **on the same two Windows VMs as the
ScadaBridge site nodes**. That deployment makes the current single-fleet-mesh design actively
wrong for sites (a site's OtOpcUa nodes would gossip across the WAN to central and elect ONE
Primary fleet-wide), and makes the ScadaBridge shape the obviously correct one: each site's two
VMs host two independent, identically-postured 2-node clusters (one per product), surviving alone
on the site LAN. It also means driver nodes must not depend on reaching central SQL — **sites
have no SQL Server**, so §6.1's "driver nodes never connect to the ConfigDb" stops being an
architectural preference and becomes a deployment requirement.

---

## Deployment topology (the co-location constraint, NEW)

Each **site**: 2 Windows VMs. Each VM runs one ScadaBridge site node AND one OtOpcUa driver node.
Two independent 2-node Akka clusters per site — they share hardware, never a mesh. **Central**:
2 Windows VMs, each running a ScadaBridge central node and an OtOpcUa central node
(`admin,driver`), again as separate pairs.

**Per-VM port allocation (no collisions — verify against actual deployment configs in Phase 6):**

| Port | Owner | Purpose |
|---|---|---|
| 8081 / 8082 | ScadaBridge | Akka remoting (central / site) |
| 8083 | ScadaBridge site | gRPC h2c (streams + LocalDb sync) |
| 8084 | ScadaBridge site | metrics |
| 5000 (+Traefik) | ScadaBridge central | UI + Inbound API |
| **4053** | OtOpcUa | Akka remoting |
| **4840** | OtOpcUa | OPC UA endpoint |
| OtOpcUa AdminUI port | OtOpcUa central (admin) | AdminUI (driver-only site nodes host no UI) |
| OtOpcUa LocalDb sync port | OtOpcUa driver | h2c LocalDb pair replication (default off/0 today — Phase 6 assigns a real per-site port) |

**Aligned HA posture (both products, per VM — already true or landing via the selfform plan):**
auto-down downing (15 s window), oldest-Up active/primary election, self-first seed ordering
(the `SelfFormAfter` self-form fallback that briefly stood in for it was retired 2026-07-22),
termination-watchdog → process exit → `sc.exe failure` restart recovery.
One failover story for operators regardless of product.

**Prerequisite ordering:** the fallback/manual-failover plan
(`docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`) executes **before** this
program — it is small, independent, and Phase 6 depends on its semantics (under per-pair meshes
every node lists itself + partner as seeds, so the fallback covers both nodes of every pair;
the site-node island guard then simply never triggers).

---

## What changes, in one table (from the design doc)

| Aspect | Today (single fleet mesh) | Target (ScadaBridge shape) |
|---|---|---|
| Mesh | one gossip ring, all nodes, seeded by central-1 | one 2-node mesh per application `Cluster`; same ActorSystem name; separation by seed-node partitioning |
| Roles | `admin` / `driver` fleet-wide | `driver` + cluster-specific `cluster-{ClusterId}`; singletons scoped to the cluster role |
| Primary election | mesh-wide oldest Up driver (Phase 0b) | pair-local by construction — same rule, correct scope |
| Command/control | 9 DPS topics + singleton over gossip | ClusterClient (one receptionist actor per side); central discovers nodes from `ClusterNode` rows |
| Deploy | notify (DPS) + node fetches from ConfigDb | notify via ClusterClient; artifact fetched **from central**, cached in LocalDb |
| Driver ConfigDb connection | direct EF connection to central SQL | **none** — LocalDb is the steady-state config store |
| Live telemetry | 7 observability DPS topics | one gRPC stream contract (`oneof` event), central dials each cluster node |
| Rig | six-node single mesh (`docker-dev`) | per-cluster meshes; rig models the real topology |

---

## Phases

Each phase below is one row of design-doc §7, expanded with entry/exit gates. **Execution recipe
per phase:** (1) invoke writing-plans in this repo to produce
`docs/plans/2026-07-2X-mesh-phase-N-<name>.md` from the scope notes here + the design doc §
references, exploring current code first; (2) execute it task-by-task; (3) run the phase's exit
gate; (4) update this file's status column and the design doc's §7 table.

### Phase 1 — `ClusterNode` address columns + DB-sourced ack set — **DONE 2026-07-22**
Shipped per `2026-07-21-per-cluster-mesh-phase1.md`; see that plan's Task 6 gate record and the
design doc's "Phase 1 as shipped" note for the two scope deviations (no cluster-scope filtering — no
such deployment exists; no per-node role column) and the one addition (`ClusterNodeAddressReconciler`).
**AdminUI node edit was deferred to Phase 2** — nothing reads the columns until then, and the
migration default plus the rig seed cover every node today.

**Scope:** `ClusterNode` gains Akka + gRPC address columns (mirroring ScadaBridge's `Site`
entity `NodeAAddress`/`GrpcNodeAAddress` pattern, but per-node rows); EF migration; AdminUI node
edit surfaces the fields; `ConfigPublishCoordinator` derives its expected-ack set from
`ClusterNode` rows instead of `Akka.Cluster.State.Members` filtered by role (design §3 fact 3 —
this removes the coordinator's one genuinely mesh-bound dependency).
**Independent of the split:** yes — safe on the current mesh.
**Exit gate:** deploy on the unchanged docker-dev rig completes with the coordinator's expected-ack
set proven DB-sourced (test: a `ClusterNode` row present but node down → deploy reports that node
missing; a node up but row absent → its ack is not expected).

### Phase 2 — Comm actors + ClusterClient transport — **DONE 2026-07-22 (live gate PASSED)**
Shipped per [`2026-07-22-mesh-phase2-clusterclient-transport.md`](2026-07-22-mesh-phase2-clusterclient-transport.md)
as a **dark switch**: `MeshTransport:Mode` defaults to `Dps`, so nothing changes on any deployment
until the flag flips. The live gate ([`2026-07-22-mesh-phase2-live-gate.md`](2026-07-22-mesh-phase2-live-gate.md))
flipped the whole rig to `ClusterClient` and passed for the transport, with four findings — most
notably that `buffer-size = 0`'s stated rationale was overstated (a node applies a `TimedOut`
deployment anyway via boot-time orphan-row replay) and that stopping both centrals strands non-seed
nodes at the membership layer (a Phase 7 drill item, transport-independent). Both comm actors are spawned and receptionist-registered in **both** modes, so
the flip is a config change rather than a redeploy. Three scope corrections landed against the
design doc (that plan's header carries them in full):

- the node path is `/user/node-communication`, not `/user/cluster-communication` (there is no
  per-cluster scoping until Phase 6);
- the substitution for `Publish` is **`SendToAll`**, not `Send` — `Send` reaches exactly one node and
  would deploy to one member of the fleet while the rest silently kept their old config;
- central creates **exactly one fleet-wide ClusterClient**. On a single mesh a receptionist serves
  its whole cluster, so a client per application `Cluster` would fan every command out once per
  cluster. Per-cluster clients become correct only when Phase 6 splits the meshes (`TODO(Phase 6)`
  in `CentralCommunicationActor.RebuildClient`).

**Exit-gate deviation — "an Ask timing out cleanly against a stopped node" cannot be run as written.**
There is no cross-boundary Ask in Phase 2: every migrated command is fire-and-forget on the wire.
`DispatchDeployment` is a `Tell` whose ack returns later as an independent `ApplyAck`;
`RestartDriver` / `ReconnectDriver` / `AlarmCommand` reply `Ok = true` from the admin node the moment
they are dispatched (`Ok` has always meant *dispatched*, not *applied* — the AdminUI string is
literally "Restart dispatched"). The honest equivalents, both in the live gate: a **stopped node**
fails the deploy at the apply deadline naming it (Phase 1 semantics), and **no reachable contact**
drops the command with a Warning and never buffers it.

**Scope:** one receptionist-registered actor per side (`/user/central-communication`,
`/user/cluster-communication`), registered **per node, not as a singleton** (contact rotation);
ClusterClient central → cluster carrying deploy notify + acks + driver-control; the ScadaBridge
idioms copied verbatim: sender-preserving `Tell(new ClusterClient.Send(...), Sender)` Ask relay,
typed-failure reply for every unhandled message, no central buffering toward unreachable clusters
(drop + warn), central discovers contacts from Phase 1's `ClusterNode` rows (60 s refresh +
admin-change refresh), clusters know central from appsettings (static, restart to change).
**Frame-size guard:** anything carrying payload sets both the frame limit and
`log-frame-size-exceeding` (design §8) — the deploy path stays payload-free by design.
**Exit gate:** on the still-single-mesh rig, deploy notify + acks and AdminUI Reconnect/Restart
flow over ClusterClient (DPS paths deleted or dark-switched), including an Ask timing out cleanly
against a stopped node.

### Phase 3 — Config fetch-and-cache from central — **CODE-COMPLETE 2026-07-23 (Tasks 0–8; live gate Task 10 pending)**
**Decisions taken:** the deploy artifact is served **by central over a gRPC fetch RPC**
(`DeploymentArtifactService.Fetch`, streamed chunks — chosen over token-gated HTTP), authenticated by
a **shared node bearer key** (`ConfigServe:ApiKey == ConfigSource:ApiKey`, `FixedTimeEquals`,
fail-closed — chosen over a per-deployment token, so **no migration** and `DispatchDeployment` stays
payload-free). Cutover is a **config-flagged dark switch** `ConfigSource:Mode` (`Direct` default |
`FetchAndCache`), per node. Plan + task ledger:
[`docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md`](2026-07-22-mesh-phase3-config-fetch-and-cache.md).
**Shipped (all green, sabotage-checked):** `ConfigSource`/`ConfigServe` options + validator; the repo's
first in-repo `.proto` + `Grpc.Tools` codegen; central `DeploymentArtifactGrpcService` (streams the
sealed blob, NotFound-hides unknown/not-sealed/empty); `ConfigServeAuthInterceptor` + a dedicated h2c
Kestrel listener merged with the LocalDb-sync listener (re-binds the existing surface once, adds both
dedicated ports); node `GrpcDeploymentArtifactFetcher` (SHA-verified reassembly, endpoint failover,
null-on-any-failure); the `FetchAndCache` apply path (`PipeTo(Self)`, fetch→cache→apply-from-bytes,
null = apply-failure keeping last-known-good) and bootstrap-from-the-LocalDb-pointer (no driver-side
config SQL read); #485 coverage; and a **real two-Host gRPC boundary test** (right-key reassembles,
wrong-key control returns null, failover). Chunking, SHA-256 verify, newest-2 retention, pair
replication carry over unchanged. **Phase boundary held:** config **reads** only — the
`NodeDeploymentState` ack-row write, `DbHealthProbe`, `EfAlarmConditionStateStore` stay for Phase 4.
**Remaining (Tasks 9–10):** rig config + docs (done) and the live gate — deploy on a site pair flipped
to `FetchAndCache` with central SQL stopped mid-fetch → retry lands; #485 last-known-good re-proven on
the new path.
**Exit gate:** live gate green on the rig; a driver node with an empty LocalDb and reachable central
boots into the current config; with central down it boots last-known-good.

### Phase 4 — Cut the driver-side ConfigDb connection
**Scope (design §6.1 audit table):** re-home `EfAlarmConditionStateStore` to LocalDb (pair-local
state, same journey as the Phase-2 alarm S&F buffer); **resolve the `DbHealthProbeActor`
question** — driver nodes have no DB to probe, and DB health currently feeds ServiceLevel tiering,
so define the replacement health input (candidate: central-reachability via the Phase 2/3
transports) — this is a client-visible ServiceLevel semantics change and must be documented in
`docs/Redundancy.md` + the interop playbook; audit `OpcUaPublishActor`'s ConfigDb use (TBD in the
design) and re-source it; registration cleanup in `ServiceCollectionExtensions`; driver-role
`Program.cs` branch registers no EF ConfigDb context at all (mirror ScadaBridge's central-only
`AddConfigurationDatabase`).
**Exit gate:** its own live gate — a driver pair runs a full deploy + alarm + historian cycle with
**no ConfigDb connection string configured at all**; grep-level proof no driver-branch service can
resolve the ConfigDb context.

### Phase 5 — gRPC stream contract for live telemetry
**Scope:** one server-streaming contract carrying a `oneof` event, **cluster nodes host the gRPC
server, central dials in** (the inverted direction is the load-bearing ScadaBridge finding —
design §2); migrate the seven observability topics (`alerts`, `driver-health`,
`driver-resilience-status`, `fleet-status`, `script-logs`, plus redundancy-state distribution and
deployment-acks if Phase 2 left them on DPS); additive-only field evolution, contract locked by
test; per-panel reconnect story for the AdminUI (design §8 — losing gossip loses free fleet
observability).
**Exit gate:** all AdminUI live panels green against a pair with DPS telemetry topics deleted;
kill-and-reconnect of the central dialer recovers every stream.

### Phase 6 — Mesh partition + co-location topology
**Scope:** per-cluster seed nodes — ~~adopt ScadaBridge's self-first ordering and RETIRE the
`SelfFormAfter` watchdog + TCP reachability guard~~ **DONE EARLY 2026-07-22** (docker-dev
`central-2` swapped to self-first, `ClusterBootstrapFallback`/`SelfFormAfter` deleted,
`AkkaClusterOptionsValidator` ports ScadaBridge's startup rule in its conditional form, and
`SelfFirstSeedBootstrapTests` replaces `SelfFormBootstrapTests`; see `docs/Redundancy.md`
§"Bootstrap: self-first seed ordering"). What remains for this phase is the per-pair
`Cluster__SeedNodes__*` matrix once the meshes actually split — every node in a pair is then a seed
of its own mesh, so the site-node exemption disappears; cluster-scoped roles `cluster-{ClusterId}` + singleton re-scoping,
central pair keeps the admin singletons; **docker-dev rig rewritten** to model the real topology —
including the co-location port table above (both products' compose files on shared per-site
networks, real LocalDb sync ports); remove the ClusterRedundancy page's mesh-scope caveat (the
election is pair-local now) and the fallback's site-node island-guard docs note (moot — every
node is a seed of its own mesh); `Cluster__SeedNodes__*` env matrix per pair.
**Exit gate:** rig up in the new shape; every existing live-gated behavior re-verified per pair
(deploy, redundancy 250/240 per pair — **two Primaries fleet-wide, one per pair, by design**);
secrets Akka replication re-verified or re-scoped (it rides DPS on the current single mesh — its
topology must be re-decided here, likely SQL-hub mode like ScadaBridge, since pub/sub cannot
cross separate meshes).

### Phase 7 — Failover drill + live gates
**Scope:** the drill ScadaBridge already has (`failover-drill.sh` analogue) run per pair, both
directions; **close the two outstanding live gates**: (a) auto-down 1-vs-1 crash-the-oldest
(deferred since Phase 0a — finally testable, every mesh is exactly two nodes), (b) self-first
lone-cold-start on the real per-pair topology; manual-failover button re-verified per pair;
operator runbook for the co-located site (one page covering both products' failover on the same
two VMs).
**Exit gate:** drill green on every pair type (central, site); runbook merged; design doc §7
table fully marked DONE. (Live gate (b) is now the **self-first cold-start-alone** drill, not
`SelfFormAfter` — the watchdog it named was retired 2026-07-22.)

---

## Risks carried from the design (§8, unchanged — re-read before each phase plan)

LocalDb becomes load-bearing for config (blast radius of the #485 class rises);
`DbHealthProbeActor` feeds a client-visible value; the rig models the doomed topology until
Phase 6; losing gossip loses free observability (7 panels); 128 KB ClusterClient frame drop is
silent unless both knobs are set; never stack app-level LWW on LocalDb's HLC. **New (this
program):** co-located VMs mean a VM loss now takes out one node of BOTH products at once — the
drill in Phase 7 must include the shared-VM failure (both products fail over together), and
resource sizing on the site VMs should be checked once both products run the full stack.

## Tracking

| Phase | Status |
|---|---|
| 0a downing strategy | DONE 2026-07-21 (live gate → Phase 7) |
| 0b oldest-Up election | DONE 2026-07-21 |
| Prereq: selfform-fallback + manual-failover plan | **DONE 2026-07-22**, merged `a78425ea`. Shipped *not* as the planned `SelfFormAfter` watchdog — the live gate caught that islanding a manually-failed-over node — but as self-first seed ordering + `AkkaClusterOptionsValidator`. Phase 6's seed-ordering scope is therefore already discharged. |
| 1 ClusterNode columns + DB ack set | **DONE 2026-07-22**, merged `7654f24d`. Deploy-seal semantics changed as flagged, but the escape hatch is **`ClusterNode.MaintenanceMode`**, not `Enabled=0` — the live gate found `Enabled=0` is rejected by `DraftValidator.ValidateClusterTopology` on any 2-node pair. Gate record: `2026-07-22-mesh-phase1-live-gate.md`. |
| 2 ClusterClient transport | **DONE 2026-07-22** (live gate PASSED, shipped dark) — shipped dark (`MeshTransport:Mode=Dps` default), both comm actors registered in both modes. Corrections: `SendToAll` not `Send`, one fleet-wide client not one per cluster, `/user/node-communication` not `/user/cluster-communication`. The gate's "Ask timing out" item does not exist to test — no cross-boundary Ask in this phase; see the phase section. |
| 3 fetch-and-cache | **CODE-COMPLETE 2026-07-23** (Tasks 0–8 merged to the phase branch, all green + sabotage-checked; live gate Task 10 pending). gRPC fetch RPC + shared node key; dark switch `ConfigSource:Mode` (Direct default). Rig flip: `OTOPCUA_CONFIG_MODE=FetchAndCache` on the site nodes. See `2026-07-22-mesh-phase3-config-fetch-and-cache.md`. |
| 4 cut driver ConfigDb | not started |
| 5 gRPC telemetry | not started |
| 6 mesh partition + co-location | not started |
| 7 drill + live gates | not started |