lmxopcua/docs/plans/2026-07-22-per-cluster-mesh-program.md

Per-Cluster Mesh Program — Align OtOpcUa's Akka Topology with ScadaBridge

For Claude: This is a PROGRAM plan (phase roadmap + gates), not a bite-sized task plan. The authoritative design is docs/plans/2026-07-21-per-cluster-mesh-design.md (decisions settled 2026-07-21; Phases 0a/0b DONE). Per that design's own instruction, each phase gets its own detailed plan, written when the phase starts (superpowers writing-plans → executing-plans), so plans are authored against current code, not against a forecast. This document sequences the phases, fixes the deployment topology (co-location with ScadaBridge — NEW constraint 2026-07-22), and defines each phase's entry/exit gates.

Goal: OtOpcUa runs the same Akka.NET mesh shape as ScadaBridge — one Akka mesh per application Cluster, two nodes max, central ↔ cluster joined by explicit transports (ClusterClient + gRPC + fetch), driver nodes holding no ConfigDb connection — so that every Primary-gated decision and every gated resource share one pair-local scope, and both products present one operational model on the shared site hardware.

Why now (2026-07-22): OtOpcUa cluster pairs will run on the same two Windows VMs as the ScadaBridge site nodes. That deployment makes the current single-fleet-mesh design actively wrong for sites (a site's OtOpcUa nodes would gossip across the WAN to central and elect ONE Primary fleet-wide), and makes the ScadaBridge shape the obviously correct one: each site's two VMs host two independent, identically-postured 2-node clusters (one per product), surviving alone on the site LAN. It also means driver nodes must not depend on reaching central SQL — sites have no SQL Server, so §6.1's "driver nodes never connect to the ConfigDb" stops being an architectural preference and becomes a deployment requirement.

---

Deployment topology (the co-location constraint, NEW)

Each site: 2 Windows VMs. Each VM runs one ScadaBridge site node AND one OtOpcUa driver node. Two independent 2-node Akka clusters per site — they share hardware, never a mesh. Central: 2 Windows VMs, each running a ScadaBridge central node and an OtOpcUa central node (admin,driver), again as separate pairs.

Per-VM port allocation (no collisions — verify against actual deployment configs in Phase 6):

Port	Owner	Purpose
8081 / 8082	ScadaBridge	Akka remoting (central / site)
8083	ScadaBridge site	gRPC h2c (streams + LocalDb sync)
8084	ScadaBridge site	metrics
5000 (+Traefik)	ScadaBridge central	UI + Inbound API
4053	OtOpcUa	Akka remoting
4840	OtOpcUa	OPC UA endpoint
OtOpcUa AdminUI port	OtOpcUa central (admin)	AdminUI (driver-only site nodes host no UI)
OtOpcUa LocalDb sync port	OtOpcUa driver	h2c LocalDb pair replication (default off/0 today — Phase 6 assigns a real per-site port)

Aligned HA posture (both products, per VM — already true or landing via the selfform plan): auto-down downing (15 s window), oldest-Up active/primary election, self-first seed ordering (the SelfFormAfter self-form fallback that briefly stood in for it was retired 2026-07-22), termination-watchdog → process exit → sc.exe failure restart recovery. One failover story for operators regardless of product.

Prerequisite ordering: the fallback/manual-failover plan (docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md) executes before this program — it is small, independent, and Phase 6 depends on its semantics (under per-pair meshes every node lists itself + partner as seeds, so the fallback covers both nodes of every pair; the site-node island guard then simply never triggers).

---

What changes, in one table (from the design doc)

Aspect	Today (single fleet mesh)	Target (ScadaBridge shape)
Mesh	one gossip ring, all nodes, seeded by central-1	one 2-node mesh per application Cluster; same ActorSystem name; separation by seed-node partitioning
Roles	admin / driver fleet-wide	driver + cluster-specific cluster-{ClusterId}; singletons scoped to the cluster role
Primary election	mesh-wide oldest Up driver (Phase 0b)	pair-local by construction — same rule, correct scope
Command/control	9 DPS topics + singleton over gossip	ClusterClient (one receptionist actor per side); central discovers nodes from ClusterNode rows
Deploy	notify (DPS) + node fetches from ConfigDb	notify via ClusterClient; artifact fetched from central, cached in LocalDb
Driver ConfigDb connection	direct EF connection to central SQL	none — LocalDb is the steady-state config store
Live telemetry	7 observability DPS topics	one gRPC stream contract (oneof event), central dials each cluster node
Rig	six-node single mesh (docker-dev)	per-cluster meshes; rig models the real topology

---

Phases

Each phase below is one row of design-doc §7, expanded with entry/exit gates. Execution recipe per phase: (1) invoke writing-plans in this repo to produce docs/plans/2026-07-2X-mesh-phase-N-<name>.md from the scope notes here + the design doc § references, exploring current code first; (2) execute it task-by-task; (3) run the phase's exit gate; (4) update this file's status column and the design doc's §7 table.

Phase 1 — ClusterNode address columns + DB-sourced ack set — DONE 2026-07-22

Shipped per 2026-07-21-per-cluster-mesh-phase1.md; see that plan's Task 6 gate record and the design doc's "Phase 1 as shipped" note for the two scope deviations (no cluster-scope filtering — no such deployment exists; no per-node role column) and the one addition (ClusterNodeAddressReconciler). AdminUI node edit was deferred to Phase 2 — nothing reads the columns until then, and the migration default plus the rig seed cover every node today.

Scope: ClusterNode gains Akka + gRPC address columns (mirroring ScadaBridge's Site entity NodeAAddress/GrpcNodeAAddress pattern, but per-node rows); EF migration; AdminUI node edit surfaces the fields; ConfigPublishCoordinator derives its expected-ack set from ClusterNode rows instead of Akka.Cluster.State.Members filtered by role (design §3 fact 3 — this removes the coordinator's one genuinely mesh-bound dependency). Independent of the split: yes — safe on the current mesh. Exit gate: deploy on the unchanged docker-dev rig completes with the coordinator's expected-ack set proven DB-sourced (test: a ClusterNode row present but node down → deploy reports that node missing; a node up but row absent → its ack is not expected).

Phase 2 — Comm actors + ClusterClient transport — DONE 2026-07-22 (live gate PASSED)

Shipped per 2026-07-22-mesh-phase2-clusterclient-transport.md as a dark switch: MeshTransport:Mode defaults to Dps, so nothing changes on any deployment until the flag flips. The live gate (2026-07-22-mesh-phase2-live-gate.md) flipped the whole rig to ClusterClient and passed for the transport, with four findings — most notably that buffer-size = 0's stated rationale was overstated (a node applies a TimedOut deployment anyway via boot-time orphan-row replay) and that stopping both centrals strands non-seed nodes at the membership layer (a Phase 7 drill item, transport-independent). Both comm actors are spawned and receptionist-registered in both modes, so the flip is a config change rather than a redeploy. Three scope corrections landed against the design doc (that plan's header carries them in full):

• the node path is /user/node-communication, not /user/cluster-communication (there is no per-cluster scoping until Phase 6);
• the substitution for Publish is SendToAll, not Send — Send reaches exactly one node and would deploy to one member of the fleet while the rest silently kept their old config;
• central creates exactly one fleet-wide ClusterClient. On a single mesh a receptionist serves its whole cluster, so a client per application Cluster would fan every command out once per cluster. Per-cluster clients become correct only when Phase 6 splits the meshes (TODO(Phase 6) in CentralCommunicationActor.RebuildClient).

Exit-gate deviation — "an Ask timing out cleanly against a stopped node" cannot be run as written. There is no cross-boundary Ask in Phase 2: every migrated command is fire-and-forget on the wire. DispatchDeployment is a Tell whose ack returns later as an independent ApplyAck; RestartDriver / ReconnectDriver / AlarmCommand reply Ok = true from the admin node the moment they are dispatched (Ok has always meant dispatched, not applied — the AdminUI string is literally "Restart dispatched"). The honest equivalents, both in the live gate: a stopped node fails the deploy at the apply deadline naming it (Phase 1 semantics), and no reachable contact drops the command with a Warning and never buffers it.

Scope: one receptionist-registered actor per side (/user/central-communication, /user/cluster-communication), registered per node, not as a singleton (contact rotation); ClusterClient central → cluster carrying deploy notify + acks + driver-control; the ScadaBridge idioms copied verbatim: sender-preserving Tell(new ClusterClient.Send(...), Sender) Ask relay, typed-failure reply for every unhandled message, no central buffering toward unreachable clusters (drop + warn), central discovers contacts from Phase 1's ClusterNode rows (60 s refresh + admin-change refresh), clusters know central from appsettings (static, restart to change). Frame-size guard: anything carrying payload sets both the frame limit and log-frame-size-exceeding (design §8) — the deploy path stays payload-free by design. Exit gate: on the still-single-mesh rig, deploy notify + acks and AdminUI Reconnect/Restart flow over ClusterClient (DPS paths deleted or dark-switched), including an Ask timing out cleanly against a stopped node.

Phase 3 — Config fetch-and-cache from central — CODE-COMPLETE 2026-07-23 (Tasks 0–8; live gate Task 10 pending)

Decisions taken: the deploy artifact is served by central over a gRPC fetch RPC (DeploymentArtifactService.Fetch, streamed chunks — chosen over token-gated HTTP), authenticated by a shared node bearer key (ConfigServe:ApiKey == ConfigSource:ApiKey, FixedTimeEquals, fail-closed — chosen over a per-deployment token, so no migration and DispatchDeployment stays payload-free). Cutover is a config-flagged dark switch ConfigSource:Mode (Direct default | FetchAndCache), per node. Plan + task ledger: docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md. Shipped (all green, sabotage-checked): ConfigSource/ConfigServe options + validator; the repo's first in-repo .proto + Grpc.Tools codegen; central DeploymentArtifactGrpcService (streams the sealed blob, NotFound-hides unknown/not-sealed/empty); ConfigServeAuthInterceptor + a dedicated h2c Kestrel listener merged with the LocalDb-sync listener (re-binds the existing surface once, adds both dedicated ports); node GrpcDeploymentArtifactFetcher (SHA-verified reassembly, endpoint failover, null-on-any-failure); the FetchAndCache apply path (PipeTo(Self), fetch→cache→apply-from-bytes, null = apply-failure keeping last-known-good) and bootstrap-from-the-LocalDb-pointer (no driver-side config SQL read); #485 coverage; and a real two-Host gRPC boundary test (right-key reassembles, wrong-key control returns null, failover). Chunking, SHA-256 verify, newest-2 retention, pair replication carry over unchanged. Phase boundary held: config reads only — the NodeDeploymentState ack-row write, DbHealthProbe, EfAlarmConditionStateStore stay for Phase 4. Remaining (Tasks 9–10): rig config + docs (done) and the live gate — deploy on a site pair flipped to FetchAndCache with central SQL stopped mid-fetch → retry lands; #485 last-known-good re-proven on the new path. Exit gate: live gate green on the rig; a driver node with an empty LocalDb and reachable central boots into the current config; with central down it boots last-known-good.

Phase 4 — Cut the driver-side ConfigDb connection

Scope (design §6.1 audit table): re-home EfAlarmConditionStateStore to LocalDb (pair-local state, same journey as the Phase-2 alarm S&F buffer); resolve the DbHealthProbeActor question — driver nodes have no DB to probe, and DB health currently feeds ServiceLevel tiering, so define the replacement health input (candidate: central-reachability via the Phase 2/3 transports) — this is a client-visible ServiceLevel semantics change and must be documented in docs/Redundancy.md + the interop playbook; audit OpcUaPublishActor's ConfigDb use (TBD in the design) and re-source it; registration cleanup in ServiceCollectionExtensions; driver-role Program.cs branch registers no EF ConfigDb context at all (mirror ScadaBridge's central-only AddConfigurationDatabase). Exit gate: its own live gate — a driver pair runs a full deploy + alarm + historian cycle with no ConfigDb connection string configured at all; grep-level proof no driver-branch service can resolve the ConfigDb context.

Phase 5 — gRPC stream contract for live telemetry

Scope: one server-streaming contract carrying a oneof event, cluster nodes host the gRPC server, central dials in (the inverted direction is the load-bearing ScadaBridge finding — design §2); migrate the seven observability topics (alerts, driver-health, driver-resilience-status, fleet-status, script-logs, plus redundancy-state distribution and deployment-acks if Phase 2 left them on DPS); additive-only field evolution, contract locked by test; per-panel reconnect story for the AdminUI (design §8 — losing gossip loses free fleet observability). Exit gate: all AdminUI live panels green against a pair with DPS telemetry topics deleted; kill-and-reconnect of the central dialer recovers every stream.

Phase 6 — Mesh partition + co-location topology

Scope: per-cluster seed nodes — adopt ScadaBridge's self-first ordering and RETIRE the SelfFormAfter watchdog + TCP reachability guard DONE EARLY 2026-07-22 (docker-dev central-2 swapped to self-first, ClusterBootstrapFallback/SelfFormAfter deleted, AkkaClusterOptionsValidator ports ScadaBridge's startup rule in its conditional form, and SelfFirstSeedBootstrapTests replaces SelfFormBootstrapTests; see docs/Redundancy.md §"Bootstrap: self-first seed ordering"). What remains for this phase is the per-pair Cluster__SeedNodes__* matrix once the meshes actually split — every node in a pair is then a seed of its own mesh, so the site-node exemption disappears; cluster-scoped roles cluster-{ClusterId} + singleton re-scoping, central pair keeps the admin singletons; docker-dev rig rewritten to model the real topology — including the co-location port table above (both products' compose files on shared per-site networks, real LocalDb sync ports); remove the ClusterRedundancy page's mesh-scope caveat (the election is pair-local now) and the fallback's site-node island-guard docs note (moot — every node is a seed of its own mesh); Cluster__SeedNodes__* env matrix per pair. Exit gate: rig up in the new shape; every existing live-gated behavior re-verified per pair (deploy, redundancy 250/240 per pair — two Primaries fleet-wide, one per pair, by design); secrets Akka replication re-verified or re-scoped (it rides DPS on the current single mesh — its topology must be re-decided here, likely SQL-hub mode like ScadaBridge, since pub/sub cannot cross separate meshes).

Phase 7 — Failover drill + live gates

Scope: the drill ScadaBridge already has (failover-drill.sh analogue) run per pair, both directions; close the two outstanding live gates: (a) auto-down 1-vs-1 crash-the-oldest (deferred since Phase 0a — finally testable, every mesh is exactly two nodes), (b) self-first lone-cold-start on the real per-pair topology; manual-failover button re-verified per pair; operator runbook for the co-located site (one page covering both products' failover on the same two VMs). Exit gate: drill green on every pair type (central, site); runbook merged; design doc §7 table fully marked DONE. (Live gate (b) is now the self-first cold-start-alone drill, not SelfFormAfter — the watchdog it named was retired 2026-07-22.)

---

Risks carried from the design (§8, unchanged — re-read before each phase plan)

LocalDb becomes load-bearing for config (blast radius of the #485 class rises); DbHealthProbeActor feeds a client-visible value; the rig models the doomed topology until Phase 6; losing gossip loses free observability (7 panels); 128 KB ClusterClient frame drop is silent unless both knobs are set; never stack app-level LWW on LocalDb's HLC. New (this program): co-located VMs mean a VM loss now takes out one node of BOTH products at once — the drill in Phase 7 must include the shared-VM failure (both products fail over together), and resource sizing on the site VMs should be checked once both products run the full stack.

Tracking

Phase	Status
0a downing strategy	DONE 2026-07-21 (live gate → Phase 7)
0b oldest-Up election	DONE 2026-07-21
Prereq: selfform-fallback + manual-failover plan	DONE 2026-07-22, merged a78425ea. Shipped not as the planned SelfFormAfter watchdog — the live gate caught that islanding a manually-failed-over node — but as self-first seed ordering + AkkaClusterOptionsValidator. Phase 6's seed-ordering scope is therefore already discharged.
1 ClusterNode columns + DB ack set	DONE 2026-07-22, merged 7654f24d. Deploy-seal semantics changed as flagged, but the escape hatch is ClusterNode.MaintenanceMode, not Enabled=0 — the live gate found Enabled=0 is rejected by DraftValidator.ValidateClusterTopology on any 2-node pair. Gate record: 2026-07-22-mesh-phase1-live-gate.md.
2 ClusterClient transport	DONE 2026-07-22 (live gate PASSED, shipped dark) — shipped dark (MeshTransport:Mode=Dps default), both comm actors registered in both modes. Corrections: SendToAll not Send, one fleet-wide client not one per cluster, /user/node-communication not /user/cluster-communication. The gate's "Ask timing out" item does not exist to test — no cross-boundary Ask in this phase; see the phase section.
3 fetch-and-cache	CODE-COMPLETE 2026-07-23 (Tasks 0–8 merged to the phase branch, all green + sabotage-checked; live gate Task 10 pending). gRPC fetch RPC + shared node key; dark switch ConfigSource:Mode (Direct default). Rig flip: OTOPCUA_CONFIG_MODE=FetchAndCache on the site nodes. See 2026-07-22-mesh-phase3-config-fetch-and-cache.md.
4 cut driver ConfigDb	not started
5 gRPC telemetry	not started
6 mesh partition + co-location	not started
7 drill + live gates	not started