lmxopcua/docs/plans/2026-07-22-per-cluster-mesh-program.md

Per-Cluster Mesh Program — Align OtOpcUa's Akka Topology with ScadaBridge

For Claude: This is a PROGRAM plan (phase roadmap + gates), not a bite-sized task plan. The authoritative design is docs/plans/2026-07-21-per-cluster-mesh-design.md (decisions settled 2026-07-21; Phases 0a/0b DONE). Per that design's own instruction, each phase gets its own detailed plan, written when the phase starts (superpowers writing-plans → executing-plans), so plans are authored against current code, not against a forecast. This document sequences the phases, fixes the deployment topology (co-location with ScadaBridge — NEW constraint 2026-07-22), and defines each phase's entry/exit gates.

Goal: OtOpcUa runs the same Akka.NET mesh shape as ScadaBridge — one Akka mesh per application Cluster, two nodes max, central ↔ cluster joined by explicit transports (ClusterClient + gRPC + fetch), driver nodes holding no ConfigDb connection — so that every Primary-gated decision and every gated resource share one pair-local scope, and both products present one operational model on the shared site hardware.

Why now (2026-07-22): OtOpcUa cluster pairs will run on the same two Windows VMs as the ScadaBridge site nodes. That deployment makes the current single-fleet-mesh design actively wrong for sites (a site's OtOpcUa nodes would gossip across the WAN to central and elect ONE Primary fleet-wide), and makes the ScadaBridge shape the obviously correct one: each site's two VMs host two independent, identically-postured 2-node clusters (one per product), surviving alone on the site LAN. It also means driver nodes must not depend on reaching central SQL — sites have no SQL Server, so §6.1's "driver nodes never connect to the ConfigDb" stops being an architectural preference and becomes a deployment requirement.

---

Deployment topology (the co-location constraint, NEW)

Each site: 2 Windows VMs. Each VM runs one ScadaBridge site node AND one OtOpcUa driver node. Two independent 2-node Akka clusters per site — they share hardware, never a mesh. Central: 2 Windows VMs, each running a ScadaBridge central node and an OtOpcUa central node (admin,driver), again as separate pairs.

Per-VM port allocation (no collisions — verify against actual deployment configs in Phase 6):

Port	Owner	Purpose
8081 / 8082	ScadaBridge	Akka remoting (central / site)
8083	ScadaBridge site	gRPC h2c (streams + LocalDb sync)
8084	ScadaBridge site	metrics
5000 (+Traefik)	ScadaBridge central	UI + Inbound API
4053	OtOpcUa	Akka remoting
4840	OtOpcUa	OPC UA endpoint
OtOpcUa AdminUI port	OtOpcUa central (admin)	AdminUI (driver-only site nodes host no UI)
OtOpcUa LocalDb sync port	OtOpcUa driver	h2c LocalDb pair replication (default off/0 today — Phase 6 assigns a real per-site port)

Aligned HA posture (both products, per VM — already true or landing via the selfform plan): auto-down downing (15 s window), oldest-Up active/primary election, self-first seed ordering (the SelfFormAfter self-form fallback that briefly stood in for it was retired 2026-07-22), termination-watchdog → process exit → sc.exe failure restart recovery. One failover story for operators regardless of product.

Prerequisite ordering: the fallback/manual-failover plan (docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md) executes before this program — it is small, independent, and Phase 6 depends on its semantics (under per-pair meshes every node lists itself + partner as seeds, so the fallback covers both nodes of every pair; the site-node island guard then simply never triggers).

---

What changes, in one table (from the design doc)

Aspect	Today (single fleet mesh)	Target (ScadaBridge shape)
Mesh	one gossip ring, all nodes, seeded by central-1	one 2-node mesh per application Cluster; same ActorSystem name; separation by seed-node partitioning
Roles	admin / driver fleet-wide	driver + cluster-specific cluster-{ClusterId}; singletons scoped to the cluster role
Primary election	mesh-wide oldest Up driver (Phase 0b)	pair-local by construction — same rule, correct scope
Command/control	9 DPS topics + singleton over gossip	ClusterClient (one receptionist actor per side); central discovers nodes from ClusterNode rows
Deploy	notify (DPS) + node fetches from ConfigDb	notify via ClusterClient; artifact fetched from central, cached in LocalDb
Driver ConfigDb connection	direct EF connection to central SQL	none — LocalDb is the steady-state config store
Live telemetry	7 observability DPS topics	one gRPC stream contract (oneof event), central dials each cluster node
Rig	six-node single mesh (docker-dev)	per-cluster meshes; rig models the real topology

---

Phases

Each phase below is one row of design-doc §7, expanded with entry/exit gates. Execution recipe per phase: (1) invoke writing-plans in this repo to produce docs/plans/2026-07-2X-mesh-phase-N-<name>.md from the scope notes here + the design doc § references, exploring current code first; (2) execute it task-by-task; (3) run the phase's exit gate; (4) update this file's status column and the design doc's §7 table.

Phase 1 — ClusterNode address columns + DB-sourced ack set — DONE 2026-07-22

Shipped per 2026-07-21-per-cluster-mesh-phase1.md; see that plan's Task 6 gate record and the design doc's "Phase 1 as shipped" note for the two scope deviations (no cluster-scope filtering — no such deployment exists; no per-node role column) and the one addition (ClusterNodeAddressReconciler). AdminUI node edit was deferred to Phase 2 — nothing reads the columns until then, and the migration default plus the rig seed cover every node today.

Scope: ClusterNode gains Akka + gRPC address columns (mirroring ScadaBridge's Site entity NodeAAddress/GrpcNodeAAddress pattern, but per-node rows); EF migration; AdminUI node edit surfaces the fields; ConfigPublishCoordinator derives its expected-ack set from ClusterNode rows instead of Akka.Cluster.State.Members filtered by role (design §3 fact 3 — this removes the coordinator's one genuinely mesh-bound dependency). Independent of the split: yes — safe on the current mesh. Exit gate: deploy on the unchanged docker-dev rig completes with the coordinator's expected-ack set proven DB-sourced (test: a ClusterNode row present but node down → deploy reports that node missing; a node up but row absent → its ack is not expected).

Phase 2 — Comm actors + ClusterClient transport — DONE 2026-07-22 (live gate PASSED)

Shipped per 2026-07-22-mesh-phase2-clusterclient-transport.md as a dark switch: MeshTransport:Mode defaults to Dps, so nothing changes on any deployment until the flag flips. The live gate (2026-07-22-mesh-phase2-live-gate.md) flipped the whole rig to ClusterClient and passed for the transport, with four findings — most notably that buffer-size = 0's stated rationale was overstated (a node applies a TimedOut deployment anyway via boot-time orphan-row replay) and that stopping both centrals strands non-seed nodes at the membership layer (a Phase 7 drill item, transport-independent). Both comm actors are spawned and receptionist-registered in both modes, so the flip is a config change rather than a redeploy. Three scope corrections landed against the design doc (that plan's header carries them in full):

• the node path is /user/node-communication, not /user/cluster-communication (there is no per-cluster scoping until Phase 6);
• the substitution for Publish is SendToAll, not Send — Send reaches exactly one node and would deploy to one member of the fleet while the rest silently kept their old config;
• central creates exactly one fleet-wide ClusterClient. On a single mesh a receptionist serves its whole cluster, so a client per application Cluster would fan every command out once per cluster. Per-cluster clients become correct only when Phase 6 splits the meshes (TODO(Phase 6) in CentralCommunicationActor.RebuildClient).

Exit-gate deviation — "an Ask timing out cleanly against a stopped node" cannot be run as written. There is no cross-boundary Ask in Phase 2: every migrated command is fire-and-forget on the wire. DispatchDeployment is a Tell whose ack returns later as an independent ApplyAck; RestartDriver / ReconnectDriver / AlarmCommand reply Ok = true from the admin node the moment they are dispatched (Ok has always meant dispatched, not applied — the AdminUI string is literally "Restart dispatched"). The honest equivalents, both in the live gate: a stopped node fails the deploy at the apply deadline naming it (Phase 1 semantics), and no reachable contact drops the command with a Warning and never buffers it.

Scope: one receptionist-registered actor per side (/user/central-communication, /user/cluster-communication), registered per node, not as a singleton (contact rotation); ClusterClient central → cluster carrying deploy notify + acks + driver-control; the ScadaBridge idioms copied verbatim: sender-preserving Tell(new ClusterClient.Send(...), Sender) Ask relay, typed-failure reply for every unhandled message, no central buffering toward unreachable clusters (drop + warn), central discovers contacts from Phase 1's ClusterNode rows (60 s refresh + admin-change refresh), clusters know central from appsettings (static, restart to change). Frame-size guard: anything carrying payload sets both the frame limit and log-frame-size-exceeding (design §8) — the deploy path stays payload-free by design. Exit gate: on the still-single-mesh rig, deploy notify + acks and AdminUI Reconnect/Restart flow over ClusterClient (DPS paths deleted or dark-switched), including an Ask timing out cleanly against a stopped node.

Phase 3 — Config fetch-and-cache from central — DONE 2026-07-23 (merged d01b0695, live gate PASSED, pushed to origin)

Decisions taken: the deploy artifact is served by central over a gRPC fetch RPC (DeploymentArtifactService.Fetch, streamed chunks — chosen over token-gated HTTP), authenticated by a shared node bearer key (ConfigServe:ApiKey == ConfigSource:ApiKey, FixedTimeEquals, fail-closed — chosen over a per-deployment token, so no migration and DispatchDeployment stays payload-free). Cutover is a config-flagged dark switch ConfigSource:Mode (Direct default | FetchAndCache), per node. Plan + task ledger: docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md. Shipped (all green, sabotage-checked): ConfigSource/ConfigServe options + validator; the repo's first in-repo .proto + Grpc.Tools codegen; central DeploymentArtifactGrpcService (streams the sealed blob, NotFound-hides unknown/not-sealed/empty); ConfigServeAuthInterceptor + a dedicated h2c Kestrel listener merged with the LocalDb-sync listener (re-binds the existing surface once, adds both dedicated ports); node GrpcDeploymentArtifactFetcher (SHA-verified reassembly, endpoint failover, null-on-any-failure); the FetchAndCache apply path (PipeTo(Self), fetch→cache→apply-from-bytes, null = apply-failure keeping last-known-good) and bootstrap-from-the-LocalDb-pointer (no driver-side config SQL read); #485 coverage; and a real two-Host gRPC boundary test (right-key reassembles, wrong-key control returns null, failover). Chunking, SHA-256 verify, newest-2 retention, pair replication carry over unchanged. Phase boundary held: config reads only — the NodeDeploymentState ack-row write, DbHealthProbe, EfAlarmConditionStateStore stay for Phase 4. Live gate PASSED: deployed on a site pair flipped to FetchAndCache with central SQL stopped mid-fetch — retry landed; #485 last-known-good re-proven on the new path. Merged to master d01b0695 and pushed to origin; the scadaproj umbrella index was updated + pushed (b5e7bc8) in the same pass. Exit gate — MET: live gate green on the rig; a driver node with an empty LocalDb and reachable central boots into the current config; with central down it boots last-known-good.

Phase 4 — Cut the driver-side ConfigDb connection — DONE 2026-07-23 (live gate PASSED; Tasks 0–8 + 1b + 10–11; Task 9 table-drop deferred)

Scope (design §6.1 audit table): re-home EfAlarmConditionStateStore to LocalDb (pair-local state, same journey as the Phase-2 alarm S&F buffer); resolve the DbHealthProbeActor question — driver nodes have no DB to probe, and DB health currently feeds ServiceLevel tiering, so define the replacement health input (candidate: central-reachability via the Phase 2/3 transports) — this is a client-visible ServiceLevel semantics change and must be documented in docs/Redundancy.md + the interop playbook; audit OpcUaPublishActor's ConfigDb use (TBD in the design) and re-source it; registration cleanup in ServiceCollectionExtensions; driver-role Program.cs branch registers no EF ConfigDb context at all (mirror ScadaBridge's central-only AddConfigurationDatabase). Shipped (branch feat/mesh-phase4, Tasks 0–7 + 1b): Program.cs gates AddOtOpcUaConfigDb + the ConfigDb health check on hasAdmin, so a driver-only node registers no ConfigDb connection string at all; ConfigSourceOptionsValidator fails host start if a driver-only node's ConfigSource:Mode is left Direct (fused admin,driver may stay Direct); the DB-health question resolved as retire, don't replace — DbHealthProbeActor is simply not spawned when no ConfigDb is present, and OpcUaPublishActor runs a dbLess mode feeding DbReachable=true constant with staleness from the redundancy-snapshot age alone (documented as the client-visible change in docs/Redundancy.md); LocalDbAlarmConditionStateStore (replicated alarm_condition_state table) replaces EfAlarmConditionStateStore on every driver-role node; DriverHostActor no longer writes NodeDeploymentState on a driver-only node (central's ConfigPublishCoordinator.PersistNodeAck was already the ack system of record); driver-only nodes register NullLdapGroupRoleMappingService so LDAP group→role falls back to the Security:Ldap:GroupToRole appsettings baseline (DB grants were never delivered to driver nodes via config anyway). Docs updated: docs/Redundancy.md, docs/Configuration.md, CLAUDE.md. Deferred: Task 9 — dropping the now-unused ScriptedAlarmState ConfigDb table (a follow-up, not a blocker; the table is simply unread). Remaining (Tasks 10–11): rig config for a driver-only node with no ConfigDb connection string, and the live gate itself — a driver pair runs a full deploy + alarm + historian cycle with no ConfigDb configured at all, confirming ServiceLevel 240/250 while central SQL is down and grep-level proof no driver-branch service can resolve the ConfigDb context. Not yet run — treat Phase 4 as code-complete, not verified, until this lands. Exit gate: its own live gate — a driver pair runs a full deploy + alarm + historian cycle with no ConfigDb connection string configured at all; grep-level proof no driver-branch service can resolve the ConfigDb context.

Phase 5 — gRPC stream contract for live telemetry — code-complete on feat/mesh-phase5, live gate pending

Scope as shipped (narrowed from the program sketch's "seven observability topics," settled with the user 2026-07-23 — mirrors how Phases 1 and 3 also honestly scoped down): one server-streaming gRPC contract carrying a oneof event with four kinds, driver nodes host the gRPC server (dedicated Kestrel h2c port), central dials in — the inverted direction is the load-bearing ScadaBridge finding (design §2), required so telemetry survives the Phase 6 mesh split without shared gossip membership.

The four migrated channels:

DPS topic	Domain record	Central sink
alerts	AlarmTransitionEvent	IInProcessBroadcaster<AlarmTransitionEvent> (+ AlertHub)
script-logs	ScriptLogEntry	IInProcessBroadcaster<ScriptLogEntry> (+ ScriptLogHub)
driver-health	DriverHealthChanged	IDriverStatusSnapshotStore
driver-resilience-status	DriverResilienceStatusChanged	IDriverResilienceStatusStore

Three deferred, each with a settled reason (NOT migrated — do not read the program sketch's original "seven" as the delivered scope):

• redundancy-state — bidirectional, built from Cluster.State, pair-local control plane driving ServiceLevel + the Primary gate. Stays on DPS in every mode; under Phase 6 it works in-mesh per pair. Central's cross-pair display of redundancy is a Phase 6 concern, not a Phase 5 panel.
• fleet-status — central-internal; FleetStatusBroadcaster builds it from the admin node's own cluster membership events and Fleet.razor polls the Config DB, ignoring the feed. Not a node→central stream at all. Revisit in Phase 6 once central loses site-node gossip visibility.
• deployment-acks — already migrated, but onto the Phase 2 ClusterClient transport, as a command-plane reply, not an observability broadcast.

Dark switch: Telemetry:Mode (node/serve, default Dps) and TelemetryDial:Mode (central/dial, default Dps) flip independently to Grpc; both code paths are compiled into every binary, so flipping is a config + restart, not a redeploy. The node always hosts the gRPC server whenever Telemetry:GrpcListenPort > 0 and always emits into its node-local hub + publishes DPS, in both modes — only central's ingest source (TelemetryDial:Mode) actually switches which of the two always-available sources feeds the AdminUI sinks. The AdminUI panels/components are unchanged; only the sinks' upstream feed swaps.

Auth from day one, fail-closed: a shared node bearer key (Telemetry:ApiKey == TelemetryDial:ApiKey), gated by TelemetryStreamAuthInterceptor (CryptographicOperations.FixedTimeEquals, PermissionDenied on mismatch, unset key rejects every call). This supersedes design §6.3's "match ScadaBridge's unauthenticated posture for now" — ScadaBridge itself has since closed that gap with this identical interceptor pattern (see the supersede note added to design §6.3, and docs/Telemetry.md § Authentication).

Reconnect story: central's TelemetryDialSupervisor runs one reconnecting dialer per driver node, discovered from ClusterNode rows (Host+GrpcPort, refreshed every TelemetryDial:ContactRefreshSeconds); immediate first retry then ~5s fixed backoff, generation-stamped so a superseded stream's late error/event is ignored; the dialer never permanently gives up (observability, not data plane). The node-local hub replays last-value snapshots for driver-health/driver-resilience-status on every (re)subscribe so those stores re-prime immediately; alerts/script-logs are append logs and tolerate the gap. The /alerts//script-log pill reflects aggregate stream health (live when ≥1 node stream is up), matching today's DPS pill semantics.

Docs shipped: docs/Telemetry.md (new, canonical guide), docs/Configuration.md § Telemetry/TelemetryDial, docs/Redundancy.md cross-reference, this section, and the design §6.3 supersede note.

Exit gate (not yet run): all AdminUI live panels green against the docker-dev rig with Telemetry:Mode=Grpc/TelemetryDial:Mode=Grpc and the four DPS telemetry bridges NOT spawned; kill-and-reconnect of a site node's dialer recovers every stream (driver-health/resilience re-prime from the node hub's snapshot replay). Treat Phase 5 as code-complete, not verified, until this lands — see docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md Task 12.

Phase 6 — Mesh partition + co-location topology

Scope: per-cluster seed nodes — adopt ScadaBridge's self-first ordering and RETIRE the SelfFormAfter watchdog + TCP reachability guard DONE EARLY 2026-07-22 (docker-dev central-2 swapped to self-first, ClusterBootstrapFallback/SelfFormAfter deleted, AkkaClusterOptionsValidator ports ScadaBridge's startup rule in its conditional form, and SelfFirstSeedBootstrapTests replaces SelfFormBootstrapTests; see docs/Redundancy.md §"Bootstrap: self-first seed ordering"). What remains for this phase is the per-pair Cluster__SeedNodes__* matrix once the meshes actually split — every node in a pair is then a seed of its own mesh, so the site-node exemption disappears; cluster-scoped roles cluster-{ClusterId} + singleton re-scoping, central pair keeps the admin singletons; docker-dev rig rewritten to model the real topology — including the co-location port table above (both products' compose files on shared per-site networks, real LocalDb sync ports); remove the ClusterRedundancy page's mesh-scope caveat (the election is pair-local now) and the fallback's site-node island-guard docs note (moot — every node is a seed of its own mesh); Cluster__SeedNodes__* env matrix per pair. Exit gate: rig up in the new shape; every existing live-gated behavior re-verified per pair (deploy, redundancy 250/240 per pair — two Primaries fleet-wide, one per pair, by design); secrets Akka replication re-verified or re-scoped (it rides DPS on the current single mesh — its topology must be re-decided here, likely SQL-hub mode like ScadaBridge, since pub/sub cannot cross separate meshes).

Phase 7 — Failover drill + live gates

Scope: the drill ScadaBridge already has (failover-drill.sh analogue) run per pair, both directions; close the two outstanding live gates: (a) auto-down 1-vs-1 crash-the-oldest (deferred since Phase 0a — finally testable, every mesh is exactly two nodes), (b) self-first lone-cold-start on the real per-pair topology; manual-failover button re-verified per pair; operator runbook for the co-located site (one page covering both products' failover on the same two VMs). Exit gate: drill green on every pair type (central, site); runbook merged; design doc §7 table fully marked DONE. (Live gate (b) is now the self-first cold-start-alone drill, not SelfFormAfter — the watchdog it named was retired 2026-07-22.)

---

Risks carried from the design (§8, unchanged — re-read before each phase plan)

LocalDb becomes load-bearing for config (blast radius of the #485 class rises); DbHealthProbeActor feeds a client-visible value; the rig models the doomed topology until Phase 6; losing gossip loses free observability (7 panels); 128 KB ClusterClient frame drop is silent unless both knobs are set; never stack app-level LWW on LocalDb's HLC. New (this program): co-located VMs mean a VM loss now takes out one node of BOTH products at once — the drill in Phase 7 must include the shared-VM failure (both products fail over together), and resource sizing on the site VMs should be checked once both products run the full stack.

Tracking

Phase	Status
0a downing strategy	DONE 2026-07-21 (live gate → Phase 7)
0b oldest-Up election	DONE 2026-07-21
Prereq: selfform-fallback + manual-failover plan	DONE 2026-07-22, merged a78425ea. Shipped not as the planned SelfFormAfter watchdog — the live gate caught that islanding a manually-failed-over node — but as self-first seed ordering + AkkaClusterOptionsValidator. Phase 6's seed-ordering scope is therefore already discharged.
1 ClusterNode columns + DB ack set	DONE 2026-07-22, merged 7654f24d. Deploy-seal semantics changed as flagged, but the escape hatch is ClusterNode.MaintenanceMode, not Enabled=0 — the live gate found Enabled=0 is rejected by DraftValidator.ValidateClusterTopology on any 2-node pair. Gate record: 2026-07-22-mesh-phase1-live-gate.md.
2 ClusterClient transport	DONE 2026-07-22 (live gate PASSED, shipped dark) — shipped dark (MeshTransport:Mode=Dps default), both comm actors registered in both modes. Corrections: SendToAll not Send, one fleet-wide client not one per cluster, /user/node-communication not /user/cluster-communication. The gate's "Ask timing out" item does not exist to test — no cross-boundary Ask in this phase; see the phase section.
3 fetch-and-cache	DONE 2026-07-23, merged d01b0695, live gate PASSED, pushed to origin (scadaproj umbrella index updated + pushed b5e7bc8). gRPC fetch RPC + shared node key; dark switch ConfigSource:Mode (Direct default). Rig flip: OTOPCUA_CONFIG_MODE=FetchAndCache on the site nodes. See 2026-07-22-mesh-phase3-config-fetch-and-cache.md.
4 cut driver ConfigDb	DONE 2026-07-23 — live gate PASSED on feat/mesh-phase4 (Tasks 0–8 + 1b + 10–11; Task 9 table-drop deferred). ConfigDb admin-only; driver-only ⇒ FetchAndCache (validator); DbHealthProbeActor not spawned driver-only (client-visible ServiceLevel 240/250 with central SQL down — proven live); alarm condition state in replicated LocalDb alarm_condition_state; central persists acks; OpcUaPublish guard split fixes the driver-only address-space wipe; driver-only LDAP maps from appsettings only (6th consumer found mid-phase). Gate: deploy sealed green w/ 4 DB-less site nodes acking, ServiceLevel held 240 w/ SQL down, restart booted last-known-good from the LocalDb pointer. See 2026-07-23-mesh-phase4-cut-driver-configdb.md + 2026-07-23-mesh-phase4-live-gate.md.
5 gRPC telemetry	code-complete (live gate pending) on feat/mesh-phase5. 4-channel scope (alerts/script-logs/driver-health/driver-resilience-status), 3 deferred with rationale (redundancy-state, fleet-status, deployment-acks); dark switch Telemetry:Mode/TelemetryDial:Mode (Dps default); auth-from-day-one fail-closed bearer key, superseding design §6.3. See docs/Telemetry.md + 2026-07-23-mesh-phase5-grpc-telemetry-stream.md.
6 mesh partition + co-location	not started
7 drill + live gates	not started