Joseph Doherty
1ec831883c
Phase 2 of the per-cluster mesh program: move the three central->node command channels (deployments, driver-control, alarm-commands) and the deployment-acks reply channel off DistributedPubSub onto Akka ClusterClient, behind a config flag. Recon of both repos turned up three corrections to the design doc's account of the sister project, all folded into the plan: - Design doc S2 claims "every unhandled message replies with a typed failure". ScadaBridge has no ReceiveAny/Unhandled override in either comm actor; the typed-failure idiom fires only for a MISSING REGISTRATION, per message type. - "No central buffering" was never a setting they wrote. They have zero akka.cluster.client HOCON and run the defaults (buffer-size 1000, reconnect-timeout off). We choose buffer-size = 0 deliberately. - Publish -> Send is the wrong substitution. Today's deploy notify is a broadcast every DriverHostActor receives (no ClusterId filter exists on the node side), so it must become SendToAll. Send would deploy to exactly one node of the fleet and seal green on partial acks. Also records the single-mesh duplicate-delivery trap (one ClusterClient in Phase 2, not one per Cluster -- a receptionist serves its whole mesh) and one deviation from the program plan's exit gate: there is no cross-boundary Ask in Phase 2, because every migrated command is fire-and-forget with a local reply, so "an Ask timing out cleanly" cannot be run as written. Marks the program tracking tables current: prereq + Phase 1 are DONE (they still read "not executed"), Phase 2 in progress. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
15 KiB
Per-Cluster Mesh Program — Align OtOpcUa's Akka Topology with ScadaBridge
For Claude: This is a PROGRAM plan (phase roadmap + gates), not a bite-sized task plan. The authoritative design is
docs/plans/2026-07-21-per-cluster-mesh-design.md(decisions settled 2026-07-21; Phases 0a/0b DONE). Per that design's own instruction, each phase gets its own detailed plan, written when the phase starts (superpowers writing-plans → executing-plans), so plans are authored against current code, not against a forecast. This document sequences the phases, fixes the deployment topology (co-location with ScadaBridge — NEW constraint 2026-07-22), and defines each phase's entry/exit gates.
Goal: OtOpcUa runs the same Akka.NET mesh shape as ScadaBridge — one Akka mesh per
application Cluster, two nodes max, central ↔ cluster joined by explicit transports
(ClusterClient + gRPC + fetch), driver nodes holding no ConfigDb connection — so that every
Primary-gated decision and every gated resource share one pair-local scope, and both products
present one operational model on the shared site hardware.
Why now (2026-07-22): OtOpcUa cluster pairs will run on the same two Windows VMs as the ScadaBridge site nodes. That deployment makes the current single-fleet-mesh design actively wrong for sites (a site's OtOpcUa nodes would gossip across the WAN to central and elect ONE Primary fleet-wide), and makes the ScadaBridge shape the obviously correct one: each site's two VMs host two independent, identically-postured 2-node clusters (one per product), surviving alone on the site LAN. It also means driver nodes must not depend on reaching central SQL — sites have no SQL Server, so §6.1's "driver nodes never connect to the ConfigDb" stops being an architectural preference and becomes a deployment requirement.
Deployment topology (the co-location constraint, NEW)
Each site: 2 Windows VMs. Each VM runs one ScadaBridge site node AND one OtOpcUa driver node.
Two independent 2-node Akka clusters per site — they share hardware, never a mesh. Central:
2 Windows VMs, each running a ScadaBridge central node and an OtOpcUa central node
(admin,driver), again as separate pairs.
Per-VM port allocation (no collisions — verify against actual deployment configs in Phase 6):
| Port | Owner | Purpose |
|---|---|---|
| 8081 / 8082 | ScadaBridge | Akka remoting (central / site) |
| 8083 | ScadaBridge site | gRPC h2c (streams + LocalDb sync) |
| 8084 | ScadaBridge site | metrics |
| 5000 (+Traefik) | ScadaBridge central | UI + Inbound API |
| 4053 | OtOpcUa | Akka remoting |
| 4840 | OtOpcUa | OPC UA endpoint |
| OtOpcUa AdminUI port | OtOpcUa central (admin) | AdminUI (driver-only site nodes host no UI) |
| OtOpcUa LocalDb sync port | OtOpcUa driver | h2c LocalDb pair replication (default off/0 today — Phase 6 assigns a real per-site port) |
Aligned HA posture (both products, per VM — already true or landing via the selfform plan):
auto-down downing (15 s window), oldest-Up active/primary election, self-first seed ordering
(the SelfFormAfter self-form fallback that briefly stood in for it was retired 2026-07-22),
termination-watchdog → process exit → sc.exe failure restart recovery.
One failover story for operators regardless of product.
Prerequisite ordering: the fallback/manual-failover plan
(docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md) executes before this
program — it is small, independent, and Phase 6 depends on its semantics (under per-pair meshes
every node lists itself + partner as seeds, so the fallback covers both nodes of every pair;
the site-node island guard then simply never triggers).
What changes, in one table (from the design doc)
| Aspect | Today (single fleet mesh) | Target (ScadaBridge shape) |
|---|---|---|
| Mesh | one gossip ring, all nodes, seeded by central-1 | one 2-node mesh per application Cluster; same ActorSystem name; separation by seed-node partitioning |
| Roles | admin / driver fleet-wide |
driver + cluster-specific cluster-{ClusterId}; singletons scoped to the cluster role |
| Primary election | mesh-wide oldest Up driver (Phase 0b) | pair-local by construction — same rule, correct scope |
| Command/control | 9 DPS topics + singleton over gossip | ClusterClient (one receptionist actor per side); central discovers nodes from ClusterNode rows |
| Deploy | notify (DPS) + node fetches from ConfigDb | notify via ClusterClient; artifact fetched from central, cached in LocalDb |
| Driver ConfigDb connection | direct EF connection to central SQL | none — LocalDb is the steady-state config store |
| Live telemetry | 7 observability DPS topics | one gRPC stream contract (oneof event), central dials each cluster node |
| Rig | six-node single mesh (docker-dev) |
per-cluster meshes; rig models the real topology |
Phases
Each phase below is one row of design-doc §7, expanded with entry/exit gates. Execution recipe
per phase: (1) invoke writing-plans in this repo to produce
docs/plans/2026-07-2X-mesh-phase-N-<name>.md from the scope notes here + the design doc §
references, exploring current code first; (2) execute it task-by-task; (3) run the phase's exit
gate; (4) update this file's status column and the design doc's §7 table.
Phase 1 — ClusterNode address columns + DB-sourced ack set — DONE 2026-07-22
Shipped per 2026-07-21-per-cluster-mesh-phase1.md; see that plan's Task 6 gate record and the
design doc's "Phase 1 as shipped" note for the two scope deviations (no cluster-scope filtering — no
such deployment exists; no per-node role column) and the one addition (ClusterNodeAddressReconciler).
AdminUI node edit was deferred to Phase 2 — nothing reads the columns until then, and the
migration default plus the rig seed cover every node today.
Scope: ClusterNode gains Akka + gRPC address columns (mirroring ScadaBridge's Site
entity NodeAAddress/GrpcNodeAAddress pattern, but per-node rows); EF migration; AdminUI node
edit surfaces the fields; ConfigPublishCoordinator derives its expected-ack set from
ClusterNode rows instead of Akka.Cluster.State.Members filtered by role (design §3 fact 3 —
this removes the coordinator's one genuinely mesh-bound dependency).
Independent of the split: yes — safe on the current mesh.
Exit gate: deploy on the unchanged docker-dev rig completes with the coordinator's expected-ack
set proven DB-sourced (test: a ClusterNode row present but node down → deploy reports that node
missing; a node up but row absent → its ack is not expected).
Phase 2 — Comm actors + ClusterClient transport
Scope: one receptionist-registered actor per side (/user/central-communication,
/user/cluster-communication), registered per node, not as a singleton (contact rotation);
ClusterClient central → cluster carrying deploy notify + acks + driver-control; the ScadaBridge
idioms copied verbatim: sender-preserving Tell(new ClusterClient.Send(...), Sender) Ask relay,
typed-failure reply for every unhandled message, no central buffering toward unreachable clusters
(drop + warn), central discovers contacts from Phase 1's ClusterNode rows (60 s refresh +
admin-change refresh), clusters know central from appsettings (static, restart to change).
Frame-size guard: anything carrying payload sets both the frame limit and
log-frame-size-exceeding (design §8) — the deploy path stays payload-free by design.
Exit gate: on the still-single-mesh rig, deploy notify + acks and AdminUI Reconnect/Restart
flow over ClusterClient (DPS paths deleted or dark-switched), including an Ask timing out cleanly
against a stopped node.
Phase 3 — Config fetch-and-cache from central
Scope: the deploy artifact is served by central (transport per its own phase plan — the ScadaBridge analogue is token-gated HTTP; decide HTTP vs a gRPC fetch RPC when planning) and cached in LocalDb; driver nodes read config exclusively from LocalDb (boot-from-cache becomes the normal path — design §6.1); chunking, SHA-256 verify, newest-2 retention, pair replication carry over unchanged. This phase changes a running data path — it gets its own live gate (design §7 note): deploy lands on a driver pair with central SQL stopped mid-fetch → retry lands; #485 last-known-good semantics re-proven on the new path. Exit gate: live gate green on the rig; a driver node with an empty LocalDb and reachable central boots into the current config; with central down it boots last-known-good.
Phase 4 — Cut the driver-side ConfigDb connection
Scope (design §6.1 audit table): re-home EfAlarmConditionStateStore to LocalDb (pair-local
state, same journey as the Phase-2 alarm S&F buffer); resolve the DbHealthProbeActor
question — driver nodes have no DB to probe, and DB health currently feeds ServiceLevel tiering,
so define the replacement health input (candidate: central-reachability via the Phase 2/3
transports) — this is a client-visible ServiceLevel semantics change and must be documented in
docs/Redundancy.md + the interop playbook; audit OpcUaPublishActor's ConfigDb use (TBD in the
design) and re-source it; registration cleanup in ServiceCollectionExtensions; driver-role
Program.cs branch registers no EF ConfigDb context at all (mirror ScadaBridge's central-only
AddConfigurationDatabase).
Exit gate: its own live gate — a driver pair runs a full deploy + alarm + historian cycle with
no ConfigDb connection string configured at all; grep-level proof no driver-branch service can
resolve the ConfigDb context.
Phase 5 — gRPC stream contract for live telemetry
Scope: one server-streaming contract carrying a oneof event, cluster nodes host the gRPC
server, central dials in (the inverted direction is the load-bearing ScadaBridge finding —
design §2); migrate the seven observability topics (alerts, driver-health,
driver-resilience-status, fleet-status, script-logs, plus redundancy-state distribution and
deployment-acks if Phase 2 left them on DPS); additive-only field evolution, contract locked by
test; per-panel reconnect story for the AdminUI (design §8 — losing gossip loses free fleet
observability).
Exit gate: all AdminUI live panels green against a pair with DPS telemetry topics deleted;
kill-and-reconnect of the central dialer recovers every stream.
Phase 6 — Mesh partition + co-location topology
Scope: per-cluster seed nodes — adopt ScadaBridge's self-first ordering and RETIRE the
DONE EARLY 2026-07-22 (docker-dev
SelfFormAfter watchdog + TCP reachability guardcentral-2 swapped to self-first, ClusterBootstrapFallback/SelfFormAfter deleted,
AkkaClusterOptionsValidator ports ScadaBridge's startup rule in its conditional form, and
SelfFirstSeedBootstrapTests replaces SelfFormBootstrapTests; see docs/Redundancy.md
§"Bootstrap: self-first seed ordering"). What remains for this phase is the per-pair
Cluster__SeedNodes__* matrix once the meshes actually split — every node in a pair is then a seed
of its own mesh, so the site-node exemption disappears; cluster-scoped roles cluster-{ClusterId} + singleton re-scoping,
central pair keeps the admin singletons; docker-dev rig rewritten to model the real topology —
including the co-location port table above (both products' compose files on shared per-site
networks, real LocalDb sync ports); remove the ClusterRedundancy page's mesh-scope caveat (the
election is pair-local now) and the fallback's site-node island-guard docs note (moot — every
node is a seed of its own mesh); Cluster__SeedNodes__* env matrix per pair.
Exit gate: rig up in the new shape; every existing live-gated behavior re-verified per pair
(deploy, redundancy 250/240 per pair — two Primaries fleet-wide, one per pair, by design);
secrets Akka replication re-verified or re-scoped (it rides DPS on the current single mesh — its
topology must be re-decided here, likely SQL-hub mode like ScadaBridge, since pub/sub cannot
cross separate meshes).
Phase 7 — Failover drill + live gates
Scope: the drill ScadaBridge already has (failover-drill.sh analogue) run per pair, both
directions; close the two outstanding live gates: (a) auto-down 1-vs-1 crash-the-oldest
(deferred since Phase 0a — finally testable, every mesh is exactly two nodes), (b) self-first
lone-cold-start on the real per-pair topology; manual-failover button re-verified per pair;
operator runbook for the co-located site (one page covering both products' failover on the same
two VMs).
Exit gate: drill green on every pair type (central, site); runbook merged; design doc §7
table fully marked DONE. (Live gate (b) is now the self-first cold-start-alone drill, not
SelfFormAfter — the watchdog it named was retired 2026-07-22.)
Risks carried from the design (§8, unchanged — re-read before each phase plan)
LocalDb becomes load-bearing for config (blast radius of the #485 class rises);
DbHealthProbeActor feeds a client-visible value; the rig models the doomed topology until
Phase 6; losing gossip loses free observability (7 panels); 128 KB ClusterClient frame drop is
silent unless both knobs are set; never stack app-level LWW on LocalDb's HLC. New (this
program): co-located VMs mean a VM loss now takes out one node of BOTH products at once — the
drill in Phase 7 must include the shared-VM failure (both products fail over together), and
resource sizing on the site VMs should be checked once both products run the full stack.
Tracking
| Phase | Status |
|---|---|
| 0a downing strategy | DONE 2026-07-21 (live gate → Phase 7) |
| 0b oldest-Up election | DONE 2026-07-21 |
| Prereq: selfform-fallback + manual-failover plan | DONE 2026-07-22, merged a78425ea. Shipped not as the planned SelfFormAfter watchdog — the live gate caught that islanding a manually-failed-over node — but as self-first seed ordering + AkkaClusterOptionsValidator. Phase 6's seed-ordering scope is therefore already discharged. |
| 1 ClusterNode columns + DB ack set | DONE 2026-07-22, merged 7654f24d. Deploy-seal semantics changed as flagged, but the escape hatch is ClusterNode.MaintenanceMode, not Enabled=0 — the live gate found Enabled=0 is rejected by DraftValidator.ValidateClusterTopology on any 2-node pair. Gate record: 2026-07-22-mesh-phase1-live-gate.md. |
| 2 ClusterClient transport | IN PROGRESS — plan being written 2026-07-22 |
| 3 fetch-and-cache | not started |
| 4 cut driver ConfigDb | not started |
| 5 gRPC telemetry | not started |
| 6 mesh partition + co-location | not started |
| 7 drill + live gates | not started |