Joseph Doherty
9af935f237
docs/Redundancy.md's bootstrap section is rewritten around the mechanism that is
actually in force: which of Akka's two bootstrap processes runs is decided by
whether seed-nodes[0] is this node's own address, so the ORDER of Cluster:SeedNodes
is the fix, not a timer. Adds the per-process table, the shipped self-first
example, why the validator is conditional (site nodes are not seeds) and why it
matches PublicHostname rather than the 0.0.0.0 bind address.
The retirement is documented rather than erased: a new subsection explains that a
watchdog outside the join handshake cannot tell "no seed answered" from "a seed
answered and the join is in flight", and the ea45ace1 live-gate finding is kept
verbatim as the evidence that produced that conclusion (InitJoinAck at 10:49:05,
old incarnation retired 10:49:06, watchdog fired 10:49:15, second cluster). The
watchdog's own earlier PASS is kept too — it did pass what it was gated on.
Also: a live gate for the NEW mechanism is registered as outstanding (not
re-drilled on the docker-dev rig since the swap), and the plan doc that shipped
the watchdog gets a superseded banner rather than an edit, so the execution record
and its Task 8 finding stay readable.
Ripple: CLAUDE.md cluster section, docs/Configuration.md + docs/v2/Cluster.md +
docs/ServiceHosting.md SeedNodes entries, the per-cluster-mesh design doc's two
"CLOSED by SelfFormAfter" notes, and mesh-program Phase 6 (which had this
convergence queued — now marked done early) + Phase 7's live-gate name.
14 KiB
Per-Cluster Mesh Program — Align OtOpcUa's Akka Topology with ScadaBridge
For Claude: This is a PROGRAM plan (phase roadmap + gates), not a bite-sized task plan. The authoritative design is
docs/plans/2026-07-21-per-cluster-mesh-design.md(decisions settled 2026-07-21; Phases 0a/0b DONE). Per that design's own instruction, each phase gets its own detailed plan, written when the phase starts (superpowers writing-plans → executing-plans), so plans are authored against current code, not against a forecast. This document sequences the phases, fixes the deployment topology (co-location with ScadaBridge — NEW constraint 2026-07-22), and defines each phase's entry/exit gates.
Goal: OtOpcUa runs the same Akka.NET mesh shape as ScadaBridge — one Akka mesh per
application Cluster, two nodes max, central ↔ cluster joined by explicit transports
(ClusterClient + gRPC + fetch), driver nodes holding no ConfigDb connection — so that every
Primary-gated decision and every gated resource share one pair-local scope, and both products
present one operational model on the shared site hardware.
Why now (2026-07-22): OtOpcUa cluster pairs will run on the same two Windows VMs as the ScadaBridge site nodes. That deployment makes the current single-fleet-mesh design actively wrong for sites (a site's OtOpcUa nodes would gossip across the WAN to central and elect ONE Primary fleet-wide), and makes the ScadaBridge shape the obviously correct one: each site's two VMs host two independent, identically-postured 2-node clusters (one per product), surviving alone on the site LAN. It also means driver nodes must not depend on reaching central SQL — sites have no SQL Server, so §6.1's "driver nodes never connect to the ConfigDb" stops being an architectural preference and becomes a deployment requirement.
Deployment topology (the co-location constraint, NEW)
Each site: 2 Windows VMs. Each VM runs one ScadaBridge site node AND one OtOpcUa driver node.
Two independent 2-node Akka clusters per site — they share hardware, never a mesh. Central:
2 Windows VMs, each running a ScadaBridge central node and an OtOpcUa central node
(admin,driver), again as separate pairs.
Per-VM port allocation (no collisions — verify against actual deployment configs in Phase 6):
| Port | Owner | Purpose |
|---|---|---|
| 8081 / 8082 | ScadaBridge | Akka remoting (central / site) |
| 8083 | ScadaBridge site | gRPC h2c (streams + LocalDb sync) |
| 8084 | ScadaBridge site | metrics |
| 5000 (+Traefik) | ScadaBridge central | UI + Inbound API |
| 4053 | OtOpcUa | Akka remoting |
| 4840 | OtOpcUa | OPC UA endpoint |
| OtOpcUa AdminUI port | OtOpcUa central (admin) | AdminUI (driver-only site nodes host no UI) |
| OtOpcUa LocalDb sync port | OtOpcUa driver | h2c LocalDb pair replication (default off/0 today — Phase 6 assigns a real per-site port) |
Aligned HA posture (both products, per VM — already true or landing via the selfform plan):
auto-down downing (15 s window), oldest-Up active/primary election, self-first seed ordering
(the SelfFormAfter self-form fallback that briefly stood in for it was retired 2026-07-22),
termination-watchdog → process exit → sc.exe failure restart recovery.
One failover story for operators regardless of product.
Prerequisite ordering: the fallback/manual-failover plan
(docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md) executes before this
program — it is small, independent, and Phase 6 depends on its semantics (under per-pair meshes
every node lists itself + partner as seeds, so the fallback covers both nodes of every pair;
the site-node island guard then simply never triggers).
What changes, in one table (from the design doc)
| Aspect | Today (single fleet mesh) | Target (ScadaBridge shape) |
|---|---|---|
| Mesh | one gossip ring, all nodes, seeded by central-1 | one 2-node mesh per application Cluster; same ActorSystem name; separation by seed-node partitioning |
| Roles | admin / driver fleet-wide |
driver + cluster-specific cluster-{ClusterId}; singletons scoped to the cluster role |
| Primary election | mesh-wide oldest Up driver (Phase 0b) | pair-local by construction — same rule, correct scope |
| Command/control | 9 DPS topics + singleton over gossip | ClusterClient (one receptionist actor per side); central discovers nodes from ClusterNode rows |
| Deploy | notify (DPS) + node fetches from ConfigDb | notify via ClusterClient; artifact fetched from central, cached in LocalDb |
| Driver ConfigDb connection | direct EF connection to central SQL | none — LocalDb is the steady-state config store |
| Live telemetry | 7 observability DPS topics | one gRPC stream contract (oneof event), central dials each cluster node |
| Rig | six-node single mesh (docker-dev) |
per-cluster meshes; rig models the real topology |
Phases
Each phase below is one row of design-doc §7, expanded with entry/exit gates. Execution recipe
per phase: (1) invoke writing-plans in this repo to produce
docs/plans/2026-07-2X-mesh-phase-N-<name>.md from the scope notes here + the design doc §
references, exploring current code first; (2) execute it task-by-task; (3) run the phase's exit
gate; (4) update this file's status column and the design doc's §7 table.
Phase 1 — ClusterNode address columns + DB-sourced ack set
Scope: ClusterNode gains Akka + gRPC address columns (mirroring ScadaBridge's Site
entity NodeAAddress/GrpcNodeAAddress pattern, but per-node rows); EF migration; AdminUI node
edit surfaces the fields; ConfigPublishCoordinator derives its expected-ack set from
ClusterNode rows instead of Akka.Cluster.State.Members filtered by role (design §3 fact 3 —
this removes the coordinator's one genuinely mesh-bound dependency).
Independent of the split: yes — safe on the current mesh.
Exit gate: deploy on the unchanged docker-dev rig completes with the coordinator's expected-ack
set proven DB-sourced (test: a ClusterNode row present but node down → deploy reports that node
missing; a node up but row absent → its ack is not expected).
Phase 2 — Comm actors + ClusterClient transport
Scope: one receptionist-registered actor per side (/user/central-communication,
/user/cluster-communication), registered per node, not as a singleton (contact rotation);
ClusterClient central → cluster carrying deploy notify + acks + driver-control; the ScadaBridge
idioms copied verbatim: sender-preserving Tell(new ClusterClient.Send(...), Sender) Ask relay,
typed-failure reply for every unhandled message, no central buffering toward unreachable clusters
(drop + warn), central discovers contacts from Phase 1's ClusterNode rows (60 s refresh +
admin-change refresh), clusters know central from appsettings (static, restart to change).
Frame-size guard: anything carrying payload sets both the frame limit and
log-frame-size-exceeding (design §8) — the deploy path stays payload-free by design.
Exit gate: on the still-single-mesh rig, deploy notify + acks and AdminUI Reconnect/Restart
flow over ClusterClient (DPS paths deleted or dark-switched), including an Ask timing out cleanly
against a stopped node.
Phase 3 — Config fetch-and-cache from central
Scope: the deploy artifact is served by central (transport per its own phase plan — the ScadaBridge analogue is token-gated HTTP; decide HTTP vs a gRPC fetch RPC when planning) and cached in LocalDb; driver nodes read config exclusively from LocalDb (boot-from-cache becomes the normal path — design §6.1); chunking, SHA-256 verify, newest-2 retention, pair replication carry over unchanged. This phase changes a running data path — it gets its own live gate (design §7 note): deploy lands on a driver pair with central SQL stopped mid-fetch → retry lands; #485 last-known-good semantics re-proven on the new path. Exit gate: live gate green on the rig; a driver node with an empty LocalDb and reachable central boots into the current config; with central down it boots last-known-good.
Phase 4 — Cut the driver-side ConfigDb connection
Scope (design §6.1 audit table): re-home EfAlarmConditionStateStore to LocalDb (pair-local
state, same journey as the Phase-2 alarm S&F buffer); resolve the DbHealthProbeActor
question — driver nodes have no DB to probe, and DB health currently feeds ServiceLevel tiering,
so define the replacement health input (candidate: central-reachability via the Phase 2/3
transports) — this is a client-visible ServiceLevel semantics change and must be documented in
docs/Redundancy.md + the interop playbook; audit OpcUaPublishActor's ConfigDb use (TBD in the
design) and re-source it; registration cleanup in ServiceCollectionExtensions; driver-role
Program.cs branch registers no EF ConfigDb context at all (mirror ScadaBridge's central-only
AddConfigurationDatabase).
Exit gate: its own live gate — a driver pair runs a full deploy + alarm + historian cycle with
no ConfigDb connection string configured at all; grep-level proof no driver-branch service can
resolve the ConfigDb context.
Phase 5 — gRPC stream contract for live telemetry
Scope: one server-streaming contract carrying a oneof event, cluster nodes host the gRPC
server, central dials in (the inverted direction is the load-bearing ScadaBridge finding —
design §2); migrate the seven observability topics (alerts, driver-health,
driver-resilience-status, fleet-status, script-logs, plus redundancy-state distribution and
deployment-acks if Phase 2 left them on DPS); additive-only field evolution, contract locked by
test; per-panel reconnect story for the AdminUI (design §8 — losing gossip loses free fleet
observability).
Exit gate: all AdminUI live panels green against a pair with DPS telemetry topics deleted;
kill-and-reconnect of the central dialer recovers every stream.
Phase 6 — Mesh partition + co-location topology
Scope: per-cluster seed nodes — adopt ScadaBridge's self-first ordering and RETIRE the
DONE EARLY 2026-07-22 (docker-dev
SelfFormAfter watchdog + TCP reachability guardcentral-2 swapped to self-first, ClusterBootstrapFallback/SelfFormAfter deleted,
AkkaClusterOptionsValidator ports ScadaBridge's startup rule in its conditional form, and
SelfFirstSeedBootstrapTests replaces SelfFormBootstrapTests; see docs/Redundancy.md
§"Bootstrap: self-first seed ordering"). What remains for this phase is the per-pair
Cluster__SeedNodes__* matrix once the meshes actually split — every node in a pair is then a seed
of its own mesh, so the site-node exemption disappears; cluster-scoped roles cluster-{ClusterId} + singleton re-scoping,
central pair keeps the admin singletons; docker-dev rig rewritten to model the real topology —
including the co-location port table above (both products' compose files on shared per-site
networks, real LocalDb sync ports); remove the ClusterRedundancy page's mesh-scope caveat (the
election is pair-local now) and the fallback's site-node island-guard docs note (moot — every
node is a seed of its own mesh); Cluster__SeedNodes__* env matrix per pair.
Exit gate: rig up in the new shape; every existing live-gated behavior re-verified per pair
(deploy, redundancy 250/240 per pair — two Primaries fleet-wide, one per pair, by design);
secrets Akka replication re-verified or re-scoped (it rides DPS on the current single mesh — its
topology must be re-decided here, likely SQL-hub mode like ScadaBridge, since pub/sub cannot
cross separate meshes).
Phase 7 — Failover drill + live gates
Scope: the drill ScadaBridge already has (failover-drill.sh analogue) run per pair, both
directions; close the two outstanding live gates: (a) auto-down 1-vs-1 crash-the-oldest
(deferred since Phase 0a — finally testable, every mesh is exactly two nodes), (b) self-first
lone-cold-start on the real per-pair topology; manual-failover button re-verified per pair;
operator runbook for the co-located site (one page covering both products' failover on the same
two VMs).
Exit gate: drill green on every pair type (central, site); runbook merged; design doc §7
table fully marked DONE. (Live gate (b) is now the self-first cold-start-alone drill, not
SelfFormAfter — the watchdog it named was retired 2026-07-22.)
Risks carried from the design (§8, unchanged — re-read before each phase plan)
LocalDb becomes load-bearing for config (blast radius of the #485 class rises);
DbHealthProbeActor feeds a client-visible value; the rig models the doomed topology until
Phase 6; losing gossip loses free observability (7 panels); 128 KB ClusterClient frame drop is
silent unless both knobs are set; never stack app-level LWW on LocalDb's HLC. New (this
program): co-located VMs mean a VM loss now takes out one node of BOTH products at once — the
drill in Phase 7 must include the shared-VM failure (both products fail over together), and
resource sizing on the site VMs should be checked once both products run the full stack.
Tracking
| Phase | Status |
|---|---|
| 0a downing strategy | DONE 2026-07-21 (live gate → Phase 7) |
| 0b oldest-Up election | DONE 2026-07-21 |
| Prereq: selfform-fallback + manual-failover plan | plan written 2026-07-22, not executed |
| 1 ClusterNode columns + DB ack set | not started — plan to be written |
| 2 ClusterClient transport | not started |
| 3 fetch-and-cache | not started |
| 4 cut driver ConfigDb | not started |
| 5 gRPC telemetry | not started |
| 6 mesh partition + co-location | not started |
| 7 drill + live gates | not started |