docs(redundancy): SelfFormAfter closes the seed-node bootstrap gap
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,194 @@
|
||||
# Per-Cluster Mesh Program — Align OtOpcUa's Akka Topology with ScadaBridge
|
||||
|
||||
> **For Claude:** This is a PROGRAM plan (phase roadmap + gates), not a bite-sized task plan.
|
||||
> The authoritative design is `docs/plans/2026-07-21-per-cluster-mesh-design.md` (decisions settled
|
||||
> 2026-07-21; Phases 0a/0b DONE). Per that design's own instruction, **each phase gets its own
|
||||
> detailed plan, written when the phase starts** (superpowers writing-plans → executing-plans),
|
||||
> so plans are authored against current code, not against a forecast. This document sequences the
|
||||
> phases, fixes the deployment topology (co-location with ScadaBridge — NEW constraint 2026-07-22),
|
||||
> and defines each phase's entry/exit gates.
|
||||
|
||||
**Goal:** OtOpcUa runs the same Akka.NET mesh shape as ScadaBridge — **one Akka mesh per
|
||||
application `Cluster`, two nodes max**, central ↔ cluster joined by explicit transports
|
||||
(ClusterClient + gRPC + fetch), driver nodes holding **no ConfigDb connection** — so that every
|
||||
Primary-gated decision and every gated resource share one pair-local scope, and both products
|
||||
present one operational model on the shared site hardware.
|
||||
|
||||
**Why now (2026-07-22):** OtOpcUa cluster pairs will run **on the same two Windows VMs as the
|
||||
ScadaBridge site nodes**. That deployment makes the current single-fleet-mesh design actively
|
||||
wrong for sites (a site's OtOpcUa nodes would gossip across the WAN to central and elect ONE
|
||||
Primary fleet-wide), and makes the ScadaBridge shape the obviously correct one: each site's two
|
||||
VMs host two independent, identically-postured 2-node clusters (one per product), surviving alone
|
||||
on the site LAN. It also means driver nodes must not depend on reaching central SQL — **sites
|
||||
have no SQL Server**, so §6.1's "driver nodes never connect to the ConfigDb" stops being an
|
||||
architectural preference and becomes a deployment requirement.
|
||||
|
||||
---
|
||||
|
||||
## Deployment topology (the co-location constraint, NEW)
|
||||
|
||||
Each **site**: 2 Windows VMs. Each VM runs one ScadaBridge site node AND one OtOpcUa driver node.
|
||||
Two independent 2-node Akka clusters per site — they share hardware, never a mesh. **Central**:
|
||||
2 Windows VMs, each running a ScadaBridge central node and an OtOpcUa central node
|
||||
(`admin,driver`), again as separate pairs.
|
||||
|
||||
**Per-VM port allocation (no collisions — verify against actual deployment configs in Phase 6):**
|
||||
|
||||
| Port | Owner | Purpose |
|
||||
|---|---|---|
|
||||
| 8081 / 8082 | ScadaBridge | Akka remoting (central / site) |
|
||||
| 8083 | ScadaBridge site | gRPC h2c (streams + LocalDb sync) |
|
||||
| 8084 | ScadaBridge site | metrics |
|
||||
| 5000 (+Traefik) | ScadaBridge central | UI + Inbound API |
|
||||
| **4053** | OtOpcUa | Akka remoting |
|
||||
| **4840** | OtOpcUa | OPC UA endpoint |
|
||||
| OtOpcUa AdminUI port | OtOpcUa central (admin) | AdminUI (driver-only site nodes host no UI) |
|
||||
| OtOpcUa LocalDb sync port | OtOpcUa driver | h2c LocalDb pair replication (default off/0 today — Phase 6 assigns a real per-site port) |
|
||||
|
||||
**Aligned HA posture (both products, per VM — already true or landing via the selfform plan):**
|
||||
auto-down downing (15 s window), oldest-Up active/primary election, `SelfFormAfter` 10 s
|
||||
self-form fallback, termination-watchdog → process exit → `sc.exe failure` restart recovery.
|
||||
One failover story for operators regardless of product.
|
||||
|
||||
**Prerequisite ordering:** the fallback/manual-failover plan
|
||||
(`docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`) executes **before** this
|
||||
program — it is small, independent, and Phase 6 depends on its semantics (under per-pair meshes
|
||||
every node lists itself + partner as seeds, so the fallback covers both nodes of every pair;
|
||||
the site-node island guard then simply never triggers).
|
||||
|
||||
---
|
||||
|
||||
## What changes, in one table (from the design doc)
|
||||
|
||||
| Aspect | Today (single fleet mesh) | Target (ScadaBridge shape) |
|
||||
|---|---|---|
|
||||
| Mesh | one gossip ring, all nodes, seeded by central-1 | one 2-node mesh per application `Cluster`; same ActorSystem name; separation by seed-node partitioning |
|
||||
| Roles | `admin` / `driver` fleet-wide | `driver` + cluster-specific `cluster-{ClusterId}`; singletons scoped to the cluster role |
|
||||
| Primary election | mesh-wide oldest Up driver (Phase 0b) | pair-local by construction — same rule, correct scope |
|
||||
| Command/control | 9 DPS topics + singleton over gossip | ClusterClient (one receptionist actor per side); central discovers nodes from `ClusterNode` rows |
|
||||
| Deploy | notify (DPS) + node fetches from ConfigDb | notify via ClusterClient; artifact fetched **from central**, cached in LocalDb |
|
||||
| Driver ConfigDb connection | direct EF connection to central SQL | **none** — LocalDb is the steady-state config store |
|
||||
| Live telemetry | 7 observability DPS topics | one gRPC stream contract (`oneof` event), central dials each cluster node |
|
||||
| Rig | six-node single mesh (`docker-dev`) | per-cluster meshes; rig models the real topology |
|
||||
|
||||
---
|
||||
|
||||
## Phases
|
||||
|
||||
Each phase below is one row of design-doc §7, expanded with entry/exit gates. **Execution recipe
|
||||
per phase:** (1) invoke writing-plans in this repo to produce
|
||||
`docs/plans/2026-07-2X-mesh-phase-N-<name>.md` from the scope notes here + the design doc §
|
||||
references, exploring current code first; (2) execute it task-by-task; (3) run the phase's exit
|
||||
gate; (4) update this file's status column and the design doc's §7 table.
|
||||
|
||||
### Phase 1 — `ClusterNode` address columns + DB-sourced ack set
|
||||
**Scope:** `ClusterNode` gains Akka + gRPC address columns (mirroring ScadaBridge's `Site`
|
||||
entity `NodeAAddress`/`GrpcNodeAAddress` pattern, but per-node rows); EF migration; AdminUI node
|
||||
edit surfaces the fields; `ConfigPublishCoordinator` derives its expected-ack set from
|
||||
`ClusterNode` rows instead of `Akka.Cluster.State.Members` filtered by role (design §3 fact 3 —
|
||||
this removes the coordinator's one genuinely mesh-bound dependency).
|
||||
**Independent of the split:** yes — safe on the current mesh.
|
||||
**Exit gate:** deploy on the unchanged docker-dev rig completes with the coordinator's expected-ack
|
||||
set proven DB-sourced (test: a `ClusterNode` row present but node down → deploy reports that node
|
||||
missing; a node up but row absent → its ack is not expected).
|
||||
|
||||
### Phase 2 — Comm actors + ClusterClient transport
|
||||
**Scope:** one receptionist-registered actor per side (`/user/central-communication`,
|
||||
`/user/cluster-communication`), registered **per node, not as a singleton** (contact rotation);
|
||||
ClusterClient central → cluster carrying deploy notify + acks + driver-control; the ScadaBridge
|
||||
idioms copied verbatim: sender-preserving `Tell(new ClusterClient.Send(...), Sender)` Ask relay,
|
||||
typed-failure reply for every unhandled message, no central buffering toward unreachable clusters
|
||||
(drop + warn), central discovers contacts from Phase 1's `ClusterNode` rows (60 s refresh +
|
||||
admin-change refresh), clusters know central from appsettings (static, restart to change).
|
||||
**Frame-size guard:** anything carrying payload sets both the frame limit and
|
||||
`log-frame-size-exceeding` (design §8) — the deploy path stays payload-free by design.
|
||||
**Exit gate:** on the still-single-mesh rig, deploy notify + acks and AdminUI Reconnect/Restart
|
||||
flow over ClusterClient (DPS paths deleted or dark-switched), including an Ask timing out cleanly
|
||||
against a stopped node.
|
||||
|
||||
### Phase 3 — Config fetch-and-cache from central
|
||||
**Scope:** the deploy artifact is served **by central** (transport per its own phase plan — the
|
||||
ScadaBridge analogue is token-gated HTTP; decide HTTP vs a gRPC fetch RPC when planning) and
|
||||
cached in LocalDb; driver nodes read config exclusively from LocalDb (boot-from-cache becomes the
|
||||
normal path — design §6.1); chunking, SHA-256 verify, newest-2 retention, pair replication carry
|
||||
over unchanged. **This phase changes a running data path — it gets its own live gate** (design §7
|
||||
note): deploy lands on a driver pair with central SQL stopped mid-fetch → retry lands; #485
|
||||
last-known-good semantics re-proven on the new path.
|
||||
**Exit gate:** live gate green on the rig; a driver node with an empty LocalDb and reachable
|
||||
central boots into the current config; with central down it boots last-known-good.
|
||||
|
||||
### Phase 4 — Cut the driver-side ConfigDb connection
|
||||
**Scope (design §6.1 audit table):** re-home `EfAlarmConditionStateStore` to LocalDb (pair-local
|
||||
state, same journey as the Phase-2 alarm S&F buffer); **resolve the `DbHealthProbeActor`
|
||||
question** — driver nodes have no DB to probe, and DB health currently feeds ServiceLevel tiering,
|
||||
so define the replacement health input (candidate: central-reachability via the Phase 2/3
|
||||
transports) — this is a client-visible ServiceLevel semantics change and must be documented in
|
||||
`docs/Redundancy.md` + the interop playbook; audit `OpcUaPublishActor`'s ConfigDb use (TBD in the
|
||||
design) and re-source it; registration cleanup in `ServiceCollectionExtensions`; driver-role
|
||||
`Program.cs` branch registers no EF ConfigDb context at all (mirror ScadaBridge's central-only
|
||||
`AddConfigurationDatabase`).
|
||||
**Exit gate:** its own live gate — a driver pair runs a full deploy + alarm + historian cycle with
|
||||
**no ConfigDb connection string configured at all**; grep-level proof no driver-branch service can
|
||||
resolve the ConfigDb context.
|
||||
|
||||
### Phase 5 — gRPC stream contract for live telemetry
|
||||
**Scope:** one server-streaming contract carrying a `oneof` event, **cluster nodes host the gRPC
|
||||
server, central dials in** (the inverted direction is the load-bearing ScadaBridge finding —
|
||||
design §2); migrate the seven observability topics (`alerts`, `driver-health`,
|
||||
`driver-resilience-status`, `fleet-status`, `script-logs`, plus redundancy-state distribution and
|
||||
deployment-acks if Phase 2 left them on DPS); additive-only field evolution, contract locked by
|
||||
test; per-panel reconnect story for the AdminUI (design §8 — losing gossip loses free fleet
|
||||
observability).
|
||||
**Exit gate:** all AdminUI live panels green against a pair with DPS telemetry topics deleted;
|
||||
kill-and-reconnect of the central dialer recovers every stream.
|
||||
|
||||
### Phase 6 — Mesh partition + co-location topology
|
||||
**Scope:** per-cluster seed nodes (each pair node lists itself + partner — the `SelfFormAfter`
|
||||
fallback then covers both), cluster-scoped roles `cluster-{ClusterId}` + singleton re-scoping,
|
||||
central pair keeps the admin singletons; **docker-dev rig rewritten** to model the real topology —
|
||||
including the co-location port table above (both products' compose files on shared per-site
|
||||
networks, real LocalDb sync ports); remove the ClusterRedundancy page's mesh-scope caveat (the
|
||||
election is pair-local now) and the fallback's site-node island-guard docs note (moot — every
|
||||
node is a seed of its own mesh); `Cluster__SeedNodes__*` env matrix per pair.
|
||||
**Exit gate:** rig up in the new shape; every existing live-gated behavior re-verified per pair
|
||||
(deploy, redundancy 250/240 per pair — **two Primaries fleet-wide, one per pair, by design**);
|
||||
secrets Akka replication re-verified or re-scoped (it rides DPS on the current single mesh — its
|
||||
topology must be re-decided here, likely SQL-hub mode like ScadaBridge, since pub/sub cannot
|
||||
cross separate meshes).
|
||||
|
||||
### Phase 7 — Failover drill + live gates
|
||||
**Scope:** the drill ScadaBridge already has (`failover-drill.sh` analogue) run per pair, both
|
||||
directions; **close the two outstanding live gates**: (a) auto-down 1-vs-1 crash-the-oldest
|
||||
(deferred since Phase 0a — finally testable, every mesh is exactly two nodes), (b) `SelfFormAfter`
|
||||
lone-cold-start on the real per-pair topology; manual-failover button re-verified per pair;
|
||||
operator runbook for the co-located site (one page covering both products' failover on the same
|
||||
two VMs).
|
||||
**Exit gate:** drill green on every pair type (central, site); runbook merged; design doc §7
|
||||
table fully marked DONE.
|
||||
|
||||
---
|
||||
|
||||
## Risks carried from the design (§8, unchanged — re-read before each phase plan)
|
||||
|
||||
LocalDb becomes load-bearing for config (blast radius of the #485 class rises);
|
||||
`DbHealthProbeActor` feeds a client-visible value; the rig models the doomed topology until
|
||||
Phase 6; losing gossip loses free observability (7 panels); 128 KB ClusterClient frame drop is
|
||||
silent unless both knobs are set; never stack app-level LWW on LocalDb's HLC. **New (this
|
||||
program):** co-located VMs mean a VM loss now takes out one node of BOTH products at once — the
|
||||
drill in Phase 7 must include the shared-VM failure (both products fail over together), and
|
||||
resource sizing on the site VMs should be checked once both products run the full stack.
|
||||
|
||||
## Tracking
|
||||
|
||||
| Phase | Status |
|
||||
|---|---|
|
||||
| 0a downing strategy | DONE 2026-07-21 (live gate → Phase 7) |
|
||||
| 0b oldest-Up election | DONE 2026-07-21 |
|
||||
| Prereq: selfform-fallback + manual-failover plan | plan written 2026-07-22, not executed |
|
||||
| 1 ClusterNode columns + DB ack set | not started — plan to be written |
|
||||
| 2 ClusterClient transport | not started |
|
||||
| 3 fetch-and-cache | not started |
|
||||
| 4 cut driver ConfigDb | not started |
|
||||
| 5 gRPC telemetry | not started |
|
||||
| 6 mesh partition + co-location | not started |
|
||||
| 7 drill + live gates | not started |
|
||||
Reference in New Issue
Block a user