From b853185cfd5870a4ee6c00c05f30680336e74a9e Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Wed, 22 Jul 2026 06:10:11 -0400 Subject: [PATCH] docs(redundancy): SelfFormAfter closes the seed-node bootstrap gap Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- CLAUDE.md | 2 + docs/Redundancy.md | 45 +- .../2026-07-21-per-cluster-mesh-design.md | 14 + .../2026-07-22-per-cluster-mesh-program.md | 194 +++++++++ ...-22-per-cluster-mesh-program.md.tasks.json | 15 + ...2-selfform-fallback-and-manual-failover.md | 388 ++++++++++++++++++ ...fallback-and-manual-failover.md.tasks.json | 13 + 7 files changed, 666 insertions(+), 5 deletions(-) create mode 100644 docs/plans/2026-07-22-per-cluster-mesh-program.md create mode 100644 docs/plans/2026-07-22-per-cluster-mesh-program.md.tasks.json create mode 100644 docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md create mode 100644 docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md.tasks.json diff --git a/CLAUDE.md b/CLAUDE.md index 86da19cd..5cca4ce7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -224,6 +224,8 @@ The server supports non-transparent warm/hot redundancy via the `Redundancy` sec - **Downing is `auto-down`, not `keep-oldest`.** `Cluster:SplitBrainResolverStrategy` (default `auto-down`) selects the provider via `ServiceCollectionExtensions.BuildDowningHocon`; an unknown value fails host start. A two-node pair running the previous `keep-oldest` + `down-if-alone` **could not survive a crash of the oldest node**: Akka's `KeepOldest.OldestDecision` only lets `down-if-alone` rescue a side holding ≥ 2 members, so the 1-vs-1 survivor downs *itself* and exits. `down-if-alone` is a 3+-node feature. The accepted trade for `auto-down` is dual-active during a genuine network partition. **Its live gate is still open** — the pathology only appears 1-vs-1 and docker-dev is a single six-node mesh, so the crash-the-oldest drill is deferred to the per-cluster mesh work. - **The Primary is the oldest Up `driver` member, not the role leader.** `RedundancyStateActor` previously used `ClusterState.RoleLeader("driver")` (lowest *address*), while `ClusterSingletonManager` places singletons on the *oldest*. They agree only on a freshly-formed cluster and diverge after any restart, so the Primary-gated data plane (writes, alarm acks, alerts emit, alarm-history drain) could enable on a node not hosting the singletons. `NodeRedundancyState.IsRoleLeaderForDriver` was renamed **`IsDriverPrimary`**. +**A third bootstrap gap closed 2026-07-22 — downing was never the whole story.** Even under `auto-down`, a node cold-starting while its peer was dead **never came Up**: Akka lets only the **first** listed seed form a *new* cluster, so every other node loops on `InitJoin` forever. `Cluster:SelfFormAfter` (`TimeSpan?`, default 10 s, `null`/`≤0` disables) arms `ClusterBootstrapFallback` from `WithOtOpcUaClusterBootstrap` via an Akka.Hosting `AddStartup` task — it waits that long for membership, then `Cluster.Join(SelfAddress)`. **It fires only when this node's own address is in its own `SeedNodes`**: a node seeded only by someone else (today's docker-dev site nodes, which list only `central-1`) must wait rather than island itself permanently. Pinned by `SelfFormBootstrapTests`, whose two negative cases each carry a positive-control self-join so they cannot pass vacuously. See `docs/Redundancy.md` §"Bootstrap: the self-form fallback". + Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file. **Still per-Akka-cluster, not per application `Cluster`.** One Primary is elected across the whole Akka mesh, so a fleet running several application clusters in one mesh (what docker-dev models) gets one Primary among all driver nodes. The fix is the per-cluster mesh workstream: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (Phases 0a/0b done; 1–7 not started, Phase 1 planned in `…-phase1.md`). diff --git a/docs/Redundancy.md b/docs/Redundancy.md index be62fc5f..2d8a5dff 100644 --- a/docs/Redundancy.md +++ b/docs/Redundancy.md @@ -159,7 +159,7 @@ Per-node identity comes from `appsettings.json` + the `OTOPCUA_ROLES` env var: OTOPCUA_ROLES=admin,driver ``` -Both nodes share the same `ConfigDb` connection string; `Cluster.PublicHostname` + `Roles` are what makes them distinct in cluster gossip. The first node bootstraps the cluster (its address goes in `SeedNodes`); the second node joins via the same `SeedNodes` list. +Both nodes share the same `ConfigDb` connection string; `Cluster.PublicHostname` + `Roles` are what makes them distinct in cluster gossip. The first node bootstraps the cluster (its address goes in `SeedNodes`); the second node joins via the same `SeedNodes` list. List **both** peers in `SeedNodes` on **both** nodes so either can cold-start alone — see [Bootstrap: the self-form fallback](#bootstrap-the-self-form-fallback) for why the seed list alone is not enough, and what `Cluster:SelfFormAfter` adds. There is no longer a `Node:NodeId` setting and no `ClusterNode.RedundancyRole` column (the V2 migration dropped it — primary/secondary is now derived from cluster membership age). NodeId is derived as `host:port` of the cluster `PublicHostname` (see `ClusterRoleInfo.LocalNode` for the formula). @@ -241,10 +241,9 @@ two-node pair that choice means *any* crash of the oldest node is a full outage. `IHostApplicationLifetime.StopApplication()` so the process exits and the supervisor restarts it instead of idling forever with a dead actor system. Under `auto-down` the *survivor* no longer needs this (it is never the one downed); the **downed** node still does. -3. **Both peers listed in `SeedNodes`** on every node, so a restarted node can re-join via either peer. - Note the residual bootstrap constraint: only the **first** seed may form a cluster alone, so a node - cold-starting while its peer is dead still waits in `InitJoin`. Auto-down removes the crash outage, not - this one. +3. **Both peers listed in `SeedNodes`** on every node, so a restarted node can re-join via either peer — + plus the **self-form fallback** below, without which a node cold-starting while its peer is dead waits + in `InitJoin` forever (auto-down removes the crash outage, not that one). > **On `HardKillFailoverTests`:** that in-process test hard-kills the oldest and asserts the survivor becomes > sole `driver` role-leader. It passed even under `keep-oldest` — because it simulates the crash with @@ -270,6 +269,42 @@ automatically. > alongside the per-cluster mesh work, which makes every mesh exactly two nodes. See > `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2 / Phase 0a. +## Bootstrap: the self-form fallback + +Listing both peers in `SeedNodes` does **not** mean either node can cold-start alone. Akka lets only the +**first** listed seed form a *new* cluster; every other node sends `InitJoin` to the seeds and retries — +forever — until one of them answers. A node that boots while its peer is dead therefore never comes Up, +and no downing strategy helps: **auto-down removes the crash outage, not this one.** + +`Cluster:SelfFormAfter` closes that gap. `ClusterBootstrapFallback.Arm` (armed for every host built through +`WithOtOpcUaClusterBootstrap`, via an Akka.Hosting `AddStartup` task) waits that long for cluster membership +and, on expiry, calls `Cluster.Join(SelfAddress)` — the node forms a cluster on itself and becomes +operational unattended. + +| `Cluster:SelfFormAfter` | Behaviour | +|---|---| +| `"00:00:10"` (**default**) | A node with no membership after 10 s self-forms. A live peer answers `InitJoin` in milliseconds, so on any normal boot the fallback never fires. | +| `null` / `≤ 0` | Disabled — the pre-2026-07-22 behaviour: wait on `InitJoin` indefinitely. | + +**The island guard is the load-bearing part.** The fallback fires **only when this node's own address +appears in its own `SeedNodes`**. A node that is not one of its own seeds is never legitimately first, and +if it self-formed, a later-booting real seed would form a second cluster the two could never merge. That is +exactly the current docker-dev topology: the site-a/site-b driver nodes list only `central-1` as a seed, so +they deliberately keep waiting rather than islanding themselves. Under the future per-cluster mesh +(`docs/plans/2026-07-21-per-cluster-mesh-design.md`) each pair node *is* a seed of its own two-node mesh, so +the fallback covers both sides of every pair and the site-node exception disappears. + +Sequential recovery is island-free by construction: Akka's join protocol prefers an existing cluster (a +booting node only self-forms when **no** seed answered), so a peer booting after the survivor self-formed +simply joins it as the youngest member. The residual risk is both pair nodes cold-starting inside the +window while mutually unreachable — both self-form, which is the same dual-active class the `auto-down` +strategy already accepts, with the same recovery (restart one side). + +Pinned by `SelfFormBootstrapTests` (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/`), which starts real hosts +through the production bootstrap: a lone non-first seed comes Up; a disabled window keeps waiting; a node +absent from its own seed list never self-forms. The two negative cases each carry a positive control (an +explicit self-join) so they cannot pass merely because the node was unformable. + ## Primary data-plane gate (writes, acks, alerts emit) Three data-plane surfaces are **Primary-gated** so a warm standby never touches a shared field device or duplicates a fleet-wide publish: inbound operator **writes** (`DriverHostActor.HandleRouteNodeWrite`), native-alarm **acks** (`HandleRouteNativeAlarmAck`), and the native/scripted **alerts emit** (`ForwardNativeAlarm` / `ScriptedAlarmHostActor.OnEngineEmission`). All three route through one policy — `PrimaryGatePolicy.ShouldServiceAsPrimary(localRole, driverMemberCount)` (`OtOpcUa.Runtime.Drivers`): diff --git a/docs/plans/2026-07-21-per-cluster-mesh-design.md b/docs/plans/2026-07-21-per-cluster-mesh-design.md index a2f9036c..71362557 100644 --- a/docs/plans/2026-07-21-per-cluster-mesh-design.md +++ b/docs/plans/2026-07-21-per-cluster-mesh-design.md @@ -85,6 +85,12 @@ Their failover drill has a mode that exists *"to make the registered gap observa it is covered."* **A two-node keep-oldest mesh has an acknowledged total-outage hole.** If OtOpcUa adopts 2-node meshes, this must be decided deliberately, not inherited. +> **CLOSED 2026-07-22.** The `InitJoin` half of this gap is fixed in both repos by the self-form +> fallback: `Cluster:SelfFormAfter` (default 10 s) lets a node that is one of its own seeds form a +> cluster on itself when no seed answers. See `docs/Redundancy.md` §"Bootstrap: the self-form +> fallback" — including the island guard that keeps today's non-seed site nodes waiting, which the +> per-cluster mesh removes by making every pair node a seed of its own mesh. + ### Both inter-cluster transports are unauthenticated Akka remoting has no TLS, no secure cookie, no `trusted-selection-paths`. The gRPC listener is h2c @@ -253,6 +259,14 @@ rather than the configured one. See `docs/Redundancy.md`. 1-vs-1 split, and `docker-dev` is a single six-node mesh. The crash-the-oldest drill therefore lands with Phase 6/7, which makes every mesh exactly two nodes. +**The bootstrap half of the gap is also closed (2026-07-22).** Downing was only ever one of the two +outage modes: even with `auto-down`, a node cold-starting while its peer was dead looped on `InitJoin` +forever, because Akka lets only the first listed seed form a new cluster. `Cluster:SelfFormAfter` +(default 10 s, `ClusterBootstrapFallback`) now self-forms in that case, guarded so a node absent from +its own seed list never islands itself. Pinned by `SelfFormBootstrapTests`; see `docs/Redundancy.md` +§"Bootstrap: the self-form fallback". Under this design each pair node is a seed of its own two-node +mesh, so the fallback covers both sides and the current site-node exception disappears. + ### 6.3 DECIDED — match ScadaBridge's auth posture for now Inter-cluster transports stay unauthenticated and unencrypted, as theirs are: the boundary assumes a diff --git a/docs/plans/2026-07-22-per-cluster-mesh-program.md b/docs/plans/2026-07-22-per-cluster-mesh-program.md new file mode 100644 index 00000000..e1ef53ef --- /dev/null +++ b/docs/plans/2026-07-22-per-cluster-mesh-program.md @@ -0,0 +1,194 @@ +# Per-Cluster Mesh Program — Align OtOpcUa's Akka Topology with ScadaBridge + +> **For Claude:** This is a PROGRAM plan (phase roadmap + gates), not a bite-sized task plan. +> The authoritative design is `docs/plans/2026-07-21-per-cluster-mesh-design.md` (decisions settled +> 2026-07-21; Phases 0a/0b DONE). Per that design's own instruction, **each phase gets its own +> detailed plan, written when the phase starts** (superpowers writing-plans → executing-plans), +> so plans are authored against current code, not against a forecast. This document sequences the +> phases, fixes the deployment topology (co-location with ScadaBridge — NEW constraint 2026-07-22), +> and defines each phase's entry/exit gates. + +**Goal:** OtOpcUa runs the same Akka.NET mesh shape as ScadaBridge — **one Akka mesh per +application `Cluster`, two nodes max**, central ↔ cluster joined by explicit transports +(ClusterClient + gRPC + fetch), driver nodes holding **no ConfigDb connection** — so that every +Primary-gated decision and every gated resource share one pair-local scope, and both products +present one operational model on the shared site hardware. + +**Why now (2026-07-22):** OtOpcUa cluster pairs will run **on the same two Windows VMs as the +ScadaBridge site nodes**. That deployment makes the current single-fleet-mesh design actively +wrong for sites (a site's OtOpcUa nodes would gossip across the WAN to central and elect ONE +Primary fleet-wide), and makes the ScadaBridge shape the obviously correct one: each site's two +VMs host two independent, identically-postured 2-node clusters (one per product), surviving alone +on the site LAN. It also means driver nodes must not depend on reaching central SQL — **sites +have no SQL Server**, so §6.1's "driver nodes never connect to the ConfigDb" stops being an +architectural preference and becomes a deployment requirement. + +--- + +## Deployment topology (the co-location constraint, NEW) + +Each **site**: 2 Windows VMs. Each VM runs one ScadaBridge site node AND one OtOpcUa driver node. +Two independent 2-node Akka clusters per site — they share hardware, never a mesh. **Central**: +2 Windows VMs, each running a ScadaBridge central node and an OtOpcUa central node +(`admin,driver`), again as separate pairs. + +**Per-VM port allocation (no collisions — verify against actual deployment configs in Phase 6):** + +| Port | Owner | Purpose | +|---|---|---| +| 8081 / 8082 | ScadaBridge | Akka remoting (central / site) | +| 8083 | ScadaBridge site | gRPC h2c (streams + LocalDb sync) | +| 8084 | ScadaBridge site | metrics | +| 5000 (+Traefik) | ScadaBridge central | UI + Inbound API | +| **4053** | OtOpcUa | Akka remoting | +| **4840** | OtOpcUa | OPC UA endpoint | +| OtOpcUa AdminUI port | OtOpcUa central (admin) | AdminUI (driver-only site nodes host no UI) | +| OtOpcUa LocalDb sync port | OtOpcUa driver | h2c LocalDb pair replication (default off/0 today — Phase 6 assigns a real per-site port) | + +**Aligned HA posture (both products, per VM — already true or landing via the selfform plan):** +auto-down downing (15 s window), oldest-Up active/primary election, `SelfFormAfter` 10 s +self-form fallback, termination-watchdog → process exit → `sc.exe failure` restart recovery. +One failover story for operators regardless of product. + +**Prerequisite ordering:** the fallback/manual-failover plan +(`docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`) executes **before** this +program — it is small, independent, and Phase 6 depends on its semantics (under per-pair meshes +every node lists itself + partner as seeds, so the fallback covers both nodes of every pair; +the site-node island guard then simply never triggers). + +--- + +## What changes, in one table (from the design doc) + +| Aspect | Today (single fleet mesh) | Target (ScadaBridge shape) | +|---|---|---| +| Mesh | one gossip ring, all nodes, seeded by central-1 | one 2-node mesh per application `Cluster`; same ActorSystem name; separation by seed-node partitioning | +| Roles | `admin` / `driver` fleet-wide | `driver` + cluster-specific `cluster-{ClusterId}`; singletons scoped to the cluster role | +| Primary election | mesh-wide oldest Up driver (Phase 0b) | pair-local by construction — same rule, correct scope | +| Command/control | 9 DPS topics + singleton over gossip | ClusterClient (one receptionist actor per side); central discovers nodes from `ClusterNode` rows | +| Deploy | notify (DPS) + node fetches from ConfigDb | notify via ClusterClient; artifact fetched **from central**, cached in LocalDb | +| Driver ConfigDb connection | direct EF connection to central SQL | **none** — LocalDb is the steady-state config store | +| Live telemetry | 7 observability DPS topics | one gRPC stream contract (`oneof` event), central dials each cluster node | +| Rig | six-node single mesh (`docker-dev`) | per-cluster meshes; rig models the real topology | + +--- + +## Phases + +Each phase below is one row of design-doc §7, expanded with entry/exit gates. **Execution recipe +per phase:** (1) invoke writing-plans in this repo to produce +`docs/plans/2026-07-2X-mesh-phase-N-.md` from the scope notes here + the design doc § +references, exploring current code first; (2) execute it task-by-task; (3) run the phase's exit +gate; (4) update this file's status column and the design doc's §7 table. + +### Phase 1 — `ClusterNode` address columns + DB-sourced ack set +**Scope:** `ClusterNode` gains Akka + gRPC address columns (mirroring ScadaBridge's `Site` +entity `NodeAAddress`/`GrpcNodeAAddress` pattern, but per-node rows); EF migration; AdminUI node +edit surfaces the fields; `ConfigPublishCoordinator` derives its expected-ack set from +`ClusterNode` rows instead of `Akka.Cluster.State.Members` filtered by role (design §3 fact 3 — +this removes the coordinator's one genuinely mesh-bound dependency). +**Independent of the split:** yes — safe on the current mesh. +**Exit gate:** deploy on the unchanged docker-dev rig completes with the coordinator's expected-ack +set proven DB-sourced (test: a `ClusterNode` row present but node down → deploy reports that node +missing; a node up but row absent → its ack is not expected). + +### Phase 2 — Comm actors + ClusterClient transport +**Scope:** one receptionist-registered actor per side (`/user/central-communication`, +`/user/cluster-communication`), registered **per node, not as a singleton** (contact rotation); +ClusterClient central → cluster carrying deploy notify + acks + driver-control; the ScadaBridge +idioms copied verbatim: sender-preserving `Tell(new ClusterClient.Send(...), Sender)` Ask relay, +typed-failure reply for every unhandled message, no central buffering toward unreachable clusters +(drop + warn), central discovers contacts from Phase 1's `ClusterNode` rows (60 s refresh + +admin-change refresh), clusters know central from appsettings (static, restart to change). +**Frame-size guard:** anything carrying payload sets both the frame limit and +`log-frame-size-exceeding` (design §8) — the deploy path stays payload-free by design. +**Exit gate:** on the still-single-mesh rig, deploy notify + acks and AdminUI Reconnect/Restart +flow over ClusterClient (DPS paths deleted or dark-switched), including an Ask timing out cleanly +against a stopped node. + +### Phase 3 — Config fetch-and-cache from central +**Scope:** the deploy artifact is served **by central** (transport per its own phase plan — the +ScadaBridge analogue is token-gated HTTP; decide HTTP vs a gRPC fetch RPC when planning) and +cached in LocalDb; driver nodes read config exclusively from LocalDb (boot-from-cache becomes the +normal path — design §6.1); chunking, SHA-256 verify, newest-2 retention, pair replication carry +over unchanged. **This phase changes a running data path — it gets its own live gate** (design §7 +note): deploy lands on a driver pair with central SQL stopped mid-fetch → retry lands; #485 +last-known-good semantics re-proven on the new path. +**Exit gate:** live gate green on the rig; a driver node with an empty LocalDb and reachable +central boots into the current config; with central down it boots last-known-good. + +### Phase 4 — Cut the driver-side ConfigDb connection +**Scope (design §6.1 audit table):** re-home `EfAlarmConditionStateStore` to LocalDb (pair-local +state, same journey as the Phase-2 alarm S&F buffer); **resolve the `DbHealthProbeActor` +question** — driver nodes have no DB to probe, and DB health currently feeds ServiceLevel tiering, +so define the replacement health input (candidate: central-reachability via the Phase 2/3 +transports) — this is a client-visible ServiceLevel semantics change and must be documented in +`docs/Redundancy.md` + the interop playbook; audit `OpcUaPublishActor`'s ConfigDb use (TBD in the +design) and re-source it; registration cleanup in `ServiceCollectionExtensions`; driver-role +`Program.cs` branch registers no EF ConfigDb context at all (mirror ScadaBridge's central-only +`AddConfigurationDatabase`). +**Exit gate:** its own live gate — a driver pair runs a full deploy + alarm + historian cycle with +**no ConfigDb connection string configured at all**; grep-level proof no driver-branch service can +resolve the ConfigDb context. + +### Phase 5 — gRPC stream contract for live telemetry +**Scope:** one server-streaming contract carrying a `oneof` event, **cluster nodes host the gRPC +server, central dials in** (the inverted direction is the load-bearing ScadaBridge finding — +design §2); migrate the seven observability topics (`alerts`, `driver-health`, +`driver-resilience-status`, `fleet-status`, `script-logs`, plus redundancy-state distribution and +deployment-acks if Phase 2 left them on DPS); additive-only field evolution, contract locked by +test; per-panel reconnect story for the AdminUI (design §8 — losing gossip loses free fleet +observability). +**Exit gate:** all AdminUI live panels green against a pair with DPS telemetry topics deleted; +kill-and-reconnect of the central dialer recovers every stream. + +### Phase 6 — Mesh partition + co-location topology +**Scope:** per-cluster seed nodes (each pair node lists itself + partner — the `SelfFormAfter` +fallback then covers both), cluster-scoped roles `cluster-{ClusterId}` + singleton re-scoping, +central pair keeps the admin singletons; **docker-dev rig rewritten** to model the real topology — +including the co-location port table above (both products' compose files on shared per-site +networks, real LocalDb sync ports); remove the ClusterRedundancy page's mesh-scope caveat (the +election is pair-local now) and the fallback's site-node island-guard docs note (moot — every +node is a seed of its own mesh); `Cluster__SeedNodes__*` env matrix per pair. +**Exit gate:** rig up in the new shape; every existing live-gated behavior re-verified per pair +(deploy, redundancy 250/240 per pair — **two Primaries fleet-wide, one per pair, by design**); +secrets Akka replication re-verified or re-scoped (it rides DPS on the current single mesh — its +topology must be re-decided here, likely SQL-hub mode like ScadaBridge, since pub/sub cannot +cross separate meshes). + +### Phase 7 — Failover drill + live gates +**Scope:** the drill ScadaBridge already has (`failover-drill.sh` analogue) run per pair, both +directions; **close the two outstanding live gates**: (a) auto-down 1-vs-1 crash-the-oldest +(deferred since Phase 0a — finally testable, every mesh is exactly two nodes), (b) `SelfFormAfter` +lone-cold-start on the real per-pair topology; manual-failover button re-verified per pair; +operator runbook for the co-located site (one page covering both products' failover on the same +two VMs). +**Exit gate:** drill green on every pair type (central, site); runbook merged; design doc §7 +table fully marked DONE. + +--- + +## Risks carried from the design (§8, unchanged — re-read before each phase plan) + +LocalDb becomes load-bearing for config (blast radius of the #485 class rises); +`DbHealthProbeActor` feeds a client-visible value; the rig models the doomed topology until +Phase 6; losing gossip loses free observability (7 panels); 128 KB ClusterClient frame drop is +silent unless both knobs are set; never stack app-level LWW on LocalDb's HLC. **New (this +program):** co-located VMs mean a VM loss now takes out one node of BOTH products at once — the +drill in Phase 7 must include the shared-VM failure (both products fail over together), and +resource sizing on the site VMs should be checked once both products run the full stack. + +## Tracking + +| Phase | Status | +|---|---| +| 0a downing strategy | DONE 2026-07-21 (live gate → Phase 7) | +| 0b oldest-Up election | DONE 2026-07-21 | +| Prereq: selfform-fallback + manual-failover plan | plan written 2026-07-22, not executed | +| 1 ClusterNode columns + DB ack set | not started — plan to be written | +| 2 ClusterClient transport | not started | +| 3 fetch-and-cache | not started | +| 4 cut driver ConfigDb | not started | +| 5 gRPC telemetry | not started | +| 6 mesh partition + co-location | not started | +| 7 drill + live gates | not started | diff --git a/docs/plans/2026-07-22-per-cluster-mesh-program.md.tasks.json b/docs/plans/2026-07-22-per-cluster-mesh-program.md.tasks.json new file mode 100644 index 00000000..697e7ac5 --- /dev/null +++ b/docs/plans/2026-07-22-per-cluster-mesh-program.md.tasks.json @@ -0,0 +1,15 @@ +{ + "planPath": "docs/plans/2026-07-22-per-cluster-mesh-program.md", + "note": "PROGRAM plan: each task = write that phase's detailed plan (writing-plans), execute it (executing-plans), run its exit gate, update the tracking tables. Prereq task 0 is the separate selfform-fallback plan in this repo.", + "tasks": [ + {"id": 0, "subject": "Prereq: execute 2026-07-22-selfform-fallback-and-manual-failover.md (7 tasks)", "status": "pending"}, + {"id": 1, "subject": "Phase 1: ClusterNode Akka+gRPC address columns; ConfigPublishCoordinator ack set from DB", "status": "pending"}, + {"id": 2, "subject": "Phase 2: comm actors + receptionist + ClusterClient transport (deploy notify/acks, driver-control)", "status": "pending", "blockedBy": [1]}, + {"id": 3, "subject": "Phase 3: config fetch-and-cache from central; LocalDb steady-state config store (live gate)", "status": "pending", "blockedBy": [2]}, + {"id": 4, "subject": "Phase 4: cut driver-side ConfigDb — EfAlarmConditionStateStore to LocalDb, DbHealthProbeActor ServiceLevel input, OpcUaPublishActor audit (live gate)", "status": "pending", "blockedBy": [3]}, + {"id": 5, "subject": "Phase 5: gRPC oneof stream contract; migrate 7 observability topics; AdminUI reconnect story", "status": "pending", "blockedBy": [2]}, + {"id": 6, "subject": "Phase 6: mesh partition — per-pair seeds, cluster-scoped roles/singletons, co-location rig rewrite, secrets replication re-scope", "status": "pending", "blockedBy": [0, 4, 5]}, + {"id": 7, "subject": "Phase 7: failover drill per pair (incl. shared-VM failure) + close auto-down 1v1 and SelfFormAfter live gates + operator runbook", "status": "pending", "blockedBy": [6]} + ], + "lastUpdated": "2026-07-22T00:00:00Z" +} diff --git a/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md b/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md new file mode 100644 index 00000000..d7d2a30d --- /dev/null +++ b/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md @@ -0,0 +1,388 @@ +# OtOpcUa: InitJoin Self-Form Fallback + Manual Failover Control — Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. +> +> Shared cross-repo design: `~/Desktop/scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md` (design rationale, MNTR assessment, behavior spec). The ScadaBridge half lives in `~/Desktop/ScadaBridge/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`. This plan is self-contained for execution. +> +> **⚠ This plan is expected to GROW** — see "Reserved for additions" at the end. Append new tasks there (and to the `.tasks.json`) rather than renumbering existing ones. + +**Goal:** (1) Either node of a 2-node OtOpcUa pair can cold-start alone and become operational, unattended — a configurable, fast (default 10 s) InitJoin timeout falls back to self-forming a cluster, with a hard guard so site nodes seeded only by central can never island themselves. (2) An admin-only "Trigger failover" control on the Cluster Redundancy page performs a graceful, audited swap of the driver Primary. + +**Architecture:** New `AkkaClusterOptions.SelfFormAfter` (`TimeSpan?`, default 10 s, `null`/`≤0` disables, bound from the `Cluster` section) arms `ClusterBootstrapFallback` via an Akka.Hosting startup task inside `WithOtOpcUaClusterBootstrap` — so production AND the Cluster.Tests hosts get it identically. Wait for membership via `RegisterOnMemberUp`; on expiry, `Cluster.Join(SelfAddress)` — only if this node's own address is in its own `SeedNodes`. Manual failover = graceful `Cluster.Leave()` via a new `IManualFailoverService` in ControlPlane, surfaced on `/clusters/{id}/redundancy`. + +**Tech Stack:** .NET 10, Akka.NET 1.5.62, Akka.Hosting 1.5.62 (`AddStartup`), Blazor Server (AdminUI, InteractiveServer), bUnit, xunit. No new packages. + +**Branch:** `feat/selfform-fallback` off `master`. + +--- + +## Design essentials (from the shared design doc) + +**The defect:** Akka only lets the FIRST listed seed self-join to form a *new* cluster; every other node loops on `InitJoin` forever — "auto-down removes the crash outage, not this one" (`docs/Redundancy.md:244-247`). A lone cold-starting `central-2` (or a pair node under the future per-cluster mesh) never comes Up. + +**Behavior spec:** + +| Scenario | Behavior with fallback | +|---|---| +| Peer alive (any boot order) | Normal seed join in ms — fallback never fires | +| Lone cold-start, self IS in own `SeedNodes` | After `SelfFormAfter`: warn log + `Cluster.Join(SelfAddress)` → Up alone (`min-nr-of-members=1`) | +| Lone cold-start, self NOT in own `SeedNodes` | Fallback inert (info log). **This is the docker-dev site-node topology** — site-a/b nodes list only `central-1`; self-forming there would island them permanently, so they must wait. | +| Peer boots after survivor self-formed | Its InitJoin is answered → joins as youngest. No island. | +| Both pair nodes cold-start simultaneously, mutually unreachable | Both self-form → dual-active (same partition class auto-down accepts; restart one side) | +| `SelfFormAfter` null/`≤0` | Disabled — today's wait-forever behavior | +| Window expires mid-join-handshake | Benign: Akka ignores `Join` once joined | + +**Manual failover rules:** graceful `Leave` of the **oldest Up `driver` member** (identical query to `RedundancyStateActor.SelectDriverPrimary` — `Member.AgeOrdering`, `Leaving` excluded via `Status == Up`), never `Down`. The leaving node hands over via the cluster-leave phases, `CoordinatedShutdown` runs, `ActorSystemTerminationWatchdog` exits the process, the supervisor restarts it, and it rejoins as youngest → the survivor is Primary (ServiceLevel 250 moves; OPC UA clients re-select). Admin-only; peer guard (<2 Up driver members → disabled); confirmation dialog; audited via the shared `ZB.MOM.WW.Audit.IAuditWriter` seam before the Leave. **Mesh-scope caveat:** until the per-cluster mesh lands, the election is mesh-wide — the button acts on THE Primary of the whole Akka cluster, not per-`ServerCluster` row; the UI must say so (mirrors the KNOWN LIMITATION in `docs/Redundancy.md:84-99`). + +**Multi-node TestKit:** assessed, NOT used — everything is deterministic in-process via real hosts through `WithOtOpcUaClusterBootstrap` (the `SplitBrainResolverActivationTests` pattern), and the Akka TestKit family is xunit-v2-only while this repo's integration tree is xunit.v3. Full verdict in the shared design doc. + +--- + +### Task 1 (B1): `SelfFormAfter` on `AkkaClusterOptions` + +**Classification:** small +**Estimated implement time:** ~3 min +**Parallelizable with:** none (first task) + +**Files:** +- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptions.cs` + +**Step 1: Branch** + +```bash +cd ~/Desktop/OtOpcUa && git checkout master && git checkout -b feat/selfform-fallback +``` + +**Step 2: Add the property** (after `SplitBrainResolverStrategy`, mirroring its XML-doc depth): + +```csharp + /// + /// Bootstrap self-form fallback window (decision 2026-07-22, scadaproj/akka_failover.md §6.1). + /// Akka only lets the FIRST listed seed form a new cluster; a non-first seed cold-starting + /// while its peer is down loops on InitJoin forever ("auto-down removes the crash outage, + /// not this one" — docs/Redundancy.md). When this node has waited longer than this window + /// without cluster membership it joins itself — but ONLY if its own address appears in + /// ; a non-seed node (e.g. a site node whose only seed is central-1) + /// stays waiting, because self-forming there creates a permanent island. Default 10s + /// (same-datacenter pair; a live peer answers InitJoin in milliseconds). null or a + /// non-positive value disables the fallback. Accepted trade: both pair nodes cold-starting + /// inside the window while mutually unreachable form two clusters — the same dual-active + /// class the auto-down strategy already accepts, same recovery (restart one side). + /// + public TimeSpan? SelfFormAfter { get; set; } = TimeSpan.FromSeconds(10); +``` + +**Step 3:** `dotnet build src/Core/ZB.MOM.WW.OtOpcUa.Cluster` — 0 warnings. Commit: + +```bash +git add -A && git commit -m "feat(cluster): SelfFormAfter option (bootstrap self-form fallback window)" +``` + +(The default-value pin test lands with Task 3's test file — the property is inert until Task 2.) + +--- + +### Task 2 (B2): `ClusterBootstrapFallback` + `AddStartup` wiring + +**Classification:** high-risk (cluster formation behavior) +**Estimated implement time:** ~5 min +**Parallelizable with:** none + +**Files:** +- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ClusterBootstrapFallback.cs` +- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs` (`WithOtOpcUaClusterBootstrap`, after the `AddHocon(BuildDowningHocon(options), HoconAddMode.Prepend)` call at line 93) + +**Step 1: Implement the fallback class** (deliberately duplicated from ScadaBridge's, like the termination watchdogs — see that repo's `ClusterBootstrapFallback.cs` for the full doc comment to adapt): + +```csharp +using Akka.Actor; +using Microsoft.Extensions.Logging; + +namespace ZB.MOM.WW.OtOpcUa.Cluster; + +/// +/// InitJoin self-form fallback (decision 2026-07-22, scadaproj/akka_failover.md §6.1). Akka only +/// lets the FIRST listed seed form a NEW cluster; every other node retries InitJoin forever — +/// "auto-down removes the crash outage, not this one" (docs/Redundancy.md). Waits +/// for membership, then joins itself. +/// Island safety: fires ONLY when this node's own address is in its own +/// — a site node seeded only by central-1 must wait, +/// not island. Sequential recovery is island-free: Akka's join protocol prefers an existing +/// cluster (a booting node only self-forms when NO seed answers InitJoin). +/// Benign race: Akka ignores Join once the node has joined a cluster this +/// incarnation. Residual risk: both pair nodes cold-starting inside the window while mutually +/// unreachable self-form separately — the same dual-active class auto-down accepts. +/// +public static class ClusterBootstrapFallback +{ + public static void Arm(ActorSystem system, AkkaClusterOptions options, ILogger logger) + { + if (options.SelfFormAfter is not { } window || window <= TimeSpan.Zero) + { + logger.LogInformation( + "Cluster self-form fallback disabled (SelfFormAfter not set) — a node cold-starting " + + "while its peer is down will wait on InitJoin indefinitely."); + return; + } + + var cluster = Akka.Cluster.Cluster.Get(system); + var self = cluster.SelfAddress; + var isSeed = options.SeedNodes.Any(s => TryParseAddress(s, out var a) && a.Equals(self)); + if (!isSeed) + { + logger.LogInformation( + "Cluster self-form fallback inactive: this node ({Self}) is not in its own seed list " + + "[{Seeds}] — self-forming here would island it from the real seeds.", + self, string.Join(", ", options.SeedNodes)); + return; + } + + var joined = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + cluster.RegisterOnMemberUp(() => joined.TrySetResult()); + + _ = Task.Run(async () => + { + var winner = await Task.WhenAny(joined.Task, Task.Delay(window)); + if (winner == joined.Task || system.WhenTerminated.IsCompleted) + return; + logger.LogWarning( + "No cluster membership after {Window} — no seed answered InitJoin (peer down at boot). " + + "Self-forming a cluster at {Self} so this node becomes operational; if the peer was " + + "merely partitioned (not dead), the pair is now dual-active — restart one side after " + + "the partition heals (accepted availability-first trade, decision 2026-07-22).", + window, self); + cluster.Join(self); + }); + } + + private static bool TryParseAddress(string seed, out Address address) + { + try { address = Address.Parse(seed); return true; } + catch { address = default!; return false; } + } +} +``` + +(If the Cluster project lacks a `Microsoft.Extensions.Logging.Abstractions` reference, add the `PackageReference` — check `Directory.Packages.props` for the pinned version first.) + +**Step 2: Wire via Akka.Hosting startup hook** — in `WithOtOpcUaClusterBootstrap`, after the downing HOCON line (~93): + +```csharp + // InitJoin self-form fallback (decision 2026-07-22): registered as an Akka.Hosting + // startup task so every host built through this bootstrap — production AND the test + // hosts in Cluster.Tests — arms it identically. Guarded inside Arm: disabled when + // SelfFormAfter is unset; inert when this node is not in its own seed list (site nodes + // seeded only by central-1 must wait, not island). + var fallbackLogger = (serviceProvider.GetService() + ?? Microsoft.Extensions.Logging.Abstractions.NullLoggerFactory.Instance) + .CreateLogger("ZB.MOM.WW.OtOpcUa.Cluster.ClusterBootstrapFallback"); + builder.AddStartup((system, _) => + { + ClusterBootstrapFallback.Arm(system, options, fallbackLogger); + return Task.CompletedTask; + }); +``` + +(Akka.Hosting 1.5.62 has `AddStartup`. If the overload signature differs, alternative: register an `IHostedService` in `AddOtOpcUaCluster` mirroring `ActorSystemTerminationWatchdog`'s lazy-accessor pattern — but prefer `AddStartup` so test hosts arm it for free.) + +**Step 3:** `dotnet build src/Core/ZB.MOM.WW.OtOpcUa.Cluster` — 0 warnings. + +**Step 4: Commit** + +```bash +git add -A && git commit -m "feat(cluster): InitJoin self-form fallback armed via Akka.Hosting startup task" +``` + +--- + +### Task 3 (B3): `SelfFormBootstrapTests` — 4 scenarios through the production bootstrap + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 4 + +**Files:** +- Create: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SelfFormBootstrapTests.cs` + +**Step 1: Write the tests** — follow `SplitBrainResolverActivationTests`' pattern (real `Host.CreateDefaultBuilder()` + `AddAkka` + `WithOtOpcUaClusterBootstrap`; that file's remarks explain why testing through the production bootstrap is mandatory — a test that re-creates the wiring pins the test's wiring, not the shipped one). Helper + scenarios: + +```csharp +public sealed class SelfFormBootstrapTests +{ + private static int FreePort() + { + var l = new System.Net.Sockets.TcpListener(System.Net.IPAddress.Loopback, 0); + l.Start(); + var p = ((System.Net.IPEndPoint)l.LocalEndpoint).Port; + l.Stop(); + return p; + } + + private static async Task StartNodeAsync( + string systemName, int selfPort, int peerPort, TimeSpan? selfFormAfter, bool selfInSeeds = true) + { + var options = new AkkaClusterOptions + { + SystemName = systemName, Hostname = "127.0.0.1", PublicHostname = "127.0.0.1", + Port = selfPort, Roles = new[] { "admin", "driver" }, + SelfFormAfter = selfFormAfter, + SeedNodes = selfInSeeds + ? new[] { $"akka.tcp://{systemName}@127.0.0.1:{peerPort}", + $"akka.tcp://{systemName}@127.0.0.1:{selfPort}" } // self SECOND — only the fallback can form it + : new[] { $"akka.tcp://{systemName}@127.0.0.1:{peerPort}" }, + }; + var builder = Host.CreateDefaultBuilder(); + builder.ConfigureServices(services => + { + services.AddSingleton>(Options.Create(options)); + services.AddAkka(systemName, (ab, sp) => ab.WithOtOpcUaClusterBootstrap(sp)); + }); + var host = builder.Build(); + await host.StartAsync(); + return host; + } + + // 1) SelfFormAfter_defaults_to_ten_seconds + // new AkkaClusterOptions().SelfFormAfter.ShouldBe(TimeSpan.FromSeconds(10)); + // 2) Lone_non_first_seed_self_forms_after_the_window + // dead peer port, window 2s → poll Cluster.Get(system).State until 1 Up member ≤ 20s. + // 3) Disabled_fallback_keeps_waiting (window null → State.Members empty after 6s; + // POSITIVE CONTROL: cluster.Join(cluster.SelfAddress) → 1 Up ≤ 20s — proves the node + // was formable and only the fallback was absent.) + // 4) Site_node_seeded_only_by_central_never_self_forms (selfInSeeds:false, window 1s → + // Members empty after 5s; positive control join. THE guard test: this is the exact + // docker-dev topology — site nodes list only central-1.) +} +``` + +Use a unique `systemName` per test (`otopcua-selfform-1` …) so concurrent hosts cannot cross-join; dispose each host in `finally` (`await host.StopAsync(); host.Dispose();`). + +**Step 2: Prove the key assertion has teeth** — before relying on scenario 2, temporarily run it with `selfFormAfter: null` and confirm it times out (red), then restore. Record that in the task notes. + +**Step 3: Run** + +```bash +dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests --filter "FullyQualifiedName~SelfFormBootstrap" +``` +Expected: 4 PASS. + +**Step 4: Commit** + +```bash +git add -A && git commit -m "test(cluster): self-form fallback — lone-seed forms, disabled waits, site-node guard" +``` + +--- + +### Task 4 (B4): Docs — Redundancy.md + mesh design note + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** Task 3 + +**Files:** +- Modify: `docs/Redundancy.md` (~lines 162, 244–247) — replace the bootstrap-constraint text ("Auto-down removes the crash outage, not this one") with `SelfFormAfter` semantics, the 10 s default, the site-node island guard (why driver-only nodes seeded by central-1 deliberately do NOT self-form under the CURRENT one-mesh topology), and the note that under the future per-cluster mesh each pair node is a seed so the fallback covers both. +- Modify: `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2/§7 — one-line note: the "registered outage gap" is now closed by `SelfFormAfter` (link Redundancy.md). +- Modify: `CLAUDE.md` — brief note in the redundancy-corrections area. + +Commit: `docs(redundancy): SelfFormAfter closes the seed-node bootstrap gap`. + +--- + +### Task 5 (B5): Full verification + optional docker-dev live check + +**Classification:** standard +**Estimated implement time:** ~5 min (suite runtime dominates) +**Parallelizable with:** none + +```bash +cd ~/Desktop/OtOpcUa +dotnet build ZB.MOM.WW.OtOpcUa.slnx # 0 warnings +dotnet test ZB.MOM.WW.OtOpcUa.slnx # green vs the pre-existing baseline +``` + +Optional live check (docker-dev six-node mesh): `docker compose stop central-1 central-2 && docker compose start central-2` → central-2 (a listed seed) self-forms in ~10 s; site nodes (non-seeds) keep waiting — the island guard on the real topology. Then `docker compose start central-1` → joins central-2's cluster, mesh re-forms. If the rig isn't running, record deferred-live. + +--- + +### Task 6 (D4): Manual failover service + cluster-level test + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** none + +**Files:** +- Create: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/IManualFailoverService.cs` +- Create: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/ManualFailoverService.cs` +- Modify: the ControlPlane/Host DI registration point (find where `RedundancyStateActor`-adjacent services register; admin-role branch of `Program.cs`) +- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/ManualFailoverServiceTests.cs` + +**Step 1: Failing tests** — reuse the two-node fixture pattern from `RedundancyPrimaryElectionTests` (node A older on the **higher** port so age ≠ address order; that fixture even has a guard test asserting the divergence — keep it honest here too): + +- `FailOver_targets_the_oldest_driver_member_not_the_role_leader`: service picks node A (oldest, higher port) even though B is `RoleLeader("driver")`; A's system terminates (graceful exit); B's next `RedundancyStateActor` snapshot names B Primary. +- `FailOver_refuses_when_no_driver_peer_exists`: single driver member → returns null, no Leave, node still Up (positive assert). +- `Parity_with_SelectDriverPrimary`: on the same fixture, the service's target equals the member `RedundancyStateActor` names Primary — pins the two 5-line queries together. + +**Step 2: Implement** — same shape as ScadaBridge's `FailOverCore` but role `"driver"`: + +```csharp +public static Address? FailOverCore(ActorSystem system, bool dryRun = false) +{ + var cluster = Akka.Cluster.Cluster.Get(system); + // Intentionally identical to RedundancyStateActor.SelectDriverPrimary (oldest Up driver, + // Member.AgeOrdering) — the node acted on must be exactly the one the election names. + // Pinned by ManualFailoverServiceTests.Parity_with_SelectDriverPrimary. + var drivers = cluster.State.Members + .Where(m => m.Status == MemberStatus.Up && m.Roles.Contains("driver")) + .OrderBy(m => m, Member.AgeOrdering) + .ToList(); + if (drivers.Count < 2) return null; + var primary = drivers[0]; + if (!dryRun) cluster.Leave(primary.Address); + return primary.Address; +} +``` + +Async wrapper `FailOverDriverPrimaryAsync(string actor)`: dry-run peer guard → audit via the shared `ZB.MOM.WW.Audit.IAuditWriter` seam (action `cluster.manual-failover`, actor, target in `DetailsJson` — copy the call shape from an existing audited AdminUI admin action) → real Leave → warn log → return address string. + +**Step 3:** Tests PASS → commit `feat(redundancy): manual failover service — graceful Leave of the driver Primary`. + +--- + +### Task 7 (D5): ClusterRedundancy page — live panel + failover button + bUnit tests + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** none + +**Files:** +- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/Clusters/ClusterRedundancy.razor` (page is already `@rendermode RenderMode.InteractiveServer` — no rendermode work; and it lives in the AdminUI host, not an RCL, so the static-router wrapper concern from lmxopcua#483 does not apply) +- Test: the AdminUI bUnit suite (place beside the existing Clusters page tests) +- Modify: `docs/Redundancy.md` — "Manual failover" subsection (button semantics, mesh-wide scope caveat) + +**Step 1: Failing bUnit tests:** (a) button hidden under the read-only policy (`AdminUiPolicies.AuthenticatedRead`), visible under the admin/write policy — check `AdminUiPolicies` for the exact policy the mutating Clusters pages use (e.g. `ClusterEdit.razor`) and use the same one; (b) disabled with tooltip when the live snapshot shows <2 Up driver members; (c) confirm flow calls a fake `IManualFailoverService` exactly once. + +**Step 2: Implement.** Add a "Live redundancy" panel above the static config table: current driver Primary + Up driver members from a `GetSnapshot()` read-only companion on `IManualFailoverService` (sourced from live `Cluster.State`; do NOT subscribe to the DPS topic for v1), the mesh-scope notice ("in the current single-mesh topology this acts on the whole mesh's Primary"), and the admin-gated **Trigger failover** button with confirm dialog (states: the Primary node restarts, ServiceLevel 250 moves to the survivor, OPC UA clients re-select). On confirm: call the service with the authenticated user name; surface the returned target address. + +**Step 3:** Tests PASS → commit `feat(adminui): manual failover control on the cluster redundancy page`. + +**Live check (fold into the Task 5 gate when docker-dev is up):** press the button, ServiceLevel 250 moves (old Primary drops off, survivor publishes 250), denial meters stay quiet, audit event recorded. + +--- + +## Reserved for additions (this plan is expected to grow) + +Append new tasks below as **Task 8+** (and extend `.tasks.json`) — do not renumber Tasks 1–7. Candidate items already flagged in the family docs, awaiting a decision on which to include: + +- **Configurable downing window** — promote `ServiceCollectionExtensions.DowningStableAfter` (hard-coded 15 s) to `AkkaClusterOptions` (e.g. `Cluster:DowningStableAfter`), keeping the pinned invariant window > `acceptable-heartbeat-pause` (`akka_failover.md` §6.2 calls this out; ScadaBridge's equivalent `Cluster:StableAfter` is already config). +- **Failure-detector tuning for faster failover** — expose `heartbeat-interval` / `acceptable-heartbeat-pause` (currently fixed in `Resources/akka.conf`) as options, per the availability-first tuning table in `akka_failover.md` §2. +- ~~Auto-down 1-vs-1 live gate~~ / ~~per-cluster mesh interplay~~ — **now owned by the per-cluster mesh PROGRAM plan** (`docs/plans/2026-07-22-per-cluster-mesh-program.md`, added 2026-07-22): Phases 1–7 align OtOpcUa's mesh with ScadaBridge's shape (one 2-node mesh per application Cluster, co-located on the ScadaBridge VMs). That program lists THIS plan as its prerequisite (execute this one first); its Phase 6 removes the ClusterRedundancy mesh-scope caveat + the site-node island-guard docs note, and Phase 7 closes the auto-down 1-vs-1 and `SelfFormAfter` live gates on the real per-pair topology. +- *(add further items here)* + +--- + +## Completion + +- Merge decision via the finishing-a-development-branch flow (family convention: ff-merge to `master` + push to gitea `lmxopcua`, or PR — ask the user). +- After merge: update `scadaproj/akka_failover.md` §6.1 status, `scadaproj/CLAUDE.md` index row, and memory `ha-availability-over-partition-safety` (tracked in the scadaproj index plan). +- Verification-before-completion applies throughout; live gates may be recorded deferred-live if docker-dev is down. diff --git a/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md.tasks.json b/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md.tasks.json new file mode 100644 index 00000000..2ca29190 --- /dev/null +++ b/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md.tasks.json @@ -0,0 +1,13 @@ +{ + "planPath": "docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md", + "tasks": [ + {"id": 0, "subject": "Task 1: SelfFormAfter on AkkaClusterOptions", "status": "completed"}, + {"id": 1, "subject": "Task 2: ClusterBootstrapFallback + AddStartup wiring", "status": "completed", "blockedBy": [0]}, + {"id": 2, "subject": "Task 3: SelfFormBootstrapTests — 4 scenarios incl. site-node island guard", "status": "completed", "blockedBy": [1], "notes": "Teeth proven: scenario 2 run with selfFormAfter:null failed (red) before restore."}, + {"id": 3, "subject": "Task 4: Docs — Redundancy.md bootstrap section + mesh design note", "status": "pending", "blockedBy": [1]}, + {"id": 4, "subject": "Task 5: Full verification + optional docker-dev live check", "status": "pending", "blockedBy": [2, 3]}, + {"id": 5, "subject": "Task 6: Manual failover service + cluster-level test", "status": "pending", "blockedBy": [4]}, + {"id": 6, "subject": "Task 7: ClusterRedundancy live panel + failover button + bUnit tests", "status": "pending", "blockedBy": [5]} + ], + "lastUpdated": "2026-07-22T00:00:00Z" +}