diff --git a/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md b/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md new file mode 100644 index 00000000..d0815009 --- /dev/null +++ b/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md @@ -0,0 +1,661 @@ +# ScadaBridge: InitJoin Self-Form Fallback + Manual Failover Control — Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. +> +> Shared cross-repo design: `~/Desktop/scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md` (design rationale, MNTR assessment, behavior spec). The OtOpcUa half lives in `~/Desktop/OtOpcUa/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`. This plan is self-contained for execution. + +**Goal:** (1) Either node of a 2-node ScadaBridge cluster can cold-start alone and become operational, unattended. (2) An admin-only "Trigger failover" control on the Health page performs a graceful, audited role swap of the central pair. + +> ## ⚠️ ARCHITECTURE REVISED DURING EXECUTION (2026-07-22) +> +> **Part 1 shipped as self-first seed ordering, NOT the `SelfFormAfter` watchdog described below.** Tasks 1–7 as originally written are superseded; what was actually built is in "Part 1 as executed". Tasks 8–10 (manual failover) are unaffected and still apply as written. +> +> **Why.** A code review of Task 2 raised, and a written test then confirmed, that the watchdog's success signal ("am I `Up` within the window?") cannot distinguish *no seed answered InitJoin* from *a seed answered and the join is in flight* — it sits outside Akka's join handshake. On a routine standby restart the peer is alive but the join stalls behind removal of the restarting node's own stale incarnation; a `Join(self)` issued during `TryingToJoin` abandons the in-flight join and forms a second cluster at the same address. **Measured: a permanent split, still unhealed after 90 s** — converting a routine restart into an outage of the previously-healthy node. That is strictly worse than the gap being closed, and it is not the boot-partition trade the design accepted. +> +> Akka's own `FirstSeedNodeProcess` already implements exactly the intended semantics — InitJoin the other seeds, self-join only if nobody answers — and, being part of the handshake, has no such race. It runs only when `seed-nodes[0]` is the node's own address. So the fix is seed **ordering**, not new runtime code. +> +> The plan's stated safety property — "a booting node only self-forms when NO seed answers InitJoin" — is true of Akka's native first-seed rule and **false** of the watchdog. That claim also appears in the shared cross-repo design doc (`scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md`) and in the OtOpcUa half; **both still need correcting** (owner deferred, 2026-07-22). + +## Part 1 as executed — self-first seed ordering + +**Architecture:** Every node lists ITSELF as `seed-nodes[0]` and its partner second. No new runtime code, no timer, no new option. `StartupValidator` enforces the ordering at boot (host **and** port comparison) because a broken ordering fails silently. Manual failover (Part 2) is unchanged: graceful `Cluster.Leave(oldest Up member)` via a new `IManualFailoverService` (CentralUI seam, Host implementation) — singleton drain, watchdog process-exit, supervisor restart, rejoin as youngest. + +**Behavior** (all rows covered by `SelfFirstSeedBootstrapTests` — real in-process clusters from production `BuildHocon` at production failure-detection timings): + +| Scenario | Behavior | +|---|---| +| Lone cold-start, peer dead | Self-joins after `seed-node-timeout` (~5 s) — operational, unattended | +| Restart into a **live** peer | Peer answers `InitJoinAck`; node rejoins, never islands | +| Both cold-start simultaneously, mutually reachable | `InitJoin` handshake converges them → **one** 2-member cluster | +| Both cold-start during a genuine boot **partition** | Each forms its own cluster — same dual-active class `auto-down` already accepts | +| Peer-first ordering (the old config) | Never forms — retained as a falsifiability control in the test suite | + +**Shipped:** +- 6 node appsettings reordered (the `*-node-b` configs; `-a` nodes were already self-first). All 14 satisfy the invariant. +- `StartupValidator` self-first rule + 3 tests; `SelfFirstSeedBootstrapTests` (4 tests). +- Docs corrected: `docker/README.md`, `docs/requirements/Component-ClusterInfrastructure.md` (new **Seed Node Ordering** section), `docs/components/ClusterInfrastructure.md` (3 passages), `docs/deployment/topology-guide.md` (incl. the stale keep-oldest claim Task 6 flagged), `CLAUDE.md`. +- Several docs had asserted self-first ordering was *unsafe* because a simultaneous cold start would produce two clusters that never merge. Disproved by test (row 3) and corrected. + +**⚠️ Ops action:** the gitignored `deploy/wonder-app-vd03/` overlay must have its `SeedNodes` reordered self-first before its next deploy, or the node will now **refuse to boot**. The validator rule is a hard gate deliberately — the alternative is the silent wedge it replaces. + +--- + +
+Original Part 1 architecture (SUPERSEDED — kept for the decision record) + +**Architecture:** New `ClusterOptions.SelfFormAfter` (`TimeSpan?`, default 10 s, `null`/`≤0` disables, appsettings-bound) arms `ClusterBootstrapFallback` right after ActorSystem creation: wait for membership via `RegisterOnMemberUp`; on expiry, `Cluster.Join(SelfAddress)`. Safety gate: fires only when this node's own address is in its own seed list. + +
+ +**Tech Stack:** .NET 10, Akka.NET 1.5.62, Blazor Server (CentralUI), bUnit, xunit. No new packages. + +**Branch:** `feat/selfform-fallback` off `main`. + +--- + +## Design essentials (from the shared design doc) + +**The defect:** Akka only lets the FIRST listed seed self-join to form a *new* cluster; every other node loops on `InitJoin` forever. Both docker central nodes list `central-a` first, so a lone cold-starting `central-b` never comes Up (the "registered outage gap" — `docker/README.md:289`). `ClusterOptions.SeedNodes`' doc comment claims "either can start first", which the deployed configs do not deliver — this plan makes it true and fixes the comment. + +**Behavior spec:** + +| Scenario | Behavior with fallback | +|---|---| +| Peer alive (any boot order) | Normal seed join in ms — fallback never fires | +| Lone cold-start, self IS in own seed list | After `SelfFormAfter`: warn log + `Cluster.Join(SelfAddress)` → Up alone, singletons start (`min-nr-of-members=1`) | +| Lone cold-start, self NOT in own seed list | Fallback inert (info log) — self-forming would island the node from the real seeds | +| Peer boots after survivor self-formed | Peer's InitJoin is answered → joins as youngest/standby. No island. | +| Both cold-start simultaneously, mutually unreachable | Both self-form → dual-active (same partition class auto-down accepts; restart one side) | +| `SelfFormAfter` null/`≤0` | Disabled — today's wait-forever behavior | +| Window expires mid-join-handshake | Benign: Akka ignores `Join` once joined | + +**Manual failover rules:** graceful `Leave`, never `Down`; admin-only (`AuthorizationPolicies.RequireAdmin`); peer guard (disabled when <2 Up `Central` members); confirmation dialog warning the Blazor circuit will drop (Traefik routes the UI to the active node — triggering failover disconnects your own page, which reconnects against the new active); audited via the app's **central** audit writer (`ICentralAuditWriter` — NOT the shared seam, see the dual-seam gotcha) before the Leave is issued. No interplay with `SelfFormAfter` (the peer is alive on this path, so the restarted node rejoins normally). + +**Multi-node TestKit:** assessed and NOT used — in-process real clusters via `TwoNodeClusterFixture` (production `BuildHocon`) cover every deterministic scenario; MNTR would need a dedicated no-parallelization test project for no added coverage. See the shared design doc for the full verdict. + +--- + +### Task 1 (A1): `SelfFormAfter` option + validator + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** none (first task) + +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptionsValidator.cs` +- Test: `tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsTests.cs` +- Test: `tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsValidatorTests.cs` + +**Step 1: Create the branch** + +```bash +cd ~/Desktop/ScadaBridge && git checkout main && git checkout -b feat/selfform-fallback +``` + +**Step 2: Write the failing tests** (append to the existing test classes, matching their assertion style — read them first) + +```csharp +// ClusterOptionsTests.cs +[Fact] +public void SelfFormAfter_defaults_to_ten_seconds() +{ + new ClusterOptions().SelfFormAfter.ShouldBe(TimeSpan.FromSeconds(10)); +} + +// ClusterOptionsValidatorTests.cs +[Fact] public void SelfFormAfter_null_passes_validation() { /* valid options + null → Succeeded */ } +[Fact] public void SelfFormAfter_zero_passes_validation() { /* zero = explicit disable → Succeeded */ } +[Fact] public void SelfFormAfter_negative_fails_validation() { /* -1s → Failed, message mentions SelfFormAfter */ } +``` + +**Step 3: Run to verify failure** + +```bash +dotnet test tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests --filter "FullyQualifiedName~SelfFormAfter" +``` +Expected: FAIL (compile error — property doesn't exist). + +**Step 4: Implement.** `ClusterOptions.cs` — add after `AllowSingleNodeCluster` (line 104): + +```csharp + /// + /// Bootstrap self-form fallback window (decision 2026-07-22, scadaproj/akka_failover.md §6.1). + /// Akka only lets the FIRST listed seed form a new cluster; a non-first seed cold-starting + /// while its peer is down loops on InitJoin forever. When this node has waited longer than + /// this window without becoming a cluster member, it forms a cluster on itself + /// (Cluster.Join(SelfAddress)) — but ONLY if its own address is in its seed list + /// (a non-seed node self-forming would create a permanent island). Default 10s: the pair + /// shares a datacenter, so a live peer answers InitJoin in milliseconds and waiting longer + /// buys nothing. null or a non-positive value disables the fallback (wait-forever). + /// Accepted trade: both nodes cold-starting within the window while mutually unreachable + /// form two clusters — the same partition class the auto-down strategy already accepts. + /// + public TimeSpan? SelfFormAfter { get; set; } = TimeSpan.FromSeconds(10); +``` + +`ClusterOptionsValidator.cs` — inside `Validate`, after the `FailureDetectionThreshold` rules: + +```csharp + builder.RequireThat(options.SelfFormAfter is null || options.SelfFormAfter.Value >= TimeSpan.Zero, + "ClusterOptions.SelfFormAfter must be null (disabled), zero (disabled) or a positive duration; " + + "a negative value is always a configuration mistake."); +``` + +**Step 5: Run tests → PASS** + +```bash +dotnet test tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests +``` + +**Step 6: Commit** + +```bash +git add -A && git commit -m "feat(cluster): SelfFormAfter option — bootstrap self-form fallback window" +``` + +--- + +### Task 2 (A2): `ClusterBootstrapFallback` + first integration test + +**Classification:** high-risk (cluster formation behavior) +**Estimated implement time:** ~5 min +**Parallelizable with:** none + +**Files:** +- Create: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/ClusterBootstrapFallback.cs` +- Create: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SelfFormBootstrapTests.cs` + +**Step 1: Write the failing test** + +```csharp +using Akka.Actor; +using Akka.Cluster; +using Akka.Configuration; +using Microsoft.Extensions.Logging.Abstractions; +using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure; +using ZB.MOM.WW.ScadaBridge.Host; +using ZB.MOM.WW.ScadaBridge.Host.Actors; + +namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster; + +/// +/// Guards the InitJoin self-form fallback (decision 2026-07-22): Akka only lets the FIRST +/// listed seed form a new cluster, so without the fallback a non-first seed cold-starting +/// alone waits on InitJoin forever — the "registered outage gap". These tests build REAL +/// single/dual-node clusters from the production BuildHocon output, exactly like +/// TwoNodeClusterFixture, and arm the production fallback. +/// +public sealed class SelfFormBootstrapTests : IAsyncLifetime +{ + private readonly List _systems = new(); + + /// Starts a node whose seed list puts the PEER first (self second, or absent), + /// so Akka's own first-seed rule can never self-form it — only the fallback can. + private ActorSystem StartNode(int selfPort, int peerPort, TimeSpan? selfFormAfter, bool selfInSeeds = true) + { + var nodeOptions = new NodeOptions { Role = "Central", NodeHostname = "127.0.0.1", RemotingPort = selfPort }; + var clusterOptions = new ClusterOptions + { + SeedNodes = selfInSeeds + ? new List + { + $"akka.tcp://scadabridge@127.0.0.1:{peerPort}", + $"akka.tcp://scadabridge@127.0.0.1:{selfPort}", + } + : new List { $"akka.tcp://scadabridge@127.0.0.1:{peerPort}" }, + SelfFormAfter = selfFormAfter, + StableAfter = TimeSpan.FromSeconds(3), + HeartbeatInterval = TimeSpan.FromMilliseconds(500), + FailureDetectionThreshold = TimeSpan.FromSeconds(2), + MinNrOfMembers = 1, + AllowSingleNodeCluster = !selfInSeeds, + }; + var hocon = AkkaHostedService.BuildHocon( + nodeOptions, clusterOptions, new[] { "Central" }, + TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(3)); + var system = ActorSystem.Create("scadabridge", ConfigurationFactory.ParseString(hocon)); + _systems.Add(system); + ClusterBootstrapFallback.Arm(system, clusterOptions, NullLogger.Instance); + return system; + } + + [Fact] + public async Task Lone_non_first_seed_self_forms_after_the_window() + { + var selfPort = TwoNodeClusterFixture.GetFreeTcpPort(); + var deadPeerPort = TwoNodeClusterFixture.GetFreeTcpPort(); // nothing listening + var node = StartNode(selfPort, deadPeerPort, selfFormAfter: TimeSpan.FromSeconds(2)); + + // Without the fallback this waits forever (Akka first-seed rule). With it, the node + // must be a 1-member Up cluster shortly after the 2s window. + await TwoNodeClusterFixture.WaitForMembersUp(node, 1, TimeSpan.FromSeconds(20)); + Cluster.Get(node).SelfMember.Status.ShouldBe(MemberStatus.Up); + } + + public Task InitializeAsync() => Task.CompletedTask; + + public async Task DisposeAsync() + { + foreach (var s in _systems) + { + try { await s.Terminate().WaitAsync(TimeSpan.FromSeconds(10)); } catch { /* teardown */ } + } + } +} +``` + +(Match the assertion library to `SbrFailoverTests.cs` — Shouldly vs xunit `Assert`.) + +**Step 2: Run to verify failure** — compile error (`ClusterBootstrapFallback` doesn't exist): + +```bash +dotnet test tests/ZB.MOM.WW.ScadaBridge.IntegrationTests --filter "FullyQualifiedName~SelfFormBootstrapTests" +``` + +**Step 3: Implement `ClusterBootstrapFallback.cs`** + +```csharp +using Akka.Actor; +using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure; + +namespace ZB.MOM.WW.ScadaBridge.Host.Actors; + +/// +/// InitJoin self-form fallback (decision 2026-07-22, scadaproj/akka_failover.md §6.1). +/// Akka only lets the FIRST listed seed form a NEW cluster; every other node retries InitJoin +/// forever. So "both nodes are seed nodes" (ClusterOptions.SeedNodes) does NOT mean either can +/// cold-start alone — a non-first seed booting while its peer is down waits indefinitely (the +/// "registered outage gap", docker/README.md). This watchdog waits +/// for membership; on expiry it forms a cluster on itself. +/// +/// Island safety. Fires ONLY when this node's own address is in its own seed list. +/// A node that is not a seed (never legitimately first) must keep waiting: if it self-formed, +/// a later-booting real seed would form a second cluster and the two can never merge. For nodes +/// that ARE seeds, sequential recovery is island-free — Akka's join protocol prefers an existing +/// cluster (a booting node only self-forms when NO seed answers InitJoin), so a peer booting +/// after this node self-formed simply joins it. +/// +/// Races are benign. If the join completes between window expiry and +/// Cluster.Join(SelfAddress), Akka ignores the join — a node joins a cluster at most once +/// per incarnation. The residual risk is both nodes cold-starting inside the window while +/// mutually unreachable (a boot-time partition): both self-form, the same dual-active class the +/// auto-down downing strategy already accepts, with the same recovery (restart one side). +/// +public static class ClusterBootstrapFallback +{ + public static void Arm(ActorSystem system, ClusterOptions options, ILogger logger) + { + if (options.SelfFormAfter is not { } window || window <= TimeSpan.Zero) + { + logger.LogInformation( + "Cluster self-form fallback disabled (SelfFormAfter not set) — a node cold-starting " + + "while its peer is down will wait on InitJoin indefinitely."); + return; + } + + var cluster = Akka.Cluster.Cluster.Get(system); + var self = cluster.SelfAddress; + var isSeed = options.SeedNodes.Any(s => TryParseAddress(s, out var a) && a.Equals(self)); + if (!isSeed) + { + logger.LogInformation( + "Cluster self-form fallback inactive: this node ({Self}) is not in its own seed list " + + "[{Seeds}] — self-forming here would island it from the real seeds.", + self, string.Join(", ", options.SeedNodes)); + return; + } + + var joined = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + cluster.RegisterOnMemberUp(() => joined.TrySetResult()); + + _ = Task.Run(async () => + { + var winner = await Task.WhenAny(joined.Task, Task.Delay(window)); + if (winner == joined.Task || system.WhenTerminated.IsCompleted) + return; + logger.LogWarning( + "No cluster membership after {Window} — no seed answered InitJoin (peer down at boot). " + + "Self-forming a cluster at {Self} so this node becomes operational; if the peer was " + + "merely partitioned (not dead), the pair is now dual-active — restart one side after " + + "the partition heals (accepted availability-first trade, decision 2026-07-22).", + window, self); + cluster.Join(self); + }); + } + + private static bool TryParseAddress(string seed, out Address address) + { + try { address = Address.Parse(seed); return true; } + catch { address = default!; return false; } + } +} +``` + +(Add the `Microsoft.Extensions.Logging` using. If non-generic `TaskCompletionSource` is unavailable, use `TaskCompletionSource` + `TrySetResult(true)`.) + +**Step 4: Run → PASS** (~5–10 s). **Step 5: Commit** + +```bash +git add -A && git commit -m "feat(cluster): InitJoin self-form fallback — lone non-first seed becomes Up" +``` + +--- + +### Task 3 (A3): Remaining fallback tests — disabled / late-peer merge / non-seed + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 4 + +**Files:** +- Modify: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SelfFormBootstrapTests.cs` + +**Step 1: Append three tests** (each absence assertion carries an in-test positive control — repo convention): + +```csharp + [Fact] + public async Task Disabled_fallback_keeps_waiting_and_the_node_was_otherwise_formable() + { + var selfPort = TwoNodeClusterFixture.GetFreeTcpPort(); + var deadPeerPort = TwoNodeClusterFixture.GetFreeTcpPort(); + var node = StartNode(selfPort, deadPeerPort, selfFormAfter: null); + + await Task.Delay(TimeSpan.FromSeconds(6)); // 3x the window used in the enabled test + var cluster = Cluster.Get(node); + cluster.State.Members.ShouldBeEmpty(); // still InitJoin-looping — today's behavior + + // POSITIVE CONTROL: prove the node COULD have formed; only the fallback was missing. + cluster.Join(cluster.SelfAddress); + await TwoNodeClusterFixture.WaitForMembersUp(node, 1, TimeSpan.FromSeconds(20)); + } + + [Fact] + public async Task Peer_booting_after_self_form_joins_the_existing_cluster_no_island() + { + var portA = TwoNodeClusterFixture.GetFreeTcpPort(); + var portB = TwoNodeClusterFixture.GetFreeTcpPort(); + + // B cold-starts alone (A dead), self-forms after 2s. + var nodeB = StartNode(portB, peerPort: portA, selfFormAfter: TimeSpan.FromSeconds(2)); + await TwoNodeClusterFixture.WaitForMembersUp(nodeB, 1, TimeSpan.FromSeconds(20)); + + // A boots later with the pair seed list. B answers InitJoin, so A must JOIN B's + // cluster instead of islanding. Generous window on A so its fallback can't race. + var nodeA = StartNode(portA, peerPort: portB, selfFormAfter: TimeSpan.FromSeconds(30)); + await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 2, TimeSpan.FromSeconds(20)); + await TwoNodeClusterFixture.WaitForMembersUp(nodeB, 2, TimeSpan.FromSeconds(20)); + } + + [Fact] + public async Task Non_seed_node_never_self_forms() + { + var selfPort = TwoNodeClusterFixture.GetFreeTcpPort(); + var deadSeedPort = TwoNodeClusterFixture.GetFreeTcpPort(); + var node = StartNode(selfPort, deadSeedPort, selfFormAfter: TimeSpan.FromSeconds(1), selfInSeeds: false); + + await Task.Delay(TimeSpan.FromSeconds(5)); // 5x the window + var cluster = Cluster.Get(node); + cluster.State.Members.ShouldBeEmpty(); // guard refused to island a non-seed + + // Positive control: the guard (not the environment) prevented formation. + cluster.Join(cluster.SelfAddress); + await TwoNodeClusterFixture.WaitForMembersUp(node, 1, TimeSpan.FromSeconds(20)); + } +``` + +**Step 2: Run** — all 4 PASS. **Step 3: Commit** `test(cluster): self-form fallback — disabled, late-peer merge, non-seed guard`. + +--- + +### Task 4 (A4): Production wiring + fix the misleading `SeedNodes` doc comment + +**Classification:** small +**Estimated implement time:** ~3 min +**Parallelizable with:** Task 3 + +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs` (inside `GetOrCreateActorSystem`, after the `WhenTerminated` continuation ending ~line 218, before `_actorSystem = system;`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs:33-38` + +**Step 1: Wire the fallback:** + +```csharp + // InitJoin self-form fallback (decision 2026-07-22): without it a non-first seed + // cold-starting while its peer is down loops on InitJoin forever — auto-down closed + // the crash-failover gap, this closes the cold-start-alone gap. Guarded inside Arm: + // disabled when SelfFormAfter is unset, inert when this node is not its own seed. + ClusterBootstrapFallback.Arm(system, _clusterOptions, _logger); +``` + +**Step 2: Fix the `SeedNodes` doc comment** (replace the ``): + +```csharp + /// + /// Akka.NET cluster seed nodes. Both nodes are seed nodes — each node lists itself and its + /// partner. NOTE: listing both is NOT sufficient for "either can start first": Akka only lets + /// the FIRST listed seed form a new cluster, so a lone non-first seed waits on InitJoin until + /// expires and the self-form fallback fires (decision 2026-07-22). + /// Must contain at least one entry. + /// +``` + +**Step 3: Build + targeted tests** + +```bash +dotnet build ZB.MOM.WW.ScadaBridge.slnx # 0 warnings (TreatWarningsAsErrors) +dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests +dotnet test tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests +``` + +**Step 4: Commit** `feat(cluster): arm self-form fallback at ActorSystem creation; honest SeedNodes doc`. + +--- + +### Task 5 (A5): appsettings sweep + +**Classification:** trivial +**Estimated implement time:** ~3 min +**Parallelizable with:** Task 6 + +Add `"SelfFormAfter": "00:00:10"` next to `"SplitBrainResolverStrategy"` in the `ScadaBridge:Cluster` section of each (explicit for operator visibility; matches the code default): + +- `src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Central.json`, `appsettings.Site.json` +- `docker/central-node-a/appsettings.Central.json`, `docker/central-node-b/appsettings.Central.json` +- `docker/site-a-node-a/appsettings.Site.json`, `docker/site-a-node-b/appsettings.Site.json` +- `docker/site-b-node-a/appsettings.Site.json`, `docker/site-b-node-b/appsettings.Site.json` +- `docker/site-c-node-a/appsettings.Site.json`, `docker/site-c-node-b/appsettings.Site.json` +- `docker-env2/` — the 4 node appsettings files + +Then `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests` and commit `config(cluster): SelfFormAfter=10s explicit in all node appsettings`. + +> Ops note (do NOT edit here): the gitignored `deploy/wonder-app-vd03/` overlay gets the same key on the next production deploy. + +--- + +### Task 6 (A6): Docs + +**Classification:** small +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 5 + +- `docs/requirements/Component-ClusterInfrastructure.md` — replace the "registered outage gap" recovery text (env-var override) with `SelfFormAfter` semantics + island guard + boot-partition trade. +- `docker/README.md` (~line 289) — same replacement; keep the partition-trade note. +- `docs/deployment/topology-guide.md:101` — **fix the stale keep-oldest claim** (still says "Keep-oldest with `down-if-alone = on`"): rewrite to auto-down default + `SelfFormAfter`, pointing at the decision records. +- `CLAUDE.md` (~line 222) — update the boot-order note: pairs no longer require the first seed for cold start; note the 10 s window. + +Commit: `docs(cluster): SelfFormAfter fallback; fix stale keep-oldest note in topology guide`. + +--- + +### Task 7 (A7): Full verification + optional docker live gate + +**Classification:** standard +**Estimated implement time:** ~5 min (suite runtime dominates) +**Parallelizable with:** none + +**Step 1:** + +```bash +cd ~/Desktop/ScadaBridge +dotnet build ZB.MOM.WW.ScadaBridge.slnx # expect 0 warnings +dotnet test ZB.MOM.WW.ScadaBridge.slnx # expect green vs pre-existing baseline +``` + +**Step 2 (LIVE GATE — run if the docker rig is available; else record deferred-live):** + +```bash +cd docker && bash deploy.sh # rebuild with the fallback +docker compose stop central-a central-b # verify service names in docker-compose.yml first +docker compose start central-b # cold-start ONLY the non-first seed +docker compose logs -f central-b | grep -m1 "Self-forming a cluster" # ≈10s after start +curl -fsS http://localhost:9002/health/active # expect 200 +docker compose start central-a # first seed returns → must JOIN, not island +docker compose logs central-a | grep -i "Welcome" +``` + +Expected: `central-b` self-forms ≈10 s and serves (previously a permanent wedge); `central-a` rejoins as youngest/standby. + +--- + +### Task 8 (D1): `IManualFailoverService` + cluster-level test + +**Classification:** high-risk (cluster behavior) +**Estimated implement time:** ~5 min +**Parallelizable with:** none + +**Files:** +- Create: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/IManualFailoverService.cs` +- Create: `src/ZB.MOM.WW.ScadaBridge.Host/Health/AkkaManualFailoverService.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` (register in the Central branch, next to `IActiveNodeGate` ~line 330) +- Test: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/ManualFailoverTests.cs` + +**Step 1: Failing tests** on `TwoNodeClusterFixture`: + +```csharp +[Fact] +public async Task Failover_makes_the_oldest_leave_and_the_survivor_take_over() +{ + await using var f = await TwoNodeClusterFixture.StartAsync(); + var oldest = Akka.Cluster.Cluster.Get(f.NodeA); // NodeA started first = oldest + var target = AkkaManualFailoverService.FailOverCore(f.NodeB, "Central"); // issued from the OTHER node + target.ShouldBe(oldest.SelfAddress); + + // Graceful exit path: the left node's own ActorSystem terminates… + await f.NodeA.WhenTerminated.WaitAsync(TimeSpan.FromSeconds(30)); + // …and the survivor becomes a 1-member cluster and the oldest-Up active node. + await TwoNodeClusterFixture.WaitForMemberRemoved(f.NodeB, oldest.SelfAddress, TimeSpan.FromSeconds(30)); + ActiveNodeEvaluator.SelfIsOldestUp(Akka.Cluster.Cluster.Get(f.NodeB)).ShouldBeTrue(); +} + +[Fact] +public void Failover_refuses_when_no_peer_exists() +{ /* 1-node cluster → FailOverCore returns null, node still Up afterwards (positive assert) */ } +``` + +**Step 2:** Run → FAIL (service missing). + +**Step 3: Implement.** Interface in CentralUI (CentralUI stays Akka-free): + +```csharp +public interface IManualFailoverService +{ + /// Gracefully fails over the central cluster: the current active (oldest Up) + /// member leaves, restarts via its supervisor, and rejoins as standby. Returns the + /// address string acted on, or null when there is no peer to fail over to. + Task FailOverCentralAsync(string actor); +} +``` + +Host implementation — static testable core + thin DI wrapper: + +```csharp +public sealed class AkkaManualFailoverService : IManualFailoverService +{ + // ctor: (AkkaHostedService akka, ICentralAuditWriter audit, ILogger logger) + + public async Task FailOverCentralAsync(string actor) + { + var system = _akka.GetOrCreateActorSystem(); + var target = FailOverCore(system, role: "Central", dryRun: true); + if (target is null) return null; // peer guard + await _audit.WriteAsync(/* canonical AuditEvent: Action=cluster.manual-failover, + Actor=actor, DetailsJson={"target": target}, Outcome=Success — copy the exact + call shape from an existing audited admin action (e.g. the Sites admin service); + use the CENTRAL audit writer, not the shared seam */); + FailOverCore(system, role: "Central"); // Cluster.Leave(target) + _logger.LogWarning("Manual failover triggered by {Actor}: {Target} is leaving the cluster.", actor, target); + return target.ToString(); + } + + /// Oldest Up member with the role leaves — mirrors ActiveNodeEvaluator's oldest-Up + /// rule so the node acted on is exactly the one hosting the singletons. Returns null when + /// fewer than 2 Up members carry the role (no peer = failover would be an outage). + public static Address? FailOverCore(ActorSystem system, string role, bool dryRun = false) + { + var cluster = Akka.Cluster.Cluster.Get(system); + var withRole = cluster.State.Members + .Where(m => m.Status == MemberStatus.Up && m.HasRole(role)) + .OrderBy(m => m, Member.AgeOrdering) + .ToList(); + if (withRole.Count < 2) return null; + var oldest = withRole[0]; + if (!dryRun) cluster.Leave(oldest.Address); + return oldest.Address; + } +} +``` + +**Step 4:** Register in the Central branch of `Program.cs`. **Step 5:** Tests PASS → commit `feat(ui): manual central failover service — graceful Leave of the oldest Up member`. + +--- + +### Task 9 (D2): Health page button + bUnit tests + runbook + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** none + +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor` (central-cluster card, near the Nodes column ~line 233) +- Test: `tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/HealthFailoverButtonTests.cs` (follow the project's existing bUnit page-test pattern) +- Modify: `docs/requirements/Component-ClusterInfrastructure.md` + `docker/README.md` — manual-failover runbook paragraph + +**Step 1: Failing bUnit tests:** (a) button absent without the admin policy; (b) present + enabled for admin with ≥2 online central nodes; (c) disabled with tooltip at 1 node; (d) confirm flow calls `IManualFailoverService.FailOverCentralAsync` exactly once (fake service). + +**Step 2: Implement:** + +```razor + + + +``` + +plus a confirmation modal (copy the page's existing dialog idiom). The warning text MUST state: the active node restarts, roles swap, and this page will briefly disconnect and reconnect against the new active node (Traefik routes to the active). On confirm: call the service with the authenticated user name; surface the returned target address. + +**Step 3:** Tests PASS → commit `feat(ui): admin manual-failover control on the health page`. + +**Live check (fold into the Task 7 gate when the rig is up):** press the button, watch `central-a` restart and `central-b`'s badge flip to Primary; audit row lands in `dbo.AuditLog`. + +--- + +### Task 10 (D3, OPTIONAL — confirm with the user before executing): site-pair failover from the central UI + +**Classification:** high-risk (new cross-cluster message contract) +**Estimated implement time:** exploration first; likely 2–3 tasks if approved + +The Health page also shows per-site node cards (Primary/Standby from heartbeats), but central and sites are **separate Akka clusters** — a site failover needs a `TriggerSiteFailover(siteId)` command over the existing central→site transport (same channel as the Retry/Discard relay), handled on the site's active node by `Cluster.Leave(SelfAddress)`. That adds a versioned message contract (rolling-upgrade surface). If approved: explore `ZB.MOM.WW.ScadaBridge.Communication` for the command path, mirror an existing command end-to-end, per-site button with the same guard/confirm/audit rules. Otherwise: file a follow-up issue and skip. + +--- + +## Completion + +- Merge decision via the finishing-a-development-branch flow (family convention: ff-merge to `main` + push to gitea, or PR — ask the user). +- After merge: update `scadaproj/akka_failover.md` §6.1 status, `scadaproj/CLAUDE.md` index row, and memory `ha-availability-over-partition-safety` (tracked as the family-docs task in the scadaproj index plan). +- Verification-before-completion applies throughout: no task is done without its command output; live gates may be recorded deferred-live if the rig is down. diff --git a/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md.tasks.json b/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md.tasks.json new file mode 100644 index 00000000..eef79770 --- /dev/null +++ b/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md.tasks.json @@ -0,0 +1,61 @@ +{ + "planPath": "docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md", + "tasks": [ + { + "id": 0, + "subject": "Part 1 (SUPERSEDED Tasks 1-4): SelfFormAfter watchdog - implemented, review+test rejected it, reverted", + "status": "completed" + }, + { + "id": 1, + "subject": "Part 1 as executed: self-first seed ordering + StartupValidator rule + SelfFirstSeedBootstrapTests + 14-config sweep + docs", + "status": "completed", + "blockedBy": [ + 0 + ] + }, + { + "id": 2, + "subject": "Task 7: full solution verification + optional docker live gate", + "status": "in_progress", + "blockedBy": [ + 1 + ] + }, + { + "id": 3, + "subject": "Task 8: IManualFailoverService + cluster-level test", + "status": "pending", + "blockedBy": [ + 2 + ] + }, + { + "id": 4, + "subject": "Task 9: Health page failover button + bUnit tests + runbook", + "status": "pending", + "blockedBy": [ + 3 + ] + }, + { + "id": 5, + "subject": "Task 10 (OPTIONAL, ask user): site-pair failover via central-to-site transport", + "status": "pending", + "blockedBy": [ + 4 + ] + }, + { + "id": 6, + "subject": "FOLLOW-UP (owner deferred): correct the false InitJoin safety claim in scadaproj + OtOpcUa design docs", + "status": "pending" + }, + { + "id": 7, + "subject": "OPS: reorder SeedNodes self-first in gitignored deploy/wonder-app-vd03/ before next deploy (hard boot gate)", + "status": "pending" + } + ], + "lastUpdated": "2026-07-22T00:00:00Z" +} \ No newline at end of file