# ScadaBridge: InitJoin Self-Form Fallback + Manual Failover Control — Implementation Plan > **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. > > Shared cross-repo design: `~/Desktop/scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md` (design rationale, MNTR assessment, behavior spec). The OtOpcUa half lives in `~/Desktop/OtOpcUa/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`. This plan is self-contained for execution. **Goal:** (1) Either node of a 2-node ScadaBridge cluster can cold-start alone and become operational, unattended. (2) An admin-only "Trigger failover" control on the Health page performs a graceful, audited role swap of the central pair. > ## ⚠️ ARCHITECTURE REVISED DURING EXECUTION (2026-07-22) > > **Part 1 shipped as self-first seed ordering, NOT the `SelfFormAfter` watchdog described below.** Tasks 1–7 as originally written are superseded; what was actually built is in "Part 1 as executed". Tasks 8–10 (manual failover) are unaffected and still apply as written. > > **Why.** A code review of Task 2 raised, and a written test then confirmed, that the watchdog's success signal ("am I `Up` within the window?") cannot distinguish *no seed answered InitJoin* from *a seed answered and the join is in flight* — it sits outside Akka's join handshake. On a routine standby restart the peer is alive but the join stalls behind removal of the restarting node's own stale incarnation; a `Join(self)` issued during `TryingToJoin` abandons the in-flight join and forms a second cluster at the same address. **Measured: a permanent split, still unhealed after 90 s** — converting a routine restart into an outage of the previously-healthy node. That is strictly worse than the gap being closed, and it is not the boot-partition trade the design accepted. > > Akka's own `FirstSeedNodeProcess` already implements exactly the intended semantics — InitJoin the other seeds, self-join only if nobody answers — and, being part of the handshake, has no such race. It runs only when `seed-nodes[0]` is the node's own address. So the fix is seed **ordering**, not new runtime code. > > The plan's stated safety property — "a booting node only self-forms when NO seed answers InitJoin" — is true of Akka's native first-seed rule and **false** of the watchdog. That claim also appears in the shared cross-repo design doc (`scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md`) and in the OtOpcUa half; **both still need correcting** (owner deferred, 2026-07-22). ## Part 1 as executed — self-first seed ordering **Architecture:** Every node lists ITSELF as `seed-nodes[0]` and its partner second. No new runtime code, no timer, no new option. `StartupValidator` enforces the ordering at boot (host **and** port comparison) because a broken ordering fails silently. Manual failover (Part 2) is unchanged: graceful `Cluster.Leave(oldest Up member)` via a new `IManualFailoverService` (CentralUI seam, Host implementation) — singleton drain, watchdog process-exit, supervisor restart, rejoin as youngest. **Behavior** (all rows covered by `SelfFirstSeedBootstrapTests` — real in-process clusters from production `BuildHocon` at production failure-detection timings): | Scenario | Behavior | |---|---| | Lone cold-start, peer dead | Self-joins after `seed-node-timeout` (~5 s) — operational, unattended | | Restart into a **live** peer | Peer answers `InitJoinAck`; node rejoins, never islands | | Both cold-start simultaneously, mutually reachable | `InitJoin` handshake converges them → **one** 2-member cluster | | Both cold-start during a genuine boot **partition** | Each forms its own cluster — same dual-active class `auto-down` already accepts | | Peer-first ordering (the old config) | Never forms — retained as a falsifiability control in the test suite | **Shipped:** - 6 node appsettings reordered (the `*-node-b` configs; `-a` nodes were already self-first). All 14 satisfy the invariant. - `StartupValidator` self-first rule + 3 tests; `SelfFirstSeedBootstrapTests` (4 tests). - Docs corrected: `docker/README.md`, `docs/requirements/Component-ClusterInfrastructure.md` (new **Seed Node Ordering** section), `docs/components/ClusterInfrastructure.md` (3 passages), `docs/deployment/topology-guide.md` (incl. the stale keep-oldest claim Task 6 flagged), `CLAUDE.md`. - Several docs had asserted self-first ordering was *unsafe* because a simultaneous cold start would produce two clusters that never merge. Disproved by test (row 3) and corrected. **⚠️ Ops action:** the gitignored `deploy/wonder-app-vd03/` overlay must have its `SeedNodes` reordered self-first before its next deploy, or the node will now **refuse to boot**. The validator rule is a hard gate deliberately — the alternative is the silent wedge it replaces. ---
Original Part 1 architecture (SUPERSEDED — kept for the decision record) **Architecture:** New `ClusterOptions.SelfFormAfter` (`TimeSpan?`, default 10 s, `null`/`≤0` disables, appsettings-bound) arms `ClusterBootstrapFallback` right after ActorSystem creation: wait for membership via `RegisterOnMemberUp`; on expiry, `Cluster.Join(SelfAddress)`. Safety gate: fires only when this node's own address is in its own seed list.
**Tech Stack:** .NET 10, Akka.NET 1.5.62, Blazor Server (CentralUI), bUnit, xunit. No new packages. **Branch:** `feat/selfform-fallback` off `main`. --- ## Design essentials (from the shared design doc) **The defect:** Akka only lets the FIRST listed seed self-join to form a *new* cluster; every other node loops on `InitJoin` forever. Both docker central nodes list `central-a` first, so a lone cold-starting `central-b` never comes Up (the "registered outage gap" — `docker/README.md:289`). `ClusterOptions.SeedNodes`' doc comment claims "either can start first", which the deployed configs do not deliver — this plan makes it true and fixes the comment. **Behavior spec:** | Scenario | Behavior with fallback | |---|---| | Peer alive (any boot order) | Normal seed join in ms — fallback never fires | | Lone cold-start, self IS in own seed list | After `SelfFormAfter`: warn log + `Cluster.Join(SelfAddress)` → Up alone, singletons start (`min-nr-of-members=1`) | | Lone cold-start, self NOT in own seed list | Fallback inert (info log) — self-forming would island the node from the real seeds | | Peer boots after survivor self-formed | Peer's InitJoin is answered → joins as youngest/standby. No island. | | Both cold-start simultaneously, mutually unreachable | Both self-form → dual-active (same partition class auto-down accepts; restart one side) | | `SelfFormAfter` null/`≤0` | Disabled — today's wait-forever behavior | | Window expires mid-join-handshake | Benign: Akka ignores `Join` once joined | **Manual failover rules:** graceful `Leave`, never `Down`; admin-only (`AuthorizationPolicies.RequireAdmin`); peer guard (disabled when <2 Up `Central` members); confirmation dialog warning the Blazor circuit will drop (Traefik routes the UI to the active node — triggering failover disconnects your own page, which reconnects against the new active); audited via the app's **central** audit writer (`ICentralAuditWriter` — NOT the shared seam, see the dual-seam gotcha) before the Leave is issued. No interplay with `SelfFormAfter` (the peer is alive on this path, so the restarted node rejoins normally). **Multi-node TestKit:** assessed and NOT used — in-process real clusters via `TwoNodeClusterFixture` (production `BuildHocon`) cover every deterministic scenario; MNTR would need a dedicated no-parallelization test project for no added coverage. See the shared design doc for the full verdict. --- ### Task 1 (A1): `SelfFormAfter` option + validator **Classification:** small **Estimated implement time:** ~4 min **Parallelizable with:** none (first task) **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs` - Modify: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptionsValidator.cs` - Test: `tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsTests.cs` - Test: `tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsValidatorTests.cs` **Step 1: Create the branch** ```bash cd ~/Desktop/ScadaBridge && git checkout main && git checkout -b feat/selfform-fallback ``` **Step 2: Write the failing tests** (append to the existing test classes, matching their assertion style — read them first) ```csharp // ClusterOptionsTests.cs [Fact] public void SelfFormAfter_defaults_to_ten_seconds() { new ClusterOptions().SelfFormAfter.ShouldBe(TimeSpan.FromSeconds(10)); } // ClusterOptionsValidatorTests.cs [Fact] public void SelfFormAfter_null_passes_validation() { /* valid options + null → Succeeded */ } [Fact] public void SelfFormAfter_zero_passes_validation() { /* zero = explicit disable → Succeeded */ } [Fact] public void SelfFormAfter_negative_fails_validation() { /* -1s → Failed, message mentions SelfFormAfter */ } ``` **Step 3: Run to verify failure** ```bash dotnet test tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests --filter "FullyQualifiedName~SelfFormAfter" ``` Expected: FAIL (compile error — property doesn't exist). **Step 4: Implement.** `ClusterOptions.cs` — add after `AllowSingleNodeCluster` (line 104): ```csharp /// /// Bootstrap self-form fallback window (decision 2026-07-22, scadaproj/akka_failover.md §6.1). /// Akka only lets the FIRST listed seed form a new cluster; a non-first seed cold-starting /// while its peer is down loops on InitJoin forever. When this node has waited longer than /// this window without becoming a cluster member, it forms a cluster on itself /// (Cluster.Join(SelfAddress)) — but ONLY if its own address is in its seed list /// (a non-seed node self-forming would create a permanent island). Default 10s: the pair /// shares a datacenter, so a live peer answers InitJoin in milliseconds and waiting longer /// buys nothing. null or a non-positive value disables the fallback (wait-forever). /// Accepted trade: both nodes cold-starting within the window while mutually unreachable /// form two clusters — the same partition class the auto-down strategy already accepts. /// public TimeSpan? SelfFormAfter { get; set; } = TimeSpan.FromSeconds(10); ``` `ClusterOptionsValidator.cs` — inside `Validate`, after the `FailureDetectionThreshold` rules: ```csharp builder.RequireThat(options.SelfFormAfter is null || options.SelfFormAfter.Value >= TimeSpan.Zero, "ClusterOptions.SelfFormAfter must be null (disabled), zero (disabled) or a positive duration; " + "a negative value is always a configuration mistake."); ``` **Step 5: Run tests → PASS** ```bash dotnet test tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests ``` **Step 6: Commit** ```bash git add -A && git commit -m "feat(cluster): SelfFormAfter option — bootstrap self-form fallback window" ``` --- ### Task 2 (A2): `ClusterBootstrapFallback` + first integration test **Classification:** high-risk (cluster formation behavior) **Estimated implement time:** ~5 min **Parallelizable with:** none **Files:** - Create: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/ClusterBootstrapFallback.cs` - Create: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SelfFormBootstrapTests.cs` **Step 1: Write the failing test** ```csharp using Akka.Actor; using Akka.Cluster; using Akka.Configuration; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure; using ZB.MOM.WW.ScadaBridge.Host; using ZB.MOM.WW.ScadaBridge.Host.Actors; namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster; /// /// Guards the InitJoin self-form fallback (decision 2026-07-22): Akka only lets the FIRST /// listed seed form a new cluster, so without the fallback a non-first seed cold-starting /// alone waits on InitJoin forever — the "registered outage gap". These tests build REAL /// single/dual-node clusters from the production BuildHocon output, exactly like /// TwoNodeClusterFixture, and arm the production fallback. /// public sealed class SelfFormBootstrapTests : IAsyncLifetime { private readonly List _systems = new(); /// Starts a node whose seed list puts the PEER first (self second, or absent), /// so Akka's own first-seed rule can never self-form it — only the fallback can. private ActorSystem StartNode(int selfPort, int peerPort, TimeSpan? selfFormAfter, bool selfInSeeds = true) { var nodeOptions = new NodeOptions { Role = "Central", NodeHostname = "127.0.0.1", RemotingPort = selfPort }; var clusterOptions = new ClusterOptions { SeedNodes = selfInSeeds ? new List { $"akka.tcp://scadabridge@127.0.0.1:{peerPort}", $"akka.tcp://scadabridge@127.0.0.1:{selfPort}", } : new List { $"akka.tcp://scadabridge@127.0.0.1:{peerPort}" }, SelfFormAfter = selfFormAfter, StableAfter = TimeSpan.FromSeconds(3), HeartbeatInterval = TimeSpan.FromMilliseconds(500), FailureDetectionThreshold = TimeSpan.FromSeconds(2), MinNrOfMembers = 1, AllowSingleNodeCluster = !selfInSeeds, }; var hocon = AkkaHostedService.BuildHocon( nodeOptions, clusterOptions, new[] { "Central" }, TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(3)); var system = ActorSystem.Create("scadabridge", ConfigurationFactory.ParseString(hocon)); _systems.Add(system); ClusterBootstrapFallback.Arm(system, clusterOptions, NullLogger.Instance); return system; } [Fact] public async Task Lone_non_first_seed_self_forms_after_the_window() { var selfPort = TwoNodeClusterFixture.GetFreeTcpPort(); var deadPeerPort = TwoNodeClusterFixture.GetFreeTcpPort(); // nothing listening var node = StartNode(selfPort, deadPeerPort, selfFormAfter: TimeSpan.FromSeconds(2)); // Without the fallback this waits forever (Akka first-seed rule). With it, the node // must be a 1-member Up cluster shortly after the 2s window. await TwoNodeClusterFixture.WaitForMembersUp(node, 1, TimeSpan.FromSeconds(20)); Cluster.Get(node).SelfMember.Status.ShouldBe(MemberStatus.Up); } public Task InitializeAsync() => Task.CompletedTask; public async Task DisposeAsync() { foreach (var s in _systems) { try { await s.Terminate().WaitAsync(TimeSpan.FromSeconds(10)); } catch { /* teardown */ } } } } ``` (Match the assertion library to `SbrFailoverTests.cs` — Shouldly vs xunit `Assert`.) **Step 2: Run to verify failure** — compile error (`ClusterBootstrapFallback` doesn't exist): ```bash dotnet test tests/ZB.MOM.WW.ScadaBridge.IntegrationTests --filter "FullyQualifiedName~SelfFormBootstrapTests" ``` **Step 3: Implement `ClusterBootstrapFallback.cs`** ```csharp using Akka.Actor; using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure; namespace ZB.MOM.WW.ScadaBridge.Host.Actors; /// /// InitJoin self-form fallback (decision 2026-07-22, scadaproj/akka_failover.md §6.1). /// Akka only lets the FIRST listed seed form a NEW cluster; every other node retries InitJoin /// forever. So "both nodes are seed nodes" (ClusterOptions.SeedNodes) does NOT mean either can /// cold-start alone — a non-first seed booting while its peer is down waits indefinitely (the /// "registered outage gap", docker/README.md). This watchdog waits /// for membership; on expiry it forms a cluster on itself. /// /// Island safety. Fires ONLY when this node's own address is in its own seed list. /// A node that is not a seed (never legitimately first) must keep waiting: if it self-formed, /// a later-booting real seed would form a second cluster and the two can never merge. For nodes /// that ARE seeds, sequential recovery is island-free — Akka's join protocol prefers an existing /// cluster (a booting node only self-forms when NO seed answers InitJoin), so a peer booting /// after this node self-formed simply joins it. /// /// Races are benign. If the join completes between window expiry and /// Cluster.Join(SelfAddress), Akka ignores the join — a node joins a cluster at most once /// per incarnation. The residual risk is both nodes cold-starting inside the window while /// mutually unreachable (a boot-time partition): both self-form, the same dual-active class the /// auto-down downing strategy already accepts, with the same recovery (restart one side). /// public static class ClusterBootstrapFallback { public static void Arm(ActorSystem system, ClusterOptions options, ILogger logger) { if (options.SelfFormAfter is not { } window || window <= TimeSpan.Zero) { logger.LogInformation( "Cluster self-form fallback disabled (SelfFormAfter not set) — a node cold-starting " + "while its peer is down will wait on InitJoin indefinitely."); return; } var cluster = Akka.Cluster.Cluster.Get(system); var self = cluster.SelfAddress; var isSeed = options.SeedNodes.Any(s => TryParseAddress(s, out var a) && a.Equals(self)); if (!isSeed) { logger.LogInformation( "Cluster self-form fallback inactive: this node ({Self}) is not in its own seed list " + "[{Seeds}] — self-forming here would island it from the real seeds.", self, string.Join(", ", options.SeedNodes)); return; } var joined = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); cluster.RegisterOnMemberUp(() => joined.TrySetResult()); _ = Task.Run(async () => { var winner = await Task.WhenAny(joined.Task, Task.Delay(window)); if (winner == joined.Task || system.WhenTerminated.IsCompleted) return; logger.LogWarning( "No cluster membership after {Window} — no seed answered InitJoin (peer down at boot). " + "Self-forming a cluster at {Self} so this node becomes operational; if the peer was " + "merely partitioned (not dead), the pair is now dual-active — restart one side after " + "the partition heals (accepted availability-first trade, decision 2026-07-22).", window, self); cluster.Join(self); }); } private static bool TryParseAddress(string seed, out Address address) { try { address = Address.Parse(seed); return true; } catch { address = default!; return false; } } } ``` (Add the `Microsoft.Extensions.Logging` using. If non-generic `TaskCompletionSource` is unavailable, use `TaskCompletionSource` + `TrySetResult(true)`.) **Step 4: Run → PASS** (~5–10 s). **Step 5: Commit** ```bash git add -A && git commit -m "feat(cluster): InitJoin self-form fallback — lone non-first seed becomes Up" ``` --- ### Task 3 (A3): Remaining fallback tests — disabled / late-peer merge / non-seed **Classification:** standard **Estimated implement time:** ~5 min **Parallelizable with:** Task 4 **Files:** - Modify: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SelfFormBootstrapTests.cs` **Step 1: Append three tests** (each absence assertion carries an in-test positive control — repo convention): ```csharp [Fact] public async Task Disabled_fallback_keeps_waiting_and_the_node_was_otherwise_formable() { var selfPort = TwoNodeClusterFixture.GetFreeTcpPort(); var deadPeerPort = TwoNodeClusterFixture.GetFreeTcpPort(); var node = StartNode(selfPort, deadPeerPort, selfFormAfter: null); await Task.Delay(TimeSpan.FromSeconds(6)); // 3x the window used in the enabled test var cluster = Cluster.Get(node); cluster.State.Members.ShouldBeEmpty(); // still InitJoin-looping — today's behavior // POSITIVE CONTROL: prove the node COULD have formed; only the fallback was missing. cluster.Join(cluster.SelfAddress); await TwoNodeClusterFixture.WaitForMembersUp(node, 1, TimeSpan.FromSeconds(20)); } [Fact] public async Task Peer_booting_after_self_form_joins_the_existing_cluster_no_island() { var portA = TwoNodeClusterFixture.GetFreeTcpPort(); var portB = TwoNodeClusterFixture.GetFreeTcpPort(); // B cold-starts alone (A dead), self-forms after 2s. var nodeB = StartNode(portB, peerPort: portA, selfFormAfter: TimeSpan.FromSeconds(2)); await TwoNodeClusterFixture.WaitForMembersUp(nodeB, 1, TimeSpan.FromSeconds(20)); // A boots later with the pair seed list. B answers InitJoin, so A must JOIN B's // cluster instead of islanding. Generous window on A so its fallback can't race. var nodeA = StartNode(portA, peerPort: portB, selfFormAfter: TimeSpan.FromSeconds(30)); await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 2, TimeSpan.FromSeconds(20)); await TwoNodeClusterFixture.WaitForMembersUp(nodeB, 2, TimeSpan.FromSeconds(20)); } [Fact] public async Task Non_seed_node_never_self_forms() { var selfPort = TwoNodeClusterFixture.GetFreeTcpPort(); var deadSeedPort = TwoNodeClusterFixture.GetFreeTcpPort(); var node = StartNode(selfPort, deadSeedPort, selfFormAfter: TimeSpan.FromSeconds(1), selfInSeeds: false); await Task.Delay(TimeSpan.FromSeconds(5)); // 5x the window var cluster = Cluster.Get(node); cluster.State.Members.ShouldBeEmpty(); // guard refused to island a non-seed // Positive control: the guard (not the environment) prevented formation. cluster.Join(cluster.SelfAddress); await TwoNodeClusterFixture.WaitForMembersUp(node, 1, TimeSpan.FromSeconds(20)); } ``` **Step 2: Run** — all 4 PASS. **Step 3: Commit** `test(cluster): self-form fallback — disabled, late-peer merge, non-seed guard`. --- ### Task 4 (A4): Production wiring + fix the misleading `SeedNodes` doc comment **Classification:** small **Estimated implement time:** ~3 min **Parallelizable with:** Task 3 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs` (inside `GetOrCreateActorSystem`, after the `WhenTerminated` continuation ending ~line 218, before `_actorSystem = system;`) - Modify: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs:33-38` **Step 1: Wire the fallback:** ```csharp // InitJoin self-form fallback (decision 2026-07-22): without it a non-first seed // cold-starting while its peer is down loops on InitJoin forever — auto-down closed // the crash-failover gap, this closes the cold-start-alone gap. Guarded inside Arm: // disabled when SelfFormAfter is unset, inert when this node is not its own seed. ClusterBootstrapFallback.Arm(system, _clusterOptions, _logger); ``` **Step 2: Fix the `SeedNodes` doc comment** (replace the ``): ```csharp /// /// Akka.NET cluster seed nodes. Both nodes are seed nodes — each node lists itself and its /// partner. NOTE: listing both is NOT sufficient for "either can start first": Akka only lets /// the FIRST listed seed form a new cluster, so a lone non-first seed waits on InitJoin until /// expires and the self-form fallback fires (decision 2026-07-22). /// Must contain at least one entry. /// ``` **Step 3: Build + targeted tests** ```bash dotnet build ZB.MOM.WW.ScadaBridge.slnx # 0 warnings (TreatWarningsAsErrors) dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests dotnet test tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests ``` **Step 4: Commit** `feat(cluster): arm self-form fallback at ActorSystem creation; honest SeedNodes doc`. --- ### Task 5 (A5): appsettings sweep **Classification:** trivial **Estimated implement time:** ~3 min **Parallelizable with:** Task 6 Add `"SelfFormAfter": "00:00:10"` next to `"SplitBrainResolverStrategy"` in the `ScadaBridge:Cluster` section of each (explicit for operator visibility; matches the code default): - `src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Central.json`, `appsettings.Site.json` - `docker/central-node-a/appsettings.Central.json`, `docker/central-node-b/appsettings.Central.json` - `docker/site-a-node-a/appsettings.Site.json`, `docker/site-a-node-b/appsettings.Site.json` - `docker/site-b-node-a/appsettings.Site.json`, `docker/site-b-node-b/appsettings.Site.json` - `docker/site-c-node-a/appsettings.Site.json`, `docker/site-c-node-b/appsettings.Site.json` - `docker-env2/` — the 4 node appsettings files Then `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests` and commit `config(cluster): SelfFormAfter=10s explicit in all node appsettings`. > Ops note (do NOT edit here): the gitignored `deploy/wonder-app-vd03/` overlay gets the same key on the next production deploy. --- ### Task 6 (A6): Docs **Classification:** small **Estimated implement time:** ~5 min **Parallelizable with:** Task 5 - `docs/requirements/Component-ClusterInfrastructure.md` — replace the "registered outage gap" recovery text (env-var override) with `SelfFormAfter` semantics + island guard + boot-partition trade. - `docker/README.md` (~line 289) — same replacement; keep the partition-trade note. - `docs/deployment/topology-guide.md:101` — **fix the stale keep-oldest claim** (still says "Keep-oldest with `down-if-alone = on`"): rewrite to auto-down default + `SelfFormAfter`, pointing at the decision records. - `CLAUDE.md` (~line 222) — update the boot-order note: pairs no longer require the first seed for cold start; note the 10 s window. Commit: `docs(cluster): SelfFormAfter fallback; fix stale keep-oldest note in topology guide`. --- ### Task 7 (A7): Full verification + optional docker live gate **Classification:** standard **Estimated implement time:** ~5 min (suite runtime dominates) **Parallelizable with:** none **Step 1:** ```bash cd ~/Desktop/ScadaBridge dotnet build ZB.MOM.WW.ScadaBridge.slnx # expect 0 warnings dotnet test ZB.MOM.WW.ScadaBridge.slnx # expect green vs pre-existing baseline ``` **Step 2 (LIVE GATE — run if the docker rig is available; else record deferred-live):** ```bash cd docker && bash deploy.sh # rebuild with the fallback docker compose stop central-a central-b # verify service names in docker-compose.yml first docker compose start central-b # cold-start ONLY the non-first seed docker compose logs -f central-b | grep -m1 "Self-forming a cluster" # ≈10s after start curl -fsS http://localhost:9002/health/active # expect 200 docker compose start central-a # first seed returns → must JOIN, not island docker compose logs central-a | grep -i "Welcome" ``` Expected: `central-b` self-forms ≈10 s and serves (previously a permanent wedge); `central-a` rejoins as youngest/standby. --- ### Task 8 (D1): `IManualFailoverService` + cluster-level test **Classification:** high-risk (cluster behavior) **Estimated implement time:** ~5 min **Parallelizable with:** none **Files:** - Create: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/IManualFailoverService.cs` - Create: `src/ZB.MOM.WW.ScadaBridge.Host/Health/AkkaManualFailoverService.cs` - Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` (register in the Central branch, next to `IActiveNodeGate` ~line 330) - Test: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/ManualFailoverTests.cs` **Step 1: Failing tests** on `TwoNodeClusterFixture`: ```csharp [Fact] public async Task Failover_makes_the_oldest_leave_and_the_survivor_take_over() { await using var f = await TwoNodeClusterFixture.StartAsync(); var oldest = Akka.Cluster.Cluster.Get(f.NodeA); // NodeA started first = oldest var target = AkkaManualFailoverService.FailOverCore(f.NodeB, "Central"); // issued from the OTHER node target.ShouldBe(oldest.SelfAddress); // Graceful exit path: the left node's own ActorSystem terminates… await f.NodeA.WhenTerminated.WaitAsync(TimeSpan.FromSeconds(30)); // …and the survivor becomes a 1-member cluster and the oldest-Up active node. await TwoNodeClusterFixture.WaitForMemberRemoved(f.NodeB, oldest.SelfAddress, TimeSpan.FromSeconds(30)); ActiveNodeEvaluator.SelfIsOldestUp(Akka.Cluster.Cluster.Get(f.NodeB)).ShouldBeTrue(); } [Fact] public void Failover_refuses_when_no_peer_exists() { /* 1-node cluster → FailOverCore returns null, node still Up afterwards (positive assert) */ } ``` **Step 2:** Run → FAIL (service missing). **Step 3: Implement.** Interface in CentralUI (CentralUI stays Akka-free): ```csharp public interface IManualFailoverService { /// Gracefully fails over the central cluster: the current active (oldest Up) /// member leaves, restarts via its supervisor, and rejoins as standby. Returns the /// address string acted on, or null when there is no peer to fail over to. Task FailOverCentralAsync(string actor); } ``` Host implementation — static testable core + thin DI wrapper: ```csharp public sealed class AkkaManualFailoverService : IManualFailoverService { // ctor: (AkkaHostedService akka, ICentralAuditWriter audit, ILogger logger) public async Task FailOverCentralAsync(string actor) { var system = _akka.GetOrCreateActorSystem(); var target = FailOverCore(system, role: "Central", dryRun: true); if (target is null) return null; // peer guard await _audit.WriteAsync(/* canonical AuditEvent: Action=cluster.manual-failover, Actor=actor, DetailsJson={"target": target}, Outcome=Success — copy the exact call shape from an existing audited admin action (e.g. the Sites admin service); use the CENTRAL audit writer, not the shared seam */); FailOverCore(system, role: "Central"); // Cluster.Leave(target) _logger.LogWarning("Manual failover triggered by {Actor}: {Target} is leaving the cluster.", actor, target); return target.ToString(); } /// Oldest Up member with the role leaves — mirrors ActiveNodeEvaluator's oldest-Up /// rule so the node acted on is exactly the one hosting the singletons. Returns null when /// fewer than 2 Up members carry the role (no peer = failover would be an outage). public static Address? FailOverCore(ActorSystem system, string role, bool dryRun = false) { var cluster = Akka.Cluster.Cluster.Get(system); var withRole = cluster.State.Members .Where(m => m.Status == MemberStatus.Up && m.HasRole(role)) .OrderBy(m => m, Member.AgeOrdering) .ToList(); if (withRole.Count < 2) return null; var oldest = withRole[0]; if (!dryRun) cluster.Leave(oldest.Address); return oldest.Address; } } ``` **Step 4:** Register in the Central branch of `Program.cs`. **Step 5:** Tests PASS → commit `feat(ui): manual central failover service — graceful Leave of the oldest Up member`. --- ### Task 9 (D2): Health page button + bUnit tests + runbook **Classification:** standard **Estimated implement time:** ~5 min **Parallelizable with:** none **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor` (central-cluster card, near the Nodes column ~line 233) - Test: `tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/HealthFailoverButtonTests.cs` (follow the project's existing bUnit page-test pattern) - Modify: `docs/requirements/Component-ClusterInfrastructure.md` + `docker/README.md` — manual-failover runbook paragraph **Step 1: Failing bUnit tests:** (a) button absent without the admin policy; (b) present + enabled for admin with ≥2 online central nodes; (c) disabled with tooltip at 1 node; (d) confirm flow calls `IManualFailoverService.FailOverCentralAsync` exactly once (fake service). **Step 2: Implement:** ```razor ``` plus a confirmation modal (copy the page's existing dialog idiom). The warning text MUST state: the active node restarts, roles swap, and this page will briefly disconnect and reconnect against the new active node (Traefik routes to the active). On confirm: call the service with the authenticated user name; surface the returned target address. **Step 3:** Tests PASS → commit `feat(ui): admin manual-failover control on the health page`. **Live check (fold into the Task 7 gate when the rig is up):** press the button, watch `central-a` restart and `central-b`'s badge flip to Primary; audit row lands in `dbo.AuditLog`. --- ### Task 10 (D3, OPTIONAL — confirm with the user before executing): site-pair failover from the central UI **Classification:** high-risk (new cross-cluster message contract) **Estimated implement time:** exploration first; likely 2–3 tasks if approved The Health page also shows per-site node cards (Primary/Standby from heartbeats), but central and sites are **separate Akka clusters** — a site failover needs a `TriggerSiteFailover(siteId)` command over the existing central→site transport (same channel as the Retry/Discard relay), handled on the site's active node by `Cluster.Leave(SelfAddress)`. That adds a versioned message contract (rolling-upgrade surface). If approved: explore `ZB.MOM.WW.ScadaBridge.Communication` for the command path, mirror an existing command end-to-end, per-site button with the same guard/confirm/audit rules. Otherwise: file a follow-up issue and skip. --- ## Completion - Merge decision via the finishing-a-development-branch flow (family convention: ff-merge to `main` + push to gitea, or PR — ask the user). - After merge: update `scadaproj/akka_failover.md` §6.1 status, `scadaproj/CLAUDE.md` index row, and memory `ha-availability-over-partition-safety` (tracked as the family-docs task in the scadaproj index plan). - Verification-before-completion applies throughout: no task is done without its command output; live gates may be recorded deferred-live if the rig is down.