ScadaBridge/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md

ScadaBridge: InitJoin Self-Form Fallback + Manual Failover Control — Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.

Shared cross-repo design: ~/Desktop/scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md (design rationale, MNTR assessment, behavior spec). The OtOpcUa half lives in ~/Desktop/OtOpcUa/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md. This plan is self-contained for execution.

Goal: (1) Either node of a 2-node ScadaBridge cluster can cold-start alone and become operational, unattended. (2) An admin-only "Trigger failover" control on the Health page performs a graceful, audited role swap of the central pair.

⚠️ ARCHITECTURE REVISED DURING EXECUTION (2026-07-22)

Part 1 shipped as self-first seed ordering, NOT the SelfFormAfter watchdog described below. Tasks 1–7 as originally written are superseded; what was actually built is in "Part 1 as executed". Tasks 8–10 (manual failover) are unaffected and still apply as written.

Why. A code review of Task 2 raised, and a written test then confirmed, that the watchdog's success signal ("am I Up within the window?") cannot distinguish no seed answered InitJoin from a seed answered and the join is in flight — it sits outside Akka's join handshake. On a routine standby restart the peer is alive but the join stalls behind removal of the restarting node's own stale incarnation; a Join(self) issued during TryingToJoin abandons the in-flight join and forms a second cluster at the same address. Measured: a permanent split, still unhealed after 90 s — converting a routine restart into an outage of the previously-healthy node. That is strictly worse than the gap being closed, and it is not the boot-partition trade the design accepted.

Akka's own FirstSeedNodeProcess already implements exactly the intended semantics — InitJoin the other seeds, self-join only if nobody answers — and, being part of the handshake, has no such race. It runs only when seed-nodes[0] is the node's own address. So the fix is seed ordering, not new runtime code.

The plan's stated safety property — "a booting node only self-forms when NO seed answers InitJoin" — is true of Akka's native first-seed rule and false of the watchdog. That claim also appears in the shared cross-repo design doc (scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md) and in the OtOpcUa half; both still need correcting (owner deferred, 2026-07-22).

Part 1 as executed — self-first seed ordering

Architecture: Every node lists ITSELF as seed-nodes[0] and its partner second. No new runtime code, no timer, no new option. StartupValidator enforces the ordering at boot (host and port comparison) because a broken ordering fails silently. Manual failover (Part 2) is unchanged: graceful Cluster.Leave(oldest Up member) via a new IManualFailoverService (CentralUI seam, Host implementation) — singleton drain, watchdog process-exit, supervisor restart, rejoin as youngest.

Behavior (all rows covered by SelfFirstSeedBootstrapTests — real in-process clusters from production BuildHocon at production failure-detection timings):

Scenario	Behavior
Lone cold-start, peer dead	Self-joins after seed-node-timeout (~5 s) — operational, unattended
Restart into a live peer	Peer answers InitJoinAck; node rejoins, never islands
Both cold-start simultaneously, mutually reachable	InitJoin handshake converges them → one 2-member cluster
Both cold-start during a genuine boot partition	Each forms its own cluster — same dual-active class auto-down already accepts
Peer-first ordering (the old config)	Never forms — retained as a falsifiability control in the test suite

Shipped:

• 6 node appsettings reordered (the *-node-b configs; -a nodes were already self-first). All 14 satisfy the invariant.
• StartupValidator self-first rule + 3 tests; SelfFirstSeedBootstrapTests (4 tests).
• Docs corrected: docker/README.md, docs/requirements/Component-ClusterInfrastructure.md (new Seed Node Ordering section), docs/components/ClusterInfrastructure.md (3 passages), docs/deployment/topology-guide.md (incl. the stale keep-oldest claim Task 6 flagged), CLAUDE.md.
• Several docs had asserted self-first ordering was unsafe because a simultaneous cold start would produce two clusters that never merge. Disproved by test (row 3) and corrected.

⚠️ Ops action: the gitignored deploy/wonder-app-vd03/ overlay must have its SeedNodes reordered self-first before its next deploy, or the node will now refuse to boot. The validator rule is a hard gate deliberately — the alternative is the silent wedge it replaces.

---

Original Part 1 architecture (SUPERSEDED — kept for the decision record)

Architecture: New ClusterOptions.SelfFormAfter (TimeSpan?, default 10 s, null/≤0 disables, appsettings-bound) arms ClusterBootstrapFallback right after ActorSystem creation: wait for membership via RegisterOnMemberUp; on expiry, Cluster.Join(SelfAddress). Safety gate: fires only when this node's own address is in its own seed list.

Tech Stack: .NET 10, Akka.NET 1.5.62, Blazor Server (CentralUI), bUnit, xunit. No new packages.

Branch: feat/selfform-fallback off main.

---

Design essentials (from the shared design doc)

The defect: Akka only lets the FIRST listed seed self-join to form a new cluster; every other node loops on InitJoin forever. Both docker central nodes list central-a first, so a lone cold-starting central-b never comes Up (the "registered outage gap" — docker/README.md:289). ClusterOptions.SeedNodes' doc comment claims "either can start first", which the deployed configs do not deliver — this plan makes it true and fixes the comment.

Behavior spec:

Scenario	Behavior with fallback
Peer alive (any boot order)	Normal seed join in ms — fallback never fires
Lone cold-start, self IS in own seed list	After SelfFormAfter: warn log + Cluster.Join(SelfAddress) → Up alone, singletons start (min-nr-of-members=1)
Lone cold-start, self NOT in own seed list	Fallback inert (info log) — self-forming would island the node from the real seeds
Peer boots after survivor self-formed	Peer's InitJoin is answered → joins as youngest/standby. No island.
Both cold-start simultaneously, mutually unreachable	Both self-form → dual-active (same partition class auto-down accepts; restart one side)
SelfFormAfter null/≤0	Disabled — today's wait-forever behavior
Window expires mid-join-handshake	Benign: Akka ignores Join once joined

Manual failover rules: graceful Leave, never Down; admin-only (AuthorizationPolicies.RequireAdmin); peer guard (disabled when <2 Up Central members); confirmation dialog warning the Blazor circuit will drop (Traefik routes the UI to the active node — triggering failover disconnects your own page, which reconnects against the new active); audited via the app's central audit writer (ICentralAuditWriter — NOT the shared seam, see the dual-seam gotcha) before the Leave is issued. No interplay with SelfFormAfter (the peer is alive on this path, so the restarted node rejoins normally).

Multi-node TestKit: assessed and NOT used — in-process real clusters via TwoNodeClusterFixture (production BuildHocon) cover every deterministic scenario; MNTR would need a dedicated no-parallelization test project for no added coverage. See the shared design doc for the full verdict.

---

Task 1 (A1): SelfFormAfter option + validator

Classification: small Estimated implement time: ~4 min Parallelizable with: none (first task)

Files:

• Modify: src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs
• Modify: src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptionsValidator.cs
• Test: tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsTests.cs
• Test: tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsValidatorTests.cs

Step 1: Create the branch

cd ~/Desktop/ScadaBridge && git checkout main && git checkout -b feat/selfform-fallback

Step 2: Write the failing tests (append to the existing test classes, matching their assertion style — read them first)

// ClusterOptionsTests.cs
[Fact]
public void SelfFormAfter_defaults_to_ten_seconds()
{
    new ClusterOptions().SelfFormAfter.ShouldBe(TimeSpan.FromSeconds(10));
}

// ClusterOptionsValidatorTests.cs
[Fact] public void SelfFormAfter_null_passes_validation() { /* valid options + null → Succeeded */ }
[Fact] public void SelfFormAfter_zero_passes_validation() { /* zero = explicit disable → Succeeded */ }
[Fact] public void SelfFormAfter_negative_fails_validation() { /* -1s → Failed, message mentions SelfFormAfter */ }

Step 3: Run to verify failure

dotnet test tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests --filter "FullyQualifiedName~SelfFormAfter"

Expected: FAIL (compile error — property doesn't exist).

Step 4: Implement. ClusterOptions.cs — add after AllowSingleNodeCluster (line 104):

    /// <summary>
    /// Bootstrap self-form fallback window (decision 2026-07-22, scadaproj/akka_failover.md §6.1).
    /// Akka only lets the FIRST listed seed form a new cluster; a non-first seed cold-starting
    /// while its peer is down loops on InitJoin forever. When this node has waited longer than
    /// this window without becoming a cluster member, it forms a cluster on itself
    /// (<c>Cluster.Join(SelfAddress)</c>) — but ONLY if its own address is in its seed list
    /// (a non-seed node self-forming would create a permanent island). Default 10s: the pair
    /// shares a datacenter, so a live peer answers InitJoin in milliseconds and waiting longer
    /// buys nothing. <c>null</c> or a non-positive value disables the fallback (wait-forever).
    /// Accepted trade: both nodes cold-starting within the window while mutually unreachable
    /// form two clusters — the same partition class the auto-down strategy already accepts.
    /// </summary>
    public TimeSpan? SelfFormAfter { get; set; } = TimeSpan.FromSeconds(10);

ClusterOptionsValidator.cs — inside Validate, after the FailureDetectionThreshold rules:

        builder.RequireThat(options.SelfFormAfter is null || options.SelfFormAfter.Value >= TimeSpan.Zero,
            "ClusterOptions.SelfFormAfter must be null (disabled), zero (disabled) or a positive duration; "
            + "a negative value is always a configuration mistake.");

Step 5: Run tests → PASS

dotnet test tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests

Step 6: Commit

git add -A && git commit -m "feat(cluster): SelfFormAfter option — bootstrap self-form fallback window"

---

Task 2 (A2): ClusterBootstrapFallback + first integration test

Classification: high-risk (cluster formation behavior) Estimated implement time: ~5 min Parallelizable with: none

Files:

• Create: src/ZB.MOM.WW.ScadaBridge.Host/Actors/ClusterBootstrapFallback.cs
• Create: tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SelfFormBootstrapTests.cs

Step 1: Write the failing test

using Akka.Actor;
using Akka.Cluster;
using Akka.Configuration;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
using ZB.MOM.WW.ScadaBridge.Host;
using ZB.MOM.WW.ScadaBridge.Host.Actors;

namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;

/// <summary>
/// Guards the InitJoin self-form fallback (decision 2026-07-22): Akka only lets the FIRST
/// listed seed form a new cluster, so without the fallback a non-first seed cold-starting
/// alone waits on InitJoin forever — the "registered outage gap". These tests build REAL
/// single/dual-node clusters from the production BuildHocon output, exactly like
/// TwoNodeClusterFixture, and arm the production fallback.
/// </summary>
public sealed class SelfFormBootstrapTests : IAsyncLifetime
{
    private readonly List<ActorSystem> _systems = new();

    /// <summary>Starts a node whose seed list puts the PEER first (self second, or absent),
    /// so Akka's own first-seed rule can never self-form it — only the fallback can.</summary>
    private ActorSystem StartNode(int selfPort, int peerPort, TimeSpan? selfFormAfter, bool selfInSeeds = true)
    {
        var nodeOptions = new NodeOptions { Role = "Central", NodeHostname = "127.0.0.1", RemotingPort = selfPort };
        var clusterOptions = new ClusterOptions
        {
            SeedNodes = selfInSeeds
                ? new List<string>
                {
                    $"akka.tcp://scadabridge@127.0.0.1:{peerPort}",
                    $"akka.tcp://scadabridge@127.0.0.1:{selfPort}",
                }
                : new List<string> { $"akka.tcp://scadabridge@127.0.0.1:{peerPort}" },
            SelfFormAfter = selfFormAfter,
            StableAfter = TimeSpan.FromSeconds(3),
            HeartbeatInterval = TimeSpan.FromMilliseconds(500),
            FailureDetectionThreshold = TimeSpan.FromSeconds(2),
            MinNrOfMembers = 1,
            AllowSingleNodeCluster = !selfInSeeds,
        };
        var hocon = AkkaHostedService.BuildHocon(
            nodeOptions, clusterOptions, new[] { "Central" },
            TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(3));
        var system = ActorSystem.Create("scadabridge", ConfigurationFactory.ParseString(hocon));
        _systems.Add(system);
        ClusterBootstrapFallback.Arm(system, clusterOptions, NullLogger.Instance);
        return system;
    }

    [Fact]
    public async Task Lone_non_first_seed_self_forms_after_the_window()
    {
        var selfPort = TwoNodeClusterFixture.GetFreeTcpPort();
        var deadPeerPort = TwoNodeClusterFixture.GetFreeTcpPort(); // nothing listening
        var node = StartNode(selfPort, deadPeerPort, selfFormAfter: TimeSpan.FromSeconds(2));

        // Without the fallback this waits forever (Akka first-seed rule). With it, the node
        // must be a 1-member Up cluster shortly after the 2s window.
        await TwoNodeClusterFixture.WaitForMembersUp(node, 1, TimeSpan.FromSeconds(20));
        Cluster.Get(node).SelfMember.Status.ShouldBe(MemberStatus.Up);
    }

    public Task InitializeAsync() => Task.CompletedTask;

    public async Task DisposeAsync()
    {
        foreach (var s in _systems)
        {
            try { await s.Terminate().WaitAsync(TimeSpan.FromSeconds(10)); } catch { /* teardown */ }
        }
    }
}

(Match the assertion library to SbrFailoverTests.cs — Shouldly vs xunit Assert.)

Step 2: Run to verify failure — compile error (ClusterBootstrapFallback doesn't exist):

dotnet test tests/ZB.MOM.WW.ScadaBridge.IntegrationTests --filter "FullyQualifiedName~SelfFormBootstrapTests"

Step 3: Implement ClusterBootstrapFallback.cs

using Akka.Actor;
using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;

namespace ZB.MOM.WW.ScadaBridge.Host.Actors;

/// <summary>
/// InitJoin self-form fallback (decision 2026-07-22, scadaproj/akka_failover.md §6.1).
/// Akka only lets the FIRST listed seed form a NEW cluster; every other node retries InitJoin
/// forever. So "both nodes are seed nodes" (ClusterOptions.SeedNodes) does NOT mean either can
/// cold-start alone — a non-first seed booting while its peer is down waits indefinitely (the
/// "registered outage gap", docker/README.md). This watchdog waits <see cref="ClusterOptions.SelfFormAfter"/>
/// for membership; on expiry it forms a cluster on itself.
///
/// <para><b>Island safety.</b> Fires ONLY when this node's own address is in its own seed list.
/// A node that is not a seed (never legitimately first) must keep waiting: if it self-formed,
/// a later-booting real seed would form a second cluster and the two can never merge. For nodes
/// that ARE seeds, sequential recovery is island-free — Akka's join protocol prefers an existing
/// cluster (a booting node only self-forms when NO seed answers InitJoin), so a peer booting
/// after this node self-formed simply joins it.</para>
///
/// <para><b>Races are benign.</b> If the join completes between window expiry and
/// <c>Cluster.Join(SelfAddress)</c>, Akka ignores the join — a node joins a cluster at most once
/// per incarnation. The residual risk is both nodes cold-starting inside the window while
/// mutually unreachable (a boot-time partition): both self-form, the same dual-active class the
/// auto-down downing strategy already accepts, with the same recovery (restart one side).</para>
/// </summary>
public static class ClusterBootstrapFallback
{
    public static void Arm(ActorSystem system, ClusterOptions options, ILogger logger)
    {
        if (options.SelfFormAfter is not { } window || window <= TimeSpan.Zero)
        {
            logger.LogInformation(
                "Cluster self-form fallback disabled (SelfFormAfter not set) — a node cold-starting "
                + "while its peer is down will wait on InitJoin indefinitely.");
            return;
        }

        var cluster = Akka.Cluster.Cluster.Get(system);
        var self = cluster.SelfAddress;
        var isSeed = options.SeedNodes.Any(s => TryParseAddress(s, out var a) && a.Equals(self));
        if (!isSeed)
        {
            logger.LogInformation(
                "Cluster self-form fallback inactive: this node ({Self}) is not in its own seed list "
                + "[{Seeds}] — self-forming here would island it from the real seeds.",
                self, string.Join(", ", options.SeedNodes));
            return;
        }

        var joined = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
        cluster.RegisterOnMemberUp(() => joined.TrySetResult());

        _ = Task.Run(async () =>
        {
            var winner = await Task.WhenAny(joined.Task, Task.Delay(window));
            if (winner == joined.Task || system.WhenTerminated.IsCompleted)
                return;
            logger.LogWarning(
                "No cluster membership after {Window} — no seed answered InitJoin (peer down at boot). "
                + "Self-forming a cluster at {Self} so this node becomes operational; if the peer was "
                + "merely partitioned (not dead), the pair is now dual-active — restart one side after "
                + "the partition heals (accepted availability-first trade, decision 2026-07-22).",
                window, self);
            cluster.Join(self);
        });
    }

    private static bool TryParseAddress(string seed, out Address address)
    {
        try { address = Address.Parse(seed); return true; }
        catch { address = default!; return false; }
    }
}

(Add the Microsoft.Extensions.Logging using. If non-generic TaskCompletionSource is unavailable, use TaskCompletionSource<bool> + TrySetResult(true).)

Step 4: Run → PASS (~5–10 s). Step 5: Commit

git add -A && git commit -m "feat(cluster): InitJoin self-form fallback — lone non-first seed becomes Up"

---

Task 3 (A3): Remaining fallback tests — disabled / late-peer merge / non-seed

Classification: standard Estimated implement time: ~5 min Parallelizable with: Task 4

Files:

• Modify: tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SelfFormBootstrapTests.cs

Step 1: Append three tests (each absence assertion carries an in-test positive control — repo convention):

    [Fact]
    public async Task Disabled_fallback_keeps_waiting_and_the_node_was_otherwise_formable()
    {
        var selfPort = TwoNodeClusterFixture.GetFreeTcpPort();
        var deadPeerPort = TwoNodeClusterFixture.GetFreeTcpPort();
        var node = StartNode(selfPort, deadPeerPort, selfFormAfter: null);

        await Task.Delay(TimeSpan.FromSeconds(6)); // 3x the window used in the enabled test
        var cluster = Cluster.Get(node);
        cluster.State.Members.ShouldBeEmpty(); // still InitJoin-looping — today's behavior

        // POSITIVE CONTROL: prove the node COULD have formed; only the fallback was missing.
        cluster.Join(cluster.SelfAddress);
        await TwoNodeClusterFixture.WaitForMembersUp(node, 1, TimeSpan.FromSeconds(20));
    }

    [Fact]
    public async Task Peer_booting_after_self_form_joins_the_existing_cluster_no_island()
    {
        var portA = TwoNodeClusterFixture.GetFreeTcpPort();
        var portB = TwoNodeClusterFixture.GetFreeTcpPort();

        // B cold-starts alone (A dead), self-forms after 2s.
        var nodeB = StartNode(portB, peerPort: portA, selfFormAfter: TimeSpan.FromSeconds(2));
        await TwoNodeClusterFixture.WaitForMembersUp(nodeB, 1, TimeSpan.FromSeconds(20));

        // A boots later with the pair seed list. B answers InitJoin, so A must JOIN B's
        // cluster instead of islanding. Generous window on A so its fallback can't race.
        var nodeA = StartNode(portA, peerPort: portB, selfFormAfter: TimeSpan.FromSeconds(30));
        await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 2, TimeSpan.FromSeconds(20));
        await TwoNodeClusterFixture.WaitForMembersUp(nodeB, 2, TimeSpan.FromSeconds(20));
    }

    [Fact]
    public async Task Non_seed_node_never_self_forms()
    {
        var selfPort = TwoNodeClusterFixture.GetFreeTcpPort();
        var deadSeedPort = TwoNodeClusterFixture.GetFreeTcpPort();
        var node = StartNode(selfPort, deadSeedPort, selfFormAfter: TimeSpan.FromSeconds(1), selfInSeeds: false);

        await Task.Delay(TimeSpan.FromSeconds(5)); // 5x the window
        var cluster = Cluster.Get(node);
        cluster.State.Members.ShouldBeEmpty(); // guard refused to island a non-seed

        // Positive control: the guard (not the environment) prevented formation.
        cluster.Join(cluster.SelfAddress);
        await TwoNodeClusterFixture.WaitForMembersUp(node, 1, TimeSpan.FromSeconds(20));
    }

Step 2: Run — all 4 PASS. Step 3: Commit test(cluster): self-form fallback — disabled, late-peer merge, non-seed guard.

---

Task 4 (A4): Production wiring + fix the misleading SeedNodes doc comment

Classification: small Estimated implement time: ~3 min Parallelizable with: Task 3

Files:

• Modify: src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs (inside GetOrCreateActorSystem, after the WhenTerminated continuation ending ~line 218, before _actorSystem = system;)
• Modify: src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs:33-38

Step 1: Wire the fallback:

            // InitJoin self-form fallback (decision 2026-07-22): without it a non-first seed
            // cold-starting while its peer is down loops on InitJoin forever — auto-down closed
            // the crash-failover gap, this closes the cold-start-alone gap. Guarded inside Arm:
            // disabled when SelfFormAfter is unset, inert when this node is not its own seed.
            ClusterBootstrapFallback.Arm(system, _clusterOptions, _logger);

Step 2: Fix the SeedNodes doc comment (replace the <summary>):

    /// <summary>
    /// Akka.NET cluster seed nodes. Both nodes are seed nodes — each node lists itself and its
    /// partner. NOTE: listing both is NOT sufficient for "either can start first": Akka only lets
    /// the FIRST listed seed form a new cluster, so a lone non-first seed waits on InitJoin until
    /// <see cref="SelfFormAfter"/> expires and the self-form fallback fires (decision 2026-07-22).
    /// Must contain at least one entry.
    /// </summary>

Step 3: Build + targeted tests

dotnet build ZB.MOM.WW.ScadaBridge.slnx      # 0 warnings (TreatWarningsAsErrors)
dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests
dotnet test tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests

Step 4: Commit feat(cluster): arm self-form fallback at ActorSystem creation; honest SeedNodes doc.

---

Task 5 (A5): appsettings sweep

Classification: trivial Estimated implement time: ~3 min Parallelizable with: Task 6

Add "SelfFormAfter": "00:00:10" next to "SplitBrainResolverStrategy" in the ScadaBridge:Cluster section of each (explicit for operator visibility; matches the code default):

• src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Central.json, appsettings.Site.json
• docker/central-node-a/appsettings.Central.json, docker/central-node-b/appsettings.Central.json
• docker/site-a-node-a/appsettings.Site.json, docker/site-a-node-b/appsettings.Site.json
• docker/site-b-node-a/appsettings.Site.json, docker/site-b-node-b/appsettings.Site.json
• docker/site-c-node-a/appsettings.Site.json, docker/site-c-node-b/appsettings.Site.json
• docker-env2/ — the 4 node appsettings files

Then dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests and commit config(cluster): SelfFormAfter=10s explicit in all node appsettings.

Ops note (do NOT edit here): the gitignored deploy/wonder-app-vd03/ overlay gets the same key on the next production deploy.

---

Task 6 (A6): Docs

Classification: small Estimated implement time: ~5 min Parallelizable with: Task 5

• docs/requirements/Component-ClusterInfrastructure.md — replace the "registered outage gap" recovery text (env-var override) with SelfFormAfter semantics + island guard + boot-partition trade.
• docker/README.md (~line 289) — same replacement; keep the partition-trade note.
• docs/deployment/topology-guide.md:101 — fix the stale keep-oldest claim (still says "Keep-oldest with down-if-alone = on"): rewrite to auto-down default + SelfFormAfter, pointing at the decision records.
• CLAUDE.md (~line 222) — update the boot-order note: pairs no longer require the first seed for cold start; note the 10 s window.

Commit: docs(cluster): SelfFormAfter fallback; fix stale keep-oldest note in topology guide.

---

Task 7 (A7): Full verification + optional docker live gate

Classification: standard Estimated implement time: ~5 min (suite runtime dominates) Parallelizable with: none

Step 1:

cd ~/Desktop/ScadaBridge
dotnet build ZB.MOM.WW.ScadaBridge.slnx   # expect 0 warnings
dotnet test ZB.MOM.WW.ScadaBridge.slnx    # expect green vs pre-existing baseline

Step 2 (LIVE GATE — run if the docker rig is available; else record deferred-live):

cd docker && bash deploy.sh               # rebuild with the fallback
docker compose stop central-a central-b   # verify service names in docker-compose.yml first
docker compose start central-b            # cold-start ONLY the non-first seed
docker compose logs -f central-b | grep -m1 "Self-forming a cluster"   # ≈10s after start
curl -fsS http://localhost:9002/health/active                          # expect 200
docker compose start central-a            # first seed returns → must JOIN, not island
docker compose logs central-a | grep -i "Welcome"

Expected: central-b self-forms ≈10 s and serves (previously a permanent wedge); central-a rejoins as youngest/standby.

---

Task 8 (D1): IManualFailoverService + cluster-level test

Classification: high-risk (cluster behavior) Estimated implement time: ~5 min Parallelizable with: none

Files:

• Create: src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/IManualFailoverService.cs
• Create: src/ZB.MOM.WW.ScadaBridge.Host/Health/AkkaManualFailoverService.cs
• Modify: src/ZB.MOM.WW.ScadaBridge.Host/Program.cs (register in the Central branch, next to IActiveNodeGate ~line 330)
• Test: tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/ManualFailoverTests.cs

Step 1: Failing tests on TwoNodeClusterFixture:

[Fact]
public async Task Failover_makes_the_oldest_leave_and_the_survivor_take_over()
{
    await using var f = await TwoNodeClusterFixture.StartAsync();
    var oldest = Akka.Cluster.Cluster.Get(f.NodeA);   // NodeA started first = oldest
    var target = AkkaManualFailoverService.FailOverCore(f.NodeB, "Central"); // issued from the OTHER node
    target.ShouldBe(oldest.SelfAddress);

    // Graceful exit path: the left node's own ActorSystem terminates…
    await f.NodeA.WhenTerminated.WaitAsync(TimeSpan.FromSeconds(30));
    // …and the survivor becomes a 1-member cluster and the oldest-Up active node.
    await TwoNodeClusterFixture.WaitForMemberRemoved(f.NodeB, oldest.SelfAddress, TimeSpan.FromSeconds(30));
    ActiveNodeEvaluator.SelfIsOldestUp(Akka.Cluster.Cluster.Get(f.NodeB)).ShouldBeTrue();
}

[Fact]
public void Failover_refuses_when_no_peer_exists()
{ /* 1-node cluster → FailOverCore returns null, node still Up afterwards (positive assert) */ }

Step 2: Run → FAIL (service missing).

Step 3: Implement. Interface in CentralUI (CentralUI stays Akka-free):

public interface IManualFailoverService
{
    /// <summary>Gracefully fails over the central cluster: the current active (oldest Up)
    /// member leaves, restarts via its supervisor, and rejoins as standby. Returns the
    /// address string acted on, or null when there is no peer to fail over to.</summary>
    Task<string?> FailOverCentralAsync(string actor);
}

Host implementation — static testable core + thin DI wrapper:

public sealed class AkkaManualFailoverService : IManualFailoverService
{
    // ctor: (AkkaHostedService akka, ICentralAuditWriter audit, ILogger<AkkaManualFailoverService> logger)

    public async Task<string?> FailOverCentralAsync(string actor)
    {
        var system = _akka.GetOrCreateActorSystem();
        var target = FailOverCore(system, role: "Central", dryRun: true);
        if (target is null) return null;                       // peer guard
        await _audit.WriteAsync(/* canonical AuditEvent: Action=cluster.manual-failover,
            Actor=actor, DetailsJson={"target": target}, Outcome=Success — copy the exact
            call shape from an existing audited admin action (e.g. the Sites admin service);
            use the CENTRAL audit writer, not the shared seam */);
        FailOverCore(system, role: "Central");                 // Cluster.Leave(target)
        _logger.LogWarning("Manual failover triggered by {Actor}: {Target} is leaving the cluster.", actor, target);
        return target.ToString();
    }

    /// <summary>Oldest Up member with the role leaves — mirrors ActiveNodeEvaluator's oldest-Up
    /// rule so the node acted on is exactly the one hosting the singletons. Returns null when
    /// fewer than 2 Up members carry the role (no peer = failover would be an outage).</summary>
    public static Address? FailOverCore(ActorSystem system, string role, bool dryRun = false)
    {
        var cluster = Akka.Cluster.Cluster.Get(system);
        var withRole = cluster.State.Members
            .Where(m => m.Status == MemberStatus.Up && m.HasRole(role))
            .OrderBy(m => m, Member.AgeOrdering)
            .ToList();
        if (withRole.Count < 2) return null;
        var oldest = withRole[0];
        if (!dryRun) cluster.Leave(oldest.Address);
        return oldest.Address;
    }
}

Step 4: Register in the Central branch of Program.cs. Step 5: Tests PASS → commit feat(ui): manual central failover service — graceful Leave of the oldest Up member.

---

Task 9 (D2): Health page button + bUnit tests + runbook

Classification: standard Estimated implement time: ~5 min Parallelizable with: none

Files:

• Modify: src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor (central-cluster card, near the Nodes column ~line 233)
• Test: tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/HealthFailoverButtonTests.cs (follow the project's existing bUnit page-test pattern)
• Modify: docs/requirements/Component-ClusterInfrastructure.md + docker/README.md — manual-failover runbook paragraph

Step 1: Failing bUnit tests: (a) button absent without the admin policy; (b) present + enabled for admin with ≥2 online central nodes; (c) disabled with tooltip at 1 node; (d) confirm flow calls IManualFailoverService.FailOverCentralAsync exactly once (fake service).

Step 2: Implement:

<AuthorizeView Policy="@AuthorizationPolicies.RequireAdmin">
    <button class="btn btn-outline-warning btn-sm"
            disabled="@(!CentralHasPeer)"
            title="@(CentralHasPeer ? "Gracefully restart the active node; the standby takes over."
                                    : "No standby available — failover would be an outage.")"
            @onclick="() => _showFailoverConfirm = true">
        Trigger failover
    </button>
</AuthorizeView>

plus a confirmation modal (copy the page's existing dialog idiom). The warning text MUST state: the active node restarts, roles swap, and this page will briefly disconnect and reconnect against the new active node (Traefik routes to the active). On confirm: call the service with the authenticated user name; surface the returned target address.

Step 3: Tests PASS → commit feat(ui): admin manual-failover control on the health page.

Live check (fold into the Task 7 gate when the rig is up): press the button, watch central-a restart and central-b's badge flip to Primary; audit row lands in dbo.AuditLog.

---

Task 10 (D3, OPTIONAL — confirm with the user before executing): site-pair failover from the central UI

Classification: high-risk (new cross-cluster message contract) Estimated implement time: exploration first; likely 2–3 tasks if approved

The Health page also shows per-site node cards (Primary/Standby from heartbeats), but central and sites are separate Akka clusters — a site failover needs a TriggerSiteFailover(siteId) command over the existing central→site transport (same channel as the Retry/Discard relay), handled on the site's active node by Cluster.Leave(SelfAddress). That adds a versioned message contract (rolling-upgrade surface). If approved: explore ZB.MOM.WW.ScadaBridge.Communication for the command path, mirror an existing command end-to-end, per-site button with the same guard/confirm/audit rules. Otherwise: file a follow-up issue and skip.

---

Completion

• Merge decision via the finishing-a-development-branch flow (family convention: ff-merge to main + push to gitea, or PR — ask the user).
• After merge: update scadaproj/akka_failover.md §6.1 status, scadaproj/CLAUDE.md index row, and memory ha-availability-over-partition-safety (tracked as the family-docs task in the scadaproj index plan).
• Verification-before-completion applies throughout: no task is done without its command output; live gates may be recorded deferred-live if the rig is down.