ScadaBridge/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md

# ScadaBridge: InitJoin Self-Form Fallback + Manual Failover Control — Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
>
> Shared cross-repo design: `~/Desktop/scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md` (design rationale, MNTR assessment, behavior spec). The OtOpcUa half lives in `~/Desktop/OtOpcUa/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`. This plan is self-contained for execution.

**Goal:** (1) Either node of a 2-node ScadaBridge cluster can cold-start alone and become operational, unattended. (2) An admin-only "Trigger failover" control on the Health page performs a graceful, audited role swap of the central pair.

> ## ⚠️ ARCHITECTURE REVISED DURING EXECUTION (2026-07-22)
>
> **Part 1 shipped as self-first seed ordering, NOT the `SelfFormAfter` watchdog described below.** Tasks 1–7 as originally written are superseded; what was actually built is in "Part 1 as executed". Tasks 8–10 (manual failover) are unaffected and still apply as written.
>
> **Why.** A code review of Task 2 raised, and a written test then confirmed, that the watchdog's success signal ("am I `Up` within the window?") cannot distinguish *no seed answered InitJoin* from *a seed answered and the join is in flight* — it sits outside Akka's join handshake. On a routine standby restart the peer is alive but the join stalls behind removal of the restarting node's own stale incarnation; a `Join(self)` issued during `TryingToJoin` abandons the in-flight join and forms a second cluster at the same address. **Measured: a permanent split, still unhealed after 90 s** — converting a routine restart into an outage of the previously-healthy node. That is strictly worse than the gap being closed, and it is not the boot-partition trade the design accepted.
>
> Akka's own `FirstSeedNodeProcess` already implements exactly the intended semantics — InitJoin the other seeds, self-join only if nobody answers — and, being part of the handshake, has no such race. It runs only when `seed-nodes[0]` is the node's own address. So the fix is seed **ordering**, not new runtime code.
>
> The plan's stated safety property — "a booting node only self-forms when NO seed answers InitJoin" — is true of Akka's native first-seed rule and **false** of the watchdog. That claim also appears in the shared cross-repo design doc (`scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md`) and in the OtOpcUa half; **both still need correcting** (owner deferred, 2026-07-22).

## Part 1 as executed — self-first seed ordering

**Architecture:** Every node lists ITSELF as `seed-nodes[0]` and its partner second. No new runtime code, no timer, no new option. `StartupValidator` enforces the ordering at boot (host **and** port comparison) because a broken ordering fails silently. Manual failover (Part 2) is unchanged: graceful `Cluster.Leave(oldest Up member)` via a new `IManualFailoverService` (CentralUI seam, Host implementation) — singleton drain, watchdog process-exit, supervisor restart, rejoin as youngest.

**Behavior** (all rows covered by `SelfFirstSeedBootstrapTests` — real in-process clusters from production `BuildHocon` at production failure-detection timings):

| Scenario | Behavior |
|---|---|
| Lone cold-start, peer dead | Self-joins after `seed-node-timeout` (~5 s) — operational, unattended |
| Restart into a **live** peer | Peer answers `InitJoinAck`; node rejoins, never islands |
| Both cold-start simultaneously, mutually reachable | `InitJoin` handshake converges them → **one** 2-member cluster |
| Both cold-start during a genuine boot **partition** | Each forms its own cluster — same dual-active class `auto-down` already accepts |
| Peer-first ordering (the old config) | Never forms — retained as a falsifiability control in the test suite |

**Shipped:**
- 6 node appsettings reordered (the `*-node-b` configs; `-a` nodes were already self-first). All 14 satisfy the invariant.
- `StartupValidator` self-first rule + 3 tests; `SelfFirstSeedBootstrapTests` (4 tests).
- Docs corrected: `docker/README.md`, `docs/requirements/Component-ClusterInfrastructure.md` (new **Seed Node Ordering** section), `docs/components/ClusterInfrastructure.md` (3 passages), `docs/deployment/topology-guide.md` (incl. the stale keep-oldest claim Task 6 flagged), `CLAUDE.md`.
- Several docs had asserted self-first ordering was *unsafe* because a simultaneous cold start would produce two clusters that never merge. Disproved by test (row 3) and corrected.

**⚠️ Ops action:** the gitignored `deploy/wonder-app-vd03/` overlay must have its `SeedNodes` reordered self-first before its next deploy, or the node will now **refuse to boot**. The validator rule is a hard gate deliberately — the alternative is the silent wedge it replaces.

---

<details>
<summary>Original Part 1 architecture (SUPERSEDED — kept for the decision record)</summary>

**Architecture:** New `ClusterOptions.SelfFormAfter` (`TimeSpan?`, default 10 s, `null`/`≤0` disables, appsettings-bound) arms `ClusterBootstrapFallback` right after ActorSystem creation: wait for membership via `RegisterOnMemberUp`; on expiry, `Cluster.Join(SelfAddress)`. Safety gate: fires only when this node's own address is in its own seed list.

</details>

**Tech Stack:** .NET 10, Akka.NET 1.5.62, Blazor Server (CentralUI), bUnit, xunit. No new packages.

**Branch:** `feat/selfform-fallback` off `main`.

---

## Design essentials (from the shared design doc)

**The defect:** Akka only lets the FIRST listed seed self-join to form a *new* cluster; every other node loops on `InitJoin` forever. Both docker central nodes list `central-a` first, so a lone cold-starting `central-b` never comes Up (the "registered outage gap" — `docker/README.md:289`). `ClusterOptions.SeedNodes`' doc comment claims "either can start first", which the deployed configs do not deliver — this plan makes it true and fixes the comment.

**Behavior spec:**

| Scenario | Behavior with fallback |
|---|---|
| Peer alive (any boot order) | Normal seed join in ms — fallback never fires |
| Lone cold-start, self IS in own seed list | After `SelfFormAfter`: warn log + `Cluster.Join(SelfAddress)` → Up alone, singletons start (`min-nr-of-members=1`) |
| Lone cold-start, self NOT in own seed list | Fallback inert (info log) — self-forming would island the node from the real seeds |
| Peer boots after survivor self-formed | Peer's InitJoin is answered → joins as youngest/standby. No island. |
| Both cold-start simultaneously, mutually unreachable | Both self-form → dual-active (same partition class auto-down accepts; restart one side) |
| `SelfFormAfter` null/`≤0` | Disabled — today's wait-forever behavior |
| Window expires mid-join-handshake | Benign: Akka ignores `Join` once joined |

**Manual failover rules:** graceful `Leave`, never `Down`; admin-only (`AuthorizationPolicies.RequireAdmin`); peer guard (disabled when <2 Up `Central` members); confirmation dialog warning the Blazor circuit will drop (Traefik routes the UI to the active node — triggering failover disconnects your own page, which reconnects against the new active); audited via the app's **central** audit writer (`ICentralAuditWriter` — NOT the shared seam, see the dual-seam gotcha) before the Leave is issued. No interplay with `SelfFormAfter` (the peer is alive on this path, so the restarted node rejoins normally).

**Multi-node TestKit:** assessed and NOT used — in-process real clusters via `TwoNodeClusterFixture` (production `BuildHocon`) cover every deterministic scenario; MNTR would need a dedicated no-parallelization test project for no added coverage. See the shared design doc for the full verdict.

---

### Task 1 (A1): `SelfFormAfter` option + validator

**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** none (first task)

**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs`
- Modify: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptionsValidator.cs`
- Test: `tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsTests.cs`
- Test: `tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsValidatorTests.cs`

**Step 1: Create the branch**

```bash
cd ~/Desktop/ScadaBridge && git checkout main && git checkout -b feat/selfform-fallback
```

**Step 2: Write the failing tests** (append to the existing test classes, matching their assertion style — read them first)

```csharp
// ClusterOptionsTests.cs
[Fact]
public void SelfFormAfter_defaults_to_ten_seconds()
{
    new ClusterOptions().SelfFormAfter.ShouldBe(TimeSpan.FromSeconds(10));
}

// ClusterOptionsValidatorTests.cs
[Fact] public void SelfFormAfter_null_passes_validation() { /* valid options + null → Succeeded */ }
[Fact] public void SelfFormAfter_zero_passes_validation() { /* zero = explicit disable → Succeeded */ }
[Fact] public void SelfFormAfter_negative_fails_validation() { /* -1s → Failed, message mentions SelfFormAfter */ }
```

**Step 3: Run to verify failure**

```bash
dotnet test tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests --filter "FullyQualifiedName~SelfFormAfter"
```
Expected: FAIL (compile error — property doesn't exist).

**Step 4: Implement.** `ClusterOptions.cs` — add after `AllowSingleNodeCluster` (line 104):

```csharp
    /// <summary>
    /// Bootstrap self-form fallback window (decision 2026-07-22, scadaproj/akka_failover.md §6.1).
    /// Akka only lets the FIRST listed seed form a new cluster; a non-first seed cold-starting
    /// while its peer is down loops on InitJoin forever. When this node has waited longer than
    /// this window without becoming a cluster member, it forms a cluster on itself
    /// (<c>Cluster.Join(SelfAddress)</c>) — but ONLY if its own address is in its seed list
    /// (a non-seed node self-forming would create a permanent island). Default 10s: the pair
    /// shares a datacenter, so a live peer answers InitJoin in milliseconds and waiting longer
    /// buys nothing. <c>null</c> or a non-positive value disables the fallback (wait-forever).
    /// Accepted trade: both nodes cold-starting within the window while mutually unreachable
    /// form two clusters — the same partition class the auto-down strategy already accepts.
    /// </summary>
    public TimeSpan? SelfFormAfter { get; set; } = TimeSpan.FromSeconds(10);
```

`ClusterOptionsValidator.cs` — inside `Validate`, after the `FailureDetectionThreshold` rules:

```csharp
        builder.RequireThat(options.SelfFormAfter is null || options.SelfFormAfter.Value >= TimeSpan.Zero,
            "ClusterOptions.SelfFormAfter must be null (disabled), zero (disabled) or a positive duration; "
            + "a negative value is always a configuration mistake.");
```

**Step 5: Run tests → PASS**

```bash
dotnet test tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests
```

**Step 6: Commit**

```bash
git add -A && git commit -m "feat(cluster): SelfFormAfter option — bootstrap self-form fallback window"
```

---

### Task 2 (A2): `ClusterBootstrapFallback` + first integration test

**Classification:** high-risk (cluster formation behavior)
**Estimated implement time:** ~5 min
**Parallelizable with:** none

**Files:**
- Create: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/ClusterBootstrapFallback.cs`
- Create: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SelfFormBootstrapTests.cs`

**Step 1: Write the failing test**

```csharp
using Akka.Actor;
using Akka.Cluster;
using Akka.Configuration;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
using ZB.MOM.WW.ScadaBridge.Host;
using ZB.MOM.WW.ScadaBridge.Host.Actors;

namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;

/// <summary>
/// Guards the InitJoin self-form fallback (decision 2026-07-22): Akka only lets the FIRST
/// listed seed form a new cluster, so without the fallback a non-first seed cold-starting
/// alone waits on InitJoin forever — the "registered outage gap". These tests build REAL
/// single/dual-node clusters from the production BuildHocon output, exactly like
/// TwoNodeClusterFixture, and arm the production fallback.
/// </summary>
public sealed class SelfFormBootstrapTests : IAsyncLifetime
{
    private readonly List<ActorSystem> _systems = new();

    /// <summary>Starts a node whose seed list puts the PEER first (self second, or absent),
    /// so Akka's own first-seed rule can never self-form it — only the fallback can.</summary>
    private ActorSystem StartNode(int selfPort, int peerPort, TimeSpan? selfFormAfter, bool selfInSeeds = true)
    {
        var nodeOptions = new NodeOptions { Role = "Central", NodeHostname = "127.0.0.1", RemotingPort = selfPort };
        var clusterOptions = new ClusterOptions
        {
            SeedNodes = selfInSeeds
                ? new List<string>
                {
                    $"akka.tcp://scadabridge@127.0.0.1:{peerPort}",
                    $"akka.tcp://scadabridge@127.0.0.1:{selfPort}",
                }
                : new List<string> { $"akka.tcp://scadabridge@127.0.0.1:{peerPort}" },
            SelfFormAfter = selfFormAfter,
            StableAfter = TimeSpan.FromSeconds(3),
            HeartbeatInterval = TimeSpan.FromMilliseconds(500),
            FailureDetectionThreshold = TimeSpan.FromSeconds(2),
            MinNrOfMembers = 1,
            AllowSingleNodeCluster = !selfInSeeds,
        };
        var hocon = AkkaHostedService.BuildHocon(
            nodeOptions, clusterOptions, new[] { "Central" },
            TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(3));
        var system = ActorSystem.Create("scadabridge", ConfigurationFactory.ParseString(hocon));
        _systems.Add(system);
        ClusterBootstrapFallback.Arm(system, clusterOptions, NullLogger.Instance);
        return system;
    }

    [Fact]
    public async Task Lone_non_first_seed_self_forms_after_the_window()
    {
        var selfPort = TwoNodeClusterFixture.GetFreeTcpPort();
        var deadPeerPort = TwoNodeClusterFixture.GetFreeTcpPort(); // nothing listening
        var node = StartNode(selfPort, deadPeerPort, selfFormAfter: TimeSpan.FromSeconds(2));

        // Without the fallback this waits forever (Akka first-seed rule). With it, the node
        // must be a 1-member Up cluster shortly after the 2s window.
        await TwoNodeClusterFixture.WaitForMembersUp(node, 1, TimeSpan.FromSeconds(20));
        Cluster.Get(node).SelfMember.Status.ShouldBe(MemberStatus.Up);
    }

    public Task InitializeAsync() => Task.CompletedTask;

    public async Task DisposeAsync()
    {
        foreach (var s in _systems)
        {
            try { await s.Terminate().WaitAsync(TimeSpan.FromSeconds(10)); } catch { /* teardown */ }
        }
    }
}
```

(Match the assertion library to `SbrFailoverTests.cs` — Shouldly vs xunit `Assert`.)

**Step 2: Run to verify failure** — compile error (`ClusterBootstrapFallback` doesn't exist):

```bash
dotnet test tests/ZB.MOM.WW.ScadaBridge.IntegrationTests --filter "FullyQualifiedName~SelfFormBootstrapTests"
```

**Step 3: Implement `ClusterBootstrapFallback.cs`**

```csharp
using Akka.Actor;
using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;

namespace ZB.MOM.WW.ScadaBridge.Host.Actors;

/// <summary>
/// InitJoin self-form fallback (decision 2026-07-22, scadaproj/akka_failover.md §6.1).
/// Akka only lets the FIRST listed seed form a NEW cluster; every other node retries InitJoin
/// forever. So "both nodes are seed nodes" (ClusterOptions.SeedNodes) does NOT mean either can
/// cold-start alone — a non-first seed booting while its peer is down waits indefinitely (the
/// "registered outage gap", docker/README.md). This watchdog waits <see cref="ClusterOptions.SelfFormAfter"/>
/// for membership; on expiry it forms a cluster on itself.
///
/// <para><b>Island safety.</b> Fires ONLY when this node's own address is in its own seed list.
/// A node that is not a seed (never legitimately first) must keep waiting: if it self-formed,
/// a later-booting real seed would form a second cluster and the two can never merge. For nodes
/// that ARE seeds, sequential recovery is island-free — Akka's join protocol prefers an existing
/// cluster (a booting node only self-forms when NO seed answers InitJoin), so a peer booting
/// after this node self-formed simply joins it.</para>
///
/// <para><b>Races are benign.</b> If the join completes between window expiry and
/// <c>Cluster.Join(SelfAddress)</c>, Akka ignores the join — a node joins a cluster at most once
/// per incarnation. The residual risk is both nodes cold-starting inside the window while
/// mutually unreachable (a boot-time partition): both self-form, the same dual-active class the
/// auto-down downing strategy already accepts, with the same recovery (restart one side).</para>
/// </summary>
public static class ClusterBootstrapFallback
{
    public static void Arm(ActorSystem system, ClusterOptions options, ILogger logger)
    {
        if (options.SelfFormAfter is not { } window || window <= TimeSpan.Zero)
        {
            logger.LogInformation(
                "Cluster self-form fallback disabled (SelfFormAfter not set) — a node cold-starting "
                + "while its peer is down will wait on InitJoin indefinitely.");
            return;
        }

        var cluster = Akka.Cluster.Cluster.Get(system);
        var self = cluster.SelfAddress;
        var isSeed = options.SeedNodes.Any(s => TryParseAddress(s, out var a) && a.Equals(self));
        if (!isSeed)
        {
            logger.LogInformation(
                "Cluster self-form fallback inactive: this node ({Self}) is not in its own seed list "
                + "[{Seeds}] — self-forming here would island it from the real seeds.",
                self, string.Join(", ", options.SeedNodes));
            return;
        }

        var joined = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
        cluster.RegisterOnMemberUp(() => joined.TrySetResult());

        _ = Task.Run(async () =>
        {
            var winner = await Task.WhenAny(joined.Task, Task.Delay(window));
            if (winner == joined.Task || system.WhenTerminated.IsCompleted)
                return;
            logger.LogWarning(
                "No cluster membership after {Window} — no seed answered InitJoin (peer down at boot). "
                + "Self-forming a cluster at {Self} so this node becomes operational; if the peer was "
                + "merely partitioned (not dead), the pair is now dual-active — restart one side after "
                + "the partition heals (accepted availability-first trade, decision 2026-07-22).",
                window, self);
            cluster.Join(self);
        });
    }

    private static bool TryParseAddress(string seed, out Address address)
    {
        try { address = Address.Parse(seed); return true; }
        catch { address = default!; return false; }
    }
}
```

(Add the `Microsoft.Extensions.Logging` using. If non-generic `TaskCompletionSource` is unavailable, use `TaskCompletionSource<bool>` + `TrySetResult(true)`.)

**Step 4: Run → PASS** (~5–10 s). **Step 5: Commit**

```bash
git add -A && git commit -m "feat(cluster): InitJoin self-form fallback — lone non-first seed becomes Up"
```

---

### Task 3 (A3): Remaining fallback tests — disabled / late-peer merge / non-seed

**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 4

**Files:**
- Modify: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SelfFormBootstrapTests.cs`

**Step 1: Append three tests** (each absence assertion carries an in-test positive control — repo convention):

```csharp
    [Fact]
    public async Task Disabled_fallback_keeps_waiting_and_the_node_was_otherwise_formable()
    {
        var selfPort = TwoNodeClusterFixture.GetFreeTcpPort();
        var deadPeerPort = TwoNodeClusterFixture.GetFreeTcpPort();
        var node = StartNode(selfPort, deadPeerPort, selfFormAfter: null);

        await Task.Delay(TimeSpan.FromSeconds(6)); // 3x the window used in the enabled test
        var cluster = Cluster.Get(node);
        cluster.State.Members.ShouldBeEmpty(); // still InitJoin-looping — today's behavior

        // POSITIVE CONTROL: prove the node COULD have formed; only the fallback was missing.
        cluster.Join(cluster.SelfAddress);
        await TwoNodeClusterFixture.WaitForMembersUp(node, 1, TimeSpan.FromSeconds(20));
    }

    [Fact]
    public async Task Peer_booting_after_self_form_joins_the_existing_cluster_no_island()
    {
        var portA = TwoNodeClusterFixture.GetFreeTcpPort();
        var portB = TwoNodeClusterFixture.GetFreeTcpPort();

        // B cold-starts alone (A dead), self-forms after 2s.
        var nodeB = StartNode(portB, peerPort: portA, selfFormAfter: TimeSpan.FromSeconds(2));
        await TwoNodeClusterFixture.WaitForMembersUp(nodeB, 1, TimeSpan.FromSeconds(20));

        // A boots later with the pair seed list. B answers InitJoin, so A must JOIN B's
        // cluster instead of islanding. Generous window on A so its fallback can't race.
        var nodeA = StartNode(portA, peerPort: portB, selfFormAfter: TimeSpan.FromSeconds(30));
        await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 2, TimeSpan.FromSeconds(20));
        await TwoNodeClusterFixture.WaitForMembersUp(nodeB, 2, TimeSpan.FromSeconds(20));
    }

    [Fact]
    public async Task Non_seed_node_never_self_forms()
    {
        var selfPort = TwoNodeClusterFixture.GetFreeTcpPort();
        var deadSeedPort = TwoNodeClusterFixture.GetFreeTcpPort();
        var node = StartNode(selfPort, deadSeedPort, selfFormAfter: TimeSpan.FromSeconds(1), selfInSeeds: false);

        await Task.Delay(TimeSpan.FromSeconds(5)); // 5x the window
        var cluster = Cluster.Get(node);
        cluster.State.Members.ShouldBeEmpty(); // guard refused to island a non-seed

        // Positive control: the guard (not the environment) prevented formation.
        cluster.Join(cluster.SelfAddress);
        await TwoNodeClusterFixture.WaitForMembersUp(node, 1, TimeSpan.FromSeconds(20));
    }
```

**Step 2: Run** — all 4 PASS. **Step 3: Commit** `test(cluster): self-form fallback — disabled, late-peer merge, non-seed guard`.

---

### Task 4 (A4): Production wiring + fix the misleading `SeedNodes` doc comment

**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** Task 3

**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs` (inside `GetOrCreateActorSystem`, after the `WhenTerminated` continuation ending ~line 218, before `_actorSystem = system;`)
- Modify: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs:33-38`

**Step 1: Wire the fallback:**

```csharp
            // InitJoin self-form fallback (decision 2026-07-22): without it a non-first seed
            // cold-starting while its peer is down loops on InitJoin forever — auto-down closed
            // the crash-failover gap, this closes the cold-start-alone gap. Guarded inside Arm:
            // disabled when SelfFormAfter is unset, inert when this node is not its own seed.
            ClusterBootstrapFallback.Arm(system, _clusterOptions, _logger);
```

**Step 2: Fix the `SeedNodes` doc comment** (replace the `<summary>`):

```csharp
    /// <summary>
    /// Akka.NET cluster seed nodes. Both nodes are seed nodes — each node lists itself and its
    /// partner. NOTE: listing both is NOT sufficient for "either can start first": Akka only lets
    /// the FIRST listed seed form a new cluster, so a lone non-first seed waits on InitJoin until
    /// <see cref="SelfFormAfter"/> expires and the self-form fallback fires (decision 2026-07-22).
    /// Must contain at least one entry.
    /// </summary>
```

**Step 3: Build + targeted tests**

```bash
dotnet build ZB.MOM.WW.ScadaBridge.slnx      # 0 warnings (TreatWarningsAsErrors)
dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests
dotnet test tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests
```

**Step 4: Commit** `feat(cluster): arm self-form fallback at ActorSystem creation; honest SeedNodes doc`.

---

### Task 5 (A5): appsettings sweep

**Classification:** trivial
**Estimated implement time:** ~3 min
**Parallelizable with:** Task 6

Add `"SelfFormAfter": "00:00:10"` next to `"SplitBrainResolverStrategy"` in the `ScadaBridge:Cluster` section of each (explicit for operator visibility; matches the code default):

- `src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Central.json`, `appsettings.Site.json`
- `docker/central-node-a/appsettings.Central.json`, `docker/central-node-b/appsettings.Central.json`
- `docker/site-a-node-a/appsettings.Site.json`, `docker/site-a-node-b/appsettings.Site.json`
- `docker/site-b-node-a/appsettings.Site.json`, `docker/site-b-node-b/appsettings.Site.json`
- `docker/site-c-node-a/appsettings.Site.json`, `docker/site-c-node-b/appsettings.Site.json`
- `docker-env2/` — the 4 node appsettings files

Then `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests` and commit `config(cluster): SelfFormAfter=10s explicit in all node appsettings`.

> Ops note (do NOT edit here): the gitignored `deploy/wonder-app-vd03/` overlay gets the same key on the next production deploy.

---

### Task 6 (A6): Docs

**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 5

- `docs/requirements/Component-ClusterInfrastructure.md` — replace the "registered outage gap" recovery text (env-var override) with `SelfFormAfter` semantics + island guard + boot-partition trade.
- `docker/README.md` (~line 289) — same replacement; keep the partition-trade note.
- `docs/deployment/topology-guide.md:101` — **fix the stale keep-oldest claim** (still says "Keep-oldest with `down-if-alone = on`"): rewrite to auto-down default + `SelfFormAfter`, pointing at the decision records.
- `CLAUDE.md` (~line 222) — update the boot-order note: pairs no longer require the first seed for cold start; note the 10 s window.

Commit: `docs(cluster): SelfFormAfter fallback; fix stale keep-oldest note in topology guide`.

---

### Task 7 (A7): Full verification + optional docker live gate

**Classification:** standard
**Estimated implement time:** ~5 min (suite runtime dominates)
**Parallelizable with:** none

**Step 1:**

```bash
cd ~/Desktop/ScadaBridge
dotnet build ZB.MOM.WW.ScadaBridge.slnx   # expect 0 warnings
dotnet test ZB.MOM.WW.ScadaBridge.slnx    # expect green vs pre-existing baseline
```

**Step 2 (LIVE GATE — run if the docker rig is available; else record deferred-live):**

```bash
cd docker && bash deploy.sh               # rebuild with the fallback
docker compose stop central-a central-b   # verify service names in docker-compose.yml first
docker compose start central-b            # cold-start ONLY the non-first seed
docker compose logs -f central-b | grep -m1 "Self-forming a cluster"   # ≈10s after start
curl -fsS http://localhost:9002/health/active                          # expect 200
docker compose start central-a            # first seed returns → must JOIN, not island
docker compose logs central-a | grep -i "Welcome"
```

Expected: `central-b` self-forms ≈10 s and serves (previously a permanent wedge); `central-a` rejoins as youngest/standby.

---

### Task 8 (D1): `IManualFailoverService` + cluster-level test

**Classification:** high-risk (cluster behavior)
**Estimated implement time:** ~5 min
**Parallelizable with:** none

**Files:**
- Create: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/IManualFailoverService.cs`
- Create: `src/ZB.MOM.WW.ScadaBridge.Host/Health/AkkaManualFailoverService.cs`
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` (register in the Central branch, next to `IActiveNodeGate` ~line 330)
- Test: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/ManualFailoverTests.cs`

**Step 1: Failing tests** on `TwoNodeClusterFixture`:

```csharp
[Fact]
public async Task Failover_makes_the_oldest_leave_and_the_survivor_take_over()
{
    await using var f = await TwoNodeClusterFixture.StartAsync();
    var oldest = Akka.Cluster.Cluster.Get(f.NodeA);   // NodeA started first = oldest
    var target = AkkaManualFailoverService.FailOverCore(f.NodeB, "Central"); // issued from the OTHER node
    target.ShouldBe(oldest.SelfAddress);

    // Graceful exit path: the left node's own ActorSystem terminates…
    await f.NodeA.WhenTerminated.WaitAsync(TimeSpan.FromSeconds(30));
    // …and the survivor becomes a 1-member cluster and the oldest-Up active node.
    await TwoNodeClusterFixture.WaitForMemberRemoved(f.NodeB, oldest.SelfAddress, TimeSpan.FromSeconds(30));
    ActiveNodeEvaluator.SelfIsOldestUp(Akka.Cluster.Cluster.Get(f.NodeB)).ShouldBeTrue();
}

[Fact]
public void Failover_refuses_when_no_peer_exists()
{ /* 1-node cluster → FailOverCore returns null, node still Up afterwards (positive assert) */ }
```

**Step 2:** Run → FAIL (service missing).

**Step 3: Implement.** Interface in CentralUI (CentralUI stays Akka-free):

```csharp
public interface IManualFailoverService
{
    /// <summary>Gracefully fails over the central cluster: the current active (oldest Up)
    /// member leaves, restarts via its supervisor, and rejoins as standby. Returns the
    /// address string acted on, or null when there is no peer to fail over to.</summary>
    Task<string?> FailOverCentralAsync(string actor);
}
```

Host implementation — static testable core + thin DI wrapper:

```csharp
public sealed class AkkaManualFailoverService : IManualFailoverService
{
    // ctor: (AkkaHostedService akka, ICentralAuditWriter audit, ILogger<AkkaManualFailoverService> logger)

    public async Task<string?> FailOverCentralAsync(string actor)
    {
        var system = _akka.GetOrCreateActorSystem();
        var target = FailOverCore(system, role: "Central", dryRun: true);
        if (target is null) return null;                       // peer guard
        await _audit.WriteAsync(/* canonical AuditEvent: Action=cluster.manual-failover,
            Actor=actor, DetailsJson={"target": target}, Outcome=Success — copy the exact
            call shape from an existing audited admin action (e.g. the Sites admin service);
            use the CENTRAL audit writer, not the shared seam */);
        FailOverCore(system, role: "Central");                 // Cluster.Leave(target)
        _logger.LogWarning("Manual failover triggered by {Actor}: {Target} is leaving the cluster.", actor, target);
        return target.ToString();
    }

    /// <summary>Oldest Up member with the role leaves — mirrors ActiveNodeEvaluator's oldest-Up
    /// rule so the node acted on is exactly the one hosting the singletons. Returns null when
    /// fewer than 2 Up members carry the role (no peer = failover would be an outage).</summary>
    public static Address? FailOverCore(ActorSystem system, string role, bool dryRun = false)
    {
        var cluster = Akka.Cluster.Cluster.Get(system);
        var withRole = cluster.State.Members
            .Where(m => m.Status == MemberStatus.Up && m.HasRole(role))
            .OrderBy(m => m, Member.AgeOrdering)
            .ToList();
        if (withRole.Count < 2) return null;
        var oldest = withRole[0];
        if (!dryRun) cluster.Leave(oldest.Address);
        return oldest.Address;
    }
}
```

**Step 4:** Register in the Central branch of `Program.cs`. **Step 5:** Tests PASS → commit `feat(ui): manual central failover service — graceful Leave of the oldest Up member`.

---

### Task 9 (D2): Health page button + bUnit tests + runbook

**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none

**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor` (central-cluster card, near the Nodes column ~line 233)
- Test: `tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/HealthFailoverButtonTests.cs` (follow the project's existing bUnit page-test pattern)
- Modify: `docs/requirements/Component-ClusterInfrastructure.md` + `docker/README.md` — manual-failover runbook paragraph

**Step 1: Failing bUnit tests:** (a) button absent without the admin policy; (b) present + enabled for admin with ≥2 online central nodes; (c) disabled with tooltip at 1 node; (d) confirm flow calls `IManualFailoverService.FailOverCentralAsync` exactly once (fake service).

**Step 2: Implement:**

```razor
<AuthorizeView Policy="@AuthorizationPolicies.RequireAdmin">
    <button class="btn btn-outline-warning btn-sm"
            disabled="@(!CentralHasPeer)"
            title="@(CentralHasPeer ? "Gracefully restart the active node; the standby takes over."
                                    : "No standby available — failover would be an outage.")"
            @onclick="() => _showFailoverConfirm = true">
        Trigger failover
    </button>
</AuthorizeView>
```

plus a confirmation modal (copy the page's existing dialog idiom). The warning text MUST state: the active node restarts, roles swap, and this page will briefly disconnect and reconnect against the new active node (Traefik routes to the active). On confirm: call the service with the authenticated user name; surface the returned target address.

**Step 3:** Tests PASS → commit `feat(ui): admin manual-failover control on the health page`.

**Live check (fold into the Task 7 gate when the rig is up):** press the button, watch `central-a` restart and `central-b`'s badge flip to Primary; audit row lands in `dbo.AuditLog`.

---

### Task 10 (D3, OPTIONAL — confirm with the user before executing): site-pair failover from the central UI

**Classification:** high-risk (new cross-cluster message contract)
**Estimated implement time:** exploration first; likely 2–3 tasks if approved

The Health page also shows per-site node cards (Primary/Standby from heartbeats), but central and sites are **separate Akka clusters** — a site failover needs a `TriggerSiteFailover(siteId)` command over the existing central→site transport (same channel as the Retry/Discard relay), handled on the site's active node by `Cluster.Leave(SelfAddress)`. That adds a versioned message contract (rolling-upgrade surface). If approved: explore `ZB.MOM.WW.ScadaBridge.Communication` for the command path, mirror an existing command end-to-end, per-site button with the same guard/confirm/audit rules. Otherwise: file a follow-up issue and skip.

---

## Completion

- Merge decision via the finishing-a-development-branch flow (family convention: ff-merge to `main` + push to gitea, or PR — ask the user).
- After merge: update `scadaproj/akka_failover.md` §6.1 status, `scadaproj/CLAUDE.md` index row, and memory `ha-availability-over-partition-safety` (tracked as the family-docs task in the scadaproj index plan).
- Verification-before-completion applies throughout: no task is done without its command output; live gates may be recorded deferred-live if the rig is down.