lmxopcua/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md

# OtOpcUa: InitJoin Self-Form Fallback + Manual Failover Control — Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
>
> Shared cross-repo design: `~/Desktop/scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md` (design rationale, MNTR assessment, behavior spec). The ScadaBridge half lives in `~/Desktop/ScadaBridge/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`. This plan is self-contained for execution.
>
> **⚠ This plan is expected to GROW** — see "Reserved for additions" at the end. Append new tasks there (and to the `.tasks.json`) rather than renumbering existing ones.

**Goal:** (1) Either node of a 2-node OtOpcUa pair can cold-start alone and become operational, unattended — a configurable, fast (default 10 s) InitJoin timeout falls back to self-forming a cluster, with a hard guard so site nodes seeded only by central can never island themselves. (2) An admin-only "Trigger failover" control on the Cluster Redundancy page performs a graceful, audited swap of the driver Primary.

**Architecture:** New `AkkaClusterOptions.SelfFormAfter` (`TimeSpan?`, default 10 s, `null`/`≤0` disables, bound from the `Cluster` section) arms `ClusterBootstrapFallback` via an Akka.Hosting startup task inside `WithOtOpcUaClusterBootstrap` — so production AND the Cluster.Tests hosts get it identically. Wait for membership via `RegisterOnMemberUp`; on expiry, `Cluster.Join(SelfAddress)` — only if this node's own address is in its own `SeedNodes`. Manual failover = graceful `Cluster.Leave(<oldest Up driver member>)` via a new `IManualFailoverService` in ControlPlane, surfaced on `/clusters/{id}/redundancy`.

**Tech Stack:** .NET 10, Akka.NET 1.5.62, Akka.Hosting 1.5.62 (`AddStartup`), Blazor Server (AdminUI, InteractiveServer), bUnit, xunit. No new packages.

**Branch:** `feat/selfform-fallback` off `master`.

---

## Design essentials (from the shared design doc)

**The defect:** Akka only lets the FIRST listed seed self-join to form a *new* cluster; every other node loops on `InitJoin` forever — "auto-down removes the crash outage, not this one" (`docs/Redundancy.md:244-247`). A lone cold-starting `central-2` (or a pair node under the future per-cluster mesh) never comes Up.

**Behavior spec:**

| Scenario | Behavior with fallback |
|---|---|
| Peer alive (any boot order) | Normal seed join in ms — fallback never fires |
| Lone cold-start, self IS in own `SeedNodes` | After `SelfFormAfter`: warn log + `Cluster.Join(SelfAddress)` → Up alone (`min-nr-of-members=1`) |
| Lone cold-start, self NOT in own `SeedNodes` | Fallback inert (info log). **This is the docker-dev site-node topology** — site-a/b nodes list only `central-1`; self-forming there would island them permanently, so they must wait. |
| Peer boots after survivor self-formed | Its InitJoin is answered → joins as youngest. No island. |
| Both pair nodes cold-start simultaneously, mutually unreachable | Both self-form → dual-active (same partition class auto-down accepts; restart one side) |
| `SelfFormAfter` null/`≤0` | Disabled — today's wait-forever behavior |
| Window expires mid-join-handshake | Benign: Akka ignores `Join` once joined |

**Manual failover rules:** graceful `Leave` of the **oldest Up `driver` member** (identical query to `RedundancyStateActor.SelectDriverPrimary` — `Member.AgeOrdering`, `Leaving` excluded via `Status == Up`), never `Down`. The leaving node hands over via the cluster-leave phases, `CoordinatedShutdown` runs, `ActorSystemTerminationWatchdog` exits the process, the supervisor restarts it, and it rejoins as youngest → the survivor is Primary (ServiceLevel 250 moves; OPC UA clients re-select). Admin-only; peer guard (<2 Up driver members → disabled); confirmation dialog; audited via the shared `ZB.MOM.WW.Audit.IAuditWriter` seam before the Leave. **Mesh-scope caveat:** until the per-cluster mesh lands, the election is mesh-wide — the button acts on THE Primary of the whole Akka cluster, not per-`ServerCluster` row; the UI must say so (mirrors the KNOWN LIMITATION in `docs/Redundancy.md:84-99`).

**Multi-node TestKit:** assessed, NOT used — everything is deterministic in-process via real hosts through `WithOtOpcUaClusterBootstrap` (the `SplitBrainResolverActivationTests` pattern), and the Akka TestKit family is xunit-v2-only while this repo's integration tree is xunit.v3. Full verdict in the shared design doc.

---

### Task 1 (B1): `SelfFormAfter` on `AkkaClusterOptions`

**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** none (first task)

**Files:**
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptions.cs`

**Step 1: Branch**

```bash
cd ~/Desktop/OtOpcUa && git checkout master && git checkout -b feat/selfform-fallback
```

**Step 2: Add the property** (after `SplitBrainResolverStrategy`, mirroring its XML-doc depth):

```csharp
    /// <summary>
    /// Bootstrap self-form fallback window (decision 2026-07-22, scadaproj/akka_failover.md §6.1).
    /// Akka only lets the FIRST listed seed form a new cluster; a non-first seed cold-starting
    /// while its peer is down loops on InitJoin forever ("auto-down removes the crash outage,
    /// not this one" — docs/Redundancy.md). When this node has waited longer than this window
    /// without cluster membership it joins itself — but ONLY if its own address appears in
    /// <see cref="SeedNodes"/>; a non-seed node (e.g. a site node whose only seed is central-1)
    /// stays waiting, because self-forming there creates a permanent island. Default 10s
    /// (same-datacenter pair; a live peer answers InitJoin in milliseconds). <c>null</c> or a
    /// non-positive value disables the fallback. Accepted trade: both pair nodes cold-starting
    /// inside the window while mutually unreachable form two clusters — the same dual-active
    /// class the auto-down strategy already accepts, same recovery (restart one side).
    /// </summary>
    public TimeSpan? SelfFormAfter { get; set; } = TimeSpan.FromSeconds(10);
```

**Step 3:** `dotnet build src/Core/ZB.MOM.WW.OtOpcUa.Cluster` — 0 warnings. Commit:

```bash
git add -A && git commit -m "feat(cluster): SelfFormAfter option (bootstrap self-form fallback window)"
```

(The default-value pin test lands with Task 3's test file — the property is inert until Task 2.)

---

### Task 2 (B2): `ClusterBootstrapFallback` + `AddStartup` wiring

**Classification:** high-risk (cluster formation behavior)
**Estimated implement time:** ~5 min
**Parallelizable with:** none

**Files:**
- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ClusterBootstrapFallback.cs`
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs` (`WithOtOpcUaClusterBootstrap`, after the `AddHocon(BuildDowningHocon(options), HoconAddMode.Prepend)` call at line 93)

**Step 1: Implement the fallback class** (deliberately duplicated from ScadaBridge's, like the termination watchdogs — see that repo's `ClusterBootstrapFallback.cs` for the full doc comment to adapt):

```csharp
using Akka.Actor;
using Microsoft.Extensions.Logging;

namespace ZB.MOM.WW.OtOpcUa.Cluster;

/// <summary>
/// InitJoin self-form fallback (decision 2026-07-22, scadaproj/akka_failover.md §6.1). Akka only
/// lets the FIRST listed seed form a NEW cluster; every other node retries InitJoin forever —
/// "auto-down removes the crash outage, not this one" (docs/Redundancy.md). Waits
/// <see cref="AkkaClusterOptions.SelfFormAfter"/> for membership, then joins itself.
/// <para><b>Island safety:</b> fires ONLY when this node's own address is in its own
/// <see cref="AkkaClusterOptions.SeedNodes"/> — a site node seeded only by central-1 must wait,
/// not island. Sequential recovery is island-free: Akka's join protocol prefers an existing
/// cluster (a booting node only self-forms when NO seed answers InitJoin).</para>
/// <para><b>Benign race:</b> Akka ignores <c>Join</c> once the node has joined a cluster this
/// incarnation. Residual risk: both pair nodes cold-starting inside the window while mutually
/// unreachable self-form separately — the same dual-active class auto-down accepts.</para>
/// </summary>
public static class ClusterBootstrapFallback
{
    public static void Arm(ActorSystem system, AkkaClusterOptions options, ILogger logger)
    {
        if (options.SelfFormAfter is not { } window || window <= TimeSpan.Zero)
        {
            logger.LogInformation(
                "Cluster self-form fallback disabled (SelfFormAfter not set) — a node cold-starting "
                + "while its peer is down will wait on InitJoin indefinitely.");
            return;
        }

        var cluster = Akka.Cluster.Cluster.Get(system);
        var self = cluster.SelfAddress;
        var isSeed = options.SeedNodes.Any(s => TryParseAddress(s, out var a) && a.Equals(self));
        if (!isSeed)
        {
            logger.LogInformation(
                "Cluster self-form fallback inactive: this node ({Self}) is not in its own seed list "
                + "[{Seeds}] — self-forming here would island it from the real seeds.",
                self, string.Join(", ", options.SeedNodes));
            return;
        }

        var joined = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
        cluster.RegisterOnMemberUp(() => joined.TrySetResult());

        _ = Task.Run(async () =>
        {
            var winner = await Task.WhenAny(joined.Task, Task.Delay(window));
            if (winner == joined.Task || system.WhenTerminated.IsCompleted)
                return;
            logger.LogWarning(
                "No cluster membership after {Window} — no seed answered InitJoin (peer down at boot). "
                + "Self-forming a cluster at {Self} so this node becomes operational; if the peer was "
                + "merely partitioned (not dead), the pair is now dual-active — restart one side after "
                + "the partition heals (accepted availability-first trade, decision 2026-07-22).",
                window, self);
            cluster.Join(self);
        });
    }

    private static bool TryParseAddress(string seed, out Address address)
    {
        try { address = Address.Parse(seed); return true; }
        catch { address = default!; return false; }
    }
}
```

(If the Cluster project lacks a `Microsoft.Extensions.Logging.Abstractions` reference, add the `PackageReference` — check `Directory.Packages.props` for the pinned version first.)

**Step 2: Wire via Akka.Hosting startup hook** — in `WithOtOpcUaClusterBootstrap`, after the downing HOCON line (~93):

```csharp
        // InitJoin self-form fallback (decision 2026-07-22): registered as an Akka.Hosting
        // startup task so every host built through this bootstrap — production AND the test
        // hosts in Cluster.Tests — arms it identically. Guarded inside Arm: disabled when
        // SelfFormAfter is unset; inert when this node is not in its own seed list (site nodes
        // seeded only by central-1 must wait, not island).
        var fallbackLogger = (serviceProvider.GetService<ILoggerFactory>()
                              ?? Microsoft.Extensions.Logging.Abstractions.NullLoggerFactory.Instance)
            .CreateLogger("ZB.MOM.WW.OtOpcUa.Cluster.ClusterBootstrapFallback");
        builder.AddStartup((system, _) =>
        {
            ClusterBootstrapFallback.Arm(system, options, fallbackLogger);
            return Task.CompletedTask;
        });
```

(Akka.Hosting 1.5.62 has `AddStartup`. If the overload signature differs, alternative: register an `IHostedService` in `AddOtOpcUaCluster` mirroring `ActorSystemTerminationWatchdog`'s lazy-accessor pattern — but prefer `AddStartup` so test hosts arm it for free.)

**Step 3:** `dotnet build src/Core/ZB.MOM.WW.OtOpcUa.Cluster` — 0 warnings.

**Step 4: Commit**

```bash
git add -A && git commit -m "feat(cluster): InitJoin self-form fallback armed via Akka.Hosting startup task"
```

---

### Task 3 (B3): `SelfFormBootstrapTests` — 4 scenarios through the production bootstrap

**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 4

**Files:**
- Create: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SelfFormBootstrapTests.cs`

**Step 1: Write the tests** — follow `SplitBrainResolverActivationTests`' pattern (real `Host.CreateDefaultBuilder()` + `AddAkka` + `WithOtOpcUaClusterBootstrap`; that file's remarks explain why testing through the production bootstrap is mandatory — a test that re-creates the wiring pins the test's wiring, not the shipped one). Helper + scenarios:

```csharp
public sealed class SelfFormBootstrapTests
{
    private static int FreePort()
    {
        var l = new System.Net.Sockets.TcpListener(System.Net.IPAddress.Loopback, 0);
        l.Start();
        var p = ((System.Net.IPEndPoint)l.LocalEndpoint).Port;
        l.Stop();
        return p;
    }

    private static async Task<IHost> StartNodeAsync(
        string systemName, int selfPort, int peerPort, TimeSpan? selfFormAfter, bool selfInSeeds = true)
    {
        var options = new AkkaClusterOptions
        {
            SystemName = systemName, Hostname = "127.0.0.1", PublicHostname = "127.0.0.1",
            Port = selfPort, Roles = new[] { "admin", "driver" },
            SelfFormAfter = selfFormAfter,
            SeedNodes = selfInSeeds
                ? new[] { $"akka.tcp://{systemName}@127.0.0.1:{peerPort}",
                          $"akka.tcp://{systemName}@127.0.0.1:{selfPort}" }   // self SECOND — only the fallback can form it
                : new[] { $"akka.tcp://{systemName}@127.0.0.1:{peerPort}" },
        };
        var builder = Host.CreateDefaultBuilder();
        builder.ConfigureServices(services =>
        {
            services.AddSingleton<IOptions<AkkaClusterOptions>>(Options.Create(options));
            services.AddAkka(systemName, (ab, sp) => ab.WithOtOpcUaClusterBootstrap(sp));
        });
        var host = builder.Build();
        await host.StartAsync();
        return host;
    }

    // 1) SelfFormAfter_defaults_to_ten_seconds
    //      new AkkaClusterOptions().SelfFormAfter.ShouldBe(TimeSpan.FromSeconds(10));
    // 2) Lone_non_first_seed_self_forms_after_the_window
    //      dead peer port, window 2s → poll Cluster.Get(system).State until 1 Up member ≤ 20s.
    // 3) Disabled_fallback_keeps_waiting  (window null → State.Members empty after 6s;
    //      POSITIVE CONTROL: cluster.Join(cluster.SelfAddress) → 1 Up ≤ 20s — proves the node
    //      was formable and only the fallback was absent.)
    // 4) Site_node_seeded_only_by_central_never_self_forms  (selfInSeeds:false, window 1s →
    //      Members empty after 5s; positive control join. THE guard test: this is the exact
    //      docker-dev topology — site nodes list only central-1.)
}
```

Use a unique `systemName` per test (`otopcua-selfform-1` …) so concurrent hosts cannot cross-join; dispose each host in `finally` (`await host.StopAsync(); host.Dispose();`).

**Step 2: Prove the key assertion has teeth** — before relying on scenario 2, temporarily run it with `selfFormAfter: null` and confirm it times out (red), then restore. Record that in the task notes.

**Step 3: Run**

```bash
dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests --filter "FullyQualifiedName~SelfFormBootstrap"
```
Expected: 4 PASS.

**Step 4: Commit**

```bash
git add -A && git commit -m "test(cluster): self-form fallback — lone-seed forms, disabled waits, site-node guard"
```

---

### Task 4 (B4): Docs — Redundancy.md + mesh design note

**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 3

**Files:**
- Modify: `docs/Redundancy.md` (~lines 162, 244–247) — replace the bootstrap-constraint text ("Auto-down removes the crash outage, not this one") with `SelfFormAfter` semantics, the 10 s default, the site-node island guard (why driver-only nodes seeded by central-1 deliberately do NOT self-form under the CURRENT one-mesh topology), and the note that under the future per-cluster mesh each pair node is a seed so the fallback covers both.
- Modify: `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2/§7 — one-line note: the "registered outage gap" is now closed by `SelfFormAfter` (link Redundancy.md).
- Modify: `CLAUDE.md` — brief note in the redundancy-corrections area.

Commit: `docs(redundancy): SelfFormAfter closes the seed-node bootstrap gap`.

---

### Task 5 (B5): Full verification + optional docker-dev live check

**Classification:** standard
**Estimated implement time:** ~5 min (suite runtime dominates)
**Parallelizable with:** none

```bash
cd ~/Desktop/OtOpcUa
dotnet build ZB.MOM.WW.OtOpcUa.slnx     # 0 warnings
dotnet test  ZB.MOM.WW.OtOpcUa.slnx     # green vs the pre-existing baseline
```

Optional live check (docker-dev six-node mesh): `docker compose stop central-1 central-2 && docker compose start central-2` → central-2 (a listed seed) self-forms in ~10 s; site nodes (non-seeds) keep waiting — the island guard on the real topology. Then `docker compose start central-1` → joins central-2's cluster, mesh re-forms. If the rig isn't running, record deferred-live.

---

### Task 6 (D4): Manual failover service + cluster-level test

**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** none

**Files:**
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/IManualFailoverService.cs`
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/ManualFailoverService.cs`
- Modify: the ControlPlane/Host DI registration point (find where `RedundancyStateActor`-adjacent services register; admin-role branch of `Program.cs`)
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/ManualFailoverServiceTests.cs`

**Step 1: Failing tests** — reuse the two-node fixture pattern from `RedundancyPrimaryElectionTests` (node A older on the **higher** port so age ≠ address order; that fixture even has a guard test asserting the divergence — keep it honest here too):

- `FailOver_targets_the_oldest_driver_member_not_the_role_leader`: service picks node A (oldest, higher port) even though B is `RoleLeader("driver")`; A's system terminates (graceful exit); B's next `RedundancyStateActor` snapshot names B Primary.
- `FailOver_refuses_when_no_driver_peer_exists`: single driver member → returns null, no Leave, node still Up (positive assert).
- `Parity_with_SelectDriverPrimary`: on the same fixture, the service's target equals the member `RedundancyStateActor` names Primary — pins the two 5-line queries together.

**Step 2: Implement** — same shape as ScadaBridge's `FailOverCore` but role `"driver"`:

```csharp
public static Address? FailOverCore(ActorSystem system, bool dryRun = false)
{
    var cluster = Akka.Cluster.Cluster.Get(system);
    // Intentionally identical to RedundancyStateActor.SelectDriverPrimary (oldest Up driver,
    // Member.AgeOrdering) — the node acted on must be exactly the one the election names.
    // Pinned by ManualFailoverServiceTests.Parity_with_SelectDriverPrimary.
    var drivers = cluster.State.Members
        .Where(m => m.Status == MemberStatus.Up && m.Roles.Contains("driver"))
        .OrderBy(m => m, Member.AgeOrdering)
        .ToList();
    if (drivers.Count < 2) return null;
    var primary = drivers[0];
    if (!dryRun) cluster.Leave(primary.Address);
    return primary.Address;
}
```

Async wrapper `FailOverDriverPrimaryAsync(string actor)`: dry-run peer guard → audit via the shared `ZB.MOM.WW.Audit.IAuditWriter` seam (action `cluster.manual-failover`, actor, target in `DetailsJson` — copy the call shape from an existing audited AdminUI admin action) → real Leave → warn log → return address string.

**Step 3:** Tests PASS → commit `feat(redundancy): manual failover service — graceful Leave of the driver Primary`.

---

### Task 7 (D5): ClusterRedundancy page — live panel + failover button + bUnit tests

**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none

**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/Clusters/ClusterRedundancy.razor` (page is already `@rendermode RenderMode.InteractiveServer` — no rendermode work; and it lives in the AdminUI host, not an RCL, so the static-router wrapper concern from lmxopcua#483 does not apply)
- Test: the AdminUI bUnit suite (place beside the existing Clusters page tests)
- Modify: `docs/Redundancy.md` — "Manual failover" subsection (button semantics, mesh-wide scope caveat)

**Step 1: Failing bUnit tests:** (a) button hidden under the read-only policy (`AdminUiPolicies.AuthenticatedRead`), visible under the admin/write policy — check `AdminUiPolicies` for the exact policy the mutating Clusters pages use (e.g. `ClusterEdit.razor`) and use the same one; (b) disabled with tooltip when the live snapshot shows <2 Up driver members; (c) confirm flow calls a fake `IManualFailoverService` exactly once.

**Step 2: Implement.** Add a "Live redundancy" panel above the static config table: current driver Primary + Up driver members from a `GetSnapshot()` read-only companion on `IManualFailoverService` (sourced from live `Cluster.State`; do NOT subscribe to the DPS topic for v1), the mesh-scope notice ("in the current single-mesh topology this acts on the whole mesh's Primary"), and the admin-gated **Trigger failover** button with confirm dialog (states: the Primary node restarts, ServiceLevel 250 moves to the survivor, OPC UA clients re-select). On confirm: call the service with the authenticated user name; surface the returned target address.

**Step 3:** Tests PASS → commit `feat(adminui): manual failover control on the cluster redundancy page`.

**Live check (fold into the Task 5 gate when docker-dev is up):** press the button, ServiceLevel 250 moves (old Primary drops off, survivor publishes 250), denial meters stay quiet, audit event recorded.

---

## Reserved for additions (this plan is expected to grow)

Append new tasks below as **Task 8+** (and extend `.tasks.json`) — do not renumber Tasks 1–7. Candidate items already flagged in the family docs, awaiting a decision on which to include:

- **Configurable downing window** — promote `ServiceCollectionExtensions.DowningStableAfter` (hard-coded 15 s) to `AkkaClusterOptions` (e.g. `Cluster:DowningStableAfter`), keeping the pinned invariant window > `acceptable-heartbeat-pause` (`akka_failover.md` §6.2 calls this out; ScadaBridge's equivalent `Cluster:StableAfter` is already config).
- **Failure-detector tuning for faster failover** — expose `heartbeat-interval` / `acceptable-heartbeat-pause` (currently fixed in `Resources/akka.conf`) as options, per the availability-first tuning table in `akka_failover.md` §2.
- ~~Auto-down 1-vs-1 live gate~~ / ~~per-cluster mesh interplay~~ — **now owned by the per-cluster mesh PROGRAM plan** (`docs/plans/2026-07-22-per-cluster-mesh-program.md`, added 2026-07-22): Phases 1–7 align OtOpcUa's mesh with ScadaBridge's shape (one 2-node mesh per application Cluster, co-located on the ScadaBridge VMs). That program lists THIS plan as its prerequisite (execute this one first); its Phase 6 removes the ClusterRedundancy mesh-scope caveat + the site-node island-guard docs note, and Phase 7 closes the auto-down 1-vs-1 and `SelfFormAfter` live gates on the real per-pair topology.
- *(add further items here)*

---

## Completion

- Merge decision via the finishing-a-development-branch flow (family convention: ff-merge to `master` + push to gitea `lmxopcua`, or PR — ask the user).
- After merge: update `scadaproj/akka_failover.md` §6.1 status, `scadaproj/CLAUDE.md` index row, and memory `ha-availability-over-partition-safety` (tracked in the scadaproj index plan).
- Verification-before-completion applies throughout; live gates may be recorded deferred-live if docker-dev is down.