lmxopcua/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md

OtOpcUa: InitJoin Self-Form Fallback + Manual Failover Control — Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.

Shared cross-repo design: ~/Desktop/scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md (design rationale, MNTR assessment, behavior spec). The ScadaBridge half lives in ~/Desktop/ScadaBridge/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md. This plan is self-contained for execution.

⚠ This plan is expected to GROW — see "Reserved for additions" at the end. Append new tasks there (and to the .tasks.json) rather than renumbering existing ones.

Goal: (1) Either node of a 2-node OtOpcUa pair can cold-start alone and become operational, unattended — a configurable, fast (default 10 s) InitJoin timeout falls back to self-forming a cluster, with a hard guard so site nodes seeded only by central can never island themselves. (2) An admin-only "Trigger failover" control on the Cluster Redundancy page performs a graceful, audited swap of the driver Primary.

Architecture: New AkkaClusterOptions.SelfFormAfter (TimeSpan?, default 10 s, null/≤0 disables, bound from the Cluster section) arms ClusterBootstrapFallback via an Akka.Hosting startup task inside WithOtOpcUaClusterBootstrap — so production AND the Cluster.Tests hosts get it identically. Wait for membership via RegisterOnMemberUp; on expiry, Cluster.Join(SelfAddress) — only if this node's own address is in its own SeedNodes. Manual failover = graceful Cluster.Leave(<oldest Up driver member>) via a new IManualFailoverService in ControlPlane, surfaced on /clusters/{id}/redundancy.

Tech Stack: .NET 10, Akka.NET 1.5.62, Akka.Hosting 1.5.62 (AddStartup), Blazor Server (AdminUI, InteractiveServer), bUnit, xunit. No new packages.

Branch: feat/selfform-fallback off master.

---

Design essentials (from the shared design doc)

The defect: Akka only lets the FIRST listed seed self-join to form a new cluster; every other node loops on InitJoin forever — "auto-down removes the crash outage, not this one" (docs/Redundancy.md:244-247). A lone cold-starting central-2 (or a pair node under the future per-cluster mesh) never comes Up.

Behavior spec:

Scenario	Behavior with fallback
Peer alive (any boot order)	Normal seed join in ms — fallback never fires
Lone cold-start, self IS in own SeedNodes	After SelfFormAfter: warn log + Cluster.Join(SelfAddress) → Up alone (min-nr-of-members=1)
Lone cold-start, self NOT in own SeedNodes	Fallback inert (info log). This is the docker-dev site-node topology — site-a/b nodes list only central-1; self-forming there would island them permanently, so they must wait.
Peer boots after survivor self-formed	Its InitJoin is answered → joins as youngest. No island.
Both pair nodes cold-start simultaneously, mutually unreachable	Both self-form → dual-active (same partition class auto-down accepts; restart one side)
SelfFormAfter null/≤0	Disabled — today's wait-forever behavior
Window expires mid-join-handshake	Benign: Akka ignores Join once joined

Manual failover rules: graceful Leave of the oldest Up driver member (identical query to RedundancyStateActor.SelectDriverPrimary — Member.AgeOrdering, Leaving excluded via Status == Up), never Down. The leaving node hands over via the cluster-leave phases, CoordinatedShutdown runs, ActorSystemTerminationWatchdog exits the process, the supervisor restarts it, and it rejoins as youngest → the survivor is Primary (ServiceLevel 250 moves; OPC UA clients re-select). Admin-only; peer guard (<2 Up driver members → disabled); confirmation dialog; audited via the shared ZB.MOM.WW.Audit.IAuditWriter seam before the Leave. Mesh-scope caveat: until the per-cluster mesh lands, the election is mesh-wide — the button acts on THE Primary of the whole Akka cluster, not per-ServerCluster row; the UI must say so (mirrors the KNOWN LIMITATION in docs/Redundancy.md:84-99).

Multi-node TestKit: assessed, NOT used — everything is deterministic in-process via real hosts through WithOtOpcUaClusterBootstrap (the SplitBrainResolverActivationTests pattern), and the Akka TestKit family is xunit-v2-only while this repo's integration tree is xunit.v3. Full verdict in the shared design doc.

---

Task 1 (B1): SelfFormAfter on AkkaClusterOptions

Classification: small Estimated implement time: ~3 min Parallelizable with: none (first task)

Files:

• Modify: src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptions.cs

Step 1: Branch

cd ~/Desktop/OtOpcUa && git checkout master && git checkout -b feat/selfform-fallback

Step 2: Add the property (after SplitBrainResolverStrategy, mirroring its XML-doc depth):

    /// <summary>
    /// Bootstrap self-form fallback window (decision 2026-07-22, scadaproj/akka_failover.md §6.1).
    /// Akka only lets the FIRST listed seed form a new cluster; a non-first seed cold-starting
    /// while its peer is down loops on InitJoin forever ("auto-down removes the crash outage,
    /// not this one" — docs/Redundancy.md). When this node has waited longer than this window
    /// without cluster membership it joins itself — but ONLY if its own address appears in
    /// <see cref="SeedNodes"/>; a non-seed node (e.g. a site node whose only seed is central-1)
    /// stays waiting, because self-forming there creates a permanent island. Default 10s
    /// (same-datacenter pair; a live peer answers InitJoin in milliseconds). <c>null</c> or a
    /// non-positive value disables the fallback. Accepted trade: both pair nodes cold-starting
    /// inside the window while mutually unreachable form two clusters — the same dual-active
    /// class the auto-down strategy already accepts, same recovery (restart one side).
    /// </summary>
    public TimeSpan? SelfFormAfter { get; set; } = TimeSpan.FromSeconds(10);

Step 3: dotnet build src/Core/ZB.MOM.WW.OtOpcUa.Cluster — 0 warnings. Commit:

git add -A && git commit -m "feat(cluster): SelfFormAfter option (bootstrap self-form fallback window)"

(The default-value pin test lands with Task 3's test file — the property is inert until Task 2.)

---

Task 2 (B2): ClusterBootstrapFallback + AddStartup wiring

Classification: high-risk (cluster formation behavior) Estimated implement time: ~5 min Parallelizable with: none

Files:

• Create: src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ClusterBootstrapFallback.cs
• Modify: src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs (WithOtOpcUaClusterBootstrap, after the AddHocon(BuildDowningHocon(options), HoconAddMode.Prepend) call at line 93)

Step 1: Implement the fallback class (deliberately duplicated from ScadaBridge's, like the termination watchdogs — see that repo's ClusterBootstrapFallback.cs for the full doc comment to adapt):

using Akka.Actor;
using Microsoft.Extensions.Logging;

namespace ZB.MOM.WW.OtOpcUa.Cluster;

/// <summary>
/// InitJoin self-form fallback (decision 2026-07-22, scadaproj/akka_failover.md §6.1). Akka only
/// lets the FIRST listed seed form a NEW cluster; every other node retries InitJoin forever —
/// "auto-down removes the crash outage, not this one" (docs/Redundancy.md). Waits
/// <see cref="AkkaClusterOptions.SelfFormAfter"/> for membership, then joins itself.
/// <para><b>Island safety:</b> fires ONLY when this node's own address is in its own
/// <see cref="AkkaClusterOptions.SeedNodes"/> — a site node seeded only by central-1 must wait,
/// not island. Sequential recovery is island-free: Akka's join protocol prefers an existing
/// cluster (a booting node only self-forms when NO seed answers InitJoin).</para>
/// <para><b>Benign race:</b> Akka ignores <c>Join</c> once the node has joined a cluster this
/// incarnation. Residual risk: both pair nodes cold-starting inside the window while mutually
/// unreachable self-form separately — the same dual-active class auto-down accepts.</para>
/// </summary>
public static class ClusterBootstrapFallback
{
    public static void Arm(ActorSystem system, AkkaClusterOptions options, ILogger logger)
    {
        if (options.SelfFormAfter is not { } window || window <= TimeSpan.Zero)
        {
            logger.LogInformation(
                "Cluster self-form fallback disabled (SelfFormAfter not set) — a node cold-starting "
                + "while its peer is down will wait on InitJoin indefinitely.");
            return;
        }

        var cluster = Akka.Cluster.Cluster.Get(system);
        var self = cluster.SelfAddress;
        var isSeed = options.SeedNodes.Any(s => TryParseAddress(s, out var a) && a.Equals(self));
        if (!isSeed)
        {
            logger.LogInformation(
                "Cluster self-form fallback inactive: this node ({Self}) is not in its own seed list "
                + "[{Seeds}] — self-forming here would island it from the real seeds.",
                self, string.Join(", ", options.SeedNodes));
            return;
        }

        var joined = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
        cluster.RegisterOnMemberUp(() => joined.TrySetResult());

        _ = Task.Run(async () =>
        {
            var winner = await Task.WhenAny(joined.Task, Task.Delay(window));
            if (winner == joined.Task || system.WhenTerminated.IsCompleted)
                return;
            logger.LogWarning(
                "No cluster membership after {Window} — no seed answered InitJoin (peer down at boot). "
                + "Self-forming a cluster at {Self} so this node becomes operational; if the peer was "
                + "merely partitioned (not dead), the pair is now dual-active — restart one side after "
                + "the partition heals (accepted availability-first trade, decision 2026-07-22).",
                window, self);
            cluster.Join(self);
        });
    }

    private static bool TryParseAddress(string seed, out Address address)
    {
        try { address = Address.Parse(seed); return true; }
        catch { address = default!; return false; }
    }
}

(If the Cluster project lacks a Microsoft.Extensions.Logging.Abstractions reference, add the PackageReference — check Directory.Packages.props for the pinned version first.)

Step 2: Wire via Akka.Hosting startup hook — in WithOtOpcUaClusterBootstrap, after the downing HOCON line (~93):

        // InitJoin self-form fallback (decision 2026-07-22): registered as an Akka.Hosting
        // startup task so every host built through this bootstrap — production AND the test
        // hosts in Cluster.Tests — arms it identically. Guarded inside Arm: disabled when
        // SelfFormAfter is unset; inert when this node is not in its own seed list (site nodes
        // seeded only by central-1 must wait, not island).
        var fallbackLogger = (serviceProvider.GetService<ILoggerFactory>()
                              ?? Microsoft.Extensions.Logging.Abstractions.NullLoggerFactory.Instance)
            .CreateLogger("ZB.MOM.WW.OtOpcUa.Cluster.ClusterBootstrapFallback");
        builder.AddStartup((system, _) =>
        {
            ClusterBootstrapFallback.Arm(system, options, fallbackLogger);
            return Task.CompletedTask;
        });

(Akka.Hosting 1.5.62 has AddStartup. If the overload signature differs, alternative: register an IHostedService in AddOtOpcUaCluster mirroring ActorSystemTerminationWatchdog's lazy-accessor pattern — but prefer AddStartup so test hosts arm it for free.)

Step 3: dotnet build src/Core/ZB.MOM.WW.OtOpcUa.Cluster — 0 warnings.

Step 4: Commit

git add -A && git commit -m "feat(cluster): InitJoin self-form fallback armed via Akka.Hosting startup task"

---

Task 3 (B3): SelfFormBootstrapTests — 4 scenarios through the production bootstrap

Classification: standard Estimated implement time: ~5 min Parallelizable with: Task 4

Files:

• Create: tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SelfFormBootstrapTests.cs

Step 1: Write the tests — follow SplitBrainResolverActivationTests' pattern (real Host.CreateDefaultBuilder() + AddAkka + WithOtOpcUaClusterBootstrap; that file's remarks explain why testing through the production bootstrap is mandatory — a test that re-creates the wiring pins the test's wiring, not the shipped one). Helper + scenarios:

public sealed class SelfFormBootstrapTests
{
    private static int FreePort()
    {
        var l = new System.Net.Sockets.TcpListener(System.Net.IPAddress.Loopback, 0);
        l.Start();
        var p = ((System.Net.IPEndPoint)l.LocalEndpoint).Port;
        l.Stop();
        return p;
    }

    private static async Task<IHost> StartNodeAsync(
        string systemName, int selfPort, int peerPort, TimeSpan? selfFormAfter, bool selfInSeeds = true)
    {
        var options = new AkkaClusterOptions
        {
            SystemName = systemName, Hostname = "127.0.0.1", PublicHostname = "127.0.0.1",
            Port = selfPort, Roles = new[] { "admin", "driver" },
            SelfFormAfter = selfFormAfter,
            SeedNodes = selfInSeeds
                ? new[] { $"akka.tcp://{systemName}@127.0.0.1:{peerPort}",
                          $"akka.tcp://{systemName}@127.0.0.1:{selfPort}" }   // self SECOND — only the fallback can form it
                : new[] { $"akka.tcp://{systemName}@127.0.0.1:{peerPort}" },
        };
        var builder = Host.CreateDefaultBuilder();
        builder.ConfigureServices(services =>
        {
            services.AddSingleton<IOptions<AkkaClusterOptions>>(Options.Create(options));
            services.AddAkka(systemName, (ab, sp) => ab.WithOtOpcUaClusterBootstrap(sp));
        });
        var host = builder.Build();
        await host.StartAsync();
        return host;
    }

    // 1) SelfFormAfter_defaults_to_ten_seconds
    //      new AkkaClusterOptions().SelfFormAfter.ShouldBe(TimeSpan.FromSeconds(10));
    // 2) Lone_non_first_seed_self_forms_after_the_window
    //      dead peer port, window 2s → poll Cluster.Get(system).State until 1 Up member ≤ 20s.
    // 3) Disabled_fallback_keeps_waiting  (window null → State.Members empty after 6s;
    //      POSITIVE CONTROL: cluster.Join(cluster.SelfAddress) → 1 Up ≤ 20s — proves the node
    //      was formable and only the fallback was absent.)
    // 4) Site_node_seeded_only_by_central_never_self_forms  (selfInSeeds:false, window 1s →
    //      Members empty after 5s; positive control join. THE guard test: this is the exact
    //      docker-dev topology — site nodes list only central-1.)
}

Use a unique systemName per test (otopcua-selfform-1 …) so concurrent hosts cannot cross-join; dispose each host in finally (await host.StopAsync(); host.Dispose();).

Step 2: Prove the key assertion has teeth — before relying on scenario 2, temporarily run it with selfFormAfter: null and confirm it times out (red), then restore. Record that in the task notes.

Step 3: Run

dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests --filter "FullyQualifiedName~SelfFormBootstrap"

Expected: 4 PASS.

Step 4: Commit

git add -A && git commit -m "test(cluster): self-form fallback — lone-seed forms, disabled waits, site-node guard"

---

Task 4 (B4): Docs — Redundancy.md + mesh design note

Classification: small Estimated implement time: ~4 min Parallelizable with: Task 3

Files:

• Modify: docs/Redundancy.md (~lines 162, 244–247) — replace the bootstrap-constraint text ("Auto-down removes the crash outage, not this one") with SelfFormAfter semantics, the 10 s default, the site-node island guard (why driver-only nodes seeded by central-1 deliberately do NOT self-form under the CURRENT one-mesh topology), and the note that under the future per-cluster mesh each pair node is a seed so the fallback covers both.
• Modify: docs/plans/2026-07-21-per-cluster-mesh-design.md §6.2/§7 — one-line note: the "registered outage gap" is now closed by SelfFormAfter (link Redundancy.md).
• Modify: CLAUDE.md — brief note in the redundancy-corrections area.

Commit: docs(redundancy): SelfFormAfter closes the seed-node bootstrap gap.

---

Task 5 (B5): Full verification + optional docker-dev live check

Classification: standard Estimated implement time: ~5 min (suite runtime dominates) Parallelizable with: none

cd ~/Desktop/OtOpcUa
dotnet build ZB.MOM.WW.OtOpcUa.slnx     # 0 warnings
dotnet test  ZB.MOM.WW.OtOpcUa.slnx     # green vs the pre-existing baseline

Optional live check (docker-dev six-node mesh): docker compose stop central-1 central-2 && docker compose start central-2 → central-2 (a listed seed) self-forms in ~10 s; site nodes (non-seeds) keep waiting — the island guard on the real topology. Then docker compose start central-1 → joins central-2's cluster, mesh re-forms. If the rig isn't running, record deferred-live.

---

Task 6 (D4): Manual failover service + cluster-level test

Classification: high-risk Estimated implement time: ~5 min Parallelizable with: none

Files:

• Create: src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/IManualFailoverService.cs
• Create: src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/ManualFailoverService.cs
• Modify: the ControlPlane/Host DI registration point (find where RedundancyStateActor-adjacent services register; admin-role branch of Program.cs)
• Test: tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/ManualFailoverServiceTests.cs

Step 1: Failing tests — reuse the two-node fixture pattern from RedundancyPrimaryElectionTests (node A older on the higher port so age ≠ address order; that fixture even has a guard test asserting the divergence — keep it honest here too):

• FailOver_targets_the_oldest_driver_member_not_the_role_leader: service picks node A (oldest, higher port) even though B is RoleLeader("driver"); A's system terminates (graceful exit); B's next RedundancyStateActor snapshot names B Primary.
• FailOver_refuses_when_no_driver_peer_exists: single driver member → returns null, no Leave, node still Up (positive assert).
• Parity_with_SelectDriverPrimary: on the same fixture, the service's target equals the member RedundancyStateActor names Primary — pins the two 5-line queries together.

Step 2: Implement — same shape as ScadaBridge's FailOverCore but role "driver":

public static Address? FailOverCore(ActorSystem system, bool dryRun = false)
{
    var cluster = Akka.Cluster.Cluster.Get(system);
    // Intentionally identical to RedundancyStateActor.SelectDriverPrimary (oldest Up driver,
    // Member.AgeOrdering) — the node acted on must be exactly the one the election names.
    // Pinned by ManualFailoverServiceTests.Parity_with_SelectDriverPrimary.
    var drivers = cluster.State.Members
        .Where(m => m.Status == MemberStatus.Up && m.Roles.Contains("driver"))
        .OrderBy(m => m, Member.AgeOrdering)
        .ToList();
    if (drivers.Count < 2) return null;
    var primary = drivers[0];
    if (!dryRun) cluster.Leave(primary.Address);
    return primary.Address;
}

Async wrapper FailOverDriverPrimaryAsync(string actor): dry-run peer guard → audit via the shared ZB.MOM.WW.Audit.IAuditWriter seam (action cluster.manual-failover, actor, target in DetailsJson — copy the call shape from an existing audited AdminUI admin action) → real Leave → warn log → return address string.

Step 3: Tests PASS → commit feat(redundancy): manual failover service — graceful Leave of the driver Primary.

---

Task 7 (D5): ClusterRedundancy page — live panel + failover button + bUnit tests

Classification: standard Estimated implement time: ~5 min Parallelizable with: none

Files:

• Modify: src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/Clusters/ClusterRedundancy.razor (page is already @rendermode RenderMode.InteractiveServer — no rendermode work; and it lives in the AdminUI host, not an RCL, so the static-router wrapper concern from lmxopcua#483 does not apply)
• Test: the AdminUI bUnit suite (place beside the existing Clusters page tests)
• Modify: docs/Redundancy.md — "Manual failover" subsection (button semantics, mesh-wide scope caveat)

Step 1: Failing bUnit tests: (a) button hidden under the read-only policy (AdminUiPolicies.AuthenticatedRead), visible under the admin/write policy — check AdminUiPolicies for the exact policy the mutating Clusters pages use (e.g. ClusterEdit.razor) and use the same one; (b) disabled with tooltip when the live snapshot shows <2 Up driver members; (c) confirm flow calls a fake IManualFailoverService exactly once.

Step 2: Implement. Add a "Live redundancy" panel above the static config table: current driver Primary + Up driver members from a GetSnapshot() read-only companion on IManualFailoverService (sourced from live Cluster.State; do NOT subscribe to the DPS topic for v1), the mesh-scope notice ("in the current single-mesh topology this acts on the whole mesh's Primary"), and the admin-gated Trigger failover button with confirm dialog (states: the Primary node restarts, ServiceLevel 250 moves to the survivor, OPC UA clients re-select). On confirm: call the service with the authenticated user name; surface the returned target address.

Step 3: Tests PASS → commit feat(adminui): manual failover control on the cluster redundancy page.

Live check (fold into the Task 5 gate when docker-dev is up): press the button, ServiceLevel 250 moves (old Primary drops off, survivor publishes 250), denial meters stay quiet, audit event recorded.

---

Reserved for additions (this plan is expected to grow)

Append new tasks below as Task 8+ (and extend .tasks.json) — do not renumber Tasks 1–7. Candidate items already flagged in the family docs, awaiting a decision on which to include:

• Configurable downing window — promote ServiceCollectionExtensions.DowningStableAfter (hard-coded 15 s) to AkkaClusterOptions (e.g. Cluster:DowningStableAfter), keeping the pinned invariant window > acceptable-heartbeat-pause (akka_failover.md §6.2 calls this out; ScadaBridge's equivalent Cluster:StableAfter is already config).
• Failure-detector tuning for faster failover — expose heartbeat-interval / acceptable-heartbeat-pause (currently fixed in Resources/akka.conf) as options, per the availability-first tuning table in akka_failover.md §2.
• Auto-down 1-vs-1 live gate / per-cluster mesh interplay — now owned by the per-cluster mesh PROGRAM plan (docs/plans/2026-07-22-per-cluster-mesh-program.md, added 2026-07-22): Phases 1–7 align OtOpcUa's mesh with ScadaBridge's shape (one 2-node mesh per application Cluster, co-located on the ScadaBridge VMs). That program lists THIS plan as its prerequisite (execute this one first); its Phase 6 removes the ClusterRedundancy mesh-scope caveat + the site-node island-guard docs note, and Phase 7 closes the auto-down 1-vs-1 and SelfFormAfter live gates on the real per-pair topology.
• (add further items here)

---

Completion

• Merge decision via the finishing-a-development-branch flow (family convention: ff-merge to master + push to gitea lmxopcua, or PR — ask the user).
• After merge: update scadaproj/akka_failover.md §6.1 status, scadaproj/CLAUDE.md index row, and memory ha-availability-over-partition-safety (tracked in the scadaproj index plan).
• Verification-before-completion applies throughout; live gates may be recorded deferred-live if docker-dev is down.