docs(plans): record the self-form watchdog rejection and the self-first ordering that replaced it
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,661 @@
|
||||
# ScadaBridge: InitJoin Self-Form Fallback + Manual Failover Control — Implementation Plan
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
|
||||
>
|
||||
> Shared cross-repo design: `~/Desktop/scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md` (design rationale, MNTR assessment, behavior spec). The OtOpcUa half lives in `~/Desktop/OtOpcUa/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`. This plan is self-contained for execution.
|
||||
|
||||
**Goal:** (1) Either node of a 2-node ScadaBridge cluster can cold-start alone and become operational, unattended. (2) An admin-only "Trigger failover" control on the Health page performs a graceful, audited role swap of the central pair.
|
||||
|
||||
> ## ⚠️ ARCHITECTURE REVISED DURING EXECUTION (2026-07-22)
|
||||
>
|
||||
> **Part 1 shipped as self-first seed ordering, NOT the `SelfFormAfter` watchdog described below.** Tasks 1–7 as originally written are superseded; what was actually built is in "Part 1 as executed". Tasks 8–10 (manual failover) are unaffected and still apply as written.
|
||||
>
|
||||
> **Why.** A code review of Task 2 raised, and a written test then confirmed, that the watchdog's success signal ("am I `Up` within the window?") cannot distinguish *no seed answered InitJoin* from *a seed answered and the join is in flight* — it sits outside Akka's join handshake. On a routine standby restart the peer is alive but the join stalls behind removal of the restarting node's own stale incarnation; a `Join(self)` issued during `TryingToJoin` abandons the in-flight join and forms a second cluster at the same address. **Measured: a permanent split, still unhealed after 90 s** — converting a routine restart into an outage of the previously-healthy node. That is strictly worse than the gap being closed, and it is not the boot-partition trade the design accepted.
|
||||
>
|
||||
> Akka's own `FirstSeedNodeProcess` already implements exactly the intended semantics — InitJoin the other seeds, self-join only if nobody answers — and, being part of the handshake, has no such race. It runs only when `seed-nodes[0]` is the node's own address. So the fix is seed **ordering**, not new runtime code.
|
||||
>
|
||||
> The plan's stated safety property — "a booting node only self-forms when NO seed answers InitJoin" — is true of Akka's native first-seed rule and **false** of the watchdog. That claim also appears in the shared cross-repo design doc (`scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md`) and in the OtOpcUa half; **both still need correcting** (owner deferred, 2026-07-22).
|
||||
|
||||
## Part 1 as executed — self-first seed ordering
|
||||
|
||||
**Architecture:** Every node lists ITSELF as `seed-nodes[0]` and its partner second. No new runtime code, no timer, no new option. `StartupValidator` enforces the ordering at boot (host **and** port comparison) because a broken ordering fails silently. Manual failover (Part 2) is unchanged: graceful `Cluster.Leave(oldest Up member)` via a new `IManualFailoverService` (CentralUI seam, Host implementation) — singleton drain, watchdog process-exit, supervisor restart, rejoin as youngest.
|
||||
|
||||
**Behavior** (all rows covered by `SelfFirstSeedBootstrapTests` — real in-process clusters from production `BuildHocon` at production failure-detection timings):
|
||||
|
||||
| Scenario | Behavior |
|
||||
|---|---|
|
||||
| Lone cold-start, peer dead | Self-joins after `seed-node-timeout` (~5 s) — operational, unattended |
|
||||
| Restart into a **live** peer | Peer answers `InitJoinAck`; node rejoins, never islands |
|
||||
| Both cold-start simultaneously, mutually reachable | `InitJoin` handshake converges them → **one** 2-member cluster |
|
||||
| Both cold-start during a genuine boot **partition** | Each forms its own cluster — same dual-active class `auto-down` already accepts |
|
||||
| Peer-first ordering (the old config) | Never forms — retained as a falsifiability control in the test suite |
|
||||
|
||||
**Shipped:**
|
||||
- 6 node appsettings reordered (the `*-node-b` configs; `-a` nodes were already self-first). All 14 satisfy the invariant.
|
||||
- `StartupValidator` self-first rule + 3 tests; `SelfFirstSeedBootstrapTests` (4 tests).
|
||||
- Docs corrected: `docker/README.md`, `docs/requirements/Component-ClusterInfrastructure.md` (new **Seed Node Ordering** section), `docs/components/ClusterInfrastructure.md` (3 passages), `docs/deployment/topology-guide.md` (incl. the stale keep-oldest claim Task 6 flagged), `CLAUDE.md`.
|
||||
- Several docs had asserted self-first ordering was *unsafe* because a simultaneous cold start would produce two clusters that never merge. Disproved by test (row 3) and corrected.
|
||||
|
||||
**⚠️ Ops action:** the gitignored `deploy/wonder-app-vd03/` overlay must have its `SeedNodes` reordered self-first before its next deploy, or the node will now **refuse to boot**. The validator rule is a hard gate deliberately — the alternative is the silent wedge it replaces.
|
||||
|
||||
---
|
||||
|
||||
<details>
|
||||
<summary>Original Part 1 architecture (SUPERSEDED — kept for the decision record)</summary>
|
||||
|
||||
**Architecture:** New `ClusterOptions.SelfFormAfter` (`TimeSpan?`, default 10 s, `null`/`≤0` disables, appsettings-bound) arms `ClusterBootstrapFallback` right after ActorSystem creation: wait for membership via `RegisterOnMemberUp`; on expiry, `Cluster.Join(SelfAddress)`. Safety gate: fires only when this node's own address is in its own seed list.
|
||||
|
||||
</details>
|
||||
|
||||
**Tech Stack:** .NET 10, Akka.NET 1.5.62, Blazor Server (CentralUI), bUnit, xunit. No new packages.
|
||||
|
||||
**Branch:** `feat/selfform-fallback` off `main`.
|
||||
|
||||
---
|
||||
|
||||
## Design essentials (from the shared design doc)
|
||||
|
||||
**The defect:** Akka only lets the FIRST listed seed self-join to form a *new* cluster; every other node loops on `InitJoin` forever. Both docker central nodes list `central-a` first, so a lone cold-starting `central-b` never comes Up (the "registered outage gap" — `docker/README.md:289`). `ClusterOptions.SeedNodes`' doc comment claims "either can start first", which the deployed configs do not deliver — this plan makes it true and fixes the comment.
|
||||
|
||||
**Behavior spec:**
|
||||
|
||||
| Scenario | Behavior with fallback |
|
||||
|---|---|
|
||||
| Peer alive (any boot order) | Normal seed join in ms — fallback never fires |
|
||||
| Lone cold-start, self IS in own seed list | After `SelfFormAfter`: warn log + `Cluster.Join(SelfAddress)` → Up alone, singletons start (`min-nr-of-members=1`) |
|
||||
| Lone cold-start, self NOT in own seed list | Fallback inert (info log) — self-forming would island the node from the real seeds |
|
||||
| Peer boots after survivor self-formed | Peer's InitJoin is answered → joins as youngest/standby. No island. |
|
||||
| Both cold-start simultaneously, mutually unreachable | Both self-form → dual-active (same partition class auto-down accepts; restart one side) |
|
||||
| `SelfFormAfter` null/`≤0` | Disabled — today's wait-forever behavior |
|
||||
| Window expires mid-join-handshake | Benign: Akka ignores `Join` once joined |
|
||||
|
||||
**Manual failover rules:** graceful `Leave`, never `Down`; admin-only (`AuthorizationPolicies.RequireAdmin`); peer guard (disabled when <2 Up `Central` members); confirmation dialog warning the Blazor circuit will drop (Traefik routes the UI to the active node — triggering failover disconnects your own page, which reconnects against the new active); audited via the app's **central** audit writer (`ICentralAuditWriter` — NOT the shared seam, see the dual-seam gotcha) before the Leave is issued. No interplay with `SelfFormAfter` (the peer is alive on this path, so the restarted node rejoins normally).
|
||||
|
||||
**Multi-node TestKit:** assessed and NOT used — in-process real clusters via `TwoNodeClusterFixture` (production `BuildHocon`) cover every deterministic scenario; MNTR would need a dedicated no-parallelization test project for no added coverage. See the shared design doc for the full verdict.
|
||||
|
||||
---
|
||||
|
||||
### Task 1 (A1): `SelfFormAfter` option + validator
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** none (first task)
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptionsValidator.cs`
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsTests.cs`
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsValidatorTests.cs`
|
||||
|
||||
**Step 1: Create the branch**
|
||||
|
||||
```bash
|
||||
cd ~/Desktop/ScadaBridge && git checkout main && git checkout -b feat/selfform-fallback
|
||||
```
|
||||
|
||||
**Step 2: Write the failing tests** (append to the existing test classes, matching their assertion style — read them first)
|
||||
|
||||
```csharp
|
||||
// ClusterOptionsTests.cs
|
||||
[Fact]
|
||||
public void SelfFormAfter_defaults_to_ten_seconds()
|
||||
{
|
||||
new ClusterOptions().SelfFormAfter.ShouldBe(TimeSpan.FromSeconds(10));
|
||||
}
|
||||
|
||||
// ClusterOptionsValidatorTests.cs
|
||||
[Fact] public void SelfFormAfter_null_passes_validation() { /* valid options + null → Succeeded */ }
|
||||
[Fact] public void SelfFormAfter_zero_passes_validation() { /* zero = explicit disable → Succeeded */ }
|
||||
[Fact] public void SelfFormAfter_negative_fails_validation() { /* -1s → Failed, message mentions SelfFormAfter */ }
|
||||
```
|
||||
|
||||
**Step 3: Run to verify failure**
|
||||
|
||||
```bash
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests --filter "FullyQualifiedName~SelfFormAfter"
|
||||
```
|
||||
Expected: FAIL (compile error — property doesn't exist).
|
||||
|
||||
**Step 4: Implement.** `ClusterOptions.cs` — add after `AllowSingleNodeCluster` (line 104):
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Bootstrap self-form fallback window (decision 2026-07-22, scadaproj/akka_failover.md §6.1).
|
||||
/// Akka only lets the FIRST listed seed form a new cluster; a non-first seed cold-starting
|
||||
/// while its peer is down loops on InitJoin forever. When this node has waited longer than
|
||||
/// this window without becoming a cluster member, it forms a cluster on itself
|
||||
/// (<c>Cluster.Join(SelfAddress)</c>) — but ONLY if its own address is in its seed list
|
||||
/// (a non-seed node self-forming would create a permanent island). Default 10s: the pair
|
||||
/// shares a datacenter, so a live peer answers InitJoin in milliseconds and waiting longer
|
||||
/// buys nothing. <c>null</c> or a non-positive value disables the fallback (wait-forever).
|
||||
/// Accepted trade: both nodes cold-starting within the window while mutually unreachable
|
||||
/// form two clusters — the same partition class the auto-down strategy already accepts.
|
||||
/// </summary>
|
||||
public TimeSpan? SelfFormAfter { get; set; } = TimeSpan.FromSeconds(10);
|
||||
```
|
||||
|
||||
`ClusterOptionsValidator.cs` — inside `Validate`, after the `FailureDetectionThreshold` rules:
|
||||
|
||||
```csharp
|
||||
builder.RequireThat(options.SelfFormAfter is null || options.SelfFormAfter.Value >= TimeSpan.Zero,
|
||||
"ClusterOptions.SelfFormAfter must be null (disabled), zero (disabled) or a positive duration; "
|
||||
+ "a negative value is always a configuration mistake.");
|
||||
```
|
||||
|
||||
**Step 5: Run tests → PASS**
|
||||
|
||||
```bash
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests
|
||||
```
|
||||
|
||||
**Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add -A && git commit -m "feat(cluster): SelfFormAfter option — bootstrap self-form fallback window"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2 (A2): `ClusterBootstrapFallback` + first integration test
|
||||
|
||||
**Classification:** high-risk (cluster formation behavior)
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** none
|
||||
|
||||
**Files:**
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/ClusterBootstrapFallback.cs`
|
||||
- Create: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SelfFormBootstrapTests.cs`
|
||||
|
||||
**Step 1: Write the failing test**
|
||||
|
||||
```csharp
|
||||
using Akka.Actor;
|
||||
using Akka.Cluster;
|
||||
using Akka.Configuration;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
|
||||
using ZB.MOM.WW.ScadaBridge.Host;
|
||||
using ZB.MOM.WW.ScadaBridge.Host.Actors;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;
|
||||
|
||||
/// <summary>
|
||||
/// Guards the InitJoin self-form fallback (decision 2026-07-22): Akka only lets the FIRST
|
||||
/// listed seed form a new cluster, so without the fallback a non-first seed cold-starting
|
||||
/// alone waits on InitJoin forever — the "registered outage gap". These tests build REAL
|
||||
/// single/dual-node clusters from the production BuildHocon output, exactly like
|
||||
/// TwoNodeClusterFixture, and arm the production fallback.
|
||||
/// </summary>
|
||||
public sealed class SelfFormBootstrapTests : IAsyncLifetime
|
||||
{
|
||||
private readonly List<ActorSystem> _systems = new();
|
||||
|
||||
/// <summary>Starts a node whose seed list puts the PEER first (self second, or absent),
|
||||
/// so Akka's own first-seed rule can never self-form it — only the fallback can.</summary>
|
||||
private ActorSystem StartNode(int selfPort, int peerPort, TimeSpan? selfFormAfter, bool selfInSeeds = true)
|
||||
{
|
||||
var nodeOptions = new NodeOptions { Role = "Central", NodeHostname = "127.0.0.1", RemotingPort = selfPort };
|
||||
var clusterOptions = new ClusterOptions
|
||||
{
|
||||
SeedNodes = selfInSeeds
|
||||
? new List<string>
|
||||
{
|
||||
$"akka.tcp://scadabridge@127.0.0.1:{peerPort}",
|
||||
$"akka.tcp://scadabridge@127.0.0.1:{selfPort}",
|
||||
}
|
||||
: new List<string> { $"akka.tcp://scadabridge@127.0.0.1:{peerPort}" },
|
||||
SelfFormAfter = selfFormAfter,
|
||||
StableAfter = TimeSpan.FromSeconds(3),
|
||||
HeartbeatInterval = TimeSpan.FromMilliseconds(500),
|
||||
FailureDetectionThreshold = TimeSpan.FromSeconds(2),
|
||||
MinNrOfMembers = 1,
|
||||
AllowSingleNodeCluster = !selfInSeeds,
|
||||
};
|
||||
var hocon = AkkaHostedService.BuildHocon(
|
||||
nodeOptions, clusterOptions, new[] { "Central" },
|
||||
TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(3));
|
||||
var system = ActorSystem.Create("scadabridge", ConfigurationFactory.ParseString(hocon));
|
||||
_systems.Add(system);
|
||||
ClusterBootstrapFallback.Arm(system, clusterOptions, NullLogger.Instance);
|
||||
return system;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Lone_non_first_seed_self_forms_after_the_window()
|
||||
{
|
||||
var selfPort = TwoNodeClusterFixture.GetFreeTcpPort();
|
||||
var deadPeerPort = TwoNodeClusterFixture.GetFreeTcpPort(); // nothing listening
|
||||
var node = StartNode(selfPort, deadPeerPort, selfFormAfter: TimeSpan.FromSeconds(2));
|
||||
|
||||
// Without the fallback this waits forever (Akka first-seed rule). With it, the node
|
||||
// must be a 1-member Up cluster shortly after the 2s window.
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(node, 1, TimeSpan.FromSeconds(20));
|
||||
Cluster.Get(node).SelfMember.Status.ShouldBe(MemberStatus.Up);
|
||||
}
|
||||
|
||||
public Task InitializeAsync() => Task.CompletedTask;
|
||||
|
||||
public async Task DisposeAsync()
|
||||
{
|
||||
foreach (var s in _systems)
|
||||
{
|
||||
try { await s.Terminate().WaitAsync(TimeSpan.FromSeconds(10)); } catch { /* teardown */ }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
(Match the assertion library to `SbrFailoverTests.cs` — Shouldly vs xunit `Assert`.)
|
||||
|
||||
**Step 2: Run to verify failure** — compile error (`ClusterBootstrapFallback` doesn't exist):
|
||||
|
||||
```bash
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.IntegrationTests --filter "FullyQualifiedName~SelfFormBootstrapTests"
|
||||
```
|
||||
|
||||
**Step 3: Implement `ClusterBootstrapFallback.cs`**
|
||||
|
||||
```csharp
|
||||
using Akka.Actor;
|
||||
using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Host.Actors;
|
||||
|
||||
/// <summary>
|
||||
/// InitJoin self-form fallback (decision 2026-07-22, scadaproj/akka_failover.md §6.1).
|
||||
/// Akka only lets the FIRST listed seed form a NEW cluster; every other node retries InitJoin
|
||||
/// forever. So "both nodes are seed nodes" (ClusterOptions.SeedNodes) does NOT mean either can
|
||||
/// cold-start alone — a non-first seed booting while its peer is down waits indefinitely (the
|
||||
/// "registered outage gap", docker/README.md). This watchdog waits <see cref="ClusterOptions.SelfFormAfter"/>
|
||||
/// for membership; on expiry it forms a cluster on itself.
|
||||
///
|
||||
/// <para><b>Island safety.</b> Fires ONLY when this node's own address is in its own seed list.
|
||||
/// A node that is not a seed (never legitimately first) must keep waiting: if it self-formed,
|
||||
/// a later-booting real seed would form a second cluster and the two can never merge. For nodes
|
||||
/// that ARE seeds, sequential recovery is island-free — Akka's join protocol prefers an existing
|
||||
/// cluster (a booting node only self-forms when NO seed answers InitJoin), so a peer booting
|
||||
/// after this node self-formed simply joins it.</para>
|
||||
///
|
||||
/// <para><b>Races are benign.</b> If the join completes between window expiry and
|
||||
/// <c>Cluster.Join(SelfAddress)</c>, Akka ignores the join — a node joins a cluster at most once
|
||||
/// per incarnation. The residual risk is both nodes cold-starting inside the window while
|
||||
/// mutually unreachable (a boot-time partition): both self-form, the same dual-active class the
|
||||
/// auto-down downing strategy already accepts, with the same recovery (restart one side).</para>
|
||||
/// </summary>
|
||||
public static class ClusterBootstrapFallback
|
||||
{
|
||||
public static void Arm(ActorSystem system, ClusterOptions options, ILogger logger)
|
||||
{
|
||||
if (options.SelfFormAfter is not { } window || window <= TimeSpan.Zero)
|
||||
{
|
||||
logger.LogInformation(
|
||||
"Cluster self-form fallback disabled (SelfFormAfter not set) — a node cold-starting "
|
||||
+ "while its peer is down will wait on InitJoin indefinitely.");
|
||||
return;
|
||||
}
|
||||
|
||||
var cluster = Akka.Cluster.Cluster.Get(system);
|
||||
var self = cluster.SelfAddress;
|
||||
var isSeed = options.SeedNodes.Any(s => TryParseAddress(s, out var a) && a.Equals(self));
|
||||
if (!isSeed)
|
||||
{
|
||||
logger.LogInformation(
|
||||
"Cluster self-form fallback inactive: this node ({Self}) is not in its own seed list "
|
||||
+ "[{Seeds}] — self-forming here would island it from the real seeds.",
|
||||
self, string.Join(", ", options.SeedNodes));
|
||||
return;
|
||||
}
|
||||
|
||||
var joined = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
cluster.RegisterOnMemberUp(() => joined.TrySetResult());
|
||||
|
||||
_ = Task.Run(async () =>
|
||||
{
|
||||
var winner = await Task.WhenAny(joined.Task, Task.Delay(window));
|
||||
if (winner == joined.Task || system.WhenTerminated.IsCompleted)
|
||||
return;
|
||||
logger.LogWarning(
|
||||
"No cluster membership after {Window} — no seed answered InitJoin (peer down at boot). "
|
||||
+ "Self-forming a cluster at {Self} so this node becomes operational; if the peer was "
|
||||
+ "merely partitioned (not dead), the pair is now dual-active — restart one side after "
|
||||
+ "the partition heals (accepted availability-first trade, decision 2026-07-22).",
|
||||
window, self);
|
||||
cluster.Join(self);
|
||||
});
|
||||
}
|
||||
|
||||
private static bool TryParseAddress(string seed, out Address address)
|
||||
{
|
||||
try { address = Address.Parse(seed); return true; }
|
||||
catch { address = default!; return false; }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
(Add the `Microsoft.Extensions.Logging` using. If non-generic `TaskCompletionSource` is unavailable, use `TaskCompletionSource<bool>` + `TrySetResult(true)`.)
|
||||
|
||||
**Step 4: Run → PASS** (~5–10 s). **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add -A && git commit -m "feat(cluster): InitJoin self-form fallback — lone non-first seed becomes Up"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3 (A3): Remaining fallback tests — disabled / late-peer merge / non-seed
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 4
|
||||
|
||||
**Files:**
|
||||
- Modify: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SelfFormBootstrapTests.cs`
|
||||
|
||||
**Step 1: Append three tests** (each absence assertion carries an in-test positive control — repo convention):
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task Disabled_fallback_keeps_waiting_and_the_node_was_otherwise_formable()
|
||||
{
|
||||
var selfPort = TwoNodeClusterFixture.GetFreeTcpPort();
|
||||
var deadPeerPort = TwoNodeClusterFixture.GetFreeTcpPort();
|
||||
var node = StartNode(selfPort, deadPeerPort, selfFormAfter: null);
|
||||
|
||||
await Task.Delay(TimeSpan.FromSeconds(6)); // 3x the window used in the enabled test
|
||||
var cluster = Cluster.Get(node);
|
||||
cluster.State.Members.ShouldBeEmpty(); // still InitJoin-looping — today's behavior
|
||||
|
||||
// POSITIVE CONTROL: prove the node COULD have formed; only the fallback was missing.
|
||||
cluster.Join(cluster.SelfAddress);
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(node, 1, TimeSpan.FromSeconds(20));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Peer_booting_after_self_form_joins_the_existing_cluster_no_island()
|
||||
{
|
||||
var portA = TwoNodeClusterFixture.GetFreeTcpPort();
|
||||
var portB = TwoNodeClusterFixture.GetFreeTcpPort();
|
||||
|
||||
// B cold-starts alone (A dead), self-forms after 2s.
|
||||
var nodeB = StartNode(portB, peerPort: portA, selfFormAfter: TimeSpan.FromSeconds(2));
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(nodeB, 1, TimeSpan.FromSeconds(20));
|
||||
|
||||
// A boots later with the pair seed list. B answers InitJoin, so A must JOIN B's
|
||||
// cluster instead of islanding. Generous window on A so its fallback can't race.
|
||||
var nodeA = StartNode(portA, peerPort: portB, selfFormAfter: TimeSpan.FromSeconds(30));
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 2, TimeSpan.FromSeconds(20));
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(nodeB, 2, TimeSpan.FromSeconds(20));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Non_seed_node_never_self_forms()
|
||||
{
|
||||
var selfPort = TwoNodeClusterFixture.GetFreeTcpPort();
|
||||
var deadSeedPort = TwoNodeClusterFixture.GetFreeTcpPort();
|
||||
var node = StartNode(selfPort, deadSeedPort, selfFormAfter: TimeSpan.FromSeconds(1), selfInSeeds: false);
|
||||
|
||||
await Task.Delay(TimeSpan.FromSeconds(5)); // 5x the window
|
||||
var cluster = Cluster.Get(node);
|
||||
cluster.State.Members.ShouldBeEmpty(); // guard refused to island a non-seed
|
||||
|
||||
// Positive control: the guard (not the environment) prevented formation.
|
||||
cluster.Join(cluster.SelfAddress);
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(node, 1, TimeSpan.FromSeconds(20));
|
||||
}
|
||||
```
|
||||
|
||||
**Step 2: Run** — all 4 PASS. **Step 3: Commit** `test(cluster): self-form fallback — disabled, late-peer merge, non-seed guard`.
|
||||
|
||||
---
|
||||
|
||||
### Task 4 (A4): Production wiring + fix the misleading `SeedNodes` doc comment
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~3 min
|
||||
**Parallelizable with:** Task 3
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs` (inside `GetOrCreateActorSystem`, after the `WhenTerminated` continuation ending ~line 218, before `_actorSystem = system;`)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs:33-38`
|
||||
|
||||
**Step 1: Wire the fallback:**
|
||||
|
||||
```csharp
|
||||
// InitJoin self-form fallback (decision 2026-07-22): without it a non-first seed
|
||||
// cold-starting while its peer is down loops on InitJoin forever — auto-down closed
|
||||
// the crash-failover gap, this closes the cold-start-alone gap. Guarded inside Arm:
|
||||
// disabled when SelfFormAfter is unset, inert when this node is not its own seed.
|
||||
ClusterBootstrapFallback.Arm(system, _clusterOptions, _logger);
|
||||
```
|
||||
|
||||
**Step 2: Fix the `SeedNodes` doc comment** (replace the `<summary>`):
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Akka.NET cluster seed nodes. Both nodes are seed nodes — each node lists itself and its
|
||||
/// partner. NOTE: listing both is NOT sufficient for "either can start first": Akka only lets
|
||||
/// the FIRST listed seed form a new cluster, so a lone non-first seed waits on InitJoin until
|
||||
/// <see cref="SelfFormAfter"/> expires and the self-form fallback fires (decision 2026-07-22).
|
||||
/// Must contain at least one entry.
|
||||
/// </summary>
|
||||
```
|
||||
|
||||
**Step 3: Build + targeted tests**
|
||||
|
||||
```bash
|
||||
dotnet build ZB.MOM.WW.ScadaBridge.slnx # 0 warnings (TreatWarningsAsErrors)
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests
|
||||
```
|
||||
|
||||
**Step 4: Commit** `feat(cluster): arm self-form fallback at ActorSystem creation; honest SeedNodes doc`.
|
||||
|
||||
---
|
||||
|
||||
### Task 5 (A5): appsettings sweep
|
||||
|
||||
**Classification:** trivial
|
||||
**Estimated implement time:** ~3 min
|
||||
**Parallelizable with:** Task 6
|
||||
|
||||
Add `"SelfFormAfter": "00:00:10"` next to `"SplitBrainResolverStrategy"` in the `ScadaBridge:Cluster` section of each (explicit for operator visibility; matches the code default):
|
||||
|
||||
- `src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Central.json`, `appsettings.Site.json`
|
||||
- `docker/central-node-a/appsettings.Central.json`, `docker/central-node-b/appsettings.Central.json`
|
||||
- `docker/site-a-node-a/appsettings.Site.json`, `docker/site-a-node-b/appsettings.Site.json`
|
||||
- `docker/site-b-node-a/appsettings.Site.json`, `docker/site-b-node-b/appsettings.Site.json`
|
||||
- `docker/site-c-node-a/appsettings.Site.json`, `docker/site-c-node-b/appsettings.Site.json`
|
||||
- `docker-env2/` — the 4 node appsettings files
|
||||
|
||||
Then `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests` and commit `config(cluster): SelfFormAfter=10s explicit in all node appsettings`.
|
||||
|
||||
> Ops note (do NOT edit here): the gitignored `deploy/wonder-app-vd03/` overlay gets the same key on the next production deploy.
|
||||
|
||||
---
|
||||
|
||||
### Task 6 (A6): Docs
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 5
|
||||
|
||||
- `docs/requirements/Component-ClusterInfrastructure.md` — replace the "registered outage gap" recovery text (env-var override) with `SelfFormAfter` semantics + island guard + boot-partition trade.
|
||||
- `docker/README.md` (~line 289) — same replacement; keep the partition-trade note.
|
||||
- `docs/deployment/topology-guide.md:101` — **fix the stale keep-oldest claim** (still says "Keep-oldest with `down-if-alone = on`"): rewrite to auto-down default + `SelfFormAfter`, pointing at the decision records.
|
||||
- `CLAUDE.md` (~line 222) — update the boot-order note: pairs no longer require the first seed for cold start; note the 10 s window.
|
||||
|
||||
Commit: `docs(cluster): SelfFormAfter fallback; fix stale keep-oldest note in topology guide`.
|
||||
|
||||
---
|
||||
|
||||
### Task 7 (A7): Full verification + optional docker live gate
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min (suite runtime dominates)
|
||||
**Parallelizable with:** none
|
||||
|
||||
**Step 1:**
|
||||
|
||||
```bash
|
||||
cd ~/Desktop/ScadaBridge
|
||||
dotnet build ZB.MOM.WW.ScadaBridge.slnx # expect 0 warnings
|
||||
dotnet test ZB.MOM.WW.ScadaBridge.slnx # expect green vs pre-existing baseline
|
||||
```
|
||||
|
||||
**Step 2 (LIVE GATE — run if the docker rig is available; else record deferred-live):**
|
||||
|
||||
```bash
|
||||
cd docker && bash deploy.sh # rebuild with the fallback
|
||||
docker compose stop central-a central-b # verify service names in docker-compose.yml first
|
||||
docker compose start central-b # cold-start ONLY the non-first seed
|
||||
docker compose logs -f central-b | grep -m1 "Self-forming a cluster" # ≈10s after start
|
||||
curl -fsS http://localhost:9002/health/active # expect 200
|
||||
docker compose start central-a # first seed returns → must JOIN, not island
|
||||
docker compose logs central-a | grep -i "Welcome"
|
||||
```
|
||||
|
||||
Expected: `central-b` self-forms ≈10 s and serves (previously a permanent wedge); `central-a` rejoins as youngest/standby.
|
||||
|
||||
---
|
||||
|
||||
### Task 8 (D1): `IManualFailoverService` + cluster-level test
|
||||
|
||||
**Classification:** high-risk (cluster behavior)
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** none
|
||||
|
||||
**Files:**
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/IManualFailoverService.cs`
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.Host/Health/AkkaManualFailoverService.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` (register in the Central branch, next to `IActiveNodeGate` ~line 330)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/ManualFailoverTests.cs`
|
||||
|
||||
**Step 1: Failing tests** on `TwoNodeClusterFixture`:
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task Failover_makes_the_oldest_leave_and_the_survivor_take_over()
|
||||
{
|
||||
await using var f = await TwoNodeClusterFixture.StartAsync();
|
||||
var oldest = Akka.Cluster.Cluster.Get(f.NodeA); // NodeA started first = oldest
|
||||
var target = AkkaManualFailoverService.FailOverCore(f.NodeB, "Central"); // issued from the OTHER node
|
||||
target.ShouldBe(oldest.SelfAddress);
|
||||
|
||||
// Graceful exit path: the left node's own ActorSystem terminates…
|
||||
await f.NodeA.WhenTerminated.WaitAsync(TimeSpan.FromSeconds(30));
|
||||
// …and the survivor becomes a 1-member cluster and the oldest-Up active node.
|
||||
await TwoNodeClusterFixture.WaitForMemberRemoved(f.NodeB, oldest.SelfAddress, TimeSpan.FromSeconds(30));
|
||||
ActiveNodeEvaluator.SelfIsOldestUp(Akka.Cluster.Cluster.Get(f.NodeB)).ShouldBeTrue();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Failover_refuses_when_no_peer_exists()
|
||||
{ /* 1-node cluster → FailOverCore returns null, node still Up afterwards (positive assert) */ }
|
||||
```
|
||||
|
||||
**Step 2:** Run → FAIL (service missing).
|
||||
|
||||
**Step 3: Implement.** Interface in CentralUI (CentralUI stays Akka-free):
|
||||
|
||||
```csharp
|
||||
public interface IManualFailoverService
|
||||
{
|
||||
/// <summary>Gracefully fails over the central cluster: the current active (oldest Up)
|
||||
/// member leaves, restarts via its supervisor, and rejoins as standby. Returns the
|
||||
/// address string acted on, or null when there is no peer to fail over to.</summary>
|
||||
Task<string?> FailOverCentralAsync(string actor);
|
||||
}
|
||||
```
|
||||
|
||||
Host implementation — static testable core + thin DI wrapper:
|
||||
|
||||
```csharp
|
||||
public sealed class AkkaManualFailoverService : IManualFailoverService
|
||||
{
|
||||
// ctor: (AkkaHostedService akka, ICentralAuditWriter audit, ILogger<AkkaManualFailoverService> logger)
|
||||
|
||||
public async Task<string?> FailOverCentralAsync(string actor)
|
||||
{
|
||||
var system = _akka.GetOrCreateActorSystem();
|
||||
var target = FailOverCore(system, role: "Central", dryRun: true);
|
||||
if (target is null) return null; // peer guard
|
||||
await _audit.WriteAsync(/* canonical AuditEvent: Action=cluster.manual-failover,
|
||||
Actor=actor, DetailsJson={"target": target}, Outcome=Success — copy the exact
|
||||
call shape from an existing audited admin action (e.g. the Sites admin service);
|
||||
use the CENTRAL audit writer, not the shared seam */);
|
||||
FailOverCore(system, role: "Central"); // Cluster.Leave(target)
|
||||
_logger.LogWarning("Manual failover triggered by {Actor}: {Target} is leaving the cluster.", actor, target);
|
||||
return target.ToString();
|
||||
}
|
||||
|
||||
/// <summary>Oldest Up member with the role leaves — mirrors ActiveNodeEvaluator's oldest-Up
|
||||
/// rule so the node acted on is exactly the one hosting the singletons. Returns null when
|
||||
/// fewer than 2 Up members carry the role (no peer = failover would be an outage).</summary>
|
||||
public static Address? FailOverCore(ActorSystem system, string role, bool dryRun = false)
|
||||
{
|
||||
var cluster = Akka.Cluster.Cluster.Get(system);
|
||||
var withRole = cluster.State.Members
|
||||
.Where(m => m.Status == MemberStatus.Up && m.HasRole(role))
|
||||
.OrderBy(m => m, Member.AgeOrdering)
|
||||
.ToList();
|
||||
if (withRole.Count < 2) return null;
|
||||
var oldest = withRole[0];
|
||||
if (!dryRun) cluster.Leave(oldest.Address);
|
||||
return oldest.Address;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Step 4:** Register in the Central branch of `Program.cs`. **Step 5:** Tests PASS → commit `feat(ui): manual central failover service — graceful Leave of the oldest Up member`.
|
||||
|
||||
---
|
||||
|
||||
### Task 9 (D2): Health page button + bUnit tests + runbook
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** none
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor` (central-cluster card, near the Nodes column ~line 233)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/HealthFailoverButtonTests.cs` (follow the project's existing bUnit page-test pattern)
|
||||
- Modify: `docs/requirements/Component-ClusterInfrastructure.md` + `docker/README.md` — manual-failover runbook paragraph
|
||||
|
||||
**Step 1: Failing bUnit tests:** (a) button absent without the admin policy; (b) present + enabled for admin with ≥2 online central nodes; (c) disabled with tooltip at 1 node; (d) confirm flow calls `IManualFailoverService.FailOverCentralAsync` exactly once (fake service).
|
||||
|
||||
**Step 2: Implement:**
|
||||
|
||||
```razor
|
||||
<AuthorizeView Policy="@AuthorizationPolicies.RequireAdmin">
|
||||
<button class="btn btn-outline-warning btn-sm"
|
||||
disabled="@(!CentralHasPeer)"
|
||||
title="@(CentralHasPeer ? "Gracefully restart the active node; the standby takes over."
|
||||
: "No standby available — failover would be an outage.")"
|
||||
@onclick="() => _showFailoverConfirm = true">
|
||||
Trigger failover
|
||||
</button>
|
||||
</AuthorizeView>
|
||||
```
|
||||
|
||||
plus a confirmation modal (copy the page's existing dialog idiom). The warning text MUST state: the active node restarts, roles swap, and this page will briefly disconnect and reconnect against the new active node (Traefik routes to the active). On confirm: call the service with the authenticated user name; surface the returned target address.
|
||||
|
||||
**Step 3:** Tests PASS → commit `feat(ui): admin manual-failover control on the health page`.
|
||||
|
||||
**Live check (fold into the Task 7 gate when the rig is up):** press the button, watch `central-a` restart and `central-b`'s badge flip to Primary; audit row lands in `dbo.AuditLog`.
|
||||
|
||||
---
|
||||
|
||||
### Task 10 (D3, OPTIONAL — confirm with the user before executing): site-pair failover from the central UI
|
||||
|
||||
**Classification:** high-risk (new cross-cluster message contract)
|
||||
**Estimated implement time:** exploration first; likely 2–3 tasks if approved
|
||||
|
||||
The Health page also shows per-site node cards (Primary/Standby from heartbeats), but central and sites are **separate Akka clusters** — a site failover needs a `TriggerSiteFailover(siteId)` command over the existing central→site transport (same channel as the Retry/Discard relay), handled on the site's active node by `Cluster.Leave(SelfAddress)`. That adds a versioned message contract (rolling-upgrade surface). If approved: explore `ZB.MOM.WW.ScadaBridge.Communication` for the command path, mirror an existing command end-to-end, per-site button with the same guard/confirm/audit rules. Otherwise: file a follow-up issue and skip.
|
||||
|
||||
---
|
||||
|
||||
## Completion
|
||||
|
||||
- Merge decision via the finishing-a-development-branch flow (family convention: ff-merge to `main` + push to gitea, or PR — ask the user).
|
||||
- After merge: update `scadaproj/akka_failover.md` §6.1 status, `scadaproj/CLAUDE.md` index row, and memory `ha-availability-over-partition-safety` (tracked as the family-docs task in the scadaproj index plan).
|
||||
- Verification-before-completion applies throughout: no task is done without its command output; live gates may be recorded deferred-live if the rig is down.
|
||||
@@ -0,0 +1,61 @@
|
||||
{
|
||||
"planPath": "docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md",
|
||||
"tasks": [
|
||||
{
|
||||
"id": 0,
|
||||
"subject": "Part 1 (SUPERSEDED Tasks 1-4): SelfFormAfter watchdog - implemented, review+test rejected it, reverted",
|
||||
"status": "completed"
|
||||
},
|
||||
{
|
||||
"id": 1,
|
||||
"subject": "Part 1 as executed: self-first seed ordering + StartupValidator rule + SelfFirstSeedBootstrapTests + 14-config sweep + docs",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
0
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"subject": "Task 7: full solution verification + optional docker live gate",
|
||||
"status": "in_progress",
|
||||
"blockedBy": [
|
||||
1
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"subject": "Task 8: IManualFailoverService + cluster-level test",
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"subject": "Task 9: Health page failover button + bUnit tests + runbook",
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
3
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"subject": "Task 10 (OPTIONAL, ask user): site-pair failover via central-to-site transport",
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
4
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"subject": "FOLLOW-UP (owner deferred): correct the false InitJoin safety claim in scadaproj + OtOpcUa design docs",
|
||||
"status": "pending"
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"subject": "OPS: reorder SeedNodes self-first in gitignored deploy/wonder-app-vd03/ before next deploy (hard boot gate)",
|
||||
"status": "pending"
|
||||
}
|
||||
],
|
||||
"lastUpdated": "2026-07-22T00:00:00Z"
|
||||
}
|
||||
Reference in New Issue
Block a user