# Cluster, Host & Failover Fixes Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.

**Goal:** Make automatic failover actually work (enable the SBR downing provider and prove it with a real two-node kill test), unify the "active node" definition on singleton-host (oldest) semantics, and close the CoordinatedShutdown, health-report-loss, offline-detection, and config-validation gaps from architecture review 01.

**Architecture:** All cluster bootstrap lives in `AkkaHostedService.BuildHocon` (hand-rolled, injection-safe HOCON) — the Critical fix is adding `akka.cluster.downing-provider-class` there, verified by a new reusable two-node in-process cluster fixture (`TwoNodeClusterFixture`) that other plans (02) also depend on. "Active node" is re-derived everywhere from a single new `ClusterActivityEvaluator` (self is the oldest `Up` member = singleton host), replacing the four divergent leader-based checks; the five copy-pasted central-singleton registration blocks are collapsed into a `CentralSingletonRegistrar` helper that also gives the two hot singletons (notification-outbox, audit-log-ingest) the drain tasks they were missing.

**Tech Stack:** C#/.NET, Akka.NET 1.5.62 (Akka.Cluster, Akka.Cluster.Tools, Akka.TestKit.Xunit2), xUnit + NSubstitute, Docker Compose + Traefik, Blazor Server (one small UI badge).

Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per-project: `dotnet test tests/<project>`. Docker rig: `bash docker/deploy.sh`.

## Findings Coverage

| # | Report finding | Severity | Task(s) / Disposition |
|---|---------------|----------|----------------------|
| S1 | Split-brain resolver never enabled — no automatic failover on crash/partition | Critical | Tasks 1, 3, 4, 21 |
| S2 | "Active node" = leader vs oldest diverge (purge on wrong node, wrong Primary label, wrong self-report, Traefik→non-singleton node) | High | Tasks 5, 6, 7, 8 |
| S3 | Partition: both central nodes return 200 on `/health/active`, Traefik serves both | High | Tasks 1 (SBR resolves partition), 7 (oldest-based + DB-required active check), 4 (behavioral proof) |
| S4 | Cluster-leave drain tasks budgeted 10s inside 5s CoordinatedShutdown phase | Medium | Task 2 |
| S5 | NotificationOutbox + AuditLogIngest singletons have no drain task | Medium | Tasks 9, 10 |
| S6 | Health-report loss recovery is dead code (fire-and-forget transport never throws) | Medium | Tasks 11, 12 |
| S7 | Offline detection keys off heartbeats, masking a dead metrics pipeline; spec/code disagree | Medium | Tasks 13, 14 |
| S8 | `appsettings.Site.json` second seed targets MetricsPort 8084; validator gap | Medium | Task 15 |
| S9 | Docker `stop_grace_period` shorter than full CoordinatedShutdown | Medium | Task 17 |
| S10 | Aggregator never forgets deleted sites | Low | Task 18 |
| S11 | Dead-letter monitor: unbounded warning volume | Low | Task 19 |
| S12 | Mandatory two-seed rule forces phantom seeds in single-node installs | Low | Task 16 |
| P1 | Readiness probes fan out 5 cluster Asks per poll | Low | **Accepted** — bounded (2s, concurrent), cheap at current poll rates; revisit only if an orchestrator tightens the probe interval |
| P2 | Startup compiles every inbound method sequentially | Low | **Deferred** — Inbound API domain (plan 06 scope); report itself says "acceptable now" |
| P3 | Per-tick allocations in health collection | Low | **Won't-fix** — report: "trivial at this cadence. No action" |
| C1 | REQ-HOST-6 mandates Akka.Hosting; code hand-rolls HOCON | Medium | Task 22 |
| C2 | `deploy/wonder-app-vd03` ships `DisableLogin: true` | High | **Deferred → Plan 07** (explicit cross-plan ownership) |
| C3 | LDAP plaintext in deploy artifact | Medium | **Deferred → Plan 07** (deploy-artifact security posture is plan 07's) |
| C4 | Secrets in checked-in docker dev configs | Medium | **Accepted** — labelled dev-only, `ConfigSecretsTests` tracks it, the wonder-app-vd03 `${...}` pattern already exists as the model |
| C5 | Stale `Component-TraefikProxy.md` references | Low | Task 23 |
| C6 | Five copy-pasted ~60-line singleton registration blocks | Low | Tasks 9, 10 |
| U1 | No real multi-node failover test anywhere | — | Tasks 3, 4, 8, 21 (the rig other plans reference) |
| U2 | Windows Service lifecycle / down-if-alone recovery story undefined | — | Task 20 |
| U3 | TLS absent at every layer of this domain | — | **Deferred** — documented lab posture; Task 23 adds an explicit "production TLS profile — not yet implemented" roadmap note so the gap is spec-visible |
| U4 | Dockerfile has no HEALTHCHECK; `/healthz` runs zero checks | — | **Deferred** — report calls this "acceptable minimalism"; prerequisite work for a future orchestrator migration, not this hardening pass |
| U5 | `NodeName` missing from wonder-app-vd03 overlays → NULL `SourceNode` | — | Task 23 |
| U6 | Known-but-open follow-ups (SecuredWrite SourceNode NULL, cert-trust persistence, docker-env2 mirrors findings) | — | **No action** — already tracked in CLAUDE.md/logged follow-ups; docker-env2 compose IS covered by Task 17 |
| U7 | Online/offline transitions unrecorded (post-incident "when did the site drop") | — | Tasks 13, 14 (`LastStatusChangeAt`) |

---

### Task 1: Enable the SBR downing provider in BuildHocon

**Classification:** high-risk (cluster failover semantics — the Critical finding)
**Estimated implement time:** ~3 min
**Parallelizable with:** 5, 11, 13, 15, 16, 17, 19, 22, 23
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs` (BuildHocon, cluster block lines 250-266)
- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HoconBuilderTests.cs`

1. Write the failing test in `HoconBuilderTests.cs` (reuse the existing `DefaultCluster()` helper):

```csharp
[Fact]
public void BuildHocon_EnablesSplitBrainResolverDowningProvider()
{
    // Review 01 [Critical]: without downing-provider-class the entire
    // split-brain-resolver section is inert (Akka default = NoDowning) and
    // singletons never migrate on a hard crash.
    var node = new NodeOptions { Role = "Central", NodeHostname = "node1", RemotingPort = 8081 };

    var hocon = AkkaHostedService.BuildHocon(
        node, DefaultCluster(), new[] { "Central" },
        TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(15));

    var config = ConfigurationFactory.ParseString(hocon);
    Assert.Equal(
        "Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster",
        config.GetString("akka.cluster.downing-provider-class"));
}
```

2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~BuildHocon_EnablesSplitBrainResolverDowningProvider"` — expect FAIL (key missing).
3. In `BuildHocon`, add one line inside the `cluster {{` block, directly above `split-brain-resolver {{` (after `min-nr-of-members = {clusterOptions.MinNrOfMembers}`):

```csharp
        downing-provider-class = ""Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster""
```

(inside the interpolated string — note doubled quotes). Also update the method's XML doc comment to state the downing provider is explicitly installed because Akka defaults to NoDowning.
4. Run the test again — expect PASS. Run the full file: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~HoconBuilderTests"` — all pass.
5. Commit:
```
git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HoconBuilderTests.cs
git commit -m "fix(cluster): enable SBR downing provider — automatic failover was inert for crashes/partitions"
```

### Task 2: Raise the cluster-leave CoordinatedShutdown phase timeout above the drain budget

**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** 5, 11, 13, 15, 16, 17, 19, 22, 23
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs` (BuildHocon, coordinated-shutdown block lines 267-269)
- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HoconBuilderTests.cs`

1. Write the failing test:

```csharp
[Fact]
public void BuildHocon_ClusterLeavePhaseTimeout_ExceedsDrainBudget()
{
    // Review 01 [Medium]: the five singleton drain tasks GracefulStop(10s)
    // inside cluster-leave, whose default phase timeout is 5s — Akka abandons
    // the task at 5s and the drain guarantee is silently halved.
    var node = new NodeOptions { Role = "Central", NodeHostname = "node1", RemotingPort = 8081 };
    var hocon = AkkaHostedService.BuildHocon(
        node, DefaultCluster(), new[] { "Central" },
        TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(15));

    var config = ConfigurationFactory.ParseString(hocon);
    var phaseTimeout = config.GetTimeSpan("akka.coordinated-shutdown.phases.cluster-leave.timeout");
    Assert.True(phaseTimeout >= TimeSpan.FromSeconds(15),
        $"cluster-leave phase timeout ({phaseTimeout}) must exceed the 10s singleton GracefulStop budget");
}
```

2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~ClusterLeavePhaseTimeout"` — expect FAIL.
3. In `BuildHocon`, extend the `coordinated-shutdown` block:

```
    coordinated-shutdown {{
        run-by-clr-shutdown-hook = on
        phases.cluster-leave.timeout = 15s
    }}
```

4. Run again — expect PASS. Also run `--filter "FullyQualifiedName~CoordinatedShutdownTests"` (source-grep tests must stay green).
5. Commit:
```
git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HoconBuilderTests.cs
git commit -m "fix(cluster): give cluster-leave phase a 15s timeout so 10s singleton drains can complete"
```

### Task 3: Two-node in-process cluster fixture (the reusable failover rig)

**Classification:** high-risk (real cluster formation in tests)
**Estimated implement time:** ~5 min
**Parallelizable with:** 5, 11, 13, 15, 16, 17, 19, 22, 23
**Files:**
- Create: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/TwoNodeClusterFixture.cs`
- Test: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/TwoNodeClusterFixtureTests.cs`

This fixture is the rig other plans (02: S&F standby gating) reference — keep its API small and public. It builds each node's config from the **production** `AkkaHostedService.BuildHocon` (so the config under test is the config that ships), overlaid with test-only crash-simulation settings.

1. Create the fixture:

```csharp
using Akka.Actor;
using Akka.Cluster;
using Akka.Configuration;
using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
using ZB.MOM.WW.ScadaBridge.Host;
using ZB.MOM.WW.ScadaBridge.Host.Actors;

namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;

/// <summary>
/// Forms a REAL two-node Akka cluster in-process from the production
/// BuildHocon output. NodeA is always started first (=> oldest member /
/// singleton host). CrashNode() terminates a node WITHOUT cluster-leave
/// (coordinated-shutdown by-terminate disabled on both nodes) to simulate
/// a hard crash — the scenario review 01 found untested.
/// </summary>
public sealed class TwoNodeClusterFixture : IAsyncDisposable
{
    public ActorSystem NodeA { get; private set; } = null!;
    public ActorSystem NodeB { get; private set; } = null!;
    public int PortA { get; private set; }
    public int PortB { get; private set; }

    public static async Task<TwoNodeClusterFixture> StartAsync(
        string role = "Central", TimeSpan? stableAfter = null)
    {
        var f = new TwoNodeClusterFixture();
        f.PortA = GetFreeTcpPort();
        f.PortB = GetFreeTcpPort();
        f.NodeA = f.StartNode(f.PortA, role, stableAfter);
        await WaitForMembersUp(f.NodeA, 1, TimeSpan.FromSeconds(20));
        f.NodeB = f.StartNode(f.PortB, role, stableAfter);
        await WaitForMembersUp(f.NodeA, 2, TimeSpan.FromSeconds(20));
        await WaitForMembersUp(f.NodeB, 2, TimeSpan.FromSeconds(20));
        return f;
    }

    /// <summary>Starts a node from production HOCON; used by StartAsync and by restart-scenarios.</summary>
    public ActorSystem StartNode(int port, string role, TimeSpan? stableAfter = null)
    {
        var nodeOptions = new NodeOptions { Role = role, NodeHostname = "127.0.0.1", RemotingPort = port };
        var clusterOptions = new ClusterOptions
        {
            SeedNodes = new List<string>
            {
                $"akka.tcp://scadabridge@127.0.0.1:{PortA}",
                $"akka.tcp://scadabridge@127.0.0.1:{PortB}",
            },
            SplitBrainResolverStrategy = "keep-oldest",
            StableAfter = stableAfter ?? TimeSpan.FromSeconds(3),
            HeartbeatInterval = TimeSpan.FromMilliseconds(500),
            FailureDetectionThreshold = TimeSpan.FromSeconds(2),
            MinNrOfMembers = 1,
            DownIfAlone = true,
        };
        var hocon = AkkaHostedService.BuildHocon(
            nodeOptions, clusterOptions, new[] { role },
            TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(3));
        // Crash simulation: Terminate() must NOT run CoordinatedShutdown
        // (no Leave gossip) or the test degenerates to the graceful path.
        var config = ConfigurationFactory
            .ParseString("akka.coordinated-shutdown.run-by-actor-system-terminate = off")
            .WithFallback(ConfigurationFactory.ParseString(hocon));
        return ActorSystem.Create("scadabridge", config);
    }

    /// <summary>Hard-crash a node: abrupt terminate, no cluster leave.</summary>
    public static Task CrashNode(ActorSystem node) => node.Terminate();

    public static async Task WaitForMembersUp(ActorSystem system, int count, TimeSpan timeout)
    {
        var cluster = Akka.Cluster.Cluster.Get(system);
        var deadline = DateTime.UtcNow + timeout;
        while (DateTime.UtcNow < deadline)
        {
            if (cluster.State.Members.Count(m => m.Status == MemberStatus.Up) == count
                && !cluster.State.Unreachable.Any())
                return;
            await Task.Delay(200);
        }
        throw new TimeoutException(
            $"Cluster on {cluster.SelfAddress} did not reach {count} Up members within {timeout}. " +
            $"Members: [{string.Join(", ", cluster.State.Members)}], " +
            $"Unreachable: [{string.Join(", ", cluster.State.Unreachable)}]");
    }

    public static async Task WaitForMemberRemoved(ActorSystem survivor, Address removed, TimeSpan timeout)
    {
        var cluster = Akka.Cluster.Cluster.Get(survivor);
        var deadline = DateTime.UtcNow + timeout;
        while (DateTime.UtcNow < deadline)
        {
            if (cluster.State.Members.All(m => m.Address != removed)
                && cluster.State.Unreachable.All(m => m.Address != removed))
                return;
            await Task.Delay(200);
        }
        throw new TimeoutException($"Member {removed} was never removed from {cluster.SelfAddress}'s view — SBR did not down it.");
    }

    private static int GetFreeTcpPort()
    {
        var listener = new System.Net.Sockets.TcpListener(System.Net.IPAddress.Loopback, 0);
        listener.Start();
        var port = ((System.Net.IPEndPoint)listener.LocalEndpoint).Port;
        listener.Stop();
        return port;
    }

    public async ValueTask DisposeAsync()
    {
        foreach (var sys in new[] { NodeA, NodeB })
        {
            if (sys is { WhenTerminated.IsCompleted: false })
            {
                try { await sys.Terminate().WaitAsync(TimeSpan.FromSeconds(10)); } catch { /* teardown */ }
            }
        }
    }
}
```

2. Write the smoke test proving the fixture forms a real cluster:

```csharp
public class TwoNodeClusterFixtureTests
{
    [Fact]
    public async Task Fixture_FormsTwoNodeCluster_NodeAIsOldest()
    {
        await using var cluster = await TwoNodeClusterFixture.StartAsync();
        var a = Akka.Cluster.Cluster.Get(cluster.NodeA);
        var b = Akka.Cluster.Cluster.Get(cluster.NodeB);
        Assert.Equal(2, a.State.Members.Count);
        Assert.Equal(2, b.State.Members.Count);
        // NodeA started first => oldest (the singleton host).
        Assert.True(a.SelfMember.IsOlderThan(b.State.Members.First(m => m.Address == b.SelfAddress)));
    }
}
```

3. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.IntegrationTests --filter "FullyQualifiedName~TwoNodeClusterFixtureTests"` — expect FAIL first (files don't exist / compile), then PASS once implemented. (This task is fixture-first; the "failing test" step is the compile failure.)
4. If `NodeOptions`/`BuildHocon` visibility blocks compilation, note that `IntegrationTests` already references the Host project (`.csproj` line 37) and both are public — no changes to `src/` are expected in this task.
5. Commit:
```
git add tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/
git commit -m "test(cluster): add TwoNodeClusterFixture — real two-node in-process cluster rig from production HOCON"
```

### Task 4: SBR behavioral proof — kill the oldest node, assert downing + singleton migration

**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** 5, 11, 13, 15, 16, 17, 19, 22, 23
**Files:**
- Create: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SbrFailoverTests.cs`

1. Write the test (this is the test that would have caught the Critical finding — it must FAIL if Task 1's HOCON line is reverted; verify that once by stashing Task 1 if cheap, otherwise trust the assertion on member removal, which is impossible under NoDowning):

```csharp
using Akka.Actor;
using Akka.Cluster.Tools.Singleton;

namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;

public class SbrFailoverTests
{
    private sealed class EchoActor : ReceiveActor
    {
        public EchoActor() => ReceiveAny(msg => Sender.Tell(msg));
    }

    private static (IActorRef manager, IActorRef proxy) StartSingleton(ActorSystem sys)
    {
        var manager = sys.ActorOf(ClusterSingletonManager.Props(
            Props.Create(() => new EchoActor()),
            PoisonPill.Instance,
            ClusterSingletonManagerSettings.Create(sys).WithSingletonName("failover-probe")),
            "failover-probe-singleton");
        var proxy = sys.ActorOf(ClusterSingletonProxy.Props(
            "/user/failover-probe-singleton",
            ClusterSingletonProxySettings.Create(sys).WithSingletonName("failover-probe")),
            "failover-probe-proxy");
        return (manager, proxy);
    }

    [Fact]
    public async Task HardCrashOfOldestNode_SbrDownsIt_AndSingletonMigratesToSurvivor()
    {
        await using var cluster = await TwoNodeClusterFixture.StartAsync();
        StartSingleton(cluster.NodeA);
        var (_, proxyB) = StartSingleton(cluster.NodeB);

        // Singleton is reachable from B while A (oldest) hosts it.
        var echo = await proxyB.Ask<string>("ping", TimeSpan.FromSeconds(20));
        Assert.Equal("ping", echo);

        var victimAddress = Akka.Cluster.Cluster.Get(cluster.NodeA).SelfAddress;
        await TwoNodeClusterFixture.CrashNode(cluster.NodeA);

        // 1) SBR must DOWN and REMOVE the crashed member (NoDowning => this
        //    times out; the member stays unreachable forever).
        // Budget: failure detection (~2s) + stable-after (3s) + gossip margin.
        await TwoNodeClusterFixture.WaitForMemberRemoved(
            cluster.NodeB, victimAddress, TimeSpan.FromSeconds(30));

        // 2) The singleton must re-host on the survivor and answer again.
        var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(30);
        Exception? last = null;
        while (DateTime.UtcNow < deadline)
        {
            try
            {
                var echo2 = await proxyB.Ask<string>("ping-after-failover", TimeSpan.FromSeconds(3));
                Assert.Equal("ping-after-failover", echo2);
                return;
            }
            catch (Exception ex) { last = ex; }
        }
        throw new Xunit.Sdk.XunitException($"Singleton never re-hosted on the survivor after SBR downing: {last}");
    }
}
```

2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.IntegrationTests --filter "FullyQualifiedName~SbrFailoverTests"` — expect PASS (Task 1 is in). To confirm the test has teeth, temporarily comment out the `downing-provider-class` line in `BuildHocon`, re-run, expect FAIL at `WaitForMemberRemoved` (timeout), then restore the line. Do NOT commit the reverted state.
3. Commit:
```
git add tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SbrFailoverTests.cs
git commit -m "test(cluster): prove SBR downs a hard-crashed oldest node and the singleton migrates"
```

### Task 5: ClusterActivityEvaluator — single oldest-member "active node" definition

**Classification:** standard
**Estimated implement time:** ~4 min
**Parallelizable with:** 1, 2, 3, 4, 11, 13, 15, 16, 17, 19, 22, 23
**Files:**
- Create: `src/ZB.MOM.WW.ScadaBridge.Host/Health/ClusterActivityEvaluator.cs`
- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ClusterActivityEvaluatorTests.cs`

1. Write the failing test (real single-node cluster via TestKit-style bootstrap, mirroring `AkkaBootstrapTests`):

```csharp
using Akka.Actor;
using Akka.Configuration;
using ZB.MOM.WW.ScadaBridge.Host.Health;

namespace ZB.MOM.WW.ScadaBridge.Host.Tests;

public class ClusterActivityEvaluatorTests : IAsyncLifetime
{
    private ActorSystem? _system;

    public async Task InitializeAsync()
    {
        var port = FreePort();
        var config = ConfigurationFactory.ParseString($@"
akka {{
    actor.provider = cluster
    remote.dot-netty.tcp {{ hostname = ""127.0.0.1"", port = {port} }}
    cluster {{
        seed-nodes = [""akka.tcp://eval-test@127.0.0.1:{port}""]
        roles = [""Central""]
        min-nr-of-members = 1
    }}
}}");
        _system = ActorSystem.Create("eval-test", config);
        var cluster = Akka.Cluster.Cluster.Get(_system);
        var deadline = DateTime.UtcNow.AddSeconds(20);
        while (cluster.SelfMember.Status != Akka.Cluster.MemberStatus.Up && DateTime.UtcNow < deadline)
            await Task.Delay(100);
    }

    public async Task DisposeAsync() { if (_system != null) await _system.Terminate(); }

    [Fact]
    public void SelfIsOldest_SoleUpMember_ReturnsTrue()
    {
        var cluster = Akka.Cluster.Cluster.Get(_system!);
        Assert.True(ClusterActivityEvaluator.SelfIsOldest(cluster));
        Assert.True(ClusterActivityEvaluator.SelfIsOldest(cluster, "Central"));
    }

    [Fact]
    public void SelfIsOldest_RoleNotHeld_ReturnsFalse()
    {
        var cluster = Akka.Cluster.Cluster.Get(_system!);
        // Self doesn't carry the role => it can never host that role's singletons.
        Assert.False(ClusterActivityEvaluator.SelfIsOldest(cluster, "site-nonexistent"));
    }

    private static int FreePort()
    {
        var l = new System.Net.Sockets.TcpListener(System.Net.IPAddress.Loopback, 0);
        l.Start(); var p = ((System.Net.IPEndPoint)l.LocalEndpoint).Port; l.Stop(); return p;
    }
}
```

2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~ClusterActivityEvaluatorTests"` — expect FAIL (type missing).
3. Implement:

```csharp
using Akka.Cluster;

namespace ZB.MOM.WW.ScadaBridge.Host.Health;

/// <summary>
/// THE single definition of "active node" (review 01 [High]): a node is active
/// when it is the OLDEST Up member (optionally within a role scope) — i.e. the
/// member the ClusterSingletonManager places singletons on. Cluster LEADERSHIP
/// (lowest address) is an Akka-internal concept that diverges from singleton
/// placement permanently once the original first node restarts and rejoins;
/// every product-level active/standby decision must use this evaluator, never
/// cluster.State.Leader.
/// </summary>
public static class ClusterActivityEvaluator
{
    /// <summary>True when self is Up and no other Up member (in the role scope) is older.</summary>
    public static bool SelfIsOldest(Cluster cluster, string? role = null)
    {
        var self = cluster.SelfMember;
        if (self.Status != MemberStatus.Up)
            return false;
        if (role != null && !self.HasRole(role))
            return false;

        return cluster.State.Members
            .Where(m => m.Status == MemberStatus.Up)
            .Where(m => role == null || m.HasRole(role))
            .All(m => m.UniqueAddress.Equals(self.UniqueAddress) || self.IsOlderThan(m));
    }

    /// <summary>The oldest Up member in the role scope, or null while none is Up. Used for Primary/Standby labelling.</summary>
    public static Member? OldestUpMember(Cluster cluster, string? role = null)
    {
        Member? oldest = null;
        foreach (var m in cluster.State.Members.Where(m => m.Status == MemberStatus.Up))
        {
            if (role != null && !m.HasRole(role)) continue;
            if (oldest == null || m.IsOlderThan(oldest)) oldest = m;
        }
        return oldest;
    }
}
```

4. Run — expect PASS.
5. Commit:
```
git add src/ZB.MOM.WW.ScadaBridge.Host/Health/ClusterActivityEvaluator.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ClusterActivityEvaluatorTests.cs
git commit -m "feat(host): ClusterActivityEvaluator — oldest-member (singleton-host) active-node definition"
```

### Task 6: Rewire ActiveNodeGate, AkkaClusterNodeProvider and Primary labels onto the evaluator

**Classification:** high-risk (changes which node the purge, self-report, inbound gate and dashboard consider active)
**Estimated implement time:** ~5 min
**Parallelizable with:** 11, 13, 15, 16, 17, 19, 22, 23
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeGate.cs` (lines 36-52)
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Health/AkkaClusterNodeProvider.cs` (lines 29-40, 43-81)
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/IClusterNodeProvider.cs` (SelfIsPrimary xmldoc, lines 15-21)
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs` (comment at lines 109-115 — text only)
- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ClusterActivityEvaluatorTests.cs` (extend)

1. Write the failing test (extend `ClusterActivityEvaluatorTests` — the single-node cluster from Task 5 is enough because with one member, leader == oldest; the *divergence* behavior is proven in Task 8's two-node test. Here assert the wiring):

```csharp
[Fact]
public void ActiveNodeGate_And_NodeProvider_AgreeWithEvaluator()
{
    // Both product gates must be backed by the oldest-member evaluator.
    // A single-node Up cluster: evaluator says true, so both must say true.
    var akkaService = /* construct AkkaHostedService test double or use the
       pattern in ActorSystemBridgeTests to wrap _system */;
    var gate = new ActiveNodeGate(akkaService);
    var provider = new AkkaClusterNodeProvider(akkaService, "Central");
    Assert.True(gate.IsActiveNode);
    Assert.True(provider.SelfIsPrimary);
}
```

Follow the construction pattern used by `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ActorSystemBridgeTests.cs` / `HealthCheckTests.cs` for obtaining an `AkkaHostedService` whose `ActorSystem` is the test system (if `AkkaHostedService` cannot be instantiated cheaply, add an internal test seam — but check those tests first; a pattern exists because `ActiveNodeGate` is already constructor-tested).
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~AgreeWithEvaluator"` — expect FAIL (or compile error) before rewiring if you assert on a divergence scenario; if single-node passes trivially pre-change, treat this as a pinning test and rely on Task 8 for the behavioral delta.
3. Implement:
   - `ActiveNodeGate.IsActiveNode`: replace the leader comparison (lines 44-51) with `return ClusterActivityEvaluator.SelfIsOldest(cluster);` (keep the null-system and `Up` guards; update the class xmldoc: "oldest Up member — the singleton host — not the cluster leader").
   - `AkkaClusterNodeProvider.SelfIsPrimary`: replace leader comparison with `return ClusterActivityEvaluator.SelfIsOldest(cluster, _siteRole);`.
   - `AkkaClusterNodeProvider.GetClusterNodes`: replace `var leader = cluster.State.Leader;` + `isLeader` with `var oldest = ClusterActivityEvaluator.OldestUpMember(cluster, _siteRole);` and `var isPrimary = oldest != null && member.UniqueAddress.Equals(oldest.UniqueAddress);` — label `"Primary"`/`"Standby"` from that.
   - `IClusterNodeProvider.SelfIsPrimary` xmldoc: change "currently the cluster leader (Primary)" to "currently the oldest Up member for the provider's role scope — the node the ClusterSingletonManager hosts singletons on. This is the canonical product 'active node' check (review 01): the site event-log purge gate, the central self-report loop, and (via plan 02) the S&F delivery gate all consume it."
   - `SiteServiceRegistration.cs` lines 109-115 comment: replace "this node is Up AND cluster leader" wording with "this node is the oldest Up member (singleton host)".
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests` and `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests` — all pass (CentralHealthReportLoop/EventLogPurge consumers are interface-driven; their fakes are unaffected).
5. Commit:
```
git add src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeGate.cs src/ZB.MOM.WW.ScadaBridge.Host/Health/AkkaClusterNodeProvider.cs src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/IClusterNodeProvider.cs src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ClusterActivityEvaluatorTests.cs
git commit -m "fix(host): unify active-node on oldest-member semantics — purge/self-report/labels now track the singleton host"
```

**Seam note for plan 02:** after this task, `IClusterNodeProvider.SelfIsPrimary` (registered in site DI at `SiteServiceRegistration.cs:101-107`) IS the canonical "this node hosts the site singletons / is active" check. Plan 02's S&F delivery gating should inject `IClusterNodeProvider` (or register a delegate mirroring `SiteEventLogActiveNodeCheck`, `SiteServiceRegistration.cs:116-120`) rather than invent a new check.

### Task 7: Oldest-based /health/active check + require DB reachability for Active

**Classification:** high-risk (changes Traefik routing behavior)
**Estimated implement time:** ~5 min
**Parallelizable with:** 11, 13, 15, 16, 17, 19, 22, 23
**Files:**
- Create: `src/ZB.MOM.WW.ScadaBridge.Host/Health/OldestNodeActiveHealthCheck.cs`
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` (health check registrations, lines 209-234; comment block lines 346-356)
- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HealthCheckTests.cs`

1. Write failing tests (follow the existing registration-introspection style in `HealthCheckTests.cs` — it already builds the central service collection and inspects `HealthCheckServiceOptions`):

```csharp
[Fact]
public void ActiveNodeCheck_IsHostLocalOldestNodeCheck()
{
    var registrations = GetCentralHealthCheckRegistrations(); // existing helper or mirror existing tests
    var active = registrations.Single(r => r.Name == "active-node");
    Assert.Contains(ZbHealthTags.Active, active.Tags);
    // The shared package's ActiveNodeHealthCheck is LEADER-based (review 01 [High]):
    // during a partition both nodes think they lead and Traefik serves both.
    var check = active.Factory(BuildCentralServiceProvider());
    Assert.IsType<OldestNodeActiveHealthCheck>(check);
}

[Fact]
public void DatabaseCheck_AlsoGatesActive()
{
    // Review 01 [High] recommendation: /health/active should require Ready
    // (DB reachable) in addition to singleton-host status, so a DB-dead
    // active node is pulled from Traefik rotation.
    var registrations = GetCentralHealthCheckRegistrations();
    var db = registrations.Single(r => r.Name == "database");
    Assert.Contains(ZbHealthTags.Ready, db.Tags);
    Assert.Contains(ZbHealthTags.Active, db.Tags);
}
```

2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~HealthCheckTests"` — expect FAIL.
3. Implement:
   - New `OldestNodeActiveHealthCheck`:

```csharp
using Akka.Actor;
using Microsoft.Extensions.Diagnostics.HealthChecks;

namespace ZB.MOM.WW.ScadaBridge.Host.Health;

/// <summary>
/// /health/active check: Healthy only on the oldest Up member — the node that
/// hosts the cluster singletons. Replaces the shared package's leader-based
/// ActiveNodeHealthCheck (review 01 [High]): leadership is address-ordered and
/// diverges from singleton placement after a restart, and during a partition
/// BOTH nodes compute themselves leader, so Traefik served both. Oldest-member
/// semantics keep exactly one active node through a partition (the non-oldest
/// side still sees the old oldest member until SBR resolves membership).
/// </summary>
public sealed class OldestNodeActiveHealthCheck : IHealthCheck
{
    private readonly ActorSystem _system;
    public OldestNodeActiveHealthCheck(ActorSystem system) => _system = system;

    public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken ct = default)
    {
        var cluster = Akka.Cluster.Cluster.Get(_system);
        return Task.FromResult(ClusterActivityEvaluator.SelfIsOldest(cluster)
            ? HealthCheckResult.Healthy("oldest Up member (singleton host)")
            : HealthCheckResult.Unhealthy("not the oldest Up member (standby)"));
    }
}
```

   - In `Program.cs`: replace `.AddTypeActivatedCheck<ActiveNodeHealthCheck>("active-node", ...)` with `.AddTypeActivatedCheck<OldestNodeActiveHealthCheck>("active-node", failureStatus: null, tags: new[] { ZbHealthTags.Active })`; change the database check's tags to `new[] { ZbHealthTags.Ready, ZbHealthTags.Active }`; update the tier-separation comment block (lines 204-208 and 346-356) to say "active-node = oldest Up member + database reachable" and remove the now-stale `ActiveNodeHealthCheck` mention. Keep the `ActiveNodeGate` registration comment (lines 252-260) in sync ("same oldest-member evaluator as OldestNodeActiveHealthCheck").
4. Run — expect PASS. Also run `--filter "FullyQualifiedName~RequiredSingletonsHealthCheckTests"` (readiness untouched) and build the solution.
5. Commit:
```
git add src/ZB.MOM.WW.ScadaBridge.Host/Health/OldestNodeActiveHealthCheck.cs src/ZB.MOM.WW.ScadaBridge.Host/Program.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HealthCheckTests.cs
git commit -m "fix(host): /health/active = oldest member + DB reachable — partition-safe Traefik routing"
```

### Task 8: Divergence proof — restarted first node must NOT become active again

**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** 11, 13, 15, 16, 17, 19, 22, 23
**Files:**
- Create: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/ActiveNodeSemanticsTests.cs`

1. Write the test using the Task 3 fixture (this is the exact scenario from the report: after the original first node restarts and rejoins, leadership returns to it by address order while the peer remains oldest — the two definitions permanently diverge):

```csharp
using ZB.MOM.WW.ScadaBridge.Host.Health;

namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;

public class ActiveNodeSemanticsTests
{
    [Fact]
    public async Task AfterOldestNodeRestarts_SurvivorStaysActive_RejoinerIsStandby()
    {
        await using var fixture = await TwoNodeClusterFixture.StartAsync();

        // Crash A (oldest), let SBR remove it, B becomes oldest.
        var victim = Akka.Cluster.Cluster.Get(fixture.NodeA).SelfAddress;
        await TwoNodeClusterFixture.CrashNode(fixture.NodeA);
        await TwoNodeClusterFixture.WaitForMemberRemoved(fixture.NodeB, victim, TimeSpan.FromSeconds(30));

        // Restart A on the same port — it rejoins as the YOUNGEST member but
        // (lowest address) may regain Akka leadership.
        var rejoined = fixture.StartNode(fixture.PortA, "Central");
        try
        {
            await TwoNodeClusterFixture.WaitForMembersUp(rejoined, 2, TimeSpan.FromSeconds(30));
            await TwoNodeClusterFixture.WaitForMembersUp(fixture.NodeB, 2, TimeSpan.FromSeconds(30));

            var clusterB = Akka.Cluster.Cluster.Get(fixture.NodeB);
            var clusterA2 = Akka.Cluster.Cluster.Get(rejoined);

            // Product active-node semantics: survivor (oldest, singleton host)
            // is active; the rejoined original node is standby — even when it
            // holds Akka leadership.
            Assert.True(ClusterActivityEvaluator.SelfIsOldest(clusterB));
            Assert.False(ClusterActivityEvaluator.SelfIsOldest(clusterA2));
        }
        finally
        {
            await rejoined.Terminate();
        }
    }
}
```

2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.IntegrationTests --filter "FullyQualifiedName~ActiveNodeSemanticsTests"` — expect PASS. Sanity: assert the OLD leader-based expression (`clusterA2.State.Leader == clusterA2.SelfAddress`) would have returned true for the rejoiner in at least one run — if it reliably does, add it as an inline comment documenting why leader-based was wrong (don't make the test depend on Akka leadership internals).
3. Commit:
```
git add tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/ActiveNodeSemanticsTests.cs
git commit -m "test(cluster): restarted original node stays standby — oldest-member active semantics proven"
```

### Task 9: CentralSingletonRegistrar helper (manager + drain task + proxy)

**Classification:** high-risk (actor lifecycle)
**Estimated implement time:** ~5 min
**Parallelizable with:** 11, 13, 15, 16, 17, 19, 22, 23
**Files:**
- Create: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/CentralSingletonRegistrar.cs`
- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CentralSingletonRegistrarTests.cs`

1. Write the failing test (single-node cluster, real CoordinatedShutdown run):

```csharp
using Akka.Actor;
using Akka.Configuration;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.Host.Actors;

namespace ZB.MOM.WW.ScadaBridge.Host.Tests;

public class CentralSingletonRegistrarTests
{
    private sealed class EchoActor : ReceiveActor
    {
        public EchoActor() => ReceiveAny(m => Sender.Tell(m));
    }

    [Fact]
    public async Task Start_CreatesManagerAndProxy_WithCanonicalNames_AndDrainTask()
    {
        using var system = CreateSingleNodeClusterSystem(); // same bootstrap as ClusterActivityEvaluatorTests
        var handle = CentralSingletonRegistrar.Start(
            system, "test-widget", Props.Create(() => new EchoActor()),
            NullLogger.Instance, drainTimeout: TimeSpan.FromSeconds(2));

        // Canonical naming preserved: <name>-singleton / <name>-proxy.
        Assert.EndsWith("/user/test-widget-singleton", handle.Manager.Path.ToString());
        Assert.EndsWith("/user/test-widget-proxy", handle.Proxy.Path.ToString());

        // Singleton answers through the proxy once the member is Up.
        var echo = await handle.Proxy.Ask<string>("hi", TimeSpan.FromSeconds(20));
        Assert.Equal("hi", echo);

        // The drain task must stop the manager during cluster-leave.
        var probeSystem = system; // watch from a test actor
        var watcher = probeSystem.ActorOf(Props.Create(() => new WatchActor(handle.Manager)));
        await Akka.Actor.CoordinatedShutdown.Get(system).Run(Akka.Actor.CoordinatedShutdown.ClrExitReason.Instance);
        // Manager terminated => drain ran (GracefulStop completed or PoisonPill fallback).
        Assert.True(handle.Manager.IsNobody() || system.WhenTerminated.IsCompleted);
    }
}
```

(Use a `WatchActor`/`TestProbe`-style Terminated assertion if `IsNobody()` proves flaky — the load-bearing assertions are the two path names and the pre-shutdown echo; the drain execution is additionally covered by the existing five singletons' behavior in Task 10's regression run.)
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~CentralSingletonRegistrarTests"` — expect FAIL (type missing).
3. Implement `CentralSingletonRegistrar`:

```csharp
using Akka.Actor;
using Akka.Cluster.Tools.Singleton;
using Microsoft.Extensions.Logging;

namespace ZB.MOM.WW.ScadaBridge.Host.Actors;

/// <summary>
/// Registers a central cluster singleton with the canonical naming scheme
/// (<c>{name}-singleton</c> / <c>{name}-proxy</c>), a PoisonPill termination
/// message, and a PhaseClusterLeave drain task that GracefulStops the manager
/// so in-flight EF work completes before handover. Extracted from five
/// copy-pasted ~60-line blocks (review 01 [Low]) whose drift left the two
/// busiest singletons (notification-outbox, audit-log-ingest) without drain
/// tasks (review 01 [Medium]).
/// </summary>
internal static class CentralSingletonRegistrar
{
    internal sealed record Handle(IActorRef Manager, IActorRef Proxy);

    internal static Handle Start(
        ActorSystem system,
        string name,
        Props singletonProps,
        ILogger logger,
        TimeSpan? drainTimeout = null)
    {
        var manager = system.ActorOf(
            ClusterSingletonManager.Props(
                singletonProps,
                PoisonPill.Instance,
                ClusterSingletonManagerSettings.Create(system).WithSingletonName(name)),
            $"{name}-singleton");

        var timeout = drainTimeout ?? TimeSpan.FromSeconds(10);
        Akka.Actor.CoordinatedShutdown.Get(system).AddTask(
            Akka.Actor.CoordinatedShutdown.PhaseClusterLeave,
            $"drain-{name}-singleton",
            async () =>
            {
                try
                {
                    await manager.GracefulStop(timeout);
                }
                catch (Exception ex)
                {
                    logger.LogWarning(ex,
                        "{Singleton} singleton did not drain within the graceful-stop timeout; "
                        + "falling through to PoisonPill handover", name);
                }
                return Akka.Done.Instance;
            });

        var proxy = system.ActorOf(
            ClusterSingletonProxy.Props(
                $"/user/{name}-singleton",
                ClusterSingletonProxySettings.Create(system).WithSingletonName(name)),
            $"{name}-proxy");

        return new Handle(manager, proxy);
    }
}
```

4. Run — expect PASS.
5. Commit:
```
git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/CentralSingletonRegistrar.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CentralSingletonRegistrarTests.cs
git commit -m "feat(host): CentralSingletonRegistrar — one registration path with drain task for central singletons"
```

### Task 10: Route all seven central singletons through the registrar (adds the missing outbox/ingest drains)

**Classification:** high-risk (actor wiring refactor)
**Estimated implement time:** ~5 min
**Parallelizable with:** 11, 13, 15, 16, 17, 19, 22, 23
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs` (lines ~395-811: notification-outbox 410-434, audit-log-ingest 451-480, site-call-audit 524-589, audit-log-purge 612-648, site-audit-reconciliation 663-700, kpi-history-recorder 722-757, pending-deployment-purge 776-811)
- Test: existing suites (regression) + `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CoordinatedShutdownTests.cs` (one new source assertion)

1. Write the failing test — extend `CoordinatedShutdownTests` in its existing source-grep style (the behavioral drain is covered by Task 9; this pins that the two hot singletons now go through the drain path):

```csharp
[Fact]
public void AllCentralSingletons_RegisterThroughRegistrarWithDrain()
{
    var hostProjectDir = FindHostProjectDirectory();
    var content = File.ReadAllText(Path.Combine(hostProjectDir!, "Actors", "AkkaHostedService.cs"));
    // Review 01 [Medium]: notification-outbox and audit-log-ingest were the
    // only central singletons WITHOUT a cluster-leave drain task.
    foreach (var name in new[] { "notification-outbox", "audit-log-ingest", "site-call-audit",
                                 "audit-log-purge", "site-audit-reconciliation",
                                 "kpi-history-recorder", "pending-deployment-purge" })
    {
        Assert.Contains($"CentralSingletonRegistrar.Start(", content);
        Assert.Contains($"\"{name}\"", content);
    }
    // No hand-rolled manager registrations remain in RegisterCentralActors.
    Assert.DoesNotContain("ClusterSingletonManager.Props(", ExtractRegisterCentralActors(content));
}
```

(Implement `ExtractRegisterCentralActors` as a substring between `RegisterCentralActors` and the next method, or simply assert `Assert.Equal(0, CountOccurrences(content, "ClusterSingletonManager.Props("))` after checking the site branch separately — the site `deployment-manager`/event-log singletons at lines 906+ use `.WithRole(...)` and stay hand-rolled; scope the assertion to avoid them, e.g. count == 2.)
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~AllCentralSingletons"` — expect FAIL.
3. Refactor `RegisterCentralActors`: replace each of the seven manager/drain/proxy blocks with

```csharp
var outbox = CentralSingletonRegistrar.Start(
    _actorSystem!, "notification-outbox",
    Props.Create(() => new ZB.MOM.WW.ScadaBridge.NotificationOutbox.NotificationOutboxActor(
        _serviceProvider, outboxOptions, outboxAuditWriter, outboxLogger)),
    _logger);
centralCommActor.Tell(new RegisterNotificationOutbox(outbox.Proxy));
commService?.SetNotificationOutbox(outbox.Proxy);
```

and equivalently for the other six (preserve every existing proxy hand-off: `RegisterAuditIngest`, `grpcServer?.SetAuditIngestActor`, `commService?.SetSiteCallAudit`, `siteCallAuditProxy.Tell(RegisterCentralCommunication…)` → `siteCallAudit.Proxy.Tell(…)`, and the log lines). Actor names and singleton names are unchanged (the registrar reproduces them exactly), so `ActorPathTests` and `RequiredSingletonsHealthCheckTests` must stay green. Preserve the comment describing the drain rationale once at the top of `RegisterCentralActors` (delete the five per-block copies and the "not added here to keep this change minimal" note at 544-549, which this task retires).
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests` (full project — ActorPath/RequiredSingletons/AuditWiring are the regression net) and `dotnet build ZB.MOM.WW.ScadaBridge.slnx` — expect PASS.
5. Commit:
```
git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CoordinatedShutdownTests.cs
git commit -m "refactor(host): central singletons via CentralSingletonRegistrar — outbox/audit-ingest gain drain tasks"
```

### Task 11: SiteHealthReportAck message + ack semantics in the communication actors

**Classification:** high-risk (actor messaging, cross-cluster contract)
**Estimated implement time:** ~5 min
**Parallelizable with:** 1, 2, 3, 4, 5, 13, 15, 16, 17, 19, 22, 23
**Files:**
- Create: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReportAck.cs`
- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteCommunicationActor.cs` (SiteHealthReport handler, lines ~393-399)
- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs` (`HandleSiteHealthReport`, lines ~360-374)
- Test: `tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/` (new file `HealthReportAckTests.cs`)

1. Write failing TestKit tests (mirror the existing `NotificationSubmit` ack tests in Communication.Tests — grep `NotificationSubmitAck` there for the harness pattern):

```csharp
[Fact]
public void SiteCommunicationActor_NoCentralClient_RepliesNotAccepted()
{
    var siteComm = Sys.ActorOf(Props.Create(() => new SiteCommunicationActor(
        "site-a", TestCommunicationOptions(), TestProbe().Ref)));
    // No RegisterCentralClient sent => _centralClient is null.
    siteComm.Tell(SampleReport(seq: 7));
    var ack = ExpectMsg<SiteHealthReportAck>();
    Assert.False(ack.Accepted);
    Assert.Equal(7, ack.SequenceNumber);
}

[Fact]
public void CentralCommunicationActor_OnReport_ProcessesAndAcks()
{
    var actor = CreateCentralCommunicationActorWithAggregator(out var aggregator);
    actor.Tell(SampleReport(seq: 3));
    var ack = ExpectMsg<SiteHealthReportAck>();
    Assert.True(ack.Accepted);
    Assert.Equal(3, ack.SequenceNumber);
    // aggregator received the report (NSubstitute: aggregator.Received().ProcessReport(...))
}
```

2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Communication.Tests --filter "FullyQualifiedName~HealthReportAckTests"` — expect FAIL (message type missing).
3. Implement:
   - Commons (additive message evolution — new type only):

```csharp
namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;

/// <summary>
/// Acknowledgement for a <see cref="SiteHealthReport"/> forwarded site→central.
/// Review 01 [Medium]: the transport was fire-and-forget, so the sender's
/// interval-counter restore logic could never observe a loss. The ack makes
/// delivery observable end-to-end (central processed the report).
/// </summary>
public sealed record SiteHealthReportAck(
    string SiteId, long SequenceNumber, bool Accepted, string? Error = null);
```

   - `SiteCommunicationActor` `Receive<SiteHealthReport>`: mirror the `NotificationSubmit` pattern exactly — when `_centralClient == null`, `Sender.Tell(new SiteHealthReportAck(msg.SiteId, msg.SequenceNumber, false, "Central ClusterClient not registered"))`; otherwise `_centralClient.Tell(new ClusterClient.Send("/user/central-communication", msg), Sender)` (forward the original Sender so the central ack routes straight back to the waiting Ask — replaces the current `Self` sender).
   - `CentralCommunicationActor.HandleSiteHealthReport`: after `ProcessLocally(report)` and the pub-sub replica publish, add `Sender.Tell(new SiteHealthReportAck(report.SiteId, report.SequenceNumber, true));` (guard with `!Sender.IsNobody()` so the peer-replica path, which arrives without an Ask, doesn't dead-letter).
4. Run — expect PASS. Run the whole Communication.Tests project.
5. Commit:
```
git add src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReportAck.cs src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteCommunicationActor.cs src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/HealthReportAckTests.cs
git commit -m "feat(comm): ack site health reports end-to-end (SiteHealthReportAck, additive contract)"
```

### Task 12: Acked async health-report transport so counter-restore actually fires

**Classification:** high-risk (changes the sender loop's failure semantics)
**Estimated implement time:** ~5 min
**Parallelizable with:** 13, 15, 16, 17, 19, 22, 23
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/IHealthReportTransport.cs`
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/AkkaHealthReportTransport.cs`
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/HealthReportSender.cs` (line 160)
- Test: `tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/HealthReportSenderTests.cs` (update fakes), `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/` (new `AkkaHealthReportTransportTests.cs`)

1. Write failing tests:
   - HealthMonitoring: update the fake transport in `HealthReportSenderTests` to the new `Task SendAsync(SiteHealthReport, CancellationToken)` signature and add/adjust:

```csharp
[Fact]
public async Task SendAsyncFailure_RestoresIntervalCounters()
{
    // The restore path (HealthReportSender.cs:158-175) was DEAD CODE against
    // the fire-and-forget production transport. With an acked async transport
    // a timeout/nack surfaces as an exception and the counters roll forward.
    var collector = NewCollectorWithErrors(scriptErrors: 4);
    var transport = new ThrowingTransport(); // SendAsync throws
    var sender = NewSender(collector, transport);
    await RunOneTick(sender);
    Assert.Equal(4, CollectorScriptErrorCount(collector)); // restored, not lost
}
```

   - Host: `AkkaHealthReportTransportTests` — TestKit: place a probe-backed actor at `/user/site-communication` that replies `SiteHealthReportAck(Accepted: true)`; assert `SendAsync` completes. Second test: reply `Accepted: false` → assert `SendAsync` throws `InvalidOperationException`. Third: no actor at the path → assert it throws (Ask timeout; use a short timeout via options or constant).
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests --filter "FullyQualifiedName~HealthReportSenderTests"` — expect FAIL (compile).
3. Implement:
   - `IHealthReportTransport`: replace `void Send(SiteHealthReport report);` with `Task SendAsync(SiteHealthReport report, CancellationToken cancellationToken);` (update xmldoc: "must THROW on non-delivery — the sender's counter-restore depends on it").
   - `AkkaHealthReportTransport`:

```csharp
public async Task SendAsync(SiteHealthReport report, CancellationToken cancellationToken)
{
    var actorSystem = _akkaService.ActorSystem
        ?? throw new InvalidOperationException("Actor system not started — health report not sent");
    var siteComm = actorSystem.ActorSelection("/user/site-communication");
    var ack = await siteComm.Ask<SiteHealthReportAck>(report, AckTimeout, cancellationToken);
    if (!ack.Accepted)
        throw new InvalidOperationException($"Health report #{report.SequenceNumber} rejected: {ack.Error}");
}
private static readonly TimeSpan AckTimeout = TimeSpan.FromSeconds(10);
```

   - `HealthReportSender` line 160: `_transport.Send(reportWithSeq);` → `await _transport.SendAsync(reportWithSeq, stoppingToken).ConfigureAwait(false);` (the surrounding try/catch + restore stays exactly as-is — it now actually fires). Trim the comment at 142-152 to note the transport is now acked.
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests` and `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~AkkaHealthReportTransportTests"` — PASS. Build the solution (interface change ripples to any other fake).
5. Commit:
```
git add src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/IHealthReportTransport.cs src/ZB.MOM.WW.ScadaBridge.Host/AkkaHealthReportTransport.cs src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/HealthReportSender.cs tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests tests/ZB.MOM.WW.ScadaBridge.Host.Tests/AkkaHealthReportTransportTests.cs
git commit -m "fix(health): acked SendAsync transport — report-loss counter restore is live, not dead code"
```

### Task 13: Metrics-staleness signal + status-transition timestamps in the aggregator

**Classification:** high-risk (lock-free CAS state machine)
**Estimated implement time:** ~5 min
**Parallelizable with:** 1, 2, 3, 4, 5, 11, 15, 16, 17, 19, 22, 23
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthState.cs`
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthAggregator.cs` (ProcessReport, CheckForOfflineSites)
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/HealthMonitoringOptions.cs` + `HealthMonitoringOptionsValidator.cs`
- Modify: `docs/requirements/Component-HealthMonitoring.md` (offline definition, lines ~43-46 + option table)
- Test: `tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/CentralHealthAggregatorTests.cs`, `HealthMonitoringOptionsValidatorTests.cs`

1. Write failing tests in `CentralHealthAggregatorTests` (use the existing `FakeTimeProvider`/manual-time pattern already in that file):

```csharp
[Fact]
public void HeartbeatingSiteWithStaleReports_IsFlaggedMetricsStale_ButStaysOnline()
{
    // Review 01 [Medium]: a site whose HealthReportSender died keeps
    // heartbeating and previously showed "online with frozen metrics forever".
    var (aggregator, time) = NewAggregator(metricsStaleTimeout: TimeSpan.FromMinutes(2));
    aggregator.ProcessReport(Report("site-a", seq: 1));
    time.Advance(TimeSpan.FromMinutes(3));
    aggregator.MarkHeartbeat("site-a", time.GetUtcNow()); // heartbeats keep flowing
    aggregator.CheckForOfflineSites();
    var state = aggregator.GetSiteState("site-a")!;
    Assert.True(state.IsOnline);        // heartbeat-based liveness unchanged
    Assert.True(state.IsMetricsStale);  // NEW distinct signal
}

[Fact]
public void FreshReport_ClearsMetricsStale()
{
    var (aggregator, time) = NewAggregator(metricsStaleTimeout: TimeSpan.FromMinutes(2));
    aggregator.ProcessReport(Report("site-a", seq: 1));
    time.Advance(TimeSpan.FromMinutes(3));
    aggregator.CheckForOfflineSites();
    aggregator.ProcessReport(Report("site-a", seq: 2));
    Assert.False(aggregator.GetSiteState("site-a")!.IsMetricsStale);
}

[Fact]
public void OnlineOfflineTransitions_RecordLastStatusChangeAt()
{
    // Review 01 underdeveloped #7: "when did the site drop" was unanswerable.
    var (aggregator, time) = NewAggregator();
    aggregator.ProcessReport(Report("site-a", seq: 1));
    time.Advance(TimeSpan.FromMinutes(5)); // > OfflineTimeout
    aggregator.CheckForOfflineSites();
    var offlineAt = aggregator.GetSiteState("site-a")!.LastStatusChangeAt;
    Assert.Equal(time.GetUtcNow(), offlineAt);
    aggregator.MarkHeartbeat("site-a", time.GetUtcNow());
    Assert.NotEqual(offlineAt, aggregator.GetSiteState("site-a")!.LastStatusChangeAt);
}
```

Validator test: `MetricsStaleTimeout` must be positive and ≥ `ReportInterval` (a stale window shorter than one report interval flags every site).
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests --filter "FullyQualifiedName~CentralHealthAggregatorTests"` — expect FAIL.
3. Implement:
   - `SiteHealthState`: add `public bool IsMetricsStale { get; init; }` and `public DateTimeOffset? LastStatusChangeAt { get; init; }` (xmldoc each: stale = online but no *report* within MetricsStaleTimeout; LastStatusChangeAt = last online↔offline flip).
   - `HealthMonitoringOptions`: add `public TimeSpan MetricsStaleTimeout { get; set; } = TimeSpan.FromMinutes(2);` — validator rules as above.
   - `CentralHealthAggregator.ProcessReport`: in both the register and update branches set `IsMetricsStale = false`; in the update branch set `LastStatusChangeAt = existing.IsOnline ? existing.LastStatusChangeAt : now`. `MarkHeartbeat`: on the offline→online promotion set `LastStatusChangeAt = now` (leave `IsMetricsStale` untouched — heartbeats say nothing about metrics).
   - `CheckForOfflineSites`: when marking offline, `offline = state with { IsOnline = false, LastStatusChangeAt = now }`. Add a second pass (same loop) for online sites: if `state.LastReportReceivedAt is { } lr && now - lr > _options.MetricsStaleTimeout && !state.IsMetricsStale`, CAS to `state with { IsMetricsStale = true }` and `LogWarning("Site {SiteId} metrics are stale — online (heartbeats) but no report for {Elapsed}s", ...)`. CAS-loss handling identical to the offline swap (lose ⇒ fresh data arrived ⇒ correct to skip).
   - `Component-HealthMonitoring.md` lines ~43-46: replace "offline = no report within 60s" with the two-signal model: **liveness** = no heartbeat/signal within `OfflineTimeout` (60s) → offline; **metrics staleness** = online but no report within `MetricsStaleTimeout` (2m) → "metrics stale" flag. Document `LastStatusChangeAt`. This closes the spec/code disagreement the report flagged.
4. Run the HealthMonitoring.Tests project — PASS.
5. Commit:
```
git add src/ZB.MOM.WW.ScadaBridge.HealthMonitoring tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests docs/requirements/Component-HealthMonitoring.md
git commit -m "feat(health): metrics-stale signal + status-transition timestamps; spec now matches heartbeat-liveness code"
```

### Task 14: Surface metrics-stale + offline-since on the Health dashboard

**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** 15, 16, 17, 19, 22, 23
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor`
- Test: `tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/` (bUnit — grep for an existing Health page test class first: `grep -rn "Health" tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests --include="*.cs" -l`; extend it if present, else create `HealthPageStalenessTests.cs` following the nearest page-test pattern)

1. Write the failing bUnit test: render the page with a site state where `IsMetricsStale = true` and assert a `.badge` (existing Bootstrap badge classes used on the page) containing text `Metrics stale` is rendered; second case: `IsOnline = false, LastStatusChangeAt = X` renders `offline since` text containing the formatted timestamp.
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests --filter "FullyQualifiedName~HealthPageStaleness"` — expect FAIL.
3. Implement in `Health.razor`, on the site card/row where the online/offline badge already renders:

```razor
@if (state.IsOnline && state.IsMetricsStale)
{
    <span class="badge bg-warning text-dark ms-1"
          title="Site is heartbeating but has sent no health report for over @StaleTimeoutDisplay — the metrics pipeline may be dead.">
        Metrics stale
    </span>
}
@if (!state.IsOnline && state.LastStatusChangeAt is { } changedAt)
{
    <small class="text-muted ms-1">offline since @changedAt.ToString("u")</small>
}
```

(Match the page's existing badge markup/classes — clean corporate Bootstrap, no new components.)
4. Run — PASS.
5. Commit:
```
git add src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests
git commit -m "feat(ui): health dashboard shows metrics-stale badge and offline-since timestamp"
```

### Task 15: Fix the metrics-port seed in appsettings.Site.json + close the validator gap

**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** 1, 2, 3, 4, 5, 11, 13, 16, 17, 19, 22, 23
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json` (lines 13-17)
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/StartupValidator.cs` (seed loop, lines 117-127)
- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/StartupValidatorTests.cs`

1. Write the failing test (mirror the existing seed-vs-GrpcPort test in `StartupValidatorTests`):

```csharp
[Fact]
public void Validate_SeedNodeTargetingMetricsPort_Fails()
{
    // Review 01 [Medium]: appsettings.Site.json shipped a seed pointing at the
    // Kestrel HTTP/1.1 metrics listener (8084) — a doomed Akka.Remote
    // association. The validator guarded GrpcPort but not MetricsPort.
    var config = SiteConfigWith(
        metricsPort: 8084,
        seedNodes: new[] { "akka.tcp://scadabridge@localhost:8082", "akka.tcp://scadabridge@localhost:8084" });
    var ex = Assert.Throws<StartupValidationException>(() => StartupValidator.Validate(config));
    Assert.Contains("must not target the metrics port", ex.Message);
}
```

(Use the existing config-building helper in that test file; match the actual exception type/validation surface used by the current seed-vs-grpc test.)
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~SeedNodeTargetingMetricsPort"` — expect FAIL.
3. Implement:
   - `StartupValidator.cs`: inside the existing `foreach (var seed in seedNodes ...)` loop add:

```csharp
p.Require("ScadaBridge:Cluster:SeedNodes",
    _ => SeedNodePort(seed) != metricsPort,
    $"entry '{seed}' must not target the metrics port " +
    $"({metricsPort}); seed nodes must reference Akka remoting ports");
```

   - `appsettings.Site.json`: change the second seed `"akka.tcp://scadabridge@localhost:8084"` → `"akka.tcp://scadabridge@localhost:8085"` and add a `"_seedNodes"` comment key: `"Host-0xx: second entry is the FUTURE node-b remoting port (8085) for a two-node localhost site. It must be an Akka remoting endpoint — never this node's GrpcPort (8083) or MetricsPort (8084); StartupValidator rejects both."`
   - Sanity-check the docker per-node site configs don't carry the same defect: `grep -n "SeedNodes" -A3 docker/site-*/appsettings.Site.json` (they seed container-host:8082 pairs — expected clean; fix identically if not).
4. Run — PASS. Run the full `StartupValidatorTests` class.
5. Commit:
```
git add src/ZB.MOM.WW.ScadaBridge.Host/StartupValidator.cs src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json tests/ZB.MOM.WW.ScadaBridge.Host.Tests/StartupValidatorTests.cs
git commit -m "fix(host): dev site seed no longer targets the metrics port; validator rejects seed-vs-MetricsPort"
```

### Task 16: Single-node installs — explicit AllowSingleNodeCluster instead of phantom seeds

**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** 1, 2, 3, 4, 5, 11, 13, 15, 17, 19, 22, 23
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs`
- Modify: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptionsValidator.cs` (lines 24-34)
- Modify: `deploy/wonder-app-vd03/appsettings.Central.json` (Cluster section, lines 9-19)
- Test: `tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsValidatorTests.cs`

1. Write failing tests:

```csharp
[Fact]
public void SingleSeed_WithoutAcknowledgement_Fails()
{
    var options = ValidOptions() with-like-mutation: SeedNodes = ["akka.tcp://scadabridge@h:8081"];
    AssertInvalid(options, "AllowSingleNodeCluster");
}

[Fact]
public void SingleSeed_WithAllowSingleNodeCluster_Passes()
{
    // Review 01 [Low]: the shipped single-node artifact satisfied the 2-seed
    // rule with a phantom seed the node dials forever. An explicit
    // acknowledgement flag replaces the fiction.
    var options = ValidOptions(); options.SeedNodes = new List<string> { "akka.tcp://scadabridge@h:8081" };
    options.AllowSingleNodeCluster = true;
    AssertValid(options);
}

[Fact]
public void EmptySeeds_FailsEvenWithFlag() { /* zero seeds always invalid */ }
```

(Adapt to the existing `ClusterOptionsValidatorTests` helper style.)
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests --filter "FullyQualifiedName~SingleSeed"` — expect FAIL.
3. Implement:
   - `ClusterOptions`: add `public bool AllowSingleNodeCluster { get; set; }` with xmldoc: "Acknowledges a deliberate single-node deployment: permits exactly one seed node (normally ≥2 are required so either node can start first). Without this flag a single-node install must list a phantom second seed, which the node dials forever — log noise and a defeated validation intent (review 01)."
   - `ClusterOptionsValidator`: replace the `>= 2` rule with:

```csharp
var minSeeds = options.AllowSingleNodeCluster ? 1 : 2;
builder.RequireThat(options.SeedNodes is not null && options.SeedNodes.Count >= minSeeds,
    options.AllowSingleNodeCluster
        ? "ClusterOptions.SeedNodes must contain at least 1 seed node."
        : "ClusterOptions.SeedNodes must contain at least 2 seed nodes "
          + "(Component-ClusterInfrastructure.md → Node Configuration: both nodes are seed nodes); "
          + "for a deliberate single-node install set ClusterOptions.AllowSingleNodeCluster = true instead of listing a phantom seed.");
```

   - `deploy/wonder-app-vd03/appsettings.Central.json`: remove the `localhost:8091` phantom seed, add `"AllowSingleNodeCluster": true`, and rewrite `_comment_Cluster` to explain the flag ("single-node install acknowledged; add node B's real seed and remove the flag when it exists"). Validate JSON: `python3 -m json.tool deploy/wonder-app-vd03/appsettings.Central.json > /dev/null`.
4. Run the ClusterInfrastructure.Tests project — PASS.
5. Commit:
```
git add src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests deploy/wonder-app-vd03/appsettings.Central.json
git commit -m "feat(cluster): AllowSingleNodeCluster flag — wonder-app-vd03 drops its phantom seed"
```

### Task 17: stop_grace_period for all app containers (both topologies)

**Classification:** trivial
**Estimated implement time:** ~3 min
**Parallelizable with:** everything except 21
**Files:**
- Modify: `docker/docker-compose.yml` (every `scadabridge:latest` service)
- Modify: `docker-env2/docker-compose.yml` (every `scadabridge:latest` service)

1. No unit test (compose config). Verification command is the "test": `docker compose -f docker/docker-compose.yml config | grep -c "stop_grace_period"` — expect 0 before, N (= number of app services, 8 in docker/, 4 in docker-env2/) after.
2. Add to each app service (central-a/b, all site nodes — NOT the traefik service, which has no CoordinatedShutdown):

```yaml
    # CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
    # actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
    # mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
    stop_grace_period: 30s
```

3. Validate: `docker compose -f docker/docker-compose.yml config > /dev/null && docker compose -f docker-env2/docker-compose.yml config > /dev/null` (syntax), then the grep from step 1 returns 8 and 4.
4. Commit:
```
git add docker/docker-compose.yml docker-env2/docker-compose.yml
git commit -m "fix(docker): 30s stop_grace_period so graceful redeploys aren't SIGKILLed mid-CoordinatedShutdown"
```

### Task 18: Evict deleted sites from the health aggregator

**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** 15, 16, 17, 19, 22, 23
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ICentralHealthAggregator.cs`
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthAggregator.cs`
- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs` (the periodic site-address refresh handler — locate with `grep -n "Refresh" src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs`; it reloads sites from the repository every 60s and on admin changes)
- Test: `tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/CentralHealthAggregatorTests.cs`, `tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/`

1. Write the failing aggregator test:

```csharp
[Fact]
public void PruneUnknownSites_RemovesDeletedSites_KeepsKnownAndCentral()
{
    // Review 01 [Low]: _siteStates only grew — a deleted site remained a
    // permanently-offline dashboard tile (and a KPI sample source) forever.
    var (aggregator, time) = NewAggregator();
    aggregator.ProcessReport(Report("site-a", 1));
    aggregator.ProcessReport(Report("site-b", 1));
    aggregator.ProcessReport(Report(CentralHealthReportLoop.CentralSiteId, 1));

    aggregator.PruneUnknownSites(new[] { "site-a" });

    Assert.NotNull(aggregator.GetSiteState("site-a"));
    Assert.Null(aggregator.GetSiteState("site-b"));
    Assert.NotNull(aggregator.GetSiteState(CentralHealthReportLoop.CentralSiteId)); // synthetic id always kept
}
```

2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests --filter "FullyQualifiedName~PruneUnknownSites"` — expect FAIL.
3. Implement:
   - Interface + aggregator:

```csharp
/// <summary>Removes tracked sites not in <paramref name="knownSiteIds"/> (deleted from configuration). The synthetic central id is always retained. Self-healing: called from the CentralCommunicationActor's periodic site refresh, so a deleted site disappears within one refresh interval without needing a deletion event.</summary>
public void PruneUnknownSites(IReadOnlyCollection<string> knownSiteIds)
{
    var known = new HashSet<string>(knownSiteIds, StringComparer.Ordinal);
    foreach (var siteId in _siteStates.Keys)
    {
        if (siteId == CentralHealthReportLoop.CentralSiteId || known.Contains(siteId))
            continue;
        if (_siteStates.TryRemove(siteId, out _))
            _logger.LogInformation("Site {SiteId} evicted from health aggregator (no longer configured)", siteId);
    }
}
```

   - In `CentralCommunicationActor`'s site-refresh handler, after the fresh site list is loaded: `_serviceProvider.GetService<ICentralHealthAggregator>()?.PruneUnknownSites(sites.Select(s => s.SiteIdentifier).ToList());` (match the identifier the aggregator is keyed on — it's the `SiteId` string sites report with; confirm by checking what `HandleSiteHealthReport` receives, and use the same property the refresh handler already uses for client-keying).
   - Add a Communication.Tests TestKit test: trigger the refresh message with a repo/enumerator fake returning `["site-a"]` and an NSubstitute aggregator; assert `aggregator.Received().PruneUnknownSites(...)`.
4. Run both test projects — PASS.
5. Commit:
```
git add src/ZB.MOM.WW.ScadaBridge.HealthMonitoring src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests tests/ZB.MOM.WW.ScadaBridge.Communication.Tests
git commit -m "fix(health): evict deleted sites from the aggregator on the periodic site refresh"
```

### Task 19: Rate-limit dead-letter warnings

**Classification:** high-risk (actor change — timers)
**Estimated implement time:** ~5 min
**Parallelizable with:** 1, 2, 3, 4, 5, 11, 13, 15, 16, 17, 22, 23
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/DeadLetterMonitorActor.cs`
- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/DeadLetterMonitorTests.cs`

1. Write the failing test (extend `DeadLetterMonitorTests`; use an NSubstitute `ILogger<DeadLetterMonitorActor>` and count `Log` invocations at `LogLevel.Warning`):

```csharp
[Fact]
public void DeadLetterStorm_LogsAtMostWindowLimit_ButCountsAll()
{
    // Review 01 [Low]: a failover/undeploy storm produced one Warning per
    // dead letter — unbounded log flood. Health metric counting is untouched.
    var logger = Substitute.For<ILogger<DeadLetterMonitorActor>>();
    var monitor = Sys.ActorOf(Props.Create(() => new DeadLetterMonitorActor(logger, null)));
    for (var i = 0; i < 50; i++)
        monitor.Tell(new DeadLetter($"m{i}", TestActor, TestActor));

    var count = monitor.Ask<DeadLetterCountResponse>(GetDeadLetterCount.Instance).Result;
    Assert.Equal(50, count.Count); // every dead letter still counted
    var warnings = logger.ReceivedCalls().Count(c =>
        c.GetMethodInfo().Name == "Log" && (LogLevel)c.GetArguments()[0]! == LogLevel.Warning);
    Assert.True(warnings <= DeadLetterMonitorActor.MaxWarningsPerWindow,
        $"expected <= {DeadLetterMonitorActor.MaxWarningsPerWindow} warnings, got {warnings}");
}
```

2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~DeadLetterStorm"` — expect FAIL (50 warnings).
3. Implement: make the actor `IWithTimers`; add `public const int MaxWarningsPerWindow = 10;` and `internal static readonly TimeSpan WarningWindow = TimeSpan.FromMinutes(1);` In the `DeadLetter` handler:

```csharp
_deadLetterCount++;
_healthCollector?.IncrementDeadLetter();
if (_warningsThisWindow < MaxWarningsPerWindow)
{
    _warningsThisWindow++;
    logger.LogWarning("Dead letter: {MessageType} from {Sender} to {Recipient}",
        dl.Message.GetType().Name, dl.Sender, dl.Recipient);
    if (_warningsThisWindow == MaxWarningsPerWindow)
        logger.LogWarning("Dead-letter warning limit ({Limit}) reached — suppressing per-letter warnings for {Window}",
            MaxWarningsPerWindow, WarningWindow);
}
else
{
    _suppressedThisWindow++;
}
```

Plus a private `WindowTick` timer message started in `PreStart` (`Timers.StartPeriodicTimer("dl-window", WindowTick.Instance, WarningWindow)`) whose handler logs one summary if `_suppressedThisWindow > 0` (`LogWarning("Suppressed {Count} dead letters in the last {Window}", ...)`) and resets both window counters. Update the class xmldoc.
4. Run — PASS (the summary tick fires only after 1 min, so the test sees ≤ 11 warnings; assert against `MaxWarningsPerWindow + 1` if the limit-reached notice counts).
5. Commit:
```
git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/DeadLetterMonitorActor.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/DeadLetterMonitorTests.cs
git commit -m "fix(host): rate-limit dead-letter warnings (10/min + suppression summary); metric counting unchanged"
```

### Task 20: Down-if-alone recovery story — exit on unexpected ActorSystem termination + service restart actions

**Classification:** high-risk (process lifecycle)
**Estimated implement time:** ~5 min
**Parallelizable with:** 15, 16, 22, 23
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs` (ctor DI + StartAsync/GetOrCreateActorSystem + StopAsync flag)
- Modify: `deploy/wonder-app-vd03/install.ps1` (service recovery actions)
- Modify: `docs/requirements/Component-ClusterInfrastructure.md` (new "Down-if-alone recovery" subsection near the split-brain section, ~lines 110-119)
- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/` (new `UnexpectedTerminationTests.cs`)

Context: with SBR enabled (Task 1), `keep-oldest down-if-alone = on` means a node that finds itself alone-and-partitioned downs ITSELF, and `run-coordinated-shutdown-when-down = on` (now live) terminates its ActorSystem. Today the Host process would keep running with a dead actor system — serving nothing, restarted by nobody. The recovery contract: unexpected ActorSystem termination ⇒ stop the host process ⇒ the supervisor (docker `restart: unless-stopped` / Windows service recovery) restarts it ⇒ it rejoins as a fresh incarnation.

1. Write the failing test:

```csharp
[Fact]
public async Task ActorSystemTerminatedOutsideStopAsync_StopsHostApplication()
{
    // Down-if-alone self-down (SBR) terminates the ActorSystem via
    // run-coordinated-shutdown-when-down. The process must exit so the
    // service supervisor restarts it — otherwise the node idles forever
    // (review 01 underdeveloped #2).
    var lifetime = Substitute.For<IHostApplicationLifetime>();
    var service = CreateAkkaHostedService(lifetime); // existing Host.Tests construction pattern
    await service.StartAsync(CancellationToken.None);
    var system = service.ActorSystem!;

    await system.Terminate(); // simulated self-down, NOT StopAsync

    await Task.Delay(500);
    lifetime.Received(1).StopApplication();
}

[Fact]
public async Task StopAsync_DoesNotTriggerStopApplication()
{
    var lifetime = Substitute.For<IHostApplicationLifetime>();
    var service = CreateAkkaHostedService(lifetime);
    await service.StartAsync(CancellationToken.None);
    await service.StopAsync(CancellationToken.None);
    lifetime.DidNotReceive().StopApplication();
}
```

(Reuse whatever construction pattern `AkkaHostedServiceAuditWiringTests`/`HostStartupTests` use for building the service with a test DI container; add `IHostApplicationLifetime` as an OPTIONAL ctor parameter — nullable, resolved via `GetService` — so every existing construction site keeps compiling.)
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~UnexpectedTermination"` — expect FAIL.
3. Implement:
   - `AkkaHostedService`: add `private volatile bool _stopRequested;` set to `true` first thing in `StopAsync`. After the system is created and published (`_actorSystem = system;` in `GetOrCreateActorSystem`), register once:

```csharp
system.WhenTerminated.ContinueWith(_ =>
{
    if (_stopRequested) return;
    _logger.LogCritical(
        "ActorSystem terminated outside host shutdown (SBR down / run-coordinated-shutdown-when-down). "
        + "Stopping the host process so the service supervisor restarts this node as a fresh incarnation.");
    _appLifetime?.StopApplication();
}, TaskContinuationOptions.ExecuteSynchronously);
```

   - `install.ps1`: after the `sc.exe create`/`New-Service` call, add recovery actions (find the actual service name in the script first):

```powershell
# Restart-on-failure: required for the SBR down-if-alone recovery contract —
# a self-downed node exits the process and MUST be restarted externally.
& sc.exe failure $ServiceName reset= 86400 actions= restart/5000/restart/30000/restart/60000
& sc.exe failureflag $ServiceName 1
```

   - `Component-ClusterInfrastructure.md`: add a "Down-if-alone recovery" subsection: self-down ⇒ CoordinatedShutdown ⇒ process exit ⇒ supervisor restart (docker `restart: unless-stopped`; Windows service recovery actions) ⇒ clean rejoin as a new incarnation; note the drill in Task 21's script.
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests` — PASS (full project; the new ctor param must not break existing tests).
5. Commit:
```
git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs deploy/wonder-app-vd03/install.ps1 docs/requirements/Component-ClusterInfrastructure.md tests/ZB.MOM.WW.ScadaBridge.Host.Tests/UnexpectedTerminationTests.cs
git commit -m "feat(host): exit process on unexpected ActorSystem termination — completes the down-if-alone recovery loop"
```

### Task 21: Docker failover drill script (the manual/CI kill test against the real topology)

**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** 22, 23
**Files:**
- Create: `docker/failover-drill.sh`
- Modify: `docker/README.md` (new "Failover drill" section), `deployments/docker-cluster.md` (cross-reference)

1. No unit test — the script IS the test. Create:

```bash
#!/usr/bin/env bash
# Failover drill against the running docker cluster (bash docker/deploy.sh first).
# Kills the ACTIVE central node (docker kill = SIGKILL, the hard-crash path that
# review 01 found had NO automatic recovery before the SBR downing provider was
# enabled) and measures how long Traefik takes to route to the survivor.
set -euo pipefail

TRAEFIK_URL="${TRAEFIK_URL:-http://localhost:9000}"
TIMEOUT_S="${TIMEOUT_S:-90}"

active_container() {
  if curl -sf -o /dev/null "http://localhost:9001/health/active"; then echo scadabridge-central-a
  elif curl -sf -o /dev/null "http://localhost:9002/health/active"; then echo scadabridge-central-b
  else echo "ERROR: no active central node found" >&2; exit 1; fi
}

VICTIM=$(active_container)
echo "Active central node: ${VICTIM} — killing it (SIGKILL, crash path)"
docker kill "${VICTIM}" > /dev/null
START=$(date +%s)

echo "Waiting for the survivor to become active through Traefik (${TRAEFIK_URL}/health/active)..."
while true; do
  ELAPSED=$(( $(date +%s) - START ))
  if curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; then
    echo "PASS: failover complete in ${ELAPSED}s (design budget ~25s + Traefik health interval)"
    break
  fi
  if (( ELAPSED > TIMEOUT_S )); then
    echo "FAIL: no active central node after ${ELAPSED}s — SBR/singleton handover did not recover" >&2
    docker start "${VICTIM}" > /dev/null
    exit 1
  fi
  sleep 1
done

SURVIVOR=$([ "${VICTIM}" = scadabridge-central-a ] && echo scadabridge-central-b || echo scadabridge-central-a)
echo "Survivor singleton evidence (last 20 matching log lines from ${SURVIVOR}):"
docker logs "${SURVIVOR}" 2>&1 | grep -Ei "singleton|oldest|Downing|Removed" | tail -20 || true

echo "Restarting ${VICTIM} and waiting for it to rejoin..."
docker start "${VICTIM}" > /dev/null
sleep 5
echo "Drill complete. Verify on the Health dashboard that both nodes show Up and the survivor is Primary."
```

2. `chmod +x docker/failover-drill.sh`. Verify statically: `bash -n docker/failover-drill.sh` (syntax). If a cluster is running (`bash docker/deploy.sh`), run the drill end-to-end and record the observed failover time in `docker/README.md`; if not available in this session, mark the README section "run after next deploy" — the script is still committed.
3. Document in `docker/README.md`: what it does, the ~25s + Traefik 5s health-interval expectation, and that it exercises S1 (SBR downing), S3 (single-active through Traefik) and Task 20 (restart/rejoin). Cross-reference from `deployments/docker-cluster.md`. Note for other plans: plan 02's duplicate-delivery checks can extend this script.
4. Commit:
```
git add docker/failover-drill.sh docker/README.md deployments/docker-cluster.md
git commit -m "test(docker): failover drill script — SIGKILL the active central node, assert Traefik recovery"
```

### Task 22: Reconcile REQ-HOST-6 (Akka.Hosting) with the hand-rolled bootstrap

**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** 15, 16, 17, 19, 21, 23
**Files:**
- Modify: `docs/requirements/Component-Host.md` (REQ-HOST-6, ~lines 106-114)
- Modify (conditional): `src/ZB.MOM.WW.ScadaBridge.Host/ZB.MOM.WW.ScadaBridge.Host.csproj` (lines 15-18) + `Directory.Packages.props`

Decision: keep the hand-rolled HOCON (it is hardened — `QuoteHocon`/`DurationHocon`, validated options — and now carries the downing provider explicitly per Task 1); amend the spec instead of a risky bootstrap migration. The report accepts either resolution.

1. Verify actual package usage: `grep -rn "using Akka.Hosting\|using Akka.Cluster.Hosting\|using Akka.Remote.Hosting\|AkkaConfigurationBuilder\|\.WithClustering\|\.WithRemoting" src/ --include="*.cs"` — expect zero hits (review found them referenced-but-unused).
2. If zero hits: remove `<PackageReference Include="Akka.Cluster.Hosting" />`, `<PackageReference Include="Akka.Hosting" />`, `<PackageReference Include="Akka.Remote.Hosting" />` from the Host csproj (keep `Akka.Cluster.Tools`), remove the matching `<PackageVersion>` lines from `Directory.Packages.props` **only if no other project references them** (`grep -rn "Akka.Hosting\|Akka.Cluster.Hosting\|Akka.Remote.Hosting" src/*/[A-Z]*.csproj tests/*/[A-Z]*.csproj`), and check whether the csproj's OpenTelemetry.Api transitive-override comment (lines ~28-32) still applies — if it referenced Akka.Hosting's pin, delete the override too.
3. Amend `Component-Host.md` REQ-HOST-6: replace "bootstrap via Akka.Hosting" with: bootstrap is a hand-assembled, injection-safe HOCON document (`AkkaHostedService.BuildHocon`) + `ActorSystem.Create`; the downing provider (`Akka.Cluster.SBR.SplitBrainResolverProvider`) is set explicitly — Akka's default is NoDowning and forgetting this line silently disables failover (this exact omission was review 01's Critical finding, caught because tests now assert the parsed key and a two-node kill test asserts the behavior). Note the deliberate rejection of Akka.Cluster.Hosting typed options with a one-line rationale (single hardened code path already in production shape; migration tracked as possible future work).
4. Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx` — expect success. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~HoconBuilderTests"` — green.
5. Commit:
```
git add docs/requirements/Component-Host.md src/ZB.MOM.WW.ScadaBridge.Host/ZB.MOM.WW.ScadaBridge.Host.csproj Directory.Packages.props
git commit -m "docs(host): REQ-HOST-6 documents the hand-rolled HOCON bootstrap; drop unused Akka.Hosting packages"
```

### Task 23: Docs & deploy-config cleanup batch (Traefik doc, NodeName overlays, TLS roadmap note)

**Classification:** trivial
**Estimated implement time:** ~4 min
**Parallelizable with:** 15, 16, 17, 19, 21, 22
**Files:**
- Modify: `docs/requirements/Component-TraefikProxy.md` (lines ~115, ~121, TLS section ~131-138)
- Modify: `deploy/wonder-app-vd03/appsettings.Central.json` (Node section), `deploy/wonder-app-vd03/appsettings.Site.json` (Node section, if the file exists — check `ls deploy/wonder-app-vd03/`)

1. `Component-TraefikProxy.md`:
   - Line ~121: replace the stale `src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeHealthCheck.cs` location with `src/ZB.MOM.WW.ScadaBridge.Host/Health/OldestNodeActiveHealthCheck.cs` (the Host-local oldest-member check from Task 7) and note `/health/active` also requires the database check.
   - Line ~115: update the `/health/ready` description from "database + Akka cluster" to "database + akka-cluster + required-singletons" (matches Program.cs and Component-Host REQ-HOST-4a).
   - TLS section (~131-138): append an explicit roadmap paragraph — "**Production TLS profile: not yet implemented.** Traefik terminates plain HTTP, Akka remoting is unencrypted TCP, and site gRPC is `http://`. Acceptable for the lab topology only; a production deployment requires TLS at all three layers and a secured Traefik dashboard. Tracked as an open roadmap item (arch review 01, underdeveloped area 3)." — this makes the accepted gap spec-visible without pretending to fix it.
2. `deploy/wonder-app-vd03/appsettings.Central.json`: add `"NodeName": "central-a"` to the `Node` section (per the base config's documented convention — without it every audit row from the one real deployment gets a NULL `SourceNode`, undermining `IX_AuditLog_Node_Occurred` and the per-node stuck KPIs). Same for the Site overlay if present (`"NodeName": "node-a"`). Validate: `python3 -m json.tool deploy/wonder-app-vd03/appsettings.Central.json > /dev/null`.
3. Cross-check no other stale `ActiveNodeHealthCheck` doc references remain: `grep -rn "ActiveNodeHealthCheck" docs/ README.md` — fix any found the same way.
4. Commit:
```
git add docs/requirements/Component-TraefikProxy.md deploy/wonder-app-vd03/
git commit -m "docs(cleanup): sync Traefik doc with shared health checks, add TLS roadmap note, NodeName in deploy overlays"
```

---

## Dependencies on other plans

- **Plan 02 (Communication & S&F) depends on THIS plan** for two facilities — design them exactly as specified here so the seam holds:
  - The canonical site-side "active node / singleton host" check: after Task 6, `IClusterNodeProvider.SelfIsPrimary` (site DI registration at `SiteServiceRegistration.cs:101-107`) means "oldest Up member in the site role = the node hosting the DeploymentManager singleton". Plan 02's standby S&F delivery gating must consume this (directly or via a registered delegate mirroring `SiteEventLogActiveNodeCheck`), not reinvent a leader check.
  - The two-node failover rig: `TwoNodeClusterFixture` (Task 3) is public and production-HOCON-based; plan 02's ClusterClient-recreate and S&F non-duplication tests should build on it, and the docker drill script (Task 21) is the shared live-cluster harness.
- **Plan 07 (UI/Management/Security) owns** the `DisableLogin: true` deploy artifact (C2) and the deploy-artifact LDAP plaintext posture (C3) — this plan touches `deploy/wonder-app-vd03/appsettings.Central.json` in Tasks 16 and 23 (Cluster + Node sections only); coordinate merges but do not touch the `Security` section here.
- No other plan blocks this one.

## Execution order

**P0 (do first, in order):** Task 1 (the Critical one-liner) → Task 3 (fixture) → Task 4 (behavioral proof). Ship these three even if nothing else lands — they convert "failover does not work" into "failover works and is regression-tested".

**Critical path:** 1 → 3 → 4 → 8 (needs 6), with 5 → 6 → 7 as the second spine (active-node unification). Task 2 must precede 9/10 (drain budget before drain refactor); 11 → 12 (message before transport); 13 → 14 and 13 → 18 (state shape before consumers). Task 20 touches `AkkaHostedService.cs` and should follow 10 to avoid merge churn; Task 21 follows 1 (drill is meaningless before SBR) and ideally 17.

**Freely parallel at any point:** 15, 16, 17, 19, 22, 23.