dohertj2/ScadaBridge

Fork 0

Files

T

Joseph Doherty c0c89a4954 docs(archreview): add architecture-review fix plans (P0 initiative baseline)

2026-07-08 14:43:07 -04:00

89 KiB

Raw Blame History

Cluster, Host & Failover Fixes Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.

Goal: Make automatic failover actually work (enable the SBR downing provider and prove it with a real two-node kill test), unify the "active node" definition on singleton-host (oldest) semantics, and close the CoordinatedShutdown, health-report-loss, offline-detection, and config-validation gaps from architecture review 01.

Architecture: All cluster bootstrap lives in AkkaHostedService.BuildHocon (hand-rolled, injection-safe HOCON) — the Critical fix is adding akka.cluster.downing-provider-class there, verified by a new reusable two-node in-process cluster fixture (TwoNodeClusterFixture) that other plans (02) also depend on. "Active node" is re-derived everywhere from a single new ClusterActivityEvaluator (self is the oldest Up member = singleton host), replacing the four divergent leader-based checks; the five copy-pasted central-singleton registration blocks are collapsed into a CentralSingletonRegistrar helper that also gives the two hot singletons (notification-outbox, audit-log-ingest) the drain tasks they were missing.

Tech Stack: C#/.NET, Akka.NET 1.5.62 (Akka.Cluster, Akka.Cluster.Tools, Akka.TestKit.Xunit2), xUnit + NSubstitute, Docker Compose + Traefik, Blazor Server (one small UI badge).

Build: dotnet build ZB.MOM.WW.ScadaBridge.slnx. Test per-project: dotnet test tests/<project>. Docker rig: bash docker/deploy.sh.

Findings Coverage

#	Report finding	Severity	Task(s) / Disposition
S1	Split-brain resolver never enabled — no automatic failover on crash/partition	Critical	Tasks 1, 3, 4, 21
S2	"Active node" = leader vs oldest diverge (purge on wrong node, wrong Primary label, wrong self-report, Traefik→non-singleton node)	High	Tasks 5, 6, 7, 8
S3	Partition: both central nodes return 200 on `/health/active`, Traefik serves both	High	Tasks 1 (SBR resolves partition), 7 (oldest-based + DB-required active check), 4 (behavioral proof)
S4	Cluster-leave drain tasks budgeted 10s inside 5s CoordinatedShutdown phase	Medium	Task 2
S5	NotificationOutbox + AuditLogIngest singletons have no drain task	Medium	Tasks 9, 10
S6	Health-report loss recovery is dead code (fire-and-forget transport never throws)	Medium	Tasks 11, 12
S7	Offline detection keys off heartbeats, masking a dead metrics pipeline; spec/code disagree	Medium	Tasks 13, 14
S8	`appsettings.Site.json` second seed targets MetricsPort 8084; validator gap	Medium	Task 15
S9	Docker `stop_grace_period` shorter than full CoordinatedShutdown	Medium	Task 17
S10	Aggregator never forgets deleted sites	Low	Task 18
S11	Dead-letter monitor: unbounded warning volume	Low	Task 19
S12	Mandatory two-seed rule forces phantom seeds in single-node installs	Low	Task 16
P1	Readiness probes fan out 5 cluster Asks per poll	Low	Accepted — bounded (2s, concurrent), cheap at current poll rates; revisit only if an orchestrator tightens the probe interval
P2	Startup compiles every inbound method sequentially	Low	Deferred — Inbound API domain (plan 06 scope); report itself says "acceptable now"
P3	Per-tick allocations in health collection	Low	Won't-fix — report: "trivial at this cadence. No action"
C1	REQ-HOST-6 mandates Akka.Hosting; code hand-rolls HOCON	Medium	Task 22
C2	`deploy/wonder-app-vd03` ships `DisableLogin: true`	High	Deferred → Plan 07 (explicit cross-plan ownership)
C3	LDAP plaintext in deploy artifact	Medium	Deferred → Plan 07 (deploy-artifact security posture is plan 07's)
C4	Secrets in checked-in docker dev configs	Medium	Accepted — labelled dev-only, `ConfigSecretsTests` tracks it, the wonder-app-vd03 `${...}` pattern already exists as the model
C5	Stale `Component-TraefikProxy.md` references	Low	Task 23
C6	Five copy-pasted ~60-line singleton registration blocks	Low	Tasks 9, 10
U1	No real multi-node failover test anywhere	—	Tasks 3, 4, 8, 21 (the rig other plans reference)
U2	Windows Service lifecycle / down-if-alone recovery story undefined	—	Task 20
U3	TLS absent at every layer of this domain	—	Deferred — documented lab posture; Task 23 adds an explicit "production TLS profile — not yet implemented" roadmap note so the gap is spec-visible
U4	Dockerfile has no HEALTHCHECK; `/healthz` runs zero checks	—	Deferred — report calls this "acceptable minimalism"; prerequisite work for a future orchestrator migration, not this hardening pass
U5	`NodeName` missing from wonder-app-vd03 overlays → NULL `SourceNode`	—	Task 23
U6	Known-but-open follow-ups (SecuredWrite SourceNode NULL, cert-trust persistence, docker-env2 mirrors findings)	—	No action — already tracked in CLAUDE.md/logged follow-ups; docker-env2 compose IS covered by Task 17
U7	Online/offline transitions unrecorded (post-incident "when did the site drop")	—	Tasks 13, 14 (`LastStatusChangeAt`)

Task 1: Enable the SBR downing provider in BuildHocon

Classification: high-risk (cluster failover semantics — the Critical finding) Estimated implement time: ~3 min Parallelizable with: 5, 11, 13, 15, 16, 17, 19, 22, 23 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs (BuildHocon, cluster block lines 250-266)
Test: tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HoconBuilderTests.cs

Write the failing test in HoconBuilderTests.cs (reuse the existing DefaultCluster() helper):

[Fact]
public void BuildHocon_EnablesSplitBrainResolverDowningProvider()
{
    // Review 01 [Critical]: without downing-provider-class the entire
    // split-brain-resolver section is inert (Akka default = NoDowning) and
    // singletons never migrate on a hard crash.
    var node = new NodeOptions { Role = "Central", NodeHostname = "node1", RemotingPort = 8081 };

    var hocon = AkkaHostedService.BuildHocon(
        node, DefaultCluster(), new[] { "Central" },
        TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(15));

    var config = ConfigurationFactory.ParseString(hocon);
    Assert.Equal(
        "Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster",
        config.GetString("akka.cluster.downing-provider-class"));
}

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~BuildHocon_EnablesSplitBrainResolverDowningProvider" — expect FAIL (key missing).
In BuildHocon, add one line inside the cluster {{ block, directly above split-brain-resolver {{ (after min-nr-of-members = {clusterOptions.MinNrOfMembers}):

        downing-provider-class = ""Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster""

(inside the interpolated string — note doubled quotes). Also update the method's XML doc comment to state the downing provider is explicitly installed because Akka defaults to NoDowning. 4. Run the test again — expect PASS. Run the full file: dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~HoconBuilderTests" — all pass. 5. Commit:

git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HoconBuilderTests.cs
git commit -m "fix(cluster): enable SBR downing provider — automatic failover was inert for crashes/partitions"

Task 2: Raise the cluster-leave CoordinatedShutdown phase timeout above the drain budget

Classification: small Estimated implement time: ~3 min Parallelizable with: 5, 11, 13, 15, 16, 17, 19, 22, 23 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs (BuildHocon, coordinated-shutdown block lines 267-269)
Test: tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HoconBuilderTests.cs

Write the failing test:

[Fact]
public void BuildHocon_ClusterLeavePhaseTimeout_ExceedsDrainBudget()
{
    // Review 01 [Medium]: the five singleton drain tasks GracefulStop(10s)
    // inside cluster-leave, whose default phase timeout is 5s — Akka abandons
    // the task at 5s and the drain guarantee is silently halved.
    var node = new NodeOptions { Role = "Central", NodeHostname = "node1", RemotingPort = 8081 };
    var hocon = AkkaHostedService.BuildHocon(
        node, DefaultCluster(), new[] { "Central" },
        TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(15));

    var config = ConfigurationFactory.ParseString(hocon);
    var phaseTimeout = config.GetTimeSpan("akka.coordinated-shutdown.phases.cluster-leave.timeout");
    Assert.True(phaseTimeout >= TimeSpan.FromSeconds(15),
        $"cluster-leave phase timeout ({phaseTimeout}) must exceed the 10s singleton GracefulStop budget");
}

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~ClusterLeavePhaseTimeout" — expect FAIL.
In BuildHocon, extend the coordinated-shutdown block:

    coordinated-shutdown {{
        run-by-clr-shutdown-hook = on
        phases.cluster-leave.timeout = 15s
    }}

Run again — expect PASS. Also run --filter "FullyQualifiedName~CoordinatedShutdownTests" (source-grep tests must stay green).
Commit:

git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HoconBuilderTests.cs
git commit -m "fix(cluster): give cluster-leave phase a 15s timeout so 10s singleton drains can complete"

Task 3: Two-node in-process cluster fixture (the reusable failover rig)

Classification: high-risk (real cluster formation in tests) Estimated implement time: ~5 min Parallelizable with: 5, 11, 13, 15, 16, 17, 19, 22, 23 Files:

Create: tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/TwoNodeClusterFixture.cs
Test: tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/TwoNodeClusterFixtureTests.cs

This fixture is the rig other plans (02: S&F standby gating) reference — keep its API small and public. It builds each node's config from the production AkkaHostedService.BuildHocon (so the config under test is the config that ships), overlaid with test-only crash-simulation settings.

Create the fixture:

using Akka.Actor;
using Akka.Cluster;
using Akka.Configuration;
using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
using ZB.MOM.WW.ScadaBridge.Host;
using ZB.MOM.WW.ScadaBridge.Host.Actors;

namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;

/// <summary>
/// Forms a REAL two-node Akka cluster in-process from the production
/// BuildHocon output. NodeA is always started first (=> oldest member /
/// singleton host). CrashNode() terminates a node WITHOUT cluster-leave
/// (coordinated-shutdown by-terminate disabled on both nodes) to simulate
/// a hard crash — the scenario review 01 found untested.
/// </summary>
public sealed class TwoNodeClusterFixture : IAsyncDisposable
{
    public ActorSystem NodeA { get; private set; } = null!;
    public ActorSystem NodeB { get; private set; } = null!;
    public int PortA { get; private set; }
    public int PortB { get; private set; }

    public static async Task<TwoNodeClusterFixture> StartAsync(
        string role = "Central", TimeSpan? stableAfter = null)
    {
        var f = new TwoNodeClusterFixture();
        f.PortA = GetFreeTcpPort();
        f.PortB = GetFreeTcpPort();
        f.NodeA = f.StartNode(f.PortA, role, stableAfter);
        await WaitForMembersUp(f.NodeA, 1, TimeSpan.FromSeconds(20));
        f.NodeB = f.StartNode(f.PortB, role, stableAfter);
        await WaitForMembersUp(f.NodeA, 2, TimeSpan.FromSeconds(20));
        await WaitForMembersUp(f.NodeB, 2, TimeSpan.FromSeconds(20));
        return f;
    }

    /// <summary>Starts a node from production HOCON; used by StartAsync and by restart-scenarios.</summary>
    public ActorSystem StartNode(int port, string role, TimeSpan? stableAfter = null)
    {
        var nodeOptions = new NodeOptions { Role = role, NodeHostname = "127.0.0.1", RemotingPort = port };
        var clusterOptions = new ClusterOptions
        {
            SeedNodes = new List<string>
            {
                $"akka.tcp://scadabridge@127.0.0.1:{PortA}",
                $"akka.tcp://scadabridge@127.0.0.1:{PortB}",
            },
            SplitBrainResolverStrategy = "keep-oldest",
            StableAfter = stableAfter ?? TimeSpan.FromSeconds(3),
            HeartbeatInterval = TimeSpan.FromMilliseconds(500),
            FailureDetectionThreshold = TimeSpan.FromSeconds(2),
            MinNrOfMembers = 1,
            DownIfAlone = true,
        };
        var hocon = AkkaHostedService.BuildHocon(
            nodeOptions, clusterOptions, new[] { role },
            TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(3));
        // Crash simulation: Terminate() must NOT run CoordinatedShutdown
        // (no Leave gossip) or the test degenerates to the graceful path.
        var config = ConfigurationFactory
            .ParseString("akka.coordinated-shutdown.run-by-actor-system-terminate = off")
            .WithFallback(ConfigurationFactory.ParseString(hocon));
        return ActorSystem.Create("scadabridge", config);
    }

    /// <summary>Hard-crash a node: abrupt terminate, no cluster leave.</summary>
    public static Task CrashNode(ActorSystem node) => node.Terminate();

    public static async Task WaitForMembersUp(ActorSystem system, int count, TimeSpan timeout)
    {
        var cluster = Akka.Cluster.Cluster.Get(system);
        var deadline = DateTime.UtcNow + timeout;
        while (DateTime.UtcNow < deadline)
        {
            if (cluster.State.Members.Count(m => m.Status == MemberStatus.Up) == count
                && !cluster.State.Unreachable.Any())
                return;
            await Task.Delay(200);
        }
        throw new TimeoutException(
            $"Cluster on {cluster.SelfAddress} did not reach {count} Up members within {timeout}. " +
            $"Members: [{string.Join(", ", cluster.State.Members)}], " +
            $"Unreachable: [{string.Join(", ", cluster.State.Unreachable)}]");
    }

    public static async Task WaitForMemberRemoved(ActorSystem survivor, Address removed, TimeSpan timeout)
    {
        var cluster = Akka.Cluster.Cluster.Get(survivor);
        var deadline = DateTime.UtcNow + timeout;
        while (DateTime.UtcNow < deadline)
        {
            if (cluster.State.Members.All(m => m.Address != removed)
                && cluster.State.Unreachable.All(m => m.Address != removed))
                return;
            await Task.Delay(200);
        }
        throw new TimeoutException($"Member {removed} was never removed from {cluster.SelfAddress}'s view — SBR did not down it.");
    }

    private static int GetFreeTcpPort()
    {
        var listener = new System.Net.Sockets.TcpListener(System.Net.IPAddress.Loopback, 0);
        listener.Start();
        var port = ((System.Net.IPEndPoint)listener.LocalEndpoint).Port;
        listener.Stop();
        return port;
    }

    public async ValueTask DisposeAsync()
    {
        foreach (var sys in new[] { NodeA, NodeB })
        {
            if (sys is { WhenTerminated.IsCompleted: false })
            {
                try { await sys.Terminate().WaitAsync(TimeSpan.FromSeconds(10)); } catch { /* teardown */ }
            }
        }
    }
}

Write the smoke test proving the fixture forms a real cluster:

public class TwoNodeClusterFixtureTests
{
    [Fact]
    public async Task Fixture_FormsTwoNodeCluster_NodeAIsOldest()
    {
        await using var cluster = await TwoNodeClusterFixture.StartAsync();
        var a = Akka.Cluster.Cluster.Get(cluster.NodeA);
        var b = Akka.Cluster.Cluster.Get(cluster.NodeB);
        Assert.Equal(2, a.State.Members.Count);
        Assert.Equal(2, b.State.Members.Count);
        // NodeA started first => oldest (the singleton host).
        Assert.True(a.SelfMember.IsOlderThan(b.State.Members.First(m => m.Address == b.SelfAddress)));
    }
}

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.IntegrationTests --filter "FullyQualifiedName~TwoNodeClusterFixtureTests" — expect FAIL first (files don't exist / compile), then PASS once implemented. (This task is fixture-first; the "failing test" step is the compile failure.)
If NodeOptions/BuildHocon visibility blocks compilation, note that IntegrationTests already references the Host project (.csproj line 37) and both are public — no changes to src/ are expected in this task.
Commit:

git add tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/
git commit -m "test(cluster): add TwoNodeClusterFixture — real two-node in-process cluster rig from production HOCON"

Task 4: SBR behavioral proof — kill the oldest node, assert downing + singleton migration

Classification: high-risk Estimated implement time: ~5 min Parallelizable with: 5, 11, 13, 15, 16, 17, 19, 22, 23 Files:

Create: tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SbrFailoverTests.cs

Write the test (this is the test that would have caught the Critical finding — it must FAIL if Task 1's HOCON line is reverted; verify that once by stashing Task 1 if cheap, otherwise trust the assertion on member removal, which is impossible under NoDowning):

using Akka.Actor;
using Akka.Cluster.Tools.Singleton;

namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;

public class SbrFailoverTests
{
    private sealed class EchoActor : ReceiveActor
    {
        public EchoActor() => ReceiveAny(msg => Sender.Tell(msg));
    }

    private static (IActorRef manager, IActorRef proxy) StartSingleton(ActorSystem sys)
    {
        var manager = sys.ActorOf(ClusterSingletonManager.Props(
            Props.Create(() => new EchoActor()),
            PoisonPill.Instance,
            ClusterSingletonManagerSettings.Create(sys).WithSingletonName("failover-probe")),
            "failover-probe-singleton");
        var proxy = sys.ActorOf(ClusterSingletonProxy.Props(
            "/user/failover-probe-singleton",
            ClusterSingletonProxySettings.Create(sys).WithSingletonName("failover-probe")),
            "failover-probe-proxy");
        return (manager, proxy);
    }

    [Fact]
    public async Task HardCrashOfOldestNode_SbrDownsIt_AndSingletonMigratesToSurvivor()
    {
        await using var cluster = await TwoNodeClusterFixture.StartAsync();
        StartSingleton(cluster.NodeA);
        var (_, proxyB) = StartSingleton(cluster.NodeB);

        // Singleton is reachable from B while A (oldest) hosts it.
        var echo = await proxyB.Ask<string>("ping", TimeSpan.FromSeconds(20));
        Assert.Equal("ping", echo);

        var victimAddress = Akka.Cluster.Cluster.Get(cluster.NodeA).SelfAddress;
        await TwoNodeClusterFixture.CrashNode(cluster.NodeA);

        // 1) SBR must DOWN and REMOVE the crashed member (NoDowning => this
        //    times out; the member stays unreachable forever).
        // Budget: failure detection (~2s) + stable-after (3s) + gossip margin.
        await TwoNodeClusterFixture.WaitForMemberRemoved(
            cluster.NodeB, victimAddress, TimeSpan.FromSeconds(30));

        // 2) The singleton must re-host on the survivor and answer again.
        var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(30);
        Exception? last = null;
        while (DateTime.UtcNow < deadline)
        {
            try
            {
                var echo2 = await proxyB.Ask<string>("ping-after-failover", TimeSpan.FromSeconds(3));
                Assert.Equal("ping-after-failover", echo2);
                return;
            }
            catch (Exception ex) { last = ex; }
        }
        throw new Xunit.Sdk.XunitException($"Singleton never re-hosted on the survivor after SBR downing: {last}");
    }
}

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.IntegrationTests --filter "FullyQualifiedName~SbrFailoverTests" — expect PASS (Task 1 is in). To confirm the test has teeth, temporarily comment out the downing-provider-class line in BuildHocon, re-run, expect FAIL at WaitForMemberRemoved (timeout), then restore the line. Do NOT commit the reverted state.
Commit:

git add tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SbrFailoverTests.cs
git commit -m "test(cluster): prove SBR downs a hard-crashed oldest node and the singleton migrates"

Task 5: ClusterActivityEvaluator — single oldest-member "active node" definition

Classification: standard Estimated implement time: ~4 min Parallelizable with: 1, 2, 3, 4, 11, 13, 15, 16, 17, 19, 22, 23 Files:

Create: src/ZB.MOM.WW.ScadaBridge.Host/Health/ClusterActivityEvaluator.cs
Test: tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ClusterActivityEvaluatorTests.cs

Write the failing test (real single-node cluster via TestKit-style bootstrap, mirroring AkkaBootstrapTests):

using Akka.Actor;
using Akka.Configuration;
using ZB.MOM.WW.ScadaBridge.Host.Health;

namespace ZB.MOM.WW.ScadaBridge.Host.Tests;

public class ClusterActivityEvaluatorTests : IAsyncLifetime
{
    private ActorSystem? _system;

    public async Task InitializeAsync()
    {
        var port = FreePort();
        var config = ConfigurationFactory.ParseString($@"
akka {{
    actor.provider = cluster
    remote.dot-netty.tcp {{ hostname = ""127.0.0.1"", port = {port} }}
    cluster {{
        seed-nodes = [""akka.tcp://eval-test@127.0.0.1:{port}""]
        roles = [""Central""]
        min-nr-of-members = 1
    }}
}}");
        _system = ActorSystem.Create("eval-test", config);
        var cluster = Akka.Cluster.Cluster.Get(_system);
        var deadline = DateTime.UtcNow.AddSeconds(20);
        while (cluster.SelfMember.Status != Akka.Cluster.MemberStatus.Up && DateTime.UtcNow < deadline)
            await Task.Delay(100);
    }

    public async Task DisposeAsync() { if (_system != null) await _system.Terminate(); }

    [Fact]
    public void SelfIsOldest_SoleUpMember_ReturnsTrue()
    {
        var cluster = Akka.Cluster.Cluster.Get(_system!);
        Assert.True(ClusterActivityEvaluator.SelfIsOldest(cluster));
        Assert.True(ClusterActivityEvaluator.SelfIsOldest(cluster, "Central"));
    }

    [Fact]
    public void SelfIsOldest_RoleNotHeld_ReturnsFalse()
    {
        var cluster = Akka.Cluster.Cluster.Get(_system!);
        // Self doesn't carry the role => it can never host that role's singletons.
        Assert.False(ClusterActivityEvaluator.SelfIsOldest(cluster, "site-nonexistent"));
    }

    private static int FreePort()
    {
        var l = new System.Net.Sockets.TcpListener(System.Net.IPAddress.Loopback, 0);
        l.Start(); var p = ((System.Net.IPEndPoint)l.LocalEndpoint).Port; l.Stop(); return p;
    }
}

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~ClusterActivityEvaluatorTests" — expect FAIL (type missing).
Implement:

using Akka.Cluster;

namespace ZB.MOM.WW.ScadaBridge.Host.Health;

/// <summary>
/// THE single definition of "active node" (review 01 [High]): a node is active
/// when it is the OLDEST Up member (optionally within a role scope) — i.e. the
/// member the ClusterSingletonManager places singletons on. Cluster LEADERSHIP
/// (lowest address) is an Akka-internal concept that diverges from singleton
/// placement permanently once the original first node restarts and rejoins;
/// every product-level active/standby decision must use this evaluator, never
/// cluster.State.Leader.
/// </summary>
public static class ClusterActivityEvaluator
{
    /// <summary>True when self is Up and no other Up member (in the role scope) is older.</summary>
    public static bool SelfIsOldest(Cluster cluster, string? role = null)
    {
        var self = cluster.SelfMember;
        if (self.Status != MemberStatus.Up)
            return false;
        if (role != null && !self.HasRole(role))
            return false;

        return cluster.State.Members
            .Where(m => m.Status == MemberStatus.Up)
            .Where(m => role == null || m.HasRole(role))
            .All(m => m.UniqueAddress.Equals(self.UniqueAddress) || self.IsOlderThan(m));
    }

    /// <summary>The oldest Up member in the role scope, or null while none is Up. Used for Primary/Standby labelling.</summary>
    public static Member? OldestUpMember(Cluster cluster, string? role = null)
    {
        Member? oldest = null;
        foreach (var m in cluster.State.Members.Where(m => m.Status == MemberStatus.Up))
        {
            if (role != null && !m.HasRole(role)) continue;
            if (oldest == null || m.IsOlderThan(oldest)) oldest = m;
        }
        return oldest;
    }
}

Run — expect PASS.
Commit:

git add src/ZB.MOM.WW.ScadaBridge.Host/Health/ClusterActivityEvaluator.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ClusterActivityEvaluatorTests.cs
git commit -m "feat(host): ClusterActivityEvaluator — oldest-member (singleton-host) active-node definition"

Task 6: Rewire ActiveNodeGate, AkkaClusterNodeProvider and Primary labels onto the evaluator

Classification: high-risk (changes which node the purge, self-report, inbound gate and dashboard consider active) Estimated implement time: ~5 min Parallelizable with: 11, 13, 15, 16, 17, 19, 22, 23 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeGate.cs (lines 36-52)
Modify: src/ZB.MOM.WW.ScadaBridge.Host/Health/AkkaClusterNodeProvider.cs (lines 29-40, 43-81)
Modify: src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/IClusterNodeProvider.cs (SelfIsPrimary xmldoc, lines 15-21)
Modify: src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs (comment at lines 109-115 — text only)
Test: tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ClusterActivityEvaluatorTests.cs (extend)

Write the failing test (extend ClusterActivityEvaluatorTests — the single-node cluster from Task 5 is enough because with one member, leader == oldest; the divergence behavior is proven in Task 8's two-node test. Here assert the wiring):

[Fact]
public void ActiveNodeGate_And_NodeProvider_AgreeWithEvaluator()
{
    // Both product gates must be backed by the oldest-member evaluator.
    // A single-node Up cluster: evaluator says true, so both must say true.
    var akkaService = /* construct AkkaHostedService test double or use the
       pattern in ActorSystemBridgeTests to wrap _system */;
    var gate = new ActiveNodeGate(akkaService);
    var provider = new AkkaClusterNodeProvider(akkaService, "Central");
    Assert.True(gate.IsActiveNode);
    Assert.True(provider.SelfIsPrimary);
}

Follow the construction pattern used by tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ActorSystemBridgeTests.cs / HealthCheckTests.cs for obtaining an AkkaHostedService whose ActorSystem is the test system (if AkkaHostedService cannot be instantiated cheaply, add an internal test seam — but check those tests first; a pattern exists because ActiveNodeGate is already constructor-tested). 2. Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~AgreeWithEvaluator" — expect FAIL (or compile error) before rewiring if you assert on a divergence scenario; if single-node passes trivially pre-change, treat this as a pinning test and rely on Task 8 for the behavioral delta. 3. Implement:

ActiveNodeGate.IsActiveNode: replace the leader comparison (lines 44-51) with return ClusterActivityEvaluator.SelfIsOldest(cluster); (keep the null-system and Up guards; update the class xmldoc: "oldest Up member — the singleton host — not the cluster leader").
AkkaClusterNodeProvider.SelfIsPrimary: replace leader comparison with return ClusterActivityEvaluator.SelfIsOldest(cluster, _siteRole);.
AkkaClusterNodeProvider.GetClusterNodes: replace var leader = cluster.State.Leader; + isLeader with var oldest = ClusterActivityEvaluator.OldestUpMember(cluster, _siteRole); and var isPrimary = oldest != null && member.UniqueAddress.Equals(oldest.UniqueAddress); — label "Primary"/"Standby" from that.
IClusterNodeProvider.SelfIsPrimary xmldoc: change "currently the cluster leader (Primary)" to "currently the oldest Up member for the provider's role scope — the node the ClusterSingletonManager hosts singletons on. This is the canonical product 'active node' check (review 01): the site event-log purge gate, the central self-report loop, and (via plan 02) the S&F delivery gate all consume it."
SiteServiceRegistration.cs lines 109-115 comment: replace "this node is Up AND cluster leader" wording with "this node is the oldest Up member (singleton host)".

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests and dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests — all pass (CentralHealthReportLoop/EventLogPurge consumers are interface-driven; their fakes are unaffected).
Commit:

git add src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeGate.cs src/ZB.MOM.WW.ScadaBridge.Host/Health/AkkaClusterNodeProvider.cs src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/IClusterNodeProvider.cs src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ClusterActivityEvaluatorTests.cs
git commit -m "fix(host): unify active-node on oldest-member semantics — purge/self-report/labels now track the singleton host"

Seam note for plan 02: after this task, IClusterNodeProvider.SelfIsPrimary (registered in site DI at SiteServiceRegistration.cs:101-107) IS the canonical "this node hosts the site singletons / is active" check. Plan 02's S&F delivery gating should inject IClusterNodeProvider (or register a delegate mirroring SiteEventLogActiveNodeCheck, SiteServiceRegistration.cs:116-120) rather than invent a new check.

Task 7: Oldest-based /health/active check + require DB reachability for Active

Classification: high-risk (changes Traefik routing behavior) Estimated implement time: ~5 min Parallelizable with: 11, 13, 15, 16, 17, 19, 22, 23 Files:

Create: src/ZB.MOM.WW.ScadaBridge.Host/Health/OldestNodeActiveHealthCheck.cs
Modify: src/ZB.MOM.WW.ScadaBridge.Host/Program.cs (health check registrations, lines 209-234; comment block lines 346-356)
Test: tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HealthCheckTests.cs

Write failing tests (follow the existing registration-introspection style in HealthCheckTests.cs — it already builds the central service collection and inspects HealthCheckServiceOptions):

[Fact]
public void ActiveNodeCheck_IsHostLocalOldestNodeCheck()
{
    var registrations = GetCentralHealthCheckRegistrations(); // existing helper or mirror existing tests
    var active = registrations.Single(r => r.Name == "active-node");
    Assert.Contains(ZbHealthTags.Active, active.Tags);
    // The shared package's ActiveNodeHealthCheck is LEADER-based (review 01 [High]):
    // during a partition both nodes think they lead and Traefik serves both.
    var check = active.Factory(BuildCentralServiceProvider());
    Assert.IsType<OldestNodeActiveHealthCheck>(check);
}

[Fact]
public void DatabaseCheck_AlsoGatesActive()
{
    // Review 01 [High] recommendation: /health/active should require Ready
    // (DB reachable) in addition to singleton-host status, so a DB-dead
    // active node is pulled from Traefik rotation.
    var registrations = GetCentralHealthCheckRegistrations();
    var db = registrations.Single(r => r.Name == "database");
    Assert.Contains(ZbHealthTags.Ready, db.Tags);
    Assert.Contains(ZbHealthTags.Active, db.Tags);
}

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~HealthCheckTests" — expect FAIL.
Implement:
- New OldestNodeActiveHealthCheck:

using Akka.Actor;
using Microsoft.Extensions.Diagnostics.HealthChecks;

namespace ZB.MOM.WW.ScadaBridge.Host.Health;

/// <summary>
/// /health/active check: Healthy only on the oldest Up member — the node that
/// hosts the cluster singletons. Replaces the shared package's leader-based
/// ActiveNodeHealthCheck (review 01 [High]): leadership is address-ordered and
/// diverges from singleton placement after a restart, and during a partition
/// BOTH nodes compute themselves leader, so Traefik served both. Oldest-member
/// semantics keep exactly one active node through a partition (the non-oldest
/// side still sees the old oldest member until SBR resolves membership).
/// </summary>
public sealed class OldestNodeActiveHealthCheck : IHealthCheck
{
    private readonly ActorSystem _system;
    public OldestNodeActiveHealthCheck(ActorSystem system) => _system = system;

    public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken ct = default)
    {
        var cluster = Akka.Cluster.Cluster.Get(_system);
        return Task.FromResult(ClusterActivityEvaluator.SelfIsOldest(cluster)
            ? HealthCheckResult.Healthy("oldest Up member (singleton host)")
            : HealthCheckResult.Unhealthy("not the oldest Up member (standby)"));
    }
}

In Program.cs: replace .AddTypeActivatedCheck<ActiveNodeHealthCheck>("active-node", ...) with .AddTypeActivatedCheck<OldestNodeActiveHealthCheck>("active-node", failureStatus: null, tags: new[] { ZbHealthTags.Active }); change the database check's tags to new[] { ZbHealthTags.Ready, ZbHealthTags.Active }; update the tier-separation comment block (lines 204-208 and 346-356) to say "active-node = oldest Up member + database reachable" and remove the now-stale ActiveNodeHealthCheck mention. Keep the ActiveNodeGate registration comment (lines 252-260) in sync ("same oldest-member evaluator as OldestNodeActiveHealthCheck").

Run — expect PASS. Also run --filter "FullyQualifiedName~RequiredSingletonsHealthCheckTests" (readiness untouched) and build the solution.
Commit:

git add src/ZB.MOM.WW.ScadaBridge.Host/Health/OldestNodeActiveHealthCheck.cs src/ZB.MOM.WW.ScadaBridge.Host/Program.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HealthCheckTests.cs
git commit -m "fix(host): /health/active = oldest member + DB reachable — partition-safe Traefik routing"

Task 8: Divergence proof — restarted first node must NOT become active again

Classification: high-risk Estimated implement time: ~5 min Parallelizable with: 11, 13, 15, 16, 17, 19, 22, 23 Files:

Create: tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/ActiveNodeSemanticsTests.cs

Write the test using the Task 3 fixture (this is the exact scenario from the report: after the original first node restarts and rejoins, leadership returns to it by address order while the peer remains oldest — the two definitions permanently diverge):

using ZB.MOM.WW.ScadaBridge.Host.Health;

namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;

public class ActiveNodeSemanticsTests
{
    [Fact]
    public async Task AfterOldestNodeRestarts_SurvivorStaysActive_RejoinerIsStandby()
    {
        await using var fixture = await TwoNodeClusterFixture.StartAsync();

        // Crash A (oldest), let SBR remove it, B becomes oldest.
        var victim = Akka.Cluster.Cluster.Get(fixture.NodeA).SelfAddress;
        await TwoNodeClusterFixture.CrashNode(fixture.NodeA);
        await TwoNodeClusterFixture.WaitForMemberRemoved(fixture.NodeB, victim, TimeSpan.FromSeconds(30));

        // Restart A on the same port — it rejoins as the YOUNGEST member but
        // (lowest address) may regain Akka leadership.
        var rejoined = fixture.StartNode(fixture.PortA, "Central");
        try
        {
            await TwoNodeClusterFixture.WaitForMembersUp(rejoined, 2, TimeSpan.FromSeconds(30));
            await TwoNodeClusterFixture.WaitForMembersUp(fixture.NodeB, 2, TimeSpan.FromSeconds(30));

            var clusterB = Akka.Cluster.Cluster.Get(fixture.NodeB);
            var clusterA2 = Akka.Cluster.Cluster.Get(rejoined);

            // Product active-node semantics: survivor (oldest, singleton host)
            // is active; the rejoined original node is standby — even when it
            // holds Akka leadership.
            Assert.True(ClusterActivityEvaluator.SelfIsOldest(clusterB));
            Assert.False(ClusterActivityEvaluator.SelfIsOldest(clusterA2));
        }
        finally
        {
            await rejoined.Terminate();
        }
    }
}

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.IntegrationTests --filter "FullyQualifiedName~ActiveNodeSemanticsTests" — expect PASS. Sanity: assert the OLD leader-based expression (clusterA2.State.Leader == clusterA2.SelfAddress) would have returned true for the rejoiner in at least one run — if it reliably does, add it as an inline comment documenting why leader-based was wrong (don't make the test depend on Akka leadership internals).
Commit:

git add tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/ActiveNodeSemanticsTests.cs
git commit -m "test(cluster): restarted original node stays standby — oldest-member active semantics proven"

Task 9: CentralSingletonRegistrar helper (manager + drain task + proxy)

Classification: high-risk (actor lifecycle) Estimated implement time: ~5 min Parallelizable with: 11, 13, 15, 16, 17, 19, 22, 23 Files:

Create: src/ZB.MOM.WW.ScadaBridge.Host/Actors/CentralSingletonRegistrar.cs
Test: tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CentralSingletonRegistrarTests.cs

Write the failing test (single-node cluster, real CoordinatedShutdown run):

using Akka.Actor;
using Akka.Configuration;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.Host.Actors;

namespace ZB.MOM.WW.ScadaBridge.Host.Tests;

public class CentralSingletonRegistrarTests
{
    private sealed class EchoActor : ReceiveActor
    {
        public EchoActor() => ReceiveAny(m => Sender.Tell(m));
    }

    [Fact]
    public async Task Start_CreatesManagerAndProxy_WithCanonicalNames_AndDrainTask()
    {
        using var system = CreateSingleNodeClusterSystem(); // same bootstrap as ClusterActivityEvaluatorTests
        var handle = CentralSingletonRegistrar.Start(
            system, "test-widget", Props.Create(() => new EchoActor()),
            NullLogger.Instance, drainTimeout: TimeSpan.FromSeconds(2));

        // Canonical naming preserved: <name>-singleton / <name>-proxy.
        Assert.EndsWith("/user/test-widget-singleton", handle.Manager.Path.ToString());
        Assert.EndsWith("/user/test-widget-proxy", handle.Proxy.Path.ToString());

        // Singleton answers through the proxy once the member is Up.
        var echo = await handle.Proxy.Ask<string>("hi", TimeSpan.FromSeconds(20));
        Assert.Equal("hi", echo);

        // The drain task must stop the manager during cluster-leave.
        var probeSystem = system; // watch from a test actor
        var watcher = probeSystem.ActorOf(Props.Create(() => new WatchActor(handle.Manager)));
        await Akka.Actor.CoordinatedShutdown.Get(system).Run(Akka.Actor.CoordinatedShutdown.ClrExitReason.Instance);
        // Manager terminated => drain ran (GracefulStop completed or PoisonPill fallback).
        Assert.True(handle.Manager.IsNobody() || system.WhenTerminated.IsCompleted);
    }
}

(Use a WatchActor/TestProbe-style Terminated assertion if IsNobody() proves flaky — the load-bearing assertions are the two path names and the pre-shutdown echo; the drain execution is additionally covered by the existing five singletons' behavior in Task 10's regression run.) 2. Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~CentralSingletonRegistrarTests" — expect FAIL (type missing). 3. Implement CentralSingletonRegistrar:

using Akka.Actor;
using Akka.Cluster.Tools.Singleton;
using Microsoft.Extensions.Logging;

namespace ZB.MOM.WW.ScadaBridge.Host.Actors;

/// <summary>
/// Registers a central cluster singleton with the canonical naming scheme
/// (<c>{name}-singleton</c> / <c>{name}-proxy</c>), a PoisonPill termination
/// message, and a PhaseClusterLeave drain task that GracefulStops the manager
/// so in-flight EF work completes before handover. Extracted from five
/// copy-pasted ~60-line blocks (review 01 [Low]) whose drift left the two
/// busiest singletons (notification-outbox, audit-log-ingest) without drain
/// tasks (review 01 [Medium]).
/// </summary>
internal static class CentralSingletonRegistrar
{
    internal sealed record Handle(IActorRef Manager, IActorRef Proxy);

    internal static Handle Start(
        ActorSystem system,
        string name,
        Props singletonProps,
        ILogger logger,
        TimeSpan? drainTimeout = null)
    {
        var manager = system.ActorOf(
            ClusterSingletonManager.Props(
                singletonProps,
                PoisonPill.Instance,
                ClusterSingletonManagerSettings.Create(system).WithSingletonName(name)),
            $"{name}-singleton");

        var timeout = drainTimeout ?? TimeSpan.FromSeconds(10);
        Akka.Actor.CoordinatedShutdown.Get(system).AddTask(
            Akka.Actor.CoordinatedShutdown.PhaseClusterLeave,
            $"drain-{name}-singleton",
            async () =>
            {
                try
                {
                    await manager.GracefulStop(timeout);
                }
                catch (Exception ex)
                {
                    logger.LogWarning(ex,
                        "{Singleton} singleton did not drain within the graceful-stop timeout; "
                        + "falling through to PoisonPill handover", name);
                }
                return Akka.Done.Instance;
            });

        var proxy = system.ActorOf(
            ClusterSingletonProxy.Props(
                $"/user/{name}-singleton",
                ClusterSingletonProxySettings.Create(system).WithSingletonName(name)),
            $"{name}-proxy");

        return new Handle(manager, proxy);
    }
}

Run — expect PASS.
Commit:

git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/CentralSingletonRegistrar.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CentralSingletonRegistrarTests.cs
git commit -m "feat(host): CentralSingletonRegistrar — one registration path with drain task for central singletons"

Task 10: Route all seven central singletons through the registrar (adds the missing outbox/ingest drains)

Classification: high-risk (actor wiring refactor) Estimated implement time: ~5 min Parallelizable with: 11, 13, 15, 16, 17, 19, 22, 23 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs (lines ~395-811: notification-outbox 410-434, audit-log-ingest 451-480, site-call-audit 524-589, audit-log-purge 612-648, site-audit-reconciliation 663-700, kpi-history-recorder 722-757, pending-deployment-purge 776-811)
Test: existing suites (regression) + tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CoordinatedShutdownTests.cs (one new source assertion)

Write the failing test — extend CoordinatedShutdownTests in its existing source-grep style (the behavioral drain is covered by Task 9; this pins that the two hot singletons now go through the drain path):

[Fact]
public void AllCentralSingletons_RegisterThroughRegistrarWithDrain()
{
    var hostProjectDir = FindHostProjectDirectory();
    var content = File.ReadAllText(Path.Combine(hostProjectDir!, "Actors", "AkkaHostedService.cs"));
    // Review 01 [Medium]: notification-outbox and audit-log-ingest were the
    // only central singletons WITHOUT a cluster-leave drain task.
    foreach (var name in new[] { "notification-outbox", "audit-log-ingest", "site-call-audit",
                                 "audit-log-purge", "site-audit-reconciliation",
                                 "kpi-history-recorder", "pending-deployment-purge" })
    {
        Assert.Contains($"CentralSingletonRegistrar.Start(", content);
        Assert.Contains($"\"{name}\"", content);
    }
    // No hand-rolled manager registrations remain in RegisterCentralActors.
    Assert.DoesNotContain("ClusterSingletonManager.Props(", ExtractRegisterCentralActors(content));
}

(Implement ExtractRegisterCentralActors as a substring between RegisterCentralActors and the next method, or simply assert Assert.Equal(0, CountOccurrences(content, "ClusterSingletonManager.Props(")) after checking the site branch separately — the site deployment-manager/event-log singletons at lines 906+ use .WithRole(...) and stay hand-rolled; scope the assertion to avoid them, e.g. count == 2.) 2. Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~AllCentralSingletons" — expect FAIL. 3. Refactor RegisterCentralActors: replace each of the seven manager/drain/proxy blocks with

var outbox = CentralSingletonRegistrar.Start(
    _actorSystem!, "notification-outbox",
    Props.Create(() => new ZB.MOM.WW.ScadaBridge.NotificationOutbox.NotificationOutboxActor(
        _serviceProvider, outboxOptions, outboxAuditWriter, outboxLogger)),
    _logger);
centralCommActor.Tell(new RegisterNotificationOutbox(outbox.Proxy));
commService?.SetNotificationOutbox(outbox.Proxy);

and equivalently for the other six (preserve every existing proxy hand-off: RegisterAuditIngest, grpcServer?.SetAuditIngestActor, commService?.SetSiteCallAudit, siteCallAuditProxy.Tell(RegisterCentralCommunication…) → siteCallAudit.Proxy.Tell(…), and the log lines). Actor names and singleton names are unchanged (the registrar reproduces them exactly), so ActorPathTests and RequiredSingletonsHealthCheckTests must stay green. Preserve the comment describing the drain rationale once at the top of RegisterCentralActors (delete the five per-block copies and the "not added here to keep this change minimal" note at 544-549, which this task retires). 4. Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests (full project — ActorPath/RequiredSingletons/AuditWiring are the regression net) and dotnet build ZB.MOM.WW.ScadaBridge.slnx — expect PASS. 5. Commit:

git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CoordinatedShutdownTests.cs
git commit -m "refactor(host): central singletons via CentralSingletonRegistrar — outbox/audit-ingest gain drain tasks"

Task 11: SiteHealthReportAck message + ack semantics in the communication actors

Classification: high-risk (actor messaging, cross-cluster contract) Estimated implement time: ~5 min Parallelizable with: 1, 2, 3, 4, 5, 13, 15, 16, 17, 19, 22, 23 Files:

Create: src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReportAck.cs
Modify: src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteCommunicationActor.cs (SiteHealthReport handler, lines ~393-399)
Modify: src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs (HandleSiteHealthReport, lines ~360-374)
Test: tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/ (new file HealthReportAckTests.cs)

Write failing TestKit tests (mirror the existing NotificationSubmit ack tests in Communication.Tests — grep NotificationSubmitAck there for the harness pattern):

[Fact]
public void SiteCommunicationActor_NoCentralClient_RepliesNotAccepted()
{
    var siteComm = Sys.ActorOf(Props.Create(() => new SiteCommunicationActor(
        "site-a", TestCommunicationOptions(), TestProbe().Ref)));
    // No RegisterCentralClient sent => _centralClient is null.
    siteComm.Tell(SampleReport(seq: 7));
    var ack = ExpectMsg<SiteHealthReportAck>();
    Assert.False(ack.Accepted);
    Assert.Equal(7, ack.SequenceNumber);
}

[Fact]
public void CentralCommunicationActor_OnReport_ProcessesAndAcks()
{
    var actor = CreateCentralCommunicationActorWithAggregator(out var aggregator);
    actor.Tell(SampleReport(seq: 3));
    var ack = ExpectMsg<SiteHealthReportAck>();
    Assert.True(ack.Accepted);
    Assert.Equal(3, ack.SequenceNumber);
    // aggregator received the report (NSubstitute: aggregator.Received().ProcessReport(...))
}

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.Communication.Tests --filter "FullyQualifiedName~HealthReportAckTests" — expect FAIL (message type missing).
Implement:
- Commons (additive message evolution — new type only):

namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;

/// <summary>
/// Acknowledgement for a <see cref="SiteHealthReport"/> forwarded site→central.
/// Review 01 [Medium]: the transport was fire-and-forget, so the sender's
/// interval-counter restore logic could never observe a loss. The ack makes
/// delivery observable end-to-end (central processed the report).
/// </summary>
public sealed record SiteHealthReportAck(
    string SiteId, long SequenceNumber, bool Accepted, string? Error = null);

SiteCommunicationActor Receive<SiteHealthReport>: mirror the NotificationSubmit pattern exactly — when _centralClient == null, Sender.Tell(new SiteHealthReportAck(msg.SiteId, msg.SequenceNumber, false, "Central ClusterClient not registered")); otherwise _centralClient.Tell(new ClusterClient.Send("/user/central-communication", msg), Sender) (forward the original Sender so the central ack routes straight back to the waiting Ask — replaces the current Self sender).
CentralCommunicationActor.HandleSiteHealthReport: after ProcessLocally(report) and the pub-sub replica publish, add Sender.Tell(new SiteHealthReportAck(report.SiteId, report.SequenceNumber, true)); (guard with !Sender.IsNobody() so the peer-replica path, which arrives without an Ask, doesn't dead-letter).

Run — expect PASS. Run the whole Communication.Tests project.
Commit:

git add src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReportAck.cs src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteCommunicationActor.cs src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/HealthReportAckTests.cs
git commit -m "feat(comm): ack site health reports end-to-end (SiteHealthReportAck, additive contract)"

Task 12: Acked async health-report transport so counter-restore actually fires

Classification: high-risk (changes the sender loop's failure semantics) Estimated implement time: ~5 min Parallelizable with: 13, 15, 16, 17, 19, 22, 23 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/IHealthReportTransport.cs
Modify: src/ZB.MOM.WW.ScadaBridge.Host/AkkaHealthReportTransport.cs
Modify: src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/HealthReportSender.cs (line 160)
Test: tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/HealthReportSenderTests.cs (update fakes), tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ (new AkkaHealthReportTransportTests.cs)

Write failing tests:
- HealthMonitoring: update the fake transport in HealthReportSenderTests to the new Task SendAsync(SiteHealthReport, CancellationToken) signature and add/adjust:

[Fact]
public async Task SendAsyncFailure_RestoresIntervalCounters()
{
    // The restore path (HealthReportSender.cs:158-175) was DEAD CODE against
    // the fire-and-forget production transport. With an acked async transport
    // a timeout/nack surfaces as an exception and the counters roll forward.
    var collector = NewCollectorWithErrors(scriptErrors: 4);
    var transport = new ThrowingTransport(); // SendAsync throws
    var sender = NewSender(collector, transport);
    await RunOneTick(sender);
    Assert.Equal(4, CollectorScriptErrorCount(collector)); // restored, not lost
}

Host: AkkaHealthReportTransportTests — TestKit: place a probe-backed actor at /user/site-communication that replies SiteHealthReportAck(Accepted: true); assert SendAsync completes. Second test: reply Accepted: false → assert SendAsync throws InvalidOperationException. Third: no actor at the path → assert it throws (Ask timeout; use a short timeout via options or constant).

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests --filter "FullyQualifiedName~HealthReportSenderTests" — expect FAIL (compile).
Implement:
- IHealthReportTransport: replace void Send(SiteHealthReport report); with Task SendAsync(SiteHealthReport report, CancellationToken cancellationToken); (update xmldoc: "must THROW on non-delivery — the sender's counter-restore depends on it").
- AkkaHealthReportTransport:

public async Task SendAsync(SiteHealthReport report, CancellationToken cancellationToken)
{
    var actorSystem = _akkaService.ActorSystem
        ?? throw new InvalidOperationException("Actor system not started — health report not sent");
    var siteComm = actorSystem.ActorSelection("/user/site-communication");
    var ack = await siteComm.Ask<SiteHealthReportAck>(report, AckTimeout, cancellationToken);
    if (!ack.Accepted)
        throw new InvalidOperationException($"Health report #{report.SequenceNumber} rejected: {ack.Error}");
}
private static readonly TimeSpan AckTimeout = TimeSpan.FromSeconds(10);

HealthReportSender line 160: _transport.Send(reportWithSeq); → await _transport.SendAsync(reportWithSeq, stoppingToken).ConfigureAwait(false); (the surrounding try/catch + restore stays exactly as-is — it now actually fires). Trim the comment at 142-152 to note the transport is now acked.

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests and dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~AkkaHealthReportTransportTests" — PASS. Build the solution (interface change ripples to any other fake).
Commit:

git add src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/IHealthReportTransport.cs src/ZB.MOM.WW.ScadaBridge.Host/AkkaHealthReportTransport.cs src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/HealthReportSender.cs tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests tests/ZB.MOM.WW.ScadaBridge.Host.Tests/AkkaHealthReportTransportTests.cs
git commit -m "fix(health): acked SendAsync transport — report-loss counter restore is live, not dead code"

Task 13: Metrics-staleness signal + status-transition timestamps in the aggregator

Classification: high-risk (lock-free CAS state machine) Estimated implement time: ~5 min Parallelizable with: 1, 2, 3, 4, 5, 11, 15, 16, 17, 19, 22, 23 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthState.cs
Modify: src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthAggregator.cs (ProcessReport, CheckForOfflineSites)
Modify: src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/HealthMonitoringOptions.cs + HealthMonitoringOptionsValidator.cs
Modify: docs/requirements/Component-HealthMonitoring.md (offline definition, lines ~43-46 + option table)
Test: tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/CentralHealthAggregatorTests.cs, HealthMonitoringOptionsValidatorTests.cs

Write failing tests in CentralHealthAggregatorTests (use the existing FakeTimeProvider/manual-time pattern already in that file):

[Fact]
public void HeartbeatingSiteWithStaleReports_IsFlaggedMetricsStale_ButStaysOnline()
{
    // Review 01 [Medium]: a site whose HealthReportSender died keeps
    // heartbeating and previously showed "online with frozen metrics forever".
    var (aggregator, time) = NewAggregator(metricsStaleTimeout: TimeSpan.FromMinutes(2));
    aggregator.ProcessReport(Report("site-a", seq: 1));
    time.Advance(TimeSpan.FromMinutes(3));
    aggregator.MarkHeartbeat("site-a", time.GetUtcNow()); // heartbeats keep flowing
    aggregator.CheckForOfflineSites();
    var state = aggregator.GetSiteState("site-a")!;
    Assert.True(state.IsOnline);        // heartbeat-based liveness unchanged
    Assert.True(state.IsMetricsStale);  // NEW distinct signal
}

[Fact]
public void FreshReport_ClearsMetricsStale()
{
    var (aggregator, time) = NewAggregator(metricsStaleTimeout: TimeSpan.FromMinutes(2));
    aggregator.ProcessReport(Report("site-a", seq: 1));
    time.Advance(TimeSpan.FromMinutes(3));
    aggregator.CheckForOfflineSites();
    aggregator.ProcessReport(Report("site-a", seq: 2));
    Assert.False(aggregator.GetSiteState("site-a")!.IsMetricsStale);
}

[Fact]
public void OnlineOfflineTransitions_RecordLastStatusChangeAt()
{
    // Review 01 underdeveloped #7: "when did the site drop" was unanswerable.
    var (aggregator, time) = NewAggregator();
    aggregator.ProcessReport(Report("site-a", seq: 1));
    time.Advance(TimeSpan.FromMinutes(5)); // > OfflineTimeout
    aggregator.CheckForOfflineSites();
    var offlineAt = aggregator.GetSiteState("site-a")!.LastStatusChangeAt;
    Assert.Equal(time.GetUtcNow(), offlineAt);
    aggregator.MarkHeartbeat("site-a", time.GetUtcNow());
    Assert.NotEqual(offlineAt, aggregator.GetSiteState("site-a")!.LastStatusChangeAt);
}

Validator test: MetricsStaleTimeout must be positive and ≥ ReportInterval (a stale window shorter than one report interval flags every site). 2. Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests --filter "FullyQualifiedName~CentralHealthAggregatorTests" — expect FAIL. 3. Implement:

SiteHealthState: add public bool IsMetricsStale { get; init; } and public DateTimeOffset? LastStatusChangeAt { get; init; } (xmldoc each: stale = online but no report within MetricsStaleTimeout; LastStatusChangeAt = last online↔offline flip).
HealthMonitoringOptions: add public TimeSpan MetricsStaleTimeout { get; set; } = TimeSpan.FromMinutes(2); — validator rules as above.
CentralHealthAggregator.ProcessReport: in both the register and update branches set IsMetricsStale = false; in the update branch set LastStatusChangeAt = existing.IsOnline ? existing.LastStatusChangeAt : now. MarkHeartbeat: on the offline→online promotion set LastStatusChangeAt = now (leave IsMetricsStale untouched — heartbeats say nothing about metrics).
CheckForOfflineSites: when marking offline, offline = state with { IsOnline = false, LastStatusChangeAt = now }. Add a second pass (same loop) for online sites: if state.LastReportReceivedAt is { } lr && now - lr > _options.MetricsStaleTimeout && !state.IsMetricsStale, CAS to state with { IsMetricsStale = true } and LogWarning("Site {SiteId} metrics are stale — online (heartbeats) but no report for {Elapsed}s", ...). CAS-loss handling identical to the offline swap (lose ⇒ fresh data arrived ⇒ correct to skip).
Component-HealthMonitoring.md lines ~43-46: replace "offline = no report within 60s" with the two-signal model: liveness = no heartbeat/signal within OfflineTimeout (60s) → offline; metrics staleness = online but no report within MetricsStaleTimeout (2m) → "metrics stale" flag. Document LastStatusChangeAt. This closes the spec/code disagreement the report flagged.

Run the HealthMonitoring.Tests project — PASS.
Commit:

git add src/ZB.MOM.WW.ScadaBridge.HealthMonitoring tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests docs/requirements/Component-HealthMonitoring.md
git commit -m "feat(health): metrics-stale signal + status-transition timestamps; spec now matches heartbeat-liveness code"

Task 14: Surface metrics-stale + offline-since on the Health dashboard

Classification: small Estimated implement time: ~4 min Parallelizable with: 15, 16, 17, 19, 22, 23 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor
Test: tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/ (bUnit — grep for an existing Health page test class first: grep -rn "Health" tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests --include="*.cs" -l; extend it if present, else create HealthPageStalenessTests.cs following the nearest page-test pattern)

Write the failing bUnit test: render the page with a site state where IsMetricsStale = true and assert a .badge (existing Bootstrap badge classes used on the page) containing text Metrics stale is rendered; second case: IsOnline = false, LastStatusChangeAt = X renders offline since text containing the formatted timestamp.
Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests --filter "FullyQualifiedName~HealthPageStaleness" — expect FAIL.
Implement in Health.razor, on the site card/row where the online/offline badge already renders:

@if (state.IsOnline && state.IsMetricsStale)
{
    <span class="badge bg-warning text-dark ms-1"
          title="Site is heartbeating but has sent no health report for over @StaleTimeoutDisplay — the metrics pipeline may be dead.">
        Metrics stale
    </span>
}
@if (!state.IsOnline && state.LastStatusChangeAt is { } changedAt)
{
    <small class="text-muted ms-1">offline since @changedAt.ToString("u")</small>
}

(Match the page's existing badge markup/classes — clean corporate Bootstrap, no new components.) 4. Run — PASS. 5. Commit:

git add src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests
git commit -m "feat(ui): health dashboard shows metrics-stale badge and offline-since timestamp"

Task 15: Fix the metrics-port seed in appsettings.Site.json + close the validator gap

Classification: small Estimated implement time: ~4 min Parallelizable with: 1, 2, 3, 4, 5, 11, 13, 16, 17, 19, 22, 23 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json (lines 13-17)
Modify: src/ZB.MOM.WW.ScadaBridge.Host/StartupValidator.cs (seed loop, lines 117-127)
Test: tests/ZB.MOM.WW.ScadaBridge.Host.Tests/StartupValidatorTests.cs

Write the failing test (mirror the existing seed-vs-GrpcPort test in StartupValidatorTests):

[Fact]
public void Validate_SeedNodeTargetingMetricsPort_Fails()
{
    // Review 01 [Medium]: appsettings.Site.json shipped a seed pointing at the
    // Kestrel HTTP/1.1 metrics listener (8084) — a doomed Akka.Remote
    // association. The validator guarded GrpcPort but not MetricsPort.
    var config = SiteConfigWith(
        metricsPort: 8084,
        seedNodes: new[] { "akka.tcp://scadabridge@localhost:8082", "akka.tcp://scadabridge@localhost:8084" });
    var ex = Assert.Throws<StartupValidationException>(() => StartupValidator.Validate(config));
    Assert.Contains("must not target the metrics port", ex.Message);
}

(Use the existing config-building helper in that test file; match the actual exception type/validation surface used by the current seed-vs-grpc test.) 2. Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~SeedNodeTargetingMetricsPort" — expect FAIL. 3. Implement:

StartupValidator.cs: inside the existing foreach (var seed in seedNodes ...) loop add:

p.Require("ScadaBridge:Cluster:SeedNodes",
    _ => SeedNodePort(seed) != metricsPort,
    $"entry '{seed}' must not target the metrics port " +
    $"({metricsPort}); seed nodes must reference Akka remoting ports");

appsettings.Site.json: change the second seed "akka.tcp://scadabridge@localhost:8084" → "akka.tcp://scadabridge@localhost:8085" and add a "_seedNodes" comment key: "Host-0xx: second entry is the FUTURE node-b remoting port (8085) for a two-node localhost site. It must be an Akka remoting endpoint — never this node's GrpcPort (8083) or MetricsPort (8084); StartupValidator rejects both."
Sanity-check the docker per-node site configs don't carry the same defect: grep -n "SeedNodes" -A3 docker/site-*/appsettings.Site.json (they seed container-host:8082 pairs — expected clean; fix identically if not).

Run — PASS. Run the full StartupValidatorTests class.
Commit:

git add src/ZB.MOM.WW.ScadaBridge.Host/StartupValidator.cs src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json tests/ZB.MOM.WW.ScadaBridge.Host.Tests/StartupValidatorTests.cs
git commit -m "fix(host): dev site seed no longer targets the metrics port; validator rejects seed-vs-MetricsPort"

Task 16: Single-node installs — explicit AllowSingleNodeCluster instead of phantom seeds

Classification: small Estimated implement time: ~4 min Parallelizable with: 1, 2, 3, 4, 5, 11, 13, 15, 17, 19, 22, 23 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs
Modify: src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptionsValidator.cs (lines 24-34)
Modify: deploy/wonder-app-vd03/appsettings.Central.json (Cluster section, lines 9-19)
Test: tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsValidatorTests.cs

Write failing tests:

[Fact]
public void SingleSeed_WithoutAcknowledgement_Fails()
{
    var options = ValidOptions() with-like-mutation: SeedNodes = ["akka.tcp://scadabridge@h:8081"];
    AssertInvalid(options, "AllowSingleNodeCluster");
}

[Fact]
public void SingleSeed_WithAllowSingleNodeCluster_Passes()
{
    // Review 01 [Low]: the shipped single-node artifact satisfied the 2-seed
    // rule with a phantom seed the node dials forever. An explicit
    // acknowledgement flag replaces the fiction.
    var options = ValidOptions(); options.SeedNodes = new List<string> { "akka.tcp://scadabridge@h:8081" };
    options.AllowSingleNodeCluster = true;
    AssertValid(options);
}

[Fact]
public void EmptySeeds_FailsEvenWithFlag() { /* zero seeds always invalid */ }

(Adapt to the existing ClusterOptionsValidatorTests helper style.) 2. Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests --filter "FullyQualifiedName~SingleSeed" — expect FAIL. 3. Implement:

ClusterOptions: add public bool AllowSingleNodeCluster { get; set; } with xmldoc: "Acknowledges a deliberate single-node deployment: permits exactly one seed node (normally ≥2 are required so either node can start first). Without this flag a single-node install must list a phantom second seed, which the node dials forever — log noise and a defeated validation intent (review 01)."
ClusterOptionsValidator: replace the >= 2 rule with:

var minSeeds = options.AllowSingleNodeCluster ? 1 : 2;
builder.RequireThat(options.SeedNodes is not null && options.SeedNodes.Count >= minSeeds,
    options.AllowSingleNodeCluster
        ? "ClusterOptions.SeedNodes must contain at least 1 seed node."
        : "ClusterOptions.SeedNodes must contain at least 2 seed nodes "
          + "(Component-ClusterInfrastructure.md → Node Configuration: both nodes are seed nodes); "
          + "for a deliberate single-node install set ClusterOptions.AllowSingleNodeCluster = true instead of listing a phantom seed.");

deploy/wonder-app-vd03/appsettings.Central.json: remove the localhost:8091 phantom seed, add "AllowSingleNodeCluster": true, and rewrite _comment_Cluster to explain the flag ("single-node install acknowledged; add node B's real seed and remove the flag when it exists"). Validate JSON: python3 -m json.tool deploy/wonder-app-vd03/appsettings.Central.json > /dev/null.

Run the ClusterInfrastructure.Tests project — PASS.
Commit:

git add src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests deploy/wonder-app-vd03/appsettings.Central.json
git commit -m "feat(cluster): AllowSingleNodeCluster flag — wonder-app-vd03 drops its phantom seed"

Task 17: stop_grace_period for all app containers (both topologies)

Classification: trivial Estimated implement time: ~3 min Parallelizable with: everything except 21 Files:

Modify: docker/docker-compose.yml (every scadabridge:latest service)
Modify: docker-env2/docker-compose.yml (every scadabridge:latest service)

No unit test (compose config). Verification command is the "test": docker compose -f docker/docker-compose.yml config | grep -c "stop_grace_period" — expect 0 before, N (= number of app services, 8 in docker/, 4 in docker-env2/) after.
Add to each app service (central-a/b, all site nodes — NOT the traefik service, which has no CoordinatedShutdown):

    # CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
    # actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
    # mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
    stop_grace_period: 30s

Validate: docker compose -f docker/docker-compose.yml config > /dev/null && docker compose -f docker-env2/docker-compose.yml config > /dev/null (syntax), then the grep from step 1 returns 8 and 4.
Commit:

git add docker/docker-compose.yml docker-env2/docker-compose.yml
git commit -m "fix(docker): 30s stop_grace_period so graceful redeploys aren't SIGKILLed mid-CoordinatedShutdown"

Task 18: Evict deleted sites from the health aggregator

Classification: standard Estimated implement time: ~5 min Parallelizable with: 15, 16, 17, 19, 22, 23 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ICentralHealthAggregator.cs
Modify: src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthAggregator.cs
Modify: src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs (the periodic site-address refresh handler — locate with grep -n "Refresh" src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs; it reloads sites from the repository every 60s and on admin changes)
Test: tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/CentralHealthAggregatorTests.cs, tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/

Write the failing aggregator test:

[Fact]
public void PruneUnknownSites_RemovesDeletedSites_KeepsKnownAndCentral()
{
    // Review 01 [Low]: _siteStates only grew — a deleted site remained a
    // permanently-offline dashboard tile (and a KPI sample source) forever.
    var (aggregator, time) = NewAggregator();
    aggregator.ProcessReport(Report("site-a", 1));
    aggregator.ProcessReport(Report("site-b", 1));
    aggregator.ProcessReport(Report(CentralHealthReportLoop.CentralSiteId, 1));

    aggregator.PruneUnknownSites(new[] { "site-a" });

    Assert.NotNull(aggregator.GetSiteState("site-a"));
    Assert.Null(aggregator.GetSiteState("site-b"));
    Assert.NotNull(aggregator.GetSiteState(CentralHealthReportLoop.CentralSiteId)); // synthetic id always kept
}

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests --filter "FullyQualifiedName~PruneUnknownSites" — expect FAIL.
Implement:
- Interface + aggregator:

/// <summary>Removes tracked sites not in <paramref name="knownSiteIds"/> (deleted from configuration). The synthetic central id is always retained. Self-healing: called from the CentralCommunicationActor's periodic site refresh, so a deleted site disappears within one refresh interval without needing a deletion event.</summary>
public void PruneUnknownSites(IReadOnlyCollection<string> knownSiteIds)
{
    var known = new HashSet<string>(knownSiteIds, StringComparer.Ordinal);
    foreach (var siteId in _siteStates.Keys)
    {
        if (siteId == CentralHealthReportLoop.CentralSiteId || known.Contains(siteId))
            continue;
        if (_siteStates.TryRemove(siteId, out _))
            _logger.LogInformation("Site {SiteId} evicted from health aggregator (no longer configured)", siteId);
    }
}

In CentralCommunicationActor's site-refresh handler, after the fresh site list is loaded: _serviceProvider.GetService<ICentralHealthAggregator>()?.PruneUnknownSites(sites.Select(s => s.SiteIdentifier).ToList()); (match the identifier the aggregator is keyed on — it's the SiteId string sites report with; confirm by checking what HandleSiteHealthReport receives, and use the same property the refresh handler already uses for client-keying).
Add a Communication.Tests TestKit test: trigger the refresh message with a repo/enumerator fake returning ["site-a"] and an NSubstitute aggregator; assert aggregator.Received().PruneUnknownSites(...).

Run both test projects — PASS.
Commit:

git add src/ZB.MOM.WW.ScadaBridge.HealthMonitoring src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests tests/ZB.MOM.WW.ScadaBridge.Communication.Tests
git commit -m "fix(health): evict deleted sites from the aggregator on the periodic site refresh"

Task 19: Rate-limit dead-letter warnings

Classification: high-risk (actor change — timers) Estimated implement time: ~5 min Parallelizable with: 1, 2, 3, 4, 5, 11, 13, 15, 16, 17, 22, 23 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.Host/Actors/DeadLetterMonitorActor.cs
Test: tests/ZB.MOM.WW.ScadaBridge.Host.Tests/DeadLetterMonitorTests.cs

Write the failing test (extend DeadLetterMonitorTests; use an NSubstitute ILogger<DeadLetterMonitorActor> and count Log invocations at LogLevel.Warning):

[Fact]
public void DeadLetterStorm_LogsAtMostWindowLimit_ButCountsAll()
{
    // Review 01 [Low]: a failover/undeploy storm produced one Warning per
    // dead letter — unbounded log flood. Health metric counting is untouched.
    var logger = Substitute.For<ILogger<DeadLetterMonitorActor>>();
    var monitor = Sys.ActorOf(Props.Create(() => new DeadLetterMonitorActor(logger, null)));
    for (var i = 0; i < 50; i++)
        monitor.Tell(new DeadLetter($"m{i}", TestActor, TestActor));

    var count = monitor.Ask<DeadLetterCountResponse>(GetDeadLetterCount.Instance).Result;
    Assert.Equal(50, count.Count); // every dead letter still counted
    var warnings = logger.ReceivedCalls().Count(c =>
        c.GetMethodInfo().Name == "Log" && (LogLevel)c.GetArguments()[0]! == LogLevel.Warning);
    Assert.True(warnings <= DeadLetterMonitorActor.MaxWarningsPerWindow,
        $"expected <= {DeadLetterMonitorActor.MaxWarningsPerWindow} warnings, got {warnings}");
}

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~DeadLetterStorm" — expect FAIL (50 warnings).
Implement: make the actor IWithTimers; add public const int MaxWarningsPerWindow = 10; and internal static readonly TimeSpan WarningWindow = TimeSpan.FromMinutes(1); In the DeadLetter handler:

_deadLetterCount++;
_healthCollector?.IncrementDeadLetter();
if (_warningsThisWindow < MaxWarningsPerWindow)
{
    _warningsThisWindow++;
    logger.LogWarning("Dead letter: {MessageType} from {Sender} to {Recipient}",
        dl.Message.GetType().Name, dl.Sender, dl.Recipient);
    if (_warningsThisWindow == MaxWarningsPerWindow)
        logger.LogWarning("Dead-letter warning limit ({Limit}) reached — suppressing per-letter warnings for {Window}",
            MaxWarningsPerWindow, WarningWindow);
}
else
{
    _suppressedThisWindow++;
}

Plus a private WindowTick timer message started in PreStart (Timers.StartPeriodicTimer("dl-window", WindowTick.Instance, WarningWindow)) whose handler logs one summary if _suppressedThisWindow > 0 (LogWarning("Suppressed {Count} dead letters in the last {Window}", ...)) and resets both window counters. Update the class xmldoc. 4. Run — PASS (the summary tick fires only after 1 min, so the test sees ≤ 11 warnings; assert against MaxWarningsPerWindow + 1 if the limit-reached notice counts). 5. Commit:

git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/DeadLetterMonitorActor.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/DeadLetterMonitorTests.cs
git commit -m "fix(host): rate-limit dead-letter warnings (10/min + suppression summary); metric counting unchanged"

Task 20: Down-if-alone recovery story — exit on unexpected ActorSystem termination + service restart actions

Classification: high-risk (process lifecycle) Estimated implement time: ~5 min Parallelizable with: 15, 16, 22, 23 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs (ctor DI + StartAsync/GetOrCreateActorSystem + StopAsync flag)
Modify: deploy/wonder-app-vd03/install.ps1 (service recovery actions)
Modify: docs/requirements/Component-ClusterInfrastructure.md (new "Down-if-alone recovery" subsection near the split-brain section, ~lines 110-119)
Test: tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ (new UnexpectedTerminationTests.cs)

Context: with SBR enabled (Task 1), keep-oldest down-if-alone = on means a node that finds itself alone-and-partitioned downs ITSELF, and run-coordinated-shutdown-when-down = on (now live) terminates its ActorSystem. Today the Host process would keep running with a dead actor system — serving nothing, restarted by nobody. The recovery contract: unexpected ActorSystem termination ⇒ stop the host process ⇒ the supervisor (docker restart: unless-stopped / Windows service recovery) restarts it ⇒ it rejoins as a fresh incarnation.

Write the failing test:

[Fact]
public async Task ActorSystemTerminatedOutsideStopAsync_StopsHostApplication()
{
    // Down-if-alone self-down (SBR) terminates the ActorSystem via
    // run-coordinated-shutdown-when-down. The process must exit so the
    // service supervisor restarts it — otherwise the node idles forever
    // (review 01 underdeveloped #2).
    var lifetime = Substitute.For<IHostApplicationLifetime>();
    var service = CreateAkkaHostedService(lifetime); // existing Host.Tests construction pattern
    await service.StartAsync(CancellationToken.None);
    var system = service.ActorSystem!;

    await system.Terminate(); // simulated self-down, NOT StopAsync

    await Task.Delay(500);
    lifetime.Received(1).StopApplication();
}

[Fact]
public async Task StopAsync_DoesNotTriggerStopApplication()
{
    var lifetime = Substitute.For<IHostApplicationLifetime>();
    var service = CreateAkkaHostedService(lifetime);
    await service.StartAsync(CancellationToken.None);
    await service.StopAsync(CancellationToken.None);
    lifetime.DidNotReceive().StopApplication();
}

(Reuse whatever construction pattern AkkaHostedServiceAuditWiringTests/HostStartupTests use for building the service with a test DI container; add IHostApplicationLifetime as an OPTIONAL ctor parameter — nullable, resolved via GetService — so every existing construction site keeps compiling.) 2. Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~UnexpectedTermination" — expect FAIL. 3. Implement:

AkkaHostedService: add private volatile bool _stopRequested; set to true first thing in StopAsync. After the system is created and published (_actorSystem = system; in GetOrCreateActorSystem), register once:

system.WhenTerminated.ContinueWith(_ =>
{
    if (_stopRequested) return;
    _logger.LogCritical(
        "ActorSystem terminated outside host shutdown (SBR down / run-coordinated-shutdown-when-down). "
        + "Stopping the host process so the service supervisor restarts this node as a fresh incarnation.");
    _appLifetime?.StopApplication();
}, TaskContinuationOptions.ExecuteSynchronously);

install.ps1: after the sc.exe create/New-Service call, add recovery actions (find the actual service name in the script first):

# Restart-on-failure: required for the SBR down-if-alone recovery contract —
# a self-downed node exits the process and MUST be restarted externally.
& sc.exe failure $ServiceName reset= 86400 actions= restart/5000/restart/30000/restart/60000
& sc.exe failureflag $ServiceName 1

Component-ClusterInfrastructure.md: add a "Down-if-alone recovery" subsection: self-down ⇒ CoordinatedShutdown ⇒ process exit ⇒ supervisor restart (docker restart: unless-stopped; Windows service recovery actions) ⇒ clean rejoin as a new incarnation; note the drill in Task 21's script.

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests — PASS (full project; the new ctor param must not break existing tests).
Commit:

git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs deploy/wonder-app-vd03/install.ps1 docs/requirements/Component-ClusterInfrastructure.md tests/ZB.MOM.WW.ScadaBridge.Host.Tests/UnexpectedTerminationTests.cs
git commit -m "feat(host): exit process on unexpected ActorSystem termination — completes the down-if-alone recovery loop"

Task 21: Docker failover drill script (the manual/CI kill test against the real topology)

Classification: small Estimated implement time: ~5 min Parallelizable with: 22, 23 Files:

Create: docker/failover-drill.sh
Modify: docker/README.md (new "Failover drill" section), deployments/docker-cluster.md (cross-reference)

No unit test — the script IS the test. Create:

#!/usr/bin/env bash
# Failover drill against the running docker cluster (bash docker/deploy.sh first).
# Kills the ACTIVE central node (docker kill = SIGKILL, the hard-crash path that
# review 01 found had NO automatic recovery before the SBR downing provider was
# enabled) and measures how long Traefik takes to route to the survivor.
set -euo pipefail

TRAEFIK_URL="${TRAEFIK_URL:-http://localhost:9000}"
TIMEOUT_S="${TIMEOUT_S:-90}"

active_container() {
  if curl -sf -o /dev/null "http://localhost:9001/health/active"; then echo scadabridge-central-a
  elif curl -sf -o /dev/null "http://localhost:9002/health/active"; then echo scadabridge-central-b
  else echo "ERROR: no active central node found" >&2; exit 1; fi
}

VICTIM=$(active_container)
echo "Active central node: ${VICTIM} — killing it (SIGKILL, crash path)"
docker kill "${VICTIM}" > /dev/null
START=$(date +%s)

echo "Waiting for the survivor to become active through Traefik (${TRAEFIK_URL}/health/active)..."
while true; do
  ELAPSED=$(( $(date +%s) - START ))
  if curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; then
    echo "PASS: failover complete in ${ELAPSED}s (design budget ~25s + Traefik health interval)"
    break
  fi
  if (( ELAPSED > TIMEOUT_S )); then
    echo "FAIL: no active central node after ${ELAPSED}s — SBR/singleton handover did not recover" >&2
    docker start "${VICTIM}" > /dev/null
    exit 1
  fi
  sleep 1
done

SURVIVOR=$([ "${VICTIM}" = scadabridge-central-a ] && echo scadabridge-central-b || echo scadabridge-central-a)
echo "Survivor singleton evidence (last 20 matching log lines from ${SURVIVOR}):"
docker logs "${SURVIVOR}" 2>&1 | grep -Ei "singleton|oldest|Downing|Removed" | tail -20 || true

echo "Restarting ${VICTIM} and waiting for it to rejoin..."
docker start "${VICTIM}" > /dev/null
sleep 5
echo "Drill complete. Verify on the Health dashboard that both nodes show Up and the survivor is Primary."

chmod +x docker/failover-drill.sh. Verify statically: bash -n docker/failover-drill.sh (syntax). If a cluster is running (bash docker/deploy.sh), run the drill end-to-end and record the observed failover time in docker/README.md; if not available in this session, mark the README section "run after next deploy" — the script is still committed.
Document in docker/README.md: what it does, the ~25s + Traefik 5s health-interval expectation, and that it exercises S1 (SBR downing), S3 (single-active through Traefik) and Task 20 (restart/rejoin). Cross-reference from deployments/docker-cluster.md. Note for other plans: plan 02's duplicate-delivery checks can extend this script.
Commit:

git add docker/failover-drill.sh docker/README.md deployments/docker-cluster.md
git commit -m "test(docker): failover drill script — SIGKILL the active central node, assert Traefik recovery"

Task 22: Reconcile REQ-HOST-6 (Akka.Hosting) with the hand-rolled bootstrap

Classification: small Estimated implement time: ~4 min Parallelizable with: 15, 16, 17, 19, 21, 23 Files:

Modify: docs/requirements/Component-Host.md (REQ-HOST-6, ~lines 106-114)
Modify (conditional): src/ZB.MOM.WW.ScadaBridge.Host/ZB.MOM.WW.ScadaBridge.Host.csproj (lines 15-18) + Directory.Packages.props

Decision: keep the hand-rolled HOCON (it is hardened — QuoteHocon/DurationHocon, validated options — and now carries the downing provider explicitly per Task 1); amend the spec instead of a risky bootstrap migration. The report accepts either resolution.

Verify actual package usage: grep -rn "using Akka.Hosting\|using Akka.Cluster.Hosting\|using Akka.Remote.Hosting\|AkkaConfigurationBuilder\|\.WithClustering\|\.WithRemoting" src/ --include="*.cs" — expect zero hits (review found them referenced-but-unused).
If zero hits: remove <PackageReference Include="Akka.Cluster.Hosting" />, <PackageReference Include="Akka.Hosting" />, <PackageReference Include="Akka.Remote.Hosting" /> from the Host csproj (keep Akka.Cluster.Tools), remove the matching <PackageVersion> lines from Directory.Packages.props only if no other project references them (grep -rn "Akka.Hosting\|Akka.Cluster.Hosting\|Akka.Remote.Hosting" src/*/[A-Z]*.csproj tests/*/[A-Z]*.csproj), and check whether the csproj's OpenTelemetry.Api transitive-override comment (lines ~28-32) still applies — if it referenced Akka.Hosting's pin, delete the override too.
Amend Component-Host.md REQ-HOST-6: replace "bootstrap via Akka.Hosting" with: bootstrap is a hand-assembled, injection-safe HOCON document (AkkaHostedService.BuildHocon) + ActorSystem.Create; the downing provider (Akka.Cluster.SBR.SplitBrainResolverProvider) is set explicitly — Akka's default is NoDowning and forgetting this line silently disables failover (this exact omission was review 01's Critical finding, caught because tests now assert the parsed key and a two-node kill test asserts the behavior). Note the deliberate rejection of Akka.Cluster.Hosting typed options with a one-line rationale (single hardened code path already in production shape; migration tracked as possible future work).
Build: dotnet build ZB.MOM.WW.ScadaBridge.slnx — expect success. Run dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~HoconBuilderTests" — green.
Commit:

git add docs/requirements/Component-Host.md src/ZB.MOM.WW.ScadaBridge.Host/ZB.MOM.WW.ScadaBridge.Host.csproj Directory.Packages.props
git commit -m "docs(host): REQ-HOST-6 documents the hand-rolled HOCON bootstrap; drop unused Akka.Hosting packages"

Task 23: Docs & deploy-config cleanup batch (Traefik doc, NodeName overlays, TLS roadmap note)

Classification: trivial Estimated implement time: ~4 min Parallelizable with: 15, 16, 17, 19, 21, 22 Files:

Modify: docs/requirements/Component-TraefikProxy.md (lines ~115, ~121, TLS section ~131-138)
Modify: deploy/wonder-app-vd03/appsettings.Central.json (Node section), deploy/wonder-app-vd03/appsettings.Site.json (Node section, if the file exists — check ls deploy/wonder-app-vd03/)

Component-TraefikProxy.md:
- Line ~121: replace the stale src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeHealthCheck.cs location with src/ZB.MOM.WW.ScadaBridge.Host/Health/OldestNodeActiveHealthCheck.cs (the Host-local oldest-member check from Task 7) and note /health/active also requires the database check.
- Line ~115: update the /health/ready description from "database + Akka cluster" to "database + akka-cluster + required-singletons" (matches Program.cs and Component-Host REQ-HOST-4a).
- TLS section (~131-138): append an explicit roadmap paragraph — "Production TLS profile: not yet implemented. Traefik terminates plain HTTP, Akka remoting is unencrypted TCP, and site gRPC is http://. Acceptable for the lab topology only; a production deployment requires TLS at all three layers and a secured Traefik dashboard. Tracked as an open roadmap item (arch review 01, underdeveloped area 3)." — this makes the accepted gap spec-visible without pretending to fix it.
deploy/wonder-app-vd03/appsettings.Central.json: add "NodeName": "central-a" to the Node section (per the base config's documented convention — without it every audit row from the one real deployment gets a NULL SourceNode, undermining IX_AuditLog_Node_Occurred and the per-node stuck KPIs). Same for the Site overlay if present ("NodeName": "node-a"). Validate: python3 -m json.tool deploy/wonder-app-vd03/appsettings.Central.json > /dev/null.
Cross-check no other stale ActiveNodeHealthCheck doc references remain: grep -rn "ActiveNodeHealthCheck" docs/ README.md — fix any found the same way.
Commit:

git add docs/requirements/Component-TraefikProxy.md deploy/wonder-app-vd03/
git commit -m "docs(cleanup): sync Traefik doc with shared health checks, add TLS roadmap note, NodeName in deploy overlays"

Dependencies on other plans

Plan 02 (Communication & S&F) depends on THIS plan for two facilities — design them exactly as specified here so the seam holds:
- The canonical site-side "active node / singleton host" check: after Task 6, IClusterNodeProvider.SelfIsPrimary (site DI registration at SiteServiceRegistration.cs:101-107) means "oldest Up member in the site role = the node hosting the DeploymentManager singleton". Plan 02's standby S&F delivery gating must consume this (directly or via a registered delegate mirroring SiteEventLogActiveNodeCheck), not reinvent a leader check.
- The two-node failover rig: TwoNodeClusterFixture (Task 3) is public and production-HOCON-based; plan 02's ClusterClient-recreate and S&F non-duplication tests should build on it, and the docker drill script (Task 21) is the shared live-cluster harness.
Plan 07 (UI/Management/Security) owns the DisableLogin: true deploy artifact (C2) and the deploy-artifact LDAP plaintext posture (C3) — this plan touches deploy/wonder-app-vd03/appsettings.Central.json in Tasks 16 and 23 (Cluster + Node sections only); coordinate merges but do not touch the Security section here.
No other plan blocks this one.

Execution order

P0 (do first, in order): Task 1 (the Critical one-liner) → Task 3 (fixture) → Task 4 (behavioral proof). Ship these three even if nothing else lands — they convert "failover does not work" into "failover works and is regression-tested".

Critical path: 1 → 3 → 4 → 8 (needs 6), with 5 → 6 → 7 as the second spine (active-node unification). Task 2 must precede 9/10 (drain budget before drain refactor); 11 → 12 (message before transport); 13 → 14 and 13 → 18 (state shape before consumers). Task 20 touches AkkaHostedService.cs and should follow 10 to avoid merge churn; Task 21 follows 1 (drill is meaningless before SBR) and ideally 17.

Freely parallel at any point: 15, 16, 17, 19, 22, 23.

89 KiB Raw Blame History