ScadaBridge/archreview/plans/PLAN-R2-01-cluster-host-failover.md

PLAN-R2-01 — Cluster, Host & Failover Round-2 Fixes Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.

Goal: Make the failover story honest — rewrite and actually RUN the failover drill so it proves what two-node keep-oldest can deliver (and measures what it cannot), apply the three wonder-app-vd03 overlay edits that were deferred on a factually wrong rationale, and close the round-2 residuals (never-reported metrics staleness, stale leader comments, undrained site singletons).

Architecture: The drill (docker/failover-drill.sh) gains a DRILL_MODE — standby (default, the survivable younger-node crash that PASSes within the ~25s SBR budget) and active (the oldest-node crash that keep-oldest cannot survive: the drill expects the outage and measures it, documenting the registered deferred user decision instead of codifying a recovery the topology cannot deliver). The in-process envelope measurement reuses TwoNodeClusterFixture (production BuildHocon, now with production-timing knobs) from the previously-skipped FailoverTimingTests. Site-role singleton drains come from generalizing CentralSingletonRegistrar into a role-aware SingletonRegistrar so deployment-manager and event-log-handler get the same PhaseClusterLeave GracefulStop the seven central singletons got in round 1.

Tech Stack: C#/.NET, Akka.NET 1.5.62 (Akka.Cluster, Akka.Cluster.Tools), xUnit, bash + Docker Compose + Traefik, JSON deploy overlays (PowerShell install.ps1).

Build: dotnet build ZB.MOM.WW.ScadaBridge.slnx. Test per-project: dotnet test tests/<project>. Docker rig: bash docker/deploy.sh.

Binding scope rulings (do not deviate):

• The two-node keep-oldest active/oldest-crash availability gap is a registered deferred USER decision (master tracker 2026-07-08 "Follow-up discovered during P0 execution"; SbrFailoverTests.cs:15-23). No task in this plan changes the SBR strategy or topology — Tasks 1–3 document and measure the gap.
• The oldest-vs-leader predicate unification inside SiteCommunicationActor/SiteReplicationActor and everything resync-related is owned by PLAN-R2-02 — do not touch those files here.
• N6 (deferred-work-register consolidation) is owned by PLAN-R2-08 — Task 7 here corrects only the factually wrong N2 prose in the master tracker.
• /deploy/ is gitignored (.gitignore:48: /deploy/) — Tasks 5 and 6 edit the on-disk artifact and deliberately do NOT git add it (never use git add -f); Task 7's tracker correction is the version-controlled record.

---

Task 1: Rewrite failover-drill.sh — standby-victim default + explicit active-victim gap mode

Classification: small Estimated implement time: ~5 min Parallelizable with: Task 4, Task 5, Task 6, Task 8, Task 9, Task 10 Files:

• Modify: docker/failover-drill.sh (full rewrite — current script is 45 lines; lines 12-19 pick the ACTIVE node as victim, which is exactly the crash keep-oldest cannot survive)

Context (round-2 N1): under the round-1 oldest-member unification, the active node IS the oldest. docker/failover-drill.sh:17-19 kills it, so the shipped drill exercises the one crash two-node keep-oldest is known not to recover from (the survivor self-downs; the restarted central-b cannot re-bootstrap because both nodes list scadabridge-central-a as FIRST seed — docker/central-node-a/appsettings.Central.json:10-13, docker/central-node-b/appsettings.Central.json:10-13 — and only the first seed may self-join to form a new cluster). The drill has also never been run (docker/README.md:288). Fix the drill so its PASS criteria match reality per direction.

1. No unit test (bash script). The verification command is the "test": bash -n docker/failover-drill.sh (must exit 0), plus DRILL_MODE=bogus bash docker/failover-drill.sh (must exit 2 with the usage error — this exercises the argument guard without needing a cluster).
2. Replace the script's entire contents with:

#!/usr/bin/env bash
# Failover drill against the running docker cluster (bash docker/deploy.sh first).
#
# ROUND-2 REWRITE (arch-review 01 round 2, N1). The original drill killed the
# ACTIVE central node — but under the unified oldest-member semantics the
# active node IS the oldest, i.e. the one crash two-node keep-oldest CANNOT
# survive (registered deferred user decision, master tracker 2026-07-08;
# SbrFailoverTests.cs XML doc). Two modes:
#
#   DRILL_MODE=standby (default) — kills the STANDBY (younger) central node.
#     The survivable direction: SBR downs the crashed member, the active node
#     keeps its singletons, and Traefik routing never goes dark. PASS = the
#     survivor logs the member removal within TIMEOUT_S (budget ~25s+: 10s
#     failure detection + 15s stable-after) while /health/active stays up.
#
#   DRILL_MODE=active — kills the ACTIVE (oldest) central node. THE EXPECTED
#     OUTCOME IS A TOTAL CENTRAL OUTAGE: keep-oldest downs the partition
#     without the oldest, so the younger survivor downs ITSELF (down-if-alone
#     cannot help — the alone-oldest is dead and cannot down itself), and the
#     self-downed survivor cannot re-form a cluster alone unless it is the
#     FIRST seed (both nodes list central-a first; only the first seed may
#     self-join). This mode measures the dark window and PASSes only when
#     central recovers AFTER the victim container is restarted. It exists to
#     make the registered gap observable — not to pretend it is covered.
set -euo pipefail

TRAEFIK_URL="${TRAEFIK_URL:-http://localhost:9000}"
TIMEOUT_S="${TIMEOUT_S:-90}"
DRILL_MODE="${DRILL_MODE:-standby}"
OUTAGE_CONFIRM_S="${OUTAGE_CONFIRM_S:-60}"

active_container() {
  if curl -sf -o /dev/null "http://localhost:9001/health/active"; then echo scadabridge-central-a
  elif curl -sf -o /dev/null "http://localhost:9002/health/active"; then echo scadabridge-central-b
  else echo "ERROR: no active central node found" >&2; exit 1; fi
}
peer_of() { [ "$1" = scadabridge-central-a ] && echo scadabridge-central-b || echo scadabridge-central-a; }

case "$DRILL_MODE" in
  standby|active) ;;
  *) echo "ERROR: DRILL_MODE must be 'standby' or 'active' (was '$DRILL_MODE')" >&2; exit 2 ;;
esac

ACTIVE=$(active_container)
if [ "$DRILL_MODE" = standby ]; then
  VICTIM=$(peer_of "$ACTIVE"); SURVIVOR="$ACTIVE"
else
  VICTIM="$ACTIVE"; SURVIVOR=$(peer_of "$ACTIVE")
fi

echo "mode=${DRILL_MODE}  active=${ACTIVE}  victim=${VICTIM}  survivor=${SURVIVOR}"
KILL_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)
docker kill "${VICTIM}" > /dev/null
START=$(date +%s)

if [ "$DRILL_MODE" = standby ]; then
  echo "Standby crash: waiting for ${SURVIVOR} to DOWN+REMOVE the dead member (SBR budget ~25s)..."
  BLIPS=0
  while true; do
    ELAPSED=$(( $(date +%s) - START ))
    curl -sf -o /dev/null "${TRAEFIK_URL}/health/active" || BLIPS=$((BLIPS + 1))
    if docker logs --since "${KILL_AT}" "${SURVIVOR}" 2>&1 | grep -Eiq "marking.*node.*down|member removed|is removed"; then
      echo "PASS: survivor removed the crashed member in ${ELAPSED}s (budget ~25s: 10s detection + 15s stable-after)."
      echo "Active-node routing blips during the drill: ${BLIPS} (expected 0 — the active node was never touched)."
      break
    fi
    if (( ELAPSED > TIMEOUT_S )); then
      echo "FAIL: no downing/removal evidence on ${SURVIVOR} after ${ELAPSED}s — SBR did not act" >&2
      docker start "${VICTIM}" > /dev/null
      exit 1
    fi
    sleep 1
  done
else
  echo "Active crash: EXPECTING a central outage (registered keep-oldest gap). Watching /health/active..."
  DARK_STREAK=0
  while true; do
    ELAPSED=$(( $(date +%s) - START ))
    if curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; then DARK_STREAK=0; else DARK_STREAK=$((DARK_STREAK + 1)); fi
    if (( DARK_STREAK >= 10 )); then
      echo "Outage confirmed at ${ELAPSED}s: no active central node — the younger survivor self-downed"
      echo "(keep-oldest downs the partition WITHOUT the oldest; this is the registered deferred gap)."
      break
    fi
    if (( ELAPSED > OUTAGE_CONFIRM_S )); then
      echo "NOTE: /health/active stayed reachable ${ELAPSED}s after killing the oldest — better than the"
      echo "registered gap predicts. Do NOT celebrate: capture both nodes' logs and investigate before trusting it."
      break
    fi
    sleep 1
  done
fi

echo "Restarting ${VICTIM}..."
docker start "${VICTIM}" > /dev/null
RESTART_AT=$(date +%s)
echo "Waiting for central to be routable again through Traefik (${TRAEFIK_URL}/health/active)..."
while true; do
  ELAPSED=$(( $(date +%s) - RESTART_AT ))
  if curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; then
    echo "Recovered: an active central node is routable ${ELAPSED}s after the victim restart."
    break
  fi
  if (( ELAPSED > 120 )); then
    echo "FAIL: central not routable 120s after restarting ${VICTIM}" >&2
    exit 1
  fi
  sleep 1
done

echo "Survivor singleton/downing evidence (last 20 matching log lines from ${SURVIVOR}):"
docker logs "${SURVIVOR}" 2>&1 | grep -Ei "singleton|oldest|downing|removed" | tail -20 || true
echo "Drill complete (${DRILL_MODE}). Verify on the Health dashboard that both nodes show Up and exactly one is Primary."

3. chmod +x docker/failover-drill.sh. Verify: bash -n docker/failover-drill.sh (exit 0) and DRILL_MODE=bogus bash docker/failover-drill.sh; echo "exit=$?" (prints the usage error, exit=2).
4. Commit:

git add docker/failover-drill.sh
git commit -m "fix(docker): failover drill kills the STANDBY by default; active mode measures the registered keep-oldest outage instead of pretending recovery (plan R2-01 T1)"

Task 2: Correct the recovery narrative + document the first-seed bootstrap constraint

Classification: trivial (doc-only) Estimated implement time: ~5 min Parallelizable with: Task 4, Task 5, Task 6, Task 8, Task 9, Task 10 Files:

• Modify: docker/README.md (Failover Testing → "Automated failover drill" section, lines 273-288)
• Modify: docs/requirements/Component-ClusterInfrastructure.md ("Down-if-alone recovery" subsection, lines ~104-116)

1. docker/README.md (lines 273-288): rewrite the drill section to match Task 1's two modes. Specifically:
   • Replace the four numbered bullets with per-mode descriptions: DRILL_MODE=standby (default) = the survivable younger-crash direction — expected result: no routing outage at all (the active node is never touched) and member removal on the survivor within ~25s (10s failure-detection threshold + 15s stable-after; the 2s heartbeat interval is not additive); DRILL_MODE=active = the oldest-crash direction — expected result: total central outage until the victim container is restarted (the registered deferred keep-oldest decision, master tracker 2026-07-08), then recovery within ~2 min of the restart.
   • Delete the line-286 promise "Expected failover: ~25s … plus the Traefik ~5s health-check interval" — it described a singleton migration on active-crash that two-node keep-oldest cannot deliver. State per-direction expectations instead.
   • Replace the line-288 "Observed failover time: run after next deploy and record here" placeholder with a two-row results table Task 3 will fill:

> **Observed results** (recorded by plan R2-01 T3, cluster @ commit <sha>):
>
> | Direction (`DRILL_MODE`) | Outcome | Measured |
> |--------------------------|---------|----------|
> | `standby` (younger-node crash) | _pending T3_ | _pending T3_ |
> | `active` (oldest-node crash) | _pending T3_ | _pending T3_ |

• Add a short "Seed-node bootstrap constraint" note to the same section: only the FIRST seed in Cluster:SeedNodes may self-join to form a new cluster; both central nodes list scadabridge-central-a first, so a lone restarted central-b (with central-a still down) loops on InitJoin forever — never Up, never /health/active 200. Operator recovery actions: (1) restart the dead first-seed node (preferred), or (2) restart the survivor with a self-first seed override (env: ScadaBridge__Cluster__SeedNodes__0=akka.tcp://scadabridge@<self-host>:8081, ...__1=<peer>). Explain why the repo does NOT ship self-first ordering per node: with both nodes self-first, a simultaneous cold start can let each self-join independently → two one-node clusters that never merge (a cold-start split-brain the identical-seed-order convention exists to prevent). The real remedy is the pending keep-oldest topology/strategy decision (deferred, owner: user).

2. docs/requirements/Component-ClusterInfrastructure.md (Down-if-alone recovery, ~104-116):
   • Amend step 4 ("The restarted process rejoins as a fresh incarnation (the keep-oldest resolver handles the rejoin cleanly…)"): add the qualifier that this holds only while a peer holding cluster state is still reachable — a lone restarted non-first-seed node cannot re-form a cluster (first-seed self-join rule) and waits for its peer.
   • Add a "Seed-node bootstrap constraint" paragraph after the numbered list (same content as the README note, spec-toned), cross-referencing the registered deferred keep-oldest decision and the operator actions.
   • Fix the closing line "The docker failover drill exercises this loop end-to-end (kill the active node, assert singleton migration + clean rejoin)" → "The docker failover drill (docker/failover-drill.sh) exercises this per direction: standby mode proves SBR downing + singleton continuity on the oldest; active mode measures the registered total-outage gap and the recovery-on-restart path."
3. Verification (doc task — no test): grep -n "DRILL_MODE" docker/README.md (≥2 hits), grep -cn "singleton migration" docs/requirements/Component-ClusterInfrastructure.md (0 hits in the recovery section), grep -n "first seed\|first-seed" docker/README.md docs/requirements/Component-ClusterInfrastructure.md (≥2 hits), grep -n "Traefik ~5s health-check interval" docker/README.md (0 hits).
4. Commit:

git add docker/README.md docs/requirements/Component-ClusterInfrastructure.md
git commit -m "docs(cluster): per-direction failover truth — first-seed bootstrap constraint + operator recovery actions; drop the undeliverable ~25s active-crash promise (plan R2-01 T2)"

Task 3: RUN the drill live (both directions) and record measured timings

Classification: standard (live-cluster verification; ~5 min of work, several minutes of wall-clock waiting) Estimated implement time: ~5 min Parallelizable with: none (exclusive use of the shared docker cluster; writes docker/README.md) Files:

• Modify: docker/README.md (fill the Task 2 results table)

The drill has NEVER been executed (round-2 N1 point 3, docker/README.md:288). This task produces the empirical record — the asymmetric result is the documentation the deferred keep-oldest decision needs.

1. Ensure infra + cluster are up and freshly built (drill needs the Task 1 script inside the checkout only — no image change required, but the cluster must be healthy):

cd infra && docker compose up -d && cd ..
bash docker/deploy.sh

Wait until curl -sf http://localhost:9000/health/active returns 200 (poll; allow ~2 min for first-boot readiness). 2. Run direction 1 and capture:

DRILL_MODE=standby bash docker/failover-drill.sh 2>&1 | tee /private/tmp/claude-501/-Users-dohertj2-Desktop-ScadaBridge/d69633c2-b169-4fe1-8fb4-0d8007a16fa1/scratchpad/drill-standby.log

Expected: PASS with member-removal time ≈ 25-30s and routing blips: 0. Wait for the victim to fully rejoin (script does this) plus ~30s settle time. 3. Run direction 2 and capture:

DRILL_MODE=active bash docker/failover-drill.sh 2>&1 | tee /private/tmp/claude-501/-Users-dohertj2-Desktop-ScadaBridge/d69633c2-b169-4fe1-8fb4-0d8007a16fa1/scratchpad/drill-active.log

Expected: "Outage confirmed at ~Ns" (survivor self-downs; central dark), then "Recovered: … Ms after the victim restart". If instead the NOTE branch fires (/health/active stayed up), capture docker logs scadabridge-central-a + -b and investigate before recording — do not record a result that contradicts the registered gap without evidence. 4. Fill the docker/README.md results table from the two logs: outcome + measured seconds per direction, and the cluster commit SHA (git rev-parse --short HEAD). Keep the raw one-line summary quotes from the script output. 5. Sanity-restore: confirm both nodes healthy afterwards — curl -sf http://localhost:9001/health/ready && curl -sf http://localhost:9002/health/ready (both 200; the standby's /health/active correctly 503). 6. Commit:

git add docker/README.md
git commit -m "test(docker): record live failover-drill results for both crash directions — standby-crash PASS timing + active-crash measured outage (plan R2-01 T3)"

Task 4: Wire FailoverTimingTests to TwoNodeClusterFixture — measure the ~25s envelope in-process

Classification: high-risk (real two-node cluster timing test) Estimated implement time: ~5 min Parallelizable with: Task 1, Task 2, Task 5, Task 6, Task 7, Task 8, Task 9, Task 10, Task 11 Files:

• Modify: tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/TwoNodeClusterFixture.cs (StartAsync lines 24-36, StartNode lines 39-65 — add production-timing knobs)
• Modify: tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/ZB.MOM.WW.ScadaBridge.PerformanceTests.csproj (add a ProjectReference)
• Modify: tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Failover/FailoverTimingTests.cs (replace the Skip placeholder — currently [Fact(Skip = "Requires the real two-node failover rig delivered by PLAN-01 …")] at lines 25-32)

This task also covers architecture-review report 08's NF2 by reference (the skipped perf placeholder): the rig it waits for exists (TwoNodeClusterFixture, PLAN-01 T3); wire it up. The fixture hard-codes scaled timings (HeartbeatInterval 500ms / FailureDetectionThreshold 2s at TwoNodeClusterFixture.cs:51-52); to measure the production ~25s envelope (2s heartbeat / 10s threshold / 15s stable-after), the knobs become optional parameters with the current values as defaults (additive — SbrFailoverTests/ActiveNodeSemanticsTests unaffected). Direction note: the measurable crash is the YOUNGER node (the survivable direction); the oldest-crash direction is the registered deferred gap and is not a recovery to time — say so in the test's XML doc.

1. Rewrite FailoverTimingTests.cs (this is the failing step — it will not compile until the fixture knobs + project reference exist):

using System.Diagnostics;
using Akka.Actor;
using Akka.Cluster.Tools.Singleton;
using Xunit.Abstractions;
using ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;

namespace ZB.MOM.WW.ScadaBridge.PerformanceTests.Failover;

/// <summary>
/// Failover-timing measurement on the real two-node in-process rig
/// (TwoNodeClusterFixture, production BuildHocon) at PRODUCTION timings:
/// 2s heartbeat / 10s failure-detection threshold / 15s SBR stable-after —
/// the CLAUDE.md "total failover ~25s" design envelope.
///
/// Measures the SURVIVABLE direction only: hard-crash of the YOUNGER node,
/// timed to the survivor's member REMOVAL (detection + stable-after + gossip)
/// with singleton continuity asserted on the oldest. The oldest/active-node
/// crash is NOT a recovery to time — two-node keep-oldest makes the younger
/// survivor down itself (total outage; registered deferred user decision,
/// see SbrFailoverTests XML doc + master tracker 2026-07-08). Covers overall
/// review P2-10 / report-08 NF2 and report-01 round-2 N1's measurement ask.
/// </summary>
public class FailoverTimingTests(ITestOutputHelper output)
{
    private sealed class EchoActor : ReceiveActor
    {
        public EchoActor() => ReceiveAny(msg => Sender.Tell(msg));
    }

    [Trait("Category", "Performance")]
    [Fact]
    public async Task HardKillOfYoungerNode_SbrRemovalWithinDesignEnvelope()
    {
        await using var cluster = await TwoNodeClusterFixture.StartAsync(
            stableAfter: TimeSpan.FromSeconds(15),
            heartbeatInterval: TimeSpan.FromSeconds(2),
            failureDetectionThreshold: TimeSpan.FromSeconds(10));

        // Singleton on the oldest (NodeA) — the continuity probe.
        var manager = cluster.NodeA.ActorOf(ClusterSingletonManager.Props(
            Props.Create(() => new EchoActor()),
            PoisonPill.Instance,
            ClusterSingletonManagerSettings.Create(cluster.NodeA).WithSingletonName("timing-probe")),
            "timing-probe-singleton");
        var proxyA = cluster.NodeA.ActorOf(ClusterSingletonProxy.Props(
            "/user/timing-probe-singleton",
            ClusterSingletonProxySettings.Create(cluster.NodeA).WithSingletonName("timing-probe")),
            "timing-probe-proxy");
        Assert.Equal("ping", await proxyA.Ask<string>("ping", TimeSpan.FromSeconds(30)));

        var victim = Akka.Cluster.Cluster.Get(cluster.NodeB).SelfAddress;
        var sw = Stopwatch.StartNew();
        await TwoNodeClusterFixture.CrashNode(cluster.NodeB);
        await TwoNodeClusterFixture.WaitForMemberRemoved(cluster.NodeA, victim, TimeSpan.FromSeconds(60));
        sw.Stop();

        output.WriteLine(
            $"MEASURED: crash -> member removed on survivor: {sw.Elapsed.TotalSeconds:F1}s " +
            "(design ~25s = 10s detection + 15s stable-after; +gossip margin)");

        // Design envelope: >= stable-after (SBR must not act early), <= 25s design + 15s margin.
        Assert.InRange(sw.Elapsed, TimeSpan.FromSeconds(15), TimeSpan.FromSeconds(40));

        // Singleton continuity on the surviving oldest node.
        Assert.Equal("ping-after-crash",
            await proxyA.Ask<string>("ping-after-crash", TimeSpan.FromSeconds(10)));
    }
}

2. Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.PerformanceTests --filter "FullyQualifiedName~FailoverTimingTests" — expect FAIL (compile error: TwoNodeClusterFixture unknown; no heartbeatInterval parameter).
3. Implement:
   • TwoNodeClusterFixture.StartAsync (line 24): signature → StartAsync(string role = "Central", TimeSpan? stableAfter = null, TimeSpan? heartbeatInterval = null, TimeSpan? failureDetectionThreshold = null); pass both new args through the two f.StartNode(...) calls.
   • TwoNodeClusterFixture.StartNode (line 39): signature → StartNode(int port, string role, TimeSpan? stableAfter = null, TimeSpan? heartbeatInterval = null, TimeSpan? failureDetectionThreshold = null); lines 51-52 become HeartbeatInterval = heartbeatInterval ?? TimeSpan.FromMilliseconds(500), / FailureDetectionThreshold = failureDetectionThreshold ?? TimeSpan.FromSeconds(2),. Extend the class XML doc: "Timing knobs default to scaled test values; pass production values (2s/10s/15s) to measure the real failover envelope (FailoverTimingTests)."
   • ZB.MOM.WW.ScadaBridge.PerformanceTests.csproj: add to the existing <ItemGroup> of ProjectReferences:

    <!-- Two-node failover rig (TwoNodeClusterFixture) — brings Host + Akka transitively. -->
    <ProjectReference Include="../../tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/ZB.MOM.WW.ScadaBridge.IntegrationTests.csproj" />

4. Run again: dotnet test tests/ZB.MOM.WW.ScadaBridge.PerformanceTests --filter "FullyQualifiedName~FailoverTimingTests" — expect PASS in ~45-90s; the test output MUST print the MEASURED: line — quote the number in the commit message. Regression: dotnet test tests/ZB.MOM.WW.ScadaBridge.IntegrationTests --filter "FullyQualifiedName~SbrFailoverTests|FullyQualifiedName~ActiveNodeSemanticsTests|FullyQualifiedName~TwoNodeClusterFixtureTests" — all green (fixture change is additive).
5. Commit:

git add tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/TwoNodeClusterFixture.cs tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/ZB.MOM.WW.ScadaBridge.PerformanceTests.csproj tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Failover/FailoverTimingTests.cs
git commit -m "test(failover): unskip FailoverTimingTests on the two-node rig at production timings — measured Xs vs ~25s design envelope (plan R2-01 T4; covers report-08 NF2)"

Task 5: Apply the wonder-app-vd03 appsettings overlay edits (NodeName + AllowSingleNodeCluster, drop phantom seeds)

Classification: trivial (config-only; NO git commit — see note) Estimated implement time: ~4 min Parallelizable with: Task 1, Task 2, Task 4, Task 6, Task 8, Task 9, Task 10, Task 11 Files:

• Modify: deploy/wonder-app-vd03/appsettings.Central.json (Node lines 3-7, Cluster lines 8-18, _comment_Cluster line 19)
• Modify: deploy/wonder-app-vd03/appsettings.Site.json (Node lines 3-9, Cluster lines 10-20, _comment_Cluster line 21)

Round-2 N2: the round-1 deferral claimed the artifact "is not tracked in this repo … the artifact directory itself is not present" — the directory IS present (merely gitignored, .gitignore:48: /deploy/) and has been actively maintained in this working tree (its _comment_Security documents the PLAN-07 edit). Apply the two appsettings edits from PLAN-01 Tasks 16/23 directly. Without NodeName, every audit row from the only production install stamps a NULL SourceNode (defeats IX_AuditLog_Node_Occurred + the per-node stuck KPIs); with the phantom seeds, each node dials a nonexistent peer (localhost:8091 / localhost:8092) forever.

1. appsettings.Central.json:
   • Node (after "Role": "Central",): add "NodeName": "central-a", (matches the docker convention, docker/central-node-a/appsettings.Central.json).
   • Cluster: drop the phantom seed "akka.tcp://scadabridge@localhost:8091" (keep only the :8081 self seed) and add "AllowSingleNodeCluster": true (flag shipped in PLAN-01 T16: ClusterOptions.cs:86, validator ClusterOptionsValidator.cs:29-35 requires it for a 1-seed list).
   • _comment_Cluster → "Single-node central install, explicitly acknowledged via AllowSingleNodeCluster (arch-review 01 S12/N2): the previous phantom second seed (localhost:8091) made the node dial a nonexistent peer forever. When a real central node B is added: list BOTH real seeds and remove AllowSingleNodeCluster."
2. appsettings.Site.json: same treatment — Node gains "NodeName": "node-a",; Cluster.SeedNodes keeps only "akka.tcp://scadabridge@localhost:8082"; add "AllowSingleNodeCluster": true; _comment_Cluster rewritten equivalently (PRESERVE the existing "Seed ports must never equal GrpcPort (8083)" sentence).
3. Verify (this is the "test" — config task):

python3 -m json.tool deploy/wonder-app-vd03/appsettings.Central.json > /dev/null && \
python3 -m json.tool deploy/wonder-app-vd03/appsettings.Site.json > /dev/null && \
grep -c "NodeName\|AllowSingleNodeCluster" deploy/wonder-app-vd03/appsettings.Central.json deploy/wonder-app-vd03/appsettings.Site.json && \
grep -c "8091\|8092" deploy/wonder-app-vd03/appsettings.Central.json deploy/wonder-app-vd03/appsettings.Site.json

Expect: both json.tool exits 0; each file counts 2 for the first grep; each file counts 0 for the phantom-port grep (grep -c exits 1 on zero matches — that is the pass signal here). 4. No git commit — /deploy/ is gitignored (.gitignore:48); do NOT use git add -f. The edit is intentionally working-tree/on-host only; Task 7 commits the version-controlled record, and the change takes effect on the host when the artifact is next synced + services restarted (RUNBOOK step added in Task 6).

Task 6: Complete the install.ps1 recovery actions (sc.exe failureflag) + RUNBOOK recovery step

Classification: trivial (script/doc-only; NO git commit — see Task 5 note) Estimated implement time: ~4 min Parallelizable with: Task 1, Task 2, Task 4, Task 5, Task 8, Task 9, Task 10, Task 11 Files:

• Modify: deploy/wonder-app-vd03/install.ps1 (section # --- 7. Auto-restart on failure, lines ~171-174)
• Modify: deploy/wonder-app-vd03/RUNBOOK.md (troubleshooting table, ~line 218 area)

Finding-vs-disk note (record this honestly): the round-2 report says the sc.exe failure recovery actions are missing, but the on-disk install.ps1 already has them (lines 171-174, all three services) — what is missing is sc.exe failureflag <svc> 1. Without the failure-actions flag, the SCM runs recovery actions only after a crash; the down-if-alone watchdog's deliberate process exit (IHostApplicationLifetime.StopApplication() after an SBR self-down — AkkaHostedService.cs:203-217) is a non-crash stop and would never be restarted, breaking the recovery contract (Component-ClusterInfrastructure.md step 3: "The service supervisor restarts it"). This is the exact second line PLAN-01 Task 20 specified.

1. In install.ps1, extend the existing loop:

# --- 7. Auto-restart on failure ------------------------------------------
foreach ($svc in 'ScadaBridge-LDAP', 'ScadaBridge-Central', 'ScadaBridge-Site') {
    sc.exe failure $svc reset= 86400 actions= restart/5000/restart/5000/restart/10000 | Out-Null
    # Required for the SBR down-if-alone recovery contract
    # (Component-ClusterInfrastructure.md "Down-if-alone recovery"): without
    # failureflag=1 the SCM applies the recovery actions only after a CRASH;
    # the Host watchdog's deliberate process exit after a self-down is a
    # normal (non-crash) stop and would otherwise never be restarted.
    sc.exe failureflag $svc 1 | Out-Null
}

2. RUNBOOK.md: add a troubleshooting-table row: | Service stopped on its own with log line "ActorSystem terminated outside host shutdown" | SBR self-down (down-if-alone). The service recovery actions restart it automatically (requires install.ps1 with sc.exe failureflag — re-run install.ps1 or run the two sc.exe lines from §7 manually on older installs). If it does not come back: Start-Service manually and check network reachability between nodes. | Also note next to the "auto-restart on failure" mention (line 87) that failureflag is part of the contract.
3. Verify: grep -n "failureflag" deploy/wonder-app-vd03/install.ps1 (3 effective invocations via the loop → 1 hit inside it) and grep -n "failureflag\|self-down" deploy/wonder-app-vd03/RUNBOOK.md (≥1 hit). If pwsh is installed, syntax-check: pwsh -NoProfile -Command "[void][System.Management.Automation.PSParser]::Tokenize((Get-Content -Raw deploy/wonder-app-vd03/install.ps1), [ref]$null)" (exit 0); otherwise the greps suffice.
4. No git commit (gitignored artifact — same rule as Task 5).

Task 7: Correct the factually wrong N2 deferral record in the master tracker

Classification: trivial (doc-only) Estimated implement time: ~3 min Parallelizable with: Task 1, Task 2, Task 4, Task 8, Task 9, Task 10, Task 11 Files:

• Modify: archreview/plans/00-MASTER-TRACKER.md (line 28 trailing sentence; "Deferred during Wave 1 PLAN-01 execution (2026-07-08)" bullet at lines 145-147)

The deferral rationale — "that production deploy artifact is not tracked in this repo … the artifact directory itself is not present in the repo" — is factually wrong (round-2 N2): deploy/wonder-app-vd03/ exists in the working tree, is merely gitignored (.gitignore:48: /deploy/), and was edited during this initiative (PLAN-07's _comment_Security). Depends on Tasks 5-6 having applied the edits.

1. Rewrite the line-147 bullet: keep the history (what was deferred and why it mattered), then correct the record: the directory is present but gitignored (production config kept out of version control — the tracking claim was wrong, the presence claim was false); the three overlay edits were applied on-disk 2026-07-12 by PLAN-R2-01 Tasks 5-6 (NodeName central-a/node-a, AllowSingleNodeCluster + phantom seeds dropped, sc.exe failureflag completing the already-present sc.exe failure actions). Remaining owner action: sync the artifact to wonder-app-vd03 and restart services (appsettings take effect on restart; failureflag on re-running install.ps1 §7 or the two sc.exe lines manually) — per the RUNBOOK row added by T6.
2. Trim the stale trailing sentence of the line-28 Wave-1 summary ("were deferred because that production artifact is not tracked in this repo") → "were deferred on a mistaken not-in-repo rationale; corrected and applied on-disk by PLAN-R2-01 (2026-07-12) — see the Deferred-during-Wave-1 entry."
3. Do NOT add rows to docs/plans/2026-07-08-deferred-work-register.md — register consolidation (round-2 N6) is owned by PLAN-R2-08; this task only stops the tracker from asserting a falsehood.
4. Verify: grep -n "not tracked in this repo" archreview/plans/00-MASTER-TRACKER.md → 0 hits; grep -n "PLAN-R2-01" archreview/plans/00-MASTER-TRACKER.md → ≥2 hits.
5. Commit:

git add archreview/plans/00-MASTER-TRACKER.md
git commit -m "docs(tracker): correct the wonder-app-vd03 deferral record — artifact is present (gitignored); overlay edits applied on-disk by R2-01 T5/T6 (plan R2-01 T7)"

Task 8: Metrics-staleness for never-reported sites (FirstSeenAt anchor)

Classification: high-risk (lock-free CAS state machine) Estimated implement time: ~5 min Parallelizable with: Task 1, Task 2, Task 4, Task 5, Task 6, Task 7, Task 9, Task 10, Task 11 Files:

• Modify: src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthState.cs (add FirstSeenAt)
• Modify: src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthAggregator.cs (register branches lines 44-54 and 114-122; staleness guard lines 300-312)
• Modify: docs/requirements/Component-HealthMonitoring.md ("Metrics staleness" bullet, line ~47)
• Test: tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/CentralHealthAggregatorTests.cs

Round-2 N3: CheckForOfflineSites guards staleness with state.LastReportReceivedAt is { } lastReport (CentralHealthAggregator.cs:300), so a site registered via heartbeat only (MarkHeartbeat registration leaves LastReportReceivedAt null, :114-122) is NEVER flagged stale — "stuck from first boot" shows online-with-no-metrics forever. LastHeartbeatAt cannot be the anchor (heartbeats keep advancing it); anchor on a new FirstSeenAt stamped at registration. No UI change needed: the badge (Health.razor:209-215) keys on IsOnline && IsMetricsStale regardless of LatestReport.

1. Write the failing tests (append to CentralHealthAggregatorTests — reuse the existing NewAggregator(metricsStaleTimeout:) and TestTimeProvider.Advance helpers, lines 48-59/22):

/// <summary>
/// Round-2 N3: a site known only via heartbeats (LastReportReceivedAt == null)
/// previously skipped the staleness check forever — a report pipeline dead
/// from FIRST BOOT showed "online with no metrics" indefinitely. FirstSeenAt
/// anchors the window instead (LastHeartbeatAt cannot: heartbeats advance it).
/// </summary>
[Fact]
public void HeartbeatOnlySite_NeverReported_IsFlaggedMetricsStale()
{
    var (aggregator, time) = NewAggregator(metricsStaleTimeout: TimeSpan.FromMinutes(2));
    aggregator.MarkHeartbeat("site-a", time.GetUtcNow());   // registered via heartbeat only
    time.Advance(TimeSpan.FromMinutes(3));
    aggregator.MarkHeartbeat("site-a", time.GetUtcNow());   // still heartbeating => online
    aggregator.CheckForOfflineSites();
    var state = aggregator.GetSiteState("site-a")!;
    Assert.True(state.IsOnline);                             // liveness unchanged
    Assert.True(state.IsMetricsStale);                       // was impossible before the fix
}

/// <summary>A heartbeat-only site inside the window is NOT falsely flagged.</summary>
[Fact]
public void HeartbeatOnlySite_WithinStaleWindow_IsNotFlagged()
{
    var (aggregator, time) = NewAggregator(metricsStaleTimeout: TimeSpan.FromMinutes(2));
    aggregator.MarkHeartbeat("site-a", time.GetUtcNow());
    time.Advance(TimeSpan.FromMinutes(1));
    aggregator.MarkHeartbeat("site-a", time.GetUtcNow());
    aggregator.CheckForOfflineSites();
    Assert.False(aggregator.GetSiteState("site-a")!.IsMetricsStale);
}

2. Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests --filter "FullyQualifiedName~HeartbeatOnlySite" — expect the first test to FAIL (IsMetricsStale stays false; the null guard skips the site).
3. Implement:
   • SiteHealthState: add

    /// <summary>
    /// Gets the instant this site was first observed by the aggregator (via a
    /// report or a heartbeat); <c>null</c> only for states not built by the
    /// aggregator (hand-constructed fixtures). Anchors metrics-staleness for a
    /// site that has NEVER delivered a report (review 01 round-2 N3): with
    /// <see cref="LastReportReceivedAt"/> null the staleness check previously
    /// skipped the site forever. <see cref="LastHeartbeatAt"/> cannot anchor
    /// this — every heartbeat advances it.
    /// </summary>
    public DateTimeOffset? FirstSeenAt { get; init; }

• CentralHealthAggregator.ProcessReport register branch (lines 44-54): add FirstSeenAt = now,. MarkHeartbeat register branch (lines 114-122): add FirstSeenAt = receivedAt,. (Update branches never touch it — with copies preserve it.)
• CheckForOfflineSites staleness guard (lines 300-312): replace the state.LastReportReceivedAt is { } lastReport condition with an anchor fallback:

            // Second signal: … (keep the existing comment, append:)
            // A site that has NEVER reported (heartbeat-only registration,
            // LastReportReceivedAt == null) anchors on FirstSeenAt instead —
            // otherwise a pipeline dead from first boot is never flagged (N3).
            var reportAnchor = state.LastReportReceivedAt ?? state.FirstSeenAt;
            if (reportAnchor is { } anchor
                && now - anchor > _options.MetricsStaleTimeout
                && !state.IsMetricsStale)
            {
                var stale = state with { IsMetricsStale = true };
                if (_siteStates.TryUpdate(kvp.Key, stale, state))
                {
                    _logger.LogWarning(
                        "Site {SiteId} metrics are stale — online (heartbeats) but no report for {Elapsed}s{NeverNote} (timeout: {Timeout}s)",
                        state.SiteId, (now - anchor).TotalSeconds,
                        state.LastReportReceivedAt is null ? " (never reported since first contact)" : string.Empty,
                        _options.MetricsStaleTimeout.TotalSeconds);
                }
                // CAS loss ⇒ a fresh report/heartbeat swapped in ⇒ correct to skip.
            }

• Component-HealthMonitoring.md "Metrics staleness" bullet (line ~47): append — "A site that has never delivered a report is anchored on its first-contact time (FirstSeenAt), so a report pipeline dead from first boot is also flagged after MetricsStaleTimeout (previously such a site showed online-with-no-metrics indefinitely)."

4. Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests — full project green (the CAS/flap/stale suites are the regression net).
5. Commit:

git add src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthState.cs src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthAggregator.cs docs/requirements/Component-HealthMonitoring.md tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/CentralHealthAggregatorTests.cs
git commit -m "fix(health): flag metrics-stale for never-reported sites via FirstSeenAt anchor — null LastReportReceivedAt no longer skips the check (plan R2-01 T8)"

Task 9: Purge the stale "cluster leader" narration from CentralHealthReportLoop

Classification: trivial (comment-only) Estimated implement time: ~2 min Parallelizable with: Task 1, Task 2, Task 4, Task 5, Task 6, Task 7, Task 8, Task 10, Task 11 Files:

• Modify: src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthReportLoop.cs (class doc line 12-14; sequence-seed comment lines 44-45)

Round-2 N4: the behavior is correct (line 93 gates on IClusterNodeProvider.SelfIsPrimary = oldest Up member since PLAN-01 T6), but the comments still narrate "cluster leader (Primary)" — re-teaching the exact leader/oldest conflation round-1 S2 eradicated.

1. Confirm scope first: grep -in "leader" src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthReportLoop.cs — expect exactly lines 12, 44, 45.
2. Edit:
   • Class doc (lines 12-14): "Only the cluster leader (Primary) generates reports — the standby's aggregator catches up on failover when it becomes Primary and starts its own loop." → "Only the active node (Primary = the oldest Up member, i.e. the singleton host — IClusterNodeProvider.SelfIsPrimary, never Akka cluster leadership) generates reports; the standby's aggregator catches up on failover when it becomes the oldest and its loop starts reporting."
   • Sequence-seed comment (lines 44-45): "reports from a newly-elected central leader always sort after reports from any prior leader" → "reports from a node that has newly become active (oldest Up member) always sort after reports from any previously-active node".
3. Verify: grep -in "leader" src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthReportLoop.cs — 0 hits. Comment-only change: dotnet build src/ZB.MOM.WW.ScadaBridge.HealthMonitoring — succeeds.
4. Commit:

git add src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthReportLoop.cs
git commit -m "docs(health): CentralHealthReportLoop comments say oldest-member, not cluster leader — stop re-teaching the S2 conflation (plan R2-01 T9)"

Task 10: Generalize the registrar — SingletonRegistrar with an optional role scope

Classification: high-risk (actor lifecycle helper used by all central singletons) Estimated implement time: ~5 min Parallelizable with: Task 1, Task 2, Task 4, Task 5, Task 6, Task 7, Task 8, Task 9 Files:

• Modify (rename): src/ZB.MOM.WW.ScadaBridge.Host/Actors/CentralSingletonRegistrar.cs → src/ZB.MOM.WW.ScadaBridge.Host/Actors/SingletonRegistrar.cs (use git mv)
• Modify: src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs (rename at the block comment lines 459-465 and the 7 central call sites: lines 466, 499, 564, 609, 632, 663, 689)
• Modify (rename): tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CentralSingletonRegistrarTests.cs → tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SingletonRegistrarTests.cs
• Modify: tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CoordinatedShutdownTests.cs (line 56 source-grep string)

Round-2 N5 first half: CentralSingletonRegistrar cannot express .WithRole(siteRole), which is the sole reason the two site singletons stay hand-rolled without drains (AkkaHostedService.cs:464-465). Add an optional role applied to BOTH the manager and proxy settings, and rename the class — "Central" becomes a lie the moment it hosts site singletons.

1. Write the failing test — in the (renamed) SingletonRegistrarTests.cs, rename the class + existing references CentralSingletonRegistrar → SingletonRegistrar, and add:

    [Fact]
    public async Task Start_WithRoleScope_CreatesRoleScopedSingleton_ThatAnswers()
    {
        // Round-2 N5: role scoping is what kept the two SITE singletons
        // (deployment-manager, event-log-handler) hand-rolled and drain-less.
        using var system = CreateSingleNodeClusterSystem(); // roles = ["Central"]
        var handle = SingletonRegistrar.Start(
            system, "role-widget", Props.Create(() => new EchoActor()),
            NullLogger.Instance, drainTimeout: TimeSpan.FromSeconds(2), role: "Central");

        Assert.EndsWith("/user/role-widget-singleton", handle.Manager.Path.ToString());
        Assert.EndsWith("/user/role-widget-proxy", handle.Proxy.Path.ToString());

        // The node holds the role => manager instantiates the singleton and
        // the role-scoped proxy resolves it.
        var echo = await handle.Proxy.Ask<string>("hi", TimeSpan.FromSeconds(20));
        Assert.Equal("hi", echo);
    }

2. Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~SingletonRegistrarTests" — expect FAIL (compile: no SingletonRegistrar type / no role parameter).
3. Implement:
   • git mv src/ZB.MOM.WW.ScadaBridge.Host/Actors/CentralSingletonRegistrar.cs src/ZB.MOM.WW.ScadaBridge.Host/Actors/SingletonRegistrar.cs; rename the class; signature → Start(ActorSystem system, string name, Props singletonProps, ILogger logger, TimeSpan? drainTimeout = null, string? role = null); apply the role to both settings:

        var managerSettings = ClusterSingletonManagerSettings.Create(system).WithSingletonName(name);
        if (role is not null) managerSettings = managerSettings.WithRole(role);
        // … ClusterSingletonManager.Props(singletonProps, PoisonPill.Instance, managerSettings) …

        var proxySettings = ClusterSingletonProxySettings.Create(system).WithSingletonName(name);
        if (role is not null) proxySettings = proxySettings.WithRole(role);
        // … ClusterSingletonProxy.Props($"/user/{name}-singleton", proxySettings) …

Update the class XML doc: drop "central"-only framing; drain rationale now reads "so in-flight EF (central) or SQLite (site) work completes before handover"; note the optional role maps to ClusterSingletonManagerSettings/ProxySettings.WithRole (review 01 round-2 N5).

• AkkaHostedService.cs: mechanical rename at the 7 call sites (466, 499, 564, 609, 632, 663, 689) and rewrite the lines-459-465 comment's last sentence ("The two SITE singletons stay hand-rolled below because they are role-scoped") → "Site singletons use the same registrar with role: siteRole (Task 11 / round-2 N5)." (Task 11 lands that; wording here is forward-referencing but true within the same plan.)
• git mv tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CentralSingletonRegistrarTests.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SingletonRegistrarTests.cs.
• CoordinatedShutdownTests.cs line 56: Assert.Contains("CentralSingletonRegistrar.Start(", content); → Assert.Contains("SingletonRegistrar.Start(", content); (also update the comment above it).
• Sanity: grep -rn "CentralSingletonRegistrar" src/ tests/ --include="*.cs" — 0 hits.

4. Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~SingletonRegistrarTests|FullyQualifiedName~CoordinatedShutdownTests" — PASS. Then dotnet build ZB.MOM.WW.ScadaBridge.slnx — clean.
5. Commit:

git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/SingletonRegistrar.cs src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SingletonRegistrarTests.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CoordinatedShutdownTests.cs
git rm --cached src/ZB.MOM.WW.ScadaBridge.Host/Actors/CentralSingletonRegistrar.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CentralSingletonRegistrarTests.cs 2>/dev/null || true
git commit -m "refactor(host): CentralSingletonRegistrar -> SingletonRegistrar with optional role scope — unblocks site-singleton drains (plan R2-01 T10)"

(git mv already stages the renames; the git rm --cached … || true line is a no-op safety net if the moves were done as delete+create.)

Task 11: Route the two site singletons through the registrar — deployment-manager + event-log-handler gain drains

Classification: high-risk (site actor wiring refactor) Estimated implement time: ~5 min Parallelizable with: Task 1, Task 2, Task 4, Task 5, Task 6, Task 7, Task 8, Task 9 Files:

• Modify: src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs (site branch: deployment-manager block lines ~791-820; event-log-handler block lines ~836-856)
• Test: tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CoordinatedShutdownTests.cs

Round-2 N5 second half: deployment-manager and event-log-handler are hand-rolled with bare PoisonPill termination and no PhaseClusterLeave drain. The round-1 S5 rationale applies identically on the site side: the DeploymentManager's SQLite writes (static overrides, native_alarm_state) should drain before handover — currently safe-but-lossy on graceful failover.

1. Write the failing test — in CoordinatedShutdownTests.cs, update the existing AllCentralSingletons_RegisterThroughRegistrarWithDrain (lines 45-70): change the final assertion Assert.Equal(2, CountOccurrences(content, "ClusterSingletonManager.Props(")); → Assert.Equal(0, …) and replace its "the only two left are the SITE singletons" comment; then add:

    [Fact]
    public void SiteSingletons_RegisterThroughRegistrarWithDrain()
    {
        var hostProjectDir = FindHostProjectDirectory();
        Assert.NotNull(hostProjectDir);
        var content = File.ReadAllText(Path.Combine(hostProjectDir!, "Actors", "AkkaHostedService.cs"));

        // Round-2 N5: the two role-scoped SITE singletons previously stayed
        // hand-rolled (bare PoisonPill, no PhaseClusterLeave drain). They now
        // go through SingletonRegistrar.Start(..., role: siteRole), which
        // always adds the GracefulStop drain — in-flight SQLite writes
        // (static overrides, native_alarm_state) complete before handover.
        Assert.Contains("\"deployment-manager\"", content);
        Assert.Contains("\"event-log-handler\"", content);
        Assert.Contains("role: siteRole", content);
        Assert.Equal(0, CountOccurrences(content, "ClusterSingletonManager.Props("));
    }

2. Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~SiteSingletons_RegisterThroughRegistrarWithDrain" — expect FAIL (ClusterSingletonManager.Props( count is 2; no role: siteRole).
3. Implement in the site branch of AkkaHostedService.cs:
   • Deployment-manager block (~791-820): replace the hand-rolled ClusterSingletonManager.Props/ClusterSingletonProxy.Props pair with

        // Deployment Manager — role-scoped singleton via SingletonRegistrar
        // (review 01 round-2 N5): previously hand-rolled with bare PoisonPill
        // termination and NO PhaseClusterLeave drain, so in-flight SQLite
        // writes (static overrides, native_alarm_state) could be cut off on
        // graceful failover. Names unchanged: deployment-manager-singleton /
        // deployment-manager-proxy.
        var dm = SingletonRegistrar.Start(
            _actorSystem!, "deployment-manager",
            Props.Create(() => new DeploymentManagerActor(
                storage, compilationService, sharedScriptLibrary, streamManager,
                siteRuntimeOptionsValue, dmLogger, dclManager, replicationActor,
                siteHealthCollector, _serviceProvider, null, deploymentConfigFetcher)),
            _logger, role: siteRole);
        var dmProxy = dm.Proxy;

(Keep the dmProxy variable name — everything downstream, SiteCommunicationActor construction and RegisterLocalHandler(LocalHandlerType.Artifacts, dmProxy), is untouched.)

• Event-log-handler block (~836-856): inside the existing if (eventLogQueryService != null):

            var eventLog = SingletonRegistrar.Start(
                _actorSystem, "event-log-handler",
                Props.Create(() => new SiteEventLogging.EventLogHandlerActor(eventLogQueryService)),
                _logger, role: siteRole);
            siteCommActor.Tell(new RegisterLocalHandler(LocalHandlerType.EventLog, eventLog.Proxy));

(Keep the existing "Event log handler — cluster singleton so queries always reach the active node…" comment above it.)

• Actor/singleton names are reproduced exactly by the registrar (deployment-manager-singleton/-proxy, event-log-handler-singleton/-proxy), so path-pinning tests and the ClusterClient receptionist wiring see no change.

4. Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests (full project — path/health-check/audit-wiring suites are the regression net) and dotnet build ZB.MOM.WW.ScadaBridge.slnx — PASS/clean.
5. Commit:

git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CoordinatedShutdownTests.cs
git commit -m "fix(host): site singletons (deployment-manager, event-log-handler) via SingletonRegistrar — PhaseClusterLeave drains for site SQLite writes (plan R2-01 T11)"

---

Dependencies on other plans

• PLAN-R2-02 owns the oldest-vs-leader predicate unification inside SiteCommunicationActor/SiteReplicationActor and everything resync-related. This plan does not touch those files; the coverage table references it.
• PLAN-R2-08 owns N6 (folding the two live PLAN-01 deferrals — the keep-oldest active-crash gap and, post-T5/T6, the "sync the artifact to the host" residue — into docs/plans/2026-07-08-deferred-work-register.md). Task 7 here only corrects the factually wrong tracker prose; it adds no register rows.
• The SBR keep-oldest active-crash availability gap stays a registered deferred USER decision — no task here changes the strategy/topology; Tasks 1-3 make the gap observable and measured, which is the input that decision needs.

Execution order

Drill spine (serial): Task 1 (script) → Task 2 (docs) → Task 3 (live run, needs the docker cluster exclusively).

Registrar spine (serial): Task 10 → Task 11.

Freely parallel at any point: Tasks 4, 5, 6, 8, 9 (all file-disjoint); Task 7 after 5+6.

Highest value if time-boxed: Task 3 (the never-run drill, run at last) + Task 11 (real durability gap) + Task 8 (real monitoring blind spot).

---

Findings Coverage

Every round-2 finding (N1-N6), the round-1 partials the report says warrant completion, and the cross-plan rulings:

#	Round-2 finding	Severity	Task(s) / Disposition
N1	Failover drill kills the wrong (active/oldest) node; first-seed bootstrap gap means the survivor's recovery story doesn't converge; drill never executed; ~25s promise undeliverable in its most likely case	High	Tasks 1 (drill rewrite: standby-victim default + explicit active-gap mode), 2 (narrative + first-seed constraint documented with operator actions — deliberately NOT reordered to self-first seeds: simultaneous cold-start self-joins would risk two independent clusters), 3 (RUN the drill live, both directions, record measured timings), 4 (wire/unskip FailoverTimingTests to TwoNodeClusterFixture, measure the ~25s envelope at production timings — also covers report-08 NF2 by reference). The keep-oldest topology/strategy remedy itself stays deferred: registered USER decision (master tracker 2026-07-08) — Tasks 1-3 document/measure it, never fix it
N2	wonder-app-vd03 overlay edits unapplied; deferral rationale factually wrong (artifact IS in the working tree, merely gitignored)	Medium	Tasks 5 (NodeName + AllowSingleNodeCluster + drop phantom seeds, both overlays), 6 (sc.exe failureflag completing the recovery actions + RUNBOOK step — note: sc.exe failure itself is already on disk, contra the report), 7 (correct the deferral record in the master tracker). No git commits for the gitignored artifact itself
N3	IsMetricsStale can never fire for a site that has never delivered a report (null LastReportReceivedAt guard)	Low	Task 8 (FirstSeenAt anchor; no UI change needed — badge already keys on the flag)
N4	CentralHealthReportLoop still narrates "cluster leader (Primary)"	Low	Task 9
N5	Site-role singletons lack drain tasks; registrar can't express role scoping	Low	Tasks 10 (role param + rename to SingletonRegistrar), 11 (route deployment-manager + event-log-handler through it)
N6	PLAN-01's two live deferrals missing from the canonical deferred-work register	Low	Reference → PLAN-R2-08 (register consolidation is its scope). Task 7 here only fixes the falsehood in the master-tracker prose
S12 (R1 partial)	Single-node artifact still ships the phantom seed, lacks AllowSingleNodeCluster	—	Task 5 (the artifact half; the flag/validator half shipped in PLAN-01 T16)
U2 (R1 partial)	install.ps1 recovery actions deferred; recovery contract doesn't converge for a non-first-seed survivor	—	Task 6 (failureflag) + Task 2 (first-seed constraint + operator actions in the spec'd recovery contract)
U5 (R1 not fixed)	NodeName missing from wonder-app-vd03 overlays → NULL SourceNode	—	Task 5
C3 (R1 partial)	LDAP Transport: None, AllowInsecure: true remains in the deploy artifact	—	Stays accepted posture per the report — bundled loopback GLAuth with documented AD-repoint instructions; a future over-the-wire AD repoint must revisit transport. No task
—	Oldest-vs-leader predicate unification in SiteCommunicationActor/SiteReplicationActor + resync work	—	Reference → PLAN-R2-02 (explicit cross-plan ownership; not touched here)