# PLAN-R2-01 — Cluster, Host & Failover Round-2 Fixes Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.

**Goal:** Make the failover story *honest* — rewrite and actually RUN the failover drill so it proves what two-node keep-oldest can deliver (and measures what it cannot), apply the three wonder-app-vd03 overlay edits that were deferred on a factually wrong rationale, and close the round-2 residuals (never-reported metrics staleness, stale leader comments, undrained site singletons).

**Architecture:** The drill (`docker/failover-drill.sh`) gains a `DRILL_MODE` — `standby` (default, the survivable younger-node crash that PASSes within the ~25s SBR budget) and `active` (the oldest-node crash that keep-oldest cannot survive: the drill *expects* the outage and measures it, documenting the registered deferred user decision instead of codifying a recovery the topology cannot deliver). The in-process envelope measurement reuses `TwoNodeClusterFixture` (production `BuildHocon`, now with production-timing knobs) from the previously-skipped `FailoverTimingTests`. Site-role singleton drains come from generalizing `CentralSingletonRegistrar` into a role-aware `SingletonRegistrar` so `deployment-manager` and `event-log-handler` get the same `PhaseClusterLeave` GracefulStop the seven central singletons got in round 1.

**Tech Stack:** C#/.NET, Akka.NET 1.5.62 (Akka.Cluster, Akka.Cluster.Tools), xUnit, bash + Docker Compose + Traefik, JSON deploy overlays (PowerShell `install.ps1`).

Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per-project: `dotnet test tests/<project>`. Docker rig: `bash docker/deploy.sh`.

**Binding scope rulings (do not deviate):**
- The two-node keep-oldest **active/oldest-crash availability gap is a registered deferred USER decision** (master tracker 2026-07-08 "Follow-up discovered during P0 execution"; `SbrFailoverTests.cs:15-23`). No task in this plan changes the SBR strategy or topology — Tasks 1–3 *document and measure* the gap.
- The oldest-vs-leader predicate unification inside `SiteCommunicationActor`/`SiteReplicationActor` and everything resync-related is **owned by PLAN-R2-02** — do not touch those files here.
- N6 (deferred-work-register consolidation) is **owned by PLAN-R2-08** — Task 7 here corrects only the factually wrong N2 prose in the master tracker.
- `/deploy/` is **gitignored** (`.gitignore:48: /deploy/`) — Tasks 5 and 6 edit the on-disk artifact and deliberately do NOT `git add` it (never use `git add -f`); Task 7's tracker correction is the version-controlled record.

---

### Task 1: Rewrite `failover-drill.sh` — standby-victim default + explicit active-victim gap mode

**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 4, Task 5, Task 6, Task 8, Task 9, Task 10
**Files:**
- Modify: `docker/failover-drill.sh` (full rewrite — current script is 45 lines; lines 12-19 pick the ACTIVE node as victim, which is exactly the crash keep-oldest cannot survive)

Context (round-2 N1): under the round-1 oldest-member unification, the *active* node IS the oldest. `docker/failover-drill.sh:17-19` kills it, so the shipped drill exercises the one crash two-node keep-oldest is *known* not to recover from (the survivor self-downs; the restarted `central-b` cannot re-bootstrap because both nodes list `scadabridge-central-a` as FIRST seed — `docker/central-node-a/appsettings.Central.json:10-13`, `docker/central-node-b/appsettings.Central.json:10-13` — and only the first seed may self-join to form a new cluster). The drill has also never been run (`docker/README.md:288`). Fix the drill so its PASS criteria match reality per direction.

1. No unit test (bash script). The verification command is the "test": `bash -n docker/failover-drill.sh` (must exit 0), plus `DRILL_MODE=bogus bash docker/failover-drill.sh` (must exit 2 with the usage error — this exercises the argument guard without needing a cluster).
2. Replace the script's entire contents with:

```bash
#!/usr/bin/env bash
# Failover drill against the running docker cluster (bash docker/deploy.sh first).
#
# ROUND-2 REWRITE (arch-review 01 round 2, N1). The original drill killed the
# ACTIVE central node — but under the unified oldest-member semantics the
# active node IS the oldest, i.e. the one crash two-node keep-oldest CANNOT
# survive (registered deferred user decision, master tracker 2026-07-08;
# SbrFailoverTests.cs XML doc). Two modes:
#
#   DRILL_MODE=standby (default) — kills the STANDBY (younger) central node.
#     The survivable direction: SBR downs the crashed member, the active node
#     keeps its singletons, and Traefik routing never goes dark. PASS = the
#     survivor logs the member removal within TIMEOUT_S (budget ~25s+: 10s
#     failure detection + 15s stable-after) while /health/active stays up.
#
#   DRILL_MODE=active — kills the ACTIVE (oldest) central node. THE EXPECTED
#     OUTCOME IS A TOTAL CENTRAL OUTAGE: keep-oldest downs the partition
#     without the oldest, so the younger survivor downs ITSELF (down-if-alone
#     cannot help — the alone-oldest is dead and cannot down itself), and the
#     self-downed survivor cannot re-form a cluster alone unless it is the
#     FIRST seed (both nodes list central-a first; only the first seed may
#     self-join). This mode measures the dark window and PASSes only when
#     central recovers AFTER the victim container is restarted. It exists to
#     make the registered gap observable — not to pretend it is covered.
set -euo pipefail

TRAEFIK_URL="${TRAEFIK_URL:-http://localhost:9000}"
TIMEOUT_S="${TIMEOUT_S:-90}"
DRILL_MODE="${DRILL_MODE:-standby}"
OUTAGE_CONFIRM_S="${OUTAGE_CONFIRM_S:-60}"

active_container() {
  if curl -sf -o /dev/null "http://localhost:9001/health/active"; then echo scadabridge-central-a
  elif curl -sf -o /dev/null "http://localhost:9002/health/active"; then echo scadabridge-central-b
  else echo "ERROR: no active central node found" >&2; exit 1; fi
}
peer_of() { [ "$1" = scadabridge-central-a ] && echo scadabridge-central-b || echo scadabridge-central-a; }

case "$DRILL_MODE" in
  standby|active) ;;
  *) echo "ERROR: DRILL_MODE must be 'standby' or 'active' (was '$DRILL_MODE')" >&2; exit 2 ;;
esac

ACTIVE=$(active_container)
if [ "$DRILL_MODE" = standby ]; then
  VICTIM=$(peer_of "$ACTIVE"); SURVIVOR="$ACTIVE"
else
  VICTIM="$ACTIVE"; SURVIVOR=$(peer_of "$ACTIVE")
fi

echo "mode=${DRILL_MODE}  active=${ACTIVE}  victim=${VICTIM}  survivor=${SURVIVOR}"
KILL_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)
docker kill "${VICTIM}" > /dev/null
START=$(date +%s)

if [ "$DRILL_MODE" = standby ]; then
  echo "Standby crash: waiting for ${SURVIVOR} to DOWN+REMOVE the dead member (SBR budget ~25s)..."
  BLIPS=0
  while true; do
    ELAPSED=$(( $(date +%s) - START ))
    curl -sf -o /dev/null "${TRAEFIK_URL}/health/active" || BLIPS=$((BLIPS + 1))
    if docker logs --since "${KILL_AT}" "${SURVIVOR}" 2>&1 | grep -Eiq "marking.*node.*down|member removed|is removed"; then
      echo "PASS: survivor removed the crashed member in ${ELAPSED}s (budget ~25s: 10s detection + 15s stable-after)."
      echo "Active-node routing blips during the drill: ${BLIPS} (expected 0 — the active node was never touched)."
      break
    fi
    if (( ELAPSED > TIMEOUT_S )); then
      echo "FAIL: no downing/removal evidence on ${SURVIVOR} after ${ELAPSED}s — SBR did not act" >&2
      docker start "${VICTIM}" > /dev/null
      exit 1
    fi
    sleep 1
  done
else
  echo "Active crash: EXPECTING a central outage (registered keep-oldest gap). Watching /health/active..."
  DARK_STREAK=0
  while true; do
    ELAPSED=$(( $(date +%s) - START ))
    if curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; then DARK_STREAK=0; else DARK_STREAK=$((DARK_STREAK + 1)); fi
    if (( DARK_STREAK >= 10 )); then
      echo "Outage confirmed at ${ELAPSED}s: no active central node — the younger survivor self-downed"
      echo "(keep-oldest downs the partition WITHOUT the oldest; this is the registered deferred gap)."
      break
    fi
    if (( ELAPSED > OUTAGE_CONFIRM_S )); then
      echo "NOTE: /health/active stayed reachable ${ELAPSED}s after killing the oldest — better than the"
      echo "registered gap predicts. Do NOT celebrate: capture both nodes' logs and investigate before trusting it."
      break
    fi
    sleep 1
  done
fi

echo "Restarting ${VICTIM}..."
docker start "${VICTIM}" > /dev/null
RESTART_AT=$(date +%s)
echo "Waiting for central to be routable again through Traefik (${TRAEFIK_URL}/health/active)..."
while true; do
  ELAPSED=$(( $(date +%s) - RESTART_AT ))
  if curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; then
    echo "Recovered: an active central node is routable ${ELAPSED}s after the victim restart."
    break
  fi
  if (( ELAPSED > 120 )); then
    echo "FAIL: central not routable 120s after restarting ${VICTIM}" >&2
    exit 1
  fi
  sleep 1
done

echo "Survivor singleton/downing evidence (last 20 matching log lines from ${SURVIVOR}):"
docker logs "${SURVIVOR}" 2>&1 | grep -Ei "singleton|oldest|downing|removed" | tail -20 || true
echo "Drill complete (${DRILL_MODE}). Verify on the Health dashboard that both nodes show Up and exactly one is Primary."
```

3. `chmod +x docker/failover-drill.sh`. Verify: `bash -n docker/failover-drill.sh` (exit 0) and `DRILL_MODE=bogus bash docker/failover-drill.sh; echo "exit=$?"` (prints the usage error, exit=2).
4. Commit:
```
git add docker/failover-drill.sh
git commit -m "fix(docker): failover drill kills the STANDBY by default; active mode measures the registered keep-oldest outage instead of pretending recovery (plan R2-01 T1)"
```

### Task 2: Correct the recovery narrative + document the first-seed bootstrap constraint

**Classification:** trivial (doc-only)
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 4, Task 5, Task 6, Task 8, Task 9, Task 10
**Files:**
- Modify: `docker/README.md` (Failover Testing → "Automated failover drill" section, lines 273-288)
- Modify: `docs/requirements/Component-ClusterInfrastructure.md` ("Down-if-alone recovery" subsection, lines ~104-116)

1. `docker/README.md` (lines 273-288): rewrite the drill section to match Task 1's two modes. Specifically:
   - Replace the four numbered bullets with per-mode descriptions: `DRILL_MODE=standby` (default) = the survivable younger-crash direction — expected result: **no routing outage at all** (the active node is never touched) and member removal on the survivor within **~25s** (10s failure-detection threshold + 15s stable-after; the 2s heartbeat interval is not additive); `DRILL_MODE=active` = the oldest-crash direction — expected result: **total central outage** until the victim container is restarted (the registered deferred keep-oldest decision, master tracker 2026-07-08), then recovery within ~2 min of the restart.
   - Delete the line-286 promise "Expected failover: **~25s** … plus the Traefik ~5s health-check interval" — it described a singleton *migration* on active-crash that two-node keep-oldest cannot deliver. State per-direction expectations instead.
   - Replace the line-288 "Observed failover time: _run after next deploy and record here_" placeholder with a two-row results table Task 3 will fill:

```markdown
> **Observed results** (recorded by plan R2-01 T3, cluster @ commit <sha>):
>
> | Direction (`DRILL_MODE`) | Outcome | Measured |
> |--------------------------|---------|----------|
> | `standby` (younger-node crash) | _pending T3_ | _pending T3_ |
> | `active` (oldest-node crash) | _pending T3_ | _pending T3_ |
```

   - Add a short **"Seed-node bootstrap constraint"** note to the same section: only the FIRST seed in `Cluster:SeedNodes` may self-join to form a *new* cluster; both central nodes list `scadabridge-central-a` first, so a lone restarted `central-b` (with `central-a` still down) loops on `InitJoin` forever — never `Up`, never `/health/active` 200. Operator recovery actions: **(1)** restart the dead first-seed node (preferred), or **(2)** restart the survivor with a self-first seed override (env: `ScadaBridge__Cluster__SeedNodes__0=akka.tcp://scadabridge@<self-host>:8081`, `...__1=<peer>`). Explain why the repo does NOT ship self-first ordering per node: with *both* nodes self-first, a simultaneous cold start can let each self-join independently → two one-node clusters that never merge (a cold-start split-brain the identical-seed-order convention exists to prevent). The real remedy is the pending keep-oldest topology/strategy decision (deferred, owner: user).
2. `docs/requirements/Component-ClusterInfrastructure.md` (Down-if-alone recovery, ~104-116):
   - Amend step 4 ("The restarted process rejoins as a **fresh incarnation** (the keep-oldest resolver handles the rejoin cleanly…)"): add the qualifier that this holds **only while a peer holding cluster state is still reachable** — a lone restarted non-first-seed node cannot re-form a cluster (first-seed self-join rule) and waits for its peer.
   - Add a "Seed-node bootstrap constraint" paragraph after the numbered list (same content as the README note, spec-toned), cross-referencing the registered deferred keep-oldest decision and the operator actions.
   - Fix the closing line "The docker failover drill exercises this loop end-to-end (kill the active node, assert singleton migration + clean rejoin)" → "The docker failover drill (`docker/failover-drill.sh`) exercises this per direction: `standby` mode proves SBR downing + singleton continuity on the oldest; `active` mode measures the registered total-outage gap and the recovery-on-restart path."
3. Verification (doc task — no test): `grep -n "DRILL_MODE" docker/README.md` (≥2 hits), `grep -cn "singleton migration" docs/requirements/Component-ClusterInfrastructure.md` (0 hits in the recovery section), `grep -n "first seed\|first-seed" docker/README.md docs/requirements/Component-ClusterInfrastructure.md` (≥2 hits), `grep -n "Traefik ~5s health-check interval" docker/README.md` (0 hits).
4. Commit:
```
git add docker/README.md docs/requirements/Component-ClusterInfrastructure.md
git commit -m "docs(cluster): per-direction failover truth — first-seed bootstrap constraint + operator recovery actions; drop the undeliverable ~25s active-crash promise (plan R2-01 T2)"
```

### Task 3: RUN the drill live (both directions) and record measured timings

**Classification:** standard (live-cluster verification; ~5 min of work, several minutes of wall-clock waiting)
**Estimated implement time:** ~5 min
**Parallelizable with:** none (exclusive use of the shared docker cluster; writes `docker/README.md`)
**Files:**
- Modify: `docker/README.md` (fill the Task 2 results table)

The drill has NEVER been executed (round-2 N1 point 3, `docker/README.md:288`). This task produces the empirical record — the asymmetric result *is* the documentation the deferred keep-oldest decision needs.

1. Ensure infra + cluster are up and freshly built (drill needs the Task 1 script inside the checkout only — no image change required, but the cluster must be healthy):
```
cd infra && docker compose up -d && cd ..
bash docker/deploy.sh
```
Wait until `curl -sf http://localhost:9000/health/active` returns 200 (poll; allow ~2 min for first-boot readiness).
2. Run direction 1 and capture:
```
DRILL_MODE=standby bash docker/failover-drill.sh 2>&1 | tee /private/tmp/claude-501/-Users-dohertj2-Desktop-ScadaBridge/d69633c2-b169-4fe1-8fb4-0d8007a16fa1/scratchpad/drill-standby.log
```
Expected: PASS with member-removal time ≈ 25-30s and `routing blips: 0`. Wait for the victim to fully rejoin (script does this) plus ~30s settle time.
3. Run direction 2 and capture:
```
DRILL_MODE=active bash docker/failover-drill.sh 2>&1 | tee /private/tmp/claude-501/-Users-dohertj2-Desktop-ScadaBridge/d69633c2-b169-4fe1-8fb4-0d8007a16fa1/scratchpad/drill-active.log
```
Expected: "Outage confirmed at ~Ns" (survivor self-downs; central dark), then "Recovered: … Ms after the victim restart". If instead the NOTE branch fires (`/health/active` stayed up), capture `docker logs scadabridge-central-a` + `-b` and investigate before recording — do not record a result that contradicts the registered gap without evidence.
4. Fill the `docker/README.md` results table from the two logs: outcome + measured seconds per direction, and the cluster commit SHA (`git rev-parse --short HEAD`). Keep the raw one-line summary quotes from the script output.
5. Sanity-restore: confirm both nodes healthy afterwards — `curl -sf http://localhost:9001/health/ready && curl -sf http://localhost:9002/health/ready` (both 200; the standby's `/health/active` correctly 503).
6. Commit:
```
git add docker/README.md
git commit -m "test(docker): record live failover-drill results for both crash directions — standby-crash PASS timing + active-crash measured outage (plan R2-01 T3)"
```

### Task 4: Wire `FailoverTimingTests` to `TwoNodeClusterFixture` — measure the ~25s envelope in-process

**Classification:** high-risk (real two-node cluster timing test)
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 1, Task 2, Task 5, Task 6, Task 7, Task 8, Task 9, Task 10, Task 11
**Files:**
- Modify: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/TwoNodeClusterFixture.cs` (StartAsync lines 24-36, StartNode lines 39-65 — add production-timing knobs)
- Modify: `tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/ZB.MOM.WW.ScadaBridge.PerformanceTests.csproj` (add a ProjectReference)
- Modify: `tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Failover/FailoverTimingTests.cs` (replace the `Skip` placeholder — currently `[Fact(Skip = "Requires the real two-node failover rig delivered by PLAN-01 …")]` at lines 25-32)

This task **also covers architecture-review report 08's NF2 by reference** (the skipped perf placeholder): the rig it waits for exists (`TwoNodeClusterFixture`, PLAN-01 T3); wire it up. The fixture hard-codes scaled timings (`HeartbeatInterval` 500ms / `FailureDetectionThreshold` 2s at `TwoNodeClusterFixture.cs:51-52`); to measure the *production* ~25s envelope (2s heartbeat / 10s threshold / 15s stable-after), the knobs become optional parameters with the current values as defaults (additive — `SbrFailoverTests`/`ActiveNodeSemanticsTests` unaffected). Direction note: the measurable crash is the YOUNGER node (the survivable direction); the oldest-crash direction is the registered deferred gap and is *not* a recovery to time — say so in the test's XML doc.

1. Rewrite `FailoverTimingTests.cs` (this is the failing step — it will not compile until the fixture knobs + project reference exist):

```csharp
using System.Diagnostics;
using Akka.Actor;
using Akka.Cluster.Tools.Singleton;
using Xunit.Abstractions;
using ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;

namespace ZB.MOM.WW.ScadaBridge.PerformanceTests.Failover;

/// <summary>
/// Failover-timing measurement on the real two-node in-process rig
/// (TwoNodeClusterFixture, production BuildHocon) at PRODUCTION timings:
/// 2s heartbeat / 10s failure-detection threshold / 15s SBR stable-after —
/// the CLAUDE.md "total failover ~25s" design envelope.
///
/// Measures the SURVIVABLE direction only: hard-crash of the YOUNGER node,
/// timed to the survivor's member REMOVAL (detection + stable-after + gossip)
/// with singleton continuity asserted on the oldest. The oldest/active-node
/// crash is NOT a recovery to time — two-node keep-oldest makes the younger
/// survivor down itself (total outage; registered deferred user decision,
/// see SbrFailoverTests XML doc + master tracker 2026-07-08). Covers overall
/// review P2-10 / report-08 NF2 and report-01 round-2 N1's measurement ask.
/// </summary>
public class FailoverTimingTests(ITestOutputHelper output)
{
    private sealed class EchoActor : ReceiveActor
    {
        public EchoActor() => ReceiveAny(msg => Sender.Tell(msg));
    }

    [Trait("Category", "Performance")]
    [Fact]
    public async Task HardKillOfYoungerNode_SbrRemovalWithinDesignEnvelope()
    {
        await using var cluster = await TwoNodeClusterFixture.StartAsync(
            stableAfter: TimeSpan.FromSeconds(15),
            heartbeatInterval: TimeSpan.FromSeconds(2),
            failureDetectionThreshold: TimeSpan.FromSeconds(10));

        // Singleton on the oldest (NodeA) — the continuity probe.
        var manager = cluster.NodeA.ActorOf(ClusterSingletonManager.Props(
            Props.Create(() => new EchoActor()),
            PoisonPill.Instance,
            ClusterSingletonManagerSettings.Create(cluster.NodeA).WithSingletonName("timing-probe")),
            "timing-probe-singleton");
        var proxyA = cluster.NodeA.ActorOf(ClusterSingletonProxy.Props(
            "/user/timing-probe-singleton",
            ClusterSingletonProxySettings.Create(cluster.NodeA).WithSingletonName("timing-probe")),
            "timing-probe-proxy");
        Assert.Equal("ping", await proxyA.Ask<string>("ping", TimeSpan.FromSeconds(30)));

        var victim = Akka.Cluster.Cluster.Get(cluster.NodeB).SelfAddress;
        var sw = Stopwatch.StartNew();
        await TwoNodeClusterFixture.CrashNode(cluster.NodeB);
        await TwoNodeClusterFixture.WaitForMemberRemoved(cluster.NodeA, victim, TimeSpan.FromSeconds(60));
        sw.Stop();

        output.WriteLine(
            $"MEASURED: crash -> member removed on survivor: {sw.Elapsed.TotalSeconds:F1}s " +
            "(design ~25s = 10s detection + 15s stable-after; +gossip margin)");

        // Design envelope: >= stable-after (SBR must not act early), <= 25s design + 15s margin.
        Assert.InRange(sw.Elapsed, TimeSpan.FromSeconds(15), TimeSpan.FromSeconds(40));

        // Singleton continuity on the surviving oldest node.
        Assert.Equal("ping-after-crash",
            await proxyA.Ask<string>("ping-after-crash", TimeSpan.FromSeconds(10)));
    }
}
```

2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.PerformanceTests --filter "FullyQualifiedName~FailoverTimingTests"` — expect FAIL (compile error: `TwoNodeClusterFixture` unknown; no `heartbeatInterval` parameter).
3. Implement:
   - `TwoNodeClusterFixture.StartAsync` (line 24): signature → `StartAsync(string role = "Central", TimeSpan? stableAfter = null, TimeSpan? heartbeatInterval = null, TimeSpan? failureDetectionThreshold = null)`; pass both new args through the two `f.StartNode(...)` calls.
   - `TwoNodeClusterFixture.StartNode` (line 39): signature → `StartNode(int port, string role, TimeSpan? stableAfter = null, TimeSpan? heartbeatInterval = null, TimeSpan? failureDetectionThreshold = null)`; lines 51-52 become `HeartbeatInterval = heartbeatInterval ?? TimeSpan.FromMilliseconds(500),` / `FailureDetectionThreshold = failureDetectionThreshold ?? TimeSpan.FromSeconds(2),`. Extend the class XML doc: "Timing knobs default to scaled test values; pass production values (2s/10s/15s) to measure the real failover envelope (FailoverTimingTests)."
   - `ZB.MOM.WW.ScadaBridge.PerformanceTests.csproj`: add to the existing `<ItemGroup>` of ProjectReferences:

```xml
    <!-- Two-node failover rig (TwoNodeClusterFixture) — brings Host + Akka transitively. -->
    <ProjectReference Include="../../tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/ZB.MOM.WW.ScadaBridge.IntegrationTests.csproj" />
```

4. Run again: `dotnet test tests/ZB.MOM.WW.ScadaBridge.PerformanceTests --filter "FullyQualifiedName~FailoverTimingTests"` — expect PASS in ~45-90s; the test output MUST print the `MEASURED:` line — quote the number in the commit message. Regression: `dotnet test tests/ZB.MOM.WW.ScadaBridge.IntegrationTests --filter "FullyQualifiedName~SbrFailoverTests|FullyQualifiedName~ActiveNodeSemanticsTests|FullyQualifiedName~TwoNodeClusterFixtureTests"` — all green (fixture change is additive).
5. Commit:
```
git add tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/TwoNodeClusterFixture.cs tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/ZB.MOM.WW.ScadaBridge.PerformanceTests.csproj tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Failover/FailoverTimingTests.cs
git commit -m "test(failover): unskip FailoverTimingTests on the two-node rig at production timings — measured Xs vs ~25s design envelope (plan R2-01 T4; covers report-08 NF2)"
```

### Task 5: Apply the wonder-app-vd03 appsettings overlay edits (NodeName + AllowSingleNodeCluster, drop phantom seeds)

**Classification:** trivial (config-only; NO git commit — see note)
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 1, Task 2, Task 4, Task 6, Task 8, Task 9, Task 10, Task 11
**Files:**
- Modify: `deploy/wonder-app-vd03/appsettings.Central.json` (`Node` lines 3-7, `Cluster` lines 8-18, `_comment_Cluster` line 19)
- Modify: `deploy/wonder-app-vd03/appsettings.Site.json` (`Node` lines 3-9, `Cluster` lines 10-20, `_comment_Cluster` line 21)

Round-2 N2: the round-1 deferral claimed the artifact "is not tracked in this repo … the artifact directory itself is not present" — **the directory IS present** (merely gitignored, `.gitignore:48: /deploy/`) and has been actively maintained in this working tree (its `_comment_Security` documents the PLAN-07 edit). Apply the two appsettings edits from PLAN-01 Tasks 16/23 directly. Without `NodeName`, every audit row from the only production install stamps a NULL `SourceNode` (defeats `IX_AuditLog_Node_Occurred` + the per-node stuck KPIs); with the phantom seeds, each node dials a nonexistent peer (`localhost:8091` / `localhost:8092`) forever.

1. `appsettings.Central.json`:
   - `Node` (after `"Role": "Central",`): add `"NodeName": "central-a",` (matches the docker convention, `docker/central-node-a/appsettings.Central.json`).
   - `Cluster`: drop the phantom seed `"akka.tcp://scadabridge@localhost:8091"` (keep only the `:8081` self seed) and add `"AllowSingleNodeCluster": true` (flag shipped in PLAN-01 T16: `ClusterOptions.cs:86`, validator `ClusterOptionsValidator.cs:29-35` requires it for a 1-seed list).
   - `_comment_Cluster` → `"Single-node central install, explicitly acknowledged via AllowSingleNodeCluster (arch-review 01 S12/N2): the previous phantom second seed (localhost:8091) made the node dial a nonexistent peer forever. When a real central node B is added: list BOTH real seeds and remove AllowSingleNodeCluster."`
2. `appsettings.Site.json`: same treatment — `Node` gains `"NodeName": "node-a",`; `Cluster.SeedNodes` keeps only `"akka.tcp://scadabridge@localhost:8082"`; add `"AllowSingleNodeCluster": true`; `_comment_Cluster` rewritten equivalently (PRESERVE the existing "Seed ports must never equal GrpcPort (8083)" sentence).
3. Verify (this is the "test" — config task):
```
python3 -m json.tool deploy/wonder-app-vd03/appsettings.Central.json > /dev/null && \
python3 -m json.tool deploy/wonder-app-vd03/appsettings.Site.json > /dev/null && \
grep -c "NodeName\|AllowSingleNodeCluster" deploy/wonder-app-vd03/appsettings.Central.json deploy/wonder-app-vd03/appsettings.Site.json && \
grep -c "8091\|8092" deploy/wonder-app-vd03/appsettings.Central.json deploy/wonder-app-vd03/appsettings.Site.json
```
Expect: both json.tool exits 0; each file counts 2 for the first grep; each file counts **0** for the phantom-port grep (grep -c exits 1 on zero matches — that is the pass signal here).
4. **No git commit** — `/deploy/` is gitignored (`.gitignore:48`); do NOT use `git add -f`. The edit is intentionally working-tree/on-host only; Task 7 commits the version-controlled record, and the change takes effect on the host when the artifact is next synced + services restarted (RUNBOOK step added in Task 6).

### Task 6: Complete the install.ps1 recovery actions (`sc.exe failureflag`) + RUNBOOK recovery step

**Classification:** trivial (script/doc-only; NO git commit — see Task 5 note)
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 1, Task 2, Task 4, Task 5, Task 8, Task 9, Task 10, Task 11
**Files:**
- Modify: `deploy/wonder-app-vd03/install.ps1` (section `# --- 7. Auto-restart on failure`, lines ~171-174)
- Modify: `deploy/wonder-app-vd03/RUNBOOK.md` (troubleshooting table, ~line 218 area)

Finding-vs-disk note (record this honestly): the round-2 report says the `sc.exe failure` recovery actions are missing, but the on-disk `install.ps1` already has them (lines 171-174, all three services) — what is missing is `sc.exe failureflag <svc> 1`. Without the failure-actions flag, the SCM runs recovery actions only after a **crash**; the down-if-alone watchdog's *deliberate* process exit (`IHostApplicationLifetime.StopApplication()` after an SBR self-down — `AkkaHostedService.cs:203-217`) is a non-crash stop and would never be restarted, breaking the recovery contract (`Component-ClusterInfrastructure.md` step 3: "The service supervisor restarts it"). This is the exact second line PLAN-01 Task 20 specified.

1. In `install.ps1`, extend the existing loop:

```powershell
# --- 7. Auto-restart on failure ------------------------------------------
foreach ($svc in 'ScadaBridge-LDAP', 'ScadaBridge-Central', 'ScadaBridge-Site') {
    sc.exe failure $svc reset= 86400 actions= restart/5000/restart/5000/restart/10000 | Out-Null
    # Required for the SBR down-if-alone recovery contract
    # (Component-ClusterInfrastructure.md "Down-if-alone recovery"): without
    # failureflag=1 the SCM applies the recovery actions only after a CRASH;
    # the Host watchdog's deliberate process exit after a self-down is a
    # normal (non-crash) stop and would otherwise never be restarted.
    sc.exe failureflag $svc 1 | Out-Null
}
```

2. `RUNBOOK.md`: add a troubleshooting-table row: `| Service stopped on its own with log line "ActorSystem terminated outside host shutdown" | SBR self-down (down-if-alone). The service recovery actions restart it automatically (requires install.ps1 with sc.exe failureflag — re-run install.ps1 or run the two sc.exe lines from §7 manually on older installs). If it does not come back: Start-Service manually and check network reachability between nodes. |` Also note next to the "auto-restart on failure" mention (line 87) that failureflag is part of the contract.
3. Verify: `grep -n "failureflag" deploy/wonder-app-vd03/install.ps1` (3 effective invocations via the loop → 1 hit inside it) and `grep -n "failureflag\|self-down" deploy/wonder-app-vd03/RUNBOOK.md` (≥1 hit). If `pwsh` is installed, syntax-check: `pwsh -NoProfile -Command "[void][System.Management.Automation.PSParser]::Tokenize((Get-Content -Raw deploy/wonder-app-vd03/install.ps1), [ref]$null)"` (exit 0); otherwise the greps suffice.
4. **No git commit** (gitignored artifact — same rule as Task 5).

### Task 7: Correct the factually wrong N2 deferral record in the master tracker

**Classification:** trivial (doc-only)
**Estimated implement time:** ~3 min
**Parallelizable with:** Task 1, Task 2, Task 4, Task 8, Task 9, Task 10, Task 11
**Files:**
- Modify: `archreview/plans/00-MASTER-TRACKER.md` (line 28 trailing sentence; "Deferred during Wave 1 PLAN-01 execution (2026-07-08)" bullet at lines 145-147)

The deferral rationale — "that production deploy artifact is not tracked in this repo … the artifact directory itself is not present in the repo" — is factually wrong (round-2 N2): `deploy/wonder-app-vd03/` exists in the working tree, is merely gitignored (`.gitignore:48: /deploy/`), and was edited during this initiative (PLAN-07's `_comment_Security`). Depends on Tasks 5-6 having applied the edits.

1. Rewrite the line-147 bullet: keep the history (what was deferred and why it mattered), then correct the record: the directory **is present but gitignored** (production config kept out of version control — the *tracking* claim was wrong, the *presence* claim was false); the three overlay edits were **applied on-disk 2026-07-12 by PLAN-R2-01 Tasks 5-6** (`NodeName` central-a/node-a, `AllowSingleNodeCluster` + phantom seeds dropped, `sc.exe failureflag` completing the already-present `sc.exe failure` actions). Remaining owner action: sync the artifact to `wonder-app-vd03` and restart services (appsettings take effect on restart; failureflag on re-running install.ps1 §7 or the two `sc.exe` lines manually) — per the RUNBOOK row added by T6.
2. Trim the stale trailing sentence of the line-28 Wave-1 summary ("were deferred because that production artifact is not tracked in this repo") → "were deferred on a mistaken not-in-repo rationale; corrected and applied on-disk by PLAN-R2-01 (2026-07-12) — see the Deferred-during-Wave-1 entry."
3. Do NOT add rows to `docs/plans/2026-07-08-deferred-work-register.md` — register consolidation (round-2 N6) is owned by **PLAN-R2-08**; this task only stops the tracker from asserting a falsehood.
4. Verify: `grep -n "not tracked in this repo" archreview/plans/00-MASTER-TRACKER.md` → 0 hits; `grep -n "PLAN-R2-01" archreview/plans/00-MASTER-TRACKER.md` → ≥2 hits.
5. Commit:
```
git add archreview/plans/00-MASTER-TRACKER.md
git commit -m "docs(tracker): correct the wonder-app-vd03 deferral record — artifact is present (gitignored); overlay edits applied on-disk by R2-01 T5/T6 (plan R2-01 T7)"
```

### Task 8: Metrics-staleness for never-reported sites (`FirstSeenAt` anchor)

**Classification:** high-risk (lock-free CAS state machine)
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 1, Task 2, Task 4, Task 5, Task 6, Task 7, Task 9, Task 10, Task 11
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthState.cs` (add `FirstSeenAt`)
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthAggregator.cs` (register branches lines 44-54 and 114-122; staleness guard lines 300-312)
- Modify: `docs/requirements/Component-HealthMonitoring.md` ("Metrics staleness" bullet, line ~47)
- Test: `tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/CentralHealthAggregatorTests.cs`

Round-2 N3: `CheckForOfflineSites` guards staleness with `state.LastReportReceivedAt is { } lastReport` (`CentralHealthAggregator.cs:300`), so a site registered via heartbeat only (`MarkHeartbeat` registration leaves `LastReportReceivedAt` null, `:114-122`) is NEVER flagged stale — "stuck from first boot" shows online-with-no-metrics forever. `LastHeartbeatAt` cannot be the anchor (heartbeats keep advancing it); anchor on a new `FirstSeenAt` stamped at registration. No UI change needed: the badge (`Health.razor:209-215`) keys on `IsOnline && IsMetricsStale` regardless of `LatestReport`.

1. Write the failing tests (append to `CentralHealthAggregatorTests` — reuse the existing `NewAggregator(metricsStaleTimeout:)` and `TestTimeProvider.Advance` helpers, lines 48-59/22):

```csharp
/// <summary>
/// Round-2 N3: a site known only via heartbeats (LastReportReceivedAt == null)
/// previously skipped the staleness check forever — a report pipeline dead
/// from FIRST BOOT showed "online with no metrics" indefinitely. FirstSeenAt
/// anchors the window instead (LastHeartbeatAt cannot: heartbeats advance it).
/// </summary>
[Fact]
public void HeartbeatOnlySite_NeverReported_IsFlaggedMetricsStale()
{
    var (aggregator, time) = NewAggregator(metricsStaleTimeout: TimeSpan.FromMinutes(2));
    aggregator.MarkHeartbeat("site-a", time.GetUtcNow());   // registered via heartbeat only
    time.Advance(TimeSpan.FromMinutes(3));
    aggregator.MarkHeartbeat("site-a", time.GetUtcNow());   // still heartbeating => online
    aggregator.CheckForOfflineSites();
    var state = aggregator.GetSiteState("site-a")!;
    Assert.True(state.IsOnline);                             // liveness unchanged
    Assert.True(state.IsMetricsStale);                       // was impossible before the fix
}

/// <summary>A heartbeat-only site inside the window is NOT falsely flagged.</summary>
[Fact]
public void HeartbeatOnlySite_WithinStaleWindow_IsNotFlagged()
{
    var (aggregator, time) = NewAggregator(metricsStaleTimeout: TimeSpan.FromMinutes(2));
    aggregator.MarkHeartbeat("site-a", time.GetUtcNow());
    time.Advance(TimeSpan.FromMinutes(1));
    aggregator.MarkHeartbeat("site-a", time.GetUtcNow());
    aggregator.CheckForOfflineSites();
    Assert.False(aggregator.GetSiteState("site-a")!.IsMetricsStale);
}
```

2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests --filter "FullyQualifiedName~HeartbeatOnlySite"` — expect the first test to FAIL (`IsMetricsStale` stays false; the null guard skips the site).
3. Implement:
   - `SiteHealthState`: add

```csharp
    /// <summary>
    /// Gets the instant this site was first observed by the aggregator (via a
    /// report or a heartbeat); <c>null</c> only for states not built by the
    /// aggregator (hand-constructed fixtures). Anchors metrics-staleness for a
    /// site that has NEVER delivered a report (review 01 round-2 N3): with
    /// <see cref="LastReportReceivedAt"/> null the staleness check previously
    /// skipped the site forever. <see cref="LastHeartbeatAt"/> cannot anchor
    /// this — every heartbeat advances it.
    /// </summary>
    public DateTimeOffset? FirstSeenAt { get; init; }
```

   - `CentralHealthAggregator.ProcessReport` register branch (lines 44-54): add `FirstSeenAt = now,`. `MarkHeartbeat` register branch (lines 114-122): add `FirstSeenAt = receivedAt,`. (Update branches never touch it — `with` copies preserve it.)
   - `CheckForOfflineSites` staleness guard (lines 300-312): replace the `state.LastReportReceivedAt is { } lastReport` condition with an anchor fallback:

```csharp
            // Second signal: … (keep the existing comment, append:)
            // A site that has NEVER reported (heartbeat-only registration,
            // LastReportReceivedAt == null) anchors on FirstSeenAt instead —
            // otherwise a pipeline dead from first boot is never flagged (N3).
            var reportAnchor = state.LastReportReceivedAt ?? state.FirstSeenAt;
            if (reportAnchor is { } anchor
                && now - anchor > _options.MetricsStaleTimeout
                && !state.IsMetricsStale)
            {
                var stale = state with { IsMetricsStale = true };
                if (_siteStates.TryUpdate(kvp.Key, stale, state))
                {
                    _logger.LogWarning(
                        "Site {SiteId} metrics are stale — online (heartbeats) but no report for {Elapsed}s{NeverNote} (timeout: {Timeout}s)",
                        state.SiteId, (now - anchor).TotalSeconds,
                        state.LastReportReceivedAt is null ? " (never reported since first contact)" : string.Empty,
                        _options.MetricsStaleTimeout.TotalSeconds);
                }
                // CAS loss ⇒ a fresh report/heartbeat swapped in ⇒ correct to skip.
            }
```

   - `Component-HealthMonitoring.md` "Metrics staleness" bullet (line ~47): append — "A site that has **never** delivered a report is anchored on its first-contact time (`FirstSeenAt`), so a report pipeline dead from first boot is also flagged after `MetricsStaleTimeout` (previously such a site showed online-with-no-metrics indefinitely)."
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests` — full project green (the CAS/flap/stale suites are the regression net).
5. Commit:
```
git add src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthState.cs src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthAggregator.cs docs/requirements/Component-HealthMonitoring.md tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/CentralHealthAggregatorTests.cs
git commit -m "fix(health): flag metrics-stale for never-reported sites via FirstSeenAt anchor — null LastReportReceivedAt no longer skips the check (plan R2-01 T8)"
```

### Task 9: Purge the stale "cluster leader" narration from `CentralHealthReportLoop`

**Classification:** trivial (comment-only)
**Estimated implement time:** ~2 min
**Parallelizable with:** Task 1, Task 2, Task 4, Task 5, Task 6, Task 7, Task 8, Task 10, Task 11
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthReportLoop.cs` (class doc line 12-14; sequence-seed comment lines 44-45)

Round-2 N4: the behavior is correct (line 93 gates on `IClusterNodeProvider.SelfIsPrimary` = oldest Up member since PLAN-01 T6), but the comments still narrate "cluster leader (Primary)" — re-teaching the exact leader/oldest conflation round-1 S2 eradicated.

1. Confirm scope first: `grep -in "leader" src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthReportLoop.cs` — expect exactly lines 12, 44, 45.
2. Edit:
   - Class doc (lines 12-14): "Only the cluster leader (Primary) generates reports — the standby's aggregator catches up on failover when it becomes Primary and starts its own loop." → "Only the active node (Primary = the **oldest Up member**, i.e. the singleton host — `IClusterNodeProvider.SelfIsPrimary`, never Akka cluster leadership) generates reports; the standby's aggregator catches up on failover when it becomes the oldest and its loop starts reporting."
   - Sequence-seed comment (lines 44-45): "reports from a newly-elected central leader always sort after reports from any prior leader" → "reports from a node that has newly become active (oldest Up member) always sort after reports from any previously-active node".
3. Verify: `grep -in "leader" src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthReportLoop.cs` — **0 hits**. Comment-only change: `dotnet build src/ZB.MOM.WW.ScadaBridge.HealthMonitoring` — succeeds.
4. Commit:
```
git add src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthReportLoop.cs
git commit -m "docs(health): CentralHealthReportLoop comments say oldest-member, not cluster leader — stop re-teaching the S2 conflation (plan R2-01 T9)"
```

### Task 10: Generalize the registrar — `SingletonRegistrar` with an optional role scope

**Classification:** high-risk (actor lifecycle helper used by all central singletons)
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 1, Task 2, Task 4, Task 5, Task 6, Task 7, Task 8, Task 9
**Files:**
- Modify (rename): `src/ZB.MOM.WW.ScadaBridge.Host/Actors/CentralSingletonRegistrar.cs` → `src/ZB.MOM.WW.ScadaBridge.Host/Actors/SingletonRegistrar.cs` (use `git mv`)
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs` (rename at the block comment lines 459-465 and the 7 central call sites: lines 466, 499, 564, 609, 632, 663, 689)
- Modify (rename): `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CentralSingletonRegistrarTests.cs` → `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SingletonRegistrarTests.cs`
- Modify: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CoordinatedShutdownTests.cs` (line 56 source-grep string)

Round-2 N5 first half: `CentralSingletonRegistrar` cannot express `.WithRole(siteRole)`, which is the sole reason the two site singletons stay hand-rolled without drains (`AkkaHostedService.cs:464-465`). Add an optional `role` applied to BOTH the manager and proxy settings, and rename the class — "Central" becomes a lie the moment it hosts site singletons.

1. Write the failing test — in the (renamed) `SingletonRegistrarTests.cs`, rename the class + existing references `CentralSingletonRegistrar` → `SingletonRegistrar`, and add:

```csharp
    [Fact]
    public async Task Start_WithRoleScope_CreatesRoleScopedSingleton_ThatAnswers()
    {
        // Round-2 N5: role scoping is what kept the two SITE singletons
        // (deployment-manager, event-log-handler) hand-rolled and drain-less.
        using var system = CreateSingleNodeClusterSystem(); // roles = ["Central"]
        var handle = SingletonRegistrar.Start(
            system, "role-widget", Props.Create(() => new EchoActor()),
            NullLogger.Instance, drainTimeout: TimeSpan.FromSeconds(2), role: "Central");

        Assert.EndsWith("/user/role-widget-singleton", handle.Manager.Path.ToString());
        Assert.EndsWith("/user/role-widget-proxy", handle.Proxy.Path.ToString());

        // The node holds the role => manager instantiates the singleton and
        // the role-scoped proxy resolves it.
        var echo = await handle.Proxy.Ask<string>("hi", TimeSpan.FromSeconds(20));
        Assert.Equal("hi", echo);
    }
```

2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~SingletonRegistrarTests"` — expect FAIL (compile: no `SingletonRegistrar` type / no `role` parameter).
3. Implement:
   - `git mv src/ZB.MOM.WW.ScadaBridge.Host/Actors/CentralSingletonRegistrar.cs src/ZB.MOM.WW.ScadaBridge.Host/Actors/SingletonRegistrar.cs`; rename the class; signature → `Start(ActorSystem system, string name, Props singletonProps, ILogger logger, TimeSpan? drainTimeout = null, string? role = null)`; apply the role to both settings:

```csharp
        var managerSettings = ClusterSingletonManagerSettings.Create(system).WithSingletonName(name);
        if (role is not null) managerSettings = managerSettings.WithRole(role);
        // … ClusterSingletonManager.Props(singletonProps, PoisonPill.Instance, managerSettings) …

        var proxySettings = ClusterSingletonProxySettings.Create(system).WithSingletonName(name);
        if (role is not null) proxySettings = proxySettings.WithRole(role);
        // … ClusterSingletonProxy.Props($"/user/{name}-singleton", proxySettings) …
```

   Update the class XML doc: drop "central"-only framing; drain rationale now reads "so in-flight EF (central) or SQLite (site) work completes before handover"; note the optional `role` maps to `ClusterSingletonManagerSettings/ProxySettings.WithRole` (review 01 round-2 N5).
   - `AkkaHostedService.cs`: mechanical rename at the 7 call sites (466, 499, 564, 609, 632, 663, 689) and rewrite the lines-459-465 comment's last sentence ("The two SITE singletons stay hand-rolled below because they are role-scoped") → "Site singletons use the same registrar with `role: siteRole` (Task 11 / round-2 N5)." (Task 11 lands that; wording here is forward-referencing but true within the same plan.)
   - `git mv tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CentralSingletonRegistrarTests.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SingletonRegistrarTests.cs`.
   - `CoordinatedShutdownTests.cs` line 56: `Assert.Contains("CentralSingletonRegistrar.Start(", content);` → `Assert.Contains("SingletonRegistrar.Start(", content);` (also update the comment above it).
   - Sanity: `grep -rn "CentralSingletonRegistrar" src/ tests/ --include="*.cs"` — 0 hits.
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~SingletonRegistrarTests|FullyQualifiedName~CoordinatedShutdownTests"` — PASS. Then `dotnet build ZB.MOM.WW.ScadaBridge.slnx` — clean.
5. Commit:
```
git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/SingletonRegistrar.cs src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SingletonRegistrarTests.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CoordinatedShutdownTests.cs
git rm --cached src/ZB.MOM.WW.ScadaBridge.Host/Actors/CentralSingletonRegistrar.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CentralSingletonRegistrarTests.cs 2>/dev/null || true
git commit -m "refactor(host): CentralSingletonRegistrar -> SingletonRegistrar with optional role scope — unblocks site-singleton drains (plan R2-01 T10)"
```
(`git mv` already stages the renames; the `git rm --cached … || true` line is a no-op safety net if the moves were done as delete+create.)

### Task 11: Route the two site singletons through the registrar — deployment-manager + event-log-handler gain drains

**Classification:** high-risk (site actor wiring refactor)
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 1, Task 2, Task 4, Task 5, Task 6, Task 7, Task 8, Task 9
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs` (site branch: deployment-manager block lines ~791-820; event-log-handler block lines ~836-856)
- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CoordinatedShutdownTests.cs`

Round-2 N5 second half: `deployment-manager` and `event-log-handler` are hand-rolled with bare `PoisonPill` termination and no `PhaseClusterLeave` drain. The round-1 S5 rationale applies identically on the site side: the DeploymentManager's SQLite writes (static overrides, `native_alarm_state`) should drain before handover — currently safe-but-lossy on graceful failover.

1. Write the failing test — in `CoordinatedShutdownTests.cs`, update the existing `AllCentralSingletons_RegisterThroughRegistrarWithDrain` (lines 45-70): change the final assertion `Assert.Equal(2, CountOccurrences(content, "ClusterSingletonManager.Props("));` → `Assert.Equal(0, …)` and replace its "the only two left are the SITE singletons" comment; then add:

```csharp
    [Fact]
    public void SiteSingletons_RegisterThroughRegistrarWithDrain()
    {
        var hostProjectDir = FindHostProjectDirectory();
        Assert.NotNull(hostProjectDir);
        var content = File.ReadAllText(Path.Combine(hostProjectDir!, "Actors", "AkkaHostedService.cs"));

        // Round-2 N5: the two role-scoped SITE singletons previously stayed
        // hand-rolled (bare PoisonPill, no PhaseClusterLeave drain). They now
        // go through SingletonRegistrar.Start(..., role: siteRole), which
        // always adds the GracefulStop drain — in-flight SQLite writes
        // (static overrides, native_alarm_state) complete before handover.
        Assert.Contains("\"deployment-manager\"", content);
        Assert.Contains("\"event-log-handler\"", content);
        Assert.Contains("role: siteRole", content);
        Assert.Equal(0, CountOccurrences(content, "ClusterSingletonManager.Props("));
    }
```

2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~SiteSingletons_RegisterThroughRegistrarWithDrain"` — expect FAIL (`ClusterSingletonManager.Props(` count is 2; no `role: siteRole`).
3. Implement in the site branch of `AkkaHostedService.cs`:
   - Deployment-manager block (~791-820): replace the hand-rolled `ClusterSingletonManager.Props`/`ClusterSingletonProxy.Props` pair with

```csharp
        // Deployment Manager — role-scoped singleton via SingletonRegistrar
        // (review 01 round-2 N5): previously hand-rolled with bare PoisonPill
        // termination and NO PhaseClusterLeave drain, so in-flight SQLite
        // writes (static overrides, native_alarm_state) could be cut off on
        // graceful failover. Names unchanged: deployment-manager-singleton /
        // deployment-manager-proxy.
        var dm = SingletonRegistrar.Start(
            _actorSystem!, "deployment-manager",
            Props.Create(() => new DeploymentManagerActor(
                storage, compilationService, sharedScriptLibrary, streamManager,
                siteRuntimeOptionsValue, dmLogger, dclManager, replicationActor,
                siteHealthCollector, _serviceProvider, null, deploymentConfigFetcher)),
            _logger, role: siteRole);
        var dmProxy = dm.Proxy;
```

   (Keep the `dmProxy` variable name — everything downstream, `SiteCommunicationActor` construction and `RegisterLocalHandler(LocalHandlerType.Artifacts, dmProxy)`, is untouched.)
   - Event-log-handler block (~836-856): inside the existing `if (eventLogQueryService != null)`:

```csharp
            var eventLog = SingletonRegistrar.Start(
                _actorSystem, "event-log-handler",
                Props.Create(() => new SiteEventLogging.EventLogHandlerActor(eventLogQueryService)),
                _logger, role: siteRole);
            siteCommActor.Tell(new RegisterLocalHandler(LocalHandlerType.EventLog, eventLog.Proxy));
```

   (Keep the existing "Event log handler — cluster singleton so queries always reach the active node…" comment above it.)
   - Actor/singleton names are reproduced exactly by the registrar (`deployment-manager-singleton`/`-proxy`, `event-log-handler-singleton`/`-proxy`), so path-pinning tests and the ClusterClient receptionist wiring see no change.
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests` (full project — path/health-check/audit-wiring suites are the regression net) and `dotnet build ZB.MOM.WW.ScadaBridge.slnx` — PASS/clean.
5. Commit:
```
git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CoordinatedShutdownTests.cs
git commit -m "fix(host): site singletons (deployment-manager, event-log-handler) via SingletonRegistrar — PhaseClusterLeave drains for site SQLite writes (plan R2-01 T11)"
```

---

## Dependencies on other plans

- **PLAN-R2-02 owns** the oldest-vs-leader predicate unification inside `SiteCommunicationActor`/`SiteReplicationActor` and everything resync-related. This plan does not touch those files; the coverage table references it.
- **PLAN-R2-08 owns** N6 (folding the two live PLAN-01 deferrals — the keep-oldest active-crash gap and, post-T5/T6, the "sync the artifact to the host" residue — into `docs/plans/2026-07-08-deferred-work-register.md`). Task 7 here only corrects the factually wrong tracker prose; it adds no register rows.
- The **SBR keep-oldest active-crash availability gap stays a registered deferred USER decision** — no task here changes the strategy/topology; Tasks 1-3 make the gap observable and measured, which is the input that decision needs.

## Execution order

**Drill spine (serial):** Task 1 (script) → Task 2 (docs) → Task 3 (live run, needs the docker cluster exclusively).

**Registrar spine (serial):** Task 10 → Task 11.

**Freely parallel at any point:** Tasks 4, 5, 6, 8, 9 (all file-disjoint); Task 7 after 5+6.

**Highest value if time-boxed:** Task 3 (the never-run drill, run at last) + Task 11 (real durability gap) + Task 8 (real monitoring blind spot).

---

## Findings Coverage

Every round-2 finding (N1-N6), the round-1 partials the report says warrant completion, and the cross-plan rulings:

| # | Round-2 finding | Severity | Task(s) / Disposition |
|---|----------------|----------|----------------------|
| N1 | Failover drill kills the wrong (active/oldest) node; first-seed bootstrap gap means the survivor's recovery story doesn't converge; drill never executed; ~25s promise undeliverable in its most likely case | High | Tasks 1 (drill rewrite: standby-victim default + explicit active-gap mode), 2 (narrative + first-seed constraint **documented** with operator actions — deliberately NOT reordered to self-first seeds: simultaneous cold-start self-joins would risk two independent clusters), 3 (RUN the drill live, both directions, record measured timings), 4 (wire/unskip `FailoverTimingTests` to `TwoNodeClusterFixture`, measure the ~25s envelope at production timings — **also covers report-08 NF2 by reference**). The keep-oldest topology/strategy remedy itself **stays deferred: registered USER decision** (master tracker 2026-07-08) — Tasks 1-3 document/measure it, never fix it |
| N2 | wonder-app-vd03 overlay edits unapplied; deferral rationale factually wrong (artifact IS in the working tree, merely gitignored) | Medium | Tasks 5 (NodeName + AllowSingleNodeCluster + drop phantom seeds, both overlays), 6 (`sc.exe failureflag` completing the recovery actions + RUNBOOK step — note: `sc.exe failure` itself is already on disk, contra the report), 7 (correct the deferral record in the master tracker). No git commits for the gitignored artifact itself |
| N3 | `IsMetricsStale` can never fire for a site that has never delivered a report (null `LastReportReceivedAt` guard) | Low | Task 8 (`FirstSeenAt` anchor; no UI change needed — badge already keys on the flag) |
| N4 | `CentralHealthReportLoop` still narrates "cluster leader (Primary)" | Low | Task 9 |
| N5 | Site-role singletons lack drain tasks; registrar can't express role scoping | Low | Tasks 10 (role param + rename to `SingletonRegistrar`), 11 (route `deployment-manager` + `event-log-handler` through it) |
| N6 | PLAN-01's two live deferrals missing from the canonical deferred-work register | Low | **Reference → PLAN-R2-08** (register consolidation is its scope). Task 7 here only fixes the falsehood in the master-tracker prose |
| S12 (R1 partial) | Single-node artifact still ships the phantom seed, lacks `AllowSingleNodeCluster` | — | Task 5 (the artifact half; the flag/validator half shipped in PLAN-01 T16) |
| U2 (R1 partial) | install.ps1 recovery actions deferred; recovery contract doesn't converge for a non-first-seed survivor | — | Task 6 (failureflag) + Task 2 (first-seed constraint + operator actions in the spec'd recovery contract) |
| U5 (R1 not fixed) | `NodeName` missing from wonder-app-vd03 overlays → NULL `SourceNode` | — | Task 5 |
| C3 (R1 partial) | LDAP `Transport: None, AllowInsecure: true` remains in the deploy artifact | — | **Stays accepted posture per the report** — bundled loopback GLAuth with documented AD-repoint instructions; a future over-the-wire AD repoint must revisit transport. No task |
| — | Oldest-vs-leader predicate unification in `SiteCommunicationActor`/`SiteReplicationActor` + resync work | — | **Reference → PLAN-R2-02** (explicit cross-plan ownership; not touched here) |