Joseph Doherty
5bbd7689fa
Re-ran all 8 domain reviews at HEAD8c888f13against theb910f5ebbaseline: every round-1 finding source-verified (168 fixed, 0 regressions, 0 false claims); 56 new findings (1 Critical / 4 High / 15 Medium / 36 Low), concentrated in post-baseline code (anti-entropy resync, KPI rollup backfill, live alarm stream) and seams the fixes exposed. Headliners: S&F resync predicate inversion can wipe the delivering node's buffer (02-N1 Critical); resync snapshot exceeds the Akka remoting frame size (02-N2); failover drill kills the one node keep-oldest can't survive (01-N1); unbounded rollup backfill per failover (04-R1); live production API key in untracked test.txt (08-NF1). Adds PLAN-R2-01..08 + .tasks.json manifests and the Round-2 board, P0 list, cross-plan mutexes, and wave order in 00-MASTER-TRACKER.
700 lines
56 KiB
Markdown
700 lines
56 KiB
Markdown
# PLAN-R2-01 — Cluster, Host & Failover Round-2 Fixes Implementation Plan
|
||
|
||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
|
||
|
||
**Goal:** Make the failover story *honest* — rewrite and actually RUN the failover drill so it proves what two-node keep-oldest can deliver (and measures what it cannot), apply the three wonder-app-vd03 overlay edits that were deferred on a factually wrong rationale, and close the round-2 residuals (never-reported metrics staleness, stale leader comments, undrained site singletons).
|
||
|
||
**Architecture:** The drill (`docker/failover-drill.sh`) gains a `DRILL_MODE` — `standby` (default, the survivable younger-node crash that PASSes within the ~25s SBR budget) and `active` (the oldest-node crash that keep-oldest cannot survive: the drill *expects* the outage and measures it, documenting the registered deferred user decision instead of codifying a recovery the topology cannot deliver). The in-process envelope measurement reuses `TwoNodeClusterFixture` (production `BuildHocon`, now with production-timing knobs) from the previously-skipped `FailoverTimingTests`. Site-role singleton drains come from generalizing `CentralSingletonRegistrar` into a role-aware `SingletonRegistrar` so `deployment-manager` and `event-log-handler` get the same `PhaseClusterLeave` GracefulStop the seven central singletons got in round 1.
|
||
|
||
**Tech Stack:** C#/.NET, Akka.NET 1.5.62 (Akka.Cluster, Akka.Cluster.Tools), xUnit, bash + Docker Compose + Traefik, JSON deploy overlays (PowerShell `install.ps1`).
|
||
|
||
Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per-project: `dotnet test tests/<project>`. Docker rig: `bash docker/deploy.sh`.
|
||
|
||
**Binding scope rulings (do not deviate):**
|
||
- The two-node keep-oldest **active/oldest-crash availability gap is a registered deferred USER decision** (master tracker 2026-07-08 "Follow-up discovered during P0 execution"; `SbrFailoverTests.cs:15-23`). No task in this plan changes the SBR strategy or topology — Tasks 1–3 *document and measure* the gap.
|
||
- The oldest-vs-leader predicate unification inside `SiteCommunicationActor`/`SiteReplicationActor` and everything resync-related is **owned by PLAN-R2-02** — do not touch those files here.
|
||
- N6 (deferred-work-register consolidation) is **owned by PLAN-R2-08** — Task 7 here corrects only the factually wrong N2 prose in the master tracker.
|
||
- `/deploy/` is **gitignored** (`.gitignore:48: /deploy/`) — Tasks 5 and 6 edit the on-disk artifact and deliberately do NOT `git add` it (never use `git add -f`); Task 7's tracker correction is the version-controlled record.
|
||
|
||
---
|
||
|
||
### Task 1: Rewrite `failover-drill.sh` — standby-victim default + explicit active-victim gap mode
|
||
|
||
**Classification:** small
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** Task 4, Task 5, Task 6, Task 8, Task 9, Task 10
|
||
**Files:**
|
||
- Modify: `docker/failover-drill.sh` (full rewrite — current script is 45 lines; lines 12-19 pick the ACTIVE node as victim, which is exactly the crash keep-oldest cannot survive)
|
||
|
||
Context (round-2 N1): under the round-1 oldest-member unification, the *active* node IS the oldest. `docker/failover-drill.sh:17-19` kills it, so the shipped drill exercises the one crash two-node keep-oldest is *known* not to recover from (the survivor self-downs; the restarted `central-b` cannot re-bootstrap because both nodes list `scadabridge-central-a` as FIRST seed — `docker/central-node-a/appsettings.Central.json:10-13`, `docker/central-node-b/appsettings.Central.json:10-13` — and only the first seed may self-join to form a new cluster). The drill has also never been run (`docker/README.md:288`). Fix the drill so its PASS criteria match reality per direction.
|
||
|
||
1. No unit test (bash script). The verification command is the "test": `bash -n docker/failover-drill.sh` (must exit 0), plus `DRILL_MODE=bogus bash docker/failover-drill.sh` (must exit 2 with the usage error — this exercises the argument guard without needing a cluster).
|
||
2. Replace the script's entire contents with:
|
||
|
||
```bash
|
||
#!/usr/bin/env bash
|
||
# Failover drill against the running docker cluster (bash docker/deploy.sh first).
|
||
#
|
||
# ROUND-2 REWRITE (arch-review 01 round 2, N1). The original drill killed the
|
||
# ACTIVE central node — but under the unified oldest-member semantics the
|
||
# active node IS the oldest, i.e. the one crash two-node keep-oldest CANNOT
|
||
# survive (registered deferred user decision, master tracker 2026-07-08;
|
||
# SbrFailoverTests.cs XML doc). Two modes:
|
||
#
|
||
# DRILL_MODE=standby (default) — kills the STANDBY (younger) central node.
|
||
# The survivable direction: SBR downs the crashed member, the active node
|
||
# keeps its singletons, and Traefik routing never goes dark. PASS = the
|
||
# survivor logs the member removal within TIMEOUT_S (budget ~25s+: 10s
|
||
# failure detection + 15s stable-after) while /health/active stays up.
|
||
#
|
||
# DRILL_MODE=active — kills the ACTIVE (oldest) central node. THE EXPECTED
|
||
# OUTCOME IS A TOTAL CENTRAL OUTAGE: keep-oldest downs the partition
|
||
# without the oldest, so the younger survivor downs ITSELF (down-if-alone
|
||
# cannot help — the alone-oldest is dead and cannot down itself), and the
|
||
# self-downed survivor cannot re-form a cluster alone unless it is the
|
||
# FIRST seed (both nodes list central-a first; only the first seed may
|
||
# self-join). This mode measures the dark window and PASSes only when
|
||
# central recovers AFTER the victim container is restarted. It exists to
|
||
# make the registered gap observable — not to pretend it is covered.
|
||
set -euo pipefail
|
||
|
||
TRAEFIK_URL="${TRAEFIK_URL:-http://localhost:9000}"
|
||
TIMEOUT_S="${TIMEOUT_S:-90}"
|
||
DRILL_MODE="${DRILL_MODE:-standby}"
|
||
OUTAGE_CONFIRM_S="${OUTAGE_CONFIRM_S:-60}"
|
||
|
||
active_container() {
|
||
if curl -sf -o /dev/null "http://localhost:9001/health/active"; then echo scadabridge-central-a
|
||
elif curl -sf -o /dev/null "http://localhost:9002/health/active"; then echo scadabridge-central-b
|
||
else echo "ERROR: no active central node found" >&2; exit 1; fi
|
||
}
|
||
peer_of() { [ "$1" = scadabridge-central-a ] && echo scadabridge-central-b || echo scadabridge-central-a; }
|
||
|
||
case "$DRILL_MODE" in
|
||
standby|active) ;;
|
||
*) echo "ERROR: DRILL_MODE must be 'standby' or 'active' (was '$DRILL_MODE')" >&2; exit 2 ;;
|
||
esac
|
||
|
||
ACTIVE=$(active_container)
|
||
if [ "$DRILL_MODE" = standby ]; then
|
||
VICTIM=$(peer_of "$ACTIVE"); SURVIVOR="$ACTIVE"
|
||
else
|
||
VICTIM="$ACTIVE"; SURVIVOR=$(peer_of "$ACTIVE")
|
||
fi
|
||
|
||
echo "mode=${DRILL_MODE} active=${ACTIVE} victim=${VICTIM} survivor=${SURVIVOR}"
|
||
KILL_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||
docker kill "${VICTIM}" > /dev/null
|
||
START=$(date +%s)
|
||
|
||
if [ "$DRILL_MODE" = standby ]; then
|
||
echo "Standby crash: waiting for ${SURVIVOR} to DOWN+REMOVE the dead member (SBR budget ~25s)..."
|
||
BLIPS=0
|
||
while true; do
|
||
ELAPSED=$(( $(date +%s) - START ))
|
||
curl -sf -o /dev/null "${TRAEFIK_URL}/health/active" || BLIPS=$((BLIPS + 1))
|
||
if docker logs --since "${KILL_AT}" "${SURVIVOR}" 2>&1 | grep -Eiq "marking.*node.*down|member removed|is removed"; then
|
||
echo "PASS: survivor removed the crashed member in ${ELAPSED}s (budget ~25s: 10s detection + 15s stable-after)."
|
||
echo "Active-node routing blips during the drill: ${BLIPS} (expected 0 — the active node was never touched)."
|
||
break
|
||
fi
|
||
if (( ELAPSED > TIMEOUT_S )); then
|
||
echo "FAIL: no downing/removal evidence on ${SURVIVOR} after ${ELAPSED}s — SBR did not act" >&2
|
||
docker start "${VICTIM}" > /dev/null
|
||
exit 1
|
||
fi
|
||
sleep 1
|
||
done
|
||
else
|
||
echo "Active crash: EXPECTING a central outage (registered keep-oldest gap). Watching /health/active..."
|
||
DARK_STREAK=0
|
||
while true; do
|
||
ELAPSED=$(( $(date +%s) - START ))
|
||
if curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; then DARK_STREAK=0; else DARK_STREAK=$((DARK_STREAK + 1)); fi
|
||
if (( DARK_STREAK >= 10 )); then
|
||
echo "Outage confirmed at ${ELAPSED}s: no active central node — the younger survivor self-downed"
|
||
echo "(keep-oldest downs the partition WITHOUT the oldest; this is the registered deferred gap)."
|
||
break
|
||
fi
|
||
if (( ELAPSED > OUTAGE_CONFIRM_S )); then
|
||
echo "NOTE: /health/active stayed reachable ${ELAPSED}s after killing the oldest — better than the"
|
||
echo "registered gap predicts. Do NOT celebrate: capture both nodes' logs and investigate before trusting it."
|
||
break
|
||
fi
|
||
sleep 1
|
||
done
|
||
fi
|
||
|
||
echo "Restarting ${VICTIM}..."
|
||
docker start "${VICTIM}" > /dev/null
|
||
RESTART_AT=$(date +%s)
|
||
echo "Waiting for central to be routable again through Traefik (${TRAEFIK_URL}/health/active)..."
|
||
while true; do
|
||
ELAPSED=$(( $(date +%s) - RESTART_AT ))
|
||
if curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; then
|
||
echo "Recovered: an active central node is routable ${ELAPSED}s after the victim restart."
|
||
break
|
||
fi
|
||
if (( ELAPSED > 120 )); then
|
||
echo "FAIL: central not routable 120s after restarting ${VICTIM}" >&2
|
||
exit 1
|
||
fi
|
||
sleep 1
|
||
done
|
||
|
||
echo "Survivor singleton/downing evidence (last 20 matching log lines from ${SURVIVOR}):"
|
||
docker logs "${SURVIVOR}" 2>&1 | grep -Ei "singleton|oldest|downing|removed" | tail -20 || true
|
||
echo "Drill complete (${DRILL_MODE}). Verify on the Health dashboard that both nodes show Up and exactly one is Primary."
|
||
```
|
||
|
||
3. `chmod +x docker/failover-drill.sh`. Verify: `bash -n docker/failover-drill.sh` (exit 0) and `DRILL_MODE=bogus bash docker/failover-drill.sh; echo "exit=$?"` (prints the usage error, exit=2).
|
||
4. Commit:
|
||
```
|
||
git add docker/failover-drill.sh
|
||
git commit -m "fix(docker): failover drill kills the STANDBY by default; active mode measures the registered keep-oldest outage instead of pretending recovery (plan R2-01 T1)"
|
||
```
|
||
|
||
### Task 2: Correct the recovery narrative + document the first-seed bootstrap constraint
|
||
|
||
**Classification:** trivial (doc-only)
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** Task 4, Task 5, Task 6, Task 8, Task 9, Task 10
|
||
**Files:**
|
||
- Modify: `docker/README.md` (Failover Testing → "Automated failover drill" section, lines 273-288)
|
||
- Modify: `docs/requirements/Component-ClusterInfrastructure.md` ("Down-if-alone recovery" subsection, lines ~104-116)
|
||
|
||
1. `docker/README.md` (lines 273-288): rewrite the drill section to match Task 1's two modes. Specifically:
|
||
- Replace the four numbered bullets with per-mode descriptions: `DRILL_MODE=standby` (default) = the survivable younger-crash direction — expected result: **no routing outage at all** (the active node is never touched) and member removal on the survivor within **~25s** (10s failure-detection threshold + 15s stable-after; the 2s heartbeat interval is not additive); `DRILL_MODE=active` = the oldest-crash direction — expected result: **total central outage** until the victim container is restarted (the registered deferred keep-oldest decision, master tracker 2026-07-08), then recovery within ~2 min of the restart.
|
||
- Delete the line-286 promise "Expected failover: **~25s** … plus the Traefik ~5s health-check interval" — it described a singleton *migration* on active-crash that two-node keep-oldest cannot deliver. State per-direction expectations instead.
|
||
- Replace the line-288 "Observed failover time: _run after next deploy and record here_" placeholder with a two-row results table Task 3 will fill:
|
||
|
||
```markdown
|
||
> **Observed results** (recorded by plan R2-01 T3, cluster @ commit <sha>):
|
||
>
|
||
> | Direction (`DRILL_MODE`) | Outcome | Measured |
|
||
> |--------------------------|---------|----------|
|
||
> | `standby` (younger-node crash) | _pending T3_ | _pending T3_ |
|
||
> | `active` (oldest-node crash) | _pending T3_ | _pending T3_ |
|
||
```
|
||
|
||
- Add a short **"Seed-node bootstrap constraint"** note to the same section: only the FIRST seed in `Cluster:SeedNodes` may self-join to form a *new* cluster; both central nodes list `scadabridge-central-a` first, so a lone restarted `central-b` (with `central-a` still down) loops on `InitJoin` forever — never `Up`, never `/health/active` 200. Operator recovery actions: **(1)** restart the dead first-seed node (preferred), or **(2)** restart the survivor with a self-first seed override (env: `ScadaBridge__Cluster__SeedNodes__0=akka.tcp://scadabridge@<self-host>:8081`, `...__1=<peer>`). Explain why the repo does NOT ship self-first ordering per node: with *both* nodes self-first, a simultaneous cold start can let each self-join independently → two one-node clusters that never merge (a cold-start split-brain the identical-seed-order convention exists to prevent). The real remedy is the pending keep-oldest topology/strategy decision (deferred, owner: user).
|
||
2. `docs/requirements/Component-ClusterInfrastructure.md` (Down-if-alone recovery, ~104-116):
|
||
- Amend step 4 ("The restarted process rejoins as a **fresh incarnation** (the keep-oldest resolver handles the rejoin cleanly…)"): add the qualifier that this holds **only while a peer holding cluster state is still reachable** — a lone restarted non-first-seed node cannot re-form a cluster (first-seed self-join rule) and waits for its peer.
|
||
- Add a "Seed-node bootstrap constraint" paragraph after the numbered list (same content as the README note, spec-toned), cross-referencing the registered deferred keep-oldest decision and the operator actions.
|
||
- Fix the closing line "The docker failover drill exercises this loop end-to-end (kill the active node, assert singleton migration + clean rejoin)" → "The docker failover drill (`docker/failover-drill.sh`) exercises this per direction: `standby` mode proves SBR downing + singleton continuity on the oldest; `active` mode measures the registered total-outage gap and the recovery-on-restart path."
|
||
3. Verification (doc task — no test): `grep -n "DRILL_MODE" docker/README.md` (≥2 hits), `grep -cn "singleton migration" docs/requirements/Component-ClusterInfrastructure.md` (0 hits in the recovery section), `grep -n "first seed\|first-seed" docker/README.md docs/requirements/Component-ClusterInfrastructure.md` (≥2 hits), `grep -n "Traefik ~5s health-check interval" docker/README.md` (0 hits).
|
||
4. Commit:
|
||
```
|
||
git add docker/README.md docs/requirements/Component-ClusterInfrastructure.md
|
||
git commit -m "docs(cluster): per-direction failover truth — first-seed bootstrap constraint + operator recovery actions; drop the undeliverable ~25s active-crash promise (plan R2-01 T2)"
|
||
```
|
||
|
||
### Task 3: RUN the drill live (both directions) and record measured timings
|
||
|
||
**Classification:** standard (live-cluster verification; ~5 min of work, several minutes of wall-clock waiting)
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** none (exclusive use of the shared docker cluster; writes `docker/README.md`)
|
||
**Files:**
|
||
- Modify: `docker/README.md` (fill the Task 2 results table)
|
||
|
||
The drill has NEVER been executed (round-2 N1 point 3, `docker/README.md:288`). This task produces the empirical record — the asymmetric result *is* the documentation the deferred keep-oldest decision needs.
|
||
|
||
1. Ensure infra + cluster are up and freshly built (drill needs the Task 1 script inside the checkout only — no image change required, but the cluster must be healthy):
|
||
```
|
||
cd infra && docker compose up -d && cd ..
|
||
bash docker/deploy.sh
|
||
```
|
||
Wait until `curl -sf http://localhost:9000/health/active` returns 200 (poll; allow ~2 min for first-boot readiness).
|
||
2. Run direction 1 and capture:
|
||
```
|
||
DRILL_MODE=standby bash docker/failover-drill.sh 2>&1 | tee /private/tmp/claude-501/-Users-dohertj2-Desktop-ScadaBridge/d69633c2-b169-4fe1-8fb4-0d8007a16fa1/scratchpad/drill-standby.log
|
||
```
|
||
Expected: PASS with member-removal time ≈ 25-30s and `routing blips: 0`. Wait for the victim to fully rejoin (script does this) plus ~30s settle time.
|
||
3. Run direction 2 and capture:
|
||
```
|
||
DRILL_MODE=active bash docker/failover-drill.sh 2>&1 | tee /private/tmp/claude-501/-Users-dohertj2-Desktop-ScadaBridge/d69633c2-b169-4fe1-8fb4-0d8007a16fa1/scratchpad/drill-active.log
|
||
```
|
||
Expected: "Outage confirmed at ~Ns" (survivor self-downs; central dark), then "Recovered: … Ms after the victim restart". If instead the NOTE branch fires (`/health/active` stayed up), capture `docker logs scadabridge-central-a` + `-b` and investigate before recording — do not record a result that contradicts the registered gap without evidence.
|
||
4. Fill the `docker/README.md` results table from the two logs: outcome + measured seconds per direction, and the cluster commit SHA (`git rev-parse --short HEAD`). Keep the raw one-line summary quotes from the script output.
|
||
5. Sanity-restore: confirm both nodes healthy afterwards — `curl -sf http://localhost:9001/health/ready && curl -sf http://localhost:9002/health/ready` (both 200; the standby's `/health/active` correctly 503).
|
||
6. Commit:
|
||
```
|
||
git add docker/README.md
|
||
git commit -m "test(docker): record live failover-drill results for both crash directions — standby-crash PASS timing + active-crash measured outage (plan R2-01 T3)"
|
||
```
|
||
|
||
### Task 4: Wire `FailoverTimingTests` to `TwoNodeClusterFixture` — measure the ~25s envelope in-process
|
||
|
||
**Classification:** high-risk (real two-node cluster timing test)
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** Task 1, Task 2, Task 5, Task 6, Task 7, Task 8, Task 9, Task 10, Task 11
|
||
**Files:**
|
||
- Modify: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/TwoNodeClusterFixture.cs` (StartAsync lines 24-36, StartNode lines 39-65 — add production-timing knobs)
|
||
- Modify: `tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/ZB.MOM.WW.ScadaBridge.PerformanceTests.csproj` (add a ProjectReference)
|
||
- Modify: `tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Failover/FailoverTimingTests.cs` (replace the `Skip` placeholder — currently `[Fact(Skip = "Requires the real two-node failover rig delivered by PLAN-01 …")]` at lines 25-32)
|
||
|
||
This task **also covers architecture-review report 08's NF2 by reference** (the skipped perf placeholder): the rig it waits for exists (`TwoNodeClusterFixture`, PLAN-01 T3); wire it up. The fixture hard-codes scaled timings (`HeartbeatInterval` 500ms / `FailureDetectionThreshold` 2s at `TwoNodeClusterFixture.cs:51-52`); to measure the *production* ~25s envelope (2s heartbeat / 10s threshold / 15s stable-after), the knobs become optional parameters with the current values as defaults (additive — `SbrFailoverTests`/`ActiveNodeSemanticsTests` unaffected). Direction note: the measurable crash is the YOUNGER node (the survivable direction); the oldest-crash direction is the registered deferred gap and is *not* a recovery to time — say so in the test's XML doc.
|
||
|
||
1. Rewrite `FailoverTimingTests.cs` (this is the failing step — it will not compile until the fixture knobs + project reference exist):
|
||
|
||
```csharp
|
||
using System.Diagnostics;
|
||
using Akka.Actor;
|
||
using Akka.Cluster.Tools.Singleton;
|
||
using Xunit.Abstractions;
|
||
using ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;
|
||
|
||
namespace ZB.MOM.WW.ScadaBridge.PerformanceTests.Failover;
|
||
|
||
/// <summary>
|
||
/// Failover-timing measurement on the real two-node in-process rig
|
||
/// (TwoNodeClusterFixture, production BuildHocon) at PRODUCTION timings:
|
||
/// 2s heartbeat / 10s failure-detection threshold / 15s SBR stable-after —
|
||
/// the CLAUDE.md "total failover ~25s" design envelope.
|
||
///
|
||
/// Measures the SURVIVABLE direction only: hard-crash of the YOUNGER node,
|
||
/// timed to the survivor's member REMOVAL (detection + stable-after + gossip)
|
||
/// with singleton continuity asserted on the oldest. The oldest/active-node
|
||
/// crash is NOT a recovery to time — two-node keep-oldest makes the younger
|
||
/// survivor down itself (total outage; registered deferred user decision,
|
||
/// see SbrFailoverTests XML doc + master tracker 2026-07-08). Covers overall
|
||
/// review P2-10 / report-08 NF2 and report-01 round-2 N1's measurement ask.
|
||
/// </summary>
|
||
public class FailoverTimingTests(ITestOutputHelper output)
|
||
{
|
||
private sealed class EchoActor : ReceiveActor
|
||
{
|
||
public EchoActor() => ReceiveAny(msg => Sender.Tell(msg));
|
||
}
|
||
|
||
[Trait("Category", "Performance")]
|
||
[Fact]
|
||
public async Task HardKillOfYoungerNode_SbrRemovalWithinDesignEnvelope()
|
||
{
|
||
await using var cluster = await TwoNodeClusterFixture.StartAsync(
|
||
stableAfter: TimeSpan.FromSeconds(15),
|
||
heartbeatInterval: TimeSpan.FromSeconds(2),
|
||
failureDetectionThreshold: TimeSpan.FromSeconds(10));
|
||
|
||
// Singleton on the oldest (NodeA) — the continuity probe.
|
||
var manager = cluster.NodeA.ActorOf(ClusterSingletonManager.Props(
|
||
Props.Create(() => new EchoActor()),
|
||
PoisonPill.Instance,
|
||
ClusterSingletonManagerSettings.Create(cluster.NodeA).WithSingletonName("timing-probe")),
|
||
"timing-probe-singleton");
|
||
var proxyA = cluster.NodeA.ActorOf(ClusterSingletonProxy.Props(
|
||
"/user/timing-probe-singleton",
|
||
ClusterSingletonProxySettings.Create(cluster.NodeA).WithSingletonName("timing-probe")),
|
||
"timing-probe-proxy");
|
||
Assert.Equal("ping", await proxyA.Ask<string>("ping", TimeSpan.FromSeconds(30)));
|
||
|
||
var victim = Akka.Cluster.Cluster.Get(cluster.NodeB).SelfAddress;
|
||
var sw = Stopwatch.StartNew();
|
||
await TwoNodeClusterFixture.CrashNode(cluster.NodeB);
|
||
await TwoNodeClusterFixture.WaitForMemberRemoved(cluster.NodeA, victim, TimeSpan.FromSeconds(60));
|
||
sw.Stop();
|
||
|
||
output.WriteLine(
|
||
$"MEASURED: crash -> member removed on survivor: {sw.Elapsed.TotalSeconds:F1}s " +
|
||
"(design ~25s = 10s detection + 15s stable-after; +gossip margin)");
|
||
|
||
// Design envelope: >= stable-after (SBR must not act early), <= 25s design + 15s margin.
|
||
Assert.InRange(sw.Elapsed, TimeSpan.FromSeconds(15), TimeSpan.FromSeconds(40));
|
||
|
||
// Singleton continuity on the surviving oldest node.
|
||
Assert.Equal("ping-after-crash",
|
||
await proxyA.Ask<string>("ping-after-crash", TimeSpan.FromSeconds(10)));
|
||
}
|
||
}
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.PerformanceTests --filter "FullyQualifiedName~FailoverTimingTests"` — expect FAIL (compile error: `TwoNodeClusterFixture` unknown; no `heartbeatInterval` parameter).
|
||
3. Implement:
|
||
- `TwoNodeClusterFixture.StartAsync` (line 24): signature → `StartAsync(string role = "Central", TimeSpan? stableAfter = null, TimeSpan? heartbeatInterval = null, TimeSpan? failureDetectionThreshold = null)`; pass both new args through the two `f.StartNode(...)` calls.
|
||
- `TwoNodeClusterFixture.StartNode` (line 39): signature → `StartNode(int port, string role, TimeSpan? stableAfter = null, TimeSpan? heartbeatInterval = null, TimeSpan? failureDetectionThreshold = null)`; lines 51-52 become `HeartbeatInterval = heartbeatInterval ?? TimeSpan.FromMilliseconds(500),` / `FailureDetectionThreshold = failureDetectionThreshold ?? TimeSpan.FromSeconds(2),`. Extend the class XML doc: "Timing knobs default to scaled test values; pass production values (2s/10s/15s) to measure the real failover envelope (FailoverTimingTests)."
|
||
- `ZB.MOM.WW.ScadaBridge.PerformanceTests.csproj`: add to the existing `<ItemGroup>` of ProjectReferences:
|
||
|
||
```xml
|
||
<!-- Two-node failover rig (TwoNodeClusterFixture) — brings Host + Akka transitively. -->
|
||
<ProjectReference Include="../../tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/ZB.MOM.WW.ScadaBridge.IntegrationTests.csproj" />
|
||
```
|
||
|
||
4. Run again: `dotnet test tests/ZB.MOM.WW.ScadaBridge.PerformanceTests --filter "FullyQualifiedName~FailoverTimingTests"` — expect PASS in ~45-90s; the test output MUST print the `MEASURED:` line — quote the number in the commit message. Regression: `dotnet test tests/ZB.MOM.WW.ScadaBridge.IntegrationTests --filter "FullyQualifiedName~SbrFailoverTests|FullyQualifiedName~ActiveNodeSemanticsTests|FullyQualifiedName~TwoNodeClusterFixtureTests"` — all green (fixture change is additive).
|
||
5. Commit:
|
||
```
|
||
git add tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/TwoNodeClusterFixture.cs tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/ZB.MOM.WW.ScadaBridge.PerformanceTests.csproj tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Failover/FailoverTimingTests.cs
|
||
git commit -m "test(failover): unskip FailoverTimingTests on the two-node rig at production timings — measured Xs vs ~25s design envelope (plan R2-01 T4; covers report-08 NF2)"
|
||
```
|
||
|
||
### Task 5: Apply the wonder-app-vd03 appsettings overlay edits (NodeName + AllowSingleNodeCluster, drop phantom seeds)
|
||
|
||
**Classification:** trivial (config-only; NO git commit — see note)
|
||
**Estimated implement time:** ~4 min
|
||
**Parallelizable with:** Task 1, Task 2, Task 4, Task 6, Task 8, Task 9, Task 10, Task 11
|
||
**Files:**
|
||
- Modify: `deploy/wonder-app-vd03/appsettings.Central.json` (`Node` lines 3-7, `Cluster` lines 8-18, `_comment_Cluster` line 19)
|
||
- Modify: `deploy/wonder-app-vd03/appsettings.Site.json` (`Node` lines 3-9, `Cluster` lines 10-20, `_comment_Cluster` line 21)
|
||
|
||
Round-2 N2: the round-1 deferral claimed the artifact "is not tracked in this repo … the artifact directory itself is not present" — **the directory IS present** (merely gitignored, `.gitignore:48: /deploy/`) and has been actively maintained in this working tree (its `_comment_Security` documents the PLAN-07 edit). Apply the two appsettings edits from PLAN-01 Tasks 16/23 directly. Without `NodeName`, every audit row from the only production install stamps a NULL `SourceNode` (defeats `IX_AuditLog_Node_Occurred` + the per-node stuck KPIs); with the phantom seeds, each node dials a nonexistent peer (`localhost:8091` / `localhost:8092`) forever.
|
||
|
||
1. `appsettings.Central.json`:
|
||
- `Node` (after `"Role": "Central",`): add `"NodeName": "central-a",` (matches the docker convention, `docker/central-node-a/appsettings.Central.json`).
|
||
- `Cluster`: drop the phantom seed `"akka.tcp://scadabridge@localhost:8091"` (keep only the `:8081` self seed) and add `"AllowSingleNodeCluster": true` (flag shipped in PLAN-01 T16: `ClusterOptions.cs:86`, validator `ClusterOptionsValidator.cs:29-35` requires it for a 1-seed list).
|
||
- `_comment_Cluster` → `"Single-node central install, explicitly acknowledged via AllowSingleNodeCluster (arch-review 01 S12/N2): the previous phantom second seed (localhost:8091) made the node dial a nonexistent peer forever. When a real central node B is added: list BOTH real seeds and remove AllowSingleNodeCluster."`
|
||
2. `appsettings.Site.json`: same treatment — `Node` gains `"NodeName": "node-a",`; `Cluster.SeedNodes` keeps only `"akka.tcp://scadabridge@localhost:8082"`; add `"AllowSingleNodeCluster": true`; `_comment_Cluster` rewritten equivalently (PRESERVE the existing "Seed ports must never equal GrpcPort (8083)" sentence).
|
||
3. Verify (this is the "test" — config task):
|
||
```
|
||
python3 -m json.tool deploy/wonder-app-vd03/appsettings.Central.json > /dev/null && \
|
||
python3 -m json.tool deploy/wonder-app-vd03/appsettings.Site.json > /dev/null && \
|
||
grep -c "NodeName\|AllowSingleNodeCluster" deploy/wonder-app-vd03/appsettings.Central.json deploy/wonder-app-vd03/appsettings.Site.json && \
|
||
grep -c "8091\|8092" deploy/wonder-app-vd03/appsettings.Central.json deploy/wonder-app-vd03/appsettings.Site.json
|
||
```
|
||
Expect: both json.tool exits 0; each file counts 2 for the first grep; each file counts **0** for the phantom-port grep (grep -c exits 1 on zero matches — that is the pass signal here).
|
||
4. **No git commit** — `/deploy/` is gitignored (`.gitignore:48`); do NOT use `git add -f`. The edit is intentionally working-tree/on-host only; Task 7 commits the version-controlled record, and the change takes effect on the host when the artifact is next synced + services restarted (RUNBOOK step added in Task 6).
|
||
|
||
### Task 6: Complete the install.ps1 recovery actions (`sc.exe failureflag`) + RUNBOOK recovery step
|
||
|
||
**Classification:** trivial (script/doc-only; NO git commit — see Task 5 note)
|
||
**Estimated implement time:** ~4 min
|
||
**Parallelizable with:** Task 1, Task 2, Task 4, Task 5, Task 8, Task 9, Task 10, Task 11
|
||
**Files:**
|
||
- Modify: `deploy/wonder-app-vd03/install.ps1` (section `# --- 7. Auto-restart on failure`, lines ~171-174)
|
||
- Modify: `deploy/wonder-app-vd03/RUNBOOK.md` (troubleshooting table, ~line 218 area)
|
||
|
||
Finding-vs-disk note (record this honestly): the round-2 report says the `sc.exe failure` recovery actions are missing, but the on-disk `install.ps1` already has them (lines 171-174, all three services) — what is missing is `sc.exe failureflag <svc> 1`. Without the failure-actions flag, the SCM runs recovery actions only after a **crash**; the down-if-alone watchdog's *deliberate* process exit (`IHostApplicationLifetime.StopApplication()` after an SBR self-down — `AkkaHostedService.cs:203-217`) is a non-crash stop and would never be restarted, breaking the recovery contract (`Component-ClusterInfrastructure.md` step 3: "The service supervisor restarts it"). This is the exact second line PLAN-01 Task 20 specified.
|
||
|
||
1. In `install.ps1`, extend the existing loop:
|
||
|
||
```powershell
|
||
# --- 7. Auto-restart on failure ------------------------------------------
|
||
foreach ($svc in 'ScadaBridge-LDAP', 'ScadaBridge-Central', 'ScadaBridge-Site') {
|
||
sc.exe failure $svc reset= 86400 actions= restart/5000/restart/5000/restart/10000 | Out-Null
|
||
# Required for the SBR down-if-alone recovery contract
|
||
# (Component-ClusterInfrastructure.md "Down-if-alone recovery"): without
|
||
# failureflag=1 the SCM applies the recovery actions only after a CRASH;
|
||
# the Host watchdog's deliberate process exit after a self-down is a
|
||
# normal (non-crash) stop and would otherwise never be restarted.
|
||
sc.exe failureflag $svc 1 | Out-Null
|
||
}
|
||
```
|
||
|
||
2. `RUNBOOK.md`: add a troubleshooting-table row: `| Service stopped on its own with log line "ActorSystem terminated outside host shutdown" | SBR self-down (down-if-alone). The service recovery actions restart it automatically (requires install.ps1 with sc.exe failureflag — re-run install.ps1 or run the two sc.exe lines from §7 manually on older installs). If it does not come back: Start-Service manually and check network reachability between nodes. |` Also note next to the "auto-restart on failure" mention (line 87) that failureflag is part of the contract.
|
||
3. Verify: `grep -n "failureflag" deploy/wonder-app-vd03/install.ps1` (3 effective invocations via the loop → 1 hit inside it) and `grep -n "failureflag\|self-down" deploy/wonder-app-vd03/RUNBOOK.md` (≥1 hit). If `pwsh` is installed, syntax-check: `pwsh -NoProfile -Command "[void][System.Management.Automation.PSParser]::Tokenize((Get-Content -Raw deploy/wonder-app-vd03/install.ps1), [ref]$null)"` (exit 0); otherwise the greps suffice.
|
||
4. **No git commit** (gitignored artifact — same rule as Task 5).
|
||
|
||
### Task 7: Correct the factually wrong N2 deferral record in the master tracker
|
||
|
||
**Classification:** trivial (doc-only)
|
||
**Estimated implement time:** ~3 min
|
||
**Parallelizable with:** Task 1, Task 2, Task 4, Task 8, Task 9, Task 10, Task 11
|
||
**Files:**
|
||
- Modify: `archreview/plans/00-MASTER-TRACKER.md` (line 28 trailing sentence; "Deferred during Wave 1 PLAN-01 execution (2026-07-08)" bullet at lines 145-147)
|
||
|
||
The deferral rationale — "that production deploy artifact is not tracked in this repo … the artifact directory itself is not present in the repo" — is factually wrong (round-2 N2): `deploy/wonder-app-vd03/` exists in the working tree, is merely gitignored (`.gitignore:48: /deploy/`), and was edited during this initiative (PLAN-07's `_comment_Security`). Depends on Tasks 5-6 having applied the edits.
|
||
|
||
1. Rewrite the line-147 bullet: keep the history (what was deferred and why it mattered), then correct the record: the directory **is present but gitignored** (production config kept out of version control — the *tracking* claim was wrong, the *presence* claim was false); the three overlay edits were **applied on-disk 2026-07-12 by PLAN-R2-01 Tasks 5-6** (`NodeName` central-a/node-a, `AllowSingleNodeCluster` + phantom seeds dropped, `sc.exe failureflag` completing the already-present `sc.exe failure` actions). Remaining owner action: sync the artifact to `wonder-app-vd03` and restart services (appsettings take effect on restart; failureflag on re-running install.ps1 §7 or the two `sc.exe` lines manually) — per the RUNBOOK row added by T6.
|
||
2. Trim the stale trailing sentence of the line-28 Wave-1 summary ("were deferred because that production artifact is not tracked in this repo") → "were deferred on a mistaken not-in-repo rationale; corrected and applied on-disk by PLAN-R2-01 (2026-07-12) — see the Deferred-during-Wave-1 entry."
|
||
3. Do NOT add rows to `docs/plans/2026-07-08-deferred-work-register.md` — register consolidation (round-2 N6) is owned by **PLAN-R2-08**; this task only stops the tracker from asserting a falsehood.
|
||
4. Verify: `grep -n "not tracked in this repo" archreview/plans/00-MASTER-TRACKER.md` → 0 hits; `grep -n "PLAN-R2-01" archreview/plans/00-MASTER-TRACKER.md` → ≥2 hits.
|
||
5. Commit:
|
||
```
|
||
git add archreview/plans/00-MASTER-TRACKER.md
|
||
git commit -m "docs(tracker): correct the wonder-app-vd03 deferral record — artifact is present (gitignored); overlay edits applied on-disk by R2-01 T5/T6 (plan R2-01 T7)"
|
||
```
|
||
|
||
### Task 8: Metrics-staleness for never-reported sites (`FirstSeenAt` anchor)
|
||
|
||
**Classification:** high-risk (lock-free CAS state machine)
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** Task 1, Task 2, Task 4, Task 5, Task 6, Task 7, Task 9, Task 10, Task 11
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthState.cs` (add `FirstSeenAt`)
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthAggregator.cs` (register branches lines 44-54 and 114-122; staleness guard lines 300-312)
|
||
- Modify: `docs/requirements/Component-HealthMonitoring.md` ("Metrics staleness" bullet, line ~47)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/CentralHealthAggregatorTests.cs`
|
||
|
||
Round-2 N3: `CheckForOfflineSites` guards staleness with `state.LastReportReceivedAt is { } lastReport` (`CentralHealthAggregator.cs:300`), so a site registered via heartbeat only (`MarkHeartbeat` registration leaves `LastReportReceivedAt` null, `:114-122`) is NEVER flagged stale — "stuck from first boot" shows online-with-no-metrics forever. `LastHeartbeatAt` cannot be the anchor (heartbeats keep advancing it); anchor on a new `FirstSeenAt` stamped at registration. No UI change needed: the badge (`Health.razor:209-215`) keys on `IsOnline && IsMetricsStale` regardless of `LatestReport`.
|
||
|
||
1. Write the failing tests (append to `CentralHealthAggregatorTests` — reuse the existing `NewAggregator(metricsStaleTimeout:)` and `TestTimeProvider.Advance` helpers, lines 48-59/22):
|
||
|
||
```csharp
|
||
/// <summary>
|
||
/// Round-2 N3: a site known only via heartbeats (LastReportReceivedAt == null)
|
||
/// previously skipped the staleness check forever — a report pipeline dead
|
||
/// from FIRST BOOT showed "online with no metrics" indefinitely. FirstSeenAt
|
||
/// anchors the window instead (LastHeartbeatAt cannot: heartbeats advance it).
|
||
/// </summary>
|
||
[Fact]
|
||
public void HeartbeatOnlySite_NeverReported_IsFlaggedMetricsStale()
|
||
{
|
||
var (aggregator, time) = NewAggregator(metricsStaleTimeout: TimeSpan.FromMinutes(2));
|
||
aggregator.MarkHeartbeat("site-a", time.GetUtcNow()); // registered via heartbeat only
|
||
time.Advance(TimeSpan.FromMinutes(3));
|
||
aggregator.MarkHeartbeat("site-a", time.GetUtcNow()); // still heartbeating => online
|
||
aggregator.CheckForOfflineSites();
|
||
var state = aggregator.GetSiteState("site-a")!;
|
||
Assert.True(state.IsOnline); // liveness unchanged
|
||
Assert.True(state.IsMetricsStale); // was impossible before the fix
|
||
}
|
||
|
||
/// <summary>A heartbeat-only site inside the window is NOT falsely flagged.</summary>
|
||
[Fact]
|
||
public void HeartbeatOnlySite_WithinStaleWindow_IsNotFlagged()
|
||
{
|
||
var (aggregator, time) = NewAggregator(metricsStaleTimeout: TimeSpan.FromMinutes(2));
|
||
aggregator.MarkHeartbeat("site-a", time.GetUtcNow());
|
||
time.Advance(TimeSpan.FromMinutes(1));
|
||
aggregator.MarkHeartbeat("site-a", time.GetUtcNow());
|
||
aggregator.CheckForOfflineSites();
|
||
Assert.False(aggregator.GetSiteState("site-a")!.IsMetricsStale);
|
||
}
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests --filter "FullyQualifiedName~HeartbeatOnlySite"` — expect the first test to FAIL (`IsMetricsStale` stays false; the null guard skips the site).
|
||
3. Implement:
|
||
- `SiteHealthState`: add
|
||
|
||
```csharp
|
||
/// <summary>
|
||
/// Gets the instant this site was first observed by the aggregator (via a
|
||
/// report or a heartbeat); <c>null</c> only for states not built by the
|
||
/// aggregator (hand-constructed fixtures). Anchors metrics-staleness for a
|
||
/// site that has NEVER delivered a report (review 01 round-2 N3): with
|
||
/// <see cref="LastReportReceivedAt"/> null the staleness check previously
|
||
/// skipped the site forever. <see cref="LastHeartbeatAt"/> cannot anchor
|
||
/// this — every heartbeat advances it.
|
||
/// </summary>
|
||
public DateTimeOffset? FirstSeenAt { get; init; }
|
||
```
|
||
|
||
- `CentralHealthAggregator.ProcessReport` register branch (lines 44-54): add `FirstSeenAt = now,`. `MarkHeartbeat` register branch (lines 114-122): add `FirstSeenAt = receivedAt,`. (Update branches never touch it — `with` copies preserve it.)
|
||
- `CheckForOfflineSites` staleness guard (lines 300-312): replace the `state.LastReportReceivedAt is { } lastReport` condition with an anchor fallback:
|
||
|
||
```csharp
|
||
// Second signal: … (keep the existing comment, append:)
|
||
// A site that has NEVER reported (heartbeat-only registration,
|
||
// LastReportReceivedAt == null) anchors on FirstSeenAt instead —
|
||
// otherwise a pipeline dead from first boot is never flagged (N3).
|
||
var reportAnchor = state.LastReportReceivedAt ?? state.FirstSeenAt;
|
||
if (reportAnchor is { } anchor
|
||
&& now - anchor > _options.MetricsStaleTimeout
|
||
&& !state.IsMetricsStale)
|
||
{
|
||
var stale = state with { IsMetricsStale = true };
|
||
if (_siteStates.TryUpdate(kvp.Key, stale, state))
|
||
{
|
||
_logger.LogWarning(
|
||
"Site {SiteId} metrics are stale — online (heartbeats) but no report for {Elapsed}s{NeverNote} (timeout: {Timeout}s)",
|
||
state.SiteId, (now - anchor).TotalSeconds,
|
||
state.LastReportReceivedAt is null ? " (never reported since first contact)" : string.Empty,
|
||
_options.MetricsStaleTimeout.TotalSeconds);
|
||
}
|
||
// CAS loss ⇒ a fresh report/heartbeat swapped in ⇒ correct to skip.
|
||
}
|
||
```
|
||
|
||
- `Component-HealthMonitoring.md` "Metrics staleness" bullet (line ~47): append — "A site that has **never** delivered a report is anchored on its first-contact time (`FirstSeenAt`), so a report pipeline dead from first boot is also flagged after `MetricsStaleTimeout` (previously such a site showed online-with-no-metrics indefinitely)."
|
||
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests` — full project green (the CAS/flap/stale suites are the regression net).
|
||
5. Commit:
|
||
```
|
||
git add src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthState.cs src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthAggregator.cs docs/requirements/Component-HealthMonitoring.md tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/CentralHealthAggregatorTests.cs
|
||
git commit -m "fix(health): flag metrics-stale for never-reported sites via FirstSeenAt anchor — null LastReportReceivedAt no longer skips the check (plan R2-01 T8)"
|
||
```
|
||
|
||
### Task 9: Purge the stale "cluster leader" narration from `CentralHealthReportLoop`
|
||
|
||
**Classification:** trivial (comment-only)
|
||
**Estimated implement time:** ~2 min
|
||
**Parallelizable with:** Task 1, Task 2, Task 4, Task 5, Task 6, Task 7, Task 8, Task 10, Task 11
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthReportLoop.cs` (class doc line 12-14; sequence-seed comment lines 44-45)
|
||
|
||
Round-2 N4: the behavior is correct (line 93 gates on `IClusterNodeProvider.SelfIsPrimary` = oldest Up member since PLAN-01 T6), but the comments still narrate "cluster leader (Primary)" — re-teaching the exact leader/oldest conflation round-1 S2 eradicated.
|
||
|
||
1. Confirm scope first: `grep -in "leader" src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthReportLoop.cs` — expect exactly lines 12, 44, 45.
|
||
2. Edit:
|
||
- Class doc (lines 12-14): "Only the cluster leader (Primary) generates reports — the standby's aggregator catches up on failover when it becomes Primary and starts its own loop." → "Only the active node (Primary = the **oldest Up member**, i.e. the singleton host — `IClusterNodeProvider.SelfIsPrimary`, never Akka cluster leadership) generates reports; the standby's aggregator catches up on failover when it becomes the oldest and its loop starts reporting."
|
||
- Sequence-seed comment (lines 44-45): "reports from a newly-elected central leader always sort after reports from any prior leader" → "reports from a node that has newly become active (oldest Up member) always sort after reports from any previously-active node".
|
||
3. Verify: `grep -in "leader" src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthReportLoop.cs` — **0 hits**. Comment-only change: `dotnet build src/ZB.MOM.WW.ScadaBridge.HealthMonitoring` — succeeds.
|
||
4. Commit:
|
||
```
|
||
git add src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthReportLoop.cs
|
||
git commit -m "docs(health): CentralHealthReportLoop comments say oldest-member, not cluster leader — stop re-teaching the S2 conflation (plan R2-01 T9)"
|
||
```
|
||
|
||
### Task 10: Generalize the registrar — `SingletonRegistrar` with an optional role scope
|
||
|
||
**Classification:** high-risk (actor lifecycle helper used by all central singletons)
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** Task 1, Task 2, Task 4, Task 5, Task 6, Task 7, Task 8, Task 9
|
||
**Files:**
|
||
- Modify (rename): `src/ZB.MOM.WW.ScadaBridge.Host/Actors/CentralSingletonRegistrar.cs` → `src/ZB.MOM.WW.ScadaBridge.Host/Actors/SingletonRegistrar.cs` (use `git mv`)
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs` (rename at the block comment lines 459-465 and the 7 central call sites: lines 466, 499, 564, 609, 632, 663, 689)
|
||
- Modify (rename): `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CentralSingletonRegistrarTests.cs` → `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SingletonRegistrarTests.cs`
|
||
- Modify: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CoordinatedShutdownTests.cs` (line 56 source-grep string)
|
||
|
||
Round-2 N5 first half: `CentralSingletonRegistrar` cannot express `.WithRole(siteRole)`, which is the sole reason the two site singletons stay hand-rolled without drains (`AkkaHostedService.cs:464-465`). Add an optional `role` applied to BOTH the manager and proxy settings, and rename the class — "Central" becomes a lie the moment it hosts site singletons.
|
||
|
||
1. Write the failing test — in the (renamed) `SingletonRegistrarTests.cs`, rename the class + existing references `CentralSingletonRegistrar` → `SingletonRegistrar`, and add:
|
||
|
||
```csharp
|
||
[Fact]
|
||
public async Task Start_WithRoleScope_CreatesRoleScopedSingleton_ThatAnswers()
|
||
{
|
||
// Round-2 N5: role scoping is what kept the two SITE singletons
|
||
// (deployment-manager, event-log-handler) hand-rolled and drain-less.
|
||
using var system = CreateSingleNodeClusterSystem(); // roles = ["Central"]
|
||
var handle = SingletonRegistrar.Start(
|
||
system, "role-widget", Props.Create(() => new EchoActor()),
|
||
NullLogger.Instance, drainTimeout: TimeSpan.FromSeconds(2), role: "Central");
|
||
|
||
Assert.EndsWith("/user/role-widget-singleton", handle.Manager.Path.ToString());
|
||
Assert.EndsWith("/user/role-widget-proxy", handle.Proxy.Path.ToString());
|
||
|
||
// The node holds the role => manager instantiates the singleton and
|
||
// the role-scoped proxy resolves it.
|
||
var echo = await handle.Proxy.Ask<string>("hi", TimeSpan.FromSeconds(20));
|
||
Assert.Equal("hi", echo);
|
||
}
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~SingletonRegistrarTests"` — expect FAIL (compile: no `SingletonRegistrar` type / no `role` parameter).
|
||
3. Implement:
|
||
- `git mv src/ZB.MOM.WW.ScadaBridge.Host/Actors/CentralSingletonRegistrar.cs src/ZB.MOM.WW.ScadaBridge.Host/Actors/SingletonRegistrar.cs`; rename the class; signature → `Start(ActorSystem system, string name, Props singletonProps, ILogger logger, TimeSpan? drainTimeout = null, string? role = null)`; apply the role to both settings:
|
||
|
||
```csharp
|
||
var managerSettings = ClusterSingletonManagerSettings.Create(system).WithSingletonName(name);
|
||
if (role is not null) managerSettings = managerSettings.WithRole(role);
|
||
// … ClusterSingletonManager.Props(singletonProps, PoisonPill.Instance, managerSettings) …
|
||
|
||
var proxySettings = ClusterSingletonProxySettings.Create(system).WithSingletonName(name);
|
||
if (role is not null) proxySettings = proxySettings.WithRole(role);
|
||
// … ClusterSingletonProxy.Props($"/user/{name}-singleton", proxySettings) …
|
||
```
|
||
|
||
Update the class XML doc: drop "central"-only framing; drain rationale now reads "so in-flight EF (central) or SQLite (site) work completes before handover"; note the optional `role` maps to `ClusterSingletonManagerSettings/ProxySettings.WithRole` (review 01 round-2 N5).
|
||
- `AkkaHostedService.cs`: mechanical rename at the 7 call sites (466, 499, 564, 609, 632, 663, 689) and rewrite the lines-459-465 comment's last sentence ("The two SITE singletons stay hand-rolled below because they are role-scoped") → "Site singletons use the same registrar with `role: siteRole` (Task 11 / round-2 N5)." (Task 11 lands that; wording here is forward-referencing but true within the same plan.)
|
||
- `git mv tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CentralSingletonRegistrarTests.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SingletonRegistrarTests.cs`.
|
||
- `CoordinatedShutdownTests.cs` line 56: `Assert.Contains("CentralSingletonRegistrar.Start(", content);` → `Assert.Contains("SingletonRegistrar.Start(", content);` (also update the comment above it).
|
||
- Sanity: `grep -rn "CentralSingletonRegistrar" src/ tests/ --include="*.cs"` — 0 hits.
|
||
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~SingletonRegistrarTests|FullyQualifiedName~CoordinatedShutdownTests"` — PASS. Then `dotnet build ZB.MOM.WW.ScadaBridge.slnx` — clean.
|
||
5. Commit:
|
||
```
|
||
git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/SingletonRegistrar.cs src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SingletonRegistrarTests.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CoordinatedShutdownTests.cs
|
||
git rm --cached src/ZB.MOM.WW.ScadaBridge.Host/Actors/CentralSingletonRegistrar.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CentralSingletonRegistrarTests.cs 2>/dev/null || true
|
||
git commit -m "refactor(host): CentralSingletonRegistrar -> SingletonRegistrar with optional role scope — unblocks site-singleton drains (plan R2-01 T10)"
|
||
```
|
||
(`git mv` already stages the renames; the `git rm --cached … || true` line is a no-op safety net if the moves were done as delete+create.)
|
||
|
||
### Task 11: Route the two site singletons through the registrar — deployment-manager + event-log-handler gain drains
|
||
|
||
**Classification:** high-risk (site actor wiring refactor)
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** Task 1, Task 2, Task 4, Task 5, Task 6, Task 7, Task 8, Task 9
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs` (site branch: deployment-manager block lines ~791-820; event-log-handler block lines ~836-856)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CoordinatedShutdownTests.cs`
|
||
|
||
Round-2 N5 second half: `deployment-manager` and `event-log-handler` are hand-rolled with bare `PoisonPill` termination and no `PhaseClusterLeave` drain. The round-1 S5 rationale applies identically on the site side: the DeploymentManager's SQLite writes (static overrides, `native_alarm_state`) should drain before handover — currently safe-but-lossy on graceful failover.
|
||
|
||
1. Write the failing test — in `CoordinatedShutdownTests.cs`, update the existing `AllCentralSingletons_RegisterThroughRegistrarWithDrain` (lines 45-70): change the final assertion `Assert.Equal(2, CountOccurrences(content, "ClusterSingletonManager.Props("));` → `Assert.Equal(0, …)` and replace its "the only two left are the SITE singletons" comment; then add:
|
||
|
||
```csharp
|
||
[Fact]
|
||
public void SiteSingletons_RegisterThroughRegistrarWithDrain()
|
||
{
|
||
var hostProjectDir = FindHostProjectDirectory();
|
||
Assert.NotNull(hostProjectDir);
|
||
var content = File.ReadAllText(Path.Combine(hostProjectDir!, "Actors", "AkkaHostedService.cs"));
|
||
|
||
// Round-2 N5: the two role-scoped SITE singletons previously stayed
|
||
// hand-rolled (bare PoisonPill, no PhaseClusterLeave drain). They now
|
||
// go through SingletonRegistrar.Start(..., role: siteRole), which
|
||
// always adds the GracefulStop drain — in-flight SQLite writes
|
||
// (static overrides, native_alarm_state) complete before handover.
|
||
Assert.Contains("\"deployment-manager\"", content);
|
||
Assert.Contains("\"event-log-handler\"", content);
|
||
Assert.Contains("role: siteRole", content);
|
||
Assert.Equal(0, CountOccurrences(content, "ClusterSingletonManager.Props("));
|
||
}
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~SiteSingletons_RegisterThroughRegistrarWithDrain"` — expect FAIL (`ClusterSingletonManager.Props(` count is 2; no `role: siteRole`).
|
||
3. Implement in the site branch of `AkkaHostedService.cs`:
|
||
- Deployment-manager block (~791-820): replace the hand-rolled `ClusterSingletonManager.Props`/`ClusterSingletonProxy.Props` pair with
|
||
|
||
```csharp
|
||
// Deployment Manager — role-scoped singleton via SingletonRegistrar
|
||
// (review 01 round-2 N5): previously hand-rolled with bare PoisonPill
|
||
// termination and NO PhaseClusterLeave drain, so in-flight SQLite
|
||
// writes (static overrides, native_alarm_state) could be cut off on
|
||
// graceful failover. Names unchanged: deployment-manager-singleton /
|
||
// deployment-manager-proxy.
|
||
var dm = SingletonRegistrar.Start(
|
||
_actorSystem!, "deployment-manager",
|
||
Props.Create(() => new DeploymentManagerActor(
|
||
storage, compilationService, sharedScriptLibrary, streamManager,
|
||
siteRuntimeOptionsValue, dmLogger, dclManager, replicationActor,
|
||
siteHealthCollector, _serviceProvider, null, deploymentConfigFetcher)),
|
||
_logger, role: siteRole);
|
||
var dmProxy = dm.Proxy;
|
||
```
|
||
|
||
(Keep the `dmProxy` variable name — everything downstream, `SiteCommunicationActor` construction and `RegisterLocalHandler(LocalHandlerType.Artifacts, dmProxy)`, is untouched.)
|
||
- Event-log-handler block (~836-856): inside the existing `if (eventLogQueryService != null)`:
|
||
|
||
```csharp
|
||
var eventLog = SingletonRegistrar.Start(
|
||
_actorSystem, "event-log-handler",
|
||
Props.Create(() => new SiteEventLogging.EventLogHandlerActor(eventLogQueryService)),
|
||
_logger, role: siteRole);
|
||
siteCommActor.Tell(new RegisterLocalHandler(LocalHandlerType.EventLog, eventLog.Proxy));
|
||
```
|
||
|
||
(Keep the existing "Event log handler — cluster singleton so queries always reach the active node…" comment above it.)
|
||
- Actor/singleton names are reproduced exactly by the registrar (`deployment-manager-singleton`/`-proxy`, `event-log-handler-singleton`/`-proxy`), so path-pinning tests and the ClusterClient receptionist wiring see no change.
|
||
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests` (full project — path/health-check/audit-wiring suites are the regression net) and `dotnet build ZB.MOM.WW.ScadaBridge.slnx` — PASS/clean.
|
||
5. Commit:
|
||
```
|
||
git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CoordinatedShutdownTests.cs
|
||
git commit -m "fix(host): site singletons (deployment-manager, event-log-handler) via SingletonRegistrar — PhaseClusterLeave drains for site SQLite writes (plan R2-01 T11)"
|
||
```
|
||
|
||
---
|
||
|
||
## Dependencies on other plans
|
||
|
||
- **PLAN-R2-02 owns** the oldest-vs-leader predicate unification inside `SiteCommunicationActor`/`SiteReplicationActor` and everything resync-related. This plan does not touch those files; the coverage table references it.
|
||
- **PLAN-R2-08 owns** N6 (folding the two live PLAN-01 deferrals — the keep-oldest active-crash gap and, post-T5/T6, the "sync the artifact to the host" residue — into `docs/plans/2026-07-08-deferred-work-register.md`). Task 7 here only corrects the factually wrong tracker prose; it adds no register rows.
|
||
- The **SBR keep-oldest active-crash availability gap stays a registered deferred USER decision** — no task here changes the strategy/topology; Tasks 1-3 make the gap observable and measured, which is the input that decision needs.
|
||
|
||
## Execution order
|
||
|
||
**Drill spine (serial):** Task 1 (script) → Task 2 (docs) → Task 3 (live run, needs the docker cluster exclusively).
|
||
|
||
**Registrar spine (serial):** Task 10 → Task 11.
|
||
|
||
**Freely parallel at any point:** Tasks 4, 5, 6, 8, 9 (all file-disjoint); Task 7 after 5+6.
|
||
|
||
**Highest value if time-boxed:** Task 3 (the never-run drill, run at last) + Task 11 (real durability gap) + Task 8 (real monitoring blind spot).
|
||
|
||
---
|
||
|
||
## Findings Coverage
|
||
|
||
Every round-2 finding (N1-N6), the round-1 partials the report says warrant completion, and the cross-plan rulings:
|
||
|
||
| # | Round-2 finding | Severity | Task(s) / Disposition |
|
||
|---|----------------|----------|----------------------|
|
||
| N1 | Failover drill kills the wrong (active/oldest) node; first-seed bootstrap gap means the survivor's recovery story doesn't converge; drill never executed; ~25s promise undeliverable in its most likely case | High | Tasks 1 (drill rewrite: standby-victim default + explicit active-gap mode), 2 (narrative + first-seed constraint **documented** with operator actions — deliberately NOT reordered to self-first seeds: simultaneous cold-start self-joins would risk two independent clusters), 3 (RUN the drill live, both directions, record measured timings), 4 (wire/unskip `FailoverTimingTests` to `TwoNodeClusterFixture`, measure the ~25s envelope at production timings — **also covers report-08 NF2 by reference**). The keep-oldest topology/strategy remedy itself **stays deferred: registered USER decision** (master tracker 2026-07-08) — Tasks 1-3 document/measure it, never fix it |
|
||
| N2 | wonder-app-vd03 overlay edits unapplied; deferral rationale factually wrong (artifact IS in the working tree, merely gitignored) | Medium | Tasks 5 (NodeName + AllowSingleNodeCluster + drop phantom seeds, both overlays), 6 (`sc.exe failureflag` completing the recovery actions + RUNBOOK step — note: `sc.exe failure` itself is already on disk, contra the report), 7 (correct the deferral record in the master tracker). No git commits for the gitignored artifact itself |
|
||
| N3 | `IsMetricsStale` can never fire for a site that has never delivered a report (null `LastReportReceivedAt` guard) | Low | Task 8 (`FirstSeenAt` anchor; no UI change needed — badge already keys on the flag) |
|
||
| N4 | `CentralHealthReportLoop` still narrates "cluster leader (Primary)" | Low | Task 9 |
|
||
| N5 | Site-role singletons lack drain tasks; registrar can't express role scoping | Low | Tasks 10 (role param + rename to `SingletonRegistrar`), 11 (route `deployment-manager` + `event-log-handler` through it) |
|
||
| N6 | PLAN-01's two live deferrals missing from the canonical deferred-work register | Low | **Reference → PLAN-R2-08** (register consolidation is its scope). Task 7 here only fixes the falsehood in the master-tracker prose |
|
||
| S12 (R1 partial) | Single-node artifact still ships the phantom seed, lacks `AllowSingleNodeCluster` | — | Task 5 (the artifact half; the flag/validator half shipped in PLAN-01 T16) |
|
||
| U2 (R1 partial) | install.ps1 recovery actions deferred; recovery contract doesn't converge for a non-first-seed survivor | — | Task 6 (failureflag) + Task 2 (first-seed constraint + operator actions in the spec'd recovery contract) |
|
||
| U5 (R1 not fixed) | `NodeName` missing from wonder-app-vd03 overlays → NULL `SourceNode` | — | Task 5 |
|
||
| C3 (R1 partial) | LDAP `Transport: None, AllowInsecure: true` remains in the deploy artifact | — | **Stays accepted posture per the report** — bundled loopback GLAuth with documented AD-repoint instructions; a future over-the-wire AD repoint must revisit transport. No task |
|
||
| — | Oldest-vs-leader predicate unification in `SiteCommunicationActor`/`SiteReplicationActor` + resync work | — | **Reference → PLAN-R2-02** (explicit cross-plan ownership; not touched here) |
|