docs(archreview): PLAN-01 complete (23/23) — sync tracker + tasks.json; log deploy-artifact deferral

docs(cleanup): sync Traefik + Host docs to oldest-member active-node semantics + DB-gated /health/active; TLS roadmap note
Renames stale ActiveNodeHealthCheck references to OldestNodeActiveHealthCheck (Task 7 rename) across the requirements + components doc sets, corrects leader→oldest-member wording and the embedded IsActiveNode snippet, and adds an explicit 'production TLS profile not yet implemented' roadmap note. deploy/wonder-app-vd03 NodeName overlay edits deferred: that production deploy artifact is not tracked in this repo (same as Tasks 16 & 20).
2026-07-08 17:12:46 -04:00 · 2026-07-08 17:09:34 -04:00 · 2026-07-08 17:05:31 -04:00 · 2026-07-08 16:59:19 -04:00 · 2026-07-08 16:56:24 -04:00 · 2026-07-08 16:52:25 -04:00
61 changed files with 1835 additions and 366 deletions
@@ -7,11 +7,8 @@
  <ItemGroup>
    <PackageVersion Include="Akka" Version="1.5.62" />
    <PackageVersion Include="Akka.Cluster" Version="1.5.62" />
-    <PackageVersion Include="Akka.Cluster.Hosting" Version="1.5.62" />
    <PackageVersion Include="Akka.Cluster.Tools" Version="1.5.62" />
-    <PackageVersion Include="Akka.Hosting" Version="1.5.62" />
    <PackageVersion Include="Akka.Remote" Version="1.5.62" />
-    <PackageVersion Include="Akka.Remote.Hosting" Version="1.5.62" />
    <PackageVersion Include="Akka.Streams" Version="1.5.62" />
    <PackageVersion Include="Akka.Streams.TestKit" Version="1.5.62" />
    <PackageVersion Include="Akka.TestKit.Xunit2" Version="1.5.62" />
@@ -10,7 +10,7 @@ Every plan follows the TDD bite-sized-task format and ships a co-located `.tasks

 | Plan | Domain | Tasks | Done | Status | Findings coverage |
 |------|--------|------:|-----:|--------|-------------------|
-| [PLAN-01](PLAN-01-cluster-host-failover.md) | Cluster, Host & Failover | 23 | 3 | 🟨 In progress (P0: T1, T3, T4) | 28 findings: 20 fixed, 3 → other plans, 3 accepted, 2 roadmap-deferred |
+| [PLAN-01](PLAN-01-cluster-host-failover.md) | Cluster, Host & Failover | 23 | 23 | ✅ Complete (all 23 tasks; branch not yet merged) | 28 findings: 20 fixed, 3 → other plans, 3 accepted, 2 roadmap-deferred |
 | [PLAN-02](PLAN-02-communication-store-and-forward.md) | Communication & S&F | 24 | 2 | 🟨 In progress (P0: T1, T2) | 28 findings: 25 fixed, 1 → plan 01, 1 deferred (serializer swap), 1 won't-fix |
 | [PLAN-03](PLAN-03-site-runtime-dcl.md) | Site Runtime & DCL | 26 | 0 | ⬜ Not started | 28 actionable: 25 fixed, 3 deferred with rationale |
 | [PLAN-04](PLAN-04-data-audit-backbone.md) | Data & Audit Backbone | 26 | 0 | ⬜ Not started | 27 items: 22 fixed, 4 deferred, 1 won't-fix (incl. NodeB reconcile from report 08) |
@@ -23,6 +23,8 @@ Status values: ⬜ Not started · 🟨 In progress · ✅ Complete · ⏸ Blocke

 **Progress:** P0 (Wave 0) complete — **10 of 192 tasks done** (2026-07-08): PLAN-01 T1/T3/T4, PLAN-02 T1/T2, PLAN-05 T1, PLAN-07 T1/T2/T3, PLAN-08 T1. Merged to `main` (`3f8e18c6`). One follow-up surfaced during execution — see the [active-node-crash SBR gap](#follow-up-discovered-during-p0-execution-2026-07-08) in the deferred registry.

+**Wave 1 — PLAN-01 COMPLETE (2026-07-08):** all 23 tasks delivered on worktree branch `worktree-archreview-wave1-plan01` (**NOT yet merged/pushed**). Beyond the P0/early-Wave-1 spine (SBR downing + two-node rig T1/T3/T4; oldest-member active-node unification via `ClusterActivityEvaluator` T5–T8; `CentralSingletonRegistrar` drains T2/T9/T10; health-report acking T11–T12), Wave 1 also landed: metrics-staleness + status-transition state machine and its dashboard surface (T13/T14), deleted-site eviction from the aggregator (T18), seed-vs-metrics-port validator + `AllowSingleNodeCluster` + docker `stop_grace_period` (T15–T17), rate-limited dead-letter warnings (T19), the down-if-alone process-exit recovery loop (T20), the `failover-drill.sh` SIGKILL drill (T21), and the REQ-HOST-6 + Traefik/Host doc reconciliation with unused-Akka.Hosting-package removal (T22/T23). PLAN-01 now **23 / 23**. Initiative total: **30 of 192**. Each task TDD + pathspec-committed; full solution builds clean (0 warnings). Three deploy-artifact sub-edits (`deploy/wonder-app-vd03/` appsettings + install.ps1, Tasks 16/20/23) were deferred because that production artifact is not tracked in this repo — logged in the deferred register below.
+
 ## P0 — Do These First (regardless of plan order)

 **✅ All six delivered 2026-07-08** on branch `archreview-p0` (worktree). TDD throughout; full solution builds clean; touched suites green (Host, Communication 244, Security 163, Transport 127+63, IntegrationTests cluster rig). One finding surfaced — see the active-node-crash SBR gap in the deferred registry below.
@@ -88,3 +90,7 @@ Consolidated in **PLAN-08 Task 11** → `docs/plans/2026-07-08-deferred-work-reg
 ### Follow-up discovered during P0 execution (2026-07-08)

 - **Active(oldest)-node hard crash is NOT covered by two-node `keep-oldest` — total central outage until restart.** PLAN-01 Task 1's `downing-provider-class` fix is verified active (a hard-crashed node now transitions `Unreachable → Down → Removed`, impossible under the pre-fix NoDowning default). **But** while building PLAN-01 Task 4's behavioral proof, the live two-node rig showed (Akka 1.5.62) that `keep-oldest` downs the partition *without* the oldest: crashing the **younger** node is survivable (oldest keeps its singletons), but crashing the **oldest** node makes the younger survivor **down itself** (`Down` ~8s, terminated ~10s) → the whole central cluster is lost until the crashed node restarts. `down-if-alone = on` does not help on a hard crash (the alone-oldest is dead and cannot down itself). Because ScadaBridge treats **oldest = active node** (PLAN-01 Task 5 `ClusterActivityEvaluator`), this means a crash of the *active* central node does not fail over to the standby — the scenario operators care most about. Task 4's test was rewritten to prove the achievable guarantee (younger-crash → SBR downs it → oldest keeps singleton) and documents this asymmetry in its XML doc. **Remedy is a topology/strategy decision (deferred, owner: user):** e.g. `keep-majority` + a 3rd/lighthouse seed node, `static-quorum`, or an explicit accepted-risk note. Not fixed in P0 to avoid silently changing the production SBR strategy.
+
+### Deferred during Wave 1 PLAN-01 execution (2026-07-08)
+
+- **`deploy/wonder-app-vd03/` overlay edits not applied — that production deploy artifact is not tracked in this repo.** PLAN-01 Tasks 16, 20, and 23 each specified a small edit to the on-host deployment overlay (`appsettings.Central.json`: add `AllowSingleNodeCluster: true` + drop the phantom seed, add `NodeName: central-a`; `install.ps1`: add `sc.exe failure` service-recovery actions for the down-if-alone restart contract). Only the *deployment-record* doc (`docs/deployment/deployment-records/2026-06-27-wonder-app-vd03.md`) is version-controlled; the `deploy/wonder-app-vd03/` artifact directory itself is not present in the repo (production config kept out of source control). The **code/validator/doc** halves of all three tasks shipped and are what make the features usable (`AllowSingleNodeCluster` flag + validator, the `IHostApplicationLifetime` process-exit watchdog, the oldest-member Traefik/Host doc reconciliation + TLS roadmap note). **Owner: whoever maintains the on-host artifact** — apply the three overlay edits out-of-band on `wonder-app-vd03` (see the referenced task steps); without `NodeName` every audit row from that deployment stamps a NULL `SourceNode`, and without the service-recovery actions a self-downed node will not auto-restart.
@@ -1,29 +1,177 @@
 {
  "planPath": "archreview/plans/PLAN-01-cluster-host-failover.md",
  "tasks": [
-    { "id": 1, "subject": "Task 1: Enable the SBR downing provider in BuildHocon", "status": "completed", "blockedBy": [] },
-    { "id": 2, "subject": "Task 2: Raise the cluster-leave CoordinatedShutdown phase timeout above the drain budget", "status": "pending", "blockedBy": [1] },
-    { "id": 3, "subject": "Task 3: Two-node in-process cluster fixture (the reusable failover rig)", "status": "completed", "blockedBy": [1] },
-    { "id": 4, "subject": "Task 4: SBR behavioral proof — DELIVERED as younger-node-crash (SBR downs it, oldest keeps singleton); the planned oldest-crash 'singleton migration' is impossible under 2-node keep-oldest — see master-tracker follow-up", "status": "completed", "blockedBy": [3] },
-    { "id": 5, "subject": "Task 5: ClusterActivityEvaluator — single oldest-member \"active node\" definition", "status": "pending", "blockedBy": [] },
-    { "id": 6, "subject": "Task 6: Rewire ActiveNodeGate, AkkaClusterNodeProvider and Primary labels onto the evaluator", "status": "pending", "blockedBy": [5] },
-    { "id": 7, "subject": "Task 7: Oldest-based /health/active check + require DB reachability for Active", "status": "pending", "blockedBy": [5] },
-    { "id": 8, "subject": "Task 8: Divergence proof — restarted first node must NOT become active again", "status": "pending", "blockedBy": [3, 6] },
-    { "id": 9, "subject": "Task 9: CentralSingletonRegistrar helper (manager + drain task + proxy)", "status": "pending", "blockedBy": [2] },
-    { "id": 10, "subject": "Task 10: Route all seven central singletons through the registrar (adds missing outbox/ingest drains)", "status": "pending", "blockedBy": [9] },
-    { "id": 11, "subject": "Task 11: SiteHealthReportAck message + ack semantics in the communication actors", "status": "pending", "blockedBy": [] },
-    { "id": 12, "subject": "Task 12: Acked async health-report transport so counter-restore actually fires", "status": "pending", "blockedBy": [11] },
-    { "id": 13, "subject": "Task 13: Metrics-staleness signal + status-transition timestamps in the aggregator", "status": "pending", "blockedBy": [] },
-    { "id": 14, "subject": "Task 14: Surface metrics-stale + offline-since on the Health dashboard", "status": "pending", "blockedBy": [13] },
-    { "id": 15, "subject": "Task 15: Fix the metrics-port seed in appsettings.Site.json + close the validator gap", "status": "pending", "blockedBy": [] },
-    { "id": 16, "subject": "Task 16: Single-node installs — explicit AllowSingleNodeCluster instead of phantom seeds", "status": "pending", "blockedBy": [] },
-    { "id": 17, "subject": "Task 17: stop_grace_period for all app containers (both topologies)", "status": "pending", "blockedBy": [] },
-    { "id": 18, "subject": "Task 18: Evict deleted sites from the health aggregator", "status": "pending", "blockedBy": [13] },
-    { "id": 19, "subject": "Task 19: Rate-limit dead-letter warnings", "status": "pending", "blockedBy": [] },
-    { "id": 20, "subject": "Task 20: Down-if-alone recovery — exit on unexpected ActorSystem termination + service restart actions", "status": "pending", "blockedBy": [10] },
-    { "id": 21, "subject": "Task 21: Docker failover drill script (manual/CI kill test against the real topology)", "status": "pending", "blockedBy": [1, 17] },
-    { "id": 22, "subject": "Task 22: Reconcile REQ-HOST-6 (Akka.Hosting) with the hand-rolled bootstrap", "status": "pending", "blockedBy": [1] },
-    { "id": 23, "subject": "Task 23: Docs & deploy-config cleanup batch (Traefik doc, NodeName overlays, TLS roadmap note)", "status": "pending", "blockedBy": [7, 16] }
+    {
+      "id": 1,
+      "subject": "Task 1: Enable the SBR downing provider in BuildHocon",
+      "status": "completed",
+      "blockedBy": []
+    },
+    {
+      "id": 2,
+      "subject": "Task 2: Raise the cluster-leave CoordinatedShutdown phase timeout above the drain budget",
+      "status": "completed",
+      "blockedBy": [
+        1
+      ]
+    },
+    {
+      "id": 3,
+      "subject": "Task 3: Two-node in-process cluster fixture (the reusable failover rig)",
+      "status": "completed",
+      "blockedBy": [
+        1
+      ]
+    },
+    {
+      "id": 4,
+      "subject": "Task 4: SBR behavioral proof \u2014 DELIVERED as younger-node-crash (SBR downs it, oldest keeps singleton); the planned oldest-crash 'singleton migration' is impossible under 2-node keep-oldest \u2014 see master-tracker follow-up",
+      "status": "completed",
+      "blockedBy": [
+        3
+      ]
+    },
+    {
+      "id": 5,
+      "subject": "Task 5: ClusterActivityEvaluator \u2014 single oldest-member \"active node\" definition",
+      "status": "completed",
+      "blockedBy": []
+    },
+    {
+      "id": 6,
+      "subject": "Task 6: Rewire ActiveNodeGate, AkkaClusterNodeProvider and Primary labels onto the evaluator",
+      "status": "completed",
+      "blockedBy": [
+        5
+      ]
+    },
+    {
+      "id": 7,
+      "subject": "Task 7: Oldest-based /health/active check + require DB reachability for Active",
+      "status": "completed",
+      "blockedBy": [
+        5
+      ]
+    },
+    {
+      "id": 8,
+      "subject": "Task 8: Divergence proof \u2014 restarted first node must NOT become active again",
+      "status": "completed",
+      "blockedBy": [
+        3,
+        6
+      ]
+    },
+    {
+      "id": 9,
+      "subject": "Task 9: CentralSingletonRegistrar helper (manager + drain task + proxy)",
+      "status": "completed",
+      "blockedBy": [
+        2
+      ]
+    },
+    {
+      "id": 10,
+      "subject": "Task 10: Route all seven central singletons through the registrar (adds missing outbox/ingest drains)",
+      "status": "completed",
+      "blockedBy": [
+        9
+      ]
+    },
+    {
+      "id": 11,
+      "subject": "Task 11: SiteHealthReportAck message + ack semantics in the communication actors",
+      "status": "completed",
+      "blockedBy": []
+    },
+    {
+      "id": 12,
+      "subject": "Task 12: Acked async health-report transport so counter-restore actually fires",
+      "status": "completed",
+      "blockedBy": [
+        11
+      ]
+    },
+    {
+      "id": 13,
+      "subject": "Task 13: Metrics-staleness signal + status-transition timestamps in the aggregator",
+      "status": "completed",
+      "blockedBy": []
+    },
+    {
+      "id": 14,
+      "subject": "Task 14: Surface metrics-stale + offline-since on the Health dashboard",
+      "status": "completed",
+      "blockedBy": [
+        13
+      ]
+    },
+    {
+      "id": 15,
+      "subject": "Task 15: Fix the metrics-port seed in appsettings.Site.json + close the validator gap",
+      "status": "completed",
+      "blockedBy": []
+    },
+    {
+      "id": 16,
+      "subject": "Task 16: Single-node installs \u2014 explicit AllowSingleNodeCluster instead of phantom seeds",
+      "status": "completed",
+      "blockedBy": []
+    },
+    {
+      "id": 17,
+      "subject": "Task 17: stop_grace_period for all app containers (both topologies)",
+      "status": "completed",
+      "blockedBy": []
+    },
+    {
+      "id": 18,
+      "subject": "Task 18: Evict deleted sites from the health aggregator",
+      "status": "completed",
+      "blockedBy": [
+        13
+      ]
+    },
+    {
+      "id": 19,
+      "subject": "Task 19: Rate-limit dead-letter warnings",
+      "status": "completed",
+      "blockedBy": []
+    },
+    {
+      "id": 20,
+      "subject": "Task 20: Down-if-alone recovery \u2014 exit on unexpected ActorSystem termination + service restart actions",
+      "status": "completed",
+      "blockedBy": [
+        10
+      ]
+    },
+    {
+      "id": 21,
+      "subject": "Task 21: Docker failover drill script (manual/CI kill test against the real topology)",
+      "status": "completed",
+      "blockedBy": [
+        1,
+        17
+      ]
+    },
+    {
+      "id": 22,
+      "subject": "Task 22: Reconcile REQ-HOST-6 (Akka.Hosting) with the hand-rolled bootstrap",
+      "status": "completed",
+      "blockedBy": [
+        1
+      ]
+    },
+    {
+      "id": 23,
+      "subject": "Task 23: Docs & deploy-config cleanup batch (Traefik doc, NodeName overlays, TLS roadmap note)",
+      "status": "completed",
+      "blockedBy": [
+        7,
+        16
+      ]
+    }
  ],
  "lastUpdated": "2026-07-08"
-}
+}
@@ -127,3 +127,4 @@ All passwords are `password`. See `infra/glauth/config.toml` for the full list.
 - Reference deployment for the project — when a change ships, it gets validated here first.
 - Concurrent with [`docker-cluster-env2`](docker-cluster-env2.md) on the same host; the two stacks share the `scadabridge-net` network and `infra/` services but use disjoint host ports (`90XX` vs `91XX`) and databases.
 - Detailed setup, failover testing, and build-cache notes live in [`docker/README.md`](../docker/README.md).
+- Automated failover drill: [`docker/failover-drill.sh`](../docker/failover-drill.sh) SIGKILLs the active central node and asserts Traefik recovers to the survivor (~25s + Traefik health interval); see the "Automated failover drill" subsection in `docker/README.md`.
@@ -1,6 +1,10 @@
 services:
  central-a:
    image: scadabridge:latest
+    # CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
+    # actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
+    # mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
+    stop_grace_period: 30s
    container_name: scadabridge-env2-central-a
    environment:
      SCADABRIDGE_CONFIG: Central
@@ -24,6 +28,10 @@ services:

  central-b:
    image: scadabridge:latest
+    # CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
+    # actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
+    # mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
+    stop_grace_period: 30s
    container_name: scadabridge-env2-central-b
    environment:
      SCADABRIDGE_CONFIG: Central
@@ -47,6 +55,10 @@ services:

  site-x-a:
    image: scadabridge:latest
+    # CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
+    # actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
+    # mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
+    stop_grace_period: 30s
    container_name: scadabridge-env2-site-x-a
    environment:
      SCADABRIDGE_CONFIG: Site
@@ -63,6 +75,10 @@ services:

  site-x-b:
    image: scadabridge:latest
+    # CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
+    # actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
+    # mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
+    stop_grace_period: 30s
    container_name: scadabridge-env2-site-x-b
    environment:
      SCADABRIDGE_CONFIG: Site
@@ -270,6 +270,23 @@ All test passwords are `password`. See `infra/glauth/config.toml` for the full l

 ## Failover Testing

+### Automated failover drill (`failover-drill.sh`)
+
+```bash
+bash docker/failover-drill.sh
+```
+
+The scripted drill is the repeatable version of the manual steps below. It:
+
+1. Finds the **active** central node (via each node's `/health/active`).
+2. **`docker kill`s it — SIGKILL, the hard-crash path.** This is the exact scenario arch-review 01 found had *no* automatic recovery before the SBR downing provider was enabled (`Akka.Cluster.SBR.SplitBrainResolverProvider` is now named explicitly in HOCON); a `docker stop` would take the graceful `CoordinatedShutdown` path instead and would not prove crash recovery.
+3. Polls Traefik (`http://localhost:9000/health/active`, override with `TRAEFIK_URL`) until the survivor is routable, printing the observed failover time. Fails after `TIMEOUT_S` (default 90s).
+4. Dumps the survivor's singleton/oldest/downing log evidence, then restarts the victim so it **rejoins as a fresh incarnation** (Task 20's recovery loop).
+
+Expected failover: **~25s** (2s heartbeat + 10s detection + 15s stable-after) **plus the Traefik ~5s health-check interval**. The drill exercises S1 (SBR downing on hard crash), S3 (single active node routed through Traefik), and the Task 20 restart/rejoin contract. Requires a running cluster (`bash docker/deploy.sh`) and `curl` + `docker` on the host.
+
+> **Observed failover time:** _run after next deploy and record here_ (the script was validated with `bash -n`; no live cluster was up when it was committed).
+
 ### Central Failover

 ```bash
@@ -1,6 +1,10 @@
 services:
  central-a:
    image: scadabridge:latest
+    # CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
+    # actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
+    # mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
+    stop_grace_period: 30s
    container_name: scadabridge-central-a
    environment:
      SCADABRIDGE_CONFIG: Central
@@ -24,6 +28,10 @@ services:

  central-b:
    image: scadabridge:latest
+    # CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
+    # actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
+    # mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
+    stop_grace_period: 30s
    container_name: scadabridge-central-b
    environment:
      SCADABRIDGE_CONFIG: Central
@@ -47,6 +55,10 @@ services:

  site-a-a:
    image: scadabridge:latest
+    # CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
+    # actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
+    # mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
+    stop_grace_period: 30s
    container_name: scadabridge-site-a-a
    environment:
      SCADABRIDGE_CONFIG: Site
@@ -63,6 +75,10 @@ services:

  site-a-b:
    image: scadabridge:latest
+    # CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
+    # actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
+    # mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
+    stop_grace_period: 30s
    container_name: scadabridge-site-a-b
    environment:
      SCADABRIDGE_CONFIG: Site
@@ -79,6 +95,10 @@ services:

  site-b-a:
    image: scadabridge:latest
+    # CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
+    # actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
+    # mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
+    stop_grace_period: 30s
    container_name: scadabridge-site-b-a
    environment:
      SCADABRIDGE_CONFIG: Site
@@ -95,6 +115,10 @@ services:

  site-b-b:
    image: scadabridge:latest
+    # CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
+    # actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
+    # mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
+    stop_grace_period: 30s
    container_name: scadabridge-site-b-b
    environment:
      SCADABRIDGE_CONFIG: Site
@@ -111,6 +135,10 @@ services:

  site-c-a:
    image: scadabridge:latest
+    # CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
+    # actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
+    # mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
+    stop_grace_period: 30s
    container_name: scadabridge-site-c-a
    environment:
      SCADABRIDGE_CONFIG: Site
@@ -127,6 +155,10 @@ services:

  site-c-b:
    image: scadabridge:latest
+    # CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
+    # actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
+    # mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
+    stop_grace_period: 30s
    container_name: scadabridge-site-c-b
    environment:
      SCADABRIDGE_CONFIG: Site
@@ -0,0 +1,44 @@
+#!/usr/bin/env bash
+# Failover drill against the running docker cluster (bash docker/deploy.sh first).
+# Kills the ACTIVE central node (docker kill = SIGKILL, the hard-crash path that
+# review 01 found had NO automatic recovery before the SBR downing provider was
+# enabled) and measures how long Traefik takes to route to the survivor.
+set -euo pipefail
+
+TRAEFIK_URL="${TRAEFIK_URL:-http://localhost:9000}"
+TIMEOUT_S="${TIMEOUT_S:-90}"
+
+active_container() {
+  if curl -sf -o /dev/null "http://localhost:9001/health/active"; then echo scadabridge-central-a
+  elif curl -sf -o /dev/null "http://localhost:9002/health/active"; then echo scadabridge-central-b
+  else echo "ERROR: no active central node found" >&2; exit 1; fi
+}
+
+VICTIM=$(active_container)
+echo "Active central node: ${VICTIM} — killing it (SIGKILL, crash path)"
+docker kill "${VICTIM}" > /dev/null
+START=$(date +%s)
+
+echo "Waiting for the survivor to become active through Traefik (${TRAEFIK_URL}/health/active)..."
+while true; do
+  ELAPSED=$(( $(date +%s) - START ))
+  if curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; then
+    echo "PASS: failover complete in ${ELAPSED}s (design budget ~25s + Traefik health interval)"
+    break
+  fi
+  if (( ELAPSED > TIMEOUT_S )); then
+    echo "FAIL: no active central node after ${ELAPSED}s — SBR/singleton handover did not recover" >&2
+    docker start "${VICTIM}" > /dev/null
+    exit 1
+  fi
+  sleep 1
+done
+
+SURVIVOR=$([ "${VICTIM}" = scadabridge-central-a ] && echo scadabridge-central-b || echo scadabridge-central-a)
+echo "Survivor singleton evidence (last 20 matching log lines from ${SURVIVOR}):"
+docker logs "${SURVIVOR}" 2>&1 | grep -Ei "singleton|oldest|Downing|Removed" | tail -20 || true
+
+echo "Restarting ${VICTIM} and waiting for it to rejoin..."
+docker start "${VICTIM}" > /dev/null
+sleep 5
+echo "Drill complete. Verify on the Health dashboard that both nodes show Up and the survivor is Primary."
@@ -51,13 +51,13 @@ The actor system name is always `scadabridge`. Site nodes carry two cluster role

 ### `/health/ready` — readiness gating

-Central nodes register `DatabaseHealthCheck<ScadaBridgeDbContext>` (tagged `Ready`) and `AkkaClusterHealthCheck` (tagged `Ready`). The `/health/ready` endpoint returns 200 only when both pass. Readiness is explicitly not tied to cluster leadership: a fully operational standby central node still reports ready because `ActiveNodeHealthCheck` carries only the `Active` tag, not `Ready`.
+Central nodes register `DatabaseHealthCheck<ScadaBridgeDbContext>` (tagged `Ready`) and `AkkaClusterHealthCheck` (tagged `Ready`). The `/health/ready` endpoint returns 200 only when both pass. Readiness is explicitly not tied to active-node (oldest-member) status: a fully operational standby central node still reports ready because `OldestNodeActiveHealthCheck` carries only the `Active` tag, not `Ready`.

 Load balancers and orchestrators should poll `/health/ready` to determine when a freshly started or failed-over node can receive traffic.

 ### `/health/active` — active-node routing for Traefik

-`ActiveNodeHealthCheck` carries the `Active` tag and is served at `/health/active`. It returns 200 only on the cluster leader. Traefik polls this endpoint and routes inbound traffic — Central UI, Inbound API, Management API — exclusively to the node that answers 200. See [TraefikProxy](./TraefikProxy.md) for the upstream routing rules.
+`OldestNodeActiveHealthCheck` carries the `Active` tag and is served at `/health/active`. It returns 200 only on the oldest Up cluster member (the singleton host) whose database check also passes — the oldest-member definition of "active" via `ClusterActivityEvaluator.SelfIsOldest`, not cluster leadership. Traefik polls this endpoint and routes inbound traffic — Central UI, Inbound API, Management API — exclusively to the node that answers 200. See [TraefikProxy](./TraefikProxy.md) for the upstream routing rules.

 The same leadership check backs `ActiveNodeGate`, the `IActiveNodeGate` implementation the Inbound API endpoint filter consults before executing a method script. A standby node therefore refuses inbound API calls even if traffic somehow reaches it directly.

@@ -217,12 +217,12 @@ All other per-component configuration sections (`ScadaBridge:Communication`, `Sc
 - [Cluster Infrastructure (#13)](./ClusterInfrastructure.md) — the Host configures the underlying Akka.NET cluster (`AkkaHostedService.BuildHocon`); ClusterInfrastructure manages it at runtime.
 - [Configuration Database (#17)](./ConfigurationDatabase.md) — the Host registers `ScadaBridgeDbContext` and calls `AddConfigurationDatabase` (Central only); the `StartupRetry`-wrapped migration step runs before traffic is accepted.
 - [Central–Site Communication (#5)](./Communication.md) — the Host creates `CentralCommunicationActor` and `SiteCommunicationActor`, registers them with `ClusterClientReceptionist`, and wires the `ClusterClient` for site→central messaging; the gRPC server is mapped at `app.MapGrpcService<SiteStreamGrpcServer>()`.
- [Health Monitoring (#11)](./HealthMonitoring.md) — the Host registers health checks (`DatabaseHealthCheck`, `AkkaClusterHealthCheck`, `ActiveNodeHealthCheck`) and mounts them via `app.MapZbHealth()` on central; site nodes register `AddSiteHealthMonitoring` and `AkkaHealthReportTransport`.
+- [Health Monitoring (#11)](./HealthMonitoring.md) — the Host registers health checks (`DatabaseHealthCheck`, `AkkaClusterHealthCheck`, `OldestNodeActiveHealthCheck`) and mounts them via `app.MapZbHealth()` on central; site nodes register `AddSiteHealthMonitoring` and `AkkaHealthReportTransport`.
 - [Audit Log (#23)](./AuditLog.md) — the Host calls `AddAuditLog` on both roles, `AddAuditLogCentralMaintenance` on central, and `AddAuditLogHealthMetricsBridge` on site; it creates the `AuditLogIngestActor` singleton and registers `SiteAuditTelemetryActor` on the dedicated dispatcher.
 - [Notification Outbox (#21)](./NotificationOutbox.md) — the Host creates the `NotificationOutboxActor` cluster singleton and hands its proxy to `CentralCommunicationActor`.
 - [Site Call Audit (#22)](./SiteCallAudit.md) — the Host creates the `SiteCallAuditActor` cluster singleton with a graceful-stop drain task registered in the `cluster-leave` coordinated-shutdown phase.
 - [Management Service (#18)](./ManagementService.md) — the Host creates `ManagementActor` and registers it with `ClusterClientReceptionist`; maps the Management and Audit HTTP APIs.
- [Traefik Proxy (#20)](./TraefikProxy.md) — Traefik polls `/health/active` to determine which central node to route traffic to; the Host implements the `ActiveNodeHealthCheck` and `ActiveNodeGate` that back this endpoint.
+- [Traefik Proxy (#20)](./TraefikProxy.md) — Traefik polls `/health/active` to determine which central node to route traffic to; the Host implements the `OldestNodeActiveHealthCheck` and `ActiveNodeGate` that back this endpoint.
 - Design spec: [Component-Host.md](../requirements/Component-Host.md).

 ## Troubleshooting
@@ -1,6 +1,6 @@
 # Traefik Proxy

-The Traefik Proxy is the reverse proxy and load balancer that fronts the central cluster's two web servers. It exposes a single stable entrypoint for all central traffic — Central UI, Management API, Inbound API — and routes exclusively to whichever central node is currently the Akka.NET cluster leader, using a health-check on each node's `/health/active` endpoint to make that determination. When the active node changes, Traefik detects the change on its next poll cycle and redirects traffic automatically, with no operator intervention.
+The Traefik Proxy is the reverse proxy and load balancer that fronts the central cluster's two web servers. It exposes a single stable entrypoint for all central traffic — Central UI, Management API, Inbound API — and routes exclusively to whichever central node is currently the active node (the oldest Up cluster member, which hosts the cluster singletons), using a health-check on each node's `/health/active` endpoint to make that determination. When the active node changes, Traefik detects the change on its next poll cycle and redirects traffic automatically, with no operator intervention.

 ## Overview

@@ -15,7 +15,7 @@ The proxy sits on the `scadabridge-net` Docker bridge network alongside both cen

 ### Active-node routing via `/health/active`

-Traefik does not know which central node is the Akka.NET cluster leader — it discovers this by polling `/health/active` on both backends. The Host registers `ActiveNodeHealthCheck` under the `Active` health tag; `app.MapZbHealth()` serves it at `/health/active`. The check returns HTTP 200 on the leader and HTTP 503 on the standby (or when the actor system has not yet reached `MemberStatus.Up`):
+Traefik does not know which central node is the active (oldest Up / singleton-host) node — it discovers this by polling `/health/active` on both backends. The Host registers `OldestNodeActiveHealthCheck` under the `Active` health tag; `app.MapZbHealth()` serves it at `/health/active`. The check returns HTTP 200 on the oldest Up member (and only when its database check also passes) and HTTP 503 on the standby (or when the actor system has not yet reached `MemberStatus.Up`):

 ```csharp
 public bool IsActiveNode
@@ -26,22 +26,19 @@ public bool IsActiveNode
        if (system == null)
            return false;

-        var cluster = Cluster.Get(system);
-        var self = cluster.SelfMember;
-        if (self.Status != MemberStatus.Up)
-            return false;
-
-        var leader = cluster.State.Leader;
-        return leader != null && leader == self.Address;
+        // Oldest Up member, NOT cluster leader: cluster singletons follow the
+        // oldest member, and leader/oldest can differ transiently during
+        // membership churn — routing must track the singleton host.
+        return ClusterActivityEvaluator.SelfIsOldest(Cluster.Get(system));
    }
 }
 ```

-The identical leadership check backs `ActiveNodeGate` — the `IActiveNodeGate` implementation the Inbound API endpoint filter consults before executing method scripts. Both surfaces agree on which node is active because they share the same Akka cluster state.
+The identical oldest-member check backs `ActiveNodeGate` — the `IActiveNodeGate` implementation the Inbound API endpoint filter consults before executing method scripts. Both surfaces agree on which node is active because they share the same `ClusterActivityEvaluator.SelfIsOldest` definition over Akka cluster state.

 ### Automatic failover

-When the active central node goes down, the Akka cluster's keep-oldest split-brain resolver promotes the surviving node to leader (roughly 25 seconds: 10-second heartbeat threshold plus a 15-second stable-after period). Once the surviving node's `ActiveNodeHealthCheck` starts returning 200, Traefik's next poll cycle — within the 5-second interval — removes the failed backend from the pool and routes all subsequent requests to the new active node. No config change or restart is required on the Traefik side.
+When the active central node goes down, the Akka cluster's keep-oldest split-brain resolver leaves the surviving node as the oldest Up member and the singletons migrate to it (roughly 25 seconds: 10-second heartbeat threshold plus a 15-second stable-after period). Once the surviving node's `OldestNodeActiveHealthCheck` starts returning 200, Traefik's next poll cycle — within the 5-second interval — removes the failed backend from the pool and routes all subsequent requests to the new active node. No config change or restart is required on the Traefik side.

 ## Architecture

@@ -158,8 +155,8 @@ http:

 ## Dependencies & Interactions

- [Host (#15)](./Host.md) — implements and serves `/health/active` via `ActiveNodeHealthCheck` (tagged `Active`, mounted by `app.MapZbHealth()`). Also implements `ActiveNodeGate`, which enforces the same active-node contract at the Inbound API filter level, providing a defence-in-depth layer if traffic reaches the standby directly.
- [Cluster Infrastructure (#13)](./ClusterInfrastructure.md) — the underlying Akka.NET cluster determines which node is the leader. Traefik's routing decision is derived entirely from cluster leadership state via the health-check poll; Traefik has no Akka dependency of its own.
+- [Host (#15)](./Host.md) — implements and serves `/health/active` via `OldestNodeActiveHealthCheck` (tagged `Active`, mounted by `app.MapZbHealth()`). Also implements `ActiveNodeGate`, which enforces the same active-node contract at the Inbound API filter level, providing a defence-in-depth layer if traffic reaches the standby directly.
+- [Cluster Infrastructure (#13)](./ClusterInfrastructure.md) — the underlying Akka.NET cluster determines which node is the oldest Up member (the singleton host). Traefik's routing decision is derived entirely from that membership state via the health-check poll; Traefik has no Akka dependency of its own.
 - [Central UI (#9)](./CentralUI.md) — Blazor Server (SignalR/WebSocket circuits) is proxied through Traefik. Traefik proxies WebSocket connections natively with no additional config. On failover, active SignalR circuits on the failed node are lost; the browser's reconnection logic re-establishes the circuit on the new active node. Session continuity is preserved because authentication uses a cookie-embedded JWT with Data Protection keys shared across both central nodes.
 - [Inbound API (#14)](./InboundAPI.md) — external API consumers target `http://localhost:9000/api/{methodName}`. Traefik routes each request to the active node; if a request reaches the standby directly (bypassing Traefik), `ActiveNodeGate` responds with HTTP 503.
 - [CLI (#19)](./CLI.md) — the CLI connects to the Management API via `http://localhost:9000` (the Traefik entrypoint) by default, so it always reaches the active central node without needing to know which node is active.
@@ -168,7 +165,7 @@ http:

 ### Both backends show unhealthy on the dashboard

-If both `central-a` and `central-b` appear red on the Traefik dashboard, neither node's `ActiveNodeHealthCheck` is returning 200. Common causes:
+If both `central-a` and `central-b` appear red on the Traefik dashboard, neither node's `OldestNodeActiveHealthCheck` is returning 200. Common causes:

 1. **Akka cluster has not formed yet** — both nodes are still starting. Wait for the cluster to stabilise (typically 10–15 seconds after both containers are up). Check the central node logs for `Cluster is now ready`.
 2. **Split-brain resolver has downed both nodes** — a network partition followed by a split-brain condition. Restart the cluster via `bash docker/deploy.sh`.
@@ -180,7 +177,7 @@ If a client receives HTTP 503 with `X-ScadaBridge-Active: false`, the request re

 ### Health check succeeds but `/health/ready` returns degraded

-`/health/active` and `/health/ready` are independent. A node can pass the active check (it is the leader) but fail the readiness check (database or Akka cluster health probe failed). Traefik only uses `/health/active`; readiness gating is for orchestration and monitoring. Check the node's structured logs for `database` or `akka-cluster` check failures.
+`/health/active` and `/health/ready` overlap on the database check but are otherwise independent. `/health/active` is 200 only on the oldest Up member **and** requires its database check to pass; `/health/ready` gates on database + akka-cluster + required-singletons. Traefik only uses `/health/active`; readiness gating is for orchestration and monitoring. Check the node's structured logs for `database` or `akka-cluster` check failures.

 ## Related Documentation

@@ -101,6 +101,17 @@ The system uses the Akka.NET **keep-oldest** split-brain resolver strategy:
 - **`down-if-alone = on`**: The keep-oldest resolver is configured with `down-if-alone` enabled. If the oldest node finds itself alone (no other reachable members), it downs itself rather than continuing as a single-node cluster. This prevents the oldest node from running in isolation during a network partition while the younger node also forms its own cluster.
 - **Why keep-oldest**: With only two nodes, quorum-based strategies (static-quorum, keep-majority) cannot distinguish "one node crashed" from "network partition" — both sides see fewer than quorum and both would down themselves, resulting in total cluster shutdown. Keep-oldest with `down-if-alone` provides safe singleton ownership — at most one node runs the cluster singleton at any time.

+### Down-if-alone recovery
+
+When a node downs itself (via `down-if-alone`, or any other SBR decision), the resolver is configured with `run-coordinated-shutdown-when-down = on`, so the self-down runs `CoordinatedShutdown` and **terminates that node's `ActorSystem`**. The Host process must not keep running with a dead actor system — it would serve nothing and be restarted by nobody. The recovery contract is:
+
+1. Self-down ⇒ `CoordinatedShutdown` ⇒ `ActorSystem` termination.
+2. The Host watches `ActorSystem.WhenTerminated`; a termination that is **not** the host's own `StopAsync` triggers `IHostApplicationLifetime.StopApplication()`, so **the process exits**.
+3. The service supervisor restarts it — docker `restart: unless-stopped`, or Windows service recovery actions (`sc.exe failure … restart/…`).
+4. The restarted process rejoins as a **fresh incarnation** (the keep-oldest resolver handles the rejoin cleanly; there is no stale membership to reconcile).
+
+The docker failover drill exercises this loop end-to-end (kill the active node, assert singleton migration + clean rejoin).
+
 ## Single-Node Operation

 `akka.cluster.min-nr-of-members` must be set to **1**. After failover, only one node is running. If set to 2, the surviving node waits for a second member before allowing the Cluster Singleton (Site Runtime Deployment Manager) to start — blocking all data collection and script execution indefinitely.
@@ -42,8 +42,11 @@ Site clusters (metric collection and reporting). Central cluster (aggregation an

 - Sites send a **health report message** to central at a configurable interval (default: **30 seconds**).
 - Each report is a **flat snapshot** containing the current values of all monitored metrics, a **monotonic sequence number**, and the **report timestamp** from the site. Central replaces the previous state for that site only if the incoming sequence number is higher than the last received — this prevents stale reports (e.g., delayed in transit or from a pre-failover node) from overwriting newer state.
- **Offline detection**: If central does not receive a report within a configurable timeout window (default: **60 seconds** — 2x the report interval), the site is marked as **offline**. This gives one missed report as grace before marking offline.
- **Online recovery**: When central receives a health report from a site that was marked offline, the site is automatically marked **online**. No manual acknowledgment required — the metrics in the report provide immediate visibility into the site's condition.
+- Central tracks two **distinct** signals per site — do not conflate them:
+  - **Liveness (online/offline)**: driven by the **last signal of any kind** (a full report *or* a transport heartbeat, the latter arriving every ~5s from any reachable site node). If central receives no signal within `OfflineTimeout` (default: **60 seconds**), the site is marked **offline**. Because heartbeats are frequent, this only fires on genuine total loss of contact — not during a single-node failover, when the surviving node keeps heartbeating. The synthetic *central* self-report site has no heartbeat source and uses the longer `CentralOfflineTimeout` (default: **3 minutes**) so a single missed self-report does not flap it offline.
+  - **Metrics staleness**: an **online** site (heartbeats still arriving) that has produced no full **report** within `MetricsStaleTimeout` (default: **2 minutes**) is flagged **metrics stale**. This surfaces the failure mode where a site's report pipeline (`HealthReportSender`) has died but the node is otherwise reachable — previously such a site showed "online with frozen metrics forever". `MetricsStaleTimeout` must be positive and no shorter than the report interval.
+- **Online recovery**: when central receives a health report from a site that was marked offline, the site is automatically marked **online** and the metrics-stale flag cleared (a fresh report proves the pipeline is alive); a heartbeat alone also restores **online** but leaves the metrics-stale flag untouched. No manual acknowledgment required.
+- **Status-transition timestamp**: each online↔offline flip stamps `LastStatusChangeAt`, so the UI can answer "since when has this site been offline/online" rather than only "is it offline now". Report/heartbeat traffic that leaves the status unchanged does not move it.

 ## Error Rate Metrics

@@ -13,7 +13,7 @@ All nodes (central and site).
 - Serve as the single entry point (`Program.cs`) for the ScadaBridge process.
 - Read and validate node configuration at startup before any actor system is created.
 - Register the correct set of component services and actors based on the configured node role.
- Bootstrap the Akka.NET actor system with Remoting, Clustering, and split-brain resolution via Akka.Hosting.
+- Bootstrap the Akka.NET actor system with Remoting, Clustering, and split-brain resolution from a hand-assembled, injection-safe HOCON document (`AkkaHostedService.BuildHocon`; see REQ-HOST-6).
 - Host ASP.NET Core web endpoints on central nodes only.
 - Configure structured logging (Serilog) with environment-specific enrichment.
 - Support running as a Windows Service in production and as a console application during development.
@@ -105,13 +105,15 @@ The Host must support running as a Windows Service via `UseWindowsService()`. Wh

 ### REQ-HOST-6: Akka.NET Bootstrap

-The Host must configure the Akka.NET actor system using Akka.Hosting with:
+The Host bootstraps the Akka.NET actor system from a **hand-assembled, injection-safe HOCON document** (`AkkaHostedService.BuildHocon`) passed to `ActorSystem.Create` — **not** Akka.Hosting's `AkkaConfigurationBuilder`/`AddAkka()`. Every interpolated value is routed through `QuoteHocon`/`DurationHocon` so a hostname, seed URI or strategy containing a quote, backslash or whitespace cannot corrupt the document, and each timing is rendered in milliseconds so sub-second values survive. The bound `ClusterOptions`/`NodeOptions`/`CommunicationOptions` are startup-validated (`ClusterOptionsValidator`, `StartupValidator`) so a misconfiguration fails fast with a key-naming message. The document configures:

- **Remoting**: Configured with the node's hostname and port from `NodeConfiguration`.
- **Clustering**: Configured with seed nodes and the node's cluster role from configuration.
- **Persistence**: Configured with the appropriate journal and snapshot store (SQL for central, SQLite for site).
- **Split-Brain Resolver**: Configured with the strategy and stable-after duration from `ClusterConfiguration`.
- **Actor registration**: Each component's actors registered via its `AddXxxActors()` extension method, conditional on the node's role.
+- **Remoting**: the node's hostname and port from `NodeOptions`.
+- **Clustering**: seed nodes and the node's cluster role(s) from configuration.
+- **Split-Brain Resolver**: the strategy, stable-after and `down-if-alone` from `ClusterOptions`, **with the downing provider named explicitly** (`akka.cluster.downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster"`). Akka's default is `NoDowning`, under which the entire SBR section is inert and singletons never migrate on a crash or partition — omitting this one line silently disables failover. This exact omission was arch-review 01's Critical finding; it is now guarded by tests asserting the parsed HOCON key **and** a two-node kill test asserting the behavior (downing + singleton migration).
+- **Coordinated shutdown**: `run-coordinated-shutdown-when-down = on` plus a cluster-leave phase timeout above the singleton drain budget (see REQ-HOST-4a / the down-if-alone recovery contract in Component-ClusterInfrastructure.md).
+- **Actor registration**: each component's actors registered conditional on the node's role.
+
+> **Bootstrap-mechanism note.** The `Akka.Hosting` / `Akka.Cluster.Hosting` / `Akka.Remote.Hosting` typed-builder packages were referenced but unused and were dropped (arch-review 01 Task 22). The hand-rolled HOCON path is a single hardened, production-shape code path with full test coverage; migrating to Akka.Hosting's typed options is deliberately **not** done here and is tracked as possible future work rather than carried as an unused dependency. Only `Akka.Cluster.Tools` (ClusterSingleton/ClusterClient, transitively Akka.Cluster) remains.

 > **Persistence note.** ScadaBridge does not use Akka.Persistence. Durable state
 > (store-and-forward buffers, site event logs, static attribute writes,
@@ -203,8 +205,7 @@ The Host's `Program.cs` calls these extension methods; the component libraries o
 ## Dependencies

 - **All 19 component libraries**: The Host references every component project to call their extension methods (excludes CLI, which is a separate executable). Audit Log (#23) ships its central+site code in `ZB.MOM.WW.ScadaBridge.AuditLog`; the Host calls `AddAuditLog()` on both roles, M2+ will add `AddAuditLogActors()`.
- **Akka.Hosting**: For `AddAkka()` and the hosting configuration builder.
- **Akka.Remote.Hosting, Akka.Cluster.Hosting**: For Akka subsystem configuration. (No Akka.Persistence plugin — see the Persistence note under REQ-HOST-6.)
+- **Akka.Cluster.Tools**: ClusterSingleton (singleton manager/proxy) and ClusterClient/ClusterClientReceptionist; transitively pulls Akka.Cluster (SBR provider) and Akka.Remote. The actor system is built from hand-assembled HOCON (REQ-HOST-6), so the `Akka.Hosting`/`Akka.Remote.Hosting`/`Akka.Cluster.Hosting` typed-builder packages are **not** referenced (dropped in arch-review 01 Task 22). No Akka.Persistence plugin — see the Persistence note under REQ-HOST-6.
 - **Serilog.AspNetCore**: For structured logging integration.
 - **Microsoft.Extensions.Hosting.WindowsServices**: For Windows Service support.
 - **ASP.NET Core** (central only): For web endpoint hosting.
@@ -13,7 +13,7 @@ Runs as a Docker container (`scadabridge-traefik`) in the cluster compose stack
 ## Responsibilities

 - Route all HTTP traffic (Central UI, Management API, Inbound API, health endpoints) to the active central node.
- Health-check both central nodes via `/health/active` to determine which is the active (cluster leader) node.
+- Health-check both central nodes via `/health/active` to determine which is the active (oldest Up member / singleton host) node.
 - Automatically fail over to the standby node when the active node goes down.
 - Provide a dashboard for monitoring routing state and backend health.

@@ -23,18 +23,18 @@ Runs as a Docker container (`scadabridge-traefik`) in the cluster compose stack

 Traefik polls `/health/active` on both central nodes every 5 seconds. This endpoint returns:

- **HTTP 200** on the active node (the Akka.NET cluster leader).
- **HTTP 503** on the standby node (or if the node is unreachable).
+- **HTTP 200** on the active node (the oldest Up cluster member — the node that hosts the cluster singletons).
+- **HTTP 503** on the standby node (or if the node is unreachable, or if its database is not reachable).

-Only the node returning 200 receives traffic. The health check is implemented by `ActiveNodeHealthCheck` in the Host project, which checks `Cluster.Get(system).State.Leader == SelfMember.Address`.
+Only the node returning 200 receives traffic. The health check is implemented by `OldestNodeActiveHealthCheck` in the Host project, which reports Healthy only when `ClusterActivityEvaluator.SelfIsOldest(cluster)` is true — the **oldest-member** definition of "active", not the cluster *leader* (leader and oldest can differ transiently during membership churn; singletons follow the oldest member, so routing must too). `/health/active` is additionally tagged with the database readiness check, so a node whose database is unreachable sheds traffic even while it is the oldest member.

 ### Failover Sequence

 1. Active node fails (crash, network partition, or graceful shutdown).
 2. Akka.NET cluster detects the failure (~10s heartbeat timeout).
 3. Split-brain resolver acts after stable-after period (~15s).
-4. Surviving node becomes cluster leader.
-5. `ActiveNodeHealthCheck` on the surviving node starts returning 200.
+4. Surviving node becomes the oldest Up member and the singletons migrate to it.
+5. `OldestNodeActiveHealthCheck` on the surviving node starts returning 200 (its database check must also pass).
 6. Traefik's next health poll (within 5s) detects the change.
 7. Traffic routes to the new active node.

@@ -112,13 +112,13 @@ The central nodes expose three health endpoints:

 | Endpoint | Purpose | Who Uses It |
 |----------|---------|-------------|
-| `/health/ready` | Readiness gate — 200 when database + Akka cluster are healthy | Kubernetes probes, monitoring |
-| `/health/active` | Active node — 200 only on cluster leader | **Traefik** (routing decisions) |
+| `/health/ready` | Readiness gate — 200 when database + akka-cluster + required-singletons are healthy | Kubernetes probes, monitoring |
+| `/health/active` | Active node — 200 only on the oldest Up member (singleton host) whose database is also reachable | **Traefik** (routing decisions) |

 ## Dependencies

 - **Central cluster nodes**: The two backends (`scadabridge-central-a`, `scadabridge-central-b`) on the `scadabridge-net` Docker network.
- **ActiveNodeHealthCheck**: Health check implementation in `src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeHealthCheck.cs` that determines cluster leader status.
+- **OldestNodeActiveHealthCheck**: Health check implementation in `src/ZB.MOM.WW.ScadaBridge.Host/Health/OldestNodeActiveHealthCheck.cs` that reports the oldest-Up-member (singleton-host) status via `ClusterActivityEvaluator.SelfIsOldest`; `/health/active` also carries the database readiness check.
 - **Docker network**: All containers must be on the shared `scadabridge-net` bridge network.

 ## Interactions
@@ -126,10 +126,12 @@ The central nodes expose three health endpoints:
 - **CLI**: Connects to `http://localhost:9000/management` — routed by Traefik to the active node.
 - **Browser (Central UI)**: Connects to `http://localhost:9000` — Blazor Server + SignalR routed to the active node.
 - **Inbound API consumers**: Connect to `http://localhost:9000/api/{methodName}` — routed to the active node.
- **Cluster Infrastructure**: The `ActiveNodeHealthCheck` relies on Akka.NET cluster gossip state to determine the leader.
+- **Cluster Infrastructure**: The `OldestNodeActiveHealthCheck` relies on Akka.NET cluster membership state to determine the oldest Up member (the singleton host).

 ## Production Considerations

+> **Production TLS profile: not yet implemented.** Traefik terminates plain HTTP, Akka remoting is unencrypted TCP, and site gRPC is `http://`. This is acceptable for the lab topology only; a production deployment requires TLS at all three layers (Traefik HTTPS entrypoint, Akka remoting TLS, gRPC over `https://`) and a secured Traefik dashboard. Tracked as an open roadmap item (arch review 01, underdeveloped area 3).
+
 The current configuration is for development/testing. In production:

 - **TLS termination**: Add HTTPS entrypoint with certificates (Let's Encrypt via Traefik's ACME provider, or static certs).
@@ -18,6 +18,7 @@
@inject CommunicationService CommunicationService
@inject IAuditLogQueryService AuditLogQueryService
@inject IKpiHistoryQueryService KpiHistory
+@inject Microsoft.Extensions.Options.IOptions<HealthMonitoringOptions> HealthOptions

 <div class="container-fluid mt-3">
    <div class="d-flex justify-content-between align-items-center mb-3">
@@ -199,6 +200,17 @@
                            <span class="badge bg-danger me-2" aria-label="State: Offline">@OfflineGlyph Offline</span>
                        }
                        <strong class="fs-5">@siteName@(isCentral ? "" : $" ({siteId})")</strong>
+                        @if (state.IsOnline && state.IsMetricsStale)
+                        {
+                            <span class="badge bg-warning text-dark ms-2"
+                                  title="Site is heartbeating but has sent no health report for over @StaleTimeoutDisplay — the metrics pipeline may be dead.">
+                                Metrics stale
+                            </span>
+                        }
+                        @if (!state.IsOnline && state.LastStatusChangeAt is { } changedAt)
+                        {
+                            <small class="text-muted ms-2">offline since @changedAt.ToString("u")</small>
+                        }
                    </div>
                    <small class="text-muted">
                        Last report: <TimestampDisplay Value="@state.LastReportReceivedAt" Format="HH:mm:ss" NullText="awaiting first report" />
@@ -419,6 +431,16 @@
    private const string PrimaryGlyph = "▲";  // ▲
    private const string StandbyGlyph = "△";  // △

+    // Human-friendly rendering of the configured metrics-stale window for the
+    // "Metrics stale" badge tooltip (e.g. "2 minutes").
+    private string StaleTimeoutDisplay =>
+        FormatDuration(HealthOptions.Value.MetricsStaleTimeout);
+
+    private static string FormatDuration(TimeSpan span) =>
+        span.TotalMinutes >= 1 && span == TimeSpan.FromMinutes(Math.Round(span.TotalMinutes))
+            ? $"{span.TotalMinutes:0} minute{(span.TotalMinutes == 1 ? "" : "s")}"
+            : $"{span.TotalSeconds:0} seconds";
+
    private IReadOnlyDictionary<string, SiteHealthState> _siteStates = new Dictionary<string, SiteHealthState>();
    private Dictionary<string, string> _siteNames = new();
    private Timer? _refreshTimer;
@@ -76,4 +76,12 @@ public class ClusterOptions
    /// other reachable members, rather than running as an isolated single-node cluster.
    /// </summary>
    public bool DownIfAlone { get; set; } = true;
+
+    /// <summary>
+    /// Acknowledges a deliberate single-node deployment: permits exactly one seed
+    /// node (normally ≥2 are required so either node can start first). Without this
+    /// flag a single-node install must list a phantom second seed, which the node
+    /// dials forever — log noise and a defeated validation intent (review 01).
+    /// </summary>
+    public bool AllowSingleNodeCluster { get; set; }
 }
@@ -26,11 +26,13 @@ public sealed class ClusterOptionsValidator : OptionsValidatorBase<ClusterOption
        // two. Accepting a single-seed configuration silently defeats the
        // "no startup ordering dependency" guarantee called out by
        // Component-ClusterInfrastructure.md (Node Configuration).
-        builder.RequireThat(options.SeedNodes is not null && options.SeedNodes.Count >= 2,
-            "ClusterOptions.SeedNodes must contain at least 2 seed nodes "
-            + "(Component-ClusterInfrastructure.md → Node Configuration: "
-            + "both nodes are seed nodes); a single-seed configuration defeats "
-            + "the no-startup-ordering-dependency guarantee.");
+        var minSeeds = options.AllowSingleNodeCluster ? 1 : 2;
+        builder.RequireThat(options.SeedNodes is not null && options.SeedNodes.Count >= minSeeds,
+            options.AllowSingleNodeCluster
+                ? "ClusterOptions.SeedNodes must contain at least 1 seed node."
+                : "ClusterOptions.SeedNodes must contain at least 2 seed nodes "
+                  + "(Component-ClusterInfrastructure.md → Node Configuration: both nodes are seed nodes); "
+                  + "for a deliberate single-node install set ClusterOptions.AllowSingleNodeCluster = true instead of listing a phantom seed.");

        builder.RequireThat(
            !string.IsNullOrWhiteSpace(options.SplitBrainResolverStrategy)
@@ -0,0 +1,10 @@
+namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
+
+/// <summary>
+/// Acknowledgement for a <see cref="SiteHealthReport"/> forwarded site→central.
+/// Review 01 [Medium]: the transport was fire-and-forget, so the sender's
+/// interval-counter restore logic could never observe a loss. The ack makes
+/// delivery observable end-to-end (central processed the report).
+/// </summary>
+public sealed record SiteHealthReportAck(
+    string SiteId, long SequenceNumber, bool Accepted, string? Error = null);
@@ -394,6 +394,15 @@ public class CentralCommunicationActor : ReceiveActor
        {
            // No-op in non-clustered hosts (TestKit).
        }
+
+        // Ack the site so its AkkaHealthReportTransport Ask completes and the
+        // report-loss counter-restore path can observe delivery (review 01
+        // [Medium]). Guarded so the peer-replica path (SiteHealthReportReplica,
+        // which arrives without an Ask sender) never dead-letters an ack.
+        if (!Sender.IsNobody())
+        {
+            Sender.Tell(new SiteHealthReportAck(report.SiteId, report.SequenceNumber, Accepted: true));
+        }
    }

    /// <summary>
@@ -490,7 +499,12 @@ public class CentralCommunicationActor : ReceiveActor
            var frozen = contacts.ToDictionary(
                kvp => kvp.Key,
                kvp => (IReadOnlyList<string>)kvp.Value.AsReadOnly());
-            return new SiteAddressCacheLoaded(frozen);
+
+            // Carry the full configured-site identifier set (not just the
+            // address-bearing subset in `frozen`) so the aggregator prunes only
+            // genuinely-deleted sites and never an addressless-but-configured one.
+            var knownSiteIds = sites.Select(s => s.SiteIdentifier).ToList();
+            return new SiteAddressCacheLoaded(frozen, knownSiteIds);
        }).PipeTo(self);
    }

@@ -561,6 +575,12 @@ public class CentralCommunicationActor : ReceiveActor
        }

        _log.Info("Site ClusterClient cache refreshed with {0} site(s)", _siteClients.Count);
+
+        // Self-healing eviction: a site deleted from configuration would otherwise
+        // linger in the aggregator as a permanently-offline tile (and live KPI
+        // sample source) forever. Prune on every refresh so it disappears within
+        // one refresh interval without needing a dedicated deletion event.
+        _serviceProvider.GetService<ICentralHealthAggregator>()?.PruneUnknownSites(msg.KnownSiteIds);
    }

    // TrackMessageForCleanup removed — the dicts it fed
@@ -641,7 +661,9 @@ public record RefreshSiteAddresses;
 /// discipline. The producer wraps the constructed buckets with
 /// <c>List&lt;T&gt;.AsReadOnly()</c> before piping to Self.
 /// </summary>
-internal record SiteAddressCacheLoaded(IReadOnlyDictionary<string, IReadOnlyList<string>> SiteContacts);
+internal record SiteAddressCacheLoaded(
+    IReadOnlyDictionary<string, IReadOnlyList<string>> SiteContacts,
+    IReadOnlyCollection<string> KnownSiteIds);

 /// <summary>
 /// Notification sent to debug view subscribers when the stream is terminated
@@ -390,11 +390,28 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
        // Internal: send heartbeat tick
        Receive<SendHeartbeat>(_ => SendHeartbeatToCentral());

-        // Internal: forward health report to central
+        // Internal: forward health report to central. The original Sender (the
+        // AkkaHealthReportTransport's Ask) is forwarded as the ClusterClient.Send
+        // sender so the central SiteHealthReportAck routes straight back to the
+        // waiting Ask — making report delivery observable end-to-end (review 01
+        // [Medium]). Mirrors the NotificationSubmit ack pattern above.
        Receive<SiteHealthReport>(msg =>
        {
-            _centralClient?.Tell(
-                new ClusterClient.Send("/user/central-communication", msg), Self);
+            if (_centralClient == null)
+            {
+                // No ClusterClient registered yet. A non-accepted ack makes the
+                // sender's counter-restore path treat this tick as a loss.
+                _log.Warning(
+                    "Cannot forward SiteHealthReport #{0} — no central ClusterClient registered",
+                    msg.SequenceNumber);
+                Sender.Tell(new SiteHealthReportAck(
+                    msg.SiteId, msg.SequenceNumber, Accepted: false,
+                    Error: "Central ClusterClient not registered"));
+                return;
+            }
+
+            _centralClient.Tell(
+                new ClusterClient.Send("/user/central-communication", msg), Sender);
        });

    }
@@ -48,7 +48,9 @@ public class CentralHealthAggregator : BackgroundService, ICentralHealthAggregat
                    LastReportReceivedAt = now,
                    LastHeartbeatAt = now,
                    LastSequenceNumber = report.SequenceNumber,
-                    IsOnline = true
+                    IsOnline = true,
+                    IsMetricsStale = false
+                    // LastStatusChangeAt stays null: first observation is not a flip.
                };

                if (_siteStates.TryAdd(report.SiteId, registered))
@@ -76,7 +78,12 @@ public class CentralHealthAggregator : BackgroundService, ICentralHealthAggregat
                LastReportReceivedAt = now,
                LastHeartbeatAt = now,
                LastSequenceNumber = report.SequenceNumber,
-                IsOnline = true
+                IsOnline = true,
+                // A fresh report is proof the metrics pipeline is alive again.
+                IsMetricsStale = false,
+                // Stamp a flip only when this report brings the site back online;
+                // a report from an already-online site leaves the timestamp be.
+                LastStatusChangeAt = existing.IsOnline ? existing.LastStatusChangeAt : now
            };

            if (_siteStates.TryUpdate(report.SiteId, updated, existing))
@@ -155,7 +162,14 @@ public class CentralHealthAggregator : BackgroundService, ICentralHealthAggregat
            var updated = existing with
            {
                LastHeartbeatAt = newHeartbeat,
-                IsOnline = true
+                IsOnline = true,
+                // Stamp the flip when this heartbeat promotes an offline site back
+                // online. IsMetricsStale is deliberately left untouched — a
+                // heartbeat says nothing about whether the metrics pipeline
+                // recovered; only a full report clears staleness.
+                LastStatusChangeAt = existing.IsOnline
+                    ? existing.LastStatusChangeAt
+                    : _timeProvider.GetUtcNow()
            };

            if (_siteStates.TryUpdate(siteId, updated, existing))
@@ -182,6 +196,20 @@ public class CentralHealthAggregator : BackgroundService, ICentralHealthAggregat
        return state;
    }

+    /// <inheritdoc />
+    public void PruneUnknownSites(IReadOnlyCollection<string> knownSiteIds)
+    {
+        var known = new HashSet<string>(knownSiteIds, StringComparer.Ordinal);
+        foreach (var siteId in _siteStates.Keys)
+        {
+            if (siteId == CentralHealthReportLoop.CentralSiteId || known.Contains(siteId))
+                continue;
+            if (_siteStates.TryRemove(siteId, out _))
+                _logger.LogInformation(
+                    "Site {SiteId} evicted from health aggregator (no longer configured)", siteId);
+        }
+    }
+
    /// <inheritdoc />
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
@@ -246,18 +274,41 @@ public class CentralHealthAggregator : BackgroundService, ICentralHealthAggregat
                : _options.OfflineTimeout;

            var elapsed = now - state.LastHeartbeatAt;
-            if (elapsed <= timeout)
-                continue;
-
-            // Atomically swap to an offline copy. If the CAS loses to a
-            // concurrent report/heartbeat the site was just heard from, so
-            // leaving it online is the correct outcome — no retry needed.
-            var offline = state with { IsOnline = false };
-            if (_siteStates.TryUpdate(kvp.Key, offline, state))
+            if (elapsed > timeout)
            {
-                _logger.LogWarning(
-                    "Site {SiteId} marked offline — no signal for {Elapsed}s (timeout: {Timeout}s)",
-                    state.SiteId, elapsed.TotalSeconds, timeout.TotalSeconds);
+                // Atomically swap to an offline copy, stamping the flip. If the
+                // CAS loses to a concurrent report/heartbeat the site was just
+                // heard from, so leaving it online is correct — no retry needed.
+                var offline = state with { IsOnline = false, LastStatusChangeAt = now };
+                if (_siteStates.TryUpdate(kvp.Key, offline, state))
+                {
+                    _logger.LogWarning(
+                        "Site {SiteId} marked offline — no signal for {Elapsed}s (timeout: {Timeout}s)",
+                        state.SiteId, elapsed.TotalSeconds, timeout.TotalSeconds);
+                }
+
+                // Whether the CAS won or lost, do not also evaluate staleness on
+                // this stale snapshot — offline supersedes metrics-stale.
+                continue;
+            }
+
+            // Second signal: the site is still live (heartbeats within timeout)
+            // but its full-report pipeline may have died. Flag it metrics-stale
+            // when no report has arrived within MetricsStaleTimeout. Distinct from
+            // offline — a site whose HealthReportSender crashed keeps heartbeating
+            // and would otherwise show "online with frozen metrics forever".
+            if (state.LastReportReceivedAt is { } lastReport
+                && now - lastReport > _options.MetricsStaleTimeout
+                && !state.IsMetricsStale)
+            {
+                var stale = state with { IsMetricsStale = true };
+                if (_siteStates.TryUpdate(kvp.Key, stale, state))
+                {
+                    _logger.LogWarning(
+                        "Site {SiteId} metrics are stale — online (heartbeats) but no report for {Elapsed}s (timeout: {Timeout}s)",
+                        state.SiteId, (now - lastReport).TotalSeconds, _options.MetricsStaleTimeout.TotalSeconds);
+                }
+                // CAS loss ⇒ a fresh report/heartbeat swapped in ⇒ correct to skip.
            }
        }
    }
@@ -19,4 +19,17 @@ public class HealthMonitoringOptions
    /// the design doc grants real sites. Default: 3x the report interval.
    /// </summary>
    public TimeSpan CentralOfflineTimeout { get; set; } = TimeSpan.FromMinutes(3);
+
+    /// <summary>
+    /// Duration an online site may go without a full health <em>report</em>
+    /// (heartbeats notwithstanding) before it is flagged
+    /// <see cref="SiteHealthState.IsMetricsStale"/>. Distinct from
+    /// <see cref="OfflineTimeout"/>, which is heartbeat-based liveness: a site
+    /// whose HealthReportSender has died keeps heartbeating and stays "online",
+    /// but its metrics freeze — this window surfaces that. Must be positive and
+    /// no shorter than <see cref="ReportInterval"/> (a window shorter than one
+    /// report interval would flag every healthy site). Default: 2 minutes
+    /// (~4 missed 30s reports).
+    /// </summary>
+    public TimeSpan MetricsStaleTimeout { get; set; } = TimeSpan.FromMinutes(2);
 }
@@ -45,5 +45,24 @@ public sealed class HealthMonitoringOptionsValidator : OptionsValidatorBase<Heal
            $"must be >= OfflineTimeout ({options.OfflineTimeout}): the synthetic 'central' site has " +
            "no heartbeat source and is fed only by the slower self-report loop, so it needs at " +
            "least as much offline grace as a real site.");
+
+        builder.RequireThat(options.MetricsStaleTimeout > TimeSpan.Zero,
+            $"ScadaBridge:HealthMonitoring:MetricsStaleTimeout must be a positive duration " +
+            $"(was {options.MetricsStaleTimeout}); it is the window an online site may go without " +
+            "a full report before being flagged metrics-stale.");
+
+        // A stale window shorter than one report interval flags every healthy
+        // site (reports simply can't arrive faster than they are emitted). Same
+        // De Morgan guard shape as the Central/Offline pair above: stay silent
+        // when either field is non-positive so the dedicated positive-duration
+        // checks own that failure rather than double-firing here.
+        builder.RequireThat(
+            !(options.ReportInterval > TimeSpan.Zero
+              && options.MetricsStaleTimeout > TimeSpan.Zero
+              && options.MetricsStaleTimeout < options.ReportInterval),
+            $"ScadaBridge:HealthMonitoring:MetricsStaleTimeout ({options.MetricsStaleTimeout}) " +
+            $"must be >= ReportInterval ({options.ReportInterval}): a stale window shorter than one " +
+            "report interval would flag every healthy site, since reports cannot arrive faster than " +
+            "they are emitted.");
    }
 }
@@ -140,10 +140,13 @@ public class HealthReportSender : BackgroundService
                var seq = Interlocked.Increment(ref _sequenceNumber);

                // CollectReport atomically read-and-resets
-                // the per-interval error counters via Interlocked.Exchange. If
-                // the Send below throws, those counts are otherwise lost
-                // forever — neither in the un-sent report nor in the now-zeroed
-                // collector. Snapshot the freshly-collected report so that on a
+                // the per-interval error counters via Interlocked.Exchange. The
+                // transport is now an ACKED round-trip (SendAsync throws on
+                // timeout/nack — review 01 [Medium]), so this restore path is live
+                // rather than dead code. If the SendAsync below throws, those
+                // counts are otherwise lost forever — neither in the un-sent
+                // report nor in the now-zeroed collector. Snapshot the
+                // freshly-collected report so that on a
                // transport failure we can atomically restore the counts back
                // into the collector via Interlocked.Add, so the next
                // successful report includes them. Concurrent increments
@@ -157,7 +160,7 @@ public class HealthReportSender : BackgroundService

                try
                {
-                    _transport.Send(reportWithSeq);
+                    await _transport.SendAsync(reportWithSeq, stoppingToken).ConfigureAwait(false);
                }
                catch
                {
@@ -39,4 +39,16 @@ public interface ICentralHealthAggregator
    /// <param name="siteId">The string identifier of the site to look up.</param>
    /// <returns>The <see cref="SiteHealthState"/> for the site, or null if not yet seen.</returns>
    SiteHealthState? GetSiteState(string siteId);
+
+    /// <summary>
+    /// Removes tracked sites not in <paramref name="knownSiteIds"/> (i.e. deleted
+    /// from configuration). The synthetic central id is always retained.
+    /// Self-healing: called from the CentralCommunicationActor's periodic site
+    /// refresh, so a deleted site disappears within one refresh interval without
+    /// needing a dedicated deletion event. Without this the in-memory state only
+    /// ever grew, leaving a deleted site as a permanently-offline dashboard tile
+    /// (and a live KPI sample source) forever.
+    /// </summary>
+    /// <param name="knownSiteIds">The identifiers of all currently-configured sites.</param>
+    void PruneUnknownSites(IReadOnlyCollection<string> knownSiteIds);
 }
@@ -13,9 +13,13 @@ public interface IClusterNodeProvider
    IReadOnlyList<NodeStatus> GetClusterNodes();

    /// <summary>
-    /// True when this node is currently the cluster leader (Primary) for the
-    /// provider's role scope. Used by the central report loop to decide which
-    /// node should generate the "central" health report.
+    /// True when this node is currently the oldest Up member for the provider's
+    /// role scope — the node the ClusterSingletonManager hosts singletons on.
+    /// This is the canonical product "active node" check (review 01): the site
+    /// event-log purge gate, the central self-report loop, and (via plan 02) the
+    /// S&amp;F delivery gate all consume it. It is deliberately NOT the cluster
+    /// leader — leadership is address-ordered and diverges from singleton
+    /// placement once the original first node restarts and rejoins.
    /// </summary>
    bool SelfIsPrimary { get; }
 }
@@ -4,13 +4,20 @@ namespace ZB.MOM.WW.ScadaBridge.HealthMonitoring;

 /// <summary>
 /// Abstraction for sending health reports to central.
-/// In production, implemented via Akka remoting (Tell, fire-and-forget).
+/// In production, implemented over Akka ClusterClient with an acked round-trip
+/// (review 01 [Medium]): the transport Asks and awaits a
+/// <see cref="SiteHealthReportAck"/>, so a lost/rejected report surfaces as a
+/// fault the sender can observe (and its per-interval counter-restore fires).
 /// </summary>
 public interface IHealthReportTransport
 {
    /// <summary>
-    /// Sends a health report to central (fire-and-forget).
+    /// Sends a health report to central and completes only once central has
+    /// acknowledged processing. MUST THROW on non-delivery (Ask timeout, actor
+    /// system not started, or a not-accepted ack) — the sender's counter-restore
+    /// path in <c>HealthReportSender</c> depends on that exception.
    /// </summary>
    /// <param name="report">The site health report to send.</param>
-    void Send(SiteHealthReport report);
+    /// <param name="cancellationToken">Cancels the in-flight send.</param>
+    Task SendAsync(SiteHealthReport report, CancellationToken cancellationToken);
 }
@@ -44,4 +44,23 @@ public sealed record SiteHealthState
    public long LastSequenceNumber { get; init; }
    /// <summary>Gets a value indicating whether the site is currently considered online.</summary>
    public bool IsOnline { get; init; }
+
+    /// <summary>
+    /// Gets a value indicating whether the site is online (heartbeats still
+    /// arriving) but has sent no full <see cref="SiteHealthReport"/> within
+    /// <see cref="HealthMonitoringOptions.MetricsStaleTimeout"/>. This is a signal
+    /// <em>distinct</em> from liveness: a site whose HealthReportSender died keeps
+    /// heartbeating and would otherwise show "online with frozen metrics forever".
+    /// Cleared whenever a fresh report is processed. Heartbeats never set or clear
+    /// it — they say nothing about the metrics pipeline.
+    /// </summary>
+    public bool IsMetricsStale { get; init; }
+
+    /// <summary>
+    /// Gets the instant of the last online↔offline flip, or <c>null</c> if the
+    /// site has never changed status since it was first observed. Answers "when
+    /// did this site drop / come back". Not moved by report/heartbeat traffic that
+    /// leaves the online/offline status unchanged, nor by the metrics-stale flag.
+    /// </summary>
+    public DateTimeOffset? LastStatusChangeAt { get; init; }
 }
@@ -33,8 +33,18 @@ public class AkkaHostedService : IHostedService
    private readonly ClusterOptions _clusterOptions;
    private readonly CommunicationOptions _communicationOptions;
    private readonly ILogger<AkkaHostedService> _logger;
+    private readonly IHostApplicationLifetime? _appLifetime;
    private ActorSystem? _actorSystem;

+    /// <summary>
+    /// Flips to <c>true</c> the moment <see cref="StopAsync"/> begins, so the
+    /// <see cref="ActorSystem.WhenTerminated"/> watchdog can tell an
+    /// intentional host shutdown apart from an SBR self-down that terminates the
+    /// system out from under a still-running process. <c>volatile</c>: written on
+    /// the shutdown thread, read on the Akka termination-callback thread.
+    /// </summary>
+    private volatile bool _stopRequested;
+
    /// <summary>
    /// Guards the one-time creation of <see cref="_actorSystem"/> in
    /// <see cref="GetOrCreateActorSystem"/> so <see cref="StartAsync"/> and a concurrent
@@ -78,13 +88,17 @@ public class AkkaHostedService : IHostedService
        IOptions<NodeOptions> nodeOptions,
        IOptions<ClusterOptions> clusterOptions,
        IOptions<CommunicationOptions> communicationOptions,
-        ILogger<AkkaHostedService> logger)
+        ILogger<AkkaHostedService> logger,
+        IHostApplicationLifetime? appLifetime = null)
    {
        _serviceProvider = serviceProvider;
        _nodeOptions = nodeOptions.Value;
        _clusterOptions = clusterOptions.Value;
        _communicationOptions = communicationOptions.Value;
        _logger = logger;
+        // Optional so existing construction sites (tests, DI variants) keep
+        // compiling. When absent the watchdog logs but can't stop the process.
+        _appLifetime = appLifetime;
    }

    /// <summary>
@@ -185,6 +199,22 @@ public class AkkaHostedService : IHostedService
                _communicationOptions.TransportHeartbeatInterval.TotalSeconds,
                _communicationOptions.TransportFailureThreshold.TotalSeconds);

+            // Down-if-alone recovery watchdog: SBR's keep-oldest down-if-alone plus
+            // run-coordinated-shutdown-when-down means a self-downed node terminates
+            // its own ActorSystem. If that happens outside our StopAsync, the Host
+            // process must exit so the service supervisor (docker
+            // `restart: unless-stopped` / Windows service recovery) restarts it and
+            // it rejoins as a fresh incarnation. Without this the node idles forever
+            // with a dead actor system (review 01 underdeveloped #2).
+            system.WhenTerminated.ContinueWith(_ =>
+            {
+                if (_stopRequested) return;
+                _logger.LogCritical(
+                    "ActorSystem terminated outside host shutdown (SBR down / run-coordinated-shutdown-when-down). "
+                    + "Stopping the host process so the service supervisor restarts this node as a fresh incarnation.");
+                _appLifetime?.StopApplication();
+            }, TaskContinuationOptions.ExecuteSynchronously);
+
            // Publish last so a concurrent reader never observes a half-constructed system.
            _actorSystem = system;
            return _actorSystem;
@@ -272,6 +302,7 @@ akka {{
    }}
    coordinated-shutdown {{
        run-by-clr-shutdown-hook = on
+        phases.cluster-leave.timeout = 15s
    }}
 }}";
    }
@@ -307,6 +338,12 @@ akka {{
    /// <returns>A task representing the asynchronous operation.</returns>
    public async Task StopAsync(CancellationToken cancellationToken)
    {
+        // Mark this as an intentional shutdown FIRST, so the WhenTerminated
+        // watchdog (wired in GetOrCreateActorSystem) does not mistake the
+        // CoordinatedShutdown below for an SBR self-down and re-trigger
+        // StopApplication on an already-stopping host.
+        _stopRequested = true;
+
        // Dispose auxiliary subscribers (e.g. SiteAuditTelemetryStalledTracker)
        // BEFORE Akka shuts down so their EventStream unsubscribe calls run
        // while the system is still alive. Per-tracker Dispose is wrapped in
@@ -413,30 +450,29 @@ akka {{
        var outboxAuditWriter = _serviceProvider
            .GetRequiredService<ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services.ICentralAuditWriter>();

-        var outboxSingletonProps = ClusterSingletonManager.Props(
-            singletonProps: Props.Create(() => new ZB.MOM.WW.ScadaBridge.NotificationOutbox.NotificationOutboxActor(
+        // All central singletons below are created through CentralSingletonRegistrar.Start,
+        // which pins the actor to the active central node (ClusterSingletonManager), exposes a
+        // stable address (ClusterSingletonProxy), and — crucially — registers a PhaseClusterLeave
+        // GracefulStop drain task so an in-flight EF write drains before handover (review 01
+        // [Medium]: notification-outbox and audit-log-ingest previously had NO drain task). Names
+        // are unchanged ({name}-singleton / {name}-proxy). The two SITE singletons stay
+        // hand-rolled below because they are role-scoped (.WithRole(siteRole)).
+        var outbox = CentralSingletonRegistrar.Start(
+            _actorSystem!, "notification-outbox",
+            Props.Create(() => new ZB.MOM.WW.ScadaBridge.NotificationOutbox.NotificationOutboxActor(
                _serviceProvider,
                outboxOptions,
                outboxAuditWriter,
                outboxLogger)),
-            terminationMessage: PoisonPill.Instance,
-            settings: ClusterSingletonManagerSettings.Create(_actorSystem!)
-                .WithSingletonName("notification-outbox"));
-        _actorSystem!.ActorOf(outboxSingletonProps, "notification-outbox-singleton");
-
-        var outboxProxyProps = ClusterSingletonProxy.Props(
-            singletonManagerPath: "/user/notification-outbox-singleton",
-            settings: ClusterSingletonProxySettings.Create(_actorSystem)
-                .WithSingletonName("notification-outbox"));
-        var outboxProxy = _actorSystem.ActorOf(outboxProxyProps, "notification-outbox-proxy");
+            _logger);

        // Hand the outbox proxy to the CentralCommunicationActor so forwarded
        // NotificationSubmit messages from sites are routed to the outbox singleton.
-        centralCommActor.Tell(new RegisterNotificationOutbox(outboxProxy));
+        centralCommActor.Tell(new RegisterNotificationOutbox(outbox.Proxy));

        // Hand the same proxy to the CommunicationService so the Central UI can
        // Ask the outbox actor directly (query, retry, discard, KPIs).
-        commService?.SetNotificationOutbox(outboxProxy);
+        commService?.SetNotificationOutbox(outbox.Proxy);
        _logger.LogInformation("NotificationOutbox singleton created and registered with CentralCommunicationActor");

        // Audit Log — central singleton mirrors the Notification Outbox
@@ -454,25 +490,17 @@ akka {{
        var auditIngestLogger = _serviceProvider.GetRequiredService<ILoggerFactory>()
            .CreateLogger<ZB.MOM.WW.ScadaBridge.AuditLog.Central.AuditLogIngestActor>();

-        var auditIngestSingletonProps = ClusterSingletonManager.Props(
-            singletonProps: Props.Create(() => new ZB.MOM.WW.ScadaBridge.AuditLog.Central.AuditLogIngestActor(
+        var auditIngest = CentralSingletonRegistrar.Start(
+            _actorSystem!, "audit-log-ingest",
+            Props.Create(() => new ZB.MOM.WW.ScadaBridge.AuditLog.Central.AuditLogIngestActor(
                _serviceProvider,
                auditIngestLogger)),
-            terminationMessage: PoisonPill.Instance,
-            settings: ClusterSingletonManagerSettings.Create(_actorSystem!)
-                .WithSingletonName("audit-log-ingest"));
-        _actorSystem!.ActorOf(auditIngestSingletonProps, "audit-log-ingest-singleton");
-
-        var auditIngestProxyProps = ClusterSingletonProxy.Props(
-            singletonManagerPath: "/user/audit-log-ingest-singleton",
-            settings: ClusterSingletonProxySettings.Create(_actorSystem)
-                .WithSingletonName("audit-log-ingest"));
-        var auditIngestProxy = _actorSystem.ActorOf(auditIngestProxyProps, "audit-log-ingest-proxy");
+            _logger);

        // Hand the audit-ingest proxy to the CentralCommunicationActor so audit
        // ingest commands forwarded by sites over ClusterClient are routed to the
        // singleton. Mirrors the RegisterNotificationOutbox wiring above.
-        centralCommActor.Tell(new RegisterAuditIngest(auditIngestProxy));
+        centralCommActor.Tell(new RegisterAuditIngest(auditIngest.Proxy));

        // Hand the proxy to the SiteStreamGrpcServer (if registered on this node)
        // so the IngestAuditEvents RPC routes incoming site batches to the singleton.
@@ -480,7 +508,7 @@ akka {{
        // node this resolves to null and the wiring is a no-op unless a future
        // central-hosted gRPC server (or a real site→central client) is added.
        var grpcServer = _serviceProvider.GetService<ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamGrpcServer>();
-        grpcServer?.SetAuditIngestActor(auditIngestProxy);
+        grpcServer?.SetAuditIngestActor(auditIngest.Proxy);
        _logger.LogInformation(
            "AuditLogIngestActor singleton created (gRPC server bound: {GrpcBound})",
            grpcServer is not null);
@@ -527,61 +555,18 @@ akka {{
        var siteCallAuditOptions = _serviceProvider
            .GetRequiredService<IOptions<ZB.MOM.WW.ScadaBridge.SiteCallAudit.SiteCallAuditOptions>>().Value;

-        var siteCallAuditSingletonProps = ClusterSingletonManager.Props(
-            singletonProps: Props.Create(() => new ZB.MOM.WW.ScadaBridge.SiteCallAudit.SiteCallAuditActor(
+        var siteCallAudit = CentralSingletonRegistrar.Start(
+            _actorSystem!, "site-call-audit",
+            Props.Create(() => new ZB.MOM.WW.ScadaBridge.SiteCallAudit.SiteCallAuditActor(
                _serviceProvider,
                siteCallAuditOptions,
                siteCallAuditLogger)),
-            terminationMessage: PoisonPill.Instance,
-            settings: ClusterSingletonManagerSettings.Create(_actorSystem!)
-                .WithSingletonName("site-call-audit"));
-        var siteCallAuditSingletonManager =
-            _actorSystem!.ActorOf(siteCallAuditSingletonProps, "site-call-audit-singleton");
-
-        // Graceful-handover hook. The default singleton handover
-        // path waits for the actor's `ReceiveAsync` task to complete before
-        // signalling `HandOverDone` to the new oldest node — so an in-flight
-        // EF `UpsertAsync` IS waited for during a *clean* coordinated shutdown
-        // (the cluster-leave phase below fires before the singleton terminates).
-        // The risk here is the seam between in-flight async work
-        // and the cluster-leave + singleton-stop sequence: we bound it by
-        // issuing an explicit `GracefulStop` to the singleton manager early
-        // in `cluster-leave`, with a timeout that lets the running upsert + SQL
-        // round-trip drain before the handover-to-other-node race window
-        // opens. The timeout is bounded so a misbehaving upsert cannot stall
-        // coordinated shutdown indefinitely — exceeding it falls through to
-        // the existing PoisonPill termination path. Same pattern is suitable
-        // for the NotificationOutbox singleton; not added here to keep this
-        // change minimal.
-        var siteCallAuditShutdown = Akka.Actor.CoordinatedShutdown.Get(_actorSystem);
-        siteCallAuditShutdown.AddTask(
-            Akka.Actor.CoordinatedShutdown.PhaseClusterLeave,
-            "drain-site-call-audit-singleton",
-            async () =>
-            {
-                try
-                {
-                    await siteCallAuditSingletonManager.GracefulStop(TimeSpan.FromSeconds(10));
-                }
-                catch (Exception ex)
-                {
-                    _logger.LogWarning(ex,
-                        "SiteCallAudit singleton did not drain within the graceful-stop "
-                        + "timeout; falling through to PoisonPill handover");
-                }
-                return Akka.Done.Instance;
-            });
-
-        var siteCallAuditProxyProps = ClusterSingletonProxy.Props(
-            singletonManagerPath: "/user/site-call-audit-singleton",
-            settings: ClusterSingletonProxySettings.Create(_actorSystem)
-                .WithSingletonName("site-call-audit"));
-        var siteCallAuditProxy = _actorSystem.ActorOf(siteCallAuditProxyProps, "site-call-audit-proxy");
+            _logger);

        // Hand the proxy to the CommunicationService so the Central UI can Ask
        // the Site Call Audit actor directly (query, KPIs, detail) — mirrors the
        // SetNotificationOutbox wiring above.
-        commService?.SetSiteCallAudit(siteCallAuditProxy);
+        commService?.SetSiteCallAudit(siteCallAudit.Proxy);

        // Hand the CentralCommunicationActor to the SiteCallAudit
        // actor so it can relay operator Retry/Discard on parked cached calls to
@@ -589,7 +574,7 @@ akka {{
        // Mirrors the RegisterAuditIngest / RegisterNotificationOutbox wiring;
        // the message is sent to the singleton proxy so it reaches whichever
        // central node currently hosts the singleton.
-        siteCallAuditProxy.Tell(
+        siteCallAudit.Proxy.Tell(
            new ZB.MOM.WW.ScadaBridge.SiteCallAudit.RegisterCentralCommunication(centralCommActor));
        _logger.LogInformation(
            "SiteCallAuditActor singleton created and registered with CentralCommunicationActor");
@@ -615,42 +600,14 @@ akka {{
        var auditLogOptions = _serviceProvider
            .GetRequiredService<IOptions<ZB.MOM.WW.ScadaBridge.AuditLog.Configuration.AuditLogOptions>>();

-        var auditPurgeSingletonProps = ClusterSingletonManager.Props(
-            singletonProps: Props.Create(() => new ZB.MOM.WW.ScadaBridge.AuditLog.Central.AuditLogPurgeActor(
+        CentralSingletonRegistrar.Start(
+            _actorSystem!, "audit-log-purge",
+            Props.Create(() => new ZB.MOM.WW.ScadaBridge.AuditLog.Central.AuditLogPurgeActor(
                _serviceProvider,
                auditPurgeOptions,
                auditLogOptions,
                auditPurgeLogger)),
-            terminationMessage: PoisonPill.Instance,
-            settings: ClusterSingletonManagerSettings.Create(_actorSystem!)
-                .WithSingletonName("audit-log-purge"));
-        var auditPurgeSingletonManager =
-            _actorSystem!.ActorOf(auditPurgeSingletonProps, "audit-log-purge-singleton");
-
-        var auditPurgeShutdown = Akka.Actor.CoordinatedShutdown.Get(_actorSystem);
-        auditPurgeShutdown.AddTask(
-            Akka.Actor.CoordinatedShutdown.PhaseClusterLeave,
-            "drain-audit-log-purge-singleton",
-            async () =>
-            {
-                try
-                {
-                    await auditPurgeSingletonManager.GracefulStop(TimeSpan.FromSeconds(10));
-                }
-                catch (Exception ex)
-                {
-                    _logger.LogWarning(ex,
-                        "AuditLogPurge singleton did not drain within the graceful-stop "
-                        + "timeout; falling through to PoisonPill handover");
-                }
-                return Akka.Done.Instance;
-            });
-
-        var auditPurgeProxyProps = ClusterSingletonProxy.Props(
-            singletonManagerPath: "/user/audit-log-purge-singleton",
-            settings: ClusterSingletonProxySettings.Create(_actorSystem)
-                .WithSingletonName("audit-log-purge"));
-        _actorSystem.ActorOf(auditPurgeProxyProps, "audit-log-purge-proxy");
+            _logger);
        _logger.LogInformation("AuditLogPurgeActor singleton created");

        // SiteAuditReconciliationActor — self-healing fallback puller. Resolves
@@ -666,43 +623,15 @@ akka {{
        var auditReconClient = _serviceProvider
            .GetRequiredService<ZB.MOM.WW.ScadaBridge.AuditLog.Central.IPullAuditEventsClient>();

-        var auditReconSingletonProps = ClusterSingletonManager.Props(
-            singletonProps: Props.Create(() => new ZB.MOM.WW.ScadaBridge.AuditLog.Central.SiteAuditReconciliationActor(
+        CentralSingletonRegistrar.Start(
+            _actorSystem!, "site-audit-reconciliation",
+            Props.Create(() => new ZB.MOM.WW.ScadaBridge.AuditLog.Central.SiteAuditReconciliationActor(
                auditReconSites,
                auditReconClient,
                _serviceProvider,
                auditReconOptions,
                auditReconLogger)),
-            terminationMessage: PoisonPill.Instance,
-            settings: ClusterSingletonManagerSettings.Create(_actorSystem!)
-                .WithSingletonName("site-audit-reconciliation"));
-        var auditReconSingletonManager =
-            _actorSystem!.ActorOf(auditReconSingletonProps, "site-audit-reconciliation-singleton");
-
-        var auditReconShutdown = Akka.Actor.CoordinatedShutdown.Get(_actorSystem);
-        auditReconShutdown.AddTask(
-            Akka.Actor.CoordinatedShutdown.PhaseClusterLeave,
-            "drain-site-audit-reconciliation-singleton",
-            async () =>
-            {
-                try
-                {
-                    await auditReconSingletonManager.GracefulStop(TimeSpan.FromSeconds(10));
-                }
-                catch (Exception ex)
-                {
-                    _logger.LogWarning(ex,
-                        "SiteAuditReconciliation singleton did not drain within the graceful-stop "
-                        + "timeout; falling through to PoisonPill handover");
-                }
-                return Akka.Done.Instance;
-            });
-
-        var auditReconProxyProps = ClusterSingletonProxy.Props(
-            singletonManagerPath: "/user/site-audit-reconciliation-singleton",
-            settings: ClusterSingletonProxySettings.Create(_actorSystem)
-                .WithSingletonName("site-audit-reconciliation"));
-        _actorSystem.ActorOf(auditReconProxyProps, "site-audit-reconciliation-proxy");
+            _logger);
        _logger.LogInformation("SiteAuditReconciliationActor singleton created");

        // KPI History — central singleton that periodically samples the
@@ -725,41 +654,13 @@ akka {{
        var kpiHistoryLogger = _serviceProvider.GetRequiredService<ILoggerFactory>()
            .CreateLogger<ZB.MOM.WW.ScadaBridge.KpiHistory.KpiHistoryRecorderActor>();

-        var kpiHistorySingletonProps = ClusterSingletonManager.Props(
-            singletonProps: Props.Create(() => new ZB.MOM.WW.ScadaBridge.KpiHistory.KpiHistoryRecorderActor(
+        CentralSingletonRegistrar.Start(
+            _actorSystem!, "kpi-history-recorder",
+            Props.Create(() => new ZB.MOM.WW.ScadaBridge.KpiHistory.KpiHistoryRecorderActor(
                _serviceProvider,
                kpiHistoryOptions,
                kpiHistoryLogger)),
-            terminationMessage: PoisonPill.Instance,
-            settings: ClusterSingletonManagerSettings.Create(_actorSystem!)
-                .WithSingletonName("kpi-history-recorder"));
-        var kpiHistorySingletonManager =
-            _actorSystem!.ActorOf(kpiHistorySingletonProps, "kpi-history-recorder-singleton");
-
-        var kpiHistoryShutdown = Akka.Actor.CoordinatedShutdown.Get(_actorSystem);
-        kpiHistoryShutdown.AddTask(
-            Akka.Actor.CoordinatedShutdown.PhaseClusterLeave,
-            "drain-kpi-history-recorder-singleton",
-            async () =>
-            {
-                try
-                {
-                    await kpiHistorySingletonManager.GracefulStop(TimeSpan.FromSeconds(10));
-                }
-                catch (Exception ex)
-                {
-                    _logger.LogWarning(ex,
-                        "KpiHistoryRecorder singleton did not drain within the graceful-stop "
-                        + "timeout; falling through to PoisonPill handover");
-                }
-                return Akka.Done.Instance;
-            });
-
-        var kpiHistoryProxyProps = ClusterSingletonProxy.Props(
-            singletonManagerPath: "/user/kpi-history-recorder-singleton",
-            settings: ClusterSingletonProxySettings.Create(_actorSystem)
-                .WithSingletonName("kpi-history-recorder"));
-        _actorSystem.ActorOf(kpiHistoryProxyProps, "kpi-history-recorder-proxy");
+            _logger);
        _logger.LogInformation("KpiHistoryRecorderActor singleton created (not readiness-gated)");

        // Notify-and-fetch deploy transport — central singleton that periodically
@@ -779,41 +680,13 @@ akka {{
        var pendingPurgeCommunicationOptions =
            _serviceProvider.GetRequiredService<IOptions<CommunicationOptions>>();

-        var pendingPurgeSingletonProps = ClusterSingletonManager.Props(
-            singletonProps: Props.Create(() => new PendingDeploymentPurgeActor(
+        CentralSingletonRegistrar.Start(
+            _actorSystem!, "pending-deployment-purge",
+            Props.Create(() => new PendingDeploymentPurgeActor(
                _serviceProvider,
                pendingPurgeCommunicationOptions,
                pendingPurgeLogger)),
-            terminationMessage: PoisonPill.Instance,
-            settings: ClusterSingletonManagerSettings.Create(_actorSystem!)
-                .WithSingletonName("pending-deployment-purge"));
-        var pendingPurgeSingletonManager =
-            _actorSystem!.ActorOf(pendingPurgeSingletonProps, "pending-deployment-purge-singleton");
-
-        var pendingPurgeShutdown = Akka.Actor.CoordinatedShutdown.Get(_actorSystem);
-        pendingPurgeShutdown.AddTask(
-            Akka.Actor.CoordinatedShutdown.PhaseClusterLeave,
-            "drain-pending-deployment-purge-singleton",
-            async () =>
-            {
-                try
-                {
-                    await pendingPurgeSingletonManager.GracefulStop(TimeSpan.FromSeconds(10));
-                }
-                catch (Exception ex)
-                {
-                    _logger.LogWarning(ex,
-                        "PendingDeploymentPurge singleton did not drain within the graceful-stop "
-                        + "timeout; falling through to PoisonPill handover");
-                }
-                return Akka.Done.Instance;
-            });
-
-        var pendingPurgeProxyProps = ClusterSingletonProxy.Props(
-            singletonManagerPath: "/user/pending-deployment-purge-singleton",
-            settings: ClusterSingletonProxySettings.Create(_actorSystem)
-                .WithSingletonName("pending-deployment-purge"));
-        _actorSystem.ActorOf(pendingPurgeProxyProps, "pending-deployment-purge-proxy");
+            _logger);
        _logger.LogInformation("PendingDeploymentPurgeActor singleton created (not readiness-gated)");

        _logger.LogInformation("Central actors registered. CentralCommunicationActor created.");
@@ -0,0 +1,61 @@
+using Akka.Actor;
+using Akka.Cluster.Tools.Singleton;
+using Microsoft.Extensions.Logging;
+
+namespace ZB.MOM.WW.ScadaBridge.Host.Actors;
+
+/// <summary>
+/// Registers a central cluster singleton with the canonical naming scheme
+/// (<c>{name}-singleton</c> / <c>{name}-proxy</c>), a PoisonPill termination
+/// message, and a PhaseClusterLeave drain task that GracefulStops the manager
+/// so in-flight EF work completes before handover. Extracted from five
+/// copy-pasted ~60-line blocks (review 01 [Low]) whose drift left the two
+/// busiest singletons (notification-outbox, audit-log-ingest) without drain
+/// tasks (review 01 [Medium]).
+/// </summary>
+internal static class CentralSingletonRegistrar
+{
+    internal sealed record Handle(IActorRef Manager, IActorRef Proxy);
+
+    internal static Handle Start(
+        ActorSystem system,
+        string name,
+        Props singletonProps,
+        ILogger logger,
+        TimeSpan? drainTimeout = null)
+    {
+        var manager = system.ActorOf(
+            ClusterSingletonManager.Props(
+                singletonProps,
+                PoisonPill.Instance,
+                ClusterSingletonManagerSettings.Create(system).WithSingletonName(name)),
+            $"{name}-singleton");
+
+        var timeout = drainTimeout ?? TimeSpan.FromSeconds(10);
+        Akka.Actor.CoordinatedShutdown.Get(system).AddTask(
+            Akka.Actor.CoordinatedShutdown.PhaseClusterLeave,
+            $"drain-{name}-singleton",
+            async () =>
+            {
+                try
+                {
+                    await manager.GracefulStop(timeout);
+                }
+                catch (Exception ex)
+                {
+                    logger.LogWarning(ex,
+                        "{Singleton} singleton did not drain within the graceful-stop timeout; "
+                        + "falling through to PoisonPill handover", name);
+                }
+                return Akka.Done.Instance;
+            });
+
+        var proxy = system.ActorOf(
+            ClusterSingletonProxy.Props(
+                $"/user/{name}-singleton",
+                ClusterSingletonProxySettings.Create(system).WithSingletonName(name)),
+            $"{name}-proxy");
+
+        return new Handle(manager, proxy);
+    }
+}
@@ -8,12 +8,31 @@ namespace ZB.MOM.WW.ScadaBridge.Host.Actors;
 /// <summary>
 /// Subscribes to Akka.NET dead letter events, logs them, and tracks count
 /// for health monitoring integration.
+///
+/// Warning logging is rate-limited to <see cref="MaxWarningsPerWindow"/> per
+/// <see cref="WarningWindow"/>: a failover/undeploy storm can produce thousands
+/// of dead letters, and one Warning per letter is an unbounded log flood
+/// (review 01 [Low]). Once the per-window cap is hit, per-letter warnings are
+/// suppressed and a single summary of the suppressed count is logged when the
+/// window rolls over. The health-metric count is <b>never</b> throttled — every
+/// dead letter is counted.
 /// </summary>
-public class DeadLetterMonitorActor : ReceiveActor
+public class DeadLetterMonitorActor : ReceiveActor, IWithTimers
 {
+    /// <summary>Maximum per-letter Warning logs emitted within one <see cref="WarningWindow"/>.</summary>
+    public const int MaxWarningsPerWindow = 10;
+
+    /// <summary>Rolling window over which <see cref="MaxWarningsPerWindow"/> applies.</summary>
+    internal static readonly TimeSpan WarningWindow = TimeSpan.FromMinutes(1);
+
    private long _deadLetterCount;
+    private int _warningsThisWindow;
+    private long _suppressedThisWindow;
    private readonly ISiteHealthCollector? _healthCollector;

+    /// <summary>Akka timer scheduler (assigned by the runtime via <see cref="IWithTimers"/>).</summary>
+    public ITimerScheduler Timers { get; set; } = null!;
+
    /// <summary>
    /// Initializes the actor and registers dead-letter message handlers.
    /// </summary>
@@ -25,13 +44,39 @@ public class DeadLetterMonitorActor : ReceiveActor

        Receive<DeadLetter>(dl =>
        {
+            // Metric counting is never throttled.
            _deadLetterCount++;
            _healthCollector?.IncrementDeadLetter();
-            logger.LogWarning(
-                "Dead letter: {MessageType} from {Sender} to {Recipient}",
-                dl.Message.GetType().Name,
-                dl.Sender,
-                dl.Recipient);
+
+            if (_warningsThisWindow < MaxWarningsPerWindow)
+            {
+                _warningsThisWindow++;
+                logger.LogWarning(
+                    "Dead letter: {MessageType} from {Sender} to {Recipient}",
+                    dl.Message.GetType().Name,
+                    dl.Sender,
+                    dl.Recipient);
+
+                if (_warningsThisWindow == MaxWarningsPerWindow)
+                    logger.LogWarning(
+                        "Dead-letter warning limit ({Limit}) reached — suppressing per-letter warnings for {Window}",
+                        MaxWarningsPerWindow, WarningWindow);
+            }
+            else
+            {
+                _suppressedThisWindow++;
+            }
+        });
+
+        Receive<WindowTick>(_ =>
+        {
+            if (_suppressedThisWindow > 0)
+                logger.LogWarning(
+                    "Suppressed {Count} dead-letter warnings in the last {Window}",
+                    _suppressedThisWindow, WarningWindow);
+
+            _warningsThisWindow = 0;
+            _suppressedThisWindow = 0;
        });

        Receive<GetDeadLetterCount>(_ => Sender.Tell(new DeadLetterCountResponse(_deadLetterCount)));
@@ -41,6 +86,7 @@ public class DeadLetterMonitorActor : ReceiveActor
    protected override void PreStart()
    {
        Context.System.EventStream.Subscribe(Self, typeof(DeadLetter));
+        Timers.StartPeriodicTimer("dl-window", WindowTick.Instance, WarningWindow);
    }

    /// <inheritdoc />
@@ -48,6 +94,13 @@ public class DeadLetterMonitorActor : ReceiveActor
    {
        Context.System.EventStream.Unsubscribe(Self, typeof(DeadLetter));
    }
+
+    /// <summary>Periodic self-message that rolls the warning-rate-limit window.</summary>
+    private sealed class WindowTick
+    {
+        public static readonly WindowTick Instance = new();
+        private WindowTick() { }
+    }
 }

 /// <summary>
@@ -6,13 +6,17 @@ using ZB.MOM.WW.ScadaBridge.Host.Actors;
 namespace ZB.MOM.WW.ScadaBridge.Host;

 /// <summary>
-/// Sends SiteHealthReport to the local SiteCommunicationActor via Akka ActorSelection.
-/// The SiteCommunicationActor forwards it to central.
+/// Sends SiteHealthReport to the local SiteCommunicationActor via Akka ActorSelection,
+/// which forwards it to central. The send is an acked round-trip (review 01 [Medium]):
+/// it Asks and awaits the central <see cref="SiteHealthReportAck"/>, so a lost or
+/// rejected report faults the sender's task (driving its counter-restore path).
 /// </summary>
 public class AkkaHealthReportTransport : IHealthReportTransport
 {
    private readonly AkkaHostedService _akkaService;

+    private static readonly TimeSpan AckTimeout = TimeSpan.FromSeconds(10);
+
    /// <summary>
    /// Initializes a new <see cref="AkkaHealthReportTransport"/> backed by the given Akka hosted service.
    /// </summary>
@@ -23,12 +27,15 @@ public class AkkaHealthReportTransport : IHealthReportTransport
    }

    /// <inheritdoc />
-    public void Send(SiteHealthReport report)
+    public async Task SendAsync(SiteHealthReport report, CancellationToken cancellationToken)
    {
-        var actorSystem = _akkaService.ActorSystem;
-        if (actorSystem == null) return;
+        var actorSystem = _akkaService.ActorSystem
+            ?? throw new InvalidOperationException("Actor system not started — health report not sent");

        var siteComm = actorSystem.ActorSelection("/user/site-communication");
-        siteComm.Tell(report, ActorRefs.NoSender);
+        var ack = await siteComm.Ask<SiteHealthReportAck>(report, AckTimeout, cancellationToken)
+            .ConfigureAwait(false);
+        if (!ack.Accepted)
+            throw new InvalidOperationException($"Health report #{report.SequenceNumber} rejected: {ack.Error}");
    }
 }
@@ -9,9 +9,12 @@ namespace ZB.MOM.WW.ScadaBridge.Host.Health;
 /// <see cref="IActiveNodeGate"/> backed by the running Akka.NET cluster.
 ///
 /// The inbound API is "Central cluster only (active node)" — a standby central
-/// node must not execute method scripts or <c>Route.To()</c> calls. This gate
-/// mirrors the leadership check in <see cref="ActiveNodeHealthCheck"/> (the
-/// node is the cluster leader, <see cref="MemberStatus.Up"/>), so
+/// node must not execute method scripts or <c>Route.To()</c> calls. Active =
+/// the oldest Up member (the singleton host), evaluated by
+/// <see cref="ClusterActivityEvaluator.SelfIsOldest(Cluster, string?)"/> — NOT
+/// the cluster leader, whose address-ordered definition diverges from singleton
+/// placement after a restart (review 01 [High]). This is the same evaluator
+/// backing <see cref="OldestNodeActiveHealthCheck"/>, so
 /// <see cref="InboundApiEndpointFilter"/> can return HTTP 503 on a standby.
 ///
 /// Registered only in the Central-role branch of <c>Program.cs</c>. The gate
@@ -42,12 +45,7 @@ public sealed class ActiveNodeGate : IActiveNodeGate
                return false;

            var cluster = Cluster.Get(system);
-            var self = cluster.SelfMember;
-            if (self.Status != MemberStatus.Up)
-                return false;
-
-            var leader = cluster.State.Leader;
-            return leader != null && leader == self.Address;
+            return ClusterActivityEvaluator.SelfIsOldest(cluster);
        }
    }
 }
@@ -33,9 +33,7 @@ public class AkkaClusterNodeProvider : IClusterNodeProvider
            var system = _akkaService.ActorSystem;
            if (system == null) return false;
            var cluster = Cluster.Get(system);
-            if (cluster.SelfMember.Status != MemberStatus.Up) return false;
-            var leader = cluster.State.Leader;
-            return leader != null && leader.Equals(cluster.SelfAddress);
+            return ClusterActivityEvaluator.SelfIsOldest(cluster, _siteRole);
        }
    }

@@ -46,8 +44,7 @@ public class AkkaClusterNodeProvider : IClusterNodeProvider
        if (system == null) return [];

        var cluster = Cluster.Get(system);
-        var selfAddress = cluster.SelfAddress;
-        var leader = cluster.State.Leader;
+        var oldest = ClusterActivityEvaluator.OldestUpMember(cluster, _siteRole);

        var nodes = new List<NodeStatus>();
        foreach (var member in cluster.State.Members)
@@ -57,8 +54,8 @@ public class AkkaClusterNodeProvider : IClusterNodeProvider

            var hostname = member.Address.Host ?? member.Address.ToString();
            var isOnline = member.Status == MemberStatus.Up;
-            var isLeader = member.Address.Equals(leader);
-            var role = isLeader ? "Primary" : "Standby";
+            var isPrimary = oldest != null && member.UniqueAddress.Equals(oldest.UniqueAddress);
+            var role = isPrimary ? "Primary" : "Standby";

            nodes.Add(new NodeStatus(hostname, isOnline, role));
        }
@@ -0,0 +1,42 @@
+using Akka.Cluster;
+
+namespace ZB.MOM.WW.ScadaBridge.Host.Health;
+
+/// <summary>
+/// THE single definition of "active node" (review 01 [High]): a node is active
+/// when it is the OLDEST Up member (optionally within a role scope) — i.e. the
+/// member the ClusterSingletonManager places singletons on. Cluster LEADERSHIP
+/// (lowest address) is an Akka-internal concept that diverges from singleton
+/// placement permanently once the original first node restarts and rejoins;
+/// every product-level active/standby decision must use this evaluator, never
+/// cluster.State.Leader.
+/// </summary>
+public static class ClusterActivityEvaluator
+{
+    /// <summary>True when self is Up and no other Up member (in the role scope) is older.</summary>
+    public static bool SelfIsOldest(Cluster cluster, string? role = null)
+    {
+        var self = cluster.SelfMember;
+        if (self.Status != MemberStatus.Up)
+            return false;
+        if (role != null && !self.HasRole(role))
+            return false;
+
+        return cluster.State.Members
+            .Where(m => m.Status == MemberStatus.Up)
+            .Where(m => role == null || m.HasRole(role))
+            .All(m => m.UniqueAddress.Equals(self.UniqueAddress) || self.IsOlderThan(m));
+    }
+
+    /// <summary>The oldest Up member in the role scope, or null while none is Up. Used for Primary/Standby labelling.</summary>
+    public static Member? OldestUpMember(Cluster cluster, string? role = null)
+    {
+        Member? oldest = null;
+        foreach (var m in cluster.State.Members.Where(m => m.Status == MemberStatus.Up))
+        {
+            if (role != null && !m.HasRole(role)) continue;
+            if (oldest == null || m.IsOlderThan(oldest)) oldest = m;
+        }
+        return oldest;
+    }
+}
@@ -0,0 +1,31 @@
+using Akka.Actor;
+using Microsoft.Extensions.Diagnostics.HealthChecks;
+
+namespace ZB.MOM.WW.ScadaBridge.Host.Health;
+
+/// <summary>
+/// /health/active check: Healthy only on the oldest Up member — the node that
+/// hosts the cluster singletons. Replaces the shared package's leader-based
+/// ActiveNodeHealthCheck (review 01 [High]): leadership is address-ordered and
+/// diverges from singleton placement after a restart, and during a partition
+/// BOTH nodes compute themselves leader, so Traefik served both. Oldest-member
+/// semantics keep exactly one active node through a partition (the non-oldest
+/// side still sees the old oldest member until SBR resolves membership).
+/// </summary>
+public sealed class OldestNodeActiveHealthCheck : IHealthCheck
+{
+    private readonly ActorSystem _system;
+
+    /// <summary>Initializes a new <see cref="OldestNodeActiveHealthCheck"/> bound to the running actor system.</summary>
+    /// <param name="system">The live cluster actor system.</param>
+    public OldestNodeActiveHealthCheck(ActorSystem system) => _system = system;
+
+    /// <inheritdoc />
+    public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken ct = default)
+    {
+        var cluster = Akka.Cluster.Cluster.Get(_system);
+        return Task.FromResult(ClusterActivityEvaluator.SelfIsOldest(cluster)
+            ? HealthCheckResult.Healthy("oldest Up member (singleton host)")
+            : HealthCheckResult.Unhealthy("not the oldest Up member (standby)"));
+    }
+}
@@ -210,15 +210,19 @@ try
        builder.Services.AddConfigurationDatabase(configDbConnectionString);

        // Health checks for readiness gating — shared ZB.MOM.WW.Health probes.
-        // Check names and the ready/active tier split are preserved: database + akka-cluster
-        // carry the Ready tag (/health/ready), active-node carries the Active tag (/health/active).
-        // The Akka checks resolve ActorSystem from DI via the transient bridge registered below;
+        // Ready tier (/health/ready): database + akka-cluster + required-singletons.
+        // Active tier (/health/active): active-node = oldest Up member (singleton host) AND
+        // database reachable (review 01 [High]) — a DB-dead active node drops from Traefik
+        // rotation, and the active-node check is the host-local OldestNodeActiveHealthCheck,
+        // NOT the shared leader-based ActiveNodeHealthCheck (leadership diverges from singleton
+        // placement after a restart and both sides claim leadership during a partition).
+        // The Akka checks resolve ActorSystem from DI via the singleton bridge registered below;
        // the DatabaseHealthCheck<TContext> resolves a scoped ScadaBridgeDbContext (no factory).
        builder.Services.AddHealthChecks()
            .AddTypeActivatedCheck<DatabaseHealthCheck<ScadaBridgeDbContext>>(
                "database",
                failureStatus: null,
-                tags: new[] { ZbHealthTags.Ready })
+                tags: new[] { ZbHealthTags.Ready, ZbHealthTags.Active })
            .AddTypeActivatedCheck<AkkaClusterHealthCheck>(
                "akka-cluster",
                failureStatus: null,
@@ -236,7 +240,9 @@ try
                "required-singletons",
                failureStatus: null,
                tags: new[] { ZbHealthTags.Ready })
-            .AddTypeActivatedCheck<ActiveNodeHealthCheck>(
+            // Host-local oldest-member check (review 01 [High]) — see OldestNodeActiveHealthCheck.
+            // Resolves ActorSystem from DI, like the akka-cluster check above.
+            .AddTypeActivatedCheck<OldestNodeActiveHealthCheck>(
                "active-node",
                failureStatus: null,
                tags: new[] { ZbHealthTags.Active });
@@ -261,9 +267,9 @@ try
        // standby-node gating is actually enforced (the InboundApiEndpointFilter
        // consults IActiveNodeGate and defaults to "allow" when none is registered,
        // which leaves the design's "central cluster only (active node)" guarantee
-        // unenforced in deployed binaries). The gate is backed by the same Akka
-        // cluster-leadership check as ActiveNodeHealthCheck above, so the inbound
-        // API and the /health/active endpoint Traefik routes against agree on
+        // unenforced in deployed binaries). The gate is backed by the same oldest-member
+        // evaluator (ClusterActivityEvaluator) as OldestNodeActiveHealthCheck above, so the
+        // inbound API and the /health/active endpoint Traefik routes against agree on
        // which node is active.
        builder.Services.AddSingleton<ZB.MOM.WW.ScadaBridge.InboundAPI.IActiveNodeGate, ActiveNodeGate>();

@@ -352,12 +358,13 @@ try
            branch => branch.UseAuditWriteMiddleware());

        // Map the canonical three-tier health endpoints in one call:
-        //   /health/ready  — Ready-tagged checks (database + akka-cluster). REQ-HOST-4a defines
-        //                    readiness as cluster membership + DB connectivity, explicitly NOT
-        //                    cluster leadership, so the leader-only active-node check is excluded
+        //   /health/ready  — Ready-tagged checks (database + akka-cluster + required-singletons).
+        //                    REQ-HOST-4a defines readiness as cluster membership + DB connectivity,
+        //                    explicitly NOT active-node status, so the active-node check is excluded
        //                    (a fully operational standby central node still reports ready).
-        //   /health/active — Active-tagged check (active-node); returns 200 only on the cluster
-        //                    leader; used by Traefik for routing.
+        //   /health/active — Active-tagged checks (active-node = oldest Up member + database
+        //                    reachable); returns 200 only on the singleton-host node with a live
+        //                    DB; used by Traefik for routing.
        //   /healthz       — bare process liveness; runs no checks (always 200 while the process
        //                    is up). New tier added by adopting the shared library.
        // All three are anonymous and use the canonical ZbHealthWriter JSON output.
@@ -109,8 +109,8 @@ public static class SiteServiceRegistration
        // The EventLogPurgeService runs on every
        // site host node but consults this optional gate each tick and early-exits on
        // the standby. Register it to delegate to IClusterNodeProvider.SelfIsPrimary
-        // (the canonical "this node is Up AND cluster leader" check) so purge runs ONLY
-        // on the active node — no duplicated cluster logic. Non-clustered test hosts that
+        // (the canonical "this node is the oldest Up member (singleton host)" check) so purge
+        // runs ONLY on the active node — no duplicated cluster logic. Non-clustered test hosts that
        // never call SiteServiceRegistration leave it unregistered, so the purge defaults
        // to always-run (the pre-fix behaviour, preserved).
        services.AddSingleton<SiteEventLogActiveNodeCheck>(sp =>
@@ -124,6 +124,14 @@ public static class StartupValidator
                        _ => SeedNodePort(seed) != grpcPort,
                        $"entry '{seed}' must not target the gRPC port " +
                        $"({grpcPort}); seed nodes must reference Akka remoting ports");
+
+                    // Same failure mode as the gRPC guard: the Kestrel HTTP/1.1
+                    // metrics listener is not an Akka.Remote endpoint, so a seed on
+                    // it dials a doomed association forever.
+                    p.Require("ScadaBridge:Cluster:SeedNodes",
+                        _ => SeedNodePort(seed) != metricsPort,
+                        $"entry '{seed}' must not target the metrics port " +
+                        $"({metricsPort}); seed nodes must reference Akka remoting ports");
                }
            })
            .ThrowIfInvalid();
@@ -12,10 +12,13 @@
  </ItemGroup>

  <ItemGroup>
-    <PackageReference Include="Akka.Cluster.Hosting" />
+    <!-- Bootstrap is a hand-assembled, injection-safe HOCON document
+         (AkkaHostedService.BuildHocon) + ActorSystem.Create, NOT Akka.Hosting's
+         AkkaConfigurationBuilder (REQ-HOST-6). The three Akka.*.Hosting packages
+         were referenced but unused; dropped in arch-review 01 Task 22. Akka.Cluster.Tools
+         supplies ClusterSingleton/ClusterClient and (transitively) Akka.Cluster,
+         which carries the SBR provider named in HOCON. -->
    <PackageReference Include="Akka.Cluster.Tools" />
-    <PackageReference Include="Akka.Hosting" />
-    <PackageReference Include="Akka.Remote.Hosting" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Design">
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
      <PrivateAssets>all</PrivateAssets>
@@ -25,9 +28,6 @@
    <PackageReference Include="Serilog.AspNetCore" />
    <PackageReference Include="Serilog.Sinks.Console" />
    <PackageReference Include="Serilog.Sinks.File" />
-    <!-- Transitive override: Akka.Hosting 1.5.62 pins OpenTelemetry.Api 1.9.0 which is flagged
-         (GHSA-g94r-2vxg-569j, GHSA-8785-wc3w-h8q6). Bumping directly clears both advisories. -->
-    <PackageReference Include="OpenTelemetry.Api" />
    <PackageReference Include="ZB.MOM.WW.Configuration" />
    <PackageReference Include="ZB.MOM.WW.Health" />
    <PackageReference Include="ZB.MOM.WW.Health.Akka" />
@@ -11,9 +11,10 @@
      "NodeName": "node-a"
    },
    "Cluster": {
+      "_seedNodes": "Host-0xx: second entry is the FUTURE node-b remoting port (8085) for a two-node localhost site. It must be an Akka remoting endpoint — never this node's GrpcPort (8083) or MetricsPort (8084); StartupValidator rejects both.",
      "SeedNodes": [
        "akka.tcp://scadabridge@localhost:8082",
-        "akka.tcp://scadabridge@localhost:8084"
+        "akka.tcp://scadabridge@localhost:8085"
      ],
      "SplitBrainResolverStrategy": "keep-oldest",
      "StableAfter": "00:00:15",
@@ -82,6 +82,11 @@ public class HealthPageTests : BunitContext
            .Returns(Task.FromResult<IReadOnlyList<Site>>(new List<Site>()));
        Services.AddSingleton(siteRepo);

+        // PLAN-01 Task 14 — the Health page now shows the metrics-stale badge with
+        // a tooltip naming the configured MetricsStaleTimeout, so it injects the
+        // health-monitoring options.
+        Services.AddSingleton(Options.Create(new HealthMonitoringOptions()));
+
        // Audit Log (#23) M7 Bundle E — the Health page now also fetches the
        // Audit KPI snapshot. Stub it with an empty point-in-time reading so
        // the existing assertions (Notification Outbox tiles, Online/Offline
@@ -275,6 +280,62 @@ public class HealthPageTests : BunitContext
        });
    }

+    [Fact]
+    public void Renders_MetricsStaleBadge_ForOnlineButStaleSite()
+    {
+        // PLAN-01 Task 14: a site that is online (heartbeating) but whose report
+        // pipeline died shows a distinct "Metrics stale" warning badge.
+        SeedStates(new SiteHealthState
+        {
+            SiteId = "site-a",
+            IsOnline = true,
+            IsMetricsStale = true,
+            LastHeartbeatAt = DateTimeOffset.UtcNow,
+            LastReportReceivedAt = DateTimeOffset.UtcNow.AddMinutes(-5),
+        });
+
+        var cut = Render<HealthPage>();
+
+        cut.WaitForAssertion(() =>
+        {
+            Assert.Contains("Metrics stale", cut.Markup);
+            // The badge uses the page's Bootstrap warning style.
+            Assert.Contains("bg-warning", cut.Markup);
+        });
+    }
+
+    [Fact]
+    public void Renders_OfflineSince_ForOfflineSiteWithStatusChange()
+    {
+        var changedAt = new DateTimeOffset(2026, 7, 8, 12, 34, 56, TimeSpan.Zero);
+        SeedStates(new SiteHealthState
+        {
+            SiteId = "site-a",
+            IsOnline = false,
+            LastHeartbeatAt = changedAt,
+            LastStatusChangeAt = changedAt,
+        });
+
+        var cut = Render<HealthPage>();
+
+        cut.WaitForAssertion(() =>
+        {
+            Assert.Contains("offline since", cut.Markup);
+            // The "u" (universal sortable) format the page renders the flip with.
+            Assert.Contains(changedAt.ToString("u"), cut.Markup);
+        });
+    }
+
+    // Re-seeds the aggregator substitute with fully-specified states (as opposed
+    // to SeedSites' minimal online placeholders).
+    private void SeedStates(params SiteHealthState[] states)
+    {
+        var aggregator = Substitute.For<ICentralHealthAggregator>();
+        aggregator.GetAllSiteStates()
+            .Returns(states.ToDictionary(s => s.SiteId));
+        Services.AddSingleton(aggregator);
+    }
+
    // Re-seeds the aggregator substitute so the trend panel's site selector has
    // options. Each site id maps to a minimal online SiteHealthState (a null
    // report is fine — the trend panel keys off the site ids, not the report).
@@ -69,6 +69,47 @@ public class ClusterOptionsValidatorTests
        Assert.Contains("SeedNodes", result.FailureMessage);
    }

+    [Fact]
+    public void SingleSeed_WithoutAcknowledgement_Fails()
+    {
+        var options = ValidOptions();
+        options.SeedNodes = new List<string> { "akka.tcp://scadabridge@h:8081" };
+
+        var result = new ClusterOptionsValidator().Validate(null, options);
+
+        Assert.True(result.Failed);
+        Assert.Contains("AllowSingleNodeCluster", result.FailureMessage);
+    }
+
+    [Fact]
+    public void SingleSeed_WithAllowSingleNodeCluster_Passes()
+    {
+        // Review 01 [Low]: the shipped single-node artifact satisfied the 2-seed
+        // rule with a phantom seed the node dials forever. An explicit
+        // acknowledgement flag replaces the fiction.
+        var options = ValidOptions();
+        options.SeedNodes = new List<string> { "akka.tcp://scadabridge@h:8081" };
+        options.AllowSingleNodeCluster = true;
+
+        var result = new ClusterOptionsValidator().Validate(null, options);
+
+        Assert.True(result.Succeeded, result.FailureMessage);
+    }
+
+    [Fact]
+    public void EmptySeeds_FailsEvenWithFlag()
+    {
+        // Zero seeds is always invalid — the flag lowers the floor to 1, not 0.
+        var options = ValidOptions();
+        options.SeedNodes = new List<string>();
+        options.AllowSingleNodeCluster = true;
+
+        var result = new ClusterOptionsValidator().Validate(null, options);
+
+        Assert.True(result.Failed);
+        Assert.Contains("SeedNodes", result.FailureMessage);
+    }
+
    [Fact]
    public void SingleSeedNode_FailsValidation()
    {
@@ -10,6 +10,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites;
 using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
 using ZB.MOM.WW.ScadaBridge.Communication;
 using ZB.MOM.WW.ScadaBridge.Communication.Actors;
+using ZB.MOM.WW.ScadaBridge.HealthMonitoring;

 namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;

@@ -37,7 +38,37 @@ public class CentralCommunicationActorClientLifecycleTests : TestKit

    private static SiteAddressCacheLoaded Load(string siteId, params string[] addrs) =>
        new(new Dictionary<string, IReadOnlyList<string>>
-            { [siteId] = addrs.ToList().AsReadOnly() });
+            { [siteId] = addrs.ToList().AsReadOnly() },
+            new[] { siteId });
+
+    [Fact]
+    public void PeriodicRefresh_PrunesDeletedSites_FromHealthAggregator()
+    {
+        // PLAN-01 Task 18: the periodic site refresh feeds the currently-configured
+        // site ids to the aggregator so a deleted site is evicted within one
+        // refresh interval. Repo returns only "site-a".
+        var siteRepo = Substitute.For<ISiteRepository>();
+        siteRepo.GetAllSitesAsync(Arg.Any<CancellationToken>())
+            .Returns(new List<Site> { new("Site A", "site-a") });
+        var aggregator = Substitute.For<ICentralHealthAggregator>();
+        var services = new ServiceCollection();
+        services.AddScoped(_ => siteRepo);
+        services.AddSingleton(aggregator);
+        var provider = services.BuildServiceProvider();
+
+        var actor = Sys.ActorOf(Props.Create(() => new CentralCommunicationActor(
+            provider, new DefaultSiteClientFactory(), null)));
+
+        // Trigger the refresh (also fires at PreStart, but drive it explicitly so
+        // the assertion is deterministic). The load runs on a detached task and
+        // pipes SiteAddressCacheLoaded back to the actor, which then prunes.
+        actor.Tell(new RefreshSiteAddresses());
+
+        AwaitAssert(
+            () => aggregator.Received().PruneUnknownSites(
+                Arg.Is<IReadOnlyCollection<string>>(ids => ids.Contains("site-a"))),
+            TimeSpan.FromSeconds(3));
+    }

    [Fact]
    public void AddressEdit_RecreatesClient_WithoutRestartLoop()
@@ -0,0 +1,66 @@
+using Akka.Actor;
+using Akka.TestKit.Xunit2;
+using Microsoft.Extensions.DependencyInjection;
+using NSubstitute;
+using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites;
+using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
+using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
+using ZB.MOM.WW.ScadaBridge.Commons.Types;
+using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
+using ZB.MOM.WW.ScadaBridge.Communication.Actors;
+using ZB.MOM.WW.ScadaBridge.HealthMonitoring;
+
+namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;
+
+/// <summary>
+/// Review 01 [Medium]: site→central health reports were fire-and-forget, so a lost
+/// report was invisible to the sender. These tests pin the new end-to-end ack:
+/// the site actor replies not-accepted when it has no central ClusterClient, and the
+/// central actor processes + acks a report it receives.
+/// </summary>
+public class HealthReportAckTests : TestKit
+{
+    private readonly CommunicationOptions _options = new();
+
+    private static SiteHealthReport SampleReport(long seq, string siteId = "site-a") =>
+        new(siteId, seq, DateTimeOffset.UtcNow,
+            new Dictionary<string, ConnectionHealth>(),
+            new Dictionary<string, TagResolutionStatus>(),
+            0, 0, new Dictionary<string, int>(), 0, 0, 0, 0);
+
+    [Fact]
+    public void SiteCommunicationActor_NoCentralClient_RepliesNotAccepted()
+    {
+        var dmProbe = CreateTestProbe();
+        var siteComm = Sys.ActorOf(Props.Create(() =>
+            new SiteCommunicationActor("site-a", _options, dmProbe.Ref)));
+        // No RegisterCentralClient sent => _centralClient is null.
+        siteComm.Tell(SampleReport(seq: 7));
+        var ack = ExpectMsg<SiteHealthReportAck>();
+        Assert.False(ack.Accepted);
+        Assert.Equal(7, ack.SequenceNumber);
+        Assert.Equal("site-a", ack.SiteId);
+    }
+
+    [Fact]
+    public void CentralCommunicationActor_OnReport_ProcessesAndAcks()
+    {
+        var mockRepo = Substitute.For<ISiteRepository>();
+        mockRepo.GetAllSitesAsync(Arg.Any<CancellationToken>()).Returns(new List<Site>());
+        var aggregator = Substitute.For<ICentralHealthAggregator>();
+
+        var services = new ServiceCollection();
+        services.AddScoped(_ => mockRepo);
+        services.AddSingleton(aggregator);
+        var sp = services.BuildServiceProvider();
+
+        var siteClientFactory = Substitute.For<ISiteClientFactory>();
+        var actor = Sys.ActorOf(Props.Create(() => new CentralCommunicationActor(sp, siteClientFactory)));
+
+        actor.Tell(SampleReport(seq: 3));
+        var ack = ExpectMsg<SiteHealthReportAck>();
+        Assert.True(ack.Accepted);
+        Assert.Equal(3, ack.SequenceNumber);
+        AwaitAssert(() => aggregator.Received().ProcessReport(Arg.Is<SiteHealthReport>(r => r.SequenceNumber == 3)));
+    }
+}
@@ -40,6 +40,23 @@ public class CentralHealthAggregatorTests
            _timeProvider);
    }

+    /// <summary>
+    /// Builds a fresh aggregator + manual time provider with an explicit
+    /// <paramref name="metricsStaleTimeout"/> so the metrics-staleness tests can
+    /// pick a window independent of the shared 60s <c>OfflineTimeout</c>.
+    /// </summary>
+    private static (CentralHealthAggregator Aggregator, TestTimeProvider Time) NewAggregator(
+        TimeSpan? metricsStaleTimeout = null)
+    {
+        var time = new TestTimeProvider(DateTimeOffset.UtcNow);
+        var opts = new HealthMonitoringOptions { OfflineTimeout = TimeSpan.FromSeconds(60) };
+        if (metricsStaleTimeout is { } mst)
+            opts.MetricsStaleTimeout = mst;
+        var aggregator = new CentralHealthAggregator(
+            Options.Create(opts), NullLogger<CentralHealthAggregator>.Instance, time);
+        return (aggregator, time);
+    }
+
    private static SiteHealthReport MakeReport(string siteId, long seq) =>
        new(
            SiteId: siteId,
@@ -423,4 +440,77 @@ public class CentralHealthAggregatorTests
        _aggregator.ProcessReport(MakeReport("site-1", 11));
        Assert.Equal(11, _aggregator.GetSiteState("site-1")!.LastSequenceNumber);
    }
+
+    /// <summary>
+    /// Review 01 [Medium]: a site whose HealthReportSender died keeps
+    /// heartbeating and previously showed "online with frozen metrics forever".
+    /// Metrics staleness is a distinct signal from liveness: heartbeats keep the
+    /// site online, but the absence of a full report for
+    /// <see cref="HealthMonitoringOptions.MetricsStaleTimeout"/> flags it stale.
+    /// </summary>
+    [Fact]
+    public void HeartbeatingSiteWithStaleReports_IsFlaggedMetricsStale_ButStaysOnline()
+    {
+        var (aggregator, time) = NewAggregator(metricsStaleTimeout: TimeSpan.FromMinutes(2));
+        aggregator.ProcessReport(MakeReport("site-a", 1));
+        time.Advance(TimeSpan.FromMinutes(3));
+        aggregator.MarkHeartbeat("site-a", time.GetUtcNow()); // heartbeats keep flowing
+        aggregator.CheckForOfflineSites();
+        var state = aggregator.GetSiteState("site-a")!;
+        Assert.True(state.IsOnline);        // heartbeat-based liveness unchanged
+        Assert.True(state.IsMetricsStale);  // NEW distinct signal
+    }
+
+    /// <summary>A fresh full report clears the metrics-stale flag.</summary>
+    [Fact]
+    public void FreshReport_ClearsMetricsStale()
+    {
+        var (aggregator, time) = NewAggregator(metricsStaleTimeout: TimeSpan.FromMinutes(2));
+        aggregator.ProcessReport(MakeReport("site-a", 1));
+        time.Advance(TimeSpan.FromMinutes(3));
+        aggregator.CheckForOfflineSites();
+        aggregator.ProcessReport(MakeReport("site-a", 2));
+        Assert.False(aggregator.GetSiteState("site-a")!.IsMetricsStale);
+    }
+
+    /// <summary>
+    /// Review 01 [Low]: <c>_siteStates</c> only ever grew — a site deleted from
+    /// configuration remained a permanently-offline dashboard tile (and a KPI
+    /// sample source) forever. Pruning against the known-site set evicts it; the
+    /// synthetic central id is always retained.
+    /// </summary>
+    [Fact]
+    public void PruneUnknownSites_RemovesDeletedSites_KeepsKnownAndCentral()
+    {
+        var (aggregator, _) = NewAggregator();
+        aggregator.ProcessReport(MakeReport("site-a", 1));
+        aggregator.ProcessReport(MakeReport("site-b", 1));
+        aggregator.ProcessReport(MakeReport(CentralHealthReportLoop.CentralSiteId, 1));
+
+        aggregator.PruneUnknownSites(new[] { "site-a" });
+
+        Assert.NotNull(aggregator.GetSiteState("site-a"));
+        Assert.Null(aggregator.GetSiteState("site-b"));
+        Assert.NotNull(aggregator.GetSiteState(CentralHealthReportLoop.CentralSiteId)); // synthetic id always kept
+    }
+
+    /// <summary>
+    /// Review 01 underdeveloped #7: "when did the site drop" was unanswerable.
+    /// Every online↔offline flip stamps <see cref="SiteHealthState.LastStatusChangeAt"/>.
+    /// </summary>
+    [Fact]
+    public void OnlineOfflineTransitions_RecordLastStatusChangeAt()
+    {
+        var (aggregator, time) = NewAggregator();
+        aggregator.ProcessReport(MakeReport("site-a", 1));
+        time.Advance(TimeSpan.FromMinutes(5)); // > OfflineTimeout
+        aggregator.CheckForOfflineSites();
+        var offlineAt = aggregator.GetSiteState("site-a")!.LastStatusChangeAt;
+        Assert.Equal(time.GetUtcNow(), offlineAt);
+        // Advance so the recovery flip lands at a demonstrably later instant than
+        // the offline flip — proves LastStatusChangeAt is re-stamped on each flip.
+        time.Advance(TimeSpan.FromSeconds(10));
+        aggregator.MarkHeartbeat("site-a", time.GetUtcNow());
+        Assert.NotEqual(offlineAt, aggregator.GetSiteState("site-a")!.LastStatusChangeAt);
+    }
 }
@@ -27,6 +27,7 @@ public class CentralHealthReportLoopTests
        public IReadOnlyDictionary<string, SiteHealthState> GetAllSiteStates() =>
            new Dictionary<string, SiteHealthState>();
        public SiteHealthState? GetSiteState(string siteId) => null;
+        public void PruneUnknownSites(IReadOnlyCollection<string> knownSiteIds) { }
    }

    /// <summary>
@@ -256,6 +257,7 @@ public class CentralHealthReportLoopTests
        public IReadOnlyDictionary<string, SiteHealthState> GetAllSiteStates() =>
            new Dictionary<string, SiteHealthState>();
        public SiteHealthState? GetSiteState(string siteId) => null;
+        public void PruneUnknownSites(IReadOnlyCollection<string> knownSiteIds) { }
    }

    [Fact]
@@ -70,4 +70,27 @@ public class HealthMonitoringOptionsValidatorTests
        Assert.True(result.Failed);
        Assert.Contains("CentralOfflineTimeout", result.FailureMessage);
    }
+
+    [Fact]
+    public void ZeroMetricsStaleTimeout_IsRejected()
+    {
+        var result = Validate(new HealthMonitoringOptions { MetricsStaleTimeout = TimeSpan.Zero });
+
+        Assert.True(result.Failed);
+        Assert.Contains("MetricsStaleTimeout", result.FailureMessage);
+    }
+
+    [Fact]
+    public void MetricsStaleTimeout_ShorterThanReportInterval_IsRejected()
+    {
+        // A stale window shorter than one report interval flags every site.
+        var result = Validate(new HealthMonitoringOptions
+        {
+            ReportInterval = TimeSpan.FromSeconds(30),
+            MetricsStaleTimeout = TimeSpan.FromSeconds(10)
+        });
+
+        Assert.True(result.Failed);
+        Assert.Contains("MetricsStaleTimeout", result.FailureMessage);
+    }
 }
@@ -13,7 +13,11 @@ public class HealthReportSenderTests
    private class FakeTransport : IHealthReportTransport
    {
        public List<SiteHealthReport> SentReports { get; } = [];
-        public void Send(SiteHealthReport report) => SentReports.Add(report);
+        public Task SendAsync(SiteHealthReport report, CancellationToken cancellationToken)
+        {
+            SentReports.Add(report);
+            return Task.CompletedTask;
+        }
    }

    private class FakeSiteIdentityProvider : ISiteIdentityProvider
@@ -387,7 +391,7 @@ public class HealthReportSenderTests
        private int _callCount;
        public List<SiteHealthReport> SentReports { get; } = [];

-        public void Send(SiteHealthReport report)
+        public Task SendAsync(SiteHealthReport report, CancellationToken cancellationToken)
        {
            var n = Interlocked.Increment(ref _callCount);
            if (n == 1)
@@ -395,6 +399,7 @@ public class HealthReportSenderTests
                throw new InvalidOperationException("transport temporarily unavailable");
            }
            SentReports.Add(report);
+            return Task.CompletedTask;
        }
    }

@@ -166,5 +166,8 @@ public class SiteHealthKpiSampleSourceTests

        public void MarkHeartbeat(string siteId, DateTimeOffset receivedAt) =>
            throw new NotSupportedException();
+
+        public void PruneUnknownSites(IReadOnlyCollection<string> knownSiteIds) =>
+            throw new NotSupportedException();
    }
 }
@@ -0,0 +1,104 @@
+using Akka.Actor;
+using Microsoft.Extensions.DependencyInjection;
+using Microsoft.Extensions.Logging.Abstractions;
+using Microsoft.Extensions.Options;
+using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
+using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
+using ZB.MOM.WW.ScadaBridge.Commons.Types;
+using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
+using ZB.MOM.WW.ScadaBridge.Communication;
+using ZB.MOM.WW.ScadaBridge.Host.Actors;
+
+namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
+
+/// <summary>
+/// Review 01 [Medium]: the health-report transport is now an ACKED round-trip.
+/// SendAsync completes when central acks Accepted:true, throws when the ack is
+/// not-accepted, and throws when no actor answers (Ask timeout / system not
+/// started) — the fault the sender's counter-restore path depends on.
+/// </summary>
+public sealed class AkkaHealthReportTransportTests
+{
+    private sealed class AckActor : ReceiveActor
+    {
+        public AckActor(bool accepted) => Receive<SiteHealthReport>(r =>
+            Sender.Tell(new SiteHealthReportAck(
+                r.SiteId, r.SequenceNumber, accepted, accepted ? null : "rejected by test")));
+    }
+
+    private static SiteHealthReport SampleReport(long seq = 1) =>
+        new("site-a", seq, DateTimeOffset.UtcNow,
+            new Dictionary<string, ConnectionHealth>(),
+            new Dictionary<string, TagResolutionStatus>(),
+            0, 0, new Dictionary<string, int>(), 0, 0, 0, 0);
+
+    private static AkkaHostedService BuildService(out ActorSystem system)
+    {
+        var port = FreePort();
+        var self = $"akka.tcp://scadabridge@127.0.0.1:{port}";
+        var svc = new AkkaHostedService(
+            new ServiceCollection().BuildServiceProvider(),
+            Options.Create(new NodeOptions { Role = "Site", NodeHostname = "127.0.0.1", RemotingPort = port }),
+            Options.Create(new ClusterOptions
+            {
+                SeedNodes = new List<string> { self },
+                SplitBrainResolverStrategy = "keep-oldest",
+                StableAfter = TimeSpan.FromSeconds(3),
+                HeartbeatInterval = TimeSpan.FromMilliseconds(500),
+                FailureDetectionThreshold = TimeSpan.FromSeconds(2),
+                MinNrOfMembers = 1,
+                DownIfAlone = true,
+            }),
+            Options.Create(new CommunicationOptions()),
+            NullLogger<AkkaHostedService>.Instance);
+        system = svc.GetOrCreateActorSystem();
+        return svc;
+    }
+
+    [Fact]
+    public async Task SendAsync_AckedAccepted_Completes()
+    {
+        var svc = BuildService(out var system);
+        try
+        {
+            system.ActorOf(Props.Create(() => new AckActor(true)), "site-communication");
+            var transport = new AkkaHealthReportTransport(svc);
+            await transport.SendAsync(SampleReport(), CancellationToken.None);
+        }
+        finally { await system.Terminate(); }
+    }
+
+    [Fact]
+    public async Task SendAsync_AckedNotAccepted_Throws()
+    {
+        var svc = BuildService(out var system);
+        try
+        {
+            system.ActorOf(Props.Create(() => new AckActor(false)), "site-communication");
+            var transport = new AkkaHealthReportTransport(svc);
+            await Assert.ThrowsAsync<InvalidOperationException>(
+                () => transport.SendAsync(SampleReport(), CancellationToken.None));
+        }
+        finally { await system.Terminate(); }
+    }
+
+    [Fact]
+    public async Task SendAsync_NoResponder_Throws()
+    {
+        var svc = BuildService(out var system);
+        try
+        {
+            // No actor at /user/site-communication => the Ask times out and faults.
+            var transport = new AkkaHealthReportTransport(svc);
+            await Assert.ThrowsAnyAsync<Exception>(
+                () => transport.SendAsync(SampleReport(), CancellationToken.None));
+        }
+        finally { await system.Terminate(); }
+    }
+
+    private static int FreePort()
+    {
+        var l = new System.Net.Sockets.TcpListener(System.Net.IPAddress.Loopback, 0);
+        l.Start(); var p = ((System.Net.IPEndPoint)l.LocalEndpoint).Port; l.Stop(); return p;
+    }
+}
@@ -0,0 +1,60 @@
+using Akka.Actor;
+using Akka.Configuration;
+using Microsoft.Extensions.Logging.Abstractions;
+using ZB.MOM.WW.ScadaBridge.Host.Actors;
+
+namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
+
+public class CentralSingletonRegistrarTests
+{
+    private sealed class EchoActor : ReceiveActor
+    {
+        public EchoActor() => ReceiveAny(m => Sender.Tell(m));
+    }
+
+    [Fact]
+    public async Task Start_CreatesManagerAndProxy_WithCanonicalNames_AndDrainTask()
+    {
+        using var system = CreateSingleNodeClusterSystem();
+        var handle = CentralSingletonRegistrar.Start(
+            system, "test-widget", Props.Create(() => new EchoActor()),
+            NullLogger.Instance, drainTimeout: TimeSpan.FromSeconds(2));
+
+        // Canonical naming preserved: <name>-singleton / <name>-proxy.
+        Assert.EndsWith("/user/test-widget-singleton", handle.Manager.Path.ToString());
+        Assert.EndsWith("/user/test-widget-proxy", handle.Proxy.Path.ToString());
+
+        // Singleton answers through the proxy once the member is Up.
+        var echo = await handle.Proxy.Ask<string>("hi", TimeSpan.FromSeconds(20));
+        Assert.Equal("hi", echo);
+
+        // The drain task registered on PhaseClusterLeave must stop the manager
+        // during coordinated shutdown.
+        await Akka.Actor.CoordinatedShutdown.Get(system).Run(
+            Akka.Actor.CoordinatedShutdown.ClrExitReason.Instance);
+        // Manager terminated => drain ran (GracefulStop completed or PoisonPill fallback).
+        Assert.True(handle.Manager.IsNobody() || system.WhenTerminated.IsCompleted);
+    }
+
+    private static ActorSystem CreateSingleNodeClusterSystem()
+    {
+        var port = FreePort();
+        var config = ConfigurationFactory.ParseString($@"
+akka {{
+    actor.provider = cluster
+    remote.dot-netty.tcp {{ hostname = ""127.0.0.1"", port = {port} }}
+    cluster {{
+        seed-nodes = [""akka.tcp://registrar-test@127.0.0.1:{port}""]
+        roles = [""Central""]
+        min-nr-of-members = 1
+    }}
+}}");
+        return ActorSystem.Create("registrar-test", config);
+    }
+
+    private static int FreePort()
+    {
+        var l = new System.Net.Sockets.TcpListener(System.Net.IPAddress.Loopback, 0);
+        l.Start(); var p = ((System.Net.IPEndPoint)l.LocalEndpoint).Port; l.Stop(); return p;
+    }
+}
@@ -0,0 +1,104 @@
+using Akka.Actor;
+using Akka.Configuration;
+using Microsoft.Extensions.DependencyInjection;
+using Microsoft.Extensions.Logging.Abstractions;
+using Microsoft.Extensions.Options;
+using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
+using ZB.MOM.WW.ScadaBridge.Communication;
+using ZB.MOM.WW.ScadaBridge.Host.Actors;
+using ZB.MOM.WW.ScadaBridge.Host.Health;
+
+namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
+
+public class ClusterActivityEvaluatorTests : IAsyncLifetime
+{
+    private ActorSystem? _system;
+
+    public async Task InitializeAsync()
+    {
+        var port = FreePort();
+        var config = ConfigurationFactory.ParseString($@"
+akka {{
+    actor.provider = cluster
+    remote.dot-netty.tcp {{ hostname = ""127.0.0.1"", port = {port} }}
+    cluster {{
+        seed-nodes = [""akka.tcp://eval-test@127.0.0.1:{port}""]
+        roles = [""Central""]
+        min-nr-of-members = 1
+    }}
+}}");
+        _system = ActorSystem.Create("eval-test", config);
+        var cluster = Akka.Cluster.Cluster.Get(_system);
+        var deadline = DateTime.UtcNow.AddSeconds(20);
+        while (cluster.SelfMember.Status != Akka.Cluster.MemberStatus.Up && DateTime.UtcNow < deadline)
+            await Task.Delay(100);
+    }
+
+    public async Task DisposeAsync() { if (_system != null) await _system.Terminate(); }
+
+    [Fact]
+    public void SelfIsOldest_SoleUpMember_ReturnsTrue()
+    {
+        var cluster = Akka.Cluster.Cluster.Get(_system!);
+        Assert.True(ClusterActivityEvaluator.SelfIsOldest(cluster));
+        Assert.True(ClusterActivityEvaluator.SelfIsOldest(cluster, "Central"));
+    }
+
+    [Fact]
+    public void SelfIsOldest_RoleNotHeld_ReturnsFalse()
+    {
+        var cluster = Akka.Cluster.Cluster.Get(_system!);
+        // Self doesn't carry the role => it can never host that role's singletons.
+        Assert.False(ClusterActivityEvaluator.SelfIsOldest(cluster, "site-nonexistent"));
+    }
+
+    [Fact]
+    public async Task ActiveNodeGate_And_NodeProvider_AgreeWithEvaluator()
+    {
+        // Both product gates must be backed by the oldest-member evaluator.
+        // A single-node Up cluster: evaluator says true, so both must say true.
+        // (Pinning test — the leader==oldest divergence is proven in Task 8's
+        // two-node integration test; here we assert the wiring holds.)
+        var port = FreePort();
+        var self = $"akka.tcp://scadabridge@127.0.0.1:{port}";
+        var akkaService = new AkkaHostedService(
+            new ServiceCollection().BuildServiceProvider(),
+            Options.Create(new NodeOptions { Role = "Central", NodeHostname = "127.0.0.1", RemotingPort = port }),
+            Options.Create(new ClusterOptions
+            {
+                SeedNodes = new List<string> { self },
+                SplitBrainResolverStrategy = "keep-oldest",
+                StableAfter = TimeSpan.FromSeconds(3),
+                HeartbeatInterval = TimeSpan.FromMilliseconds(500),
+                FailureDetectionThreshold = TimeSpan.FromSeconds(2),
+                MinNrOfMembers = 1,
+                DownIfAlone = true,
+            }),
+            Options.Create(new CommunicationOptions()),
+            NullLogger<AkkaHostedService>.Instance);
+
+        var system = akkaService.GetOrCreateActorSystem();
+        try
+        {
+            var cluster = Akka.Cluster.Cluster.Get(system);
+            var deadline = DateTime.UtcNow.AddSeconds(20);
+            while (cluster.SelfMember.Status != Akka.Cluster.MemberStatus.Up && DateTime.UtcNow < deadline)
+                await Task.Delay(100);
+
+            var gate = new ActiveNodeGate(akkaService);
+            var provider = new AkkaClusterNodeProvider(akkaService, "Central");
+            Assert.True(gate.IsActiveNode);
+            Assert.True(provider.SelfIsPrimary);
+        }
+        finally
+        {
+            await system.Terminate();
+        }
+    }
+
+    private static int FreePort()
+    {
+        var l = new System.Net.Sockets.TcpListener(System.Net.IPAddress.Loopback, 0);
+        l.Start(); var p = ((System.Net.IPEndPoint)l.LocalEndpoint).Port; l.Stop(); return p;
+    }
+}
@@ -42,6 +42,43 @@ public class CoordinatedShutdownTests
        Assert.Contains("run-coordinated-shutdown-when-down = on", content);
    }

+    [Fact]
+    public void AllCentralSingletons_RegisterThroughRegistrarWithDrain()
+    {
+        var hostProjectDir = FindHostProjectDirectory();
+        Assert.NotNull(hostProjectDir);
+        var content = File.ReadAllText(Path.Combine(hostProjectDir!, "Actors", "AkkaHostedService.cs"));
+
+        // Review 01 [Medium]: notification-outbox and audit-log-ingest were the
+        // only central singletons WITHOUT a cluster-leave drain task. All seven
+        // now go through CentralSingletonRegistrar.Start, which always adds the
+        // PhaseClusterLeave GracefulStop drain.
+        Assert.Contains("CentralSingletonRegistrar.Start(", content);
+        foreach (var name in new[] { "notification-outbox", "audit-log-ingest", "site-call-audit",
+                                     "audit-log-purge", "site-audit-reconciliation",
+                                     "kpi-history-recorder", "pending-deployment-purge" })
+        {
+            Assert.Contains($"\"{name}\"", content);
+        }
+
+        // No hand-rolled ClusterSingletonManager registrations remain in the
+        // CENTRAL branch. The only two left are the SITE singletons
+        // (deployment-manager, event-log-handler), which stay hand-rolled because
+        // they are role-scoped (.WithRole(siteRole)).
+        Assert.Equal(2, CountOccurrences(content, "ClusterSingletonManager.Props("));
+    }
+
+    private static int CountOccurrences(string haystack, string needle)
+    {
+        int count = 0, i = 0;
+        while ((i = haystack.IndexOf(needle, i, StringComparison.Ordinal)) >= 0)
+        {
+            count++;
+            i += needle.Length;
+        }
+        return count;
+    }
+
    private static string? FindHostProjectDirectory()
    {
        var assemblyDir = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location)!;
@@ -1,8 +1,10 @@
+using System.Linq;
 using Akka.Actor;
 using Akka.Event;
 using Akka.TestKit.Xunit2;
 using Microsoft.Extensions.Logging;
 using Microsoft.Extensions.Logging.Abstractions;
+using NSubstitute;
 using ZB.MOM.WW.ScadaBridge.Host.Actors;

 namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
@@ -48,6 +50,33 @@ public class DeadLetterMonitorTests : TestKit
        });
    }

+    [Fact]
+    public async Task DeadLetterStorm_LogsAtMostWindowLimit_ButCountsAll()
+    {
+        // Review 01 [Low]: a failover/undeploy storm produced one Warning per
+        // dead letter — unbounded log flood. Health metric counting is untouched.
+        var logger = Substitute.For<ILogger<DeadLetterMonitorActor>>();
+        var monitor = Sys.ActorOf(Props.Create(() => new DeadLetterMonitorActor(logger, null)));
+
+        // Ensure the actor has started before the storm.
+        monitor.Tell(GetDeadLetterCount.Instance);
+        ExpectMsg<DeadLetterCountResponse>();
+
+        for (var i = 0; i < 50; i++)
+            monitor.Tell(new DeadLetter($"m{i}", TestActor, TestActor));
+
+        var count = await monitor.Ask<DeadLetterCountResponse>(GetDeadLetterCount.Instance);
+        Assert.Equal(50, count.Count); // every dead letter still counted
+
+        var warnings = logger.ReceivedCalls().Count(c =>
+            c.GetMethodInfo().Name == "Log"
+            && (Microsoft.Extensions.Logging.LogLevel)c.GetArguments()[0]!
+                == Microsoft.Extensions.Logging.LogLevel.Warning);
+        // The per-letter cap plus the one "limit reached" notice.
+        Assert.True(warnings <= DeadLetterMonitorActor.MaxWarningsPerWindow + 1,
+            $"expected <= {DeadLetterMonitorActor.MaxWarningsPerWindow + 1} warnings, got {warnings}");
+    }
+
    [Fact]
    public void DeadLetterMonitor_CountAccumulates()
    {
@@ -5,6 +5,7 @@ using Microsoft.Extensions.DependencyInjection;
 using Microsoft.Extensions.Diagnostics.HealthChecks;
 using Microsoft.Extensions.Options;
 using ZB.MOM.WW.Health;
+using ZB.MOM.WW.ScadaBridge.Host.Health;

 namespace ZB.MOM.WW.ScadaBridge.Host.Tests;

@@ -177,10 +178,11 @@ public class HealthCheckTests : IDisposable
    }

    [Fact]
-    public void ActiveTier_Carries_Only_ActiveNode()
+    public void ActiveTier_Carries_ActiveNode_And_Database()
    {
-        // The active-node leader check carries the Active tag (→ /health/active); the readiness
-        // checks do not, so /health/active reports leadership alone.
+        // Review 01 [High]: /health/active = oldest Up member AND database reachable, so a
+        // DB-dead active node is pulled from Traefik rotation. active-node + database carry
+        // the Active tag; akka-cluster does not (readiness-only).
        var previousEnv = Environment.GetEnvironmentVariable("DOTNET_ENVIRONMENT");
        try
        {
@@ -192,7 +194,8 @@ public class HealthCheckTests : IDisposable
            Assert.True(registrations.ContainsKey("active-node"), "Expected an 'active-node' health check.");
            Assert.Contains(ZbHealthTags.Active, registrations["active-node"].Tags);

-            Assert.DoesNotContain(ZbHealthTags.Active, registrations["database"].Tags);
+            // Database gates active too (DB-dead active node must drop out of rotation).
+            Assert.Contains(ZbHealthTags.Active, registrations["database"].Tags);
            Assert.DoesNotContain(ZbHealthTags.Active, registrations["akka-cluster"].Tags);
        }
        finally
@@ -200,4 +203,51 @@ public class HealthCheckTests : IDisposable
            Environment.SetEnvironmentVariable("DOTNET_ENVIRONMENT", previousEnv);
        }
    }
+
+    [Fact]
+    public void ActiveNodeCheck_IsHostLocalOldestNodeCheck()
+    {
+        // The shared package's ActiveNodeHealthCheck is LEADER-based (review 01 [High]):
+        // during a partition both nodes think they lead and Traefik serves both. The
+        // active-node check must be the host-local oldest-member check instead.
+        var previousEnv = Environment.GetEnvironmentVariable("DOTNET_ENVIRONMENT");
+        try
+        {
+            Environment.SetEnvironmentVariable("DOTNET_ENVIRONMENT", "Central");
+            var factory = CreateCentralFactory();
+
+            var active = Registrations(factory).Single(r => r.Name == "active-node");
+            Assert.Contains(ZbHealthTags.Active, active.Tags);
+
+            var check = active.Factory(factory.Services);
+            Assert.IsType<OldestNodeActiveHealthCheck>(check);
+        }
+        finally
+        {
+            Environment.SetEnvironmentVariable("DOTNET_ENVIRONMENT", previousEnv);
+        }
+    }
+
+    [Fact]
+    public void DatabaseCheck_AlsoGatesActive()
+    {
+        // Review 01 [High] recommendation: /health/active should require Ready
+        // (DB reachable) in addition to singleton-host status, so a DB-dead
+        // active node is pulled from Traefik rotation.
+        var previousEnv = Environment.GetEnvironmentVariable("DOTNET_ENVIRONMENT");
+        try
+        {
+            Environment.SetEnvironmentVariable("DOTNET_ENVIRONMENT", "Central");
+            var factory = CreateCentralFactory();
+
+            var registrations = Registrations(factory).ToDictionary(r => r.Name);
+            var db = registrations["database"];
+            Assert.Contains(ZbHealthTags.Ready, db.Tags);
+            Assert.Contains(ZbHealthTags.Active, db.Tags);
+        }
+        finally
+        {
+            Environment.SetEnvironmentVariable("DOTNET_ENVIRONMENT", previousEnv);
+        }
+    }
 }
@@ -26,6 +26,23 @@ public class HoconBuilderTests
        FailureDetectionThreshold = TimeSpan.FromSeconds(10),
    };

+    [Fact]
+    public void BuildHocon_ClusterLeavePhaseTimeout_ExceedsDrainBudget()
+    {
+        // Review 01 [Medium]: the five singleton drain tasks GracefulStop(10s)
+        // inside cluster-leave, whose default phase timeout is 5s — Akka abandons
+        // the task at 5s and the drain guarantee is silently halved.
+        var node = new NodeOptions { Role = "Central", NodeHostname = "node1", RemotingPort = 8081 };
+        var hocon = AkkaHostedService.BuildHocon(
+            node, DefaultCluster(), new[] { "Central" },
+            TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(15));
+
+        var config = ConfigurationFactory.ParseString(hocon);
+        var phaseTimeout = config.GetTimeSpan("akka.coordinated-shutdown.phases.cluster-leave.timeout");
+        Assert.True(phaseTimeout >= TimeSpan.FromSeconds(15),
+            $"cluster-leave phase timeout ({phaseTimeout}) must exceed the 10s singleton GracefulStop budget");
+    }
+
    [Fact]
    public void BuildHocon_PlainValues_ParsesAndPreservesValues()
    {
@@ -319,6 +319,36 @@ public class StartupValidatorTests
        Assert.Contains("must not target the gRPC port", ex.Message);
    }

+    [Fact]
+    public void Site_SeedNodeOnMetricsPort_FailsValidation()
+    {
+        // Review 01 [Medium]: appsettings.Site.json shipped a seed pointing at the
+        // Kestrel HTTP/1.1 metrics listener (8084) — a doomed Akka.Remote
+        // association. The validator guarded GrpcPort but not MetricsPort.
+        var values = ValidSiteConfig();
+        values["ScadaBridge:Node:GrpcPort"] = "8083";
+        values["ScadaBridge:Node:MetricsPort"] = "8084";
+        values["ScadaBridge:Cluster:SeedNodes:1"] = "akka.tcp://scadabridge@site-a-node1:8084";
+        var config = BuildConfig(values);
+
+        var ex = Assert.Throws<InvalidOperationException>(() => StartupValidator.Validate(config));
+        Assert.Contains("must not target the metrics port", ex.Message);
+    }
+
+    [Fact]
+    public void Site_SeedNodeOnDefaultMetricsPort_FailsValidation()
+    {
+        // MetricsPort absent => NodeOptions default 8084. A seed on 8084 must
+        // still be rejected. Keep GrpcPort distinct so only the metrics rule fires.
+        var values = ValidSiteConfig();
+        values["ScadaBridge:Node:GrpcPort"] = "8083";
+        values["ScadaBridge:Cluster:SeedNodes:1"] = "akka.tcp://scadabridge@site-a-node2:8084";
+        var config = BuildConfig(values);
+
+        var ex = Assert.Throws<InvalidOperationException>(() => StartupValidator.Validate(config));
+        Assert.Contains("must not target the metrics port", ex.Message);
+    }
+
    [Fact]
    public void Site_SeedNodeOnDefaultGrpcPort_FailsValidation()
    {
@@ -0,0 +1,84 @@
+using Akka.Actor;
+using Microsoft.Extensions.DependencyInjection;
+using Microsoft.Extensions.Hosting;
+using Microsoft.Extensions.Logging.Abstractions;
+using Microsoft.Extensions.Options;
+using NSubstitute;
+using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
+using ZB.MOM.WW.ScadaBridge.Communication;
+using ZB.MOM.WW.ScadaBridge.Host.Actors;
+
+namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
+
+/// <summary>
+/// PLAN-01 Task 20 (review 01 underdeveloped #2): with SBR down-if-alone +
+/// run-coordinated-shutdown-when-down live, a self-downed node terminates its
+/// ActorSystem out from under the still-running Host process. Unless the process
+/// then exits, the node idles forever — serving nothing, restarted by nobody.
+/// The recovery contract: unexpected ActorSystem termination ⇒ StopApplication()
+/// ⇒ the service supervisor restarts the process ⇒ it rejoins as a fresh
+/// incarnation. A termination that IS the host's own StopAsync must NOT trip it.
+/// </summary>
+public sealed class UnexpectedTerminationTests
+{
+    private static int FreePort()
+    {
+        var l = new System.Net.Sockets.TcpListener(System.Net.IPAddress.Loopback, 0);
+        l.Start();
+        var p = ((System.Net.IPEndPoint)l.LocalEndpoint).Port;
+        l.Stop();
+        return p;
+    }
+
+    private static AkkaHostedService CreateAkkaHostedService(IHostApplicationLifetime lifetime)
+    {
+        var port = FreePort();
+        var self = $"akka.tcp://scadabridge@127.0.0.1:{port}";
+        return new AkkaHostedService(
+            new ServiceCollection().BuildServiceProvider(),
+            Options.Create(new NodeOptions { Role = "Site", NodeHostname = "127.0.0.1", RemotingPort = port, SiteId = "site-a" }),
+            Options.Create(new ClusterOptions
+            {
+                SeedNodes = new List<string> { self },
+                AllowSingleNodeCluster = true,
+                SplitBrainResolverStrategy = "keep-oldest",
+                StableAfter = TimeSpan.FromSeconds(3),
+                HeartbeatInterval = TimeSpan.FromMilliseconds(500),
+                FailureDetectionThreshold = TimeSpan.FromSeconds(2),
+                MinNrOfMembers = 1,
+                DownIfAlone = true,
+            }),
+            Options.Create(new CommunicationOptions()),
+            NullLogger<AkkaHostedService>.Instance,
+            lifetime);
+    }
+
+    [Fact]
+    public async Task ActorSystemTerminatedOutsideStopAsync_StopsHostApplication()
+    {
+        var lifetime = Substitute.For<IHostApplicationLifetime>();
+        var service = CreateAkkaHostedService(lifetime);
+        // Create the system (StartAsync registers actors; the system is created
+        // by GetOrCreateActorSystem, which wires the WhenTerminated watchdog).
+        service.GetOrCreateActorSystem();
+        var system = service.ActorSystem!;
+
+        await system.Terminate(); // simulated self-down, NOT StopAsync
+        await Task.Delay(500);
+
+        lifetime.Received(1).StopApplication();
+    }
+
+    [Fact]
+    public async Task StopAsync_DoesNotTriggerStopApplication()
+    {
+        var lifetime = Substitute.For<IHostApplicationLifetime>();
+        var service = CreateAkkaHostedService(lifetime);
+        service.GetOrCreateActorSystem();
+
+        await service.StopAsync(CancellationToken.None);
+        await Task.Delay(500);
+
+        lifetime.DidNotReceive().StopApplication();
+    }
+}
@@ -18,6 +18,7 @@
    <PackageReference Include="Microsoft.AspNetCore.Mvc.Testing" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.InMemory" />
    <PackageReference Include="Microsoft.NET.Test.Sdk" />
+    <PackageReference Include="NSubstitute" />
    <PackageReference Include="Serilog" />
    <PackageReference Include="xunit" />
    <PackageReference Include="xunit.runner.visualstudio" />
@@ -0,0 +1,69 @@
+using Akka.Cluster;
+using Xunit;
+using ZB.MOM.WW.ScadaBridge.Host.Health;
+
+namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;
+
+/// <summary>
+/// The exact scenario from arch-review 01 [High]: after the original first node
+/// (oldest = singleton host) leaves and rejoins, it comes back as the YOUNGEST
+/// member while the peer is now oldest — yet by Akka's address-ordered rule the
+/// rejoiner may regain cluster LEADERSHIP. The two definitions permanently
+/// diverge, which is why every product active/standby decision must use
+/// <see cref="ClusterActivityEvaluator.SelfIsOldest(Cluster, string?)"/> (oldest
+/// member = singleton host) and never <c>cluster.State.Leader</c>.
+///
+/// DEVIATION FROM THE PLAN (documented): the plan drafted this as a hard CRASH of
+/// the oldest node. Under two-node <c>keep-oldest</c> (verified empirically in
+/// <c>SbrFailoverTests</c>, Akka 1.5.62) a hard crash of the oldest makes the
+/// younger survivor down ITSELF — total cluster loss — so the survivor could not
+/// be asserted to stay up. A graceful cluster LEAVE of the oldest instead models a
+/// rolling restart / redeploy of the first node (the realistic way the original
+/// oldest cycles out) and hands oldest-ship to the survivor cleanly, without any
+/// unreachable-partition SBR decision. The divergence assertions are identical and
+/// leadership-independent either way.
+/// </summary>
+public class ActiveNodeSemanticsTests
+{
+    [Fact]
+    public async Task AfterOldestNodeRestarts_SurvivorStaysActive_RejoinerIsStandby()
+    {
+        await using var fixture = await TwoNodeClusterFixture.StartAsync();
+
+        // Gracefully remove A (oldest); B takes over as oldest. A graceful Leave
+        // (not a hard crash) is required because two-node keep-oldest downs the
+        // survivor on an oldest-node partition — see the class-doc deviation note.
+        var clusterA = Akka.Cluster.Cluster.Get(fixture.NodeA);
+        var victim = clusterA.SelfAddress;
+        clusterA.Leave(victim);
+        await TwoNodeClusterFixture.WaitForMemberRemoved(fixture.NodeB, victim, TimeSpan.FromSeconds(30));
+
+        // Release the port so the "restarted" original node can rebind it.
+        await fixture.NodeA.Terminate();
+
+        // Restart A on the same port — it rejoins as the YOUNGEST member but
+        // (lowest address) may regain Akka leadership.
+        var rejoined = fixture.StartNode(fixture.PortA, "Central");
+        try
+        {
+            await TwoNodeClusterFixture.WaitForMembersUp(rejoined, 2, TimeSpan.FromSeconds(30));
+            await TwoNodeClusterFixture.WaitForMembersUp(fixture.NodeB, 2, TimeSpan.FromSeconds(30));
+
+            var clusterB = Akka.Cluster.Cluster.Get(fixture.NodeB);
+            var clusterA2 = Akka.Cluster.Cluster.Get(rejoined);
+
+            // Product active-node semantics: survivor (oldest, singleton host)
+            // is active; the rejoined original node is standby — even when it
+            // holds Akka leadership.
+            Assert.True(ClusterActivityEvaluator.SelfIsOldest(clusterB),
+                "Survivor B must be the oldest Up member (the active/singleton-host node).");
+            Assert.False(ClusterActivityEvaluator.SelfIsOldest(clusterA2),
+                "Rejoined original node is the youngest member — it must NOT be active, " +
+                "even if it holds address-ordered Akka leadership (that is the divergence).");
+        }
+        finally
+        {
+            await rejoined.Terminate();
+        }
+    }
+}
Author	SHA1	Message	Date
Joseph Doherty	ec47cb5612	docs(archreview): PLAN-01 complete (23/23) — sync tracker + tasks.json; log deploy-artifact deferral	2026-07-08 17:12:46 -04:00
Joseph Doherty	dc7a613dae	docs(cleanup): sync Traefik + Host docs to oldest-member active-node semantics + DB-gated /health/active; TLS roadmap note Renames stale ActiveNodeHealthCheck references to OldestNodeActiveHealthCheck (Task 7 rename) across the requirements + components doc sets, corrects leader→oldest-member wording and the embedded IsActiveNode snippet, and adds an explicit 'production TLS profile not yet implemented' roadmap note. deploy/wonder-app-vd03 NodeName overlay edits deferred: that production deploy artifact is not tracked in this repo (same as Tasks 16 & 20).	2026-07-08 17:09:34 -04:00
Joseph Doherty	5117fa97b9	test(docker): failover drill script — SIGKILL the active central node, assert Traefik recovery	2026-07-08 17:05:31 -04:00
Joseph Doherty	1f1dbd916d	docs(host): REQ-HOST-6 documents the hand-rolled HOCON bootstrap; drop unused Akka.Hosting packages	2026-07-08 16:59:19 -04:00
Joseph Doherty	48e97fee01	feat(host): exit process on unexpected ActorSystem termination — completes the down-if-alone recovery loop WhenTerminated watchdog calls IHostApplicationLifetime.StopApplication() when the ActorSystem dies outside StopAsync (SBR self-down + run-coordinated-shutdown-when-down), so the service supervisor restarts the node as a fresh incarnation. Optional ctor param keeps every existing construction site compiling. Plan's deploy/wonder-app-vd03/install.ps1 service-recovery-actions edit is skipped: that production deploy artifact is not tracked in this repo (Task 16 skipped its appsettings.Central.json for the same reason).	2026-07-08 16:56:24 -04:00
Joseph Doherty	dea69842d5	fix(host): rate-limit dead-letter warnings (10/min + suppression summary); metric counting unchanged	2026-07-08 16:52:25 -04:00
Joseph Doherty	3ce21734b8	fix(docker): 30s stop_grace_period so graceful redeploys aren't SIGKILLed mid-CoordinatedShutdown	2026-07-08 16:40:17 -04:00
Joseph Doherty	ce66e194a8	feat(cluster): AllowSingleNodeCluster flag — single-seed installs drop the phantom seed Validator lowers the seed-count floor to 1 when AllowSingleNodeCluster is set, otherwise still requires 2 and points operators at the flag. The plan's deploy/wonder-app-vd03/appsettings.Central.json edit is skipped: that production deploy artifact is not tracked in this repo (only its deployment record doc is).	2026-07-08 16:39:12 -04:00
Joseph Doherty	0e3c1df1a7	fix(host): dev site seed no longer targets the metrics port; validator rejects seed-vs-MetricsPort	2026-07-08 16:37:43 -04:00
Joseph Doherty	d962c77bb7	fix(health): evict deleted sites from the aggregator on the periodic site refresh	2026-07-08 16:27:38 -04:00
Joseph Doherty	87c7255912	feat(ui): health dashboard shows metrics-stale badge and offline-since timestamp	2026-07-08 16:23:43 -04:00
Joseph Doherty	c73b7faa11	feat(health): metrics-stale signal + status-transition timestamps; spec now matches heartbeat-liveness code	2026-07-08 16:21:30 -04:00
Joseph Doherty	15d91d760f	docs(archreview): sync PLAN-01 tracker + tasks.json — Wave 1 T2,T5–T12 done (12/23)	2026-07-08 16:14:46 -04:00
Joseph Doherty	6300d6d399	fix(health): acked SendAsync transport — report-loss counter restore is live, not dead code	2026-07-08 16:12:48 -04:00
Joseph Doherty	17af376a8e	feat(comm): ack site health reports end-to-end (SiteHealthReportAck, additive contract)	2026-07-08 16:09:08 -04:00
Joseph Doherty	f7b9d342e4	refactor(host): central singletons via CentralSingletonRegistrar — outbox/audit-ingest gain drain tasks	2026-07-08 16:05:52 -04:00
Joseph Doherty	c255ec31c9	feat(host): CentralSingletonRegistrar — canonical singleton manager+proxy+drain helper	2026-07-08 15:59:01 -04:00
Joseph Doherty	7138d47630	test(cluster): restarted original node stays standby — oldest-member active semantics proven (graceful-leave variant per 2-node keep-oldest gap)	2026-07-08 15:57:45 -04:00
Joseph Doherty	f164f62a8b	fix(cluster): give cluster-leave phase a 15s timeout so 10s singleton drains can complete	2026-07-08 15:55:06 -04:00
Joseph Doherty	ff89e3145c	fix(host): /health/active = oldest member + DB reachable — partition-safe Traefik routing	2026-07-08 15:51:56 -04:00
Joseph Doherty	da6d5ae12c	fix(host): unify active-node on oldest-member semantics — purge/self-report/labels now track the singleton host	2026-07-08 15:48:07 -04:00
Joseph Doherty	6320be9954	feat(host): ClusterActivityEvaluator — oldest-member (singleton-host) active-node definition	2026-07-08 15:43:13 -04:00