# Architecture Review 01 — Cluster Infrastructure, Host, Failover, Health Monitoring, Deployment Topology Reviewer domain: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure`, `src/ZB.MOM.WW.ScadaBridge.Host`, `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring`, `docker/`, `docker-env2/`, `deploy/wonder-app-vd03/`, Traefik config, and the corresponding design docs (`Component-ClusterInfrastructure.md`, `Component-Host.md`, `Component-HealthMonitoring.md`, `Component-TraefikProxy.md`) and test projects. ## Scope Read in full: `AkkaHostedService.cs` (1,203 lines — HOCON builder, singleton wiring, coordinated shutdown), `Program.cs` (both role branches), `StartupValidator.cs`, `StartupRetry.cs`, `SiteServiceRegistration.cs`, `NodeOptions.cs`, `ClusterOptions.cs` + validator + `ServiceCollectionExtensions.cs`, all Host `Health/` checks (`RequiredSingletonsHealthCheck`, `ActiveNodeGate`, `AkkaClusterNodeProvider`), `DeadLetterMonitorActor.cs`, `AkkaHealthReportTransport.cs`, the entire HealthMonitoring project (`HealthReportSender`, `CentralHealthAggregator`, `CentralHealthReportLoop`, `SiteHealthCollector`, `SiteEventLogFailureCountReporter`, `SiteHealthKpiSampleSource`, options + validator), the docker compose topologies + per-node appsettings + Traefik static/dynamic config + Dockerfile, the on-host deploy artifacts (`deploy/wonder-app-vd03/appsettings.*.json`), the four design docs, the repo's own Akka.NET reference notes (`AkkaDotNet/03-Cluster.md`), and the relevant test projects (`Host.Tests`, `ClusterInfrastructure.Tests`, `HealthMonitoring.Tests`, `IntegrationTests` failover/readiness suites). Health-report ingestion on the central side was cross-checked in `Communication/Actors/CentralCommunicationActor.cs:349-392`. ## Maturity Verdict This slice of the codebase is unusually well-commented, defensively coded at the micro level (options validators with `ValidateOnStart`, CAS loops in the aggregator, sequence-number seeding for failover, HOCON escaping, bounded startup retry), and clearly the product of many careful review passes — but it has **one macro-level defect that undoes the entire failover design: the split-brain resolver is configured but never enabled** (no `akka.cluster.downing-provider-class`), so unreachable nodes are never automatically downed and cluster singletons never migrate on a hard crash. Every document, comment, and validator in the domain reasons meticulously about keep-oldest semantics that are inert at runtime, and the test suite asserts the HOCON *strings* rather than the *behavior*, which is exactly how the gap survived. Secondary structural issues cluster around the ambiguous definition of "active node" (cluster *leader* vs. singleton-hosting *oldest* member, which diverge after restarts) and dev-vs-prod configuration drift. Verdict: **high polish, one critical foundation crack; not production-ready for unattended failover until the downing provider is fixed and verified with a real two-node test.** --- ## 1. Stability ### [Critical] Split-brain resolver is never enabled — automatic failover does not work for crashes/partitions `AkkaHostedService.BuildHocon` (src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs:225-270) emits: ```hocon cluster { ... split-brain-resolver { active-strategy = "keep-oldest" stable-after = 15000ms keep-oldest { down-if-alone = on } } failure-detector { ... } run-coordinated-shutdown-when-down = on } ``` but never sets `akka.cluster.downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster"`. In Akka.NET (1.5.62 per `Directory.Packages.props:8-17`), the default downing provider is **NoDowning**; the whole `split-brain-resolver` section is only read by `SplitBrainResolverProvider`, which must be opted into via `downing-provider-class`. A repo-wide grep confirms the string `SplitBrainResolverProvider` appears **nowhere in `src/`** — only in the project's own reference notes, which show the required line (AkkaDotNet/03-Cluster.md:56, also 15-Coordination.md:69 and 18-MultiNodeTestRunner.md:100). Failure scenario: the active site node loses power. The standby's failure detector marks it unreachable (~10s), but **nothing ever downs it**. The unreachable member stays in the membership forever, the `ClusterSingletonManager` cannot hand over (singleton migration requires the oldest member's *removal*, not mere unreachability), and the site's DeploymentManager singleton — and with it all data collection, script execution, and S&F delivery — never restarts on the survivor. The same applies to all six central singletons (notification-outbox, audit-log-ingest, site-call-audit, audit-log-purge, site-audit-reconciliation, kpi-history-recorder). The documented "~25 seconds total failover" (docs/requirements/Component-ClusterInfrastructure.md:110-119) is only achievable today for a **graceful** stop (CoordinatedShutdown leave); crash and partition scenarios hang indefinitely until an operator downs the node or restarts the survivor. Corroborating evidence that this was never exercised: - `HoconBuilderTests.cs:45,112,155` assert only that the SBR *strings* appear in the config. - `CoordinatedShutdownTests.cs:11-43` are source-text greps (`Assert.Contains("run-by-clr-shutdown-hook = on", content)`). - The "failover" integration tests are single-process simulations — e.g. `CentralFailoverTests.cs:27-47` validates a JWT across two `JwtTokenService` instances; no test forms a two-node cluster and kills the oldest node. - `run-coordinated-shutdown-when-down = on` (AkkaHostedService.cs:265) is dead config: a node is never transitioned to Down by anything. - `ClusterOptionsValidator` (src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptionsValidator.cs:35-63) rigorously rejects any strategy other than `keep-oldest` and demands `DownIfAlone == true` — validating knobs on a resolver that is not installed. Fix: add the `downing-provider-class` line to `BuildHocon` (one line), or better, migrate the bootstrap to `Akka.Cluster.Hosting`'s typed options (already referenced, ZB.MOM.WW.ScadaBridge.Host.csproj:15) whose `SplitBrainResolverOption` wires the provider automatically — and add a real two-node in-process cluster test that kills the oldest member and asserts the singleton re-hosts within the expected window. ### [High] "Active node" means two different things — cluster leader vs. oldest member — and they diverge Two distinct node-selection concepts are used interchangeably: - **Leader-based** (address ordering, *not* age): `ActiveNodeGate.IsActiveNode` (src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeGate.cs:36-52), `AkkaClusterNodeProvider.SelfIsPrimary` and the "Primary"/"Standby" labels (src/ZB.MOM.WW.ScadaBridge.Host/Health/AkkaClusterNodeProvider.cs:29-40,61), the `active-node` health check Traefik routes on (Program.cs:231-234, docker/traefik/dynamic.yml:12-18), the `CentralHealthReportLoop` leader gate (CentralHealthReportLoop.cs:93-97), and the site event-log purge gate (`SiteEventLogActiveNodeCheck` → `SelfIsPrimary`, SiteServiceRegistration.cs:116-120). - **Oldest-based** (singleton placement): every `ClusterSingletonManager` (AkkaHostedService.cs:410+, 906-925), and the site health "IsActiveNode" flag, which is set by the DeploymentManager *singleton* on start/stop (src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs:243,258). In Akka the leader is the lowest-address `Up` member; the singleton host is the *oldest* member. They coincide at first boot but diverge permanently after the original first node is restarted and rejoins (it regains leadership by address order while the peer remains oldest). Concrete consequences once diverged: 1. **Site event-log retention is unenforced on the node that writes events.** Events are recorded by the singleton hierarchy on the oldest node; the purge (`EventLogPurgeService.cs:21,36`) early-exits unless `SelfIsPrimary` (leader). The active/oldest node's SQLite grows past the 30-day/1GB cap while the leader purges an idle database. 2. **Health dashboard labels the wrong node "Primary"** (AkkaClusterNodeProvider.cs:61): the node named Primary hosts no singletons; the node doing all the work shows Standby. 3. **Central self-report is emitted by the leader** (CentralHealthReportLoop.cs:93) while the NotificationOutbox/audit singletons run on the other node — the `$central` health card describes the wrong node. 4. **Traefik routes UI/API to the leader** while every singleton `Ask` then hops to the peer — functionally correct via the proxies, but it contradicts the design docs' "one active node runs all central components" model (Component-ClusterInfrastructure.md:44-47) and doubles intra-cluster latency on every outbox/audit page load. Recommendation: pick one definition. Simplest coherent choice: derive "active" from singleton-host status (e.g. compare self against the oldest `Up` member, or gate on reachability of a local singleton), and use it consistently for `ActiveNodeGate`, `SelfIsPrimary`, the purge gate, and the `/health/active` check — leadership is an Akka-internal concept that only accidentally matches the product's active/standby model. ### [High] During a central network partition, both nodes report `/health/active = 200` and Traefik serves both `ActiveNodeGate`/the active-node check compute `cluster.State.Leader` from each node's own gossip view. When central-a and central-b cannot see each other but Traefik can reach both (a plausible partial partition), each side sees the peer as unreachable and computes **itself** as leader — both return 200 on `/health/active`, and Traefik's load balancer (docker/traefik/dynamic.yml:10-18) round-robins requests between two nodes that each believe they are active. Blazor circuits and the Inbound API's active-node 503 gate (Program.cs:256-260) are then split across both. Because the SBR is not enabled (Critical finding above), nothing ever resolves the partition — this state persists until manual intervention. Singletons do *not* duplicate (that requires member removal), but every leader-gated behavior does. Enabling keep-oldest + `down-if-alone` fixes the persistence; consider also making `/health/active` require `Ready` (DB reachable) in addition to leadership so a DB-dead leader is pulled from rotation. ### [Medium] Cluster-leave drain tasks are budgeted 10s inside a 5s CoordinatedShutdown phase Five singletons add a `PhaseClusterLeave` task that runs `GracefulStop(TimeSpan.FromSeconds(10))` (AkkaHostedService.cs:551-567 site-call-audit, 624-641 audit-log-purge, 676-693 site-audit-reconciliation, 733-750 kpi-history-recorder, 787-804 pending-deployment-purge). Akka's `cluster-leave` phase uses the default phase timeout of **5 seconds**; the phase abandons the task at 5s (default `recover = on`) and proceeds, so the 10s graceful-stop can never complete its full window and the `catch` logging in those tasks never observes the overrun (the task is orphaned, not faulted). The intended "let the in-flight EF upsert drain before handover" guarantee is silently halved. Fix: set `akka.coordinated-shutdown.phases.cluster-leave.timeout` in `BuildHocon` to exceed the graceful-stop budget, or shrink the `GracefulStop` timeout below 5s. ### [Medium] The two busiest singletons have no drain task at all The `NotificationOutboxActor` and `AuditLogIngestActor` singletons are terminated by bare `PoisonPill` with no cluster-leave drain (AkkaHostedService.cs:410-419, 451-458); the comment at AkkaHostedService.cs:544-549 explicitly notes the pattern "is suitable for the NotificationOutbox singleton; not added here to keep this change minimal". These are precisely the actors with in-flight EF transactions on every message (dispatch sweeps, dual-write ingest). The idempotent DB design (insert-if-not-exists on `NotificationId`/`EventId`) makes this safe-but-lossy rather than corrupting, but the inconsistency (drain for the low-traffic maintenance singletons, none for the hot ones) should be resolved. ### [Medium] Health-report loss recovery is dead code on the production transport `HealthReportSender` carefully restores interval counters if `_transport.Send` throws (HealthReportSender.cs:158-175), and `SiteHealthCollector.AddIntervalCounters` (SiteHealthCollector.cs:153-171) exists to support it. But the only production transport is `AkkaHealthReportTransport.Send` (src/ZB.MOM.WW.ScadaBridge.Host/AkkaHealthReportTransport.cs:26-33) — a fire-and-forget `Tell` to an `ActorSelection`, which **never throws** and silently returns when the actor system is null. Any downstream loss (central unreachable, ClusterClient reconnecting, dead-lettered selection) drops the report *and* the interval error counters it carried — the scenario the restore logic was built for is exactly the one it can't see. Either make the transport Ask-with-ack (the site→central notification forward already uses this pattern over the same actor) or accept the loss and delete the restore machinery; the current shape gives false confidence. ### [Medium] Offline detection deviates from the spec and can mask a dead metrics pipeline `Component-HealthMonitoring.md:43-46` defines offline as "no *report* within 60s". `CentralHealthAggregator.CheckForOfflineSites` (CentralHealthAggregator.cs:224-263) instead keys off `LastHeartbeatAt`, which is also advanced by the ~5s transport heartbeats (`CentralCommunicationActor.HandleHeartbeat` → `MarkHeartbeat`, CentralCommunicationActor.cs:349-353). A site whose `HealthReportSender` loop has died (or whose DeploymentManager singleton is stuck, so `IsActiveNode` is false on both nodes and reports are skipped — HealthReportSender.cs:85) keeps heartbeating and shows **online with silently frozen metrics forever**; nothing surfaces `LastReportReceivedAt` staleness. The heartbeat-based liveness choice is defensible (the code comment argues it), but then the dashboard needs a distinct "metrics stale" signal, and the design doc should be updated — currently spec and code disagree. ### [Medium] Second seed node in the checked-in site dev config targets the metrics port; validator has a gap `src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json:14-17` seeds `akka.tcp://scadabridge@localhost:8082` and `akka.tcp://scadabridge@localhost:8084`, while the same file sets `MetricsPort: 8084` (line 10). The second seed points at the Kestrel HTTP/1.1 metrics listener — a node attempting to join via that seed performs a doomed Akka.Remote association against Kestrel (handshake noise, retry loops). `StartupValidator` explicitly guards seed-vs-GrpcPort (StartupValidator.cs:121-127) but not seed-vs-MetricsPort, so this config validates cleanly. Fix the dev config (8084 → the intended second node's remoting port) and extend the seed-port loop to also reject `MetricsPort`. ### [Medium] Docker stop grace period is shorter than a full CoordinatedShutdown `docker/docker-compose.yml` sets no `stop_grace_period` (default 10s SIGTERM→SIGKILL) and the Dockerfile runs `dotnet` as PID 1 (docker/Dockerfile, final stage). A graceful shutdown runs cluster-leave (5s budget) + cluster-exiting (10s) + actor-system-terminate plus Serilog flush — easily exceeding 10s on a loaded node, so `bash docker/deploy.sh` recreates can SIGKILL mid-drain, turning every "graceful" redeploy into the crash path (which, per the Critical finding, currently has no automatic recovery on the surviving node). Add `stop_grace_period: 30s` to the app services. ### [Low] Aggregator never forgets sites `CentralHealthAggregator._siteStates` (CentralHealthAggregator.cs:16) only grows: a site deleted from configuration remains a permanently-offline dashboard tile (and continues to be enumerated by `SiteHealthKpiSampleSource.CollectAsync`, SiteHealthKpiSampleSource.cs:74-85) until process restart. Needs an eviction path wired to site deletion. ### [Low] Dead-letter monitor: unbounded warning volume and restart-reset count `DeadLetterMonitorActor` logs one Warning per dead letter with no sampling/rate-limiting and keeps its count in actor state that resets on restart (src/ZB.MOM.WW.ScadaBridge.Host/Actors/DeadLetterMonitorActor.cs:14,26-35). A dead-letter storm (mass undeploy, failover, shutdown) floods the log; the health metric path (`IncrementDeadLetter`) is fine because the collector counter is interval-based. ### [Low] Mandatory two-seed rule forces phantom seeds in single-node installs `ClusterOptionsValidator` requires ≥2 seed nodes (ClusterOptionsValidator.cs:29-33); the shipped single-node production artifact satisfies it with a documented placeholder (`deploy/wonder-app-vd03/appsettings.Central.json:9-19`, second seed `localhost:8091` "harmless while absent"). The node dials the phantom seed forever (log noise, periodic connect attempts), and the rule's intent — no startup-ordering dependency — is defeated by fiction. Consider allowing 1 seed with a warning, or a `SingleNode` acknowledgement flag. ## 2. Performance This domain is largely cold-path and correctly engineered for it; nothing here touches the data-plane hot path. Observations: - **[Low] Readiness probes fan out five cluster Asks per poll.** `RequiredSingletonsHealthCheck` sends 5 concurrent `Identify` Asks through singleton proxies every `/health/ready` poll (RequiredSingletonsHealthCheck.cs:66-78,111-117). When the singletons live on the peer node each probe is a remoting round trip. Bounded (2s, concurrent) and cheap at typical poll rates, but a tight orchestration probe interval multiplies cluster chatter. - **[Low] Startup compiles every inbound API method sequentially before serving** (Program.cs:379-388). Startup latency scales linearly with method count; a slow Roslyn compile of many methods delays readiness on the *standby* too (it runs the same branch). Acceptable now; parallelize or defer-compile if method counts grow. - **[Low] Per-tick allocations in health collection** (dictionary snapshots per 30s report, `GetAllSiteStates` copy per dashboard poll — CentralHealthAggregator.cs:173-176) are trivial at this cadence. No action. - **Positive:** the site audit telemetry actor is pinned to a dedicated dispatcher (`audit-telemetry-dispatcher`, AkkaHostedService.cs:226-232,1150-1158) so batch SQLite/gRPC work never contends with hot-path actors; `StoreAndForwardService.StartAsync` is awaited rather than sync-blocked (AkkaHostedService.cs:976-981); the ActorSystem DI bridge is a root singleton precisely to avoid per-scope disposal (Program.cs:240-250) — all correct. ## 3. Conventions - **[Medium] REQ-HOST-6 says "Akka.NET bootstrap via Akka.Hosting"; the code hand-rolls HOCON.** `Component-Host.md` (REQ-HOST-6, line 106-114) mandates Akka.Hosting; the actual bootstrap is string-interpolated HOCON + `ActorSystem.Create` (AkkaHostedService.cs:171-176,214-271) with `Akka.Hosting`/`Akka.Cluster.Hosting`/`Akka.Remote.Hosting` referenced but unused for cluster bring-up (ZB.MOM.WW.ScadaBridge.Host.csproj:15-18). This is not just doc drift: Akka.Cluster.Hosting's typed `SplitBrainResolverOption` sets `downing-provider-class` for you — using the mandated API would have prevented the Critical finding. Either implement REQ-HOST-6 or amend it and drop the unused packages. - **[High] Production deploy artifact ships with authentication disabled.** `deploy/wonder-app-vd03/appsettings.Central.json:39-42` sets `"DisableLogin": true` — every request auto-authenticates as `multi-role` with all roles, and the management API on that host is unauthenticated. The base config documents this as "DEV/TEST ONLY — never enable in production" with "no environment guard; a startup warning is the only protection" (src/ZB.MOM.WW.ScadaBridge.Host/appsettings.json:6). A `deploy/` artifact with an install runbook is a production surface; at minimum gate `DisableLogin` on `Environment.IsDevelopment()` or require an explicit `I-understand` companion flag. - **[Medium] LDAP transport is plaintext everywhere, contradicting the security spec.** The design requires "LDAPS/StartTLS required" (CLAUDE.md Security & Auth; Component-Security), but every shipped config — docker (docker/central-node-a/appsettings.Central.json:26-33) and the on-host deploy (deploy/wonder-app-vd03/appsettings.Central.json:26-34) — sets `"Transport": "None", "AllowInsecure": true` and sends the service-account bind password in clear. Dev GLAuth is a fair excuse for docker; the deploy artifact should not normalize it. - **[Medium] Secrets in checked-in dev configs.** SQL credentials, LDAP service-account password and the JWT signing key are committed in `docker/central-node-a/appsettings.Central.json:21-38` (labelled dev-only; the compose pepper is likewise flagged, docker-compose.yml:9-14). The wonder-app-vd03 artifact handles this correctly via `${...}` placeholders + env injection (appsettings.Central.json:20-24) — good pattern, worth backporting a scrubbed variant to docker if these ever leave a lab network. Also note `ConfigSecretsTests.cs` exists in Host.Tests, so the team is tracking this. - **[Low] Stale doc references.** `Component-TraefikProxy.md:121` locates `ActiveNodeHealthCheck` at `src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeHealthCheck.cs`; the check now comes from the shared `ZB.MOM.WW.Health.Akka` package (Program.cs:214-234) and no such file exists. `Component-TraefikProxy.md:115` also still describes `/health/ready` as "database + Akka cluster" — it now includes `required-singletons` (Program.cs:227-230; Component-Host.md REQ-HOST-4a *is* current). Keep the Traefik doc in sync. - **[Low] Five copy-pasted ~60-line singleton registration blocks.** AkkaHostedService.RegisterCentralActors repeats the manager/drain-task/proxy triple five times (AkkaHostedService.cs:399-811) with only names and Props varying. Extract a `StartCentralSingleton(name, props, drainTimeout)` helper — the copy-paste already caused drift (outbox/ingest lack drain tasks). - **Positive conventions compliance:** Options pattern is followed rigorously (component-owned options classes, `ValidateOnStart`, dedicated validators with key-naming messages — ClusterOptionsValidator, HealthMonitoringOptionsValidator); HOCON interpolation is injection-safe (`QuoteHocon`/`DurationHocon`, AkkaHostedService.cs:280-295); Serilog enrichment matches REQ-HOST-8; startup validation matches REQ-HOST-4 including cross-field port-collision rules; single-binary role dispatch matches REQ-HOST-1/2; the `/health/ready` vs `/health/active` tier separation (leadership deliberately excluded from readiness) is well-reasoned and documented in place (Program.cs:346-356). ## 4. Underdeveloped Areas 1. **No real multi-node failover test anywhere.** `CentralFailoverTests` are single-process simulations (JWT across two service instances, CentralFailoverTests.cs:27-47); `SiteFailoverTests`/`DualNodeRecoveryTests` similarly exercise component state recovery, not cluster membership; `CoordinatedShutdownTests` grep source text (CoordinatedShutdownTests.cs:11-43); `HoconBuilderTests` assert config strings. The repo's own notes describe the MultiNode test runner (AkkaDotNet/18-MultiNodeTestRunner.md) but it is unused. This is the direct cause of the Critical finding surviving: nothing ever killed the oldest node and waited for the singleton to move. Even a plain xunit test spinning two `ActorSystem`s in-process and calling `cluster.Down()`/terminating one would have caught it. 2. **Windows Service lifecycle management is thin.** `Component-ClusterInfrastructure.md:19` claims "Manage Windows service lifecycle (start, stop, restart)"; the implementation is `UseWindowsService()` (Program.cs:74,401) plus install scripts. Service recovery actions (restart-on-failure), the watchdog implied by `down-if-alone` ("an external mechanism must restart both nodes" — AkkaDotNet/03-Cluster.md:64) do not exist in the repo; after a down-if-alone self-down the process presumably exits (or worse, idles) with nothing to restart it. The down-if-alone recovery story needs an explicit design (does the process exit? does the service restart it? does it rejoin cleanly?) and a drill. 3. **TLS is absent at every layer of this domain**: Traefik terminates plain HTTP with an unauthenticated dashboard (docker/traefik/traefik.yml:5-7 — documented as dev-only in Component-TraefikProxy.md:131-138), Akka remoting is unencrypted TCP, site gRPC is `http://`. Fine for the lab; there is no production TLS profile or documented path yet. 4. **Dockerfile has no `HEALTHCHECK`** and compose has no health-based dependency ordering; liveness `/healthz` runs zero checks (Program.cs:353-355) so a wedged actor system still reports live. Acceptable minimalism, but an orchestrator migration will need both. 5. **`NodeName` is missing from the wonder-app-vd03 deploy overlays** (deploy/wonder-app-vd03/appsettings.Central.json has no `Node.NodeName`), so audit rows from that install get a NULL `SourceNode` (the base file documents the normalisation, src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Central.json:3) — undermines the per-node audit/KPI features (`IX_AuditLog_Node_Occurred`, per-node stuck KPIs) on the one real deployment. 6. **Known-but-open follow-ups acknowledged in code/docs:** SecuredWrite audit rows leave `SourceNode` NULL (CLAUDE.md notes it as logged); no central persistence of OPC UA cert trust; `docker-env2` mirrors the main topology (verified: same Traefik config shape, same seed/port conventions) so it inherits every finding above. 7. **Health monitoring has no history by design** (Component-HealthMonitoring.md:93-97) — now partially superseded by KPI History (#26), which the docs handle; but online/offline *transitions* are still unrecorded anywhere (doc line 97 defers it), so post-incident "when did the site drop" questions remain unanswerable beyond log scraping. ## Prioritized Recommendations 1. **[Critical, one line + tests] Enable the split-brain resolver.** Add `downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster"` to `BuildHocon` (AkkaHostedService.cs:250) — or move the bootstrap to Akka.Cluster.Hosting per REQ-HOST-6, which sets it via typed options. Then add a two-node in-process failover test (kill oldest → assert singleton re-hosted, member removed within stable-after + margin) and a documented manual drill against the docker cluster (`docker kill scadabridge-central-a`, watch Traefik + singleton logs). 2. **[High] Unify the "active node" definition** on singleton-host (oldest) semantics for `ActiveNodeGate`, `AkkaClusterNodeProvider.SelfIsPrimary`, the event-log purge gate, and the `/health/active` check; today's leader-based checks purge the wrong node's event log and label the wrong node Primary once leader ≠ oldest. 3. **[High] Remove `DisableLogin: true` from `deploy/wonder-app-vd03`** or add a hard environment guard around the flag. 4. **[Medium] Fix the CoordinatedShutdown drain budget** (`cluster-leave` phase timeout ≥ graceful-stop timeout) and add drain tasks for the notification-outbox and audit-log-ingest singletons — or delete all five drains if the idempotent-DB argument makes them unnecessary. 5. **[Medium] Fix `appsettings.Site.json` seed `localhost:8084`** and extend `StartupValidator`'s seed-port loop to reject `MetricsPort` collisions. 6. **[Medium] Make health-report delivery observable**: switch `AkkaHealthReportTransport` to an acked Ask (reusing the existing site→central forward pattern) so the counter-restore logic in `HealthReportSender` actually fires, and surface `LastReportReceivedAt` staleness on the dashboard so heartbeat-alive/metrics-dead sites are visible. 7. **[Medium] Add `stop_grace_period: 30s`** to the app services in both compose files. 8. **[Low] Housekeeping:** evict deleted sites from `CentralHealthAggregator`; rate-limit dead-letter warnings; extract the singleton-registration helper; sync `Component-TraefikProxy.md` with the shared health-check library; add `NodeName` to the wonder-app-vd03 overlays; reconsider the mandatory-two-seeds rule for single-node installs.