# Architecture Review 01 — Cluster Infrastructure, Host, Failover, Health Monitoring, Deployment Topology

Reviewer domain: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure`, `src/ZB.MOM.WW.ScadaBridge.Host`,
`src/ZB.MOM.WW.ScadaBridge.HealthMonitoring`, `docker/`, `docker-env2/`, `deploy/wonder-app-vd03/`,
Traefik config, and the corresponding design docs (`Component-ClusterInfrastructure.md`,
`Component-Host.md`, `Component-HealthMonitoring.md`, `Component-TraefikProxy.md`) and test projects.

## Scope

Read in full: `AkkaHostedService.cs` (1,203 lines — HOCON builder, singleton wiring, coordinated
shutdown), `Program.cs` (both role branches), `StartupValidator.cs`, `StartupRetry.cs`,
`SiteServiceRegistration.cs`, `NodeOptions.cs`, `ClusterOptions.cs` + validator +
`ServiceCollectionExtensions.cs`, all Host `Health/` checks (`RequiredSingletonsHealthCheck`,
`ActiveNodeGate`, `AkkaClusterNodeProvider`), `DeadLetterMonitorActor.cs`,
`AkkaHealthReportTransport.cs`, the entire HealthMonitoring project (`HealthReportSender`,
`CentralHealthAggregator`, `CentralHealthReportLoop`, `SiteHealthCollector`,
`SiteEventLogFailureCountReporter`, `SiteHealthKpiSampleSource`, options + validator), the docker
compose topologies + per-node appsettings + Traefik static/dynamic config + Dockerfile, the
on-host deploy artifacts (`deploy/wonder-app-vd03/appsettings.*.json`), the four design docs, the
repo's own Akka.NET reference notes (`AkkaDotNet/03-Cluster.md`), and the relevant test projects
(`Host.Tests`, `ClusterInfrastructure.Tests`, `HealthMonitoring.Tests`, `IntegrationTests`
failover/readiness suites). Health-report ingestion on the central side was cross-checked in
`Communication/Actors/CentralCommunicationActor.cs:349-392`.

## Maturity Verdict

This slice of the codebase is unusually well-commented, defensively coded at the micro level
(options validators with `ValidateOnStart`, CAS loops in the aggregator, sequence-number seeding
for failover, HOCON escaping, bounded startup retry), and clearly the product of many careful
review passes — but it has **one macro-level defect that undoes the entire failover design: the
split-brain resolver is configured but never enabled** (no `akka.cluster.downing-provider-class`),
so unreachable nodes are never automatically downed and cluster singletons never migrate on a hard
crash. Every document, comment, and validator in the domain reasons meticulously about keep-oldest
semantics that are inert at runtime, and the test suite asserts the HOCON *strings* rather than the
*behavior*, which is exactly how the gap survived. Secondary structural issues cluster around the
ambiguous definition of "active node" (cluster *leader* vs. singleton-hosting *oldest* member,
which diverge after restarts) and dev-vs-prod configuration drift. Verdict: **high polish, one
critical foundation crack; not production-ready for unattended failover until the downing provider
is fixed and verified with a real two-node test.**

---

## 1. Stability

### [Critical] Split-brain resolver is never enabled — automatic failover does not work for crashes/partitions

`AkkaHostedService.BuildHocon` (src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs:225-270)
emits:

```hocon
cluster {
    ...
    split-brain-resolver {
        active-strategy = "keep-oldest"
        stable-after = 15000ms
        keep-oldest { down-if-alone = on }
    }
    failure-detector { ... }
    run-coordinated-shutdown-when-down = on
}
```

but never sets `akka.cluster.downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider,
Akka.Cluster"`. In Akka.NET (1.5.62 per `Directory.Packages.props:8-17`), the default downing
provider is **NoDowning**; the whole `split-brain-resolver` section is only read by
`SplitBrainResolverProvider`, which must be opted into via `downing-provider-class`. A repo-wide
grep confirms the string `SplitBrainResolverProvider` appears **nowhere in `src/`** — only in the
project's own reference notes, which show the required line
(AkkaDotNet/03-Cluster.md:56, also 15-Coordination.md:69 and 18-MultiNodeTestRunner.md:100).

Failure scenario: the active site node loses power. The standby's failure detector marks it
unreachable (~10s), but **nothing ever downs it**. The unreachable member stays in the membership
forever, the `ClusterSingletonManager` cannot hand over (singleton migration requires the oldest
member's *removal*, not mere unreachability), and the site's DeploymentManager singleton — and with
it all data collection, script execution, and S&F delivery — never restarts on the survivor. The
same applies to all six central singletons (notification-outbox, audit-log-ingest, site-call-audit,
audit-log-purge, site-audit-reconciliation, kpi-history-recorder). The documented "~25 seconds
total failover" (docs/requirements/Component-ClusterInfrastructure.md:110-119) is only achievable
today for a **graceful** stop (CoordinatedShutdown leave); crash and partition scenarios hang
indefinitely until an operator downs the node or restarts the survivor.

Corroborating evidence that this was never exercised:

- `HoconBuilderTests.cs:45,112,155` assert only that the SBR *strings* appear in the config.
- `CoordinatedShutdownTests.cs:11-43` are source-text greps (`Assert.Contains("run-by-clr-shutdown-hook = on", content)`).
- The "failover" integration tests are single-process simulations — e.g.
  `CentralFailoverTests.cs:27-47` validates a JWT across two `JwtTokenService` instances; no test
  forms a two-node cluster and kills the oldest node.
- `run-coordinated-shutdown-when-down = on` (AkkaHostedService.cs:265) is dead config: a node is
  never transitioned to Down by anything.
- `ClusterOptionsValidator` (src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptionsValidator.cs:35-63)
  rigorously rejects any strategy other than `keep-oldest` and demands `DownIfAlone == true` —
  validating knobs on a resolver that is not installed.

Fix: add the `downing-provider-class` line to `BuildHocon` (one line), or better, migrate the
bootstrap to `Akka.Cluster.Hosting`'s typed options (already referenced,
ZB.MOM.WW.ScadaBridge.Host.csproj:15) whose `SplitBrainResolverOption` wires the provider
automatically — and add a real two-node in-process cluster test that kills the oldest member and
asserts the singleton re-hosts within the expected window.

### [High] "Active node" means two different things — cluster leader vs. oldest member — and they diverge

Two distinct node-selection concepts are used interchangeably:

- **Leader-based** (address ordering, *not* age): `ActiveNodeGate.IsActiveNode`
  (src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeGate.cs:36-52),
  `AkkaClusterNodeProvider.SelfIsPrimary` and the "Primary"/"Standby" labels
  (src/ZB.MOM.WW.ScadaBridge.Host/Health/AkkaClusterNodeProvider.cs:29-40,61), the `active-node`
  health check Traefik routes on (Program.cs:231-234, docker/traefik/dynamic.yml:12-18), the
  `CentralHealthReportLoop` leader gate (CentralHealthReportLoop.cs:93-97), and the site
  event-log purge gate (`SiteEventLogActiveNodeCheck` → `SelfIsPrimary`,
  SiteServiceRegistration.cs:116-120).
- **Oldest-based** (singleton placement): every `ClusterSingletonManager` (AkkaHostedService.cs:410+,
  906-925), and the site health "IsActiveNode" flag, which is set by the DeploymentManager
  *singleton* on start/stop (src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs:243,258).

In Akka the leader is the lowest-address `Up` member; the singleton host is the *oldest* member.
They coincide at first boot but diverge permanently after the original first node is restarted and
rejoins (it regains leadership by address order while the peer remains oldest). Concrete
consequences once diverged:

1. **Site event-log retention is unenforced on the node that writes events.** Events are recorded
   by the singleton hierarchy on the oldest node; the purge (`EventLogPurgeService.cs:21,36`)
   early-exits unless `SelfIsPrimary` (leader). The active/oldest node's SQLite grows past the
   30-day/1GB cap while the leader purges an idle database.
2. **Health dashboard labels the wrong node "Primary"** (AkkaClusterNodeProvider.cs:61): the node
   named Primary hosts no singletons; the node doing all the work shows Standby.
3. **Central self-report is emitted by the leader** (CentralHealthReportLoop.cs:93) while the
   NotificationOutbox/audit singletons run on the other node — the `$central` health card describes
   the wrong node.
4. **Traefik routes UI/API to the leader** while every singleton `Ask` then hops to the peer —
   functionally correct via the proxies, but it contradicts the design docs' "one active node runs
   all central components" model (Component-ClusterInfrastructure.md:44-47) and doubles
   intra-cluster latency on every outbox/audit page load.

Recommendation: pick one definition. Simplest coherent choice: derive "active" from singleton-host
status (e.g. compare self against the oldest `Up` member, or gate on reachability of a local
singleton), and use it consistently for `ActiveNodeGate`, `SelfIsPrimary`, the purge gate, and the
`/health/active` check — leadership is an Akka-internal concept that only accidentally matches the
product's active/standby model.

### [High] During a central network partition, both nodes report `/health/active = 200` and Traefik serves both

`ActiveNodeGate`/the active-node check compute `cluster.State.Leader` from each node's own gossip
view. When central-a and central-b cannot see each other but Traefik can reach both (a plausible
partial partition), each side sees the peer as unreachable and computes **itself** as leader — both
return 200 on `/health/active`, and Traefik's load balancer (docker/traefik/dynamic.yml:10-18)
round-robins requests between two nodes that each believe they are active. Blazor circuits and the
Inbound API's active-node 503 gate (Program.cs:256-260) are then split across both. Because the SBR
is not enabled (Critical finding above), nothing ever resolves the partition — this state persists
until manual intervention. Singletons do *not* duplicate (that requires member removal), but every
leader-gated behavior does. Enabling keep-oldest + `down-if-alone` fixes the persistence; consider
also making `/health/active` require `Ready` (DB reachable) in addition to leadership so a
DB-dead leader is pulled from rotation.

### [Medium] Cluster-leave drain tasks are budgeted 10s inside a 5s CoordinatedShutdown phase

Five singletons add a `PhaseClusterLeave` task that runs `GracefulStop(TimeSpan.FromSeconds(10))`
(AkkaHostedService.cs:551-567 site-call-audit, 624-641 audit-log-purge, 676-693
site-audit-reconciliation, 733-750 kpi-history-recorder, 787-804 pending-deployment-purge). Akka's
`cluster-leave` phase uses the default phase timeout of **5 seconds**; the phase abandons the task
at 5s (default `recover = on`) and proceeds, so the 10s graceful-stop can never complete its full
window and the `catch` logging in those tasks never observes the overrun (the task is orphaned, not
faulted). The intended "let the in-flight EF upsert drain before handover" guarantee is silently
halved. Fix: set `akka.coordinated-shutdown.phases.cluster-leave.timeout` in `BuildHocon` to
exceed the graceful-stop budget, or shrink the `GracefulStop` timeout below 5s.

### [Medium] The two busiest singletons have no drain task at all

The `NotificationOutboxActor` and `AuditLogIngestActor` singletons are terminated by bare
`PoisonPill` with no cluster-leave drain (AkkaHostedService.cs:410-419, 451-458); the comment at
AkkaHostedService.cs:544-549 explicitly notes the pattern "is suitable for the NotificationOutbox
singleton; not added here to keep this change minimal". These are precisely the actors with
in-flight EF transactions on every message (dispatch sweeps, dual-write ingest). The idempotent
DB design (insert-if-not-exists on `NotificationId`/`EventId`) makes this safe-but-lossy rather
than corrupting, but the inconsistency (drain for the low-traffic maintenance singletons, none for
the hot ones) should be resolved.

### [Medium] Health-report loss recovery is dead code on the production transport

`HealthReportSender` carefully restores interval counters if `_transport.Send` throws
(HealthReportSender.cs:158-175), and `SiteHealthCollector.AddIntervalCounters`
(SiteHealthCollector.cs:153-171) exists to support it. But the only production transport is
`AkkaHealthReportTransport.Send` (src/ZB.MOM.WW.ScadaBridge.Host/AkkaHealthReportTransport.cs:26-33)
— a fire-and-forget `Tell` to an `ActorSelection`, which **never throws** and silently returns when
the actor system is null. Any downstream loss (central unreachable, ClusterClient reconnecting,
dead-lettered selection) drops the report *and* the interval error counters it carried — the
scenario the restore logic was built for is exactly the one it can't see. Either make the transport
Ask-with-ack (the site→central notification forward already uses this pattern over the same actor)
or accept the loss and delete the restore machinery; the current shape gives false confidence.

### [Medium] Offline detection deviates from the spec and can mask a dead metrics pipeline

`Component-HealthMonitoring.md:43-46` defines offline as "no *report* within 60s".
`CentralHealthAggregator.CheckForOfflineSites` (CentralHealthAggregator.cs:224-263) instead keys
off `LastHeartbeatAt`, which is also advanced by the ~5s transport heartbeats
(`CentralCommunicationActor.HandleHeartbeat` → `MarkHeartbeat`, CentralCommunicationActor.cs:349-353).
A site whose `HealthReportSender` loop has died (or whose DeploymentManager singleton is stuck, so
`IsActiveNode` is false on both nodes and reports are skipped — HealthReportSender.cs:85) keeps
heartbeating and shows **online with silently frozen metrics forever**; nothing surfaces
`LastReportReceivedAt` staleness. The heartbeat-based liveness choice is defensible (the code
comment argues it), but then the dashboard needs a distinct "metrics stale" signal, and the design
doc should be updated — currently spec and code disagree.

### [Medium] Second seed node in the checked-in site dev config targets the metrics port; validator has a gap

`src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json:14-17` seeds
`akka.tcp://scadabridge@localhost:8082` and `akka.tcp://scadabridge@localhost:8084`, while the same
file sets `MetricsPort: 8084` (line 10). The second seed points at the Kestrel HTTP/1.1 metrics
listener — a node attempting to join via that seed performs a doomed Akka.Remote association
against Kestrel (handshake noise, retry loops). `StartupValidator` explicitly guards seed-vs-GrpcPort
(StartupValidator.cs:121-127) but not seed-vs-MetricsPort, so this config validates cleanly. Fix
the dev config (8084 → the intended second node's remoting port) and extend the seed-port loop to
also reject `MetricsPort`.

### [Medium] Docker stop grace period is shorter than a full CoordinatedShutdown

`docker/docker-compose.yml` sets no `stop_grace_period` (default 10s SIGTERM→SIGKILL) and the
Dockerfile runs `dotnet` as PID 1 (docker/Dockerfile, final stage). A graceful shutdown runs
cluster-leave (5s budget) + cluster-exiting (10s) + actor-system-terminate plus Serilog flush —
easily exceeding 10s on a loaded node, so `bash docker/deploy.sh` recreates can SIGKILL mid-drain,
turning every "graceful" redeploy into the crash path (which, per the Critical finding, currently
has no automatic recovery on the surviving node). Add `stop_grace_period: 30s` to the app services.

### [Low] Aggregator never forgets sites

`CentralHealthAggregator._siteStates` (CentralHealthAggregator.cs:16) only grows: a site deleted
from configuration remains a permanently-offline dashboard tile (and continues to be enumerated by
`SiteHealthKpiSampleSource.CollectAsync`, SiteHealthKpiSampleSource.cs:74-85) until process
restart. Needs an eviction path wired to site deletion.

### [Low] Dead-letter monitor: unbounded warning volume and restart-reset count

`DeadLetterMonitorActor` logs one Warning per dead letter with no sampling/rate-limiting and keeps
its count in actor state that resets on restart
(src/ZB.MOM.WW.ScadaBridge.Host/Actors/DeadLetterMonitorActor.cs:14,26-35). A dead-letter storm
(mass undeploy, failover, shutdown) floods the log; the health metric path
(`IncrementDeadLetter`) is fine because the collector counter is interval-based.

### [Low] Mandatory two-seed rule forces phantom seeds in single-node installs

`ClusterOptionsValidator` requires ≥2 seed nodes
(ClusterOptionsValidator.cs:29-33); the shipped single-node production artifact satisfies it with
a documented placeholder (`deploy/wonder-app-vd03/appsettings.Central.json:9-19`, second seed
`localhost:8091` "harmless while absent"). The node dials the phantom seed forever (log noise,
periodic connect attempts), and the rule's intent — no startup-ordering dependency — is defeated
by fiction. Consider allowing 1 seed with a warning, or a `SingleNode` acknowledgement flag.

## 2. Performance

This domain is largely cold-path and correctly engineered for it; nothing here touches the
data-plane hot path. Observations:

- **[Low] Readiness probes fan out five cluster Asks per poll.**
  `RequiredSingletonsHealthCheck` sends 5 concurrent `Identify` Asks through singleton proxies
  every `/health/ready` poll (RequiredSingletonsHealthCheck.cs:66-78,111-117). When the singletons
  live on the peer node each probe is a remoting round trip. Bounded (2s, concurrent) and cheap at
  typical poll rates, but a tight orchestration probe interval multiplies cluster chatter.
- **[Low] Startup compiles every inbound API method sequentially before serving**
  (Program.cs:379-388). Startup latency scales linearly with method count; a slow Roslyn compile of
  many methods delays readiness on the *standby* too (it runs the same branch). Acceptable now;
  parallelize or defer-compile if method counts grow.
- **[Low] Per-tick allocations in health collection** (dictionary snapshots per 30s report,
  `GetAllSiteStates` copy per dashboard poll — CentralHealthAggregator.cs:173-176) are trivial at
  this cadence. No action.
- **Positive:** the site audit telemetry actor is pinned to a dedicated dispatcher
  (`audit-telemetry-dispatcher`, AkkaHostedService.cs:226-232,1150-1158) so batch SQLite/gRPC work
  never contends with hot-path actors; `StoreAndForwardService.StartAsync` is awaited rather than
  sync-blocked (AkkaHostedService.cs:976-981); the ActorSystem DI bridge is a root singleton
  precisely to avoid per-scope disposal (Program.cs:240-250) — all correct.

## 3. Conventions

- **[Medium] REQ-HOST-6 says "Akka.NET bootstrap via Akka.Hosting"; the code hand-rolls HOCON.**
  `Component-Host.md` (REQ-HOST-6, line 106-114) mandates Akka.Hosting; the actual bootstrap is
  string-interpolated HOCON + `ActorSystem.Create` (AkkaHostedService.cs:171-176,214-271) with
  `Akka.Hosting`/`Akka.Cluster.Hosting`/`Akka.Remote.Hosting` referenced but unused for cluster
  bring-up (ZB.MOM.WW.ScadaBridge.Host.csproj:15-18). This is not just doc drift: Akka.Cluster.Hosting's
  typed `SplitBrainResolverOption` sets `downing-provider-class` for you — using the mandated API
  would have prevented the Critical finding. Either implement REQ-HOST-6 or amend it and drop the
  unused packages.
- **[High] Production deploy artifact ships with authentication disabled.**
  `deploy/wonder-app-vd03/appsettings.Central.json:39-42` sets `"DisableLogin": true` — every
  request auto-authenticates as `multi-role` with all roles, and the management API on that host is
  unauthenticated. The base config documents this as "DEV/TEST ONLY — never enable in production"
  with "no environment guard; a startup warning is the only protection"
  (src/ZB.MOM.WW.ScadaBridge.Host/appsettings.json:6). A `deploy/` artifact with an install
  runbook is a production surface; at minimum gate `DisableLogin` on
  `Environment.IsDevelopment()` or require an explicit `I-understand` companion flag.
- **[Medium] LDAP transport is plaintext everywhere, contradicting the security spec.** The design
  requires "LDAPS/StartTLS required" (CLAUDE.md Security & Auth; Component-Security), but every
  shipped config — docker (docker/central-node-a/appsettings.Central.json:26-33) and the on-host
  deploy (deploy/wonder-app-vd03/appsettings.Central.json:26-34) — sets `"Transport": "None",
  "AllowInsecure": true` and sends the service-account bind password in clear. Dev GLAuth is a fair
  excuse for docker; the deploy artifact should not normalize it.
- **[Medium] Secrets in checked-in dev configs.** SQL credentials, LDAP service-account password
  and the JWT signing key are committed in `docker/central-node-a/appsettings.Central.json:21-38`
  (labelled dev-only; the compose pepper is likewise flagged, docker-compose.yml:9-14). The
  wonder-app-vd03 artifact handles this correctly via `${...}` placeholders + env injection
  (appsettings.Central.json:20-24) — good pattern, worth backporting a scrubbed variant to docker
  if these ever leave a lab network. Also note `ConfigSecretsTests.cs` exists in Host.Tests, so the
  team is tracking this.
- **[Low] Stale doc references.** `Component-TraefikProxy.md:121` locates `ActiveNodeHealthCheck`
  at `src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeHealthCheck.cs`; the check now comes from the
  shared `ZB.MOM.WW.Health.Akka` package (Program.cs:214-234) and no such file exists.
  `Component-TraefikProxy.md:115` also still describes `/health/ready` as "database + Akka
  cluster" — it now includes `required-singletons` (Program.cs:227-230; Component-Host.md REQ-HOST-4a
  *is* current). Keep the Traefik doc in sync.
- **[Low] Five copy-pasted ~60-line singleton registration blocks.**
  AkkaHostedService.RegisterCentralActors repeats the manager/drain-task/proxy triple five times
  (AkkaHostedService.cs:399-811) with only names and Props varying. Extract a
  `StartCentralSingleton(name, props, drainTimeout)` helper — the copy-paste already caused drift
  (outbox/ingest lack drain tasks).
- **Positive conventions compliance:** Options pattern is followed rigorously (component-owned
  options classes, `ValidateOnStart`, dedicated validators with key-naming messages —
  ClusterOptionsValidator, HealthMonitoringOptionsValidator); HOCON interpolation is
  injection-safe (`QuoteHocon`/`DurationHocon`, AkkaHostedService.cs:280-295); Serilog enrichment
  matches REQ-HOST-8; startup validation matches REQ-HOST-4 including cross-field port-collision
  rules; single-binary role dispatch matches REQ-HOST-1/2; the `/health/ready` vs `/health/active`
  tier separation (leadership deliberately excluded from readiness) is well-reasoned and
  documented in place (Program.cs:346-356).

## 4. Underdeveloped Areas

1. **No real multi-node failover test anywhere.** `CentralFailoverTests` are single-process
   simulations (JWT across two service instances, CentralFailoverTests.cs:27-47);
   `SiteFailoverTests`/`DualNodeRecoveryTests` similarly exercise component state recovery, not
   cluster membership; `CoordinatedShutdownTests` grep source text (CoordinatedShutdownTests.cs:11-43);
   `HoconBuilderTests` assert config strings. The repo's own notes describe the MultiNode test
   runner (AkkaDotNet/18-MultiNodeTestRunner.md) but it is unused. This is the direct cause of the
   Critical finding surviving: nothing ever killed the oldest node and waited for the singleton to
   move. Even a plain xunit test spinning two `ActorSystem`s in-process and calling
   `cluster.Down()`/terminating one would have caught it.
2. **Windows Service lifecycle management is thin.** `Component-ClusterInfrastructure.md:19`
   claims "Manage Windows service lifecycle (start, stop, restart)"; the implementation is
   `UseWindowsService()` (Program.cs:74,401) plus install scripts. Service recovery actions
   (restart-on-failure), the watchdog implied by `down-if-alone` ("an external mechanism must
   restart both nodes" — AkkaDotNet/03-Cluster.md:64) do not exist in the repo; after a
   down-if-alone self-down the process presumably exits (or worse, idles) with nothing to restart
   it. The down-if-alone recovery story needs an explicit design (does the process exit? does the
   service restart it? does it rejoin cleanly?) and a drill.
3. **TLS is absent at every layer of this domain**: Traefik terminates plain HTTP with an
   unauthenticated dashboard (docker/traefik/traefik.yml:5-7 — documented as dev-only in
   Component-TraefikProxy.md:131-138), Akka remoting is unencrypted TCP, site gRPC is `http://`.
   Fine for the lab; there is no production TLS profile or documented path yet.
4. **Dockerfile has no `HEALTHCHECK`** and compose has no health-based dependency ordering;
   liveness `/healthz` runs zero checks (Program.cs:353-355) so a wedged actor system still
   reports live. Acceptable minimalism, but an orchestrator migration will need both.
5. **`NodeName` is missing from the wonder-app-vd03 deploy overlays**
   (deploy/wonder-app-vd03/appsettings.Central.json has no `Node.NodeName`), so audit rows from
   that install get a NULL `SourceNode` (the base file documents the normalisation,
   src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Central.json:3) — undermines the per-node audit/KPI
   features (`IX_AuditLog_Node_Occurred`, per-node stuck KPIs) on the one real deployment.
6. **Known-but-open follow-ups acknowledged in code/docs:** SecuredWrite audit rows leave
   `SourceNode` NULL (CLAUDE.md notes it as logged); no central persistence of OPC UA cert trust;
   `docker-env2` mirrors the main topology (verified: same Traefik config shape, same
   seed/port conventions) so it inherits every finding above.
7. **Health monitoring has no history by design** (Component-HealthMonitoring.md:93-97) — now
   partially superseded by KPI History (#26), which the docs handle; but online/offline
   *transitions* are still unrecorded anywhere (doc line 97 defers it), so post-incident "when did
   the site drop" questions remain unanswerable beyond log scraping.

## Prioritized Recommendations

1. **[Critical, one line + tests] Enable the split-brain resolver.** Add
   `downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster"` to
   `BuildHocon` (AkkaHostedService.cs:250) — or move the bootstrap to Akka.Cluster.Hosting per
   REQ-HOST-6, which sets it via typed options. Then add a two-node in-process failover test
   (kill oldest → assert singleton re-hosted, member removed within stable-after + margin) and a
   documented manual drill against the docker cluster (`docker kill scadabridge-central-a`, watch
   Traefik + singleton logs).
2. **[High] Unify the "active node" definition** on singleton-host (oldest) semantics for
   `ActiveNodeGate`, `AkkaClusterNodeProvider.SelfIsPrimary`, the event-log purge gate, and the
   `/health/active` check; today's leader-based checks purge the wrong node's event log and label
   the wrong node Primary once leader ≠ oldest.
3. **[High] Remove `DisableLogin: true` from `deploy/wonder-app-vd03`** or add a hard environment
   guard around the flag.
4. **[Medium] Fix the CoordinatedShutdown drain budget** (`cluster-leave` phase timeout ≥
   graceful-stop timeout) and add drain tasks for the notification-outbox and audit-log-ingest
   singletons — or delete all five drains if the idempotent-DB argument makes them unnecessary.
5. **[Medium] Fix `appsettings.Site.json` seed `localhost:8084`** and extend `StartupValidator`'s
   seed-port loop to reject `MetricsPort` collisions.
6. **[Medium] Make health-report delivery observable**: switch `AkkaHealthReportTransport` to an
   acked Ask (reusing the existing site→central forward pattern) so the counter-restore logic in
   `HealthReportSender` actually fires, and surface `LastReportReceivedAt` staleness on the
   dashboard so heartbeat-alive/metrics-dead sites are visible.
7. **[Medium] Add `stop_grace_period: 30s`** to the app services in both compose files.
8. **[Low] Housekeeping:** evict deleted sites from `CentralHealthAggregator`; rate-limit
   dead-letter warnings; extract the singleton-registration helper; sync
   `Component-TraefikProxy.md` with the shared health-check library; add `NodeName` to the
   wonder-app-vd03 overlays; reconsider the mandatory-two-seeds rule for single-node installs.
