dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases Wiki Activity
Files
9e60347bde5772273819c4e82a6ae48b6bf56eab
ScadaBridge/archreview/01-cluster-host-failover.md
T

28 KiB
Raw Blame History

Architecture Review 01 — Cluster Infrastructure, Host, Failover, Health Monitoring, Deployment Topology

Reviewer domain: src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure, src/ZB.MOM.WW.ScadaBridge.Host, src/ZB.MOM.WW.ScadaBridge.HealthMonitoring, docker/, docker-env2/, deploy/wonder-app-vd03/, Traefik config, and the corresponding design docs (Component-ClusterInfrastructure.md, Component-Host.md, Component-HealthMonitoring.md, Component-TraefikProxy.md) and test projects.

Scope

Read in full: AkkaHostedService.cs (1,203 lines — HOCON builder, singleton wiring, coordinated shutdown), Program.cs (both role branches), StartupValidator.cs, StartupRetry.cs, SiteServiceRegistration.cs, NodeOptions.cs, ClusterOptions.cs + validator + ServiceCollectionExtensions.cs, all Host Health/ checks (RequiredSingletonsHealthCheck, ActiveNodeGate, AkkaClusterNodeProvider), DeadLetterMonitorActor.cs, AkkaHealthReportTransport.cs, the entire HealthMonitoring project (HealthReportSender, CentralHealthAggregator, CentralHealthReportLoop, SiteHealthCollector, SiteEventLogFailureCountReporter, SiteHealthKpiSampleSource, options + validator), the docker compose topologies + per-node appsettings + Traefik static/dynamic config + Dockerfile, the on-host deploy artifacts (deploy/wonder-app-vd03/appsettings.*.json), the four design docs, the repo's own Akka.NET reference notes (AkkaDotNet/03-Cluster.md), and the relevant test projects (Host.Tests, ClusterInfrastructure.Tests, HealthMonitoring.Tests, IntegrationTests failover/readiness suites). Health-report ingestion on the central side was cross-checked in Communication/Actors/CentralCommunicationActor.cs:349-392.

Maturity Verdict

This slice of the codebase is unusually well-commented, defensively coded at the micro level (options validators with ValidateOnStart, CAS loops in the aggregator, sequence-number seeding for failover, HOCON escaping, bounded startup retry), and clearly the product of many careful review passes — but it has one macro-level defect that undoes the entire failover design: the split-brain resolver is configured but never enabled (no akka.cluster.downing-provider-class), so unreachable nodes are never automatically downed and cluster singletons never migrate on a hard crash. Every document, comment, and validator in the domain reasons meticulously about keep-oldest semantics that are inert at runtime, and the test suite asserts the HOCON strings rather than the behavior, which is exactly how the gap survived. Secondary structural issues cluster around the ambiguous definition of "active node" (cluster leader vs. singleton-hosting oldest member, which diverge after restarts) and dev-vs-prod configuration drift. Verdict: high polish, one critical foundation crack; not production-ready for unattended failover until the downing provider is fixed and verified with a real two-node test.

1. Stability

[Critical] Split-brain resolver is never enabled — automatic failover does not work for crashes/partitions

AkkaHostedService.BuildHocon (src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs:225-270) emits:

cluster {
    ...
    split-brain-resolver {
        active-strategy = "keep-oldest"
        stable-after = 15000ms
        keep-oldest { down-if-alone = on }
    }
    failure-detector { ... }
    run-coordinated-shutdown-when-down = on
}

but never sets akka.cluster.downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster". In Akka.NET (1.5.62 per Directory.Packages.props:8-17), the default downing provider is NoDowning; the whole split-brain-resolver section is only read by SplitBrainResolverProvider, which must be opted into via downing-provider-class. A repo-wide grep confirms the string SplitBrainResolverProvider appears nowhere in src/ — only in the project's own reference notes, which show the required line (AkkaDotNet/03-Cluster.md:56, also 15-Coordination.md:69 and 18-MultiNodeTestRunner.md:100).

Failure scenario: the active site node loses power. The standby's failure detector marks it unreachable (~10s), but nothing ever downs it. The unreachable member stays in the membership forever, the ClusterSingletonManager cannot hand over (singleton migration requires the oldest member's removal, not mere unreachability), and the site's DeploymentManager singleton — and with it all data collection, script execution, and S&F delivery — never restarts on the survivor. The same applies to all six central singletons (notification-outbox, audit-log-ingest, site-call-audit, audit-log-purge, site-audit-reconciliation, kpi-history-recorder). The documented "~25 seconds total failover" (docs/requirements/Component-ClusterInfrastructure.md:110-119) is only achievable today for a graceful stop (CoordinatedShutdown leave); crash and partition scenarios hang indefinitely until an operator downs the node or restarts the survivor.

Corroborating evidence that this was never exercised:

  • HoconBuilderTests.cs:45,112,155 assert only that the SBR strings appear in the config.
  • CoordinatedShutdownTests.cs:11-43 are source-text greps (Assert.Contains("run-by-clr-shutdown-hook = on", content)).
  • The "failover" integration tests are single-process simulations — e.g. CentralFailoverTests.cs:27-47 validates a JWT across two JwtTokenService instances; no test forms a two-node cluster and kills the oldest node.
  • run-coordinated-shutdown-when-down = on (AkkaHostedService.cs:265) is dead config: a node is never transitioned to Down by anything.
  • ClusterOptionsValidator (src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptionsValidator.cs:35-63) rigorously rejects any strategy other than keep-oldest and demands DownIfAlone == true — validating knobs on a resolver that is not installed.

Fix: add the downing-provider-class line to BuildHocon (one line), or better, migrate the bootstrap to Akka.Cluster.Hosting's typed options (already referenced, ZB.MOM.WW.ScadaBridge.Host.csproj:15) whose SplitBrainResolverOption wires the provider automatically — and add a real two-node in-process cluster test that kills the oldest member and asserts the singleton re-hosts within the expected window.

[High] "Active node" means two different things — cluster leader vs. oldest member — and they diverge

Two distinct node-selection concepts are used interchangeably:

  • Leader-based (address ordering, not age): ActiveNodeGate.IsActiveNode (src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeGate.cs:36-52), AkkaClusterNodeProvider.SelfIsPrimary and the "Primary"/"Standby" labels (src/ZB.MOM.WW.ScadaBridge.Host/Health/AkkaClusterNodeProvider.cs:29-40,61), the active-node health check Traefik routes on (Program.cs:231-234, docker/traefik/dynamic.yml:12-18), the CentralHealthReportLoop leader gate (CentralHealthReportLoop.cs:93-97), and the site event-log purge gate (SiteEventLogActiveNodeCheck → SelfIsPrimary, SiteServiceRegistration.cs:116-120).
  • Oldest-based (singleton placement): every ClusterSingletonManager (AkkaHostedService.cs:410+, 906-925), and the site health "IsActiveNode" flag, which is set by the DeploymentManager singleton on start/stop (src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs:243,258).

In Akka the leader is the lowest-address Up member; the singleton host is the oldest member. They coincide at first boot but diverge permanently after the original first node is restarted and rejoins (it regains leadership by address order while the peer remains oldest). Concrete consequences once diverged:

  1. Site event-log retention is unenforced on the node that writes events. Events are recorded by the singleton hierarchy on the oldest node; the purge (EventLogPurgeService.cs:21,36) early-exits unless SelfIsPrimary (leader). The active/oldest node's SQLite grows past the 30-day/1GB cap while the leader purges an idle database.
  2. Health dashboard labels the wrong node "Primary" (AkkaClusterNodeProvider.cs:61): the node named Primary hosts no singletons; the node doing all the work shows Standby.
  3. Central self-report is emitted by the leader (CentralHealthReportLoop.cs:93) while the NotificationOutbox/audit singletons run on the other node — the $central health card describes the wrong node.
  4. Traefik routes UI/API to the leader while every singleton Ask then hops to the peer — functionally correct via the proxies, but it contradicts the design docs' "one active node runs all central components" model (Component-ClusterInfrastructure.md:44-47) and doubles intra-cluster latency on every outbox/audit page load.

Recommendation: pick one definition. Simplest coherent choice: derive "active" from singleton-host status (e.g. compare self against the oldest Up member, or gate on reachability of a local singleton), and use it consistently for ActiveNodeGate, SelfIsPrimary, the purge gate, and the /health/active check — leadership is an Akka-internal concept that only accidentally matches the product's active/standby model.

[High] During a central network partition, both nodes report /health/active = 200 and Traefik serves both

ActiveNodeGate/the active-node check compute cluster.State.Leader from each node's own gossip view. When central-a and central-b cannot see each other but Traefik can reach both (a plausible partial partition), each side sees the peer as unreachable and computes itself as leader — both return 200 on /health/active, and Traefik's load balancer (docker/traefik/dynamic.yml:10-18) round-robins requests between two nodes that each believe they are active. Blazor circuits and the Inbound API's active-node 503 gate (Program.cs:256-260) are then split across both. Because the SBR is not enabled (Critical finding above), nothing ever resolves the partition — this state persists until manual intervention. Singletons do not duplicate (that requires member removal), but every leader-gated behavior does. Enabling keep-oldest + down-if-alone fixes the persistence; consider also making /health/active require Ready (DB reachable) in addition to leadership so a DB-dead leader is pulled from rotation.

[Medium] Cluster-leave drain tasks are budgeted 10s inside a 5s CoordinatedShutdown phase

Five singletons add a PhaseClusterLeave task that runs GracefulStop(TimeSpan.FromSeconds(10)) (AkkaHostedService.cs:551-567 site-call-audit, 624-641 audit-log-purge, 676-693 site-audit-reconciliation, 733-750 kpi-history-recorder, 787-804 pending-deployment-purge). Akka's cluster-leave phase uses the default phase timeout of 5 seconds; the phase abandons the task at 5s (default recover = on) and proceeds, so the 10s graceful-stop can never complete its full window and the catch logging in those tasks never observes the overrun (the task is orphaned, not faulted). The intended "let the in-flight EF upsert drain before handover" guarantee is silently halved. Fix: set akka.coordinated-shutdown.phases.cluster-leave.timeout in BuildHocon to exceed the graceful-stop budget, or shrink the GracefulStop timeout below 5s.

[Medium] The two busiest singletons have no drain task at all

The NotificationOutboxActor and AuditLogIngestActor singletons are terminated by bare PoisonPill with no cluster-leave drain (AkkaHostedService.cs:410-419, 451-458); the comment at AkkaHostedService.cs:544-549 explicitly notes the pattern "is suitable for the NotificationOutbox singleton; not added here to keep this change minimal". These are precisely the actors with in-flight EF transactions on every message (dispatch sweeps, dual-write ingest). The idempotent DB design (insert-if-not-exists on NotificationId/EventId) makes this safe-but-lossy rather than corrupting, but the inconsistency (drain for the low-traffic maintenance singletons, none for the hot ones) should be resolved.

[Medium] Health-report loss recovery is dead code on the production transport

HealthReportSender carefully restores interval counters if _transport.Send throws (HealthReportSender.cs:158-175), and SiteHealthCollector.AddIntervalCounters (SiteHealthCollector.cs:153-171) exists to support it. But the only production transport is AkkaHealthReportTransport.Send (src/ZB.MOM.WW.ScadaBridge.Host/AkkaHealthReportTransport.cs:26-33) — a fire-and-forget Tell to an ActorSelection, which never throws and silently returns when the actor system is null. Any downstream loss (central unreachable, ClusterClient reconnecting, dead-lettered selection) drops the report and the interval error counters it carried — the scenario the restore logic was built for is exactly the one it can't see. Either make the transport Ask-with-ack (the site→central notification forward already uses this pattern over the same actor) or accept the loss and delete the restore machinery; the current shape gives false confidence.

[Medium] Offline detection deviates from the spec and can mask a dead metrics pipeline

Component-HealthMonitoring.md:43-46 defines offline as "no report within 60s". CentralHealthAggregator.CheckForOfflineSites (CentralHealthAggregator.cs:224-263) instead keys off LastHeartbeatAt, which is also advanced by the ~5s transport heartbeats (CentralCommunicationActor.HandleHeartbeat → MarkHeartbeat, CentralCommunicationActor.cs:349-353). A site whose HealthReportSender loop has died (or whose DeploymentManager singleton is stuck, so IsActiveNode is false on both nodes and reports are skipped — HealthReportSender.cs:85) keeps heartbeating and shows online with silently frozen metrics forever; nothing surfaces LastReportReceivedAt staleness. The heartbeat-based liveness choice is defensible (the code comment argues it), but then the dashboard needs a distinct "metrics stale" signal, and the design doc should be updated — currently spec and code disagree.

[Medium] Second seed node in the checked-in site dev config targets the metrics port; validator has a gap

src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json:14-17 seeds akka.tcp://scadabridge@localhost:8082 and akka.tcp://scadabridge@localhost:8084, while the same file sets MetricsPort: 8084 (line 10). The second seed points at the Kestrel HTTP/1.1 metrics listener — a node attempting to join via that seed performs a doomed Akka.Remote association against Kestrel (handshake noise, retry loops). StartupValidator explicitly guards seed-vs-GrpcPort (StartupValidator.cs:121-127) but not seed-vs-MetricsPort, so this config validates cleanly. Fix the dev config (8084 → the intended second node's remoting port) and extend the seed-port loop to also reject MetricsPort.

[Medium] Docker stop grace period is shorter than a full CoordinatedShutdown

docker/docker-compose.yml sets no stop_grace_period (default 10s SIGTERM→SIGKILL) and the Dockerfile runs dotnet as PID 1 (docker/Dockerfile, final stage). A graceful shutdown runs cluster-leave (5s budget) + cluster-exiting (10s) + actor-system-terminate plus Serilog flush — easily exceeding 10s on a loaded node, so bash docker/deploy.sh recreates can SIGKILL mid-drain, turning every "graceful" redeploy into the crash path (which, per the Critical finding, currently has no automatic recovery on the surviving node). Add stop_grace_period: 30s to the app services.

[Low] Aggregator never forgets sites

CentralHealthAggregator._siteStates (CentralHealthAggregator.cs:16) only grows: a site deleted from configuration remains a permanently-offline dashboard tile (and continues to be enumerated by SiteHealthKpiSampleSource.CollectAsync, SiteHealthKpiSampleSource.cs:74-85) until process restart. Needs an eviction path wired to site deletion.

[Low] Dead-letter monitor: unbounded warning volume and restart-reset count

DeadLetterMonitorActor logs one Warning per dead letter with no sampling/rate-limiting and keeps its count in actor state that resets on restart (src/ZB.MOM.WW.ScadaBridge.Host/Actors/DeadLetterMonitorActor.cs:14,26-35). A dead-letter storm (mass undeploy, failover, shutdown) floods the log; the health metric path (IncrementDeadLetter) is fine because the collector counter is interval-based.

[Low] Mandatory two-seed rule forces phantom seeds in single-node installs

ClusterOptionsValidator requires ≥2 seed nodes (ClusterOptionsValidator.cs:29-33); the shipped single-node production artifact satisfies it with a documented placeholder (deploy/wonder-app-vd03/appsettings.Central.json:9-19, second seed localhost:8091 "harmless while absent"). The node dials the phantom seed forever (log noise, periodic connect attempts), and the rule's intent — no startup-ordering dependency — is defeated by fiction. Consider allowing 1 seed with a warning, or a SingleNode acknowledgement flag.

2. Performance

This domain is largely cold-path and correctly engineered for it; nothing here touches the data-plane hot path. Observations:

  • [Low] Readiness probes fan out five cluster Asks per poll. RequiredSingletonsHealthCheck sends 5 concurrent Identify Asks through singleton proxies every /health/ready poll (RequiredSingletonsHealthCheck.cs:66-78,111-117). When the singletons live on the peer node each probe is a remoting round trip. Bounded (2s, concurrent) and cheap at typical poll rates, but a tight orchestration probe interval multiplies cluster chatter.
  • [Low] Startup compiles every inbound API method sequentially before serving (Program.cs:379-388). Startup latency scales linearly with method count; a slow Roslyn compile of many methods delays readiness on the standby too (it runs the same branch). Acceptable now; parallelize or defer-compile if method counts grow.
  • [Low] Per-tick allocations in health collection (dictionary snapshots per 30s report, GetAllSiteStates copy per dashboard poll — CentralHealthAggregator.cs:173-176) are trivial at this cadence. No action.
  • Positive: the site audit telemetry actor is pinned to a dedicated dispatcher (audit-telemetry-dispatcher, AkkaHostedService.cs:226-232,1150-1158) so batch SQLite/gRPC work never contends with hot-path actors; StoreAndForwardService.StartAsync is awaited rather than sync-blocked (AkkaHostedService.cs:976-981); the ActorSystem DI bridge is a root singleton precisely to avoid per-scope disposal (Program.cs:240-250) — all correct.

3. Conventions

  • [Medium] REQ-HOST-6 says "Akka.NET bootstrap via Akka.Hosting"; the code hand-rolls HOCON. Component-Host.md (REQ-HOST-6, line 106-114) mandates Akka.Hosting; the actual bootstrap is string-interpolated HOCON + ActorSystem.Create (AkkaHostedService.cs:171-176,214-271) with Akka.Hosting/Akka.Cluster.Hosting/Akka.Remote.Hosting referenced but unused for cluster bring-up (ZB.MOM.WW.ScadaBridge.Host.csproj:15-18). This is not just doc drift: Akka.Cluster.Hosting's typed SplitBrainResolverOption sets downing-provider-class for you — using the mandated API would have prevented the Critical finding. Either implement REQ-HOST-6 or amend it and drop the unused packages.
  • [High] Production deploy artifact ships with authentication disabled. deploy/wonder-app-vd03/appsettings.Central.json:39-42 sets "DisableLogin": true — every request auto-authenticates as multi-role with all roles, and the management API on that host is unauthenticated. The base config documents this as "DEV/TEST ONLY — never enable in production" with "no environment guard; a startup warning is the only protection" (src/ZB.MOM.WW.ScadaBridge.Host/appsettings.json:6). A deploy/ artifact with an install runbook is a production surface; at minimum gate DisableLogin on Environment.IsDevelopment() or require an explicit I-understand companion flag.
  • [Medium] LDAP transport is plaintext everywhere, contradicting the security spec. The design requires "LDAPS/StartTLS required" (CLAUDE.md Security & Auth; Component-Security), but every shipped config — docker (docker/central-node-a/appsettings.Central.json:26-33) and the on-host deploy (deploy/wonder-app-vd03/appsettings.Central.json:26-34) — sets "Transport": "None", "AllowInsecure": true and sends the service-account bind password in clear. Dev GLAuth is a fair excuse for docker; the deploy artifact should not normalize it.
  • [Medium] Secrets in checked-in dev configs. SQL credentials, LDAP service-account password and the JWT signing key are committed in docker/central-node-a/appsettings.Central.json:21-38 (labelled dev-only; the compose pepper is likewise flagged, docker-compose.yml:9-14). The wonder-app-vd03 artifact handles this correctly via ${...} placeholders + env injection (appsettings.Central.json:20-24) — good pattern, worth backporting a scrubbed variant to docker if these ever leave a lab network. Also note ConfigSecretsTests.cs exists in Host.Tests, so the team is tracking this.
  • [Low] Stale doc references. Component-TraefikProxy.md:121 locates ActiveNodeHealthCheck at src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeHealthCheck.cs; the check now comes from the shared ZB.MOM.WW.Health.Akka package (Program.cs:214-234) and no such file exists. Component-TraefikProxy.md:115 also still describes /health/ready as "database + Akka cluster" — it now includes required-singletons (Program.cs:227-230; Component-Host.md REQ-HOST-4a is current). Keep the Traefik doc in sync.
  • [Low] Five copy-pasted ~60-line singleton registration blocks. AkkaHostedService.RegisterCentralActors repeats the manager/drain-task/proxy triple five times (AkkaHostedService.cs:399-811) with only names and Props varying. Extract a StartCentralSingleton(name, props, drainTimeout) helper — the copy-paste already caused drift (outbox/ingest lack drain tasks).
  • Positive conventions compliance: Options pattern is followed rigorously (component-owned options classes, ValidateOnStart, dedicated validators with key-naming messages — ClusterOptionsValidator, HealthMonitoringOptionsValidator); HOCON interpolation is injection-safe (QuoteHocon/DurationHocon, AkkaHostedService.cs:280-295); Serilog enrichment matches REQ-HOST-8; startup validation matches REQ-HOST-4 including cross-field port-collision rules; single-binary role dispatch matches REQ-HOST-1/2; the /health/ready vs /health/active tier separation (leadership deliberately excluded from readiness) is well-reasoned and documented in place (Program.cs:346-356).

4. Underdeveloped Areas

  1. No real multi-node failover test anywhere. CentralFailoverTests are single-process simulations (JWT across two service instances, CentralFailoverTests.cs:27-47); SiteFailoverTests/DualNodeRecoveryTests similarly exercise component state recovery, not cluster membership; CoordinatedShutdownTests grep source text (CoordinatedShutdownTests.cs:11-43); HoconBuilderTests assert config strings. The repo's own notes describe the MultiNode test runner (AkkaDotNet/18-MultiNodeTestRunner.md) but it is unused. This is the direct cause of the Critical finding surviving: nothing ever killed the oldest node and waited for the singleton to move. Even a plain xunit test spinning two ActorSystems in-process and calling cluster.Down()/terminating one would have caught it.
  2. Windows Service lifecycle management is thin. Component-ClusterInfrastructure.md:19 claims "Manage Windows service lifecycle (start, stop, restart)"; the implementation is UseWindowsService() (Program.cs:74,401) plus install scripts. Service recovery actions (restart-on-failure), the watchdog implied by down-if-alone ("an external mechanism must restart both nodes" — AkkaDotNet/03-Cluster.md:64) do not exist in the repo; after a down-if-alone self-down the process presumably exits (or worse, idles) with nothing to restart it. The down-if-alone recovery story needs an explicit design (does the process exit? does the service restart it? does it rejoin cleanly?) and a drill.
  3. TLS is absent at every layer of this domain: Traefik terminates plain HTTP with an unauthenticated dashboard (docker/traefik/traefik.yml:5-7 — documented as dev-only in Component-TraefikProxy.md:131-138), Akka remoting is unencrypted TCP, site gRPC is http://. Fine for the lab; there is no production TLS profile or documented path yet.
  4. Dockerfile has no HEALTHCHECK and compose has no health-based dependency ordering; liveness /healthz runs zero checks (Program.cs:353-355) so a wedged actor system still reports live. Acceptable minimalism, but an orchestrator migration will need both.
  5. NodeName is missing from the wonder-app-vd03 deploy overlays (deploy/wonder-app-vd03/appsettings.Central.json has no Node.NodeName), so audit rows from that install get a NULL SourceNode (the base file documents the normalisation, src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Central.json:3) — undermines the per-node audit/KPI features (IX_AuditLog_Node_Occurred, per-node stuck KPIs) on the one real deployment.
  6. Known-but-open follow-ups acknowledged in code/docs: SecuredWrite audit rows leave SourceNode NULL (CLAUDE.md notes it as logged); no central persistence of OPC UA cert trust; docker-env2 mirrors the main topology (verified: same Traefik config shape, same seed/port conventions) so it inherits every finding above.
  7. Health monitoring has no history by design (Component-HealthMonitoring.md:93-97) — now partially superseded by KPI History (#26), which the docs handle; but online/offline transitions are still unrecorded anywhere (doc line 97 defers it), so post-incident "when did the site drop" questions remain unanswerable beyond log scraping.

Prioritized Recommendations

  1. [Critical, one line + tests] Enable the split-brain resolver. Add downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster" to BuildHocon (AkkaHostedService.cs:250) — or move the bootstrap to Akka.Cluster.Hosting per REQ-HOST-6, which sets it via typed options. Then add a two-node in-process failover test (kill oldest → assert singleton re-hosted, member removed within stable-after + margin) and a documented manual drill against the docker cluster (docker kill scadabridge-central-a, watch Traefik + singleton logs).
  2. [High] Unify the "active node" definition on singleton-host (oldest) semantics for ActiveNodeGate, AkkaClusterNodeProvider.SelfIsPrimary, the event-log purge gate, and the /health/active check; today's leader-based checks purge the wrong node's event log and label the wrong node Primary once leader ≠ oldest.
  3. [High] Remove DisableLogin: true from deploy/wonder-app-vd03 or add a hard environment guard around the flag.
  4. [Medium] Fix the CoordinatedShutdown drain budget (cluster-leave phase timeout ≥ graceful-stop timeout) and add drain tasks for the notification-outbox and audit-log-ingest singletons — or delete all five drains if the idempotent-DB argument makes them unnecessary.
  5. [Medium] Fix appsettings.Site.json seed localhost:8084 and extend StartupValidator's seed-port loop to reject MetricsPort collisions.
  6. [Medium] Make health-report delivery observable: switch AkkaHealthReportTransport to an acked Ask (reusing the existing site→central forward pattern) so the counter-restore logic in HealthReportSender actually fires, and surface LastReportReceivedAt staleness on the dashboard so heartbeat-alive/metrics-dead sites are visible.
  7. [Medium] Add stop_grace_period: 30s to the app services in both compose files.
  8. [Low] Housekeeping: evict deleted sites from CentralHealthAggregator; rate-limit dead-letter warnings; extract the singleton-registration helper; sync Component-TraefikProxy.md with the shared health-check library; add NodeName to the wonder-app-vd03 overlays; reconsider the mandatory-two-seeds rule for single-node installs.
Reference in New Issue View Git Blame Copy Permalink