docs(archreview): add architecture-review fix plans (P0 initiative baseline)
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,391 @@
|
||||
# Architecture Review 01 — Cluster Infrastructure, Host, Failover, Health Monitoring, Deployment Topology
|
||||
|
||||
Reviewer domain: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure`, `src/ZB.MOM.WW.ScadaBridge.Host`,
|
||||
`src/ZB.MOM.WW.ScadaBridge.HealthMonitoring`, `docker/`, `docker-env2/`, `deploy/wonder-app-vd03/`,
|
||||
Traefik config, and the corresponding design docs (`Component-ClusterInfrastructure.md`,
|
||||
`Component-Host.md`, `Component-HealthMonitoring.md`, `Component-TraefikProxy.md`) and test projects.
|
||||
|
||||
## Scope
|
||||
|
||||
Read in full: `AkkaHostedService.cs` (1,203 lines — HOCON builder, singleton wiring, coordinated
|
||||
shutdown), `Program.cs` (both role branches), `StartupValidator.cs`, `StartupRetry.cs`,
|
||||
`SiteServiceRegistration.cs`, `NodeOptions.cs`, `ClusterOptions.cs` + validator +
|
||||
`ServiceCollectionExtensions.cs`, all Host `Health/` checks (`RequiredSingletonsHealthCheck`,
|
||||
`ActiveNodeGate`, `AkkaClusterNodeProvider`), `DeadLetterMonitorActor.cs`,
|
||||
`AkkaHealthReportTransport.cs`, the entire HealthMonitoring project (`HealthReportSender`,
|
||||
`CentralHealthAggregator`, `CentralHealthReportLoop`, `SiteHealthCollector`,
|
||||
`SiteEventLogFailureCountReporter`, `SiteHealthKpiSampleSource`, options + validator), the docker
|
||||
compose topologies + per-node appsettings + Traefik static/dynamic config + Dockerfile, the
|
||||
on-host deploy artifacts (`deploy/wonder-app-vd03/appsettings.*.json`), the four design docs, the
|
||||
repo's own Akka.NET reference notes (`AkkaDotNet/03-Cluster.md`), and the relevant test projects
|
||||
(`Host.Tests`, `ClusterInfrastructure.Tests`, `HealthMonitoring.Tests`, `IntegrationTests`
|
||||
failover/readiness suites). Health-report ingestion on the central side was cross-checked in
|
||||
`Communication/Actors/CentralCommunicationActor.cs:349-392`.
|
||||
|
||||
## Maturity Verdict
|
||||
|
||||
This slice of the codebase is unusually well-commented, defensively coded at the micro level
|
||||
(options validators with `ValidateOnStart`, CAS loops in the aggregator, sequence-number seeding
|
||||
for failover, HOCON escaping, bounded startup retry), and clearly the product of many careful
|
||||
review passes — but it has **one macro-level defect that undoes the entire failover design: the
|
||||
split-brain resolver is configured but never enabled** (no `akka.cluster.downing-provider-class`),
|
||||
so unreachable nodes are never automatically downed and cluster singletons never migrate on a hard
|
||||
crash. Every document, comment, and validator in the domain reasons meticulously about keep-oldest
|
||||
semantics that are inert at runtime, and the test suite asserts the HOCON *strings* rather than the
|
||||
*behavior*, which is exactly how the gap survived. Secondary structural issues cluster around the
|
||||
ambiguous definition of "active node" (cluster *leader* vs. singleton-hosting *oldest* member,
|
||||
which diverge after restarts) and dev-vs-prod configuration drift. Verdict: **high polish, one
|
||||
critical foundation crack; not production-ready for unattended failover until the downing provider
|
||||
is fixed and verified with a real two-node test.**
|
||||
|
||||
---
|
||||
|
||||
## 1. Stability
|
||||
|
||||
### [Critical] Split-brain resolver is never enabled — automatic failover does not work for crashes/partitions
|
||||
|
||||
`AkkaHostedService.BuildHocon` (src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs:225-270)
|
||||
emits:
|
||||
|
||||
```hocon
|
||||
cluster {
|
||||
...
|
||||
split-brain-resolver {
|
||||
active-strategy = "keep-oldest"
|
||||
stable-after = 15000ms
|
||||
keep-oldest { down-if-alone = on }
|
||||
}
|
||||
failure-detector { ... }
|
||||
run-coordinated-shutdown-when-down = on
|
||||
}
|
||||
```
|
||||
|
||||
but never sets `akka.cluster.downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider,
|
||||
Akka.Cluster"`. In Akka.NET (1.5.62 per `Directory.Packages.props:8-17`), the default downing
|
||||
provider is **NoDowning**; the whole `split-brain-resolver` section is only read by
|
||||
`SplitBrainResolverProvider`, which must be opted into via `downing-provider-class`. A repo-wide
|
||||
grep confirms the string `SplitBrainResolverProvider` appears **nowhere in `src/`** — only in the
|
||||
project's own reference notes, which show the required line
|
||||
(AkkaDotNet/03-Cluster.md:56, also 15-Coordination.md:69 and 18-MultiNodeTestRunner.md:100).
|
||||
|
||||
Failure scenario: the active site node loses power. The standby's failure detector marks it
|
||||
unreachable (~10s), but **nothing ever downs it**. The unreachable member stays in the membership
|
||||
forever, the `ClusterSingletonManager` cannot hand over (singleton migration requires the oldest
|
||||
member's *removal*, not mere unreachability), and the site's DeploymentManager singleton — and with
|
||||
it all data collection, script execution, and S&F delivery — never restarts on the survivor. The
|
||||
same applies to all six central singletons (notification-outbox, audit-log-ingest, site-call-audit,
|
||||
audit-log-purge, site-audit-reconciliation, kpi-history-recorder). The documented "~25 seconds
|
||||
total failover" (docs/requirements/Component-ClusterInfrastructure.md:110-119) is only achievable
|
||||
today for a **graceful** stop (CoordinatedShutdown leave); crash and partition scenarios hang
|
||||
indefinitely until an operator downs the node or restarts the survivor.
|
||||
|
||||
Corroborating evidence that this was never exercised:
|
||||
|
||||
- `HoconBuilderTests.cs:45,112,155` assert only that the SBR *strings* appear in the config.
|
||||
- `CoordinatedShutdownTests.cs:11-43` are source-text greps (`Assert.Contains("run-by-clr-shutdown-hook = on", content)`).
|
||||
- The "failover" integration tests are single-process simulations — e.g.
|
||||
`CentralFailoverTests.cs:27-47` validates a JWT across two `JwtTokenService` instances; no test
|
||||
forms a two-node cluster and kills the oldest node.
|
||||
- `run-coordinated-shutdown-when-down = on` (AkkaHostedService.cs:265) is dead config: a node is
|
||||
never transitioned to Down by anything.
|
||||
- `ClusterOptionsValidator` (src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptionsValidator.cs:35-63)
|
||||
rigorously rejects any strategy other than `keep-oldest` and demands `DownIfAlone == true` —
|
||||
validating knobs on a resolver that is not installed.
|
||||
|
||||
Fix: add the `downing-provider-class` line to `BuildHocon` (one line), or better, migrate the
|
||||
bootstrap to `Akka.Cluster.Hosting`'s typed options (already referenced,
|
||||
ZB.MOM.WW.ScadaBridge.Host.csproj:15) whose `SplitBrainResolverOption` wires the provider
|
||||
automatically — and add a real two-node in-process cluster test that kills the oldest member and
|
||||
asserts the singleton re-hosts within the expected window.
|
||||
|
||||
### [High] "Active node" means two different things — cluster leader vs. oldest member — and they diverge
|
||||
|
||||
Two distinct node-selection concepts are used interchangeably:
|
||||
|
||||
- **Leader-based** (address ordering, *not* age): `ActiveNodeGate.IsActiveNode`
|
||||
(src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeGate.cs:36-52),
|
||||
`AkkaClusterNodeProvider.SelfIsPrimary` and the "Primary"/"Standby" labels
|
||||
(src/ZB.MOM.WW.ScadaBridge.Host/Health/AkkaClusterNodeProvider.cs:29-40,61), the `active-node`
|
||||
health check Traefik routes on (Program.cs:231-234, docker/traefik/dynamic.yml:12-18), the
|
||||
`CentralHealthReportLoop` leader gate (CentralHealthReportLoop.cs:93-97), and the site
|
||||
event-log purge gate (`SiteEventLogActiveNodeCheck` → `SelfIsPrimary`,
|
||||
SiteServiceRegistration.cs:116-120).
|
||||
- **Oldest-based** (singleton placement): every `ClusterSingletonManager` (AkkaHostedService.cs:410+,
|
||||
906-925), and the site health "IsActiveNode" flag, which is set by the DeploymentManager
|
||||
*singleton* on start/stop (src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs:243,258).
|
||||
|
||||
In Akka the leader is the lowest-address `Up` member; the singleton host is the *oldest* member.
|
||||
They coincide at first boot but diverge permanently after the original first node is restarted and
|
||||
rejoins (it regains leadership by address order while the peer remains oldest). Concrete
|
||||
consequences once diverged:
|
||||
|
||||
1. **Site event-log retention is unenforced on the node that writes events.** Events are recorded
|
||||
by the singleton hierarchy on the oldest node; the purge (`EventLogPurgeService.cs:21,36`)
|
||||
early-exits unless `SelfIsPrimary` (leader). The active/oldest node's SQLite grows past the
|
||||
30-day/1GB cap while the leader purges an idle database.
|
||||
2. **Health dashboard labels the wrong node "Primary"** (AkkaClusterNodeProvider.cs:61): the node
|
||||
named Primary hosts no singletons; the node doing all the work shows Standby.
|
||||
3. **Central self-report is emitted by the leader** (CentralHealthReportLoop.cs:93) while the
|
||||
NotificationOutbox/audit singletons run on the other node — the `$central` health card describes
|
||||
the wrong node.
|
||||
4. **Traefik routes UI/API to the leader** while every singleton `Ask` then hops to the peer —
|
||||
functionally correct via the proxies, but it contradicts the design docs' "one active node runs
|
||||
all central components" model (Component-ClusterInfrastructure.md:44-47) and doubles
|
||||
intra-cluster latency on every outbox/audit page load.
|
||||
|
||||
Recommendation: pick one definition. Simplest coherent choice: derive "active" from singleton-host
|
||||
status (e.g. compare self against the oldest `Up` member, or gate on reachability of a local
|
||||
singleton), and use it consistently for `ActiveNodeGate`, `SelfIsPrimary`, the purge gate, and the
|
||||
`/health/active` check — leadership is an Akka-internal concept that only accidentally matches the
|
||||
product's active/standby model.
|
||||
|
||||
### [High] During a central network partition, both nodes report `/health/active = 200` and Traefik serves both
|
||||
|
||||
`ActiveNodeGate`/the active-node check compute `cluster.State.Leader` from each node's own gossip
|
||||
view. When central-a and central-b cannot see each other but Traefik can reach both (a plausible
|
||||
partial partition), each side sees the peer as unreachable and computes **itself** as leader — both
|
||||
return 200 on `/health/active`, and Traefik's load balancer (docker/traefik/dynamic.yml:10-18)
|
||||
round-robins requests between two nodes that each believe they are active. Blazor circuits and the
|
||||
Inbound API's active-node 503 gate (Program.cs:256-260) are then split across both. Because the SBR
|
||||
is not enabled (Critical finding above), nothing ever resolves the partition — this state persists
|
||||
until manual intervention. Singletons do *not* duplicate (that requires member removal), but every
|
||||
leader-gated behavior does. Enabling keep-oldest + `down-if-alone` fixes the persistence; consider
|
||||
also making `/health/active` require `Ready` (DB reachable) in addition to leadership so a
|
||||
DB-dead leader is pulled from rotation.
|
||||
|
||||
### [Medium] Cluster-leave drain tasks are budgeted 10s inside a 5s CoordinatedShutdown phase
|
||||
|
||||
Five singletons add a `PhaseClusterLeave` task that runs `GracefulStop(TimeSpan.FromSeconds(10))`
|
||||
(AkkaHostedService.cs:551-567 site-call-audit, 624-641 audit-log-purge, 676-693
|
||||
site-audit-reconciliation, 733-750 kpi-history-recorder, 787-804 pending-deployment-purge). Akka's
|
||||
`cluster-leave` phase uses the default phase timeout of **5 seconds**; the phase abandons the task
|
||||
at 5s (default `recover = on`) and proceeds, so the 10s graceful-stop can never complete its full
|
||||
window and the `catch` logging in those tasks never observes the overrun (the task is orphaned, not
|
||||
faulted). The intended "let the in-flight EF upsert drain before handover" guarantee is silently
|
||||
halved. Fix: set `akka.coordinated-shutdown.phases.cluster-leave.timeout` in `BuildHocon` to
|
||||
exceed the graceful-stop budget, or shrink the `GracefulStop` timeout below 5s.
|
||||
|
||||
### [Medium] The two busiest singletons have no drain task at all
|
||||
|
||||
The `NotificationOutboxActor` and `AuditLogIngestActor` singletons are terminated by bare
|
||||
`PoisonPill` with no cluster-leave drain (AkkaHostedService.cs:410-419, 451-458); the comment at
|
||||
AkkaHostedService.cs:544-549 explicitly notes the pattern "is suitable for the NotificationOutbox
|
||||
singleton; not added here to keep this change minimal". These are precisely the actors with
|
||||
in-flight EF transactions on every message (dispatch sweeps, dual-write ingest). The idempotent
|
||||
DB design (insert-if-not-exists on `NotificationId`/`EventId`) makes this safe-but-lossy rather
|
||||
than corrupting, but the inconsistency (drain for the low-traffic maintenance singletons, none for
|
||||
the hot ones) should be resolved.
|
||||
|
||||
### [Medium] Health-report loss recovery is dead code on the production transport
|
||||
|
||||
`HealthReportSender` carefully restores interval counters if `_transport.Send` throws
|
||||
(HealthReportSender.cs:158-175), and `SiteHealthCollector.AddIntervalCounters`
|
||||
(SiteHealthCollector.cs:153-171) exists to support it. But the only production transport is
|
||||
`AkkaHealthReportTransport.Send` (src/ZB.MOM.WW.ScadaBridge.Host/AkkaHealthReportTransport.cs:26-33)
|
||||
— a fire-and-forget `Tell` to an `ActorSelection`, which **never throws** and silently returns when
|
||||
the actor system is null. Any downstream loss (central unreachable, ClusterClient reconnecting,
|
||||
dead-lettered selection) drops the report *and* the interval error counters it carried — the
|
||||
scenario the restore logic was built for is exactly the one it can't see. Either make the transport
|
||||
Ask-with-ack (the site→central notification forward already uses this pattern over the same actor)
|
||||
or accept the loss and delete the restore machinery; the current shape gives false confidence.
|
||||
|
||||
### [Medium] Offline detection deviates from the spec and can mask a dead metrics pipeline
|
||||
|
||||
`Component-HealthMonitoring.md:43-46` defines offline as "no *report* within 60s".
|
||||
`CentralHealthAggregator.CheckForOfflineSites` (CentralHealthAggregator.cs:224-263) instead keys
|
||||
off `LastHeartbeatAt`, which is also advanced by the ~5s transport heartbeats
|
||||
(`CentralCommunicationActor.HandleHeartbeat` → `MarkHeartbeat`, CentralCommunicationActor.cs:349-353).
|
||||
A site whose `HealthReportSender` loop has died (or whose DeploymentManager singleton is stuck, so
|
||||
`IsActiveNode` is false on both nodes and reports are skipped — HealthReportSender.cs:85) keeps
|
||||
heartbeating and shows **online with silently frozen metrics forever**; nothing surfaces
|
||||
`LastReportReceivedAt` staleness. The heartbeat-based liveness choice is defensible (the code
|
||||
comment argues it), but then the dashboard needs a distinct "metrics stale" signal, and the design
|
||||
doc should be updated — currently spec and code disagree.
|
||||
|
||||
### [Medium] Second seed node in the checked-in site dev config targets the metrics port; validator has a gap
|
||||
|
||||
`src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json:14-17` seeds
|
||||
`akka.tcp://scadabridge@localhost:8082` and `akka.tcp://scadabridge@localhost:8084`, while the same
|
||||
file sets `MetricsPort: 8084` (line 10). The second seed points at the Kestrel HTTP/1.1 metrics
|
||||
listener — a node attempting to join via that seed performs a doomed Akka.Remote association
|
||||
against Kestrel (handshake noise, retry loops). `StartupValidator` explicitly guards seed-vs-GrpcPort
|
||||
(StartupValidator.cs:121-127) but not seed-vs-MetricsPort, so this config validates cleanly. Fix
|
||||
the dev config (8084 → the intended second node's remoting port) and extend the seed-port loop to
|
||||
also reject `MetricsPort`.
|
||||
|
||||
### [Medium] Docker stop grace period is shorter than a full CoordinatedShutdown
|
||||
|
||||
`docker/docker-compose.yml` sets no `stop_grace_period` (default 10s SIGTERM→SIGKILL) and the
|
||||
Dockerfile runs `dotnet` as PID 1 (docker/Dockerfile, final stage). A graceful shutdown runs
|
||||
cluster-leave (5s budget) + cluster-exiting (10s) + actor-system-terminate plus Serilog flush —
|
||||
easily exceeding 10s on a loaded node, so `bash docker/deploy.sh` recreates can SIGKILL mid-drain,
|
||||
turning every "graceful" redeploy into the crash path (which, per the Critical finding, currently
|
||||
has no automatic recovery on the surviving node). Add `stop_grace_period: 30s` to the app services.
|
||||
|
||||
### [Low] Aggregator never forgets sites
|
||||
|
||||
`CentralHealthAggregator._siteStates` (CentralHealthAggregator.cs:16) only grows: a site deleted
|
||||
from configuration remains a permanently-offline dashboard tile (and continues to be enumerated by
|
||||
`SiteHealthKpiSampleSource.CollectAsync`, SiteHealthKpiSampleSource.cs:74-85) until process
|
||||
restart. Needs an eviction path wired to site deletion.
|
||||
|
||||
### [Low] Dead-letter monitor: unbounded warning volume and restart-reset count
|
||||
|
||||
`DeadLetterMonitorActor` logs one Warning per dead letter with no sampling/rate-limiting and keeps
|
||||
its count in actor state that resets on restart
|
||||
(src/ZB.MOM.WW.ScadaBridge.Host/Actors/DeadLetterMonitorActor.cs:14,26-35). A dead-letter storm
|
||||
(mass undeploy, failover, shutdown) floods the log; the health metric path
|
||||
(`IncrementDeadLetter`) is fine because the collector counter is interval-based.
|
||||
|
||||
### [Low] Mandatory two-seed rule forces phantom seeds in single-node installs
|
||||
|
||||
`ClusterOptionsValidator` requires ≥2 seed nodes
|
||||
(ClusterOptionsValidator.cs:29-33); the shipped single-node production artifact satisfies it with
|
||||
a documented placeholder (`deploy/wonder-app-vd03/appsettings.Central.json:9-19`, second seed
|
||||
`localhost:8091` "harmless while absent"). The node dials the phantom seed forever (log noise,
|
||||
periodic connect attempts), and the rule's intent — no startup-ordering dependency — is defeated
|
||||
by fiction. Consider allowing 1 seed with a warning, or a `SingleNode` acknowledgement flag.
|
||||
|
||||
## 2. Performance
|
||||
|
||||
This domain is largely cold-path and correctly engineered for it; nothing here touches the
|
||||
data-plane hot path. Observations:
|
||||
|
||||
- **[Low] Readiness probes fan out five cluster Asks per poll.**
|
||||
`RequiredSingletonsHealthCheck` sends 5 concurrent `Identify` Asks through singleton proxies
|
||||
every `/health/ready` poll (RequiredSingletonsHealthCheck.cs:66-78,111-117). When the singletons
|
||||
live on the peer node each probe is a remoting round trip. Bounded (2s, concurrent) and cheap at
|
||||
typical poll rates, but a tight orchestration probe interval multiplies cluster chatter.
|
||||
- **[Low] Startup compiles every inbound API method sequentially before serving**
|
||||
(Program.cs:379-388). Startup latency scales linearly with method count; a slow Roslyn compile of
|
||||
many methods delays readiness on the *standby* too (it runs the same branch). Acceptable now;
|
||||
parallelize or defer-compile if method counts grow.
|
||||
- **[Low] Per-tick allocations in health collection** (dictionary snapshots per 30s report,
|
||||
`GetAllSiteStates` copy per dashboard poll — CentralHealthAggregator.cs:173-176) are trivial at
|
||||
this cadence. No action.
|
||||
- **Positive:** the site audit telemetry actor is pinned to a dedicated dispatcher
|
||||
(`audit-telemetry-dispatcher`, AkkaHostedService.cs:226-232,1150-1158) so batch SQLite/gRPC work
|
||||
never contends with hot-path actors; `StoreAndForwardService.StartAsync` is awaited rather than
|
||||
sync-blocked (AkkaHostedService.cs:976-981); the ActorSystem DI bridge is a root singleton
|
||||
precisely to avoid per-scope disposal (Program.cs:240-250) — all correct.
|
||||
|
||||
## 3. Conventions
|
||||
|
||||
- **[Medium] REQ-HOST-6 says "Akka.NET bootstrap via Akka.Hosting"; the code hand-rolls HOCON.**
|
||||
`Component-Host.md` (REQ-HOST-6, line 106-114) mandates Akka.Hosting; the actual bootstrap is
|
||||
string-interpolated HOCON + `ActorSystem.Create` (AkkaHostedService.cs:171-176,214-271) with
|
||||
`Akka.Hosting`/`Akka.Cluster.Hosting`/`Akka.Remote.Hosting` referenced but unused for cluster
|
||||
bring-up (ZB.MOM.WW.ScadaBridge.Host.csproj:15-18). This is not just doc drift: Akka.Cluster.Hosting's
|
||||
typed `SplitBrainResolverOption` sets `downing-provider-class` for you — using the mandated API
|
||||
would have prevented the Critical finding. Either implement REQ-HOST-6 or amend it and drop the
|
||||
unused packages.
|
||||
- **[High] Production deploy artifact ships with authentication disabled.**
|
||||
`deploy/wonder-app-vd03/appsettings.Central.json:39-42` sets `"DisableLogin": true` — every
|
||||
request auto-authenticates as `multi-role` with all roles, and the management API on that host is
|
||||
unauthenticated. The base config documents this as "DEV/TEST ONLY — never enable in production"
|
||||
with "no environment guard; a startup warning is the only protection"
|
||||
(src/ZB.MOM.WW.ScadaBridge.Host/appsettings.json:6). A `deploy/` artifact with an install
|
||||
runbook is a production surface; at minimum gate `DisableLogin` on
|
||||
`Environment.IsDevelopment()` or require an explicit `I-understand` companion flag.
|
||||
- **[Medium] LDAP transport is plaintext everywhere, contradicting the security spec.** The design
|
||||
requires "LDAPS/StartTLS required" (CLAUDE.md Security & Auth; Component-Security), but every
|
||||
shipped config — docker (docker/central-node-a/appsettings.Central.json:26-33) and the on-host
|
||||
deploy (deploy/wonder-app-vd03/appsettings.Central.json:26-34) — sets `"Transport": "None",
|
||||
"AllowInsecure": true` and sends the service-account bind password in clear. Dev GLAuth is a fair
|
||||
excuse for docker; the deploy artifact should not normalize it.
|
||||
- **[Medium] Secrets in checked-in dev configs.** SQL credentials, LDAP service-account password
|
||||
and the JWT signing key are committed in `docker/central-node-a/appsettings.Central.json:21-38`
|
||||
(labelled dev-only; the compose pepper is likewise flagged, docker-compose.yml:9-14). The
|
||||
wonder-app-vd03 artifact handles this correctly via `${...}` placeholders + env injection
|
||||
(appsettings.Central.json:20-24) — good pattern, worth backporting a scrubbed variant to docker
|
||||
if these ever leave a lab network. Also note `ConfigSecretsTests.cs` exists in Host.Tests, so the
|
||||
team is tracking this.
|
||||
- **[Low] Stale doc references.** `Component-TraefikProxy.md:121` locates `ActiveNodeHealthCheck`
|
||||
at `src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeHealthCheck.cs`; the check now comes from the
|
||||
shared `ZB.MOM.WW.Health.Akka` package (Program.cs:214-234) and no such file exists.
|
||||
`Component-TraefikProxy.md:115` also still describes `/health/ready` as "database + Akka
|
||||
cluster" — it now includes `required-singletons` (Program.cs:227-230; Component-Host.md REQ-HOST-4a
|
||||
*is* current). Keep the Traefik doc in sync.
|
||||
- **[Low] Five copy-pasted ~60-line singleton registration blocks.**
|
||||
AkkaHostedService.RegisterCentralActors repeats the manager/drain-task/proxy triple five times
|
||||
(AkkaHostedService.cs:399-811) with only names and Props varying. Extract a
|
||||
`StartCentralSingleton(name, props, drainTimeout)` helper — the copy-paste already caused drift
|
||||
(outbox/ingest lack drain tasks).
|
||||
- **Positive conventions compliance:** Options pattern is followed rigorously (component-owned
|
||||
options classes, `ValidateOnStart`, dedicated validators with key-naming messages —
|
||||
ClusterOptionsValidator, HealthMonitoringOptionsValidator); HOCON interpolation is
|
||||
injection-safe (`QuoteHocon`/`DurationHocon`, AkkaHostedService.cs:280-295); Serilog enrichment
|
||||
matches REQ-HOST-8; startup validation matches REQ-HOST-4 including cross-field port-collision
|
||||
rules; single-binary role dispatch matches REQ-HOST-1/2; the `/health/ready` vs `/health/active`
|
||||
tier separation (leadership deliberately excluded from readiness) is well-reasoned and
|
||||
documented in place (Program.cs:346-356).
|
||||
|
||||
## 4. Underdeveloped Areas
|
||||
|
||||
1. **No real multi-node failover test anywhere.** `CentralFailoverTests` are single-process
|
||||
simulations (JWT across two service instances, CentralFailoverTests.cs:27-47);
|
||||
`SiteFailoverTests`/`DualNodeRecoveryTests` similarly exercise component state recovery, not
|
||||
cluster membership; `CoordinatedShutdownTests` grep source text (CoordinatedShutdownTests.cs:11-43);
|
||||
`HoconBuilderTests` assert config strings. The repo's own notes describe the MultiNode test
|
||||
runner (AkkaDotNet/18-MultiNodeTestRunner.md) but it is unused. This is the direct cause of the
|
||||
Critical finding surviving: nothing ever killed the oldest node and waited for the singleton to
|
||||
move. Even a plain xunit test spinning two `ActorSystem`s in-process and calling
|
||||
`cluster.Down()`/terminating one would have caught it.
|
||||
2. **Windows Service lifecycle management is thin.** `Component-ClusterInfrastructure.md:19`
|
||||
claims "Manage Windows service lifecycle (start, stop, restart)"; the implementation is
|
||||
`UseWindowsService()` (Program.cs:74,401) plus install scripts. Service recovery actions
|
||||
(restart-on-failure), the watchdog implied by `down-if-alone` ("an external mechanism must
|
||||
restart both nodes" — AkkaDotNet/03-Cluster.md:64) do not exist in the repo; after a
|
||||
down-if-alone self-down the process presumably exits (or worse, idles) with nothing to restart
|
||||
it. The down-if-alone recovery story needs an explicit design (does the process exit? does the
|
||||
service restart it? does it rejoin cleanly?) and a drill.
|
||||
3. **TLS is absent at every layer of this domain**: Traefik terminates plain HTTP with an
|
||||
unauthenticated dashboard (docker/traefik/traefik.yml:5-7 — documented as dev-only in
|
||||
Component-TraefikProxy.md:131-138), Akka remoting is unencrypted TCP, site gRPC is `http://`.
|
||||
Fine for the lab; there is no production TLS profile or documented path yet.
|
||||
4. **Dockerfile has no `HEALTHCHECK`** and compose has no health-based dependency ordering;
|
||||
liveness `/healthz` runs zero checks (Program.cs:353-355) so a wedged actor system still
|
||||
reports live. Acceptable minimalism, but an orchestrator migration will need both.
|
||||
5. **`NodeName` is missing from the wonder-app-vd03 deploy overlays**
|
||||
(deploy/wonder-app-vd03/appsettings.Central.json has no `Node.NodeName`), so audit rows from
|
||||
that install get a NULL `SourceNode` (the base file documents the normalisation,
|
||||
src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Central.json:3) — undermines the per-node audit/KPI
|
||||
features (`IX_AuditLog_Node_Occurred`, per-node stuck KPIs) on the one real deployment.
|
||||
6. **Known-but-open follow-ups acknowledged in code/docs:** SecuredWrite audit rows leave
|
||||
`SourceNode` NULL (CLAUDE.md notes it as logged); no central persistence of OPC UA cert trust;
|
||||
`docker-env2` mirrors the main topology (verified: same Traefik config shape, same
|
||||
seed/port conventions) so it inherits every finding above.
|
||||
7. **Health monitoring has no history by design** (Component-HealthMonitoring.md:93-97) — now
|
||||
partially superseded by KPI History (#26), which the docs handle; but online/offline
|
||||
*transitions* are still unrecorded anywhere (doc line 97 defers it), so post-incident "when did
|
||||
the site drop" questions remain unanswerable beyond log scraping.
|
||||
|
||||
## Prioritized Recommendations
|
||||
|
||||
1. **[Critical, one line + tests] Enable the split-brain resolver.** Add
|
||||
`downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster"` to
|
||||
`BuildHocon` (AkkaHostedService.cs:250) — or move the bootstrap to Akka.Cluster.Hosting per
|
||||
REQ-HOST-6, which sets it via typed options. Then add a two-node in-process failover test
|
||||
(kill oldest → assert singleton re-hosted, member removed within stable-after + margin) and a
|
||||
documented manual drill against the docker cluster (`docker kill scadabridge-central-a`, watch
|
||||
Traefik + singleton logs).
|
||||
2. **[High] Unify the "active node" definition** on singleton-host (oldest) semantics for
|
||||
`ActiveNodeGate`, `AkkaClusterNodeProvider.SelfIsPrimary`, the event-log purge gate, and the
|
||||
`/health/active` check; today's leader-based checks purge the wrong node's event log and label
|
||||
the wrong node Primary once leader ≠ oldest.
|
||||
3. **[High] Remove `DisableLogin: true` from `deploy/wonder-app-vd03`** or add a hard environment
|
||||
guard around the flag.
|
||||
4. **[Medium] Fix the CoordinatedShutdown drain budget** (`cluster-leave` phase timeout ≥
|
||||
graceful-stop timeout) and add drain tasks for the notification-outbox and audit-log-ingest
|
||||
singletons — or delete all five drains if the idempotent-DB argument makes them unnecessary.
|
||||
5. **[Medium] Fix `appsettings.Site.json` seed `localhost:8084`** and extend `StartupValidator`'s
|
||||
seed-port loop to reject `MetricsPort` collisions.
|
||||
6. **[Medium] Make health-report delivery observable**: switch `AkkaHealthReportTransport` to an
|
||||
acked Ask (reusing the existing site→central forward pattern) so the counter-restore logic in
|
||||
`HealthReportSender` actually fires, and surface `LastReportReceivedAt` staleness on the
|
||||
dashboard so heartbeat-alive/metrics-dead sites are visible.
|
||||
7. **[Medium] Add `stop_grace_period: 30s`** to the app services in both compose files.
|
||||
8. **[Low] Housekeeping:** evict deleted sites from `CentralHealthAggregator`; rate-limit
|
||||
dead-letter warnings; extract the singleton-registration helper; sync
|
||||
`Component-TraefikProxy.md` with the shared health-check library; add `NodeName` to the
|
||||
wonder-app-vd03 overlays; reconsider the mandatory-two-seeds rule for single-node installs.
|
||||
Reference in New Issue
Block a user