docs(archreview): round-2 re-review (2026-07-12) + 8 fix plans (86 tasks)
Re-ran all 8 domain reviews at HEAD8c888f13against theb910f5ebbaseline: every round-1 finding source-verified (168 fixed, 0 regressions, 0 false claims); 56 new findings (1 Critical / 4 High / 15 Medium / 36 Low), concentrated in post-baseline code (anti-entropy resync, KPI rollup backfill, live alarm stream) and seams the fixes exposed. Headliners: S&F resync predicate inversion can wipe the delivering node's buffer (02-N1 Critical); resync snapshot exceeds the Akka remoting frame size (02-N2); failover drill kills the one node keep-oldest can't survive (01-N1); unbounded rollup backfill per failover (04-R1); live production API key in untracked test.txt (08-NF1). Adds PLAN-R2-01..08 + .tasks.json manifests and the Round-2 board, P0 list, cross-plan mutexes, and wave order in 00-MASTER-TRACKER.
This commit is contained in:
Joseph Doherty
@@ -1,391 +1,118 @@
|
||||
# Architecture Review 01 — Cluster Infrastructure, Host, Failover, Health Monitoring, Deployment Topology
|
||||
# Architecture Review 01 — Cluster Infrastructure, Host, Failover, Health Monitoring, Deployment Topology (Round 2)
|
||||
|
||||
Reviewer domain: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure`, `src/ZB.MOM.WW.ScadaBridge.Host`,
|
||||
`src/ZB.MOM.WW.ScadaBridge.HealthMonitoring`, `docker/`, `docker-env2/`, `deploy/wonder-app-vd03/`,
|
||||
Traefik config, and the corresponding design docs (`Component-ClusterInfrastructure.md`,
|
||||
`Component-Host.md`, `Component-HealthMonitoring.md`, `Component-TraefikProxy.md`) and test projects.
|
||||
**Date:** 2026-07-12 (round 2; round-1 report 2026-07-08, baseline commit `b910f5eb`; current HEAD `8c888f13`)
|
||||
|
||||
## Scope
|
||||
|
||||
Read in full: `AkkaHostedService.cs` (1,203 lines — HOCON builder, singleton wiring, coordinated
|
||||
shutdown), `Program.cs` (both role branches), `StartupValidator.cs`, `StartupRetry.cs`,
|
||||
`SiteServiceRegistration.cs`, `NodeOptions.cs`, `ClusterOptions.cs` + validator +
|
||||
`ServiceCollectionExtensions.cs`, all Host `Health/` checks (`RequiredSingletonsHealthCheck`,
|
||||
`ActiveNodeGate`, `AkkaClusterNodeProvider`), `DeadLetterMonitorActor.cs`,
|
||||
`AkkaHealthReportTransport.cs`, the entire HealthMonitoring project (`HealthReportSender`,
|
||||
`CentralHealthAggregator`, `CentralHealthReportLoop`, `SiteHealthCollector`,
|
||||
`SiteEventLogFailureCountReporter`, `SiteHealthKpiSampleSource`, options + validator), the docker
|
||||
compose topologies + per-node appsettings + Traefik static/dynamic config + Dockerfile, the
|
||||
on-host deploy artifacts (`deploy/wonder-app-vd03/appsettings.*.json`), the four design docs, the
|
||||
repo's own Akka.NET reference notes (`AkkaDotNet/03-Cluster.md`), and the relevant test projects
|
||||
(`Host.Tests`, `ClusterInfrastructure.Tests`, `HealthMonitoring.Tests`, `IntegrationTests`
|
||||
failover/readiness suites). Health-report ingestion on the central side was cross-checked in
|
||||
`Communication/Actors/CentralCommunicationActor.cs:349-392`.
|
||||
Same domain as round 1: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure`, `src/ZB.MOM.WW.ScadaBridge.Host`,
|
||||
`src/ZB.MOM.WW.ScadaBridge.HealthMonitoring`, `docker/`, `docker-env2/`, `deploy/wonder-app-vd03/`,
|
||||
Traefik config, the four design docs (`Component-ClusterInfrastructure.md`, `Component-Host.md`,
|
||||
`Component-HealthMonitoring.md`, `Component-TraefikProxy.md`), and the test projects.
|
||||
|
||||
## Maturity Verdict
|
||||
## Method
|
||||
|
||||
This slice of the codebase is unusually well-commented, defensively coded at the micro level
|
||||
(options validators with `ValidateOnStart`, CAS loops in the aggregator, sequence-number seeding
|
||||
for failover, HOCON escaping, bounded startup retry), and clearly the product of many careful
|
||||
review passes — but it has **one macro-level defect that undoes the entire failover design: the
|
||||
split-brain resolver is configured but never enabled** (no `akka.cluster.downing-provider-class`),
|
||||
so unreachable nodes are never automatically downed and cluster singletons never migrate on a hard
|
||||
crash. Every document, comment, and validator in the domain reasons meticulously about keep-oldest
|
||||
semantics that are inert at runtime, and the test suite asserts the HOCON *strings* rather than the
|
||||
*behavior*, which is exactly how the gap survived. Secondary structural issues cluster around the
|
||||
ambiguous definition of "active node" (cluster *leader* vs. singleton-hosting *oldest* member,
|
||||
which diverge after restarts) and dev-vs-prod configuration drift. Verdict: **high polish, one
|
||||
critical foundation crack; not production-ready for unattended failover until the downing provider
|
||||
is fixed and verified with a real two-node test.**
|
||||
1. Re-read the round-1 report and `archreview/plans/PLAN-01-cluster-host-failover.md` (findings-coverage table) plus the master tracker's cross-plan notes.
|
||||
2. Verified **every** round-1 finding against the current source (not the plan's claims), with file:line evidence.
|
||||
3. Fresh sweep over `git diff b910f5eb..HEAD --stat` scoped to the domain paths (~2,550 insertions across 68 files: `ClusterActivityEvaluator`, `CentralSingletonRegistrar`, `OldestNodeActiveHealthCheck`, the SBR downing line, acked health transport, metrics-staleness state machine, `failover-drill.sh`, options validators, the two-node cluster test rig, and adjacent new code that landed in this domain from other plans — `InProcessScriptArtifactChangeBus`, `CentralHealthAuditBacklogProvider`).
|
||||
|
||||
**One-line verdict:** Round 1 found one Critical foundation crack (SBR never enabled) under high polish; round 2 finds the crack fixed and behaviorally proven, 18 of 28 findings fully fixed and verified, the active-node model coherently unified — but the *shipped failover drill and recovery narrative overpromise*: a hard crash of the active (oldest) central node still ends in total central outage (registered deferred decision), and the never-executed drill script codifies a ~25s recovery that topology (first-seed bootstrap + keep-oldest asymmetry) cannot deliver in its most likely case.
|
||||
|
||||
---
|
||||
|
||||
## 1. Stability
|
||||
## Round-1 Finding Disposition
|
||||
|
||||
### [Critical] Split-brain resolver is never enabled — automatic failover does not work for crashes/partitions
|
||||
Every disposition below was verified by reading the current source; plan claims were not trusted.
|
||||
|
||||
`AkkaHostedService.BuildHocon` (src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs:225-270)
|
||||
emits:
|
||||
| # | Round-1 finding | R1 severity | Disposition | Evidence (file:line) |
|
||||
|---|----------------|-------------|-------------|----------------------|
|
||||
| S1 | SBR downing provider never enabled — no automatic failover on crash/partition | Critical | **Fixed (verified)** — provider named in HOCON; behaviorally proven by a real two-node kill test. Residual: two-node keep-oldest cannot survive an *oldest*-node crash — a **registered deferred user decision**, honestly documented in the test's XML doc | `AkkaHostedService.cs:290` (`downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster"`), doc at `:235-238`; `SbrFailoverTests.cs:46-80` (crash younger → member removed + singleton survives), asymmetry documented at `:15-23`; fixture builds from production `BuildHocon` (`TwoNodeClusterFixture.cs:56-64`) |
|
||||
| S2 | "Active node" = leader vs oldest diverge (wrong purge node, wrong Primary label, wrong self-report) | High | **Fixed (verified)** — single oldest-member evaluator consumed by every product gate; divergence proven by a rejoin test | `ClusterActivityEvaluator.cs:20-47`; `ActiveNodeGate.cs:47-48`; `AkkaClusterNodeProvider.cs:36` (SelfIsPrimary), `:47-58` (Primary label from oldest); purge gate delegates to `SelfIsPrimary` (`SiteServiceRegistration.cs:118-122`); central self-report gate (`CentralHealthReportLoop.cs:93`); S&F delivery gate reuses it (`AkkaHostedService.cs:878-882`); `ActiveNodeSemanticsTests.cs:29-68` |
|
||||
| S3 | Partition: both centrals return 200 on `/health/active`; Traefik serves both | High | **Fixed (verified)** — active = oldest Up member (exactly one in both partition views, since an unreachable member stays in gossip) AND database reachable | `OldestNodeActiveHealthCheck.cs:27-33`; `Program.cs:231-234` (database check tagged `Ready` + `Active`), `:252-257` (active-node = `OldestNodeActiveHealthCheck`); `Component-TraefikProxy.md:29,116` |
|
||||
| S4 | 10s singleton drains inside the 5s `cluster-leave` phase | Medium | **Fixed (verified)** | `AkkaHostedService.cs:304-307` (`phases.cluster-leave.timeout = 15s`); parsed-config test in `HoconBuilderTests.cs` |
|
||||
| S5 | NotificationOutbox + AuditLogIngest singletons had no drain task | Medium | **Fixed (verified)** — all seven central singletons routed through the registrar, which always adds a `PhaseClusterLeave` `GracefulStop` drain | `CentralSingletonRegistrar.cs:46-62`; call sites `AkkaHostedService.cs:466` (outbox), `:499` (ingest), `:564`, `:609`, `:632`, `:663`, `:689`; rationale comment `:459-465` |
|
||||
| S6 | Health-report loss recovery was dead code (fire-and-forget transport never throws) | Medium | **Fixed (verified)** — transport is now an acked Ask round-trip; the counter-restore path is live | `AkkaHealthReportTransport.cs:30-40` (Ask + throw on nack/timeout); acks at `SiteCommunicationActor.cs:395-407` and `CentralCommunicationActor.cs:425`; restore fires on throw (`HealthReportSender.cs:170-187`) |
|
||||
| S7 | Offline detection keyed off heartbeats masks a dead metrics pipeline; spec/code disagreed | Medium | **Fixed (verified)** — two-signal model: heartbeat liveness + `IsMetricsStale` on report silence; options cross-validated; spec updated to match. Residual: never-reported sites can't be flagged (new Low finding N3) | `CentralHealthAggregator.cs:295-312`; `HealthMonitoringOptions.cs` (`MetricsStaleTimeout`, `CentralOfflineTimeout`); validator cross-field rules (`HealthMonitoringOptionsValidator.cs`); `Component-HealthMonitoring.md:46-49`; UI badge `Health.razor:209` |
|
||||
| S8 | `appsettings.Site.json` second seed targeted MetricsPort 8084; validator gap | Medium | **Fixed (verified)** — seed now 8085 with an explanatory `_seedNodes` note; validator rejects seeds on GrpcPort *and* MetricsPort | `appsettings.Site.json` (`SeedNodes` → `localhost:8085`); `StartupValidator.cs:117-134` |
|
||||
| S9 | Docker stop grace shorter than CoordinatedShutdown | Medium | **Fixed (verified)** — `stop_grace_period: 30s` on all 8 `docker/` app services and all 4 `docker-env2/` services, each with a rationale comment | `docker/docker-compose.yml:7,34,61,81,101,121,141,161`; `docker-env2/docker-compose.yml:7,34,61,81` |
|
||||
| S10 | Aggregator never forgets deleted sites | Low | **Fixed (verified)** — eviction piggybacks on the existing 60s site-address-cache refresh, no dedicated deletion event needed | `CentralHealthAggregator.cs:200-211` (`PruneUnknownSites`, protects the synthetic `$central` entry); caller `CentralCommunicationActor.cs:599-605` |
|
||||
| S11 | Dead-letter monitor: unbounded warning volume | Low | **Fixed (verified)** — 10 warnings/min window, suppressed-count summary on rollover, metric counting never throttled | `DeadLetterMonitorActor.cs:22-26,51-80,89` |
|
||||
| S12 | Mandatory two-seed rule forces phantom seeds in single-node installs | Low | **Partially fixed** — `AllowSingleNodeCluster` flag + validator with a phantom-seed-naming message shipped; the one real single-node artifact still ships the phantom seed and lacks the flag (overlay edit deferred out-of-band; see N2) | `ClusterOptions.cs:86`; `ClusterOptionsValidator.cs:29-35`; unfixed artifact: `deploy/wonder-app-vd03/appsettings.Central.json` (seeds still `localhost:8081` + phantom `localhost:8091`, no `AllowSingleNodeCluster`) |
|
||||
| P1 | Readiness probes fan out 5 cluster Asks per poll | Low | **Accepted (per plan)** — unchanged, bounded 2s concurrent Asks | `RequiredSingletonsHealthCheck.cs` (only doc-comment change since baseline) |
|
||||
| P2 | Startup compiles every inbound method sequentially | Low | **Fixed (verified)** (delivered via plan 06) — parallel compile with a thread-safety rationale | `Program.cs:412-415` (`Parallel.ForEach(methods, …)`) |
|
||||
| P3 | Per-tick allocations in health collection | Low | **Won't-fix (per report's own "no action")** — unchanged | `CentralHealthAggregator.cs:187-190` |
|
||||
| C1 | REQ-HOST-6 mandated Akka.Hosting; code hand-rolls HOCON | Medium | **Fixed (verified)** — resolved by amending the spec (hand-rolled HOCON is now the documented mechanism, with rationale) and dropping the three unused `Akka.*.Hosting` packages | `Component-Host.md:108-116` (bootstrap-mechanism note); `ZB.MOM.WW.ScadaBridge.Host.csproj:15-21` (packages removed, comment); `Component-Host.md:208` |
|
||||
| C2 | `deploy/wonder-app-vd03` shipped `DisableLogin: true` | High | **Fixed (verified)** (delivered via plan 07) — fail-fast startup guard refuses `DisableLogin` outside Development without an explicit second acknowledgement key; the artifact no longer sets it and its `_comment_Security` documents the removal + a bundled localhost GLAuth for real logins | `DisableLoginGuard.cs:17-29`; call in `Program.cs` (Central branch, before `AddSecurity`); `deploy/wonder-app-vd03/appsettings.Central.json:40` |
|
||||
| C3 | LDAP transport plaintext everywhere, contradicting "LDAPS/StartTLS required" | Medium | **Partially fixed / accepted posture** — docker remains dev-only plaintext (accepted); the deploy artifact now binds a *bundled loopback GLAuth* (`localhost:3893`) rather than plaintext across a network, with documented instructions to repoint to corporate AD — but `Transport: None, AllowInsecure: true` remains and a future AD repoint over the wire would still be plaintext unless changed | `deploy/wonder-app-vd03/appsettings.Central.json:29-30,40` |
|
||||
| C4 | Secrets in checked-in docker dev configs | Medium | **Accepted (per plan)** — labelled dev-only; `ConfigSecretsTests` still tracks; `${...}` env-injection pattern remains the model in the deploy artifact | unchanged |
|
||||
| C5 | Stale `Component-TraefikProxy.md` references | Low | **Fixed (verified)** — doc now names `OldestNodeActiveHealthCheck` with the correct path, documents `/health/ready` = database + akka-cluster + required-singletons, and the active check's DB tag | `Component-TraefikProxy.md:29,115-121,129` |
|
||||
| C6 | Five copy-pasted ~60-line singleton registration blocks | Low | **Fixed (verified)** for the central role — `CentralSingletonRegistrar` collapses manager/drain/proxy; the two *site* singletons remain hand-rolled by documented intent (role-scoped; see N5) | `CentralSingletonRegistrar.cs:31-71`; `AkkaHostedService.cs:459-465` |
|
||||
| U1 | No real multi-node failover test anywhere | — | **Fixed (verified)** — real two-node in-process cluster rig built from production HOCON, SBR kill test, active-node divergence test, plus a scripted docker drill (but see N1 — the drill itself was never executed) | `TwoNodeClusterFixture.cs`; `SbrFailoverTests.cs`; `ActiveNodeSemanticsTests.cs`; `CentralSingletonRegistrarTests.cs`; `docker/failover-drill.sh` |
|
||||
| U2 | Windows Service lifecycle / down-if-alone recovery story undefined | — | **Partially fixed** — `WhenTerminated` watchdog stops the host process on an out-of-band ActorSystem termination so a supervisor restarts it (`UnexpectedTerminationTests` cover it); the recovery *contract* is now spec'd. But the `install.ps1` `sc.exe failure` recovery actions were deferred to the artifact owner, and the contract does not converge for a non-first-seed survivor (see N1) | `AkkaHostedService.cs:203-217` (watchdog), `:342-346` (`_stopRequested` guard); `Component-ClusterInfrastructure.md:104-111`; tracker "Deferred during Wave 1" |
|
||||
| U3 | TLS absent at every layer | — | **Deferred (accepted, spec-visible)** — explicit "Production TLS profile: not yet implemented" roadmap note added as planned | `Component-TraefikProxy.md:133` |
|
||||
| U4 | No Dockerfile HEALTHCHECK; `/healthz` runs zero checks | — | **Deferred (accepted per plan)** — unchanged (`grep -c HEALTHCHECK docker/Dockerfile` = 0; `Program.cs:381`) | — |
|
||||
| U5 | `NodeName` missing from wonder-app-vd03 overlays → NULL `SourceNode` | — | **Not fixed** — the artifact's `Node` section still has no `NodeName` (verified by parsing the JSON). Deferred out-of-band with an owner; see N2 for why that rationale is shaky | `deploy/wonder-app-vd03/appsettings.Central.json` (`Node`: Role/NodeHostname/RemotingPort only) |
|
||||
| U6 | Known-but-open follow-ups (SecuredWrite SourceNode NULL; cert-trust persistence; docker-env2 mirrors findings) | — | **Mixed, all tracked** — SecuredWrite `SourceNode` fixed (register row 7, via `ICentralAuditWriter`); cert-trust persistence still deferred with trigger (register row 11); docker-env2 got `stop_grace_period` | `docs/plans/2026-07-08-deferred-work-register.md:23,35`; `docker-env2/docker-compose.yml:7-81` |
|
||||
| U7 | Online/offline transitions unrecorded | — | **Fixed (verified)** — `LastStatusChangeAt` stamped on every online↔offline flip (report-recovery, heartbeat-recovery, offline-detection), never moved by status-preserving traffic; surfaced as "offline since" in the UI; spec'd | `SiteHealthState.cs` (field + doc); `CentralHealthAggregator.cs:86,170-172,282`; `Health.razor:216`; `Component-HealthMonitoring.md:49` |
|
||||
|
||||
```hocon
|
||||
cluster {
|
||||
...
|
||||
split-brain-resolver {
|
||||
active-strategy = "keep-oldest"
|
||||
stable-after = 15000ms
|
||||
keep-oldest { down-if-alone = on }
|
||||
}
|
||||
failure-detector { ... }
|
||||
run-coordinated-shutdown-when-down = on
|
||||
}
|
||||
```
|
||||
**Disposition counts:** 18 Fixed (verified) · 3 Partially fixed (S12, C3, U2) · 1 Not fixed (U5, deferred out-of-band with owner) · 5 Deferred/accepted with rationale (P1, P3, C4, U3, U4) · 1 Mixed-tracked (U6). Plus one registered residual design gap on S1 (keep-oldest active-crash, deferred user decision).
|
||||
|
||||
but never sets `akka.cluster.downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider,
|
||||
Akka.Cluster"`. In Akka.NET (1.5.62 per `Directory.Packages.props:8-17`), the default downing
|
||||
provider is **NoDowning**; the whole `split-brain-resolver` section is only read by
|
||||
`SplitBrainResolverProvider`, which must be opted into via `downing-provider-class`. A repo-wide
|
||||
grep confirms the string `SplitBrainResolverProvider` appears **nowhere in `src/`** — only in the
|
||||
project's own reference notes, which show the required line
|
||||
(AkkaDotNet/03-Cluster.md:56, also 15-Coordination.md:69 and 18-MultiNodeTestRunner.md:100).
|
||||
---
|
||||
|
||||
Failure scenario: the active site node loses power. The standby's failure detector marks it
|
||||
unreachable (~10s), but **nothing ever downs it**. The unreachable member stays in the membership
|
||||
forever, the `ClusterSingletonManager` cannot hand over (singleton migration requires the oldest
|
||||
member's *removal*, not mere unreachability), and the site's DeploymentManager singleton — and with
|
||||
it all data collection, script execution, and S&F delivery — never restarts on the survivor. The
|
||||
same applies to all six central singletons (notification-outbox, audit-log-ingest, site-call-audit,
|
||||
audit-log-purge, site-audit-reconciliation, kpi-history-recorder). The documented "~25 seconds
|
||||
total failover" (docs/requirements/Component-ClusterInfrastructure.md:110-119) is only achievable
|
||||
today for a **graceful** stop (CoordinatedShutdown leave); crash and partition scenarios hang
|
||||
indefinitely until an operator downs the node or restarts the survivor.
|
||||
## New Findings (Round 2)
|
||||
|
||||
Corroborating evidence that this was never exercised:
|
||||
### [High] N1 — The down-if-alone recovery story does not converge for the non-first-seed survivor, and the shipped failover drill codifies a recovery the topology cannot deliver
|
||||
|
||||
- `HoconBuilderTests.cs:45,112,155` assert only that the SBR *strings* appear in the config.
|
||||
- `CoordinatedShutdownTests.cs:11-43` are source-text greps (`Assert.Contains("run-by-clr-shutdown-hook = on", content)`).
|
||||
- The "failover" integration tests are single-process simulations — e.g.
|
||||
`CentralFailoverTests.cs:27-47` validates a JWT across two `JwtTokenService` instances; no test
|
||||
forms a two-node cluster and kills the oldest node.
|
||||
- `run-coordinated-shutdown-when-down = on` (AkkaHostedService.cs:265) is dead config: a node is
|
||||
never transitioned to Down by anything.
|
||||
- `ClusterOptionsValidator` (src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptionsValidator.cs:35-63)
|
||||
rigorously rejects any strategy other than `keep-oldest` and demands `DownIfAlone == true` —
|
||||
validating knobs on a resolver that is not installed.
|
||||
The keep-oldest asymmetry itself is a *registered deferred user decision* (master tracker, 2026-07-08: crashing the **oldest/active** central node makes the younger survivor down itself — total central outage) and is honestly documented in `SbrFailoverTests.cs:15-23`. What is **new** in this round is that the artifacts built on top of that gap contradict it:
|
||||
|
||||
Fix: add the `downing-provider-class` line to `BuildHocon` (one line), or better, migrate the
|
||||
bootstrap to `Akka.Cluster.Hosting`'s typed options (already referenced,
|
||||
ZB.MOM.WW.ScadaBridge.Host.csproj:15) whose `SplitBrainResolverOption` wires the provider
|
||||
automatically — and add a real two-node in-process cluster test that kills the oldest member and
|
||||
asserts the singleton re-hosts within the expected window.
|
||||
1. **`docker/failover-drill.sh:12-16` kills the ACTIVE node** — under the now-unified oldest-member semantics, the active node *is* the oldest, i.e. the drill exercises exactly the crash keep-oldest cannot survive. The intended safety net is Task 20's restart loop: survivor self-downs → `run-coordinated-shutdown-when-down` terminates its ActorSystem → the watchdog (`AkkaHostedService.cs:203-217`) exits the process → docker `restart: unless-stopped` (`docker-compose.yml:27,54`) restarts the container.
|
||||
2. **But the restarted survivor can only re-bootstrap if it is the FIRST seed.** Both central nodes list `scadabridge-central-a` first (`docker/central-node-a/appsettings.Central.json:10-13`, `central-node-b/…:10-13`). Akka's seed-join rule lets only the *first* seed self-join to form a new cluster; a restarted `central-b` with `central-a` still dead loops on `InitJoin` forever — never `Up`, never `/health/active` 200. So when the drill's victim is `central-a` (the likelier oldest, since it is listed first and probed first by `active_container`), the drill fails at its 90s timeout and central stays dark until the victim returns. Only a `central-b` victim (or a younger-node crash) produces the advertised PASS.
|
||||
3. **The drill has never been executed**: `docker/README.md:288` — "*run after next deploy and record here* (the script was validated with `bash -n`; no live cluster was up when it was committed)" — yet `docker/README.md:286` promises "Expected failover: **~25s** … plus the Traefik ~5s health-check interval", and `Component-ClusterInfrastructure.md:111` claims "the restarted process rejoins as a fresh incarnation (the keep-oldest resolver handles the rejoin cleanly)" — true only while the peer is reachable.
|
||||
|
||||
### [High] "Active node" means two different things — cluster leader vs. oldest member — and they diverge
|
||||
**Recommendation:** (a) run the drill live once per direction (victim = a, victim = b) and record both outcomes in the README — the asymmetric result is itself the documentation the deferred decision needs; (b) fix the drill/README narrative to state which crash it can prove; (c) as a cheap partial mitigation independent of the strategy decision, make each node list *itself* first in its own `SeedNodes` (Akka tolerates asymmetric seed ordering) or document the operator action ("restart the dead node / restart the survivor with a self-first seed override") so a lone surviving central can actually re-form; (d) fold this into the pending keep-oldest topology/strategy decision (`keep-majority` + lighthouse node, `static-quorum`, or an accepted-risk note).
|
||||
|
||||
Two distinct node-selection concepts are used interchangeably:
|
||||
### [Medium] N2 — The wonder-app-vd03 overlay edits remain unapplied while the artifact sits editable in the working tree; the deferral rationale is factually wrong
|
||||
|
||||
- **Leader-based** (address ordering, *not* age): `ActiveNodeGate.IsActiveNode`
|
||||
(src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeGate.cs:36-52),
|
||||
`AkkaClusterNodeProvider.SelfIsPrimary` and the "Primary"/"Standby" labels
|
||||
(src/ZB.MOM.WW.ScadaBridge.Host/Health/AkkaClusterNodeProvider.cs:29-40,61), the `active-node`
|
||||
health check Traefik routes on (Program.cs:231-234, docker/traefik/dynamic.yml:12-18), the
|
||||
`CentralHealthReportLoop` leader gate (CentralHealthReportLoop.cs:93-97), and the site
|
||||
event-log purge gate (`SiteEventLogActiveNodeCheck` → `SelfIsPrimary`,
|
||||
SiteServiceRegistration.cs:116-120).
|
||||
- **Oldest-based** (singleton placement): every `ClusterSingletonManager` (AkkaHostedService.cs:410+,
|
||||
906-925), and the site health "IsActiveNode" flag, which is set by the DeploymentManager
|
||||
*singleton* on start/stop (src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs:243,258).
|
||||
Three PLAN-01 sub-edits (Tasks 16/20/23) were deferred with the rationale "that production deploy artifact is not tracked in this repo … the artifact directory itself is not present" (master tracker, Deferred during Wave 1). The directory **is** present at `deploy/wonder-app-vd03/` — it is merely gitignored (`.gitignore:48: /deploy/`) — and was clearly edited during this initiative (its `_comment_Security` at `appsettings.Central.json:40` describes the PLAN-07 DisableLogin removal and the new bundled GLAuth). So the artifact is being maintained in this working tree, yet still lacks:
|
||||
|
||||
In Akka the leader is the lowest-address `Up` member; the singleton host is the *oldest* member.
|
||||
They coincide at first boot but diverge permanently after the original first node is restarted and
|
||||
rejoins (it regains leadership by address order while the peer remains oldest). Concrete
|
||||
consequences once diverged:
|
||||
- `Node.NodeName` → every audit row from the **only production install** stamps a NULL `SourceNode`, defeating `IX_AuditLog_Node_Occurred` and the per-node stuck KPIs (round-1 U5, unchanged);
|
||||
- `Cluster.AllowSingleNodeCluster: true` + removal of the phantom seed `localhost:8091` (round-1 S12's artifact half — the node still dials a nonexistent seed forever);
|
||||
- the `install.ps1` `sc.exe failure` service-recovery actions that the down-if-alone watchdog contract *requires* ("the Host process must exit **so the service supervisor restarts it**" — `Component-ClusterInfrastructure.md:106-110`); without them, a self-downed single node on that host stays down.
|
||||
|
||||
1. **Site event-log retention is unenforced on the node that writes events.** Events are recorded
|
||||
by the singleton hierarchy on the oldest node; the purge (`EventLogPurgeService.cs:21,36`)
|
||||
early-exits unless `SelfIsPrimary` (leader). The active/oldest node's SQLite grows past the
|
||||
30-day/1GB cap while the leader purges an idle database.
|
||||
2. **Health dashboard labels the wrong node "Primary"** (AkkaClusterNodeProvider.cs:61): the node
|
||||
named Primary hosts no singletons; the node doing all the work shows Standby.
|
||||
3. **Central self-report is emitted by the leader** (CentralHealthReportLoop.cs:93) while the
|
||||
NotificationOutbox/audit singletons run on the other node — the `$central` health card describes
|
||||
the wrong node.
|
||||
4. **Traefik routes UI/API to the leader** while every singleton `Ask` then hops to the peer —
|
||||
functionally correct via the proxies, but it contradicts the design docs' "one active node runs
|
||||
all central components" model (Component-ClusterInfrastructure.md:44-47) and doubles
|
||||
intra-cluster latency on every outbox/audit page load.
|
||||
**Recommendation:** apply the three documented edits directly to the on-disk artifact (they are one-liners specified in PLAN-01 Tasks 16/20/23), or correct the deferral record and add a RUNBOOK step so the gap is operator-visible.
|
||||
|
||||
Recommendation: pick one definition. Simplest coherent choice: derive "active" from singleton-host
|
||||
status (e.g. compare self against the oldest `Up` member, or gate on reachability of a local
|
||||
singleton), and use it consistently for `ActiveNodeGate`, `SelfIsPrimary`, the purge gate, and the
|
||||
`/health/active` check — leadership is an Akka-internal concept that only accidentally matches the
|
||||
product's active/standby model.
|
||||
### [Low] N3 — Metrics-staleness can never fire for a site that has never delivered a report
|
||||
|
||||
### [High] During a central network partition, both nodes report `/health/active = 200` and Traefik serves both
|
||||
`CentralHealthAggregator.CheckForOfflineSites` guards the staleness check with `state.LastReportReceivedAt is { } lastReport` (`CentralHealthAggregator.cs:300-302`), so a site registered via heartbeat only (`MarkHeartbeat` registration path, `:114-122`, leaves `LastReportReceivedAt` null) is never flagged `IsMetricsStale` and never logs the staleness warning. This is the "stuck from first boot" variant of round-1 S7: a site whose DeploymentManager singleton never starts (so `IsActiveNode` is false on both nodes and `HealthReportSender` skips every report — `HealthReportSender.cs:85-86`) shows online-with-no-metrics indefinitely. Mitigation: the UI renders "awaiting first report" for the null timestamp (`Health.razor:222`), so the state is at least visible — but it carries no warning styling or aggregator log line, unlike the stale case. Fix: treat `LastReportReceivedAt == null` with `now - LastHeartbeatAt > MetricsStaleTimeout` (or first-seen time) as stale too.
|
||||
|
||||
`ActiveNodeGate`/the active-node check compute `cluster.State.Leader` from each node's own gossip
|
||||
view. When central-a and central-b cannot see each other but Traefik can reach both (a plausible
|
||||
partial partition), each side sees the peer as unreachable and computes **itself** as leader — both
|
||||
return 200 on `/health/active`, and Traefik's load balancer (docker/traefik/dynamic.yml:10-18)
|
||||
round-robins requests between two nodes that each believe they are active. Blazor circuits and the
|
||||
Inbound API's active-node 503 gate (Program.cs:256-260) are then split across both. Because the SBR
|
||||
is not enabled (Critical finding above), nothing ever resolves the partition — this state persists
|
||||
until manual intervention. Singletons do *not* duplicate (that requires member removal), but every
|
||||
leader-gated behavior does. Enabling keep-oldest + `down-if-alone` fixes the persistence; consider
|
||||
also making `/health/active` require `Ready` (DB reachable) in addition to leadership so a
|
||||
DB-dead leader is pulled from rotation.
|
||||
### [Low] N4 — `CentralHealthReportLoop` still narrates "cluster leader (Primary)"
|
||||
|
||||
### [Medium] Cluster-leave drain tasks are budgeted 10s inside a 5s CoordinatedShutdown phase
|
||||
The class doc ("Only the cluster leader (Primary) generates reports", `CentralHealthReportLoop.cs:12-14`) and the sequence-seed comment ("reports from a newly-elected central leader", `:44-45`) survived the S2 rewiring. The behavior is correct (`:93` gates on `IClusterNodeProvider.SelfIsPrimary`, now oldest-member), but the wording re-teaches the exact leader/oldest conflation the fix eradicated. Two-line comment fix.
|
||||
|
||||
Five singletons add a `PhaseClusterLeave` task that runs `GracefulStop(TimeSpan.FromSeconds(10))`
|
||||
(AkkaHostedService.cs:551-567 site-call-audit, 624-641 audit-log-purge, 676-693
|
||||
site-audit-reconciliation, 733-750 kpi-history-recorder, 787-804 pending-deployment-purge). Akka's
|
||||
`cluster-leave` phase uses the default phase timeout of **5 seconds**; the phase abandons the task
|
||||
at 5s (default `recover = on`) and proceeds, so the 10s graceful-stop can never complete its full
|
||||
window and the `catch` logging in those tasks never observes the overrun (the task is orphaned, not
|
||||
faulted). The intended "let the in-flight EF upsert drain before handover" guarantee is silently
|
||||
halved. Fix: set `akka.coordinated-shutdown.phases.cluster-leave.timeout` in `BuildHocon` to
|
||||
exceed the graceful-stop budget, or shrink the `GracefulStop` timeout below 5s.
|
||||
### [Low] N5 — Site-role singletons still lack drain tasks; the registrar can't express role scoping
|
||||
|
||||
### [Medium] The two busiest singletons have no drain task at all
|
||||
`CentralSingletonRegistrar` deliberately excludes the two site singletons because they need `.WithRole(siteRole)` (`AkkaHostedService.cs:464-465`), so `deployment-manager` (`:791-810`) and `event-log-handler` (`:839-845`) remain hand-rolled with bare `PoisonPill` termination and no `PhaseClusterLeave` drain. The S5 rationale (let in-flight writes drain before handover) applies to the DeploymentManager's SQLite static-override/native-alarm persistence just as it did to the central EF writers; the site side is safe-but-lossy on graceful failover. Adding an optional `role` parameter to `CentralSingletonRegistrar.Start` (and renaming it `SingletonRegistrar`) closes the asymmetry for a few lines.
|
||||
|
||||
The `NotificationOutboxActor` and `AuditLogIngestActor` singletons are terminated by bare
|
||||
`PoisonPill` with no cluster-leave drain (AkkaHostedService.cs:410-419, 451-458); the comment at
|
||||
AkkaHostedService.cs:544-549 explicitly notes the pattern "is suitable for the NotificationOutbox
|
||||
singleton; not added here to keep this change minimal". These are precisely the actors with
|
||||
in-flight EF transactions on every message (dispatch sweeps, dual-write ingest). The idempotent
|
||||
DB design (insert-if-not-exists on `NotificationId`/`EventId`) makes this safe-but-lossy rather
|
||||
than corrupting, but the inconsistency (drain for the low-traffic maintenance singletons, none for
|
||||
the hot ones) should be resolved.
|
||||
### [Low] N6 — PLAN-01's two live deferrals are missing from the canonical deferred-work register
|
||||
|
||||
### [Medium] Health-report loss recovery is dead code on the production transport
|
||||
`docs/plans/2026-07-08-deferred-work-register.md` (established by PLAN-08 T11 as the triage home, with rationale + revisit-trigger columns) contains neither (a) the keep-oldest active-crash availability gap nor (b) the wonder-app-vd03 overlay edits — both live only in `archreview/plans/00-MASTER-TRACKER.md` prose ("Follow-up discovered during P0 execution", "Deferred during Wave 1"). The register's structure (owner, rationale, revisit trigger) is exactly what the SBR gap — an availability-critical open decision — needs; today a reader of the register alone would conclude this domain has no open items.
|
||||
|
||||
`HealthReportSender` carefully restores interval counters if `_transport.Send` throws
|
||||
(HealthReportSender.cs:158-175), and `SiteHealthCollector.AddIntervalCounters`
|
||||
(SiteHealthCollector.cs:153-171) exists to support it. But the only production transport is
|
||||
`AkkaHealthReportTransport.Send` (src/ZB.MOM.WW.ScadaBridge.Host/AkkaHealthReportTransport.cs:26-33)
|
||||
— a fire-and-forget `Tell` to an `ActorSelection`, which **never throws** and silently returns when
|
||||
the actor system is null. Any downstream loss (central unreachable, ClusterClient reconnecting,
|
||||
dead-lettered selection) drops the report *and* the interval error counters it carried — the
|
||||
scenario the restore logic was built for is exactly the one it can't see. Either make the transport
|
||||
Ask-with-ack (the site→central notification forward already uses this pattern over the same actor)
|
||||
or accept the loss and delete the restore machinery; the current shape gives false confidence.
|
||||
---
|
||||
|
||||
### [Medium] Offline detection deviates from the spec and can mask a dead metrics pipeline
|
||||
## What's Genuinely Good
|
||||
|
||||
`Component-HealthMonitoring.md:43-46` defines offline as "no *report* within 60s".
|
||||
`CentralHealthAggregator.CheckForOfflineSites` (CentralHealthAggregator.cs:224-263) instead keys
|
||||
off `LastHeartbeatAt`, which is also advanced by the ~5s transport heartbeats
|
||||
(`CentralCommunicationActor.HandleHeartbeat` → `MarkHeartbeat`, CentralCommunicationActor.cs:349-353).
|
||||
A site whose `HealthReportSender` loop has died (or whose DeploymentManager singleton is stuck, so
|
||||
`IsActiveNode` is false on both nodes and reports are skipped — HealthReportSender.cs:85) keeps
|
||||
heartbeating and shows **online with silently frozen metrics forever**; nothing surfaces
|
||||
`LastReportReceivedAt` staleness. The heartbeat-based liveness choice is defensible (the code
|
||||
comment argues it), but then the dashboard needs a distinct "metrics stale" signal, and the design
|
||||
doc should be updated — currently spec and code disagree.
|
||||
- **The Critical fix is proven, not just present.** The downing provider is one HOCON line (`AkkaHostedService.cs:290`), but the team built a real two-node in-process cluster rig from *production* `BuildHocon` output (`TwoNodeClusterFixture.cs:56-64`) and asserted member removal + singleton survival after an abrupt terminate — the exact test shape whose absence let round-1's Critical survive. `HoconBuilderTests` now parse the config instead of grepping strings.
|
||||
- **Honest empiricism over plan compliance.** When the live rig revealed that killing the oldest node makes the survivor self-down, Task 4 was rewritten to the *achievable* guarantee and the asymmetry documented in the test XML doc (`SbrFailoverTests.cs:15-23`) and `ActiveNodeSemanticsTests.cs:16-24` (which switched to a graceful Leave with a documented deviation note) rather than shipping a green-but-lying test.
|
||||
- **The active-node unification is genuinely single-sourced.** One 48-line static evaluator (`ClusterActivityEvaluator`) now backs the inbound-API gate, `/health/active`, the Primary/Standby labels, the event-log purge gate, the central self-report loop, and (via the `SelfIsPrimary` seam) plan 02's S&F delivery gate — with a comment in each consumer pointing back at the review finding. The seam design let a *different* plan consume it without inventing a parallel check (`AkkaHostedService.cs:873-882`).
|
||||
- **`CentralSingletonRegistrar`** removed ~300 lines of copy-paste *and* fixed the drift it caused (missing drains) in one move — the correct root-cause response to a copy-paste bug.
|
||||
- **The health state machine is careful engineering:** immutable `SiteHealthState` records swapped via CAS loops with correct lost-race semantics (a lost CAS on offline-marking means the site was just heard from — deliberately not retried, `CentralHealthAggregator.cs:279-292`), clock-skew-anchored heartbeat recovery (`:145-156`), a collision-proof `$central` synthetic key, and cross-field options validation with key-naming messages (`HealthMonitoringOptionsValidator`).
|
||||
- **Doc reconciliation was done properly:** REQ-HOST-6 was amended with a rationale note *and* the unused packages dropped (instead of quietly deleting the requirement); the Traefik doc, the health-monitoring spec, and the cluster-infrastructure recovery contract all now describe the code as built.
|
||||
- **`DisableLoginGuard`** is the right shape for the C2 fix: fail-fast at boot, environment-gated, with an explicit consciously-named acknowledgement key — a durable control rather than a one-time artifact edit.
|
||||
|
||||
### [Medium] Second seed node in the checked-in site dev config targets the metrics port; validator has a gap
|
||||
## Severity Tally — New Findings Only
|
||||
|
||||
`src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json:14-17` seeds
|
||||
`akka.tcp://scadabridge@localhost:8082` and `akka.tcp://scadabridge@localhost:8084`, while the same
|
||||
file sets `MetricsPort: 8084` (line 10). The second seed points at the Kestrel HTTP/1.1 metrics
|
||||
listener — a node attempting to join via that seed performs a doomed Akka.Remote association
|
||||
against Kestrel (handshake noise, retry loops). `StartupValidator` explicitly guards seed-vs-GrpcPort
|
||||
(StartupValidator.cs:121-127) but not seed-vs-MetricsPort, so this config validates cleanly. Fix
|
||||
the dev config (8084 → the intended second node's remoting port) and extend the seed-port loop to
|
||||
also reject `MetricsPort`.
|
||||
|
||||
### [Medium] Docker stop grace period is shorter than a full CoordinatedShutdown
|
||||
|
||||
`docker/docker-compose.yml` sets no `stop_grace_period` (default 10s SIGTERM→SIGKILL) and the
|
||||
Dockerfile runs `dotnet` as PID 1 (docker/Dockerfile, final stage). A graceful shutdown runs
|
||||
cluster-leave (5s budget) + cluster-exiting (10s) + actor-system-terminate plus Serilog flush —
|
||||
easily exceeding 10s on a loaded node, so `bash docker/deploy.sh` recreates can SIGKILL mid-drain,
|
||||
turning every "graceful" redeploy into the crash path (which, per the Critical finding, currently
|
||||
has no automatic recovery on the surviving node). Add `stop_grace_period: 30s` to the app services.
|
||||
|
||||
### [Low] Aggregator never forgets sites
|
||||
|
||||
`CentralHealthAggregator._siteStates` (CentralHealthAggregator.cs:16) only grows: a site deleted
|
||||
from configuration remains a permanently-offline dashboard tile (and continues to be enumerated by
|
||||
`SiteHealthKpiSampleSource.CollectAsync`, SiteHealthKpiSampleSource.cs:74-85) until process
|
||||
restart. Needs an eviction path wired to site deletion.
|
||||
|
||||
### [Low] Dead-letter monitor: unbounded warning volume and restart-reset count
|
||||
|
||||
`DeadLetterMonitorActor` logs one Warning per dead letter with no sampling/rate-limiting and keeps
|
||||
its count in actor state that resets on restart
|
||||
(src/ZB.MOM.WW.ScadaBridge.Host/Actors/DeadLetterMonitorActor.cs:14,26-35). A dead-letter storm
|
||||
(mass undeploy, failover, shutdown) floods the log; the health metric path
|
||||
(`IncrementDeadLetter`) is fine because the collector counter is interval-based.
|
||||
|
||||
### [Low] Mandatory two-seed rule forces phantom seeds in single-node installs
|
||||
|
||||
`ClusterOptionsValidator` requires ≥2 seed nodes
|
||||
(ClusterOptionsValidator.cs:29-33); the shipped single-node production artifact satisfies it with
|
||||
a documented placeholder (`deploy/wonder-app-vd03/appsettings.Central.json:9-19`, second seed
|
||||
`localhost:8091` "harmless while absent"). The node dials the phantom seed forever (log noise,
|
||||
periodic connect attempts), and the rule's intent — no startup-ordering dependency — is defeated
|
||||
by fiction. Consider allowing 1 seed with a warning, or a `SingleNode` acknowledgement flag.
|
||||
|
||||
## 2. Performance
|
||||
|
||||
This domain is largely cold-path and correctly engineered for it; nothing here touches the
|
||||
data-plane hot path. Observations:
|
||||
|
||||
- **[Low] Readiness probes fan out five cluster Asks per poll.**
|
||||
`RequiredSingletonsHealthCheck` sends 5 concurrent `Identify` Asks through singleton proxies
|
||||
every `/health/ready` poll (RequiredSingletonsHealthCheck.cs:66-78,111-117). When the singletons
|
||||
live on the peer node each probe is a remoting round trip. Bounded (2s, concurrent) and cheap at
|
||||
typical poll rates, but a tight orchestration probe interval multiplies cluster chatter.
|
||||
- **[Low] Startup compiles every inbound API method sequentially before serving**
|
||||
(Program.cs:379-388). Startup latency scales linearly with method count; a slow Roslyn compile of
|
||||
many methods delays readiness on the *standby* too (it runs the same branch). Acceptable now;
|
||||
parallelize or defer-compile if method counts grow.
|
||||
- **[Low] Per-tick allocations in health collection** (dictionary snapshots per 30s report,
|
||||
`GetAllSiteStates` copy per dashboard poll — CentralHealthAggregator.cs:173-176) are trivial at
|
||||
this cadence. No action.
|
||||
- **Positive:** the site audit telemetry actor is pinned to a dedicated dispatcher
|
||||
(`audit-telemetry-dispatcher`, AkkaHostedService.cs:226-232,1150-1158) so batch SQLite/gRPC work
|
||||
never contends with hot-path actors; `StoreAndForwardService.StartAsync` is awaited rather than
|
||||
sync-blocked (AkkaHostedService.cs:976-981); the ActorSystem DI bridge is a root singleton
|
||||
precisely to avoid per-scope disposal (Program.cs:240-250) — all correct.
|
||||
|
||||
## 3. Conventions
|
||||
|
||||
- **[Medium] REQ-HOST-6 says "Akka.NET bootstrap via Akka.Hosting"; the code hand-rolls HOCON.**
|
||||
`Component-Host.md` (REQ-HOST-6, line 106-114) mandates Akka.Hosting; the actual bootstrap is
|
||||
string-interpolated HOCON + `ActorSystem.Create` (AkkaHostedService.cs:171-176,214-271) with
|
||||
`Akka.Hosting`/`Akka.Cluster.Hosting`/`Akka.Remote.Hosting` referenced but unused for cluster
|
||||
bring-up (ZB.MOM.WW.ScadaBridge.Host.csproj:15-18). This is not just doc drift: Akka.Cluster.Hosting's
|
||||
typed `SplitBrainResolverOption` sets `downing-provider-class` for you — using the mandated API
|
||||
would have prevented the Critical finding. Either implement REQ-HOST-6 or amend it and drop the
|
||||
unused packages.
|
||||
- **[High] Production deploy artifact ships with authentication disabled.**
|
||||
`deploy/wonder-app-vd03/appsettings.Central.json:39-42` sets `"DisableLogin": true` — every
|
||||
request auto-authenticates as `multi-role` with all roles, and the management API on that host is
|
||||
unauthenticated. The base config documents this as "DEV/TEST ONLY — never enable in production"
|
||||
with "no environment guard; a startup warning is the only protection"
|
||||
(src/ZB.MOM.WW.ScadaBridge.Host/appsettings.json:6). A `deploy/` artifact with an install
|
||||
runbook is a production surface; at minimum gate `DisableLogin` on
|
||||
`Environment.IsDevelopment()` or require an explicit `I-understand` companion flag.
|
||||
- **[Medium] LDAP transport is plaintext everywhere, contradicting the security spec.** The design
|
||||
requires "LDAPS/StartTLS required" (CLAUDE.md Security & Auth; Component-Security), but every
|
||||
shipped config — docker (docker/central-node-a/appsettings.Central.json:26-33) and the on-host
|
||||
deploy (deploy/wonder-app-vd03/appsettings.Central.json:26-34) — sets `"Transport": "None",
|
||||
"AllowInsecure": true` and sends the service-account bind password in clear. Dev GLAuth is a fair
|
||||
excuse for docker; the deploy artifact should not normalize it.
|
||||
- **[Medium] Secrets in checked-in dev configs.** SQL credentials, LDAP service-account password
|
||||
and the JWT signing key are committed in `docker/central-node-a/appsettings.Central.json:21-38`
|
||||
(labelled dev-only; the compose pepper is likewise flagged, docker-compose.yml:9-14). The
|
||||
wonder-app-vd03 artifact handles this correctly via `${...}` placeholders + env injection
|
||||
(appsettings.Central.json:20-24) — good pattern, worth backporting a scrubbed variant to docker
|
||||
if these ever leave a lab network. Also note `ConfigSecretsTests.cs` exists in Host.Tests, so the
|
||||
team is tracking this.
|
||||
- **[Low] Stale doc references.** `Component-TraefikProxy.md:121` locates `ActiveNodeHealthCheck`
|
||||
at `src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeHealthCheck.cs`; the check now comes from the
|
||||
shared `ZB.MOM.WW.Health.Akka` package (Program.cs:214-234) and no such file exists.
|
||||
`Component-TraefikProxy.md:115` also still describes `/health/ready` as "database + Akka
|
||||
cluster" — it now includes `required-singletons` (Program.cs:227-230; Component-Host.md REQ-HOST-4a
|
||||
*is* current). Keep the Traefik doc in sync.
|
||||
- **[Low] Five copy-pasted ~60-line singleton registration blocks.**
|
||||
AkkaHostedService.RegisterCentralActors repeats the manager/drain-task/proxy triple five times
|
||||
(AkkaHostedService.cs:399-811) with only names and Props varying. Extract a
|
||||
`StartCentralSingleton(name, props, drainTimeout)` helper — the copy-paste already caused drift
|
||||
(outbox/ingest lack drain tasks).
|
||||
- **Positive conventions compliance:** Options pattern is followed rigorously (component-owned
|
||||
options classes, `ValidateOnStart`, dedicated validators with key-naming messages —
|
||||
ClusterOptionsValidator, HealthMonitoringOptionsValidator); HOCON interpolation is
|
||||
injection-safe (`QuoteHocon`/`DurationHocon`, AkkaHostedService.cs:280-295); Serilog enrichment
|
||||
matches REQ-HOST-8; startup validation matches REQ-HOST-4 including cross-field port-collision
|
||||
rules; single-binary role dispatch matches REQ-HOST-1/2; the `/health/ready` vs `/health/active`
|
||||
tier separation (leadership deliberately excluded from readiness) is well-reasoned and
|
||||
documented in place (Program.cs:346-356).
|
||||
|
||||
## 4. Underdeveloped Areas
|
||||
|
||||
1. **No real multi-node failover test anywhere.** `CentralFailoverTests` are single-process
|
||||
simulations (JWT across two service instances, CentralFailoverTests.cs:27-47);
|
||||
`SiteFailoverTests`/`DualNodeRecoveryTests` similarly exercise component state recovery, not
|
||||
cluster membership; `CoordinatedShutdownTests` grep source text (CoordinatedShutdownTests.cs:11-43);
|
||||
`HoconBuilderTests` assert config strings. The repo's own notes describe the MultiNode test
|
||||
runner (AkkaDotNet/18-MultiNodeTestRunner.md) but it is unused. This is the direct cause of the
|
||||
Critical finding surviving: nothing ever killed the oldest node and waited for the singleton to
|
||||
move. Even a plain xunit test spinning two `ActorSystem`s in-process and calling
|
||||
`cluster.Down()`/terminating one would have caught it.
|
||||
2. **Windows Service lifecycle management is thin.** `Component-ClusterInfrastructure.md:19`
|
||||
claims "Manage Windows service lifecycle (start, stop, restart)"; the implementation is
|
||||
`UseWindowsService()` (Program.cs:74,401) plus install scripts. Service recovery actions
|
||||
(restart-on-failure), the watchdog implied by `down-if-alone` ("an external mechanism must
|
||||
restart both nodes" — AkkaDotNet/03-Cluster.md:64) do not exist in the repo; after a
|
||||
down-if-alone self-down the process presumably exits (or worse, idles) with nothing to restart
|
||||
it. The down-if-alone recovery story needs an explicit design (does the process exit? does the
|
||||
service restart it? does it rejoin cleanly?) and a drill.
|
||||
3. **TLS is absent at every layer of this domain**: Traefik terminates plain HTTP with an
|
||||
unauthenticated dashboard (docker/traefik/traefik.yml:5-7 — documented as dev-only in
|
||||
Component-TraefikProxy.md:131-138), Akka remoting is unencrypted TCP, site gRPC is `http://`.
|
||||
Fine for the lab; there is no production TLS profile or documented path yet.
|
||||
4. **Dockerfile has no `HEALTHCHECK`** and compose has no health-based dependency ordering;
|
||||
liveness `/healthz` runs zero checks (Program.cs:353-355) so a wedged actor system still
|
||||
reports live. Acceptable minimalism, but an orchestrator migration will need both.
|
||||
5. **`NodeName` is missing from the wonder-app-vd03 deploy overlays**
|
||||
(deploy/wonder-app-vd03/appsettings.Central.json has no `Node.NodeName`), so audit rows from
|
||||
that install get a NULL `SourceNode` (the base file documents the normalisation,
|
||||
src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Central.json:3) — undermines the per-node audit/KPI
|
||||
features (`IX_AuditLog_Node_Occurred`, per-node stuck KPIs) on the one real deployment.
|
||||
6. **Known-but-open follow-ups acknowledged in code/docs:** SecuredWrite audit rows leave
|
||||
`SourceNode` NULL (CLAUDE.md notes it as logged); no central persistence of OPC UA cert trust;
|
||||
`docker-env2` mirrors the main topology (verified: same Traefik config shape, same
|
||||
seed/port conventions) so it inherits every finding above.
|
||||
7. **Health monitoring has no history by design** (Component-HealthMonitoring.md:93-97) — now
|
||||
partially superseded by KPI History (#26), which the docs handle; but online/offline
|
||||
*transitions* are still unrecorded anywhere (doc line 97 defers it), so post-incident "when did
|
||||
the site drop" questions remain unanswerable beyond log scraping.
|
||||
|
||||
## Prioritized Recommendations
|
||||
|
||||
1. **[Critical, one line + tests] Enable the split-brain resolver.** Add
|
||||
`downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster"` to
|
||||
`BuildHocon` (AkkaHostedService.cs:250) — or move the bootstrap to Akka.Cluster.Hosting per
|
||||
REQ-HOST-6, which sets it via typed options. Then add a two-node in-process failover test
|
||||
(kill oldest → assert singleton re-hosted, member removed within stable-after + margin) and a
|
||||
documented manual drill against the docker cluster (`docker kill scadabridge-central-a`, watch
|
||||
Traefik + singleton logs).
|
||||
2. **[High] Unify the "active node" definition** on singleton-host (oldest) semantics for
|
||||
`ActiveNodeGate`, `AkkaClusterNodeProvider.SelfIsPrimary`, the event-log purge gate, and the
|
||||
`/health/active` check; today's leader-based checks purge the wrong node's event log and label
|
||||
the wrong node Primary once leader ≠ oldest.
|
||||
3. **[High] Remove `DisableLogin: true` from `deploy/wonder-app-vd03`** or add a hard environment
|
||||
guard around the flag.
|
||||
4. **[Medium] Fix the CoordinatedShutdown drain budget** (`cluster-leave` phase timeout ≥
|
||||
graceful-stop timeout) and add drain tasks for the notification-outbox and audit-log-ingest
|
||||
singletons — or delete all five drains if the idempotent-DB argument makes them unnecessary.
|
||||
5. **[Medium] Fix `appsettings.Site.json` seed `localhost:8084`** and extend `StartupValidator`'s
|
||||
seed-port loop to reject `MetricsPort` collisions.
|
||||
6. **[Medium] Make health-report delivery observable**: switch `AkkaHealthReportTransport` to an
|
||||
acked Ask (reusing the existing site→central forward pattern) so the counter-restore logic in
|
||||
`HealthReportSender` actually fires, and surface `LastReportReceivedAt` staleness on the
|
||||
dashboard so heartbeat-alive/metrics-dead sites are visible.
|
||||
7. **[Medium] Add `stop_grace_period: 30s`** to the app services in both compose files.
|
||||
8. **[Low] Housekeeping:** evict deleted sites from `CentralHealthAggregator`; rate-limit
|
||||
dead-letter warnings; extract the singleton-registration helper; sync
|
||||
`Component-TraefikProxy.md` with the shared health-check library; add `NodeName` to the
|
||||
wonder-app-vd03 overlays; reconsider the mandatory-two-seeds rule for single-node installs.
|
||||
| Severity | Count | Findings |
|
||||
|----------|-------|----------|
|
||||
| Critical | 0 | — |
|
||||
| High | 1 | N1 (recovery story doesn't converge for non-first-seed survivor; never-run drill codifies an undeliverable ~25s recovery) |
|
||||
| Medium | 1 | N2 (wonder-app-vd03 overlay edits unapplied; deferral rationale factually wrong) |
|
||||
| Low | 4 | N3 (metrics-stale null-report gap), N4 (stale "leader" comments), N5 (site singletons undrained / registrar role gap), N6 (deferrals missing from canonical register) |
|
||||
|
||||
Reference in New Issue
Block a user