dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs(archreview): round-2 re-review (2026-07-12) + 8 fix plans (86 tasks)

Browse Source 
Re-ran all 8 domain reviews at HEAD 8c888f13 against the b910f5eb baseline:
every round-1 finding source-verified (168 fixed, 0 regressions, 0 false
claims); 56 new findings (1 Critical / 4 High / 15 Medium / 36 Low),
concentrated in post-baseline code (anti-entropy resync, KPI rollup
backfill, live alarm stream) and seams the fixes exposed.

Headliners: S&F resync predicate inversion can wipe the delivering node's
buffer (02-N1 Critical); resync snapshot exceeds the Akka remoting frame
size (02-N2); failover drill kills the one node keep-oldest can't survive
(01-N1); unbounded rollup backfill per failover (04-R1); live production
API key in untracked test.txt (08-NF1).

Adds PLAN-R2-01..08 + .tasks.json manifests and the Round-2 board,
P0 list, cross-plan mutexes, and wave order in 00-MASTER-TRACKER.
This commit is contained in:
Joseph Doherty
2026-07-12 23:52:10 -04:00
parent 8c888f13a0
commit 5bbd7689fa
26 changed files with 7236 additions and 1321 deletions
Download Patch File Download Diff File Expand all files Collapse all files
archreview/00-OVERALL.md
+84 -84
View File
@@ -1,122 +1,122 @@
# ScadaBridge Deep Architecture Review — Overall Report
# ScadaBridge Deep Architecture Review — Overall Report (Round 2)
**Date:** 2026-07-08
**Scope:** Full system — all 27 components, docker/deploy topology, tests, and design docs
**Method:** 8 parallel domain reviews, each reading the component design docs first and then the actual source (findings are grounded in `file:line` citations in the domain reports). Dimensions: **stability, performance, conventions, underdeveloped areas.**
**Date:** 2026-07-12 (round 2; round 1 dated 2026-07-08)
**Scope:** Full system — all 27 components, docker/deploy topology, tests, and design docs, at HEAD `8c888f13`
**Method:** 8 parallel domain re-reviews. Each re-read its round-1 report and fix plan (PLAN-01…08, all complete — 191/192 tasks merged by 2026-07-10), then **verified every round-1 finding's disposition against current source** (trusting code, not plan claims), then swept the ~268 commits since the pre-initiative baseline `b910f5eb` — the fix-plan code itself plus the two post-initiative features nobody had reviewed (aggregated live alarm stream, KPI hourly rollups) — for **new** findings. Dimensions: stability, performance, conventions, underdeveloped areas; all findings carry `file:line` citations in the domain reports.
## Report Index
| # | Report | Domain | Verdict (one line) |
|---|--------|--------|--------------------|
| 01 | [Cluster, Host & Failover](01-cluster-host-failover.md) | ClusterInfrastructure, Host, HealthMonitoring, docker/Traefik/deploy | High micro-level polish undone by one critical foundation crack: the split-brain resolver is configured but never enabled — automatic failover does not work for hard crashes. |
| 02 | [Communication & Store-and-Forward](02-communication-store-and-forward.md) | ClusterClient, gRPC streaming, S&F engine | High code quality in-the-small; mid maturity as a distributed system — serious lifecycle/topology defects the test suite is structurally blind to. |
| 03 | [Site Runtime & DCL](03-site-runtime-dcl.md) | SiteRuntime, DataConnectionLayer, SiteEventLogging | Mature and unusually carefully engineered; residual risk in adapter reconnect lifecycle, script containment, and deployment honesty. |
| 04 | [Data & Audit Backbone](04-data-audit-backbone.md) | ConfigurationDatabase, AuditLog, NotificationOutbox, SiteCallAudit, KpiHistory | Genuinely mature data layer with scale-dependent operational time bombs that will surface within months of production volume. |
| 05 | [Templates, Deployment & Transport](05-templates-deployment-transport.md) | TemplateEngine, DeploymentManager, Transport, ScriptAnalysis | Failure-path engineering is production-grade, but the Transport import apply path and newest template features contain multiple silent-data-loss and staleness defects. |
| 06 | [Edge Integrations](06-edge-integrations.md) | InboundAPI, ExternalSystemGateway, NotificationService, DelmiaNotifier | Mature, heavily-hardened edge layer; risk concentrated in one silent cache-coherence bug and acknowledged ESG spec gaps. |
| 07 | [UI, Management & Security](07-ui-management-security.md) | CentralUI, ManagementService, CLI, Security | Mature, carefully engineered surface undermined by sharp asymmetries: an auth-disabled deploy artifact, ignored deploy scoping, inconsistent secret elision. |
| 08 | [Conventions, Tests & Underdeveloped Areas](08-conventions-tests-underdeveloped.md) | Cross-cutting | Unusually disciplined codebase — the documented conventions are actually followed; gaps are narrow, specific, and mostly consciously logged. |
| # | Report | Domain | Round-2 verdict (one line) |
|---|--------|--------|----------------------------|
| 01 | [Cluster, Host & Failover](01-cluster-host-failover.md) | ClusterInfrastructure, Host, HealthMonitoring, docker/Traefik/deploy | Round 1's Critical (SBR never enabled) fixed and behaviorally proven; active-node model coherently unified — but the never-run failover drill kills the one node keep-oldest can't survive, and seed-order blocks the recovery loop when central-b is the survivor. |
| 02 | [Communication & Store-and-Forward](02-communication-store-and-forward.md) | ClusterClient, gRPC streaming, S&F engine, live alarm stream | Round 1's Critical and all three Highs genuinely fixed and test-backed; the new live alarm stream is solid — but the new anti-entropy peer resync was built on the wrong active-node predicate and an undeliverable-size Akka message: 1 new Critical + 1 new High, both confined to that feature. |
| 03 | [Site Runtime & DCL](03-site-runtime-dcl.md) | SiteRuntime, DataConnectionLayer, SiteEventLogging | All three round-1 risk concentrations (adapter reconnect, script containment, deployment honesty) genuinely fixed — 26/28 verified; one Medium residual on the MxGateway reconnect fault path. |
| 04 | [Data & Audit Backbone](04-data-audit-backbone.md) | ConfigurationDatabase, AuditLog, NotificationOutbox, SiteCallAudit, KpiHistory | All round-1 time bombs genuinely defused and verified; the one new material risk is the brand-new KPI rollup **backfill**, which re-introduces at startup the exact unbounded year-scale single-pass shape the plan eliminated everywhere else. |
| 05 | [Templates, Deployment & Transport](05-templates-deployment-transport.md) | TemplateEngine, DeploymentManager, Transport, ScriptAnalysis | All 26 actionable round-1 findings — including the Critical inheritance-edge loss and the whole silent-data-loss family — fixed in code with zero regressions; residue is 1 Medium perf leftover and 5 Lows. |
| 06 | [Edge Integrations](06-edge-integrations.md) | InboundAPI, ExternalSystemGateway, NotificationService, DelmiaNotifier | All round-1 risk genuinely fixed and source-verified (20/25 fixed, 0 regressions); new residual risk is 1 Medium + 4 Low — the Medium being the script-artifact invalidation bus publishing to zero subscribers. |
| 07 | [UI, Management & Security](07-ui-management-security.md) | CentralUI, ManagementService, CLI, Security | Both round-1 Criticals dead, 17/24 fully fixed, 0 regressions; remaining new findings are Medium-and-below — mostly security seams the fixes themselves exposed (proxy-blind throttle keying, unthrottled hub bind, unscoped secured writes). |
| 08 | [Conventions, Tests & Underdeveloped](08-conventions-tests-underdeveloped.md) | Cross-cutting | Every actionable round-1 finding genuinely fixed and conventions held through 268 commits of churn; new residue is small and mostly hygiene — except a live production API key sitting in an untracked working-tree file. |
## Severity Tally (tag occurrences per report)
## Round-1 Finding Disposition (verified against source, not plan claims)
| Report | Tracked | Fixed (verified) | Partial | Deferred/accepted | Not fixed | Regressed |
|--------|--------:|-----------------:|--------:|------------------:|----------:|----------:|
| 01 Cluster/Host/Failover | 28 | 18 | 3 | 6 | 1¹ | 0 |
| 02 Communication/S&F | 26 | 22 | 2 | 2 | 0 | 0 |
| 03 Site Runtime/DCL | 28 | 26 | 0 | 2 | 0 | 0 |
| 04 Data/Audit Backbone | 26 | 19 | 0 | 7² | 0 | 0 |
| 05 Templates/Deploy/Transport | 32 | 26 | 1 | 5 | 0 | 0 |
| 06 Edge Integrations | 25 | 20 | 2 | 3 | 0 | 0 |
| 07 UI/Management/Security | 24 | 17 | 6 | 1³ | 0 | 0 |
| 08 Conventions/Tests | 30 | 20 | 3 | 7 | 0 | 0 |
| **Total** | **~219** | **168** | **17** | **~33** | **1** | **0** |
¹ U5 (NodeName in the wonder-app-vd03 overlay) — deferred out-of-band with an owner, but see new finding 01-N2.
² Includes 1 won't-fix (S10). ³ UA7 was a false premise (No longer applicable).
**Headline: zero regressions, zero false "fixed" claims found.** Every disposition was re-derived from current source with `file:line` evidence. Several fixes exceeded their tickets (content-based inbound handler invalidation, two-signal deploy join, trigger-body trust coverage); PLAN-05's reflection round-trip guard structurally forecloses the entire DTO-drift class that caused round 1's Transport Critical.
## New-Findings Severity Tally (round 2 only)
| Report | Critical | High | Medium | Low |
|--------|---------:|-----:|-------:|----:|
| 01 Cluster/Host/Failover | 1 | 5 | 13 | 9 |
| 02 Communication/S&F | 2 | 6 | 11 | 11 |
| 03 Site Runtime/DCL | 0 | 4 | 6 | 11 |
| 04 Data/Audit Backbone | 0 | 6 | 11 | 6 |
| 05 Templates/Deploy/Transport | 2 | 13 | 15 | 5 |
| 06 Edge Integrations | 0 | 4 | 11 | 11 |
| 07 UI/Management/Security | 2 | 8 | 12 | 5 |
| 08 Conventions/Tests | 0 | 4 | 6 | 10 |
| **Total (approx.)** | **7** | **50** | **85** | **68** |
| 01 Cluster/Host/Failover | 0 | 1 | 1 | 4 |
| 02 Communication/S&F | 1 | 1 | 2 | 5 |
| 03 Site Runtime/DCL | 0 | 0 | 1 | 5 |
| 04 Data/Audit Backbone | 0 | 1 | 3 | 3 |
| 05 Templates/Deploy/Transport | 0 | 0 | 1 | 5 |
| 06 Edge Integrations | 0 | 0 | 1 | 4 |
| 07 UI/Management/Security | 0 | 0 | 4 | 5 |
| 08 Conventions/Tests | 0 | 1 | 2 | 5 |
| **Total** | **1** | **4** | **15** | **36** |
*(Counts are raw tag occurrences; a few findings are cited in more than one report — e.g. the `DisableLogin` deploy artifact appears in both 01 and 07.)*
Round 1 closed with 7 Critical / 50 High / 85 Medium / 68 Low. Round 2 opens with 1 / 4 / 15 / 36 — an ~80% reduction at every severity, and the residue is concentrated almost entirely in **code written after round 1**, not in anything round 1 flagged.
## Overall Verdict
ScadaBridge is an unusually disciplined codebase at the micro level. The conventions documented in CLAUDE.md are **actually followed** — verified, not assumed: zero `DateTime.Now` in `src/`, persistence-ignorant Commons entities, a clean project-reference graph with no site→central contamination, correlation IDs on request/response messages, additive message contracts, idempotent ingest keyed on `EventId`/`NotificationId`, timing-oracle-free API-key auth, and only two real TODOs in ~190k LOC. Test volume is real (~5,050 behavioral tests across 30 projects — Akka TestKit, bUnit, Playwright, MSSQL-backed integration tests).
The arch-review initiative worked: the fix plans delivered what they claimed, verified in source, with zero regressions. The system's round-1 risk profile — the failover crack, the Transport silent data loss, the data-layer time bombs, the cache-coherence family, the security projection leaks — is genuinely closed.
The system's risk is **not** diffuse sloppiness. It concentrates in exactly the four places a single-process test suite cannot see:
The round-2 risk profile is different in kind: **the new findings live almost exclusively in the newest code** — the anti-entropy resync (built by PLAN-02 itself), the KPI rollup backfill and the live alarm stream (both shipped post-initiative, 2026-07-10), and operational seams the fixes exposed. This is the expected shape for a healthy hardening cycle: the reviewed-and-fixed code held; the code that had never been reviewed is where the defects are.
1. **Real multi-node failover** — the paths that only execute when a node hard-crashes, a partition forms, or the standby takes over.
2. **Cache coherence across process/node boundaries** — compiled-script and staleness caches that are only invalidated on the node that made the change.
3. **Cross-environment data movement** — the Transport bundle import apply path, where DTOs, spec, and apply code drifted apart.
4. **Production-scale data volume** — purge/retention/KPI paths tested only at small scale.
**Bottom line: the codebase graduated from "production-grade engineering, not production-proven HA" to "one new feature (peer resync) must not ship as-is, one secret must be rotated today, and the failover drill still needs to actually be run."**
**Bottom line: production-grade engineering, not yet production-proven HA.** The single most important sentence in this review: *automatic failover for hard crashes does not currently work* (finding C1 below), and no test in the repo would have caught it.
## The Findings That Matter Most (fix before anything else)
## The Critical Findings (fix before anything else)
**C1 (new). The S&F anti-entropy resync can wipe the delivering node's live buffer.** The delivery gate uses oldest-Up (`SelfIsPrimary`) but the resync authority check uses leader+Up (lowest address). After a routine rolling restart of the lower-address site node the two predicates diverge persistently: the *delivering* node requests a resync from the stale rejoined leader and `ReplaceAllAsync`-wipes its own live buffer — silently losing every notification/cached call buffered during the outage and resurrecting delivered rows as duplicates. *(Report 02 N1; `SiteReplicationActor.cs:190-198` vs `AkkaHostedService.cs:881`)*
**C1. The split-brain resolver is configured but never enabled.** `BuildHocon` never sets `akka.cluster.downing-provider-class`, so the keep-oldest SBR settings are inert: unreachable nodes are never downed, and cluster singletons (site DeploymentManager = all data collection; all six central singletons) never migrate on a hard crash or partition. The documented ~25s failover only works for graceful stops. The repo's own Akka reference notes (`AkkaDotNet/03-Cluster.md`) show the required line. *(Report 01; `Host/Actors/AkkaHostedService.cs:225-270`)*
**H1 (new). The resync snapshot is undeliverable at exactly the scale it exists for.** Up to 10,000 full rows travel as ONE Akka remoting message with no `maximum-frame-size` override (default 128 KB) — silently dropped for any realistic backlog. *(Report 02 N2; `SiteReplicationActor.cs:40,407-410,460`)*
**C2. Editing a site's addresses crash-loops central routing to all sites.** `CentralCommunicationActor` stops and immediately recreates a *named top-level* ClusterClient in the same message handling; `Context.Stop` is async, so the name collides and the actor crash-restarts into a permanent loop. Tests mock the client factory, so this is never exercised. *(Report 02; `CentralCommunicationActor.cs:514-521`)*
**H2 (new). A live production Inbound API key sits in untracked `test.txt` at the repo root** (`sbk_eb5acc…` for `wonder-app-vd03.zmr.zimmer.com:8085`) — one `git add -A` from history. Rotate the key and delete the file **today**. *(Report 08 NF1; `test.txt:3`)*
**C3. Transport bundle import silently drops template inheritance.** `TemplateDto.BaseTemplateName` is exported and diffed but never consumed at apply — derived templates land as root templates and flatten to amputated configs in the target environment. *(Report 05; `Transport/Import/BundleImporter.cs:1524`)*
**H3 (new). The failover drill codifies a recovery the topology cannot deliver.** `failover-drill.sh` kills the ACTIVE(=oldest) central node — the one crash keep-oldest cannot survive (registered deferred decision) — and the restart recovery loop cannot converge when central-b is the survivor because both nodes list central-a first in `SeedNodes` (Akka's first-seed rule blocks a non-first seed from self-bootstrapping). The README admits the drill was never run live yet promises ~25s failover; the ~25s envelope also remains the last unmeasured headline design claim (the perf harness's skip precondition — the two-node rig — landed 2026-07-08). *(Reports 01 N1, 08 NF2)*
**C4. The on-host production deploy artifact ships `DisableLogin: true`.** Every surface (UI, `/management`, audit REST, debug hub) auto-authenticates with ALL roles — including Operator+Verifier, nullifying the two-person secured-write control. *(Reports 01 & 07; `deploy/wonder-app-vd03/appsettings.Central.json:39-42`)*
**H4 (new). The KPI rollup backfill re-introduces the unbounded single-pass shape at startup.** The one-shot backfill folds the full 90-day retention window in one tracked `ToListAsync` (potentially tens of millions of rows) + one giant `SaveChanges`, re-runs on **every** singleton failover, and blocks periodic folds while it runs. *(Report 04 R1; `KpiHistoryRecorderActor.cs:510-511`, `KpiHistoryRepository.cs:104-167`)*
## Cross-Cutting Themes
## Cross-Cutting Themes (round 2)
### Theme 1 — Failover is designed but unverified, and multiple independent defects hide on that path (stability)
### Theme 1 — The oldest-vs-leader predicate split is still alive, and it just caused the new Critical
PLAN-01 unified "active node" on oldest-member semantics, but the helper was never swapped into `SiteCommunicationActor`/`SiteReplicationActor` (a known partial), and the new resync feature then keyed its authority check on the *leader* predicate — producing C1. One shared `IsActiveNode` predicate, used by every gate, would close this class permanently. *(Reports 01, 02)*
Beyond C1/C2, the failover path accumulates: leader-vs-oldest confusion ("active node" = cluster leader by address order, but singletons live on the *oldest* node — the event-log purge gate runs on the wrong node and Traefik can route to a node hosting no singletons); during a central partition **both** nodes return 200 on `/health/active` so Traefik serves both indefinitely; the standby site node runs the **full S&F delivery pipeline** with no active-node gate, so replicated Pending rows are delivered by both nodes (systematic duplicate external calls and DB writes); singleton drain tasks run `GracefulStop(10s)` inside a 5s CoordinatedShutdown phase; and inbound-API compiled handlers go stale across central failover (the standby serves startup-time delegates with no revision check — an updated method silently serves the old script with HTTP 200).
### Theme 2 — Post-initiative features carry the new risk
The aggregated live alarm stream and KPI rollups shipped after the review baseline with no adversarial review: the rollup backfill (H4), rate-unit discontinuity at the raw/rollup routing boundary (~60× magnitude jump) and a bucketer re-fold error on rate metrics, per-delta full-snapshot copies + per-circuit fan-out with no coalescing in the live alarm cache, sticky `IsLive` grafting frozen snapshots after aggregator death, and a stale-site poll race on the Alarm Summary page. None are Critical individually; together they say **new features need the round-1 treatment before they accumulate.** *(Reports 02, 04, 07)*
**Root cause is shared:** no test anywhere forms a real two-node cluster and kills a node. "Failover" integration tests are single-process simulations; HOCON tests assert strings, not behavior; the risky collaborators (real ClusterClients, the standby node, shared gRPC channels) are mocked away. *(Reports 01, 02, 06)*
### Theme 3 — The script-artifact invalidation bus publishes into the void
The PLAN-05→06 handoff never completed: `BundleImporter` publishes `ScriptArtifactsChanged`, but zero `Subscribe` call sites exist repo-wide — the Host comment claiming the Inbound API consumer "wired in plan 06" is factually wrong. Correctness currently self-heals via the revision-checked handler cache, so this is a Medium not a Critical — but the contract everyone believes exists, doesn't. Wire the consumer or delete the claim. *(Reports 06 N1, 05 N3)*
### Theme 2 — A family of cache-coherence/staleness bugs around compiled scripts (stability)
### Theme 4 — The eager-options-validation convention has a tail
PLAN-08 validated 12 components, but knobs added *by the fix plans themselves* missed the sweep: `TagSubscribeRetryIntervalMs` (0 hot-loops), `StuckScriptGraceMs` (negative throws), `MaxConcurrentImportSessions` (0 permanently blocks imports), 6 AuditLog sub-options, and Host's `NodeOptions` (empty NodeName silently NULLs audit `SourceNode`). One follow-up batch closes it. *(Reports 03 N3, 05 N6, 08 NF4)*
The same defect shape appears at least four times: (a) inbound compiled handlers stale across failover; (b) inbound compiled-handler and known-bad-method caches stale after a bundle import overwrites an `ApiMethod` (old code serves HTTP 200 until restart); (c) `_knownBadMethods` never cleared on out-of-band fixes (known, in project memory); (d) `RevisionHashService` omits `NativeAlarmSources`, so native-alarm edits never flag instances stale and sites mirror old sources indefinitely. Related: every staleness probe / Deployments-page validation compiles every script via Roslyn into the non-collectible AssemblyLoadContext with no caching — CPU spikes plus an unbounded assembly leak. **A single "script artifact changed → invalidate everywhere" invalidation contract would close the whole family.** *(Reports 05, 06)*
### Theme 5 — Security seams the fixes exposed
`LoginThrottle` landed everywhere except the DebugStreamHub bind; no `ForwardedHeaders` handling anywhere means behind Traefik the throttle key collapses to `username|proxy-IP` — anyone can lock out `admin` for all operators with 5 bad passwords; and none of the four secured-write handlers enforces site scope, so a site-scoped Operator/Verifier can submit/approve MxGateway device writes fleet-wide — the same class of gap C2 (round 1) just closed, on a higher-consequence path. *(Report 07 N1–N3)*
### Theme 3 — Transport import is the least-finished shipped feature: silent data loss (underdeveloped)
Beyond C3: imported scripts silently lose `MinTimeBetweenRuns`/`ExecutionTimeoutSeconds` (and `LockedInDerived` isn't in the DTOs at all); template `NativeAlarmSources` aren't transported while their instance-level overrides *are* (dangling overrides silently dropped at flatten); the exporter hardcodes `AreaName: null` while docs and CLAUDE.md claim Area-by-name travels (the importer's resolution machinery is fully built — a half-shipped feature presented as done); the import blocker heuristic hard-blocks valid imports on PascalCase false positives with no operator override. The bundle **crypto** is solid; it's the **apply path** where spec, DTOs, and code drifted. *(Reports 05, 08)*
### Theme 4 — Scale-dependent time bombs in the data layer (performance/stability)
Each is invisible at test scale and arrives on a clock: the spec'd site SQLite 7-day audit purge **does not exist** (site audit DBs grow unbounded); `SwitchOutPartitionAsync` rebuilds the full-table `UX_AuditLog_EventId` unique index in one transaction with the default 30s `CommandTimeout` — once the table is large, the daily purge fails permanently and retention silently stops; SiteCalls KPI predicates full-scan the 365-day table ~15×/minute; `GetExecutionTreeAsync` does a `SELECT DISTINCT` over the entire AuditLog per drilldown; the S&F retry sweep is serial, unbounded, and single-laned (central down + 200-message backlog ≈ 100 min/sweep with head-of-line blocking); site SQLite runs without WAL/busy-timeout under many concurrent writers; no EF `EnableRetryOnFailure` anywhere; the persisted `backlogTotal` KPI trend is hardcoded zero. *(Reports 02, 03, 04)*
### Theme 5 — Security posture is strong in design, leaky in projection (stability/conventions)
The session/JWT design, secured-write CAS race guards, and timing-oracle-free API-key auth are exemplary. But: C4 (`DisableLogin` artifact); DataConnection configs (plaintext OPC UA passwords), ExternalSystem `AuthConfiguration`, and DB connection strings are returned **raw to any authenticated user** via ungated List/Get commands — while SMTP/SMS/API keys are properly elided; `deploy artifacts --site-id` is silently ignored and deploys fleet-wide with no site-scope check; pending secured writes never expire server-side (the "Expired" status exists only in the UI — a stale setpoint can be approved and written to a live device days later); every management/auth request does a full LDAP bind with zero throttling (password-spray oracle); role-casing split-brain between UI policies (case-sensitive) and the ManagementActor (case-insensitive). *(Report 07)*
### Theme 6 — Script containment is cooperative, not enforced (stability)
Script execution timeout is cooperative-only: a script blocked in synchronous I/O permanently consumes one of the 8 dedicated `ScriptExecutionScheduler` threads — 8 hung scripts silently halt all script **and alarm on-trigger** execution site-wide, with zero observability (no stuck-thread health metric). Relatedly: trigger expressions run Roslyn synchronously on the default dispatcher per attribute change; the inbound method timeout abandons the orphaned task while disposing its DI scope (use-after-dispose); site-side compile failures don't reject the deployment — central is still told Success, contradicting the spec. *(Reports 03, 06)*
### Theme 7 — Adapter/channel lifecycle leaks on reconnect (stability)
`OpcUaDataConnection`/`MxGatewayDataConnection` overwrite `_client` on every `ConnectAsync` without dispose/detach — session/socket leak per flap, plus a stale `ConnectionLost` handler that can flap the *new* healthy connection. One debug session's gRPC failover disposes the site's **shared** cached channel, cancelling every other session's subscriptions (concurrent sessions ping-pong and terminate each other). S&F replication is `Task.Run` fire-and-forget with possible Add/Remove reordering. *(Reports 02, 03)*
### Theme 6 — Deferral tracking is fragmenting again
The canonical deferred-work register is substantially accurate (all 10 "Resolved 2026-07-10" claims spot-verified true), but the SBR oldest-crash gap and the wonder-app overlay edits live only in the master tracker, the perf-envelope row's revisit trigger fired unnoticed, an untracked `deferred.md` snapshot has drifted, and the CHANGELOG (last touched 2026-06-02) is now factually wrong. One tracking home, enforced. *(Reports 01 N6, 08 NF3/NF5)*
## What's Genuinely Good (keep doing this)
- **Conventions hold under audit** — UTC everywhere, POCO/repository split, options pattern (where validated), additive contracts, clean reference graph, correlation IDs. (Report 08)
- **Idempotency and atomicity where it matters** — EventId-keyed audit ingest, NotificationId insert-if-not-exists, verified-atomic AuditLog+SiteCalls dual-write, CAS on deployment status and secured-write approval. (Reports 04, 07)
- **Error hygiene at the edges** — consistent transient/permanent classification, poison-message parking, enumeration-safe 403s, `IHttpClientFactory` throughout, bounded audit capture. (Report 06)
- **Hardening passes are visible** — Become/Stash lifecycle machines, redeploy termination-watch buffering, adapter-generation guards, HOCON escaping, DebugView disposal/thread-marshalling. (Reports 01, 03, 07)
- **Deferred work is honestly logged** — 23-item inventory in report 08; only ~4 warrant near-term action.
- **The fix plans held under adversarial re-verification** — 168 findings re-proven fixed from source, zero regressions, zero false claims; several fixes exceeded scope.
- **Structural guards over point fixes**: the reflection round-trip suite (Transport), the frozen 138-command authz matrix + dispatch-coverage pair, contract-lock tests (which surfaced the real Newtonsoft default(T) wire behavior), and the design-invariant DI lock tests.
- **The live alarm cache lifecycle engineering** (ref-counting, linger, version-checked timers, fail-safe viewer cap, self-healing start retry) is exemplary — the defects around it are in the fan-out economics, not the lifecycle.
- **NodeB-failover reconciliation pull clients and the composite keyset cursor** dodge every classic trap (transport-fault-only failover, per-row fault isolation, `DateTime.MinValue` underflow).
- **Conventions survived 268 commits of churn**: UTC discipline, Commons purity, additive contracts, clean reference graph all re-verified.
## Prioritized Recommendations
**P0 — before the next production deployment**
1. Enable the SBR downing provider in `BuildHocon` and prove it with a real two-node kill test (see P2-10). *(C1)*
2. Remove `DisableLogin: true` from `deploy/wonder-app-vd03` (or add an environment guard that refuses it outside dev). *(C4)*
3. Fix the ClusterClient stop/recreate name collision (await termination or use generation-suffixed names). *(C2)*
4. Gate the S&F delivery pipeline (and event-log purge gate) on singleton-host/active-node status — standby must be passive. *(Theme 1)*
**P0 — immediately**
1. Rotate the exposed Inbound API key and delete `test.txt`. *(H2)*
2. Fix the resync predicate inversion — one shared oldest-member active-node predicate for delivery gate AND resync authority. *(C1)*
3. Chunk the anti-entropy snapshot (or page it) so it fits Akka remoting frames; add a delivery-confirmation check. *(H1)*
**P1 — correctness of shipped features**
5. Fix the Transport import apply path: consume `BaseTemplateName`, carry the dropped script fields + `LockedInDerived`, transport template `NativeAlarmSources`, wire `AreaName`. Add round-trip export→import equivalence tests per entity type. *(C3, Theme 3)*
6. Define one script-artifact invalidation contract: bundle import, failover activation, and out-of-band updates must all invalidate compiled-handler/known-bad caches; add `NativeAlarmSources` to `RevisionHashService`; cache validation compiles (collectible ALC). *(Theme 2)*
7. Close the secrets-projection gap: elide DataConnection/ExternalSystem/DB-connection secrets in List/Get like SMTP/SMS already are; honor `SiteId` in `HandleDeployArtifacts` with a site-scope check; add server-side secured-write expiry. *(Theme 5)*
8. Make site-side compile failure reject the deployment (per spec) instead of reporting Success. *(Theme 6)*
**P1 — before relying on the affected features**
4. Bound the rollup backfill (time-sliced like the purges, `AsNoTracking`, skip-on-failover-if-current) and fix the rate-metric unit discontinuity at the raw/rollup boundary. *(H4, Report 04 R2/R3)*
5. Make the failover drill honest: kill the *younger* node in the scripted drill (document the oldest-crash gap), fix seed-list ordering or document the constraint, then **actually run it** and measure the ~25s envelope with the now-landed two-node rig. *(H3)*
6. Enforce site scope in the four secured-write handlers; throttle the DebugStreamHub bind; add `ForwardedHeaders` handling so throttle keying survives Traefik. *(Theme 5)*
7. Wire the Inbound API subscriber to `IScriptArtifactChangeBus` (or remove the claim from Host and the tracker). *(Theme 3)*
**P2 — before production data volume / as hardening**
9. Data-layer time bombs: implement the site SQLite audit purge; set an explicit long `CommandTimeout` on partition switch; add the missing SiteCalls KPI index; bound the execution-tree CTE; turn on WAL + busy_timeout for site SQLite; `EnableRetryOnFailure`; fix the hardcoded-zero backlog KPI. *(Theme 4)*
10. Build a real two-node failover test rig (docker already provides the topology): kill -9 a node, assert singleton migration, S&F non-duplication, Traefik single-active, inbound handler freshness. This one rig would have caught C1, C2, and three Theme-1 findings. *(Theme 1)*
11. Script containment: watchdog metric for stuck `ScriptExecutionScheduler` threads (health-surface it), move trigger-expression evaluation off the default dispatcher, fix the inbound DI-scope use-after-dispose. *(Theme 6)*
12. Adapter lifecycle: dispose/detach old OPC UA/MxGateway clients on reconnect; per-session (or ref-counted) gRPC channel handling for debug streams; make S&F replication ordered or reconciled. *(Theme 7)*
13. Housekeeping: add CLI.Tests to the slnx (one line, long overdue); extend `ValidateOnStart` options validation to the ~14 unvalidated components; batch/parallelize the S&F retry sweep with a per-sweep LIMIT; fix the docs drift catalogued in reports 04 and 08.
**P2 — hardening batch**
8. Close the options-validation tail (5 spots, Theme 4); guard the MxGateway reconnect generic-catch with `ct.IsCancellationRequested` (Report 03 N1); add failure mapping to the expression-eval `PipeTo`s (03 N2); batch the SiteCalls terminal purge (04 R6); cache Expression-trigger compiles on read-only validation paths (05 N1); coalesce live-alarm deltas (02 N6); fix the Alarm Summary stale-site poll race (07 N4).
9. Consolidate deferral tracking into the canonical register; refresh or delete the CHANGELOG; sweep the doc drift each report catalogues (stale "leader" comments, "off-thread" compile-gate claim, "throttling complete" claim, Host bus comment).
## Reading Order
If you read only three reports: **01** (the failover crack), **05** (Transport data loss), **04** (the data-layer clocks). Report **08** is the best single picture of overall codebase health and carries the consolidated 23-item deferred-work inventory.
If you read only three reports this round: **02** (the new Critical, plus the live-alarm stream review), **01** (what the failover story still owes), **04** (the rollup backfill). Report **08** again carries the best overall-health picture, the refreshed test inventory (~30 projects, ~6,300 executed cases), and the deferred-register audit.