dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 1 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

docs(archreview): round-2 re-review (2026-07-12) + 8 fix plans (86 tasks)

Browse Source 
Re-ran all 8 domain reviews at HEAD 8c888f13 against the b910f5eb baseline:
every round-1 finding source-verified (168 fixed, 0 regressions, 0 false
claims); 56 new findings (1 Critical / 4 High / 15 Medium / 36 Low),
concentrated in post-baseline code (anti-entropy resync, KPI rollup
backfill, live alarm stream) and seams the fixes exposed.

Headliners: S&F resync predicate inversion can wipe the delivering node's
buffer (02-N1 Critical); resync snapshot exceeds the Akka remoting frame
size (02-N2); failover drill kills the one node keep-oldest can't survive
(01-N1); unbounded rollup backfill per failover (04-R1); live production
API key in untracked test.txt (08-NF1).

Adds PLAN-R2-01..08 + .tasks.json manifests and the Round-2 board,
P0 list, cross-plan mutexes, and wave order in 00-MASTER-TRACKER.
This commit is contained in:
Joseph Doherty
2026-07-12 23:52:10 -04:00
parent 8c888f13a0
commit 5bbd7689fa
26 changed files with 7236 additions and 1321 deletions
Download Patch File Download Diff File Expand all files Collapse all files
archreview/00-OVERALL.md
+84 -84
View File
@@ -1,122 +1,122 @@
# ScadaBridge Deep Architecture Review — Overall Report
# ScadaBridge Deep Architecture Review — Overall Report (Round 2)
**Date:** 2026-07-08
**Scope:** Full system — all 27 components, docker/deploy topology, tests, and design docs
**Method:** 8 parallel domain reviews, each reading the component design docs first and then the actual source (findings are grounded in `file:line` citations in the domain reports). Dimensions: **stability, performance, conventions, underdeveloped areas.**
**Date:** 2026-07-12 (round 2; round 1 dated 2026-07-08)
**Scope:** Full system — all 27 components, docker/deploy topology, tests, and design docs, at HEAD `8c888f13`
**Method:** 8 parallel domain re-reviews. Each re-read its round-1 report and fix plan (PLAN-01…08, all complete — 191/192 tasks merged by 2026-07-10), then **verified every round-1 finding's disposition against current source** (trusting code, not plan claims), then swept the ~268 commits since the pre-initiative baseline `b910f5eb` — the fix-plan code itself plus the two post-initiative features nobody had reviewed (aggregated live alarm stream, KPI hourly rollups) — for **new** findings. Dimensions: stability, performance, conventions, underdeveloped areas; all findings carry `file:line` citations in the domain reports.
## Report Index
| # | Report | Domain | Verdict (one line) |
|---|--------|--------|--------------------|
| 01 | [Cluster, Host & Failover](01-cluster-host-failover.md) | ClusterInfrastructure, Host, HealthMonitoring, docker/Traefik/deploy | High micro-level polish undone by one critical foundation crack: the split-brain resolver is configured but never enabled — automatic failover does not work for hard crashes. |
| 02 | [Communication & Store-and-Forward](02-communication-store-and-forward.md) | ClusterClient, gRPC streaming, S&F engine | High code quality in-the-small; mid maturity as a distributed system — serious lifecycle/topology defects the test suite is structurally blind to. |
| 03 | [Site Runtime & DCL](03-site-runtime-dcl.md) | SiteRuntime, DataConnectionLayer, SiteEventLogging | Mature and unusually carefully engineered; residual risk in adapter reconnect lifecycle, script containment, and deployment honesty. |
| 04 | [Data & Audit Backbone](04-data-audit-backbone.md) | ConfigurationDatabase, AuditLog, NotificationOutbox, SiteCallAudit, KpiHistory | Genuinely mature data layer with scale-dependent operational time bombs that will surface within months of production volume. |
| 05 | [Templates, Deployment & Transport](05-templates-deployment-transport.md) | TemplateEngine, DeploymentManager, Transport, ScriptAnalysis | Failure-path engineering is production-grade, but the Transport import apply path and newest template features contain multiple silent-data-loss and staleness defects. |
| 06 | [Edge Integrations](06-edge-integrations.md) | InboundAPI, ExternalSystemGateway, NotificationService, DelmiaNotifier | Mature, heavily-hardened edge layer; risk concentrated in one silent cache-coherence bug and acknowledged ESG spec gaps. |
| 07 | [UI, Management & Security](07-ui-management-security.md) | CentralUI, ManagementService, CLI, Security | Mature, carefully engineered surface undermined by sharp asymmetries: an auth-disabled deploy artifact, ignored deploy scoping, inconsistent secret elision. |
| 08 | [Conventions, Tests & Underdeveloped Areas](08-conventions-tests-underdeveloped.md) | Cross-cutting | Unusually disciplined codebase — the documented conventions are actually followed; gaps are narrow, specific, and mostly consciously logged. |
| # | Report | Domain | Round-2 verdict (one line) |
|---|--------|--------|----------------------------|
| 01 | [Cluster, Host & Failover](01-cluster-host-failover.md) | ClusterInfrastructure, Host, HealthMonitoring, docker/Traefik/deploy | Round 1's Critical (SBR never enabled) fixed and behaviorally proven; active-node model coherently unified — but the never-run failover drill kills the one node keep-oldest can't survive, and seed-order blocks the recovery loop when central-b is the survivor. |
| 02 | [Communication & Store-and-Forward](02-communication-store-and-forward.md) | ClusterClient, gRPC streaming, S&F engine, live alarm stream | Round 1's Critical and all three Highs genuinely fixed and test-backed; the new live alarm stream is solid — but the new anti-entropy peer resync was built on the wrong active-node predicate and an undeliverable-size Akka message: 1 new Critical + 1 new High, both confined to that feature. |
| 03 | [Site Runtime & DCL](03-site-runtime-dcl.md) | SiteRuntime, DataConnectionLayer, SiteEventLogging | All three round-1 risk concentrations (adapter reconnect, script containment, deployment honesty) genuinely fixed — 26/28 verified; one Medium residual on the MxGateway reconnect fault path. |
| 04 | [Data & Audit Backbone](04-data-audit-backbone.md) | ConfigurationDatabase, AuditLog, NotificationOutbox, SiteCallAudit, KpiHistory | All round-1 time bombs genuinely defused and verified; the one new material risk is the brand-new KPI rollup **backfill**, which re-introduces at startup the exact unbounded year-scale single-pass shape the plan eliminated everywhere else. |
| 05 | [Templates, Deployment & Transport](05-templates-deployment-transport.md) | TemplateEngine, DeploymentManager, Transport, ScriptAnalysis | All 26 actionable round-1 findings — including the Critical inheritance-edge loss and the whole silent-data-loss family — fixed in code with zero regressions; residue is 1 Medium perf leftover and 5 Lows. |
| 06 | [Edge Integrations](06-edge-integrations.md) | InboundAPI, ExternalSystemGateway, NotificationService, DelmiaNotifier | All round-1 risk genuinely fixed and source-verified (20/25 fixed, 0 regressions); new residual risk is 1 Medium + 4 Low — the Medium being the script-artifact invalidation bus publishing to zero subscribers. |
| 07 | [UI, Management & Security](07-ui-management-security.md) | CentralUI, ManagementService, CLI, Security | Both round-1 Criticals dead, 17/24 fully fixed, 0 regressions; remaining new findings are Medium-and-below — mostly security seams the fixes themselves exposed (proxy-blind throttle keying, unthrottled hub bind, unscoped secured writes). |
| 08 | [Conventions, Tests & Underdeveloped](08-conventions-tests-underdeveloped.md) | Cross-cutting | Every actionable round-1 finding genuinely fixed and conventions held through 268 commits of churn; new residue is small and mostly hygiene — except a live production API key sitting in an untracked working-tree file. |
## Severity Tally (tag occurrences per report)
## Round-1 Finding Disposition (verified against source, not plan claims)
| Report | Tracked | Fixed (verified) | Partial | Deferred/accepted | Not fixed | Regressed |
|--------|--------:|-----------------:|--------:|------------------:|----------:|----------:|
| 01 Cluster/Host/Failover | 28 | 18 | 3 | 6 | 1¹ | 0 |
| 02 Communication/S&F | 26 | 22 | 2 | 2 | 0 | 0 |
| 03 Site Runtime/DCL | 28 | 26 | 0 | 2 | 0 | 0 |
| 04 Data/Audit Backbone | 26 | 19 | 0 | 7² | 0 | 0 |
| 05 Templates/Deploy/Transport | 32 | 26 | 1 | 5 | 0 | 0 |
| 06 Edge Integrations | 25 | 20 | 2 | 3 | 0 | 0 |
| 07 UI/Management/Security | 24 | 17 | 6 | 1³ | 0 | 0 |
| 08 Conventions/Tests | 30 | 20 | 3 | 7 | 0 | 0 |
| **Total** | **~219** | **168** | **17** | **~33** | **1** | **0** |
¹ U5 (NodeName in the wonder-app-vd03 overlay) — deferred out-of-band with an owner, but see new finding 01-N2.
² Includes 1 won't-fix (S10). ³ UA7 was a false premise (No longer applicable).
**Headline: zero regressions, zero false "fixed" claims found.** Every disposition was re-derived from current source with `file:line` evidence. Several fixes exceeded their tickets (content-based inbound handler invalidation, two-signal deploy join, trigger-body trust coverage); PLAN-05's reflection round-trip guard structurally forecloses the entire DTO-drift class that caused round 1's Transport Critical.
## New-Findings Severity Tally (round 2 only)
| Report | Critical | High | Medium | Low |
|--------|---------:|-----:|-------:|----:|
| 01 Cluster/Host/Failover | 1 | 5 | 13 | 9 |
| 02 Communication/S&F | 2 | 6 | 11 | 11 |
| 03 Site Runtime/DCL | 0 | 4 | 6 | 11 |
| 04 Data/Audit Backbone | 0 | 6 | 11 | 6 |
| 05 Templates/Deploy/Transport | 2 | 13 | 15 | 5 |
| 06 Edge Integrations | 0 | 4 | 11 | 11 |
| 07 UI/Management/Security | 2 | 8 | 12 | 5 |
| 08 Conventions/Tests | 0 | 4 | 6 | 10 |
| **Total (approx.)** | **7** | **50** | **85** | **68** |
| 01 Cluster/Host/Failover | 0 | 1 | 1 | 4 |
| 02 Communication/S&F | 1 | 1 | 2 | 5 |
| 03 Site Runtime/DCL | 0 | 0 | 1 | 5 |
| 04 Data/Audit Backbone | 0 | 1 | 3 | 3 |
| 05 Templates/Deploy/Transport | 0 | 0 | 1 | 5 |
| 06 Edge Integrations | 0 | 0 | 1 | 4 |
| 07 UI/Management/Security | 0 | 0 | 4 | 5 |
| 08 Conventions/Tests | 0 | 1 | 2 | 5 |
| **Total** | **1** | **4** | **15** | **36** |
*(Counts are raw tag occurrences; a few findings are cited in more than one report — e.g. the `DisableLogin` deploy artifact appears in both 01 and 07.)*
Round 1 closed with 7 Critical / 50 High / 85 Medium / 68 Low. Round 2 opens with 1 / 4 / 15 / 36 — an ~80% reduction at every severity, and the residue is concentrated almost entirely in **code written after round 1**, not in anything round 1 flagged.
## Overall Verdict
ScadaBridge is an unusually disciplined codebase at the micro level. The conventions documented in CLAUDE.md are **actually followed** — verified, not assumed: zero `DateTime.Now` in `src/`, persistence-ignorant Commons entities, a clean project-reference graph with no site→central contamination, correlation IDs on request/response messages, additive message contracts, idempotent ingest keyed on `EventId`/`NotificationId`, timing-oracle-free API-key auth, and only two real TODOs in ~190k LOC. Test volume is real (~5,050 behavioral tests across 30 projects — Akka TestKit, bUnit, Playwright, MSSQL-backed integration tests).
The arch-review initiative worked: the fix plans delivered what they claimed, verified in source, with zero regressions. The system's round-1 risk profile — the failover crack, the Transport silent data loss, the data-layer time bombs, the cache-coherence family, the security projection leaks — is genuinely closed.
The system's risk is **not** diffuse sloppiness. It concentrates in exactly the four places a single-process test suite cannot see:
The round-2 risk profile is different in kind: **the new findings live almost exclusively in the newest code** — the anti-entropy resync (built by PLAN-02 itself), the KPI rollup backfill and the live alarm stream (both shipped post-initiative, 2026-07-10), and operational seams the fixes exposed. This is the expected shape for a healthy hardening cycle: the reviewed-and-fixed code held; the code that had never been reviewed is where the defects are.
1. **Real multi-node failover** — the paths that only execute when a node hard-crashes, a partition forms, or the standby takes over.
2. **Cache coherence across process/node boundaries** — compiled-script and staleness caches that are only invalidated on the node that made the change.
3. **Cross-environment data movement** — the Transport bundle import apply path, where DTOs, spec, and apply code drifted apart.
4. **Production-scale data volume** — purge/retention/KPI paths tested only at small scale.
**Bottom line: the codebase graduated from "production-grade engineering, not production-proven HA" to "one new feature (peer resync) must not ship as-is, one secret must be rotated today, and the failover drill still needs to actually be run."**
**Bottom line: production-grade engineering, not yet production-proven HA.** The single most important sentence in this review: *automatic failover for hard crashes does not currently work* (finding C1 below), and no test in the repo would have caught it.
## The Findings That Matter Most (fix before anything else)
## The Critical Findings (fix before anything else)
**C1 (new). The S&F anti-entropy resync can wipe the delivering node's live buffer.** The delivery gate uses oldest-Up (`SelfIsPrimary`) but the resync authority check uses leader+Up (lowest address). After a routine rolling restart of the lower-address site node the two predicates diverge persistently: the *delivering* node requests a resync from the stale rejoined leader and `ReplaceAllAsync`-wipes its own live buffer — silently losing every notification/cached call buffered during the outage and resurrecting delivered rows as duplicates. *(Report 02 N1; `SiteReplicationActor.cs:190-198` vs `AkkaHostedService.cs:881`)*
**C1. The split-brain resolver is configured but never enabled.** `BuildHocon` never sets `akka.cluster.downing-provider-class`, so the keep-oldest SBR settings are inert: unreachable nodes are never downed, and cluster singletons (site DeploymentManager = all data collection; all six central singletons) never migrate on a hard crash or partition. The documented ~25s failover only works for graceful stops. The repo's own Akka reference notes (`AkkaDotNet/03-Cluster.md`) show the required line. *(Report 01; `Host/Actors/AkkaHostedService.cs:225-270`)*
**H1 (new). The resync snapshot is undeliverable at exactly the scale it exists for.** Up to 10,000 full rows travel as ONE Akka remoting message with no `maximum-frame-size` override (default 128 KB) — silently dropped for any realistic backlog. *(Report 02 N2; `SiteReplicationActor.cs:40,407-410,460`)*
**C2. Editing a site's addresses crash-loops central routing to all sites.** `CentralCommunicationActor` stops and immediately recreates a *named top-level* ClusterClient in the same message handling; `Context.Stop` is async, so the name collides and the actor crash-restarts into a permanent loop. Tests mock the client factory, so this is never exercised. *(Report 02; `CentralCommunicationActor.cs:514-521`)*
**H2 (new). A live production Inbound API key sits in untracked `test.txt` at the repo root** (`sbk_eb5acc…` for `wonder-app-vd03.zmr.zimmer.com:8085`) — one `git add -A` from history. Rotate the key and delete the file **today**. *(Report 08 NF1; `test.txt:3`)*
**C3. Transport bundle import silently drops template inheritance.** `TemplateDto.BaseTemplateName` is exported and diffed but never consumed at apply — derived templates land as root templates and flatten to amputated configs in the target environment. *(Report 05; `Transport/Import/BundleImporter.cs:1524`)*
**H3 (new). The failover drill codifies a recovery the topology cannot deliver.** `failover-drill.sh` kills the ACTIVE(=oldest) central node — the one crash keep-oldest cannot survive (registered deferred decision) — and the restart recovery loop cannot converge when central-b is the survivor because both nodes list central-a first in `SeedNodes` (Akka's first-seed rule blocks a non-first seed from self-bootstrapping). The README admits the drill was never run live yet promises ~25s failover; the ~25s envelope also remains the last unmeasured headline design claim (the perf harness's skip precondition — the two-node rig — landed 2026-07-08). *(Reports 01 N1, 08 NF2)*
**C4. The on-host production deploy artifact ships `DisableLogin: true`.** Every surface (UI, `/management`, audit REST, debug hub) auto-authenticates with ALL roles — including Operator+Verifier, nullifying the two-person secured-write control. *(Reports 01 & 07; `deploy/wonder-app-vd03/appsettings.Central.json:39-42`)*
**H4 (new). The KPI rollup backfill re-introduces the unbounded single-pass shape at startup.** The one-shot backfill folds the full 90-day retention window in one tracked `ToListAsync` (potentially tens of millions of rows) + one giant `SaveChanges`, re-runs on **every** singleton failover, and blocks periodic folds while it runs. *(Report 04 R1; `KpiHistoryRecorderActor.cs:510-511`, `KpiHistoryRepository.cs:104-167`)*
## Cross-Cutting Themes
## Cross-Cutting Themes (round 2)
### Theme 1 — Failover is designed but unverified, and multiple independent defects hide on that path (stability)
### Theme 1 — The oldest-vs-leader predicate split is still alive, and it just caused the new Critical
PLAN-01 unified "active node" on oldest-member semantics, but the helper was never swapped into `SiteCommunicationActor`/`SiteReplicationActor` (a known partial), and the new resync feature then keyed its authority check on the *leader* predicate — producing C1. One shared `IsActiveNode` predicate, used by every gate, would close this class permanently. *(Reports 01, 02)*
Beyond C1/C2, the failover path accumulates: leader-vs-oldest confusion ("active node" = cluster leader by address order, but singletons live on the *oldest* node — the event-log purge gate runs on the wrong node and Traefik can route to a node hosting no singletons); during a central partition **both** nodes return 200 on `/health/active` so Traefik serves both indefinitely; the standby site node runs the **full S&F delivery pipeline** with no active-node gate, so replicated Pending rows are delivered by both nodes (systematic duplicate external calls and DB writes); singleton drain tasks run `GracefulStop(10s)` inside a 5s CoordinatedShutdown phase; and inbound-API compiled handlers go stale across central failover (the standby serves startup-time delegates with no revision check — an updated method silently serves the old script with HTTP 200).
### Theme 2 — Post-initiative features carry the new risk
The aggregated live alarm stream and KPI rollups shipped after the review baseline with no adversarial review: the rollup backfill (H4), rate-unit discontinuity at the raw/rollup routing boundary (~60× magnitude jump) and a bucketer re-fold error on rate metrics, per-delta full-snapshot copies + per-circuit fan-out with no coalescing in the live alarm cache, sticky `IsLive` grafting frozen snapshots after aggregator death, and a stale-site poll race on the Alarm Summary page. None are Critical individually; together they say **new features need the round-1 treatment before they accumulate.** *(Reports 02, 04, 07)*
**Root cause is shared:** no test anywhere forms a real two-node cluster and kills a node. "Failover" integration tests are single-process simulations; HOCON tests assert strings, not behavior; the risky collaborators (real ClusterClients, the standby node, shared gRPC channels) are mocked away. *(Reports 01, 02, 06)*
### Theme 3 — The script-artifact invalidation bus publishes into the void
The PLAN-05→06 handoff never completed: `BundleImporter` publishes `ScriptArtifactsChanged`, but zero `Subscribe` call sites exist repo-wide — the Host comment claiming the Inbound API consumer "wired in plan 06" is factually wrong. Correctness currently self-heals via the revision-checked handler cache, so this is a Medium not a Critical — but the contract everyone believes exists, doesn't. Wire the consumer or delete the claim. *(Reports 06 N1, 05 N3)*
### Theme 2 — A family of cache-coherence/staleness bugs around compiled scripts (stability)
### Theme 4 — The eager-options-validation convention has a tail
PLAN-08 validated 12 components, but knobs added *by the fix plans themselves* missed the sweep: `TagSubscribeRetryIntervalMs` (0 hot-loops), `StuckScriptGraceMs` (negative throws), `MaxConcurrentImportSessions` (0 permanently blocks imports), 6 AuditLog sub-options, and Host's `NodeOptions` (empty NodeName silently NULLs audit `SourceNode`). One follow-up batch closes it. *(Reports 03 N3, 05 N6, 08 NF4)*
The same defect shape appears at least four times: (a) inbound compiled handlers stale across failover; (b) inbound compiled-handler and known-bad-method caches stale after a bundle import overwrites an `ApiMethod` (old code serves HTTP 200 until restart); (c) `_knownBadMethods` never cleared on out-of-band fixes (known, in project memory); (d) `RevisionHashService` omits `NativeAlarmSources`, so native-alarm edits never flag instances stale and sites mirror old sources indefinitely. Related: every staleness probe / Deployments-page validation compiles every script via Roslyn into the non-collectible AssemblyLoadContext with no caching — CPU spikes plus an unbounded assembly leak. **A single "script artifact changed → invalidate everywhere" invalidation contract would close the whole family.** *(Reports 05, 06)*
### Theme 5 — Security seams the fixes exposed
`LoginThrottle` landed everywhere except the DebugStreamHub bind; no `ForwardedHeaders` handling anywhere means behind Traefik the throttle key collapses to `username|proxy-IP` — anyone can lock out `admin` for all operators with 5 bad passwords; and none of the four secured-write handlers enforces site scope, so a site-scoped Operator/Verifier can submit/approve MxGateway device writes fleet-wide — the same class of gap C2 (round 1) just closed, on a higher-consequence path. *(Report 07 N1–N3)*
### Theme 3 — Transport import is the least-finished shipped feature: silent data loss (underdeveloped)
Beyond C3: imported scripts silently lose `MinTimeBetweenRuns`/`ExecutionTimeoutSeconds` (and `LockedInDerived` isn't in the DTOs at all); template `NativeAlarmSources` aren't transported while their instance-level overrides *are* (dangling overrides silently dropped at flatten); the exporter hardcodes `AreaName: null` while docs and CLAUDE.md claim Area-by-name travels (the importer's resolution machinery is fully built — a half-shipped feature presented as done); the import blocker heuristic hard-blocks valid imports on PascalCase false positives with no operator override. The bundle **crypto** is solid; it's the **apply path** where spec, DTOs, and code drifted. *(Reports 05, 08)*
### Theme 4 — Scale-dependent time bombs in the data layer (performance/stability)
Each is invisible at test scale and arrives on a clock: the spec'd site SQLite 7-day audit purge **does not exist** (site audit DBs grow unbounded); `SwitchOutPartitionAsync` rebuilds the full-table `UX_AuditLog_EventId` unique index in one transaction with the default 30s `CommandTimeout` — once the table is large, the daily purge fails permanently and retention silently stops; SiteCalls KPI predicates full-scan the 365-day table ~15×/minute; `GetExecutionTreeAsync` does a `SELECT DISTINCT` over the entire AuditLog per drilldown; the S&F retry sweep is serial, unbounded, and single-laned (central down + 200-message backlog ≈ 100 min/sweep with head-of-line blocking); site SQLite runs without WAL/busy-timeout under many concurrent writers; no EF `EnableRetryOnFailure` anywhere; the persisted `backlogTotal` KPI trend is hardcoded zero. *(Reports 02, 03, 04)*
### Theme 5 — Security posture is strong in design, leaky in projection (stability/conventions)
The session/JWT design, secured-write CAS race guards, and timing-oracle-free API-key auth are exemplary. But: C4 (`DisableLogin` artifact); DataConnection configs (plaintext OPC UA passwords), ExternalSystem `AuthConfiguration`, and DB connection strings are returned **raw to any authenticated user** via ungated List/Get commands — while SMTP/SMS/API keys are properly elided; `deploy artifacts --site-id` is silently ignored and deploys fleet-wide with no site-scope check; pending secured writes never expire server-side (the "Expired" status exists only in the UI — a stale setpoint can be approved and written to a live device days later); every management/auth request does a full LDAP bind with zero throttling (password-spray oracle); role-casing split-brain between UI policies (case-sensitive) and the ManagementActor (case-insensitive). *(Report 07)*
### Theme 6 — Script containment is cooperative, not enforced (stability)
Script execution timeout is cooperative-only: a script blocked in synchronous I/O permanently consumes one of the 8 dedicated `ScriptExecutionScheduler` threads — 8 hung scripts silently halt all script **and alarm on-trigger** execution site-wide, with zero observability (no stuck-thread health metric). Relatedly: trigger expressions run Roslyn synchronously on the default dispatcher per attribute change; the inbound method timeout abandons the orphaned task while disposing its DI scope (use-after-dispose); site-side compile failures don't reject the deployment — central is still told Success, contradicting the spec. *(Reports 03, 06)*
### Theme 7 — Adapter/channel lifecycle leaks on reconnect (stability)
`OpcUaDataConnection`/`MxGatewayDataConnection` overwrite `_client` on every `ConnectAsync` without dispose/detach — session/socket leak per flap, plus a stale `ConnectionLost` handler that can flap the *new* healthy connection. One debug session's gRPC failover disposes the site's **shared** cached channel, cancelling every other session's subscriptions (concurrent sessions ping-pong and terminate each other). S&F replication is `Task.Run` fire-and-forget with possible Add/Remove reordering. *(Reports 02, 03)*
### Theme 6 — Deferral tracking is fragmenting again
The canonical deferred-work register is substantially accurate (all 10 "Resolved 2026-07-10" claims spot-verified true), but the SBR oldest-crash gap and the wonder-app overlay edits live only in the master tracker, the perf-envelope row's revisit trigger fired unnoticed, an untracked `deferred.md` snapshot has drifted, and the CHANGELOG (last touched 2026-06-02) is now factually wrong. One tracking home, enforced. *(Reports 01 N6, 08 NF3/NF5)*
## What's Genuinely Good (keep doing this)
- **Conventions hold under audit** — UTC everywhere, POCO/repository split, options pattern (where validated), additive contracts, clean reference graph, correlation IDs. (Report 08)
- **Idempotency and atomicity where it matters** — EventId-keyed audit ingest, NotificationId insert-if-not-exists, verified-atomic AuditLog+SiteCalls dual-write, CAS on deployment status and secured-write approval. (Reports 04, 07)
- **Error hygiene at the edges** — consistent transient/permanent classification, poison-message parking, enumeration-safe 403s, `IHttpClientFactory` throughout, bounded audit capture. (Report 06)
- **Hardening passes are visible** — Become/Stash lifecycle machines, redeploy termination-watch buffering, adapter-generation guards, HOCON escaping, DebugView disposal/thread-marshalling. (Reports 01, 03, 07)
- **Deferred work is honestly logged** — 23-item inventory in report 08; only ~4 warrant near-term action.
- **The fix plans held under adversarial re-verification** — 168 findings re-proven fixed from source, zero regressions, zero false claims; several fixes exceeded scope.
- **Structural guards over point fixes**: the reflection round-trip suite (Transport), the frozen 138-command authz matrix + dispatch-coverage pair, contract-lock tests (which surfaced the real Newtonsoft default(T) wire behavior), and the design-invariant DI lock tests.
- **The live alarm cache lifecycle engineering** (ref-counting, linger, version-checked timers, fail-safe viewer cap, self-healing start retry) is exemplary — the defects around it are in the fan-out economics, not the lifecycle.
- **NodeB-failover reconciliation pull clients and the composite keyset cursor** dodge every classic trap (transport-fault-only failover, per-row fault isolation, `DateTime.MinValue` underflow).
- **Conventions survived 268 commits of churn**: UTC discipline, Commons purity, additive contracts, clean reference graph all re-verified.
## Prioritized Recommendations
**P0 — before the next production deployment**
1. Enable the SBR downing provider in `BuildHocon` and prove it with a real two-node kill test (see P2-10). *(C1)*
2. Remove `DisableLogin: true` from `deploy/wonder-app-vd03` (or add an environment guard that refuses it outside dev). *(C4)*
3. Fix the ClusterClient stop/recreate name collision (await termination or use generation-suffixed names). *(C2)*
4. Gate the S&F delivery pipeline (and event-log purge gate) on singleton-host/active-node status — standby must be passive. *(Theme 1)*
**P0 — immediately**
1. Rotate the exposed Inbound API key and delete `test.txt`. *(H2)*
2. Fix the resync predicate inversion — one shared oldest-member active-node predicate for delivery gate AND resync authority. *(C1)*
3. Chunk the anti-entropy snapshot (or page it) so it fits Akka remoting frames; add a delivery-confirmation check. *(H1)*
**P1 — correctness of shipped features**
5. Fix the Transport import apply path: consume `BaseTemplateName`, carry the dropped script fields + `LockedInDerived`, transport template `NativeAlarmSources`, wire `AreaName`. Add round-trip export→import equivalence tests per entity type. *(C3, Theme 3)*
6. Define one script-artifact invalidation contract: bundle import, failover activation, and out-of-band updates must all invalidate compiled-handler/known-bad caches; add `NativeAlarmSources` to `RevisionHashService`; cache validation compiles (collectible ALC). *(Theme 2)*
7. Close the secrets-projection gap: elide DataConnection/ExternalSystem/DB-connection secrets in List/Get like SMTP/SMS already are; honor `SiteId` in `HandleDeployArtifacts` with a site-scope check; add server-side secured-write expiry. *(Theme 5)*
8. Make site-side compile failure reject the deployment (per spec) instead of reporting Success. *(Theme 6)*
**P1 — before relying on the affected features**
4. Bound the rollup backfill (time-sliced like the purges, `AsNoTracking`, skip-on-failover-if-current) and fix the rate-metric unit discontinuity at the raw/rollup boundary. *(H4, Report 04 R2/R3)*
5. Make the failover drill honest: kill the *younger* node in the scripted drill (document the oldest-crash gap), fix seed-list ordering or document the constraint, then **actually run it** and measure the ~25s envelope with the now-landed two-node rig. *(H3)*
6. Enforce site scope in the four secured-write handlers; throttle the DebugStreamHub bind; add `ForwardedHeaders` handling so throttle keying survives Traefik. *(Theme 5)*
7. Wire the Inbound API subscriber to `IScriptArtifactChangeBus` (or remove the claim from Host and the tracker). *(Theme 3)*
**P2 — before production data volume / as hardening**
9. Data-layer time bombs: implement the site SQLite audit purge; set an explicit long `CommandTimeout` on partition switch; add the missing SiteCalls KPI index; bound the execution-tree CTE; turn on WAL + busy_timeout for site SQLite; `EnableRetryOnFailure`; fix the hardcoded-zero backlog KPI. *(Theme 4)*
10. Build a real two-node failover test rig (docker already provides the topology): kill -9 a node, assert singleton migration, S&F non-duplication, Traefik single-active, inbound handler freshness. This one rig would have caught C1, C2, and three Theme-1 findings. *(Theme 1)*
11. Script containment: watchdog metric for stuck `ScriptExecutionScheduler` threads (health-surface it), move trigger-expression evaluation off the default dispatcher, fix the inbound DI-scope use-after-dispose. *(Theme 6)*
12. Adapter lifecycle: dispose/detach old OPC UA/MxGateway clients on reconnect; per-session (or ref-counted) gRPC channel handling for debug streams; make S&F replication ordered or reconciled. *(Theme 7)*
13. Housekeeping: add CLI.Tests to the slnx (one line, long overdue); extend `ValidateOnStart` options validation to the ~14 unvalidated components; batch/parallelize the S&F retry sweep with a per-sweep LIMIT; fix the docs drift catalogued in reports 04 and 08.
**P2 — hardening batch**
8. Close the options-validation tail (5 spots, Theme 4); guard the MxGateway reconnect generic-catch with `ct.IsCancellationRequested` (Report 03 N1); add failure mapping to the expression-eval `PipeTo`s (03 N2); batch the SiteCalls terminal purge (04 R6); cache Expression-trigger compiles on read-only validation paths (05 N1); coalesce live-alarm deltas (02 N6); fix the Alarm Summary stale-site poll race (07 N4).
9. Consolidate deferral tracking into the canonical register; refresh or delete the CHANGELOG; sweep the doc drift each report catalogues (stale "leader" comments, "off-thread" compile-gate claim, "throttling complete" claim, Host bus comment).
## Reading Order
If you read only three reports: **01** (the failover crack), **05** (Transport data loss), **04** (the data-layer clocks). Report **08** is the best single picture of overall codebase health and carries the consolidated 23-item deferred-work inventory.
If you read only three reports this round: **02** (the new Critical, plus the live-alarm stream review), **01** (what the failover story still owes), **04** (the rollup backfill). Report **08** again carries the best overall-health picture, the refreshed test inventory (~30 projects, ~6,300 executed cases), and the deferred-register audit.
archreview/01-cluster-host-failover.md
+83 -356
View File
@@ -1,391 +1,118 @@
# Architecture Review 01 — Cluster Infrastructure, Host, Failover, Health Monitoring, Deployment Topology
# Architecture Review 01 — Cluster Infrastructure, Host, Failover, Health Monitoring, Deployment Topology (Round 2)
Reviewer domain: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure`, `src/ZB.MOM.WW.ScadaBridge.Host`,
`src/ZB.MOM.WW.ScadaBridge.HealthMonitoring`, `docker/`, `docker-env2/`, `deploy/wonder-app-vd03/`,
Traefik config, and the corresponding design docs (`Component-ClusterInfrastructure.md`,
`Component-Host.md`, `Component-HealthMonitoring.md`, `Component-TraefikProxy.md`) and test projects.
**Date:** 2026-07-12 (round 2; round-1 report 2026-07-08, baseline commit `b910f5eb`; current HEAD `8c888f13`)
## Scope
Read in full: `AkkaHostedService.cs` (1,203 lines — HOCON builder, singleton wiring, coordinated
shutdown), `Program.cs` (both role branches), `StartupValidator.cs`, `StartupRetry.cs`,
`SiteServiceRegistration.cs`, `NodeOptions.cs`, `ClusterOptions.cs` + validator +
`ServiceCollectionExtensions.cs`, all Host `Health/` checks (`RequiredSingletonsHealthCheck`,
`ActiveNodeGate`, `AkkaClusterNodeProvider`), `DeadLetterMonitorActor.cs`,
`AkkaHealthReportTransport.cs`, the entire HealthMonitoring project (`HealthReportSender`,
`CentralHealthAggregator`, `CentralHealthReportLoop`, `SiteHealthCollector`,
`SiteEventLogFailureCountReporter`, `SiteHealthKpiSampleSource`, options + validator), the docker
compose topologies + per-node appsettings + Traefik static/dynamic config + Dockerfile, the
on-host deploy artifacts (`deploy/wonder-app-vd03/appsettings.*.json`), the four design docs, the
repo's own Akka.NET reference notes (`AkkaDotNet/03-Cluster.md`), and the relevant test projects
(`Host.Tests`, `ClusterInfrastructure.Tests`, `HealthMonitoring.Tests`, `IntegrationTests`
failover/readiness suites). Health-report ingestion on the central side was cross-checked in
`Communication/Actors/CentralCommunicationActor.cs:349-392`.
Same domain as round 1: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure`, `src/ZB.MOM.WW.ScadaBridge.Host`,
`src/ZB.MOM.WW.ScadaBridge.HealthMonitoring`, `docker/`, `docker-env2/`, `deploy/wonder-app-vd03/`,
Traefik config, the four design docs (`Component-ClusterInfrastructure.md`, `Component-Host.md`,
`Component-HealthMonitoring.md`, `Component-TraefikProxy.md`), and the test projects.
## Maturity Verdict
## Method
This slice of the codebase is unusually well-commented, defensively coded at the micro level
(options validators with `ValidateOnStart`, CAS loops in the aggregator, sequence-number seeding
for failover, HOCON escaping, bounded startup retry), and clearly the product of many careful
review passes — but it has **one macro-level defect that undoes the entire failover design: the
split-brain resolver is configured but never enabled** (no `akka.cluster.downing-provider-class`),
so unreachable nodes are never automatically downed and cluster singletons never migrate on a hard
crash. Every document, comment, and validator in the domain reasons meticulously about keep-oldest
semantics that are inert at runtime, and the test suite asserts the HOCON *strings* rather than the
*behavior*, which is exactly how the gap survived. Secondary structural issues cluster around the
ambiguous definition of "active node" (cluster *leader* vs. singleton-hosting *oldest* member,
which diverge after restarts) and dev-vs-prod configuration drift. Verdict: **high polish, one
critical foundation crack; not production-ready for unattended failover until the downing provider
is fixed and verified with a real two-node test.**
1. Re-read the round-1 report and `archreview/plans/PLAN-01-cluster-host-failover.md` (findings-coverage table) plus the master tracker's cross-plan notes.
2. Verified **every** round-1 finding against the current source (not the plan's claims), with file:line evidence.
3. Fresh sweep over `git diff b910f5eb..HEAD --stat` scoped to the domain paths (~2,550 insertions across 68 files: `ClusterActivityEvaluator`, `CentralSingletonRegistrar`, `OldestNodeActiveHealthCheck`, the SBR downing line, acked health transport, metrics-staleness state machine, `failover-drill.sh`, options validators, the two-node cluster test rig, and adjacent new code that landed in this domain from other plans — `InProcessScriptArtifactChangeBus`, `CentralHealthAuditBacklogProvider`).
**One-line verdict:** Round 1 found one Critical foundation crack (SBR never enabled) under high polish; round 2 finds the crack fixed and behaviorally proven, 18 of 28 findings fully fixed and verified, the active-node model coherently unified — but the *shipped failover drill and recovery narrative overpromise*: a hard crash of the active (oldest) central node still ends in total central outage (registered deferred decision), and the never-executed drill script codifies a ~25s recovery that topology (first-seed bootstrap + keep-oldest asymmetry) cannot deliver in its most likely case.
---
## 1. Stability
## Round-1 Finding Disposition
### [Critical] Split-brain resolver is never enabled — automatic failover does not work for crashes/partitions
Every disposition below was verified by reading the current source; plan claims were not trusted.
`AkkaHostedService.BuildHocon` (src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs:225-270)
emits:
| # | Round-1 finding | R1 severity | Disposition | Evidence (file:line) |
|---|----------------|-------------|-------------|----------------------|
| S1 | SBR downing provider never enabled — no automatic failover on crash/partition | Critical | **Fixed (verified)** — provider named in HOCON; behaviorally proven by a real two-node kill test. Residual: two-node keep-oldest cannot survive an *oldest*-node crash — a **registered deferred user decision**, honestly documented in the test's XML doc | `AkkaHostedService.cs:290` (`downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster"`), doc at `:235-238`; `SbrFailoverTests.cs:46-80` (crash younger → member removed + singleton survives), asymmetry documented at `:15-23`; fixture builds from production `BuildHocon` (`TwoNodeClusterFixture.cs:56-64`) |
| S2 | "Active node" = leader vs oldest diverge (wrong purge node, wrong Primary label, wrong self-report) | High | **Fixed (verified)** — single oldest-member evaluator consumed by every product gate; divergence proven by a rejoin test | `ClusterActivityEvaluator.cs:20-47`; `ActiveNodeGate.cs:47-48`; `AkkaClusterNodeProvider.cs:36` (SelfIsPrimary), `:47-58` (Primary label from oldest); purge gate delegates to `SelfIsPrimary` (`SiteServiceRegistration.cs:118-122`); central self-report gate (`CentralHealthReportLoop.cs:93`); S&F delivery gate reuses it (`AkkaHostedService.cs:878-882`); `ActiveNodeSemanticsTests.cs:29-68` |
| S3 | Partition: both centrals return 200 on `/health/active`; Traefik serves both | High | **Fixed (verified)** — active = oldest Up member (exactly one in both partition views, since an unreachable member stays in gossip) AND database reachable | `OldestNodeActiveHealthCheck.cs:27-33`; `Program.cs:231-234` (database check tagged `Ready` + `Active`), `:252-257` (active-node = `OldestNodeActiveHealthCheck`); `Component-TraefikProxy.md:29,116` |
| S4 | 10s singleton drains inside the 5s `cluster-leave` phase | Medium | **Fixed (verified)** | `AkkaHostedService.cs:304-307` (`phases.cluster-leave.timeout = 15s`); parsed-config test in `HoconBuilderTests.cs` |
| S5 | NotificationOutbox + AuditLogIngest singletons had no drain task | Medium | **Fixed (verified)** — all seven central singletons routed through the registrar, which always adds a `PhaseClusterLeave` `GracefulStop` drain | `CentralSingletonRegistrar.cs:46-62`; call sites `AkkaHostedService.cs:466` (outbox), `:499` (ingest), `:564`, `:609`, `:632`, `:663`, `:689`; rationale comment `:459-465` |
| S6 | Health-report loss recovery was dead code (fire-and-forget transport never throws) | Medium | **Fixed (verified)** — transport is now an acked Ask round-trip; the counter-restore path is live | `AkkaHealthReportTransport.cs:30-40` (Ask + throw on nack/timeout); acks at `SiteCommunicationActor.cs:395-407` and `CentralCommunicationActor.cs:425`; restore fires on throw (`HealthReportSender.cs:170-187`) |
| S7 | Offline detection keyed off heartbeats masks a dead metrics pipeline; spec/code disagreed | Medium | **Fixed (verified)** — two-signal model: heartbeat liveness + `IsMetricsStale` on report silence; options cross-validated; spec updated to match. Residual: never-reported sites can't be flagged (new Low finding N3) | `CentralHealthAggregator.cs:295-312`; `HealthMonitoringOptions.cs` (`MetricsStaleTimeout`, `CentralOfflineTimeout`); validator cross-field rules (`HealthMonitoringOptionsValidator.cs`); `Component-HealthMonitoring.md:46-49`; UI badge `Health.razor:209` |
| S8 | `appsettings.Site.json` second seed targeted MetricsPort 8084; validator gap | Medium | **Fixed (verified)** — seed now 8085 with an explanatory `_seedNodes` note; validator rejects seeds on GrpcPort *and* MetricsPort | `appsettings.Site.json` (`SeedNodes` → `localhost:8085`); `StartupValidator.cs:117-134` |
| S9 | Docker stop grace shorter than CoordinatedShutdown | Medium | **Fixed (verified)** — `stop_grace_period: 30s` on all 8 `docker/` app services and all 4 `docker-env2/` services, each with a rationale comment | `docker/docker-compose.yml:7,34,61,81,101,121,141,161`; `docker-env2/docker-compose.yml:7,34,61,81` |
| S10 | Aggregator never forgets deleted sites | Low | **Fixed (verified)** — eviction piggybacks on the existing 60s site-address-cache refresh, no dedicated deletion event needed | `CentralHealthAggregator.cs:200-211` (`PruneUnknownSites`, protects the synthetic `$central` entry); caller `CentralCommunicationActor.cs:599-605` |
| S11 | Dead-letter monitor: unbounded warning volume | Low | **Fixed (verified)** — 10 warnings/min window, suppressed-count summary on rollover, metric counting never throttled | `DeadLetterMonitorActor.cs:22-26,51-80,89` |
| S12 | Mandatory two-seed rule forces phantom seeds in single-node installs | Low | **Partially fixed** — `AllowSingleNodeCluster` flag + validator with a phantom-seed-naming message shipped; the one real single-node artifact still ships the phantom seed and lacks the flag (overlay edit deferred out-of-band; see N2) | `ClusterOptions.cs:86`; `ClusterOptionsValidator.cs:29-35`; unfixed artifact: `deploy/wonder-app-vd03/appsettings.Central.json` (seeds still `localhost:8081` + phantom `localhost:8091`, no `AllowSingleNodeCluster`) |
| P1 | Readiness probes fan out 5 cluster Asks per poll | Low | **Accepted (per plan)** — unchanged, bounded 2s concurrent Asks | `RequiredSingletonsHealthCheck.cs` (only doc-comment change since baseline) |
| P2 | Startup compiles every inbound method sequentially | Low | **Fixed (verified)** (delivered via plan 06) — parallel compile with a thread-safety rationale | `Program.cs:412-415` (`Parallel.ForEach(methods, …)`) |
| P3 | Per-tick allocations in health collection | Low | **Won't-fix (per report's own "no action")** — unchanged | `CentralHealthAggregator.cs:187-190` |
| C1 | REQ-HOST-6 mandated Akka.Hosting; code hand-rolls HOCON | Medium | **Fixed (verified)** — resolved by amending the spec (hand-rolled HOCON is now the documented mechanism, with rationale) and dropping the three unused `Akka.*.Hosting` packages | `Component-Host.md:108-116` (bootstrap-mechanism note); `ZB.MOM.WW.ScadaBridge.Host.csproj:15-21` (packages removed, comment); `Component-Host.md:208` |
| C2 | `deploy/wonder-app-vd03` shipped `DisableLogin: true` | High | **Fixed (verified)** (delivered via plan 07) — fail-fast startup guard refuses `DisableLogin` outside Development without an explicit second acknowledgement key; the artifact no longer sets it and its `_comment_Security` documents the removal + a bundled localhost GLAuth for real logins | `DisableLoginGuard.cs:17-29`; call in `Program.cs` (Central branch, before `AddSecurity`); `deploy/wonder-app-vd03/appsettings.Central.json:40` |
| C3 | LDAP transport plaintext everywhere, contradicting "LDAPS/StartTLS required" | Medium | **Partially fixed / accepted posture** — docker remains dev-only plaintext (accepted); the deploy artifact now binds a *bundled loopback GLAuth* (`localhost:3893`) rather than plaintext across a network, with documented instructions to repoint to corporate AD — but `Transport: None, AllowInsecure: true` remains and a future AD repoint over the wire would still be plaintext unless changed | `deploy/wonder-app-vd03/appsettings.Central.json:29-30,40` |
| C4 | Secrets in checked-in docker dev configs | Medium | **Accepted (per plan)** — labelled dev-only; `ConfigSecretsTests` still tracks; `${...}` env-injection pattern remains the model in the deploy artifact | unchanged |
| C5 | Stale `Component-TraefikProxy.md` references | Low | **Fixed (verified)** — doc now names `OldestNodeActiveHealthCheck` with the correct path, documents `/health/ready` = database + akka-cluster + required-singletons, and the active check's DB tag | `Component-TraefikProxy.md:29,115-121,129` |
| C6 | Five copy-pasted ~60-line singleton registration blocks | Low | **Fixed (verified)** for the central role — `CentralSingletonRegistrar` collapses manager/drain/proxy; the two *site* singletons remain hand-rolled by documented intent (role-scoped; see N5) | `CentralSingletonRegistrar.cs:31-71`; `AkkaHostedService.cs:459-465` |
| U1 | No real multi-node failover test anywhere | — | **Fixed (verified)** — real two-node in-process cluster rig built from production HOCON, SBR kill test, active-node divergence test, plus a scripted docker drill (but see N1 — the drill itself was never executed) | `TwoNodeClusterFixture.cs`; `SbrFailoverTests.cs`; `ActiveNodeSemanticsTests.cs`; `CentralSingletonRegistrarTests.cs`; `docker/failover-drill.sh` |
| U2 | Windows Service lifecycle / down-if-alone recovery story undefined | — | **Partially fixed** — `WhenTerminated` watchdog stops the host process on an out-of-band ActorSystem termination so a supervisor restarts it (`UnexpectedTerminationTests` cover it); the recovery *contract* is now spec'd. But the `install.ps1` `sc.exe failure` recovery actions were deferred to the artifact owner, and the contract does not converge for a non-first-seed survivor (see N1) | `AkkaHostedService.cs:203-217` (watchdog), `:342-346` (`_stopRequested` guard); `Component-ClusterInfrastructure.md:104-111`; tracker "Deferred during Wave 1" |
| U3 | TLS absent at every layer | — | **Deferred (accepted, spec-visible)** — explicit "Production TLS profile: not yet implemented" roadmap note added as planned | `Component-TraefikProxy.md:133` |
| U4 | No Dockerfile HEALTHCHECK; `/healthz` runs zero checks | — | **Deferred (accepted per plan)** — unchanged (`grep -c HEALTHCHECK docker/Dockerfile` = 0; `Program.cs:381`) | — |
| U5 | `NodeName` missing from wonder-app-vd03 overlays → NULL `SourceNode` | — | **Not fixed** — the artifact's `Node` section still has no `NodeName` (verified by parsing the JSON). Deferred out-of-band with an owner; see N2 for why that rationale is shaky | `deploy/wonder-app-vd03/appsettings.Central.json` (`Node`: Role/NodeHostname/RemotingPort only) |
| U6 | Known-but-open follow-ups (SecuredWrite SourceNode NULL; cert-trust persistence; docker-env2 mirrors findings) | — | **Mixed, all tracked** — SecuredWrite `SourceNode` fixed (register row 7, via `ICentralAuditWriter`); cert-trust persistence still deferred with trigger (register row 11); docker-env2 got `stop_grace_period` | `docs/plans/2026-07-08-deferred-work-register.md:23,35`; `docker-env2/docker-compose.yml:7-81` |
| U7 | Online/offline transitions unrecorded | — | **Fixed (verified)** — `LastStatusChangeAt` stamped on every online↔offline flip (report-recovery, heartbeat-recovery, offline-detection), never moved by status-preserving traffic; surfaced as "offline since" in the UI; spec'd | `SiteHealthState.cs` (field + doc); `CentralHealthAggregator.cs:86,170-172,282`; `Health.razor:216`; `Component-HealthMonitoring.md:49` |
```hocon
cluster {
...
split-brain-resolver {
active-strategy = "keep-oldest"
stable-after = 15000ms
keep-oldest { down-if-alone = on }
}
failure-detector { ... }
run-coordinated-shutdown-when-down = on
}
```
**Disposition counts:** 18 Fixed (verified) · 3 Partially fixed (S12, C3, U2) · 1 Not fixed (U5, deferred out-of-band with owner) · 5 Deferred/accepted with rationale (P1, P3, C4, U3, U4) · 1 Mixed-tracked (U6). Plus one registered residual design gap on S1 (keep-oldest active-crash, deferred user decision).
but never sets `akka.cluster.downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider,
Akka.Cluster"`. In Akka.NET (1.5.62 per `Directory.Packages.props:8-17`), the default downing
provider is **NoDowning**; the whole `split-brain-resolver` section is only read by
`SplitBrainResolverProvider`, which must be opted into via `downing-provider-class`. A repo-wide
grep confirms the string `SplitBrainResolverProvider` appears **nowhere in `src/`** — only in the
project's own reference notes, which show the required line
(AkkaDotNet/03-Cluster.md:56, also 15-Coordination.md:69 and 18-MultiNodeTestRunner.md:100).
---
Failure scenario: the active site node loses power. The standby's failure detector marks it
unreachable (~10s), but **nothing ever downs it**. The unreachable member stays in the membership
forever, the `ClusterSingletonManager` cannot hand over (singleton migration requires the oldest
member's *removal*, not mere unreachability), and the site's DeploymentManager singleton — and with
it all data collection, script execution, and S&F delivery — never restarts on the survivor. The
same applies to all six central singletons (notification-outbox, audit-log-ingest, site-call-audit,
audit-log-purge, site-audit-reconciliation, kpi-history-recorder). The documented "~25 seconds
total failover" (docs/requirements/Component-ClusterInfrastructure.md:110-119) is only achievable
today for a **graceful** stop (CoordinatedShutdown leave); crash and partition scenarios hang
indefinitely until an operator downs the node or restarts the survivor.
## New Findings (Round 2)
Corroborating evidence that this was never exercised:
### [High] N1 — The down-if-alone recovery story does not converge for the non-first-seed survivor, and the shipped failover drill codifies a recovery the topology cannot deliver
- `HoconBuilderTests.cs:45,112,155` assert only that the SBR *strings* appear in the config.
- `CoordinatedShutdownTests.cs:11-43` are source-text greps (`Assert.Contains("run-by-clr-shutdown-hook = on", content)`).
- The "failover" integration tests are single-process simulations — e.g.
`CentralFailoverTests.cs:27-47` validates a JWT across two `JwtTokenService` instances; no test
forms a two-node cluster and kills the oldest node.
- `run-coordinated-shutdown-when-down = on` (AkkaHostedService.cs:265) is dead config: a node is
never transitioned to Down by anything.
- `ClusterOptionsValidator` (src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptionsValidator.cs:35-63)
rigorously rejects any strategy other than `keep-oldest` and demands `DownIfAlone == true` —
validating knobs on a resolver that is not installed.
The keep-oldest asymmetry itself is a *registered deferred user decision* (master tracker, 2026-07-08: crashing the **oldest/active** central node makes the younger survivor down itself — total central outage) and is honestly documented in `SbrFailoverTests.cs:15-23`. What is **new** in this round is that the artifacts built on top of that gap contradict it:
Fix: add the `downing-provider-class` line to `BuildHocon` (one line), or better, migrate the
bootstrap to `Akka.Cluster.Hosting`'s typed options (already referenced,
ZB.MOM.WW.ScadaBridge.Host.csproj:15) whose `SplitBrainResolverOption` wires the provider
automatically — and add a real two-node in-process cluster test that kills the oldest member and
asserts the singleton re-hosts within the expected window.
1. **`docker/failover-drill.sh:12-16` kills the ACTIVE node** — under the now-unified oldest-member semantics, the active node *is* the oldest, i.e. the drill exercises exactly the crash keep-oldest cannot survive. The intended safety net is Task 20's restart loop: survivor self-downs → `run-coordinated-shutdown-when-down` terminates its ActorSystem → the watchdog (`AkkaHostedService.cs:203-217`) exits the process → docker `restart: unless-stopped` (`docker-compose.yml:27,54`) restarts the container.
2. **But the restarted survivor can only re-bootstrap if it is the FIRST seed.** Both central nodes list `scadabridge-central-a` first (`docker/central-node-a/appsettings.Central.json:10-13`, `central-node-b/…:10-13`). Akka's seed-join rule lets only the *first* seed self-join to form a new cluster; a restarted `central-b` with `central-a` still dead loops on `InitJoin` forever — never `Up`, never `/health/active` 200. So when the drill's victim is `central-a` (the likelier oldest, since it is listed first and probed first by `active_container`), the drill fails at its 90s timeout and central stays dark until the victim returns. Only a `central-b` victim (or a younger-node crash) produces the advertised PASS.
3. **The drill has never been executed**: `docker/README.md:288` — "*run after next deploy and record here* (the script was validated with `bash -n`; no live cluster was up when it was committed)" — yet `docker/README.md:286` promises "Expected failover: **~25s** … plus the Traefik ~5s health-check interval", and `Component-ClusterInfrastructure.md:111` claims "the restarted process rejoins as a fresh incarnation (the keep-oldest resolver handles the rejoin cleanly)" — true only while the peer is reachable.
### [High] "Active node" means two different things — cluster leader vs. oldest member — and they diverge
**Recommendation:** (a) run the drill live once per direction (victim = a, victim = b) and record both outcomes in the README — the asymmetric result is itself the documentation the deferred decision needs; (b) fix the drill/README narrative to state which crash it can prove; (c) as a cheap partial mitigation independent of the strategy decision, make each node list *itself* first in its own `SeedNodes` (Akka tolerates asymmetric seed ordering) or document the operator action ("restart the dead node / restart the survivor with a self-first seed override") so a lone surviving central can actually re-form; (d) fold this into the pending keep-oldest topology/strategy decision (`keep-majority` + lighthouse node, `static-quorum`, or an accepted-risk note).
Two distinct node-selection concepts are used interchangeably:
### [Medium] N2 — The wonder-app-vd03 overlay edits remain unapplied while the artifact sits editable in the working tree; the deferral rationale is factually wrong
- **Leader-based** (address ordering, *not* age): `ActiveNodeGate.IsActiveNode`
(src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeGate.cs:36-52),
`AkkaClusterNodeProvider.SelfIsPrimary` and the "Primary"/"Standby" labels
(src/ZB.MOM.WW.ScadaBridge.Host/Health/AkkaClusterNodeProvider.cs:29-40,61), the `active-node`
health check Traefik routes on (Program.cs:231-234, docker/traefik/dynamic.yml:12-18), the
`CentralHealthReportLoop` leader gate (CentralHealthReportLoop.cs:93-97), and the site
event-log purge gate (`SiteEventLogActiveNodeCheck` → `SelfIsPrimary`,
SiteServiceRegistration.cs:116-120).
- **Oldest-based** (singleton placement): every `ClusterSingletonManager` (AkkaHostedService.cs:410+,
906-925), and the site health "IsActiveNode" flag, which is set by the DeploymentManager
*singleton* on start/stop (src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs:243,258).
Three PLAN-01 sub-edits (Tasks 16/20/23) were deferred with the rationale "that production deploy artifact is not tracked in this repo … the artifact directory itself is not present" (master tracker, Deferred during Wave 1). The directory **is** present at `deploy/wonder-app-vd03/` — it is merely gitignored (`.gitignore:48: /deploy/`) — and was clearly edited during this initiative (its `_comment_Security` at `appsettings.Central.json:40` describes the PLAN-07 DisableLogin removal and the new bundled GLAuth). So the artifact is being maintained in this working tree, yet still lacks:
In Akka the leader is the lowest-address `Up` member; the singleton host is the *oldest* member.
They coincide at first boot but diverge permanently after the original first node is restarted and
rejoins (it regains leadership by address order while the peer remains oldest). Concrete
consequences once diverged:
- `Node.NodeName` → every audit row from the **only production install** stamps a NULL `SourceNode`, defeating `IX_AuditLog_Node_Occurred` and the per-node stuck KPIs (round-1 U5, unchanged);
- `Cluster.AllowSingleNodeCluster: true` + removal of the phantom seed `localhost:8091` (round-1 S12's artifact half — the node still dials a nonexistent seed forever);
- the `install.ps1` `sc.exe failure` service-recovery actions that the down-if-alone watchdog contract *requires* ("the Host process must exit **so the service supervisor restarts it**" — `Component-ClusterInfrastructure.md:106-110`); without them, a self-downed single node on that host stays down.
1. **Site event-log retention is unenforced on the node that writes events.** Events are recorded
by the singleton hierarchy on the oldest node; the purge (`EventLogPurgeService.cs:21,36`)
early-exits unless `SelfIsPrimary` (leader). The active/oldest node's SQLite grows past the
30-day/1GB cap while the leader purges an idle database.
2. **Health dashboard labels the wrong node "Primary"** (AkkaClusterNodeProvider.cs:61): the node
named Primary hosts no singletons; the node doing all the work shows Standby.
3. **Central self-report is emitted by the leader** (CentralHealthReportLoop.cs:93) while the
NotificationOutbox/audit singletons run on the other node — the `$central` health card describes
the wrong node.
4. **Traefik routes UI/API to the leader** while every singleton `Ask` then hops to the peer —
functionally correct via the proxies, but it contradicts the design docs' "one active node runs
all central components" model (Component-ClusterInfrastructure.md:44-47) and doubles
intra-cluster latency on every outbox/audit page load.
**Recommendation:** apply the three documented edits directly to the on-disk artifact (they are one-liners specified in PLAN-01 Tasks 16/20/23), or correct the deferral record and add a RUNBOOK step so the gap is operator-visible.
Recommendation: pick one definition. Simplest coherent choice: derive "active" from singleton-host
status (e.g. compare self against the oldest `Up` member, or gate on reachability of a local
singleton), and use it consistently for `ActiveNodeGate`, `SelfIsPrimary`, the purge gate, and the
`/health/active` check — leadership is an Akka-internal concept that only accidentally matches the
product's active/standby model.
### [Low] N3 — Metrics-staleness can never fire for a site that has never delivered a report
### [High] During a central network partition, both nodes report `/health/active = 200` and Traefik serves both
`CentralHealthAggregator.CheckForOfflineSites` guards the staleness check with `state.LastReportReceivedAt is { } lastReport` (`CentralHealthAggregator.cs:300-302`), so a site registered via heartbeat only (`MarkHeartbeat` registration path, `:114-122`, leaves `LastReportReceivedAt` null) is never flagged `IsMetricsStale` and never logs the staleness warning. This is the "stuck from first boot" variant of round-1 S7: a site whose DeploymentManager singleton never starts (so `IsActiveNode` is false on both nodes and `HealthReportSender` skips every report — `HealthReportSender.cs:85-86`) shows online-with-no-metrics indefinitely. Mitigation: the UI renders "awaiting first report" for the null timestamp (`Health.razor:222`), so the state is at least visible — but it carries no warning styling or aggregator log line, unlike the stale case. Fix: treat `LastReportReceivedAt == null` with `now - LastHeartbeatAt > MetricsStaleTimeout` (or first-seen time) as stale too.
`ActiveNodeGate`/the active-node check compute `cluster.State.Leader` from each node's own gossip
view. When central-a and central-b cannot see each other but Traefik can reach both (a plausible
partial partition), each side sees the peer as unreachable and computes **itself** as leader — both
return 200 on `/health/active`, and Traefik's load balancer (docker/traefik/dynamic.yml:10-18)
round-robins requests between two nodes that each believe they are active. Blazor circuits and the
Inbound API's active-node 503 gate (Program.cs:256-260) are then split across both. Because the SBR
is not enabled (Critical finding above), nothing ever resolves the partition — this state persists
until manual intervention. Singletons do *not* duplicate (that requires member removal), but every
leader-gated behavior does. Enabling keep-oldest + `down-if-alone` fixes the persistence; consider
also making `/health/active` require `Ready` (DB reachable) in addition to leadership so a
DB-dead leader is pulled from rotation.
### [Low] N4 — `CentralHealthReportLoop` still narrates "cluster leader (Primary)"
### [Medium] Cluster-leave drain tasks are budgeted 10s inside a 5s CoordinatedShutdown phase
The class doc ("Only the cluster leader (Primary) generates reports", `CentralHealthReportLoop.cs:12-14`) and the sequence-seed comment ("reports from a newly-elected central leader", `:44-45`) survived the S2 rewiring. The behavior is correct (`:93` gates on `IClusterNodeProvider.SelfIsPrimary`, now oldest-member), but the wording re-teaches the exact leader/oldest conflation the fix eradicated. Two-line comment fix.
Five singletons add a `PhaseClusterLeave` task that runs `GracefulStop(TimeSpan.FromSeconds(10))`
(AkkaHostedService.cs:551-567 site-call-audit, 624-641 audit-log-purge, 676-693
site-audit-reconciliation, 733-750 kpi-history-recorder, 787-804 pending-deployment-purge). Akka's
`cluster-leave` phase uses the default phase timeout of **5 seconds**; the phase abandons the task
at 5s (default `recover = on`) and proceeds, so the 10s graceful-stop can never complete its full
window and the `catch` logging in those tasks never observes the overrun (the task is orphaned, not
faulted). The intended "let the in-flight EF upsert drain before handover" guarantee is silently
halved. Fix: set `akka.coordinated-shutdown.phases.cluster-leave.timeout` in `BuildHocon` to
exceed the graceful-stop budget, or shrink the `GracefulStop` timeout below 5s.
### [Low] N5 — Site-role singletons still lack drain tasks; the registrar can't express role scoping
### [Medium] The two busiest singletons have no drain task at all
`CentralSingletonRegistrar` deliberately excludes the two site singletons because they need `.WithRole(siteRole)` (`AkkaHostedService.cs:464-465`), so `deployment-manager` (`:791-810`) and `event-log-handler` (`:839-845`) remain hand-rolled with bare `PoisonPill` termination and no `PhaseClusterLeave` drain. The S5 rationale (let in-flight writes drain before handover) applies to the DeploymentManager's SQLite static-override/native-alarm persistence just as it did to the central EF writers; the site side is safe-but-lossy on graceful failover. Adding an optional `role` parameter to `CentralSingletonRegistrar.Start` (and renaming it `SingletonRegistrar`) closes the asymmetry for a few lines.
The `NotificationOutboxActor` and `AuditLogIngestActor` singletons are terminated by bare
`PoisonPill` with no cluster-leave drain (AkkaHostedService.cs:410-419, 451-458); the comment at
AkkaHostedService.cs:544-549 explicitly notes the pattern "is suitable for the NotificationOutbox
singleton; not added here to keep this change minimal". These are precisely the actors with
in-flight EF transactions on every message (dispatch sweeps, dual-write ingest). The idempotent
DB design (insert-if-not-exists on `NotificationId`/`EventId`) makes this safe-but-lossy rather
than corrupting, but the inconsistency (drain for the low-traffic maintenance singletons, none for
the hot ones) should be resolved.
### [Low] N6 — PLAN-01's two live deferrals are missing from the canonical deferred-work register
### [Medium] Health-report loss recovery is dead code on the production transport
`docs/plans/2026-07-08-deferred-work-register.md` (established by PLAN-08 T11 as the triage home, with rationale + revisit-trigger columns) contains neither (a) the keep-oldest active-crash availability gap nor (b) the wonder-app-vd03 overlay edits — both live only in `archreview/plans/00-MASTER-TRACKER.md` prose ("Follow-up discovered during P0 execution", "Deferred during Wave 1"). The register's structure (owner, rationale, revisit trigger) is exactly what the SBR gap — an availability-critical open decision — needs; today a reader of the register alone would conclude this domain has no open items.
`HealthReportSender` carefully restores interval counters if `_transport.Send` throws
(HealthReportSender.cs:158-175), and `SiteHealthCollector.AddIntervalCounters`
(SiteHealthCollector.cs:153-171) exists to support it. But the only production transport is
`AkkaHealthReportTransport.Send` (src/ZB.MOM.WW.ScadaBridge.Host/AkkaHealthReportTransport.cs:26-33)
— a fire-and-forget `Tell` to an `ActorSelection`, which **never throws** and silently returns when
the actor system is null. Any downstream loss (central unreachable, ClusterClient reconnecting,
dead-lettered selection) drops the report *and* the interval error counters it carried — the
scenario the restore logic was built for is exactly the one it can't see. Either make the transport
Ask-with-ack (the site→central notification forward already uses this pattern over the same actor)
or accept the loss and delete the restore machinery; the current shape gives false confidence.
---
### [Medium] Offline detection deviates from the spec and can mask a dead metrics pipeline
## What's Genuinely Good
`Component-HealthMonitoring.md:43-46` defines offline as "no *report* within 60s".
`CentralHealthAggregator.CheckForOfflineSites` (CentralHealthAggregator.cs:224-263) instead keys
off `LastHeartbeatAt`, which is also advanced by the ~5s transport heartbeats
(`CentralCommunicationActor.HandleHeartbeat` → `MarkHeartbeat`, CentralCommunicationActor.cs:349-353).
A site whose `HealthReportSender` loop has died (or whose DeploymentManager singleton is stuck, so
`IsActiveNode` is false on both nodes and reports are skipped — HealthReportSender.cs:85) keeps
heartbeating and shows **online with silently frozen metrics forever**; nothing surfaces
`LastReportReceivedAt` staleness. The heartbeat-based liveness choice is defensible (the code
comment argues it), but then the dashboard needs a distinct "metrics stale" signal, and the design
doc should be updated — currently spec and code disagree.
- **The Critical fix is proven, not just present.** The downing provider is one HOCON line (`AkkaHostedService.cs:290`), but the team built a real two-node in-process cluster rig from *production* `BuildHocon` output (`TwoNodeClusterFixture.cs:56-64`) and asserted member removal + singleton survival after an abrupt terminate — the exact test shape whose absence let round-1's Critical survive. `HoconBuilderTests` now parse the config instead of grepping strings.
- **Honest empiricism over plan compliance.** When the live rig revealed that killing the oldest node makes the survivor self-down, Task 4 was rewritten to the *achievable* guarantee and the asymmetry documented in the test XML doc (`SbrFailoverTests.cs:15-23`) and `ActiveNodeSemanticsTests.cs:16-24` (which switched to a graceful Leave with a documented deviation note) rather than shipping a green-but-lying test.
- **The active-node unification is genuinely single-sourced.** One 48-line static evaluator (`ClusterActivityEvaluator`) now backs the inbound-API gate, `/health/active`, the Primary/Standby labels, the event-log purge gate, the central self-report loop, and (via the `SelfIsPrimary` seam) plan 02's S&F delivery gate — with a comment in each consumer pointing back at the review finding. The seam design let a *different* plan consume it without inventing a parallel check (`AkkaHostedService.cs:873-882`).
- **`CentralSingletonRegistrar`** removed ~300 lines of copy-paste *and* fixed the drift it caused (missing drains) in one move — the correct root-cause response to a copy-paste bug.
- **The health state machine is careful engineering:** immutable `SiteHealthState` records swapped via CAS loops with correct lost-race semantics (a lost CAS on offline-marking means the site was just heard from — deliberately not retried, `CentralHealthAggregator.cs:279-292`), clock-skew-anchored heartbeat recovery (`:145-156`), a collision-proof `$central` synthetic key, and cross-field options validation with key-naming messages (`HealthMonitoringOptionsValidator`).
- **Doc reconciliation was done properly:** REQ-HOST-6 was amended with a rationale note *and* the unused packages dropped (instead of quietly deleting the requirement); the Traefik doc, the health-monitoring spec, and the cluster-infrastructure recovery contract all now describe the code as built.
- **`DisableLoginGuard`** is the right shape for the C2 fix: fail-fast at boot, environment-gated, with an explicit consciously-named acknowledgement key — a durable control rather than a one-time artifact edit.
### [Medium] Second seed node in the checked-in site dev config targets the metrics port; validator has a gap
## Severity Tally — New Findings Only
`src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json:14-17` seeds
`akka.tcp://scadabridge@localhost:8082` and `akka.tcp://scadabridge@localhost:8084`, while the same
file sets `MetricsPort: 8084` (line 10). The second seed points at the Kestrel HTTP/1.1 metrics
listener — a node attempting to join via that seed performs a doomed Akka.Remote association
against Kestrel (handshake noise, retry loops). `StartupValidator` explicitly guards seed-vs-GrpcPort
(StartupValidator.cs:121-127) but not seed-vs-MetricsPort, so this config validates cleanly. Fix
the dev config (8084 → the intended second node's remoting port) and extend the seed-port loop to
also reject `MetricsPort`.
### [Medium] Docker stop grace period is shorter than a full CoordinatedShutdown
`docker/docker-compose.yml` sets no `stop_grace_period` (default 10s SIGTERM→SIGKILL) and the
Dockerfile runs `dotnet` as PID 1 (docker/Dockerfile, final stage). A graceful shutdown runs
cluster-leave (5s budget) + cluster-exiting (10s) + actor-system-terminate plus Serilog flush —
easily exceeding 10s on a loaded node, so `bash docker/deploy.sh` recreates can SIGKILL mid-drain,
turning every "graceful" redeploy into the crash path (which, per the Critical finding, currently
has no automatic recovery on the surviving node). Add `stop_grace_period: 30s` to the app services.
### [Low] Aggregator never forgets sites
`CentralHealthAggregator._siteStates` (CentralHealthAggregator.cs:16) only grows: a site deleted
from configuration remains a permanently-offline dashboard tile (and continues to be enumerated by
`SiteHealthKpiSampleSource.CollectAsync`, SiteHealthKpiSampleSource.cs:74-85) until process
restart. Needs an eviction path wired to site deletion.
### [Low] Dead-letter monitor: unbounded warning volume and restart-reset count
`DeadLetterMonitorActor` logs one Warning per dead letter with no sampling/rate-limiting and keeps
its count in actor state that resets on restart
(src/ZB.MOM.WW.ScadaBridge.Host/Actors/DeadLetterMonitorActor.cs:14,26-35). A dead-letter storm
(mass undeploy, failover, shutdown) floods the log; the health metric path
(`IncrementDeadLetter`) is fine because the collector counter is interval-based.
### [Low] Mandatory two-seed rule forces phantom seeds in single-node installs
`ClusterOptionsValidator` requires ≥2 seed nodes
(ClusterOptionsValidator.cs:29-33); the shipped single-node production artifact satisfies it with
a documented placeholder (`deploy/wonder-app-vd03/appsettings.Central.json:9-19`, second seed
`localhost:8091` "harmless while absent"). The node dials the phantom seed forever (log noise,
periodic connect attempts), and the rule's intent — no startup-ordering dependency — is defeated
by fiction. Consider allowing 1 seed with a warning, or a `SingleNode` acknowledgement flag.
## 2. Performance
This domain is largely cold-path and correctly engineered for it; nothing here touches the
data-plane hot path. Observations:
- **[Low] Readiness probes fan out five cluster Asks per poll.**
`RequiredSingletonsHealthCheck` sends 5 concurrent `Identify` Asks through singleton proxies
every `/health/ready` poll (RequiredSingletonsHealthCheck.cs:66-78,111-117). When the singletons
live on the peer node each probe is a remoting round trip. Bounded (2s, concurrent) and cheap at
typical poll rates, but a tight orchestration probe interval multiplies cluster chatter.
- **[Low] Startup compiles every inbound API method sequentially before serving**
(Program.cs:379-388). Startup latency scales linearly with method count; a slow Roslyn compile of
many methods delays readiness on the *standby* too (it runs the same branch). Acceptable now;
parallelize or defer-compile if method counts grow.
- **[Low] Per-tick allocations in health collection** (dictionary snapshots per 30s report,
`GetAllSiteStates` copy per dashboard poll — CentralHealthAggregator.cs:173-176) are trivial at
this cadence. No action.
- **Positive:** the site audit telemetry actor is pinned to a dedicated dispatcher
(`audit-telemetry-dispatcher`, AkkaHostedService.cs:226-232,1150-1158) so batch SQLite/gRPC work
never contends with hot-path actors; `StoreAndForwardService.StartAsync` is awaited rather than
sync-blocked (AkkaHostedService.cs:976-981); the ActorSystem DI bridge is a root singleton
precisely to avoid per-scope disposal (Program.cs:240-250) — all correct.
## 3. Conventions
- **[Medium] REQ-HOST-6 says "Akka.NET bootstrap via Akka.Hosting"; the code hand-rolls HOCON.**
`Component-Host.md` (REQ-HOST-6, line 106-114) mandates Akka.Hosting; the actual bootstrap is
string-interpolated HOCON + `ActorSystem.Create` (AkkaHostedService.cs:171-176,214-271) with
`Akka.Hosting`/`Akka.Cluster.Hosting`/`Akka.Remote.Hosting` referenced but unused for cluster
bring-up (ZB.MOM.WW.ScadaBridge.Host.csproj:15-18). This is not just doc drift: Akka.Cluster.Hosting's
typed `SplitBrainResolverOption` sets `downing-provider-class` for you — using the mandated API
would have prevented the Critical finding. Either implement REQ-HOST-6 or amend it and drop the
unused packages.
- **[High] Production deploy artifact ships with authentication disabled.**
`deploy/wonder-app-vd03/appsettings.Central.json:39-42` sets `"DisableLogin": true` — every
request auto-authenticates as `multi-role` with all roles, and the management API on that host is
unauthenticated. The base config documents this as "DEV/TEST ONLY — never enable in production"
with "no environment guard; a startup warning is the only protection"
(src/ZB.MOM.WW.ScadaBridge.Host/appsettings.json:6). A `deploy/` artifact with an install
runbook is a production surface; at minimum gate `DisableLogin` on
`Environment.IsDevelopment()` or require an explicit `I-understand` companion flag.
- **[Medium] LDAP transport is plaintext everywhere, contradicting the security spec.** The design
requires "LDAPS/StartTLS required" (CLAUDE.md Security & Auth; Component-Security), but every
shipped config — docker (docker/central-node-a/appsettings.Central.json:26-33) and the on-host
deploy (deploy/wonder-app-vd03/appsettings.Central.json:26-34) — sets `"Transport": "None",
"AllowInsecure": true` and sends the service-account bind password in clear. Dev GLAuth is a fair
excuse for docker; the deploy artifact should not normalize it.
- **[Medium] Secrets in checked-in dev configs.** SQL credentials, LDAP service-account password
and the JWT signing key are committed in `docker/central-node-a/appsettings.Central.json:21-38`
(labelled dev-only; the compose pepper is likewise flagged, docker-compose.yml:9-14). The
wonder-app-vd03 artifact handles this correctly via `${...}` placeholders + env injection
(appsettings.Central.json:20-24) — good pattern, worth backporting a scrubbed variant to docker
if these ever leave a lab network. Also note `ConfigSecretsTests.cs` exists in Host.Tests, so the
team is tracking this.
- **[Low] Stale doc references.** `Component-TraefikProxy.md:121` locates `ActiveNodeHealthCheck`
at `src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeHealthCheck.cs`; the check now comes from the
shared `ZB.MOM.WW.Health.Akka` package (Program.cs:214-234) and no such file exists.
`Component-TraefikProxy.md:115` also still describes `/health/ready` as "database + Akka
cluster" — it now includes `required-singletons` (Program.cs:227-230; Component-Host.md REQ-HOST-4a
*is* current). Keep the Traefik doc in sync.
- **[Low] Five copy-pasted ~60-line singleton registration blocks.**
AkkaHostedService.RegisterCentralActors repeats the manager/drain-task/proxy triple five times
(AkkaHostedService.cs:399-811) with only names and Props varying. Extract a
`StartCentralSingleton(name, props, drainTimeout)` helper — the copy-paste already caused drift
(outbox/ingest lack drain tasks).
- **Positive conventions compliance:** Options pattern is followed rigorously (component-owned
options classes, `ValidateOnStart`, dedicated validators with key-naming messages —
ClusterOptionsValidator, HealthMonitoringOptionsValidator); HOCON interpolation is
injection-safe (`QuoteHocon`/`DurationHocon`, AkkaHostedService.cs:280-295); Serilog enrichment
matches REQ-HOST-8; startup validation matches REQ-HOST-4 including cross-field port-collision
rules; single-binary role dispatch matches REQ-HOST-1/2; the `/health/ready` vs `/health/active`
tier separation (leadership deliberately excluded from readiness) is well-reasoned and
documented in place (Program.cs:346-356).
## 4. Underdeveloped Areas
1. **No real multi-node failover test anywhere.** `CentralFailoverTests` are single-process
simulations (JWT across two service instances, CentralFailoverTests.cs:27-47);
`SiteFailoverTests`/`DualNodeRecoveryTests` similarly exercise component state recovery, not
cluster membership; `CoordinatedShutdownTests` grep source text (CoordinatedShutdownTests.cs:11-43);
`HoconBuilderTests` assert config strings. The repo's own notes describe the MultiNode test
runner (AkkaDotNet/18-MultiNodeTestRunner.md) but it is unused. This is the direct cause of the
Critical finding surviving: nothing ever killed the oldest node and waited for the singleton to
move. Even a plain xunit test spinning two `ActorSystem`s in-process and calling
`cluster.Down()`/terminating one would have caught it.
2. **Windows Service lifecycle management is thin.** `Component-ClusterInfrastructure.md:19`
claims "Manage Windows service lifecycle (start, stop, restart)"; the implementation is
`UseWindowsService()` (Program.cs:74,401) plus install scripts. Service recovery actions
(restart-on-failure), the watchdog implied by `down-if-alone` ("an external mechanism must
restart both nodes" — AkkaDotNet/03-Cluster.md:64) do not exist in the repo; after a
down-if-alone self-down the process presumably exits (or worse, idles) with nothing to restart
it. The down-if-alone recovery story needs an explicit design (does the process exit? does the
service restart it? does it rejoin cleanly?) and a drill.
3. **TLS is absent at every layer of this domain**: Traefik terminates plain HTTP with an
unauthenticated dashboard (docker/traefik/traefik.yml:5-7 — documented as dev-only in
Component-TraefikProxy.md:131-138), Akka remoting is unencrypted TCP, site gRPC is `http://`.
Fine for the lab; there is no production TLS profile or documented path yet.
4. **Dockerfile has no `HEALTHCHECK`** and compose has no health-based dependency ordering;
liveness `/healthz` runs zero checks (Program.cs:353-355) so a wedged actor system still
reports live. Acceptable minimalism, but an orchestrator migration will need both.
5. **`NodeName` is missing from the wonder-app-vd03 deploy overlays**
(deploy/wonder-app-vd03/appsettings.Central.json has no `Node.NodeName`), so audit rows from
that install get a NULL `SourceNode` (the base file documents the normalisation,
src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Central.json:3) — undermines the per-node audit/KPI
features (`IX_AuditLog_Node_Occurred`, per-node stuck KPIs) on the one real deployment.
6. **Known-but-open follow-ups acknowledged in code/docs:** SecuredWrite audit rows leave
`SourceNode` NULL (CLAUDE.md notes it as logged); no central persistence of OPC UA cert trust;
`docker-env2` mirrors the main topology (verified: same Traefik config shape, same
seed/port conventions) so it inherits every finding above.
7. **Health monitoring has no history by design** (Component-HealthMonitoring.md:93-97) — now
partially superseded by KPI History (#26), which the docs handle; but online/offline
*transitions* are still unrecorded anywhere (doc line 97 defers it), so post-incident "when did
the site drop" questions remain unanswerable beyond log scraping.
## Prioritized Recommendations
1. **[Critical, one line + tests] Enable the split-brain resolver.** Add
`downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster"` to
`BuildHocon` (AkkaHostedService.cs:250) — or move the bootstrap to Akka.Cluster.Hosting per
REQ-HOST-6, which sets it via typed options. Then add a two-node in-process failover test
(kill oldest → assert singleton re-hosted, member removed within stable-after + margin) and a
documented manual drill against the docker cluster (`docker kill scadabridge-central-a`, watch
Traefik + singleton logs).
2. **[High] Unify the "active node" definition** on singleton-host (oldest) semantics for
`ActiveNodeGate`, `AkkaClusterNodeProvider.SelfIsPrimary`, the event-log purge gate, and the
`/health/active` check; today's leader-based checks purge the wrong node's event log and label
the wrong node Primary once leader ≠ oldest.
3. **[High] Remove `DisableLogin: true` from `deploy/wonder-app-vd03`** or add a hard environment
guard around the flag.
4. **[Medium] Fix the CoordinatedShutdown drain budget** (`cluster-leave` phase timeout ≥
graceful-stop timeout) and add drain tasks for the notification-outbox and audit-log-ingest
singletons — or delete all five drains if the idempotent-DB argument makes them unnecessary.
5. **[Medium] Fix `appsettings.Site.json` seed `localhost:8084`** and extend `StartupValidator`'s
seed-port loop to reject `MetricsPort` collisions.
6. **[Medium] Make health-report delivery observable**: switch `AkkaHealthReportTransport` to an
acked Ask (reusing the existing site→central forward pattern) so the counter-restore logic in
`HealthReportSender` actually fires, and surface `LastReportReceivedAt` staleness on the
dashboard so heartbeat-alive/metrics-dead sites are visible.
7. **[Medium] Add `stop_grace_period: 30s`** to the app services in both compose files.
8. **[Low] Housekeeping:** evict deleted sites from `CentralHealthAggregator`; rate-limit
dead-letter warnings; extract the singleton-registration helper; sync
`Component-TraefikProxy.md` with the shared health-check library; add `NodeName` to the
wonder-app-vd03 overlays; reconsider the mandatory-two-seeds rule for single-node installs.
| Severity | Count | Findings |
|----------|-------|----------|
| Critical | 0 | — |
| High | 1 | N1 (recovery story doesn't converge for non-first-seed survivor; never-run drill codifies an undeliverable ~25s recovery) |
| Medium | 1 | N2 (wonder-app-vd03 overlay edits unapplied; deferral rationale factually wrong) |
| Low | 4 | N3 (metrics-stale null-report gap), N4 (stale "leader" comments), N5 (site singletons undrained / registrar role gap), N6 (deferrals missing from canonical register) |
archreview/02-communication-store-and-forward.md
+97 -122
View File
@@ -1,160 +1,135 @@
# Architecture Review — Central–Site Communication & Store-and-Forward
# Architecture Review — Central–Site Communication & Store-and-Forward (Round 2)
Reviewer domain: `src/ZB.MOM.WW.ScadaBridge.Communication`, `src/ZB.MOM.WW.ScadaBridge.StoreAndForward`, the telemetry transport paths that ride them, and the corresponding test projects.
Date: 2026-07-08.
Reviewer domain: `src/ZB.MOM.WW.ScadaBridge.Communication`, `src/ZB.MOM.WW.ScadaBridge.StoreAndForward`, the telemetry transport paths that ride them, the Host wiring that composes them, and — new this round — the **aggregated live alarm stream** shipped 2026-07-10 (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`): site-wide alarm-only `SubscribeSite` gRPC stream, central `ISiteAlarmLiveCache`/`SiteAlarmLiveCacheService`/`SiteAlarmAggregatorActor`, seed-then-stream, and reconnect telemetry.
## Scope
Date: 2026-07-12 (round 2). Round-1 report dated 2026-07-08 at baseline `b910f5eb`; this round re-reviews everything landed since (PLAN-02's 24 tasks, PLAN-08 options validation, and the post-initiative live-alarm-stream feature) on `main` @ `8c888f13`.
Read in full: `Component-Communication.md`, `Component-StoreAndForward.md`; all 9 S&F source files (`StoreAndForwardService.cs`, `StoreAndForwardStorage.cs`, `ReplicationService.cs`, `NotificationForwarder.cs`, `ParkedMessageHandlerActor.cs`, options/messages/DI); all Communication source (`CentralCommunicationActor.cs`, `SiteCommunicationActor.cs`, `CommunicationService.cs`, `DebugStreamService.cs`, `DebugStreamBridgeActor.cs`, `StreamRelayActor.cs`, `SiteStreamGrpcServer.cs`, `SiteStreamGrpcClient.cs`, `SiteStreamGrpcClientFactory.cs`, `ReconcileService.cs`, `PendingDeploymentPurgeActor.cs`, `CommunicationOptions.cs`); the transport ends of the telemetry pipelines (`AuditLog/Site/Telemetry/SiteAuditTelemetryActor.cs`, `ClusterClientSiteAuditClient.cs`); and the Host wiring that composes them (`Host/Actors/AkkaHostedService.cs` — `BuildHocon`, `RegisterCentralActors`, `RegisterSiteActorsAsync`), plus `SiteRuntime/Actors/SiteReplicationActor.cs` and the `Notify.Send` enqueue path in `SiteRuntime/Scripts/ScriptRuntimeContext.cs`. Central-side ingest internals (NotificationOutboxActor, AuditLogIngestActor, SiteCallAuditActor) are another reviewer's domain; only their transport contracts are assessed here.
## Scope & Method
## Maturity Verdict
- Read the round-1 report and `archreview/plans/PLAN-02-communication-store-and-forward.md` (findings-coverage table), then **verified every round-1 finding against the current source** — no plan claim was trusted without file:line evidence.
- Fresh sweep over `git diff b910f5eb..HEAD` scoped to the Communication and StoreAndForward projects and their tests (~4,900 insertions across 50 files, 32 commits), plus the SiteRuntime/Host/CentralUI files the new code touches (`SiteReplicationActor`, `SiteStreamManager`, `AkkaHostedService`, `ScriptRuntimeContext`, `AlarmSummary.razor`).
- Static review only (suites verified green 2026-07-10). Known context: the `SiteStreamGrpcServerTests.SiteConnectionUpGauge_*` observable-gauge timing flake pre-exists on main and is not a regression from these plans.
This is a carefully engineered, unusually well-documented subsystem that is strong *in the small* — idempotency keys threaded end-to-end (NotificationId, EventId, TrackedOperationId), Sender captured before every PipeTo, compare-and-swap storage updates guarding the sweep against operator races, additive-only proto discipline with contract tests, and deliberate, written-down failure semantics on nearly every path. However, it has a small number of serious *in-the-large* lifecycle/topology defects that the (otherwise broad) test suite is structurally blind to, because the risky collaborators are mocked away: the ClusterClient recreate-on-address-change path will almost certainly crash-loop the `CentralCommunicationActor` in production, the standby site node runs the full S&F delivery pipeline with no active-node gate (systematic duplicate-delivery exposure the spec explicitly says should only occur on failover), and one debug session's gRPC failover disposes the shared channel out from under every other session to the same site. The retry sweep's serial, unbounded design also collapses under a realistic central outage backlog. Verdict: **high code quality, mid maturity as a distributed system — 3–4 targeted fixes away from solid.**
**One-line verdict:** round 1's Critical and all three Highs are genuinely fixed and well-tested — but the **new** anti-entropy resync was built on the *wrong* active-node predicate (leader instead of the oldest-Up helper the delivery gate uses) and can wipe the delivering node's live buffer after a routine rolling restart, and its snapshot message cannot fit through Akka remoting's default frame size — so round 2 finds one new Critical and one new High, both confined to the peer-resync feature; everything else, including the brand-new live alarm stream, is solid work.
---
## Findings — Stability
## Round-1 Finding Disposition
### [Critical] ClusterClient recreate-on-address-change causes actor-name collision and a permanent restart loop
`CentralCommunicationActor.HandleSiteAddressCacheLoaded` stops the old client and immediately creates a new one under the same name:
| Round-1 finding | R1 severity | Disposition | Evidence (file:line) |
|---|---|---|---|
| ClusterClient recreate-on-address-change → name collision → restart loop | Critical | **Fixed (verified)** | `CentralCommunicationActor.cs:33-67` — `DefaultSiteClientFactory` mints `site-client-{sanitized}-{generation}` via `Interlocked.Increment` (48); real-factory lifecycle tests exist (`tests/.../DefaultSiteClientFactoryTests.cs`, `CentralCommunicationActorClientLifecycleTests.cs`) |
| `siteId` interpolated unvalidated into actor name (sub-finding) | Critical (sub) | **Fixed (verified)** | `CentralCommunicationActor.cs:59-66` — `SanitizeForActorName` maps to valid path elements, falls back to `"site"`; `Create` additionally wrapped in try/catch that logs and skips the site (582-593), stale entry removed before recreate (572-580) |
| Standby site node runs full S&F delivery pipeline (systematic duplicates) | High | **Fixed (verified)** | `StoreAndForwardService.cs:89-103` (gate seam), 682-698 (re-evaluated per sweep tick, throwing gate = standby); Host wires it to the shared singleton-host helper `IClusterNodeProvider.SelfIsPrimary` (`AkkaHostedService.cs:868-884`); unit tests cover standby-skip / flip-active / throwing-gate. **But see NEW finding N2** — the *resync* feature added alongside uses a *different* predicate |
| Debug-session gRPC failover disposes shared channel; `CleanupGrpc` creates channels during teardown | High | **Fixed (verified)** | `SiteStreamGrpcClientFactory.cs:26,61-73` — cache keyed by `(site, endpoint)`, `GetOrCreate` never disposes, `TryGet` never creates; `RemoveSiteAsync` (95-102) is the only shared-disposal path; `DebugStreamBridgeActor.cs:469-473, 495-498` — both teardown paths use `TryGet(...)?.Unsubscribe(...)` |
| Retry sweep serial, unbounded, single-laned — collapses under backlog | High (perf) | **Fixed (verified)** | `StoreAndForwardOptions.cs:31` `SweepBatchLimit` (500) → `StoreAndForwardStorage.cs:392-397` SQL `LIMIT`; per-`(category,target)` lane short-circuit on first transient failure (`StoreAndForwardService.cs:721-730`, `RetryOutcome` 752); parallel lanes capped at `SweepTargetParallelism` 4, sequential within lane (710-734) |
| Replication ops applied out of order (`Task.Run` fire-and-forget) | Medium | **Fixed (verified)** | `ReplicationService.cs:142-161` — handler invoked inline; ordering preserved by construction, faults observed via `ContinueWith(OnlyOnFaulted)` |
| Replication failures logged at Debug — dead channel invisible | Medium | **Fixed (verified)** | `ReplicationService.cs:153-168` — Warning + `ScadaBridgeTelemetry.RecordReplicationFailure()`; `SiteReplicationActor.cs:225-234` — no-peer drop also Warning + counter |
| Park/Requeue replication blind UPDATE — lost on standby missing the row | Medium | **Fixed (verified)** | `ReplicationService.cs:115-140` — Add/Park/Requeue applied via `StoreAndForwardStorage.UpsertMessageAsync` (`ON CONFLICT(id) DO UPDATE`, `StoreAndForwardStorage.cs:319-369`, `created_at` preserved) |
| Notifications park after ~25 min; `maxRetries: 0` escape hatch unused | Medium | **Fixed (verified)** | `ScriptRuntimeContext.cs:2209` — `Notify.Send` enqueues with `maxRetries: 0`; park-on-exhaustion is impossible (`StoreAndForwardService.cs:830` guards `MaxRetries > 0`) |
| Corrupt buffered notification discarded and reported as delivered | Medium | **Fixed (verified)** | `NotificationForwarder.cs:82-95` — undeserialisable payload returns `false` → engine **parks** the row; Warning log with capped payload preview (170-187) |
| Spec-vs-code drift: standby-passive model; `Notify.Send` blocks up to 30 s | Medium | **Fixed (verified)** | `Component-StoreAndForward.md:80-83` — gate + failover-window-only duplicates documented; `:52` — enqueue-only `deferToSweep` semantics documented; code matches (`StoreAndForwardService.cs:588-594` deferToSweep path never invokes the handler inline, kicks `TriggerSweep`) |
| No explicit Akka serializer — wire format unpinned | Medium (perf) | **Partially fixed / deferred (accepted)** | Round-trip pin tests added (`tests/.../WireSerializationPinTests.cs` — 191 lines, `ReplicationWireSerializationPinTests.cs`; PLAN-08 T9 additionally found the Newtonsoft ctor-optional-default gotcha and locked contracts). The serializer *swap* remains deferred by design (rolling-compat migration) — reasonable |
| Central heartbeat state not replicated to peer central node | Low | **Fixed (verified)** | `CentralCommunicationActor.cs:369-398` — `SiteHeartbeatReplica` fanned over DistributedPubSub, idempotent local mark (194-196, 390-398) |
| `SubscribeInstance` concurrency check-then-act (comment only asked) | Low | **Fixed as specified** | `SiteStreamGrpcServer.cs:287-292` — deliberate check-then-act documented with bounded-overshoot rationale |
| S&F SQLite: no WAL, no busy_timeout | Low | **Fixed (verified)** | `StoreAndForwardStorage.cs:43-47` (WAL in `InitializeAsync`), 177-185 (5 s busy_timeout per pooled open) |
| Due-check per-row `julianday()` parsing | Low | **Fixed (verified)** | Additive `last_attempt_at_ms` column + one-time backfill + `(status, last_attempt_at_ms)` index (`StoreAndForwardStorage.cs:88-114`); due predicate is pure integer math (386-400) |
| Audit-observer notification awaited inline in the sweep | Low | **Fixed (verified)** | `StoreAndForwardService.cs:117-135, 898-922` — ordered single-reader channel + pump; inline fallback preserved when not started; drained on `StopAsync` (451-468) |
| `StreamRelayActor.WriteToChannel` dead branch + silent DropOldest loss | Low | **Fixed (verified)** | `StreamRelayActor.cs:105-115` — comment corrected (guards closed channel only); real eviction counted by the bounded channel's `itemDropped` callback with first + every-500th Warning (`SiteStreamGrpcServer.cs:303-315`) |
| Duplicated Ask-timeout constants between the two ingest transports | Low | **Fixed (verified)** | Single source of truth `SiteStreamGrpcServer.AuditIngestAskTimeout` (`SiteStreamGrpcServer.cs:47`), consumed by `CentralCommunicationActor.cs:140-150, 174` |
| Heartbeat `IsActive` leader-check vs oldest-node singleton placement | Low | **Partially fixed — and now materially worse** | PLAN-01 shipped the oldest-Up helper (`ClusterActivityEvaluator.SelfIsOldest`, `Host/Health/ClusterActivityEvaluator.cs:20-31`; `AkkaClusterNodeProvider.SelfIsPrimary:29-37`) and the S&F delivery gate uses it — but `SiteCommunicationActor.DefaultIsActiveCheck` (517-526) and `SiteReplicationActor.DefaultIsActive` (190-198) still use **leader+Up** with "swap point" comments never swapped. The new resync feature was built on the leader predicate → **NEW finding N2 (Critical)** |
| `TransportHeartbeatInterval` double duty | Low | **Fixed (verified)** | `CommunicationOptions.cs:59-65` — dedicated `ApplicationHeartbeatInterval` (default 5 s, no behavior change), used at `SiteCommunicationActor.cs:444`; validated (`CommunicationOptionsValidator.cs:54-55`) |
| U1: Dual transports for the same ingest operations | Underdev | **Deferred (accepted)** | Both paths still exist (`SiteStreamGrpcServer.cs:388-491` gRPC vs `CentralCommunicationActor.cs:282-325` ClusterClient) with the shared timeout constant removing the tuning drift; consolidation remains a registered follow-up. Acceptable |
| U2: No real-collaborator test for ClusterClient cache lifecycle | Underdev | **Fixed (verified)** | `DefaultSiteClientFactoryTests.cs` (real factory, collision + sanitize), `CentralCommunicationActorClientLifecycleTests.cs` (address-edit ×3, throwing factory survives) |
| U3: No test/design treatment of standby-node sweep behavior | Underdev | **Fixed (unit level)** | Gate tests in `StoreAndForwardServiceTests.cs` (skip/resume/throwing-gate); doc updated; the live two-node kill rig landed in PLAN-01 (`TwoNodeClusterFixture`) |
| U4: Replication has no anti-entropy/resync | Underdev | **Fixed — with two new defects** | `SiteReplicationActor.cs:111-113, 175-181, 398-444` + `StoreAndForwardStorage.GetAllMessagesAsync`/`ReplaceAllAsync` (256-306). The mechanism exists and is authority-checked and capped — but see **N1 (frame size)** and **N2 (predicate inversion)** below |
| U5: Parked rows unbounded retention, no aging signal | Underdev | **Fixed (verified)** | `StoreAndForwardStorage.GetOldestParkedCreatedAtAsync` (675-685) → `HealthReportSender.cs:114-115` → additive `SiteHealthReport.OldestParkedMessageAgeSeconds` (`SiteHealthReport.cs:57`) |
| U6: Reconciliation cursor edge (`>` vs `>=`) unpinned | Underdev | **Fixed (verified, exceeded)** | `PullSiteCalls` grew an additive composite-keyset cursor (`after_id`, `SiteStreamGrpcServer.cs:598-641`); pinned by `SiteStreamPullSiteCallsTests.cs` |
| U7: `DebugStreamService._sessions` lost silently on central restart | Underdev | **Documented (accepted)** | `Component-Communication.md:67` — explicit "Accepted limitation" paragraph citing this review |
- `CentralCommunicationActor.cs:514-521` — `Context.Stop(_siteClients[siteId].Client)` followed synchronously by `_siteClientFactory.Create(...)`.
- `CentralCommunicationActor.cs:36-40` (`DefaultSiteClientFactory`) — `system.ActorOf(ClusterClient.Props(settings), $"site-client-{siteId}")`: a **named top-level actor**.
`Context.Stop` is asynchronous; the name `site-client-{siteId}` stays reserved until the old actor fully terminates. Creating the replacement in the same message handling will throw `InvalidActorNameException` essentially every time. The `Create` call is **outside** the surrounding try/catch (which only guards `ActorPath.Parse`, `CentralCommunicationActor.cs:492-505`), so the exception escapes the handler → the user-guardian default directive **restarts** the actor → the new incarnation's `_siteClients` dictionary is empty, but all old `site-client-*` actors are still alive at the system level. Every subsequent 60-second refresh (`PreStart`, `CentralCommunicationActor.cs:564-569`) then tries to create clients for **all** sites, collides on the reserved names, and crashes again. Failure scenario: an operator edits one site's NodeA address in the Central UI → central loses ClusterClient routing to *every* site (all `SiteEnvelope`s dropped at `HandleSiteEnvelope`, `CentralCommunicationActor.cs:398-413`) until the central process is restarted. This is the exact path the design doc promises works: "ClusterClient instances are recreated when contact points change" (`Component-Communication.md:238`).
Secondary issue on the same lines: `siteId` is interpolated into an actor name unvalidated. A `SiteIdentifier` containing `/`, whitespace, or other non-path characters throws `InvalidActorNameException` on first creation with the same restart-loop consequence — contrast with `SiteStreamGrpcServer.SubscribeInstance`, which validates `correlation_id` with `ActorPath.IsValidPathElement` for precisely this reason (`SiteStreamGrpcServer.cs:229-234`).
Why tests miss it: `CentralCommunicationActorTests` substitutes `ISiteClientFactory` with an NSubstitute mock returning TestProbes (`tests/.../CentralCommunicationActorTests.cs:42-51`) — the real naming/lifecycle machinery is never exercised anywhere.
Fix shape: unique names (`site-client-{siteId}-{seq}` via a counter, like `SiteStreamGrpcServer._actorCounter`), or watch `Terminated` and defer recreation, plus sanitize/validate the site id, plus one test using `DefaultSiteClientFactory`.
### [High] Standby site node runs the full S&F delivery pipeline — duplicate delivery is systematic, not a failover residue
The design says the standby only *applies* replicated operations and "on failover, the new active node resumes delivery" (`Component-StoreAndForward.md:77-80`). The code starts delivery on **both** nodes:
- `AkkaHostedService.cs:981` — `await storeAndForwardService.StartAsync()` runs in `RegisterSiteActorsAsync` on **every** site node, starting the retry timer (`StoreAndForwardService.cs:337-341`).
- `AkkaHostedService.cs:989, 998, 1033` — all three delivery handlers (ExternalSystem, CachedDbWrite, Notification) are registered unconditionally on every node.
- `SiteReplicationActor.cs:280-291` — replicated `Add` operations are applied to the standby's SQLite, so the standby's sweep has real Pending rows to deliver.
There is no active-node/leader gate anywhere in `StoreAndForwardService.RetryPendingMessagesAsync` (`StoreAndForwardService.cs:563-589`) or in the delivery handlers (`ExternalSystemClient.DeliverBufferedAsync`, `ExternalSystemGateway/ExternalSystemClient.cs:173-224` — checked, no gate). Consequences:
1. **Duplicate deliveries in steady operation.** Both nodes' sweeps become due for the same message at the same wall-clock moment (the replicated row carries the same `last_attempt_at`). Whichever delivers first replicates a `Remove`, but if both attempts are in flight concurrently — a window the size of the delivery duration (an HTTP call, often ≥ hundreds of ms) plus replication latency — the target receives the call **twice**. Over a long outage with many buffered messages, duplicates recur every recovery. Notifications survive this (central dedups on `NotificationId` insert-if-not-exists), but `ExternalSystem` calls and `CachedDbWrite`s do not — "idempotency is the caller's responsibility" was scoped to *rare failover* duplicates, not steady-state ones.
2. **Doubled retry traffic** against an already-struggling target during every outage.
3. Both nodes emit `CachedCallTelemetry` attempts for the same operation with independent retry counts, muddying the central audit picture.
Fix shape: gate the sweep (and ideally the queue-depth gauge registration) behind the same leader-check used by `SiteCommunicationActor.DefaultIsActiveCheck` (`SiteCommunicationActor.cs:498-507`), re-evaluated per sweep tick so failover resumes delivery automatically.
### [High] One debug session's gRPC failover disposes the shared channel under every other session to the same site
`SiteStreamGrpcClientFactory` caches **one** `SiteStreamGrpcClient` per site, keyed by site id, replacing (and disposing) it whenever a caller asks for a different endpoint (`SiteStreamGrpcClientFactory.cs:52-78`). `DebugStreamBridgeActor` holds **per-session** node preference and flips endpoint on every stream error (`DebugStreamBridgeActor.cs:416, 468-483`). With two concurrent debug sessions to the same site:
- Session A (on NodeA) hits an error and flips to NodeB → `GetOrCreate(site, NodeB)` **disposes** the NodeA-bound client. `SiteStreamGrpcClient.ReleaseResources` cancels **every** registered subscription CTS and disposes the channel (`SiteStreamGrpcClient.cs:314-324`) — killing session B's healthy stream.
- Session B's `SubscribeAsync` observes the cancellation as an error → B flips to NodeA → disposes the NodeB client → kills A's brand-new stream. The two sessions ping-pong, burning both sessions' 3-retry budgets (`DebugStreamBridgeActor.cs:45`) and terminating both.
Also, `CleanupGrpc` (`DebugStreamBridgeActor.cs:486-495`) calls `GetOrCreate` during teardown — if the cached client is bound to the other endpoint, cleanup *creates a fresh channel* (and disposes the cached one) merely to call `Unsubscribe`, with the same collateral effect.
Fix shape: key the cache by `(siteId, endpoint)` (both node channels can coexist), or reference-count clients, or give each bridge actor its own client instance. The single-flip failover for a *site-wide* address change can stay in the factory; per-session failover must not mutate shared state.
### [Medium] Replication operations can be applied out of order
`ReplicationService.FireAndForget` wraps each operation in its own `Task.Run` (`ReplicationService.cs:134-149`). Two operations for the same message issued in quick succession (Add then Remove; Park then operator Requeue) are handed to the thread pool as independent tasks — the pool does not preserve ordering, so the `replicationActor.Tell(...)` calls (handler wired at `AkkaHostedService.cs:897-901`) can happen inverted. Akka's per-sender/receiver ordering guarantee only applies *after* the Tells, so the standby can apply `Remove` (no-op, row absent) then `Add` → an orphan Pending row that the ungated standby sweep (finding above) or a failover will re-deliver. The `Task.Run` buys nothing — the handler is a non-blocking Tell returning `Task.CompletedTask`; invoking it inline (with a try/catch) preserves ordering by construction.
### [Medium] Replication failures are logged at Debug — a dead replication channel is invisible
`ReplicationService.cs:144-147` logs a failed replication at `LogDebug`. If the handler starts failing systematically (peer unreachable in a way `SendToPeer` doesn't detect, serialization fault of `StoreAndForwardMessage`), the standby's buffer silently diverges and the next failover loses the entire un-replicated delta — with zero operator-visible signal at default log levels and no counter metric. `SiteReplicationActor.SendToPeer`'s no-peer drop is also Debug (`SiteReplicationActor.cs:139-149`). At minimum Warning + a telemetry counter; the repo already has `ScadaBridgeTelemetry` plumbing for exactly this kind of gauge.
### [Medium] Park/Requeue replication is a blind UPDATE — lost on a standby missing the row
`ApplyReplicatedOperationAsync` applies `Park` and `Requeue` via `UpdateMessageAsync` (`ReplicationService.cs:121-130`), which is `UPDATE ... WHERE id = @id` (`StoreAndForwardStorage.cs:210-232`) — zero rows affected is silent. If the original `Add` was lost (fire-and-forget), the park state never materializes on the standby: after failover the message is simply *gone* from the parked view (operator can't retry/discard it) even though the full message payload was in hand in the replication operation. An upsert (INSERT OR REPLACE) for Park/Requeue would self-heal the lost-Add case for free.
### [Medium] Notifications park after ~25 minutes of central unreachability, and the "no limit" escape hatch is unused
`Notify.Send` enqueues with no `maxRetries` (`ScriptRuntimeContext.cs:2197-2203`) → `DefaultMaxRetries = 50` (`StoreAndForwardOptions.cs:21`) at the 30 s default interval = ~25 minutes of central outage before a notification parks and requires per-message operator action. `Component-StoreAndForward.md:52` explicitly documents `maxRetries: 0` as the escape hatch for callers that "genuinely require unbounded retry" — but the *only* notification producer doesn't use it, and no host config raises the cap for the Notification category specifically. A central maintenance window longer than 25 minutes strands every notification raised during it. Compounding: the serial sweep (Performance #1) makes effective intervals much longer than 30 s under backlog, but retries are counted per sweep-attempt, so the wall-clock park horizon is at least honest — the operational cliff is the manual un-parking.
### [Medium] Corrupt buffered notification is discarded and reported as delivered
`NotificationForwarder.DeliverAsync` returns `true` (→ engine removes the row) when the payload fails to deserialize (`NotificationForwarder.cs:79-87`). The message is permanently lost with only a site-log Warning: no parked row, no central `Notifications` row, no audit row. Returning `false` (permanent failure → **park**) would preserve the payload for forensics at zero protocol cost — parking is exactly the designed home for undeliverable-but-preserved messages.
### [Low] Central heartbeat state is not replicated to the peer central node
`HandleSiteHealthReport` fans reports to the peer via DistributedPubSub (`CentralCommunicationActor.cs:360-373`), but `HandleHeartbeat` marks only the local aggregator (`CentralCommunicationActor.cs:349-353`). ClusterClient delivers each site's heartbeat to one central node; the other's aggregator sees none. After a central failover the new active node briefly shows all sites offline until fresh heartbeats arrive (≤ heartbeat interval + offline threshold). Minor, but inconsistent with the care taken for health reports.
### [Low] `SubscribeInstance` concurrency-limit check is check-then-act
`SiteStreamGrpcServer.cs:243-254`: `_activeStreams.Count >= _maxConcurrentStreams` then insert — two concurrent subscribes at the boundary both pass. Bounded overshoot only; not worth locking, worth a comment.
**Disposition counts:** 22 Fixed (verified) · 2 Partially fixed (serializer pin-only by design; leader-vs-oldest helper shipped but not swapped into all call sites) · 2 Deferred/accepted (U1 dual transports, U7 session locality) · 0 Not fixed · 0 Regressed. Round-1 Critical **C2/C1 (ClusterClient recreate) specially verified: genuinely fixed**, with real-collaborator tests exactly where round 1 said the blind spot was.
---
## Findings — Performance
## NEW Findings — Store-and-Forward & Replication
### [High] The retry sweep is serial, unbounded, and single-laned across categories — it collapses under backlog
Three compounding design choices:
### [Critical] N1 — Anti-entropy resync authority uses the *leader* predicate while delivery uses *oldest-Up*: after a routine rolling restart the stale rejoining node wipes the delivering node's live buffer
- `GetMessagesForRetryAsync` has **no LIMIT** — it loads every due row into memory each sweep (`StoreAndForwardStorage.cs:183-203`).
- `RetryPendingMessagesAsync` delivers strictly one-at-a-time (`StoreAndForwardService.cs:571-579`) while `_retryInProgress` blocks the next sweep (`StoreAndForwardService.cs:566`).
- Every notification forward attempt against a down central costs a full 30 s Ask timeout (`NotificationForwardTimeout`, `CommunicationOptions.cs:36`; `NotificationForwarder.cs:93-95`).
The delivery gate installed by PLAN-02 T4 correctly uses PLAN-01's singleton-host helper: `AkkaHostedService.cs:881` → `AkkaClusterNodeProvider.SelfIsPrimary` → `ClusterActivityEvaluator.SelfIsOldest` (`ClusterActivityEvaluator.cs:20-31`, up-number/`IsOlderThan` based — matches where cluster singletons actually run). But the **peer-join resync roles** added in the same plan use a different predicate: `SiteReplicationActor.DefaultIsActive` (`SiteReplicationActor.cs:190-198`) is **leader + Up**, with a comment calling itself the "swap point for plan 01's shared helper" that was never swapped.
Failure scenario: central is down for an hour and scripts have buffered 200 notifications. One sweep = 200 × 30 s ≈ 100 minutes of serial timeouts — during which no `ExternalSystem` or `CachedDbWrite` retries happen either (single lane, head-of-line blocking across categories and targets: one slow external system delays notification forwarding to a *healthy* central and vice versa). There is no per-target short-circuit — N messages to the same dead target incur N consecutive timeouts per sweep. The queue drains far slower than it fills; retry counts tick toward parking on a wildly distorted cadence.
Akka.NET's cluster leader is the **lowest-address** reachable Up member — not the oldest. The two predicates diverge *persistently* (not just transiently) whenever the lower-address node is not the oldest, which is exactly the state produced by the most routine operation there is: a rolling restart (or crash + rejoin) of the lower-address site node. Sequence, with `node-a` (lower address) restarted while `node-b` keeps running:
Fix shape: `LIMIT` on the due query; group by (category, target) and short-circuit remaining messages for a target after its first transient failure in a sweep; optionally parallelize across targets with a small cap. Any one of the three helps materially.
1. `node-b` is now oldest → `SelfIsPrimary` true → node-b runs the delivery sweep and owns the live buffer. `node-a` rejoins with a fresh (higher) up-number but becomes **leader** by address ordering.
2. `node-b` observes `MemberUp(node-a)` → `TryTrackPeer` → `OnPeerTracked` (`SiteReplicationActor.cs:175-181`): `SafeIsActive()` is **false** on node-b (it is not leader) → the *delivering* node sends `RequestSfBufferResync` to the stale peer.
3. `node-a` handles it (`HandleRequestSfBufferResync`, 398-411): `SafeIsActive()` is **true** on node-a (leader+Up) → it answers with a snapshot of its **stale, pre-outage buffer**.
4. `node-b` handles the snapshot (`HandleSfBufferSnapshot`, 420-444): `SafeIsActive()` false → it calls `ReplaceAllAsync` — the method whose own doc says "**Never call on an active node** — it discards every in-flight row" (`StoreAndForwardStorage.cs:281-283`) — **on the node that is actively delivering**.
### [Medium] No explicit Akka serializer configuration — every cross-cluster contract rides default reflective JSON
`BuildHocon` (`AkkaHostedService.cs:214-271`) configures remoting, SBR, and failure detectors, but **no** `akka.actor.serializers` / `serialization-bindings`. All command/control and telemetry traffic — deployment configs, `IngestAuditEventsCommand` batches (entities converted *from* compact proto DTOs back to POCOs before the Akka hop, `ClusterClientSiteAuditClient.cs:72-85`), `ReplicationOperation` with full message payloads — is serialized by Akka.NET's default Newtonsoft JSON serializer with embedded CLR type manifests. Costs: (a) hot-path CPU/allocation and payload bloat on the highest-volume channel (audit telemetry); (b) the documented "additive-only message contract evolution" rule (`CLAUDE.md`, Commons conventions) is only half the story — wire compatibility is also coupled to *type and namespace identity*, so a rename/move of a Commons message type breaks rolling cross-version central↔site communication with no test to catch it. `MessageContractTests` asserts correlation-id presence only (`tests/.../MessageContractTests.cs`), not serialized round-trip stability. Ironic detail: the telemetry path builds efficient proto DTOs, then converts them back to POCOs specifically to cross the ClusterClient hop.
Consequences: every message buffered on node-b during and since node-a's outage is **deleted** (notifications enqueued with `maxRetries: 0` are silently gone — the one category the design promises is never lost short of payload corruption), and rows node-b already delivered are **resurrected** from node-a's stale copy → guaranteed duplicate deliveries on subsequent sweeps. `ReplicationEnabled` defaults to true (`StoreAndForwardOptions.cs:12`), so production 2-node sites are exposed by default.
### [Low] S&F SQLite: no WAL, no explicit busy_timeout, connection-per-operation
Every `StoreAndForwardStorage` method opens a fresh `SqliteConnection` (e.g. `StoreAndForwardStorage.cs:137-139, 185-186`) with a bare `Data Source=` connection string (`StoreAndForward/ServiceCollectionExtensions.cs:22-24`) — default rollback-journal mode. Concurrent writers exist by design: script-thread enqueues, the sweep, replication applies (standby), and gRPC `PullSiteCalls`/parked-query reads. WAL + `busy_timeout` pragma would remove writer/reader blocking; Microsoft.Data.Sqlite's pooling makes connection-per-op tolerable but the pragmas are one line each in `InitializeAsync`.
Fix shape: swap both remaining leader-based predicates (`SiteReplicationActor.cs:190-198`, `SiteCommunicationActor.cs:517-526`) to the shared oldest-Up helper so *one* definition of "active" governs delivery, resync-request, resync-answer, and heartbeat IsActive; add a two-node rig test that rolls the lower-address node and asserts the surviving node's buffer is untouched. (Also consider a belt-and-braces guard: refuse `ReplaceAllAsync` while this node's *delivery gate* reports active.)
### [Low] Due-check does per-row `julianday()` string parsing on ISO-8601 timestamps
`StoreAndForwardStorage.cs:195-197` computes due-ness with `(julianday('now') - julianday(last_attempt_at)) * 86400000` against `"O"`-format strings with 7-digit fractional seconds and `+00:00` offsets — relying on SQLite's lenient datetime parser, non-indexable, and re-parsed for every Pending row on every 10 s sweep. Storing epoch milliseconds would be cheaper, indexable, and immune to format drift.
### [High] N2 — The resync snapshot rides ONE Akka remoting message with no frame-size headroom: it is silently undeliverable for any realistic backlog
### [Low] Audit-observer notification is awaited inline in the sweep
`NotifyCachedCallObserverAsync` is awaited per attempt inside `RetryMessageAsync` (`StoreAndForwardService.cs:620-626, 688-694, 711-717`). It's exception-isolated (good), but not latency-isolated: a slow observer (SQLite audit write) stretches the already-serial sweep. Fire-and-forget with isolation, or post to a channel, would decouple it.
`HandleRequestSfBufferResync` pipes up to **10,000 full rows** (id + target + complete `payload_json` + provenance) into a single `SfBufferSnapshot` record and Tells it across Akka remoting (`SiteReplicationActor.cs:40, 407-410, 460`). `BuildHocon` sets no `akka.remote.dot-netty.tcp.maximum-frame-size` (grep across `src/` and `docker/` finds none), so the default **128,000 bytes** applies. That is ~12 bytes per row at the cap — even a few hundred typical cached-call payloads exceed it. An oversized remote message is dropped by the transport (an error in the sender's log; no exception surfaces to the `PipeTo`), the standby never receives the snapshot, nothing retries until the *next* peer-track event, and no telemetry counts it. Net effect: **the anti-entropy fix works only for near-empty buffers and silently fails in exactly the sustained-outage/backlog scenario it was built for** (U4's "standby down for an hour"). Ironically this also caps N1's blast radius — large stale snapshots can't be delivered either — but small-buffer wipes still go through.
Fix shape: chunk the snapshot (reuse the pull-style paging the audit reconciliation already has), or stream rows as a sequence of upsert ops bracketed by begin/end markers with a completion-driven delete-absent pass; count send failures. Also reconsider the 10k in-memory `List` on the active node.
### [Medium] N3 — `_sweepTask` handoff is clobbered by every timer tick, defeating the shutdown drain it exists for
Both the timer callback (`StoreAndForwardService.cs:396-400`) and `TriggerSweep` (664-668) unconditionally `Volatile.Write(ref _sweepTask, RetryPendingMessagesAsync())`. When a sweep is still in flight, the new call no-ops via the `_retryInProgress` CAS and returns a **completed** task — which overwrites `_sweepTask`. `StopAsync` (422-431) then awaits the no-op and proceeds with shutdown while the real sweep (one that has outlived a 10 s tick — i.e. precisely the backlog case the drain was added for) is still mid-delivery. Consequence is bounded (the sweep's own catch logs; storage is connection-per-op), but the "wait for the in-flight sweep" guarantee is illusory whenever it matters. Fix: only publish the task when the CAS is won (e.g. have `RetryPendingMessagesAsync` register itself), or CAS `_sweepTask` from completed→new.
### [Low] N4 — New sweep options missing from the eager validator
`StoreAndForwardOptionsValidator` (17-34) validates path/intervals/max-retries but not the two options PLAN-02 added: a negative `SweepBatchLimit` silently means "unlimited (legacy)" (`StoreAndForwardStorage.cs:392` checks `> 0`) and a non-positive `SweepTargetParallelism` is silently clamped to 1 (`StoreAndForwardService.cs:715`). Both deserve fail-fast messages per the §1.5 eager-validation convention the same release adopted elsewhere.
### [Low] N5 — Resync snapshot vs in-flight replication op race leaves orphan rows on the standby
A replicated `Remove` sent after the active node read its snapshot but before the snapshot send is ordered *before* the snapshot on the wire (same sender/receiver pair), so the stale snapshot re-adds the removed row on the standby → an orphan Pending row that a later failover re-delivers. Bounded and self-correcting at the next resync, and inherent to no-ack replication — worth a code comment on `HandleSfBufferSnapshot` (`SiteReplicationActor.cs:420-444`) so nobody "fixes" it into something worse.
---
## Findings — Conventions
## NEW Findings — Aggregated Live Alarm Stream (post-initiative, first review)
### [Medium] Spec-vs-code drift in the S&F component doc
Two material divergences from `Component-StoreAndForward.md`:
1. **Standby-passive model** (doc lines 77-80) vs. the ungated standby delivery pipeline (Stability #2). The doc's "a few duplicate deliveries [on failover]... acceptable trade-off" framing materially understates actual behavior.
2. **`Notify.Send` latency**: the doc/CLAUDE.md say Send "returns a NotificationId status handle immediately", but `EnqueueAsync`'s immediate-delivery attempt (`StoreAndForwardService.cs:500-529`) runs the `NotificationForwarder` Ask inline on the script's await — when central is down, every `Notify.Send` blocks its script execution for up to `NotificationForwardTimeout` (30 s) before buffering. Either document the worst case or make the Notification category enqueue with `attemptImmediateDelivery: false` + an immediate sweep kick.
The feature is well-shaped overall: one gRPC stream per site (not per circuit), reference-counted lifecycle with linger, stream-first seed with ordered delta buffering and per-key dedup (correctly copied from `DebugStreamBridgeActor`, including the inclusive-boundary rule), authority-split with the poll fallback (`IsLive` false until first seed publishes — `SiteAlarmLiveCacheService.cs:131-136, 306-323`), reconcile as the drop-detection backstop (`SiteAlarmAggregatorActor.cs:276-304` rebuilds authoritatively so disappeared rows are dropped), self-healing on both start failure (bounded retry timer, `SiteAlarmLiveCacheService.cs:249-275`) and stream give-up (reconcile tick reopens, `SiteAlarmAggregatorActor.cs:224-236`), balanced telemetry gauges in PreStart/PostStop (188, 212), viewer-cap fail-safe returning a no-op handle instead of throwing on a Blazor render path (100-107), and eager options validation (`CommunicationOptionsValidator.cs:69-87`). Findings:
### [Low] `StreamRelayActor.WriteToChannel` has a dead branch and silent loss
`TryWrite` on a bounded channel in `DropOldest` mode never returns `false` — it evicts the oldest item and succeeds. The "Channel full, dropping event" warning (`StreamRelayActor.cs:105-111`) can never fire, while the *actual* loss (the evicted oldest event) is completely unlogged. Lossy-under-backpressure is fine for debug view per spec (`Component-Communication.md:59`), but the observability is inverted: dead warning, silent real drop. Count evictions (channel `ItemDropped` callback via `CreateBounded(options, itemDropped)`) or log periodically.
### [Medium] N6 — No delta coalescing: every applied alarm delta copies the whole site cache and fans out to every viewer circuit
### [Low] Duplicated Ask-timeout constants and divergent fault semantics between the two ingest transports
`CentralCommunicationActor.DefaultAuditIngestAskTimeout` copies `SiteStreamGrpcServer.AuditIngestAskTimeout` by hand (`CentralCommunicationActor.cs:114-124` vs `SiteStreamGrpcServer.cs:45`), and the two handlers deliberately differ on fault behavior (propagate `Status.Failure` vs swallow-to-empty-ack — documented at `CentralCommunicationActor.cs:104-111`). Both are acknowledged in comments, but two transports for the same operation with different failure semantics and copy-pasted tuning is drift waiting to happen (see Underdeveloped, dual transports).
`HandleLiveDelta` → `Publish()` materialises a fresh `_cache.Values.ToList()` **per delta** (`SiteAlarmAggregatorActor.cs:365-367, 391-403`), and `OnPublish` invokes every subscriber's `onChanged` (`SiteAlarmLiveCacheService.cs:386-411`), each of which does a Blazor `InvokeAsync` + `GetCurrentAlarms` + re-render (`AlarmSummary.razor:369-388`). An alarm flood — the canonical SCADA storm scenario, hundreds of condition transitions per second on a large site — becomes O(cacheSize) list copies per transition on the actor thread plus a render queue entry per circuit per transition. The seed path already batches; the live path should too (e.g. a ~250 ms publish-coalescing timer, or publish-on-tick-if-dirty). Everything stays correct (last write wins), so this is performance, not correctness.
### [Low] Heartbeat `IsActive` uses leader-check while singletons follow oldest-node
`SiteCommunicationActor.DefaultIsActiveCheck` reports active = cluster leader (`SiteCommunicationActor.cs:498-507`). Akka cluster singletons run on the *oldest* qualifying node; leader and oldest usually coincide in a 2-node stable cluster but can diverge transiently. The comment says it deliberately mirrors `ActiveNodeGate`, so it's a consistent repo convention — but the convention itself can misreport which node hosts the DeploymentManager singleton. Worth one shared helper with an oldest-member check.
### [Low] N7 — Aggregator lifecycle nits (bundled)
### [Low] `TransportHeartbeatInterval` does double duty
The Akka.Remote transport failure-detector interval (`BuildHocon`, `AkkaHostedService.cs:246-249`) is also the *application-level* site→central heartbeat cadence (`SiteCommunicationActor.cs:421-425`). These are unrelated concerns (spec treats them separately: `Component-Communication.md:268-274` vs health reporting); tuning one silently retunes the other.
1. **Reconnect re-seed can be skipped:** the post-failover-flip `StartFanout(isInitial: false)` (`SiteAlarmAggregatorActor.cs:472`) no-ops if a reconcile fan-out is already in flight (247-253) — deltas missed between stream death and the in-flight snapshot's read-time stay stale until the *next* reconcile (up to 60 s), despite the "never silently serve stale" comment.
2. **No stream-generation stamp on `GrpcAlarmStreamError`:** a late error raced out of the *previous* cancelled stream (the `RpcException(Cancelled)` filter at `SiteStreamGrpcClient.cs:256-258` covers the normal path, but a genuine socket fault can beat the cancel) is indistinguishable from a failure of the current stream and burns retry budget / double-flips. The window is small; a monotonic stream-id in the error message would close it.
3. **Static mutable test seams:** `ReconnectDelay` / `StabilityWindow` are `internal static` mutable properties (`SiteAlarmAggregatorActor.cs:61-69`) — process-global state that parallel test classes can trample; instance/ctor parameters would match how the rest of the file injects tuning.
### Positives worth recording
- Tell/Ask discipline matches the repo rule everywhere: Ask only at system boundaries (`CommunicationService`, forwarder, telemetry clients), Tell/Forward on hot paths with Sender preservation for reply routing (`SiteCommunicationActor.cs:270-388`, `CentralCommunicationActor.cs:398-413`).
- Correlation IDs on every request/response contract, verified by tests.
- The at-least-once notification handoff is textbook: ack-after-persist at central, script-generated `NotificationId` as buffered-row id *and* idempotency key, re-stamped provenance at the forwarder (`NotificationForwarder.cs:123-160`).
- The sweep's CAS updates (`UpdateMessageIfStatusAsync`, `StoreAndForwardStorage.cs:243-269`) correctly close the operator-retry/discard vs sweep race, replicated with correct captured state (`StoreAndForwardService.cs:861-892`).
- gRPC server stream lifecycle is thorough: readiness + shutdown gating, correlation-id validation, duplicate-stream replacement, session-lifetime cap, leak-proofed Subscribe failure cleanup, own-entry-only removal (`SiteStreamGrpcServer.cs:211-327`).
- Stream-first debug subscription with buffered gap-window replay and per-entity timestamp dedup (`DebugStreamBridgeActor.cs:134-176, 286-329`) is a correct solution to the snapshot/stream ordering problem, including the flapping-stream retry-budget stability window.
- Additive proto evolution is real and tested (`ProtoContractTests`, `ProtoRoundtripTests`), with the manual regen process documented.
### [Low] N8 — "Lives only on the active central node" is aspirational, and deleted sites leak idle channels
`SetActorSystem` is wired on every central node (`AkkaHostedService.cs:430`), so browsing the standby node directly (port 9002, documented for diagnostics) starts a second, fully functional aggregator + gRPC stream there — harmless (read-only) but contradicting the `[PERM]` doc claim in `ISiteAlarmLiveCache.cs:13-17`. Separately, `SiteStreamGrpcClientFactory.RemoveSiteAsync` — designed as "the disposal path when a site record is deleted" (`SiteStreamGrpcClientFactory.cs:88-102`) — has **no production caller** (grep: tests only), so a deleted site's channels (and an aggregator kept alive by an open viewer, which then reconciles to an empty snapshot forever) persist until process restart. Bounded, but wire the deletion path or document the gap.
### [Low] N9 — A site with only one configured gRPC endpoint can never go live
`ResolveSiteAsync` requires **both** `GrpcNodeAAddress` and `GrpcNodeBAddress` (`SiteAlarmLiveCacheService.cs:294-300`); a single-node site (or one with only NodeA populated) silently never starts an aggregator and polls forever, with only a log Warning. The aggregator itself would work fine flipping between one address; accept a single endpoint and disable the flip.
---
## Underdeveloped Areas
## What's genuinely good (round 2)
1. **Dual transports for the same ingest operations.** `IngestAuditEvents`/`IngestCachedTelemetry` exist both as gRPC unary RPCs (`SiteStreamGrpcServer.cs:330-433`) and as ClusterClient commands (`CentralCommunicationActor.cs:262-305`); production push uses ClusterClient (`ClusterClientSiteAuditClient`), leaving the gRPC ingest surface production-idle (doc admits: "the gRPC-receiving counterpart", `Component-Communication.md:87`). Two code paths, two fault semantics, double the test/maintenance surface — consolidation is pending and should be scheduled.
2. **No real-collaborator test for the ClusterClient cache lifecycle.** `DefaultSiteClientFactory` is untested; address-change recreation is only tested through mocks — precisely where the Critical finding lives.
3. **No test or design treatment of standby-node sweep behavior.** Neither the S&F tests nor an integration test asserts that only the active node delivers; the docker-cluster smoke topology would expose it with a two-node site and a delayed target.
4. **Replication has no anti-entropy/resync.** A standby down for an hour rejoins and applies only *new* operations; the missed delta diverges forever (until rows drain naturally). The doc covers only the "last few operations" loss case. A periodic checksum or full-row reconciliation (the pattern already exists for audit rows via `PullAuditEvents`) is absent for the S&F buffer itself.
5. **Parked rows have unbounded retention.** By design parked messages persist until operator action (`Component-StoreAndForward.md:110-116`), but there is no aging alert, cap, or archival — a forgotten site accumulates parked rows indefinitely with only a health-metric count.
6. **Reconciliation cursor edge.** `PullAuditEvents`/`PullSiteCalls` batch continuation relies on central advancing a `since_utc` cursor from the last returned timestamp (`SiteStreamGrpcServer.cs:478-484, 563-572`); rows sharing a boundary timestamp are protected only by EventId dedup / upsert-on-newer at central — fine, but the site-side read contract (`>` vs `>=`) isn't pinned by a test in this project.
7. **`DebugStreamService._sessions` on central failover** — sessions are process-local with no persistence (documented as acceptable: engineer re-establishes), but the SignalR consumer contract (`OnStreamTerminated`) is the only notification; a central restart drops sessions with no signal at all. Known/accepted, listed for completeness.
- **The fix quality is high and test-backed.** Every round-1 fix landed with the failing-test-first shape the plan prescribed; the two places round 1 called out as structurally untested (real `DefaultSiteClientFactory`, standby sweep behavior) now have real-collaborator tests. The wire-format pin tests even caught a real Newtonsoft deserialization gotcha (ctor optional-param defaults) during PLAN-08.
- **The sweep redesign is careful.** Skipped lane rows keep `RetryCount`/`LastAttemptAt` untouched so the park horizon reflects real attempts; `GroupBy` stability preserves per-target FIFO; the observer pump preserves per-operation event order while latency-isolating the sweep; the gate is fail-safe (throwing → standby) and re-evaluated per tick.
- **Storage hygiene done right:** additive `last_attempt_at_ms` with one-time guarded backfill and a covering `(status, due)` index; shared `InsertMessageSql` + `BindMessageParameters` so column lists can't drift; defensive `Guid.TryParse` on read so one corrupt row can't abort a sweep.
- **The live alarm stream reuses instead of reinventing:** `SubscribeSite` rides the same hardened `RunSubscriptionStreamAsync` pipeline (readiness/shutdown gates, correlation-id validation, duplicate replacement, DropOldest with counted evictions, leak-proof subscribe failure cleanup — `SiteStreamGrpcServer.cs:253-385`); the client shares the proto→domain mapping; the actor copies the proven seed-then-stream/buffer/dedup/stability-window design from `DebugStreamBridgeActor` including its NUL-delimited three-part alarm key; the proto change is purely additive (new RPC + new request message).
- **Failure-mode thinking is visible everywhere in the new code:** seed fan-out degrades per-instance rather than failing whole; a failed *initial* seed keeps `IsLive` false so the page keeps polling; publish callbacks are exception-isolated outside the lock; the viewer cap rejects with a no-op disposable rather than throwing into a render.
- Docs were genuinely synchronized: `Component-StoreAndForward.md` (gate, resync, enqueue-only Send) and `Component-Communication.md` (U7 accepted-limitation) now match the code they describe.
---
## Prioritized Recommendations
## Severity tally — NEW findings only
1. **[Critical] Fix ClusterClient recreation** (`CentralCommunicationActor.cs:514-521`, `DefaultSiteClientFactory`): unique per-incarnation actor names (suffix a counter) or Terminated-watch before re-create; validate/sanitize `siteId` for actor-name safety; wrap `Create` in the per-site try/catch; add a test using the real factory that edits a site's address twice.
2. **[High] Gate the S&F retry sweep to the active node** — check leader/oldest status at the top of `RetryPendingMessagesAsync` (or gate `StartAsync`'s timer), re-evaluated per tick so failover resumes delivery within one sweep interval. Add a two-node test.
3. **[High] Stop disposing shared gRPC clients on per-session failover** — key `SiteStreamGrpcClientFactory` by `(site, endpoint)` (allow both node channels concurrently), reserve dispose-and-replace for site removal/address edits driven by the address cache, not by individual bridge actors.
4. **[High] Bound and de-serialize the retry sweep** — `LIMIT` the due query, short-circuit per target after the first transient failure in a sweep, and consider a small parallelism cap across distinct targets; separately consider a much shorter forward timeout for the notification category or `attemptImmediateDelivery: false` on `Notify.Send` to unblock script threads.
5. **[Medium] Make replication trustworthy**: replace `Task.Run` fire-and-forget with an inline Tell (ordering), log failures at Warning with a counter metric, and make Park/Requeue applies upserts so a lost Add self-heals.
6. **[Medium] Pin the wire format**: configure an explicit serializer (or at least add serializer round-trip compatibility tests for the cross-cluster Commons contracts) so the additive-only rule is actually enforced end-to-end; long-term, stop converting proto DTOs back to POCOs for the ClusterClient hop.
7. **[Medium] Unstrand notifications**: pass `maxRetries: 0` (documented no-limit) or a category-specific high cap on the `Notify.Send` enqueue path, or auto-requeue parked Notification-category rows when central connectivity recovers.
8. **[Medium] Park (don't silently drop) corrupt notification payloads** in `NotificationForwarder.DeliverAsync` — return `false` instead of `true`.
9. **[Low] SQLite hygiene**: enable WAL + busy_timeout in `InitializeAsync`; migrate timestamps to epoch ms for the due-check.
10. **[Low] Observability polish**: count DropOldest evictions in `StreamRelayActor`; replicate heartbeat marks to the peer central node (or document the failover blind window); split the app heartbeat interval from the transport failure-detector setting.
| Severity | Count | Findings |
|---|---|---|
| Critical | 1 | N1 (resync predicate inversion wipes the delivering node's buffer) |
| High | 1 | N2 (resync snapshot exceeds Akka remoting frame size — silently undeliverable) |
| Medium | 2 | N3 (`_sweepTask` clobbering defeats shutdown drain), N6 (no live-delta coalescing) |
| Low | 5 | N4 (validator gaps), N5 (resync/op race), N7 (aggregator lifecycle nits), N8 (standby aggregator + channel leak on site delete), N9 (dual-endpoint requirement) |
Both the Critical and the High are confined to the **peer-join anti-entropy resync** (PLAN-02 T20/T21) — the one round-2 feature that shipped without a two-node integration test — and both are small, well-localized fixes: swap two predicates to the existing shared helper, and chunk one message.
archreview/03-site-runtime-dcl.md
+69 -130
View File
@@ -1,155 +1,94 @@
# Architecture Review 03 — Site Runtime & Data Connection Layer
# Architecture Review 03 — Site Runtime & Data Connection Layer (Round 2)
Reviewer domain: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime`, `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer`, `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging`, matched against `docs/requirements/Component-SiteRuntime.md`, `Component-DataConnectionLayer.md`, `Component-SiteEventLogging.md`, and `HighLevelReqs.md`. All findings are grounded in direct reads of the cited files.
## Scope
- **Site Runtime actors**: `DeploymentManagerActor` (1,720 LOC), `InstanceActor` (1,572), `ScriptActor` (594), `ScriptExecutionActor` (305), `AlarmActor` (780), `AlarmExecutionActor`, `NativeAlarmActor` (448), `CertStoreActor` (189), `SiteReplicationActor` (292), `SiteReconciliationActor`.
- **Script subsystem**: `ScriptCompilationService`, `SharedScriptLibrary`, `ScriptExecutionScheduler`, `ScriptRuntimeContext` (2,384 — skimmed for Ask/blocking patterns), trigger expressions.
- **Streaming**: `SiteStreamManager` (BroadcastHub + per-subscriber buffers).
- **Persistence**: `SiteStorageService` (site SQLite: deployed configs, static overrides, artifacts, `native_alarm_state`), `OperationTrackingStore` (not deep-read).
- **DCL**: `DataConnectionManagerActor` (345), `DataConnectionActor` (2,000), `OpcUaDataConnection` (401), `RealOpcUaClient` (1,313 — lifecycle sections), `MxGatewayDataConnection` (297 — connect path), alarm mappers, browse/search/verify paths.
- **Site Event Logging**: `SiteEventLogger` (326), purge/query services.
- **Tests**: inventoried `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests`, `...DataConnectionLayer.Tests`, `...SiteEventLogging.Tests`.
## Maturity Verdict
This is a mature, unusually carefully engineered subsystem — the Become/Stash state machine, in-flight subscribe bookkeeping, adapter-generation guards, seed-read retry, redeploy termination-watch buffering, and per-source native-alarm cap/eviction all show evidence of multiple hardening passes, and the code matches the design docs closely with extensive rationale comments and broad targeted test coverage. The residual risk is concentrated in three places: **adapter lifecycle on reconnect** (the previous OPC UA / MxGateway client object is silently abandoned on every `ConnectAsync`, leaking sessions and leaving a stale `ConnectionLost` handler that can flap a healthy connection), **script containment** (the execution timeout is cooperative-only, so a hung blocking script permanently consumes one of the 8 dedicated script threads; trigger expressions block actor threads on the default dispatcher), and **deployment honesty** (a site-side script compile failure does not reject the deployment as the spec requires — the instance starts with the script silently missing and central is told Success). None of these break the happy path; all of them are the kind of defect that surfaces weeks into production under flaky networks or a bad script.
**Date:** 2026-07-12
**Scope:** `src/ZB.MOM.WW.ScadaBridge.SiteRuntime` (#3: DeploymentManager singleton, Instance/Script/Alarm/NativeAlarm actors, script execution scheduler, site-wide Akka stream), `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer` (#4: OPC UA + MxGateway adapters, connection actors, alarm subscription seam, CertStoreActor), `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging` (#12); matched against `docs/requirements/Component-SiteRuntime.md`, `Component-DataConnectionLayer.md`, `Component-SiteEventLogging.md`.
**Method:** Re-verified every round-1 finding against current source (HEAD `8c888f13`) with file:line evidence — plan claims were not trusted; each fix was read in the code. Fresh sweep over `git diff b910f5eb..HEAD` scoped to the three projects (38 files, +2,205/−640) covering the PLAN-03 fix commits, the PLAN-08 options-validator/notification-excision commits, and the 2026-07-10 aggregated-live-alarm-stream site-side piece (`SubscribeSiteAlarms`).
**Round-1 vs round-2 verdict:** Round 1 found a mature subsystem with three concentrated risks (adapter reconnect lifecycle, script containment, deployment honesty); round 2 confirms all three are genuinely fixed in code — 26 of 28 actionable findings verified fixed, 2 deferred with sound rationale — leaving only one Medium residual (a stale MxGateway event-loop fault can still flap a fresh connection through the generic catch) and a handful of Low polish items.
---
## Findings — Stability
## Round-1 Finding Disposition
### S1. [High] Reconnect abandons the previous protocol client undisposed, and its stale `ConnectionLost` handler can flap a healthy connection
`OpcUaDataConnection.ConnectAsync` (`src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/OpcUaDataConnection.cs:99-101`) does:
Every disposition below was verified by direct read of the cited current source, not the plan's claims.
```csharp
_client = _clientFactory.Create();
_client.ConnectionLost += OnClientConnectionLost;
await _client.ConnectAsync(...);
```
| Finding | R1 severity | Disposition | Evidence (file:line) |
|---|---|---|---|
| S1 — reconnect abandons previous client; stale `ConnectionLost` flaps healthy connection | High | **Fixed (verified)** — with one Medium residual on the MxGateway fault path (new finding N1) | `OpcUaDataConnection.cs:99-118` detaches `ConnectionLost`, fire-and-forget-disposes the previous client with fault logging, and `StopHeartbeatMonitor()` (:107) kills the old `StaleTagMonitor`; `MxGatewayDataConnection.cs:71-91` cancels+disposes the previous event loop/client before the guard reset. Tests exist: `OpcUaDataConnectionTests.cs` (`Reconnect_DisposesAndDetachesPreviousClient`, `StaleClientConnectionLost_…`), `Adapters/MxGatewayDataConnectionReconnectTests.cs` |
| S2 — cooperative-only timeout; no stuck-thread observability | High | **Fixed (verified)** (observability shipped; ADO.NET command-timeout sub-suggestion deferred with rationale — scripts own their commands, 30 s default) | Gauges: `ScriptExecutionScheduler.cs:29,80-113` (`QueueDepth`/`BusyThreadCount`/`OldestBusyAge` via per-worker `_busySinceTicks`, volatile writes in `WorkerLoop` :121-128). Watchdog: `ScriptExecutionActor.cs:131-154` names the script whose thread never came back after timeout+`StuckScriptGraceMs`, logs + site event; `completed` flipped in the finally (:326). Reporter: `ScriptSchedulerStatsReporter.cs:71-74`, registered `ServiceCollectionExtensions.cs:76-79`; health surface `SiteHealthCollector.cs:151-162,260-262`; additive `SiteHealthReport.cs:65-75` |
| S3 — site compile failure doesn't reject deployment (spec drift) | High | **Fixed (verified)** — deliberate deviation: gate is synchronous on the singleton, not off-thread (see N4/N5) | `DeployCompileValidator.cs:37-116` compiles every script, alarm on-trigger script, and Expression trigger (malformed JSON is itself an error, fronting the poison-config case); gate at `DeploymentManagerActor.cs:508-537` replies `DeploymentStatus.Failed` with the collected errors and returns before any actor creation or persistence. Synchronous-on-mailbox rationale documented in code (:510-521): off-thread piping would reorder deploys vs. delete/disable and break the redeploy-supersede FIFO. Test: `DeploymentManagerActorTests.cs` (`Deploy_WithNonCompilingScript…`) |
| S4 — no retry on failed tag subscription; NativeAlarm lost-response gap | Medium | **Fixed (verified)** — stronger than asked: retry arms on *send*, so lost responses are covered too | `InstanceActor.cs:1047-1068` `SendTagSubscribe` arms a per-connection `TagSubscribeRetryIntervalMs` timer on every send with correlation-id supersede; `HandleSubscribeTagsResponse` (:1075-1099) cancels on Success, logs + emits a site event and leaves the timer armed on failure; timers cancelled in PostStop (:296-298). `NativeAlarmActor.cs:114,128,147,319` — response-timeout retry armed in `SendSubscribe`, cancelled on Success |
| S5 — startup SQLite load failure aborts silently, no retry | Medium | **Fixed (verified)** | `DeploymentManagerActor.cs:286-290,345-356,400-415` — failed load schedules `RetryStartupLoad` at a fixed interval (default 5 s, ctor-overridable), unlimited, with a startup-complete flag so a stale retry cannot re-run after success |
| S6 — dead child ref in `_instanceActors` + Success reported for dead instance | Medium | **Fixed (verified)** — redesigned beyond the plan: two-signal deploy join, not swallow-only | Every Instance Actor is watched (`DeploymentManagerActor.cs:1995`); unexpected `Terminated` evicts the dead ref (:618+). Deploy `Success` requires `InFlightDeploy.Persisted && Initialized` (:872-892, class :2093-2104) where `Initialized` is the new `InstanceActorInitialized` readiness message sent at the end of `InstanceActor.PreStart` (`InstanceActor.cs:280-286`, `Messages/InstanceActorInitialized.cs`); init-failure row rollback is deferred until the store commits (`_initFailedPendingRowRemoval`, :135,657,824) so it can never race the optimistic write. Design-doc capture: `Component-SiteRuntime.md:109` |
| S7 — background tasks read mutable `_adapter` field off-thread | Medium | **Fixed (verified)** | Adapter captured into a local on the actor thread before every background task: `DataConnectionActor.cs:704` (subscribe), `:1155` (write), `:1199` (trigger write), `:1638` (re-subscribe, symmetric with the pre-existing `reseedAdapter`) |
| S8 — site SQLite: no WAL/busy-timeout | Low | **Fixed (verified)** | `SiteStorageService.cs:23` (busy-timeout floor pinned in the normalized connection string), `:63-74` (`PRAGMA journal_mode=WAL` set once in `InitializeAsync`, warning logged on fallback) |
| S9 — fire-and-forget adapter dispose in `PostStop` unobserved | Low | **Fixed (verified)** | `DataConnectionActor.cs:232-237` — `ContinueWith(OnlyOnFaulted, log)` on the dispose task |
| S10 — alarm condition filter last-subscriber-wins, silent overwrite | Low | **Fixed (verified)** | `DataConnectionActor.cs:1840-1851` — warning logged when an incoming filter differs from the stored one; filter also parsed once into `_alarmSourceFilterPredicate` (:126) |
| P1 — trigger expressions block dispatcher threads per attribute change | High | **Fixed (verified)** — with one Low residual (new finding N2) | `ScriptActor.cs:281-334` and `AlarmActor.cs:520-577`: evaluation runs on `ScriptExecutionScheduler.Shared` against a point-in-time snapshot, result piped back as `ExpressionEvalResult`, edge state (`_lastExpressionResult`, WhileTrue timers) applied only on the actor thread, bursts coalesced (one in flight + one pending) |
| P2 — attribute-change fan-out broadcast to every child | Medium | **Fixed (verified)** | New `Scripts/TriggerRouting.cs` (three-way monitored/all-changes/none classification, fail-open on unparseable config); `InstanceActor.PublishAndNotifyChildren` (:1270-1289) routes to `_allChangeChildren` + `_childrenByMonitoredAttribute[attr]` only; `RouteChild` files children at creation (:1451-1469, wired at :1516/:1576) |
| P3 — DCL tag-value fan-out scans all instances per update | Medium | **Fixed (verified)** | `DataConnectionActor.cs:1745-1764` `IndexTag`/`UnindexTag` maintain the `_instancesByTag` reverse index at every subscription mutation site (incl. instance release, :1076); `HandleTagValueReceived` (:1778-1790) fans out O(subscribers-of-tag) |
| P4 — per-transition SQLite upserts on native-alarm mirror | Low | **Fixed (verified)** | `NativeAlarmActor.cs:57-70,115,469-476` — latest-transition-per-source coalescing map, batched flush on a 100 ms timer (`FlushPersistence`), best-effort final flush in PostStop (:139-143) |
| P5 — per-attribute script read is an Ask round-trip | Low | **Deferred (accepted)** | `ScriptRuntimeContext` unchanged on this path; plan rationale (spec-consistent, no profiling evidence, batch `GetAttributes` as follow-up) holds |
| P6 — Roslyn compiles inside `InstanceActor.PreStart` during staggered startup | Low | **Deferred (accepted, partially mitigated)** | Compiles still run in `PreStart` → `CreateChildActors` (`InstanceActor.cs:260-280,1472-1499`); the deploy-path gate (S3) validates earlier but PreStart recompiles — see new finding N4. Risk remains failover time-to-recover only |
| C1 — positive: Akka discipline | — | Still true; the new code keeps the discipline (PipeTo everywhere, captured `Self`, actor-confined state) | e.g. `ScriptActor.cs:289` captured `self`; `DataConnectionActor.cs:704` captured adapter |
| C2 — stale "unbounded channel" comment in SiteEventLogger | Low | **Fixed (verified)** | `SiteEventLogger.cs:21-22,69,215-216` — comments now correctly describe bounded/DropOldest semantics |
| C3 — spec self-contradiction on reconnect-interval scope | Low | **Fixed (verified)** | `Component-DataConnectionLayer.md:304` — "shared setting for all data connections (`ReconnectInterval` in the Shared Settings table)" |
| C4 — quality-only change not delivered to children vs. spec wording | Low | **Fixed (verified)** | `Component-DataConnectionLayer.md:303` — spec now states children are "deliberately NOT re-notified on a quality-only change" with the spurious-firing rationale; divergence is now a documented decision |
| C5 — shared scripts run Root scope, undocumented | Low | **Fixed (verified)** | `Component-SiteRuntime.md:344` — "Shared scripts always execute with Root scope (C5)… Callers needing module-scoped access must pass the values in as parameters" |
| C6 — `EvaluateCondition` fallback uses current-culture `ToString` | Low | **Fixed (verified)** | `ScriptActor.cs:513-516` — threshold rendered with `CultureInfo.InvariantCulture`, `StringComparison.Ordinal` |
| UA1 — cert-trust convergence after node downtime | Underdev. | **Fixed (verified)** — additive reconcile-on-join; removal-reconcile + cross-node list aggregation remain deferred to the documented central-persistence follow-up | `CertStoreActor.cs:139+` exports the trusted set (thumbprint+DER); `DeploymentManagerActor.cs:277-281,321-328` singleton subscribes to `ClusterEvent.MemberUp`; `:1296-1365` re-reads the local store and pushes each cert to the joiner's CertStoreActor (idempotent by thumbprint, additive union only, fire-and-forget with heal-on-next-join). Divergence direction is always singleton→joiner since trust commands flow through the singleton, so the additive push closes the real hole |
| UA2 — `ConfigFetchRetryCount` dead option | Underdev. | **Fixed (verified)** | `SiteRuntimeOptions.cs:68` (default 3), consumed at `SiteReplicationActor.cs:86` (floor 1) with a fixed 2 s inter-attempt delay (:270); validated ≥ 0 (`SiteRuntimeOptionsValidator.cs:56-58`) |
| UA3 — no aggregated live alarm stream | Underdev. | **No longer applicable — shipped post-plan (2026-07-10)** | Deferred by PLAN-03 as central-scope; the aggregated-live-alarm-stream plan delivered it. Site-side piece reviewed: `SiteStreamManager.SubscribeSiteAlarms` (`SiteStreamManager.cs:134-159`) — alarm-only filter off the same BroadcastHub, per-subscriber DropHead buffer, KillSwitch teardown shared with `Unsubscribe`/`RemoveSubscriber`; clean |
| UA4 — native-alarm rehydration loses display metadata | Underdev. | **Fixed (verified)** | `SiteStorageService.cs:174-175,975` — `MetadataJson` (type/category/message/values) persisted alongside `condition_json` and restored on rehydration |
| UA5 — no scheduler stuck-thread/queue-depth observability | Underdev. | **Fixed (verified)** (same fix as S2) | See S2 row |
| UA6 — tag-subscription failure path has no retry/site event | Underdev. | **Fixed (verified)** (same fix as S4; site event emitted at `InstanceActor.cs:1093-1096`) | See S4 row |
| UA7 — test gaps (reconnect-stale-handler, compile-fail deploy status, subscribe race) | Underdev. | **Fixed (verified)** | `OpcUaDataConnectionTests.cs` (both S1 tests present), `Adapters/MxGatewayDataConnectionReconnectTests.cs`, `DeploymentManagerActorTests.cs` (compile-fail deploy + `InstanceActorInitialized` handshake tests) |
The **previous** `_client` is neither `DisposeAsync`'d nor detached. `DataConnectionActor`'s reconnect loop calls `_adapter.ConnectAsync` repeatedly on the *same adapter* (`DataConnectionActor.cs:537-547`); only the failover branch replaces the adapter. `MxGatewayDataConnection.ConnectAsync` has the same pattern (`MxGatewayDataConnection.cs:70-88`) and additionally overwrites `_eventLoopCts` without cancelling the previous event loop.
Failure scenarios:
1. **Session/socket leak per disconnect cycle.** After a keep-alive-detected drop, the old `RealOpcUaClient` still holds its `ISession` + subscription (`RealOpcUaClient.cs:133-160`); `Session.CloseAsync` only runs from `DisconnectAsync` (`RealOpcUaClient.cs:392-396`), which nothing calls on this path. A site whose PLC link flaps hourly accumulates orphaned SDK session objects, keep-alive timers, and sockets indefinitely.
2. **Spurious disconnect of a healthy connection.** If the drop was detected *reactively* (a `ReadAsync` failure → `RaiseDisconnected`, `OpcUaDataConnection.cs:222-227`), the old client's own once-only guard (`_connectionLostFired`) has **not** fired. After a successful reconnect resets the adapter's `_disconnectFired` (`OpcUaDataConnection.cs:104`), the abandoned session's keep-alive eventually fails, fires the *old* client's `ConnectionLost`, which is still wired to `OnClientConnectionLost` → `RaiseDisconnected()` → `AdapterDisconnected` → `BecomeReconnecting()` on a perfectly healthy new session. Each such cycle abandons another client — self-sustaining reconnect churn.
Fix shape: in both adapters' `ConnectAsync`, detach + dispose the old client before creating the new one (mirroring what the actor's failover branch already does at the adapter level, `DataConnectionActor.cs:421-431`).
### S2. [High] Script execution timeout is cooperative-only; hung scripts permanently consume the bounded script-thread pool
`ScriptExecutionActor` runs the body on the dedicated `ScriptExecutionScheduler` (default **8 threads**, `SiteRuntimeOptions.cs:46`) and enforces the timeout via `CancellationTokenSource(timeout)` passed to `compiledScript.RunAsync` (`ScriptExecutionActor.cs:129, 246`). Roslyn only observes the token between statements; a script blocked in synchronous I/O (`Database.Connection` ADO.NET call, a socket in a wedged state) or a CPU loop never observes cancellation. The `OperationCanceledException` path replies/logs, but the *thread* is only released when the blocking call itself returns. `ScriptExecutionScheduler.WorkerLoop` (`ScriptExecutionScheduler.cs:71-84`) has no watchdog, no thread replacement, no "N tasks stuck > timeout" metric.
Failure scenario: a site DB connection black-holes (firewall drops packets, no RST). Eight interval-triggered scripts each block ~30+ min in a synchronous read. All subsequent script *and alarm on-trigger* executions site-wide queue in `_queue` forever — alarms still transition (evaluation is on actor threads) but no on-trigger scripts run, and nothing on the health dashboard says why. The spec's claim that "Exceeding the timeout cancels the script" (`Component-SiteRuntime.md:411`) overstates what is enforced.
Mitigations worth considering: a scheduler gauge (queue depth + busy-thread age) surfaced through `ISiteHealthCollector`; per-script-thread stuck detection that logs the script name; ADO.NET command timeouts injected into `Database.Connection`.
### S3. [High] Spec drift: site-side script compile failure does not reject the deployment — central is told Success with the script silently missing
`Component-SiteRuntime.md:474-477`: "If script compilation fails when a deployment is received, the entire deployment for that instance is **rejected**. No partial state is applied. The failure is reported back to central as a failed deployment."
Implementation: `InstanceActor.CreateChildActors` logs and `continue`s past a failed compile (`InstanceActor.cs:1362-1369`; same for alarm on-trigger scripts at 1418-1423 and trigger expressions at 1524-1531). The instance starts with the failing script absent; `DeploymentManagerActor.HandleDeployPersistenceResult` reports `DeploymentStatus.Success` purely on SQLite persistence (`DeploymentManagerActor.cs:596-610`). The XML doc on `CreateChildActors` even states the contradiction: "Compilation errors reject entire instance deployment (logged but actor still starts)" (`InstanceActor.cs:1339`).
The design-time gate in Template Engine makes this rare, but the spec explicitly calls the site the last line ("Site-side compilation failures indicate an unexpected issue") — and version skew between central's Roslyn surface and the site's is exactly the case where the gates disagree. Today the operator sees a green deploy and a dead script discoverable only in the site event log.
### S4. [Medium] Instance actor never retries a failed tag subscription; a subscribe/create ordering race strands attributes at Uncertain forever
`InstanceActor` registers `Receive<SubscribeTagsResponse>(_ => { })` — an explicit no-op (`InstanceActor.cs:207`). If the DCL replies `Success=false` there is **no retry and no logging**. Two ways to get there:
- `DataConnectionManagerActor.HandleRoute` replies "Unknown connection" when the `SubscribeTagsRequest` arrives before the `CreateConnectionCommand` has been processed (`DataConnectionManagerActor.cs:104-115`). The Deployment Manager Tells `CreateConnectionCommand` and then creates the Instance Actor (`DeploymentManagerActor.cs:543-546`), whose `PreStart` Tells `SubscribeTagsRequest` — but the two messages come from **different senders**, so Akka's per-sender-pair ordering guarantee does not formally order them at the manager's mailbox. If the race fires, the instance's data-sourced attributes stay Uncertain until redeploy/restart.
- The connection-level-failure reply ("connection unavailable — will re-subscribe on reconnect", `DataConnectionActor.cs:1009-1012`) is self-healing because `_subscriptionsByInstance` was already updated and `ReSubscribeAll` re-derives from it — that case is fine. The unknown-connection case leaves **no state anywhere**.
Contrast: `NativeAlarmActor` retries a failed subscribe on a timer (`NativeAlarmActor.cs:269-286`). The tag path deserves the same treatment — at minimum, log the failure and schedule a bounded re-subscribe.
Related sub-gap: `NativeAlarmActor`'s retry only arms on a *failure response*. A subscribe whose response is **lost** (DCL manager restarted between request and reply; message dead-lettered) never retries — `SendSubscribe` is a bare `Tell` with no response timeout (`NativeAlarmActor.cs:123-131`). Rehydrated SQLite state masks the gap in the debug view while the mirror is actually dead.
### S5. [Medium] Deployment Manager startup aborts silently on a SQLite read failure — site runs zero instances with no retry
`HandleStartupConfigsLoaded`: on error, `_logger.LogError(...); return;` (`DeploymentManagerActor.cs:291-297`). One transient SQLite failure (locked file during a crash-recovery race, disk hiccup) at singleton start/failover leaves the active node with **no Instance Actors** until a human restarts the process. There is no retry timer, no health flag beyond the log line (health instance counts will read 0 deployed, which at least shows on the dashboard, but nothing distinguishes "genuinely nothing deployed" from "startup load failed"). A bounded retry (the same `Timers.StartSingleTimer` idiom used two methods later) would close this.
### S6. [Medium] A throwing `InstanceActor` constructor leaves a dead ref in `_instanceActors` and still reports the deployment Success
`InstanceActor`'s constructor deserializes the config JSON (`InstanceActor.cs:138`) and builds the resolved-attribute index; a malformed/oversized config throws → `ActorInitializationException` → Deployment Manager's decider Stops the child (`DeploymentManagerActor.cs:268-274`). But:
- `CreateInstanceActor` already stored the ref in `_instanceActors` (`DeploymentManagerActor.cs:1628-1629`) and nothing watches for this failure mode (the `Context.Watch` only happens on the redeploy path), so the map holds a dead ref — debug view, inbound-API routes, etc. dead-letter with 30s Ask timeouts instead of "not found".
- `ApplyDeployment`'s persistence continues and `HandleDeployPersistenceResult` reports `Success` (`DeploymentManagerActor.cs:596-610`) — central shows a healthy deployment for an instance that is not running.
A `Context.Watch` on every created child (removing from `_instanceActors` on unexpected `Terminated`) would fix both the stale map and give a hook to fail the pending deployment.
### S7. [Medium] Subscribe background task reads the mutable `_adapter` field off-thread, violating its own confinement contract
`HandleSubscribe`'s `Task.Run` body awaits `_adapter.SubscribeAsync(...)` per tag and calls `SeedTagsAsync(_adapter, ...)` (`DataConnectionActor.cs:707-758`) — reading the actor field from a thread-pool thread, despite the comment eight lines earlier: "The background task below must NOT read or mutate actor state — these partitioned lists are the only state it sees" (`DataConnectionActor.cs:689-690`). If a failover swaps `_adapter` mid-loop (`CountFailureAndMaybeFailover`, `DataConnectionActor.cs:610-635`), later iterations subscribe on the **new** adapter while stamping the **old** generation — the resulting values are dropped by the generation guard and the monitored items are orphaned on the new adapter (never in `_subscriptionIds` for that generation… actually worse: `SubscribeCompleted` will store them against current state, mixing generations). `ReSubscribeAll` (`DataConnectionActor.cs:1596`) and `HandleWriteBatch`'s local async function (`DataConnectionActor.cs:1161-1201`) have the same field-read pattern. Capture the adapter into a local before spawning the task (as the re-seed block at `DataConnectionActor.cs:1620` correctly does with `reseedAdapter`).
### S8. [Low] Site SQLite: connection-per-call, no WAL / busy-timeout tuning, many concurrent writers
`SiteStorageService` opens a fresh `SqliteConnection` per operation with a bare `Data Source=` string (`SiteStorageService.cs:34-44, 163-166`) — no `journal_mode=WAL`, relying on Microsoft.Data.Sqlite's default command-timeout-as-busy-handler. Concurrent writers against the same file include: deploy persistence, static-override fire-and-forget writes (`InstanceActor.cs:433-441`), per-transition native-alarm upserts (`NativeAlarmActor.cs:420-428`), replication applies (`SiteReplicationActor.cs:182-216`), and reconciliation. Under a native-alarm storm plus a deploy, rollback-journal locking serializes everything and a slow write can push others toward `SQLITE_BUSY`. WAL + explicit busy_timeout is a one-line hardening. (Contrast: `SiteEventLogger` got the full treatment — single owned connection, lock, bounded write queue.)
### S9. [Low] Fire-and-forget adapter disposal in `PostStop`
`DataConnectionActor.PostStop` discards the `DisposeAsync` task (`DataConnectionActor.cs:218-221`); a hanging/faulting dispose (OPC UA `CloseAsync` against a dead server can block for the operation timeout) is unobserved. Acceptable trade-off (documented elsewhere in the file), but pair it with a `ContinueWith(OnlyOnFaulted, log)` like the other fire-and-forget paths in this codebase do.
### S10. [Low] Alarm condition filter is last-subscriber-wins across co-subscribers of one source
`_alarmSourceFilter[request.SourceReference] = request.ConditionFilter` overwrites unconditionally (`DataConnectionActor.cs:104-108, 1763-1766`); two instances binding the same source with different filters silently re-gate each other's transitions. The field's XML doc honestly declares the sharp edge, but nothing *validates* agreement or logs the overwrite — a mismatch is undiagnosable in the field. Log a warning when the incoming filter differs from the stored one.
**Disposition counts:** 26 Fixed (verified) · 2 Deferred (accepted: P5, P6) · 0 Not fixed · 0 Regressed. (UA3 counted Fixed via the post-plan 2026-07-10 delivery; UA1's removal-reconcile sub-scope remains an explicitly documented deferral inside an otherwise-fixed item.)
---
## Findings — Performance
## New Findings (Round 2)
### P1. [High] Trigger expressions block actor threads on the default dispatcher — per attribute change
`ScriptActor.EvaluateExpressionTrigger` runs the compiled Roslyn expression with `.GetAwaiter().GetResult()` under a 2-second CTS (`ScriptActor.cs:283-287`); `AlarmActor.EvaluateExpression` is identical (`AlarmActor.cs:503-506`). Both execute **on the actor's default-dispatcher thread**, and both run on *every* `AttributeValueChanged` the actor receives (Expression triggers have no monitored-attribute gate — `ScriptActor.cs:255-258`, `AlarmActor.cs:215-218`). A site with a handful of Expression-triggered scripts/alarms on a chatty instance (e.g., 50 tags at 1 Hz) synchronously executes 100s of Roslyn script invocations per second on dispatcher threads; one pathological expression (2 s timeout, treated-as-false) stalls its actor's mailbox and steals a dispatcher thread for the duration. The `ScriptExecutionScheduler` exists precisely to keep script code off these threads — trigger expressions bypass it. The in-code justification ("trusted Design-role users, compile-checked") addresses malice, not cost.
### N1. [Medium] MxGateway: a stale event loop's *non-OCE* fault can still flap the fresh connection — the S1 fix closed the OPC UA path but left this one open
The S1 fix in `MxGatewayDataConnection.ConnectAsync` cancels the previous loop's CTS and disposes the previous client (`MxGatewayDataConnection.cs:77-89`), then resets the disconnect guard on the very next line (`Interlocked.Exchange(ref _disconnectFired, 0)`, `:91`) — *before* the old loop has actually observed the cancellation. The comment claims a teardown fault "is absorbed by the old guard cycle", but `Cancel()` is not synchronous with the loop's exit: the old loop's `await foreach (… _session!.StreamEventsAsync(_lastSeq, ct))` (`RealMxGatewayClient.cs:278-294`) surfaces its failure milliseconds later, while the new `ConnectAsync` is still awaiting. `RunEventLoopAsync`'s catch (`MxGatewayDataConnection.cs:121-129`) only absorbs `OperationCanceledException`; a cancelled/disposed gRPC stream commonly faults with `RpcException(StatusCode.Cancelled)` (Grpc.Net default without `ThrowOperationCanceledOnCancellation`, which the wrapped `MxGatewayClient` package is not shown to set) or `ObjectDisposedException` from the concurrent `DisposeAsync` — both fall into the generic `catch` → `RaiseDisconnected()` against the *reset* guard → `AdapterDisconnected` → reconnect → which cancels/disposes again → potentially self-sustaining churn, the exact S1 scenario-2 mechanism. Unit tests can't catch it (the reconnect test substitutes the client and never faults the loop), and there is no real gateway in the local env.
**Fix shape:** in the generic catch, treat `ct.IsCancellationRequested == true` as normal shutdown (no `RaiseDisconnected`). Secondary hardening in the same method: `Task.Run(() => RunEventLoopAsync(_eventLoopCts.Token))` (`:108`) reads the `_eventLoopCts` *field* lazily — a rapid double-reconnect can hand loop #1 loop #2's token; capture the token into a local before the lambda.
### P2. [Medium] Attribute-change fan-out is broadcast to every child, filtering in the receiver
`InstanceActor.PublishAndNotifyChildren` Tells **every** Script Actor and **every** Alarm Actor for every attribute change (`InstanceActor.cs:1184-1194`); `Component-SiteRuntime.md:191` describes children subscribing "for the specific monitored attribute". An instance with 30 scripts + 20 alarms and 100 Hz aggregate tag updates generates 5,000 child messages/sec, ~98% of which are discarded by the child's trigger check (`ScriptActor.cs:229-233`, `AlarmActor.cs:219-222`) — pure mailbox/dispatch overhead, plus each `ScriptActor` maintains an `_attributeSnapshot` copy of every value (`ScriptActor.cs:227`). A per-attribute subscription map on the Instance Actor (attributeName → interested children; Expression-trigger children in an "all" bucket) would cut this by an order of magnitude on realistic configs.
### N2. [Low] Trigger-expression `PipeTo` has a success mapping only — a faulted scheduler task permanently kills the actor's expression trigger
`ScriptActor.StartExpressionEvaluation` (`ScriptActor.cs:311-313`) and `AlarmActor.StartExpressionEvaluation` (`AlarmActor.cs:552-554`) pipe the evaluation back with `PipeTo(self, success: r => new ExpressionEvalResult(r))` and no `failure:` mapping. The inner body catches all exceptions and returns `false`, so the only fault sources are the outer `Task.Factory.StartNew`/`QueueTask` itself (e.g. `ObjectDisposedException` from a disposed `ScriptExecutionScheduler` during shutdown or in test teardown). If that ever fires, the actor receives an unhandled `Status.Failure` and — critically — `_evalInFlight` stays `true` forever (`ScriptActor.cs:284-285`, `AlarmActor.cs:523-524`), so every subsequent attribute change parks in `_evalPending` and the expression trigger is dead for the actor's lifetime with no log. One-line fix: add `failure: ex => new ExpressionEvalResult(false)` (or a dedicated failure message that logs).
### P3. [Medium] DCL tag-value fan-out scans all instances per update
`HandleTagValueReceived` iterates `_subscriptionsByInstance` (every instance) with a per-instance `HashSet.Contains` for every incoming tag value (`DataConnectionActor.cs:1706-1716`). The actor already maintains `_tagSubscriberCount` (tag → count) for O(1) unsubscribe; the matching tag → *instances* reverse index is missing. With 500 instances on one connection at high sample rates this is the hottest loop in the DCL. Also `_healthCollector.UpdateTagQuality` is invoked on **every** update (`DataConnectionActor.cs:1737`) — cheap if the collector is a plain field write, but worth confirming it isn't lock-contended.
### N3. [Low] New PLAN-03 option knobs are missing from the eager options validator this same initiative introduced
`SiteRuntimeOptionsValidator.cs:20-58` validates ten options, but not the two PLAN-03 added: `TagSubscribeRetryIntervalMs` (`SiteRuntimeOptions.cs:76`, drives `ScheduleTellOnceCancelable` in `InstanceActor.SendTagSubscribe` `:1065-1067` — a 0 hot-loops the subscribe retry, a negative value throws inside the actor) and `StuckScriptGraceMs` (`SiteRuntimeOptions.cs:86`, a negative value throws `ArgumentOutOfRangeException` inside the watchdog's `Task.Delay`, `ScriptExecutionActor.cs:144`). Inconsistent with the PLAN-08 §1.5 "fail fast at boot with a key-naming message" convention; two `RequireThat` lines close it.
### P4. [Low] Per-transition SQLite upserts on the native-alarm mirror
`NativeAlarmActor.PersistUpsert` issues one async SQLite write per accepted transition (`NativeAlarmActor.cs:420-428`), each opening a fresh connection (S8). An A&C alarm storm (hundreds of transitions/sec across sources) turns into hundreds of single-row upsert connections/sec. Bounded by `MirroredAlarmCapPerSource` (1000) per source in state size but not in write rate. Batching/coalescing (persist latest per SourceReference on a short timer) would be a proportionate hardening if storms are expected.
### N4. [Low] Every deploy now compiles each script twice — synchronously on the singleton mailbox at the gate, then again in `InstanceActor.PreStart` — with no compile cache
The S3 gate runs `_deployCompileValidator.Validate` on the singleton's actor thread (`DeploymentManagerActor.cs:522`; deliberate and well-argued for mailbox-FIFO ordering, :510-521), and the created Instance Actor then recompiles the identical scripts/expressions in `CreateChildActors` (`InstanceActor.cs:1485,1497-1498`). `ScriptCompilationService` has no compile cache, so a site-wide redeploy of N instances serializes roughly 2×N script-set Roslyn compiles through the singleton + instance start path (each 50–300 ms). Single-instance deploys are fine; bulk redeploy latency and central Ask timeouts are the exposure. The master tracker already earmarks adopting PLAN-05's compile-cache abstraction for `DeployCompileValidator` ("later refactor, not a blocker") — this finding is the concrete reason to actually do it, and a keyed cache would also shrink the P6 failover-recompile cost for free.
### P5. [Low] Every script attribute read is an actor Ask round-trip
`ScriptRuntimeContext` Asks the Instance Actor per `GetAttribute` (`ScriptRuntimeContext.cs:375`) — a temp actor allocation + two mailbox hops per read. Correct (serialized through the source of truth) and consistent with the spec, but a script reading 50 attributes in a loop pays 50 round-trips; a `GetAttributes` batch message (the DeploymentManager routing path already fans out N Asks — `DeploymentManagerActor.cs:1360-1367` — which would also benefit) is an easy win if profiling ever flags it.
### N5. [Low] Spec drift (reintroduced): `Component-SiteRuntime.md` says the compile gate runs "off-thread" — the shipped gate is deliberately synchronous
`Component-SiteRuntime.md:489`: "…the Deployment Manager compiles all of the instance's scripts … **off-thread**; any compile failure rejects the deployment…". The implementation intentionally validates *on* the singleton thread (`DeploymentManagerActor.cs:510-522`), a deviation captured in the master tracker and the in-code comment but never folded back into the component doc. Given this repo's docs-travel-with-code rule — and that round 1's S3 was itself a doc-vs-code honesty finding — the sentence should say "synchronously, as a pure prefix step of the deploy handler (preserving mailbox FIFO ordering)".
### P6. [Low] Roslyn compiles run inside `InstanceActor.PreStart` on the default dispatcher during staggered startup
Each instance compiles all its scripts + alarm scripts + trigger expressions synchronously in `CreateChildActors` (called from `PreStart`, `InstanceActor.cs:250-255, 1348+`). The startup stagger (20 instances / 100 ms, `SiteRuntimeOptions.cs:13-19`) is tuned for OPC UA connection storms, not compile cost — 500 instances × several scripts each at ~50–300 ms/compile will occupy default-dispatcher threads for minutes after failover. The Deployment Manager already moved *shared*-script compilation off-thread (`DeploymentManagerActor.cs:1168-1191`); per-instance compiles did not get the same treatment. Failover time-to-recover is the metric at risk.
### N6. [Low] Observation: trigger expressions now share the 8-thread bounded script scheduler with (possibly stuck) script bodies — saturation silently stalls all Expression triggers and Expression alarms site-wide
The P1 fix routes every expression evaluation through `ScriptExecutionScheduler.Shared` (`ScriptActor.cs:312`, `AlarmActor.cs:553`). Before, a pathological expression stalled a dispatcher thread (bad) but expressions always ran; now, eight stuck script bodies (the S2 scenario) also mean **no Expression-triggered script or alarm evaluates anywhere on the site** until a thread frees — alarm activation latency becomes unbounded during scheduler saturation. The coalescing (one in-flight + one pending per actor) correctly bounds queue growth, and the new gauges/watchdog make the state visible, so this is an acceptable and arguably correct trade-off — but it is a behavioral coupling the component doc's Alarm section should state (one sentence: "Expression alarm evaluation shares the bounded script-execution pool; scheduler saturation delays alarm transitions").
---
## Findings — Conventions
## What's Genuinely Good
### C1. [Positive] Akka.NET discipline is exemplary
Worth stating because it is the dominant impression: Become/Stash exactly as the spec draws it (`DataConnectionActor.cs:232-533`); Tell on hot paths, Ask only at boundaries (`Component-SiteRuntime.md:430-442` ↔ implementation); `Sender`/`Self` captured before every continuation (e.g., `DeploymentManagerActor.cs:464-470`, `DataConnectionActor.cs:149-153`); all async results marshalled via `PipeTo`; supervision matches the spec table precisely — DM `OneForOne` Resume/Stop-on-init (`DeploymentManagerActor.cs:263-279`), Instance Resume (`InstanceActor.cs:281-293`), Script/Alarm coordinators Stop their execution children (`ScriptActor.cs:180-192`, `AlarmActor.cs:192-204`). The redeploy Terminated-watch buffering with the name-shadow map and its disable/delete supersede handling (`DeploymentManagerActor.cs:60-90, 389-435, 641-698, 762-831`) is genuinely hard concurrency done correctly. The in-flight subscribe/orphan-release guards in the DCL (`DataConnectionActor.cs:77-90, 836-985, 1795-1858`) close race classes most codebases never notice.
### C2. [Low] Stale comment: SiteEventLogger claims its channel is unbounded
`SiteEventLogger.cs:216-218` — "The channel is unbounded, so the only way TryWrite fails is…" — the channel became bounded (`Channel.CreateBounded`, `SiteEventLogger.cs:74-88`), and `TryWrite` on a bounded DropOldest channel still only fails when completed, so the *behavior* is right but the stated reason is wrong. Trivial fix; flagged because this file's comments are otherwise load-bearing.
### C3. [Low] Spec is internally inconsistent on the reconnect interval
`Component-DataConnectionLayer.md:303` says "The retry interval is defined **per data connection**", but the same document's Shared Settings table (`Component-DataConnectionLayer.md:170-177`) and the implementation (`DataConnectionOptions.ReconnectInterval`, one value for all connections — `DataConnectionActor.cs:463`) make it global. Fix the sentence at :303 (per the CLAUDE.md rule that the design doc travels with the code).
### C4. [Low] Quality-only change on disconnect is not delivered to Alarm/Script actors — deliberate, but diverges from the spec's wording
`HandleConnectionQualityChanged` updates qualities and the stream but explicitly does **not** notify children ("firing scripts/alarms would cause spurious evaluations", `InstanceActor.cs:924-956`). The DCL spec promises "Instance Actors and their downstream consumers (**alarms, scripts checking quality**) see the staleness immediately" (`Component-DataConnectionLayer.md:302`). Practical consequence: an Active alarm computed from a now-dead connection stays Active (arguably correct — last-known-state), and a Conditional/Expression script never learns quality went Bad. The implementation choice is defensible; the spec sentence should be qualified so the divergence is a decision, not drift.
### C5. [Low] Shared scripts execute without the caller's scope and without a `Scope` assignment
`SharedScriptLibrary.ExecuteAsync` builds `ScriptGlobals` without setting `Scope` (`SharedScriptLibrary.cs:98-103`), so a shared script always runs `ScriptScope.Root` (`ScriptCompilationService.cs:200-201`) even when invoked from a composed-module script. If a shared script uses `Attributes[...]`, it addresses **root** attributes regardless of caller scope. Possibly intended (shared scripts are instance-agnostic), but it is undocumented in `Component-SiteRuntime.md`'s Shared Script Library section and is a surprise waiting for a template author.
### C6. [Low] `EvaluateCondition` fallback compares `value.ToString()` to `Threshold.ToString()` with current-culture double formatting
`ScriptActor.cs:474-477`: the non-numeric fallback stringifies the double threshold with the host culture (the numeric path was carefully invariant-culture'd; the fallback wasn't). A `"1.5"` threshold on a de-DE host renders `"1,5"`. Edge-case only (non-parseable values), but inconsistent with the deliberate invariant-culture work two lines up.
- **The fix quality matches the codebase's standard.** Every PLAN-03 fix reads like the surrounding hardened code: retries use the existing fixed-interval cancelable-timer idiom with supersede correlation ids (`InstanceActor.cs:1047-1068`), all off-thread work returns via `PipeTo` with state applied on the actor thread, and health metrics ride the additive-only `SiteHealthReport` contract.
- **Two fixes shipped *better* than the plan asked.** S4's retry arms on *send* (covering lost responses, not just failure replies — the plan's sub-gap), and S6 was redesigned mid-execution when the implementer discovered the store commits before `PreStart` throws: the `InstanceActorInitialized` two-signal join (`DeploymentManagerActor.cs:2085-2104`) plus commit-ordered rollback is deterministic where the plan's swallow-only guard would have silently kept the bug. The deviation is documented in the master tracker *and* the component doc (`Component-SiteRuntime.md:109`).
- **The S3 synchronous-gate decision is correctly reasoned.** The in-code comment (`DeploymentManagerActor.cs:510-521`) explains precisely why off-thread validation would break the redeploy-supersede/delete ordering that depends on mailbox FIFO — the kind of concurrency reasoning this file consistently gets right. (It just needs the doc sentence fixed — N5.)
- **Instrumentation is lock-free and cheap.** The scheduler gauges use per-worker `Volatile` slots read without locks (`ScriptExecutionScheduler.cs:29,83-113`); the stuck-watchdog deliberately runs its grace delay on the thread pool, not the possibly-saturated script scheduler (`ScriptExecutionActor.cs:138-139`).
- **The reverse indexes are maintained at every mutation site.** Both P2's child-routing maps (filed at child creation, `InstanceActor.cs:1451-1469`) and P3's `_instancesByTag` (add/remove helpers called from subscribe, unsubscribe, and instance-release paths incl. `:1076`) show the discipline that makes index-based fan-out safe.
- **`TriggerRouting` fails open.** Unparseable trigger configs route to the all-changes bucket rather than silencing a child (`TriggerRouting.cs:81-105`) — routing is an optimization, the child's own gate stays the correctness check, exactly the right layering.
- **Docs kept pace.** C3/C4/C5 spec fixes landed, the S6 deviation is in the component doc, and the shared-script Root-scope surprise is now a documented contract (`Component-SiteRuntime.md:344`).
---
## Underdeveloped Areas
## Severity Tally (new findings only)
1. **Cert-trust convergence after node downtime.** The trust broadcast reaches only currently-Up nodes (`DeploymentManagerActor.BroadcastToSiteCertStores`, :962-1007); a node that is down during the broadcast rejoins with a divergent PKI store and there is no reconcile-on-join, no periodic store diff, and `ListServerCertsCommand` reads only the singleton's own node (:936-953) so the divergence is invisible from central. Central persistence of trust decisions is the documented follow-up (`Component-SiteRuntime.md:125`); until then a join-time pull from the peer's `CertStoreActor` would close the failover hole.
2. **Dead option**: `SiteRuntimeOptions.ConfigFetchRetryCount` — "Reserved — consumed by the standby replication fetch in a later task; not yet wired" (`SiteRuntimeOptions.cs:63-67`). The standby fetch is genuinely single-attempt (`SiteReplicationActor.cs:178-216`), leaning entirely on reconciliation as backstop.
3. **No aggregated live alarm stream** — the M7 operator Alarm Summary polls per-instance `DebugViewSnapshot` Asks; documented as a follow-up in CLAUDE.md/M7 plan. Fine for now; scale limit is the SemaphoreSlim-capped fan-out on the central side.
4. **Native alarm rehydration loses display metadata** — only `condition_json` + timestamps persist (`SiteStorageService` `native_alarm_state`, :117-124), so a rehydrated condition renders with empty type/category/message until the first source snapshot (`NativeAlarmActor.cs:161-167`). Acknowledged in comments; worth a line in the component doc's operator-facing section.
5. **No stuck-thread/queue-depth observability for the script scheduler** (see S2) — `ScriptExecutionScheduler` exposes nothing to `ISiteHealthCollector`.
6. **Tag-subscription failure path has no retry or site-event entry** (see S4) — the spec's Tag Path Resolution retry covers resolution failures inside the DCL, but the instance-side response handling is a hole.
7. **Tests**: coverage breadth is very good (redeploy races, waiter races, native-alarm swap/cap, seed reads, scheduler, trust model — 17 actor test files, 17 script test files, dedicated DCL alarm/browse/verify suites, plus `OpcUaAlarmLiveSmokeTests`). Gaps that match the findings above: no test exercises reconnect-then-stale-`ConnectionLost` (S1), compile-failure-deploy-status (S3), the unknown-connection subscribe race (S4), or sustained-churn fan-out cost (the `PerformanceTests` project was not examined for these paths).
| Severity | Count | Findings |
|---|---|---|
| Critical | 0 | — |
| High | 0 | — |
| Medium | 1 | N1 |
| Low | 5 | N2, N3, N4, N5, N6 |
---
## Prioritized Recommendations
1. **(S1)** Dispose + detach the previous client in `OpcUaDataConnection.ConnectAsync` / `MxGatewayDataConnection.ConnectAsync` (and cancel the old MxGateway event loop). This is the single highest-value fix: it removes both a per-flap resource leak and a healthy-connection flapping mechanism.
2. **(S3)** Make site-side compile failure fail the deployment, per spec: have `CreateChildActors` report compile outcomes to the Deployment Manager (or pre-compile in `ApplyDeployment` before actor creation, which also fixes P6 by moving compiles off the instance-start path) and return `DeploymentStatus.Failed` with the errors.
3. **(S2)** Add stuck-execution observability (scheduler queue depth + oldest-busy-thread age as health metrics; log the script name on timeout-with-thread-still-busy), and inject command timeouts into script-facing DB connections.
4. **(P1)** Move trigger-expression evaluation off the actor thread — evaluate on the `ScriptExecutionScheduler` and post the boolean result back to the actor (edge-state stays actor-confined), or precompile expressions to delegates rather than re-running Roslyn scripts.
5. **(S4)** Handle `SubscribeTagsResponse.Success == false` in `InstanceActor` with a logged, bounded re-subscribe timer (mirror `NativeAlarmActor`'s retry); add a response-timeout retry to `NativeAlarmActor.SendSubscribe`.
6. **(S5/S6)** Retry the startup config load on failure; `Context.Watch` all Instance Actors so an init-failure removes the dead ref and fails the in-flight deployment.
7. **(P2/P3)** Introduce per-attribute child-subscription routing in `InstanceActor` and a tag→instances reverse index in `DataConnectionActor` before scaling to the 500-instance design point.
8. **(S8)** Enable WAL + explicit busy timeout on the site SQLite stores.
9. **(S7)** Capture `_adapter` into locals before every background task in `DataConnectionActor` (three sites).
10. **(C3/C4/C5)** Doc fixes: reconcile the reconnect-interval wording, qualify the disconnect-quality-visibility promise, and document shared-script scope semantics — per the repo rule that design docs and code travel together.
Round-1 residue: P5 and P6 remain deferred with accepted rationale (P6's cost is now partially addressable via N4's compile cache); UA1's removal-reconcile/list-aggregation sub-scope stays parked behind the documented central-persistence-of-cert-trust follow-up (`Component-SiteRuntime.md:125`).
archreview/04-data-audit-backbone.md
+85 -139
View File
@@ -1,180 +1,126 @@
# Architecture Review 04 — Central Data Layer & Audit/KPI Backbone
# Architecture Review 04 — Central Data Layer & Audit/KPI Backbone (Round 2)
**Date:** 2026-07-12 (round 2; round-1 baseline commit `b910f5eb`, current HEAD `8c888f13`)
**Scope reviewed:**
- `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase` — EF Core mappings, repositories, migrations, `IAuditService`, partition maintenance
- `src/ZB.MOM.WW.ScadaBridge.Commons` — as the data-shape layer (entities, repository interfaces, audit enums/types, KPI types)
- `src/ZB.MOM.WW.ScadaBridge.AuditLog` — central ingest/purge/reconciliation actors, `CentralAuditWriter`, site SQLite writer + telemetry
- `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox` — `NotificationOutboxActor`, repository, delivery adapters, KPI source
- `src/ZB.MOM.WW.ScadaBridge.SiteCallAudit` — `SiteCallAuditActor`, monotonic upsert repository, reconciliation, relay
- `src/ZB.MOM.WW.ScadaBridge.KpiHistory` — `KpiHistoryRecorderActor`, `KpiHistoryRepository`, bucketer
- Design docs: `docs/requirements/Component-ConfigurationDatabase.md`, `Component-AuditLog.md`, `Component-NotificationOutbox.md`, `Component-SiteCallAudit.md`, `Component-KpiHistory.md`, `Component-Commons.md`
- Test projects: `AuditLog.Tests`, `NotificationOutbox.Tests`, `SiteCallAudit.Tests`, `KpiHistory.Tests`, `ConfigurationDatabase.Tests`
- `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase` — EF Core mappings, repositories, migrations (5 new since baseline), `IAuditService`, partition maintenance
- `src/ZB.MOM.WW.ScadaBridge.AuditLog` — central ingest/purge/reconciliation actors, pull clients, site SQLite writer + telemetry + new retention job
- `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox` — dispatcher (now bounded-parallel), repository (single-query KPIs), delivery adapters
- `src/ZB.MOM.WW.ScadaBridge.SiteCallAudit` — actor (composite keyset reconciliation cursor, pinned event, relay identity), repository (same-rank tiebreaker, RECOMPILE)
- `src/ZB.MOM.WW.ScadaBridge.KpiHistory` — recorder actor **including the post-initiative hourly-rollup subsystem shipped 2026-07-10** (`KpiRollupHourly`, `kpi-rollup` tick, backfill, `KpiMetricAggregationCatalog`, raw-vs-rollup routing) — brand-new code no round-1 review saw
- Design docs: `Component-ConfigurationDatabase.md`, `Component-AuditLog.md`, `Component-NotificationOutbox.md`, `Component-SiteCallAudit.md`, `Component-KpiHistory.md`, `Component-Commons.md`
## Maturity Verdict
**Method:** read the round-1 report and `archreview/plans/PLAN-04-data-audit-backbone.md` in full; verified every round-1 finding's true disposition in current source (file:line, not trusting the plan's claims); fresh static sweep of `git diff b910f5eb..HEAD` scoped to the five projects + new migrations, with the rollup subsystem reviewed line-by-line. No builds/tests run (suites verified green 2026-07-10).
This is an unusually mature data layer for a system at this stage: idempotent ingest is implemented correctly and defensively everywhere (duplicate-key race handling in all three insert-if-not-exists paths), the combined cached-telemetry dual-write really is transactional (verified: `AuditLogIngestActor.OnCachedTelemetryAsync` shares one scoped `DbContext` across both repositories under `BeginTransactionAsync`), UTC discipline is enforced with converters and `SpecifyKind` re-tagging at every ADO.NET boundary, actor/DB interaction follows a consistent scope-per-message + PipeTo pattern with in-flight guards, and MSSQL-backed integration tests exist for the hard parts (partition purge, execution-tree walks, outage reconciliation). The weaknesses are concentrated in **scale-dependent operational behavior** rather than logic: the partition purge's full-table unique-index rebuild will start failing on default command timeouts as the table grows, several KPI predicates have no supporting index and full-scan year-sized tables every 60 seconds, the execution-tree query scans the entire AuditLog per invocation, the spec'd site SQLite 7-day retention purge simply does not exist, the append-only DB-role story is DDL-only (one connection principal runs both writer and maintenance paths), and the persisted `backlogTotal` KPI trend is hardwired to zero. None of these bites at demo scale; several will bite within months of production volume.
**Round-1 vs round-2 verdict:** every round-1 time bomb was genuinely defused — all 12 Fixed dispositions verified in code, deferrals are honestly documented in the design docs rather than silently dropped — and the fix quality is uniformly high (the NodeB-failover pull clients and the keyset cursor are exemplary); the one new material risk is concentrated in the brand-new rollup **backfill** path, which re-introduces at startup exactly the "unbounded single-pass over a year-scale table" shape the plan spent five tasks eliminating elsewhere.
---
## Findings — Stability
## Round-1 Finding Disposition
### [High] Site SQLite 7-day retention purge is specified but not implemented — unbounded site DB growth
Every disposition below was verified against current source, not the plan's coverage table.
`Component-AuditLog.md:412-414` specifies: "Sites: daily site job; default 7-day retention (configurable, min 1, max 90). Respects the hard `ForwardState` invariant." No such job exists. There is no `DELETE` against `audit_event` / `audit_forward_state` anywhere in `src/` — the only destructive statement is the one-time legacy-table `DROP TABLE IF EXISTS AuditLog` at startup (`src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriter.cs:198`). The writer's own comments treat the purge as future work: `SqliteAuditWriter.cs:145` ("a future `PRAGMA incremental_vacuum` shrink the file after the 7-day retention purge") and `:33` claims the store is "recreated per deployment", which is not a retention mechanism.
| # | Round-1 finding | R1 sev | Disposition | Evidence (file:line) |
|---|-----------------|--------|-------------|----------------------|
| S1/U2 | Site SQLite 7-day retention purge missing | High | **Fixed (verified)** | `SqliteAuditWriter.PurgeExpiredAsync` — temp-table purge honoring the ForwardState invariant (`AuditLog/Site/SqliteAuditWriter.cs:858-966`), post-purge `incremental_vacuum` (:949-961); `SiteAuditRetentionService` hosted loop (`Site/SiteAuditRetentionService.cs:62-117`); options bound (`AuditLog/ServiceCollectionExtensions.cs:107`) and service registered (:335); clamped options (`Site/SiteAuditRetentionOptions.cs`); tests `SqliteAuditWriterRetentionTests.cs`, `SiteAuditRetentionServiceTests.cs`, `SiteAuditRetentionOptionsTests.cs` |
| S2 | `SwitchOutPartitionAsync` no CommandTimeout; purge failure log-only | High | **Fixed (verified)** | `commandTimeout` parameter + `SetCommandTimeout` with finally-restore (`ConfigurationDatabase/Repositories/AuditLogRepository.cs:230, 368, 389-402`), same on per-channel `DELETE TOP` (:472); actor passes `ResolvedMaintenanceCommandTimeout` (default 30 min, 1-min floor — `AuditLogPurgeOptions.cs:73-81`) at `AuditLogPurgeActor.cs:200, 295`; failure now publishes `AuditLogPurgeFailedEvent` + increments the `IAuditPurgeFailureCounter` health counter (`AuditLogPurgeActor.cs:235-247`, `Central/AuditLogPurgeFailedEvent.cs`, `Central/IAuditPurgeFailureCounter.cs`) |
| S3 | Append-only role DDL-only; purger grants insufficient | High | **Fixed (verified)** | Migration `20260709105112_FixAuditPurgerRoleGrants.cs:25-33` grants `CREATE TABLE` + scoped `DELETE ON dbo.AuditLog` (idempotent, reversible Down); docs now state honestly that default deployments enforce append-only via the CI grep guard, with the roles as optional DBA hardening (`Component-AuditLog.md:489-495`; `Component-ConfigurationDatabase.md:342` retracts the stale "no row DELETE even for purge" claim) |
| S4 | SiteCalls upsert freezes RetryCount/LastError at rank | Medium | **Fixed (verified)** | Same-rank freshness tiebreaker on `UpdatedAtUtc`, with terminal rows deliberately immutable so a later terminal never overwrites an earlier one (`SiteCallAuditRepository.cs:106-165`) |
| S5 | No EF `EnableRetryOnFailure` | Medium | **Fixed (verified)** | `EnableRetryOnFailure(5, 30s)` fleet-wide (`ConfigurationDatabase/ServiceCollectionExtensions.cs:33-36`); the combined-telemetry dual-write transaction is wrapped in `CreateExecutionStrategy().ExecuteAsync` per entry (`AuditLogIngestActor.cs:263-298`) |
| S6 | Operator Retry emits no audit row; relay lacks operator identity | Medium | **Fixed (verified)** | `RetryAsync` emits a `Submitted` NotifyDeliver row with `RequestedBy` as actor (`NotificationOutboxActor.cs:1078-1126`); Discard passes it too (:1183); `SiteCallAuditActor` takes an optional `ICentralAuditWriter` and emits relay-identity rows (`SiteCallAuditActor.cs:104, 199, 288-291`) |
| S7 | Reconciliation dead-ends (cursor pin; permanent abandonment) | Medium | **Fixed (verified)** | Composite `(UpdatedAtUtc, TrackedOperationId)` keyset cursor per site (`SiteCallAuditActor.cs:135-159, 635-713`); legacy-site pin now publishes `SiteCallReconciliationPinnedChanged` on transitions (:737-746, `SiteCallReconciliationPinnedChanged.cs`); site store honors strict keyset with legacy `>=` fallback (`SiteRuntime/Tracking/OperationTrackingStore.cs:365-395`; proto `sitestream.proto:187-192`; server maps empty→null `SiteStreamGrpcServer.cs:607-613`); abandonment writes a durable synthetic `AuditKind.ReconciliationAbandoned` row (`SiteAuditReconciliationActor.cs:333-385`, `AuditKind.cs:43`) |
| S8 | Outbox repository dual SQLite/T-SQL dialect | Medium | **Deferred (accepted, documented)** | Convention recorded: new repositories are MSSQL-only with MSSQL fixtures; `NotificationOutboxRepository` is the named legacy exception, "do not copy" (`Component-ConfigurationDatabase.md:69`) |
| S9 | Purge timers' first tick waits a full interval | Low | **Fixed (verified)** | Shared `PurgeTimerSchedule.InitialDelay` (`Commons/Types/PurgeTimerSchedule.cs`) used by all three: `AuditLogPurgeActor.cs:100`, `SiteCallAuditActor.cs:422`, `KpiHistoryRecorderActor.cs:191` |
| S10 | `SqliteAuditWriter.Dispose` sync-over-async | Low | **Won't-fix (accepted)** | Per plan; round-1 itself judged the thread-pool-hop mitigation correct and documented |
| P1 | SiteCalls KPI predicates full-scan (no `TerminalAtUtc` index) | High | **Fixed (verified)** | Filtered `IX_SiteCalls_NonTerminal` on `CreatedAtUtc WHERE [TerminalAtUtc] IS NULL` with `SourceSite`/`SourceNode`/`Status` includes (`SiteCallEntityTypeConfiguration.cs:90-99`; migration `20260709110614_AddSiteCallsNonTerminalIndex.cs`) |
| P2 | `GetExecutionTreeAsync` scans entire AuditLog | High | **Fixed (verified)** | Edges anchor bounded to `[rootFirst − 1h, rootFirst + 7d)` (`AuditLogRepository.cs:682-683, 756, 846-852`); row-less-root degenerate case deliberately keeps the unbounded form |
| P3 | AuditLog clustered key leads with random GUID | Medium | **Deferred (accepted, tracked)** | Recorded as a tracked benchmark follow-up with the partition-scheme rationale (`Component-ConfigurationDatabase.md:70`) |
| P4 | KpiSample volume + unbatched purge | Medium | **Fixed (purge) / Deferred (cadence, tracked)** | Time-sliced (≤1h/DELETE) batched purge (`KpiHistoryRepository.cs:65-84`), backed by the standalone `IX_KpiSample_Captured` index (`KpiSampleEntityTypeConfiguration.cs:54-55`); per-node sampling cadence recorded as a tracked follow-up (`Component-ConfigurationDatabase.md:71`) |
| P5 | Catch-all `(@p IS NULL OR col=@p)` predicates | Medium | **Fixed (verified)** | `OPTION (RECOMPILE)` on the SiteCalls query page (`SiteCallAuditRepository.cs:236`) |
| P6 | Outbox KPI 5-7 round trips; unbounded oldest materialization; offset paging | Medium | **Fixed (KPIs) / Deferred (paging, tracked)** | Global/per-site/per-node snapshots are each one grouped conditional-aggregation query with server-side `MIN`-over-CASE — no entity materialization (`NotificationOutboxRepository.cs:244-270, 286-315, 330-374`); offset→keyset page conversion deferred per plan, recorded in `Component-ConfigurationDatabase.md:71` |
| P7 | Non-persisted `IngestedAtUtc` computed column | Low | **Fixed (guard)** | "NEVER use in a WHERE predicate" comment guard (`AuditLogEntityTypeConfiguration.cs:159-164`) |
| P8 | `MarkForwarded/ReconciledAsync` near SQLite param limit | Low | **Fixed (verified)** | ≤500-id chunks inside one transaction, both methods (`SqliteAuditWriter.cs:61-63, 657-673, 759-775`) |
| C1 | Design docs drifted (enums, Teams, per-channel, SiteCalls schema) | Medium | **Fixed (verified)** | `AuditChannel` 5 values / `AuditKind` 15 incl. `ReconciliationAbandoned` (`Component-Commons.md:46-47`); `NotificationType` Email+Sms with Teams-dropped rationale (:90); per-channel retention correctly described (`Component-ConfigurationDatabase.md:110, 408`); SiteCalls schema rewritten to shipped Channel/AuditStatus shape with explicit "no Kind/TargetSummary/provenance columns" (`Component-SiteCallAudit.md:41-72`); no load-bearing "Teams" references remain |
| C2 | `AuditLogRow` lives in ConfigurationDatabase, not Commons | Medium | **Deferred (accepted, documented)** | Deviation documented as intentional persistence-only projection of the external `ZB.MOM.WW.Audit.AuditEvent` contract, "do not lift it into Commons" (`Component-Commons.md:51`) |
| C3 | Mixed timestamp CLR types | Low | **Deferred (accepted, documented)** | Convention recorded: new tables use UTC `DateTime` + converter; `Notification`'s `DateTimeOffset` is the documented legacy exception (`Component-ConfigurationDatabase.md:68`) |
| C4 | Uncharted KPI metric names as string literals | Low | **Fixed (verified)** | `KpiMetrics.NotificationOutbox.StuckCount`/`OldestPendingAgeSeconds` constants consumed by the source (`NotificationOutboxKpiSampleSource.cs:39-43`) — but see new finding R4: the pattern partially recurs inside `KpiMetricAggregationCatalog` |
| U1 | `backlogTotal` KPI history hardwired to zero | High (underdev.) | **Fixed (verified)** | `IAuditBacklogProvider` seam consumed with snapshot fallback (`AuditLogKpiSampleSource.cs:47, 66, 95`); `CentralHealthAuditBacklogProvider` registered centrally (`HealthMonitoring/ServiceCollectionExtensions.cs:72`); test `AuditLogKpiSampleSourceTests.cs` |
| U3 | Hash-chain / Parquet archival | — | **Deferred (accepted)** | Unchanged v1.x deferral; no drift |
| U4 | SecuredWrite rows leave `SourceNode` NULL | — | **Fixed (elsewhere, verified seam)** | `CentralAuditWriter` stamps `SourceNode` via optional `INodeIdentityProvider` (`Central/CentralAuditWriter.cs:47, 80`) — landed under plan 07 as predicted; doc under-promise on per-channel keys fixed in C1 pass |
| U5 | Reconciliation keyset upgrade untracked | — | **Fixed (SiteCalls) / Deferred (AuditLog pull, tracked)** | SiteCalls keyset shipped (see S7); `PullAuditEvents` remains inclusive-`>=` (idempotent on `EventId`) with the deferral recorded (`Component-ConfigurationDatabase.md:71`) |
| U6 | Dispatcher throughput ceiling (sequential delivery) | — | **Fixed (verified)** | `SemaphoreSlim`-gated `Task.WhenAll` with per-notification DI scope, `MaxParallelDeliveries` default 4 clamped ≥1 (`NotificationOutboxActor.cs:288-408`; `NotificationOutboxOptions.cs:62-69`); fault isolation per delivery so `WhenAll` never throws |
| U7 | Test gaps mirror findings | — | **Fixed (verified)** | Retention tests exist (S1 row); `PartitionPurgeTests` extended for the timeout path per plan; backlog-provider test exists |
| X1 | Reconciliation dials site NodeA only (owned from report 08) | — | **Fixed (verified)** | `SiteEntry.FallbackGrpcEndpoint` populated from NodeB (`Central/SiteEnumerator.cs:34-41`); both pull clients retry once against the fallback on transport faults only, never on mapping faults, skipping when cancelled (`GrpcPullSiteCallsClient.cs:110-127, 172-211`; `GrpcPullAuditEventsClient.cs:194`) |
**Failure scenario:** a long-lived site node accumulates every Forwarded/Reconciled audit row forever. At even modest script activity (10 events/s ≈ 860k rows/day) the SQLite file grows by GBs per month; `GetBacklogStatsAsync`'s `COUNT(*)` stays cheap (it filters `Pending`), so nothing surfaces on the health dashboard — the disk just fills. The `auto_vacuum = INCREMENTAL` pragma (`SqliteAuditWriter.cs:149`) is set up for a purge that never runs.
### [High] `SwitchOutPartitionAsync` — full-table unique-index rebuild inside one transaction, no command timeout
`src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/AuditLogRepository.cs:257-332`: the monthly partition switch drops `UX_AuditLog_EventId`, switches, then `CREATE UNIQUE NONCLUSTERED INDEX UX_AuditLog_EventId ON dbo.AuditLog (EventId)` — a scan+sort over **every remaining row of the table** (up to 365 days of audit events), all inside a single transaction executed via `ExecuteSqlRawAsync`.
Three compounding problems:
1. **Default 30s command timeout.** No `CommandTimeout` is configured anywhere (`ServiceCollectionExtensions.cs:29-34` sets only `UseSqlServer(connectionString)`; repo-wide grep confirms no `CommandTimeout` on the data layer). Once the table is large enough that the index rebuild exceeds 30 seconds, the batch times out, the CATCH block rolls back and re-`THROW`s, `AuditLogPurgeActor.OnTickAsync` logs and moves on (`AuditLogPurgeActor.cs:198-206`) — and the same failure repeats every daily tick forever. Retention silently stops; the table grows past 365 days; the rebuild gets *slower* each day. This is a self-reinforcing failure with no alerting beyond an error log line.
2. **Write blockage.** The transaction holds Sch-M from the DROP INDEX through the CREATE INDEX; all ingest (`InsertIfNotExistsAsync`, dual-writes, direct writes) blocks for the full rebuild duration. The actors tolerate this (rows stay Pending at sites), but central direct-write paths (Inbound API middleware, dispatcher audit emission) will eat the latency or the swallow-and-count path.
3. **Uniqueness window.** While `UX_AuditLog_EventId` is absent, the `IF NOT EXISTS` probe becomes an unindexed scan and single-column EventId uniqueness is enforced only via the composite PK. In practice a retried event carries the same `OccurredAtUtc` so the PK still dedupes; the residual risk is small but the per-insert scan cost during the window is not.
**Recommendation:** set a long `CommandTimeout` for the maintenance path (or run the rebuild with `ONLINE = ON` where edition allows, outside the switch transaction), and monitor purge success as a health metric, not just a log line. Long term, consider making `UX_AuditLog_EventId` unnecessary (see the clustered-key finding under Performance).
### [High] Append-only DB-role enforcement is DDL-only — one principal runs writer and maintenance paths
The migrations create `scadabridge_audit_writer` (INSERT+SELECT, DENY UPDATE/DELETE) and `scadabridge_audit_purger` (SELECT + ALTER ON SCHEMA::dbo) — `20260520142214_AddAuditLogTable.cs:146-158`, re-granted in `20260602174346_CollapseAuditLogToCanonical.cs:212-223`. But the application registers **one** `ScadaBridgeDbContext` with **one** connection string (`ServiceCollectionExtensions.cs:29-45`); the same principal executes `InsertIfNotExistsAsync` (writer path), `PurgeChannelOlderThanAsync`'s `DELETE TOP` (`AuditLogRepository.cs:409-410`), `BackfillSourceNodeAsync`'s `UPDATE TOP` (`AuditLogRepository.cs:851-852`), and the switch-out DDL. Consequences:
- If the runtime login were actually placed in `scadabridge_audit_writer`, the DENY would break the purge/backfill on the same connection. So in any real deployment the login must hold INSERT+UPDATE+DELETE+DDL — meaning the append-only guarantee is enforced **only** by the CI grep guard (`tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/AuditLogAppendOnlyGuardTests.cs`), not by the database.
- The purger role as granted cannot even perform the purge: `CREATE TABLE` (staging) requires the database-level `CREATE TABLE` permission, which `ALTER ON SCHEMA::dbo` alone does not confer.
`Component-AuditLog.md:417-424` presents the role separation as the enforcement mechanism ("The application accesses AuditLog via a dedicated DB role"). Spec and code have drifted: either wire a second connection string/principal for the maintenance path (and fix the purger grants), or re-document enforcement honestly as "CI guard + code review, roles provided for DBAs who choose to segregate principals."
### [Medium] SiteCalls monotonic upsert drops mid-lifecycle progress (RetryCount / LastError frozen at rank)
`SiteCallAuditRepository.cs:30-40` ranks `Attempted == Skipped == 2` and the UPDATE fires only when `incomingRank > storedRank` (`:128-138`). A cached call retrying 20 times emits 20 `Attempted` telemetry packets with increasing `RetryCount` and fresher `LastError`/`HttpStatus` — every one after the first is a **no-op** ("first-write-wins at each rank" per the comment). The Central UI Site Calls page therefore shows `RetryCount = 1` and the first error for the entire retry phase, until the terminal packet lands. The spec's intent (`Component-SiteCallAudit.md:76-78`, "upsert-on-newer-status... at-least-once and out-of-order telemetry are harmless") is about status regression, not about freezing progress fields. A safe refinement: within equal rank, apply the update when `incoming.UpdatedAtUtc > stored.UpdatedAtUtc` (still idempotent and regression-proof).
### [Medium] No EF Core connection resiliency (`EnableRetryOnFailure`)
`ServiceCollectionExtensions.cs:31` configures `UseSqlServer(connectionString)` with no execution strategy. The audit/dispatcher actors compensate with their own retry loops (rows stay Pending, next tick retries), so the backbone self-heals — but every *read-side* path (KPI requests, outbox queries, `Notify.Status` round-trips, execution-tree queries) surfaces transient faults (failover of the SQL instance, connection pool churn) directly to the caller as a failed response. One line of `sqlOptions.EnableRetryOnFailure(...)` would cover the fleet. (Note: retry strategies and user-initiated transactions interact — `OnCachedTelemetryAsync`'s `BeginTransactionAsync` would need `CreateExecutionStrategy` wrapping — so adopt deliberately.)
### [Medium] Operator Retry on a parked notification emits no audit row
`Component-NotificationOutbox.md:128`: "Operator Retry/Discard still mutates only the Notifications row, and each transition emits the corresponding `Notification.Attempt` / `Notification.Terminal` audit row." `DiscardAsync` does emit (`NotificationOutboxActor.cs:1021`), but `RetryAsync` (`NotificationOutboxActor.cs:951-977`) resets Parked→Pending with **no** audit emission. Forensically, a parked notification that an operator retried and that then delivered shows an unexplained `Parked → Attempted → Delivered` sequence with no record of who un-parked it or when. Same gap for `SiteCallAudit`'s retry/discard relay (site-side telemetry covers the state change but not the operator identity).
### [Medium] Reconciliation edge cases are detected but dead-ended
- **Single-timestamp cursor pin** (`SiteCallAuditActor.cs:636-654`): when > `ReconciliationBatchSize` rows share one `UpdatedAtUtc` the cursor cannot advance; the code correctly detects and logs it, but the backlog tail then **never** reconciles ("will not reconcile until those rows' timestamps differ") and nothing surfaces beyond a log Warning — no health metric, no stalled event (unlike the sibling `SiteAuditReconciliationActor`, which publishes `SiteAuditTelemetryStalledChanged`). The fix the log message itself names — a composite `(timestamp, id)` keyset in the pull contract — is deferred without a tracking artifact.
- **Permanent row abandonment** (`SiteAuditReconciliationActor.cs:283-293`): after 5 failed inserts a row is dropped with `LogCritical` and the cursor advances. For a compliance-oriented audit log, "we permanently lost EventId X" deserves a durable record (e.g., a synthetic `Skipped` audit row or a health counter), not only a log line that rotates away.
### [Medium] `NotificationOutboxActor` ingest ack path can ack before the row is *committed* under a SQLite test provider mismatch — verify provider parity
`NotificationOutboxRepository.InsertIfNotExistsAsync` maintains two dialects (`NotificationOutboxRepository.cs:70-95` SQLite `INSERT OR IGNORE`, `:99-117` T-SQL `IF NOT EXISTS`). This is correct today but is production code carrying a test-only branch, and the two branches have subtly different concurrency semantics (the SQLite path can never observe the duplicate-key race the SQL Server path handles). The AuditLog and SiteCall repositories chose the opposite strategy (MSSQL-only SQL, MSSQL-backed test fixtures). Pick one convention; the divergence invites the next repository to guess.
### [Low] Purge actors' first tick waits a full interval
`AuditLogPurgeActor.PreStart` (`:95-100`), `SiteCallAuditActor.StartPurgeTimer` (`:380-386`), and `KpiHistoryRecorderActor` purge timer (`KpiHistoryRecorderActor.cs:124-127`) all use `initialDelay = interval` (24h). A central node that restarts daily (crash loop, scheduled recycling) would *never* purge. Low likelihood, cheap fix (short initial delay; purges are idempotent).
### [Low] `SqliteAuditWriter.Dispose` sync-over-async
`SqliteAuditWriter.cs:890-894` blocks on `DisposeAsync` via `Task.Run(...).GetAwaiter().GetResult()`. The mitigation (thread-pool hop) is correct and documented; flagged only because it remains a blocking dispose on a hot-path component.
**Disposition counts:** 19 Fixed (verified), 2 Fixed-with-documented-deferral remainder (P4, P6, U5 counted once each as Fixed), 6 Deferred (accepted, all now documented/tracked: S8, P3, C2, C3, U3), 1 Won't-fix (accepted: S10), 0 Not fixed, 0 Regressed. No plan claim was found to be false.
---
## Findings — Performance
## New Findings — KPI Rollup Subsystem (post-initiative, 2026-07-10)
### [High] SiteCalls KPI predicates full-scan the table — no index on `TerminalAtUtc`
The rollup design is sound where it was thought through — the gauge-vs-rate catalog with its safe-Gauge default and the corrected `*LastHour` classification (`KpiMetricAggregation.cs:85-96`) is genuinely careful work, the `HasFilter(null)` suppression of EF's default filtered unique index (`KpiRollupHourlyEntityTypeConfiguration.cs:57-63`) dodges a real null-ScopeKey uniqueness trap, and the exclusive-upper-bound / in-progress-hour exclusion is correct. The problems are concentrated where the periodic-fold assumptions meet the backfill path.
The "schema-honest" non-terminal predicate is `TerminalAtUtc IS NULL` (`SiteCallAuditRepository.cs:224-232`), used by `ComputeKpisAsync` (buffered `:243-244`, stuck `:259-260`, oldest `:262-268`), the per-site and per-node variants (`:286-307`, `:336-362`), and `PurgeTerminalAsync` (`:213-218`). But the table's only indexes are `(SourceSite, CreatedAtUtc)` and `(Status, UpdatedAtUtc)` (`SiteCallEntityTypeConfiguration.cs:79-87`) — **nothing covers `TerminalAtUtc`**. Every such count is a clustered scan of a table retained for 365 days (`SiteCallAuditOptions.RetentionDays = 365`, potentially millions of rows for busy sites). These queries run:
### [High] R1 — The startup backfill folds the entire raw-retention window in one unbounded pass, on every failover
- every 60 s from `SiteCallAuditKpiSampleSource` (global + per-site + per-node = ~9 scanning queries per recorder tick),
- on every Health-dashboard poll (10 s cadence per the UI design) via `SiteCallKpiRequest`.
`FoldHourlyRollupsAsync` loads its whole window with a single **tracked** `ToListAsync` and then issues one `FirstOrDefaultAsync` per (series, hour) group plus one giant `SaveChanges` (`ConfigurationDatabase/Repositories/KpiHistoryRepository.cs:104-106, 133-139, 167`). Its own comment says "The caller passes a small trailing lookback (e.g. 3 h), so the window is bounded" — but its second caller is the one-shot backfill, which passes `[TruncateToHour(now) − RetentionDays, TruncateToHour(now))` — **90 days by default** (`KpiHistory/KpiHistoryRecorderActor.cs:510-511`).
**Failure scenario:** at ~2M rows, each scan is hundreds of ms of CPU/IO; the KPI backbone alone drives a near-continuous scan load on the central DB, and the KPI Ask timeouts start failing intermittently. **Fix:** a filtered index `CREATE INDEX IX_SiteCalls_NonTerminal ON SiteCalls (CreatedAtUtc) WHERE TerminalAtUtc IS NULL` (plus `SourceSite`/`SourceNode` includes) makes every one of these predicates a small seek, since the non-terminal population is the live queue, not the archive.
At round-1's own fleet estimate (~500 samples/min ≈ 700k rows/day for 10 sites), the backfill materializes tens of millions of tracked `KpiSample` entities into one `List<>` — a multi-GB allocation on the active central node that no `catch` can contain if it OOMs — then runs ~(series-count × 2,160 hours) single-row upsert probes and commits everything in one `SaveChanges`. Even a modest single-site deployment (≈26 samples/min) materializes ~3.4M tracked rows. And because `_backfillDone` is per actor lifetime (`KpiHistoryRecorderActor.cs:129-136`), **every singleton failover/restart re-runs the full-window fetch** — the pre-existing rollup rows shrink the writes but not the raw read. While it grinds, `_backfillInFlight` also blocks every periodic fold (`:435-441`), so fresh hours stall behind the historical pass. `Component-KpiHistory.md:93` calls the backfill "cheap and safe to re-run" — the "safe" is true (idempotent), the "cheap" is not.
### [High] `GetExecutionTreeAsync` scans the entire AuditLog per invocation
**Fix:** slice the backfill in the actor into bounded windows (e.g. one day per `FoldHourlyRollupsAsync` call, oldest-first, yielding between slices) — the repository method then never sees more than 24 hours; this also releases the fold path between slices. Alternatively (or additionally) skip the backfill when the newest `KpiRollupHourly.HourStartUtc` is already near `now` (the failover case, which is the common one). This is the same "bound the maintenance pass" discipline Tasks 4/19 applied to every other year-scale sweep in this domain.
`AuditLogRepository.cs:704-756`: the recursive CTE's `Edges` anchor is `SELECT DISTINCT ExecutionId, ParentExecutionId FROM dbo.AuditLog WHERE ExecutionId IS NOT NULL`. Neither `IX_AuditLog_Execution` nor `IX_AuditLog_ParentExecution` covers the *other* column, so satisfying the DISTINCT requires scanning an entire index (or the clustered index) across **all partitions, all 365 days** — on every `audit tree` CLI call and every UI execution-tree drilldown. The upward walk (`:652-677`) is fine (seeks on `IX_AuditLog_Execution`), and the STRING_AGG subqueries seek per node; it is the edge materialization that is O(table).
### [Medium] R2 — Fold fetch is change-tracked (`AsNoTracking` missing)
**Fix options:** (a) add `ParentExecutionId` as an INCLUDE on `IX_AuditLog_Execution` (index-only DISTINCT scan — better but still O(table)); (b) constrain the Edges CTE to a time window derived from the root's `FirstOccurredAtUtc` (execution trees span minutes, not years — a `OccurredAtUtc >= root - slack` predicate gets partition elimination); (c) seed the recursion by seeking children per level (`WHERE ParentExecutionId IN (…)`) instead of pre-materializing a global edge set.
The sample fetch in `FoldHourlyRollupsAsync` (`KpiHistoryRepository.cs:104-106`) reads `KpiSample` rows it will never modify, without `AsNoTracking()`. Even on the healthy periodic path (3-hour lookback), that is ~5k-90k read-only entities registered in the change tracker every hour, all re-scanned by `DetectChanges` on the final `SaveChanges` alongside the rollup entities that actually changed. One-word fix; it also cuts the R1 backfill's memory overhead roughly in half (tracking snapshots) even before R1 is properly sliced.
### [Medium] AuditLog clustered key leads with a random GUID
### [Medium] R3 — Rate metrics change units ~60× across the raw/rollup routing boundary, and the bucketer re-introduces the fold error the catalog warns about
`AuditLogEntityTypeConfiguration.cs:172` / migration `CollapseAuditLogToCanonical.cs:75`: clustered PK `(EventId, OccurredAtUtc)` with `EventId = Guid.NewGuid()`. Inserts land randomly across the current month's partition → page splits, fragmentation, and larger-than-necessary buffer pool footprint on the system's highest-volume table. The composite order also means the PK is useless for time-range queries (all served by secondary indexes). A clustered key of `(OccurredAtUtc, EventId)` would give append-locality and make `IX_AuditLog_OccurredAtUtc` redundant; `EventId` uniqueness would still need the non-aligned unique index (which exists anyway). This interacts with the switch-out dance either way; worth a deliberate benchmark before the table gets big — changing it later is a full rebuild.
`KpiHistoryQueryService.FetchSeriesAsync` routes windows ≤168h to raw per-minute samples and wider windows to hourly rollups (`CentralUI/Services/KpiHistoryQueryService.cs:107-116`). For Gauge metrics both paths agree (last-value either way). For **Rate** metrics they do not: the raw path plots per-minute deltas, the rollup path plots **sum-per-hour** values — so the same `deliveredLastInterval` chart jumps by up to ~60× in magnitude when the operator widens the range from 7d to 30d, with no normalization in the bucketer or `KpiTrendChart`. Compounding it, `KpiSeriesBucketer.Bucket` takes **last-value-per-bucket** on top of the rollup series (`Component-KpiHistory.md:151`): a 90d/200-point chart keeps 1 of ~11 hourly sums per bucket and discards the rest — precisely the "keep one delta, discard the others" mistake `KpiMetricAggregationCatalog`'s doc warns silently corrupts rate trends (`KpiMetricAggregation.cs:26-28`), re-introduced one layer up. **Fix:** teach the bucketer (or the query service) a per-metric reduction — sum-per-bucket for Rate series — and normalize both paths to a common unit (e.g. per-hour rate) so the boundary is visually seamless; the catalog needed to drive this already exists in Commons.
### [Medium] KpiSample volume and unbatched purge
### [Low] R4 — `KpiMetricAggregationCatalog` re-creates the private-metric-literal drift hazard C4 just fixed
Per recorder tick (60 s), sample count ≈ NotificationOutbox(5 × (1 global + N sites + M nodes)) + SiteCallAudit(6 × (1 + N + M)) + AuditLog(3) + SiteHealth(12 × N). For 10 sites / 20 site-nodes this is ~500 rows/min ≈ **700k rows/day ≈ 60M+ rows at the 90-day default retention** — with a 5-column composite index (`IX_KpiSample_Series`) maintained on every insert. Meanwhile `PurgeOlderThanAsync` is a single unbatched `ExecuteDeleteAsync` (`KpiHistoryRepository.cs:65-71`): in steady state it deletes ~700k rows in **one transaction** daily — a lock/log spike on the same DB serving the operational tables; after any purge outage the catch-up delete is worse (and subject to the same default 30s command timeout as the partition switch). Batch the delete (`TOP (n)` loop, mirroring `PurgeChannelOlderThanAsync`), and consider whether per-node scope sampling every 60 s is worth 40% of the row volume.
The catalog hardcodes `"deliveredLastInterval"` (SiteCallAudit), `"alarmEvalErrors"`, and `"eventLogWriteFailures"` as its own private literals because the emitting sources keep those names private (`Commons/Types/Kpi/KpiMetricAggregation.cs:60-64`). If an emitter's literal ever drifts, the pair silently resolves to Gauge and the hourly sum quietly becomes a last-value — no error, just an under-counted long-range trend. The Gauge default makes this degradation safe-ish, but silent. A lock-in test asserting every `RatePairs` entry matches a metric actually emitted by the corresponding `IKpiSampleSource` (or promoting the three literals to `KpiMetrics`) closes it.
### [Medium] Catch-all `(@p IS NULL OR col = @p)` predicates in SiteCalls query
### [Low] R5 — Overlapping-singleton fold race faults an entire fold batch
`SiteCallAuditRepository.QueryAsync` (`:186-202`) builds one static SQL shape where every filter is `({param} IS NULL OR col = {param})`. This is the classic optional-parameter anti-pattern: SQL Server compiles one plan for all filter combinations, generally a scan ordered by `CreatedAtUtc DESC`. Fine while the table is small; on a year of rows, a filtered query (e.g. `Status = 'Parked'`) will scan rather than use `IX_SiteCalls_Status_Updated`. `OPTION (RECOMPILE)` or dynamic predicate composition (as `AuditLogRepository.QueryAsync` does with LINQ, `:117-199`) fixes it.
### [Medium] Notification outbox query page uses offset paging + double query + unsargable LIKE
`NotificationOutboxRepository.QueryAsync` (`:165-228`): `COUNT` + `Skip((page-1)*size)` — deep pages re-scan; `Subject.Contains(keyword)` translates to `LIKE '%kw%'` (unsargable). The sibling AuditLog and SiteCalls pages both use keyset paging. For a 365-day table this page will degrade first in the UI. Also `ComputeKpisAsync`/`ComputePerSiteKpisAsync`/`ComputePerNodeKpisAsync` issue 5-7 sequential round trips each (`:239-283`, `:286-334`, `:337-391`), and the per-site/per-node "oldest" computations materialize every non-terminal row into memory (`:312-318`, `:368-375`) — bounded by live-queue size in normal operation, unbounded during a delivery outage (exactly when the KPI page gets watched). A single `GROUP BY` with conditional aggregation would serve each snapshot in one query.
### [Low] Non-persisted `IngestedAtUtc` computed column re-parses `DetailsJson` per row read
`AuditLogEntityTypeConfiguration.cs:158-164`: every materialization of `AuditLogRow` (e.g. `QueryAsync` pages) evaluates `SWITCHOFFSET(CAST(JSON_VALUE(DetailsJson …)))` over an `nvarchar(max)` per row. Page-sized reads make this tolerable; just avoid ever using `IngestedAtUtc` in a predicate (it would force a full-table JSON parse).
### [Low] `MarkForwardedAsync` / `MarkReconciledAsync` build per-id IN lists
`SqliteAuditWriter.cs:644-664, 735-753`: parameter-per-id is safe at the default telemetry batch of 256 (`SiteAuditTelemetryOptions.BatchSize = 256`) but silently approaches SQLite's 999-parameter limit if an operator raises the batch size ~4×. Worth a guard or chunking.
All groups in a fold commit through one `SaveChanges` (`KpiHistoryRepository.cs:167`). During a failover overlap window (old incarnation's in-flight fold vs. the new node's first fold), both may `Add` the same `(series, hour)` row; the unique `IX_KpiRollupHourly_Series` then faults the loser's **whole** `SaveChanges`, discarding every group in that pass, not just the contested row. Self-heals on the next tick (idempotent re-fold), and the in-actor guards prevent same-node races, so this is noise rather than loss — but worth knowing the failure grain is the pass, not the row, when reading fold-failure logs after a failover.
---
## Findings — Conventions
## New Findings — Other post-baseline code
### [Medium] Design docs have drifted from the code in several load-bearing places
### [Medium] R6 — `PurgeTerminalAsync` on SiteCalls is the one remaining unbatched, unindexed year-scale maintenance DELETE
- `Component-Commons.md:47` lists `AuditKind` with 10 values; the code has 14 (`Commons/Types/Enums/AuditKind.cs:10-26`, adds the four `SecuredWrite*` kinds). `:46` lists `AuditChannel` with 4 values; the code has 5 (`AuditChannel.cs:8-15`, adds `SecuredWrite`).
- `Component-Commons.md:89` still says `NotificationType`: "Email — currently the only value; Teams and other channels are planned but not yet present" — `Sms` shipped 2026-06-19 and the Teams plan was dropped (per CLAUDE.md).
- `Component-ConfigurationDatabase.md:58` describes the NotificationList `Type` discriminator as "`Email` / `Teams` / …".
- `Component-ConfigurationDatabase.md:399` states AuditLog retention is a "single global value in v1, **no per-channel overrides**", directly contradicting the shipped `PerChannelRetentionDays` feature (and `Component-AuditLog.md:396-411`, which documents it).
- `Component-SiteCallAudit.md:44-51` describes the `SiteCalls` schema with `Kind`/`TargetSummary`/`Provenance` columns and a `Pending/Retrying/...` status set; the shipped schema (`Component-ConfigurationDatabase.md:64` and `SiteCallEntityTypeConfiguration.cs`) uses `Channel`/`Target` and `AuditStatus`-derived strings with no instance/script provenance columns. The ConfigDb doc is current; the SiteCallAudit doc's own table section is stale.
`SiteCallAuditRepository.PurgeTerminalAsync` is still a single unbatched `DELETE FROM dbo.SiteCalls WHERE TerminalAtUtc IS NOT NULL AND TerminalAtUtc < cutoff` (`SiteCallAuditRepository.cs:247-252`). Nothing indexes non-null `TerminalAtUtc` — the new `IX_SiteCalls_NonTerminal` is filtered to `IS NULL` and unusable here — so the daily purge full-scans the 365-day table, and its steady-state day-of-terminal-rows (plus any catch-up after an outage) deletes in one transaction under the default command timeout. This is exactly the shape Task 19 batched for `KpiSample` and Task 4 time-boxed for `AuditLog`; SiteCalls was left out. Volume is lower than either sibling, so Medium not High — but it inherits both prior failure modes (lock/log spike; a 30s-timeout loop after an outage backlog). Fix: `TOP (n)` loop like `PurgeChannelOlderThanAsync` plus either an index on `TerminalAtUtc` or a `(Status, UpdatedAtUtc)`-driven predicate.
Per the repo's own rule ("design doc and code travel together"), these should be reconciled in one pass.
### [Low] R7 — `SiteAuditRetentionService.StopAsync` surfaces the loop's cancellation to the host
### [Medium] `AuditLogRow` entity lives in ConfigurationDatabase, not Commons
`src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Entities/AuditLogRow.cs` breaks the "POCO entities in Commons / mappings in ConfigurationDatabase" convention that every sibling table follows (`SiteCall`, `Notification`, `KpiSample`, `AuditLogEntry` are all Commons POCOs). There is a defensible rationale — the *contract* type is the external `ZB.MOM.WW.Audit.AuditEvent` and `AuditLogRow` is a persistence-only projection — but the deviation is undocumented in `Component-Commons.md`/`Component-ConfigurationDatabase.md`, and `IAuditLogRepository` (in Commons) consequently traffics in `AuditEvent` while `ISiteCallAuditRepository` traffics in a Commons entity. Document the deviation or move the row shape.
### [Low] Mixed timestamp CLR types across sibling tables
`Notification` uses `DateTimeOffset` (`Commons/Entities/Notifications/Notification.cs:68-77`) while `SiteCall`, `KpiSample`, and `AuditLogRow` use UTC `DateTime`. Both satisfy the UTC convention, but the `DateTimeOffset` choice is what forced the awkward in-memory Min reductions and "converter makes SQL Min awkward" workarounds in `NotificationOutboxRepository` (`:262-274`, `:308-318`) that its `DateTime`-based siblings didn't need. New tables should standardize (the `DateTime`+`Utc`-converter pattern in `AuditLogEntityTypeConfiguration.cs:32-40` is the cleaner precedent).
### [Low] Uncharted KPI metric names are string literals while charted ones use the catalog
`NotificationOutboxKpiSampleSource.cs:40-43`: `"stuckCount"` / `"oldestPendingAgeSeconds"` are inline literals with a comment explaining the split, while charted metrics use `KpiMetrics.*` constants. Consistent cataloging (all persisted names in `KpiMetrics`) would remove a rename hazard on values that are, per the code's own comment, "persisted; do not rename."
### Positive observations (conventions done right)
- Options pattern with dedicated validators and clamped intervals everywhere (`SiteCallAuditOptions.ResolvedReconciliationInterval` guarding the Akka zero-interval spin footgun, `AuditLogOptionsValidator`, `KpiHistoryOptionsValidator`).
- The `Sender`-capture-before-await and `Context.System.EventStream`-capture-before-await disciplines are applied consistently across all five actors — a subtle Akka correctness point handled uniformly.
- Duplicate-key race hardening (2601/2627 swallow) is uniform across all three idempotent-insert repositories, with the check-then-act window explicitly documented at each site.
- `AuditService` stages within the caller's unit of work exactly as the spec's transactional-audit guarantee requires, with cycle-tolerant serialization that can't roll back the business operation (`AuditService.cs:20-24, 71-85`).
- The dual-write transaction (`AuditLogIngestActor.cs:262-311`) is genuinely atomic per entry with per-entry isolation, and `IngestedAtUtc` is stamped from one instant across both rows.
`SafePurgeAsync` deliberately rethrows `OperationCanceledException` (`Site/SiteAuditRetentionService.cs:106-110`), but the two `await SafePurgeAsync(ct)` call sites in `RunLoopAsync` (:76, :89) are outside any try/catch, so a shutdown that lands mid-purge cancels the `_loop` task — and `StopAsync` returns that canceled task directly (:122-126), making the host's `StopAsync` await throw `TaskCanceledException`. The generic host logs and continues, so this is shutdown-log noise, not a hang — but the sibling `SiteAuditBacklogReporter` pattern it mirrors is worth aligning: catch OCE at the loop level (or `Task.WhenAny`-guard in `StopAsync`).
---
## Underdeveloped Areas
## What's genuinely good
1. **`backlogTotal` KPI history series is permanently zero.** `AuditLogRepository.GetKpiSnapshotAsync` hardcodes `BacklogTotal: 0L` (`AuditLogRepository.cs:618`) — documented in the interface as "left at zero" — and the Central UI's live tile compensates by summing site backlog reports (`CentralUI/Services/AuditLogQueryService.cs:125-155`). But `AuditLogKpiSampleSource` persists the repo snapshot **directly** (`AuditLogKpiSampleSource.cs:69-85`), so every recorded `backlogTotal` sample is 0, and the Audit Log page's backlog trend chart (`AuditLogPage.razor.cs:297`) renders a flat zero line regardless of actual backlog. The one KPI whose *history* matters most during an outage post-mortem (how big did the backlog get, how fast did it drain) records nothing. Fix: inject the same health-aggregator read the query service uses into the sample source.
2. **Site SQLite retention purge** — spec'd, commented as future, absent (see Stability [High] above).
3. **Hash-chain tamper evidence (T1) and Parquet archival (T2)** — explicitly deferred to v1.x per `Component-AuditLog.md:433-437`; the `verify-chain` CLI is a documented no-op placeholder. Consistent with plan; no drift.
4. **SecuredWrite audit rows leave `SourceNode` NULL** — documented as a logged follow-up (`Component-AuditLog.md:154-157`); also note `PerChannelRetentionDays` validation accepts `SecuredWrite` as a key (the enum has it) even though the design doc's config section (`:396-399`) enumerates only the four original channels — harmless, but the doc under-promises.
5. **Reconciliation cursor keyset upgrade** — the `(timestamp, id)` composite cursor that would eliminate the single-timestamp pin is acknowledged in a log message (`SiteCallAuditActor.cs:640-654`) but not tracked anywhere; the pull contracts (`CachedCallReconcileRequest`, `PullAuditEvents`) would need an additive field.
6. **Dispatcher throughput ceiling** — notification delivery is strictly sequential within a sweep (batch 100, 10 s interval, blocking SMTP/Twilio per row: `NotificationOutboxActor.RunDispatchPass:321-347`). A slow SMTP endpoint (5 s/attempt) caps throughput at ~1 notification/5 s regardless of batch size, and the in-flight guard means subsequent ticks drop. The spec's "dedicated blocking-I/O dispatcher" (`Component-NotificationOutbox.md:23`) is not literally implemented (delivery runs on the thread pool off the actor thread — functionally acceptable, doc drift), but there is no per-notification parallelism at all. Fine for alarm-notification volume; a burst (alarm storm fanning to hundreds of notifications) drains slowly.
7. **Test coverage** is strong overall (MSSQL-backed integration tests for partition purge, execution trees, outage reconciliation, dual-write; per-actor TestKit suites; options validators tested). Gaps mirror the findings: no test asserts a site-SQLite retention purge (it can't — the feature is missing), no test exercises `SwitchOutPartitionAsync` at a size that would expose the command-timeout failure, and no test asserts `backlogTotal` history is non-zero under backlog.
- **The fix execution matches the fix claims.** All 19 verified-Fixed items are real, tested, and in several cases exceed the plan: the purge-failure path got a dedicated no-op-defaulted counter seam (`IAuditPurgeFailureCounter` + `NoOpAuditPurgeFailureCounter`), and the S1 purge encodes the ForwardState invariant *in the SQL itself* with a param-limit-proof temp table (`SqliteAuditWriter.cs:877-897`).
- **The NodeB-failover pull clients are exemplary defensive code**: transport-fault-only failover (a mapping fault correctly does not retry on the other node, "since a second node would hit the same non-transport fault"), per-row DTO fault isolation, race-safe channel cache with loser-dispose, and the `DateTime.MinValue`/`ToUniversalTime` underflow trap explicitly dodged and documented (`GrpcPullSiteCallsClient.cs:110-127, 236-242, 309-325`).
- **The keyset cursor landed end-to-end with honest legacy behavior**: proto additive field, empty-string-as-null mapping at the server, strict-keyset vs. legacy-inclusive branches in the site store, and a latched pinned-state event that fires only on transitions (`OperationTrackingStore.cs:384-395`, `SiteCallAuditActor.cs:737-746`).
- **Deferral hygiene**: every accepted deferral (S8, P3, P4-cadence, P6-paging, C2, C3, U5) now has a written home in a design doc — the round-1 complaint that dead-ends were "deferred without a tracking artifact" is fully answered.
- **The rollup catalog's semantics section** (`KpiMetricAggregation.cs:85-96`) is a model of the kind of reasoning that prevents silent data corruption: it overrides the plan's tentative Rate classification for the `*LastHour` metrics after checking the actual source semantics, and documents why in place.
- **Execution-strategy adoption was done deliberately**, exactly as round-1 cautioned: the dual-write's user-initiated transaction is wrapped per entry with the idempotency argument stated (`AuditLogIngestActor.cs:263-298`), rather than a blind `EnableRetryOnFailure` toggle.
---
## Underdeveloped areas
## Prioritized Recommendations
1. **Backfill scale testing** — no test exercises `FoldHourlyRollupsAsync` at a window larger than a few hours; the R1 failure mode (90-day materialization) is invisible to the current suites, mirroring how the round-1 partition-switch timeout was invisible until sized. A `SkippableFact` MSSQL test at representative volume, or a unit test asserting the actor slices the backfill, would lock in the R1 fix.
2. **Rate-metric presentation across the routing boundary** (R3) — the storage layer has full fidelity (`MinValue`/`MaxValue`/`SampleCount` are stored but uncharted); the presentation semantics are the unfinished half of the rollup feature.
3. **AuditLog pull keyset** (U5 residue) — tracked, additive, and lower-urgency; fine as-is, noting it here so the tracker's entry doesn't age out.
1. **Implement the site SQLite 7-day retention purge** (daily timer on `SiteAuditTelemetryActor` or a dedicated job; `DELETE` joined on `audit_forward_state.ForwardState IN ('Forwarded','Reconciled')` + age, followed by `PRAGMA incremental_vacuum`) — closes the unbounded-growth hole and honors the spec'd `ForwardState` invariant.
2. **Fix the `backlogTotal` KPI sample source** to read the health aggregator like `AuditLogQueryService` does — one-file change, restores a shipped feature.
3. **Harden `SwitchOutPartitionAsync` operationally**: explicit long `CommandTimeout` on the maintenance commands, a health metric/event on purge *failure* (the success event exists; the failure path is log-only), and an integration test at representative volume.
4. **Add the filtered non-terminal index on `SiteCalls`** (`WHERE TerminalAtUtc IS NULL`) and turn each KPI snapshot into a single conditional-aggregation query — removes the dominant recurring scan load.
5. **Bound the `GetExecutionTreeAsync` edge scan** with a time-window predicate (partition elimination) or per-level child seeks.
6. **Decide the DB-role story honestly**: either a second maintenance connection string/principal (and grant `CREATE TABLE` to the purger), or update `Component-AuditLog.md`/`Component-ConfigurationDatabase.md` to state that append-only enforcement is CI-guard-based with roles offered as an optional DBA hardening.
7. **Allow same-rank freshness updates in the SiteCalls upsert** (`UpdatedAtUtc` tiebreaker) so retrying calls show live RetryCount/LastError.
8. **Batch the KpiSample purge** and reconsider per-node sampling cadence before fleet size multiplies row volume.
9. **Emit an audit row for operator Retry** on parked notifications (and consider operator identity on the SiteCalls relay), matching the spec's transition-audit promise.
10. **Reconcile the stale design-doc passages** (AuditKind/AuditChannel/NotificationType enum lists, `Teams` references, "no per-channel overrides", SiteCalls schema section) in one documentation pass, per the repo's docs-and-code-travel-together rule.
## New-Findings Severity Tally (round 2 only)
| Severity | Count | Findings |
|----------|-------|----------|
| Critical | 0 | — |
| High | 1 | R1 (unbounded rollup backfill, re-run per failover) |
| Medium | 3 | R2 (tracked fold fetch), R3 (rate-unit discontinuity + bucketer re-fold error), R6 (SiteCalls terminal purge unbatched/unindexed) |
| Low | 3 | R4 (catalog literal drift), R5 (fold-batch failover race grain), R7 (retention-service shutdown OCE) |
**Round-2 bottom line:** the round-1 debt is paid — verified, not just claimed. The new debt is small and localized: bound the rollup backfill (R1) and add `AsNoTracking` (R2) before any production deployment accumulates months of `KpiSample` rows, decide the rate-metric presentation (R3) before operators learn to distrust the 30d charts, and give `PurgeTerminalAsync` the same batching its two siblings received (R6).
archreview/05-templates-deployment-transport.md
+84 -115
View File
@@ -1,145 +1,114 @@
# Architecture Review 05 — Design-Time Pipeline: Templates, Deployment, Transport, Script Analysis
# Architecture Review 05 — Design-Time Pipeline: Templates, Deployment, Transport, Script Analysis (Round 2)
Reviewer domain: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine`, `src/ZB.MOM.WW.ScadaBridge.DeploymentManager`, `src/ZB.MOM.WW.ScadaBridge.Transport`, `src/ZB.MOM.WW.ScadaBridge.ScriptAnalysis`, their design docs (`docs/requirements/Component-TemplateEngine.md`, `Component-Transport.md`, `Component-ScriptAnalysis.md`, `Component-DeploymentManager.md`) and test projects. All findings cite code actually read.
**Date:** 2026-07-12 (round 2; round-1 baseline commit `b910f5eb`, re-review at HEAD `8c888f13`)
## Scope
- **Template Engine**: flattening (`Flattening/FlatteningService.cs`), collision detection (`CollisionDetector.cs`), acyclicity (`CycleDetector.cs`), revision hashing (`Flattening/RevisionHashService.cs`), lock rules (`LockEnforcer.cs`), diffs (`Flattening/DiffService.cs`), validation (`Validation/ValidationService.cs`, `Validation/SemanticValidator.cs`, `Validation/ScriptCompiler.cs`), authoring services (`TemplateService.cs`, `TemplateInheritanceResolver.cs`).
- **Deployment Manager**: `DeploymentService.cs` (deploy/lifecycle pipeline, idempotency reconciliation), `OperationLockManager.cs`, `FlatteningPipeline.cs`, `StateTransitionValidator.cs`, `StaleInstanceProbe.cs`, `ArtifactDeploymentService.cs`.
- **Transport**: `Import/BundleImporter.cs` (3,832 lines — load/preview/apply), `Export/BundleExporter.cs`, `Export/DependencyResolver.cs` (skimmed), `Serialization/EntitySerializer.cs`/`EntityDtos.cs`, `Encryption/BundleSecretEncryptor.cs` + `BundleManifestAad.cs`, `Import/LineDiffer.cs`, `Import/ArtifactDiff.cs`, `Import/BundleSessionStore.cs`, `TransportOptions.cs`.
- **Script Analysis**: `ScriptTrustPolicy.cs`, `ScriptTrustValidator.cs`, `RoslynScriptCompiler.cs`, `ScriptCompileSurface.cs`.
- Cross-component seam checked: Inbound API compiled-handler cache (`src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundScriptExecutor.cs`) as a consumer of Transport-imported `ApiMethod` rows; `ManagementActor` native-alarm-source override handlers.
`src/ZB.MOM.WW.ScadaBridge.TemplateEngine`, `src/ZB.MOM.WW.ScadaBridge.DeploymentManager`, `src/ZB.MOM.WW.ScadaBridge.Transport`, `src/ZB.MOM.WW.ScadaBridge.ScriptAnalysis`, their design docs (`docs/requirements/Component-TemplateEngine.md`, `Component-Transport.md`, `Component-ScriptAnalysis.md`, `Component-DeploymentManager.md`), and the test projects. Cross-component seams re-checked: Inbound API compiled-handler cache (`InboundScriptExecutor`) as the consumer of the plan-05/plan-06 invalidation work; `ManagementActor` native-alarm-source override handlers.
## Maturity Verdict
## Method
This slice is **unusually well-engineered in its failure-path plumbing** — the deploy pipeline's query-before-redeploy reconciliation, cancellation-token-safe failure writes, ref-counted operation locks, AES-GCM+AAD bundle crypto with zip-bomb caps and passphrase lockout, and the fused two-pass script trust validator are all production-grade and well beyond typical internal-tool quality. However, the **Transport import apply path has serious correctness holes** that undercut the whole promotion story: imported templates silently lose their inheritance edge, their per-script cadence/timeout fields, and their native alarm sources — three distinct silent-data-loss paths, none covered by the otherwise substantial integration test suite. The flattener has one real algorithmic bug (repeated composed templates lose grandchild members), the revision hash omits native alarm sources (breaking staleness detection for that member class), and the central node's validation path leaks Roslyn script assemblies on every Deployments-page comparison. The core design is sound; the defects are concentrated in the newest surface area (M8 Transport, native alarms) where spec, DTOs, and apply code drifted apart.
1. Re-read the round-1 report (the "silent data loss in Transport import" report: 1 Critical, 7 High, 10 Medium, 5 Low, 7 underdeveloped areas) and `archreview/plans/PLAN-05-templates-deployment-transport.md` (27 fixed / 6 deferred claimed).
2. Verified every round-1 finding's true disposition in current source — code read directly, plan claims not trusted. All dispositions below carry file:line evidence from HEAD.
3. Fresh sweep of `git diff b910f5eb..HEAD` scoped to the four projects (~1,900 insertions across 36 files; 31 commits) for new findings: stability, performance, security, conventions.
**One-line verdict:** Round 1 found a Transport import that silently amputated template inheritance, cadence/timeout, lock flags, and native alarm sources; round 2 finds all 26 actionable findings genuinely fixed in code (0 regressions, 5 recorded-decision deferrals intact), guarded by a reflection round-trip suite that itself caught 5 further bugs — the residue is 1 Medium performance leftover (Expression-trigger Roslyn compiles on read-only paths) and 5 Lows.
---
## Findings — Stability & Correctness
## Round-1 Finding Disposition
### [Critical] Bundle import drops template inheritance edges — derived templates land as root templates
`TemplateDto.BaseTemplateName` (`Serialization/EntityDtos.cs:115`) is populated at export (`Serialization/EntitySerializer.cs:55`) and even diffed in preview (`Import/ArtifactDiff.cs:65`), but **the apply path never consumes it**. `BundleImporter.BuildTemplate` (`Import/BundleImporter.cs:1524-1562`) never sets `ParentTemplateId`; the Overwrite branch (`BundleImporter.cs:1466-1484`) updates only `Description`/`FolderId` and the three child collections; the two second-pass rewires cover alarm→script FKs (`ResolveAlarmScriptLinksAsync`, line 1907) and compositions (`ResolveCompositionEdgesAsync`, line 1993) — there is **no inheritance-edge pass** (`grep -n 'Parent' BundleImporter.cs` returns nothing). The only code that wires `ParentTemplateId` from `BaseTemplateName`, `EntitySerializer.FromBundleContent` (`EntitySerializer.cs:357-360`), is invoked **only from tests** (`tests/.../EntitySerializerTests.cs`).
Legend: **F** = Fixed (verified in source), **P** = Partially fixed, **D** = Deferred (accepted, recorded), **N/A** = no longer applicable.
**Failure scenario:** export `Pump` + derived `Pump.WaterPump` from dev (the manifest explicitly records `dependsOn: ["Template:Pump"]` — `Component-Transport.md:79-80`); import into prod. `WaterPump` is created with `ParentTemplateId = null`. `FlatteningPipeline.BuildTemplateChainAsync` (`DeploymentManager/FlatteningPipeline.cs:176-192`) produces a one-element chain; every inherited attribute/alarm/script vanishes from the flattened config. Because validation validates *what's there*, an instance with no bindings on the missing members can deploy "successfully" with a silently amputated configuration. Neither `tests/ZB.MOM.WW.ScadaBridge.Transport.Tests` nor `Transport.IntegrationTests` contains any inheritance round-trip test (grep for `BaseTemplateName|ParentTemplateId` hits only serializer unit tests).
| # | Round-1 finding | R1 sev | Disposition | Evidence (file:line) |
|---|---|---|---|---|
| 1 | Bundle import drops template inheritance edges (derived templates land as roots) | Critical | **F** | `ResolveInheritanceEdgesAsync` (`Transport/Import/BundleImporter.cs:2542-2616`), called from `ApplyAsync` at `:1261` after the composition rewire; honours Rename via the resolution map (`:2576-2580`), clears stale edges for root-template bundles (`:2566-2569`), unresolvable base → null edge + `BundleImportBaseTemplateUnresolved` audit row, never throws (`:2593-2612`). Round-trip + rename tests: `tests/.../Transport.IntegrationTests/Import/InheritanceImportTests.cs` |
| 1b | (Rec. #1 rider) importer performs no acyclicity check — crafted bundle can persist a cycle | Critical (rider) | **F** | `EnsureNoTemplateGraphCycles` (`BundleImporter.cs:2633-2670`) runs `CycleDetector.DetectInheritanceCycle`/`DetectCompositionCycle` over the merged change-tracker graph, called before any SaveChanges of the staged edges (`:1275-1287`); completeness argument sound — any edge to a pre-existing template forces a tracked `GetAllTemplatesAsync` load, which Includes `Compositions`/`NativeAlarmSources` (`ConfigurationDatabase/Repositories/TemplateEngineRepository.cs:69-79`) |
| 2 | Imported scripts lose `MinTimeBetweenRuns`/`ExecutionTimeoutSeconds`; `LockedInDerived` absent from DTOs | High | **F** | DTOs carry all three (`Transport/Serialization/EntityDtos.cs:210, 221, 231-236`); `BuildTemplate` copies them (`BundleImporter.cs:1832, 1844, 1856-1858`); all four `SyncTemplate*Async` overwrite paths copy them (`:1941, 2067, 2169-2171, 2277`); fidelity tests `Import/TemplateScriptFidelityTests.cs` |
| 3 | Template `NativeAlarmSources` not transported; carried instance overrides dangle | High | **F** | `TemplateNativeAlarmSourceDto` (`EntityDtos.cs:187-195`) with back-compat empty default (`:168-178`); export + `BuildTemplate` (`BundleImporter.cs:1866-1872`) + `SyncTemplateNativeAlarmSourcesAsync` (`:2237-2330`, called from Overwrite at `:1726`); preview diff via `DiffChildren` (`Import/ArtifactDiff.cs:123-128`); tests `Import/NativeAlarmSourceImportTests.cs` incl. the override-no-longer-dangles case |
| 4 | Flattener drops grandchild members on repeated composition; CollisionDetector blind | High | **F** | All four `ResolveComposed*Recursive` methods guard on the recursion **path** (add-on-entry / remove-in-finally): attributes `TemplateEngine/Flattening/FlatteningService.cs:286-307`, alarms `:659-680`, native sources `:782-803`, scripts `:947+`; `CollisionDetector.CollectComposedMembers` same conversion with explicit rationale comment (`CollisionDetector.cs:126-155`) |
| 5 | Revision hash and diff omit `NativeAlarmSources` — no staleness signal | High | **F** | `HashableNativeAlarmSource` (`Flattening/RevisionHashService.cs:280-298`), folded null-when-empty so native-source-free hashes stay byte-identical (`:103-114` — surgical migration, only affected instances flip stale once); `DiffService` sweeps native sources (`Flattening/DiffService.cs:47-50, 149`); reflective alphabetical guard auto-covers the new record (`tests/.../RevisionHashServiceTests.cs:103-132`) plus `ComputeHash_NativeAlarmSourceChange_ChangesHash` (`:371`) |
| 6 | Bundle import leaves stale compiled Inbound API handlers serving old code | High | **F** | Two-layer fix. Contract + publisher (plan-05 T14): `IScriptArtifactChangeBus` (`Commons/Interfaces/Scripting/IScriptArtifactChangeBus.cs:17`), post-commit publish from `ApplyAsync` (`BundleImporter.cs:1340, 1777-1816`), swallow-and-log. Consumer (plan-06) went stronger than name-eviction: `InboundScriptExecutor` compares cached handler script text against the current DB row and recompiles on mismatch, and `_knownBadMethods` fast-fails only when the *current* text equals the recorded bad text (`InboundAPI/InboundScriptExecutor.cs:370-404`) — content-based self-healing that also covers failover/direct-SQL edits |
| 7 | Locked native alarm sources overridable at instance level — lock enforced nowhere | High | **F** | `ResolvedNativeAlarmSource.IsLocked` propagated incl. derived-shadow locks (`FlatteningService.cs:722`); `ApplyInstanceNativeAlarmSourceOverrides` skips locked (`:815`); `ManagementActor` rejects on locked source for both single and batch handlers, and rejects dangling canonical names (`ManagementService/ManagementActor.cs:953-1043`, throws at `:978`, `:1037`) |
| 8 | Post-success audit failure flips committed Success deployment to Failed | Medium | **F** | Deploy audit routed through guarded `TryLogAuditAsync` explicitly outside the failure path (`DeploymentManager/DeploymentService.cs:329-336`, rationale comment in place) |
| 9 | Deleting a `NotDeployed` instance requires a live site round-trip | Medium | **F** | `DeleteInstanceAsync` short-circuits `NotDeployed` to a central-only record delete + audit with `SiteRoundTripSkipped = true`, no communication-service call (`DeploymentService.cs:567-595`); doc aligned (`Component-DeploymentManager.md`) |
| 10 | Import blocker heuristic hard-blocks valid imports, no override | Medium | **F** | Three-part fix: locally-declared functions/methods excluded per-origin (`BundleImporter.cs:792-811`, `CollectLocalDeclarations` Roslyn script-parse at `:921-944`); denylist extended; severity split — template-script findings are `ConflictKind.Warning`/`ImportResult.Warnings` (advisory, deploy gate re-validates), ApiMethod findings stay `Blocker`/hard error (`:819, :842-846`, apply side `:4444-4460`) |
| 11 | `OperationLockManager` single-node in-memory; standby-node ops not excluded | Medium | **D** (accepted) | Invariant recorded as a decision block (`Component-DeploymentManager.md:94`); structural active-node gating deferred to the cluster/UI plans (01/07) per PLAN-05 coverage table — matches what shipped |
| 12 | Every validation run Roslyn-compiles every script — CPU + non-collectible assembly load on read paths | High (perf) | **F** (residual → new finding N1) | Verdict cache keyed on SHA-256 of code, name-free errors, 4096-entry bound (`TemplateEngine/Validation/ScriptCompileVerdictCache.cs:28-69`; consumed at `ScriptCompiler.cs:42-60`); read-only paths skip the compile stage entirely — `GetDeploymentComparisonAsync` (`DeploymentService.cs:690`) and `StaleInstanceProbe` (`StaleInstanceProbe.cs:37`) pass `validateScripts: false` through `FlatteningPipeline.cs:58, 162-174`. Script *bodies* fully covered; Expression-**trigger** compiles remain on read paths — see N1 |
| 13 | `LineDiffer` O((N+M)²) memory, no input cap | Medium (perf) | **F** | `MaxInputLines = 4000` (`Transport/Import/LineDiffer.cs:62`); oversized inputs short-circuit to summary-only before the Myers trace (`:84`, `SummaryOnlyResult` `:119`); documented (`Component-Transport.md:23`) |
| 14 | Templates with folder/parent/composition never diff `Identical` (placeholder comparisons) | Medium (perf) | **F** | `CompareTemplate` takes optional `folderNameById`/`templateNameById` maps (`ArtifactDiff.cs:64-65, 75-76`), resolvers fall back to placeholders only when a map misses (`:679-700`); `PreviewAsync` passes the maps it already builds (`BundleImporter.cs:395`) |
| 15 | `LoadAsync` O(bundle) even for manifest; unbounded session count | Low (perf) | **P** | Session cap shipped: `MaxConcurrentImportSessions = 8` (`TransportOptions.cs:37`) enforced with evict-expired-first in `BundleSessionStore.Open` (`BundleSessionStore.cs:78-88`). Manifest-peek `ReadManifestAsync` deferred as planned, honestly documented (`Component-Transport.md:92`) |
| 16 | Import apply is one long EF transaction | Low (perf) | **D** (accepted) | Per plan: dominant cost (Roslyn in `StaleInstanceProbe`) removed by #12; restructuring the single-transaction rollback contract judged high-risk/low-gain. Transaction shape unchanged (`BundleImporter.cs` apply path) — accepted |
| 17 | Deny-list gaps: `Environment.Exit`/`FailFast`/`GetEnvironmentVariable`, `GC` | Medium (sec) | **F** | `ForbiddenScopes` += `System.Environment`, `System.GC` with rationale comments (`ScriptAnalysis/ScriptTrustPolicy.cs:45-52`). Bonus beyond plan: `ScriptTrustValidator` now compiles the analysis with the runtime's `DefaultImports` (`WithUsings`), closing the bare-identifier blind spot where `Environment.Exit(0)` under `using System;` under-resolved (`ScriptTrustValidator.cs:118-132`) |
| 18 | `System.Data` allowance is an unbounded network hole (concrete providers) | Medium (sec) | **F** | `Microsoft.Data`, `System.Data.SqlClient`, `System.Data.Odbc`, `System.Data.OleDb` denied; `System.Data.Common` abstractions deliberately kept (Database helper posture documented in the code comment and `Component-ScriptAnalysis.md`) (`ScriptTrustPolicy.cs:53-61`) |
| 19 | Reflection-gateway member list incomplete (`GetTypes`, `Invoke`, `EntryPoint`, `Declared*`) | Medium (sec) | **F** | `GetTypes`, `EntryPoint`, `DeclaredMethods/Members/Constructors`, `DynamicInvoke` added; `Invoke` deliberately excluded with recorded rationale (delegate `Invoke` legitimate; `MethodInfo.Invoke` caught semantically) (`ScriptTrustPolicy.cs:105-116`) |
| 20 | Transport import bypasses the script trust gate entirely | Medium (sec) | **F** | Pass 0 trust gate at apply (`BundleImporter.cs:4301-4335`) and preview (`:849-905`): `ScriptTrustValidator.FindViolations` over every non-Skip template/shared/ApiMethod body **and** template script + alarm Expression-trigger bodies (`EnumerateTrustGatedScripts` `:957-1001`, `ExtractTriggerExpression` `:1004-1035`); hard error for all kinds; fail-closed on validator throw. `Component-ScriptAnalysis.md:5, 173, 193` names Transport the fifth call site |
| 21 | Instance alarm overrides implement less than spec (Description / On-Trigger ref) | Medium (conv) | **F** (doc aligned) | Spec now records the narrower granularity as the decision — only `TriggerConfigurationOverride`/`PriorityLevelOverride` overridable; Description/On-Trigger ref are template-level; adding columns is future feature work (`Component-TemplateEngine.md:113`) |
| 22 | `ArtifactDiff` equality predicates drift from sync predicates (both directions) | Medium (conv) | **F** | `TemplateChildEquality` single source of truth — complete writable-field comparers for attributes/alarms/scripts/native sources incl. `ElementDataType`, `LockedInDerived`, on-trigger-script-by-name resolver, cadence/timeout (`Transport/Import/TemplateChildEquality.cs:35-100`); both `ArtifactDiff` (`:118-128`) and all four sync `changed` predicates route through it (`BundleImporter.cs:1931, 2045, 2160, 2268`) |
| 23 | Spec advertises unimplemented `scripts/` bundle dir vs `MaxBundleEntryCount = 4` | Low | **F** | Spec corrected: scripts travel inside the content blob, a well-formed bundle has exactly two zip entries, separate `scripts/` dir explicitly not part of the format (`Component-Transport.md:49-51`) |
| 24 | `TryCompile` surfaces only the first violation/error | Low | **F** (residual → N2) | `ScriptCompiler.TryCompile` joins ALL violations / ALL compile errors (`ScriptCompiler.cs:46-58`). The trigger-expression path was not covered — see N2 |
| 25 | `RevisionHashService` determinism contract fragile | Low | **F** | Guard test enumerates every nested `Hashable*` record **reflectively**, so new records are auto-covered (`RevisionHashServiceTests.cs:103-132`); migration note in the class doc (`RevisionHashService.cs:23`) |
| U1 | Template fidelity shipped behind site/instance wave | — | **F** | Tasks 1–8 landed; the reflection round-trip equivalence suite (`tests/.../Transport.IntegrationTests/RoundTripEquivalenceTests.cs`) structurally guards every transported entity type and itself caught 5 further silent-loss bugs, fixed in the importer/exporter (commit `7c74bbe4`: ES retry/methods-on-Add, notification recipients on export, folder `ParentFolderId` on Add) |
| U2 | No derived-template import test | — | **F** | `Import/InheritanceImportTests.cs` (Add + Overwrite + renamed-base cases) |
| U3 | No repeated-composition flattening/collision test | — | **F** | Tests extended with the `y1.z.*` / `y2.z.*` two-slot cases + cycle-termination regression guard (commits `730bf191`, `874e32a9`) |
| U4 | Deferred-but-load-bearing: rename call-site rewriting, Transport-012 UI, `ReadManifestAsync` | — | **D** (accepted) | Rename limitation now explicit and load-bearing in the spec; Transport-012 filter UI documented as deferred with the CLI workaround (`Component-Transport.md:354`); `ReadManifestAsync` deferral documented (`:92`) |
| U5 | Preview→apply window has no optimistic concurrency (version fields null) | — | **D** (accepted) | Recorded decision paragraph — resolutions keyed by `(EntityType, Name)`, single-operator workflow accepted for v1, version fields reserved (`Component-Transport.md:138`; `ArtifactDiff.cs:33-37`) |
| U6 | Three hand-maintained compile-surface mirrors | — | **D** (accepted) | Guard tests exist; source-generator remains an improvement, not a defect. `ScriptCompileSurface` changes since baseline are XML-doc only (verified via diff) |
| U7 | `SemanticValidator` leaf-name fallback silently accepts wrong-child calls | — | **F** | Leaf-name-only matches now emit `ValidationEntry.Warning(CallTargetNotFound, "... matched by composed leaf name only — child path not verified")` (`TemplateEngine/Validation/SemanticValidator.cs:123-131`) |
### [High] Imported template scripts lose `MinTimeBetweenRuns` and `ExecutionTimeoutSeconds`
`TemplateScriptDto` carries both fields (`EntityDtos.cs:148-151`), but the importer drops them on **both** paths: `BuildTemplate` (Add/Rename — `BundleImporter.cs:1550-1560`) and `SyncTemplateScriptsAsync` (Overwrite — the `changed` comparison and field copy at `BundleImporter.cs:~1836-1851` cover Code/TriggerType/TriggerConfiguration/Params/Return/IsLocked only). Both fields participate in the revision hash (`RevisionHashService.cs:87-90`), so the drift is invisible until redeploy, at which point a WhileTrue script's re-fire cadence or a per-script timeout silently reverts to default on the target environment. Same-class omissions: `LockedInDerived` is not carried in the DTO at all (no hits in `EntityDtos.cs`/`EntitySerializer.cs`), so a base template's derived-lock policy does not survive promotion.
### [High] Template `NativeAlarmSources` are not transported, but instance-level overrides of them are
There is no `TemplateNativeAlarmSourceDto`; `BuildTemplate` copies only attributes/alarms/scripts, and none of the `SyncTemplate*` overwrite helpers touch native alarm sources — yet `InstanceNativeAlarmSourceOverrideDto` **does** travel and is applied (`EntityDtos.cs:292`, `BundleImporter.cs:3325-3338`). Consequences: (a) a template imported as **Add** has no native alarm mirrors at all; (b) an **Overwrite** leaves the target's existing sources untouched and the divergence is invisible because `ArtifactDiff.CompareTemplate` (`Import/ArtifactDiff.cs:54-105`) doesn't diff them either; (c) the carried instance overrides dangle and are *silently ignored* at flatten (`FlatteningService.ApplyInstanceNativeAlarmSourceOverrides`, `FlatteningService.cs:781` — missing canonical name → `continue`). This contradicts `Component-TemplateEngine.md:59-64` which makes native alarm sources first-class template members.
### [High] Flattening silently drops grandchild members when the same template is composed twice in one subtree
The recursive composed-member resolvers share one `visited` set down the recursion (`FlatteningService.cs:285-297` for attributes; identical pattern for alarms `:649-660`, scripts `:913-924`, native sources `:761-772`). The direct members of a repeated module still resolve (added before the `visited` check), but the **descent into its nested compositions is skipped** on the second occurrence.
**Failure scenario:** template X composes Y twice (slots `y1`, `y2`); Y composes Z. Flattening produces `y1.*`, `y1.z.*`, `y2.*` — but **not `y2.z.*`**. No validation error fires (the members simply don't exist in the flattened config), so the instance deploys with the second slot's nested sensor/alarm/script members missing. `CollisionDetector.CollectComposedMembers` (`CollisionDetector.cs:126-143`) has the same shared-`visited` early-return, and there it skips even the repeated module's *direct* members — so collision detection is also blind under the second slot. The `visited` set is the right tool for cycle-guarding but the wrong tool for slot deduplication: composition is by-slot, not by-template. Fix is to key the guard on the *path* (prefix) or track only the current recursion stack.
### [High] Revision hash and diff omit `NativeAlarmSources` — no staleness signal for native-alarm changes
`RevisionHashService.HashableConfiguration` (`RevisionHashService.cs:116-152`) hashes Attributes, Alarms, Scripts, Connections — **not** `FlattenedConfiguration.NativeAlarmSources` (populated at `FlatteningService.cs:135`). Editing a template's native alarm source (connection, source reference, condition filter) or an instance override therefore does not change the revision hash: the Deployments page never flags the instance stale, `DeploymentService.GetDeploymentComparisonAsync` reports `IsStale = false`, and sites keep mirroring the old alarm source indefinitely until some unrelated change forces a redeploy. This contradicts the spec's own claim that overrides "participate in the revision hash — changes … re-deploy as expected" (`Component-TemplateEngine.md:215`), which is implemented for connection-binding overrides but not for the native-alarm member class. `DiffService` should be audited for the same omission.
### [High] Bundle import leaves stale compiled Inbound API handlers serving old code
`ApplyApiMethodsAsync` (`BundleImporter.cs:2625`) writes `ApiMethod` rows through `IInboundApiRepository` — a DB-level write. The Inbound API's singleton `InboundScriptExecutor` caches compiled delegates **by method name with no content-based invalidation** (`InboundAPI/InboundScriptExecutor.cs:311-331`: `_scriptHandlers.TryGetValue(method.Name, …)` → serve cached delegate) and caches known-bad methods the same way (`:39, :58-62`). An import that Overwrites an ApiMethod script keeps the **old delegate** serving (HTTP 200 with stale behavior); an import that *fixes* a previously-broken method keeps returning "Script compilation failed" from `_knownBadMethods` until restart. This is the exact failure mode already documented in project memory for DB-direct edits — Transport import is a DB-direct edit at scale, and nothing in `ApplyAsync` signals the executor. Cross-cluster deployments make this worse: the cache lives per central node.
### [High] Locked native alarm sources can be overridden at instance level — lock not enforced anywhere
Spec: "Lock applies to the entire source" (`Component-TemplateEngine.md:114`). Reality: `ApplyInstanceNativeAlarmSourceOverrides` (`FlatteningService.cs:775-792`) applies the override with **no lock check** — contrast the attribute path (`:309 if (existing.IsLocked) continue`) and alarm path (`:337`). `ResolvedNativeAlarmSource` doesn't even carry `IsLocked` (the inherit-time `lockedNames` set at `:676` is local and discarded). The management command path has no check either: `ManagementActor.HandleSetInstanceNativeAlarmSourceOverride` (`ManagementService/ManagementActor.cs:882-909`) upserts unconditionally after site-scope authorization. A site-scoped operator can repoint a *locked* native alarm binding to an arbitrary connection/source reference — precisely what locking exists to prevent on a read-only alarm mirror.
### [Medium] Post-success audit failure flips a committed Success deployment record to Failed
`DeployInstanceAsync` commits the terminal Success status (`DeploymentService.cs:304-306`), runs post-success side effects (correctly swallowed, `:970-1005`), then calls `_auditService.LogAsync` **inside the same try** (`:330-332`). If that audit write throws (transient DB fault, serialization), the catch block (`:343-397`) unconditionally sets `record.Status = Failed` and persists it — central now reports Failed while the site runs the new config, the exact divergence the surrounding comments (`:298-303`) were written to prevent. It also violates the repo convention "audit-write failure NEVER aborts the user-facing action". The audit call should be outside the try or individually guarded like `TryLogLifecycleTimeoutAsync` (`:1034-1066`) already is for lifecycle timeouts.
### [Medium] Deleting a `NotDeployed` instance still requires a live site round-trip
`StateTransitionValidator.CanDelete` explicitly allows delete-from-NotDeployed as "central-side record cleanup (no live site config to tear down)" (`StateTransitionValidator.cs:40-49`), but `DeploymentService.DeleteInstanceAsync` (`DeploymentService.cs:546-585`) unconditionally sends `DeleteInstanceCommand` to the site and fails on timeout. A Transport-imported instance (always lands `NotDeployed`) targeted at a site that is unreachable or not yet commissioned is **undeletable** through the normal path. The two components disagree about the same design sentence.
### [Medium] Import blocker heuristic can hard-block a valid import with no operator override
The Pass-1 name scan (`RunSemanticValidationAsync`, `BundleImporter.cs:3603-3692`; shared with `DetectBlockersAsync:751-818`) flags any `PascalCase(`-shaped token not in the enumerative `KnownNonReferenceNames` denylist (`:853-874`) as a missing SharedScript/ExternalSystem and **throws `SemanticValidationException`** at apply — rolling back the whole import. Any script that defines a local PascalCase method (`decimal ComputeRate(...)` then `ComputeRate(x)`), uses a stdlib type not on the list (`Regex.Match`, `StringBuilder`, `Json`), or embeds SQL keywords outside the nine listed, is a false positive with no "proceed anyway" escape hatch — the operator must edit scripts or re-cut the bundle. The denylist will drift forever. Conversely `Regex`/`StringBuilder` being absent from the list means the heuristic fires on innocent code today. Consider downgrading Pass-1 findings on *template scripts* to warnings (the deploy gate re-validates authoritatively) and keeping hard-block only for `ApiMethod` scripts, or adding an explicit ignore resolution.
### [Medium] `OperationLockManager` is single-node in-memory; concurrent ops via the standby node aren't excluded
Documented as acceptable ("lost on central failover", `OperationLockManager.cs:12-13`) and the implementation itself is careful (ref-counting, reservation-before-wait, exception-filter cleanup — `:46-129`). Residual risk: nothing structurally prevents both central nodes from executing mutating instance ops concurrently if traffic bypasses Traefik's active-node routing (direct ports 9001/9002 are documented in CLAUDE.md). Optimistic concurrency on `DeploymentRecord` is the backstop, but instance-state writes (`DisableInstanceAsync:456-458`) are last-write-wins. Low probability, but worth an explicit invariant (reject management ops on the non-active node).
**Disposition counts:** 26 Fixed (verified) · 1 Partially fixed (#15 — remainder is the doc-acknowledged `ReadManifestAsync` deferral) · 5 Deferred (accepted, all recorded as decisions in the design docs) · 0 Not fixed · 0 Regressed · 0 No longer applicable.
---
## Findings — Performance
## New Findings (round 2)
### [High] Every validation run compiles every script with Roslyn — CPU heavy and leaks loaded assemblies on the central node
`ValidationService.ValidateScriptCompilation` (`ValidationService.cs:256-273`) calls `ScriptCompiler.TryCompile` per script; that runs `ScriptTrustValidator.FindViolations` (a fresh `CSharpCompilation.CreateScriptCompilation` over the **entire TPA reference set**, `ScriptTrustValidator.cs:114-124`) *plus* `RoslynScriptCompiler.Compile` → `CSharpScript.Create(...).Compile()` (`RoslynScriptCompiler.cs:70-71`). `Script.Compile()` on success emits and loads the script assembly via Roslyn's `InteractiveAssemblyLoader`, which is **not collectible** — each compiled script permanently occupies loader/metadata memory in the central host.
Fresh review of the ~1,900 lines added across the four projects since `b910f5eb` (31 commits: the PLAN-05 fix wave, PLAN-07's `DeployToSiteAsync` site-scope slice, PLAN-06's ESG `TimeoutSeconds` pipeline ride-along, PLAN-08's options validators, and a docs/fixdocs sweep).
This sits on hot paths: `FlatteningPipeline.FlattenAndValidateAsync` (`FlatteningPipeline.cs:161-167`) is invoked (a) per deploy, (b) per instance by `DeploymentService.GetDeploymentComparisonAsync` (`DeploymentService.cs:652`) — i.e. **at query time whenever the Deployments page computes staleness**, and (c) per deployed instance by `StaleInstanceProbe` during every template-overwriting bundle import (`BundleImporter.ComputeStaleInstanceIdsAsync:1242-1327`). A 50-instance environment with 10 scripts each = 500 full trust-compilations + 500 script-assembly loads per Deployments-page staleness sweep, unbounded over process lifetime. There is **no compile-result cache**. Fix: cache `TryCompile` verdicts keyed by SHA-256 of code (the verdict is a pure function of code + policy), and/or skip `ValidateScriptCompilation` on read-only comparison paths (staleness needs the hash, not a compile).
### Performance
### [Medium] `LineDiffer` worst case is O((N+M)²) memory with no input cap
The Myers implementation snapshots the full V frontier per edit distance (`LineDiffer.cs:157-204`: `trace.Add((int[])v.Clone())` with `v` of length `2(N+M)+1`, up to `N+M` rounds). Two ~10,000-line scripts with no common lines ⇒ ~20k rounds × 160 KB ≈ **3+ GB** transient allocation during `PreviewAsync`. `maxLines` (400) caps only the *output* (`:66-92`); nothing caps the *inputs*, and the bundle size cap (100 MB) comfortably admits such scripts. A crafted (or merely bloated) bundle can OOM the active central node from the import preview. Cap input line counts (fall back to the `<N lines>` summary beyond a threshold) or use the linear-space Myers variant.
#### [Medium] N1 — Expression-trigger validation still Roslyn-compiles on read-only staleness paths, uncached
The `validateScripts: false` fix (#12) gates only `ValidateScriptCompilation`; `ValidateExpressionTriggers` runs unconditionally in the `Validate` pipeline (`TemplateEngine/Validation/ValidationService.cs:128-134`), and its `CheckExpressionSyntax` performs a fresh `ScriptTrustValidator.FindViolations` (a full `CSharpCompilation` over the TPA reference set) **plus** `RoslynScriptCompiler.Compile` per Expression-triggered script *and* alarm, with no verdict cache (`ValidationService.cs:531-546`). For an environment that uses Expression triggers, every Deployments-page staleness sweep (`DeploymentService.GetDeploymentComparisonAsync:690`) and every template-overwriting bundle import (`StaleInstanceProbe.cs:37`) still pays N×(trust-compilation + script compile) — the exact hot-path cost class round 1 flagged, surviving in a smaller member class. Fix: either gate `ValidateExpressionTriggers`' syntax check behind the same `validateScriptCompilation` flag (the attribute-reference scan can stay), or cache the verdict — **if the `ScriptCompileVerdictCache` is reused, the key must gain the globals-surface discriminator**: it is currently keyed on code alone (`ScriptCompileVerdictCache.cs:53`), which is sound only while `ScriptCompiler` (always `ScriptCompileSurface`) is its sole writer; a trigger expression valid against `TriggerCompileSurface` is not interchangeable with a script-body verdict.
### [Medium] Any template with a folder, parent, or composition can never diff `Identical`
`ArtifactDiff.CompareTemplate` compares `FolderNameOf`/`BaseTemplateNameOf`/`CompositionTargetNameOf` placeholder strings (`"<id:5>"`) against real incoming names (`ArtifactDiff.cs:64-65, 100, 674-693`), so `FolderName`/`BaseTemplateName`/`Compositions.*` always register as changed for such templates. Effect: the wizard's "Identical → auto-skipped" contract (`Component-Transport.md:222`) never applies to structured templates; every re-import of an unchanged bundle shows spurious Modified rows and, if bulk-Overwrite is chosen, generates needless audit rows and stale-instance probing. The preview loop already hydrates templates (`BundleImporter.cs:379-388`); passing the folder/template name maps it also builds (`:365-366`) into `CompareTemplate` would fix all three placeholders cheaply.
### Stability & Correctness
### [Low] `LoadAsync` is O(bundle) even when only the manifest is needed, and each load double-opens the zip
Acknowledged in the design doc itself (`Component-Transport.md:90` — deferred `ReadManifestAsync`). Combined with sessions holding the full decrypted content (`BundleSession.DecryptedContent`) for up to 30 minutes and no cap on concurrent session count (`BundleSessionStore.cs:38`), N concurrent 100 MB uploads pin N×~200 MB. The T-007 zeroing on apply/failure (`BundleImporter.cs:1105, 1194`) is good; a session-count cap would round it out.
#### [Low] N2 — `CheckExpressionSyntax` surfaces only the first violation / first compile error
`ValidationService.cs:534-544` returns `violations[0]` / `errors[0]`. Task 24 fixed exactly this shape in `ScriptCompiler.TryCompile` (full joined lists) but the trigger-expression twin was missed — an operator fixing a multi-error expression gets one deploy round-trip per error. Same one-line `string.Join` fix.
### [Low] Import apply is one long EF transaction with per-entity audit rows and four intermediate flushes
`ApplyAsync` (`BundleImporter.cs:960-1098`) holds a relational transaction across full semantic validation (Roslyn-free — fine), all entity staging, per-row `IAuditService.LogAsync` calls, and stale-instance probing (which itself re-flattens per instance, see the Roslyn finding). For a large bundle this is minutes of open-transaction lock footprint on the shared config DB while the Blazor circuit waits. The CLI's raised 5-minute timeout acknowledges the symptom.
#### [Low] N3 — `PublishScriptArtifactChanges` skips `Add` resolutions
The post-commit publisher notifies only `Overwrite`/`Rename` resolutions (`BundleImporter.cs:1798-1803`). Edge case: an ApiMethod is deleted on the target, then re-imported as `Add` under the same name — any node still holding a `_knownBadMethods`/handler entry for that name receives no notification. Mitigated to Low because the plan-06 consumer self-heals by content comparison (`InboundScriptExecutor.cs:370-379`), which is exactly the contract's stated design (bus is advisory, consumers must self-heal); noting it so a future *non*-self-healing subscriber doesn't inherit the gap silently.
#### [Low] N4 — Import persists silently-inert instance overrides on locked members
`ApplyInstancesAsync` writes `InstanceAttributeOverride` / `InstanceNativeAlarmSourceOverride` rows straight from the bundle (`BundleImporter.cs:4001, 4022`) with none of the lock checks the `ManagementActor` enforces for the same writes (`ManagementActor.cs:898, 978`). No security bypass — the flattener drops locked-member overrides (`FlatteningService.cs:319, 815`) — but the imported rows are dead data and the operator gets no warning that part of the bundle's instance config will never take effect. A `ConflictKind.Warning` row (the Task-19 machinery is right there) would surface it in the wizard.
### Security / Trust Model
#### [Low] N5 — Import trust gate does not scan instance-override Expression-trigger bodies
`EnumerateTrustGatedScripts` covers template script bodies, template script/alarm trigger expressions, shared scripts, and ApiMethods (`BundleImporter.cs:957-1001`) — but not `InstanceAlarmOverrideDto.TriggerConfigurationOverride`, which can carry an `{"expression": ...}` body when the overridden alarm is Expression-triggered. Not exploitable pre-runtime: imported instances land `NotDeployed`, and the deploy gate's `ValidateExpressionTriggers` runs on the *flattened* config (post-override) with the same authoritative `FindViolations` (`ValidationService.cs:531-539`), so a forbidden expression is rejected before it can execute. This is an import-*review* completeness gap only — the operator learns at deploy time instead of in the wizard, the exact UX the Task-20 gate was built to improve. Cheap add: run `ExtractTriggerExpression` over instance alarm-override configs in the enumerator.
### Conventions
#### [Low] N6 — `TransportOptionsValidator` misses the knob its own fix wave added
The validator covers ten options including the PBKDF2 floor (`TransportOptionsValidator.cs:25-66`) but not `MaxConcurrentImportSessions` — configured to `0`/negative, `BundleSessionStore.Open`'s `_sessions.Count >= cap` (`BundleSessionStore.cs:78-88`) rejects **every** import forever with "Too many concurrent import sessions (0)". Fail-closed, so no safety issue, but it contradicts the validator's stated purpose ("a zero/negative value would either disable a cap [or] crash the import pipeline") and is a one-line `RequireThat`.
---
## Findings — Security / Trust Model
## What's genuinely good
### [Medium] Deny-list gaps: `Environment.Exit`/`FailFast` (site-node kill), `Environment.GetEnvironmentVariable` (secret read)
`ScriptTrustPolicy.ForbiddenScopes` (`ScriptTrustPolicy.cs:36-45`) blocks IO/Process/Threading/Reflection/Net/Interop/Win32, but the `System` namespace root types remain fully available: a template script containing `Environment.Exit(0)` compiles cleanly against the surface and **terminates the site-runtime process** on first trigger — a one-line denial of service of an entire site node from the Design role. `Environment.FailFast`, `Environment.GetEnvironmentVariable` (exfiltrate `SCADABRIDGE_API_KEY`-class secrets via `Notify`/`ExternalSystem.Call`), and `GC.Collect` loops are similarly unblocked. These are cheap adds to `ForbiddenScopes` (as fully-qualified type-or-member entries) or to `ForbiddenIdentifiers`.
### [Medium] `System.Data` allowance is an intentional but unbounded network hole
Site scripts may use raw ADO.NET (`Database.Connection` returns `DbConnection`; `System.Data.Common` is necessarily bindable — `ScriptCompileSurface.cs:119`). `System.Data` is absent from `ForbiddenScopes`, so if the site-runtime execution reference set includes any concrete provider (e.g. `Microsoft.Data.SqlClient`, which the Database helper itself needs), a script can `new SqlConnection("Server=attacker;...")` — an arbitrary-host network channel that the `System.Net` deny was meant to close. The semantic pass will not flag it (SqlClient is not under a forbidden scope). Worth either denying provider namespaces (`Microsoft.Data`, `System.Data.SqlClient`, `System.Data.Odbc`, …) while keeping `System.Data.Common` types reachable only via the `Database` helper, or documenting the accepted risk explicitly in `Component-ScriptAnalysis.md`.
### [Medium] Reflection-gateway member list misses `GetTypes`, `Invoke`, `EntryPoint`, `DeclaredMembers`
`ReflectionGatewayMembers` (`ScriptTrustPolicy.cs:66-88`) blocks `GetMethod`/`InvokeMember`/etc. but not `Invoke` (on `MethodInfo`/`Delegate`), `GetTypes`, `EntryPoint`, `DeclaredMethods`. Pass 1 usually compensates — any expression whose resolved member's containing type is under `System.Reflection` is flagged semantically (`ScriptTrustValidator.cs:180-212`) — but in the documented **TPA-degraded fallback mode** (`ScriptTrustPolicy.cs:207-245`) resolution weakens and Pass 2 becomes the primary defence for chained accesses; the syntactic list should be closed under the obvious invocation members since it exists precisely for that mode. Cheap hardening.
### [Medium] Transport import bypasses the script trust gate entirely
`ApplyAsync` writes template scripts, shared scripts, and API-method scripts to the DB without ever invoking `ScriptTrustValidator` (`RunSemanticValidationAsync` at `BundleImporter.cs:3603` is name-resolution + `SemanticValidator` only). Compensating controls exist downstream — the deploy gate re-validates template scripts (`ScriptCompiler.TryCompile`), the Inbound API re-checks trust at lazy compile — but the layering claim in `Component-ScriptAnalysis.md` ("all call sites delegate…") now has a fifth write path with no gate, and the Inbound API's check happens at *serve time* on the production node rather than at *import review time*. Running `FindViolations` over imported script bodies during Pass 2 (they're already in memory) would restore the design-time boundary and give the operator the finding in the wizard instead of a runtime 500.
### Positive observations (crypto & envelope)
`BundleSecretEncryptor` is textbook: AES-256-GCM with 16-byte salt / 12-byte nonce / 16-byte tag, PBKDF2-SHA256 600k iterations, fresh salt+nonce per encrypt (`BundleSecretEncryptor.cs:14-58`). The T-005 AAD binding of non-derivative manifest fields (`BundleManifestAad.cs:45-55`, verified in `LoadAsync` at `BundleImporter.cs:252-255`) makes the source-environment confirmation gate tamper-evident — a genuinely thoughtful touch. T-006 zip-bomb caps validate central-directory headers before any decompression (`BundleImporter.cs:296-338`); T-003 lockout is keyed by content hash so a second tab/CLI can't sidestep it (`BundleSessionStore.cs:40-47, 122-167`); T-007 zeroes decrypted plaintext on both success and failure. Key management is passphrase-only with no persistence — appropriate for the file-carry threat model, with the residual (documented) weakness that unencrypted export is allowed and only audit-flagged.
- **The round-trip reflection guard is the standout of the fix wave.** `RoundTripEquivalenceTests` compares every public writable property of every transported entity across export→import, so the *class* of bug that produced round-1's Critical + two Highs (hand-written DTO↔entity mapping drifting) is now structurally impossible to reintroduce silently — and it proved itself immediately by catching 5 residual fidelity holes the per-field unit tests had all passed over (commit `7c74bbe4`).
- **Fixes routinely exceeded their tickets.** The trust gate covers Expression-trigger bodies, not just script bodies (`BundleImporter.cs:957-1001`); the deny-list work also fixed a real analyzer blind spot (`WithUsings(ScriptTrustPolicy.DefaultImports)` so bare `Environment`/`GC` resolve to their forbidden types — `ScriptTrustValidator.cs:118-132`); the Inbound API consumer chose content-based invalidation over name eviction, which also self-heals failover and direct-SQL edits (`InboundScriptExecutor.cs:370-404`); the `ManagementActor` lock fix also rejects dangling canonical names, closing the create-inert-override path at the management surface (`ManagementActor.cs:961-980`).
- **Deferrals are honest and recorded where the next reader will look.** The preview→apply concurrency window (`Component-Transport.md:138`), the `OperationLockManager` node-affinity invariant (`Component-DeploymentManager.md:94`), the rename-doesn't-rewrite-call-sites limitation, and the `ReadManifestAsync` cost note are all written as explicit decisions with rationale, not silently dropped.
- **The import-time acyclicity guard's completeness reasoning is documented in the code** (`BundleImporter.cs:2633` remarks) and holds up under adversarial reading — the rewire passes' full-load fallback plus `GetAllTemplatesAsync`'s `Compositions` Include guarantee every node of any candidate cycle is tracked.
- **Single-sourcing done right:** `TemplateChildEquality` carries the "adding a writable field means adding it here, once" contract in its doc comment, and both consumers verifiably route through it; the revision-hash alphabetical guard enumerates nested records reflectively so new member classes are covered by construction.
---
## Findings — Conventions & Spec Drift
## Severity tally — NEW findings only
### [Medium] Instance alarm overrides implement less than the spec grants
`Component-TemplateEngine.md:113` says instance-overridable alarm fields include **Description and On-Trigger Script reference**; `ApplyInstanceAlarmOverrides` (`FlatteningService.cs:328-355`) applies only `TriggerConfigurationOverride` and `PriorityLevelOverride`. Either the doc or the entity/flattener should move.
| Severity | Count | Findings |
|---|---|---|
| Critical | 0 | — |
| High | 0 | — |
| Medium | 1 | N1 (Expression-trigger compiles on read-only paths, uncached) |
| Low | 5 | N2 (first-error-only trigger syntax), N3 (publisher skips Add), N4 (inert locked-member overrides on import), N5 (instance-override trigger bodies not import-gated), N6 (session-cap knob unvalidated) |
### [Medium] `ArtifactDiff` child-equality predicates miss fields that the import sync *does* write
`AttributesEqual` omits `ElementDataType` and `LockedInDerived` (`ArtifactDiff.cs:646-651`); `AlarmsEqual` omits the on-trigger script reference (`:653-658`); `ScriptsEqual` omits `MinTimeBetweenRuns`/`ExecutionTimeoutSeconds` (`:660-666`). A bundle differing only in those fields classifies **Identical → auto-skipped → silently not imported** (for the fields the sync would have written) — the mirror image of the false-Modified problem above. Diff predicates and sync-helper `changed` predicates should be generated from one field list per entity so they cannot drift (they already have, in both directions).
### [Low] Bundle format spec advertises a `scripts/` directory that is never produced, while `MaxBundleEntryCount` defaults to 4
`Component-Transport.md:45-47` documents optional `scripts/template-{id}-{name}.cs` entries; no code writes or reads them, and `TransportOptions.MaxBundleEntryCount = 4` (`TransportOptions.cs:22`) would reject a bundle that actually used the feature for more than two scripts. Remove from the spec or note it as unimplemented.
### [Low] `ScriptCompiler.TryCompile` surfaces only the first violation/error
`Validation/ScriptCompiler.cs:42,47` return `violations[0]` / `errors[0]`; an operator fixing a script with five issues gets five deploy round-trips. Both underlying APIs return full lists — join them.
### [Low] `RevisionHashService` determinism contract is fragile but guarded
Hash stability depends on alphabetical property declaration order in the `Hashable*` records (`RevisionHashService.cs:12-19, 113-115`); the guard test (`HashableRecords_PropertiesDeclaredAlphabetically`) exists, which is the right mitigation. Note for maintainers: adding `NativeAlarmSources` (see the High finding) will change all hashes — plan it as a deliberate migration (one-time global staleness) rather than a patch.
### Positive observations (conventions)
`DeploymentService` is exemplary on the repo's own rules: `CancellationToken.None` for failure/audit writes after cancellation (`DeploymentService.cs:355-372`), timeout-audit entries for all lifecycle verbs, reconciliation preserving operator-intended Disabled state (`:895-907`), terminal-status-before-side-effects ordering (`:298-306`), and the notify-and-fetch PendingDeployment TTL rationale (`:283-290`) is documented where the decision lives. `CycleDetector.BuildLookup` duplicate-Id tolerance and the Id-0-is-legitimate handling (`CycleDetector.cs:21-27, 56-59`) show real defect-driven hardening. `ScriptTrustPolicy`'s `DefaultReferences` vs `AnalysisReferences` separation with the loud TPA-degradation flag (`ScriptTrustPolicy.cs:207-245`) is a well-reasoned two-layer defence.
---
## Underdeveloped Areas
1. **Transport site/instance wave (M8) shipped ahead of its template-fidelity basics.** Instances, connections, name-mapping, FK rewiring, and stale-probing are all implemented and tested (`SiteInstanceImportTests.cs`, `CompositionImportTests.cs`, rollback-failure tests), but the older template payload silently loses inheritance, cadence/timeout, derived-lock flags, and native alarm sources (findings above). The DTO layer (`EntitySerializer.FromBundleContent`) actually implements more fidelity than the apply path uses — the importer re-implements DTO→entity mapping by hand (`BuildTemplate`) and drifted.
2. **No import test exercises a derived template.** `Transport.IntegrationTests` covers round-trip, conflict resolution, composition edges, semantic validation failure — but grep shows zero inheritance coverage. A single `export(base+derived) → import → flatten` test would have caught the Critical finding.
3. **No flattening test for repeated composition.** `TemplateEngine.Tests/Flattening` exists but the shared-`visited` semantics (same template in two slots of one subtree) is untested; same for `CollisionDetector`.
4. **Deferred items acknowledged in docs but load-bearing:** manifest-only `ReadManifestAsync` (perf), Configuration-Audit-Log "Bundle Import" filter UI (Transport-012), rename call-site rewriting ("call sites are not rewritten in v1" — `BundleImporter.cs:1519-1522`, meaning a Rename resolution on a shared script produces a bundle whose importing templates still call the *old* name, satisfied only because the name-set registers both — the *target DB* ends up with scripts calling a name that may not exist there; only Pass-1's dual registration hides it at import time).
5. **`ArtifactDiff` version fields are permanently null** (`ArtifactDiff.cs:33-37`) — the manifest's per-artifact `version` numbers are display-only; no optimistic concurrency exists between preview and apply, so a target edited between Preview and Apply is silently overwritten (session TTL is 30 min — a real window).
6. **Compile-surface parity relies on three hand-maintained mirrors** (`ScriptCompileSurface`, `TriggerCompileSurface`, Central-UI `SandboxScriptHost`), guarded by reflection tests for two of them and representative-script tests for the third (`Component-ScriptAnalysis.md:141-145`). Adequate but fragile; a source-generator would eliminate the class of drift.
7. **`SemanticValidator` call-target extraction is string-based** and self-admittedly cannot resolve dynamic child names (`SemanticValidator.cs:43-56` — leaf-name fallback accepts any composed script with a matching leaf, so a wrong-child `Children[x].CallScript("Y")` passes design-time validation). Documented tradeoff, but worth a warning-level finding rather than silence.
---
## Prioritized Recommendations
1. **[Critical] Wire template inheritance in the import apply path.** Add a `ResolveInheritanceEdgesAsync` pass mirroring `ResolveCompositionEdgesAsync` (resolve `BaseTemplateName` against Local + target by name, honouring Rename), run it after the template flush, and run `CycleDetector.DetectInheritanceCycle`/`DetectCompositionCycle` over the merged graph before commit (the importer currently performs **no** acyclicity check — a crafted bundle can persist a composition cycle that the authoring path would reject). Add a derived-template round-trip integration test.
2. **[High] Close the import data-loss set in one sweep:** carry `MinTimeBetweenRuns`/`ExecutionTimeoutSeconds` through `BuildTemplate` + `SyncTemplateScriptsAsync`; add `LockedInDerived` to the attribute/alarm/script DTOs and syncs; add `TemplateNativeAlarmSourceDto` (or block export of templates that have native alarm sources until supported). Drive diff predicates and sync predicates from a single per-entity field list.
3. **[High] Fix the repeated-composition `visited` bug** in the four `ResolveComposed*Recursive` methods and `CollisionDetector.CollectComposedMembers` — guard on recursion *path*, not global template Id.
4. **[High] Add `NativeAlarmSources` to `RevisionHashService` (and `DiffService`)** as a planned hash-migration, restoring staleness detection for native-alarm edits.
5. **[High] Cache script-compile verdicts by code hash** and stop loading script assemblies on validation-only paths; skip `ValidateScriptCompilation` in `GetDeploymentComparisonAsync`/`StaleInstanceProbe` (staleness needs only the hash). This removes both the CPU spike and the non-collectible assembly leak from the Deployments page and bundle imports.
6. **[High] Invalidate Inbound API handler caches on import** — publish an ApiMethod-changed notification from `ApplyAsync` (or key `_scriptHandlers` by `(name, scriptHash)`), covering the management-path gotcha already in project memory at the same time.
7. **[High] Enforce locks on native alarm source overrides** at both `ManagementActor.HandleSetInstanceNativeAlarmSourceOverride` and `FlatteningService.ApplyInstanceNativeAlarmSourceOverrides` (carry `IsLocked` on `ResolvedNativeAlarmSource`).
8. **[Medium] Harden the deny-list:** add `System.Environment` members (`Exit`, `FailFast`, `GetEnvironmentVariable(s)`, `SetEnvironmentVariable`), close the reflection-gateway list (`Invoke`, `GetTypes`, `EntryPoint`), and decide/document the `System.Data` provider posture.
9. **[Medium] Deployment audit-write isolation:** move the post-success `LogAsync` in `DeployInstanceAsync` out of the try (or guard it) so an audit fault can never flip a committed Success record to Failed; skip the site round-trip when deleting `NotDeployed` instances.
10. **[Medium] Bound `LineDiffer` inputs; fix `ArtifactDiff` placeholder comparisons** (pass folder/template name maps into `CompareTemplate`) so Identical classification works for structured templates; consider an "ignore blocker" resolution for the Pass-1 heuristic on template scripts.
**Prioritized recommendations:** (1) gate or cache the Expression-trigger syntax check on read-only paths — keying any shared cache by globals surface (N1); (2) fold N2/N6 into the next low-severity sweep (two one-liners); (3) extend `EnumerateTrustGatedScripts` to instance alarm-override expressions and emit a wizard warning for locked-member overrides (N4+N5 share the same code region).
archreview/06-edge-integrations.md
+72 -111
View File
@@ -1,143 +1,104 @@
# Architecture Review 06 — Edge Integrations (Inbound & Outbound Boundaries)
# Architecture Review 06 — Edge Integrations (Inbound & Outbound Boundaries) — Round 2
Reviewer domain: the four components where ScadaBridge touches external systems.
**Date:** 2026-07-12 (round 2; round-1 report dated pre-2026-07-08, baseline commit `b910f5eb`; everything through HEAD `8c888f13` re-reviewed)
## Scope
Deep source review (every non-generated `.cs` file read) of:
Same four components as round 1:
- `src/ZB.MOM.WW.ScadaBridge.InboundAPI` — `POST /api/{methodName}`, API-key auth, Roslyn script handlers, parameter/return validation, audit middleware, routed site calls.
- `src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway` — outbound HTTP client (`Call`/`CachedCall`), `DatabaseGateway` (`Connection`/`CachedWrite`), HTTP + SQL error classifiers, S&F handoff.
- `src/ZB.MOM.WW.ScadaBridge.NotificationService` + the delivery adapters in `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/Delivery` — SMTP (MailKit) wrapper, OAuth2 token service, SMTP/SMS error classifiers, credential redactor, Email/SMS adapters.
- `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier` — Native-AOT `WWNotifier.exe` client (arg parser, config loader, failover loop, YES/NO reporter).
- `src/ZB.MOM.WW.ScadaBridge.InboundAPI` — `POST /api/{methodName}`, API-key auth, Roslyn script handlers, parameter/return validation, routed site calls.
- `src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway` — outbound HTTP client (`Call`/`CachedCall`), error classifiers, S&F handoff.
- `src/ZB.MOM.WW.ScadaBridge.NotificationService` + `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/Delivery` — SMTP/OAuth2, SMS (Twilio) adapter, outbox dispatch/retry policy.
- `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier` — Native-AOT `WWNotifier.exe` client (spec: project README + `docs/plans/2026-06-26-delmia-recipe-notifier-design.md`).
Specs compared: `docs/requirements/Component-InboundAPI.md`, `Component-ExternalSystemGateway.md`, `Component-NotificationService.md`, `docs/plans/2026-06-26-delmia-recipe-notifier-design.md`. Test projects examined: `tests/ZB.MOM.WW.ScadaBridge.{InboundAPI,ExternalSystemGateway,NotificationService,NotificationOutbox,DelmiaNotifier}.Tests`.
## Method
## Maturity Verdict
1. Read the round-1 report and `archreview/plans/PLAN-06-edge-integrations.md` (24 tasks; tracker claims 21 findings fixed, 3 deferred/won't-fix, merged to main `3e964ac2`).
2. Verified **every** round-1 finding against current source with file:line evidence — no plan claim taken on trust. Key verifications: the revision-checked handler cache, `InvalidateMethod`/change-bus wiring, deferred DI-scope disposal, the full `TimeoutSeconds` vertical slice (entity → EF migration → artifact contract → site SQLite → client → mgmt → CLI → UI → doc), `{param}` path templates, bounded response buffering, JSON `AuthConfiguration` with legacy fallback, SMS retry-policy wiring, OAuth2 authority/scope, deterministic SMTP selection.
3. Fresh sweep of `git diff b910f5eb..HEAD` scoped to the four projects, their test projects, the two PLAN-06 EF migrations (`20260710075422_AddExternalSystemTimeoutSeconds`, `20260710080636_AddSmtpOAuth2AuthorityScope`), plus the touched shared surfaces (Host `Program.cs`, `ManagementActor.cs`, `StoreAndForwardService.cs` interaction, `ScadaBridgeTelemetry.cs`). Static review only (suites verified green 2026-07-10).
This is a mature, unusually well-hardened edge layer. The code shows evidence of multiple hardening passes: enumeration-safe auth responses, bounded body capture with UTF-8-safe truncation, ordered cancellation-vs-timeout catch discrimination repeated consistently across HTTP, SQL, SMTP, and SMS paths, poison-message parking on both S&F retry paths, credential redaction on every error string, and secrets encrypted at rest (`ExternalSystemConfiguration.cs:27-29,83-85`). Error classification matches the spec (5xx/408/429/connection = transient) on all four channels and is centralized in dedicated classifiers with tests. The remaining risk is concentrated in three places: **cache coherence of compiled inbound script handlers across central failover** (a silent-stale-code bug, the one genuinely High-severity defect found), **two acknowledged spec-vs-code gaps in the External System Gateway** (per-system timeout and path templating), and **unbounded buffering of outbound HTTP response bodies**. DelmiaNotifier is small, faithful to its design doc, and fully tested at the logic level.
## Verdict
**Round 1:** mature edge layer with risk concentrated in the stale compiled-handler cache (High), two ESG spec gaps (High/Medium), and unbounded response buffering (Medium). **Round 2:** all of that is genuinely fixed and verified in source — 20 of 25 findings fully fixed, 2 partially, 3 deferred as agreed; the only notable residue is one Medium (the script-artifact invalidation bus publishes to zero subscribers while a Host comment claims the Inbound API subscribes) plus four Lows at the edges of otherwise-correct fixes. No Critical or High new findings. This domain is now in the best shape of any round-1 state I compared against.
---
## Findings — Dimension 1: Stability
## Round-1 Finding Disposition
### [High] Compiled inbound handler cache goes stale across central failover — old script serves silently with HTTP 200
| # | Finding (round 1) | R1 severity | Disposition | Evidence |
|---|---|---|---|---|
| S1 | Compiled inbound handler cache stale across failover (+ known-bad mirror) | High | **Fixed (verified)** | `InboundScriptExecutor.cs:31` (`CachedHandler(Script, Handler)` record), `:371-405` per-request ordinal revision check + known-bad script-text compare + recompile-in-place; known-bad keyed name→script `:52`, overwrite-newer `:71-83`; DB row authoritative per doc-comment `:157-168`. Tests: `InboundScriptExecutorStalenessTests.cs:31,49,67,80` |
| S2 | Method timeout abandons script; DI scope disposed underneath it | Medium | **Fixed (verified)** | `InboundScriptExecutor.cs:328` hoisted `handlerTask`; `:473-494` finally defers scope disposal to a `ContinueWith` that observes the fault; orphan gauge `_abandonedExecutions` `:92-100`; counter `ScadaBridgeTelemetry.cs:104`. (Gauge is internal + OTel counter only — not on the health snapshot, which the plan scoped as "promote later".) |
| S3 | ESG API-key `AuthConfiguration` colon-splitting ambiguous | Medium | **Fixed (verified)** | `ExternalSystemClient.cs:594-599` JSON-first sniff (`{`-prefix), apikey JSON form `:603-625`, basic JSON form `:641-658`, legacy colon fallback preserved `:627-637,661-677`, tolerant parser `TryParseJsonAuth` `:701-722`. Spec updated `Component-ExternalSystemGateway.md:40-41` (colon-in-key preserved verbatim). Tests `ExternalSystemClientTests.cs:832,855,880,906` |
| S4 | DelmiaNotifier failover duplicates undocumented (at-least-once) | Medium | **Fixed (verified — docs)** | `docs/plans/2026-06-26-delmia-recipe-notifier-design.md:108-129` new "Delivery semantics: at-least-once" section + MUST-be-idempotent requirement on `(MachineCode, DownloadPath, WorkOrderNumber)`; `DelmiaNotifier/README.md:57-62` |
| S5 | SMS adapter keeps sending after first transient — duplicate amplification | Medium | **Fixed (verified)** | `SmsNotificationDeliveryAdapter.cs:158-165` breaks recipient loop at first transient; roll-up doc `:293-298`. Test `SmsNotificationDeliveryAdapterTests.cs:303` (`MidListTransient_ShortCircuits_RemainingRecipientsNotAttempted`) |
| S6 | `ErrorClassifier.IsTransient(Exception)` treats any OCE as transient | Low | **Fixed (verified)** | `ErrorClassifier.cs:52-57` cancellation-aware overload; token-less overload carries steer-away doc `:29-33`; call site uses it `ExternalSystemClient.cs:343` |
| S7 | First-of-many SMTP configuration nondeterministic | Low | **Fixed (verified)** | `EmailNotificationDeliveryAdapter.cs:78` `OrderBy(c => c.Id).FirstOrDefault()` + multi-row warning `:83-89`; same in retry-policy resolver `NotificationOutboxActor.cs:450-465` |
| S8 | OAuth2 token cache field visibility | Low | **Fixed (verified)** | `OAuth2TokenService.cs:143-152` immutable `TokenValue` record published through a single `volatile` reference; lock-free read `:61-65`, double-check under per-entry lock `:71-75` |
| S9 | Non-JSON content-type with body yields misleading 400 | Low | **Fixed (verified)** | `EndpointExtensions.cs:186-194` returns 415 `UNSUPPORTED_MEDIA_TYPE`; body with no Content-Type still lenient-parsed. (Chunked-body edge → new finding N5.) |
| P1 | Outbound HTTP responses buffered unbounded | Medium | **Fixed (verified)** | `ExternalSystemClient.cs:329-330` `ResponseHeadersRead`; `ReadBodyBoundedAsync` `:451-483` Content-Length fast-fail + streamed cap+1 abort; `ExternalSystemGatewayOptions.MaxResponseBodyBytes` (10 MiB default) `ExternalSystemGatewayOptions.cs:21`, validated `ExternalSystemGatewayOptionsValidator.cs:29`. Test `ExternalSystemClientTests.cs:1047` |
| P2 | Serial per-recipient Twilio POSTs inside a serial sweep | Medium | **Fixed (verified, beyond plan)** | Plan deferred bounded parallelism, but the outbox dispatcher gained it anyway (arch-review-04 stream): `NotificationOutboxActor.cs` `DeliverGatedAsync` with `SemaphoreSlim(ResolvedMaxParallelDeliveries)` + per-delivery DI scope (`NotificationOutboxOptions.cs:69`); combined with S5's first-transient short-circuit the black-hole worst case is bounded to ~1 timeout per notification |
| P3 | Per-delivery TCP+TLS SMTP handshake | Low | **Won't-fix (accepted)** | Deliberate, documented tradeoff; pooling seam still open in `MailKitSmtpClientWrapper` |
| P4 | Startup compiles every inbound method serially | Low | **Fixed (verified)** | `Host/Program.cs:415` `Parallel.ForEach(methods, method => executor.CompileAndRegister(method))` |
| C1 | Spec drift: per-system ESG timeout specified but not implemented | High | **Fixed (verified — full slice)** | Entity `ExternalSystemDefinition.cs:25`; EF `ExternalSystemConfiguration.cs:32`; migration `20260710075422_AddExternalSystemTimeoutSeconds.cs:13-15` (`AddColumn<int>` on `ExternalSystemDefinitions`); additive artifact `ExternalSystemArtifact.cs:9` (`int TimeoutSeconds = 0` trailing); site SQLite `SiteStorageService.cs:180,735-742` + `SiteExternalSystemRepository.cs:39,71`; client honor `ExternalSystemClient.cs:315-317`; mgmt `ManagementActor.cs:1878,1898`; CLI `ExternalSystemCommands.cs:52,93`; UI `ExternalSystemForm.razor`; doc `Component-ExternalSystemGateway.md:44,107`. Test `ExternalSystemClientTests.cs:975` |
| C2 | Spec drift: path templates never substituted | Medium | **Fixed (verified)** | `ExternalSystemClient.cs:500-501` identifier-only `PathParamRegex`; `:525-543` escaped substitution + consumed-set; excluded from body `:295-298` and query `:556`; doc `Component-ExternalSystemGateway.md:49,81`. Test `:1273`. (Buffered-retry edge → new finding N2.) |
| C3 | Inbound error responses lack documented `code` field / SITE_UNREACHABLE as generic 500 | Low | **Fixed (verified)** | `Error(code, message, status)` helper `EndpointExtensions.cs:66-67` used on every branch (`UNAUTHORIZED :126`, `NOT_APPROVED :166`, `UNSUPPORTED_MEDIA_TYPE :191`, `INVALID_JSON :216`, `VALIDATION_FAILED :235`, passthrough `:279-282`); filter bodies `InboundApiEndpointFilter.cs:59,73` (`STANDBY_NODE`/`BODY_TOO_LARGE`); executor codes `InboundScriptExecutor.cs:387,399,451,463,469` incl. `SITE_UNREACHABLE` via typed `SiteUnreachableException` (`RouteHelper.cs:16-20,386-395`); doc table `Component-InboundAPI.md:111-121`. (Classification mechanism → new finding N3.) |
| C4 | OAuth2 SMTP Microsoft-365-only despite broader spec | Low | **Fixed (verified)** | `OAuth2TokenService.cs:48-56,89-96` optional authority (`{tenant}` template) + scope, cache keyed by credentials\|authority\|scope; entity `SmtpConfiguration.cs:34,40`; migration `20260710080636_AddSmtpOAuth2AuthorityScope.cs:13-22`; adapter passthrough `EmailNotificationDeliveryAdapter.cs:198-201`; CLI `NotificationCommands.cs:178-184`; UI `SmtpConfiguration.razor`; doc `Component-NotificationService.md:49-50` |
| C5 | `InboundApiEndpointFilter` freezes options at construction | Low | **Fixed (verified)** | `InboundApiEndpointFilter.cs:27,34,66` `IOptionsMonitor<InboundApiOptions>` read per request |
| U1 | Script trust boundary static-only (no runtime sandbox) | Underdeveloped | **Deferred (accepted)** | Unchanged by design; `ForbiddenApiChecker` still the static gate (`InboundScriptExecutor.cs:213-223`). Logged in the deferred-work register |
| U2 | No cache-coherence mechanism for compiled handlers | Underdeveloped | **Partially fixed** | The revision-check self-heal (S1) is the real fix and works standalone; `InvalidateMethod` seam exists (`InboundScriptExecutor.cs:136-140`) and the ManagementActor delete path uses it (`ManagementActor.cs:2953`). **But** the claimed bus consumer was never wired — see new finding N1 |
| U3 | Per-SMS retry settings are dead fields | Underdeveloped | **Fixed (verified)** | `NotificationOutboxActor.cs:469-489` `ResolveSmsPolicyAsync` reads `SmsConfiguration.MaxRetries/RetryDelay` (SMTP fallback when absent), per-type selection `:591-595`, non-positive clamp `:494-518`; doc `Component-NotificationService.md:67`. Tests `NotificationOutboxActorRetryEmissionTests.cs` |
| U4 | Twilio accept-only + `AccountSid` un-escaped in URI | Underdeveloped | **Partially fixed (as planned)** | SID format regex at config save `ManagementActor.cs:2156-2161` (`^AC[0-9a-fA-F]{32}$`). Status-callback webhook / per-recipient state: deferred as documented (`Component-NotificationService.md:95`). Pre-validation rows are not retro-checked (minor) |
| U5 | Delmia duplicate semantics undocumented; no CI artifact for shipped binary | Underdeveloped | **Fixed (docs) / Deferred (CI, accepted)** | Docs verified under S4; Windows AOT CI artifact deferred — repo has no Windows runner |
| U6 | Test gaps: handler staleness, oversized outbound body, mid-list SMS transient | Underdeveloped | **Fixed (verified)** | `InboundScriptExecutorStalenessTests.cs` (5 tests), `ExternalSystemClientTests.cs:1047` (oversized body), `SmsNotificationDeliveryAdapterTests.cs:303` (mid-list transient) — exactly the three named gaps |
| U7 | API key rotation procedure unwritten | Underdeveloped | **Fixed (verified)** | `Component-InboundAPI.md:75-77+` "Key rotation" section documenting the dual-key create → migrate → disable procedure |
`InboundScriptExecutor` is a per-node singleton (`ServiceCollectionExtensions.cs:19`) whose `_scriptHandlers` and `_knownBadMethods` caches (`InboundScriptExecutor.cs:26,39`) are mutated only by:
**Disposition counts:** Fixed (verified) 20 · Partially fixed 2 (U2, U4) · Deferred/won't-fix (accepted) 3 (P3, U1, U5-CI) · Not fixed 0 · Regressed 0.
- the node's own startup compile of all methods (`Host/Program.cs:379-388`), and
- `ManagementActor.HandleUpdateApiMethod`/`HandleDeleteApiMethod` → `CompileAndRegister`/`RemoveHandler` (`ManagementActor.cs:2631,2650`) — which run **only on the node hosting the ManagementActor singleton (the active node)**.
Also verified: T2's DB-authoritative semantics propagated to the management warning strings (`ManagementActor.cs:2979-2980`) and the rewritten Direct SQL Warning (`Component-InboundAPI.md:158-162`); the two EF migrations are non-empty and additive; PLAN-08's eager options validation landed for all three components (`InboundApiOptionsValidator.cs`, `ExternalSystemGatewayOptionsValidator.cs`, `NotificationOptionsValidator.cs`, registered with `ValidateOnStart`, e.g. `Host/Program.cs:298-300`); the DelmiaNotifier code diff since baseline is XML doc-comments only — the binary's behavior is unchanged.
The standby central node compiles every method at *its* startup and then never hears about updates. `ExecuteAsync` looks the handler up by **name only** (`InboundScriptExecutor.cs:311`) — no revision/hash check — even though the endpoint fetches the fresh `ApiMethod` row (with the current `Script` text) from the DB on every request (`EndpointExtensions.cs:142`).
---
Concrete failure scenario:
1. Method `GetOrderStatus` updated via CLI on active node A → A recompiles, serves new logic.
2. Weeks later A fails over to B. B still holds the delegate compiled from the script text as of B's process start.
3. B serves the **old** script indefinitely — HTTP 200, correct-looking JSON, `UpdateApiMethod` long ago reported success. Nothing recompiles until B restarts.
## New Findings (Round 2)
The mirror-image variant: a method that was broken at B's startup landed in B's `_knownBadMethods`; it was later fixed via management update (clears the record on A only). After failover, B answers `"Script compilation failed for this method"` for a method that compiles fine.
### [Medium] N1 — The script-artifact invalidation bus publishes into the void; the Host comment claims a consumer that does not exist
This is the cluster-wide generalization of the already-known "direct SQL edit doesn't clear the cache" issue (documented as a Direct SQL Warning, `Component-InboundAPI.md:129-131`) — but here the write went through the *supported* management path and is still lost. A per-request script-hash comparison against the freshly-fetched row would fix failover staleness, direct-SQL staleness, and known-bad staleness in one move at negligible cost (the row is already in hand).
`Host/Program.cs:96-99` registers `InProcessScriptArtifactChangeBus` with the comment "the Inbound API compiled-handler cache **subscribes as the consumer (wired in plan 06)**" — but a repo-wide search finds **zero** `Subscribe` call sites: the only producer is the Transport bundle importer (`BundleImporter.cs:1811`), and nothing consumes `ScriptArtifactsChanged` for ApiMethods, SharedScripts, *or* Templates. The master tracker's Wave-4 summary makes the same claim ("`InvalidateMethod` seam consuming PLAN-05's `IScriptArtifactChangeBus` contract"). What actually shipped is the seam (`InboundScriptExecutor.InvalidateMethod`, `InboundScriptExecutor.cs:136-140`) plus one direct call on the delete path (`ManagementActor.cs:2953`).
Deleted methods are safe (fresh DB lookup returns null → 403); methods *created* after standby startup are safe (lazy compile, `InboundScriptExecutor.cs:314-330`).
Correctness is **not** broken for the Inbound API — the per-request revision check (S1) is the designed fallback and fully self-heals bundle-imported ApiMethod scripts. But (a) the Host comment and tracker are factually wrong and will misdirect the next person debugging cache coherence; (b) the PLAN-05→PLAN-06 handoff ("when both land, wire the bus to the invalidate seam", tracker §98) is still open; (c) for the other published `ArtifactKind`s there is no consumer and no documented self-heal claim — if any cache ever keys on SharedScript/Template text at central (e.g. a future compile-verdict cache), this silent no-op bus is a trap. Either wire the one-line subscription in the Host composition root (`bus.Subscribe(n => { if (n.Kind == ApiMethod) executor.InvalidateMethod(n.Name); })`) or fix the comment/tracker to say the fallback is the mechanism.
### [Medium] Method timeout abandons the running script; its DI scope is disposed underneath it
### [Low] N2 — A now-unresolvable path template on a buffered message is classified transient and retried to exhaustion
On timeout, `handler(context).WaitAsync(cts.Token)` (`InboundScriptExecutor.cs:333`) returns control but does not stop the underlying script task — cancellation is cooperative via `ctx.CancellationToken` only. The `finally` block then disposes the per-execution DI scope (`InboundScriptExecutor.cs:389-393`) while a non-cooperative script (e.g. mid-flight in a poorly-authored loop, or blocked in a routed call that ignores the token) may still be running and holding `InboundDatabaseHelper` → the scoped `IDatabaseGateway`. Consequences: `ObjectDisposedException`s surfacing as unobserved background faults, and — since the inbound API deliberately has no rate limiting (`Component-InboundAPI.md:140`) — repeated timeouts of a slow method accumulate orphaned executions with no counter or cap. Recommend at minimum a metric for abandoned executions, and deferring scope disposal until the handler task actually completes (`ContinueWith` on the abandoned task).
`DeliverBufferedAsync` deliberately parks deterministic failures (malformed JSON payload → `return false`, `ExternalSystemClient.cs:186-195`), but the **new** `{param}` substitution throws `ArgumentException` for a placeholder with no matching/non-null parameter (`ExternalSystemClient.cs:535-537`), and `ValidateHttpMethod` throws the same type (`:427-436`). Neither is caught in `DeliverBufferedAsync` (`:174-225` catches only `Permanent…`; everything else propagates), and the S&F retry sweep's catch-all treats any exception as transient (`StoreAndForwardService.cs:822-830`). Scenario: a method's path is edited to add `{id}` while calls are buffered — every already-buffered message now deterministically throws, gets counted as a *transient* failure, and burns the full default 50 retries (`StoreAndForwardOptions.cs:21`) before parking with a misleading "transient" `LastError`. Same shape as the JsonException poison fix in the same method; catch `ArgumentException` there and `return false` (park immediately).
### [Medium] ESG API-key `AuthConfiguration` colon-splitting is ambiguous — a key containing `:` silently breaks auth
### [Low] N3 — `SITE_UNREACHABLE` classification is substring-sniffing on error message text
`ExternalSystemClient.ApplyAuth` (`ExternalSystemClient.cs:489-499`) interprets `AuthConfiguration` as `"HeaderName:KeyValue"` when it contains a colon, else as a bare key with default header `X-API-Key`. An API key whose *value* legitimately contains a colon (not rare for opaque tokens) is silently misparsed: the prefix becomes a bogus header name and the request goes out mis-authenticated with only a remote 401 to debug from. The Basic branch (`:504-519`) is fine (split-2 keeps colons in the password), but the apikey branch has no escape hatch. Structured JSON config (`{"header":"X-API-Key","key":"..."}`) — which the entity doc-comment (`ExternalSystemDefinition.cs:13`, "JSON-serialized authentication configuration") already implies — would eliminate the ambiguity.
`RouteHelper.IsUnreachable` (`RouteHelper.cs:391-395`) decides between `SiteUnreachableException` and plain `InvalidOperationException` by matching `"unreachable"` / `"no contact"` in the failure message produced by `IInstanceRouter`/`IInstanceLocator` — a different component. A harmless rewording of those messages silently downgrades the spec'd `SITE_UNREACHABLE` code to `SCRIPT_ERROR` with no test failing at the boundary. A typed reachability flag on the router result (or a shared const for the message) would make the contract explicit. All five throw sites already funnel through `RoutingFailure` (`:189,237,301,352,372`), so the change is localized.
### [Medium] DelmiaNotifier failover can duplicate a processed notification
### [Low] N4 — Bounded response read decodes as UTF-8 unconditionally, dropping charset honoring
`HttpRecipeSender.cs:36-45` maps **any** `HttpRequestException` and any non-caller timeout to `ConnectFailed`, which the loop rolls over to the next URL (`Notifier.cs`, `continue`). Both cases include "the server received and possibly processed the request, but the response never arrived" (connection reset after send; slow response exceeding `TimeoutSeconds`). The next node then receives a second `DelmiaRecipeDownload` POST for the same recipe. This matches the design table (`2026-06-26-delmia-recipe-notifier-design.md:97-99` lists timeout as a failover trigger) — so this is a *design-level* at-least-once semantics that is nowhere documented as such; the inbound `DelmiaRecipeDownload` method must be idempotent on `(MachineCode, DownloadPath, WorkOrderNumber)` or duplicates will surface downstream. Worth one paragraph in the design doc and a check of the deployed method script.
Round 1's `ReadAsStringAsync` honored the response's declared charset; the replacement `ReadBodyBoundedAsync` ends with `Encoding.UTF8.GetString(...)` (`ExternalSystemClient.cs:482`) regardless of `Content-Type: ...; charset=iso-8859-1`. For a legacy external system replying in a non-UTF-8 single-byte encoding, response text (and error bodies embedded into script-visible messages) becomes mojibake — a subtle behavior regression smuggled in by an otherwise-correct fix. Reading `content.Headers.ContentType?.CharSet` and falling back to UTF-8 restores parity at negligible cost.
### [Medium] SMS adapter keeps sending after the first transient failure — inflating duplicate texts on retry
### [Low] N5 — The new 415 guard is bypassed by chunked (no Content-Length) bodies
In `SmsNotificationDeliveryAdapter.DeliverAsync` (`SmsNotificationDeliveryAdapter.cs:148-170`), a transient result does not break the loop; the remaining recipients are still attempted. Since *any* transient rolls the **whole notification** up to `Transient` (`RollUp`, `:297-303`) and retry re-texts everyone, each additional number accepted after the first transient is a guaranteed duplicate SMS on the retry. Short-circuiting on the first transient would reduce duplicates at no delivery cost (the retry re-attempts all numbers anyway). The re-send-to-all characteristic itself is documented (`Component-NotificationService.md:102`), but this amplification is avoidable within v1's no-per-recipient-state constraint.
`EndpointExtensions.cs:187` gates the 415 on `ContentLength > 0`; a chunked non-JSON body has null `ContentLength`, skips the 415, then also skips JSON parsing (the parse condition `:206-207` is `ContentLength > 0 || ContentType contains json`), so `body = null` and the caller gets `VALIDATION_FAILED` "missing required parameter" — the same misleading-error class S9 set out to fix, now confined to the chunked edge. Checking `Request.Headers.ContentLength ?? (Request.Body != null && Request.Headers.TransferEncoding.Count > 0)` — or simply keying the 415 on a present non-JSON `ContentType` alone — closes it.
### [Low] `ErrorClassifier.IsTransient(Exception)` treats any `OperationCanceledException` as transient
## What's Genuinely Good
`ErrorClassifier.cs:32-38` classifies bare `OperationCanceledException`/`TaskCanceledException` as transient. Every current call site is protected by ordered catch filters that peel off caller-cancellation first (`ExternalSystemClient.cs:314-327`), so behaviour is correct today — but the predicate is public and a future call site that skips the ordering would buffer caller-cancelled work into S&F. A `cancellationToken` parameter (as `SmtpErrorClassifier.Classify` and `SmsErrorClassifier.Classify` already take) would make it misuse-resistant.
- **The S1 fix is the textbook version of itself.** The DB row is authoritative, the marginal cost is one ordinal string compare of a row already in hand, the known-bad cache became revision-aware in the same move (name→failing-script map, `InboundScriptExecutor.cs:52,71-83`), the test seam is explicitly exempt and documented (`:113-126`), and the semantics change (broken save now 500s consistently on every node instead of node-divergent stale success) was propagated to the management warning strings *and* the component doc — code, warning text, and spec all tell the same story.
- **The `TimeoutSeconds` slice is complete in the way this repo demands.** Ten distinct layers (entity → EF → migration → additive wire contract → site SQLite migration+upsert+read → client resolution order → management → CLI → UI → spec) all present and mutually consistent, with the artifact field additive-trailing per the message-evolution rule.
- **Bounded response reading reuses the house pattern.** Content-Length fast-fail plus streamed cap+1 abort, oversize classified permanent (correct: retrying will not shrink it, and buffering into S&F would defeat the cap), timeout token threaded through the body read so "round-trip" means round-trip (`ExternalSystemClient.cs:348-366`).
- **The JSON auth migration is genuinely backward compatible** — `{`-sniff, tolerant parse that falls through to a warn-and-send-unauthenticated path (never throws on hostile config), legacy colon behavior bit-for-bit preserved, and the never-log-the-value discipline maintained on every new branch.
- **Notification retry policy resolution is now channel-typed, clamped, deterministic, and cheap** (resolved once per sweep, selected per notification), and the outbox dispatcher's bounded parallelism uses a per-delivery DI scope so EF `DbContext` non-thread-safety is respected — the concurrency comment block reads like it was written by someone who has been burned before.
- **Docs kept their promises**: the at-least-once section in the Delmia design doc names the exact idempotency key and puts the obligation on the receiving side where it belongs; the key-rotation procedure is written against the actual `sbk_` token design.
### [Low] First-of-many SMTP configuration is nondeterministic
## New-Finding Severity Tally (round 2 only)
`EmailNotificationDeliveryAdapter.cs:75-76` and `NotificationOutboxActor.ResolveRetryPolicyAsync` (`NotificationOutboxActor.cs:373-375`) both use `GetAllSmtpConfigurationsAsync()` + `FirstOrDefault()`. With more than one SMTP row the selected server and the retry policy depend on repository ordering. Either enforce single-row semantics or add an explicit "active" flag.
| Severity | Count |
|---|---|
| Critical | 0 |
| High | 0 |
| Medium | 1 (N1) |
| Low | 4 (N2, N3, N4, N5) |
### [Low] OAuth2 token cache: unsynchronized field visibility and unbounded growth
`OAuth2TokenService.CacheEntry.Token/Expiry` are plain fields read outside the per-entry lock (`OAuth2TokenService.cs:48-51`) and written inside it (`:93-94`). The races are benign (worst case: one redundant token fetch), but `volatile`/`Volatile.Read` would make it airtight. The cache is also never evicted — bounded in practice by the number of distinct SMTP credential triples, so acceptable.
### [Low] Non-JSON content-type with a body yields a misleading 400
`EndpointExtensions.cs:181-194`: `ContentLength > 0 || ContentType contains "json"` means a `text/plain` or `application/x-www-form-urlencoded` body is force-parsed as JSON and rejected as "Invalid JSON in request body" rather than a 415. Cosmetic, but the error steers the caller at the wrong problem.
## Findings — Dimension 2: Performance
### [Medium] Outbound HTTP responses are buffered unbounded
`ExternalSystemClient.InvokeHttpAsync` uses `client.SendAsync(request, token)` with the default `ResponseContentRead` completion option (`ExternalSystemClient.cs:312`) and then `ReadAsStringAsync` (`:337`) — the *entire* response body is materialized in memory with **no size cap**. `MaxErrorBodyChars` (`:386`) truncates only the string embedded into error messages, *after* the full read. The inbound side caps request bodies at 1 MiB precisely because "no rate limiting… an explicit, modest cap bounds per-request allocations" (`InboundApiOptions.cs:16-19`); the outbound side extends no such protection against a misbehaving or compromised external system streaming hundreds of MB into a site node (which also runs the instance actors). Use `HttpCompletionOption.ResponseHeadersRead` plus a bounded read (the audit middleware's cap+1 pattern is directly reusable).
### [Medium] Serial per-recipient Twilio POSTs inside a serial dispatch sweep
`SmsNotificationDeliveryAdapter` sends one POST per phone number sequentially (`SmsNotificationDeliveryAdapter.cs:148-170`), and the outbox dispatcher delivers each due notification sequentially within a sweep (`NotificationOutboxActor.cs:320-345`). Worst case for one 50-recipient list against a black-holing endpoint: 50 × 30 s = 25 minutes inside one sweep, during which *all* other due notifications (including email) queue behind it and appear "stuck" on the KPI tiles. Bounded parallelism per notification (e.g. `Parallel.ForEachAsync` with 4–8 DOP — Twilio tolerates it) and/or first-transient short-circuit (see Stability above) would bound the sweep. At current scale (small lists) this is latent, not live.
### [Low] Per-delivery TCP+TLS SMTP handshake
`MailKitSmtpClientWrapper` is deliberately one-connection-per-delivery, extensively documented with the factory seam left open for pooling (`MailKitSmtpClientWrapper.cs:12-38`). Correct tradeoff at current volume; noted only as the first knob if notification volume grows.
### [Low] Startup compiles every inbound method serially
`Host/Program.cs:383-387` runs Roslyn compilation per method in a foreach before `app.RunAsync()`. Fine at tens of methods; would stretch central startup (and failover-recovery window) at hundreds. `Parallel.ForEach` or lazy-only compilation are cheap options later.
### [Good] The hot-path hygiene elsewhere is exemplary
Bounded audit capture at the source with `ArrayPool` scratch buffers and cap+1 over-read detection (`AuditWriteMiddleware.cs:442-502`), fire-and-forget audit write with fault observation (`:358-359,380-398`), `EnableBuffering` skipped for bodyless requests (`:169-191`), name-keyed indexed repository lookups on the ESG hot path (`ExternalSystemClient.cs:540-550`), known-bad-method cache preventing per-request Roslyn recompiles with a 1000-entry flood cap (`InboundScriptExecutor.cs:38-63`), and per-system `HttpClient` factory clients with `MaxConnectionsPerServer` scoped to gateway-owned names only — explicitly avoiding process-global handler replacement (`ExternalSystemGateway/ServiceCollectionExtensions.cs:28-44,92-105`). No socket-exhaustion patterns anywhere: all four components use `IHttpClientFactory` (DelmiaNotifier's single-shot process makes its one `new HttpClient` correct).
## Findings — Dimension 3: Conventions & Spec Fidelity
### [High] Spec drift: per-system ESG timeout is specified but not implemented
`Component-ExternalSystemGateway.md:38` ("**Timeout**: Per-system timeout for all method calls") and `:101` are explicit, but `ExternalSystemDefinition` has no timeout field (`Commons/Entities/ExternalSystems/ExternalSystemDefinition.cs`) and the client acknowledges it: "ExternalSystemDefinition has no per-system Timeout field yet, so the configured DefaultHttpTimeout is the effective round-trip limit" (`ExternalSystemClient.cs:300-305`). Every external system — a fast local MES and a slow remote recipe manager alike — shares one global 30 s (`ExternalSystemGatewayOptions.cs:9`). Repo rules say the design doc is the spec and doc+code travel together; either add the column (entity + EF config + migration + CLI/UI) or amend the spec to the global-timeout reality.
### [Medium] Spec drift: path templates (`/recipes/{id}`) are never substituted
The spec's method definition shows a templated relative path (`Component-ExternalSystemGateway.md:43`), but `BuildUrl` (`ExternalSystemClient.cs:431-463`) concatenates the path verbatim and routes parameters exclusively to the query string (GET/DELETE) or JSON body (POST/PUT/PATCH). A method authored with `{id}` in its path literally calls `.../recipes/{id}`. Either implement `{param}` substitution (with the consumed parameter removed from body/query) or strike the example from the spec before someone authors against it.
### [Low] Spec drift: inbound error responses lack the documented `code` field
`Component-InboundAPI.md:91-97` documents failure bodies as `{"error": ..., "code": "SITE_UNREACHABLE"}`; every actual error body is `{ error }` only (`EndpointExtensions.cs:111-113,153-155,191-193,212-214,253-255`) and a routed site-unreachable failure surfaces as a generic 500 "Internal script error" (`InboundScriptExecutor.cs:382-387` swallows the `InvalidOperationException` from `RouteTarget.Call`, `RouteHelper.cs:172-174`). Machine-readable error codes matter to exactly the audience of this API (external integrators); implement or de-spec.
### [Low] OAuth2 SMTP is Microsoft-365-only despite a broader spec
`OAuth2TokenService` hardcodes `https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token` and the `outlook.office365.com/.default` scope (`OAuth2TokenService.cs:73,80`); the spec says "For Microsoft 365 **and other modern SMTP providers** that require OAuth2" (`Component-NotificationService.md:48`). A configurable authority/scope on `SmtpConfiguration` is a small change; until then the spec overpromises.
### [Low] `InboundApiEndpointFilter` freezes its options at construction
The filter is a singleton taking `IOptions<InboundApiOptions>` (`InboundApiEndpointFilter.cs:32-38`), so a live change to `MaxRequestBodyBytes` is ignored until restart — while the sibling `AuditWriteMiddleware` deliberately hot-reads `IOptionsMonitor<AuditLogOptions>` per request (`AuditWriteMiddleware.cs:97,149`). Inconsistent; use the monitor.
### [Good] Convention adherence is otherwise strong
Options pattern with startup validators throughout (`SmsOptionsValidator`, `NotificationOptions`); additive message-contract evolution respected and even documented at the parameter level ("this optional parameter trails the CancellationToken because it was appended additively", `InboundScriptExecutor.cs:243-246`); secrets consistently encrypted at rest via `EncryptedStringConverter` (ESG `AuthConfiguration` + DB `ConnectionString`: `Configurations/ExternalSystemConfiguration.cs:27-29,83-85`; SMTP credentials + Twilio AuthToken: `Configurations/NotificationConfiguration.cs:69,118`); connection strings never leak into error text (`DatabaseGateway.cs:298`, enforced in messages); credential scrubbing shared, not duplicated (`CredentialRedactor.cs`); API-key auth delegated to the shared `ZB.MOM.WW.Auth.ApiKeys` package (`InboundAPI.csproj:34`) with peppered-HMAC constant-time verification per spec (`Component-InboundAPI.md:66` — the package itself is outside this repo and was not independently verified), generic 401 messages, and byte-identical 403s for not-found vs not-in-scope with the DB lookup ordered *after* the in-memory scope check to close the timing oracle (`EndpointExtensions.cs:117-156`).
## Underdeveloped Areas
1. **Script trust boundary is static-only.** `ForbiddenApiChecker` is candid: semantic + reflection-gateway hardening, "not a true sandbox… genuine containment needs a runtime boundary" (`ForbiddenApiChecker.cs:24-29`). Inbound scripts run in-process on the central node with `Microsoft.CSharp` dynamic binder referenced (`InboundScriptExecutor.cs:172-184`). The deferred restricted-`AssemblyLoadContext`/out-of-process option remains the single biggest security-hardening gap on this boundary.
2. **No cache-coherence mechanism for compiled handlers** (the High finding above) — no revision hash, no cluster invalidation message, no recompile-on-activation hook in `IActiveNodeGate`.
3. **Per-SMS retry settings are dead fields.** `SmsConfiguration.MaxRetries/RetryDelay` are persisted and transported but never read; SMTP config governs retry for *all* notification types (`NotificationOutboxActor.cs:358-401`; documented as deferred in `Component-NotificationService.md:65`).
4. **Twilio delivery is accept-only.** No status-callback webhook, no per-recipient delivery state (both documented as out-of-scope/future — `Component-NotificationService.md:93,102`). Also `AccountSid` is interpolated un-escaped into the request URI (`SmsNotificationDeliveryAdapter.cs:131`) — harmless for a valid SID, but a validation regex at config-save time would close the door.
5. **DelmiaNotifier duplicate-delivery semantics undocumented** (Stability finding); AOT publish is Windows-only and the live smoke is manual (`2026-06-26-delmia-recipe-notifier-design.md:133-136,150-151`) — no CI artifact for the actual shipped binary.
6. **Test gaps** (coverage is otherwise broad and genuinely good across all five test projects — endpoint gating, content-type case-sensitivity, schema-`$ref` runtime resolution, failover loop, both adapters, all four classifiers): nothing exercises the failover/staleness scenario for compiled handlers, nothing exercises an oversized outbound response body, and no test covers a multi-recipient SMS list with a mid-list transient (the duplicate-amplification path).
7. **API key rotation** is enable/disable + scope-set only (`ManagementActor.cs:2685+`); a documented dual-key rotation procedure (create → migrate caller → disable) would round out the ops story — the `sbk_<keyId>_<secret>` design supports it, it's just unwritten.
## Prioritized Recommendations
1. **[High] Make the inbound handler cache self-healing.** Store the compiled delegate together with a hash of the script text; on each request compare against the freshly-fetched `ApiMethod.Script` and recompile on mismatch (also purging `_knownBadMethods`). One change fixes failover staleness, direct-SQL staleness, and stale-bad-method staleness simultaneously; the DB row is already fetched per request so the marginal cost is one string hash.
2. **[High] Close the two ESG spec gaps** — per-system `Timeout` column (entity + EF migration + honored in `InvokeHttpAsync`) and either implement `{param}` path templating or amend `Component-ExternalSystemGateway.md`. Per repo rules, doc and code must travel together.
3. **[Medium] Cap outbound response buffering** in `ExternalSystemClient` (`ResponseHeadersRead` + bounded read, reusing the audit-capture pattern).
4. **[Medium] Bound the SMS blast radius**: short-circuit the recipient loop on first transient, and add bounded parallelism (or per-notification concurrency in the dispatch sweep) so one dead endpoint cannot stall the whole outbox.
5. **[Medium] Replace colon-split `AuthConfiguration` parsing with structured JSON** (the entity already claims to be JSON) with a backward-compatible fallback.
6. **[Medium] Handle abandoned inbound executions**: defer DI-scope disposal to actual handler completion, and add an orphaned-execution counter to the health snapshot.
7. **[Low] Document DelmiaNotifier's at-least-once failover semantics** and verify the deployed `DelmiaRecipeDownload` script is idempotent.
8. **[Low] Small consistency fixes**: `IOptionsMonitor` in `InboundApiEndpointFilter`; token parameter on `ErrorClassifier.IsTransient(Exception)`; configurable OAuth2 authority/scope; single-active-SMTP-config enforcement; implement or de-spec the inbound `code` error field.
Round-1 tally for comparison: 2 High, 6 Medium, 9 Low + 7 underdeveloped areas. The domain's residual risk has collapsed to one wiring/documentation gap and four edge-case polish items.
archreview/07-ui-management-security.md
+94 -101
View File
@@ -1,126 +1,119 @@
# Architecture Review 07 — Management & Presentation Surface
# Architecture Review 07 — Management & Presentation Surface (Round 2)
**Domain:** Central UI (Blazor Server), Management Service (ManagementActor + HTTP API), CLI, Security & Auth
**Reviewed:** `src/ZB.MOM.WW.ScadaBridge.CentralUI`, `src/ZB.MOM.WW.ScadaBridge.ManagementService`, `src/ZB.MOM.WW.ScadaBridge.CLI`, `src/ZB.MOM.WW.ScadaBridge.Security`, design docs `docs/requirements/Component-CentralUI.md`, `Component-ManagementService.md`, `Component-CLI.md`, `Component-Security.md`, and the corresponding test projects.
**Date:** 2026-07-12
**Domain:** Central UI (Blazor Server), Management Service (ManagementActor + HTTP API), CLI, Security & Auth — plus the post-initiative **Alarm Summary aggregated live-alarm stream** (central `ISiteAlarmLiveCache` / Blazor-circuit push side; the gRPC/site side belongs to review 02).
**Baseline:** round-1 report at commit `b910f5eb`; this round re-reviews everything through HEAD `8c888f13` (PLAN-07 execution `74c295e3..d8a39f3c` plus post-initiative work: live alarm stream `b91ed3c8`/`8c888f13`, Test-Run WaitForAttribute `22136d36`, PLAN-06 edge spillover into UI forms).
## Method
1. Read the round-1 report and `archreview/plans/PLAN-07-ui-management-security.md` in full.
2. Verified every round-1 finding's disposition **in current source** (no trust in plan claims) — every disposition below carries file:line evidence read this session.
3. Fresh sweep of `git diff b910f5eb..HEAD` scoped to `CentralUI`, `ManagementService`, `CLI`, `Security` (54 files, +2,397/−231), with a focused review of the new Alarm Summary live path (`AlarmSummary.razor`, `AlarmSummaryService.BuildFromLiveAlarms`, `ISiteAlarmLiveCache`, `SiteAlarmLiveCacheService`).
4. Static review only (suites verified green 2026-07-10); no builds/tests run.
## Round-1 vs Round-2 Verdict
**Round 1:** mature surface with sharp asymmetries — an auth-disabled production artifact, a silently-ignored deploy option, inconsistent secret projection, a fictional secured-write "Expired" state. **Round 2:** the fix plan genuinely landed — 17 of 24 findings fully fixed and verified in source (including both criticals), 6 partially fixed with the residue consciously deferred and documented, 1 was a false premise; the new code (frozen 139-entry authz matrix, dispatch-coverage guard, TTL'd secured writes, live alarm cache) is of high quality, and the remaining new findings are Medium-and-below — mostly seams the fixes themselves exposed (an unthrottled debug-hub bind the docs claim is covered, throttle keying behind Traefik, and site-scope not enforced on secured writes despite the doc saying scoping "is layered on at the LDAP-mapping level").
---
## Scope
## Round-1 Finding Disposition
Deep read of the ManagementActor (all 3,157 lines), the HTTP management/audit endpoints and authenticator, the full Security project (JWT service, cookie session validator, authorization policies, role mapper, service registration), the secured-writes end-to-end path (UI page → `SecuredWriteService` → actor handlers), the debug-stream path (`DebugStreamHub`, `DebugView.razor`), representative Blazor pages (Alarm Summary, Health, Secured Writes, Notification Report), UI-side trust-boundary services (`BrowseService`), the CLI HTTP client and command wiring, and deployed configuration in `docker/` and `deploy/wonder-app-vd03/`. Spec-vs-code comparison against the four component documents.
| Finding | R1 Severity | Disposition | Evidence (file:line) |
|---|---|---|---|
| C1 — `DisableLogin: true` shipped in production deploy artifact; no environment guard | Critical | **Fixed (verified)** | `Security/Auth/DisableLoginGuard.cs:17-28` (refuses outside Development without `AllowOutsideDevelopment` ack); wired fail-fast at the composition root `Host/Program.cs:149-154` *before* `AddSecurity`; `deploy/wonder-app-vd03/appsettings.Central.json` no longer contains `DisableLogin` (repo-wide grep: no non-`false` occurrence under `deploy/`, `docker/`, `docker-env2/`), and its `_comment_Security` documents the new posture |
| C2 — `deploy artifacts --site-id` silently ignored; fleet-wide deploy with no scope check | High | **Fixed (verified)** | `ArtifactDeploymentService.cs:228-255` (`DeployToAllSitesAsync`/`DeployToSiteAsync` over shared `DeployCoreAsync`); `ManagementActor.cs:2295-2316` honors `cmd.SiteId`, adds the explicit fleet-wide guard for site-scoped non-admins (`SiteScopeViolationException`, :2300-2306) **and** `EnforceSiteScope` (:2307) |
| C3 — DataConnection / ExternalSystem / DatabaseConnection secrets leak in List/Get + audit | High | **Fixed (verified)** | `ConfigSecretScrubber.cs` (recursive JSON elision + sentinel merge); `DataConnectionPublicShape` `ManagementActor.cs:1632-1647` applied to List/Get/Create/Update **and** audit afterState (:1653-1654, :1663, :1677-1678, :1697-1698), sentinel-preserving update :1692-1693; `ExternalSystemPublicShape` :1847-1857 + preserve-if-null :1897; `DatabaseConnectionPublicShape` :2835-2841 applied :2845-2878. Note: UI editors write via repositories directly (e.g. `ExternalSystemForm.razor`), so UI round-tripping was never at risk; CLI round-trips are sentinel-protected |
| S1 — 30 s server Ask vs 5-min bundle operations; import keeps applying after 504 | High | **Partially fixed (verified; residue deferred)** | `ManagementEndpoints.cs:41-58` — `ResolveAskTimeout(options, commandType)` gives Import/Preview/Export/DeployArtifacts/DeployInstance a 5-min default (`ManagementServiceOptions.cs:9`), call site :126; `ManagementService` timeout config shipped to all three topologies (`docker/central-node-a/appsettings.Central.json:51`, `docker-env2/.../:48`, `deploy/wonder-app-vd03/.../:49`); `ManagementServiceOptionsValidator` fail-fast. **Still open (accepted, plan-05 domain):** `ImportBundleCommand` (`TransportCommands.cs:121-128`) has no client-supplied idempotency id — a timeout-then-retry can still double-apply; the `TransportGate` single-flight (:3203) only narrows the window on one node |
| S2 — Pending secured writes never expire; "Expired" fictional | High | **Fixed (verified)** | `ISecuredWriteRepository.cs:100` (`TryMarkExpiredAsync` CAS), :111 (`CountAsync`); `TryExpireIfStaleAsync` `ManagementActor.cs:1224-1241` (TTL default 24 h, `ManagementServiceOptions.cs:12`; non-positive disables; emits `AuditKind.SecuredWriteExpire`); enforced **before** self-approval/CAS at approve (:1296-1302) and reject (:1453-1458); opportunistic sweep on list (:1493-1506); approve dialog now shows submission age (`SecuredWrites.razor:415`); spec'd in `Component-Security.md:145` |
| S3 — Poll timers overlap their own refreshes | Medium | **Fixed (verified)** | `CentralUI/Services/PollGate.cs` (Interlocked CAS gate); wired in `AlarmSummary.razor:315-326` and `Health.razor:552-562` |
| S4 — 200 MB bodies buffered as strings, ~4 copies resident | Medium | **Partially fixed (verified; residue deferred)** | `TransportGate` single-flight semaphore `ManagementActor.cs:3203-3213` wraps all three bundle handlers (released in `finally` :3317/:3361/:3442); export copy eliminated via `ms.GetBuffer().AsSpan(...)` :3309-3312. Body is still `StreamReader.ReadToEndAsync()` into a string (`ManagementEndpoints.cs:103-107`) with the 200 MB cap (:75); streaming multipart remains the deferred long-term shape |
| S5 — Fire-and-forget SignalR sends swallow failures | Low | **Fixed (verified)** | `DebugStreamHub.cs:221-232` — `ContinueWith(OnlyOnFaulted)` logs push failures on both the event and termination paths while keeping fire-and-forget semantics |
| P1 — Full LDAP bind per request; no throttle/lockout (spray oracle) | Medium | **Partially fixed (verified — one surface missed, see N1)** | `Security/LoginThrottle.cs` (fixed-window, per `username\|ip`, bounded at 10k keys, TimeProvider-driven); wired in `ManagementAuthenticator.cs:110-129` (covers `POST /management` + audit REST via `AuditEndpoints.cs:525`), `/auth/login` `AuthEndpoints.cs:38-58`, `/auth/token` :125-147. **`DebugStreamHub.OnConnectedAsync` still binds LDAP directly with no throttle** (`DebugStreamHub.cs:119-121`) — see new finding N1. Per-request bind cost itself unchanged (documented stateless design) |
| P2 — Unbounded list handlers materialize whole tables | Medium | **Partially fixed (verified; residue accepted)** | Additive `Skip`/`Take` on `ListTemplatesCommand`/`ListInstancesCommand` (`TemplateCommands.cs:5`, `InstanceCommands.cs:5`); shared clamped `Page` helper `ManagementActor.cs:576-580`; instances paged **after** the site-scope filter :797-799. `QueryDeployments` unfiltered branch still loads all records + instances (:2334-2339) and `ExportBundle` still loads all-entities for name resolution — documented deferral, bounded by fleet size |
| P3 — `ListSecuredWrites` hard-caps at 200, no paging | Medium | **Fixed (verified)** | `ListSecuredWritesCommand(…, int Skip = 0, int Take = 200)` `SecuredWriteCommands.cs:63`; handler clamps (1..500) + `CountAsync` total `ManagementActor.cs:1488-1491`; UI history pager with real bounds `SecuredWrites.razor:224-236, 343-349, 352-362` |
| P4 — Alarm Summary re-sorts/filters per render | Low | **Fixed (verified)** | Memoized `_visibleRows` (`AlarmSummary.razor:217-220`), `RecomputeVisibleRows()` :401 invoked only from refresh/filter (`@bind:after`, :114-156)/sort (:461) |
| C4 — Browse/Verify/Cert commands documented as actor commands but UI-only; no CLI parity | Medium | **Fixed (verified)** | Six actor handlers `ManagementActor.cs:3103-3161` (SiteIdentifier required + `EnforceSiteScopeForIdentifier` + relay via `CommunicationService`); dispatch arms :453-458; role gates: browse/search/verify → Designer (:258), cert trust → Admin (:219-223); additive `SiteIdentifier` on the six Commons records; CLI `Commands/ConnectionBrowseCommands.cs` (browse/search/verify-endpoint/certs list\|trust\|remove) registered in `CLI/Program.cs:26-27` |
| C5 — Role-casing asymmetry (UI ordinal vs actor ignore-case) | Medium | **Fixed (verified)** | `CanonicalizeMappingRole` `ManagementActor.cs:2203-2210` — stored casing canonicalized against `Roles.All` at both mapping write paths (:2212, :2223); invalid-role rejection message preserved |
| C6 — Area role gate contradicts three docs | Medium | **Fixed (verified)** | Reconciled on any-of `[Designer, Deployer]`: `AreaManagers` `ManagementActor.cs:171-178`, arm :228; `Component-ManagementService.md:224` documents the any-of gate and the rationale (docs' "Admin" was the stale side) |
| C7 — `InstanceName` passed into slot commented "InstanceId filter" | Low | **Fixed (verified)** | Behavior was correct (site `instance_id` stores UniqueName); comment now says so explicitly `ManagementActor.cs:3086-3089` |
| C8 — Stale CLAUDE.md note on SecuredWrite `SourceNode` | Low | **Fixed (verified)** | CLAUDE.md Security & Auth section now reads "(SecuredWrite audit rows stamp `SourceNode` via `ICentralAuditWriter`/`INodeIdentityProvider`.)" |
| UA1 — Secured-write lifecycle (no expiry, no paging, history ungated, dual-role hazard) | UA | **Fixed (verified)** | Expiry + paging per S2/P3; `ListSecuredWrites` gated to `SecuredWriteReaders` = Operator/Verifier/Admin (`ManagementActor.cs:184-185, :290`) with matching UI policy `RequireSecuredWriteAccess` (`AuthorizationPolicies.cs:99-106, :179-180`; page attribute `SecuredWrites.razor:5`); dual-role deployment hazard restated in `Component-Security.md:143` |
| UA2 — Test depth: no authz matrix over ~150 commands | UA | **Fixed (verified)** | `RequiredRoleMatrixTests.cs` — frozen ~139-entry `Expected` table + `EveryRegisteredCommand_IsInTheFrozenTable` reflection guard (:26, :194); `DispatchCoverageTests.cs` — every registered command must reach a dispatch arm (NotSupportedException sentinel; documented exclusions for `ResolveRolesCommand` and the site-routed `ReadTagValuesCommand`) |
| UA3 — CLI.Tests not in slnx | UA | **Fixed (verified)** | `ZB.MOM.WW.ScadaBridge.slnx:52` (landed via PLAN-08 T1; PLAN-07 T33 correctly skipped as duplicate) |
| UA4 — No single deferred-work tracking list | UA | **Fixed (verified)** | `docs/plans/2026-07-08-deferred-work-register.md` — actively maintained (deferred #10/#22/#12/#14 resolved through it post-initiative) |
| UA5 — Login endpoints lack throttling; user-enumeration check | UA | **Partially fixed (verified)** | Throttling as per P1 (with the hub gap, N1). Enumeration verified safe: `LdapAuthFailureMessages.cs:79-80` maps `BadCredentials` and `UserNotFound` to the identical "Invalid username or password." |
| UA6 — Accessibility debt on sortable headers | UA | **Partially fixed (accepted deferral)** | AlarmSummary headers: `tabindex="0"`, `aria-sort` (`AriaSortFor` :469-470), Enter/Space activation with Space preventDefault (:479-487, markup :182-191). Fleet-wide grid sweep explicitly deferred and logged in the component comment (:175-179) |
| UA7 — "Committed" runtime logs under docker topology | UA | **No longer applicable (false premise)** | `git check-ignore docker/central-node-a/logs` → ignored; `git ls-files docker docker-env2 \| grep -i logs/` → empty. Untracked local output; nothing to scrub |
## Maturity Verdict
This is a **mature, carefully engineered surface with a small number of sharp edges**. The security layer is genuinely good — fail-fast key validation, a single shared claim builder for login and refresh, a deliberate and well-documented idle/refresh model, DB-persisted Data Protection keys, curated-vs-generic error mapping that prevents internal-detail leaks, and a secured-writes implementation with a real CAS race guard, server-side no-self-approval, and a TOCTOU protocol re-check at execute time. The Blazor code shows unusual discipline for disposal, thread marshalling, and best-effort degradation. The problems that remain are not sloppiness but *asymmetries*: a production deployment artifact that ships with authentication disabled, a per-site artifact-deploy option that is silently ignored server-side, secret-projection hygiene applied to SMTP/SMS/API keys but not to data connections / external systems / DB connection strings, and a secured-write "Expired" state that exists in the UI and spec but nowhere in the code. Test depth is thin relative to the actor's ~150-command surface.
**Disposition counts:** Fixed (verified) **17** · Partially fixed **6** (S1, S4, P1, P2, UA5, UA6 — all residues consciously deferred and documented except the P1/UA5 hub gap, which is new finding N1) · No longer applicable **1** (UA7) · Not fixed / Regressed **0**.
---
## Findings — Dimension 1: Stability
## New Findings — Round 2
### [High] S1 — Bundle import via `/management` can 504 while the import keeps applying
`ManagementEndpoints.HandleRequest` Asks the actor with a configurable timeout that defaults to **30 s** (`ManagementEndpoints.cs:18`, `ResolveAskTimeout` at `:28-33`), and neither `docker/central-node-a/appsettings.Central.json` nor `deploy/wonder-app-vd03/appsettings.Central.json` sets `ScadaBridge:ManagementService:CommandTimeout`. The CLI's bundle commands use a **5-minute client timeout** (`CLI/Commands/BundleCommands.cs:16`) precisely because imports are slow — but the server-side Ask expires at 30 s, returns `504 TIMEOUT` (`ManagementEndpoints.cs:107-113`), **and the actor's `ProcessCommand` task keeps running** (PipeTo is not cancelled), so `HandleImportBundle` (`ManagementActor.cs:2949-3015`) continues applying the bundle after the caller was told it timed out. Failure scenario: an operator imports a large bundle, gets a 504, retries — under `--on-conflict rename` the second import produces a duplicate `-imported-<stamp>` artifact set; under `overwrite` it double-applies. The same ambiguity applies to fleet-wide `DeployArtifacts`. Fix: raise the configured timeout for transport/deploy commands (per-command timeout in the envelope), or make import idempotent on a client-supplied import id.
### A. Security & Auth
### [High] S2 — Pending secured writes never expire; the "Expired" status is fictional
The UI's terminal-status list and badge mapping include `"Expired"` (`CentralUI/Components/Pages/Operations/SecuredWrites.razor:265, 411`), and Component-CentralUI.md §Secured Writes lists "Executed / Failed / Rejected / Expired" — but **no code anywhere sets a row to Expired** (repo-wide grep: the only two hits are the UI file). `HandleApproveSecuredWrite` (`ManagementActor.cs:1099-1246`) will approve a `Pending` row of any age. Failure scenario on a live SCADA surface: an Operator submits a setpoint write, it sits unverified over a weekend while the process moves to a different state, and a Verifier approves the stale value days later — the confirmation dialog (`SecuredWrites.razor:342-347`) shows the value but not its age. A pending-write TTL (with a sweeper or an at-approve age check) is the obvious missing piece; the status enum already reserved the name.
#### [Medium] N1 — DebugStreamHub still binds LDAP with no throttle; Component-Security.md falsely claims complete coverage
`DebugStreamHub.OnConnectedAsync` extracts Basic-Auth credentials itself and calls `ILdapAuthService.AuthenticateAsync` directly (`DebugStreamHub.cs:118-124`) — it uses `ManagementAuthenticator` **only** for the DisableLogin bypass (`TryDisableLoginUser`, :82), not for the bind, so the `LoginThrottle` never sees hub connection attempts. Meanwhile `Component-Security.md:208` states the throttle scope includes "the debug-stream hub … No bind happens outside these paths, so throttling coverage is complete" — a doc statement that is simply untrue in code. A SignalR negotiate + connect per guess is clumsier than `POST /auth/token`, but it remains an unthrottled LDAP-bind oracle, and it also skips the `RecordSuccess`/`RecordFailure` bookkeeping that would let hub failures contribute to a shared lockout. Fix: route the hub's credential path through `ManagementAuthenticator.AuthenticateAsync` (it already returns roles + site scopes) or replicate the three throttle calls; correct the doc either way.
### [Medium] S3 — Poll timers can overlap their own refreshes
`Health.razor:521-528` and `AlarmSummary.razor:283-291` both use `System.Threading.Timer` with a fixed period whose callback does `InvokeAsync(async () => { await RefreshAsync(); ... })` with no reentrancy guard. `AlarmSummary.RefreshAsync` fans out per-instance snapshot Asks (site offline ⇒ each fetch waits its full Ask timeout); with a 15 s period and a >15 s degraded fetch, ticks stack up on the dispatcher and the page issues overlapping fan-outs against an already-struggling site. A `PeriodicTimer` loop or an `Interlocked` in-flight flag would bound this. (Disposal itself is handled correctly on both pages — `Health.razor:757-760`, `AlarmSummary.razor:382`.)
#### [Medium] N2 — LoginThrottle keys collapse behind Traefik: no ForwardedHeaders handling anywhere
Every throttle surface keys on `context.Connection.RemoteIpAddress` (`ManagementAuthenticator.cs:111`, `AuthEndpoints.cs:39, :126`), and no `UseForwardedHeaders`/`ForwardedHeadersOptions` exists in the Host, CentralUI, or ManagementService (repo grep: zero hits). The documented topology fronts both the UI and the management API with Traefik (`CLAUDE.md` CLI quick reference: management URL is the LB on :9000), so in production **every client shares Traefik's IP** and the key degenerates to `username|<proxy-ip>`. Two consequences: (a) the per-IP isolation the design intends ("`alice@10.0.0.2` is not locked out by `alice@10.0.0.1`'s failures" — `LoginThrottleTests` asserts exactly this) does not exist in the shipped topology; (b) any network-adjacent actor can lock out any username — five wrong passwords for `admin` via the LB locks `admin` out for all real operators for `LoginLockoutMinutes`, repeatable indefinitely: a cheap login-DoS against a SCADA control surface. Fix: honor `X-Forwarded-For` from the trusted proxy only (`ForwardedHeadersOptions.KnownProxies`), or document the trade-off and consider a success-path bypass (a correct password unlocks) to blunt the DoS.
### [Medium] S4 — 200 MB request bodies are buffered as strings, then multiplied
The management endpoint raises the body cap to **200 MB** for transport bundles (`ManagementEndpoints.cs:50, 58-62`) and then reads the whole body with `StreamReader.ReadToEndAsync()` into a string (`:77-81`); the actor then base64-decodes to a `byte[]` (`ManagementActor.cs:3105-3119`), wraps in a `MemoryStream`, and export goes the other way — `stream.CopyToAsync(ms); ms.ToArray(); Convert.ToBase64String(bytes)` (`ManagementActor.cs:2902-2905`), which is ~4 copies of the artifact resident simultaneously (bytes, MemoryStream buffer, base64 string, JSON envelope string). A concurrent pair of large imports on a central node is a realistic OOM/LOH-fragmentation path. Streaming multipart upload (as the audit export already streams downloads — `AuditEndpoints.cs:183+`) is the right long-term shape.
#### [Medium] N3 — Secured-write handlers ignore site scope, while the spec says scoping "is layered on at the LDAP-mapping level"
None of the four secured-write handlers calls `EnforceSiteScope`/`EnforceSiteScopeForIdentifier` (verified: zero occurrences in `ManagementActor.cs:1243-1510`): `HandleSubmitSecuredWrite` resolves the target site (:1247) but never checks `user.PermittedSiteIds`; approve/reject/list likewise. `Component-Security.md:141` explicitly says Operator/Verifier are "coarse global roles like the others; **any site scoping is layered on at the LDAP-mapping level**" — but a mapping row that grants Operator with `PermittedSiteIds=["3"]` still lets that principal submit (and a scoped Verifier approve) MxGateway **device writes** to every site. This is the same class of gap C2 just closed for `DeployArtifacts`, on a higher-consequence path. Fix: `EnforceSiteScopeForIdentifier(sp, user, cmd.SiteId)` at submit and `row.SiteId` at approve/reject (+ scope-filter the list), or amend the spec to state that secured-write roles are exempt from site scoping — code or doc, one must move.
### [Low] S5 — Fire-and-forget SignalR sends swallow failures silently
`DebugStreamHub.SubscribeInstance` pushes events with `_ = hubClients.Client(connectionId).SendAsync(...)` (`DebugStreamHub.cs:204-221`). Deliberate (documented) — but a persistently failing client produces zero telemetry. Elsewhere the hub is exemplary: transient-hub-instance pitfall avoided via `IHubContext` (`:198-200`), unsubscribe on disconnect (`:255-259`).
### B. Alarm Summary live path (`ISiteAlarmLiveCache` → Blazor circuit)
### Positive stability observations
- `DebugView.razor` is a model Blazor Server component: `_disposed` flag set before stream stop, `SafeInvokeAsync` guarding `ObjectDisposedException`, dictionary mutation marshalled onto the render thread with the race explicitly documented (`DebugView.razor:511-544, 589-611`).
- Failover transparency is real: Data Protection keys persist to the config DB (`ConfigurationDatabase/ServiceCollectionExtensions.cs:75-76`), the cookie principal is self-contained, and the ManagementActor runs on every central node (stateless, per spec).
- `AlarmSummaryService` implements the spec'd fan-out faithfully: `SemaphoreSlim(8)` cap, partial-results tolerance, caller-cancel propagation (`AlarmSummaryService.cs:26, 68-84, 112-123`).
The design is genuinely good (see "What's genuinely good"), but the page-side reconciliation has seams:
#### [Medium] N4 — `RefreshAsync` applies results without re-checking the selected site: cross-site data displayed after a site switch
`AlarmSummary.razor:283-297` captures `siteId` at entry, awaits the multi-second fan-out (`GetSiteAlarmsAsync`), then unconditionally assigns `_rows`/`_notReporting`/`_rollup`. A poll tick for site A in flight when the operator switches to site B completes **after** B's initial refresh and overwrites the page with A's alarms (and A's not-reporting list) labeled under B's picker — on an operator alarm page. The live path already has exactly the right guard (`OnLiveAlarmsChanged` drops callbacks when `_selectedSiteId != siteId`, :374); the poll path needs the same one line before assignment (`if (_selectedSiteId != siteId) return;`). Self-corrects within ≤15 s, but wrong-site alarm data for that window is not acceptable on this surface.
#### [Low] N5 — When live, a slower poll snapshot can regress fresher live state
The 15 s poll always rebuilds `_rows` from the fan-out even when the cache is live (`AlarmSummary.razor:293-297`); the comment claims the poll "simply re-affirms the same snapshot" (:344-347), but a poll whose fan-out started before a live delta lands **after** it and momentarily reverts the delta (an alarm flickers back to its prior state until the next delta/poll). Cheap fix: when `LiveAlarmCache.IsLive(siteId)`, let the poll update only `_notReporting` (its unique authority) and skip the row rebuild — this also removes redundant per-instance snapshot load while live.
#### [Low] N6 — `IsLive` is sticky: `HasPublished` never resets on aggregator death or stream loss
`SiteEntry.HasPublished` is set on first publish (`SiteAlarmLiveCacheService.cs:392-393`) and never cleared; `IsLive` (:132-136) therefore reports `true` forever, even if the aggregator actor later dies or the site goes dark and `entry.Current` freezes. Exposure is bounded (the poll keeps overwriting rows, and reconcile re-seeds a *living* aggregator), but `OnSiteChangedAsync` deliberately applies the live snapshot **on top of** the fresh poll when `IsLive` (`AlarmSummary.razor:277-280`) — with a dead aggregator that grafts an arbitrarily stale snapshot over correct data for up to 15 s. Consider clearing `HasPublished` (or publishing a degraded flag) when the actor terminates (a `Context.Watch`/deathwatch hook from the service) and having the interface expose staleness, not just "seeded once".
#### [Low] N7 — Live-callback `InvokeAsync` lacks the house disposal guard
`OnLiveAlarmsChanged` does `_ = InvokeAsync(...)` with no `_disposed` flag (`AlarmSummary.razor:367-382`); a callback racing `Dispose()` can fault the discarded task with `ObjectDisposedException` from `StateHasChanged` on a torn-down renderer. `DebugView.razor` established the pattern for exactly this (`SafeInvokeAsync` + `_disposed` set before teardown — cited in round 1 as exemplary); the new page should match it. `SiteAlarmLiveCacheService.OnPublish` catches and logs synchronous callback throws (:399-410) but cannot observe the async fault.
### C. Management / Security implementation details
#### [Low] N8 — `LoginThrottle.Prune` scans (and may sort) the whole map on every failed bind
`RecordFailure` → `Prune` enumerates all entries and, when over cap, runs `OrderBy` over the entire dictionary (`LoginThrottle.cs:121, :146-168`). During the exact scenario the throttle exists for — a spray — every failed request pays an O(n) scan (n up to 10,000) plus a possible full sort, on the request path. Harmless at this cap, but the pruning would be better amortized (e.g. every Nth write or on a timer). Counterpoint noted: the cap and opportunistic pruning are exactly what keeps memory bounded, and the constants are sane.
#### [Low] N9 — `ConfigSecretScrubber.MergeSentinels` matches array elements by index; fragment list is name-based
Sentinel restoration inside arrays pairs `incoming[i]` with `stored[i]` (`ConfigSecretScrubber.cs:200-221`): a client that reorders an array of credentialed objects while round-tripping sentinels grafts the *wrong* stored secret into each slot (silent misconfiguration, not disclosure). And detection is name-fragment-based (`password|secret|token|apikey|credential|passphrase`, :18-19) — current config types are fully covered (`OpcUaUserIdentityConfig.Password`/`CertificatePassword`, `MxGatewayEndpointConfig.ApiKey`), but a future field named e.g. `SigningKey` or `Pin` would silently leak; worth a lock-in test enumerating the DataConnections config types' string properties against the fragment list, mirroring the transport round-trip-guard idea.
---
## Findings — Dimension 2: Performance
## What's genuinely good
### [Medium] P1 — Every CLI/management/audit HTTP request performs a full LDAP bind
`ManagementAuthenticator.AuthenticateAsync` binds against LDAP and re-runs role mapping on **every** request (`ManagementAuthenticator.cs:107-118`); the audit endpoints and `DebugStreamHub.OnConnectedAsync` (`DebugStreamHub.cs:118-137`) do the same. This is the documented stateless design, but there is no negative-result cache, rate limit, or lockout — so (a) a scripted CLI loop hammers the directory once per command, and (b) `POST /management` and `POST /auth/token` (`AuthEndpoints.cs:99-148`) are unauthenticated LDAP-bind oracles usable for password spraying at whatever rate the LDAP server tolerates. A short-TTL bind cache or a per-IP failure throttle would address both.
### [Medium] P2 — Unbounded list handlers materialize whole tables
Several read handlers return `GetAll*` with no paging: `HandleListTemplates` (`ManagementActor.cs:507-511`), `HandleListInstances` (`:718-729`, post-filtered in memory for scoped users), `HandleQueryDeployments` unfiltered branch loads *all* deployment records *and all instances* (`:2031-2060` — the N+1 was fixed, but it is still two full-table loads per query), and `HandleExportBundle` loads every template/script/system/site/instance to resolve names (`:2818-2831`). Fine at current fleet size; a 10k-instance estate will feel it on every CLI `instance list`. The audit surface shows the house already knows the right pattern (keyset paging, `MaxPageSize` 1000 — `AuditEndpoints.cs:59-66`).
### [Medium] P3 — `ListSecuredWrites` hard-caps at 200 rows with no paging
`HandleListSecuredWrites` does `QueryAsync(status, siteId, skip: 0, take: 200)` (`ManagementActor.cs:1280-1286`) and the page loads *everything* then splits pending/history client-side (`SecuredWrites.razor:303-308`). Once >200 rows exist, history silently truncates — an **audit-adjacent** table silently losing visibility of older decisions. No pagination controls exist on the page.
### [Low] P4 — Alarm Summary re-sorts/filters per render
`FilteredRows().ToList()` runs inside the render body (`AlarmSummary.razor:170`) — recomputed on every `StateHasChanged`, including each 15 s tick. Cheap at hundreds of rows; worth memoizing if sites grow. No virtualization on any table in the domain, but page sizes are mostly bounded (Notification Report pages at 50 — `NotificationReport.razor:352`).
### Positive performance observations
- Audit export streams server-side in 1000-row pages and the CLI streams to disk with `ResponseHeadersRead` (`AuditEndpoints.cs:66`, `ManagementHttpClient.cs:209-210`).
- The actor never blocks its dispatcher: `HandleEnvelope` role-checks synchronously then `PipeTo`s the async work (`ManagementActor.cs:97-120`), so the single actor instance is not a mailbox bottleneck for I/O-bound commands — commands execute concurrently on the thread pool with per-command DI scopes (`:127-132`). This is the correct pattern and worth calling out as such.
- **The authorization table is now a frozen, mechanically-enforced artifact.** `GetRequiredRoles` is a single any-of switch over named static role sets with per-arm rationale comments (`ManagementActor.cs:163-294`), backed by the ~139-entry `RequiredRoleMatrixTests` table that fails on any unlisted command, plus `DispatchCoverageTests` proving every registered command reaches a real handler. A C2- or C4-class regression is now a CI failure, not an audit finding.
- **The secured-write lifecycle is defensively complete.** TTL checked before self-approval before CAS (`ManagementActor.cs:1296-1315`), CAS-idempotent multi-node expiry, expiry audited as `SecuredWriteExpire`, protocol re-checked at execute, list gated and paged, submission age surfaced in the approve dialog — and the docs restate both the TTL and the dual-role deployment hazard precisely (`Component-Security.md:143-145`).
- **The live alarm cache is a model piece of lifecycle engineering**: reference-counted per-site aggregators with linger-delayed stop and version-checked timers (`SiteAlarmLiveCacheService.cs:160-190`), fail-safe viewer cap returning a no-op handle on a render path (:100-107), race-checked start with self-healing bounded retry (:219-274), callbacks invoked outside the lock with per-viewer fault isolation (:397-410), immutable snapshot swaps, and validated options (`CommunicationOptionsValidator.cs:72-87`). The page's seed-then-stream reconciliation comments (:337-351) show real thought about ordering.
- **Secret hygiene is now symmetric**: the `*PublicShape` pattern covers SMTP, SMS, API keys, data connections, external systems, and DB connections, on responses *and* audit afterState, with sentinel-preserving round-trips — and the elision reuses one tested scrubber instead of six bespoke projections.
- **`DisableLoginGuard` is the right shape**: fail-fast at the composition root before any service wiring, environment-gated with an explicit double-ack escape hatch, and the error message tells the operator all three ways out (`DisableLoginGuard.cs:22-27`).
- **Consistency discipline held under a 739-line actor diff**: paging clamped through one shared helper applied after scope filters, curated `ManagementCommandException` messages everywhere, additive-only contract evolution on all six browse/cert records and the paging params, and docs updated in the same commits (verified for C4/C5/C6/S2/throttle sections).
---
## Findings — Dimension 3: Conventions & Correctness
## Prioritized Recommendations (new findings only)
### [Critical] C1 — Production deployment artifact ships with authentication disabled
`deploy/wonder-app-vd03/appsettings.Central.json:40` sets `"DisableLogin": true` (and `:38` `"RequireHttpsCookie": false`). Per `Component-Security.md` §Dev Disable-Login Flag there is **no environment guard** — only a startup log warning — and the bypass is honoured by *every* surface: the cookie scheme (`AutoLoginAuthenticationHandler`), `POST /management` (`ManagementAuthenticator.cs:52-76`), the audit REST endpoints, and the debug-stream hub (`DebugStreamHub.cs:82-91`). On this host, every anonymous HTTP request is authenticated as `multi-role` with **all roles system-wide — including Operator and Verifier**, which nullifies the two-person secured-write control on a box that can relay MxGateway device writes. `deploy/` is described by CLAUDE.md as "Production/on-host deployment artifacts". Even if wonder-app-vd03 is a staging box today, an unauthenticated SCADA control surface should not be one config-file copy away from production. Recommendation: add an environment guard (refuse `DisableLogin=true` unless `ASPNETCORE_ENVIRONMENT=Development` or an explicit `IUnderstandThisDisablesAuth` second key), and scrub the deploy artifact.
1. **[Medium] Throttle the debug hub's LDAP bind and fix the doc** — route `DebugStreamHub.OnConnectedAsync` through `ManagementAuthenticator` (or replicate the three throttle calls); correct `Component-Security.md:208`. (N1)
2. **[Medium] Decide the proxy-IP question for `LoginThrottle`** — trusted-proxy `ForwardedHeaders` (KnownProxies = Traefik) so keys are real client IPs, or document the username-lockout-DoS trade-off deliberately. (N2)
3. **[Medium] Enforce (or explicitly exempt) site scope on secured writes** — `EnforceSiteScopeForIdentifier` at submit/approve/reject + scoped list, or amend `Component-Security.md:141`; code and spec currently disagree. (N3)
4. **[Medium] One-line staleness guard in `AlarmSummary.RefreshAsync`** — re-check `_selectedSiteId == siteId` before assigning results, mirroring the live path's guard. (N4)
5. **[Low] Live-path polish** — poll updates only `_notReporting` while `IsLive` (N5); reset/expose liveness on aggregator death (N6); adopt the `DebugView` disposal-guard pattern for the live callback (N7).
6. **[Low] Hygiene** — amortize `LoginThrottle.Prune` (N8); add a lock-in test binding the scrubber's fragment list to the DataConnections config types' string properties (N9).
### [High] C2 — `deploy artifacts --site-id` is silently ignored; artifacts always deploy fleet-wide
`MgmtDeployArtifactsCommand` carries `int? SiteId` (`Commons/Messages/Management/DeploymentCommands.cs:3`) and both CLI commands send it (`CLI/Commands/SiteCommands.cs:209`, `CLI/Commands/DeployCommands.cs:51`; documented in `Component-CLI.md` and the CLI README as `[--site-id <id>]`). But `HandleDeployArtifacts` never reads `cmd.SiteId` — it calls `svc.DeployToAllSitesAsync(user)` unconditionally (`ManagementActor.cs:2006-2013`), and `ArtifactDeploymentService.DeployToAllSitesAsync(string user, CancellationToken)` (`ArtifactDeploymentService.cs:227-229`) has no site parameter. Two consequences: (1) **contract violation** — an operator targeting one site pushes shared scripts/external systems/DB connections to *every* site; (2) **authorization gap** — the command also has no `EnforceSiteScope` call, so a *site-scoped* Deployer can trigger a fleet-wide artifact deployment. The Central UI's per-site "Deploy Artifacts" button presumably uses a different path; the management surface needs `DeployToSiteAsync(siteId)` plus a scope check.
## Severity tally — NEW findings (round 2)
### [High] C3 — Secret projection is inconsistent: connection credentials leak to any authenticated user and into the audit log
The actor is scrupulous about SMTP credentials, Twilio auth tokens, and API-key hashes — all projected to credential-free shapes before response *and* audit (`SmtpConfigPublicShape` `ManagementActor.cs:1787-1801`, `SmsConfigPublicShape` `:1841-1853`, `HandleListApiKeys` `:1955-1967`). But three sibling entity families get no such treatment:
- **Data connections** — `HandleListDataConnections`/`HandleGetDataConnection` (`:1401-1416`) return the raw entity, whose `PrimaryConfiguration`/`BackupConfiguration` JSON embeds `OpcUaUserIdentityConfig.Password` and `CertificatePassword` in plaintext (`Commons/Types/DataConnections/OpcUaUserIdentityConfig.cs:13,17`). List/Get are **read-only commands with no role gate** (`GetRequiredRole` default `:240-241`), so a Viewer- or Operator-only principal can read every site's OPC UA credentials via one CLI call. Create/Update also audit the full entity as afterState (`:1429, :1445`), persisting the passwords into the audit log readable by `OperationalAudit` roles.
- **External systems** — `AuthConfiguration` (API keys / Basic credentials) returned raw to any authenticated user and audited verbatim (`:1587-1624`).
- **Database connections** — full `ConnectionString` (SQL credentials) returned raw in List/Get (`:2526-2536`); the audit shape here *was* trimmed to `{Id, Name}` (`:2544, :2557`), which shows the team knows the pattern — it just wasn't applied to the response.
Recommendation: apply the established `*PublicShape` pattern (config-with-secrets-elided + `HasCredentials` flag) to all three, and gate Get-with-secrets behind Admin if the UI editor needs round-tripping.
### [Medium] C4 — Spec drift: browse/verify/cert-trust commands are documented as ManagementActor commands but only exist UI-side
`Component-ManagementService.md` §Remote Queries lists `BrowseNodeCommand`, `SearchAddressSpaceCommand`, `VerifyEndpointCommand`, `TrustServerCertCommand`/`RemoveServerCertCommand`/`ListServerCertsCommand` with Design/Admin role gates. The actor's `DispatchCommand` has **no handlers for any of them** — they fall through to `NotSupportedException` (`ManagementActor.cs:423`), and since `ManagementCommandRegistry` auto-registers every `*Command` in the namespace by reflection (`ManagementCommandRegistry.cs:66-79`), a CLI `POST /management {"command":"TrustServerCert"}` deserializes fine and then returns a generic `COMMAND_FAILED` internal error. The real implementations live in Central UI services (`BrowseService.cs`, `EndpointVerificationService.cs`, `CertManagementService.cs`) with hand-rolled `HasClaim` role guards (`BrowseService.cs:48-55, 103-110`). Net effect: the CLI cannot browse address spaces or manage cert trust (a gap for the automation story the CLI exists for), and the component doc misdescribes the enforcement point. Fix the doc or add actor handlers.
### [Medium] C5 — Role-name comparison asymmetry between UI policies and the actor (acknowledged, still live)
ASP.NET `RequireClaim` values are compared **ordinal case-sensitive** (`AuthorizationPolicies.cs:143-169`), while the ManagementActor compares roles **OrdinalIgnoreCase** (`ManagementActor.cs:105, 440` etc.), and `ValidateMappingRole` deliberately preserves the stored row's verbatim casing while validating membership case-insensitively — the comment at `ManagementActor.cs:1904-1911` explicitly defers the asymmetry. Failure scenario: an admin creates a mapping row `deployer` (lowercase) via CLI; the affected users pass every ManagementActor gate but fail every `RequireDeployment` UI policy — a confusing split-brain where CLI works and the UI 403s. Since the write path already validates against `Roles.All`, the cheap fix is to **canonicalize the stored casing** at that single write path.
### [Medium] C6 — Area-management role gate contradicts both design docs
`CreateAreaCommand`/`UpdateAreaCommand`/`DeleteAreaCommand` are gated **Designer** (`ManagementActor.cs:187, 207`), but `Component-ManagementService.md` §Authorization ("Admin role required for: site management, area management…") and `Component-Security.md` §Administrator ("Manage area definitions per site") both assign areas to **Admin**, and Component-CentralUI.md titles the workflow "Area Management (Admin Role)". Either the code or three documents are wrong; per the repo's own propagate-together rule this should be reconciled in one change.
### [Low] C7 — `HandleQueryEventLogs` passes `InstanceName` into a positional slot commented "InstanceId filter"
`ManagementActor.cs:2757-2766` builds `EventLogQueryRequest(..., cmd.InstanceName, /* InstanceId filter */ ...)`. If the site-side filter expects an id, name-based CLI filtering silently matches nothing; if it expects a name, the comment/param name is wrong. Worth a five-minute verification against `SiteEventLogging`.
### [Low] C8 — Stale CLAUDE.md note on SecuredWrite `SourceNode`
CLAUDE.md says "SecuredWrite audit rows currently leave `SourceNode` NULL — a logged follow-up", but `EmitSecuredWriteAuditAsync` now routes through `ICentralAuditWriter`, which stamps `SourceNode` from `INodeIdentityProvider` (documented at `ManagementActor.cs:999-1013`). The follow-up appears done; the project memory should be updated so future sessions don't re-fix it.
### Positive convention observations
- **No third-party Blazor component libraries** — the CentralUI csproj references only Roslyn packages and the in-family `ZB.MOM.WW.Theme` (`ZB.MOM.WW.ScadaBridge.CentralUI.csproj:19-21`); tables, tree views, dialogs, toasts, and the KPI chart are all custom, per the repo rule.
- Options pattern honoured throughout; component libraries take no `IConfiguration` (`Security/ServiceCollectionExtensions.cs:26-37`); `SecurityOptionsValidator` fail-fast with a genuinely well-reasoned invariant (refresh threshold < idle window, `SecurityOptions.cs:118-142`).
- Error-surface hygiene is exemplary: `MapFault` returns curated `ManagementCommandException` messages verbatim and collapses everything else to a correlation-id-bearing generic message (`ManagementActor.cs:137-158`) — no raw exception text reaches the CLI/UI for unanticipated faults.
- The secured-write approve handler is defensive in exactly the right places: no-self-approval **before** the CAS (`:1113-1115`), CAS via `TryMarkApprovedAsync` (`:1121`), MxGateway protocol **re-checked at execute** against submit-time reconfiguration (`:1138-1165`), value-type and decode failures contained so a row is never stuck `Approved` (`:1171-1202`), transport exceptions contained (`:1206-1226`).
- `ResolveRolesCommand` deliberately not dispatched with the security rationale documented (`:418-422`) — good trust-boundary thinking.
- JWT implementation pins issuer/audience, zero clock skew, disables in/outbound claim mapping symmetrically, enforces ≥256-bit keys at construction (`JwtTokenService.cs:54-78, 124-171`); the cookie path's login and mid-session refresh share `SessionClaimBuilder` so claim shapes cannot drift (`CookieSessionValidator.cs:200-210`, `AuthEndpoints.cs:84-89`); refresh failures fail-open to *current* roles and only idle-timeout rejects (`CookieSessionValidator.cs:132-152`) — matching the documented policy exactly.
- LDAP is delegated to the shared hardened `ZB.MOM.WW.Auth.Ldap` library (bind-then-search, escaping, fail-closed — per `Security/ServiceCollectionExtensions.cs:49-59`); no bespoke LDAP string assembly exists in this repo, so injection surface is centralized in the audited library.
---
## Findings — Dimension 4: Underdeveloped Areas
1. **Secured-write lifecycle** — no expiry (S2), no paging (P3), history visible to any authenticated user (`ListSecuredWritesCommand` falls through to the no-role gate, `ManagementActor.cs:238-241`) — tag values on the history table may themselves be process-sensitive. Also nothing prevents the *same person* holding both Operator and Verifier roles via two LDAP groups; the control is only "not the same row's submitter", which is the documented design but worth restating as a deployment-configuration hazard.
2. **ManagementService test depth** — 11 test files (`tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/`) against a 3,157-line actor dispatching ~150 commands. `SecuredWriteHandlerTests` exists (good — covers the flagship control), but there is no visible exhaustive matrix test asserting `GetRequiredRole` for every command type — precisely the test that would have caught C2's missing scope check and would freeze the authz table against regressions. Security.Tests is 9 files; CookieSessionValidator is covered.
3. **CLI.Tests is not in the slnx** (`ZB.MOM.WW.ScadaBridge.slnx:24` includes only the CLI project) — a documented gotcha, but it remains a standing risk that a green solution run hides never-compiled CLI tests.
4. **Deferred follow-ups visible in code/docs**: aggregated live alarm stream for Alarm Summary (poll-only today — Component-CentralUI.md §Alarm Summary "logged as a follow-up"); central persistence of cert trust; live LDAP group re-query mid-session (Component-Security.md "Residual limitation"); Parquet export returns 501 (`AuditEndpoints.cs:202-210`); `audit verify-chain` is a no-op pending the hash chain. All are honestly documented — the gap is that no single tracking list ties them together.
5. **Login endpoints lack throttling** — `/auth/login`, `/auth/token`, `/management`, and the debug hub all do per-request LDAP binds with no failure throttle or lockout (P1). `/auth/login` also reflects the LDAP failure kind into the redirect query string (`AuthEndpoints.cs:41-45`); confirm `LdapAuthFailureMessages` does not distinguish "unknown user" from "bad password" (user enumeration).
6. **Accessibility/UX debt** — sortable headers are `<th role="button" @onclick>` without keyboard handlers or `aria-sort` (`AlarmSummary.razor:176-179`); status conveyed by color+text badges is fine, but the custom tables have no `scope`/caption markup. Minor for an internal tool, but it is uniform debt across every custom grid.
7. **`docker/central-node-a/logs/` committed runtime logs** sit inside the deploy topology folder (noticed while scanning configs) — noise that will eventually leak something.
---
## Prioritized Recommendations
1. **[Critical] Guard `DisableLogin` and scrub `deploy/wonder-app-vd03`** — refuse the flag outside a Development environment (or require a second explicit ack key); remove `DisableLogin: true` from the on-host deploy artifact. (C1)
2. **[High] Implement per-site artifact deployment or reject the option** — honour `MgmtDeployArtifactsCommand.SiteId` (add `DeployToSiteAsync`) and add `EnforceSiteScope`; until then, make the handler *fail* when SiteId is supplied rather than silently deploying fleet-wide. (C2)
3. **[High] Apply the `*PublicShape` secret-elision pattern to DataConnection, ExternalSystem, and DatabaseConnection reads and audits**; consider role-gating those List/Get commands. (C3)
4. **[High] Add a pending-secured-write TTL** that transitions stale rows to the already-reserved `Expired` status, plus paging on the list. (S2, P3)
5. **[High] Fix the long-command timeout mismatch** — per-command Ask timeouts (transport/deploy get minutes), configure `CommandTimeout` in shipped configs, and/or make bundle import idempotent on an import id so a 504-then-retry is safe. (S1)
6. **[Medium] Add a `GetRequiredRole` matrix test** enumerating every registered command and asserting its gate — freezes the authorization table and catches future C2-style gaps mechanically.
7. **[Medium] Canonicalize role casing at the mapping write path** to close the UI-vs-actor case-sensitivity split. (C5)
8. **[Medium] Reconcile the area-management role gate** (code says Designer, three docs say Admin) and the browse/cert-command placement (docs say actor, code says UI-only) in one doc+code pass. (C4, C6)
9. **[Medium] Add LDAP-bind failure throttling** (per-IP or per-username backoff) in `ManagementAuthenticator` and the auth endpoints. (P1)
10. **[Low] Housekeeping** — reentrancy guards on the two poll timers (S3), update the stale SecuredWrite `SourceNode` memory note (C8), verify the `InstanceName`/`InstanceId` event-log filter slot (C7), pull `docker/*/logs` out of the tree, and put CLI.Tests into the slnx or CI explicitly.
| Severity | Count | Findings |
|---|---|---|
| Critical | 0 | — |
| High | 0 | — |
| Medium | 4 | N1 (hub bind unthrottled + doc drift), N2 (throttle keying behind Traefik), N3 (secured-write site scope), N4 (Alarm Summary stale-site race) |
| Low | 5 | N5 (poll-over-live regression), N6 (sticky IsLive), N7 (disposal guard), N8 (Prune cost), N9 (scrubber merge/fragment coverage) |
archreview/08-conventions-tests-underdeveloped.md
+165 -163
View File
@@ -1,185 +1,187 @@
# Architecture Review 08 — Cross-Cutting Conventions, Test Posture, Underdeveloped Areas
# Architecture Review 08 — Cross-Cutting Conventions, Test Posture, Underdeveloped Areas (Round 2)
Reviewer domain: cross-cutting conventions, code organization, test posture, repo-wide underdeveloped-areas sweep, docs-code drift. Individual subsystems are covered in depth by the other seven reviews; this one covers the seams and the whole.
**Date:** 2026-07-12 (round 2; round 1 dated 2026-07-08, baseline commit `b910f5eb`)
**Repo state reviewed:** `main` @ `8c888f13` — 268 commits past the round-1 baseline (all eight archreview fix plans, PLAN-01…PLAN-08, plus the two post-initiative features: aggregated live alarm stream and KPI hourly rollups, both merged 2026-07-10).
Reviewer domain: cross-cutting conventions, code organization, test posture, docs-code drift, deferred-work inventory. Individual subsystems are covered in depth by the other seven reviews; this one covers the seams and the whole.
## Scope
- `src/` — all 26 projects: Commons structure, project reference graph, options pattern, POCO/repository conventions, UTC and correlation-ID discipline, message-contract evolution.
- `tests/` — all 30 test projects: breadth, quality sampling, integration/performance posture, slnx membership.
- Repo-wide grep sweep for TODO / FIXME / HACK / NotImplementedException / "deferred" / "follow-up" / "not shipped", plus deferred-work inventory from `docs/plans/` and `CLAUDE.md`.
- Docs-code sync spot checks: README component table, `docs/requirements/Component-*.md` (KpiHistory, StoreAndForward, Transport, ClusterInfrastructure), `docs/components/` reference docs.
- Disposition audit: every round-1 finding re-verified against current source with file:line evidence (plan claims not trusted — checked directly).
- Fresh convention sweep over `src/` (26 projects) and `tests/` (30 projects): UTC discipline, TODO/HACK/NotImplementedException surface, Commons purity, options-validation coverage, correlation IDs, csproj hygiene, message-contract enforcement.
- Drift audit of the 268 post-baseline commits: new options classes vs validators, new wire contracts vs contract locks, CHANGELOG/README/CLAUDE.md claims vs code, register accuracy.
- Deferred-work register (`docs/plans/2026-07-08-deferred-work-register.md`, created by PLAN-08 T11) — item-by-item accuracy check, including the two register rows that shipped post-initiative (#10 live alarm stream, #22 KPI rollups).
Method: `.slnx` and `.csproj` graph extraction, directory-structure listing, targeted greps over `src/`, `tests/`, `docs/`, and direct reads of the files behind every claim below. Read-only except this report.
Method: targeted greps and direct file reads only (static review; no builds/test runs — suites verified green 2026-07-10 per the master tracker). Read-only except this report.
## Maturity Verdict
## Verdict
This is an unusually disciplined codebase for its size (~190k src LOC, ~156k test LOC, ~5,000 test cases across 30 projects). The documented conventions are not aspirational — they are actually followed: Commons is genuinely persistence-ignorant, the Types/Interfaces/Entities/Messages hierarchy is real, UTC discipline is total (zero `DateTime.Now` in src), correlation IDs are on every cross-cluster request/response plus the `ManagementEnvelope`, and the project reference graph has no site→central contamination. The TODO surface is astonishingly small (two real TODOs in all of src) and almost every deferral is deliberately documented at the deferral site with a pointer to the design doc. The gaps that exist are narrow and specific: one genuine spec-vs-code contradiction (Transport claims Area membership travels in bundles; the exporter always emits `null`), the CLI.Tests slnx exclusion that keeps 279 tests out of every solution-level test run, options-validation coverage on only ~6 of ~20 option classes, and a thin PerformanceTests project that is a correctness-at-small-scale suite rather than a performance suite. Deferred-work inventory is sizeable (~20 tracked items) but almost entirely intentional and logged; only two or three items carry real operational risk.
**Round 1 vs round 2 in one line:** every actionable round-1 finding was genuinely fixed (17 fixed, 3 partially, 6 consciously deferred with register rows, 0 regressed) and the conventions that were already clean stayed clean through 268 commits of fix-plan churn; the new residue is small and mostly hygiene — one real embarrassment (a live production API key sitting in an untracked `test.txt` at the repo root), a failover-perf placeholder whose stated precondition has already been satisfied, and an abandoned CHANGELOG that is now factually wrong.
---
## 1. Conventions & Organization
## 1. Round-1 Finding Disposition
### 1.1 Commons hierarchy — followed, with undocumented extensions [Low]
Every round-1 finding, verified in current source. "Fixed (verified)" means the fix was confirmed at the cited location, not taken from plan claims.
`src/ZB.MOM.WW.ScadaBridge.Commons/` follows the documented `Types/`, `Interfaces/`, `Entities/`, `Messages/` layout with domain-area subfolders exactly as `Component-Commons.md` describes (284 .cs files; e.g. `Entities/{Audit,Deployment,ExternalSystems,InboundApi,Instances,Kpi,Notifications,Schemas,Scripts,SecuredWrites,Security,Sites,Templates}`, `Messages/{Artifacts,Audit,Communication,DataConnection,DebugView,Deployment,Health,InboundApi,Instance,Integration,Lifecycle,Management,Notification,RemoteQuery,ScriptExecution,Streaming}`).
Three top-level folders exist beyond the documented four: `Observability/`, `Serialization/` (protocol endpoint-config serializers), and `Validators/`. These are reasonable homes, but the convention doc lists only the four canonical folders. Either document them in `Component-Commons.md` or fold them in (`Serialization` arguably belongs under `Types/`).
### 1.2 POCO persistence-ignorance — verified clean [Pass]
Grep for EF Core / DataAnnotations attributes across Commons returns only one hit, and it is an XML-doc `<see cref>` to `DbUpdateConcurrencyException` in `src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IDeploymentManagerRepository.cs:64` — a documentation reference, not a dependency. The Commons csproj has zero EF package references (its only PackageReference is the shared `ZB.MOM.WW.Audit` lib). Entities are genuinely persistence-ignorant.
### 1.3 Repository interfaces vs implementations — 1:1, with a deliberate site-side exception [Low]
14 repository interfaces in `Commons/Interfaces/Repositories/`, 14 implementation files in `ConfigurationDatabase/Repositories/` — exact match. Two additional implementations live in `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/` (`SiteExternalSystemRepository`, `SiteNotificationRepository`): read-only SQLite-backed site-side implementations that throw `NotSupportedException` on writes. This is a sensible pattern (same interface, site-local store), but it is an undocumented exception to the "implementations in Configuration Database" rule.
**Vestigial seam** [Low/Medium]: `SiteNotificationRepository` (registered scoped in `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs:65`) reads the site-local `notification_lists` table — but per the central-only notification design, `SiteStorageService.PurgeCentralOnlyNotificationConfigAsync` (`src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs:703`) deliberately empties that table on every artifact apply. So the registered repository can only ever return empty results, and `StoreNotificationListAsync` (line 721) still exists as a write path into a table the design says must stay empty. The purge is well-commented and correct as a security cleanup; the live read/write surface around an intentionally-empty table is dead weight and a trap for a future maintainer. Recommend removing `StoreNotificationListAsync`/list-read paths or marking them `[Obsolete]` with the design rationale.
### 1.4 Project reference graph — sane; no site→central contamination [Pass, one Low note]
Extracted from all 26 csproj files:
- **Commons is a leaf** (no project refs). Everything depends on it; it depends on nothing.
- **Site-side projects never reference central-only projects.** `SiteRuntime` → Commons, Communication, DataConnectionLayer, ScriptAnalysis, HealthMonitoring, SiteEventLogging, StoreAndForward — critically, **no** reference to `ConfigurationDatabase` (EF/central SQL), `ManagementService`, `CentralUI`, or `TemplateEngine`. Same for DataConnectionLayer, StoreAndForward, SiteEventLogging.
- **Central-only stack is correctly layered**: AuditLog/SiteCallAudit/Transport → ConfigurationDatabase; CentralUI → the central components + ManagementService; Host references everything (composition root, as designed).
- **CLI → Commons only** — correctly thin (HTTP client to the Management API).
- **DelmiaNotifier → nothing** — correctly standalone/BCL-only.
One smell [Low]: `Communication` → `HealthMonitoring` (`src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs:13`). A transport layer depending on a monitoring component is a mild layering inversion; the health-aggregator abstraction it consumes could live in Commons. Cosmetic [Low]: `ManagementService.csproj` uses a Windows-style `..\` separator for its InboundAPI reference while every other reference in the repo uses `../` — harmless to MSBuild, inconsistent style.
### 1.5 Options pattern — ownership right, validation coverage partial [Medium]
Every component owns its options class in its own project (31 `*Options.cs` files across src; none in Commons) — the ownership convention is fully honored. But **eager validation is wired for only ~6 components**: `IValidateOptions<>`/`ValidateDataAnnotations`/`ValidateOnStart` appear only in AuditLog, KpiHistory, NotificationOutbox (+SmsOptions), HealthMonitoring, Security, ClusterInfrastructure, plus the Host `StartupValidator`. Components binding options without a validator include Communication, DataConnectionLayer, DeploymentManager, ExternalSystemGateway, InboundAPI, ManagementService, NotificationService, SiteCallAudit, SiteEventLogging, SiteRuntime, StoreAndForward, and Transport (e.g. `src/ZB.MOM.WW.ScadaBridge.Transport/ServiceCollectionExtensions.cs:25` — `AddOptions<TransportOptions>().BindConfiguration(...)` with no `.ValidateOnStart()`). A malformed `appsettings` section in those components surfaces at first use rather than at startup, which is exactly what the Host's readiness-gating philosophy is supposed to prevent. Recommend a sweep adding validators (the HealthMonitoring/ClusterInfrastructure `IValidateOptions` implementations are ready-made patterns to copy).
### 1.6 UTC timestamps — total compliance [Pass]
Zero occurrences of `DateTime.Now`/`DateTimeOffset.Now` (non-Utc) in all of src (grep across .cs and .razor). Every timestamp uses `UtcNow` or stores `datetime2`/ISO-8601 UTC.
### 1.7 Correlation IDs — convention followed [Pass]
96 `CorrelationId` declarations across `Commons/Messages/` cover the full cross-cluster request/response surface (RouteToInstance, Get/SetAttribute(s), Subscribe*, WriteTag*, ParkedMessage*, NotificationOutboxQueries, SiteCallQueries, EventLogQuery, DeploymentStateQuery, WaitForAttribute, etc.). Management commands themselves carry no per-command field, but they always travel inside `ManagementEnvelope(User, Command, CorrelationId)` with `ManagementSuccess`/`ManagementError`/`ManagementUnauthorized` echoing it (`Commons/Messages/Management/ManagementEnvelope.cs:7-11`) — the convention holds at the envelope level.
### 1.8 Message contract evolution — convention by discipline, not enforcement [Low]
Messages are plain C# records with defaulted trailing parameters (additive-friendly, e.g. `CreateDataConnectionCommand(..., string? BackupConfiguration = null, int FailoverRetryCount = 3)` in `Commons/Messages/Management/DataConnectionCommands.cs`). The Akka HOCON in `Host/Actors/AkkaHostedService.cs:233` configures no custom serializer, so ClusterClient traffic rides Akka.NET's default JSON serialization — tolerant of additive fields. The gRPC side does have contract-lock tests (`tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Protos/CachedTelemetryProtoTests.cs`, `PullAuditEventsProtoTests.cs`), but there is no equivalent guard (snapshot/round-trip tests against old shapes) for the ClusterClient record contracts. Given the "system-wide artifact version skew across sites is supported" design decision, a small contract-compatibility test suite for the highest-traffic ClusterClient messages would convert the additive-only rule from convention to enforcement.
### 1.9 Naming/namespace consistency [Pass]
Consistent `ZB.MOM.WW.ScadaBridge.<Component>` namespaces mirroring folder structure in every file sampled (SiteRuntime, Transport, Commons, KpiHistory, ManagementService). Test projects mirror src names 1:1 plus IntegrationTests/PerformanceTests/PlaywrightTests/Transport.IntegrationTests.
---
## 2. Test Posture & Quality
### 2.1 Breadth — every component has a test project; volume is high
30 test projects; ~5,050 `[Fact]`/`[Theory]` methods; ~156k test LOC against ~190k src LOC (src figure inflated by ~75k in ConfigurationDatabase, mostly EF migrations). Per-project counts (files / LOC / test methods):
| Project | Tests | Notes |
|---|---|---|
| CentralUI.Tests | 829 | bUnit page/component tests — largest suite |
| Commons.Tests | 465 | includes KpiSeriesBucketer, codecs, validators |
| SiteRuntime.Tests | 457 | Akka TestKit behavioral (see 2.2) |
| TemplateEngine.Tests | 433 | |
| CLI.Tests | 279 | **excluded from slnx** (see 2.4) |
| AuditLog.Tests | 237 | |
| ManagementService.Tests | 234 | |
| ConfigurationDatabase.Tests | 219 | incl. migration tests |
| InboundAPI.Tests | 218 | |
| Communication.Tests | 211 | incl. proto contract locks |
| DataConnectionLayer.Tests | 190 | |
| Host.Tests | 142 | incl. CompositionRootTests |
| Security.Tests | 133 | |
| NotificationOutbox.Tests / Transport.Tests | 122 / 124 | |
| StoreAndForward.Tests | 116 | |
| DeploymentManager.Tests | 98 | |
| HealthMonitoring.Tests | 84 | |
| IntegrationTests / ExternalSystemGateway.Tests | 69 / 69 | |
| Transport.IntegrationTests | 60 | |
| SiteEventLogging.Tests | 54 | |
| ScriptAnalysis.Tests / NotificationService.Tests / CentralUI.PlaywrightTests | 40 / 36 / 36 | |
| SiteCallAudit.Tests | 31 | thinnest per src LOC (1,642 loc / 31 tests) |
| DelmiaNotifier.Tests | 23 | |
| ClusterInfrastructure.Tests | 15 | src is only 191 LOC (wiring lives in Host) — proportionate |
| KpiHistory.Tests | 14 | bucketer/query/chart tests live with owners (Commons.Tests, CentralUI.Tests) — coverage is distributed, not missing |
| PerformanceTests | 10 | see 2.3 |
Relatively thin spots: **SiteCallAudit.Tests** (31 tests for a reconciliation/KPI/relay component) and **DeploymentManager.Tests** (98 tests for the deployment pipeline, though much of its logic is exercised via ManagementService/Host/Integration suites).
### 2.2 Quality sampling — real behavioral tests, not shallow
`tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/AlarmActorTests.cs` is representative: full Akka TestKit harness, real compiled script library, `TestProbe` as the parent InstanceActor, asserts actual `AlarmStateChanged` messages with state/name payloads — genuine behavior, not constructor smoke tests. The SiteRuntime.Tests directory also carries targeted regression suites (`InstanceActorChildAttributeRaceTests`, `DeploymentManagerRedeployTests`, `DeploymentManagerMediumFindingsTests` — named for review findings, indicating review feedback gets locked in as tests). IntegrationTests covers real cross-cutting concerns: `CentralFailoverTests`, `SiteFailoverTests`, `DualNodeRecoveryTests`, `RecoveryDrillTests`, `ReadinessTests`, `AuthFlowTests`, `SecurityHardeningTests`, `AuditTransactionTests`, `NotificationOutboxFlowTests`, gRPC tests, plus a `ScadaBridgeWebApplicationFactory`. A 36-test Playwright E2E suite exists for the Central UI.
### 2.3 PerformanceTests is real but minimal — a misnomer more than a stub [Medium]
`tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/` has 3 test files / 10 tests: `HealthAggregationTests` (10 sites × 100 updates, **correctness assertions only, no timing**), `StaggeredStartupTests` (one `Stopwatch` assertion < 1000ms), and `AuditLog/HotPathLatencyTests` (the best of the three: `Stopwatch.GetTimestamp` p95-under-threshold on the site audit hot path; its own doc comment acknowledges "no BenchmarkDotNet" as a deliberate choice). Nothing here exercises the load-bearing performance claims of the design — 25s failover, gRPC streaming throughput, S&F drain rate, per-subscriber stream backpressure. Either rename the project to reflect its actual scope or grow it toward the design's stated performance envelope (failover time and stream throughput being the two highest-value additions).
### 2.4 CLI.Tests still excluded from the slnx [High]
Verified against `ZB.MOM.WW.ScadaBridge.slnx`: 29 of 30 test projects are members; `tests/ZB.MOM.WW.ScadaBridge.CLI.Tests` is absent (the src `CLI.csproj` *is* in the slnx). This is a documented gotcha in CLAUDE.md and memory, but the durable fix — adding one `<Project>` line — has not been made, so `dotnet test ZB.MOM.WW.ScadaBridge.slnx` silently skips 279 tests and a green solution run can mask CLI tests that never compiled. There is no evident reason to keep it out; add it.
---
## 3. Underdeveloped Areas & Deferred-Work Inventory
The in-code TODO surface is remarkably small: exactly **two real TODOs** in all of src (both the same issue — Transport area export, `EntitySerializer.cs:251/518`), zero `NotImplementedException` in production code paths (the one that existed was deliberately deleted — see the comment in `ClusterInfrastructure/ServiceCollectionExtensions.cs:34-47`), and zero FIXME/HACK. 39 "deferred/follow-up" comment mentions are almost all self-documenting pointers to logged decisions.
Consolidated inventory (item, where noted, risk of leaving it):
| # | Item | Where noted | Risk if left |
| Round-1 finding | Sev | Disposition | Evidence (current source) |
|---|---|---|---|
| 1 | **Transport: Area membership doesn't travel in bundles** — exporter hardcodes `AreaName: null`; importer machinery is fully built (`BundleImporter.cs:3104-3168` resolves/creates areas) | `src/.../Transport/Serialization/EntitySerializer.cs:251,518` (TODO); contradicts `Component-Transport.md:21` + CLAUDE.md #24 | **High** — spec and CLAUDE.md claim it ships; cross-env imports silently drop area organization; half-built feature invites confusion |
| 2 | CLI.Tests not in slnx | `ZB.MOM.WW.ScadaBridge.slnx`; CLAUDE.md gotcha | **High** — 279 tests invisible to solution-level CI/test runs |
| 3 | AuditLog reconciliation dials site NodeA only; NodeB failover endpoint selection deferred | `src/.../AuditLog/Central/SiteEnumerator.cs:32,64`, `ISiteEnumerator.cs:16` | **Medium** — during a site NodeA outage, telemetry-loss reconciliation (the safety net) is also unavailable; audit gaps persist until NodeA returns |
| 4 | Options validation missing on ~14 components (§1.5) | e.g. `Transport/ServiceCollectionExtensions.cs:25` | **Medium** — config typos surface at first use, not startup |
| 5 | `SmsConfiguration.MaxRetryCount` stored but not honored at dispatch (SMTP retry policy governs SMS) | `Commons/Entities/Notifications/SmsConfiguration.cs:36` | **Medium** — admin-visible knob that does nothing; misleading UI |
| 6 | Role-check case-sensitivity asymmetry: UI `RequireClaim` policies vs ManagementActor's case-insensitive check | `ManagementService/ManagementActor.cs:1910` | **Medium** — a role stored with non-canonical casing could pass one gate and fail the other; deliberately unaltered, separately deferred |
| 7 | SecuredWrite audit rows leave `SourceNode` NULL | CLAUDE.md (Security section, logged follow-up) | **Low/Medium** — per-node audit forensics blind spot for two-person writes specifically |
| 8 | Hash-chain tamper evidence (T1) | CLAUDE.md; `docs/plans/2026-05-20-audit-log-code-roadmap.md:12`; CLI `verify-chain` is a shipped no-op stub (`CLI/Commands/AuditVerifyChainHelpers.cs:7`) | **Low/Medium** — deferred to v1.x by locked decision; append-only DB roles are the current control; the no-op CLI verb slightly overstates capability |
| 9 | Parquet audit archival/export (T2) — endpoint returns 501 | `ManagementService/AuditEndpoints.cs:204-208`; CLAUDE.md | **Low** — deferred v1.x; 501 + CLI messaging handle it honestly |
| 10 | Aggregated **live** alarm stream for Alarm Summary (currently snapshot fan-out + poll) | `docs/plans/2026-06-18-m7-opcua-mxgateway-ux-design.md:252` | **Low/Medium** — operator page scales with instance count; fan-out Ask per refresh is the known cost |
| 11 | Central-persisted, auditable OPC UA server-cert trust (v1 is site-local PKI only) | m7 design follow-ups; CLAUDE.md | **Low** — broadcast-to-both-nodes covers HA; no central governance/audit of trust decisions |
| 12 | Native-alarm-source-override CSV bulk import | m7 design follow-ups; CLAUDE.md (M7 UX) | **Low** — attribute-override CSV shipped; parity gap only |
| 13 | `WaitForAttribute` quality-gated ("Good"-only) mode | `SiteRuntime/Scripts/ScriptRuntimeContext.cs:405` | **Low** — planned enhancement per spec §4.2 |
| 14 | `WaitForAttribute` not wired into CentralUI Test-Run sandbox | `docs/plans/2026-06-17-waitfor-deferred-items.md:222` | **Low** — Test-Run parity gap; production scripts unaffected |
| 15 | `BrowseNext` continuation: server's final-page signal not surfaced (client may issue one extra empty pull) | `Commons/Messages/Management/BrowseCommands.cs:34` | **Low** — one wasted round-trip |
| 16 | `StubOpcUaClient` throws on browse — test-infra gap for browse/search unit tests | m7 design "risks" (`2026-06-18-m7-opcua-mxgateway-ux-design.md:245`) | **Low** — limits offline coverage of browse UI |
| 17 | Unified notifications+site-calls outbox page | `2026-06-15-stillpending-completion-design.md:118` (M9 deferrals) | **Low** — two pages remain; explicit decision |
| 18 | Folder drag-drop | same, `[PERM]` permanently deferred | None — closed decision; menu reorder shipped |
| 19 | Bundle signing (asymmetric, non-repudiation), direct cluster-to-cluster pull, differential bundles | `docs/plans/2026-05-24-transport-design.md:402-408` | **Low** — v1 manifest hash + passphrase AES-GCM held sufficient |
| 20 | Deployment EXPIRED-but-unpurged row purge | `ConfigurationDatabase/Repositories/DeploymentManagerRepository.cs:310` ("deferred TODO" — workaround in place) | **Low** — code already compensates when reading |
| 21 | `SiteAuditBacklogReporter` threshold consolidation into `SqliteAuditWriterOptions` | `AuditLog/Site/SiteAuditBacklogReporter.cs:28` | **Low** — config-shape cleanup |
| 22 | KPI history downsampling (hourly rollups) | `Component-KpiHistory.md` (YAGNI) | **Low** — 90-day retention bounds table size |
| 23 | Vestigial site-side notification list read/write surface (§1.3) | `SiteRuntime/Persistence/SiteStorageService.cs:721`, `Repositories/SiteNotificationRepository.cs` | **Low/Medium** — dead-but-live API around an intentionally empty table |
| §1.1 Commons folders beyond documented four (`Observability/` etc.) | Low | **Fixed** | `docs/requirements/Component-Commons.md:294` documents `Observability/` in the layout tree (PLAN-08 T10 had narrowed this: `Serialization/`/`Validators/` were already documented) |
| §1.2 POCO persistence-ignorance | Pass | **Still Pass** | 0 EF/DataAnnotations refs in Commons (excluding `cref`); csproj's only PackageReference remains `ZB.MOM.WW.Audit` |
| §1.3 Site-side repository exception undocumented | Low | **Fixed** | `docs/requirements/Component-ConfigurationDatabase.md:170` documents the `SiteExternalSystemRepository` exception with the `NotSupportedException` rationale |
| §1.3/#23 Vestigial `SiteNotificationRepository` + dead write paths | Low/Med | **Fixed** | File deleted (0 hits in `src/`); only historical why-comments remain (`SiteRuntime/Persistence/SiteStorageService.cs:804-805`); design-invariant lock `Site_NotificationRepository_IsNotRegistered` at `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CompositionRootTests.cs:446` |
| §1.4 Communication → HealthMonitoring layering inversion | Low | **Deferred (accepted)** | Still present (`Communication/Actors/CentralCommunicationActor.cs:13`); register row with rationale + revisit trigger (`2026-07-08-deferred-work-register.md:48`) |
| §1.4 `..\` csproj path separator | Low | **Fixed** | 0 backslash project references across all csprojs |
| §1.5 Options validation on only ~6 of ~20 option classes | Med | **Partially fixed** | 16 `OptionsValidatorBase<T>` validators now exist (all 12 PLAN-08 target components + the prior 4), wired `.ValidateOnStart()` + `TryAddEnumerable`; **residue remains** — see New Finding NF4 (AuditLog's 6 sub-options, Host's Node/Database/Logging options) |
| §1.6 UTC discipline | Pass | **Still Pass** | 0 non-Utc `DateTime.Now`/`DateTimeOffset.Now` in src (.cs + .razor) |
| §1.7 Correlation IDs | Pass | **Still Pass** | 99 `CorrelationId` declarations in `Commons/Messages/` (was 96 — new messages kept the convention) |
| §1.8 No contract-lock tests for ClusterClient records | Low | **Fixed** | `tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Messages/ClusterClientContractLockTests.cs` — 9 tests over 7 records (old-shape, unknown-field, round-trip); locks the real wire finding that Newtonsoft ignores C# ctor optional-param defaults → missing trailing fields deserialize to `default(T)` (lines 57–60); Newtonsoft pinned 13.0.3 in `Directory.Packages.props:54` |
| §1.9 Naming/namespaces | Pass | **Still Pass** | spot checks consistent |
| §2.3 PerformanceTests a correctness-at-small-scale suite | Med | **Partially fixed** | Real throughput test shipped: `PerformanceTests/Streaming/SiteStreamThroughputTests.cs` (100k events, asserts ≥10k events/s + ≥90% delivery, lines 50–94). Failover timing is still a `Skip` placeholder — and its skip reason is now stale (New Finding NF2) |
| §2.4 CLI.Tests excluded from slnx | High | **Fixed** | `ZB.MOM.WW.ScadaBridge.slnx` contains all 30 test projects incl. CLI.Tests; CLAUDE.md gotcha retired; CLI suite (359 cases) runs at solution level |
| §3.1 / §4 Transport `AreaName: null` vs doc claim | High | **Fixed** | Exporter now carries area by name: `Transport/Serialization/EntitySerializer.cs:273` resolves `AreaId` → `AreaNameById`; importer resolves-or-creates. Residual stale comment at :553–554 (New Finding NF6) |
| §3.3 AuditLog reconciliation dials NodeA only | Med | **Fixed** | `AuditLog/Central/SiteEnumerator.cs:31-33, 64-77` — primary = NodeA else NodeB (NodeB-only sites no longer skipped), distinct NodeB rides along as failover endpoint |
| §3.5 `SmsConfiguration.MaxRetryCount` stored but unused | Med | **Fixed** | `NotificationOutbox/NotificationOutboxActor.cs:470, 485` honors `SmsConfiguration.MaxRetries`/`RetryDelay` (PLAN-06 T18) |
| §3.6 Role-check case-sensitivity asymmetry | Med | **Fixed** | `ManagementService/ManagementActor.cs:108, 499, 515, 530` — `StringComparer.OrdinalIgnoreCase` throughout; PLAN-07 T12 canonicalized casing at write |
| §3.7 SecuredWrite audit rows `SourceNode` NULL | Low/Med | **Fixed** | `ManagementActor.cs:1208` routes secured-write audit through `ICentralAuditWriter`, which stamps `SourceNode` from `INodeIdentityProvider` |
| §3.8 Hash-chain tamper evidence (no-op `verify-chain`) | Low/Med | **Deferred (accepted)** | Register row 8 (v1.x locked decision) |
| §3.9 Parquet archival (501) | Low | **Deferred (accepted)** | Register row 9 |
| §3.10 Aggregated live alarm stream | Low/Med | **Fixed (shipped 2026-07-10)** | `Communication/ISiteAlarmLiveCache.cs`, `Communication/Actors/SiteAlarmAggregatorActor.cs`, `SubscribeSite` rpc in `Communication/Protos/sitestream.proto:13`; options eagerly validated (`CommunicationOptionsValidator.cs:72-87`); no central store (in-memory only, no EF migration) — register row moved to Resolved |
| §3.11 Central-persisted cert-trust audit | Low | **Deferred (accepted)** | Register row 11 |
| §3.12 Native-alarm-source-override CSV import | Low | **Partially fixed** | Parser + ManagementActor handler + CLI `instance native-alarm-source import --file` shipped 2026-07-10; only the Central UI upload affordance remains (register row 12 updated accordingly) |
| §3.13 `WaitForAttribute` quality-gated mode | Low | **Fixed** | `Commons/Messages/InboundApi/RouteToInstanceRequest.cs:117` — `RequireGoodQuality` param, enforced site-side, tested |
| §3.14 `WaitForAttribute` in Test-Run sandbox | Low | **Fixed** | Sandbox `Attributes.WaitAsync`/`WaitForAsync` route via `ISandboxInstanceGateway`; predicate-form waits documented unsupported (register Resolved row 14) |
| §3.15 `BrowseNext` final-page signal | Low | **Fixed / no longer applicable** | `Commons/Messages/Management/BrowseCommands.cs:18-36` — `ContinuationToken` + `Truncated` on `BrowseNodeResult`; "Load more" renders only while a token remains |
| §3.16 `StubOpcUaClient` throws on browse | Low | **Fixed** | Browse + search supported; `tests/.../DataConnectionLayer.Tests/StubOpcUaClientBrowseTests.cs`, `StubOpcUaClientSearchTests.cs` |
| §3.17 Unified outbox page | Low | **Deferred (accepted)** | Register row 17 (explicit M9 decision) |
| §3.18 Folder drag-drop | — | **Closed (permanent)** | Register row 18 `[PERM]` |
| §3.19 Bundle signing / cluster pull / differential bundles | Low | **Deferred (accepted)** | Register row 19 |
| §3.20 Deployment EXPIRED-row purge | Low | **Fixed** | `Communication/Actors/PendingDeploymentPurgeActor.cs` (central singleton, options-validated interval; PLAN-04) |
| §3.21 `SiteAuditBacklogReporter` threshold consolidation | Low | **Fixed** | `SqliteAuditWriterOptions.BacklogPollIntervalSeconds` drives the reporter cadence (`AuditLog/Site/SiteAuditBacklogReporter.cs:61-87`) |
| §3.22 KPI hourly rollups | Low | **Fixed (shipped 2026-07-10)** | `Commons/Entities/Kpi/KpiRollupHourly.cs`; `KpiHistoryOptions.cs:46-73` (RollupInterval 1h, Lookback 3h, Retention 365d, Threshold 168h); validator bounds (`KpiHistoryOptionsValidator.cs:31-44`); `Component-KpiHistory.md` updated (24 rollup mentions) — register row moved to Resolved |
| §4 `docs/components/` lags 3 components vs README claim | Low | **Fixed (via scoping)** | `README.md:111` + `docs/components/README.md:8-9` now state "24 of the 27… ScriptAnalysis/KpiHistory/DelmiaNotifier pending", pointing at the register; actual count verified: 24 component docs + TreeView + index = 26 files. Backfill itself remains deferred (register) |
| §4 DelmiaNotifier breaks `Component-<Name>.md` convention | Low | **Fixed (documented exception)** | CLAUDE.md Document Conventions now carries the explicit DelmiaNotifier exception bullet |
Overall: ~20 open items, of which only #1–#4 warrant action before the next release; the rest are consciously logged, mostly with honest in-product behavior (501s, no-op stubs that say so).
**Disposition counts: 20 Fixed (verified) · 3 Partially fixed · 6 Deferred (accepted, register rows) · 1 Closed-permanent · 0 Not fixed · 0 Regressed.** (The 7 always-Pass items stayed Pass.)
---
## 4. Docs-Code Drift
## 2. Refreshed Convention Audit (round-2 sweep)
Drift is exceptionally low for a docs-as-spec repo. Spot checks:
- **Component-KpiHistory.md vs code** — verified in-sync: `KpiHistoryOptions` defaults (SampleInterval 60s, RetentionDays 90 [1,3650], PurgeInterval 1d, DefaultMaxSeriesPoints 200 [2,5000]) match `src/.../KpiHistory/KpiHistoryOptions.cs` exactly, including range constraints; source placement (`IKpiSampleSource` impls with owners, bucketer in Commons, query service in CentralUI) matches the actual file layout and test placement.
- **Component-StoreAndForward.md vs code** — verified in-sync: the doc's nuanced notification-parking semantics (`DefaultMaxRetries` cap, `maxRetries: 0` escape hatch), `ICachedCallLifecycleObserver` telemetry hook, and tracking-table-vs-buffer split all correspond to the shipped files (`NotificationForwarder.cs`, `ParkedMessageHandlerActor.cs`, `StoreAndForwardService.cs`, `ReplicationService.cs`).
- **Component-ClusterInfrastructure.md vs code** — in-sync, and exemplary: the code carries a comment (`ClusterInfrastructure/ServiceCollectionExtensions.cs:34-47`) explaining that a dead throwing extension was removed *because* the doc's "Code Placement" section settled ownership in Host — docs and code cross-reference each other.
- **[High] Component-Transport.md:21 and CLAUDE.md #24 claim `Area` membership travels "carried by name"** — but the exporter emits `AreaName: null` unconditionally (`Transport/Serialization/EntitySerializer.cs:251-254`) with a TODO, and `BundleImporter.cs:511-518` confirms "Areas don't travel (see export TODO)". The import-side resolution (create-area-if-missing, `BundleImporter.cs:3104-3168`) is fully built, making the doc claim look shipped on casual inspection. Either finish the export (small: add Areas to `EntityAggregate` + the id→name lookup) or correct the doc/CLAUDE.md to say area membership is not yet carried.
- **[Low] `docs/components/` reference docs lag by three components**: 24 per-component reference docs exist but ScriptAnalysis (#25), KpiHistory (#26), and DelmiaNotifier (#27) are missing, while `README.md:111` claims "One doc per component (plus the shared TreeView)".
- **[Low] DelmiaNotifier breaks the requirements-doc naming convention**: no `docs/requirements/Component-DelmiaNotifier.md`; the README component table row 27 links to the project README instead. Defensible for an external tool, but CLAUDE.md's own Document Conventions say component documents live in `docs/requirements/` as `Component-<Name>.md` — either add a thin spec doc or note the exception in the conventions section.
- **README component table** — all 27 rows present and links resolve to existing files; component descriptions match CLAUDE.md's component list. TreeView correctly footnoted as a sub-component. No stale cross-references found in the sampled docs.
- Historical `docs/plans/` design docs contain superseded "future work" lists (e.g. transport-design §18 lists site-scoped transport as future, which M8 later shipped) — acceptable since they are dated decision records, and the living docs (Component-*.md, CLAUDE.md, README) reflect the shipped state.
- **UTC**: 0 non-Utc `DateTime.Now`/`DateTimeOffset.Now` across src `.cs`/`.razor` — total compliance held through 268 commits.
- **TODO surface**: 0 `// TODO` / `// FIXME` / `// HACK` in all of src (round 1 had two, both the Transport area export — now shipped). 0 `NotImplementedException` in production code. 29 "deferred/follow-up" comment mentions (was 39), sampled — all are pointer-style documentation of logged decisions, none load-bearing stubs.
- **Commons purity**: intact — 0 EF/DataAnnotations dependencies; Commons csproj still references only `ZB.MOM.WW.Audit`.
- **Correlation IDs**: 99 declarations in `Commons/Messages/` — the new cross-cluster messages added by the plans (per-node KPI requests, secured-write paging, `RouteToWaitForAttributeRequest`, browse `SiteIdentifier` additions) all kept the convention or ride `ManagementEnvelope`.
- **Options pattern**: 16 eager validators (`OptionsValidatorBase<T>` + `ValidateOnStart` + `TryAddEnumerable`) covering every primary per-component options class, including the two post-initiative features (live-alarm-cache fields validated at `CommunicationOptionsValidator.cs:72-87`; rollup fields bounded in `KpiHistoryOptionsValidator.cs:31-44`). Residue: NF4 below.
- **Message-contract enforcement**: additive-only is now *enforced* for the 7 highest-traffic ClusterClient records (`ClusterClientContractLockTests`) and the gRPC protos (existing proto contract tests; the new `SubscribeSite` stream has behavioral coverage in `Communication.Tests/Grpc/`). The lock suite deliberately covers "top" records, not all — new records added by the plans are outside it, which is the accepted design of the guard.
- **csproj hygiene**: 0 `Version=` attributes on PackageReferences in src/tests (central package management honored); 0 backslash path separators; 0 third-party Blazor component frameworks (MudBlazor/Radzen/Blazorise/Syncfusion) — the "custom components only" UI rule held.
- **Slnx**: all 30 test projects are members (the 31st `tests/` match is the `<Folder Name="/tests/">` element).
- **CLAUDE.md accuracy spot-checks**: the claims audited all match code — CLI.Tests slnx note, live-alarm-stream description (`ISiteAlarmLiveCache`, seed-then-stream, 15s poll fallback, no persisted store — no EF migration exists for it), KPI rollup knobs (168h threshold / 365d retention match `KpiHistoryOptions.cs:65-73`), SecuredWrite `SourceNode` stamping, SMS retry policy.
---
## Prioritized Recommendations
## 3. Refreshed Test Inventory
1. **[High] Add `tests/ZB.MOM.WW.ScadaBridge.CLI.Tests` to `ZB.MOM.WW.ScadaBridge.slnx`.** One line; removes a standing CI blind spot of 279 tests and retires a documented footgun.
2. **[High] Resolve the Transport Area contradiction** — either implement area export (importer already complete; add Areas to `EntityAggregate` and the id→name lookup in `EntitySerializer.cs:251`) or amend `Component-Transport.md:21` + CLAUDE.md #24 to state area membership does not travel.
3. **[Medium] Options-validation sweep**: add `IValidateOptions<>`/`ValidateOnStart` to the ~14 unvalidated option classes (Transport, StoreAndForward, SiteRuntime, Communication, InboundAPI, etc.), copying the existing HealthMonitoring/ClusterInfrastructure validator pattern.
4. **[Medium] Close the AuditLog reconciliation NodeB gap** (`SiteEnumerator.cs:64`): dial NodeB when NodeA is unreachable — the shape (`SiteEntry`) already supports it, and reconciliation is the pipeline's loss-recovery safety net.
5. **[Medium] Honor or remove `SmsConfiguration.MaxRetryCount`** (`Commons/Entities/Notifications/SmsConfiguration.cs:36`) — an admin-editable field with no effect is worse than no field.
6. **[Medium] Decide the PerformanceTests story**: rename to reflect its correctness-at-scale scope, or add the two highest-value real measurements (failover time, gRPC stream throughput) behind the existing `Category=Performance` trait.
7. **[Low/Medium] Excise the vestigial site-side notification-list surface** (`SiteNotificationRepository` reads + `StoreNotificationListAsync`) now that the purge guarantees the table stays empty.
8. **[Low] Unify role-string casing handling** between UI `RequireClaim` policies and the ManagementActor check (`ManagementActor.cs:1910`) — normalize at write (already validated canonically) and compare consistently.
9. **[Low] Backfill the three missing `docs/components/` reference docs** (ScriptAnalysis, KpiHistory, DelmiaNotifier) or scope the README claim.
10. **[Low] Add contract-lock tests for the top ClusterClient message records** (mirror the existing proto contract tests) to enforce, not just intend, additive-only evolution under version skew.
30 test projects (all in the slnx), **~5,620 `[Fact]`/`[Theory]` attribute sites** (round 1: ~5,050; +~570 from the fix plans), ~170k test LOC against ~212k src LOC. Executed-case counts run higher with Theory expansion — the 2026-07-10 terminal sweeps report e.g. CentralUI 898, Commons 609, SiteRuntime 494, ManagementService 448, CLI 359 — roughly **6,300+ executed cases**. Attribute counts per project (grep-based, approximate):
| Project | Attrs | Δ vs r1 | Notes |
|---|---|---|---|
| CentralUI.Tests | 860 | +31 | still the largest suite |
| Commons.Tests | 506 | +41 | now incl. ClusterClientContractLockTests |
| SiteRuntime.Tests | 497 | +40 | |
| TemplateEngine.Tests | 452 | +19 | |
| ManagementService.Tests | 306 | +72 | frozen 138-command authz matrix + dispatch-coverage guard (PLAN-07 UA2) |
| CLI.Tests | 300 | +21 | **now in the slnx** |
| Communication.Tests | 269 | +58 | live-alarm-cache/aggregator suites added |
| AuditLog.Tests | 263 | +26 | |
| InboundAPI.Tests / ConfigurationDatabase.Tests | 234 / 234 | +16 / +15 | |
| DataConnectionLayer.Tests | 200 | +10 | incl. Stub browse/search tests |
| Host.Tests | 164 | +22 | incl. the site-notification design-invariant lock |
| Transport.Tests / Transport.IntegrationTests | 148 / 96 | +24 / +36 | round-trip equivalence guard (PLAN-05 T8) |
| StoreAndForward.Tests / Security.Tests | 144 / 144 | +28 / +11 | |
| NotificationOutbox.Tests | 132 | +10 | |
| DeploymentManager.Tests | 112 | +14 | still thin relative to scope (register row) |
| HealthMonitoring.Tests | 94 | +10 | |
| ExternalSystemGateway.Tests | 85 | +16 | |
| IntegrationTests | 74 | +5 | incl. `Cluster/TwoNodeClusterFixture` + failover/recovery drills |
| SiteEventLogging.Tests | 58 | +4 | |
| SiteCallAudit.Tests | 44 | +13 | grew ~40% but still the thinnest per src LOC (register row) |
| ScriptAnalysis.Tests / NotificationService.Tests | 43 / 43 | +3 / +7 | |
| CentralUI.PlaywrightTests | 36 | 0 | 8 env-only failures documented (need live docker cluster) |
| KpiHistory.Tests | 29 | +15 | doubled with rollup coverage |
| DelmiaNotifier.Tests | 23 | 0 | |
| ClusterInfrastructure.Tests | 18 | +3 | |
| PerformanceTests | 12 | +2 | see below |
**PLAN-08's new tests are real, with one asterisk.** The contract-lock suite (9 tests) does exactly what the review asked and surfaced a genuine wire-behavior finding (Newtonsoft `default(T)` on missing trailing fields — now locked as documented reality). The stream-throughput test is a genuine measurement (100k events through the real `SiteStreamManager` BroadcastHub, asserting ≥10k events/s and ≥90% delivery). The asterisk is the failover-timing placeholder: it shipped as designed (a documented, skipped protocol), but its skip precondition is already satisfied — see NF2.
---
## 4. Deferred-Work Register Assessment
The register (`docs/plans/2026-07-08-deferred-work-register.md`) exists, is committed, and is substantially accurate — including a "Resolved (verified against the code 2026-07-10)" section whose ten resolution claims I spot-verified (items 7, 10, 13, 14, 15, 16, 20, 21, 22, 12-CLI: all confirmed in source, citations in §1 above). Post-initiative work genuinely retired two register rows (#10 live alarm stream, #22 KPI rollups) and most of a third (#12 CSV import — only the Blazor upload affordance remains).
**Remaining open deferrals (verified still-accurate):** #8 hash-chain (CLI `verify-chain` still a no-op), #9 Parquet (501), #11 central cert-trust persistence, #12 (UI upload button only), #17 unified outbox page, #19 bundle signing/pull/differential, plus the four "new deferrals from review 08" rows (layering inversion — still present at `CentralCommunicationActor.cs:13`; three reference docs; SiteCallAudit/DeploymentManager test backfill; perf envelope). #18 is permanently closed. **Net: ~10 open deferrals, all with rationale + trigger, none mislabeled.**
Two register-integrity problems, though:
1. **One revisit trigger has already fired unnoticed** — the perf-envelope row's trigger is "PLAN-01 rig landing", and `tests/.../IntegrationTests/Cluster/TwoNodeClusterFixture.cs` landed 2026-07-08, *before* the placeholder itself shipped (2026-07-10). Nobody revisited (NF2).
2. **The register is no longer the single place.** The master tracker grew its own initiative-level "Deferred / Won't-Fix Registry" (`archreview/plans/00-MASTER-TRACKER.md:137+`) holding items absent from the register — most notably the **SBR oldest-crash total-outage gap** (a real operational risk, owner: user decision) and the unapplied `deploy/wonder-app-vd03` overlay edits (whose absence NULLs `SourceNode` on that deployment's audit rows). An untracked root-level `deferred.md` snapshot duplicates the register a third time and has already drifted from it (NF5). The register's own Fix-now table also violates its stated remove-when-landed rule — all 7 rows landed but are still listed as open.
---
## 5. New Findings (round 2)
### NF1 [High] — Live production API key in plaintext at the repo root
`test.txt` (untracked, repo root) contains a working Inbound API key for the production host `wonder-app-vd03.zmr.zimmer.com:8085` (`sbk_eb5acc0b…`, full key present). It is one `git add -A` (or one run of the `pushit` skill, which stages everything) away from entering history, and it is a live credential regardless. **Rotate/revoke the key** (the dual-key rotation procedure from PLAN-06 T24 exists for exactly this), delete the file, and consider a `.gitignore`/pre-commit secret-scan guard for the root. (File: `/Users/dohertj2/Desktop/ScadaBridge/test.txt`.)
### NF2 [Medium] — Failover-timing test's skip precondition is already satisfied; the 25s envelope remains unmeasured
`tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Failover/FailoverTimingTests.cs:26-28` skips with "Requires the real two-node failover rig delivered by PLAN-01 … wire it to the rig when PLAN-01 lands" — but PLAN-01's rig landed 2026-07-08 (`tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/TwoNodeClusterFixture.cs`), two days before this placeholder shipped. The design's single most-quoted performance claim (~25s failover) is still unverified by any test, the register's revisit trigger for this row has fired, and the skip message is now misleading. Wire the protocol to the fixture (or update the skip reason + register row with the real blocker, e.g. "in-proc rig can't hard-kill a process").
### NF3 [Medium] — CHANGELOG.md abandoned and now factually wrong
`CHANGELOG.md` was last touched 2026-06-02 (`b104760b`), 268+ commits and the entire arch-review initiative ago; its only entry sits under `[Unreleased]`. Worse, that entry asserts "`Operator`/`Engineer` are unused" (`CHANGELOG.md:11-12`) — false since M7 shipped the `Operator`/`Verifier` secured-write roles. A changelog that is both stale and wrong is worse than none: either backfill it from the merge history (the master tracker's wave summaries are ready-made material) or delete it and declare the tracker + `docs/plans/` the release record.
### NF4 [Low] — Options-validation residue: AuditLog sub-options and Host-owned options still bind without validation
PLAN-08 covered the 12 primary component options, but six AuditLog sub-option classes bind via `AddOptions<T>().Bind(...)` with no `.ValidateOnStart()` and no validator: `SqliteAuditWriterOptions`, `SiteAuditTelemetryOptions`, `SiteAuditRetentionOptions` (`AuditLog/ServiceCollectionExtensions.cs:100-108`) and `AuditLogPartitionMaintenanceOptions`, `AuditLogPurgeOptions`, `SiteAuditReconciliationOptions` (`:368-390`). Host-owned `NodeOptions`, `DatabaseOptions`, `LoggingOptions` remain plain `services.Configure<>` (`Host/SiteServiceRegistration.cs:160-172`) — notable because an empty `NodeOptions.NodeName` silently NULLs the `SourceNode` audit column (the exact failure mode the tracker logged for wonder-app-vd03). Also, `OperationTrackingOptions` has no binding site at all (only consumed at `SiteRuntime/Tracking/OperationTrackingStore.cs:59`) — defaults-only, not configurable from appsettings; either bind it or document that it is code-configured.
### NF5 [Low] — Deferral tracking has re-scattered into three places, one of them untracked and drifting
The register's charter ("new deferrals get appended here instead of scattering") is already violated: (a) the master tracker's own registry (`archreview/plans/00-MASTER-TRACKER.md:137-150`) holds the **SBR oldest-crash gap** and the **wonder-app-vd03 overlay edits** — arguably the two highest-risk open items in the whole program — with no register rows; (b) an untracked root `deferred.md` snapshot duplicates the register and has drifted (it counts "12 open deferrals" vs the register's ~10; different section layout); (c) the register's Fix-now table still lists all 7 landed items despite its remove-when-landed rule. Fold the tracker's two live items into the register, delete or commit-and-subordinate `deferred.md`, and clear the Fix-now table.
### NF6 [Low] — Stale comment contradicts the shipped Transport area export
`Transport/Serialization/EntitySerializer.cs:553-554`: "Areas don't travel (see export TODO), so AreaId stays null" — but the export TODO was fixed (area travels by name, `:268-273`) and no TODO exists anymore. The comment sits in `FromBundleContent`, which still materializes `AreaId = null`; it is off the importer's path (BundleImporter walks the raw DTO), so this is comment drift plus a round-trip-fidelity trap in the helper rather than a functional bug — but it is precisely the "half-fixed-looking" confusion round 1 flagged for this feature. Update the comment (and either populate `AreaName`-awareness in the helper or say why it intentionally doesn't).
### NF7 [Low] — Untracked working-tree residue, including an uncommitted design plan
Beyond `test.txt` (NF1): `docs/plans/2026-06-30-mes-alarm-status-api.md` is a 12-day-old draft plan sitting untracked inside `docs/plans/` — the repo's docs-as-spec convention says plan docs are committed decision records; untracked, it is invisible to every other session and one `git clean` from gone. `ScadaBridge-docs-fixed.md` / `ScadaBridge-docs-issues.md` (root-level generated reports) and `.claire/` (tool worktrees) are also untracked clutter. Commit the plan doc (marked Draft), and remove or relocate the rest.
### NF8 [Low] — Duplicate `DataConnectionOptions` binding in the Host
`Host/SiteServiceRegistration.cs:140` binds `DataConnectionOptions` via `services.Configure<>` even though `AddDataConnectionLayer()` (called at `:51`) already binds it with `.ValidateOnStart()` (`DataConnectionLayer/ServiceCollectionExtensions.cs:17-19`). Both bindings compose so validation still runs, but the duplicate invites section-name drift between the two sites. Delete the Host copy (same applies to the `CommunicationOptions`/`NotificationOptions` `Configure<>` calls in `BindSharedOptions` at `:166/:171`, which are redundant on roles where the owning SCE runs).
---
## 6. What's Genuinely Good
- **The fix plans actually fixed things.** 20 of 23 actionable round-1 items verified fixed at the exact seams the review pointed at, with zero regressions found in this sweep — and several fixes exceeded the ask (the contract-lock suite surfaced a real wire-behavior discovery; the round-trip equivalence guard in Transport caught 5 silent-data-loss bugs the per-field tests missed).
- **Convention discipline survived 268 commits of high-velocity parallel-agent churn.** UTC still total, Commons still pure, TODO count went 2 → 0, correlation IDs on every new message, central package management unviolated, no third-party UI frameworks crept in.
- **The two post-initiative features shipped convention-complete on day one**: both came with validated options (live-cache fields in `CommunicationOptionsValidator`, rollup bounds in `KpiHistoryOptionsValidator`), tests, design-doc updates (`Component-KpiHistory.md`, the 2026-07-10 plan docs), CLAUDE.md updates, and register-row closure — the "docs + code + tests together" rule working as designed, unprompted.
- **Deferral honesty**: shipped no-ops and 501s still say what they are; the register's Resolved section shows items being verified against code before being marked done, and two register rows were retired by real shipped work within 48 hours of the register's creation.
- **Test posture improved where it was weakest**: KpiHistory.Tests doubled, SiteCallAudit.Tests +40%, the frozen 138-command authorization matrix turned the largest unaudited security surface into a locked contract, and CLI.Tests finally runs at the solution level.
---
## 7. Prioritized Recommendations (round 2)
1. **[High] Rotate the exposed wonder-app-vd03 API key and delete `test.txt`** (NF1); consider a root-level secret-scan/gitignore guard.
2. **[Medium] Wire `FailoverTimingTests` to `TwoNodeClusterFixture`** or update the skip reason + register row with the true blocker (NF2) — the 25s envelope is the last unmeasured headline design claim.
3. **[Medium] Fix or kill CHANGELOG.md** (NF3) — at minimum correct the now-false Operator/Engineer claim.
4. **[Low] Finish the options sweep's tail**: ValidateOnStart + validators for the 6 AuditLog sub-options and a minimal `NodeOptions` validator (non-empty `NodeName` when auditing is active) (NF4); drop the duplicate Host bindings (NF8).
5. **[Low] Re-consolidate deferral tracking** (NF5): move the SBR gap + wonder-app overlay items into the register, clear the landed Fix-now rows, delete/subordinate root `deferred.md`.
6. **[Low] Commit `docs/plans/2026-06-30-mes-alarm-status-api.md` and clean root clutter** (NF7); fix the stale `EntitySerializer.cs:553` comment (NF6).
## Severity Tally — NEW findings only
| Severity | Count | Findings |
|---|---|---|
| Critical | 0 | — |
| High | 1 | NF1 (production API key in working tree) |
| Medium | 2 | NF2 (failover envelope unmeasured, stale skip), NF3 (CHANGELOG abandoned + wrong) |
| Low | 5 | NF4 (options residue), NF5 (deferral scattering), NF6 (stale Transport comment), NF7 (untracked plan doc + clutter), NF8 (duplicate bindings) |
archreview/plans/00-MASTER-TRACKER.md
+51
View File
@@ -6,6 +6,57 @@
Every plan follows the TDD bite-sized-task format and ships a co-located `.tasks.json` manifest. Execute a plan with the `superpowers-extended-cc:executing-plans` skill (pass the plan path); it reads/updates the `.tasks.json`. **When a plan's status changes, update the table below** — this file is the single source of truth for initiative progress.
---
# ROUND 2 (2026-07-12)
**Source:** the round-2 re-review (all 8 `archreview/0X-*.md` reports rewritten 2026-07-12 at HEAD `8c888f13`): every round-1 finding source-verified (168 fixed, 0 regressions, 0 false claims), plus **56 new findings** (1 Critical / 4 High / 15 Medium / 36 Low) concentrated in post-baseline code. Round-2 plans: **8 plans, 86 tasks.**
## Round-2 Status Board
| Plan | Domain | Tasks | Done | Status | Findings coverage |
|------|--------|------:|-----:|--------|-------------------|
| [PLAN-R2-01](PLAN-R2-01-cluster-host-failover.md) | Cluster, Host & Failover | 11 | 0 | ⬜ Not started | N1→T1–T4 (incl. live drill run + envelope measurement, covers R2-08's NF2); N2→T5–T7 (`needs-user`-adjacent: deploy overlay edits, no git add); N3→T8; N4→T9; N5→T10/T11; N6→R2-08 |
| [PLAN-R2-02](PLAN-R2-02-communication-store-and-forward.md) | Communication & S&F + live alarm stream | 15 | 0 | ⬜ Not started | **N1 Critical→T1–T4** (shared oldest-Up predicate, failing-first repro); **N2 High→T5–T7** (chunked resync protocol); N3→T8; N4→T9; N5→T6; N6→T10 (MUTEX w/ R2-07); N7→T11/T12; N8→T13/T14; N9→T15 |
| [PLAN-R2-03](PLAN-R2-03-site-runtime-dcl.md) | Site Runtime & DCL | 7 | 0 | ⬜ Not started | N1→T1; N2→T2/T3; N3→T4; N4→T5/T6 (full compile-cache adoption); N5/N6→T7 |
| [PLAN-R2-04](PLAN-R2-04-data-audit-backbone.md) | Data & Audit Backbone + KPI rollups | 13 | 0 | ⬜ Not started | **R1 High→T2–T4** (sliced backfill + watermark fast-path); R2→T1; R3→T5–T7; R4→T8; R5→T9; R6→T10/T11 (1 EF migration, build-first gotcha noted); R7→T12; final verify T13 |
| [PLAN-R2-05](PLAN-R2-05-templates-deployment-transport.md) | Templates, Deployment & Transport | 9 | 0 | ⬜ Not started | N1→T2–T4 (verdict-cache globals-surface keying); N2→T1; N3→T5 (publisher; subscriber = R2-06); N4→T7; N5→T6 (trust-gate extension, hard-error); N6→T8; docs T9 |
| [PLAN-R2-06](PLAN-R2-06-edge-integrations.md) | Edge Integrations | 6 | 0 | ⬜ Not started | N1→T1/T2 (**wires the change-bus subscriber**, closes round-1 U2 partial); N2→T3; N3→T5 (typed SITE_UNREACHABLE); N4→T4; N5→T6 |
| [PLAN-R2-07](PLAN-R2-07-ui-management-security.md) | UI, Management & Security | 12 | 0 | ⬜ Not started | N1→T1 (hub throttle); N2→T2/T3 (ForwardedHeaders); N3→T4–T6 (secured-write site scope, ManagementActor lane); N4→T7; N5→T8; N6→T10 (MUTEX w/ R2-02); N7→T9; N8→T11; N9→T12 |
| [PLAN-R2-08](PLAN-R2-08-conventions-hygiene.md) | Conventions & Hygiene | 13 | 0 | ⬜ Not started | **NF1 High P0→T1–T3** (`needs-user`: key rotation + secret-file deletion — ldap/sql login files hold LIVE secrets too); NF3→T4; NF4→T5–T8 (T8 = possible live `IOperationTrackingStore` DI gap, verify-then-fix); NF5→T10/T11; NF6→T12; NF7→T13; NF8→T9 (Host duplicates are the LIVE bindings — canonicalize SCE sections first) |
**Round-2 progress: 0 of 86 tasks.**
## Round-2 P0 (do first, any order)
1. **R2-08 T1/T2** — rotate the exposed wonder-app-vd03 API key; delete `test.txt` + the three root credential files (ALL contain live secrets incl. a production sysadmin password). `needs-user`.
2. **R2-02 T1–T4** — the Critical: unify the active-node predicate; failing-first buffer-wipe repro then fix.
3. **R2-02 T5–T7** — the resync frame-size High (chunked protocol).
4. **R2-04 T2–T4** — bound the rollup backfill before the next central failover at data volume.
## Round-2 Cross-Plan Mutexes & Dedupe Rulings
| Surface / finding | Ruling |
|---|---|
| `SiteAlarmLiveCacheService.cs` | R2-02 T10 (coalescing) and R2-07 T10 (aggregator deathwatch/sticky IsLive) must NOT run concurrently — serialize, either order. R2-02 T12/T15 also touch adjacent files; keep the whole live-alarm slice single-lane across plans. |
| `ManagementActor.cs` | Single-writer lane (round-1 rule stands): R2-07 T4/T6 and R2-02 T13 (`HandleDeleteSite`) serialize against each other. |
| `IScriptArtifactChangeBus` | Publisher Add-fix = R2-05 T5; subscriber wiring + Host comment = R2-06 T1/T2. Independent, no cross-plan blockedBy. |
| Failover envelope measurement (report-08 NF2) | Owned by R2-01 T4; R2-08 T10 only repoints the register row. |
| Deferral-register consolidation (reports 01 N6 / 08 NF5) | Owned by R2-08 T10/T11. |
| Options-validation tail | Split by owner: SiteRuntime knobs = R2-03 T4; Transport knob = R2-05 T8; S&F sweep knobs = R2-02 T9; AuditLog/Host = R2-08 T5–T8. |
| SBR keep-oldest active-crash topology gap | Still a registered deferred USER decision — no round-2 plan fixes it; R2-01 documents/measures it, R2-08 T10 moves it into the canonical register. |
## Round-2 Recommended Waves
- **Wave 0 (P0):** R2-08 T1–T3 (`needs-user`) ∥ R2-02 T1–T7 ∥ R2-04 T1–T4.
- **Wave 1:** rest of R2-02 ∥ R2-03 ∥ R2-05 ∥ R2-06 (all file-disjoint domains); R2-01 T1–T4 (drill + measurement, needs live docker cluster).
- **Wave 2:** R2-07 (after R2-02's live-alarm slice lands, honoring the mutex) ∥ rest of R2-04 ∥ rest of R2-01 ∥ rest of R2-08.
- Full-suite sweep + docker redeploy at the end, per round-1 practice (watch for stale enum-count lock-in tests — bit PLAN-04 and PLAN-07).
---
# ROUND 1 (2026-07-08) — COMPLETE
## Status Board
| Plan | Domain | Tasks | Done | Status | Findings coverage |
archreview/plans/PLAN-R2-01-cluster-host-failover.md
+699
View File
@@ -0,0 +1,699 @@
# PLAN-R2-01 — Cluster, Host & Failover Round-2 Fixes Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
**Goal:** Make the failover story *honest* — rewrite and actually RUN the failover drill so it proves what two-node keep-oldest can deliver (and measures what it cannot), apply the three wonder-app-vd03 overlay edits that were deferred on a factually wrong rationale, and close the round-2 residuals (never-reported metrics staleness, stale leader comments, undrained site singletons).
**Architecture:** The drill (`docker/failover-drill.sh`) gains a `DRILL_MODE` — `standby` (default, the survivable younger-node crash that PASSes within the ~25s SBR budget) and `active` (the oldest-node crash that keep-oldest cannot survive: the drill *expects* the outage and measures it, documenting the registered deferred user decision instead of codifying a recovery the topology cannot deliver). The in-process envelope measurement reuses `TwoNodeClusterFixture` (production `BuildHocon`, now with production-timing knobs) from the previously-skipped `FailoverTimingTests`. Site-role singleton drains come from generalizing `CentralSingletonRegistrar` into a role-aware `SingletonRegistrar` so `deployment-manager` and `event-log-handler` get the same `PhaseClusterLeave` GracefulStop the seven central singletons got in round 1.
**Tech Stack:** C#/.NET, Akka.NET 1.5.62 (Akka.Cluster, Akka.Cluster.Tools), xUnit, bash + Docker Compose + Traefik, JSON deploy overlays (PowerShell `install.ps1`).
Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per-project: `dotnet test tests/<project>`. Docker rig: `bash docker/deploy.sh`.
**Binding scope rulings (do not deviate):**
- The two-node keep-oldest **active/oldest-crash availability gap is a registered deferred USER decision** (master tracker 2026-07-08 "Follow-up discovered during P0 execution"; `SbrFailoverTests.cs:15-23`). No task in this plan changes the SBR strategy or topology — Tasks 1–3 *document and measure* the gap.
- The oldest-vs-leader predicate unification inside `SiteCommunicationActor`/`SiteReplicationActor` and everything resync-related is **owned by PLAN-R2-02** — do not touch those files here.
- N6 (deferred-work-register consolidation) is **owned by PLAN-R2-08** — Task 7 here corrects only the factually wrong N2 prose in the master tracker.
- `/deploy/` is **gitignored** (`.gitignore:48: /deploy/`) — Tasks 5 and 6 edit the on-disk artifact and deliberately do NOT `git add` it (never use `git add -f`); Task 7's tracker correction is the version-controlled record.
---
### Task 1: Rewrite `failover-drill.sh` — standby-victim default + explicit active-victim gap mode
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 4, Task 5, Task 6, Task 8, Task 9, Task 10
**Files:**
- Modify: `docker/failover-drill.sh` (full rewrite — current script is 45 lines; lines 12-19 pick the ACTIVE node as victim, which is exactly the crash keep-oldest cannot survive)
Context (round-2 N1): under the round-1 oldest-member unification, the *active* node IS the oldest. `docker/failover-drill.sh:17-19` kills it, so the shipped drill exercises the one crash two-node keep-oldest is *known* not to recover from (the survivor self-downs; the restarted `central-b` cannot re-bootstrap because both nodes list `scadabridge-central-a` as FIRST seed — `docker/central-node-a/appsettings.Central.json:10-13`, `docker/central-node-b/appsettings.Central.json:10-13` — and only the first seed may self-join to form a new cluster). The drill has also never been run (`docker/README.md:288`). Fix the drill so its PASS criteria match reality per direction.
1. No unit test (bash script). The verification command is the "test": `bash -n docker/failover-drill.sh` (must exit 0), plus `DRILL_MODE=bogus bash docker/failover-drill.sh` (must exit 2 with the usage error — this exercises the argument guard without needing a cluster).
2. Replace the script's entire contents with:
```bash
#!/usr/bin/env bash
# Failover drill against the running docker cluster (bash docker/deploy.sh first).
#
# ROUND-2 REWRITE (arch-review 01 round 2, N1). The original drill killed the
# ACTIVE central node — but under the unified oldest-member semantics the
# active node IS the oldest, i.e. the one crash two-node keep-oldest CANNOT
# survive (registered deferred user decision, master tracker 2026-07-08;
# SbrFailoverTests.cs XML doc). Two modes:
#
# DRILL_MODE=standby (default) — kills the STANDBY (younger) central node.
# The survivable direction: SBR downs the crashed member, the active node
# keeps its singletons, and Traefik routing never goes dark. PASS = the
# survivor logs the member removal within TIMEOUT_S (budget ~25s+: 10s
# failure detection + 15s stable-after) while /health/active stays up.
#
# DRILL_MODE=active — kills the ACTIVE (oldest) central node. THE EXPECTED
# OUTCOME IS A TOTAL CENTRAL OUTAGE: keep-oldest downs the partition
# without the oldest, so the younger survivor downs ITSELF (down-if-alone
# cannot help — the alone-oldest is dead and cannot down itself), and the
# self-downed survivor cannot re-form a cluster alone unless it is the
# FIRST seed (both nodes list central-a first; only the first seed may
# self-join). This mode measures the dark window and PASSes only when
# central recovers AFTER the victim container is restarted. It exists to
# make the registered gap observable — not to pretend it is covered.
set -euo pipefail
TRAEFIK_URL="${TRAEFIK_URL:-http://localhost:9000}"
TIMEOUT_S="${TIMEOUT_S:-90}"
DRILL_MODE="${DRILL_MODE:-standby}"
OUTAGE_CONFIRM_S="${OUTAGE_CONFIRM_S:-60}"
active_container() {
if curl -sf -o /dev/null "http://localhost:9001/health/active"; then echo scadabridge-central-a
elif curl -sf -o /dev/null "http://localhost:9002/health/active"; then echo scadabridge-central-b
else echo "ERROR: no active central node found" >&2; exit 1; fi
}
peer_of() { [ "$1" = scadabridge-central-a ] && echo scadabridge-central-b || echo scadabridge-central-a; }
case "$DRILL_MODE" in
standby|active) ;;
*) echo "ERROR: DRILL_MODE must be 'standby' or 'active' (was '$DRILL_MODE')" >&2; exit 2 ;;
esac
ACTIVE=$(active_container)
if [ "$DRILL_MODE" = standby ]; then
VICTIM=$(peer_of "$ACTIVE"); SURVIVOR="$ACTIVE"
else
VICTIM="$ACTIVE"; SURVIVOR=$(peer_of "$ACTIVE")
fi
echo "mode=${DRILL_MODE} active=${ACTIVE} victim=${VICTIM} survivor=${SURVIVOR}"
KILL_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)
docker kill "${VICTIM}" > /dev/null
START=$(date +%s)
if [ "$DRILL_MODE" = standby ]; then
echo "Standby crash: waiting for ${SURVIVOR} to DOWN+REMOVE the dead member (SBR budget ~25s)..."
BLIPS=0
while true; do
ELAPSED=$(( $(date +%s) - START ))
curl -sf -o /dev/null "${TRAEFIK_URL}/health/active" || BLIPS=$((BLIPS + 1))
if docker logs --since "${KILL_AT}" "${SURVIVOR}" 2>&1 | grep -Eiq "marking.*node.*down|member removed|is removed"; then
echo "PASS: survivor removed the crashed member in ${ELAPSED}s (budget ~25s: 10s detection + 15s stable-after)."
echo "Active-node routing blips during the drill: ${BLIPS} (expected 0 — the active node was never touched)."
break
fi
if (( ELAPSED > TIMEOUT_S )); then
echo "FAIL: no downing/removal evidence on ${SURVIVOR} after ${ELAPSED}s — SBR did not act" >&2
docker start "${VICTIM}" > /dev/null
exit 1
fi
sleep 1
done
else
echo "Active crash: EXPECTING a central outage (registered keep-oldest gap). Watching /health/active..."
DARK_STREAK=0
while true; do
ELAPSED=$(( $(date +%s) - START ))
if curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; then DARK_STREAK=0; else DARK_STREAK=$((DARK_STREAK + 1)); fi
if (( DARK_STREAK >= 10 )); then
echo "Outage confirmed at ${ELAPSED}s: no active central node — the younger survivor self-downed"
echo "(keep-oldest downs the partition WITHOUT the oldest; this is the registered deferred gap)."
break
fi
if (( ELAPSED > OUTAGE_CONFIRM_S )); then
echo "NOTE: /health/active stayed reachable ${ELAPSED}s after killing the oldest — better than the"
echo "registered gap predicts. Do NOT celebrate: capture both nodes' logs and investigate before trusting it."
break
fi
sleep 1
done
fi
echo "Restarting ${VICTIM}..."
docker start "${VICTIM}" > /dev/null
RESTART_AT=$(date +%s)
echo "Waiting for central to be routable again through Traefik (${TRAEFIK_URL}/health/active)..."
while true; do
ELAPSED=$(( $(date +%s) - RESTART_AT ))
if curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; then
echo "Recovered: an active central node is routable ${ELAPSED}s after the victim restart."
break
fi
if (( ELAPSED > 120 )); then
echo "FAIL: central not routable 120s after restarting ${VICTIM}" >&2
exit 1
fi
sleep 1
done
echo "Survivor singleton/downing evidence (last 20 matching log lines from ${SURVIVOR}):"
docker logs "${SURVIVOR}" 2>&1 | grep -Ei "singleton|oldest|downing|removed" | tail -20 || true
echo "Drill complete (${DRILL_MODE}). Verify on the Health dashboard that both nodes show Up and exactly one is Primary."
```
3. `chmod +x docker/failover-drill.sh`. Verify: `bash -n docker/failover-drill.sh` (exit 0) and `DRILL_MODE=bogus bash docker/failover-drill.sh; echo "exit=$?"` (prints the usage error, exit=2).
4. Commit:
```
git add docker/failover-drill.sh
git commit -m "fix(docker): failover drill kills the STANDBY by default; active mode measures the registered keep-oldest outage instead of pretending recovery (plan R2-01 T1)"
```
### Task 2: Correct the recovery narrative + document the first-seed bootstrap constraint
**Classification:** trivial (doc-only)
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 4, Task 5, Task 6, Task 8, Task 9, Task 10
**Files:**
- Modify: `docker/README.md` (Failover Testing → "Automated failover drill" section, lines 273-288)
- Modify: `docs/requirements/Component-ClusterInfrastructure.md` ("Down-if-alone recovery" subsection, lines ~104-116)
1. `docker/README.md` (lines 273-288): rewrite the drill section to match Task 1's two modes. Specifically:
- Replace the four numbered bullets with per-mode descriptions: `DRILL_MODE=standby` (default) = the survivable younger-crash direction — expected result: **no routing outage at all** (the active node is never touched) and member removal on the survivor within **~25s** (10s failure-detection threshold + 15s stable-after; the 2s heartbeat interval is not additive); `DRILL_MODE=active` = the oldest-crash direction — expected result: **total central outage** until the victim container is restarted (the registered deferred keep-oldest decision, master tracker 2026-07-08), then recovery within ~2 min of the restart.
- Delete the line-286 promise "Expected failover: **~25s** … plus the Traefik ~5s health-check interval" — it described a singleton *migration* on active-crash that two-node keep-oldest cannot deliver. State per-direction expectations instead.
- Replace the line-288 "Observed failover time: _run after next deploy and record here_" placeholder with a two-row results table Task 3 will fill:
```markdown
> **Observed results** (recorded by plan R2-01 T3, cluster @ commit <sha>):
>
> | Direction (`DRILL_MODE`) | Outcome | Measured |
> |--------------------------|---------|----------|
> | `standby` (younger-node crash) | _pending T3_ | _pending T3_ |
> | `active` (oldest-node crash) | _pending T3_ | _pending T3_ |
```
- Add a short **"Seed-node bootstrap constraint"** note to the same section: only the FIRST seed in `Cluster:SeedNodes` may self-join to form a *new* cluster; both central nodes list `scadabridge-central-a` first, so a lone restarted `central-b` (with `central-a` still down) loops on `InitJoin` forever — never `Up`, never `/health/active` 200. Operator recovery actions: **(1)** restart the dead first-seed node (preferred), or **(2)** restart the survivor with a self-first seed override (env: `ScadaBridge__Cluster__SeedNodes__0=akka.tcp://scadabridge@<self-host>:8081`, `...__1=<peer>`). Explain why the repo does NOT ship self-first ordering per node: with *both* nodes self-first, a simultaneous cold start can let each self-join independently → two one-node clusters that never merge (a cold-start split-brain the identical-seed-order convention exists to prevent). The real remedy is the pending keep-oldest topology/strategy decision (deferred, owner: user).
2. `docs/requirements/Component-ClusterInfrastructure.md` (Down-if-alone recovery, ~104-116):
- Amend step 4 ("The restarted process rejoins as a **fresh incarnation** (the keep-oldest resolver handles the rejoin cleanly…)"): add the qualifier that this holds **only while a peer holding cluster state is still reachable** — a lone restarted non-first-seed node cannot re-form a cluster (first-seed self-join rule) and waits for its peer.
- Add a "Seed-node bootstrap constraint" paragraph after the numbered list (same content as the README note, spec-toned), cross-referencing the registered deferred keep-oldest decision and the operator actions.
- Fix the closing line "The docker failover drill exercises this loop end-to-end (kill the active node, assert singleton migration + clean rejoin)" → "The docker failover drill (`docker/failover-drill.sh`) exercises this per direction: `standby` mode proves SBR downing + singleton continuity on the oldest; `active` mode measures the registered total-outage gap and the recovery-on-restart path."
3. Verification (doc task — no test): `grep -n "DRILL_MODE" docker/README.md` (≥2 hits), `grep -cn "singleton migration" docs/requirements/Component-ClusterInfrastructure.md` (0 hits in the recovery section), `grep -n "first seed\|first-seed" docker/README.md docs/requirements/Component-ClusterInfrastructure.md` (≥2 hits), `grep -n "Traefik ~5s health-check interval" docker/README.md` (0 hits).
4. Commit:
```
git add docker/README.md docs/requirements/Component-ClusterInfrastructure.md
git commit -m "docs(cluster): per-direction failover truth — first-seed bootstrap constraint + operator recovery actions; drop the undeliverable ~25s active-crash promise (plan R2-01 T2)"
```
### Task 3: RUN the drill live (both directions) and record measured timings
**Classification:** standard (live-cluster verification; ~5 min of work, several minutes of wall-clock waiting)
**Estimated implement time:** ~5 min
**Parallelizable with:** none (exclusive use of the shared docker cluster; writes `docker/README.md`)
**Files:**
- Modify: `docker/README.md` (fill the Task 2 results table)
The drill has NEVER been executed (round-2 N1 point 3, `docker/README.md:288`). This task produces the empirical record — the asymmetric result *is* the documentation the deferred keep-oldest decision needs.
1. Ensure infra + cluster are up and freshly built (drill needs the Task 1 script inside the checkout only — no image change required, but the cluster must be healthy):
```
cd infra && docker compose up -d && cd ..
bash docker/deploy.sh
```
Wait until `curl -sf http://localhost:9000/health/active` returns 200 (poll; allow ~2 min for first-boot readiness).
2. Run direction 1 and capture:
```
DRILL_MODE=standby bash docker/failover-drill.sh 2>&1 | tee /private/tmp/claude-501/-Users-dohertj2-Desktop-ScadaBridge/d69633c2-b169-4fe1-8fb4-0d8007a16fa1/scratchpad/drill-standby.log
```
Expected: PASS with member-removal time ≈ 25-30s and `routing blips: 0`. Wait for the victim to fully rejoin (script does this) plus ~30s settle time.
3. Run direction 2 and capture:
```
DRILL_MODE=active bash docker/failover-drill.sh 2>&1 | tee /private/tmp/claude-501/-Users-dohertj2-Desktop-ScadaBridge/d69633c2-b169-4fe1-8fb4-0d8007a16fa1/scratchpad/drill-active.log
```
Expected: "Outage confirmed at ~Ns" (survivor self-downs; central dark), then "Recovered: … Ms after the victim restart". If instead the NOTE branch fires (`/health/active` stayed up), capture `docker logs scadabridge-central-a` + `-b` and investigate before recording — do not record a result that contradicts the registered gap without evidence.
4. Fill the `docker/README.md` results table from the two logs: outcome + measured seconds per direction, and the cluster commit SHA (`git rev-parse --short HEAD`). Keep the raw one-line summary quotes from the script output.
5. Sanity-restore: confirm both nodes healthy afterwards — `curl -sf http://localhost:9001/health/ready && curl -sf http://localhost:9002/health/ready` (both 200; the standby's `/health/active` correctly 503).
6. Commit:
```
git add docker/README.md
git commit -m "test(docker): record live failover-drill results for both crash directions — standby-crash PASS timing + active-crash measured outage (plan R2-01 T3)"
```
### Task 4: Wire `FailoverTimingTests` to `TwoNodeClusterFixture` — measure the ~25s envelope in-process
**Classification:** high-risk (real two-node cluster timing test)
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 1, Task 2, Task 5, Task 6, Task 7, Task 8, Task 9, Task 10, Task 11
**Files:**
- Modify: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/TwoNodeClusterFixture.cs` (StartAsync lines 24-36, StartNode lines 39-65 — add production-timing knobs)
- Modify: `tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/ZB.MOM.WW.ScadaBridge.PerformanceTests.csproj` (add a ProjectReference)
- Modify: `tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Failover/FailoverTimingTests.cs` (replace the `Skip` placeholder — currently `[Fact(Skip = "Requires the real two-node failover rig delivered by PLAN-01 …")]` at lines 25-32)
This task **also covers architecture-review report 08's NF2 by reference** (the skipped perf placeholder): the rig it waits for exists (`TwoNodeClusterFixture`, PLAN-01 T3); wire it up. The fixture hard-codes scaled timings (`HeartbeatInterval` 500ms / `FailureDetectionThreshold` 2s at `TwoNodeClusterFixture.cs:51-52`); to measure the *production* ~25s envelope (2s heartbeat / 10s threshold / 15s stable-after), the knobs become optional parameters with the current values as defaults (additive — `SbrFailoverTests`/`ActiveNodeSemanticsTests` unaffected). Direction note: the measurable crash is the YOUNGER node (the survivable direction); the oldest-crash direction is the registered deferred gap and is *not* a recovery to time — say so in the test's XML doc.
1. Rewrite `FailoverTimingTests.cs` (this is the failing step — it will not compile until the fixture knobs + project reference exist):
```csharp
using System.Diagnostics;
using Akka.Actor;
using Akka.Cluster.Tools.Singleton;
using Xunit.Abstractions;
using ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;
namespace ZB.MOM.WW.ScadaBridge.PerformanceTests.Failover;
/// <summary>
/// Failover-timing measurement on the real two-node in-process rig
/// (TwoNodeClusterFixture, production BuildHocon) at PRODUCTION timings:
/// 2s heartbeat / 10s failure-detection threshold / 15s SBR stable-after —
/// the CLAUDE.md "total failover ~25s" design envelope.
///
/// Measures the SURVIVABLE direction only: hard-crash of the YOUNGER node,
/// timed to the survivor's member REMOVAL (detection + stable-after + gossip)
/// with singleton continuity asserted on the oldest. The oldest/active-node
/// crash is NOT a recovery to time — two-node keep-oldest makes the younger
/// survivor down itself (total outage; registered deferred user decision,
/// see SbrFailoverTests XML doc + master tracker 2026-07-08). Covers overall
/// review P2-10 / report-08 NF2 and report-01 round-2 N1's measurement ask.
/// </summary>
public class FailoverTimingTests(ITestOutputHelper output)
{
private sealed class EchoActor : ReceiveActor
{
public EchoActor() => ReceiveAny(msg => Sender.Tell(msg));
}
[Trait("Category", "Performance")]
[Fact]
public async Task HardKillOfYoungerNode_SbrRemovalWithinDesignEnvelope()
{
await using var cluster = await TwoNodeClusterFixture.StartAsync(
stableAfter: TimeSpan.FromSeconds(15),
heartbeatInterval: TimeSpan.FromSeconds(2),
failureDetectionThreshold: TimeSpan.FromSeconds(10));
// Singleton on the oldest (NodeA) — the continuity probe.
var manager = cluster.NodeA.ActorOf(ClusterSingletonManager.Props(
Props.Create(() => new EchoActor()),
PoisonPill.Instance,
ClusterSingletonManagerSettings.Create(cluster.NodeA).WithSingletonName("timing-probe")),
"timing-probe-singleton");
var proxyA = cluster.NodeA.ActorOf(ClusterSingletonProxy.Props(
"/user/timing-probe-singleton",
ClusterSingletonProxySettings.Create(cluster.NodeA).WithSingletonName("timing-probe")),
"timing-probe-proxy");
Assert.Equal("ping", await proxyA.Ask<string>("ping", TimeSpan.FromSeconds(30)));
var victim = Akka.Cluster.Cluster.Get(cluster.NodeB).SelfAddress;
var sw = Stopwatch.StartNew();
await TwoNodeClusterFixture.CrashNode(cluster.NodeB);
await TwoNodeClusterFixture.WaitForMemberRemoved(cluster.NodeA, victim, TimeSpan.FromSeconds(60));
sw.Stop();
output.WriteLine(
$"MEASURED: crash -> member removed on survivor: {sw.Elapsed.TotalSeconds:F1}s " +
"(design ~25s = 10s detection + 15s stable-after; +gossip margin)");
// Design envelope: >= stable-after (SBR must not act early), <= 25s design + 15s margin.
Assert.InRange(sw.Elapsed, TimeSpan.FromSeconds(15), TimeSpan.FromSeconds(40));
// Singleton continuity on the surviving oldest node.
Assert.Equal("ping-after-crash",
await proxyA.Ask<string>("ping-after-crash", TimeSpan.FromSeconds(10)));
}
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.PerformanceTests --filter "FullyQualifiedName~FailoverTimingTests"` — expect FAIL (compile error: `TwoNodeClusterFixture` unknown; no `heartbeatInterval` parameter).
3. Implement:
- `TwoNodeClusterFixture.StartAsync` (line 24): signature → `StartAsync(string role = "Central", TimeSpan? stableAfter = null, TimeSpan? heartbeatInterval = null, TimeSpan? failureDetectionThreshold = null)`; pass both new args through the two `f.StartNode(...)` calls.
- `TwoNodeClusterFixture.StartNode` (line 39): signature → `StartNode(int port, string role, TimeSpan? stableAfter = null, TimeSpan? heartbeatInterval = null, TimeSpan? failureDetectionThreshold = null)`; lines 51-52 become `HeartbeatInterval = heartbeatInterval ?? TimeSpan.FromMilliseconds(500),` / `FailureDetectionThreshold = failureDetectionThreshold ?? TimeSpan.FromSeconds(2),`. Extend the class XML doc: "Timing knobs default to scaled test values; pass production values (2s/10s/15s) to measure the real failover envelope (FailoverTimingTests)."
- `ZB.MOM.WW.ScadaBridge.PerformanceTests.csproj`: add to the existing `<ItemGroup>` of ProjectReferences:
```xml
<!-- Two-node failover rig (TwoNodeClusterFixture) — brings Host + Akka transitively. -->
<ProjectReference Include="../../tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/ZB.MOM.WW.ScadaBridge.IntegrationTests.csproj" />
```
4. Run again: `dotnet test tests/ZB.MOM.WW.ScadaBridge.PerformanceTests --filter "FullyQualifiedName~FailoverTimingTests"` — expect PASS in ~45-90s; the test output MUST print the `MEASURED:` line — quote the number in the commit message. Regression: `dotnet test tests/ZB.MOM.WW.ScadaBridge.IntegrationTests --filter "FullyQualifiedName~SbrFailoverTests|FullyQualifiedName~ActiveNodeSemanticsTests|FullyQualifiedName~TwoNodeClusterFixtureTests"` — all green (fixture change is additive).
5. Commit:
```
git add tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/TwoNodeClusterFixture.cs tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/ZB.MOM.WW.ScadaBridge.PerformanceTests.csproj tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Failover/FailoverTimingTests.cs
git commit -m "test(failover): unskip FailoverTimingTests on the two-node rig at production timings — measured Xs vs ~25s design envelope (plan R2-01 T4; covers report-08 NF2)"
```
### Task 5: Apply the wonder-app-vd03 appsettings overlay edits (NodeName + AllowSingleNodeCluster, drop phantom seeds)
**Classification:** trivial (config-only; NO git commit — see note)
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 1, Task 2, Task 4, Task 6, Task 8, Task 9, Task 10, Task 11
**Files:**
- Modify: `deploy/wonder-app-vd03/appsettings.Central.json` (`Node` lines 3-7, `Cluster` lines 8-18, `_comment_Cluster` line 19)
- Modify: `deploy/wonder-app-vd03/appsettings.Site.json` (`Node` lines 3-9, `Cluster` lines 10-20, `_comment_Cluster` line 21)
Round-2 N2: the round-1 deferral claimed the artifact "is not tracked in this repo … the artifact directory itself is not present" — **the directory IS present** (merely gitignored, `.gitignore:48: /deploy/`) and has been actively maintained in this working tree (its `_comment_Security` documents the PLAN-07 edit). Apply the two appsettings edits from PLAN-01 Tasks 16/23 directly. Without `NodeName`, every audit row from the only production install stamps a NULL `SourceNode` (defeats `IX_AuditLog_Node_Occurred` + the per-node stuck KPIs); with the phantom seeds, each node dials a nonexistent peer (`localhost:8091` / `localhost:8092`) forever.
1. `appsettings.Central.json`:
- `Node` (after `"Role": "Central",`): add `"NodeName": "central-a",` (matches the docker convention, `docker/central-node-a/appsettings.Central.json`).
- `Cluster`: drop the phantom seed `"akka.tcp://scadabridge@localhost:8091"` (keep only the `:8081` self seed) and add `"AllowSingleNodeCluster": true` (flag shipped in PLAN-01 T16: `ClusterOptions.cs:86`, validator `ClusterOptionsValidator.cs:29-35` requires it for a 1-seed list).
- `_comment_Cluster` → `"Single-node central install, explicitly acknowledged via AllowSingleNodeCluster (arch-review 01 S12/N2): the previous phantom second seed (localhost:8091) made the node dial a nonexistent peer forever. When a real central node B is added: list BOTH real seeds and remove AllowSingleNodeCluster."`
2. `appsettings.Site.json`: same treatment — `Node` gains `"NodeName": "node-a",`; `Cluster.SeedNodes` keeps only `"akka.tcp://scadabridge@localhost:8082"`; add `"AllowSingleNodeCluster": true`; `_comment_Cluster` rewritten equivalently (PRESERVE the existing "Seed ports must never equal GrpcPort (8083)" sentence).
3. Verify (this is the "test" — config task):
```
python3 -m json.tool deploy/wonder-app-vd03/appsettings.Central.json > /dev/null && \
python3 -m json.tool deploy/wonder-app-vd03/appsettings.Site.json > /dev/null && \
grep -c "NodeName\|AllowSingleNodeCluster" deploy/wonder-app-vd03/appsettings.Central.json deploy/wonder-app-vd03/appsettings.Site.json && \
grep -c "8091\|8092" deploy/wonder-app-vd03/appsettings.Central.json deploy/wonder-app-vd03/appsettings.Site.json
```
Expect: both json.tool exits 0; each file counts 2 for the first grep; each file counts **0** for the phantom-port grep (grep -c exits 1 on zero matches — that is the pass signal here).
4. **No git commit** — `/deploy/` is gitignored (`.gitignore:48`); do NOT use `git add -f`. The edit is intentionally working-tree/on-host only; Task 7 commits the version-controlled record, and the change takes effect on the host when the artifact is next synced + services restarted (RUNBOOK step added in Task 6).
### Task 6: Complete the install.ps1 recovery actions (`sc.exe failureflag`) + RUNBOOK recovery step
**Classification:** trivial (script/doc-only; NO git commit — see Task 5 note)
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 1, Task 2, Task 4, Task 5, Task 8, Task 9, Task 10, Task 11
**Files:**
- Modify: `deploy/wonder-app-vd03/install.ps1` (section `# --- 7. Auto-restart on failure`, lines ~171-174)
- Modify: `deploy/wonder-app-vd03/RUNBOOK.md` (troubleshooting table, ~line 218 area)
Finding-vs-disk note (record this honestly): the round-2 report says the `sc.exe failure` recovery actions are missing, but the on-disk `install.ps1` already has them (lines 171-174, all three services) — what is missing is `sc.exe failureflag <svc> 1`. Without the failure-actions flag, the SCM runs recovery actions only after a **crash**; the down-if-alone watchdog's *deliberate* process exit (`IHostApplicationLifetime.StopApplication()` after an SBR self-down — `AkkaHostedService.cs:203-217`) is a non-crash stop and would never be restarted, breaking the recovery contract (`Component-ClusterInfrastructure.md` step 3: "The service supervisor restarts it"). This is the exact second line PLAN-01 Task 20 specified.
1. In `install.ps1`, extend the existing loop:
```powershell
# --- 7. Auto-restart on failure ------------------------------------------
foreach ($svc in 'ScadaBridge-LDAP', 'ScadaBridge-Central', 'ScadaBridge-Site') {
sc.exe failure $svc reset= 86400 actions= restart/5000/restart/5000/restart/10000 | Out-Null
# Required for the SBR down-if-alone recovery contract
# (Component-ClusterInfrastructure.md "Down-if-alone recovery"): without
# failureflag=1 the SCM applies the recovery actions only after a CRASH;
# the Host watchdog's deliberate process exit after a self-down is a
# normal (non-crash) stop and would otherwise never be restarted.
sc.exe failureflag $svc 1 | Out-Null
}
```
2. `RUNBOOK.md`: add a troubleshooting-table row: `| Service stopped on its own with log line "ActorSystem terminated outside host shutdown" | SBR self-down (down-if-alone). The service recovery actions restart it automatically (requires install.ps1 with sc.exe failureflag — re-run install.ps1 or run the two sc.exe lines from §7 manually on older installs). If it does not come back: Start-Service manually and check network reachability between nodes. |` Also note next to the "auto-restart on failure" mention (line 87) that failureflag is part of the contract.
3. Verify: `grep -n "failureflag" deploy/wonder-app-vd03/install.ps1` (3 effective invocations via the loop → 1 hit inside it) and `grep -n "failureflag\|self-down" deploy/wonder-app-vd03/RUNBOOK.md` (≥1 hit). If `pwsh` is installed, syntax-check: `pwsh -NoProfile -Command "[void][System.Management.Automation.PSParser]::Tokenize((Get-Content -Raw deploy/wonder-app-vd03/install.ps1), [ref]$null)"` (exit 0); otherwise the greps suffice.
4. **No git commit** (gitignored artifact — same rule as Task 5).
### Task 7: Correct the factually wrong N2 deferral record in the master tracker
**Classification:** trivial (doc-only)
**Estimated implement time:** ~3 min
**Parallelizable with:** Task 1, Task 2, Task 4, Task 8, Task 9, Task 10, Task 11
**Files:**
- Modify: `archreview/plans/00-MASTER-TRACKER.md` (line 28 trailing sentence; "Deferred during Wave 1 PLAN-01 execution (2026-07-08)" bullet at lines 145-147)
The deferral rationale — "that production deploy artifact is not tracked in this repo … the artifact directory itself is not present in the repo" — is factually wrong (round-2 N2): `deploy/wonder-app-vd03/` exists in the working tree, is merely gitignored (`.gitignore:48: /deploy/`), and was edited during this initiative (PLAN-07's `_comment_Security`). Depends on Tasks 5-6 having applied the edits.
1. Rewrite the line-147 bullet: keep the history (what was deferred and why it mattered), then correct the record: the directory **is present but gitignored** (production config kept out of version control — the *tracking* claim was wrong, the *presence* claim was false); the three overlay edits were **applied on-disk 2026-07-12 by PLAN-R2-01 Tasks 5-6** (`NodeName` central-a/node-a, `AllowSingleNodeCluster` + phantom seeds dropped, `sc.exe failureflag` completing the already-present `sc.exe failure` actions). Remaining owner action: sync the artifact to `wonder-app-vd03` and restart services (appsettings take effect on restart; failureflag on re-running install.ps1 §7 or the two `sc.exe` lines manually) — per the RUNBOOK row added by T6.
2. Trim the stale trailing sentence of the line-28 Wave-1 summary ("were deferred because that production artifact is not tracked in this repo") → "were deferred on a mistaken not-in-repo rationale; corrected and applied on-disk by PLAN-R2-01 (2026-07-12) — see the Deferred-during-Wave-1 entry."
3. Do NOT add rows to `docs/plans/2026-07-08-deferred-work-register.md` — register consolidation (round-2 N6) is owned by **PLAN-R2-08**; this task only stops the tracker from asserting a falsehood.
4. Verify: `grep -n "not tracked in this repo" archreview/plans/00-MASTER-TRACKER.md` → 0 hits; `grep -n "PLAN-R2-01" archreview/plans/00-MASTER-TRACKER.md` → ≥2 hits.
5. Commit:
```
git add archreview/plans/00-MASTER-TRACKER.md
git commit -m "docs(tracker): correct the wonder-app-vd03 deferral record — artifact is present (gitignored); overlay edits applied on-disk by R2-01 T5/T6 (plan R2-01 T7)"
```
### Task 8: Metrics-staleness for never-reported sites (`FirstSeenAt` anchor)
**Classification:** high-risk (lock-free CAS state machine)
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 1, Task 2, Task 4, Task 5, Task 6, Task 7, Task 9, Task 10, Task 11
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthState.cs` (add `FirstSeenAt`)
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthAggregator.cs` (register branches lines 44-54 and 114-122; staleness guard lines 300-312)
- Modify: `docs/requirements/Component-HealthMonitoring.md` ("Metrics staleness" bullet, line ~47)
- Test: `tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/CentralHealthAggregatorTests.cs`
Round-2 N3: `CheckForOfflineSites` guards staleness with `state.LastReportReceivedAt is { } lastReport` (`CentralHealthAggregator.cs:300`), so a site registered via heartbeat only (`MarkHeartbeat` registration leaves `LastReportReceivedAt` null, `:114-122`) is NEVER flagged stale — "stuck from first boot" shows online-with-no-metrics forever. `LastHeartbeatAt` cannot be the anchor (heartbeats keep advancing it); anchor on a new `FirstSeenAt` stamped at registration. No UI change needed: the badge (`Health.razor:209-215`) keys on `IsOnline && IsMetricsStale` regardless of `LatestReport`.
1. Write the failing tests (append to `CentralHealthAggregatorTests` — reuse the existing `NewAggregator(metricsStaleTimeout:)` and `TestTimeProvider.Advance` helpers, lines 48-59/22):
```csharp
/// <summary>
/// Round-2 N3: a site known only via heartbeats (LastReportReceivedAt == null)
/// previously skipped the staleness check forever — a report pipeline dead
/// from FIRST BOOT showed "online with no metrics" indefinitely. FirstSeenAt
/// anchors the window instead (LastHeartbeatAt cannot: heartbeats advance it).
/// </summary>
[Fact]
public void HeartbeatOnlySite_NeverReported_IsFlaggedMetricsStale()
{
var (aggregator, time) = NewAggregator(metricsStaleTimeout: TimeSpan.FromMinutes(2));
aggregator.MarkHeartbeat("site-a", time.GetUtcNow()); // registered via heartbeat only
time.Advance(TimeSpan.FromMinutes(3));
aggregator.MarkHeartbeat("site-a", time.GetUtcNow()); // still heartbeating => online
aggregator.CheckForOfflineSites();
var state = aggregator.GetSiteState("site-a")!;
Assert.True(state.IsOnline); // liveness unchanged
Assert.True(state.IsMetricsStale); // was impossible before the fix
}
/// <summary>A heartbeat-only site inside the window is NOT falsely flagged.</summary>
[Fact]
public void HeartbeatOnlySite_WithinStaleWindow_IsNotFlagged()
{
var (aggregator, time) = NewAggregator(metricsStaleTimeout: TimeSpan.FromMinutes(2));
aggregator.MarkHeartbeat("site-a", time.GetUtcNow());
time.Advance(TimeSpan.FromMinutes(1));
aggregator.MarkHeartbeat("site-a", time.GetUtcNow());
aggregator.CheckForOfflineSites();
Assert.False(aggregator.GetSiteState("site-a")!.IsMetricsStale);
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests --filter "FullyQualifiedName~HeartbeatOnlySite"` — expect the first test to FAIL (`IsMetricsStale` stays false; the null guard skips the site).
3. Implement:
- `SiteHealthState`: add
```csharp
/// <summary>
/// Gets the instant this site was first observed by the aggregator (via a
/// report or a heartbeat); <c>null</c> only for states not built by the
/// aggregator (hand-constructed fixtures). Anchors metrics-staleness for a
/// site that has NEVER delivered a report (review 01 round-2 N3): with
/// <see cref="LastReportReceivedAt"/> null the staleness check previously
/// skipped the site forever. <see cref="LastHeartbeatAt"/> cannot anchor
/// this — every heartbeat advances it.
/// </summary>
public DateTimeOffset? FirstSeenAt { get; init; }
```
- `CentralHealthAggregator.ProcessReport` register branch (lines 44-54): add `FirstSeenAt = now,`. `MarkHeartbeat` register branch (lines 114-122): add `FirstSeenAt = receivedAt,`. (Update branches never touch it — `with` copies preserve it.)
- `CheckForOfflineSites` staleness guard (lines 300-312): replace the `state.LastReportReceivedAt is { } lastReport` condition with an anchor fallback:
```csharp
// Second signal: … (keep the existing comment, append:)
// A site that has NEVER reported (heartbeat-only registration,
// LastReportReceivedAt == null) anchors on FirstSeenAt instead —
// otherwise a pipeline dead from first boot is never flagged (N3).
var reportAnchor = state.LastReportReceivedAt ?? state.FirstSeenAt;
if (reportAnchor is { } anchor
&& now - anchor > _options.MetricsStaleTimeout
&& !state.IsMetricsStale)
{
var stale = state with { IsMetricsStale = true };
if (_siteStates.TryUpdate(kvp.Key, stale, state))
{
_logger.LogWarning(
"Site {SiteId} metrics are stale — online (heartbeats) but no report for {Elapsed}s{NeverNote} (timeout: {Timeout}s)",
state.SiteId, (now - anchor).TotalSeconds,
state.LastReportReceivedAt is null ? " (never reported since first contact)" : string.Empty,
_options.MetricsStaleTimeout.TotalSeconds);
}
// CAS loss ⇒ a fresh report/heartbeat swapped in ⇒ correct to skip.
}
```
- `Component-HealthMonitoring.md` "Metrics staleness" bullet (line ~47): append — "A site that has **never** delivered a report is anchored on its first-contact time (`FirstSeenAt`), so a report pipeline dead from first boot is also flagged after `MetricsStaleTimeout` (previously such a site showed online-with-no-metrics indefinitely)."
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests` — full project green (the CAS/flap/stale suites are the regression net).
5. Commit:
```
git add src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthState.cs src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthAggregator.cs docs/requirements/Component-HealthMonitoring.md tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/CentralHealthAggregatorTests.cs
git commit -m "fix(health): flag metrics-stale for never-reported sites via FirstSeenAt anchor — null LastReportReceivedAt no longer skips the check (plan R2-01 T8)"
```
### Task 9: Purge the stale "cluster leader" narration from `CentralHealthReportLoop`
**Classification:** trivial (comment-only)
**Estimated implement time:** ~2 min
**Parallelizable with:** Task 1, Task 2, Task 4, Task 5, Task 6, Task 7, Task 8, Task 10, Task 11
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthReportLoop.cs` (class doc line 12-14; sequence-seed comment lines 44-45)
Round-2 N4: the behavior is correct (line 93 gates on `IClusterNodeProvider.SelfIsPrimary` = oldest Up member since PLAN-01 T6), but the comments still narrate "cluster leader (Primary)" — re-teaching the exact leader/oldest conflation round-1 S2 eradicated.
1. Confirm scope first: `grep -in "leader" src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthReportLoop.cs` — expect exactly lines 12, 44, 45.
2. Edit:
- Class doc (lines 12-14): "Only the cluster leader (Primary) generates reports — the standby's aggregator catches up on failover when it becomes Primary and starts its own loop." → "Only the active node (Primary = the **oldest Up member**, i.e. the singleton host — `IClusterNodeProvider.SelfIsPrimary`, never Akka cluster leadership) generates reports; the standby's aggregator catches up on failover when it becomes the oldest and its loop starts reporting."
- Sequence-seed comment (lines 44-45): "reports from a newly-elected central leader always sort after reports from any prior leader" → "reports from a node that has newly become active (oldest Up member) always sort after reports from any previously-active node".
3. Verify: `grep -in "leader" src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthReportLoop.cs` — **0 hits**. Comment-only change: `dotnet build src/ZB.MOM.WW.ScadaBridge.HealthMonitoring` — succeeds.
4. Commit:
```
git add src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthReportLoop.cs
git commit -m "docs(health): CentralHealthReportLoop comments say oldest-member, not cluster leader — stop re-teaching the S2 conflation (plan R2-01 T9)"
```
### Task 10: Generalize the registrar — `SingletonRegistrar` with an optional role scope
**Classification:** high-risk (actor lifecycle helper used by all central singletons)
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 1, Task 2, Task 4, Task 5, Task 6, Task 7, Task 8, Task 9
**Files:**
- Modify (rename): `src/ZB.MOM.WW.ScadaBridge.Host/Actors/CentralSingletonRegistrar.cs` → `src/ZB.MOM.WW.ScadaBridge.Host/Actors/SingletonRegistrar.cs` (use `git mv`)
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs` (rename at the block comment lines 459-465 and the 7 central call sites: lines 466, 499, 564, 609, 632, 663, 689)
- Modify (rename): `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CentralSingletonRegistrarTests.cs` → `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SingletonRegistrarTests.cs`
- Modify: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CoordinatedShutdownTests.cs` (line 56 source-grep string)
Round-2 N5 first half: `CentralSingletonRegistrar` cannot express `.WithRole(siteRole)`, which is the sole reason the two site singletons stay hand-rolled without drains (`AkkaHostedService.cs:464-465`). Add an optional `role` applied to BOTH the manager and proxy settings, and rename the class — "Central" becomes a lie the moment it hosts site singletons.
1. Write the failing test — in the (renamed) `SingletonRegistrarTests.cs`, rename the class + existing references `CentralSingletonRegistrar` → `SingletonRegistrar`, and add:
```csharp
[Fact]
public async Task Start_WithRoleScope_CreatesRoleScopedSingleton_ThatAnswers()
{
// Round-2 N5: role scoping is what kept the two SITE singletons
// (deployment-manager, event-log-handler) hand-rolled and drain-less.
using var system = CreateSingleNodeClusterSystem(); // roles = ["Central"]
var handle = SingletonRegistrar.Start(
system, "role-widget", Props.Create(() => new EchoActor()),
NullLogger.Instance, drainTimeout: TimeSpan.FromSeconds(2), role: "Central");
Assert.EndsWith("/user/role-widget-singleton", handle.Manager.Path.ToString());
Assert.EndsWith("/user/role-widget-proxy", handle.Proxy.Path.ToString());
// The node holds the role => manager instantiates the singleton and
// the role-scoped proxy resolves it.
var echo = await handle.Proxy.Ask<string>("hi", TimeSpan.FromSeconds(20));
Assert.Equal("hi", echo);
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~SingletonRegistrarTests"` — expect FAIL (compile: no `SingletonRegistrar` type / no `role` parameter).
3. Implement:
- `git mv src/ZB.MOM.WW.ScadaBridge.Host/Actors/CentralSingletonRegistrar.cs src/ZB.MOM.WW.ScadaBridge.Host/Actors/SingletonRegistrar.cs`; rename the class; signature → `Start(ActorSystem system, string name, Props singletonProps, ILogger logger, TimeSpan? drainTimeout = null, string? role = null)`; apply the role to both settings:
```csharp
var managerSettings = ClusterSingletonManagerSettings.Create(system).WithSingletonName(name);
if (role is not null) managerSettings = managerSettings.WithRole(role);
// … ClusterSingletonManager.Props(singletonProps, PoisonPill.Instance, managerSettings) …
var proxySettings = ClusterSingletonProxySettings.Create(system).WithSingletonName(name);
if (role is not null) proxySettings = proxySettings.WithRole(role);
// … ClusterSingletonProxy.Props($"/user/{name}-singleton", proxySettings) …
```
Update the class XML doc: drop "central"-only framing; drain rationale now reads "so in-flight EF (central) or SQLite (site) work completes before handover"; note the optional `role` maps to `ClusterSingletonManagerSettings/ProxySettings.WithRole` (review 01 round-2 N5).
- `AkkaHostedService.cs`: mechanical rename at the 7 call sites (466, 499, 564, 609, 632, 663, 689) and rewrite the lines-459-465 comment's last sentence ("The two SITE singletons stay hand-rolled below because they are role-scoped") → "Site singletons use the same registrar with `role: siteRole` (Task 11 / round-2 N5)." (Task 11 lands that; wording here is forward-referencing but true within the same plan.)
- `git mv tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CentralSingletonRegistrarTests.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SingletonRegistrarTests.cs`.
- `CoordinatedShutdownTests.cs` line 56: `Assert.Contains("CentralSingletonRegistrar.Start(", content);` → `Assert.Contains("SingletonRegistrar.Start(", content);` (also update the comment above it).
- Sanity: `grep -rn "CentralSingletonRegistrar" src/ tests/ --include="*.cs"` — 0 hits.
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~SingletonRegistrarTests|FullyQualifiedName~CoordinatedShutdownTests"` — PASS. Then `dotnet build ZB.MOM.WW.ScadaBridge.slnx` — clean.
5. Commit:
```
git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/SingletonRegistrar.cs src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SingletonRegistrarTests.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CoordinatedShutdownTests.cs
git rm --cached src/ZB.MOM.WW.ScadaBridge.Host/Actors/CentralSingletonRegistrar.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CentralSingletonRegistrarTests.cs 2>/dev/null || true
git commit -m "refactor(host): CentralSingletonRegistrar -> SingletonRegistrar with optional role scope — unblocks site-singleton drains (plan R2-01 T10)"
```
(`git mv` already stages the renames; the `git rm --cached … || true` line is a no-op safety net if the moves were done as delete+create.)
### Task 11: Route the two site singletons through the registrar — deployment-manager + event-log-handler gain drains
**Classification:** high-risk (site actor wiring refactor)
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 1, Task 2, Task 4, Task 5, Task 6, Task 7, Task 8, Task 9
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs` (site branch: deployment-manager block lines ~791-820; event-log-handler block lines ~836-856)
- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CoordinatedShutdownTests.cs`
Round-2 N5 second half: `deployment-manager` and `event-log-handler` are hand-rolled with bare `PoisonPill` termination and no `PhaseClusterLeave` drain. The round-1 S5 rationale applies identically on the site side: the DeploymentManager's SQLite writes (static overrides, `native_alarm_state`) should drain before handover — currently safe-but-lossy on graceful failover.
1. Write the failing test — in `CoordinatedShutdownTests.cs`, update the existing `AllCentralSingletons_RegisterThroughRegistrarWithDrain` (lines 45-70): change the final assertion `Assert.Equal(2, CountOccurrences(content, "ClusterSingletonManager.Props("));` → `Assert.Equal(0, …)` and replace its "the only two left are the SITE singletons" comment; then add:
```csharp
[Fact]
public void SiteSingletons_RegisterThroughRegistrarWithDrain()
{
var hostProjectDir = FindHostProjectDirectory();
Assert.NotNull(hostProjectDir);
var content = File.ReadAllText(Path.Combine(hostProjectDir!, "Actors", "AkkaHostedService.cs"));
// Round-2 N5: the two role-scoped SITE singletons previously stayed
// hand-rolled (bare PoisonPill, no PhaseClusterLeave drain). They now
// go through SingletonRegistrar.Start(..., role: siteRole), which
// always adds the GracefulStop drain — in-flight SQLite writes
// (static overrides, native_alarm_state) complete before handover.
Assert.Contains("\"deployment-manager\"", content);
Assert.Contains("\"event-log-handler\"", content);
Assert.Contains("role: siteRole", content);
Assert.Equal(0, CountOccurrences(content, "ClusterSingletonManager.Props("));
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~SiteSingletons_RegisterThroughRegistrarWithDrain"` — expect FAIL (`ClusterSingletonManager.Props(` count is 2; no `role: siteRole`).
3. Implement in the site branch of `AkkaHostedService.cs`:
- Deployment-manager block (~791-820): replace the hand-rolled `ClusterSingletonManager.Props`/`ClusterSingletonProxy.Props` pair with
```csharp
// Deployment Manager — role-scoped singleton via SingletonRegistrar
// (review 01 round-2 N5): previously hand-rolled with bare PoisonPill
// termination and NO PhaseClusterLeave drain, so in-flight SQLite
// writes (static overrides, native_alarm_state) could be cut off on
// graceful failover. Names unchanged: deployment-manager-singleton /
// deployment-manager-proxy.
var dm = SingletonRegistrar.Start(
_actorSystem!, "deployment-manager",
Props.Create(() => new DeploymentManagerActor(
storage, compilationService, sharedScriptLibrary, streamManager,
siteRuntimeOptionsValue, dmLogger, dclManager, replicationActor,
siteHealthCollector, _serviceProvider, null, deploymentConfigFetcher)),
_logger, role: siteRole);
var dmProxy = dm.Proxy;
```
(Keep the `dmProxy` variable name — everything downstream, `SiteCommunicationActor` construction and `RegisterLocalHandler(LocalHandlerType.Artifacts, dmProxy)`, is untouched.)
- Event-log-handler block (~836-856): inside the existing `if (eventLogQueryService != null)`:
```csharp
var eventLog = SingletonRegistrar.Start(
_actorSystem, "event-log-handler",
Props.Create(() => new SiteEventLogging.EventLogHandlerActor(eventLogQueryService)),
_logger, role: siteRole);
siteCommActor.Tell(new RegisterLocalHandler(LocalHandlerType.EventLog, eventLog.Proxy));
```
(Keep the existing "Event log handler — cluster singleton so queries always reach the active node…" comment above it.)
- Actor/singleton names are reproduced exactly by the registrar (`deployment-manager-singleton`/`-proxy`, `event-log-handler-singleton`/`-proxy`), so path-pinning tests and the ClusterClient receptionist wiring see no change.
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests` (full project — path/health-check/audit-wiring suites are the regression net) and `dotnet build ZB.MOM.WW.ScadaBridge.slnx` — PASS/clean.
5. Commit:
```
git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CoordinatedShutdownTests.cs
git commit -m "fix(host): site singletons (deployment-manager, event-log-handler) via SingletonRegistrar — PhaseClusterLeave drains for site SQLite writes (plan R2-01 T11)"
```
---
## Dependencies on other plans
- **PLAN-R2-02 owns** the oldest-vs-leader predicate unification inside `SiteCommunicationActor`/`SiteReplicationActor` and everything resync-related. This plan does not touch those files; the coverage table references it.
- **PLAN-R2-08 owns** N6 (folding the two live PLAN-01 deferrals — the keep-oldest active-crash gap and, post-T5/T6, the "sync the artifact to the host" residue — into `docs/plans/2026-07-08-deferred-work-register.md`). Task 7 here only corrects the factually wrong tracker prose; it adds no register rows.
- The **SBR keep-oldest active-crash availability gap stays a registered deferred USER decision** — no task here changes the strategy/topology; Tasks 1-3 make the gap observable and measured, which is the input that decision needs.
## Execution order
**Drill spine (serial):** Task 1 (script) → Task 2 (docs) → Task 3 (live run, needs the docker cluster exclusively).
**Registrar spine (serial):** Task 10 → Task 11.
**Freely parallel at any point:** Tasks 4, 5, 6, 8, 9 (all file-disjoint); Task 7 after 5+6.
**Highest value if time-boxed:** Task 3 (the never-run drill, run at last) + Task 11 (real durability gap) + Task 8 (real monitoring blind spot).
---
## Findings Coverage
Every round-2 finding (N1-N6), the round-1 partials the report says warrant completion, and the cross-plan rulings:
| # | Round-2 finding | Severity | Task(s) / Disposition |
|---|----------------|----------|----------------------|
| N1 | Failover drill kills the wrong (active/oldest) node; first-seed bootstrap gap means the survivor's recovery story doesn't converge; drill never executed; ~25s promise undeliverable in its most likely case | High | Tasks 1 (drill rewrite: standby-victim default + explicit active-gap mode), 2 (narrative + first-seed constraint **documented** with operator actions — deliberately NOT reordered to self-first seeds: simultaneous cold-start self-joins would risk two independent clusters), 3 (RUN the drill live, both directions, record measured timings), 4 (wire/unskip `FailoverTimingTests` to `TwoNodeClusterFixture`, measure the ~25s envelope at production timings — **also covers report-08 NF2 by reference**). The keep-oldest topology/strategy remedy itself **stays deferred: registered USER decision** (master tracker 2026-07-08) — Tasks 1-3 document/measure it, never fix it |
| N2 | wonder-app-vd03 overlay edits unapplied; deferral rationale factually wrong (artifact IS in the working tree, merely gitignored) | Medium | Tasks 5 (NodeName + AllowSingleNodeCluster + drop phantom seeds, both overlays), 6 (`sc.exe failureflag` completing the recovery actions + RUNBOOK step — note: `sc.exe failure` itself is already on disk, contra the report), 7 (correct the deferral record in the master tracker). No git commits for the gitignored artifact itself |
| N3 | `IsMetricsStale` can never fire for a site that has never delivered a report (null `LastReportReceivedAt` guard) | Low | Task 8 (`FirstSeenAt` anchor; no UI change needed — badge already keys on the flag) |
| N4 | `CentralHealthReportLoop` still narrates "cluster leader (Primary)" | Low | Task 9 |
| N5 | Site-role singletons lack drain tasks; registrar can't express role scoping | Low | Tasks 10 (role param + rename to `SingletonRegistrar`), 11 (route `deployment-manager` + `event-log-handler` through it) |
| N6 | PLAN-01's two live deferrals missing from the canonical deferred-work register | Low | **Reference → PLAN-R2-08** (register consolidation is its scope). Task 7 here only fixes the falsehood in the master-tracker prose |
| S12 (R1 partial) | Single-node artifact still ships the phantom seed, lacks `AllowSingleNodeCluster` | — | Task 5 (the artifact half; the flag/validator half shipped in PLAN-01 T16) |
| U2 (R1 partial) | install.ps1 recovery actions deferred; recovery contract doesn't converge for a non-first-seed survivor | — | Task 6 (failureflag) + Task 2 (first-seed constraint + operator actions in the spec'd recovery contract) |
| U5 (R1 not fixed) | `NodeName` missing from wonder-app-vd03 overlays → NULL `SourceNode` | — | Task 5 |
| C3 (R1 partial) | LDAP `Transport: None, AllowInsecure: true` remains in the deploy artifact | — | **Stays accepted posture per the report** — bundled loopback GLAuth with documented AD-repoint instructions; a future over-the-wire AD repoint must revisit transport. No task |
| — | Oldest-vs-leader predicate unification in `SiteCommunicationActor`/`SiteReplicationActor` + resync work | — | **Reference → PLAN-R2-02** (explicit cross-plan ownership; not touched here) |
archreview/plans/PLAN-R2-01-cluster-host-failover.md.tasks.json
+82
View File
@@ -0,0 +1,82 @@
{
"planPath": "archreview/plans/PLAN-R2-01-cluster-host-failover.md",
"tasks": [
{
"id": 1,
"subject": "Task 1: Rewrite failover-drill.sh — standby-victim default + explicit active-victim gap mode",
"status": "pending",
"blockedBy": []
},
{
"id": 2,
"subject": "Task 2: Correct the recovery narrative + document the first-seed bootstrap constraint",
"status": "pending",
"blockedBy": [
1
]
},
{
"id": 3,
"subject": "Task 3: RUN the drill live (both directions) and record measured timings",
"status": "pending",
"blockedBy": [
1,
2
]
},
{
"id": 4,
"subject": "Task 4: Wire FailoverTimingTests to TwoNodeClusterFixture — measure the ~25s envelope in-process",
"status": "pending",
"blockedBy": []
},
{
"id": 5,
"subject": "Task 5: Apply the wonder-app-vd03 appsettings overlay edits (NodeName + AllowSingleNodeCluster, drop phantom seeds)",
"status": "pending",
"blockedBy": []
},
{
"id": 6,
"subject": "Task 6: Complete the install.ps1 recovery actions (sc.exe failureflag) + RUNBOOK recovery step",
"status": "pending",
"blockedBy": []
},
{
"id": 7,
"subject": "Task 7: Correct the factually wrong N2 deferral record in the master tracker",
"status": "pending",
"blockedBy": [
5,
6
]
},
{
"id": 8,
"subject": "Task 8: Metrics-staleness for never-reported sites (FirstSeenAt anchor)",
"status": "pending",
"blockedBy": []
},
{
"id": 9,
"subject": "Task 9: Purge the stale \"cluster leader\" narration from CentralHealthReportLoop",
"status": "pending",
"blockedBy": []
},
{
"id": 10,
"subject": "Task 10: Generalize the registrar — SingletonRegistrar with an optional role scope",
"status": "pending",
"blockedBy": []
},
{
"id": 11,
"subject": "Task 11: Route the two site singletons through the registrar — deployment-manager + event-log-handler gain drains",
"status": "pending",
"blockedBy": [
10
]
}
],
"lastUpdated": "2026-07-13T03:25:44Z"
}
archreview/plans/PLAN-R2-02-communication-store-and-forward.md
+1340
View File
File diff suppressed because it is too large Load Diff
archreview/plans/PLAN-R2-02-communication-store-and-forward.md.tasks.json
+115
View File
@@ -0,0 +1,115 @@
{
"planPath": "archreview/plans/PLAN-R2-02-communication-store-and-forward.md",
"tasks": [
{
"id": 1,
"subject": "Task 1: Shared oldest-Up active-node evaluator, reachable from Communication and SiteRuntime",
"status": "pending",
"blockedBy": []
},
{
"id": 2,
"subject": "Task 2: Two-node divergence repro — leader≠oldest wipes the delivering node's buffer (failing first)",
"status": "pending",
"blockedBy": []
},
{
"id": 3,
"subject": "Task 3: Swap SiteReplicationActor to the shared oldest-Up predicate + Host wires the delivery-gate delegate",
"status": "pending",
"blockedBy": [
1,
2
]
},
{
"id": 4,
"subject": "Task 4: Swap SiteCommunicationActor.DefaultIsActiveCheck + Host wiring + doc sync",
"status": "pending",
"blockedBy": [
3
]
},
{
"id": 5,
"subject": "Task 5: Chunked resync protocol — additive messages, byte-budgeted chunker, active-side chunked answer",
"status": "pending",
"blockedBy": [
3
]
},
{
"id": 6,
"subject": "Task 6: Standby-side chunk assembly, atomic apply, ack + the N5 race comment",
"status": "pending",
"blockedBy": [
5
]
},
{
"id": 7,
"subject": "Task 7: Resync delivery confirmation on the active node + telemetry + doc rewrite",
"status": "pending",
"blockedBy": [
6
]
},
{
"id": 8,
"subject": "Task 8: Publish _sweepTask only when the sweep CAS is won (unclobber the shutdown drain)",
"status": "pending",
"blockedBy": []
},
{
"id": 9,
"subject": "Task 9: Eager validation for SweepBatchLimit / SweepTargetParallelism",
"status": "pending",
"blockedBy": []
},
{
"id": 10,
"subject": "Task 10: Coalesce live-alarm delta publishes (bound the per-circuit fan-out) [MUTEX with PLAN-R2-07]",
"status": "pending",
"blockedBy": []
},
{
"id": 11,
"subject": "Task 11: Aggregator lifecycle — queued re-seed after reconnect + stream-generation stamp",
"status": "pending",
"blockedBy": [
10
]
},
{
"id": 12,
"subject": "Task 12: Instance-injected ReconnectDelay/StabilityWindow (kill the process-global test seams)",
"status": "pending",
"blockedBy": [
11
]
},
{
"id": 13,
"subject": "Task 13: Production caller for RemoveSiteAsync — dispose a deleted site's gRPC channels",
"status": "pending",
"blockedBy": []
},
{
"id": 14,
"subject": "Task 14: Documented acceptance — standby-node aggregators + deleted-site viewer behavior",
"status": "pending",
"blockedBy": [
4
]
},
{
"id": 15,
"subject": "Task 15: Single-endpoint sites can go live [MUTEX with PLAN-R2-07]",
"status": "pending",
"blockedBy": [
12
]
}
],
"lastUpdated": "2026-07-13T03:28:50Z"
}
archreview/plans/PLAN-R2-03-site-runtime-dcl.md
+485
View File
@@ -0,0 +1,485 @@
# PLAN-R2-03 — Site Runtime & DCL Round-2 Hardening Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
**Goal:** Fix the round-2 findings in `archreview/03-site-runtime-dcl.md` (dated 2026-07-12) — the one Medium residual on the MxGateway reconnect path (a stale event loop's non-OCE fault can still flap the fresh connection through the generic catch) plus five Low polish items: missing failure mapping on the expression-eval `PipeTo`, two PLAN-03 option knobs absent from the eager options validator, the deploy-path double-compile (adopt the earmarked compile cache), the reintroduced "off-thread" spec-drift sentence, and the undocumented Expression-trigger/script-scheduler coupling. Round-1 deferrals P5 and P6 stay deferred (coverage-table rows only; Tasks 5/6's compile cache shrinks P6's warm-process cost as a side effect).
**Architecture:** All changes are site-side, inside `ZB.MOM.WW.ScadaBridge.SiteRuntime` and `ZB.MOM.WW.ScadaBridge.DataConnectionLayer`, plus their design doc. The established discipline is preserved: adapter lifecycle faults are classified on the *loop's own* cancellation token (never the mutable fields a newer generation may have replaced), every off-thread result — including a *faulted* task — returns to the actor via `PipeTo` and is applied on the actor thread, and options fail fast at boot with key-naming messages per the PLAN-08 §1.5 convention. The N4 compile cache mirrors TemplateEngine's `ScriptCompileVerdictCache` (SHA-256-keyed, bounded, clear-on-overflow) but is site-local and stores the full `ScriptCompilationResult` — sound because Roslyn `Script<T>` objects are immutable and thread-safe and the result's error text is name-free — so the synchronous deploy gate (deliberately on the singleton mailbox, see N5) and the `InstanceActor.PreStart` recompile share one Roslyn compile per script body per process lifetime. Docs travel with code.
**Tech Stack:** C#/.NET, Akka.NET (ReceiveActor, TestKit.Xunit2), Roslyn scripting, gRPC (Grpc.Core exception shapes via the `ZB.MOM.WW.MxGateway.Client` package), xunit + NSubstitute.
Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per-project: `dotnet test tests/<project> --filter <name>`.
---
### Task 1: MxGatewayDataConnection — a stale event loop's fault on a cancelled token must not signal Disconnected
**Classification:** high-risk (adapter reconnect lifecycle, concurrency with the dying loop)
**Estimated implement time:** ~5 min
**Parallelizable with:** 2, 3, 4, 5
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MxGatewayDataConnection.cs` (`ConnectAsync` ~lines 106–108, `RunEventLoopAsync` ~lines 111–130)
- Test: `tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/MxGatewayDataConnectionReconnectTests.cs`
The S1 fix cancels the previous loop's CTS and disposes the previous client (`:77–89`), then resets `_disconnectFired` (`:91`) — but `Cancel()` is not synchronous with the loop's exit. The old loop's `await foreach` over the gRPC stream surfaces its failure milliseconds later, and a cancelled/disposed Grpc.Net stream commonly faults with `RpcException(StatusCode.Cancelled)` (the default without `ThrowOperationCanceledOnCancellation`) or `ObjectDisposedException` from the concurrent `DisposeAsync` — neither is an `OperationCanceledException`, so both fall into the generic `catch` (`:125–129`) → `RaiseDisconnected()` against the *reset* guard → the exact S1 scenario-2 flap, potentially self-sustaining.
1. Add failing tests to the existing reconnect test class (extend its per-client substitute fixture so each client's `RunEventLoopAsync` returns a controllable `TaskCompletionSource` task instead of `Task.Delay(Timeout.Infinite)`):
```csharp
[Fact]
public async Task StaleEventLoopRpcFault_AfterReconnect_DoesNotSignalDisconnected()
{
var clients = new List<IMxGatewayClient>();
var loops = new List<TaskCompletionSource>();
var loopTokens = new List<CancellationToken>();
var factory = Substitute.For<IMxGatewayClientFactory>();
factory.Create().Returns(_ =>
{
var c = Substitute.For<IMxGatewayClient>();
var tcs = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
loops.Add(tcs);
c.RunEventLoopAsync(Arg.Any<Action<MxValueUpdate>>(), Arg.Do<CancellationToken>(t => loopTokens.Add(t)))
.Returns(tcs.Task);
clients.Add(c);
return c;
});
var adapter = new MxGatewayDataConnection(factory, NullLogger<MxGatewayDataConnection>.Instance);
var disconnects = 0;
adapter.Disconnected += () => Interlocked.Increment(ref disconnects);
var details = new Dictionary<string, string> { ["Endpoint"] = "http://gw:5000", ["ApiKey"] = "k" };
await adapter.ConnectAsync(details);
await WaitUntilAsync(() => loopTokens.Count == 1);
await adapter.ConnectAsync(details); // reconnect: cancels loop #0, resets _disconnectFired
await WaitUntilAsync(() => loopTokens.Count == 2);
// The OLD loop observes its cancellation as a gRPC fault, not an OCE —
// the Grpc.Net default without ThrowOperationCanceledOnCancellation.
loops[0].SetException(new RpcException(new Status(StatusCode.Cancelled, "call cancelled")));
await Task.Delay(200);
Assert.Equal(0, disconnects); // stale fault absorbed — the fresh connection must not flap
// The CURRENT loop's genuine fault must still signal, exactly once.
loops[1].SetException(new RpcException(new Status(StatusCode.Unavailable, "gateway gone")));
await WaitUntilAsync(() => disconnects == 1);
}
[Fact]
public async Task StaleEventLoopObjectDisposedFault_AfterReconnect_DoesNotSignalDisconnected()
{
// Identical drive, but the old loop faults with the concurrent-DisposeAsync shape.
// … same fixture …
loops[0].SetException(new ObjectDisposedException("MxGatewayClient"));
await Task.Delay(200);
Assert.Equal(0, disconnects);
}
```
(`RpcException`/`Status`/`StatusCode` come from `Grpc.Core`, transitively via the `ZB.MOM.WW.MxGateway.Client` package the test project already references — `RealMxGatewayClient.cs:227` catches `RpcException` in the main project. If the test assembly cannot resolve them, add a `Grpc.Net.Client` PackageReference; the version is already pinned in `Directory.Packages.props`.)
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~StaleEventLoopRpcFault_AfterReconnect|FullyQualifiedName~StaleEventLoopObjectDisposedFault_AfterReconnect"` → expect **FAIL** (the generic catch raises `Disconnected` against the reset guard: `disconnects == 1` after the stale fault).
3. Implement:
- Change `RunEventLoopAsync` to take the client AND token as parameters — `private async Task RunEventLoopAsync(IMxGatewayClient client, CancellationToken ct)` — and use `client.RunEventLoopAsync(…)` instead of the `_client!` field read (the old loop must never bind a newer generation's client).
- Add a filtered catch **before** the generic one:
```csharp
catch (Exception ex) when (ct.IsCancellationRequested)
{
// Teardown fault of a superseded loop (N1): Cancel() is not synchronous with
// the loop's exit — a cancelled/disposed gRPC stream commonly surfaces as
// RpcException(StatusCode.Cancelled) (Grpc.Net default without
// ThrowOperationCanceledOnCancellation) or ObjectDisposedException from the
// concurrent DisposeAsync, milliseconds AFTER ConnectAsync has already reset
// _disconnectFired. Any fault on a cancelled token is normal shutdown;
// raising Disconnected here would flap the NEW healthy session.
_logger.LogDebug(ex,
"MxGateway event loop faulted after cancellation — superseded loop teardown; not signalling disconnect");
}
```
- In `ConnectAsync`, capture the token and client into locals before the lambda (a rapid double-reconnect must not hand loop #1 loop #2's token/client — the current `Task.Run(() => RunEventLoopAsync(_eventLoopCts.Token))` reads the *field* lazily):
```csharp
_eventLoopCts = new CancellationTokenSource();
var loopToken = _eventLoopCts.Token; // capture NOW: the field may be replaced by a
var loopClient = _client; // subsequent reconnect before Task.Run executes (N1)
_ = Task.Run(() => RunEventLoopAsync(loopClient, loopToken));
```
4. Run the two new tests → **PASS**; then the full adapter suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~MxGatewayDataConnection"` → all green (the existing `Reconnect_DisposesPreviousClient_AndCancelsPreviousEventLoop` test pins the S1 behavior and must survive the signature change).
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MxGatewayDataConnection.cs tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/MxGatewayDataConnectionReconnectTests.cs && git commit -m "fix(dcl): stale MxGateway event-loop fault on a cancelled token no longer flaps the fresh connection (plan R2-03 T1)"`
### Task 2: ScriptActor — failure mapping on the expression-eval `PipeTo` (a faulted scheduler task must not kill the trigger)
**Classification:** high-risk (actor concurrency, trigger liveness)
**Estimated implement time:** ~5 min
**Parallelizable with:** 1, 3, 4, 5, 6
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs` (Receive block ~line 161, `StartExpressionEvaluation` PipeTo ~lines 311–313, internal messages ~line 623)
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ScriptActorTests.cs`
`StartExpressionEvaluation` pipes with `success:` only (`:312–313`). The inner body catches everything, so the only fault source is the outer `Task.Factory.StartNew` itself (e.g. `ObjectDisposedException` from a disposed `ScriptExecutionScheduler` during shutdown/test teardown) — but if it ever fires, the actor gets an unhandled `Status.Failure` and `_evalInFlight` stays `true` forever (`:284–285`): every subsequent change parks in `_evalPending` and the expression trigger is dead for the actor's lifetime with no log.
1. Failing test. Add a public static gate hook to the test file (the `CompileRawTriggerExpression` helper compiles *without* the trust validator, so the expression may reference it — same trick as the P1 thread-name tests):
```csharp
public static class EvalGate
{
public static readonly SemaphoreSlim Gate = new(0);
private static int _entries;
public static int Entries => Volatile.Read(ref _entries);
public static bool Block() { Interlocked.Increment(ref _entries); Gate.Wait(); return false; }
}
[Fact]
public void ExpressionEvalTaskFault_ClearsInFlight_AndDrainsPendingEvaluation()
{
var expr = CompileRawTriggerExpression(
"ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors.EvalGate.Block()");
var (actor, _) = CreateTriggeredActor(
"ExprFault", "Expression", "{\"expression\":\"true\",\"mode\":\"OnTrue\"}", null, expr);
try
{
actor.Tell(Change("A", "1")); // eval starts on the scheduler and BLOCKS → _evalInFlight = true
AwaitAssert(() => Assert.Equal(1, EvalGate.Entries), TimeSpan.FromSeconds(10));
actor.Tell(Change("A", "2")); // coalesces → _evalPending = true
// Simulate the faulted scheduler task the PipeTo failure mapping now surfaces
// (e.g. ObjectDisposedException from a disposed ScriptExecutionScheduler).
actor.Tell(new ScriptActor.ExpressionEvalFailed(new ObjectDisposedException("scheduler")));
// In-flight cleared + pending drained ⇒ a SECOND evaluation starts and blocks too.
AwaitAssert(() => Assert.Equal(2, EvalGate.Entries), TimeSpan.FromSeconds(10));
}
finally
{
EvalGate.Gate.Release(EvalGate.Entries); // ALWAYS free the shared scheduler threads
}
}
```
(Note the expression must reference the gate's fully-qualified name; the raw compile references `typeof(object).Assembly` only, so also add the test assembly to `CompileRawTriggerExpression`'s `.WithReferences(...)` for this test — or add an overload taking extra references. The `finally` release is mandatory: the gate blocks threads of the *process-wide* `ScriptExecutionScheduler.Shared`.)
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ExpressionEvalTaskFault_ClearsInFlight"` → expect **FAIL** — the RED step is the compile failure (`ScriptActor.ExpressionEvalFailed` does not exist); behaviorally, without the fix the second evaluation never starts either.
3. Implement:
- New message next to `ExpressionEvalResult` (`:623`) — `internal` (not `private`) so the TestKit can Tell it, same visibility idiom as `IntervalTick`/`ScriptExecutionCompleted`:
```csharp
/// <summary>
/// Piped back to self when the off-dispatcher evaluation TASK itself faults
/// (e.g. ObjectDisposedException from a disposed ScriptExecutionScheduler
/// during shutdown) — the inner body's catch never sees that. Without this
/// mapping the actor receives an unhandled Status.Failure and _evalInFlight
/// stays true forever, permanently parking the expression trigger (N2).
/// </summary>
internal sealed record ExpressionEvalFailed(Exception Cause);
```
- Ctor Receive block (after the `ExpressionEvalResult` registration at `:161`): `Receive<ExpressionEvalFailed>(HandleExpressionEvalFailed);`
- PipeTo (`:312–313`): `…​.PipeTo(self, success: r => new ExpressionEvalResult(r), failure: ex => new ExpressionEvalFailed(ex));`
- Handler — log via the existing `LogExpressionError` (`:397`, already increments the health counter), then reuse the result path with `false` (consistent with the inner catch's treated-as-false semantics — clears in-flight, applies the false edge, drains pending):
```csharp
private void HandleExpressionEvalFailed(ExpressionEvalFailed msg)
{
LogExpressionError(msg.Cause);
HandleExpressionEvalResult(new ExpressionEvalResult(false));
}
```
4. Run the new test → **PASS**; then `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ScriptActorTests"` → all green (the P1 expression/WhileTrue suites are the regression canary).
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ScriptActorTests.cs && git commit -m "fix(siteruntime): faulted expression-eval task clears _evalInFlight instead of killing the ScriptActor trigger (plan R2-03 T2)"`
### Task 3: AlarmActor — same failure mapping on the expression-eval `PipeTo`
**Classification:** high-risk (alarm state machine liveness)
**Estimated implement time:** ~4 min
**Parallelizable with:** 1, 2, 4, 5, 6
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/AlarmActor.cs` (Receive block ~line 188, `StartExpressionEvaluation` PipeTo ~lines 552–554, internal messages ~line 797)
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/AlarmActorTests.cs`
Structurally identical to Task 2 — `AlarmActor.cs:552–554` pipes success-only, `:523–524` sets `_evalInFlight`, so a faulted task permanently kills the Expression alarm.
1. Failing test (reuse the `ExpressionAlarm_EvaluatesOnSchedulerThread_AndActivates` harness in `AlarmActorTests.cs:1031` — same `ResolvedAlarm`/`instanceProbe` construction):
```csharp
[Fact]
public void ExpressionEvalTaskFault_DoesNotKillTheAlarmTrigger()
{
var expr = CompileRawTriggerExpression("true");
var alarmConfig = new ResolvedAlarm
{
CanonicalName = "ExprFaultAlarm",
TriggerType = "Expression",
TriggerConfiguration = "{\"expression\":\"true\"}",
PriorityLevel = 1
};
var instanceProbe = CreateTestProbe();
var alarm = ActorOf(Props.Create(() => new AlarmActor(
"ExprFaultAlarm", "Pump1", instanceProbe.Ref, alarmConfig,
null, _sharedLibrary, _options,
NullLogger<AlarmActor>.Instance, expr)));
// Simulate the faulted scheduler task: must be handled (logged, in-flight
// cleared, false applied) — NOT an unhandled Status.Failure.
alarm.Tell(new AlarmActor.ExpressionEvalFailed(new ObjectDisposedException("scheduler")));
// The trigger must still be alive: the next change evaluates and activates.
alarm.Tell(new AttributeValueChanged("Pump1", "A", "A", 1, "Good", DateTimeOffset.UtcNow));
var change = instanceProbe.ExpectMsg<AlarmStateChanged>(TimeSpan.FromSeconds(10));
Assert.Equal(AlarmState.Active, change.State);
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ExpressionEvalTaskFault_DoesNotKillTheAlarmTrigger"` → expect **FAIL** (compile error — `AlarmActor.ExpressionEvalFailed` does not exist).
3. Implement, mirroring Task 2:
- `internal sealed record ExpressionEvalFailed(Exception Cause);` next to the existing `ExpressionEvalResult` (`:797`), with the same N2 XML doc.
- `Receive<ExpressionEvalFailed>(HandleExpressionEvalFailed);` after `:188`.
- PipeTo (`:553–554`) gains `failure: ex => new ExpressionEvalFailed(ex)`.
- Handler — same shape as the inner catch's error path, then delegate to the false result (the class doc already promises "a throwing … expression is treated as false so that the state machine still runs — an Active alarm correctly clears"):
```csharp
private void HandleExpressionEvalFailed(ExpressionEvalFailed msg)
{
_healthCollector?.IncrementAlarmError();
_logger.LogError(msg.Cause,
"Alarm {Alarm} trigger-expression evaluation task faulted on {Instance}; treated as false",
_alarmName, _instanceName);
HandleExpressionEvalResult(new ExpressionEvalResult(false));
}
```
4. Run the new test → **PASS**; then `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~AlarmActorTests"` → all green.
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/AlarmActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/AlarmActorTests.cs && git commit -m "fix(siteruntime): faulted expression-eval task clears _evalInFlight instead of killing the AlarmActor trigger (plan R2-03 T3)"`
### Task 4: Eagerly validate `TagSubscribeRetryIntervalMs` and `StuckScriptGraceMs`
**Classification:** small (additive validator rules per the PLAN-08 §1.5 convention)
**Estimated implement time:** ~3 min
**Parallelizable with:** 1, 2, 3, 5, 6
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs` (append after the `ConfigFetchRetryCount` rule, lines 56–58)
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/SiteRuntimeOptionsValidatorTests.cs`
The validator covers ten options but not the two PLAN-03 added: `TagSubscribeRetryIntervalMs` (`SiteRuntimeOptions.cs:76` — a 0 hot-loops the subscribe retry, a negative value throws inside `InstanceActor.SendTagSubscribe`'s `ScheduleTellOnceCancelable`) and `StuckScriptGraceMs` (`SiteRuntimeOptions.cs:86` — a negative value throws `ArgumentOutOfRangeException` inside the watchdog's `Task.Delay`, `ScriptExecutionActor.cs:144`).
1. Failing tests (mirror the existing `Validate(...)` helper and assertion shape in the test file):
```csharp
[Fact]
public void ZeroTagSubscribeRetryIntervalMs_IsRejected()
{
var result = Validate(new SiteRuntimeOptions { TagSubscribeRetryIntervalMs = 0 });
Assert.True(result.Failed);
Assert.Contains("TagSubscribeRetryIntervalMs", result.FailureMessage);
}
[Fact]
public void NegativeStuckScriptGraceMs_IsRejected()
{
var result = Validate(new SiteRuntimeOptions { StuckScriptGraceMs = -1 });
Assert.True(result.Failed);
Assert.Contains("StuckScriptGraceMs", result.FailureMessage);
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~SiteRuntimeOptionsValidatorTests"` → the two new tests **FAIL** (validator succeeds on both).
3. Implement — two `RequireThat` lines matching the file's key-naming message style:
```csharp
builder.RequireThat(options.TagSubscribeRetryIntervalMs > 0,
$"ScadaBridge:SiteRuntime:TagSubscribeRetryIntervalMs must be greater than 0 " +
$"(was {options.TagSubscribeRetryIntervalMs}); a zero interval hot-loops the tag-subscribe " +
"retry and a negative one throws inside the Instance Actor's retry scheduling.");
builder.RequireThat(options.StuckScriptGraceMs >= 0,
$"ScadaBridge:SiteRuntime:StuckScriptGraceMs must be >= 0 " +
$"(was {options.StuckScriptGraceMs}); a negative grace throws inside the stuck-script watchdog's delay.");
```
4. Run → **PASS**, including `DefaultOptions_AreValid` (defaults 5000/30000 satisfy both rules).
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/SiteRuntimeOptionsValidatorTests.cs && git commit -m "fix(siteruntime): eagerly validate TagSubscribeRetryIntervalMs + StuckScriptGraceMs at boot (plan R2-03 T4)"`
### Task 5: SiteScriptCompileCache — bounded process-wide compiled-script cache (N4 groundwork)
**Classification:** standard (pure new class, no consumers yet)
**Estimated implement time:** ~5 min
**Parallelizable with:** 1, 2, 3, 4
**Files:**
- Create: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/SiteScriptCompileCache.cs`
- Test: create `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/SiteScriptCompileCacheTests.cs`
This is the site-side adoption of the compile-cache abstraction the master tracker earmarked when PLAN-03's Task 3/4 shipped ("later refactor, not a blocker") — TemplateEngine's `ScriptCompileVerdictCache` pattern, but storing the full `ScriptCompilationResult` so both the deploy gate and the `PreStart` recompile reuse the compiled `Script<object?>` itself. Soundness: the result is a pure function of (code, globals type, compile-time-static trust policy); the stored error text is name-free (`Diagnostic.GetMessage()` / `"Compilation exception: …"` — callers prefix their own context); Roslyn `Script<T>` is immutable and thread-safe (`RunAsync` creates per-run state), so one instance may be shared across actors.
1. Failing tests (the type does not exist — compile failure is the RED step). The cache is process-wide static state: put this test class and Task 6's `ScriptCompilationServiceTests` additions in one xunit collection (`[Collection("SiteScriptCompileCache")]`) and call `Clear()` at the top of each test, so parallel classes cannot skew `Hits`/`Count`:
```csharp
[Collection("SiteScriptCompileCache")]
public class SiteScriptCompileCacheTests
{
private static ScriptCompilationResult Ok() =>
ScriptCompilationResult.Succeeded(CSharpScript.Create<object?>("1", ScriptOptions.Default, typeof(ScriptGlobals)));
[Fact]
public void GetOrAdd_SameCodeAndGlobals_ComputesOnce()
{
SiteScriptCompileCache.Clear();
var factoryCalls = 0;
ScriptCompilationResult Factory() { factoryCalls++; return Ok(); }
var r1 = SiteScriptCompileCache.GetOrAdd("return 1;", typeof(ScriptGlobals), Factory);
var r2 = SiteScriptCompileCache.GetOrAdd("return 1;", typeof(ScriptGlobals), Factory);
Assert.Equal(1, factoryCalls);
Assert.Same(r1, r2);
Assert.Equal(1, SiteScriptCompileCache.Hits);
}
[Fact]
public void GetOrAdd_SameCode_DifferentGlobals_AreSeparateEntries()
{
SiteScriptCompileCache.Clear();
SiteScriptCompileCache.GetOrAdd("1 > 0", typeof(ScriptGlobals), Ok);
SiteScriptCompileCache.GetOrAdd("1 > 0", typeof(TriggerExpressionGlobals), Ok);
Assert.Equal(2, SiteScriptCompileCache.Count);
Assert.Equal(0, SiteScriptCompileCache.Hits); // no cross-globals hit
}
[Fact]
public void Overflow_ClearsWholesale()
{
SiteScriptCompileCache.Clear();
for (var i = 0; i <= SiteScriptCompileCache.MaxEntries; i++)
SiteScriptCompileCache.GetOrAdd($"return {i};", typeof(ScriptGlobals), Ok);
Assert.True(SiteScriptCompileCache.Count <= SiteScriptCompileCache.MaxEntries);
}
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~SiteScriptCompileCacheTests"` → **FAIL** (type does not exist).
3. Implement, mirroring `ScriptCompileVerdictCache` (`src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ScriptCompileVerdictCache.cs`): `internal static class SiteScriptCompileCache` (InternalsVisibleTo already covers the test assembly) with
- `internal const int MaxEntries = 1024;` — deliberately smaller than the verdict cache's 4096: entries pin compiled assemblies, not just verdict strings; on overflow the cache is cleared wholesale (results are recomputable, coarse reset avoids eviction bookkeeping).
- Key = `globalsType.FullName + ":" + Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(code)))` — the globals discriminator keeps `ScriptGlobals` and `TriggerExpressionGlobals` compiles of identical source apart.
- `public static ScriptCompilationResult GetOrAdd(string code, Type globalsType, Func<ScriptCompilationResult> factory)` over a `ConcurrentDictionary<string, ScriptCompilationResult>`, incrementing `Hits` on a hit; `Hits`/`Count`/`Clear()` exposed for tests and diagnostics — same surface as the TemplateEngine cache.
- XML doc stating the three soundness invariants above (pure function of code+globals+static policy; name-free errors; `Script<T>` immutability) and that **failures are cached too** — a bad script body fails identically for every caller.
4. Run → **PASS**.
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/SiteScriptCompileCache.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/SiteScriptCompileCacheTests.cs && git commit -m "feat(siteruntime): bounded process-wide compiled-script cache, keyed on code hash + globals type (plan R2-03 T5)"`
### Task 6: Wire the cache into ScriptCompilationService — deploy gate + PreStart share one compile per script per revision
**Classification:** standard (single choke point; all call sites inherit)
**Estimated implement time:** ~4 min
**Parallelizable with:** 1, 2, 3, 4 (needs 5 merged)
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptCompilationService.cs` (`CompileCore`, ~lines 98–139)
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs` (gate rationale comment ~lines 510–521 — comment only)
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/ScriptCompilationServiceTests.cs`
`CompileCore` is the single funnel for every site-side compile — the `DeployCompileValidator` gate (`DeploymentManagerActor.cs:522`), the `InstanceActor.CreateChildActors` recompile (`InstanceActor.cs:1485, 1497–1498`), and `SharedScriptLibrary` all flow through it — so caching here collapses the 2×N-compiles-per-bulk-redeploy cost (N4) and shrinks P6's warm-process staggered-startup recompiles for free, with zero call-site changes.
1. Failing test (add to `ScriptCompilationServiceTests`, tagged `[Collection("SiteScriptCompileCache")]` per Task 5's isolation note):
```csharp
[Fact]
public void Compile_SameCodeTwice_SharesOneRoslynCompile()
{
SiteScriptCompileCache.Clear();
var r1 = _service.Compile("deploy-gate-copy", "return 1 + 1;");
var r2 = _service.Compile("prestart-copy", "return 1 + 1;");
Assert.True(r1.IsSuccess);
Assert.Same(r1.CompiledScript, r2.CompiledScript); // one compile, shared Script<T> (N4)
Assert.Equal(1, SiteScriptCompileCache.Hits);
}
[Fact]
public void Compile_ScriptAndTriggerExpression_DoNotCrossContaminate()
{
SiteScriptCompileCache.Clear();
var script = _service.Compile("s", "1 > 0");
var trigger = _service.CompileTriggerExpression("t", "1 > 0");
Assert.NotSame(script.CompiledScript, trigger.CompiledScript); // different globals surfaces
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~Compile_SameCodeTwice_SharesOneRoslynCompile|FullyQualifiedName~Compile_ScriptAndTriggerExpression_DoNotCrossContaminate"` → **FAIL** (`Assert.Same` fails: two independent compiles today).
3. Implement:
- Rename the current `CompileCore` body to `private ScriptCompilationResult CompileUncached(string name, string code, Type globalsType)` (byte-identical, including the trust-model check — the verdict is deterministic on code, so it is cacheable with the compile).
- New `CompileCore`:
```csharp
private ScriptCompilationResult CompileCore(string name, string code, Type globalsType)
{
var result = SiteScriptCompileCache.GetOrAdd(code, globalsType,
() => CompileUncached(name, code, globalsType));
if (!result.IsSuccess)
_logger.LogDebug(
"Script {Script}: returning cached compile failure ({ErrorCount} error(s))",
name, result.Errors.Count); // miss-path logs name the FIRST caller; keep this caller visible
return result;
}
```
- Update the class XML doc: results (success AND failure) are memoised process-wide in `SiteScriptCompileCache`; error text is name-free by construction, so cached failures render correctly for every caller.
- Append one sentence to the `HandleDeploy` gate comment (`DeploymentManagerActor.cs:519–521`): "Redeploys and multi-instance deploys of unchanged scripts hit the process-wide compile cache (`SiteScriptCompileCache`), so the synchronous gate recompiles only genuinely new code and the Instance Actor's PreStart reuses the gate's compile (N4)."
4. Run the two new tests → **PASS**; then the funnel's consumer suites: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ScriptCompilationServiceTests|FullyQualifiedName~DeployCompileValidatorTests|FullyQualifiedName~SharedScriptLibraryTests|FullyQualifiedName~TrustModelSemanticTests"` → all green (these pin that caching changed no verdicts).
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptCompilationService.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/ScriptCompilationServiceTests.cs && git commit -m "perf(siteruntime): deploy gate + InstanceActor PreStart share one Roslyn compile via the compile cache (plan R2-03 T6)"`
### Task 7: Design-doc sync — synchronous compile gate (N5), Expression/scheduler coupling (N6), P6/N4 note refresh
**Classification:** trivial (docs + one stale XML doc; repo rule: docs travel with code)
**Estimated implement time:** ~4 min
**Parallelizable with:** none (write after the behavior tasks land so the docs describe reality)
**Files:**
- Modify: `docs/requirements/Component-SiteRuntime.md` (lines ~489, ~234–246, ~99)
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Deployment/DeployCompileValidator.cs` (class XML doc, ~lines 16–18)
1. **N5** — `Component-SiteRuntime.md:489`: the sentence claims the gate compiles "**off-thread**"; the shipped gate is deliberately synchronous (`DeploymentManagerActor.cs:510–522`). Replace the off-thread clause so the bullet reads: "…the Deployment Manager compiles **all** of the instance's scripts, alarm on-trigger scripts, and trigger expressions **synchronously, as a pure prefix step of the deploy handler on the singleton's actor thread** — deliberately not off-thread, because piping the verdict back to self would reorder concurrent deploys against delete/disable and break the redeploy-supersede mailbox-FIFO ordering (see the in-code rationale in `DeploymentManagerActor.HandleDeploy`); unchanged script bodies hit the process-wide compile cache, so the gate recompiles only genuinely new code. Any compile failure rejects the deployment with the collected compile errors…" (keep the existing honesty tail verbatim).
2. **N6** — `Component-SiteRuntime.md` Alarm Evaluation section (~line 236, after the trigger-type list): add one bullet: "**Expression** trigger evaluation runs on the bounded script-execution thread pool shared with script bodies (and Expression *script* triggers): scheduler saturation — e.g. stuck scripts occupying all threads — delays Expression alarm transitions site-wide until a thread frees. This is a deliberate trade-off (evaluation no longer blocks dispatcher threads; per-actor coalescing bounds queue growth) and the scheduler gauges + stuck-script watchdog make the saturated state visible."
3. **P6/N4 note refresh** — `Component-SiteRuntime.md:99` (the existing P6 follow-up note): append "As of the round-2 hardening, a process-wide compile cache dedupes identical script bodies within a node's process lifetime (the deploy gate's compile is reused by Instance Actor start), shrinking the recompile cost; the *first* compile after process start still runs inside Instance Actor start, so the deferral stands."
4. **N5 (code-comment drift)** — `DeployCompileValidator.cs:16–18`: the XML doc still says "it is safe to run off the singleton mailbox on a `Task.Run` thread — the deploy path never blocks on Roslyn", which describes the plan's original shape, not the shipped synchronous gate. Reword: "Because it holds no actor state and only calls the (stateless) `ScriptCompilationService`, it is pure and thread-safe; the Deployment Manager nevertheless invokes it *synchronously on the singleton's actor thread* as a pure prefix step, preserving mailbox-FIFO deploy ordering (see `HandleDeploy`). Compiles of unchanged script bodies are deduped by the process-wide compile cache."
5. Review with `git diff docs/requirements/Component-SiteRuntime.md src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Deployment/DeployCompileValidator.cs` — verify no other "off-thread" claim about the gate remains (`grep -n "off-thread" docs/requirements/Component-SiteRuntime.md` should return only the line-99 P6 note, which refers to the *startup* compiles, not the gate).
6. Commit: `git add docs/requirements/Component-SiteRuntime.md src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Deployment/DeployCompileValidator.cs && git commit -m "docs(siteruntime): compile gate is synchronous by design; Expression alarms share the bounded script pool (plan R2-03 T7)"`
---
## Dependencies on other plans
- **R2-05 (Templates/Deployment/Transport round 2)** owns the Transport options-validator knob; **R2-08** owns the AuditLog/Host validator gaps — Task 4 here covers ONLY the two SiteRuntime knobs (`TagSubscribeRetryIntervalMs`, `StuckScriptGraceMs`), per the cross-plan ruling.
- **TemplateEngine's `ScriptCompileVerdictCache`** stays untouched: it caches *verdicts* for central validation; Task 5/6's `SiteScriptCompileCache` caches *compiled scripts* for site execution. The two are intentionally separate (different result shapes, different assemblies); no shared abstraction is introduced.
- The site-side live-alarm-stream code (`SiteStreamManager.SubscribeSiteAlarms`) was reviewed clean in round 2 — nothing from the aggregated-live-alarm-stream plan lands here.
## Execution order
**P0 (do first):** Task 1 (N1 — the only Medium; closes the last open S1 mechanism).
**P1 (parallel, file-disjoint):** Tasks 2, 3 (N2 — liveness of every Expression trigger/alarm), Task 4 (N3), Task 5 (N4 groundwork).
**P2:** Task 6 (after 5 — consumes the cache; comment-only touch on `DeploymentManagerActor.cs`).
**Last:** Task 7 (docs describe the landed behavior).
Same-file sequencing summary: `ScriptCompilationService.cs`: 6 only. `DeploymentManagerActor.cs`: 6 only (comment). `DeployCompileValidator.cs`: 7 only (XML doc). No file is touched by two tasks.
After the final task: `dotnet build ZB.MOM.WW.ScadaBridge.slnx && dotnet test ZB.MOM.WW.ScadaBridge.slnx`, and for cluster-runtime verification rebuild the docker image (`bash docker/deploy.sh`).
---
## Findings Coverage
| Finding | Severity | Task(s) / Disposition |
|---|---|---|
| N1 — MxGateway: stale event loop's non-OCE fault (`RpcException(Cancelled)`/`ObjectDisposedException`) can flap the fresh connection through the generic catch; lazy `_eventLoopCts` field read in the `Task.Run` lambda | Medium | 1 (ct-guarded catch + token/client captured into locals; churn scenario regression-tested with both fault shapes) |
| N2 — expression-eval `PipeTo` has a success mapping only; a faulted scheduler task permanently kills the trigger with `_evalInFlight` stuck | Low | 2 (ScriptActor), 3 (AlarmActor) |
| N3 — `TagSubscribeRetryIntervalMs` / `StuckScriptGraceMs` missing from the eager options validator | Low | 4 (SiteRuntime knobs only — Transport → R2-05, AuditLog/Host → R2-08 per the cross-plan ruling) |
| N4 — every deploy compiles each script twice (synchronous gate + `PreStart` recompile), no compile cache | Low | 5, 6 — **full adoption ships here** (one compile per script body per process; gate, PreStart, and SharedScriptLibrary all funnel through the cached `CompileCore`), so no deferred-register remainder. The gate itself stays synchronous by design (N5). |
| N5 — spec drift (reintroduced): `Component-SiteRuntime.md:489` says the gate runs "off-thread"; the shipped gate is deliberately synchronous | Low | 7 (spec sentence + the matching stale `DeployCompileValidator` XML doc) |
| N6 — Expression triggers/alarms share the bounded script scheduler; saturation silently stalls all Expression evaluation site-wide | Low | 7 (documented as the accepted trade-off the report judged it to be — gauges/watchdog already make it visible; no behavior change) |
| P5 — per-attribute script read is an Ask round-trip (round-1 deferral) | Low | **Deferred (accepted, unchanged)** — spec-consistent, no profiling evidence; batch `GetAttributes` remains the profiling-gated follow-up. No task. |
| P6 — Roslyn compiles inside `InstanceActor.PreStart` during staggered startup (round-1 deferral) | Low | **Deferred (accepted, further mitigated)** — Tasks 5/6's cache dedupes identical bodies within a warm process (the report's own observation); the first compile after process start still runs in `PreStart`, and moving it out stays out of scope. Doc note refreshed in 7. |
| UA1 removal-reconcile / cross-node list-aggregation sub-scope (round-1 residue) | — | **Deferred** behind the documented central-persistence-of-cert-trust follow-up (`Component-SiteRuntime.md:125`). No task. |
| S1–S10, P1–P4, C2–C6, UA1–UA7 (round-1 findings) | — | **Verified fixed** by the round-2 report's disposition table (26 fixed, evidence-read) — no action. |
archreview/plans/PLAN-R2-03-site-runtime-dcl.md.tasks.json
+13
View File
@@ -0,0 +1,13 @@
{
"planPath": "archreview/plans/PLAN-R2-03-site-runtime-dcl.md",
"tasks": [
{ "id": 1, "subject": "Task 1: MxGatewayDataConnection — stale event-loop fault on a cancelled token must not signal Disconnected (N1)", "status": "pending", "blockedBy": [] },
{ "id": 2, "subject": "Task 2: ScriptActor — failure mapping on the expression-eval PipeTo clears _evalInFlight (N2)", "status": "pending", "blockedBy": [] },
{ "id": 3, "subject": "Task 3: AlarmActor — failure mapping on the expression-eval PipeTo clears _evalInFlight (N2)", "status": "pending", "blockedBy": [] },
{ "id": 4, "subject": "Task 4: Eagerly validate TagSubscribeRetryIntervalMs + StuckScriptGraceMs (N3)", "status": "pending", "blockedBy": [] },
{ "id": 5, "subject": "Task 5: SiteScriptCompileCache — bounded process-wide compiled-script cache (N4 groundwork)", "status": "pending", "blockedBy": [] },
{ "id": 6, "subject": "Task 6: Wire the compile cache into ScriptCompilationService — deploy gate + PreStart share one compile (N4)", "status": "pending", "blockedBy": [5] },
{ "id": 7, "subject": "Task 7: Design-doc sync — synchronous compile gate (N5), Expression/scheduler coupling (N6), P6/N4 note refresh", "status": "pending", "blockedBy": [1, 2, 3, 4, 5, 6] }
],
"lastUpdated": "2026-07-13T03:23:02Z"
}
archreview/plans/PLAN-R2-04-data-audit-backbone.md
+679
View File
@@ -0,0 +1,679 @@
# PLAN-R2-04 — Data & Audit Backbone Round-2 Fixes Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
**Goal:** Close the seven NEW findings of the round-2 architecture review (`archreview/04-data-audit-backbone.md`, 2026-07-12): bound the KPI rollup backfill that currently re-folds the entire 90-day raw window in one tracked pass on every singleton failover (R1), make the fold fetch untracked and projected (R2), fix the ~60× rate-unit discontinuity at the raw/rollup routing boundary and the last-value-per-bucket re-fold error the aggregation catalog itself warns about (R3), close the catalog metric-literal drift hazard (R4), classify the failover fold-race failure grain (R5), give the SiteCalls terminal purge the same time-sliced batching + index support its two siblings received in round 1 (R6), and stop the site retention service from surfacing its own shutdown cancellation to the host (R7). Every accepted round-1 deferral stays deferred — coverage rows only, no code.
**Architecture:** All fixes stay inside the existing component boundaries. The rollup backfill/fold hardening lives in the KpiHistory recorder actor + `KpiHistoryRepository` (one additive `IKpiHistoryRepository` watermark method in Commons); the rate-presentation fix lives in the Commons `KpiSeriesBucketer` (additive optional parameter — existing call sites compile unchanged) driven by the already-shipped `KpiMetricAggregationCatalog`, wired at the `KpiHistoryQueryService` boundary; the SiteCalls purge fix rides one EF Core migration (a filtered `TerminalAtUtc` index) plus a repository-local time-sliced DELETE loop mirroring the round-1 `KpiSample` purge; the retention-service fix is a loop-boundary OCE catch. Design docs are updated in the same task as the code they describe (repo rule: doc + code + tests travel together). No new tables, no contract-breaking changes.
**Tech Stack:** C#/.NET 8, Akka.NET (actors/TestKit), EF Core + SQL Server (central; SQLite in-memory harness for the KpiHistory repository suite), xUnit. Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per-project with targeted filters only. MSSQL-backed `SkippableFact` suites (`SiteCallAuditRepositoryTests`) need `cd infra && docker compose up -d`. **EF migrations gotcha (repo-documented):** always build first and run `dotnet ef migrations add <Name>` WITHOUT `--no-build`, or it scaffolds an empty migration off the stale DLL — delete the empty files if it happens (`migrations remove` needs a live DB).
---
### Task 1: Untracked, projected fold fetch (R2)
**Classification:** standard
**Estimated implement time:** ~3 min
**Parallelizable with:** 3, 5, 6, 7, 8, 10, 11, 12 (NOT 2/9 — same repository file)
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs` (`FoldHourlyRollupsAsync` fetch at :104-106; group lambdas :112-128)
- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/KpiHistoryRepositoryTests.cs` (extend)
1. Write the failing test (SQLite in-memory harness — pattern-match `NewContext()`/`Sample()` already in the file; clear the tracker after seeding so the assertion sees only the fold's own work):
```csharp
[Fact]
public async Task FoldHourlyRollups_DoesNotTrack_KpiSampleEntities()
{
await using var ctx = NewContext();
var repo = new KpiHistoryRepository(ctx);
await repo.RecordSamplesAsync(new[]
{
Sample("NotificationOutbox", "queueDepth", "Global", null, 5, Base.AddMinutes(10)),
Sample("NotificationOutbox", "queueDepth", "Global", null, 7, Base.AddMinutes(50)),
});
ctx.ChangeTracker.Clear(); // isolate the fold's tracking behavior from the seed
await repo.FoldHourlyRollupsAsync(Base, Base.AddHours(1));
// The fold READS samples and WRITES rollups; the read must not register
// thousands of read-only KpiSample entries for DetectChanges to re-scan
// (arch-review 04 round 2, R2).
Assert.Empty(ctx.ChangeTracker.Entries<KpiSample>());
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter FoldHourlyRollups_DoesNotTrack_KpiSampleEntities` — expect FAIL (the fetch is a tracked `ToListAsync` today).
3. Replace the fetch at :104-106 with a projection (a `Select()` projection is never tracked, and it also drops the unneeded `Id` column, roughly halving per-row memory — which shrinks the R1 backfill's footprint even before Task 3 slices it):
```csharp
// Projected, untracked fetch: the fold only reads these six fields and never
// mutates a KpiSample. A tracked ToListAsync registered ~5k-90k read-only
// entities in the change tracker per healthy 3h fold — all re-scanned by
// DetectChanges on the final SaveChanges (arch-review 04 round 2, R2). A
// projection is inherently untracked and materializes no entity at all.
var samples = await _context.KpiSamples
.Where(s => s.CapturedAtUtc >= from && s.CapturedAtUtc < to)
.Select(s => new FoldSample(s.Source, s.Metric, s.Scope, s.ScopeKey, s.CapturedAtUtc, s.Value))
.ToListAsync(cancellationToken);
```
with, at the bottom of the class next to `SeriesHourKey`:
```csharp
/// <summary>Narrow, untracked projection of one KpiSample row for the fold (R2).</summary>
private readonly record struct FoldSample(
string Source, string Metric, string Scope, string? ScopeKey, DateTime CapturedAtUtc, double Value);
```
The grouping/aggregation lambdas (:112-128) compile unchanged — `FoldSample` carries the same member names.
4. Run the new filter — PASS. Then run the whole fold suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter KpiHistoryRepositoryTests` — PASS (all existing fold/purge/query tests must stay green; this is a pure fetch-shape change).
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/KpiHistoryRepositoryTests.cs && git commit -m "fix(kpi): untracked projected fold fetch — fold reads no longer flood the change tracker (plan R2-04 T1)"`
### Task 2: Rollup watermark seam — `GetLatestRollupHourAsync` (R1, part 1)
**Classification:** standard
**Estimated implement time:** ~4 min
**Parallelizable with:** 5, 8, 10, 11, 12 (NOT 1/9 — same repository file; NOT 3/4/6 — the interface addition ripples into their test fakes)
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IKpiHistoryRepository.cs` (additive method)
- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs` (implement)
- Modify: every test fake implementing `IKpiHistoryRepository` — locate with `grep -rln ": IKpiHistoryRepository" tests/` (expect the recorder-actor fake in `tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests/KpiHistoryRecorderActorTests.cs` and the query-service stub in `tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Services/KpiHistoryQueryServiceTests.cs`) — add a trivial `Task.FromResult<DateTime?>(null)` member so they compile
- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/KpiHistoryRepositoryTests.cs` (extend)
1. Failing tests:
```csharp
[Fact]
public async Task GetLatestRollupHour_ReturnsNull_WhenNoRollupsExist()
{
await using var ctx = NewContext();
var repo = new KpiHistoryRepository(ctx);
Assert.Null(await repo.GetLatestRollupHourAsync());
}
[Fact]
public async Task GetLatestRollupHour_ReturnsNewestHourStart_AcrossAllSeries()
{
await using var ctx = NewContext();
var repo = new KpiHistoryRepository(ctx);
await repo.RecordSamplesAsync(new[]
{
Sample("NotificationOutbox", "queueDepth", "Global", null, 5, Base.AddMinutes(10)),
Sample("SiteCallAudit", "buffered", "Global", null, 2, Base.AddHours(3).AddMinutes(10)),
});
await repo.FoldHourlyRollupsAsync(Base, Base.AddHours(4));
Assert.Equal(Base.AddHours(3), await repo.GetLatestRollupHourAsync());
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter GetLatestRollupHour` — expect FAIL (method does not exist / does not compile — comment the tests out to prove the baseline if needed, standard TDD-on-new-member flow).
3. Add to `IKpiHistoryRepository` (after `GetHourlySeriesAsync`, ~:108):
```csharp
/// <summary>
/// Returns the newest <c>KpiRollupHourly.HourStartUtc</c> across all series, or
/// <c>null</c> when no rollups exist. The recorder's one-shot backfill consults
/// this watermark to skip (or shrink to) the un-rolled tail instead of re-folding
/// the whole raw-retention window on every failover (arch-review 04 round 2, R1).
/// </summary>
Task<DateTime?> GetLatestRollupHourAsync(CancellationToken cancellationToken = default);
```
Implement in `KpiHistoryRepository`: `return await _context.KpiRollupHourly.MaxAsync(r => (DateTime?)r.HourStartUtc, cancellationToken);` — a single MAX over the unique `IX_KpiRollupHourly_Series` population (rollup table is small: one row per series-hour). Add the null-returning member to each located test fake.
4. Run the filter — PASS. Build the solution (`dotnet build ZB.MOM.WW.ScadaBridge.slnx`) to prove no other implementor was missed.
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IKpiHistoryRepository.cs src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs tests && git commit -m "feat(kpi): GetLatestRollupHourAsync watermark seam for the backfill fast-path (plan R2-04 T2)"`
### Task 3: Slice the rollup backfill into bounded day windows (R1, part 2)
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** 5, 6, 7, 8, 10, 11, 12 (NOT 4 — same actor file; requires Task 2's fake update)
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.KpiHistory/KpiHistoryRecorderActor.cs` (`HandleBackfillTick` :494-526, `RunBackfillPass` :536-558, the `BackfillComplete` receive :163-167)
- Test: `tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests/KpiHistoryRecorderActorTests.cs` (extend + update the two existing backfill tests)
1. Failing test (reuse the gated-fold fake-repository rig the existing `BackfillTick_FoldsFullRetentionWindow_Once` / `BackfillInFlight_SkipsPeriodicRollupTick` tests drive by hand):
```csharp
[Fact]
public void Backfill_SlicesRetentionWindow_IntoBoundedDayFolds_OldestFirst()
{
// RetentionDays = 3 → expect 3+ fold calls, each window ≤ 24 h, contiguous,
// oldest-first, whose union is exactly [TruncateToHour(now) − 3 d, TruncateToHour(now)).
// Today: FAIL — a single fold call spans the whole 3-day window.
}
[Fact]
public void PeriodicRollupTick_Interleaves_BetweenBackfillSlices()
{
// Gate slice 1 open; release it; before dequeuing slice 2 send a RollupTick —
// the periodic fold must claim the fold path (the backfill defers via its
// re-arm branch) and ALL backfill slices must still complete afterwards.
// Today: FAIL — _backfillInFlight blocks every periodic fold for the whole pass.
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests --filter "Backfill_SlicesRetentionWindow|PeriodicRollupTick_Interleaves"` — expect FAIL.
3. Implementation — convert the single full-window pass into a slice queue processed one bounded window per fold, releasing the fold path between slices:
- New constant + state:
```csharp
/// <summary>
/// Width of one backfill fold slice (24 h). The repository fold materializes its whole
/// window in memory, so the one-shot backfill must never hand it the full raw-retention
/// window (90 d ≈ tens of millions of rows at fleet volume — arch-review 04 round 2, R1).
/// One day per fold bounds each pass to roughly what a steady-state day writes, and the
/// guard is lowered between slices so periodic folds interleave instead of stalling
/// behind the historical pass.
/// </summary>
private static readonly TimeSpan BackfillSliceWidth = TimeSpan.FromHours(24);
private readonly Queue<(DateTime FromHourUtc, DateTime ToHourUtc)> _backfillSlices = new();
```
- `HandleBackfillTick`: keep the `_backfillDone || _backfillInFlight` and `_rollupInFlight`-defer branches verbatim. When `_backfillSlices` is empty, build the plan — enumerate `[toHourUtc − RetentionDays, toHourUtc)` into ≤24h windows oldest-first and enqueue them (log the one "backfill starting" line with the slice count); when it is non-empty, dequeue ONE slice, raise `_backfillInFlight`, and run `RunBackfillPass(slice.FromHourUtc, slice.ToHourUtc, ct)` piping `BackfillSliceComplete.Instance` on both success and (logged) failure.
- Replace the `BackfillComplete` receive with:
```csharp
Receive<BackfillSliceComplete>(_ =>
{
_backfillInFlight = false; // release the fold path BETWEEN slices (R1)
if (_backfillSlices.Count == 0)
{
_backfillDone = true; // latch: at most once per actor lifetime (unchanged semantics)
_logger.LogInformation("KPI rollup backfill completed — all slices folded.");
return;
}
// Next slice via the timer, not inline: a RollupTick already queued in the mailbox
// is processed first, so periodic folds interleave; the existing _rollupInFlight
// deferral branch in HandleBackfillTick then re-arms the backfill behind it.
Timers.StartSingleTimer(BackfillTimerKey, BackfillTick.Instance, TimeSpan.Zero);
});
```
Rename the `BackfillComplete` message class to `BackfillSliceComplete` (internal — used only by this actor and its tests). Move the "backfill completed" wall-time log from `RunBackfillPass` (which now logs per-slice at Debug) to the queue-drained branch. Ordering note in the class doc: periodic folds and backfill slices strictly alternate through the single `_rollupInFlight`/`_backfillInFlight` gate pair — the two idempotent upserts still never race on overlapping rows.
- Update `BackfillTick_FoldsFullRetentionWindow_Once` (assert the union of slice windows covers the retention window and a post-completion `BackfillTick` is a no-op) and `BackfillInFlight_SkipsPeriodicRollupTick` (unchanged semantics — a RollupTick arriving while a *slice* is in flight is still skipped).
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests --filter KpiHistoryRecorderActorTests` — PASS (all pre-existing sample/purge/rollup tests must stay green).
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.KpiHistory/KpiHistoryRecorderActor.cs tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests/KpiHistoryRecorderActorTests.cs && git commit -m "fix(kpi): slice the rollup backfill into bounded 24h folds that yield to periodic folds (plan R2-04 T3)"`
### Task 4: Backfill failover fast-path via the rollup watermark + doc correction (R1, part 3)
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** 5, 6, 8, 10, 11, 12 (NOT 3 — same actor file, do 3 first; NOT 7/9 — same design-doc file)
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.KpiHistory/KpiHistoryRecorderActor.cs` (plan branch of `HandleBackfillTick` from Task 3)
- Modify: `docs/requirements/Component-KpiHistory.md` (:93, the "One-shot backfill" paragraph)
- Test: `tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests/KpiHistoryRecorderActorTests.cs` (extend)
1. Failing tests (the fake repository's `GetLatestRollupHourAsync` — added in Task 2 — gains a settable return value):
```csharp
[Fact]
public void Backfill_Skips_WhenRollupsAlreadyCurrent()
{
// Fake watermark = TruncateToHour(now) − 1 h (i.e. within RollupLookbackHours of now —
// the failover case, which is the common one): expect ZERO fold calls, the done-latch
// set, and an Information log; the periodic fold alone covers the lookback tail.
}
[Fact]
public void Backfill_ShrinksToWatermark_InsteadOfRetentionFloor()
{
// RetentionDays = 90, fake watermark = 2 days ago: expect the first slice to start at
// the WATERMARK HOUR (inclusive — re-folding the newest already-folded hour is a cheap
// idempotent safety margin), not 90 days ago, and ≤ 3 slices total.
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests --filter "Backfill_Skips_WhenRollupsAlreadyCurrent|Backfill_ShrinksToWatermark"` — expect FAIL (the plan branch always enumerates from the retention floor).
3. Implementation — make the plan branch asynchronous: when `_backfillSlices` is empty, raise `_backfillInFlight` and run a plan pass that resolves `IKpiHistoryRepository` in a fresh scope and returns `GetLatestRollupHourAsync()` (never-faulting wrapper like the other passes), piping `BackfillPlan(DateTime? Watermark)` back to Self. In the `Receive<BackfillPlan>` handler (actor thread):
```csharp
var toHourUtc = TruncateToHour(DateTime.UtcNow);
var retentionFloor = toHourUtc - TimeSpan.FromDays(_options.RetentionDays);
// Shrink to the un-rolled tail; re-fold the watermark hour itself (idempotent) as a
// safety margin. A null watermark (fresh install) keeps the full retention window.
var floor = plan.Watermark is { } w && w > retentionFloor ? w : retentionFloor;
if (floor >= toHourUtc - TimeSpan.FromHours(_options.RollupLookbackHours))
{
// Failover fast-path: rollups are already current — the periodic fold's lookback
// window covers everything from the watermark forward, so the full historical
// pass is skipped entirely (arch-review 04 round 2, R1).
_backfillInFlight = false;
_backfillDone = true;
_logger.LogInformation(
"KPI rollup backfill skipped — rollups current through {Watermark:o}.", plan.Watermark);
return;
}
// enqueue ≤24h slices over [floor, toHourUtc) oldest-first (Task 3's builder),
// lower _backfillInFlight, then self-arm BackfillTick to dequeue the first slice.
```
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests --filter KpiHistoryRecorderActorTests` — PASS.
5. Doc: rewrite `Component-KpiHistory.md:93` — the backfill folds the window in **≤24h slices oldest-first**, yielding to periodic folds between slices; on a failover restart it consults the `GetLatestRollupHourAsync` watermark and **skips** (rollups current) or **shrinks to the un-rolled tail**. Drop the "cheap and safe to re-run" claim — replace with "safe (idempotent) to re-run; the watermark fast-path makes the failover re-run cheap as well" (the review's exact complaint: the "safe" was true, the "cheap" was not).
6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.KpiHistory/KpiHistoryRecorderActor.cs docs/requirements/Component-KpiHistory.md tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests/KpiHistoryRecorderActorTests.cs && git commit -m "fix(kpi): backfill skips or shrinks to the rollup watermark on failover restarts (plan R2-04 T4)"`
### Task 5: Per-metric reduction in the bucketer — sum-per-bucket for Rate series (R3, part 1)
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** 1, 2, 3, 4, 8, 10, 11, 12
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Types/Kpi/KpiSeriesBucketer.cs`
- Test: `tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Kpi/KpiSeriesBucketerTests.cs` (extend)
1. Failing tests:
```csharp
[Fact]
public void Bucket_RateSeries_SumsPerBucket_InsteadOfLastValue()
{
// 6 points of value 10 across a window bucketed to 2 → each bucket = 30 (sum),
// not 10 (last value). Today: FAIL (last-value keeps 1 of 3 deltas per bucket —
// the exact "keep one delta, discard the others" fold error the catalog warns
// about, re-introduced one layer up: arch-review 04 round 2, R3).
}
[Fact]
public void Bucket_RateSeries_ShortSeries_StillSumsClusteredPoints()
{
// 3 points (< maxPoints) with two sharing a bucket → the shared bucket is their SUM.
// Rate series skip the raw.Count <= maxPoints early return so totals stay truthful.
}
[Fact]
public void Bucket_RateSeries_PreservesSeriesTotal()
{
// Strong invariant: sum(output values) == sum(in-window input values) for Rate.
}
[Fact]
public void Bucket_DefaultAggregation_IsGauge_AndBehaviorUnchanged()
{
// Calling the existing 4-arg shape produces byte-identical output to before.
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter KpiSeriesBucketerTests` — expect FAIL.
3. Implementation — additive optional parameter (every existing call site compiles unchanged):
```csharp
public static IReadOnlyList<KpiSeriesPoint> Bucket(
IReadOnlyList<KpiSeriesPoint> raw,
DateTime fromUtc,
DateTime toUtc,
int maxPoints,
KpiRollupAggregation aggregation = KpiRollupAggregation.Gauge)
```
- Keep the argument guards and the null/empty return unchanged. Restrict the `raw.Count <= maxPoints` early return to **Gauge only**: for Rate, two deltas landing in one bucket must sum even in a short series (for well-spaced short series the per-point buckets make sum == identity, so output ≈ input anyway).
- In the loop, branch per aggregation: Gauge keeps the existing last-value candidate logic verbatim; Rate accumulates `sums[bucketIndex] += point.Value` (plus the same `occupied` flag), emitting `new KpiSeriesPoint(bucketStart, sums[i])` in the collection pass.
- Update the class/method XML doc: "last-value-per-bucket / gauge semantics" becomes per-metric — Gauge = last value in the bucket (unchanged default), Rate = **sum-per-bucket** (each raw point of a Rate series is a per-interval delta, so the bucket's true total is the sum of its deltas; both the raw-minute and hourly-rollup paths then chart the same unit, "events per chart bucket", eliminating the ~60× jump at the routing boundary — R3).
4. Run the filter — PASS. Also `dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter "FullyQualifiedName~Kpi"` — PASS (catalog + point tests untouched).
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Commons/Types/Kpi/KpiSeriesBucketer.cs tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Kpi/KpiSeriesBucketerTests.cs && git commit -m "fix(kpi): per-metric bucket reduction — Rate series sum per bucket instead of keeping one delta (plan R2-04 T5)"`
### Task 6: Catalog-driven aggregation at the query-service boundary (R3, part 2)
**Classification:** standard
**Estimated implement time:** ~4 min
**Parallelizable with:** 1, 3, 4, 8, 10, 11, 12 (requires Task 5)
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/KpiHistoryQueryService.cs` (`GetSeriesAsync` :73-94, `FetchSeriesAsync` doc :96-106)
- Test: `tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Services/KpiHistoryQueryServiceTests.cs` (extend)
1. Failing tests (stub-repository rig already in the file):
```csharp
[Fact]
public async Task GetSeriesAsync_RateMetric_SumsPerBucket_OnRollupPath()
{
// (KpiSources.SiteCallAudit, KpiMetrics.SiteCallAudit.FailedLastInterval) is a
// catalog Rate pair. Stub GetHourlySeriesAsync with 400 hourly points of value 1
// over a >168h window, maxPoints 200 → every output value is 2.0 (sum of the two
// hourly sums per bucket), not 1.0 (last value). Today: FAIL.
}
[Fact]
public async Task GetSeriesAsync_RateMetric_PreservesTotal_AcrossRoutingBoundary()
{
// Same total event count served as per-minute deltas (raw path, window == threshold)
// and as hourly sums (rollup path, window just over threshold): the summed chart
// total must be equal on both paths — the boundary is now visually seamless (R3).
}
[Fact]
public async Task GetSeriesAsync_GaugeMetric_KeepsLastValuePerBucket()
{
// queueDepth (uncatalogued → Gauge) keeps the existing last-value output unchanged.
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests --filter KpiHistoryQueryServiceTests` — expect FAIL.
3. Implementation in `GetSeriesAsync` (both ctor paths):
```csharp
// Per-metric reduction intent: the same catalog that drives the hourly fold drives
// the chart-time bucket reduction, so a Rate series is summed per bucket on BOTH the
// raw-minute and hourly-rollup routes — one unit ("events per chart bucket") on both
// sides of the RollupThresholdHours routing boundary (arch-review 04 round 2, R3).
var aggregation = KpiMetricAggregationCatalog.Resolve(source, metric);
...
return KpiSeriesBucketer.Bucket(injectedSeries, fromUtc, toUtc, effectiveMax, aggregation);
...
return KpiSeriesBucketer.Bucket(series, fromUtc, toUtc, effectiveMax, aggregation);
```
Extend the `FetchSeriesAsync` doc comment: the routing boundary no longer changes a Rate chart's magnitude — both paths reduce to per-bucket sums (a residual ≤~1.2× native-resolution difference exists only for an unbucketed just-over-threshold rollup series vs. an at-threshold bucketed raw series; documented in Task 7).
4. Run the filter — PASS (the pre-existing routing/forwarding tests must stay green — Gauge default preserves their expectations).
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/KpiHistoryQueryService.cs tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Services/KpiHistoryQueryServiceTests.cs && git commit -m "fix(kpi): query service applies catalog aggregation at the bucketer — rate charts keep one unit across the raw/rollup boundary (plan R2-04 T6)"`
### Task 7: Truthful trend presentation — chart doc + Component-KpiHistory.md (R3, part 3)
**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** 1, 2, 3, 8, 10, 11, 12 (requires Task 6; NOT 4/9 — same design-doc file)
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Shared/KpiTrendChart.razor.cs` (XML docs on `Points` :~55 and `Unit` :~62 — doc-only, no markup/behavior change; the component stays custom-SVG, no third-party libs)
- Modify: `docs/requirements/Component-KpiHistory.md` (:151 "Bucketed query", the "Raw-vs-rollup range routing" section :~153-155, and one line under ":157 Aggregation intent")
- Modify: trend-panel `Unit` strings only where they now misstate the semantics — locate with `grep -rn "Unit=" src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages` and fix any Rate-metric panel whose unit text claims a per-minute/per-interval reading (e.g. the `deliveredLastInterval` / `failedLastInterval` panels) to say per-bucket totals (e.g. `Unit="delivered/bucket"`); leave Gauge panels untouched
1. `KpiTrendChart.razor.cs` doc: on `Points`, note that Rate-classified series (per `KpiMetricAggregationCatalog`) now carry **per-bucket totals** (sum reduction), while Gauge series carry last-value-per-bucket readings; on `Unit`, note the supplied text must state the per-bucket semantics for Rate metrics.
2. `Component-KpiHistory.md:151`: replace "returns the **last value per bucket**" with the per-metric reduction — `KpiSeriesBucketer.Bucket(series, fromUtc, toUtc, maxPoints, aggregation)` keeps the last value for Gauge series and **sums per bucket** for Rate series, driven by the same `KpiMetricAggregationCatalog` the fold consults (the previous last-value-on-rollups behavior kept 1 of ~11 hourly sums per 90d bucket — the fold error the catalog's own doc warns about, fixed per arch-review 04 round 2, R3). In the range-routing section add: "Rate charts keep one unit (events per chart bucket) on both sides of the boundary; widening 7d→30d no longer changes the magnitude ~60×. Residual: an unbucketed just-over-threshold rollup series plots native per-hour sums vs. the at-threshold raw path's ~50-minute bucket sums (≤~1.2×) — accepted and documented, not hidden." Under Aggregation intent, note the catalog now has two consumers: the hourly fold and the chart-time bucket reduction.
3. Verify no stale claim remains: `grep -n "last value per bucket" docs/requirements/Component-KpiHistory.md` returns nothing.
4. Run the chart component suite to prove the doc-only touch broke nothing: `dotnet test tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests --filter KpiTrendChartTests` — PASS.
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.CentralUI docs/requirements/Component-KpiHistory.md && git commit -m "docs(kpi): trend charts state per-bucket-total semantics for rate metrics (plan R2-04 T7)"`
### Task 8: Close the catalog metric-literal drift hazard (R4)
**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** 1, 2, 3, 4, 5, 10, 11, 12
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Types/Kpi/KpiMetrics.cs` (add three constants + amend the class remark)
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Types/Kpi/KpiMetricAggregation.cs` (:60-64 private literals; :98-107 `RatePairs`)
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/Kpi/SiteCallAuditKpiSampleSource.cs` (:45)
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/Kpi/SiteHealthKpiSampleSource.cs` (:43, :51)
- Test: `tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Kpi/KpiMetricAggregationTests.cs` (extend)
1. Failing test (does not compile until the constants exist — the TDD gate for symbol promotion):
```csharp
[Fact]
public void PromotedMetricConstants_LockHistoricalPersistedValues()
{
// Persisted data contract — these are symbol promotions, NOT renames.
Assert.Equal("deliveredLastInterval", KpiMetrics.SiteCallAudit.DeliveredLastInterval);
Assert.Equal("alarmEvalErrors", KpiMetrics.SiteHealth.AlarmEvalErrors);
Assert.Equal("eventLogWriteFailures", KpiMetrics.SiteHealth.EventLogWriteFailures);
}
[Theory]
[InlineData(KpiSources.SiteCallAudit, KpiMetrics.SiteCallAudit.DeliveredLastInterval)]
[InlineData(KpiSources.SiteHealth, KpiMetrics.SiteHealth.AlarmEvalErrors)]
[InlineData(KpiSources.SiteHealth, KpiMetrics.SiteHealth.EventLogWriteFailures)]
public void Resolve_PromotedRatePairs_StillClassifyAsRate(string source, string metric)
=> Assert.Equal(KpiRollupAggregation.Rate, KpiMetricAggregationCatalog.Resolve(source, metric));
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter KpiMetricAggregationTests` — expect FAIL (compile error: constants absent).
3. Implementation — compile-time coupling in the only dependency-legal direction (Commons cannot reference the source projects, so the shared constant lives in `KpiMetrics` and both the catalog and the emitting source consume it):
- `KpiMetrics.SiteCallAudit`: add `public const string DeliveredLastInterval = "deliveredLastInterval";`; `KpiMetrics.SiteHealth`: add `AlarmEvalErrors = "alarmEvalErrors"` and `EventLogWriteFailures = "eventLogWriteFailures"` (EXACT existing string values — persisted, renames forbidden).
- Amend the `KpiMetrics` class remark (:20-27): the catalog now also carries metrics the **rollup aggregation catalog** classifies, not only charted ones — the C4/R4 lesson is that any metric named in two places needs one shared symbol.
- `KpiMetricAggregationCatalog`: delete the three private literals (:62-64) and reference the constants in `RatePairs`; update the "otherwise the source's own private literal" remark (:46-48) — every catalogued name is now a shared `KpiMetrics` constant, so an emitter rename is a compile error, not a silent Gauge downgrade.
- The two sources keep their private-const style with the value redefined in terms of the shared symbol (tiny diff, zero call-site churn): `private const string MetricDeliveredLastInterval = KpiMetrics.SiteCallAudit.DeliveredLastInterval;` (SiteCallAudit :45) and likewise for the two SiteHealth consts (:43, :51).
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter KpiMetricAggregationTests && dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests --filter SiteCallAuditKpiSampleSource && dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests --filter SiteHealthKpiSampleSource` — PASS (emitted strings unchanged).
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Commons src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/Kpi/SiteCallAuditKpiSampleSource.cs src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/Kpi/SiteHealthKpiSampleSource.cs tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Kpi/KpiMetricAggregationTests.cs && git commit -m "fix(kpi): promote catalog metric literals to shared KpiMetrics constants — drift becomes a compile error (plan R2-04 T8)"`
### Task 9: Classify the failover fold-race failure grain (R5)
**Classification:** standard
**Estimated implement time:** ~4 min
**Parallelizable with:** 3, 5, 6, 8, 10, 11, 12 (NOT 1/2 — same repository file, do them first; NOT 4/7 — same design-doc file)
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs` (ctor + the fold's `SaveChangesAsync` at :167)
- Modify: `docs/requirements/Component-KpiHistory.md` (one sentence in the "Hourly rollup tick" paragraph, :~91)
- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/KpiHistoryRepositoryTests.cs` (extend)
1. Failing test — force a REAL unique violation deterministically with a `SaveChangesInterceptor` that, on the fold's first save, inserts the conflicting `(series, hour)` rollup row through a separate command on the same in-memory SQLite connection (simulating the old singleton incarnation's in-flight fold winning the race):
```csharp
[Fact]
public async Task Fold_LosingFailoverUpsertRace_DoesNotThrow_AndNextFoldSelfHeals()
{
// ctx built with .AddInterceptors(new InsertConflictingRollupOnFirstSave(connection))
// seed one sample in a complete hour; FoldHourlyRollupsAsync must NOT propagate
// DbUpdateException (today: FAIL — the whole pass faults);
// a second FoldHourlyRollupsAsync over the same window must land the row (self-heal).
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter Fold_LosingFailoverUpsertRace` — expect FAIL.
3. Implementation:
- Ctor gains `ILogger<KpiHistoryRepository>? logger = null` defaulting to `NullLogger` (mirrors `SiteCallAuditRepository` — MS.DI resolves `ILogger<>` automatically, no registration churn).
- Wrap the fold's final `SaveChangesAsync`:
```csharp
try
{
await _context.SaveChangesAsync(cancellationToken);
}
catch (DbUpdateException ex)
{
// Failover-overlap race: the old singleton incarnation's in-flight fold and the
// new node's first fold can both Add the same (series, hour) row; the unique
// IX_KpiRollupHourly_Series then faults the loser's ENTIRE SaveChanges — the
// failure grain is the PASS, not the row (arch-review 04 round 2, R5). The fold
// only ever writes KpiRollupHourly rows, so any save fault here is a lost upsert
// race or a transient the next idempotent re-fold repairs identically; classify
// at Information instead of surfacing a scary error for a self-healing no-op.
_logger.LogInformation(ex,
"KPI rollup fold lost a failover-overlap upsert race; pass discarded, next fold self-heals.");
}
```
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter KpiHistoryRepositoryTests` — PASS.
5. Doc: add to the rollup-tick paragraph of `Component-KpiHistory.md`: "During a failover overlap the loser of the `(series, hour)` upsert race discards its whole fold pass (single `SaveChanges`) — logged at Information, repaired by the next idempotent fold; when reading fold logs after a failover, the failure grain is the pass, not the row."
6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs docs/requirements/Component-KpiHistory.md tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/KpiHistoryRepositoryTests.cs && git commit -m "fix(kpi): classify the failover fold-race pass loss as a self-healing Information event (plan R2-04 T9)"`
### Task 10: SiteCalls filtered terminal index (migration) (R6, part 1)
**Classification:** high-risk
**Estimated implement time:** ~4 min
**Parallelizable with:** 1, 2, 3, 4, 5, 6, 7, 8, 9, 12 (the plan's only migration — but serialize against any OTHER plan adding ConfigurationDatabase migrations: model-snapshot ordering)
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/SiteCallEntityTypeConfiguration.cs` (after the `IX_SiteCalls_NonTerminal` block ending :~99)
- Create: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/<timestamp>_AddSiteCallsTerminalIndex.cs` (scaffolded)
- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/SchemaConfigurationTests.cs` (extend)
1. Failing model test (SchemaConfigurationTests already asserts index shapes):
```csharp
[Fact]
public void SiteCalls_Has_Filtered_Terminal_Index()
{
var entity = Model.FindEntityType(typeof(SiteCall))!;
var index = entity.GetIndexes().Single(i => i.GetDatabaseName() == "IX_SiteCalls_Terminal");
Assert.Equal(new[] { nameof(SiteCall.TerminalAtUtc) }, index.Properties.Select(p => p.Name).ToArray());
Assert.Equal("[TerminalAtUtc] IS NOT NULL", index.GetFilter());
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter SiteCalls_Has_Filtered_Terminal_Index` — expect FAIL.
3. Add to `SiteCallEntityTypeConfiguration.Configure`:
```csharp
// Terminal backs the daily retention purge's predicate (TerminalAtUtc IS NOT NULL
// AND TerminalAtUtc < cutoff) and Task 11's MIN() slice anchor: filtered to the
// terminal population so the purge seeks the expired tail instead of full-scanning
// the 365-day table (arch-review 04 round 2, R6). Complements IX_SiteCalls_NonTerminal,
// which is filtered to IS NULL and unusable for this predicate.
builder.HasIndex(s => s.TerminalAtUtc)
.HasDatabaseName("IX_SiteCalls_Terminal")
.HasFilter("[TerminalAtUtc] IS NOT NULL");
```
4. Scaffold the migration — **repo gotcha: BUILD FIRST, never `dotnet ef migrations add --no-build`** (it scaffolds an EMPTY migration off the stale previously-built DLL; if that happens, delete the empty `Migrations/` files — `migrations remove` needs a live DB):
```bash
dotnet build ZB.MOM.WW.ScadaBridge.slnx
cd src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase
dotnet ef migrations add AddSiteCallsTerminalIndex
```
Verify the scaffold contains the `CreateIndex` with the `IS NOT NULL` filter — if it is empty, the gotcha bit; delete and redo.
5. Run the model test — PASS. With infra up (`cd infra && docker compose up -d`), run the MSSQL-backed suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter SiteCallAuditRepositoryTests` — PASS. Production script (repo convention — manual SQL for production): `dotnet ef migrations script AddKpiRollupHourlyTable AddSiteCallsTerminalIndex --idempotent --output ../../docs/plans/sql/AddSiteCallsTerminalIndex.sql` (verify the from-migration is still the latest applied; review the output).
6. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/SchemaConfigurationTests.cs docs/plans/sql && git commit -m "perf(sitecallaudit): filtered IX_SiteCalls_Terminal index backs the retention purge predicate (plan R2-04 T10)"`
### Task 11: Time-sliced SiteCalls terminal purge (R6, part 2)
**Classification:** standard
**Estimated implement time:** ~4 min
**Parallelizable with:** 1, 2, 3, 4, 5, 6, 7, 8, 9, 12 (requires Task 10 — the MIN() anchor and DELETE predicate seek its index)
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs` (`PurgeTerminalAsync` :247-252)
- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/SiteCallAuditRepositoryTests.cs` (extend; MSSQL `SkippableFact` — infra up)
1. Failing test (pattern-match the fixture usage of the existing `SkippableFact` tests in the file; assert slicing via an EF `DbCommandInterceptor` capturing DELETE statements, or SQL capture matching how sibling tests inspect SQL):
```csharp
[SkippableFact]
public async Task PurgeTerminal_SlicesMultiDayBacklog_IntoBoundedDeletes()
{
// Seed terminal rows spread across 3 distinct days beyond the cutoff, one terminal
// row inside the window, and one OLD non-terminal row.
// Purge → returns 3-day total; the fresh terminal row and the old NON-terminal row
// survive (non-terminal rows are NEVER purged on age — sites remain source of truth);
// captured SQL shows MORE THAN ONE DELETE statement for the multi-day backlog.
// Today: FAIL — a single unbatched DELETE is issued.
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter PurgeTerminal_SlicesMultiDayBacklog` — expect FAIL.
3. Implement time-sliced batching mirroring `KpiHistoryRepository.PurgeOlderThanAsync` (the round-1 Task 19 shape), keeping the raw-T-SQL MSSQL-only style this repository already uses:
```csharp
public async Task<int> PurgeTerminalAsync(DateTime olderThanUtc, CancellationToken ct = default)
{
// Time-sliced batches (arch-review 04 round 2, R6 — the one maintenance DELETE
// that missed round 1's batching pass): each DELETE covers at most one DAY of
// terminal rows, capping the lock/log footprint per statement. Steady state
// (daily purge, 365-day retention) is a single slice; only catch-up after an
// outage runs several. One-day (not one-hour) slices are proportionate to
// SiteCalls volume, which is far below KpiSample's. The MIN() anchor and the
// DELETE predicate both seek IX_SiteCalls_Terminal (filtered IS NOT NULL).
var total = 0;
var floor = await _context.SiteCalls
.Where(s => s.TerminalAtUtc != null && s.TerminalAtUtc < olderThanUtc)
.MinAsync(s => s.TerminalAtUtc, ct);
while (floor is not null && floor < olderThanUtc)
{
var ceiling = floor.Value.AddDays(1) < olderThanUtc ? floor.Value.AddDays(1) : olderThanUtc;
total += await _context.Database.ExecuteSqlInterpolatedAsync(
$"DELETE FROM dbo.SiteCalls WHERE TerminalAtUtc IS NOT NULL AND TerminalAtUtc < {ceiling};",
ct);
floor = await _context.SiteCalls
.Where(s => s.TerminalAtUtc != null && s.TerminalAtUtc < olderThanUtc)
.MinAsync(s => s.TerminalAtUtc, ct);
}
return total;
}
```
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter SiteCallAuditRepositoryTests` — PASS (upsert/query/KPI tests untouched). Then the purge-actor contract: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests --filter SiteCallAuditPurgeTests` — PASS (return-total contract unchanged).
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/SiteCallAuditRepositoryTests.cs && git commit -m "fix(sitecallaudit): time-sliced terminal purge replaces the single year-scale DELETE (plan R2-04 T11)"`
### Task 12: Retention-service shutdown no longer surfaces cancellation (R7)
**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SiteAuditRetentionService.cs` (`RunLoopAsync` :62-91)
- Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SiteAuditRetentionServiceTests.cs` (extend; the `RecordingSiteAuditQueue` stub gains a block-until-cancelled mode)
1. Failing test:
```csharp
[Fact]
public async Task StopAsync_MidPurge_CompletesCleanly_WithoutSurfacingCancellation()
{
var queue = new RecordingSiteAuditQueue { BlockUntilCancelled = true }; // PurgeExpiredAsync: await Task.Delay(Timeout.Infinite, ct)
var options = Options.Create(new SiteAuditRetentionOptions
{ RetentionDays = 7, PurgeInterval = TimeSpan.FromHours(24), InitialDelay = TimeSpan.Zero });
using var svc = new SiteAuditRetentionService(queue, options, NullLogger<SiteAuditRetentionService>.Instance);
await svc.StartAsync(CancellationToken.None);
await WaitUntilAsync(() => queue.PurgeCalls.Count >= 1, TimeSpan.FromSeconds(5));
// Today: FAIL — the OCE SafePurgeAsync deliberately rethrows escapes RunLoopAsync
// (the two awaits at :76/:89 sit outside any try/catch), cancels _loop, and
// StopAsync returns that canceled task straight to the host (:122-126), whose
// await throws TaskCanceledException into every shutdown log (R7).
await svc.StopAsync(CancellationToken.None); // must NOT throw
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter StopAsync_MidPurge` — expect FAIL (`TaskCanceledException`).
3. Implementation — catch OCE at the loop boundary (the report's preferred alignment with the sibling `SiteAuditBacklogReporter` pattern), leaving `SafePurgeAsync`'s deliberate rethrow (:106-110) untouched so a mid-purge shutdown still aborts the purge promptly:
```csharp
private async Task RunLoopAsync(CancellationToken ct)
{
try
{
// ... existing body verbatim (initial delay, first purge, delay/purge loop) ...
}
catch (OperationCanceledException)
{
// Shutdown landed mid-purge: SafePurgeAsync rethrows OCE by design so the purge
// aborts promptly, but the loop task must complete CLEANLY — StopAsync hands
// _loop straight to the host, and a canceled task there is shutdown-log noise
// (arch-review 04 round 2, R7). Cancellation here IS the clean exit.
}
}
```
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter SiteAuditRetentionServiceTests` — PASS (the two existing tick tests must stay green).
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SiteAuditRetentionService.cs tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SiteAuditRetentionServiceTests.cs && git commit -m "fix(audit-log): retention loop absorbs shutdown cancellation instead of faulting host StopAsync (plan R2-04 T12)"`
---
## Dependencies on other plans
- None of the round-1 plans have open tasks (the whole arch-review initiative closed 191/192 on 2026-07-10); no cross-plan file contention is expected.
- Task 10 is this plan's **only EF migration**. No other plan currently adds ConfigurationDatabase migrations, but if one appears, serialize on the model snapshot (same rule as round-1 Tasks 6→8).
- The `IKpiHistoryRepository` addition (Task 2) is additive; any concurrently developed fake/stub implementing the interface must add the member — build the solution after Task 2 to surface stragglers immediately.
## Execution order
**P0 (the High + its enablers, in order):** 1 → 2 → 3 → 4 (fold fetch, watermark seam, sliced backfill, failover fast-path).
**P1:** 5 → 6 → 7 (rate presentation chain), 10 → 11 (SiteCalls purge chain).
**P2 (low-severity cleanup):** 8, 9, 12 — broadly parallel per each task's "Parallelizable with" contract (watch the two serialization lanes: `KpiHistoryRepository.cs` = 1 → 2 → 9; `Component-KpiHistory.md` = 4 → 7 → 9 in whatever order they run, never concurrently).
Finish with `dotnet build ZB.MOM.WW.ScadaBridge.slnx` plus the targeted suites this plan touched — `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter "KpiHistoryRepositoryTests|SiteCallAuditRepositoryTests|SchemaConfigurationTests"` (infra up for the MSSQL `SkippableFact`s), `dotnet test tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests`, `dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter "FullyQualifiedName~Kpi"`, `dotnet test tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests --filter "KpiHistoryQueryServiceTests|KpiTrendChartTests"`, `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests`, `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests --filter SiteHealthKpiSampleSource`, and `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter "FullyQualifiedName~Site"` — before declaring the plan done.
## Findings Coverage
| # | Report finding (severity) | Task(s) |
|---|---------------------------|---------|
| R1 | Startup backfill folds the entire raw-retention window in one unbounded pass, re-run on every failover (High) | 2, 3, 4 |
| R2 | Fold fetch is change-tracked — `AsNoTracking`/projection missing (Medium) | 1 |
| R3 | Rate metrics change units ~60× across the raw/rollup routing boundary; bucketer re-introduces the last-value fold error on hourly Rate sums (Medium) | 5, 6, 7 |
| R4 | `KpiMetricAggregationCatalog` re-creates the private-metric-literal drift hazard C4 fixed (Low) | 8 |
| R5 | Overlapping-singleton fold race faults an entire fold batch — failure grain is the pass, not the row (Low) | 9 |
| R6 | `PurgeTerminalAsync` on SiteCalls is the one remaining unbatched, unindexed year-scale maintenance DELETE (Medium) | 10, 11 |
| R7 | `SiteAuditRetentionService.StopAsync` surfaces the loop's cancellation to the host (Low) | 12 |
| S8 | Outbox repository dual SQLite/T-SQL dialect | No action — accepted round-1 deferral; round-2 re-verified the documented convention (`Component-ConfigurationDatabase.md:69`, "do not copy") |
| S10 | `SqliteAuditWriter.Dispose` sync-over-async | No action — accepted round-1 won't-fix; round-2 re-verified |
| P3 | AuditLog clustered key leads with random GUID | No action — accepted round-1 deferral, tracked benchmark follow-up (`Component-ConfigurationDatabase.md:70`) |
| P4 (cadence) | Per-node KPI sampling cadence | No action — accepted round-1 deferral, tracked (`Component-ConfigurationDatabase.md:71`); the purge half shipped in round 1 |
| P6 (paging) | Outbox offset→keyset page conversion | No action — accepted round-1 deferral, tracked (`Component-ConfigurationDatabase.md:71`); the KPI half shipped in round 1 |
| C2 | `AuditLogRow` lives in ConfigurationDatabase, not Commons | No action — accepted round-1 deviation, documented (`Component-Commons.md:51`, "do not lift it into Commons") |
| C3 | Mixed timestamp CLR types | No action — accepted round-1 convention note (`Component-ConfigurationDatabase.md:68`) |
| U3 | Hash-chain tamper evidence / Parquet archival | No action — unchanged v1.x deferral; round-2 confirms no drift |
| U5 (residue) | AuditLog `PullAuditEvents` keyset upgrade | No action — accepted round-1 deferral, tracked (`Component-ConfigurationDatabase.md:71`); idempotent on `EventId`, lower urgency; round-2 lists it only so the tracker entry doesn't age out |
archreview/plans/PLAN-R2-04-data-audit-backbone.md.tasks.json
+19
View File
@@ -0,0 +1,19 @@
{
"planPath": "archreview/plans/PLAN-R2-04-data-audit-backbone.md",
"tasks": [
{ "id": 1, "subject": "Task 1: Untracked, projected fold fetch (R2)", "status": "pending", "blockedBy": [] },
{ "id": 2, "subject": "Task 2: Rollup watermark seam — GetLatestRollupHourAsync (R1, part 1)", "status": "pending", "blockedBy": [1] },
{ "id": 3, "subject": "Task 3: Slice the rollup backfill into bounded day windows (R1, part 2)", "status": "pending", "blockedBy": [2] },
{ "id": 4, "subject": "Task 4: Backfill failover fast-path via the rollup watermark + doc correction (R1, part 3)", "status": "pending", "blockedBy": [2, 3] },
{ "id": 5, "subject": "Task 5: Per-metric reduction in the bucketer — sum-per-bucket for Rate series (R3, part 1)", "status": "pending", "blockedBy": [] },
{ "id": 6, "subject": "Task 6: Catalog-driven aggregation at the query-service boundary (R3, part 2)", "status": "pending", "blockedBy": [5] },
{ "id": 7, "subject": "Task 7: Truthful trend presentation — chart doc + Component-KpiHistory.md (R3, part 3)", "status": "pending", "blockedBy": [6] },
{ "id": 8, "subject": "Task 8: Close the catalog metric-literal drift hazard (R4)", "status": "pending", "blockedBy": [] },
{ "id": 9, "subject": "Task 9: Classify the failover fold-race failure grain (R5)", "status": "pending", "blockedBy": [1, 2] },
{ "id": 10, "subject": "Task 10: SiteCalls filtered terminal index (migration) (R6, part 1)", "status": "pending", "blockedBy": [] },
{ "id": 11, "subject": "Task 11: Time-sliced SiteCalls terminal purge (R6, part 2)", "status": "pending", "blockedBy": [10] },
{ "id": 12, "subject": "Task 12: Retention-service shutdown no longer surfaces cancellation (R7)", "status": "pending", "blockedBy": [] },
{ "id": 13, "subject": "Final verification: solution build + targeted suites this plan touched (infra up for MSSQL SkippableFacts)", "status": "pending", "blockedBy": [4, 7, 8, 9, 11, 12] }
],
"lastUpdated": "2026-07-13T03:25:01Z"
}
archreview/plans/PLAN-R2-05-templates-deployment-transport.md
+429
View File
@@ -0,0 +1,429 @@
# PLAN-R2-05 — Templates, Deployment & Transport Round-2 Fix Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
**Goal:** Close the six NEW findings (N1–N6) in the round-2 report `archreview/05-templates-deployment-transport.md` (2026-07-12, re-review at HEAD `8c888f13`): the Expression-trigger Roslyn-compile leftover on read-only staleness paths (N1 — the sole Medium, the surviving member class of round-1 #12), the first-error-only trigger syntax message (N2 — the trigger twin of the T24 fix), the change-bus publisher skipping `Add` resolutions (N3), silently-inert locked-member instance overrides on import (N4), the instance-alarm-override gap in the import trust gate (N5), and the unvalidated `MaxConcurrentImportSessions` knob (N6). All round-1 accepted deferrals (incl. the #15 `ReadManifestAsync` residue) stay deferred — coverage rows only, no tasks.
**Architecture:** Two independent lanes. **TemplateEngine lane (N1+N2):** `ValidationService.ValidateExpressionTriggers` today runs `ScriptTrustValidator.FindViolations` + `RoslynScriptCompiler.Compile` per Expression-triggered script/alarm unconditionally and uncached (`ValidationService.cs:131, 531-544`) — the fix mirrors round-1 #12 exactly: memoise the verdict in `ScriptCompileVerdictCache` (whose key must first gain a **globals-surface discriminator** — it is keyed on code alone at `ScriptCompileVerdictCache.cs:53`, sound only while `ScriptCompiler`/`ScriptCompileSurface` is its sole writer; a `TriggerCompileSurface` verdict is not interchangeable), then gate the syntax/compile stage behind the same `validateScriptCompilation` flag that already carries `false` from `GetDeploymentComparisonAsync` and `StaleInstanceProbe` through `FlatteningPipeline` — the blank-expression check and attribute-reference scan stay unconditional, and the deploy gate keeps the default (`true`). **Transport lane (N3+N4+N5):** three surgical `BundleImporter` changes — publish `ScriptArtifactsChanged` for non-Skip resolutions including `Add` (`:1798-1803`); extend `EnumerateTrustGatedScripts` (`:957-1001`, the single enumerator both the apply-time Pass-0 gate at `:4309` and the preview gate at `:849` share, so one change covers both) to `InstanceAlarmOverrideDto.TriggerConfigurationOverride` expression bodies; and emit best-effort warnings (`ConflictKind.Warning` at preview + `ImportResult.Warnings` at apply) when an instance override targets a locked template member, WITHOUT changing what gets written or how the flattener resolves it. N6 is a one-line `RequireThat` in `TransportOptionsValidator`.
**Tech Stack:** C#/.NET, EF Core (in-memory + MSSQL integration fixtures), Roslyn (`Microsoft.CodeAnalysis.CSharp.Scripting`), xUnit. Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per project: `dotnet test tests/<project>` — targeted filters only per task; no full-suite runs until the plan's terminal verification.
## Parallelization
**Concurrent lanes (dispatch as separate implementers):**
- **TemplateEngine lane (serialize — shared `ValidationService.cs` + `ScriptCompilerTests.cs`):** T1 → T2 → T3 → T4.
- **Transport `BundleImporter.cs` mutex (serialize):** T5 → T6 → T7.
- **Free / file-disjoint:** T8 (`TransportOptionsValidator.cs`) runs concurrently with everything.
- **Last:** T9 (docs sweep, blockedBy all).
**Cross-plan cautions:** T5 is the **publisher half** of N3 only — the subscriber side (Inbound API wiring + the wrong Host comment) is owned by **PLAN-R2-06**. The publisher fix is independent and self-contained (the plan-06 consumer already self-heals by content comparison, `InboundScriptExecutor.cs:370-404`), so there is **no blockedBy across plans** — either plan can land first; the coverage table records the split so both halves land coherently.
---
### Task 1: Full violation/error lists in the trigger-expression syntax check (N2)
**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** 5, 6, 7, 8
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ValidationService.cs` (`CheckExpressionSyntax` :536, :541)
- Test: `tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Validation/ScriptCompilerTests.cs` (extend — the existing `CheckExpressionSyntax_*` facts live here, :134-148)
1. Write failing tests:
```csharp
[Fact]
public void CheckExpressionSyntax_MultipleForbiddenApis_ReportsAll()
{
var error = ValidationService.CheckExpressionSyntax(
"System.IO.File.Exists(\"x\") && System.Diagnostics.Process.GetProcesses().Length > 0");
Assert.NotNull(error);
Assert.Contains("System.IO", error); // FAILS today: only violations[0] is surfaced
Assert.Contains("Process", error);
}
[Fact]
public void CheckExpressionSyntax_MultipleCompileErrors_ReportsAll()
{
var error = ValidationService.CheckExpressionSyntax("NoSuchThingA > 1 && NoSuchThingB < 2");
Assert.NotNull(error);
Assert.Contains("NoSuchThingA", error); // FAILS today: only errors[0] is surfaced
Assert.Contains("NoSuchThingB", error);
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests --filter FullyQualifiedName~ScriptCompilerTests` → expect **FAIL** (second violation/error missing from the message).
3. Implement — the exact `string.Join` shape Task 24 gave `ScriptCompiler.TryCompile` (`ScriptCompiler.cs:50, 55`), applied to the trigger twin at `ValidationService.cs:534-544`:
```csharp
var violations = ScriptTrustValidator.FindViolations(expression);
if (violations.Count > 0)
return $"uses forbidden API: {string.Join("; ", violations)}";
var errors = RoslynScriptCompiler.Compile(expression, typeof(TriggerCompileSurface));
if (errors.Count > 0)
return $"is not a valid expression: {string.Join("; ", errors)}";
```
An operator fixing a multi-error expression sees every finding in one deploy round-trip, matching the "Report ALL violations, not just the first" comment `ScriptCompiler.cs:46-47` already carries.
4. Run → expect **PASS**. Re-run the whole filter to confirm the existing `CheckExpressionSyntax_*` facts still pass.
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ValidationService.cs tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Validation/ScriptCompilerTests.cs && git commit -m "fix(template-engine): trigger-expression syntax check reports ALL violations/compile errors, not just the first (plan R2-05 T1)"`
### Task 2: Add a globals-surface discriminator to the verdict-cache key (N1 part 1)
**Classification:** high-risk (keying of a cache that stores script-trust verdicts — a cross-surface verdict reuse would return a stale "clean" for code never vetted against that surface)
**Estimated implement time:** ~4 min
**Parallelizable with:** 5, 6, 7, 8 (blockedBy 1 — shared `ScriptCompilerTests.cs`)
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ScriptCompileVerdictCache.cs` (`GetOrAdd` :51-69 gains a leading `string surface` parameter; key at :53 becomes `surface + ":" + hash`)
- Modify: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ScriptCompiler.cs` (:42 — pass `nameof(ScriptCompileSurface)`)
- Test: `tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Validation/ScriptCompilerTests.cs` (extend)
1. Write failing test (compile-red first — the signature changes — then behavioural red):
```csharp
[Fact]
public void GetOrAdd_SameCodeDifferentSurface_ComputesSeparateVerdicts()
{
ScriptCompileVerdictCache.Clear();
var a = ScriptCompileVerdictCache.GetOrAdd("SurfaceA", "return 1;", () => (true, null));
var hitsAfterA = ScriptCompileVerdictCache.Hits;
var b = ScriptCompileVerdictCache.GetOrAdd("SurfaceB", "return 1;", () => (false, "err"));
Assert.True(a.Ok);
Assert.False(b.Ok); // code-only key would return SurfaceA's verdict
Assert.Equal(hitsAfterA, ScriptCompileVerdictCache.Hits); // no false cross-surface hit
Assert.Equal(2, ScriptCompileVerdictCache.Count);
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests --filter FullyQualifiedName~ScriptCompilerTests` → expect **FAIL** (compile error on the new parameter — that counts; fix the signature, then the behavioural assertions go red under a code-only key).
3. Implement:
- `GetOrAdd(string surface, string code, Func<(bool Ok, string? Error)> factory)`; key = `surface + ":" + Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(code)))`. Update the class doc-comment: *the verdict is a pure function of code + policy + globals surface, so the surface is part of the key — a trigger expression valid against `TriggerCompileSurface` is NOT interchangeable with a `ScriptCompileSurface` script-body verdict* (the exact hazard the round-2 report calls out for N1).
- `ScriptCompiler.TryCompile` passes `nameof(ScriptCompileSurface)` at :42. No behaviour change for script bodies — same single writer, now under an explicit key segment.
4. Run the filter → expect **PASS** (including the pre-existing `TryCompile_SameCodeTwice_SecondCallIsCacheHit` cache facts at :157-171 — they exercise one surface and must be unaffected).
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ScriptCompileVerdictCache.cs src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ScriptCompiler.cs tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Validation/ScriptCompilerTests.cs && git commit -m "fix(template-engine): verdict-cache key gains the globals-surface discriminator — cross-surface verdict reuse structurally impossible (plan R2-05 T2)"`
### Task 3: Cache Expression-trigger verdicts under the trigger surface key (N1 part 2)
**Classification:** high-risk (trust-gate path — the trigger syntax check is a forbidden-API verdict; a caching bug here weakens the gate)
**Estimated implement time:** ~4 min
**Parallelizable with:** 5, 6, 7, 8 (blockedBy 2 — same files)
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ValidationService.cs` (`CheckExpressionSyntax` :531-544 routes through `ScriptCompileVerdictCache`)
- Test: `tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Validation/ScriptCompilerTests.cs` (extend)
1. Write failing test plus the cross-surface lock-in negative:
```csharp
[Fact]
public void CheckExpressionSyntax_SameExpressionTwice_SecondCallIsCacheHit()
{
ScriptCompileVerdictCache.Clear();
ValidationService.CheckExpressionSyntax("Attributes[\"Temp\"] != null");
var hitsBefore = ScriptCompileVerdictCache.Hits;
var error = ValidationService.CheckExpressionSyntax("Attributes[\"Temp\"] != null");
Assert.Null(error);
Assert.Equal(hitsBefore + 1, ScriptCompileVerdictCache.Hits); // FAILS today: no cache use
}
[Fact]
public void CheckExpressionSyntax_ScriptSurfaceVerdict_NotReusedForTriggerSurface()
{
// "Notify != null" resolves on ScriptCompileSurface (Notify is a script global,
// ScriptCompileSurface.cs:53) but NOT on TriggerCompileSurface — a shared
// code-only cache entry would wrongly report the trigger expression clean.
ScriptCompileVerdictCache.Clear();
Assert.True(new ScriptCompiler().TryCompile("Notify != null", "S").IsSuccess); // warms the SCRIPT-surface entry
var error = ValidationService.CheckExpressionSyntax("Notify != null");
Assert.NotNull(error); // green today (no cache), and MUST STAY green after T3 — the T2 key makes it structural
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests --filter FullyQualifiedName~ScriptCompilerTests` → expect **FAIL** (the cache-hit fact; the cross-surface fact is the lock-in guard that must never flip).
3. Implement — wrap the T1-joined factory in the surface-keyed cache:
```csharp
internal static string? CheckExpressionSyntax(string expression)
{
// Memoise the verdict under the TRIGGER surface key (see ScriptCompileVerdictCache):
// the verdict is a pure function of expression + policy + TriggerCompileSurface, and
// it is the exact hot-path cost N1 flagged — every staleness sweep / import probe of
// an Expression-triggered config re-ran a full trust compilation + script compile.
var (_, error) = ScriptCompileVerdictCache.GetOrAdd(nameof(TriggerCompileSurface), expression, () =>
{
var violations = ScriptTrustValidator.FindViolations(expression);
if (violations.Count > 0)
return (false, $"uses forbidden API: {string.Join("; ", violations)}");
var errors = RoslynScriptCompiler.Compile(expression, typeof(TriggerCompileSurface));
if (errors.Count > 0)
return (false, $"is not a valid expression: {string.Join("; ", errors)}");
return (true, (string?)null);
});
return error;
}
```
The stored error is already entity-name-free — `CheckExpressionTrigger` formats the entity name in at :449-451, matching the cache's name-free contract.
4. Run the filter → expect **PASS** (both new facts + all pre-existing `CheckExpressionSyntax_*`/`TryCompile_*` facts).
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ValidationService.cs tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Validation/ScriptCompilerTests.cs && git commit -m "perf(template-engine): cache Expression-trigger compile verdicts under the trigger-surface key — unchanged expressions compile once per process (plan R2-05 T3)"`
### Task 4: Skip the trigger syntax check on read-only staleness/comparison paths (N1 part 3)
**Classification:** standard (read paths only; the deploy gate keeps the authoritative compile by default — the same shape round-1 #12/T16 shipped for script bodies)
**Estimated implement time:** ~4 min
**Parallelizable with:** 5, 6, 7, 8 (blockedBy 3 — same files)
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ValidationService.cs` (`Validate` :131 threads the flag; `ValidateExpressionTriggers` :370-400 + `CheckExpressionTrigger` :420-463 gain the gate; `validateScriptCompilation` doc-comment :98-105 updated)
- Test: `tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Validation/ValidationServiceTests.cs` (extend)
1. Write failing tests (build the config the way the file's existing facts do — a `FlattenedConfiguration` with one script whose `TriggerType = "Expression"` and `TriggerConfiguration = """{"expression":"this is not C# (("}"""`):
```csharp
[Fact]
public void Validate_SkipCompilation_SkipsExpressionTriggerSyntaxCheck()
{
var config = ConfigWithExpressionTriggerScript("this is not C# ((");
var gated = new ValidationService().Validate(config, validateScriptCompilation: false);
Assert.DoesNotContain(gated.Errors, e => e.Message.Contains("failed validation")); // FAILS today
var full = new ValidationService().Validate(config);
Assert.Contains(full.Errors, e => e.Message.Contains("failed validation")); // deploy gate unchanged
}
[Fact]
public void Validate_SkipCompilation_StillChecksExpressionAttributeReferences()
{
// expression "Attributes[\"Ghost\"] != null" referencing a missing attribute:
// the reference scan is cheap string work and MUST survive the gate.
var config = ConfigWithExpressionTriggerScript("Attributes[\"Ghost\"] != null");
var gated = new ValidationService().Validate(config, validateScriptCompilation: false);
Assert.Contains(gated.Errors, e => e.Message.Contains("Ghost"));
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests --filter FullyQualifiedName~ValidationServiceTests` → expect **FAIL** (the gated run still emits the syntax error).
3. Implement:
- `ValidateExpressionTriggers(FlattenedConfiguration configuration, bool validateExpressionSyntax = true)` — additive default parameter, so every existing direct caller/test stays source-compatible.
- `Validate` passes `validateScriptCompilation` through at :131 — the same flag `FlatteningPipeline` already threads `false` into from `GetDeploymentComparisonAsync` (`DeploymentService.cs:690`) and `StaleInstanceProbe` (`StaleInstanceProbe.cs:37`), so **no DeploymentManager/Transport change is needed**: both read paths stop paying the per-expression trust-compilation + compile the moment this lands.
- Thread the flag into `CheckExpressionTrigger`; when `false`, skip ONLY the `CheckExpressionSyntax` call (:446-452) — the blank-expression warning/error (:432-444) and the attribute-reference scan (:454-462) still run, per the report's "the attribute-reference scan can stay".
- Extend the `validateScriptCompilation` doc-comment (:98-105): the flag now gates both the script-body compile stage AND the Expression-trigger syntax/compile check; read-only callers skip both, the deploy gate runs both.
4. Run the filter, then the full TemplateEngine project (`dotnet test tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests`) → expect **PASS**, no regressions.
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ValidationService.cs tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Validation/ValidationServiceTests.cs && git commit -m "perf(template-engine): read-only staleness/comparison paths skip Expression-trigger compiles — deploy gate remains the authoritative check (plan R2-05 T4)"`
### Task 5: Publish `ScriptArtifactsChanged` for `Add` resolutions too (N3)
**Classification:** standard
**Estimated implement time:** ~3 min
**Parallelizable with:** 1, 2, 3, 4, 8
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs` (`PublishScriptArtifactChanges` filter :1802 + the doc-comment :1768-1776)
- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/Import/BundleImporterApplyTests.cs` (extend — the `RecordingScriptArtifactChangeBus` fixture and the `#05-T14` publish fact at :1336-1455 are already there)
1. Write failing test alongside `ApplyAsync_publishes_ScriptArtifactsChanged_per_kind_after_commit` (:1339):
```csharp
[Fact]
public async Task ApplyAsync_publishes_ScriptArtifactsChanged_for_Add_resolutions()
{
// ApiMethod imported as Add into a target where the name does not exist.
// Delete-then-reimport-as-Add is the N3 edge: a node can still hold a
// _knownBadMethods / compiled-handler entry under that very name, so Adds
// must notify too (over-approximation is safe per the bus contract).
var result = await ApplyBundleWithApiMethodAsync(action: ResolutionAction.Add);
var notification = Assert.Single(_artifactBus.Received,
n => n.ArtifactKind == ScriptArtifactKinds.ApiMethod);
Assert.Contains("DelmiaRecipeDownload", notification.Names); // FAILS today: Adds are filtered out
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests --filter FullyQualifiedName~BundleImporterApplyTests` → expect **FAIL** (no ApiMethod notification — `names.Count == 0` short-circuits).
3. Implement — one condition at :1802 plus the comment that justified the old behaviour:
```csharp
.Where(r => string.Equals(r.EntityType, entityType, StringComparison.Ordinal)
&& r.Action is not ResolutionAction.Skip)
```
Rewrite the doc-comment (:1768-1776): *publishes for every non-Skip resolution — Overwrite, Rename, AND Add. An Add is not "nothing cached yet": an artifact deleted on the target and re-imported as Add under the same name can still have a stale compiled-handler/`_knownBadMethods` entry on any node. Over-approximation stays safe (the bus is advisory, at-least-once; consumers self-heal), which is the contract's stated design.* Keep the Rename → `RenameTo` name projection at :1803 as-is.
4. Run the filter → expect **PASS** (including the existing per-kind Overwrite fact and the failed-apply-publishes-nothing fact — the post-commit placement at :1340 is untouched).
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/Import/BundleImporterApplyTests.cs && git commit -m "fix(transport): script-artifact change publisher covers Add resolutions — delete-then-reimport no longer starves non-self-healing subscribers (plan R2-05 T5)"`
### Task 6: Trust-gate instance alarm-override trigger expressions (N5)
**Classification:** high-risk (script trust gate — 5th `ScriptTrustValidator` call-site completeness; security-adjacent)
**Estimated implement time:** ~5 min
**Parallelizable with:** 1, 2, 3, 4, 8 (blockedBy 5 — `BundleImporter.cs` mutex)
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs` (`EnumerateTrustGatedScripts` :957-1001 gains the instance pass; `ExtractTriggerExpression` :1009-1033 refactored to expose a JSON-only `ExtractExpressionBody`)
- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/SemanticValidatorImportTests.cs` (extend — sits beside the template twin `Apply_AlarmExpressionTriggerWithForbiddenApi_HardBlocks` at :371)
**Severity decision (recorded here, per the review convention):** instance-override trigger findings are **hard errors**, matching every other trust-gate surface. The advisory-vs-hard severity split is a property of the *name-resolution heuristic* only (false-positive-prone, so template findings are warnings — `:819, :842-846`); the trust gate's semantic verdict is authoritative with no false-positive channel, and Task 20 fixed its severity as *"a HARD error for all kinds"* (`:4301-4308`) — template script bodies and template trigger expressions already hard-block today. An instance override is the same executable surface (it *replaces* the template's trigger configuration in the flattened config and compiles/executes at the site), so it takes the same severity. The deploy gate (`ValidateExpressionTriggers` over the flattened, post-override config) stays the authoritative backstop — this closes the import-*review* completeness gap so the operator learns in the wizard, not at deploy time.
1. Write failing tests (negative tests required for a trust-gate change):
```csharp
[Fact]
public async Task Apply_InstanceAlarmOverrideExpressionWithForbiddenApi_HardBlocks()
{
// Bundle: template with an Expression-triggered alarm + an instance whose
// InstanceAlarmOverrideDto.TriggerConfigurationOverride carries
// {"expression":"System.Diagnostics.Process.GetProcesses().Length > 0"}.
var ex = await Assert.ThrowsAsync<SemanticValidationException>(
() => ApplyBundleAsync(bundle)); // FAILS today: imports clean
Assert.Contains(ex.Errors, e => e.Contains("Process") && e.Contains("trigger override"));
// Rollback contract: nothing persisted (mirror the template-twin's assertions).
}
[Fact]
public async Task Preview_InstanceAlarmOverrideExpressionWithForbiddenApi_SurfacesBlocker()
{
var preview = await PreviewBundleAsync(bundle);
Assert.Contains(preview.Blockers, b =>
b.Kind == ConflictKind.Blocker && b.EntityType == "Instance"
&& b.BlockerReason!.Contains("trust violation")); // FAILS today
}
[Fact]
public async Task Apply_InstanceAlarmOverrideExpression_Clean_Imports()
{
// {"expression":"Attributes[\"Temp\"] != null"} → import succeeds, override row persisted verbatim.
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests --filter FullyQualifiedName~SemanticValidatorImportTests` → expect **FAIL** (both forbidden-API cases import/preview clean).
3. Implement:
- Split `ExtractTriggerExpression` (:1009-1033): the trigger-type guard stays in `ExtractTriggerExpression`, the JSON `{"expression":"…"}` parse moves to a new `private static string? ExtractExpressionBody(string? triggerConfigJson)` it delegates to.
- Append the instance pass to `EnumerateTrustGatedScripts` after the ApiMethod loop (:993-1000):
```csharp
foreach (var i in content.Instances)
{
if (IsSkipResolution(resolutionMap, "Instance", i.UniqueName)) continue;
foreach (var o in i.AlarmOverrides)
{
// The DTO carries no trigger type (the type lives on the template
// alarm), so gate ANY override config carrying an {"expression":...}
// string body: if the overridden alarm is not Expression-triggered
// the body never executes, but vetting it anyway is fail-safe — the
// structured (HiLo/threshold) configs have no "expression" key, so
// there is no false-positive channel.
var expr = ExtractExpressionBody(o.TriggerConfigurationOverride);
if (!string.IsNullOrEmpty(expr))
{
yield return ("Instance", i.UniqueName,
$"{o.AlarmCanonicalName} (alarm trigger override expression)", expr);
}
}
}
```
- No gate-side change needed: the apply-time Pass 0 (:4309) and the preview gate (:849) both iterate this enumerator, so one edit covers both, including the fail-closed validator-throw handling each already has. Update the enumerator's doc-comment (:946-956) to name instance alarm-override expressions in the covered-surface list.
4. Run the filter, then `dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests --filter FullyQualifiedName~BundleImporterPreviewTests` (preview parity) → expect **PASS**.
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/SemanticValidatorImportTests.cs && git commit -m "fix(security): import trust gate covers instance alarm-override trigger expressions — 5th call site is surface-complete (plan R2-05 T6)"`
### Task 7: Warn when import persists overrides on locked template members (N4)
**Classification:** standard (warning-only UX; no write-path or flattener behaviour change — the flattener already drops locked-member overrides, `FlatteningService.cs:319, :337, :815`)
**Estimated implement time:** ~5 min
**Parallelizable with:** 1, 2, 3, 4, 8 (blockedBy 6 — `BundleImporter.cs` mutex)
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs` (new best-effort lock scan called from `RunSemanticValidationAsync` (:4293, appending to `warnings`) and from `DetectBlockersAsync` (:670, emitting `ConflictKind.Warning` rows); generalize the apply-side warning log wording at :1204-1208)
- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/Import/SiteInstanceImportTests.cs` (extend)
1. Write failing tests:
```csharp
[Fact]
public async Task Import_OverrideOnLockedAttribute_EmitsWarning_ImportStillSucceeds()
{
// Template attribute "SetPoint" IsLocked=true; instance carries an
// InstanceAttributeOverrideDto for "SetPoint".
var result = await ApplyBundleAsync(bundle);
Assert.Contains(result.Warnings, w => w.Contains("SetPoint") && w.Contains("locked")); // FAILS today: silent
// Behaviour unchanged: the row IS still written (bundle fidelity) — the
// flattener is what ignores it; assert the persisted override row exists.
}
[Fact]
public async Task Preview_OverrideOnLockedNativeAlarmSource_ShowsWarningRow()
{
var preview = await PreviewBundleAsync(bundle);
Assert.Contains(preview.Blockers, b =>
b.Kind == ConflictKind.Warning && b.EntityType == "Instance"
&& b.BlockerReason!.Contains("locked")); // FAILS today
}
[Fact]
public async Task Import_OverrideOnUnlockedAttribute_NoLockWarning()
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests --filter FullyQualifiedName~SiteInstanceImportTests` → expect **FAIL**.
3. Implement one shared best-effort scan, `CollectLockedOverrideWarnings(BundleContentDto content, Dictionary<(string,string), ImportResolution>? resolutionMap, IReadOnlyList<Template> targetTemplates)`:
- For each non-Skip instance, resolve its template by `dto.TemplateName` — bundle DTO first (the version about to be written), else the pre-existing target template (`GetAllTemplatesAsync` already Includes the child collections).
- Locked-member name sets from that template's rows: attributes/alarms/native sources with `IsLocked == true` (bundle DTOs carry `IsLocked`; derived templates carry inherited placeholder rows, so own + inherited members match by name). Check `AttributeOverrides` (→ `FlatteningService.cs:319` drop), `AlarmOverrides` (→ `:337`), and `NativeAlarmSourceOverrides` (→ `:815`) — the alarm loop is the same pattern the report's two cited collections use, included for completeness.
- Warning text: `Instance '{unique}' {kind} override '{name}' targets a LOCKED template member — the flattener ignores it, so this part of the bundle's instance config will never take effect.`
- **Best-effort, documented:** overrides addressing composed path-qualified members (`y1.z.Val`) or derived-shadow locks (`LockedInDerived` on a base the placeholder row doesn't reflect) may not match a direct row — an unmatched name emits NO warning (never a false positive; the `ManagementActor` and flattener remain the enforcement points, `ManagementActor.cs:898, :978`).
- Wire it twice: `RunSemanticValidationAsync` appends to `warnings` (they ride `validationWarnings` → `ImportResult.Warnings` at :1359 — generalize the "advisory template-script reference warning(s)" log wording at :1206-1208 to "advisory import warning(s)"); `DetectBlockersAsync` maps each to an `ImportPreviewItem` with `EntityType: "Instance"`, `Kind: ConflictKind.Warning` — the Task-19 machinery, so the wizard renders it with the existing Warning badge, no UI change.
- Do **NOT** block the import and do **NOT** change what `PopulateInstanceChildren` writes (:3999-4029) — bundle fidelity keeps the row; the flattener stays the behavioural authority (binding scope ruling: no flattener change).
4. Run the filter + `--filter FullyQualifiedName~BundleImporterPreviewTests` → expect **PASS**.
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/Import/SiteInstanceImportTests.cs && git commit -m "fix(transport): warn on import of instance overrides targeting locked template members — inert rows no longer silent (plan R2-05 T7)"`
### Task 8: Validate `MaxConcurrentImportSessions` at startup (N6)
**Classification:** small
**Estimated implement time:** ~2 min
**Parallelizable with:** 1, 2, 3, 4, 5, 6, 7 (file-disjoint with everything)
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/TransportOptionsValidator.cs` (one `RequireThat` in `Validate` :25-66)
- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.Tests/TransportOptionsValidatorTests.cs` (extend — mirror `ZeroMaxBundleSizeMb_IsRejected` :34)
1. Write failing tests:
```csharp
[Fact]
public void ZeroMaxConcurrentImportSessions_IsRejected()
{
var result = Validate(new TransportOptions { MaxConcurrentImportSessions = 0 });
Assert.True(result.Failed);
Assert.Contains("MaxConcurrentImportSessions", result.FailureMessage); // FAILS today: validates clean
}
[Fact]
public void NegativeMaxConcurrentImportSessions_IsRejected()
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.Tests --filter FullyQualifiedName~TransportOptionsValidatorTests` → expect **FAIL**.
3. Implement — the knob its own fix wave added (`TransportOptions.cs:37`, enforced at `BundleSessionStore.cs:78-88` where a `0`/negative cap rejects **every** import forever with "Too many concurrent import sessions (0)"):
```csharp
builder.RequireThat(options.MaxConcurrentImportSessions > 0,
$"ScadaBridge:Transport:MaxConcurrentImportSessions must be positive " +
$"(was {options.MaxConcurrentImportSessions}); a zero/negative cap rejects every import session forever.");
```
4. Run the filter → expect **PASS** (`DefaultOptions_AreValid` must stay green — the default is 8).
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Transport/TransportOptionsValidator.cs tests/ZB.MOM.WW.ScadaBridge.Transport.Tests/TransportOptionsValidatorTests.cs && git commit -m "fix(transport): validate MaxConcurrentImportSessions at startup — a zero cap no longer bricks imports silently (plan R2-05 T8)"`
### Task 9: Design-doc sync sweep
**Classification:** small (docs only)
**Estimated implement time:** ~4 min
**Parallelizable with:** none (run last)
**Files:**
- Modify: `docs/requirements/Component-Transport.md`, `docs/requirements/Component-ScriptAnalysis.md`, `docs/requirements/Component-TemplateEngine.md`, `/Users/dohertj2/Desktop/ScadaBridge/CLAUDE.md` (component #24 blurb + Transport bullet)
No tests; concrete edits, then a stale-cross-reference sweep per repo convention:
1. **Component-Transport.md:** (a) trust-gate section — the gated-surface list gains *instance alarm-override trigger expressions* (T6), same hard severity, with the severity-decision sentence (semantic verdict authoritative; the advisory split remains name-heuristic-only); (b) import-warning section — locked-member override warnings (T7): advisory `ConflictKind.Warning` at preview + `ImportResult.Warnings` at apply, best-effort by direct member name, rows still written, flattener remains the behavioural authority; (c) change-bus paragraph — publisher now notifies every non-Skip resolution including Add (T5), and note the subscriber side lives with the Inbound API (**PLAN-R2-06** owns the subscriber wiring + the stale Host comment); (d) options table — `MaxConcurrentImportSessions` is now startup-validated (T8). Leave the `ReadManifestAsync` deferral note (:92) untouched — still deferred.
2. **Component-ScriptAnalysis.md:** fifth-call-site (Transport) surface list gains instance alarm-override trigger expressions; note the verdict cache is keyed by (globals surface, code hash) so script-body and trigger-expression verdicts never cross (T2/T3).
3. **Component-TemplateEngine.md:** validation-pipeline section — `validateScriptCompilation: false` now also skips the Expression-trigger syntax/compile check on read-only staleness/comparison paths (blank + attribute-reference checks still run; deploy gate authoritative, T4); trigger syntax findings report all violations/errors (T1).
4. **CLAUDE.md:** component #24 blurb / Transport Key-Design-Decisions bullet — extend the script-trust-gate sentence: Expression-trigger gating covers template scripts, alarms, **and instance alarm-override expressions**. No `../scadaproj/CLAUDE.md` change (no wire-relationship/stack change) — verify and note in the commit message.
5. `git diff` review, then commit: `git add docs/requirements/Component-Transport.md docs/requirements/Component-ScriptAnalysis.md docs/requirements/Component-TemplateEngine.md CLAUDE.md && git commit -m "docs: sync Transport/ScriptAnalysis/TemplateEngine specs + CLAUDE.md with the round-2 fix wave (plan R2-05 T9)"`
---
## Dependencies on other plans
- **PLAN-R2-06 (Edge Integrations round 2):** owns the **subscriber side** of the change bus (Inbound API wiring + the wrong Host comment). T5's publisher fix is independent — the shipped plan-06 consumer self-heals by content comparison (`InboundScriptExecutor.cs:370-404`), so no cross-plan `blockedBy` exists in either direction; the two halves land coherently in any order. R2-06 should read T5's updated publisher doc-comment (Adds now publish) when wiring any non-self-healing subscriber.
- **No other cross-plan edges.** N1's gating rides the `validateScriptCompilation` flag PLAN-05 T16 already threads from DeploymentManager — no DeploymentManager file is touched.
## Execution order
**P0 (start immediately, in parallel):** Task 1 (TemplateEngine lane opener), Task 5 (BundleImporter lane opener), Task 8 (free).
**Lane A (serialize — shared files):** 1 → 2 → 3 → 4.
**Lane B (serialize — `BundleImporter.cs` mutex):** 5 → 6 → 7.
**Last:** 9 (docs sweep), then terminal verification: `dotnet build ZB.MOM.WW.ScadaBridge.slnx` + `dotnet test tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests && dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.Tests && dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests && dotnet test tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests` (DeploymentManager consumes the `Validate` flag) before declaring the plan done.
## Findings Coverage
| Report finding | Severity | Task(s) |
|---|---|---|
| N1 — Expression-trigger validation Roslyn-compiles on read-only staleness paths, uncached; any shared cache must key by globals surface | Medium (perf) | 2 (surface-discriminated cache key), 3 (cache trigger verdicts), 4 (skip on comparison/probe paths via the existing `validateScriptCompilation` flag) |
| N2 — `CheckExpressionSyntax` surfaces only the first violation / first compile error | Low | 1 (the T24 `string.Join` fix mirrored onto the trigger twin) |
| N3 — `PublishScriptArtifactChanges` skips `Add` resolutions | Low | 5 (publisher publishes every non-Skip resolution); **subscriber side (Inbound API wiring + wrong Host comment) → PLAN-R2-06** — publisher fix is independent, no cross-plan blockedBy |
| N4 — Import persists silently-inert instance overrides on locked members | Low | 7 (`ConflictKind.Warning` preview rows + `ImportResult.Warnings` at apply; best-effort direct-member match; rows still written, flattener behaviour unchanged per ruling) |
| N5 — Import trust gate misses `InstanceAlarmOverrideDto.TriggerConfigurationOverride` expression bodies | Low (sec) | 6 (enumerator covers instance overrides; **hard error** — trust-gate severity is uniform, the advisory split is name-heuristic-only; deploy gate stays authoritative backstop; negative tests included) |
| N6 — `TransportOptionsValidator` misses `MaxConcurrentImportSessions` | Low (conv) | 8 (one `RequireThat` > 0) |
| Docs drift from the above | — | 9 |
| Accepted deferral — #11 `OperationLockManager` single-node in-memory | Medium (R1) | **Deferred** (unchanged) — invariant recorded `Component-DeploymentManager.md:94`; structural active-node gating owned by plans 01/07 |
| Accepted deferral — #15 residue: `LoadAsync` manifest-peek (`ReadManifestAsync`) | Low (perf, R1) | **Deferred** (unchanged) — doc-acknowledged `Component-Transport.md:92`; pure optimisation (session cap already shipped, and N6/T8 now validates it) |
| Accepted deferral — #16 import apply is one long EF transaction | Low (perf, R1) | **Deferred** (unchanged) — dominant cost removed by #12/N1; restructuring the rollback contract stays high-risk/low-gain |
| Accepted deferral — U4: rename call-site rewriting, Transport-012 filter UI | — | **Deferred** (unchanged) — recorded limitations, `Component-Transport.md:354` |
| Accepted deferral — U5: preview→apply optimistic concurrency window | — | **Deferred** (unchanged) — recorded decision, `Component-Transport.md:138`; version fields reserved (`ArtifactDiff.cs:33-37`) |
| Accepted deferral — U6: three hand-maintained compile-surface mirrors | — | **Deferred** (unchanged) — guard tests exist; source-generator is an improvement, not a defect |
archreview/plans/PLAN-R2-05-templates-deployment-transport.md.tasks.json
+15
View File
@@ -0,0 +1,15 @@
{
"planPath": "archreview/plans/PLAN-R2-05-templates-deployment-transport.md",
"tasks": [
{ "id": 1, "subject": "Task 1: Full violation/error lists in the trigger-expression syntax check (N2)", "status": "pending", "blockedBy": [] },
{ "id": 2, "subject": "Task 2: Add a globals-surface discriminator to the verdict-cache key (N1 part 1)", "status": "pending", "blockedBy": [1] },
{ "id": 3, "subject": "Task 3: Cache Expression-trigger verdicts under the trigger surface key (N1 part 2)", "status": "pending", "blockedBy": [2] },
{ "id": 4, "subject": "Task 4: Skip the trigger syntax check on read-only staleness/comparison paths (N1 part 3)", "status": "pending", "blockedBy": [3] },
{ "id": 5, "subject": "Task 5: Publish ScriptArtifactsChanged for Add resolutions too (N3)", "status": "pending", "blockedBy": [] },
{ "id": 6, "subject": "Task 6: Trust-gate instance alarm-override trigger expressions (N5)", "status": "pending", "blockedBy": [5] },
{ "id": 7, "subject": "Task 7: Warn when import persists overrides on locked template members (N4)", "status": "pending", "blockedBy": [6] },
{ "id": 8, "subject": "Task 8: Validate MaxConcurrentImportSessions at startup (N6)", "status": "pending", "blockedBy": [] },
{ "id": 9, "subject": "Task 9: Design-doc sync sweep", "status": "pending", "blockedBy": [1, 2, 3, 4, 5, 6, 7, 8] }
],
"lastUpdated": "2026-07-13T03:23:07Z"
}
archreview/plans/PLAN-R2-06-edge-integrations.md
+809
View File
@@ -0,0 +1,809 @@
# PLAN-R2-06 — Edge Integrations Round-2 Fixes
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
**Goal:** Close every NEW finding in `archreview/06-edge-integrations.md` (round 2, 2026-07-12): wire the Inbound API compiled-handler cache as the real `IScriptArtifactChangeBus` consumer the Host comment and master tracker already claim exists (N1, the round-1 U2 residue) and sweep the false claims out of the docs; park (not retry-to-exhaustion) buffered ESG calls that fail deterministically with `ArgumentException` (N2); replace `SITE_UNREACHABLE` message-substring sniffing with a typed signal at the Ask boundary (N3); honor the declared response charset in the bounded body read (N4); and close the chunked-body bypass of the 415 guard (N5).
**Architecture:** The N1 consumer is a small `IHostedService` (`ScriptArtifactChangeSubscriber`) owned by the InboundAPI component and registered in `AddInboundAPI`: on start it calls `IScriptArtifactChangeBus.Subscribe` and, for every `ApiMethod` notification, calls the existing `InboundScriptExecutor.InvalidateMethod` seam (`InboundScriptExecutor.cs:136-140`) per name — the per-request revision check stays the correctness fallback per the contract's normative semantics (`docs/plans/2026-07-08-script-artifact-invalidation-contract.md` §4), so the subscriber is a fast-path, never a correctness dependency. The bus dependency is optional (`IScriptArtifactChangeBus? = null`, mirroring `BundleImporter.cs:134`) so site roles and plain test hosts no-op. N3 deliberately chooses the **typed-exception** form over an additive error-code field: unreachability manifests at the communication layer as `Akka.Actor.AskTimeoutException` (the site never answered — see `CentralCommunicationActor.HandleSiteEnvelope`'s no-ClusterClient path, which deliberately lets the Ask expire), so `RouteHelper` types it into `SiteUnreachableException` at the four router awaits and the message sniff (`RouteHelper.cs:391-395`) is deleted — a `Success=false` response means the site *answered*, which is by definition not unreachability. **No Commons message contract changes anywhere in this plan** — the PLAN-08 contract-lock suite (`ClusterClientContractLockTests.cs`) is untouched; the `RouteTo*Response` records are not among the locked seven in any case, and this plan does not modify them. N2/N4/N5 are each one-file classification/decoding/guard fixes with the failing test being exactly the report's scenario.
**Tech Stack:** C#/.NET 9, xUnit + NSubstitute, ASP.NET Core TestServer (inbound endpoint tests), Roslyn scripting (real compiles in executor tests), Akka.NET (`AskTimeoutException`, transitively referenced via Communication). No EF migrations, no `ManagementActor.cs` edits, no Commons message-contract edits.
Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per-project: `dotnet test tests/<project>` — targeted filters only, per task.
## Parallelization (incomplete tasks) — 2026-07-13
Nothing complete yet. **Ready now** (`blockedBy` empty): **{1, 3, 5, 6}**.
**Concurrent lanes (dispatch as separate implementers, fully file-disjoint):**
- **Inbound consumer lane:** T1 (`ScriptArtifactChangeSubscriber.cs` new + `ServiceCollectionExtensions.cs` + the `Host/Program.cs:95-98` comment) → T2 (docs/tracker truth sweep — no code).
- **ESG client lane (single-writer on `ExternalSystemClient.cs` + `ExternalSystemClientTests.cs`):** T3 → T4, serial.
- **Inbound routing lane:** T5 (`RouteHelper.cs` + executor/route tests).
- **Inbound endpoint lane:** T6 (`EndpointExtensions.cs` + `EndpointContentTypeTests.cs`).
**Cross-plan file notes:** T1 touches `Host/Program.cs` (comment only, lines 95-98) — coordinate if another active plan holds the `Program.cs` mutex. Nothing here touches `ManagementActor.cs`, `BundleImporter.cs`, or the EF model snapshot.
---
### Task 1: `ScriptArtifactChangeSubscriber` — wire the Inbound API as the bus's `ApiMethod` consumer
**Classification:** standard (new consumer wiring on the hot handler cache; safe-by-design — the per-request revision check remains the correctness fallback, so a missed/duplicate notification is harmless)
**Estimated implement time:** ~5 min
**Parallelizable with:** 3, 4, 5, 6
**Files:**
- Create: `src/ZB.MOM.WW.ScadaBridge.InboundAPI/ScriptArtifactChangeSubscriber.cs`
- Modify: `src/ZB.MOM.WW.ScadaBridge.InboundAPI/ServiceCollectionExtensions.cs` (lines 12-31: register the hosted service)
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` (lines 95-98: rewrite the factually-wrong comment)
- Test (create): `tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/ScriptArtifactChangeSubscriberTests.cs`
The bus contract shipped in PLAN-05 T14 (`IScriptArtifactChangeBus`, `InProcessScriptArtifactChangeBus`, the `BundleImporter` post-commit publisher at `BundleImporter.cs:1811`), and PLAN-06 T3 shipped the `InvalidateMethod` seam — but a repo-wide search confirms **zero `Subscribe` call sites**: the bus publishes into the void while `Host/Program.cs:96-98` claims "the Inbound API compiled-handler cache subscribes as the consumer (wired in plan 06)". This task ships the one real consumer: an `IHostedService` in the InboundAPI project (component-owned, unit-testable — unlike a lambda in the Host composition root). `SharedScript`/`Template` notifications are ignored by this consumer (the executor caches only ApiMethod handlers); their no-consumer status is documented honestly in Task 2.
**Steps:**
1. Write the failing tests:
```csharp
// tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/ScriptArtifactChangeSubscriberTests.cs
using Microsoft.Extensions.Logging.Abstractions;
using NSubstitute;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.InboundApi;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Scripting;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Scripting;
namespace ZB.MOM.WW.ScadaBridge.InboundAPI.Tests;
/// <summary>
/// Arch-review R2 N1: the Inbound API must be a REAL subscriber of
/// IScriptArtifactChangeBus — a bundle-import publish must invalidate a cached
/// handler immediately, without waiting for the per-request revision-check
/// self-heal (which remains the correctness fallback per the invalidation
/// contract's normative semantics).
/// </summary>
public class ScriptArtifactChangeSubscriberTests
{
/// <summary>Minimal in-test bus: records subscriptions, delivers synchronously.</summary>
private sealed class RecordingBus : IScriptArtifactChangeBus
{
private readonly List<Action<ScriptArtifactsChanged>> _handlers = new();
public int SubscriberCount => _handlers.Count;
public void Publish(ScriptArtifactsChanged notification)
{
foreach (var handler in _handlers.ToArray()) handler(notification);
}
public IDisposable Subscribe(Action<ScriptArtifactsChanged> handler)
{
_handlers.Add(handler);
return new Token(() => _handlers.Remove(handler));
}
private sealed class Token(Action dispose) : IDisposable
{
public void Dispose() => dispose();
}
}
private readonly InboundScriptExecutor _executor = new(
NullLogger<InboundScriptExecutor>.Instance, Substitute.For<IServiceProvider>());
private readonly RecordingBus _bus = new();
private readonly RouteHelper _route = new(
Substitute.For<IInstanceLocator>(), Substitute.For<IInstanceRouter>());
private ScriptArtifactChangeSubscriber CreateSubscriber(IScriptArtifactChangeBus? bus) =>
new(_executor, NullLogger<ScriptArtifactChangeSubscriber>.Instance, bus);
private Task<InboundScriptResult> Run(ApiMethod m) => _executor.ExecuteAsync(
m, new Dictionary<string, object?>(), _route, TimeSpan.FromSeconds(10));
private static ScriptArtifactsChanged ApiMethodChanged(params string[] names) =>
new(ScriptArtifactKinds.ApiMethod, names, "BundleImport", DateTimeOffset.UtcNow);
[Fact]
public async Task ApiMethodPublish_InvalidatesCachedHandler_WithoutRevisionCheckSelfHeal()
{
var subscriber = CreateSubscriber(_bus);
await subscriber.StartAsync(CancellationToken.None);
// Pin a SCRIPT-LESS handler: the RegisterHandler seam is exempt from the
// per-request revision check, so the ONLY mechanism that can stop it
// serving is an explicit InvalidateMethod — exactly what the bus must trigger.
_executor.RegisterHandler("m", async _ => { await Task.CompletedTask; return 99; });
var row = new ApiMethod("m", "return 1;") { Id = 1, TimeoutSeconds = 10 };
var before = await Run(row);
Assert.Contains("99", before.ResultJson); // pinned handler serves; no self-heal fires
_bus.Publish(ApiMethodChanged("m"));
var after = await Run(row);
Assert.True(after.Success);
Assert.Contains("1", after.ResultJson); // recompiled from the fresh row — invalidation, not self-heal
}
[Fact]
public async Task ApiMethodPublish_PurgesKnownBadRecord()
{
var subscriber = CreateSubscriber(_bus);
await subscriber.StartAsync(CancellationToken.None);
var broken = new ApiMethod("bad", "%%% not C# %%%") { Id = 2, TimeoutSeconds = 10 };
Assert.False(_executor.CompileAndRegister(broken));
Assert.Equal(1, _executor.KnownBadMethodCount);
_bus.Publish(ApiMethodChanged("bad"));
Assert.Equal(0, _executor.KnownBadMethodCount);
}
[Fact]
public async Task NonApiMethodKinds_AreIgnored()
{
var subscriber = CreateSubscriber(_bus);
await subscriber.StartAsync(CancellationToken.None);
_executor.RegisterHandler("m", async _ => { await Task.CompletedTask; return 99; });
var row = new ApiMethod("m", "return 1;") { Id = 3, TimeoutSeconds = 10 };
_bus.Publish(new ScriptArtifactsChanged(
ScriptArtifactKinds.SharedScript, new[] { "m" }, "BundleImport", DateTimeOffset.UtcNow));
_bus.Publish(new ScriptArtifactsChanged(
ScriptArtifactKinds.Template, new[] { "m" }, "BundleImport", DateTimeOffset.UtcNow));
var result = await Run(row);
Assert.Contains("99", result.ResultJson); // pinned handler untouched
}
[Fact]
public async Task StopAsync_Unsubscribes()
{
var subscriber = CreateSubscriber(_bus);
await subscriber.StartAsync(CancellationToken.None);
Assert.Equal(1, _bus.SubscriberCount);
await subscriber.StopAsync(CancellationToken.None);
Assert.Equal(0, _bus.SubscriberCount);
}
[Fact]
public async Task NullBus_NoOps()
{
// Site roles / plain test hosts have no bus registered — the hosted
// service must start and stop cleanly (the revision check alone applies).
var subscriber = CreateSubscriber(bus: null);
await subscriber.StartAsync(CancellationToken.None);
await subscriber.StopAsync(CancellationToken.None);
}
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests --filter ScriptArtifactChangeSubscriberTests` — expect **FAIL** (compile error: no `ScriptArtifactChangeSubscriber` type).
3. Implement `src/ZB.MOM.WW.ScadaBridge.InboundAPI/ScriptArtifactChangeSubscriber.cs`:
```csharp
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Scripting;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Scripting;
namespace ZB.MOM.WW.ScadaBridge.InboundAPI;
/// <summary>
/// N1 (arch-review 06 round 2): the Inbound API consumer of the script-artifact
/// invalidation contract (docs/plans/2026-07-08-script-artifact-invalidation-contract.md,
/// handoff item (a)). Subscribes to <see cref="IScriptArtifactChangeBus"/> on host
/// start and, for every <c>ApiMethod</c> notification (e.g. a Transport bundle
/// import overwriting method scripts, published post-commit), drops the compiled
/// handler AND the known-bad record via
/// <see cref="InboundScriptExecutor.InvalidateMethod"/>, so the next request
/// recompiles from the fresh row without waiting for the per-request
/// revision-check self-heal.
/// <para>
/// Fast-path only: the revision check in <c>ExecuteAsync</c> remains the
/// correctness fallback — this bus is in-process/per-node and at-least-once
/// (contract §4), so a missed or duplicate notification must always be harmless.
/// The bus is optional (site roles and plain test hosts register none) — the
/// subscriber then no-ops. SharedScript/Template notifications are ignored: this
/// executor caches only ApiMethod handlers.
/// </para>
/// </summary>
public sealed class ScriptArtifactChangeSubscriber : IHostedService
{
private readonly InboundScriptExecutor _executor;
private readonly ILogger<ScriptArtifactChangeSubscriber> _logger;
private readonly IScriptArtifactChangeBus? _bus;
private IDisposable? _subscription;
/// <summary>Initializes the subscriber.</summary>
/// <param name="executor">The compiled-handler cache to invalidate.</param>
/// <param name="logger">Logger instance.</param>
/// <param name="bus">The change bus, or null when the host registers none (site roles, tests).</param>
public ScriptArtifactChangeSubscriber(
InboundScriptExecutor executor,
ILogger<ScriptArtifactChangeSubscriber> logger,
IScriptArtifactChangeBus? bus = null)
{
_executor = executor;
_logger = logger;
_bus = bus;
}
/// <inheritdoc />
public Task StartAsync(CancellationToken cancellationToken)
{
if (_bus is null)
{
_logger.LogDebug(
"No IScriptArtifactChangeBus registered; inbound handler-cache freshness relies on the per-request revision check alone.");
return Task.CompletedTask;
}
_subscription = _bus.Subscribe(OnChanged);
return Task.CompletedTask;
}
/// <inheritdoc />
public Task StopAsync(CancellationToken cancellationToken)
{
_subscription?.Dispose();
_subscription = null;
return Task.CompletedTask;
}
private void OnChanged(ScriptArtifactsChanged notification)
{
if (!string.Equals(
notification.ArtifactKind, ScriptArtifactKinds.ApiMethod, StringComparison.Ordinal))
{
return; // only ApiMethod handlers are cached by this executor
}
foreach (var name in notification.Names)
{
_executor.InvalidateMethod(name);
}
_logger.LogInformation(
"Invalidated {Count} inbound API method handler(s) after a {Source} script-artifact change.",
notification.Names.Count, notification.Source);
}
}
```
In `ServiceCollectionExtensions.AddInboundAPI` (after the `InboundApiEndpointFilter` registration):
```csharp
// N1: the ApiMethod consumer of the script-artifact invalidation bus.
// The bus parameter is optional — hosts without a registered
// IScriptArtifactChangeBus (site roles, plain test hosts) start it as a no-op
// and the executor's per-request revision check alone keeps handlers fresh.
services.AddHostedService<ScriptArtifactChangeSubscriber>();
```
In `Host/Program.cs` replace the comment at lines 95-98 with the truth:
```csharp
// Script-artifact invalidation bus: in-process, per-node pub/sub
// for ScriptArtifactsChanged. The Transport bundle importer publishes
// post-commit; the Inbound API's ScriptArtifactChangeSubscriber (a hosted
// service registered by AddInboundAPI below) consumes ApiMethod notifications
// and invalidates the compiled-handler cache (wired in plan R2-06 T1; the
// per-request revision check remains the correctness fallback).
// SharedScript/Template notifications currently have NO consumer — nothing at
// central caches compiled artifacts of those kinds. Central-only.
```
4. Run the filter → **PASS**, then the whole project: `dotnet test tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests` → all green (the existing TestServer-based hosts call `AddInboundAPI` + `StartAsync` with no bus registered — the optional-parameter default keeps them green; this is the same optional-dependency pattern `BundleImporter` already uses). Then `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests` → green (Program.cs change is comment-only).
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.InboundAPI/ScriptArtifactChangeSubscriber.cs src/ZB.MOM.WW.ScadaBridge.InboundAPI/ServiceCollectionExtensions.cs src/ZB.MOM.WW.ScadaBridge.Host/Program.cs tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/ScriptArtifactChangeSubscriberTests.cs && git commit -m "fix(inbound): wire the compiled-handler cache as the IScriptArtifactChangeBus ApiMethod consumer (plan R2-06 T1)"`
### Task 2: Truth sweep — tracker / contract-doc / CLAUDE.md claims about the bus consumer
**Classification:** small (documentation)
**Estimated implement time:** ~4 min
**Parallelizable with:** 3, 4, 5, 6 (after 1)
**Files:**
- Modify: `archreview/plans/00-MASTER-TRACKER.md` (line 36 Wave-4 summary: the "`InvalidateMethod` seam consuming PLAN-05's `IScriptArtifactChangeBus` contract (T3)" claim; line 98 cross-plan handoff bullet)
- Modify: `docs/plans/2026-07-08-script-artifact-invalidation-contract.md` (line 3 status; §"Handoff to plan 06" rows (a)/(b)/(c), line 58 "no subscribers" sentence)
- Modify: `CLAUDE.md` (Templates & Deployment section, the "Plan-06 handoff: … (consumers = plan 06)" sentence)
N1's second half: the report calls the Host comment (fixed in Task 1) *and* the tracker/doc claims factually wrong — "will misdirect the next person debugging cache coherence." After Task 1 the consumer genuinely exists, so this task makes every claim match reality, including the honest residue: **(b) Management producers are still unwired by design** (the delete path calls `InvalidateMethod` directly at `ManagementActor.cs:2953`; update/edit paths self-heal via the per-request revision check — no notification needed for correctness), and **(c) is satisfied by the revision check** (PLAN-06 T1). SharedScript/Template kinds still publish with no consumer — that is now *documented* as safe (nothing at central caches compiled artifacts of those kinds) instead of silently misdescribed.
**Steps:**
1. `00-MASTER-TRACKER.md` line 36: change "`InvalidateMethod` seam consuming PLAN-05's `IScriptArtifactChangeBus` contract (T3)" to "`InvalidateMethod` seam (T3; the bus *subscriber* was NOT wired here — arch-review 06 round-2 N1 — and shipped later via PLAN-R2-06 T1)". Line 98: append to the PLAN-05 T14 ↔ PLAN-06 T3 bullet: "**Closed 2026-07-13 (PLAN-R2-06 T1):** `ScriptArtifactChangeSubscriber` (InboundAPI hosted service) subscribes and invalidates on `ApiMethod` notifications."
2. `docs/plans/2026-07-08-script-artifact-invalidation-contract.md`: line 3 status → "Seam + first producer shipped in PLAN-05 (T14). **Consumer (a) shipped in PLAN-R2-06 T1** (`ScriptArtifactChangeSubscriber`). (b) Management producers remain unwired by design — delete calls `InvalidateMethod` directly; edits self-heal via the revision check. (c) is satisfied by the per-request revision check (PLAN-06 T1)." Update the handoff table rows to the same dispositions and rewrite line 58 ("Until (a)–(c) land, plan 05's publisher is a no-op in practice…") to past tense with the shipped/unwired split.
3. `CLAUDE.md`: change "(consumers = plan 06)" in the Transport bullet to "(ApiMethod consumer = InboundAPI `ScriptArtifactChangeSubscriber`, wired in plan R2-06; SharedScript/Template publish with no consumer — nothing at central caches those kinds)".
4. Verify no stale claim remains: `grep -rn "wired in plan 06\|consumers = plan 06" src docs CLAUDE.md archreview/plans/00-MASTER-TRACKER.md` → zero hits (Task 1 already rewrote the Program.cs comment).
5. Commit: `git add archreview/plans/00-MASTER-TRACKER.md docs/plans/2026-07-08-script-artifact-invalidation-contract.md CLAUDE.md && git commit -m "docs(inbound): correct bus-consumer claims in tracker + invalidation-contract doc + CLAUDE.md (plan R2-06 T2)"`
### Task 3: `DeliverBufferedAsync` parks deterministic `ArgumentException` failures (path template / verb)
**Classification:** small (classification-only change on the buffered path, mirroring the existing `JsonException` poison fix in the same method)
**Estimated implement time:** ~4 min
**Parallelizable with:** 1, 2, 5, 6
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemClient.cs` (`DeliverBufferedAsync`, lines 214-225: add the catch)
- Test: `tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs` (append)
N2: `BuildUrl`'s `{param}` substitution throws `ArgumentException` for a placeholder with no matching/non-null parameter (`ExternalSystemClient.cs:535-537`) and `ValidateHttpMethod` throws the same type (`:423-438`). Neither is caught in `DeliverBufferedAsync` (its try at `:214-224` catches only `PermanentExternalSystemException`), and the S&F retry sweep's catch-all treats **any** thrown exception as transient (`StoreAndForwardService.cs:822-830`) — so a method path edited to add `{id}` while calls are buffered burns the full default 50 retries (`StoreAndForwardOptions.cs:21`) with a misleading "transient" `LastError` before parking. Re-running the same substitution against the same stored payload throws deterministically — same poison shape as the `JsonException` fix at `:184-195`. No `StoreAndForwardService` change: the classification decision belongs to the delivery handler, exactly like the existing cases.
**Steps:**
1. Failing tests (append; reuse the existing `StubResolution`/`BufferedCall`/`MockHttpMessageHandler` helpers — `BufferedCall` builds a payload with `Parameters: null`, which is exactly the unresolvable-placeholder case):
```csharp
[Fact]
public async Task DeliverBuffered_PathTemplateNowUnresolvable_ReturnsFalseSoMessageParks()
{
// Arch-review R2 N2: a method whose path gained a {param} placeholder AFTER
// calls were buffered throws ArgumentException deterministically for every
// already-buffered message — it must park immediately (permanent), not burn
// MaxRetries "transient" attempts (the S&F sweep's catch-all retries any throw).
var system = new ExternalSystemDefinition("TestAPI", "https://api.example.com", "none") { Id = 1 };
var method = new ExternalSystemMethod("getRecipe", "GET", "/recipes/{id}") { Id = 1, ExternalSystemDefinitionId = 1 };
StubResolution(system, method);
_httpClientFactory.CreateClient(Arg.Any<string>())
.Returns(new HttpClient(new MockHttpMessageHandler(HttpStatusCode.OK, "{}")));
var client = new ExternalSystemClient(
_httpClientFactory, _repository, NullLogger<ExternalSystemClient>.Instance);
// BufferedCall carries Parameters:null — the {id} placeholder has no value.
var delivered = await client.DeliverBufferedAsync(BufferedCall("TestAPI", "getRecipe"));
Assert.False(delivered);
}
[Fact]
public async Task DeliverBuffered_UnsupportedVerbOnStoredMethod_ReturnsFalseSoMessageParks()
{
// ValidateHttpMethod throws the same deterministic ArgumentException shape.
var system = new ExternalSystemDefinition("TestAPI", "https://api.example.com", "none") { Id = 1 };
var method = new ExternalSystemMethod("oddVerb", "FOO", "/p") { Id = 1, ExternalSystemDefinitionId = 1 };
StubResolution(system, method);
var client = new ExternalSystemClient(
_httpClientFactory, _repository, NullLogger<ExternalSystemClient>.Instance);
var delivered = await client.DeliverBufferedAsync(BufferedCall("TestAPI", "oddVerb"));
Assert.False(delivered);
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests --filter "DeliverBuffered_PathTemplateNowUnresolvable_ReturnsFalseSoMessageParks|DeliverBuffered_UnsupportedVerbOnStoredMethod_ReturnsFalseSoMessageParks"` → **FAIL** (both tests die on the propagating `ArgumentException` instead of returning `false`).
3. Implement — in `DeliverBufferedAsync`, after the `catch (PermanentExternalSystemException …)` branch (line 223), add:
```csharp
catch (ArgumentException ex)
{
// N2: same poison-message shape as the JsonException catch above. BuildUrl's
// {param} path-template substitution (placeholder with no matching/non-null
// parameter) and ValidateHttpMethod both throw ArgumentException
// deterministically for the SAME stored payload — the method definition
// changed underneath the buffered call, and retrying can never succeed.
// Return false so the S&F engine parks the message instead of counting the
// throw as transient and burning MaxRetries.
_logger.LogError(
ex,
"Buffered call to '{System}'/'{Method}' fails deterministically against the current method definition; parking.",
payload.SystemName, payload.MethodName);
return false;
}
```
Also extend the method's `<summary>` doc-comment (lines 165-173) to name `ArgumentException` alongside malformed JSON as a park-immediately case.
4. Run the filter → **PASS**; then the full project `dotnet test tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests` → green; consumer-suite check (no S&F files touched, classification contract only): `dotnet test tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests` → green.
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemClient.cs tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs && git commit -m "fix(esg): park buffered calls that fail deterministically with ArgumentException (path template/verb) instead of retrying to exhaustion (plan R2-06 T3)"`
### Task 4: Honor the declared response charset in `ReadBodyBoundedAsync`
**Classification:** small (decode-only; restores round-1 `ReadAsStringAsync` parity)
**Estimated implement time:** ~4 min
**Parallelizable with:** 1, 2, 5, 6 (NOT 3 — same file; run after 3)
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemClient.cs` (`ReadBodyBoundedAsync`, line 482 + new private helper)
- Test: `tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs` (append)
N4: the bounded read ends with `Encoding.UTF8.GetString(...)` (`:482`) regardless of `Content-Type: …; charset=iso-8859-1` — a legacy external system replying in a non-UTF-8 single-byte encoding becomes mojibake in script-visible response text and error bodies. The pre-P1 `ReadAsStringAsync` honored the charset; restore parity: read `content.Headers.ContentType?.CharSet`, `Encoding.GetEncoding` it (strip optional quotes), fall back to UTF-8 on unknown/invalid values (mojibake beats a hard failure for a cosmetic header).
**Steps:**
1. Failing tests (append; add the small byte-body handler beside `MockHttpMessageHandler` at line ~1100):
```csharp
[Fact]
public async Task Call_Latin1Charset_DecodesUsingDeclaredCharset()
{
// Arch-review R2 N4: the bounded read must honor the declared charset —
// UTF-8-decoding an iso-8859-1 body turns every non-ASCII byte into U+FFFD.
var system = new ExternalSystemDefinition("TestAPI", "https://api.example.com", "none") { Id = 1 };
var method = new ExternalSystemMethod("getData", "GET", "/data") { Id = 1, ExternalSystemDefinitionId = 1 };
StubResolution(system, method);
var handler = new ByteBodyHandler(
HttpStatusCode.OK,
Encoding.Latin1.GetBytes("température 25°C"),
"text/plain; charset=iso-8859-1");
_httpClientFactory.CreateClient(Arg.Any<string>()).Returns(new HttpClient(handler));
var client = new ExternalSystemClient(
_httpClientFactory, _repository, NullLogger<ExternalSystemClient>.Instance);
var result = await client.CallAsync("TestAPI", "getData");
Assert.True(result.Success);
Assert.Contains("température 25°C", result.ResponseJson);
}
[Fact]
public async Task Call_UnknownCharset_FallsBackToUtf8()
{
var system = new ExternalSystemDefinition("TestAPI", "https://api.example.com", "none") { Id = 1 };
var method = new ExternalSystemMethod("getData", "GET", "/data") { Id = 1, ExternalSystemDefinitionId = 1 };
StubResolution(system, method);
var handler = new ByteBodyHandler(
HttpStatusCode.OK, Encoding.UTF8.GetBytes("ok"), "text/plain; charset=not-a-charset");
_httpClientFactory.CreateClient(Arg.Any<string>()).Returns(new HttpClient(handler));
var client = new ExternalSystemClient(
_httpClientFactory, _repository, NullLogger<ExternalSystemClient>.Instance);
var result = await client.CallAsync("TestAPI", "getData");
Assert.True(result.Success);
Assert.Contains("ok", result.ResponseJson);
}
/// <summary>Test helper: returns a fixed byte body with an explicit Content-Type.</summary>
private sealed class ByteBodyHandler(
HttpStatusCode statusCode, byte[] body, string contentType) : HttpMessageHandler
{
protected override Task<HttpResponseMessage> SendAsync(
HttpRequestMessage request, CancellationToken cancellationToken)
{
var response = new HttpResponseMessage(statusCode) { Content = new ByteArrayContent(body) };
response.Content.Headers.ContentType = MediaTypeHeaderValue.Parse(contentType);
return Task.FromResult(response);
}
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests --filter "Call_Latin1Charset_DecodesUsingDeclaredCharset|Call_UnknownCharset_FallsBackToUtf8"` → **FAIL** (`Call_Latin1Charset…`: UTF-8 decode yields U+FFFD for `é`/`°`; the fallback test passes already — it pins the tolerance so the fix cannot regress it into a throw).
3. Implement — replace line 482 with:
```csharp
// N4: honor the response's declared charset (parity with the pre-bounded-read
// ReadAsStringAsync). Unknown/invalid charset → UTF-8 fallback: mojibake beats
// failing the call over a cosmetic header.
return ResolveEncoding(content.Headers.ContentType?.CharSet)
.GetString(buffer.GetBuffer(), 0, (int)buffer.Length);
```
and add below `ReadBodyBoundedAsync`:
```csharp
/// <summary>
/// Resolves the declared response charset to an <see cref="Encoding"/>, tolerating
/// the quoted form some servers emit (<c>charset="iso-8859-1"</c>). Null, empty,
/// or unrecognized values fall back to UTF-8.
/// </summary>
private static Encoding ResolveEncoding(string? charset)
{
if (string.IsNullOrWhiteSpace(charset))
{
return Encoding.UTF8;
}
try
{
return Encoding.GetEncoding(charset.Trim().Trim('"'));
}
catch (ArgumentException)
{
return Encoding.UTF8;
}
}
```
Also update the `ReadBodyBoundedAsync` doc-comment (lines 440-450) to state the charset behavior.
4. Run the filter → **PASS**; full project run `dotnet test tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests` → green (the oversized-body and timeout tests at `:1047` are decode-independent).
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemClient.cs tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs && git commit -m "fix(esg): honor the declared response charset in ReadBodyBoundedAsync with UTF-8 fallback (plan R2-06 T4)"`
### Task 5: Typed `SITE_UNREACHABLE` — `AskTimeoutException` classification replaces message sniffing
**Classification:** standard (reclassifies the routed-failure taxonomy for every inbound routed call; the change also makes SITE_UNREACHABLE fire for the *real* unreachable path for the first time)
**Estimated implement time:** ~5 min
**Parallelizable with:** 1, 2, 3, 4, 6
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.InboundAPI/RouteHelper.cs` (lines 16-21: inner-exception ctor; 185, 233, 297, 348: the four router awaits; 364-377: `ResolveSiteAsync` comment; 379-395: `RoutingFailure` + delete `IsUnreachable`)
- Modify: `docs/requirements/Component-InboundAPI.md` (error-code table: `SITE_UNREACHABLE` = "the site did not respond to the routed request (no contact / down / partitioned)")
- Test: `tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/InboundScriptExecutorTests.cs` (rewrite the sniff-pinning test at lines 100-124, add the anti-sniff test), `tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/RouteHelperTests.cs` (append)
N3: `RouteHelper.IsUnreachable` (`:391-395`) matches `"unreachable"`/`"no contact"` substrings in `ErrorMessage` text produced by another component — a harmless rewording silently downgrades `SITE_UNREACHABLE` to `SCRIPT_ERROR` with no test failing at the boundary. The typed signal already exists in the communication layer: an unreachable site means the ClusterClient Ask **expires** (`CentralCommunicationActor.HandleSiteEnvelope` deliberately drops enveloped messages for sites with no client so "the Ask will timeout on the caller side"; `CommunicationService.RouteTo*Async` are plain bounded Asks), surfacing as `Akka.Actor.AskTimeoutException` — which today falls into the executor's generic catch as `SCRIPT_ERROR`. Conversely, a `Success=false` *response* means the site **answered** — by definition not unreachability. So: type `AskTimeoutException → SiteUnreachableException` at the four router awaits, and make `RoutingFailure` always a plain `InvalidOperationException`. **No Commons message-contract change** (the typed-exception form was chosen over an additive error-code field on the `RouteTo*Response` records, so the PLAN-08 contract-lock suite needs no update; those records are not among the locked seven anyway, and additive-only + default(T)-safe rules would apply if a future change adds such a field).
**Steps:**
1. Failing tests. In `InboundScriptExecutorTests.cs`, **rewrite** `RoutedSiteUnreachable_Returns500WithSiteUnreachableCode` (lines 100-124 — it currently pins the sniff by stubbing `ErrorMessage: "Site unreachable"`) and add the anti-sniff case:
```csharp
[Fact]
public async Task RoutedAskTimeout_Returns500WithSiteUnreachableCode()
{
// N3: unreachability is the Ask EXPIRING (site never answered) — typed as
// AskTimeoutException by the communication layer — not a substring in a
// failure message. Pre-fix this fell into the generic catch as SCRIPT_ERROR.
var locator = Substitute.For<IInstanceLocator>();
locator.GetSiteIdForInstanceAsync("X", Arg.Any<CancellationToken>()).Returns("SiteA");
var router = Substitute.For<IInstanceRouter>();
router.RouteToCallAsync("SiteA", Arg.Any<RouteToCallRequest>(), Arg.Any<CancellationToken>())
.Returns(Task.FromException<RouteToCallResponse>(
new Akka.Actor.AskTimeoutException("Timeout after 00:00:30")));
var route = new RouteHelper(locator, router);
var method = new ApiMethod("routed", "return await Route.To(\"X\").Call(\"s\");")
{ Id = 7, TimeoutSeconds = 10 };
var result = await _executor.ExecuteAsync(
method, new Dictionary<string, object?>(), route, TimeSpan.FromSeconds(10));
Assert.False(result.Success);
Assert.Equal("SITE_UNREACHABLE", result.ErrorCode);
}
[Fact]
public async Task SiteRespondedFailure_MessageMentioningUnreachable_IsScriptErrorNotSiteUnreachable()
{
// N3 anti-sniff: a site that RESPONDED with Success=false is reachable by
// definition — even when its error text happens to contain "unreachable".
// Pre-fix the substring sniff misclassified this as SITE_UNREACHABLE.
var locator = Substitute.For<IInstanceLocator>();
locator.GetSiteIdForInstanceAsync("X", Arg.Any<CancellationToken>()).Returns("SiteA");
var router = Substitute.For<IInstanceRouter>();
router.RouteToCallAsync("SiteA", Arg.Any<RouteToCallRequest>(), Arg.Any<CancellationToken>())
.Returns(ci => new RouteToCallResponse(
((RouteToCallRequest)ci[1]).CorrelationId,
Success: false, ReturnValue: null,
ErrorMessage: "PLC device unreachable behind the site gateway",
DateTimeOffset.UtcNow));
var route = new RouteHelper(locator, router);
var method = new ApiMethod("routed2", "return await Route.To(\"X\").Call(\"s\");")
{ Id = 8, TimeoutSeconds = 10 };
var result = await _executor.ExecuteAsync(
method, new Dictionary<string, object?>(), route, TimeSpan.FromSeconds(10));
Assert.False(result.Success);
Assert.Equal("SCRIPT_ERROR", result.ErrorCode);
}
```
In `RouteHelperTests.cs` (uses the existing `_locator`/`_router`/`CreateHelper`/`SiteResolves` scaffolding), append:
```csharp
[Fact]
public async Task Call_AskTimeout_ThrowsSiteUnreachableException()
{
SiteResolves("inst", "SiteA");
_router.RouteToCallAsync("SiteA", Arg.Any<RouteToCallRequest>(), Arg.Any<CancellationToken>())
.Returns(Task.FromException<RouteToCallResponse>(
new Akka.Actor.AskTimeoutException("Timeout after 00:00:30")));
await Assert.ThrowsAsync<SiteUnreachableException>(
() => CreateHelper().To("inst").Call("s"));
}
[Fact]
public async Task GetAttributes_AskTimeout_ThrowsSiteUnreachableException()
{
SiteResolves("inst", "SiteA");
_router.RouteToGetAttributesAsync("SiteA", Arg.Any<RouteToGetAttributesRequest>(), Arg.Any<CancellationToken>())
.Returns(Task.FromException<RouteToGetAttributesResponse>(
new Akka.Actor.AskTimeoutException("Timeout after 00:00:30")));
await Assert.ThrowsAsync<SiteUnreachableException>(
() => CreateHelper().To("inst").GetAttributes(new[] { "a" }));
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests --filter "RoutedAskTimeout_Returns500WithSiteUnreachableCode|SiteRespondedFailure_MessageMentioningUnreachable|AskTimeout_ThrowsSiteUnreachableException"` → **FAIL** (Ask-timeout cases surface `SCRIPT_ERROR`/`AskTimeoutException`; the anti-sniff case returns `SITE_UNREACHABLE`).
3. Implement in `RouteHelper.cs` (add `using Akka.Actor;`):
- `SiteUnreachableException` gains `public SiteUnreachableException(string message, Exception innerException) : base(message, innerException) { }` and its class doc-comment is rewritten: thrown when the routed Ask **expires** (typed `AskTimeoutException` from the communication layer) — never inferred from message text.
- Add to `RouteTarget`:
```csharp
/// <summary>
/// N3: awaits a routed Ask and TYPES unreachability. AskTimeoutException means
/// the site never answered (no ClusterClient contact, site down, partition) —
/// the communication layer's own typed signal (CentralCommunicationActor drops
/// envelopes for contactless sites so the Ask expires caller-side) — and
/// surfaces as SiteUnreachableException so the executor maps it to the spec's
/// SITE_UNREACHABLE code. A Success=false RESPONSE means the site answered and
/// is therefore reachable; it flows through RoutingFailure as a plain
/// InvalidOperationException (SCRIPT_ERROR).
/// </summary>
private static async Task<TResponse> RouteOrUnreachable<TResponse>(Task<TResponse> routedAsk)
{
try
{
return await routedAsk;
}
catch (AskTimeoutException ex)
{
throw new SiteUnreachableException(
"Site did not respond to the routed request within the routing timeout — unreachable or not connected.",
ex);
}
}
```
- Wrap the four router awaits, e.g. line 185: `var response = await RouteOrUnreachable(_instanceRouter.RouteToCallAsync(siteId, request, token));` (same at 233, 297, 348).
- Replace `RoutingFailure` + delete `IsUnreachable` (lines 379-395):
```csharp
/// <summary>
/// C3/N3: a routing-level failure REPORTED BY the site or the locator. The
/// remote end answered (or the instance simply has no site), so this is never
/// unreachability — always a plain <see cref="InvalidOperationException"/>
/// (SCRIPT_ERROR at the endpoint). Unreachability is typed at the Ask boundary
/// by <c>RouteOrUnreachable</c> (AskTimeoutException → SiteUnreachableException),
/// replacing the old message-substring sniff that a harmless rewording in the
/// communication layer could silently defeat.
/// </summary>
private static InvalidOperationException RoutingFailure(string message) => new(message);
```
- Fix the now-stale comment inside `ResolveSiteAsync` (lines 368-371) — a null site id is a plain not-found, never sniffed.
- Update the `Component-InboundAPI.md` error-code table row for `SITE_UNREACHABLE` to the typed semantics ("the target site did not respond to the routed request within the routing timeout").
4. Run the filter → **PASS**; full project `dotnet test tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests` → green (no other test stubs unreachable-flavored `ErrorMessage` text).
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.InboundAPI/RouteHelper.cs docs/requirements/Component-InboundAPI.md tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/InboundScriptExecutorTests.cs tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/RouteHelperTests.cs && git commit -m "fix(inbound): type SITE_UNREACHABLE on AskTimeoutException at the router await instead of message-substring sniffing (plan R2-06 T5)"`
### Task 6: 415 guard covers chunked (no Content-Length) bodies
**Classification:** small (guard-condition fix on the endpoint; also fixes lenient parsing of chunked bodies)
**Estimated implement time:** ~4 min
**Parallelizable with:** 1, 2, 3, 4, 5
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.InboundAPI/EndpointExtensions.cs` (lines 182-212: the 415 guard at 187-194 and the parse condition at 206-207)
- Modify: `docs/requirements/Component-InboundAPI.md` (the 415 row: "a body — declared by Content-Length or Transfer-Encoding (chunked)")
- Test: `tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/EndpointContentTypeTests.cs` (append)
N5: the 415 guard keys on `ContentLength > 0` (`:187`); a chunked non-JSON body has null `ContentLength`, skips the 415, then also skips JSON parsing (the parse condition `:206-207` is `ContentLength > 0 || ContentType contains json`), so `body = null` and the caller gets a misleading `VALIDATION_FAILED` — the same error class S9 set out to fix, now confined to the chunked edge. Fix: compute `hasBody` once (Content-Length positive, OR Content-Length absent with a `Transfer-Encoding` header present) and use it in **both** the 415 guard and the parse condition — so a chunked JSON/no-Content-Type body also parses leniently, matching the fixed-length behavior. The round-1 leniencies are preserved: no body → no 415; body with no Content-Type → lenient JSON parse.
**Steps:**
1. Failing tests (append to `EndpointContentTypeTests`, reusing `BuildHostAsync`/`SeedKeyAsync`; add the non-seekable content helper so the client can never auto-compute a Content-Length):
```csharp
[Fact]
public async Task ChunkedNonJsonBody_Returns415()
{
// Arch-review R2 N5: a chunked body has NULL ContentLength — it must not
// slip past the 415 guard into a misleading VALIDATION_FAILED.
const string methodName = "echoChunked";
var method = new ApiMethod(methodName, "return Parameters[\"value\"];")
{
Id = 1,
TimeoutSeconds = 10,
ParameterDefinitions = """[{"name":"value","type":"Integer","required":true}]""",
};
var repo = Substitute.For<IInboundApiRepository>();
repo.GetMethodByNameAsync(methodName, Arg.Any<CancellationToken>()).Returns(method);
using var host = await BuildHostAsync(repo);
var token = await SeedKeyAsync(host, methodName);
var client = host.GetTestClient();
var request = new HttpRequestMessage(HttpMethod.Post, "/api/" + methodName)
{
Content = new ChunkedByteContent(Encoding.UTF8.GetBytes("hello")),
};
request.Content.Headers.ContentType = MediaTypeHeaderValue.Parse("text/plain");
request.Headers.TransferEncodingChunked = true;
request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", token);
var response = await client.SendAsync(request);
var body = await response.Content.ReadAsStringAsync();
Assert.Equal(HttpStatusCode.UnsupportedMediaType, response.StatusCode);
Assert.Contains("UNSUPPORTED_MEDIA_TYPE", body);
}
[Fact]
public async Task ChunkedJsonBody_NoContentType_StillParsesLeniently()
{
// The parse condition must also learn about chunked bodies: a chunked JSON
// body with no Content-Type previously skipped parsing entirely (body=null →
// misleading "missing required parameter").
const string methodName = "echoChunkedJson";
var method = new ApiMethod(methodName, "return Parameters[\"value\"];")
{
Id = 1,
TimeoutSeconds = 10,
ParameterDefinitions = """[{"name":"value","type":"Integer","required":true}]""",
};
var repo = Substitute.For<IInboundApiRepository>();
repo.GetMethodByNameAsync(methodName, Arg.Any<CancellationToken>()).Returns(method);
using var host = await BuildHostAsync(repo);
var token = await SeedKeyAsync(host, methodName);
var client = host.GetTestClient();
var request = new HttpRequestMessage(HttpMethod.Post, "/api/" + methodName)
{
Content = new ChunkedByteContent(Encoding.UTF8.GetBytes("{\"value\":42}")),
};
request.Content.Headers.ContentType = null;
request.Headers.TransferEncodingChunked = true;
request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", token);
var response = await client.SendAsync(request);
var body = await response.Content.ReadAsStringAsync();
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
Assert.Contains("42", body);
}
/// <summary>
/// Content whose length is never computable, forcing HttpClient to transmit it
/// chunked (no Content-Length header ever reaches the server).
/// </summary>
private sealed class ChunkedByteContent(byte[] bytes) : HttpContent
{
protected override Task SerializeToStreamAsync(
Stream stream, System.Net.TransportContext? context) =>
stream.WriteAsync(bytes, 0, bytes.Length);
protected override bool TryComputeLength(out long length)
{
length = -1;
return false;
}
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests --filter "ChunkedNonJsonBody_Returns415|ChunkedJsonBody_NoContentType_StillParsesLeniently"` → **FAIL** (first: 400 `VALIDATION_FAILED` instead of 415; second: 400 "missing required parameter" instead of 200).
3. Implement in `EndpointExtensions.cs` — above the S9 block (line ~182), compute the body signal once, then use it in both places:
```csharp
// N5: a body is present when Content-Length says so, OR when the request is
// chunked — Transfer-Encoding present and Content-Length (necessarily) absent.
// Computed once and used by BOTH the 415 guard and the JSON parse condition so
// chunked and fixed-length bodies behave identically.
var requestHeaders = httpContext.Request.Headers;
var hasBody = httpContext.Request.ContentLength > 0
|| (httpContext.Request.ContentLength is null && requestHeaders.TransferEncoding.Count > 0);
```
The 415 guard (lines 187-189) becomes `if (hasBody && !string.IsNullOrEmpty(httpContext.Request.ContentType) && httpContext.Request.ContentType.Contains("json", StringComparison.OrdinalIgnoreCase) == false)`; the parse condition (lines 206-207) becomes `if (hasBody || httpContext.Request.ContentType?.Contains("json", StringComparison.OrdinalIgnoreCase) == true)`. Update the S9 comment block (lines 182-186) to name the chunked case. Update the doc's 415 row.
4. Run the filter → **PASS**; full project `dotnet test tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests` → green (`NonJsonContentType_WithBody_Returns415` and the lenient no-Content-Type case at `EndpointContentTypeTests.cs:96-146` still pass — fixed-length behavior is unchanged, and a zero-length body still never 415s).
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.InboundAPI/EndpointExtensions.cs docs/requirements/Component-InboundAPI.md tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/EndpointContentTypeTests.cs && git commit -m "fix(inbound): 415 guard and lenient JSON parse cover chunked (no Content-Length) bodies (plan R2-06 T6)"`
---
## Dependencies on other plans
- **PLAN-R2-05 owns the publisher-side Add-skip fix.** `BundleImporter.PublishScriptArtifactChanges` (`BundleImporter.cs:1777-1817`, publish at `:1811`) deliberately excludes `Add` resolutions ("nothing is cached under a brand-new name yet"); revisiting that rationale is PLAN-R2-05's finding. Task 1's subscriber has **no `blockedBy`** on it — the bus contract shipped in PLAN-05 T14, the subscriber consumes whatever is published, and `InvalidateMethod` is safe for unknown names, so when PLAN-R2-05 broadens the publisher the consumer needs zero changes. Files are disjoint (`BundleImporter.cs` vs `InboundAPI/*`) — safe to run both plans concurrently.
- **Contract-lock (PLAN-08 T9).** This plan touches no Commons message contracts, so `ClusterClientContractLockTests.cs` (locked records: `CreateDataConnectionCommand`, `DeployInstanceCommand`, `NotificationSubmit`, `AttributeValueChanged`, `GetAttributeRequest`, `SetStaticAttributeCommand`, `ManagementEnvelope`) needs no update. Task 5 deliberately chose the typed-exception form over an additive error-code field on the `RouteTo*Response` records; those records are not among the locked seven, but any future additive field there must still be safe at `default(T)` (the locked reality: Akka's Newtonsoft serializer ignores C# optional-parameter defaults on missing fields).
- **`Host/Program.cs` mutex.** Task 1's edit is comment-only (lines 95-98) — coordinate only if another active plan is editing `Program.cs` in the same window.
## Execution order
**P0 (start immediately, independent, file-disjoint):** Tasks 1, 3, 5, 6 in parallel lanes.
- **Inbound consumer lane:** 1 → 2 (docs truth sweep last, so it records what actually shipped).
- **ESG client lane:** 3 → 4 (both edit `ExternalSystemClient.cs` + the same test file — serial).
- **Inbound routing lane:** 5 (free-floating).
- **Inbound endpoint lane:** 6 (free-floating).
Intra-plan critical path: **3 → 4** and **1 → 2** (both two deep); everything else is single-task. No EF migrations, no `ManagementActor.cs`, no initiative-wide mutexes beyond the `Program.cs` comment note above.
## Findings Coverage
| # | Finding (report section) | Severity | Task(s) |
|---|--------------------------|----------|---------|
| N1 | Script-artifact invalidation bus publishes into the void; Host comment + master tracker claim a consumer that does not exist (the open PLAN-05→PLAN-06 handoff) | Medium | 1 (subscriber + Program.cs comment), 2 (tracker/contract-doc/CLAUDE.md truth sweep) |
| N2 | Now-unresolvable path template (or bad verb) on a buffered message classified transient and retried to exhaustion (50 retries) before parking with a misleading `LastError` | Low | 3 |
| N3 | `SITE_UNREACHABLE` classification is substring-sniffing on another component's error-message text; a rewording silently downgrades it to `SCRIPT_ERROR` | Low | 5 (typed `AskTimeoutException → SiteUnreachableException` at the router await; sniff deleted; no Commons contract change → contract-lock suite untouched) |
| N4 | Bounded response read decodes UTF-8 unconditionally, dropping the round-1 charset honoring (mojibake for non-UTF-8 legacy systems) | Low | 4 |
| N5 | 415 guard keyed on `ContentLength > 0` is bypassed by chunked non-JSON bodies, resurfacing the misleading-error class S9 fixed | Low | 6 |
| U2 (round-1 partial) | No cache-coherence mechanism for compiled handlers — the revision check shipped but the claimed bus consumer was never wired | Underdeveloped | Same as N1 → 1, 2 (the report folds the U2 residue into N1) |
| U4 (round-1 partial) | Twilio accept-only: status-callback webhook / per-recipient delivery state | Underdeveloped | **Deferred (unchanged)** — stays out of scope per the round-1 ruling (documented in `Component-NotificationService.md:95`): an inbound webhook endpoint is new attack surface warranting its own design pass. The residual minor (SID rows saved before validation are not retro-checked) is accepted. No task. |
archreview/plans/PLAN-R2-06-edge-integrations.md.tasks.json
+12
View File
@@ -0,0 +1,12 @@
{
"planPath": "archreview/plans/PLAN-R2-06-edge-integrations.md",
"tasks": [
{ "id": 1, "subject": "Task 1: ScriptArtifactChangeSubscriber — wire the Inbound API as the bus's ApiMethod consumer", "status": "pending", "blockedBy": [] },
{ "id": 2, "subject": "Task 2: Truth sweep — tracker / contract-doc / CLAUDE.md claims about the bus consumer", "status": "pending", "blockedBy": [1] },
{ "id": 3, "subject": "Task 3: DeliverBufferedAsync parks deterministic ArgumentException failures (path template / verb)", "status": "pending", "blockedBy": [] },
{ "id": 4, "subject": "Task 4: Honor the declared response charset in ReadBodyBoundedAsync", "status": "pending", "blockedBy": [3] },
{ "id": 5, "subject": "Task 5: Typed SITE_UNREACHABLE — AskTimeoutException classification replaces message sniffing", "status": "pending", "blockedBy": [] },
{ "id": 6, "subject": "Task 6: 415 guard covers chunked (no Content-Length) bodies", "status": "pending", "blockedBy": [] }
],
"lastUpdated": "2026-07-13T03:23:07Z"
}
archreview/plans/PLAN-R2-07-ui-management-security.md
+980
View File
@@ -0,0 +1,980 @@
# PLAN-R2-07 — UI, Management & Security Round-2 Hardening Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
**Goal** — Close every NEW (round-2) finding from architecture review 07 round 2 (`archreview/07-ui-management-security.md`, dated 2026-07-12): throttle the last unthrottled LDAP-bind surface (the debug-stream hub) and fix the doc that falsely claims coverage is complete (N1); make `LoginThrottle` keys survive the Traefik topology via trusted-proxy `ForwardedHeaders` and document the username-lockout-DoS trade-off (N2); enforce LDAP-mapping site scope on all four secured-write handlers, where the spec already promises it (N3); stop the Alarm Summary poll from displaying cross-site data after a site switch (N4) and from regressing fresher live state (N5); un-stick `IsLive` when a live-cache aggregator dies (N6); adopt the house disposal guard on the live callback (N7); amortize `LoginThrottle.Prune` (N8); and freeze the secret-scrubber's fragment coverage with a lock-in test (N9). Round-1 partials with documented deferrals (S1 import idempotency, S4 streaming multipart, P2 QueryDeployments/ExportBundle internals, UA6 fleet-wide grid a11y) **stay deferred** — coverage rows only, no tasks.
**Architecture** — All fixes stay inside the seams round 1 built: `ManagementAuthenticator` is already the shared Basic→LDAP+throttle flow, so N1 is *deleting* the hub's bespoke bind and delegating; `LoginThrottle` and `SecurityOptions` already live in the Security component, so N2 adds one small pure setup class (`ForwardedHeadersSetup`) wired at the Host composition root ahead of the middleware pipeline; N3 reuses the existing `EnforceSiteScope`/`EnforceSiteScopeForIdentifier` helpers (`ManagementActor.cs:495-536`) exactly as C2's fix did for `DeployArtifacts`; N4/N5/N7 mirror guards that already exist elsewhere in the same file or in `DebugView.razor`; N6 adds a deathwatch hook to the (already reference-counted, lock-guarded) `SiteAlarmLiveCacheService`. Message-contract and repository changes are strictly additive (optional trailing params). Every spec-affecting change updates `docs/requirements/Component-Security.md` / `Component-CentralUI.md` / `Component-Communication.md` in the same task as the code.
**Tech Stack** — C#/.NET 10, Akka.NET (TestKit.Xunit2 + NSubstitute for actor tests), ASP.NET Core minimal endpoints + SignalR + Blazor Server (bUnit), EF Core (ConfigurationDatabase). Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per-project: `dotnet test tests/<project>` (CLI.Tests is in the slnx since PLAN-08 T1 — no special handling).
---
## Parallelization & mutex rules
**ManagementActor single-writer lane (initiative-wide rule, same as PLAN-07):** every task that edits `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` serializes into ONE lane — among themselves **and** against any other plan's ManagementActor task. In this round-2 wave only PLAN-R2-07 touches the file (**Tasks 4 and 6** — run T4 → T6, never concurrently), but the rule stands: check the round-2 master tracker before starting either, and if another plan later gains a ManagementActor task, this lane serializes with it.
**`SiteAlarmLiveCacheService.cs` mutex with PLAN-R2-02:** PLAN-R2-02 also modifies `src/ZB.MOM.WW.ScadaBridge.Communication/SiteAlarmLiveCacheService.cs` (delta coalescing). **Task 10 (N6) and R2-02's coalescing task must NOT run concurrently** — whichever plan's task starts first finishes and lands its commit before the other begins, and the second task rebases on the first's committed state. Coordinate through the round-2 tracker.
**`AlarmSummary.razor` internal lane:** Tasks 7, 8, 9 all edit the same file/methods — run strictly 7 → 8 → 9.
**Concurrent lanes at plan start:**
- **Free / file-disjoint:** T1 (`DebugStreamHub.cs`), T2 (`Security/ForwardedHeadersSetup.cs` + `Host/Program.cs`), T5 (`ISecuredWriteRepository` + EF repo), T7 (first of the AlarmSummary lane), T10 (subject to the R2-02 mutex), T11 (`LoginThrottle.cs`), T12 (test-only + `ConfigSecretScrubber.cs` xmldoc).
- **Ordering constraints:** 3←2; 6←(4,5); 8←7; 9←8. T4 and T6 share the ManagementActor lane (T4 first).
---
### Task 1: Route the DebugStreamHub LDAP bind through `ManagementAuthenticator` (throttled) + fix the false doc claim
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** 2, 4, 5, 7, 10, 11, 12
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/DebugStreamHub.cs` (lines 78–146: replace the bespoke bypass + Basic-decode + LDAP-bind + role-map block in `OnConnectedAsync`)
- Modify: `docs/requirements/Component-Security.md` (line 208, "Scope — every LDAP-bind surface" bullet)
- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/DebugStreamHubTests.cs` (extend — currently only tests the static `IsInstanceAccessAllowed`)
The hub currently uses `ManagementAuthenticator` **only** for the DisableLogin bypass (`TryDisableLoginUser`, :82) and then runs its own Basic decode + `ILdapAuthService.AuthenticateAsync` (:119-120) + `RoleMapper` (:129-130) — so the `LoginThrottle` never sees hub connection attempts, and hub failures don't feed the shared lockout. `ManagementAuthenticator.AuthenticateAsync` (`ManagementAuthenticator.cs:70-146`) already does all of it — bypass, decode, `IsLockedOut` refusal, bind, `RecordFailure`/`RecordSuccess`, role resolution, and system-wide `PermittedSiteIds` normalization. Delegate to it.
1. Write the failing tests. Harness: a real `DefaultHttpContext` whose `RequestServices` carries `IOptions<AuthDisableLoginOptions>` (off), `IOptions<SecurityOptions>` (defaults), a `FakeTimeProvider`-style clock as `TimeProvider`, a singleton `LoginThrottle`, and a substituted `ILdapAuthService` — mirror `ManagementAuthenticatorTests.BasicAuthContext` exactly. Wrap it for SignalR with a minimal `HubCallerContext` fake:
```csharp
private sealed class TestHubCallerContext : HubCallerContext
{
private readonly FeatureCollection _features = new();
public TestHubCallerContext(HttpContext http) =>
_features.Set<IHttpContextFeature>(new TestHttpContextFeature { HttpContext = http });
public bool Aborted { get; private set; }
public override string ConnectionId => "test-conn";
public override string? UserIdentifier => null;
public override System.Security.Claims.ClaimsPrincipal? User => null;
public override IDictionary<object, object?> Items { get; } = new Dictionary<object, object?>();
public override IFeatureCollection Features => _features;
public override CancellationToken ConnectionAborted => CancellationToken.None;
public override void Abort() => Aborted = true;
private sealed class TestHttpContextFeature : IHttpContextFeature { public HttpContext? HttpContext { get; set; } }
}
```
Build the hub with real light-weight collaborators (`DebugStreamService` needs `CommunicationService` — construct it directly: `new CommunicationService(Options.Create(new CommunicationOptions()), NullLogger<CommunicationService>.Instance)` — plus a `new SiteStreamGrpcClientFactory(NullLoggerFactory.Instance)` and a bare `ServiceCollection().BuildServiceProvider()`; `IHubContext<DebugStreamHub>` is `Substitute.For`), assign `hub.Context = testContext`, set `context.Connection.RemoteIpAddress = IPAddress.Parse("10.0.0.1")` and a Basic header for `alice:wrong`. Tests:
```csharp
[Fact]
public async Task OnConnectedAsync_FailedBinds_FeedTheSharedLoginThrottle()
{
// 5 failed hub connects for alice@10.0.0.1 must arm the same lockout the
// /management and /auth surfaces consult (arch-review R2 N1).
for (var i = 0; i < 5; i++) await ConnectAsync("alice", "wrong"); // each Aborted
Assert.True(_throttle.IsLockedOut("alice", "10.0.0.1"));
}
[Fact]
public async Task OnConnectedAsync_LockedOutKey_AbortsWithoutContactingLdap()
{
for (var i = 0; i < 5; i++) _throttle.RecordFailure("alice", "10.0.0.1");
_ldap.ClearReceivedCalls();
var ctx = await ConnectAsync("alice", "whatever");
Assert.True(ctx.Aborted);
await _ldap.DidNotReceiveWithAnyArgs().AuthenticateAsync(default!, default!, default);
}
```
(`ConnectAsync` = build hub + context, `await hub.OnConnectedAsync()`, return the fake context.)
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests --filter DebugStreamHub` — new tests FAIL (the hub binds directly; the throttle never locks and LDAP is contacted while "locked").
3. Implement — in `OnConnectedAsync`, delete lines 78–137 (the `TryDisableLoginUser` block, the Basic-decode block, the direct `ILdapAuthService` bind, and the `RoleMapper` call) and replace with:
```csharp
// Authenticate via the SAME shared Basic→LDAP flow as POST /management and the
// audit REST (arch-review R2 N1): ManagementAuthenticator applies the DisableLogin
// bypass, the LoginThrottle lockout check BEFORE the bind, and RecordFailure/
// RecordSuccess bookkeeping — so hub connection attempts can no longer be used as
// an unthrottled LDAP-bind oracle, and hub failures feed the shared lockout.
var outcome = await ManagementAuthenticator.AuthenticateAsync(httpContext);
if (outcome.User is null)
{
_logger.LogWarning("DebugStreamHub connection rejected: authentication failed or throttled");
Context.Abort();
return;
}
var user = outcome.User;
// Role check — Deployer role required (unchanged policy).
if (!user.Roles.Contains(Roles.Deployer))
{
_logger.LogWarning("DebugStreamHub connection rejected: {Username} lacks Deployer role", user.Username);
Context.Abort();
return;
}
// Persist the resolved identity on the connection so per-instance site-scope
// enforcement can be applied to SubscribeInstance calls. PermittedSiteIds is the
// authenticator's normalized form (empty == system-wide), same as every other surface.
Context.Items[RolesKey] = user.Roles;
Context.Items[PermittedSiteIdsKey] = user.PermittedSiteIds;
_logger.LogInformation("DebugStreamHub connection established for {Username}", user.Username);
await base.OnConnectedAsync();
```
Drop the now-unused `using System.Text;` / `ZB.MOM.WW.Auth.Abstractions.Ldap` imports if nothing else uses them.
4. Run the filter — PASS. Run the full project: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests` (the five `IsInstanceAccessAllowed` tests and `ManagementAuthenticatorTests` must stay green).
5. Fix `Component-Security.md:208`: the claim was false while the hub bound directly — with this change it becomes true. Reword the bullet so it *describes the mechanism* rather than asserting completeness by fiat, e.g.: "…and the HTTP-Basic management/CLI surfaces fronted by `ManagementAuthenticator` — `POST /management`, the audit REST endpoints, **and the debug-stream hub (`DebugStreamHub.OnConnectedAsync` delegates its whole credential path to `ManagementAuthenticator.AuthenticateAsync`, closing the round-2 N1 gap where the hub bound LDAP directly and bypassed the throttle)**. No bind happens outside these paths."
6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ManagementService/DebugStreamHub.cs tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/DebugStreamHubTests.cs docs/requirements/Component-Security.md && git commit -m "fix(security): throttle DebugStreamHub LDAP bind via ManagementAuthenticator + correct coverage doc (plan R2-07 T1)"`
### Task 2: `ForwardedHeadersSetup` — trusted-proxy client-IP resolution for the throttle keys
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** 1, 4, 5, 7, 10, 11, 12
**Files:**
- Create: `src/ZB.MOM.WW.ScadaBridge.Security/ForwardedHeadersSetup.cs` (options class + pure builder)
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` (central pipeline — insert immediately before `app.UseWebSockets()` at line 340)
- Test: `tests/ZB.MOM.WW.ScadaBridge.Security.Tests/ForwardedHeadersSetupTests.cs` (create)
Every throttle surface keys on `context.Connection.RemoteIpAddress` (`ManagementAuthenticator.cs:111`, `AuthEndpoints.cs:39, :126`), and the repo has zero `UseForwardedHeaders` hits — behind Traefik every client shares the proxy IP and the key degenerates to `username|<proxy-ip>` (N2). Fix: honor `X-Forwarded-For` **from the trusted proxy only**.
1. Failing tests:
```csharp
using Microsoft.AspNetCore.HttpOverrides;
using ZB.MOM.WW.ScadaBridge.Security;
using Xunit;
namespace ZB.MOM.WW.ScadaBridge.Security.Tests;
public class ForwardedHeadersSetupTests
{
[Fact]
public void Build_Disabled_ReturnsNull()
=> Assert.Null(ForwardedHeadersSetup.Build(new ForwardedHeadersSecurityOptions { Enabled = false }));
[Fact]
public void Build_Enabled_ParsesProxiesAndNetworks_AndClearsDefaults()
{
var built = ForwardedHeadersSetup.Build(new ForwardedHeadersSecurityOptions
{
Enabled = true,
KnownProxies = ["10.5.0.9"],
KnownNetworks = ["172.16.0.0/12"],
})!;
Assert.Equal(ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto, built.ForwardedHeaders);
Assert.Equal(1, built.ForwardLimit);
Assert.Single(built.KnownProxies); // ONLY the configured proxy —
Assert.DoesNotContain(System.Net.IPAddress.IPv6Loopback, // the loopback default is cleared
built.KnownProxies);
Assert.Single(built.KnownNetworks);
}
[Theory]
[InlineData("not-an-ip", "172.16.0.0/12")]
[InlineData("10.5.0.9", "not-a-cidr")]
public void Build_MalformedEntry_ThrowsAtStartup(string proxy, string network)
=> Assert.ThrowsAny<Exception>(() => ForwardedHeadersSetup.Build(
new ForwardedHeadersSecurityOptions { Enabled = true, KnownProxies = [proxy], KnownNetworks = [network] }));
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Security.Tests --filter ForwardedHeadersSetup` — FAIL (types missing).
3. Implement `ForwardedHeadersSetup.cs`:
```csharp
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.HttpOverrides;
using System.Net;
namespace ZB.MOM.WW.ScadaBridge.Security;
/// <summary>
/// Config shape for trusted-proxy forwarded-header handling
/// (<c>ScadaBridge:Security:ForwardedHeaders</c>). Disabled by default: a deployment
/// with no reverse proxy must NOT honor X-Forwarded-For (a client could spoof its
/// throttle key). The documented Traefik topologies enable it with the docker
/// network range so LoginThrottle keys on the REAL client IP (arch-review R2 N2).
/// </summary>
public sealed class ForwardedHeadersSecurityOptions
{
public const string SectionName = "ScadaBridge:Security:ForwardedHeaders";
public bool Enabled { get; set; }
/// <summary>Exact proxy IPs allowed to supply X-Forwarded-For.</summary>
public string[] KnownProxies { get; set; } = [];
/// <summary>CIDR networks allowed to supply X-Forwarded-For (e.g. "172.16.0.0/12").</summary>
public string[] KnownNetworks { get; set; } = [];
}
/// <summary>
/// Builds the <see cref="ForwardedHeadersOptions"/> for <c>app.UseForwardedHeaders</c>.
/// Pure and fail-fast: malformed entries throw at startup rather than silently
/// trusting nothing/everything. Defaults (loopback) are CLEARED — only the
/// explicitly configured proxies/networks are trusted, and ForwardLimit=1 means
/// only the entry the trusted proxy itself appended is honored.
/// </summary>
public static class ForwardedHeadersSetup
{
public static ForwardedHeadersOptions? Build(ForwardedHeadersSecurityOptions options)
{
if (!options.Enabled) return null;
var built = new ForwardedHeadersOptions
{
ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto,
ForwardLimit = 1,
};
built.KnownProxies.Clear();
built.KnownNetworks.Clear();
foreach (var proxy in options.KnownProxies)
built.KnownProxies.Add(IPAddress.Parse(proxy));
foreach (var network in options.KnownNetworks)
{
var parts = network.Split('/');
built.KnownNetworks.Add(new(IPAddress.Parse(parts[0]), int.Parse(parts[1])));
}
return built;
}
}
```
(`KnownNetworks` element type: use whatever the installed ASP.NET Core surface expects — `Microsoft.AspNetCore.HttpOverrides.IPNetwork` historically, `System.Net.IPNetwork` on newer TFMs; the collection initializer `new(...)` resolves either. Match the compiler.)
4. Wire in `Host/Program.cs` — insert immediately **before** `app.UseWebSockets();` (line 340), first middleware in the central pipeline so everything downstream (auth endpoints, `ManagementAuthenticator`, hub) reads the substituted `Connection.RemoteIpAddress`:
```csharp
// Trusted-proxy forwarded headers (arch-review R2 N2): behind Traefik every client
// shares the proxy's IP, collapsing the LoginThrottle's {username}|{ip} keys onto
// one address (per-IP isolation gone + cheap username-lockout DoS). When enabled,
// X-Forwarded-For from the CONFIGURED proxies only is honored. MUST run first —
// everything downstream keys on Connection.RemoteIpAddress.
var forwardedOptions = builder.Configuration
.GetSection(ZB.MOM.WW.ScadaBridge.Security.ForwardedHeadersSecurityOptions.SectionName)
.Get<ZB.MOM.WW.ScadaBridge.Security.ForwardedHeadersSecurityOptions>() ?? new();
if (ZB.MOM.WW.ScadaBridge.Security.ForwardedHeadersSetup.Build(forwardedOptions) is { } fho)
app.UseForwardedHeaders(fho);
```
5. Run the filter — PASS; `dotnet build ZB.MOM.WW.ScadaBridge.slnx` — clean; `dotnet test tests/ZB.MOM.WW.ScadaBridge.Security.Tests` — no regressions.
6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Security/ForwardedHeadersSetup.cs src/ZB.MOM.WW.ScadaBridge.Host/Program.cs tests/ZB.MOM.WW.ScadaBridge.Security.Tests/ForwardedHeadersSetupTests.cs && git commit -m "fix(security): trusted-proxy ForwardedHeaders so LoginThrottle keys on the real client IP (plan R2-07 T2)"`
### Task 3: Ship the ForwardedHeaders config to the Traefik topologies + document the lockout-DoS trade-off
**Classification:** high-risk
**Estimated implement time:** ~4 min
**Parallelizable with:** 1, 4, 5, 7, 10, 11, 12 (waits on Task 2)
**Files:**
- Modify: `docker/central-node-a/appsettings.Central.json`, `docker/central-node-b/appsettings.Central.json` (inside the existing `"Security"` object)
- Modify: `docker-env2/central-node-a/appsettings.Central.json`, `docker-env2/central-node-b/appsettings.Central.json` (same)
- Modify: `deploy/wonder-app-vd03/appsettings.Central.json` (explicit disabled block + comment; no proxy fronts that host)
- Modify: `docs/requirements/Component-Security.md` (§Login throttling — key derivation + new trade-off paragraph)
- Modify: `tests/ZB.MOM.WW.ScadaBridge.Security.Tests/LoginThrottleTests.cs` (comment only, line 37)
1. Add to both docker central nodes' `"Security"` section (site nodes don't serve the UI/management surface — untouched):
```json
"ForwardedHeaders": {
"_comment": "Traefik fronts this node on the external scadabridge-net docker network. Trust X-Forwarded-For from the docker bridge address pool so LoginThrottle keys on the real client IP (arch-review R2 N2). Narrow to the Traefik container IP if the network is pinned.",
"Enabled": true,
"KnownNetworks": [ "172.16.0.0/12" ]
}
```
Same block in both `docker-env2` central nodes. In `deploy/wonder-app-vd03/appsettings.Central.json` add the block with `"Enabled": false` and a comment: "No reverse proxy fronts this host today. If one is introduced, set Enabled=true and list ONLY the proxy's IP in KnownProxies — never enable without it, or clients can spoof their throttle key."
2. Validate all five JSONs: `for f in docker/central-node-{a,b}/appsettings.Central.json docker-env2/central-node-{a,b}/appsettings.Central.json deploy/wonder-app-vd03/appsettings.Central.json; do python3 -c "import json,sys;json.load(open(sys.argv[1]))" "$f" || echo "BAD $f"; done`
3. `Component-Security.md` §Login throttling — update the "Key and window" bullet: the IP half of the key is the **forwarded-header-resolved client IP** when `ScadaBridge:Security:ForwardedHeaders` names a trusted proxy, otherwise the raw connection peer. Then append a new bullet documenting the trade-off (this text is the deliverable — keep the substance):
- **Proxy topologies and the username-lockout DoS.** The key is deliberately `{username}|{IP}`, not `{username}` alone: keying on username alone would let any network-adjacent actor lock any operator out of a SCADA control surface with five wrong passwords, repeatable indefinitely. That isolation only exists when the throttle sees real client IPs — behind a proxy **without** ForwardedHeaders trust, all clients collapse onto the proxy's IP and the DoS returns. Mitigations: (a) the shipped Traefik topologies enable trusted-proxy ForwardedHeaders (so per-IP isolation is real in the documented deployment); (b) the lockout window is short by default (`LoginLockoutMinutes` = 5) and never touches the directory account itself; (c) residual risk — an attacker spraying from their *own* IP locking out a victim username *at that IP only* — is the throttle working as designed. A success-path bypass ("a correct password unlocks") was considered and rejected: it would let an attacker who has the password bypass the lockout entirely, and it converts the throttle into a password oracle during the lockout window.
4. `LoginThrottleTests.cs:37` — extend the `// per username+IP key` comment: `// per username+IP key — real isolation in production requires the trusted-proxy ForwardedHeaders config (R2 N2); without it all clients share the proxy IP`.
5. `dotnet test tests/ZB.MOM.WW.ScadaBridge.Security.Tests --filter LoginThrottle` — still green (comment-only). Optional smoke after the next `bash docker/deploy.sh`: `docker logs scadabridge-central-a` shows no ForwardedHeaders startup fault, and a failed CLI login from the host still 401s (not 429) on the first attempt.
6. Commit: `git add docker docker-env2 deploy/wonder-app-vd03/appsettings.Central.json docs/requirements/Component-Security.md tests/ZB.MOM.WW.ScadaBridge.Security.Tests/LoginThrottleTests.cs && git commit -m "fix(security): ship trusted-proxy ForwardedHeaders config for Traefik topologies + document lockout-DoS trade-off (plan R2-07 T3)"`
### Task 4: Enforce site scope on secured-write submit / approve / reject
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** 1, 2, 3, 5, 7, 10, 11, 12 (ManagementActor.cs — single-writer lane; run before Task 6)
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (`HandleSubmitSecuredWrite` :1243-1283, `HandleApproveSecuredWrite` :1285-1440, `HandleRejectSecuredWrite` :1442-1479)
- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/SecuredWriteHandlerTests.cs`
None of the four secured-write handlers checks `user.PermittedSiteIds` (verified: zero `EnforceSiteScope` occurrences in :1243-1510), while `Component-Security.md:141` says scoping "is layered on at the LDAP-mapping level" — a Verifier scoped to site 3 can today approve a **device write** to any site (N3, the same class C2 closed for `DeployArtifacts`). Ruling: **code moves, not the doc** — enforce.
1. Add a scoped-envelope helper next to the existing `Envelope` helper (`SecuredWriteHandlerTests.cs:151-153`):
```csharp
private static ManagementEnvelope ScopedEnvelope(object command, string username, string[] permittedSiteIds, params string[] roles) =>
new(new AuthenticatedUser(username, username, roles, permittedSiteIds), command, Guid.NewGuid().ToString("N"));
```
Write the failing negative (and positive) tests — every security task needs the out-of-scope rejection asserted:
```csharp
[Fact]
public void Submit_SiteScopedOperator_OutOfScopeSite_Unauthorized_NoRowInserted()
{
SeedSiteWithConnection(1, "SITE1", "Gw1", "MxGateway");
var actor = CreateActor();
actor.Tell(ScopedEnvelope(
new SubmitSecuredWriteCommand("SITE1", "Gw1", "Tag.A", "true", "Boolean", "go"),
"alice", ["3"], "Operator"));
ExpectMsg<ManagementUnauthorized>(TimeSpan.FromSeconds(5));
_securedWriteRepo.DidNotReceiveWithAnyArgs().AddAsync(default!, default);
}
[Fact]
public void Submit_SiteScopedOperator_InScopeSite_Succeeds()
{
SeedSiteWithConnection(1, "SITE1", "Gw1", "MxGateway");
var actor = CreateActor();
actor.Tell(ScopedEnvelope(
new SubmitSecuredWriteCommand("SITE1", "Gw1", "Tag.A", "true", "Boolean", "go"),
"alice", ["1"], "Operator"));
ExpectMsg<ManagementSuccess>(TimeSpan.FromSeconds(5));
}
[Fact]
public void Approve_SiteScopedVerifier_OutOfScopeSite_Unauthorized_NeverRelays()
{
SeedSiteWithConnection(1, "SITE1", "Gw1", "MxGateway");
_securedWriteRepo.GetAsync(42, Arg.Any<CancellationToken>()).Returns(new PendingSecuredWrite
{
Id = 42, SiteId = "SITE1", ConnectionName = "Gw1", TagPath = "Tag.A",
ValueJson = "true", ValueType = "Boolean", Status = "Pending",
OperatorUser = "alice", SubmittedAtUtc = DateTime.UtcNow,
});
var actor = CreateActor();
actor.Tell(ScopedEnvelope(new ApproveSecuredWriteCommand(42, "ok"), "bob", ["3"], "Verifier"));
ExpectMsg<ManagementUnauthorized>(TimeSpan.FromSeconds(5));
_securedWriteRepo.DidNotReceiveWithAnyArgs().TryMarkApprovedAsync(default, default!, default, default, default);
Assert.Equal(0, _comms.CallCount); // the device write is NEVER relayed
}
[Fact]
public void Reject_SiteScopedVerifier_OutOfScopeSite_Unauthorized() { /* same seed, RejectSecuredWriteCommand(42, "no") → ManagementUnauthorized; UpdateAsync not received */ }
[Fact]
public void Approve_ScopedAdministrator_BypassesSiteScope() { /* ScopedEnvelope(..., "carol", ["3"], "Verifier", "Administrator") on the SITE1 row → NOT ManagementUnauthorized (EnforceSiteScope admin bypass, ManagementActor.cs:499) */ }
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests --filter SecuredWrite` — new tests FAIL (no scope check exists).
3. Implement — three insertions, all **before** any state mutation or audit emission:
- `HandleSubmitSecuredWrite`: the site entity is already loaded at :1247-1248, so use the cheap overload right after it:
```csharp
// Site scope (arch-review R2 N3): an LDAP-mapping-scoped Operator may only
// submit device writes to their permitted sites — same rule C2 applies to
// DeployArtifacts, on a higher-consequence path. Checked before the row exists.
EnforceSiteScope(user, site.Id);
```
- `HandleApproveSecuredWrite`: after the row load + Pending check (:1289-1294), **before** the TTL/self-approval/CAS chain:
```csharp
// Site scope (arch-review R2 N3): the approving Verifier must be permitted on
// the row's target site — checked BEFORE the TTL/self-approval/CAS chain so an
// out-of-scope verifier can neither consume the Pending transition nor relay.
await EnforceSiteScopeForIdentifier(sp, user, row.SiteId);
```
- `HandleRejectSecuredWrite`: same line after :1447-1451, against `entity.SiteId`.
4. Run the filter — PASS. Run the full project (`dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests`) — the existing submit/approve/reject tests use the system-wide `Envelope` helper (`PermittedSiteIds` empty), so `EnforceSiteScope` no-ops and they stay green. `RequiredRoleMatrixTests` is untouched: no role-set changes, only in-handler scope enforcement (the frozen table freezes *roles*, not scope — see its header comment "Do not regenerate it mechanically"). If it fails, STOP: something else changed.
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/SecuredWriteHandlerTests.cs && git commit -m "fix(security): enforce LDAP-mapping site scope on secured-write submit/approve/reject (plan R2-07 T4)"`
### Task 5: `ISecuredWriteRepository` — additive permitted-sites filter for scoped listing
**Classification:** standard
**Estimated implement time:** ~4 min
**Parallelizable with:** 1, 2, 3, 4, 7, 10, 11, 12
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/ISecuredWriteRepository.cs` (`QueryAsync` :51, `CountAsync` :112)
- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SecuredWriteRepository.cs` (`QueryAsync` :55, `CountAsync` :131)
- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/SecuredWriteRepositoryTests.cs` (extend)
Task 6 needs to scope the *unfiltered* list for a site-scoped caller at the query level (so rows **and** `totalCount` are both scoped, and paging stays correct). Give both query methods an additive optional filter.
1. Failing tests (mirror the existing `SecuredWriteRepositoryTests` seeding — rows across site identifiers `"SITE1"`, `"SITE2"`, `"SITE3"`):
```csharp
[Fact]
public async Task QueryAsync_PermittedSiteIds_ReturnsOnlyPermittedRows()
{
// seed 2 rows on SITE1, 1 on SITE2, 1 on SITE3 (existing seeding helper)
var rows = await _repo.QueryAsync(status: null, siteId: null, skip: 0, take: 100,
permittedSiteIds: new[] { "SITE1", "SITE3" });
Assert.Equal(3, rows.Count);
Assert.DoesNotContain(rows, r => r.SiteId == "SITE2");
}
[Fact]
public async Task CountAsync_PermittedSiteIds_CountsOnlyPermittedRows()
{
var count = await _repo.CountAsync(status: null, siteId: null,
permittedSiteIds: new[] { "SITE1", "SITE3" });
Assert.Equal(3, count);
}
[Fact]
public async Task QueryAsync_NullPermittedSiteIds_IsUnfiltered_BackCompat()
{ /* null → all 4 rows, exactly today's behavior */ }
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter SecuredWriteRepository` — FAIL (no such parameter).
3. Implement — additive optional parameter before the `CancellationToken` on both members (interface + impl):
```csharp
Task<IReadOnlyList<PendingSecuredWrite>> QueryAsync(
string? status, string? siteId, int skip, int take,
IReadOnlyCollection<string>? permittedSiteIds = null, CancellationToken ct = default);
Task<int> CountAsync(
string? status, string? siteId,
IReadOnlyCollection<string>? permittedSiteIds = null, CancellationToken ct = default);
```
xmldoc: "`permittedSiteIds` — when non-null, only rows whose `SiteId` (site identifier) is in the set match; `null` matches every site. Applied IN ADDITION to `siteId`. Used by the ManagementActor to constrain an unfiltered list to a site-scoped caller's permitted sites at the query level (arch-review R2 N3), so paging and totals stay correct." EF impl, in both methods' filter composition:
```csharp
if (permittedSiteIds is not null)
query = query.Where(w => permittedSiteIds.Contains(w.SiteId));
```
4. Run the filter — PASS; full project run (existing positional callers compile unchanged — the parameter is trailing-optional).
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/ISecuredWriteRepository.cs src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SecuredWriteRepository.cs tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/SecuredWriteRepositoryTests.cs && git commit -m "fix(security): additive permitted-sites filter on secured-write query/count (plan R2-07 T5)"`
### Task 6: Scope-filter `HandleListSecuredWrites` + amend `Component-Security.md:141` + matrix verification
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** none (ManagementActor.cs — single-writer lane; waits on Tasks 4 and 5)
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (`HandleListSecuredWrites` :1481-1509; dispatch arm :470)
- Modify: `docs/requirements/Component-Security.md` (line 141 — the "any site scoping is layered on at the LDAP-mapping level" sentence)
- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/SecuredWriteHandlerTests.cs`
1. Failing tests:
```csharp
[Fact]
public void List_SiteScopedReader_ExplicitOutOfScopeSiteFilter_Unauthorized()
{
SeedSiteWithConnection(1, "SITE1", "Gw1", "MxGateway");
var actor = CreateActor();
actor.Tell(ScopedEnvelope(new ListSecuredWritesCommand(null, "SITE1"), "bob", ["3"], "Verifier"));
ExpectMsg<ManagementUnauthorized>(TimeSpan.FromSeconds(5));
}
[Fact]
public void List_SiteScopedReader_Unfiltered_QueriesOnlyPermittedIdentifiers()
{
_siteRepo.GetAllSitesAsync(Arg.Any<CancellationToken>()).Returns(new List<Site>
{
new("Site1", "SITE1") { Id = 1 },
new("Site3", "SITE3") { Id = 3 },
});
_securedWriteRepo.QueryAsync(null, null, 0, Arg.Any<int>(),
Arg.Any<IReadOnlyCollection<string>?>(), Arg.Any<CancellationToken>())
.Returns(new List<PendingSecuredWrite>());
var actor = CreateActor();
actor.Tell(ScopedEnvelope(new ListSecuredWritesCommand(null, null), "bob", ["3"], "Verifier"));
ExpectMsg<ManagementSuccess>(TimeSpan.FromSeconds(5));
_securedWriteRepo.Received(1).QueryAsync(null, null, 0, Arg.Any<int>(),
Arg.Is<IReadOnlyCollection<string>?>(s => s != null && s.Single() == "SITE3"),
Arg.Any<CancellationToken>());
}
[Fact]
public void List_SystemWideReader_PassesNullPermittedFilter() { /* Envelope(...) → QueryAsync received with permittedSiteIds == null */ }
```
(Match the real `ListSecuredWritesCommand` parameter order — read `SecuredWriteCommands.cs:63`: `(Status, SiteId, Skip, Take)`.)
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests --filter List_SiteScoped` — FAIL.
3. Implement — change the handler signature to take the user and the dispatch arm at :470 from `HandleListSecuredWrites(sp, cmd)` to `HandleListSecuredWrites(sp, cmd, user)`:
```csharp
private static async Task<object?> HandleListSecuredWrites(
IServiceProvider sp, ListSecuredWritesCommand cmd, AuthenticatedUser user)
{
// Site scoping (arch-review R2 N3): an explicit SiteId filter is scope-checked
// like every other identifier-keyed command; an UNFILTERED list from a
// site-scoped non-admin caller is constrained to their permitted sites at the
// query level (rows AND totalCount both scoped — see ISecuredWriteRepository).
IReadOnlyCollection<string>? permittedIdentifiers = null;
if (cmd.SiteId is not null)
{
await EnforceSiteScopeForIdentifier(sp, user, cmd.SiteId);
}
else if (user.PermittedSiteIds.Length > 0
&& !user.Roles.Contains(Roles.Administrator, StringComparer.OrdinalIgnoreCase))
{
var siteRepo = sp.GetRequiredService<ISiteRepository>();
var permitted = new HashSet<string>(user.PermittedSiteIds);
permittedIdentifiers = (await siteRepo.GetAllSitesAsync())
.Where(s => permitted.Contains(s.Id.ToString()))
.Select(s => s.SiteIdentifier)
.ToList();
}
var repo = sp.GetRequiredService<ISecuredWriteRepository>();
var take = Math.Clamp(cmd.Take, 1, 500);
var skip = Math.Max(0, cmd.Skip);
var rows = await repo.QueryAsync(cmd.Status, cmd.SiteId, skip, take, permittedIdentifiers);
var totalCount = await repo.CountAsync(cmd.Status, cmd.SiteId, permittedIdentifiers);
// ... (opportunistic expiry sweep unchanged)
```
4. Run the filter — PASS. Then the lock-in suites — the frozen table needs **no regeneration** (role sets untouched; per its header, the table freezes the `GetRequiredRoles` switch, which this task does not edit) and the dispatch arm still reaches a handler:
`dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests --filter "RequiredRoleMatrixTests|DispatchCoverageTests"` — green. If `RequiredRoleMatrixTests` DOES fail, follow its documented update procedure (hand-author the changed entry against the switch + cross-check `Component-ManagementService.md` §Authorization — never regenerate mechanically). Full project run.
5. Amend `Component-Security.md:141` — replace "Both are coarse global roles like the others; any site scoping is layered on at the LDAP-mapping level." with: "Both are coarse global roles like the others. **Site scoping from the LDAP mapping is enforced server-side on every secured-write command** (arch-review R2 N3): submit checks the target site, approve/reject check the row's site (before the TTL/self-approval/CAS chain), and the list constrains a scoped caller to their permitted sites — mirroring the deployment commands' `EnforceSiteScope` rule. Administrators and system-wide principals (empty permitted-site set) are unrestricted."
6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/SecuredWriteHandlerTests.cs docs/requirements/Component-Security.md && git commit -m "fix(security): scope-filter secured-write listing + align spec with enforced site scoping (plan R2-07 T6)"`
### Task 7: `AlarmSummary.RefreshAsync` stale-site guard — never display cross-site data
**Classification:** standard
**Estimated implement time:** ~4 min
**Parallelizable with:** 1, 2, 3, 4, 5, 10, 11, 12 (AlarmSummary.razor lane — run before Tasks 8, 9)
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/AlarmSummary.razor` (`RefreshAsync` :283-308)
- Test: `tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Monitoring/AlarmSummaryRenderTests.cs`
`RefreshAsync` captures `siteId` at entry (:285), awaits the multi-second fan-out (:293), then unconditionally assigns `_rows`/`_notReporting`/`_rollup` (:294-296) — a site-A poll completing after a switch to site B overwrites B's alarms with A's, on an operator alarm page (N4). The live path already has exactly this guard (`OnLiveAlarmsChanged`, :374).
1. Failing bUnit test (the harness's constructor stubs `GetSiteAlarmsAsync(Arg.Any<int>, ...)`; override per-site inside the test):
```csharp
[Fact]
public void InFlightPollForPreviousSite_DoesNotOverwriteNewSitesRows()
{
// Site 1's fan-out hangs on a TCS; site 2 answers immediately (arch-review R2 N4).
var site1Tcs = new TaskCompletionSource<AlarmSummaryResult>();
_summary.GetSiteAlarmsAsync(1, Arg.Any<CancellationToken>()).Returns(site1Tcs.Task);
_summary.GetSiteAlarmsAsync(2, Arg.Any<CancellationToken>()).Returns(Task.FromResult(
new AlarmSummaryResult(
new List<AlarmSummaryRow>
{
new("Site2Instance", new AlarmStateChanged("Site2Instance", "S2-alarm", AlarmState.Active, 100, DateTimeOffset.UtcNow)),
},
Array.Empty<string>())));
var cut = Render<AlarmSummaryPage>();
cut.Find("[data-test='alarm-summary-site']").Change("1"); // poll in flight, hung
cut.Find("[data-test='alarm-summary-site']").Change("2"); // site 2 renders
cut.WaitForAssertion(() => Assert.Equal("Site2Instance", FirstRowInstance(cut)));
// The stale site-1 fan-out now completes — it must be dropped, not rendered.
site1Tcs.SetResult(new AlarmSummaryResult(
new List<AlarmSummaryRow>
{
new("StaleSite1", new AlarmStateChanged("StaleSite1", "S1-alarm", AlarmState.Active, 999, DateTimeOffset.UtcNow)),
},
new[] { "StaleNotReporting" }));
cut.WaitForAssertion(() =>
{
Assert.Equal("Site2Instance", FirstRowInstance(cut));
Assert.DoesNotContain("StaleNotReporting", cut.Markup);
});
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests --filter AlarmSummary` — new test FAILS (StaleSite1 overwrites the page).
3. Implement — one guard in `RefreshAsync`, immediately after the await at :293:
```csharp
var result = await AlarmSummaryService.GetSiteAlarmsAsync(siteId);
// Stale-site guard (arch-review R2 N4): the operator may have switched sites
// while this fan-out was in flight — drop the result rather than labeling
// site A's alarms under site B's picker. Mirrors OnLiveAlarmsChanged (:374).
if (_selectedSiteId != siteId)
{
return;
}
```
(`_loading` is reset in the existing `finally` — correct either way, since a switch re-enters `RefreshAsync` for the new site.)
4. Run the filter — PASS, plus the pre-existing AlarmSummary tests stay green.
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/AlarmSummary.razor tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Monitoring/AlarmSummaryRenderTests.cs && git commit -m "fix(ui): drop stale-site poll results on Alarm Summary site switch (plan R2-07 T7)"`
### Task 8: While live, the poll updates only `_notReporting` — never regresses live rows
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none (AlarmSummary.razor lane; waits on Task 7)
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/AlarmSummary.razor` (`RefreshAsync` :283-308; reconciliation comment block :337-351)
- Modify: `docs/requirements/Component-CentralUI.md` (line 194, the Refresh bullet)
- Test: `tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Monitoring/AlarmSummaryRenderTests.cs`
The 15 s poll always rebuilds `_rows` from the fan-out even when the cache is live; a poll whose fan-out started before a live delta lands after it and momentarily reverts the delta (N5). The comment at :344-347 ("the poll simply re-affirms the same snapshot") is wrong — fix both.
1. Failing bUnit test (uses the manual Refresh button, `data-test="alarm-summary-refresh"`, to drive the poll path deterministically — it invokes the same `RefreshAsync` the timer does):
```csharp
[Fact]
public void PollWhileLive_UpdatesNotReportingOnly_DoesNotRegressLiveRows()
{
// Poll snapshot = Zeta/Alpha (constructor stub) + a not-reporting instance.
_summary.GetSiteAlarmsAsync(Arg.Any<int>(), Arg.Any<CancellationToken>())
.Returns(ci => Task.FromResult(new AlarmSummaryResult(
new List<AlarmSummaryRow>
{
new("Zeta", new AlarmStateChanged("Zeta", "Z-alarm", AlarmState.Active, 900, DateTimeOffset.UtcNow)),
},
new[] { "OfflineInstance" })));
var cut = RenderWithSiteSelected();
// Go live with a fresher delta: Gamma only (arch-review R2 N5).
_liveCache.PushAlarms(new List<AlarmStateChanged>
{
new("Gamma", "G-alarm", AlarmState.Active, 300, DateTimeOffset.UtcNow),
});
cut.WaitForAssertion(() => Assert.Equal("Gamma", FirstRowInstance(cut)));
// A poll now completes: it owns _notReporting but must NOT rebuild the rows.
cut.Find("[data-test='alarm-summary-refresh']").Click();
cut.WaitForAssertion(() =>
{
Assert.Contains("OfflineInstance", cut.Markup); // notReporting refreshed
Assert.Equal("Gamma", FirstRowInstance(cut)); // live rows NOT regressed
Assert.Single(cut.FindAll("tr[data-test='alarm-summary-row']"));
});
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests --filter AlarmSummary` — FAILS (the poll rebuilds Zeta over Gamma).
3. Implement — in `RefreshAsync`, after Task 7's stale-site guard, replace the unconditional assignments:
```csharp
// _notReporting is the poll's unique authority — the alarm-only live cache
// cannot compute it — so it is always refreshed.
_notReporting = result.NotReportingInstances;
// While the cache is live, the live deltas own the row set: a poll whose fan-out
// started BEFORE a delta must not land after it and momentarily revert the alarm
// state (arch-review R2 N5). When not live (pre-seed / degraded stream / dead
// aggregator — see R2 N6), the poll remains the full-rebuild safety net.
if (!LiveAlarmCache.IsLive(siteId))
{
_rows = result.Alarms;
_rollup = AlarmSummaryService.ComputeRollup(_rows);
}
RecomputeVisibleRows();
```
Rewrite the stale sentence in the reconciliation comment block (:344-347): the poll is "the authority for `_notReporting` and the full-rebuild safety net **only while the cache is not live**; when live it deliberately leaves `_rows` to the delta path".
4. Run the filter — PASS. Note interplay checks: `PollFallbackRenders_WhenCacheNotLiveYet` still passes (not live → full rebuild); `OnChangedDelta_RebuildsRenderedRowsFromLiveSnapshot` still passes (live path untouched).
5. Update `Component-CentralUI.md:194`: append "…when live, the poll updates only the `NotReporting` list and leaves the row set to the delta path, so a slow fan-out can never momentarily revert a fresher live delta (R2 N5)."
6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/AlarmSummary.razor tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Monitoring/AlarmSummaryRenderTests.cs docs/requirements/Component-CentralUI.md && git commit -m "fix(ui): Alarm Summary poll defers row rebuilds to the live path while IsLive (plan R2-07 T8)"`
### Task 9: House disposal guard on the live callback
**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** none (AlarmSummary.razor lane; waits on Task 8)
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/AlarmSummary.razor` (`OnLiveAlarmsChanged` :367-382, `Dispose` :507-511)
- Test: `tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Monitoring/AlarmSummaryRenderTests.cs` (+ add a `CapturedCallback` accessor to `FakeSiteAlarmLiveCache`)
`OnLiveAlarmsChanged` does `_ = InvokeAsync(...)` with no `_disposed` flag; a callback racing `Dispose()` can fault the discarded task with `ObjectDisposedException` from `StateHasChanged` on a torn-down renderer (N7). `DebugView.razor` established the pattern (`_disposed` set before teardown + checked before AND inside the marshalled work — :312, :430-433, :606).
1. Failing test — expose the raw registered callback on the fake so the test can simulate a delta already in flight when `Dispose` runs (bUnit's renderer stays alive after `cut.Instance.Dispose()`, so pre-fix the late callback observably rebuilds the rows):
```csharp
// In FakeSiteAlarmLiveCache:
public Action? CapturedCallback => _onChanged;
[Fact]
public void LiveCallbackRacingDispose_IsDropped_NoRebuildNoThrow()
{
var cut = RenderWithSiteSelected();
_liveCache.PushAlarms(new List<AlarmStateChanged>
{
new("Gamma", "G-alarm", AlarmState.Active, 300, DateTimeOffset.UtcNow),
});
cut.WaitForAssertion(() => Assert.Equal("Gamma", FirstRowInstance(cut)));
// Snapshot the callback BEFORE disposal — this models a delta thread that
// already read the delegate when Dispose ran (arch-review R2 N7).
var inFlight = _liveCache.CapturedCallback!;
cut.Instance.Dispose();
_liveCache.PushAlarms(new List<AlarmStateChanged>()); // mutate the fake's snapshot
var ex = Record.Exception(inFlight);
Assert.Null(ex);
// The disposed page must not have re-applied the (now empty) live snapshot.
Assert.Equal("Gamma", FirstRowInstance(cut));
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests --filter AlarmSummary` — FAILS (the late callback re-applies the empty snapshot: zero rows).
3. Implement — mirror `DebugView.razor`:
```csharp
// N7: set BEFORE teardown so a live callback racing Dispose is dropped both
// before the InvokeAsync marshal and inside it (mirrors DebugView.razor).
private volatile bool _disposed;
private void OnLiveAlarmsChanged(int siteId)
{
if (_disposed) return;
_ = InvokeAsync(() =>
{
if (_disposed || _selectedSiteId != siteId || !LiveAlarmCache.IsLive(siteId))
{
return;
}
ApplyLiveSnapshot(siteId);
StateHasChanged();
});
}
public void Dispose()
{
_disposed = true;
StopTimer();
DisposeLiveSubscription();
}
```
4. Run the filter — PASS (including `DisposingComponent_UnsubscribesFromTheLiveCache`).
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/AlarmSummary.razor tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Monitoring/AlarmSummaryRenderTests.cs && git commit -m "fix(ui): disposal guard on Alarm Summary live callback, matching the DebugView pattern (plan R2-07 T9)"`
### Task 10: Un-stick `IsLive` — deathwatch resets liveness when the aggregator terminates
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** 1, 2, 3, 4, 5, 7, 11, 12 — **MUTEX: must NOT run concurrently with PLAN-R2-02's `SiteAlarmLiveCacheService` delta-coalescing task** (see "Parallelization & mutex rules"); whichever starts first lands its commit before the other begins
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/SiteAlarmLiveCacheService.cs` (`StartAggregatorAsync` :219-243; new nested watcher actor + `OnAggregatorTerminated`; `SiteEntry` :416-453)
- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/ISiteAlarmLiveCache.cs` (`IsLive` xmldoc :53-61)
- Modify: `docs/requirements/Component-CentralUI.md` (line 194 — "when a stream is unhealthy" wording gains the aggregator-death case)
- Test: `tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/SiteAlarmLiveCacheServiceTests.cs`
`SiteEntry.HasPublished` is set on first publish (:392-393) and never cleared, so `IsLive` (:132-136) reports `true` forever — and `OnSiteChangedAsync` deliberately applies the live snapshot **on top of** the fresh poll when `IsLive` (`AlarmSummary.razor:277-280`), grafting an arbitrarily stale frozen snapshot over correct data if the aggregator died (N6). Fix: deathwatch the aggregator and reset liveness (which also makes the page's poll the authority again, via Task 8's `!IsLive` branch).
1. Failing tests (the harness runs a real TestKit `ActorSystem` — `SiteAlarmLiveCacheServiceTests` — and the aggregator is a real child at `/user/site-alarm-aggregator-{siteId}-{guid}`):
```csharp
[Fact]
public async Task Aggregator_Death_Resets_IsLive_And_Clears_The_Snapshot()
{
var service = CreateService(TimeSpan.FromMinutes(5), out _);
using var sub = service.Subscribe(SiteId, () => { });
AwaitAssert(() => Assert.True(service.IsLive(SiteId)), TimeSpan.FromSeconds(10));
// Kill the aggregator out from under the service (crash-death, NOT a linger stop).
var aggregator = await Sys.ActorSelection($"/user/site-alarm-aggregator-{SiteId}-*")
.ResolveOne(TimeSpan.FromSeconds(5));
Sys.Stop(aggregator);
AwaitAssert(() =>
{
Assert.False(service.IsLive(SiteId)); // sticky no more (R2 N6)
Assert.Empty(service.GetCurrentAlarms(SiteId)); // frozen snapshot never grafted
}, TimeSpan.FromSeconds(10));
}
[Fact]
public void Linger_Stop_Does_Not_Resurrect_The_Aggregator()
{
// Last viewer leaves, linger fires, entry removed: the deathwatch on the
// deliberately-stopped actor must be a no-op (entry gone), not a restart.
var service = CreateService(TimeSpan.FromMilliseconds(50), out var factory);
var sub = service.Subscribe(SiteId, () => { });
AwaitAssert(() => Assert.True(service.IsLive(SiteId)), TimeSpan.FromSeconds(10));
sub.Dispose();
AwaitAssert(() => Assert.False(service.IsLive(SiteId)), TimeSpan.FromSeconds(10));
var startsAfterStop = factory.GetOrCreateCount;
Thread.Sleep(300);
Assert.Equal(startsAfterStop, factory.GetOrCreateCount); // no self-heal restart fired
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Communication.Tests --filter SiteAlarmLiveCache` — first test FAILS (`IsLive` stays true after the stop).
3. Implement:
- In `StartAggregatorAsync`, inside the lock right after `entry.Actor = system.ActorOf(props, ...)` (:238), capture and watch:
```csharp
var aggregator = entry.Actor;
// Deathwatch (arch-review R2 N6): if the aggregator terminates for any reason,
// liveness must reset — a dead feed must never keep IsLive == true, or the page
// grafts a frozen snapshot over fresh poll data for up to 15 s.
system.ActorOf(Props.Create(() => new AggregatorTerminationWatcher(
aggregator!, () => OnAggregatorTerminated(entry.SiteId, aggregator!))));
```
- New members on the service:
```csharp
/// <summary>Watches one aggregator and fires a callback exactly once on Terminated.</summary>
private sealed class AggregatorTerminationWatcher : ReceiveActor
{
public AggregatorTerminationWatcher(IActorRef watched, Action onTerminated)
{
Context.Watch(watched);
Receive<Terminated>(_ =>
{
onTerminated();
Context.Stop(Self);
});
}
}
/// <summary>
/// Terminated hook (R2 N6). A DELIBERATE stop (linger fired, entry already removed
/// from _sites) is a no-op — the ReferenceEquals guard also ignores a stale watcher
/// firing after a newer aggregator replaced the dead one. A crash-death with live
/// viewers resets liveness (IsLive → false, snapshot cleared, so the page's poll is
/// authoritative again) and arms the existing bounded start-retry so the feature
/// self-heals, mirroring StartAggregatorAsync's transient-failure path.
/// </summary>
private void OnAggregatorTerminated(int siteId, IActorRef deadActor)
{
lock (_lock)
{
if (!_sites.TryGetValue(siteId, out var entry) || !ReferenceEquals(entry.Actor, deadActor))
return;
entry.Actor = null;
entry.HasPublished = false;
entry.Current = Empty;
if (entry.Subscribers.Count > 0 && !entry.Starting)
{
entry.Starting = true;
entry.StartRetryTimer?.Dispose();
entry.StartRetryTimer = new Timer(
_ => { _ = StartAggregatorAsync(entry); },
state: null,
dueTime: _options.LiveAlarmCacheReconcileInterval,
period: Timeout.InfiniteTimeSpan);
}
}
_logger.LogWarning(
"Live alarm aggregator for site {SiteId} terminated unexpectedly; liveness reset " +
"(page falls back to polling) and a restart is armed while viewers remain", siteId);
}
```
- `ISiteAlarmLiveCache.IsLive` xmldoc: replace "true once the site's aggregator has seeded and published at least once" with "true while a **living** aggregator has seeded and published; reverts to false if the aggregator terminates (R2 N6) — a frozen snapshot is never reported as live".
4. Run the filter — PASS (all pre-existing lifecycle tests — shared start, linger stop, re-subscribe, cap, self-heal — must stay green; the linger-stop path removes the entry before the actor stops, so the new hook's `TryGetValue` guard keeps it inert there).
5. Update `Component-CentralUI.md:194`: "…when a stream is unhealthy, **the aggregator has died (deathwatch resets `IsLive`)**, or a site has not yet seeded, the poll keeps the page fresh…"
6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Communication/SiteAlarmLiveCacheService.cs src/ZB.MOM.WW.ScadaBridge.Communication/ISiteAlarmLiveCache.cs tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/SiteAlarmLiveCacheServiceTests.cs docs/requirements/Component-CentralUI.md && git commit -m "fix(sitestream): deathwatch resets live-cache liveness on aggregator termination (plan R2-07 T10)"`
### Task 11: Amortize `LoginThrottle.Prune` off the failure hot path
**Classification:** standard
**Estimated implement time:** ~4 min
**Parallelizable with:** 1, 2, 4, 5, 7, 10, 12
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.Security/LoginThrottle.cs` (`RecordFailure` :121, `Prune` :146-169; new counter field)
- Test: `tests/ZB.MOM.WW.ScadaBridge.Security.Tests/LoginThrottleTests.cs`
`RecordFailure` → `Prune` enumerates all entries (and may `OrderBy` the whole map) on **every** failed bind — during the exact spray the throttle exists for, every request pays an O(n≤10k) scan (N8). Amortize: sweep every Nth failure, plus immediately whenever the map exceeds the hard cap (the cap stays the memory-safety invariant; only the expired-entry housekeeping is batched).
1. Failing test (needs a diagnostics accessor — `LoginThrottle` already has `InternalsVisibleTo` for Security.Tests via the csproj):
```csharp
[Fact]
public void KeySpray_MapStaysBounded_AtTheHardCap()
{
var (throttle, _) = Create(); // existing helper: throttle over a fake clock
// Spray far past the cap: the over-cap eviction must run on EVERY write once
// the cap is exceeded, regardless of the amortized cadence (arch-review R2 N8).
for (var i = 0; i < LoginThrottle.MaxTrackedKeys + 500; i++)
throttle.RecordFailure($"user{i}", "10.0.0.1");
Assert.True(throttle.TrackedKeyCount <= LoginThrottle.MaxTrackedKeys);
}
[Fact]
public void ExpiredEntries_AreEventuallySwept_OnTheAmortizedCadence()
{
var (throttle, clock) = Create();
throttle.RecordFailure("stale", "10.0.0.1");
clock.Advance(TimeSpan.FromMinutes(10)); // window fully expired
for (var i = 0; i < LoginThrottle.PruneEveryNFailures; i++)
throttle.RecordFailure($"fresh{i}", "10.0.0.1"); // cadence tick fires within these
Assert.False(throttle.IsLockedOut("stale", "10.0.0.1"));
Assert.True(throttle.TrackedKeyCount <= LoginThrottle.PruneEveryNFailures); // "stale" swept
}
```
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Security.Tests --filter LoginThrottle` — FAIL (compile: `TrackedKeyCount`/`PruneEveryNFailures` missing).
3. Implement:
```csharp
/// <summary>Amortization cadence: a full prune sweep runs every Nth failure write
/// (plus immediately whenever the map exceeds <see cref="MaxTrackedKeys"/>), so a
/// spray does not pay an O(n) scan — and a possible full sort — on every failed
/// request (arch-review R2 N8). Lockout SEMANTICS are unaffected: IsLockedOut reads
/// window/lockout expiry directly and never depends on pruning.</summary>
internal const int PruneEveryNFailures = 64;
/// <summary>Diagnostics/test accessor: number of currently tracked keys.</summary>
internal int TrackedKeyCount => _entries.Count;
private int _failureWrites;
```
Replace the unconditional `Prune(now);` at :121 with:
```csharp
// Amortized prune (R2 N8): every Nth failure, or immediately when over the hard
// cap — the cap is the memory-safety invariant and must never wait for the cadence.
if (Interlocked.Increment(ref _failureWrites) % PruneEveryNFailures == 0
|| _entries.Count > MaxTrackedKeys)
{
Prune(now);
}
```
4. Run the filter — PASS, and ALL seven pre-existing throttle tests must stay green unchanged (they assert lockout semantics, which never depended on pruning).
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Security/LoginThrottle.cs tests/ZB.MOM.WW.ScadaBridge.Security.Tests/LoginThrottleTests.cs && git commit -m "fix(security): amortize LoginThrottle.Prune off the failed-bind hot path (plan R2-07 T11)"`
### Task 12: Lock in the scrubber's fragment coverage + document the array-merge limitation
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** 1, 2, 4, 5, 7, 10, 11
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ConfigSecretScrubber.cs` (`IsSecretName` :21 → `internal`; `MergeSentinels` xmldoc :121-126)
- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/ConfigSecretScrubberTests.cs` (extend)
Detection is name-fragment-based (`password|secret|token|apikey|credential|passphrase`, :18-19). Current config types are fully covered, but a future field named `SigningKey` or `Pin` would silently leak (N9). Freeze the coverage the same way `RequiredRoleMatrixTests` froze the authz table: enumerate every string property on the DataConnections config types and demand each is *either* fragment-matched (scrubbed) *or* on a deliberate non-secret allowlist.
1. Change `IsSecretName` from `private` to `internal` (the csproj already has `InternalsVisibleTo` for the test project). Write the failing test:
```csharp
using System.Reflection;
using ZB.MOM.WW.ScadaBridge.Commons.Types.DataConnections;
// Frozen non-secret allowlist (arch-review R2 N9), mirroring RequiredRoleMatrixTests:
// every string-typed property on the DataConnections config types must EITHER match
// the scrubber's secret fragments (it gets elided) OR be listed here after a
// deliberate "this is not a secret" decision. A new string property with neither
// fails EveryConfigStringProperty_IsScrubbedOrDeliberatelyNonSecret, so a future
// "SigningKey"/"Pin" cannot silently leak through List/Get/audit.
private static readonly Dictionary<Type, string[]> KnownNonSecretStringProperties = new()
{
[typeof(OpcUaEndpointConfig)] = ["EndpointUrl", "SubscriptionDisplayName"],
[typeof(OpcUaUserIdentityConfig)] = ["Username", "CertificatePath"],
[typeof(MxGatewayEndpointConfig)] = ["Endpoint", "ClientName", "CaFile", "ServerName"],
[typeof(OpcUaHeartbeatConfig)] = [/* enumerate on first run; likely none */],
[typeof(OpcUaDeadbandConfig)] = [/* likely none */],
};
[Fact]
public void EveryConfigStringProperty_IsScrubbedOrDeliberatelyNonSecret()
{
foreach (var (type, allowlist) in KnownNonSecretStringProperties)
{
var stringProps = type.GetProperties(BindingFlags.Public | BindingFlags.Instance)
.Where(p => p.PropertyType == typeof(string))
.Select(p => p.Name);
foreach (var name in stringProps)
{
var scrubbed = ConfigSecretScrubber.IsSecretName(name);
var allowlisted = allowlist.Contains(name);
Assert.True(scrubbed ^ allowlisted,
$"{type.Name}.{name}: must be EITHER fragment-scrubbed OR explicitly " +
"allowlisted as non-secret (and never both) — see arch-review R2 N9.");
}
}
}
[Theory]
[InlineData("Password")]
[InlineData("CertificatePassword")]
[InlineData("ApiKey")]
public void KnownSecretProperties_MatchTheFragmentList(string name)
=> Assert.True(ConfigSecretScrubber.IsSecretName(name));
```
Cross-check the allowlist against the actual types before committing (`src/ZB.MOM.WW.ScadaBridge.Commons/Types/DataConnections/` — `OpcUaEndpointConfig.cs`, `OpcUaUserIdentityConfig.cs`, `MxGatewayEndpointConfig.cs`, `OpcUaHeartbeatConfig.cs`, `OpcUaDeadbandConfig.cs`); the XOR also locks the *secret* side (a rename of `Password` off the fragment list fails the test).
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests --filter ConfigSecretScrubber` — FAIL first on compile (`IsSecretName` private), then green the assertions.
3. Append to the `MergeSentinels` xmldoc (:121-126) the N9 limitation, so the by-index behavior is a documented contract rather than a surprise:
```
/// <b>Array limitation (arch-review R2 N9):</b> sentinel restoration inside arrays
/// pairs incoming[i] with stored[i] BY INDEX. A client that reorders an array of
/// credentialed objects while round-tripping sentinels grafts the wrong stored
/// secret into each slot — a silent misconfiguration (never a disclosure). Clients
/// must not reorder arrays that carry sentinels (the CLI round-trip does not);
/// key-based matching is a possible future refinement if a config type ever nests
/// credentialed arrays.
```
4. Run the filter — PASS; full `dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests` run.
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ManagementService/ConfigSecretScrubber.cs tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/ConfigSecretScrubberTests.cs && git commit -m "test(management): freeze scrubber fragment coverage over DataConnection config types + document array-merge limit (plan R2-07 T12)"`
---
## Dependencies on other plans
- **PLAN-R2-02 (Communication / site stream):** shares `SiteAlarmLiveCacheService.cs` — **Task 10 and R2-02's delta-coalescing task are mutually exclusive in time** (see the mutex rule at the top). The gRPC/site half of the live-alarm path (stream reconnect, `SubscribeSite` semantics) belongs to R2-02; Task 10 touches only the central service's liveness bookkeeping.
- **ManagementActor single-writer lane:** only this plan (Tasks 4, 6) touches `ManagementActor.cs` in round 2, but the initiative-wide rule stands — serialize against any other plan's ManagementActor task that may appear.
- **Round-1 documented deferrals stay put:** S1's client-supplied import-id idempotency remains plan-05 domain; S4 streaming multipart, P2 QueryDeployments/ExportBundle projections, and the UA6 fleet-wide grid a11y sweep remain in `docs/plans/2026-07-08-deferred-work-register.md`. No round-2 task may "helpfully" absorb them.
## Execution order
**P0 (security-correctness first):** Task 1 (N1) and Task 2 → 3 (N2) and Task 4 → 6 (N3, with Task 5 landing any time before 6) — these five are the Medium security findings. Then Task 7 (N4, the operator-facing Medium) → 8 → 9 on the AlarmSummary lane; Tasks 10, 11, 12 land any time (10 subject to the R2-02 mutex).
**Ordering constraints:** 3←2; 6←(4,5); 8←7; 9←8. Serialize {4, 6} (ManagementActor lane) and {7, 8, 9} (AlarmSummary.razor). Everything else is file-disjoint.
**Milestone verification:** at plan completion run `dotnet build ZB.MOM.WW.ScadaBridge.slnx` and `dotnet test ZB.MOM.WW.ScadaBridge.slnx` (targeted filters during tasks; the full sweep only here), then rebuild the docker cluster (`bash docker/deploy.sh`) for a smoke: a wrong-password CLI login 401s then 429s on the 6th try (throttle through Traefik with the real client IP), `secured-write list` under `multi-role` still works, and the Alarm Summary page live-updates then survives a site switch mid-poll.
---
## Findings Coverage
| Finding | Severity | Disposition |
|---|---|---|
| N1 — DebugStreamHub binds LDAP with no throttle; `Component-Security.md:208` falsely claims complete coverage | Medium | Task 1 (hub delegates to `ManagementAuthenticator.AuthenticateAsync`; doc corrected in the same task) |
| N2 — LoginThrottle keys collapse behind Traefik: no ForwardedHeaders handling anywhere | Medium | Tasks 2, 3 (trusted-proxy `UseForwardedHeaders` bound from config; docker/docker-env2/deploy config shipped; lockout-DoS trade-off + mitigation documented; throttle-test assumption noted) |
| N3 — Secured-write handlers ignore site scope while the spec says scoping is layered on at the LDAP-mapping level | Medium | Tasks 4, 5, 6 (enforce at submit/approve/reject via existing `EnforceSiteScope*` helpers; scoped list at the query level; `Component-Security.md:141` amended to match; frozen matrix verified — no role changes, no regeneration needed) |
| N4 — `RefreshAsync` applies fan-out results without re-checking the selected site (cross-site data after a switch) | Medium | Task 7 (one-line guard mirroring the live path's `:374` check + bUnit regression) |
| N5 — When live, a slower poll snapshot can regress fresher live state | Low | Task 8 (poll updates only `_notReporting` while `IsLive`; stale comment fixed) |
| N6 — `IsLive` is sticky: `HasPublished` never resets on aggregator death | Low | Task 10 (deathwatch → liveness reset + snapshot clear + self-heal restart; interface doc updated). **Mutex with PLAN-R2-02's coalescing task.** |
| N7 — Live-callback `InvokeAsync` lacks the house disposal guard | Low | Task 9 (`_disposed` set before teardown, checked before and inside the marshal — DebugView pattern) |
| N8 — `LoginThrottle.Prune` scans (and may sort) the whole map on every failed bind | Low | Task 11 (every-Nth-failure cadence + immediate over-cap sweep; semantics unchanged) |
| N9 — Scrubber array merge is by-index; fragment list has no lock-in against future config fields | Low | Task 12 (frozen XOR coverage test over the DataConnections config types; array-by-index limitation documented as contract) |
| *Round-1 residue* S1 — import idempotency on a client-supplied id | High (residual) | **Remains deferred** (round-1 documented deferral; plan-05 / Transport apply-path domain; tracked in the deferred-work register) |
| *Round-1 residue* S4 — streaming multipart bundle upload | Medium (residual) | **Remains deferred** (bounded by the round-1 `TransportGate` single-flight; deferred-work register) |
| *Round-1 residue* P2 — QueryDeployments/ExportBundle load-all internals | Medium (residual) | **Remains deferred** (bounded by fleet size; needs repo-level projections; deferred-work register) |
| *Round-1 residue* UA6 — fleet-wide sortable-grid a11y sweep | UA (residual) | **Remains deferred** (uniform low-severity UX debt; logged in the AlarmSummary component comment + deferred-work register) |
| *Round-1 residue* P1/UA5 — the one missed bind surface | Medium (residual) | **Owned here as N1** → Task 1 (no separate row-work) |
archreview/plans/PLAN-R2-07-ui-management-security.md.tasks.json
+87
View File
@@ -0,0 +1,87 @@
{
"planPath": "archreview/plans/PLAN-R2-07-ui-management-security.md",
"tasks": [
{
"id": 1,
"subject": "Task 1: Route the DebugStreamHub LDAP bind through ManagementAuthenticator (throttled) + fix the false doc claim",
"status": "pending",
"blockedBy": []
},
{
"id": 2,
"subject": "Task 2: ForwardedHeadersSetup — trusted-proxy client-IP resolution for the throttle keys",
"status": "pending",
"blockedBy": []
},
{
"id": 3,
"subject": "Task 3: Ship the ForwardedHeaders config to the Traefik topologies + document the lockout-DoS trade-off",
"status": "pending",
"blockedBy": [
2
]
},
{
"id": 4,
"subject": "Task 4: Enforce site scope on secured-write submit / approve / reject",
"status": "pending",
"blockedBy": []
},
{
"id": 5,
"subject": "Task 5: ISecuredWriteRepository — additive permitted-sites filter for scoped listing",
"status": "pending",
"blockedBy": []
},
{
"id": 6,
"subject": "Task 6: Scope-filter HandleListSecuredWrites + amend Component-Security.md:141 + matrix verification",
"status": "pending",
"blockedBy": [
4,
5
]
},
{
"id": 7,
"subject": "Task 7: AlarmSummary.RefreshAsync stale-site guard — never display cross-site data",
"status": "pending",
"blockedBy": []
},
{
"id": 8,
"subject": "Task 8: While live, the poll updates only _notReporting — never regresses live rows",
"status": "pending",
"blockedBy": [
7
]
},
{
"id": 9,
"subject": "Task 9: House disposal guard on the live callback",
"status": "pending",
"blockedBy": [
8
]
},
{
"id": 10,
"subject": "Task 10: Un-stick IsLive — deathwatch resets liveness when the aggregator terminates (MUTEX with PLAN-R2-02 SiteAlarmLiveCacheService task)",
"status": "pending",
"blockedBy": []
},
{
"id": 11,
"subject": "Task 11: Amortize LoginThrottle.Prune off the failure hot path",
"status": "pending",
"blockedBy": []
},
{
"id": 12,
"subject": "Task 12: Lock in the scrubber's fragment coverage + document the array-merge limitation",
"status": "pending",
"blockedBy": []
}
],
"lastUpdated": "2026-07-13T03:26:45Z"
}
archreview/plans/PLAN-R2-08-conventions-hygiene.md
+504
View File
@@ -0,0 +1,504 @@
# PLAN-R2-08 — Conventions & Hygiene (Round-2 Findings NF1–NF8)
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
**Goal:** Close the eight NEW findings from architecture review 08 round 2 (`archreview/08-conventions-tests-underdeveloped.md`, 2026-07-12) — rotate the live production Inbound API key sitting in `test.txt` at the repo root and purge the other plaintext credential captures (NF1, High), refresh the abandoned-and-now-wrong CHANGELOG (NF3), finish the options-validation tail (AuditLog's 6 sub-options + Host Node/Database/Logging + OperationTracking — NF4), re-consolidate deferral tracking into the canonical register (NF5), fix the stale Transport area comment (NF6), commit the orphaned MES plan doc + clean root clutter (NF7), and kill the duplicate/drifted Host options bindings (NF8). NF2 (failover envelope measurement) is **owned by PLAN-R2-01** — this plan only repairs the register row whose revisit trigger fired.
**Architecture:** No new runtime components, no schema changes, no wire-contract changes. The options work copies the shipped `OptionsValidatorBase<T>` + `AddValidatedOptions<T, TValidator>` pattern (`ZB.MOM.WW.Configuration`; see `AuditLog/ServiceCollectionExtensions.cs:81` and `KpiHistory/ServiceCollectionExtensions.cs:40`). The credential work is operational (dual-key rotation per `docs/requirements/Component-InboundAPI.md` §Key rotation, shipped by PLAN-06 T24) plus working-tree hygiene with a `.gitignore`/pre-commit guard. One discovery made while verifying NF4 is folded in as Task 8: the site-only `IOperationTrackingStore` appears to have **no DI registration anywhere in `src/`**, despite two comments claiming `AddSiteRuntime` registers it — Task 8 verifies and fixes that seam alongside the missing `OperationTrackingOptions` binding.
**Tech Stack:** C# / .NET 10, xUnit, `Microsoft.Extensions.Options` (`IValidateOptions<>` + `ValidateOnStart`), `ZB.MOM.WW.Configuration` (`OptionsValidatorBase<T>`, `AddValidatedOptions`), SQLite (site tracking store), ScadaBridge CLI (`security api-key` group), git/bash (hygiene tasks).
**Build:** `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Tests: `dotnet test ZB.MOM.WW.ScadaBridge.slnx` (all 30 test projects, incl. CLI.Tests).
**Baseline:** `main` @ `8c888f13` (2026-07-12). Re-verify cited line numbers before editing — they were checked at plan-authoring time against this commit.
## Parallelization
- **Task 1 FIRST** (NF1 is the P0; it needs the user/host, so start it immediately and let the wait overlap other tasks). Task 2 follows Task 1 (rotate before destroying the local record of what was exposed).
- **Mutually parallel (file-disjoint):** 1, 3, 4, 5, 7, 10, 12, 13.
- **Serial chains (shared files):** 5 → 6 (both edit `AuditLog/ServiceCollectionExtensions.cs`); 7 → 8 → 9 (all edit `Host/SiteServiceRegistration.cs`); 10 → 11 (delete the snapshot only after the register has absorbed its content).
- Tasks 1, 2, 11, and the deletion half of 13 are flagged **`needs-user`** — they require production-host access and/or destroy untracked (unrecoverable) files, so the executor must show the user what is being done and get confirmation.
---
### Task 1: Rotate the exposed wonder-app-vd03 Inbound API key (dual-key flow) — `needs-user`
**Classification:** NF1 — high-risk security hygiene; **`needs-user`** (production environment access; the executor runs the commands only with the user present/consenting, and cannot complete step 3's caller-cutover without the user)
**Estimated implement time:** ~5 min of active work (cutover/quarantine waits excluded)
**Parallelizable with:** 3, 4, 5, 7, 10, 12, 13
**Files:**
- Modify: `docs/deployment/deployment-records/2026-06-27-wonder-app-vd03.md` (append a dated rotation record — the only repo artifact this task produces)
**Context (verified):** `test.txt` (untracked, repo root) contains a **working production Inbound API key** for `http://wonder-app-vd03.zmr.zimmer.com:8085` — `sbk_eb5acc0b…` (full token on line 3) scoped at least to `MesMoveOut`. The key is live regardless of whether the file ever enters git history, so it must be **rotated**, not merely deleted. The zero-downtime dual-key procedure is already documented (`docs/requirements/Component-InboundAPI.md:75-86`, PLAN-06 T24): create → cutover → disable → quarantine → delete. This task is that procedure instantiated against wonder-app-vd03 as an executable checklist. Host facts (auto-memory `wonder-app-vd03-host-access`): the deployment runs with `DisableLogin=true`, so `:8085/management` is unauthenticated and the CLI works with just `--url`; host shell access (if needed) is servecli SSH port 2222.
**Checklist (run each step, record output):**
1. Enumerate keys and identify the exposed one (match the `eb5acc0b` key-id prefix from `test.txt`; note its **name, key-id, and scope list**):
```bash
scadabridge --url http://wonder-app-vd03.zmr.zimmer.com:8085 security api-key list
```
2. Create the replacement with the **same scope set** (copy the old key's method list verbatim from step 1 — scopes are copied at creation only; drift between the pair is operator error):
```bash
scadabridge --url http://wonder-app-vd03.zmr.zimmer.com:8085 security api-key create \
--name "<old-name>-rot-2026-07" --methods "<exact scope list from step 1>"
```
Capture the one-time `sbk_…` token from the output immediately (it is shown once). Do **not** write it to any file in this repo — hand it to the user directly in the conversation or via their chosen secret channel.
3. **USER ACTION — cutover:** the user delivers the new token to the calling system's owner (the MES/DELMIA-side integration that drives `MesMoveOut`) and confirms the caller has switched. Verify cutover from the audit log (the Actor field records the serving key per call): `scadabridge --url http://wonder-app-vd03.zmr.zimmer.com:8085 audit list --channel InboundApi --limit 20` (adjust flags to the real `audit` verbs) — recent `MesMoveOut` rows must show the **new** key id.
4. Disable (don't delete) the old key once cutover is confirmed:
```bash
scadabridge --url http://wonder-app-vd03.zmr.zimmer.com:8085 security api-key update --key-id <old-key-id> --enabled false
```
5. Schedule deletion after a quarantine window (user judgment; suggest ≥ 1 week):
```bash
scadabridge --url http://wonder-app-vd03.zmr.zimmer.com:8085 security api-key delete --key-id <old-key-id>
```
6. Append a dated "API-key rotation 2026-07" subsection to `docs/deployment/deployment-records/2026-06-27-wonder-app-vd03.md`: why (key exposed in a local scratch file, arch-review 08 round 2 NF1), old key id (id only — never the secret), new key name/id, cutover-confirmed date, disable date, planned delete date. Do NOT include any token material.
7. Verify (hygiene task — verification replaces a test): step 1 re-run shows the old key `Enabled=false` (or absent after step 5) and the new key present; `grep -c "sbk_" docs/deployment/deployment-records/2026-06-27-wonder-app-vd03.md` returns `0`.
8. Commit:
```bash
git add docs/deployment/deployment-records/2026-06-27-wonder-app-vd03.md
git commit -m "chore(hygiene): record wonder-app-vd03 inbound API-key rotation — exposed key retired via dual-key flow (plan R2-08 T1)"
```
If the user cannot reach wonder-app-vd03 in this session, record the checklist state (which steps are pending) in the deployment record with a `PENDING` marker and still commit — the rotation must not be silently forgotten. Task 2 may proceed regardless (deleting `test.txt` does not need the old secret; disable/delete key operations only need the key-id from `list`).
---
### Task 2: Delete `test.txt` and triage the root credential files — `needs-user`
**Classification:** NF1 — high-risk security hygiene; **`needs-user`** (deletes untracked files — unrecoverable; user must see and confirm each)
**Estimated implement time:** ~3 min
**Parallelizable with:** 3, 4, 5, 7, 10, 12, 13 (blocked by 1 — rotate before destroying the local record; see Task 1's closing note for the pending-rotation escape hatch)
**Files:**
- Delete (untracked): `test.txt`
- Delete (untracked+ignored, user-confirmed): `ldap_login.txt`, `sql_login.txt`, `sqllogin.txt`
**Context (verified by reading each file):**
- `test.txt` — the live `sbk_eb5acc0b…` production API key (Task 1's subject) plus an ad-hoc test note. Untracked and NOT gitignored — one `git add -A` (or the `pushit` skill, which stages everything) from entering history.
- `ldap_login.txt` — **live AD credentials**: `zimdom.zimmer.com:3268` bind with the `SVC-DNC` service-account password AND the user's personal domain password. Ignored via `.gitignore:45` (`*login*.txt`) so unstageable, but plaintext on disk.
- `sql_login.txt` — the wonder-app-vd03 SQL Server `wwadmin` password (the production ScadaBridge config DB). Ignored, plaintext.
- `sqllogin.txt` — a provisioning transcript that creates `wwadmin` **with sysadmin** using the same password. Ignored, plaintext.
None of the four are in git history (verified: untracked/ignored; `git log` shows no commits touching them), so this is disk cleanup, not history surgery.
**Steps:**
1. Show the user the four files' contents (Read each aloud in the conversation) and state plainly: they are untracked — deletion is unrecoverable. Get explicit confirmation per file. If the user wants any kept, move it OUTSIDE the repo (e.g. `~/secrets/`) instead of leaving it in the working tree.
2. On confirmation:
```bash
rm /Users/dohertj2/Desktop/ScadaBridge/test.txt
rm /Users/dohertj2/Desktop/ScadaBridge/ldap_login.txt /Users/dohertj2/Desktop/ScadaBridge/sql_login.txt /Users/dohertj2/Desktop/ScadaBridge/sqllogin.txt
```
3. **Advisory to relay to the user (do not action without them):** the SVC-DNC, personal-domain, and `wwadmin` passwords in those files were exposed on disk for ~7 weeks; recommend rotating them through the normal AD/SQL channels. The `wwadmin` password also appears in the on-host deployment; rotating it means updating `appsettings` connection strings on wonder-app-vd03 (`deploy/wonder-app-vd03/RUNBOOK.md` procedure).
4. Verify: `ls test.txt ldap_login.txt sql_login.txt sqllogin.txt 2>&1` — all four report "No such file". `git status --short` no longer lists `test.txt`.
5. No commit — the deletions touch only untracked files (Task 3 commits the guard that prevents recurrence).
---
### Task 3: Root secret-capture guards — `.gitignore` patterns + pre-commit secret scan
**Classification:** small (hygiene guard)
**Estimated implement time:** ~5 min
**Parallelizable with:** 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12
**Files:**
- Modify: `.gitignore` (currently 48 lines; the `*login*.txt` rule is at line 45)
- Create: `tools/hooks/pre-commit` (secret-scan hook) and `tools/hooks/README.md` (2-line install note)
**Steps:**
1. Extend `.gitignore` — after the existing `*login*.txt` block (line 45), add:
```gitignore
# Ad-hoc scratch/credential captures at the repo root — never commit
# (arch-review 08 round 2 NF1: a live API key sat in an untracked test.txt)
/test.txt
/*credentials*
/*secret*
# Claude tool worktrees (NF7)
.claire/
```
Keep patterns root-anchored (`/`) so legitimate `src/`/`tests/` files named e.g. `SecretsBlock.cs` are unaffected.
2. Create `tools/hooks/pre-commit` (mode 755) — a ~20-line POSIX-sh scan of the **staged diff** that blocks the commit when it introduces obvious credential material:
- an inbound API token: `sbk_[0-9a-f]{16,}_[A-Za-z0-9_-]{20,}`
- a plaintext password assignment in root-level `.txt`/`.md` additions: `^\+.*[Pp]assword[0-9]*\s*[:=]\s*\S`
Exit non-zero with a message naming the offending file/pattern and how to bypass (`git commit --no-verify`) for a knowing false positive. Scan only `git diff --cached -U0` output — never the whole tree (speed + no false hits on committed test fixtures).
3. `tools/hooks/README.md`: two lines — what the hook does, and the opt-in install command `git config core.hooksPath tools/hooks` (hooks are not tracked by git; this is a suggestion the user enables once, exactly the "pre-commit suggestion" the report asked for).
4. Verify (guard task — a real fail→pass exercise replaces a unit test):
```bash
git check-ignore -v test.txt .claire/ # both match the new rules
printf 'api key: sbk_%s_%s\n' "$(printf 'a%.0s' {1..32})" "$(printf 'b%.0s' {1..40})" > /tmp/nf1-probe.txt
cp /tmp/nf1-probe.txt probe-secret-scan.md && git add probe-secret-scan.md
sh tools/hooks/pre-commit; echo "hook exit: $?" # expect non-zero (blocked)
git reset probe-secret-scan.md && rm probe-secret-scan.md /tmp/nf1-probe.txt
sh tools/hooks/pre-commit; echo "hook exit: $?" # clean index → expect 0
```
5. Commit:
```bash
git add .gitignore tools/hooks
git commit -m "chore(hygiene): root secret-capture guards — .gitignore patterns + opt-in pre-commit secret scan (plan R2-08 T3)"
```
---
### Task 4: CHANGELOG.md — correct the false role claim and refresh to reality
**Classification:** trivial (documentation)
**Estimated implement time:** ~5 min
**Parallelizable with:** 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13
**Files:**
- Modify: `CHANGELOG.md` (last touched 2026-06-02, `b104760b`; the false claim is at lines 11-12)
**Steps:**
1. Fix the falsehood first — lines 11-12 assert "`Operator`/`Engineer` are unused by ScadaBridge", false since M7 shipped the secured-write flow. Replace the parenthetical with:
> (`Engineer` remains unused by ScadaBridge; `Operator` has since been adopted by the M7 two-person secured-write flow, together with the new `Verifier` role — see the M7 entry above)
2. Add new entries ABOVE the existing `[Unreleased]` content (keep-a-changelog style, newest first, still under `[Unreleased]` since nothing is version-tagged). Keep it honest and SHORT — 3 summary blocks, a few bullets each, each pointing at the authoritative record rather than duplicating it:
- **2026-07-08 → 2026-07-10 — Architecture-review hardening initiative (PLAN-01…PLAN-08).** One paragraph: 8 reviews → 8 fix plans, 191/192 tasks landed (options validation across 12+ components, ClusterClient contract-lock tests, secured-write TTL, login throttle, transport single-flight, S&F sweep bounding, deploy-readiness handshake, vestigial site-notification surface excised, deferred-work register established). Authoritative record: `archreview/plans/00-MASTER-TRACKER.md` + the per-plan docs.
- **M8 — Transport.** Encrypted bundle export/import extended to site/instance-scoped config with `BundleNameMap` reconciliation, per-line Myers diff, script trust gate at import. Post-initiative: aggregated live alarm stream + KPI hourly rollups (both 2026-07-10).
- **M7 — Native alarms + operational UX.** Native alarm mirroring (OPC UA A&C + MxGateway), Alarm Summary page, OPC UA browse/search/cert-trust, **two-person secured writes introducing the `Operator`/`Verifier` roles**, SMS notification adapter.
3. Add one closing line under the changelog intro: detailed per-change history lives in `git log`, `archreview/plans/00-MASTER-TRACKER.md`, and `docs/plans/` — this file records operator-facing/breaking changes only.
4. Verify: `grep -n "Engineer remains unused\|Verifier" CHANGELOG.md` hits; `grep -n "are unused by ScadaBridge" CHANGELOG.md` → no hit for the old combined claim; the two BREAKING sections (roles, inbound-API auth) are preserved untouched below the new entries.
5. Commit:
```bash
git add CHANGELOG.md
git commit -m "docs(changelog): refresh to reality — fix false Operator/Engineer claim, summarize arch-review initiative + M7/M8 (plan R2-08 T4)"
```
---
### Task 5: Options validation — AuditLog site sub-options (SiteWriter, SiteTelemetry, SiteRetention)
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** 1, 2, 3, 4, 7, 10, 11, 12, 13 (NOT 6 — both edit `AuditLog/ServiceCollectionExtensions.cs`)
**Files:**
- Create: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriterOptionsValidator.cs`, `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/SiteAuditTelemetryOptionsValidator.cs`, `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SiteAuditRetentionOptionsValidator.cs`
- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/ServiceCollectionExtensions.cs` (lines 100-108 — the three `AddOptions<T>().Bind(...)` calls)
- Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SqliteAuditWriterOptionsValidatorTests.cs`, `.../Site/SiteAuditTelemetryOptionsValidatorTests.cs`, `.../Site/SiteAuditRetentionOptionsValidatorTests.cs`
The pattern to copy is in the SAME file: `AuditLog/ServiceCollectionExtensions.cs:81` already does `services.AddValidatedOptions<AuditLogOptions, AuditLogOptionsValidator>(config, ConfigSectionName)` — the shared helper (scadaproj `ZB.MOM.WW.Configuration/ServiceCollectionExtensions.cs:28-41`) binds via `config.GetSection(sectionPath)` (preserving the file's documented partial-config-root rationale at lines 96-99), chains `.ValidateOnStart()`, and `TryAddEnumerable`s the validator. Validator + test shape: `AuditLog/Configuration/AuditLogOptionsValidator.cs` / `AuditLog.Tests/Configuration/AuditLogOptionsValidatorTests.cs`.
1. Write failing tests first (one `DefaultOptions_AreValid` + 2-3 rejection cases per validator, `new XValidator().Validate(Options.DefaultName, options)` shape). Rejection cases (fields verified against the option classes):
- `SqliteAuditWriterOptions`: `DatabasePath = ""`, `ChannelCapacity = 0`, `BatchSize = 0`, `FlushIntervalMs = 0`. **Do NOT validate `BacklogPollIntervalSeconds`** — non-positive has a documented fall-back-to-30s contract (register row 21 resolution + cadence tests); note this in a validator comment.
- `SiteAuditTelemetryOptions`: `BatchSize = 0`, `BusyIntervalSeconds = 0`, `IdleIntervalSeconds = 0`. Read `SiteAuditTelemetryActor`'s tick usage first; if idle < busy is genuinely nonsensical there, add the `IdleIntervalSeconds >= BusyIntervalSeconds` cross-check too.
- `SiteAuditRetentionOptions`: `RetentionDays = 0`, `PurgeInterval = TimeSpan.Zero`, `InitialDelay = TimeSpan.FromSeconds(-1)` (require `>= TimeSpan.Zero`). Leave `PurgeIntervalOverride` unvalidated — the class remarks mark it test-only, not config-set.
2. Run, expect FAIL (compile error on missing validator types counts):
```bash
dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter "FullyQualifiedName~SqliteAuditWriterOptionsValidator"
```
3. Implement the three validators (`OptionsValidatorBase<T>`, key-naming messages using the real section names from the SCE constants — `AuditLog:SiteWriter`, `AuditLog:SiteTelemetry`, `AuditLog:SiteRetention`). Convert the bindings at lines 100-108 to:
```csharp
services.AddValidatedOptions<SqliteAuditWriterOptions, SqliteAuditWriterOptionsValidator>(config, SiteWriterSectionName);
services.AddValidatedOptions<SiteAuditTelemetryOptions, SiteAuditTelemetryOptionsValidator>(config, SiteTelemetrySectionName);
services.AddValidatedOptions<SiteAuditRetentionOptions, SiteAuditRetentionOptionsValidator>(config, SiteAuditRetentionOptions.SectionName);
```
Keep (condense, don't delete) the surrounding why-comments — the partial-config-root note and the "inert on central" retention note still hold.
4. Run, expect PASS; then guard both composition roots (central also calls `AddAuditLog`, so defaults must validate clean there):
```bash
dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter "FullyQualifiedName~OptionsValidator"
dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~CompositionRoot"
```
5. Commit:
```bash
git add src/ZB.MOM.WW.ScadaBridge.AuditLog tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests
git commit -m "fix(options): eager startup validation for AuditLog site sub-options — SiteWriter, SiteTelemetry, SiteRetention (plan R2-08 T5, arch-review 08r2 NF4)"
```
---
### Task 6: Options validation — AuditLog central sub-options (PartitionMaintenance, Purge, Reconciliation)
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** 1, 2, 3, 4, 7, 10, 11, 12, 13 (blocked by 5 — same `ServiceCollectionExtensions.cs`)
**Files:**
- Create: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPartitionMaintenanceOptionsValidator.cs`, `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeOptionsValidator.cs`, `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/SiteAuditReconciliationOptionsValidator.cs`
- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/ServiceCollectionExtensions.cs` (lines 368-390 in `AddAuditLogCentralMaintenance` — re-check offsets after Task 5's edit)
- Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/AuditLogPartitionMaintenanceOptionsValidatorTests.cs`, `.../Central/AuditLogPurgeOptionsValidatorTests.cs`, `.../Central/SiteAuditReconciliationOptionsValidatorTests.cs`
Same recipe as Task 5. Rejection cases (fields verified):
- `AuditLogPartitionMaintenanceOptions` (`AuditLog:PartitionMaintenance`): `IntervalSeconds = 0`, `LookaheadMonths = 0`.
- `AuditLogPurgeOptions` (`AuditLog:Purge`): `IntervalHours = 0`, `ChannelPurgeBatchSizeConfigured = 0`, `MaintenanceCommandTimeoutMinutes = 0`. Leave `IntervalOverride` unvalidated (test-only per class remarks).
- `SiteAuditReconciliationOptions` (`AuditLog:Reconciliation`): `ReconciliationIntervalSeconds = 0`, `BatchSize = 0`, `StalledAfterNonDrainingCycles = 0`. Leave `ReconciliationIntervalOverride` unvalidated.
1. Failing tests → 2. run one filter, expect FAIL → 3. implement validators + convert the three `AddOptions<T>().Bind(...)` at :368-390 to `AddValidatedOptions` with the existing section constants (`PartitionMaintenanceSectionName`, `PurgeSectionName`, `ReconciliationSectionName`); **keep the long I1-review comment block at :372-384** — it explains WHY the two singleton options bind here, which is unchanged → 4. run, expect PASS:
```bash
dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter "FullyQualifiedName~OptionsValidator"
dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~CompositionRoot"
```
2. Commit:
```bash
git add src/ZB.MOM.WW.ScadaBridge.AuditLog tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests
git commit -m "fix(options): eager startup validation for AuditLog central sub-options — PartitionMaintenance, Purge, Reconciliation (plan R2-08 T6, arch-review 08r2 NF4)"
```
---
### Task 7: Options validation — Host NodeOptions / DatabaseOptions / LoggingOptions (empty NodeName fails fast)
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** 1, 2, 3, 4, 5, 6, 10, 11, 12, 13 (NOT 8/9 — all three edit `Host/SiteServiceRegistration.cs`)
**Files:**
- Create: `src/ZB.MOM.WW.ScadaBridge.Host/NodeOptionsValidator.cs`, `src/ZB.MOM.WW.ScadaBridge.Host/DatabaseOptionsValidator.cs`, `src/ZB.MOM.WW.ScadaBridge.Host/LoggingOptionsValidator.cs`
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs` (`BindSharedOptions`, lines 160/165/172 — the plain `services.Configure<>` calls for these three)
- Modify tests: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CompositionRootTests.cs` (in-memory config dicts at ~:111 and ~:350), `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/MetricsEndpointTests.cs` (~:43) — none currently sets `ScadaBridge:Node:NodeName`
- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/NodeOptionsValidatorTests.cs` (one file covering all three validators is fine)
This is the SourceNode failure mode the tracker logged for wonder-app-vd03: an empty `NodeOptions.NodeName` "normalises to a NULL SourceNode" (the appsettings `_nodeName` comment says so verbatim) — it must fail the host at boot, not silently degrade the audit trail. The Host csproj already references `ZB.MOM.WW.Configuration` transitively via components, but add the explicit `<PackageReference Include="ZB.MOM.WW.Configuration" />` (no Version — central package management) if `dotnet build` says otherwise.
1. Write failing validator tests. Rejection cases:
- `NodeOptions`: `NodeName = ""` (and whitespace) rejected — message must say what breaks ("SourceNode audit column stamps NULL"). `RemotingPort`/`GrpcPort`/`MetricsPort` outside `0..65535` rejected (**0 must stay VALID** — `CompositionRootTests` uses `RemotingPort = 0` for dynamic ports).
- `DatabaseOptions`: all three fields are nullable and role-dependent — reject only whitespace-non-null values (e.g. `ConfigurationDb = " "`); null stays valid.
- `LoggingOptions`: `MinimumLevel = "Chatty"` rejected. First read how `MinimumLevel` is consumed (Program.cs Serilog bootstrap) and validate against that parser's accepted set (Serilog: Verbose/Debug/Information/Warning/Error/Fatal, case-insensitive).
2. Run, expect FAIL:
```bash
dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~NodeOptionsValidator"
```
3. Implement the three validators (`OptionsValidatorBase<T>`, key prefix `ScadaBridge:Node` / `ScadaBridge:Database` / `ScadaBridge:Logging`). Convert `BindSharedOptions` lines 160/165/172 to the `AddOptions<T>().Bind(...).ValidateOnStart()` + `TryAddEnumerable` shape (exactly what the same method already does for `ClusterOptions` at :164 and `HealthMonitoringOptions` at :170 — mirror those, comments included).
4. Add `["ScadaBridge:Node:NodeName"] = "central-a"` (central fixtures) / `"node-a"` (site fixtures) to every in-memory Host-boot config dict — grep the test project for `ScadaBridge:Node:` to find them all; the production `appsettings.{Central,Site}.json` and all docker per-node configs already set NodeName (verified), so no config-file changes are needed.
5. Run, expect PASS:
```bash
dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~NodeOptionsValidator|FullyQualifiedName~CompositionRoot|FullyQualifiedName~MetricsEndpoint"
dotnet build ZB.MOM.WW.ScadaBridge.slnx
```
6. Commit:
```bash
git add src/ZB.MOM.WW.ScadaBridge.Host tests/ZB.MOM.WW.ScadaBridge.Host.Tests
git commit -m "fix(options): validate Host Node/Database/Logging options at startup — empty NodeName fails fast instead of NULLing SourceNode (plan R2-08 T7, arch-review 08r2 NF4)"
```
---
### Task 8: OperationTrackingOptions binding + the missing site `IOperationTrackingStore` registration (verify-then-fix)
**Classification:** standard — carries a **plan-authoring discovery** that may be a live functional gap, so investigate before coding
**Estimated implement time:** ~5 min (after the ~5 min verification below confirms the gap)
**Parallelizable with:** 1, 2, 3, 4, 5, 6, 10, 11, 12, 13 (blocked by 7 — same `Host/SiteServiceRegistration.cs`)
**Files:**
- Create: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingOptionsValidator.cs`
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs` (`AddSiteRuntime(connectionString)`, lines 36-82 — add the store registration)
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs` (site-options block ~:134-152 — add the binding)
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Tracking/OperationTrackingOptionsValidatorTests.cs`; extend `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CompositionRootTests.cs` (site-role fixture)
**Discovery (verified at plan time, main @ 8c888f13):** NF4 notes `OperationTrackingOptions` has no binding site. Verifying that surfaced something bigger: **no code anywhere in `src/` registers `IOperationTrackingStore` in DI** — yet `Host/Actors/AkkaHostedService.cs:1044` says "The store is registered by AddSiteRuntime on site composition roots" and `AuditLog/ServiceCollectionExtensions.cs:158-159` says "the operational tracking store is SITE-ONLY (registered by ZB.MOM.WW.ScadaBridge.SiteRuntime)". `AddSiteRuntime` (SiteRuntime SCE :36-82) registers no such thing. Every consumer resolves it with `GetService` (null-tolerant): `AkkaHostedService.cs:1048` (cached-drain scheduler never armed when null), `:1097-1099` (`PullSiteCalls` reconciliation seam never wired), `ScriptExecutionActor.cs:198` (`Tracking.Status` degraded), `AuditLog SCE :166` (forwarder degrades to audit-only). If the registration is truly absent, site-local cached-call tracking has been running in its degraded mode.
1. **Verify first** (do not trust this plan's grep over live behavior): `grep -rn "IOperationTrackingStore" src --include="*.cs"` and confirm no `AddSingleton`/descriptor registration exists. If one exists somewhere this plan missed — this task shrinks to the options binding + validator only; update the task notes accordingly.
2. Write the failing composition-root lock — in the Host.Tests **site-role** fixture:
```csharp
[Fact]
public void Site_Resolves_IOperationTrackingStore()
{
// Site Call Audit's site-local source of truth. AkkaHostedService and
// ScriptRuntimeContext resolve this with GetService and silently degrade
// when absent (audit-only telemetry, no cached-drain, no PullSiteCalls) —
// so its presence on the SITE composition root must be locked here.
var store = _host.Services.GetService<IOperationTrackingStore>();
Assert.NotNull(store);
}
```
Plus validator rejection tests: `ConnectionString = ""`, `RetentionDays = 0` (SiteRuntime.Tests; SiteRuntime already references `ZB.MOM.WW.Configuration` — `SiteRuntimeOptionsValidator` lives there).
3. Run, expect FAIL:
```bash
dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~Site_Resolves_IOperationTrackingStore"
```
4. Implement:
- `OperationTrackingOptionsValidator` (`ScadaBridge:OperationTracking` key prefix; `ConnectionString` non-empty, `RetentionDays > 0`).
- In `AddSiteRuntime(connectionString)`: `services.AddSingleton<IOperationTrackingStore, OperationTrackingStore>();` (ctor takes `IOptions<OperationTrackingOptions>` + `ILogger` — verified at `OperationTrackingStore.cs:58-60`). Note in a comment that this makes the two existing "registered by AddSiteRuntime" claims true again.
- In `Host/SiteServiceRegistration.cs` site-options block (~:134-152): `AddOptions<OperationTrackingOptions>().Bind(config.GetSection("ScadaBridge:OperationTracking")).ValidateOnStart()` + `TryAddEnumerable` the validator — same shape as its `SiteRuntimeOptions` neighbor.
- Check `OperationTrackingStore`'s construction path: if the ctor eagerly opens/creates the SQLite file, point the Host.Tests site fixture at a temp path (`["ScadaBridge:OperationTracking:ConnectionString"] = $"Data Source={Path.Combine(Path.GetTempPath(), ...)}"`) so test runs don't litter `site-tracking.db` in the test cwd.
5. Run, expect PASS — and because this un-degrades real behavior, run the consumer suites too:
```bash
dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~CompositionRoot"
dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~Tracking"
dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter "FullyQualifiedName~Telemetry"
```
6. If the gap was real (step 1 confirmed), flag it in the commit body — it is a functional fix riding a hygiene plan, and the user should know:
```bash
git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests tests/ZB.MOM.WW.ScadaBridge.Host.Tests
git commit -m "fix(options): bind + validate OperationTrackingOptions; register the site IOperationTrackingStore that two comments claimed AddSiteRuntime provides (plan R2-08 T8, arch-review 08r2 NF4)"
```
---
### Task 9: NF8 — canonicalize the Communication/DataConnection section names, drop the duplicate Host bindings
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** 1, 2, 3, 4, 5, 6, 10, 11, 12, 13 (blocked by 8 — same `Host/SiteServiceRegistration.cs`)
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/ServiceCollectionExtensions.cs` (line 16), `src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptionsValidator.cs` (message prefixes)
- Modify: `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/ServiceCollectionExtensions.cs` (lines 18, 27-31), `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/DataConnectionOptionsValidator.cs` (message prefixes, if any name the section)
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs` (delete lines 140, 166, 171 — re-check offsets after Tasks 7/8)
- Test: extend `tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/CommunicationOptionsValidatorTests.cs` (or a new `CommunicationOptionsBindingTests.cs`) and the DCL equivalent
**Context (verified — the drift NF8 warned about has already happened):** every real appsettings file nests these sections under `ScadaBridge:` (`Host/appsettings.Site.json:2` root, `Communication` at :37, `DataConnection` at :28; same in all docker per-node configs). But `AddCommunication()` binds `BindConfiguration("Communication")` (Communication SCE :16) and `AddDataConnectionLayer()` binds `"DataConnectionLayer"` (DCL SCE :18) — **sections that exist nowhere**. The options only get real values because the Host's duplicate `Configure<>` calls (`SiteServiceRegistration.cs:140/:166`) bind the true `ScadaBridge:*` sections. So "delete the Host copy" alone (the report's shorthand) would silently revert both components to defaults — the fix is: canonicalize the SCE section names to the real ones, THEN delete the Host duplicates. `NotificationOptions` is the clean case: its SCE already binds `"ScadaBridge:Notification"` (NotificationService SCE :23), so the Host duplicate at :171 is purely redundant — but `AddNotificationService()` is central-only (site path comment at `SiteServiceRegistration.cs:37-41`), so before deleting :171, grep for any site-path consumer of `IOptions<NotificationOptions>`; if none exists (expected — sites don't deliver), delete it.
1. Write the failing binding test (Communication.Tests):
```csharp
[Fact]
public void AddCommunication_BindsTheScadaBridgeCommunicationSection()
{
// The real appsettings nest this section under "ScadaBridge:" — binding a
// root-level "Communication" section reads config that no deployment writes
// (arch-review 08 round 2 NF8: the Host duplicate was silently supplying the
// real values). AddCommunication alone must bind the canonical section.
var config = new ConfigurationBuilder().AddInMemoryCollection(new Dictionary<string, string?>
{
["ScadaBridge:Communication:DeploymentTimeout"] = "00:01:23",
}).Build();
var services = new ServiceCollection();
services.AddSingleton<IConfiguration>(config);
services.AddCommunication();
using var sp = services.BuildServiceProvider();
Assert.Equal(TimeSpan.FromSeconds(83), sp.GetRequiredService<IOptions<CommunicationOptions>>().Value.DeploymentTimeout);
}
```
Mirror for DCL (`ScadaBridge:DataConnection:ReconnectInterval`). Run, expect FAIL (the SCE binds the wrong section, so the value stays at default).
2. Implement:
- Communication SCE :16 → `.BindConfiguration("ScadaBridge:Communication")`; update every `Communication:<Field>` message prefix in `CommunicationOptionsValidator` to `ScadaBridge:Communication:<Field>` (the messages exist to name the real config key).
- DCL SCE :18 → `.BindConfiguration("ScadaBridge:DataConnection")`; same prefix sweep in its validator. For `OpcUaGlobalOptions`/`MxGatewayGlobalOptions` (:27-31, bare `"OpcUa"`/`"MxGateway"`): grep repo-wide (appsettings + tests + docs) for those bare sections first; no hit is expected (verified: none in any appsettings) — then canonicalize to `"ScadaBridge:OpcUa"`/`"ScadaBridge:MxGateway"` and note the sections in the two options classes' XML docs.
- Delete the three Host duplicates (`SiteServiceRegistration.cs:140` DataConnection, `:166` Communication, `:171` Notification — after the consumer grep above).
3. Run, expect PASS — plus both composition roots (this is exactly the change that could silently zero a timeout if the section name is wrong):
```bash
dotnet test tests/ZB.MOM.WW.ScadaBridge.Communication.Tests --filter "FullyQualifiedName~Options"
dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~Options"
dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~CompositionRoot"
dotnet build ZB.MOM.WW.ScadaBridge.slnx
```
4. Live-config sanity (cheap, catches a typo'd section name better than any unit test): `bash docker/deploy.sh` is NOT required for this plan, but at minimum re-read one docker site config and one central config and confirm every key under `ScadaBridge:Communication` / `ScadaBridge:DataConnection` corresponds to a `CommunicationOptions`/`DataConnectionOptions` property.
5. Commit:
```bash
git add src/ZB.MOM.WW.ScadaBridge.Communication src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs tests/ZB.MOM.WW.ScadaBridge.Communication.Tests tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests
git commit -m "fix(options): canonicalize Communication/DataConnection config sections to ScadaBridge:*; drop the duplicate Host bindings that were masking the drift (plan R2-08 T9, arch-review 08r2 NF8)"
```
---
### Task 10: Re-consolidate deferral tracking into the canonical register
**Classification:** trivial (documentation)
**Estimated implement time:** ~5 min
**Parallelizable with:** 1, 2, 3, 4, 5, 6, 7, 8, 9, 12, 13
**Files:**
- Modify: `docs/plans/2026-07-08-deferred-work-register.md` (Fix-now table lines 7-16; "New deferrals" table lines 45-51)
- Modify: `archreview/plans/00-MASTER-TRACKER.md` (registry section, lines 137-147)
Four edits, all specified by NF5 + the register's own rules:
1. **Prune the landed Fix-now rows** (the register's stated rule: "removed from this table when that plan's task lands" — all 7 landed). Replace the Fix-now table body with a single line: `All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round 2, 2026-07-12) — rows removed per the rule above; see the round-2 report §1 for the per-item evidence.`
2. **Add the two tracker-only live items as first-class register rows** (a new `## Deferred — operational risk (from the initiative tracker, folded in 2026-07-12)` section, same column shape as the Deferred table):
- **SBR oldest-crash total-outage gap** — 2-node `keep-oldest` downs the partition without the oldest, so a hard crash of the ACTIVE (oldest) central node makes the standby self-down (~10s) → total central outage until restart; only a younger-node crash fails over. Where noted: `archreview/plans/00-MASTER-TRACKER.md:143` + auto-memory. Rationale: remedy is a production SBR topology/strategy decision (keep-majority + 3rd lighthouse node, static-quorum, or an accepted-risk note) — **owner: user decision**, not silently changeable. Revisit trigger: before the next production deployment that adds a central node, or the first real active-node crash.
- **`deploy/wonder-app-vd03/` overlay edits unapplied** — `appsettings.Central.json` needs `AllowSingleNodeCluster: true` + phantom-seed removal + `NodeName: central-a`; `install.ps1` needs `sc.exe failure` recovery actions. The artifact directory is intentionally untracked (production config out of source control), so the repo cannot ship the fix. Where noted: `00-MASTER-TRACKER.md:147` (PLAN-01 T16/T20/T23). Rationale: needs on-host access; without `NodeName` that deployment's audit rows stamp NULL `SourceNode` (partially mitigated once Task 7 lands — the host will now **fail at boot** with a key-naming error instead of silently NULLing, so applying the overlay becomes mandatory at next upgrade). Owner: whoever maintains the host (user). Revisit trigger: next wonder-app-vd03 deployment/upgrade — **the Task 7 validator makes this row unskippable then**.
3. **Fix the fired revisit trigger** — the "Failover-timing + broader perf envelope" row (line 51): its trigger "PLAN-01 rig landing" fired on 2026-07-08 (`tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/TwoNodeClusterFixture.cs`), before the placeholder even shipped, and nobody noticed (NF2). Rewrite the row: status → **trigger fired 2026-07-08**; owner → **PLAN-R2-01** (the round-2 cluster/host/failover plan owns wiring `FailoverTimingTests` to the fixture or documenting the true blocker — reference its failover-envelope measurement task by number once that plan file exists; if PLAN-R2-01 is not yet written when this task executes, cite it as `archreview/plans/PLAN-R2-01-*.md (failover-envelope measurement task)`).
4. **Subordinate the tracker's registry** — in `00-MASTER-TRACKER.md`, immediately under the `## Deferred / Won't-Fix Registry (initiative-level)` heading (line 137), add: `> 2026-07-12 (plan R2-08 T10): the two live items below (SBR oldest-crash gap, wonder-app-vd03 overlay) now have canonical rows in docs/plans/2026-07-08-deferred-work-register.md — that register is the single tracking place; the subsections below remain as the historical narrative.` Do not delete the narrative subsections (they carry the full evidence).
5. Verify: `grep -n "SBR\|wonder-app-vd03" docs/plans/2026-07-08-deferred-work-register.md` → both new rows present; `grep -c "PLAN-08 Task 1 |" docs/plans/2026-07-08-deferred-work-register.md` → 0 (Fix-now rows gone); `grep -n "PLAN-R2-01" docs/plans/2026-07-08-deferred-work-register.md` → the repointed trigger row; `grep -n "single tracking place" archreview/plans/00-MASTER-TRACKER.md` → the subordination note.
6. Commit:
```bash
git add docs/plans/2026-07-08-deferred-work-register.md archreview/plans/00-MASTER-TRACKER.md
git commit -m "docs(plans): re-consolidate deferral tracking — SBR gap + wonder-app overlay rows into the register, prune landed Fix-now rows, repoint the fired perf-envelope trigger at PLAN-R2-01 (plan R2-08 T10, arch-review 08r2 NF5)"
```
---
### Task 11: Delete the drifted root `deferred.md` snapshot — `needs-user`
**Classification:** trivial (hygiene); **`needs-user`** (deletes an untracked file — unrecoverable)
**Estimated implement time:** ~2 min
**Parallelizable with:** 1, 2, 3, 4, 5, 6, 7, 8, 9, 12, 13 (blocked by 10 — the register must absorb/verify everything first)
**Files:**
- Delete (untracked): `deferred.md` (repo root)
`deferred.md` is a 2026-07-10 snapshot of the register that has already drifted (it counts "12 open deferrals" vs the register's ~10, restructures the sections, and its two ✅-SHIPPED rows duplicate the register's Resolved table). Duplicated state is how the trigger in NF2 got missed — it goes.
1. Diff it against the register for the user's benefit and confirm **nothing in `deferred.md` is absent from the post-Task-10 register** (the only unique content at plan-authoring time was cosmetic regrouping + the shipped-row prose, both already in the register's Resolved section). If the executor finds a genuinely unique fact, fold it into the register first (amend Task 10's commit).
2. Show the user the file (it is untracked — deletion is unrecoverable) and get confirmation.
3. `rm /Users/dohertj2/Desktop/ScadaBridge/deferred.md`
4. Verify: `ls deferred.md 2>&1` → no such file; `git status --short` no longer lists it. No commit (untracked file).
---
### Task 12: Fix the stale Transport area comment in `EntitySerializer.FromBundleContent`
**Classification:** trivial (comment drift)
**Estimated implement time:** ~3 min
**Parallelizable with:** 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/EntitySerializer.cs` (lines 553-554)
`:553-554` still says "Areas don't travel (see export TODO), so AreaId stays null" — but areas DO travel by name since PLAN-05 (export resolves `AreaId` → `AreaNameById` at `:265-273`; the importer resolves-or-creates), and no TODO exists anymore. The `AreaId = null` at `:572` is still intentionally correct for this helper — `FromBundleContent` is off the importer's path (BundleImporter walks the raw DTO) and has no target-DB area-name→id map, exactly like the placeholder `DataConnectionId = 0` the same comment block already explains for connection bindings.
1. Replace the stale sentence with the truth, riding the existing explanation:
```csharp
// Instances: resolve template + site by name/identifier. Areas DO travel —
// by NAME on InstanceDto.AreaName (export resolves AreaId→AreaNameById; the
// importer resolves-or-creates under the target site). This helper still
// materializes AreaId = null for the same reason the connection-binding FK
// below is a placeholder: FromBundleContent is NOT on the importer's path
// (BundleImporter walks the raw DTO) and has no target-DB name→id map to
// resolve against. Round-trip fidelity for the area lives on the DTO, not
// on this materialized entity. Connection bindings keep their ...
```
(Splice so the existing binding explanation at :555-562 continues unduplicated.)
2. Verify (comment task — verification replaces a test):
```bash
grep -n "export TODO" src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/EntitySerializer.cs # expect: no hits
grep -c "Areas DO travel" src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/EntitySerializer.cs # expect: 1
dotnet build src/ZB.MOM.WW.ScadaBridge.Transport
```
3. Commit:
```bash
git add src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/EntitySerializer.cs
git commit -m "chore(hygiene): fix stale area-export comment in EntitySerializer.FromBundleContent — areas travel by name since PLAN-05 (plan R2-08 T12, arch-review 08r2 NF6)"
```
---
### Task 13: Commit the orphaned MES plan doc; remove the generated root reports — `needs-user` (deletions only)
**Classification:** trivial (hygiene); the two report deletions are **`needs-user`** (untracked — unrecoverable)
**Estimated implement time:** ~3 min
**Parallelizable with:** 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12
**Files:**
- Commit (currently untracked): `docs/plans/2026-06-30-mes-alarm-status-api.md`
- Delete (untracked, user-confirmed): `ScadaBridge-docs-fixed.md`, `ScadaBridge-docs-issues.md` (root)
- (`.claire/` is handled by Task 3's `.gitignore` entry — do not delete the directory; it holds tool worktrees)
1. The MES plan doc already carries the right header (`**Status:** Draft plan — NOT yet executed`), so it needs no edits — the repo's docs-as-spec convention says plan docs are committed decision records, and untracked it is invisible to every other session and one `git clean` from gone:
```bash
git add docs/plans/2026-06-30-mes-alarm-status-api.md
git commit -m "docs(plans): commit the 2026-06-30 MES alarm-status API draft plan — untracked decision record (plan R2-08 T13, arch-review 08r2 NF7)"
```
2. `ScadaBridge-docs-fixed.md` (4 KB) / `ScadaBridge-docs-issues.md` (166 KB) are generated CommentChecker-style reports from 2026-07-10, already consumed. Show the user their first ~20 lines each, state they are untracked/unrecoverable, and on confirmation:
```bash
rm /Users/dohertj2/Desktop/ScadaBridge/ScadaBridge-docs-fixed.md /Users/dohertj2/Desktop/ScadaBridge/ScadaBridge-docs-issues.md
```
If the user wants them kept, move them to the session scratchpad or `~/` — not the repo root.
3. Verify: `git status --short` shows neither report nor the plan doc as untracked; `git log --oneline -1 -- docs/plans/2026-06-30-mes-alarm-status-api.md` shows the new commit.
---
## Dependencies on other plans
- **PLAN-R2-01 (cluster/host/failover round 2)** owns NF2 in full: wiring `tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Failover/FailoverTimingTests.cs:26-28` (whose skip reason is now stale) to the `TwoNodeClusterFixture` rig — or updating the skip text + register row with the true blocker. This plan's only NF2 touchpoint is Task 10 step 3, which repoints the register row's fired trigger at that plan's measurement task. **No file overlap** with PLAN-R2-01 is expected except `00-MASTER-TRACKER.md` (Task 10 adds one blockquote line) — coordinate merge order if both plans run concurrently.
- No other round-2 plan shares files with this one at plan-authoring time; Tasks 7-9 all edit `Host/SiteServiceRegistration.cs`, which round-1 plans also touched — they are merged, so only the intra-plan 7→8→9 serialization matters.
## Execution order
1. **Task 1 first** (P0; starts the user-dependent rotation clock), Task 2 after it.
2. **Tasks 3, 4, 5, 7, 10, 12, 13 in parallel** (disjoint files).
3. **Task 6 after 5; Tasks 8 then 9 after 7; Task 11 after 10.**
4. Finish with `dotnet build ZB.MOM.WW.ScadaBridge.slnx && dotnet test ZB.MOM.WW.ScadaBridge.slnx` — expect all 30 test projects green (the known environmental exceptions: the 8 Playwright E2E that need a live docker cluster, and the pre-existing `SiteConnectionUpGauge_*` gRPC gauge flake).
## Findings Coverage
| Report finding (08-conventions-tests-underdeveloped.md, round 2) | Severity | Coverage |
|---|---|---|
| NF1 — live production Inbound API key in `test.txt` at the repo root (plus, found on triage: live AD/SQL credentials in the gitignored `ldap_login.txt`/`sql_login.txt`/`sqllogin.txt`) | High | Tasks 1 (rotate via the shipped dual-key flow, `needs-user`), 2 (delete the credential files, `needs-user`), 3 (`.gitignore` + pre-commit guard) |
| NF2 — failover envelope unmeasured; `FailoverTimingTests` skip precondition already satisfied | Medium | **Owned by PLAN-R2-01** (rig wiring / measurement). This plan: Task 10 step 3 repairs the register row whose "PLAN-01 rig landing" trigger fired unnoticed, repointing it at PLAN-R2-01's measurement task |
| NF3 — CHANGELOG.md abandoned and factually wrong (`Operator`/`Engineer` claim false since M7) | Medium | Task 4 |
| NF4 — options-validation residue: 6 AuditLog sub-options, Host Node/Database/Logging, unbound `OperationTrackingOptions` | Low | Tasks 5, 6 (AuditLog site + central sub-options), 7 (Host options; empty `NodeName` fails fast), 8 (OperationTracking binding + the discovered missing site `IOperationTrackingStore` registration) |
| NF5 — deferral tracking re-scattered into three places (tracker registry, drifted root `deferred.md`, stale Fix-now rows) | Low | Tasks 10 (fold SBR gap + wonder-app overlay into the register, prune Fix-now, fix fired trigger, subordinate the tracker), 11 (delete root `deferred.md`, `needs-user`) |
| NF6 — stale `EntitySerializer.cs:553-554` comment contradicts the shipped area export | Low | Task 12 |
| NF7 — untracked residue: MES draft plan doc, root generated reports, `.claire/` | Low | Task 13 (commit the plan doc, remove the reports `needs-user`); Task 3 gitignores `.claire/`; `test.txt` itself is NF1/Task 2 |
| NF8 — duplicate `DataConnectionOptions`/`CommunicationOptions`/`NotificationOptions` bindings in the Host (and, verified at plan time: the owning SCEs bind section names that exist in NO appsettings — the Host duplicates were the live bindings) | Low | Task 9 (canonicalize SCE section names to `ScadaBridge:*`, then delete the Host duplicates, with a binding-test lock) |
archreview/plans/PLAN-R2-08-conventions-hygiene.md.tasks.json
+84
View File
@@ -0,0 +1,84 @@
{
"planPath": "archreview/plans/PLAN-R2-08-conventions-hygiene.md",
"tasks": [
{
"id": 1,
"subject": "Task 1: Rotate the exposed wonder-app-vd03 Inbound API key (dual-key flow) — needs-user",
"status": "pending",
"blockedBy": []
},
{
"id": 2,
"subject": "Task 2: Delete test.txt and triage the root credential files — needs-user",
"status": "pending",
"blockedBy": [1]
},
{
"id": 3,
"subject": "Task 3: Root secret-capture guards — .gitignore patterns + pre-commit secret scan",
"status": "pending",
"blockedBy": []
},
{
"id": 4,
"subject": "Task 4: CHANGELOG.md — correct the false role claim and refresh to reality",
"status": "pending",
"blockedBy": []
},
{
"id": 5,
"subject": "Task 5: Options validation — AuditLog site sub-options (SiteWriter, SiteTelemetry, SiteRetention)",
"status": "pending",
"blockedBy": []
},
{
"id": 6,
"subject": "Task 6: Options validation — AuditLog central sub-options (PartitionMaintenance, Purge, Reconciliation)",
"status": "pending",
"blockedBy": [5]
},
{
"id": 7,
"subject": "Task 7: Options validation — Host NodeOptions / DatabaseOptions / LoggingOptions (empty NodeName fails fast)",
"status": "pending",
"blockedBy": []
},
{
"id": 8,
"subject": "Task 8: OperationTrackingOptions binding + the missing site IOperationTrackingStore registration (verify-then-fix)",
"status": "pending",
"blockedBy": [7]
},
{
"id": 9,
"subject": "Task 9: NF8 — canonicalize the Communication/DataConnection section names, drop the duplicate Host bindings",
"status": "pending",
"blockedBy": [8]
},
{
"id": 10,
"subject": "Task 10: Re-consolidate deferral tracking into the canonical register",
"status": "pending",
"blockedBy": []
},
{
"id": 11,
"subject": "Task 11: Delete the drifted root deferred.md snapshot — needs-user",
"status": "pending",
"blockedBy": [10]
},
{
"id": 12,
"subject": "Task 12: Fix the stale Transport area comment in EntitySerializer.FromBundleContent",
"status": "pending",
"blockedBy": []
},
{
"id": 13,
"subject": "Task 13: Commit the orphaned MES plan doc; remove the generated root reports — needs-user (deletions only)",
"status": "pending",
"blockedBy": []
}
],
"lastUpdated": "2026-07-13T03:30:22Z"
}