diff --git a/archreview/00-OVERALL.md b/archreview/00-OVERALL.md new file mode 100644 index 00000000..926ebd9d --- /dev/null +++ b/archreview/00-OVERALL.md @@ -0,0 +1,122 @@ +# ScadaBridge Deep Architecture Review — Overall Report + +**Date:** 2026-07-08 +**Scope:** Full system — all 27 components, docker/deploy topology, tests, and design docs +**Method:** 8 parallel domain reviews, each reading the component design docs first and then the actual source (findings are grounded in `file:line` citations in the domain reports). Dimensions: **stability, performance, conventions, underdeveloped areas.** + +## Report Index + +| # | Report | Domain | Verdict (one line) | +|---|--------|--------|--------------------| +| 01 | [Cluster, Host & Failover](01-cluster-host-failover.md) | ClusterInfrastructure, Host, HealthMonitoring, docker/Traefik/deploy | High micro-level polish undone by one critical foundation crack: the split-brain resolver is configured but never enabled — automatic failover does not work for hard crashes. | +| 02 | [Communication & Store-and-Forward](02-communication-store-and-forward.md) | ClusterClient, gRPC streaming, S&F engine | High code quality in-the-small; mid maturity as a distributed system — serious lifecycle/topology defects the test suite is structurally blind to. | +| 03 | [Site Runtime & DCL](03-site-runtime-dcl.md) | SiteRuntime, DataConnectionLayer, SiteEventLogging | Mature and unusually carefully engineered; residual risk in adapter reconnect lifecycle, script containment, and deployment honesty. | +| 04 | [Data & Audit Backbone](04-data-audit-backbone.md) | ConfigurationDatabase, AuditLog, NotificationOutbox, SiteCallAudit, KpiHistory | Genuinely mature data layer with scale-dependent operational time bombs that will surface within months of production volume. | +| 05 | [Templates, Deployment & Transport](05-templates-deployment-transport.md) | TemplateEngine, DeploymentManager, Transport, ScriptAnalysis | Failure-path engineering is production-grade, but the Transport import apply path and newest template features contain multiple silent-data-loss and staleness defects. | +| 06 | [Edge Integrations](06-edge-integrations.md) | InboundAPI, ExternalSystemGateway, NotificationService, DelmiaNotifier | Mature, heavily-hardened edge layer; risk concentrated in one silent cache-coherence bug and acknowledged ESG spec gaps. | +| 07 | [UI, Management & Security](07-ui-management-security.md) | CentralUI, ManagementService, CLI, Security | Mature, carefully engineered surface undermined by sharp asymmetries: an auth-disabled deploy artifact, ignored deploy scoping, inconsistent secret elision. | +| 08 | [Conventions, Tests & Underdeveloped Areas](08-conventions-tests-underdeveloped.md) | Cross-cutting | Unusually disciplined codebase — the documented conventions are actually followed; gaps are narrow, specific, and mostly consciously logged. | + +## Severity Tally (tag occurrences per report) + +| Report | Critical | High | Medium | Low | +|--------|---------:|-----:|-------:|----:| +| 01 Cluster/Host/Failover | 1 | 5 | 13 | 9 | +| 02 Communication/S&F | 2 | 6 | 11 | 11 | +| 03 Site Runtime/DCL | 0 | 4 | 6 | 11 | +| 04 Data/Audit Backbone | 0 | 6 | 11 | 6 | +| 05 Templates/Deploy/Transport | 2 | 13 | 15 | 5 | +| 06 Edge Integrations | 0 | 4 | 11 | 11 | +| 07 UI/Management/Security | 2 | 8 | 12 | 5 | +| 08 Conventions/Tests | 0 | 4 | 6 | 10 | +| **Total (approx.)** | **7** | **50** | **85** | **68** | + +*(Counts are raw tag occurrences; a few findings are cited in more than one report — e.g. the `DisableLogin` deploy artifact appears in both 01 and 07.)* + +## Overall Verdict + +ScadaBridge is an unusually disciplined codebase at the micro level. The conventions documented in CLAUDE.md are **actually followed** — verified, not assumed: zero `DateTime.Now` in `src/`, persistence-ignorant Commons entities, a clean project-reference graph with no site→central contamination, correlation IDs on request/response messages, additive message contracts, idempotent ingest keyed on `EventId`/`NotificationId`, timing-oracle-free API-key auth, and only two real TODOs in ~190k LOC. Test volume is real (~5,050 behavioral tests across 30 projects — Akka TestKit, bUnit, Playwright, MSSQL-backed integration tests). + +The system's risk is **not** diffuse sloppiness. It concentrates in exactly the four places a single-process test suite cannot see: + +1. **Real multi-node failover** — the paths that only execute when a node hard-crashes, a partition forms, or the standby takes over. +2. **Cache coherence across process/node boundaries** — compiled-script and staleness caches that are only invalidated on the node that made the change. +3. **Cross-environment data movement** — the Transport bundle import apply path, where DTOs, spec, and apply code drifted apart. +4. **Production-scale data volume** — purge/retention/KPI paths tested only at small scale. + +**Bottom line: production-grade engineering, not yet production-proven HA.** The single most important sentence in this review: *automatic failover for hard crashes does not currently work* (finding C1 below), and no test in the repo would have caught it. + +## The Critical Findings (fix before anything else) + +**C1. The split-brain resolver is configured but never enabled.** `BuildHocon` never sets `akka.cluster.downing-provider-class`, so the keep-oldest SBR settings are inert: unreachable nodes are never downed, and cluster singletons (site DeploymentManager = all data collection; all six central singletons) never migrate on a hard crash or partition. The documented ~25s failover only works for graceful stops. The repo's own Akka reference notes (`AkkaDotNet/03-Cluster.md`) show the required line. *(Report 01; `Host/Actors/AkkaHostedService.cs:225-270`)* + +**C2. Editing a site's addresses crash-loops central routing to all sites.** `CentralCommunicationActor` stops and immediately recreates a *named top-level* ClusterClient in the same message handling; `Context.Stop` is async, so the name collides and the actor crash-restarts into a permanent loop. Tests mock the client factory, so this is never exercised. *(Report 02; `CentralCommunicationActor.cs:514-521`)* + +**C3. Transport bundle import silently drops template inheritance.** `TemplateDto.BaseTemplateName` is exported and diffed but never consumed at apply — derived templates land as root templates and flatten to amputated configs in the target environment. *(Report 05; `Transport/Import/BundleImporter.cs:1524`)* + +**C4. The on-host production deploy artifact ships `DisableLogin: true`.** Every surface (UI, `/management`, audit REST, debug hub) auto-authenticates with ALL roles — including Operator+Verifier, nullifying the two-person secured-write control. *(Reports 01 & 07; `deploy/wonder-app-vd03/appsettings.Central.json:39-42`)* + +## Cross-Cutting Themes + +### Theme 1 — Failover is designed but unverified, and multiple independent defects hide on that path (stability) + +Beyond C1/C2, the failover path accumulates: leader-vs-oldest confusion ("active node" = cluster leader by address order, but singletons live on the *oldest* node — the event-log purge gate runs on the wrong node and Traefik can route to a node hosting no singletons); during a central partition **both** nodes return 200 on `/health/active` so Traefik serves both indefinitely; the standby site node runs the **full S&F delivery pipeline** with no active-node gate, so replicated Pending rows are delivered by both nodes (systematic duplicate external calls and DB writes); singleton drain tasks run `GracefulStop(10s)` inside a 5s CoordinatedShutdown phase; and inbound-API compiled handlers go stale across central failover (the standby serves startup-time delegates with no revision check — an updated method silently serves the old script with HTTP 200). + +**Root cause is shared:** no test anywhere forms a real two-node cluster and kills a node. "Failover" integration tests are single-process simulations; HOCON tests assert strings, not behavior; the risky collaborators (real ClusterClients, the standby node, shared gRPC channels) are mocked away. *(Reports 01, 02, 06)* + +### Theme 2 — A family of cache-coherence/staleness bugs around compiled scripts (stability) + +The same defect shape appears at least four times: (a) inbound compiled handlers stale across failover; (b) inbound compiled-handler and known-bad-method caches stale after a bundle import overwrites an `ApiMethod` (old code serves HTTP 200 until restart); (c) `_knownBadMethods` never cleared on out-of-band fixes (known, in project memory); (d) `RevisionHashService` omits `NativeAlarmSources`, so native-alarm edits never flag instances stale and sites mirror old sources indefinitely. Related: every staleness probe / Deployments-page validation compiles every script via Roslyn into the non-collectible AssemblyLoadContext with no caching — CPU spikes plus an unbounded assembly leak. **A single "script artifact changed → invalidate everywhere" invalidation contract would close the whole family.** *(Reports 05, 06)* + +### Theme 3 — Transport import is the least-finished shipped feature: silent data loss (underdeveloped) + +Beyond C3: imported scripts silently lose `MinTimeBetweenRuns`/`ExecutionTimeoutSeconds` (and `LockedInDerived` isn't in the DTOs at all); template `NativeAlarmSources` aren't transported while their instance-level overrides *are* (dangling overrides silently dropped at flatten); the exporter hardcodes `AreaName: null` while docs and CLAUDE.md claim Area-by-name travels (the importer's resolution machinery is fully built — a half-shipped feature presented as done); the import blocker heuristic hard-blocks valid imports on PascalCase false positives with no operator override. The bundle **crypto** is solid; it's the **apply path** where spec, DTOs, and code drifted. *(Reports 05, 08)* + +### Theme 4 — Scale-dependent time bombs in the data layer (performance/stability) + +Each is invisible at test scale and arrives on a clock: the spec'd site SQLite 7-day audit purge **does not exist** (site audit DBs grow unbounded); `SwitchOutPartitionAsync` rebuilds the full-table `UX_AuditLog_EventId` unique index in one transaction with the default 30s `CommandTimeout` — once the table is large, the daily purge fails permanently and retention silently stops; SiteCalls KPI predicates full-scan the 365-day table ~15×/minute; `GetExecutionTreeAsync` does a `SELECT DISTINCT` over the entire AuditLog per drilldown; the S&F retry sweep is serial, unbounded, and single-laned (central down + 200-message backlog ≈ 100 min/sweep with head-of-line blocking); site SQLite runs without WAL/busy-timeout under many concurrent writers; no EF `EnableRetryOnFailure` anywhere; the persisted `backlogTotal` KPI trend is hardcoded zero. *(Reports 02, 03, 04)* + +### Theme 5 — Security posture is strong in design, leaky in projection (stability/conventions) + +The session/JWT design, secured-write CAS race guards, and timing-oracle-free API-key auth are exemplary. But: C4 (`DisableLogin` artifact); DataConnection configs (plaintext OPC UA passwords), ExternalSystem `AuthConfiguration`, and DB connection strings are returned **raw to any authenticated user** via ungated List/Get commands — while SMTP/SMS/API keys are properly elided; `deploy artifacts --site-id` is silently ignored and deploys fleet-wide with no site-scope check; pending secured writes never expire server-side (the "Expired" status exists only in the UI — a stale setpoint can be approved and written to a live device days later); every management/auth request does a full LDAP bind with zero throttling (password-spray oracle); role-casing split-brain between UI policies (case-sensitive) and the ManagementActor (case-insensitive). *(Report 07)* + +### Theme 6 — Script containment is cooperative, not enforced (stability) + +Script execution timeout is cooperative-only: a script blocked in synchronous I/O permanently consumes one of the 8 dedicated `ScriptExecutionScheduler` threads — 8 hung scripts silently halt all script **and alarm on-trigger** execution site-wide, with zero observability (no stuck-thread health metric). Relatedly: trigger expressions run Roslyn synchronously on the default dispatcher per attribute change; the inbound method timeout abandons the orphaned task while disposing its DI scope (use-after-dispose); site-side compile failures don't reject the deployment — central is still told Success, contradicting the spec. *(Reports 03, 06)* + +### Theme 7 — Adapter/channel lifecycle leaks on reconnect (stability) + +`OpcUaDataConnection`/`MxGatewayDataConnection` overwrite `_client` on every `ConnectAsync` without dispose/detach — session/socket leak per flap, plus a stale `ConnectionLost` handler that can flap the *new* healthy connection. One debug session's gRPC failover disposes the site's **shared** cached channel, cancelling every other session's subscriptions (concurrent sessions ping-pong and terminate each other). S&F replication is `Task.Run` fire-and-forget with possible Add/Remove reordering. *(Reports 02, 03)* + +## What's Genuinely Good (keep doing this) + +- **Conventions hold under audit** — UTC everywhere, POCO/repository split, options pattern (where validated), additive contracts, clean reference graph, correlation IDs. (Report 08) +- **Idempotency and atomicity where it matters** — EventId-keyed audit ingest, NotificationId insert-if-not-exists, verified-atomic AuditLog+SiteCalls dual-write, CAS on deployment status and secured-write approval. (Reports 04, 07) +- **Error hygiene at the edges** — consistent transient/permanent classification, poison-message parking, enumeration-safe 403s, `IHttpClientFactory` throughout, bounded audit capture. (Report 06) +- **Hardening passes are visible** — Become/Stash lifecycle machines, redeploy termination-watch buffering, adapter-generation guards, HOCON escaping, DebugView disposal/thread-marshalling. (Reports 01, 03, 07) +- **Deferred work is honestly logged** — 23-item inventory in report 08; only ~4 warrant near-term action. + +## Prioritized Recommendations + +**P0 — before the next production deployment** +1. Enable the SBR downing provider in `BuildHocon` and prove it with a real two-node kill test (see P2-10). *(C1)* +2. Remove `DisableLogin: true` from `deploy/wonder-app-vd03` (or add an environment guard that refuses it outside dev). *(C4)* +3. Fix the ClusterClient stop/recreate name collision (await termination or use generation-suffixed names). *(C2)* +4. Gate the S&F delivery pipeline (and event-log purge gate) on singleton-host/active-node status — standby must be passive. *(Theme 1)* + +**P1 — correctness of shipped features** +5. Fix the Transport import apply path: consume `BaseTemplateName`, carry the dropped script fields + `LockedInDerived`, transport template `NativeAlarmSources`, wire `AreaName`. Add round-trip export→import equivalence tests per entity type. *(C3, Theme 3)* +6. Define one script-artifact invalidation contract: bundle import, failover activation, and out-of-band updates must all invalidate compiled-handler/known-bad caches; add `NativeAlarmSources` to `RevisionHashService`; cache validation compiles (collectible ALC). *(Theme 2)* +7. Close the secrets-projection gap: elide DataConnection/ExternalSystem/DB-connection secrets in List/Get like SMTP/SMS already are; honor `SiteId` in `HandleDeployArtifacts` with a site-scope check; add server-side secured-write expiry. *(Theme 5)* +8. Make site-side compile failure reject the deployment (per spec) instead of reporting Success. *(Theme 6)* + +**P2 — before production data volume / as hardening** +9. Data-layer time bombs: implement the site SQLite audit purge; set an explicit long `CommandTimeout` on partition switch; add the missing SiteCalls KPI index; bound the execution-tree CTE; turn on WAL + busy_timeout for site SQLite; `EnableRetryOnFailure`; fix the hardcoded-zero backlog KPI. *(Theme 4)* +10. Build a real two-node failover test rig (docker already provides the topology): kill -9 a node, assert singleton migration, S&F non-duplication, Traefik single-active, inbound handler freshness. This one rig would have caught C1, C2, and three Theme-1 findings. *(Theme 1)* +11. Script containment: watchdog metric for stuck `ScriptExecutionScheduler` threads (health-surface it), move trigger-expression evaluation off the default dispatcher, fix the inbound DI-scope use-after-dispose. *(Theme 6)* +12. Adapter lifecycle: dispose/detach old OPC UA/MxGateway clients on reconnect; per-session (or ref-counted) gRPC channel handling for debug streams; make S&F replication ordered or reconciled. *(Theme 7)* +13. Housekeeping: add CLI.Tests to the slnx (one line, long overdue); extend `ValidateOnStart` options validation to the ~14 unvalidated components; batch/parallelize the S&F retry sweep with a per-sweep LIMIT; fix the docs drift catalogued in reports 04 and 08. + +## Reading Order + +If you read only three reports: **01** (the failover crack), **05** (Transport data loss), **04** (the data-layer clocks). Report **08** is the best single picture of overall codebase health and carries the consolidated 23-item deferred-work inventory. diff --git a/archreview/01-cluster-host-failover.md b/archreview/01-cluster-host-failover.md new file mode 100644 index 00000000..fd45dcf6 --- /dev/null +++ b/archreview/01-cluster-host-failover.md @@ -0,0 +1,391 @@ +# Architecture Review 01 — Cluster Infrastructure, Host, Failover, Health Monitoring, Deployment Topology + +Reviewer domain: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure`, `src/ZB.MOM.WW.ScadaBridge.Host`, +`src/ZB.MOM.WW.ScadaBridge.HealthMonitoring`, `docker/`, `docker-env2/`, `deploy/wonder-app-vd03/`, +Traefik config, and the corresponding design docs (`Component-ClusterInfrastructure.md`, +`Component-Host.md`, `Component-HealthMonitoring.md`, `Component-TraefikProxy.md`) and test projects. + +## Scope + +Read in full: `AkkaHostedService.cs` (1,203 lines — HOCON builder, singleton wiring, coordinated +shutdown), `Program.cs` (both role branches), `StartupValidator.cs`, `StartupRetry.cs`, +`SiteServiceRegistration.cs`, `NodeOptions.cs`, `ClusterOptions.cs` + validator + +`ServiceCollectionExtensions.cs`, all Host `Health/` checks (`RequiredSingletonsHealthCheck`, +`ActiveNodeGate`, `AkkaClusterNodeProvider`), `DeadLetterMonitorActor.cs`, +`AkkaHealthReportTransport.cs`, the entire HealthMonitoring project (`HealthReportSender`, +`CentralHealthAggregator`, `CentralHealthReportLoop`, `SiteHealthCollector`, +`SiteEventLogFailureCountReporter`, `SiteHealthKpiSampleSource`, options + validator), the docker +compose topologies + per-node appsettings + Traefik static/dynamic config + Dockerfile, the +on-host deploy artifacts (`deploy/wonder-app-vd03/appsettings.*.json`), the four design docs, the +repo's own Akka.NET reference notes (`AkkaDotNet/03-Cluster.md`), and the relevant test projects +(`Host.Tests`, `ClusterInfrastructure.Tests`, `HealthMonitoring.Tests`, `IntegrationTests` +failover/readiness suites). Health-report ingestion on the central side was cross-checked in +`Communication/Actors/CentralCommunicationActor.cs:349-392`. + +## Maturity Verdict + +This slice of the codebase is unusually well-commented, defensively coded at the micro level +(options validators with `ValidateOnStart`, CAS loops in the aggregator, sequence-number seeding +for failover, HOCON escaping, bounded startup retry), and clearly the product of many careful +review passes — but it has **one macro-level defect that undoes the entire failover design: the +split-brain resolver is configured but never enabled** (no `akka.cluster.downing-provider-class`), +so unreachable nodes are never automatically downed and cluster singletons never migrate on a hard +crash. Every document, comment, and validator in the domain reasons meticulously about keep-oldest +semantics that are inert at runtime, and the test suite asserts the HOCON *strings* rather than the +*behavior*, which is exactly how the gap survived. Secondary structural issues cluster around the +ambiguous definition of "active node" (cluster *leader* vs. singleton-hosting *oldest* member, +which diverge after restarts) and dev-vs-prod configuration drift. Verdict: **high polish, one +critical foundation crack; not production-ready for unattended failover until the downing provider +is fixed and verified with a real two-node test.** + +--- + +## 1. Stability + +### [Critical] Split-brain resolver is never enabled — automatic failover does not work for crashes/partitions + +`AkkaHostedService.BuildHocon` (src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs:225-270) +emits: + +```hocon +cluster { + ... + split-brain-resolver { + active-strategy = "keep-oldest" + stable-after = 15000ms + keep-oldest { down-if-alone = on } + } + failure-detector { ... } + run-coordinated-shutdown-when-down = on +} +``` + +but never sets `akka.cluster.downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider, +Akka.Cluster"`. In Akka.NET (1.5.62 per `Directory.Packages.props:8-17`), the default downing +provider is **NoDowning**; the whole `split-brain-resolver` section is only read by +`SplitBrainResolverProvider`, which must be opted into via `downing-provider-class`. A repo-wide +grep confirms the string `SplitBrainResolverProvider` appears **nowhere in `src/`** — only in the +project's own reference notes, which show the required line +(AkkaDotNet/03-Cluster.md:56, also 15-Coordination.md:69 and 18-MultiNodeTestRunner.md:100). + +Failure scenario: the active site node loses power. The standby's failure detector marks it +unreachable (~10s), but **nothing ever downs it**. The unreachable member stays in the membership +forever, the `ClusterSingletonManager` cannot hand over (singleton migration requires the oldest +member's *removal*, not mere unreachability), and the site's DeploymentManager singleton — and with +it all data collection, script execution, and S&F delivery — never restarts on the survivor. The +same applies to all six central singletons (notification-outbox, audit-log-ingest, site-call-audit, +audit-log-purge, site-audit-reconciliation, kpi-history-recorder). The documented "~25 seconds +total failover" (docs/requirements/Component-ClusterInfrastructure.md:110-119) is only achievable +today for a **graceful** stop (CoordinatedShutdown leave); crash and partition scenarios hang +indefinitely until an operator downs the node or restarts the survivor. + +Corroborating evidence that this was never exercised: + +- `HoconBuilderTests.cs:45,112,155` assert only that the SBR *strings* appear in the config. +- `CoordinatedShutdownTests.cs:11-43` are source-text greps (`Assert.Contains("run-by-clr-shutdown-hook = on", content)`). +- The "failover" integration tests are single-process simulations — e.g. + `CentralFailoverTests.cs:27-47` validates a JWT across two `JwtTokenService` instances; no test + forms a two-node cluster and kills the oldest node. +- `run-coordinated-shutdown-when-down = on` (AkkaHostedService.cs:265) is dead config: a node is + never transitioned to Down by anything. +- `ClusterOptionsValidator` (src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptionsValidator.cs:35-63) + rigorously rejects any strategy other than `keep-oldest` and demands `DownIfAlone == true` — + validating knobs on a resolver that is not installed. + +Fix: add the `downing-provider-class` line to `BuildHocon` (one line), or better, migrate the +bootstrap to `Akka.Cluster.Hosting`'s typed options (already referenced, +ZB.MOM.WW.ScadaBridge.Host.csproj:15) whose `SplitBrainResolverOption` wires the provider +automatically — and add a real two-node in-process cluster test that kills the oldest member and +asserts the singleton re-hosts within the expected window. + +### [High] "Active node" means two different things — cluster leader vs. oldest member — and they diverge + +Two distinct node-selection concepts are used interchangeably: + +- **Leader-based** (address ordering, *not* age): `ActiveNodeGate.IsActiveNode` + (src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeGate.cs:36-52), + `AkkaClusterNodeProvider.SelfIsPrimary` and the "Primary"/"Standby" labels + (src/ZB.MOM.WW.ScadaBridge.Host/Health/AkkaClusterNodeProvider.cs:29-40,61), the `active-node` + health check Traefik routes on (Program.cs:231-234, docker/traefik/dynamic.yml:12-18), the + `CentralHealthReportLoop` leader gate (CentralHealthReportLoop.cs:93-97), and the site + event-log purge gate (`SiteEventLogActiveNodeCheck` → `SelfIsPrimary`, + SiteServiceRegistration.cs:116-120). +- **Oldest-based** (singleton placement): every `ClusterSingletonManager` (AkkaHostedService.cs:410+, + 906-925), and the site health "IsActiveNode" flag, which is set by the DeploymentManager + *singleton* on start/stop (src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs:243,258). + +In Akka the leader is the lowest-address `Up` member; the singleton host is the *oldest* member. +They coincide at first boot but diverge permanently after the original first node is restarted and +rejoins (it regains leadership by address order while the peer remains oldest). Concrete +consequences once diverged: + +1. **Site event-log retention is unenforced on the node that writes events.** Events are recorded + by the singleton hierarchy on the oldest node; the purge (`EventLogPurgeService.cs:21,36`) + early-exits unless `SelfIsPrimary` (leader). The active/oldest node's SQLite grows past the + 30-day/1GB cap while the leader purges an idle database. +2. **Health dashboard labels the wrong node "Primary"** (AkkaClusterNodeProvider.cs:61): the node + named Primary hosts no singletons; the node doing all the work shows Standby. +3. **Central self-report is emitted by the leader** (CentralHealthReportLoop.cs:93) while the + NotificationOutbox/audit singletons run on the other node — the `$central` health card describes + the wrong node. +4. **Traefik routes UI/API to the leader** while every singleton `Ask` then hops to the peer — + functionally correct via the proxies, but it contradicts the design docs' "one active node runs + all central components" model (Component-ClusterInfrastructure.md:44-47) and doubles + intra-cluster latency on every outbox/audit page load. + +Recommendation: pick one definition. Simplest coherent choice: derive "active" from singleton-host +status (e.g. compare self against the oldest `Up` member, or gate on reachability of a local +singleton), and use it consistently for `ActiveNodeGate`, `SelfIsPrimary`, the purge gate, and the +`/health/active` check — leadership is an Akka-internal concept that only accidentally matches the +product's active/standby model. + +### [High] During a central network partition, both nodes report `/health/active = 200` and Traefik serves both + +`ActiveNodeGate`/the active-node check compute `cluster.State.Leader` from each node's own gossip +view. When central-a and central-b cannot see each other but Traefik can reach both (a plausible +partial partition), each side sees the peer as unreachable and computes **itself** as leader — both +return 200 on `/health/active`, and Traefik's load balancer (docker/traefik/dynamic.yml:10-18) +round-robins requests between two nodes that each believe they are active. Blazor circuits and the +Inbound API's active-node 503 gate (Program.cs:256-260) are then split across both. Because the SBR +is not enabled (Critical finding above), nothing ever resolves the partition — this state persists +until manual intervention. Singletons do *not* duplicate (that requires member removal), but every +leader-gated behavior does. Enabling keep-oldest + `down-if-alone` fixes the persistence; consider +also making `/health/active` require `Ready` (DB reachable) in addition to leadership so a +DB-dead leader is pulled from rotation. + +### [Medium] Cluster-leave drain tasks are budgeted 10s inside a 5s CoordinatedShutdown phase + +Five singletons add a `PhaseClusterLeave` task that runs `GracefulStop(TimeSpan.FromSeconds(10))` +(AkkaHostedService.cs:551-567 site-call-audit, 624-641 audit-log-purge, 676-693 +site-audit-reconciliation, 733-750 kpi-history-recorder, 787-804 pending-deployment-purge). Akka's +`cluster-leave` phase uses the default phase timeout of **5 seconds**; the phase abandons the task +at 5s (default `recover = on`) and proceeds, so the 10s graceful-stop can never complete its full +window and the `catch` logging in those tasks never observes the overrun (the task is orphaned, not +faulted). The intended "let the in-flight EF upsert drain before handover" guarantee is silently +halved. Fix: set `akka.coordinated-shutdown.phases.cluster-leave.timeout` in `BuildHocon` to +exceed the graceful-stop budget, or shrink the `GracefulStop` timeout below 5s. + +### [Medium] The two busiest singletons have no drain task at all + +The `NotificationOutboxActor` and `AuditLogIngestActor` singletons are terminated by bare +`PoisonPill` with no cluster-leave drain (AkkaHostedService.cs:410-419, 451-458); the comment at +AkkaHostedService.cs:544-549 explicitly notes the pattern "is suitable for the NotificationOutbox +singleton; not added here to keep this change minimal". These are precisely the actors with +in-flight EF transactions on every message (dispatch sweeps, dual-write ingest). The idempotent +DB design (insert-if-not-exists on `NotificationId`/`EventId`) makes this safe-but-lossy rather +than corrupting, but the inconsistency (drain for the low-traffic maintenance singletons, none for +the hot ones) should be resolved. + +### [Medium] Health-report loss recovery is dead code on the production transport + +`HealthReportSender` carefully restores interval counters if `_transport.Send` throws +(HealthReportSender.cs:158-175), and `SiteHealthCollector.AddIntervalCounters` +(SiteHealthCollector.cs:153-171) exists to support it. But the only production transport is +`AkkaHealthReportTransport.Send` (src/ZB.MOM.WW.ScadaBridge.Host/AkkaHealthReportTransport.cs:26-33) +— a fire-and-forget `Tell` to an `ActorSelection`, which **never throws** and silently returns when +the actor system is null. Any downstream loss (central unreachable, ClusterClient reconnecting, +dead-lettered selection) drops the report *and* the interval error counters it carried — the +scenario the restore logic was built for is exactly the one it can't see. Either make the transport +Ask-with-ack (the site→central notification forward already uses this pattern over the same actor) +or accept the loss and delete the restore machinery; the current shape gives false confidence. + +### [Medium] Offline detection deviates from the spec and can mask a dead metrics pipeline + +`Component-HealthMonitoring.md:43-46` defines offline as "no *report* within 60s". +`CentralHealthAggregator.CheckForOfflineSites` (CentralHealthAggregator.cs:224-263) instead keys +off `LastHeartbeatAt`, which is also advanced by the ~5s transport heartbeats +(`CentralCommunicationActor.HandleHeartbeat` → `MarkHeartbeat`, CentralCommunicationActor.cs:349-353). +A site whose `HealthReportSender` loop has died (or whose DeploymentManager singleton is stuck, so +`IsActiveNode` is false on both nodes and reports are skipped — HealthReportSender.cs:85) keeps +heartbeating and shows **online with silently frozen metrics forever**; nothing surfaces +`LastReportReceivedAt` staleness. The heartbeat-based liveness choice is defensible (the code +comment argues it), but then the dashboard needs a distinct "metrics stale" signal, and the design +doc should be updated — currently spec and code disagree. + +### [Medium] Second seed node in the checked-in site dev config targets the metrics port; validator has a gap + +`src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json:14-17` seeds +`akka.tcp://scadabridge@localhost:8082` and `akka.tcp://scadabridge@localhost:8084`, while the same +file sets `MetricsPort: 8084` (line 10). The second seed points at the Kestrel HTTP/1.1 metrics +listener — a node attempting to join via that seed performs a doomed Akka.Remote association +against Kestrel (handshake noise, retry loops). `StartupValidator` explicitly guards seed-vs-GrpcPort +(StartupValidator.cs:121-127) but not seed-vs-MetricsPort, so this config validates cleanly. Fix +the dev config (8084 → the intended second node's remoting port) and extend the seed-port loop to +also reject `MetricsPort`. + +### [Medium] Docker stop grace period is shorter than a full CoordinatedShutdown + +`docker/docker-compose.yml` sets no `stop_grace_period` (default 10s SIGTERM→SIGKILL) and the +Dockerfile runs `dotnet` as PID 1 (docker/Dockerfile, final stage). A graceful shutdown runs +cluster-leave (5s budget) + cluster-exiting (10s) + actor-system-terminate plus Serilog flush — +easily exceeding 10s on a loaded node, so `bash docker/deploy.sh` recreates can SIGKILL mid-drain, +turning every "graceful" redeploy into the crash path (which, per the Critical finding, currently +has no automatic recovery on the surviving node). Add `stop_grace_period: 30s` to the app services. + +### [Low] Aggregator never forgets sites + +`CentralHealthAggregator._siteStates` (CentralHealthAggregator.cs:16) only grows: a site deleted +from configuration remains a permanently-offline dashboard tile (and continues to be enumerated by +`SiteHealthKpiSampleSource.CollectAsync`, SiteHealthKpiSampleSource.cs:74-85) until process +restart. Needs an eviction path wired to site deletion. + +### [Low] Dead-letter monitor: unbounded warning volume and restart-reset count + +`DeadLetterMonitorActor` logs one Warning per dead letter with no sampling/rate-limiting and keeps +its count in actor state that resets on restart +(src/ZB.MOM.WW.ScadaBridge.Host/Actors/DeadLetterMonitorActor.cs:14,26-35). A dead-letter storm +(mass undeploy, failover, shutdown) floods the log; the health metric path +(`IncrementDeadLetter`) is fine because the collector counter is interval-based. + +### [Low] Mandatory two-seed rule forces phantom seeds in single-node installs + +`ClusterOptionsValidator` requires ≥2 seed nodes +(ClusterOptionsValidator.cs:29-33); the shipped single-node production artifact satisfies it with +a documented placeholder (`deploy/wonder-app-vd03/appsettings.Central.json:9-19`, second seed +`localhost:8091` "harmless while absent"). The node dials the phantom seed forever (log noise, +periodic connect attempts), and the rule's intent — no startup-ordering dependency — is defeated +by fiction. Consider allowing 1 seed with a warning, or a `SingleNode` acknowledgement flag. + +## 2. Performance + +This domain is largely cold-path and correctly engineered for it; nothing here touches the +data-plane hot path. Observations: + +- **[Low] Readiness probes fan out five cluster Asks per poll.** + `RequiredSingletonsHealthCheck` sends 5 concurrent `Identify` Asks through singleton proxies + every `/health/ready` poll (RequiredSingletonsHealthCheck.cs:66-78,111-117). When the singletons + live on the peer node each probe is a remoting round trip. Bounded (2s, concurrent) and cheap at + typical poll rates, but a tight orchestration probe interval multiplies cluster chatter. +- **[Low] Startup compiles every inbound API method sequentially before serving** + (Program.cs:379-388). Startup latency scales linearly with method count; a slow Roslyn compile of + many methods delays readiness on the *standby* too (it runs the same branch). Acceptable now; + parallelize or defer-compile if method counts grow. +- **[Low] Per-tick allocations in health collection** (dictionary snapshots per 30s report, + `GetAllSiteStates` copy per dashboard poll — CentralHealthAggregator.cs:173-176) are trivial at + this cadence. No action. +- **Positive:** the site audit telemetry actor is pinned to a dedicated dispatcher + (`audit-telemetry-dispatcher`, AkkaHostedService.cs:226-232,1150-1158) so batch SQLite/gRPC work + never contends with hot-path actors; `StoreAndForwardService.StartAsync` is awaited rather than + sync-blocked (AkkaHostedService.cs:976-981); the ActorSystem DI bridge is a root singleton + precisely to avoid per-scope disposal (Program.cs:240-250) — all correct. + +## 3. Conventions + +- **[Medium] REQ-HOST-6 says "Akka.NET bootstrap via Akka.Hosting"; the code hand-rolls HOCON.** + `Component-Host.md` (REQ-HOST-6, line 106-114) mandates Akka.Hosting; the actual bootstrap is + string-interpolated HOCON + `ActorSystem.Create` (AkkaHostedService.cs:171-176,214-271) with + `Akka.Hosting`/`Akka.Cluster.Hosting`/`Akka.Remote.Hosting` referenced but unused for cluster + bring-up (ZB.MOM.WW.ScadaBridge.Host.csproj:15-18). This is not just doc drift: Akka.Cluster.Hosting's + typed `SplitBrainResolverOption` sets `downing-provider-class` for you — using the mandated API + would have prevented the Critical finding. Either implement REQ-HOST-6 or amend it and drop the + unused packages. +- **[High] Production deploy artifact ships with authentication disabled.** + `deploy/wonder-app-vd03/appsettings.Central.json:39-42` sets `"DisableLogin": true` — every + request auto-authenticates as `multi-role` with all roles, and the management API on that host is + unauthenticated. The base config documents this as "DEV/TEST ONLY — never enable in production" + with "no environment guard; a startup warning is the only protection" + (src/ZB.MOM.WW.ScadaBridge.Host/appsettings.json:6). A `deploy/` artifact with an install + runbook is a production surface; at minimum gate `DisableLogin` on + `Environment.IsDevelopment()` or require an explicit `I-understand` companion flag. +- **[Medium] LDAP transport is plaintext everywhere, contradicting the security spec.** The design + requires "LDAPS/StartTLS required" (CLAUDE.md Security & Auth; Component-Security), but every + shipped config — docker (docker/central-node-a/appsettings.Central.json:26-33) and the on-host + deploy (deploy/wonder-app-vd03/appsettings.Central.json:26-34) — sets `"Transport": "None", + "AllowInsecure": true` and sends the service-account bind password in clear. Dev GLAuth is a fair + excuse for docker; the deploy artifact should not normalize it. +- **[Medium] Secrets in checked-in dev configs.** SQL credentials, LDAP service-account password + and the JWT signing key are committed in `docker/central-node-a/appsettings.Central.json:21-38` + (labelled dev-only; the compose pepper is likewise flagged, docker-compose.yml:9-14). The + wonder-app-vd03 artifact handles this correctly via `${...}` placeholders + env injection + (appsettings.Central.json:20-24) — good pattern, worth backporting a scrubbed variant to docker + if these ever leave a lab network. Also note `ConfigSecretsTests.cs` exists in Host.Tests, so the + team is tracking this. +- **[Low] Stale doc references.** `Component-TraefikProxy.md:121` locates `ActiveNodeHealthCheck` + at `src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeHealthCheck.cs`; the check now comes from the + shared `ZB.MOM.WW.Health.Akka` package (Program.cs:214-234) and no such file exists. + `Component-TraefikProxy.md:115` also still describes `/health/ready` as "database + Akka + cluster" — it now includes `required-singletons` (Program.cs:227-230; Component-Host.md REQ-HOST-4a + *is* current). Keep the Traefik doc in sync. +- **[Low] Five copy-pasted ~60-line singleton registration blocks.** + AkkaHostedService.RegisterCentralActors repeats the manager/drain-task/proxy triple five times + (AkkaHostedService.cs:399-811) with only names and Props varying. Extract a + `StartCentralSingleton(name, props, drainTimeout)` helper — the copy-paste already caused drift + (outbox/ingest lack drain tasks). +- **Positive conventions compliance:** Options pattern is followed rigorously (component-owned + options classes, `ValidateOnStart`, dedicated validators with key-naming messages — + ClusterOptionsValidator, HealthMonitoringOptionsValidator); HOCON interpolation is + injection-safe (`QuoteHocon`/`DurationHocon`, AkkaHostedService.cs:280-295); Serilog enrichment + matches REQ-HOST-8; startup validation matches REQ-HOST-4 including cross-field port-collision + rules; single-binary role dispatch matches REQ-HOST-1/2; the `/health/ready` vs `/health/active` + tier separation (leadership deliberately excluded from readiness) is well-reasoned and + documented in place (Program.cs:346-356). + +## 4. Underdeveloped Areas + +1. **No real multi-node failover test anywhere.** `CentralFailoverTests` are single-process + simulations (JWT across two service instances, CentralFailoverTests.cs:27-47); + `SiteFailoverTests`/`DualNodeRecoveryTests` similarly exercise component state recovery, not + cluster membership; `CoordinatedShutdownTests` grep source text (CoordinatedShutdownTests.cs:11-43); + `HoconBuilderTests` assert config strings. The repo's own notes describe the MultiNode test + runner (AkkaDotNet/18-MultiNodeTestRunner.md) but it is unused. This is the direct cause of the + Critical finding surviving: nothing ever killed the oldest node and waited for the singleton to + move. Even a plain xunit test spinning two `ActorSystem`s in-process and calling + `cluster.Down()`/terminating one would have caught it. +2. **Windows Service lifecycle management is thin.** `Component-ClusterInfrastructure.md:19` + claims "Manage Windows service lifecycle (start, stop, restart)"; the implementation is + `UseWindowsService()` (Program.cs:74,401) plus install scripts. Service recovery actions + (restart-on-failure), the watchdog implied by `down-if-alone` ("an external mechanism must + restart both nodes" — AkkaDotNet/03-Cluster.md:64) do not exist in the repo; after a + down-if-alone self-down the process presumably exits (or worse, idles) with nothing to restart + it. The down-if-alone recovery story needs an explicit design (does the process exit? does the + service restart it? does it rejoin cleanly?) and a drill. +3. **TLS is absent at every layer of this domain**: Traefik terminates plain HTTP with an + unauthenticated dashboard (docker/traefik/traefik.yml:5-7 — documented as dev-only in + Component-TraefikProxy.md:131-138), Akka remoting is unencrypted TCP, site gRPC is `http://`. + Fine for the lab; there is no production TLS profile or documented path yet. +4. **Dockerfile has no `HEALTHCHECK`** and compose has no health-based dependency ordering; + liveness `/healthz` runs zero checks (Program.cs:353-355) so a wedged actor system still + reports live. Acceptable minimalism, but an orchestrator migration will need both. +5. **`NodeName` is missing from the wonder-app-vd03 deploy overlays** + (deploy/wonder-app-vd03/appsettings.Central.json has no `Node.NodeName`), so audit rows from + that install get a NULL `SourceNode` (the base file documents the normalisation, + src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Central.json:3) — undermines the per-node audit/KPI + features (`IX_AuditLog_Node_Occurred`, per-node stuck KPIs) on the one real deployment. +6. **Known-but-open follow-ups acknowledged in code/docs:** SecuredWrite audit rows leave + `SourceNode` NULL (CLAUDE.md notes it as logged); no central persistence of OPC UA cert trust; + `docker-env2` mirrors the main topology (verified: same Traefik config shape, same + seed/port conventions) so it inherits every finding above. +7. **Health monitoring has no history by design** (Component-HealthMonitoring.md:93-97) — now + partially superseded by KPI History (#26), which the docs handle; but online/offline + *transitions* are still unrecorded anywhere (doc line 97 defers it), so post-incident "when did + the site drop" questions remain unanswerable beyond log scraping. + +## Prioritized Recommendations + +1. **[Critical, one line + tests] Enable the split-brain resolver.** Add + `downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster"` to + `BuildHocon` (AkkaHostedService.cs:250) — or move the bootstrap to Akka.Cluster.Hosting per + REQ-HOST-6, which sets it via typed options. Then add a two-node in-process failover test + (kill oldest → assert singleton re-hosted, member removed within stable-after + margin) and a + documented manual drill against the docker cluster (`docker kill scadabridge-central-a`, watch + Traefik + singleton logs). +2. **[High] Unify the "active node" definition** on singleton-host (oldest) semantics for + `ActiveNodeGate`, `AkkaClusterNodeProvider.SelfIsPrimary`, the event-log purge gate, and the + `/health/active` check; today's leader-based checks purge the wrong node's event log and label + the wrong node Primary once leader ≠ oldest. +3. **[High] Remove `DisableLogin: true` from `deploy/wonder-app-vd03`** or add a hard environment + guard around the flag. +4. **[Medium] Fix the CoordinatedShutdown drain budget** (`cluster-leave` phase timeout ≥ + graceful-stop timeout) and add drain tasks for the notification-outbox and audit-log-ingest + singletons — or delete all five drains if the idempotent-DB argument makes them unnecessary. +5. **[Medium] Fix `appsettings.Site.json` seed `localhost:8084`** and extend `StartupValidator`'s + seed-port loop to reject `MetricsPort` collisions. +6. **[Medium] Make health-report delivery observable**: switch `AkkaHealthReportTransport` to an + acked Ask (reusing the existing site→central forward pattern) so the counter-restore logic in + `HealthReportSender` actually fires, and surface `LastReportReceivedAt` staleness on the + dashboard so heartbeat-alive/metrics-dead sites are visible. +7. **[Medium] Add `stop_grace_period: 30s`** to the app services in both compose files. +8. **[Low] Housekeeping:** evict deleted sites from `CentralHealthAggregator`; rate-limit + dead-letter warnings; extract the singleton-registration helper; sync + `Component-TraefikProxy.md` with the shared health-check library; add `NodeName` to the + wonder-app-vd03 overlays; reconsider the mandatory-two-seeds rule for single-node installs. diff --git a/archreview/02-communication-store-and-forward.md b/archreview/02-communication-store-and-forward.md new file mode 100644 index 00000000..bd7f6769 --- /dev/null +++ b/archreview/02-communication-store-and-forward.md @@ -0,0 +1,160 @@ +# Architecture Review — Central–Site Communication & Store-and-Forward + +Reviewer domain: `src/ZB.MOM.WW.ScadaBridge.Communication`, `src/ZB.MOM.WW.ScadaBridge.StoreAndForward`, the telemetry transport paths that ride them, and the corresponding test projects. +Date: 2026-07-08. + +## Scope + +Read in full: `Component-Communication.md`, `Component-StoreAndForward.md`; all 9 S&F source files (`StoreAndForwardService.cs`, `StoreAndForwardStorage.cs`, `ReplicationService.cs`, `NotificationForwarder.cs`, `ParkedMessageHandlerActor.cs`, options/messages/DI); all Communication source (`CentralCommunicationActor.cs`, `SiteCommunicationActor.cs`, `CommunicationService.cs`, `DebugStreamService.cs`, `DebugStreamBridgeActor.cs`, `StreamRelayActor.cs`, `SiteStreamGrpcServer.cs`, `SiteStreamGrpcClient.cs`, `SiteStreamGrpcClientFactory.cs`, `ReconcileService.cs`, `PendingDeploymentPurgeActor.cs`, `CommunicationOptions.cs`); the transport ends of the telemetry pipelines (`AuditLog/Site/Telemetry/SiteAuditTelemetryActor.cs`, `ClusterClientSiteAuditClient.cs`); and the Host wiring that composes them (`Host/Actors/AkkaHostedService.cs` — `BuildHocon`, `RegisterCentralActors`, `RegisterSiteActorsAsync`), plus `SiteRuntime/Actors/SiteReplicationActor.cs` and the `Notify.Send` enqueue path in `SiteRuntime/Scripts/ScriptRuntimeContext.cs`. Central-side ingest internals (NotificationOutboxActor, AuditLogIngestActor, SiteCallAuditActor) are another reviewer's domain; only their transport contracts are assessed here. + +## Maturity Verdict + +This is a carefully engineered, unusually well-documented subsystem that is strong *in the small* — idempotency keys threaded end-to-end (NotificationId, EventId, TrackedOperationId), Sender captured before every PipeTo, compare-and-swap storage updates guarding the sweep against operator races, additive-only proto discipline with contract tests, and deliberate, written-down failure semantics on nearly every path. However, it has a small number of serious *in-the-large* lifecycle/topology defects that the (otherwise broad) test suite is structurally blind to, because the risky collaborators are mocked away: the ClusterClient recreate-on-address-change path will almost certainly crash-loop the `CentralCommunicationActor` in production, the standby site node runs the full S&F delivery pipeline with no active-node gate (systematic duplicate-delivery exposure the spec explicitly says should only occur on failover), and one debug session's gRPC failover disposes the shared channel out from under every other session to the same site. The retry sweep's serial, unbounded design also collapses under a realistic central outage backlog. Verdict: **high code quality, mid maturity as a distributed system — 3–4 targeted fixes away from solid.** + +--- + +## Findings — Stability + +### [Critical] ClusterClient recreate-on-address-change causes actor-name collision and a permanent restart loop +`CentralCommunicationActor.HandleSiteAddressCacheLoaded` stops the old client and immediately creates a new one under the same name: + +- `CentralCommunicationActor.cs:514-521` — `Context.Stop(_siteClients[siteId].Client)` followed synchronously by `_siteClientFactory.Create(...)`. +- `CentralCommunicationActor.cs:36-40` (`DefaultSiteClientFactory`) — `system.ActorOf(ClusterClient.Props(settings), $"site-client-{siteId}")`: a **named top-level actor**. + +`Context.Stop` is asynchronous; the name `site-client-{siteId}` stays reserved until the old actor fully terminates. Creating the replacement in the same message handling will throw `InvalidActorNameException` essentially every time. The `Create` call is **outside** the surrounding try/catch (which only guards `ActorPath.Parse`, `CentralCommunicationActor.cs:492-505`), so the exception escapes the handler → the user-guardian default directive **restarts** the actor → the new incarnation's `_siteClients` dictionary is empty, but all old `site-client-*` actors are still alive at the system level. Every subsequent 60-second refresh (`PreStart`, `CentralCommunicationActor.cs:564-569`) then tries to create clients for **all** sites, collides on the reserved names, and crashes again. Failure scenario: an operator edits one site's NodeA address in the Central UI → central loses ClusterClient routing to *every* site (all `SiteEnvelope`s dropped at `HandleSiteEnvelope`, `CentralCommunicationActor.cs:398-413`) until the central process is restarted. This is the exact path the design doc promises works: "ClusterClient instances are recreated when contact points change" (`Component-Communication.md:238`). + +Secondary issue on the same lines: `siteId` is interpolated into an actor name unvalidated. A `SiteIdentifier` containing `/`, whitespace, or other non-path characters throws `InvalidActorNameException` on first creation with the same restart-loop consequence — contrast with `SiteStreamGrpcServer.SubscribeInstance`, which validates `correlation_id` with `ActorPath.IsValidPathElement` for precisely this reason (`SiteStreamGrpcServer.cs:229-234`). + +Why tests miss it: `CentralCommunicationActorTests` substitutes `ISiteClientFactory` with an NSubstitute mock returning TestProbes (`tests/.../CentralCommunicationActorTests.cs:42-51`) — the real naming/lifecycle machinery is never exercised anywhere. + +Fix shape: unique names (`site-client-{siteId}-{seq}` via a counter, like `SiteStreamGrpcServer._actorCounter`), or watch `Terminated` and defer recreation, plus sanitize/validate the site id, plus one test using `DefaultSiteClientFactory`. + +### [High] Standby site node runs the full S&F delivery pipeline — duplicate delivery is systematic, not a failover residue +The design says the standby only *applies* replicated operations and "on failover, the new active node resumes delivery" (`Component-StoreAndForward.md:77-80`). The code starts delivery on **both** nodes: + +- `AkkaHostedService.cs:981` — `await storeAndForwardService.StartAsync()` runs in `RegisterSiteActorsAsync` on **every** site node, starting the retry timer (`StoreAndForwardService.cs:337-341`). +- `AkkaHostedService.cs:989, 998, 1033` — all three delivery handlers (ExternalSystem, CachedDbWrite, Notification) are registered unconditionally on every node. +- `SiteReplicationActor.cs:280-291` — replicated `Add` operations are applied to the standby's SQLite, so the standby's sweep has real Pending rows to deliver. + +There is no active-node/leader gate anywhere in `StoreAndForwardService.RetryPendingMessagesAsync` (`StoreAndForwardService.cs:563-589`) or in the delivery handlers (`ExternalSystemClient.DeliverBufferedAsync`, `ExternalSystemGateway/ExternalSystemClient.cs:173-224` — checked, no gate). Consequences: + +1. **Duplicate deliveries in steady operation.** Both nodes' sweeps become due for the same message at the same wall-clock moment (the replicated row carries the same `last_attempt_at`). Whichever delivers first replicates a `Remove`, but if both attempts are in flight concurrently — a window the size of the delivery duration (an HTTP call, often ≥ hundreds of ms) plus replication latency — the target receives the call **twice**. Over a long outage with many buffered messages, duplicates recur every recovery. Notifications survive this (central dedups on `NotificationId` insert-if-not-exists), but `ExternalSystem` calls and `CachedDbWrite`s do not — "idempotency is the caller's responsibility" was scoped to *rare failover* duplicates, not steady-state ones. +2. **Doubled retry traffic** against an already-struggling target during every outage. +3. Both nodes emit `CachedCallTelemetry` attempts for the same operation with independent retry counts, muddying the central audit picture. + +Fix shape: gate the sweep (and ideally the queue-depth gauge registration) behind the same leader-check used by `SiteCommunicationActor.DefaultIsActiveCheck` (`SiteCommunicationActor.cs:498-507`), re-evaluated per sweep tick so failover resumes delivery automatically. + +### [High] One debug session's gRPC failover disposes the shared channel under every other session to the same site +`SiteStreamGrpcClientFactory` caches **one** `SiteStreamGrpcClient` per site, keyed by site id, replacing (and disposing) it whenever a caller asks for a different endpoint (`SiteStreamGrpcClientFactory.cs:52-78`). `DebugStreamBridgeActor` holds **per-session** node preference and flips endpoint on every stream error (`DebugStreamBridgeActor.cs:416, 468-483`). With two concurrent debug sessions to the same site: + +- Session A (on NodeA) hits an error and flips to NodeB → `GetOrCreate(site, NodeB)` **disposes** the NodeA-bound client. `SiteStreamGrpcClient.ReleaseResources` cancels **every** registered subscription CTS and disposes the channel (`SiteStreamGrpcClient.cs:314-324`) — killing session B's healthy stream. +- Session B's `SubscribeAsync` observes the cancellation as an error → B flips to NodeA → disposes the NodeB client → kills A's brand-new stream. The two sessions ping-pong, burning both sessions' 3-retry budgets (`DebugStreamBridgeActor.cs:45`) and terminating both. + +Also, `CleanupGrpc` (`DebugStreamBridgeActor.cs:486-495`) calls `GetOrCreate` during teardown — if the cached client is bound to the other endpoint, cleanup *creates a fresh channel* (and disposes the cached one) merely to call `Unsubscribe`, with the same collateral effect. + +Fix shape: key the cache by `(siteId, endpoint)` (both node channels can coexist), or reference-count clients, or give each bridge actor its own client instance. The single-flip failover for a *site-wide* address change can stay in the factory; per-session failover must not mutate shared state. + +### [Medium] Replication operations can be applied out of order +`ReplicationService.FireAndForget` wraps each operation in its own `Task.Run` (`ReplicationService.cs:134-149`). Two operations for the same message issued in quick succession (Add then Remove; Park then operator Requeue) are handed to the thread pool as independent tasks — the pool does not preserve ordering, so the `replicationActor.Tell(...)` calls (handler wired at `AkkaHostedService.cs:897-901`) can happen inverted. Akka's per-sender/receiver ordering guarantee only applies *after* the Tells, so the standby can apply `Remove` (no-op, row absent) then `Add` → an orphan Pending row that the ungated standby sweep (finding above) or a failover will re-deliver. The `Task.Run` buys nothing — the handler is a non-blocking Tell returning `Task.CompletedTask`; invoking it inline (with a try/catch) preserves ordering by construction. + +### [Medium] Replication failures are logged at Debug — a dead replication channel is invisible +`ReplicationService.cs:144-147` logs a failed replication at `LogDebug`. If the handler starts failing systematically (peer unreachable in a way `SendToPeer` doesn't detect, serialization fault of `StoreAndForwardMessage`), the standby's buffer silently diverges and the next failover loses the entire un-replicated delta — with zero operator-visible signal at default log levels and no counter metric. `SiteReplicationActor.SendToPeer`'s no-peer drop is also Debug (`SiteReplicationActor.cs:139-149`). At minimum Warning + a telemetry counter; the repo already has `ScadaBridgeTelemetry` plumbing for exactly this kind of gauge. + +### [Medium] Park/Requeue replication is a blind UPDATE — lost on a standby missing the row +`ApplyReplicatedOperationAsync` applies `Park` and `Requeue` via `UpdateMessageAsync` (`ReplicationService.cs:121-130`), which is `UPDATE ... WHERE id = @id` (`StoreAndForwardStorage.cs:210-232`) — zero rows affected is silent. If the original `Add` was lost (fire-and-forget), the park state never materializes on the standby: after failover the message is simply *gone* from the parked view (operator can't retry/discard it) even though the full message payload was in hand in the replication operation. An upsert (INSERT OR REPLACE) for Park/Requeue would self-heal the lost-Add case for free. + +### [Medium] Notifications park after ~25 minutes of central unreachability, and the "no limit" escape hatch is unused +`Notify.Send` enqueues with no `maxRetries` (`ScriptRuntimeContext.cs:2197-2203`) → `DefaultMaxRetries = 50` (`StoreAndForwardOptions.cs:21`) at the 30 s default interval = ~25 minutes of central outage before a notification parks and requires per-message operator action. `Component-StoreAndForward.md:52` explicitly documents `maxRetries: 0` as the escape hatch for callers that "genuinely require unbounded retry" — but the *only* notification producer doesn't use it, and no host config raises the cap for the Notification category specifically. A central maintenance window longer than 25 minutes strands every notification raised during it. Compounding: the serial sweep (Performance #1) makes effective intervals much longer than 30 s under backlog, but retries are counted per sweep-attempt, so the wall-clock park horizon is at least honest — the operational cliff is the manual un-parking. + +### [Medium] Corrupt buffered notification is discarded and reported as delivered +`NotificationForwarder.DeliverAsync` returns `true` (→ engine removes the row) when the payload fails to deserialize (`NotificationForwarder.cs:79-87`). The message is permanently lost with only a site-log Warning: no parked row, no central `Notifications` row, no audit row. Returning `false` (permanent failure → **park**) would preserve the payload for forensics at zero protocol cost — parking is exactly the designed home for undeliverable-but-preserved messages. + +### [Low] Central heartbeat state is not replicated to the peer central node +`HandleSiteHealthReport` fans reports to the peer via DistributedPubSub (`CentralCommunicationActor.cs:360-373`), but `HandleHeartbeat` marks only the local aggregator (`CentralCommunicationActor.cs:349-353`). ClusterClient delivers each site's heartbeat to one central node; the other's aggregator sees none. After a central failover the new active node briefly shows all sites offline until fresh heartbeats arrive (≤ heartbeat interval + offline threshold). Minor, but inconsistent with the care taken for health reports. + +### [Low] `SubscribeInstance` concurrency-limit check is check-then-act +`SiteStreamGrpcServer.cs:243-254`: `_activeStreams.Count >= _maxConcurrentStreams` then insert — two concurrent subscribes at the boundary both pass. Bounded overshoot only; not worth locking, worth a comment. + +--- + +## Findings — Performance + +### [High] The retry sweep is serial, unbounded, and single-laned across categories — it collapses under backlog +Three compounding design choices: + +- `GetMessagesForRetryAsync` has **no LIMIT** — it loads every due row into memory each sweep (`StoreAndForwardStorage.cs:183-203`). +- `RetryPendingMessagesAsync` delivers strictly one-at-a-time (`StoreAndForwardService.cs:571-579`) while `_retryInProgress` blocks the next sweep (`StoreAndForwardService.cs:566`). +- Every notification forward attempt against a down central costs a full 30 s Ask timeout (`NotificationForwardTimeout`, `CommunicationOptions.cs:36`; `NotificationForwarder.cs:93-95`). + +Failure scenario: central is down for an hour and scripts have buffered 200 notifications. One sweep = 200 × 30 s ≈ 100 minutes of serial timeouts — during which no `ExternalSystem` or `CachedDbWrite` retries happen either (single lane, head-of-line blocking across categories and targets: one slow external system delays notification forwarding to a *healthy* central and vice versa). There is no per-target short-circuit — N messages to the same dead target incur N consecutive timeouts per sweep. The queue drains far slower than it fills; retry counts tick toward parking on a wildly distorted cadence. + +Fix shape: `LIMIT` on the due query; group by (category, target) and short-circuit remaining messages for a target after its first transient failure in a sweep; optionally parallelize across targets with a small cap. Any one of the three helps materially. + +### [Medium] No explicit Akka serializer configuration — every cross-cluster contract rides default reflective JSON +`BuildHocon` (`AkkaHostedService.cs:214-271`) configures remoting, SBR, and failure detectors, but **no** `akka.actor.serializers` / `serialization-bindings`. All command/control and telemetry traffic — deployment configs, `IngestAuditEventsCommand` batches (entities converted *from* compact proto DTOs back to POCOs before the Akka hop, `ClusterClientSiteAuditClient.cs:72-85`), `ReplicationOperation` with full message payloads — is serialized by Akka.NET's default Newtonsoft JSON serializer with embedded CLR type manifests. Costs: (a) hot-path CPU/allocation and payload bloat on the highest-volume channel (audit telemetry); (b) the documented "additive-only message contract evolution" rule (`CLAUDE.md`, Commons conventions) is only half the story — wire compatibility is also coupled to *type and namespace identity*, so a rename/move of a Commons message type breaks rolling cross-version central↔site communication with no test to catch it. `MessageContractTests` asserts correlation-id presence only (`tests/.../MessageContractTests.cs`), not serialized round-trip stability. Ironic detail: the telemetry path builds efficient proto DTOs, then converts them back to POCOs specifically to cross the ClusterClient hop. + +### [Low] S&F SQLite: no WAL, no explicit busy_timeout, connection-per-operation +Every `StoreAndForwardStorage` method opens a fresh `SqliteConnection` (e.g. `StoreAndForwardStorage.cs:137-139, 185-186`) with a bare `Data Source=` connection string (`StoreAndForward/ServiceCollectionExtensions.cs:22-24`) — default rollback-journal mode. Concurrent writers exist by design: script-thread enqueues, the sweep, replication applies (standby), and gRPC `PullSiteCalls`/parked-query reads. WAL + `busy_timeout` pragma would remove writer/reader blocking; Microsoft.Data.Sqlite's pooling makes connection-per-op tolerable but the pragmas are one line each in `InitializeAsync`. + +### [Low] Due-check does per-row `julianday()` string parsing on ISO-8601 timestamps +`StoreAndForwardStorage.cs:195-197` computes due-ness with `(julianday('now') - julianday(last_attempt_at)) * 86400000` against `"O"`-format strings with 7-digit fractional seconds and `+00:00` offsets — relying on SQLite's lenient datetime parser, non-indexable, and re-parsed for every Pending row on every 10 s sweep. Storing epoch milliseconds would be cheaper, indexable, and immune to format drift. + +### [Low] Audit-observer notification is awaited inline in the sweep +`NotifyCachedCallObserverAsync` is awaited per attempt inside `RetryMessageAsync` (`StoreAndForwardService.cs:620-626, 688-694, 711-717`). It's exception-isolated (good), but not latency-isolated: a slow observer (SQLite audit write) stretches the already-serial sweep. Fire-and-forget with isolation, or post to a channel, would decouple it. + +--- + +## Findings — Conventions + +### [Medium] Spec-vs-code drift in the S&F component doc +Two material divergences from `Component-StoreAndForward.md`: +1. **Standby-passive model** (doc lines 77-80) vs. the ungated standby delivery pipeline (Stability #2). The doc's "a few duplicate deliveries [on failover]... acceptable trade-off" framing materially understates actual behavior. +2. **`Notify.Send` latency**: the doc/CLAUDE.md say Send "returns a NotificationId status handle immediately", but `EnqueueAsync`'s immediate-delivery attempt (`StoreAndForwardService.cs:500-529`) runs the `NotificationForwarder` Ask inline on the script's await — when central is down, every `Notify.Send` blocks its script execution for up to `NotificationForwardTimeout` (30 s) before buffering. Either document the worst case or make the Notification category enqueue with `attemptImmediateDelivery: false` + an immediate sweep kick. + +### [Low] `StreamRelayActor.WriteToChannel` has a dead branch and silent loss +`TryWrite` on a bounded channel in `DropOldest` mode never returns `false` — it evicts the oldest item and succeeds. The "Channel full, dropping event" warning (`StreamRelayActor.cs:105-111`) can never fire, while the *actual* loss (the evicted oldest event) is completely unlogged. Lossy-under-backpressure is fine for debug view per spec (`Component-Communication.md:59`), but the observability is inverted: dead warning, silent real drop. Count evictions (channel `ItemDropped` callback via `CreateBounded(options, itemDropped)`) or log periodically. + +### [Low] Duplicated Ask-timeout constants and divergent fault semantics between the two ingest transports +`CentralCommunicationActor.DefaultAuditIngestAskTimeout` copies `SiteStreamGrpcServer.AuditIngestAskTimeout` by hand (`CentralCommunicationActor.cs:114-124` vs `SiteStreamGrpcServer.cs:45`), and the two handlers deliberately differ on fault behavior (propagate `Status.Failure` vs swallow-to-empty-ack — documented at `CentralCommunicationActor.cs:104-111`). Both are acknowledged in comments, but two transports for the same operation with different failure semantics and copy-pasted tuning is drift waiting to happen (see Underdeveloped, dual transports). + +### [Low] Heartbeat `IsActive` uses leader-check while singletons follow oldest-node +`SiteCommunicationActor.DefaultIsActiveCheck` reports active = cluster leader (`SiteCommunicationActor.cs:498-507`). Akka cluster singletons run on the *oldest* qualifying node; leader and oldest usually coincide in a 2-node stable cluster but can diverge transiently. The comment says it deliberately mirrors `ActiveNodeGate`, so it's a consistent repo convention — but the convention itself can misreport which node hosts the DeploymentManager singleton. Worth one shared helper with an oldest-member check. + +### [Low] `TransportHeartbeatInterval` does double duty +The Akka.Remote transport failure-detector interval (`BuildHocon`, `AkkaHostedService.cs:246-249`) is also the *application-level* site→central heartbeat cadence (`SiteCommunicationActor.cs:421-425`). These are unrelated concerns (spec treats them separately: `Component-Communication.md:268-274` vs health reporting); tuning one silently retunes the other. + +### Positives worth recording +- Tell/Ask discipline matches the repo rule everywhere: Ask only at system boundaries (`CommunicationService`, forwarder, telemetry clients), Tell/Forward on hot paths with Sender preservation for reply routing (`SiteCommunicationActor.cs:270-388`, `CentralCommunicationActor.cs:398-413`). +- Correlation IDs on every request/response contract, verified by tests. +- The at-least-once notification handoff is textbook: ack-after-persist at central, script-generated `NotificationId` as buffered-row id *and* idempotency key, re-stamped provenance at the forwarder (`NotificationForwarder.cs:123-160`). +- The sweep's CAS updates (`UpdateMessageIfStatusAsync`, `StoreAndForwardStorage.cs:243-269`) correctly close the operator-retry/discard vs sweep race, replicated with correct captured state (`StoreAndForwardService.cs:861-892`). +- gRPC server stream lifecycle is thorough: readiness + shutdown gating, correlation-id validation, duplicate-stream replacement, session-lifetime cap, leak-proofed Subscribe failure cleanup, own-entry-only removal (`SiteStreamGrpcServer.cs:211-327`). +- Stream-first debug subscription with buffered gap-window replay and per-entity timestamp dedup (`DebugStreamBridgeActor.cs:134-176, 286-329`) is a correct solution to the snapshot/stream ordering problem, including the flapping-stream retry-budget stability window. +- Additive proto evolution is real and tested (`ProtoContractTests`, `ProtoRoundtripTests`), with the manual regen process documented. + +--- + +## Underdeveloped Areas + +1. **Dual transports for the same ingest operations.** `IngestAuditEvents`/`IngestCachedTelemetry` exist both as gRPC unary RPCs (`SiteStreamGrpcServer.cs:330-433`) and as ClusterClient commands (`CentralCommunicationActor.cs:262-305`); production push uses ClusterClient (`ClusterClientSiteAuditClient`), leaving the gRPC ingest surface production-idle (doc admits: "the gRPC-receiving counterpart", `Component-Communication.md:87`). Two code paths, two fault semantics, double the test/maintenance surface — consolidation is pending and should be scheduled. +2. **No real-collaborator test for the ClusterClient cache lifecycle.** `DefaultSiteClientFactory` is untested; address-change recreation is only tested through mocks — precisely where the Critical finding lives. +3. **No test or design treatment of standby-node sweep behavior.** Neither the S&F tests nor an integration test asserts that only the active node delivers; the docker-cluster smoke topology would expose it with a two-node site and a delayed target. +4. **Replication has no anti-entropy/resync.** A standby down for an hour rejoins and applies only *new* operations; the missed delta diverges forever (until rows drain naturally). The doc covers only the "last few operations" loss case. A periodic checksum or full-row reconciliation (the pattern already exists for audit rows via `PullAuditEvents`) is absent for the S&F buffer itself. +5. **Parked rows have unbounded retention.** By design parked messages persist until operator action (`Component-StoreAndForward.md:110-116`), but there is no aging alert, cap, or archival — a forgotten site accumulates parked rows indefinitely with only a health-metric count. +6. **Reconciliation cursor edge.** `PullAuditEvents`/`PullSiteCalls` batch continuation relies on central advancing a `since_utc` cursor from the last returned timestamp (`SiteStreamGrpcServer.cs:478-484, 563-572`); rows sharing a boundary timestamp are protected only by EventId dedup / upsert-on-newer at central — fine, but the site-side read contract (`>` vs `>=`) isn't pinned by a test in this project. +7. **`DebugStreamService._sessions` on central failover** — sessions are process-local with no persistence (documented as acceptable: engineer re-establishes), but the SignalR consumer contract (`OnStreamTerminated`) is the only notification; a central restart drops sessions with no signal at all. Known/accepted, listed for completeness. + +--- + +## Prioritized Recommendations + +1. **[Critical] Fix ClusterClient recreation** (`CentralCommunicationActor.cs:514-521`, `DefaultSiteClientFactory`): unique per-incarnation actor names (suffix a counter) or Terminated-watch before re-create; validate/sanitize `siteId` for actor-name safety; wrap `Create` in the per-site try/catch; add a test using the real factory that edits a site's address twice. +2. **[High] Gate the S&F retry sweep to the active node** — check leader/oldest status at the top of `RetryPendingMessagesAsync` (or gate `StartAsync`'s timer), re-evaluated per tick so failover resumes delivery within one sweep interval. Add a two-node test. +3. **[High] Stop disposing shared gRPC clients on per-session failover** — key `SiteStreamGrpcClientFactory` by `(site, endpoint)` (allow both node channels concurrently), reserve dispose-and-replace for site removal/address edits driven by the address cache, not by individual bridge actors. +4. **[High] Bound and de-serialize the retry sweep** — `LIMIT` the due query, short-circuit per target after the first transient failure in a sweep, and consider a small parallelism cap across distinct targets; separately consider a much shorter forward timeout for the notification category or `attemptImmediateDelivery: false` on `Notify.Send` to unblock script threads. +5. **[Medium] Make replication trustworthy**: replace `Task.Run` fire-and-forget with an inline Tell (ordering), log failures at Warning with a counter metric, and make Park/Requeue applies upserts so a lost Add self-heals. +6. **[Medium] Pin the wire format**: configure an explicit serializer (or at least add serializer round-trip compatibility tests for the cross-cluster Commons contracts) so the additive-only rule is actually enforced end-to-end; long-term, stop converting proto DTOs back to POCOs for the ClusterClient hop. +7. **[Medium] Unstrand notifications**: pass `maxRetries: 0` (documented no-limit) or a category-specific high cap on the `Notify.Send` enqueue path, or auto-requeue parked Notification-category rows when central connectivity recovers. +8. **[Medium] Park (don't silently drop) corrupt notification payloads** in `NotificationForwarder.DeliverAsync` — return `false` instead of `true`. +9. **[Low] SQLite hygiene**: enable WAL + busy_timeout in `InitializeAsync`; migrate timestamps to epoch ms for the due-check. +10. **[Low] Observability polish**: count DropOldest evictions in `StreamRelayActor`; replicate heartbeat marks to the peer central node (or document the failover blind window); split the app heartbeat interval from the transport failure-detector setting. diff --git a/archreview/03-site-runtime-dcl.md b/archreview/03-site-runtime-dcl.md new file mode 100644 index 00000000..60494282 --- /dev/null +++ b/archreview/03-site-runtime-dcl.md @@ -0,0 +1,155 @@ +# Architecture Review 03 — Site Runtime & Data Connection Layer + +Reviewer domain: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime`, `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer`, `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging`, matched against `docs/requirements/Component-SiteRuntime.md`, `Component-DataConnectionLayer.md`, `Component-SiteEventLogging.md`, and `HighLevelReqs.md`. All findings are grounded in direct reads of the cited files. + +## Scope + +- **Site Runtime actors**: `DeploymentManagerActor` (1,720 LOC), `InstanceActor` (1,572), `ScriptActor` (594), `ScriptExecutionActor` (305), `AlarmActor` (780), `AlarmExecutionActor`, `NativeAlarmActor` (448), `CertStoreActor` (189), `SiteReplicationActor` (292), `SiteReconciliationActor`. +- **Script subsystem**: `ScriptCompilationService`, `SharedScriptLibrary`, `ScriptExecutionScheduler`, `ScriptRuntimeContext` (2,384 — skimmed for Ask/blocking patterns), trigger expressions. +- **Streaming**: `SiteStreamManager` (BroadcastHub + per-subscriber buffers). +- **Persistence**: `SiteStorageService` (site SQLite: deployed configs, static overrides, artifacts, `native_alarm_state`), `OperationTrackingStore` (not deep-read). +- **DCL**: `DataConnectionManagerActor` (345), `DataConnectionActor` (2,000), `OpcUaDataConnection` (401), `RealOpcUaClient` (1,313 — lifecycle sections), `MxGatewayDataConnection` (297 — connect path), alarm mappers, browse/search/verify paths. +- **Site Event Logging**: `SiteEventLogger` (326), purge/query services. +- **Tests**: inventoried `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests`, `...DataConnectionLayer.Tests`, `...SiteEventLogging.Tests`. + +## Maturity Verdict + +This is a mature, unusually carefully engineered subsystem — the Become/Stash state machine, in-flight subscribe bookkeeping, adapter-generation guards, seed-read retry, redeploy termination-watch buffering, and per-source native-alarm cap/eviction all show evidence of multiple hardening passes, and the code matches the design docs closely with extensive rationale comments and broad targeted test coverage. The residual risk is concentrated in three places: **adapter lifecycle on reconnect** (the previous OPC UA / MxGateway client object is silently abandoned on every `ConnectAsync`, leaking sessions and leaving a stale `ConnectionLost` handler that can flap a healthy connection), **script containment** (the execution timeout is cooperative-only, so a hung blocking script permanently consumes one of the 8 dedicated script threads; trigger expressions block actor threads on the default dispatcher), and **deployment honesty** (a site-side script compile failure does not reject the deployment as the spec requires — the instance starts with the script silently missing and central is told Success). None of these break the happy path; all of them are the kind of defect that surfaces weeks into production under flaky networks or a bad script. + +--- + +## Findings — Stability + +### S1. [High] Reconnect abandons the previous protocol client undisposed, and its stale `ConnectionLost` handler can flap a healthy connection +`OpcUaDataConnection.ConnectAsync` (`src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/OpcUaDataConnection.cs:99-101`) does: + +```csharp +_client = _clientFactory.Create(); +_client.ConnectionLost += OnClientConnectionLost; +await _client.ConnectAsync(...); +``` + +The **previous** `_client` is neither `DisposeAsync`'d nor detached. `DataConnectionActor`'s reconnect loop calls `_adapter.ConnectAsync` repeatedly on the *same adapter* (`DataConnectionActor.cs:537-547`); only the failover branch replaces the adapter. `MxGatewayDataConnection.ConnectAsync` has the same pattern (`MxGatewayDataConnection.cs:70-88`) and additionally overwrites `_eventLoopCts` without cancelling the previous event loop. + +Failure scenarios: +1. **Session/socket leak per disconnect cycle.** After a keep-alive-detected drop, the old `RealOpcUaClient` still holds its `ISession` + subscription (`RealOpcUaClient.cs:133-160`); `Session.CloseAsync` only runs from `DisconnectAsync` (`RealOpcUaClient.cs:392-396`), which nothing calls on this path. A site whose PLC link flaps hourly accumulates orphaned SDK session objects, keep-alive timers, and sockets indefinitely. +2. **Spurious disconnect of a healthy connection.** If the drop was detected *reactively* (a `ReadAsync` failure → `RaiseDisconnected`, `OpcUaDataConnection.cs:222-227`), the old client's own once-only guard (`_connectionLostFired`) has **not** fired. After a successful reconnect resets the adapter's `_disconnectFired` (`OpcUaDataConnection.cs:104`), the abandoned session's keep-alive eventually fails, fires the *old* client's `ConnectionLost`, which is still wired to `OnClientConnectionLost` → `RaiseDisconnected()` → `AdapterDisconnected` → `BecomeReconnecting()` on a perfectly healthy new session. Each such cycle abandons another client — self-sustaining reconnect churn. + +Fix shape: in both adapters' `ConnectAsync`, detach + dispose the old client before creating the new one (mirroring what the actor's failover branch already does at the adapter level, `DataConnectionActor.cs:421-431`). + +### S2. [High] Script execution timeout is cooperative-only; hung scripts permanently consume the bounded script-thread pool +`ScriptExecutionActor` runs the body on the dedicated `ScriptExecutionScheduler` (default **8 threads**, `SiteRuntimeOptions.cs:46`) and enforces the timeout via `CancellationTokenSource(timeout)` passed to `compiledScript.RunAsync` (`ScriptExecutionActor.cs:129, 246`). Roslyn only observes the token between statements; a script blocked in synchronous I/O (`Database.Connection` ADO.NET call, a socket in a wedged state) or a CPU loop never observes cancellation. The `OperationCanceledException` path replies/logs, but the *thread* is only released when the blocking call itself returns. `ScriptExecutionScheduler.WorkerLoop` (`ScriptExecutionScheduler.cs:71-84`) has no watchdog, no thread replacement, no "N tasks stuck > timeout" metric. + +Failure scenario: a site DB connection black-holes (firewall drops packets, no RST). Eight interval-triggered scripts each block ~30+ min in a synchronous read. All subsequent script *and alarm on-trigger* executions site-wide queue in `_queue` forever — alarms still transition (evaluation is on actor threads) but no on-trigger scripts run, and nothing on the health dashboard says why. The spec's claim that "Exceeding the timeout cancels the script" (`Component-SiteRuntime.md:411`) overstates what is enforced. + +Mitigations worth considering: a scheduler gauge (queue depth + busy-thread age) surfaced through `ISiteHealthCollector`; per-script-thread stuck detection that logs the script name; ADO.NET command timeouts injected into `Database.Connection`. + +### S3. [High] Spec drift: site-side script compile failure does not reject the deployment — central is told Success with the script silently missing +`Component-SiteRuntime.md:474-477`: "If script compilation fails when a deployment is received, the entire deployment for that instance is **rejected**. No partial state is applied. The failure is reported back to central as a failed deployment." + +Implementation: `InstanceActor.CreateChildActors` logs and `continue`s past a failed compile (`InstanceActor.cs:1362-1369`; same for alarm on-trigger scripts at 1418-1423 and trigger expressions at 1524-1531). The instance starts with the failing script absent; `DeploymentManagerActor.HandleDeployPersistenceResult` reports `DeploymentStatus.Success` purely on SQLite persistence (`DeploymentManagerActor.cs:596-610`). The XML doc on `CreateChildActors` even states the contradiction: "Compilation errors reject entire instance deployment (logged but actor still starts)" (`InstanceActor.cs:1339`). + +The design-time gate in Template Engine makes this rare, but the spec explicitly calls the site the last line ("Site-side compilation failures indicate an unexpected issue") — and version skew between central's Roslyn surface and the site's is exactly the case where the gates disagree. Today the operator sees a green deploy and a dead script discoverable only in the site event log. + +### S4. [Medium] Instance actor never retries a failed tag subscription; a subscribe/create ordering race strands attributes at Uncertain forever +`InstanceActor` registers `Receive(_ => { })` — an explicit no-op (`InstanceActor.cs:207`). If the DCL replies `Success=false` there is **no retry and no logging**. Two ways to get there: + +- `DataConnectionManagerActor.HandleRoute` replies "Unknown connection" when the `SubscribeTagsRequest` arrives before the `CreateConnectionCommand` has been processed (`DataConnectionManagerActor.cs:104-115`). The Deployment Manager Tells `CreateConnectionCommand` and then creates the Instance Actor (`DeploymentManagerActor.cs:543-546`), whose `PreStart` Tells `SubscribeTagsRequest` — but the two messages come from **different senders**, so Akka's per-sender-pair ordering guarantee does not formally order them at the manager's mailbox. If the race fires, the instance's data-sourced attributes stay Uncertain until redeploy/restart. +- The connection-level-failure reply ("connection unavailable — will re-subscribe on reconnect", `DataConnectionActor.cs:1009-1012`) is self-healing because `_subscriptionsByInstance` was already updated and `ReSubscribeAll` re-derives from it — that case is fine. The unknown-connection case leaves **no state anywhere**. + +Contrast: `NativeAlarmActor` retries a failed subscribe on a timer (`NativeAlarmActor.cs:269-286`). The tag path deserves the same treatment — at minimum, log the failure and schedule a bounded re-subscribe. + +Related sub-gap: `NativeAlarmActor`'s retry only arms on a *failure response*. A subscribe whose response is **lost** (DCL manager restarted between request and reply; message dead-lettered) never retries — `SendSubscribe` is a bare `Tell` with no response timeout (`NativeAlarmActor.cs:123-131`). Rehydrated SQLite state masks the gap in the debug view while the mirror is actually dead. + +### S5. [Medium] Deployment Manager startup aborts silently on a SQLite read failure — site runs zero instances with no retry +`HandleStartupConfigsLoaded`: on error, `_logger.LogError(...); return;` (`DeploymentManagerActor.cs:291-297`). One transient SQLite failure (locked file during a crash-recovery race, disk hiccup) at singleton start/failover leaves the active node with **no Instance Actors** until a human restarts the process. There is no retry timer, no health flag beyond the log line (health instance counts will read 0 deployed, which at least shows on the dashboard, but nothing distinguishes "genuinely nothing deployed" from "startup load failed"). A bounded retry (the same `Timers.StartSingleTimer` idiom used two methods later) would close this. + +### S6. [Medium] A throwing `InstanceActor` constructor leaves a dead ref in `_instanceActors` and still reports the deployment Success +`InstanceActor`'s constructor deserializes the config JSON (`InstanceActor.cs:138`) and builds the resolved-attribute index; a malformed/oversized config throws → `ActorInitializationException` → Deployment Manager's decider Stops the child (`DeploymentManagerActor.cs:268-274`). But: +- `CreateInstanceActor` already stored the ref in `_instanceActors` (`DeploymentManagerActor.cs:1628-1629`) and nothing watches for this failure mode (the `Context.Watch` only happens on the redeploy path), so the map holds a dead ref — debug view, inbound-API routes, etc. dead-letter with 30s Ask timeouts instead of "not found". +- `ApplyDeployment`'s persistence continues and `HandleDeployPersistenceResult` reports `Success` (`DeploymentManagerActor.cs:596-610`) — central shows a healthy deployment for an instance that is not running. + +A `Context.Watch` on every created child (removing from `_instanceActors` on unexpected `Terminated`) would fix both the stale map and give a hook to fail the pending deployment. + +### S7. [Medium] Subscribe background task reads the mutable `_adapter` field off-thread, violating its own confinement contract +`HandleSubscribe`'s `Task.Run` body awaits `_adapter.SubscribeAsync(...)` per tag and calls `SeedTagsAsync(_adapter, ...)` (`DataConnectionActor.cs:707-758`) — reading the actor field from a thread-pool thread, despite the comment eight lines earlier: "The background task below must NOT read or mutate actor state — these partitioned lists are the only state it sees" (`DataConnectionActor.cs:689-690`). If a failover swaps `_adapter` mid-loop (`CountFailureAndMaybeFailover`, `DataConnectionActor.cs:610-635`), later iterations subscribe on the **new** adapter while stamping the **old** generation — the resulting values are dropped by the generation guard and the monitored items are orphaned on the new adapter (never in `_subscriptionIds` for that generation… actually worse: `SubscribeCompleted` will store them against current state, mixing generations). `ReSubscribeAll` (`DataConnectionActor.cs:1596`) and `HandleWriteBatch`'s local async function (`DataConnectionActor.cs:1161-1201`) have the same field-read pattern. Capture the adapter into a local before spawning the task (as the re-seed block at `DataConnectionActor.cs:1620` correctly does with `reseedAdapter`). + +### S8. [Low] Site SQLite: connection-per-call, no WAL / busy-timeout tuning, many concurrent writers +`SiteStorageService` opens a fresh `SqliteConnection` per operation with a bare `Data Source=` string (`SiteStorageService.cs:34-44, 163-166`) — no `journal_mode=WAL`, relying on Microsoft.Data.Sqlite's default command-timeout-as-busy-handler. Concurrent writers against the same file include: deploy persistence, static-override fire-and-forget writes (`InstanceActor.cs:433-441`), per-transition native-alarm upserts (`NativeAlarmActor.cs:420-428`), replication applies (`SiteReplicationActor.cs:182-216`), and reconciliation. Under a native-alarm storm plus a deploy, rollback-journal locking serializes everything and a slow write can push others toward `SQLITE_BUSY`. WAL + explicit busy_timeout is a one-line hardening. (Contrast: `SiteEventLogger` got the full treatment — single owned connection, lock, bounded write queue.) + +### S9. [Low] Fire-and-forget adapter disposal in `PostStop` +`DataConnectionActor.PostStop` discards the `DisposeAsync` task (`DataConnectionActor.cs:218-221`); a hanging/faulting dispose (OPC UA `CloseAsync` against a dead server can block for the operation timeout) is unobserved. Acceptable trade-off (documented elsewhere in the file), but pair it with a `ContinueWith(OnlyOnFaulted, log)` like the other fire-and-forget paths in this codebase do. + +### S10. [Low] Alarm condition filter is last-subscriber-wins across co-subscribers of one source +`_alarmSourceFilter[request.SourceReference] = request.ConditionFilter` overwrites unconditionally (`DataConnectionActor.cs:104-108, 1763-1766`); two instances binding the same source with different filters silently re-gate each other's transitions. The field's XML doc honestly declares the sharp edge, but nothing *validates* agreement or logs the overwrite — a mismatch is undiagnosable in the field. Log a warning when the incoming filter differs from the stored one. + +--- + +## Findings — Performance + +### P1. [High] Trigger expressions block actor threads on the default dispatcher — per attribute change +`ScriptActor.EvaluateExpressionTrigger` runs the compiled Roslyn expression with `.GetAwaiter().GetResult()` under a 2-second CTS (`ScriptActor.cs:283-287`); `AlarmActor.EvaluateExpression` is identical (`AlarmActor.cs:503-506`). Both execute **on the actor's default-dispatcher thread**, and both run on *every* `AttributeValueChanged` the actor receives (Expression triggers have no monitored-attribute gate — `ScriptActor.cs:255-258`, `AlarmActor.cs:215-218`). A site with a handful of Expression-triggered scripts/alarms on a chatty instance (e.g., 50 tags at 1 Hz) synchronously executes 100s of Roslyn script invocations per second on dispatcher threads; one pathological expression (2 s timeout, treated-as-false) stalls its actor's mailbox and steals a dispatcher thread for the duration. The `ScriptExecutionScheduler` exists precisely to keep script code off these threads — trigger expressions bypass it. The in-code justification ("trusted Design-role users, compile-checked") addresses malice, not cost. + +### P2. [Medium] Attribute-change fan-out is broadcast to every child, filtering in the receiver +`InstanceActor.PublishAndNotifyChildren` Tells **every** Script Actor and **every** Alarm Actor for every attribute change (`InstanceActor.cs:1184-1194`); `Component-SiteRuntime.md:191` describes children subscribing "for the specific monitored attribute". An instance with 30 scripts + 20 alarms and 100 Hz aggregate tag updates generates 5,000 child messages/sec, ~98% of which are discarded by the child's trigger check (`ScriptActor.cs:229-233`, `AlarmActor.cs:219-222`) — pure mailbox/dispatch overhead, plus each `ScriptActor` maintains an `_attributeSnapshot` copy of every value (`ScriptActor.cs:227`). A per-attribute subscription map on the Instance Actor (attributeName → interested children; Expression-trigger children in an "all" bucket) would cut this by an order of magnitude on realistic configs. + +### P3. [Medium] DCL tag-value fan-out scans all instances per update +`HandleTagValueReceived` iterates `_subscriptionsByInstance` (every instance) with a per-instance `HashSet.Contains` for every incoming tag value (`DataConnectionActor.cs:1706-1716`). The actor already maintains `_tagSubscriberCount` (tag → count) for O(1) unsubscribe; the matching tag → *instances* reverse index is missing. With 500 instances on one connection at high sample rates this is the hottest loop in the DCL. Also `_healthCollector.UpdateTagQuality` is invoked on **every** update (`DataConnectionActor.cs:1737`) — cheap if the collector is a plain field write, but worth confirming it isn't lock-contended. + +### P4. [Low] Per-transition SQLite upserts on the native-alarm mirror +`NativeAlarmActor.PersistUpsert` issues one async SQLite write per accepted transition (`NativeAlarmActor.cs:420-428`), each opening a fresh connection (S8). An A&C alarm storm (hundreds of transitions/sec across sources) turns into hundreds of single-row upsert connections/sec. Bounded by `MirroredAlarmCapPerSource` (1000) per source in state size but not in write rate. Batching/coalescing (persist latest per SourceReference on a short timer) would be a proportionate hardening if storms are expected. + +### P5. [Low] Every script attribute read is an actor Ask round-trip +`ScriptRuntimeContext` Asks the Instance Actor per `GetAttribute` (`ScriptRuntimeContext.cs:375`) — a temp actor allocation + two mailbox hops per read. Correct (serialized through the source of truth) and consistent with the spec, but a script reading 50 attributes in a loop pays 50 round-trips; a `GetAttributes` batch message (the DeploymentManager routing path already fans out N Asks — `DeploymentManagerActor.cs:1360-1367` — which would also benefit) is an easy win if profiling ever flags it. + +### P6. [Low] Roslyn compiles run inside `InstanceActor.PreStart` on the default dispatcher during staggered startup +Each instance compiles all its scripts + alarm scripts + trigger expressions synchronously in `CreateChildActors` (called from `PreStart`, `InstanceActor.cs:250-255, 1348+`). The startup stagger (20 instances / 100 ms, `SiteRuntimeOptions.cs:13-19`) is tuned for OPC UA connection storms, not compile cost — 500 instances × several scripts each at ~50–300 ms/compile will occupy default-dispatcher threads for minutes after failover. The Deployment Manager already moved *shared*-script compilation off-thread (`DeploymentManagerActor.cs:1168-1191`); per-instance compiles did not get the same treatment. Failover time-to-recover is the metric at risk. + +--- + +## Findings — Conventions + +### C1. [Positive] Akka.NET discipline is exemplary +Worth stating because it is the dominant impression: Become/Stash exactly as the spec draws it (`DataConnectionActor.cs:232-533`); Tell on hot paths, Ask only at boundaries (`Component-SiteRuntime.md:430-442` ↔ implementation); `Sender`/`Self` captured before every continuation (e.g., `DeploymentManagerActor.cs:464-470`, `DataConnectionActor.cs:149-153`); all async results marshalled via `PipeTo`; supervision matches the spec table precisely — DM `OneForOne` Resume/Stop-on-init (`DeploymentManagerActor.cs:263-279`), Instance Resume (`InstanceActor.cs:281-293`), Script/Alarm coordinators Stop their execution children (`ScriptActor.cs:180-192`, `AlarmActor.cs:192-204`). The redeploy Terminated-watch buffering with the name-shadow map and its disable/delete supersede handling (`DeploymentManagerActor.cs:60-90, 389-435, 641-698, 762-831`) is genuinely hard concurrency done correctly. The in-flight subscribe/orphan-release guards in the DCL (`DataConnectionActor.cs:77-90, 836-985, 1795-1858`) close race classes most codebases never notice. + +### C2. [Low] Stale comment: SiteEventLogger claims its channel is unbounded +`SiteEventLogger.cs:216-218` — "The channel is unbounded, so the only way TryWrite fails is…" — the channel became bounded (`Channel.CreateBounded`, `SiteEventLogger.cs:74-88`), and `TryWrite` on a bounded DropOldest channel still only fails when completed, so the *behavior* is right but the stated reason is wrong. Trivial fix; flagged because this file's comments are otherwise load-bearing. + +### C3. [Low] Spec is internally inconsistent on the reconnect interval +`Component-DataConnectionLayer.md:303` says "The retry interval is defined **per data connection**", but the same document's Shared Settings table (`Component-DataConnectionLayer.md:170-177`) and the implementation (`DataConnectionOptions.ReconnectInterval`, one value for all connections — `DataConnectionActor.cs:463`) make it global. Fix the sentence at :303 (per the CLAUDE.md rule that the design doc travels with the code). + +### C4. [Low] Quality-only change on disconnect is not delivered to Alarm/Script actors — deliberate, but diverges from the spec's wording +`HandleConnectionQualityChanged` updates qualities and the stream but explicitly does **not** notify children ("firing scripts/alarms would cause spurious evaluations", `InstanceActor.cs:924-956`). The DCL spec promises "Instance Actors and their downstream consumers (**alarms, scripts checking quality**) see the staleness immediately" (`Component-DataConnectionLayer.md:302`). Practical consequence: an Active alarm computed from a now-dead connection stays Active (arguably correct — last-known-state), and a Conditional/Expression script never learns quality went Bad. The implementation choice is defensible; the spec sentence should be qualified so the divergence is a decision, not drift. + +### C5. [Low] Shared scripts execute without the caller's scope and without a `Scope` assignment +`SharedScriptLibrary.ExecuteAsync` builds `ScriptGlobals` without setting `Scope` (`SharedScriptLibrary.cs:98-103`), so a shared script always runs `ScriptScope.Root` (`ScriptCompilationService.cs:200-201`) even when invoked from a composed-module script. If a shared script uses `Attributes[...]`, it addresses **root** attributes regardless of caller scope. Possibly intended (shared scripts are instance-agnostic), but it is undocumented in `Component-SiteRuntime.md`'s Shared Script Library section and is a surprise waiting for a template author. + +### C6. [Low] `EvaluateCondition` fallback compares `value.ToString()` to `Threshold.ToString()` with current-culture double formatting +`ScriptActor.cs:474-477`: the non-numeric fallback stringifies the double threshold with the host culture (the numeric path was carefully invariant-culture'd; the fallback wasn't). A `"1.5"` threshold on a de-DE host renders `"1,5"`. Edge-case only (non-parseable values), but inconsistent with the deliberate invariant-culture work two lines up. + +--- + +## Underdeveloped Areas + +1. **Cert-trust convergence after node downtime.** The trust broadcast reaches only currently-Up nodes (`DeploymentManagerActor.BroadcastToSiteCertStores`, :962-1007); a node that is down during the broadcast rejoins with a divergent PKI store and there is no reconcile-on-join, no periodic store diff, and `ListServerCertsCommand` reads only the singleton's own node (:936-953) so the divergence is invisible from central. Central persistence of trust decisions is the documented follow-up (`Component-SiteRuntime.md:125`); until then a join-time pull from the peer's `CertStoreActor` would close the failover hole. +2. **Dead option**: `SiteRuntimeOptions.ConfigFetchRetryCount` — "Reserved — consumed by the standby replication fetch in a later task; not yet wired" (`SiteRuntimeOptions.cs:63-67`). The standby fetch is genuinely single-attempt (`SiteReplicationActor.cs:178-216`), leaning entirely on reconciliation as backstop. +3. **No aggregated live alarm stream** — the M7 operator Alarm Summary polls per-instance `DebugViewSnapshot` Asks; documented as a follow-up in CLAUDE.md/M7 plan. Fine for now; scale limit is the SemaphoreSlim-capped fan-out on the central side. +4. **Native alarm rehydration loses display metadata** — only `condition_json` + timestamps persist (`SiteStorageService` `native_alarm_state`, :117-124), so a rehydrated condition renders with empty type/category/message until the first source snapshot (`NativeAlarmActor.cs:161-167`). Acknowledged in comments; worth a line in the component doc's operator-facing section. +5. **No stuck-thread/queue-depth observability for the script scheduler** (see S2) — `ScriptExecutionScheduler` exposes nothing to `ISiteHealthCollector`. +6. **Tag-subscription failure path has no retry or site-event entry** (see S4) — the spec's Tag Path Resolution retry covers resolution failures inside the DCL, but the instance-side response handling is a hole. +7. **Tests**: coverage breadth is very good (redeploy races, waiter races, native-alarm swap/cap, seed reads, scheduler, trust model — 17 actor test files, 17 script test files, dedicated DCL alarm/browse/verify suites, plus `OpcUaAlarmLiveSmokeTests`). Gaps that match the findings above: no test exercises reconnect-then-stale-`ConnectionLost` (S1), compile-failure-deploy-status (S3), the unknown-connection subscribe race (S4), or sustained-churn fan-out cost (the `PerformanceTests` project was not examined for these paths). + +--- + +## Prioritized Recommendations + +1. **(S1)** Dispose + detach the previous client in `OpcUaDataConnection.ConnectAsync` / `MxGatewayDataConnection.ConnectAsync` (and cancel the old MxGateway event loop). This is the single highest-value fix: it removes both a per-flap resource leak and a healthy-connection flapping mechanism. +2. **(S3)** Make site-side compile failure fail the deployment, per spec: have `CreateChildActors` report compile outcomes to the Deployment Manager (or pre-compile in `ApplyDeployment` before actor creation, which also fixes P6 by moving compiles off the instance-start path) and return `DeploymentStatus.Failed` with the errors. +3. **(S2)** Add stuck-execution observability (scheduler queue depth + oldest-busy-thread age as health metrics; log the script name on timeout-with-thread-still-busy), and inject command timeouts into script-facing DB connections. +4. **(P1)** Move trigger-expression evaluation off the actor thread — evaluate on the `ScriptExecutionScheduler` and post the boolean result back to the actor (edge-state stays actor-confined), or precompile expressions to delegates rather than re-running Roslyn scripts. +5. **(S4)** Handle `SubscribeTagsResponse.Success == false` in `InstanceActor` with a logged, bounded re-subscribe timer (mirror `NativeAlarmActor`'s retry); add a response-timeout retry to `NativeAlarmActor.SendSubscribe`. +6. **(S5/S6)** Retry the startup config load on failure; `Context.Watch` all Instance Actors so an init-failure removes the dead ref and fails the in-flight deployment. +7. **(P2/P3)** Introduce per-attribute child-subscription routing in `InstanceActor` and a tag→instances reverse index in `DataConnectionActor` before scaling to the 500-instance design point. +8. **(S8)** Enable WAL + explicit busy timeout on the site SQLite stores. +9. **(S7)** Capture `_adapter` into locals before every background task in `DataConnectionActor` (three sites). +10. **(C3/C4/C5)** Doc fixes: reconcile the reconnect-interval wording, qualify the disconnect-quality-visibility promise, and document shared-script scope semantics — per the repo rule that design docs and code travel together. diff --git a/archreview/04-data-audit-backbone.md b/archreview/04-data-audit-backbone.md new file mode 100644 index 00000000..96383177 --- /dev/null +++ b/archreview/04-data-audit-backbone.md @@ -0,0 +1,180 @@ +# Architecture Review 04 — Central Data Layer & Audit/KPI Backbone + +**Scope reviewed:** + +- `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase` — EF Core mappings, repositories, migrations, `IAuditService`, partition maintenance +- `src/ZB.MOM.WW.ScadaBridge.Commons` — as the data-shape layer (entities, repository interfaces, audit enums/types, KPI types) +- `src/ZB.MOM.WW.ScadaBridge.AuditLog` — central ingest/purge/reconciliation actors, `CentralAuditWriter`, site SQLite writer + telemetry +- `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox` — `NotificationOutboxActor`, repository, delivery adapters, KPI source +- `src/ZB.MOM.WW.ScadaBridge.SiteCallAudit` — `SiteCallAuditActor`, monotonic upsert repository, reconciliation, relay +- `src/ZB.MOM.WW.ScadaBridge.KpiHistory` — `KpiHistoryRecorderActor`, `KpiHistoryRepository`, bucketer +- Design docs: `docs/requirements/Component-ConfigurationDatabase.md`, `Component-AuditLog.md`, `Component-NotificationOutbox.md`, `Component-SiteCallAudit.md`, `Component-KpiHistory.md`, `Component-Commons.md` +- Test projects: `AuditLog.Tests`, `NotificationOutbox.Tests`, `SiteCallAudit.Tests`, `KpiHistory.Tests`, `ConfigurationDatabase.Tests` + +## Maturity Verdict + +This is an unusually mature data layer for a system at this stage: idempotent ingest is implemented correctly and defensively everywhere (duplicate-key race handling in all three insert-if-not-exists paths), the combined cached-telemetry dual-write really is transactional (verified: `AuditLogIngestActor.OnCachedTelemetryAsync` shares one scoped `DbContext` across both repositories under `BeginTransactionAsync`), UTC discipline is enforced with converters and `SpecifyKind` re-tagging at every ADO.NET boundary, actor/DB interaction follows a consistent scope-per-message + PipeTo pattern with in-flight guards, and MSSQL-backed integration tests exist for the hard parts (partition purge, execution-tree walks, outage reconciliation). The weaknesses are concentrated in **scale-dependent operational behavior** rather than logic: the partition purge's full-table unique-index rebuild will start failing on default command timeouts as the table grows, several KPI predicates have no supporting index and full-scan year-sized tables every 60 seconds, the execution-tree query scans the entire AuditLog per invocation, the spec'd site SQLite 7-day retention purge simply does not exist, the append-only DB-role story is DDL-only (one connection principal runs both writer and maintenance paths), and the persisted `backlogTotal` KPI trend is hardwired to zero. None of these bites at demo scale; several will bite within months of production volume. + +--- + +## Findings — Stability + +### [High] Site SQLite 7-day retention purge is specified but not implemented — unbounded site DB growth + +`Component-AuditLog.md:412-414` specifies: "Sites: daily site job; default 7-day retention (configurable, min 1, max 90). Respects the hard `ForwardState` invariant." No such job exists. There is no `DELETE` against `audit_event` / `audit_forward_state` anywhere in `src/` — the only destructive statement is the one-time legacy-table `DROP TABLE IF EXISTS AuditLog` at startup (`src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriter.cs:198`). The writer's own comments treat the purge as future work: `SqliteAuditWriter.cs:145` ("a future `PRAGMA incremental_vacuum` shrink the file after the 7-day retention purge") and `:33` claims the store is "recreated per deployment", which is not a retention mechanism. + +**Failure scenario:** a long-lived site node accumulates every Forwarded/Reconciled audit row forever. At even modest script activity (10 events/s ≈ 860k rows/day) the SQLite file grows by GBs per month; `GetBacklogStatsAsync`'s `COUNT(*)` stays cheap (it filters `Pending`), so nothing surfaces on the health dashboard — the disk just fills. The `auto_vacuum = INCREMENTAL` pragma (`SqliteAuditWriter.cs:149`) is set up for a purge that never runs. + +### [High] `SwitchOutPartitionAsync` — full-table unique-index rebuild inside one transaction, no command timeout + +`src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/AuditLogRepository.cs:257-332`: the monthly partition switch drops `UX_AuditLog_EventId`, switches, then `CREATE UNIQUE NONCLUSTERED INDEX UX_AuditLog_EventId ON dbo.AuditLog (EventId)` — a scan+sort over **every remaining row of the table** (up to 365 days of audit events), all inside a single transaction executed via `ExecuteSqlRawAsync`. + +Three compounding problems: + +1. **Default 30s command timeout.** No `CommandTimeout` is configured anywhere (`ServiceCollectionExtensions.cs:29-34` sets only `UseSqlServer(connectionString)`; repo-wide grep confirms no `CommandTimeout` on the data layer). Once the table is large enough that the index rebuild exceeds 30 seconds, the batch times out, the CATCH block rolls back and re-`THROW`s, `AuditLogPurgeActor.OnTickAsync` logs and moves on (`AuditLogPurgeActor.cs:198-206`) — and the same failure repeats every daily tick forever. Retention silently stops; the table grows past 365 days; the rebuild gets *slower* each day. This is a self-reinforcing failure with no alerting beyond an error log line. +2. **Write blockage.** The transaction holds Sch-M from the DROP INDEX through the CREATE INDEX; all ingest (`InsertIfNotExistsAsync`, dual-writes, direct writes) blocks for the full rebuild duration. The actors tolerate this (rows stay Pending at sites), but central direct-write paths (Inbound API middleware, dispatcher audit emission) will eat the latency or the swallow-and-count path. +3. **Uniqueness window.** While `UX_AuditLog_EventId` is absent, the `IF NOT EXISTS` probe becomes an unindexed scan and single-column EventId uniqueness is enforced only via the composite PK. In practice a retried event carries the same `OccurredAtUtc` so the PK still dedupes; the residual risk is small but the per-insert scan cost during the window is not. + +**Recommendation:** set a long `CommandTimeout` for the maintenance path (or run the rebuild with `ONLINE = ON` where edition allows, outside the switch transaction), and monitor purge success as a health metric, not just a log line. Long term, consider making `UX_AuditLog_EventId` unnecessary (see the clustered-key finding under Performance). + +### [High] Append-only DB-role enforcement is DDL-only — one principal runs writer and maintenance paths + +The migrations create `scadabridge_audit_writer` (INSERT+SELECT, DENY UPDATE/DELETE) and `scadabridge_audit_purger` (SELECT + ALTER ON SCHEMA::dbo) — `20260520142214_AddAuditLogTable.cs:146-158`, re-granted in `20260602174346_CollapseAuditLogToCanonical.cs:212-223`. But the application registers **one** `ScadaBridgeDbContext` with **one** connection string (`ServiceCollectionExtensions.cs:29-45`); the same principal executes `InsertIfNotExistsAsync` (writer path), `PurgeChannelOlderThanAsync`'s `DELETE TOP` (`AuditLogRepository.cs:409-410`), `BackfillSourceNodeAsync`'s `UPDATE TOP` (`AuditLogRepository.cs:851-852`), and the switch-out DDL. Consequences: + +- If the runtime login were actually placed in `scadabridge_audit_writer`, the DENY would break the purge/backfill on the same connection. So in any real deployment the login must hold INSERT+UPDATE+DELETE+DDL — meaning the append-only guarantee is enforced **only** by the CI grep guard (`tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/AuditLogAppendOnlyGuardTests.cs`), not by the database. +- The purger role as granted cannot even perform the purge: `CREATE TABLE` (staging) requires the database-level `CREATE TABLE` permission, which `ALTER ON SCHEMA::dbo` alone does not confer. + +`Component-AuditLog.md:417-424` presents the role separation as the enforcement mechanism ("The application accesses AuditLog via a dedicated DB role"). Spec and code have drifted: either wire a second connection string/principal for the maintenance path (and fix the purger grants), or re-document enforcement honestly as "CI guard + code review, roles provided for DBAs who choose to segregate principals." + +### [Medium] SiteCalls monotonic upsert drops mid-lifecycle progress (RetryCount / LastError frozen at rank) + +`SiteCallAuditRepository.cs:30-40` ranks `Attempted == Skipped == 2` and the UPDATE fires only when `incomingRank > storedRank` (`:128-138`). A cached call retrying 20 times emits 20 `Attempted` telemetry packets with increasing `RetryCount` and fresher `LastError`/`HttpStatus` — every one after the first is a **no-op** ("first-write-wins at each rank" per the comment). The Central UI Site Calls page therefore shows `RetryCount = 1` and the first error for the entire retry phase, until the terminal packet lands. The spec's intent (`Component-SiteCallAudit.md:76-78`, "upsert-on-newer-status... at-least-once and out-of-order telemetry are harmless") is about status regression, not about freezing progress fields. A safe refinement: within equal rank, apply the update when `incoming.UpdatedAtUtc > stored.UpdatedAtUtc` (still idempotent and regression-proof). + +### [Medium] No EF Core connection resiliency (`EnableRetryOnFailure`) + +`ServiceCollectionExtensions.cs:31` configures `UseSqlServer(connectionString)` with no execution strategy. The audit/dispatcher actors compensate with their own retry loops (rows stay Pending, next tick retries), so the backbone self-heals — but every *read-side* path (KPI requests, outbox queries, `Notify.Status` round-trips, execution-tree queries) surfaces transient faults (failover of the SQL instance, connection pool churn) directly to the caller as a failed response. One line of `sqlOptions.EnableRetryOnFailure(...)` would cover the fleet. (Note: retry strategies and user-initiated transactions interact — `OnCachedTelemetryAsync`'s `BeginTransactionAsync` would need `CreateExecutionStrategy` wrapping — so adopt deliberately.) + +### [Medium] Operator Retry on a parked notification emits no audit row + +`Component-NotificationOutbox.md:128`: "Operator Retry/Discard still mutates only the Notifications row, and each transition emits the corresponding `Notification.Attempt` / `Notification.Terminal` audit row." `DiscardAsync` does emit (`NotificationOutboxActor.cs:1021`), but `RetryAsync` (`NotificationOutboxActor.cs:951-977`) resets Parked→Pending with **no** audit emission. Forensically, a parked notification that an operator retried and that then delivered shows an unexplained `Parked → Attempted → Delivered` sequence with no record of who un-parked it or when. Same gap for `SiteCallAudit`'s retry/discard relay (site-side telemetry covers the state change but not the operator identity). + +### [Medium] Reconciliation edge cases are detected but dead-ended + +- **Single-timestamp cursor pin** (`SiteCallAuditActor.cs:636-654`): when > `ReconciliationBatchSize` rows share one `UpdatedAtUtc` the cursor cannot advance; the code correctly detects and logs it, but the backlog tail then **never** reconciles ("will not reconcile until those rows' timestamps differ") and nothing surfaces beyond a log Warning — no health metric, no stalled event (unlike the sibling `SiteAuditReconciliationActor`, which publishes `SiteAuditTelemetryStalledChanged`). The fix the log message itself names — a composite `(timestamp, id)` keyset in the pull contract — is deferred without a tracking artifact. +- **Permanent row abandonment** (`SiteAuditReconciliationActor.cs:283-293`): after 5 failed inserts a row is dropped with `LogCritical` and the cursor advances. For a compliance-oriented audit log, "we permanently lost EventId X" deserves a durable record (e.g., a synthetic `Skipped` audit row or a health counter), not only a log line that rotates away. + +### [Medium] `NotificationOutboxActor` ingest ack path can ack before the row is *committed* under a SQLite test provider mismatch — verify provider parity + +`NotificationOutboxRepository.InsertIfNotExistsAsync` maintains two dialects (`NotificationOutboxRepository.cs:70-95` SQLite `INSERT OR IGNORE`, `:99-117` T-SQL `IF NOT EXISTS`). This is correct today but is production code carrying a test-only branch, and the two branches have subtly different concurrency semantics (the SQLite path can never observe the duplicate-key race the SQL Server path handles). The AuditLog and SiteCall repositories chose the opposite strategy (MSSQL-only SQL, MSSQL-backed test fixtures). Pick one convention; the divergence invites the next repository to guess. + +### [Low] Purge actors' first tick waits a full interval + +`AuditLogPurgeActor.PreStart` (`:95-100`), `SiteCallAuditActor.StartPurgeTimer` (`:380-386`), and `KpiHistoryRecorderActor` purge timer (`KpiHistoryRecorderActor.cs:124-127`) all use `initialDelay = interval` (24h). A central node that restarts daily (crash loop, scheduled recycling) would *never* purge. Low likelihood, cheap fix (short initial delay; purges are idempotent). + +### [Low] `SqliteAuditWriter.Dispose` sync-over-async + +`SqliteAuditWriter.cs:890-894` blocks on `DisposeAsync` via `Task.Run(...).GetAwaiter().GetResult()`. The mitigation (thread-pool hop) is correct and documented; flagged only because it remains a blocking dispose on a hot-path component. + +--- + +## Findings — Performance + +### [High] SiteCalls KPI predicates full-scan the table — no index on `TerminalAtUtc` + +The "schema-honest" non-terminal predicate is `TerminalAtUtc IS NULL` (`SiteCallAuditRepository.cs:224-232`), used by `ComputeKpisAsync` (buffered `:243-244`, stuck `:259-260`, oldest `:262-268`), the per-site and per-node variants (`:286-307`, `:336-362`), and `PurgeTerminalAsync` (`:213-218`). But the table's only indexes are `(SourceSite, CreatedAtUtc)` and `(Status, UpdatedAtUtc)` (`SiteCallEntityTypeConfiguration.cs:79-87`) — **nothing covers `TerminalAtUtc`**. Every such count is a clustered scan of a table retained for 365 days (`SiteCallAuditOptions.RetentionDays = 365`, potentially millions of rows for busy sites). These queries run: + +- every 60 s from `SiteCallAuditKpiSampleSource` (global + per-site + per-node = ~9 scanning queries per recorder tick), +- on every Health-dashboard poll (10 s cadence per the UI design) via `SiteCallKpiRequest`. + +**Failure scenario:** at ~2M rows, each scan is hundreds of ms of CPU/IO; the KPI backbone alone drives a near-continuous scan load on the central DB, and the KPI Ask timeouts start failing intermittently. **Fix:** a filtered index `CREATE INDEX IX_SiteCalls_NonTerminal ON SiteCalls (CreatedAtUtc) WHERE TerminalAtUtc IS NULL` (plus `SourceSite`/`SourceNode` includes) makes every one of these predicates a small seek, since the non-terminal population is the live queue, not the archive. + +### [High] `GetExecutionTreeAsync` scans the entire AuditLog per invocation + +`AuditLogRepository.cs:704-756`: the recursive CTE's `Edges` anchor is `SELECT DISTINCT ExecutionId, ParentExecutionId FROM dbo.AuditLog WHERE ExecutionId IS NOT NULL`. Neither `IX_AuditLog_Execution` nor `IX_AuditLog_ParentExecution` covers the *other* column, so satisfying the DISTINCT requires scanning an entire index (or the clustered index) across **all partitions, all 365 days** — on every `audit tree` CLI call and every UI execution-tree drilldown. The upward walk (`:652-677`) is fine (seeks on `IX_AuditLog_Execution`), and the STRING_AGG subqueries seek per node; it is the edge materialization that is O(table). + +**Fix options:** (a) add `ParentExecutionId` as an INCLUDE on `IX_AuditLog_Execution` (index-only DISTINCT scan — better but still O(table)); (b) constrain the Edges CTE to a time window derived from the root's `FirstOccurredAtUtc` (execution trees span minutes, not years — a `OccurredAtUtc >= root - slack` predicate gets partition elimination); (c) seed the recursion by seeking children per level (`WHERE ParentExecutionId IN (…)`) instead of pre-materializing a global edge set. + +### [Medium] AuditLog clustered key leads with a random GUID + +`AuditLogEntityTypeConfiguration.cs:172` / migration `CollapseAuditLogToCanonical.cs:75`: clustered PK `(EventId, OccurredAtUtc)` with `EventId = Guid.NewGuid()`. Inserts land randomly across the current month's partition → page splits, fragmentation, and larger-than-necessary buffer pool footprint on the system's highest-volume table. The composite order also means the PK is useless for time-range queries (all served by secondary indexes). A clustered key of `(OccurredAtUtc, EventId)` would give append-locality and make `IX_AuditLog_OccurredAtUtc` redundant; `EventId` uniqueness would still need the non-aligned unique index (which exists anyway). This interacts with the switch-out dance either way; worth a deliberate benchmark before the table gets big — changing it later is a full rebuild. + +### [Medium] KpiSample volume and unbatched purge + +Per recorder tick (60 s), sample count ≈ NotificationOutbox(5 × (1 global + N sites + M nodes)) + SiteCallAudit(6 × (1 + N + M)) + AuditLog(3) + SiteHealth(12 × N). For 10 sites / 20 site-nodes this is ~500 rows/min ≈ **700k rows/day ≈ 60M+ rows at the 90-day default retention** — with a 5-column composite index (`IX_KpiSample_Series`) maintained on every insert. Meanwhile `PurgeOlderThanAsync` is a single unbatched `ExecuteDeleteAsync` (`KpiHistoryRepository.cs:65-71`): in steady state it deletes ~700k rows in **one transaction** daily — a lock/log spike on the same DB serving the operational tables; after any purge outage the catch-up delete is worse (and subject to the same default 30s command timeout as the partition switch). Batch the delete (`TOP (n)` loop, mirroring `PurgeChannelOlderThanAsync`), and consider whether per-node scope sampling every 60 s is worth 40% of the row volume. + +### [Medium] Catch-all `(@p IS NULL OR col = @p)` predicates in SiteCalls query + +`SiteCallAuditRepository.QueryAsync` (`:186-202`) builds one static SQL shape where every filter is `({param} IS NULL OR col = {param})`. This is the classic optional-parameter anti-pattern: SQL Server compiles one plan for all filter combinations, generally a scan ordered by `CreatedAtUtc DESC`. Fine while the table is small; on a year of rows, a filtered query (e.g. `Status = 'Parked'`) will scan rather than use `IX_SiteCalls_Status_Updated`. `OPTION (RECOMPILE)` or dynamic predicate composition (as `AuditLogRepository.QueryAsync` does with LINQ, `:117-199`) fixes it. + +### [Medium] Notification outbox query page uses offset paging + double query + unsargable LIKE + +`NotificationOutboxRepository.QueryAsync` (`:165-228`): `COUNT` + `Skip((page-1)*size)` — deep pages re-scan; `Subject.Contains(keyword)` translates to `LIKE '%kw%'` (unsargable). The sibling AuditLog and SiteCalls pages both use keyset paging. For a 365-day table this page will degrade first in the UI. Also `ComputeKpisAsync`/`ComputePerSiteKpisAsync`/`ComputePerNodeKpisAsync` issue 5-7 sequential round trips each (`:239-283`, `:286-334`, `:337-391`), and the per-site/per-node "oldest" computations materialize every non-terminal row into memory (`:312-318`, `:368-375`) — bounded by live-queue size in normal operation, unbounded during a delivery outage (exactly when the KPI page gets watched). A single `GROUP BY` with conditional aggregation would serve each snapshot in one query. + +### [Low] Non-persisted `IngestedAtUtc` computed column re-parses `DetailsJson` per row read + +`AuditLogEntityTypeConfiguration.cs:158-164`: every materialization of `AuditLogRow` (e.g. `QueryAsync` pages) evaluates `SWITCHOFFSET(CAST(JSON_VALUE(DetailsJson …)))` over an `nvarchar(max)` per row. Page-sized reads make this tolerable; just avoid ever using `IngestedAtUtc` in a predicate (it would force a full-table JSON parse). + +### [Low] `MarkForwardedAsync` / `MarkReconciledAsync` build per-id IN lists + +`SqliteAuditWriter.cs:644-664, 735-753`: parameter-per-id is safe at the default telemetry batch of 256 (`SiteAuditTelemetryOptions.BatchSize = 256`) but silently approaches SQLite's 999-parameter limit if an operator raises the batch size ~4×. Worth a guard or chunking. + +--- + +## Findings — Conventions + +### [Medium] Design docs have drifted from the code in several load-bearing places + +- `Component-Commons.md:47` lists `AuditKind` with 10 values; the code has 14 (`Commons/Types/Enums/AuditKind.cs:10-26`, adds the four `SecuredWrite*` kinds). `:46` lists `AuditChannel` with 4 values; the code has 5 (`AuditChannel.cs:8-15`, adds `SecuredWrite`). +- `Component-Commons.md:89` still says `NotificationType`: "Email — currently the only value; Teams and other channels are planned but not yet present" — `Sms` shipped 2026-06-19 and the Teams plan was dropped (per CLAUDE.md). +- `Component-ConfigurationDatabase.md:58` describes the NotificationList `Type` discriminator as "`Email` / `Teams` / …". +- `Component-ConfigurationDatabase.md:399` states AuditLog retention is a "single global value in v1, **no per-channel overrides**", directly contradicting the shipped `PerChannelRetentionDays` feature (and `Component-AuditLog.md:396-411`, which documents it). +- `Component-SiteCallAudit.md:44-51` describes the `SiteCalls` schema with `Kind`/`TargetSummary`/`Provenance` columns and a `Pending/Retrying/...` status set; the shipped schema (`Component-ConfigurationDatabase.md:64` and `SiteCallEntityTypeConfiguration.cs`) uses `Channel`/`Target` and `AuditStatus`-derived strings with no instance/script provenance columns. The ConfigDb doc is current; the SiteCallAudit doc's own table section is stale. + +Per the repo's own rule ("design doc and code travel together"), these should be reconciled in one pass. + +### [Medium] `AuditLogRow` entity lives in ConfigurationDatabase, not Commons + +`src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Entities/AuditLogRow.cs` breaks the "POCO entities in Commons / mappings in ConfigurationDatabase" convention that every sibling table follows (`SiteCall`, `Notification`, `KpiSample`, `AuditLogEntry` are all Commons POCOs). There is a defensible rationale — the *contract* type is the external `ZB.MOM.WW.Audit.AuditEvent` and `AuditLogRow` is a persistence-only projection — but the deviation is undocumented in `Component-Commons.md`/`Component-ConfigurationDatabase.md`, and `IAuditLogRepository` (in Commons) consequently traffics in `AuditEvent` while `ISiteCallAuditRepository` traffics in a Commons entity. Document the deviation or move the row shape. + +### [Low] Mixed timestamp CLR types across sibling tables + +`Notification` uses `DateTimeOffset` (`Commons/Entities/Notifications/Notification.cs:68-77`) while `SiteCall`, `KpiSample`, and `AuditLogRow` use UTC `DateTime`. Both satisfy the UTC convention, but the `DateTimeOffset` choice is what forced the awkward in-memory Min reductions and "converter makes SQL Min awkward" workarounds in `NotificationOutboxRepository` (`:262-274`, `:308-318`) that its `DateTime`-based siblings didn't need. New tables should standardize (the `DateTime`+`Utc`-converter pattern in `AuditLogEntityTypeConfiguration.cs:32-40` is the cleaner precedent). + +### [Low] Uncharted KPI metric names are string literals while charted ones use the catalog + +`NotificationOutboxKpiSampleSource.cs:40-43`: `"stuckCount"` / `"oldestPendingAgeSeconds"` are inline literals with a comment explaining the split, while charted metrics use `KpiMetrics.*` constants. Consistent cataloging (all persisted names in `KpiMetrics`) would remove a rename hazard on values that are, per the code's own comment, "persisted; do not rename." + +### Positive observations (conventions done right) + +- Options pattern with dedicated validators and clamped intervals everywhere (`SiteCallAuditOptions.ResolvedReconciliationInterval` guarding the Akka zero-interval spin footgun, `AuditLogOptionsValidator`, `KpiHistoryOptionsValidator`). +- The `Sender`-capture-before-await and `Context.System.EventStream`-capture-before-await disciplines are applied consistently across all five actors — a subtle Akka correctness point handled uniformly. +- Duplicate-key race hardening (2601/2627 swallow) is uniform across all three idempotent-insert repositories, with the check-then-act window explicitly documented at each site. +- `AuditService` stages within the caller's unit of work exactly as the spec's transactional-audit guarantee requires, with cycle-tolerant serialization that can't roll back the business operation (`AuditService.cs:20-24, 71-85`). +- The dual-write transaction (`AuditLogIngestActor.cs:262-311`) is genuinely atomic per entry with per-entry isolation, and `IngestedAtUtc` is stamped from one instant across both rows. + +--- + +## Underdeveloped Areas + +1. **`backlogTotal` KPI history series is permanently zero.** `AuditLogRepository.GetKpiSnapshotAsync` hardcodes `BacklogTotal: 0L` (`AuditLogRepository.cs:618`) — documented in the interface as "left at zero" — and the Central UI's live tile compensates by summing site backlog reports (`CentralUI/Services/AuditLogQueryService.cs:125-155`). But `AuditLogKpiSampleSource` persists the repo snapshot **directly** (`AuditLogKpiSampleSource.cs:69-85`), so every recorded `backlogTotal` sample is 0, and the Audit Log page's backlog trend chart (`AuditLogPage.razor.cs:297`) renders a flat zero line regardless of actual backlog. The one KPI whose *history* matters most during an outage post-mortem (how big did the backlog get, how fast did it drain) records nothing. Fix: inject the same health-aggregator read the query service uses into the sample source. +2. **Site SQLite retention purge** — spec'd, commented as future, absent (see Stability [High] above). +3. **Hash-chain tamper evidence (T1) and Parquet archival (T2)** — explicitly deferred to v1.x per `Component-AuditLog.md:433-437`; the `verify-chain` CLI is a documented no-op placeholder. Consistent with plan; no drift. +4. **SecuredWrite audit rows leave `SourceNode` NULL** — documented as a logged follow-up (`Component-AuditLog.md:154-157`); also note `PerChannelRetentionDays` validation accepts `SecuredWrite` as a key (the enum has it) even though the design doc's config section (`:396-399`) enumerates only the four original channels — harmless, but the doc under-promises. +5. **Reconciliation cursor keyset upgrade** — the `(timestamp, id)` composite cursor that would eliminate the single-timestamp pin is acknowledged in a log message (`SiteCallAuditActor.cs:640-654`) but not tracked anywhere; the pull contracts (`CachedCallReconcileRequest`, `PullAuditEvents`) would need an additive field. +6. **Dispatcher throughput ceiling** — notification delivery is strictly sequential within a sweep (batch 100, 10 s interval, blocking SMTP/Twilio per row: `NotificationOutboxActor.RunDispatchPass:321-347`). A slow SMTP endpoint (5 s/attempt) caps throughput at ~1 notification/5 s regardless of batch size, and the in-flight guard means subsequent ticks drop. The spec's "dedicated blocking-I/O dispatcher" (`Component-NotificationOutbox.md:23`) is not literally implemented (delivery runs on the thread pool off the actor thread — functionally acceptable, doc drift), but there is no per-notification parallelism at all. Fine for alarm-notification volume; a burst (alarm storm fanning to hundreds of notifications) drains slowly. +7. **Test coverage** is strong overall (MSSQL-backed integration tests for partition purge, execution trees, outage reconciliation, dual-write; per-actor TestKit suites; options validators tested). Gaps mirror the findings: no test asserts a site-SQLite retention purge (it can't — the feature is missing), no test exercises `SwitchOutPartitionAsync` at a size that would expose the command-timeout failure, and no test asserts `backlogTotal` history is non-zero under backlog. + +--- + +## Prioritized Recommendations + +1. **Implement the site SQLite 7-day retention purge** (daily timer on `SiteAuditTelemetryActor` or a dedicated job; `DELETE` joined on `audit_forward_state.ForwardState IN ('Forwarded','Reconciled')` + age, followed by `PRAGMA incremental_vacuum`) — closes the unbounded-growth hole and honors the spec'd `ForwardState` invariant. +2. **Fix the `backlogTotal` KPI sample source** to read the health aggregator like `AuditLogQueryService` does — one-file change, restores a shipped feature. +3. **Harden `SwitchOutPartitionAsync` operationally**: explicit long `CommandTimeout` on the maintenance commands, a health metric/event on purge *failure* (the success event exists; the failure path is log-only), and an integration test at representative volume. +4. **Add the filtered non-terminal index on `SiteCalls`** (`WHERE TerminalAtUtc IS NULL`) and turn each KPI snapshot into a single conditional-aggregation query — removes the dominant recurring scan load. +5. **Bound the `GetExecutionTreeAsync` edge scan** with a time-window predicate (partition elimination) or per-level child seeks. +6. **Decide the DB-role story honestly**: either a second maintenance connection string/principal (and grant `CREATE TABLE` to the purger), or update `Component-AuditLog.md`/`Component-ConfigurationDatabase.md` to state that append-only enforcement is CI-guard-based with roles offered as an optional DBA hardening. +7. **Allow same-rank freshness updates in the SiteCalls upsert** (`UpdatedAtUtc` tiebreaker) so retrying calls show live RetryCount/LastError. +8. **Batch the KpiSample purge** and reconsider per-node sampling cadence before fleet size multiplies row volume. +9. **Emit an audit row for operator Retry** on parked notifications (and consider operator identity on the SiteCalls relay), matching the spec's transition-audit promise. +10. **Reconcile the stale design-doc passages** (AuditKind/AuditChannel/NotificationType enum lists, `Teams` references, "no per-channel overrides", SiteCalls schema section) in one documentation pass, per the repo's docs-and-code-travel-together rule. diff --git a/archreview/05-templates-deployment-transport.md b/archreview/05-templates-deployment-transport.md new file mode 100644 index 00000000..795e4c86 --- /dev/null +++ b/archreview/05-templates-deployment-transport.md @@ -0,0 +1,145 @@ +# Architecture Review 05 — Design-Time Pipeline: Templates, Deployment, Transport, Script Analysis + +Reviewer domain: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine`, `src/ZB.MOM.WW.ScadaBridge.DeploymentManager`, `src/ZB.MOM.WW.ScadaBridge.Transport`, `src/ZB.MOM.WW.ScadaBridge.ScriptAnalysis`, their design docs (`docs/requirements/Component-TemplateEngine.md`, `Component-Transport.md`, `Component-ScriptAnalysis.md`, `Component-DeploymentManager.md`) and test projects. All findings cite code actually read. + +## Scope + +- **Template Engine**: flattening (`Flattening/FlatteningService.cs`), collision detection (`CollisionDetector.cs`), acyclicity (`CycleDetector.cs`), revision hashing (`Flattening/RevisionHashService.cs`), lock rules (`LockEnforcer.cs`), diffs (`Flattening/DiffService.cs`), validation (`Validation/ValidationService.cs`, `Validation/SemanticValidator.cs`, `Validation/ScriptCompiler.cs`), authoring services (`TemplateService.cs`, `TemplateInheritanceResolver.cs`). +- **Deployment Manager**: `DeploymentService.cs` (deploy/lifecycle pipeline, idempotency reconciliation), `OperationLockManager.cs`, `FlatteningPipeline.cs`, `StateTransitionValidator.cs`, `StaleInstanceProbe.cs`, `ArtifactDeploymentService.cs`. +- **Transport**: `Import/BundleImporter.cs` (3,832 lines — load/preview/apply), `Export/BundleExporter.cs`, `Export/DependencyResolver.cs` (skimmed), `Serialization/EntitySerializer.cs`/`EntityDtos.cs`, `Encryption/BundleSecretEncryptor.cs` + `BundleManifestAad.cs`, `Import/LineDiffer.cs`, `Import/ArtifactDiff.cs`, `Import/BundleSessionStore.cs`, `TransportOptions.cs`. +- **Script Analysis**: `ScriptTrustPolicy.cs`, `ScriptTrustValidator.cs`, `RoslynScriptCompiler.cs`, `ScriptCompileSurface.cs`. +- Cross-component seam checked: Inbound API compiled-handler cache (`src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundScriptExecutor.cs`) as a consumer of Transport-imported `ApiMethod` rows; `ManagementActor` native-alarm-source override handlers. + +## Maturity Verdict + +This slice is **unusually well-engineered in its failure-path plumbing** — the deploy pipeline's query-before-redeploy reconciliation, cancellation-token-safe failure writes, ref-counted operation locks, AES-GCM+AAD bundle crypto with zip-bomb caps and passphrase lockout, and the fused two-pass script trust validator are all production-grade and well beyond typical internal-tool quality. However, the **Transport import apply path has serious correctness holes** that undercut the whole promotion story: imported templates silently lose their inheritance edge, their per-script cadence/timeout fields, and their native alarm sources — three distinct silent-data-loss paths, none covered by the otherwise substantial integration test suite. The flattener has one real algorithmic bug (repeated composed templates lose grandchild members), the revision hash omits native alarm sources (breaking staleness detection for that member class), and the central node's validation path leaks Roslyn script assemblies on every Deployments-page comparison. The core design is sound; the defects are concentrated in the newest surface area (M8 Transport, native alarms) where spec, DTOs, and apply code drifted apart. + +--- + +## Findings — Stability & Correctness + +### [Critical] Bundle import drops template inheritance edges — derived templates land as root templates +`TemplateDto.BaseTemplateName` (`Serialization/EntityDtos.cs:115`) is populated at export (`Serialization/EntitySerializer.cs:55`) and even diffed in preview (`Import/ArtifactDiff.cs:65`), but **the apply path never consumes it**. `BundleImporter.BuildTemplate` (`Import/BundleImporter.cs:1524-1562`) never sets `ParentTemplateId`; the Overwrite branch (`BundleImporter.cs:1466-1484`) updates only `Description`/`FolderId` and the three child collections; the two second-pass rewires cover alarm→script FKs (`ResolveAlarmScriptLinksAsync`, line 1907) and compositions (`ResolveCompositionEdgesAsync`, line 1993) — there is **no inheritance-edge pass** (`grep -n 'Parent' BundleImporter.cs` returns nothing). The only code that wires `ParentTemplateId` from `BaseTemplateName`, `EntitySerializer.FromBundleContent` (`EntitySerializer.cs:357-360`), is invoked **only from tests** (`tests/.../EntitySerializerTests.cs`). + +**Failure scenario:** export `Pump` + derived `Pump.WaterPump` from dev (the manifest explicitly records `dependsOn: ["Template:Pump"]` — `Component-Transport.md:79-80`); import into prod. `WaterPump` is created with `ParentTemplateId = null`. `FlatteningPipeline.BuildTemplateChainAsync` (`DeploymentManager/FlatteningPipeline.cs:176-192`) produces a one-element chain; every inherited attribute/alarm/script vanishes from the flattened config. Because validation validates *what's there*, an instance with no bindings on the missing members can deploy "successfully" with a silently amputated configuration. Neither `tests/ZB.MOM.WW.ScadaBridge.Transport.Tests` nor `Transport.IntegrationTests` contains any inheritance round-trip test (grep for `BaseTemplateName|ParentTemplateId` hits only serializer unit tests). + +### [High] Imported template scripts lose `MinTimeBetweenRuns` and `ExecutionTimeoutSeconds` +`TemplateScriptDto` carries both fields (`EntityDtos.cs:148-151`), but the importer drops them on **both** paths: `BuildTemplate` (Add/Rename — `BundleImporter.cs:1550-1560`) and `SyncTemplateScriptsAsync` (Overwrite — the `changed` comparison and field copy at `BundleImporter.cs:~1836-1851` cover Code/TriggerType/TriggerConfiguration/Params/Return/IsLocked only). Both fields participate in the revision hash (`RevisionHashService.cs:87-90`), so the drift is invisible until redeploy, at which point a WhileTrue script's re-fire cadence or a per-script timeout silently reverts to default on the target environment. Same-class omissions: `LockedInDerived` is not carried in the DTO at all (no hits in `EntityDtos.cs`/`EntitySerializer.cs`), so a base template's derived-lock policy does not survive promotion. + +### [High] Template `NativeAlarmSources` are not transported, but instance-level overrides of them are +There is no `TemplateNativeAlarmSourceDto`; `BuildTemplate` copies only attributes/alarms/scripts, and none of the `SyncTemplate*` overwrite helpers touch native alarm sources — yet `InstanceNativeAlarmSourceOverrideDto` **does** travel and is applied (`EntityDtos.cs:292`, `BundleImporter.cs:3325-3338`). Consequences: (a) a template imported as **Add** has no native alarm mirrors at all; (b) an **Overwrite** leaves the target's existing sources untouched and the divergence is invisible because `ArtifactDiff.CompareTemplate` (`Import/ArtifactDiff.cs:54-105`) doesn't diff them either; (c) the carried instance overrides dangle and are *silently ignored* at flatten (`FlatteningService.ApplyInstanceNativeAlarmSourceOverrides`, `FlatteningService.cs:781` — missing canonical name → `continue`). This contradicts `Component-TemplateEngine.md:59-64` which makes native alarm sources first-class template members. + +### [High] Flattening silently drops grandchild members when the same template is composed twice in one subtree +The recursive composed-member resolvers share one `visited` set down the recursion (`FlatteningService.cs:285-297` for attributes; identical pattern for alarms `:649-660`, scripts `:913-924`, native sources `:761-772`). The direct members of a repeated module still resolve (added before the `visited` check), but the **descent into its nested compositions is skipped** on the second occurrence. + +**Failure scenario:** template X composes Y twice (slots `y1`, `y2`); Y composes Z. Flattening produces `y1.*`, `y1.z.*`, `y2.*` — but **not `y2.z.*`**. No validation error fires (the members simply don't exist in the flattened config), so the instance deploys with the second slot's nested sensor/alarm/script members missing. `CollisionDetector.CollectComposedMembers` (`CollisionDetector.cs:126-143`) has the same shared-`visited` early-return, and there it skips even the repeated module's *direct* members — so collision detection is also blind under the second slot. The `visited` set is the right tool for cycle-guarding but the wrong tool for slot deduplication: composition is by-slot, not by-template. Fix is to key the guard on the *path* (prefix) or track only the current recursion stack. + +### [High] Revision hash and diff omit `NativeAlarmSources` — no staleness signal for native-alarm changes +`RevisionHashService.HashableConfiguration` (`RevisionHashService.cs:116-152`) hashes Attributes, Alarms, Scripts, Connections — **not** `FlattenedConfiguration.NativeAlarmSources` (populated at `FlatteningService.cs:135`). Editing a template's native alarm source (connection, source reference, condition filter) or an instance override therefore does not change the revision hash: the Deployments page never flags the instance stale, `DeploymentService.GetDeploymentComparisonAsync` reports `IsStale = false`, and sites keep mirroring the old alarm source indefinitely until some unrelated change forces a redeploy. This contradicts the spec's own claim that overrides "participate in the revision hash — changes … re-deploy as expected" (`Component-TemplateEngine.md:215`), which is implemented for connection-binding overrides but not for the native-alarm member class. `DiffService` should be audited for the same omission. + +### [High] Bundle import leaves stale compiled Inbound API handlers serving old code +`ApplyApiMethodsAsync` (`BundleImporter.cs:2625`) writes `ApiMethod` rows through `IInboundApiRepository` — a DB-level write. The Inbound API's singleton `InboundScriptExecutor` caches compiled delegates **by method name with no content-based invalidation** (`InboundAPI/InboundScriptExecutor.cs:311-331`: `_scriptHandlers.TryGetValue(method.Name, …)` → serve cached delegate) and caches known-bad methods the same way (`:39, :58-62`). An import that Overwrites an ApiMethod script keeps the **old delegate** serving (HTTP 200 with stale behavior); an import that *fixes* a previously-broken method keeps returning "Script compilation failed" from `_knownBadMethods` until restart. This is the exact failure mode already documented in project memory for DB-direct edits — Transport import is a DB-direct edit at scale, and nothing in `ApplyAsync` signals the executor. Cross-cluster deployments make this worse: the cache lives per central node. + +### [High] Locked native alarm sources can be overridden at instance level — lock not enforced anywhere +Spec: "Lock applies to the entire source" (`Component-TemplateEngine.md:114`). Reality: `ApplyInstanceNativeAlarmSourceOverrides` (`FlatteningService.cs:775-792`) applies the override with **no lock check** — contrast the attribute path (`:309 if (existing.IsLocked) continue`) and alarm path (`:337`). `ResolvedNativeAlarmSource` doesn't even carry `IsLocked` (the inherit-time `lockedNames` set at `:676` is local and discarded). The management command path has no check either: `ManagementActor.HandleSetInstanceNativeAlarmSourceOverride` (`ManagementService/ManagementActor.cs:882-909`) upserts unconditionally after site-scope authorization. A site-scoped operator can repoint a *locked* native alarm binding to an arbitrary connection/source reference — precisely what locking exists to prevent on a read-only alarm mirror. + +### [Medium] Post-success audit failure flips a committed Success deployment record to Failed +`DeployInstanceAsync` commits the terminal Success status (`DeploymentService.cs:304-306`), runs post-success side effects (correctly swallowed, `:970-1005`), then calls `_auditService.LogAsync` **inside the same try** (`:330-332`). If that audit write throws (transient DB fault, serialization), the catch block (`:343-397`) unconditionally sets `record.Status = Failed` and persists it — central now reports Failed while the site runs the new config, the exact divergence the surrounding comments (`:298-303`) were written to prevent. It also violates the repo convention "audit-write failure NEVER aborts the user-facing action". The audit call should be outside the try or individually guarded like `TryLogLifecycleTimeoutAsync` (`:1034-1066`) already is for lifecycle timeouts. + +### [Medium] Deleting a `NotDeployed` instance still requires a live site round-trip +`StateTransitionValidator.CanDelete` explicitly allows delete-from-NotDeployed as "central-side record cleanup (no live site config to tear down)" (`StateTransitionValidator.cs:40-49`), but `DeploymentService.DeleteInstanceAsync` (`DeploymentService.cs:546-585`) unconditionally sends `DeleteInstanceCommand` to the site and fails on timeout. A Transport-imported instance (always lands `NotDeployed`) targeted at a site that is unreachable or not yet commissioned is **undeletable** through the normal path. The two components disagree about the same design sentence. + +### [Medium] Import blocker heuristic can hard-block a valid import with no operator override +The Pass-1 name scan (`RunSemanticValidationAsync`, `BundleImporter.cs:3603-3692`; shared with `DetectBlockersAsync:751-818`) flags any `PascalCase(`-shaped token not in the enumerative `KnownNonReferenceNames` denylist (`:853-874`) as a missing SharedScript/ExternalSystem and **throws `SemanticValidationException`** at apply — rolling back the whole import. Any script that defines a local PascalCase method (`decimal ComputeRate(...)` then `ComputeRate(x)`), uses a stdlib type not on the list (`Regex.Match`, `StringBuilder`, `Json`), or embeds SQL keywords outside the nine listed, is a false positive with no "proceed anyway" escape hatch — the operator must edit scripts or re-cut the bundle. The denylist will drift forever. Conversely `Regex`/`StringBuilder` being absent from the list means the heuristic fires on innocent code today. Consider downgrading Pass-1 findings on *template scripts* to warnings (the deploy gate re-validates authoritatively) and keeping hard-block only for `ApiMethod` scripts, or adding an explicit ignore resolution. + +### [Medium] `OperationLockManager` is single-node in-memory; concurrent ops via the standby node aren't excluded +Documented as acceptable ("lost on central failover", `OperationLockManager.cs:12-13`) and the implementation itself is careful (ref-counting, reservation-before-wait, exception-filter cleanup — `:46-129`). Residual risk: nothing structurally prevents both central nodes from executing mutating instance ops concurrently if traffic bypasses Traefik's active-node routing (direct ports 9001/9002 are documented in CLAUDE.md). Optimistic concurrency on `DeploymentRecord` is the backstop, but instance-state writes (`DisableInstanceAsync:456-458`) are last-write-wins. Low probability, but worth an explicit invariant (reject management ops on the non-active node). + +--- + +## Findings — Performance + +### [High] Every validation run compiles every script with Roslyn — CPU heavy and leaks loaded assemblies on the central node +`ValidationService.ValidateScriptCompilation` (`ValidationService.cs:256-273`) calls `ScriptCompiler.TryCompile` per script; that runs `ScriptTrustValidator.FindViolations` (a fresh `CSharpCompilation.CreateScriptCompilation` over the **entire TPA reference set**, `ScriptTrustValidator.cs:114-124`) *plus* `RoslynScriptCompiler.Compile` → `CSharpScript.Create(...).Compile()` (`RoslynScriptCompiler.cs:70-71`). `Script.Compile()` on success emits and loads the script assembly via Roslyn's `InteractiveAssemblyLoader`, which is **not collectible** — each compiled script permanently occupies loader/metadata memory in the central host. + +This sits on hot paths: `FlatteningPipeline.FlattenAndValidateAsync` (`FlatteningPipeline.cs:161-167`) is invoked (a) per deploy, (b) per instance by `DeploymentService.GetDeploymentComparisonAsync` (`DeploymentService.cs:652`) — i.e. **at query time whenever the Deployments page computes staleness**, and (c) per deployed instance by `StaleInstanceProbe` during every template-overwriting bundle import (`BundleImporter.ComputeStaleInstanceIdsAsync:1242-1327`). A 50-instance environment with 10 scripts each = 500 full trust-compilations + 500 script-assembly loads per Deployments-page staleness sweep, unbounded over process lifetime. There is **no compile-result cache**. Fix: cache `TryCompile` verdicts keyed by SHA-256 of code (the verdict is a pure function of code + policy), and/or skip `ValidateScriptCompilation` on read-only comparison paths (staleness needs the hash, not a compile). + +### [Medium] `LineDiffer` worst case is O((N+M)²) memory with no input cap +The Myers implementation snapshots the full V frontier per edit distance (`LineDiffer.cs:157-204`: `trace.Add((int[])v.Clone())` with `v` of length `2(N+M)+1`, up to `N+M` rounds). Two ~10,000-line scripts with no common lines ⇒ ~20k rounds × 160 KB ≈ **3+ GB** transient allocation during `PreviewAsync`. `maxLines` (400) caps only the *output* (`:66-92`); nothing caps the *inputs*, and the bundle size cap (100 MB) comfortably admits such scripts. A crafted (or merely bloated) bundle can OOM the active central node from the import preview. Cap input line counts (fall back to the `` summary beyond a threshold) or use the linear-space Myers variant. + +### [Medium] Any template with a folder, parent, or composition can never diff `Identical` +`ArtifactDiff.CompareTemplate` compares `FolderNameOf`/`BaseTemplateNameOf`/`CompositionTargetNameOf` placeholder strings (`""`) against real incoming names (`ArtifactDiff.cs:64-65, 100, 674-693`), so `FolderName`/`BaseTemplateName`/`Compositions.*` always register as changed for such templates. Effect: the wizard's "Identical → auto-skipped" contract (`Component-Transport.md:222`) never applies to structured templates; every re-import of an unchanged bundle shows spurious Modified rows and, if bulk-Overwrite is chosen, generates needless audit rows and stale-instance probing. The preview loop already hydrates templates (`BundleImporter.cs:379-388`); passing the folder/template name maps it also builds (`:365-366`) into `CompareTemplate` would fix all three placeholders cheaply. + +### [Low] `LoadAsync` is O(bundle) even when only the manifest is needed, and each load double-opens the zip +Acknowledged in the design doc itself (`Component-Transport.md:90` — deferred `ReadManifestAsync`). Combined with sessions holding the full decrypted content (`BundleSession.DecryptedContent`) for up to 30 minutes and no cap on concurrent session count (`BundleSessionStore.cs:38`), N concurrent 100 MB uploads pin N×~200 MB. The T-007 zeroing on apply/failure (`BundleImporter.cs:1105, 1194`) is good; a session-count cap would round it out. + +### [Low] Import apply is one long EF transaction with per-entity audit rows and four intermediate flushes +`ApplyAsync` (`BundleImporter.cs:960-1098`) holds a relational transaction across full semantic validation (Roslyn-free — fine), all entity staging, per-row `IAuditService.LogAsync` calls, and stale-instance probing (which itself re-flattens per instance, see the Roslyn finding). For a large bundle this is minutes of open-transaction lock footprint on the shared config DB while the Blazor circuit waits. The CLI's raised 5-minute timeout acknowledges the symptom. + +--- + +## Findings — Security / Trust Model + +### [Medium] Deny-list gaps: `Environment.Exit`/`FailFast` (site-node kill), `Environment.GetEnvironmentVariable` (secret read) +`ScriptTrustPolicy.ForbiddenScopes` (`ScriptTrustPolicy.cs:36-45`) blocks IO/Process/Threading/Reflection/Net/Interop/Win32, but the `System` namespace root types remain fully available: a template script containing `Environment.Exit(0)` compiles cleanly against the surface and **terminates the site-runtime process** on first trigger — a one-line denial of service of an entire site node from the Design role. `Environment.FailFast`, `Environment.GetEnvironmentVariable` (exfiltrate `SCADABRIDGE_API_KEY`-class secrets via `Notify`/`ExternalSystem.Call`), and `GC.Collect` loops are similarly unblocked. These are cheap adds to `ForbiddenScopes` (as fully-qualified type-or-member entries) or to `ForbiddenIdentifiers`. + +### [Medium] `System.Data` allowance is an intentional but unbounded network hole +Site scripts may use raw ADO.NET (`Database.Connection` returns `DbConnection`; `System.Data.Common` is necessarily bindable — `ScriptCompileSurface.cs:119`). `System.Data` is absent from `ForbiddenScopes`, so if the site-runtime execution reference set includes any concrete provider (e.g. `Microsoft.Data.SqlClient`, which the Database helper itself needs), a script can `new SqlConnection("Server=attacker;...")` — an arbitrary-host network channel that the `System.Net` deny was meant to close. The semantic pass will not flag it (SqlClient is not under a forbidden scope). Worth either denying provider namespaces (`Microsoft.Data`, `System.Data.SqlClient`, `System.Data.Odbc`, …) while keeping `System.Data.Common` types reachable only via the `Database` helper, or documenting the accepted risk explicitly in `Component-ScriptAnalysis.md`. + +### [Medium] Reflection-gateway member list misses `GetTypes`, `Invoke`, `EntryPoint`, `DeclaredMembers` +`ReflectionGatewayMembers` (`ScriptTrustPolicy.cs:66-88`) blocks `GetMethod`/`InvokeMember`/etc. but not `Invoke` (on `MethodInfo`/`Delegate`), `GetTypes`, `EntryPoint`, `DeclaredMethods`. Pass 1 usually compensates — any expression whose resolved member's containing type is under `System.Reflection` is flagged semantically (`ScriptTrustValidator.cs:180-212`) — but in the documented **TPA-degraded fallback mode** (`ScriptTrustPolicy.cs:207-245`) resolution weakens and Pass 2 becomes the primary defence for chained accesses; the syntactic list should be closed under the obvious invocation members since it exists precisely for that mode. Cheap hardening. + +### [Medium] Transport import bypasses the script trust gate entirely +`ApplyAsync` writes template scripts, shared scripts, and API-method scripts to the DB without ever invoking `ScriptTrustValidator` (`RunSemanticValidationAsync` at `BundleImporter.cs:3603` is name-resolution + `SemanticValidator` only). Compensating controls exist downstream — the deploy gate re-validates template scripts (`ScriptCompiler.TryCompile`), the Inbound API re-checks trust at lazy compile — but the layering claim in `Component-ScriptAnalysis.md` ("all call sites delegate…") now has a fifth write path with no gate, and the Inbound API's check happens at *serve time* on the production node rather than at *import review time*. Running `FindViolations` over imported script bodies during Pass 2 (they're already in memory) would restore the design-time boundary and give the operator the finding in the wizard instead of a runtime 500. + +### Positive observations (crypto & envelope) +`BundleSecretEncryptor` is textbook: AES-256-GCM with 16-byte salt / 12-byte nonce / 16-byte tag, PBKDF2-SHA256 600k iterations, fresh salt+nonce per encrypt (`BundleSecretEncryptor.cs:14-58`). The T-005 AAD binding of non-derivative manifest fields (`BundleManifestAad.cs:45-55`, verified in `LoadAsync` at `BundleImporter.cs:252-255`) makes the source-environment confirmation gate tamper-evident — a genuinely thoughtful touch. T-006 zip-bomb caps validate central-directory headers before any decompression (`BundleImporter.cs:296-338`); T-003 lockout is keyed by content hash so a second tab/CLI can't sidestep it (`BundleSessionStore.cs:40-47, 122-167`); T-007 zeroes decrypted plaintext on both success and failure. Key management is passphrase-only with no persistence — appropriate for the file-carry threat model, with the residual (documented) weakness that unencrypted export is allowed and only audit-flagged. + +--- + +## Findings — Conventions & Spec Drift + +### [Medium] Instance alarm overrides implement less than the spec grants +`Component-TemplateEngine.md:113` says instance-overridable alarm fields include **Description and On-Trigger Script reference**; `ApplyInstanceAlarmOverrides` (`FlatteningService.cs:328-355`) applies only `TriggerConfigurationOverride` and `PriorityLevelOverride`. Either the doc or the entity/flattener should move. + +### [Medium] `ArtifactDiff` child-equality predicates miss fields that the import sync *does* write +`AttributesEqual` omits `ElementDataType` and `LockedInDerived` (`ArtifactDiff.cs:646-651`); `AlarmsEqual` omits the on-trigger script reference (`:653-658`); `ScriptsEqual` omits `MinTimeBetweenRuns`/`ExecutionTimeoutSeconds` (`:660-666`). A bundle differing only in those fields classifies **Identical → auto-skipped → silently not imported** (for the fields the sync would have written) — the mirror image of the false-Modified problem above. Diff predicates and sync-helper `changed` predicates should be generated from one field list per entity so they cannot drift (they already have, in both directions). + +### [Low] Bundle format spec advertises a `scripts/` directory that is never produced, while `MaxBundleEntryCount` defaults to 4 +`Component-Transport.md:45-47` documents optional `scripts/template-{id}-{name}.cs` entries; no code writes or reads them, and `TransportOptions.MaxBundleEntryCount = 4` (`TransportOptions.cs:22`) would reject a bundle that actually used the feature for more than two scripts. Remove from the spec or note it as unimplemented. + +### [Low] `ScriptCompiler.TryCompile` surfaces only the first violation/error +`Validation/ScriptCompiler.cs:42,47` return `violations[0]` / `errors[0]`; an operator fixing a script with five issues gets five deploy round-trips. Both underlying APIs return full lists — join them. + +### [Low] `RevisionHashService` determinism contract is fragile but guarded +Hash stability depends on alphabetical property declaration order in the `Hashable*` records (`RevisionHashService.cs:12-19, 113-115`); the guard test (`HashableRecords_PropertiesDeclaredAlphabetically`) exists, which is the right mitigation. Note for maintainers: adding `NativeAlarmSources` (see the High finding) will change all hashes — plan it as a deliberate migration (one-time global staleness) rather than a patch. + +### Positive observations (conventions) +`DeploymentService` is exemplary on the repo's own rules: `CancellationToken.None` for failure/audit writes after cancellation (`DeploymentService.cs:355-372`), timeout-audit entries for all lifecycle verbs, reconciliation preserving operator-intended Disabled state (`:895-907`), terminal-status-before-side-effects ordering (`:298-306`), and the notify-and-fetch PendingDeployment TTL rationale (`:283-290`) is documented where the decision lives. `CycleDetector.BuildLookup` duplicate-Id tolerance and the Id-0-is-legitimate handling (`CycleDetector.cs:21-27, 56-59`) show real defect-driven hardening. `ScriptTrustPolicy`'s `DefaultReferences` vs `AnalysisReferences` separation with the loud TPA-degradation flag (`ScriptTrustPolicy.cs:207-245`) is a well-reasoned two-layer defence. + +--- + +## Underdeveloped Areas + +1. **Transport site/instance wave (M8) shipped ahead of its template-fidelity basics.** Instances, connections, name-mapping, FK rewiring, and stale-probing are all implemented and tested (`SiteInstanceImportTests.cs`, `CompositionImportTests.cs`, rollback-failure tests), but the older template payload silently loses inheritance, cadence/timeout, derived-lock flags, and native alarm sources (findings above). The DTO layer (`EntitySerializer.FromBundleContent`) actually implements more fidelity than the apply path uses — the importer re-implements DTO→entity mapping by hand (`BuildTemplate`) and drifted. +2. **No import test exercises a derived template.** `Transport.IntegrationTests` covers round-trip, conflict resolution, composition edges, semantic validation failure — but grep shows zero inheritance coverage. A single `export(base+derived) → import → flatten` test would have caught the Critical finding. +3. **No flattening test for repeated composition.** `TemplateEngine.Tests/Flattening` exists but the shared-`visited` semantics (same template in two slots of one subtree) is untested; same for `CollisionDetector`. +4. **Deferred items acknowledged in docs but load-bearing:** manifest-only `ReadManifestAsync` (perf), Configuration-Audit-Log "Bundle Import" filter UI (Transport-012), rename call-site rewriting ("call sites are not rewritten in v1" — `BundleImporter.cs:1519-1522`, meaning a Rename resolution on a shared script produces a bundle whose importing templates still call the *old* name, satisfied only because the name-set registers both — the *target DB* ends up with scripts calling a name that may not exist there; only Pass-1's dual registration hides it at import time). +5. **`ArtifactDiff` version fields are permanently null** (`ArtifactDiff.cs:33-37`) — the manifest's per-artifact `version` numbers are display-only; no optimistic concurrency exists between preview and apply, so a target edited between Preview and Apply is silently overwritten (session TTL is 30 min — a real window). +6. **Compile-surface parity relies on three hand-maintained mirrors** (`ScriptCompileSurface`, `TriggerCompileSurface`, Central-UI `SandboxScriptHost`), guarded by reflection tests for two of them and representative-script tests for the third (`Component-ScriptAnalysis.md:141-145`). Adequate but fragile; a source-generator would eliminate the class of drift. +7. **`SemanticValidator` call-target extraction is string-based** and self-admittedly cannot resolve dynamic child names (`SemanticValidator.cs:43-56` — leaf-name fallback accepts any composed script with a matching leaf, so a wrong-child `Children[x].CallScript("Y")` passes design-time validation). Documented tradeoff, but worth a warning-level finding rather than silence. + +--- + +## Prioritized Recommendations + +1. **[Critical] Wire template inheritance in the import apply path.** Add a `ResolveInheritanceEdgesAsync` pass mirroring `ResolveCompositionEdgesAsync` (resolve `BaseTemplateName` against Local + target by name, honouring Rename), run it after the template flush, and run `CycleDetector.DetectInheritanceCycle`/`DetectCompositionCycle` over the merged graph before commit (the importer currently performs **no** acyclicity check — a crafted bundle can persist a composition cycle that the authoring path would reject). Add a derived-template round-trip integration test. +2. **[High] Close the import data-loss set in one sweep:** carry `MinTimeBetweenRuns`/`ExecutionTimeoutSeconds` through `BuildTemplate` + `SyncTemplateScriptsAsync`; add `LockedInDerived` to the attribute/alarm/script DTOs and syncs; add `TemplateNativeAlarmSourceDto` (or block export of templates that have native alarm sources until supported). Drive diff predicates and sync predicates from a single per-entity field list. +3. **[High] Fix the repeated-composition `visited` bug** in the four `ResolveComposed*Recursive` methods and `CollisionDetector.CollectComposedMembers` — guard on recursion *path*, not global template Id. +4. **[High] Add `NativeAlarmSources` to `RevisionHashService` (and `DiffService`)** as a planned hash-migration, restoring staleness detection for native-alarm edits. +5. **[High] Cache script-compile verdicts by code hash** and stop loading script assemblies on validation-only paths; skip `ValidateScriptCompilation` in `GetDeploymentComparisonAsync`/`StaleInstanceProbe` (staleness needs only the hash). This removes both the CPU spike and the non-collectible assembly leak from the Deployments page and bundle imports. +6. **[High] Invalidate Inbound API handler caches on import** — publish an ApiMethod-changed notification from `ApplyAsync` (or key `_scriptHandlers` by `(name, scriptHash)`), covering the management-path gotcha already in project memory at the same time. +7. **[High] Enforce locks on native alarm source overrides** at both `ManagementActor.HandleSetInstanceNativeAlarmSourceOverride` and `FlatteningService.ApplyInstanceNativeAlarmSourceOverrides` (carry `IsLocked` on `ResolvedNativeAlarmSource`). +8. **[Medium] Harden the deny-list:** add `System.Environment` members (`Exit`, `FailFast`, `GetEnvironmentVariable(s)`, `SetEnvironmentVariable`), close the reflection-gateway list (`Invoke`, `GetTypes`, `EntryPoint`), and decide/document the `System.Data` provider posture. +9. **[Medium] Deployment audit-write isolation:** move the post-success `LogAsync` in `DeployInstanceAsync` out of the try (or guard it) so an audit fault can never flip a committed Success record to Failed; skip the site round-trip when deleting `NotDeployed` instances. +10. **[Medium] Bound `LineDiffer` inputs; fix `ArtifactDiff` placeholder comparisons** (pass folder/template name maps into `CompareTemplate`) so Identical classification works for structured templates; consider an "ignore blocker" resolution for the Pass-1 heuristic on template scripts. diff --git a/archreview/06-edge-integrations.md b/archreview/06-edge-integrations.md new file mode 100644 index 00000000..b94ab72b --- /dev/null +++ b/archreview/06-edge-integrations.md @@ -0,0 +1,143 @@ +# Architecture Review 06 — Edge Integrations (Inbound & Outbound Boundaries) + +Reviewer domain: the four components where ScadaBridge touches external systems. + +## Scope + +Deep source review (every non-generated `.cs` file read) of: + +- `src/ZB.MOM.WW.ScadaBridge.InboundAPI` — `POST /api/{methodName}`, API-key auth, Roslyn script handlers, parameter/return validation, audit middleware, routed site calls. +- `src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway` — outbound HTTP client (`Call`/`CachedCall`), `DatabaseGateway` (`Connection`/`CachedWrite`), HTTP + SQL error classifiers, S&F handoff. +- `src/ZB.MOM.WW.ScadaBridge.NotificationService` + the delivery adapters in `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/Delivery` — SMTP (MailKit) wrapper, OAuth2 token service, SMTP/SMS error classifiers, credential redactor, Email/SMS adapters. +- `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier` — Native-AOT `WWNotifier.exe` client (arg parser, config loader, failover loop, YES/NO reporter). + +Specs compared: `docs/requirements/Component-InboundAPI.md`, `Component-ExternalSystemGateway.md`, `Component-NotificationService.md`, `docs/plans/2026-06-26-delmia-recipe-notifier-design.md`. Test projects examined: `tests/ZB.MOM.WW.ScadaBridge.{InboundAPI,ExternalSystemGateway,NotificationService,NotificationOutbox,DelmiaNotifier}.Tests`. + +## Maturity Verdict + +This is a mature, unusually well-hardened edge layer. The code shows evidence of multiple hardening passes: enumeration-safe auth responses, bounded body capture with UTF-8-safe truncation, ordered cancellation-vs-timeout catch discrimination repeated consistently across HTTP, SQL, SMTP, and SMS paths, poison-message parking on both S&F retry paths, credential redaction on every error string, and secrets encrypted at rest (`ExternalSystemConfiguration.cs:27-29,83-85`). Error classification matches the spec (5xx/408/429/connection = transient) on all four channels and is centralized in dedicated classifiers with tests. The remaining risk is concentrated in three places: **cache coherence of compiled inbound script handlers across central failover** (a silent-stale-code bug, the one genuinely High-severity defect found), **two acknowledged spec-vs-code gaps in the External System Gateway** (per-system timeout and path templating), and **unbounded buffering of outbound HTTP response bodies**. DelmiaNotifier is small, faithful to its design doc, and fully tested at the logic level. + +--- + +## Findings — Dimension 1: Stability + +### [High] Compiled inbound handler cache goes stale across central failover — old script serves silently with HTTP 200 + +`InboundScriptExecutor` is a per-node singleton (`ServiceCollectionExtensions.cs:19`) whose `_scriptHandlers` and `_knownBadMethods` caches (`InboundScriptExecutor.cs:26,39`) are mutated only by: + +- the node's own startup compile of all methods (`Host/Program.cs:379-388`), and +- `ManagementActor.HandleUpdateApiMethod`/`HandleDeleteApiMethod` → `CompileAndRegister`/`RemoveHandler` (`ManagementActor.cs:2631,2650`) — which run **only on the node hosting the ManagementActor singleton (the active node)**. + +The standby central node compiles every method at *its* startup and then never hears about updates. `ExecuteAsync` looks the handler up by **name only** (`InboundScriptExecutor.cs:311`) — no revision/hash check — even though the endpoint fetches the fresh `ApiMethod` row (with the current `Script` text) from the DB on every request (`EndpointExtensions.cs:142`). + +Concrete failure scenario: +1. Method `GetOrderStatus` updated via CLI on active node A → A recompiles, serves new logic. +2. Weeks later A fails over to B. B still holds the delegate compiled from the script text as of B's process start. +3. B serves the **old** script indefinitely — HTTP 200, correct-looking JSON, `UpdateApiMethod` long ago reported success. Nothing recompiles until B restarts. + +The mirror-image variant: a method that was broken at B's startup landed in B's `_knownBadMethods`; it was later fixed via management update (clears the record on A only). After failover, B answers `"Script compilation failed for this method"` for a method that compiles fine. + +This is the cluster-wide generalization of the already-known "direct SQL edit doesn't clear the cache" issue (documented as a Direct SQL Warning, `Component-InboundAPI.md:129-131`) — but here the write went through the *supported* management path and is still lost. A per-request script-hash comparison against the freshly-fetched row would fix failover staleness, direct-SQL staleness, and known-bad staleness in one move at negligible cost (the row is already in hand). + +Deleted methods are safe (fresh DB lookup returns null → 403); methods *created* after standby startup are safe (lazy compile, `InboundScriptExecutor.cs:314-330`). + +### [Medium] Method timeout abandons the running script; its DI scope is disposed underneath it + +On timeout, `handler(context).WaitAsync(cts.Token)` (`InboundScriptExecutor.cs:333`) returns control but does not stop the underlying script task — cancellation is cooperative via `ctx.CancellationToken` only. The `finally` block then disposes the per-execution DI scope (`InboundScriptExecutor.cs:389-393`) while a non-cooperative script (e.g. mid-flight in a poorly-authored loop, or blocked in a routed call that ignores the token) may still be running and holding `InboundDatabaseHelper` → the scoped `IDatabaseGateway`. Consequences: `ObjectDisposedException`s surfacing as unobserved background faults, and — since the inbound API deliberately has no rate limiting (`Component-InboundAPI.md:140`) — repeated timeouts of a slow method accumulate orphaned executions with no counter or cap. Recommend at minimum a metric for abandoned executions, and deferring scope disposal until the handler task actually completes (`ContinueWith` on the abandoned task). + +### [Medium] ESG API-key `AuthConfiguration` colon-splitting is ambiguous — a key containing `:` silently breaks auth + +`ExternalSystemClient.ApplyAuth` (`ExternalSystemClient.cs:489-499`) interprets `AuthConfiguration` as `"HeaderName:KeyValue"` when it contains a colon, else as a bare key with default header `X-API-Key`. An API key whose *value* legitimately contains a colon (not rare for opaque tokens) is silently misparsed: the prefix becomes a bogus header name and the request goes out mis-authenticated with only a remote 401 to debug from. The Basic branch (`:504-519`) is fine (split-2 keeps colons in the password), but the apikey branch has no escape hatch. Structured JSON config (`{"header":"X-API-Key","key":"..."}`) — which the entity doc-comment (`ExternalSystemDefinition.cs:13`, "JSON-serialized authentication configuration") already implies — would eliminate the ambiguity. + +### [Medium] DelmiaNotifier failover can duplicate a processed notification + +`HttpRecipeSender.cs:36-45` maps **any** `HttpRequestException` and any non-caller timeout to `ConnectFailed`, which the loop rolls over to the next URL (`Notifier.cs`, `continue`). Both cases include "the server received and possibly processed the request, but the response never arrived" (connection reset after send; slow response exceeding `TimeoutSeconds`). The next node then receives a second `DelmiaRecipeDownload` POST for the same recipe. This matches the design table (`2026-06-26-delmia-recipe-notifier-design.md:97-99` lists timeout as a failover trigger) — so this is a *design-level* at-least-once semantics that is nowhere documented as such; the inbound `DelmiaRecipeDownload` method must be idempotent on `(MachineCode, DownloadPath, WorkOrderNumber)` or duplicates will surface downstream. Worth one paragraph in the design doc and a check of the deployed method script. + +### [Medium] SMS adapter keeps sending after the first transient failure — inflating duplicate texts on retry + +In `SmsNotificationDeliveryAdapter.DeliverAsync` (`SmsNotificationDeliveryAdapter.cs:148-170`), a transient result does not break the loop; the remaining recipients are still attempted. Since *any* transient rolls the **whole notification** up to `Transient` (`RollUp`, `:297-303`) and retry re-texts everyone, each additional number accepted after the first transient is a guaranteed duplicate SMS on the retry. Short-circuiting on the first transient would reduce duplicates at no delivery cost (the retry re-attempts all numbers anyway). The re-send-to-all characteristic itself is documented (`Component-NotificationService.md:102`), but this amplification is avoidable within v1's no-per-recipient-state constraint. + +### [Low] `ErrorClassifier.IsTransient(Exception)` treats any `OperationCanceledException` as transient + +`ErrorClassifier.cs:32-38` classifies bare `OperationCanceledException`/`TaskCanceledException` as transient. Every current call site is protected by ordered catch filters that peel off caller-cancellation first (`ExternalSystemClient.cs:314-327`), so behaviour is correct today — but the predicate is public and a future call site that skips the ordering would buffer caller-cancelled work into S&F. A `cancellationToken` parameter (as `SmtpErrorClassifier.Classify` and `SmsErrorClassifier.Classify` already take) would make it misuse-resistant. + +### [Low] First-of-many SMTP configuration is nondeterministic + +`EmailNotificationDeliveryAdapter.cs:75-76` and `NotificationOutboxActor.ResolveRetryPolicyAsync` (`NotificationOutboxActor.cs:373-375`) both use `GetAllSmtpConfigurationsAsync()` + `FirstOrDefault()`. With more than one SMTP row the selected server and the retry policy depend on repository ordering. Either enforce single-row semantics or add an explicit "active" flag. + +### [Low] OAuth2 token cache: unsynchronized field visibility and unbounded growth + +`OAuth2TokenService.CacheEntry.Token/Expiry` are plain fields read outside the per-entry lock (`OAuth2TokenService.cs:48-51`) and written inside it (`:93-94`). The races are benign (worst case: one redundant token fetch), but `volatile`/`Volatile.Read` would make it airtight. The cache is also never evicted — bounded in practice by the number of distinct SMTP credential triples, so acceptable. + +### [Low] Non-JSON content-type with a body yields a misleading 400 + +`EndpointExtensions.cs:181-194`: `ContentLength > 0 || ContentType contains "json"` means a `text/plain` or `application/x-www-form-urlencoded` body is force-parsed as JSON and rejected as "Invalid JSON in request body" rather than a 415. Cosmetic, but the error steers the caller at the wrong problem. + +## Findings — Dimension 2: Performance + +### [Medium] Outbound HTTP responses are buffered unbounded + +`ExternalSystemClient.InvokeHttpAsync` uses `client.SendAsync(request, token)` with the default `ResponseContentRead` completion option (`ExternalSystemClient.cs:312`) and then `ReadAsStringAsync` (`:337`) — the *entire* response body is materialized in memory with **no size cap**. `MaxErrorBodyChars` (`:386`) truncates only the string embedded into error messages, *after* the full read. The inbound side caps request bodies at 1 MiB precisely because "no rate limiting… an explicit, modest cap bounds per-request allocations" (`InboundApiOptions.cs:16-19`); the outbound side extends no such protection against a misbehaving or compromised external system streaming hundreds of MB into a site node (which also runs the instance actors). Use `HttpCompletionOption.ResponseHeadersRead` plus a bounded read (the audit middleware's cap+1 pattern is directly reusable). + +### [Medium] Serial per-recipient Twilio POSTs inside a serial dispatch sweep + +`SmsNotificationDeliveryAdapter` sends one POST per phone number sequentially (`SmsNotificationDeliveryAdapter.cs:148-170`), and the outbox dispatcher delivers each due notification sequentially within a sweep (`NotificationOutboxActor.cs:320-345`). Worst case for one 50-recipient list against a black-holing endpoint: 50 × 30 s = 25 minutes inside one sweep, during which *all* other due notifications (including email) queue behind it and appear "stuck" on the KPI tiles. Bounded parallelism per notification (e.g. `Parallel.ForEachAsync` with 4–8 DOP — Twilio tolerates it) and/or first-transient short-circuit (see Stability above) would bound the sweep. At current scale (small lists) this is latent, not live. + +### [Low] Per-delivery TCP+TLS SMTP handshake + +`MailKitSmtpClientWrapper` is deliberately one-connection-per-delivery, extensively documented with the factory seam left open for pooling (`MailKitSmtpClientWrapper.cs:12-38`). Correct tradeoff at current volume; noted only as the first knob if notification volume grows. + +### [Low] Startup compiles every inbound method serially + +`Host/Program.cs:383-387` runs Roslyn compilation per method in a foreach before `app.RunAsync()`. Fine at tens of methods; would stretch central startup (and failover-recovery window) at hundreds. `Parallel.ForEach` or lazy-only compilation are cheap options later. + +### [Good] The hot-path hygiene elsewhere is exemplary + +Bounded audit capture at the source with `ArrayPool` scratch buffers and cap+1 over-read detection (`AuditWriteMiddleware.cs:442-502`), fire-and-forget audit write with fault observation (`:358-359,380-398`), `EnableBuffering` skipped for bodyless requests (`:169-191`), name-keyed indexed repository lookups on the ESG hot path (`ExternalSystemClient.cs:540-550`), known-bad-method cache preventing per-request Roslyn recompiles with a 1000-entry flood cap (`InboundScriptExecutor.cs:38-63`), and per-system `HttpClient` factory clients with `MaxConnectionsPerServer` scoped to gateway-owned names only — explicitly avoiding process-global handler replacement (`ExternalSystemGateway/ServiceCollectionExtensions.cs:28-44,92-105`). No socket-exhaustion patterns anywhere: all four components use `IHttpClientFactory` (DelmiaNotifier's single-shot process makes its one `new HttpClient` correct). + +## Findings — Dimension 3: Conventions & Spec Fidelity + +### [High] Spec drift: per-system ESG timeout is specified but not implemented + +`Component-ExternalSystemGateway.md:38` ("**Timeout**: Per-system timeout for all method calls") and `:101` are explicit, but `ExternalSystemDefinition` has no timeout field (`Commons/Entities/ExternalSystems/ExternalSystemDefinition.cs`) and the client acknowledges it: "ExternalSystemDefinition has no per-system Timeout field yet, so the configured DefaultHttpTimeout is the effective round-trip limit" (`ExternalSystemClient.cs:300-305`). Every external system — a fast local MES and a slow remote recipe manager alike — shares one global 30 s (`ExternalSystemGatewayOptions.cs:9`). Repo rules say the design doc is the spec and doc+code travel together; either add the column (entity + EF config + migration + CLI/UI) or amend the spec to the global-timeout reality. + +### [Medium] Spec drift: path templates (`/recipes/{id}`) are never substituted + +The spec's method definition shows a templated relative path (`Component-ExternalSystemGateway.md:43`), but `BuildUrl` (`ExternalSystemClient.cs:431-463`) concatenates the path verbatim and routes parameters exclusively to the query string (GET/DELETE) or JSON body (POST/PUT/PATCH). A method authored with `{id}` in its path literally calls `.../recipes/{id}`. Either implement `{param}` substitution (with the consumed parameter removed from body/query) or strike the example from the spec before someone authors against it. + +### [Low] Spec drift: inbound error responses lack the documented `code` field + +`Component-InboundAPI.md:91-97` documents failure bodies as `{"error": ..., "code": "SITE_UNREACHABLE"}`; every actual error body is `{ error }` only (`EndpointExtensions.cs:111-113,153-155,191-193,212-214,253-255`) and a routed site-unreachable failure surfaces as a generic 500 "Internal script error" (`InboundScriptExecutor.cs:382-387` swallows the `InvalidOperationException` from `RouteTarget.Call`, `RouteHelper.cs:172-174`). Machine-readable error codes matter to exactly the audience of this API (external integrators); implement or de-spec. + +### [Low] OAuth2 SMTP is Microsoft-365-only despite a broader spec + +`OAuth2TokenService` hardcodes `https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token` and the `outlook.office365.com/.default` scope (`OAuth2TokenService.cs:73,80`); the spec says "For Microsoft 365 **and other modern SMTP providers** that require OAuth2" (`Component-NotificationService.md:48`). A configurable authority/scope on `SmtpConfiguration` is a small change; until then the spec overpromises. + +### [Low] `InboundApiEndpointFilter` freezes its options at construction + +The filter is a singleton taking `IOptions` (`InboundApiEndpointFilter.cs:32-38`), so a live change to `MaxRequestBodyBytes` is ignored until restart — while the sibling `AuditWriteMiddleware` deliberately hot-reads `IOptionsMonitor` per request (`AuditWriteMiddleware.cs:97,149`). Inconsistent; use the monitor. + +### [Good] Convention adherence is otherwise strong + +Options pattern with startup validators throughout (`SmsOptionsValidator`, `NotificationOptions`); additive message-contract evolution respected and even documented at the parameter level ("this optional parameter trails the CancellationToken because it was appended additively", `InboundScriptExecutor.cs:243-246`); secrets consistently encrypted at rest via `EncryptedStringConverter` (ESG `AuthConfiguration` + DB `ConnectionString`: `Configurations/ExternalSystemConfiguration.cs:27-29,83-85`; SMTP credentials + Twilio AuthToken: `Configurations/NotificationConfiguration.cs:69,118`); connection strings never leak into error text (`DatabaseGateway.cs:298`, enforced in messages); credential scrubbing shared, not duplicated (`CredentialRedactor.cs`); API-key auth delegated to the shared `ZB.MOM.WW.Auth.ApiKeys` package (`InboundAPI.csproj:34`) with peppered-HMAC constant-time verification per spec (`Component-InboundAPI.md:66` — the package itself is outside this repo and was not independently verified), generic 401 messages, and byte-identical 403s for not-found vs not-in-scope with the DB lookup ordered *after* the in-memory scope check to close the timing oracle (`EndpointExtensions.cs:117-156`). + +## Underdeveloped Areas + +1. **Script trust boundary is static-only.** `ForbiddenApiChecker` is candid: semantic + reflection-gateway hardening, "not a true sandbox… genuine containment needs a runtime boundary" (`ForbiddenApiChecker.cs:24-29`). Inbound scripts run in-process on the central node with `Microsoft.CSharp` dynamic binder referenced (`InboundScriptExecutor.cs:172-184`). The deferred restricted-`AssemblyLoadContext`/out-of-process option remains the single biggest security-hardening gap on this boundary. +2. **No cache-coherence mechanism for compiled handlers** (the High finding above) — no revision hash, no cluster invalidation message, no recompile-on-activation hook in `IActiveNodeGate`. +3. **Per-SMS retry settings are dead fields.** `SmsConfiguration.MaxRetries/RetryDelay` are persisted and transported but never read; SMTP config governs retry for *all* notification types (`NotificationOutboxActor.cs:358-401`; documented as deferred in `Component-NotificationService.md:65`). +4. **Twilio delivery is accept-only.** No status-callback webhook, no per-recipient delivery state (both documented as out-of-scope/future — `Component-NotificationService.md:93,102`). Also `AccountSid` is interpolated un-escaped into the request URI (`SmsNotificationDeliveryAdapter.cs:131`) — harmless for a valid SID, but a validation regex at config-save time would close the door. +5. **DelmiaNotifier duplicate-delivery semantics undocumented** (Stability finding); AOT publish is Windows-only and the live smoke is manual (`2026-06-26-delmia-recipe-notifier-design.md:133-136,150-151`) — no CI artifact for the actual shipped binary. +6. **Test gaps** (coverage is otherwise broad and genuinely good across all five test projects — endpoint gating, content-type case-sensitivity, schema-`$ref` runtime resolution, failover loop, both adapters, all four classifiers): nothing exercises the failover/staleness scenario for compiled handlers, nothing exercises an oversized outbound response body, and no test covers a multi-recipient SMS list with a mid-list transient (the duplicate-amplification path). +7. **API key rotation** is enable/disable + scope-set only (`ManagementActor.cs:2685+`); a documented dual-key rotation procedure (create → migrate caller → disable) would round out the ops story — the `sbk__` design supports it, it's just unwritten. + +## Prioritized Recommendations + +1. **[High] Make the inbound handler cache self-healing.** Store the compiled delegate together with a hash of the script text; on each request compare against the freshly-fetched `ApiMethod.Script` and recompile on mismatch (also purging `_knownBadMethods`). One change fixes failover staleness, direct-SQL staleness, and stale-bad-method staleness simultaneously; the DB row is already fetched per request so the marginal cost is one string hash. +2. **[High] Close the two ESG spec gaps** — per-system `Timeout` column (entity + EF migration + honored in `InvokeHttpAsync`) and either implement `{param}` path templating or amend `Component-ExternalSystemGateway.md`. Per repo rules, doc and code must travel together. +3. **[Medium] Cap outbound response buffering** in `ExternalSystemClient` (`ResponseHeadersRead` + bounded read, reusing the audit-capture pattern). +4. **[Medium] Bound the SMS blast radius**: short-circuit the recipient loop on first transient, and add bounded parallelism (or per-notification concurrency in the dispatch sweep) so one dead endpoint cannot stall the whole outbox. +5. **[Medium] Replace colon-split `AuthConfiguration` parsing with structured JSON** (the entity already claims to be JSON) with a backward-compatible fallback. +6. **[Medium] Handle abandoned inbound executions**: defer DI-scope disposal to actual handler completion, and add an orphaned-execution counter to the health snapshot. +7. **[Low] Document DelmiaNotifier's at-least-once failover semantics** and verify the deployed `DelmiaRecipeDownload` script is idempotent. +8. **[Low] Small consistency fixes**: `IOptionsMonitor` in `InboundApiEndpointFilter`; token parameter on `ErrorClassifier.IsTransient(Exception)`; configurable OAuth2 authority/scope; single-active-SMTP-config enforcement; implement or de-spec the inbound `code` error field. diff --git a/archreview/07-ui-management-security.md b/archreview/07-ui-management-security.md new file mode 100644 index 00000000..5555186a --- /dev/null +++ b/archreview/07-ui-management-security.md @@ -0,0 +1,126 @@ +# Architecture Review 07 — Management & Presentation Surface + +**Domain:** Central UI (Blazor Server), Management Service (ManagementActor + HTTP API), CLI, Security & Auth +**Reviewed:** `src/ZB.MOM.WW.ScadaBridge.CentralUI`, `src/ZB.MOM.WW.ScadaBridge.ManagementService`, `src/ZB.MOM.WW.ScadaBridge.CLI`, `src/ZB.MOM.WW.ScadaBridge.Security`, design docs `docs/requirements/Component-CentralUI.md`, `Component-ManagementService.md`, `Component-CLI.md`, `Component-Security.md`, and the corresponding test projects. + +--- + +## Scope + +Deep read of the ManagementActor (all 3,157 lines), the HTTP management/audit endpoints and authenticator, the full Security project (JWT service, cookie session validator, authorization policies, role mapper, service registration), the secured-writes end-to-end path (UI page → `SecuredWriteService` → actor handlers), the debug-stream path (`DebugStreamHub`, `DebugView.razor`), representative Blazor pages (Alarm Summary, Health, Secured Writes, Notification Report), UI-side trust-boundary services (`BrowseService`), the CLI HTTP client and command wiring, and deployed configuration in `docker/` and `deploy/wonder-app-vd03/`. Spec-vs-code comparison against the four component documents. + +## Maturity Verdict + +This is a **mature, carefully engineered surface with a small number of sharp edges**. The security layer is genuinely good — fail-fast key validation, a single shared claim builder for login and refresh, a deliberate and well-documented idle/refresh model, DB-persisted Data Protection keys, curated-vs-generic error mapping that prevents internal-detail leaks, and a secured-writes implementation with a real CAS race guard, server-side no-self-approval, and a TOCTOU protocol re-check at execute time. The Blazor code shows unusual discipline for disposal, thread marshalling, and best-effort degradation. The problems that remain are not sloppiness but *asymmetries*: a production deployment artifact that ships with authentication disabled, a per-site artifact-deploy option that is silently ignored server-side, secret-projection hygiene applied to SMTP/SMS/API keys but not to data connections / external systems / DB connection strings, and a secured-write "Expired" state that exists in the UI and spec but nowhere in the code. Test depth is thin relative to the actor's ~150-command surface. + +--- + +## Findings — Dimension 1: Stability + +### [High] S1 — Bundle import via `/management` can 504 while the import keeps applying +`ManagementEndpoints.HandleRequest` Asks the actor with a configurable timeout that defaults to **30 s** (`ManagementEndpoints.cs:18`, `ResolveAskTimeout` at `:28-33`), and neither `docker/central-node-a/appsettings.Central.json` nor `deploy/wonder-app-vd03/appsettings.Central.json` sets `ScadaBridge:ManagementService:CommandTimeout`. The CLI's bundle commands use a **5-minute client timeout** (`CLI/Commands/BundleCommands.cs:16`) precisely because imports are slow — but the server-side Ask expires at 30 s, returns `504 TIMEOUT` (`ManagementEndpoints.cs:107-113`), **and the actor's `ProcessCommand` task keeps running** (PipeTo is not cancelled), so `HandleImportBundle` (`ManagementActor.cs:2949-3015`) continues applying the bundle after the caller was told it timed out. Failure scenario: an operator imports a large bundle, gets a 504, retries — under `--on-conflict rename` the second import produces a duplicate `-imported-` artifact set; under `overwrite` it double-applies. The same ambiguity applies to fleet-wide `DeployArtifacts`. Fix: raise the configured timeout for transport/deploy commands (per-command timeout in the envelope), or make import idempotent on a client-supplied import id. + +### [High] S2 — Pending secured writes never expire; the "Expired" status is fictional +The UI's terminal-status list and badge mapping include `"Expired"` (`CentralUI/Components/Pages/Operations/SecuredWrites.razor:265, 411`), and Component-CentralUI.md §Secured Writes lists "Executed / Failed / Rejected / Expired" — but **no code anywhere sets a row to Expired** (repo-wide grep: the only two hits are the UI file). `HandleApproveSecuredWrite` (`ManagementActor.cs:1099-1246`) will approve a `Pending` row of any age. Failure scenario on a live SCADA surface: an Operator submits a setpoint write, it sits unverified over a weekend while the process moves to a different state, and a Verifier approves the stale value days later — the confirmation dialog (`SecuredWrites.razor:342-347`) shows the value but not its age. A pending-write TTL (with a sweeper or an at-approve age check) is the obvious missing piece; the status enum already reserved the name. + +### [Medium] S3 — Poll timers can overlap their own refreshes +`Health.razor:521-528` and `AlarmSummary.razor:283-291` both use `System.Threading.Timer` with a fixed period whose callback does `InvokeAsync(async () => { await RefreshAsync(); ... })` with no reentrancy guard. `AlarmSummary.RefreshAsync` fans out per-instance snapshot Asks (site offline ⇒ each fetch waits its full Ask timeout); with a 15 s period and a >15 s degraded fetch, ticks stack up on the dispatcher and the page issues overlapping fan-outs against an already-struggling site. A `PeriodicTimer` loop or an `Interlocked` in-flight flag would bound this. (Disposal itself is handled correctly on both pages — `Health.razor:757-760`, `AlarmSummary.razor:382`.) + +### [Medium] S4 — 200 MB request bodies are buffered as strings, then multiplied +The management endpoint raises the body cap to **200 MB** for transport bundles (`ManagementEndpoints.cs:50, 58-62`) and then reads the whole body with `StreamReader.ReadToEndAsync()` into a string (`:77-81`); the actor then base64-decodes to a `byte[]` (`ManagementActor.cs:3105-3119`), wraps in a `MemoryStream`, and export goes the other way — `stream.CopyToAsync(ms); ms.ToArray(); Convert.ToBase64String(bytes)` (`ManagementActor.cs:2902-2905`), which is ~4 copies of the artifact resident simultaneously (bytes, MemoryStream buffer, base64 string, JSON envelope string). A concurrent pair of large imports on a central node is a realistic OOM/LOH-fragmentation path. Streaming multipart upload (as the audit export already streams downloads — `AuditEndpoints.cs:183+`) is the right long-term shape. + +### [Low] S5 — Fire-and-forget SignalR sends swallow failures silently +`DebugStreamHub.SubscribeInstance` pushes events with `_ = hubClients.Client(connectionId).SendAsync(...)` (`DebugStreamHub.cs:204-221`). Deliberate (documented) — but a persistently failing client produces zero telemetry. Elsewhere the hub is exemplary: transient-hub-instance pitfall avoided via `IHubContext` (`:198-200`), unsubscribe on disconnect (`:255-259`). + +### Positive stability observations +- `DebugView.razor` is a model Blazor Server component: `_disposed` flag set before stream stop, `SafeInvokeAsync` guarding `ObjectDisposedException`, dictionary mutation marshalled onto the render thread with the race explicitly documented (`DebugView.razor:511-544, 589-611`). +- Failover transparency is real: Data Protection keys persist to the config DB (`ConfigurationDatabase/ServiceCollectionExtensions.cs:75-76`), the cookie principal is self-contained, and the ManagementActor runs on every central node (stateless, per spec). +- `AlarmSummaryService` implements the spec'd fan-out faithfully: `SemaphoreSlim(8)` cap, partial-results tolerance, caller-cancel propagation (`AlarmSummaryService.cs:26, 68-84, 112-123`). + +--- + +## Findings — Dimension 2: Performance + +### [Medium] P1 — Every CLI/management/audit HTTP request performs a full LDAP bind +`ManagementAuthenticator.AuthenticateAsync` binds against LDAP and re-runs role mapping on **every** request (`ManagementAuthenticator.cs:107-118`); the audit endpoints and `DebugStreamHub.OnConnectedAsync` (`DebugStreamHub.cs:118-137`) do the same. This is the documented stateless design, but there is no negative-result cache, rate limit, or lockout — so (a) a scripted CLI loop hammers the directory once per command, and (b) `POST /management` and `POST /auth/token` (`AuthEndpoints.cs:99-148`) are unauthenticated LDAP-bind oracles usable for password spraying at whatever rate the LDAP server tolerates. A short-TTL bind cache or a per-IP failure throttle would address both. + +### [Medium] P2 — Unbounded list handlers materialize whole tables +Several read handlers return `GetAll*` with no paging: `HandleListTemplates` (`ManagementActor.cs:507-511`), `HandleListInstances` (`:718-729`, post-filtered in memory for scoped users), `HandleQueryDeployments` unfiltered branch loads *all* deployment records *and all instances* (`:2031-2060` — the N+1 was fixed, but it is still two full-table loads per query), and `HandleExportBundle` loads every template/script/system/site/instance to resolve names (`:2818-2831`). Fine at current fleet size; a 10k-instance estate will feel it on every CLI `instance list`. The audit surface shows the house already knows the right pattern (keyset paging, `MaxPageSize` 1000 — `AuditEndpoints.cs:59-66`). + +### [Medium] P3 — `ListSecuredWrites` hard-caps at 200 rows with no paging +`HandleListSecuredWrites` does `QueryAsync(status, siteId, skip: 0, take: 200)` (`ManagementActor.cs:1280-1286`) and the page loads *everything* then splits pending/history client-side (`SecuredWrites.razor:303-308`). Once >200 rows exist, history silently truncates — an **audit-adjacent** table silently losing visibility of older decisions. No pagination controls exist on the page. + +### [Low] P4 — Alarm Summary re-sorts/filters per render +`FilteredRows().ToList()` runs inside the render body (`AlarmSummary.razor:170`) — recomputed on every `StateHasChanged`, including each 15 s tick. Cheap at hundreds of rows; worth memoizing if sites grow. No virtualization on any table in the domain, but page sizes are mostly bounded (Notification Report pages at 50 — `NotificationReport.razor:352`). + +### Positive performance observations +- Audit export streams server-side in 1000-row pages and the CLI streams to disk with `ResponseHeadersRead` (`AuditEndpoints.cs:66`, `ManagementHttpClient.cs:209-210`). +- The actor never blocks its dispatcher: `HandleEnvelope` role-checks synchronously then `PipeTo`s the async work (`ManagementActor.cs:97-120`), so the single actor instance is not a mailbox bottleneck for I/O-bound commands — commands execute concurrently on the thread pool with per-command DI scopes (`:127-132`). This is the correct pattern and worth calling out as such. + +--- + +## Findings — Dimension 3: Conventions & Correctness + +### [Critical] C1 — Production deployment artifact ships with authentication disabled +`deploy/wonder-app-vd03/appsettings.Central.json:40` sets `"DisableLogin": true` (and `:38` `"RequireHttpsCookie": false`). Per `Component-Security.md` §Dev Disable-Login Flag there is **no environment guard** — only a startup log warning — and the bypass is honoured by *every* surface: the cookie scheme (`AutoLoginAuthenticationHandler`), `POST /management` (`ManagementAuthenticator.cs:52-76`), the audit REST endpoints, and the debug-stream hub (`DebugStreamHub.cs:82-91`). On this host, every anonymous HTTP request is authenticated as `multi-role` with **all roles system-wide — including Operator and Verifier**, which nullifies the two-person secured-write control on a box that can relay MxGateway device writes. `deploy/` is described by CLAUDE.md as "Production/on-host deployment artifacts". Even if wonder-app-vd03 is a staging box today, an unauthenticated SCADA control surface should not be one config-file copy away from production. Recommendation: add an environment guard (refuse `DisableLogin=true` unless `ASPNETCORE_ENVIRONMENT=Development` or an explicit `IUnderstandThisDisablesAuth` second key), and scrub the deploy artifact. + +### [High] C2 — `deploy artifacts --site-id` is silently ignored; artifacts always deploy fleet-wide +`MgmtDeployArtifactsCommand` carries `int? SiteId` (`Commons/Messages/Management/DeploymentCommands.cs:3`) and both CLI commands send it (`CLI/Commands/SiteCommands.cs:209`, `CLI/Commands/DeployCommands.cs:51`; documented in `Component-CLI.md` and the CLI README as `[--site-id ]`). But `HandleDeployArtifacts` never reads `cmd.SiteId` — it calls `svc.DeployToAllSitesAsync(user)` unconditionally (`ManagementActor.cs:2006-2013`), and `ArtifactDeploymentService.DeployToAllSitesAsync(string user, CancellationToken)` (`ArtifactDeploymentService.cs:227-229`) has no site parameter. Two consequences: (1) **contract violation** — an operator targeting one site pushes shared scripts/external systems/DB connections to *every* site; (2) **authorization gap** — the command also has no `EnforceSiteScope` call, so a *site-scoped* Deployer can trigger a fleet-wide artifact deployment. The Central UI's per-site "Deploy Artifacts" button presumably uses a different path; the management surface needs `DeployToSiteAsync(siteId)` plus a scope check. + +### [High] C3 — Secret projection is inconsistent: connection credentials leak to any authenticated user and into the audit log +The actor is scrupulous about SMTP credentials, Twilio auth tokens, and API-key hashes — all projected to credential-free shapes before response *and* audit (`SmtpConfigPublicShape` `ManagementActor.cs:1787-1801`, `SmsConfigPublicShape` `:1841-1853`, `HandleListApiKeys` `:1955-1967`). But three sibling entity families get no such treatment: +- **Data connections** — `HandleListDataConnections`/`HandleGetDataConnection` (`:1401-1416`) return the raw entity, whose `PrimaryConfiguration`/`BackupConfiguration` JSON embeds `OpcUaUserIdentityConfig.Password` and `CertificatePassword` in plaintext (`Commons/Types/DataConnections/OpcUaUserIdentityConfig.cs:13,17`). List/Get are **read-only commands with no role gate** (`GetRequiredRole` default `:240-241`), so a Viewer- or Operator-only principal can read every site's OPC UA credentials via one CLI call. Create/Update also audit the full entity as afterState (`:1429, :1445`), persisting the passwords into the audit log readable by `OperationalAudit` roles. +- **External systems** — `AuthConfiguration` (API keys / Basic credentials) returned raw to any authenticated user and audited verbatim (`:1587-1624`). +- **Database connections** — full `ConnectionString` (SQL credentials) returned raw in List/Get (`:2526-2536`); the audit shape here *was* trimmed to `{Id, Name}` (`:2544, :2557`), which shows the team knows the pattern — it just wasn't applied to the response. +Recommendation: apply the established `*PublicShape` pattern (config-with-secrets-elided + `HasCredentials` flag) to all three, and gate Get-with-secrets behind Admin if the UI editor needs round-tripping. + +### [Medium] C4 — Spec drift: browse/verify/cert-trust commands are documented as ManagementActor commands but only exist UI-side +`Component-ManagementService.md` §Remote Queries lists `BrowseNodeCommand`, `SearchAddressSpaceCommand`, `VerifyEndpointCommand`, `TrustServerCertCommand`/`RemoveServerCertCommand`/`ListServerCertsCommand` with Design/Admin role gates. The actor's `DispatchCommand` has **no handlers for any of them** — they fall through to `NotSupportedException` (`ManagementActor.cs:423`), and since `ManagementCommandRegistry` auto-registers every `*Command` in the namespace by reflection (`ManagementCommandRegistry.cs:66-79`), a CLI `POST /management {"command":"TrustServerCert"}` deserializes fine and then returns a generic `COMMAND_FAILED` internal error. The real implementations live in Central UI services (`BrowseService.cs`, `EndpointVerificationService.cs`, `CertManagementService.cs`) with hand-rolled `HasClaim` role guards (`BrowseService.cs:48-55, 103-110`). Net effect: the CLI cannot browse address spaces or manage cert trust (a gap for the automation story the CLI exists for), and the component doc misdescribes the enforcement point. Fix the doc or add actor handlers. + +### [Medium] C5 — Role-name comparison asymmetry between UI policies and the actor (acknowledged, still live) +ASP.NET `RequireClaim` values are compared **ordinal case-sensitive** (`AuthorizationPolicies.cs:143-169`), while the ManagementActor compares roles **OrdinalIgnoreCase** (`ManagementActor.cs:105, 440` etc.), and `ValidateMappingRole` deliberately preserves the stored row's verbatim casing while validating membership case-insensitively — the comment at `ManagementActor.cs:1904-1911` explicitly defers the asymmetry. Failure scenario: an admin creates a mapping row `deployer` (lowercase) via CLI; the affected users pass every ManagementActor gate but fail every `RequireDeployment` UI policy — a confusing split-brain where CLI works and the UI 403s. Since the write path already validates against `Roles.All`, the cheap fix is to **canonicalize the stored casing** at that single write path. + +### [Medium] C6 — Area-management role gate contradicts both design docs +`CreateAreaCommand`/`UpdateAreaCommand`/`DeleteAreaCommand` are gated **Designer** (`ManagementActor.cs:187, 207`), but `Component-ManagementService.md` §Authorization ("Admin role required for: site management, area management…") and `Component-Security.md` §Administrator ("Manage area definitions per site") both assign areas to **Admin**, and Component-CentralUI.md titles the workflow "Area Management (Admin Role)". Either the code or three documents are wrong; per the repo's own propagate-together rule this should be reconciled in one change. + +### [Low] C7 — `HandleQueryEventLogs` passes `InstanceName` into a positional slot commented "InstanceId filter" +`ManagementActor.cs:2757-2766` builds `EventLogQueryRequest(..., cmd.InstanceName, /* InstanceId filter */ ...)`. If the site-side filter expects an id, name-based CLI filtering silently matches nothing; if it expects a name, the comment/param name is wrong. Worth a five-minute verification against `SiteEventLogging`. + +### [Low] C8 — Stale CLAUDE.md note on SecuredWrite `SourceNode` +CLAUDE.md says "SecuredWrite audit rows currently leave `SourceNode` NULL — a logged follow-up", but `EmitSecuredWriteAuditAsync` now routes through `ICentralAuditWriter`, which stamps `SourceNode` from `INodeIdentityProvider` (documented at `ManagementActor.cs:999-1013`). The follow-up appears done; the project memory should be updated so future sessions don't re-fix it. + +### Positive convention observations +- **No third-party Blazor component libraries** — the CentralUI csproj references only Roslyn packages and the in-family `ZB.MOM.WW.Theme` (`ZB.MOM.WW.ScadaBridge.CentralUI.csproj:19-21`); tables, tree views, dialogs, toasts, and the KPI chart are all custom, per the repo rule. +- Options pattern honoured throughout; component libraries take no `IConfiguration` (`Security/ServiceCollectionExtensions.cs:26-37`); `SecurityOptionsValidator` fail-fast with a genuinely well-reasoned invariant (refresh threshold < idle window, `SecurityOptions.cs:118-142`). +- Error-surface hygiene is exemplary: `MapFault` returns curated `ManagementCommandException` messages verbatim and collapses everything else to a correlation-id-bearing generic message (`ManagementActor.cs:137-158`) — no raw exception text reaches the CLI/UI for unanticipated faults. +- The secured-write approve handler is defensive in exactly the right places: no-self-approval **before** the CAS (`:1113-1115`), CAS via `TryMarkApprovedAsync` (`:1121`), MxGateway protocol **re-checked at execute** against submit-time reconfiguration (`:1138-1165`), value-type and decode failures contained so a row is never stuck `Approved` (`:1171-1202`), transport exceptions contained (`:1206-1226`). +- `ResolveRolesCommand` deliberately not dispatched with the security rationale documented (`:418-422`) — good trust-boundary thinking. +- JWT implementation pins issuer/audience, zero clock skew, disables in/outbound claim mapping symmetrically, enforces ≥256-bit keys at construction (`JwtTokenService.cs:54-78, 124-171`); the cookie path's login and mid-session refresh share `SessionClaimBuilder` so claim shapes cannot drift (`CookieSessionValidator.cs:200-210`, `AuthEndpoints.cs:84-89`); refresh failures fail-open to *current* roles and only idle-timeout rejects (`CookieSessionValidator.cs:132-152`) — matching the documented policy exactly. +- LDAP is delegated to the shared hardened `ZB.MOM.WW.Auth.Ldap` library (bind-then-search, escaping, fail-closed — per `Security/ServiceCollectionExtensions.cs:49-59`); no bespoke LDAP string assembly exists in this repo, so injection surface is centralized in the audited library. + +--- + +## Findings — Dimension 4: Underdeveloped Areas + +1. **Secured-write lifecycle** — no expiry (S2), no paging (P3), history visible to any authenticated user (`ListSecuredWritesCommand` falls through to the no-role gate, `ManagementActor.cs:238-241`) — tag values on the history table may themselves be process-sensitive. Also nothing prevents the *same person* holding both Operator and Verifier roles via two LDAP groups; the control is only "not the same row's submitter", which is the documented design but worth restating as a deployment-configuration hazard. +2. **ManagementService test depth** — 11 test files (`tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/`) against a 3,157-line actor dispatching ~150 commands. `SecuredWriteHandlerTests` exists (good — covers the flagship control), but there is no visible exhaustive matrix test asserting `GetRequiredRole` for every command type — precisely the test that would have caught C2's missing scope check and would freeze the authz table against regressions. Security.Tests is 9 files; CookieSessionValidator is covered. +3. **CLI.Tests is not in the slnx** (`ZB.MOM.WW.ScadaBridge.slnx:24` includes only the CLI project) — a documented gotcha, but it remains a standing risk that a green solution run hides never-compiled CLI tests. +4. **Deferred follow-ups visible in code/docs**: aggregated live alarm stream for Alarm Summary (poll-only today — Component-CentralUI.md §Alarm Summary "logged as a follow-up"); central persistence of cert trust; live LDAP group re-query mid-session (Component-Security.md "Residual limitation"); Parquet export returns 501 (`AuditEndpoints.cs:202-210`); `audit verify-chain` is a no-op pending the hash chain. All are honestly documented — the gap is that no single tracking list ties them together. +5. **Login endpoints lack throttling** — `/auth/login`, `/auth/token`, `/management`, and the debug hub all do per-request LDAP binds with no failure throttle or lockout (P1). `/auth/login` also reflects the LDAP failure kind into the redirect query string (`AuthEndpoints.cs:41-45`); confirm `LdapAuthFailureMessages` does not distinguish "unknown user" from "bad password" (user enumeration). +6. **Accessibility/UX debt** — sortable headers are `` without keyboard handlers or `aria-sort` (`AlarmSummary.razor:176-179`); status conveyed by color+text badges is fine, but the custom tables have no `scope`/caption markup. Minor for an internal tool, but it is uniform debt across every custom grid. +7. **`docker/central-node-a/logs/` committed runtime logs** sit inside the deploy topology folder (noticed while scanning configs) — noise that will eventually leak something. + +--- + +## Prioritized Recommendations + +1. **[Critical] Guard `DisableLogin` and scrub `deploy/wonder-app-vd03`** — refuse the flag outside a Development environment (or require a second explicit ack key); remove `DisableLogin: true` from the on-host deploy artifact. (C1) +2. **[High] Implement per-site artifact deployment or reject the option** — honour `MgmtDeployArtifactsCommand.SiteId` (add `DeployToSiteAsync`) and add `EnforceSiteScope`; until then, make the handler *fail* when SiteId is supplied rather than silently deploying fleet-wide. (C2) +3. **[High] Apply the `*PublicShape` secret-elision pattern to DataConnection, ExternalSystem, and DatabaseConnection reads and audits**; consider role-gating those List/Get commands. (C3) +4. **[High] Add a pending-secured-write TTL** that transitions stale rows to the already-reserved `Expired` status, plus paging on the list. (S2, P3) +5. **[High] Fix the long-command timeout mismatch** — per-command Ask timeouts (transport/deploy get minutes), configure `CommandTimeout` in shipped configs, and/or make bundle import idempotent on an import id so a 504-then-retry is safe. (S1) +6. **[Medium] Add a `GetRequiredRole` matrix test** enumerating every registered command and asserting its gate — freezes the authorization table and catches future C2-style gaps mechanically. +7. **[Medium] Canonicalize role casing at the mapping write path** to close the UI-vs-actor case-sensitivity split. (C5) +8. **[Medium] Reconcile the area-management role gate** (code says Designer, three docs say Admin) and the browse/cert-command placement (docs say actor, code says UI-only) in one doc+code pass. (C4, C6) +9. **[Medium] Add LDAP-bind failure throttling** (per-IP or per-username backoff) in `ManagementAuthenticator` and the auth endpoints. (P1) +10. **[Low] Housekeeping** — reentrancy guards on the two poll timers (S3), update the stale SecuredWrite `SourceNode` memory note (C8), verify the `InstanceName`/`InstanceId` event-log filter slot (C7), pull `docker/*/logs` out of the tree, and put CLI.Tests into the slnx or CI explicitly. diff --git a/archreview/08-conventions-tests-underdeveloped.md b/archreview/08-conventions-tests-underdeveloped.md new file mode 100644 index 00000000..a4c4fa41 --- /dev/null +++ b/archreview/08-conventions-tests-underdeveloped.md @@ -0,0 +1,185 @@ +# Architecture Review 08 — Cross-Cutting Conventions, Test Posture, Underdeveloped Areas + +Reviewer domain: cross-cutting conventions, code organization, test posture, repo-wide underdeveloped-areas sweep, docs-code drift. Individual subsystems are covered in depth by the other seven reviews; this one covers the seams and the whole. + +## Scope + +- `src/` — all 26 projects: Commons structure, project reference graph, options pattern, POCO/repository conventions, UTC and correlation-ID discipline, message-contract evolution. +- `tests/` — all 30 test projects: breadth, quality sampling, integration/performance posture, slnx membership. +- Repo-wide grep sweep for TODO / FIXME / HACK / NotImplementedException / "deferred" / "follow-up" / "not shipped", plus deferred-work inventory from `docs/plans/` and `CLAUDE.md`. +- Docs-code sync spot checks: README component table, `docs/requirements/Component-*.md` (KpiHistory, StoreAndForward, Transport, ClusterInfrastructure), `docs/components/` reference docs. + +Method: `.slnx` and `.csproj` graph extraction, directory-structure listing, targeted greps over `src/`, `tests/`, `docs/`, and direct reads of the files behind every claim below. Read-only except this report. + +## Maturity Verdict + +This is an unusually disciplined codebase for its size (~190k src LOC, ~156k test LOC, ~5,000 test cases across 30 projects). The documented conventions are not aspirational — they are actually followed: Commons is genuinely persistence-ignorant, the Types/Interfaces/Entities/Messages hierarchy is real, UTC discipline is total (zero `DateTime.Now` in src), correlation IDs are on every cross-cluster request/response plus the `ManagementEnvelope`, and the project reference graph has no site→central contamination. The TODO surface is astonishingly small (two real TODOs in all of src) and almost every deferral is deliberately documented at the deferral site with a pointer to the design doc. The gaps that exist are narrow and specific: one genuine spec-vs-code contradiction (Transport claims Area membership travels in bundles; the exporter always emits `null`), the CLI.Tests slnx exclusion that keeps 279 tests out of every solution-level test run, options-validation coverage on only ~6 of ~20 option classes, and a thin PerformanceTests project that is a correctness-at-small-scale suite rather than a performance suite. Deferred-work inventory is sizeable (~20 tracked items) but almost entirely intentional and logged; only two or three items carry real operational risk. + +--- + +## 1. Conventions & Organization + +### 1.1 Commons hierarchy — followed, with undocumented extensions [Low] + +`src/ZB.MOM.WW.ScadaBridge.Commons/` follows the documented `Types/`, `Interfaces/`, `Entities/`, `Messages/` layout with domain-area subfolders exactly as `Component-Commons.md` describes (284 .cs files; e.g. `Entities/{Audit,Deployment,ExternalSystems,InboundApi,Instances,Kpi,Notifications,Schemas,Scripts,SecuredWrites,Security,Sites,Templates}`, `Messages/{Artifacts,Audit,Communication,DataConnection,DebugView,Deployment,Health,InboundApi,Instance,Integration,Lifecycle,Management,Notification,RemoteQuery,ScriptExecution,Streaming}`). + +Three top-level folders exist beyond the documented four: `Observability/`, `Serialization/` (protocol endpoint-config serializers), and `Validators/`. These are reasonable homes, but the convention doc lists only the four canonical folders. Either document them in `Component-Commons.md` or fold them in (`Serialization` arguably belongs under `Types/`). + +### 1.2 POCO persistence-ignorance — verified clean [Pass] + +Grep for EF Core / DataAnnotations attributes across Commons returns only one hit, and it is an XML-doc `` to `DbUpdateConcurrencyException` in `src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IDeploymentManagerRepository.cs:64` — a documentation reference, not a dependency. The Commons csproj has zero EF package references (its only PackageReference is the shared `ZB.MOM.WW.Audit` lib). Entities are genuinely persistence-ignorant. + +### 1.3 Repository interfaces vs implementations — 1:1, with a deliberate site-side exception [Low] + +14 repository interfaces in `Commons/Interfaces/Repositories/`, 14 implementation files in `ConfigurationDatabase/Repositories/` — exact match. Two additional implementations live in `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/` (`SiteExternalSystemRepository`, `SiteNotificationRepository`): read-only SQLite-backed site-side implementations that throw `NotSupportedException` on writes. This is a sensible pattern (same interface, site-local store), but it is an undocumented exception to the "implementations in Configuration Database" rule. + +**Vestigial seam** [Low/Medium]: `SiteNotificationRepository` (registered scoped in `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs:65`) reads the site-local `notification_lists` table — but per the central-only notification design, `SiteStorageService.PurgeCentralOnlyNotificationConfigAsync` (`src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs:703`) deliberately empties that table on every artifact apply. So the registered repository can only ever return empty results, and `StoreNotificationListAsync` (line 721) still exists as a write path into a table the design says must stay empty. The purge is well-commented and correct as a security cleanup; the live read/write surface around an intentionally-empty table is dead weight and a trap for a future maintainer. Recommend removing `StoreNotificationListAsync`/list-read paths or marking them `[Obsolete]` with the design rationale. + +### 1.4 Project reference graph — sane; no site→central contamination [Pass, one Low note] + +Extracted from all 26 csproj files: + +- **Commons is a leaf** (no project refs). Everything depends on it; it depends on nothing. +- **Site-side projects never reference central-only projects.** `SiteRuntime` → Commons, Communication, DataConnectionLayer, ScriptAnalysis, HealthMonitoring, SiteEventLogging, StoreAndForward — critically, **no** reference to `ConfigurationDatabase` (EF/central SQL), `ManagementService`, `CentralUI`, or `TemplateEngine`. Same for DataConnectionLayer, StoreAndForward, SiteEventLogging. +- **Central-only stack is correctly layered**: AuditLog/SiteCallAudit/Transport → ConfigurationDatabase; CentralUI → the central components + ManagementService; Host references everything (composition root, as designed). +- **CLI → Commons only** — correctly thin (HTTP client to the Management API). +- **DelmiaNotifier → nothing** — correctly standalone/BCL-only. + +One smell [Low]: `Communication` → `HealthMonitoring` (`src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs:13`). A transport layer depending on a monitoring component is a mild layering inversion; the health-aggregator abstraction it consumes could live in Commons. Cosmetic [Low]: `ManagementService.csproj` uses a Windows-style `..\` separator for its InboundAPI reference while every other reference in the repo uses `../` — harmless to MSBuild, inconsistent style. + +### 1.5 Options pattern — ownership right, validation coverage partial [Medium] + +Every component owns its options class in its own project (31 `*Options.cs` files across src; none in Commons) — the ownership convention is fully honored. But **eager validation is wired for only ~6 components**: `IValidateOptions<>`/`ValidateDataAnnotations`/`ValidateOnStart` appear only in AuditLog, KpiHistory, NotificationOutbox (+SmsOptions), HealthMonitoring, Security, ClusterInfrastructure, plus the Host `StartupValidator`. Components binding options without a validator include Communication, DataConnectionLayer, DeploymentManager, ExternalSystemGateway, InboundAPI, ManagementService, NotificationService, SiteCallAudit, SiteEventLogging, SiteRuntime, StoreAndForward, and Transport (e.g. `src/ZB.MOM.WW.ScadaBridge.Transport/ServiceCollectionExtensions.cs:25` — `AddOptions().BindConfiguration(...)` with no `.ValidateOnStart()`). A malformed `appsettings` section in those components surfaces at first use rather than at startup, which is exactly what the Host's readiness-gating philosophy is supposed to prevent. Recommend a sweep adding validators (the HealthMonitoring/ClusterInfrastructure `IValidateOptions` implementations are ready-made patterns to copy). + +### 1.6 UTC timestamps — total compliance [Pass] + +Zero occurrences of `DateTime.Now`/`DateTimeOffset.Now` (non-Utc) in all of src (grep across .cs and .razor). Every timestamp uses `UtcNow` or stores `datetime2`/ISO-8601 UTC. + +### 1.7 Correlation IDs — convention followed [Pass] + +96 `CorrelationId` declarations across `Commons/Messages/` cover the full cross-cluster request/response surface (RouteToInstance, Get/SetAttribute(s), Subscribe*, WriteTag*, ParkedMessage*, NotificationOutboxQueries, SiteCallQueries, EventLogQuery, DeploymentStateQuery, WaitForAttribute, etc.). Management commands themselves carry no per-command field, but they always travel inside `ManagementEnvelope(User, Command, CorrelationId)` with `ManagementSuccess`/`ManagementError`/`ManagementUnauthorized` echoing it (`Commons/Messages/Management/ManagementEnvelope.cs:7-11`) — the convention holds at the envelope level. + +### 1.8 Message contract evolution — convention by discipline, not enforcement [Low] + +Messages are plain C# records with defaulted trailing parameters (additive-friendly, e.g. `CreateDataConnectionCommand(..., string? BackupConfiguration = null, int FailoverRetryCount = 3)` in `Commons/Messages/Management/DataConnectionCommands.cs`). The Akka HOCON in `Host/Actors/AkkaHostedService.cs:233` configures no custom serializer, so ClusterClient traffic rides Akka.NET's default JSON serialization — tolerant of additive fields. The gRPC side does have contract-lock tests (`tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Protos/CachedTelemetryProtoTests.cs`, `PullAuditEventsProtoTests.cs`), but there is no equivalent guard (snapshot/round-trip tests against old shapes) for the ClusterClient record contracts. Given the "system-wide artifact version skew across sites is supported" design decision, a small contract-compatibility test suite for the highest-traffic ClusterClient messages would convert the additive-only rule from convention to enforcement. + +### 1.9 Naming/namespace consistency [Pass] + +Consistent `ZB.MOM.WW.ScadaBridge.` namespaces mirroring folder structure in every file sampled (SiteRuntime, Transport, Commons, KpiHistory, ManagementService). Test projects mirror src names 1:1 plus IntegrationTests/PerformanceTests/PlaywrightTests/Transport.IntegrationTests. + +--- + +## 2. Test Posture & Quality + +### 2.1 Breadth — every component has a test project; volume is high + +30 test projects; ~5,050 `[Fact]`/`[Theory]` methods; ~156k test LOC against ~190k src LOC (src figure inflated by ~75k in ConfigurationDatabase, mostly EF migrations). Per-project counts (files / LOC / test methods): + +| Project | Tests | Notes | +|---|---|---| +| CentralUI.Tests | 829 | bUnit page/component tests — largest suite | +| Commons.Tests | 465 | includes KpiSeriesBucketer, codecs, validators | +| SiteRuntime.Tests | 457 | Akka TestKit behavioral (see 2.2) | +| TemplateEngine.Tests | 433 | | +| CLI.Tests | 279 | **excluded from slnx** (see 2.4) | +| AuditLog.Tests | 237 | | +| ManagementService.Tests | 234 | | +| ConfigurationDatabase.Tests | 219 | incl. migration tests | +| InboundAPI.Tests | 218 | | +| Communication.Tests | 211 | incl. proto contract locks | +| DataConnectionLayer.Tests | 190 | | +| Host.Tests | 142 | incl. CompositionRootTests | +| Security.Tests | 133 | | +| NotificationOutbox.Tests / Transport.Tests | 122 / 124 | | +| StoreAndForward.Tests | 116 | | +| DeploymentManager.Tests | 98 | | +| HealthMonitoring.Tests | 84 | | +| IntegrationTests / ExternalSystemGateway.Tests | 69 / 69 | | +| Transport.IntegrationTests | 60 | | +| SiteEventLogging.Tests | 54 | | +| ScriptAnalysis.Tests / NotificationService.Tests / CentralUI.PlaywrightTests | 40 / 36 / 36 | | +| SiteCallAudit.Tests | 31 | thinnest per src LOC (1,642 loc / 31 tests) | +| DelmiaNotifier.Tests | 23 | | +| ClusterInfrastructure.Tests | 15 | src is only 191 LOC (wiring lives in Host) — proportionate | +| KpiHistory.Tests | 14 | bucketer/query/chart tests live with owners (Commons.Tests, CentralUI.Tests) — coverage is distributed, not missing | +| PerformanceTests | 10 | see 2.3 | + +Relatively thin spots: **SiteCallAudit.Tests** (31 tests for a reconciliation/KPI/relay component) and **DeploymentManager.Tests** (98 tests for the deployment pipeline, though much of its logic is exercised via ManagementService/Host/Integration suites). + +### 2.2 Quality sampling — real behavioral tests, not shallow + +`tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/AlarmActorTests.cs` is representative: full Akka TestKit harness, real compiled script library, `TestProbe` as the parent InstanceActor, asserts actual `AlarmStateChanged` messages with state/name payloads — genuine behavior, not constructor smoke tests. The SiteRuntime.Tests directory also carries targeted regression suites (`InstanceActorChildAttributeRaceTests`, `DeploymentManagerRedeployTests`, `DeploymentManagerMediumFindingsTests` — named for review findings, indicating review feedback gets locked in as tests). IntegrationTests covers real cross-cutting concerns: `CentralFailoverTests`, `SiteFailoverTests`, `DualNodeRecoveryTests`, `RecoveryDrillTests`, `ReadinessTests`, `AuthFlowTests`, `SecurityHardeningTests`, `AuditTransactionTests`, `NotificationOutboxFlowTests`, gRPC tests, plus a `ScadaBridgeWebApplicationFactory`. A 36-test Playwright E2E suite exists for the Central UI. + +### 2.3 PerformanceTests is real but minimal — a misnomer more than a stub [Medium] + +`tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/` has 3 test files / 10 tests: `HealthAggregationTests` (10 sites × 100 updates, **correctness assertions only, no timing**), `StaggeredStartupTests` (one `Stopwatch` assertion < 1000ms), and `AuditLog/HotPathLatencyTests` (the best of the three: `Stopwatch.GetTimestamp` p95-under-threshold on the site audit hot path; its own doc comment acknowledges "no BenchmarkDotNet" as a deliberate choice). Nothing here exercises the load-bearing performance claims of the design — 25s failover, gRPC streaming throughput, S&F drain rate, per-subscriber stream backpressure. Either rename the project to reflect its actual scope or grow it toward the design's stated performance envelope (failover time and stream throughput being the two highest-value additions). + +### 2.4 CLI.Tests still excluded from the slnx [High] + +Verified against `ZB.MOM.WW.ScadaBridge.slnx`: 29 of 30 test projects are members; `tests/ZB.MOM.WW.ScadaBridge.CLI.Tests` is absent (the src `CLI.csproj` *is* in the slnx). This is a documented gotcha in CLAUDE.md and memory, but the durable fix — adding one `` line — has not been made, so `dotnet test ZB.MOM.WW.ScadaBridge.slnx` silently skips 279 tests and a green solution run can mask CLI tests that never compiled. There is no evident reason to keep it out; add it. + +--- + +## 3. Underdeveloped Areas & Deferred-Work Inventory + +The in-code TODO surface is remarkably small: exactly **two real TODOs** in all of src (both the same issue — Transport area export, `EntitySerializer.cs:251/518`), zero `NotImplementedException` in production code paths (the one that existed was deliberately deleted — see the comment in `ClusterInfrastructure/ServiceCollectionExtensions.cs:34-47`), and zero FIXME/HACK. 39 "deferred/follow-up" comment mentions are almost all self-documenting pointers to logged decisions. + +Consolidated inventory (item, where noted, risk of leaving it): + +| # | Item | Where noted | Risk if left | +|---|---|---|---| +| 1 | **Transport: Area membership doesn't travel in bundles** — exporter hardcodes `AreaName: null`; importer machinery is fully built (`BundleImporter.cs:3104-3168` resolves/creates areas) | `src/.../Transport/Serialization/EntitySerializer.cs:251,518` (TODO); contradicts `Component-Transport.md:21` + CLAUDE.md #24 | **High** — spec and CLAUDE.md claim it ships; cross-env imports silently drop area organization; half-built feature invites confusion | +| 2 | CLI.Tests not in slnx | `ZB.MOM.WW.ScadaBridge.slnx`; CLAUDE.md gotcha | **High** — 279 tests invisible to solution-level CI/test runs | +| 3 | AuditLog reconciliation dials site NodeA only; NodeB failover endpoint selection deferred | `src/.../AuditLog/Central/SiteEnumerator.cs:32,64`, `ISiteEnumerator.cs:16` | **Medium** — during a site NodeA outage, telemetry-loss reconciliation (the safety net) is also unavailable; audit gaps persist until NodeA returns | +| 4 | Options validation missing on ~14 components (§1.5) | e.g. `Transport/ServiceCollectionExtensions.cs:25` | **Medium** — config typos surface at first use, not startup | +| 5 | `SmsConfiguration.MaxRetryCount` stored but not honored at dispatch (SMTP retry policy governs SMS) | `Commons/Entities/Notifications/SmsConfiguration.cs:36` | **Medium** — admin-visible knob that does nothing; misleading UI | +| 6 | Role-check case-sensitivity asymmetry: UI `RequireClaim` policies vs ManagementActor's case-insensitive check | `ManagementService/ManagementActor.cs:1910` | **Medium** — a role stored with non-canonical casing could pass one gate and fail the other; deliberately unaltered, separately deferred | +| 7 | SecuredWrite audit rows leave `SourceNode` NULL | CLAUDE.md (Security section, logged follow-up) | **Low/Medium** — per-node audit forensics blind spot for two-person writes specifically | +| 8 | Hash-chain tamper evidence (T1) | CLAUDE.md; `docs/plans/2026-05-20-audit-log-code-roadmap.md:12`; CLI `verify-chain` is a shipped no-op stub (`CLI/Commands/AuditVerifyChainHelpers.cs:7`) | **Low/Medium** — deferred to v1.x by locked decision; append-only DB roles are the current control; the no-op CLI verb slightly overstates capability | +| 9 | Parquet audit archival/export (T2) — endpoint returns 501 | `ManagementService/AuditEndpoints.cs:204-208`; CLAUDE.md | **Low** — deferred v1.x; 501 + CLI messaging handle it honestly | +| 10 | Aggregated **live** alarm stream for Alarm Summary (currently snapshot fan-out + poll) | `docs/plans/2026-06-18-m7-opcua-mxgateway-ux-design.md:252` | **Low/Medium** — operator page scales with instance count; fan-out Ask per refresh is the known cost | +| 11 | Central-persisted, auditable OPC UA server-cert trust (v1 is site-local PKI only) | m7 design follow-ups; CLAUDE.md | **Low** — broadcast-to-both-nodes covers HA; no central governance/audit of trust decisions | +| 12 | Native-alarm-source-override CSV bulk import | m7 design follow-ups; CLAUDE.md (M7 UX) | **Low** — attribute-override CSV shipped; parity gap only | +| 13 | `WaitForAttribute` quality-gated ("Good"-only) mode | `SiteRuntime/Scripts/ScriptRuntimeContext.cs:405` | **Low** — planned enhancement per spec §4.2 | +| 14 | `WaitForAttribute` not wired into CentralUI Test-Run sandbox | `docs/plans/2026-06-17-waitfor-deferred-items.md:222` | **Low** — Test-Run parity gap; production scripts unaffected | +| 15 | `BrowseNext` continuation: server's final-page signal not surfaced (client may issue one extra empty pull) | `Commons/Messages/Management/BrowseCommands.cs:34` | **Low** — one wasted round-trip | +| 16 | `StubOpcUaClient` throws on browse — test-infra gap for browse/search unit tests | m7 design "risks" (`2026-06-18-m7-opcua-mxgateway-ux-design.md:245`) | **Low** — limits offline coverage of browse UI | +| 17 | Unified notifications+site-calls outbox page | `2026-06-15-stillpending-completion-design.md:118` (M9 deferrals) | **Low** — two pages remain; explicit decision | +| 18 | Folder drag-drop | same, `[PERM]` permanently deferred | None — closed decision; menu reorder shipped | +| 19 | Bundle signing (asymmetric, non-repudiation), direct cluster-to-cluster pull, differential bundles | `docs/plans/2026-05-24-transport-design.md:402-408` | **Low** — v1 manifest hash + passphrase AES-GCM held sufficient | +| 20 | Deployment EXPIRED-but-unpurged row purge | `ConfigurationDatabase/Repositories/DeploymentManagerRepository.cs:310` ("deferred TODO" — workaround in place) | **Low** — code already compensates when reading | +| 21 | `SiteAuditBacklogReporter` threshold consolidation into `SqliteAuditWriterOptions` | `AuditLog/Site/SiteAuditBacklogReporter.cs:28` | **Low** — config-shape cleanup | +| 22 | KPI history downsampling (hourly rollups) | `Component-KpiHistory.md` (YAGNI) | **Low** — 90-day retention bounds table size | +| 23 | Vestigial site-side notification list read/write surface (§1.3) | `SiteRuntime/Persistence/SiteStorageService.cs:721`, `Repositories/SiteNotificationRepository.cs` | **Low/Medium** — dead-but-live API around an intentionally empty table | + +Overall: ~20 open items, of which only #1–#4 warrant action before the next release; the rest are consciously logged, mostly with honest in-product behavior (501s, no-op stubs that say so). + +--- + +## 4. Docs-Code Drift + +Drift is exceptionally low for a docs-as-spec repo. Spot checks: + +- **Component-KpiHistory.md vs code** — verified in-sync: `KpiHistoryOptions` defaults (SampleInterval 60s, RetentionDays 90 [1,3650], PurgeInterval 1d, DefaultMaxSeriesPoints 200 [2,5000]) match `src/.../KpiHistory/KpiHistoryOptions.cs` exactly, including range constraints; source placement (`IKpiSampleSource` impls with owners, bucketer in Commons, query service in CentralUI) matches the actual file layout and test placement. +- **Component-StoreAndForward.md vs code** — verified in-sync: the doc's nuanced notification-parking semantics (`DefaultMaxRetries` cap, `maxRetries: 0` escape hatch), `ICachedCallLifecycleObserver` telemetry hook, and tracking-table-vs-buffer split all correspond to the shipped files (`NotificationForwarder.cs`, `ParkedMessageHandlerActor.cs`, `StoreAndForwardService.cs`, `ReplicationService.cs`). +- **Component-ClusterInfrastructure.md vs code** — in-sync, and exemplary: the code carries a comment (`ClusterInfrastructure/ServiceCollectionExtensions.cs:34-47`) explaining that a dead throwing extension was removed *because* the doc's "Code Placement" section settled ownership in Host — docs and code cross-reference each other. +- **[High] Component-Transport.md:21 and CLAUDE.md #24 claim `Area` membership travels "carried by name"** — but the exporter emits `AreaName: null` unconditionally (`Transport/Serialization/EntitySerializer.cs:251-254`) with a TODO, and `BundleImporter.cs:511-518` confirms "Areas don't travel (see export TODO)". The import-side resolution (create-area-if-missing, `BundleImporter.cs:3104-3168`) is fully built, making the doc claim look shipped on casual inspection. Either finish the export (small: add Areas to `EntityAggregate` + the id→name lookup) or correct the doc/CLAUDE.md to say area membership is not yet carried. +- **[Low] `docs/components/` reference docs lag by three components**: 24 per-component reference docs exist but ScriptAnalysis (#25), KpiHistory (#26), and DelmiaNotifier (#27) are missing, while `README.md:111` claims "One doc per component (plus the shared TreeView)". +- **[Low] DelmiaNotifier breaks the requirements-doc naming convention**: no `docs/requirements/Component-DelmiaNotifier.md`; the README component table row 27 links to the project README instead. Defensible for an external tool, but CLAUDE.md's own Document Conventions say component documents live in `docs/requirements/` as `Component-.md` — either add a thin spec doc or note the exception in the conventions section. +- **README component table** — all 27 rows present and links resolve to existing files; component descriptions match CLAUDE.md's component list. TreeView correctly footnoted as a sub-component. No stale cross-references found in the sampled docs. +- Historical `docs/plans/` design docs contain superseded "future work" lists (e.g. transport-design §18 lists site-scoped transport as future, which M8 later shipped) — acceptable since they are dated decision records, and the living docs (Component-*.md, CLAUDE.md, README) reflect the shipped state. + +--- + +## Prioritized Recommendations + +1. **[High] Add `tests/ZB.MOM.WW.ScadaBridge.CLI.Tests` to `ZB.MOM.WW.ScadaBridge.slnx`.** One line; removes a standing CI blind spot of 279 tests and retires a documented footgun. +2. **[High] Resolve the Transport Area contradiction** — either implement area export (importer already complete; add Areas to `EntityAggregate` and the id→name lookup in `EntitySerializer.cs:251`) or amend `Component-Transport.md:21` + CLAUDE.md #24 to state area membership does not travel. +3. **[Medium] Options-validation sweep**: add `IValidateOptions<>`/`ValidateOnStart` to the ~14 unvalidated option classes (Transport, StoreAndForward, SiteRuntime, Communication, InboundAPI, etc.), copying the existing HealthMonitoring/ClusterInfrastructure validator pattern. +4. **[Medium] Close the AuditLog reconciliation NodeB gap** (`SiteEnumerator.cs:64`): dial NodeB when NodeA is unreachable — the shape (`SiteEntry`) already supports it, and reconciliation is the pipeline's loss-recovery safety net. +5. **[Medium] Honor or remove `SmsConfiguration.MaxRetryCount`** (`Commons/Entities/Notifications/SmsConfiguration.cs:36`) — an admin-editable field with no effect is worse than no field. +6. **[Medium] Decide the PerformanceTests story**: rename to reflect its correctness-at-scale scope, or add the two highest-value real measurements (failover time, gRPC stream throughput) behind the existing `Category=Performance` trait. +7. **[Low/Medium] Excise the vestigial site-side notification-list surface** (`SiteNotificationRepository` reads + `StoreNotificationListAsync`) now that the purge guarantees the table stays empty. +8. **[Low] Unify role-string casing handling** between UI `RequireClaim` policies and the ManagementActor check (`ManagementActor.cs:1910`) — normalize at write (already validated canonically) and compare consistently. +9. **[Low] Backfill the three missing `docs/components/` reference docs** (ScriptAnalysis, KpiHistory, DelmiaNotifier) or scope the README claim. +10. **[Low] Add contract-lock tests for the top ClusterClient message records** (mirror the existing proto contract tests) to enforce, not just intend, additive-only evolution under version skew. diff --git a/archreview/plans/00-MASTER-TRACKER.md b/archreview/plans/00-MASTER-TRACKER.md new file mode 100644 index 00000000..4629392e --- /dev/null +++ b/archreview/plans/00-MASTER-TRACKER.md @@ -0,0 +1,82 @@ +# Architecture Review Fix Plans — Master Tracker + +**Created:** 2026-07-08 +**Source:** [`archreview/00-OVERALL.md`](../00-OVERALL.md) and domain reports 01–08 +**Plans:** 8 implementation plans, **192 tasks** (191 net of one cross-plan duplicate — see Dedupe Rulings) + +Every plan follows the TDD bite-sized-task format and ships a co-located `.tasks.json` manifest. Execute a plan with the `superpowers-extended-cc:executing-plans` skill (pass the plan path); it reads/updates the `.tasks.json`. **When a plan's status changes, update the table below** — this file is the single source of truth for initiative progress. + +## Status Board + +| Plan | Domain | Tasks | Done | Status | Findings coverage | +|------|--------|------:|-----:|--------|-------------------| +| [PLAN-01](PLAN-01-cluster-host-failover.md) | Cluster, Host & Failover | 23 | 0 | ⬜ Not started | 28 findings: 20 fixed, 3 → other plans, 3 accepted, 2 roadmap-deferred | +| [PLAN-02](PLAN-02-communication-store-and-forward.md) | Communication & S&F | 24 | 0 | ⬜ Not started | 28 findings: 25 fixed, 1 → plan 01, 1 deferred (serializer swap), 1 won't-fix | +| [PLAN-03](PLAN-03-site-runtime-dcl.md) | Site Runtime & DCL | 26 | 0 | ⬜ Not started | 28 actionable: 25 fixed, 3 deferred with rationale | +| [PLAN-04](PLAN-04-data-audit-backbone.md) | Data & Audit Backbone | 26 | 0 | ⬜ Not started | 27 items: 22 fixed, 4 deferred, 1 won't-fix (incl. NodeB reconcile from report 08) | +| [PLAN-05](PLAN-05-templates-deployment-transport.md) | Templates, Deployment & Transport | 25 | 0 | ⬜ Not started | 33 line-items: 27 fixed, 6 deferred with rationale | +| [PLAN-06](PLAN-06-edge-integrations.md) | Edge Integrations | 24 | 0 | ⬜ Not started | 24 findings: 21 fixed, 3 deferred/won't-fix | +| [PLAN-07](PLAN-07-ui-management-security.md) | UI, Management & Security | 33 | 0 | ⬜ Not started | 24 items: 20 fixed, 3 partial (deferrals documented), 1 won't-fix | +| [PLAN-08](PLAN-08-conventions-tests.md) | Conventions & Tests | 11 | 0 | ⬜ Not started | 13 covered, 4 owned by other plans, ~20 triaged into deferred-work register | + +Status values: ⬜ Not started · 🟨 In progress · ✅ Complete · ⏸ Blocked + +## P0 — Do These First (regardless of plan order) + +1. **PLAN-01 Task 1** — enable the SBR downing provider (the system-wide Critical; one line + parsed-config test). +2. **PLAN-01 Tasks 3–4** — the two-node kill-test rig (`TwoNodeClusterFixture`); integration-proves Task 1 and later PLAN-02 work. +3. **PLAN-07 Tasks 1–3** — `DisableLoginGuard` + scrub `deploy/wonder-app-vd03` (Critical; unauthenticated prod surface). +4. **PLAN-02 Tasks 1–2** — ClusterClient generation-suffixed names (Critical crash-loop dropping routing to all sites). +5. **PLAN-05 Task 1** — bundle import consumes `BaseTemplateName` (Critical silent data loss). +6. **PLAN-08 Task 1** — CLI.Tests into the slnx (protects every subsequent plan's test runs). + +## Cross-Plan Dependency Map + +**Hard-ish sequencing (execute earlier):** +- **PLAN-01 Tasks 5–7 → PLAN-02 Tasks 3–4.** PLAN-02's standby S&F delivery gate is a `Func` seam with marked swap points; it works standalone but its final wiring consumes PLAN-01's `ClusterActivityEvaluator` / `SelfIsPrimary` (singleton-host semantics). +- **PLAN-01 Tasks 3–4 → PLAN-02 integration proof.** The two-node rig is how PLAN-02's Critical fixes get integration-verified. +- **PLAN-04 Task 13 (additive `RequestedBy` message change) before any PLAN-07 sender refactors** touching the same messages. + +**Soft seams (either order works, coordinate):** +- **PLAN-05 Task 14 (`IScriptArtifactChangeBus`) ↔ PLAN-06 Task 3 (`InboundScriptExecutor.InvalidateMethod`).** PLAN-06 Task 1's per-request revision check is the designed correctness fallback, so there is no hard sequencing; when both land, wire the bus to the invalidate seam. +- **PLAN-03's `DeployCompileValidator` should adopt PLAN-05's compile-cache abstraction** once it exists (later refactor, not a blocker). +- **PLAN-03 Task 8's stuck-thread watchdog pattern is reusable by PLAN-06 Task 4** (inbound orphan counter). + +**EF Core migrations — serialize globally.** Migration-adding tasks exist in PLAN-04 (Tasks 6, 8), PLAN-06 (Tasks 8, 19), and possibly PLAN-07 (secured-write expiry). All share one model snapshot: **never run two migration tasks concurrently**, and always build first (never `--no-build` on `dotnet ef migrations add` — known empty-scaffold gotcha). + +**Shared-file merge surfaces (don't run these plans' overlapping tasks concurrently):** +| File / area | Plans touching it | +|---|---| +| `ManagementActor.cs` | PLAN-07 (15 tasks, serialized within plan), PLAN-05 (management-path publish), PLAN-06 (method-update paths) | +| `deploy/wonder-app-vd03/appsettings.Central.json` | PLAN-07 (auth), PLAN-01 (Cluster/Node sections) | +| `NotificationOutboxActor.cs` | PLAN-04, PLAN-06 (SMS retry policy) | +| `StoreAndForwardService.cs` / site tracking store | PLAN-02 (primary owner), PLAN-04 Task 15 (keyset query) | +| `Host/Program.cs` + service registration | PLAN-01, PLAN-08 (options validation) | +| `SiteHealthReport` (Commons record, additive fields) | PLAN-03 Task 6, PLAN-04 health-event fields | +| Transport `BundleImporter.cs` | PLAN-05 (primary, serialized chain), PLAN-07 (import idempotency note) | + +## Dedupe Rulings (one owner per finding) + +| Finding | Owner | Others must skip | +|---|---|---| +| CLI.Tests into slnx | **PLAN-08 Task 1** (also retires CLAUDE.md gotcha + memory note) | PLAN-07 Task 33 — **skip as duplicate**; mark `skipped-duplicate` in its tasks.json | +| `DisableLogin: true` artifact | **PLAN-07 Tasks 1–3** | PLAN-01 references only | +| AuditLog reconciliation NodeB failover | **PLAN-04 Task 18** | PLAN-08 pointer only | +| `SmsConfiguration.MaxRetryCount` unused | **PLAN-06 Task 18** | PLAN-08 pointer only | +| Transport `AreaName: null` | **PLAN-05 Task 7** | PLAN-08 pointer only | +| Role-casing normalization | **PLAN-07 Task 10 chain** | PLAN-08 pointer only | +| Two-node failover rig | **PLAN-01 Tasks 3–4** | PLAN-08 Task 8 is a documented placeholder; PLAN-02 consumes | +| Script-artifact invalidation contract | **PLAN-05 Task 14** (design + publisher) | PLAN-06 implements the inbound consumer | + +## Recommended Execution Waves + +- **Wave 0 (P0 list above).** Independent files; all six items can run in parallel. ~1 focused session. +- **Wave 1 — stability spine.** Rest of PLAN-01 (active-node semantics, drains); PLAN-02 Tasks 3–9 (standby gate, gRPC channel keying, sweep bounding); PLAN-03 Tasks 1–4 (reconnect leak, compile gate); PLAN-04 Tasks 1–5 (purge chains); PLAN-05 import chain Tasks 2–6; PLAN-06 Tasks 1–4 + 8; PLAN-07 Tasks 4–9 (deploy scoping, secret elision). +- **Wave 2 — remainder.** All other tasks per intra-plan `blockedBy` edges; migration tasks serialized globally; docs-update tasks land with their code tasks (repo rule: doc + code + tests together). +- **Wave 3 — closeout.** PLAN-08 Task 11 (deferred-work register), each plan's doc-sweep task, then a full `dotnet build ZB.MOM.WW.ScadaBridge.slnx` + `dotnet test` (which now includes CLI.Tests) + `bash docker/deploy.sh` + the two-node drill from PLAN-01. + +**Execution hygiene:** work in a dedicated worktree (not the primary checkout); one plan per executor session where possible; for parallel implementers use pathspec-scoped commits and verify every commit landed on HEAD afterward (both are known failure modes in this repo's history). Re-verify a finding's line numbers before editing — reports and plans cite the code as of 2026-07-08. + +## Deferred / Won't-Fix Registry (initiative-level) + +Consolidated in **PLAN-08 Task 11** → `docs/plans/2026-07-08-deferred-work-register.md` (triages the 23-item inventory from report 08 plus each plan's coverage-table deferrals: serializer swap, hash-chain/Parquet, aggregated live alarm stream, central cert-trust persistence, runtime script sandbox, clustered-GUID benchmark, TLS profile, Dockerfile HEALTHCHECK, etc.). Nothing from the review is silently dropped: every report finding is either a task, an ownership pointer, or a registered deferral with rationale. diff --git a/archreview/plans/PLAN-01-cluster-host-failover.md b/archreview/plans/PLAN-01-cluster-host-failover.md new file mode 100644 index 00000000..536dcdf2 --- /dev/null +++ b/archreview/plans/PLAN-01-cluster-host-failover.md @@ -0,0 +1,1556 @@ +# Cluster, Host & Failover Fixes Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. + +**Goal:** Make automatic failover actually work (enable the SBR downing provider and prove it with a real two-node kill test), unify the "active node" definition on singleton-host (oldest) semantics, and close the CoordinatedShutdown, health-report-loss, offline-detection, and config-validation gaps from architecture review 01. + +**Architecture:** All cluster bootstrap lives in `AkkaHostedService.BuildHocon` (hand-rolled, injection-safe HOCON) — the Critical fix is adding `akka.cluster.downing-provider-class` there, verified by a new reusable two-node in-process cluster fixture (`TwoNodeClusterFixture`) that other plans (02) also depend on. "Active node" is re-derived everywhere from a single new `ClusterActivityEvaluator` (self is the oldest `Up` member = singleton host), replacing the four divergent leader-based checks; the five copy-pasted central-singleton registration blocks are collapsed into a `CentralSingletonRegistrar` helper that also gives the two hot singletons (notification-outbox, audit-log-ingest) the drain tasks they were missing. + +**Tech Stack:** C#/.NET, Akka.NET 1.5.62 (Akka.Cluster, Akka.Cluster.Tools, Akka.TestKit.Xunit2), xUnit + NSubstitute, Docker Compose + Traefik, Blazor Server (one small UI badge). + +Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per-project: `dotnet test tests/`. Docker rig: `bash docker/deploy.sh`. + +## Findings Coverage + +| # | Report finding | Severity | Task(s) / Disposition | +|---|---------------|----------|----------------------| +| S1 | Split-brain resolver never enabled — no automatic failover on crash/partition | Critical | Tasks 1, 3, 4, 21 | +| S2 | "Active node" = leader vs oldest diverge (purge on wrong node, wrong Primary label, wrong self-report, Traefik→non-singleton node) | High | Tasks 5, 6, 7, 8 | +| S3 | Partition: both central nodes return 200 on `/health/active`, Traefik serves both | High | Tasks 1 (SBR resolves partition), 7 (oldest-based + DB-required active check), 4 (behavioral proof) | +| S4 | Cluster-leave drain tasks budgeted 10s inside 5s CoordinatedShutdown phase | Medium | Task 2 | +| S5 | NotificationOutbox + AuditLogIngest singletons have no drain task | Medium | Tasks 9, 10 | +| S6 | Health-report loss recovery is dead code (fire-and-forget transport never throws) | Medium | Tasks 11, 12 | +| S7 | Offline detection keys off heartbeats, masking a dead metrics pipeline; spec/code disagree | Medium | Tasks 13, 14 | +| S8 | `appsettings.Site.json` second seed targets MetricsPort 8084; validator gap | Medium | Task 15 | +| S9 | Docker `stop_grace_period` shorter than full CoordinatedShutdown | Medium | Task 17 | +| S10 | Aggregator never forgets deleted sites | Low | Task 18 | +| S11 | Dead-letter monitor: unbounded warning volume | Low | Task 19 | +| S12 | Mandatory two-seed rule forces phantom seeds in single-node installs | Low | Task 16 | +| P1 | Readiness probes fan out 5 cluster Asks per poll | Low | **Accepted** — bounded (2s, concurrent), cheap at current poll rates; revisit only if an orchestrator tightens the probe interval | +| P2 | Startup compiles every inbound method sequentially | Low | **Deferred** — Inbound API domain (plan 06 scope); report itself says "acceptable now" | +| P3 | Per-tick allocations in health collection | Low | **Won't-fix** — report: "trivial at this cadence. No action" | +| C1 | REQ-HOST-6 mandates Akka.Hosting; code hand-rolls HOCON | Medium | Task 22 | +| C2 | `deploy/wonder-app-vd03` ships `DisableLogin: true` | High | **Deferred → Plan 07** (explicit cross-plan ownership) | +| C3 | LDAP plaintext in deploy artifact | Medium | **Deferred → Plan 07** (deploy-artifact security posture is plan 07's) | +| C4 | Secrets in checked-in docker dev configs | Medium | **Accepted** — labelled dev-only, `ConfigSecretsTests` tracks it, the wonder-app-vd03 `${...}` pattern already exists as the model | +| C5 | Stale `Component-TraefikProxy.md` references | Low | Task 23 | +| C6 | Five copy-pasted ~60-line singleton registration blocks | Low | Tasks 9, 10 | +| U1 | No real multi-node failover test anywhere | — | Tasks 3, 4, 8, 21 (the rig other plans reference) | +| U2 | Windows Service lifecycle / down-if-alone recovery story undefined | — | Task 20 | +| U3 | TLS absent at every layer of this domain | — | **Deferred** — documented lab posture; Task 23 adds an explicit "production TLS profile — not yet implemented" roadmap note so the gap is spec-visible | +| U4 | Dockerfile has no HEALTHCHECK; `/healthz` runs zero checks | — | **Deferred** — report calls this "acceptable minimalism"; prerequisite work for a future orchestrator migration, not this hardening pass | +| U5 | `NodeName` missing from wonder-app-vd03 overlays → NULL `SourceNode` | — | Task 23 | +| U6 | Known-but-open follow-ups (SecuredWrite SourceNode NULL, cert-trust persistence, docker-env2 mirrors findings) | — | **No action** — already tracked in CLAUDE.md/logged follow-ups; docker-env2 compose IS covered by Task 17 | +| U7 | Online/offline transitions unrecorded (post-incident "when did the site drop") | — | Tasks 13, 14 (`LastStatusChangeAt`) | + +--- + +### Task 1: Enable the SBR downing provider in BuildHocon + +**Classification:** high-risk (cluster failover semantics — the Critical finding) +**Estimated implement time:** ~3 min +**Parallelizable with:** 5, 11, 13, 15, 16, 17, 19, 22, 23 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs` (BuildHocon, cluster block lines 250-266) +- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HoconBuilderTests.cs` + +1. Write the failing test in `HoconBuilderTests.cs` (reuse the existing `DefaultCluster()` helper): + +```csharp +[Fact] +public void BuildHocon_EnablesSplitBrainResolverDowningProvider() +{ + // Review 01 [Critical]: without downing-provider-class the entire + // split-brain-resolver section is inert (Akka default = NoDowning) and + // singletons never migrate on a hard crash. + var node = new NodeOptions { Role = "Central", NodeHostname = "node1", RemotingPort = 8081 }; + + var hocon = AkkaHostedService.BuildHocon( + node, DefaultCluster(), new[] { "Central" }, + TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(15)); + + var config = ConfigurationFactory.ParseString(hocon); + Assert.Equal( + "Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster", + config.GetString("akka.cluster.downing-provider-class")); +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~BuildHocon_EnablesSplitBrainResolverDowningProvider"` — expect FAIL (key missing). +3. In `BuildHocon`, add one line inside the `cluster {{` block, directly above `split-brain-resolver {{` (after `min-nr-of-members = {clusterOptions.MinNrOfMembers}`): + +```csharp + downing-provider-class = ""Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster"" +``` + +(inside the interpolated string — note doubled quotes). Also update the method's XML doc comment to state the downing provider is explicitly installed because Akka defaults to NoDowning. +4. Run the test again — expect PASS. Run the full file: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~HoconBuilderTests"` — all pass. +5. Commit: +``` +git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HoconBuilderTests.cs +git commit -m "fix(cluster): enable SBR downing provider — automatic failover was inert for crashes/partitions" +``` + +### Task 2: Raise the cluster-leave CoordinatedShutdown phase timeout above the drain budget + +**Classification:** small +**Estimated implement time:** ~3 min +**Parallelizable with:** 5, 11, 13, 15, 16, 17, 19, 22, 23 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs` (BuildHocon, coordinated-shutdown block lines 267-269) +- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HoconBuilderTests.cs` + +1. Write the failing test: + +```csharp +[Fact] +public void BuildHocon_ClusterLeavePhaseTimeout_ExceedsDrainBudget() +{ + // Review 01 [Medium]: the five singleton drain tasks GracefulStop(10s) + // inside cluster-leave, whose default phase timeout is 5s — Akka abandons + // the task at 5s and the drain guarantee is silently halved. + var node = new NodeOptions { Role = "Central", NodeHostname = "node1", RemotingPort = 8081 }; + var hocon = AkkaHostedService.BuildHocon( + node, DefaultCluster(), new[] { "Central" }, + TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(15)); + + var config = ConfigurationFactory.ParseString(hocon); + var phaseTimeout = config.GetTimeSpan("akka.coordinated-shutdown.phases.cluster-leave.timeout"); + Assert.True(phaseTimeout >= TimeSpan.FromSeconds(15), + $"cluster-leave phase timeout ({phaseTimeout}) must exceed the 10s singleton GracefulStop budget"); +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~ClusterLeavePhaseTimeout"` — expect FAIL. +3. In `BuildHocon`, extend the `coordinated-shutdown` block: + +``` + coordinated-shutdown {{ + run-by-clr-shutdown-hook = on + phases.cluster-leave.timeout = 15s + }} +``` + +4. Run again — expect PASS. Also run `--filter "FullyQualifiedName~CoordinatedShutdownTests"` (source-grep tests must stay green). +5. Commit: +``` +git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HoconBuilderTests.cs +git commit -m "fix(cluster): give cluster-leave phase a 15s timeout so 10s singleton drains can complete" +``` + +### Task 3: Two-node in-process cluster fixture (the reusable failover rig) + +**Classification:** high-risk (real cluster formation in tests) +**Estimated implement time:** ~5 min +**Parallelizable with:** 5, 11, 13, 15, 16, 17, 19, 22, 23 +**Files:** +- Create: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/TwoNodeClusterFixture.cs` +- Test: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/TwoNodeClusterFixtureTests.cs` + +This fixture is the rig other plans (02: S&F standby gating) reference — keep its API small and public. It builds each node's config from the **production** `AkkaHostedService.BuildHocon` (so the config under test is the config that ships), overlaid with test-only crash-simulation settings. + +1. Create the fixture: + +```csharp +using Akka.Actor; +using Akka.Cluster; +using Akka.Configuration; +using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure; +using ZB.MOM.WW.ScadaBridge.Host; +using ZB.MOM.WW.ScadaBridge.Host.Actors; + +namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster; + +/// +/// Forms a REAL two-node Akka cluster in-process from the production +/// BuildHocon output. NodeA is always started first (=> oldest member / +/// singleton host). CrashNode() terminates a node WITHOUT cluster-leave +/// (coordinated-shutdown by-terminate disabled on both nodes) to simulate +/// a hard crash — the scenario review 01 found untested. +/// +public sealed class TwoNodeClusterFixture : IAsyncDisposable +{ + public ActorSystem NodeA { get; private set; } = null!; + public ActorSystem NodeB { get; private set; } = null!; + public int PortA { get; private set; } + public int PortB { get; private set; } + + public static async Task StartAsync( + string role = "Central", TimeSpan? stableAfter = null) + { + var f = new TwoNodeClusterFixture(); + f.PortA = GetFreeTcpPort(); + f.PortB = GetFreeTcpPort(); + f.NodeA = f.StartNode(f.PortA, role, stableAfter); + await WaitForMembersUp(f.NodeA, 1, TimeSpan.FromSeconds(20)); + f.NodeB = f.StartNode(f.PortB, role, stableAfter); + await WaitForMembersUp(f.NodeA, 2, TimeSpan.FromSeconds(20)); + await WaitForMembersUp(f.NodeB, 2, TimeSpan.FromSeconds(20)); + return f; + } + + /// Starts a node from production HOCON; used by StartAsync and by restart-scenarios. + public ActorSystem StartNode(int port, string role, TimeSpan? stableAfter = null) + { + var nodeOptions = new NodeOptions { Role = role, NodeHostname = "127.0.0.1", RemotingPort = port }; + var clusterOptions = new ClusterOptions + { + SeedNodes = new List + { + $"akka.tcp://scadabridge@127.0.0.1:{PortA}", + $"akka.tcp://scadabridge@127.0.0.1:{PortB}", + }, + SplitBrainResolverStrategy = "keep-oldest", + StableAfter = stableAfter ?? TimeSpan.FromSeconds(3), + HeartbeatInterval = TimeSpan.FromMilliseconds(500), + FailureDetectionThreshold = TimeSpan.FromSeconds(2), + MinNrOfMembers = 1, + DownIfAlone = true, + }; + var hocon = AkkaHostedService.BuildHocon( + nodeOptions, clusterOptions, new[] { role }, + TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(3)); + // Crash simulation: Terminate() must NOT run CoordinatedShutdown + // (no Leave gossip) or the test degenerates to the graceful path. + var config = ConfigurationFactory + .ParseString("akka.coordinated-shutdown.run-by-actor-system-terminate = off") + .WithFallback(ConfigurationFactory.ParseString(hocon)); + return ActorSystem.Create("scadabridge", config); + } + + /// Hard-crash a node: abrupt terminate, no cluster leave. + public static Task CrashNode(ActorSystem node) => node.Terminate(); + + public static async Task WaitForMembersUp(ActorSystem system, int count, TimeSpan timeout) + { + var cluster = Akka.Cluster.Cluster.Get(system); + var deadline = DateTime.UtcNow + timeout; + while (DateTime.UtcNow < deadline) + { + if (cluster.State.Members.Count(m => m.Status == MemberStatus.Up) == count + && !cluster.State.Unreachable.Any()) + return; + await Task.Delay(200); + } + throw new TimeoutException( + $"Cluster on {cluster.SelfAddress} did not reach {count} Up members within {timeout}. " + + $"Members: [{string.Join(", ", cluster.State.Members)}], " + + $"Unreachable: [{string.Join(", ", cluster.State.Unreachable)}]"); + } + + public static async Task WaitForMemberRemoved(ActorSystem survivor, Address removed, TimeSpan timeout) + { + var cluster = Akka.Cluster.Cluster.Get(survivor); + var deadline = DateTime.UtcNow + timeout; + while (DateTime.UtcNow < deadline) + { + if (cluster.State.Members.All(m => m.Address != removed) + && cluster.State.Unreachable.All(m => m.Address != removed)) + return; + await Task.Delay(200); + } + throw new TimeoutException($"Member {removed} was never removed from {cluster.SelfAddress}'s view — SBR did not down it."); + } + + private static int GetFreeTcpPort() + { + var listener = new System.Net.Sockets.TcpListener(System.Net.IPAddress.Loopback, 0); + listener.Start(); + var port = ((System.Net.IPEndPoint)listener.LocalEndpoint).Port; + listener.Stop(); + return port; + } + + public async ValueTask DisposeAsync() + { + foreach (var sys in new[] { NodeA, NodeB }) + { + if (sys is { WhenTerminated.IsCompleted: false }) + { + try { await sys.Terminate().WaitAsync(TimeSpan.FromSeconds(10)); } catch { /* teardown */ } + } + } + } +} +``` + +2. Write the smoke test proving the fixture forms a real cluster: + +```csharp +public class TwoNodeClusterFixtureTests +{ + [Fact] + public async Task Fixture_FormsTwoNodeCluster_NodeAIsOldest() + { + await using var cluster = await TwoNodeClusterFixture.StartAsync(); + var a = Akka.Cluster.Cluster.Get(cluster.NodeA); + var b = Akka.Cluster.Cluster.Get(cluster.NodeB); + Assert.Equal(2, a.State.Members.Count); + Assert.Equal(2, b.State.Members.Count); + // NodeA started first => oldest (the singleton host). + Assert.True(a.SelfMember.IsOlderThan(b.State.Members.First(m => m.Address == b.SelfAddress))); + } +} +``` + +3. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.IntegrationTests --filter "FullyQualifiedName~TwoNodeClusterFixtureTests"` — expect FAIL first (files don't exist / compile), then PASS once implemented. (This task is fixture-first; the "failing test" step is the compile failure.) +4. If `NodeOptions`/`BuildHocon` visibility blocks compilation, note that `IntegrationTests` already references the Host project (`.csproj` line 37) and both are public — no changes to `src/` are expected in this task. +5. Commit: +``` +git add tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/ +git commit -m "test(cluster): add TwoNodeClusterFixture — real two-node in-process cluster rig from production HOCON" +``` + +### Task 4: SBR behavioral proof — kill the oldest node, assert downing + singleton migration + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** 5, 11, 13, 15, 16, 17, 19, 22, 23 +**Files:** +- Create: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SbrFailoverTests.cs` + +1. Write the test (this is the test that would have caught the Critical finding — it must FAIL if Task 1's HOCON line is reverted; verify that once by stashing Task 1 if cheap, otherwise trust the assertion on member removal, which is impossible under NoDowning): + +```csharp +using Akka.Actor; +using Akka.Cluster.Tools.Singleton; + +namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster; + +public class SbrFailoverTests +{ + private sealed class EchoActor : ReceiveActor + { + public EchoActor() => ReceiveAny(msg => Sender.Tell(msg)); + } + + private static (IActorRef manager, IActorRef proxy) StartSingleton(ActorSystem sys) + { + var manager = sys.ActorOf(ClusterSingletonManager.Props( + Props.Create(() => new EchoActor()), + PoisonPill.Instance, + ClusterSingletonManagerSettings.Create(sys).WithSingletonName("failover-probe")), + "failover-probe-singleton"); + var proxy = sys.ActorOf(ClusterSingletonProxy.Props( + "/user/failover-probe-singleton", + ClusterSingletonProxySettings.Create(sys).WithSingletonName("failover-probe")), + "failover-probe-proxy"); + return (manager, proxy); + } + + [Fact] + public async Task HardCrashOfOldestNode_SbrDownsIt_AndSingletonMigratesToSurvivor() + { + await using var cluster = await TwoNodeClusterFixture.StartAsync(); + StartSingleton(cluster.NodeA); + var (_, proxyB) = StartSingleton(cluster.NodeB); + + // Singleton is reachable from B while A (oldest) hosts it. + var echo = await proxyB.Ask("ping", TimeSpan.FromSeconds(20)); + Assert.Equal("ping", echo); + + var victimAddress = Akka.Cluster.Cluster.Get(cluster.NodeA).SelfAddress; + await TwoNodeClusterFixture.CrashNode(cluster.NodeA); + + // 1) SBR must DOWN and REMOVE the crashed member (NoDowning => this + // times out; the member stays unreachable forever). + // Budget: failure detection (~2s) + stable-after (3s) + gossip margin. + await TwoNodeClusterFixture.WaitForMemberRemoved( + cluster.NodeB, victimAddress, TimeSpan.FromSeconds(30)); + + // 2) The singleton must re-host on the survivor and answer again. + var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(30); + Exception? last = null; + while (DateTime.UtcNow < deadline) + { + try + { + var echo2 = await proxyB.Ask("ping-after-failover", TimeSpan.FromSeconds(3)); + Assert.Equal("ping-after-failover", echo2); + return; + } + catch (Exception ex) { last = ex; } + } + throw new Xunit.Sdk.XunitException($"Singleton never re-hosted on the survivor after SBR downing: {last}"); + } +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.IntegrationTests --filter "FullyQualifiedName~SbrFailoverTests"` — expect PASS (Task 1 is in). To confirm the test has teeth, temporarily comment out the `downing-provider-class` line in `BuildHocon`, re-run, expect FAIL at `WaitForMemberRemoved` (timeout), then restore the line. Do NOT commit the reverted state. +3. Commit: +``` +git add tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SbrFailoverTests.cs +git commit -m "test(cluster): prove SBR downs a hard-crashed oldest node and the singleton migrates" +``` + +### Task 5: ClusterActivityEvaluator — single oldest-member "active node" definition + +**Classification:** standard +**Estimated implement time:** ~4 min +**Parallelizable with:** 1, 2, 3, 4, 11, 13, 15, 16, 17, 19, 22, 23 +**Files:** +- Create: `src/ZB.MOM.WW.ScadaBridge.Host/Health/ClusterActivityEvaluator.cs` +- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ClusterActivityEvaluatorTests.cs` + +1. Write the failing test (real single-node cluster via TestKit-style bootstrap, mirroring `AkkaBootstrapTests`): + +```csharp +using Akka.Actor; +using Akka.Configuration; +using ZB.MOM.WW.ScadaBridge.Host.Health; + +namespace ZB.MOM.WW.ScadaBridge.Host.Tests; + +public class ClusterActivityEvaluatorTests : IAsyncLifetime +{ + private ActorSystem? _system; + + public async Task InitializeAsync() + { + var port = FreePort(); + var config = ConfigurationFactory.ParseString($@" +akka {{ + actor.provider = cluster + remote.dot-netty.tcp {{ hostname = ""127.0.0.1"", port = {port} }} + cluster {{ + seed-nodes = [""akka.tcp://eval-test@127.0.0.1:{port}""] + roles = [""Central""] + min-nr-of-members = 1 + }} +}}"); + _system = ActorSystem.Create("eval-test", config); + var cluster = Akka.Cluster.Cluster.Get(_system); + var deadline = DateTime.UtcNow.AddSeconds(20); + while (cluster.SelfMember.Status != Akka.Cluster.MemberStatus.Up && DateTime.UtcNow < deadline) + await Task.Delay(100); + } + + public async Task DisposeAsync() { if (_system != null) await _system.Terminate(); } + + [Fact] + public void SelfIsOldest_SoleUpMember_ReturnsTrue() + { + var cluster = Akka.Cluster.Cluster.Get(_system!); + Assert.True(ClusterActivityEvaluator.SelfIsOldest(cluster)); + Assert.True(ClusterActivityEvaluator.SelfIsOldest(cluster, "Central")); + } + + [Fact] + public void SelfIsOldest_RoleNotHeld_ReturnsFalse() + { + var cluster = Akka.Cluster.Cluster.Get(_system!); + // Self doesn't carry the role => it can never host that role's singletons. + Assert.False(ClusterActivityEvaluator.SelfIsOldest(cluster, "site-nonexistent")); + } + + private static int FreePort() + { + var l = new System.Net.Sockets.TcpListener(System.Net.IPAddress.Loopback, 0); + l.Start(); var p = ((System.Net.IPEndPoint)l.LocalEndpoint).Port; l.Stop(); return p; + } +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~ClusterActivityEvaluatorTests"` — expect FAIL (type missing). +3. Implement: + +```csharp +using Akka.Cluster; + +namespace ZB.MOM.WW.ScadaBridge.Host.Health; + +/// +/// THE single definition of "active node" (review 01 [High]): a node is active +/// when it is the OLDEST Up member (optionally within a role scope) — i.e. the +/// member the ClusterSingletonManager places singletons on. Cluster LEADERSHIP +/// (lowest address) is an Akka-internal concept that diverges from singleton +/// placement permanently once the original first node restarts and rejoins; +/// every product-level active/standby decision must use this evaluator, never +/// cluster.State.Leader. +/// +public static class ClusterActivityEvaluator +{ + /// True when self is Up and no other Up member (in the role scope) is older. + public static bool SelfIsOldest(Cluster cluster, string? role = null) + { + var self = cluster.SelfMember; + if (self.Status != MemberStatus.Up) + return false; + if (role != null && !self.HasRole(role)) + return false; + + return cluster.State.Members + .Where(m => m.Status == MemberStatus.Up) + .Where(m => role == null || m.HasRole(role)) + .All(m => m.UniqueAddress.Equals(self.UniqueAddress) || self.IsOlderThan(m)); + } + + /// The oldest Up member in the role scope, or null while none is Up. Used for Primary/Standby labelling. + public static Member? OldestUpMember(Cluster cluster, string? role = null) + { + Member? oldest = null; + foreach (var m in cluster.State.Members.Where(m => m.Status == MemberStatus.Up)) + { + if (role != null && !m.HasRole(role)) continue; + if (oldest == null || m.IsOlderThan(oldest)) oldest = m; + } + return oldest; + } +} +``` + +4. Run — expect PASS. +5. Commit: +``` +git add src/ZB.MOM.WW.ScadaBridge.Host/Health/ClusterActivityEvaluator.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ClusterActivityEvaluatorTests.cs +git commit -m "feat(host): ClusterActivityEvaluator — oldest-member (singleton-host) active-node definition" +``` + +### Task 6: Rewire ActiveNodeGate, AkkaClusterNodeProvider and Primary labels onto the evaluator + +**Classification:** high-risk (changes which node the purge, self-report, inbound gate and dashboard consider active) +**Estimated implement time:** ~5 min +**Parallelizable with:** 11, 13, 15, 16, 17, 19, 22, 23 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeGate.cs` (lines 36-52) +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Health/AkkaClusterNodeProvider.cs` (lines 29-40, 43-81) +- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/IClusterNodeProvider.cs` (SelfIsPrimary xmldoc, lines 15-21) +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs` (comment at lines 109-115 — text only) +- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ClusterActivityEvaluatorTests.cs` (extend) + +1. Write the failing test (extend `ClusterActivityEvaluatorTests` — the single-node cluster from Task 5 is enough because with one member, leader == oldest; the *divergence* behavior is proven in Task 8's two-node test. Here assert the wiring): + +```csharp +[Fact] +public void ActiveNodeGate_And_NodeProvider_AgreeWithEvaluator() +{ + // Both product gates must be backed by the oldest-member evaluator. + // A single-node Up cluster: evaluator says true, so both must say true. + var akkaService = /* construct AkkaHostedService test double or use the + pattern in ActorSystemBridgeTests to wrap _system */; + var gate = new ActiveNodeGate(akkaService); + var provider = new AkkaClusterNodeProvider(akkaService, "Central"); + Assert.True(gate.IsActiveNode); + Assert.True(provider.SelfIsPrimary); +} +``` + +Follow the construction pattern used by `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ActorSystemBridgeTests.cs` / `HealthCheckTests.cs` for obtaining an `AkkaHostedService` whose `ActorSystem` is the test system (if `AkkaHostedService` cannot be instantiated cheaply, add an internal test seam — but check those tests first; a pattern exists because `ActiveNodeGate` is already constructor-tested). +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~AgreeWithEvaluator"` — expect FAIL (or compile error) before rewiring if you assert on a divergence scenario; if single-node passes trivially pre-change, treat this as a pinning test and rely on Task 8 for the behavioral delta. +3. Implement: + - `ActiveNodeGate.IsActiveNode`: replace the leader comparison (lines 44-51) with `return ClusterActivityEvaluator.SelfIsOldest(cluster);` (keep the null-system and `Up` guards; update the class xmldoc: "oldest Up member — the singleton host — not the cluster leader"). + - `AkkaClusterNodeProvider.SelfIsPrimary`: replace leader comparison with `return ClusterActivityEvaluator.SelfIsOldest(cluster, _siteRole);`. + - `AkkaClusterNodeProvider.GetClusterNodes`: replace `var leader = cluster.State.Leader;` + `isLeader` with `var oldest = ClusterActivityEvaluator.OldestUpMember(cluster, _siteRole);` and `var isPrimary = oldest != null && member.UniqueAddress.Equals(oldest.UniqueAddress);` — label `"Primary"`/`"Standby"` from that. + - `IClusterNodeProvider.SelfIsPrimary` xmldoc: change "currently the cluster leader (Primary)" to "currently the oldest Up member for the provider's role scope — the node the ClusterSingletonManager hosts singletons on. This is the canonical product 'active node' check (review 01): the site event-log purge gate, the central self-report loop, and (via plan 02) the S&F delivery gate all consume it." + - `SiteServiceRegistration.cs` lines 109-115 comment: replace "this node is Up AND cluster leader" wording with "this node is the oldest Up member (singleton host)". +4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests` and `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests` — all pass (CentralHealthReportLoop/EventLogPurge consumers are interface-driven; their fakes are unaffected). +5. Commit: +``` +git add src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeGate.cs src/ZB.MOM.WW.ScadaBridge.Host/Health/AkkaClusterNodeProvider.cs src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/IClusterNodeProvider.cs src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ClusterActivityEvaluatorTests.cs +git commit -m "fix(host): unify active-node on oldest-member semantics — purge/self-report/labels now track the singleton host" +``` + +**Seam note for plan 02:** after this task, `IClusterNodeProvider.SelfIsPrimary` (registered in site DI at `SiteServiceRegistration.cs:101-107`) IS the canonical "this node hosts the site singletons / is active" check. Plan 02's S&F delivery gating should inject `IClusterNodeProvider` (or register a delegate mirroring `SiteEventLogActiveNodeCheck`, `SiteServiceRegistration.cs:116-120`) rather than invent a new check. + +### Task 7: Oldest-based /health/active check + require DB reachability for Active + +**Classification:** high-risk (changes Traefik routing behavior) +**Estimated implement time:** ~5 min +**Parallelizable with:** 11, 13, 15, 16, 17, 19, 22, 23 +**Files:** +- Create: `src/ZB.MOM.WW.ScadaBridge.Host/Health/OldestNodeActiveHealthCheck.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` (health check registrations, lines 209-234; comment block lines 346-356) +- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HealthCheckTests.cs` + +1. Write failing tests (follow the existing registration-introspection style in `HealthCheckTests.cs` — it already builds the central service collection and inspects `HealthCheckServiceOptions`): + +```csharp +[Fact] +public void ActiveNodeCheck_IsHostLocalOldestNodeCheck() +{ + var registrations = GetCentralHealthCheckRegistrations(); // existing helper or mirror existing tests + var active = registrations.Single(r => r.Name == "active-node"); + Assert.Contains(ZbHealthTags.Active, active.Tags); + // The shared package's ActiveNodeHealthCheck is LEADER-based (review 01 [High]): + // during a partition both nodes think they lead and Traefik serves both. + var check = active.Factory(BuildCentralServiceProvider()); + Assert.IsType(check); +} + +[Fact] +public void DatabaseCheck_AlsoGatesActive() +{ + // Review 01 [High] recommendation: /health/active should require Ready + // (DB reachable) in addition to singleton-host status, so a DB-dead + // active node is pulled from Traefik rotation. + var registrations = GetCentralHealthCheckRegistrations(); + var db = registrations.Single(r => r.Name == "database"); + Assert.Contains(ZbHealthTags.Ready, db.Tags); + Assert.Contains(ZbHealthTags.Active, db.Tags); +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~HealthCheckTests"` — expect FAIL. +3. Implement: + - New `OldestNodeActiveHealthCheck`: + +```csharp +using Akka.Actor; +using Microsoft.Extensions.Diagnostics.HealthChecks; + +namespace ZB.MOM.WW.ScadaBridge.Host.Health; + +/// +/// /health/active check: Healthy only on the oldest Up member — the node that +/// hosts the cluster singletons. Replaces the shared package's leader-based +/// ActiveNodeHealthCheck (review 01 [High]): leadership is address-ordered and +/// diverges from singleton placement after a restart, and during a partition +/// BOTH nodes compute themselves leader, so Traefik served both. Oldest-member +/// semantics keep exactly one active node through a partition (the non-oldest +/// side still sees the old oldest member until SBR resolves membership). +/// +public sealed class OldestNodeActiveHealthCheck : IHealthCheck +{ + private readonly ActorSystem _system; + public OldestNodeActiveHealthCheck(ActorSystem system) => _system = system; + + public Task CheckHealthAsync(HealthCheckContext context, CancellationToken ct = default) + { + var cluster = Akka.Cluster.Cluster.Get(_system); + return Task.FromResult(ClusterActivityEvaluator.SelfIsOldest(cluster) + ? HealthCheckResult.Healthy("oldest Up member (singleton host)") + : HealthCheckResult.Unhealthy("not the oldest Up member (standby)")); + } +} +``` + + - In `Program.cs`: replace `.AddTypeActivatedCheck("active-node", ...)` with `.AddTypeActivatedCheck("active-node", failureStatus: null, tags: new[] { ZbHealthTags.Active })`; change the database check's tags to `new[] { ZbHealthTags.Ready, ZbHealthTags.Active }`; update the tier-separation comment block (lines 204-208 and 346-356) to say "active-node = oldest Up member + database reachable" and remove the now-stale `ActiveNodeHealthCheck` mention. Keep the `ActiveNodeGate` registration comment (lines 252-260) in sync ("same oldest-member evaluator as OldestNodeActiveHealthCheck"). +4. Run — expect PASS. Also run `--filter "FullyQualifiedName~RequiredSingletonsHealthCheckTests"` (readiness untouched) and build the solution. +5. Commit: +``` +git add src/ZB.MOM.WW.ScadaBridge.Host/Health/OldestNodeActiveHealthCheck.cs src/ZB.MOM.WW.ScadaBridge.Host/Program.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HealthCheckTests.cs +git commit -m "fix(host): /health/active = oldest member + DB reachable — partition-safe Traefik routing" +``` + +### Task 8: Divergence proof — restarted first node must NOT become active again + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** 11, 13, 15, 16, 17, 19, 22, 23 +**Files:** +- Create: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/ActiveNodeSemanticsTests.cs` + +1. Write the test using the Task 3 fixture (this is the exact scenario from the report: after the original first node restarts and rejoins, leadership returns to it by address order while the peer remains oldest — the two definitions permanently diverge): + +```csharp +using ZB.MOM.WW.ScadaBridge.Host.Health; + +namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster; + +public class ActiveNodeSemanticsTests +{ + [Fact] + public async Task AfterOldestNodeRestarts_SurvivorStaysActive_RejoinerIsStandby() + { + await using var fixture = await TwoNodeClusterFixture.StartAsync(); + + // Crash A (oldest), let SBR remove it, B becomes oldest. + var victim = Akka.Cluster.Cluster.Get(fixture.NodeA).SelfAddress; + await TwoNodeClusterFixture.CrashNode(fixture.NodeA); + await TwoNodeClusterFixture.WaitForMemberRemoved(fixture.NodeB, victim, TimeSpan.FromSeconds(30)); + + // Restart A on the same port — it rejoins as the YOUNGEST member but + // (lowest address) may regain Akka leadership. + var rejoined = fixture.StartNode(fixture.PortA, "Central"); + try + { + await TwoNodeClusterFixture.WaitForMembersUp(rejoined, 2, TimeSpan.FromSeconds(30)); + await TwoNodeClusterFixture.WaitForMembersUp(fixture.NodeB, 2, TimeSpan.FromSeconds(30)); + + var clusterB = Akka.Cluster.Cluster.Get(fixture.NodeB); + var clusterA2 = Akka.Cluster.Cluster.Get(rejoined); + + // Product active-node semantics: survivor (oldest, singleton host) + // is active; the rejoined original node is standby — even when it + // holds Akka leadership. + Assert.True(ClusterActivityEvaluator.SelfIsOldest(clusterB)); + Assert.False(ClusterActivityEvaluator.SelfIsOldest(clusterA2)); + } + finally + { + await rejoined.Terminate(); + } + } +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.IntegrationTests --filter "FullyQualifiedName~ActiveNodeSemanticsTests"` — expect PASS. Sanity: assert the OLD leader-based expression (`clusterA2.State.Leader == clusterA2.SelfAddress`) would have returned true for the rejoiner in at least one run — if it reliably does, add it as an inline comment documenting why leader-based was wrong (don't make the test depend on Akka leadership internals). +3. Commit: +``` +git add tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/ActiveNodeSemanticsTests.cs +git commit -m "test(cluster): restarted original node stays standby — oldest-member active semantics proven" +``` + +### Task 9: CentralSingletonRegistrar helper (manager + drain task + proxy) + +**Classification:** high-risk (actor lifecycle) +**Estimated implement time:** ~5 min +**Parallelizable with:** 11, 13, 15, 16, 17, 19, 22, 23 +**Files:** +- Create: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/CentralSingletonRegistrar.cs` +- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CentralSingletonRegistrarTests.cs` + +1. Write the failing test (single-node cluster, real CoordinatedShutdown run): + +```csharp +using Akka.Actor; +using Akka.Configuration; +using Microsoft.Extensions.Logging.Abstractions; +using ZB.MOM.WW.ScadaBridge.Host.Actors; + +namespace ZB.MOM.WW.ScadaBridge.Host.Tests; + +public class CentralSingletonRegistrarTests +{ + private sealed class EchoActor : ReceiveActor + { + public EchoActor() => ReceiveAny(m => Sender.Tell(m)); + } + + [Fact] + public async Task Start_CreatesManagerAndProxy_WithCanonicalNames_AndDrainTask() + { + using var system = CreateSingleNodeClusterSystem(); // same bootstrap as ClusterActivityEvaluatorTests + var handle = CentralSingletonRegistrar.Start( + system, "test-widget", Props.Create(() => new EchoActor()), + NullLogger.Instance, drainTimeout: TimeSpan.FromSeconds(2)); + + // Canonical naming preserved: -singleton / -proxy. + Assert.EndsWith("/user/test-widget-singleton", handle.Manager.Path.ToString()); + Assert.EndsWith("/user/test-widget-proxy", handle.Proxy.Path.ToString()); + + // Singleton answers through the proxy once the member is Up. + var echo = await handle.Proxy.Ask("hi", TimeSpan.FromSeconds(20)); + Assert.Equal("hi", echo); + + // The drain task must stop the manager during cluster-leave. + var probeSystem = system; // watch from a test actor + var watcher = probeSystem.ActorOf(Props.Create(() => new WatchActor(handle.Manager))); + await Akka.Actor.CoordinatedShutdown.Get(system).Run(Akka.Actor.CoordinatedShutdown.ClrExitReason.Instance); + // Manager terminated => drain ran (GracefulStop completed or PoisonPill fallback). + Assert.True(handle.Manager.IsNobody() || system.WhenTerminated.IsCompleted); + } +} +``` + +(Use a `WatchActor`/`TestProbe`-style Terminated assertion if `IsNobody()` proves flaky — the load-bearing assertions are the two path names and the pre-shutdown echo; the drain execution is additionally covered by the existing five singletons' behavior in Task 10's regression run.) +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~CentralSingletonRegistrarTests"` — expect FAIL (type missing). +3. Implement `CentralSingletonRegistrar`: + +```csharp +using Akka.Actor; +using Akka.Cluster.Tools.Singleton; +using Microsoft.Extensions.Logging; + +namespace ZB.MOM.WW.ScadaBridge.Host.Actors; + +/// +/// Registers a central cluster singleton with the canonical naming scheme +/// ({name}-singleton / {name}-proxy), a PoisonPill termination +/// message, and a PhaseClusterLeave drain task that GracefulStops the manager +/// so in-flight EF work completes before handover. Extracted from five +/// copy-pasted ~60-line blocks (review 01 [Low]) whose drift left the two +/// busiest singletons (notification-outbox, audit-log-ingest) without drain +/// tasks (review 01 [Medium]). +/// +internal static class CentralSingletonRegistrar +{ + internal sealed record Handle(IActorRef Manager, IActorRef Proxy); + + internal static Handle Start( + ActorSystem system, + string name, + Props singletonProps, + ILogger logger, + TimeSpan? drainTimeout = null) + { + var manager = system.ActorOf( + ClusterSingletonManager.Props( + singletonProps, + PoisonPill.Instance, + ClusterSingletonManagerSettings.Create(system).WithSingletonName(name)), + $"{name}-singleton"); + + var timeout = drainTimeout ?? TimeSpan.FromSeconds(10); + Akka.Actor.CoordinatedShutdown.Get(system).AddTask( + Akka.Actor.CoordinatedShutdown.PhaseClusterLeave, + $"drain-{name}-singleton", + async () => + { + try + { + await manager.GracefulStop(timeout); + } + catch (Exception ex) + { + logger.LogWarning(ex, + "{Singleton} singleton did not drain within the graceful-stop timeout; " + + "falling through to PoisonPill handover", name); + } + return Akka.Done.Instance; + }); + + var proxy = system.ActorOf( + ClusterSingletonProxy.Props( + $"/user/{name}-singleton", + ClusterSingletonProxySettings.Create(system).WithSingletonName(name)), + $"{name}-proxy"); + + return new Handle(manager, proxy); + } +} +``` + +4. Run — expect PASS. +5. Commit: +``` +git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/CentralSingletonRegistrar.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CentralSingletonRegistrarTests.cs +git commit -m "feat(host): CentralSingletonRegistrar — one registration path with drain task for central singletons" +``` + +### Task 10: Route all seven central singletons through the registrar (adds the missing outbox/ingest drains) + +**Classification:** high-risk (actor wiring refactor) +**Estimated implement time:** ~5 min +**Parallelizable with:** 11, 13, 15, 16, 17, 19, 22, 23 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs` (lines ~395-811: notification-outbox 410-434, audit-log-ingest 451-480, site-call-audit 524-589, audit-log-purge 612-648, site-audit-reconciliation 663-700, kpi-history-recorder 722-757, pending-deployment-purge 776-811) +- Test: existing suites (regression) + `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CoordinatedShutdownTests.cs` (one new source assertion) + +1. Write the failing test — extend `CoordinatedShutdownTests` in its existing source-grep style (the behavioral drain is covered by Task 9; this pins that the two hot singletons now go through the drain path): + +```csharp +[Fact] +public void AllCentralSingletons_RegisterThroughRegistrarWithDrain() +{ + var hostProjectDir = FindHostProjectDirectory(); + var content = File.ReadAllText(Path.Combine(hostProjectDir!, "Actors", "AkkaHostedService.cs")); + // Review 01 [Medium]: notification-outbox and audit-log-ingest were the + // only central singletons WITHOUT a cluster-leave drain task. + foreach (var name in new[] { "notification-outbox", "audit-log-ingest", "site-call-audit", + "audit-log-purge", "site-audit-reconciliation", + "kpi-history-recorder", "pending-deployment-purge" }) + { + Assert.Contains($"CentralSingletonRegistrar.Start(", content); + Assert.Contains($"\"{name}\"", content); + } + // No hand-rolled manager registrations remain in RegisterCentralActors. + Assert.DoesNotContain("ClusterSingletonManager.Props(", ExtractRegisterCentralActors(content)); +} +``` + +(Implement `ExtractRegisterCentralActors` as a substring between `RegisterCentralActors` and the next method, or simply assert `Assert.Equal(0, CountOccurrences(content, "ClusterSingletonManager.Props("))` after checking the site branch separately — the site `deployment-manager`/event-log singletons at lines 906+ use `.WithRole(...)` and stay hand-rolled; scope the assertion to avoid them, e.g. count == 2.) +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~AllCentralSingletons"` — expect FAIL. +3. Refactor `RegisterCentralActors`: replace each of the seven manager/drain/proxy blocks with + +```csharp +var outbox = CentralSingletonRegistrar.Start( + _actorSystem!, "notification-outbox", + Props.Create(() => new ZB.MOM.WW.ScadaBridge.NotificationOutbox.NotificationOutboxActor( + _serviceProvider, outboxOptions, outboxAuditWriter, outboxLogger)), + _logger); +centralCommActor.Tell(new RegisterNotificationOutbox(outbox.Proxy)); +commService?.SetNotificationOutbox(outbox.Proxy); +``` + +and equivalently for the other six (preserve every existing proxy hand-off: `RegisterAuditIngest`, `grpcServer?.SetAuditIngestActor`, `commService?.SetSiteCallAudit`, `siteCallAuditProxy.Tell(RegisterCentralCommunication…)` → `siteCallAudit.Proxy.Tell(…)`, and the log lines). Actor names and singleton names are unchanged (the registrar reproduces them exactly), so `ActorPathTests` and `RequiredSingletonsHealthCheckTests` must stay green. Preserve the comment describing the drain rationale once at the top of `RegisterCentralActors` (delete the five per-block copies and the "not added here to keep this change minimal" note at 544-549, which this task retires). +4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests` (full project — ActorPath/RequiredSingletons/AuditWiring are the regression net) and `dotnet build ZB.MOM.WW.ScadaBridge.slnx` — expect PASS. +5. Commit: +``` +git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CoordinatedShutdownTests.cs +git commit -m "refactor(host): central singletons via CentralSingletonRegistrar — outbox/audit-ingest gain drain tasks" +``` + +### Task 11: SiteHealthReportAck message + ack semantics in the communication actors + +**Classification:** high-risk (actor messaging, cross-cluster contract) +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 3, 4, 5, 13, 15, 16, 17, 19, 22, 23 +**Files:** +- Create: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReportAck.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteCommunicationActor.cs` (SiteHealthReport handler, lines ~393-399) +- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs` (`HandleSiteHealthReport`, lines ~360-374) +- Test: `tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/` (new file `HealthReportAckTests.cs`) + +1. Write failing TestKit tests (mirror the existing `NotificationSubmit` ack tests in Communication.Tests — grep `NotificationSubmitAck` there for the harness pattern): + +```csharp +[Fact] +public void SiteCommunicationActor_NoCentralClient_RepliesNotAccepted() +{ + var siteComm = Sys.ActorOf(Props.Create(() => new SiteCommunicationActor( + "site-a", TestCommunicationOptions(), TestProbe().Ref))); + // No RegisterCentralClient sent => _centralClient is null. + siteComm.Tell(SampleReport(seq: 7)); + var ack = ExpectMsg(); + Assert.False(ack.Accepted); + Assert.Equal(7, ack.SequenceNumber); +} + +[Fact] +public void CentralCommunicationActor_OnReport_ProcessesAndAcks() +{ + var actor = CreateCentralCommunicationActorWithAggregator(out var aggregator); + actor.Tell(SampleReport(seq: 3)); + var ack = ExpectMsg(); + Assert.True(ack.Accepted); + Assert.Equal(3, ack.SequenceNumber); + // aggregator received the report (NSubstitute: aggregator.Received().ProcessReport(...)) +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Communication.Tests --filter "FullyQualifiedName~HealthReportAckTests"` — expect FAIL (message type missing). +3. Implement: + - Commons (additive message evolution — new type only): + +```csharp +namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Health; + +/// +/// Acknowledgement for a forwarded site→central. +/// Review 01 [Medium]: the transport was fire-and-forget, so the sender's +/// interval-counter restore logic could never observe a loss. The ack makes +/// delivery observable end-to-end (central processed the report). +/// +public sealed record SiteHealthReportAck( + string SiteId, long SequenceNumber, bool Accepted, string? Error = null); +``` + + - `SiteCommunicationActor` `Receive`: mirror the `NotificationSubmit` pattern exactly — when `_centralClient == null`, `Sender.Tell(new SiteHealthReportAck(msg.SiteId, msg.SequenceNumber, false, "Central ClusterClient not registered"))`; otherwise `_centralClient.Tell(new ClusterClient.Send("/user/central-communication", msg), Sender)` (forward the original Sender so the central ack routes straight back to the waiting Ask — replaces the current `Self` sender). + - `CentralCommunicationActor.HandleSiteHealthReport`: after `ProcessLocally(report)` and the pub-sub replica publish, add `Sender.Tell(new SiteHealthReportAck(report.SiteId, report.SequenceNumber, true));` (guard with `!Sender.IsNobody()` so the peer-replica path, which arrives without an Ask, doesn't dead-letter). +4. Run — expect PASS. Run the whole Communication.Tests project. +5. Commit: +``` +git add src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReportAck.cs src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteCommunicationActor.cs src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/HealthReportAckTests.cs +git commit -m "feat(comm): ack site health reports end-to-end (SiteHealthReportAck, additive contract)" +``` + +### Task 12: Acked async health-report transport so counter-restore actually fires + +**Classification:** high-risk (changes the sender loop's failure semantics) +**Estimated implement time:** ~5 min +**Parallelizable with:** 13, 15, 16, 17, 19, 22, 23 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/IHealthReportTransport.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/AkkaHealthReportTransport.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/HealthReportSender.cs` (line 160) +- Test: `tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/HealthReportSenderTests.cs` (update fakes), `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/` (new `AkkaHealthReportTransportTests.cs`) + +1. Write failing tests: + - HealthMonitoring: update the fake transport in `HealthReportSenderTests` to the new `Task SendAsync(SiteHealthReport, CancellationToken)` signature and add/adjust: + +```csharp +[Fact] +public async Task SendAsyncFailure_RestoresIntervalCounters() +{ + // The restore path (HealthReportSender.cs:158-175) was DEAD CODE against + // the fire-and-forget production transport. With an acked async transport + // a timeout/nack surfaces as an exception and the counters roll forward. + var collector = NewCollectorWithErrors(scriptErrors: 4); + var transport = new ThrowingTransport(); // SendAsync throws + var sender = NewSender(collector, transport); + await RunOneTick(sender); + Assert.Equal(4, CollectorScriptErrorCount(collector)); // restored, not lost +} +``` + + - Host: `AkkaHealthReportTransportTests` — TestKit: place a probe-backed actor at `/user/site-communication` that replies `SiteHealthReportAck(Accepted: true)`; assert `SendAsync` completes. Second test: reply `Accepted: false` → assert `SendAsync` throws `InvalidOperationException`. Third: no actor at the path → assert it throws (Ask timeout; use a short timeout via options or constant). +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests --filter "FullyQualifiedName~HealthReportSenderTests"` — expect FAIL (compile). +3. Implement: + - `IHealthReportTransport`: replace `void Send(SiteHealthReport report);` with `Task SendAsync(SiteHealthReport report, CancellationToken cancellationToken);` (update xmldoc: "must THROW on non-delivery — the sender's counter-restore depends on it"). + - `AkkaHealthReportTransport`: + +```csharp +public async Task SendAsync(SiteHealthReport report, CancellationToken cancellationToken) +{ + var actorSystem = _akkaService.ActorSystem + ?? throw new InvalidOperationException("Actor system not started — health report not sent"); + var siteComm = actorSystem.ActorSelection("/user/site-communication"); + var ack = await siteComm.Ask(report, AckTimeout, cancellationToken); + if (!ack.Accepted) + throw new InvalidOperationException($"Health report #{report.SequenceNumber} rejected: {ack.Error}"); +} +private static readonly TimeSpan AckTimeout = TimeSpan.FromSeconds(10); +``` + + - `HealthReportSender` line 160: `_transport.Send(reportWithSeq);` → `await _transport.SendAsync(reportWithSeq, stoppingToken).ConfigureAwait(false);` (the surrounding try/catch + restore stays exactly as-is — it now actually fires). Trim the comment at 142-152 to note the transport is now acked. +4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests` and `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~AkkaHealthReportTransportTests"` — PASS. Build the solution (interface change ripples to any other fake). +5. Commit: +``` +git add src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/IHealthReportTransport.cs src/ZB.MOM.WW.ScadaBridge.Host/AkkaHealthReportTransport.cs src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/HealthReportSender.cs tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests tests/ZB.MOM.WW.ScadaBridge.Host.Tests/AkkaHealthReportTransportTests.cs +git commit -m "fix(health): acked SendAsync transport — report-loss counter restore is live, not dead code" +``` + +### Task 13: Metrics-staleness signal + status-transition timestamps in the aggregator + +**Classification:** high-risk (lock-free CAS state machine) +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 3, 4, 5, 11, 15, 16, 17, 19, 22, 23 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthState.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthAggregator.cs` (ProcessReport, CheckForOfflineSites) +- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/HealthMonitoringOptions.cs` + `HealthMonitoringOptionsValidator.cs` +- Modify: `docs/requirements/Component-HealthMonitoring.md` (offline definition, lines ~43-46 + option table) +- Test: `tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/CentralHealthAggregatorTests.cs`, `HealthMonitoringOptionsValidatorTests.cs` + +1. Write failing tests in `CentralHealthAggregatorTests` (use the existing `FakeTimeProvider`/manual-time pattern already in that file): + +```csharp +[Fact] +public void HeartbeatingSiteWithStaleReports_IsFlaggedMetricsStale_ButStaysOnline() +{ + // Review 01 [Medium]: a site whose HealthReportSender died keeps + // heartbeating and previously showed "online with frozen metrics forever". + var (aggregator, time) = NewAggregator(metricsStaleTimeout: TimeSpan.FromMinutes(2)); + aggregator.ProcessReport(Report("site-a", seq: 1)); + time.Advance(TimeSpan.FromMinutes(3)); + aggregator.MarkHeartbeat("site-a", time.GetUtcNow()); // heartbeats keep flowing + aggregator.CheckForOfflineSites(); + var state = aggregator.GetSiteState("site-a")!; + Assert.True(state.IsOnline); // heartbeat-based liveness unchanged + Assert.True(state.IsMetricsStale); // NEW distinct signal +} + +[Fact] +public void FreshReport_ClearsMetricsStale() +{ + var (aggregator, time) = NewAggregator(metricsStaleTimeout: TimeSpan.FromMinutes(2)); + aggregator.ProcessReport(Report("site-a", seq: 1)); + time.Advance(TimeSpan.FromMinutes(3)); + aggregator.CheckForOfflineSites(); + aggregator.ProcessReport(Report("site-a", seq: 2)); + Assert.False(aggregator.GetSiteState("site-a")!.IsMetricsStale); +} + +[Fact] +public void OnlineOfflineTransitions_RecordLastStatusChangeAt() +{ + // Review 01 underdeveloped #7: "when did the site drop" was unanswerable. + var (aggregator, time) = NewAggregator(); + aggregator.ProcessReport(Report("site-a", seq: 1)); + time.Advance(TimeSpan.FromMinutes(5)); // > OfflineTimeout + aggregator.CheckForOfflineSites(); + var offlineAt = aggregator.GetSiteState("site-a")!.LastStatusChangeAt; + Assert.Equal(time.GetUtcNow(), offlineAt); + aggregator.MarkHeartbeat("site-a", time.GetUtcNow()); + Assert.NotEqual(offlineAt, aggregator.GetSiteState("site-a")!.LastStatusChangeAt); +} +``` + +Validator test: `MetricsStaleTimeout` must be positive and ≥ `ReportInterval` (a stale window shorter than one report interval flags every site). +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests --filter "FullyQualifiedName~CentralHealthAggregatorTests"` — expect FAIL. +3. Implement: + - `SiteHealthState`: add `public bool IsMetricsStale { get; init; }` and `public DateTimeOffset? LastStatusChangeAt { get; init; }` (xmldoc each: stale = online but no *report* within MetricsStaleTimeout; LastStatusChangeAt = last online↔offline flip). + - `HealthMonitoringOptions`: add `public TimeSpan MetricsStaleTimeout { get; set; } = TimeSpan.FromMinutes(2);` — validator rules as above. + - `CentralHealthAggregator.ProcessReport`: in both the register and update branches set `IsMetricsStale = false`; in the update branch set `LastStatusChangeAt = existing.IsOnline ? existing.LastStatusChangeAt : now`. `MarkHeartbeat`: on the offline→online promotion set `LastStatusChangeAt = now` (leave `IsMetricsStale` untouched — heartbeats say nothing about metrics). + - `CheckForOfflineSites`: when marking offline, `offline = state with { IsOnline = false, LastStatusChangeAt = now }`. Add a second pass (same loop) for online sites: if `state.LastReportReceivedAt is { } lr && now - lr > _options.MetricsStaleTimeout && !state.IsMetricsStale`, CAS to `state with { IsMetricsStale = true }` and `LogWarning("Site {SiteId} metrics are stale — online (heartbeats) but no report for {Elapsed}s", ...)`. CAS-loss handling identical to the offline swap (lose ⇒ fresh data arrived ⇒ correct to skip). + - `Component-HealthMonitoring.md` lines ~43-46: replace "offline = no report within 60s" with the two-signal model: **liveness** = no heartbeat/signal within `OfflineTimeout` (60s) → offline; **metrics staleness** = online but no report within `MetricsStaleTimeout` (2m) → "metrics stale" flag. Document `LastStatusChangeAt`. This closes the spec/code disagreement the report flagged. +4. Run the HealthMonitoring.Tests project — PASS. +5. Commit: +``` +git add src/ZB.MOM.WW.ScadaBridge.HealthMonitoring tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests docs/requirements/Component-HealthMonitoring.md +git commit -m "feat(health): metrics-stale signal + status-transition timestamps; spec now matches heartbeat-liveness code" +``` + +### Task 14: Surface metrics-stale + offline-since on the Health dashboard + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** 15, 16, 17, 19, 22, 23 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor` +- Test: `tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/` (bUnit — grep for an existing Health page test class first: `grep -rn "Health" tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests --include="*.cs" -l`; extend it if present, else create `HealthPageStalenessTests.cs` following the nearest page-test pattern) + +1. Write the failing bUnit test: render the page with a site state where `IsMetricsStale = true` and assert a `.badge` (existing Bootstrap badge classes used on the page) containing text `Metrics stale` is rendered; second case: `IsOnline = false, LastStatusChangeAt = X` renders `offline since` text containing the formatted timestamp. +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests --filter "FullyQualifiedName~HealthPageStaleness"` — expect FAIL. +3. Implement in `Health.razor`, on the site card/row where the online/offline badge already renders: + +```razor +@if (state.IsOnline && state.IsMetricsStale) +{ + + Metrics stale + +} +@if (!state.IsOnline && state.LastStatusChangeAt is { } changedAt) +{ + offline since @changedAt.ToString("u") +} +``` + +(Match the page's existing badge markup/classes — clean corporate Bootstrap, no new components.) +4. Run — PASS. +5. Commit: +``` +git add src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests +git commit -m "feat(ui): health dashboard shows metrics-stale badge and offline-since timestamp" +``` + +### Task 15: Fix the metrics-port seed in appsettings.Site.json + close the validator gap + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** 1, 2, 3, 4, 5, 11, 13, 16, 17, 19, 22, 23 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json` (lines 13-17) +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/StartupValidator.cs` (seed loop, lines 117-127) +- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/StartupValidatorTests.cs` + +1. Write the failing test (mirror the existing seed-vs-GrpcPort test in `StartupValidatorTests`): + +```csharp +[Fact] +public void Validate_SeedNodeTargetingMetricsPort_Fails() +{ + // Review 01 [Medium]: appsettings.Site.json shipped a seed pointing at the + // Kestrel HTTP/1.1 metrics listener (8084) — a doomed Akka.Remote + // association. The validator guarded GrpcPort but not MetricsPort. + var config = SiteConfigWith( + metricsPort: 8084, + seedNodes: new[] { "akka.tcp://scadabridge@localhost:8082", "akka.tcp://scadabridge@localhost:8084" }); + var ex = Assert.Throws(() => StartupValidator.Validate(config)); + Assert.Contains("must not target the metrics port", ex.Message); +} +``` + +(Use the existing config-building helper in that test file; match the actual exception type/validation surface used by the current seed-vs-grpc test.) +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~SeedNodeTargetingMetricsPort"` — expect FAIL. +3. Implement: + - `StartupValidator.cs`: inside the existing `foreach (var seed in seedNodes ...)` loop add: + +```csharp +p.Require("ScadaBridge:Cluster:SeedNodes", + _ => SeedNodePort(seed) != metricsPort, + $"entry '{seed}' must not target the metrics port " + + $"({metricsPort}); seed nodes must reference Akka remoting ports"); +``` + + - `appsettings.Site.json`: change the second seed `"akka.tcp://scadabridge@localhost:8084"` → `"akka.tcp://scadabridge@localhost:8085"` and add a `"_seedNodes"` comment key: `"Host-0xx: second entry is the FUTURE node-b remoting port (8085) for a two-node localhost site. It must be an Akka remoting endpoint — never this node's GrpcPort (8083) or MetricsPort (8084); StartupValidator rejects both."` + - Sanity-check the docker per-node site configs don't carry the same defect: `grep -n "SeedNodes" -A3 docker/site-*/appsettings.Site.json` (they seed container-host:8082 pairs — expected clean; fix identically if not). +4. Run — PASS. Run the full `StartupValidatorTests` class. +5. Commit: +``` +git add src/ZB.MOM.WW.ScadaBridge.Host/StartupValidator.cs src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json tests/ZB.MOM.WW.ScadaBridge.Host.Tests/StartupValidatorTests.cs +git commit -m "fix(host): dev site seed no longer targets the metrics port; validator rejects seed-vs-MetricsPort" +``` + +### Task 16: Single-node installs — explicit AllowSingleNodeCluster instead of phantom seeds + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** 1, 2, 3, 4, 5, 11, 13, 15, 17, 19, 22, 23 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptionsValidator.cs` (lines 24-34) +- Modify: `deploy/wonder-app-vd03/appsettings.Central.json` (Cluster section, lines 9-19) +- Test: `tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsValidatorTests.cs` + +1. Write failing tests: + +```csharp +[Fact] +public void SingleSeed_WithoutAcknowledgement_Fails() +{ + var options = ValidOptions() with-like-mutation: SeedNodes = ["akka.tcp://scadabridge@h:8081"]; + AssertInvalid(options, "AllowSingleNodeCluster"); +} + +[Fact] +public void SingleSeed_WithAllowSingleNodeCluster_Passes() +{ + // Review 01 [Low]: the shipped single-node artifact satisfied the 2-seed + // rule with a phantom seed the node dials forever. An explicit + // acknowledgement flag replaces the fiction. + var options = ValidOptions(); options.SeedNodes = new List { "akka.tcp://scadabridge@h:8081" }; + options.AllowSingleNodeCluster = true; + AssertValid(options); +} + +[Fact] +public void EmptySeeds_FailsEvenWithFlag() { /* zero seeds always invalid */ } +``` + +(Adapt to the existing `ClusterOptionsValidatorTests` helper style.) +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests --filter "FullyQualifiedName~SingleSeed"` — expect FAIL. +3. Implement: + - `ClusterOptions`: add `public bool AllowSingleNodeCluster { get; set; }` with xmldoc: "Acknowledges a deliberate single-node deployment: permits exactly one seed node (normally ≥2 are required so either node can start first). Without this flag a single-node install must list a phantom second seed, which the node dials forever — log noise and a defeated validation intent (review 01)." + - `ClusterOptionsValidator`: replace the `>= 2` rule with: + +```csharp +var minSeeds = options.AllowSingleNodeCluster ? 1 : 2; +builder.RequireThat(options.SeedNodes is not null && options.SeedNodes.Count >= minSeeds, + options.AllowSingleNodeCluster + ? "ClusterOptions.SeedNodes must contain at least 1 seed node." + : "ClusterOptions.SeedNodes must contain at least 2 seed nodes " + + "(Component-ClusterInfrastructure.md → Node Configuration: both nodes are seed nodes); " + + "for a deliberate single-node install set ClusterOptions.AllowSingleNodeCluster = true instead of listing a phantom seed."); +``` + + - `deploy/wonder-app-vd03/appsettings.Central.json`: remove the `localhost:8091` phantom seed, add `"AllowSingleNodeCluster": true`, and rewrite `_comment_Cluster` to explain the flag ("single-node install acknowledged; add node B's real seed and remove the flag when it exists"). Validate JSON: `python3 -m json.tool deploy/wonder-app-vd03/appsettings.Central.json > /dev/null`. +4. Run the ClusterInfrastructure.Tests project — PASS. +5. Commit: +``` +git add src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests deploy/wonder-app-vd03/appsettings.Central.json +git commit -m "feat(cluster): AllowSingleNodeCluster flag — wonder-app-vd03 drops its phantom seed" +``` + +### Task 17: stop_grace_period for all app containers (both topologies) + +**Classification:** trivial +**Estimated implement time:** ~3 min +**Parallelizable with:** everything except 21 +**Files:** +- Modify: `docker/docker-compose.yml` (every `scadabridge:latest` service) +- Modify: `docker-env2/docker-compose.yml` (every `scadabridge:latest` service) + +1. No unit test (compose config). Verification command is the "test": `docker compose -f docker/docker-compose.yml config | grep -c "stop_grace_period"` — expect 0 before, N (= number of app services, 8 in docker/, 4 in docker-env2/) after. +2. Add to each app service (central-a/b, all site nodes — NOT the traefik service, which has no CoordinatedShutdown): + +```yaml + # CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting + + # actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed + # mid-drain, turning every redeploy into the crash path (review 01 [Medium]). + stop_grace_period: 30s +``` + +3. Validate: `docker compose -f docker/docker-compose.yml config > /dev/null && docker compose -f docker-env2/docker-compose.yml config > /dev/null` (syntax), then the grep from step 1 returns 8 and 4. +4. Commit: +``` +git add docker/docker-compose.yml docker-env2/docker-compose.yml +git commit -m "fix(docker): 30s stop_grace_period so graceful redeploys aren't SIGKILLed mid-CoordinatedShutdown" +``` + +### Task 18: Evict deleted sites from the health aggregator + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 15, 16, 17, 19, 22, 23 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ICentralHealthAggregator.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthAggregator.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs` (the periodic site-address refresh handler — locate with `grep -n "Refresh" src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs`; it reloads sites from the repository every 60s and on admin changes) +- Test: `tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/CentralHealthAggregatorTests.cs`, `tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/` + +1. Write the failing aggregator test: + +```csharp +[Fact] +public void PruneUnknownSites_RemovesDeletedSites_KeepsKnownAndCentral() +{ + // Review 01 [Low]: _siteStates only grew — a deleted site remained a + // permanently-offline dashboard tile (and a KPI sample source) forever. + var (aggregator, time) = NewAggregator(); + aggregator.ProcessReport(Report("site-a", 1)); + aggregator.ProcessReport(Report("site-b", 1)); + aggregator.ProcessReport(Report(CentralHealthReportLoop.CentralSiteId, 1)); + + aggregator.PruneUnknownSites(new[] { "site-a" }); + + Assert.NotNull(aggregator.GetSiteState("site-a")); + Assert.Null(aggregator.GetSiteState("site-b")); + Assert.NotNull(aggregator.GetSiteState(CentralHealthReportLoop.CentralSiteId)); // synthetic id always kept +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests --filter "FullyQualifiedName~PruneUnknownSites"` — expect FAIL. +3. Implement: + - Interface + aggregator: + +```csharp +/// Removes tracked sites not in (deleted from configuration). The synthetic central id is always retained. Self-healing: called from the CentralCommunicationActor's periodic site refresh, so a deleted site disappears within one refresh interval without needing a deletion event. +public void PruneUnknownSites(IReadOnlyCollection knownSiteIds) +{ + var known = new HashSet(knownSiteIds, StringComparer.Ordinal); + foreach (var siteId in _siteStates.Keys) + { + if (siteId == CentralHealthReportLoop.CentralSiteId || known.Contains(siteId)) + continue; + if (_siteStates.TryRemove(siteId, out _)) + _logger.LogInformation("Site {SiteId} evicted from health aggregator (no longer configured)", siteId); + } +} +``` + + - In `CentralCommunicationActor`'s site-refresh handler, after the fresh site list is loaded: `_serviceProvider.GetService()?.PruneUnknownSites(sites.Select(s => s.SiteIdentifier).ToList());` (match the identifier the aggregator is keyed on — it's the `SiteId` string sites report with; confirm by checking what `HandleSiteHealthReport` receives, and use the same property the refresh handler already uses for client-keying). + - Add a Communication.Tests TestKit test: trigger the refresh message with a repo/enumerator fake returning `["site-a"]` and an NSubstitute aggregator; assert `aggregator.Received().PruneUnknownSites(...)`. +4. Run both test projects — PASS. +5. Commit: +``` +git add src/ZB.MOM.WW.ScadaBridge.HealthMonitoring src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests tests/ZB.MOM.WW.ScadaBridge.Communication.Tests +git commit -m "fix(health): evict deleted sites from the aggregator on the periodic site refresh" +``` + +### Task 19: Rate-limit dead-letter warnings + +**Classification:** high-risk (actor change — timers) +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 3, 4, 5, 11, 13, 15, 16, 17, 22, 23 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/DeadLetterMonitorActor.cs` +- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/DeadLetterMonitorTests.cs` + +1. Write the failing test (extend `DeadLetterMonitorTests`; use an NSubstitute `ILogger` and count `Log` invocations at `LogLevel.Warning`): + +```csharp +[Fact] +public void DeadLetterStorm_LogsAtMostWindowLimit_ButCountsAll() +{ + // Review 01 [Low]: a failover/undeploy storm produced one Warning per + // dead letter — unbounded log flood. Health metric counting is untouched. + var logger = Substitute.For>(); + var monitor = Sys.ActorOf(Props.Create(() => new DeadLetterMonitorActor(logger, null))); + for (var i = 0; i < 50; i++) + monitor.Tell(new DeadLetter($"m{i}", TestActor, TestActor)); + + var count = monitor.Ask(GetDeadLetterCount.Instance).Result; + Assert.Equal(50, count.Count); // every dead letter still counted + var warnings = logger.ReceivedCalls().Count(c => + c.GetMethodInfo().Name == "Log" && (LogLevel)c.GetArguments()[0]! == LogLevel.Warning); + Assert.True(warnings <= DeadLetterMonitorActor.MaxWarningsPerWindow, + $"expected <= {DeadLetterMonitorActor.MaxWarningsPerWindow} warnings, got {warnings}"); +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~DeadLetterStorm"` — expect FAIL (50 warnings). +3. Implement: make the actor `IWithTimers`; add `public const int MaxWarningsPerWindow = 10;` and `internal static readonly TimeSpan WarningWindow = TimeSpan.FromMinutes(1);` In the `DeadLetter` handler: + +```csharp +_deadLetterCount++; +_healthCollector?.IncrementDeadLetter(); +if (_warningsThisWindow < MaxWarningsPerWindow) +{ + _warningsThisWindow++; + logger.LogWarning("Dead letter: {MessageType} from {Sender} to {Recipient}", + dl.Message.GetType().Name, dl.Sender, dl.Recipient); + if (_warningsThisWindow == MaxWarningsPerWindow) + logger.LogWarning("Dead-letter warning limit ({Limit}) reached — suppressing per-letter warnings for {Window}", + MaxWarningsPerWindow, WarningWindow); +} +else +{ + _suppressedThisWindow++; +} +``` + +Plus a private `WindowTick` timer message started in `PreStart` (`Timers.StartPeriodicTimer("dl-window", WindowTick.Instance, WarningWindow)`) whose handler logs one summary if `_suppressedThisWindow > 0` (`LogWarning("Suppressed {Count} dead letters in the last {Window}", ...)`) and resets both window counters. Update the class xmldoc. +4. Run — PASS (the summary tick fires only after 1 min, so the test sees ≤ 11 warnings; assert against `MaxWarningsPerWindow + 1` if the limit-reached notice counts). +5. Commit: +``` +git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/DeadLetterMonitorActor.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/DeadLetterMonitorTests.cs +git commit -m "fix(host): rate-limit dead-letter warnings (10/min + suppression summary); metric counting unchanged" +``` + +### Task 20: Down-if-alone recovery story — exit on unexpected ActorSystem termination + service restart actions + +**Classification:** high-risk (process lifecycle) +**Estimated implement time:** ~5 min +**Parallelizable with:** 15, 16, 22, 23 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs` (ctor DI + StartAsync/GetOrCreateActorSystem + StopAsync flag) +- Modify: `deploy/wonder-app-vd03/install.ps1` (service recovery actions) +- Modify: `docs/requirements/Component-ClusterInfrastructure.md` (new "Down-if-alone recovery" subsection near the split-brain section, ~lines 110-119) +- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/` (new `UnexpectedTerminationTests.cs`) + +Context: with SBR enabled (Task 1), `keep-oldest down-if-alone = on` means a node that finds itself alone-and-partitioned downs ITSELF, and `run-coordinated-shutdown-when-down = on` (now live) terminates its ActorSystem. Today the Host process would keep running with a dead actor system — serving nothing, restarted by nobody. The recovery contract: unexpected ActorSystem termination ⇒ stop the host process ⇒ the supervisor (docker `restart: unless-stopped` / Windows service recovery) restarts it ⇒ it rejoins as a fresh incarnation. + +1. Write the failing test: + +```csharp +[Fact] +public async Task ActorSystemTerminatedOutsideStopAsync_StopsHostApplication() +{ + // Down-if-alone self-down (SBR) terminates the ActorSystem via + // run-coordinated-shutdown-when-down. The process must exit so the + // service supervisor restarts it — otherwise the node idles forever + // (review 01 underdeveloped #2). + var lifetime = Substitute.For(); + var service = CreateAkkaHostedService(lifetime); // existing Host.Tests construction pattern + await service.StartAsync(CancellationToken.None); + var system = service.ActorSystem!; + + await system.Terminate(); // simulated self-down, NOT StopAsync + + await Task.Delay(500); + lifetime.Received(1).StopApplication(); +} + +[Fact] +public async Task StopAsync_DoesNotTriggerStopApplication() +{ + var lifetime = Substitute.For(); + var service = CreateAkkaHostedService(lifetime); + await service.StartAsync(CancellationToken.None); + await service.StopAsync(CancellationToken.None); + lifetime.DidNotReceive().StopApplication(); +} +``` + +(Reuse whatever construction pattern `AkkaHostedServiceAuditWiringTests`/`HostStartupTests` use for building the service with a test DI container; add `IHostApplicationLifetime` as an OPTIONAL ctor parameter — nullable, resolved via `GetService` — so every existing construction site keeps compiling.) +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~UnexpectedTermination"` — expect FAIL. +3. Implement: + - `AkkaHostedService`: add `private volatile bool _stopRequested;` set to `true` first thing in `StopAsync`. After the system is created and published (`_actorSystem = system;` in `GetOrCreateActorSystem`), register once: + +```csharp +system.WhenTerminated.ContinueWith(_ => +{ + if (_stopRequested) return; + _logger.LogCritical( + "ActorSystem terminated outside host shutdown (SBR down / run-coordinated-shutdown-when-down). " + + "Stopping the host process so the service supervisor restarts this node as a fresh incarnation."); + _appLifetime?.StopApplication(); +}, TaskContinuationOptions.ExecuteSynchronously); +``` + + - `install.ps1`: after the `sc.exe create`/`New-Service` call, add recovery actions (find the actual service name in the script first): + +```powershell +# Restart-on-failure: required for the SBR down-if-alone recovery contract — +# a self-downed node exits the process and MUST be restarted externally. +& sc.exe failure $ServiceName reset= 86400 actions= restart/5000/restart/30000/restart/60000 +& sc.exe failureflag $ServiceName 1 +``` + + - `Component-ClusterInfrastructure.md`: add a "Down-if-alone recovery" subsection: self-down ⇒ CoordinatedShutdown ⇒ process exit ⇒ supervisor restart (docker `restart: unless-stopped`; Windows service recovery actions) ⇒ clean rejoin as a new incarnation; note the drill in Task 21's script. +4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests` — PASS (full project; the new ctor param must not break existing tests). +5. Commit: +``` +git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs deploy/wonder-app-vd03/install.ps1 docs/requirements/Component-ClusterInfrastructure.md tests/ZB.MOM.WW.ScadaBridge.Host.Tests/UnexpectedTerminationTests.cs +git commit -m "feat(host): exit process on unexpected ActorSystem termination — completes the down-if-alone recovery loop" +``` + +### Task 21: Docker failover drill script (the manual/CI kill test against the real topology) + +**Classification:** small +**Estimated implement time:** ~5 min +**Parallelizable with:** 22, 23 +**Files:** +- Create: `docker/failover-drill.sh` +- Modify: `docker/README.md` (new "Failover drill" section), `deployments/docker-cluster.md` (cross-reference) + +1. No unit test — the script IS the test. Create: + +```bash +#!/usr/bin/env bash +# Failover drill against the running docker cluster (bash docker/deploy.sh first). +# Kills the ACTIVE central node (docker kill = SIGKILL, the hard-crash path that +# review 01 found had NO automatic recovery before the SBR downing provider was +# enabled) and measures how long Traefik takes to route to the survivor. +set -euo pipefail + +TRAEFIK_URL="${TRAEFIK_URL:-http://localhost:9000}" +TIMEOUT_S="${TIMEOUT_S:-90}" + +active_container() { + if curl -sf -o /dev/null "http://localhost:9001/health/active"; then echo scadabridge-central-a + elif curl -sf -o /dev/null "http://localhost:9002/health/active"; then echo scadabridge-central-b + else echo "ERROR: no active central node found" >&2; exit 1; fi +} + +VICTIM=$(active_container) +echo "Active central node: ${VICTIM} — killing it (SIGKILL, crash path)" +docker kill "${VICTIM}" > /dev/null +START=$(date +%s) + +echo "Waiting for the survivor to become active through Traefik (${TRAEFIK_URL}/health/active)..." +while true; do + ELAPSED=$(( $(date +%s) - START )) + if curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; then + echo "PASS: failover complete in ${ELAPSED}s (design budget ~25s + Traefik health interval)" + break + fi + if (( ELAPSED > TIMEOUT_S )); then + echo "FAIL: no active central node after ${ELAPSED}s — SBR/singleton handover did not recover" >&2 + docker start "${VICTIM}" > /dev/null + exit 1 + fi + sleep 1 +done + +SURVIVOR=$([ "${VICTIM}" = scadabridge-central-a ] && echo scadabridge-central-b || echo scadabridge-central-a) +echo "Survivor singleton evidence (last 20 matching log lines from ${SURVIVOR}):" +docker logs "${SURVIVOR}" 2>&1 | grep -Ei "singleton|oldest|Downing|Removed" | tail -20 || true + +echo "Restarting ${VICTIM} and waiting for it to rejoin..." +docker start "${VICTIM}" > /dev/null +sleep 5 +echo "Drill complete. Verify on the Health dashboard that both nodes show Up and the survivor is Primary." +``` + +2. `chmod +x docker/failover-drill.sh`. Verify statically: `bash -n docker/failover-drill.sh` (syntax). If a cluster is running (`bash docker/deploy.sh`), run the drill end-to-end and record the observed failover time in `docker/README.md`; if not available in this session, mark the README section "run after next deploy" — the script is still committed. +3. Document in `docker/README.md`: what it does, the ~25s + Traefik 5s health-interval expectation, and that it exercises S1 (SBR downing), S3 (single-active through Traefik) and Task 20 (restart/rejoin). Cross-reference from `deployments/docker-cluster.md`. Note for other plans: plan 02's duplicate-delivery checks can extend this script. +4. Commit: +``` +git add docker/failover-drill.sh docker/README.md deployments/docker-cluster.md +git commit -m "test(docker): failover drill script — SIGKILL the active central node, assert Traefik recovery" +``` + +### Task 22: Reconcile REQ-HOST-6 (Akka.Hosting) with the hand-rolled bootstrap + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** 15, 16, 17, 19, 21, 23 +**Files:** +- Modify: `docs/requirements/Component-Host.md` (REQ-HOST-6, ~lines 106-114) +- Modify (conditional): `src/ZB.MOM.WW.ScadaBridge.Host/ZB.MOM.WW.ScadaBridge.Host.csproj` (lines 15-18) + `Directory.Packages.props` + +Decision: keep the hand-rolled HOCON (it is hardened — `QuoteHocon`/`DurationHocon`, validated options — and now carries the downing provider explicitly per Task 1); amend the spec instead of a risky bootstrap migration. The report accepts either resolution. + +1. Verify actual package usage: `grep -rn "using Akka.Hosting\|using Akka.Cluster.Hosting\|using Akka.Remote.Hosting\|AkkaConfigurationBuilder\|\.WithClustering\|\.WithRemoting" src/ --include="*.cs"` — expect zero hits (review found them referenced-but-unused). +2. If zero hits: remove ``, ``, `` from the Host csproj (keep `Akka.Cluster.Tools`), remove the matching `` lines from `Directory.Packages.props` **only if no other project references them** (`grep -rn "Akka.Hosting\|Akka.Cluster.Hosting\|Akka.Remote.Hosting" src/*/[A-Z]*.csproj tests/*/[A-Z]*.csproj`), and check whether the csproj's OpenTelemetry.Api transitive-override comment (lines ~28-32) still applies — if it referenced Akka.Hosting's pin, delete the override too. +3. Amend `Component-Host.md` REQ-HOST-6: replace "bootstrap via Akka.Hosting" with: bootstrap is a hand-assembled, injection-safe HOCON document (`AkkaHostedService.BuildHocon`) + `ActorSystem.Create`; the downing provider (`Akka.Cluster.SBR.SplitBrainResolverProvider`) is set explicitly — Akka's default is NoDowning and forgetting this line silently disables failover (this exact omission was review 01's Critical finding, caught because tests now assert the parsed key and a two-node kill test asserts the behavior). Note the deliberate rejection of Akka.Cluster.Hosting typed options with a one-line rationale (single hardened code path already in production shape; migration tracked as possible future work). +4. Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx` — expect success. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~HoconBuilderTests"` — green. +5. Commit: +``` +git add docs/requirements/Component-Host.md src/ZB.MOM.WW.ScadaBridge.Host/ZB.MOM.WW.ScadaBridge.Host.csproj Directory.Packages.props +git commit -m "docs(host): REQ-HOST-6 documents the hand-rolled HOCON bootstrap; drop unused Akka.Hosting packages" +``` + +### Task 23: Docs & deploy-config cleanup batch (Traefik doc, NodeName overlays, TLS roadmap note) + +**Classification:** trivial +**Estimated implement time:** ~4 min +**Parallelizable with:** 15, 16, 17, 19, 21, 22 +**Files:** +- Modify: `docs/requirements/Component-TraefikProxy.md` (lines ~115, ~121, TLS section ~131-138) +- Modify: `deploy/wonder-app-vd03/appsettings.Central.json` (Node section), `deploy/wonder-app-vd03/appsettings.Site.json` (Node section, if the file exists — check `ls deploy/wonder-app-vd03/`) + +1. `Component-TraefikProxy.md`: + - Line ~121: replace the stale `src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeHealthCheck.cs` location with `src/ZB.MOM.WW.ScadaBridge.Host/Health/OldestNodeActiveHealthCheck.cs` (the Host-local oldest-member check from Task 7) and note `/health/active` also requires the database check. + - Line ~115: update the `/health/ready` description from "database + Akka cluster" to "database + akka-cluster + required-singletons" (matches Program.cs and Component-Host REQ-HOST-4a). + - TLS section (~131-138): append an explicit roadmap paragraph — "**Production TLS profile: not yet implemented.** Traefik terminates plain HTTP, Akka remoting is unencrypted TCP, and site gRPC is `http://`. Acceptable for the lab topology only; a production deployment requires TLS at all three layers and a secured Traefik dashboard. Tracked as an open roadmap item (arch review 01, underdeveloped area 3)." — this makes the accepted gap spec-visible without pretending to fix it. +2. `deploy/wonder-app-vd03/appsettings.Central.json`: add `"NodeName": "central-a"` to the `Node` section (per the base config's documented convention — without it every audit row from the one real deployment gets a NULL `SourceNode`, undermining `IX_AuditLog_Node_Occurred` and the per-node stuck KPIs). Same for the Site overlay if present (`"NodeName": "node-a"`). Validate: `python3 -m json.tool deploy/wonder-app-vd03/appsettings.Central.json > /dev/null`. +3. Cross-check no other stale `ActiveNodeHealthCheck` doc references remain: `grep -rn "ActiveNodeHealthCheck" docs/ README.md` — fix any found the same way. +4. Commit: +``` +git add docs/requirements/Component-TraefikProxy.md deploy/wonder-app-vd03/ +git commit -m "docs(cleanup): sync Traefik doc with shared health checks, add TLS roadmap note, NodeName in deploy overlays" +``` + +--- + +## Dependencies on other plans + +- **Plan 02 (Communication & S&F) depends on THIS plan** for two facilities — design them exactly as specified here so the seam holds: + - The canonical site-side "active node / singleton host" check: after Task 6, `IClusterNodeProvider.SelfIsPrimary` (site DI registration at `SiteServiceRegistration.cs:101-107`) means "oldest Up member in the site role = the node hosting the DeploymentManager singleton". Plan 02's standby S&F delivery gating must consume this (directly or via a registered delegate mirroring `SiteEventLogActiveNodeCheck`), not reinvent a leader check. + - The two-node failover rig: `TwoNodeClusterFixture` (Task 3) is public and production-HOCON-based; plan 02's ClusterClient-recreate and S&F non-duplication tests should build on it, and the docker drill script (Task 21) is the shared live-cluster harness. +- **Plan 07 (UI/Management/Security) owns** the `DisableLogin: true` deploy artifact (C2) and the deploy-artifact LDAP plaintext posture (C3) — this plan touches `deploy/wonder-app-vd03/appsettings.Central.json` in Tasks 16 and 23 (Cluster + Node sections only); coordinate merges but do not touch the `Security` section here. +- No other plan blocks this one. + +## Execution order + +**P0 (do first, in order):** Task 1 (the Critical one-liner) → Task 3 (fixture) → Task 4 (behavioral proof). Ship these three even if nothing else lands — they convert "failover does not work" into "failover works and is regression-tested". + +**Critical path:** 1 → 3 → 4 → 8 (needs 6), with 5 → 6 → 7 as the second spine (active-node unification). Task 2 must precede 9/10 (drain budget before drain refactor); 11 → 12 (message before transport); 13 → 14 and 13 → 18 (state shape before consumers). Task 20 touches `AkkaHostedService.cs` and should follow 10 to avoid merge churn; Task 21 follows 1 (drill is meaningless before SBR) and ideally 17. + +**Freely parallel at any point:** 15, 16, 17, 19, 22, 23. diff --git a/archreview/plans/PLAN-01-cluster-host-failover.md.tasks.json b/archreview/plans/PLAN-01-cluster-host-failover.md.tasks.json new file mode 100644 index 00000000..f218f584 --- /dev/null +++ b/archreview/plans/PLAN-01-cluster-host-failover.md.tasks.json @@ -0,0 +1,29 @@ +{ + "planPath": "archreview/plans/PLAN-01-cluster-host-failover.md", + "tasks": [ + { "id": 1, "subject": "Task 1: Enable the SBR downing provider in BuildHocon", "status": "pending", "blockedBy": [] }, + { "id": 2, "subject": "Task 2: Raise the cluster-leave CoordinatedShutdown phase timeout above the drain budget", "status": "pending", "blockedBy": [1] }, + { "id": 3, "subject": "Task 3: Two-node in-process cluster fixture (the reusable failover rig)", "status": "pending", "blockedBy": [1] }, + { "id": 4, "subject": "Task 4: SBR behavioral proof — kill the oldest node, assert downing + singleton migration", "status": "pending", "blockedBy": [3] }, + { "id": 5, "subject": "Task 5: ClusterActivityEvaluator — single oldest-member \"active node\" definition", "status": "pending", "blockedBy": [] }, + { "id": 6, "subject": "Task 6: Rewire ActiveNodeGate, AkkaClusterNodeProvider and Primary labels onto the evaluator", "status": "pending", "blockedBy": [5] }, + { "id": 7, "subject": "Task 7: Oldest-based /health/active check + require DB reachability for Active", "status": "pending", "blockedBy": [5] }, + { "id": 8, "subject": "Task 8: Divergence proof — restarted first node must NOT become active again", "status": "pending", "blockedBy": [3, 6] }, + { "id": 9, "subject": "Task 9: CentralSingletonRegistrar helper (manager + drain task + proxy)", "status": "pending", "blockedBy": [2] }, + { "id": 10, "subject": "Task 10: Route all seven central singletons through the registrar (adds missing outbox/ingest drains)", "status": "pending", "blockedBy": [9] }, + { "id": 11, "subject": "Task 11: SiteHealthReportAck message + ack semantics in the communication actors", "status": "pending", "blockedBy": [] }, + { "id": 12, "subject": "Task 12: Acked async health-report transport so counter-restore actually fires", "status": "pending", "blockedBy": [11] }, + { "id": 13, "subject": "Task 13: Metrics-staleness signal + status-transition timestamps in the aggregator", "status": "pending", "blockedBy": [] }, + { "id": 14, "subject": "Task 14: Surface metrics-stale + offline-since on the Health dashboard", "status": "pending", "blockedBy": [13] }, + { "id": 15, "subject": "Task 15: Fix the metrics-port seed in appsettings.Site.json + close the validator gap", "status": "pending", "blockedBy": [] }, + { "id": 16, "subject": "Task 16: Single-node installs — explicit AllowSingleNodeCluster instead of phantom seeds", "status": "pending", "blockedBy": [] }, + { "id": 17, "subject": "Task 17: stop_grace_period for all app containers (both topologies)", "status": "pending", "blockedBy": [] }, + { "id": 18, "subject": "Task 18: Evict deleted sites from the health aggregator", "status": "pending", "blockedBy": [13] }, + { "id": 19, "subject": "Task 19: Rate-limit dead-letter warnings", "status": "pending", "blockedBy": [] }, + { "id": 20, "subject": "Task 20: Down-if-alone recovery — exit on unexpected ActorSystem termination + service restart actions", "status": "pending", "blockedBy": [10] }, + { "id": 21, "subject": "Task 21: Docker failover drill script (manual/CI kill test against the real topology)", "status": "pending", "blockedBy": [1, 17] }, + { "id": 22, "subject": "Task 22: Reconcile REQ-HOST-6 (Akka.Hosting) with the hand-rolled bootstrap", "status": "pending", "blockedBy": [1] }, + { "id": 23, "subject": "Task 23: Docs & deploy-config cleanup batch (Traefik doc, NodeName overlays, TLS roadmap note)", "status": "pending", "blockedBy": [7, 16] } + ], + "lastUpdated": "2026-07-08" +} diff --git a/archreview/plans/PLAN-02-communication-store-and-forward.md b/archreview/plans/PLAN-02-communication-store-and-forward.md new file mode 100644 index 00000000..681a804f --- /dev/null +++ b/archreview/plans/PLAN-02-communication-store-and-forward.md @@ -0,0 +1,1632 @@ +# Communication & Store-and-Forward Fix Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. + +**Goal:** Fix every finding in `archreview/02-communication-store-and-forward.md` — the ClusterClient recreate crash-loop, the ungated standby S&F delivery pipeline, the shared-gRPC-channel disposal, the collapsing retry sweep, replication ordering/upsert/observability, notification parking/blocking semantics, corrupt-payload handling, and the SQLite/serializer/observability hygiene items. + +**Architecture:** The fixes stay inside the existing seams: `CentralCommunicationActor`/`DefaultSiteClientFactory` get collision-free generation-suffixed actor names; `StoreAndForwardService` gets an injectable active-node delivery gate (a `Func` seam that plan 01's shared singleton-host helper will later plug into), a bounded/short-circuiting/parallel-lane sweep, and a defer-to-sweep enqueue path so `Notify.Send` never blocks a script thread; `SiteStreamGrpcClientFactory` is re-keyed by `(site, endpoint)` so per-session failover cannot dispose shared channels; `ReplicationService` dispatches inline (ordered) with Warning-level observability and upsert-based standby applies, plus a peer-join buffer resync as anti-entropy. + +**Tech Stack:** C#/.NET 10, Akka.NET (TestKit.Xunit2, ClusterClient, DistributedPubSub), Microsoft.Data.Sqlite, Grpc.Net, xUnit + NSubstitute. Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per-project: `dotnet test tests/`. + +## Findings Coverage + +| Report finding | Severity | Task(s) | +|---|---|---| +| ClusterClient recreate-on-address-change → name collision → permanent restart loop | Critical | 1, 2 | +| `siteId` interpolated unvalidated into actor name (secondary, same lines) | Critical (sub) | 1 | +| Standby site node runs full S&F delivery pipeline (systematic duplicate delivery) | High | 3, 4 | +| Debug-session gRPC failover disposes shared channel under other sessions; `CleanupGrpc` creates channels during teardown | High | 5, 6 | +| Retry sweep serial, unbounded, single-laned — collapses under backlog | High (perf) | 7, 8, 9 | +| Replication operations applied out of order (`Task.Run` fire-and-forget) | Medium | 10 | +| Replication failures logged at Debug — dead channel invisible | Medium | 10 | +| Park/Requeue replication is a blind UPDATE — lost on standby missing the row | Medium | 11 | +| Notifications park after ~25 min of central unreachability; `maxRetries: 0` escape hatch unused | Medium | 13 | +| Corrupt buffered notification discarded and reported as delivered | Medium | 14 | +| Spec-vs-code drift: standby-passive model; `Notify.Send` blocks script up to 30 s | Medium | 4 (doc), 12, 13 | +| No explicit Akka serializer — wire format unpinned | Medium (perf) | 15 (round-trip pin tests). Serializer *swap* Deferred: changing the wire format breaks rolling cross-version central↔site compat; needs its own migration plan (flagged for a follow-up design session) | +| Central heartbeat state not replicated to peer central node | Low | 16 | +| `SubscribeInstance` concurrency-limit check-then-act (comment only) | Low | 22 | +| S&F SQLite: no WAL, no busy_timeout | Low | 17 | +| Due-check does per-row `julianday()` string parsing | Low | 18 | +| Audit-observer notification awaited inline in the sweep | Low | 19 | +| `StreamRelayActor.WriteToChannel` dead branch + silent DropOldest loss | Low | 22 | +| Duplicated Ask-timeout constants between the two ingest transports | Low | 22 | +| Heartbeat `IsActive` leader-check vs oldest-node singleton placement | Low | **Deferred → plan 01** (owns the shared active-node/singleton-host helper; Task 3's `Func` seam is designed to accept it) | +| `TransportHeartbeatInterval` double duty (transport FD + app heartbeat) | Low | 22 | +| U1: Dual transports for the same ingest operations | Underdev | **Won't-fix here** — consolidation spans the central ingest actors (another reviewer's domain); Task 22's shared timeout constant removes the copy-paste drift; consolidation flagged as a scheduled follow-up in the Task 22 doc note | +| U2: No real-collaborator test for ClusterClient cache lifecycle | Underdev | 2 | +| U3: No test/design treatment of standby-node sweep behavior | Underdev | 3, 4 (unit-level); real two-node kill rig → plan 01 dependency | +| U4: Replication has no anti-entropy/resync | Underdev | 20, 21 | +| U5: Parked rows have unbounded retention, no aging signal | Underdev | 23 | +| U6: Reconciliation cursor edge (`>` vs `>=`) unpinned | Underdev | 24 | +| U7: `DebugStreamService._sessions` lost silently on central restart | Underdev | 22 (documented as accepted limitation in `Component-Communication.md`) | + +--- + +### Task 1: Generation-suffixed, sanitized ClusterClient actor names + +**Classification:** high-risk (actor lifecycle/naming) +**Estimated implement time:** ~5 min +**Parallelizable with:** 3, 5, 10, 14, 15, 24 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs` (lines 33-41, `DefaultSiteClientFactory`) +- Test: `tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/DefaultSiteClientFactoryTests.cs` (create) + +**Design decision — generation suffix over await-termination:** `Context.Stop` is async and the name stays reserved until the old ClusterClient fully terminates. Watching `Terminated` and deferring recreation requires a pending-recreate state machine in the actor and *lengthens* the routing gap (envelopes for that site are dropped until the new client exists either way). A per-incarnation generation suffix makes creation collision-free by construction, is synchronous, and mirrors the existing repo precedent (`SiteStreamGrpcServer._actorCounter`, `SiteStreamGrpcServer.cs:36,259`). Sanitization handles validity; the counter guarantees uniqueness even if two site ids sanitize identically. + +1. Write the failing test: + +```csharp +// tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/DefaultSiteClientFactoryTests.cs +using System.Collections.Immutable; +using Akka.Actor; +using Akka.Cluster.Tools.Client; +using Akka.Configuration; +using Akka.TestKit.Xunit2; +using Xunit; +using ZB.MOM.WW.ScadaBridge.Communication.Actors; + +public class DefaultSiteClientFactoryTests : TestKit +{ + private static readonly Config TestConfig = ConfigurationFactory.ParseString(@" + akka.actor.provider = cluster + akka.remote.dot-netty.tcp.port = 0 + akka.remote.dot-netty.tcp.hostname = localhost") + .WithFallback(ClusterClientReceptionist.DefaultConfig()); + + public DefaultSiteClientFactoryTests() : base(TestConfig) { } + + private static ImmutableHashSet Contacts() => + ImmutableHashSet.Create(ActorPath.Parse("akka.tcp://other@localhost:2552/system/receptionist")); + + [Fact] + public void Create_TwiceForSameSite_DoesNotCollide() + { + var factory = new DefaultSiteClientFactory(); + var first = factory.Create(Sys, "site-a", Contacts()); + var second = factory.Create(Sys, "site-a", Contacts()); // pre-fix: InvalidActorNameException + Assert.NotEqual(first.Path, second.Path); + } + + [Theory] + [InlineData("site/with/slashes")] + [InlineData("site with spaces")] + [InlineData("näme#!")] + public void Create_WithUnsafeSiteId_SanitizesAndSucceeds(string siteId) + { + var factory = new DefaultSiteClientFactory(); + var client = factory.Create(Sys, siteId, Contacts()); // pre-fix: InvalidActorNameException + Assert.NotNull(client); + } + + [Theory] + [InlineData("plant-01", "plant-01")] + [InlineData("a/b c", "a_b_c")] + [InlineData("", "site")] + public void SanitizeForActorName_ProducesValidPathElement(string input, string expected) + { + Assert.Equal(expected, DefaultSiteClientFactory.SanitizeForActorName(input)); + Assert.True(ActorPath.IsValidPathElement( + DefaultSiteClientFactory.SanitizeForActorName(input))); + } +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Communication.Tests --filter DefaultSiteClientFactoryTests` — expect FAIL (collision throws / sanitize method missing). +3. Implement in `CentralCommunicationActor.cs`, replacing the `DefaultSiteClientFactory` body: + +```csharp +public class DefaultSiteClientFactory : ISiteClientFactory +{ + /// + /// Per-incarnation generation counter. Context.Stop of the previous client is + /// asynchronous — its actor name stays reserved until termination completes — + /// so a same-named recreate in the same message handling throws + /// InvalidActorNameException. A generation suffix makes every incarnation's + /// name unique by construction (mirrors SiteStreamGrpcServer._actorCounter). + /// + private long _generation; + + /// + public IActorRef Create(ActorSystem system, string siteId, ImmutableHashSet contacts) + { + var settings = ClusterClientSettings.Create(system).WithInitialContacts(contacts); + var name = $"site-client-{SanitizeForActorName(siteId)}-{Interlocked.Increment(ref _generation)}"; + return system.ActorOf(ClusterClient.Props(settings), name); + } + + /// + /// Maps an arbitrary SiteIdentifier onto a valid Akka actor-path element: + /// letters/digits/'-'/'_' pass through, everything else becomes '_'. Uniqueness + /// is NOT required here (the generation suffix guarantees it); only validity is. + /// + internal static string SanitizeForActorName(string siteId) + { + if (string.IsNullOrEmpty(siteId)) return "site"; + var sanitized = new string(siteId + .Select(c => char.IsAsciiLetterOrDigit(c) || c is '-' or '_' ? c : '_') + .ToArray()); + return ActorPath.IsValidPathElement(sanitized) ? sanitized : "site"; + } +} +``` + + (`internal` is visible to the test project via the existing `InternalsVisibleTo` in the csproj.) +4. Run the filter again — expect PASS. Also run the full project: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Communication.Tests`. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/DefaultSiteClientFactoryTests.cs && git commit -m "fix(communication): generation-suffixed, sanitized ClusterClient actor names to prevent recreate name collision"` + +### Task 2: Guard client creation in the refresh loop + real-factory address-edit lifecycle test + +**Classification:** high-risk (actor restart semantics) +**Estimated implement time:** ~5 min +**Parallelizable with:** 3, 5, 7, 10, 14 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs` (`HandleSiteAddressCacheLoaded`, lines 507-526) +- Test: `tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/CentralCommunicationActorClientLifecycleTests.cs` (create) + +Depends on Task 1 (same file, uses the fixed factory). + +1. Write the failing test — the address-edit scenario with the **real** `DefaultSiteClientFactory` (this is the exact production path the report says no test exercises; follow the DI-stub pattern from `CentralCommunicationActorTests` and the TestKit config from Task 1): + +```csharp +// tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/CentralCommunicationActorClientLifecycleTests.cs +public class CentralCommunicationActorClientLifecycleTests : TestKit +{ + // same TestConfig as DefaultSiteClientFactoryTests + + private static SiteAddressCacheLoaded Load(string siteId, params string[] addrs) => + new(new Dictionary> + { [siteId] = addrs.ToList().AsReadOnly() }); + + [Fact] + public void AddressEdit_RecreatesClient_WithoutRestartLoop() + { + var sp = new ServiceCollection().BuildServiceProvider(); // no repo needed: we inject cache loads directly + var actor = Sys.ActorOf(Props.Create(() => new CentralCommunicationActor( + sp, new DefaultSiteClientFactory(), null))); + + // First load creates the client; second load (edited NodeA address) stops + // the old one and creates a replacement in the same message handling. + // Pre-fix this throws InvalidActorNameException and restarts the actor. + EventFilter.Exception().Expect(0, () => + { + actor.Tell(Load("site-a", "akka.tcp://scadabridge@node-a:8081")); + actor.Tell(Load("site-a", "akka.tcp://scadabridge@node-a-edited:8081")); + actor.Tell(Load("site-a", "akka.tcp://scadabridge@node-a:8081")); // and back — third generation + }); + + // The actor must still be alive and routing (not crash-looping): + // an envelope for an unknown site produces the "No ClusterClient" warning, + // proving the Receive pipeline is healthy. + EventFilter.Warning(contains: "No ClusterClient for site").ExpectOne(() => + actor.Tell(new SiteEnvelope("unknown-site", new object()))); + } + + [Fact] + public void FactoryThrow_SkipsSite_DoesNotCrashActor() + { + var throwingFactory = Substitute.For(); + throwingFactory.Create(Arg.Any(), Arg.Any(), Arg.Any>()) + .Returns(_ => throw new InvalidOperationException("boom")); + var sp = new ServiceCollection().BuildServiceProvider(); + var actor = Sys.ActorOf(Props.Create(() => new CentralCommunicationActor( + sp, throwingFactory, null))); + + EventFilter.Error(contains: "Failed to create ClusterClient").ExpectOne(() => + actor.Tell(Load("site-a", "akka.tcp://scadabridge@node-a:8081"))); + + // Actor survived — a subsequent load with a healthy factory-free path still processes. + EventFilter.Warning(contains: "No ClusterClient for site").ExpectOne(() => + actor.Tell(new SiteEnvelope("site-a", new object()))); + } +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Communication.Tests --filter CentralCommunicationActorClientLifecycleTests` — expect FAIL (`FactoryThrow` case escapes the handler and restarts the actor; pre-Task-1 the first case also fails). +3. Implement — in `HandleSiteAddressCacheLoaded`, wrap the create (currently line 520) so a failure logs, cleans the entry, and skips the site instead of crashing the actor: + +```csharp + // Stop old client if addresses changed + if (_siteClients.ContainsKey(siteId)) + { + _log.Info("Updating ClusterClient for site {0} (addresses changed)", siteId); + Context.Stop(_siteClients[siteId].Client); + // Remove now: if the replacement create below fails, a stale entry + // would route envelopes to a stopping actor. + _siteClients.Remove(siteId); + } + + IActorRef client; + try + { + client = _siteClientFactory.Create(Context.System, siteId, contactPaths); + } + catch (Exception ex) + { + _log.Error(ex, + "Failed to create ClusterClient for site {0}; site is unroutable until the next refresh", + siteId); + continue; + } + _siteClients[siteId] = (client, contactStrings); +``` + +4. Run the filter — expect PASS. Run full project tests. +5. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.Communication tests/ZB.MOM.WW.ScadaBridge.Communication.Tests && git commit -m "fix(communication): guard per-site ClusterClient creation; add real-factory address-edit lifecycle test"` + +### Task 3: Active-node delivery gate on the S&F retry sweep + +**Classification:** high-risk (failover semantics) +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 5, 14, 15, 16, 24 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardService.cs` (fields near line 85; `RetryPendingMessagesAsync`, lines 563-589) +- Test: `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardServiceTests.cs` (append) + +The gate is a `Func` seam re-evaluated on **every sweep tick** — failover resumes delivery within one `RetryTimerInterval` (10 s) with no re-wiring. Plan 01 owns the shared active-node/singleton-host helper; this seam is where it plugs in (Task 4 wires the interim leader-check). + +1. Write the failing tests (append to `StoreAndForwardServiceTests.cs`, reusing its temp-SQLite construction pattern): + +```csharp +[Fact] +public async Task RetrySweep_SkipsDelivery_WhenDeliveryGateReportsStandby() +{ + var service = CreateService(); // existing helper pattern: storage + options + service + var delivered = 0; + service.RegisterDeliveryHandler(StoreAndForwardCategory.ExternalSystem, + _ => { delivered++; return Task.FromResult(true); }); + service.SetDeliveryGate(() => false); // standby + + await service.EnqueueAsync(StoreAndForwardCategory.ExternalSystem, "target-x", "{}", + attemptImmediateDelivery: false); + await service.RetryPendingMessagesAsync(); + + Assert.Equal(0, delivered); + Assert.NotNull(await service.GetMessageByIdAsync( + (await GetOnlyPendingId(service)))); // row untouched, still Pending +} + +[Fact] +public async Task RetrySweep_ResumesDelivery_WhenGateFlipsActive() +{ + var service = CreateService(); + var delivered = 0; + var active = false; + service.RegisterDeliveryHandler(StoreAndForwardCategory.ExternalSystem, + _ => { delivered++; return Task.FromResult(true); }); + service.SetDeliveryGate(() => active); + + await service.EnqueueAsync(StoreAndForwardCategory.ExternalSystem, "target-x", "{}", + attemptImmediateDelivery: false, retryInterval: TimeSpan.Zero); + + await service.RetryPendingMessagesAsync(); + Assert.Equal(0, delivered); + + active = true; // failover: this node became active + await service.RetryPendingMessagesAsync(); + Assert.Equal(1, delivered); +} + +[Fact] +public async Task RetrySweep_TreatsThrowingGateAsStandby() +{ + var service = CreateService(); + var delivered = 0; + service.RegisterDeliveryHandler(StoreAndForwardCategory.ExternalSystem, + _ => { delivered++; return Task.FromResult(true); }); + service.SetDeliveryGate(() => throw new InvalidOperationException("cluster not ready")); + + await service.EnqueueAsync(StoreAndForwardCategory.ExternalSystem, "t", "{}", + attemptImmediateDelivery: false, retryInterval: TimeSpan.Zero); + await service.RetryPendingMessagesAsync(); // must not throw + Assert.Equal(0, delivered); +} +``` + + Note: `attemptImmediateDelivery: false` sets `LastAttemptAt = now`, so pass `retryInterval: TimeSpan.Zero` (or use the storage `retry_interval_ms = 0` always-due path) to make the row due immediately. +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests --filter RetrySweep_` — expect FAIL (`SetDeliveryGate` missing). +3. Implement in `StoreAndForwardService`: + +```csharp + /// + /// Active-node delivery gate. When set, the retry sweep runs ONLY when the + /// gate reports true — the standby node applies replicated buffer operations + /// but must never deliver (Component-StoreAndForward.md "standby-passive" + /// model). Re-evaluated on every sweep tick so a failover resumes delivery + /// within one RetryTimerInterval without re-wiring. Null (tests, central + /// hosts) preserves the ungated legacy behaviour. A throwing gate is treated + /// as standby — safe-by-default, mirroring SiteCommunicationActor's + /// DefaultIsActiveCheck fallback. + /// + private Func? _deliveryGate; + + /// Installs the active-node delivery gate (see ). + public void SetDeliveryGate(Func gate) => _deliveryGate = gate; +``` + + And at the top of `RetryPendingMessagesAsync`, immediately after the `_retryInProgress` CAS succeeds (inside the `try`): + +```csharp + var gate = _deliveryGate; + if (gate != null) + { + bool isActive; + try { isActive = gate(); } + catch (Exception ex) + { + _logger.LogWarning(ex, + "S&F delivery gate threw; treating this node as standby for this sweep"); + isActive = false; + } + if (!isActive) + { + _logger.LogDebug("S&F retry sweep skipped: this node is not the active site node"); + return; + } + } +``` + +4. Run the filter — expect PASS. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests`. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardService.cs tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardServiceTests.cs && git commit -m "fix(store-and-forward): gate the retry sweep behind an active-node delivery gate (standby must be passive)"` + +### Task 4: Wire the delivery gate in the Host + align the S&F design doc + +**Classification:** small (wiring + docs; behavior covered by Task 3's tests) +**Estimated implement time:** ~3 min +**Parallelizable with:** 5, 6, 10, 11, 14 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs` (in `RegisterSiteActorsAsync`, immediately after `await storeAndForwardService.StartAsync();` at line 981) +- Modify: `docs/requirements/Component-StoreAndForward.md` (Persistence section, the failover paragraphs at lines ~77-80) + +Depends on Task 3. **Dependency on plan 01:** plan 01 delivers the shared active-node/singleton-host helper (and the leader-vs-oldest correction). Until it lands, wire the same leader+Up check the repo already uses (`ActiveNodeGate`, `SiteCommunicationActor.DefaultIsActiveCheck`); the lambda body is the single swap point. + +1. Implement the wiring in `AkkaHostedService.RegisterSiteActorsAsync`: + +```csharp + await storeAndForwardService.StartAsync(); + + // Standby must be passive: only the active site node runs the S&F + // delivery sweep. Without this gate BOTH nodes deliver the same + // replicated Pending rows — systematic duplicate external calls and + // DB writes (arch review 02, Stability #2). Re-evaluated per sweep + // tick so failover resumes delivery within one RetryTimerInterval. + // NOTE: leader+Up mirrors ActiveNodeGate / ActiveNodeHealthCheck / + // SiteCommunicationActor.DefaultIsActiveCheck. When the shared + // singleton-host helper from the cluster-infrastructure fix plan + // lands, replace this lambda body with it. + var gateSystem = _actorSystem!; + storeAndForwardService.SetDeliveryGate(() => + { + var cluster = Akka.Cluster.Cluster.Get(gateSystem); + var self = cluster.SelfMember; + return self.Status == Akka.Cluster.MemberStatus.Up + && cluster.State.Leader != null + && cluster.State.Leader == self.Address; + }); +``` + +2. Update `docs/requirements/Component-StoreAndForward.md` Persistence/failover text: replace the sentence "On failover, the new active node resumes delivery from its local copy." with an explicit statement that (a) the delivery sweep is **gated to the active node** and the standby only applies replicated operations, and (b) duplicate deliveries are confined to the failover window (an in-flight delivery whose `Remove` was not yet replicated), not steady-state operation. +3. Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx` — expect success. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests` (regression only). +4. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs docs/requirements/Component-StoreAndForward.md && git commit -m "fix(host): wire S&F active-node delivery gate on site nodes; align S&F doc standby-passive wording"` + +### Task 5: Key the gRPC client cache by (site, endpoint) + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 3, 10, 14, 15 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClientFactory.cs` (whole cache surface, lines 14, 52-137) +- Test: `tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcClientFactoryTests.cs` (modify — the dispose-on-endpoint-change assertions invert), `tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcClientFactoryDisposeTests.cs` (modify) + +Both node channels coexist per site; per-session failover never disposes shared state. Site *removal* (`RemoveSiteAsync`) disposes all of a site's entries — that remains the only shared-disposal path. Trade-off (documented in the class doc): an edited gRPC address leaves the old endpoint's idle channel cached until site removal or process shutdown — bounded at a handful of entries per site (2 nodes + rare edits), each an idle HTTP/2 channel. + +1. Modify the existing factory tests (they use the `protected virtual CreateClient` tracking-client override): the test that asserts "requesting a different endpoint disposes the cached client" becomes: + +```csharp +[Fact] +public void GetOrCreate_DifferentEndpointSameSite_KeepsBothClientsAlive() +{ + var factory = new TrackingFactory(LoggerFactory); + var a = factory.GetOrCreate("site-1", "http://node-a:8083"); + var b = factory.GetOrCreate("site-1", "http://node-b:8083"); + + Assert.NotSame(a, b); + Assert.False(((TrackingClient)a).Disposed); // pre-fix: a is disposed here + Assert.Same(a, factory.GetOrCreate("site-1", "http://node-a:8083")); // still cached +} + +[Fact] +public void TryGet_ReturnsCachedClientOrNull_WithoutCreating() +{ + var factory = new TrackingFactory(LoggerFactory); + Assert.Null(factory.TryGet("site-1", "http://node-a:8083")); + var a = factory.GetOrCreate("site-1", "http://node-a:8083"); + Assert.Same(a, factory.TryGet("site-1", "http://node-a:8083")); + Assert.Null(factory.TryGet("site-1", "http://node-b:8083")); + Assert.Equal(1, factory.CreatedCount); // TryGet never creates +} + +[Fact] +public async Task RemoveSiteAsync_DisposesAllEndpointsForTheSite_OnlyThatSite() +{ + var factory = new TrackingFactory(LoggerFactory); + var a = (TrackingClient)factory.GetOrCreate("site-1", "http://node-a:8083"); + var b = (TrackingClient)factory.GetOrCreate("site-1", "http://node-b:8083"); + var other = (TrackingClient)factory.GetOrCreate("site-2", "http://node-a:8083"); + + await factory.RemoveSiteAsync("site-1"); + Assert.True(a.Disposed); + Assert.True(b.Disposed); + Assert.False(other.Disposed); +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Communication.Tests --filter SiteStreamGrpcClientFactory` — expect FAIL. +3. Implement: + +```csharp + private readonly ConcurrentDictionary<(string Site, string Endpoint), SiteStreamGrpcClient> _clients = new(); + + public virtual SiteStreamGrpcClient GetOrCreate(string siteIdentifier, string grpcEndpoint) => + _clients.GetOrAdd((siteIdentifier, grpcEndpoint), _ => CreateClient(grpcEndpoint)); + + /// + /// Returns the cached client for (site, endpoint), or null — never creates. + /// Teardown paths (DebugStreamBridgeActor.CleanupGrpc / failed-stream + /// unsubscribe) use this so cleanup can never open a fresh channel. + /// + public virtual SiteStreamGrpcClient? TryGet(string siteIdentifier, string grpcEndpoint) => + _clients.TryGetValue((siteIdentifier, grpcEndpoint), out var client) ? client : null; + + public async Task RemoveSiteAsync(string siteIdentifier) + { + foreach (var key in _clients.Keys.Where(k => k.Site == siteIdentifier).ToList()) + { + if (_clients.TryRemove(key, out var client)) + await client.DisposeAsync(); + } + } +``` + + `DisposeAsync`/`Dispose` iterate `_clients.Values` unchanged. Update the class `` (lines 40-48) — GetOrCreate no longer disposes on endpoint mismatch; both node channels coexist; site removal is the disposal path; note the idle-stale-channel trade-off. Fix any `RemoveSiteAsync` caller expectations (grep: only tests today). +4. Run the filter — expect PASS. Run the whole project (DebugStreamBridgeActorTests may still pass pre-Task-6 since GetOrCreate keeps working). +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClientFactory.cs tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc && git commit -m "fix(communication): key gRPC client cache by (site, endpoint) so per-session failover cannot dispose shared channels"` + +### Task 6: Endpoint-safe unsubscribe in DebugStreamBridgeActor + concurrent-session regression test + +**Classification:** high-risk (actor/stream lifecycle) +**Estimated implement time:** ~5 min +**Parallelizable with:** 3, 4, 7, 10, 14 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/Actors/DebugStreamBridgeActor.cs` (`HandleGrpcError` lines 463-470, `CleanupGrpc` lines 486-495) +- Test: `tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/DebugStreamBridgeActorTests.cs` (append) + +Depends on Task 5 (`TryGet`). + +1. Write the failing test (use the existing tracking-factory pattern in `DebugStreamBridgeActorTests`): + +```csharp +[Fact] +public void SessionFailover_DoesNotDisposeOrTouch_OtherEndpointClient() +{ + var factory = new TrackingFactory(LoggerFactory); + // Session A on node-a; session B's healthy client on node-b already cached. + var bClient = (TrackingClient)factory.GetOrCreate("site-1", "http://node-b:8083"); + + var bridge = CreateBridge(factory, nodeA: "http://node-a:8083", nodeB: "http://node-b:8083"); + bridge.Tell(new GrpcStreamError(new Exception("stream fault"))); // A flips a->b + + AwaitAssert(() => + { + Assert.False(bClient.Disposed); // pre-Task-5 fix: disposed + Assert.Equal(0, bClient.UnsubscribeCallsForOtherCorrelations); // untouched + }); +} + +[Fact] +public void CleanupGrpc_NeverCreatesAChannel() +{ + var factory = new TrackingFactory(LoggerFactory); + var bridge = CreateBridge(factory, nodeA: "http://node-a:8083", nodeB: "http://node-b:8083"); + var createdBefore = factory.CreatedCount; // 1: PreStart opened node-a stream + + bridge.Tell(new StopDebugStream()); // teardown path + AwaitAssert(() => Assert.Equal(createdBefore, factory.CreatedCount)); // pre-fix: GetOrCreate in CleanupGrpc creates +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Communication.Tests --filter "SessionFailover_DoesNotDispose|CleanupGrpc_NeverCreates"` — expect FAIL on the create-count assertion. +3. Implement — replace both `GetOrCreate` teardown uses with `TryGet`: + +```csharp + // HandleGrpcError (was lines 468-470): + var previousEndpoint = _useNodeA ? _grpcNodeAAddress : _grpcNodeBAddress; + // TryGet, not GetOrCreate: unsubscribing a failed stream must never open a + // fresh channel (and, pre-(site,endpoint)-keying, must never dispose another + // session's healthy channel). Absent client => the channel is already gone + // and the site-side relay will be reaped by keepalive/session-lifetime. + _grpcFactory.TryGet(_siteIdentifier, previousEndpoint)?.Unsubscribe(_correlationId); +``` + +```csharp + private void CleanupGrpc() + { + _grpcCts?.Cancel(); + _grpcCts?.Dispose(); + _grpcCts = null; + + var endpoint = _useNodeA ? _grpcNodeAAddress : _grpcNodeBAddress; + _grpcFactory.TryGet(_siteIdentifier, endpoint)?.Unsubscribe(_correlationId); + } +``` + +4. Run the filter, then the full project — expect PASS. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Communication/Actors/DebugStreamBridgeActor.cs tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/DebugStreamBridgeActorTests.cs && git commit -m "fix(communication): debug-stream teardown/failover unsubscribes via TryGet, never creates or disposes shared channels"` + +### Task 7: Bound the retry sweep with a batch LIMIT + +**Classification:** standard +**Estimated implement time:** ~4 min +**Parallelizable with:** 1, 2, 5, 6, 14, 16 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs` (`GetMessagesForRetryAsync`, lines 183-203), `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptions.cs`, `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardService.cs` (line 571) +- Test: `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardStorageTests.cs` (append), `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardOptionsTests.cs` (append) + +1. Failing tests: + +```csharp +// StoreAndForwardStorageTests +[Fact] +public async Task GetMessagesForRetry_HonorsLimit_OldestFirst() +{ + for (var i = 0; i < 5; i++) + await _storage.EnqueueAsync(NewPendingMessage($"m{i}", createdAt: Base.AddSeconds(i))); + + var page = await _storage.GetMessagesForRetryAsync(limit: 3); + Assert.Equal(3, page.Count); + Assert.Equal(new[] { "m0", "m1", "m2" }, page.Select(m => m.Id)); +} + +// StoreAndForwardOptionsTests +[Fact] +public void SweepBatchLimit_Defaults_To_500() => + Assert.Equal(500, new StoreAndForwardOptions().SweepBatchLimit); +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests --filter "GetMessagesForRetry_HonorsLimit|SweepBatchLimit_Defaults"` — expect FAIL (no `limit` parameter / no option). +3. Implement: + - `StoreAndForwardOptions`: `/// Maximum due rows loaded per retry sweep. Bounds sweep memory and duration under backlog; the remainder is picked up next tick (10 s). 0 = unlimited (legacy). public int SweepBatchLimit { get; set; } = 500;` + - `GetMessagesForRetryAsync(int limit = 0)`: append `LIMIT @limit` when `limit > 0` (`cmd.CommandText += "\n LIMIT @limit"; cmd.Parameters.AddWithValue("@limit", limit);`). Keep `ORDER BY created_at ASC` (oldest-first fairness). + - `RetryPendingMessagesAsync` line 571: `var messages = await _storage.GetMessagesForRetryAsync(_options.SweepBatchLimit);` +4. Run the filter, then the full project — expect PASS. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.StoreAndForward tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests && git commit -m "perf(store-and-forward): bound retry sweep with SweepBatchLimit (default 500) on the due-rows query"` + +### Task 8: Per-target short-circuit in the retry sweep + +**Classification:** high-risk (retry-count semantics) +**Estimated implement time:** ~5 min +**Parallelizable with:** 2, 5, 6, 14, 16 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardService.cs` (`RetryPendingMessagesAsync` lines 563-589, `RetryMessageAsync` signature/returns, lines 591-720) +- Test: `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardServiceTests.cs` (append) + +Depends on Task 7 (same methods). After the first *transient* failure to a `(category, target)` in a sweep, remaining messages for that pair are skipped — N messages to a dead target cost one timeout per sweep, not N. Skipped messages are NOT counted as retry attempts (their `RetryCount`/`LastAttemptAt` are untouched), so the park horizon reflects real attempts. + +1. Failing test: + +```csharp +[Fact] +public async Task RetrySweep_ShortCircuitsTarget_AfterFirstTransientFailure() +{ + var service = CreateService(); + var attemptsByTarget = new Dictionary(); + service.RegisterDeliveryHandler(StoreAndForwardCategory.ExternalSystem, msg => + { + attemptsByTarget[msg.Target] = attemptsByTarget.GetValueOrDefault(msg.Target) + 1; + if (msg.Target == "dead-target") throw new TimeoutException("down"); + return Task.FromResult(true); + }); + + for (var i = 0; i < 3; i++) + await service.EnqueueAsync(StoreAndForwardCategory.ExternalSystem, "dead-target", "{}", + attemptImmediateDelivery: false, retryInterval: TimeSpan.Zero); + await service.EnqueueAsync(StoreAndForwardCategory.ExternalSystem, "healthy-target", "{}", + attemptImmediateDelivery: false, retryInterval: TimeSpan.Zero); + + await service.RetryPendingMessagesAsync(); + + Assert.Equal(1, attemptsByTarget["dead-target"]); // pre-fix: 3 + Assert.Equal(1, attemptsByTarget["healthy-target"]); // healthy lane unaffected +} + +[Fact] +public async Task RetrySweep_SkippedMessages_DoNotAccrueRetryCount() +{ + var service = CreateService(); + service.RegisterDeliveryHandler(StoreAndForwardCategory.ExternalSystem, + _ => throw new TimeoutException("down")); + await service.EnqueueAsync(StoreAndForwardCategory.ExternalSystem, "dead", "{}", maxRetries: 5, + attemptImmediateDelivery: false, retryInterval: TimeSpan.Zero); + var skippedId = (await service.EnqueueAsync(StoreAndForwardCategory.ExternalSystem, "dead", "{}", + maxRetries: 5, attemptImmediateDelivery: false, retryInterval: TimeSpan.Zero)).MessageId; + + await service.RetryPendingMessagesAsync(); + var skipped = await service.GetMessageByIdAsync(skippedId); + Assert.Equal(0, skipped!.RetryCount); // only the attempted message accrued a retry +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests --filter "ShortCircuitsTarget|DoNotAccrueRetryCount"` — expect FAIL. +3. Implement: + - Add `internal enum RetryOutcome { Delivered, Parked, TransientFailure, Skipped }` and change `RetryMessageAsync` to `private async Task RetryMessageAsync(StoreAndForwardMessage message)` returning `Delivered` on success, `Parked` on both park branches, `TransientFailure` on the non-park transient branch, `Skipped` on no-handler/CAS-skip paths. + - In `RetryPendingMessagesAsync`: + +```csharp + var failedTargets = new HashSet<(StoreAndForwardCategory, string)>(); + foreach (var message in messages) + { + // One transient failure per (category, target) per sweep: the target + // is down — burning a full timeout per remaining message serializes + // the sweep into hours under backlog (arch review 02, Performance #1). + // Skipped rows keep their RetryCount/LastAttemptAt untouched. + if (failedTargets.Contains((message.Category, message.Target))) + continue; + + var outcome = await RetryMessageAsync(message); + if (outcome == RetryOutcome.TransientFailure) + failedTargets.Add((message.Category, message.Target)); + } +``` + +4. Run the filter + full project — expect PASS. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardService.cs tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardServiceTests.cs && git commit -m "perf(store-and-forward): short-circuit a (category,target) lane after its first transient failure per sweep"` + +### Task 9: Parallel per-target lanes with a small concurrency cap + +**Classification:** high-risk (concurrency) +**Estimated implement time:** ~5 min +**Parallelizable with:** 2, 5, 6, 14, 16 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardService.cs` (`RetryPendingMessagesAsync`), `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptions.cs` +- Test: `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardServiceTests.cs` (append) + +Depends on Task 8. Removes head-of-line blocking *across* targets (one dead external system no longer delays notification forwarding to a healthy central). Ordering **within** a `(category, target)` lane stays strictly sequential — per-target FIFO is preserved. + +1. Failing test: + +```csharp +[Fact] +public async Task RetrySweep_SlowTarget_DoesNotBlockOtherTargets() +{ + var service = CreateService(); // options.SweepTargetParallelism default 4 + var slowGate = new TaskCompletionSource(); + var healthyDelivered = new TaskCompletionSource(); + service.RegisterDeliveryHandler(StoreAndForwardCategory.ExternalSystem, async msg => + { + if (msg.Target == "slow") { await slowGate.Task; return true; } + healthyDelivered.TrySetResult(); + return true; + }); + await service.EnqueueAsync(StoreAndForwardCategory.ExternalSystem, "slow", "{}", + attemptImmediateDelivery: false, retryInterval: TimeSpan.Zero); + await service.EnqueueAsync(StoreAndForwardCategory.ExternalSystem, "healthy", "{}", + attemptImmediateDelivery: false, retryInterval: TimeSpan.Zero); + + var sweep = service.RetryPendingMessagesAsync(); + // Healthy lane completes while the slow lane is still blocked — pre-fix this + // times out because delivery is strictly serial across targets. + await healthyDelivered.Task.WaitAsync(TimeSpan.FromSeconds(5)); + slowGate.SetResult(); + await sweep; +} + +[Fact] +public async Task RetrySweep_WithinTargetLane_StaysSequentialOldestFirst() +{ + var service = CreateService(); + var order = new System.Collections.Concurrent.ConcurrentQueue(); + service.RegisterDeliveryHandler(StoreAndForwardCategory.ExternalSystem, msg => + { + order.Enqueue(msg.Id); return Task.FromResult(true); + }); + for (var i = 0; i < 3; i++) + await service.EnqueueAsync(StoreAndForwardCategory.ExternalSystem, "t", "{}", + attemptImmediateDelivery: false, retryInterval: TimeSpan.Zero, messageId: $"m{i}"); + await service.RetryPendingMessagesAsync(); + Assert.Equal(new[] { "m0", "m1", "m2" }, order.ToArray()); +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests --filter "SlowTarget_DoesNotBlock|StaysSequentialOldestFirst"` — expect FAIL (first test). +3. Implement: + - Option: `/// Max (category,target) lanes delivered concurrently per sweep. 1 = legacy serial. Within a lane delivery stays sequential (per-target FIFO). public int SweepTargetParallelism { get; set; } = 4;` + - In `RetryPendingMessagesAsync`, replace the flat loop (keeping Task 8's short-circuit *inside* each lane): + +```csharp + var lanes = messages + .GroupBy(m => (m.Category, m.Target)) + .Select(g => g.ToList()) + .ToList(); + + using var laneCap = new SemaphoreSlim(Math.Max(1, _options.SweepTargetParallelism)); + var laneTasks = lanes.Select(async lane => + { + await laneCap.WaitAsync(); + try + { + foreach (var message in lane) + { + var outcome = await RetryMessageAsync(message); + if (outcome == RetryOutcome.TransientFailure) + return; // short-circuit the rest of this lane this sweep + } + } + finally { laneCap.Release(); } + }).ToList(); + await Task.WhenAll(laneTasks); +``` + + (`_bufferedCount` uses `Interlocked` and each storage call opens its own connection, so lanes are safe to run concurrently; the `failedTargets` set from Task 8 collapses into the per-lane `return`.) +4. Run the filter + full StoreAndForward.Tests project (watch for timing-sensitive existing sweep tests). +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.StoreAndForward tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests && git commit -m "perf(store-and-forward): parallel per-target sweep lanes (cap 4), sequential within lane"` + +### Task 10: Ordered, observable replication dispatch + +**Classification:** high-risk (replication ordering) +**Estimated implement time:** ~4 min +**Parallelizable with:** 1, 2, 5, 6, 7, 14 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ReplicationService.cs` (`FireAndForget`, lines 134-149), `src/ZB.MOM.WW.ScadaBridge.Commons/Observability/ScadaBridgeTelemetry.cs` (add counter), `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/SiteReplicationActor.cs` (`SendToPeer`, lines 139-149) +- Test: `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationServiceTests.cs` (append) + +`Task.Run` bought nothing (the handler is a non-blocking Akka `Tell` returning `Task.CompletedTask`) and cost ordering: Add/Remove pairs invert on the thread pool. Invoking inline preserves ordering by construction; Akka then guarantees per-sender/receiver order. + +1. Failing test: + +```csharp +[Fact] +public void ReplicationOperations_AreDispatchedInIssueOrder() +{ + var seen = new List<(ReplicationOperationType, string)>(); + var service = new ReplicationService(new StoreAndForwardOptions(), NullLogger.Instance); + service.SetReplicationHandler(op => + { + seen.Add((op.OperationType, op.MessageId)); + return Task.CompletedTask; + }); + + for (var i = 0; i < 200; i++) + { + service.ReplicateEnqueue(new StoreAndForwardMessage { Id = $"m{i}" }); + service.ReplicateRemove($"m{i}"); + } + + // Inline dispatch: by the time the calls return, every op was handed to the + // handler, Add strictly before Remove per id. Pre-fix (Task.Run) this is + // both racy-in-count and racy-in-order. + Assert.Equal(400, seen.Count); + for (var i = 0; i < 200; i++) + { + Assert.Equal((ReplicationOperationType.Add, $"m{i}"), seen[2 * i]); + Assert.Equal((ReplicationOperationType.Remove, $"m{i}"), seen[2 * i + 1]); + } +} + +[Fact] +public void ReplicationHandlerThrow_IsSwallowed_AndLoggedAtWarning() +{ + var logger = new TestLogger(); // xunit-friendly capture, or use a Meter listener + var service = new ReplicationService(new StoreAndForwardOptions(), logger); + service.SetReplicationHandler(_ => throw new InvalidOperationException("peer gone")); + service.ReplicateRemove("m1"); // must not throw + Assert.Contains(logger.Entries, e => e.Level == LogLevel.Warning); +} +``` + + (If no `TestLogger` helper exists in the test project, add a minimal `List<(LogLevel, string)>`-capturing `ILogger` to the test file.) +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests --filter "ReplicationOperations_AreDispatchedInIssueOrder|ReplicationHandlerThrow"` — expect FAIL. +3. Implement: + - `ScadaBridgeTelemetry`: add alongside the existing counters: + +```csharp + private static readonly Counter _replicationFailures = + Meter.CreateCounter("scadabridge.store_and_forward.replication.failures", unit: "1", + description: "S&F buffer replication operations that failed to dispatch/deliver to the peer node"); + /// Records one failed S&F replication dispatch. + public static void RecordReplicationFailure() => _replicationFailures.Add(1); +``` + + - `ReplicationService.FireAndForget` (rename to `DispatchInline`, update the three callers or keep the name with a corrected doc comment): + +```csharp + private void FireAndForget(ReplicationOperation operation) + { + // Invoked inline, NOT via Task.Run: the handler is a non-blocking Akka + // Tell, and thread-pool hand-off destroyed Add/Remove ordering for the + // same message id. Inline invocation preserves issue order; Akka's + // per-sender/receiver guarantee preserves it on the wire. + try + { + var task = _replicationHandler!.Invoke(operation); + if (!task.IsCompletedSuccessfully) + { + task.ContinueWith(t => + { + ScadaBridgeTelemetry.RecordReplicationFailure(); + _logger.LogWarning(t.Exception, + "Replication of {OpType} for message {MessageId} failed (best-effort); standby buffer may be diverging", + operation.OperationType, operation.MessageId); + }, + TaskContinuationOptions.OnlyOnFaulted); + } + } + catch (Exception ex) + { + ScadaBridgeTelemetry.RecordReplicationFailure(); + _logger.LogWarning(ex, + "Replication of {OpType} for message {MessageId} failed (best-effort); standby buffer may be diverging", + operation.OperationType, operation.MessageId); + } + } +``` + + - `SiteReplicationActor.SendToPeer` no-peer drop: raise `LogDebug` → `LogWarning` and call `ScadaBridgeTelemetry.RecordReplicationFailure()` (a dropped op is a lost delta; when the peer is legitimately absent — single-node dev — the warning is rate-tolerable at one per op; acceptable per review). +4. Run the filter + `dotnet test tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests` + `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter SiteReplication`. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ReplicationService.cs src/ZB.MOM.WW.ScadaBridge.Commons/Observability/ScadaBridgeTelemetry.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/SiteReplicationActor.cs tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationServiceTests.cs && git commit -m "fix(store-and-forward): inline ordered replication dispatch; Warning + counter on replication failures"` + +### Task 11: Upsert-based standby applies for Add/Park/Requeue + +**Classification:** standard +**Estimated implement time:** ~4 min +**Parallelizable with:** 2, 5, 6, 14, 16 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs` (add `UpsertMessageAsync`), `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ReplicationService.cs` (`ApplyReplicatedOperationAsync`, lines 107-132) +- Test: `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardReplicationTests.cs` (append) + +Depends on Task 10 (same `ReplicationService.cs`) and Task 7 (same storage file). A Park/Requeue whose original Add was lost self-heals (the full message rides in the operation); a duplicate Add after Task 21's resync no longer violates the PK. + +1. Failing tests: + +```csharp +[Fact] +public async Task ApplyReplicatedPark_WhenAddWasLost_MaterializesTheParkedRow() +{ + var service = new ReplicationService(OptionsWithReplication(), NullLogger.Instance); + var msg = NewMessage("lost-add-1"); // never Added on this (standby) storage + msg.Status = StoreAndForwardMessageStatus.Parked; + + await service.ApplyReplicatedOperationAsync( + new ReplicationOperation(ReplicationOperationType.Park, msg.Id, msg), _storage); + + var row = await _storage.GetMessageByIdAsync("lost-add-1"); + Assert.NotNull(row); // pre-fix: blind UPDATE affected 0 rows, row is gone forever + Assert.Equal(StoreAndForwardMessageStatus.Parked, row!.Status); +} + +[Fact] +public async Task ApplyReplicatedAdd_Twice_IsIdempotent_NewestWins() +{ + var service = new ReplicationService(OptionsWithReplication(), NullLogger.Instance); + var msg = NewMessage("dup-add"); + await service.ApplyReplicatedOperationAsync( + new ReplicationOperation(ReplicationOperationType.Add, msg.Id, msg), _storage); + msg.RetryCount = 3; + await service.ApplyReplicatedOperationAsync( // pre-fix: SqliteException PK violation + new ReplicationOperation(ReplicationOperationType.Add, msg.Id, msg), _storage); + + Assert.Equal(3, (await _storage.GetMessageByIdAsync("dup-add"))!.RetryCount); +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests --filter "ApplyReplicatedPark_WhenAddWasLost|ApplyReplicatedAdd_Twice"` — expect FAIL. +3. Implement: + - `StoreAndForwardStorage.UpsertMessageAsync(StoreAndForwardMessage message)`: same column list/parameters as `EnqueueAsync` but `INSERT INTO sf_messages (...) VALUES (...) ON CONFLICT(id) DO UPDATE SET category=excluded.category, target=excluded.target, payload_json=excluded.payload_json, retry_count=excluded.retry_count, max_retries=excluded.max_retries, retry_interval_ms=excluded.retry_interval_ms, last_attempt_at=excluded.last_attempt_at, status=excluded.status, last_error=excluded.last_error, origin_instance=excluded.origin_instance, execution_id=excluded.execution_id, source_script=excluded.source_script, parent_execution_id=excluded.parent_execution_id` (keep the original `created_at` — do not overwrite it, it orders the sweep). + - `ApplyReplicatedOperationAsync`: `Add`, `Park`, and `Requeue` all call `storage.UpsertMessageAsync(operation.Message)` (after the existing Status/RetryCount mutations for Park/Requeue). `Remove` unchanged. Update the doc comment: replicated applies are upserts so a lost Add self-heals from any later full-message operation. +4. Run the filter + full project. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.StoreAndForward tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests && git commit -m "fix(store-and-forward): standby applies Add/Park/Requeue as upserts so a lost Add self-heals"` + +### Task 12: `deferToSweep` enqueue mode + immediate sweep kick + +**Classification:** high-risk (delivery-path semantics) +**Estimated implement time:** ~5 min +**Parallelizable with:** 2, 5, 6, 14, 16 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardService.cs` (`EnqueueAsync` signature + tail, lines 467-546; new `TriggerSweep`) +- Test: `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardServiceTests.cs` (append) + +Depends on Tasks 8/9 (same file). Groundwork for Task 13: `Notify.Send` must never run the 30 s central Ask inline on the script's await. `deferToSweep: true` buffers without invoking the handler and **without** stamping `LastAttemptAt` (row is due immediately), then kicks a background sweep so the healthy-path latency is milliseconds, not one timer interval. + +1. Failing tests: + +```csharp +[Fact] +public async Task Enqueue_DeferToSweep_DoesNotInvokeHandlerInline_AndRowIsDueImmediately() +{ + var service = CreateService(); + var inlineInvoked = false; + service.RegisterDeliveryHandler(StoreAndForwardCategory.Notification, + _ => { inlineInvoked = true; return Task.FromResult(true); }); + + var result = await service.EnqueueAsync(StoreAndForwardCategory.Notification, "ops", "{}", + deferToSweep: true, messageId: "n1"); + + Assert.False(inlineInvoked); + Assert.True(result.WasBuffered); + var row = await service.GetMessageByIdAsync("n1"); + Assert.Null(row!.LastAttemptAt); // due on the very next sweep, not after RetryInterval +} + +[Fact] +public async Task Enqueue_DeferToSweep_AfterStart_KicksAnImmediateSweep() +{ + var service = CreateService(retryTimerInterval: TimeSpan.FromHours(1)); // timer will never fire in-test + var delivered = new TaskCompletionSource(); + service.RegisterDeliveryHandler(StoreAndForwardCategory.Notification, + _ => { delivered.TrySetResult(); return Task.FromResult(true); }); + await service.StartAsync(); + try + { + await service.EnqueueAsync(StoreAndForwardCategory.Notification, "ops", "{}", deferToSweep: true); + await delivered.Task.WaitAsync(TimeSpan.FromSeconds(5)); // pre-fix: times out + } + finally { await service.StopAsync(); } +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests --filter "Enqueue_DeferToSweep"` — expect FAIL (no parameter). +3. Implement: + - Add parameter `bool deferToSweep = false` to `EnqueueAsync` (after `attemptImmediateDelivery`), XML-doc: *"When true, the handler is never invoked inline — the message is buffered due-immediately (`LastAttemptAt` stays null) and a background sweep is kicked. Use for callers on latency-sensitive threads (script dispatchers) whose delivery involves a long Ask timeout. Unlike `attemptImmediateDelivery: false`, no delivery attempt is presumed to have been made."* + - Guard: `if (deferToSweep) { await BufferAsync(message); RaiseActivity("Queued", category, $"Deferred to sweep: {target}"); TriggerSweep(); return new StoreAndForwardResult(true, message.Id, true); }` placed **before** the immediate-delivery block. (`LastAttemptAt` intentionally not set.) + - Add: + +```csharp + /// + /// Kicks a background retry sweep now (fire-and-forget). No-op before + /// StartAsync (storage may be uninitialized). Overlap-safe: the sweep's + /// _retryInProgress CAS makes a concurrent kick a cheap no-op. + /// + public void TriggerSweep() + { + if (_retryTimer == null) return; + Volatile.Write(ref _sweepTask, RetryPendingMessagesAsync()); + } +``` + + - `CreateService` helper: add optional `retryTimerInterval` parameter feeding `StoreAndForwardOptions.RetryTimerInterval` if not already supported. +4. Run the filter + full project. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardService.cs tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardServiceTests.cs && git commit -m "feat(store-and-forward): deferToSweep enqueue mode with immediate sweep kick for latency-sensitive callers"` + +### Task 13: Unstrand and unblock `Notify.Send` (maxRetries: 0 + deferToSweep) + docs + +**Classification:** high-risk (script-facing semantics) +**Estimated implement time:** ~5 min +**Parallelizable with:** 5, 6, 14, 16 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptRuntimeContext.cs` (the `EnqueueAsync` call at lines ~2197-2203) +- Modify: `docs/requirements/Component-StoreAndForward.md` (notification-delivery paragraph, ~line 52 region), `CLAUDE.md` (the "`Notify.Send` is async — returns … immediately" bullet: note it is now enqueue-only + swept, unbounded retries) +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/NotifyHelperTests.cs` (append) + +Depends on Task 12. Two changes at the single notification producer: `maxRetries: 0` (the documented no-limit escape hatch — a central maintenance window can no longer strand notifications into per-message operator unparking) and `deferToSweep: true` (a `Notify.Send` during a central outage returns in milliseconds instead of blocking a scarce script thread for the 30 s `NotificationForwardTimeout`). + +1. Failing test (the `NotifyHelperTests` harness constructs a real `StoreAndForwardService` at line 45): + +```csharp +[Fact] +public async Task NotifySend_BuffersUnbounded_AndNeverInvokesForwarderInline() +{ + var inlineInvoked = false; + _saf.RegisterDeliveryHandler(StoreAndForwardCategory.Notification, + _ => { inlineInvoked = true; throw new TimeoutException("central down"); }); + + var notificationId = await SendViaNotifyHelper("ops-list", "hello"); // existing helper pattern + + Assert.False(inlineInvoked); // pre-fix: Send ran the forwarder Ask inline + var row = await _saf.GetMessageByIdAsync(notificationId); + Assert.NotNull(row); + Assert.Equal(0, row!.MaxRetries); // pre-fix: 50 → parked after ~25 min of outage + Assert.Equal(StoreAndForwardMessageStatus.Pending, row.Status); +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter NotifySend_BuffersUnbounded` — expect FAIL. +3. Implement — the `ScriptRuntimeContext` call becomes: + +```csharp + await _storeAndForward.EnqueueAsync( + StoreAndForwardCategory.Notification, + target: _listName, + payloadJson: payloadJson, + originInstanceName: _instanceName, + // 0 = the documented "no limit" escape hatch (StoreAndForward-015): + // notifications are retried until central acks and are never parked + // for retry exhaustion — a long central outage must not strand them + // behind per-message operator unparking. + maxRetries: 0, + // Never run the forwarder's 30s central Ask inline on the script + // thread: buffer due-immediately and kick the sweep. Send returns + // in milliseconds whether central is up or down. + deferToSweep: true, + messageId: notificationId); +``` + +4. Update the docs: `Component-StoreAndForward.md` — rewrite the "bounded by DefaultMaxRetries … will park the buffered notification" passage: `Notify.Send` now enqueues with `maxRetries: 0`; notifications never park for retry exhaustion (corrupt payloads still park — Task 14); `Notify.Send` is enqueue-only (no inline forward attempt) with an immediate sweep kick, so its worst-case latency is the local SQLite insert. `CLAUDE.md`: adjust the `Notify.Send` bullet accordingly. +5. Run the filter + `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter Notify` — expect PASS (fix any existing Notify tests that asserted the inline-attempt behavior). +6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptRuntimeContext.cs docs/requirements/Component-StoreAndForward.md CLAUDE.md tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/NotifyHelperTests.cs && git commit -m "fix(notifications): Notify.Send enqueues unbounded (maxRetries 0) and defers to sweep — no 30s script-thread block, no stranding"` + +### Task 14: Park (don't silently drop) corrupt buffered notifications + +**Classification:** standard +**Estimated implement time:** ~4 min +**Parallelizable with:** 1, 2, 3, 5, 7, 10, 16 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/NotificationForwarder.cs` (lines 77-106) +- Test: `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/NotificationForwarderTests.cs` (modify lines 201-255 — the two StoreAndForward-018 regression tests invert) +- Modify: `docs/requirements/Component-StoreAndForward.md` (note superseding StoreAndForward-018) + +**Deliberate reversal of StoreAndForward-018.** That decision discarded corrupt payloads to honour "notifications do not park". The doc itself now frames "do not park" as happy-path intent, and Task 13 makes retry-exhaustion parking impossible (`maxRetries: 0`) — so parking is exactly the designed home for an *undeliverable-but-preserved* payload: the operator gets a parked row with the payload for forensics instead of a silent site-log-only vanish that central reports as delivered. + +1. Rewrite the two tests (rename; flip assertions): + +```csharp +[Fact] +public async Task Deliver_CorruptJsonPayload_ReturnsFalse_ParksForForensics_AndDoesNotForward() +{ + // Supersedes StoreAndForward-018 (discard-on-corrupt). Returning false is the + // handler contract's permanent-failure signal: the engine parks the row, which + // preserves the payload for operator forensics. Retrying a corrupt payload is + // pointless; deleting it silently (and reporting Delivered) loses data with no + // parked row, no central Notifications row, and no audit trail. + var centralProbe = CreateTestProbe(); + var forwarder = new NotificationForwarder(centralProbe.Ref, "site-7", ForwardTimeout); + var corrupt = new StoreAndForwardMessage + { + Id = "msg-corrupt", Category = StoreAndForwardCategory.Notification, + Target = "Operators", PayloadJson = "{not-valid-json", OriginInstanceName = "Plant.Pump3", + }; + + Assert.False(await forwarder.DeliverAsync(corrupt)); + centralProbe.ExpectNoMsg(TimeSpan.FromMilliseconds(200)); +} +``` + + (Mirror for the `"null"` payload test.) +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests --filter "Deliver_CorruptJsonPayload|Deliver_NullDeserializedPayload"` — expect FAIL. +3. Implement in `DeliverAsync`: + +```csharp + if (!TryBuildSubmit(message, out var submit)) + { + _logger.LogWarning( + "Parking corrupt buffered notification {NotificationId} (payload is not deserialisable as NotificationSubmit); " + + "the row is preserved for operator forensics. Payload preview: {PayloadPreview}", + message.Id, + PreviewPayload(message.PayloadJson)); + // false = permanent failure by the delivery-handler contract → the + // engine parks the row (payload preserved, operator can inspect or + // discard). Supersedes StoreAndForward-018's silent discard, which + // reported the notification as delivered while losing it entirely. + return false; + } +``` + + Update the class XML doc list (`true`/throws/`false` outcomes) to include the park case. +4. Update `Component-StoreAndForward.md`: corrupt buffered notifications park (StoreAndForward-018 superseded); note the two parking causes for notifications are now *corrupt payload only* (retry exhaustion is off via `maxRetries: 0`). +5. Run the filter + full project. +6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.StoreAndForward/NotificationForwarder.cs tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/NotificationForwarderTests.cs docs/requirements/Component-StoreAndForward.md && git commit -m "fix(notifications): park corrupt buffered notification payloads instead of silently discarding as delivered (supersedes StoreAndForward-018)"` + +### Task 15: Wire-format pin tests for cross-cluster contracts + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 3, 5, 7, 10, 14 +**Files:** +- Test: `tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/WireSerializationPinTests.cs` (create), `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationWireSerializationPinTests.cs` (create) + +No production code change. The Akka default reflective-JSON wire format couples compatibility to *type and namespace identity*; nothing tests it (`MessageContractTests` asserts correlation-ids only). These tests pin (a) round-trip fidelity and (b) the fully-qualified type identity of the highest-traffic cross-cluster contracts, so a rename/move that would break rolling central↔site compat fails a test instead of production. The serializer *swap* (proto/explicit bindings) is **deferred** — it changes the wire format and needs a coordinated rolling-upgrade plan. + +1. Write the tests (they pass immediately if the wire behavior is sane — the value is the pin; expect PASS on first run, and that's fine: the "failing" phase here is meaningless, this is characterization): + +```csharp +// tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/WireSerializationPinTests.cs +public class WireSerializationPinTests : TestKit +{ + private T RoundTrip(T message) + { + var serialization = Sys.Serialization; + var serializer = serialization.FindSerializerFor(message); + var bytes = serializer.ToBinary(message); + return (T)serialization.Deserialize(bytes, serializer.Identifier, + message!.GetType().TypeQualifiedName()); + } + + [Fact] + public void HeartbeatMessage_RoundTripsOnTheWire() + { + var original = new HeartbeatMessage("site-1", "host-a", true, DateTimeOffset.UtcNow); + var back = RoundTrip(original); + Assert.Equal(original, back); + } + + [Fact] + public void NotificationSubmit_RoundTripsOnTheWire() { /* construct with all fields non-default; assert field equality */ } + + [Fact] + public void IngestAuditEventsCommand_RoundTripsOnTheWire() { /* one fully-populated AuditEvent entity */ } + + [Fact] + public void SiteHealthReport_RoundTripsOnTheWire() { /* minimal + one dictionary entry per collection */ } + + // Type-identity pins: the wire embeds CLR type manifests; a rename/move of any + // of these types breaks rolling cross-version central↔site communication. + // If one of these assertions fails, you are making a wire-breaking change — + // coordinate a full-fleet upgrade or add an explicit serialization binding. + [Theory] + [InlineData(typeof(HeartbeatMessage), "ZB.MOM.WW.ScadaBridge.Commons.Messages.Health.HeartbeatMessage")] + [InlineData(typeof(NotificationSubmit), "ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification.NotificationSubmit")] + [InlineData(typeof(IngestAuditEventsCommand), "ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit.IngestAuditEventsCommand")] + public void CrossClusterContract_TypeIdentity_IsPinned(Type type, string expectedFullName) => + Assert.Equal(expectedFullName, type.FullName); +} +``` + + And in `ReplicationWireSerializationPinTests.cs` (StoreAndForward.Tests, which references the S&F assembly): round-trip `ReplicationOperation` carrying a fully-populated `StoreAndForwardMessage` (every property non-default, including `ExecutionId`/`ParentExecutionId`) and pin its type identity. +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Communication.Tests --filter WireSerializationPinTests` and `dotnet test tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests --filter ReplicationWireSerializationPinTests` — expect PASS. If any round-trip FAILS, that is a live wire-fidelity bug: fix the message type (missing ctor for deserialization, get-only property), not the test. +3. Commit: `git add tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/WireSerializationPinTests.cs tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationWireSerializationPinTests.cs && git commit -m "test(communication): pin wire round-trip + type identity of cross-cluster message contracts"` + +### Task 16: Replicate heartbeat marks to the peer central node + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** 3, 5, 7, 10, 14 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs` (`HandleHeartbeat` lines 349-353; new record + Receive) +- Test: `tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/CentralCommunicationActorTests.cs` (append) + +Depends on Task 2 (same file). Mirrors the existing `SiteHealthReportReplica` pattern exactly; `MarkHeartbeat` is idempotent (timestamp overwrite), so self-delivery via pub-sub is harmless. + +1. Failing test (follow the existing DI-stub pattern that substitutes `ICentralHealthAggregator`): + +```csharp +[Fact] +public void HeartbeatReplica_MarksLocalAggregator_WithoutRebroadcast() +{ + var aggregator = Substitute.For(); + var actor = CreateActorWith(aggregator); // existing helper pattern + var ts = DateTimeOffset.UtcNow; + + actor.Tell(new SiteHeartbeatReplica(new HeartbeatMessage("site-1", "host-a", true, ts))); + + AwaitAssert(() => aggregator.Received(1).MarkHeartbeat("site-1", ts)); +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Communication.Tests --filter HeartbeatReplica_MarksLocalAggregator` — expect FAIL (record missing). +3. Implement: + - Record (bottom of file, next to the other registration records): `/// Peer-replication envelope for a site heartbeat, fanned out over the same DistributedPubSub topic as SiteHealthReportReplica so both central aggregators mark heartbeats regardless of which node the site's ClusterClient delivered to. public sealed record SiteHeartbeatReplica(HeartbeatMessage Heartbeat);` + - Constructor: `Receive(r => MarkHeartbeatLocally(r.Heartbeat));` + - `HandleHeartbeat`: extract `MarkHeartbeatLocally(heartbeat)` (the existing two lines), then publish `new Publish(HealthReportTopic, new SiteHeartbeatReplica(heartbeat))` inside the same try/catch-no-op-for-TestKit pattern used by `HandleSiteHealthReport`. +4. Run the filter + full project. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/CentralCommunicationActorTests.cs && git commit -m "fix(communication): replicate heartbeat marks to the peer central node (closes post-failover offline blind window)"` + +### Task 17: S&F SQLite hygiene — WAL + busy_timeout + +**Classification:** standard +**Estimated implement time:** ~4 min +**Parallelizable with:** 2, 5, 6, 13, 14, 16 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs` (`InitializeAsync` line 32-79; new `OpenConnectionAsync` helper; replace every `new SqliteConnection` + `OpenAsync` pair — ~12 sites) +- Test: `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardStorageTests.cs` (append) + +Depends on Task 11 (same file — includes the new `UpsertMessageAsync`). + +1. Failing test: + +```csharp +[Fact] +public async Task Initialize_EnablesWalJournalMode_OnFileDatabase() +{ + var path = Path.Combine(_tempDir, "wal-test.db"); + var storage = new StoreAndForwardStorage($"Data Source={path}", NullLogger.Instance); + await storage.InitializeAsync(); + + await using var conn = new SqliteConnection($"Data Source={path}"); + await conn.OpenAsync(); + await using var cmd = conn.CreateCommand(); + cmd.CommandText = "PRAGMA journal_mode"; + Assert.Equal("wal", (string)(await cmd.ExecuteScalarAsync())!); +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests --filter Initialize_EnablesWalJournalMode` — expect FAIL (`delete`). +3. Implement: + - `InitializeAsync`, right after `OpenAsync()`: execute `PRAGMA journal_mode=WAL;` (persistent, file-scoped; in-memory DBs return `memory` — harmless, don't assert). + - Add and use everywhere: + +```csharp + /// + /// Opens a connection with a 5s busy_timeout. Concurrent writers exist by + /// design (script enqueues, the sweep's lanes, standby replication applies, + /// central pull queries); with connection-per-operation the pragma must be + /// set per open (it is per-connection, and Microsoft.Data.Sqlite pooling + /// makes the extra statement cheap on a pooled physical connection). + /// + private async Task OpenConnectionAsync() + { + var connection = new SqliteConnection(_connectionString); + await connection.OpenAsync(); + await using var pragma = connection.CreateCommand(); + pragma.CommandText = "PRAGMA busy_timeout = 5000"; + await pragma.ExecuteNonQueryAsync(); + return connection; + } +``` + + - Mechanical replace in every method: `await using var connection = new SqliteConnection(_connectionString); await connection.OpenAsync();` → `await using var connection = await OpenConnectionAsync();`. +4. Run the filter + full project. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardStorageTests.cs && git commit -m "perf(store-and-forward): enable WAL + per-connection busy_timeout on the S&F SQLite store"` + +### Task 18: Epoch-ms due-check (replace per-row julianday parsing) + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 2, 5, 6, 13, 14, 16 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs` (schema migration in `InitializeAsync`; writers `EnqueueAsync`/`UpdateMessageAsync`/`UpdateMessageIfStatusAsync`/`UpsertMessageAsync`/`RetryParkedMessageAsync`; due query lines 189-198) +- Test: `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardStorageTests.cs` (append) + +Depends on Task 17 (same file). Additive column, existing `AddColumnIfMissingAsync` pattern; the ISO-8601 text column stays authoritative for reads/back-compat, the ms column drives the due predicate. + +1. Failing tests: + +```csharp +[Fact] +public async Task GetMessagesForRetry_UsesEpochMs_RespectsInterval() +{ + var msg = NewPendingMessage("due-check", retryIntervalMs: 60_000); + msg.LastAttemptAt = DateTimeOffset.UtcNow.AddSeconds(-30); // not yet due + await _storage.EnqueueAsync(msg); + Assert.Empty(await _storage.GetMessagesForRetryAsync()); + + msg.LastAttemptAt = DateTimeOffset.UtcNow.AddSeconds(-90); // due + await _storage.UpdateMessageAsync(msg); + Assert.Single(await _storage.GetMessagesForRetryAsync()); +} + +[Fact] +public async Task Initialize_BackfillsEpochMs_ForLegacyRows() +{ + await _storage.InitializeAsync(); + // Simulate a legacy row: text timestamp present, ms column NULL. + await ExecRaw("INSERT INTO sf_messages (id, category, target, payload_json, created_at, last_attempt_at, status) " + + "VALUES ('legacy', 0, 't', '{}', @c, @l, 0)", + ("@c", DateTimeOffset.UtcNow.ToString("O")), + ("@l", DateTimeOffset.UtcNow.AddHours(-1).ToString("O"))); + await _storage.InitializeAsync(); // second init runs the backfill + var ms = await ScalarRaw("SELECT last_attempt_at_ms FROM sf_messages WHERE id='legacy'"); + Assert.NotNull(ms); + Assert.InRange(ms!.Value, DateTimeOffset.UtcNow.AddHours(-1).AddMinutes(-1).ToUnixTimeMilliseconds(), + DateTimeOffset.UtcNow.ToUnixTimeMilliseconds()); +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests --filter "GetMessagesForRetry_UsesEpochMs|Initialize_BackfillsEpochMs"` — expect FAIL (column missing). +3. Implement: + - `InitializeAsync`: `await AddColumnIfMissingAsync(connection, "last_attempt_at_ms", "INTEGER");` then a one-time backfill: `UPDATE sf_messages SET last_attempt_at_ms = CAST((julianday(last_attempt_at) - 2440587.5) * 86400000 AS INTEGER) WHERE last_attempt_at IS NOT NULL AND last_attempt_at_ms IS NULL` (the only remaining julianday use — runs once per legacy DB, then never matches). Add `CREATE INDEX IF NOT EXISTS idx_sf_messages_status_due ON sf_messages(status, last_attempt_at_ms);`. + - Every writer that sets `@lastAttempt` also sets `@lastAttemptMs` (`message.LastAttemptAt?.ToUnixTimeMilliseconds()` or DBNull); `RetryParkedMessageAsync` sets `last_attempt_at_ms = NULL`. + - Due query: `WHERE status = @pending AND (last_attempt_at_ms IS NULL OR retry_interval_ms = 0 OR (@nowMs - last_attempt_at_ms) >= retry_interval_ms)` with `cmd.Parameters.AddWithValue("@nowMs", DateTimeOffset.UtcNow.ToUnixTimeMilliseconds())`. +4. Run the filter + full project (existing due-check tests must still pass). +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardStorageTests.cs && git commit -m "perf(store-and-forward): epoch-ms due-check with additive column, backfill, and (status, due) index"` + +### Task 19: Decouple audit-observer notification from the sweep (ordered channel) + +**Classification:** high-risk (concurrency; audit-event ordering) +**Estimated implement time:** ~5 min +**Parallelizable with:** 2, 5, 6, 14, 16 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardService.cs` (`StartAsync`/`StopAsync`; the three `NotifyCachedCallObserverAsync` await sites at lines ~620-626, 688-694, 711-717 and the transient site ~711) +- Test: `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/CachedCallAttemptEmissionTests.cs` (append) + +Depends on Task 12 (same file). A slow observer (SQLite audit write) must not stretch the sweep; a single-reader unbounded channel preserves per-operation event order. + +1. Failing test: + +```csharp +[Fact] +public async Task SlowObserver_DoesNotBlockTheSweep_AndEventsArriveInOrder() +{ + var gate = new TaskCompletionSource(); + var observed = new List(); + var observer = new DelegatingObserver(async ctx => + { + await gate.Task; // first call blocks until released + lock (observed) observed.Add(ctx.Outcome); + }); + var service = CreateServiceWithObserver(observer); // existing harness pattern + await service.StartAsync(); + try + { + service.RegisterDeliveryHandler(StoreAndForwardCategory.ExternalSystem, + _ => Task.FromResult(true)); + await service.EnqueueAsync(StoreAndForwardCategory.ExternalSystem, "t", "{}", + attemptImmediateDelivery: false, retryInterval: TimeSpan.Zero, + messageId: TrackedOperationId.New().ToString()); + + var sweep = service.RetryPendingMessagesAsync(); + await sweep.WaitAsync(TimeSpan.FromSeconds(5)); // pre-fix: blocks on gate → times out + + gate.SetResult(); + await AssertEventually(() => { lock (observed) return observed.Count == 1; }); + } + finally { await service.StopAsync(); } +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests --filter SlowObserver_DoesNotBlockTheSweep` — expect FAIL (timeout). +3. Implement: + - Fields: `private readonly Channel> _observerQueue = Channel.CreateUnbounded>(new UnboundedChannelOptions { SingleReader = true }); private Task? _observerPump;` + - `StartAsync`: `_observerPump = Task.Run(async () => { await foreach (var work in _observerQueue.Reader.ReadAllAsync()) { try { await work(); } catch (Exception ex) { _logger.LogWarning(ex, "Cached-call audit observer threw; ignored (best-effort per alog.md §7)"); } } });` + - The three sweep call sites become non-awaiting posts: `PostObserverNotification(message, outcome, lastError, httpStatus, attemptStartUtc, durationMs);` where `PostObserverNotification` snapshots nothing extra (each branch finishes mutating `message` before posting; the sweep re-reads fresh row instances next tick) and does `_observerQueue.Writer.TryWrite(() => NotifyCachedCallObserverAsync(...));`. + - `StopAsync` (before clearing the gauge): `_observerQueue.Writer.TryComplete(); if (_observerPump is not null) { try { await _observerPump.WaitAsync(SweepShutdownWaitTimeout); } catch (TimeoutException) { _logger.LogWarning("Audit-observer pump did not drain within {Timeout}", SweepShutdownWaitTimeout); } }` — note: after TryComplete a restarted service needs a fresh channel; make the channel a non-readonly field re-created in `StartAsync` if existing tests restart the same instance. + - XML-doc on `NotifyCachedCallObserverAsync`: now invoked from the single-reader pump — latency-isolated from the sweep, order preserved per posting sequence. +4. Run the filter + full project (especially `CachedCallAttemptEmissionTests` — they may need `StartAsync`/`StopAsync` bracketing or an explicit pump-drain helper; keep their ordering assertions intact). +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardService.cs tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/CachedCallAttemptEmissionTests.cs && git commit -m "perf(store-and-forward): post audit-observer notifications to an ordered single-reader channel, latency-isolating the sweep"` + +### Task 20: Buffer resync storage primitives (`GetAllMessagesAsync` / `ReplaceAllAsync`) + +**Classification:** standard +**Estimated implement time:** ~4 min +**Parallelizable with:** 2, 5, 6, 13, 14, 16, 19 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs` +- Test: `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardStorageTests.cs` (append) + +Depends on Task 18 (same file). Anti-entropy groundwork (report Underdeveloped #4). + +1. Failing tests: + +```csharp +[Fact] +public async Task GetAllMessages_ReturnsEveryStatus_OldestFirst_UpToLimit() +{ + await _storage.EnqueueAsync(NewMessage("p1", status: StoreAndForwardMessageStatus.Pending, createdAt: Base)); + await _storage.EnqueueAsync(NewMessage("k1", status: StoreAndForwardMessageStatus.Parked, createdAt: Base.AddSeconds(1))); + await _storage.EnqueueAsync(NewMessage("p2", status: StoreAndForwardMessageStatus.Pending, createdAt: Base.AddSeconds(2))); + + var (all, truncated) = await _storage.GetAllMessagesAsync(limit: 2); + Assert.Equal(new[] { "p1", "k1" }, all.Select(m => m.Id)); + Assert.True(truncated); +} + +[Fact] +public async Task ReplaceAll_SwapsTheEntireBuffer_Atomically() +{ + await _storage.EnqueueAsync(NewMessage("stale")); + await _storage.ReplaceAllAsync(new[] { NewMessage("fresh-1"), NewMessage("fresh-2") }); + + Assert.Null(await _storage.GetMessageByIdAsync("stale")); + Assert.NotNull(await _storage.GetMessageByIdAsync("fresh-1")); + Assert.NotNull(await _storage.GetMessageByIdAsync("fresh-2")); +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests --filter "GetAllMessages_ReturnsEveryStatus|ReplaceAll_SwapsTheEntireBuffer"` — expect FAIL. +3. Implement: + - `GetAllMessagesAsync(int limit)`: the standard SELECT column list, no status filter, `ORDER BY created_at ASC LIMIT @limitPlusOne` — fetch `limit + 1`, return `(rows.Take(limit).ToList(), rows.Count > limit)`. + - `ReplaceAllAsync(IReadOnlyList messages)`: one transaction — `DELETE FROM sf_messages`, then insert each row (extract the parameter-binding block shared with `EnqueueAsync` into a private `BindMessageParameters(SqliteCommand, StoreAndForwardMessage)` helper), commit. XML-doc: standby-side anti-entropy apply; never call on an active node. +4. Run the filter + full project. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardStorageTests.cs && git commit -m "feat(store-and-forward): GetAllMessagesAsync/ReplaceAllAsync storage primitives for standby buffer resync"` + +### Task 21: Peer-join S&F buffer resync (anti-entropy) + +**Classification:** high-risk (cluster protocol) +**Estimated implement time:** ~5 min +**Parallelizable with:** 2, 5, 6, 14, 16 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/SiteReplicationActor.cs` (new messages, handlers, trigger in `TryTrackPeer`; ctor gains `Func? isActiveOverride = null` seam) +- Modify: `docs/requirements/Component-StoreAndForward.md` (Persistence section — document the resync) +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReplicationActorTests.cs` (append; follow the existing `SendToPeer`-override subclass pattern) + +Depends on Tasks 10 (same file) and 20 (primitives). Closes "a standby down for an hour rejoins and diverges forever": on tracking a peer, a **standby** requests a full-buffer snapshot; the **active** node answers with up to `MaxResyncRows` (10 000) rows; the standby replaces its buffer wholesale. Combined with Task 11's upserts, later replicated ops land cleanly on the resynced state. + +1. Failing tests: + +```csharp +[Fact] +public void StandbyTrackingPeer_SendsResyncRequest() +{ + var actor = CreateActor(isActive: () => false, sendToPeerCapture: out var sent); + SimulatePeerUp(actor); // existing MemberUp-injection pattern + AwaitAssert(() => Assert.Contains(sent, m => m is RequestSfBufferResync)); +} + +[Fact] +public void ActiveTrackingPeer_DoesNotRequestResync() +{ + var actor = CreateActor(isActive: () => true, sendToPeerCapture: out var sent); + SimulatePeerUp(actor); + ExpectNoMsg(200.Milliseconds()); + Assert.DoesNotContain(sent, m => m is RequestSfBufferResync); +} + +[Fact] +public async Task ActiveNode_AnswersResyncRequest_WithFullBufferSnapshot() +{ + await _sfStorage.EnqueueAsync(NewMessage("m1")); + var actor = CreateActor(isActive: () => true, sendToPeerCapture: out _); + actor.Tell(new RequestSfBufferResync(), TestActor); + var snapshot = ExpectMsg(); + Assert.Single(snapshot.Messages); + Assert.False(snapshot.Truncated); +} + +[Fact] +public async Task StandbyNode_AppliesSnapshot_ReplacingItsBuffer() +{ + await _sfStorage.EnqueueAsync(NewMessage("stale")); + var actor = CreateActor(isActive: () => false, sendToPeerCapture: out _); + actor.Tell(new SfBufferSnapshot(new List { NewMessage("fresh") }, false)); + await AwaitAssertAsync(async () => + { + Assert.Null(await _sfStorage.GetMessageByIdAsync("stale")); + Assert.NotNull(await _sfStorage.GetMessageByIdAsync("fresh")); + }); +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "ResyncRequest|ResyncRequest_WithFullBufferSnapshot|AppliesSnapshot_Replacing"` — expect FAIL (messages missing). +3. Implement in `SiteReplicationActor.cs`: + - Messages (same file, public — they cross Akka remoting; POCO payload rides the default serializer, pinned by Task 15's pattern): + +```csharp +/// Standby→active: request a full S&F buffer snapshot for anti-entropy resync (sent when a standby (re)tracks a peer). +public sealed record RequestSfBufferResync; + +/// Active→standby: full-buffer snapshot; Truncated=true when the buffer exceeded MaxResyncRows (standby logs a Warning — divergence beyond the cap drains naturally). +public sealed record SfBufferSnapshot(List Messages, bool Truncated); +``` + + - Ctor: `Func? isActiveOverride = null` stored as `_isActive = isActiveOverride ?? DefaultIsActive;` where `DefaultIsActive` is the repo-standard leader+Up check via `_cluster` (same body as `SiteCommunicationActor.DefaultIsActiveCheck`; swap point for plan 01's helper). + - `TryTrackPeer`: after setting `_peerAddress`, `if (!SafeIsActive()) SendToPeer(new RequestSfBufferResync());` (`SafeIsActive` wraps `_isActive()` in try/catch→false). + - `Receive`: only when `SafeIsActive()`; capture `var replyTo = Sender;` then pipe `_sfStorage.GetAllMessagesAsync(MaxResyncRows)` (const `int MaxResyncRows = 10_000`) into `replyTo.Tell(new SfBufferSnapshot(rows, truncated))` via `Task.Run(...).PipeTo(replyTo, success: t => new SfBufferSnapshot(t.Rows, t.Truncated))`. Standby receiving a request ignores it (Debug log). + - `Receive`: only when `!SafeIsActive()`; fire-and-forget `_sfStorage.ReplaceAllAsync(msg.Messages)` with a `ContinueWith` error log (matches `HandleApplyStoreAndForward`'s pattern); Warning when `msg.Truncated`. +4. Update `Component-StoreAndForward.md` Persistence: document the peer-join resync (trigger, direction, 10k cap, replace-all semantics, why upserts make trailing ops safe). +5. Run the filter + `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests`. +6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/SiteReplicationActor.cs docs/requirements/Component-StoreAndForward.md tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests && git commit -m "feat(store-and-forward): peer-join full-buffer resync — standby anti-entropy for the S&F buffer"` + +### Task 22: Conventions & observability cleanup batch (Lows) + +**Classification:** small (batched Lows, concrete edits) +**Estimated implement time:** ~5 min +**Parallelizable with:** 4, 13, 17, 18, 20 (blocked only by 16 — shared `CentralCommunicationActor.cs`) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcServer.cs` (lines 243-258), `src/ZB.MOM.WW.ScadaBridge.Communication/Actors/StreamRelayActor.cs` (lines 105-111), `src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs` (lines 114-124), `src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptions.cs`, `src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteCommunicationActor.cs` (line 425), `docs/requirements/Component-Communication.md` +- Test: `tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/CommunicationOptionsTests.cs` (append) + +Five concrete edits: + +1. **[Low: check-then-act]** `SiteStreamGrpcServer.cs:243-245` — add a comment above the `_activeStreams.Count >= _maxConcurrentStreams` check: `// Deliberately check-then-act: two concurrent subscribes at the boundary can both pass, overshooting the cap by at most (concurrent subscribers - 1). Bounded and benign — not worth a lock on this path.` +2. **[Low: DropOldest observability]** `SiteStreamGrpcServer.cs:256-257` — count evictions via the `itemDropped` overload: + +```csharp + long dropped = 0; + var correlationId = request.CorrelationId; + var channel = Channel.CreateBounded( + new BoundedChannelOptions(1000) { FullMode = BoundedChannelFullMode.DropOldest }, + _ => + { + // Lossy-under-backpressure is the debug-view spec; make the real + // loss visible: first eviction + every 500th thereafter. + var n = Interlocked.Increment(ref dropped); + if (n == 1 || n % 500 == 0) + _logger.LogWarning( + "Debug stream {CorrelationId} backpressure: {Dropped} oldest event(s) evicted so far", + correlationId, n); + }); +``` + + And in `StreamRelayActor.WriteToChannel` (lines 105-111), replace the dead warning: `// TryWrite on a DropOldest bounded channel never returns false — eviction of the oldest item is counted by the channel's itemDropped callback (SiteStreamGrpcServer). This branch guards only a completed/closed channel.` and demote the log to Debug ("channel closed, dropping event"). +3. **[Low: duplicated timeout constants]** `SiteStreamGrpcServer.cs:45` — change `private static readonly TimeSpan AuditIngestAskTimeout` to `internal static readonly` with a doc note that `CentralCommunicationActor` shares it; `CentralCommunicationActor.cs:114-124` — delete `DefaultAuditIngestAskTimeout` and initialize `_auditIngestAskTimeout = auditIngestAskTimeout ?? Grpc.SiteStreamGrpcServer.AuditIngestAskTimeout;` (same assembly). Keep the fault-semantics divergence comment but append: `// Consolidating the two ingest transports (gRPC vs ClusterClient) is a scheduled follow-up — see arch review 02, Underdeveloped #1.` +4. **[Low: interval double duty]** `CommunicationOptions.cs` — add `/// Application-level site→central heartbeat cadence. Distinct from TransportHeartbeatInterval (the Akka.Remote failure-detector setting) — tuning the transport FD must not silently retune the health heartbeat. public TimeSpan ApplicationHeartbeatInterval { get; set; } = TimeSpan.FromSeconds(5);` and switch `SiteCommunicationActor.PreStart` (line 425) to `_options.ApplicationHeartbeatInterval`. Default equals the old effective value — no behavior change. Test: `[Fact] public void ApplicationHeartbeatInterval_DefaultsTo5s_IndependentOfTransport() { var o = new CommunicationOptions(); Assert.Equal(TimeSpan.FromSeconds(5), o.ApplicationHeartbeatInterval); o.TransportHeartbeatInterval = TimeSpan.FromSeconds(30); Assert.Equal(TimeSpan.FromSeconds(5), o.ApplicationHeartbeatInterval); }` +5. **[U7 doc]** `Component-Communication.md` (debug-stream section): add an explicit accepted-limitation note — debug sessions are process-local on the central node; a central restart/failover drops them with no `OnStreamTerminated` signal; the engineer re-establishes the session. + +Steps: write the `CommunicationOptionsTests` test first → `dotnet test tests/ZB.MOM.WW.ScadaBridge.Communication.Tests --filter ApplicationHeartbeatInterval` (FAIL) → apply all five edits → run full `dotnet test tests/ZB.MOM.WW.ScadaBridge.Communication.Tests` (PASS) → commit: `git add src/ZB.MOM.WW.ScadaBridge.Communication docs/requirements/Component-Communication.md tests/ZB.MOM.WW.ScadaBridge.Communication.Tests && git commit -m "chore(communication): low-severity cleanup — eviction counting, shared ingest timeout, split app heartbeat interval, doc notes"` + +### Task 23: Oldest-parked-age health metric (parked-retention aging signal) + +**Classification:** standard (additive message contract) +**Estimated implement time:** ~5 min +**Parallelizable with:** 2, 5, 6, 13, 16, 21 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs` (add `GetOldestParkedCreatedAtAsync`), `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReport.cs` (additive field), `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/HealthReportSender.cs` (lines ~105-118, next to the parked-count block) + the `ISiteHealthCollector` interface/implementation it calls +- Test: `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardStorageTests.cs` (append) + +Depends on Task 20 (same storage file). Parked rows persist until operator action by design; this gives operators the aging signal the review flagged as absent (a forgotten site's parked backlog now shows its age, not just its count). Central UI tile rendering is a display-only follow-up once the field flows. + +1. Failing test: + +```csharp +[Fact] +public async Task GetOldestParkedCreatedAt_ReturnsOldestParkedRow_OrNull() +{ + Assert.Null(await _storage.GetOldestParkedCreatedAtAsync()); + + var old = NewMessage("old-parked", status: StoreAndForwardMessageStatus.Parked, + createdAt: DateTimeOffset.UtcNow.AddDays(-3)); + var recent = NewMessage("new-parked", status: StoreAndForwardMessageStatus.Parked, + createdAt: DateTimeOffset.UtcNow.AddHours(-1)); + var pending = NewMessage("pending", status: StoreAndForwardMessageStatus.Pending, + createdAt: DateTimeOffset.UtcNow.AddDays(-9)); // must not count + await _storage.EnqueueAsync(old); + await _storage.EnqueueAsync(recent); + await _storage.EnqueueAsync(pending); + + var oldest = await _storage.GetOldestParkedCreatedAtAsync(); + Assert.Equal(old.CreatedAt, oldest!.Value, TimeSpan.FromSeconds(1)); +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests --filter GetOldestParkedCreatedAt` — expect FAIL. +3. Implement: + - Storage: `SELECT MIN(created_at) FROM sf_messages WHERE status = @parked` → parse to `DateTimeOffset?` (null when no rows). + - `SiteHealthReport`: append additive default parameter `double? OldestParkedMessageAgeSeconds = null` with a doc comment (age computed at report time on the site; null = no parked rows; additive-only contract evolution). + - `ISiteHealthCollector`: add `SetOldestParkedMessageAgeSeconds(double? ageSeconds);` implement in the collector (store; include in the built report). + - `HealthReportSender`: inside the existing parked-count `try` block: `var oldest = await _sfStorage.GetOldestParkedCreatedAtAsync(); _collector.SetOldestParkedMessageAgeSeconds(oldest.HasValue ? (DateTimeOffset.UtcNow - oldest.Value).TotalSeconds : null);` +4. Build the solution (`dotnet build ZB.MOM.WW.ScadaBridge.slnx` — the additive default param must not break any constructor caller) and run `dotnet test tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests` + `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests`. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReport.cs src/ZB.MOM.WW.ScadaBridge.HealthMonitoring tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests && git commit -m "feat(health): oldest-parked-message age on the site health report (parked-retention aging signal)"` + +### Task 24: Pin the reconciliation cursor boundary contract (`>=` at the site read) + +**Classification:** standard (test-first; possible one-operator fix) +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 3, 5, 7, 10, 14 +**Files:** +- Test: `tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/SiteStreamPullAuditEventsTests.cs` (append; mirror in `SiteStreamPullSiteCallsTests.cs`) +- Possibly modify: the site-side since-cursor query invoked by `SiteStreamGrpcServer.PullAuditEvents` (`SiteStreamGrpcServer.cs:478-484` — follow the Ask/store call into the site audit store in `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/`; same for `PullSiteCalls` at `:563-572` into the SiteRuntime tracking store) + +Central advances `since_utc` from the last returned row's timestamp; rows *sharing* that boundary timestamp are only safe if the site-side read is **inclusive** (`>=`) — duplicates are absorbed by central's EventId dedup / upsert-on-newer, but an *exclusive* read silently loses same-timestamp rows forever. + +1. Locate the site-side query behind `PullAuditEvents` (grep `since` in the store the gRPC handler Asks) and note the current operator. +2. Write the failing/pinning test in `SiteStreamPullAuditEventsTests` (its harness already stands up the server with a real/fake store — follow it): + +```csharp +[Fact] +public async Task PullAuditEvents_BoundaryTimestamp_IsInclusive_EventIdDedupAbsorbsDuplicates() +{ + // Two rows share one OccurredAtUtc; a pull whose since_utc equals that + // timestamp MUST return both (inclusive read). Central dedups on EventId, + // so duplication is safe; exclusion is silent permanent loss. + var boundary = DateTime.UtcNow; + await SeedAuditRow(eventId: _e1, occurredAtUtc: boundary); + await SeedAuditRow(eventId: _e2, occurredAtUtc: boundary); + + var reply = await PullSince(boundary); + Assert.Equal(2, reply.Events.Count); +} +``` + +3. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Communication.Tests --filter PullAuditEvents_BoundaryTimestamp` — if the current operator is `>=` it PASSES (the pin is the deliverable); if it FAILS, change the site-side query operator from `>` to `>=` (one-character fix + doc comment: *"inclusive: central's cursor is the last returned timestamp; EventId idempotency absorbs the re-sent boundary rows, exclusivity would lose same-timestamp rows"*), rerun, expect PASS. +4. Mirror the same boundary test for `PullSiteCalls` (upsert-on-newer absorbs duplicates there). +5. Run both filters + full project. +6. Commit: `git add -A tests/ZB.MOM.WW.ScadaBridge.Communication.Tests src/ZB.MOM.WW.ScadaBridge.AuditLog src/ZB.MOM.WW.ScadaBridge.SiteRuntime && git commit -m "test(communication): pin inclusive (>=) reconciliation cursor reads at the site; EventId/upsert dedup absorbs boundary duplicates"` + +--- + +## Dependencies on other plans + +- **Plan 01 (Cluster/Host/Failover)** — owns the general active-node/singleton-host facility and the leader-vs-oldest correction. Task 3's gate is a `Func` seam and Task 4/Task 21 wire the interim repo-standard leader+Up check with an explicit swap-point comment; when plan 01's shared helper lands, replace those two lambda bodies (and `SiteCommunicationActor.DefaultIsActiveCheck`, which this plan defers to plan 01). Plan 01 also owns the **real two-node failover test rig**, which is the integration-level proof for Tasks 1-4 (kill -9 a node → assert single-node S&F delivery and ClusterClient survival); this plan's unit tests are the fast-feedback layer. +- **Plan 04 (Data/Audit backbone)** — owns central-side ingest idempotency and data-layer indexes; Task 24 only pins the *site-side* read contract and relies on plan-04-owned central dedup staying as-is. +- No other plan touches `src/ZB.MOM.WW.ScadaBridge.StoreAndForward` or `src/ZB.MOM.WW.ScadaBridge.Communication`; merge conflicts are expected only in `AkkaHostedService.cs` (plan 01 edits `BuildHocon`; Task 4 edits `RegisterSiteActorsAsync` — different regions). + +## Execution order + +**P0 (fix before the next production deployment):** Tasks 1 → 2 (Critical crash-loop), 3 → 4 (standby gate), 5 → 6 (shared-channel disposal), 7 → 8 (sweep collapse). These four pairs are mutually independent — run them as four parallel tracks. + +**Critical path (file-serialized chains):** +- `StoreAndForwardService.cs`: 3 → 7 → 8 → 9 → 12 → 13 → 19 +- `StoreAndForwardStorage.cs`: 7 → 11 → 17 → 18 → 20 → 23 +- `ReplicationService.cs` / `SiteReplicationActor.cs`: 10 → 11, 10+20 → 21 +- `CentralCommunicationActor.cs`: 1 → 2 → 16 → 22 + +**P1:** 10, 11, 12, 13, 14, 15, 16. **P2 (hardening/underdeveloped):** 9, 17, 18, 19, 20, 21, 22, 23, 24. + +After each task: `dotnet build ZB.MOM.WW.ScadaBridge.slnx` + the named per-project test command. At plan end: full `dotnet test ZB.MOM.WW.ScadaBridge.slnx` **plus** `dotnet test tests/ZB.MOM.WW.ScadaBridge.CLI.Tests` (not in the slnx), and rebuild the docker cluster (`bash docker/deploy.sh`) since Host/actor runtime changed. diff --git a/archreview/plans/PLAN-02-communication-store-and-forward.md.tasks.json b/archreview/plans/PLAN-02-communication-store-and-forward.md.tasks.json new file mode 100644 index 00000000..3b46c719 --- /dev/null +++ b/archreview/plans/PLAN-02-communication-store-and-forward.md.tasks.json @@ -0,0 +1,30 @@ +{ + "planPath": "archreview/plans/PLAN-02-communication-store-and-forward.md", + "tasks": [ + { "id": 1, "subject": "Task 1: Generation-suffixed, sanitized ClusterClient actor names", "status": "pending", "blockedBy": [] }, + { "id": 2, "subject": "Task 2: Guard client creation in the refresh loop + real-factory address-edit lifecycle test", "status": "pending", "blockedBy": [1] }, + { "id": 3, "subject": "Task 3: Active-node delivery gate on the S&F retry sweep", "status": "pending", "blockedBy": [] }, + { "id": 4, "subject": "Task 4: Wire the delivery gate in the Host + align the S&F design doc", "status": "pending", "blockedBy": [3] }, + { "id": 5, "subject": "Task 5: Key the gRPC client cache by (site, endpoint)", "status": "pending", "blockedBy": [] }, + { "id": 6, "subject": "Task 6: Endpoint-safe unsubscribe in DebugStreamBridgeActor + concurrent-session regression test", "status": "pending", "blockedBy": [5] }, + { "id": 7, "subject": "Task 7: Bound the retry sweep with a batch LIMIT", "status": "pending", "blockedBy": [3] }, + { "id": 8, "subject": "Task 8: Per-target short-circuit in the retry sweep", "status": "pending", "blockedBy": [7] }, + { "id": 9, "subject": "Task 9: Parallel per-target lanes with a small concurrency cap", "status": "pending", "blockedBy": [8] }, + { "id": 10, "subject": "Task 10: Ordered, observable replication dispatch", "status": "pending", "blockedBy": [] }, + { "id": 11, "subject": "Task 11: Upsert-based standby applies for Add/Park/Requeue", "status": "pending", "blockedBy": [7, 10] }, + { "id": 12, "subject": "Task 12: deferToSweep enqueue mode + immediate sweep kick", "status": "pending", "blockedBy": [9] }, + { "id": 13, "subject": "Task 13: Unstrand and unblock Notify.Send (maxRetries 0 + deferToSweep) + docs", "status": "pending", "blockedBy": [12] }, + { "id": 14, "subject": "Task 14: Park (don't silently drop) corrupt buffered notifications", "status": "pending", "blockedBy": [] }, + { "id": 15, "subject": "Task 15: Wire-format pin tests for cross-cluster contracts", "status": "pending", "blockedBy": [] }, + { "id": 16, "subject": "Task 16: Replicate heartbeat marks to the peer central node", "status": "pending", "blockedBy": [2] }, + { "id": 17, "subject": "Task 17: S&F SQLite hygiene — WAL + busy_timeout", "status": "pending", "blockedBy": [11] }, + { "id": 18, "subject": "Task 18: Epoch-ms due-check (replace per-row julianday parsing)", "status": "pending", "blockedBy": [17] }, + { "id": 19, "subject": "Task 19: Decouple audit-observer notification from the sweep (ordered channel)", "status": "pending", "blockedBy": [12] }, + { "id": 20, "subject": "Task 20: Buffer resync storage primitives (GetAllMessagesAsync / ReplaceAllAsync)", "status": "pending", "blockedBy": [18] }, + { "id": 21, "subject": "Task 21: Peer-join S&F buffer resync (anti-entropy)", "status": "pending", "blockedBy": [10, 20] }, + { "id": 22, "subject": "Task 22: Conventions & observability cleanup batch (Lows)", "status": "pending", "blockedBy": [16] }, + { "id": 23, "subject": "Task 23: Oldest-parked-age health metric (parked-retention aging signal)", "status": "pending", "blockedBy": [20] }, + { "id": 24, "subject": "Task 24: Pin the reconciliation cursor boundary contract (>= at the site read)", "status": "pending", "blockedBy": [] } + ], + "lastUpdated": "2026-07-08" +} diff --git a/archreview/plans/PLAN-03-site-runtime-dcl.md b/archreview/plans/PLAN-03-site-runtime-dcl.md new file mode 100644 index 00000000..81856b7c --- /dev/null +++ b/archreview/plans/PLAN-03-site-runtime-dcl.md @@ -0,0 +1,1297 @@ +# Site Runtime & DCL Hardening Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. + +**Goal:** Fix every finding in `archreview/03-site-runtime-dcl.md` — adapter reconnect lifecycle leaks, script-containment observability, deployment honesty (site-side compile failure must reject the deployment per spec), subscription/startup retry holes, dead-child bookkeeping, hot-path fan-out cost, and site SQLite hardening — with design docs updated alongside the code. + +**Architecture:** All changes are site-side, inside `ZB.MOM.WW.ScadaBridge.SiteRuntime` and `ZB.MOM.WW.ScadaBridge.DataConnectionLayer` (plus additive message/report contracts in Commons/HealthMonitoring). The actor model is preserved: every fix keeps state mutation on the actor thread (background work returns via `PipeTo`), retries use the existing fixed-interval timer idiom, and new health metrics ride the existing `ISiteHealthCollector` → `SiteHealthReport` path with additive-only contract changes. The deploy compile gate runs off-thread before actor creation so the singleton mailbox never blocks on Roslyn. + +**Tech Stack:** C#/.NET, Akka.NET (ReceiveActor/UntypedActor, TestKit.Xunit2), Roslyn scripting, Microsoft.Data.Sqlite, xunit + NSubstitute. + +Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per-project: `dotnet test tests/ --filter `. + +## Findings Coverage + +| Finding | Severity | Task(s) / Disposition | +|---|---|---| +| S1 — reconnect abandons previous client, stale `ConnectionLost` flaps healthy connection | High | 1 (OPC UA), 2 (MxGateway) | +| S2 — cooperative-only timeout; no stuck-thread observability | High | 5, 6, 7, 8. Sub-suggestion "inject ADO.NET command timeouts into Database.Connection" **deferred**: scripts own their ADO.NET commands (`SqlCommand.CommandTimeout` defaults to 30 s already); the watchdog + gauges make the residual case visible. | +| S3 — site compile failure doesn't reject deployment (spec drift) | High | 3, 4 | +| S4 — no retry on failed tag subscription; NativeAlarm lost-response gap | Medium | 11 (InstanceActor), 12 (NativeAlarmActor) | +| S5 — startup SQLite load failure aborts silently, no retry | Medium | 13 | +| S6 — dead child ref in `_instanceActors` + Success reported for dead instance | Medium | 14, 15 | +| S7 — background tasks read mutable `_adapter` field off-thread | Medium | 16 | +| S8 — site SQLite: no WAL/busy-timeout | Low | 19 | +| S9 — fire-and-forget adapter dispose in `PostStop` unobserved | Low | 25 | +| S10 — alarm condition filter last-subscriber-wins, silent overwrite | Low | 25 | +| P1 — trigger expressions block dispatcher threads per attribute change | High | 9 (ScriptActor), 10 (AlarmActor) | +| P2 — attribute-change fan-out broadcast to every child | Medium | 18 | +| P3 — DCL tag-value fan-out scans all instances per update | Medium | 17 | +| P4 — per-transition SQLite upserts on native-alarm mirror | Low | 20 | +| P5 — per-attribute script read is an Ask round-trip | Low | **Deferred** — correct and spec-consistent; no profiling evidence yet; batch `GetAttributes` is a follow-up when profiling flags it. | +| P6 — Roslyn compiles inside `InstanceActor.PreStart` during staggered startup | Low | **Deferred** (partially mitigated) — Task 3/4 moves *deploy-path* compiles off the singleton mailbox; moving *startup-path* compiles out of `PreStart` requires an async child-creation phase (messages arriving before children exist) and is out of scope; risk is failover time-to-recover only. Logged as follow-up in Task 26 doc note. | +| C1 — positive (Akka discipline) | — | No action. | +| C2 — stale "unbounded channel" comment in SiteEventLogger | Low | 25 | +| C3 — spec self-contradiction on reconnect interval scope | Low | 26 | +| C4 — quality-only change not delivered to children vs. spec wording | Low | 26 | +| C5 — shared scripts run Root scope, undocumented | Low | 26 | +| C6 — `EvaluateCondition` fallback uses current-culture `ToString` | Low | 25 | +| UA1 — cert-trust convergence after node downtime | Underdev. | 22, 23 (+ doc in 26). List-aggregation across both nodes and removal-reconcile **deferred** to the documented central-persistence follow-up (`Component-SiteRuntime.md:125`). | +| UA2 — `ConfigFetchRetryCount` dead option | Underdev. | 24 | +| UA3 — no aggregated live alarm stream | Underdev. | **Deferred** — documented M7 follow-up, central-side; out of this plan's site-runtime scope. | +| UA4 — native-alarm rehydration loses display metadata | Underdev. | 21 | +| UA5 — no scheduler stuck-thread/queue-depth observability | Underdev. | 5, 6, 7, 8 (same fix as S2) | +| UA6 — tag-subscription failure path has no retry/site event | Underdev. | 11, 12 (same fix as S4) | +| UA7 — test gaps (reconnect-stale-handler, compile-fail deploy status, subscribe race) | Underdev. | Covered by the TDD tests in Tasks 1, 2, 4, 11 | + +--- + +### Task 1: OpcUaDataConnection — dispose + detach the previous client on reconnect + +**Classification:** high-risk (adapter lifecycle, concurrency with SDK callback threads) +**Estimated implement time:** ~5 min +**Parallelizable with:** 2, 3, 5, 9, 10, 12, 13, 19, 22, 24 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/OpcUaDataConnection.cs` (ConnectAsync, ~lines 97–108) +- Test: `tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/OpcUaDataConnectionTests.cs` + +1. Add failing tests (reuse the existing `Substitute.For()` / `IOpcUaClientFactory` fixture; note the fixture returns one `_mockClient` — for these tests configure the factory to return a **fresh substitute per `Create()` call**): + +```csharp +[Fact] +public async Task Reconnect_DisposesAndDetachesPreviousClient() +{ + var clients = new List(); + _mockFactory.Create().Returns(_ => + { + var c = Substitute.For(); + clients.Add(c); + return c; + }); + var details = new Dictionary { ["EndpointUrl"] = "opc.tcp://x:4840" }; + + await _adapter.ConnectAsync(details); // client[0] + await _adapter.ConnectAsync(details); // reconnect -> client[1] + + Assert.Equal(2, clients.Count); + await clients[0].Received(1).DisposeAsync(); + await clients[1].DidNotReceive().DisposeAsync(); +} + +[Fact] +public async Task StaleClientConnectionLost_DoesNotRaiseDisconnected_AfterReconnect() +{ + var clients = new List(); + _mockFactory.Create().Returns(_ => + { + var c = Substitute.For(); + clients.Add(c); + return c; + }); + var details = new Dictionary { ["EndpointUrl"] = "opc.tcp://x:4840" }; + var disconnects = 0; + _adapter.Disconnected += () => Interlocked.Increment(ref disconnects); + + await _adapter.ConnectAsync(details); + await _adapter.ConnectAsync(details); // reconnect resets _disconnectFired + + // The OLD client's keep-alive dies later — must be a no-op now. + clients[0].ConnectionLost += Raise.Event(); + Assert.Equal(0, disconnects); + + // The CURRENT client's loss must still signal. + clients[1].ConnectionLost += Raise.Event(); + Assert.Equal(1, disconnects); +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~Reconnect_DisposesAndDetachesPreviousClient|FullyQualifiedName~StaleClientConnectionLost_DoesNotRaiseDisconnected"` → expect **FAIL** (old client never disposed; stale handler still wired). +3. Implement in `ConnectAsync`, immediately before `_client = _clientFactory.Create();`: + +```csharp +// Reconnect reuses this adapter instance: detach + dispose the previous +// client so (a) its session/keep-alive/socket do not leak per flap and +// (b) its late ConnectionLost cannot flap the NEW healthy session after +// _disconnectFired is reset below. Dispose is best-effort fire-and-forget +// (CloseAsync against a dead server can block for the operation timeout); +// faults are logged, never thrown into the connect path. +if (_client is { } previousClient) +{ + previousClient.ConnectionLost -= OnClientConnectionLost; + _ = previousClient.DisposeAsync().AsTask().ContinueWith( + t => _logger.LogWarning(t.Exception?.GetBaseException(), + "Disposing previous OPC UA client failed for {Endpoint}", _endpointUrl), + TaskContinuationOptions.OnlyOnFaulted); +} +``` + + Also stop the previous heartbeat monitor if active (call the existing `StopHeartbeatMonitor()` before creating the new client) so the old `StaleTagMonitor` cannot fire against the new session. +4. Run the two tests → expect **PASS**. Run the full adapter suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~OpcUaDataConnectionTests"` → all green. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/OpcUaDataConnection.cs tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/OpcUaDataConnectionTests.cs && git commit -m "fix(dcl): dispose+detach previous OPC UA client on reconnect (S1) — stops session leak and stale ConnectionLost flapping"` + +### Task 2: MxGatewayDataConnection — dispose previous client + cancel previous event loop on reconnect + +**Classification:** high-risk (adapter lifecycle) +**Estimated implement time:** ~4 min +**Parallelizable with:** 1, 3, 5, 9, 10, 12, 13, 19, 22, 24 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MxGatewayDataConnection.cs` (ConnectAsync, ~lines 67–88) +- Test: create `tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/MxGatewayDataConnectionReconnectTests.cs` + +1. Failing test (mirror the OPC UA fixture shape with `Substitute.For()` / `IMxGatewayClientFactory`; capture the `CancellationToken` handed to `RunEventLoopAsync`): + +```csharp +[Fact] +public async Task Reconnect_DisposesPreviousClient_AndCancelsPreviousEventLoop() +{ + var clients = new List(); + var loopTokens = new List(); + var factory = Substitute.For(); + factory.Create().Returns(_ => + { + var c = Substitute.For(); + c.RunEventLoopAsync(Arg.Any>(), Arg.Do(t => loopTokens.Add(t))) + .Returns(Task.Delay(Timeout.Infinite)); + clients.Add(c); + return c; + }); + var adapter = new MxGatewayDataConnection(factory, NullLogger.Instance); + var details = new Dictionary { ["Endpoint"] = "localhost:5000", ["ApiKey"] = "k" }; + + await adapter.ConnectAsync(details); + await adapter.ConnectAsync(details); // reconnect + + await clients[0].Received(1).DisposeAsync(); + Assert.True(loopTokens[0].IsCancellationRequested); // old loop cancelled + Assert.False(loopTokens[1].IsCancellationRequested); // new loop live +} +``` + + (Adjust the `RunEventLoopAsync` signature/`MxGatewayValueUpdate` names to the real `IMxGatewayClient` members — check `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/IMxGatewayClient.cs`.) +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~MxGatewayDataConnectionReconnectTests"` → **FAIL**. +3. Implement at the top of `ConnectAsync` (before `_client = _clientFactory.Create();`): + +```csharp +// Reconnect reuses this adapter: cancel the previous event loop and dispose +// the previous client so a flapping gateway cannot accumulate orphaned +// streams/sockets, and the old loop's fault path cannot signal Disconnected +// against the new session after the guard resets below. +if (_eventLoopCts is { } previousLoopCts) +{ + previousLoopCts.Cancel(); + previousLoopCts.Dispose(); + _eventLoopCts = null; +} +if (_client is { } previousClient) +{ + _ = previousClient.DisposeAsync().AsTask().ContinueWith( + t => _logger.LogWarning(t.Exception?.GetBaseException(), + "Disposing previous MxGateway client failed"), + TaskContinuationOptions.OnlyOnFaulted); +} +``` + + Note ordering: keep `Interlocked.Exchange(ref _disconnectFired, 0)` AFTER the cancel/dispose, so a fault raised by the dying loop during teardown is absorbed by the *old* guard cycle. Also clear `_subs`/`_tagToSub` is NOT needed (re-subscribe repopulates via the actor), but verify `RunEventLoopAsync`'s catch respects the cancelled token → `OperationCanceledException` path, which it already does. +4. Run → **PASS**; then full project: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests` → green. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MxGatewayDataConnection.cs tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/MxGatewayDataConnectionReconnectTests.cs && git commit -m "fix(dcl): cancel old event loop + dispose old MxGateway client on reconnect (S1)"` + +### Task 3: Pure deploy-time compile validator for a flattened config + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 5, 9, 10, 12, 19, 22, 24 +**Files:** +- Create: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Deployment/DeployCompileValidator.cs` +- Test: create `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Deployment/DeployCompileValidatorTests.cs` + +1. Failing test: + +```csharp +public class DeployCompileValidatorTests +{ + private static string ConfigWith(string scriptCode) => JsonSerializer.Serialize(new FlattenedConfiguration + { + Scripts = new List + { + new() { CanonicalName = "S1", Code = scriptCode, TriggerType = "Call" } + }, + Attributes = new List(), + Alarms = new List(), + NativeAlarmSources = new List() + }); + + [Fact] + public void BrokenScript_ReturnsErrors() + { + var validator = new DeployCompileValidator(new ScriptCompilationService(/* mirror ctor args used in ScriptCompilationServiceTests */)); + var errors = validator.Validate(ConfigWith("this is not C# ~~~")); + Assert.NotEmpty(errors); + Assert.Contains(errors, e => e.Contains("S1")); + } + + [Fact] + public void ValidScript_ReturnsNoErrors() + { + var validator = new DeployCompileValidator(new ScriptCompilationService(/* as above */)); + Assert.Empty(validator.Validate(ConfigWith("return 1 + 1;"))); + } + + [Fact] + public void BrokenTriggerExpression_ReturnsErrors() + { + var cfg = /* config with one Expression-triggered script whose TriggerConfiguration + JSON carries expression: "1 +" (mirror shape used in ScriptActorTests) */; + var validator = new DeployCompileValidator(new ScriptCompilationService(/* as above */)); + Assert.NotEmpty(validator.Validate(cfg)); + } +} +``` + + (Mirror `FlattenedConfiguration`/`ResolvedScript` construction and the `ScriptCompilationService` ctor from the existing `ScriptCompilationServiceTests` / `InstanceActorTests` fixtures.) +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~DeployCompileValidatorTests"` → **FAIL** (type does not exist). +3. Implement `DeployCompileValidator`: a small class taking the same compilation-service type `InstanceActor` uses; method `IReadOnlyList Validate(string configJson)`: + - Deserialize `FlattenedConfiguration`; malformed JSON → single error `"Configuration JSON failed to deserialize: …"` (this also fronts Task 15's poison-config case). + - For each `config.Scripts`: `_compilationService.Compile(script.CanonicalName, script.Code)`; on failure add `$"Script '{script.CanonicalName}': {string.Join("; ", result.Errors)}"`. + - For each alarm with `OnTriggerScriptCanonicalName`: resolve the script def (missing def → error) and compile it (mirroring `InstanceActor.CreateChildActors` lines 1404–1424). + - For each Expression-trigger script/alarm: extract via `TriggerExpressionGlobals.ExtractExpression` and `CompileTriggerExpression`; failures are errors (mirroring `InstanceActor.CompileTriggerExpression`). + - Pure and thread-safe (no actor state) — callable from `Task.Run`. +4. Run → **PASS**. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Deployment/DeployCompileValidator.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Deployment/DeployCompileValidatorTests.cs && git commit -m "feat(site-runtime): pure deploy-time compile validator for flattened configs (S3 groundwork)"` + +### Task 4: Reject the deployment on site-side compile failure (per spec) + +**Classification:** high-risk (deployment pipeline, singleton actor) +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 12, 19, 22, 24 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs` (HandleDeploy entry + new piped message) +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs` (fix the contradictory XML doc at line 1339) +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs` + +1. Failing test (reuse the existing DeploymentManagerActor TestKit harness/fixture in that file): + +```csharp +[Fact] +public void Deploy_WithNonCompilingScript_ReportsFailed_AndCreatesNoInstanceActor() +{ + var badConfig = /* FlattenedConfiguration JSON with one script whose Code = "not C# ~~~" — + reuse the config-builder helper the existing deploy tests use */; + var dm = CreateDeploymentManager(); // existing harness factory + + dm.Tell(new DeployInstanceCommand(/* deploymentId */ Guid.NewGuid().ToString(), + "bad-instance", badConfig, /* revisionHash */ "r1" /*, …existing args*/)); + + var resp = ExpectMsg(TimeSpan.FromSeconds(10)); + Assert.Equal(DeploymentStatus.Failed, resp.Status); + Assert.Contains("compile", resp.ErrorMessage, StringComparison.OrdinalIgnoreCase); + + // No partial state applied: instance actor must not exist, config must not persist. + Sys.ActorSelection(dm.Path / "bad-instance").Tell(new Identify(1)); + Assert.Null(ExpectMsg().Subject); + _storage.DidNotReceive().StoreDeployedConfigAsync("bad-instance", + Arg.Any(), Arg.Any(), Arg.Any(), Arg.Any()); +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~Deploy_WithNonCompilingScript"` → **FAIL** (today it reports Success). +3. Implement: + - Add internal message `private sealed record DeployValidationResult(DeployInstanceCommand Cmd, IActorRef ReplyTo, IReadOnlyList Errors);` and `Receive(HandleDeployValidationResult);`. + - Rename the current `HandleDeploy(DeployInstanceCommand)` body to `ProceedWithDeploy(DeployInstanceCommand command, IActorRef sender)` (this is the code that decides redeploy-buffer vs `ApplyDeployment` — keep it byte-identical). + - New `HandleDeploy`: capture `Sender`; run `Task.Run(() => _deployCompileValidator.Validate(command.FlattenedConfigurationJson))` → `ContinueWith` into `DeployValidationResult` → `PipeTo(Self)`. Validation is pure, so the singleton mailbox never blocks on Roslyn, and results are applied on the actor thread in arrival order (the existing redeploy-supersede handling still governs ordering). + - `HandleDeployValidationResult`: if `Errors.Count > 0` → `LogDeploymentEvent("Error", …, "compile validation failed")`, reply `DeploymentStatusResponse(DeploymentId, Instance, DeploymentStatus.Failed, "Script compilation failed on site: " + string.Join(" | ", errors), UtcNow)` and **return** (no actor creation, no persistence — spec: "No partial state is applied"). Else call `ProceedWithDeploy(msg.Cmd, msg.ReplyTo)`. + - Construct `_deployCompileValidator = new DeployCompileValidator(_compilationService)` in the ctor. + - Fix `InstanceActor.cs:1339` XML doc: replace "Compilation errors reject entire instance deployment (logged but actor still starts)" with "Compile failures are rejected at deploy time by the Deployment Manager's validation gate; a failure reaching this point (startup load of a legacy config) is logged and the script skipped." +4. Run the new test → **PASS**; then the deploy suites: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~DeploymentManager"` → green (existing deploy/redeploy/disable-supersede tests must still pass — they exercise `ProceedWithDeploy` through the new gate with compiling configs). +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs && git commit -m "fix(site-runtime): site-side compile failure now rejects the deployment per spec (S3) — off-thread validation gate before actor creation"` + +### Task 5: ScriptExecutionScheduler gauges — queue depth, busy threads, oldest-busy age + +**Classification:** small (concurrency-adjacent but additive instrumentation) +**Estimated implement time:** ~4 min +**Parallelizable with:** 1, 2, 3, 9, 10, 11, 12, 13, 19, 22, 24 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptExecutionScheduler.cs` +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/ScriptExecutionSchedulerTests.cs` + +1. Failing test: + +```csharp +[Fact] +public async Task Gauges_ReflectBusyThreadAndQueueDepth() +{ + using var scheduler = new ScriptExecutionScheduler(1); + using var gate = new ManualResetEventSlim(false); + + var blocking = Task.Factory.StartNew(() => gate.Wait(), + CancellationToken.None, TaskCreationOptions.None, scheduler); + var queued = Task.Factory.StartNew(() => { }, + CancellationToken.None, TaskCreationOptions.None, scheduler); + + await WaitUntilAsync(() => scheduler.BusyThreadCount == 1); + Assert.Equal(1, scheduler.QueueDepth); // second task waiting + Assert.True(scheduler.OldestBusyAge > TimeSpan.Zero); + + gate.Set(); + await Task.WhenAll(blocking, queued); + await WaitUntilAsync(() => scheduler.BusyThreadCount == 0); + Assert.Null(scheduler.OldestBusyAge); +} +// WaitUntilAsync: small poll helper (100×50ms) — add locally if the test file has none. +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~Gauges_ReflectBusyThreadAndQueueDepth"` → **FAIL** (members don't exist). +3. Implement: + - `private readonly long[] _busySinceTicks;` sized `threadCount`, initialized 0. Give each worker its index (change `new Thread(WorkerLoop)` to `new Thread(() => WorkerLoop(index))` with a captured local). + - In `WorkerLoop(int index)`: around `TryExecuteTask(task)` do `Volatile.Write(ref _busySinceTicks[index], Environment.TickCount64);` before and `Volatile.Write(ref _busySinceTicks[index], 0);` in a `finally`. + - Public gauges: `public int QueueDepth => _queue.Count;` `public int BusyThreadCount` (count of non-zero slots), `public TimeSpan? OldestBusyAge` (max of `now - since` over non-zero slots, else null). +4. Run → **PASS**. Run full scheduler suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ScriptExecutionScheduler"`. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptExecutionScheduler.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/ScriptExecutionSchedulerTests.cs && git commit -m "feat(site-runtime): script scheduler gauges — queue depth, busy threads, oldest-busy age (S2/UA5)"` + +### Task 6: Surface scheduler stats through ISiteHealthCollector + SiteHealthReport (additive) + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** 1, 2, 3, 4, 9, 10, 11, 12, 13, 19, 22, 24 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ISiteHealthCollector.cs` (new default-no-op method) +- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthCollector.cs` (store + include in `CollectReport`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReport.cs` (additive optional properties) +- Test: `tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/` (add to the existing SiteHealthCollector test file) + +1. Failing test (in the existing collector test class): + +```csharp +[Fact] +public void CollectReport_CarriesScriptSchedulerStats() +{ + var collector = CreateCollector(); // existing fixture helper + collector.SetScriptSchedulerStats(queueDepth: 7, busyThreads: 3, oldestBusyAgeSeconds: 42.5); + var report = collector.CollectReport("site-1"); + Assert.Equal(7, report.ScriptQueueDepth); + Assert.Equal(3, report.ScriptBusyThreads); + Assert.Equal(42.5, report.ScriptOldestBusyAgeSeconds); +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests --filter "FullyQualifiedName~CollectReport_CarriesScriptSchedulerStats"` → **FAIL**. +3. Implement: + - Interface: `void SetScriptSchedulerStats(int queueDepth, int busyThreads, double? oldestBusyAgeSeconds) { }` (default no-op, matching the `SetSiteEventLogWriteFailures` pattern so test fakes keep compiling). + - Collector: store via `Interlocked`/volatile fields; include in `CollectReport`. + - `SiteHealthReport`: **additive** optional init properties with safe defaults (`public int ScriptQueueDepth { get; init; }`, `public int ScriptBusyThreads { get; init; }`, `public double? ScriptOldestBusyAgeSeconds { get; init; }`) — do NOT change the positional parameter list (message-contract additive-only rule). +4. Run → **PASS**; run full projects: `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests` and `dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests`. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ISiteHealthCollector.cs src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthCollector.cs src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReport.cs tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests && git commit -m "feat(health): additive script-scheduler stats on ISiteHealthCollector + SiteHealthReport (S2/UA5)"` + +### Task 7: Periodic scheduler-stats reporter (hosted service) + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** 1, 2, 4, 9, 10, 11, 12, 13, 19, 22, 24 (needs 5 + 6 merged) +**Files:** +- Create: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptSchedulerStatsReporter.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs` (register hosted service on site hosts) +- Test: create `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/ScriptSchedulerStatsReporterTests.cs` + +1. Failing test: + +```csharp +[Fact] +public async Task Reporter_PushesStatsToCollector() +{ + var collector = Substitute.For(); + var options = new SiteRuntimeOptions { ScriptExecutionThreadCount = 1 }; + using var reporter = new ScriptSchedulerStatsReporter( + collector, options, NullLogger.Instance, + pollInterval: TimeSpan.FromMilliseconds(50)); + + await reporter.StartAsync(CancellationToken.None); + await Task.Delay(300); + await reporter.StopAsync(CancellationToken.None); + + collector.ReceivedWithAnyArgs().SetScriptSchedulerStats(default, default, default); +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ScriptSchedulerStatsReporterTests"` → **FAIL**. +3. Implement a `BackgroundService` mirroring `SiteEventLogFailureCountReporter` (`src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteEventLogFailureCountReporter.cs`): every 10 s (test-overridable ctor param) read `ScriptExecutionScheduler.Shared(options)` gauges and call `collector.SetScriptSchedulerStats(...)`. Register it in the SiteRuntime `ServiceCollectionExtensions` next to the existing site-role hosted services. Never throw from the loop (catch + log, matching the sibling reporter). +4. Run → **PASS**. Build the solution once (`dotnet build ZB.MOM.WW.ScadaBridge.slnx`) to confirm Host wiring compiles. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptSchedulerStatsReporter.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/ScriptSchedulerStatsReporterTests.cs && git commit -m "feat(site-runtime): periodic scheduler-stats reporter feeding site health (S2/UA5)"` + +### Task 8: Stuck-script watchdog — name the script whose thread never came back + +**Classification:** high-risk (script execution path) +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 4, 9, 10, 12, 13, 19, 22, 24 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptExecutionActor.cs` (~lines 122–260) +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs` (new `StuckScriptGraceMs`, default 30000) +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ExecutionActorTests.cs` + +1. Failing test (follow the existing `ExecutionActorTests` pattern for compiling a `Script` directly with `CSharpScript.Create(code, options, typeof(ScriptGlobals))` — test-compiled, so the trust validator is not in the path): + +```csharp +[Fact] +public async Task TimedOutScript_WithBlockedThread_EmitsStuckThreadSiteEvent() +{ + var gate = new SemaphoreSlim(0); + StuckTestHooks.Gate = gate; // small public static test hook class in the test assembly + var script = CompileRaw("ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors.StuckTestHooks.Gate.Wait(); return null;"); + var siteEvents = new RecordingSiteEventLogger(); // reuse/extend the existing fake in TestSupport + var options = new SiteRuntimeOptions { ScriptExecutionTimeoutSeconds = 1, StuckScriptGraceMs = 200 }; + + SpawnExecutionActor(script, options, siteEvents); // existing harness helper + + await WaitUntilAsync(() => siteEvents.Events.Any(e => + e.Category == "script" && e.Message.Contains("still executing", StringComparison.OrdinalIgnoreCase)), + timeout: TimeSpan.FromSeconds(10)); + + gate.Release(); // free the scheduler thread so the test run stays clean +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~TimedOutScript_WithBlockedThread_EmitsStuckThreadSiteEvent"` → **FAIL**. +3. Implement inside the execution lambda in `ScriptExecutionActor`: + - `var completed = 0;` before the `try`; `Interlocked.Exchange(ref completed, 1);` in the existing `finally` (add one if missing — the scope-dispose finally exists). + - Immediately after `using var cts = new CancellationTokenSource(timeout);`: + +```csharp +// Cooperative-only timeout containment (S2): the CTS firing does NOT free a +// thread blocked in synchronous I/O. When the timeout elapses, wait a grace +// period on the thread pool and, if the body still hasn't returned, name the +// script loudly — this is the only signal an operator gets that one of the +// bounded script-execution threads is gone. +var graceMs = options.StuckScriptGraceMs; +cts.Token.Register(() => _ = Task.Run(async () => +{ + await Task.Delay(graceMs); + if (Volatile.Read(ref completed) == 0) + { + var msg = $"Script '{scriptName}' on instance '{instanceName}' exceeded its " + + $"{timeout.TotalSeconds:F0}s timeout and is STILL EXECUTING — its dedicated " + + "script-execution thread is blocked (cooperative cancellation not observed)."; + logger.LogError(msg); + _ = siteEventLogger?.LogEventAsync("script", "Error", instanceName, + $"ScriptActor:{scriptName}", msg); + } +})); +``` + + - Note: `Register` callback fires on cancellation only; normal completion disposes the CTS with no callback. `Task.Run`/`Task.Delay` run on the thread pool, not the (possibly saturated) script scheduler — deliberate. +4. Run → **PASS**; run `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ExecutionActorTests"`. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptExecutionActor.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ExecutionActorTests.cs && git commit -m "feat(site-runtime): stuck-script watchdog names the script holding a scheduler thread past timeout (S2)"` + +### Task 9: ScriptActor — trigger-expression evaluation off the dispatcher (coalesced, actor-confined edge state) + +**Classification:** high-risk (actor concurrency, trigger semantics) +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 3, 5, 10, 12, 13, 19, 22, 24 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs` (~lines 255–307) +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ScriptActorTests.cs` + +1. Failing test (expression that only returns true on a scheduler thread — proves evaluation left the dispatcher; reuse the ScriptActorTests harness that builds an Expression-triggered ScriptActor with a compiled trigger expression and a script body observable from the test): + +```csharp +[Fact] +public void ExpressionTrigger_EvaluatesOnScriptSchedulerThread_AndStillFires() +{ + // Expression is TRUE only when evaluated on a script-execution thread. + var expr = CompileTriggerExpression( + "System.Threading.Thread.CurrentThread.Name != null && " + + "System.Threading.Thread.CurrentThread.Name.StartsWith(\"script-execution-\")"); + var fired = CreateExpressionScriptActorWithObservableBody(expr); // harness helper returning a signal + + _scriptActor.Tell(new AttributeValueChanged("inst", "A", "A", 1, "Good", DateTimeOffset.UtcNow)); + + AwaitAssert(() => Assert.True(fired.IsSet), TimeSpan.FromSeconds(10)); +} + +[Fact] +public void ExpressionTrigger_OnTrue_StillFiresOncePerFalseTrueEdge() +{ + // Regression: async evaluation must preserve edge semantics. + // expr: (int)Attributes["A"] > 5, OnTrue mode. + SendChange("A", 1); SendChange("A", 10); SendChange("A", 11); // true stays true + AwaitAssert(() => Assert.Equal(1, ExecutionCount()), TimeSpan.FromSeconds(10)); + SendChange("A", 1); SendChange("A", 9); + AwaitAssert(() => Assert.Equal(2, ExecutionCount()), TimeSpan.FromSeconds(10)); +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ExpressionTrigger_EvaluatesOnScriptSchedulerThread"` → **FAIL** (evaluation currently happens synchronously on the dispatcher thread, whose name is not `script-execution-*`). +3. Implement: + - New internal message: `private sealed record ExpressionEvalResult(bool Result);` + `Receive(HandleExpressionEvalResult);` in the ctor. + - Fields: `private bool _evalInFlight; private bool _evalPending;` + - Replace the body of `EvaluateExpressionTrigger()` with `StartExpressionEvaluation()`: + +```csharp +private void StartExpressionEvaluation() +{ + if (_compiledTriggerExpression == null || _triggerConfig is not ExpressionTriggerConfig) return; + if (_evalInFlight) { _evalPending = true; return; } // coalesce bursts: one in flight, one pending + _evalInFlight = true; + + var snapshot = new Dictionary(_attributeSnapshot); // point-in-time copy, actor thread + var expression = _compiledTriggerExpression; + var self = Self; + Task.Factory.StartNew(async () => + { + try + { + using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(2)); + var state = await expression.RunAsync(new TriggerExpressionGlobals(snapshot), cancellationToken: cts.Token); + return state.ReturnValue is bool b && b; + } + catch (Exception ex) { LogExpressionError(ex); return false; } + }, CancellationToken.None, TaskCreationOptions.DenyChildAttach, + ScriptExecutionScheduler.Shared(_options)).Unwrap().PipeTo(self, + success: r => new ExpressionEvalResult(r)); +} + +private void HandleExpressionEvalResult(ExpressionEvalResult msg) +{ + _evalInFlight = false; + if (_triggerConfig is ExpressionTriggerConfig exprConfig) + { + if (exprConfig.Mode == TriggerMode.WhileTrue) HandleWhileTrueTransition(msg.Result, _lastExpressionResult); + else if (msg.Result && !_lastExpressionResult) TrySpawnExecution(null); + _lastExpressionResult = msg.Result; + } + if (_evalPending) { _evalPending = false; StartExpressionEvaluation(); } +} +``` + + - Edge state (`_lastExpressionResult`, timers) is touched only on the actor thread — the concurrency contract holds. `LogExpressionError` is called from the background task: it only touches `_healthCollector`/`_logger`/`_serviceProvider` (thread-safe singletons) — verify, and if in doubt route the error through the result message instead. +4. Run both new tests + `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ScriptActorTests"` → **PASS** (existing expression tests must survive; they may need `AwaitAssert` in place of synchronous asserts since evaluation is now async). +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ScriptActorTests.cs && git commit -m "perf(site-runtime): ScriptActor trigger expressions evaluate on the script scheduler, coalesced, edge state actor-confined (P1)"` + +### Task 10: AlarmActor — same async trigger-expression evaluation + +**Classification:** high-risk (actor concurrency, alarm state machine) +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 3, 5, 9, 12, 13, 19, 22, 24 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/AlarmActor.cs` (~lines 210–240, 491–519) +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/AlarmActorTests.cs` + +1. Failing test (mirror Task 9's thread-name expression trick with an Expression-triggered alarm; assert the alarm transitions to Active via the existing `AlarmStateChanged` parent-probe pattern in AlarmActorTests): + +```csharp +[Fact] +public void ExpressionAlarm_EvaluatesOnSchedulerThread_AndActivates() +{ + var expr = CompileTriggerExpression( + "System.Threading.Thread.CurrentThread.Name != null && " + + "System.Threading.Thread.CurrentThread.Name.StartsWith(\"script-execution-\")"); + var alarm = CreateExpressionAlarmActor(expr, parentProbe); // existing harness + + alarm.Tell(new AttributeValueChanged("inst", "A", "A", 1, "Good", DateTimeOffset.UtcNow)); + + var change = parentProbe.ExpectMsg(TimeSpan.FromSeconds(10)); + Assert.Equal(AlarmState.Active, change.State); +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ExpressionAlarm_EvaluatesOnSchedulerThread"` → **FAIL**. +3. Implement, structurally identical to Task 9: + - Extract the existing Normal↔Active transition block in `HandleAttributeValueChanged` (the `isTriggered && _currentState == AlarmState.Normal` / clear branches) into `private void ApplyTriggeredState(bool isTriggered)`. + - In `HandleAttributeValueChanged`, the `AlarmTriggerType.Expression` arm no longer computes `EvaluateExpression()` inline — it calls `StartExpressionEvaluation()` (in-flight + pending coalescing, snapshot copy, run on `ScriptExecutionScheduler.Shared(_options)`, `PipeTo` self as `ExpressionEvalResult`). `Receive` → `ApplyTriggeredState(msg.Result)` + drain pending. + - Non-expression trigger types keep their synchronous path untouched (they are cheap numeric comparisons). + - Delete the now-unused synchronous `EvaluateExpression()` (or keep as the body invoked inside the task). +4. Run new test + `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~AlarmActorTests"` → **PASS** (existing expression-alarm tests may need `AwaitAssert`). +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/AlarmActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/AlarmActorTests.cs && git commit -m "perf(site-runtime): AlarmActor trigger expressions evaluate on the script scheduler (P1)"` + +### Task 11: InstanceActor — retry failed/lost tag subscriptions per connection + +**Classification:** high-risk (actor timers, DCL handshake) +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 4, 9, 10, 12, 13, 19, 22, 24 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs` (line 207 no-op handler; `SubscribeToDcl` ~962–1010) +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs` (new `TagSubscribeRetryIntervalMs`, default 5000) +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorTests.cs` + +1. Failing tests (TestKit; the InstanceActor harness passes a `TestProbe` as `dclManager`): + +```csharp +[Fact] +public void FailedSubscribeResponse_IsRetried() +{ + var options = new SiteRuntimeOptions { TagSubscribeRetryIntervalMs = 200 }; + var (instance, dclProbe) = CreateInstanceWithDataSourcedAttribute(options); // existing harness + + var first = dclProbe.ExpectMsg(); + instance.Tell(new SubscribeTagsResponse(first.CorrelationId, first.InstanceUniqueName, + false, "Unknown connection: conn-1", DateTimeOffset.UtcNow), dclProbe.Ref); + + var retry = dclProbe.ExpectMsg(TimeSpan.FromSeconds(5)); // S4 core + Assert.Equal(first.ConnectionName, retry.ConnectionName); + Assert.Equal(first.TagPaths, retry.TagPaths); +} + +[Fact] +public void LostSubscribeResponse_IsRetried_AndSuccessStopsRetrying() +{ + var options = new SiteRuntimeOptions { TagSubscribeRetryIntervalMs = 200 }; + var (instance, dclProbe) = CreateInstanceWithDataSourcedAttribute(options); + + var first = dclProbe.ExpectMsg(); + // No reply at all (dead-lettered response) -> must re-send. + var retry = dclProbe.ExpectMsg(TimeSpan.FromSeconds(5)); + + instance.Tell(new SubscribeTagsResponse(retry.CorrelationId, retry.InstanceUniqueName, + true, null, DateTimeOffset.UtcNow), dclProbe.Ref); + dclProbe.ExpectNoMsg(TimeSpan.FromMilliseconds(600)); // retry timer cancelled +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~FailedSubscribeResponse_IsRetried|FullyQualifiedName~LostSubscribeResponse_IsRetried"` → **FAIL** (response is a no-op; no re-send). +3. Implement: + - Refactor `SubscribeToDcl` to build the per-connection groups once into a field `Dictionary> _tagsByConnection`, then call `SendTagSubscribe(connectionName)` per connection. + - `SendTagSubscribe(string conn)`: mint a correlation id, record `_pendingTagSubscribes[correlationId] = conn`, Tell the DCL manager, and (re)arm a per-connection retry: `_tagSubscribeRetryTimers[conn]?.Cancel(); _tagSubscribeRetryTimers[conn] = Context.System.Scheduler.ScheduleTellOnceCancelable(TimeSpan.FromMilliseconds(_options.TagSubscribeRetryIntervalMs), Self, new RetryTagSubscribe(conn), Self);` — arming on **send** covers both the failure reply AND the lost reply with one mechanism (mirrors `NativeAlarmActor`'s timer but closes its lost-response gap too). + - Replace `Receive(_ => { })` with a real handler: resolve `conn` from `_pendingTagSubscribes` (remove the entry); on `Success` → cancel the connection's retry timer, `LogDebug`; on failure → `LogWarning` + fire-and-forget an `instance_lifecycle`/`connection` site event ("tag subscribe failed … will retry") and leave the armed timer to re-send. Unknown correlation id (stale retry's superseded reply) → ignore. + - `Receive(msg => SendTagSubscribe(msg.ConnectionName));` (internal record). `PostStop`: cancel all retry timers. + - Note the DCL's connection-level-failure reply (`Success=false, "connection unavailable — will re-subscribe on reconnect"`) now ALSO retriggers a subscribe — harmless: the DCL treats already-subscribed tags as `AlreadySubscribed` and its `_subscriptionsByInstance` update is idempotent (verified `HandleSubscribe`/`HandleSubscribeCompleted`). +4. Run → **PASS**; run `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~InstanceActor"` → green. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorTests.cs && git commit -m "fix(site-runtime): InstanceActor retries failed/lost tag subscriptions per connection (S4/UA6) — closes the unknown-connection ordering race"` + +### Task 12: NativeAlarmActor — retry on lost subscribe response + +**Classification:** small +**Estimated implement time:** ~3 min +**Parallelizable with:** 1, 2, 3, 4, 5, 9, 10, 11, 13, 19, 22, 24 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/NativeAlarmActor.cs` (`SendSubscribe` ~123–132, `HandleSubscribeResponse` ~269–286) +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs` + +1. Failing test: + +```csharp +[Fact] +public void LostSubscribeResponse_ResendsSubscribe() +{ + var options = new SiteRuntimeOptions { NativeAlarmRetryIntervalMs = 200 }; + var (actor, dclProbe) = CreateNativeAlarmActor(options); // existing harness + + dclProbe.ExpectMsg(); + // Swallow the response entirely — today the actor never retries. + dclProbe.ExpectMsg(TimeSpan.FromSeconds(5)); +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~LostSubscribeResponse_ResendsSubscribe"` → **FAIL**. +3. Implement: in `SendSubscribe()`, after the Tell, arm the response-timeout retry using the existing `_retryTimer` field: `_retryTimer?.Cancel(); _retryTimer = Context.System.Scheduler.ScheduleTellOnceCancelable(TimeSpan.FromMilliseconds(_options.NativeAlarmRetryIntervalMs * 2), Self, RetrySubscribe.Instance, Self);`. In `HandleSubscribeResponse` success branch, add `_retryTimer?.Cancel();` before returning. The failure branch already re-arms (keep it — it re-arms at 1× interval, which now simply overrides the 2× response-timeout arm). +4. Run → **PASS**; run `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~NativeAlarmActorTests"`. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/NativeAlarmActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs && git commit -m "fix(site-runtime): NativeAlarmActor retries a lost subscribe response, not only a failed one (S4)"` + +### Task 13: DeploymentManager — retry the startup config load + +**Classification:** standard +**Estimated implement time:** ~4 min +**Parallelizable with:** 1, 2, 3, 5, 9, 10, 11, 12, 19, 22, 24 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs` (PreStart ~240–253, `HandleStartupConfigsLoaded` ~291–297) +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs` + +1. Failing test: + +```csharp +[Fact] +public void StartupLoadFailure_IsRetried_UntilSuccess() +{ + var calls = 0; + _storage.GetAllDeployedConfigsAsync().Returns(_ => + ++calls == 1 + ? Task.FromException>(new IOException("db locked")) + : Task.FromResult(new List + { NewDeployedInstance("inst-1", ValidConfigJson()) })); // harness helpers + + var dm = CreateDeploymentManager(); // retry interval overridable to ~200ms for the test + + AwaitAssert(() => + { + Assert.True(calls >= 2); + Sys.ActorSelection(dm.Path / "inst-1").Tell(new Identify(1)); + Assert.NotNull(ExpectMsg().Subject); + }, TimeSpan.FromSeconds(10)); +} +``` + + (If `_storage` is the concrete `SiteStorageService` in the harness rather than a substitute, follow the harness's existing seam — several DM tests already fake persistence failures for `DeployPersistenceResult`; reuse that approach.) +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~StartupLoadFailure_IsRetried"` → **FAIL** (one attempt, then dead). +3. Implement: + - Extract the PreStart load block into `private void LoadDeployedConfigs()` and call it from `PreStart`. + - In `HandleStartupConfigsLoaded`, replace the error `return;` with: log the error, then `Timers.StartSingleTimer("startup-load-retry", RetryStartupLoad.Instance, StartupLoadRetryInterval);` (private static readonly `TimeSpan StartupLoadRetryInterval = TimeSpan.FromSeconds(5)` — internal-settable or ctor-overridable for the test, matching how other DM test knobs are exposed). Fixed-interval, unlimited — consistent with the system's fixed-interval retry philosophy; each failure logs. + - `Receive(_ => LoadDeployedConfigs());` (internal singleton message). Guard: if configs already loaded (`_deployedInstanceNames.Count > 0` after a successful load — add a `_startupLoadCompleted` bool set in the success path), ignore stale retries. +4. Run → **PASS**; run the DM suite. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs && git commit -m "fix(site-runtime): retry the startup deployed-config load on SQLite failure (S5) — no more silent zero-instance node"` + +### Task 14: Watch all Instance Actors; clean the map on unexpected termination + +**Classification:** high-risk (supervision/termination flow interacts with redeploy buffering) +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 5, 9, 10, 12, 19, 22, 24 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs` (`CreateInstanceActor` ~1605–1632, `HandleTerminated`) +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs` + +1. Failing test: + +```csharp +[Fact] +public void InstanceActorInitFailure_RemovesDeadRefFromMap() +{ + var poisonConfig = "{ not json"; // InstanceActor ctor JsonSerializer.Deserialize throws + var dm = CreateDeploymentManager(); + var healthProbe = _healthCollector; // fake collector already captured by the harness + + dm.Tell(new DeployInstanceCommand(Guid.NewGuid().ToString(), "poison", poisonConfig, "r1" /*…*/)); + + // After the child Stops (decider: ActorInitializationException -> Stop), the map + // must not hold a dead ref: instance counts must return to 0 deployed. + AwaitAssert(() => Assert.Equal(0, healthProbe.LastDeployedCount), TimeSpan.FromSeconds(10)); +} +``` + + **Note:** with Task 4 in place, `"{ not json"` is rejected by the compile validator (deserialize error) before an actor is created — so drive this test through a config that *deserializes and validates* but makes the ctor throw (e.g. a config whose `Attributes` list contains an entry that trips `DecodeAttributeValue`'s non-guarded path, or temporarily bypass via the startup path: seed storage with the poison config and let startup create the actor). Prefer the **startup path** (storage returns the poison row): it does not pass through the deploy validation gate and is exactly the "dead ref" scenario S6 describes. +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~InstanceActorInitFailure_RemovesDeadRefFromMap"` → **FAIL** (map keeps the dead ref; counts stay 1). +3. Implement: + - `CreateInstanceActor`: after `Context.ActorOf`, add `Context.Watch(actorRef);`. + - `HandleTerminated(Terminated t)`: keep the existing redeploy-buffer logic FIRST (it matches on its own bookkeeping). Add a fallback branch at the end: if `_instanceActors.TryGetValue(t.ActorRef.Path.Name, out var mapped) && mapped.Equals(t.ActorRef)` → remove from `_instanceActors`, `UpdateInstanceCounts()`, `LogError` + fire-and-forget a `deployment` site event (`"Instance {name} terminated unexpectedly (init failure or crash) — removed from routing"`). Guard: expected stops (disable/delete/redeploy) remove or replace the map entry *before* the child stops, so `mapped.Equals(t.ActorRef)` is false for them and the fallback is inert — verify against `HandleDisable`/`HandleDelete`/the redeploy shadow map and adjust ordering only if a path removes after stop. + - Redeploy path already `Context.Watch`es; double-watch is idempotent in Akka.NET (single Terminated), but confirm the redeploy handler consumes its Terminated *before* the fallback branch runs (order the if/else so the buffered-redeploy match wins). +4. Run → **PASS**; run `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~DeploymentManager"` (the redeploy race suites are the regression canary here — all must stay green). +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs && git commit -m "fix(site-runtime): watch every Instance Actor; unexpected termination evicts the dead ref from routing (S6)"` + +### Task 15: Fail the in-flight deployment when its Instance Actor dies during init + +**Classification:** high-risk (deployment status protocol) +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 5, 9, 10, 12, 19, 22, 24 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs` (`ApplyDeployment` ~538–587, `HandleDeployPersistenceResult` ~596–636, `HandleTerminated`) +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs` + +1. Failing test: + +```csharp +[Fact] +public void Deploy_WhoseActorDiesDuringInit_ReportsFailed_NotSuccess() +{ + // Config that passes compile validation but whose InstanceActor ctor throws — + // reuse/extend the poison-config helper from Task 14 (ctor-throwing but valid JSON), + // or a config helper stub if none exists (e.g. attribute row that trips ctor indexing). + var dm = CreateDeploymentManager(); + dm.Tell(new DeployInstanceCommand(Guid.NewGuid().ToString(), "dies-on-init", CtorThrowingConfig(), "r1" /*…*/)); + + var resp = ExpectMsg(TimeSpan.FromSeconds(10)); + Assert.Equal(DeploymentStatus.Failed, resp.Status); +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~Deploy_WhoseActorDiesDuringInit"` → **FAIL** (Success is reported once SQLite persists). +3. Implement: + - Field: `private readonly Dictionary _inFlightDeployments = new();` keyed by instance name — store a small record `InFlightDeploy(string DeploymentId, IActorRef ReplyTo)` instead (clearer): `private readonly Dictionary _inFlightDeployments = new();` + - `ApplyDeployment`: before spawning the persistence task, `_inFlightDeployments[instanceName] = new InFlightDeploy(command.DeploymentId, sender);` + - Task 14's unexpected-`Terminated` fallback branch: after evicting the map entry, if `_inFlightDeployments.Remove(name, out var inflight)` → `inflight.ReplyTo.Tell(new DeploymentStatusResponse(inflight.DeploymentId, name, DeploymentStatus.Failed, "Instance actor failed during initialization — see site event log", DateTimeOffset.UtcNow));` and remove the optimistically-persisted row (`Task.Run(() => _storage.RemoveDeployedConfigAsync(name))` fire-and-forget with fault log) + `_deployedInstanceNames.Remove(name)` + `UpdateInstanceCounts()`. + - `HandleDeployPersistenceResult`: only reply if `_inFlightDeployments.Remove(result.InstanceName)` returns true (both success and failure branches) — if the Terminated path already failed the deployment, swallow the late persistence Success (log Debug). Both orders (Terminated-then-persistence, persistence-then-Terminated) now produce exactly one reply. +4. Run → **PASS**; re-run the full DM suite plus Task 4's test (the validation-gate reply path must not double-reply): `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~DeploymentManager"`. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs && git commit -m "fix(site-runtime): deployment whose Instance Actor dies during init reports Failed, exactly-once reply (S6)"` + +### Task 16: DataConnectionActor — capture `_adapter` into locals before every background task + +**Classification:** high-risk (actor-state confinement) +**Estimated implement time:** ~4 min +**Parallelizable with:** 3, 5, 9, 10, 11, 12, 13, 19, 21, 22, 24 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs` (`HandleSubscribe` ~707–758, `HandleWriteBatch` ~1155–1202, `HandleWrite`, `ReSubscribeAll` ~1594–1605; audit every `Task.Run`/local-async-function for `_adapter` reads) +- Test: `tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/DataConnectionActorTests.cs` + +1. Failing test (deterministic via a gated adapter — the in-flight subscribe must finish against the adapter that STARTED it, even after a failover swaps `_adapter`): + +```csharp +[Fact] +public async Task InFlightSubscribe_ContinuesOnOriginalAdapter_AfterFailover() +{ + var subscribeGate = new TaskCompletionSource(); + var primaryAdapter = Substitute.For(); + primaryAdapter.ConnectAsync(default!).ReturnsForAnyArgs(Task.CompletedTask); + primaryAdapter.SubscribeAsync(default!, default!).ReturnsForAnyArgs( + _ => subscribeGate.Task, // tag 1 blocks mid-flight + _ => Task.FromResult("sub-2")); // tag 2 resumes after failover + var backupAdapter = Substitute.For(); + var factory = Substitute.For(); + factory.Create(default!, default!).ReturnsForAnyArgs(backupAdapter); + + var actor = CreateConnectedActor(primaryAdapter, factory, backupConfig: SomeBackupConfig()); // harness + actor.Tell(new SubscribeTagsRequest(NewCorrelation(), "inst", "conn", new[] { "t1", "t2" }, Now())); + + TriggerFailoverToBackup(actor); // drive AdapterDisconnected + failed reconnects past retry count + + subscribeGate.SetResult("sub-1"); // release the in-flight background loop + AwaitAssert(async () => + { + await primaryAdapter.ReceivedWithAnyArgs(2).SubscribeAsync(default!, default!); + await backupAdapter.DidNotReceiveWithAnyArgs().SubscribeAsync(default!, default!); + }); +} +``` + + (Build on the existing `DataConnectionActorTests` failover harness — it already drives `CountFailureAndMaybeFailover` via consecutive `ConnectResult` failures.) +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~InFlightSubscribe_ContinuesOnOriginalAdapter"` → **FAIL** (later loop iterations read the swapped `_adapter` field and hit the backup with the old generation). +3. Implement — one pattern, four sites: + - `HandleSubscribe`: `var adapter = _adapter;` on the actor thread before `Task.Run`; use `adapter.SubscribeAsync(...)` and `SeedTagsAsync(adapter, tagsToSeed)` inside the task. Update the confinement comment at ~689 to note the adapter is captured too. + - `HandleWriteBatch`: `var adapter = _adapter;` before `RunAsync()`; use `adapter.WriteBatchAsync` / `adapter.WriteAsync` (the second read currently happens after the first `await`, off-thread). + - `HandleWrite`: same audit/fix (the single-write path has the same local-async-function shape). + - `ReSubscribeAll`: the per-tag `SubscribeAsync` loop runs on the actor thread (safe), but capture `var subscribeAdapter = _adapter;` and use it in the loop anyway for symmetry with `reseedAdapter` — zero cost, removes the last field read reachable from this method's continuations. +4. Run → **PASS**; full DCL actor suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~DataConnectionActor"`. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/DataConnectionActorTests.cs && git commit -m "fix(dcl): capture adapter into locals before background tasks — no off-thread reads of the mutable _adapter field (S7)"` + +### Task 17: DCL — tag→instances reverse index for the value fan-out hot path + +**Classification:** high-risk (hottest DCL loop, index-consistency invariants) +**Estimated implement time:** ~5 min +**Parallelizable with:** 5, 9, 10, 11, 12, 13, 19, 21, 22, 24 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs` (`HandleTagValueReceived` ~1693–1738; every `_subscriptionsByInstance` mutation site) +- Test: `tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/DataConnectionActorTests.cs` + +1. Failing test (behavioral — index correctness across unsubscribe, which a naive index gets wrong): + +```csharp +[Fact] +public void TagFanOut_AfterOneInstanceUnsubscribes_OnlyRemainingSubscriberReceives() +{ + var actor = CreateConnectedActorWithInstantSubscribes(); // adapter substitute returns sub ids immediately + var probeA = CreateTestProbe(); var probeB = CreateTestProbe(); + + actor.Tell(SubscribeRequest("inst-A", new[] { "shared-tag" }), probeA.Ref); + actor.Tell(SubscribeRequest("inst-B", new[] { "shared-tag" }), probeB.Ref); + probeA.ExpectMsg(); probeB.ExpectMsg(); + probeA.FishForMessage(m => m is TagValueUpdate, hint: "seed"); // drain seeds as the harness does + + actor.Tell(UnsubscribeRequest("inst-A", new[] { "shared-tag" }), probeA.Ref); + DeliverTagValue(actor, "shared-tag", 42); // helper: Tell TagValueReceived with current generation + + probeB.ExpectMsg(u => u.TagPath == "shared-tag"); + probeA.ExpectNoMsg(TimeSpan.FromMilliseconds(300)); +} +``` + + This passes today via the O(N) scan — write it FIRST and confirm it passes (pin the behavior), then convert `HandleTagValueReceived` to the index and confirm it STILL passes. Add one more assertion-style test that also passes pre-change (two instances, disjoint tags, each receives only its own) — these are the regression net for the index bookkeeping. +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~TagFanOut_"` → expect **PASS** pre-change (documented pin), then proceed. +3. Implement: + - Field: `private readonly Dictionary> _instancesByTag = new();` with XML doc mirroring `_tagSubscriberCount`'s rationale. + - Add/remove alongside **every** `_subscriptionsByInstance[inst]` tag-set mutation. Grep `_subscriptionsByInstance` and cover: `HandleSubscribeCompleted` (tags added), `HandleUnsubscribe` (tags removed + instance entry removal), the orphan-release guards (~836–985, 1795–1858 region), and any instance-removal path. Write two private helpers `IndexTag(string tag, string inst)` / `UnindexTag(string tag, string inst)` (removing the tag key when its set empties) and use them at every site — no inline mutation. + - `HandleTagValueReceived`: replace the `_subscriptionsByInstance` scan with `if (_instancesByTag.TryGetValue(msg.TagPath, out var insts)) foreach (var inst in insts) if (_subscribers.TryGetValue(inst, out var s)) s.Tell(...)`. + - `ReSubscribeAll` preserves `_subscriptionsByInstance` — so it must NOT clear `_instancesByTag` (it is derived from the same preserved state). Note this in a comment. +4. Run the fan-out tests + full DCL suite → **PASS**. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/DataConnectionActorTests.cs && git commit -m "perf(dcl): tag->instances reverse index replaces per-update all-instances scan (P3)"` + +### Task 18: InstanceActor — per-attribute child-subscription routing + +**Classification:** high-risk (hot-path routing semantics for scripts AND alarms) +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 5, 10, 12, 13, 17, 19, 22, 24 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs` (`CreateChildActors` ~1348–1488, `PublishAndNotifyChildren` ~1179–1209) +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorChildAttributeRaceTests.cs` (or a new `InstanceActorChildRoutingTests.cs`) + +1. Failing test: + +```csharp +[Fact] +public void AttributeChange_IsRoutedOnlyToInterestedChildren() +{ + // Instance with: script S_A (ValueChange on "A"), script S_B (ValueChange on "B"), + // script S_expr (Expression trigger). Script bodies record executions via the + // static-recording pattern the existing tests use. + var instance = CreateInstance(ScriptsAB_and_Expression()); + + SetStatic(instance, "A", 1); // publishes AttributeValueChanged for A + + AwaitAssert(() => + { + Assert.Equal(1, Executions("S_A")); + Assert.Equal(0, Executions("S_B")); // FAILS pre-change only via mailbox counting… + }); +} +``` + + `S_B` never *executes* today either (its own trigger check discards) — so assert on **message delivery**, not execution: give the routing seam a test hook. Concretely: make the routing maps `internal` and unit-test map construction directly (`InstanceActor` exposes `internal IReadOnlyDictionary> ChildrenByMonitoredAttribute` / `internal IReadOnlyList AllChangeChildren` next to the existing `ScriptActorCount` diagnostics), plus a behavioral test that the Expression script still fires on ANY attribute change and the ValueChange script still fires on its own attribute: + +```csharp +[Fact] +public void RoutingMaps_AreBuiltFromTriggerConfigs() +{ + var actor = CreateInstanceActorRef(ScriptsAB_and_Expression()); + var underlying = GetUnderlyingActor(actor); // TestActorRef pattern used elsewhere in this file + Assert.Single(underlying.ChildrenByMonitoredAttribute["A"]); + Assert.Single(underlying.ChildrenByMonitoredAttribute["B"]); + Assert.Single(underlying.AllChangeChildren); // the Expression script +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~RoutingMaps_AreBuiltFromTriggerConfigs"` → **FAIL**. +3. Implement: + - Fields: `private readonly Dictionary> _childrenByMonitoredAttribute = new(); private readonly List _allChangeChildren = new();` + - In `CreateChildActors`, after creating each script/alarm actor, classify: + - Script trigger `valuechange`/`conditional` → parse `attributeName` from `TriggerConfiguration` JSON (reuse/extract the parse `ScriptActor.ParseTriggerConfig` does — lift a tiny static helper `TriggerRouting.TryGetMonitoredAttribute(triggerType, triggerConfigJson, out string attr)` into `Scripts/` so ScriptActor and InstanceActor share one parser). Add to `_childrenByMonitoredAttribute[attr]`. + - Script trigger `expression` → `_allChangeChildren` (its snapshot must stay current). + - Script trigger `interval`/`call`/unparseable → **`_allChangeChildren` if unparseable (fail-open, log warning), else no routing** (interval/call scripts never react to attribute changes; their `_attributeSnapshot` is only read by expression evaluation, which they don't have — verify this before excluding, it is true today at `ScriptActor` where the snapshot is read only in `EvaluateExpressionTrigger`). + - Alarms: same classification via the alarm's trigger type — Expression → all bucket; HiLo/ValueMatch/RangeViolation/RateOfChange → monitored attribute (parse from the alarm config the same way `AlarmActor.IsMonitoredAttribute` derives it); unparseable → all bucket. + - `PublishAndNotifyChildren`: replace the two full loops with `foreach (var c in _allChangeChildren) c.Tell(changed);` + `if (_childrenByMonitoredAttribute.TryGetValue(changed.AttributeName, out var interested)) foreach (var c in interested) c.Tell(changed);`. Children keep their own trigger gates (defense in depth — routing is an optimization, not the correctness gate). + - Note: `changed.AttributeName` on the DCL path is the canonical name (`HandleTagValueUpdate` maps tag→attributes before publishing) — key the map by canonical name; confirm against `HandleTagValueUpdate`. +4. Run new tests + the whole InstanceActor/ScriptActor/AlarmActor suites: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~InstanceActor|FullyQualifiedName~ScriptActor|FullyQualifiedName~AlarmActor"` → **PASS**. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests && git commit -m "perf(site-runtime): per-attribute child routing in InstanceActor — attribute changes reach only interested children (P2)"` + +### Task 19: SiteStorageService — WAL journal mode + busy timeout + +**Classification:** small +**Estimated implement time:** ~3 min +**Parallelizable with:** 1, 2, 3, 4, 5, 9, 10, 11, 12, 13, 22, 24 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs` (ctor ~21–25, `InitializeAsync` ~41–132) +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/` (add to the existing SiteStorageService test file) + +1. Failing test: + +```csharp +[Fact] +public async Task Initialize_EnablesWalJournalMode() +{ + var service = CreateTempFileStorageService(); // existing temp-SQLite fixture pattern + await service.InitializeAsync(); + + await using var conn = service.CreateConnection(); + await conn.OpenAsync(); + await using var cmd = conn.CreateCommand(); + cmd.CommandText = "PRAGMA journal_mode;"; + Assert.Equal("wal", ((string)(await cmd.ExecuteScalarAsync())!).ToLowerInvariant()); +} +``` + + Note: WAL requires a file-backed DB — if the existing fixture uses `:memory:`, use the temp-file variant (several persistence tests already do). +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~Initialize_EnablesWalJournalMode"` → **FAIL** (`delete`). +3. Implement: + - In `InitializeAsync`, before creating tables: execute `PRAGMA journal_mode=WAL;` (persistent, database-level — one-time). Log the resulting mode; `:memory:`/unsupported filesystems fall back gracefully (don't throw if the pragma returns non-wal, just warn). + - In the ctor, normalize the connection string through `SqliteConnectionStringBuilder` and ensure `DefaultTimeout` is at least 5 s (Microsoft.Data.Sqlite uses the command timeout as its busy handler — an explicit floor documents intent even though the library default is 30): only raise it if the caller set something lower. Add an XML remark on `CreateConnection()` documenting WAL + busy-handling so the repositories layered on it inherit the story. +4. Run → **PASS**; run the persistence folder suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~SiteStorageService"`. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence && git commit -m "fix(site-runtime): site SQLite runs WAL with an explicit busy-timeout floor (S8) — concurrent writers stop serializing on the rollback journal"` + +### Task 20: NativeAlarmActor — coalesce per-transition SQLite upserts + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 4, 5, 9, 10, 11, 13, 17, 22, 24 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/NativeAlarmActor.cs` (`PersistUpsert` ~420–428, timer plumbing) +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs` (new `UpsertNativeAlarmsAsync` batch — one connection + transaction) +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs` + +1. Failing test (against the temp-SQLite storage the existing NativeAlarmActor tests use): + +```csharp +[Fact] +public async Task AlarmStorm_CoalescesPersistence_LatestPerSourceRefWins() +{ + var (actor, storage) = CreateNativeAlarmActorWithTempStorage(flushIntervalMs: 100); + + for (var i = 0; i < 50; i++) + DeliverLiveTransition(actor, sourceRef: $"ref-{i % 5}", severity: i); // 50 transitions, 5 refs + + await Task.Delay(500); // > flush interval + var rows = await storage.GetNativeAlarmsAsync("inst", "src"); + Assert.Equal(5, rows.Count); + // Latest transition per ref persisted (severity of the LAST write for ref-4 is 49): + Assert.Contains(rows, r => r.ConditionJson.Contains("49")); +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~AlarmStorm_CoalescesPersistence"` → **FAIL** only on the *mechanism* (row content passes either way) — so also assert the batch API is used: add `UpsertNativeAlarmsAsync` first (it won't exist → compile-fail = the failing signal), or count connections via a storage wrapper. Simplest honest failing signal: write the test calling `storage.UpsertNativeAlarmsAsync(...)` shape via the actor and assert rows — compile failure is the RED step. +3. Implement: + - `SiteStorageService.UpsertNativeAlarmsAsync(string instance, string source, IReadOnlyList<(string SourceRef, string ConditionJson, DateTimeOffset At)> rows)` — one connection, one transaction, one prepared command re-bound per row. + - `NativeAlarmActor`: replace direct `PersistUpsert(t)` calls (in `ApplyLiveTransition` and `ApplySnapshotSwap`) with `_dirtyUpserts[t.SourceReference] = t;` + arm a 100 ms single-shot flush timer if not armed (`ScheduleTellOnceCancelable`, internal `FlushPersistence` message). The flush handler snapshots + clears `_dirtyUpserts`, calls the batch API fire-and-forget with the existing `OnlyOnFaulted` logging. `PersistDelete` stays immediate (rare; also remove the ref from `_dirtyUpserts` so a delete isn't resurrected by a pending upsert). `PostStop`: best-effort final flush (fire-and-forget). + - Rehydration semantics unchanged: worst case a crash loses ≤100 ms of mirror writes; the (re)subscribe snapshot swap reconciles — note this in the class doc. +4. Run → **PASS**; full NativeAlarmActor suite green. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/NativeAlarmActor.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs && git commit -m "perf(site-runtime): coalesce native-alarm mirror persistence into batched flushes (P4)"` + +### Task 21: Persist native-alarm display metadata for rehydration + +**Classification:** standard (SQLite schema migration — additive column) +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 4, 5, 9, 10, 11, 13, 22, 24 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs` (`native_alarm_state` schema ~117–124, `MigrateSchemaAsync`, upsert/read methods) +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/NativeAlarmActor.cs` (`PersistUpsert`/flush ~420–428, `HandleRehydration` ~134–174) +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs` + +1. Failing test: + +```csharp +[Fact] +public async Task Rehydration_RestoresDisplayMetadata() +{ + var (actor1, storage) = CreateNativeAlarmActorWithTempStorage(); + DeliverLiveTransition(actor1, sourceRef: "ref-1", + alarmTypeName: "HighLevelAlarm", category: "Process", message: "Tank overflow"); + await FlushAndStop(actor1); + + var (actor2, parentProbe) = RecreateActorOnSameStorage(storage); // rehydrates in PreStart + var emitted = parentProbe.FishForMessage(m => m.SourceReference == "ref-1"); + Assert.Equal("HighLevelAlarm", emitted.AlarmTypeName); // empty string today + Assert.Equal("Process", emitted.Category); + Assert.Equal("Tank overflow", emitted.Message); +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~Rehydration_RestoresDisplayMetadata"` → **FAIL** (metadata comes back `string.Empty`). +3. Implement: + - Schema: add `metadata_json TEXT` to the `CREATE TABLE native_alarm_state` DDL AND `await TryAddColumnAsync(connection, "native_alarm_state", "metadata_json", "TEXT");` in `MigrateSchemaAsync` (existing-DB upgrade). + - Persist: serialize a small record `NativeAlarmMetadata(string AlarmTypeName, string Category, string Message, string CurrentValue, string LimitValue)` from the transition into `metadata_json` in the upsert (single + Task 20 batch). `NativeAlarmRow` gains `string? MetadataJson`. + - Rehydrate: in `HandleRehydration`, deserialize `MetadataJson` (null/malformed → empty strings, current behavior) and build the placeholder `NativeAlarmTransition` with the restored fields instead of `string.Empty`. +4. Run → **PASS**; full NativeAlarmActor + persistence suites. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/NativeAlarmActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs && git commit -m "feat(site-runtime): persist native-alarm display metadata so rehydrated conditions render fully (UA4)"` + +### Task 22: CertStoreActor — export trusted certs (thumbprint + DER) for reconciliation + +**Classification:** standard (additive Commons message contract) +**Estimated implement time:** ~4 min +**Parallelizable with:** 1, 2, 3, 4, 5, 9, 10, 11, 12, 13, 16, 17, 19, 24 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Management/` (the file holding `WriteCertToLocalStore`/`ListLocalCerts` — add `ExportLocalTrustedCerts` + `LocalCertExport`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/CertStoreActor.cs` (new handler) +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/CertStoreActorTests.cs` + +1. Failing test (reuse the temp-dir PKI fixture in `CertStoreActorTests`): + +```csharp +[Fact] +public void ExportLocalTrustedCerts_ReturnsThumbprintAndDerBytes() +{ + var (actor, testCertDer, thumbprint) = CreateCertStoreWithOneTrustedCert(); // existing fixture helpers + actor.Tell(new ExportLocalTrustedCerts()); + var export = ExpectMsg(); + var entry = Assert.Single(export.Certs); + Assert.Equal(thumbprint, entry.Thumbprint); + Assert.Equal(testCertDer, entry.DerBytes); +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ExportLocalTrustedCerts_ReturnsThumbprintAndDerBytes"` → **FAIL** (types don't exist). +3. Implement: + - Commons (additive, next to the existing cert messages): `public sealed record ExportLocalTrustedCerts(); public sealed record ExportedCert(string Thumbprint, byte[] DerBytes); public sealed record LocalCertExport(bool Success, string? Error, IReadOnlyList Certs);` + - `CertStoreActor`: `Receive` — enumerate `*.der` in `_trustedStoreDir`, load each via `X509CertificateLoader`/`new X509Certificate2(bytes)` (match how `HandleList` reads them today) to compute the thumbprint, reply `LocalCertExport(true, null, list)`; any IO error → `LocalCertExport(false, ex.Message, [])`. +4. Run → **PASS**. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Commons src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/CertStoreActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/CertStoreActorTests.cs && git commit -m "feat(site-runtime): CertStoreActor exports trusted certs (thumbprint+DER) for node reconciliation (UA1 groundwork)"` + +### Task 23: Cert-trust reconcile-on-join — singleton pushes its trusted set to a (re)joining site node + +**Classification:** high-risk (cluster membership events) +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 5, 9, 10, 12, 17, 19, 24 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs` (PreStart/PostStop cluster subscription; new handlers near the cert section ~900–1007) +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs` + +1. Failing test — test the reconcile flow via the internal message (constructing a real `ClusterEvent.MemberUp` in a unit test is impractical; the MemberUp handler itself is a two-liner producing this message). Use the same "forwarding actor named `cert-store` → TestProbe" trick the existing cert broadcast tests use: + +```csharp +[Fact] +public void SiteNodeJoined_PushesLocalTrustedCertsToJoinedNode() +{ + var certStoreProbe = InstallCertStoreForwarder(); // /user/cert-store -> probe (existing pattern) + var dm = CreateDeploymentManager(); + + dm.Tell(new DeploymentManagerActor.SiteNodeJoined(Cluster.Get(Sys).SelfAddress)); + + certStoreProbe.ExpectMsg(); // 1) read own store + dm.Tell(new LocalCertExport(true, null, + new[] { new ExportedCert("ABC123", new byte[] { 1, 2, 3 }) }), certStoreProbe.Ref); + // 2) pushed to the joined node's cert-store (self address in-test -> same selection): + var write = certStoreProbe.ExpectMsg(TimeSpan.FromSeconds(5)); + Assert.Equal("ABC123", write.Thumbprint); +} +``` + + (Match `WriteCertToLocalStore`'s real field names/args from Commons when writing the assertion.) +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~SiteNodeJoined_PushesLocalTrustedCerts"` → **FAIL**. +3. Implement: + - `PreStart`: `Cluster.Get(Context.System).Subscribe(Self, ClusterEvent.SubscriptionInitialStateMode.InitialStateAsEvents, typeof(ClusterEvent.MemberUp));` `PostStop`: `Cluster.Get(Context.System).Unsubscribe(Self);` + - `Receive`: if the member has the site role and is not the singleton's own node → `Self.Tell(new SiteNodeJoined(member.Address))` (public/internal record for testability). + - `Receive`: capture the address; ask the LOCAL cert store (`Context.ActorSelection($"/user/{CertStoreActor.WellKnownName}")`) with `ExportLocalTrustedCerts` (short `CertBroadcastTimeout` Ask, `ContinueWith` → pipe a wrapper carrying the target address). On the piped `LocalCertExport` (wrap it: `CertReconcileExport(Address Target, LocalCertExport Export)`): for each cert, `Tell` the target node's cert store (`new RootActorPath(target) / "user" / CertStoreActor.WellKnownName`) a `WriteCertToLocalStore(...)` — fire-and-forget, additive union (documented: removals do NOT reconcile until central persistence lands; the write is idempotent — `CertStoreActor.HandleWrite` overwrites by thumbprint). + - Log one info line per reconcile (`"Cert-trust reconcile: pushed {N} trusted cert(s) to joining node {Address}"`); export failure → warning, no retry (the next MemberUp or manual trust re-broadcast heals). + - Simplification for the test flow: since the test drives `LocalCertExport` directly at the actor, have `Receive` be the handler and also accept a bare `LocalCertExport` only if you keep a single pending target field — prefer the wrapper message; adjust the test to send the wrapper if needed. +4. Run → **PASS**; full DM suite green. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs && git commit -m "feat(site-runtime): cert-trust reconcile-on-join — singleton pushes trusted certs to (re)joining site nodes (UA1)"` + +### Task 24: Wire `ConfigFetchRetryCount` into the standby replication fetch + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** 1, 2, 3, 4, 5, 9, 10, 12, 13, 16, 17, 19, 22 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/SiteReplicationActor.cs` (`HandleApplyConfigDeploy` ~153–217; ctor gains `SiteRuntimeOptions? options = null`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs` (fix the "Reserved — not yet wired" XML doc, lines 63–67) +- Modify: the `SiteReplicationActor` Props construction site (Host/ServiceCollectionExtensions — grep `new SiteReplicationActor(` / `Props.Create(() => new SiteReplicationActor`) to pass options +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReplicationActorTests.cs` + +1. Failing test: + +```csharp +[Fact] +public void ReplicatedFetch_RetriesUpToConfigFetchRetryCount() +{ + var attempts = 0; + var fetcher = Substitute.For(); + fetcher.FetchAsync(default!, default!, default!, default).ReturnsForAnyArgs(_ => + ++attempts < 3 + ? Task.FromException(new HttpRequestException("central hiccup")) + : Task.FromResult("{\"config\":true}")); + var actor = CreateReplicationActor(fetcher, + new SiteRuntimeOptions { ConfigFetchRetryCount = 3 }); // existing harness + new arg + + actor.Tell(new ApplyConfigDeploy("inst", "dep-1", "r1", true, "http://central", "tok")); + + AwaitAssert(() => + { + Assert.Equal(3, attempts); + _storage.Received(1).StoreDeployedConfigIfNewerAsync("inst", "{\"config\":true}", "dep-1", "r1", true); + }, TimeSpan.FromSeconds(10)); +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ReplicatedFetch_RetriesUpToConfigFetchRetryCount"` → **FAIL** (single attempt). +3. Implement: replace the single `FetchAsync(...).ContinueWith(...)` with a local async function launched fire-and-forget (same best-effort model): + +```csharp +var maxAttempts = Math.Max(1, _options?.ConfigFetchRetryCount ?? 1); +_ = FetchWithRetryAsync(); // errors fully handled inside; observes every fault +async Task FetchWithRetryAsync() +{ + for (var attempt = 1; attempt <= maxAttempts; attempt++) + { + try + { + var json = await _configFetcher.FetchAsync(msg.CentralFetchBaseUrl, msg.DeploymentId, msg.FetchToken, CancellationToken.None); + await _storage.StoreDeployedConfigIfNewerAsync(msg.InstanceName, json, msg.DeploymentId, msg.RevisionHash, msg.IsEnabled); + return; + } + catch (DeploymentConfigFetchException fex) when (fex.IsSuperseded) + { /* keep the existing "superseded" info log */ return; } + catch (Exception ex) + { + // keep existing per-failure log wording; add attempt counters + if (attempt < maxAttempts) await Task.Delay(TimeSpan.FromSeconds(2)); + else _logger.LogError(ex, "Replicated config fetch failed after {Attempts} attempt(s) for {Instance} (deployment {DeploymentId}) — reconciliation will backstop", maxAttempts, msg.InstanceName, msg.DeploymentId); + } + } +} +``` + + Update the `SiteRuntimeOptions.ConfigFetchRetryCount` XML doc: remove "Reserved … not yet wired"; state "Bounded attempt count (including the first) for the standby's replicated-config fetch; 2 s fixed delay between attempts; superseded fetches never retry." +4. Run → **PASS**; run the replication suite. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/SiteReplicationActor.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReplicationActorTests.cs && git commit -m "feat(site-runtime): wire ConfigFetchRetryCount into the standby replicated-config fetch (UA2)"` (include the Props-site file in the `git add` if touched) + +### Task 25: Low-severity code cleanups — S9, S10, C2, C6 + +**Classification:** small (batched Low findings; concrete edits below) +**Estimated implement time:** ~5 min +**Parallelizable with:** 13, 19, 21, 22, 24 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs` (PostStop ~216–221; `HandleSubscribeAlarms` ~1763) +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs` (comment ~215–218) +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs` (`EvaluateCondition` fallback ~474–477) +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ScriptActorTests.cs`, `tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/DataConnectionActorAlarmTests.cs` + +1. **C6 failing test** (culture-independence of the non-numeric fallback): + +```csharp +[Fact] +public void ConditionalTrigger_NonNumericFallback_IsCultureInvariant() +{ + var original = CultureInfo.CurrentCulture; + try + { + CultureInfo.CurrentCulture = new CultureInfo("de-DE"); // 1.5 -> "1,5" + // Conditional trigger: operator "==", Threshold = 1.5; value "1,5" fails + // invariant numeric parse -> string fallback. Old code compares against + // Threshold.ToString() == "1,5" under de-DE and FIRES; invariant must NOT. + var fired = DriveConditionalTrigger(threshold: 1.5, op: "==", value: "1,5"); // harness helper + Assert.False(fired); + } + finally { CultureInfo.CurrentCulture = original; } +} +``` + +2. **S10 failing test** (EventFilter on the warning): + +```csharp +[Fact] +public void SecondAlarmSubscriber_WithDifferentFilter_LogsOverwriteWarning() +{ + var actor = CreateConnectedAlarmCapableActor(); // existing alarm-test harness + actor.Tell(AlarmSubscribe("src-1", filter: "HighAlarm"), probeA.Ref); + EventFilter.Warning(contains: "condition filter").ExpectOne(() => + actor.Tell(AlarmSubscribe("src-1", filter: "LowAlarm"), probeB.Ref)); +} +``` + +3. Run both → **FAIL** (`dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ConditionalTrigger_NonNumericFallback"` and `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~SecondAlarmSubscriber_WithDifferentFilter"`). +4. Implement all four edits: + - **C6** (`ScriptActor.cs:476`): `return string.Equals(value.ToString(), config.Threshold.ToString(CultureInfo.InvariantCulture), StringComparison.Ordinal);` — matches the invariant-culture handling two lines up. + - **S10** (`DataConnectionActor.cs`, before line 1763's overwrite): `if (_alarmSourceFilter.TryGetValue(request.SourceReference, out var existingFilter) && !string.Equals(existingFilter, request.ConditionFilter, StringComparison.Ordinal)) _log.Warning("[{0}] Alarm condition filter for source {1} overwritten: '{2}' -> '{3}' (last subscriber wins; co-subscribers must agree)", _connectionName, request.SourceReference, existingFilter, request.ConditionFilter);` + - **S9** (`DataConnectionActor.PostStop`): replace `_ = _adapter.DisposeAsync().AsTask();` with `_adapter.DisposeAsync().AsTask().ContinueWith(t => _log.Warning("[{0}] Adapter dispose faulted on stop: {1}", _connectionName, t.Exception?.GetBaseException().Message), TaskContinuationOptions.OnlyOnFaulted);` + - **C2** (`SiteEventLogger.cs:215-218`): rewrite the comment: "The channel is bounded with DropOldest, so TryWrite only returns false once the channel is completed (shutdown) — full-buffer pressure evicts the oldest pending event instead of failing the write." +5. Run both tests → **PASS**; run both touched projects' suites. +6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ScriptActorTests.cs tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/DataConnectionActorAlarmTests.cs && git commit -m "fix(site-runtime,dcl): low-severity cleanups — observed adapter dispose (S9), alarm-filter overwrite warning (S10), bounded-channel comment (C2), invariant-culture condition fallback (C6)"` + +### Task 26: Design-doc updates — C3, C4, C5 + behavior notes for this plan's changes + +**Classification:** trivial (docs only; repo rule: docs travel with code) +**Estimated implement time:** ~5 min +**Parallelizable with:** none (write after the behavior tasks land so the docs describe reality) +**Files:** +- Modify: `docs/requirements/Component-DataConnectionLayer.md` +- Modify: `docs/requirements/Component-SiteRuntime.md` + +1. `Component-DataConnectionLayer.md`: + - **C3** (line ~303): change "The retry interval is defined **per data connection**" to "The retry interval is a **shared setting for all data connections** (`ReconnectInterval` in the Shared Settings table)" — matching the table at lines 170–177 and `DataConnectionOptions`. + - **C4** (line ~302, item 1): qualify — "Instance Actors see the staleness immediately and publish it to the site stream/debug view; **script and alarm actors are deliberately NOT re-notified on a quality-only change** (no value changed — re-evaluating triggers would cause spurious firings). Scripts observe quality on their next attribute read; a computed alarm holds its last-known state until fresh values arrive." + - **S1 note** (connection lifecycle section): add "On every (re)connect the adapter detaches and disposes the previous protocol client (session, keep-alive timers, event loop) before creating a new one — a stale client can neither leak nor signal a disconnect against the new session." + - **S10 note**: add one line documenting the last-subscriber-wins alarm filter + the logged warning on mismatch. +2. `Component-SiteRuntime.md`: + - **C5** (Shared Script Library section): "Shared scripts always execute with **Root scope** — `Attributes[...]` inside a shared script addresses root attributes even when invoked from a composed-module script. Callers needing module-scoped access must pass values as parameters." + - **S2/UA5** (Script Trust Model / execution-timeout paragraph ~409–412): qualify "Exceeding the timeout cancels the script" → "…cancels the script **cooperatively**: a script blocked in synchronous I/O or a CPU loop does not observe cancellation and continues to occupy its dedicated script-execution thread. A watchdog logs the script by name when its thread has not returned within a grace period after timeout, and the scheduler's queue depth / busy-thread-age gauges are reported through Health Monitoring." + - **S3** (Script Compilation Errors section ~474–477): add "Enforced at the site by a pre-compile validation gate in the Deployment Manager: all scripts, alarm on-trigger scripts, and trigger expressions are compiled off-thread before the Instance Actor is created or the config persisted; any failure rejects the deployment with the compile errors." + - **S4/UA6** (Instance Actor / DCL interaction): add "A failed or unanswered tag-subscription handshake is retried on a fixed interval per connection (default 5 s), mirroring the native-alarm subscribe retry." + - **S5**: note the fixed-interval retry of the startup deployed-config load. + - **S6**: note that all Instance Actors are watched and an unexpected termination evicts the ref and fails any in-flight deployment. + - **UA1** (cert trust section ~125): add "On a site node (re)joining the cluster, the Deployment Manager singleton pushes its trusted-cert set to the joining node's CertStore (additive union). Removal reconciliation and central persistence of trust decisions remain the documented follow-up." + - **UA4** (native alarm mirror section): "Persisted mirror rows include display metadata (type/category/message), so rehydrated conditions render fully before the first source snapshot." + - **P2** note (attribute change fan-out ~191): "The Instance Actor routes attribute changes per monitored attribute (Expression-trigger children receive all changes); child actors keep their own trigger gates as defense in depth." + - **P6 follow-up note** (deployment/startup section): "Follow-up: per-instance Roslyn compiles still run inside Instance Actor start during staggered startup; moving them off-thread is deferred (affects failover time-to-recover only)." +3. Review with `git diff docs/` — verify no stale cross-references (e.g., the C3 sentence is the only "per data connection" retry claim). +4. Commit: `git add docs/requirements/Component-DataConnectionLayer.md docs/requirements/Component-SiteRuntime.md && git commit -m "docs(requirements): sync SiteRuntime + DCL specs with hardening changes (C3/C4/C5 + S1-S6, P2, UA1/UA4 behavior notes)"` + +--- + +## Dependencies on other plans + +- **Plan 05 (Templates/Deployment/Transport)** owns script-trust-policy and Roslyn compile **caching** in central validation paths. Task 3/4 here adds a *site-side* compile gate that duplicates compile work per deploy — if plan 05 introduces a shared compile-cache/collectible-ALC abstraction, the `DeployCompileValidator` should adopt it later; no ordering dependency (this plan's gate is self-contained). +- **Plan 02 (Communication/S&F)** owns the S&F store SQLite (`StoreAndForwardStorage`) — Task 19 deliberately touches only `SiteStorageService`. If plan 02 also enables WAL on the S&F store, the two changes are independent files. +- **Plan 06 (Edge integrations)** owns inbound script execution (DI-scope use-after-dispose). Task 8's watchdog pattern (`completed` flag + grace check) is reusable there; no ordering dependency. +- **Plan 01 (Cluster/Host/Failover)** owns SBR enablement. Task 23 (reconcile-on-join) becomes *more* valuable once real failover works, but functions independently. +- Health-report additive fields (Task 6) touch `Commons/Messages/Health/SiteHealthReport.cs` — coordinate with any other plan touching that record (additive-only on both sides makes merges trivial). + +## Execution order + +**P0 (do first, in order):** Task 1, Task 2 (S1 — the single highest-value fix: stops the leak *and* the healthy-connection flapping), then Task 3 → Task 4 (S3 — deployment honesty, matches OVERALL P1 recommendation #8). + +**P1:** Tasks 5 → 6 → 7 (scheduler observability chain), Task 8 (watchdog), Tasks 9, 10 (P1 dispatcher relief), Task 11 → 12 (subscription retries), Task 13 (startup retry), Task 14 → 15 (dead-child bookkeeping — 15 depends on 14's Terminated fallback). + +**P2:** Task 16 → 17 (both edit `DataConnectionActor.cs`, sequential), Task 18 (after 11 — both edit `InstanceActor.cs`), Task 19 → 20 → 21 (19 and 21 both edit `SiteStorageService.cs`; 20 and 21 both edit `NativeAlarmActor.cs` — run 19, 20, 21 in that order; 12 must precede 20 for `NativeAlarmActor.cs`), Task 22 → 23 (23 needs 22's messages and follows 15 in `DeploymentManagerActor.cs`), Task 24. + +**Last:** Task 25 (after 9 and 17 — shared files), then Task 26 (docs describe the landed behavior). + +Same-file sequencing summary: `DeploymentManagerActor.cs`: 4 → 13 → 14 → 15 → 23. `InstanceActor.cs`: 4 (doc comment) → 11 → 18. `NativeAlarmActor.cs`: 12 → 20 → 21. `SiteStorageService.cs`: 19 → 20 → 21. `DataConnectionActor.cs`: 16 → 17 → 25. `ScriptActor.cs`: 9 → 18 (helper extraction) → 25. `SiteRuntimeOptions.cs`: 8 → 11 → 24. + +After the final task: `dotnet build ZB.MOM.WW.ScadaBridge.slnx && dotnet test ZB.MOM.WW.ScadaBridge.slnx`, and for cluster-runtime verification rebuild the docker image (`bash docker/deploy.sh`). diff --git a/archreview/plans/PLAN-03-site-runtime-dcl.md.tasks.json b/archreview/plans/PLAN-03-site-runtime-dcl.md.tasks.json new file mode 100644 index 00000000..d136d832 --- /dev/null +++ b/archreview/plans/PLAN-03-site-runtime-dcl.md.tasks.json @@ -0,0 +1,32 @@ +{ + "planPath": "archreview/plans/PLAN-03-site-runtime-dcl.md", + "tasks": [ + { "id": 1, "subject": "Task 1: OpcUaDataConnection — dispose + detach the previous client on reconnect (S1)", "status": "pending", "blockedBy": [] }, + { "id": 2, "subject": "Task 2: MxGatewayDataConnection — dispose previous client + cancel previous event loop on reconnect (S1)", "status": "pending", "blockedBy": [] }, + { "id": 3, "subject": "Task 3: Pure deploy-time compile validator for a flattened config (S3 groundwork)", "status": "pending", "blockedBy": [] }, + { "id": 4, "subject": "Task 4: Reject the deployment on site-side compile failure per spec (S3)", "status": "pending", "blockedBy": [3] }, + { "id": 5, "subject": "Task 5: ScriptExecutionScheduler gauges — queue depth, busy threads, oldest-busy age (S2/UA5)", "status": "pending", "blockedBy": [] }, + { "id": 6, "subject": "Task 6: Surface scheduler stats through ISiteHealthCollector + SiteHealthReport (S2/UA5)", "status": "pending", "blockedBy": [5] }, + { "id": 7, "subject": "Task 7: Periodic scheduler-stats reporter hosted service (S2/UA5)", "status": "pending", "blockedBy": [5, 6] }, + { "id": 8, "subject": "Task 8: Stuck-script watchdog — name the script whose thread never came back (S2)", "status": "pending", "blockedBy": [] }, + { "id": 9, "subject": "Task 9: ScriptActor — trigger-expression evaluation off the dispatcher, coalesced (P1)", "status": "pending", "blockedBy": [] }, + { "id": 10, "subject": "Task 10: AlarmActor — async trigger-expression evaluation on the script scheduler (P1)", "status": "pending", "blockedBy": [] }, + { "id": 11, "subject": "Task 11: InstanceActor — retry failed/lost tag subscriptions per connection (S4/UA6)", "status": "pending", "blockedBy": [4, 8] }, + { "id": 12, "subject": "Task 12: NativeAlarmActor — retry on lost subscribe response (S4)", "status": "pending", "blockedBy": [] }, + { "id": 13, "subject": "Task 13: DeploymentManager — retry the startup config load (S5)", "status": "pending", "blockedBy": [4] }, + { "id": 14, "subject": "Task 14: Watch all Instance Actors; clean the map on unexpected termination (S6)", "status": "pending", "blockedBy": [13] }, + { "id": 15, "subject": "Task 15: Fail the in-flight deployment when its Instance Actor dies during init (S6)", "status": "pending", "blockedBy": [14] }, + { "id": 16, "subject": "Task 16: DataConnectionActor — capture _adapter into locals before background tasks (S7)", "status": "pending", "blockedBy": [] }, + { "id": 17, "subject": "Task 17: DCL — tag→instances reverse index for the value fan-out hot path (P3)", "status": "pending", "blockedBy": [16] }, + { "id": 18, "subject": "Task 18: InstanceActor — per-attribute child-subscription routing (P2)", "status": "pending", "blockedBy": [9, 11] }, + { "id": 19, "subject": "Task 19: SiteStorageService — WAL journal mode + busy timeout (S8)", "status": "pending", "blockedBy": [] }, + { "id": 20, "subject": "Task 20: NativeAlarmActor — coalesce per-transition SQLite upserts (P4)", "status": "pending", "blockedBy": [12, 19] }, + { "id": 21, "subject": "Task 21: Persist native-alarm display metadata for rehydration (UA4)", "status": "pending", "blockedBy": [20] }, + { "id": 22, "subject": "Task 22: CertStoreActor — export trusted certs (thumbprint + DER) for reconciliation (UA1 groundwork)", "status": "pending", "blockedBy": [] }, + { "id": 23, "subject": "Task 23: Cert-trust reconcile-on-join — singleton pushes trusted set to (re)joining site node (UA1)", "status": "pending", "blockedBy": [15, 22] }, + { "id": 24, "subject": "Task 24: Wire ConfigFetchRetryCount into the standby replication fetch (UA2)", "status": "pending", "blockedBy": [11] }, + { "id": 25, "subject": "Task 25: Low-severity code cleanups — S9, S10, C2, C6", "status": "pending", "blockedBy": [9, 17] }, + { "id": 26, "subject": "Task 26: Design-doc updates — C3, C4, C5 + behavior notes for landed changes", "status": "pending", "blockedBy": [1, 2, 4, 7, 8, 11, 13, 15, 18, 21, 23, 24, 25] } + ], + "lastUpdated": "2026-07-08" +} diff --git a/archreview/plans/PLAN-04-data-audit-backbone.md b/archreview/plans/PLAN-04-data-audit-backbone.md new file mode 100644 index 00000000..c9127ec3 --- /dev/null +++ b/archreview/plans/PLAN-04-data-audit-backbone.md @@ -0,0 +1,774 @@ +# Data & Audit Backbone Fixes Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. + +**Goal:** Defuse the scale-dependent operational time bombs in the central data/audit layer (missing site SQLite retention purge, partition-switch timeout self-lock, unindexed KPI scans, table-wide execution-tree scans, hardcoded-zero backlog KPI) and close the audit-integrity/consistency gaps (append-only role drift, frozen SiteCalls progress fields, missing operator-Retry audit rows, reconciliation dead-ends, NodeA-only reconciliation) identified in `archreview/04-data-audit-backbone.md`. + +**Architecture:** All fixes stay inside the existing component boundaries: site-side purge lives in the AuditLog component's site half (SqliteAuditWriter + a new hosted retention job), central maintenance hardening lives in AuditLogRepository/AuditLogPurgeActor, index/DDL changes ride EF Core migrations in ConfigurationDatabase (with idempotent SQL scripts for production), and contract changes (keyset cursor, RequestedBy, SiteEntry fallback endpoint) are strictly additive per the repo's message-evolution rule. Design docs are updated in the same task as the code they describe (repo rule: doc + code + tests travel together). + +**Tech Stack:** C#/.NET 8, Akka.NET (actors/TestKit), EF Core + SQL Server (central), Microsoft.Data.Sqlite (site), xUnit. Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per-project: `dotnet test tests/ --filter `. MSSQL-backed integration tests need `cd infra && docker compose up -d`. **EF migrations gotcha (repo-documented):** always build first and run `dotnet ef migrations add ` WITHOUT `--no-build`, or it scaffolds an empty migration off the stale DLL. + +## Findings Coverage + +| # | Report finding (severity) | Task(s) | +|---|---------------------------|---------| +| S1 | Site SQLite 7-day retention purge missing (High) | 1, 2, 3 | +| S2 | `SwitchOutPartitionAsync` no CommandTimeout / log-only failure (High) | 4, 5 | +| S3 | Append-only DB-role enforcement DDL-only; purger grants insufficient (High) | 6, 7 | +| S4 | SiteCalls upsert freezes RetryCount/LastError at rank (Medium) | 11 | +| S5 | No EF `EnableRetryOnFailure` (Medium) | 12 | +| S6 | Operator Retry on parked notification emits no audit row (+SiteCalls relay identity) (Medium) | 13, 14 | +| S7 | Reconciliation edge cases dead-ended (cursor pin; permanent abandonment) (Medium) | 15, 16, 17 | +| S8 | NotificationOutboxRepository dual SQLite/T-SQL dialect — provider parity (Medium) | 24 (documented convention decision: MSSQL-only SQL + MSSQL fixtures is the norm for new repositories; existing dual-dialect kept as a documented legacy exception — rewriting the shipped repository + its whole test suite is not worth the churn) | +| S9 | Purge actors' first tick waits a full interval (Low) | 22 | +| S10 | `SqliteAuditWriter.Dispose` sync-over-async (Low) | Won't-fix — report itself notes the thread-pool-hop mitigation is correct and documented; flagged-only | +| P1 | SiteCalls KPI predicates full-scan — no `TerminalAtUtc` index (High) | 8 | +| P2 | `GetExecutionTreeAsync` scans entire AuditLog (High) | 9 | +| P3 | AuditLog clustered key leads with random GUID (Medium) | Deferred — report says "worth a deliberate benchmark before the table gets big"; changing it is a full-table rebuild. Task 24 records it as a tracked design decision in Component-ConfigurationDatabase.md so it is no longer untracked | +| P4 | KpiSample volume + unbatched purge (Medium) | 19 (batched purge); per-node sampling cadence recorded as tracked follow-up in 24 | +| P5 | Catch-all `(@p IS NULL OR col=@p)` predicates in SiteCalls query (Medium) | 21 | +| P6 | Outbox offset paging + double query + LIKE; KPI 5-7 round trips; unbounded oldest materialization (Medium) | 20 (KPI single query + bounded oldest); offset→keyset conversion Deferred — it changes the page/TotalCount UI contract for a table whose live-queue portion stays small under Task 19/existing purge; recorded in 24 | +| P7 | Non-persisted `IngestedAtUtc` computed column re-parse (Low) | 23 (predicate-ban comment guard) | +| P8 | `MarkForwardedAsync`/`MarkReconciledAsync` per-id IN lists near SQLite param limit (Low) | 23 | +| C1 | Design docs drifted (enum lists, Teams, per-channel overrides, SiteCalls schema) (Medium) | 24 | +| C2 | `AuditLogRow` entity lives in ConfigurationDatabase not Commons (Medium) | 24 (document the deviation — the rationale in the report is accepted: contract type is external `ZB.MOM.WW.Audit.AuditEvent`, row is a persistence-only projection) | +| C3 | Mixed timestamp CLR types across sibling tables (Low) | 24 (convention note: new tables use `DateTime`+Utc converter) | +| C4 | Uncharted KPI metric names are string literals (Low) | 23 | +| U1 | `backlogTotal` KPI history permanently zero | 10 | +| U2 | Site SQLite retention purge | 1, 2, 3 | +| U3 | Hash-chain tamper evidence / Parquet archival | No action — explicitly deferred to v1.x per Component-AuditLog.md; report confirms "consistent with plan; no drift" | +| U4 | SecuredWrite audit rows leave `SourceNode` NULL; doc under-promises per-channel keys | SourceNode fix Deferred — already a logged follow-up in Component-AuditLog.md:154-157 and the SecuredWrite emission path belongs to plan 07's domain. Doc under-promise fixed in 24 | +| U5 | Reconciliation cursor keyset upgrade untracked | 15, 16 (SiteCalls keyset implemented); AuditLog-pull keyset recorded as tracked follow-up in Component-AuditLog.md (24) | +| U6 | Dispatcher throughput ceiling (sequential delivery) | 25 | +| U7 | Test-coverage gaps mirror findings | Covered inside tasks 1, 4, 10 (the report's three named gaps each get a test) | +| X1 | AuditLog reconciliation dials site NodeA only (flagged in report 08, owned here) | 18 | + +--- + +### Task 1: Site SQLite retention purge — `PurgeExpiredAsync` on the writer + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 4, 6, 8, 11, 12, 19, 20, 21 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Services/ISiteAuditQueue.cs` (add method) +- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriter.cs` (implement; the `auto_vacuum = INCREMENTAL` pragma at ~line 149 already anticipates this) +- Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SqliteAuditWriterRetentionTests.cs` (create) + +1. Write the failing test (pattern-match the fixture setup from `SqliteAuditWriterBacklogStatsTests.cs`): + +```csharp +public class SqliteAuditWriterRetentionTests +{ + [Fact] + public async Task PurgeExpired_DeletesForwardedAndReconciled_OlderThanCutoff() + { + await using var writer = CreateWriter(); // same in-memory/temp-file helper the sibling tests use + var oldForwarded = await WriteEventAsync(writer, occurredAtUtc: DateTime.UtcNow.AddDays(-10)); + var oldReconciled = await WriteEventAsync(writer, occurredAtUtc: DateTime.UtcNow.AddDays(-10)); + var oldPending = await WriteEventAsync(writer, occurredAtUtc: DateTime.UtcNow.AddDays(-10)); + var fresh = await WriteEventAsync(writer, occurredAtUtc: DateTime.UtcNow); + await writer.MarkForwardedAsync(new[] { oldForwarded, oldReconciled, fresh }); + await writer.MarkReconciledAsync(new[] { oldReconciled }); + + var purged = await writer.PurgeExpiredAsync(DateTime.UtcNow.AddDays(-7)); + + Assert.Equal(2, purged); // oldForwarded + oldReconciled + // Hard ForwardState invariant: the old *Pending* row survives on age alone. + var pending = await writer.ReadPendingAsync(10); + Assert.Contains(pending, e => e.EventId == oldPending); + // Fresh forwarded row inside the window survives. + var stats = await writer.GetBacklogStatsAsync(); + Assert.Equal(1, stats.PendingCount); + } + + [Fact] + public async Task PurgeExpired_IsIdempotent() + { + await using var writer = CreateWriter(); + var id = await WriteEventAsync(writer, occurredAtUtc: DateTime.UtcNow.AddDays(-10)); + await writer.MarkForwardedAsync(new[] { id }); + Assert.Equal(1, await writer.PurgeExpiredAsync(DateTime.UtcNow.AddDays(-7))); + Assert.Equal(0, await writer.PurgeExpiredAsync(DateTime.UtcNow.AddDays(-7))); + } +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter SqliteAuditWriterRetentionTests` — expect FAIL (method does not exist). +3. Add to `ISiteAuditQueue`: `Task PurgeExpiredAsync(DateTime olderThanUtc, CancellationToken ct = default);` with XML doc stating the hard invariant (`Pending` rows are never purged on age alone — only `Forwarded`/`Reconciled`). Implement in `SqliteAuditWriter` under `_writeLock` (mirror `MarkForwardedAsync`'s lock + `ObjectDisposedException.ThrowIf` shape), one transaction: + +```sql +CREATE TEMP TABLE IF NOT EXISTS purge_ids AS + SELECT EventId FROM audit_forward_state + WHERE ForwardState IN ('Forwarded','Reconciled') AND OccurredAtUtc < $cutoff; +DELETE FROM audit_forward_state WHERE EventId IN (SELECT EventId FROM purge_ids); +DELETE FROM audit_event WHERE EventId IN (SELECT EventId FROM purge_ids); +DROP TABLE purge_ids; +``` + +Delete the sidecar first (it is the FK child of `audit_event`). `$cutoff` uses the same round-trip `"o"` timestamp format the writer already uses. Return `changes()` from the `audit_event` DELETE. After COMMIT, run `PRAGMA incremental_vacuum;` (the pragma the ctor set up — reference the existing comment at `SqliteAuditWriter.cs:145` and update it: the purge now exists). +4. Run the filter again — expect PASS. Also run the whole site suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter "FullyQualifiedName~Site"` — expect PASS. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Services/ISiteAuditQueue.cs src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriter.cs tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SqliteAuditWriterRetentionTests.cs && git commit -m "feat(audit-log): add site SQLite PurgeExpiredAsync honoring the ForwardState invariant"` + +### Task 2: Site retention options + +**Classification:** small +**Estimated implement time:** ~3 min +**Parallelizable with:** 1, 4, 6, 8, 11, 12, 19, 20, 21 +**Files:** +- Create: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SiteAuditRetentionOptions.cs` +- Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SiteAuditRetentionOptionsTests.cs` (create) + +1. Failing test: + +```csharp +public class SiteAuditRetentionOptionsTests +{ + [Theory] + [InlineData(0, 1)] // clamp floor + [InlineData(7, 7)] // default passthrough + [InlineData(365, 90)] // clamp ceiling (spec: min 1, max 90) + public void ResolvedRetentionDays_Clamps(int configured, int expected) + => Assert.Equal(expected, new SiteAuditRetentionOptions { RetentionDays = configured }.ResolvedRetentionDays); + + [Fact] + public void Defaults_Are_SevenDays_DailyPurge_FiveMinuteInitialDelay() + { + var o = new SiteAuditRetentionOptions(); + Assert.Equal(7, o.ResolvedRetentionDays); + Assert.Equal(TimeSpan.FromHours(24), o.ResolvedPurgeInterval); + Assert.Equal(TimeSpan.FromMinutes(5), o.InitialDelay); + } +} +``` + +2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter SiteAuditRetentionOptionsTests` — FAIL. +3. Implement the options class following the `SiteCallAuditOptions.ResolvedReconciliationInterval` clamp pattern: `RetentionDays` (default 7; `ResolvedRetentionDays` clamps to `[1, 90]`), `PurgeInterval` (default 24h; `ResolvedPurgeInterval` clamps to min 1 minute — Akka zero-interval spin footgun), `InitialDelay` (default 5 min — short so a daily-restarting node still purges). Config section: `ScadaBridge:AuditLog:SiteRetention`. +4. Run filter — PASS. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SiteAuditRetentionOptions.cs tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SiteAuditRetentionOptionsTests.cs && git commit -m "feat(audit-log): SiteAuditRetentionOptions (7-day default, clamped 1-90)"` + +### Task 3: Site retention hosted job + registration + design doc + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 4, 6, 8, 11, 12, 19, 20, 21 +**Files:** +- Create: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SiteAuditRetentionService.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/ServiceCollectionExtensions.cs` (site registration path, near the `ISiteAuditQueue` registration at ~line 111) +- Modify: `docs/requirements/Component-AuditLog.md` (~lines 412-414, the "Sites: daily site job" bullet) +- Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SiteAuditRetentionServiceTests.cs` (create) + +1. Failing test (fake `ISiteAuditQueue`, short intervals): + +```csharp +public class SiteAuditRetentionServiceTests +{ + [Fact] + public async Task Tick_Purges_With_RetentionCutoff() + { + var queue = new RecordingSiteAuditQueue(); // stub: records PurgeExpiredAsync calls + var options = Options.Create(new SiteAuditRetentionOptions + { RetentionDays = 7, PurgeInterval = TimeSpan.FromMilliseconds(50), InitialDelay = TimeSpan.Zero }); + using var svc = new SiteAuditRetentionService(queue, options, NullLogger.Instance); + await svc.StartAsync(CancellationToken.None); + await WaitUntilAsync(() => queue.PurgeCalls.Count >= 1, TimeSpan.FromSeconds(5)); + await svc.StopAsync(CancellationToken.None); + var cutoff = queue.PurgeCalls[0]; + Assert.InRange(cutoff, DateTime.UtcNow.AddDays(-7).AddMinutes(-1), DateTime.UtcNow.AddDays(-7).AddMinutes(1)); + } + + [Fact] + public async Task Tick_SwallowsExceptions_AndKeepsTicking() + { + var queue = new RecordingSiteAuditQueue { ThrowOnFirstCall = true }; + // ... start, wait for >= 2 calls, assert no unobserved exception + } +} +``` + +2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter SiteAuditRetentionServiceTests` — FAIL. +3. Implement `SiteAuditRetentionService : IHostedService, IDisposable` mirroring `SiteAuditBacklogReporter.cs` (same file layout: `System.Threading.Timer`, `InitialDelay` as due time, `ResolvedPurgeInterval` as period, per-tick try/catch that logs and continues, purge count logged at Information). Register in the site path of `ServiceCollectionExtensions`: bind `SiteAuditRetentionOptions` from `ScadaBridge:AuditLog:SiteRetention` and `services.AddHostedService();`. +4. Run filter — PASS. Then `dotnet build ZB.MOM.WW.ScadaBridge.slnx` — expect clean. +5. Update `Component-AuditLog.md` ~412-414: replace the aspirational bullet with the shipped mechanism — `SiteAuditRetentionService` hosted job, config keys (`ScadaBridge:AuditLog:SiteRetention:RetentionDays|PurgeInterval`), the `PurgeExpiredAsync` invariant, and the post-purge `incremental_vacuum`. +6. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.AuditLog tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site docs/requirements/Component-AuditLog.md && git commit -m "feat(audit-log): daily site SQLite retention purge job (closes unbounded site DB growth)"` + +### Task 4: Maintenance command timeout on partition switch + per-channel purge + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 3, 8 (different files), 11, 20 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeOptions.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IAuditLogRepository.cs` (additive optional parameter) +- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/AuditLogRepository.cs` (`SwitchOutPartitionAsync` ~257-371, `PurgeChannelOlderThanAsync` ~374-445) +- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeActor.cs` (pass the timeout) +- Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/AuditLogPurgeActorTests.cs` (extend), `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/PartitionPurgeTests.cs` (extend, MSSQL) + +1. Failing unit test in `AuditLogPurgeActorTests` (these tests already drive the actor against a mocked `IAuditLogRepository`): assert the actor calls `SwitchOutPartitionAsync(boundary, commandTimeout)` with the options value: + +```csharp +[Fact] +public async Task PurgeTick_Passes_MaintenanceCommandTimeout_To_Repository() +{ + // options: MaintenanceCommandTimeoutMinutes = 45 + // mock repo: capture the TimeSpan? argument + // drive one tick; Assert.Equal(TimeSpan.FromMinutes(45), captured); +} +``` + +Plus an options test: `MaintenanceCommandTimeoutMinutes` default 30, clamped to min 1 (`ResolvedMaintenanceCommandTimeout`). +2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter AuditLogPurgeActorTests` — FAIL (no such parameter). +3. Implementation: + - `AuditLogPurgeOptions`: add `public int MaintenanceCommandTimeoutMinutes { get; set; } = 30;` + `ResolvedMaintenanceCommandTimeout` clamp (min 1 minute), XML-doc'd against the 30s-default-timeout failure mode from the review. + - `IAuditLogRepository`: `Task SwitchOutPartitionAsync(DateTime monthBoundary, TimeSpan? commandTimeout = null, CancellationToken ct = default);` and `PurgeChannelOlderThanAsync(..., TimeSpan? commandTimeout = null, ...)` — additive defaults, existing callers compile unchanged. + - `AuditLogRepository.SwitchOutPartitionAsync`: when `commandTimeout` is non-null, set `sampleCmd.CommandTimeout = (int)commandTimeout.Value.TotalSeconds;` and wrap the `ExecuteSqlRawAsync` with `_context.Database.SetCommandTimeout(commandTimeout)` (the context is scoped per purge tick — see `AuditLogPurgeActor.OnTickAsync`'s `CreateAsyncScope` — so no restore is needed, but restore the previous value in a `finally` anyway for hygiene). Same `cmd.CommandTimeout` treatment on the per-channel `DELETE TOP` command. + - `AuditLogPurgeActor.OnTickAsync`: pass `_purgeOptions.ResolvedMaintenanceCommandTimeout` to both repository calls. +4. Run the actor filter — PASS. Extend `PartitionPurgeTests` (MSSQL; `cd infra && docker compose up -d` first) with one case that calls `SwitchOutPartitionAsync(boundary, TimeSpan.FromMinutes(30))` and asserts success — proving the timeout path executes against real SQL Server: `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter PartitionPurgeTests`. +5. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.AuditLog src/ZB.MOM.WW.ScadaBridge.Commons src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests && git commit -m "fix(audit-log): explicit long CommandTimeout on partition-switch and per-channel purge maintenance paths"` + +### Task 5: Purge-failure health event (not just a log line) + +**Classification:** standard +**Estimated implement time:** ~4 min +**Parallelizable with:** 8, 11, 19, 20, 21 +**Files:** +- Create: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeFailedEvent.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeActor.cs` (catch block ~198-206 and the boundary-enumeration catch) +- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditCentralHealthSnapshot.cs` + `IAuditCentralHealthSnapshot.cs` (add `PurgeFailures` counter, mirroring the existing counter members) +- Modify: `docs/requirements/Component-AuditLog.md` (retention section: purge failure is now a health metric) +- Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/AuditLogPurgeActorTests.cs` (extend) + +1. Failing test: mock repository whose `SwitchOutPartitionAsync` throws; subscribe a TestProbe to `AuditLogPurgeFailedEvent` on the EventStream (same pattern the existing tests use for `AuditLogPurgedEvent`); drive a tick; expect one `AuditLogPurgeFailedEvent(boundary, errorMessage, elapsedMs)` and assert the health snapshot's `PurgeFailures` incremented. +2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter AuditLogPurgeActorTests` — FAIL. +3. Implement: `public sealed record AuditLogPurgeFailedEvent(DateTime MonthBoundary, string Error, long ElapsedMilliseconds);` — publish it in the per-boundary catch (alongside the existing `LogError`) using the already-captured `eventStream`; increment the new snapshot counter (inject `IAuditCentralHealthSnapshot` the same way sibling actors take their counters — follow the existing ctor DI shape). The success event (`AuditLogPurgedEvent`) already exists; this is its symmetric failure twin. +4. Run filter — PASS. +5. Doc: add to Component-AuditLog.md retention section: "Purge failure publishes `AuditLogPurgeFailedEvent` and increments the `PurgeFailures` central health counter — a silently failing retention job is a health-surface condition, not just a log line." +6. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.AuditLog tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central docs/requirements/Component-AuditLog.md && git commit -m "feat(audit-log): surface partition-purge failure as health event + counter"` + +### Task 6: Fix purger role grants (migration) + +**Classification:** high-risk +**Estimated implement time:** ~4 min +**Parallelizable with:** 1, 2, 3, 11, 20 (NOT with Task 8 — both add migrations, serialize on the model snapshot) +**Files:** +- Create: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/_FixAuditPurgerRoleGrants.cs` (scaffolded) +- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Migrations/` (add `FixAuditPurgerRoleGrantsTests.cs`), plus re-run `AuditLogAppendOnlyGuardTests` + +1. Failing test first — a migration-content guard in the style of the existing migration tests: assert the migration source contains `GRANT CREATE TABLE TO scadabridge_audit_purger` and `GRANT DELETE ON dbo.AuditLog TO scadabridge_audit_purger`. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter FixAuditPurgerRoleGrants` — FAIL (file absent). +2. Scaffold (repo gotcha — build first, NO `--no-build`): + +```bash +dotnet build ZB.MOM.WW.ScadaBridge.slnx +cd src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase +dotnet ef migrations add FixAuditPurgerRoleGrants +``` + +Verify the scaffold is model-empty (no schema change) and add the SQL by hand in `Up`: + +```csharp +migrationBuilder.Sql(@" +IF DATABASE_PRINCIPAL_ID('scadabridge_audit_purger') IS NULL + EXEC sp_executesql N'CREATE ROLE scadabridge_audit_purger'; +-- The switch-out dance CREATEs a staging table; ALTER ON SCHEMA::dbo alone does not +-- confer the database-level CREATE TABLE permission (review finding S3). +GRANT CREATE TABLE TO scadabridge_audit_purger; +-- The per-channel retention override (PurgeChannelOlderThanAsync) is a bounded row +-- DELETE on the maintenance path; a segregated purger principal needs this grant. +GRANT DELETE ON dbo.AuditLog TO scadabridge_audit_purger;"); // AUDIT-PURGE-ALLOWED: role grant for the maintenance principal +``` + +`Down`: `REVOKE` both grants (guarded on principal existence). +3. Run the new test — PASS. Then run `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter AuditLogAppendOnlyGuardTests` — if the CI grep guard trips on the `GRANT DELETE ON dbo.AuditLog` text, extend the guard's allowlist to permit `GRANT`/`DENY`/`REVOKE` statements (they are permission DDL, not data mutation) — keep the guard strict for `DELETE FROM`/`UPDATE `. +4. Production script step (repo convention — manual SQL for production): `dotnet ef migrations script AddPendingDeployment FixAuditPurgerRoleGrants --idempotent --output ../../docs/plans/sql/FixAuditPurgerRoleGrants.sql` (adjust the from-migration to the actual latest; review the output). +5. MSSQL sanity: with infra up, `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter Migrations` — expect PASS. +6. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests docs/plans/sql && git commit -m "fix(config-db): grant CREATE TABLE + scoped DELETE to scadabridge_audit_purger so a segregated maintenance principal can actually run the purge"` + +### Task 7: Honest append-only enforcement story (docs) + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** everything except 24 (same doc files) +**Files:** +- Modify: `docs/requirements/Component-AuditLog.md` (~417-424, "Security & Tamper-Evidence") +- Modify: `docs/requirements/Component-ConfigurationDatabase.md` (~399-402, AuditLog Table Purge section) + +1. Rewrite Component-AuditLog.md §Security "Append-only enforcement" to state reality (review finding S3): the application runs with **one** connection principal for both writer and maintenance paths, so in the default deployment append-only is enforced by (a) the CI grep guard (`AuditLogAppendOnlyGuardTests`) with its single marked DELETE exemption, and (b) code review. The `scadabridge_audit_writer` / `scadabridge_audit_purger` roles are **optional DBA hardening**: a deployment that wants database-level enforcement must provision two logins (runtime → writer role; a maintenance job/second connection → purger role) — and note the purger role now carries `CREATE TABLE` + scoped `DELETE` (Task 6) so it can genuinely execute the switch-out and per-channel purge. Remove the false claim "row-level DELETE is not granted even to purge" (stale since `PerChannelRetentionDays` shipped). +2. In Component-ConfigurationDatabase.md fix the same passage's stale "single global value in v1, no per-channel overrides" claim while in the file (coordinates with Task 24 — do this bullet here since it is in the same paragraph; Task 24 skips it if already fixed). +3. Verify no other doc contradicts: `grep -rn "audit_writer\|audit_purger" docs/`. +4. Commit: `git add docs/requirements/Component-AuditLog.md docs/requirements/Component-ConfigurationDatabase.md && git commit -m "docs(audit-log): document append-only enforcement honestly (CI guard default; DB roles are optional DBA hardening)"` + +### Task 8: SiteCalls filtered non-terminal index (migration) + +**Classification:** high-risk +**Estimated implement time:** ~4 min +**Parallelizable with:** 1, 2, 3, 5, 11 (NOT with Task 6 — migration serialization) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/SiteCallEntityTypeConfiguration.cs` (~66-88; also fix the now-stale "No index — … per-site, not per-node" SourceNode comment) +- Create: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/_AddSiteCallsNonTerminalIndex.cs` (scaffolded) +- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/SchemaConfigurationTests.cs` (extend) + +1. Failing model test (SchemaConfigurationTests already asserts index shapes): + +```csharp +[Fact] +public void SiteCalls_Has_Filtered_NonTerminal_Index() +{ + var entity = Model.FindEntityType(typeof(SiteCall))!; + var index = entity.GetIndexes().Single(i => i.GetDatabaseName() == "IX_SiteCalls_NonTerminal"); + Assert.Equal(new[] { nameof(SiteCall.CreatedAtUtc) }, index.Properties.Select(p => p.Name)); + Assert.Equal("[TerminalAtUtc] IS NULL", index.GetFilter()); + Assert.Equal(new[] { "SourceSite", "SourceNode", "Status" }, index.GetIncludeProperties()); +} +``` + +2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter SiteCalls_Has_Filtered_NonTerminal_Index` — FAIL. +3. Add to `SiteCallEntityTypeConfiguration.Configure`: + +```csharp +// NonTerminal backs every "live queue" KPI predicate (TerminalAtUtc IS NULL): +// buffered/stuck/oldest counts run every 60 s from the KPI recorder plus every +// 10 s Health-dashboard poll. Filtered: the non-terminal population is the live +// queue, not the 365-day archive — each count becomes a small seek instead of a +// clustered scan (arch-review 04, P1). +builder.HasIndex(s => s.CreatedAtUtc) + .HasDatabaseName("IX_SiteCalls_NonTerminal") + .HasFilter("[TerminalAtUtc] IS NULL") + .IncludeProperties(nameof(SiteCall.SourceSite), nameof(SiteCall.SourceNode), nameof(SiteCall.Status)); +``` + +4. Scaffold the migration (build first, NO `--no-build`): `dotnet build ZB.MOM.WW.ScadaBridge.slnx && cd src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase && dotnet ef migrations add AddSiteCallsNonTerminalIndex` — verify the scaffold contains the `CreateIndex` with filter (if it is empty, the build/`--no-build` gotcha bit — delete and redo). +5. Run the model test — PASS. With infra up, run the MSSQL-backed repo suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter SiteCallAuditRepositoryTests` — PASS. +6. Production script: `dotnet ef migrations script FixAuditPurgerRoleGrants AddSiteCallsNonTerminalIndex --idempotent --output ../../docs/plans/sql/AddSiteCallsNonTerminalIndex.sql`. +7. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests docs/plans/sql && git commit -m "perf(site-call-audit): filtered IX_SiteCalls_NonTerminal index — KPI predicates seek the live queue instead of scanning 365 days"` + +### Task 9: Bound the execution-tree edge scan with a root time window + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 3, 11, 13, 18, 19 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/AuditLogRepository.cs` (`GetExecutionTreeAsync`, ~632-790) +- Modify: `docs/requirements/Component-AuditLog.md` (audit-tree section: document the 7-day traversal bound) +- Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/ExecutionIdCorrelationTests.cs` or the existing execution-tree test file (extend; MSSQL) + +1. Failing test (add to the existing MSSQL execution-tree suite — locate with `grep -rln "GetExecutionTreeAsync" tests/`): + +```csharp +[Fact] +public async Task ExecutionTree_Bounds_EdgeScan_To_Root_Window() +{ + // Root execution at T0, child at T0+5min → both returned. + // Unrelated execution pair at T0+30 days sharing no ancestry → NOT returned (was never returned; + // this asserts correctness is preserved), AND a genuine descendant stamped at T0+30 days + // (beyond the 7-day traversal bound) is documented-excluded: assert it is absent. +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter ExecutionTree_Bounds_EdgeScan` — FAIL (descendant beyond bound is currently returned). +3. Implementation in `GetExecutionTreeAsync`, between Phase 1 and Phase 2: + - New constants: `private static readonly TimeSpan ExecutionTreeWindowSlack = TimeSpan.FromHours(1);` and `private static readonly TimeSpan ExecutionTreeMaxSpan = TimeSpan.FromDays(7);` with a comment: execution trees span minutes, not years; the window enables partition elimination on the Edges anchor (arch-review 04, P2). + - Query `SELECT MIN(OccurredAtUtc) FROM dbo.AuditLog WHERE ExecutionId = @root;` (a seek on `IX_AuditLog_Execution`). + - If non-NULL, parameterize the Edges CTE anchor: `WHERE ExecutionId IS NOT NULL AND OccurredAtUtc >= @winStart AND OccurredAtUtc < @winEnd` with `@winStart = rootFirst - slack`, `@winEnd = rootFirst + maxSpan`. If NULL (row-less stub root), keep the unbounded form (current behavior) — correctness over speed for the degenerate case. +4. Run the execution-tree filter plus the full existing suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter "FullyQualifiedName~ExecutionTree|FullyQualifiedName~ExecutionId"` — PASS. +5. Doc: Component-AuditLog.md audit-tree section — "tree traversal is bounded to a 7-day window anchored at the root's first event (configurable constant); descendants beyond the window are excluded by design." +6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/AuditLogRepository.cs docs/requirements/Component-AuditLog.md tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests && git commit -m "perf(audit-log): bound GetExecutionTreeAsync edge scan to a root-anchored time window (partition elimination)"` + +### Task 10: Fix the hardcoded-zero backlogTotal KPI history + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 3, 6, 8, 11, 19, 21 +**Files:** +- Create: `src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Kpi/IAuditBacklogProvider.cs` +- Create: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/Kpi/CentralHealthAuditBacklogProvider.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ServiceCollectionExtensions.cs` (central registration; locate the file — `grep -n AddSingleton src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ServiceCollectionExtensions.cs`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Kpi/AuditLogKpiSampleSource.cs` +- Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Kpi/AuditLogKpiSampleSourceTests.cs` (extend/create), `tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests` (provider test) + +1. Failing test: + +```csharp +[Fact] +public async Task Collect_Uses_BacklogProvider_When_Present() +{ + var repo = new FakeAuditLogRepository(new AuditLogKpiSnapshot(10, 1, BacklogTotal: 0, DateTime.UtcNow)); + var source = new AuditLogKpiSampleSource(repo, new FakeBacklogProvider(42)); + var samples = await source.CollectAsync(DateTime.UtcNow); + Assert.Equal(42, samples.Single(s => s.Metric == KpiMetrics.AuditLog.BacklogTotal).Value); +} + +[Fact] +public async Task Collect_FallsBack_To_Snapshot_When_Provider_Absent() +{ + // ctor with provider: null → BacklogTotal sample = snapshot value (0) — unchanged behavior +} +``` + +2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter AuditLogKpiSampleSource` — FAIL. +3. Implementation: + - Commons: `public interface IAuditBacklogProvider { long GetPendingBacklogTotal(); }` (XML doc: sums latest per-site `SiteAuditBacklog.PendingCount` health reports; best-effort — sites not yet reporting contribute zero). + - HealthMonitoring: `CentralHealthAuditBacklogProvider(ICentralHealthAggregator)` — port the exact summation from `CentralUI/Services/AuditLogQueryService.GetKpiSnapshotAsync` (~lines 145-155: `state.LatestReport?.SiteAuditBacklog?.PendingCount`, `> 0` guard). Register `services.AddSingleton();` in the central HealthMonitoring registration. + - `AuditLogKpiSampleSource`: ctor gains `IAuditBacklogProvider? backlogProvider = null` (MS.DI resolves the default when unregistered — sites/tests unaffected); backlog sample value becomes `_backlogProvider?.GetPendingBacklogTotal() ?? snapshot.BacklogTotal`. + - Optionally refactor `AuditLogQueryService` to consume the provider (skip if it drags UI DI churn — the live tile already works; the *history* was the broken surface). +4. Run filter — PASS. Build the solution. +5. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.Commons src/ZB.MOM.WW.ScadaBridge.HealthMonitoring src/ZB.MOM.WW.ScadaBridge.AuditLog tests && git commit -m "fix(kpi-history): backlogTotal trend records the real site-backlog aggregate instead of a hardwired zero"` + +### Task 11: SiteCalls same-rank freshness upsert (unfreeze RetryCount/LastError) + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 3, 4, 8, 9, 10 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs` (rank comment ~27-40; UPDATE predicate ~117-139) +- Modify: `docs/requirements/Component-SiteCallAudit.md` (~76-78 upsert contract wording) +- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/SiteCallAuditRepositoryTests.cs` (extend; MSSQL — infra up) + +1. Failing tests: + +```csharp +[Fact] +public async Task Upsert_SameRank_NewerUpdatedAt_RefreshesProgressFields() +{ + var id = NewId(); + await Repo.UpsertAsync(Call(id, "Attempted", retryCount: 1, lastError: "timeout #1", updatedAt: T0)); + await Repo.UpsertAsync(Call(id, "Attempted", retryCount: 5, lastError: "timeout #5", updatedAt: T0.AddMinutes(5))); + var row = await Repo.GetAsync(id); + Assert.Equal(5, row!.RetryCount); + Assert.Equal("timeout #5", row.LastError); +} + +[Fact] +public async Task Upsert_SameRank_OlderUpdatedAt_IsNoOp() { /* retry 5 then out-of-order retry 2 with older stamp → stays 5 */ } + +[Fact] +public async Task Upsert_LowerRank_IsStillNoOp() { /* Delivered then Attempted (newer stamp) → stays Delivered */ } +``` + +2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter SiteCallAuditRepositoryTests` — FAIL (same-rank is a no-op today). +3. Change the UPDATE guard from `incomingRank > CASE...` to: + +```sql +AND ( {incomingRank} > (CASE Status ... END) + OR ({incomingRank} = (CASE Status ... END) AND UpdatedAtUtc < {siteCall.UpdatedAtUtc}) ) +``` + +(repeat the CASE expression — both branches parameterize identically). Update the class comment: "monotonic on rank; within equal rank, newest `UpdatedAtUtc` wins (still idempotent — equal stamps are a no-op — and still regression-proof)". This unfreezes `RetryCount`/`LastError`/`HttpStatus` during the Attempted phase and lets a genuinely-newer terminal correction land. +4. Run filter — PASS (all pre-existing monotonic tests must stay green). +5. Doc: Component-SiteCallAudit.md upsert paragraph — replace "first-write-wins at each rank" semantics with "insert-if-not-exists, then upsert on newer status rank, with a newest-`UpdatedAtUtc` tiebreaker within equal rank so retrying calls show live RetryCount/LastError". +6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs docs/requirements/Component-SiteCallAudit.md tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests && git commit -m "fix(site-call-audit): same-rank freshness tiebreaker — retrying calls no longer freeze RetryCount/LastError at first write"` + +### Task 12: EF Core connection resiliency (`EnableRetryOnFailure`) + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 3, 8, 11 (do NOT run concurrently with any task executing MSSQL integration tests — behavior-wide change) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ServiceCollectionExtensions.cs` (~29-34) +- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogIngestActor.cs` (`OnCachedTelemetryAsync` dual-write transaction ~262-311) +- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/ServiceCollectionExtensionsTests.cs` (extend), `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/CombinedTelemetryIdempotencyTests.cs` (must stay green) + +1. First inventory every user-initiated transaction against `ScadaBridgeDbContext`: `grep -rn "BeginTransaction" src/ --include="*.cs"` — expect `AuditLogIngestActor.OnCachedTelemetryAsync`; treat any others found identically. +2. Failing test in `ServiceCollectionExtensionsTests`: + +```csharp +[Fact] +public void AddConfigurationDatabase_Configures_RetryingExecutionStrategy() +{ + var services = new ServiceCollection(); + services.AddConfigurationDatabase("Server=unused;Database=unused;"); + using var sp = services.BuildServiceProvider(); + var options = sp.GetRequiredService>(); + var sqlExt = options.Extensions.OfType().Single(); + Assert.NotNull(sqlExt.ExecutionStrategyFactory); // retry strategy configured +} +``` + +3. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter Configures_RetryingExecutionStrategy` — FAIL. +4. Implementation: + - `options.UseSqlServer(connectionString, sql => sql.EnableRetryOnFailure(maxRetryCount: 5, maxRetryDelay: TimeSpan.FromSeconds(30), errorNumbersToAdd: null))` — comment: covers every read-side path (KPI asks, outbox queries, `Notify.Status`, execution trees) that previously surfaced transient faults raw (arch-review 04, S5). + - `OnCachedTelemetryAsync`: wrap the per-entry `BeginTransactionAsync` block in the execution strategy — `var strategy = context.Database.CreateExecutionStrategy(); await strategy.ExecuteAsync(async () => { await using var tx = await context.Database.BeginTransactionAsync(); ... await tx.CommitAsync(); });` — the block is already idempotent (insert-if-not-exists + monotonic upsert), so a retried unit is safe; say so in the comment. + - Note in `SwitchOutPartitionAsync`'s comment: the raw batch carries its own server-side BEGIN TRAN/CATCH and is IF-EXISTS-guarded idempotent, so strategy-level replay of the whole batch is safe. +5. Run: the new filter → PASS; then with infra up, `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter "CombinedTelemetry"` → PASS (proves the dual-write still commits atomically under the strategy). +6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ServiceCollectionExtensions.cs src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogIngestActor.cs tests && git commit -m "fix(config-db): EnableRetryOnFailure fleet-wide + execution-strategy wrap of the combined-telemetry dual-write"` + +### Task 13: Audit row for operator Retry on parked notifications + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 8, 9, 11, 18, 19, 21 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Notification/NotificationOutboxQueries.cs` (`RetryNotificationRequest` ~54, `DiscardNotificationRequest` ~69 — additive `string? RequestedBy = null`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/NotificationOutboxActor.cs` (`RetryAsync` ~951-977; `BuildNotifyDeliverEvent` ~675-714 gains `string? actorOverride = null`; `EmitTerminalAuditAsync` passes it for discard) +- Modify: callers that send these requests — locate with `grep -rln "RetryNotificationRequest(" src/` (expect the Central UI outbox service and/or ManagementActor) and pass the authenticated username +- Modify: `docs/requirements/Component-NotificationOutbox.md` (~128) +- Test: `tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests/NotificationOutboxActorRetryEmissionTests.cs` (create; clone the audit-writer capture rig from `NotificationOutboxActorTerminalEmissionTests.cs`) + +1. Failing test: + +```csharp +[Fact] +public async Task Retry_OnParked_Emits_Submitted_NotifyDeliver_AuditRow_With_Operator() +{ + // seed a Parked notification; send RetryNotificationRequest(corr, id, RequestedBy: "jdoe") + // expect RetryNotificationResponse Success + // assert captured audit writer received exactly one event: + // Kind NotifyDeliver, Status Submitted, Actor "jdoe", CorrelationId == NotificationId +} + +[Fact] +public async Task Retry_AuditFailure_DoesNotFail_TheRetry() { /* throwing audit writer → response still Success (audit is best-effort) */ } +``` + +2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests --filter RetryEmission` — FAIL. +3. Implementation: + - Add `string? RequestedBy = null` to both request records (additive-only contract evolution — existing senders compile unchanged). + - `BuildNotifyDeliverEvent(..., string? actorOverride = null)`: `actor: actorOverride ?? SystemActor`. + - In `RetryAsync`, after `repository.UpdateAsync(notification)`: emit `BuildNotifyDeliverEvent(notification, now, AuditStatus.Submitted, errorMessage: null, actorOverride: request.RequestedBy)` via `_auditWriter.WriteAsync` inside the same try/catch-and-log shape as `EmitTerminalAuditAsync` (audit failure never aborts the retry). `Submitted` = "operator re-queued it" — the forensic gap the review names (`Parked → Attempted → Delivered` with no record of who un-parked). + - `DiscardAsync`: pass `request.RequestedBy` through to the terminal row's actor. + - Plumb the username at the senders (the UI/management layer knows the authenticated user; pass `null` where no identity exists). +4. Run filter + `dotnet test tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests` — PASS. +5. Doc: Component-NotificationOutbox.md ~128 — "…each transition emits the corresponding audit row: operator **Retry** emits a `Notification`-channel `NotifyDeliver` row with status `Submitted` and the operator as `Actor`; Discard emits the `Terminal` row (operator as `Actor`)." +6. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.Commons src/ZB.MOM.WW.ScadaBridge.NotificationOutbox src/ZB.MOM.WW.ScadaBridge.CentralUI src/ZB.MOM.WW.ScadaBridge.ManagementService docs/requirements/Component-NotificationOutbox.md tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests && git commit -m "feat(notification-outbox): operator Retry/Discard emit audit rows with operator identity"` + +### Task 14: Operator identity on the SiteCalls Retry/Discard relay + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 8, 9, 10, 19, 20 +**Files:** +- Modify: the relay request records — locate with `grep -rn "record RetryParkedOperation\|record DiscardParkedOperation" src/ZB.MOM.WW.ScadaBridge.Commons` (additive `string? RequestedBy = null`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/SiteCallAuditActor.cs` (relay handlers — see `SiteCallRelayTests.cs` for the handler names) +- Modify: `docs/requirements/Component-SiteCallAudit.md` (relay section) +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests/SiteCallRelayTests.cs` (extend) + +1. Failing test: relay a Retry for a parked row with `RequestedBy: "jdoe"`; assert the injected central audit writer (same `ICentralAuditWriter` seam `NotificationOutboxActor` uses — confirm with `grep -rn "ICentralAuditWriter" src/ZB.MOM.WW.ScadaBridge.NotificationOutbox src/ZB.MOM.WW.ScadaBridge.Commons`) received one event: `Kind = CachedResolve`, `Status = Submitted`, `Actor = "jdoe"`, `CorrelationId = TrackedOperationId`, channel taken from the stored `SiteCalls` row's `Channel`. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests --filter SiteCallRelayTests` — FAIL. +2. Implementation: inject `ICentralAuditWriter` into `SiteCallAuditActor` (optional ctor param defaulting to null → no-op, so existing tests construct unchanged); in the Retry/Discard relay handlers, after the site relay succeeds, fetch the row (`ISiteCallAuditRepository.GetAsync`) for its `Channel`, build the event via `ScadaBridgeAuditEventFactory.Create` (mirror `BuildNotifyDeliverEvent`'s shape), write best-effort (try/catch + log — the relay outcome is authoritative, audit never aborts it). Site-side telemetry still records the state change itself; this row adds *who asked*. +3. Run filter — PASS. +4. Doc: Component-SiteCallAudit.md relay section — the relay now emits a central direct-write audit row carrying operator identity; sites remain the source of truth for the state change. +5. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.Commons src/ZB.MOM.WW.ScadaBridge.SiteCallAudit docs/requirements/Component-SiteCallAudit.md tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests && git commit -m "feat(site-call-audit): relay Retry/Discard emit operator-identity audit rows"` + +### Task 15: Composite keyset in the SiteCalls reconciliation pull contract (site side) + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 3, 8, 13 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto` (additive `string after_id` on the SiteCalls pull request message — locate exact message with `grep -n "PullSiteCalls" src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto`) +- Modify: the Commons pull-request message (`grep -rn "record CachedCallReconcileRequest\|PullSiteCallsRequest" src/ZB.MOM.WW.ScadaBridge.Commons`) — additive `string? AfterId = null` +- Modify: the site-side pull handler + tracking-store query (`grep -rn "ReadSince\|PullSiteCalls" src/ZB.MOM.WW.ScadaBridge.StoreAndForward src/ZB.MOM.WW.ScadaBridge.Communication --include="*.cs"`) — order by `(UpdatedAtUtc, TrackedOperationId)`, filter `UpdatedAtUtc > since OR (UpdatedAtUtc = since AND TrackedOperationId > afterId)` (when `afterId` present; absent → legacy `>= since` behavior) +- Test: site-side handler/store tests (same project as the handler; locate via the greps above) + +1. Failing test on the site store/handler: seed `batchSize + 2` rows sharing one `UpdatedAtUtc`; pull with `since = thatTimestamp, afterId = `; assert page 2 returns the remaining rows (today it returns the same page). Run the owning test project with `--filter ` — FAIL. +2. Implement the additive proto field (field number = next free; additive-only rule), regenerate/compile, map it through the DTO, and apply the keyset predicate + deterministic `(UpdatedAtUtc, TrackedOperationId)` ordering in the store query. Absent `AfterId` preserves the exact legacy contract (old central against new site: unchanged). +3. Run the owning project's full suite — PASS. +4. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.Communication src/ZB.MOM.WW.ScadaBridge.Commons src/ZB.MOM.WW.ScadaBridge.StoreAndForward tests && git commit -m "feat(site-call-audit): additive (UpdatedAtUtc, TrackedOperationId) keyset in the reconciliation pull contract"` + +### Task 16: Central keyset cursor — un-pin the SiteCalls reconciliation loop + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** 5, 8, 9, 10 (requires Task 15) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/SiteCallAuditActor.cs` (reconciliation loop ~600-670: cursor becomes `(DateTime, string?)`; pin branch ~636-654) +- Create: `src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/SiteCallReconciliationPinnedChanged.cs` (EventStream record, mirroring `SiteAuditTelemetryStalledChanged`) +- Modify: `docs/requirements/Component-SiteCallAudit.md` (reconciliation cursor paragraph) +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests/SiteCallAuditReconciliationTests.cs` (extend) + +1. Failing test: fake pull client honoring the keyset; seed `2 × batchSize` rows sharing one `UpdatedAtUtc`; drive one tick; assert **all** rows were upserted (today the tail never reconciles). Second test: a fake *legacy* client that ignores `AfterId` (returns the same page) → assert the actor publishes `SiteCallReconciliationPinnedChanged(siteId, pinned: true)` on the EventStream instead of silently logging. `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests --filter Reconciliation` — FAIL. +2. Implement: per-site cursor `(DateTime since, string? afterId)`; after each page set `afterId` to the max `(UpdatedAtUtc, TrackedOperationId)` row's id; pass it on the next pull. Keep the no-progress detection as the legacy-site fallback, but publish the new event (and clear it with `pinned: false` when progress resumes) — the dead-end becomes a health-observable condition. +3. Run filter — PASS; run the full SiteCallAudit.Tests suite — PASS. +4. Doc: Component-SiteCallAudit.md — cursor is now a composite `(UpdatedAtUtc, TrackedOperationId)` keyset (additive contract field; legacy sites fall back to the timestamp cursor and surface `SiteCallReconciliationPinnedChanged` when pinned). Also add one line to Component-AuditLog.md's reconciliation section: the analogous keyset for the `PullAuditEvents` cursor is a tracked follow-up (same shape; lower urgency because the audit cursor already re-pulls idempotently). +5. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.SiteCallAudit docs/requirements tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests && git commit -m "fix(site-call-audit): composite keyset cursor eliminates the single-timestamp reconciliation pin; pinned state now a published event"` + +### Task 17: Durable record for permanently-abandoned reconciliation rows + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 8, 11, 19, 20, 21 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Types/Enums/AuditKind.cs` (additive `ReconciliationAbandoned` value) +- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/SiteAuditReconciliationActor.cs` (abandonment branch ~283-293) +- Modify: `docs/requirements/Component-AuditLog.md` (kind vocabulary + reconciliation section) +- Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/SiteAuditReconciliationActorTests.cs` (extend) + +1. Failing test: repository stub whose `InsertIfNotExistsAsync` throws for one specific EventId across `MaxPermanentInsertAttempts` ticks but accepts everything else; after the abandonment tick, assert a **synthetic** audit event was inserted (new EventId; `Kind = ReconciliationAbandoned`; `DetailsJson`/Extra carrying the abandoned `EventId`, the site id, and the final error) and the cursor advanced. `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter SiteAuditReconciliationActorTests` — FAIL. +2. Implement in the `attempts >= MaxPermanentInsertAttempts` branch: alongside the existing `LogCritical`, build a minimal synthetic event (fresh `Guid.NewGuid()` EventId, `OccurredAtUtc = now`, Actor = system, Action `"Audit.ReconciliationAbandoned"`, Category preserved from the lost event where parseable, Extra = `{ abandonedEventId, sourceSiteId, error }` — use the same factory/projection helpers the ingest path uses) and `InsertIfNotExistsAsync` it in its own try/catch (the synthetic row is small and well-formed, so the row-specific failure that killed the original won't recur; if it does, log and continue — never block the cursor twice). The permanent loss is now queryable in the audit log itself, not only in a rotating log file. +3. Run filter — PASS. +4. Doc: add `ReconciliationAbandoned` to the kind vocabulary in Component-AuditLog.md (Task 24 updates the Commons enum listing — coordinate: this task adds the value, 24 fixes the counts). +5. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.Commons src/ZB.MOM.WW.ScadaBridge.AuditLog docs/requirements/Component-AuditLog.md tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests && git commit -m "feat(audit-log): permanently-abandoned reconciliation rows leave a durable ReconciliationAbandoned audit record"` + +### Task 18: Reconciliation NodeB failover (owned from report 08) + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 9, 11, 13, 19, 20, 21 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/ISiteEnumerator.cs` (`SiteEntry` record ~37: additive `string? FallbackGrpcEndpoint = null`; fix the stale "NodeA-only first cut" doc comment) +- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/SiteEnumerator.cs` (~60-75: populate from `GrpcNodeBAddress`; include sites with blank NodeA but valid NodeB, primary = NodeA else NodeB) +- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/GrpcPullAuditEventsClient.cs` + `GrpcPullSiteCallsClient.cs` (retry-once-on-fallback) +- Modify: `docs/requirements/Component-AuditLog.md` (reconciliation endpoint-resolution paragraph) +- Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/SiteEnumeratorTests.cs`, `GrpcPullAuditEventsClientTests.cs`, `GrpcPullSiteCallsClientTests.cs` (extend) + +1. Failing tests: (a) `SiteEnumeratorTests` — site with both addresses yields `SiteEntry(id, nodeA, FallbackGrpcEndpoint: nodeB)`; site with only NodeB is no longer skipped; (b) `GrpcPullAuditEventsClientTests` — fake invoker throws `Unavailable` for the primary endpoint and succeeds for the fallback → the pull returns the fallback's events (today: collapses to empty). `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter "SiteEnumeratorTests|GrpcPull"` — FAIL. +2. Implement: in both pull clients' transport-fault catch (the `Unavailable`/`DeadlineExceeded`/`Cancelled`/`HttpRequestException` set already enumerated there), if `FallbackGrpcEndpoint` is non-blank and differs from the primary, invoke once against it before collapsing to empty; log the failover at Information. The invoker's per-endpoint channel cache already handles the second endpoint. +3. Run filter + full Central test folder — PASS. +4. Doc: Component-AuditLog.md — reconciliation dials NodeA first, fails over to NodeB per call; during a NodeA outage the loss-recovery safety net stays available. +5. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.AuditLog docs/requirements/Component-AuditLog.md tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests && git commit -m "fix(audit-log): reconciliation pulls fail over to site NodeB when NodeA is unreachable"` + +### Task 19: Batch the KpiSample purge + +**Classification:** standard +**Estimated implement time:** ~4 min +**Parallelizable with:** 1, 2, 3, 8, 9, 13, 18 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs` (`PurgeOlderThanAsync` ~65-71) +- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IKpiHistoryRepository.cs` (doc update; signature unchanged) +- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/KpiHistoryRepositoryTests.cs` (extend) + +1. Failing test: seed samples spread across 5 days; purge with a 2-day-old cutoff; assert the correct rows are gone and the return value equals the total deleted — then assert via an EF command interceptor (or SQL capture, matching how sibling tests inspect SQL) that **more than one** DELETE statement was issued for a multi-window purge. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter KpiHistoryRepositoryTests` — FAIL (single `ExecuteDeleteAsync` today). +2. Implement provider-portable time-slice batching (avoids `DELETE TOP` dialect divergence — the convention Task 24 documents): + +```csharp +public async Task PurgeOlderThanAsync(DateTime before, CancellationToken cancellationToken = default) +{ + // Time-sliced batches: each DELETE covers at most one hour of samples, capping the + // lock/log footprint per statement (arch-review 04, P4 — steady state is ~1 day of + // rows/day; after an outage the catch-up would otherwise be one giant transaction). + var total = 0; + var floor = await _context.KpiSamples.Where(s => s.CapturedAtUtc < before) + .MinAsync(s => (DateTime?)s.CapturedAtUtc, cancellationToken); + while (floor is not null && floor < before) + { + var ceiling = floor.Value.AddHours(1) < before ? floor.Value.AddHours(1) : before; + total += await _context.KpiSamples + .Where(s => s.CapturedAtUtc < ceiling) + .ExecuteDeleteAsync(cancellationToken); + floor = await _context.KpiSamples.Where(s => s.CapturedAtUtc < before) + .MinAsync(s => (DateTime?)s.CapturedAtUtc, cancellationToken); + } + return total; +} +``` + +3. Run filter — PASS; run `dotnet test tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests` (recorder actor contract unchanged) — PASS. +4. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IKpiHistoryRepository.cs tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests && git commit -m "perf(kpi-history): time-sliced batched purge replaces the single daily mega-DELETE"` + +### Task 20: NotificationOutbox KPI — single-query aggregation, bounded oldest + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 3, 8, 9, 14, 18, 19 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/NotificationOutboxRepository.cs` (`ComputeKpisAsync` ~239-283, `ComputePerSiteKpisAsync` ~286-334, `ComputePerNodeKpisAsync` ~337-391) +- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/NotificationOutboxRepositoryPerSiteKpiTests.cs`, `...PerNodeKpiTests.cs`, and the global KPI tests (all must stay green — this is a pure refactor of query shape) + +1. Add one new failing test capturing the *bounded-oldest* behavior: seed 3 non-terminal rows; assert the oldest-pending computation issues no query that materializes more than 1 row (assert via command interceptor / captured SQL: the oldest query is `ORDER BY … OFFSET 0 ROWS FETCH NEXT 1` or `TOP(1)`, not a full non-terminal SELECT). Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter NotificationOutboxRepository` — FAIL (per-site/per-node paths materialize every non-terminal row today, ~lines 312-318, 368-375). +2. Refactor each snapshot to: + - **Counts:** one conditional-aggregation query per scope — `GroupBy(n => 1)` (global) / `GroupBy(n => n.SourceSiteId)` / `GroupBy(n => n.SourceNode)` with `g.Count(n => )` per metric, replacing the 5-7 sequential round trips. + - **Oldest pending:** global — `Where(nonTerminal).OrderBy(n => n.CreatedAt).Select(n => (DateTimeOffset?)n.CreatedAt).FirstOrDefaultAsync()`; per-site/per-node — try server-side `GroupBy(...).Select(g => new { g.Key, Oldest = g.Min(n => n.CreatedAt) })` first; if the `DateTimeOffset` converter blocks translation (the review's noted awkwardness), fall back to projecting only `(Key, CreatedAt)` pairs of non-terminal rows (narrow projection, no entity materialization) and reduce in memory — either way the full-entity materialization is gone. +3. Run the full NotificationOutboxRepository test set + `dotnet test tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests --filter Kpi` — PASS (identical snapshot values). +4. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/NotificationOutboxRepository.cs tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests && git commit -m "perf(notification-outbox): single-query KPI aggregation; oldest-pending no longer materializes the live queue"` + +### Task 21: `OPTION (RECOMPILE)` on the SiteCalls catch-all query + +**Classification:** small +**Estimated implement time:** ~2 min +**Parallelizable with:** 1, 2, 3, 8, 9, 13, 18, 19, 20 (NOT 11 — same file) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs` (`QueryAsync` SQL ~186-202) +- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/SiteCallAuditRepositoryTests.cs` (existing query tests must pass) + +1. Append `OPTION (RECOMPILE)` after the `ORDER BY` line of the `FormattableString` (with a comment: every filter is the `(@p IS NULL OR col=@p)` optional-parameter shape; RECOMPILE lets the optimizer prune dead predicates per invocation and pick `IX_SiteCalls_Status_Updated`/`IX_SiteCalls_NonTerminal` — arch-review 04, P5; per-invocation compile cost is fine at UI-page cadence). +2. With infra up: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter SiteCallAuditRepositoryTests` — PASS. +3. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs && git commit -m "perf(site-call-audit): OPTION (RECOMPILE) on the optional-parameter query page"` + +### Task 22: Short first-tick delay on the three purge timers + +**Classification:** small +**Estimated implement time:** ~3 min +**Parallelizable with:** 1, 2, 6, 8, 19, 20, 21 (NOT 4/5 — AuditLogPurgeActor; NOT 16 — SiteCallAuditActor) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeActor.cs` (`PreStart` ~95-100) +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/SiteCallAuditActor.cs` (`StartPurgeTimer` ~374-387) +- Modify: `src/ZB.MOM.WW.ScadaBridge.KpiHistory/KpiHistoryRecorderActor.cs` (purge timer ~124-128) +- Test: extend each actor's existing test file with one fast-first-tick case + +1. Failing test (one per actor, using each suite's existing scheduler/interval-override rig): configure a long interval (24h) and assert a purge tick arrives within the short initial delay window rather than never. +2. Implement uniformly: `var initialDelay = interval < ShortFirstTick ? interval : ShortFirstTick;` with `private static readonly TimeSpan ShortFirstTick = TimeSpan.FromMinutes(5);` and a shared comment: purges are idempotent; `initialDelay = interval` meant a node recycled daily would never purge (arch-review 04, S9). Taking `min(interval, 5min)` keeps millisecond test cadences unchanged. +3. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter AuditLogPurgeActorTests && dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests --filter Purge && dotnet test tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests` — PASS. +4. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.AuditLog src/ZB.MOM.WW.ScadaBridge.SiteCallAudit src/ZB.MOM.WW.ScadaBridge.KpiHistory tests && git commit -m "fix(purge): short first tick on all three purge timers so daily-recycled nodes still purge"` + +### Task 23: Low-severity cleanup batch (param chunking, KPI catalog, computed-column guard) + +**Classification:** small +**Estimated implement time:** ~5 min +**Parallelizable with:** 4, 5, 8, 9, 11, 20 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriter.cs` (`MarkForwardedAsync` ~644-664, `MarkReconciledAsync` ~735-753) +- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Types/Kpi/KpiMetrics.cs` (locate: `grep -rn "class KpiMetrics" src/ZB.MOM.WW.ScadaBridge.Commons`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/Kpi/NotificationOutboxKpiSampleSource.cs` (~40-43) +- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/AuditLogEntityTypeConfiguration.cs` (~158-164) +- Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SqliteAuditWriterWriteTests.cs` (extend), `tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests/Kpi` (extend) + +1. **P8 — parameter chunking:** failing test — write 1200 events, `MarkForwardedAsync(all 1200 ids)`, assert all flip to Forwarded (today: one statement with 1200+ parameters risks/violates SQLite's 999 limit as batch sizes grow). Implement: chunk `eventIds` into slices of 500 inside the existing `_writeLock` (one transaction around all chunks so the batch stays atomic); same in `MarkReconciledAsync`. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter SqliteAuditWriter` — PASS. +2. **C4 — KPI metric catalog:** add `KpiMetrics.NotificationOutbox.StuckCount = "stuckCount"` and `OldestPendingAgeSeconds = "oldestPendingAgeSeconds"` (EXACT existing string values — persisted, renames forbidden); replace the inline literals in `NotificationOutboxKpiSampleSource` with the constants; add a test asserting the constant values equal the historical strings. +3. **P7 — computed-column guard:** extend the `IngestedAtUtc` comment in `AuditLogEntityTypeConfiguration`: "NEVER use IngestedAtUtc in a WHERE predicate — non-persisted, it forces a full-table `JSON_VALUE` parse; read-side projection only (arch-review 04, P7)." +4. Run both test filters + build — PASS. +5. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.AuditLog src/ZB.MOM.WW.ScadaBridge.Commons src/ZB.MOM.WW.ScadaBridge.NotificationOutbox src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase tests && git commit -m "chore(data-layer): low-severity cleanups — SQLite param chunking, KPI metric catalog, computed-column predicate guard"` + +### Task 24: Design-doc reconciliation pass + +**Classification:** small +**Estimated implement time:** ~5 min +**Parallelizable with:** all code tasks except 7 (same files); run AFTER 17 (AuditKind gains a value) +**Files:** +- Modify: `docs/requirements/Component-Commons.md` (~46-47, ~89) +- Modify: `docs/requirements/Component-ConfigurationDatabase.md` (~58; timestamp-convention + clustered-key + repository-dialect notes) +- Modify: `docs/requirements/Component-SiteCallAudit.md` (~44-51 schema section) +- Modify: `docs/requirements/Component-AuditLog.md` (~396-399 per-channel config channel list) +- Modify: `docs/requirements/Component-NotificationOutbox.md` (~23 "dedicated blocking-I/O dispatcher" phrasing — align with Task 25 if it has landed, else describe the actual thread-pool off-actor delivery) + +Concrete edits (each verifiable against the code cited in the review): +1. Component-Commons.md ~47: `AuditKind` list → the current enum (`grep -A20 "enum AuditKind" src/ZB.MOM.WW.ScadaBridge.Commons/Types/Enums/AuditKind.cs`): the original 10 + `SecuredWriteSubmitted/Approved/Rejected/Executed` (4) + `ReconciliationAbandoned` (Task 17). ~46: `AuditChannel` → 5 values incl. `SecuredWrite`. +2. Component-Commons.md ~89: `NotificationType` → "`Email`, `Sms` (shipped 2026-06-19; the Teams plan was evaluated and dropped)". +3. Component-ConfigurationDatabase.md ~58: NotificationList Type discriminator → "`Email` / `Sms`". +4. Component-ConfigurationDatabase.md ~399: "single global value in v1, no per-channel overrides" → per-channel `AuditLog:PerChannelRetentionDays` overrides exist (skip if Task 7 already fixed it). +5. Component-SiteCallAudit.md ~44-51: rewrite the schema table to the shipped columns (`Channel`/`Target`/`AuditStatus`-derived `Status` strings, `SourceNode`, no provenance columns) — align with `SiteCallEntityTypeConfiguration.cs` and Component-ConfigurationDatabase.md:64 (the current source of truth per the review). +6. Component-AuditLog.md ~396-399: per-channel retention config enumerates all five channels including `SecuredWrite` (the validator already accepts it — doc under-promised). +7. Component-Commons.md / Component-ConfigurationDatabase.md: document the `AuditLogRow` deviation (persistence-only projection of the external `ZB.MOM.WW.Audit.AuditEvent` contract — deliberately not a Commons POCO) and two conventions: **new tables use UTC `DateTime` + the Utc converter** (the `DateTimeOffset` choice on `Notification` is the documented exception that forced in-memory Min workarounds), and **new repositories write MSSQL-only SQL with MSSQL-backed test fixtures** (the NotificationOutboxRepository dual SQLite/T-SQL dialect is a documented legacy exception — finding S8). Record two tracked follow-ups: the AuditLog clustered-key `(OccurredAtUtc, EventId)` benchmark question (P3) and the per-node KPI sampling cadence / outbox keyset-paging deferrals (P4/P6). +8. Verify: `grep -rn "Teams" docs/requirements/ | grep -iv "dropped\|evaluated"` returns nothing load-bearing; `grep -n "no per-channel" docs/requirements/Component-ConfigurationDatabase.md` returns nothing. +9. Commit: `git add docs/requirements && git commit -m "docs: reconcile design docs with shipped code (enums, Sms, per-channel retention, SiteCalls schema, conventions)"` + +### Task 25: Bounded parallel notification delivery + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** 8, 9, 11, 18, 19 (NOT 13 — same actor file; do 13 first) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/NotificationOutboxOptions.cs` (add `MaxParallelDeliveries`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/NotificationOutboxActor.cs` (`RunDispatchPass` ~321-347) +- Modify: `docs/requirements/Component-NotificationOutbox.md` (~23 dispatcher paragraph) +- Test: `tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests/NotificationOutboxActorDispatchTests.cs` (extend) + +1. Failing test: adapter stub whose `Deliver` blocks on a gate and records concurrent-invocation high-water mark; enqueue 8 pending notifications; with `MaxParallelDeliveries = 4` assert high-water ≥ 2 and all 8 reach terminal status; with `MaxParallelDeliveries = 1` assert high-water == 1 (existing sequential behavior preserved). `dotnet test tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests --filter Dispatch` — FAIL. +2. Implement: `MaxParallelDeliveries` (default **4**, clamp min 1 — the review's alarm-storm scenario: sequential 5 s SMTP attempts cap throughput at ~0.2/s regardless of batch size). In `RunDispatchPass`, replace the sequential foreach with `SemaphoreSlim(maxParallel)` + `Task.WhenAll`, giving **each notification its own DI scope/repository** (rows are claimed per-notification, so no shared-context concurrency; the existing in-flight pass guard is untouched). Audit emission stays per-notification inside its task. +3. Run the full outbox suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests` — PASS (attempt/terminal emission tests prove per-row semantics survived). +4. Doc: Component-NotificationOutbox.md dispatcher section — delivery runs off the actor thread on the thread pool with bounded per-sweep parallelism (`MaxParallelDeliveries`, default 4); drop the inaccurate "dedicated blocking-I/O dispatcher" phrasing (doc drift the review flagged). +5. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.NotificationOutbox docs/requirements/Component-NotificationOutbox.md tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests && git commit -m "perf(notification-outbox): bounded parallel delivery within a dispatch sweep (default 4)"` + +--- + +## Dependencies on other plans + +- **Plan 02 (Communication/S&F)** owns the S&F retry-sweep batching — Task 15's site-side tracking-store query touches adjacent code in the StoreAndForward project; coordinate file ownership if both run concurrently. +- **Plan 07 (UI/Management/Security)** owns SecuredWrite emission paths — the `SourceNode`-NULL follow-up (U4) stays with them; Task 24 only fixes the doc under-promise. +- **Plan 08 (Conventions/Tests)** flagged the reconciliation NodeB gap; it is implemented here (Task 18) — plan 08 should not duplicate it. +- Tasks 13/14 plumb `RequestedBy` through UI/ManagementActor senders — if plan 07 refactors those senders simultaneously, land this plan's additive message change first (defaults keep old senders compiling). +- Migration serialization is **intra-plan only** (Tasks 6 → 8); no other plan currently adds ConfigurationDatabase migrations, but if one does, coordinate snapshot ordering. + +## Execution order + +**P0 (the operational time bombs — do first, in this order):** 1 → 2 → 3 (site purge chain), 4 → 5 (partition-switch hardening), 6 → 8 (migrations, serialized), 9, 10. +**P1:** 11, 12, 13, 15 → 16, 17, 18. +**P2 (hardening/cleanup):** 14, 19, 20, 21, 22, 23, 25, then 7 and 24 (docs last so they describe what actually landed). +Intra-plan critical path: **1 → 2 → 3** and **15 → 16**; everything else is broadly parallelizable per each task's "Parallelizable with" contract. Finish with a full `dotnet build ZB.MOM.WW.ScadaBridge.slnx && dotnet test ZB.MOM.WW.ScadaBridge.slnx` (infra compose up for the MSSQL suites) before declaring the plan done. diff --git a/archreview/plans/PLAN-04-data-audit-backbone.md.tasks.json b/archreview/plans/PLAN-04-data-audit-backbone.md.tasks.json new file mode 100644 index 00000000..48e401e2 --- /dev/null +++ b/archreview/plans/PLAN-04-data-audit-backbone.md.tasks.json @@ -0,0 +1,32 @@ +{ + "planPath": "archreview/plans/PLAN-04-data-audit-backbone.md", + "tasks": [ + { "id": 1, "subject": "Task 1: Site SQLite retention purge — PurgeExpiredAsync on the writer", "status": "pending", "blockedBy": [] }, + { "id": 2, "subject": "Task 2: Site retention options", "status": "pending", "blockedBy": [] }, + { "id": 3, "subject": "Task 3: Site retention hosted job + registration + design doc", "status": "pending", "blockedBy": [1, 2] }, + { "id": 4, "subject": "Task 4: Maintenance command timeout on partition switch + per-channel purge", "status": "pending", "blockedBy": [] }, + { "id": 5, "subject": "Task 5: Purge-failure health event (not just a log line)", "status": "pending", "blockedBy": [4] }, + { "id": 6, "subject": "Task 6: Fix purger role grants (migration)", "status": "pending", "blockedBy": [] }, + { "id": 7, "subject": "Task 7: Honest append-only enforcement story (docs)", "status": "pending", "blockedBy": [6] }, + { "id": 8, "subject": "Task 8: SiteCalls filtered non-terminal index (migration)", "status": "pending", "blockedBy": [6] }, + { "id": 9, "subject": "Task 9: Bound the execution-tree edge scan with a root time window", "status": "pending", "blockedBy": [] }, + { "id": 10, "subject": "Task 10: Fix the hardcoded-zero backlogTotal KPI history", "status": "pending", "blockedBy": [] }, + { "id": 11, "subject": "Task 11: SiteCalls same-rank freshness upsert (unfreeze RetryCount/LastError)", "status": "pending", "blockedBy": [] }, + { "id": 12, "subject": "Task 12: EF Core connection resiliency (EnableRetryOnFailure)", "status": "pending", "blockedBy": [] }, + { "id": 13, "subject": "Task 13: Audit row for operator Retry on parked notifications", "status": "pending", "blockedBy": [] }, + { "id": 14, "subject": "Task 14: Operator identity on the SiteCalls Retry/Discard relay", "status": "pending", "blockedBy": [13] }, + { "id": 15, "subject": "Task 15: Composite keyset in the SiteCalls reconciliation pull contract (site side)", "status": "pending", "blockedBy": [] }, + { "id": 16, "subject": "Task 16: Central keyset cursor — un-pin the SiteCalls reconciliation loop", "status": "pending", "blockedBy": [15] }, + { "id": 17, "subject": "Task 17: Durable record for permanently-abandoned reconciliation rows", "status": "pending", "blockedBy": [] }, + { "id": 18, "subject": "Task 18: Reconciliation NodeB failover (owned from report 08)", "status": "pending", "blockedBy": [] }, + { "id": 19, "subject": "Task 19: Batch the KpiSample purge", "status": "pending", "blockedBy": [] }, + { "id": 20, "subject": "Task 20: NotificationOutbox KPI — single-query aggregation, bounded oldest", "status": "pending", "blockedBy": [] }, + { "id": 21, "subject": "Task 21: OPTION (RECOMPILE) on the SiteCalls catch-all query", "status": "pending", "blockedBy": [11] }, + { "id": 22, "subject": "Task 22: Short first-tick delay on the three purge timers", "status": "pending", "blockedBy": [5, 16] }, + { "id": 23, "subject": "Task 23: Low-severity cleanup batch (param chunking, KPI catalog, computed-column guard)", "status": "pending", "blockedBy": [1] }, + { "id": 24, "subject": "Task 24: Design-doc reconciliation pass", "status": "pending", "blockedBy": [7, 17] }, + { "id": 25, "subject": "Task 25: Bounded parallel notification delivery", "status": "pending", "blockedBy": [13] }, + { "id": 26, "subject": "Final verification: full solution build + test sweep (infra up for MSSQL suites)", "status": "pending", "blockedBy": [3, 8, 9, 10, 12, 14, 18, 19, 20, 21, 22, 23, 24, 25] } + ], + "lastUpdated": "2026-07-08" +} diff --git a/archreview/plans/PLAN-05-templates-deployment-transport.md b/archreview/plans/PLAN-05-templates-deployment-transport.md new file mode 100644 index 00000000..b7ad7398 --- /dev/null +++ b/archreview/plans/PLAN-05-templates-deployment-transport.md @@ -0,0 +1,632 @@ +# Templates, Deployment & Transport Fix Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. + +**Goal:** Close every finding in `archreview/05-templates-deployment-transport.md` — the Transport import silent-data-loss family (inheritance, script fields, native alarm sources, AreaName), the flattener/collision repeated-composition bug, revision-hash staleness blindness, native-alarm lock enforcement, Roslyn compile cost/leak on read paths, deployment audit/delete defects, script-trust hardening, and the cross-cutting script-artifact invalidation contract. + +**Architecture:** The Transport import apply path (`BundleImporter`) re-implements DTO→entity mapping by hand and drifted from the DTO layer; fixes restore field-for-field fidelity, add the missing inheritance/native-source passes, and single-source the diff/sync equality predicates so they cannot drift again, guarded structurally by a per-entity round-trip equivalence test suite. TemplateEngine fixes convert the shared cycle-guard `visited` set to a recursion-path set (composition is by-slot, not by-template), extend the revision hash/diff to the native-alarm member class, and thread `IsLocked` through `ResolvedNativeAlarmSource`. A new Commons seam (`IScriptArtifactChangeBus` + `ScriptArtifactsChanged`) defines the "script artifact changed → invalidate everywhere" contract; Transport publishes post-commit, plan 06 consumes it in the Inbound API. + +**Tech Stack:** C#/.NET, EF Core (in-memory + MSSQL integration fixtures), Roslyn (`Microsoft.CodeAnalysis.CSharp.Scripting`), xUnit. Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per project: `dotnet test tests/`. + +## Findings Coverage + +| Report finding | Severity | Task(s) | +|---|---|---| +| Bundle import drops template inheritance edges | Critical | 1, 2 | +| Imported scripts lose `MinTimeBetweenRuns`/`ExecutionTimeoutSeconds`; `LockedInDerived` not in DTOs | High | 3, 4 | +| Template `NativeAlarmSources` not transported (overrides dangle) | High | 5 | +| Flattener drops grandchild members on repeated composition; CollisionDetector blind | High | 9, 10 | +| Revision hash and diff omit `NativeAlarmSources` | High | 11, 12 | +| Import leaves stale compiled Inbound API handlers | High | 14 (contract + Transport publisher; consumer → **plan 06**) | +| Locked native alarm sources overridable at instance level | High | 13 | +| Post-success audit failure flips Success→Failed | Medium | 17 | +| Deleting `NotDeployed` instance requires live site round-trip | Medium | 18 | +| Import blocker heuristic hard-blocks on false positives | Medium | 19 | +| `OperationLockManager` single-node in-memory | Medium | 25 (documented invariant); active-node gating **deferred to plans 01/07** (cluster routing ownership) | +| Every validation run compiles every script (CPU + assembly leak) | High (perf) | 15, 16 | +| `LineDiffer` O((N+M)²) memory, no input cap | Medium (perf) | 22 | +| Templates with folder/parent/composition never diff `Identical` | Medium (perf) | 23 | +| `LoadAsync` O(bundle) + unbounded session count | Low (perf) | 24 (session cap); manifest-peek `ReadManifestAsync` **Deferred** — already doc-acknowledged (`Component-Transport.md:90`), pure optimisation | +| Import apply is one long EF transaction | Low (perf) | **Deferred** — Task 16 removes the dominant cost (Roslyn in `StaleInstanceProbe`); restructuring the single-transaction rollback contract is high-risk for low residual gain | +| Deny-list gaps: `Environment.Exit`/`FailFast`/`GetEnvironmentVariable` | Medium (sec) | 21 | +| `System.Data` provider namespaces unbounded network hole | Medium (sec) | 21 | +| Reflection-gateway member list incomplete | Medium (sec) | 21 | +| Transport import bypasses script trust gate | Medium (sec) | 20 | +| Instance alarm overrides implement less than spec | Medium | 25 (doc aligned to implementation — adding `DescriptionOverride`/`OnTriggerScriptOverride` columns is a feature, not a fix; decision recorded) | +| `ArtifactDiff` equality predicates drift from sync predicates | Medium | 6 | +| Spec advertises unimplemented `scripts/` bundle dir; `MaxBundleEntryCount = 4` | Low | 25 | +| `TryCompile` surfaces only first violation/error | Low | 24 | +| RevisionHash determinism contract fragile | Low | 11 (guard test extended; migration note) | +| Underdeveloped 1: template fidelity shipped behind site/instance wave | — | 1–8 | +| Underdeveloped 2: no derived-template import test | — | 1, 8 | +| Underdeveloped 3: no repeated-composition flattening test | — | 9, 10 | +| Underdeveloped 4: deferred-but-load-bearing (rename call-site rewriting, Transport-012 UI, `ReadManifestAsync`) | — | 25 documents the rename limitation; Transport-012 UI + `ReadManifestAsync` **Deferred** (acknowledged backlog, not correctness) | +| Underdeveloped 5: preview→apply window has no optimistic concurrency | — | **Deferred** — version fields are documented as reserved (`ArtifactDiff.cs:33-37`); 30-min TTL window accepted, recorded in Task 25 doc sweep | +| Underdeveloped 6: three hand-maintained compile-surface mirrors | — | **Deferred** — reflection/representative-script guard tests already exist; source-generator is an improvement, not a defect | +| Underdeveloped 7: `SemanticValidator` leaf-name fallback silently accepts wrong-child calls | — | 24 (emit warning) | + +--- + +### Task 1: Wire template inheritance edges in the import apply path + +**Classification:** high-risk (Critical data-loss fix on the import data contract) +**Estimated implement time:** ~5 min +**Parallelizable with:** 9, 10, 11, 12, 13, 15, 16, 17, 18, 21, 22 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs` (add `ResolveInheritanceEdgesAsync`; call it from `ApplyAsync` after line 1044 `ResolveCompositionEdgesAsync`; extend Overwrite branch ~1466-1484) +- Test (new): `tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/Import/InheritanceImportTests.cs` + +1. Write failing test (follow the fixture pattern of `Import/SiteInstanceImportTests.cs` — source context → `BundleExporter.ExportAsync` → target context → `LoadAsync`/`PreviewAsync`/`ApplyAsync`): +```csharp +[Fact] +public async Task Import_DerivedTemplate_WiresParentTemplateId() +{ + // Source: base "Pump" + derived "Pump.WaterPump" (ParentTemplateId -> Pump) + // with one inherited-from-base attribute on Pump. + var bundle = await ExportTemplatesFromSourceAsync("Pump", "Pump.WaterPump"); + await ImportIntoEmptyTargetAsync(bundle, ResolutionAction.Add); + + var target = await TargetTemplateRepo.GetAllTemplatesAsync(); + var basePump = target.Single(t => t.Name == "Pump"); + var derived = target.Single(t => t.Name == "Pump.WaterPump"); + Assert.Equal(basePump.Id, derived.ParentTemplateId); // FAILS today: null +} + +[Fact] +public async Task Import_Overwrite_UpdatesParentEdge_AndRenamedBaseIsHonoured() +{ + // Bundle base renamed to "Pump2" via ResolutionAction.Rename; derived's + // BaseTemplateName ("Pump") must resolve through the rename map to Pump2. + ... + Assert.Equal(pump2.Id, derived.ParentTemplateId); +} +``` +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests --filter FullyQualifiedName~InheritanceImportTests` → expect **FAIL** (ParentTemplateId null). +3. Implement `ResolveInheritanceEdgesAsync(IReadOnlyList dtos, Dictionary<(string,string), ImportResolution> resolutionMap, string user, CancellationToken ct)` mirroring `ResolveCompositionEdgesAsync` (line 1993): + - For each non-Skip template DTO: compute the template's *persisted* name (Rename → `RenameTo`), load the tracked entity by name. + - If `dto.BaseTemplateName is null` → set `ParentTemplateId = null` (an Overwrite from a root-template bundle must clear a stale edge). + - Else resolve the base name through the resolution map (`ResolveOrDefault(resolutionMap, "Template", dto.BaseTemplateName)`; Rename → `RenameTo`), then look up by name across staged-imported + pre-existing target templates (same lookup set `ResolveCompositionEdgesAsync` builds). Set `ParentTemplateId`. + - Unresolvable base → leave null and emit a `BundleImportBaseTemplateUnresolved` audit row (mirror `BundleImportAlarmScriptUnresolved`), never throw. + - Call from `ApplyAsync` immediately after `ResolveCompositionEdgesAsync` (line 1044). +4. Run test → expect **PASS**. Run the whole Transport suites: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.Tests && dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests`. +5. Update `docs/requirements/Component-Transport.md` apply-pass list (the second-pass rewire section) to name the inheritance pass alongside alarm-script and composition rewires. +6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Transport tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests docs/requirements/Component-Transport.md && git commit -m "fix(transport): wire template inheritance edges on bundle import (C3) — derived templates no longer land as roots"` + +### Task 2: Import-time acyclicity check over the merged template graph + +**Classification:** standard +**Estimated implement time:** ~4 min +**Parallelizable with:** 9, 10, 11, 12, 13, 15, 16, 17, 18, 21, 22 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs` (after the Task-1 inheritance pass in `ApplyAsync`) +- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/Import/InheritanceImportTests.cs` (extend) + +1. Write failing test: craft a bundle whose templates form an inheritance cycle against a pre-existing target template (target `A` has parent `B`; bundle overwrites `B` with `BaseTemplateName: "A"`). Assert `ApplyAsync` throws `SemanticValidationException` and the transaction rolled back (target `B.ParentTemplateId` unchanged). Add the composition-cycle twin (bundle composition edge closing a loop through a target template). +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests --filter FullyQualifiedName~InheritanceImportTests` → **FAIL** (cycle persists today; importer performs no acyclicity check). +3. Implement: after both rewire passes, load all templates + compositions from the tracked context and run TemplateEngine `CycleDetector.DetectInheritanceCycle` / `DetectCompositionCycle` (Transport already references TemplateEngine). On a detected cycle, throw `SemanticValidationException(new[] {"Import would create a template inheritance/composition cycle: "})` — the existing catch path rolls back and reports. +4. Run test → **PASS**. Commit: `git commit -m "fix(transport): reject bundle imports that would persist an inheritance/composition cycle"` (with the touched paths in `git add`). + +### Task 3: Carry `MinTimeBetweenRuns` / `ExecutionTimeoutSeconds` through import + +**Classification:** small +**Estimated implement time:** ~3 min +**Parallelizable with:** 9–13, 15–18, 21, 22 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs` (`BuildTemplate` ~1550-1560; `SyncTemplateScriptsAsync` `changed` predicate + copies ~1836-1851 and the add branch ~1870-1877) +- Test (new): `tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/Import/TemplateScriptFidelityTests.cs` + +1. Failing test: export a template whose script has `MinTimeBetweenRuns = TimeSpan.FromSeconds(30)` and `ExecutionTimeoutSeconds = 42`; import as Add into empty target, assert both fields on the persisted `TemplateScript`. Second test: Overwrite a target whose script differs *only* in `ExecutionTimeoutSeconds` → assert field updated and a `TemplateScriptUpdated` audit row emitted. +2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests --filter FullyQualifiedName~TemplateScriptFidelityTests` → **FAIL**. +3. Implement: add `MinTimeBetweenRuns = s.MinTimeBetweenRuns, ExecutionTimeoutSeconds = s.ExecutionTimeoutSeconds` to the `TemplateScript` initializers in `BuildTemplate` and the `SyncTemplateScriptsAsync` add branch; add `|| current.MinTimeBetweenRuns != scriptDto.MinTimeBetweenRuns || current.ExecutionTimeoutSeconds != scriptDto.ExecutionTimeoutSeconds` to the `changed` predicate and copy both fields in the update block. +4. Run → **PASS**. Commit: `git commit -m "fix(transport): carry script cadence/timeout fields through bundle import (was silently reset to defaults)"` + +### Task 4: Add `LockedInDerived` to template child DTOs and apply paths + +**Classification:** high-risk (bundle data contract change — must stay additive) +**Estimated implement time:** ~5 min +**Parallelizable with:** 9–13, 15–18, 21, 22 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/EntityDtos.cs` (`TemplateAttributeDto`, `TemplateAlarmDto`, `TemplateScriptDto` — trailing `bool LockedInDerived = false`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/EntitySerializer.cs` (export ~55-90; `FromBundleContent` ~330-360) +- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs` (`BuildTemplate`, three `SyncTemplate*Async` predicates/copies) +- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.Tests/Serialization/EntitySerializerTests.cs` (extend), `tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/Import/TemplateScriptFidelityTests.cs` (extend) + +1. Failing unit test in `EntitySerializerTests`: entity with `LockedInDerived = true` on an attribute, alarm, and script → `ToBundleContent` DTOs carry it; `FromBundleContent` round-trips it. Failing integration test: import as Add → persisted flags true; old-form JSON (no field) still deserializes with `false` (backward-compat assertion — deserialize a hand-written DTO JSON blob missing the property). +2. Run both test projects with `--filter LockedInDerived` → **FAIL** (compile errors first — that counts; fix signatures then re-run for behavioural red). +3. Implement: trailing optional parameter on each record (mirrors the `ExecutionTimeoutSeconds = null` precedent — additive, no `schemaVersion` bump needed); populate at export; consume in `BuildTemplate`, `FromBundleContent`, and all three sync helpers (`changed` predicate + copy). +4. Run → **PASS**. Commit: `git commit -m "fix(transport): transport LockedInDerived on template attributes/alarms/scripts (additive DTO fields)"` + +### Task 5: Transport template `NativeAlarmSources` end-to-end + +**Classification:** high-risk (new bundle payload class + new sync pass) +**Estimated implement time:** ~5 min (DTO+export+apply) — if it runs long, split diff wiring into its own commit +**Parallelizable with:** 9–13, 15–18, 21, 22 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/EntityDtos.cs` (new `TemplateNativeAlarmSourceDto`; `TemplateDto` gains trailing `IReadOnlyList NativeAlarmSources` defaulting empty) +- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/EntitySerializer.cs` (export + `FromBundleContent`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs` (`BuildTemplate`; new `SyncTemplateNativeAlarmSourcesAsync` called from the Overwrite branch at ~1482) +- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/ArtifactDiff.cs` (`CompareTemplate` — `DiffChildren` over native sources) +- Test (new): `tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/Import/NativeAlarmSourceImportTests.cs`; extend `EntitySerializerTests` + +1. Failing tests: +```csharp +[Fact] public async Task Import_Add_CarriesTemplateNativeAlarmSources() +// template with one source (all fields set incl. IsLocked, IsInherited, LockedInDerived) +// → export → import Add → persisted TemplateNativeAlarmSource field-equal. + +[Fact] public async Task Import_Overwrite_SyncsNativeAlarmSources() +// target source differs in ConditionFilter; bundle also drops one source and adds one +// → assert update/add/delete applied + audit rows (TemplateNativeAlarmSource{Added,Updated,Deleted}). + +[Fact] public async Task Preview_DiffsNativeAlarmSources() +// changed source shows a "NativeAlarmSources." FieldChange; identical → no change row. + +[Fact] public async Task Import_InstanceOverride_NoLongerDangles() +// export template-with-source + instance override of it → import → flatten target instance +// → resolved config contains the source with the override applied (today: silently dropped). +``` +2. Run `--filter NativeAlarmSourceImportTests` → **FAIL**. +3. Implement: + - DTO: `record TemplateNativeAlarmSourceDto(string Name, string? Description, string ConnectionName, string SourceReference, string? ConditionFilter, bool IsLocked, bool IsInherited, bool LockedInDerived)`. + - Export in `EntitySerializer.ToBundleContent` (include `IsInherited` placeholder rows — the collision detector depends on them) and consume in `FromBundleContent`. + - `BuildTemplate`: copy the collection. New `SyncTemplateNativeAlarmSourcesAsync` mirrors `SyncTemplateAttributesAsync` (name-keyed add/update/delete; repository has `DeleteTemplateNativeAlarmSourceAsync`-style members — verify exact name on `ITemplateEngineRepository` and add if missing). + - `ArtifactDiff.CompareTemplate`: `DiffChildren(existing.NativeAlarmSources, incoming.NativeAlarmSources, e => e.Name, i => i.Name, NativeAlarmSourcesEqual, "NativeAlarmSources", changes)`. +4. Run → **PASS**; run full Transport suites. Update `docs/requirements/Component-Transport.md` transported-entity table + `CLAUDE.md` component #24 blurb (template native alarm sources now travel). +5. Commit: `git commit -m "fix(transport): transport template NativeAlarmSources — imports no longer amputate native alarm mirrors nor dangle instance overrides"` + +### Task 6: Single-source per-entity field comparers for diff + sync + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 9–13, 15–18, 21, 22 +**Files:** +- Create: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/TemplateChildEquality.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/ArtifactDiff.cs` (replace `AttributesEqual`/`AlarmsEqual`/`ScriptsEqual` at 646-666; `CompareTemplate` passes a script-name resolver for the alarm on-trigger ref) +- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs` (three sync `changed` predicates delegate to the same class) +- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.Tests/Import/ArtifactDiffTests.cs` (extend) + +1. Failing tests (today's drift, both directions): +```csharp +[Fact] public void CompareTemplate_AttributeElementDataTypeChange_IsModified() // omitted today → false Identical +[Fact] public void CompareTemplate_ScriptExecutionTimeoutChange_IsModified() // omitted today +[Fact] public void CompareTemplate_AlarmOnTriggerScriptChange_IsModified() // omitted today +[Fact] public void CompareTemplate_LockedInDerivedChange_IsModified() +``` +2. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.Tests --filter ArtifactDiffTests` → **FAIL**. +3. Implement `TemplateChildEquality`: one static method per entity comparing the **complete** writable field list (attributes: Value/DataType/ElementDataType/IsLocked/LockedInDerived/Description/DataSourceReference; alarms: + `OnTriggerScriptName` via a `Func scriptNameById` resolver built from `existing.Scripts`; scripts: + MinTimeBetweenRuns/ExecutionTimeoutSeconds/LockedInDerived; native sources per Task 5). XML-doc each with: *"This is the single source of truth for ' changed'; ArtifactDiff and SyncTemplate\*Async must both call it."* Point `ArtifactDiff` and the three (four after Task 5) sync predicates at it. +4. Run Transport unit + integration suites → **PASS**. +5. Commit: `git commit -m "refactor(transport): single-source template child equality — diff and overwrite-sync can no longer drift"` + +### Task 7: Export real `AreaName` (finish the half-shipped Area transport) + +**Classification:** standard +**Estimated implement time:** ~4 min +**Parallelizable with:** 1–4, 9–13, 15–18, 21, 22 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Export/DependencyResolver.cs` / `ResolvedExport.cs` (aggregate gains `areaNameById` for exported instances; exporter has `ScadaBridgeDbContext` — load `Areas` for the instances' `AreaId`s) +- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/EntitySerializer.cs` (~247-256: replace the `AreaName: null` TODO with the lookup) +- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/Import/SiteInstanceImportTests.cs` (extend) or new `AreaTransportTests.cs` + +1. Failing test: source instance assigned to Area "Line-1" under its site → export → import into target with `--create-missing` site mapping → assert target instance's `AreaId` resolves to an Area named "Line-1" under the target site (the importer's `GetOrCreate` machinery at `BundleImporter.cs:3168` is already built and tested — only the exporter feeds it null). +2. Run → **FAIL** (imported `AreaId` is null). +3. Implement: extend the export aggregate with the instances' area id→name map (single query over `Areas` filtered to referenced ids) and emit `AreaName: areaNameById.TryGetValue(inst.AreaId ?? -1, out var an) ? an : null`. Delete the TODO comment. +4. Run → **PASS**. Update `docs/requirements/Component-Transport.md` (Area-by-name now actually travels; remove any "importer leaves AreaId null" caveat). CLAUDE.md #24 already claims this — now true, no edit needed (verify wording). +5. Commit: `git commit -m "fix(transport): export instance AreaName — Area-by-name reconciliation was half-shipped (exporter hardcoded null)"` + +### Task 8: Round-trip export→import equivalence test suite (Theme-3 structural guard) + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 9–18, 21, 22 (test-only; depends on Tasks 3–5, 7 landing first) +**Files:** +- Create: `tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/RoundTripEquivalenceTests.cs` + +1. For **every** transported entity type (Template + attribute/alarm/script/native-source/composition children, SharedScript, ExternalSystem + methods, DatabaseConnection, NotificationList, SmtpConfig, SmsConfig, ApiMethod, TemplateFolder, Site, DataConnection, Instance + override children + bindings), seed a source DB with a *maximally populated* entity (every writable property non-default), export, import into a fresh target, reload, and compare **by reflection**: every public writable property of the entity must be equal, except an explicit per-type exclusion list (`Id`, FK id columns remapped by design, timestamps). The reflection sweep is the point — a future entity property that no DTO carries fails the test instead of silently vanishing (exactly how `LockedInDerived`, cadence/timeout, and native sources were lost). +```csharp +private static void AssertRoundTripEqual(T source, T imported, params string[] excluded) +{ + foreach (var p in typeof(T).GetProperties().Where(p => p.CanWrite + && !excluded.Contains(p.Name) + && p.PropertyType.IsValueType || p.PropertyType == typeof(string))) + Assert.True(Equals(p.GetValue(source), p.GetValue(imported)), + $"{typeof(T).Name}.{p.Name} did not survive export→import"); +} +``` +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests --filter RoundTripEquivalenceTests` — expect **PASS** if Tasks 1–7 are complete (any failure here is a residual fidelity hole: fix it in the importer/exporter, not the test). +3. Commit: `git commit -m "test(transport): per-entity reflection round-trip equivalence suite — structural guard against silent import data loss"` + +### Task 9: Fix repeated-composition member loss in `FlatteningService` + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 1–8, 15–18, 21, 22 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Flattening/FlatteningService.cs` (the four `ResolveComposed*Recursive` methods: attributes ~285-297, alarms ~648-660, native sources ~761-772, scripts ~913-924) +- Test: `tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Flattening/FlatteningServiceTests.cs` (extend) + +1. Failing test: +```csharp +[Fact] +public void Flatten_SameTemplateComposedTwice_ResolvesNestedMembersUnderBothSlots() +{ + // X composes Y as "y1" and "y2"; Y composes Z as "z"; Z has attribute "Val", + // alarm "Alm", script "Run", native source "Src". + var result = Flatten(x); + Assert.Contains(result.Attributes, a => a.CanonicalName == "y1.z.Val"); + Assert.Contains(result.Attributes, a => a.CanonicalName == "y2.z.Val"); // FAILS today + // + same assertions for Alarms / Scripts / NativeAlarmSources +} + +[Fact] +public void Flatten_CompositionCycle_StillTerminates() // regression guard for the cycle role of `visited` +``` +2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests --filter Flatten_SameTemplateComposedTwice` → **FAIL** (`y2.z.Val` missing). +3. Implement: convert the shared `visited` set into a **recursion-path** set in all four methods. Pattern (identical in each): +```csharp +// Guard on the recursion PATH, not globally: composition is by-slot, so a template +// legitimately appears twice under different prefixes; only a template already on +// the CURRENT path is a cycle. +var pushed = new List(); +foreach (var composedTemplate in composedChain) + if (path.Add(composedTemplate.Id)) pushed.Add(composedTemplate.Id); +try +{ + foreach (var composedTemplate in composedChain) + { + if (!compositionMap.TryGetValue(composedTemplate.Id, out var nested)) continue; + foreach (var n in nested) + if (!path.Contains(n.ComposedTemplateId)) // cycle guard + ResolveComposed…Recursive(n, $"{prefix}.{n.InstanceName}", …, path); + } +} +finally { foreach (var id in pushed) path.Remove(id); } +``` +(Rename the parameter `visited` → `path` so the semantics are explicit.) +4. Run the full TemplateEngine test project → **PASS**, no regressions. +5. Commit: `git commit -m "fix(template-engine): repeated composition no longer drops nested members — cycle guard keyed on recursion path, not global template id"` + +### Task 10: Fix `CollisionDetector` blindness under repeated composition + +**Classification:** small +**Estimated implement time:** ~3 min +**Parallelizable with:** 1–8, 15–18, 21, 22 (different file from Task 9) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/CollisionDetector.cs` (`CollectComposedMembers` ~119-144) +- Test: `tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/CollisionDetectorTests.cs` (extend) + +1. Failing test: X composes Y twice (`y1`, `y2`); Y composes Z; X also declares a direct attribute named `y2.z.Val` (path-collision). Assert the detector reports the collision (today the second slot's subtree is never collected, so it cannot). +2. Run `--filter CollisionDetectorTests` → **FAIL**. +3. Implement the same path-set conversion as Task 9: `visited` becomes the current recursion path — add `template.Id` on entry, remove in `finally`; the early-return then only fires on genuine cycles. +4. Run → **PASS**. Commit: `git commit -m "fix(template-engine): collision detection sees all slots of a repeatedly-composed template"` + +### Task 11: Add `NativeAlarmSources` to `RevisionHashService` (planned hash migration) + +**Classification:** high-risk (revision-hash change ⇒ staleness flags flip for native-source-bearing instances) +**Estimated implement time:** ~4 min +**Parallelizable with:** 1–8, 15–18, 21, 22 — **after Task 13** (hash includes `IsLocked`) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Flattening/RevisionHashService.cs` (new `HashableNativeAlarmSource`; `HashableConfiguration` gains the property in alphabetical position) +- Test: `tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Flattening/RevisionHashServiceTests.cs` (extend, incl. the alphabetical-guard test) + +1. Failing tests: +```csharp +[Fact] public void ComputeHash_NativeAlarmSourceChange_ChangesHash() // FAILS today +[Fact] public void ComputeHash_NoNativeAlarmSources_HashUnchangedFromBaseline() +// pin a literal known-good hash for a native-source-free config so the migration +// is surgical: only instances that HAVE native sources flip stale. +``` +2. Run `--filter RevisionHashServiceTests` → **FAIL**. +3. Implement: `HashableNativeAlarmSource` with alphabetical properties (`CanonicalName`, `ConditionFilter`, `ConnectionName`, `IsLocked`, `SourceReference`); on `HashableConfiguration` add `public List? NativeAlarmSources { get; init; }` in alphabetical slot (between `InstanceUniqueName` and `Scripts`), populated **null-when-empty** so `WhenWritingNull` keeps every native-source-free config's hash byte-identical (no global staleness storm — the migration only flags the instances the fix is about). Extend the `HashableRecords_PropertiesDeclaredAlphabetically` guard to the new record. Add a migration note to the class doc-comment and `docs/requirements/Component-TemplateEngine.md` (revision-hash section): *native-source-bearing instances will report stale once after upgrade; redeploy clears it — deliberate.* +4. Run → **PASS**. Commit: `git commit -m "fix(template-engine): native alarm sources participate in the revision hash — staleness detection restored (deliberate one-time stale flip for affected instances)"` + +### Task 12: Add `NativeAlarmSources` to `DiffService` + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** 1–8, 15–18, 21, 22 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Flattening/DiffService.cs` (add added/removed/changed sweep — file currently has zero `NativeAlarm` references) +- Modify: the `ConfigurationDiff` type in Commons (`src/ZB.MOM.WW.ScadaBridge.Commons/Types/Flattening/` — additive section) +- Test: `tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Flattening/DiffServiceTests.cs` (extend) + +1. Failing test: two flattened configs differing only in one native source's `SourceReference` → diff reports one changed native source; add/remove cases too. +2. Run `--filter DiffServiceTests` → **FAIL**. +3. Implement following the exact pattern DiffService uses for Alarms (keyed on `CanonicalName`, field-compare `ConnectionName`/`SourceReference`/`ConditionFilter`/`IsLocked`). Additive `NativeAlarmSources` collection on `ConfigurationDiff` (message-contract rule: additive only). +4. Run → **PASS**; also run `tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests` (`DeploymentComparisonTests` consume the diff). Commit: `git commit -m "fix(template-engine): DiffService covers native alarm sources — Deployments diff view no longer blind to native-alarm edits"` + +### Task 13: Enforce locks on native alarm source overrides + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 1–8, 15–18, 21, 22 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Types/Flattening/FlattenedConfiguration.cs` (`ResolvedNativeAlarmSource` gains `public bool IsLocked { get; init; }` — additive) +- Modify: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Flattening/FlatteningService.cs` (`ResolveInheritedNativeAlarmSources` ~670-710 sets `IsLocked`; `ApplyInstanceNativeAlarmSourceOverrides` ~775-792 adds the lock check) +- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (`HandleSetInstanceNativeAlarmSourceOverride` ~882-909) +- Test: `tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Flattening/FlatteningServiceTests.cs`, ManagementService test project (mirror an existing handler test) + +1. Failing tests: +```csharp +[Fact] public void Flatten_LockedNativeAlarmSource_IgnoresInstanceOverride() +// template source IsLocked=true + instance override repointing ConnectionName +// → resolved source keeps template values, Source stays "Template". FAILS today. + +[Fact] public async Task SetNativeAlarmSourceOverride_OnLockedSource_Throws() +// ManagementActor handler rejects with ManagementCommandException. FAILS today. +``` +2. Run TemplateEngine + ManagementService test filters → **FAIL**. +3. Implement: + - `ResolveInheritedNativeAlarmSources`: set `IsLocked = binding.IsLocked || lockedNames.Contains(binding.Name)` on the resolved record (keep `lockedNames` derived-shadow logic). + - `ApplyInstanceNativeAlarmSourceOverrides`: after the `TryGetValue`, `if (existing.IsLocked) continue;` — exactly mirroring the attribute (`:309`) and alarm (`:337`) paths. + - `HandleSetInstanceNativeAlarmSourceOverride`: resolve `IFlatteningPipeline` from `sp` (ManagementService already references DeploymentManager), flatten the instance, find the source by `cmd.SourceCanonicalName`; throw `ManagementCommandException($"Native alarm source '{…}' is locked at the template level and cannot be overridden.")` when locked, and also reject when the canonical name doesn't resolve at all (prevents creating dangling overrides — matching the flattener's cannot-add rule). +4. Run → **PASS**. Update `docs/requirements/Component-TemplateEngine.md` §Native-alarm override rules if wording needs the enforcement point named. Commit: `git commit -m "fix(security): enforce template locks on native alarm source overrides at flatten and management-command level"` + +### Task 14: Script-artifact invalidation contract — Commons seam + Transport publisher (plan-06 handoff) + +**Classification:** high-risk (cross-component contract) +**Estimated implement time:** ~5 min +**Parallelizable with:** 9–13, 15–18, 21, 22 +**Files:** +- Create: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Scripting/ScriptArtifactsChanged.cs` +- Create: `src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/IScriptArtifactChangeBus.cs` +- Create: `src/ZB.MOM.WW.ScadaBridge.Host/Services/InProcessScriptArtifactChangeBus.cs` (+ singleton registration in the Host's central-role service wiring) +- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs` (optional ctor dep `IScriptArtifactChangeBus?`; publish after `tx.CommitAsync` at ~1098) +- Create: `docs/plans/2026-07-08-script-artifact-invalidation-contract.md` +- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/Import/BundleImporterApplyTests.cs` (extend), new Host unit test for the bus + +1. Failing test: apply a bundle that Overwrites an `ApiMethod`, a `SharedScript`, and a `Template`; a recording fake `IScriptArtifactChangeBus` handed to the importer must receive, **after** commit, one `ScriptArtifactsChanged` per kind with the post-resolution (renamed) names. Second test: a failed apply (semantic validation throw) publishes **nothing**. +2. Run → **FAIL** (types don't exist — compile-red, then behavioural red). +3. Implement the contract: +```csharp +// Commons/Messages/Scripting/ScriptArtifactsChanged.cs +/// Advisory, at-least-once, published only AFTER the mutating transaction +/// commits. Consumers must invalidate any compiled/cached artifact keyed by these +/// names, and must ALSO self-heal without the notification (content-hash checks) — +/// this bus is in-process per node; cross-node propagation is the consumer's +/// responsibility (see docs/plans/2026-07-08-script-artifact-invalidation-contract.md). +public sealed record ScriptArtifactsChanged( + string ArtifactKind, // ScriptArtifactKinds.ApiMethod | SharedScript | Template + IReadOnlyList Names, // post-resolution (renamed) names + string Source, // "BundleImport" | "Management" | "FailoverActivation" + DateTimeOffset OccurredAtUtc); +public static class ScriptArtifactKinds { public const string ApiMethod = "ApiMethod"; /* … */ } + +// Commons/Interfaces/IScriptArtifactChangeBus.cs +public interface IScriptArtifactChangeBus +{ + void Publish(ScriptArtifactsChanged notification); + IDisposable Subscribe(Action handler); +} +``` + `InProcessScriptArtifactChangeBus`: lock-free copy-on-write handler list; `Publish` swallows-and-logs handler exceptions (a bad consumer must never fail an import). `BundleImporter` collects overwritten/renamed names during apply and publishes once per kind **after** `tx.CommitAsync` (never inside the transaction — a notification for a rolled-back write is worse than a missed one). +4. Write the contract doc: semantics above + the **explicit handoff table**: *plan 06 implements (a) the `InboundScriptExecutor` subscription that evicts `_scriptHandlers` and `_knownBadMethods` by name, (b) publishing from the ManagementActor `UpdateApiMethod`/shared-script paths (closes the project-memory gotchas), (c) cross-node/failover freshness (revision-keyed handler cache).* Plan 05 ships the seam + the Transport publisher only. +5. Run → **PASS**; `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Commit: `git commit -m "feat(commons/transport): ScriptArtifactsChanged invalidation contract + post-commit publish from bundle import (consumer lands in plan 06)"` + +### Task 15: Cache script-compile verdicts by code hash + +**Classification:** standard +**Estimated implement time:** ~4 min +**Parallelizable with:** 1–14, 17, 18, 21, 22 +**Files:** +- Create: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ScriptCompileVerdictCache.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ScriptCompiler.cs` (`TryCompile` consults the cache) +- Test: `tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Validation/ScriptCompilerTests.cs` (extend) + +1. Failing test: +```csharp +[Fact] +public void TryCompile_SameCodeTwice_SecondCallIsCacheHit() +{ + ScriptCompileVerdictCache.Clear(); + var c = new ScriptCompiler(); + c.TryCompile("return 1;", "A"); + var hitsBefore = ScriptCompileVerdictCache.Hits; + var r = c.TryCompile("return 1;", "B"); // same code, different name + Assert.True(r.IsSuccess); + Assert.Equal(hitsBefore + 1, ScriptCompileVerdictCache.Hits); +} +``` +2. Run `--filter ScriptCompilerTests` → **FAIL**. +3. Implement: static `ConcurrentDictionary` keyed `Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(code)))` — the verdict is a pure function of code + policy, so a process-wide cache is sound; the policy is compile-time static. Bound at 4096 entries (on overflow, `Clear()` — simple and safe). Expose `Hits`/`Count`/`Clear()` for tests and diagnostics. `TryCompile` keys the cache on code only; the script *name* is formatted into the returned message at read time (store the name-free error, format on return). This removes both the repeat-CPU and the repeat assembly load: an unchanged script compiles exactly once per process lifetime. +4. Run → **PASS**; run the full TemplateEngine test project. Commit: `git commit -m "perf(template-engine): cache script compile verdicts by code hash — unchanged scripts compile once per process"` + +### Task 16: Skip script compilation on read-only staleness/comparison paths + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 1–15, 21, 22 +**Files:** +- Modify: `IFlatteningPipeline` + `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/FlatteningPipeline.cs` (`FlattenAndValidateAsync(int instanceId, bool validateScripts = true, CancellationToken ct = default)` — additive default param) +- Modify: `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentService.cs` (`GetDeploymentComparisonAsync` ~652 passes `validateScripts: false`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/StaleInstanceProbe.cs` (~34 passes `validateScripts: false`) +- Test: `tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests/DeploymentComparisonTests.cs`, `DeploymentServiceTests.cs` (extend) + +1. Failing test: with a template script that does **not** compile, `GetDeploymentComparisonAsync` must still return a comparison (staleness needs the hash, not a compile) — today `FlattenAndValidateAsync` fails and the comparison errors out; and a mock-pipeline assertion that the comparison/probe paths request `validateScripts: false` while `DeployInstanceAsync` keeps `true`. +2. Run `--filter DeploymentComparisonTests` → **FAIL**. +3. Implement: `FlatteningPipeline` skips `ValidationService.ValidateScriptCompilation` (only that validator — semantic/structural validation stays) when `validateScripts` is false. Deploy path unchanged (`true` default). This removes 100% of Roslyn work (and the non-collectible `InteractiveAssemblyLoader` loads) from the Deployments page sweep and from `BundleImporter.ComputeStaleInstanceIdsAsync`'s per-instance probing. +4. Run DeploymentManager + Transport integration suites → **PASS**. Update `docs/requirements/Component-DeploymentManager.md` (comparison path is hash-only by design; deploy gate remains authoritative). +5. Commit: `git commit -m "perf(deployment): staleness/comparison paths no longer Roslyn-compile every script — deploy gate remains the authoritative compile"` + +### Task 17: Isolate the post-success deployment audit write + +**Classification:** small +**Estimated implement time:** ~3 min +**Parallelizable with:** 1–16, 21, 22 (same file as 16/18 — coordinate order, regions don't overlap) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentService.cs` (~330-332) +- Test: `tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests/DeploymentServiceTests.cs` (extend) + +1. Failing test: audit-service mock throws on the `"Deploy"` `LogAsync`; site responds Success → assert result is Success and the persisted record's `Status` **stays** `Success` (today the catch at :343 flips it to `Failed` — the exact divergence the surrounding comments prevent, and a violation of "audit-write failure NEVER aborts the user-facing action"). +2. Run → **FAIL**. +3. Implement: wrap the `LogAsync` at :330-332 in its own try/catch that only `_logger.LogError`s (mirror `TryLogLifecycleTimeoutAsync` at :1034). Same guard for any other post-terminal-status audit writes in the lifecycle verbs (audit-only; scan `DisableInstanceAsync`/`EnableInstanceAsync` for the same shape and guard if present). +4. Run → **PASS**. Commit: `git commit -m "fix(deployment): audit-write fault can no longer flip a committed Success deployment to Failed"` + +### Task 18: Delete `NotDeployed` instances without a site round-trip + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** 1–16, 21, 22 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentService.cs` (`DeleteInstanceAsync` ~546-585) +- Test: `tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests/DeploymentServiceTests.cs` (extend) + +1. Failing test: instance in `NotDeployed`, communication-service mock configured to throw (site unreachable) → `DeleteInstanceAsync` must succeed and remove the record without ever calling the communication service (assert `Verify(..., Times.Never)`); Transport-imported instances (always `NotDeployed`) become deletable against uncommissioned sites. +2. Run → **FAIL**. +3. Implement: after the lock acquisition, `if (instance.State == InstanceState.NotDeployed)` → skip the site command entirely, delete the central record (reuse the existing guarded delete block), audit `"Delete"` with `new { CommandId = (string?)null, LocalOnly = true }`, return success. Matches `StateTransitionValidator.CanDelete`'s documented rationale (:40-49) — the two components currently disagree about the same design sentence. +4. Run → **PASS**. Update the matching sentence in `docs/requirements/Component-DeploymentManager.md`. Commit: `git commit -m "fix(deployment): delete of a NotDeployed instance is central-side only — no site round-trip, unreachable sites can't block cleanup"` + +### Task 19: Import blocker heuristic — local-declaration exclusion, denylist additions, warning severity for template scripts + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 9–18, 21, 22 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs` (`RunSemanticValidationAsync` ~3680-3692; `DetectBlockersAsync` ~751-818; `KnownNonReferenceNames` ~853-874; `CollectCallIdentifiers`) +- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/SemanticValidatorImportTests.cs` (extend) + +1. Failing tests: +```csharp +[Fact] public async Task Apply_TemplateScriptWithLocalPascalCaseMethod_DoesNotHardBlock() +// script: "decimal ComputeRate(decimal x) => x * 2; return ComputeRate(Attributes[\"A\"]);" +// → ApplyAsync succeeds (today: SemanticValidationException on 'ComputeRate'). + +[Fact] public async Task Apply_TemplateScriptWithRegexAndStringBuilder_DoesNotHardBlock() + +[Fact] public async Task Apply_ApiMethodWithUnresolvedReference_StillHardBlocks() +// ApiMethod script calling a genuinely-missing ExternalSystem keeps throwing. +``` +2. Run `--filter SemanticValidatorImportTests` → **FAIL**. +3. Implement, three edges: + - **Locally-declared names:** before the candidate loop, Roslyn-parse each scanned script (`CSharpSyntaxTree.ParseText(code, new CSharpParseOptions(kind: SourceCodeKind.Script))` — Transport gets it transitively; add the package ref if not), collect `LocalFunctionStatementSyntax`/`MethodDeclarationSyntax` identifier names, and exclude them from `referenced` for that script's contributions. + - **Denylist additions:** add to `KnownNonReferenceNames`: `Regex, Match, Matches, IsMatch, Replace, Split, StringBuilder, Append, AppendLine, Parse, TryParse, Format, Join, Abs, Round, Min, Max, Floor, Ceiling, Pow, Sqrt, Json, Serialize, Deserialize, Where, Select, First, FirstOrDefault, Any, All, Count, Sum, Average, OrderBy, OrderByDescending, GroupBy, Distinct, Add, Remove, Clear, Contains, ContainsKey, TryGetValue, StartsWith, EndsWith, Substring, Trim, ToUpper, ToLower, HAVING, VALUES, DELETE, DISTINCT, LIMIT` (comment: *the list will still drift — that is why template-script findings are warnings, below*). + - **Severity split:** track which candidate names came from *template* scripts vs *ApiMethod* scripts. Template-script findings become **warnings** — logged + appended to the `ImportResult`/preview blocker list with a non-blocking flag (the deploy gate re-validates authoritatively at deploy time, `ScriptCompiler.TryCompile`); **ApiMethod** findings remain hard errors (`SemanticValidationException`) because no downstream design-time gate exists for them. Apply the identical split in `DetectBlockersAsync` so preview and apply agree. +4. Run → **PASS**; full Transport integration suite. Update `docs/requirements/Component-Transport.md` blocker-row section (template-script name findings are advisory; ApiMethod findings block). +5. Commit: `git commit -m "fix(transport): blocker heuristic no longer hard-blocks valid imports — local declarations excluded, denylist extended, template-script findings downgraded to warnings"` + +### Task 20: Run the script trust gate on bundle import + +**Classification:** standard +**Estimated implement time:** ~4 min +**Parallelizable with:** 9–18, 21, 22 (same `BundleImporter` region as Task 19 — do after 19) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs` (`RunSemanticValidationAsync` Pass 2, ~3694+; `DetectBlockersAsync` for preview parity) +- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/SemanticValidatorImportTests.cs` (extend) + +1. Failing test: bundle whose template script (and, second case, ApiMethod script) contains `System.Diagnostics.Process.Start("cmd")` → `ApplyAsync` throws `SemanticValidationException` naming the forbidden API; preview surfaces the same as a blocker row. Today both import clean — the fifth write path with no trust gate. +2. Run → **FAIL**. +3. Implement: in Pass 2, for every non-Skip template script, shared script, and ApiMethod script body, call `ScriptTrustValidator.FindViolations(code)` (ScriptAnalysis is referenced via TemplateEngine; add the direct project reference to `ZB.MOM.WW.ScadaBridge.Transport.csproj` if needed). Any violation → **hard error** for all three kinds (trust violations are not the false-positive-prone name heuristic — the semantic verdict is authoritative, same severity everywhere). The bodies are already in memory; Task 15's verdict cache keeps repeat cost nil. +4. Run → **PASS**. Update `docs/requirements/Component-ScriptAnalysis.md` call-site list (Transport import is now the fifth delegating call site) and `Component-Transport.md`. +5. Commit: `git commit -m "fix(security): bundle import runs the script trust gate — forbidden-API scripts rejected at import review, not at runtime"` + +### Task 21: Harden the `ScriptTrustPolicy` deny-list + +**Classification:** high-risk (security policy; false-positive blast radius on existing deployed scripts) +**Estimated implement time:** ~5 min +**Parallelizable with:** 1–20, 22 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ScriptAnalysis/ScriptTrustPolicy.cs` (`ForbiddenScopes` :36-45, `ReflectionGatewayMembers` :66-88) +- Test: `tests/ZB.MOM.WW.ScadaBridge.ScriptAnalysis.Tests/ScriptTrustValidatorTests.cs` (extend) + +1. Failing tests: +```csharp +[Theory] +[InlineData("Environment.Exit(0);")] +[InlineData("Environment.FailFast(\"x\");")] +[InlineData("var s = Environment.GetEnvironmentVariable(\"SCADABRIDGE_API_KEY\");")] +[InlineData("var c = new Microsoft.Data.SqlClient.SqlConnection(\"Server=attacker\");")] +public void FindViolations_Flags(string code) => Assert.NotEmpty(ScriptTrustValidator.FindViolations(code)); + +[Theory] // syntactic gateway closure (matters in TPA-degraded fallback mode) +[InlineData("typeof(string).Assembly.GetTypes()")] +[InlineData("asm.EntryPoint")] +public void SyntacticPass_Flags(string code) ... + +[Theory] // must NOT regress legitimate scripts +[InlineData("await Task.Delay(1);")] +[InlineData("Func f = x => x; f.Invoke(3);")] // delegate Invoke stays allowed — see step 3 +public void FindViolations_Allows(string code) => Assert.Empty(ScriptTrustValidator.FindViolations(code)); +``` +2. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.ScriptAnalysis.Tests --filter ScriptTrustValidatorTests` → **FAIL**. +3. Implement: + - `ForbiddenScopes` += `"System.Environment"` (whole type — no legitimate script use; `Environment.NewLine` callers can use `"\n"`), `"System.GC"`, and the ADO.NET provider namespaces: `"Microsoft.Data"`, `"System.Data.SqlClient"`, `"System.Data.Odbc"`, `"System.Data.OleDb"` (scripts reach `DbConnection` only via the `Database` helper, whose `System.Data.Common` abstract types stay allowed — closes the arbitrary-host `new SqlConnection(...)` channel the `System.Net` deny was meant to cover). + - `ReflectionGatewayMembers` += `"GetTypes"`, `"EntryPoint"`, `"DeclaredMethods"`, `"DeclaredMembers"`, `"DeclaredConstructors"`, `"DynamicInvoke"`. **Deliberately exclude `"Invoke"`**: the syntactic pass rejects regardless of receiver, and `delegate.Invoke()` is legitimate; `MethodInfo.Invoke` is already caught semantically (`System.Reflection` scope) — record this rationale in the code comment. +4. Run → **PASS**; also run the SiteRuntime/InboundAPI representative-script suites if they exercise the surface (`dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter Script` — any newly-flagged legitimate pattern is a deliberate decision to surface, not silently absorb). +5. Update `docs/requirements/Component-ScriptAnalysis.md` deny-list table + the `System.Data` posture paragraph. Commit: `git commit -m "fix(security): deny Environment/GC and ADO.NET provider namespaces; close reflection-gateway list (GetTypes/EntryPoint/Declared*/DynamicInvoke)"` + +### Task 22: Bound `LineDiffer` input size + +**Classification:** small +**Estimated implement time:** ~3 min +**Parallelizable with:** 1–21 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/LineDiffer.cs` +- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.Tests/Import/LineDifferTests.cs` (extend) + +1. Failing test: two 10,000-line inputs with zero common lines → `Diff` returns within a small time budget with `Truncated = true` and a summary-only result (hunk list empty or `` placeholders), and never materializes the Myers trace. Assert e.g. total runtime < 2s (today: ~20k rounds × 160 KB V-snapshots ≈ multi-GB transient allocation — the assertion red-lines as timeout/OOM). +2. Run `--filter LineDifferTests` → **FAIL** (or pathological slowness — treat as fail). +3. Implement: `private const int MaxInputLines = 4000;` at the top of `Diff`: `if (aLines.Length + bLines.Length > MaxInputLines) return SummaryOnlyResult(aLines.Length, bLines.Length);` producing the same shape the `maxLines` output cap already emits (`Truncated = true`, add/remove totals = full line counts, no hunks). The 400-line *output* cap stays; this bounds the *algorithm input* so a bloated or crafted bundle cannot OOM the active central node from the import preview. +4. Run → **PASS**. Commit: `git commit -m "fix(transport): cap LineDiffer input size — oversized script diffs degrade to summary instead of O((N+M)^2) memory"` + +### Task 23: Fix `ArtifactDiff` placeholder-identity comparisons + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** 9–18, 21, 22 (touches `ArtifactDiff` + preview call site — after Tasks 5/6) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/ArtifactDiff.cs` (`CompareTemplate` gains optional `IReadOnlyDictionary? folderNameById`, `IReadOnlyDictionary? templateNameById`; `FolderNameOf`/`BaseTemplateNameOf`/`CompositionTargetNameOf` at 674-693 use them) +- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs` (`PreviewAsync` ~379-388 passes the maps it already builds at ~365-366) +- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.Tests/Import/ArtifactDiffTests.cs` (extend) + +1. Failing test: existing template with `FolderId`, `ParentTemplateId`, and one composition; incoming DTO carries the *matching names* → `CompareTemplate(dto, existing, folderMap, templateMap)` classifies **Identical** (today: `"" != "Pump"` ⇒ spurious Modified on every re-import, defeating the wizard's auto-skip contract, `Component-Transport.md:222`). +2. Run → **FAIL**. +3. Implement: resolve real names through the maps, falling back to the `` placeholder only when a map is absent/misses (keeps existing unit tests valid). Wire the two maps through from `PreviewAsync`. +4. Run Transport unit + integration suites → **PASS**. Commit: `git commit -m "fix(transport): template diffs resolve folder/base/composition names — unchanged structured templates classify Identical again"` + +### Task 24: Low-severity cleanup batch + +**Classification:** small +**Estimated implement time:** ~5 min +**Parallelizable with:** none (touches files owned by Tasks 15 and 19 — run after both) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleSessionStore.cs` + `src/ZB.MOM.WW.ScadaBridge.Transport/TransportOptions.cs` (session cap) +- Modify: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ScriptCompiler.cs` (:42, :47 — full error lists) +- Modify: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/SemanticValidator.cs` (~43-56 — leaf-fallback warning) +- Tests: `BundleSessionStoreTests.cs`, `ScriptCompilerTests.cs`, `SemanticValidatorTests` (Validation folder) + +Three concrete edits, each TDD'd: +1. **Session cap:** `TransportOptions.MaxConcurrentImportSessions = 8`. Failing test: 9th `Add` throws `InvalidOperationException("Too many concurrent import sessions …")`; removing one admits a new one. Implement the count check in `BundleSessionStore.Add` (:38). Bounds the N×~200 MB decrypted-content pinning. +2. **Full error lists:** failing test: script with two forbidden APIs → failure message contains both. Implement: `string.Join("; ", violations)` / `string.Join("; ", errors)` instead of `violations[0]`/`errors[0]` (coordinate with Task 15's cached-error format — store the joined string). +3. **Leaf-fallback warning:** failing test: composed script call resolved only by leaf-name fallback yields a `ValidationEntry.Warning(ValidationCategory.…, "call target '…' matched by leaf name only — child path not verified")` instead of silent acceptance. Implement in the fallback branch at `SemanticValidator.cs:43-56` (return-shape permitting; if the method returns bool, thread a warnings collector the way `ValidationService` already aggregates warnings). +4. Run all three filters → **PASS**. Commit: `git commit -m "chore: low-severity sweep — import session cap, full compile-error lists, leaf-name-fallback validation warning"` + +### Task 25: Design-doc + CLAUDE.md sync sweep and deferred-decision ledger + +**Classification:** small (docs only) +**Estimated implement time:** ~5 min +**Parallelizable with:** none (run last) +**Files:** +- Modify: `docs/requirements/Component-Transport.md`, `Component-TemplateEngine.md`, `Component-DeploymentManager.md`, `Component-ScriptAnalysis.md`, `/Users/dohertj2/Desktop/ScadaBridge/CLAUDE.md` (component #24 blurb + Transport bullet in Key Design Decisions) + +No tests; concrete edits (several were partially made inside their behavior tasks — this pass verifies and completes them, then does a stale-cross-reference sweep per repo convention): +1. **Component-Transport.md:** (a) remove the `scripts/` bundle-directory section (:45-47) or mark it explicitly *"not implemented — `MaxBundleEntryCount` default 4 would reject it"*; (b) transported-entity list gains template `NativeAlarmSources`, `LockedInDerived`, script cadence/timeout, real `AreaName`; (c) blocker section documents the warning/error severity split (Task 19) and the LineDiffer input cap (Task 22); (d) make the *rename call-site limitation* explicit and load-bearing: *a Rename resolution does not rewrite call sites in importing scripts — the target DB can end up with scripts calling the old name; Pass-1 registers both names so import passes, but deploy-time validation on the target is the real gate*; (e) note the accepted preview→apply window (no optimistic concurrency; version fields reserved — Underdeveloped 5) as a recorded decision. +2. **Component-TemplateEngine.md:** align the alarm-override granularity sentence (:113) to the implementation — only Trigger Definition thresholds and Priority Level are instance-overridable; Description and On-Trigger Script reference are **not** (`InstanceAlarmOverride` carries only `TriggerConfigurationOverride`/`PriorityLevelOverride`); record it as the decision (adding the two override columns is future feature work, not spec debt). Add the revision-hash native-source migration note (Task 11) if not already present. +3. **Component-DeploymentManager.md:** `NotDeployed` delete is central-side only (Task 18); post-success audit isolation (Task 17); add the explicit `OperationLockManager` invariant paragraph: *locks are per-node in-memory; mutual exclusion assumes management traffic reaches only the active central node (Traefik). Direct standby-port mutations are unsupported — structural rejection of ops on the non-active node is owned by the cluster/UI plans (01/07).* +4. **Component-ScriptAnalysis.md:** deny-list additions + `System.Data` provider posture + Transport import as the fifth delegating call site (Tasks 20, 21). +5. **CLAUDE.md:** component #24 blurb — add template native-alarm-source transport and confirm the Area-by-name claim is now true; adjust the "Transport (#24, M8)" Key Design Decisions bullet likewise. Per the umbrella-index rule, no `../scadaproj/CLAUDE.md` change is needed (no wire-relationship/stack change) — verify and note in the commit message. +6. `git diff` review, then commit: `git commit -m "docs: sync Transport/TemplateEngine/DeploymentManager/ScriptAnalysis specs + CLAUDE.md with the arch-review fix wave (plan 05)"` + +--- + +## Dependencies on other plans + +- **Plan 06 (Edge Integrations) — explicit handoff:** Task 14 ships the `ScriptArtifactsChanged` message, `IScriptArtifactChangeBus` seam, in-process bus, contract doc, and the Transport post-commit publisher. Plan 06 implements the consumers: `InboundScriptExecutor` eviction of `_scriptHandlers`/`_knownBadMethods`, publishing from the ManagementActor ApiMethod/shared-script update paths, and cross-node/failover handler freshness. Plan 06 should not start its consumer work until Task 14's contract doc is committed. +- **Plans 01/07:** structural rejection of management mutations on the non-active central node (the `OperationLockManager` residual risk) belongs to the cluster-routing/management-surface owners; Task 25 records the invariant only. +- **Plan 08:** also flagged the `AreaName: null` half-shipped feature — the fix is owned here (Task 7); plan 08 should not duplicate it. +- **Plan 03:** runtime script execution/containment is untouched by this plan (reference only). + +## Execution order + +**P0 (start immediately, in parallel):** Task 1 → 2 (the Critical, sequential), Task 14 (unblocks plan 06), Task 9 → 10 (flattener correctness), Task 11 (after 13), Task 13. +**Wave 2 (BundleImporter chain — same file, serialize):** 3 → 4 → 5 → 6 → 19 → 20 → 23, with 7 in parallel (exporter files) and 15/16/17/18/21/22 in parallel (disjoint projects). +**Wave 3:** 8 (equivalence suite — after 3–7), 12, 24. +**Last:** 25 (doc sweep), then a full `dotnet build ZB.MOM.WW.ScadaBridge.slnx && dotnet test ZB.MOM.WW.ScadaBridge.slnx` plus the direct `dotnet test tests/ZB.MOM.WW.ScadaBridge.CLI.Tests` run (slnx-exclusion gotcha) before declaring the plan done. diff --git a/archreview/plans/PLAN-05-templates-deployment-transport.md.tasks.json b/archreview/plans/PLAN-05-templates-deployment-transport.md.tasks.json new file mode 100644 index 00000000..d34bc328 --- /dev/null +++ b/archreview/plans/PLAN-05-templates-deployment-transport.md.tasks.json @@ -0,0 +1,31 @@ +{ + "planPath": "archreview/plans/PLAN-05-templates-deployment-transport.md", + "tasks": [ + { "id": 1, "subject": "Task 1: Wire template inheritance edges in the import apply path", "status": "pending", "blockedBy": [] }, + { "id": 2, "subject": "Task 2: Import-time acyclicity check over the merged template graph", "status": "pending", "blockedBy": [1] }, + { "id": 3, "subject": "Task 3: Carry MinTimeBetweenRuns/ExecutionTimeoutSeconds through import", "status": "pending", "blockedBy": [2] }, + { "id": 4, "subject": "Task 4: Add LockedInDerived to template child DTOs and apply paths", "status": "pending", "blockedBy": [3] }, + { "id": 5, "subject": "Task 5: Transport template NativeAlarmSources end-to-end", "status": "pending", "blockedBy": [4] }, + { "id": 6, "subject": "Task 6: Single-source per-entity field comparers for diff + sync", "status": "pending", "blockedBy": [5] }, + { "id": 7, "subject": "Task 7: Export real AreaName (finish half-shipped Area transport)", "status": "pending", "blockedBy": [] }, + { "id": 8, "subject": "Task 8: Round-trip export→import equivalence test suite", "status": "pending", "blockedBy": [5, 6, 7] }, + { "id": 9, "subject": "Task 9: Fix repeated-composition member loss in FlatteningService", "status": "pending", "blockedBy": [] }, + { "id": 10, "subject": "Task 10: Fix CollisionDetector blindness under repeated composition", "status": "pending", "blockedBy": [9] }, + { "id": 11, "subject": "Task 11: Add NativeAlarmSources to RevisionHashService (planned hash migration)", "status": "pending", "blockedBy": [13] }, + { "id": 12, "subject": "Task 12: Add NativeAlarmSources to DiffService", "status": "pending", "blockedBy": [13] }, + { "id": 13, "subject": "Task 13: Enforce locks on native alarm source overrides", "status": "pending", "blockedBy": [] }, + { "id": 14, "subject": "Task 14: Script-artifact invalidation contract — Commons seam + Transport publisher (plan-06 handoff)", "status": "pending", "blockedBy": [] }, + { "id": 15, "subject": "Task 15: Cache script compile verdicts by code hash", "status": "pending", "blockedBy": [] }, + { "id": 16, "subject": "Task 16: Skip script compilation on read-only staleness/comparison paths", "status": "pending", "blockedBy": [] }, + { "id": 17, "subject": "Task 17: Isolate the post-success deployment audit write", "status": "pending", "blockedBy": [16] }, + { "id": 18, "subject": "Task 18: Delete NotDeployed instances without a site round-trip", "status": "pending", "blockedBy": [17] }, + { "id": 19, "subject": "Task 19: Import blocker heuristic — local-declaration exclusion, denylist additions, warning severity", "status": "pending", "blockedBy": [6] }, + { "id": 20, "subject": "Task 20: Run the script trust gate on bundle import", "status": "pending", "blockedBy": [19] }, + { "id": 21, "subject": "Task 21: Harden the ScriptTrustPolicy deny-list", "status": "pending", "blockedBy": [] }, + { "id": 22, "subject": "Task 22: Bound LineDiffer input size", "status": "pending", "blockedBy": [] }, + { "id": 23, "subject": "Task 23: Fix ArtifactDiff placeholder-identity comparisons", "status": "pending", "blockedBy": [20] }, + { "id": 24, "subject": "Task 24: Low-severity cleanup batch (session cap, full error lists, semantic-validator warning)", "status": "pending", "blockedBy": [15, 19] }, + { "id": 25, "subject": "Task 25: Design-doc + CLAUDE.md sync sweep and deferred-decision ledger", "status": "pending", "blockedBy": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24] } + ], + "lastUpdated": "2026-07-08" +} diff --git a/archreview/plans/PLAN-06-edge-integrations.md b/archreview/plans/PLAN-06-edge-integrations.md new file mode 100644 index 00000000..152a794f --- /dev/null +++ b/archreview/plans/PLAN-06-edge-integrations.md @@ -0,0 +1,820 @@ +# Edge Integrations Fix Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. + +**Goal:** Fix every finding in `archreview/06-edge-integrations.md` — make the inbound compiled-handler cache self-healing (revision-checked, failover/direct-SQL/import-safe), close both ESG spec gaps (per-system timeout, path templating), bound outbound response buffering and SMS blast radius, wire the dead SMS retry fields, and clear the report's Medium/Low stability, performance, and convention findings across InboundAPI, ExternalSystemGateway, NotificationService/Outbox delivery, and DelmiaNotifier. + +**Architecture:** The core stability fix replaces the name-keyed compiled-handler cache in `InboundScriptExecutor` with a script-revision-checked cache (the fresh `ApiMethod` row is already in hand on every request — one ordinal string compare detects staleness and triggers recompile), plus a public `InvalidateMethod` seam that plan 05's "script artifact changed → invalidate" contract will consume. The ESG per-system timeout is a full vertical slice per repo rules: Commons entity → EF mapping + migration → artifact message contract (additive) → site SQLite column → client honor → management command → CLI → UI → design doc. Notification fixes are localized to the delivery adapters and the outbox dispatcher's retry-policy resolution. + +**Tech Stack:** C#/.NET 9, xUnit + NSubstitute, EF Core migrations (MS SQL central), Microsoft.Data.Sqlite (site), Roslyn scripting, System.CommandLine (CLI), Blazor Server (UI). + +Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per-project: `dotnet test tests/`. EF migrations: **build first, never `--no-build`**; `dotnet ef migrations add --project src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase --startup-project src/ZB.MOM.WW.ScadaBridge.Host`. If the migration scaffolds empty, delete the files and rebuild first. + +## Findings Coverage + +| # | Finding (report section) | Severity | Task(s) | +|---|--------------------------|----------|---------| +| S1 | Compiled inbound handler cache stale across central failover (+ known-bad mirror image) | High | 1, 2, 3 | +| S2 | Method timeout abandons running script; DI scope disposed underneath (use-after-dispose + orphan accumulation, no counter) | Medium | 4 | +| S3 | ESG API-key `AuthConfiguration` colon-splitting ambiguous (also silently misparses the JSON shapes the UI placeholder already suggests) | Medium | 15 | +| S4 | DelmiaNotifier failover can duplicate a processed notification (at-least-once undocumented) | Medium | 23 | +| S5 | SMS adapter keeps sending after first transient — duplicate amplification on retry | Medium | 17 | +| S6 | `ErrorClassifier.IsTransient(Exception)` treats any OCE as transient (misuse-prone public predicate) | Low | 16 | +| S7 | First-of-many SMTP configuration nondeterministic (`FirstOrDefault` in adapter + retry-policy resolver) | Low | 21 | +| S8 | OAuth2 token cache: unsynchronized field visibility (growth is bounded in practice — accepted) | Low | 21 | +| S9 | Non-JSON content-type with body yields misleading 400 instead of 415 | Low | 5 | +| P1 | Outbound HTTP responses buffered unbounded | Medium | 14 | +| P2 | Serial per-recipient Twilio POSTs inside serial dispatch sweep | Medium | 17 (first-transient short-circuit bounds the black-hole worst case; bounded parallelism **deferred** — at current list sizes short-circuit alone caps the sweep at ~1 timeout) | +| P3 | Per-delivery TCP+TLS SMTP handshake | Low | **Won't-fix** — deliberate, documented tradeoff with the pooling seam already left open (`MailKitSmtpClientWrapper.cs:12-38`); report itself says "correct tradeoff at current volume" | +| P4 | Startup compiles every inbound method serially | Low | 7 | +| C1 | Spec drift: per-system ESG timeout specified but not implemented | High | 8, 9, 10, 11, 12 | +| C2 | Spec drift: path templates (`/recipes/{id}`) never substituted | Medium | 13 | +| C3 | Spec drift: inbound error responses lack documented `code` field (incl. SITE_UNREACHABLE surfacing as generic 500) | Low | 5 | +| C4 | OAuth2 SMTP is Microsoft-365-only despite broader spec | Low | 19, 20 | +| C5 | `InboundApiEndpointFilter` freezes options at construction | Low | 6 | +| U1 | Script trust boundary is static-only (no runtime sandbox) | Underdeveloped | **Deferred** — restricted-ALC/out-of-process containment is an architecture change already logged as deferred to v1.x in the codebase's own deferred-work inventory; interacts with plan 05's Script Analysis ownership. Not a bite-sized fix. | +| U2 | No cache-coherence mechanism for compiled handlers | Underdeveloped | 1, 3 (revision-check fallback works standalone; Task 3 provides the consumer seam for plan 05's invalidation contract) | +| U3 | Per-SMS retry settings are dead fields (`SmsConfiguration.MaxRetries`/`RetryDelay` never read) | Underdeveloped | 18 | +| U4 | Twilio accept-only + `AccountSid` interpolated un-escaped into URI | Underdeveloped | 22 (SID format validation at save). Status-callback webhook / per-recipient delivery state: **Deferred** — documented out-of-scope in `Component-NotificationService.md:93,102`; needs an inbound webhook endpoint (new attack surface) that warrants its own design pass. | +| U5 | DelmiaNotifier duplicate-delivery semantics undocumented; no CI artifact for shipped binary | Underdeveloped | 23 (docs). Windows AOT CI artifact: **Deferred** — repo has no Windows CI runner; the design doc's manual-smoke note stands. | +| U6 | Test gaps: handler failover/staleness, oversized outbound body, mid-list SMS transient | Underdeveloped | 1, 14, 17 (each task's failing-test step is exactly the missing test) | +| U7 | API key rotation procedure unwritten | Underdeveloped | 24 | + +--- + +### Task 1: Revision-checked inbound handler cache (self-healing across failover / direct SQL / known-bad) + +**Classification:** high-risk (concurrency; changes the serving semantics of every inbound API method) +**Estimated implement time:** 5 min +**Parallelizable with:** 6, 7, 8, 17, 18, 23, 24 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundScriptExecutor.cs` (lines 26, 38-63, 118-143, 311-331) +- Test (create): `tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/InboundScriptExecutorStalenessTests.cs` + +Design: the cache value becomes `CachedHandler(string? Script, Func> Handler)`; `_knownBadMethods` becomes `ConcurrentDictionary` (method name → the exact script text that failed). On every `ExecuteAsync`, the freshly-fetched `method.Script` is ordinal-compared against the cached script; mismatch → recompile in place (and purge the known-bad record). A known-bad record only short-circuits when the *current* script is the one that failed — a changed script always gets a fresh compile. `RegisterHandler(name, handler)` (test seam) stores `Script = null`, meaning "no staleness info, serve as-is" — existing tests keep passing; all production compiles flow through `CompileAndRegister`, which always records the script text. This fixes failover staleness, direct-SQL-edit staleness, and stale known-bad in one move; marginal per-request cost is one ordinal string compare of a few-KB script. + +Behavioral consequence (deliberate, see Task 2): the DB row becomes authoritative. A saved-but-broken script now returns HTTP 500 "Script compilation failed" on every node instead of the old delegate silently serving on the node that compiled it — honest failure replaces node-divergent stale success. + +**Steps:** + +1. Write the failing tests: + +```csharp +// tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/InboundScriptExecutorStalenessTests.cs +using Microsoft.Extensions.Logging.Abstractions; +using NSubstitute; +using ZB.MOM.WW.ScadaBridge.Commons.Entities.InboundApi; +using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services; + +namespace ZB.MOM.WW.ScadaBridge.InboundAPI.Tests; + +/// +/// Arch-review S1: the compiled-handler cache must be revision-checked against the +/// freshly-fetched ApiMethod.Script so a standby node (whose cache was populated at +/// its own startup) never serves a stale delegate after failover, and a method fixed +/// out-of-band is never stuck behind a stale known-bad record. +/// +public class InboundScriptExecutorStalenessTests +{ + private readonly InboundScriptExecutor _executor; + private readonly RouteHelper _route; + + public InboundScriptExecutorStalenessTests() + { + _executor = new InboundScriptExecutor( + NullLogger.Instance, Substitute.For()); + _route = new RouteHelper( + Substitute.For(), Substitute.For()); + } + + private Task Run(ApiMethod m) => _executor.ExecuteAsync( + m, new Dictionary(), _route, TimeSpan.FromSeconds(10)); + + [Fact] + public async Task UpdatedScript_RecompilesInsteadOfServingStaleHandler() + { + // Simulates the failover scenario: this node compiled v1 at startup... + var v1 = new ApiMethod("stale", "return 1;") { Id = 1, TimeoutSeconds = 10 }; + Assert.True(_executor.CompileAndRegister(v1)); + var r1 = await Run(v1); + Assert.Contains("1", r1.ResultJson); + + // ...the active node updated the method (this node never heard about it); + // the per-request DB fetch hands ExecuteAsync the NEW row. + var v2 = new ApiMethod("stale", "return 2;") { Id = 1, TimeoutSeconds = 10 }; + var r2 = await Run(v2); + + Assert.True(r2.Success); + Assert.Contains("2", r2.ResultJson); // old cache would have returned 1 + } + + [Fact] + public async Task KnownBadMethod_FixedScript_CompilesInsteadOfFastFailing() + { + // The mirror-image variant: broken at this node's startup... + var broken = new ApiMethod("fixme", "%%% not C# %%%") { Id = 2, TimeoutSeconds = 10 }; + Assert.False(_executor.CompileAndRegister(broken)); + var r1 = await Run(broken); + Assert.False(r1.Success); + + // ...fixed via the management path on the OTHER node; this node's fresh + // DB fetch carries the fixed script. + var fixedMethod = new ApiMethod("fixme", "return 42;") { Id = 2, TimeoutSeconds = 10 }; + var r2 = await Run(fixedMethod); + + Assert.True(r2.Success); + Assert.Contains("42", r2.ResultJson); + } + + [Fact] + public async Task KnownBadMethod_SameScript_StillFastFails() + { + var broken = new ApiMethod("stillbad", "%%% not C# %%%") { Id = 3, TimeoutSeconds = 10 }; + Assert.False(_executor.CompileAndRegister(broken)); + + // Same script text → the fast-fail record must hold (no per-request Roslyn). + var again = new ApiMethod("stillbad", "%%% not C# %%%") { Id = 3, TimeoutSeconds = 10 }; + var r = await Run(again); + Assert.False(r.Success); + Assert.Contains("Script compilation failed", r.ErrorMessage); + } + + [Fact] + public async Task TestSeamRegisterHandler_WithoutScript_IsNotRevisionChecked() + { + // RegisterHandler (script-less test seam) pins the handler regardless of row content. + var m = new ApiMethod("pinned", "return 0;") { Id = 4, TimeoutSeconds = 10 }; + _executor.RegisterHandler("pinned", async _ => { await Task.CompletedTask; return 99; }); + var r = await Run(m); + Assert.True(r.Success); + Assert.Contains("99", r.ResultJson); + } +} +``` + +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests --filter InboundScriptExecutorStalenessTests` — expect **FAIL** (`UpdatedScript_RecompilesInsteadOfServingStaleHandler` returns 1; `KnownBadMethod_FixedScript…` fast-fails). + +3. Minimal implementation in `InboundScriptExecutor.cs`: + - Add `private sealed record CachedHandler(string? Script, Func> Handler);` and change `_scriptHandlers` to `ConcurrentDictionary`. + - Change `_knownBadMethods` to `ConcurrentDictionary`; `TryRecordBadMethod(string methodName, string script)` stores the failing script text (cap logic unchanged; `_knownBadMethods[methodName] = script` when already present so a *newer* bad script replaces the record). + - `RegisterHandler(name, handler)` → `_scriptHandlers[name] = new CachedHandler(null, handler);` (doc-comment it as the script-less test seam, exempt from revision checks). + - `Register(methodName, handler)` → `Register(ApiMethod method, handler)` storing `new CachedHandler(method.Script, handler)`. + - In `ExecuteAsync`, replace the lookup block (lines 311-331) with: + +```csharp +_scriptHandlers.TryGetValue(method.Name, out var cached); +// Revision check: a cached handler compiled from different script text is stale +// (standby-after-failover, direct SQL edit, bundle import). Null Script = test seam, exempt. +if (cached is { Script: not null } + && !string.Equals(cached.Script, method.Script, StringComparison.Ordinal)) +{ + _logger.LogInformation( + "API method {Method} script changed since it was cached; recompiling", method.Name); + cached = null; +} + +if (cached == null) +{ + // Fast-fail only when the CURRENT script text is the one already known bad. + if (_knownBadMethods.TryGetValue(method.Name, out var badScript) + && string.Equals(badScript, method.Script, StringComparison.Ordinal)) + return new InboundScriptResult(false, null, "Script compilation failed for this method"); + + var (compiled, _) = Compile(method); + if (compiled == null) + { + TryRecordBadMethod(method.Name, method.Script ?? string.Empty); + return new InboundScriptResult(false, null, "Script compilation failed for this method"); + } + + _knownBadMethods.TryRemove(method.Name, out _); + cached = new CachedHandler(method.Script, compiled); + _scriptHandlers[method.Name] = cached; // last-write-wins; concurrent first-callers race benignly +} + +var result = await cached.Handler(context).WaitAsync(cts.Token); +``` + + - In `CompileAndRegister` update the failure branch to `TryRecordBadMethod(method.Name, method.Script ?? string.Empty)`. + +4. Run the new filter → **PASS**, then the whole project: `dotnet test tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests` → all green (existing `RegisterHandler`-based tests keep passing via the null-script seam). + +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundScriptExecutor.cs tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/InboundScriptExecutorStalenessTests.cs && git commit -m "fix(inbound-api): revision-check compiled handler cache against the fresh ApiMethod row — heals failover/direct-SQL/known-bad staleness"` + +### Task 2: DB-authoritative compile semantics — management warning text + design doc + +**Classification:** small +**Estimated implement time:** 3 min +**Parallelizable with:** 6, 7, 8, 17, 18, 23 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (`TryCompileAndDescribe`, ~lines 2646-2660: warning strings) +- Modify: `docs/requirements/Component-InboundAPI.md` (the "Direct SQL Warning" section, ~lines 129-131, plus the broken-save behavior description) + +Task 1 changed the truth: after a broken save, the *old* handler no longer keeps serving (the per-request revision check bypasses it and the method 500s with a compile error on every node consistently). The management warning and the component doc must say so, or they now lie. + +**Steps:** + +1. In `TryCompileAndDescribe`, replace the `isUpdate` warning string with: `"Script saved but failed to compile; the method will return a compilation error on every request until a compiling version is saved. Compilation errors: {detail}"` (the create-path string is already correct). Update the `CompileAndRegister` XML doc-comment in `InboundScriptExecutor.cs` (lines 105-114) if Task 1 did not already: the "previously registered handler is left in place" sentence must be replaced with the DB-authoritative behavior. +2. In `Component-InboundAPI.md`, rewrite the Direct SQL Warning section: direct SQL edits are now picked up on the next request via the per-request script revision check (no restart / no management round-trip needed); note that a broken script serves a 500 compile-failure consistently on all central nodes. +3. Verify no stale text remains: `grep -rn "previously registered" src/ZB.MOM.WW.ScadaBridge.ManagementService src/ZB.MOM.WW.ScadaBridge.InboundAPI docs/requirements/Component-InboundAPI.md` — every remaining hit must describe the new semantics. +4. `dotnet build ZB.MOM.WW.ScadaBridge.slnx` → green; `dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests` (some tests may assert the old warning text — update them to the new string). +5. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.ManagementService src/ZB.MOM.WW.ScadaBridge.InboundAPI tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests docs/requirements/Component-InboundAPI.md && git commit -m "docs(inbound-api): DB row is authoritative for method scripts — update broken-save warning + Direct SQL Warning for the revision-checked cache"` + +### Task 3: `InvalidateMethod` seam — the Inbound API consumer of plan 05's invalidation contract + +**Classification:** small +**Estimated implement time:** 3 min +**Parallelizable with:** 8, 9, 10, 13-17, 19, 23, 24 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundScriptExecutor.cs` (after Task 1) +- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (`HandleDeleteApiMethod`, ~line 2631: swap `RemoveHandler` for `InvalidateMethod`) (after Task 2) +- Test: `tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/InboundScriptExecutorStalenessTests.cs` (append) + +Plan 05 designs the general "script artifact changed → invalidate everywhere" contract (bundle import is its consumer to wire). This task gives the Inbound API one obvious, complete entry point: `InvalidateMethod(string methodName)` drops **both** the compiled handler and the known-bad record. Even before plan 05 lands, Task 1's revision check makes invalidation a fast-path optimization rather than a correctness requirement — that is the designed fallback. + +**Steps:** + +1. Failing test (append to `InboundScriptExecutorStalenessTests`): + +```csharp +[Fact] +public async Task InvalidateMethod_DropsHandlerAndKnownBadRecord() +{ + var good = new ApiMethod("inv", "return 5;") { Id = 5, TimeoutSeconds = 10 }; + Assert.True(_executor.CompileAndRegister(good)); + var broken = new ApiMethod("invbad", "%%%") { Id = 6, TimeoutSeconds = 10 }; + Assert.False(_executor.CompileAndRegister(broken)); + + _executor.InvalidateMethod("inv"); + _executor.InvalidateMethod("invbad"); + + Assert.Equal(0, _executor.KnownBadMethodCount); + // After invalidation the method lazily recompiles from the row — still serves. + var r = await Run(good); + Assert.True(r.Success); +} +``` + +2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests --filter InvalidateMethod_DropsHandlerAndKnownBadRecord` → **FAIL** (no such method). +3. Implement: + +```csharp +/// +/// Drops the compiled handler AND any known-bad record for the method. The +/// consumption seam for the script-artifact invalidation contract (bundle import, +/// out-of-band updates, failover activation). Safe to call for unknown names. +/// The per-request revision check in ExecuteAsync remains the correctness +/// fallback when an invalidation is missed. +/// +public void InvalidateMethod(string methodName) +{ + _scriptHandlers.TryRemove(methodName, out _); + _knownBadMethods.TryRemove(methodName, out _); +} +``` + + Keep `RemoveHandler` as a one-line delegate to `InvalidateMethod` (public API preserved), and change `HandleDeleteApiMethod` to call `InvalidateMethod(method.Name)`. +4. Run filter → **PASS**; run full `dotnet test tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests`. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundScriptExecutor.cs src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/InboundScriptExecutorStalenessTests.cs && git commit -m "feat(inbound-api): InvalidateMethod seam purging handler + known-bad — consumer entry point for the script-artifact invalidation contract (plan 05)"` + +### Task 4: Abandoned inbound executions — defer DI-scope disposal + orphan telemetry + +**Classification:** high-risk (concurrency; per-request lifecycle) +**Estimated implement time:** 5 min +**Parallelizable with:** 8-16, 17-22 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundScriptExecutor.cs` (`ExecuteAsync` body + finally, lines 333, 388-394) (after Task 3) +- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Observability/ScadaBridgeTelemetry.cs` (add abandoned-execution counter, mirroring `RecordInboundApiRequest`) +- Test: `tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/InboundScriptExecutorTests.cs` (append) + +On timeout, `WaitAsync` returns but the handler task keeps running; the `finally` disposes the DI scope underneath it → `ObjectDisposedException` background faults, and orphans accumulate invisibly (no rate limiting on this API). Fix: capture the handler task; in `finally`, dispose the scope only if the task completed — otherwise increment an orphan gauge, log, and attach a continuation that observes the fault, disposes the scope, and decrements the gauge. + +**Steps:** + +1. Failing test: + +```csharp +[Fact] +public async Task TimedOutScript_ScopeDisposalDeferred_NoOrphanLeak() +{ + // Handler ignores the token and keeps running past the method timeout. + var release = new TaskCompletionSource(); + _executor.RegisterHandler("slow", async _ => { await release.Task; return 1; }); + var method = new ApiMethod("slow", "unused") { Id = 9, TimeoutSeconds = 1 }; + + var result = await _executor.ExecuteAsync( + method, new Dictionary(), _route, TimeSpan.FromMilliseconds(50)); + + Assert.False(result.Success); + Assert.Contains("timed out", result.ErrorMessage); + Assert.Equal(1, _executor.AbandonedExecutionCount); // orphan is COUNTED + + release.SetResult(); // orphan finishes; continuation cleans up + await Task.Delay(200); + Assert.Equal(0, _executor.AbandonedExecutionCount); // ...and DECREMENTED +} +``` + +2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests --filter TimedOutScript_ScopeDisposalDeferred_NoOrphanLeak` → **FAIL** (no `AbandonedExecutionCount`). +3. Implement in `ExecuteAsync`: + - Add `private long _abandonedExecutions;` and `internal long AbandonedExecutionCount => Interlocked.Read(ref _abandonedExecutions);` (public if the health surface needs it later). + - Hoist the handler task: `Task? handlerTask = null;` declared next to `scope`; replace line 333 with `handlerTask = cached.Handler(context); var result = await handlerTask.WaitAsync(cts.Token);`. + - Replace the `finally` block: + +```csharp +finally +{ + if (handlerTask is null || handlerTask.IsCompleted) + { + scope?.Dispose(); + } + else + { + // The handler outlived the request (non-cooperative script after a + // timeout/client abort). Do NOT dispose the scope underneath it — + // defer cleanup to the task's own completion, and count the orphan + // so repeated timeouts of a slow method are visible. + Interlocked.Increment(ref _abandonedExecutions); + ScadaBridgeTelemetry.RecordInboundAbandonedExecution(method.Name); + _logger.LogWarning( + "Abandoned inbound execution for method {Method} is still running; DI scope disposal deferred", method.Name); + var abandonedScope = scope; + _ = handlerTask.ContinueWith(t => + { + _ = t.Exception; // observe the fault so it never surfaces as unobserved + abandonedScope?.Dispose(); + Interlocked.Decrement(ref _abandonedExecutions); + }, TaskScheduler.Default); + } +} +``` + + - In `ScadaBridgeTelemetry`, add `RecordInboundAbandonedExecution(string method)` incrementing a counter (`scadabridge.inbound_api.abandoned_executions`) with a bounded `method` tag, copying the existing `RecordInboundApiRequest` pattern. +4. Run filter → **PASS**; run full project. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundScriptExecutor.cs src/ZB.MOM.WW.ScadaBridge.Commons/Observability/ScadaBridgeTelemetry.cs tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/InboundScriptExecutorTests.cs && git commit -m "fix(inbound-api): defer DI-scope disposal to handler completion and count abandoned executions — closes the timeout use-after-dispose"` + +### Task 5: Inbound error contract — machine-readable `code` field + SITE_UNREACHABLE + 415 for non-JSON bodies + +**Classification:** standard +**Estimated implement time:** 5 min +**Parallelizable with:** 8-16, 17-22 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.InboundAPI/EndpointExtensions.cs` (all `Results.Json(new { error = … })` sites: lines 111-113, 153-155, 184-194, 212-214, 253-255) +- Modify: `src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundScriptExecutor.cs` (`InboundScriptResult` + catch blocks) (after Task 4) +- Modify: `src/ZB.MOM.WW.ScadaBridge.InboundAPI/RouteHelper.cs` (~line 172: typed exception) +- Modify: `src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundApiEndpointFilter.cs` (503/413 bodies) (after Task 6) +- Modify: `docs/requirements/Component-InboundAPI.md` (error-code table, ~lines 91-97) +- Test: `tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/EndpointExtensionsTests.cs`, `tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/EndpointContentTypeTests.cs` (append) + +Spec documents `{"error": …, "code": "SITE_UNREACHABLE"}`; code emits `{error}` only, and routed site-unreachable failures collapse into a generic 500. Implement the spec (the audience is external integrators). Codes: `UNAUTHORIZED` (401), `NOT_APPROVED` (403 — one shared code for both indistinguishable negatives, preserving enumeration-safety), `INVALID_JSON`, `VALIDATION_FAILED`, `UNSUPPORTED_MEDIA_TYPE` (new 415), `BODY_TOO_LARGE` (413), `STANDBY_NODE` (503), `TIMEOUT`, `SITE_UNREACHABLE`, `SCRIPT_ERROR` (500 catch-all). + +**Steps:** + +1. Failing tests (follow the existing patterns in `EndpointExtensionsTests`/`EndpointContentTypeTests`, which drive the endpoint via a test host): + - `ErrorBodies_CarryMachineReadableCode`: unauthorized request → body has `code == "UNAUTHORIZED"`; in-scope-but-invalid-JSON body → `code == "INVALID_JSON"`; missing required parameter → `code == "VALIDATION_FAILED"`. + - `NonJsonContentType_WithBody_Returns415`: `POST` with `Content-Type: text/plain` and body `"hello"` → HTTP 415, `code == "UNSUPPORTED_MEDIA_TYPE"`. A request with a body and **no** Content-Type header still parses as JSON (lenient, preserves existing callers). + - `RoutedSiteUnreachable_Returns500WithSiteUnreachableCode` (unit-level on the executor): stub `IInstanceRouter.RouteToCallAsync` to return `Success=false, ErrorMessage="Site unreachable"`; script does `await Route.To("X").Call("s")`; assert `result.ErrorCode == "SITE_UNREACHABLE"`. +2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests --filter "ErrorBodies_CarryMachineReadableCode|NonJsonContentType_WithBody_Returns415|RoutedSiteUnreachable"` → **FAIL**. +3. Implement: + - `RouteHelper.cs`: add `public sealed class SiteUnreachableException : InvalidOperationException` and throw it (instead of bare `InvalidOperationException`) when `RouteToCallAsync` reports failure whose message indicates unreachable/no-contact (and from the site-resolution failure path). + - `InboundScriptResult` gains `string? ErrorCode = null` (additive positional-with-default). Catch blocks: timeout → `"TIMEOUT"`; add `catch (SiteUnreachableException ex)` before the generic catch → `new InboundScriptResult(false, null, "Site unreachable", "SITE_UNREACHABLE")`; generic → `"SCRIPT_ERROR"`; compile failure → `"SCRIPT_COMPILE_FAILED"`. + - `EndpointExtensions.cs`: add a local helper `static IResult Error(string code, string message, int status) => Results.Json(new { error = message, code }, statusCode: status);` and route every error return through it. Content-type branch: if `ContentLength > 0` and `ContentType` is non-empty and does not contain `"json"` (ordinal-ignore-case) → `Error("UNSUPPORTED_MEDIA_TYPE", "Content-Type must be application/json", StatusCodes.Status415UnsupportedMediaType)`. Script-failure branch passes `scriptResult.ErrorCode ?? "SCRIPT_ERROR"` through. + - `InboundApiEndpointFilter.cs`: add `code = "STANDBY_NODE"` / `"BODY_TOO_LARGE"` to its two bodies. + - Update the doc's failure-body examples/table to the implemented code set. +4. Run filters → **PASS**; run the full InboundAPI test project (existing error-shape assertions that check only `error` still pass — the field is additive). +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.InboundAPI tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests docs/requirements/Component-InboundAPI.md && git commit -m "feat(inbound-api): machine-readable error codes per spec (incl. SITE_UNREACHABLE) + 415 for non-JSON bodies"` + +### Task 6: `InboundApiEndpointFilter` hot-reads options via `IOptionsMonitor` + +**Classification:** trivial +**Estimated implement time:** 2 min +**Parallelizable with:** 1, 2, 7, 8, 17, 18, 23, 24 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundApiEndpointFilter.cs` (lines 26-38, 66) +- Test: `tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/EndpointGatingTests.cs` (append) + +**Steps:** + +1. Failing test: `BodySizeCap_HotReloads_WithoutRestart` — construct the filter with a test `IOptionsMonitor` whose `CurrentValue` is swapped between two invocations (cap 10 bytes → reject; cap 1 MiB → accept); assert the second request passes. (A minimal `TestOptionsMonitor` stub: implements `CurrentValue` from a mutable field, `Get`→CurrentValue, `OnChange`→null.) +2. Run filter test → **FAIL** (ctor takes `IOptions`). +3. Change the ctor parameter to `IOptionsMonitor options`, store the monitor, and read `_options.CurrentValue.MaxRequestBodyBytes` per request (matches the sibling `AuditWriteMiddleware` convention). +4. Run → **PASS**; `dotnet test tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests`. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundApiEndpointFilter.cs tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/EndpointGatingTests.cs && git commit -m "fix(inbound-api): endpoint filter hot-reads MaxRequestBodyBytes via IOptionsMonitor"` + +### Task 7: Parallelize startup compile of inbound methods + +**Classification:** trivial +**Estimated implement time:** 2 min +**Parallelizable with:** 1, 2, 6, 8, 17, 18, 23, 24 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` (lines 383-387) + +`CompileAndRegister` is safe for concurrent callers (ConcurrentDictionary caches, no shared mutable compile state). Replace the foreach with: + +```csharp +Parallel.ForEach(methods, method => executor.CompileAndRegister(method)); +``` + +**Steps:** No practical unit test (Host composition root; behavior identical, latency only). 1. Make the edit with a comment citing the review finding (serial Roslyn compiles stretch the failover-recovery window at hundreds of methods). 2. `dotnet build ZB.MOM.WW.ScadaBridge.slnx` → green. 3. `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests` → green. 4. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Host/Program.cs && git commit -m "perf(host): compile inbound API methods in parallel at startup"` + +### Task 8: `ExternalSystemDefinition.TimeoutSeconds` — entity + EF mapping + migration + +**Classification:** high-risk (schema migration) +**Estimated implement time:** 5 min +**Parallelizable with:** 1, 2, 6, 7, 17, 23, 24 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Entities/ExternalSystems/ExternalSystemDefinition.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/ExternalSystemConfiguration.cs` +- Create: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/_AddExternalSystemTimeoutSeconds.cs` (+ Designer, + snapshot update — generated) +- Test: `tests/ZB.MOM.WW.ScadaBridge.Commons.Tests` (entity default), `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests` (mapping round-trip, follow the existing external-system repository test pattern) + +**Steps:** + +1. Failing test (Commons.Tests): `ExternalSystemDefinition_TimeoutSeconds_DefaultsToZeroMeaningUseGatewayDefault` — construct the entity, assert `TimeoutSeconds == 0`. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter ExternalSystemDefinition_TimeoutSeconds` → **FAIL** (no property). +2. Add to the entity: + +```csharp +/// +/// Per-system HTTP round-trip timeout in seconds for all method calls +/// (spec: Component-ExternalSystemGateway "Timeout"). 0 = unset — the gateway's +/// configured DefaultHttpTimeout applies. +/// +public int TimeoutSeconds { get; set; } +``` + +3. EF mapping: in `ExternalSystemConfiguration`, add `builder.Property(e => e.TimeoutSeconds).HasDefaultValue(0);`. +4. Generate the migration — **build first**: `dotnet build ZB.MOM.WW.ScadaBridge.slnx` then `dotnet ef migrations add AddExternalSystemTimeoutSeconds --project src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase --startup-project src/ZB.MOM.WW.ScadaBridge.Host`. Open the generated file and verify it is NOT empty (one `AddColumn` on the external-systems table, default 0); if empty, delete the generated files, rebuild, regenerate. +5. Run tests: the Commons filter → **PASS**; `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests` → green. +6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Commons/Entities/ExternalSystems/ExternalSystemDefinition.cs src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase && git add tests/ZB.MOM.WW.ScadaBridge.Commons.Tests && git commit -m "feat(esg): per-system TimeoutSeconds column on ExternalSystemDefinition (spec Component-ExternalSystemGateway Timeout) — entity + EF + migration"` + +### Task 9: Honor the per-system timeout in `ExternalSystemClient.InvokeHttpAsync` + +**Classification:** standard +**Estimated implement time:** 4 min +**Parallelizable with:** 10, 11, 12 (Files disjoint) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemClient.cs` (lines 300-307, 322-323, 345-346) +- Test: `tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs` (append) + +**Steps:** + +1. Failing test (use the project's existing stub-handler `HttpClient` pattern): + +```csharp +[Fact] +public async Task PerSystemTimeout_OverridesGatewayDefault() +{ + // System timeout 1s; handler black-holes for 5s; gateway default is 30s. + var system = new ExternalSystemDefinition("slow", "http://x", "none") { TimeoutSeconds = 1 }; + var method = new ExternalSystemMethod("m", "GET", "/p"); + SetupRepository(system, method); // existing helper pattern + var handler = new DelegateHandler(async (_, ct) => + { + await Task.Delay(TimeSpan.FromSeconds(5), ct); + return new HttpResponseMessage(HttpStatusCode.OK); + }); + _httpClientFactory.CreateClient(Arg.Any()).Returns(new HttpClient(handler)); + var client = new ExternalSystemClient(_httpClientFactory, _repository, NullLogger.Instance); + + var sw = System.Diagnostics.Stopwatch.StartNew(); + var result = await client.CallAsync("slow", "m"); + sw.Stop(); + + Assert.False(result.Success); + Assert.Contains("Timeout", result.ErrorMessage); + Assert.True(sw.Elapsed < TimeSpan.FromSeconds(4), "per-system 1s timeout must apply, not the 30s default"); +} +``` + +2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests --filter PerSystemTimeout_OverridesGatewayDefault` → **FAIL** (~30s default would apply; the test waits 5s then sees OK → success). +3. Implement: before the CTS creation, `var effectiveTimeout = system.TimeoutSeconds > 0 ? TimeSpan.FromSeconds(system.TimeoutSeconds) : _options.DefaultHttpTimeout;` — use `effectiveTimeout` in `new CancellationTokenSource(effectiveTimeout)` and in both timeout error messages (replacing `_options.DefaultHttpTimeout.TotalSeconds`). Update the stale comment block at lines 300-305 ("has no per-system Timeout field yet") to describe the new resolution order. +4. Run → **PASS**; full project run. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemClient.cs tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs && git commit -m "feat(esg): honor per-system TimeoutSeconds in InvokeHttpAsync (falls back to DefaultHttpTimeout)"` + +### Task 10: Carry TimeoutSeconds through the artifact pipeline to sites + +**Classification:** high-risk (message contract + site SQLite schema) +**Estimated implement time:** 5 min +**Parallelizable with:** 9, 11, 12 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Artifacts/ExternalSystemArtifact.cs` (additive trailing param `int TimeoutSeconds = 0`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/ArtifactDeploymentService.cs` (~line 194: pass `es.TimeoutSeconds`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs` (`MigrateSchemaAsync`: `TryAddColumnAsync(connection, "external_systems", "timeout_seconds", "INTEGER NOT NULL DEFAULT 0")`; `StoreExternalSystemAsync`: new `int timeoutSeconds` param + column in the upsert) +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs` (~line 1533: pass `es.TimeoutSeconds`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/SiteExternalSystemRepository.cs` (`MapExternalSystem` + its SELECTs: read `timeout_seconds` into `TimeoutSeconds`) +- Check/Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/SiteReplicationActor.cs` (it re-applies `ReplicateArtifacts` — verify its `StoreExternalSystemAsync` call site compiles with the new parameter) +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests` (site storage round-trip test — follow the existing `StoreExternalSystemAsync` test if present, else add `SiteStorage_ExternalSystem_TimeoutSeconds_RoundTrips`) + +The message contract change is **additive** (trailing positional with default), per the repo's additive-only evolution rule — an old central sending to a new site yields 0 (= use default), and vice versa deserializes cleanly. + +**Steps:** + +1. Failing test (SiteRuntime.Tests, in-memory/temp-file SQLite): store an external system with `timeoutSeconds: 7` via `SiteStorageService.StoreExternalSystemAsync`, read it back via `SiteExternalSystemRepository.GetExternalSystemByNameAsync`, assert `TimeoutSeconds == 7`. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter TimeoutSeconds_RoundTrips` → **FAIL** (compile error / missing column). +2. Make the six edits above: add the artifact record's trailing `int TimeoutSeconds = 0`; thread it central-build → site-store → site-read. The SQLite upsert gains `timeout_seconds = excluded.timeout_seconds`. +3. Run → **PASS**; then `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests` and `dotnet test tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests` (artifact-build assertions may enumerate record fields — update). +4. `dotnet build ZB.MOM.WW.ScadaBridge.slnx` → green (catches any other `ExternalSystemArtifact` construction site). +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Artifacts/ExternalSystemArtifact.cs src/ZB.MOM.WW.ScadaBridge.DeploymentManager/ArtifactDeploymentService.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime tests/ && git commit -m "feat(esg): TimeoutSeconds travels the artifact pipeline (additive contract) into site SQLite so site-side calls honor it"` + +### Task 11: TimeoutSeconds management commands + CLI + +**Classification:** small +**Estimated implement time:** 4 min +**Parallelizable with:** 9, 10, 12 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Management/ExternalSystemCommands.cs` (Create/Update records: trailing `int? TimeoutSeconds = null`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (`HandleCreateExternalSystem` ~1599, `HandleUpdateExternalSystem` ~1612: apply `cmd.TimeoutSeconds ?? 0` / preserve-on-null semantics for update) +- Modify: `src/ZB.MOM.WW.ScadaBridge.CLI/Commands/ExternalSystemCommands.cs` (create/update: `--timeout-seconds` `Option`) +- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests` (append to the existing external-system handler tests); `tests/ZB.MOM.WW.ScadaBridge.CLI.Tests` (option parse test — **note: CLI.Tests is not in the slnx; test the csproj directly**) + +**Steps:** + +1. Failing test (ManagementService.Tests): `CreateExternalSystem_PersistsTimeoutSeconds` — send `CreateExternalSystemCommand(..., TimeoutSeconds: 15)`, assert the persisted definition has `TimeoutSeconds == 15`; and `UpdateExternalSystem_NullTimeout_PreservesExisting`. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests --filter TimeoutSeconds` → **FAIL**. +2. Implement: additive record params; create handler sets `TimeoutSeconds = cmd.TimeoutSeconds ?? 0`; update handler `if (cmd.TimeoutSeconds is { } ts) def.TimeoutSeconds = ts;`. CLI: add the option to both `create` and `update`, pass through, add `--timeout-seconds` to the table projection if the list output enumerates fields. +3. Run → **PASS**; run `dotnet test tests/ZB.MOM.WW.ScadaBridge.CLI.Tests` **directly** (not via slnx). +4. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Management/ExternalSystemCommands.cs src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs src/ZB.MOM.WW.ScadaBridge.CLI tests/ && git commit -m "feat(esg): --timeout-seconds on external-system create/update (management command + CLI, additive contract)"` + +### Task 12: TimeoutSeconds in Central UI form + spec doc sync + +**Classification:** small +**Estimated implement time:** 3 min +**Parallelizable with:** 9, 10, 11 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Design/ExternalSystemForm.razor` (numeric input next to the existing `_maxRetries`/`_retryDelaySeconds` fields; load in `OnInitializedAsync`, save in both create/update branches) +- Modify: `docs/requirements/Component-ExternalSystemGateway.md` (line 38 area + §"Call Timeout & Error Handling" ~line 99: document per-system `TimeoutSeconds` with 0 = gateway `DefaultHttpTimeout` fallback and that it now ships to sites in the artifact) +- Test: `tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests` (bUnit — append to the existing ExternalSystemForm test if one exists; otherwise assert the rendered form contains the `Timeout (seconds)` field) + +**Steps:** + +1. Failing bUnit test: render `ExternalSystemForm` (existing test scaffolding), assert a labeled `Timeout (seconds)` input exists and that saving with value 20 passes `TimeoutSeconds == 20` to the repository/command. Run → **FAIL**. +2. Implement: `private int _timeoutSeconds;` bound field with helper text "0 = use gateway default"; wire load + both save paths (the form writes the entity directly per the existing pattern). Update the doc. +3. Run → **PASS**; `dotnet test tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests`. +4. Commit: `git add src/ZB.MOM.WW.ScadaBridge.CentralUI docs/requirements/Component-ExternalSystemGateway.md tests/ && git commit -m "feat(esg): TimeoutSeconds in ExternalSystem UI form + Component-ExternalSystemGateway doc sync — closes the spec-drift gap"` + +### Task 13: Path-template substitution (`/recipes/{id}`) in `BuildUrl` + +**Classification:** standard +**Estimated implement time:** 5 min +**Parallelizable with:** 10, 11, 12 (NOT 9/14/15/16 — same file; run after 9) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemClient.cs` (`BuildUrl` lines 431-463, `InvokeHttpAsync` body-serialization branch lines 286-298) +- Modify: `docs/requirements/Component-ExternalSystemGateway.md` (path/method-definition section: document `{param}` substitution + consumed-parameter removal + missing-parameter error) +- Test: `tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs` (append) + +Design: `BuildUrl` becomes `BuildUrl(baseUrl, path, parameters, httpMethod, out ISet consumedParams)` (or returns a tuple). Regex `\{([A-Za-z_][A-Za-z0-9_]*)\}` over the trimmed path: each placeholder is replaced with `Uri.EscapeDataString(value?.ToString())` from `parameters` and its name recorded as consumed; a placeholder with no matching parameter (or a null value) throws `ArgumentException("Path template parameter '{id}' has no value …")` — a clear authoring error, mirroring `ValidateHttpMethod`. Consumed names are excluded from the GET/DELETE query string AND from the POST/PUT/PATCH JSON body. + +**Steps:** + +1. Failing tests: + - `PathTemplate_SubstitutesAndConsumesParameter`: method path `/recipes/{id}`, GET, parameters `{ id = "R 1", filter = "x" }` → captured request URI is `.../recipes/R%201?filter=x` (no `id` in the query). + - `PathTemplate_Post_ConsumedParamExcludedFromBody`: POST `/orders/{orderId}/lines`, parameters `{ orderId = 5, qty = 2 }` → body JSON contains `qty` but not `orderId`. + - `PathTemplate_MissingParameter_ThrowsClearError`: GET `/recipes/{id}` with no `id` → `ArgumentException` naming `id`. +2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests --filter PathTemplate` → **FAIL** (URL contains literal `{id}`). +3. Implement as designed (a `private static readonly Regex PathParamRegex` + substitution loop; thread `consumedParams` into the query-string LINQ filter and into a `parameters.Where(p => !consumed.Contains(p.Key))` projection before body serialization). +4. Run → **PASS**; full project run. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemClient.cs docs/requirements/Component-ExternalSystemGateway.md tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs && git commit -m "feat(esg): {param} path-template substitution with consumed-param removal — implements the spec'd /recipes/{id} form"` + +### Task 14: Bound outbound HTTP response buffering + +**Classification:** standard +**Estimated implement time:** 5 min +**Parallelizable with:** 10, 11, 12 (same-file chain: after 13) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemClient.cs` (line 312 `SendAsync`, lines 334-347 body read) +- Modify: `src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemGatewayOptions.cs` (add `MaxResponseBodyBytes`, default `10 * 1024 * 1024`) +- Test: `tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs` (append) + +Design: `SendAsync(request, HttpCompletionOption.ResponseHeadersRead, linkedCts.Token)`; pre-check `Content-Length` header when present (fail fast); then a bounded read (cap+1 pattern): stream into a `MemoryStream` in 16 KiB chunks, aborting once `MaxResponseBodyBytes` is exceeded. Oversize → `PermanentExternalSystemException` ("response exceeded N bytes" — retrying won't shrink it; buffering it into S&F would defeat the cap). + +**Steps:** + +1. Failing test: `OversizedResponseBody_FailsPermanently_WithoutBufferingWholeBody` — options with `MaxResponseBodyBytes = 1024`; stub handler returns 2 MiB of `'x'`; `CallAsync` → `result.Success == false`, error contains `"exceeded"`; and a same-shape `CachedCallAsync` does NOT enqueue to S&F (assert via the existing S&F test double if the suite has one, else assert `WasBuffered == false`). +2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests --filter OversizedResponseBody` → **FAIL** (full body read succeeds). +3. Implement: add the option; replace `ReadAsStringAsync` with a private `ReadBodyBoundedAsync(HttpContent content, long maxBytes, CancellationToken token)` returning `string` or throwing `PermanentExternalSystemException`; keep the existing ordered OCE catch filters around the read. +4. Run → **PASS**; full project run. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs && git commit -m "perf(esg): bound outbound response bodies (ResponseHeadersRead + MaxResponseBodyBytes cap; oversize = permanent failure)"` + +### Task 15: Structured JSON `AuthConfiguration` with legacy colon fallback + +**Classification:** high-risk (auth parsing; existing rows must keep working) +**Estimated implement time:** 5 min +**Parallelizable with:** 10, 11, 12 (same-file chain: after 14) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemClient.cs` (`ApplyAuth`, lines 465-533) +- Modify: `docs/requirements/Component-ExternalSystemGateway.md` (auth-configuration format section) +- Test: `tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs` (append) + +The entity doc-comment ("JSON-serialized authentication configuration") and the UI placeholders (`{"key":"xyz"}` / `{"username":"u","password":"p"}`) already promise JSON — but `ApplyAuth` colon-splits, so a JSON-shaped config is silently misparsed **today** (a `{"key":"xyz"}` apikey config produces header name `{"key"`), and any API key containing `:` breaks. Parse order per branch: trimmed value starting with `{` → JSON (`apikey`: `{"header"?: string, "key": string}`, default header `X-API-Key`; `basic`: `{"username": string, "password": string}`); otherwise the legacy colon-split/bare-key behavior, unchanged. Malformed JSON → the existing "malformed config" warning path (send without header), never an exception. + +**Steps:** + +1. Failing tests: + - `ApiKeyAuth_JsonConfig_SetsConfiguredHeaderAndKeepsColons`: `AuthConfiguration = "{\"header\":\"X-Auth\",\"key\":\"ab:cd:ef\"}"` → captured request has header `X-Auth: ab:cd:ef`. + - `ApiKeyAuth_JsonConfig_DefaultHeader`: `{"key":"xyz"}` → `X-API-Key: xyz` (this is the UI-placeholder shape that is broken today). + - `BasicAuth_JsonConfig_EncodesUsernamePassword`: `{"username":"u","password":"p:w"}` → `Authorization: Basic` of `u:p:w`. + - `ApiKeyAuth_LegacyColonConfig_StillWorks`: `"X-Custom:secret"` → `X-Custom: secret` (backward compatibility pinned). +2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests --filter "JsonConfig|LegacyColonConfig"` → **FAIL**. +3. Implement a private `static bool TryParseJsonAuth(string config, out Dictionary fields)` (System.Text.Json, case-insensitive property names, catch `JsonException` → false) and branch `apikey`/`basic` on it before the legacy split. Log a warning (values never logged) when JSON parsing yields no usable `key`/`username`. Update the doc section with both accepted formats and the deprecation note on the colon form. +4. Run → **PASS**; full project run. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemClient.cs docs/requirements/Component-ExternalSystemGateway.md tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs && git commit -m "fix(esg): structured JSON AuthConfiguration (matches entity doc + UI placeholders) with legacy colon fallback — keys containing ':' no longer misparse"` + +### Task 16: Cancellation-aware `ErrorClassifier.IsTransient` overload + +**Classification:** small +**Estimated implement time:** 3 min +**Parallelizable with:** 10, 11, 12, 17-24 (same-file chain: after 15) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ErrorClassifier.cs` (lines 27-38) +- Modify: `src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemClient.cs` (line 325 call site → pass the caller token) +- Test: `tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ErrorClassifierTests.cs` (append) + +**Steps:** + +1. Failing test: `IsTransient_CallerCancelledOce_IsNotTransient` — `var cts = new CancellationTokenSource(); cts.Cancel(); Assert.False(ErrorClassifier.IsTransient(new OperationCanceledException(), cts.Token)); Assert.True(ErrorClassifier.IsTransient(new OperationCanceledException(), CancellationToken.None));` → **FAIL** (no overload). +2. Implement (mirrors `SmtpErrorClassifier`/`SmsErrorClassifier`): + +```csharp +/// +/// Cancellation-aware overload: a caller-requested cancellation is NOT transient +/// (buffering caller-abandoned work into S&F would be wrong). Prefer this overload +/// at call sites that hold the caller's token. +/// +public static bool IsTransient(Exception exception, CancellationToken cancellationToken) +{ + if (exception is OperationCanceledException && cancellationToken.IsCancellationRequested) + return false; + return IsTransient(exception); +} +``` + + Switch the `catch (Exception ex) when (ErrorClassifier.IsTransient(ex))` filter in `InvokeHttpAsync` to `ErrorClassifier.IsTransient(ex, cancellationToken)` (behavior identical — the ordered filters already peel caller-cancel — but the predicate is now misuse-resistant); add a doc-comment on the old overload pointing to the new one. +3. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests --filter IsTransient_CallerCancelledOce` → **PASS**; full project run. +4. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ErrorClassifierTests.cs && git commit -m "fix(esg): cancellation-aware ErrorClassifier.IsTransient overload — caller-cancelled work can never classify transient"` + +### Task 17: SMS first-transient short-circuit (duplicate amplification + sweep bound) + +**Classification:** standard +**Estimated implement time:** 4 min +**Parallelizable with:** 1-16, 19, 22, 23, 24 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/Delivery/SmsNotificationDeliveryAdapter.cs` (recipient loop, lines 148-172; `RollUp` doc-comment) +- Modify: `docs/requirements/Component-NotificationService.md` (the re-send-to-all note ~line 102: add the short-circuit) +- Test: `tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests/Delivery/` (append to the existing SMS adapter test file) + +Any transient rolls the whole notification up to retry-all, so every recipient accepted *after* the first transient is a guaranteed duplicate text on retry. Breaking the loop on the first transient costs nothing (retry re-attempts everyone) and — because a black-holed endpoint's first request eats the full per-request timeout — also bounds the worst-case sweep occupancy to ~one timeout instead of recipients × timeout, which is the substance of the P2 performance finding at current scale. Bounded parallelism is deferred (see coverage table). + +**Steps:** + +1. Failing test: `MidListTransient_ShortCircuits_RemainingRecipientsNotAttempted` — list of 3 numbers; stub handler: first number → 500 (transient), and a request counter. Assert `DeliveryOutcome.Result == Transient` and **exactly 1** HTTP request was made (current code makes 3). +2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests --filter MidListTransient_ShortCircuits` → **FAIL** (3 requests). +3. Implement: in the `switch`, the `Transient` case sets `firstTransientError ??= attempt.Detail;` then `break`s out of the `foreach` (convert to `goto` -free form: set a flag and `break` the loop after the switch, or restructure to `if/else if`). Add the rationale comment: *"every recipient accepted after the first transient is a guaranteed duplicate on retry (the whole notification re-sends); stopping here also bounds a black-holed endpoint to one timeout per sweep."* Update `RollUp`'s doc-comment and the component doc line. +4. Run → **PASS**; full `dotnet test tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests`. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/Delivery/SmsNotificationDeliveryAdapter.cs docs/requirements/Component-NotificationService.md tests/ && git commit -m "fix(notifications): SMS adapter short-circuits on first transient — kills duplicate amplification and bounds the dispatch sweep"` + +### Task 18: Honor `SmsConfiguration.MaxRetries`/`RetryDelay` — per-type retry policy + +**Classification:** high-risk (behavioral change to dispatch/parking) +**Estimated implement time:** 5 min +**Parallelizable with:** 1-16, 23, 24 (NOT 17/21 — shared test dirs are fine, but 21 edits the same actor file: run 18 before 21) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/NotificationOutboxActor.cs` (`RunDispatchSweepAsync` ~line 319, `DeliverOneAsync` signature, `ResolveRetryPolicyAsync` lines 370-401) +- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Entities/Notifications/SmsConfiguration.cs` (lines 28-45: delete the RESERVED doc-comments, document the live behavior) +- Modify: `docs/requirements/Component-NotificationService.md` (~line 65 deferred note), `CLAUDE.md` (the "Notification Outbox retry reuses central SMTP max-retry-count and fixed interval" decision line → "…for Email; Sms uses SmsConfiguration.MaxRetries/RetryDelay, falling back to the SMTP policy when unset") +- Test: `tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests/NotificationOutboxActorDispatchTests.cs` (append) + +Design: `ResolveRetryPolicyAsync` becomes `ResolveRetryPoliciesAsync` returning a small `RetryPolicies(SmtpPolicy, SmsPolicy)` record, resolved once per sweep: the SMTP policy exactly as today; the SMS policy from `INotificationRepository.GetSmsConfigurationAsync()` with the same non-positive-clamp logic, falling back to the SMTP policy when no SMS config row exists. `DeliverOneAsync` selects by `notification.Type` (`Sms` → SMS policy, everything else → SMTP policy). This is plan 08's flagged dead-field finding, owned here. + +**Steps:** + +1. Failing test (follow the existing dispatch-test scaffolding, which drives the actor with substitute repositories): `SmsNotification_TransientFailure_UsesSmsRetryPolicy` — SMTP config `MaxRetries = 10`; SMS config `MaxRetries = 1, RetryDelay = 5 min`; an SMS notification with `RetryCount = 1` and a transient adapter outcome → row is **Parked** (SMS policy exhausted) not Retrying; a parallel `EmailNotification_StillUsesSmtpPolicy` asserts an email row with `RetryCount = 1` goes to Retrying with the SMTP delay. +2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests --filter "UsesSmsRetryPolicy|StillUsesSmtpPolicy"` → **FAIL** (SMS row retries under SMTP policy). +3. Implement as designed; keep the clamp warnings (parameterized by which config produced them). Update the entity doc-comments, component doc, and the CLAUDE.md decision line in the same commit (docs travel with code). +4. Run → **PASS**; full NotificationOutbox.Tests run. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/NotificationOutboxActor.cs src/ZB.MOM.WW.ScadaBridge.Commons/Entities/Notifications/SmsConfiguration.cs docs/requirements/Component-NotificationService.md CLAUDE.md tests/ && git commit -m "feat(notifications): SMS notifications retry under SmsConfiguration.MaxRetries/RetryDelay (SMTP policy remains the Email + fallback policy) — wires the dead fields"` + +### Task 19: Configurable OAuth2 authority/scope — entity + EF + migration + token service + +**Classification:** high-risk (schema migration + auth flow) +**Estimated implement time:** 5 min +**Parallelizable with:** 1-7, 17, 23, 24 (**migration serialization: run after Task 8** — both regenerate the EF model snapshot) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Entities/Notifications/SmtpConfiguration.cs` (add `string? OAuth2Authority`, `string? OAuth2Scope`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/NotificationConfiguration.cs` (map both nullable columns) +- Create: migration `AddSmtpOAuth2AuthorityScope` (same command pattern as Task 8; **build first**, verify non-empty) +- Modify: `src/ZB.MOM.WW.ScadaBridge.NotificationService/OAuth2TokenService.cs` (`GetTokenAsync` gains optional `string? authority = null, string? scope = null`; defaults preserve today's M365 URL/scope; cache key incorporates authority+scope so two configs with the same credentials but different authorities never share a token) +- Modify: `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/Delivery/EmailNotificationDeliveryAdapter.cs` (line 188: pass `smtpConfig.OAuth2Authority`, `smtpConfig.OAuth2Scope`) +- Test: `tests/ZB.MOM.WW.ScadaBridge.NotificationService.Tests` (append to the OAuth2 token service tests) + +Authority template: the stored value is the full token-endpoint URL with an optional `{tenant}` placeholder (e.g. `https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token`); null → today's hardcoded M365 endpoint. Scope: null → `https://outlook.office365.com/.default`. + +**Steps:** + +1. Failing tests: `GetTokenAsync_CustomAuthorityAndScope_PostsToConfiguredEndpoint` — stub HTTP handler captures the request; call with `authority: "https://idp.example/{tenant}/token", scope: "smtp.send"`; assert the POST went to `https://idp.example/T1/token` with form `scope=smtp.send`. And `GetTokenAsync_DefaultsPreserved_M365` (no authority/scope → today's URL + scope). Run filter → **FAIL** (no parameters). +2. Implement entity + EF + migration (verify non-empty), then the token-service parameters and the adapter call site; cache key: `CredentialKey($"{credentials}|{authority}|{scope}")`. +3. Run → **PASS**; `dotnet test tests/ZB.MOM.WW.ScadaBridge.NotificationService.Tests` and `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests`. +4. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Commons/Entities/Notifications/SmtpConfiguration.cs src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase src/ZB.MOM.WW.ScadaBridge.NotificationService/OAuth2TokenService.cs src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/Delivery/EmailNotificationDeliveryAdapter.cs tests/ && git commit -m "feat(notifications): configurable OAuth2 authority + scope per SMTP configuration (defaults preserve M365) — entity + EF + migration + token service"` + +### Task 20: OAuth2 authority/scope surfaces — management command + CLI + UI + doc + +**Classification:** small +**Estimated implement time:** 4 min +**Parallelizable with:** 9-16, 21, 22 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Management/NotificationCommands.cs` (`UpdateSmtpConfigCommand`: trailing `string? OAuth2Authority = null, string? OAuth2Scope = null`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (the SMTP update handler: apply when non-null) +- Modify: `src/ZB.MOM.WW.ScadaBridge.CLI/Commands/NotificationCommands.cs` (`smtp update`: `--oauth2-authority`, `--oauth2-scope` options, ~line 127+ group) +- Modify: the Central UI SMTP page (`src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Notifications/` — the SMTP form component; two text inputs shown only when AuthType is OAuth2) +- Modify: `docs/requirements/Component-NotificationService.md` (~line 48: the "and other modern SMTP providers" sentence now true — document the two fields and their M365 defaults) +- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests` (handler applies fields), `tests/ZB.MOM.WW.ScadaBridge.CLI.Tests` (option parse — test csproj directly) + +**Steps:** 1. Failing handler test `UpdateSmtpConfig_AppliesOAuth2AuthorityAndScope` → run filter → **FAIL**. 2. Implement the additive command params, handler application (preserve-on-null), CLI options, UI inputs, doc paragraph. 3. Run filters → **PASS**; run ManagementService.Tests + CLI.Tests (direct). 4. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.Commons/Messages src/ZB.MOM.WW.ScadaBridge.ManagementService src/ZB.MOM.WW.ScadaBridge.CLI src/ZB.MOM.WW.ScadaBridge.CentralUI docs/requirements/Component-NotificationService.md tests/ && git commit -m "feat(notifications): OAuth2 authority/scope management + CLI + UI surfaces"` + +### Task 21: Notification low-severity cleanups — deterministic SMTP selection + OAuth2 cache visibility + +**Classification:** small +**Estimated implement time:** 4 min +**Parallelizable with:** 9-16, 20, 22 (after 18 and 19 — shared files) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/Delivery/EmailNotificationDeliveryAdapter.cs` (lines 75-76) +- Modify: `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/NotificationOutboxActor.cs` (retry-policy resolution — post-Task-18 shape) +- Modify: `src/ZB.MOM.WW.ScadaBridge.NotificationService/OAuth2TokenService.cs` (`CacheEntry`) +- Test: `tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests/Delivery/` (append) + +Concrete edits (S7, S8): +1. **Deterministic SMTP pick**: in both call sites replace `FirstOrDefault()` / `configurations[0]` with `configurations.OrderBy(c => c.Id).FirstOrDefault()` and, when `configurations.Count > 1`, log a warning: `"Multiple SMTP configurations exist; using the lowest-Id row ({Id}). Enforce a single row or delete extras."` Failing test first: `MultipleSmtpConfigs_LowestIdWins` — repository returns rows Id 7 then Id 3 (that order); assert the adapter connects with row 3's server. Run → **FAIL** (row 7 picked) → implement → **PASS**. +2. **OAuth2 cache visibility**: replace `CacheEntry`'s mutable `Token`/`Expiry` fields with a single `volatile TokenValue? Current;` where `private sealed record TokenValue(string Token, DateTimeOffset Expiry);` — reads outside the lock see a consistent pair; writes inside the lock assign one reference. (Growth stays bounded by distinct credential triples — accepted per report; no eviction added.) Existing OAuth2 tests must stay green. + +Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests` and `tests/ZB.MOM.WW.ScadaBridge.NotificationService.Tests`. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.NotificationOutbox src/ZB.MOM.WW.ScadaBridge.NotificationService tests/ && git commit -m "fix(notifications): deterministic lowest-Id SMTP config selection (+ multi-row warning); atomic OAuth2 token cache entry"` + +### Task 22: Validate Twilio `AccountSid` format at config save + +**Classification:** small +**Estimated implement time:** 3 min +**Parallelizable with:** 9, 10, 12, 13-17, 20, 21, 23, 24 (after 11 — same `ManagementActor.cs`) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (the `UpdateSmsConfig` handler: reject a non-matching SID) +- Modify: the Central UI SMS form (`/notifications/sms` page component: same client-side validation message) +- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests` (append) + +`AccountSid` is interpolated un-escaped into the Twilio request URI (`SmsNotificationDeliveryAdapter.cs:131`); a validation regex at save time closes the door on URI-injection via a malformed SID. + +**Steps:** + +1. Failing test: `UpdateSmsConfig_MalformedAccountSid_Rejected` — send the SMS-config update command with `AccountSid = "../evil?x="`; assert `ManagementCommandException` containing `"Account SID"`; and `UpdateSmsConfig_ValidSid_Accepted` with `"AC" + new string('a', 32)`. +2. Run filter → **FAIL**. +3. Implement in the handler, before persisting: `if (!System.Text.RegularExpressions.Regex.IsMatch(cmd.AccountSid, "^AC[0-9a-fA-F]{32}$")) throw new ManagementCommandException("Account SID must match Twilio's format: 'AC' followed by 32 hex characters.");` Mirror the message in the UI form's validation. +4. Run → **PASS**; run ManagementService.Tests. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs src/ZB.MOM.WW.ScadaBridge.CentralUI tests/ && git commit -m "fix(notifications): validate Twilio AccountSid format at save — closes the un-escaped URI interpolation door"` + +### Task 23: Document DelmiaNotifier at-least-once semantics + idempotency requirement + +**Classification:** small (documentation) +**Estimated implement time:** 4 min +**Parallelizable with:** 1-22 +**Files:** +- Modify: `docs/plans/2026-06-26-delmia-recipe-notifier-design.md` (after the failover table, ~line 100) +- Modify: `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md` (operator-facing note) +- Modify: `docs/requirements/Component-InboundAPI.md` (authoring note for the `DelmiaRecipeDownload` method) + +The behavior is by design (timeout → failover), so no code change — but the consequence (at-least-once: a connection reset *after* the server processed the POST, or a slow response exceeding `TimeoutSeconds`, both roll over to the next URL and re-POST) is nowhere stated, and the deployed inbound method carries the burden. + +**Steps:** + +1. Add a "Delivery semantics: at-least-once" section to the design doc: enumerate the two duplicate-producing cases; state the requirement — *the `DelmiaRecipeDownload` inbound method MUST be idempotent on `(MachineCode, DownloadPath, WorkOrderNumber)`; a duplicate must return the same `Result`/`ResultText` as the original, not double-apply side effects.* Cross-link from the failover table row for timeout. +2. Mirror one paragraph in the DelmiaNotifier README (`## Delivery semantics`) and add the authoring note to `Component-InboundAPI.md` where the method is referenced. +3. Manual verification step (record the outcome in the commit message): fetch the deployed `DelmiaRecipeDownload` script (`scadabridge api-method get` via the CLI against the target environment, or read the row) and confirm it is idempotent on the natural key; if it is not, file/fix in the environment — this plan only mandates and documents the contract. +4. `grep -n "at-least-once" docs/plans/2026-06-26-delmia-recipe-notifier-design.md src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md` → both hit. +5. Commit: `git add docs/plans/2026-06-26-delmia-recipe-notifier-design.md src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md docs/requirements/Component-InboundAPI.md && git commit -m "docs(delmia): document at-least-once failover semantics + DelmiaRecipeDownload idempotency requirement"` + +### Task 24: Document the dual-key API-key rotation procedure + +**Classification:** trivial (documentation) +**Estimated implement time:** 3 min +**Parallelizable with:** 1, 3-22 (after 2 and 5 — same doc file) +**Files:** +- Modify: `docs/requirements/Component-InboundAPI.md` (new "### Key rotation" subsection under the API-key auth section, ~line 62+) + +**Steps:** + +1. Add the procedure the `sbk__` design already supports: (1) create a new key with the same scope set (`apikey create`), (2) deliver the new token to the caller and confirm cutover (the audit log's Actor field shows which key serves traffic), (3) disable — don't delete — the old key (`apikey update --enabled false`; reversible), (4) delete after a quarantine period. Note that scopes are copied at creation (scope drift between the pair is an operator error) and that both keys verifying simultaneously is the intended overlap window. +2. Verify cross-references: the CLI command names used must match `src/ZB.MOM.WW.ScadaBridge.CLI/Commands/SecurityCommands.cs` (adjust wording to the real verbs). +3. Commit: `git add docs/requirements/Component-InboundAPI.md && git commit -m "docs(inbound-api): dual-key API-key rotation procedure (create -> migrate -> disable -> delete)"` + +--- + +## Dependencies on other plans + +- **Plan 05 (Templates/Deployment/Transport)** designs the general "script artifact changed → invalidate everywhere" contract and owns the **bundle-import** side of cache invalidation. This plan's Task 3 provides the Inbound API consumer entry point (`InboundScriptExecutor.InvalidateMethod`) that plan 05's import path must call for every overwritten `ApiMethod`. **Deliberate resilience:** Task 1's per-request revision check is the correctness fallback — inbound methods heal even if an invalidation call is missed or plan 05 lands later, so there is no hard sequencing requirement in either direction. +- **Plan 04 (Data/Audit backbone)** owns NotificationOutbox dispatcher/audit-row gaps. Tasks 17/18/21 touch `NotificationOutboxActor.cs` and the delivery adapters — if plan 04 is executed concurrently, coordinate on that file (this plan's edits are confined to `RunDispatchSweepAsync`'s policy resolution, `DeliverOneAsync`'s policy selection, and the two adapters). +- **Plan 08** flagged `SmsConfiguration.MaxRetryCount` unused — ownership of the wiring is **here** (Task 18); plan 08 should not duplicate it. +- Tasks 8 and 19 both add EF migrations against the shared model snapshot — they are sequenced (19 after 8) within this plan; if any other plan adds migrations in the same window, serialize migration-generation across plans. + +## Execution order + +**P0 (start immediately, independent):** Task 1 (the High-severity stale-handler fix) and Task 8 (the High-severity spec-gap migration). + +- **Inbound chain:** 1 → 2 → 3 → 4 → 6 → 5 → 24 (2/3 also gate 11's ManagementActor edits ordering; 24 last because it shares `Component-InboundAPI.md` with 2 and 5). +- **ESG chain:** 8 → {9, 10, 11, 12 in parallel} → 13 → 14 → 15 → 16 (9/13/14/15/16 serialize on `ExternalSystemClient.cs`; 10/11/12 are parallel with all of them). +- **Notifications:** 17 (anytime) ; 18 → 21 ; 19 (after 8, migration serialization) → 20 → 21 ; 22 after 11 (shared `ManagementActor.cs`). +- **Docs:** 23 anytime; 24 after 2/5. + +Intra-plan critical path: **1 → 2 → 3 → 4 → 5** (inbound executor file chain) and **8 → 9 → 13 → 14 → 15 → 16** (ESG client file chain) — roughly equal length; run them in parallel lanes. Everything else hangs off those two spines or is free-floating. diff --git a/archreview/plans/PLAN-06-edge-integrations.md.tasks.json b/archreview/plans/PLAN-06-edge-integrations.md.tasks.json new file mode 100644 index 00000000..4704bdfb --- /dev/null +++ b/archreview/plans/PLAN-06-edge-integrations.md.tasks.json @@ -0,0 +1,30 @@ +{ + "planPath": "archreview/plans/PLAN-06-edge-integrations.md", + "tasks": [ + { "id": 1, "subject": "Task 1: Revision-checked inbound handler cache (self-healing across failover / direct SQL / known-bad)", "status": "pending", "blockedBy": [] }, + { "id": 2, "subject": "Task 2: DB-authoritative compile semantics — management warning text + design doc", "status": "pending", "blockedBy": [1] }, + { "id": 3, "subject": "Task 3: InvalidateMethod seam — the Inbound API consumer of plan 05's invalidation contract", "status": "pending", "blockedBy": [1, 2] }, + { "id": 4, "subject": "Task 4: Abandoned inbound executions — defer DI-scope disposal + orphan telemetry", "status": "pending", "blockedBy": [3] }, + { "id": 5, "subject": "Task 5: Inbound error contract — code field + SITE_UNREACHABLE + 415 for non-JSON bodies", "status": "pending", "blockedBy": [4, 6] }, + { "id": 6, "subject": "Task 6: InboundApiEndpointFilter hot-reads options via IOptionsMonitor", "status": "pending", "blockedBy": [] }, + { "id": 7, "subject": "Task 7: Parallelize startup compile of inbound methods", "status": "pending", "blockedBy": [] }, + { "id": 8, "subject": "Task 8: ExternalSystemDefinition.TimeoutSeconds — entity + EF mapping + migration", "status": "pending", "blockedBy": [] }, + { "id": 9, "subject": "Task 9: Honor the per-system timeout in ExternalSystemClient.InvokeHttpAsync", "status": "pending", "blockedBy": [8] }, + { "id": 10, "subject": "Task 10: Carry TimeoutSeconds through the artifact pipeline to sites", "status": "pending", "blockedBy": [8] }, + { "id": 11, "subject": "Task 11: TimeoutSeconds management commands + CLI", "status": "pending", "blockedBy": [8, 3] }, + { "id": 12, "subject": "Task 12: TimeoutSeconds in Central UI form + spec doc sync", "status": "pending", "blockedBy": [8] }, + { "id": 13, "subject": "Task 13: Path-template substitution (/recipes/{id}) in BuildUrl", "status": "pending", "blockedBy": [9] }, + { "id": 14, "subject": "Task 14: Bound outbound HTTP response buffering", "status": "pending", "blockedBy": [13] }, + { "id": 15, "subject": "Task 15: Structured JSON AuthConfiguration with legacy colon fallback", "status": "pending", "blockedBy": [14] }, + { "id": 16, "subject": "Task 16: Cancellation-aware ErrorClassifier.IsTransient overload", "status": "pending", "blockedBy": [15] }, + { "id": 17, "subject": "Task 17: SMS first-transient short-circuit (duplicate amplification + sweep bound)", "status": "pending", "blockedBy": [] }, + { "id": 18, "subject": "Task 18: Honor SmsConfiguration.MaxRetries/RetryDelay — per-type retry policy", "status": "pending", "blockedBy": [] }, + { "id": 19, "subject": "Task 19: Configurable OAuth2 authority/scope — entity + EF + migration + token service", "status": "pending", "blockedBy": [8] }, + { "id": 20, "subject": "Task 20: OAuth2 authority/scope surfaces — management command + CLI + UI + doc", "status": "pending", "blockedBy": [19] }, + { "id": 21, "subject": "Task 21: Notification low-severity cleanups — deterministic SMTP selection + OAuth2 cache visibility", "status": "pending", "blockedBy": [18, 19] }, + { "id": 22, "subject": "Task 22: Validate Twilio AccountSid format at config save", "status": "pending", "blockedBy": [11] }, + { "id": 23, "subject": "Task 23: Document DelmiaNotifier at-least-once semantics + idempotency requirement", "status": "pending", "blockedBy": [] }, + { "id": 24, "subject": "Task 24: Document the dual-key API-key rotation procedure", "status": "pending", "blockedBy": [2, 5] } + ], + "lastUpdated": "2026-07-08" +} diff --git a/archreview/plans/PLAN-07-ui-management-security.md b/archreview/plans/PLAN-07-ui-management-security.md new file mode 100644 index 00000000..211fcada --- /dev/null +++ b/archreview/plans/PLAN-07-ui-management-security.md @@ -0,0 +1,1262 @@ +# UI, Management & Security Hardening Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. + +**Goal** — Close every finding from architecture review 07 (`archreview/07-ui-management-security.md`): kill the auth-disabled production artifact, honor per-site artifact deployment with a site-scope check, elide connection/external-system/DB secrets everywhere SMTP/SMS already are, make secured-write expiry real server-side, fix the 30s-vs-5min Ask-timeout mismatch, give the CLI actor-side browse/verify/cert-trust parity, canonicalize role casing, throttle LDAP binds, and freeze the authorization table with a matrix test. + +**Architecture** — All fixes stay inside the existing management/presentation seam: the `ManagementActor` (single dispatch switch + `GetRequiredRole` gate + `PipeTo`), the `ManagementAuthenticator` Basic→LDAP flow shared by every CLI surface, the Security component's cookie/JWT/policy layer, and the Blazor Central UI. New primitives are small and testable in isolation (`DisableLoginGuard`, `ConfigSecretScrubber`, `LoginThrottle`, `PollGate`), wired at the existing composition points. Message-contract changes are strictly additive (optional trailing record params, new enum members) per the repo's contract-evolution rule; every spec-affecting change updates `docs/requirements/Component-Security.md` / `Component-ManagementService.md` / `Component-CentralUI.md` / `Component-CLI.md` in the same task as the code. + +**Tech Stack** — C#/.NET 10, Akka.NET (TestKit.Xunit2 + NSubstitute for actor tests), ASP.NET Core minimal endpoints + Blazor Server (bUnit), EF Core (ConfigurationDatabase), System.CommandLine (CLI). Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per-project: `dotnet test tests/` — **CLI tests must be run directly** (`dotnet test tests/ZB.MOM.WW.ScadaBridge.CLI.Tests`) until Task 33 lands. + +--- + +## Findings Coverage + +| Finding | Severity | Disposition | +|---|---|---| +| C1 — `DisableLogin: true` in production deploy artifact, no environment guard | Critical | Tasks 1, 2, 3 | +| C2 — `deploy artifacts --site-id` silently ignored; no site-scope check | High | Tasks 4, 5 | +| C3 — DataConnection/ExternalSystem/DB-connection secrets leak in List/Get + audit | High | Tasks 6, 7, 8, 9 | +| S1 — 30s server Ask vs 5-min bundle/deploy operations; import keeps applying after 504 | High | Tasks 19, 20. Client-supplied import-id idempotency: **Deferred** — Transport apply-path change, plan 05's domain (coordinate; single-flight in Task 30 also narrows the double-apply window) | +| S2 — Pending secured writes never expire; "Expired" status fictional | High | Tasks 14, 15, 16, 18 | +| S3 — Poll timers can overlap their own refreshes | Medium | Task 29 | +| S4 — 200 MB bodies buffered as strings, ~4 copies resident | Medium | Task 30 (single-flight + copy elimination). Streaming multipart upload: **Deferred** — transport re-plumb of the JSON envelope; bounded risk after Task 30 | +| S5 — Fire-and-forget SignalR sends swallow failures | Low | Task 31 | +| P1 — Full LDAP bind per request, no throttle/lockout (password-spray oracle) | Medium | Tasks 26, 27 | +| P2 — Unbounded list handlers materialize whole tables | Medium | Task 28 (ListTemplates/ListInstances additive paging). QueryDeployments/ExportBundle internals: **Deferred** — bounded by fleet size today; needs repo-level projections, low risk | +| P3 — `ListSecuredWrites` hard-caps at 200, no paging | Medium | Tasks 14, 17, 18 | +| P4 — Alarm Summary re-sorts/filters per render | Low | Task 32 | +| C4 — Browse/Verify/Cert commands doc'd as actor commands but UI-only (no CLI parity) | Medium | Tasks 21, 22, 23, 24, 25 | +| C5 — Role-casing asymmetry (UI ordinal vs actor ignore-case), acknowledged deferral | Medium | Task 12 | +| C6 — Area role gate: code says Designer, three docs say Admin, UI page says Deployer | Medium | Task 11 | +| C7 — `InstanceName` passed into slot commented "InstanceId filter" | Low | Task 31 (verified: site `instance_id` column stores the instance **UniqueName** — `InstanceActor.LogLifecycleEvent` passes `_instanceUniqueName` — so behavior is correct; comment/xmldoc are wrong) | +| C8 — Stale CLAUDE.md note on SecuredWrite `SourceNode` | Low | Task 31 | +| UA1 — Secured-write lifecycle (no expiry, no paging, history ungated, dual-role hazard) | UA | Tasks 10, 14–18; dual-role deployment hazard restated in docs in Task 15 | +| UA2 — ManagementService test depth (~150 commands, no authz matrix test) | UA | Tasks 13, 24 (+ handler tests added by every task above) | +| UA3 — CLI.Tests not in slnx | UA | Task 33 | +| UA4 — No single tracking list for deferred follow-ups | UA | **Deferred to plan 08**, which owns the consolidated 23-item deferred-work inventory | +| UA5 — Login endpoints lack throttling; user-enumeration check | UA | Tasks 26, 27. Enumeration check **verified safe, no change**: `LdapAuthFailureMessages` maps `UserNotFound` and `BadCredentials` to the identical "Invalid username or password." | +| UA6 — Accessibility debt on sortable headers/custom grids | UA | Task 32 (AlarmSummary, the cited file). Fleet-wide grid a11y sweep: **Deferred** — uniform low-severity UX debt, log as follow-up | +| UA7 — "Committed" runtime logs under docker topology | UA | **Won't-fix** — verified false premise: `git check-ignore docker/central-node-a/logs` → ignored, `git ls-files docker | grep -i log` → empty. Logs are untracked local output; nothing to scrub | + +--- + +### Task 1: `DisableLoginGuard` — refuse the flag outside Development + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** 4, 6, 14, 19, 21, 26, 29 +**Files:** +- Create: `src/ZB.MOM.WW.ScadaBridge.Security/Auth/DisableLoginGuard.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.Security/Auth/AuthDisableLoginOptions.cs` (add `AllowOutsideDevelopment` property) +- Test: `tests/ZB.MOM.WW.ScadaBridge.Security.Tests/DisableLoginGuardTests.cs` (create) + +1. Write the failing test: +```csharp +using ZB.MOM.WW.ScadaBridge.Security.Auth; +using Xunit; + +namespace ZB.MOM.WW.ScadaBridge.Security.Tests; + +public class DisableLoginGuardTests +{ + [Theory] + [InlineData("Production")] + [InlineData("Staging")] + [InlineData("")] + public void Validate_DisableLoginOutsideDevelopment_WithoutAck_Throws(string env) + { + var ex = Assert.Throws( + () => DisableLoginGuard.Validate(disableLogin: true, environmentName: env, allowOutsideDevelopment: false)); + Assert.Contains("DisableLogin", ex.Message); + Assert.Contains("Development", ex.Message); + } + + [Theory] + [InlineData("Development", false)] // dev env: allowed without ack + [InlineData("development", false)] // case-insensitive env name + [InlineData("Production", true)] // explicit second ack key + public void Validate_Allowed_DoesNotThrow(string env, bool ack) + => DisableLoginGuard.Validate(true, env, ack); + + [Fact] + public void Validate_FlagOff_NeverThrows() + => DisableLoginGuard.Validate(false, "Production", false); +} +``` +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Security.Tests --filter DisableLoginGuardTests` — expect FAIL (type missing). +3. Implement `DisableLoginGuard.cs`: +```csharp +namespace ZB.MOM.WW.ScadaBridge.Security.Auth; + +/// +/// Startup guard for the dev/test flag. +/// DisableLogin authenticates EVERY request with ALL roles (including Operator+Verifier, +/// nullifying the two-person secured-write control) on a SCADA control surface, so it is +/// refused outside the Development environment unless the operator sets the explicit +/// second acknowledgement key . +/// Called by the Host composition root before AddSecurity(); fail-fast at boot. +/// +public static class DisableLoginGuard +{ + public static void Validate(bool disableLogin, string environmentName, bool allowOutsideDevelopment) + { + if (!disableLogin) return; + if (string.Equals(environmentName, "Development", StringComparison.OrdinalIgnoreCase)) return; + if (allowOutsideDevelopment) return; + throw new InvalidOperationException( + "ScadaBridge:Security:Auth:DisableLogin=true disables ALL authentication on a SCADA control " + + $"surface and is refused outside the Development environment (current: '{environmentName}'). " + + "Remove the flag, run with ASPNETCORE_ENVIRONMENT=Development, or — only if you fully accept an " + + "unauthenticated deployment — additionally set " + + "ScadaBridge:Security:Auth:AllowOutsideDevelopment=true."); + } +} +``` +Add to `AuthDisableLoginOptions`: +```csharp +/// Explicit second acknowledgement key: permits DisableLogin outside Development. Default false. +public bool AllowOutsideDevelopment { get; set; } +``` +4. Run the filter again — expect PASS. Run full project: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Security.Tests`. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Security/Auth tests/ZB.MOM.WW.ScadaBridge.Security.Tests/DisableLoginGuardTests.cs && git commit -m "feat(security): DisableLoginGuard refuses DisableLogin outside Development without explicit ack"` + +### Task 2: Wire the guard at the Host composition root + doc + +**Classification:** high-risk +**Estimated implement time:** ~4 min +**Parallelizable with:** none (waits on Task 1) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` (~lines 130–138, immediately after `disableLogin` is read) +- Modify: `docs/requirements/Component-Security.md` §Dev Disable-Login Flag (~line 27) +- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/` — inspect harness; if no config-composition test seam exists, verification is the Security.Tests guard suite (Task 1) + build + step 4 smoke + +1. In `Program.cs`, after line 137 (`GetValue(...DisableLogin)`), insert: +```csharp +// Fail-fast guard: DisableLogin outside Development requires the explicit +// AllowOutsideDevelopment acknowledgement key. See DisableLoginGuard. +ZB.MOM.WW.ScadaBridge.Security.Auth.DisableLoginGuard.Validate( + disableLogin, + builder.Environment.EnvironmentName, + builder.Configuration + .GetSection(ZB.MOM.WW.ScadaBridge.Security.Auth.AuthDisableLoginOptions.SectionName) + .GetValue(nameof(ZB.MOM.WW.ScadaBridge.Security.Auth.AuthDisableLoginOptions.AllowOutsideDevelopment))); +``` +2. Update `Component-Security.md` §Dev Disable-Login Flag: document the guard, the `AllowOutsideDevelopment` ack key, and fix the stale "all four roles" → "all six roles (Administrator, Designer, Deployer, Viewer, Operator, Verifier)" while in the section. +3. `dotnet build ZB.MOM.WW.ScadaBridge.slnx` — expect clean. +4. Smoke (docker cluster is unaffected — `docker/docker-compose.yml` sets `ASPNETCORE_ENVIRONMENT: Development` and both central nodes have `DisableLogin: false`): optionally run the Host locally with `ASPNETCORE_ENVIRONMENT=Production` and `ScadaBridge__Security__Auth__DisableLogin=true`, assert it exits with the guard message. +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Host/Program.cs docs/requirements/Component-Security.md && git commit -m "feat(host): fail-fast DisableLogin environment guard at composition root"` + +### Task 3: Scrub `deploy/wonder-app-vd03` — remove `DisableLogin: true` + +**Classification:** small +**Estimated implement time:** ~3 min +**Parallelizable with:** 1, 4, 6, 14, 19, 21, 26, 29 +**Files:** +- Modify: `deploy/wonder-app-vd03/appsettings.Central.json` (lines 39–42 `"Auth"` block; line 44 `_comment_Security`) +- Modify: `deploy/wonder-app-vd03/RUNBOOK.md` (add a login note) + +1. Delete the block: +```json + "Auth": { + "DisableLogin": true, + "User": "multi-role" + } +``` +(and the now-trailing comma on `"RequireHttpsCookie": false`'s following line structure — validate JSON). +2. Append to `_comment_Security`: "Login is REQUIRED on this host: authenticate as one of the GLAuth test users (e.g. multi-role/password) or repoint to corporate AD. DisableLogin is refused outside Development by a startup guard." +3. In `RUNBOOK.md`, add a short "Authentication" note: the UI (:8080) and management API (:8085) now require credentials; CLI calls need `--username/--password`; the previous unauthenticated posture was removed (arch-review C1). +4. Validate: `python3 -c "import json;json.load(open('deploy/wonder-app-vd03/appsettings.Central.json'))"` — expect no error. +5. Commit: `git add deploy/wonder-app-vd03 && git commit -m "fix(deploy): remove DisableLogin=true from wonder-app-vd03 production artifact (arch-review C1)"` +6. Note for the operator relaying this plan: the user-memory file `wonder-app-vd03-host-access.md` records "DisableLogin=true makes :8085 /management unauthenticated" — that note becomes stale on the next redeploy of this artifact. + +### Task 4: `ArtifactDeploymentService.DeployToSiteAsync` + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 3, 6, 14, 19, 21, 26, 29 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/ArtifactDeploymentService.cs` (refactor `DeployToAllSitesAsync` ~lines 227–345) +- Test: `tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests/` — extend the existing `ArtifactDeploymentService*` test file (locate with `grep -rl ArtifactDeploymentService tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests`) + +1. Write failing tests (mirror the existing harness's mocks for `ISiteRepository`/`CommunicationService`): +```csharp +[Fact] +public async Task DeployToSiteAsync_DeploysOnlyToTargetSite() +{ + // harness: two sites (Id 1 "SITE1", Id 2 "SITE2") in the site repo mock + var result = await _service.DeployToSiteAsync(1, "tester"); + Assert.True(result.IsSuccess); + Assert.Single(result.Value.SiteResults); + Assert.Equal("SITE1", result.Value.SiteResults[0].SiteId); + await _communicationService.Received(1).DeployArtifactsAsync( + "SITE1", Arg.Any(), Arg.Any()); + await _communicationService.DidNotReceive().DeployArtifactsAsync( + "SITE2", Arg.Any(), Arg.Any()); +} + +[Fact] +public async Task DeployToSiteAsync_UnknownSite_ReturnsFailure() +{ + var result = await _service.DeployToSiteAsync(999, "tester"); + Assert.False(result.IsSuccess); + Assert.Contains("999", result.Error); +} +``` +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests --filter DeployToSiteAsync` — expect FAIL (no such method). +3. Implement: extract the body of `DeployToAllSitesAsync` after the `sites` fetch into `private async Task> DeployCoreAsync(IReadOnlyList sites, string user, CancellationToken ct)` (everything from `deploymentId` creation down, unchanged). Then: +```csharp +public async Task> DeployToAllSitesAsync( + string user, CancellationToken cancellationToken = default) +{ + var sites = await _siteRepo.GetAllSitesAsync(cancellationToken); + if (sites.Count == 0) + return Result.Failure("No sites configured."); + return await DeployCoreAsync(sites, user, cancellationToken); +} + +/// Deploys system-wide artifacts to a single site (management-API --site-id path). +public async Task> DeployToSiteAsync( + int siteId, string user, CancellationToken cancellationToken = default) +{ + var site = await _siteRepo.GetSiteByIdAsync(siteId, cancellationToken); + if (site is null) + return Result.Failure($"Site with ID {siteId} not found."); + return await DeployCoreAsync([site], user, cancellationToken); +} +``` +(Match the actual `ISiteRepository.GetSiteByIdAsync` signature — check whether it takes a CancellationToken.) +4. Run the filter — expect PASS. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests` (no regression in the all-sites tests). +5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.DeploymentManager tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests && git commit -m "feat(deployment): DeployToSiteAsync for single-site artifact deployment"` + +### Task 5: `HandleDeployArtifacts` honors `SiteId` + site-scope enforcement + docs + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** none (waits on Task 4; touches ManagementActor.cs — serialize with 7–13, 15–18, 22, 23, 30) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (`HandleDeployArtifacts` lines 2006–2013; dispatch arm ~line where `MgmtDeployArtifactsCommand cmd =>` appears, currently passes `user.Username` — pass `user`) +- Modify: `docs/requirements/Component-ManagementService.md` (deployment commands section), `docs/requirements/Component-CLI.md` + `src/ZB.MOM.WW.ScadaBridge.CLI/README.md` (`--site-id` semantics: now honored, and fleet-wide requires system-wide deployment rights) +- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/ManagementActorTests.cs` + +1. Write failing tests (using the existing `Envelope`/`ScopedEnvelope` helpers; register a substituted `ArtifactDeploymentService` in `_services` — if it is a concrete class, extract-or-virtualize is out of scope: register the real class with mocked dependencies exactly as the DeploymentManager tests construct it, or Substitute if members are virtual; follow whatever `SecuredWriteHandlerTests` does for `CommunicationService`): +```csharp +[Fact] +public void DeployArtifacts_WithSiteId_CallsSingleSiteDeploy() +{ + var actor = CreateActor(); + actor.Tell(Envelope(new MgmtDeployArtifactsCommand(SiteId: 7), "Deployer")); + ExpectMsg(TimeSpan.FromSeconds(5)); + _artifactService.Received(1).DeployToSiteAsync(7, "testuser", Arg.Any()); +} + +[Fact] +public void DeployArtifacts_SiteScopedUser_OutOfScopeSite_Unauthorized() +{ + var actor = CreateActor(); + actor.Tell(ScopedEnvelope(new MgmtDeployArtifactsCommand(SiteId: 7), ["3"], "Deployer")); + ExpectMsg(TimeSpan.FromSeconds(5)); +} + +[Fact] +public void DeployArtifacts_SiteScopedUser_FleetWide_Unauthorized() +{ + var actor = CreateActor(); + actor.Tell(ScopedEnvelope(new MgmtDeployArtifactsCommand(SiteId: null), ["3"], "Deployer")); + var resp = ExpectMsg(TimeSpan.FromSeconds(5)); + Assert.Contains("fleet-wide", resp.Message, StringComparison.OrdinalIgnoreCase); +} +``` +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests --filter DeployArtifacts` — expect FAIL. +3. Implement: +```csharp +private static async Task HandleDeployArtifacts( + IServiceProvider sp, MgmtDeployArtifactsCommand cmd, AuthenticatedUser user) +{ + // A site-scoped Deployer may not trigger a FLEET-WIDE artifact deployment: + // EnforceSiteScope(null) is a no-op by design, so the fleet-wide case needs + // its own guard (arch-review C2 authorization gap). + if (cmd.SiteId is null + && user.PermittedSiteIds.Length > 0 + && !user.Roles.Contains(Roles.Administrator, StringComparer.OrdinalIgnoreCase)) + { + throw new SiteScopeViolationException( + "Site-scoped users cannot deploy artifacts fleet-wide; specify a target site id."); + } + EnforceSiteScope(user, cmd.SiteId); + + var svc = sp.GetRequiredService(); + var result = cmd.SiteId is int siteId + ? await svc.DeployToSiteAsync(siteId, user.Username) + : await svc.DeployToAllSitesAsync(user.Username); + return result.IsSuccess + ? result.Value + : throw new ManagementCommandException(result.Error); +} +``` +Change the dispatch arm from `HandleDeployArtifacts(sp, cmd, user.Username)` to `HandleDeployArtifacts(sp, cmd, user)`. (Check `SiteScopeViolationException`'s ctor; match `EnforceSiteScope`'s usage.) +4. Run the filter — expect PASS; run the full project. +5. Update the three docs (step Files list) — `--site-id` now deploys to exactly that site; fleet-wide requires system-wide Deployer or Administrator. +6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests docs/requirements/Component-ManagementService.md docs/requirements/Component-CLI.md src/ZB.MOM.WW.ScadaBridge.CLI/README.md && git commit -m "fix(management): honor MgmtDeployArtifactsCommand.SiteId + enforce site scope (arch-review C2)"` + +### Task 6: `ConfigSecretScrubber` — JSON secret elision + sentinel merge + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 3, 4, 14, 19, 21, 26, 29 +**Files:** +- Create: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ConfigSecretScrubber.cs` +- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/ConfigSecretScrubberTests.cs` (create) + +1. Failing tests: +```csharp +public class ConfigSecretScrubberTests +{ + [Fact] + public void Scrub_RedactsPasswordFields_AtAnyDepth() + { + var json = """{"Endpoint":"opc.tcp://x","UserIdentity":{"Username":"u","Password":"p"},"CertificatePassword":"cp"}"""; + var (scrubbed, had) = ConfigSecretScrubber.Scrub(json); + Assert.True(had); + Assert.DoesNotContain("\"p\"", scrubbed); + Assert.DoesNotContain("cp", scrubbed); + Assert.Contains(ConfigSecretScrubber.Sentinel, scrubbed); + Assert.Contains("opc.tcp://x", scrubbed); // non-secrets untouched + } + + [Fact] + public void Scrub_NoSecrets_ReturnsFalseFlag() + { + var (_, had) = ConfigSecretScrubber.Scrub("""{"Endpoint":"opc.tcp://x"}"""); + Assert.False(had); + } + + [Theory] + [InlineData(null)] + [InlineData("")] + [InlineData("not-json{")] + public void Scrub_NullEmptyOrMalformed_PassesThrough(string? input) + { + var (scrubbed, had) = ConfigSecretScrubber.Scrub(input); + Assert.Equal(input, scrubbed); + Assert.False(had); + } + + [Fact] + public void MergeSentinels_RestoresStoredSecrets_KeepsEdits() + { + var stored = """{"Endpoint":"old","UserIdentity":{"Password":"real"}}"""; + var incoming = $$"""{"Endpoint":"new","UserIdentity":{"Password":"{{ConfigSecretScrubber.Sentinel}}"}}"""; + var merged = ConfigSecretScrubber.MergeSentinels(incoming, stored)!; + Assert.Contains("\"new\"", merged); + Assert.Contains("\"real\"", merged); + Assert.DoesNotContain(ConfigSecretScrubber.Sentinel, merged); + } +} +``` +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests --filter ConfigSecretScrubberTests` — FAIL. +3. Implement with `System.Text.Json.Nodes.JsonNode` recursion: +```csharp +/// +/// Elides secret-bearing values inside opaque connection-config JSON blobs before +/// they leave the management boundary (responses AND audit afterState), mirroring +/// the SmtpConfigPublicShape/SmsConfigPublicShape pattern for structured entities. +/// A property is secret-bearing when its name contains one of the fragments below +/// (case-insensitive) and its value is a JSON string. MergeSentinels lets a client +/// round-trip a scrubbed config through update without wiping stored secrets. +/// +internal static class ConfigSecretScrubber +{ + public const string Sentinel = "***REDACTED***"; + private static readonly string[] SecretNameFragments = + ["password", "secret", "token", "apikey", "credential", "passphrase"]; + // Scrub: parse (return input unchanged on parse failure), walk objects/arrays, + // replace matching string values with Sentinel, return (json, hadSecrets). + // MergeSentinels: walk incoming; wherever value == Sentinel, substitute the value + // at the same path in storedJson (if absent, leave Sentinel — update will store it + // verbatim, which is inert). Null/empty inputs pass through. +} +``` +4. Run — PASS. 5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ManagementService/ConfigSecretScrubber.cs tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/ConfigSecretScrubberTests.cs && git commit -m "feat(management): ConfigSecretScrubber for connection-config secret elision"` + +### Task 7: DataConnection List/Get/audit secret elision + sentinel-preserving update + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** none (ManagementActor.cs; waits on Task 6) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (lines 1401–1456: `HandleListDataConnections`, `HandleGetDataConnection`, `HandleCreateDataConnection`, `HandleUpdateDataConnection`) +- Modify: `docs/requirements/Component-ManagementService.md` (data-connection command responses are secret-elided; update merges the redaction sentinel) +- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/ManagementActorTests.cs` (or a new `SecretProjectionTests.cs`) + +1. Failing tests (harness: `ISiteRepository` substitute returning a `DataConnection` whose `PrimaryConfiguration` contains `"Password":"opc-secret"`): +```csharp +[Fact] +public void GetDataConnection_ResponseOmitsPasswords() +{ + var actor = CreateActor(); + actor.Tell(Envelope(new GetDataConnectionCommand(1), "Viewer")); + var resp = ExpectMsg(TimeSpan.FromSeconds(5)); + Assert.DoesNotContain("opc-secret", resp.JsonData); + Assert.Contains("hasSecrets", resp.JsonData); // camelCase serializer +} + +[Fact] +public void UpdateDataConnection_SentinelConfig_PreservesStoredSecret() +{ + // stored config has Password "opc-secret"; incoming config carries the sentinel + var actor = CreateActor(); + var incoming = """{"Endpoint":"opc.tcp://new","UserIdentity":{"Password":"***REDACTED***"}}"""; + actor.Tell(Envelope(new UpdateDataConnectionCommand(1, "c1", "OpcUa", incoming, null, 3), "Designer")); + ExpectMsg(TimeSpan.FromSeconds(5)); + await _siteRepo.Received(1).UpdateDataConnectionAsync( + Arg.Is(c => c.PrimaryConfiguration!.Contains("opc-secret") + && !c.PrimaryConfiguration.Contains("***REDACTED***"))); +} + +[Fact] +public void UpdateDataConnection_AuditAfterState_OmitsPasswords() +{ + var actor = CreateActor(); + actor.Tell(Envelope(new UpdateDataConnectionCommand(1, "c1", "OpcUa", "{\"Password\":\"newpw\"}", null, 3), "Designer")); + ExpectMsg(TimeSpan.FromSeconds(5)); + // AuditAsync serializes afterState via IAuditService — assert the captured object graph has no "newpw" + _auditService.Received().LogAsync(Arg.Any(), "Update", "DataConnection", + Arg.Any(), Arg.Any(), + Arg.Is(o => !ManagementActor.SerializeResult(o).Contains("newpw")), Arg.Any()); +} +``` +(Match `UpdateDataConnectionCommand`'s and `IAuditService.LogAsync`'s real signatures — read them first.) +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests --filter DataConnection` — new tests FAIL. +3. Implement in `ManagementActor.cs`: +```csharp +/// Secret-elided projection of a DataConnection (arch-review C3): the raw +/// PrimaryConfiguration/BackupConfiguration JSON can embed OPC UA user + certificate +/// passwords, and List/Get are readable by any authenticated user. Mirrors +/// SmtpConfigPublicShape. Non-config fields are projected verbatim. +private static object DataConnectionPublicShape(DataConnection c) +{ + var (primary, s1) = ConfigSecretScrubber.Scrub(c.PrimaryConfiguration); + var (backup, s2) = ConfigSecretScrubber.Scrub(c.BackupConfiguration); + return new + { + c.Id, c.Name, c.Protocol, c.SiteId, c.FailoverRetryCount, + PrimaryConfiguration = primary, BackupConfiguration = backup, + HasSecrets = s1 || s2, + }; + // NOTE: project every additional simple property present on the entity + // (read Commons/Entities .../DataConnection.cs and include all non-config fields). +} +``` +- `HandleListDataConnections`: `return (await ...).Select(DataConnectionPublicShape).ToList();` (both branches). +- `HandleGetDataConnection`: `return conn is null ? null : DataConnectionPublicShape(conn);` (after the scope check). +- `HandleCreateDataConnection`: audit + return `DataConnectionPublicShape(conn)`. +- `HandleUpdateDataConnection`: before assignment, `cmd = cmd with { }` is unnecessary — merge directly: +```csharp +conn.PrimaryConfiguration = ConfigSecretScrubber.MergeSentinels(cmd.PrimaryConfiguration, conn.PrimaryConfiguration); +conn.BackupConfiguration = ConfigSecretScrubber.MergeSentinels(cmd.BackupConfiguration, conn.BackupConfiguration); +``` +(assign Name/Protocol/FailoverRetryCount as today); audit + return the public shape. +4. Run filter — PASS; full project run. Note: the Central UI data-connection editor talks to repositories directly (not the actor), so UI round-tripping is unaffected; CLI round-trips are protected by the sentinel merge. +5. Update `Component-ManagementService.md`. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.ManagementService tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests docs/requirements/Component-ManagementService.md && git commit -m "fix(management): elide DataConnection config secrets from responses and audit (arch-review C3)"` + +### Task 8: ExternalSystem `AuthConfiguration` elision + preserve-if-null update + +**Classification:** high-risk +**Estimated implement time:** ~4 min +**Parallelizable with:** none (ManagementActor.cs) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (lines 1587–1625) +- Modify: `docs/requirements/Component-ManagementService.md` +- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/` (same file as Task 7) + +1. Failing tests: `ListExternalSystems`/`GetExternalSystem` response must not contain the mocked `AuthConfiguration` value (e.g. `"apiKey":"es-secret"`), must contain `hasAuthConfiguration`; `UpdateExternalSystem` with `AuthConfiguration = null` must preserve the stored value; Create/Update audit afterState must not contain the secret. Same test pattern as Task 7. +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests --filter ExternalSystem` — FAIL. +3. Implement `ExternalSystemPublicShape`: +```csharp +/// Secret-elided ExternalSystemDefinition projection: AuthConfiguration +/// (API keys / Basic credentials) never leaves this boundary. Mirrors SmtpConfigPublicShape. +private static object ExternalSystemPublicShape(ExternalSystemDefinition d) => new +{ + d.Id, d.Name, d.EndpointUrl, d.AuthType, + HasAuthConfiguration = !string.IsNullOrEmpty(d.AuthConfiguration), +}; +``` +List → `.Select(ExternalSystemPublicShape)`; Get → projected-or-null; Create/Update → audit + return projection; Update gains preserve-if-null: `if (cmd.AuthConfiguration is not null) def.AuthConfiguration = cmd.AuthConfiguration;` (comment mirrors the SMTP preserve-if-null rationale). +4. PASS + full run. 5. Doc + commit: `git commit -m "fix(management): elide ExternalSystem AuthConfiguration from responses and audit (arch-review C3)"` + +### Task 9: DatabaseConnection `ConnectionString` elision + preserve-if-null update + +**Classification:** high-risk +**Estimated implement time:** ~4 min +**Parallelizable with:** none (ManagementActor.cs) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (lines 2526–2559) +- Modify: `docs/requirements/Component-ManagementService.md` +- Test: same file as Tasks 7–8 + +1. Failing tests: List/Get response contains `id`/`name`/`hasConnectionString` but not the mocked connection string (`"Server=db;Password=sql-secret"`); Update with `ConnectionString = null` preserves stored value. Check `UpdateDatabaseConnectionDefCommand.ConnectionString` nullability first — if non-nullable, make it `string?` (additive JSON-wire-compatible). +2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests --filter DatabaseConnection` — FAIL. +3. Implement `private static object DatabaseConnectionPublicShape(DatabaseConnectionDefinition d) => new { d.Id, d.Name, HasConnectionString = !string.IsNullOrEmpty(d.ConnectionString) };` and apply to List/Get/Create/Update returns (audit shape at :2544/:2557 already `{Id, Name}` — leave). Update handler: `if (cmd.ConnectionString is not null) def.ConnectionString = cmd.ConnectionString;` +4. PASS + full run. CLI note: `db-connection` list/get output shape changes (no connection string) — update `Component-CLI.md`/CLI README sample output if shown. +5. Commit: `git commit -m "fix(management): elide DatabaseConnection ConnectionString from List/Get (arch-review C3)"` + +### Task 10: `GetRequiredRoles` any-of refactor + gate `ListSecuredWrites` + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** none (ManagementActor.cs) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (lines 97–242: `HandleEnvelope` + `GetRequiredRole`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.Security/AuthorizationPolicies.cs` (new `RequireSecuredWriteAccess` policy) +- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Operations/SecuredWrites.razor` (page `@attribute [Authorize(Policy = ...)]` — check current policy first and align) +- Modify: `docs/requirements/Component-ManagementService.md` §Authorization, `docs/requirements/Component-CentralUI.md` §Secured Writes +- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/ManagementActorTests.cs`, `tests/ZB.MOM.WW.ScadaBridge.Security.Tests/SecurityTests.cs` (policy registration) + +1. Failing tests: +```csharp +[Fact] +public void ListSecuredWrites_ViewerOnly_Unauthorized() +{ + var actor = CreateActor(); + actor.Tell(Envelope(new ListSecuredWritesCommand(null, null), "Viewer")); + ExpectMsg(TimeSpan.FromSeconds(5)); +} + +[Theory] +[InlineData("Operator")] +[InlineData("Verifier")] +[InlineData("Administrator")] +public void ListSecuredWrites_SecuredWriteRoles_Allowed(string role) +{ + var actor = CreateActor(); // ISecuredWriteRepository substitute returns empty list + actor.Tell(Envelope(new ListSecuredWritesCommand(null, null), role)); + ExpectMsg(TimeSpan.FromSeconds(5)); +} +``` +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests --filter ListSecuredWrites` — FAIL (currently ungated). +3. Implement: + - Rename `GetRequiredRole` → `internal static IReadOnlyList? GetRequiredRoles(object command)` (internal for Task 13's matrix test). Wrap the existing arms with static readonly arrays: `private static readonly string[] AdminOnly = [Roles.Administrator];`, `DesignerOnly`, `DeployerOnly`, `OperatorOnly`, `VerifierOnly`, and new `SecuredWriteReaders = [Roles.Operator, Roles.Verifier, Roles.Administrator];`. Add arm: `ListSecuredWritesCommand => SecuredWriteReaders,` with a comment (tag values on the history table are process-sensitive — arch-review UA1). + - `HandleEnvelope`: +```csharp +var requiredRoles = GetRequiredRoles(envelope.Command); +if (requiredRoles is not null + && !requiredRoles.Any(r => user.Roles.Contains(r, StringComparer.OrdinalIgnoreCase))) +{ + sender.Tell(new ManagementUnauthorized(correlationId, + $"One of roles [{string.Join(", ", requiredRoles)}] required for {envelope.Command.GetType().Name}")); + return; +} +``` +(The existing `CreateSiteCommand_WithDesignRole_ReturnsUnauthorized` asserts `Contains("Administrator")` — still passes.) + - `AuthorizationPolicies.cs`: add `RequireSecuredWriteAccess` = `RequireClaim(JwtTokenService.RoleClaimType, Roles.Operator, Roles.Verifier, Roles.Administrator)` and apply to the SecuredWrites page attribute (check the page's current attribute; per Component-CentralUI it may already be operator-gated — align page, service, and actor). +4. PASS + full ManagementService.Tests + Security.Tests runs. +5. Docs, then commit: `git commit -m "feat(management): any-of role gates; restrict ListSecuredWrites to Operator/Verifier/Admin (arch-review UA1)"` + +### Task 11: Reconcile the area-management role gate (code ⇄ three docs) + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** none (ManagementActor.cs; waits on Task 10) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (Area command arms, lines 186/207) +- Modify: `docs/requirements/Component-ManagementService.md` §Authorization (~line 213), `docs/requirements/Component-Security.md` §Administrator (~line 88), `docs/requirements/Component-CentralUI.md` §Area Management (~line 106) +- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/ManagementActorTests.cs` + +**Design decision** (record in the commit message): the Central UI creates areas inside the Deployment Topology workflow (`Topology.razor` is `RequireDeployment`), and the actor gates them Designer — the three docs' "Admin" assignment is stale. Reconcile on **any-of [Designer, Deployer]** (covers both real surfaces; Administrator is not implicitly included in the actor gate, matching current behavior for other Designer commands), and update all three docs to match. + +1. Failing test: +```csharp +[Theory] +[InlineData("Designer")] +[InlineData("Deployer")] +public void CreateArea_DesignerOrDeployer_Allowed(string role) { /* Envelope(new CreateAreaCommand(...), role) → not ManagementUnauthorized */ } + +[Fact] +public void CreateArea_ViewerOnly_Unauthorized() { /* → ManagementUnauthorized */ } +``` +2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests --filter CreateArea` — Deployer case FAILS today. +3. Move `CreateAreaCommand or UpdateAreaCommand or DeleteAreaCommand` out of the Designer arm into a new arm `=> AreaManagers` where `private static readonly string[] AreaManagers = [Roles.Designer, Roles.Deployer];` with a comment citing the reconciliation. Update the three docs: area management = Designer or Deployer (management API), Central UI exposes it inside the Deployment Topology workflow (`RequireDeployment`); remove "area management" from the Admin lists. +4. PASS. 5. Commit: `git commit -m "fix(management): reconcile area role gate to Designer|Deployer across actor and all three docs (arch-review C6)"` + +### Task 12: Canonicalize role-mapping casing at the write path + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** 14, 19, 21, 26, 29 (ManagementActor.cs region 1904–1944 — do not run concurrently with other ManagementActor tasks) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (lines 1904–1944: `ValidateMappingRole`, `HandleCreateRoleMapping`, `HandleUpdateRoleMapping`) +- Modify: `docs/requirements/Component-Security.md` (role-mapping section: stored casing is canonicalized) +- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/RoleMappingValidationTests.cs` + +1. Failing test: +```csharp +[Fact] +public void CreateRoleMapping_LowercaseRole_StoresCanonicalCasing() +{ + var actor = CreateActor(); + actor.Tell(Envelope(new CreateRoleMappingCommand("ldap-group", "deployer"), "Administrator")); + ExpectMsg(TimeSpan.FromSeconds(5)); + await _securityRepo.Received(1).AddMappingAsync( + Arg.Is(m => m.Role == "Deployer")); // canonical, not verbatim +} +``` +2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests --filter RoleMapping` — FAIL. +3. Replace `ValidateMappingRole` with: +```csharp +// The stored Role must be CANONICAL casing (Roles.All): ASP.NET RequireClaim policies +// compare ordinal case-sensitively while the actor compares OrdinalIgnoreCase, so a +// row stored as "deployer" passes every actor gate but fails every UI policy — a +// CLI-works/UI-403s split-brain. Canonicalizing at this single write path closes the +// asymmetry (arch-review C5; closes the deferral formerly documented here). +private static string CanonicalizeMappingRole(string role) +{ + var canonical = Roles.All.FirstOrDefault(r => string.Equals(r, role, StringComparison.OrdinalIgnoreCase)); + return canonical ?? throw new ManagementCommandException( + $"Role '{role}' is not a recognized role. Valid roles are: {string.Join(", ", Roles.All)}."); +} +``` +Use `var role = CanonicalizeMappingRole(cmd.Role);` in both create/update handlers and store `role`. Keep the invalid-role rejection tests green (message unchanged). +4. PASS + full RoleMappingValidationTests run. 5. Doc + commit: `git commit -m "fix(management): canonicalize role-mapping casing at write path, closing UI/actor case split (arch-review C5)"` + +### Task 13: `GetRequiredRoles` matrix test — freeze the authorization table + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** none (waits on Tasks 10, 11, 22, 23 — final gate table) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ZB.MOM.WW.ScadaBridge.ManagementService.csproj` (add ``) +- Create: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/RequiredRoleMatrixTests.cs` + +1. Write the test (it IS the deliverable — no implementation change beyond the csproj line): +```csharp +using System.Runtime.CompilerServices; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management; +using ZB.MOM.WW.ScadaBridge.Security; + +public class RequiredRoleMatrixTests +{ + // FROZEN authorization table: every registered management command MUST have an + // entry here. A new command without one fails EveryRegisteredCommand_IsInTheFrozenTable, + // forcing a deliberate authz decision (arch-review UA2; would have caught C2/C4). + private static readonly Dictionary Expected = new(StringComparer.OrdinalIgnoreCase) + { + ["CreateSite"] = [Roles.Administrator], + ["CreateTemplate"] = [Roles.Designer], + ["MgmtDeployArtifacts"] = [Roles.Deployer], + ["SubmitSecuredWrite"] = [Roles.Operator], + ["ApproveSecuredWrite"] = [Roles.Verifier], + ["ListSecuredWrites"] = [Roles.Operator, Roles.Verifier, Roles.Administrator], + ["CreateArea"] = [Roles.Designer, Roles.Deployer], + ["ListTemplates"] = null, // read-only: any authenticated user + // ... generate the FULL ~150-entry table: dump + // ManagementCommandRegistry-registered names with their current + // GetRequiredRoles(...) result once (scratch console or temp test), + // eyeball it against Component-ManagementService.md §Authorization, + // then paste it here verbatim and never regenerate mechanically. + }; + + public static IEnumerable AllRegisteredCommands() + => typeof(ManagementEnvelope).Assembly.GetTypes() + .Where(t => t.Namespace == typeof(ManagementEnvelope).Namespace + && t.Name.EndsWith("Command", StringComparison.Ordinal) && !t.IsAbstract) + .Select(t => new object[] { t }); + + [Theory, MemberData(nameof(AllRegisteredCommands))] + public void EveryRegisteredCommand_IsInTheFrozenTable(Type commandType) + { + var name = commandType.Name[..^"Command".Length]; + Assert.True(Expected.ContainsKey(name), + $"Command '{name}' has no entry in the frozen authorization table — add one deliberately."); + var instance = RuntimeHelpers.GetUninitializedObject(commandType); + var actual = ManagementActor.GetRequiredRoles(instance); + var expected = Expected[name]; + if (expected is null) Assert.Null(actual); + else Assert.Equal(expected.OrderBy(x => x), actual!.OrderBy(x => x)); + } +} +``` +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests --filter RequiredRoleMatrixTests` — FAIL until the table is complete and `InternalsVisibleTo` + `internal` visibility (Task 10) are in place. +3. Fill the table to green. Any surprise found while filling (a command gated differently than `Component-ManagementService.md` documents) gets fixed in this task — doc or gate, whichever is wrong — and noted in the commit message. +4. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ManagementService/ZB.MOM.WW.ScadaBridge.ManagementService.csproj tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/RequiredRoleMatrixTests.cs && git commit -m "test(management): frozen GetRequiredRoles matrix over every registered command (arch-review UA2)"` + +### Task 14: SecuredWrite repository — `TryMarkExpiredAsync` + `CountAsync` + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 3, 4, 6, 19, 21, 26, 29 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/ISecuredWriteRepository.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SecuredWriteRepository.cs` +- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/` — extend the existing SecuredWriteRepository test file (locate via `grep -rl SecuredWriteRepository tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests`; mirror its harness — in-memory/SQLite or MSSQL fixture) + +1. Failing tests: +```csharp +[Fact] +public async Task TryMarkExpiredAsync_PendingRow_FlipsToExpired_Once() +{ + var id = await _repo.AddAsync(NewPendingWrite()); + Assert.True(await _repo.TryMarkExpiredAsync(id, DateTime.UtcNow)); + Assert.False(await _repo.TryMarkExpiredAsync(id, DateTime.UtcNow)); // CAS: second caller loses + var row = await _repo.GetAsync(id); + Assert.Equal("Expired", row!.Status); + Assert.NotNull(row.DecidedAtUtc); + Assert.Null(row.VerifierUser); // system decision, no verifier +} + +[Fact] +public async Task TryMarkExpiredAsync_DecidedRow_ReturnsFalse() +{ + var id = await _repo.AddAsync(NewPendingWrite()); + await _repo.TryMarkApprovedAsync(id, "v", null, DateTime.UtcNow); + Assert.False(await _repo.TryMarkExpiredAsync(id, DateTime.UtcNow)); +} + +[Fact] +public async Task CountAsync_FiltersLikeQueryAsync() +{ + await _repo.AddAsync(NewPendingWrite()); + await _repo.AddAsync(NewPendingWrite()); + Assert.Equal(2, await _repo.CountAsync(status: "Pending", siteId: null)); + Assert.Equal(0, await _repo.CountAsync(status: "Executed", siteId: null)); +} +``` +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter "TryMarkExpired|CountAsync_Filters"` — FAIL. +3. Add to the interface (xmldoc mirrors `TryMarkApprovedAsync`'s CAS wording — this is the multi-node-safe expiry primitive): +```csharp +Task TryMarkExpiredAsync(long id, DateTime expiredAtUtc, CancellationToken ct = default); +Task CountAsync(string? status, string? siteId, CancellationToken ct = default); +``` +Implement with the same conditional-update pattern `TryMarkApprovedAsync` uses (`WHERE Status = 'Pending'`), setting `Status='Expired'`, `DecidedAtUtc=expiredAtUtc`, leaving `VerifierUser` NULL; `CountAsync` mirrors `QueryAsync`'s filter composition with `CountAsync()`. +4. PASS + full project run. 5. Commit: `git commit -m "feat(config-db): secured-write TryMarkExpiredAsync CAS + CountAsync (arch-review S2/P3)"` + +### Task 15: Server-side secured-write TTL — enforce at approve/reject + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** none (ManagementActor.cs; waits on Task 14) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementServiceOptions.cs` (add `SecuredWritePendingTtl`, default 24 h) +- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Types/Enums/AuditKind.cs` (additive member `SecuredWriteExpire`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (`HandleApproveSecuredWrite` ~1106, `HandleRejectSecuredWrite` ~1255) +- Modify: `docs/requirements/Component-ManagementService.md`, `docs/requirements/Component-Security.md` (two-person section: TTL + restate the dual-role Operator+Verifier deployment-configuration hazard), `CLAUDE.md` Security & Auth bullet (append "pending secured writes expire server-side after a configurable TTL, default 24 h") +- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/SecuredWriteHandlerTests.cs` + +1. Failing tests (extend the existing `SecuredWriteHandlerTests` harness; register `IOptions` in its service collection): +```csharp +[Fact] +public void Approve_PendingRowOlderThanTtl_ExpiresInsteadOfExecuting() +{ + var row = PendingRow(submittedAtUtc: DateTime.UtcNow.AddHours(-25)); // TTL default 24h + _securedWriteRepo.GetAsync(1).Returns(row); + _securedWriteRepo.TryMarkExpiredAsync(1, Arg.Any()).Returns(true); + + var actor = CreateActor(); + actor.Tell(Envelope(new ApproveSecuredWriteCommand(1, null), "Verifier")); + + var resp = ExpectMsg(TimeSpan.FromSeconds(5)); + Assert.Contains("expired", resp.Error, StringComparison.OrdinalIgnoreCase); + _securedWriteRepo.DidNotReceive().TryMarkApprovedAsync( + Arg.Any(), Arg.Any(), Arg.Any(), Arg.Any(), Arg.Any()); + _commService.DidNotReceiveWithAnyArgs().WriteTagAsync(default!, default!); // stale value NEVER relayed +} + +[Fact] +public void Approve_FreshPendingRow_StillExecutes() { /* submitted 1h ago → existing approve path unchanged */ } +``` +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests --filter SecuredWrite` — new tests FAIL. +3. Implement: + - Options: `/// Age after which a Pending secured write can no longer be approved; it is transitioned to Expired at the next decision/list touch. Default 24 hours. public TimeSpan SecuredWritePendingTtl { get; set; } = TimeSpan.FromHours(24);` + - `AuditKind`: add `SecuredWriteExpire` (additive enum member — message contracts are additive-only). + - Shared helper in `ManagementActor`: +```csharp +/// If the Pending row is older than the configured TTL, CAS it to Expired +/// (multi-node idempotent), audit best-effort, and return true. A stale setpoint +/// must never be approvable days later (arch-review S2). +private static async Task TryExpireIfStaleAsync( + IServiceProvider sp, ISecuredWriteRepository repo, PendingSecuredWrite row) +{ + var ttl = sp.GetService>()?.Value.SecuredWritePendingTtl + ?? TimeSpan.FromHours(24); + if (ttl <= TimeSpan.Zero || DateTime.UtcNow - row.SubmittedAtUtc <= ttl) return false; + var expiredAt = DateTime.UtcNow; + if (await repo.TryMarkExpiredAsync(row.Id, expiredAt)) + { + row.Status = "Expired"; + row.DecidedAtUtc = expiredAt; + await EmitSecuredWriteAuditAsync( + sp, AuditKind.SecuredWriteExpire, AuditStatus.Discarded, row, actor: "system"); + } + return true; +} +``` + - In `HandleApproveSecuredWrite` and `HandleRejectSecuredWrite`, immediately after the `"Pending"` status check: `if (await TryExpireIfStaleAsync(sp, repo, row)) throw new ManagementCommandException($"Secured write {cmd.Id} expired (pending longer than the configured TTL) and can no longer be approved or rejected.");` + (Check `AuditStatus` members; if `Discarded` is inappropriate, use the closest terminal status the enum offers and note it.) +4. PASS + full SecuredWriteHandlerTests. 5. Docs + CLAUDE.md, commit: `git commit -m "feat(management): server-side secured-write TTL — stale pending writes expire, never execute (arch-review S2)"` + +### Task 16: Opportunistic expiry sweep in `HandleListSecuredWrites` + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** none (ManagementActor.cs; waits on Task 15) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (`HandleListSecuredWrites` ~1280) +- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/SecuredWriteHandlerTests.cs` + +1. Failing test: repo returns one fresh Pending + one 25h-old Pending; `ListSecuredWritesCommand` result marks the old one `Expired` and `TryMarkExpiredAsync` was called exactly once (for the stale row only). +2. Run `--filter SecuredWrite` — FAIL. +3. In `HandleListSecuredWrites`, before returning: iterate the queried rows, `await TryExpireIfStaleAsync(sp, repo, row)` for each `Pending` row (helper self-filters by age; CAS makes the multi-node double-run harmless — the ManagementActor runs on every central node by design, so the sweep is deliberately lazy/opportunistic rather than a new timer singleton; document this in a comment). +4. PASS. 5. Commit: `git commit -m "feat(management): opportunistic expiry sweep on secured-write list (arch-review S2)"` + +### Task 17: Secured-write list paging — command, handler, CLI + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** none (ManagementActor.cs; waits on Task 14) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Management/SecuredWriteCommands.cs` (line 59 `ListSecuredWritesCommand`, line 83 `SecuredWriteListResult`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (`HandleListSecuredWrites`) +- Modify: CLI secured-write list command (locate: `grep -rl ListSecuredWritesCommand src/ZB.MOM.WW.ScadaBridge.CLI`) — add `--skip/--take` +- Modify: `docs/requirements/Component-ManagementService.md`, `docs/requirements/Component-CLI.md` +- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/SecuredWriteHandlerTests.cs`, `tests/ZB.MOM.WW.ScadaBridge.CLI.Tests` (parse test) + +1. Failing test: `ListSecuredWritesCommand(null, null, Skip: 10, Take: 25)` → repo `QueryAsync(null, null, 10, 25)` received; result JSON contains `totalCount` sourced from `CountAsync`. +2. Run `--filter SecuredWrite` — FAIL (record has no such params). +3. Implement (additive, defaults preserve today's wire behavior): +```csharp +public record ListSecuredWritesCommand(string? Status, string? SiteId, int Skip = 0, int Take = 200); +public record SecuredWriteListResult(IReadOnlyList Items, int TotalCount); +``` +Handler: clamp `var take = Math.Clamp(cmd.Take, 1, 500); var skip = Math.Max(0, cmd.Skip);`, `QueryAsync(cmd.Status, cmd.SiteId, skip, take)` + `CountAsync(cmd.Status, cmd.SiteId)`. Update the one UI construction site (`SecuredWriteService`) for the new result shape (compile will point at it). CLI: add the two int options. +4. PASS; `dotnet test tests/ZB.MOM.WW.ScadaBridge.CLI.Tests` (direct — not in slnx yet). +5. Commit: `git commit -m "feat(management): secured-write list paging with TotalCount (arch-review P3)"` + +### Task 18: SecuredWrites page — history pager + age in the approve dialog + docs + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 19, 21, 26, 28, 29 (waits on Task 17) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Operations/SecuredWrites.razor` (RefreshListsAsync ~line 303; confirm dialog ~line 342; history table) +- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/ISecuredWriteService.cs` + `SecuredWriteService.cs` (paging params + TotalCount pass-through) +- Modify: `docs/requirements/Component-CentralUI.md` §Secured Writes (Expired is real + TTL + paging) +- Test: `tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/` (bUnit — follow an existing page test's auth/service-fake setup) + +1. Failing bUnit test: render the page with a fake `ISecuredWriteService` returning 60 terminal rows + `TotalCount = 120`; assert a "Next" pager button exists and clicking it requests the second page (`ListAsync(..., skip: 50, ...)` received). Second test: the approve confirmation dialog for a row submitted 26 hours ago renders text matching `Submitted .* ago` (age surfaced next to the value — the reviewer's stale-setpoint scenario). +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests --filter SecuredWrites` — FAIL. +3. Implement: page-size 50 history pager (`_historyPage`, Prev/Next buttons disabled at bounds using TotalCount); dialog gains a `Submitted {FormatAge(row.SubmittedAtUtc)} ago ({row.SubmittedAtUtc:u})` line. Pending list stays unpaged (bounded by TTL now). +4. PASS. 5. Doc + commit: `git commit -m "feat(ui): secured-write history paging + submission age in approve dialog (arch-review S2/P3)"` + +### Task 19: Per-command long-running Ask timeout + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 3, 4, 6, 14, 21, 26, 29 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementServiceOptions.cs` (add `LongRunningCommandTimeout`, default 5 min) +- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementEndpoints.cs` (lines 18–33 `ResolveAskTimeout`; line 101 call site) +- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/ManagementEndpointsTests.cs` + +1. Failing tests: +```csharp +[Theory] +[InlineData(typeof(ImportBundleCommand))] +[InlineData(typeof(ExportBundleCommand))] +[InlineData(typeof(PreviewBundleCommand))] +[InlineData(typeof(MgmtDeployArtifactsCommand))] +[InlineData(typeof(MgmtDeployInstanceCommand))] +public void ResolveAskTimeout_LongRunningCommand_UsesLongTimeout(Type cmd) + => Assert.Equal(TimeSpan.FromMinutes(5), ManagementEndpoints.ResolveAskTimeout(null, cmd)); + +[Fact] +public void ResolveAskTimeout_OrdinaryCommand_UsesDefault() + => Assert.Equal(TimeSpan.FromSeconds(30), ManagementEndpoints.ResolveAskTimeout(null, typeof(ListTemplatesCommand))); + +[Fact] +public void ResolveAskTimeout_ConfiguredLongTimeout_Honored() + => Assert.Equal(TimeSpan.FromMinutes(10), ManagementEndpoints.ResolveAskTimeout( + new ManagementServiceOptions { LongRunningCommandTimeout = TimeSpan.FromMinutes(10) }, + typeof(ImportBundleCommand))); +``` +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests --filter ResolveAskTimeout` — FAIL. +3. Implement: +```csharp +private static readonly TimeSpan DefaultLongRunningTimeout = TimeSpan.FromMinutes(5); + +/// Transport/deploy commands legitimately run for minutes (the CLI already +/// uses a 5-minute client timeout for bundles); a 30 s server-side Ask 504s while the +/// piped ProcessCommand keeps applying — the caller then retries and double-applies +/// (arch-review S1). These command types get LongRunningCommandTimeout instead. +private static readonly HashSet LongRunningCommands = +[ + typeof(ImportBundleCommand), typeof(PreviewBundleCommand), typeof(ExportBundleCommand), + typeof(MgmtDeployArtifactsCommand), typeof(MgmtDeployInstanceCommand), +]; + +public static TimeSpan ResolveAskTimeout(ManagementServiceOptions? options, Type commandType) + => LongRunningCommands.Contains(commandType) + ? (options is { LongRunningCommandTimeout.Ticks: > 0 } o ? o.LongRunningCommandTimeout : DefaultLongRunningTimeout) + : ResolveAskTimeout(options); +``` +Call site: `ResolveAskTimeout(options, command.GetType())`. Keep the existing single-arg overload (used by other callers/tests). +4. PASS + full ManagementEndpointsTests. 5. Commit: `git commit -m "fix(management): 5-min Ask timeout for transport/deploy commands (arch-review S1)"` + +### Task 20: Ship `ManagementService` timeout config + doc + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** 18, 21, 26, 28, 29 (waits on Task 19) +**Files:** +- Modify: `docker/central-node-a/appsettings.Central.json`, `docker/central-node-b/appsettings.Central.json`, the two `docker-env2` central-node appsettings (glob `docker-env2/central-node-*/appsettings.Central.json`), `deploy/wonder-app-vd03/appsettings.Central.json` +- Modify: `docs/requirements/Component-ManagementService.md` §Configuration (~line 247) + +1. Add under the `"ScadaBridge"` object in each central-node config: +```json +"ManagementService": { + "CommandTimeout": "00:00:30", + "LongRunningCommandTimeout": "00:05:00" +} +``` +2. Extend the doc's configuration table row with `LongRunningCommandTimeout` (TimeSpan, default 5 min, applied to ImportBundle/PreviewBundle/ExportBundle/MgmtDeployArtifacts/MgmtDeployInstance). +3. Validate each edited JSON (`python3 -c ...` as in Task 3). Cluster-runtime change: rebuild with `bash docker/deploy.sh` when convenient (config-only; note in commit). +4. Commit: `git commit -m "chore(deploy): ship ManagementService command-timeout config to all central-node topologies (arch-review S1)"` + +### Task 21: Browse/Verify/Cert commands gain optional `SiteIdentifier` + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** 1, 3, 4, 6, 14, 19, 26, 29 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Management/BrowseCommands.cs` (`BrowseNodeCommand` :24, `SearchAddressSpaceCommand` :62), `VerifyEndpointCommands.cs` (`VerifyEndpointCommand` :102), `CertTrustCommands.cs` (`TrustServerCertCommand` :192, `RemoveServerCertCommand` :199, `ListServerCertsCommand` :206) +- Test: `tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/` (serialization round-trip) + +1. Failing test (Commons.Tests): deserialize legacy JSON without the field → `SiteIdentifier == null`; round-trip with the field set survives. (Use `System.Text.Json` with `PropertyNameCaseInsensitive = true`, matching `ManagementEndpoints.ParseCommand`.) +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter SiteIdentifier` — FAIL. +3. Append an optional trailing positional param to each of the six records, e.g.: +```csharp +public record BrowseNodeCommand( + string ConnectionName, string? ParentNodeId, string? ContinuationToken, + string? SiteIdentifier = null); +``` +xmldoc: "Target site identifier — REQUIRED when the command is invoked via the management HTTP API/CLI (the actor must route it to a site); ignored on the site side, where routing already happened. Additive per the message-contract evolution rules." Existing positional construction sites (UI services, Communication) compile unchanged. +4. PASS + `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. 5. Commit: `git commit -m "feat(commons): additive SiteIdentifier on browse/verify/cert-trust commands for management-API routing (arch-review C4)"` + +### Task 22: Actor handlers — BrowseNode / SearchAddressSpace / VerifyEndpoint + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** none (ManagementActor.cs; waits on Tasks 10, 21) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (dispatch switch ~:398 "Remote Queries" region; `GetRequiredRoles` Designer arm; new handlers near `HandleQueryEventLogs` ~:2753) +- Modify: `docs/requirements/Component-ManagementService.md` §Remote Queries (note the commands are now actor-dispatched — the doc's existing description at lines 204–207 becomes true) +- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/ManagementActorTests.cs` + +1. Failing tests: +```csharp +[Fact] +public void BrowseNode_WithoutSiteIdentifier_ReturnsError() +{ + var actor = CreateActor(); + actor.Tell(Envelope(new BrowseNodeCommand("conn", null, null), "Designer")); + var resp = ExpectMsg(TimeSpan.FromSeconds(5)); + Assert.Contains("SiteIdentifier", resp.Error); +} + +[Fact] +public void BrowseNode_Designer_RelaysToSite() +{ + var actor = CreateActor(); // CommunicationService fake per SecuredWriteHandlerTests' pattern + actor.Tell(Envelope(new BrowseNodeCommand("conn", null, null, "SITE1"), "Designer")); + ExpectMsg(TimeSpan.FromSeconds(5)); + _commService.Received(1).BrowseNodeAsync("SITE1", Arg.Any(), Arg.Any()); +} + +[Fact] +public void BrowseNode_ViewerOnly_Unauthorized() { /* → ManagementUnauthorized */ } +``` +2. Run `--filter BrowseNode` — FAIL (falls through to NotSupportedException today). +3. Implement three handlers following `HandleQueryEventLogs`'s shape: +```csharp +private static async Task HandleBrowseNode(IServiceProvider sp, BrowseNodeCommand cmd, AuthenticatedUser user) +{ + var site = cmd.SiteIdentifier + ?? throw new ManagementCommandException("SiteIdentifier is required when browsing via the management API."); + await EnforceSiteScopeForIdentifier(sp, user, site); + var comm = sp.GetRequiredService(); + return await comm.BrowseNodeAsync(site, cmd); +} +``` +(same for `HandleSearchAddressSpace` → `SearchAddressSpaceAsync`, `HandleVerifyEndpoint` → `VerifyEndpointAsync`; match the real `CommunicationService` signatures at :366/:388/:410). Dispatch arms in the Remote Queries region; `GetRequiredRoles`: add the three commands to the Designer arm (doc line 204/206 already says Design). Update the Task 13 matrix table if it already exists. +4. PASS + full run. 5. Doc + commit: `git commit -m "feat(management): actor dispatch for BrowseNode/SearchAddressSpace/VerifyEndpoint — CLI parity (arch-review C4)"` + +### Task 23: Actor handlers — TrustServerCert / RemoveServerCert / ListServerCerts (Admin) + +**Classification:** high-risk +**Estimated implement time:** ~4 min +**Parallelizable with:** none (ManagementActor.cs; waits on Tasks 10, 21) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (dispatch + `GetRequiredRoles` Admin arm + three handlers) +- Modify: `docs/requirements/Component-ManagementService.md` (line 218's Admin gating is now enforced at the actor) +- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/ManagementActorTests.cs` + +1. Failing tests mirroring Task 22: Designer-only caller gets `ManagementUnauthorized` for `TrustServerCertCommand`; Admin with `SiteIdentifier` relays via `CommunicationService.TrustServerCertAsync`; missing `SiteIdentifier` → curated error. +2. Run `--filter ServerCert` — FAIL. +3. Implement three handlers (delegate to `TrustServerCertAsync`/`RemoveServerCertAsync`/`ListServerCertsAsync` at CommunicationService :435/:477/:456); `GetRequiredRoles`: add the three cert commands to the Administrator arm. Comment: "Mutates the site's trusted-peer PKI store — Admin, matching the Admin-gated UI cert page. Site-side reconcile semantics are owned by the site CertStoreActor (plan 03); this handler only relays." +4. PASS + full run. 5. Doc + commit: `git commit -m "feat(management): Admin-gated actor dispatch for cert-trust commands (arch-review C4)"` + +### Task 24: Dispatch-coverage test — every registered command reaches a handler + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 18, 20, 25, 26, 28, 29 (waits on Tasks 22, 23) +**Files:** +- Create: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/DispatchCoverageTests.cs` + +1. Write the test: for every `ManagementCommandRegistry`-registered type (except a documented exclusion list: `ResolveRolesCommand` — intentionally undispatched, see ManagementActor :418), send an envelope carrying `RuntimeHelpers.GetUninitializedObject(type)` with ALL roles to an actor whose `IServiceProvider` is a substitute wired so `CreateScope()` yields a scoped provider that throws a `SentinelException` from `GetService` (mock `IServiceScopeFactory`/`IServiceScope`). Capture the exception MapFault logs via a recording `ILogger` (small test logger that stores `LogError` exceptions keyed by correlation id). Assert: for no command is the captured exception a `NotSupportedException` (which is the "no dispatch arm" signature). Commands that succeed without touching the provider (e.g. `GetHealthSummaryCommand`) simply produce no captured exception — also a pass. +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests --filter DispatchCoverageTests` — should PASS immediately after Tasks 22/23; if it fails, the failure list IS the remaining C4-style gap — add the missing arms or (with justification) the exclusion entry. +3. Commit: `git commit -m "test(management): dispatch-coverage test — every registered command has a handler (arch-review UA2)"` + +### Task 25: CLI parity — browse / search / verify-endpoint / certs commands + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 18, 20, 24, 26, 28, 29 (waits on Tasks 22, 23) +**Files:** +- Create: `src/ZB.MOM.WW.ScadaBridge.CLI/Commands/ConnectionBrowseCommands.cs` +- Modify: CLI `Program.cs` (register the group), `src/ZB.MOM.WW.ScadaBridge.CLI/README.md`, `docs/requirements/Component-CLI.md` +- Test: `tests/ZB.MOM.WW.ScadaBridge.CLI.Tests/` (new `ConnectionBrowseCommandsTests.cs`) + +1. Failing parse tests (mirror an existing CLI.Tests command-parse test): `data-connection browse --site SITE1 --connection conn1 --node-id ns=2;s=X` parses and builds a `BrowseNodeCommand` with `SiteIdentifier == "SITE1"`; same for `search` (`--query --max-depth --max-results`), `verify-endpoint` (`--protocol --config-file ` reading ConfigJson from the file), `certs list`, `certs trust --der-file --thumbprint ` (base64-encodes the file), `certs remove --thumbprint`. +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.CLI.Tests --filter ConnectionBrowse` — FAIL. +3. Implement the command group with `CommandHelpers.ExecuteCommandAsync` like `SiteCommands.BuildDeployArtifacts`; wire under the existing `data-connection` group. Update README + Component-CLI.md command tables. +4. PASS: `dotnet test tests/ZB.MOM.WW.ScadaBridge.CLI.Tests` (direct — remember the slnx gotcha until Task 33). +5. Commit: `git commit -m "feat(cli): browse/search/verify-endpoint/cert-trust commands — actor parity (arch-review C4)"` + +### Task 26: `LoginThrottle` — fixed-window LDAP-bind failure lockout + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 3, 4, 6, 14, 19, 21, 29 +**Files:** +- Create: `src/ZB.MOM.WW.ScadaBridge.Security/LoginThrottle.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.Security/SecurityOptions.cs` (three additive settings + validator rules) +- Modify: `src/ZB.MOM.WW.ScadaBridge.Security/ServiceCollectionExtensions.cs` (`services.AddSingleton();` in `AddSecurity`) +- Test: `tests/ZB.MOM.WW.ScadaBridge.Security.Tests/LoginThrottleTests.cs` (create) + +1. Failing tests (drive time with a fake `TimeProvider` — the codebase already injects `TimeProvider` for `CookieSessionValidator`; reuse that pattern): +```csharp +[Fact] +public void FiveFailuresInWindow_LocksOut_ThenWindowExpiryUnlocks() +{ + var clock = new FakeTimeProvider(); // Microsoft.Extensions.TimeProvider.Testing, already used by Security.Tests — verify, else hand-roll + var throttle = new LoginThrottle(clock, Options.Create(new SecurityOptions())); + for (var i = 0; i < 5; i++) throttle.RecordFailure("alice", "10.0.0.1"); + Assert.True(throttle.IsLockedOut("alice", "10.0.0.1")); + Assert.False(throttle.IsLockedOut("alice", "10.0.0.2")); // per username+IP key + Assert.False(throttle.IsLockedOut("bob", "10.0.0.1")); + clock.Advance(TimeSpan.FromMinutes(6)); // lockout window passed + Assert.False(throttle.IsLockedOut("alice", "10.0.0.1")); +} + +[Fact] +public void SuccessResets() { /* 4 failures + RecordSuccess → not locked after a 5th failure */ } +``` +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Security.Tests --filter LoginThrottleTests` — FAIL. +3. Implement: `ConcurrentDictionary` keyed `{username.ToLowerInvariant()}|{ip}`; fixed window; opportunistic pruning of expired entries on write; bounded (drop-oldest above e.g. 10 000 keys to cap memory under spray). `SecurityOptions` additive: `MaxLoginFailuresPerWindow = 5`, `LoginFailureWindowMinutes = 5`, `LoginLockoutMinutes = 5` (xmldoc: "0 disables throttling"); extend `SecurityOptionsValidator` (non-negative). Register singleton in `AddSecurity`. +4. PASS + full Security.Tests. 5. Commit: `git commit -m "feat(security): LoginThrottle — per-username+IP LDAP-bind failure lockout (arch-review P1)"` + +### Task 27: Wire the throttle — ManagementAuthenticator + auth endpoints + doc + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** none (waits on Task 26) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementAuthenticator.cs` (`AuthenticateAsync`, lines 101–115) +- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Auth/AuthEndpoints.cs` (`/auth/login` + `/auth/token`, before the LDAP bind) +- Modify: `docs/requirements/Component-Security.md` (new "Login throttling" subsection: scope = every LDAP-bind surface — `/auth/login`, `/auth/token`, `POST /management`, audit REST, debug hub, the last three via ManagementAuthenticator) +- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/ManagementAuthenticatorTests.cs` + +1. Failing test (existing harness builds a `DefaultHttpContext` with `RequestServices`; add `LoginThrottle` + real `TimeProvider` to its service collection): +```csharp +[Fact] +public async Task AuthenticateAsync_LockedOutKey_Returns429WithoutLdapBind() +{ + var throttle = GetThrottleFromServices(); + for (var i = 0; i < 5; i++) throttle.RecordFailure("alice", "127.0.0.1"); + var context = BasicAuthContext("alice", "wrong"); // helper: sets Authorization + Connection.RemoteIpAddress + var outcome = await ManagementAuthenticator.AuthenticateAsync(context); + Assert.Null(outcome.User); + // execute outcome.Failure and assert 429 + code AUTH_THROTTLED + await _ldapAuth.DidNotReceiveWithAnyArgs().AuthenticateAsync(default!, default!, default); +} + +[Fact] +public async Task AuthenticateAsync_FailedBind_RecordsFailure() { /* bind fails → next 4 failures lock out */ } +``` +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests --filter ManagementAuthenticator` — FAIL. +3. Implement in `AuthenticateAsync` after username/password extraction, before the bind: +```csharp +var throttle = context.RequestServices.GetRequiredService(); +var remoteIp = context.Connection.RemoteIpAddress?.ToString(); +if (throttle.IsLockedOut(username, remoteIp)) +{ + return new AuthOutcome(null, Results.Json( + new { error = "Too many failed authentication attempts. Try again later.", code = "AUTH_THROTTLED" }, + statusCode: 429)); +} +``` +After the bind: `if (!authResult.Succeeded) { throttle.RecordFailure(username, remoteIp); ...existing 401... }` else `throttle.RecordSuccess(username, remoteIp);`. Same three lines in `/auth/login` (locked → `Redirect("/login?error=Too+many+failed+attempts.+Try+again+later.")`) and `/auth/token` (429 JSON). The DisableLogin bypass path stays above the throttle (dev only, guarded by Task 1). +4. PASS + full ManagementService.Tests + CentralUI.Tests (auth endpoint tests if present). +5. Doc + commit: `git commit -m "feat(security): wire LoginThrottle into management Basic-Auth and /auth/login|token — password-spray guard (arch-review P1/UA5)"` + +### Task 28: Additive paging for ListTemplates / ListInstances + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 18, 20, 24, 25, 29 (touches ManagementActor.cs — serialize with 5, 7–13, 15–17, 22, 23, 30) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Management/` (the file declaring `ListTemplatesCommand` / `ListInstancesCommand` — locate via grep; add `int Skip = 0, int? Take = null`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (`HandleListTemplates` :507, `HandleListInstances` :718) +- Modify: CLI `template list` / `instance list` commands (add `--skip/--take`), `docs/requirements/Component-CLI.md`, `Component-ManagementService.md` +- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/ManagementActorTests.cs`, `tests/ZB.MOM.WW.ScadaBridge.CLI.Tests` + +1. Failing test: repo returns 30 templates; `ListTemplatesCommand(Skip: 10, Take: 5)` yields exactly items 11–15 (post-sort order preserved); `ListInstancesCommand` paging applies AFTER the site-scope filter (scoped user with 3 permitted-site instances and `Take: 2` sees 2 of *their* instances, never an out-of-scope one). +2. Run `--filter Paging` — FAIL. +3. Implement: `Take = null` ⇒ unlimited (today's behavior, so existing callers/UI are untouched); apply `.Skip(Math.Max(0, cmd.Skip)).Take(cmd.Take is > 0 and <= 1000 ? cmd.Take.Value : int.MaxValue)` after any in-memory scope filtering. CLI options pass through. Note in `Component-ManagementService.md` that `QueryDeployments`/`ExportBundle` full-table internals remain a logged scale follow-up (arch-review P2, deferred). +4. PASS (both test projects — CLI directly). 5. Commit: `git commit -m "feat(management): additive Skip/Take paging on template/instance lists (arch-review P2)"` + +### Task 29: `PollGate` — non-reentrant poll timers on Health + Alarm Summary + +**Classification:** small +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 3, 4, 6, 14, 19, 21, 26 +**Files:** +- Create: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/PollGate.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/AlarmSummary.razor` (`StartTimer` ~line 280), `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor` (timer ~line 521) +- Test: `tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/PollGateTests.cs` (create) + +1. Failing tests: +```csharp +public class PollGateTests +{ + [Fact] + public void TryEnter_SecondCallWhileHeld_ReturnsFalse() + { + var gate = new PollGate(); + Assert.True(gate.TryEnter()); + Assert.False(gate.TryEnter()); + gate.Exit(); + Assert.True(gate.TryEnter()); + } +} +``` +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests --filter PollGateTests` — FAIL. +3. Implement: +```csharp +/// Reentrancy guard for poll timers: a fixed-period Timer whose refresh outlives +/// the period stacks overlapping fan-outs against an already-degraded site (arch-review S3). +/// Callers TryEnter at tick start and skip the tick when the previous refresh is in flight. +public sealed class PollGate +{ + private int _inFlight; + public bool TryEnter() => Interlocked.CompareExchange(ref _inFlight, 1, 0) == 0; + public void Exit() => Volatile.Write(ref _inFlight, 0); +} +``` +Wire both timers: `if (!_pollGate.TryEnter()) return;` at the top of the callback lambda, `try { await RefreshAsync(); StateHasChanged(); } finally { _pollGate.Exit(); }` inside the `InvokeAsync`. +4. PASS + `dotnet test tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests` (no page-test regressions). +5. Commit: `git commit -m "fix(ui): reentrancy guard on Health/AlarmSummary poll timers (arch-review S3)"` + +### Task 30: Transport single-flight + base64 copy elimination + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** none (ManagementActor.cs) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (`HandleExportBundle` ~2902–2905, `HandlePreviewBundle`, `HandleImportBundle` ~2949) +- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/` (new `TransportGateTests.cs`) + +1. Failing test: substitute `IBundleImporter` whose `LoadAsync` awaits a `TaskCompletionSource`; send two `ImportBundleCommand` envelopes; assert `LoadAsync` received exactly 1 call while the TCS is unset, then complete it and assert the second proceeds (both eventually answer). +2. Run `--filter TransportGate` — FAIL. +3. Implement: +```csharp +/// Single-flight gate for transport bundle commands: a 200 MB import holds +/// ~4 concurrent copies (base64 string, byte[], MemoryStream, JSON envelope), so two +/// concurrent imports on one central node are a realistic OOM path (arch-review S4). +/// Commands queue here instead of running concurrently on the thread pool. +private static readonly SemaphoreSlim TransportGate = new(1, 1); +``` +Wrap the bodies of the three handlers in `await TransportGate.WaitAsync(); try { ... } finally { TransportGate.Release(); }`. In `HandleExportBundle`, replace `var bytes = ms.ToArray(); ... Convert.ToBase64String(bytes)` with `var base64 = Convert.ToBase64String(ms.GetBuffer().AsSpan(0, (int)ms.Length)); return new ExportBundleResult(base64, (int)ms.Length);` (one whole-artifact copy eliminated). Leave a comment that streaming multipart (like the audit export path, `AuditEndpoints.cs:183+`) is the deferred long-term shape. +4. PASS + full ManagementService.Tests + `dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.Tests` (no behavioral change expected). +5. Commit: `git commit -m "fix(management): single-flight transport bundle commands + drop redundant export copy (arch-review S4)"` + +### Task 31: Low cleanup A — hub send fault logging (S5), event-log filter comment (C7), stale CLAUDE.md note (C8) + +**Classification:** small +**Estimated implement time:** ~5 min +**Parallelizable with:** 32, 33 (and any non-overlapping task) +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/DebugStreamHub.cs` (~lines 204–221) +- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (line 2762 comment), `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/RemoteQuery/EventLogQueryRequest.cs` (xmldoc for `InstanceId`) +- Modify: `CLAUDE.md` (Security & Auth section, the SecuredWrite `SourceNode` parenthetical) +- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/DebugStreamHubTests.cs` + +1. **S5**: change the fire-and-forget send to log failures (keep fire-and-forget semantics — deliberate): +```csharp +_ = hubClients.Client(connectionId).SendAsync(method, payload).ContinueWith( + t => logger.LogWarning(t.Exception?.GetBaseException(), + "Debug stream push to connection {ConnectionId} failed", connectionId), + CancellationToken.None, TaskContinuationOptions.OnlyOnFaulted, TaskScheduler.Default); +``` +(adapt to the hub's actual logger access; add a bUnit-free unit test in `DebugStreamHubTests` if the harness exposes the push path — otherwise verify by build + existing hub tests green). +2. **C7** (verified: the site event-log `instance_id` column stores the instance **UniqueName** — `InstanceActor.LogLifecycleEvent` passes `_instanceUniqueName`, `EventLogQueryService.cs:107-111` matches on it): change the `ManagementActor.cs:2762` comment to `cmd.InstanceName, // instance unique-name filter — the site event log's instance_id column stores UniqueName (see InstanceActor.LogLifecycleEvent)`; add matching `` xmldoc on `EventLogQueryRequest.InstanceId`. +3. **C8** (verified: `EmitSecuredWriteAuditAsync` routes through `ICentralAuditWriter` which stamps `SourceNode` — confirm at `ManagementActor.cs:999-1013` before editing): replace the CLAUDE.md parenthetical "(SecuredWrite audit rows currently leave `SourceNode` NULL — a logged follow-up.)" with "(SecuredWrite audit rows stamp `SourceNode` via `ICentralAuditWriter`/`INodeIdentityProvider`.)". +4. `dotnet build ZB.MOM.WW.ScadaBridge.slnx` + `dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests`. +5. Commit: `git commit -m "chore(management): log debug-stream push failures; fix event-log filter docs; refresh stale SourceNode note (arch-review S5/C7/C8)"` + +### Task 32: Low cleanup B — Alarm Summary memoization (P4) + sortable-header a11y (UA6) + +**Classification:** small +**Estimated implement time:** ~5 min +**Parallelizable with:** 31, 33 +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/AlarmSummary.razor` (render body ~line 170; header markup ~lines 176–179) +- Test: `tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/` (extend the existing AlarmSummary page test if present, else create `AlarmSummaryRenderTests.cs`) + +1. Failing bUnit test: rendered sortable `` elements carry `tabindex="0"` and `aria-sort` (`ascending`/`descending` on the active column, `none` otherwise); pressing Enter on a header changes the sort (assert row order flips, mirroring the click test). +2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests --filter AlarmSummary` — FAIL. +3. Implement: (a) **P4** — replace the in-render `FilteredRows().ToList()` with a cached `_visibleRows` field recomputed in `RefreshAsync` and in each filter/sort setter (one `RecomputeVisibleRows()` helper); (b) **UA6** — sortable ``: add `tabindex="0"`, `aria-sort="@AriaSortFor(col)"`, `@onkeydown` handler toggling sort on `Enter`/`Space` (`@onkeydown:preventDefault` for Space). Leave a note that the same pattern applies to the other custom grids (deferred fleet-wide sweep, logged). +4. PASS. 5. Commit: `git commit -m "chore(ui): memoize AlarmSummary rows + keyboard/aria-sort on sortable headers (arch-review P4/UA6)"` + +### Task 33: Add CLI.Tests to the slnx + retire the gotcha + +**Classification:** trivial +**Estimated implement time:** ~3 min +**Parallelizable with:** 31, 32 +**Files:** +- Modify: `ZB.MOM.WW.ScadaBridge.slnx` (tests ItemGroup, after line 59) +- Modify: `CLAUDE.md` (Editing Rules bullet about CLI.Tests exclusion; CLI Quick Reference untouched) + +1. Add: `` +2. Verify: `dotnet build ZB.MOM.WW.ScadaBridge.slnx` then `dotnet test ZB.MOM.WW.ScadaBridge.slnx --filter FullyQualifiedName~CLI.Tests` — CLI tests now discovered and green. If the solution build breaks (an undocumented reason for the historical exclusion), revert and record the reason in CLAUDE.md instead — do not leave the solution red. +3. Update the CLAUDE.md Editing Rules bullet: CLI.Tests is now in the slnx; the "silently skipped" gotcha is retired (keep one sentence of history so old memory notes make sense). The user-memory note `cli-tests-not-in-slnx.md` becomes stale — flag it in your final report. +4. Commit: `git add ZB.MOM.WW.ScadaBridge.slnx CLAUDE.md && git commit -m "chore(build): include CLI.Tests in the solution — retire the silent-skip gotcha (arch-review UA3)"` + +--- + +## Dependencies on other plans + +- **Plan 01 (Cluster/Host/Failover):** also flags the `DisableLogin` artifact — **this plan owns the fix** (Tasks 1–3); plan 01 should not duplicate. General active-node/`/health/active` semantics stay with plan 01. +- **Plan 03 (Site Runtime & DCL):** cert-trust **site-side reconcile** (central persistence of trust, PKI-store consistency on failover) is plan 03's; Tasks 21/23 here only add the central relay + Admin gate and do not change site `CertStoreActor` behavior. +- **Plan 05 (Templates/Deployment/Transport):** bundle-import **idempotency on a client-supplied import id** (the other half of S1) belongs to the Transport apply path plan 05 is rewriting — coordinate so `HandleImportBundle` merges cleanly (this plan touches it only to add the Task 30 single-flight gate). +- **Plan 08 (Conventions/Tests):** UA4 (single deferred-work tracking list) is deferred to plan 08's consolidated inventory; Task 33 (slnx) is also listed in the overall report's P2-13 housekeeping — this plan does it, plan 08 should skip it. + +## Execution order + +**P0 (do first, in order):** Task 1 → 2 → 3 (Critical C1 — the auth-disabled artifact). Then the security-correctness wave: 4 → 5 (C2), 6 → 7 → 8 → 9 (C3), 10 (gate refactor) → 11, 12. + +**Ordering constraints:** 2←1; 5←4; 7←6; 11←10; 13←(10,11,22,23); 15←14; 16←15; 17←14; 18←17; 20←19; 22←(10,21); 23←(10,21); 24←(22,23); 25←(22,23); 27←26. + +**Serialize every task that edits `ManagementActor.cs`** (5, 7, 8, 9, 10, 11, 12, 15, 16, 17, 22, 23, 28, 30, 31) — do not run these concurrently even where logically independent. Tasks 1/3/4/6/14/19/21/26/29 are safe to parallelize early; 31/32/33 can land any time. + +**Milestone verification:** after the P0 wave and again at plan completion, run `dotnet build ZB.MOM.WW.ScadaBridge.slnx`, `dotnet test ZB.MOM.WW.ScadaBridge.slnx`, **plus** `dotnet test tests/ZB.MOM.WW.ScadaBridge.CLI.Tests` (until Task 33), and rebuild the docker cluster (`bash docker/deploy.sh`) for a CLI smoke (`--username multi-role --password password`: `deploy artifacts --site-id`, `data-connection get` secret elision, secured-write list gating, a browse call). diff --git a/archreview/plans/PLAN-07-ui-management-security.md.tasks.json b/archreview/plans/PLAN-07-ui-management-security.md.tasks.json new file mode 100644 index 00000000..1df0971b --- /dev/null +++ b/archreview/plans/PLAN-07-ui-management-security.md.tasks.json @@ -0,0 +1,39 @@ +{ + "planPath": "archreview/plans/PLAN-07-ui-management-security.md", + "tasks": [ + { "id": 1, "subject": "Task 1: DisableLoginGuard — refuse the flag outside Development", "status": "pending", "blockedBy": [] }, + { "id": 2, "subject": "Task 2: Wire the guard at the Host composition root + doc", "status": "pending", "blockedBy": [1] }, + { "id": 3, "subject": "Task 3: Scrub deploy/wonder-app-vd03 — remove DisableLogin: true", "status": "pending", "blockedBy": [] }, + { "id": 4, "subject": "Task 4: ArtifactDeploymentService.DeployToSiteAsync", "status": "pending", "blockedBy": [] }, + { "id": 5, "subject": "Task 5: HandleDeployArtifacts honors SiteId + site-scope enforcement + docs", "status": "pending", "blockedBy": [4] }, + { "id": 6, "subject": "Task 6: ConfigSecretScrubber — JSON secret elision + sentinel merge", "status": "pending", "blockedBy": [] }, + { "id": 7, "subject": "Task 7: DataConnection List/Get/audit secret elision + sentinel-preserving update", "status": "pending", "blockedBy": [6] }, + { "id": 8, "subject": "Task 8: ExternalSystem AuthConfiguration elision + preserve-if-null update", "status": "pending", "blockedBy": [] }, + { "id": 9, "subject": "Task 9: DatabaseConnection ConnectionString elision + preserve-if-null update", "status": "pending", "blockedBy": [] }, + { "id": 10, "subject": "Task 10: GetRequiredRoles any-of refactor + gate ListSecuredWrites", "status": "pending", "blockedBy": [] }, + { "id": 11, "subject": "Task 11: Reconcile the area-management role gate (code + three docs)", "status": "pending", "blockedBy": [10] }, + { "id": 12, "subject": "Task 12: Canonicalize role-mapping casing at the write path", "status": "pending", "blockedBy": [] }, + { "id": 13, "subject": "Task 13: GetRequiredRoles matrix test — freeze the authorization table", "status": "pending", "blockedBy": [10, 11, 22, 23] }, + { "id": 14, "subject": "Task 14: SecuredWrite repository — TryMarkExpiredAsync + CountAsync", "status": "pending", "blockedBy": [] }, + { "id": 15, "subject": "Task 15: Server-side secured-write TTL — enforce at approve/reject", "status": "pending", "blockedBy": [14] }, + { "id": 16, "subject": "Task 16: Opportunistic expiry sweep in HandleListSecuredWrites", "status": "pending", "blockedBy": [15] }, + { "id": 17, "subject": "Task 17: Secured-write list paging — command, handler, CLI", "status": "pending", "blockedBy": [14] }, + { "id": 18, "subject": "Task 18: SecuredWrites page — history pager + age in approve dialog + docs", "status": "pending", "blockedBy": [17] }, + { "id": 19, "subject": "Task 19: Per-command long-running Ask timeout", "status": "pending", "blockedBy": [] }, + { "id": 20, "subject": "Task 20: Ship ManagementService timeout config + doc", "status": "pending", "blockedBy": [19] }, + { "id": 21, "subject": "Task 21: Browse/Verify/Cert commands gain optional SiteIdentifier", "status": "pending", "blockedBy": [] }, + { "id": 22, "subject": "Task 22: Actor handlers — BrowseNode / SearchAddressSpace / VerifyEndpoint", "status": "pending", "blockedBy": [10, 21] }, + { "id": 23, "subject": "Task 23: Actor handlers — TrustServerCert / RemoveServerCert / ListServerCerts (Admin)", "status": "pending", "blockedBy": [10, 21] }, + { "id": 24, "subject": "Task 24: Dispatch-coverage test — every registered command reaches a handler", "status": "pending", "blockedBy": [22, 23] }, + { "id": 25, "subject": "Task 25: CLI parity — browse / search / verify-endpoint / certs commands", "status": "pending", "blockedBy": [22, 23] }, + { "id": 26, "subject": "Task 26: LoginThrottle — fixed-window LDAP-bind failure lockout", "status": "pending", "blockedBy": [] }, + { "id": 27, "subject": "Task 27: Wire the throttle — ManagementAuthenticator + auth endpoints + doc", "status": "pending", "blockedBy": [26] }, + { "id": 28, "subject": "Task 28: Additive paging for ListTemplates / ListInstances", "status": "pending", "blockedBy": [] }, + { "id": 29, "subject": "Task 29: PollGate — non-reentrant poll timers on Health + Alarm Summary", "status": "pending", "blockedBy": [] }, + { "id": 30, "subject": "Task 30: Transport single-flight + base64 copy elimination", "status": "pending", "blockedBy": [] }, + { "id": 31, "subject": "Task 31: Low cleanup A — hub send fault logging, event-log filter comment, stale CLAUDE.md note", "status": "pending", "blockedBy": [] }, + { "id": 32, "subject": "Task 32: Low cleanup B — AlarmSummary memoization + sortable-header a11y", "status": "pending", "blockedBy": [] }, + { "id": 33, "subject": "Task 33: Add CLI.Tests to the slnx + retire the gotcha", "status": "pending", "blockedBy": [] } + ], + "lastUpdated": "2026-07-08" +} diff --git a/archreview/plans/PLAN-08-conventions-tests.md b/archreview/plans/PLAN-08-conventions-tests.md new file mode 100644 index 00000000..893279e1 --- /dev/null +++ b/archreview/plans/PLAN-08-conventions-tests.md @@ -0,0 +1,739 @@ +# Conventions, Tests & Underdeveloped Areas Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. + +**Goal:** Close the cross-cutting gaps from architecture review 08 — the CLI.Tests slnx blind spot, the ~12 components binding options without startup validation, the vestigial site-side notification surface, the misnamed-thin PerformanceTests project, the docs-code drift owned by no other plan, and a triaged register for the 23-item deferred-work inventory. + +**Architecture:** All changes are convention-hardening around existing seams: eager options validation copies the shipped `OptionsValidatorBase` pattern (`ZB.MOM.WW.Configuration` package) already used by HealthMonitoring/ClusterInfrastructure/AuditLog/KpiHistory; the vestigial notification removal relies on the existing `PurgeCentralOnlyNotificationConfigAsync` guarantee (which stays); new tests slot into existing test projects (PerformanceTests, Commons.Tests). No new runtime components, no schema changes, no wire-contract changes. + +**Tech Stack:** C# / .NET 10, xUnit, Akka.NET TestKit/Streams, `Microsoft.Extensions.Options` (`IValidateOptions<>` + `ValidateOnStart`), Newtonsoft.Json (contract-lock tests only), SQLite (site storage). + +**Build:** `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Tests: `dotnet test ZB.MOM.WW.ScadaBridge.slnx` — after Task 1 this picks up CLI.Tests too. + +## Findings Coverage + +| Report finding (08-conventions-tests-underdeveloped.md) | Severity | Coverage | +|---|---|---| +| §1.1 Commons folders beyond the documented four | Low | Task 10 — **narrowed on re-verify**: `Serialization/` and `Validators/` are already in `Component-Commons.md`'s layout tree (lines 293–294); only `Observability/` is missing | +| §1.2 POCO persistence-ignorance | Pass | No action | +| §1.3 Site-side repository-implementation exception undocumented | Low | Task 10 | +| §1.3 / §3 item 23 Vestigial `SiteNotificationRepository` + `StoreNotificationListAsync` write path | Low/Medium | Task 6 | +| §1.4 `Communication` → `HealthMonitoring` layering inversion | Low | **Deferred** (Task 11 register entry): moving `ICentralHealthAggregator` + `SiteHealthState` to Commons ripples across HealthMonitoring, Communication, ManagementService, CentralUI, AuditLog + their tests — disproportionate to a cosmetic note; documented as accepted with revisit trigger | +| §1.4 `ManagementService.csproj` `..\` path-separator inconsistency | Low | Task 10 | +| §1.5 Options validation on only ~6 of ~20 option classes | Medium | Tasks 2, 3, 4, 5 | +| §1.6 UTC / §1.7 correlation IDs / §1.9 naming | Pass | No action | +| §1.8 No contract-lock tests for ClusterClient record contracts (also Rec 10) | Low | Task 9 | +| §2.1 Thin test spots (SiteCallAudit.Tests, DeploymentManager.Tests) | Info | **Deferred** (Task 11 register entry): no defect identified; backfill tracked with revisit trigger | +| §2.3 PerformanceTests is a correctness-at-small-scale suite | Medium | Tasks 7 (streaming throughput) + 8 (failover-timing harness placeholder, delegating the rig to Plan 01) | +| §2.4 CLI.Tests excluded from slnx | High | Task 1 | +| §3 item 1 / §4 Transport `AreaName: null` vs doc claim | High | **Owned by plan 05** | +| §3 item 3 AuditLog reconciliation dials NodeA only | Medium | **Owned by plan 04** | +| §3 item 5 `SmsConfiguration.MaxRetryCount` stored but unused | Medium | **Owned by plan 06** | +| §3 item 6 Role-check case-sensitivity asymmetry | Medium | **Owned by plan 07** | +| §3 items 7–22 (hash-chain, Parquet, live alarm stream, cert-trust central persistence, CSV parity, WaitForAttribute, BrowseNext, StubOpcUaClient, unified outbox page, folder drag-drop, bundle signing, EXPIRED purge, backlog-reporter options, KPI downsampling) | Low | Task 11 — each gets an explicit register row with defer rationale + revisit trigger (none needs code in this plan) | +| §4 `docs/components/` lags 3 components vs README claim | Low | Task 10 (scope the claim; backfill tracked in Task 11 register) | +| §4 DelmiaNotifier breaks `Component-.md` convention | Low | Task 10 (document the exception) | +| §4 Component-KpiHistory / StoreAndForward / ClusterInfrastructure doc sync | Pass | No action | + +--- + +### Task 1: Add CLI.Tests to the slnx and retire the gotcha documentation + +**Classification:** small +**Estimated implement time:** ~3 min +**Parallelizable with:** 2, 3, 4, 5, 6, 7, 8, 9, 11 +**Files:** +- Modify: `ZB.MOM.WW.ScadaBridge.slnx` (insert one line after line 51, the ManagementService.Tests entry, mirroring src ordering) +- Modify: `CLAUDE.md` (line 69, the Editing Rules test-command bullet) +- Modify: `/Users/dohertj2/.claude/projects/-Users-dohertj2-Desktop-ScadaBridge/memory/MEMORY.md` (index entry "CLI.Tests is not in the slnx") and `/Users/dohertj2/.claude/projects/-Users-dohertj2-Desktop-ScadaBridge/memory/cli-tests-not-in-slnx.md` + +1. Verify current state (expect CLI.Tests absent): + ```bash + grep -c "CLI.Tests" ZB.MOM.WW.ScadaBridge.slnx # expect: 0 + ``` +2. Edit `ZB.MOM.WW.ScadaBridge.slnx`: after the line + ` ` + insert: + ```xml + + ``` +3. Verify the solution builds and the CLI tests now run at the solution level: + ```bash + dotnet build ZB.MOM.WW.ScadaBridge.slnx + dotnet test ZB.MOM.WW.ScadaBridge.slnx --no-build --filter "FullyQualifiedName~ScadaBridge.CLI.Tests" + ``` + Expected: build succeeds; the test run lists `ZB.MOM.WW.ScadaBridge.CLI.Tests` and reports ~279 passing tests (0 failed). If any CLI test fails to compile, fix forward — that is exactly the rot this task exists to expose. +4. Update `CLAUDE.md:69`. Replace the bullet: + > `- Run tests with \`dotnet test ZB.MOM.WW.ScadaBridge.slnx\`, **but \`ZB.MOM.WW.ScadaBridge.CLI.Tests\` is not in the slnx** and is silently skipped — when touching the CLI, test that project directly (\`dotnet test tests/ZB.MOM.WW.ScadaBridge.CLI.Tests\`), or a green solution run can hide CLI tests that never compiled.` + + with: + > `- Run tests with \`dotnet test ZB.MOM.WW.ScadaBridge.slnx\`. All 30 test projects, including \`ZB.MOM.WW.ScadaBridge.CLI.Tests\`, are members of the slnx (CLI.Tests was added 2026-07; the old "silently skipped" gotcha no longer applies).` +5. Update the auto-memory: in `MEMORY.md`, change the index line for `cli-tests-not-in-slnx.md` to note the gotcha is fixed (e.g. "FIXED 2026-07: CLI.Tests is now in the slnx; entry kept as history"), and add a first-line status note to `cli-tests-not-in-slnx.md` itself. +6. Commit: + ```bash + git add ZB.MOM.WW.ScadaBridge.slnx CLAUDE.md + git commit -m "build(slnx): add CLI.Tests to the solution — retire the silently-skipped-279-tests gotcha (arch-review 08 §2.4)" + ``` + (The memory files live outside the repo and are not committed.) + +--- + +### Task 2: Options validation — Communication + DataConnectionLayer + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 3, 4, 5, 6, 7, 8, 9, 10, 11 +**Files:** +- Create: `src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptionsValidator.cs` +- Create: `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/DataConnectionOptionsValidator.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/ServiceCollectionExtensions.cs` (lines 13–14), `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/ServiceCollectionExtensions.cs` (lines 15–22) +- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/ZB.MOM.WW.ScadaBridge.Communication.csproj`, `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.csproj` (add `` — no Version attribute; central package management already has it, see `HealthMonitoring.csproj:15`) +- Test: `tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/CommunicationOptionsValidatorTests.cs`, `tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/DataConnectionOptionsValidatorTests.cs` + +The pattern to copy is `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/HealthMonitoringOptionsValidator.cs` (an `OptionsValidatorBase` from the `ZB.MOM.WW.Configuration` package with `builder.RequireThat(...)` per field) registered via `TryAddEnumerable(ServiceDescriptor.Singleton, TValidator>())` + `.ValidateOnStart()` on the `AddOptions` chain (`HealthMonitoring/ServiceCollectionExtensions.cs:157-162`). Tests copy `tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/HealthMonitoringOptionsValidatorTests.cs`. + +1. Write failing tests first — `tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/CommunicationOptionsValidatorTests.cs`: + ```csharp + using Microsoft.Extensions.Options; + + namespace ZB.MOM.WW.ScadaBridge.Communication.Tests; + + public class CommunicationOptionsValidatorTests + { + private static ValidateOptionsResult Validate(CommunicationOptions options) => + new CommunicationOptionsValidator().Validate(Options.DefaultName, options); + + [Fact] + public void DefaultOptions_AreValid() + { + var result = Validate(new CommunicationOptions()); + Assert.True(result.Succeeded, result.FailureMessage); + } + + [Fact] + public void ZeroDeploymentTimeout_IsRejected() + { + var result = Validate(new CommunicationOptions { DeploymentTimeout = TimeSpan.Zero }); + Assert.True(result.Failed); + Assert.Contains("DeploymentTimeout", result.FailureMessage); + } + + [Fact] + public void NonPositiveGrpcMaxConcurrentStreams_IsRejected() + { + var result = Validate(new CommunicationOptions { GrpcMaxConcurrentStreams = 0 }); + Assert.True(result.Failed); + Assert.Contains("GrpcMaxConcurrentStreams", result.FailureMessage); + } + } + ``` + Mirror the same three-test shape for `DataConnectionOptionsValidator` (`ReconnectInterval`, `SeedReadMaxAttempts`). +2. Run, expect FAIL (validator types don't exist → compile error, which counts): + ```bash + dotnet test tests/ZB.MOM.WW.ScadaBridge.Communication.Tests --filter "FullyQualifiedName~CommunicationOptionsValidator" + ``` +3. Implement `CommunicationOptionsValidator` (positive-duration checks for every `TimeSpan` on `CommunicationOptions` — `DeploymentTimeout`, `LifecycleTimeout`, `ArtifactDeploymentTimeout`, `QueryTimeout`, `IntegrationTimeout`, `DebugViewTimeout`, `HealthReportTimeout`, `NotificationForwardTimeout`, `GrpcKeepAlivePingDelay`, `GrpcKeepAlivePingTimeout`, `GrpcMaxStreamLifetime`, `TransportHeartbeatInterval`, `TransportFailureThreshold` — plus `GrpcMaxConcurrentStreams > 0`). Error messages must name the config key (`Communication:`), same style as `HealthMonitoringOptionsValidator`: + ```csharp + using ZB.MOM.WW.Configuration; + + namespace ZB.MOM.WW.ScadaBridge.Communication; + + /// + /// Validates at startup (ValidateOnStart) so a + /// malformed "Communication" appsettings section fails fast at boot with a + /// key-naming message instead of surfacing at first Ask/gRPC use. + /// + public sealed class CommunicationOptionsValidator : OptionsValidatorBase + { + protected override void Validate(ValidationBuilder builder, CommunicationOptions options) + { + builder.RequireThat(options.DeploymentTimeout > TimeSpan.Zero, + $"Communication:DeploymentTimeout must be a positive duration (was {options.DeploymentTimeout})."); + // ... one RequireThat per field listed above ... + builder.RequireThat(options.GrpcMaxConcurrentStreams > 0, + $"Communication:GrpcMaxConcurrentStreams must be positive (was {options.GrpcMaxConcurrentStreams})."); + } + } + ``` + `DataConnectionOptionsValidator` likewise: `ReconnectInterval`, `TagResolutionRetryInterval`, `WriteTimeout`, `SeedReadTimeout`, `StableConnectionThreshold`, `SeedReadRetryDelay` positive durations; `SeedReadMaxAttempts > 0`. Read `OpcUaGlobalOptions`/`MxGatewayGlobalOptions` (`src/.../DataConnectionLayer/`) and add `RequireThat` checks for any numeric/duration fields they carry (add validators for them only if they have such fields). +4. Wire registration. `Communication/ServiceCollectionExtensions.cs:13-14` becomes: + ```csharp + services.AddOptions() + .BindConfiguration("Communication") + .ValidateOnStart(); + services.TryAddEnumerable( + ServiceDescriptor.Singleton, CommunicationOptionsValidator>()); + ``` + Same shape for the three `AddOptions` calls in `DataConnectionLayer/ServiceCollectionExtensions.cs:15-22`. Add the `ZB.MOM.WW.Configuration` PackageReference to both csprojs. +5. Run, expect PASS: + ```bash + dotnet test tests/ZB.MOM.WW.ScadaBridge.Communication.Tests --filter "FullyQualifiedName~OptionsValidator" + dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~OptionsValidator" + dotnet build ZB.MOM.WW.ScadaBridge.slnx + ``` +6. Commit: + ```bash + git add src/ZB.MOM.WW.ScadaBridge.Communication src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer tests/ZB.MOM.WW.ScadaBridge.Communication.Tests tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests + git commit -m "feat(options): eager startup validation for Communication + DataConnectionLayer options (arch-review 08 §1.5)" + ``` + +--- + +### Task 3: Options validation — site pipeline (SiteRuntime, StoreAndForward, SiteEventLogging) + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 4, 5, 7, 8, 9, 10, 11 (NOT 6 — both touch the SiteRuntime project; run sequentially to keep csproj/test churn separate) +**Files:** +- Create: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs`, `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptionsValidator.cs`, `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogOptionsValidator.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs` (lines 124–127 — the `services.Configure(config.GetSection(...))` calls for these three options) +- Modify: the three component csprojs (add ``) +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/SiteRuntimeOptionsValidatorTests.cs`, `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardOptionsValidatorTests.cs`, `tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLogOptionsValidatorTests.cs` + +These three options classes are bound in the Host, not in the component SCEs (`Host/SiteServiceRegistration.cs:124-127`), so binding conversion happens there; validator classes live with their owners per the "options classes owned by component projects" convention. + +1. Write failing tests (same shape as Task 2 step 1; one `DefaultOptions_AreValid` + 2–3 rejection cases per validator). Key rejection cases: + - `SiteRuntimeOptions`: `StartupBatchSize = 0`, `ScriptExecutionThreadCount = 0`, `StreamBufferSize = 0` (all feed thread-pool/stream sizing; also validate `StartupBatchDelayMs >= 0`, `MaxScriptCallDepth > 0`, `ScriptExecutionTimeoutSeconds > 0`, `MirroredAlarmCapPerSource > 0`, `NativeAlarmRetryIntervalMs > 0`, `ConfigFetchTimeoutSeconds > 0`, `ConfigFetchRetryCount >= 0`). + - `StoreAndForwardOptions`: `DefaultRetryInterval = TimeSpan.Zero`, `RetryTimerInterval = TimeSpan.Zero`, `SqliteDbPath = ""` (also `DefaultMaxRetries >= 0`). + - `SiteEventLogOptions`: `RetentionDays = 0`, `PurgeInterval = TimeSpan.Zero`, `MaxQueryPageSize < QueryPageSize` cross-check (also `MaxStorageMb > 0`, `WriteQueueCapacity > 0`, `DatabasePath` non-empty). +2. Run, expect FAIL (compile error on missing validator types): + ```bash + dotnet test tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests --filter "FullyQualifiedName~OptionsValidator" + ``` +3. Implement the three validators (`OptionsValidatorBase`, key-naming messages using the `ScadaBridge:SiteRuntime` / `ScadaBridge:StoreAndForward` / `ScadaBridge:SiteEventLog` prefixes). Convert the Host bindings — `Host/SiteServiceRegistration.cs:124-127` pattern per option: + ```csharp + services.AddOptions() + .Bind(config.GetSection("ScadaBridge:SiteRuntime")) + .ValidateOnStart(); + services.TryAddEnumerable( + ServiceDescriptor.Singleton, SiteRuntimeOptionsValidator>()); + ``` + (Keep any other `Configure<>` calls in that block untouched.) +4. Run, expect PASS; then guard against startup regressions (Host composition-root tests boot both roles): + ```bash + dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~OptionsValidator" + dotnet test tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests --filter "FullyQualifiedName~OptionsValidator" + dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests --filter "FullyQualifiedName~OptionsValidator" + dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~CompositionRoot" + ``` +5. Commit: + ```bash + git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime src/ZB.MOM.WW.ScadaBridge.StoreAndForward src/ZB.MOM.WW.ScadaBridge.SiteEventLogging src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests + git commit -m "feat(options): eager startup validation for site-pipeline options (SiteRuntime, S&F, SiteEventLog) (arch-review 08 §1.5)" + ``` + +--- + +### Task 4: Options validation — edge + management (InboundAPI, ExternalSystemGateway, NotificationService, ManagementService) + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 3, 5, 6, 7, 8, 9, 10, 11 +**Files:** +- Create: `src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundApiOptionsValidator.cs`, `src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemGatewayOptionsValidator.cs`, `src/ZB.MOM.WW.ScadaBridge.NotificationService/NotificationOptionsValidator.cs`, `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementServiceOptionsValidator.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` (line 273 — `Configure` → `AddOptions().Bind().ValidateOnStart()` + validator registration), `src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ServiceCollectionExtensions.cs` (lines 23–24), `src/ZB.MOM.WW.ScadaBridge.NotificationService/ServiceCollectionExtensions.cs` (lines 20–21), `src/ZB.MOM.WW.ScadaBridge.ManagementService/ServiceCollectionExtensions.cs` (lines 15–16) +- Modify: the four component csprojs (add `ZB.MOM.WW.Configuration` PackageReference) +- Test: `tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/InboundApiOptionsValidatorTests.cs`, `tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemGatewayOptionsValidatorTests.cs`, `tests/ZB.MOM.WW.ScadaBridge.NotificationService.Tests/NotificationOptionsValidatorTests.cs`, `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/ManagementServiceOptionsValidatorTests.cs` + +1. Write failing tests (Task 2 shape). Rejection cases: + - `InboundApiOptions`: `DefaultMethodTimeout = TimeSpan.Zero`, `MaxRequestBodyBytes = 0`. Do **not** require `ApiKeyPepper` non-empty (empty is a legitimate dev default — check `Host/Program.cs` usage before deciding; if a non-empty pepper is required in production the `StartupValidator` owns that, not this validator). + - `ExternalSystemGatewayOptions`: `DefaultHttpTimeout = TimeSpan.Zero`, `MaxConcurrentConnectionsPerSystem = 0`. + - `NotificationOptions`: `ConnectionTimeoutSeconds = 0`, `MaxConcurrentConnections = 0`. + - `ManagementServiceOptions`: `CommandTimeout = TimeSpan.Zero`. +2. Run one project, expect FAIL: + ```bash + dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests --filter "FullyQualifiedName~OptionsValidator" + ``` +3. Implement the four validators (key prefixes `ScadaBridge:InboundApi`, `ScadaBridge:ExternalSystemGateway`, `ScadaBridge:Notification`, `ScadaBridge:ManagementService`) and convert each binding site to `AddOptions().BindConfiguration(...)/.Bind(...).ValidateOnStart()` + `TryAddEnumerable` validator registration (exactly the Task 2 step 4 shape). +4. Run, expect PASS: + ```bash + dotnet test tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests --filter "FullyQualifiedName~OptionsValidator" + dotnet test tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests --filter "FullyQualifiedName~OptionsValidator" + dotnet test tests/ZB.MOM.WW.ScadaBridge.NotificationService.Tests --filter "FullyQualifiedName~OptionsValidator" + dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests --filter "FullyQualifiedName~OptionsValidator" + dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~CompositionRoot" + ``` +5. Commit: + ```bash + git add src/ZB.MOM.WW.ScadaBridge.InboundAPI src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway src/ZB.MOM.WW.ScadaBridge.NotificationService src/ZB.MOM.WW.ScadaBridge.ManagementService src/ZB.MOM.WW.ScadaBridge.Host/Program.cs tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests tests/ZB.MOM.WW.ScadaBridge.NotificationService.Tests tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests + git commit -m "feat(options): eager startup validation for edge/management options (InboundAPI, ESG, Notification, ManagementService) (arch-review 08 §1.5)" + ``` + +--- + +### Task 5: Options validation — central components (Transport, SiteCallAudit, DeploymentManager) + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 3, 4, 6, 7, 8, 9, 10, 11 +**Files:** +- Create: `src/ZB.MOM.WW.ScadaBridge.Transport/TransportOptionsValidator.cs`, `src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/SiteCallAuditOptionsValidator.cs`, `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentManagerOptionsValidator.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/ServiceCollectionExtensions.cs` (line 25), `src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/ServiceCollectionExtensions.cs` (lines 40–41), `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/ServiceCollectionExtensions.cs` (line 31) +- Modify: the three component csprojs (add `ZB.MOM.WW.Configuration` PackageReference) +- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.Tests/TransportOptionsValidatorTests.cs`, `tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests/SiteCallAuditOptionsValidatorTests.cs`, `tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests/DeploymentManagerOptionsValidatorTests.cs` + +1. Write failing tests (Task 2 shape). Rejection cases: + - `TransportOptions` (`ScadaBridge:Transport`): `BundleSessionTtlMinutes = 0`, `MaxBundleSizeMb = 0`, `Pbkdf2Iterations = 0` (security-sensitive: also require `Pbkdf2Iterations >= 100_000` so a config typo cannot silently weaken bundle key derivation), `MaxUnlockAttemptsPerSession = 0`, plus positivity for `MaxBundleEntryDecompressedMb`, `MaxBundleEntryCount`, `MaxBundleEntryCompressionRatio`, `MaxUnlockAttemptsPerIpPerHour`, `SchemaVersionMajor`, and `SourceEnvironment` non-empty. + - `SiteCallAuditOptions` (`ScadaBridge:SiteCallAudit`): `StuckAgeThreshold = TimeSpan.Zero`, `KpiInterval = TimeSpan.Zero`, `ReconciliationBatchSize = 0`, `RetentionDays = 0`, plus positivity for `RelayTimeout`, `ReconciliationInterval`, `PurgeInterval`. Note: validate the base properties, not the `Resolved*` computed properties. + - `DeploymentManagerOptions` (`ScadaBridge:DeploymentManager` — confirm the section name at the binding site; `ServiceCollectionExtensions.cs:31` uses bare `AddOptions()`, i.e. code-configured defaults, so ValidateOnStart still applies): `LifecycleCommandTimeout = TimeSpan.Zero`, `ArtifactDeploymentTimeoutPerSite = TimeSpan.Zero`, `OperationLockTimeout = TimeSpan.Zero`. +2. Run one project, expect FAIL: + ```bash + dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.Tests --filter "FullyQualifiedName~OptionsValidator" + ``` +3. Implement the three validators + registration conversion (Task 2 step 4 shape). Transport becomes: + ```csharp + services.AddOptions().BindConfiguration(OptionsSection).ValidateOnStart(); + services.TryAddEnumerable( + ServiceDescriptor.Singleton, TransportOptionsValidator>()); + ``` +4. Run, expect PASS: + ```bash + dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.Tests --filter "FullyQualifiedName~OptionsValidator" + dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests --filter "FullyQualifiedName~OptionsValidator" + dotnet test tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests --filter "FullyQualifiedName~OptionsValidator" + dotnet build ZB.MOM.WW.ScadaBridge.slnx + ``` +5. Commit: + ```bash + git add src/ZB.MOM.WW.ScadaBridge.Transport src/ZB.MOM.WW.ScadaBridge.SiteCallAudit src/ZB.MOM.WW.ScadaBridge.DeploymentManager tests/ZB.MOM.WW.ScadaBridge.Transport.Tests tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests + git commit -m "feat(options): eager startup validation for central-component options (Transport, SiteCallAudit, DeploymentManager) (arch-review 08 §1.5)" + ``` + +--- + +### Task 6: Remove the vestigial site-side notification-list surface + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 4, 5, 7, 8, 9, 11 (NOT 3 — same project; NOT 10 — both touch `docs/components/SiteRuntime.md`? No: Task 10 does not touch that file. Parallelizable with 10 too, but keep 3 sequential.) +**Files:** +- Delete: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/SiteNotificationRepository.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs` (line 65 — remove `services.AddScoped();`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs` — remove `StoreNotificationListAsync` (lines 715–739) and `StoreSmtpConfigurationAsync` (lines ~741–785). **Keep** `PurgeCentralOnlyNotificationConfigAsync` (lines 691–713) and the table schemas — the purge is the security cleanup for DBs written by older builds. +- Modify tests: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CompositionRootTests.cs` (lines 444–451), `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Repositories/SiteRepositoryTests.cs` (~lines 185–200), `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/ArtifactStorageTests.cs` (all `StoreNotificationListAsync`/`StoreSmtpConfigurationAsync`/`SiteNotificationRepository` usages: ~lines 125–195) +- Modify: `docs/components/SiteRuntime.md` (line 15 — drop `SiteNotificationRepository` from the `Repositories/` listing and note why) + +Context (verified): the central-only notification design means `PurgeCentralOnlyNotificationConfigAsync` empties `notification_lists` + `smtp_configurations` on every artifact apply (`DeploymentManagerActor.cs:1556`, `SiteReplicationActor.cs:267`), so the registered `SiteNotificationRepository` can only ever read empty tables, and the two `Store*` methods have **zero production callers** (grep: only tests). `SiteExternalSystemRepository` is NOT vestigial — external system definitions do ship to sites — leave it alone. + +1. Write the failing test first — replace `Site_NotificationRepository_IsSiteImplementation` in `CompositionRootTests.cs:444-451` with a design-invariant lock: + ```csharp + [Fact] + public void Site_NotificationRepository_IsNotRegistered() + { + // Notification delivery is central-only (CLAUDE.md "Notification delivery is + // central-only"); the site-local notification_lists/smtp_configurations tables + // are purged on every artifact apply, so a site-side INotificationRepository + // could only ever serve empty results. It must not be registered at all. + using var scope = _host.Services.CreateScope(); + var repo = scope.ServiceProvider.GetService(); + Assert.Null(repo); + } + ``` +2. Run, expect FAIL (repo is still registered): + ```bash + dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~Site_NotificationRepository" + ``` +3. Implement: delete `SiteNotificationRepository.cs`; remove the DI registration at `SiteRuntime/ServiceCollectionExtensions.cs:65`; delete `StoreNotificationListAsync` + `StoreSmtpConfigurationAsync` from `SiteStorageService.cs`; extend the doc comment on `PurgeCentralOnlyNotificationConfigAsync` with one sentence noting the write paths and site repository were removed (with date + review pointer) so a future maintainer doesn't reintroduce them. Check whether `SiteRuntime/Repositories/SyntheticId.cs` is still used by `SiteExternalSystemRepository` — if yes keep it, if the notification repo was its last consumer delete it too. +4. Delete/rewrite the orphaned tests: the notification-list blocks in `SiteRepositoryTests.cs` and `ArtifactStorageTests.cs` (they exercise deleted API). Where a test asserted persistence-across-restart semantics generically, keep the external-system variant only. +5. Run, expect PASS: + ```bash + dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~CompositionRoot" + dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests + dotnet build ZB.MOM.WW.ScadaBridge.slnx + ``` +6. Update `docs/components/SiteRuntime.md:15` (`Repositories/` bullet → `SiteExternalSystemRepository` only, with a one-line note that the notification variant was removed because notification config never lives on sites). +7. Commit: + ```bash + git add -A src/ZB.MOM.WW.ScadaBridge.SiteRuntime tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests tests/ZB.MOM.WW.ScadaBridge.Host.Tests docs/components/SiteRuntime.md + git commit -m "refactor(site-runtime): excise vestigial site-side notification-list surface — repo, DI registration, dead write paths (arch-review 08 §1.3/#23)" + ``` + +--- + +### Task 7: PerformanceTests — site-stream throughput test + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 3, 4, 5, 6, 9, 10, 11 (and 8 — different new files) +**Files:** +- Create: `tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Streaming/SiteStreamThroughputTests.cs` +- (No csproj change needed: `PerformanceTests.csproj` already references SiteRuntime and Commons.) + +This adds the first of the two highest-value real measurements the review called for (§2.3): sustained throughput of the site-wide Akka stream (`SiteStreamManager`, `Source.ActorRef` + `BroadcastHub`, per-subscriber buffering) that carries every attribute value and alarm state change on a site. Follows the existing `HotPathLatencyTests` style: `Stopwatch`-based, `[Trait("Category", "Performance")]`, conservative thresholds so CI stays green on slow machines. + +1. Write the test (this is the implementation — a perf test is its own assertion): + ```csharp + using System.Diagnostics; + using Akka.Actor; + using Akka.TestKit.Xunit2; + using Microsoft.Extensions.Logging.Abstractions; + using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming; + using ZB.MOM.WW.ScadaBridge.SiteRuntime; + using ZB.MOM.WW.ScadaBridge.SiteRuntime.Streaming; + + namespace ZB.MOM.WW.ScadaBridge.PerformanceTests.Streaming; + + /// + /// Throughput measurement for the site-wide attribute/alarm stream + /// (SiteStreamManager: Source.ActorRef → BroadcastHub with per-subscriber + /// buffering, DropHead overflow). Design envelope: this stream carries every + /// attribute value change on a site; it must comfortably sustain tens of + /// thousands of events/sec. Thresholds are deliberately conservative + /// (CI-machine safe) — this test exists to catch order-of-magnitude + /// regressions, not to benchmark. + /// + public class SiteStreamThroughputTests : TestKit + { + private sealed class CountingActor : ReceiveActor + { + public static long Received; + public CountingActor() + { + Receive(_ => Interlocked.Increment(ref Received)); + ReceiveAny(_ => { }); + } + } + + [Trait("Category", "Performance")] + [Fact] + public async Task SiteStream_SustainsAtLeast10kEventsPerSecond_ToOneSubscriber() + { + const int eventCount = 100_000; + CountingActor.Received = 0; + + var manager = new SiteStreamManager( + new SiteRuntimeOptions { StreamBufferSize = eventCount }, + NullLogger.Instance); + manager.Initialize(Sys); + + var counter = Sys.ActorOf(Props.Create()); + manager.Subscribe("PerfInstance", counter); + + var sw = Stopwatch.StartNew(); + for (var i = 0; i < eventCount; i++) + { + manager.PublishAttributeValueChanged(new AttributeValueChanged( + "PerfInstance", "Attr", "Attr", i, "Good", DateTimeOffset.UtcNow)); + } + + // Drain: wait until the counter goes quiet (no growth across a 200ms window). + long last = -1; + var deadline = DateTime.UtcNow.AddSeconds(30); + while (DateTime.UtcNow < deadline) + { + var now = Interlocked.Read(ref CountingActor.Received); + if (now == last && now > 0) break; + last = now; + await Task.Delay(200); + } + sw.Stop(); + + var received = Interlocked.Read(ref CountingActor.Received); + var eventsPerSecond = received / sw.Elapsed.TotalSeconds; + + // DropHead means under extreme pressure some events may drop; with the + // buffer sized to the burst, deliver ratio should be ~1.0. + Assert.True(received >= eventCount * 0.9, + $"Expected >=90% delivery, got {received}/{eventCount}"); + Assert.True(eventsPerSecond >= 10_000, + $"Expected >=10k events/s sustained, got {eventsPerSecond:F0}/s over {sw.Elapsed.TotalSeconds:F1}s"); + } + } + ``` + Adjust to actual constructor/subscribe signatures on read (`SiteStreamManager.Subscribe(string instanceName, IActorRef subscriber)` — verified; subscription filters by `InstanceUniqueName`). If subscriber delivery turns out to route a wrapper message rather than the raw record, match on the wrapper. +2. Run, expect PASS (and eyeball the reported rate in the failure-message format by temporarily inverting the assertion if you want the number): + ```bash + dotnet test tests/ZB.MOM.WW.ScadaBridge.PerformanceTests --filter "FullyQualifiedName~SiteStreamThroughput" + ``` +3. Commit: + ```bash + git add tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Streaming + git commit -m "test(perf): site-wide stream throughput measurement — first real perf-envelope test (arch-review 08 §2.3)" + ``` + +--- + +### Task 8: PerformanceTests — failover-timing benchmark harness placeholder + +**Classification:** small +**Estimated implement time:** ~3 min +**Parallelizable with:** 1, 2, 3, 4, 5, 6, 7, 9, 10, 11 +**Files:** +- Create: `tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Failover/FailoverTimingTests.cs` + +The design claims ~25s total failover (CLAUDE.md "Cluster & Failover"). Measuring it needs a real two-node cluster with a hard kill — **that rig is owned by Plan 01** (the two-node failover test rig from overall recommendation P2-10). This task ships the measurement protocol as an explicitly-skipped harness so the perf suite documents the envelope and Plan 01's rig has a ready seat to plug into; it must NOT duplicate the rig. + +1. Create the placeholder: + ```csharp + namespace ZB.MOM.WW.ScadaBridge.PerformanceTests.Failover; + + /// + /// Failover-timing benchmark harness (placeholder). + /// + /// Design envelope under test (CLAUDE.md "Cluster & Failover"): failure + /// detection 2s heartbeat / 10s threshold, SBR stable-after 15s, total + /// failover ~25s for a hard node loss. + /// + /// Measurement protocol (to be wired to the two-node cluster rig delivered + /// by PLAN-01 — see archreview/plans/PLAN-01-*.md; do not duplicate that rig + /// here): + /// 1. Form a real 2-node cluster (docker/ topology or Akka.Remote in-proc + /// multi-ActorSystem rig from PLAN-01). + /// 2. Confirm a cluster singleton (e.g. site DeploymentManager) is hosted + /// on node A; record T0. + /// 3. Hard-kill node A (process kill / ActorSystem.Abort — not + /// CoordinatedShutdown; graceful stop exercises a different path). + /// 4. Poll node B for singleton re-host; record T1 at first successful + /// response from the migrated singleton. + /// 5. Assert T1 - T0 <= 40s (25s design + margin) and report the number. + /// + public class FailoverTimingTests + { + [Trait("Category", "Performance")] + [Fact(Skip = "Requires the real two-node failover rig delivered by PLAN-01 " + + "(overall review P2-10). This class documents the measurement " + + "protocol; wire it to the rig when PLAN-01 lands.")] + public void HardKillFailover_SingletonRehostedWithinDesignEnvelope() + { + // Intentionally empty until the PLAN-01 rig exists. + } + } + ``` +2. Verify it compiles and shows as skipped: + ```bash + dotnet test tests/ZB.MOM.WW.ScadaBridge.PerformanceTests --filter "FullyQualifiedName~FailoverTiming" + ``` + Expected: 1 skipped, 0 failed. +3. Commit: + ```bash + git add tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Failover + git commit -m "test(perf): failover-timing harness placeholder documenting the 25s envelope protocol, pending PLAN-01 two-node rig (arch-review 08 §2.3)" + ``` + +--- + +### Task 9: Contract-lock tests for top ClusterClient message records + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 3, 4, 5, 6, 7, 8, 10, 11 +**Files:** +- Create: `tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Messages/ClusterClientContractLockTests.cs` +- Modify: `Directory.Packages.props` (add `` — first check `dotnet list tests/ZB.MOM.WW.ScadaBridge.Commons.Tests package --include-transitive | grep -i newtonsoft` and pin to the version Akka already pulls transitively, to avoid a version bump ride-along) +- Modify: `tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/ZB.MOM.WW.ScadaBridge.Commons.Tests.csproj` (add ``) + +Rationale (§1.8, Rec 10): ClusterClient traffic rides Akka.NET's default Newtonsoft-JSON serialization; "additive-only evolution" is currently discipline, not enforcement — the gRPC side has proto contract-lock tests (`Communication.Tests/Protos/*ProtoTests.cs`, `Grpc/ProtoContractTests.cs`) but the record contracts have none. These tests lock the version-skew guarantees: (a) an old-shape payload (missing newer trailing optional fields) still deserializes with defaults, (b) unknown fields from a *newer* peer are ignored, (c) round-trip preserves values. + +Records to lock (highest-traffic cross-cluster surface): `ManagementEnvelope` (`Commons/Messages/Management/ManagementEnvelope.cs`), `DeployInstanceCommand` (`Messages/Deployment/DeployInstanceCommand.cs` — has documented optional trailing fields `CentralFetchBaseUrl`/`FetchToken`), `GetAttributeRequest` + `SetStaticAttributeCommand` (`Messages/Instance/`), `AttributeValueChanged` (`Messages/Streaming/`), one notification-forward record from `Messages/Notification/NotificationMessages.cs`, and `CreateDataConnectionCommand` (`Messages/Management/DataConnectionCommands.cs` — the report's own example of defaulted trailing parameters). + +1. Write the failing test file (fails to compile until the package reference lands — that is the FAIL step): + ```csharp + using Newtonsoft.Json; + using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management; + using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming; + + namespace ZB.MOM.WW.ScadaBridge.Commons.Tests.Messages; + + /// + /// Contract-lock tests for the highest-traffic ClusterClient record contracts. + /// ClusterClient traffic uses Akka.NET's default Newtonsoft-JSON serializer and + /// the design supports system-wide artifact version skew across sites, so these + /// records must obey additive-only evolution: old-shape payloads deserialize + /// (missing new fields → defaults) and unknown fields are ignored. If a change + /// breaks one of these tests, it breaks rolling cross-cluster compatibility — + /// do not "fix the test"; make the change additive. + /// + public class ClusterClientContractLockTests + { + private static readonly JsonSerializerSettings AkkaLikeSettings = new() + { + MissingMemberHandling = MissingMemberHandling.Ignore, + }; + + [Fact] + public void CreateDataConnectionCommand_OldShape_WithoutNewerTrailingFields_Deserializes() + { + // Shape as emitted before BackupConfiguration/FailoverRetryCount existed. + // (Adjust property list to the record's actual required leading fields.) + const string oldShape = + """{"Name":"conn1","SiteId":3,"Protocol":"OpcUa","Configuration":"{}"}"""; + + var cmd = JsonConvert.DeserializeObject(oldShape, AkkaLikeSettings); + + Assert.NotNull(cmd); + Assert.Null(cmd!.BackupConfiguration); // default applied + Assert.Equal(3, cmd.FailoverRetryCount); // default applied + } + + [Fact] + public void AttributeValueChanged_UnknownFieldFromNewerPeer_IsIgnored() + { + const string newerShape = + """{"InstanceUniqueName":"i","AttributePath":"a","AttributeName":"a","Value":1,"Quality":"Good","Timestamp":"2026-07-08T00:00:00+00:00","SomeFutureField":"x"}"""; + + var evt = JsonConvert.DeserializeObject(newerShape, AkkaLikeSettings); + + Assert.NotNull(evt); + Assert.Equal("Good", evt!.Quality); + } + + [Fact] + public void ManagementEnvelope_RoundTrip_PreservesCorrelationId() + { + // Construct with the real AuthenticatedUser shape (read Commons/…/AuthenticatedUser). + var envelope = /* new ManagementEnvelope(user, command: new object(), "corr-1") */ default(ManagementEnvelope); + // serialize → deserialize → assert CorrelationId round-trips. + } + } + ``` + Flesh out one old-shape + one unknown-field + one round-trip case per listed record (read each record first; the literals above must match real constructor signatures — that is part of the task, and any mismatch found IS the drift this suite exists to surface). +2. Run, expect FAIL (missing Newtonsoft reference → compile error): + ```bash + dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter "FullyQualifiedName~ClusterClientContractLock" + ``` +3. Add the `PackageVersion` to `Directory.Packages.props` and the `PackageReference` to the Commons.Tests csproj; finish the test bodies. +4. Run, expect PASS: + ```bash + dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter "FullyQualifiedName~ClusterClientContractLock" + ``` +5. Commit: + ```bash + git add Directory.Packages.props tests/ZB.MOM.WW.ScadaBridge.Commons.Tests + git commit -m "test(commons): contract-lock tests for top ClusterClient records — enforce additive-only evolution under version skew (arch-review 08 §1.8)" + ``` + +--- + +### Task 10: Docs-code drift + conventions cleanup batch (Low findings, consolidated) + +**Classification:** small +**Estimated implement time:** ~5 min +**Parallelizable with:** 2, 3, 4, 5, 6, 7, 8, 9, 11 (NOT 1 — both edit CLAUDE.md) +**Files:** +- Modify: `docs/requirements/Component-Commons.md` (layout tree, lines ~293–294) +- Modify: `docs/requirements/Component-ConfigurationDatabase.md` (repositories section) +- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ZB.MOM.WW.ScadaBridge.ManagementService.csproj` (line 22) +- Modify: `README.md` (line ~111, the "One doc per component" claim) +- Modify: `docs/components/README.md` (index) +- Modify: `CLAUDE.md` (Document Conventions section) + +Concrete edits (each verified against current file state): + +1. **Component-Commons.md — add `Observability/`** (§1.1, narrowed: `Serialization/` and `Validators/` are already documented at lines 293–294; only `Observability/` — which contains `ScadaBridgeTelemetry.cs` — is missing). In the layout tree, before the `├── Serialization/` line add: + ``` + ├── Observability/ # ScadaBridgeTelemetry (ActivitySource/metrics names) + ``` + (Adjust the comment after reading `src/ZB.MOM.WW.ScadaBridge.Commons/Observability/ScadaBridgeTelemetry.cs` — describe what it actually holds.) +2. **Component-ConfigurationDatabase.md — document the site-side repository exception** (§1.3). In the section stating repository implementations live in Configuration Database, add: + > Site-side exception: read-only SQLite-backed implementations of shared repository interfaces live in `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/` (currently `SiteExternalSystemRepository`) — same Commons interface, site-local store, writes throw `NotSupportedException` because site artifacts are managed exclusively by deployment from Central. + + (If Task 6 has not run yet, mention `SiteNotificationRepository` too and let Task 6's doc step remove it; if running after Task 6, list only the external-system repository.) +3. **ManagementService.csproj line 22** — change `..\ZB.MOM.WW.ScadaBridge.InboundAPI\ZB.MOM.WW.ScadaBridge.InboundAPI.csproj` to `../ZB.MOM.WW.ScadaBridge.InboundAPI/ZB.MOM.WW.ScadaBridge.InboundAPI.csproj` (§1.4 cosmetic; every other reference in the repo uses `/`). +4. **README.md ~line 111** (§4): replace the claim "One doc per component (plus the shared TreeView)" with an accurate scope, e.g.: + > Reference docs exist for 24 of the 27 components (plus the shared TreeView); ScriptAnalysis (#25), KpiHistory (#26), and DelmiaNotifier (#27) are pending — tracked in `docs/plans/2026-07-08-deferred-work-register.md`. +5. **docs/components/README.md**: add the same three-pending note to the index so the two lists can't drift independently. +6. **CLAUDE.md Document Conventions** (§4 DelmiaNotifier): add one bullet: + > - Exception: DelmiaNotifier (#27) is an external client tool, not a cluster component; its spec is the project README (`src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md`) plus `docs/plans/2026-06-26-delmia-recipe-notifier-design.md` — there is intentionally no `Component-DelmiaNotifier.md`. +7. Verify (config/docs task — verification replaces the failing test): + ```bash + dotnet build src/ZB.MOM.WW.ScadaBridge.ManagementService # csproj separator change still builds + grep -n "Observability" docs/requirements/Component-Commons.md # expect: 1 hit in the tree + grep -rn '\\\\ZB.MOM' src --include="*.csproj" # expect: no hits + grep -n "24 of the 27\|pending" README.md docs/components/README.md # expect: scoped claim present + grep -n "Component-DelmiaNotifier" CLAUDE.md # expect: the exception bullet + ``` +8. Commit: + ```bash + git add docs/requirements/Component-Commons.md docs/requirements/Component-ConfigurationDatabase.md src/ZB.MOM.WW.ScadaBridge.ManagementService/ZB.MOM.WW.ScadaBridge.ManagementService.csproj README.md docs/components/README.md CLAUDE.md + git commit -m "docs: close arch-review 08 drift batch — Commons Observability folder, site-repo exception, csproj separator, docs/components scope, DelmiaNotifier convention exception" + ``` + +--- + +### Task 11: Deferred-work register — triage the 23-item inventory into a tracked list + +**Classification:** small +**Estimated implement time:** ~5 min +**Parallelizable with:** 1, 2, 3, 4, 5, 6, 7, 8, 9 (write after 10 only if you want the README pointer target to exist first — create this file before or simultaneously; the README in Task 10 points here) +**Files:** +- Create: `docs/plans/2026-07-08-deferred-work-register.md` + +Turn the review's §3 inventory (23 items) into the explicit tracked register the review asked for: every item gets a row with **status** (`fix-now → plan NN` / `deferred`), **owner/where-noted**, **rationale**, and a **revisit trigger** (the concrete condition that reopens it). This is the single place the "consciously logged" deferrals live from now on; new deferrals get appended here instead of scattering. + +1. Create `docs/plans/2026-07-08-deferred-work-register.md` with this exact structure and content (source: `archreview/08-conventions-tests-underdeveloped.md` §3; carry each item's file/line citation over): + + ```markdown + # Deferred-Work Register (established 2026-07-08, from architecture review 08) + + Single tracked list of consciously-deferred work. Rules: every deferral gets a row + (rationale + revisit trigger); fix-now items reference the archreview plan that owns + them and are removed from this table when that plan's task lands. + + ## Fix-now (owned by archreview plans) + | # | Item | Owner | + |---|------|-------| + | 1 | Transport: Area membership doesn't travel in bundles (EntitySerializer.cs:251,518) | PLAN-05 | + | 2 | CLI.Tests not in slnx | PLAN-08 Task 1 | + | 3 | AuditLog reconciliation dials site NodeA only (SiteEnumerator.cs:32,64) | PLAN-04 | + | 4 | Options validation missing on ~12 components | PLAN-08 Tasks 2–5 | + | 5 | SmsConfiguration.MaxRetryCount stored but not honored | PLAN-06 | + | 6 | Role-check case-sensitivity asymmetry (ManagementActor.cs:1910) | PLAN-07 | + | 23 | Vestigial site-side notification-list surface | PLAN-08 Task 6 | + + ## Deferred (with rationale + revisit trigger) + | # | Item | Where noted | Rationale for deferral | Revisit trigger | + |---|------|-------------|------------------------|-----------------| + | 7 | SecuredWrite audit rows leave SourceNode NULL | CLAUDE.md Security | Logged follow-up; low-volume channel | First per-node forensic query against secured writes | + | 8 | Hash-chain tamper evidence (T1); CLI verify-chain is a no-op stub | audit-log roadmap :12 | v1.x by locked decision; append-only DB roles are the control | Compliance requirement for cryptographic tamper evidence | + | 9 | Parquet audit archival (T2); endpoint returns 501 | AuditEndpoints.cs:204 | v1.x; 501 + CLI messaging are honest | AuditLog partition volume nears retention ceiling | + | 10 | Aggregated live alarm stream for Alarm Summary | m7 design :252 | Snapshot fan-out acceptable at current instance counts | Alarm Summary latency complaints or >~50 instances/site | + | 11 | Central-persisted OPC UA cert-trust audit | m7 follow-ups | Broadcast-to-both-nodes covers HA | Governance/audit requirement for trust decisions | + | 12 | Native-alarm-source-override CSV import | m7 follow-ups | Parity gap only; attribute CSV shipped | First bulk native-alarm rollout request | + | 13 | WaitForAttribute quality-gated mode | ScriptRuntimeContext.cs:405 | Planned enhancement per spec §4.2 | First script needing Good-only waits | + | 14 | WaitForAttribute in Test-Run sandbox | waitfor-deferred :222 | Test-Run parity gap; production unaffected | Author feedback on Test-Run fidelity | + | 15 | BrowseNext final-page signal not surfaced | BrowseCommands.cs:34 | One wasted round-trip | UX complaint on browse paging | + | 16 | StubOpcUaClient throws on browse | m7 design :245 | Limits offline UI coverage only | Next browse/search UI regression | + | 17 | Unified notifications+site-calls outbox page | stillpending :118 | Explicit M9 decision to keep two pages | Operator confusion reports | + | 18 | Folder drag-drop | same, [PERM] | Permanently closed; menu reorder shipped | — (closed) | + | 19 | Bundle signing / cluster-to-cluster pull / differential bundles | transport-design :402 | v1 manifest hash + AES-GCM held sufficient | Non-repudiation requirement across orgs | + | 20 | Deployment EXPIRED-row purge | DeploymentManagerRepository.cs:310 | Read path already compensates | EXPIRED rows visible in ops queries | + | 21 | SiteAuditBacklogReporter threshold consolidation | SiteAuditBacklogReporter.cs:28 | Config-shape cleanup only | Next SqliteAuditWriterOptions change | + | 22 | KPI history hourly rollups | Component-KpiHistory.md | YAGNI; 90-day retention bounds table | KpiSample query latency on dashboards | + + ## New deferrals from review 08 (this plan) + | Item | Rationale | Revisit trigger | + |------|-----------|-----------------| + | Communication → HealthMonitoring layering (ICentralHealthAggregator consumed by CentralCommunicationActor.cs:351) | Moving the interface + SiteHealthState to Commons ripples across 5 projects for a cosmetic inversion | Next breaking change to ICentralHealthAggregator | + | docs/components reference docs for ScriptAnalysis, KpiHistory, DelmiaNotifier | Reference docs are substantial (StyleGuide-conformant); README claim scoped instead (PLAN-08 Task 10) | Next doc-writing session touching those components | + | Test-coverage backfill: SiteCallAudit.Tests (31 tests/1.6k LOC), DeploymentManager.Tests | No defect identified; coverage partly lives in ManagementService/Host/Integration suites | First regression escaping either component | + | Failover-timing + broader perf envelope (S&F drain rate, per-subscriber backpressure) | Needs PLAN-01 two-node rig; placeholder harness shipped (PLAN-08 Task 8) | PLAN-01 rig landing | + ``` +2. Verify (docs task): + ```bash + grep -c "^| " docs/plans/2026-07-08-deferred-work-register.md + ``` + Expected: ≥ 30 table rows (7 fix-now + 16 deferred + 4 new + headers). +3. Commit: + ```bash + git add docs/plans/2026-07-08-deferred-work-register.md + git commit -m "docs(plans): deferred-work register — triage the 23-item arch-review inventory into fix-now (plan-owned) vs deferred-with-rationale" + ``` + +--- + +## Dependencies on other plans + +- **Plan 01 (Cluster/Host/Failover):** Task 8 is a placeholder that documents the failover-timing measurement protocol; the actual two-node kill rig is Plan 01's (overall review P2-10). When Plan 01's rig lands, un-skip `FailoverTimingTests` and wire it in. No file overlap expected, but Plan 01 may touch `Host/Program.cs` (Task 4 also edits it — coordinate merge order). +- **Plan 05 (Templates/Deploy/Transport):** owns the Transport `AreaName: null` contradiction (§3 item 1 / §4). Plan 05 also touches `Transport/`; Task 5 here only touches `Transport/ServiceCollectionExtensions.cs` + a new validator file — small merge-conflict surface. +- **Plan 04 (Data/Audit):** owns AuditLog NodeA-only reconciliation (§3 item 3). +- **Plan 06 (Edge integrations):** owns `SmsConfiguration.MaxRetryCount` (§3 item 5). +- **Plan 07 (UI/Management/Security):** owns role-casing asymmetry (§3 item 6). Plan 07 may also touch `ManagementActor.cs`; Task 4 here touches only `ManagementService/ServiceCollectionExtensions.cs` + csproj. +- The register (Task 11) cites plan numbers 04/05/06/07 — content-only references, no execution dependency. + +## Execution order + +1. **Task 1 first** (slnx) — it changes what `dotnet test ZB.MOM.WW.ScadaBridge.slnx` covers, so every later task's solution-level verification includes CLI.Tests. +2. **Tasks 2, 4, 5, 7, 8, 9, 11 in parallel** (disjoint files). +3. **Task 3 then Task 6 sequentially** (both live in the SiteRuntime project; either order works, but running 6 after 3 keeps the CompositionRoot test churn in one direction). +4. **Task 10 last among docs** (it edits CLAUDE.md — serialize with Task 1's CLAUDE.md edit — and its README pointer targets the Task 11 register, so run after 11). +5. Finish with a full `dotnet build ZB.MOM.WW.ScadaBridge.slnx && dotnet test ZB.MOM.WW.ScadaBridge.slnx` — expect all 30 test projects (now including CLI.Tests' ~279) green. diff --git a/archreview/plans/PLAN-08-conventions-tests.md.tasks.json b/archreview/plans/PLAN-08-conventions-tests.md.tasks.json new file mode 100644 index 00000000..78b9e9e2 --- /dev/null +++ b/archreview/plans/PLAN-08-conventions-tests.md.tasks.json @@ -0,0 +1,72 @@ +{ + "planPath": "archreview/plans/PLAN-08-conventions-tests.md", + "tasks": [ + { + "id": 1, + "subject": "Task 1: Add CLI.Tests to the slnx and retire the gotcha documentation", + "status": "pending", + "blockedBy": [] + }, + { + "id": 2, + "subject": "Task 2: Options validation — Communication + DataConnectionLayer", + "status": "pending", + "blockedBy": [] + }, + { + "id": 3, + "subject": "Task 3: Options validation — site pipeline (SiteRuntime, StoreAndForward, SiteEventLogging)", + "status": "pending", + "blockedBy": [] + }, + { + "id": 4, + "subject": "Task 4: Options validation — edge + management (InboundAPI, ExternalSystemGateway, NotificationService, ManagementService)", + "status": "pending", + "blockedBy": [] + }, + { + "id": 5, + "subject": "Task 5: Options validation — central components (Transport, SiteCallAudit, DeploymentManager)", + "status": "pending", + "blockedBy": [] + }, + { + "id": 6, + "subject": "Task 6: Remove the vestigial site-side notification-list surface", + "status": "pending", + "blockedBy": [3] + }, + { + "id": 7, + "subject": "Task 7: PerformanceTests — site-stream throughput test", + "status": "pending", + "blockedBy": [] + }, + { + "id": 8, + "subject": "Task 8: PerformanceTests — failover-timing benchmark harness placeholder", + "status": "pending", + "blockedBy": [] + }, + { + "id": 9, + "subject": "Task 9: Contract-lock tests for top ClusterClient message records", + "status": "pending", + "blockedBy": [] + }, + { + "id": 10, + "subject": "Task 10: Docs-code drift + conventions cleanup batch (Low findings, consolidated)", + "status": "pending", + "blockedBy": [1, 11] + }, + { + "id": 11, + "subject": "Task 11: Deferred-work register — triage the 23-item inventory into a tracked list", + "status": "pending", + "blockedBy": [] + } + ], + "lastUpdated": "2026-07-08" +}