docs(archreview): add architecture-review fix plans (P0 initiative baseline)
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,122 @@
|
||||
# ScadaBridge Deep Architecture Review — Overall Report
|
||||
|
||||
**Date:** 2026-07-08
|
||||
**Scope:** Full system — all 27 components, docker/deploy topology, tests, and design docs
|
||||
**Method:** 8 parallel domain reviews, each reading the component design docs first and then the actual source (findings are grounded in `file:line` citations in the domain reports). Dimensions: **stability, performance, conventions, underdeveloped areas.**
|
||||
|
||||
## Report Index
|
||||
|
||||
| # | Report | Domain | Verdict (one line) |
|
||||
|---|--------|--------|--------------------|
|
||||
| 01 | [Cluster, Host & Failover](01-cluster-host-failover.md) | ClusterInfrastructure, Host, HealthMonitoring, docker/Traefik/deploy | High micro-level polish undone by one critical foundation crack: the split-brain resolver is configured but never enabled — automatic failover does not work for hard crashes. |
|
||||
| 02 | [Communication & Store-and-Forward](02-communication-store-and-forward.md) | ClusterClient, gRPC streaming, S&F engine | High code quality in-the-small; mid maturity as a distributed system — serious lifecycle/topology defects the test suite is structurally blind to. |
|
||||
| 03 | [Site Runtime & DCL](03-site-runtime-dcl.md) | SiteRuntime, DataConnectionLayer, SiteEventLogging | Mature and unusually carefully engineered; residual risk in adapter reconnect lifecycle, script containment, and deployment honesty. |
|
||||
| 04 | [Data & Audit Backbone](04-data-audit-backbone.md) | ConfigurationDatabase, AuditLog, NotificationOutbox, SiteCallAudit, KpiHistory | Genuinely mature data layer with scale-dependent operational time bombs that will surface within months of production volume. |
|
||||
| 05 | [Templates, Deployment & Transport](05-templates-deployment-transport.md) | TemplateEngine, DeploymentManager, Transport, ScriptAnalysis | Failure-path engineering is production-grade, but the Transport import apply path and newest template features contain multiple silent-data-loss and staleness defects. |
|
||||
| 06 | [Edge Integrations](06-edge-integrations.md) | InboundAPI, ExternalSystemGateway, NotificationService, DelmiaNotifier | Mature, heavily-hardened edge layer; risk concentrated in one silent cache-coherence bug and acknowledged ESG spec gaps. |
|
||||
| 07 | [UI, Management & Security](07-ui-management-security.md) | CentralUI, ManagementService, CLI, Security | Mature, carefully engineered surface undermined by sharp asymmetries: an auth-disabled deploy artifact, ignored deploy scoping, inconsistent secret elision. |
|
||||
| 08 | [Conventions, Tests & Underdeveloped Areas](08-conventions-tests-underdeveloped.md) | Cross-cutting | Unusually disciplined codebase — the documented conventions are actually followed; gaps are narrow, specific, and mostly consciously logged. |
|
||||
|
||||
## Severity Tally (tag occurrences per report)
|
||||
|
||||
| Report | Critical | High | Medium | Low |
|
||||
|--------|---------:|-----:|-------:|----:|
|
||||
| 01 Cluster/Host/Failover | 1 | 5 | 13 | 9 |
|
||||
| 02 Communication/S&F | 2 | 6 | 11 | 11 |
|
||||
| 03 Site Runtime/DCL | 0 | 4 | 6 | 11 |
|
||||
| 04 Data/Audit Backbone | 0 | 6 | 11 | 6 |
|
||||
| 05 Templates/Deploy/Transport | 2 | 13 | 15 | 5 |
|
||||
| 06 Edge Integrations | 0 | 4 | 11 | 11 |
|
||||
| 07 UI/Management/Security | 2 | 8 | 12 | 5 |
|
||||
| 08 Conventions/Tests | 0 | 4 | 6 | 10 |
|
||||
| **Total (approx.)** | **7** | **50** | **85** | **68** |
|
||||
|
||||
*(Counts are raw tag occurrences; a few findings are cited in more than one report — e.g. the `DisableLogin` deploy artifact appears in both 01 and 07.)*
|
||||
|
||||
## Overall Verdict
|
||||
|
||||
ScadaBridge is an unusually disciplined codebase at the micro level. The conventions documented in CLAUDE.md are **actually followed** — verified, not assumed: zero `DateTime.Now` in `src/`, persistence-ignorant Commons entities, a clean project-reference graph with no site→central contamination, correlation IDs on request/response messages, additive message contracts, idempotent ingest keyed on `EventId`/`NotificationId`, timing-oracle-free API-key auth, and only two real TODOs in ~190k LOC. Test volume is real (~5,050 behavioral tests across 30 projects — Akka TestKit, bUnit, Playwright, MSSQL-backed integration tests).
|
||||
|
||||
The system's risk is **not** diffuse sloppiness. It concentrates in exactly the four places a single-process test suite cannot see:
|
||||
|
||||
1. **Real multi-node failover** — the paths that only execute when a node hard-crashes, a partition forms, or the standby takes over.
|
||||
2. **Cache coherence across process/node boundaries** — compiled-script and staleness caches that are only invalidated on the node that made the change.
|
||||
3. **Cross-environment data movement** — the Transport bundle import apply path, where DTOs, spec, and apply code drifted apart.
|
||||
4. **Production-scale data volume** — purge/retention/KPI paths tested only at small scale.
|
||||
|
||||
**Bottom line: production-grade engineering, not yet production-proven HA.** The single most important sentence in this review: *automatic failover for hard crashes does not currently work* (finding C1 below), and no test in the repo would have caught it.
|
||||
|
||||
## The Critical Findings (fix before anything else)
|
||||
|
||||
**C1. The split-brain resolver is configured but never enabled.** `BuildHocon` never sets `akka.cluster.downing-provider-class`, so the keep-oldest SBR settings are inert: unreachable nodes are never downed, and cluster singletons (site DeploymentManager = all data collection; all six central singletons) never migrate on a hard crash or partition. The documented ~25s failover only works for graceful stops. The repo's own Akka reference notes (`AkkaDotNet/03-Cluster.md`) show the required line. *(Report 01; `Host/Actors/AkkaHostedService.cs:225-270`)*
|
||||
|
||||
**C2. Editing a site's addresses crash-loops central routing to all sites.** `CentralCommunicationActor` stops and immediately recreates a *named top-level* ClusterClient in the same message handling; `Context.Stop` is async, so the name collides and the actor crash-restarts into a permanent loop. Tests mock the client factory, so this is never exercised. *(Report 02; `CentralCommunicationActor.cs:514-521`)*
|
||||
|
||||
**C3. Transport bundle import silently drops template inheritance.** `TemplateDto.BaseTemplateName` is exported and diffed but never consumed at apply — derived templates land as root templates and flatten to amputated configs in the target environment. *(Report 05; `Transport/Import/BundleImporter.cs:1524`)*
|
||||
|
||||
**C4. The on-host production deploy artifact ships `DisableLogin: true`.** Every surface (UI, `/management`, audit REST, debug hub) auto-authenticates with ALL roles — including Operator+Verifier, nullifying the two-person secured-write control. *(Reports 01 & 07; `deploy/wonder-app-vd03/appsettings.Central.json:39-42`)*
|
||||
|
||||
## Cross-Cutting Themes
|
||||
|
||||
### Theme 1 — Failover is designed but unverified, and multiple independent defects hide on that path (stability)
|
||||
|
||||
Beyond C1/C2, the failover path accumulates: leader-vs-oldest confusion ("active node" = cluster leader by address order, but singletons live on the *oldest* node — the event-log purge gate runs on the wrong node and Traefik can route to a node hosting no singletons); during a central partition **both** nodes return 200 on `/health/active` so Traefik serves both indefinitely; the standby site node runs the **full S&F delivery pipeline** with no active-node gate, so replicated Pending rows are delivered by both nodes (systematic duplicate external calls and DB writes); singleton drain tasks run `GracefulStop(10s)` inside a 5s CoordinatedShutdown phase; and inbound-API compiled handlers go stale across central failover (the standby serves startup-time delegates with no revision check — an updated method silently serves the old script with HTTP 200).
|
||||
|
||||
**Root cause is shared:** no test anywhere forms a real two-node cluster and kills a node. "Failover" integration tests are single-process simulations; HOCON tests assert strings, not behavior; the risky collaborators (real ClusterClients, the standby node, shared gRPC channels) are mocked away. *(Reports 01, 02, 06)*
|
||||
|
||||
### Theme 2 — A family of cache-coherence/staleness bugs around compiled scripts (stability)
|
||||
|
||||
The same defect shape appears at least four times: (a) inbound compiled handlers stale across failover; (b) inbound compiled-handler and known-bad-method caches stale after a bundle import overwrites an `ApiMethod` (old code serves HTTP 200 until restart); (c) `_knownBadMethods` never cleared on out-of-band fixes (known, in project memory); (d) `RevisionHashService` omits `NativeAlarmSources`, so native-alarm edits never flag instances stale and sites mirror old sources indefinitely. Related: every staleness probe / Deployments-page validation compiles every script via Roslyn into the non-collectible AssemblyLoadContext with no caching — CPU spikes plus an unbounded assembly leak. **A single "script artifact changed → invalidate everywhere" invalidation contract would close the whole family.** *(Reports 05, 06)*
|
||||
|
||||
### Theme 3 — Transport import is the least-finished shipped feature: silent data loss (underdeveloped)
|
||||
|
||||
Beyond C3: imported scripts silently lose `MinTimeBetweenRuns`/`ExecutionTimeoutSeconds` (and `LockedInDerived` isn't in the DTOs at all); template `NativeAlarmSources` aren't transported while their instance-level overrides *are* (dangling overrides silently dropped at flatten); the exporter hardcodes `AreaName: null` while docs and CLAUDE.md claim Area-by-name travels (the importer's resolution machinery is fully built — a half-shipped feature presented as done); the import blocker heuristic hard-blocks valid imports on PascalCase false positives with no operator override. The bundle **crypto** is solid; it's the **apply path** where spec, DTOs, and code drifted. *(Reports 05, 08)*
|
||||
|
||||
### Theme 4 — Scale-dependent time bombs in the data layer (performance/stability)
|
||||
|
||||
Each is invisible at test scale and arrives on a clock: the spec'd site SQLite 7-day audit purge **does not exist** (site audit DBs grow unbounded); `SwitchOutPartitionAsync` rebuilds the full-table `UX_AuditLog_EventId` unique index in one transaction with the default 30s `CommandTimeout` — once the table is large, the daily purge fails permanently and retention silently stops; SiteCalls KPI predicates full-scan the 365-day table ~15×/minute; `GetExecutionTreeAsync` does a `SELECT DISTINCT` over the entire AuditLog per drilldown; the S&F retry sweep is serial, unbounded, and single-laned (central down + 200-message backlog ≈ 100 min/sweep with head-of-line blocking); site SQLite runs without WAL/busy-timeout under many concurrent writers; no EF `EnableRetryOnFailure` anywhere; the persisted `backlogTotal` KPI trend is hardcoded zero. *(Reports 02, 03, 04)*
|
||||
|
||||
### Theme 5 — Security posture is strong in design, leaky in projection (stability/conventions)
|
||||
|
||||
The session/JWT design, secured-write CAS race guards, and timing-oracle-free API-key auth are exemplary. But: C4 (`DisableLogin` artifact); DataConnection configs (plaintext OPC UA passwords), ExternalSystem `AuthConfiguration`, and DB connection strings are returned **raw to any authenticated user** via ungated List/Get commands — while SMTP/SMS/API keys are properly elided; `deploy artifacts --site-id` is silently ignored and deploys fleet-wide with no site-scope check; pending secured writes never expire server-side (the "Expired" status exists only in the UI — a stale setpoint can be approved and written to a live device days later); every management/auth request does a full LDAP bind with zero throttling (password-spray oracle); role-casing split-brain between UI policies (case-sensitive) and the ManagementActor (case-insensitive). *(Report 07)*
|
||||
|
||||
### Theme 6 — Script containment is cooperative, not enforced (stability)
|
||||
|
||||
Script execution timeout is cooperative-only: a script blocked in synchronous I/O permanently consumes one of the 8 dedicated `ScriptExecutionScheduler` threads — 8 hung scripts silently halt all script **and alarm on-trigger** execution site-wide, with zero observability (no stuck-thread health metric). Relatedly: trigger expressions run Roslyn synchronously on the default dispatcher per attribute change; the inbound method timeout abandons the orphaned task while disposing its DI scope (use-after-dispose); site-side compile failures don't reject the deployment — central is still told Success, contradicting the spec. *(Reports 03, 06)*
|
||||
|
||||
### Theme 7 — Adapter/channel lifecycle leaks on reconnect (stability)
|
||||
|
||||
`OpcUaDataConnection`/`MxGatewayDataConnection` overwrite `_client` on every `ConnectAsync` without dispose/detach — session/socket leak per flap, plus a stale `ConnectionLost` handler that can flap the *new* healthy connection. One debug session's gRPC failover disposes the site's **shared** cached channel, cancelling every other session's subscriptions (concurrent sessions ping-pong and terminate each other). S&F replication is `Task.Run` fire-and-forget with possible Add/Remove reordering. *(Reports 02, 03)*
|
||||
|
||||
## What's Genuinely Good (keep doing this)
|
||||
|
||||
- **Conventions hold under audit** — UTC everywhere, POCO/repository split, options pattern (where validated), additive contracts, clean reference graph, correlation IDs. (Report 08)
|
||||
- **Idempotency and atomicity where it matters** — EventId-keyed audit ingest, NotificationId insert-if-not-exists, verified-atomic AuditLog+SiteCalls dual-write, CAS on deployment status and secured-write approval. (Reports 04, 07)
|
||||
- **Error hygiene at the edges** — consistent transient/permanent classification, poison-message parking, enumeration-safe 403s, `IHttpClientFactory` throughout, bounded audit capture. (Report 06)
|
||||
- **Hardening passes are visible** — Become/Stash lifecycle machines, redeploy termination-watch buffering, adapter-generation guards, HOCON escaping, DebugView disposal/thread-marshalling. (Reports 01, 03, 07)
|
||||
- **Deferred work is honestly logged** — 23-item inventory in report 08; only ~4 warrant near-term action.
|
||||
|
||||
## Prioritized Recommendations
|
||||
|
||||
**P0 — before the next production deployment**
|
||||
1. Enable the SBR downing provider in `BuildHocon` and prove it with a real two-node kill test (see P2-10). *(C1)*
|
||||
2. Remove `DisableLogin: true` from `deploy/wonder-app-vd03` (or add an environment guard that refuses it outside dev). *(C4)*
|
||||
3. Fix the ClusterClient stop/recreate name collision (await termination or use generation-suffixed names). *(C2)*
|
||||
4. Gate the S&F delivery pipeline (and event-log purge gate) on singleton-host/active-node status — standby must be passive. *(Theme 1)*
|
||||
|
||||
**P1 — correctness of shipped features**
|
||||
5. Fix the Transport import apply path: consume `BaseTemplateName`, carry the dropped script fields + `LockedInDerived`, transport template `NativeAlarmSources`, wire `AreaName`. Add round-trip export→import equivalence tests per entity type. *(C3, Theme 3)*
|
||||
6. Define one script-artifact invalidation contract: bundle import, failover activation, and out-of-band updates must all invalidate compiled-handler/known-bad caches; add `NativeAlarmSources` to `RevisionHashService`; cache validation compiles (collectible ALC). *(Theme 2)*
|
||||
7. Close the secrets-projection gap: elide DataConnection/ExternalSystem/DB-connection secrets in List/Get like SMTP/SMS already are; honor `SiteId` in `HandleDeployArtifacts` with a site-scope check; add server-side secured-write expiry. *(Theme 5)*
|
||||
8. Make site-side compile failure reject the deployment (per spec) instead of reporting Success. *(Theme 6)*
|
||||
|
||||
**P2 — before production data volume / as hardening**
|
||||
9. Data-layer time bombs: implement the site SQLite audit purge; set an explicit long `CommandTimeout` on partition switch; add the missing SiteCalls KPI index; bound the execution-tree CTE; turn on WAL + busy_timeout for site SQLite; `EnableRetryOnFailure`; fix the hardcoded-zero backlog KPI. *(Theme 4)*
|
||||
10. Build a real two-node failover test rig (docker already provides the topology): kill -9 a node, assert singleton migration, S&F non-duplication, Traefik single-active, inbound handler freshness. This one rig would have caught C1, C2, and three Theme-1 findings. *(Theme 1)*
|
||||
11. Script containment: watchdog metric for stuck `ScriptExecutionScheduler` threads (health-surface it), move trigger-expression evaluation off the default dispatcher, fix the inbound DI-scope use-after-dispose. *(Theme 6)*
|
||||
12. Adapter lifecycle: dispose/detach old OPC UA/MxGateway clients on reconnect; per-session (or ref-counted) gRPC channel handling for debug streams; make S&F replication ordered or reconciled. *(Theme 7)*
|
||||
13. Housekeeping: add CLI.Tests to the slnx (one line, long overdue); extend `ValidateOnStart` options validation to the ~14 unvalidated components; batch/parallelize the S&F retry sweep with a per-sweep LIMIT; fix the docs drift catalogued in reports 04 and 08.
|
||||
|
||||
## Reading Order
|
||||
|
||||
If you read only three reports: **01** (the failover crack), **05** (Transport data loss), **04** (the data-layer clocks). Report **08** is the best single picture of overall codebase health and carries the consolidated 23-item deferred-work inventory.
|
||||
@@ -0,0 +1,391 @@
|
||||
# Architecture Review 01 — Cluster Infrastructure, Host, Failover, Health Monitoring, Deployment Topology
|
||||
|
||||
Reviewer domain: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure`, `src/ZB.MOM.WW.ScadaBridge.Host`,
|
||||
`src/ZB.MOM.WW.ScadaBridge.HealthMonitoring`, `docker/`, `docker-env2/`, `deploy/wonder-app-vd03/`,
|
||||
Traefik config, and the corresponding design docs (`Component-ClusterInfrastructure.md`,
|
||||
`Component-Host.md`, `Component-HealthMonitoring.md`, `Component-TraefikProxy.md`) and test projects.
|
||||
|
||||
## Scope
|
||||
|
||||
Read in full: `AkkaHostedService.cs` (1,203 lines — HOCON builder, singleton wiring, coordinated
|
||||
shutdown), `Program.cs` (both role branches), `StartupValidator.cs`, `StartupRetry.cs`,
|
||||
`SiteServiceRegistration.cs`, `NodeOptions.cs`, `ClusterOptions.cs` + validator +
|
||||
`ServiceCollectionExtensions.cs`, all Host `Health/` checks (`RequiredSingletonsHealthCheck`,
|
||||
`ActiveNodeGate`, `AkkaClusterNodeProvider`), `DeadLetterMonitorActor.cs`,
|
||||
`AkkaHealthReportTransport.cs`, the entire HealthMonitoring project (`HealthReportSender`,
|
||||
`CentralHealthAggregator`, `CentralHealthReportLoop`, `SiteHealthCollector`,
|
||||
`SiteEventLogFailureCountReporter`, `SiteHealthKpiSampleSource`, options + validator), the docker
|
||||
compose topologies + per-node appsettings + Traefik static/dynamic config + Dockerfile, the
|
||||
on-host deploy artifacts (`deploy/wonder-app-vd03/appsettings.*.json`), the four design docs, the
|
||||
repo's own Akka.NET reference notes (`AkkaDotNet/03-Cluster.md`), and the relevant test projects
|
||||
(`Host.Tests`, `ClusterInfrastructure.Tests`, `HealthMonitoring.Tests`, `IntegrationTests`
|
||||
failover/readiness suites). Health-report ingestion on the central side was cross-checked in
|
||||
`Communication/Actors/CentralCommunicationActor.cs:349-392`.
|
||||
|
||||
## Maturity Verdict
|
||||
|
||||
This slice of the codebase is unusually well-commented, defensively coded at the micro level
|
||||
(options validators with `ValidateOnStart`, CAS loops in the aggregator, sequence-number seeding
|
||||
for failover, HOCON escaping, bounded startup retry), and clearly the product of many careful
|
||||
review passes — but it has **one macro-level defect that undoes the entire failover design: the
|
||||
split-brain resolver is configured but never enabled** (no `akka.cluster.downing-provider-class`),
|
||||
so unreachable nodes are never automatically downed and cluster singletons never migrate on a hard
|
||||
crash. Every document, comment, and validator in the domain reasons meticulously about keep-oldest
|
||||
semantics that are inert at runtime, and the test suite asserts the HOCON *strings* rather than the
|
||||
*behavior*, which is exactly how the gap survived. Secondary structural issues cluster around the
|
||||
ambiguous definition of "active node" (cluster *leader* vs. singleton-hosting *oldest* member,
|
||||
which diverge after restarts) and dev-vs-prod configuration drift. Verdict: **high polish, one
|
||||
critical foundation crack; not production-ready for unattended failover until the downing provider
|
||||
is fixed and verified with a real two-node test.**
|
||||
|
||||
---
|
||||
|
||||
## 1. Stability
|
||||
|
||||
### [Critical] Split-brain resolver is never enabled — automatic failover does not work for crashes/partitions
|
||||
|
||||
`AkkaHostedService.BuildHocon` (src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs:225-270)
|
||||
emits:
|
||||
|
||||
```hocon
|
||||
cluster {
|
||||
...
|
||||
split-brain-resolver {
|
||||
active-strategy = "keep-oldest"
|
||||
stable-after = 15000ms
|
||||
keep-oldest { down-if-alone = on }
|
||||
}
|
||||
failure-detector { ... }
|
||||
run-coordinated-shutdown-when-down = on
|
||||
}
|
||||
```
|
||||
|
||||
but never sets `akka.cluster.downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider,
|
||||
Akka.Cluster"`. In Akka.NET (1.5.62 per `Directory.Packages.props:8-17`), the default downing
|
||||
provider is **NoDowning**; the whole `split-brain-resolver` section is only read by
|
||||
`SplitBrainResolverProvider`, which must be opted into via `downing-provider-class`. A repo-wide
|
||||
grep confirms the string `SplitBrainResolverProvider` appears **nowhere in `src/`** — only in the
|
||||
project's own reference notes, which show the required line
|
||||
(AkkaDotNet/03-Cluster.md:56, also 15-Coordination.md:69 and 18-MultiNodeTestRunner.md:100).
|
||||
|
||||
Failure scenario: the active site node loses power. The standby's failure detector marks it
|
||||
unreachable (~10s), but **nothing ever downs it**. The unreachable member stays in the membership
|
||||
forever, the `ClusterSingletonManager` cannot hand over (singleton migration requires the oldest
|
||||
member's *removal*, not mere unreachability), and the site's DeploymentManager singleton — and with
|
||||
it all data collection, script execution, and S&F delivery — never restarts on the survivor. The
|
||||
same applies to all six central singletons (notification-outbox, audit-log-ingest, site-call-audit,
|
||||
audit-log-purge, site-audit-reconciliation, kpi-history-recorder). The documented "~25 seconds
|
||||
total failover" (docs/requirements/Component-ClusterInfrastructure.md:110-119) is only achievable
|
||||
today for a **graceful** stop (CoordinatedShutdown leave); crash and partition scenarios hang
|
||||
indefinitely until an operator downs the node or restarts the survivor.
|
||||
|
||||
Corroborating evidence that this was never exercised:
|
||||
|
||||
- `HoconBuilderTests.cs:45,112,155` assert only that the SBR *strings* appear in the config.
|
||||
- `CoordinatedShutdownTests.cs:11-43` are source-text greps (`Assert.Contains("run-by-clr-shutdown-hook = on", content)`).
|
||||
- The "failover" integration tests are single-process simulations — e.g.
|
||||
`CentralFailoverTests.cs:27-47` validates a JWT across two `JwtTokenService` instances; no test
|
||||
forms a two-node cluster and kills the oldest node.
|
||||
- `run-coordinated-shutdown-when-down = on` (AkkaHostedService.cs:265) is dead config: a node is
|
||||
never transitioned to Down by anything.
|
||||
- `ClusterOptionsValidator` (src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptionsValidator.cs:35-63)
|
||||
rigorously rejects any strategy other than `keep-oldest` and demands `DownIfAlone == true` —
|
||||
validating knobs on a resolver that is not installed.
|
||||
|
||||
Fix: add the `downing-provider-class` line to `BuildHocon` (one line), or better, migrate the
|
||||
bootstrap to `Akka.Cluster.Hosting`'s typed options (already referenced,
|
||||
ZB.MOM.WW.ScadaBridge.Host.csproj:15) whose `SplitBrainResolverOption` wires the provider
|
||||
automatically — and add a real two-node in-process cluster test that kills the oldest member and
|
||||
asserts the singleton re-hosts within the expected window.
|
||||
|
||||
### [High] "Active node" means two different things — cluster leader vs. oldest member — and they diverge
|
||||
|
||||
Two distinct node-selection concepts are used interchangeably:
|
||||
|
||||
- **Leader-based** (address ordering, *not* age): `ActiveNodeGate.IsActiveNode`
|
||||
(src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeGate.cs:36-52),
|
||||
`AkkaClusterNodeProvider.SelfIsPrimary` and the "Primary"/"Standby" labels
|
||||
(src/ZB.MOM.WW.ScadaBridge.Host/Health/AkkaClusterNodeProvider.cs:29-40,61), the `active-node`
|
||||
health check Traefik routes on (Program.cs:231-234, docker/traefik/dynamic.yml:12-18), the
|
||||
`CentralHealthReportLoop` leader gate (CentralHealthReportLoop.cs:93-97), and the site
|
||||
event-log purge gate (`SiteEventLogActiveNodeCheck` → `SelfIsPrimary`,
|
||||
SiteServiceRegistration.cs:116-120).
|
||||
- **Oldest-based** (singleton placement): every `ClusterSingletonManager` (AkkaHostedService.cs:410+,
|
||||
906-925), and the site health "IsActiveNode" flag, which is set by the DeploymentManager
|
||||
*singleton* on start/stop (src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs:243,258).
|
||||
|
||||
In Akka the leader is the lowest-address `Up` member; the singleton host is the *oldest* member.
|
||||
They coincide at first boot but diverge permanently after the original first node is restarted and
|
||||
rejoins (it regains leadership by address order while the peer remains oldest). Concrete
|
||||
consequences once diverged:
|
||||
|
||||
1. **Site event-log retention is unenforced on the node that writes events.** Events are recorded
|
||||
by the singleton hierarchy on the oldest node; the purge (`EventLogPurgeService.cs:21,36`)
|
||||
early-exits unless `SelfIsPrimary` (leader). The active/oldest node's SQLite grows past the
|
||||
30-day/1GB cap while the leader purges an idle database.
|
||||
2. **Health dashboard labels the wrong node "Primary"** (AkkaClusterNodeProvider.cs:61): the node
|
||||
named Primary hosts no singletons; the node doing all the work shows Standby.
|
||||
3. **Central self-report is emitted by the leader** (CentralHealthReportLoop.cs:93) while the
|
||||
NotificationOutbox/audit singletons run on the other node — the `$central` health card describes
|
||||
the wrong node.
|
||||
4. **Traefik routes UI/API to the leader** while every singleton `Ask` then hops to the peer —
|
||||
functionally correct via the proxies, but it contradicts the design docs' "one active node runs
|
||||
all central components" model (Component-ClusterInfrastructure.md:44-47) and doubles
|
||||
intra-cluster latency on every outbox/audit page load.
|
||||
|
||||
Recommendation: pick one definition. Simplest coherent choice: derive "active" from singleton-host
|
||||
status (e.g. compare self against the oldest `Up` member, or gate on reachability of a local
|
||||
singleton), and use it consistently for `ActiveNodeGate`, `SelfIsPrimary`, the purge gate, and the
|
||||
`/health/active` check — leadership is an Akka-internal concept that only accidentally matches the
|
||||
product's active/standby model.
|
||||
|
||||
### [High] During a central network partition, both nodes report `/health/active = 200` and Traefik serves both
|
||||
|
||||
`ActiveNodeGate`/the active-node check compute `cluster.State.Leader` from each node's own gossip
|
||||
view. When central-a and central-b cannot see each other but Traefik can reach both (a plausible
|
||||
partial partition), each side sees the peer as unreachable and computes **itself** as leader — both
|
||||
return 200 on `/health/active`, and Traefik's load balancer (docker/traefik/dynamic.yml:10-18)
|
||||
round-robins requests between two nodes that each believe they are active. Blazor circuits and the
|
||||
Inbound API's active-node 503 gate (Program.cs:256-260) are then split across both. Because the SBR
|
||||
is not enabled (Critical finding above), nothing ever resolves the partition — this state persists
|
||||
until manual intervention. Singletons do *not* duplicate (that requires member removal), but every
|
||||
leader-gated behavior does. Enabling keep-oldest + `down-if-alone` fixes the persistence; consider
|
||||
also making `/health/active` require `Ready` (DB reachable) in addition to leadership so a
|
||||
DB-dead leader is pulled from rotation.
|
||||
|
||||
### [Medium] Cluster-leave drain tasks are budgeted 10s inside a 5s CoordinatedShutdown phase
|
||||
|
||||
Five singletons add a `PhaseClusterLeave` task that runs `GracefulStop(TimeSpan.FromSeconds(10))`
|
||||
(AkkaHostedService.cs:551-567 site-call-audit, 624-641 audit-log-purge, 676-693
|
||||
site-audit-reconciliation, 733-750 kpi-history-recorder, 787-804 pending-deployment-purge). Akka's
|
||||
`cluster-leave` phase uses the default phase timeout of **5 seconds**; the phase abandons the task
|
||||
at 5s (default `recover = on`) and proceeds, so the 10s graceful-stop can never complete its full
|
||||
window and the `catch` logging in those tasks never observes the overrun (the task is orphaned, not
|
||||
faulted). The intended "let the in-flight EF upsert drain before handover" guarantee is silently
|
||||
halved. Fix: set `akka.coordinated-shutdown.phases.cluster-leave.timeout` in `BuildHocon` to
|
||||
exceed the graceful-stop budget, or shrink the `GracefulStop` timeout below 5s.
|
||||
|
||||
### [Medium] The two busiest singletons have no drain task at all
|
||||
|
||||
The `NotificationOutboxActor` and `AuditLogIngestActor` singletons are terminated by bare
|
||||
`PoisonPill` with no cluster-leave drain (AkkaHostedService.cs:410-419, 451-458); the comment at
|
||||
AkkaHostedService.cs:544-549 explicitly notes the pattern "is suitable for the NotificationOutbox
|
||||
singleton; not added here to keep this change minimal". These are precisely the actors with
|
||||
in-flight EF transactions on every message (dispatch sweeps, dual-write ingest). The idempotent
|
||||
DB design (insert-if-not-exists on `NotificationId`/`EventId`) makes this safe-but-lossy rather
|
||||
than corrupting, but the inconsistency (drain for the low-traffic maintenance singletons, none for
|
||||
the hot ones) should be resolved.
|
||||
|
||||
### [Medium] Health-report loss recovery is dead code on the production transport
|
||||
|
||||
`HealthReportSender` carefully restores interval counters if `_transport.Send` throws
|
||||
(HealthReportSender.cs:158-175), and `SiteHealthCollector.AddIntervalCounters`
|
||||
(SiteHealthCollector.cs:153-171) exists to support it. But the only production transport is
|
||||
`AkkaHealthReportTransport.Send` (src/ZB.MOM.WW.ScadaBridge.Host/AkkaHealthReportTransport.cs:26-33)
|
||||
— a fire-and-forget `Tell` to an `ActorSelection`, which **never throws** and silently returns when
|
||||
the actor system is null. Any downstream loss (central unreachable, ClusterClient reconnecting,
|
||||
dead-lettered selection) drops the report *and* the interval error counters it carried — the
|
||||
scenario the restore logic was built for is exactly the one it can't see. Either make the transport
|
||||
Ask-with-ack (the site→central notification forward already uses this pattern over the same actor)
|
||||
or accept the loss and delete the restore machinery; the current shape gives false confidence.
|
||||
|
||||
### [Medium] Offline detection deviates from the spec and can mask a dead metrics pipeline
|
||||
|
||||
`Component-HealthMonitoring.md:43-46` defines offline as "no *report* within 60s".
|
||||
`CentralHealthAggregator.CheckForOfflineSites` (CentralHealthAggregator.cs:224-263) instead keys
|
||||
off `LastHeartbeatAt`, which is also advanced by the ~5s transport heartbeats
|
||||
(`CentralCommunicationActor.HandleHeartbeat` → `MarkHeartbeat`, CentralCommunicationActor.cs:349-353).
|
||||
A site whose `HealthReportSender` loop has died (or whose DeploymentManager singleton is stuck, so
|
||||
`IsActiveNode` is false on both nodes and reports are skipped — HealthReportSender.cs:85) keeps
|
||||
heartbeating and shows **online with silently frozen metrics forever**; nothing surfaces
|
||||
`LastReportReceivedAt` staleness. The heartbeat-based liveness choice is defensible (the code
|
||||
comment argues it), but then the dashboard needs a distinct "metrics stale" signal, and the design
|
||||
doc should be updated — currently spec and code disagree.
|
||||
|
||||
### [Medium] Second seed node in the checked-in site dev config targets the metrics port; validator has a gap
|
||||
|
||||
`src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json:14-17` seeds
|
||||
`akka.tcp://scadabridge@localhost:8082` and `akka.tcp://scadabridge@localhost:8084`, while the same
|
||||
file sets `MetricsPort: 8084` (line 10). The second seed points at the Kestrel HTTP/1.1 metrics
|
||||
listener — a node attempting to join via that seed performs a doomed Akka.Remote association
|
||||
against Kestrel (handshake noise, retry loops). `StartupValidator` explicitly guards seed-vs-GrpcPort
|
||||
(StartupValidator.cs:121-127) but not seed-vs-MetricsPort, so this config validates cleanly. Fix
|
||||
the dev config (8084 → the intended second node's remoting port) and extend the seed-port loop to
|
||||
also reject `MetricsPort`.
|
||||
|
||||
### [Medium] Docker stop grace period is shorter than a full CoordinatedShutdown
|
||||
|
||||
`docker/docker-compose.yml` sets no `stop_grace_period` (default 10s SIGTERM→SIGKILL) and the
|
||||
Dockerfile runs `dotnet` as PID 1 (docker/Dockerfile, final stage). A graceful shutdown runs
|
||||
cluster-leave (5s budget) + cluster-exiting (10s) + actor-system-terminate plus Serilog flush —
|
||||
easily exceeding 10s on a loaded node, so `bash docker/deploy.sh` recreates can SIGKILL mid-drain,
|
||||
turning every "graceful" redeploy into the crash path (which, per the Critical finding, currently
|
||||
has no automatic recovery on the surviving node). Add `stop_grace_period: 30s` to the app services.
|
||||
|
||||
### [Low] Aggregator never forgets sites
|
||||
|
||||
`CentralHealthAggregator._siteStates` (CentralHealthAggregator.cs:16) only grows: a site deleted
|
||||
from configuration remains a permanently-offline dashboard tile (and continues to be enumerated by
|
||||
`SiteHealthKpiSampleSource.CollectAsync`, SiteHealthKpiSampleSource.cs:74-85) until process
|
||||
restart. Needs an eviction path wired to site deletion.
|
||||
|
||||
### [Low] Dead-letter monitor: unbounded warning volume and restart-reset count
|
||||
|
||||
`DeadLetterMonitorActor` logs one Warning per dead letter with no sampling/rate-limiting and keeps
|
||||
its count in actor state that resets on restart
|
||||
(src/ZB.MOM.WW.ScadaBridge.Host/Actors/DeadLetterMonitorActor.cs:14,26-35). A dead-letter storm
|
||||
(mass undeploy, failover, shutdown) floods the log; the health metric path
|
||||
(`IncrementDeadLetter`) is fine because the collector counter is interval-based.
|
||||
|
||||
### [Low] Mandatory two-seed rule forces phantom seeds in single-node installs
|
||||
|
||||
`ClusterOptionsValidator` requires ≥2 seed nodes
|
||||
(ClusterOptionsValidator.cs:29-33); the shipped single-node production artifact satisfies it with
|
||||
a documented placeholder (`deploy/wonder-app-vd03/appsettings.Central.json:9-19`, second seed
|
||||
`localhost:8091` "harmless while absent"). The node dials the phantom seed forever (log noise,
|
||||
periodic connect attempts), and the rule's intent — no startup-ordering dependency — is defeated
|
||||
by fiction. Consider allowing 1 seed with a warning, or a `SingleNode` acknowledgement flag.
|
||||
|
||||
## 2. Performance
|
||||
|
||||
This domain is largely cold-path and correctly engineered for it; nothing here touches the
|
||||
data-plane hot path. Observations:
|
||||
|
||||
- **[Low] Readiness probes fan out five cluster Asks per poll.**
|
||||
`RequiredSingletonsHealthCheck` sends 5 concurrent `Identify` Asks through singleton proxies
|
||||
every `/health/ready` poll (RequiredSingletonsHealthCheck.cs:66-78,111-117). When the singletons
|
||||
live on the peer node each probe is a remoting round trip. Bounded (2s, concurrent) and cheap at
|
||||
typical poll rates, but a tight orchestration probe interval multiplies cluster chatter.
|
||||
- **[Low] Startup compiles every inbound API method sequentially before serving**
|
||||
(Program.cs:379-388). Startup latency scales linearly with method count; a slow Roslyn compile of
|
||||
many methods delays readiness on the *standby* too (it runs the same branch). Acceptable now;
|
||||
parallelize or defer-compile if method counts grow.
|
||||
- **[Low] Per-tick allocations in health collection** (dictionary snapshots per 30s report,
|
||||
`GetAllSiteStates` copy per dashboard poll — CentralHealthAggregator.cs:173-176) are trivial at
|
||||
this cadence. No action.
|
||||
- **Positive:** the site audit telemetry actor is pinned to a dedicated dispatcher
|
||||
(`audit-telemetry-dispatcher`, AkkaHostedService.cs:226-232,1150-1158) so batch SQLite/gRPC work
|
||||
never contends with hot-path actors; `StoreAndForwardService.StartAsync` is awaited rather than
|
||||
sync-blocked (AkkaHostedService.cs:976-981); the ActorSystem DI bridge is a root singleton
|
||||
precisely to avoid per-scope disposal (Program.cs:240-250) — all correct.
|
||||
|
||||
## 3. Conventions
|
||||
|
||||
- **[Medium] REQ-HOST-6 says "Akka.NET bootstrap via Akka.Hosting"; the code hand-rolls HOCON.**
|
||||
`Component-Host.md` (REQ-HOST-6, line 106-114) mandates Akka.Hosting; the actual bootstrap is
|
||||
string-interpolated HOCON + `ActorSystem.Create` (AkkaHostedService.cs:171-176,214-271) with
|
||||
`Akka.Hosting`/`Akka.Cluster.Hosting`/`Akka.Remote.Hosting` referenced but unused for cluster
|
||||
bring-up (ZB.MOM.WW.ScadaBridge.Host.csproj:15-18). This is not just doc drift: Akka.Cluster.Hosting's
|
||||
typed `SplitBrainResolverOption` sets `downing-provider-class` for you — using the mandated API
|
||||
would have prevented the Critical finding. Either implement REQ-HOST-6 or amend it and drop the
|
||||
unused packages.
|
||||
- **[High] Production deploy artifact ships with authentication disabled.**
|
||||
`deploy/wonder-app-vd03/appsettings.Central.json:39-42` sets `"DisableLogin": true` — every
|
||||
request auto-authenticates as `multi-role` with all roles, and the management API on that host is
|
||||
unauthenticated. The base config documents this as "DEV/TEST ONLY — never enable in production"
|
||||
with "no environment guard; a startup warning is the only protection"
|
||||
(src/ZB.MOM.WW.ScadaBridge.Host/appsettings.json:6). A `deploy/` artifact with an install
|
||||
runbook is a production surface; at minimum gate `DisableLogin` on
|
||||
`Environment.IsDevelopment()` or require an explicit `I-understand` companion flag.
|
||||
- **[Medium] LDAP transport is plaintext everywhere, contradicting the security spec.** The design
|
||||
requires "LDAPS/StartTLS required" (CLAUDE.md Security & Auth; Component-Security), but every
|
||||
shipped config — docker (docker/central-node-a/appsettings.Central.json:26-33) and the on-host
|
||||
deploy (deploy/wonder-app-vd03/appsettings.Central.json:26-34) — sets `"Transport": "None",
|
||||
"AllowInsecure": true` and sends the service-account bind password in clear. Dev GLAuth is a fair
|
||||
excuse for docker; the deploy artifact should not normalize it.
|
||||
- **[Medium] Secrets in checked-in dev configs.** SQL credentials, LDAP service-account password
|
||||
and the JWT signing key are committed in `docker/central-node-a/appsettings.Central.json:21-38`
|
||||
(labelled dev-only; the compose pepper is likewise flagged, docker-compose.yml:9-14). The
|
||||
wonder-app-vd03 artifact handles this correctly via `${...}` placeholders + env injection
|
||||
(appsettings.Central.json:20-24) — good pattern, worth backporting a scrubbed variant to docker
|
||||
if these ever leave a lab network. Also note `ConfigSecretsTests.cs` exists in Host.Tests, so the
|
||||
team is tracking this.
|
||||
- **[Low] Stale doc references.** `Component-TraefikProxy.md:121` locates `ActiveNodeHealthCheck`
|
||||
at `src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeHealthCheck.cs`; the check now comes from the
|
||||
shared `ZB.MOM.WW.Health.Akka` package (Program.cs:214-234) and no such file exists.
|
||||
`Component-TraefikProxy.md:115` also still describes `/health/ready` as "database + Akka
|
||||
cluster" — it now includes `required-singletons` (Program.cs:227-230; Component-Host.md REQ-HOST-4a
|
||||
*is* current). Keep the Traefik doc in sync.
|
||||
- **[Low] Five copy-pasted ~60-line singleton registration blocks.**
|
||||
AkkaHostedService.RegisterCentralActors repeats the manager/drain-task/proxy triple five times
|
||||
(AkkaHostedService.cs:399-811) with only names and Props varying. Extract a
|
||||
`StartCentralSingleton(name, props, drainTimeout)` helper — the copy-paste already caused drift
|
||||
(outbox/ingest lack drain tasks).
|
||||
- **Positive conventions compliance:** Options pattern is followed rigorously (component-owned
|
||||
options classes, `ValidateOnStart`, dedicated validators with key-naming messages —
|
||||
ClusterOptionsValidator, HealthMonitoringOptionsValidator); HOCON interpolation is
|
||||
injection-safe (`QuoteHocon`/`DurationHocon`, AkkaHostedService.cs:280-295); Serilog enrichment
|
||||
matches REQ-HOST-8; startup validation matches REQ-HOST-4 including cross-field port-collision
|
||||
rules; single-binary role dispatch matches REQ-HOST-1/2; the `/health/ready` vs `/health/active`
|
||||
tier separation (leadership deliberately excluded from readiness) is well-reasoned and
|
||||
documented in place (Program.cs:346-356).
|
||||
|
||||
## 4. Underdeveloped Areas
|
||||
|
||||
1. **No real multi-node failover test anywhere.** `CentralFailoverTests` are single-process
|
||||
simulations (JWT across two service instances, CentralFailoverTests.cs:27-47);
|
||||
`SiteFailoverTests`/`DualNodeRecoveryTests` similarly exercise component state recovery, not
|
||||
cluster membership; `CoordinatedShutdownTests` grep source text (CoordinatedShutdownTests.cs:11-43);
|
||||
`HoconBuilderTests` assert config strings. The repo's own notes describe the MultiNode test
|
||||
runner (AkkaDotNet/18-MultiNodeTestRunner.md) but it is unused. This is the direct cause of the
|
||||
Critical finding surviving: nothing ever killed the oldest node and waited for the singleton to
|
||||
move. Even a plain xunit test spinning two `ActorSystem`s in-process and calling
|
||||
`cluster.Down()`/terminating one would have caught it.
|
||||
2. **Windows Service lifecycle management is thin.** `Component-ClusterInfrastructure.md:19`
|
||||
claims "Manage Windows service lifecycle (start, stop, restart)"; the implementation is
|
||||
`UseWindowsService()` (Program.cs:74,401) plus install scripts. Service recovery actions
|
||||
(restart-on-failure), the watchdog implied by `down-if-alone` ("an external mechanism must
|
||||
restart both nodes" — AkkaDotNet/03-Cluster.md:64) do not exist in the repo; after a
|
||||
down-if-alone self-down the process presumably exits (or worse, idles) with nothing to restart
|
||||
it. The down-if-alone recovery story needs an explicit design (does the process exit? does the
|
||||
service restart it? does it rejoin cleanly?) and a drill.
|
||||
3. **TLS is absent at every layer of this domain**: Traefik terminates plain HTTP with an
|
||||
unauthenticated dashboard (docker/traefik/traefik.yml:5-7 — documented as dev-only in
|
||||
Component-TraefikProxy.md:131-138), Akka remoting is unencrypted TCP, site gRPC is `http://`.
|
||||
Fine for the lab; there is no production TLS profile or documented path yet.
|
||||
4. **Dockerfile has no `HEALTHCHECK`** and compose has no health-based dependency ordering;
|
||||
liveness `/healthz` runs zero checks (Program.cs:353-355) so a wedged actor system still
|
||||
reports live. Acceptable minimalism, but an orchestrator migration will need both.
|
||||
5. **`NodeName` is missing from the wonder-app-vd03 deploy overlays**
|
||||
(deploy/wonder-app-vd03/appsettings.Central.json has no `Node.NodeName`), so audit rows from
|
||||
that install get a NULL `SourceNode` (the base file documents the normalisation,
|
||||
src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Central.json:3) — undermines the per-node audit/KPI
|
||||
features (`IX_AuditLog_Node_Occurred`, per-node stuck KPIs) on the one real deployment.
|
||||
6. **Known-but-open follow-ups acknowledged in code/docs:** SecuredWrite audit rows leave
|
||||
`SourceNode` NULL (CLAUDE.md notes it as logged); no central persistence of OPC UA cert trust;
|
||||
`docker-env2` mirrors the main topology (verified: same Traefik config shape, same
|
||||
seed/port conventions) so it inherits every finding above.
|
||||
7. **Health monitoring has no history by design** (Component-HealthMonitoring.md:93-97) — now
|
||||
partially superseded by KPI History (#26), which the docs handle; but online/offline
|
||||
*transitions* are still unrecorded anywhere (doc line 97 defers it), so post-incident "when did
|
||||
the site drop" questions remain unanswerable beyond log scraping.
|
||||
|
||||
## Prioritized Recommendations
|
||||
|
||||
1. **[Critical, one line + tests] Enable the split-brain resolver.** Add
|
||||
`downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster"` to
|
||||
`BuildHocon` (AkkaHostedService.cs:250) — or move the bootstrap to Akka.Cluster.Hosting per
|
||||
REQ-HOST-6, which sets it via typed options. Then add a two-node in-process failover test
|
||||
(kill oldest → assert singleton re-hosted, member removed within stable-after + margin) and a
|
||||
documented manual drill against the docker cluster (`docker kill scadabridge-central-a`, watch
|
||||
Traefik + singleton logs).
|
||||
2. **[High] Unify the "active node" definition** on singleton-host (oldest) semantics for
|
||||
`ActiveNodeGate`, `AkkaClusterNodeProvider.SelfIsPrimary`, the event-log purge gate, and the
|
||||
`/health/active` check; today's leader-based checks purge the wrong node's event log and label
|
||||
the wrong node Primary once leader ≠ oldest.
|
||||
3. **[High] Remove `DisableLogin: true` from `deploy/wonder-app-vd03`** or add a hard environment
|
||||
guard around the flag.
|
||||
4. **[Medium] Fix the CoordinatedShutdown drain budget** (`cluster-leave` phase timeout ≥
|
||||
graceful-stop timeout) and add drain tasks for the notification-outbox and audit-log-ingest
|
||||
singletons — or delete all five drains if the idempotent-DB argument makes them unnecessary.
|
||||
5. **[Medium] Fix `appsettings.Site.json` seed `localhost:8084`** and extend `StartupValidator`'s
|
||||
seed-port loop to reject `MetricsPort` collisions.
|
||||
6. **[Medium] Make health-report delivery observable**: switch `AkkaHealthReportTransport` to an
|
||||
acked Ask (reusing the existing site→central forward pattern) so the counter-restore logic in
|
||||
`HealthReportSender` actually fires, and surface `LastReportReceivedAt` staleness on the
|
||||
dashboard so heartbeat-alive/metrics-dead sites are visible.
|
||||
7. **[Medium] Add `stop_grace_period: 30s`** to the app services in both compose files.
|
||||
8. **[Low] Housekeeping:** evict deleted sites from `CentralHealthAggregator`; rate-limit
|
||||
dead-letter warnings; extract the singleton-registration helper; sync
|
||||
`Component-TraefikProxy.md` with the shared health-check library; add `NodeName` to the
|
||||
wonder-app-vd03 overlays; reconsider the mandatory-two-seeds rule for single-node installs.
|
||||
@@ -0,0 +1,160 @@
|
||||
# Architecture Review — Central–Site Communication & Store-and-Forward
|
||||
|
||||
Reviewer domain: `src/ZB.MOM.WW.ScadaBridge.Communication`, `src/ZB.MOM.WW.ScadaBridge.StoreAndForward`, the telemetry transport paths that ride them, and the corresponding test projects.
|
||||
Date: 2026-07-08.
|
||||
|
||||
## Scope
|
||||
|
||||
Read in full: `Component-Communication.md`, `Component-StoreAndForward.md`; all 9 S&F source files (`StoreAndForwardService.cs`, `StoreAndForwardStorage.cs`, `ReplicationService.cs`, `NotificationForwarder.cs`, `ParkedMessageHandlerActor.cs`, options/messages/DI); all Communication source (`CentralCommunicationActor.cs`, `SiteCommunicationActor.cs`, `CommunicationService.cs`, `DebugStreamService.cs`, `DebugStreamBridgeActor.cs`, `StreamRelayActor.cs`, `SiteStreamGrpcServer.cs`, `SiteStreamGrpcClient.cs`, `SiteStreamGrpcClientFactory.cs`, `ReconcileService.cs`, `PendingDeploymentPurgeActor.cs`, `CommunicationOptions.cs`); the transport ends of the telemetry pipelines (`AuditLog/Site/Telemetry/SiteAuditTelemetryActor.cs`, `ClusterClientSiteAuditClient.cs`); and the Host wiring that composes them (`Host/Actors/AkkaHostedService.cs` — `BuildHocon`, `RegisterCentralActors`, `RegisterSiteActorsAsync`), plus `SiteRuntime/Actors/SiteReplicationActor.cs` and the `Notify.Send` enqueue path in `SiteRuntime/Scripts/ScriptRuntimeContext.cs`. Central-side ingest internals (NotificationOutboxActor, AuditLogIngestActor, SiteCallAuditActor) are another reviewer's domain; only their transport contracts are assessed here.
|
||||
|
||||
## Maturity Verdict
|
||||
|
||||
This is a carefully engineered, unusually well-documented subsystem that is strong *in the small* — idempotency keys threaded end-to-end (NotificationId, EventId, TrackedOperationId), Sender captured before every PipeTo, compare-and-swap storage updates guarding the sweep against operator races, additive-only proto discipline with contract tests, and deliberate, written-down failure semantics on nearly every path. However, it has a small number of serious *in-the-large* lifecycle/topology defects that the (otherwise broad) test suite is structurally blind to, because the risky collaborators are mocked away: the ClusterClient recreate-on-address-change path will almost certainly crash-loop the `CentralCommunicationActor` in production, the standby site node runs the full S&F delivery pipeline with no active-node gate (systematic duplicate-delivery exposure the spec explicitly says should only occur on failover), and one debug session's gRPC failover disposes the shared channel out from under every other session to the same site. The retry sweep's serial, unbounded design also collapses under a realistic central outage backlog. Verdict: **high code quality, mid maturity as a distributed system — 3–4 targeted fixes away from solid.**
|
||||
|
||||
---
|
||||
|
||||
## Findings — Stability
|
||||
|
||||
### [Critical] ClusterClient recreate-on-address-change causes actor-name collision and a permanent restart loop
|
||||
`CentralCommunicationActor.HandleSiteAddressCacheLoaded` stops the old client and immediately creates a new one under the same name:
|
||||
|
||||
- `CentralCommunicationActor.cs:514-521` — `Context.Stop(_siteClients[siteId].Client)` followed synchronously by `_siteClientFactory.Create(...)`.
|
||||
- `CentralCommunicationActor.cs:36-40` (`DefaultSiteClientFactory`) — `system.ActorOf(ClusterClient.Props(settings), $"site-client-{siteId}")`: a **named top-level actor**.
|
||||
|
||||
`Context.Stop` is asynchronous; the name `site-client-{siteId}` stays reserved until the old actor fully terminates. Creating the replacement in the same message handling will throw `InvalidActorNameException` essentially every time. The `Create` call is **outside** the surrounding try/catch (which only guards `ActorPath.Parse`, `CentralCommunicationActor.cs:492-505`), so the exception escapes the handler → the user-guardian default directive **restarts** the actor → the new incarnation's `_siteClients` dictionary is empty, but all old `site-client-*` actors are still alive at the system level. Every subsequent 60-second refresh (`PreStart`, `CentralCommunicationActor.cs:564-569`) then tries to create clients for **all** sites, collides on the reserved names, and crashes again. Failure scenario: an operator edits one site's NodeA address in the Central UI → central loses ClusterClient routing to *every* site (all `SiteEnvelope`s dropped at `HandleSiteEnvelope`, `CentralCommunicationActor.cs:398-413`) until the central process is restarted. This is the exact path the design doc promises works: "ClusterClient instances are recreated when contact points change" (`Component-Communication.md:238`).
|
||||
|
||||
Secondary issue on the same lines: `siteId` is interpolated into an actor name unvalidated. A `SiteIdentifier` containing `/`, whitespace, or other non-path characters throws `InvalidActorNameException` on first creation with the same restart-loop consequence — contrast with `SiteStreamGrpcServer.SubscribeInstance`, which validates `correlation_id` with `ActorPath.IsValidPathElement` for precisely this reason (`SiteStreamGrpcServer.cs:229-234`).
|
||||
|
||||
Why tests miss it: `CentralCommunicationActorTests` substitutes `ISiteClientFactory` with an NSubstitute mock returning TestProbes (`tests/.../CentralCommunicationActorTests.cs:42-51`) — the real naming/lifecycle machinery is never exercised anywhere.
|
||||
|
||||
Fix shape: unique names (`site-client-{siteId}-{seq}` via a counter, like `SiteStreamGrpcServer._actorCounter`), or watch `Terminated` and defer recreation, plus sanitize/validate the site id, plus one test using `DefaultSiteClientFactory`.
|
||||
|
||||
### [High] Standby site node runs the full S&F delivery pipeline — duplicate delivery is systematic, not a failover residue
|
||||
The design says the standby only *applies* replicated operations and "on failover, the new active node resumes delivery" (`Component-StoreAndForward.md:77-80`). The code starts delivery on **both** nodes:
|
||||
|
||||
- `AkkaHostedService.cs:981` — `await storeAndForwardService.StartAsync()` runs in `RegisterSiteActorsAsync` on **every** site node, starting the retry timer (`StoreAndForwardService.cs:337-341`).
|
||||
- `AkkaHostedService.cs:989, 998, 1033` — all three delivery handlers (ExternalSystem, CachedDbWrite, Notification) are registered unconditionally on every node.
|
||||
- `SiteReplicationActor.cs:280-291` — replicated `Add` operations are applied to the standby's SQLite, so the standby's sweep has real Pending rows to deliver.
|
||||
|
||||
There is no active-node/leader gate anywhere in `StoreAndForwardService.RetryPendingMessagesAsync` (`StoreAndForwardService.cs:563-589`) or in the delivery handlers (`ExternalSystemClient.DeliverBufferedAsync`, `ExternalSystemGateway/ExternalSystemClient.cs:173-224` — checked, no gate). Consequences:
|
||||
|
||||
1. **Duplicate deliveries in steady operation.** Both nodes' sweeps become due for the same message at the same wall-clock moment (the replicated row carries the same `last_attempt_at`). Whichever delivers first replicates a `Remove`, but if both attempts are in flight concurrently — a window the size of the delivery duration (an HTTP call, often ≥ hundreds of ms) plus replication latency — the target receives the call **twice**. Over a long outage with many buffered messages, duplicates recur every recovery. Notifications survive this (central dedups on `NotificationId` insert-if-not-exists), but `ExternalSystem` calls and `CachedDbWrite`s do not — "idempotency is the caller's responsibility" was scoped to *rare failover* duplicates, not steady-state ones.
|
||||
2. **Doubled retry traffic** against an already-struggling target during every outage.
|
||||
3. Both nodes emit `CachedCallTelemetry` attempts for the same operation with independent retry counts, muddying the central audit picture.
|
||||
|
||||
Fix shape: gate the sweep (and ideally the queue-depth gauge registration) behind the same leader-check used by `SiteCommunicationActor.DefaultIsActiveCheck` (`SiteCommunicationActor.cs:498-507`), re-evaluated per sweep tick so failover resumes delivery automatically.
|
||||
|
||||
### [High] One debug session's gRPC failover disposes the shared channel under every other session to the same site
|
||||
`SiteStreamGrpcClientFactory` caches **one** `SiteStreamGrpcClient` per site, keyed by site id, replacing (and disposing) it whenever a caller asks for a different endpoint (`SiteStreamGrpcClientFactory.cs:52-78`). `DebugStreamBridgeActor` holds **per-session** node preference and flips endpoint on every stream error (`DebugStreamBridgeActor.cs:416, 468-483`). With two concurrent debug sessions to the same site:
|
||||
|
||||
- Session A (on NodeA) hits an error and flips to NodeB → `GetOrCreate(site, NodeB)` **disposes** the NodeA-bound client. `SiteStreamGrpcClient.ReleaseResources` cancels **every** registered subscription CTS and disposes the channel (`SiteStreamGrpcClient.cs:314-324`) — killing session B's healthy stream.
|
||||
- Session B's `SubscribeAsync` observes the cancellation as an error → B flips to NodeA → disposes the NodeB client → kills A's brand-new stream. The two sessions ping-pong, burning both sessions' 3-retry budgets (`DebugStreamBridgeActor.cs:45`) and terminating both.
|
||||
|
||||
Also, `CleanupGrpc` (`DebugStreamBridgeActor.cs:486-495`) calls `GetOrCreate` during teardown — if the cached client is bound to the other endpoint, cleanup *creates a fresh channel* (and disposes the cached one) merely to call `Unsubscribe`, with the same collateral effect.
|
||||
|
||||
Fix shape: key the cache by `(siteId, endpoint)` (both node channels can coexist), or reference-count clients, or give each bridge actor its own client instance. The single-flip failover for a *site-wide* address change can stay in the factory; per-session failover must not mutate shared state.
|
||||
|
||||
### [Medium] Replication operations can be applied out of order
|
||||
`ReplicationService.FireAndForget` wraps each operation in its own `Task.Run` (`ReplicationService.cs:134-149`). Two operations for the same message issued in quick succession (Add then Remove; Park then operator Requeue) are handed to the thread pool as independent tasks — the pool does not preserve ordering, so the `replicationActor.Tell(...)` calls (handler wired at `AkkaHostedService.cs:897-901`) can happen inverted. Akka's per-sender/receiver ordering guarantee only applies *after* the Tells, so the standby can apply `Remove` (no-op, row absent) then `Add` → an orphan Pending row that the ungated standby sweep (finding above) or a failover will re-deliver. The `Task.Run` buys nothing — the handler is a non-blocking Tell returning `Task.CompletedTask`; invoking it inline (with a try/catch) preserves ordering by construction.
|
||||
|
||||
### [Medium] Replication failures are logged at Debug — a dead replication channel is invisible
|
||||
`ReplicationService.cs:144-147` logs a failed replication at `LogDebug`. If the handler starts failing systematically (peer unreachable in a way `SendToPeer` doesn't detect, serialization fault of `StoreAndForwardMessage`), the standby's buffer silently diverges and the next failover loses the entire un-replicated delta — with zero operator-visible signal at default log levels and no counter metric. `SiteReplicationActor.SendToPeer`'s no-peer drop is also Debug (`SiteReplicationActor.cs:139-149`). At minimum Warning + a telemetry counter; the repo already has `ScadaBridgeTelemetry` plumbing for exactly this kind of gauge.
|
||||
|
||||
### [Medium] Park/Requeue replication is a blind UPDATE — lost on a standby missing the row
|
||||
`ApplyReplicatedOperationAsync` applies `Park` and `Requeue` via `UpdateMessageAsync` (`ReplicationService.cs:121-130`), which is `UPDATE ... WHERE id = @id` (`StoreAndForwardStorage.cs:210-232`) — zero rows affected is silent. If the original `Add` was lost (fire-and-forget), the park state never materializes on the standby: after failover the message is simply *gone* from the parked view (operator can't retry/discard it) even though the full message payload was in hand in the replication operation. An upsert (INSERT OR REPLACE) for Park/Requeue would self-heal the lost-Add case for free.
|
||||
|
||||
### [Medium] Notifications park after ~25 minutes of central unreachability, and the "no limit" escape hatch is unused
|
||||
`Notify.Send` enqueues with no `maxRetries` (`ScriptRuntimeContext.cs:2197-2203`) → `DefaultMaxRetries = 50` (`StoreAndForwardOptions.cs:21`) at the 30 s default interval = ~25 minutes of central outage before a notification parks and requires per-message operator action. `Component-StoreAndForward.md:52` explicitly documents `maxRetries: 0` as the escape hatch for callers that "genuinely require unbounded retry" — but the *only* notification producer doesn't use it, and no host config raises the cap for the Notification category specifically. A central maintenance window longer than 25 minutes strands every notification raised during it. Compounding: the serial sweep (Performance #1) makes effective intervals much longer than 30 s under backlog, but retries are counted per sweep-attempt, so the wall-clock park horizon is at least honest — the operational cliff is the manual un-parking.
|
||||
|
||||
### [Medium] Corrupt buffered notification is discarded and reported as delivered
|
||||
`NotificationForwarder.DeliverAsync` returns `true` (→ engine removes the row) when the payload fails to deserialize (`NotificationForwarder.cs:79-87`). The message is permanently lost with only a site-log Warning: no parked row, no central `Notifications` row, no audit row. Returning `false` (permanent failure → **park**) would preserve the payload for forensics at zero protocol cost — parking is exactly the designed home for undeliverable-but-preserved messages.
|
||||
|
||||
### [Low] Central heartbeat state is not replicated to the peer central node
|
||||
`HandleSiteHealthReport` fans reports to the peer via DistributedPubSub (`CentralCommunicationActor.cs:360-373`), but `HandleHeartbeat` marks only the local aggregator (`CentralCommunicationActor.cs:349-353`). ClusterClient delivers each site's heartbeat to one central node; the other's aggregator sees none. After a central failover the new active node briefly shows all sites offline until fresh heartbeats arrive (≤ heartbeat interval + offline threshold). Minor, but inconsistent with the care taken for health reports.
|
||||
|
||||
### [Low] `SubscribeInstance` concurrency-limit check is check-then-act
|
||||
`SiteStreamGrpcServer.cs:243-254`: `_activeStreams.Count >= _maxConcurrentStreams` then insert — two concurrent subscribes at the boundary both pass. Bounded overshoot only; not worth locking, worth a comment.
|
||||
|
||||
---
|
||||
|
||||
## Findings — Performance
|
||||
|
||||
### [High] The retry sweep is serial, unbounded, and single-laned across categories — it collapses under backlog
|
||||
Three compounding design choices:
|
||||
|
||||
- `GetMessagesForRetryAsync` has **no LIMIT** — it loads every due row into memory each sweep (`StoreAndForwardStorage.cs:183-203`).
|
||||
- `RetryPendingMessagesAsync` delivers strictly one-at-a-time (`StoreAndForwardService.cs:571-579`) while `_retryInProgress` blocks the next sweep (`StoreAndForwardService.cs:566`).
|
||||
- Every notification forward attempt against a down central costs a full 30 s Ask timeout (`NotificationForwardTimeout`, `CommunicationOptions.cs:36`; `NotificationForwarder.cs:93-95`).
|
||||
|
||||
Failure scenario: central is down for an hour and scripts have buffered 200 notifications. One sweep = 200 × 30 s ≈ 100 minutes of serial timeouts — during which no `ExternalSystem` or `CachedDbWrite` retries happen either (single lane, head-of-line blocking across categories and targets: one slow external system delays notification forwarding to a *healthy* central and vice versa). There is no per-target short-circuit — N messages to the same dead target incur N consecutive timeouts per sweep. The queue drains far slower than it fills; retry counts tick toward parking on a wildly distorted cadence.
|
||||
|
||||
Fix shape: `LIMIT` on the due query; group by (category, target) and short-circuit remaining messages for a target after its first transient failure in a sweep; optionally parallelize across targets with a small cap. Any one of the three helps materially.
|
||||
|
||||
### [Medium] No explicit Akka serializer configuration — every cross-cluster contract rides default reflective JSON
|
||||
`BuildHocon` (`AkkaHostedService.cs:214-271`) configures remoting, SBR, and failure detectors, but **no** `akka.actor.serializers` / `serialization-bindings`. All command/control and telemetry traffic — deployment configs, `IngestAuditEventsCommand` batches (entities converted *from* compact proto DTOs back to POCOs before the Akka hop, `ClusterClientSiteAuditClient.cs:72-85`), `ReplicationOperation` with full message payloads — is serialized by Akka.NET's default Newtonsoft JSON serializer with embedded CLR type manifests. Costs: (a) hot-path CPU/allocation and payload bloat on the highest-volume channel (audit telemetry); (b) the documented "additive-only message contract evolution" rule (`CLAUDE.md`, Commons conventions) is only half the story — wire compatibility is also coupled to *type and namespace identity*, so a rename/move of a Commons message type breaks rolling cross-version central↔site communication with no test to catch it. `MessageContractTests` asserts correlation-id presence only (`tests/.../MessageContractTests.cs`), not serialized round-trip stability. Ironic detail: the telemetry path builds efficient proto DTOs, then converts them back to POCOs specifically to cross the ClusterClient hop.
|
||||
|
||||
### [Low] S&F SQLite: no WAL, no explicit busy_timeout, connection-per-operation
|
||||
Every `StoreAndForwardStorage` method opens a fresh `SqliteConnection` (e.g. `StoreAndForwardStorage.cs:137-139, 185-186`) with a bare `Data Source=` connection string (`StoreAndForward/ServiceCollectionExtensions.cs:22-24`) — default rollback-journal mode. Concurrent writers exist by design: script-thread enqueues, the sweep, replication applies (standby), and gRPC `PullSiteCalls`/parked-query reads. WAL + `busy_timeout` pragma would remove writer/reader blocking; Microsoft.Data.Sqlite's pooling makes connection-per-op tolerable but the pragmas are one line each in `InitializeAsync`.
|
||||
|
||||
### [Low] Due-check does per-row `julianday()` string parsing on ISO-8601 timestamps
|
||||
`StoreAndForwardStorage.cs:195-197` computes due-ness with `(julianday('now') - julianday(last_attempt_at)) * 86400000` against `"O"`-format strings with 7-digit fractional seconds and `+00:00` offsets — relying on SQLite's lenient datetime parser, non-indexable, and re-parsed for every Pending row on every 10 s sweep. Storing epoch milliseconds would be cheaper, indexable, and immune to format drift.
|
||||
|
||||
### [Low] Audit-observer notification is awaited inline in the sweep
|
||||
`NotifyCachedCallObserverAsync` is awaited per attempt inside `RetryMessageAsync` (`StoreAndForwardService.cs:620-626, 688-694, 711-717`). It's exception-isolated (good), but not latency-isolated: a slow observer (SQLite audit write) stretches the already-serial sweep. Fire-and-forget with isolation, or post to a channel, would decouple it.
|
||||
|
||||
---
|
||||
|
||||
## Findings — Conventions
|
||||
|
||||
### [Medium] Spec-vs-code drift in the S&F component doc
|
||||
Two material divergences from `Component-StoreAndForward.md`:
|
||||
1. **Standby-passive model** (doc lines 77-80) vs. the ungated standby delivery pipeline (Stability #2). The doc's "a few duplicate deliveries [on failover]... acceptable trade-off" framing materially understates actual behavior.
|
||||
2. **`Notify.Send` latency**: the doc/CLAUDE.md say Send "returns a NotificationId status handle immediately", but `EnqueueAsync`'s immediate-delivery attempt (`StoreAndForwardService.cs:500-529`) runs the `NotificationForwarder` Ask inline on the script's await — when central is down, every `Notify.Send` blocks its script execution for up to `NotificationForwardTimeout` (30 s) before buffering. Either document the worst case or make the Notification category enqueue with `attemptImmediateDelivery: false` + an immediate sweep kick.
|
||||
|
||||
### [Low] `StreamRelayActor.WriteToChannel` has a dead branch and silent loss
|
||||
`TryWrite` on a bounded channel in `DropOldest` mode never returns `false` — it evicts the oldest item and succeeds. The "Channel full, dropping event" warning (`StreamRelayActor.cs:105-111`) can never fire, while the *actual* loss (the evicted oldest event) is completely unlogged. Lossy-under-backpressure is fine for debug view per spec (`Component-Communication.md:59`), but the observability is inverted: dead warning, silent real drop. Count evictions (channel `ItemDropped` callback via `CreateBounded(options, itemDropped)`) or log periodically.
|
||||
|
||||
### [Low] Duplicated Ask-timeout constants and divergent fault semantics between the two ingest transports
|
||||
`CentralCommunicationActor.DefaultAuditIngestAskTimeout` copies `SiteStreamGrpcServer.AuditIngestAskTimeout` by hand (`CentralCommunicationActor.cs:114-124` vs `SiteStreamGrpcServer.cs:45`), and the two handlers deliberately differ on fault behavior (propagate `Status.Failure` vs swallow-to-empty-ack — documented at `CentralCommunicationActor.cs:104-111`). Both are acknowledged in comments, but two transports for the same operation with different failure semantics and copy-pasted tuning is drift waiting to happen (see Underdeveloped, dual transports).
|
||||
|
||||
### [Low] Heartbeat `IsActive` uses leader-check while singletons follow oldest-node
|
||||
`SiteCommunicationActor.DefaultIsActiveCheck` reports active = cluster leader (`SiteCommunicationActor.cs:498-507`). Akka cluster singletons run on the *oldest* qualifying node; leader and oldest usually coincide in a 2-node stable cluster but can diverge transiently. The comment says it deliberately mirrors `ActiveNodeGate`, so it's a consistent repo convention — but the convention itself can misreport which node hosts the DeploymentManager singleton. Worth one shared helper with an oldest-member check.
|
||||
|
||||
### [Low] `TransportHeartbeatInterval` does double duty
|
||||
The Akka.Remote transport failure-detector interval (`BuildHocon`, `AkkaHostedService.cs:246-249`) is also the *application-level* site→central heartbeat cadence (`SiteCommunicationActor.cs:421-425`). These are unrelated concerns (spec treats them separately: `Component-Communication.md:268-274` vs health reporting); tuning one silently retunes the other.
|
||||
|
||||
### Positives worth recording
|
||||
- Tell/Ask discipline matches the repo rule everywhere: Ask only at system boundaries (`CommunicationService`, forwarder, telemetry clients), Tell/Forward on hot paths with Sender preservation for reply routing (`SiteCommunicationActor.cs:270-388`, `CentralCommunicationActor.cs:398-413`).
|
||||
- Correlation IDs on every request/response contract, verified by tests.
|
||||
- The at-least-once notification handoff is textbook: ack-after-persist at central, script-generated `NotificationId` as buffered-row id *and* idempotency key, re-stamped provenance at the forwarder (`NotificationForwarder.cs:123-160`).
|
||||
- The sweep's CAS updates (`UpdateMessageIfStatusAsync`, `StoreAndForwardStorage.cs:243-269`) correctly close the operator-retry/discard vs sweep race, replicated with correct captured state (`StoreAndForwardService.cs:861-892`).
|
||||
- gRPC server stream lifecycle is thorough: readiness + shutdown gating, correlation-id validation, duplicate-stream replacement, session-lifetime cap, leak-proofed Subscribe failure cleanup, own-entry-only removal (`SiteStreamGrpcServer.cs:211-327`).
|
||||
- Stream-first debug subscription with buffered gap-window replay and per-entity timestamp dedup (`DebugStreamBridgeActor.cs:134-176, 286-329`) is a correct solution to the snapshot/stream ordering problem, including the flapping-stream retry-budget stability window.
|
||||
- Additive proto evolution is real and tested (`ProtoContractTests`, `ProtoRoundtripTests`), with the manual regen process documented.
|
||||
|
||||
---
|
||||
|
||||
## Underdeveloped Areas
|
||||
|
||||
1. **Dual transports for the same ingest operations.** `IngestAuditEvents`/`IngestCachedTelemetry` exist both as gRPC unary RPCs (`SiteStreamGrpcServer.cs:330-433`) and as ClusterClient commands (`CentralCommunicationActor.cs:262-305`); production push uses ClusterClient (`ClusterClientSiteAuditClient`), leaving the gRPC ingest surface production-idle (doc admits: "the gRPC-receiving counterpart", `Component-Communication.md:87`). Two code paths, two fault semantics, double the test/maintenance surface — consolidation is pending and should be scheduled.
|
||||
2. **No real-collaborator test for the ClusterClient cache lifecycle.** `DefaultSiteClientFactory` is untested; address-change recreation is only tested through mocks — precisely where the Critical finding lives.
|
||||
3. **No test or design treatment of standby-node sweep behavior.** Neither the S&F tests nor an integration test asserts that only the active node delivers; the docker-cluster smoke topology would expose it with a two-node site and a delayed target.
|
||||
4. **Replication has no anti-entropy/resync.** A standby down for an hour rejoins and applies only *new* operations; the missed delta diverges forever (until rows drain naturally). The doc covers only the "last few operations" loss case. A periodic checksum or full-row reconciliation (the pattern already exists for audit rows via `PullAuditEvents`) is absent for the S&F buffer itself.
|
||||
5. **Parked rows have unbounded retention.** By design parked messages persist until operator action (`Component-StoreAndForward.md:110-116`), but there is no aging alert, cap, or archival — a forgotten site accumulates parked rows indefinitely with only a health-metric count.
|
||||
6. **Reconciliation cursor edge.** `PullAuditEvents`/`PullSiteCalls` batch continuation relies on central advancing a `since_utc` cursor from the last returned timestamp (`SiteStreamGrpcServer.cs:478-484, 563-572`); rows sharing a boundary timestamp are protected only by EventId dedup / upsert-on-newer at central — fine, but the site-side read contract (`>` vs `>=`) isn't pinned by a test in this project.
|
||||
7. **`DebugStreamService._sessions` on central failover** — sessions are process-local with no persistence (documented as acceptable: engineer re-establishes), but the SignalR consumer contract (`OnStreamTerminated`) is the only notification; a central restart drops sessions with no signal at all. Known/accepted, listed for completeness.
|
||||
|
||||
---
|
||||
|
||||
## Prioritized Recommendations
|
||||
|
||||
1. **[Critical] Fix ClusterClient recreation** (`CentralCommunicationActor.cs:514-521`, `DefaultSiteClientFactory`): unique per-incarnation actor names (suffix a counter) or Terminated-watch before re-create; validate/sanitize `siteId` for actor-name safety; wrap `Create` in the per-site try/catch; add a test using the real factory that edits a site's address twice.
|
||||
2. **[High] Gate the S&F retry sweep to the active node** — check leader/oldest status at the top of `RetryPendingMessagesAsync` (or gate `StartAsync`'s timer), re-evaluated per tick so failover resumes delivery within one sweep interval. Add a two-node test.
|
||||
3. **[High] Stop disposing shared gRPC clients on per-session failover** — key `SiteStreamGrpcClientFactory` by `(site, endpoint)` (allow both node channels concurrently), reserve dispose-and-replace for site removal/address edits driven by the address cache, not by individual bridge actors.
|
||||
4. **[High] Bound and de-serialize the retry sweep** — `LIMIT` the due query, short-circuit per target after the first transient failure in a sweep, and consider a small parallelism cap across distinct targets; separately consider a much shorter forward timeout for the notification category or `attemptImmediateDelivery: false` on `Notify.Send` to unblock script threads.
|
||||
5. **[Medium] Make replication trustworthy**: replace `Task.Run` fire-and-forget with an inline Tell (ordering), log failures at Warning with a counter metric, and make Park/Requeue applies upserts so a lost Add self-heals.
|
||||
6. **[Medium] Pin the wire format**: configure an explicit serializer (or at least add serializer round-trip compatibility tests for the cross-cluster Commons contracts) so the additive-only rule is actually enforced end-to-end; long-term, stop converting proto DTOs back to POCOs for the ClusterClient hop.
|
||||
7. **[Medium] Unstrand notifications**: pass `maxRetries: 0` (documented no-limit) or a category-specific high cap on the `Notify.Send` enqueue path, or auto-requeue parked Notification-category rows when central connectivity recovers.
|
||||
8. **[Medium] Park (don't silently drop) corrupt notification payloads** in `NotificationForwarder.DeliverAsync` — return `false` instead of `true`.
|
||||
9. **[Low] SQLite hygiene**: enable WAL + busy_timeout in `InitializeAsync`; migrate timestamps to epoch ms for the due-check.
|
||||
10. **[Low] Observability polish**: count DropOldest evictions in `StreamRelayActor`; replicate heartbeat marks to the peer central node (or document the failover blind window); split the app heartbeat interval from the transport failure-detector setting.
|
||||
@@ -0,0 +1,155 @@
|
||||
# Architecture Review 03 — Site Runtime & Data Connection Layer
|
||||
|
||||
Reviewer domain: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime`, `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer`, `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging`, matched against `docs/requirements/Component-SiteRuntime.md`, `Component-DataConnectionLayer.md`, `Component-SiteEventLogging.md`, and `HighLevelReqs.md`. All findings are grounded in direct reads of the cited files.
|
||||
|
||||
## Scope
|
||||
|
||||
- **Site Runtime actors**: `DeploymentManagerActor` (1,720 LOC), `InstanceActor` (1,572), `ScriptActor` (594), `ScriptExecutionActor` (305), `AlarmActor` (780), `AlarmExecutionActor`, `NativeAlarmActor` (448), `CertStoreActor` (189), `SiteReplicationActor` (292), `SiteReconciliationActor`.
|
||||
- **Script subsystem**: `ScriptCompilationService`, `SharedScriptLibrary`, `ScriptExecutionScheduler`, `ScriptRuntimeContext` (2,384 — skimmed for Ask/blocking patterns), trigger expressions.
|
||||
- **Streaming**: `SiteStreamManager` (BroadcastHub + per-subscriber buffers).
|
||||
- **Persistence**: `SiteStorageService` (site SQLite: deployed configs, static overrides, artifacts, `native_alarm_state`), `OperationTrackingStore` (not deep-read).
|
||||
- **DCL**: `DataConnectionManagerActor` (345), `DataConnectionActor` (2,000), `OpcUaDataConnection` (401), `RealOpcUaClient` (1,313 — lifecycle sections), `MxGatewayDataConnection` (297 — connect path), alarm mappers, browse/search/verify paths.
|
||||
- **Site Event Logging**: `SiteEventLogger` (326), purge/query services.
|
||||
- **Tests**: inventoried `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests`, `...DataConnectionLayer.Tests`, `...SiteEventLogging.Tests`.
|
||||
|
||||
## Maturity Verdict
|
||||
|
||||
This is a mature, unusually carefully engineered subsystem — the Become/Stash state machine, in-flight subscribe bookkeeping, adapter-generation guards, seed-read retry, redeploy termination-watch buffering, and per-source native-alarm cap/eviction all show evidence of multiple hardening passes, and the code matches the design docs closely with extensive rationale comments and broad targeted test coverage. The residual risk is concentrated in three places: **adapter lifecycle on reconnect** (the previous OPC UA / MxGateway client object is silently abandoned on every `ConnectAsync`, leaking sessions and leaving a stale `ConnectionLost` handler that can flap a healthy connection), **script containment** (the execution timeout is cooperative-only, so a hung blocking script permanently consumes one of the 8 dedicated script threads; trigger expressions block actor threads on the default dispatcher), and **deployment honesty** (a site-side script compile failure does not reject the deployment as the spec requires — the instance starts with the script silently missing and central is told Success). None of these break the happy path; all of them are the kind of defect that surfaces weeks into production under flaky networks or a bad script.
|
||||
|
||||
---
|
||||
|
||||
## Findings — Stability
|
||||
|
||||
### S1. [High] Reconnect abandons the previous protocol client undisposed, and its stale `ConnectionLost` handler can flap a healthy connection
|
||||
`OpcUaDataConnection.ConnectAsync` (`src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/OpcUaDataConnection.cs:99-101`) does:
|
||||
|
||||
```csharp
|
||||
_client = _clientFactory.Create();
|
||||
_client.ConnectionLost += OnClientConnectionLost;
|
||||
await _client.ConnectAsync(...);
|
||||
```
|
||||
|
||||
The **previous** `_client` is neither `DisposeAsync`'d nor detached. `DataConnectionActor`'s reconnect loop calls `_adapter.ConnectAsync` repeatedly on the *same adapter* (`DataConnectionActor.cs:537-547`); only the failover branch replaces the adapter. `MxGatewayDataConnection.ConnectAsync` has the same pattern (`MxGatewayDataConnection.cs:70-88`) and additionally overwrites `_eventLoopCts` without cancelling the previous event loop.
|
||||
|
||||
Failure scenarios:
|
||||
1. **Session/socket leak per disconnect cycle.** After a keep-alive-detected drop, the old `RealOpcUaClient` still holds its `ISession` + subscription (`RealOpcUaClient.cs:133-160`); `Session.CloseAsync` only runs from `DisconnectAsync` (`RealOpcUaClient.cs:392-396`), which nothing calls on this path. A site whose PLC link flaps hourly accumulates orphaned SDK session objects, keep-alive timers, and sockets indefinitely.
|
||||
2. **Spurious disconnect of a healthy connection.** If the drop was detected *reactively* (a `ReadAsync` failure → `RaiseDisconnected`, `OpcUaDataConnection.cs:222-227`), the old client's own once-only guard (`_connectionLostFired`) has **not** fired. After a successful reconnect resets the adapter's `_disconnectFired` (`OpcUaDataConnection.cs:104`), the abandoned session's keep-alive eventually fails, fires the *old* client's `ConnectionLost`, which is still wired to `OnClientConnectionLost` → `RaiseDisconnected()` → `AdapterDisconnected` → `BecomeReconnecting()` on a perfectly healthy new session. Each such cycle abandons another client — self-sustaining reconnect churn.
|
||||
|
||||
Fix shape: in both adapters' `ConnectAsync`, detach + dispose the old client before creating the new one (mirroring what the actor's failover branch already does at the adapter level, `DataConnectionActor.cs:421-431`).
|
||||
|
||||
### S2. [High] Script execution timeout is cooperative-only; hung scripts permanently consume the bounded script-thread pool
|
||||
`ScriptExecutionActor` runs the body on the dedicated `ScriptExecutionScheduler` (default **8 threads**, `SiteRuntimeOptions.cs:46`) and enforces the timeout via `CancellationTokenSource(timeout)` passed to `compiledScript.RunAsync` (`ScriptExecutionActor.cs:129, 246`). Roslyn only observes the token between statements; a script blocked in synchronous I/O (`Database.Connection` ADO.NET call, a socket in a wedged state) or a CPU loop never observes cancellation. The `OperationCanceledException` path replies/logs, but the *thread* is only released when the blocking call itself returns. `ScriptExecutionScheduler.WorkerLoop` (`ScriptExecutionScheduler.cs:71-84`) has no watchdog, no thread replacement, no "N tasks stuck > timeout" metric.
|
||||
|
||||
Failure scenario: a site DB connection black-holes (firewall drops packets, no RST). Eight interval-triggered scripts each block ~30+ min in a synchronous read. All subsequent script *and alarm on-trigger* executions site-wide queue in `_queue` forever — alarms still transition (evaluation is on actor threads) but no on-trigger scripts run, and nothing on the health dashboard says why. The spec's claim that "Exceeding the timeout cancels the script" (`Component-SiteRuntime.md:411`) overstates what is enforced.
|
||||
|
||||
Mitigations worth considering: a scheduler gauge (queue depth + busy-thread age) surfaced through `ISiteHealthCollector`; per-script-thread stuck detection that logs the script name; ADO.NET command timeouts injected into `Database.Connection`.
|
||||
|
||||
### S3. [High] Spec drift: site-side script compile failure does not reject the deployment — central is told Success with the script silently missing
|
||||
`Component-SiteRuntime.md:474-477`: "If script compilation fails when a deployment is received, the entire deployment for that instance is **rejected**. No partial state is applied. The failure is reported back to central as a failed deployment."
|
||||
|
||||
Implementation: `InstanceActor.CreateChildActors` logs and `continue`s past a failed compile (`InstanceActor.cs:1362-1369`; same for alarm on-trigger scripts at 1418-1423 and trigger expressions at 1524-1531). The instance starts with the failing script absent; `DeploymentManagerActor.HandleDeployPersistenceResult` reports `DeploymentStatus.Success` purely on SQLite persistence (`DeploymentManagerActor.cs:596-610`). The XML doc on `CreateChildActors` even states the contradiction: "Compilation errors reject entire instance deployment (logged but actor still starts)" (`InstanceActor.cs:1339`).
|
||||
|
||||
The design-time gate in Template Engine makes this rare, but the spec explicitly calls the site the last line ("Site-side compilation failures indicate an unexpected issue") — and version skew between central's Roslyn surface and the site's is exactly the case where the gates disagree. Today the operator sees a green deploy and a dead script discoverable only in the site event log.
|
||||
|
||||
### S4. [Medium] Instance actor never retries a failed tag subscription; a subscribe/create ordering race strands attributes at Uncertain forever
|
||||
`InstanceActor` registers `Receive<SubscribeTagsResponse>(_ => { })` — an explicit no-op (`InstanceActor.cs:207`). If the DCL replies `Success=false` there is **no retry and no logging**. Two ways to get there:
|
||||
|
||||
- `DataConnectionManagerActor.HandleRoute` replies "Unknown connection" when the `SubscribeTagsRequest` arrives before the `CreateConnectionCommand` has been processed (`DataConnectionManagerActor.cs:104-115`). The Deployment Manager Tells `CreateConnectionCommand` and then creates the Instance Actor (`DeploymentManagerActor.cs:543-546`), whose `PreStart` Tells `SubscribeTagsRequest` — but the two messages come from **different senders**, so Akka's per-sender-pair ordering guarantee does not formally order them at the manager's mailbox. If the race fires, the instance's data-sourced attributes stay Uncertain until redeploy/restart.
|
||||
- The connection-level-failure reply ("connection unavailable — will re-subscribe on reconnect", `DataConnectionActor.cs:1009-1012`) is self-healing because `_subscriptionsByInstance` was already updated and `ReSubscribeAll` re-derives from it — that case is fine. The unknown-connection case leaves **no state anywhere**.
|
||||
|
||||
Contrast: `NativeAlarmActor` retries a failed subscribe on a timer (`NativeAlarmActor.cs:269-286`). The tag path deserves the same treatment — at minimum, log the failure and schedule a bounded re-subscribe.
|
||||
|
||||
Related sub-gap: `NativeAlarmActor`'s retry only arms on a *failure response*. A subscribe whose response is **lost** (DCL manager restarted between request and reply; message dead-lettered) never retries — `SendSubscribe` is a bare `Tell` with no response timeout (`NativeAlarmActor.cs:123-131`). Rehydrated SQLite state masks the gap in the debug view while the mirror is actually dead.
|
||||
|
||||
### S5. [Medium] Deployment Manager startup aborts silently on a SQLite read failure — site runs zero instances with no retry
|
||||
`HandleStartupConfigsLoaded`: on error, `_logger.LogError(...); return;` (`DeploymentManagerActor.cs:291-297`). One transient SQLite failure (locked file during a crash-recovery race, disk hiccup) at singleton start/failover leaves the active node with **no Instance Actors** until a human restarts the process. There is no retry timer, no health flag beyond the log line (health instance counts will read 0 deployed, which at least shows on the dashboard, but nothing distinguishes "genuinely nothing deployed" from "startup load failed"). A bounded retry (the same `Timers.StartSingleTimer` idiom used two methods later) would close this.
|
||||
|
||||
### S6. [Medium] A throwing `InstanceActor` constructor leaves a dead ref in `_instanceActors` and still reports the deployment Success
|
||||
`InstanceActor`'s constructor deserializes the config JSON (`InstanceActor.cs:138`) and builds the resolved-attribute index; a malformed/oversized config throws → `ActorInitializationException` → Deployment Manager's decider Stops the child (`DeploymentManagerActor.cs:268-274`). But:
|
||||
- `CreateInstanceActor` already stored the ref in `_instanceActors` (`DeploymentManagerActor.cs:1628-1629`) and nothing watches for this failure mode (the `Context.Watch` only happens on the redeploy path), so the map holds a dead ref — debug view, inbound-API routes, etc. dead-letter with 30s Ask timeouts instead of "not found".
|
||||
- `ApplyDeployment`'s persistence continues and `HandleDeployPersistenceResult` reports `Success` (`DeploymentManagerActor.cs:596-610`) — central shows a healthy deployment for an instance that is not running.
|
||||
|
||||
A `Context.Watch` on every created child (removing from `_instanceActors` on unexpected `Terminated`) would fix both the stale map and give a hook to fail the pending deployment.
|
||||
|
||||
### S7. [Medium] Subscribe background task reads the mutable `_adapter` field off-thread, violating its own confinement contract
|
||||
`HandleSubscribe`'s `Task.Run` body awaits `_adapter.SubscribeAsync(...)` per tag and calls `SeedTagsAsync(_adapter, ...)` (`DataConnectionActor.cs:707-758`) — reading the actor field from a thread-pool thread, despite the comment eight lines earlier: "The background task below must NOT read or mutate actor state — these partitioned lists are the only state it sees" (`DataConnectionActor.cs:689-690`). If a failover swaps `_adapter` mid-loop (`CountFailureAndMaybeFailover`, `DataConnectionActor.cs:610-635`), later iterations subscribe on the **new** adapter while stamping the **old** generation — the resulting values are dropped by the generation guard and the monitored items are orphaned on the new adapter (never in `_subscriptionIds` for that generation… actually worse: `SubscribeCompleted` will store them against current state, mixing generations). `ReSubscribeAll` (`DataConnectionActor.cs:1596`) and `HandleWriteBatch`'s local async function (`DataConnectionActor.cs:1161-1201`) have the same field-read pattern. Capture the adapter into a local before spawning the task (as the re-seed block at `DataConnectionActor.cs:1620` correctly does with `reseedAdapter`).
|
||||
|
||||
### S8. [Low] Site SQLite: connection-per-call, no WAL / busy-timeout tuning, many concurrent writers
|
||||
`SiteStorageService` opens a fresh `SqliteConnection` per operation with a bare `Data Source=` string (`SiteStorageService.cs:34-44, 163-166`) — no `journal_mode=WAL`, relying on Microsoft.Data.Sqlite's default command-timeout-as-busy-handler. Concurrent writers against the same file include: deploy persistence, static-override fire-and-forget writes (`InstanceActor.cs:433-441`), per-transition native-alarm upserts (`NativeAlarmActor.cs:420-428`), replication applies (`SiteReplicationActor.cs:182-216`), and reconciliation. Under a native-alarm storm plus a deploy, rollback-journal locking serializes everything and a slow write can push others toward `SQLITE_BUSY`. WAL + explicit busy_timeout is a one-line hardening. (Contrast: `SiteEventLogger` got the full treatment — single owned connection, lock, bounded write queue.)
|
||||
|
||||
### S9. [Low] Fire-and-forget adapter disposal in `PostStop`
|
||||
`DataConnectionActor.PostStop` discards the `DisposeAsync` task (`DataConnectionActor.cs:218-221`); a hanging/faulting dispose (OPC UA `CloseAsync` against a dead server can block for the operation timeout) is unobserved. Acceptable trade-off (documented elsewhere in the file), but pair it with a `ContinueWith(OnlyOnFaulted, log)` like the other fire-and-forget paths in this codebase do.
|
||||
|
||||
### S10. [Low] Alarm condition filter is last-subscriber-wins across co-subscribers of one source
|
||||
`_alarmSourceFilter[request.SourceReference] = request.ConditionFilter` overwrites unconditionally (`DataConnectionActor.cs:104-108, 1763-1766`); two instances binding the same source with different filters silently re-gate each other's transitions. The field's XML doc honestly declares the sharp edge, but nothing *validates* agreement or logs the overwrite — a mismatch is undiagnosable in the field. Log a warning when the incoming filter differs from the stored one.
|
||||
|
||||
---
|
||||
|
||||
## Findings — Performance
|
||||
|
||||
### P1. [High] Trigger expressions block actor threads on the default dispatcher — per attribute change
|
||||
`ScriptActor.EvaluateExpressionTrigger` runs the compiled Roslyn expression with `.GetAwaiter().GetResult()` under a 2-second CTS (`ScriptActor.cs:283-287`); `AlarmActor.EvaluateExpression` is identical (`AlarmActor.cs:503-506`). Both execute **on the actor's default-dispatcher thread**, and both run on *every* `AttributeValueChanged` the actor receives (Expression triggers have no monitored-attribute gate — `ScriptActor.cs:255-258`, `AlarmActor.cs:215-218`). A site with a handful of Expression-triggered scripts/alarms on a chatty instance (e.g., 50 tags at 1 Hz) synchronously executes 100s of Roslyn script invocations per second on dispatcher threads; one pathological expression (2 s timeout, treated-as-false) stalls its actor's mailbox and steals a dispatcher thread for the duration. The `ScriptExecutionScheduler` exists precisely to keep script code off these threads — trigger expressions bypass it. The in-code justification ("trusted Design-role users, compile-checked") addresses malice, not cost.
|
||||
|
||||
### P2. [Medium] Attribute-change fan-out is broadcast to every child, filtering in the receiver
|
||||
`InstanceActor.PublishAndNotifyChildren` Tells **every** Script Actor and **every** Alarm Actor for every attribute change (`InstanceActor.cs:1184-1194`); `Component-SiteRuntime.md:191` describes children subscribing "for the specific monitored attribute". An instance with 30 scripts + 20 alarms and 100 Hz aggregate tag updates generates 5,000 child messages/sec, ~98% of which are discarded by the child's trigger check (`ScriptActor.cs:229-233`, `AlarmActor.cs:219-222`) — pure mailbox/dispatch overhead, plus each `ScriptActor` maintains an `_attributeSnapshot` copy of every value (`ScriptActor.cs:227`). A per-attribute subscription map on the Instance Actor (attributeName → interested children; Expression-trigger children in an "all" bucket) would cut this by an order of magnitude on realistic configs.
|
||||
|
||||
### P3. [Medium] DCL tag-value fan-out scans all instances per update
|
||||
`HandleTagValueReceived` iterates `_subscriptionsByInstance` (every instance) with a per-instance `HashSet.Contains` for every incoming tag value (`DataConnectionActor.cs:1706-1716`). The actor already maintains `_tagSubscriberCount` (tag → count) for O(1) unsubscribe; the matching tag → *instances* reverse index is missing. With 500 instances on one connection at high sample rates this is the hottest loop in the DCL. Also `_healthCollector.UpdateTagQuality` is invoked on **every** update (`DataConnectionActor.cs:1737`) — cheap if the collector is a plain field write, but worth confirming it isn't lock-contended.
|
||||
|
||||
### P4. [Low] Per-transition SQLite upserts on the native-alarm mirror
|
||||
`NativeAlarmActor.PersistUpsert` issues one async SQLite write per accepted transition (`NativeAlarmActor.cs:420-428`), each opening a fresh connection (S8). An A&C alarm storm (hundreds of transitions/sec across sources) turns into hundreds of single-row upsert connections/sec. Bounded by `MirroredAlarmCapPerSource` (1000) per source in state size but not in write rate. Batching/coalescing (persist latest per SourceReference on a short timer) would be a proportionate hardening if storms are expected.
|
||||
|
||||
### P5. [Low] Every script attribute read is an actor Ask round-trip
|
||||
`ScriptRuntimeContext` Asks the Instance Actor per `GetAttribute` (`ScriptRuntimeContext.cs:375`) — a temp actor allocation + two mailbox hops per read. Correct (serialized through the source of truth) and consistent with the spec, but a script reading 50 attributes in a loop pays 50 round-trips; a `GetAttributes` batch message (the DeploymentManager routing path already fans out N Asks — `DeploymentManagerActor.cs:1360-1367` — which would also benefit) is an easy win if profiling ever flags it.
|
||||
|
||||
### P6. [Low] Roslyn compiles run inside `InstanceActor.PreStart` on the default dispatcher during staggered startup
|
||||
Each instance compiles all its scripts + alarm scripts + trigger expressions synchronously in `CreateChildActors` (called from `PreStart`, `InstanceActor.cs:250-255, 1348+`). The startup stagger (20 instances / 100 ms, `SiteRuntimeOptions.cs:13-19`) is tuned for OPC UA connection storms, not compile cost — 500 instances × several scripts each at ~50–300 ms/compile will occupy default-dispatcher threads for minutes after failover. The Deployment Manager already moved *shared*-script compilation off-thread (`DeploymentManagerActor.cs:1168-1191`); per-instance compiles did not get the same treatment. Failover time-to-recover is the metric at risk.
|
||||
|
||||
---
|
||||
|
||||
## Findings — Conventions
|
||||
|
||||
### C1. [Positive] Akka.NET discipline is exemplary
|
||||
Worth stating because it is the dominant impression: Become/Stash exactly as the spec draws it (`DataConnectionActor.cs:232-533`); Tell on hot paths, Ask only at boundaries (`Component-SiteRuntime.md:430-442` ↔ implementation); `Sender`/`Self` captured before every continuation (e.g., `DeploymentManagerActor.cs:464-470`, `DataConnectionActor.cs:149-153`); all async results marshalled via `PipeTo`; supervision matches the spec table precisely — DM `OneForOne` Resume/Stop-on-init (`DeploymentManagerActor.cs:263-279`), Instance Resume (`InstanceActor.cs:281-293`), Script/Alarm coordinators Stop their execution children (`ScriptActor.cs:180-192`, `AlarmActor.cs:192-204`). The redeploy Terminated-watch buffering with the name-shadow map and its disable/delete supersede handling (`DeploymentManagerActor.cs:60-90, 389-435, 641-698, 762-831`) is genuinely hard concurrency done correctly. The in-flight subscribe/orphan-release guards in the DCL (`DataConnectionActor.cs:77-90, 836-985, 1795-1858`) close race classes most codebases never notice.
|
||||
|
||||
### C2. [Low] Stale comment: SiteEventLogger claims its channel is unbounded
|
||||
`SiteEventLogger.cs:216-218` — "The channel is unbounded, so the only way TryWrite fails is…" — the channel became bounded (`Channel.CreateBounded`, `SiteEventLogger.cs:74-88`), and `TryWrite` on a bounded DropOldest channel still only fails when completed, so the *behavior* is right but the stated reason is wrong. Trivial fix; flagged because this file's comments are otherwise load-bearing.
|
||||
|
||||
### C3. [Low] Spec is internally inconsistent on the reconnect interval
|
||||
`Component-DataConnectionLayer.md:303` says "The retry interval is defined **per data connection**", but the same document's Shared Settings table (`Component-DataConnectionLayer.md:170-177`) and the implementation (`DataConnectionOptions.ReconnectInterval`, one value for all connections — `DataConnectionActor.cs:463`) make it global. Fix the sentence at :303 (per the CLAUDE.md rule that the design doc travels with the code).
|
||||
|
||||
### C4. [Low] Quality-only change on disconnect is not delivered to Alarm/Script actors — deliberate, but diverges from the spec's wording
|
||||
`HandleConnectionQualityChanged` updates qualities and the stream but explicitly does **not** notify children ("firing scripts/alarms would cause spurious evaluations", `InstanceActor.cs:924-956`). The DCL spec promises "Instance Actors and their downstream consumers (**alarms, scripts checking quality**) see the staleness immediately" (`Component-DataConnectionLayer.md:302`). Practical consequence: an Active alarm computed from a now-dead connection stays Active (arguably correct — last-known-state), and a Conditional/Expression script never learns quality went Bad. The implementation choice is defensible; the spec sentence should be qualified so the divergence is a decision, not drift.
|
||||
|
||||
### C5. [Low] Shared scripts execute without the caller's scope and without a `Scope` assignment
|
||||
`SharedScriptLibrary.ExecuteAsync` builds `ScriptGlobals` without setting `Scope` (`SharedScriptLibrary.cs:98-103`), so a shared script always runs `ScriptScope.Root` (`ScriptCompilationService.cs:200-201`) even when invoked from a composed-module script. If a shared script uses `Attributes[...]`, it addresses **root** attributes regardless of caller scope. Possibly intended (shared scripts are instance-agnostic), but it is undocumented in `Component-SiteRuntime.md`'s Shared Script Library section and is a surprise waiting for a template author.
|
||||
|
||||
### C6. [Low] `EvaluateCondition` fallback compares `value.ToString()` to `Threshold.ToString()` with current-culture double formatting
|
||||
`ScriptActor.cs:474-477`: the non-numeric fallback stringifies the double threshold with the host culture (the numeric path was carefully invariant-culture'd; the fallback wasn't). A `"1.5"` threshold on a de-DE host renders `"1,5"`. Edge-case only (non-parseable values), but inconsistent with the deliberate invariant-culture work two lines up.
|
||||
|
||||
---
|
||||
|
||||
## Underdeveloped Areas
|
||||
|
||||
1. **Cert-trust convergence after node downtime.** The trust broadcast reaches only currently-Up nodes (`DeploymentManagerActor.BroadcastToSiteCertStores`, :962-1007); a node that is down during the broadcast rejoins with a divergent PKI store and there is no reconcile-on-join, no periodic store diff, and `ListServerCertsCommand` reads only the singleton's own node (:936-953) so the divergence is invisible from central. Central persistence of trust decisions is the documented follow-up (`Component-SiteRuntime.md:125`); until then a join-time pull from the peer's `CertStoreActor` would close the failover hole.
|
||||
2. **Dead option**: `SiteRuntimeOptions.ConfigFetchRetryCount` — "Reserved — consumed by the standby replication fetch in a later task; not yet wired" (`SiteRuntimeOptions.cs:63-67`). The standby fetch is genuinely single-attempt (`SiteReplicationActor.cs:178-216`), leaning entirely on reconciliation as backstop.
|
||||
3. **No aggregated live alarm stream** — the M7 operator Alarm Summary polls per-instance `DebugViewSnapshot` Asks; documented as a follow-up in CLAUDE.md/M7 plan. Fine for now; scale limit is the SemaphoreSlim-capped fan-out on the central side.
|
||||
4. **Native alarm rehydration loses display metadata** — only `condition_json` + timestamps persist (`SiteStorageService` `native_alarm_state`, :117-124), so a rehydrated condition renders with empty type/category/message until the first source snapshot (`NativeAlarmActor.cs:161-167`). Acknowledged in comments; worth a line in the component doc's operator-facing section.
|
||||
5. **No stuck-thread/queue-depth observability for the script scheduler** (see S2) — `ScriptExecutionScheduler` exposes nothing to `ISiteHealthCollector`.
|
||||
6. **Tag-subscription failure path has no retry or site-event entry** (see S4) — the spec's Tag Path Resolution retry covers resolution failures inside the DCL, but the instance-side response handling is a hole.
|
||||
7. **Tests**: coverage breadth is very good (redeploy races, waiter races, native-alarm swap/cap, seed reads, scheduler, trust model — 17 actor test files, 17 script test files, dedicated DCL alarm/browse/verify suites, plus `OpcUaAlarmLiveSmokeTests`). Gaps that match the findings above: no test exercises reconnect-then-stale-`ConnectionLost` (S1), compile-failure-deploy-status (S3), the unknown-connection subscribe race (S4), or sustained-churn fan-out cost (the `PerformanceTests` project was not examined for these paths).
|
||||
|
||||
---
|
||||
|
||||
## Prioritized Recommendations
|
||||
|
||||
1. **(S1)** Dispose + detach the previous client in `OpcUaDataConnection.ConnectAsync` / `MxGatewayDataConnection.ConnectAsync` (and cancel the old MxGateway event loop). This is the single highest-value fix: it removes both a per-flap resource leak and a healthy-connection flapping mechanism.
|
||||
2. **(S3)** Make site-side compile failure fail the deployment, per spec: have `CreateChildActors` report compile outcomes to the Deployment Manager (or pre-compile in `ApplyDeployment` before actor creation, which also fixes P6 by moving compiles off the instance-start path) and return `DeploymentStatus.Failed` with the errors.
|
||||
3. **(S2)** Add stuck-execution observability (scheduler queue depth + oldest-busy-thread age as health metrics; log the script name on timeout-with-thread-still-busy), and inject command timeouts into script-facing DB connections.
|
||||
4. **(P1)** Move trigger-expression evaluation off the actor thread — evaluate on the `ScriptExecutionScheduler` and post the boolean result back to the actor (edge-state stays actor-confined), or precompile expressions to delegates rather than re-running Roslyn scripts.
|
||||
5. **(S4)** Handle `SubscribeTagsResponse.Success == false` in `InstanceActor` with a logged, bounded re-subscribe timer (mirror `NativeAlarmActor`'s retry); add a response-timeout retry to `NativeAlarmActor.SendSubscribe`.
|
||||
6. **(S5/S6)** Retry the startup config load on failure; `Context.Watch` all Instance Actors so an init-failure removes the dead ref and fails the in-flight deployment.
|
||||
7. **(P2/P3)** Introduce per-attribute child-subscription routing in `InstanceActor` and a tag→instances reverse index in `DataConnectionActor` before scaling to the 500-instance design point.
|
||||
8. **(S8)** Enable WAL + explicit busy timeout on the site SQLite stores.
|
||||
9. **(S7)** Capture `_adapter` into locals before every background task in `DataConnectionActor` (three sites).
|
||||
10. **(C3/C4/C5)** Doc fixes: reconcile the reconnect-interval wording, qualify the disconnect-quality-visibility promise, and document shared-script scope semantics — per the repo rule that design docs and code travel together.
|
||||
@@ -0,0 +1,180 @@
|
||||
# Architecture Review 04 — Central Data Layer & Audit/KPI Backbone
|
||||
|
||||
**Scope reviewed:**
|
||||
|
||||
- `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase` — EF Core mappings, repositories, migrations, `IAuditService`, partition maintenance
|
||||
- `src/ZB.MOM.WW.ScadaBridge.Commons` — as the data-shape layer (entities, repository interfaces, audit enums/types, KPI types)
|
||||
- `src/ZB.MOM.WW.ScadaBridge.AuditLog` — central ingest/purge/reconciliation actors, `CentralAuditWriter`, site SQLite writer + telemetry
|
||||
- `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox` — `NotificationOutboxActor`, repository, delivery adapters, KPI source
|
||||
- `src/ZB.MOM.WW.ScadaBridge.SiteCallAudit` — `SiteCallAuditActor`, monotonic upsert repository, reconciliation, relay
|
||||
- `src/ZB.MOM.WW.ScadaBridge.KpiHistory` — `KpiHistoryRecorderActor`, `KpiHistoryRepository`, bucketer
|
||||
- Design docs: `docs/requirements/Component-ConfigurationDatabase.md`, `Component-AuditLog.md`, `Component-NotificationOutbox.md`, `Component-SiteCallAudit.md`, `Component-KpiHistory.md`, `Component-Commons.md`
|
||||
- Test projects: `AuditLog.Tests`, `NotificationOutbox.Tests`, `SiteCallAudit.Tests`, `KpiHistory.Tests`, `ConfigurationDatabase.Tests`
|
||||
|
||||
## Maturity Verdict
|
||||
|
||||
This is an unusually mature data layer for a system at this stage: idempotent ingest is implemented correctly and defensively everywhere (duplicate-key race handling in all three insert-if-not-exists paths), the combined cached-telemetry dual-write really is transactional (verified: `AuditLogIngestActor.OnCachedTelemetryAsync` shares one scoped `DbContext` across both repositories under `BeginTransactionAsync`), UTC discipline is enforced with converters and `SpecifyKind` re-tagging at every ADO.NET boundary, actor/DB interaction follows a consistent scope-per-message + PipeTo pattern with in-flight guards, and MSSQL-backed integration tests exist for the hard parts (partition purge, execution-tree walks, outage reconciliation). The weaknesses are concentrated in **scale-dependent operational behavior** rather than logic: the partition purge's full-table unique-index rebuild will start failing on default command timeouts as the table grows, several KPI predicates have no supporting index and full-scan year-sized tables every 60 seconds, the execution-tree query scans the entire AuditLog per invocation, the spec'd site SQLite 7-day retention purge simply does not exist, the append-only DB-role story is DDL-only (one connection principal runs both writer and maintenance paths), and the persisted `backlogTotal` KPI trend is hardwired to zero. None of these bites at demo scale; several will bite within months of production volume.
|
||||
|
||||
---
|
||||
|
||||
## Findings — Stability
|
||||
|
||||
### [High] Site SQLite 7-day retention purge is specified but not implemented — unbounded site DB growth
|
||||
|
||||
`Component-AuditLog.md:412-414` specifies: "Sites: daily site job; default 7-day retention (configurable, min 1, max 90). Respects the hard `ForwardState` invariant." No such job exists. There is no `DELETE` against `audit_event` / `audit_forward_state` anywhere in `src/` — the only destructive statement is the one-time legacy-table `DROP TABLE IF EXISTS AuditLog` at startup (`src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriter.cs:198`). The writer's own comments treat the purge as future work: `SqliteAuditWriter.cs:145` ("a future `PRAGMA incremental_vacuum` shrink the file after the 7-day retention purge") and `:33` claims the store is "recreated per deployment", which is not a retention mechanism.
|
||||
|
||||
**Failure scenario:** a long-lived site node accumulates every Forwarded/Reconciled audit row forever. At even modest script activity (10 events/s ≈ 860k rows/day) the SQLite file grows by GBs per month; `GetBacklogStatsAsync`'s `COUNT(*)` stays cheap (it filters `Pending`), so nothing surfaces on the health dashboard — the disk just fills. The `auto_vacuum = INCREMENTAL` pragma (`SqliteAuditWriter.cs:149`) is set up for a purge that never runs.
|
||||
|
||||
### [High] `SwitchOutPartitionAsync` — full-table unique-index rebuild inside one transaction, no command timeout
|
||||
|
||||
`src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/AuditLogRepository.cs:257-332`: the monthly partition switch drops `UX_AuditLog_EventId`, switches, then `CREATE UNIQUE NONCLUSTERED INDEX UX_AuditLog_EventId ON dbo.AuditLog (EventId)` — a scan+sort over **every remaining row of the table** (up to 365 days of audit events), all inside a single transaction executed via `ExecuteSqlRawAsync`.
|
||||
|
||||
Three compounding problems:
|
||||
|
||||
1. **Default 30s command timeout.** No `CommandTimeout` is configured anywhere (`ServiceCollectionExtensions.cs:29-34` sets only `UseSqlServer(connectionString)`; repo-wide grep confirms no `CommandTimeout` on the data layer). Once the table is large enough that the index rebuild exceeds 30 seconds, the batch times out, the CATCH block rolls back and re-`THROW`s, `AuditLogPurgeActor.OnTickAsync` logs and moves on (`AuditLogPurgeActor.cs:198-206`) — and the same failure repeats every daily tick forever. Retention silently stops; the table grows past 365 days; the rebuild gets *slower* each day. This is a self-reinforcing failure with no alerting beyond an error log line.
|
||||
2. **Write blockage.** The transaction holds Sch-M from the DROP INDEX through the CREATE INDEX; all ingest (`InsertIfNotExistsAsync`, dual-writes, direct writes) blocks for the full rebuild duration. The actors tolerate this (rows stay Pending at sites), but central direct-write paths (Inbound API middleware, dispatcher audit emission) will eat the latency or the swallow-and-count path.
|
||||
3. **Uniqueness window.** While `UX_AuditLog_EventId` is absent, the `IF NOT EXISTS` probe becomes an unindexed scan and single-column EventId uniqueness is enforced only via the composite PK. In practice a retried event carries the same `OccurredAtUtc` so the PK still dedupes; the residual risk is small but the per-insert scan cost during the window is not.
|
||||
|
||||
**Recommendation:** set a long `CommandTimeout` for the maintenance path (or run the rebuild with `ONLINE = ON` where edition allows, outside the switch transaction), and monitor purge success as a health metric, not just a log line. Long term, consider making `UX_AuditLog_EventId` unnecessary (see the clustered-key finding under Performance).
|
||||
|
||||
### [High] Append-only DB-role enforcement is DDL-only — one principal runs writer and maintenance paths
|
||||
|
||||
The migrations create `scadabridge_audit_writer` (INSERT+SELECT, DENY UPDATE/DELETE) and `scadabridge_audit_purger` (SELECT + ALTER ON SCHEMA::dbo) — `20260520142214_AddAuditLogTable.cs:146-158`, re-granted in `20260602174346_CollapseAuditLogToCanonical.cs:212-223`. But the application registers **one** `ScadaBridgeDbContext` with **one** connection string (`ServiceCollectionExtensions.cs:29-45`); the same principal executes `InsertIfNotExistsAsync` (writer path), `PurgeChannelOlderThanAsync`'s `DELETE TOP` (`AuditLogRepository.cs:409-410`), `BackfillSourceNodeAsync`'s `UPDATE TOP` (`AuditLogRepository.cs:851-852`), and the switch-out DDL. Consequences:
|
||||
|
||||
- If the runtime login were actually placed in `scadabridge_audit_writer`, the DENY would break the purge/backfill on the same connection. So in any real deployment the login must hold INSERT+UPDATE+DELETE+DDL — meaning the append-only guarantee is enforced **only** by the CI grep guard (`tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/AuditLogAppendOnlyGuardTests.cs`), not by the database.
|
||||
- The purger role as granted cannot even perform the purge: `CREATE TABLE` (staging) requires the database-level `CREATE TABLE` permission, which `ALTER ON SCHEMA::dbo` alone does not confer.
|
||||
|
||||
`Component-AuditLog.md:417-424` presents the role separation as the enforcement mechanism ("The application accesses AuditLog via a dedicated DB role"). Spec and code have drifted: either wire a second connection string/principal for the maintenance path (and fix the purger grants), or re-document enforcement honestly as "CI guard + code review, roles provided for DBAs who choose to segregate principals."
|
||||
|
||||
### [Medium] SiteCalls monotonic upsert drops mid-lifecycle progress (RetryCount / LastError frozen at rank)
|
||||
|
||||
`SiteCallAuditRepository.cs:30-40` ranks `Attempted == Skipped == 2` and the UPDATE fires only when `incomingRank > storedRank` (`:128-138`). A cached call retrying 20 times emits 20 `Attempted` telemetry packets with increasing `RetryCount` and fresher `LastError`/`HttpStatus` — every one after the first is a **no-op** ("first-write-wins at each rank" per the comment). The Central UI Site Calls page therefore shows `RetryCount = 1` and the first error for the entire retry phase, until the terminal packet lands. The spec's intent (`Component-SiteCallAudit.md:76-78`, "upsert-on-newer-status... at-least-once and out-of-order telemetry are harmless") is about status regression, not about freezing progress fields. A safe refinement: within equal rank, apply the update when `incoming.UpdatedAtUtc > stored.UpdatedAtUtc` (still idempotent and regression-proof).
|
||||
|
||||
### [Medium] No EF Core connection resiliency (`EnableRetryOnFailure`)
|
||||
|
||||
`ServiceCollectionExtensions.cs:31` configures `UseSqlServer(connectionString)` with no execution strategy. The audit/dispatcher actors compensate with their own retry loops (rows stay Pending, next tick retries), so the backbone self-heals — but every *read-side* path (KPI requests, outbox queries, `Notify.Status` round-trips, execution-tree queries) surfaces transient faults (failover of the SQL instance, connection pool churn) directly to the caller as a failed response. One line of `sqlOptions.EnableRetryOnFailure(...)` would cover the fleet. (Note: retry strategies and user-initiated transactions interact — `OnCachedTelemetryAsync`'s `BeginTransactionAsync` would need `CreateExecutionStrategy` wrapping — so adopt deliberately.)
|
||||
|
||||
### [Medium] Operator Retry on a parked notification emits no audit row
|
||||
|
||||
`Component-NotificationOutbox.md:128`: "Operator Retry/Discard still mutates only the Notifications row, and each transition emits the corresponding `Notification.Attempt` / `Notification.Terminal` audit row." `DiscardAsync` does emit (`NotificationOutboxActor.cs:1021`), but `RetryAsync` (`NotificationOutboxActor.cs:951-977`) resets Parked→Pending with **no** audit emission. Forensically, a parked notification that an operator retried and that then delivered shows an unexplained `Parked → Attempted → Delivered` sequence with no record of who un-parked it or when. Same gap for `SiteCallAudit`'s retry/discard relay (site-side telemetry covers the state change but not the operator identity).
|
||||
|
||||
### [Medium] Reconciliation edge cases are detected but dead-ended
|
||||
|
||||
- **Single-timestamp cursor pin** (`SiteCallAuditActor.cs:636-654`): when > `ReconciliationBatchSize` rows share one `UpdatedAtUtc` the cursor cannot advance; the code correctly detects and logs it, but the backlog tail then **never** reconciles ("will not reconcile until those rows' timestamps differ") and nothing surfaces beyond a log Warning — no health metric, no stalled event (unlike the sibling `SiteAuditReconciliationActor`, which publishes `SiteAuditTelemetryStalledChanged`). The fix the log message itself names — a composite `(timestamp, id)` keyset in the pull contract — is deferred without a tracking artifact.
|
||||
- **Permanent row abandonment** (`SiteAuditReconciliationActor.cs:283-293`): after 5 failed inserts a row is dropped with `LogCritical` and the cursor advances. For a compliance-oriented audit log, "we permanently lost EventId X" deserves a durable record (e.g., a synthetic `Skipped` audit row or a health counter), not only a log line that rotates away.
|
||||
|
||||
### [Medium] `NotificationOutboxActor` ingest ack path can ack before the row is *committed* under a SQLite test provider mismatch — verify provider parity
|
||||
|
||||
`NotificationOutboxRepository.InsertIfNotExistsAsync` maintains two dialects (`NotificationOutboxRepository.cs:70-95` SQLite `INSERT OR IGNORE`, `:99-117` T-SQL `IF NOT EXISTS`). This is correct today but is production code carrying a test-only branch, and the two branches have subtly different concurrency semantics (the SQLite path can never observe the duplicate-key race the SQL Server path handles). The AuditLog and SiteCall repositories chose the opposite strategy (MSSQL-only SQL, MSSQL-backed test fixtures). Pick one convention; the divergence invites the next repository to guess.
|
||||
|
||||
### [Low] Purge actors' first tick waits a full interval
|
||||
|
||||
`AuditLogPurgeActor.PreStart` (`:95-100`), `SiteCallAuditActor.StartPurgeTimer` (`:380-386`), and `KpiHistoryRecorderActor` purge timer (`KpiHistoryRecorderActor.cs:124-127`) all use `initialDelay = interval` (24h). A central node that restarts daily (crash loop, scheduled recycling) would *never* purge. Low likelihood, cheap fix (short initial delay; purges are idempotent).
|
||||
|
||||
### [Low] `SqliteAuditWriter.Dispose` sync-over-async
|
||||
|
||||
`SqliteAuditWriter.cs:890-894` blocks on `DisposeAsync` via `Task.Run(...).GetAwaiter().GetResult()`. The mitigation (thread-pool hop) is correct and documented; flagged only because it remains a blocking dispose on a hot-path component.
|
||||
|
||||
---
|
||||
|
||||
## Findings — Performance
|
||||
|
||||
### [High] SiteCalls KPI predicates full-scan the table — no index on `TerminalAtUtc`
|
||||
|
||||
The "schema-honest" non-terminal predicate is `TerminalAtUtc IS NULL` (`SiteCallAuditRepository.cs:224-232`), used by `ComputeKpisAsync` (buffered `:243-244`, stuck `:259-260`, oldest `:262-268`), the per-site and per-node variants (`:286-307`, `:336-362`), and `PurgeTerminalAsync` (`:213-218`). But the table's only indexes are `(SourceSite, CreatedAtUtc)` and `(Status, UpdatedAtUtc)` (`SiteCallEntityTypeConfiguration.cs:79-87`) — **nothing covers `TerminalAtUtc`**. Every such count is a clustered scan of a table retained for 365 days (`SiteCallAuditOptions.RetentionDays = 365`, potentially millions of rows for busy sites). These queries run:
|
||||
|
||||
- every 60 s from `SiteCallAuditKpiSampleSource` (global + per-site + per-node = ~9 scanning queries per recorder tick),
|
||||
- on every Health-dashboard poll (10 s cadence per the UI design) via `SiteCallKpiRequest`.
|
||||
|
||||
**Failure scenario:** at ~2M rows, each scan is hundreds of ms of CPU/IO; the KPI backbone alone drives a near-continuous scan load on the central DB, and the KPI Ask timeouts start failing intermittently. **Fix:** a filtered index `CREATE INDEX IX_SiteCalls_NonTerminal ON SiteCalls (CreatedAtUtc) WHERE TerminalAtUtc IS NULL` (plus `SourceSite`/`SourceNode` includes) makes every one of these predicates a small seek, since the non-terminal population is the live queue, not the archive.
|
||||
|
||||
### [High] `GetExecutionTreeAsync` scans the entire AuditLog per invocation
|
||||
|
||||
`AuditLogRepository.cs:704-756`: the recursive CTE's `Edges` anchor is `SELECT DISTINCT ExecutionId, ParentExecutionId FROM dbo.AuditLog WHERE ExecutionId IS NOT NULL`. Neither `IX_AuditLog_Execution` nor `IX_AuditLog_ParentExecution` covers the *other* column, so satisfying the DISTINCT requires scanning an entire index (or the clustered index) across **all partitions, all 365 days** — on every `audit tree` CLI call and every UI execution-tree drilldown. The upward walk (`:652-677`) is fine (seeks on `IX_AuditLog_Execution`), and the STRING_AGG subqueries seek per node; it is the edge materialization that is O(table).
|
||||
|
||||
**Fix options:** (a) add `ParentExecutionId` as an INCLUDE on `IX_AuditLog_Execution` (index-only DISTINCT scan — better but still O(table)); (b) constrain the Edges CTE to a time window derived from the root's `FirstOccurredAtUtc` (execution trees span minutes, not years — a `OccurredAtUtc >= root - slack` predicate gets partition elimination); (c) seed the recursion by seeking children per level (`WHERE ParentExecutionId IN (…)`) instead of pre-materializing a global edge set.
|
||||
|
||||
### [Medium] AuditLog clustered key leads with a random GUID
|
||||
|
||||
`AuditLogEntityTypeConfiguration.cs:172` / migration `CollapseAuditLogToCanonical.cs:75`: clustered PK `(EventId, OccurredAtUtc)` with `EventId = Guid.NewGuid()`. Inserts land randomly across the current month's partition → page splits, fragmentation, and larger-than-necessary buffer pool footprint on the system's highest-volume table. The composite order also means the PK is useless for time-range queries (all served by secondary indexes). A clustered key of `(OccurredAtUtc, EventId)` would give append-locality and make `IX_AuditLog_OccurredAtUtc` redundant; `EventId` uniqueness would still need the non-aligned unique index (which exists anyway). This interacts with the switch-out dance either way; worth a deliberate benchmark before the table gets big — changing it later is a full rebuild.
|
||||
|
||||
### [Medium] KpiSample volume and unbatched purge
|
||||
|
||||
Per recorder tick (60 s), sample count ≈ NotificationOutbox(5 × (1 global + N sites + M nodes)) + SiteCallAudit(6 × (1 + N + M)) + AuditLog(3) + SiteHealth(12 × N). For 10 sites / 20 site-nodes this is ~500 rows/min ≈ **700k rows/day ≈ 60M+ rows at the 90-day default retention** — with a 5-column composite index (`IX_KpiSample_Series`) maintained on every insert. Meanwhile `PurgeOlderThanAsync` is a single unbatched `ExecuteDeleteAsync` (`KpiHistoryRepository.cs:65-71`): in steady state it deletes ~700k rows in **one transaction** daily — a lock/log spike on the same DB serving the operational tables; after any purge outage the catch-up delete is worse (and subject to the same default 30s command timeout as the partition switch). Batch the delete (`TOP (n)` loop, mirroring `PurgeChannelOlderThanAsync`), and consider whether per-node scope sampling every 60 s is worth 40% of the row volume.
|
||||
|
||||
### [Medium] Catch-all `(@p IS NULL OR col = @p)` predicates in SiteCalls query
|
||||
|
||||
`SiteCallAuditRepository.QueryAsync` (`:186-202`) builds one static SQL shape where every filter is `({param} IS NULL OR col = {param})`. This is the classic optional-parameter anti-pattern: SQL Server compiles one plan for all filter combinations, generally a scan ordered by `CreatedAtUtc DESC`. Fine while the table is small; on a year of rows, a filtered query (e.g. `Status = 'Parked'`) will scan rather than use `IX_SiteCalls_Status_Updated`. `OPTION (RECOMPILE)` or dynamic predicate composition (as `AuditLogRepository.QueryAsync` does with LINQ, `:117-199`) fixes it.
|
||||
|
||||
### [Medium] Notification outbox query page uses offset paging + double query + unsargable LIKE
|
||||
|
||||
`NotificationOutboxRepository.QueryAsync` (`:165-228`): `COUNT` + `Skip((page-1)*size)` — deep pages re-scan; `Subject.Contains(keyword)` translates to `LIKE '%kw%'` (unsargable). The sibling AuditLog and SiteCalls pages both use keyset paging. For a 365-day table this page will degrade first in the UI. Also `ComputeKpisAsync`/`ComputePerSiteKpisAsync`/`ComputePerNodeKpisAsync` issue 5-7 sequential round trips each (`:239-283`, `:286-334`, `:337-391`), and the per-site/per-node "oldest" computations materialize every non-terminal row into memory (`:312-318`, `:368-375`) — bounded by live-queue size in normal operation, unbounded during a delivery outage (exactly when the KPI page gets watched). A single `GROUP BY` with conditional aggregation would serve each snapshot in one query.
|
||||
|
||||
### [Low] Non-persisted `IngestedAtUtc` computed column re-parses `DetailsJson` per row read
|
||||
|
||||
`AuditLogEntityTypeConfiguration.cs:158-164`: every materialization of `AuditLogRow` (e.g. `QueryAsync` pages) evaluates `SWITCHOFFSET(CAST(JSON_VALUE(DetailsJson …)))` over an `nvarchar(max)` per row. Page-sized reads make this tolerable; just avoid ever using `IngestedAtUtc` in a predicate (it would force a full-table JSON parse).
|
||||
|
||||
### [Low] `MarkForwardedAsync` / `MarkReconciledAsync` build per-id IN lists
|
||||
|
||||
`SqliteAuditWriter.cs:644-664, 735-753`: parameter-per-id is safe at the default telemetry batch of 256 (`SiteAuditTelemetryOptions.BatchSize = 256`) but silently approaches SQLite's 999-parameter limit if an operator raises the batch size ~4×. Worth a guard or chunking.
|
||||
|
||||
---
|
||||
|
||||
## Findings — Conventions
|
||||
|
||||
### [Medium] Design docs have drifted from the code in several load-bearing places
|
||||
|
||||
- `Component-Commons.md:47` lists `AuditKind` with 10 values; the code has 14 (`Commons/Types/Enums/AuditKind.cs:10-26`, adds the four `SecuredWrite*` kinds). `:46` lists `AuditChannel` with 4 values; the code has 5 (`AuditChannel.cs:8-15`, adds `SecuredWrite`).
|
||||
- `Component-Commons.md:89` still says `NotificationType`: "Email — currently the only value; Teams and other channels are planned but not yet present" — `Sms` shipped 2026-06-19 and the Teams plan was dropped (per CLAUDE.md).
|
||||
- `Component-ConfigurationDatabase.md:58` describes the NotificationList `Type` discriminator as "`Email` / `Teams` / …".
|
||||
- `Component-ConfigurationDatabase.md:399` states AuditLog retention is a "single global value in v1, **no per-channel overrides**", directly contradicting the shipped `PerChannelRetentionDays` feature (and `Component-AuditLog.md:396-411`, which documents it).
|
||||
- `Component-SiteCallAudit.md:44-51` describes the `SiteCalls` schema with `Kind`/`TargetSummary`/`Provenance` columns and a `Pending/Retrying/...` status set; the shipped schema (`Component-ConfigurationDatabase.md:64` and `SiteCallEntityTypeConfiguration.cs`) uses `Channel`/`Target` and `AuditStatus`-derived strings with no instance/script provenance columns. The ConfigDb doc is current; the SiteCallAudit doc's own table section is stale.
|
||||
|
||||
Per the repo's own rule ("design doc and code travel together"), these should be reconciled in one pass.
|
||||
|
||||
### [Medium] `AuditLogRow` entity lives in ConfigurationDatabase, not Commons
|
||||
|
||||
`src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Entities/AuditLogRow.cs` breaks the "POCO entities in Commons / mappings in ConfigurationDatabase" convention that every sibling table follows (`SiteCall`, `Notification`, `KpiSample`, `AuditLogEntry` are all Commons POCOs). There is a defensible rationale — the *contract* type is the external `ZB.MOM.WW.Audit.AuditEvent` and `AuditLogRow` is a persistence-only projection — but the deviation is undocumented in `Component-Commons.md`/`Component-ConfigurationDatabase.md`, and `IAuditLogRepository` (in Commons) consequently traffics in `AuditEvent` while `ISiteCallAuditRepository` traffics in a Commons entity. Document the deviation or move the row shape.
|
||||
|
||||
### [Low] Mixed timestamp CLR types across sibling tables
|
||||
|
||||
`Notification` uses `DateTimeOffset` (`Commons/Entities/Notifications/Notification.cs:68-77`) while `SiteCall`, `KpiSample`, and `AuditLogRow` use UTC `DateTime`. Both satisfy the UTC convention, but the `DateTimeOffset` choice is what forced the awkward in-memory Min reductions and "converter makes SQL Min awkward" workarounds in `NotificationOutboxRepository` (`:262-274`, `:308-318`) that its `DateTime`-based siblings didn't need. New tables should standardize (the `DateTime`+`Utc`-converter pattern in `AuditLogEntityTypeConfiguration.cs:32-40` is the cleaner precedent).
|
||||
|
||||
### [Low] Uncharted KPI metric names are string literals while charted ones use the catalog
|
||||
|
||||
`NotificationOutboxKpiSampleSource.cs:40-43`: `"stuckCount"` / `"oldestPendingAgeSeconds"` are inline literals with a comment explaining the split, while charted metrics use `KpiMetrics.*` constants. Consistent cataloging (all persisted names in `KpiMetrics`) would remove a rename hazard on values that are, per the code's own comment, "persisted; do not rename."
|
||||
|
||||
### Positive observations (conventions done right)
|
||||
|
||||
- Options pattern with dedicated validators and clamped intervals everywhere (`SiteCallAuditOptions.ResolvedReconciliationInterval` guarding the Akka zero-interval spin footgun, `AuditLogOptionsValidator`, `KpiHistoryOptionsValidator`).
|
||||
- The `Sender`-capture-before-await and `Context.System.EventStream`-capture-before-await disciplines are applied consistently across all five actors — a subtle Akka correctness point handled uniformly.
|
||||
- Duplicate-key race hardening (2601/2627 swallow) is uniform across all three idempotent-insert repositories, with the check-then-act window explicitly documented at each site.
|
||||
- `AuditService` stages within the caller's unit of work exactly as the spec's transactional-audit guarantee requires, with cycle-tolerant serialization that can't roll back the business operation (`AuditService.cs:20-24, 71-85`).
|
||||
- The dual-write transaction (`AuditLogIngestActor.cs:262-311`) is genuinely atomic per entry with per-entry isolation, and `IngestedAtUtc` is stamped from one instant across both rows.
|
||||
|
||||
---
|
||||
|
||||
## Underdeveloped Areas
|
||||
|
||||
1. **`backlogTotal` KPI history series is permanently zero.** `AuditLogRepository.GetKpiSnapshotAsync` hardcodes `BacklogTotal: 0L` (`AuditLogRepository.cs:618`) — documented in the interface as "left at zero" — and the Central UI's live tile compensates by summing site backlog reports (`CentralUI/Services/AuditLogQueryService.cs:125-155`). But `AuditLogKpiSampleSource` persists the repo snapshot **directly** (`AuditLogKpiSampleSource.cs:69-85`), so every recorded `backlogTotal` sample is 0, and the Audit Log page's backlog trend chart (`AuditLogPage.razor.cs:297`) renders a flat zero line regardless of actual backlog. The one KPI whose *history* matters most during an outage post-mortem (how big did the backlog get, how fast did it drain) records nothing. Fix: inject the same health-aggregator read the query service uses into the sample source.
|
||||
2. **Site SQLite retention purge** — spec'd, commented as future, absent (see Stability [High] above).
|
||||
3. **Hash-chain tamper evidence (T1) and Parquet archival (T2)** — explicitly deferred to v1.x per `Component-AuditLog.md:433-437`; the `verify-chain` CLI is a documented no-op placeholder. Consistent with plan; no drift.
|
||||
4. **SecuredWrite audit rows leave `SourceNode` NULL** — documented as a logged follow-up (`Component-AuditLog.md:154-157`); also note `PerChannelRetentionDays` validation accepts `SecuredWrite` as a key (the enum has it) even though the design doc's config section (`:396-399`) enumerates only the four original channels — harmless, but the doc under-promises.
|
||||
5. **Reconciliation cursor keyset upgrade** — the `(timestamp, id)` composite cursor that would eliminate the single-timestamp pin is acknowledged in a log message (`SiteCallAuditActor.cs:640-654`) but not tracked anywhere; the pull contracts (`CachedCallReconcileRequest`, `PullAuditEvents`) would need an additive field.
|
||||
6. **Dispatcher throughput ceiling** — notification delivery is strictly sequential within a sweep (batch 100, 10 s interval, blocking SMTP/Twilio per row: `NotificationOutboxActor.RunDispatchPass:321-347`). A slow SMTP endpoint (5 s/attempt) caps throughput at ~1 notification/5 s regardless of batch size, and the in-flight guard means subsequent ticks drop. The spec's "dedicated blocking-I/O dispatcher" (`Component-NotificationOutbox.md:23`) is not literally implemented (delivery runs on the thread pool off the actor thread — functionally acceptable, doc drift), but there is no per-notification parallelism at all. Fine for alarm-notification volume; a burst (alarm storm fanning to hundreds of notifications) drains slowly.
|
||||
7. **Test coverage** is strong overall (MSSQL-backed integration tests for partition purge, execution trees, outage reconciliation, dual-write; per-actor TestKit suites; options validators tested). Gaps mirror the findings: no test asserts a site-SQLite retention purge (it can't — the feature is missing), no test exercises `SwitchOutPartitionAsync` at a size that would expose the command-timeout failure, and no test asserts `backlogTotal` history is non-zero under backlog.
|
||||
|
||||
---
|
||||
|
||||
## Prioritized Recommendations
|
||||
|
||||
1. **Implement the site SQLite 7-day retention purge** (daily timer on `SiteAuditTelemetryActor` or a dedicated job; `DELETE` joined on `audit_forward_state.ForwardState IN ('Forwarded','Reconciled')` + age, followed by `PRAGMA incremental_vacuum`) — closes the unbounded-growth hole and honors the spec'd `ForwardState` invariant.
|
||||
2. **Fix the `backlogTotal` KPI sample source** to read the health aggregator like `AuditLogQueryService` does — one-file change, restores a shipped feature.
|
||||
3. **Harden `SwitchOutPartitionAsync` operationally**: explicit long `CommandTimeout` on the maintenance commands, a health metric/event on purge *failure* (the success event exists; the failure path is log-only), and an integration test at representative volume.
|
||||
4. **Add the filtered non-terminal index on `SiteCalls`** (`WHERE TerminalAtUtc IS NULL`) and turn each KPI snapshot into a single conditional-aggregation query — removes the dominant recurring scan load.
|
||||
5. **Bound the `GetExecutionTreeAsync` edge scan** with a time-window predicate (partition elimination) or per-level child seeks.
|
||||
6. **Decide the DB-role story honestly**: either a second maintenance connection string/principal (and grant `CREATE TABLE` to the purger), or update `Component-AuditLog.md`/`Component-ConfigurationDatabase.md` to state that append-only enforcement is CI-guard-based with roles offered as an optional DBA hardening.
|
||||
7. **Allow same-rank freshness updates in the SiteCalls upsert** (`UpdatedAtUtc` tiebreaker) so retrying calls show live RetryCount/LastError.
|
||||
8. **Batch the KpiSample purge** and reconsider per-node sampling cadence before fleet size multiplies row volume.
|
||||
9. **Emit an audit row for operator Retry** on parked notifications (and consider operator identity on the SiteCalls relay), matching the spec's transition-audit promise.
|
||||
10. **Reconcile the stale design-doc passages** (AuditKind/AuditChannel/NotificationType enum lists, `Teams` references, "no per-channel overrides", SiteCalls schema section) in one documentation pass, per the repo's docs-and-code-travel-together rule.
|
||||
@@ -0,0 +1,145 @@
|
||||
# Architecture Review 05 — Design-Time Pipeline: Templates, Deployment, Transport, Script Analysis
|
||||
|
||||
Reviewer domain: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine`, `src/ZB.MOM.WW.ScadaBridge.DeploymentManager`, `src/ZB.MOM.WW.ScadaBridge.Transport`, `src/ZB.MOM.WW.ScadaBridge.ScriptAnalysis`, their design docs (`docs/requirements/Component-TemplateEngine.md`, `Component-Transport.md`, `Component-ScriptAnalysis.md`, `Component-DeploymentManager.md`) and test projects. All findings cite code actually read.
|
||||
|
||||
## Scope
|
||||
|
||||
- **Template Engine**: flattening (`Flattening/FlatteningService.cs`), collision detection (`CollisionDetector.cs`), acyclicity (`CycleDetector.cs`), revision hashing (`Flattening/RevisionHashService.cs`), lock rules (`LockEnforcer.cs`), diffs (`Flattening/DiffService.cs`), validation (`Validation/ValidationService.cs`, `Validation/SemanticValidator.cs`, `Validation/ScriptCompiler.cs`), authoring services (`TemplateService.cs`, `TemplateInheritanceResolver.cs`).
|
||||
- **Deployment Manager**: `DeploymentService.cs` (deploy/lifecycle pipeline, idempotency reconciliation), `OperationLockManager.cs`, `FlatteningPipeline.cs`, `StateTransitionValidator.cs`, `StaleInstanceProbe.cs`, `ArtifactDeploymentService.cs`.
|
||||
- **Transport**: `Import/BundleImporter.cs` (3,832 lines — load/preview/apply), `Export/BundleExporter.cs`, `Export/DependencyResolver.cs` (skimmed), `Serialization/EntitySerializer.cs`/`EntityDtos.cs`, `Encryption/BundleSecretEncryptor.cs` + `BundleManifestAad.cs`, `Import/LineDiffer.cs`, `Import/ArtifactDiff.cs`, `Import/BundleSessionStore.cs`, `TransportOptions.cs`.
|
||||
- **Script Analysis**: `ScriptTrustPolicy.cs`, `ScriptTrustValidator.cs`, `RoslynScriptCompiler.cs`, `ScriptCompileSurface.cs`.
|
||||
- Cross-component seam checked: Inbound API compiled-handler cache (`src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundScriptExecutor.cs`) as a consumer of Transport-imported `ApiMethod` rows; `ManagementActor` native-alarm-source override handlers.
|
||||
|
||||
## Maturity Verdict
|
||||
|
||||
This slice is **unusually well-engineered in its failure-path plumbing** — the deploy pipeline's query-before-redeploy reconciliation, cancellation-token-safe failure writes, ref-counted operation locks, AES-GCM+AAD bundle crypto with zip-bomb caps and passphrase lockout, and the fused two-pass script trust validator are all production-grade and well beyond typical internal-tool quality. However, the **Transport import apply path has serious correctness holes** that undercut the whole promotion story: imported templates silently lose their inheritance edge, their per-script cadence/timeout fields, and their native alarm sources — three distinct silent-data-loss paths, none covered by the otherwise substantial integration test suite. The flattener has one real algorithmic bug (repeated composed templates lose grandchild members), the revision hash omits native alarm sources (breaking staleness detection for that member class), and the central node's validation path leaks Roslyn script assemblies on every Deployments-page comparison. The core design is sound; the defects are concentrated in the newest surface area (M8 Transport, native alarms) where spec, DTOs, and apply code drifted apart.
|
||||
|
||||
---
|
||||
|
||||
## Findings — Stability & Correctness
|
||||
|
||||
### [Critical] Bundle import drops template inheritance edges — derived templates land as root templates
|
||||
`TemplateDto.BaseTemplateName` (`Serialization/EntityDtos.cs:115`) is populated at export (`Serialization/EntitySerializer.cs:55`) and even diffed in preview (`Import/ArtifactDiff.cs:65`), but **the apply path never consumes it**. `BundleImporter.BuildTemplate` (`Import/BundleImporter.cs:1524-1562`) never sets `ParentTemplateId`; the Overwrite branch (`BundleImporter.cs:1466-1484`) updates only `Description`/`FolderId` and the three child collections; the two second-pass rewires cover alarm→script FKs (`ResolveAlarmScriptLinksAsync`, line 1907) and compositions (`ResolveCompositionEdgesAsync`, line 1993) — there is **no inheritance-edge pass** (`grep -n 'Parent' BundleImporter.cs` returns nothing). The only code that wires `ParentTemplateId` from `BaseTemplateName`, `EntitySerializer.FromBundleContent` (`EntitySerializer.cs:357-360`), is invoked **only from tests** (`tests/.../EntitySerializerTests.cs`).
|
||||
|
||||
**Failure scenario:** export `Pump` + derived `Pump.WaterPump` from dev (the manifest explicitly records `dependsOn: ["Template:Pump"]` — `Component-Transport.md:79-80`); import into prod. `WaterPump` is created with `ParentTemplateId = null`. `FlatteningPipeline.BuildTemplateChainAsync` (`DeploymentManager/FlatteningPipeline.cs:176-192`) produces a one-element chain; every inherited attribute/alarm/script vanishes from the flattened config. Because validation validates *what's there*, an instance with no bindings on the missing members can deploy "successfully" with a silently amputated configuration. Neither `tests/ZB.MOM.WW.ScadaBridge.Transport.Tests` nor `Transport.IntegrationTests` contains any inheritance round-trip test (grep for `BaseTemplateName|ParentTemplateId` hits only serializer unit tests).
|
||||
|
||||
### [High] Imported template scripts lose `MinTimeBetweenRuns` and `ExecutionTimeoutSeconds`
|
||||
`TemplateScriptDto` carries both fields (`EntityDtos.cs:148-151`), but the importer drops them on **both** paths: `BuildTemplate` (Add/Rename — `BundleImporter.cs:1550-1560`) and `SyncTemplateScriptsAsync` (Overwrite — the `changed` comparison and field copy at `BundleImporter.cs:~1836-1851` cover Code/TriggerType/TriggerConfiguration/Params/Return/IsLocked only). Both fields participate in the revision hash (`RevisionHashService.cs:87-90`), so the drift is invisible until redeploy, at which point a WhileTrue script's re-fire cadence or a per-script timeout silently reverts to default on the target environment. Same-class omissions: `LockedInDerived` is not carried in the DTO at all (no hits in `EntityDtos.cs`/`EntitySerializer.cs`), so a base template's derived-lock policy does not survive promotion.
|
||||
|
||||
### [High] Template `NativeAlarmSources` are not transported, but instance-level overrides of them are
|
||||
There is no `TemplateNativeAlarmSourceDto`; `BuildTemplate` copies only attributes/alarms/scripts, and none of the `SyncTemplate*` overwrite helpers touch native alarm sources — yet `InstanceNativeAlarmSourceOverrideDto` **does** travel and is applied (`EntityDtos.cs:292`, `BundleImporter.cs:3325-3338`). Consequences: (a) a template imported as **Add** has no native alarm mirrors at all; (b) an **Overwrite** leaves the target's existing sources untouched and the divergence is invisible because `ArtifactDiff.CompareTemplate` (`Import/ArtifactDiff.cs:54-105`) doesn't diff them either; (c) the carried instance overrides dangle and are *silently ignored* at flatten (`FlatteningService.ApplyInstanceNativeAlarmSourceOverrides`, `FlatteningService.cs:781` — missing canonical name → `continue`). This contradicts `Component-TemplateEngine.md:59-64` which makes native alarm sources first-class template members.
|
||||
|
||||
### [High] Flattening silently drops grandchild members when the same template is composed twice in one subtree
|
||||
The recursive composed-member resolvers share one `visited` set down the recursion (`FlatteningService.cs:285-297` for attributes; identical pattern for alarms `:649-660`, scripts `:913-924`, native sources `:761-772`). The direct members of a repeated module still resolve (added before the `visited` check), but the **descent into its nested compositions is skipped** on the second occurrence.
|
||||
|
||||
**Failure scenario:** template X composes Y twice (slots `y1`, `y2`); Y composes Z. Flattening produces `y1.*`, `y1.z.*`, `y2.*` — but **not `y2.z.*`**. No validation error fires (the members simply don't exist in the flattened config), so the instance deploys with the second slot's nested sensor/alarm/script members missing. `CollisionDetector.CollectComposedMembers` (`CollisionDetector.cs:126-143`) has the same shared-`visited` early-return, and there it skips even the repeated module's *direct* members — so collision detection is also blind under the second slot. The `visited` set is the right tool for cycle-guarding but the wrong tool for slot deduplication: composition is by-slot, not by-template. Fix is to key the guard on the *path* (prefix) or track only the current recursion stack.
|
||||
|
||||
### [High] Revision hash and diff omit `NativeAlarmSources` — no staleness signal for native-alarm changes
|
||||
`RevisionHashService.HashableConfiguration` (`RevisionHashService.cs:116-152`) hashes Attributes, Alarms, Scripts, Connections — **not** `FlattenedConfiguration.NativeAlarmSources` (populated at `FlatteningService.cs:135`). Editing a template's native alarm source (connection, source reference, condition filter) or an instance override therefore does not change the revision hash: the Deployments page never flags the instance stale, `DeploymentService.GetDeploymentComparisonAsync` reports `IsStale = false`, and sites keep mirroring the old alarm source indefinitely until some unrelated change forces a redeploy. This contradicts the spec's own claim that overrides "participate in the revision hash — changes … re-deploy as expected" (`Component-TemplateEngine.md:215`), which is implemented for connection-binding overrides but not for the native-alarm member class. `DiffService` should be audited for the same omission.
|
||||
|
||||
### [High] Bundle import leaves stale compiled Inbound API handlers serving old code
|
||||
`ApplyApiMethodsAsync` (`BundleImporter.cs:2625`) writes `ApiMethod` rows through `IInboundApiRepository` — a DB-level write. The Inbound API's singleton `InboundScriptExecutor` caches compiled delegates **by method name with no content-based invalidation** (`InboundAPI/InboundScriptExecutor.cs:311-331`: `_scriptHandlers.TryGetValue(method.Name, …)` → serve cached delegate) and caches known-bad methods the same way (`:39, :58-62`). An import that Overwrites an ApiMethod script keeps the **old delegate** serving (HTTP 200 with stale behavior); an import that *fixes* a previously-broken method keeps returning "Script compilation failed" from `_knownBadMethods` until restart. This is the exact failure mode already documented in project memory for DB-direct edits — Transport import is a DB-direct edit at scale, and nothing in `ApplyAsync` signals the executor. Cross-cluster deployments make this worse: the cache lives per central node.
|
||||
|
||||
### [High] Locked native alarm sources can be overridden at instance level — lock not enforced anywhere
|
||||
Spec: "Lock applies to the entire source" (`Component-TemplateEngine.md:114`). Reality: `ApplyInstanceNativeAlarmSourceOverrides` (`FlatteningService.cs:775-792`) applies the override with **no lock check** — contrast the attribute path (`:309 if (existing.IsLocked) continue`) and alarm path (`:337`). `ResolvedNativeAlarmSource` doesn't even carry `IsLocked` (the inherit-time `lockedNames` set at `:676` is local and discarded). The management command path has no check either: `ManagementActor.HandleSetInstanceNativeAlarmSourceOverride` (`ManagementService/ManagementActor.cs:882-909`) upserts unconditionally after site-scope authorization. A site-scoped operator can repoint a *locked* native alarm binding to an arbitrary connection/source reference — precisely what locking exists to prevent on a read-only alarm mirror.
|
||||
|
||||
### [Medium] Post-success audit failure flips a committed Success deployment record to Failed
|
||||
`DeployInstanceAsync` commits the terminal Success status (`DeploymentService.cs:304-306`), runs post-success side effects (correctly swallowed, `:970-1005`), then calls `_auditService.LogAsync` **inside the same try** (`:330-332`). If that audit write throws (transient DB fault, serialization), the catch block (`:343-397`) unconditionally sets `record.Status = Failed` and persists it — central now reports Failed while the site runs the new config, the exact divergence the surrounding comments (`:298-303`) were written to prevent. It also violates the repo convention "audit-write failure NEVER aborts the user-facing action". The audit call should be outside the try or individually guarded like `TryLogLifecycleTimeoutAsync` (`:1034-1066`) already is for lifecycle timeouts.
|
||||
|
||||
### [Medium] Deleting a `NotDeployed` instance still requires a live site round-trip
|
||||
`StateTransitionValidator.CanDelete` explicitly allows delete-from-NotDeployed as "central-side record cleanup (no live site config to tear down)" (`StateTransitionValidator.cs:40-49`), but `DeploymentService.DeleteInstanceAsync` (`DeploymentService.cs:546-585`) unconditionally sends `DeleteInstanceCommand` to the site and fails on timeout. A Transport-imported instance (always lands `NotDeployed`) targeted at a site that is unreachable or not yet commissioned is **undeletable** through the normal path. The two components disagree about the same design sentence.
|
||||
|
||||
### [Medium] Import blocker heuristic can hard-block a valid import with no operator override
|
||||
The Pass-1 name scan (`RunSemanticValidationAsync`, `BundleImporter.cs:3603-3692`; shared with `DetectBlockersAsync:751-818`) flags any `PascalCase(`-shaped token not in the enumerative `KnownNonReferenceNames` denylist (`:853-874`) as a missing SharedScript/ExternalSystem and **throws `SemanticValidationException`** at apply — rolling back the whole import. Any script that defines a local PascalCase method (`decimal ComputeRate(...)` then `ComputeRate(x)`), uses a stdlib type not on the list (`Regex.Match`, `StringBuilder`, `Json`), or embeds SQL keywords outside the nine listed, is a false positive with no "proceed anyway" escape hatch — the operator must edit scripts or re-cut the bundle. The denylist will drift forever. Conversely `Regex`/`StringBuilder` being absent from the list means the heuristic fires on innocent code today. Consider downgrading Pass-1 findings on *template scripts* to warnings (the deploy gate re-validates authoritatively) and keeping hard-block only for `ApiMethod` scripts, or adding an explicit ignore resolution.
|
||||
|
||||
### [Medium] `OperationLockManager` is single-node in-memory; concurrent ops via the standby node aren't excluded
|
||||
Documented as acceptable ("lost on central failover", `OperationLockManager.cs:12-13`) and the implementation itself is careful (ref-counting, reservation-before-wait, exception-filter cleanup — `:46-129`). Residual risk: nothing structurally prevents both central nodes from executing mutating instance ops concurrently if traffic bypasses Traefik's active-node routing (direct ports 9001/9002 are documented in CLAUDE.md). Optimistic concurrency on `DeploymentRecord` is the backstop, but instance-state writes (`DisableInstanceAsync:456-458`) are last-write-wins. Low probability, but worth an explicit invariant (reject management ops on the non-active node).
|
||||
|
||||
---
|
||||
|
||||
## Findings — Performance
|
||||
|
||||
### [High] Every validation run compiles every script with Roslyn — CPU heavy and leaks loaded assemblies on the central node
|
||||
`ValidationService.ValidateScriptCompilation` (`ValidationService.cs:256-273`) calls `ScriptCompiler.TryCompile` per script; that runs `ScriptTrustValidator.FindViolations` (a fresh `CSharpCompilation.CreateScriptCompilation` over the **entire TPA reference set**, `ScriptTrustValidator.cs:114-124`) *plus* `RoslynScriptCompiler.Compile` → `CSharpScript.Create(...).Compile()` (`RoslynScriptCompiler.cs:70-71`). `Script.Compile()` on success emits and loads the script assembly via Roslyn's `InteractiveAssemblyLoader`, which is **not collectible** — each compiled script permanently occupies loader/metadata memory in the central host.
|
||||
|
||||
This sits on hot paths: `FlatteningPipeline.FlattenAndValidateAsync` (`FlatteningPipeline.cs:161-167`) is invoked (a) per deploy, (b) per instance by `DeploymentService.GetDeploymentComparisonAsync` (`DeploymentService.cs:652`) — i.e. **at query time whenever the Deployments page computes staleness**, and (c) per deployed instance by `StaleInstanceProbe` during every template-overwriting bundle import (`BundleImporter.ComputeStaleInstanceIdsAsync:1242-1327`). A 50-instance environment with 10 scripts each = 500 full trust-compilations + 500 script-assembly loads per Deployments-page staleness sweep, unbounded over process lifetime. There is **no compile-result cache**. Fix: cache `TryCompile` verdicts keyed by SHA-256 of code (the verdict is a pure function of code + policy), and/or skip `ValidateScriptCompilation` on read-only comparison paths (staleness needs the hash, not a compile).
|
||||
|
||||
### [Medium] `LineDiffer` worst case is O((N+M)²) memory with no input cap
|
||||
The Myers implementation snapshots the full V frontier per edit distance (`LineDiffer.cs:157-204`: `trace.Add((int[])v.Clone())` with `v` of length `2(N+M)+1`, up to `N+M` rounds). Two ~10,000-line scripts with no common lines ⇒ ~20k rounds × 160 KB ≈ **3+ GB** transient allocation during `PreviewAsync`. `maxLines` (400) caps only the *output* (`:66-92`); nothing caps the *inputs*, and the bundle size cap (100 MB) comfortably admits such scripts. A crafted (or merely bloated) bundle can OOM the active central node from the import preview. Cap input line counts (fall back to the `<N lines>` summary beyond a threshold) or use the linear-space Myers variant.
|
||||
|
||||
### [Medium] Any template with a folder, parent, or composition can never diff `Identical`
|
||||
`ArtifactDiff.CompareTemplate` compares `FolderNameOf`/`BaseTemplateNameOf`/`CompositionTargetNameOf` placeholder strings (`"<id:5>"`) against real incoming names (`ArtifactDiff.cs:64-65, 100, 674-693`), so `FolderName`/`BaseTemplateName`/`Compositions.*` always register as changed for such templates. Effect: the wizard's "Identical → auto-skipped" contract (`Component-Transport.md:222`) never applies to structured templates; every re-import of an unchanged bundle shows spurious Modified rows and, if bulk-Overwrite is chosen, generates needless audit rows and stale-instance probing. The preview loop already hydrates templates (`BundleImporter.cs:379-388`); passing the folder/template name maps it also builds (`:365-366`) into `CompareTemplate` would fix all three placeholders cheaply.
|
||||
|
||||
### [Low] `LoadAsync` is O(bundle) even when only the manifest is needed, and each load double-opens the zip
|
||||
Acknowledged in the design doc itself (`Component-Transport.md:90` — deferred `ReadManifestAsync`). Combined with sessions holding the full decrypted content (`BundleSession.DecryptedContent`) for up to 30 minutes and no cap on concurrent session count (`BundleSessionStore.cs:38`), N concurrent 100 MB uploads pin N×~200 MB. The T-007 zeroing on apply/failure (`BundleImporter.cs:1105, 1194`) is good; a session-count cap would round it out.
|
||||
|
||||
### [Low] Import apply is one long EF transaction with per-entity audit rows and four intermediate flushes
|
||||
`ApplyAsync` (`BundleImporter.cs:960-1098`) holds a relational transaction across full semantic validation (Roslyn-free — fine), all entity staging, per-row `IAuditService.LogAsync` calls, and stale-instance probing (which itself re-flattens per instance, see the Roslyn finding). For a large bundle this is minutes of open-transaction lock footprint on the shared config DB while the Blazor circuit waits. The CLI's raised 5-minute timeout acknowledges the symptom.
|
||||
|
||||
---
|
||||
|
||||
## Findings — Security / Trust Model
|
||||
|
||||
### [Medium] Deny-list gaps: `Environment.Exit`/`FailFast` (site-node kill), `Environment.GetEnvironmentVariable` (secret read)
|
||||
`ScriptTrustPolicy.ForbiddenScopes` (`ScriptTrustPolicy.cs:36-45`) blocks IO/Process/Threading/Reflection/Net/Interop/Win32, but the `System` namespace root types remain fully available: a template script containing `Environment.Exit(0)` compiles cleanly against the surface and **terminates the site-runtime process** on first trigger — a one-line denial of service of an entire site node from the Design role. `Environment.FailFast`, `Environment.GetEnvironmentVariable` (exfiltrate `SCADABRIDGE_API_KEY`-class secrets via `Notify`/`ExternalSystem.Call`), and `GC.Collect` loops are similarly unblocked. These are cheap adds to `ForbiddenScopes` (as fully-qualified type-or-member entries) or to `ForbiddenIdentifiers`.
|
||||
|
||||
### [Medium] `System.Data` allowance is an intentional but unbounded network hole
|
||||
Site scripts may use raw ADO.NET (`Database.Connection` returns `DbConnection`; `System.Data.Common` is necessarily bindable — `ScriptCompileSurface.cs:119`). `System.Data` is absent from `ForbiddenScopes`, so if the site-runtime execution reference set includes any concrete provider (e.g. `Microsoft.Data.SqlClient`, which the Database helper itself needs), a script can `new SqlConnection("Server=attacker;...")` — an arbitrary-host network channel that the `System.Net` deny was meant to close. The semantic pass will not flag it (SqlClient is not under a forbidden scope). Worth either denying provider namespaces (`Microsoft.Data`, `System.Data.SqlClient`, `System.Data.Odbc`, …) while keeping `System.Data.Common` types reachable only via the `Database` helper, or documenting the accepted risk explicitly in `Component-ScriptAnalysis.md`.
|
||||
|
||||
### [Medium] Reflection-gateway member list misses `GetTypes`, `Invoke`, `EntryPoint`, `DeclaredMembers`
|
||||
`ReflectionGatewayMembers` (`ScriptTrustPolicy.cs:66-88`) blocks `GetMethod`/`InvokeMember`/etc. but not `Invoke` (on `MethodInfo`/`Delegate`), `GetTypes`, `EntryPoint`, `DeclaredMethods`. Pass 1 usually compensates — any expression whose resolved member's containing type is under `System.Reflection` is flagged semantically (`ScriptTrustValidator.cs:180-212`) — but in the documented **TPA-degraded fallback mode** (`ScriptTrustPolicy.cs:207-245`) resolution weakens and Pass 2 becomes the primary defence for chained accesses; the syntactic list should be closed under the obvious invocation members since it exists precisely for that mode. Cheap hardening.
|
||||
|
||||
### [Medium] Transport import bypasses the script trust gate entirely
|
||||
`ApplyAsync` writes template scripts, shared scripts, and API-method scripts to the DB without ever invoking `ScriptTrustValidator` (`RunSemanticValidationAsync` at `BundleImporter.cs:3603` is name-resolution + `SemanticValidator` only). Compensating controls exist downstream — the deploy gate re-validates template scripts (`ScriptCompiler.TryCompile`), the Inbound API re-checks trust at lazy compile — but the layering claim in `Component-ScriptAnalysis.md` ("all call sites delegate…") now has a fifth write path with no gate, and the Inbound API's check happens at *serve time* on the production node rather than at *import review time*. Running `FindViolations` over imported script bodies during Pass 2 (they're already in memory) would restore the design-time boundary and give the operator the finding in the wizard instead of a runtime 500.
|
||||
|
||||
### Positive observations (crypto & envelope)
|
||||
`BundleSecretEncryptor` is textbook: AES-256-GCM with 16-byte salt / 12-byte nonce / 16-byte tag, PBKDF2-SHA256 600k iterations, fresh salt+nonce per encrypt (`BundleSecretEncryptor.cs:14-58`). The T-005 AAD binding of non-derivative manifest fields (`BundleManifestAad.cs:45-55`, verified in `LoadAsync` at `BundleImporter.cs:252-255`) makes the source-environment confirmation gate tamper-evident — a genuinely thoughtful touch. T-006 zip-bomb caps validate central-directory headers before any decompression (`BundleImporter.cs:296-338`); T-003 lockout is keyed by content hash so a second tab/CLI can't sidestep it (`BundleSessionStore.cs:40-47, 122-167`); T-007 zeroes decrypted plaintext on both success and failure. Key management is passphrase-only with no persistence — appropriate for the file-carry threat model, with the residual (documented) weakness that unencrypted export is allowed and only audit-flagged.
|
||||
|
||||
---
|
||||
|
||||
## Findings — Conventions & Spec Drift
|
||||
|
||||
### [Medium] Instance alarm overrides implement less than the spec grants
|
||||
`Component-TemplateEngine.md:113` says instance-overridable alarm fields include **Description and On-Trigger Script reference**; `ApplyInstanceAlarmOverrides` (`FlatteningService.cs:328-355`) applies only `TriggerConfigurationOverride` and `PriorityLevelOverride`. Either the doc or the entity/flattener should move.
|
||||
|
||||
### [Medium] `ArtifactDiff` child-equality predicates miss fields that the import sync *does* write
|
||||
`AttributesEqual` omits `ElementDataType` and `LockedInDerived` (`ArtifactDiff.cs:646-651`); `AlarmsEqual` omits the on-trigger script reference (`:653-658`); `ScriptsEqual` omits `MinTimeBetweenRuns`/`ExecutionTimeoutSeconds` (`:660-666`). A bundle differing only in those fields classifies **Identical → auto-skipped → silently not imported** (for the fields the sync would have written) — the mirror image of the false-Modified problem above. Diff predicates and sync-helper `changed` predicates should be generated from one field list per entity so they cannot drift (they already have, in both directions).
|
||||
|
||||
### [Low] Bundle format spec advertises a `scripts/` directory that is never produced, while `MaxBundleEntryCount` defaults to 4
|
||||
`Component-Transport.md:45-47` documents optional `scripts/template-{id}-{name}.cs` entries; no code writes or reads them, and `TransportOptions.MaxBundleEntryCount = 4` (`TransportOptions.cs:22`) would reject a bundle that actually used the feature for more than two scripts. Remove from the spec or note it as unimplemented.
|
||||
|
||||
### [Low] `ScriptCompiler.TryCompile` surfaces only the first violation/error
|
||||
`Validation/ScriptCompiler.cs:42,47` return `violations[0]` / `errors[0]`; an operator fixing a script with five issues gets five deploy round-trips. Both underlying APIs return full lists — join them.
|
||||
|
||||
### [Low] `RevisionHashService` determinism contract is fragile but guarded
|
||||
Hash stability depends on alphabetical property declaration order in the `Hashable*` records (`RevisionHashService.cs:12-19, 113-115`); the guard test (`HashableRecords_PropertiesDeclaredAlphabetically`) exists, which is the right mitigation. Note for maintainers: adding `NativeAlarmSources` (see the High finding) will change all hashes — plan it as a deliberate migration (one-time global staleness) rather than a patch.
|
||||
|
||||
### Positive observations (conventions)
|
||||
`DeploymentService` is exemplary on the repo's own rules: `CancellationToken.None` for failure/audit writes after cancellation (`DeploymentService.cs:355-372`), timeout-audit entries for all lifecycle verbs, reconciliation preserving operator-intended Disabled state (`:895-907`), terminal-status-before-side-effects ordering (`:298-306`), and the notify-and-fetch PendingDeployment TTL rationale (`:283-290`) is documented where the decision lives. `CycleDetector.BuildLookup` duplicate-Id tolerance and the Id-0-is-legitimate handling (`CycleDetector.cs:21-27, 56-59`) show real defect-driven hardening. `ScriptTrustPolicy`'s `DefaultReferences` vs `AnalysisReferences` separation with the loud TPA-degradation flag (`ScriptTrustPolicy.cs:207-245`) is a well-reasoned two-layer defence.
|
||||
|
||||
---
|
||||
|
||||
## Underdeveloped Areas
|
||||
|
||||
1. **Transport site/instance wave (M8) shipped ahead of its template-fidelity basics.** Instances, connections, name-mapping, FK rewiring, and stale-probing are all implemented and tested (`SiteInstanceImportTests.cs`, `CompositionImportTests.cs`, rollback-failure tests), but the older template payload silently loses inheritance, cadence/timeout, derived-lock flags, and native alarm sources (findings above). The DTO layer (`EntitySerializer.FromBundleContent`) actually implements more fidelity than the apply path uses — the importer re-implements DTO→entity mapping by hand (`BuildTemplate`) and drifted.
|
||||
2. **No import test exercises a derived template.** `Transport.IntegrationTests` covers round-trip, conflict resolution, composition edges, semantic validation failure — but grep shows zero inheritance coverage. A single `export(base+derived) → import → flatten` test would have caught the Critical finding.
|
||||
3. **No flattening test for repeated composition.** `TemplateEngine.Tests/Flattening` exists but the shared-`visited` semantics (same template in two slots of one subtree) is untested; same for `CollisionDetector`.
|
||||
4. **Deferred items acknowledged in docs but load-bearing:** manifest-only `ReadManifestAsync` (perf), Configuration-Audit-Log "Bundle Import" filter UI (Transport-012), rename call-site rewriting ("call sites are not rewritten in v1" — `BundleImporter.cs:1519-1522`, meaning a Rename resolution on a shared script produces a bundle whose importing templates still call the *old* name, satisfied only because the name-set registers both — the *target DB* ends up with scripts calling a name that may not exist there; only Pass-1's dual registration hides it at import time).
|
||||
5. **`ArtifactDiff` version fields are permanently null** (`ArtifactDiff.cs:33-37`) — the manifest's per-artifact `version` numbers are display-only; no optimistic concurrency exists between preview and apply, so a target edited between Preview and Apply is silently overwritten (session TTL is 30 min — a real window).
|
||||
6. **Compile-surface parity relies on three hand-maintained mirrors** (`ScriptCompileSurface`, `TriggerCompileSurface`, Central-UI `SandboxScriptHost`), guarded by reflection tests for two of them and representative-script tests for the third (`Component-ScriptAnalysis.md:141-145`). Adequate but fragile; a source-generator would eliminate the class of drift.
|
||||
7. **`SemanticValidator` call-target extraction is string-based** and self-admittedly cannot resolve dynamic child names (`SemanticValidator.cs:43-56` — leaf-name fallback accepts any composed script with a matching leaf, so a wrong-child `Children[x].CallScript("Y")` passes design-time validation). Documented tradeoff, but worth a warning-level finding rather than silence.
|
||||
|
||||
---
|
||||
|
||||
## Prioritized Recommendations
|
||||
|
||||
1. **[Critical] Wire template inheritance in the import apply path.** Add a `ResolveInheritanceEdgesAsync` pass mirroring `ResolveCompositionEdgesAsync` (resolve `BaseTemplateName` against Local + target by name, honouring Rename), run it after the template flush, and run `CycleDetector.DetectInheritanceCycle`/`DetectCompositionCycle` over the merged graph before commit (the importer currently performs **no** acyclicity check — a crafted bundle can persist a composition cycle that the authoring path would reject). Add a derived-template round-trip integration test.
|
||||
2. **[High] Close the import data-loss set in one sweep:** carry `MinTimeBetweenRuns`/`ExecutionTimeoutSeconds` through `BuildTemplate` + `SyncTemplateScriptsAsync`; add `LockedInDerived` to the attribute/alarm/script DTOs and syncs; add `TemplateNativeAlarmSourceDto` (or block export of templates that have native alarm sources until supported). Drive diff predicates and sync predicates from a single per-entity field list.
|
||||
3. **[High] Fix the repeated-composition `visited` bug** in the four `ResolveComposed*Recursive` methods and `CollisionDetector.CollectComposedMembers` — guard on recursion *path*, not global template Id.
|
||||
4. **[High] Add `NativeAlarmSources` to `RevisionHashService` (and `DiffService`)** as a planned hash-migration, restoring staleness detection for native-alarm edits.
|
||||
5. **[High] Cache script-compile verdicts by code hash** and stop loading script assemblies on validation-only paths; skip `ValidateScriptCompilation` in `GetDeploymentComparisonAsync`/`StaleInstanceProbe` (staleness needs only the hash). This removes both the CPU spike and the non-collectible assembly leak from the Deployments page and bundle imports.
|
||||
6. **[High] Invalidate Inbound API handler caches on import** — publish an ApiMethod-changed notification from `ApplyAsync` (or key `_scriptHandlers` by `(name, scriptHash)`), covering the management-path gotcha already in project memory at the same time.
|
||||
7. **[High] Enforce locks on native alarm source overrides** at both `ManagementActor.HandleSetInstanceNativeAlarmSourceOverride` and `FlatteningService.ApplyInstanceNativeAlarmSourceOverrides` (carry `IsLocked` on `ResolvedNativeAlarmSource`).
|
||||
8. **[Medium] Harden the deny-list:** add `System.Environment` members (`Exit`, `FailFast`, `GetEnvironmentVariable(s)`, `SetEnvironmentVariable`), close the reflection-gateway list (`Invoke`, `GetTypes`, `EntryPoint`), and decide/document the `System.Data` provider posture.
|
||||
9. **[Medium] Deployment audit-write isolation:** move the post-success `LogAsync` in `DeployInstanceAsync` out of the try (or guard it) so an audit fault can never flip a committed Success record to Failed; skip the site round-trip when deleting `NotDeployed` instances.
|
||||
10. **[Medium] Bound `LineDiffer` inputs; fix `ArtifactDiff` placeholder comparisons** (pass folder/template name maps into `CompareTemplate`) so Identical classification works for structured templates; consider an "ignore blocker" resolution for the Pass-1 heuristic on template scripts.
|
||||
@@ -0,0 +1,143 @@
|
||||
# Architecture Review 06 — Edge Integrations (Inbound & Outbound Boundaries)
|
||||
|
||||
Reviewer domain: the four components where ScadaBridge touches external systems.
|
||||
|
||||
## Scope
|
||||
|
||||
Deep source review (every non-generated `.cs` file read) of:
|
||||
|
||||
- `src/ZB.MOM.WW.ScadaBridge.InboundAPI` — `POST /api/{methodName}`, API-key auth, Roslyn script handlers, parameter/return validation, audit middleware, routed site calls.
|
||||
- `src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway` — outbound HTTP client (`Call`/`CachedCall`), `DatabaseGateway` (`Connection`/`CachedWrite`), HTTP + SQL error classifiers, S&F handoff.
|
||||
- `src/ZB.MOM.WW.ScadaBridge.NotificationService` + the delivery adapters in `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/Delivery` — SMTP (MailKit) wrapper, OAuth2 token service, SMTP/SMS error classifiers, credential redactor, Email/SMS adapters.
|
||||
- `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier` — Native-AOT `WWNotifier.exe` client (arg parser, config loader, failover loop, YES/NO reporter).
|
||||
|
||||
Specs compared: `docs/requirements/Component-InboundAPI.md`, `Component-ExternalSystemGateway.md`, `Component-NotificationService.md`, `docs/plans/2026-06-26-delmia-recipe-notifier-design.md`. Test projects examined: `tests/ZB.MOM.WW.ScadaBridge.{InboundAPI,ExternalSystemGateway,NotificationService,NotificationOutbox,DelmiaNotifier}.Tests`.
|
||||
|
||||
## Maturity Verdict
|
||||
|
||||
This is a mature, unusually well-hardened edge layer. The code shows evidence of multiple hardening passes: enumeration-safe auth responses, bounded body capture with UTF-8-safe truncation, ordered cancellation-vs-timeout catch discrimination repeated consistently across HTTP, SQL, SMTP, and SMS paths, poison-message parking on both S&F retry paths, credential redaction on every error string, and secrets encrypted at rest (`ExternalSystemConfiguration.cs:27-29,83-85`). Error classification matches the spec (5xx/408/429/connection = transient) on all four channels and is centralized in dedicated classifiers with tests. The remaining risk is concentrated in three places: **cache coherence of compiled inbound script handlers across central failover** (a silent-stale-code bug, the one genuinely High-severity defect found), **two acknowledged spec-vs-code gaps in the External System Gateway** (per-system timeout and path templating), and **unbounded buffering of outbound HTTP response bodies**. DelmiaNotifier is small, faithful to its design doc, and fully tested at the logic level.
|
||||
|
||||
---
|
||||
|
||||
## Findings — Dimension 1: Stability
|
||||
|
||||
### [High] Compiled inbound handler cache goes stale across central failover — old script serves silently with HTTP 200
|
||||
|
||||
`InboundScriptExecutor` is a per-node singleton (`ServiceCollectionExtensions.cs:19`) whose `_scriptHandlers` and `_knownBadMethods` caches (`InboundScriptExecutor.cs:26,39`) are mutated only by:
|
||||
|
||||
- the node's own startup compile of all methods (`Host/Program.cs:379-388`), and
|
||||
- `ManagementActor.HandleUpdateApiMethod`/`HandleDeleteApiMethod` → `CompileAndRegister`/`RemoveHandler` (`ManagementActor.cs:2631,2650`) — which run **only on the node hosting the ManagementActor singleton (the active node)**.
|
||||
|
||||
The standby central node compiles every method at *its* startup and then never hears about updates. `ExecuteAsync` looks the handler up by **name only** (`InboundScriptExecutor.cs:311`) — no revision/hash check — even though the endpoint fetches the fresh `ApiMethod` row (with the current `Script` text) from the DB on every request (`EndpointExtensions.cs:142`).
|
||||
|
||||
Concrete failure scenario:
|
||||
1. Method `GetOrderStatus` updated via CLI on active node A → A recompiles, serves new logic.
|
||||
2. Weeks later A fails over to B. B still holds the delegate compiled from the script text as of B's process start.
|
||||
3. B serves the **old** script indefinitely — HTTP 200, correct-looking JSON, `UpdateApiMethod` long ago reported success. Nothing recompiles until B restarts.
|
||||
|
||||
The mirror-image variant: a method that was broken at B's startup landed in B's `_knownBadMethods`; it was later fixed via management update (clears the record on A only). After failover, B answers `"Script compilation failed for this method"` for a method that compiles fine.
|
||||
|
||||
This is the cluster-wide generalization of the already-known "direct SQL edit doesn't clear the cache" issue (documented as a Direct SQL Warning, `Component-InboundAPI.md:129-131`) — but here the write went through the *supported* management path and is still lost. A per-request script-hash comparison against the freshly-fetched row would fix failover staleness, direct-SQL staleness, and known-bad staleness in one move at negligible cost (the row is already in hand).
|
||||
|
||||
Deleted methods are safe (fresh DB lookup returns null → 403); methods *created* after standby startup are safe (lazy compile, `InboundScriptExecutor.cs:314-330`).
|
||||
|
||||
### [Medium] Method timeout abandons the running script; its DI scope is disposed underneath it
|
||||
|
||||
On timeout, `handler(context).WaitAsync(cts.Token)` (`InboundScriptExecutor.cs:333`) returns control but does not stop the underlying script task — cancellation is cooperative via `ctx.CancellationToken` only. The `finally` block then disposes the per-execution DI scope (`InboundScriptExecutor.cs:389-393`) while a non-cooperative script (e.g. mid-flight in a poorly-authored loop, or blocked in a routed call that ignores the token) may still be running and holding `InboundDatabaseHelper` → the scoped `IDatabaseGateway`. Consequences: `ObjectDisposedException`s surfacing as unobserved background faults, and — since the inbound API deliberately has no rate limiting (`Component-InboundAPI.md:140`) — repeated timeouts of a slow method accumulate orphaned executions with no counter or cap. Recommend at minimum a metric for abandoned executions, and deferring scope disposal until the handler task actually completes (`ContinueWith` on the abandoned task).
|
||||
|
||||
### [Medium] ESG API-key `AuthConfiguration` colon-splitting is ambiguous — a key containing `:` silently breaks auth
|
||||
|
||||
`ExternalSystemClient.ApplyAuth` (`ExternalSystemClient.cs:489-499`) interprets `AuthConfiguration` as `"HeaderName:KeyValue"` when it contains a colon, else as a bare key with default header `X-API-Key`. An API key whose *value* legitimately contains a colon (not rare for opaque tokens) is silently misparsed: the prefix becomes a bogus header name and the request goes out mis-authenticated with only a remote 401 to debug from. The Basic branch (`:504-519`) is fine (split-2 keeps colons in the password), but the apikey branch has no escape hatch. Structured JSON config (`{"header":"X-API-Key","key":"..."}`) — which the entity doc-comment (`ExternalSystemDefinition.cs:13`, "JSON-serialized authentication configuration") already implies — would eliminate the ambiguity.
|
||||
|
||||
### [Medium] DelmiaNotifier failover can duplicate a processed notification
|
||||
|
||||
`HttpRecipeSender.cs:36-45` maps **any** `HttpRequestException` and any non-caller timeout to `ConnectFailed`, which the loop rolls over to the next URL (`Notifier.cs`, `continue`). Both cases include "the server received and possibly processed the request, but the response never arrived" (connection reset after send; slow response exceeding `TimeoutSeconds`). The next node then receives a second `DelmiaRecipeDownload` POST for the same recipe. This matches the design table (`2026-06-26-delmia-recipe-notifier-design.md:97-99` lists timeout as a failover trigger) — so this is a *design-level* at-least-once semantics that is nowhere documented as such; the inbound `DelmiaRecipeDownload` method must be idempotent on `(MachineCode, DownloadPath, WorkOrderNumber)` or duplicates will surface downstream. Worth one paragraph in the design doc and a check of the deployed method script.
|
||||
|
||||
### [Medium] SMS adapter keeps sending after the first transient failure — inflating duplicate texts on retry
|
||||
|
||||
In `SmsNotificationDeliveryAdapter.DeliverAsync` (`SmsNotificationDeliveryAdapter.cs:148-170`), a transient result does not break the loop; the remaining recipients are still attempted. Since *any* transient rolls the **whole notification** up to `Transient` (`RollUp`, `:297-303`) and retry re-texts everyone, each additional number accepted after the first transient is a guaranteed duplicate SMS on the retry. Short-circuiting on the first transient would reduce duplicates at no delivery cost (the retry re-attempts all numbers anyway). The re-send-to-all characteristic itself is documented (`Component-NotificationService.md:102`), but this amplification is avoidable within v1's no-per-recipient-state constraint.
|
||||
|
||||
### [Low] `ErrorClassifier.IsTransient(Exception)` treats any `OperationCanceledException` as transient
|
||||
|
||||
`ErrorClassifier.cs:32-38` classifies bare `OperationCanceledException`/`TaskCanceledException` as transient. Every current call site is protected by ordered catch filters that peel off caller-cancellation first (`ExternalSystemClient.cs:314-327`), so behaviour is correct today — but the predicate is public and a future call site that skips the ordering would buffer caller-cancelled work into S&F. A `cancellationToken` parameter (as `SmtpErrorClassifier.Classify` and `SmsErrorClassifier.Classify` already take) would make it misuse-resistant.
|
||||
|
||||
### [Low] First-of-many SMTP configuration is nondeterministic
|
||||
|
||||
`EmailNotificationDeliveryAdapter.cs:75-76` and `NotificationOutboxActor.ResolveRetryPolicyAsync` (`NotificationOutboxActor.cs:373-375`) both use `GetAllSmtpConfigurationsAsync()` + `FirstOrDefault()`. With more than one SMTP row the selected server and the retry policy depend on repository ordering. Either enforce single-row semantics or add an explicit "active" flag.
|
||||
|
||||
### [Low] OAuth2 token cache: unsynchronized field visibility and unbounded growth
|
||||
|
||||
`OAuth2TokenService.CacheEntry.Token/Expiry` are plain fields read outside the per-entry lock (`OAuth2TokenService.cs:48-51`) and written inside it (`:93-94`). The races are benign (worst case: one redundant token fetch), but `volatile`/`Volatile.Read` would make it airtight. The cache is also never evicted — bounded in practice by the number of distinct SMTP credential triples, so acceptable.
|
||||
|
||||
### [Low] Non-JSON content-type with a body yields a misleading 400
|
||||
|
||||
`EndpointExtensions.cs:181-194`: `ContentLength > 0 || ContentType contains "json"` means a `text/plain` or `application/x-www-form-urlencoded` body is force-parsed as JSON and rejected as "Invalid JSON in request body" rather than a 415. Cosmetic, but the error steers the caller at the wrong problem.
|
||||
|
||||
## Findings — Dimension 2: Performance
|
||||
|
||||
### [Medium] Outbound HTTP responses are buffered unbounded
|
||||
|
||||
`ExternalSystemClient.InvokeHttpAsync` uses `client.SendAsync(request, token)` with the default `ResponseContentRead` completion option (`ExternalSystemClient.cs:312`) and then `ReadAsStringAsync` (`:337`) — the *entire* response body is materialized in memory with **no size cap**. `MaxErrorBodyChars` (`:386`) truncates only the string embedded into error messages, *after* the full read. The inbound side caps request bodies at 1 MiB precisely because "no rate limiting… an explicit, modest cap bounds per-request allocations" (`InboundApiOptions.cs:16-19`); the outbound side extends no such protection against a misbehaving or compromised external system streaming hundreds of MB into a site node (which also runs the instance actors). Use `HttpCompletionOption.ResponseHeadersRead` plus a bounded read (the audit middleware's cap+1 pattern is directly reusable).
|
||||
|
||||
### [Medium] Serial per-recipient Twilio POSTs inside a serial dispatch sweep
|
||||
|
||||
`SmsNotificationDeliveryAdapter` sends one POST per phone number sequentially (`SmsNotificationDeliveryAdapter.cs:148-170`), and the outbox dispatcher delivers each due notification sequentially within a sweep (`NotificationOutboxActor.cs:320-345`). Worst case for one 50-recipient list against a black-holing endpoint: 50 × 30 s = 25 minutes inside one sweep, during which *all* other due notifications (including email) queue behind it and appear "stuck" on the KPI tiles. Bounded parallelism per notification (e.g. `Parallel.ForEachAsync` with 4–8 DOP — Twilio tolerates it) and/or first-transient short-circuit (see Stability above) would bound the sweep. At current scale (small lists) this is latent, not live.
|
||||
|
||||
### [Low] Per-delivery TCP+TLS SMTP handshake
|
||||
|
||||
`MailKitSmtpClientWrapper` is deliberately one-connection-per-delivery, extensively documented with the factory seam left open for pooling (`MailKitSmtpClientWrapper.cs:12-38`). Correct tradeoff at current volume; noted only as the first knob if notification volume grows.
|
||||
|
||||
### [Low] Startup compiles every inbound method serially
|
||||
|
||||
`Host/Program.cs:383-387` runs Roslyn compilation per method in a foreach before `app.RunAsync()`. Fine at tens of methods; would stretch central startup (and failover-recovery window) at hundreds. `Parallel.ForEach` or lazy-only compilation are cheap options later.
|
||||
|
||||
### [Good] The hot-path hygiene elsewhere is exemplary
|
||||
|
||||
Bounded audit capture at the source with `ArrayPool` scratch buffers and cap+1 over-read detection (`AuditWriteMiddleware.cs:442-502`), fire-and-forget audit write with fault observation (`:358-359,380-398`), `EnableBuffering` skipped for bodyless requests (`:169-191`), name-keyed indexed repository lookups on the ESG hot path (`ExternalSystemClient.cs:540-550`), known-bad-method cache preventing per-request Roslyn recompiles with a 1000-entry flood cap (`InboundScriptExecutor.cs:38-63`), and per-system `HttpClient` factory clients with `MaxConnectionsPerServer` scoped to gateway-owned names only — explicitly avoiding process-global handler replacement (`ExternalSystemGateway/ServiceCollectionExtensions.cs:28-44,92-105`). No socket-exhaustion patterns anywhere: all four components use `IHttpClientFactory` (DelmiaNotifier's single-shot process makes its one `new HttpClient` correct).
|
||||
|
||||
## Findings — Dimension 3: Conventions & Spec Fidelity
|
||||
|
||||
### [High] Spec drift: per-system ESG timeout is specified but not implemented
|
||||
|
||||
`Component-ExternalSystemGateway.md:38` ("**Timeout**: Per-system timeout for all method calls") and `:101` are explicit, but `ExternalSystemDefinition` has no timeout field (`Commons/Entities/ExternalSystems/ExternalSystemDefinition.cs`) and the client acknowledges it: "ExternalSystemDefinition has no per-system Timeout field yet, so the configured DefaultHttpTimeout is the effective round-trip limit" (`ExternalSystemClient.cs:300-305`). Every external system — a fast local MES and a slow remote recipe manager alike — shares one global 30 s (`ExternalSystemGatewayOptions.cs:9`). Repo rules say the design doc is the spec and doc+code travel together; either add the column (entity + EF config + migration + CLI/UI) or amend the spec to the global-timeout reality.
|
||||
|
||||
### [Medium] Spec drift: path templates (`/recipes/{id}`) are never substituted
|
||||
|
||||
The spec's method definition shows a templated relative path (`Component-ExternalSystemGateway.md:43`), but `BuildUrl` (`ExternalSystemClient.cs:431-463`) concatenates the path verbatim and routes parameters exclusively to the query string (GET/DELETE) or JSON body (POST/PUT/PATCH). A method authored with `{id}` in its path literally calls `.../recipes/{id}`. Either implement `{param}` substitution (with the consumed parameter removed from body/query) or strike the example from the spec before someone authors against it.
|
||||
|
||||
### [Low] Spec drift: inbound error responses lack the documented `code` field
|
||||
|
||||
`Component-InboundAPI.md:91-97` documents failure bodies as `{"error": ..., "code": "SITE_UNREACHABLE"}`; every actual error body is `{ error }` only (`EndpointExtensions.cs:111-113,153-155,191-193,212-214,253-255`) and a routed site-unreachable failure surfaces as a generic 500 "Internal script error" (`InboundScriptExecutor.cs:382-387` swallows the `InvalidOperationException` from `RouteTarget.Call`, `RouteHelper.cs:172-174`). Machine-readable error codes matter to exactly the audience of this API (external integrators); implement or de-spec.
|
||||
|
||||
### [Low] OAuth2 SMTP is Microsoft-365-only despite a broader spec
|
||||
|
||||
`OAuth2TokenService` hardcodes `https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token` and the `outlook.office365.com/.default` scope (`OAuth2TokenService.cs:73,80`); the spec says "For Microsoft 365 **and other modern SMTP providers** that require OAuth2" (`Component-NotificationService.md:48`). A configurable authority/scope on `SmtpConfiguration` is a small change; until then the spec overpromises.
|
||||
|
||||
### [Low] `InboundApiEndpointFilter` freezes its options at construction
|
||||
|
||||
The filter is a singleton taking `IOptions<InboundApiOptions>` (`InboundApiEndpointFilter.cs:32-38`), so a live change to `MaxRequestBodyBytes` is ignored until restart — while the sibling `AuditWriteMiddleware` deliberately hot-reads `IOptionsMonitor<AuditLogOptions>` per request (`AuditWriteMiddleware.cs:97,149`). Inconsistent; use the monitor.
|
||||
|
||||
### [Good] Convention adherence is otherwise strong
|
||||
|
||||
Options pattern with startup validators throughout (`SmsOptionsValidator`, `NotificationOptions`); additive message-contract evolution respected and even documented at the parameter level ("this optional parameter trails the CancellationToken because it was appended additively", `InboundScriptExecutor.cs:243-246`); secrets consistently encrypted at rest via `EncryptedStringConverter` (ESG `AuthConfiguration` + DB `ConnectionString`: `Configurations/ExternalSystemConfiguration.cs:27-29,83-85`; SMTP credentials + Twilio AuthToken: `Configurations/NotificationConfiguration.cs:69,118`); connection strings never leak into error text (`DatabaseGateway.cs:298`, enforced in messages); credential scrubbing shared, not duplicated (`CredentialRedactor.cs`); API-key auth delegated to the shared `ZB.MOM.WW.Auth.ApiKeys` package (`InboundAPI.csproj:34`) with peppered-HMAC constant-time verification per spec (`Component-InboundAPI.md:66` — the package itself is outside this repo and was not independently verified), generic 401 messages, and byte-identical 403s for not-found vs not-in-scope with the DB lookup ordered *after* the in-memory scope check to close the timing oracle (`EndpointExtensions.cs:117-156`).
|
||||
|
||||
## Underdeveloped Areas
|
||||
|
||||
1. **Script trust boundary is static-only.** `ForbiddenApiChecker` is candid: semantic + reflection-gateway hardening, "not a true sandbox… genuine containment needs a runtime boundary" (`ForbiddenApiChecker.cs:24-29`). Inbound scripts run in-process on the central node with `Microsoft.CSharp` dynamic binder referenced (`InboundScriptExecutor.cs:172-184`). The deferred restricted-`AssemblyLoadContext`/out-of-process option remains the single biggest security-hardening gap on this boundary.
|
||||
2. **No cache-coherence mechanism for compiled handlers** (the High finding above) — no revision hash, no cluster invalidation message, no recompile-on-activation hook in `IActiveNodeGate`.
|
||||
3. **Per-SMS retry settings are dead fields.** `SmsConfiguration.MaxRetries/RetryDelay` are persisted and transported but never read; SMTP config governs retry for *all* notification types (`NotificationOutboxActor.cs:358-401`; documented as deferred in `Component-NotificationService.md:65`).
|
||||
4. **Twilio delivery is accept-only.** No status-callback webhook, no per-recipient delivery state (both documented as out-of-scope/future — `Component-NotificationService.md:93,102`). Also `AccountSid` is interpolated un-escaped into the request URI (`SmsNotificationDeliveryAdapter.cs:131`) — harmless for a valid SID, but a validation regex at config-save time would close the door.
|
||||
5. **DelmiaNotifier duplicate-delivery semantics undocumented** (Stability finding); AOT publish is Windows-only and the live smoke is manual (`2026-06-26-delmia-recipe-notifier-design.md:133-136,150-151`) — no CI artifact for the actual shipped binary.
|
||||
6. **Test gaps** (coverage is otherwise broad and genuinely good across all five test projects — endpoint gating, content-type case-sensitivity, schema-`$ref` runtime resolution, failover loop, both adapters, all four classifiers): nothing exercises the failover/staleness scenario for compiled handlers, nothing exercises an oversized outbound response body, and no test covers a multi-recipient SMS list with a mid-list transient (the duplicate-amplification path).
|
||||
7. **API key rotation** is enable/disable + scope-set only (`ManagementActor.cs:2685+`); a documented dual-key rotation procedure (create → migrate caller → disable) would round out the ops story — the `sbk_<keyId>_<secret>` design supports it, it's just unwritten.
|
||||
|
||||
## Prioritized Recommendations
|
||||
|
||||
1. **[High] Make the inbound handler cache self-healing.** Store the compiled delegate together with a hash of the script text; on each request compare against the freshly-fetched `ApiMethod.Script` and recompile on mismatch (also purging `_knownBadMethods`). One change fixes failover staleness, direct-SQL staleness, and stale-bad-method staleness simultaneously; the DB row is already fetched per request so the marginal cost is one string hash.
|
||||
2. **[High] Close the two ESG spec gaps** — per-system `Timeout` column (entity + EF migration + honored in `InvokeHttpAsync`) and either implement `{param}` path templating or amend `Component-ExternalSystemGateway.md`. Per repo rules, doc and code must travel together.
|
||||
3. **[Medium] Cap outbound response buffering** in `ExternalSystemClient` (`ResponseHeadersRead` + bounded read, reusing the audit-capture pattern).
|
||||
4. **[Medium] Bound the SMS blast radius**: short-circuit the recipient loop on first transient, and add bounded parallelism (or per-notification concurrency in the dispatch sweep) so one dead endpoint cannot stall the whole outbox.
|
||||
5. **[Medium] Replace colon-split `AuthConfiguration` parsing with structured JSON** (the entity already claims to be JSON) with a backward-compatible fallback.
|
||||
6. **[Medium] Handle abandoned inbound executions**: defer DI-scope disposal to actual handler completion, and add an orphaned-execution counter to the health snapshot.
|
||||
7. **[Low] Document DelmiaNotifier's at-least-once failover semantics** and verify the deployed `DelmiaRecipeDownload` script is idempotent.
|
||||
8. **[Low] Small consistency fixes**: `IOptionsMonitor` in `InboundApiEndpointFilter`; token parameter on `ErrorClassifier.IsTransient(Exception)`; configurable OAuth2 authority/scope; single-active-SMTP-config enforcement; implement or de-spec the inbound `code` error field.
|
||||
@@ -0,0 +1,126 @@
|
||||
# Architecture Review 07 — Management & Presentation Surface
|
||||
|
||||
**Domain:** Central UI (Blazor Server), Management Service (ManagementActor + HTTP API), CLI, Security & Auth
|
||||
**Reviewed:** `src/ZB.MOM.WW.ScadaBridge.CentralUI`, `src/ZB.MOM.WW.ScadaBridge.ManagementService`, `src/ZB.MOM.WW.ScadaBridge.CLI`, `src/ZB.MOM.WW.ScadaBridge.Security`, design docs `docs/requirements/Component-CentralUI.md`, `Component-ManagementService.md`, `Component-CLI.md`, `Component-Security.md`, and the corresponding test projects.
|
||||
|
||||
---
|
||||
|
||||
## Scope
|
||||
|
||||
Deep read of the ManagementActor (all 3,157 lines), the HTTP management/audit endpoints and authenticator, the full Security project (JWT service, cookie session validator, authorization policies, role mapper, service registration), the secured-writes end-to-end path (UI page → `SecuredWriteService` → actor handlers), the debug-stream path (`DebugStreamHub`, `DebugView.razor`), representative Blazor pages (Alarm Summary, Health, Secured Writes, Notification Report), UI-side trust-boundary services (`BrowseService`), the CLI HTTP client and command wiring, and deployed configuration in `docker/` and `deploy/wonder-app-vd03/`. Spec-vs-code comparison against the four component documents.
|
||||
|
||||
## Maturity Verdict
|
||||
|
||||
This is a **mature, carefully engineered surface with a small number of sharp edges**. The security layer is genuinely good — fail-fast key validation, a single shared claim builder for login and refresh, a deliberate and well-documented idle/refresh model, DB-persisted Data Protection keys, curated-vs-generic error mapping that prevents internal-detail leaks, and a secured-writes implementation with a real CAS race guard, server-side no-self-approval, and a TOCTOU protocol re-check at execute time. The Blazor code shows unusual discipline for disposal, thread marshalling, and best-effort degradation. The problems that remain are not sloppiness but *asymmetries*: a production deployment artifact that ships with authentication disabled, a per-site artifact-deploy option that is silently ignored server-side, secret-projection hygiene applied to SMTP/SMS/API keys but not to data connections / external systems / DB connection strings, and a secured-write "Expired" state that exists in the UI and spec but nowhere in the code. Test depth is thin relative to the actor's ~150-command surface.
|
||||
|
||||
---
|
||||
|
||||
## Findings — Dimension 1: Stability
|
||||
|
||||
### [High] S1 — Bundle import via `/management` can 504 while the import keeps applying
|
||||
`ManagementEndpoints.HandleRequest` Asks the actor with a configurable timeout that defaults to **30 s** (`ManagementEndpoints.cs:18`, `ResolveAskTimeout` at `:28-33`), and neither `docker/central-node-a/appsettings.Central.json` nor `deploy/wonder-app-vd03/appsettings.Central.json` sets `ScadaBridge:ManagementService:CommandTimeout`. The CLI's bundle commands use a **5-minute client timeout** (`CLI/Commands/BundleCommands.cs:16`) precisely because imports are slow — but the server-side Ask expires at 30 s, returns `504 TIMEOUT` (`ManagementEndpoints.cs:107-113`), **and the actor's `ProcessCommand` task keeps running** (PipeTo is not cancelled), so `HandleImportBundle` (`ManagementActor.cs:2949-3015`) continues applying the bundle after the caller was told it timed out. Failure scenario: an operator imports a large bundle, gets a 504, retries — under `--on-conflict rename` the second import produces a duplicate `-imported-<stamp>` artifact set; under `overwrite` it double-applies. The same ambiguity applies to fleet-wide `DeployArtifacts`. Fix: raise the configured timeout for transport/deploy commands (per-command timeout in the envelope), or make import idempotent on a client-supplied import id.
|
||||
|
||||
### [High] S2 — Pending secured writes never expire; the "Expired" status is fictional
|
||||
The UI's terminal-status list and badge mapping include `"Expired"` (`CentralUI/Components/Pages/Operations/SecuredWrites.razor:265, 411`), and Component-CentralUI.md §Secured Writes lists "Executed / Failed / Rejected / Expired" — but **no code anywhere sets a row to Expired** (repo-wide grep: the only two hits are the UI file). `HandleApproveSecuredWrite` (`ManagementActor.cs:1099-1246`) will approve a `Pending` row of any age. Failure scenario on a live SCADA surface: an Operator submits a setpoint write, it sits unverified over a weekend while the process moves to a different state, and a Verifier approves the stale value days later — the confirmation dialog (`SecuredWrites.razor:342-347`) shows the value but not its age. A pending-write TTL (with a sweeper or an at-approve age check) is the obvious missing piece; the status enum already reserved the name.
|
||||
|
||||
### [Medium] S3 — Poll timers can overlap their own refreshes
|
||||
`Health.razor:521-528` and `AlarmSummary.razor:283-291` both use `System.Threading.Timer` with a fixed period whose callback does `InvokeAsync(async () => { await RefreshAsync(); ... })` with no reentrancy guard. `AlarmSummary.RefreshAsync` fans out per-instance snapshot Asks (site offline ⇒ each fetch waits its full Ask timeout); with a 15 s period and a >15 s degraded fetch, ticks stack up on the dispatcher and the page issues overlapping fan-outs against an already-struggling site. A `PeriodicTimer` loop or an `Interlocked` in-flight flag would bound this. (Disposal itself is handled correctly on both pages — `Health.razor:757-760`, `AlarmSummary.razor:382`.)
|
||||
|
||||
### [Medium] S4 — 200 MB request bodies are buffered as strings, then multiplied
|
||||
The management endpoint raises the body cap to **200 MB** for transport bundles (`ManagementEndpoints.cs:50, 58-62`) and then reads the whole body with `StreamReader.ReadToEndAsync()` into a string (`:77-81`); the actor then base64-decodes to a `byte[]` (`ManagementActor.cs:3105-3119`), wraps in a `MemoryStream`, and export goes the other way — `stream.CopyToAsync(ms); ms.ToArray(); Convert.ToBase64String(bytes)` (`ManagementActor.cs:2902-2905`), which is ~4 copies of the artifact resident simultaneously (bytes, MemoryStream buffer, base64 string, JSON envelope string). A concurrent pair of large imports on a central node is a realistic OOM/LOH-fragmentation path. Streaming multipart upload (as the audit export already streams downloads — `AuditEndpoints.cs:183+`) is the right long-term shape.
|
||||
|
||||
### [Low] S5 — Fire-and-forget SignalR sends swallow failures silently
|
||||
`DebugStreamHub.SubscribeInstance` pushes events with `_ = hubClients.Client(connectionId).SendAsync(...)` (`DebugStreamHub.cs:204-221`). Deliberate (documented) — but a persistently failing client produces zero telemetry. Elsewhere the hub is exemplary: transient-hub-instance pitfall avoided via `IHubContext` (`:198-200`), unsubscribe on disconnect (`:255-259`).
|
||||
|
||||
### Positive stability observations
|
||||
- `DebugView.razor` is a model Blazor Server component: `_disposed` flag set before stream stop, `SafeInvokeAsync` guarding `ObjectDisposedException`, dictionary mutation marshalled onto the render thread with the race explicitly documented (`DebugView.razor:511-544, 589-611`).
|
||||
- Failover transparency is real: Data Protection keys persist to the config DB (`ConfigurationDatabase/ServiceCollectionExtensions.cs:75-76`), the cookie principal is self-contained, and the ManagementActor runs on every central node (stateless, per spec).
|
||||
- `AlarmSummaryService` implements the spec'd fan-out faithfully: `SemaphoreSlim(8)` cap, partial-results tolerance, caller-cancel propagation (`AlarmSummaryService.cs:26, 68-84, 112-123`).
|
||||
|
||||
---
|
||||
|
||||
## Findings — Dimension 2: Performance
|
||||
|
||||
### [Medium] P1 — Every CLI/management/audit HTTP request performs a full LDAP bind
|
||||
`ManagementAuthenticator.AuthenticateAsync` binds against LDAP and re-runs role mapping on **every** request (`ManagementAuthenticator.cs:107-118`); the audit endpoints and `DebugStreamHub.OnConnectedAsync` (`DebugStreamHub.cs:118-137`) do the same. This is the documented stateless design, but there is no negative-result cache, rate limit, or lockout — so (a) a scripted CLI loop hammers the directory once per command, and (b) `POST /management` and `POST /auth/token` (`AuthEndpoints.cs:99-148`) are unauthenticated LDAP-bind oracles usable for password spraying at whatever rate the LDAP server tolerates. A short-TTL bind cache or a per-IP failure throttle would address both.
|
||||
|
||||
### [Medium] P2 — Unbounded list handlers materialize whole tables
|
||||
Several read handlers return `GetAll*` with no paging: `HandleListTemplates` (`ManagementActor.cs:507-511`), `HandleListInstances` (`:718-729`, post-filtered in memory for scoped users), `HandleQueryDeployments` unfiltered branch loads *all* deployment records *and all instances* (`:2031-2060` — the N+1 was fixed, but it is still two full-table loads per query), and `HandleExportBundle` loads every template/script/system/site/instance to resolve names (`:2818-2831`). Fine at current fleet size; a 10k-instance estate will feel it on every CLI `instance list`. The audit surface shows the house already knows the right pattern (keyset paging, `MaxPageSize` 1000 — `AuditEndpoints.cs:59-66`).
|
||||
|
||||
### [Medium] P3 — `ListSecuredWrites` hard-caps at 200 rows with no paging
|
||||
`HandleListSecuredWrites` does `QueryAsync(status, siteId, skip: 0, take: 200)` (`ManagementActor.cs:1280-1286`) and the page loads *everything* then splits pending/history client-side (`SecuredWrites.razor:303-308`). Once >200 rows exist, history silently truncates — an **audit-adjacent** table silently losing visibility of older decisions. No pagination controls exist on the page.
|
||||
|
||||
### [Low] P4 — Alarm Summary re-sorts/filters per render
|
||||
`FilteredRows().ToList()` runs inside the render body (`AlarmSummary.razor:170`) — recomputed on every `StateHasChanged`, including each 15 s tick. Cheap at hundreds of rows; worth memoizing if sites grow. No virtualization on any table in the domain, but page sizes are mostly bounded (Notification Report pages at 50 — `NotificationReport.razor:352`).
|
||||
|
||||
### Positive performance observations
|
||||
- Audit export streams server-side in 1000-row pages and the CLI streams to disk with `ResponseHeadersRead` (`AuditEndpoints.cs:66`, `ManagementHttpClient.cs:209-210`).
|
||||
- The actor never blocks its dispatcher: `HandleEnvelope` role-checks synchronously then `PipeTo`s the async work (`ManagementActor.cs:97-120`), so the single actor instance is not a mailbox bottleneck for I/O-bound commands — commands execute concurrently on the thread pool with per-command DI scopes (`:127-132`). This is the correct pattern and worth calling out as such.
|
||||
|
||||
---
|
||||
|
||||
## Findings — Dimension 3: Conventions & Correctness
|
||||
|
||||
### [Critical] C1 — Production deployment artifact ships with authentication disabled
|
||||
`deploy/wonder-app-vd03/appsettings.Central.json:40` sets `"DisableLogin": true` (and `:38` `"RequireHttpsCookie": false`). Per `Component-Security.md` §Dev Disable-Login Flag there is **no environment guard** — only a startup log warning — and the bypass is honoured by *every* surface: the cookie scheme (`AutoLoginAuthenticationHandler`), `POST /management` (`ManagementAuthenticator.cs:52-76`), the audit REST endpoints, and the debug-stream hub (`DebugStreamHub.cs:82-91`). On this host, every anonymous HTTP request is authenticated as `multi-role` with **all roles system-wide — including Operator and Verifier**, which nullifies the two-person secured-write control on a box that can relay MxGateway device writes. `deploy/` is described by CLAUDE.md as "Production/on-host deployment artifacts". Even if wonder-app-vd03 is a staging box today, an unauthenticated SCADA control surface should not be one config-file copy away from production. Recommendation: add an environment guard (refuse `DisableLogin=true` unless `ASPNETCORE_ENVIRONMENT=Development` or an explicit `IUnderstandThisDisablesAuth` second key), and scrub the deploy artifact.
|
||||
|
||||
### [High] C2 — `deploy artifacts --site-id` is silently ignored; artifacts always deploy fleet-wide
|
||||
`MgmtDeployArtifactsCommand` carries `int? SiteId` (`Commons/Messages/Management/DeploymentCommands.cs:3`) and both CLI commands send it (`CLI/Commands/SiteCommands.cs:209`, `CLI/Commands/DeployCommands.cs:51`; documented in `Component-CLI.md` and the CLI README as `[--site-id <id>]`). But `HandleDeployArtifacts` never reads `cmd.SiteId` — it calls `svc.DeployToAllSitesAsync(user)` unconditionally (`ManagementActor.cs:2006-2013`), and `ArtifactDeploymentService.DeployToAllSitesAsync(string user, CancellationToken)` (`ArtifactDeploymentService.cs:227-229`) has no site parameter. Two consequences: (1) **contract violation** — an operator targeting one site pushes shared scripts/external systems/DB connections to *every* site; (2) **authorization gap** — the command also has no `EnforceSiteScope` call, so a *site-scoped* Deployer can trigger a fleet-wide artifact deployment. The Central UI's per-site "Deploy Artifacts" button presumably uses a different path; the management surface needs `DeployToSiteAsync(siteId)` plus a scope check.
|
||||
|
||||
### [High] C3 — Secret projection is inconsistent: connection credentials leak to any authenticated user and into the audit log
|
||||
The actor is scrupulous about SMTP credentials, Twilio auth tokens, and API-key hashes — all projected to credential-free shapes before response *and* audit (`SmtpConfigPublicShape` `ManagementActor.cs:1787-1801`, `SmsConfigPublicShape` `:1841-1853`, `HandleListApiKeys` `:1955-1967`). But three sibling entity families get no such treatment:
|
||||
- **Data connections** — `HandleListDataConnections`/`HandleGetDataConnection` (`:1401-1416`) return the raw entity, whose `PrimaryConfiguration`/`BackupConfiguration` JSON embeds `OpcUaUserIdentityConfig.Password` and `CertificatePassword` in plaintext (`Commons/Types/DataConnections/OpcUaUserIdentityConfig.cs:13,17`). List/Get are **read-only commands with no role gate** (`GetRequiredRole` default `:240-241`), so a Viewer- or Operator-only principal can read every site's OPC UA credentials via one CLI call. Create/Update also audit the full entity as afterState (`:1429, :1445`), persisting the passwords into the audit log readable by `OperationalAudit` roles.
|
||||
- **External systems** — `AuthConfiguration` (API keys / Basic credentials) returned raw to any authenticated user and audited verbatim (`:1587-1624`).
|
||||
- **Database connections** — full `ConnectionString` (SQL credentials) returned raw in List/Get (`:2526-2536`); the audit shape here *was* trimmed to `{Id, Name}` (`:2544, :2557`), which shows the team knows the pattern — it just wasn't applied to the response.
|
||||
Recommendation: apply the established `*PublicShape` pattern (config-with-secrets-elided + `HasCredentials` flag) to all three, and gate Get-with-secrets behind Admin if the UI editor needs round-tripping.
|
||||
|
||||
### [Medium] C4 — Spec drift: browse/verify/cert-trust commands are documented as ManagementActor commands but only exist UI-side
|
||||
`Component-ManagementService.md` §Remote Queries lists `BrowseNodeCommand`, `SearchAddressSpaceCommand`, `VerifyEndpointCommand`, `TrustServerCertCommand`/`RemoveServerCertCommand`/`ListServerCertsCommand` with Design/Admin role gates. The actor's `DispatchCommand` has **no handlers for any of them** — they fall through to `NotSupportedException` (`ManagementActor.cs:423`), and since `ManagementCommandRegistry` auto-registers every `*Command` in the namespace by reflection (`ManagementCommandRegistry.cs:66-79`), a CLI `POST /management {"command":"TrustServerCert"}` deserializes fine and then returns a generic `COMMAND_FAILED` internal error. The real implementations live in Central UI services (`BrowseService.cs`, `EndpointVerificationService.cs`, `CertManagementService.cs`) with hand-rolled `HasClaim` role guards (`BrowseService.cs:48-55, 103-110`). Net effect: the CLI cannot browse address spaces or manage cert trust (a gap for the automation story the CLI exists for), and the component doc misdescribes the enforcement point. Fix the doc or add actor handlers.
|
||||
|
||||
### [Medium] C5 — Role-name comparison asymmetry between UI policies and the actor (acknowledged, still live)
|
||||
ASP.NET `RequireClaim` values are compared **ordinal case-sensitive** (`AuthorizationPolicies.cs:143-169`), while the ManagementActor compares roles **OrdinalIgnoreCase** (`ManagementActor.cs:105, 440` etc.), and `ValidateMappingRole` deliberately preserves the stored row's verbatim casing while validating membership case-insensitively — the comment at `ManagementActor.cs:1904-1911` explicitly defers the asymmetry. Failure scenario: an admin creates a mapping row `deployer` (lowercase) via CLI; the affected users pass every ManagementActor gate but fail every `RequireDeployment` UI policy — a confusing split-brain where CLI works and the UI 403s. Since the write path already validates against `Roles.All`, the cheap fix is to **canonicalize the stored casing** at that single write path.
|
||||
|
||||
### [Medium] C6 — Area-management role gate contradicts both design docs
|
||||
`CreateAreaCommand`/`UpdateAreaCommand`/`DeleteAreaCommand` are gated **Designer** (`ManagementActor.cs:187, 207`), but `Component-ManagementService.md` §Authorization ("Admin role required for: site management, area management…") and `Component-Security.md` §Administrator ("Manage area definitions per site") both assign areas to **Admin**, and Component-CentralUI.md titles the workflow "Area Management (Admin Role)". Either the code or three documents are wrong; per the repo's own propagate-together rule this should be reconciled in one change.
|
||||
|
||||
### [Low] C7 — `HandleQueryEventLogs` passes `InstanceName` into a positional slot commented "InstanceId filter"
|
||||
`ManagementActor.cs:2757-2766` builds `EventLogQueryRequest(..., cmd.InstanceName, /* InstanceId filter */ ...)`. If the site-side filter expects an id, name-based CLI filtering silently matches nothing; if it expects a name, the comment/param name is wrong. Worth a five-minute verification against `SiteEventLogging`.
|
||||
|
||||
### [Low] C8 — Stale CLAUDE.md note on SecuredWrite `SourceNode`
|
||||
CLAUDE.md says "SecuredWrite audit rows currently leave `SourceNode` NULL — a logged follow-up", but `EmitSecuredWriteAuditAsync` now routes through `ICentralAuditWriter`, which stamps `SourceNode` from `INodeIdentityProvider` (documented at `ManagementActor.cs:999-1013`). The follow-up appears done; the project memory should be updated so future sessions don't re-fix it.
|
||||
|
||||
### Positive convention observations
|
||||
- **No third-party Blazor component libraries** — the CentralUI csproj references only Roslyn packages and the in-family `ZB.MOM.WW.Theme` (`ZB.MOM.WW.ScadaBridge.CentralUI.csproj:19-21`); tables, tree views, dialogs, toasts, and the KPI chart are all custom, per the repo rule.
|
||||
- Options pattern honoured throughout; component libraries take no `IConfiguration` (`Security/ServiceCollectionExtensions.cs:26-37`); `SecurityOptionsValidator` fail-fast with a genuinely well-reasoned invariant (refresh threshold < idle window, `SecurityOptions.cs:118-142`).
|
||||
- Error-surface hygiene is exemplary: `MapFault` returns curated `ManagementCommandException` messages verbatim and collapses everything else to a correlation-id-bearing generic message (`ManagementActor.cs:137-158`) — no raw exception text reaches the CLI/UI for unanticipated faults.
|
||||
- The secured-write approve handler is defensive in exactly the right places: no-self-approval **before** the CAS (`:1113-1115`), CAS via `TryMarkApprovedAsync` (`:1121`), MxGateway protocol **re-checked at execute** against submit-time reconfiguration (`:1138-1165`), value-type and decode failures contained so a row is never stuck `Approved` (`:1171-1202`), transport exceptions contained (`:1206-1226`).
|
||||
- `ResolveRolesCommand` deliberately not dispatched with the security rationale documented (`:418-422`) — good trust-boundary thinking.
|
||||
- JWT implementation pins issuer/audience, zero clock skew, disables in/outbound claim mapping symmetrically, enforces ≥256-bit keys at construction (`JwtTokenService.cs:54-78, 124-171`); the cookie path's login and mid-session refresh share `SessionClaimBuilder` so claim shapes cannot drift (`CookieSessionValidator.cs:200-210`, `AuthEndpoints.cs:84-89`); refresh failures fail-open to *current* roles and only idle-timeout rejects (`CookieSessionValidator.cs:132-152`) — matching the documented policy exactly.
|
||||
- LDAP is delegated to the shared hardened `ZB.MOM.WW.Auth.Ldap` library (bind-then-search, escaping, fail-closed — per `Security/ServiceCollectionExtensions.cs:49-59`); no bespoke LDAP string assembly exists in this repo, so injection surface is centralized in the audited library.
|
||||
|
||||
---
|
||||
|
||||
## Findings — Dimension 4: Underdeveloped Areas
|
||||
|
||||
1. **Secured-write lifecycle** — no expiry (S2), no paging (P3), history visible to any authenticated user (`ListSecuredWritesCommand` falls through to the no-role gate, `ManagementActor.cs:238-241`) — tag values on the history table may themselves be process-sensitive. Also nothing prevents the *same person* holding both Operator and Verifier roles via two LDAP groups; the control is only "not the same row's submitter", which is the documented design but worth restating as a deployment-configuration hazard.
|
||||
2. **ManagementService test depth** — 11 test files (`tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/`) against a 3,157-line actor dispatching ~150 commands. `SecuredWriteHandlerTests` exists (good — covers the flagship control), but there is no visible exhaustive matrix test asserting `GetRequiredRole` for every command type — precisely the test that would have caught C2's missing scope check and would freeze the authz table against regressions. Security.Tests is 9 files; CookieSessionValidator is covered.
|
||||
3. **CLI.Tests is not in the slnx** (`ZB.MOM.WW.ScadaBridge.slnx:24` includes only the CLI project) — a documented gotcha, but it remains a standing risk that a green solution run hides never-compiled CLI tests.
|
||||
4. **Deferred follow-ups visible in code/docs**: aggregated live alarm stream for Alarm Summary (poll-only today — Component-CentralUI.md §Alarm Summary "logged as a follow-up"); central persistence of cert trust; live LDAP group re-query mid-session (Component-Security.md "Residual limitation"); Parquet export returns 501 (`AuditEndpoints.cs:202-210`); `audit verify-chain` is a no-op pending the hash chain. All are honestly documented — the gap is that no single tracking list ties them together.
|
||||
5. **Login endpoints lack throttling** — `/auth/login`, `/auth/token`, `/management`, and the debug hub all do per-request LDAP binds with no failure throttle or lockout (P1). `/auth/login` also reflects the LDAP failure kind into the redirect query string (`AuthEndpoints.cs:41-45`); confirm `LdapAuthFailureMessages` does not distinguish "unknown user" from "bad password" (user enumeration).
|
||||
6. **Accessibility/UX debt** — sortable headers are `<th role="button" @onclick>` without keyboard handlers or `aria-sort` (`AlarmSummary.razor:176-179`); status conveyed by color+text badges is fine, but the custom tables have no `scope`/caption markup. Minor for an internal tool, but it is uniform debt across every custom grid.
|
||||
7. **`docker/central-node-a/logs/` committed runtime logs** sit inside the deploy topology folder (noticed while scanning configs) — noise that will eventually leak something.
|
||||
|
||||
---
|
||||
|
||||
## Prioritized Recommendations
|
||||
|
||||
1. **[Critical] Guard `DisableLogin` and scrub `deploy/wonder-app-vd03`** — refuse the flag outside a Development environment (or require a second explicit ack key); remove `DisableLogin: true` from the on-host deploy artifact. (C1)
|
||||
2. **[High] Implement per-site artifact deployment or reject the option** — honour `MgmtDeployArtifactsCommand.SiteId` (add `DeployToSiteAsync`) and add `EnforceSiteScope`; until then, make the handler *fail* when SiteId is supplied rather than silently deploying fleet-wide. (C2)
|
||||
3. **[High] Apply the `*PublicShape` secret-elision pattern to DataConnection, ExternalSystem, and DatabaseConnection reads and audits**; consider role-gating those List/Get commands. (C3)
|
||||
4. **[High] Add a pending-secured-write TTL** that transitions stale rows to the already-reserved `Expired` status, plus paging on the list. (S2, P3)
|
||||
5. **[High] Fix the long-command timeout mismatch** — per-command Ask timeouts (transport/deploy get minutes), configure `CommandTimeout` in shipped configs, and/or make bundle import idempotent on an import id so a 504-then-retry is safe. (S1)
|
||||
6. **[Medium] Add a `GetRequiredRole` matrix test** enumerating every registered command and asserting its gate — freezes the authorization table and catches future C2-style gaps mechanically.
|
||||
7. **[Medium] Canonicalize role casing at the mapping write path** to close the UI-vs-actor case-sensitivity split. (C5)
|
||||
8. **[Medium] Reconcile the area-management role gate** (code says Designer, three docs say Admin) and the browse/cert-command placement (docs say actor, code says UI-only) in one doc+code pass. (C4, C6)
|
||||
9. **[Medium] Add LDAP-bind failure throttling** (per-IP or per-username backoff) in `ManagementAuthenticator` and the auth endpoints. (P1)
|
||||
10. **[Low] Housekeeping** — reentrancy guards on the two poll timers (S3), update the stale SecuredWrite `SourceNode` memory note (C8), verify the `InstanceName`/`InstanceId` event-log filter slot (C7), pull `docker/*/logs` out of the tree, and put CLI.Tests into the slnx or CI explicitly.
|
||||
@@ -0,0 +1,185 @@
|
||||
# Architecture Review 08 — Cross-Cutting Conventions, Test Posture, Underdeveloped Areas
|
||||
|
||||
Reviewer domain: cross-cutting conventions, code organization, test posture, repo-wide underdeveloped-areas sweep, docs-code drift. Individual subsystems are covered in depth by the other seven reviews; this one covers the seams and the whole.
|
||||
|
||||
## Scope
|
||||
|
||||
- `src/` — all 26 projects: Commons structure, project reference graph, options pattern, POCO/repository conventions, UTC and correlation-ID discipline, message-contract evolution.
|
||||
- `tests/` — all 30 test projects: breadth, quality sampling, integration/performance posture, slnx membership.
|
||||
- Repo-wide grep sweep for TODO / FIXME / HACK / NotImplementedException / "deferred" / "follow-up" / "not shipped", plus deferred-work inventory from `docs/plans/` and `CLAUDE.md`.
|
||||
- Docs-code sync spot checks: README component table, `docs/requirements/Component-*.md` (KpiHistory, StoreAndForward, Transport, ClusterInfrastructure), `docs/components/` reference docs.
|
||||
|
||||
Method: `.slnx` and `.csproj` graph extraction, directory-structure listing, targeted greps over `src/`, `tests/`, `docs/`, and direct reads of the files behind every claim below. Read-only except this report.
|
||||
|
||||
## Maturity Verdict
|
||||
|
||||
This is an unusually disciplined codebase for its size (~190k src LOC, ~156k test LOC, ~5,000 test cases across 30 projects). The documented conventions are not aspirational — they are actually followed: Commons is genuinely persistence-ignorant, the Types/Interfaces/Entities/Messages hierarchy is real, UTC discipline is total (zero `DateTime.Now` in src), correlation IDs are on every cross-cluster request/response plus the `ManagementEnvelope`, and the project reference graph has no site→central contamination. The TODO surface is astonishingly small (two real TODOs in all of src) and almost every deferral is deliberately documented at the deferral site with a pointer to the design doc. The gaps that exist are narrow and specific: one genuine spec-vs-code contradiction (Transport claims Area membership travels in bundles; the exporter always emits `null`), the CLI.Tests slnx exclusion that keeps 279 tests out of every solution-level test run, options-validation coverage on only ~6 of ~20 option classes, and a thin PerformanceTests project that is a correctness-at-small-scale suite rather than a performance suite. Deferred-work inventory is sizeable (~20 tracked items) but almost entirely intentional and logged; only two or three items carry real operational risk.
|
||||
|
||||
---
|
||||
|
||||
## 1. Conventions & Organization
|
||||
|
||||
### 1.1 Commons hierarchy — followed, with undocumented extensions [Low]
|
||||
|
||||
`src/ZB.MOM.WW.ScadaBridge.Commons/` follows the documented `Types/`, `Interfaces/`, `Entities/`, `Messages/` layout with domain-area subfolders exactly as `Component-Commons.md` describes (284 .cs files; e.g. `Entities/{Audit,Deployment,ExternalSystems,InboundApi,Instances,Kpi,Notifications,Schemas,Scripts,SecuredWrites,Security,Sites,Templates}`, `Messages/{Artifacts,Audit,Communication,DataConnection,DebugView,Deployment,Health,InboundApi,Instance,Integration,Lifecycle,Management,Notification,RemoteQuery,ScriptExecution,Streaming}`).
|
||||
|
||||
Three top-level folders exist beyond the documented four: `Observability/`, `Serialization/` (protocol endpoint-config serializers), and `Validators/`. These are reasonable homes, but the convention doc lists only the four canonical folders. Either document them in `Component-Commons.md` or fold them in (`Serialization` arguably belongs under `Types/`).
|
||||
|
||||
### 1.2 POCO persistence-ignorance — verified clean [Pass]
|
||||
|
||||
Grep for EF Core / DataAnnotations attributes across Commons returns only one hit, and it is an XML-doc `<see cref>` to `DbUpdateConcurrencyException` in `src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IDeploymentManagerRepository.cs:64` — a documentation reference, not a dependency. The Commons csproj has zero EF package references (its only PackageReference is the shared `ZB.MOM.WW.Audit` lib). Entities are genuinely persistence-ignorant.
|
||||
|
||||
### 1.3 Repository interfaces vs implementations — 1:1, with a deliberate site-side exception [Low]
|
||||
|
||||
14 repository interfaces in `Commons/Interfaces/Repositories/`, 14 implementation files in `ConfigurationDatabase/Repositories/` — exact match. Two additional implementations live in `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/` (`SiteExternalSystemRepository`, `SiteNotificationRepository`): read-only SQLite-backed site-side implementations that throw `NotSupportedException` on writes. This is a sensible pattern (same interface, site-local store), but it is an undocumented exception to the "implementations in Configuration Database" rule.
|
||||
|
||||
**Vestigial seam** [Low/Medium]: `SiteNotificationRepository` (registered scoped in `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs:65`) reads the site-local `notification_lists` table — but per the central-only notification design, `SiteStorageService.PurgeCentralOnlyNotificationConfigAsync` (`src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs:703`) deliberately empties that table on every artifact apply. So the registered repository can only ever return empty results, and `StoreNotificationListAsync` (line 721) still exists as a write path into a table the design says must stay empty. The purge is well-commented and correct as a security cleanup; the live read/write surface around an intentionally-empty table is dead weight and a trap for a future maintainer. Recommend removing `StoreNotificationListAsync`/list-read paths or marking them `[Obsolete]` with the design rationale.
|
||||
|
||||
### 1.4 Project reference graph — sane; no site→central contamination [Pass, one Low note]
|
||||
|
||||
Extracted from all 26 csproj files:
|
||||
|
||||
- **Commons is a leaf** (no project refs). Everything depends on it; it depends on nothing.
|
||||
- **Site-side projects never reference central-only projects.** `SiteRuntime` → Commons, Communication, DataConnectionLayer, ScriptAnalysis, HealthMonitoring, SiteEventLogging, StoreAndForward — critically, **no** reference to `ConfigurationDatabase` (EF/central SQL), `ManagementService`, `CentralUI`, or `TemplateEngine`. Same for DataConnectionLayer, StoreAndForward, SiteEventLogging.
|
||||
- **Central-only stack is correctly layered**: AuditLog/SiteCallAudit/Transport → ConfigurationDatabase; CentralUI → the central components + ManagementService; Host references everything (composition root, as designed).
|
||||
- **CLI → Commons only** — correctly thin (HTTP client to the Management API).
|
||||
- **DelmiaNotifier → nothing** — correctly standalone/BCL-only.
|
||||
|
||||
One smell [Low]: `Communication` → `HealthMonitoring` (`src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs:13`). A transport layer depending on a monitoring component is a mild layering inversion; the health-aggregator abstraction it consumes could live in Commons. Cosmetic [Low]: `ManagementService.csproj` uses a Windows-style `..\` separator for its InboundAPI reference while every other reference in the repo uses `../` — harmless to MSBuild, inconsistent style.
|
||||
|
||||
### 1.5 Options pattern — ownership right, validation coverage partial [Medium]
|
||||
|
||||
Every component owns its options class in its own project (31 `*Options.cs` files across src; none in Commons) — the ownership convention is fully honored. But **eager validation is wired for only ~6 components**: `IValidateOptions<>`/`ValidateDataAnnotations`/`ValidateOnStart` appear only in AuditLog, KpiHistory, NotificationOutbox (+SmsOptions), HealthMonitoring, Security, ClusterInfrastructure, plus the Host `StartupValidator`. Components binding options without a validator include Communication, DataConnectionLayer, DeploymentManager, ExternalSystemGateway, InboundAPI, ManagementService, NotificationService, SiteCallAudit, SiteEventLogging, SiteRuntime, StoreAndForward, and Transport (e.g. `src/ZB.MOM.WW.ScadaBridge.Transport/ServiceCollectionExtensions.cs:25` — `AddOptions<TransportOptions>().BindConfiguration(...)` with no `.ValidateOnStart()`). A malformed `appsettings` section in those components surfaces at first use rather than at startup, which is exactly what the Host's readiness-gating philosophy is supposed to prevent. Recommend a sweep adding validators (the HealthMonitoring/ClusterInfrastructure `IValidateOptions` implementations are ready-made patterns to copy).
|
||||
|
||||
### 1.6 UTC timestamps — total compliance [Pass]
|
||||
|
||||
Zero occurrences of `DateTime.Now`/`DateTimeOffset.Now` (non-Utc) in all of src (grep across .cs and .razor). Every timestamp uses `UtcNow` or stores `datetime2`/ISO-8601 UTC.
|
||||
|
||||
### 1.7 Correlation IDs — convention followed [Pass]
|
||||
|
||||
96 `CorrelationId` declarations across `Commons/Messages/` cover the full cross-cluster request/response surface (RouteToInstance, Get/SetAttribute(s), Subscribe*, WriteTag*, ParkedMessage*, NotificationOutboxQueries, SiteCallQueries, EventLogQuery, DeploymentStateQuery, WaitForAttribute, etc.). Management commands themselves carry no per-command field, but they always travel inside `ManagementEnvelope(User, Command, CorrelationId)` with `ManagementSuccess`/`ManagementError`/`ManagementUnauthorized` echoing it (`Commons/Messages/Management/ManagementEnvelope.cs:7-11`) — the convention holds at the envelope level.
|
||||
|
||||
### 1.8 Message contract evolution — convention by discipline, not enforcement [Low]
|
||||
|
||||
Messages are plain C# records with defaulted trailing parameters (additive-friendly, e.g. `CreateDataConnectionCommand(..., string? BackupConfiguration = null, int FailoverRetryCount = 3)` in `Commons/Messages/Management/DataConnectionCommands.cs`). The Akka HOCON in `Host/Actors/AkkaHostedService.cs:233` configures no custom serializer, so ClusterClient traffic rides Akka.NET's default JSON serialization — tolerant of additive fields. The gRPC side does have contract-lock tests (`tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Protos/CachedTelemetryProtoTests.cs`, `PullAuditEventsProtoTests.cs`), but there is no equivalent guard (snapshot/round-trip tests against old shapes) for the ClusterClient record contracts. Given the "system-wide artifact version skew across sites is supported" design decision, a small contract-compatibility test suite for the highest-traffic ClusterClient messages would convert the additive-only rule from convention to enforcement.
|
||||
|
||||
### 1.9 Naming/namespace consistency [Pass]
|
||||
|
||||
Consistent `ZB.MOM.WW.ScadaBridge.<Component>` namespaces mirroring folder structure in every file sampled (SiteRuntime, Transport, Commons, KpiHistory, ManagementService). Test projects mirror src names 1:1 plus IntegrationTests/PerformanceTests/PlaywrightTests/Transport.IntegrationTests.
|
||||
|
||||
---
|
||||
|
||||
## 2. Test Posture & Quality
|
||||
|
||||
### 2.1 Breadth — every component has a test project; volume is high
|
||||
|
||||
30 test projects; ~5,050 `[Fact]`/`[Theory]` methods; ~156k test LOC against ~190k src LOC (src figure inflated by ~75k in ConfigurationDatabase, mostly EF migrations). Per-project counts (files / LOC / test methods):
|
||||
|
||||
| Project | Tests | Notes |
|
||||
|---|---|---|
|
||||
| CentralUI.Tests | 829 | bUnit page/component tests — largest suite |
|
||||
| Commons.Tests | 465 | includes KpiSeriesBucketer, codecs, validators |
|
||||
| SiteRuntime.Tests | 457 | Akka TestKit behavioral (see 2.2) |
|
||||
| TemplateEngine.Tests | 433 | |
|
||||
| CLI.Tests | 279 | **excluded from slnx** (see 2.4) |
|
||||
| AuditLog.Tests | 237 | |
|
||||
| ManagementService.Tests | 234 | |
|
||||
| ConfigurationDatabase.Tests | 219 | incl. migration tests |
|
||||
| InboundAPI.Tests | 218 | |
|
||||
| Communication.Tests | 211 | incl. proto contract locks |
|
||||
| DataConnectionLayer.Tests | 190 | |
|
||||
| Host.Tests | 142 | incl. CompositionRootTests |
|
||||
| Security.Tests | 133 | |
|
||||
| NotificationOutbox.Tests / Transport.Tests | 122 / 124 | |
|
||||
| StoreAndForward.Tests | 116 | |
|
||||
| DeploymentManager.Tests | 98 | |
|
||||
| HealthMonitoring.Tests | 84 | |
|
||||
| IntegrationTests / ExternalSystemGateway.Tests | 69 / 69 | |
|
||||
| Transport.IntegrationTests | 60 | |
|
||||
| SiteEventLogging.Tests | 54 | |
|
||||
| ScriptAnalysis.Tests / NotificationService.Tests / CentralUI.PlaywrightTests | 40 / 36 / 36 | |
|
||||
| SiteCallAudit.Tests | 31 | thinnest per src LOC (1,642 loc / 31 tests) |
|
||||
| DelmiaNotifier.Tests | 23 | |
|
||||
| ClusterInfrastructure.Tests | 15 | src is only 191 LOC (wiring lives in Host) — proportionate |
|
||||
| KpiHistory.Tests | 14 | bucketer/query/chart tests live with owners (Commons.Tests, CentralUI.Tests) — coverage is distributed, not missing |
|
||||
| PerformanceTests | 10 | see 2.3 |
|
||||
|
||||
Relatively thin spots: **SiteCallAudit.Tests** (31 tests for a reconciliation/KPI/relay component) and **DeploymentManager.Tests** (98 tests for the deployment pipeline, though much of its logic is exercised via ManagementService/Host/Integration suites).
|
||||
|
||||
### 2.2 Quality sampling — real behavioral tests, not shallow
|
||||
|
||||
`tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/AlarmActorTests.cs` is representative: full Akka TestKit harness, real compiled script library, `TestProbe` as the parent InstanceActor, asserts actual `AlarmStateChanged` messages with state/name payloads — genuine behavior, not constructor smoke tests. The SiteRuntime.Tests directory also carries targeted regression suites (`InstanceActorChildAttributeRaceTests`, `DeploymentManagerRedeployTests`, `DeploymentManagerMediumFindingsTests` — named for review findings, indicating review feedback gets locked in as tests). IntegrationTests covers real cross-cutting concerns: `CentralFailoverTests`, `SiteFailoverTests`, `DualNodeRecoveryTests`, `RecoveryDrillTests`, `ReadinessTests`, `AuthFlowTests`, `SecurityHardeningTests`, `AuditTransactionTests`, `NotificationOutboxFlowTests`, gRPC tests, plus a `ScadaBridgeWebApplicationFactory`. A 36-test Playwright E2E suite exists for the Central UI.
|
||||
|
||||
### 2.3 PerformanceTests is real but minimal — a misnomer more than a stub [Medium]
|
||||
|
||||
`tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/` has 3 test files / 10 tests: `HealthAggregationTests` (10 sites × 100 updates, **correctness assertions only, no timing**), `StaggeredStartupTests` (one `Stopwatch` assertion < 1000ms), and `AuditLog/HotPathLatencyTests` (the best of the three: `Stopwatch.GetTimestamp` p95-under-threshold on the site audit hot path; its own doc comment acknowledges "no BenchmarkDotNet" as a deliberate choice). Nothing here exercises the load-bearing performance claims of the design — 25s failover, gRPC streaming throughput, S&F drain rate, per-subscriber stream backpressure. Either rename the project to reflect its actual scope or grow it toward the design's stated performance envelope (failover time and stream throughput being the two highest-value additions).
|
||||
|
||||
### 2.4 CLI.Tests still excluded from the slnx [High]
|
||||
|
||||
Verified against `ZB.MOM.WW.ScadaBridge.slnx`: 29 of 30 test projects are members; `tests/ZB.MOM.WW.ScadaBridge.CLI.Tests` is absent (the src `CLI.csproj` *is* in the slnx). This is a documented gotcha in CLAUDE.md and memory, but the durable fix — adding one `<Project>` line — has not been made, so `dotnet test ZB.MOM.WW.ScadaBridge.slnx` silently skips 279 tests and a green solution run can mask CLI tests that never compiled. There is no evident reason to keep it out; add it.
|
||||
|
||||
---
|
||||
|
||||
## 3. Underdeveloped Areas & Deferred-Work Inventory
|
||||
|
||||
The in-code TODO surface is remarkably small: exactly **two real TODOs** in all of src (both the same issue — Transport area export, `EntitySerializer.cs:251/518`), zero `NotImplementedException` in production code paths (the one that existed was deliberately deleted — see the comment in `ClusterInfrastructure/ServiceCollectionExtensions.cs:34-47`), and zero FIXME/HACK. 39 "deferred/follow-up" comment mentions are almost all self-documenting pointers to logged decisions.
|
||||
|
||||
Consolidated inventory (item, where noted, risk of leaving it):
|
||||
|
||||
| # | Item | Where noted | Risk if left |
|
||||
|---|---|---|---|
|
||||
| 1 | **Transport: Area membership doesn't travel in bundles** — exporter hardcodes `AreaName: null`; importer machinery is fully built (`BundleImporter.cs:3104-3168` resolves/creates areas) | `src/.../Transport/Serialization/EntitySerializer.cs:251,518` (TODO); contradicts `Component-Transport.md:21` + CLAUDE.md #24 | **High** — spec and CLAUDE.md claim it ships; cross-env imports silently drop area organization; half-built feature invites confusion |
|
||||
| 2 | CLI.Tests not in slnx | `ZB.MOM.WW.ScadaBridge.slnx`; CLAUDE.md gotcha | **High** — 279 tests invisible to solution-level CI/test runs |
|
||||
| 3 | AuditLog reconciliation dials site NodeA only; NodeB failover endpoint selection deferred | `src/.../AuditLog/Central/SiteEnumerator.cs:32,64`, `ISiteEnumerator.cs:16` | **Medium** — during a site NodeA outage, telemetry-loss reconciliation (the safety net) is also unavailable; audit gaps persist until NodeA returns |
|
||||
| 4 | Options validation missing on ~14 components (§1.5) | e.g. `Transport/ServiceCollectionExtensions.cs:25` | **Medium** — config typos surface at first use, not startup |
|
||||
| 5 | `SmsConfiguration.MaxRetryCount` stored but not honored at dispatch (SMTP retry policy governs SMS) | `Commons/Entities/Notifications/SmsConfiguration.cs:36` | **Medium** — admin-visible knob that does nothing; misleading UI |
|
||||
| 6 | Role-check case-sensitivity asymmetry: UI `RequireClaim` policies vs ManagementActor's case-insensitive check | `ManagementService/ManagementActor.cs:1910` | **Medium** — a role stored with non-canonical casing could pass one gate and fail the other; deliberately unaltered, separately deferred |
|
||||
| 7 | SecuredWrite audit rows leave `SourceNode` NULL | CLAUDE.md (Security section, logged follow-up) | **Low/Medium** — per-node audit forensics blind spot for two-person writes specifically |
|
||||
| 8 | Hash-chain tamper evidence (T1) | CLAUDE.md; `docs/plans/2026-05-20-audit-log-code-roadmap.md:12`; CLI `verify-chain` is a shipped no-op stub (`CLI/Commands/AuditVerifyChainHelpers.cs:7`) | **Low/Medium** — deferred to v1.x by locked decision; append-only DB roles are the current control; the no-op CLI verb slightly overstates capability |
|
||||
| 9 | Parquet audit archival/export (T2) — endpoint returns 501 | `ManagementService/AuditEndpoints.cs:204-208`; CLAUDE.md | **Low** — deferred v1.x; 501 + CLI messaging handle it honestly |
|
||||
| 10 | Aggregated **live** alarm stream for Alarm Summary (currently snapshot fan-out + poll) | `docs/plans/2026-06-18-m7-opcua-mxgateway-ux-design.md:252` | **Low/Medium** — operator page scales with instance count; fan-out Ask per refresh is the known cost |
|
||||
| 11 | Central-persisted, auditable OPC UA server-cert trust (v1 is site-local PKI only) | m7 design follow-ups; CLAUDE.md | **Low** — broadcast-to-both-nodes covers HA; no central governance/audit of trust decisions |
|
||||
| 12 | Native-alarm-source-override CSV bulk import | m7 design follow-ups; CLAUDE.md (M7 UX) | **Low** — attribute-override CSV shipped; parity gap only |
|
||||
| 13 | `WaitForAttribute` quality-gated ("Good"-only) mode | `SiteRuntime/Scripts/ScriptRuntimeContext.cs:405` | **Low** — planned enhancement per spec §4.2 |
|
||||
| 14 | `WaitForAttribute` not wired into CentralUI Test-Run sandbox | `docs/plans/2026-06-17-waitfor-deferred-items.md:222` | **Low** — Test-Run parity gap; production scripts unaffected |
|
||||
| 15 | `BrowseNext` continuation: server's final-page signal not surfaced (client may issue one extra empty pull) | `Commons/Messages/Management/BrowseCommands.cs:34` | **Low** — one wasted round-trip |
|
||||
| 16 | `StubOpcUaClient` throws on browse — test-infra gap for browse/search unit tests | m7 design "risks" (`2026-06-18-m7-opcua-mxgateway-ux-design.md:245`) | **Low** — limits offline coverage of browse UI |
|
||||
| 17 | Unified notifications+site-calls outbox page | `2026-06-15-stillpending-completion-design.md:118` (M9 deferrals) | **Low** — two pages remain; explicit decision |
|
||||
| 18 | Folder drag-drop | same, `[PERM]` permanently deferred | None — closed decision; menu reorder shipped |
|
||||
| 19 | Bundle signing (asymmetric, non-repudiation), direct cluster-to-cluster pull, differential bundles | `docs/plans/2026-05-24-transport-design.md:402-408` | **Low** — v1 manifest hash + passphrase AES-GCM held sufficient |
|
||||
| 20 | Deployment EXPIRED-but-unpurged row purge | `ConfigurationDatabase/Repositories/DeploymentManagerRepository.cs:310` ("deferred TODO" — workaround in place) | **Low** — code already compensates when reading |
|
||||
| 21 | `SiteAuditBacklogReporter` threshold consolidation into `SqliteAuditWriterOptions` | `AuditLog/Site/SiteAuditBacklogReporter.cs:28` | **Low** — config-shape cleanup |
|
||||
| 22 | KPI history downsampling (hourly rollups) | `Component-KpiHistory.md` (YAGNI) | **Low** — 90-day retention bounds table size |
|
||||
| 23 | Vestigial site-side notification list read/write surface (§1.3) | `SiteRuntime/Persistence/SiteStorageService.cs:721`, `Repositories/SiteNotificationRepository.cs` | **Low/Medium** — dead-but-live API around an intentionally empty table |
|
||||
|
||||
Overall: ~20 open items, of which only #1–#4 warrant action before the next release; the rest are consciously logged, mostly with honest in-product behavior (501s, no-op stubs that say so).
|
||||
|
||||
---
|
||||
|
||||
## 4. Docs-Code Drift
|
||||
|
||||
Drift is exceptionally low for a docs-as-spec repo. Spot checks:
|
||||
|
||||
- **Component-KpiHistory.md vs code** — verified in-sync: `KpiHistoryOptions` defaults (SampleInterval 60s, RetentionDays 90 [1,3650], PurgeInterval 1d, DefaultMaxSeriesPoints 200 [2,5000]) match `src/.../KpiHistory/KpiHistoryOptions.cs` exactly, including range constraints; source placement (`IKpiSampleSource` impls with owners, bucketer in Commons, query service in CentralUI) matches the actual file layout and test placement.
|
||||
- **Component-StoreAndForward.md vs code** — verified in-sync: the doc's nuanced notification-parking semantics (`DefaultMaxRetries` cap, `maxRetries: 0` escape hatch), `ICachedCallLifecycleObserver` telemetry hook, and tracking-table-vs-buffer split all correspond to the shipped files (`NotificationForwarder.cs`, `ParkedMessageHandlerActor.cs`, `StoreAndForwardService.cs`, `ReplicationService.cs`).
|
||||
- **Component-ClusterInfrastructure.md vs code** — in-sync, and exemplary: the code carries a comment (`ClusterInfrastructure/ServiceCollectionExtensions.cs:34-47`) explaining that a dead throwing extension was removed *because* the doc's "Code Placement" section settled ownership in Host — docs and code cross-reference each other.
|
||||
- **[High] Component-Transport.md:21 and CLAUDE.md #24 claim `Area` membership travels "carried by name"** — but the exporter emits `AreaName: null` unconditionally (`Transport/Serialization/EntitySerializer.cs:251-254`) with a TODO, and `BundleImporter.cs:511-518` confirms "Areas don't travel (see export TODO)". The import-side resolution (create-area-if-missing, `BundleImporter.cs:3104-3168`) is fully built, making the doc claim look shipped on casual inspection. Either finish the export (small: add Areas to `EntityAggregate` + the id→name lookup) or correct the doc/CLAUDE.md to say area membership is not yet carried.
|
||||
- **[Low] `docs/components/` reference docs lag by three components**: 24 per-component reference docs exist but ScriptAnalysis (#25), KpiHistory (#26), and DelmiaNotifier (#27) are missing, while `README.md:111` claims "One doc per component (plus the shared TreeView)".
|
||||
- **[Low] DelmiaNotifier breaks the requirements-doc naming convention**: no `docs/requirements/Component-DelmiaNotifier.md`; the README component table row 27 links to the project README instead. Defensible for an external tool, but CLAUDE.md's own Document Conventions say component documents live in `docs/requirements/` as `Component-<Name>.md` — either add a thin spec doc or note the exception in the conventions section.
|
||||
- **README component table** — all 27 rows present and links resolve to existing files; component descriptions match CLAUDE.md's component list. TreeView correctly footnoted as a sub-component. No stale cross-references found in the sampled docs.
|
||||
- Historical `docs/plans/` design docs contain superseded "future work" lists (e.g. transport-design §18 lists site-scoped transport as future, which M8 later shipped) — acceptable since they are dated decision records, and the living docs (Component-*.md, CLAUDE.md, README) reflect the shipped state.
|
||||
|
||||
---
|
||||
|
||||
## Prioritized Recommendations
|
||||
|
||||
1. **[High] Add `tests/ZB.MOM.WW.ScadaBridge.CLI.Tests` to `ZB.MOM.WW.ScadaBridge.slnx`.** One line; removes a standing CI blind spot of 279 tests and retires a documented footgun.
|
||||
2. **[High] Resolve the Transport Area contradiction** — either implement area export (importer already complete; add Areas to `EntityAggregate` and the id→name lookup in `EntitySerializer.cs:251`) or amend `Component-Transport.md:21` + CLAUDE.md #24 to state area membership does not travel.
|
||||
3. **[Medium] Options-validation sweep**: add `IValidateOptions<>`/`ValidateOnStart` to the ~14 unvalidated option classes (Transport, StoreAndForward, SiteRuntime, Communication, InboundAPI, etc.), copying the existing HealthMonitoring/ClusterInfrastructure validator pattern.
|
||||
4. **[Medium] Close the AuditLog reconciliation NodeB gap** (`SiteEnumerator.cs:64`): dial NodeB when NodeA is unreachable — the shape (`SiteEntry`) already supports it, and reconciliation is the pipeline's loss-recovery safety net.
|
||||
5. **[Medium] Honor or remove `SmsConfiguration.MaxRetryCount`** (`Commons/Entities/Notifications/SmsConfiguration.cs:36`) — an admin-editable field with no effect is worse than no field.
|
||||
6. **[Medium] Decide the PerformanceTests story**: rename to reflect its correctness-at-scale scope, or add the two highest-value real measurements (failover time, gRPC stream throughput) behind the existing `Category=Performance` trait.
|
||||
7. **[Low/Medium] Excise the vestigial site-side notification-list surface** (`SiteNotificationRepository` reads + `StoreNotificationListAsync`) now that the purge guarantees the table stays empty.
|
||||
8. **[Low] Unify role-string casing handling** between UI `RequireClaim` policies and the ManagementActor check (`ManagementActor.cs:1910`) — normalize at write (already validated canonically) and compare consistently.
|
||||
9. **[Low] Backfill the three missing `docs/components/` reference docs** (ScriptAnalysis, KpiHistory, DelmiaNotifier) or scope the README claim.
|
||||
10. **[Low] Add contract-lock tests for the top ClusterClient message records** (mirror the existing proto contract tests) to enforce, not just intend, additive-only evolution under version skew.
|
||||
@@ -0,0 +1,82 @@
|
||||
# Architecture Review Fix Plans — Master Tracker
|
||||
|
||||
**Created:** 2026-07-08
|
||||
**Source:** [`archreview/00-OVERALL.md`](../00-OVERALL.md) and domain reports 01–08
|
||||
**Plans:** 8 implementation plans, **192 tasks** (191 net of one cross-plan duplicate — see Dedupe Rulings)
|
||||
|
||||
Every plan follows the TDD bite-sized-task format and ships a co-located `.tasks.json` manifest. Execute a plan with the `superpowers-extended-cc:executing-plans` skill (pass the plan path); it reads/updates the `.tasks.json`. **When a plan's status changes, update the table below** — this file is the single source of truth for initiative progress.
|
||||
|
||||
## Status Board
|
||||
|
||||
| Plan | Domain | Tasks | Done | Status | Findings coverage |
|
||||
|------|--------|------:|-----:|--------|-------------------|
|
||||
| [PLAN-01](PLAN-01-cluster-host-failover.md) | Cluster, Host & Failover | 23 | 0 | ⬜ Not started | 28 findings: 20 fixed, 3 → other plans, 3 accepted, 2 roadmap-deferred |
|
||||
| [PLAN-02](PLAN-02-communication-store-and-forward.md) | Communication & S&F | 24 | 0 | ⬜ Not started | 28 findings: 25 fixed, 1 → plan 01, 1 deferred (serializer swap), 1 won't-fix |
|
||||
| [PLAN-03](PLAN-03-site-runtime-dcl.md) | Site Runtime & DCL | 26 | 0 | ⬜ Not started | 28 actionable: 25 fixed, 3 deferred with rationale |
|
||||
| [PLAN-04](PLAN-04-data-audit-backbone.md) | Data & Audit Backbone | 26 | 0 | ⬜ Not started | 27 items: 22 fixed, 4 deferred, 1 won't-fix (incl. NodeB reconcile from report 08) |
|
||||
| [PLAN-05](PLAN-05-templates-deployment-transport.md) | Templates, Deployment & Transport | 25 | 0 | ⬜ Not started | 33 line-items: 27 fixed, 6 deferred with rationale |
|
||||
| [PLAN-06](PLAN-06-edge-integrations.md) | Edge Integrations | 24 | 0 | ⬜ Not started | 24 findings: 21 fixed, 3 deferred/won't-fix |
|
||||
| [PLAN-07](PLAN-07-ui-management-security.md) | UI, Management & Security | 33 | 0 | ⬜ Not started | 24 items: 20 fixed, 3 partial (deferrals documented), 1 won't-fix |
|
||||
| [PLAN-08](PLAN-08-conventions-tests.md) | Conventions & Tests | 11 | 0 | ⬜ Not started | 13 covered, 4 owned by other plans, ~20 triaged into deferred-work register |
|
||||
|
||||
Status values: ⬜ Not started · 🟨 In progress · ✅ Complete · ⏸ Blocked
|
||||
|
||||
## P0 — Do These First (regardless of plan order)
|
||||
|
||||
1. **PLAN-01 Task 1** — enable the SBR downing provider (the system-wide Critical; one line + parsed-config test).
|
||||
2. **PLAN-01 Tasks 3–4** — the two-node kill-test rig (`TwoNodeClusterFixture`); integration-proves Task 1 and later PLAN-02 work.
|
||||
3. **PLAN-07 Tasks 1–3** — `DisableLoginGuard` + scrub `deploy/wonder-app-vd03` (Critical; unauthenticated prod surface).
|
||||
4. **PLAN-02 Tasks 1–2** — ClusterClient generation-suffixed names (Critical crash-loop dropping routing to all sites).
|
||||
5. **PLAN-05 Task 1** — bundle import consumes `BaseTemplateName` (Critical silent data loss).
|
||||
6. **PLAN-08 Task 1** — CLI.Tests into the slnx (protects every subsequent plan's test runs).
|
||||
|
||||
## Cross-Plan Dependency Map
|
||||
|
||||
**Hard-ish sequencing (execute earlier):**
|
||||
- **PLAN-01 Tasks 5–7 → PLAN-02 Tasks 3–4.** PLAN-02's standby S&F delivery gate is a `Func<bool>` seam with marked swap points; it works standalone but its final wiring consumes PLAN-01's `ClusterActivityEvaluator` / `SelfIsPrimary` (singleton-host semantics).
|
||||
- **PLAN-01 Tasks 3–4 → PLAN-02 integration proof.** The two-node rig is how PLAN-02's Critical fixes get integration-verified.
|
||||
- **PLAN-04 Task 13 (additive `RequestedBy` message change) before any PLAN-07 sender refactors** touching the same messages.
|
||||
|
||||
**Soft seams (either order works, coordinate):**
|
||||
- **PLAN-05 Task 14 (`IScriptArtifactChangeBus`) ↔ PLAN-06 Task 3 (`InboundScriptExecutor.InvalidateMethod`).** PLAN-06 Task 1's per-request revision check is the designed correctness fallback, so there is no hard sequencing; when both land, wire the bus to the invalidate seam.
|
||||
- **PLAN-03's `DeployCompileValidator` should adopt PLAN-05's compile-cache abstraction** once it exists (later refactor, not a blocker).
|
||||
- **PLAN-03 Task 8's stuck-thread watchdog pattern is reusable by PLAN-06 Task 4** (inbound orphan counter).
|
||||
|
||||
**EF Core migrations — serialize globally.** Migration-adding tasks exist in PLAN-04 (Tasks 6, 8), PLAN-06 (Tasks 8, 19), and possibly PLAN-07 (secured-write expiry). All share one model snapshot: **never run two migration tasks concurrently**, and always build first (never `--no-build` on `dotnet ef migrations add` — known empty-scaffold gotcha).
|
||||
|
||||
**Shared-file merge surfaces (don't run these plans' overlapping tasks concurrently):**
|
||||
| File / area | Plans touching it |
|
||||
|---|---|
|
||||
| `ManagementActor.cs` | PLAN-07 (15 tasks, serialized within plan), PLAN-05 (management-path publish), PLAN-06 (method-update paths) |
|
||||
| `deploy/wonder-app-vd03/appsettings.Central.json` | PLAN-07 (auth), PLAN-01 (Cluster/Node sections) |
|
||||
| `NotificationOutboxActor.cs` | PLAN-04, PLAN-06 (SMS retry policy) |
|
||||
| `StoreAndForwardService.cs` / site tracking store | PLAN-02 (primary owner), PLAN-04 Task 15 (keyset query) |
|
||||
| `Host/Program.cs` + service registration | PLAN-01, PLAN-08 (options validation) |
|
||||
| `SiteHealthReport` (Commons record, additive fields) | PLAN-03 Task 6, PLAN-04 health-event fields |
|
||||
| Transport `BundleImporter.cs` | PLAN-05 (primary, serialized chain), PLAN-07 (import idempotency note) |
|
||||
|
||||
## Dedupe Rulings (one owner per finding)
|
||||
|
||||
| Finding | Owner | Others must skip |
|
||||
|---|---|---|
|
||||
| CLI.Tests into slnx | **PLAN-08 Task 1** (also retires CLAUDE.md gotcha + memory note) | PLAN-07 Task 33 — **skip as duplicate**; mark `skipped-duplicate` in its tasks.json |
|
||||
| `DisableLogin: true` artifact | **PLAN-07 Tasks 1–3** | PLAN-01 references only |
|
||||
| AuditLog reconciliation NodeB failover | **PLAN-04 Task 18** | PLAN-08 pointer only |
|
||||
| `SmsConfiguration.MaxRetryCount` unused | **PLAN-06 Task 18** | PLAN-08 pointer only |
|
||||
| Transport `AreaName: null` | **PLAN-05 Task 7** | PLAN-08 pointer only |
|
||||
| Role-casing normalization | **PLAN-07 Task 10 chain** | PLAN-08 pointer only |
|
||||
| Two-node failover rig | **PLAN-01 Tasks 3–4** | PLAN-08 Task 8 is a documented placeholder; PLAN-02 consumes |
|
||||
| Script-artifact invalidation contract | **PLAN-05 Task 14** (design + publisher) | PLAN-06 implements the inbound consumer |
|
||||
|
||||
## Recommended Execution Waves
|
||||
|
||||
- **Wave 0 (P0 list above).** Independent files; all six items can run in parallel. ~1 focused session.
|
||||
- **Wave 1 — stability spine.** Rest of PLAN-01 (active-node semantics, drains); PLAN-02 Tasks 3–9 (standby gate, gRPC channel keying, sweep bounding); PLAN-03 Tasks 1–4 (reconnect leak, compile gate); PLAN-04 Tasks 1–5 (purge chains); PLAN-05 import chain Tasks 2–6; PLAN-06 Tasks 1–4 + 8; PLAN-07 Tasks 4–9 (deploy scoping, secret elision).
|
||||
- **Wave 2 — remainder.** All other tasks per intra-plan `blockedBy` edges; migration tasks serialized globally; docs-update tasks land with their code tasks (repo rule: doc + code + tests together).
|
||||
- **Wave 3 — closeout.** PLAN-08 Task 11 (deferred-work register), each plan's doc-sweep task, then a full `dotnet build ZB.MOM.WW.ScadaBridge.slnx` + `dotnet test` (which now includes CLI.Tests) + `bash docker/deploy.sh` + the two-node drill from PLAN-01.
|
||||
|
||||
**Execution hygiene:** work in a dedicated worktree (not the primary checkout); one plan per executor session where possible; for parallel implementers use pathspec-scoped commits and verify every commit landed on HEAD afterward (both are known failure modes in this repo's history). Re-verify a finding's line numbers before editing — reports and plans cite the code as of 2026-07-08.
|
||||
|
||||
## Deferred / Won't-Fix Registry (initiative-level)
|
||||
|
||||
Consolidated in **PLAN-08 Task 11** → `docs/plans/2026-07-08-deferred-work-register.md` (triages the 23-item inventory from report 08 plus each plan's coverage-table deferrals: serializer swap, hash-chain/Parquet, aggregated live alarm stream, central cert-trust persistence, runtime script sandbox, clustered-GUID benchmark, TLS profile, Dockerfile HEALTHCHECK, etc.). Nothing from the review is silently dropped: every report finding is either a task, an ownership pointer, or a registered deferral with rationale.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"planPath": "archreview/plans/PLAN-01-cluster-host-failover.md",
|
||||
"tasks": [
|
||||
{ "id": 1, "subject": "Task 1: Enable the SBR downing provider in BuildHocon", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 2, "subject": "Task 2: Raise the cluster-leave CoordinatedShutdown phase timeout above the drain budget", "status": "pending", "blockedBy": [1] },
|
||||
{ "id": 3, "subject": "Task 3: Two-node in-process cluster fixture (the reusable failover rig)", "status": "pending", "blockedBy": [1] },
|
||||
{ "id": 4, "subject": "Task 4: SBR behavioral proof — kill the oldest node, assert downing + singleton migration", "status": "pending", "blockedBy": [3] },
|
||||
{ "id": 5, "subject": "Task 5: ClusterActivityEvaluator — single oldest-member \"active node\" definition", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 6, "subject": "Task 6: Rewire ActiveNodeGate, AkkaClusterNodeProvider and Primary labels onto the evaluator", "status": "pending", "blockedBy": [5] },
|
||||
{ "id": 7, "subject": "Task 7: Oldest-based /health/active check + require DB reachability for Active", "status": "pending", "blockedBy": [5] },
|
||||
{ "id": 8, "subject": "Task 8: Divergence proof — restarted first node must NOT become active again", "status": "pending", "blockedBy": [3, 6] },
|
||||
{ "id": 9, "subject": "Task 9: CentralSingletonRegistrar helper (manager + drain task + proxy)", "status": "pending", "blockedBy": [2] },
|
||||
{ "id": 10, "subject": "Task 10: Route all seven central singletons through the registrar (adds missing outbox/ingest drains)", "status": "pending", "blockedBy": [9] },
|
||||
{ "id": 11, "subject": "Task 11: SiteHealthReportAck message + ack semantics in the communication actors", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 12, "subject": "Task 12: Acked async health-report transport so counter-restore actually fires", "status": "pending", "blockedBy": [11] },
|
||||
{ "id": 13, "subject": "Task 13: Metrics-staleness signal + status-transition timestamps in the aggregator", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 14, "subject": "Task 14: Surface metrics-stale + offline-since on the Health dashboard", "status": "pending", "blockedBy": [13] },
|
||||
{ "id": 15, "subject": "Task 15: Fix the metrics-port seed in appsettings.Site.json + close the validator gap", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 16, "subject": "Task 16: Single-node installs — explicit AllowSingleNodeCluster instead of phantom seeds", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 17, "subject": "Task 17: stop_grace_period for all app containers (both topologies)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 18, "subject": "Task 18: Evict deleted sites from the health aggregator", "status": "pending", "blockedBy": [13] },
|
||||
{ "id": 19, "subject": "Task 19: Rate-limit dead-letter warnings", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 20, "subject": "Task 20: Down-if-alone recovery — exit on unexpected ActorSystem termination + service restart actions", "status": "pending", "blockedBy": [10] },
|
||||
{ "id": 21, "subject": "Task 21: Docker failover drill script (manual/CI kill test against the real topology)", "status": "pending", "blockedBy": [1, 17] },
|
||||
{ "id": 22, "subject": "Task 22: Reconcile REQ-HOST-6 (Akka.Hosting) with the hand-rolled bootstrap", "status": "pending", "blockedBy": [1] },
|
||||
{ "id": 23, "subject": "Task 23: Docs & deploy-config cleanup batch (Traefik doc, NodeName overlays, TLS roadmap note)", "status": "pending", "blockedBy": [7, 16] }
|
||||
],
|
||||
"lastUpdated": "2026-07-08"
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"planPath": "archreview/plans/PLAN-02-communication-store-and-forward.md",
|
||||
"tasks": [
|
||||
{ "id": 1, "subject": "Task 1: Generation-suffixed, sanitized ClusterClient actor names", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 2, "subject": "Task 2: Guard client creation in the refresh loop + real-factory address-edit lifecycle test", "status": "pending", "blockedBy": [1] },
|
||||
{ "id": 3, "subject": "Task 3: Active-node delivery gate on the S&F retry sweep", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 4, "subject": "Task 4: Wire the delivery gate in the Host + align the S&F design doc", "status": "pending", "blockedBy": [3] },
|
||||
{ "id": 5, "subject": "Task 5: Key the gRPC client cache by (site, endpoint)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 6, "subject": "Task 6: Endpoint-safe unsubscribe in DebugStreamBridgeActor + concurrent-session regression test", "status": "pending", "blockedBy": [5] },
|
||||
{ "id": 7, "subject": "Task 7: Bound the retry sweep with a batch LIMIT", "status": "pending", "blockedBy": [3] },
|
||||
{ "id": 8, "subject": "Task 8: Per-target short-circuit in the retry sweep", "status": "pending", "blockedBy": [7] },
|
||||
{ "id": 9, "subject": "Task 9: Parallel per-target lanes with a small concurrency cap", "status": "pending", "blockedBy": [8] },
|
||||
{ "id": 10, "subject": "Task 10: Ordered, observable replication dispatch", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 11, "subject": "Task 11: Upsert-based standby applies for Add/Park/Requeue", "status": "pending", "blockedBy": [7, 10] },
|
||||
{ "id": 12, "subject": "Task 12: deferToSweep enqueue mode + immediate sweep kick", "status": "pending", "blockedBy": [9] },
|
||||
{ "id": 13, "subject": "Task 13: Unstrand and unblock Notify.Send (maxRetries 0 + deferToSweep) + docs", "status": "pending", "blockedBy": [12] },
|
||||
{ "id": 14, "subject": "Task 14: Park (don't silently drop) corrupt buffered notifications", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 15, "subject": "Task 15: Wire-format pin tests for cross-cluster contracts", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 16, "subject": "Task 16: Replicate heartbeat marks to the peer central node", "status": "pending", "blockedBy": [2] },
|
||||
{ "id": 17, "subject": "Task 17: S&F SQLite hygiene — WAL + busy_timeout", "status": "pending", "blockedBy": [11] },
|
||||
{ "id": 18, "subject": "Task 18: Epoch-ms due-check (replace per-row julianday parsing)", "status": "pending", "blockedBy": [17] },
|
||||
{ "id": 19, "subject": "Task 19: Decouple audit-observer notification from the sweep (ordered channel)", "status": "pending", "blockedBy": [12] },
|
||||
{ "id": 20, "subject": "Task 20: Buffer resync storage primitives (GetAllMessagesAsync / ReplaceAllAsync)", "status": "pending", "blockedBy": [18] },
|
||||
{ "id": 21, "subject": "Task 21: Peer-join S&F buffer resync (anti-entropy)", "status": "pending", "blockedBy": [10, 20] },
|
||||
{ "id": 22, "subject": "Task 22: Conventions & observability cleanup batch (Lows)", "status": "pending", "blockedBy": [16] },
|
||||
{ "id": 23, "subject": "Task 23: Oldest-parked-age health metric (parked-retention aging signal)", "status": "pending", "blockedBy": [20] },
|
||||
{ "id": 24, "subject": "Task 24: Pin the reconciliation cursor boundary contract (>= at the site read)", "status": "pending", "blockedBy": [] }
|
||||
],
|
||||
"lastUpdated": "2026-07-08"
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"planPath": "archreview/plans/PLAN-03-site-runtime-dcl.md",
|
||||
"tasks": [
|
||||
{ "id": 1, "subject": "Task 1: OpcUaDataConnection — dispose + detach the previous client on reconnect (S1)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 2, "subject": "Task 2: MxGatewayDataConnection — dispose previous client + cancel previous event loop on reconnect (S1)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 3, "subject": "Task 3: Pure deploy-time compile validator for a flattened config (S3 groundwork)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 4, "subject": "Task 4: Reject the deployment on site-side compile failure per spec (S3)", "status": "pending", "blockedBy": [3] },
|
||||
{ "id": 5, "subject": "Task 5: ScriptExecutionScheduler gauges — queue depth, busy threads, oldest-busy age (S2/UA5)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 6, "subject": "Task 6: Surface scheduler stats through ISiteHealthCollector + SiteHealthReport (S2/UA5)", "status": "pending", "blockedBy": [5] },
|
||||
{ "id": 7, "subject": "Task 7: Periodic scheduler-stats reporter hosted service (S2/UA5)", "status": "pending", "blockedBy": [5, 6] },
|
||||
{ "id": 8, "subject": "Task 8: Stuck-script watchdog — name the script whose thread never came back (S2)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 9, "subject": "Task 9: ScriptActor — trigger-expression evaluation off the dispatcher, coalesced (P1)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 10, "subject": "Task 10: AlarmActor — async trigger-expression evaluation on the script scheduler (P1)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 11, "subject": "Task 11: InstanceActor — retry failed/lost tag subscriptions per connection (S4/UA6)", "status": "pending", "blockedBy": [4, 8] },
|
||||
{ "id": 12, "subject": "Task 12: NativeAlarmActor — retry on lost subscribe response (S4)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 13, "subject": "Task 13: DeploymentManager — retry the startup config load (S5)", "status": "pending", "blockedBy": [4] },
|
||||
{ "id": 14, "subject": "Task 14: Watch all Instance Actors; clean the map on unexpected termination (S6)", "status": "pending", "blockedBy": [13] },
|
||||
{ "id": 15, "subject": "Task 15: Fail the in-flight deployment when its Instance Actor dies during init (S6)", "status": "pending", "blockedBy": [14] },
|
||||
{ "id": 16, "subject": "Task 16: DataConnectionActor — capture _adapter into locals before background tasks (S7)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 17, "subject": "Task 17: DCL — tag→instances reverse index for the value fan-out hot path (P3)", "status": "pending", "blockedBy": [16] },
|
||||
{ "id": 18, "subject": "Task 18: InstanceActor — per-attribute child-subscription routing (P2)", "status": "pending", "blockedBy": [9, 11] },
|
||||
{ "id": 19, "subject": "Task 19: SiteStorageService — WAL journal mode + busy timeout (S8)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 20, "subject": "Task 20: NativeAlarmActor — coalesce per-transition SQLite upserts (P4)", "status": "pending", "blockedBy": [12, 19] },
|
||||
{ "id": 21, "subject": "Task 21: Persist native-alarm display metadata for rehydration (UA4)", "status": "pending", "blockedBy": [20] },
|
||||
{ "id": 22, "subject": "Task 22: CertStoreActor — export trusted certs (thumbprint + DER) for reconciliation (UA1 groundwork)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 23, "subject": "Task 23: Cert-trust reconcile-on-join — singleton pushes trusted set to (re)joining site node (UA1)", "status": "pending", "blockedBy": [15, 22] },
|
||||
{ "id": 24, "subject": "Task 24: Wire ConfigFetchRetryCount into the standby replication fetch (UA2)", "status": "pending", "blockedBy": [11] },
|
||||
{ "id": 25, "subject": "Task 25: Low-severity code cleanups — S9, S10, C2, C6", "status": "pending", "blockedBy": [9, 17] },
|
||||
{ "id": 26, "subject": "Task 26: Design-doc updates — C3, C4, C5 + behavior notes for landed changes", "status": "pending", "blockedBy": [1, 2, 4, 7, 8, 11, 13, 15, 18, 21, 23, 24, 25] }
|
||||
],
|
||||
"lastUpdated": "2026-07-08"
|
||||
}
|
||||
@@ -0,0 +1,774 @@
|
||||
# Data & Audit Backbone Fixes Implementation Plan
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Defuse the scale-dependent operational time bombs in the central data/audit layer (missing site SQLite retention purge, partition-switch timeout self-lock, unindexed KPI scans, table-wide execution-tree scans, hardcoded-zero backlog KPI) and close the audit-integrity/consistency gaps (append-only role drift, frozen SiteCalls progress fields, missing operator-Retry audit rows, reconciliation dead-ends, NodeA-only reconciliation) identified in `archreview/04-data-audit-backbone.md`.
|
||||
|
||||
**Architecture:** All fixes stay inside the existing component boundaries: site-side purge lives in the AuditLog component's site half (SqliteAuditWriter + a new hosted retention job), central maintenance hardening lives in AuditLogRepository/AuditLogPurgeActor, index/DDL changes ride EF Core migrations in ConfigurationDatabase (with idempotent SQL scripts for production), and contract changes (keyset cursor, RequestedBy, SiteEntry fallback endpoint) are strictly additive per the repo's message-evolution rule. Design docs are updated in the same task as the code they describe (repo rule: doc + code + tests travel together).
|
||||
|
||||
**Tech Stack:** C#/.NET 8, Akka.NET (actors/TestKit), EF Core + SQL Server (central), Microsoft.Data.Sqlite (site), xUnit. Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per-project: `dotnet test tests/<project> --filter <name>`. MSSQL-backed integration tests need `cd infra && docker compose up -d`. **EF migrations gotcha (repo-documented):** always build first and run `dotnet ef migrations add <Name>` WITHOUT `--no-build`, or it scaffolds an empty migration off the stale DLL.
|
||||
|
||||
## Findings Coverage
|
||||
|
||||
| # | Report finding (severity) | Task(s) |
|
||||
|---|---------------------------|---------|
|
||||
| S1 | Site SQLite 7-day retention purge missing (High) | 1, 2, 3 |
|
||||
| S2 | `SwitchOutPartitionAsync` no CommandTimeout / log-only failure (High) | 4, 5 |
|
||||
| S3 | Append-only DB-role enforcement DDL-only; purger grants insufficient (High) | 6, 7 |
|
||||
| S4 | SiteCalls upsert freezes RetryCount/LastError at rank (Medium) | 11 |
|
||||
| S5 | No EF `EnableRetryOnFailure` (Medium) | 12 |
|
||||
| S6 | Operator Retry on parked notification emits no audit row (+SiteCalls relay identity) (Medium) | 13, 14 |
|
||||
| S7 | Reconciliation edge cases dead-ended (cursor pin; permanent abandonment) (Medium) | 15, 16, 17 |
|
||||
| S8 | NotificationOutboxRepository dual SQLite/T-SQL dialect — provider parity (Medium) | 24 (documented convention decision: MSSQL-only SQL + MSSQL fixtures is the norm for new repositories; existing dual-dialect kept as a documented legacy exception — rewriting the shipped repository + its whole test suite is not worth the churn) |
|
||||
| S9 | Purge actors' first tick waits a full interval (Low) | 22 |
|
||||
| S10 | `SqliteAuditWriter.Dispose` sync-over-async (Low) | Won't-fix — report itself notes the thread-pool-hop mitigation is correct and documented; flagged-only |
|
||||
| P1 | SiteCalls KPI predicates full-scan — no `TerminalAtUtc` index (High) | 8 |
|
||||
| P2 | `GetExecutionTreeAsync` scans entire AuditLog (High) | 9 |
|
||||
| P3 | AuditLog clustered key leads with random GUID (Medium) | Deferred — report says "worth a deliberate benchmark before the table gets big"; changing it is a full-table rebuild. Task 24 records it as a tracked design decision in Component-ConfigurationDatabase.md so it is no longer untracked |
|
||||
| P4 | KpiSample volume + unbatched purge (Medium) | 19 (batched purge); per-node sampling cadence recorded as tracked follow-up in 24 |
|
||||
| P5 | Catch-all `(@p IS NULL OR col=@p)` predicates in SiteCalls query (Medium) | 21 |
|
||||
| P6 | Outbox offset paging + double query + LIKE; KPI 5-7 round trips; unbounded oldest materialization (Medium) | 20 (KPI single query + bounded oldest); offset→keyset conversion Deferred — it changes the page/TotalCount UI contract for a table whose live-queue portion stays small under Task 19/existing purge; recorded in 24 |
|
||||
| P7 | Non-persisted `IngestedAtUtc` computed column re-parse (Low) | 23 (predicate-ban comment guard) |
|
||||
| P8 | `MarkForwardedAsync`/`MarkReconciledAsync` per-id IN lists near SQLite param limit (Low) | 23 |
|
||||
| C1 | Design docs drifted (enum lists, Teams, per-channel overrides, SiteCalls schema) (Medium) | 24 |
|
||||
| C2 | `AuditLogRow` entity lives in ConfigurationDatabase not Commons (Medium) | 24 (document the deviation — the rationale in the report is accepted: contract type is external `ZB.MOM.WW.Audit.AuditEvent`, row is a persistence-only projection) |
|
||||
| C3 | Mixed timestamp CLR types across sibling tables (Low) | 24 (convention note: new tables use `DateTime`+Utc converter) |
|
||||
| C4 | Uncharted KPI metric names are string literals (Low) | 23 |
|
||||
| U1 | `backlogTotal` KPI history permanently zero | 10 |
|
||||
| U2 | Site SQLite retention purge | 1, 2, 3 |
|
||||
| U3 | Hash-chain tamper evidence / Parquet archival | No action — explicitly deferred to v1.x per Component-AuditLog.md; report confirms "consistent with plan; no drift" |
|
||||
| U4 | SecuredWrite audit rows leave `SourceNode` NULL; doc under-promises per-channel keys | SourceNode fix Deferred — already a logged follow-up in Component-AuditLog.md:154-157 and the SecuredWrite emission path belongs to plan 07's domain. Doc under-promise fixed in 24 |
|
||||
| U5 | Reconciliation cursor keyset upgrade untracked | 15, 16 (SiteCalls keyset implemented); AuditLog-pull keyset recorded as tracked follow-up in Component-AuditLog.md (24) |
|
||||
| U6 | Dispatcher throughput ceiling (sequential delivery) | 25 |
|
||||
| U7 | Test-coverage gaps mirror findings | Covered inside tasks 1, 4, 10 (the report's three named gaps each get a test) |
|
||||
| X1 | AuditLog reconciliation dials site NodeA only (flagged in report 08, owned here) | 18 |
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Site SQLite retention purge — `PurgeExpiredAsync` on the writer
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 4, 6, 8, 11, 12, 19, 20, 21
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Services/ISiteAuditQueue.cs` (add method)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriter.cs` (implement; the `auto_vacuum = INCREMENTAL` pragma at ~line 149 already anticipates this)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SqliteAuditWriterRetentionTests.cs` (create)
|
||||
|
||||
1. Write the failing test (pattern-match the fixture setup from `SqliteAuditWriterBacklogStatsTests.cs`):
|
||||
|
||||
```csharp
|
||||
public class SqliteAuditWriterRetentionTests
|
||||
{
|
||||
[Fact]
|
||||
public async Task PurgeExpired_DeletesForwardedAndReconciled_OlderThanCutoff()
|
||||
{
|
||||
await using var writer = CreateWriter(); // same in-memory/temp-file helper the sibling tests use
|
||||
var oldForwarded = await WriteEventAsync(writer, occurredAtUtc: DateTime.UtcNow.AddDays(-10));
|
||||
var oldReconciled = await WriteEventAsync(writer, occurredAtUtc: DateTime.UtcNow.AddDays(-10));
|
||||
var oldPending = await WriteEventAsync(writer, occurredAtUtc: DateTime.UtcNow.AddDays(-10));
|
||||
var fresh = await WriteEventAsync(writer, occurredAtUtc: DateTime.UtcNow);
|
||||
await writer.MarkForwardedAsync(new[] { oldForwarded, oldReconciled, fresh });
|
||||
await writer.MarkReconciledAsync(new[] { oldReconciled });
|
||||
|
||||
var purged = await writer.PurgeExpiredAsync(DateTime.UtcNow.AddDays(-7));
|
||||
|
||||
Assert.Equal(2, purged); // oldForwarded + oldReconciled
|
||||
// Hard ForwardState invariant: the old *Pending* row survives on age alone.
|
||||
var pending = await writer.ReadPendingAsync(10);
|
||||
Assert.Contains(pending, e => e.EventId == oldPending);
|
||||
// Fresh forwarded row inside the window survives.
|
||||
var stats = await writer.GetBacklogStatsAsync();
|
||||
Assert.Equal(1, stats.PendingCount);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task PurgeExpired_IsIdempotent()
|
||||
{
|
||||
await using var writer = CreateWriter();
|
||||
var id = await WriteEventAsync(writer, occurredAtUtc: DateTime.UtcNow.AddDays(-10));
|
||||
await writer.MarkForwardedAsync(new[] { id });
|
||||
Assert.Equal(1, await writer.PurgeExpiredAsync(DateTime.UtcNow.AddDays(-7)));
|
||||
Assert.Equal(0, await writer.PurgeExpiredAsync(DateTime.UtcNow.AddDays(-7)));
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter SqliteAuditWriterRetentionTests` — expect FAIL (method does not exist).
|
||||
3. Add to `ISiteAuditQueue`: `Task<int> PurgeExpiredAsync(DateTime olderThanUtc, CancellationToken ct = default);` with XML doc stating the hard invariant (`Pending` rows are never purged on age alone — only `Forwarded`/`Reconciled`). Implement in `SqliteAuditWriter` under `_writeLock` (mirror `MarkForwardedAsync`'s lock + `ObjectDisposedException.ThrowIf` shape), one transaction:
|
||||
|
||||
```sql
|
||||
CREATE TEMP TABLE IF NOT EXISTS purge_ids AS
|
||||
SELECT EventId FROM audit_forward_state
|
||||
WHERE ForwardState IN ('Forwarded','Reconciled') AND OccurredAtUtc < $cutoff;
|
||||
DELETE FROM audit_forward_state WHERE EventId IN (SELECT EventId FROM purge_ids);
|
||||
DELETE FROM audit_event WHERE EventId IN (SELECT EventId FROM purge_ids);
|
||||
DROP TABLE purge_ids;
|
||||
```
|
||||
|
||||
Delete the sidecar first (it is the FK child of `audit_event`). `$cutoff` uses the same round-trip `"o"` timestamp format the writer already uses. Return `changes()` from the `audit_event` DELETE. After COMMIT, run `PRAGMA incremental_vacuum;` (the pragma the ctor set up — reference the existing comment at `SqliteAuditWriter.cs:145` and update it: the purge now exists).
|
||||
4. Run the filter again — expect PASS. Also run the whole site suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter "FullyQualifiedName~Site"` — expect PASS.
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Services/ISiteAuditQueue.cs src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriter.cs tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SqliteAuditWriterRetentionTests.cs && git commit -m "feat(audit-log): add site SQLite PurgeExpiredAsync honoring the ForwardState invariant"`
|
||||
|
||||
### Task 2: Site retention options
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~3 min
|
||||
**Parallelizable with:** 1, 4, 6, 8, 11, 12, 19, 20, 21
|
||||
**Files:**
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SiteAuditRetentionOptions.cs`
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SiteAuditRetentionOptionsTests.cs` (create)
|
||||
|
||||
1. Failing test:
|
||||
|
||||
```csharp
|
||||
public class SiteAuditRetentionOptionsTests
|
||||
{
|
||||
[Theory]
|
||||
[InlineData(0, 1)] // clamp floor
|
||||
[InlineData(7, 7)] // default passthrough
|
||||
[InlineData(365, 90)] // clamp ceiling (spec: min 1, max 90)
|
||||
public void ResolvedRetentionDays_Clamps(int configured, int expected)
|
||||
=> Assert.Equal(expected, new SiteAuditRetentionOptions { RetentionDays = configured }.ResolvedRetentionDays);
|
||||
|
||||
[Fact]
|
||||
public void Defaults_Are_SevenDays_DailyPurge_FiveMinuteInitialDelay()
|
||||
{
|
||||
var o = new SiteAuditRetentionOptions();
|
||||
Assert.Equal(7, o.ResolvedRetentionDays);
|
||||
Assert.Equal(TimeSpan.FromHours(24), o.ResolvedPurgeInterval);
|
||||
Assert.Equal(TimeSpan.FromMinutes(5), o.InitialDelay);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter SiteAuditRetentionOptionsTests` — FAIL.
|
||||
3. Implement the options class following the `SiteCallAuditOptions.ResolvedReconciliationInterval` clamp pattern: `RetentionDays` (default 7; `ResolvedRetentionDays` clamps to `[1, 90]`), `PurgeInterval` (default 24h; `ResolvedPurgeInterval` clamps to min 1 minute — Akka zero-interval spin footgun), `InitialDelay` (default 5 min — short so a daily-restarting node still purges). Config section: `ScadaBridge:AuditLog:SiteRetention`.
|
||||
4. Run filter — PASS.
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SiteAuditRetentionOptions.cs tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SiteAuditRetentionOptionsTests.cs && git commit -m "feat(audit-log): SiteAuditRetentionOptions (7-day default, clamped 1-90)"`
|
||||
|
||||
### Task 3: Site retention hosted job + registration + design doc
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 4, 6, 8, 11, 12, 19, 20, 21
|
||||
**Files:**
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SiteAuditRetentionService.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/ServiceCollectionExtensions.cs` (site registration path, near the `ISiteAuditQueue` registration at ~line 111)
|
||||
- Modify: `docs/requirements/Component-AuditLog.md` (~lines 412-414, the "Sites: daily site job" bullet)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SiteAuditRetentionServiceTests.cs` (create)
|
||||
|
||||
1. Failing test (fake `ISiteAuditQueue`, short intervals):
|
||||
|
||||
```csharp
|
||||
public class SiteAuditRetentionServiceTests
|
||||
{
|
||||
[Fact]
|
||||
public async Task Tick_Purges_With_RetentionCutoff()
|
||||
{
|
||||
var queue = new RecordingSiteAuditQueue(); // stub: records PurgeExpiredAsync calls
|
||||
var options = Options.Create(new SiteAuditRetentionOptions
|
||||
{ RetentionDays = 7, PurgeInterval = TimeSpan.FromMilliseconds(50), InitialDelay = TimeSpan.Zero });
|
||||
using var svc = new SiteAuditRetentionService(queue, options, NullLogger<SiteAuditRetentionService>.Instance);
|
||||
await svc.StartAsync(CancellationToken.None);
|
||||
await WaitUntilAsync(() => queue.PurgeCalls.Count >= 1, TimeSpan.FromSeconds(5));
|
||||
await svc.StopAsync(CancellationToken.None);
|
||||
var cutoff = queue.PurgeCalls[0];
|
||||
Assert.InRange(cutoff, DateTime.UtcNow.AddDays(-7).AddMinutes(-1), DateTime.UtcNow.AddDays(-7).AddMinutes(1));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Tick_SwallowsExceptions_AndKeepsTicking()
|
||||
{
|
||||
var queue = new RecordingSiteAuditQueue { ThrowOnFirstCall = true };
|
||||
// ... start, wait for >= 2 calls, assert no unobserved exception
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter SiteAuditRetentionServiceTests` — FAIL.
|
||||
3. Implement `SiteAuditRetentionService : IHostedService, IDisposable` mirroring `SiteAuditBacklogReporter.cs` (same file layout: `System.Threading.Timer`, `InitialDelay` as due time, `ResolvedPurgeInterval` as period, per-tick try/catch that logs and continues, purge count logged at Information). Register in the site path of `ServiceCollectionExtensions`: bind `SiteAuditRetentionOptions` from `ScadaBridge:AuditLog:SiteRetention` and `services.AddHostedService<SiteAuditRetentionService>();`.
|
||||
4. Run filter — PASS. Then `dotnet build ZB.MOM.WW.ScadaBridge.slnx` — expect clean.
|
||||
5. Update `Component-AuditLog.md` ~412-414: replace the aspirational bullet with the shipped mechanism — `SiteAuditRetentionService` hosted job, config keys (`ScadaBridge:AuditLog:SiteRetention:RetentionDays|PurgeInterval`), the `PurgeExpiredAsync` invariant, and the post-purge `incremental_vacuum`.
|
||||
6. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.AuditLog tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site docs/requirements/Component-AuditLog.md && git commit -m "feat(audit-log): daily site SQLite retention purge job (closes unbounded site DB growth)"`
|
||||
|
||||
### Task 4: Maintenance command timeout on partition switch + per-channel purge
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 2, 3, 8 (different files), 11, 20
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeOptions.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IAuditLogRepository.cs` (additive optional parameter)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/AuditLogRepository.cs` (`SwitchOutPartitionAsync` ~257-371, `PurgeChannelOlderThanAsync` ~374-445)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeActor.cs` (pass the timeout)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/AuditLogPurgeActorTests.cs` (extend), `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/PartitionPurgeTests.cs` (extend, MSSQL)
|
||||
|
||||
1. Failing unit test in `AuditLogPurgeActorTests` (these tests already drive the actor against a mocked `IAuditLogRepository`): assert the actor calls `SwitchOutPartitionAsync(boundary, commandTimeout)` with the options value:
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task PurgeTick_Passes_MaintenanceCommandTimeout_To_Repository()
|
||||
{
|
||||
// options: MaintenanceCommandTimeoutMinutes = 45
|
||||
// mock repo: capture the TimeSpan? argument
|
||||
// drive one tick; Assert.Equal(TimeSpan.FromMinutes(45), captured);
|
||||
}
|
||||
```
|
||||
|
||||
Plus an options test: `MaintenanceCommandTimeoutMinutes` default 30, clamped to min 1 (`ResolvedMaintenanceCommandTimeout`).
|
||||
2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter AuditLogPurgeActorTests` — FAIL (no such parameter).
|
||||
3. Implementation:
|
||||
- `AuditLogPurgeOptions`: add `public int MaintenanceCommandTimeoutMinutes { get; set; } = 30;` + `ResolvedMaintenanceCommandTimeout` clamp (min 1 minute), XML-doc'd against the 30s-default-timeout failure mode from the review.
|
||||
- `IAuditLogRepository`: `Task<long> SwitchOutPartitionAsync(DateTime monthBoundary, TimeSpan? commandTimeout = null, CancellationToken ct = default);` and `PurgeChannelOlderThanAsync(..., TimeSpan? commandTimeout = null, ...)` — additive defaults, existing callers compile unchanged.
|
||||
- `AuditLogRepository.SwitchOutPartitionAsync`: when `commandTimeout` is non-null, set `sampleCmd.CommandTimeout = (int)commandTimeout.Value.TotalSeconds;` and wrap the `ExecuteSqlRawAsync` with `_context.Database.SetCommandTimeout(commandTimeout)` (the context is scoped per purge tick — see `AuditLogPurgeActor.OnTickAsync`'s `CreateAsyncScope` — so no restore is needed, but restore the previous value in a `finally` anyway for hygiene). Same `cmd.CommandTimeout` treatment on the per-channel `DELETE TOP` command.
|
||||
- `AuditLogPurgeActor.OnTickAsync`: pass `_purgeOptions.ResolvedMaintenanceCommandTimeout` to both repository calls.
|
||||
4. Run the actor filter — PASS. Extend `PartitionPurgeTests` (MSSQL; `cd infra && docker compose up -d` first) with one case that calls `SwitchOutPartitionAsync(boundary, TimeSpan.FromMinutes(30))` and asserts success — proving the timeout path executes against real SQL Server: `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter PartitionPurgeTests`.
|
||||
5. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.AuditLog src/ZB.MOM.WW.ScadaBridge.Commons src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests && git commit -m "fix(audit-log): explicit long CommandTimeout on partition-switch and per-channel purge maintenance paths"`
|
||||
|
||||
### Task 5: Purge-failure health event (not just a log line)
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 8, 11, 19, 20, 21
|
||||
**Files:**
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeFailedEvent.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeActor.cs` (catch block ~198-206 and the boundary-enumeration catch)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditCentralHealthSnapshot.cs` + `IAuditCentralHealthSnapshot.cs` (add `PurgeFailures` counter, mirroring the existing counter members)
|
||||
- Modify: `docs/requirements/Component-AuditLog.md` (retention section: purge failure is now a health metric)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/AuditLogPurgeActorTests.cs` (extend)
|
||||
|
||||
1. Failing test: mock repository whose `SwitchOutPartitionAsync` throws; subscribe a TestProbe to `AuditLogPurgeFailedEvent` on the EventStream (same pattern the existing tests use for `AuditLogPurgedEvent`); drive a tick; expect one `AuditLogPurgeFailedEvent(boundary, errorMessage, elapsedMs)` and assert the health snapshot's `PurgeFailures` incremented.
|
||||
2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter AuditLogPurgeActorTests` — FAIL.
|
||||
3. Implement: `public sealed record AuditLogPurgeFailedEvent(DateTime MonthBoundary, string Error, long ElapsedMilliseconds);` — publish it in the per-boundary catch (alongside the existing `LogError`) using the already-captured `eventStream`; increment the new snapshot counter (inject `IAuditCentralHealthSnapshot` the same way sibling actors take their counters — follow the existing ctor DI shape). The success event (`AuditLogPurgedEvent`) already exists; this is its symmetric failure twin.
|
||||
4. Run filter — PASS.
|
||||
5. Doc: add to Component-AuditLog.md retention section: "Purge failure publishes `AuditLogPurgeFailedEvent` and increments the `PurgeFailures` central health counter — a silently failing retention job is a health-surface condition, not just a log line."
|
||||
6. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.AuditLog tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central docs/requirements/Component-AuditLog.md && git commit -m "feat(audit-log): surface partition-purge failure as health event + counter"`
|
||||
|
||||
### Task 6: Fix purger role grants (migration)
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 1, 2, 3, 11, 20 (NOT with Task 8 — both add migrations, serialize on the model snapshot)
|
||||
**Files:**
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/<timestamp>_FixAuditPurgerRoleGrants.cs` (scaffolded)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Migrations/` (add `FixAuditPurgerRoleGrantsTests.cs`), plus re-run `AuditLogAppendOnlyGuardTests`
|
||||
|
||||
1. Failing test first — a migration-content guard in the style of the existing migration tests: assert the migration source contains `GRANT CREATE TABLE TO scadabridge_audit_purger` and `GRANT DELETE ON dbo.AuditLog TO scadabridge_audit_purger`. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter FixAuditPurgerRoleGrants` — FAIL (file absent).
|
||||
2. Scaffold (repo gotcha — build first, NO `--no-build`):
|
||||
|
||||
```bash
|
||||
dotnet build ZB.MOM.WW.ScadaBridge.slnx
|
||||
cd src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase
|
||||
dotnet ef migrations add FixAuditPurgerRoleGrants
|
||||
```
|
||||
|
||||
Verify the scaffold is model-empty (no schema change) and add the SQL by hand in `Up`:
|
||||
|
||||
```csharp
|
||||
migrationBuilder.Sql(@"
|
||||
IF DATABASE_PRINCIPAL_ID('scadabridge_audit_purger') IS NULL
|
||||
EXEC sp_executesql N'CREATE ROLE scadabridge_audit_purger';
|
||||
-- The switch-out dance CREATEs a staging table; ALTER ON SCHEMA::dbo alone does not
|
||||
-- confer the database-level CREATE TABLE permission (review finding S3).
|
||||
GRANT CREATE TABLE TO scadabridge_audit_purger;
|
||||
-- The per-channel retention override (PurgeChannelOlderThanAsync) is a bounded row
|
||||
-- DELETE on the maintenance path; a segregated purger principal needs this grant.
|
||||
GRANT DELETE ON dbo.AuditLog TO scadabridge_audit_purger;"); // AUDIT-PURGE-ALLOWED: role grant for the maintenance principal
|
||||
```
|
||||
|
||||
`Down`: `REVOKE` both grants (guarded on principal existence).
|
||||
3. Run the new test — PASS. Then run `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter AuditLogAppendOnlyGuardTests` — if the CI grep guard trips on the `GRANT DELETE ON dbo.AuditLog` text, extend the guard's allowlist to permit `GRANT`/`DENY`/`REVOKE` statements (they are permission DDL, not data mutation) — keep the guard strict for `DELETE FROM`/`UPDATE `.
|
||||
4. Production script step (repo convention — manual SQL for production): `dotnet ef migrations script AddPendingDeployment FixAuditPurgerRoleGrants --idempotent --output ../../docs/plans/sql/FixAuditPurgerRoleGrants.sql` (adjust the from-migration to the actual latest; review the output).
|
||||
5. MSSQL sanity: with infra up, `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter Migrations` — expect PASS.
|
||||
6. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests docs/plans/sql && git commit -m "fix(config-db): grant CREATE TABLE + scoped DELETE to scadabridge_audit_purger so a segregated maintenance principal can actually run the purge"`
|
||||
|
||||
### Task 7: Honest append-only enforcement story (docs)
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** everything except 24 (same doc files)
|
||||
**Files:**
|
||||
- Modify: `docs/requirements/Component-AuditLog.md` (~417-424, "Security & Tamper-Evidence")
|
||||
- Modify: `docs/requirements/Component-ConfigurationDatabase.md` (~399-402, AuditLog Table Purge section)
|
||||
|
||||
1. Rewrite Component-AuditLog.md §Security "Append-only enforcement" to state reality (review finding S3): the application runs with **one** connection principal for both writer and maintenance paths, so in the default deployment append-only is enforced by (a) the CI grep guard (`AuditLogAppendOnlyGuardTests`) with its single marked DELETE exemption, and (b) code review. The `scadabridge_audit_writer` / `scadabridge_audit_purger` roles are **optional DBA hardening**: a deployment that wants database-level enforcement must provision two logins (runtime → writer role; a maintenance job/second connection → purger role) — and note the purger role now carries `CREATE TABLE` + scoped `DELETE` (Task 6) so it can genuinely execute the switch-out and per-channel purge. Remove the false claim "row-level DELETE is not granted even to purge" (stale since `PerChannelRetentionDays` shipped).
|
||||
2. In Component-ConfigurationDatabase.md fix the same passage's stale "single global value in v1, no per-channel overrides" claim while in the file (coordinates with Task 24 — do this bullet here since it is in the same paragraph; Task 24 skips it if already fixed).
|
||||
3. Verify no other doc contradicts: `grep -rn "audit_writer\|audit_purger" docs/`.
|
||||
4. Commit: `git add docs/requirements/Component-AuditLog.md docs/requirements/Component-ConfigurationDatabase.md && git commit -m "docs(audit-log): document append-only enforcement honestly (CI guard default; DB roles are optional DBA hardening)"`
|
||||
|
||||
### Task 8: SiteCalls filtered non-terminal index (migration)
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 1, 2, 3, 5, 11 (NOT with Task 6 — migration serialization)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/SiteCallEntityTypeConfiguration.cs` (~66-88; also fix the now-stale "No index — … per-site, not per-node" SourceNode comment)
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/<timestamp>_AddSiteCallsNonTerminalIndex.cs` (scaffolded)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/SchemaConfigurationTests.cs` (extend)
|
||||
|
||||
1. Failing model test (SchemaConfigurationTests already asserts index shapes):
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public void SiteCalls_Has_Filtered_NonTerminal_Index()
|
||||
{
|
||||
var entity = Model.FindEntityType(typeof(SiteCall))!;
|
||||
var index = entity.GetIndexes().Single(i => i.GetDatabaseName() == "IX_SiteCalls_NonTerminal");
|
||||
Assert.Equal(new[] { nameof(SiteCall.CreatedAtUtc) }, index.Properties.Select(p => p.Name));
|
||||
Assert.Equal("[TerminalAtUtc] IS NULL", index.GetFilter());
|
||||
Assert.Equal(new[] { "SourceSite", "SourceNode", "Status" }, index.GetIncludeProperties());
|
||||
}
|
||||
```
|
||||
|
||||
2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter SiteCalls_Has_Filtered_NonTerminal_Index` — FAIL.
|
||||
3. Add to `SiteCallEntityTypeConfiguration.Configure`:
|
||||
|
||||
```csharp
|
||||
// NonTerminal backs every "live queue" KPI predicate (TerminalAtUtc IS NULL):
|
||||
// buffered/stuck/oldest counts run every 60 s from the KPI recorder plus every
|
||||
// 10 s Health-dashboard poll. Filtered: the non-terminal population is the live
|
||||
// queue, not the 365-day archive — each count becomes a small seek instead of a
|
||||
// clustered scan (arch-review 04, P1).
|
||||
builder.HasIndex(s => s.CreatedAtUtc)
|
||||
.HasDatabaseName("IX_SiteCalls_NonTerminal")
|
||||
.HasFilter("[TerminalAtUtc] IS NULL")
|
||||
.IncludeProperties(nameof(SiteCall.SourceSite), nameof(SiteCall.SourceNode), nameof(SiteCall.Status));
|
||||
```
|
||||
|
||||
4. Scaffold the migration (build first, NO `--no-build`): `dotnet build ZB.MOM.WW.ScadaBridge.slnx && cd src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase && dotnet ef migrations add AddSiteCallsNonTerminalIndex` — verify the scaffold contains the `CreateIndex` with filter (if it is empty, the build/`--no-build` gotcha bit — delete and redo).
|
||||
5. Run the model test — PASS. With infra up, run the MSSQL-backed repo suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter SiteCallAuditRepositoryTests` — PASS.
|
||||
6. Production script: `dotnet ef migrations script FixAuditPurgerRoleGrants AddSiteCallsNonTerminalIndex --idempotent --output ../../docs/plans/sql/AddSiteCallsNonTerminalIndex.sql`.
|
||||
7. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests docs/plans/sql && git commit -m "perf(site-call-audit): filtered IX_SiteCalls_NonTerminal index — KPI predicates seek the live queue instead of scanning 365 days"`
|
||||
|
||||
### Task 9: Bound the execution-tree edge scan with a root time window
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 2, 3, 11, 13, 18, 19
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/AuditLogRepository.cs` (`GetExecutionTreeAsync`, ~632-790)
|
||||
- Modify: `docs/requirements/Component-AuditLog.md` (audit-tree section: document the 7-day traversal bound)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/ExecutionIdCorrelationTests.cs` or the existing execution-tree test file (extend; MSSQL)
|
||||
|
||||
1. Failing test (add to the existing MSSQL execution-tree suite — locate with `grep -rln "GetExecutionTreeAsync" tests/`):
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task ExecutionTree_Bounds_EdgeScan_To_Root_Window()
|
||||
{
|
||||
// Root execution at T0, child at T0+5min → both returned.
|
||||
// Unrelated execution pair at T0+30 days sharing no ancestry → NOT returned (was never returned;
|
||||
// this asserts correctness is preserved), AND a genuine descendant stamped at T0+30 days
|
||||
// (beyond the 7-day traversal bound) is documented-excluded: assert it is absent.
|
||||
}
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter ExecutionTree_Bounds_EdgeScan` — FAIL (descendant beyond bound is currently returned).
|
||||
3. Implementation in `GetExecutionTreeAsync`, between Phase 1 and Phase 2:
|
||||
- New constants: `private static readonly TimeSpan ExecutionTreeWindowSlack = TimeSpan.FromHours(1);` and `private static readonly TimeSpan ExecutionTreeMaxSpan = TimeSpan.FromDays(7);` with a comment: execution trees span minutes, not years; the window enables partition elimination on the Edges anchor (arch-review 04, P2).
|
||||
- Query `SELECT MIN(OccurredAtUtc) FROM dbo.AuditLog WHERE ExecutionId = @root;` (a seek on `IX_AuditLog_Execution`).
|
||||
- If non-NULL, parameterize the Edges CTE anchor: `WHERE ExecutionId IS NOT NULL AND OccurredAtUtc >= @winStart AND OccurredAtUtc < @winEnd` with `@winStart = rootFirst - slack`, `@winEnd = rootFirst + maxSpan`. If NULL (row-less stub root), keep the unbounded form (current behavior) — correctness over speed for the degenerate case.
|
||||
4. Run the execution-tree filter plus the full existing suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter "FullyQualifiedName~ExecutionTree|FullyQualifiedName~ExecutionId"` — PASS.
|
||||
5. Doc: Component-AuditLog.md audit-tree section — "tree traversal is bounded to a 7-day window anchored at the root's first event (configurable constant); descendants beyond the window are excluded by design."
|
||||
6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/AuditLogRepository.cs docs/requirements/Component-AuditLog.md tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests && git commit -m "perf(audit-log): bound GetExecutionTreeAsync edge scan to a root-anchored time window (partition elimination)"`
|
||||
|
||||
### Task 10: Fix the hardcoded-zero backlogTotal KPI history
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 2, 3, 6, 8, 11, 19, 21
|
||||
**Files:**
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Kpi/IAuditBacklogProvider.cs`
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/Kpi/CentralHealthAuditBacklogProvider.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ServiceCollectionExtensions.cs` (central registration; locate the file — `grep -n AddSingleton src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ServiceCollectionExtensions.cs`)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Kpi/AuditLogKpiSampleSource.cs`
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Kpi/AuditLogKpiSampleSourceTests.cs` (extend/create), `tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests` (provider test)
|
||||
|
||||
1. Failing test:
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task Collect_Uses_BacklogProvider_When_Present()
|
||||
{
|
||||
var repo = new FakeAuditLogRepository(new AuditLogKpiSnapshot(10, 1, BacklogTotal: 0, DateTime.UtcNow));
|
||||
var source = new AuditLogKpiSampleSource(repo, new FakeBacklogProvider(42));
|
||||
var samples = await source.CollectAsync(DateTime.UtcNow);
|
||||
Assert.Equal(42, samples.Single(s => s.Metric == KpiMetrics.AuditLog.BacklogTotal).Value);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Collect_FallsBack_To_Snapshot_When_Provider_Absent()
|
||||
{
|
||||
// ctor with provider: null → BacklogTotal sample = snapshot value (0) — unchanged behavior
|
||||
}
|
||||
```
|
||||
|
||||
2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter AuditLogKpiSampleSource` — FAIL.
|
||||
3. Implementation:
|
||||
- Commons: `public interface IAuditBacklogProvider { long GetPendingBacklogTotal(); }` (XML doc: sums latest per-site `SiteAuditBacklog.PendingCount` health reports; best-effort — sites not yet reporting contribute zero).
|
||||
- HealthMonitoring: `CentralHealthAuditBacklogProvider(ICentralHealthAggregator)` — port the exact summation from `CentralUI/Services/AuditLogQueryService.GetKpiSnapshotAsync` (~lines 145-155: `state.LatestReport?.SiteAuditBacklog?.PendingCount`, `> 0` guard). Register `services.AddSingleton<IAuditBacklogProvider, CentralHealthAuditBacklogProvider>();` in the central HealthMonitoring registration.
|
||||
- `AuditLogKpiSampleSource`: ctor gains `IAuditBacklogProvider? backlogProvider = null` (MS.DI resolves the default when unregistered — sites/tests unaffected); backlog sample value becomes `_backlogProvider?.GetPendingBacklogTotal() ?? snapshot.BacklogTotal`.
|
||||
- Optionally refactor `AuditLogQueryService` to consume the provider (skip if it drags UI DI churn — the live tile already works; the *history* was the broken surface).
|
||||
4. Run filter — PASS. Build the solution.
|
||||
5. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.Commons src/ZB.MOM.WW.ScadaBridge.HealthMonitoring src/ZB.MOM.WW.ScadaBridge.AuditLog tests && git commit -m "fix(kpi-history): backlogTotal trend records the real site-backlog aggregate instead of a hardwired zero"`
|
||||
|
||||
### Task 11: SiteCalls same-rank freshness upsert (unfreeze RetryCount/LastError)
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 2, 3, 4, 8, 9, 10
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs` (rank comment ~27-40; UPDATE predicate ~117-139)
|
||||
- Modify: `docs/requirements/Component-SiteCallAudit.md` (~76-78 upsert contract wording)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/SiteCallAuditRepositoryTests.cs` (extend; MSSQL — infra up)
|
||||
|
||||
1. Failing tests:
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task Upsert_SameRank_NewerUpdatedAt_RefreshesProgressFields()
|
||||
{
|
||||
var id = NewId();
|
||||
await Repo.UpsertAsync(Call(id, "Attempted", retryCount: 1, lastError: "timeout #1", updatedAt: T0));
|
||||
await Repo.UpsertAsync(Call(id, "Attempted", retryCount: 5, lastError: "timeout #5", updatedAt: T0.AddMinutes(5)));
|
||||
var row = await Repo.GetAsync(id);
|
||||
Assert.Equal(5, row!.RetryCount);
|
||||
Assert.Equal("timeout #5", row.LastError);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Upsert_SameRank_OlderUpdatedAt_IsNoOp() { /* retry 5 then out-of-order retry 2 with older stamp → stays 5 */ }
|
||||
|
||||
[Fact]
|
||||
public async Task Upsert_LowerRank_IsStillNoOp() { /* Delivered then Attempted (newer stamp) → stays Delivered */ }
|
||||
```
|
||||
|
||||
2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter SiteCallAuditRepositoryTests` — FAIL (same-rank is a no-op today).
|
||||
3. Change the UPDATE guard from `incomingRank > CASE...` to:
|
||||
|
||||
```sql
|
||||
AND ( {incomingRank} > (CASE Status ... END)
|
||||
OR ({incomingRank} = (CASE Status ... END) AND UpdatedAtUtc < {siteCall.UpdatedAtUtc}) )
|
||||
```
|
||||
|
||||
(repeat the CASE expression — both branches parameterize identically). Update the class comment: "monotonic on rank; within equal rank, newest `UpdatedAtUtc` wins (still idempotent — equal stamps are a no-op — and still regression-proof)". This unfreezes `RetryCount`/`LastError`/`HttpStatus` during the Attempted phase and lets a genuinely-newer terminal correction land.
|
||||
4. Run filter — PASS (all pre-existing monotonic tests must stay green).
|
||||
5. Doc: Component-SiteCallAudit.md upsert paragraph — replace "first-write-wins at each rank" semantics with "insert-if-not-exists, then upsert on newer status rank, with a newest-`UpdatedAtUtc` tiebreaker within equal rank so retrying calls show live RetryCount/LastError".
|
||||
6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs docs/requirements/Component-SiteCallAudit.md tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests && git commit -m "fix(site-call-audit): same-rank freshness tiebreaker — retrying calls no longer freeze RetryCount/LastError at first write"`
|
||||
|
||||
### Task 12: EF Core connection resiliency (`EnableRetryOnFailure`)
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 2, 3, 8, 11 (do NOT run concurrently with any task executing MSSQL integration tests — behavior-wide change)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ServiceCollectionExtensions.cs` (~29-34)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogIngestActor.cs` (`OnCachedTelemetryAsync` dual-write transaction ~262-311)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/ServiceCollectionExtensionsTests.cs` (extend), `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/CombinedTelemetryIdempotencyTests.cs` (must stay green)
|
||||
|
||||
1. First inventory every user-initiated transaction against `ScadaBridgeDbContext`: `grep -rn "BeginTransaction" src/ --include="*.cs"` — expect `AuditLogIngestActor.OnCachedTelemetryAsync`; treat any others found identically.
|
||||
2. Failing test in `ServiceCollectionExtensionsTests`:
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public void AddConfigurationDatabase_Configures_RetryingExecutionStrategy()
|
||||
{
|
||||
var services = new ServiceCollection();
|
||||
services.AddConfigurationDatabase("Server=unused;Database=unused;");
|
||||
using var sp = services.BuildServiceProvider();
|
||||
var options = sp.GetRequiredService<DbContextOptions<ScadaBridgeDbContext>>();
|
||||
var sqlExt = options.Extensions.OfType<SqlServerOptionsExtension>().Single();
|
||||
Assert.NotNull(sqlExt.ExecutionStrategyFactory); // retry strategy configured
|
||||
}
|
||||
```
|
||||
|
||||
3. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter Configures_RetryingExecutionStrategy` — FAIL.
|
||||
4. Implementation:
|
||||
- `options.UseSqlServer(connectionString, sql => sql.EnableRetryOnFailure(maxRetryCount: 5, maxRetryDelay: TimeSpan.FromSeconds(30), errorNumbersToAdd: null))` — comment: covers every read-side path (KPI asks, outbox queries, `Notify.Status`, execution trees) that previously surfaced transient faults raw (arch-review 04, S5).
|
||||
- `OnCachedTelemetryAsync`: wrap the per-entry `BeginTransactionAsync` block in the execution strategy — `var strategy = context.Database.CreateExecutionStrategy(); await strategy.ExecuteAsync(async () => { await using var tx = await context.Database.BeginTransactionAsync(); ... await tx.CommitAsync(); });` — the block is already idempotent (insert-if-not-exists + monotonic upsert), so a retried unit is safe; say so in the comment.
|
||||
- Note in `SwitchOutPartitionAsync`'s comment: the raw batch carries its own server-side BEGIN TRAN/CATCH and is IF-EXISTS-guarded idempotent, so strategy-level replay of the whole batch is safe.
|
||||
5. Run: the new filter → PASS; then with infra up, `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter "CombinedTelemetry"` → PASS (proves the dual-write still commits atomically under the strategy).
|
||||
6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ServiceCollectionExtensions.cs src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogIngestActor.cs tests && git commit -m "fix(config-db): EnableRetryOnFailure fleet-wide + execution-strategy wrap of the combined-telemetry dual-write"`
|
||||
|
||||
### Task 13: Audit row for operator Retry on parked notifications
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 8, 9, 11, 18, 19, 21
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Notification/NotificationOutboxQueries.cs` (`RetryNotificationRequest` ~54, `DiscardNotificationRequest` ~69 — additive `string? RequestedBy = null`)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/NotificationOutboxActor.cs` (`RetryAsync` ~951-977; `BuildNotifyDeliverEvent` ~675-714 gains `string? actorOverride = null`; `EmitTerminalAuditAsync` passes it for discard)
|
||||
- Modify: callers that send these requests — locate with `grep -rln "RetryNotificationRequest(" src/` (expect the Central UI outbox service and/or ManagementActor) and pass the authenticated username
|
||||
- Modify: `docs/requirements/Component-NotificationOutbox.md` (~128)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests/NotificationOutboxActorRetryEmissionTests.cs` (create; clone the audit-writer capture rig from `NotificationOutboxActorTerminalEmissionTests.cs`)
|
||||
|
||||
1. Failing test:
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task Retry_OnParked_Emits_Submitted_NotifyDeliver_AuditRow_With_Operator()
|
||||
{
|
||||
// seed a Parked notification; send RetryNotificationRequest(corr, id, RequestedBy: "jdoe")
|
||||
// expect RetryNotificationResponse Success
|
||||
// assert captured audit writer received exactly one event:
|
||||
// Kind NotifyDeliver, Status Submitted, Actor "jdoe", CorrelationId == NotificationId
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Retry_AuditFailure_DoesNotFail_TheRetry() { /* throwing audit writer → response still Success (audit is best-effort) */ }
|
||||
```
|
||||
|
||||
2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests --filter RetryEmission` — FAIL.
|
||||
3. Implementation:
|
||||
- Add `string? RequestedBy = null` to both request records (additive-only contract evolution — existing senders compile unchanged).
|
||||
- `BuildNotifyDeliverEvent(..., string? actorOverride = null)`: `actor: actorOverride ?? SystemActor`.
|
||||
- In `RetryAsync`, after `repository.UpdateAsync(notification)`: emit `BuildNotifyDeliverEvent(notification, now, AuditStatus.Submitted, errorMessage: null, actorOverride: request.RequestedBy)` via `_auditWriter.WriteAsync` inside the same try/catch-and-log shape as `EmitTerminalAuditAsync` (audit failure never aborts the retry). `Submitted` = "operator re-queued it" — the forensic gap the review names (`Parked → Attempted → Delivered` with no record of who un-parked).
|
||||
- `DiscardAsync`: pass `request.RequestedBy` through to the terminal row's actor.
|
||||
- Plumb the username at the senders (the UI/management layer knows the authenticated user; pass `null` where no identity exists).
|
||||
4. Run filter + `dotnet test tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests` — PASS.
|
||||
5. Doc: Component-NotificationOutbox.md ~128 — "…each transition emits the corresponding audit row: operator **Retry** emits a `Notification`-channel `NotifyDeliver` row with status `Submitted` and the operator as `Actor`; Discard emits the `Terminal` row (operator as `Actor`)."
|
||||
6. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.Commons src/ZB.MOM.WW.ScadaBridge.NotificationOutbox src/ZB.MOM.WW.ScadaBridge.CentralUI src/ZB.MOM.WW.ScadaBridge.ManagementService docs/requirements/Component-NotificationOutbox.md tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests && git commit -m "feat(notification-outbox): operator Retry/Discard emit audit rows with operator identity"`
|
||||
|
||||
### Task 14: Operator identity on the SiteCalls Retry/Discard relay
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 8, 9, 10, 19, 20
|
||||
**Files:**
|
||||
- Modify: the relay request records — locate with `grep -rn "record RetryParkedOperation\|record DiscardParkedOperation" src/ZB.MOM.WW.ScadaBridge.Commons` (additive `string? RequestedBy = null`)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/SiteCallAuditActor.cs` (relay handlers — see `SiteCallRelayTests.cs` for the handler names)
|
||||
- Modify: `docs/requirements/Component-SiteCallAudit.md` (relay section)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests/SiteCallRelayTests.cs` (extend)
|
||||
|
||||
1. Failing test: relay a Retry for a parked row with `RequestedBy: "jdoe"`; assert the injected central audit writer (same `ICentralAuditWriter` seam `NotificationOutboxActor` uses — confirm with `grep -rn "ICentralAuditWriter" src/ZB.MOM.WW.ScadaBridge.NotificationOutbox src/ZB.MOM.WW.ScadaBridge.Commons`) received one event: `Kind = CachedResolve`, `Status = Submitted`, `Actor = "jdoe"`, `CorrelationId = TrackedOperationId`, channel taken from the stored `SiteCalls` row's `Channel`. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests --filter SiteCallRelayTests` — FAIL.
|
||||
2. Implementation: inject `ICentralAuditWriter` into `SiteCallAuditActor` (optional ctor param defaulting to null → no-op, so existing tests construct unchanged); in the Retry/Discard relay handlers, after the site relay succeeds, fetch the row (`ISiteCallAuditRepository.GetAsync`) for its `Channel`, build the event via `ScadaBridgeAuditEventFactory.Create` (mirror `BuildNotifyDeliverEvent`'s shape), write best-effort (try/catch + log — the relay outcome is authoritative, audit never aborts it). Site-side telemetry still records the state change itself; this row adds *who asked*.
|
||||
3. Run filter — PASS.
|
||||
4. Doc: Component-SiteCallAudit.md relay section — the relay now emits a central direct-write audit row carrying operator identity; sites remain the source of truth for the state change.
|
||||
5. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.Commons src/ZB.MOM.WW.ScadaBridge.SiteCallAudit docs/requirements/Component-SiteCallAudit.md tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests && git commit -m "feat(site-call-audit): relay Retry/Discard emit operator-identity audit rows"`
|
||||
|
||||
### Task 15: Composite keyset in the SiteCalls reconciliation pull contract (site side)
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 2, 3, 8, 13
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto` (additive `string after_id` on the SiteCalls pull request message — locate exact message with `grep -n "PullSiteCalls" src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto`)
|
||||
- Modify: the Commons pull-request message (`grep -rn "record CachedCallReconcileRequest\|PullSiteCallsRequest" src/ZB.MOM.WW.ScadaBridge.Commons`) — additive `string? AfterId = null`
|
||||
- Modify: the site-side pull handler + tracking-store query (`grep -rn "ReadSince\|PullSiteCalls" src/ZB.MOM.WW.ScadaBridge.StoreAndForward src/ZB.MOM.WW.ScadaBridge.Communication --include="*.cs"`) — order by `(UpdatedAtUtc, TrackedOperationId)`, filter `UpdatedAtUtc > since OR (UpdatedAtUtc = since AND TrackedOperationId > afterId)` (when `afterId` present; absent → legacy `>= since` behavior)
|
||||
- Test: site-side handler/store tests (same project as the handler; locate via the greps above)
|
||||
|
||||
1. Failing test on the site store/handler: seed `batchSize + 2` rows sharing one `UpdatedAtUtc`; pull with `since = thatTimestamp, afterId = <id of last row of page 1>`; assert page 2 returns the remaining rows (today it returns the same page). Run the owning test project with `--filter <new test name>` — FAIL.
|
||||
2. Implement the additive proto field (field number = next free; additive-only rule), regenerate/compile, map it through the DTO, and apply the keyset predicate + deterministic `(UpdatedAtUtc, TrackedOperationId)` ordering in the store query. Absent `AfterId` preserves the exact legacy contract (old central against new site: unchanged).
|
||||
3. Run the owning project's full suite — PASS.
|
||||
4. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.Communication src/ZB.MOM.WW.ScadaBridge.Commons src/ZB.MOM.WW.ScadaBridge.StoreAndForward tests && git commit -m "feat(site-call-audit): additive (UpdatedAtUtc, TrackedOperationId) keyset in the reconciliation pull contract"`
|
||||
|
||||
### Task 16: Central keyset cursor — un-pin the SiteCalls reconciliation loop
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 5, 8, 9, 10 (requires Task 15)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/SiteCallAuditActor.cs` (reconciliation loop ~600-670: cursor becomes `(DateTime, string?)`; pin branch ~636-654)
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/SiteCallReconciliationPinnedChanged.cs` (EventStream record, mirroring `SiteAuditTelemetryStalledChanged`)
|
||||
- Modify: `docs/requirements/Component-SiteCallAudit.md` (reconciliation cursor paragraph)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests/SiteCallAuditReconciliationTests.cs` (extend)
|
||||
|
||||
1. Failing test: fake pull client honoring the keyset; seed `2 × batchSize` rows sharing one `UpdatedAtUtc`; drive one tick; assert **all** rows were upserted (today the tail never reconciles). Second test: a fake *legacy* client that ignores `AfterId` (returns the same page) → assert the actor publishes `SiteCallReconciliationPinnedChanged(siteId, pinned: true)` on the EventStream instead of silently logging. `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests --filter Reconciliation` — FAIL.
|
||||
2. Implement: per-site cursor `(DateTime since, string? afterId)`; after each page set `afterId` to the max `(UpdatedAtUtc, TrackedOperationId)` row's id; pass it on the next pull. Keep the no-progress detection as the legacy-site fallback, but publish the new event (and clear it with `pinned: false` when progress resumes) — the dead-end becomes a health-observable condition.
|
||||
3. Run filter — PASS; run the full SiteCallAudit.Tests suite — PASS.
|
||||
4. Doc: Component-SiteCallAudit.md — cursor is now a composite `(UpdatedAtUtc, TrackedOperationId)` keyset (additive contract field; legacy sites fall back to the timestamp cursor and surface `SiteCallReconciliationPinnedChanged` when pinned). Also add one line to Component-AuditLog.md's reconciliation section: the analogous keyset for the `PullAuditEvents` cursor is a tracked follow-up (same shape; lower urgency because the audit cursor already re-pulls idempotently).
|
||||
5. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.SiteCallAudit docs/requirements tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests && git commit -m "fix(site-call-audit): composite keyset cursor eliminates the single-timestamp reconciliation pin; pinned state now a published event"`
|
||||
|
||||
### Task 17: Durable record for permanently-abandoned reconciliation rows
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 8, 11, 19, 20, 21
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Types/Enums/AuditKind.cs` (additive `ReconciliationAbandoned` value)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/SiteAuditReconciliationActor.cs` (abandonment branch ~283-293)
|
||||
- Modify: `docs/requirements/Component-AuditLog.md` (kind vocabulary + reconciliation section)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/SiteAuditReconciliationActorTests.cs` (extend)
|
||||
|
||||
1. Failing test: repository stub whose `InsertIfNotExistsAsync` throws for one specific EventId across `MaxPermanentInsertAttempts` ticks but accepts everything else; after the abandonment tick, assert a **synthetic** audit event was inserted (new EventId; `Kind = ReconciliationAbandoned`; `DetailsJson`/Extra carrying the abandoned `EventId`, the site id, and the final error) and the cursor advanced. `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter SiteAuditReconciliationActorTests` — FAIL.
|
||||
2. Implement in the `attempts >= MaxPermanentInsertAttempts` branch: alongside the existing `LogCritical`, build a minimal synthetic event (fresh `Guid.NewGuid()` EventId, `OccurredAtUtc = now`, Actor = system, Action `"Audit.ReconciliationAbandoned"`, Category preserved from the lost event where parseable, Extra = `{ abandonedEventId, sourceSiteId, error }` — use the same factory/projection helpers the ingest path uses) and `InsertIfNotExistsAsync` it in its own try/catch (the synthetic row is small and well-formed, so the row-specific failure that killed the original won't recur; if it does, log and continue — never block the cursor twice). The permanent loss is now queryable in the audit log itself, not only in a rotating log file.
|
||||
3. Run filter — PASS.
|
||||
4. Doc: add `ReconciliationAbandoned` to the kind vocabulary in Component-AuditLog.md (Task 24 updates the Commons enum listing — coordinate: this task adds the value, 24 fixes the counts).
|
||||
5. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.Commons src/ZB.MOM.WW.ScadaBridge.AuditLog docs/requirements/Component-AuditLog.md tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests && git commit -m "feat(audit-log): permanently-abandoned reconciliation rows leave a durable ReconciliationAbandoned audit record"`
|
||||
|
||||
### Task 18: Reconciliation NodeB failover (owned from report 08)
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 9, 11, 13, 19, 20, 21
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/ISiteEnumerator.cs` (`SiteEntry` record ~37: additive `string? FallbackGrpcEndpoint = null`; fix the stale "NodeA-only first cut" doc comment)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/SiteEnumerator.cs` (~60-75: populate from `GrpcNodeBAddress`; include sites with blank NodeA but valid NodeB, primary = NodeA else NodeB)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/GrpcPullAuditEventsClient.cs` + `GrpcPullSiteCallsClient.cs` (retry-once-on-fallback)
|
||||
- Modify: `docs/requirements/Component-AuditLog.md` (reconciliation endpoint-resolution paragraph)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/SiteEnumeratorTests.cs`, `GrpcPullAuditEventsClientTests.cs`, `GrpcPullSiteCallsClientTests.cs` (extend)
|
||||
|
||||
1. Failing tests: (a) `SiteEnumeratorTests` — site with both addresses yields `SiteEntry(id, nodeA, FallbackGrpcEndpoint: nodeB)`; site with only NodeB is no longer skipped; (b) `GrpcPullAuditEventsClientTests` — fake invoker throws `Unavailable` for the primary endpoint and succeeds for the fallback → the pull returns the fallback's events (today: collapses to empty). `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter "SiteEnumeratorTests|GrpcPull"` — FAIL.
|
||||
2. Implement: in both pull clients' transport-fault catch (the `Unavailable`/`DeadlineExceeded`/`Cancelled`/`HttpRequestException` set already enumerated there), if `FallbackGrpcEndpoint` is non-blank and differs from the primary, invoke once against it before collapsing to empty; log the failover at Information. The invoker's per-endpoint channel cache already handles the second endpoint.
|
||||
3. Run filter + full Central test folder — PASS.
|
||||
4. Doc: Component-AuditLog.md — reconciliation dials NodeA first, fails over to NodeB per call; during a NodeA outage the loss-recovery safety net stays available.
|
||||
5. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.AuditLog docs/requirements/Component-AuditLog.md tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests && git commit -m "fix(audit-log): reconciliation pulls fail over to site NodeB when NodeA is unreachable"`
|
||||
|
||||
### Task 19: Batch the KpiSample purge
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 1, 2, 3, 8, 9, 13, 18
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs` (`PurgeOlderThanAsync` ~65-71)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IKpiHistoryRepository.cs` (doc update; signature unchanged)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/KpiHistoryRepositoryTests.cs` (extend)
|
||||
|
||||
1. Failing test: seed samples spread across 5 days; purge with a 2-day-old cutoff; assert the correct rows are gone and the return value equals the total deleted — then assert via an EF command interceptor (or SQL capture, matching how sibling tests inspect SQL) that **more than one** DELETE statement was issued for a multi-window purge. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter KpiHistoryRepositoryTests` — FAIL (single `ExecuteDeleteAsync` today).
|
||||
2. Implement provider-portable time-slice batching (avoids `DELETE TOP` dialect divergence — the convention Task 24 documents):
|
||||
|
||||
```csharp
|
||||
public async Task<int> PurgeOlderThanAsync(DateTime before, CancellationToken cancellationToken = default)
|
||||
{
|
||||
// Time-sliced batches: each DELETE covers at most one hour of samples, capping the
|
||||
// lock/log footprint per statement (arch-review 04, P4 — steady state is ~1 day of
|
||||
// rows/day; after an outage the catch-up would otherwise be one giant transaction).
|
||||
var total = 0;
|
||||
var floor = await _context.KpiSamples.Where(s => s.CapturedAtUtc < before)
|
||||
.MinAsync(s => (DateTime?)s.CapturedAtUtc, cancellationToken);
|
||||
while (floor is not null && floor < before)
|
||||
{
|
||||
var ceiling = floor.Value.AddHours(1) < before ? floor.Value.AddHours(1) : before;
|
||||
total += await _context.KpiSamples
|
||||
.Where(s => s.CapturedAtUtc < ceiling)
|
||||
.ExecuteDeleteAsync(cancellationToken);
|
||||
floor = await _context.KpiSamples.Where(s => s.CapturedAtUtc < before)
|
||||
.MinAsync(s => (DateTime?)s.CapturedAtUtc, cancellationToken);
|
||||
}
|
||||
return total;
|
||||
}
|
||||
```
|
||||
|
||||
3. Run filter — PASS; run `dotnet test tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests` (recorder actor contract unchanged) — PASS.
|
||||
4. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IKpiHistoryRepository.cs tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests && git commit -m "perf(kpi-history): time-sliced batched purge replaces the single daily mega-DELETE"`
|
||||
|
||||
### Task 20: NotificationOutbox KPI — single-query aggregation, bounded oldest
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 2, 3, 8, 9, 14, 18, 19
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/NotificationOutboxRepository.cs` (`ComputeKpisAsync` ~239-283, `ComputePerSiteKpisAsync` ~286-334, `ComputePerNodeKpisAsync` ~337-391)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/NotificationOutboxRepositoryPerSiteKpiTests.cs`, `...PerNodeKpiTests.cs`, and the global KPI tests (all must stay green — this is a pure refactor of query shape)
|
||||
|
||||
1. Add one new failing test capturing the *bounded-oldest* behavior: seed 3 non-terminal rows; assert the oldest-pending computation issues no query that materializes more than 1 row (assert via command interceptor / captured SQL: the oldest query is `ORDER BY … OFFSET 0 ROWS FETCH NEXT 1` or `TOP(1)`, not a full non-terminal SELECT). Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter NotificationOutboxRepository` — FAIL (per-site/per-node paths materialize every non-terminal row today, ~lines 312-318, 368-375).
|
||||
2. Refactor each snapshot to:
|
||||
- **Counts:** one conditional-aggregation query per scope — `GroupBy(n => 1)` (global) / `GroupBy(n => n.SourceSiteId)` / `GroupBy(n => n.SourceNode)` with `g.Count(n => <predicate>)` per metric, replacing the 5-7 sequential round trips.
|
||||
- **Oldest pending:** global — `Where(nonTerminal).OrderBy(n => n.CreatedAt).Select(n => (DateTimeOffset?)n.CreatedAt).FirstOrDefaultAsync()`; per-site/per-node — try server-side `GroupBy(...).Select(g => new { g.Key, Oldest = g.Min(n => n.CreatedAt) })` first; if the `DateTimeOffset` converter blocks translation (the review's noted awkwardness), fall back to projecting only `(Key, CreatedAt)` pairs of non-terminal rows (narrow projection, no entity materialization) and reduce in memory — either way the full-entity materialization is gone.
|
||||
3. Run the full NotificationOutboxRepository test set + `dotnet test tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests --filter Kpi` — PASS (identical snapshot values).
|
||||
4. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/NotificationOutboxRepository.cs tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests && git commit -m "perf(notification-outbox): single-query KPI aggregation; oldest-pending no longer materializes the live queue"`
|
||||
|
||||
### Task 21: `OPTION (RECOMPILE)` on the SiteCalls catch-all query
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~2 min
|
||||
**Parallelizable with:** 1, 2, 3, 8, 9, 13, 18, 19, 20 (NOT 11 — same file)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs` (`QueryAsync` SQL ~186-202)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/SiteCallAuditRepositoryTests.cs` (existing query tests must pass)
|
||||
|
||||
1. Append `OPTION (RECOMPILE)` after the `ORDER BY` line of the `FormattableString` (with a comment: every filter is the `(@p IS NULL OR col=@p)` optional-parameter shape; RECOMPILE lets the optimizer prune dead predicates per invocation and pick `IX_SiteCalls_Status_Updated`/`IX_SiteCalls_NonTerminal` — arch-review 04, P5; per-invocation compile cost is fine at UI-page cadence).
|
||||
2. With infra up: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter SiteCallAuditRepositoryTests` — PASS.
|
||||
3. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs && git commit -m "perf(site-call-audit): OPTION (RECOMPILE) on the optional-parameter query page"`
|
||||
|
||||
### Task 22: Short first-tick delay on the three purge timers
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~3 min
|
||||
**Parallelizable with:** 1, 2, 6, 8, 19, 20, 21 (NOT 4/5 — AuditLogPurgeActor; NOT 16 — SiteCallAuditActor)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeActor.cs` (`PreStart` ~95-100)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/SiteCallAuditActor.cs` (`StartPurgeTimer` ~374-387)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.KpiHistory/KpiHistoryRecorderActor.cs` (purge timer ~124-128)
|
||||
- Test: extend each actor's existing test file with one fast-first-tick case
|
||||
|
||||
1. Failing test (one per actor, using each suite's existing scheduler/interval-override rig): configure a long interval (24h) and assert a purge tick arrives within the short initial delay window rather than never.
|
||||
2. Implement uniformly: `var initialDelay = interval < ShortFirstTick ? interval : ShortFirstTick;` with `private static readonly TimeSpan ShortFirstTick = TimeSpan.FromMinutes(5);` and a shared comment: purges are idempotent; `initialDelay = interval` meant a node recycled daily would never purge (arch-review 04, S9). Taking `min(interval, 5min)` keeps millisecond test cadences unchanged.
|
||||
3. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter AuditLogPurgeActorTests && dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests --filter Purge && dotnet test tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests` — PASS.
|
||||
4. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.AuditLog src/ZB.MOM.WW.ScadaBridge.SiteCallAudit src/ZB.MOM.WW.ScadaBridge.KpiHistory tests && git commit -m "fix(purge): short first tick on all three purge timers so daily-recycled nodes still purge"`
|
||||
|
||||
### Task 23: Low-severity cleanup batch (param chunking, KPI catalog, computed-column guard)
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 4, 5, 8, 9, 11, 20
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriter.cs` (`MarkForwardedAsync` ~644-664, `MarkReconciledAsync` ~735-753)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Types/Kpi/KpiMetrics.cs` (locate: `grep -rn "class KpiMetrics" src/ZB.MOM.WW.ScadaBridge.Commons`)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/Kpi/NotificationOutboxKpiSampleSource.cs` (~40-43)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/AuditLogEntityTypeConfiguration.cs` (~158-164)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SqliteAuditWriterWriteTests.cs` (extend), `tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests/Kpi` (extend)
|
||||
|
||||
1. **P8 — parameter chunking:** failing test — write 1200 events, `MarkForwardedAsync(all 1200 ids)`, assert all flip to Forwarded (today: one statement with 1200+ parameters risks/violates SQLite's 999 limit as batch sizes grow). Implement: chunk `eventIds` into slices of 500 inside the existing `_writeLock` (one transaction around all chunks so the batch stays atomic); same in `MarkReconciledAsync`. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter SqliteAuditWriter` — PASS.
|
||||
2. **C4 — KPI metric catalog:** add `KpiMetrics.NotificationOutbox.StuckCount = "stuckCount"` and `OldestPendingAgeSeconds = "oldestPendingAgeSeconds"` (EXACT existing string values — persisted, renames forbidden); replace the inline literals in `NotificationOutboxKpiSampleSource` with the constants; add a test asserting the constant values equal the historical strings.
|
||||
3. **P7 — computed-column guard:** extend the `IngestedAtUtc` comment in `AuditLogEntityTypeConfiguration`: "NEVER use IngestedAtUtc in a WHERE predicate — non-persisted, it forces a full-table `JSON_VALUE` parse; read-side projection only (arch-review 04, P7)."
|
||||
4. Run both test filters + build — PASS.
|
||||
5. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.AuditLog src/ZB.MOM.WW.ScadaBridge.Commons src/ZB.MOM.WW.ScadaBridge.NotificationOutbox src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase tests && git commit -m "chore(data-layer): low-severity cleanups — SQLite param chunking, KPI metric catalog, computed-column predicate guard"`
|
||||
|
||||
### Task 24: Design-doc reconciliation pass
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** all code tasks except 7 (same files); run AFTER 17 (AuditKind gains a value)
|
||||
**Files:**
|
||||
- Modify: `docs/requirements/Component-Commons.md` (~46-47, ~89)
|
||||
- Modify: `docs/requirements/Component-ConfigurationDatabase.md` (~58; timestamp-convention + clustered-key + repository-dialect notes)
|
||||
- Modify: `docs/requirements/Component-SiteCallAudit.md` (~44-51 schema section)
|
||||
- Modify: `docs/requirements/Component-AuditLog.md` (~396-399 per-channel config channel list)
|
||||
- Modify: `docs/requirements/Component-NotificationOutbox.md` (~23 "dedicated blocking-I/O dispatcher" phrasing — align with Task 25 if it has landed, else describe the actual thread-pool off-actor delivery)
|
||||
|
||||
Concrete edits (each verifiable against the code cited in the review):
|
||||
1. Component-Commons.md ~47: `AuditKind` list → the current enum (`grep -A20 "enum AuditKind" src/ZB.MOM.WW.ScadaBridge.Commons/Types/Enums/AuditKind.cs`): the original 10 + `SecuredWriteSubmitted/Approved/Rejected/Executed` (4) + `ReconciliationAbandoned` (Task 17). ~46: `AuditChannel` → 5 values incl. `SecuredWrite`.
|
||||
2. Component-Commons.md ~89: `NotificationType` → "`Email`, `Sms` (shipped 2026-06-19; the Teams plan was evaluated and dropped)".
|
||||
3. Component-ConfigurationDatabase.md ~58: NotificationList Type discriminator → "`Email` / `Sms`".
|
||||
4. Component-ConfigurationDatabase.md ~399: "single global value in v1, no per-channel overrides" → per-channel `AuditLog:PerChannelRetentionDays` overrides exist (skip if Task 7 already fixed it).
|
||||
5. Component-SiteCallAudit.md ~44-51: rewrite the schema table to the shipped columns (`Channel`/`Target`/`AuditStatus`-derived `Status` strings, `SourceNode`, no provenance columns) — align with `SiteCallEntityTypeConfiguration.cs` and Component-ConfigurationDatabase.md:64 (the current source of truth per the review).
|
||||
6. Component-AuditLog.md ~396-399: per-channel retention config enumerates all five channels including `SecuredWrite` (the validator already accepts it — doc under-promised).
|
||||
7. Component-Commons.md / Component-ConfigurationDatabase.md: document the `AuditLogRow` deviation (persistence-only projection of the external `ZB.MOM.WW.Audit.AuditEvent` contract — deliberately not a Commons POCO) and two conventions: **new tables use UTC `DateTime` + the Utc converter** (the `DateTimeOffset` choice on `Notification` is the documented exception that forced in-memory Min workarounds), and **new repositories write MSSQL-only SQL with MSSQL-backed test fixtures** (the NotificationOutboxRepository dual SQLite/T-SQL dialect is a documented legacy exception — finding S8). Record two tracked follow-ups: the AuditLog clustered-key `(OccurredAtUtc, EventId)` benchmark question (P3) and the per-node KPI sampling cadence / outbox keyset-paging deferrals (P4/P6).
|
||||
8. Verify: `grep -rn "Teams" docs/requirements/ | grep -iv "dropped\|evaluated"` returns nothing load-bearing; `grep -n "no per-channel" docs/requirements/Component-ConfigurationDatabase.md` returns nothing.
|
||||
9. Commit: `git add docs/requirements && git commit -m "docs: reconcile design docs with shipped code (enums, Sms, per-channel retention, SiteCalls schema, conventions)"`
|
||||
|
||||
### Task 25: Bounded parallel notification delivery
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 8, 9, 11, 18, 19 (NOT 13 — same actor file; do 13 first)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/NotificationOutboxOptions.cs` (add `MaxParallelDeliveries`)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/NotificationOutboxActor.cs` (`RunDispatchPass` ~321-347)
|
||||
- Modify: `docs/requirements/Component-NotificationOutbox.md` (~23 dispatcher paragraph)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests/NotificationOutboxActorDispatchTests.cs` (extend)
|
||||
|
||||
1. Failing test: adapter stub whose `Deliver` blocks on a gate and records concurrent-invocation high-water mark; enqueue 8 pending notifications; with `MaxParallelDeliveries = 4` assert high-water ≥ 2 and all 8 reach terminal status; with `MaxParallelDeliveries = 1` assert high-water == 1 (existing sequential behavior preserved). `dotnet test tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests --filter Dispatch` — FAIL.
|
||||
2. Implement: `MaxParallelDeliveries` (default **4**, clamp min 1 — the review's alarm-storm scenario: sequential 5 s SMTP attempts cap throughput at ~0.2/s regardless of batch size). In `RunDispatchPass`, replace the sequential foreach with `SemaphoreSlim(maxParallel)` + `Task.WhenAll`, giving **each notification its own DI scope/repository** (rows are claimed per-notification, so no shared-context concurrency; the existing in-flight pass guard is untouched). Audit emission stays per-notification inside its task.
|
||||
3. Run the full outbox suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests` — PASS (attempt/terminal emission tests prove per-row semantics survived).
|
||||
4. Doc: Component-NotificationOutbox.md dispatcher section — delivery runs off the actor thread on the thread pool with bounded per-sweep parallelism (`MaxParallelDeliveries`, default 4); drop the inaccurate "dedicated blocking-I/O dispatcher" phrasing (doc drift the review flagged).
|
||||
5. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.NotificationOutbox docs/requirements/Component-NotificationOutbox.md tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests && git commit -m "perf(notification-outbox): bounded parallel delivery within a dispatch sweep (default 4)"`
|
||||
|
||||
---
|
||||
|
||||
## Dependencies on other plans
|
||||
|
||||
- **Plan 02 (Communication/S&F)** owns the S&F retry-sweep batching — Task 15's site-side tracking-store query touches adjacent code in the StoreAndForward project; coordinate file ownership if both run concurrently.
|
||||
- **Plan 07 (UI/Management/Security)** owns SecuredWrite emission paths — the `SourceNode`-NULL follow-up (U4) stays with them; Task 24 only fixes the doc under-promise.
|
||||
- **Plan 08 (Conventions/Tests)** flagged the reconciliation NodeB gap; it is implemented here (Task 18) — plan 08 should not duplicate it.
|
||||
- Tasks 13/14 plumb `RequestedBy` through UI/ManagementActor senders — if plan 07 refactors those senders simultaneously, land this plan's additive message change first (defaults keep old senders compiling).
|
||||
- Migration serialization is **intra-plan only** (Tasks 6 → 8); no other plan currently adds ConfigurationDatabase migrations, but if one does, coordinate snapshot ordering.
|
||||
|
||||
## Execution order
|
||||
|
||||
**P0 (the operational time bombs — do first, in this order):** 1 → 2 → 3 (site purge chain), 4 → 5 (partition-switch hardening), 6 → 8 (migrations, serialized), 9, 10.
|
||||
**P1:** 11, 12, 13, 15 → 16, 17, 18.
|
||||
**P2 (hardening/cleanup):** 14, 19, 20, 21, 22, 23, 25, then 7 and 24 (docs last so they describe what actually landed).
|
||||
Intra-plan critical path: **1 → 2 → 3** and **15 → 16**; everything else is broadly parallelizable per each task's "Parallelizable with" contract. Finish with a full `dotnet build ZB.MOM.WW.ScadaBridge.slnx && dotnet test ZB.MOM.WW.ScadaBridge.slnx` (infra compose up for the MSSQL suites) before declaring the plan done.
|
||||
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"planPath": "archreview/plans/PLAN-04-data-audit-backbone.md",
|
||||
"tasks": [
|
||||
{ "id": 1, "subject": "Task 1: Site SQLite retention purge — PurgeExpiredAsync on the writer", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 2, "subject": "Task 2: Site retention options", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 3, "subject": "Task 3: Site retention hosted job + registration + design doc", "status": "pending", "blockedBy": [1, 2] },
|
||||
{ "id": 4, "subject": "Task 4: Maintenance command timeout on partition switch + per-channel purge", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 5, "subject": "Task 5: Purge-failure health event (not just a log line)", "status": "pending", "blockedBy": [4] },
|
||||
{ "id": 6, "subject": "Task 6: Fix purger role grants (migration)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 7, "subject": "Task 7: Honest append-only enforcement story (docs)", "status": "pending", "blockedBy": [6] },
|
||||
{ "id": 8, "subject": "Task 8: SiteCalls filtered non-terminal index (migration)", "status": "pending", "blockedBy": [6] },
|
||||
{ "id": 9, "subject": "Task 9: Bound the execution-tree edge scan with a root time window", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 10, "subject": "Task 10: Fix the hardcoded-zero backlogTotal KPI history", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 11, "subject": "Task 11: SiteCalls same-rank freshness upsert (unfreeze RetryCount/LastError)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 12, "subject": "Task 12: EF Core connection resiliency (EnableRetryOnFailure)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 13, "subject": "Task 13: Audit row for operator Retry on parked notifications", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 14, "subject": "Task 14: Operator identity on the SiteCalls Retry/Discard relay", "status": "pending", "blockedBy": [13] },
|
||||
{ "id": 15, "subject": "Task 15: Composite keyset in the SiteCalls reconciliation pull contract (site side)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 16, "subject": "Task 16: Central keyset cursor — un-pin the SiteCalls reconciliation loop", "status": "pending", "blockedBy": [15] },
|
||||
{ "id": 17, "subject": "Task 17: Durable record for permanently-abandoned reconciliation rows", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 18, "subject": "Task 18: Reconciliation NodeB failover (owned from report 08)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 19, "subject": "Task 19: Batch the KpiSample purge", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 20, "subject": "Task 20: NotificationOutbox KPI — single-query aggregation, bounded oldest", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 21, "subject": "Task 21: OPTION (RECOMPILE) on the SiteCalls catch-all query", "status": "pending", "blockedBy": [11] },
|
||||
{ "id": 22, "subject": "Task 22: Short first-tick delay on the three purge timers", "status": "pending", "blockedBy": [5, 16] },
|
||||
{ "id": 23, "subject": "Task 23: Low-severity cleanup batch (param chunking, KPI catalog, computed-column guard)", "status": "pending", "blockedBy": [1] },
|
||||
{ "id": 24, "subject": "Task 24: Design-doc reconciliation pass", "status": "pending", "blockedBy": [7, 17] },
|
||||
{ "id": 25, "subject": "Task 25: Bounded parallel notification delivery", "status": "pending", "blockedBy": [13] },
|
||||
{ "id": 26, "subject": "Final verification: full solution build + test sweep (infra up for MSSQL suites)", "status": "pending", "blockedBy": [3, 8, 9, 10, 12, 14, 18, 19, 20, 21, 22, 23, 24, 25] }
|
||||
],
|
||||
"lastUpdated": "2026-07-08"
|
||||
}
|
||||
@@ -0,0 +1,632 @@
|
||||
# Templates, Deployment & Transport Fix Implementation Plan
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Close every finding in `archreview/05-templates-deployment-transport.md` — the Transport import silent-data-loss family (inheritance, script fields, native alarm sources, AreaName), the flattener/collision repeated-composition bug, revision-hash staleness blindness, native-alarm lock enforcement, Roslyn compile cost/leak on read paths, deployment audit/delete defects, script-trust hardening, and the cross-cutting script-artifact invalidation contract.
|
||||
|
||||
**Architecture:** The Transport import apply path (`BundleImporter`) re-implements DTO→entity mapping by hand and drifted from the DTO layer; fixes restore field-for-field fidelity, add the missing inheritance/native-source passes, and single-source the diff/sync equality predicates so they cannot drift again, guarded structurally by a per-entity round-trip equivalence test suite. TemplateEngine fixes convert the shared cycle-guard `visited` set to a recursion-path set (composition is by-slot, not by-template), extend the revision hash/diff to the native-alarm member class, and thread `IsLocked` through `ResolvedNativeAlarmSource`. A new Commons seam (`IScriptArtifactChangeBus` + `ScriptArtifactsChanged`) defines the "script artifact changed → invalidate everywhere" contract; Transport publishes post-commit, plan 06 consumes it in the Inbound API.
|
||||
|
||||
**Tech Stack:** C#/.NET, EF Core (in-memory + MSSQL integration fixtures), Roslyn (`Microsoft.CodeAnalysis.CSharp.Scripting`), xUnit. Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per project: `dotnet test tests/<project>`.
|
||||
|
||||
## Findings Coverage
|
||||
|
||||
| Report finding | Severity | Task(s) |
|
||||
|---|---|---|
|
||||
| Bundle import drops template inheritance edges | Critical | 1, 2 |
|
||||
| Imported scripts lose `MinTimeBetweenRuns`/`ExecutionTimeoutSeconds`; `LockedInDerived` not in DTOs | High | 3, 4 |
|
||||
| Template `NativeAlarmSources` not transported (overrides dangle) | High | 5 |
|
||||
| Flattener drops grandchild members on repeated composition; CollisionDetector blind | High | 9, 10 |
|
||||
| Revision hash and diff omit `NativeAlarmSources` | High | 11, 12 |
|
||||
| Import leaves stale compiled Inbound API handlers | High | 14 (contract + Transport publisher; consumer → **plan 06**) |
|
||||
| Locked native alarm sources overridable at instance level | High | 13 |
|
||||
| Post-success audit failure flips Success→Failed | Medium | 17 |
|
||||
| Deleting `NotDeployed` instance requires live site round-trip | Medium | 18 |
|
||||
| Import blocker heuristic hard-blocks on false positives | Medium | 19 |
|
||||
| `OperationLockManager` single-node in-memory | Medium | 25 (documented invariant); active-node gating **deferred to plans 01/07** (cluster routing ownership) |
|
||||
| Every validation run compiles every script (CPU + assembly leak) | High (perf) | 15, 16 |
|
||||
| `LineDiffer` O((N+M)²) memory, no input cap | Medium (perf) | 22 |
|
||||
| Templates with folder/parent/composition never diff `Identical` | Medium (perf) | 23 |
|
||||
| `LoadAsync` O(bundle) + unbounded session count | Low (perf) | 24 (session cap); manifest-peek `ReadManifestAsync` **Deferred** — already doc-acknowledged (`Component-Transport.md:90`), pure optimisation |
|
||||
| Import apply is one long EF transaction | Low (perf) | **Deferred** — Task 16 removes the dominant cost (Roslyn in `StaleInstanceProbe`); restructuring the single-transaction rollback contract is high-risk for low residual gain |
|
||||
| Deny-list gaps: `Environment.Exit`/`FailFast`/`GetEnvironmentVariable` | Medium (sec) | 21 |
|
||||
| `System.Data` provider namespaces unbounded network hole | Medium (sec) | 21 |
|
||||
| Reflection-gateway member list incomplete | Medium (sec) | 21 |
|
||||
| Transport import bypasses script trust gate | Medium (sec) | 20 |
|
||||
| Instance alarm overrides implement less than spec | Medium | 25 (doc aligned to implementation — adding `DescriptionOverride`/`OnTriggerScriptOverride` columns is a feature, not a fix; decision recorded) |
|
||||
| `ArtifactDiff` equality predicates drift from sync predicates | Medium | 6 |
|
||||
| Spec advertises unimplemented `scripts/` bundle dir; `MaxBundleEntryCount = 4` | Low | 25 |
|
||||
| `TryCompile` surfaces only first violation/error | Low | 24 |
|
||||
| RevisionHash determinism contract fragile | Low | 11 (guard test extended; migration note) |
|
||||
| Underdeveloped 1: template fidelity shipped behind site/instance wave | — | 1–8 |
|
||||
| Underdeveloped 2: no derived-template import test | — | 1, 8 |
|
||||
| Underdeveloped 3: no repeated-composition flattening test | — | 9, 10 |
|
||||
| Underdeveloped 4: deferred-but-load-bearing (rename call-site rewriting, Transport-012 UI, `ReadManifestAsync`) | — | 25 documents the rename limitation; Transport-012 UI + `ReadManifestAsync` **Deferred** (acknowledged backlog, not correctness) |
|
||||
| Underdeveloped 5: preview→apply window has no optimistic concurrency | — | **Deferred** — version fields are documented as reserved (`ArtifactDiff.cs:33-37`); 30-min TTL window accepted, recorded in Task 25 doc sweep |
|
||||
| Underdeveloped 6: three hand-maintained compile-surface mirrors | — | **Deferred** — reflection/representative-script guard tests already exist; source-generator is an improvement, not a defect |
|
||||
| Underdeveloped 7: `SemanticValidator` leaf-name fallback silently accepts wrong-child calls | — | 24 (emit warning) |
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Wire template inheritance edges in the import apply path
|
||||
|
||||
**Classification:** high-risk (Critical data-loss fix on the import data contract)
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 9, 10, 11, 12, 13, 15, 16, 17, 18, 21, 22
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs` (add `ResolveInheritanceEdgesAsync`; call it from `ApplyAsync` after line 1044 `ResolveCompositionEdgesAsync`; extend Overwrite branch ~1466-1484)
|
||||
- Test (new): `tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/Import/InheritanceImportTests.cs`
|
||||
|
||||
1. Write failing test (follow the fixture pattern of `Import/SiteInstanceImportTests.cs` — source context → `BundleExporter.ExportAsync` → target context → `LoadAsync`/`PreviewAsync`/`ApplyAsync`):
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task Import_DerivedTemplate_WiresParentTemplateId()
|
||||
{
|
||||
// Source: base "Pump" + derived "Pump.WaterPump" (ParentTemplateId -> Pump)
|
||||
// with one inherited-from-base attribute on Pump.
|
||||
var bundle = await ExportTemplatesFromSourceAsync("Pump", "Pump.WaterPump");
|
||||
await ImportIntoEmptyTargetAsync(bundle, ResolutionAction.Add);
|
||||
|
||||
var target = await TargetTemplateRepo.GetAllTemplatesAsync();
|
||||
var basePump = target.Single(t => t.Name == "Pump");
|
||||
var derived = target.Single(t => t.Name == "Pump.WaterPump");
|
||||
Assert.Equal(basePump.Id, derived.ParentTemplateId); // FAILS today: null
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Import_Overwrite_UpdatesParentEdge_AndRenamedBaseIsHonoured()
|
||||
{
|
||||
// Bundle base renamed to "Pump2" via ResolutionAction.Rename; derived's
|
||||
// BaseTemplateName ("Pump") must resolve through the rename map to Pump2.
|
||||
...
|
||||
Assert.Equal(pump2.Id, derived.ParentTemplateId);
|
||||
}
|
||||
```
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests --filter FullyQualifiedName~InheritanceImportTests` → expect **FAIL** (ParentTemplateId null).
|
||||
3. Implement `ResolveInheritanceEdgesAsync(IReadOnlyList<TemplateDto> dtos, Dictionary<(string,string), ImportResolution> resolutionMap, string user, CancellationToken ct)` mirroring `ResolveCompositionEdgesAsync` (line 1993):
|
||||
- For each non-Skip template DTO: compute the template's *persisted* name (Rename → `RenameTo`), load the tracked entity by name.
|
||||
- If `dto.BaseTemplateName is null` → set `ParentTemplateId = null` (an Overwrite from a root-template bundle must clear a stale edge).
|
||||
- Else resolve the base name through the resolution map (`ResolveOrDefault(resolutionMap, "Template", dto.BaseTemplateName)`; Rename → `RenameTo`), then look up by name across staged-imported + pre-existing target templates (same lookup set `ResolveCompositionEdgesAsync` builds). Set `ParentTemplateId`.
|
||||
- Unresolvable base → leave null and emit a `BundleImportBaseTemplateUnresolved` audit row (mirror `BundleImportAlarmScriptUnresolved`), never throw.
|
||||
- Call from `ApplyAsync` immediately after `ResolveCompositionEdgesAsync` (line 1044).
|
||||
4. Run test → expect **PASS**. Run the whole Transport suites: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.Tests && dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests`.
|
||||
5. Update `docs/requirements/Component-Transport.md` apply-pass list (the second-pass rewire section) to name the inheritance pass alongside alarm-script and composition rewires.
|
||||
6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Transport tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests docs/requirements/Component-Transport.md && git commit -m "fix(transport): wire template inheritance edges on bundle import (C3) — derived templates no longer land as roots"`
|
||||
|
||||
### Task 2: Import-time acyclicity check over the merged template graph
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 9, 10, 11, 12, 13, 15, 16, 17, 18, 21, 22
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs` (after the Task-1 inheritance pass in `ApplyAsync`)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/Import/InheritanceImportTests.cs` (extend)
|
||||
|
||||
1. Write failing test: craft a bundle whose templates form an inheritance cycle against a pre-existing target template (target `A` has parent `B`; bundle overwrites `B` with `BaseTemplateName: "A"`). Assert `ApplyAsync` throws `SemanticValidationException` and the transaction rolled back (target `B.ParentTemplateId` unchanged). Add the composition-cycle twin (bundle composition edge closing a loop through a target template).
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests --filter FullyQualifiedName~InheritanceImportTests` → **FAIL** (cycle persists today; importer performs no acyclicity check).
|
||||
3. Implement: after both rewire passes, load all templates + compositions from the tracked context and run TemplateEngine `CycleDetector.DetectInheritanceCycle` / `DetectCompositionCycle` (Transport already references TemplateEngine). On a detected cycle, throw `SemanticValidationException(new[] {"Import would create a template inheritance/composition cycle: <path>"})` — the existing catch path rolls back and reports.
|
||||
4. Run test → **PASS**. Commit: `git commit -m "fix(transport): reject bundle imports that would persist an inheritance/composition cycle"` (with the touched paths in `git add`).
|
||||
|
||||
### Task 3: Carry `MinTimeBetweenRuns` / `ExecutionTimeoutSeconds` through import
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~3 min
|
||||
**Parallelizable with:** 9–13, 15–18, 21, 22
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs` (`BuildTemplate` ~1550-1560; `SyncTemplateScriptsAsync` `changed` predicate + copies ~1836-1851 and the add branch ~1870-1877)
|
||||
- Test (new): `tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/Import/TemplateScriptFidelityTests.cs`
|
||||
|
||||
1. Failing test: export a template whose script has `MinTimeBetweenRuns = TimeSpan.FromSeconds(30)` and `ExecutionTimeoutSeconds = 42`; import as Add into empty target, assert both fields on the persisted `TemplateScript`. Second test: Overwrite a target whose script differs *only* in `ExecutionTimeoutSeconds` → assert field updated and a `TemplateScriptUpdated` audit row emitted.
|
||||
2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests --filter FullyQualifiedName~TemplateScriptFidelityTests` → **FAIL**.
|
||||
3. Implement: add `MinTimeBetweenRuns = s.MinTimeBetweenRuns, ExecutionTimeoutSeconds = s.ExecutionTimeoutSeconds` to the `TemplateScript` initializers in `BuildTemplate` and the `SyncTemplateScriptsAsync` add branch; add `|| current.MinTimeBetweenRuns != scriptDto.MinTimeBetweenRuns || current.ExecutionTimeoutSeconds != scriptDto.ExecutionTimeoutSeconds` to the `changed` predicate and copy both fields in the update block.
|
||||
4. Run → **PASS**. Commit: `git commit -m "fix(transport): carry script cadence/timeout fields through bundle import (was silently reset to defaults)"`
|
||||
|
||||
### Task 4: Add `LockedInDerived` to template child DTOs and apply paths
|
||||
|
||||
**Classification:** high-risk (bundle data contract change — must stay additive)
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 9–13, 15–18, 21, 22
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/EntityDtos.cs` (`TemplateAttributeDto`, `TemplateAlarmDto`, `TemplateScriptDto` — trailing `bool LockedInDerived = false`)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/EntitySerializer.cs` (export ~55-90; `FromBundleContent` ~330-360)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs` (`BuildTemplate`, three `SyncTemplate*Async` predicates/copies)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.Tests/Serialization/EntitySerializerTests.cs` (extend), `tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/Import/TemplateScriptFidelityTests.cs` (extend)
|
||||
|
||||
1. Failing unit test in `EntitySerializerTests`: entity with `LockedInDerived = true` on an attribute, alarm, and script → `ToBundleContent` DTOs carry it; `FromBundleContent` round-trips it. Failing integration test: import as Add → persisted flags true; old-form JSON (no field) still deserializes with `false` (backward-compat assertion — deserialize a hand-written DTO JSON blob missing the property).
|
||||
2. Run both test projects with `--filter LockedInDerived` → **FAIL** (compile errors first — that counts; fix signatures then re-run for behavioural red).
|
||||
3. Implement: trailing optional parameter on each record (mirrors the `ExecutionTimeoutSeconds = null` precedent — additive, no `schemaVersion` bump needed); populate at export; consume in `BuildTemplate`, `FromBundleContent`, and all three sync helpers (`changed` predicate + copy).
|
||||
4. Run → **PASS**. Commit: `git commit -m "fix(transport): transport LockedInDerived on template attributes/alarms/scripts (additive DTO fields)"`
|
||||
|
||||
### Task 5: Transport template `NativeAlarmSources` end-to-end
|
||||
|
||||
**Classification:** high-risk (new bundle payload class + new sync pass)
|
||||
**Estimated implement time:** ~5 min (DTO+export+apply) — if it runs long, split diff wiring into its own commit
|
||||
**Parallelizable with:** 9–13, 15–18, 21, 22
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/EntityDtos.cs` (new `TemplateNativeAlarmSourceDto`; `TemplateDto` gains trailing `IReadOnlyList<TemplateNativeAlarmSourceDto> NativeAlarmSources` defaulting empty)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/EntitySerializer.cs` (export + `FromBundleContent`)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs` (`BuildTemplate`; new `SyncTemplateNativeAlarmSourcesAsync` called from the Overwrite branch at ~1482)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/ArtifactDiff.cs` (`CompareTemplate` — `DiffChildren` over native sources)
|
||||
- Test (new): `tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/Import/NativeAlarmSourceImportTests.cs`; extend `EntitySerializerTests`
|
||||
|
||||
1. Failing tests:
|
||||
```csharp
|
||||
[Fact] public async Task Import_Add_CarriesTemplateNativeAlarmSources()
|
||||
// template with one source (all fields set incl. IsLocked, IsInherited, LockedInDerived)
|
||||
// → export → import Add → persisted TemplateNativeAlarmSource field-equal.
|
||||
|
||||
[Fact] public async Task Import_Overwrite_SyncsNativeAlarmSources()
|
||||
// target source differs in ConditionFilter; bundle also drops one source and adds one
|
||||
// → assert update/add/delete applied + audit rows (TemplateNativeAlarmSource{Added,Updated,Deleted}).
|
||||
|
||||
[Fact] public async Task Preview_DiffsNativeAlarmSources()
|
||||
// changed source shows a "NativeAlarmSources.<name>" FieldChange; identical → no change row.
|
||||
|
||||
[Fact] public async Task Import_InstanceOverride_NoLongerDangles()
|
||||
// export template-with-source + instance override of it → import → flatten target instance
|
||||
// → resolved config contains the source with the override applied (today: silently dropped).
|
||||
```
|
||||
2. Run `--filter NativeAlarmSourceImportTests` → **FAIL**.
|
||||
3. Implement:
|
||||
- DTO: `record TemplateNativeAlarmSourceDto(string Name, string? Description, string ConnectionName, string SourceReference, string? ConditionFilter, bool IsLocked, bool IsInherited, bool LockedInDerived)`.
|
||||
- Export in `EntitySerializer.ToBundleContent` (include `IsInherited` placeholder rows — the collision detector depends on them) and consume in `FromBundleContent`.
|
||||
- `BuildTemplate`: copy the collection. New `SyncTemplateNativeAlarmSourcesAsync` mirrors `SyncTemplateAttributesAsync` (name-keyed add/update/delete; repository has `DeleteTemplateNativeAlarmSourceAsync`-style members — verify exact name on `ITemplateEngineRepository` and add if missing).
|
||||
- `ArtifactDiff.CompareTemplate`: `DiffChildren(existing.NativeAlarmSources, incoming.NativeAlarmSources, e => e.Name, i => i.Name, NativeAlarmSourcesEqual, "NativeAlarmSources", changes)`.
|
||||
4. Run → **PASS**; run full Transport suites. Update `docs/requirements/Component-Transport.md` transported-entity table + `CLAUDE.md` component #24 blurb (template native alarm sources now travel).
|
||||
5. Commit: `git commit -m "fix(transport): transport template NativeAlarmSources — imports no longer amputate native alarm mirrors nor dangle instance overrides"`
|
||||
|
||||
### Task 6: Single-source per-entity field comparers for diff + sync
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 9–13, 15–18, 21, 22
|
||||
**Files:**
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/TemplateChildEquality.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/ArtifactDiff.cs` (replace `AttributesEqual`/`AlarmsEqual`/`ScriptsEqual` at 646-666; `CompareTemplate` passes a script-name resolver for the alarm on-trigger ref)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs` (three sync `changed` predicates delegate to the same class)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.Tests/Import/ArtifactDiffTests.cs` (extend)
|
||||
|
||||
1. Failing tests (today's drift, both directions):
|
||||
```csharp
|
||||
[Fact] public void CompareTemplate_AttributeElementDataTypeChange_IsModified() // omitted today → false Identical
|
||||
[Fact] public void CompareTemplate_ScriptExecutionTimeoutChange_IsModified() // omitted today
|
||||
[Fact] public void CompareTemplate_AlarmOnTriggerScriptChange_IsModified() // omitted today
|
||||
[Fact] public void CompareTemplate_LockedInDerivedChange_IsModified()
|
||||
```
|
||||
2. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.Tests --filter ArtifactDiffTests` → **FAIL**.
|
||||
3. Implement `TemplateChildEquality`: one static method per entity comparing the **complete** writable field list (attributes: Value/DataType/ElementDataType/IsLocked/LockedInDerived/Description/DataSourceReference; alarms: + `OnTriggerScriptName` via a `Func<int?, string?> scriptNameById` resolver built from `existing.Scripts`; scripts: + MinTimeBetweenRuns/ExecutionTimeoutSeconds/LockedInDerived; native sources per Task 5). XML-doc each with: *"This is the single source of truth for '<entity> changed'; ArtifactDiff and SyncTemplate\*Async must both call it."* Point `ArtifactDiff` and the three (four after Task 5) sync predicates at it.
|
||||
4. Run Transport unit + integration suites → **PASS**.
|
||||
5. Commit: `git commit -m "refactor(transport): single-source template child equality — diff and overwrite-sync can no longer drift"`
|
||||
|
||||
### Task 7: Export real `AreaName` (finish the half-shipped Area transport)
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 1–4, 9–13, 15–18, 21, 22
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Export/DependencyResolver.cs` / `ResolvedExport.cs` (aggregate gains `areaNameById` for exported instances; exporter has `ScadaBridgeDbContext` — load `Areas` for the instances' `AreaId`s)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/EntitySerializer.cs` (~247-256: replace the `AreaName: null` TODO with the lookup)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/Import/SiteInstanceImportTests.cs` (extend) or new `AreaTransportTests.cs`
|
||||
|
||||
1. Failing test: source instance assigned to Area "Line-1" under its site → export → import into target with `--create-missing` site mapping → assert target instance's `AreaId` resolves to an Area named "Line-1" under the target site (the importer's `GetOrCreate` machinery at `BundleImporter.cs:3168` is already built and tested — only the exporter feeds it null).
|
||||
2. Run → **FAIL** (imported `AreaId` is null).
|
||||
3. Implement: extend the export aggregate with the instances' area id→name map (single query over `Areas` filtered to referenced ids) and emit `AreaName: areaNameById.TryGetValue(inst.AreaId ?? -1, out var an) ? an : null`. Delete the TODO comment.
|
||||
4. Run → **PASS**. Update `docs/requirements/Component-Transport.md` (Area-by-name now actually travels; remove any "importer leaves AreaId null" caveat). CLAUDE.md #24 already claims this — now true, no edit needed (verify wording).
|
||||
5. Commit: `git commit -m "fix(transport): export instance AreaName — Area-by-name reconciliation was half-shipped (exporter hardcoded null)"`
|
||||
|
||||
### Task 8: Round-trip export→import equivalence test suite (Theme-3 structural guard)
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 9–18, 21, 22 (test-only; depends on Tasks 3–5, 7 landing first)
|
||||
**Files:**
|
||||
- Create: `tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/RoundTripEquivalenceTests.cs`
|
||||
|
||||
1. For **every** transported entity type (Template + attribute/alarm/script/native-source/composition children, SharedScript, ExternalSystem + methods, DatabaseConnection, NotificationList, SmtpConfig, SmsConfig, ApiMethod, TemplateFolder, Site, DataConnection, Instance + override children + bindings), seed a source DB with a *maximally populated* entity (every writable property non-default), export, import into a fresh target, reload, and compare **by reflection**: every public writable property of the entity must be equal, except an explicit per-type exclusion list (`Id`, FK id columns remapped by design, timestamps). The reflection sweep is the point — a future entity property that no DTO carries fails the test instead of silently vanishing (exactly how `LockedInDerived`, cadence/timeout, and native sources were lost).
|
||||
```csharp
|
||||
private static void AssertRoundTripEqual<T>(T source, T imported, params string[] excluded)
|
||||
{
|
||||
foreach (var p in typeof(T).GetProperties().Where(p => p.CanWrite
|
||||
&& !excluded.Contains(p.Name)
|
||||
&& p.PropertyType.IsValueType || p.PropertyType == typeof(string)))
|
||||
Assert.True(Equals(p.GetValue(source), p.GetValue(imported)),
|
||||
$"{typeof(T).Name}.{p.Name} did not survive export→import");
|
||||
}
|
||||
```
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests --filter RoundTripEquivalenceTests` — expect **PASS** if Tasks 1–7 are complete (any failure here is a residual fidelity hole: fix it in the importer/exporter, not the test).
|
||||
3. Commit: `git commit -m "test(transport): per-entity reflection round-trip equivalence suite — structural guard against silent import data loss"`
|
||||
|
||||
### Task 9: Fix repeated-composition member loss in `FlatteningService`
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1–8, 15–18, 21, 22
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Flattening/FlatteningService.cs` (the four `ResolveComposed*Recursive` methods: attributes ~285-297, alarms ~648-660, native sources ~761-772, scripts ~913-924)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Flattening/FlatteningServiceTests.cs` (extend)
|
||||
|
||||
1. Failing test:
|
||||
```csharp
|
||||
[Fact]
|
||||
public void Flatten_SameTemplateComposedTwice_ResolvesNestedMembersUnderBothSlots()
|
||||
{
|
||||
// X composes Y as "y1" and "y2"; Y composes Z as "z"; Z has attribute "Val",
|
||||
// alarm "Alm", script "Run", native source "Src".
|
||||
var result = Flatten(x);
|
||||
Assert.Contains(result.Attributes, a => a.CanonicalName == "y1.z.Val");
|
||||
Assert.Contains(result.Attributes, a => a.CanonicalName == "y2.z.Val"); // FAILS today
|
||||
// + same assertions for Alarms / Scripts / NativeAlarmSources
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Flatten_CompositionCycle_StillTerminates() // regression guard for the cycle role of `visited`
|
||||
```
|
||||
2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests --filter Flatten_SameTemplateComposedTwice` → **FAIL** (`y2.z.Val` missing).
|
||||
3. Implement: convert the shared `visited` set into a **recursion-path** set in all four methods. Pattern (identical in each):
|
||||
```csharp
|
||||
// Guard on the recursion PATH, not globally: composition is by-slot, so a template
|
||||
// legitimately appears twice under different prefixes; only a template already on
|
||||
// the CURRENT path is a cycle.
|
||||
var pushed = new List<int>();
|
||||
foreach (var composedTemplate in composedChain)
|
||||
if (path.Add(composedTemplate.Id)) pushed.Add(composedTemplate.Id);
|
||||
try
|
||||
{
|
||||
foreach (var composedTemplate in composedChain)
|
||||
{
|
||||
if (!compositionMap.TryGetValue(composedTemplate.Id, out var nested)) continue;
|
||||
foreach (var n in nested)
|
||||
if (!path.Contains(n.ComposedTemplateId)) // cycle guard
|
||||
ResolveComposed…Recursive(n, $"{prefix}.{n.InstanceName}", …, path);
|
||||
}
|
||||
}
|
||||
finally { foreach (var id in pushed) path.Remove(id); }
|
||||
```
|
||||
(Rename the parameter `visited` → `path` so the semantics are explicit.)
|
||||
4. Run the full TemplateEngine test project → **PASS**, no regressions.
|
||||
5. Commit: `git commit -m "fix(template-engine): repeated composition no longer drops nested members — cycle guard keyed on recursion path, not global template id"`
|
||||
|
||||
### Task 10: Fix `CollisionDetector` blindness under repeated composition
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~3 min
|
||||
**Parallelizable with:** 1–8, 15–18, 21, 22 (different file from Task 9)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/CollisionDetector.cs` (`CollectComposedMembers` ~119-144)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/CollisionDetectorTests.cs` (extend)
|
||||
|
||||
1. Failing test: X composes Y twice (`y1`, `y2`); Y composes Z; X also declares a direct attribute named `y2.z.Val` (path-collision). Assert the detector reports the collision (today the second slot's subtree is never collected, so it cannot).
|
||||
2. Run `--filter CollisionDetectorTests` → **FAIL**.
|
||||
3. Implement the same path-set conversion as Task 9: `visited` becomes the current recursion path — add `template.Id` on entry, remove in `finally`; the early-return then only fires on genuine cycles.
|
||||
4. Run → **PASS**. Commit: `git commit -m "fix(template-engine): collision detection sees all slots of a repeatedly-composed template"`
|
||||
|
||||
### Task 11: Add `NativeAlarmSources` to `RevisionHashService` (planned hash migration)
|
||||
|
||||
**Classification:** high-risk (revision-hash change ⇒ staleness flags flip for native-source-bearing instances)
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 1–8, 15–18, 21, 22 — **after Task 13** (hash includes `IsLocked`)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Flattening/RevisionHashService.cs` (new `HashableNativeAlarmSource`; `HashableConfiguration` gains the property in alphabetical position)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Flattening/RevisionHashServiceTests.cs` (extend, incl. the alphabetical-guard test)
|
||||
|
||||
1. Failing tests:
|
||||
```csharp
|
||||
[Fact] public void ComputeHash_NativeAlarmSourceChange_ChangesHash() // FAILS today
|
||||
[Fact] public void ComputeHash_NoNativeAlarmSources_HashUnchangedFromBaseline()
|
||||
// pin a literal known-good hash for a native-source-free config so the migration
|
||||
// is surgical: only instances that HAVE native sources flip stale.
|
||||
```
|
||||
2. Run `--filter RevisionHashServiceTests` → **FAIL**.
|
||||
3. Implement: `HashableNativeAlarmSource` with alphabetical properties (`CanonicalName`, `ConditionFilter`, `ConnectionName`, `IsLocked`, `SourceReference`); on `HashableConfiguration` add `public List<HashableNativeAlarmSource>? NativeAlarmSources { get; init; }` in alphabetical slot (between `InstanceUniqueName` and `Scripts`), populated **null-when-empty** so `WhenWritingNull` keeps every native-source-free config's hash byte-identical (no global staleness storm — the migration only flags the instances the fix is about). Extend the `HashableRecords_PropertiesDeclaredAlphabetically` guard to the new record. Add a migration note to the class doc-comment and `docs/requirements/Component-TemplateEngine.md` (revision-hash section): *native-source-bearing instances will report stale once after upgrade; redeploy clears it — deliberate.*
|
||||
4. Run → **PASS**. Commit: `git commit -m "fix(template-engine): native alarm sources participate in the revision hash — staleness detection restored (deliberate one-time stale flip for affected instances)"`
|
||||
|
||||
### Task 12: Add `NativeAlarmSources` to `DiffService`
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 1–8, 15–18, 21, 22
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Flattening/DiffService.cs` (add added/removed/changed sweep — file currently has zero `NativeAlarm` references)
|
||||
- Modify: the `ConfigurationDiff` type in Commons (`src/ZB.MOM.WW.ScadaBridge.Commons/Types/Flattening/` — additive section)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Flattening/DiffServiceTests.cs` (extend)
|
||||
|
||||
1. Failing test: two flattened configs differing only in one native source's `SourceReference` → diff reports one changed native source; add/remove cases too.
|
||||
2. Run `--filter DiffServiceTests` → **FAIL**.
|
||||
3. Implement following the exact pattern DiffService uses for Alarms (keyed on `CanonicalName`, field-compare `ConnectionName`/`SourceReference`/`ConditionFilter`/`IsLocked`). Additive `NativeAlarmSources` collection on `ConfigurationDiff` (message-contract rule: additive only).
|
||||
4. Run → **PASS**; also run `tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests` (`DeploymentComparisonTests` consume the diff). Commit: `git commit -m "fix(template-engine): DiffService covers native alarm sources — Deployments diff view no longer blind to native-alarm edits"`
|
||||
|
||||
### Task 13: Enforce locks on native alarm source overrides
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1–8, 15–18, 21, 22
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Types/Flattening/FlattenedConfiguration.cs` (`ResolvedNativeAlarmSource` gains `public bool IsLocked { get; init; }` — additive)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Flattening/FlatteningService.cs` (`ResolveInheritedNativeAlarmSources` ~670-710 sets `IsLocked`; `ApplyInstanceNativeAlarmSourceOverrides` ~775-792 adds the lock check)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (`HandleSetInstanceNativeAlarmSourceOverride` ~882-909)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Flattening/FlatteningServiceTests.cs`, ManagementService test project (mirror an existing handler test)
|
||||
|
||||
1. Failing tests:
|
||||
```csharp
|
||||
[Fact] public void Flatten_LockedNativeAlarmSource_IgnoresInstanceOverride()
|
||||
// template source IsLocked=true + instance override repointing ConnectionName
|
||||
// → resolved source keeps template values, Source stays "Template". FAILS today.
|
||||
|
||||
[Fact] public async Task SetNativeAlarmSourceOverride_OnLockedSource_Throws()
|
||||
// ManagementActor handler rejects with ManagementCommandException. FAILS today.
|
||||
```
|
||||
2. Run TemplateEngine + ManagementService test filters → **FAIL**.
|
||||
3. Implement:
|
||||
- `ResolveInheritedNativeAlarmSources`: set `IsLocked = binding.IsLocked || lockedNames.Contains(binding.Name)` on the resolved record (keep `lockedNames` derived-shadow logic).
|
||||
- `ApplyInstanceNativeAlarmSourceOverrides`: after the `TryGetValue`, `if (existing.IsLocked) continue;` — exactly mirroring the attribute (`:309`) and alarm (`:337`) paths.
|
||||
- `HandleSetInstanceNativeAlarmSourceOverride`: resolve `IFlatteningPipeline` from `sp` (ManagementService already references DeploymentManager), flatten the instance, find the source by `cmd.SourceCanonicalName`; throw `ManagementCommandException($"Native alarm source '{…}' is locked at the template level and cannot be overridden.")` when locked, and also reject when the canonical name doesn't resolve at all (prevents creating dangling overrides — matching the flattener's cannot-add rule).
|
||||
4. Run → **PASS**. Update `docs/requirements/Component-TemplateEngine.md` §Native-alarm override rules if wording needs the enforcement point named. Commit: `git commit -m "fix(security): enforce template locks on native alarm source overrides at flatten and management-command level"`
|
||||
|
||||
### Task 14: Script-artifact invalidation contract — Commons seam + Transport publisher (plan-06 handoff)
|
||||
|
||||
**Classification:** high-risk (cross-component contract)
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 9–13, 15–18, 21, 22
|
||||
**Files:**
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Scripting/ScriptArtifactsChanged.cs`
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/IScriptArtifactChangeBus.cs`
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.Host/Services/InProcessScriptArtifactChangeBus.cs` (+ singleton registration in the Host's central-role service wiring)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs` (optional ctor dep `IScriptArtifactChangeBus?`; publish after `tx.CommitAsync` at ~1098)
|
||||
- Create: `docs/plans/2026-07-08-script-artifact-invalidation-contract.md`
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/Import/BundleImporterApplyTests.cs` (extend), new Host unit test for the bus
|
||||
|
||||
1. Failing test: apply a bundle that Overwrites an `ApiMethod`, a `SharedScript`, and a `Template`; a recording fake `IScriptArtifactChangeBus` handed to the importer must receive, **after** commit, one `ScriptArtifactsChanged` per kind with the post-resolution (renamed) names. Second test: a failed apply (semantic validation throw) publishes **nothing**.
|
||||
2. Run → **FAIL** (types don't exist — compile-red, then behavioural red).
|
||||
3. Implement the contract:
|
||||
```csharp
|
||||
// Commons/Messages/Scripting/ScriptArtifactsChanged.cs
|
||||
/// <summary>Advisory, at-least-once, published only AFTER the mutating transaction
|
||||
/// commits. Consumers must invalidate any compiled/cached artifact keyed by these
|
||||
/// names, and must ALSO self-heal without the notification (content-hash checks) —
|
||||
/// this bus is in-process per node; cross-node propagation is the consumer's
|
||||
/// responsibility (see docs/plans/2026-07-08-script-artifact-invalidation-contract.md).</summary>
|
||||
public sealed record ScriptArtifactsChanged(
|
||||
string ArtifactKind, // ScriptArtifactKinds.ApiMethod | SharedScript | Template
|
||||
IReadOnlyList<string> Names, // post-resolution (renamed) names
|
||||
string Source, // "BundleImport" | "Management" | "FailoverActivation"
|
||||
DateTimeOffset OccurredAtUtc);
|
||||
public static class ScriptArtifactKinds { public const string ApiMethod = "ApiMethod"; /* … */ }
|
||||
|
||||
// Commons/Interfaces/IScriptArtifactChangeBus.cs
|
||||
public interface IScriptArtifactChangeBus
|
||||
{
|
||||
void Publish(ScriptArtifactsChanged notification);
|
||||
IDisposable Subscribe(Action<ScriptArtifactsChanged> handler);
|
||||
}
|
||||
```
|
||||
`InProcessScriptArtifactChangeBus`: lock-free copy-on-write handler list; `Publish` swallows-and-logs handler exceptions (a bad consumer must never fail an import). `BundleImporter` collects overwritten/renamed names during apply and publishes once per kind **after** `tx.CommitAsync` (never inside the transaction — a notification for a rolled-back write is worse than a missed one).
|
||||
4. Write the contract doc: semantics above + the **explicit handoff table**: *plan 06 implements (a) the `InboundScriptExecutor` subscription that evicts `_scriptHandlers` and `_knownBadMethods` by name, (b) publishing from the ManagementActor `UpdateApiMethod`/shared-script paths (closes the project-memory gotchas), (c) cross-node/failover freshness (revision-keyed handler cache).* Plan 05 ships the seam + the Transport publisher only.
|
||||
5. Run → **PASS**; `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Commit: `git commit -m "feat(commons/transport): ScriptArtifactsChanged invalidation contract + post-commit publish from bundle import (consumer lands in plan 06)"`
|
||||
|
||||
### Task 15: Cache script-compile verdicts by code hash
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 1–14, 17, 18, 21, 22
|
||||
**Files:**
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ScriptCompileVerdictCache.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ScriptCompiler.cs` (`TryCompile` consults the cache)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Validation/ScriptCompilerTests.cs` (extend)
|
||||
|
||||
1. Failing test:
|
||||
```csharp
|
||||
[Fact]
|
||||
public void TryCompile_SameCodeTwice_SecondCallIsCacheHit()
|
||||
{
|
||||
ScriptCompileVerdictCache.Clear();
|
||||
var c = new ScriptCompiler();
|
||||
c.TryCompile("return 1;", "A");
|
||||
var hitsBefore = ScriptCompileVerdictCache.Hits;
|
||||
var r = c.TryCompile("return 1;", "B"); // same code, different name
|
||||
Assert.True(r.IsSuccess);
|
||||
Assert.Equal(hitsBefore + 1, ScriptCompileVerdictCache.Hits);
|
||||
}
|
||||
```
|
||||
2. Run `--filter ScriptCompilerTests` → **FAIL**.
|
||||
3. Implement: static `ConcurrentDictionary<string,(bool ok, string? error)>` keyed `Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(code)))` — the verdict is a pure function of code + policy, so a process-wide cache is sound; the policy is compile-time static. Bound at 4096 entries (on overflow, `Clear()` — simple and safe). Expose `Hits`/`Count`/`Clear()` for tests and diagnostics. `TryCompile` keys the cache on code only; the script *name* is formatted into the returned message at read time (store the name-free error, format on return). This removes both the repeat-CPU and the repeat assembly load: an unchanged script compiles exactly once per process lifetime.
|
||||
4. Run → **PASS**; run the full TemplateEngine test project. Commit: `git commit -m "perf(template-engine): cache script compile verdicts by code hash — unchanged scripts compile once per process"`
|
||||
|
||||
### Task 16: Skip script compilation on read-only staleness/comparison paths
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1–15, 21, 22
|
||||
**Files:**
|
||||
- Modify: `IFlatteningPipeline` + `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/FlatteningPipeline.cs` (`FlattenAndValidateAsync(int instanceId, bool validateScripts = true, CancellationToken ct = default)` — additive default param)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentService.cs` (`GetDeploymentComparisonAsync` ~652 passes `validateScripts: false`)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/StaleInstanceProbe.cs` (~34 passes `validateScripts: false`)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests/DeploymentComparisonTests.cs`, `DeploymentServiceTests.cs` (extend)
|
||||
|
||||
1. Failing test: with a template script that does **not** compile, `GetDeploymentComparisonAsync` must still return a comparison (staleness needs the hash, not a compile) — today `FlattenAndValidateAsync` fails and the comparison errors out; and a mock-pipeline assertion that the comparison/probe paths request `validateScripts: false` while `DeployInstanceAsync` keeps `true`.
|
||||
2. Run `--filter DeploymentComparisonTests` → **FAIL**.
|
||||
3. Implement: `FlatteningPipeline` skips `ValidationService.ValidateScriptCompilation` (only that validator — semantic/structural validation stays) when `validateScripts` is false. Deploy path unchanged (`true` default). This removes 100% of Roslyn work (and the non-collectible `InteractiveAssemblyLoader` loads) from the Deployments page sweep and from `BundleImporter.ComputeStaleInstanceIdsAsync`'s per-instance probing.
|
||||
4. Run DeploymentManager + Transport integration suites → **PASS**. Update `docs/requirements/Component-DeploymentManager.md` (comparison path is hash-only by design; deploy gate remains authoritative).
|
||||
5. Commit: `git commit -m "perf(deployment): staleness/comparison paths no longer Roslyn-compile every script — deploy gate remains the authoritative compile"`
|
||||
|
||||
### Task 17: Isolate the post-success deployment audit write
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~3 min
|
||||
**Parallelizable with:** 1–16, 21, 22 (same file as 16/18 — coordinate order, regions don't overlap)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentService.cs` (~330-332)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests/DeploymentServiceTests.cs` (extend)
|
||||
|
||||
1. Failing test: audit-service mock throws on the `"Deploy"` `LogAsync`; site responds Success → assert result is Success and the persisted record's `Status` **stays** `Success` (today the catch at :343 flips it to `Failed` — the exact divergence the surrounding comments prevent, and a violation of "audit-write failure NEVER aborts the user-facing action").
|
||||
2. Run → **FAIL**.
|
||||
3. Implement: wrap the `LogAsync` at :330-332 in its own try/catch that only `_logger.LogError`s (mirror `TryLogLifecycleTimeoutAsync` at :1034). Same guard for any other post-terminal-status audit writes in the lifecycle verbs (audit-only; scan `DisableInstanceAsync`/`EnableInstanceAsync` for the same shape and guard if present).
|
||||
4. Run → **PASS**. Commit: `git commit -m "fix(deployment): audit-write fault can no longer flip a committed Success deployment to Failed"`
|
||||
|
||||
### Task 18: Delete `NotDeployed` instances without a site round-trip
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 1–16, 21, 22
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentService.cs` (`DeleteInstanceAsync` ~546-585)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests/DeploymentServiceTests.cs` (extend)
|
||||
|
||||
1. Failing test: instance in `NotDeployed`, communication-service mock configured to throw (site unreachable) → `DeleteInstanceAsync` must succeed and remove the record without ever calling the communication service (assert `Verify(..., Times.Never)`); Transport-imported instances (always `NotDeployed`) become deletable against uncommissioned sites.
|
||||
2. Run → **FAIL**.
|
||||
3. Implement: after the lock acquisition, `if (instance.State == InstanceState.NotDeployed)` → skip the site command entirely, delete the central record (reuse the existing guarded delete block), audit `"Delete"` with `new { CommandId = (string?)null, LocalOnly = true }`, return success. Matches `StateTransitionValidator.CanDelete`'s documented rationale (:40-49) — the two components currently disagree about the same design sentence.
|
||||
4. Run → **PASS**. Update the matching sentence in `docs/requirements/Component-DeploymentManager.md`. Commit: `git commit -m "fix(deployment): delete of a NotDeployed instance is central-side only — no site round-trip, unreachable sites can't block cleanup"`
|
||||
|
||||
### Task 19: Import blocker heuristic — local-declaration exclusion, denylist additions, warning severity for template scripts
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 9–18, 21, 22
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs` (`RunSemanticValidationAsync` ~3680-3692; `DetectBlockersAsync` ~751-818; `KnownNonReferenceNames` ~853-874; `CollectCallIdentifiers`)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/SemanticValidatorImportTests.cs` (extend)
|
||||
|
||||
1. Failing tests:
|
||||
```csharp
|
||||
[Fact] public async Task Apply_TemplateScriptWithLocalPascalCaseMethod_DoesNotHardBlock()
|
||||
// script: "decimal ComputeRate(decimal x) => x * 2; return ComputeRate(Attributes[\"A\"]);"
|
||||
// → ApplyAsync succeeds (today: SemanticValidationException on 'ComputeRate').
|
||||
|
||||
[Fact] public async Task Apply_TemplateScriptWithRegexAndStringBuilder_DoesNotHardBlock()
|
||||
|
||||
[Fact] public async Task Apply_ApiMethodWithUnresolvedReference_StillHardBlocks()
|
||||
// ApiMethod script calling a genuinely-missing ExternalSystem keeps throwing.
|
||||
```
|
||||
2. Run `--filter SemanticValidatorImportTests` → **FAIL**.
|
||||
3. Implement, three edges:
|
||||
- **Locally-declared names:** before the candidate loop, Roslyn-parse each scanned script (`CSharpSyntaxTree.ParseText(code, new CSharpParseOptions(kind: SourceCodeKind.Script))` — Transport gets it transitively; add the package ref if not), collect `LocalFunctionStatementSyntax`/`MethodDeclarationSyntax` identifier names, and exclude them from `referenced` for that script's contributions.
|
||||
- **Denylist additions:** add to `KnownNonReferenceNames`: `Regex, Match, Matches, IsMatch, Replace, Split, StringBuilder, Append, AppendLine, Parse, TryParse, Format, Join, Abs, Round, Min, Max, Floor, Ceiling, Pow, Sqrt, Json, Serialize, Deserialize, Where, Select, First, FirstOrDefault, Any, All, Count, Sum, Average, OrderBy, OrderByDescending, GroupBy, Distinct, Add, Remove, Clear, Contains, ContainsKey, TryGetValue, StartsWith, EndsWith, Substring, Trim, ToUpper, ToLower, HAVING, VALUES, DELETE, DISTINCT, LIMIT` (comment: *the list will still drift — that is why template-script findings are warnings, below*).
|
||||
- **Severity split:** track which candidate names came from *template* scripts vs *ApiMethod* scripts. Template-script findings become **warnings** — logged + appended to the `ImportResult`/preview blocker list with a non-blocking flag (the deploy gate re-validates authoritatively at deploy time, `ScriptCompiler.TryCompile`); **ApiMethod** findings remain hard errors (`SemanticValidationException`) because no downstream design-time gate exists for them. Apply the identical split in `DetectBlockersAsync` so preview and apply agree.
|
||||
4. Run → **PASS**; full Transport integration suite. Update `docs/requirements/Component-Transport.md` blocker-row section (template-script name findings are advisory; ApiMethod findings block).
|
||||
5. Commit: `git commit -m "fix(transport): blocker heuristic no longer hard-blocks valid imports — local declarations excluded, denylist extended, template-script findings downgraded to warnings"`
|
||||
|
||||
### Task 20: Run the script trust gate on bundle import
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 9–18, 21, 22 (same `BundleImporter` region as Task 19 — do after 19)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs` (`RunSemanticValidationAsync` Pass 2, ~3694+; `DetectBlockersAsync` for preview parity)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/SemanticValidatorImportTests.cs` (extend)
|
||||
|
||||
1. Failing test: bundle whose template script (and, second case, ApiMethod script) contains `System.Diagnostics.Process.Start("cmd")` → `ApplyAsync` throws `SemanticValidationException` naming the forbidden API; preview surfaces the same as a blocker row. Today both import clean — the fifth write path with no trust gate.
|
||||
2. Run → **FAIL**.
|
||||
3. Implement: in Pass 2, for every non-Skip template script, shared script, and ApiMethod script body, call `ScriptTrustValidator.FindViolations(code)` (ScriptAnalysis is referenced via TemplateEngine; add the direct project reference to `ZB.MOM.WW.ScadaBridge.Transport.csproj` if needed). Any violation → **hard error** for all three kinds (trust violations are not the false-positive-prone name heuristic — the semantic verdict is authoritative, same severity everywhere). The bodies are already in memory; Task 15's verdict cache keeps repeat cost nil.
|
||||
4. Run → **PASS**. Update `docs/requirements/Component-ScriptAnalysis.md` call-site list (Transport import is now the fifth delegating call site) and `Component-Transport.md`.
|
||||
5. Commit: `git commit -m "fix(security): bundle import runs the script trust gate — forbidden-API scripts rejected at import review, not at runtime"`
|
||||
|
||||
### Task 21: Harden the `ScriptTrustPolicy` deny-list
|
||||
|
||||
**Classification:** high-risk (security policy; false-positive blast radius on existing deployed scripts)
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1–20, 22
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ScriptAnalysis/ScriptTrustPolicy.cs` (`ForbiddenScopes` :36-45, `ReflectionGatewayMembers` :66-88)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ScriptAnalysis.Tests/ScriptTrustValidatorTests.cs` (extend)
|
||||
|
||||
1. Failing tests:
|
||||
```csharp
|
||||
[Theory]
|
||||
[InlineData("Environment.Exit(0);")]
|
||||
[InlineData("Environment.FailFast(\"x\");")]
|
||||
[InlineData("var s = Environment.GetEnvironmentVariable(\"SCADABRIDGE_API_KEY\");")]
|
||||
[InlineData("var c = new Microsoft.Data.SqlClient.SqlConnection(\"Server=attacker\");")]
|
||||
public void FindViolations_Flags(string code) => Assert.NotEmpty(ScriptTrustValidator.FindViolations(code));
|
||||
|
||||
[Theory] // syntactic gateway closure (matters in TPA-degraded fallback mode)
|
||||
[InlineData("typeof(string).Assembly.GetTypes()")]
|
||||
[InlineData("asm.EntryPoint")]
|
||||
public void SyntacticPass_Flags(string code) ...
|
||||
|
||||
[Theory] // must NOT regress legitimate scripts
|
||||
[InlineData("await Task.Delay(1);")]
|
||||
[InlineData("Func<int,int> f = x => x; f.Invoke(3);")] // delegate Invoke stays allowed — see step 3
|
||||
public void FindViolations_Allows(string code) => Assert.Empty(ScriptTrustValidator.FindViolations(code));
|
||||
```
|
||||
2. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.ScriptAnalysis.Tests --filter ScriptTrustValidatorTests` → **FAIL**.
|
||||
3. Implement:
|
||||
- `ForbiddenScopes` += `"System.Environment"` (whole type — no legitimate script use; `Environment.NewLine` callers can use `"\n"`), `"System.GC"`, and the ADO.NET provider namespaces: `"Microsoft.Data"`, `"System.Data.SqlClient"`, `"System.Data.Odbc"`, `"System.Data.OleDb"` (scripts reach `DbConnection` only via the `Database` helper, whose `System.Data.Common` abstract types stay allowed — closes the arbitrary-host `new SqlConnection(...)` channel the `System.Net` deny was meant to cover).
|
||||
- `ReflectionGatewayMembers` += `"GetTypes"`, `"EntryPoint"`, `"DeclaredMethods"`, `"DeclaredMembers"`, `"DeclaredConstructors"`, `"DynamicInvoke"`. **Deliberately exclude `"Invoke"`**: the syntactic pass rejects regardless of receiver, and `delegate.Invoke()` is legitimate; `MethodInfo.Invoke` is already caught semantically (`System.Reflection` scope) — record this rationale in the code comment.
|
||||
4. Run → **PASS**; also run the SiteRuntime/InboundAPI representative-script suites if they exercise the surface (`dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter Script` — any newly-flagged legitimate pattern is a deliberate decision to surface, not silently absorb).
|
||||
5. Update `docs/requirements/Component-ScriptAnalysis.md` deny-list table + the `System.Data` posture paragraph. Commit: `git commit -m "fix(security): deny Environment/GC and ADO.NET provider namespaces; close reflection-gateway list (GetTypes/EntryPoint/Declared*/DynamicInvoke)"`
|
||||
|
||||
### Task 22: Bound `LineDiffer` input size
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~3 min
|
||||
**Parallelizable with:** 1–21
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/LineDiffer.cs`
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.Tests/Import/LineDifferTests.cs` (extend)
|
||||
|
||||
1. Failing test: two 10,000-line inputs with zero common lines → `Diff` returns within a small time budget with `Truncated = true` and a summary-only result (hunk list empty or `<N lines>` placeholders), and never materializes the Myers trace. Assert e.g. total runtime < 2s (today: ~20k rounds × 160 KB V-snapshots ≈ multi-GB transient allocation — the assertion red-lines as timeout/OOM).
|
||||
2. Run `--filter LineDifferTests` → **FAIL** (or pathological slowness — treat as fail).
|
||||
3. Implement: `private const int MaxInputLines = 4000;` at the top of `Diff`: `if (aLines.Length + bLines.Length > MaxInputLines) return SummaryOnlyResult(aLines.Length, bLines.Length);` producing the same shape the `maxLines` output cap already emits (`Truncated = true`, add/remove totals = full line counts, no hunks). The 400-line *output* cap stays; this bounds the *algorithm input* so a bloated or crafted bundle cannot OOM the active central node from the import preview.
|
||||
4. Run → **PASS**. Commit: `git commit -m "fix(transport): cap LineDiffer input size — oversized script diffs degrade to summary instead of O((N+M)^2) memory"`
|
||||
|
||||
### Task 23: Fix `ArtifactDiff` placeholder-identity comparisons
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 9–18, 21, 22 (touches `ArtifactDiff` + preview call site — after Tasks 5/6)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/ArtifactDiff.cs` (`CompareTemplate` gains optional `IReadOnlyDictionary<int,string>? folderNameById`, `IReadOnlyDictionary<int,string>? templateNameById`; `FolderNameOf`/`BaseTemplateNameOf`/`CompositionTargetNameOf` at 674-693 use them)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs` (`PreviewAsync` ~379-388 passes the maps it already builds at ~365-366)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.Tests/Import/ArtifactDiffTests.cs` (extend)
|
||||
|
||||
1. Failing test: existing template with `FolderId`, `ParentTemplateId`, and one composition; incoming DTO carries the *matching names* → `CompareTemplate(dto, existing, folderMap, templateMap)` classifies **Identical** (today: `"<id:5>" != "Pump"` ⇒ spurious Modified on every re-import, defeating the wizard's auto-skip contract, `Component-Transport.md:222`).
|
||||
2. Run → **FAIL**.
|
||||
3. Implement: resolve real names through the maps, falling back to the `<id:N>` placeholder only when a map is absent/misses (keeps existing unit tests valid). Wire the two maps through from `PreviewAsync`.
|
||||
4. Run Transport unit + integration suites → **PASS**. Commit: `git commit -m "fix(transport): template diffs resolve folder/base/composition names — unchanged structured templates classify Identical again"`
|
||||
|
||||
### Task 24: Low-severity cleanup batch
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** none (touches files owned by Tasks 15 and 19 — run after both)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleSessionStore.cs` + `src/ZB.MOM.WW.ScadaBridge.Transport/TransportOptions.cs` (session cap)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ScriptCompiler.cs` (:42, :47 — full error lists)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/SemanticValidator.cs` (~43-56 — leaf-fallback warning)
|
||||
- Tests: `BundleSessionStoreTests.cs`, `ScriptCompilerTests.cs`, `SemanticValidatorTests` (Validation folder)
|
||||
|
||||
Three concrete edits, each TDD'd:
|
||||
1. **Session cap:** `TransportOptions.MaxConcurrentImportSessions = 8`. Failing test: 9th `Add` throws `InvalidOperationException("Too many concurrent import sessions …")`; removing one admits a new one. Implement the count check in `BundleSessionStore.Add` (:38). Bounds the N×~200 MB decrypted-content pinning.
|
||||
2. **Full error lists:** failing test: script with two forbidden APIs → failure message contains both. Implement: `string.Join("; ", violations)` / `string.Join("; ", errors)` instead of `violations[0]`/`errors[0]` (coordinate with Task 15's cached-error format — store the joined string).
|
||||
3. **Leaf-fallback warning:** failing test: composed script call resolved only by leaf-name fallback yields a `ValidationEntry.Warning(ValidationCategory.…, "call target '…' matched by leaf name only — child path not verified")` instead of silent acceptance. Implement in the fallback branch at `SemanticValidator.cs:43-56` (return-shape permitting; if the method returns bool, thread a warnings collector the way `ValidationService` already aggregates warnings).
|
||||
4. Run all three filters → **PASS**. Commit: `git commit -m "chore: low-severity sweep — import session cap, full compile-error lists, leaf-name-fallback validation warning"`
|
||||
|
||||
### Task 25: Design-doc + CLAUDE.md sync sweep and deferred-decision ledger
|
||||
|
||||
**Classification:** small (docs only)
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** none (run last)
|
||||
**Files:**
|
||||
- Modify: `docs/requirements/Component-Transport.md`, `Component-TemplateEngine.md`, `Component-DeploymentManager.md`, `Component-ScriptAnalysis.md`, `/Users/dohertj2/Desktop/ScadaBridge/CLAUDE.md` (component #24 blurb + Transport bullet in Key Design Decisions)
|
||||
|
||||
No tests; concrete edits (several were partially made inside their behavior tasks — this pass verifies and completes them, then does a stale-cross-reference sweep per repo convention):
|
||||
1. **Component-Transport.md:** (a) remove the `scripts/` bundle-directory section (:45-47) or mark it explicitly *"not implemented — `MaxBundleEntryCount` default 4 would reject it"*; (b) transported-entity list gains template `NativeAlarmSources`, `LockedInDerived`, script cadence/timeout, real `AreaName`; (c) blocker section documents the warning/error severity split (Task 19) and the LineDiffer input cap (Task 22); (d) make the *rename call-site limitation* explicit and load-bearing: *a Rename resolution does not rewrite call sites in importing scripts — the target DB can end up with scripts calling the old name; Pass-1 registers both names so import passes, but deploy-time validation on the target is the real gate*; (e) note the accepted preview→apply window (no optimistic concurrency; version fields reserved — Underdeveloped 5) as a recorded decision.
|
||||
2. **Component-TemplateEngine.md:** align the alarm-override granularity sentence (:113) to the implementation — only Trigger Definition thresholds and Priority Level are instance-overridable; Description and On-Trigger Script reference are **not** (`InstanceAlarmOverride` carries only `TriggerConfigurationOverride`/`PriorityLevelOverride`); record it as the decision (adding the two override columns is future feature work, not spec debt). Add the revision-hash native-source migration note (Task 11) if not already present.
|
||||
3. **Component-DeploymentManager.md:** `NotDeployed` delete is central-side only (Task 18); post-success audit isolation (Task 17); add the explicit `OperationLockManager` invariant paragraph: *locks are per-node in-memory; mutual exclusion assumes management traffic reaches only the active central node (Traefik). Direct standby-port mutations are unsupported — structural rejection of ops on the non-active node is owned by the cluster/UI plans (01/07).*
|
||||
4. **Component-ScriptAnalysis.md:** deny-list additions + `System.Data` provider posture + Transport import as the fifth delegating call site (Tasks 20, 21).
|
||||
5. **CLAUDE.md:** component #24 blurb — add template native-alarm-source transport and confirm the Area-by-name claim is now true; adjust the "Transport (#24, M8)" Key Design Decisions bullet likewise. Per the umbrella-index rule, no `../scadaproj/CLAUDE.md` change is needed (no wire-relationship/stack change) — verify and note in the commit message.
|
||||
6. `git diff` review, then commit: `git commit -m "docs: sync Transport/TemplateEngine/DeploymentManager/ScriptAnalysis specs + CLAUDE.md with the arch-review fix wave (plan 05)"`
|
||||
|
||||
---
|
||||
|
||||
## Dependencies on other plans
|
||||
|
||||
- **Plan 06 (Edge Integrations) — explicit handoff:** Task 14 ships the `ScriptArtifactsChanged` message, `IScriptArtifactChangeBus` seam, in-process bus, contract doc, and the Transport post-commit publisher. Plan 06 implements the consumers: `InboundScriptExecutor` eviction of `_scriptHandlers`/`_knownBadMethods`, publishing from the ManagementActor ApiMethod/shared-script update paths, and cross-node/failover handler freshness. Plan 06 should not start its consumer work until Task 14's contract doc is committed.
|
||||
- **Plans 01/07:** structural rejection of management mutations on the non-active central node (the `OperationLockManager` residual risk) belongs to the cluster-routing/management-surface owners; Task 25 records the invariant only.
|
||||
- **Plan 08:** also flagged the `AreaName: null` half-shipped feature — the fix is owned here (Task 7); plan 08 should not duplicate it.
|
||||
- **Plan 03:** runtime script execution/containment is untouched by this plan (reference only).
|
||||
|
||||
## Execution order
|
||||
|
||||
**P0 (start immediately, in parallel):** Task 1 → 2 (the Critical, sequential), Task 14 (unblocks plan 06), Task 9 → 10 (flattener correctness), Task 11 (after 13), Task 13.
|
||||
**Wave 2 (BundleImporter chain — same file, serialize):** 3 → 4 → 5 → 6 → 19 → 20 → 23, with 7 in parallel (exporter files) and 15/16/17/18/21/22 in parallel (disjoint projects).
|
||||
**Wave 3:** 8 (equivalence suite — after 3–7), 12, 24.
|
||||
**Last:** 25 (doc sweep), then a full `dotnet build ZB.MOM.WW.ScadaBridge.slnx && dotnet test ZB.MOM.WW.ScadaBridge.slnx` plus the direct `dotnet test tests/ZB.MOM.WW.ScadaBridge.CLI.Tests` run (slnx-exclusion gotcha) before declaring the plan done.
|
||||
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"planPath": "archreview/plans/PLAN-05-templates-deployment-transport.md",
|
||||
"tasks": [
|
||||
{ "id": 1, "subject": "Task 1: Wire template inheritance edges in the import apply path", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 2, "subject": "Task 2: Import-time acyclicity check over the merged template graph", "status": "pending", "blockedBy": [1] },
|
||||
{ "id": 3, "subject": "Task 3: Carry MinTimeBetweenRuns/ExecutionTimeoutSeconds through import", "status": "pending", "blockedBy": [2] },
|
||||
{ "id": 4, "subject": "Task 4: Add LockedInDerived to template child DTOs and apply paths", "status": "pending", "blockedBy": [3] },
|
||||
{ "id": 5, "subject": "Task 5: Transport template NativeAlarmSources end-to-end", "status": "pending", "blockedBy": [4] },
|
||||
{ "id": 6, "subject": "Task 6: Single-source per-entity field comparers for diff + sync", "status": "pending", "blockedBy": [5] },
|
||||
{ "id": 7, "subject": "Task 7: Export real AreaName (finish half-shipped Area transport)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 8, "subject": "Task 8: Round-trip export→import equivalence test suite", "status": "pending", "blockedBy": [5, 6, 7] },
|
||||
{ "id": 9, "subject": "Task 9: Fix repeated-composition member loss in FlatteningService", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 10, "subject": "Task 10: Fix CollisionDetector blindness under repeated composition", "status": "pending", "blockedBy": [9] },
|
||||
{ "id": 11, "subject": "Task 11: Add NativeAlarmSources to RevisionHashService (planned hash migration)", "status": "pending", "blockedBy": [13] },
|
||||
{ "id": 12, "subject": "Task 12: Add NativeAlarmSources to DiffService", "status": "pending", "blockedBy": [13] },
|
||||
{ "id": 13, "subject": "Task 13: Enforce locks on native alarm source overrides", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 14, "subject": "Task 14: Script-artifact invalidation contract — Commons seam + Transport publisher (plan-06 handoff)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 15, "subject": "Task 15: Cache script compile verdicts by code hash", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 16, "subject": "Task 16: Skip script compilation on read-only staleness/comparison paths", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 17, "subject": "Task 17: Isolate the post-success deployment audit write", "status": "pending", "blockedBy": [16] },
|
||||
{ "id": 18, "subject": "Task 18: Delete NotDeployed instances without a site round-trip", "status": "pending", "blockedBy": [17] },
|
||||
{ "id": 19, "subject": "Task 19: Import blocker heuristic — local-declaration exclusion, denylist additions, warning severity", "status": "pending", "blockedBy": [6] },
|
||||
{ "id": 20, "subject": "Task 20: Run the script trust gate on bundle import", "status": "pending", "blockedBy": [19] },
|
||||
{ "id": 21, "subject": "Task 21: Harden the ScriptTrustPolicy deny-list", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 22, "subject": "Task 22: Bound LineDiffer input size", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 23, "subject": "Task 23: Fix ArtifactDiff placeholder-identity comparisons", "status": "pending", "blockedBy": [20] },
|
||||
{ "id": 24, "subject": "Task 24: Low-severity cleanup batch (session cap, full error lists, semantic-validator warning)", "status": "pending", "blockedBy": [15, 19] },
|
||||
{ "id": 25, "subject": "Task 25: Design-doc + CLAUDE.md sync sweep and deferred-decision ledger", "status": "pending", "blockedBy": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24] }
|
||||
],
|
||||
"lastUpdated": "2026-07-08"
|
||||
}
|
||||
@@ -0,0 +1,820 @@
|
||||
# Edge Integrations Fix Implementation Plan
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Fix every finding in `archreview/06-edge-integrations.md` — make the inbound compiled-handler cache self-healing (revision-checked, failover/direct-SQL/import-safe), close both ESG spec gaps (per-system timeout, path templating), bound outbound response buffering and SMS blast radius, wire the dead SMS retry fields, and clear the report's Medium/Low stability, performance, and convention findings across InboundAPI, ExternalSystemGateway, NotificationService/Outbox delivery, and DelmiaNotifier.
|
||||
|
||||
**Architecture:** The core stability fix replaces the name-keyed compiled-handler cache in `InboundScriptExecutor` with a script-revision-checked cache (the fresh `ApiMethod` row is already in hand on every request — one ordinal string compare detects staleness and triggers recompile), plus a public `InvalidateMethod` seam that plan 05's "script artifact changed → invalidate" contract will consume. The ESG per-system timeout is a full vertical slice per repo rules: Commons entity → EF mapping + migration → artifact message contract (additive) → site SQLite column → client honor → management command → CLI → UI → design doc. Notification fixes are localized to the delivery adapters and the outbox dispatcher's retry-policy resolution.
|
||||
|
||||
**Tech Stack:** C#/.NET 9, xUnit + NSubstitute, EF Core migrations (MS SQL central), Microsoft.Data.Sqlite (site), Roslyn scripting, System.CommandLine (CLI), Blazor Server (UI).
|
||||
|
||||
Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per-project: `dotnet test tests/<project>`. EF migrations: **build first, never `--no-build`**; `dotnet ef migrations add <Name> --project src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase --startup-project src/ZB.MOM.WW.ScadaBridge.Host`. If the migration scaffolds empty, delete the files and rebuild first.
|
||||
|
||||
## Findings Coverage
|
||||
|
||||
| # | Finding (report section) | Severity | Task(s) |
|
||||
|---|--------------------------|----------|---------|
|
||||
| S1 | Compiled inbound handler cache stale across central failover (+ known-bad mirror image) | High | 1, 2, 3 |
|
||||
| S2 | Method timeout abandons running script; DI scope disposed underneath (use-after-dispose + orphan accumulation, no counter) | Medium | 4 |
|
||||
| S3 | ESG API-key `AuthConfiguration` colon-splitting ambiguous (also silently misparses the JSON shapes the UI placeholder already suggests) | Medium | 15 |
|
||||
| S4 | DelmiaNotifier failover can duplicate a processed notification (at-least-once undocumented) | Medium | 23 |
|
||||
| S5 | SMS adapter keeps sending after first transient — duplicate amplification on retry | Medium | 17 |
|
||||
| S6 | `ErrorClassifier.IsTransient(Exception)` treats any OCE as transient (misuse-prone public predicate) | Low | 16 |
|
||||
| S7 | First-of-many SMTP configuration nondeterministic (`FirstOrDefault` in adapter + retry-policy resolver) | Low | 21 |
|
||||
| S8 | OAuth2 token cache: unsynchronized field visibility (growth is bounded in practice — accepted) | Low | 21 |
|
||||
| S9 | Non-JSON content-type with body yields misleading 400 instead of 415 | Low | 5 |
|
||||
| P1 | Outbound HTTP responses buffered unbounded | Medium | 14 |
|
||||
| P2 | Serial per-recipient Twilio POSTs inside serial dispatch sweep | Medium | 17 (first-transient short-circuit bounds the black-hole worst case; bounded parallelism **deferred** — at current list sizes short-circuit alone caps the sweep at ~1 timeout) |
|
||||
| P3 | Per-delivery TCP+TLS SMTP handshake | Low | **Won't-fix** — deliberate, documented tradeoff with the pooling seam already left open (`MailKitSmtpClientWrapper.cs:12-38`); report itself says "correct tradeoff at current volume" |
|
||||
| P4 | Startup compiles every inbound method serially | Low | 7 |
|
||||
| C1 | Spec drift: per-system ESG timeout specified but not implemented | High | 8, 9, 10, 11, 12 |
|
||||
| C2 | Spec drift: path templates (`/recipes/{id}`) never substituted | Medium | 13 |
|
||||
| C3 | Spec drift: inbound error responses lack documented `code` field (incl. SITE_UNREACHABLE surfacing as generic 500) | Low | 5 |
|
||||
| C4 | OAuth2 SMTP is Microsoft-365-only despite broader spec | Low | 19, 20 |
|
||||
| C5 | `InboundApiEndpointFilter` freezes options at construction | Low | 6 |
|
||||
| U1 | Script trust boundary is static-only (no runtime sandbox) | Underdeveloped | **Deferred** — restricted-ALC/out-of-process containment is an architecture change already logged as deferred to v1.x in the codebase's own deferred-work inventory; interacts with plan 05's Script Analysis ownership. Not a bite-sized fix. |
|
||||
| U2 | No cache-coherence mechanism for compiled handlers | Underdeveloped | 1, 3 (revision-check fallback works standalone; Task 3 provides the consumer seam for plan 05's invalidation contract) |
|
||||
| U3 | Per-SMS retry settings are dead fields (`SmsConfiguration.MaxRetries`/`RetryDelay` never read) | Underdeveloped | 18 |
|
||||
| U4 | Twilio accept-only + `AccountSid` interpolated un-escaped into URI | Underdeveloped | 22 (SID format validation at save). Status-callback webhook / per-recipient delivery state: **Deferred** — documented out-of-scope in `Component-NotificationService.md:93,102`; needs an inbound webhook endpoint (new attack surface) that warrants its own design pass. |
|
||||
| U5 | DelmiaNotifier duplicate-delivery semantics undocumented; no CI artifact for shipped binary | Underdeveloped | 23 (docs). Windows AOT CI artifact: **Deferred** — repo has no Windows CI runner; the design doc's manual-smoke note stands. |
|
||||
| U6 | Test gaps: handler failover/staleness, oversized outbound body, mid-list SMS transient | Underdeveloped | 1, 14, 17 (each task's failing-test step is exactly the missing test) |
|
||||
| U7 | API key rotation procedure unwritten | Underdeveloped | 24 |
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Revision-checked inbound handler cache (self-healing across failover / direct SQL / known-bad)
|
||||
|
||||
**Classification:** high-risk (concurrency; changes the serving semantics of every inbound API method)
|
||||
**Estimated implement time:** 5 min
|
||||
**Parallelizable with:** 6, 7, 8, 17, 18, 23, 24
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundScriptExecutor.cs` (lines 26, 38-63, 118-143, 311-331)
|
||||
- Test (create): `tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/InboundScriptExecutorStalenessTests.cs`
|
||||
|
||||
Design: the cache value becomes `CachedHandler(string? Script, Func<InboundScriptContext, Task<object?>> Handler)`; `_knownBadMethods` becomes `ConcurrentDictionary<string, string>` (method name → the exact script text that failed). On every `ExecuteAsync`, the freshly-fetched `method.Script` is ordinal-compared against the cached script; mismatch → recompile in place (and purge the known-bad record). A known-bad record only short-circuits when the *current* script is the one that failed — a changed script always gets a fresh compile. `RegisterHandler(name, handler)` (test seam) stores `Script = null`, meaning "no staleness info, serve as-is" — existing tests keep passing; all production compiles flow through `CompileAndRegister`, which always records the script text. This fixes failover staleness, direct-SQL-edit staleness, and stale known-bad in one move; marginal per-request cost is one ordinal string compare of a few-KB script.
|
||||
|
||||
Behavioral consequence (deliberate, see Task 2): the DB row becomes authoritative. A saved-but-broken script now returns HTTP 500 "Script compilation failed" on every node instead of the old delegate silently serving on the node that compiled it — honest failure replaces node-divergent stale success.
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Write the failing tests:
|
||||
|
||||
```csharp
|
||||
// tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/InboundScriptExecutorStalenessTests.cs
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using NSubstitute;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Entities.InboundApi;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.InboundAPI.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Arch-review S1: the compiled-handler cache must be revision-checked against the
|
||||
/// freshly-fetched ApiMethod.Script so a standby node (whose cache was populated at
|
||||
/// its own startup) never serves a stale delegate after failover, and a method fixed
|
||||
/// out-of-band is never stuck behind a stale known-bad record.
|
||||
/// </summary>
|
||||
public class InboundScriptExecutorStalenessTests
|
||||
{
|
||||
private readonly InboundScriptExecutor _executor;
|
||||
private readonly RouteHelper _route;
|
||||
|
||||
public InboundScriptExecutorStalenessTests()
|
||||
{
|
||||
_executor = new InboundScriptExecutor(
|
||||
NullLogger<InboundScriptExecutor>.Instance, Substitute.For<IServiceProvider>());
|
||||
_route = new RouteHelper(
|
||||
Substitute.For<IInstanceLocator>(), Substitute.For<IInstanceRouter>());
|
||||
}
|
||||
|
||||
private Task<InboundScriptResult> Run(ApiMethod m) => _executor.ExecuteAsync(
|
||||
m, new Dictionary<string, object?>(), _route, TimeSpan.FromSeconds(10));
|
||||
|
||||
[Fact]
|
||||
public async Task UpdatedScript_RecompilesInsteadOfServingStaleHandler()
|
||||
{
|
||||
// Simulates the failover scenario: this node compiled v1 at startup...
|
||||
var v1 = new ApiMethod("stale", "return 1;") { Id = 1, TimeoutSeconds = 10 };
|
||||
Assert.True(_executor.CompileAndRegister(v1));
|
||||
var r1 = await Run(v1);
|
||||
Assert.Contains("1", r1.ResultJson);
|
||||
|
||||
// ...the active node updated the method (this node never heard about it);
|
||||
// the per-request DB fetch hands ExecuteAsync the NEW row.
|
||||
var v2 = new ApiMethod("stale", "return 2;") { Id = 1, TimeoutSeconds = 10 };
|
||||
var r2 = await Run(v2);
|
||||
|
||||
Assert.True(r2.Success);
|
||||
Assert.Contains("2", r2.ResultJson); // old cache would have returned 1
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task KnownBadMethod_FixedScript_CompilesInsteadOfFastFailing()
|
||||
{
|
||||
// The mirror-image variant: broken at this node's startup...
|
||||
var broken = new ApiMethod("fixme", "%%% not C# %%%") { Id = 2, TimeoutSeconds = 10 };
|
||||
Assert.False(_executor.CompileAndRegister(broken));
|
||||
var r1 = await Run(broken);
|
||||
Assert.False(r1.Success);
|
||||
|
||||
// ...fixed via the management path on the OTHER node; this node's fresh
|
||||
// DB fetch carries the fixed script.
|
||||
var fixedMethod = new ApiMethod("fixme", "return 42;") { Id = 2, TimeoutSeconds = 10 };
|
||||
var r2 = await Run(fixedMethod);
|
||||
|
||||
Assert.True(r2.Success);
|
||||
Assert.Contains("42", r2.ResultJson);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task KnownBadMethod_SameScript_StillFastFails()
|
||||
{
|
||||
var broken = new ApiMethod("stillbad", "%%% not C# %%%") { Id = 3, TimeoutSeconds = 10 };
|
||||
Assert.False(_executor.CompileAndRegister(broken));
|
||||
|
||||
// Same script text → the fast-fail record must hold (no per-request Roslyn).
|
||||
var again = new ApiMethod("stillbad", "%%% not C# %%%") { Id = 3, TimeoutSeconds = 10 };
|
||||
var r = await Run(again);
|
||||
Assert.False(r.Success);
|
||||
Assert.Contains("Script compilation failed", r.ErrorMessage);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task TestSeamRegisterHandler_WithoutScript_IsNotRevisionChecked()
|
||||
{
|
||||
// RegisterHandler (script-less test seam) pins the handler regardless of row content.
|
||||
var m = new ApiMethod("pinned", "return 0;") { Id = 4, TimeoutSeconds = 10 };
|
||||
_executor.RegisterHandler("pinned", async _ => { await Task.CompletedTask; return 99; });
|
||||
var r = await Run(m);
|
||||
Assert.True(r.Success);
|
||||
Assert.Contains("99", r.ResultJson);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests --filter InboundScriptExecutorStalenessTests` — expect **FAIL** (`UpdatedScript_RecompilesInsteadOfServingStaleHandler` returns 1; `KnownBadMethod_FixedScript…` fast-fails).
|
||||
|
||||
3. Minimal implementation in `InboundScriptExecutor.cs`:
|
||||
- Add `private sealed record CachedHandler(string? Script, Func<InboundScriptContext, Task<object?>> Handler);` and change `_scriptHandlers` to `ConcurrentDictionary<string, CachedHandler>`.
|
||||
- Change `_knownBadMethods` to `ConcurrentDictionary<string, string>`; `TryRecordBadMethod(string methodName, string script)` stores the failing script text (cap logic unchanged; `_knownBadMethods[methodName] = script` when already present so a *newer* bad script replaces the record).
|
||||
- `RegisterHandler(name, handler)` → `_scriptHandlers[name] = new CachedHandler(null, handler);` (doc-comment it as the script-less test seam, exempt from revision checks).
|
||||
- `Register(methodName, handler)` → `Register(ApiMethod method, handler)` storing `new CachedHandler(method.Script, handler)`.
|
||||
- In `ExecuteAsync`, replace the lookup block (lines 311-331) with:
|
||||
|
||||
```csharp
|
||||
_scriptHandlers.TryGetValue(method.Name, out var cached);
|
||||
// Revision check: a cached handler compiled from different script text is stale
|
||||
// (standby-after-failover, direct SQL edit, bundle import). Null Script = test seam, exempt.
|
||||
if (cached is { Script: not null }
|
||||
&& !string.Equals(cached.Script, method.Script, StringComparison.Ordinal))
|
||||
{
|
||||
_logger.LogInformation(
|
||||
"API method {Method} script changed since it was cached; recompiling", method.Name);
|
||||
cached = null;
|
||||
}
|
||||
|
||||
if (cached == null)
|
||||
{
|
||||
// Fast-fail only when the CURRENT script text is the one already known bad.
|
||||
if (_knownBadMethods.TryGetValue(method.Name, out var badScript)
|
||||
&& string.Equals(badScript, method.Script, StringComparison.Ordinal))
|
||||
return new InboundScriptResult(false, null, "Script compilation failed for this method");
|
||||
|
||||
var (compiled, _) = Compile(method);
|
||||
if (compiled == null)
|
||||
{
|
||||
TryRecordBadMethod(method.Name, method.Script ?? string.Empty);
|
||||
return new InboundScriptResult(false, null, "Script compilation failed for this method");
|
||||
}
|
||||
|
||||
_knownBadMethods.TryRemove(method.Name, out _);
|
||||
cached = new CachedHandler(method.Script, compiled);
|
||||
_scriptHandlers[method.Name] = cached; // last-write-wins; concurrent first-callers race benignly
|
||||
}
|
||||
|
||||
var result = await cached.Handler(context).WaitAsync(cts.Token);
|
||||
```
|
||||
|
||||
- In `CompileAndRegister` update the failure branch to `TryRecordBadMethod(method.Name, method.Script ?? string.Empty)`.
|
||||
|
||||
4. Run the new filter → **PASS**, then the whole project: `dotnet test tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests` → all green (existing `RegisterHandler`-based tests keep passing via the null-script seam).
|
||||
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundScriptExecutor.cs tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/InboundScriptExecutorStalenessTests.cs && git commit -m "fix(inbound-api): revision-check compiled handler cache against the fresh ApiMethod row — heals failover/direct-SQL/known-bad staleness"`
|
||||
|
||||
### Task 2: DB-authoritative compile semantics — management warning text + design doc
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** 3 min
|
||||
**Parallelizable with:** 6, 7, 8, 17, 18, 23
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (`TryCompileAndDescribe`, ~lines 2646-2660: warning strings)
|
||||
- Modify: `docs/requirements/Component-InboundAPI.md` (the "Direct SQL Warning" section, ~lines 129-131, plus the broken-save behavior description)
|
||||
|
||||
Task 1 changed the truth: after a broken save, the *old* handler no longer keeps serving (the per-request revision check bypasses it and the method 500s with a compile error on every node consistently). The management warning and the component doc must say so, or they now lie.
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. In `TryCompileAndDescribe`, replace the `isUpdate` warning string with: `"Script saved but failed to compile; the method will return a compilation error on every request until a compiling version is saved. Compilation errors: {detail}"` (the create-path string is already correct). Update the `CompileAndRegister` XML doc-comment in `InboundScriptExecutor.cs` (lines 105-114) if Task 1 did not already: the "previously registered handler is left in place" sentence must be replaced with the DB-authoritative behavior.
|
||||
2. In `Component-InboundAPI.md`, rewrite the Direct SQL Warning section: direct SQL edits are now picked up on the next request via the per-request script revision check (no restart / no management round-trip needed); note that a broken script serves a 500 compile-failure consistently on all central nodes.
|
||||
3. Verify no stale text remains: `grep -rn "previously registered" src/ZB.MOM.WW.ScadaBridge.ManagementService src/ZB.MOM.WW.ScadaBridge.InboundAPI docs/requirements/Component-InboundAPI.md` — every remaining hit must describe the new semantics.
|
||||
4. `dotnet build ZB.MOM.WW.ScadaBridge.slnx` → green; `dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests` (some tests may assert the old warning text — update them to the new string).
|
||||
5. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.ManagementService src/ZB.MOM.WW.ScadaBridge.InboundAPI tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests docs/requirements/Component-InboundAPI.md && git commit -m "docs(inbound-api): DB row is authoritative for method scripts — update broken-save warning + Direct SQL Warning for the revision-checked cache"`
|
||||
|
||||
### Task 3: `InvalidateMethod` seam — the Inbound API consumer of plan 05's invalidation contract
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** 3 min
|
||||
**Parallelizable with:** 8, 9, 10, 13-17, 19, 23, 24
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundScriptExecutor.cs` (after Task 1)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (`HandleDeleteApiMethod`, ~line 2631: swap `RemoveHandler` for `InvalidateMethod`) (after Task 2)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/InboundScriptExecutorStalenessTests.cs` (append)
|
||||
|
||||
Plan 05 designs the general "script artifact changed → invalidate everywhere" contract (bundle import is its consumer to wire). This task gives the Inbound API one obvious, complete entry point: `InvalidateMethod(string methodName)` drops **both** the compiled handler and the known-bad record. Even before plan 05 lands, Task 1's revision check makes invalidation a fast-path optimization rather than a correctness requirement — that is the designed fallback.
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Failing test (append to `InboundScriptExecutorStalenessTests`):
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task InvalidateMethod_DropsHandlerAndKnownBadRecord()
|
||||
{
|
||||
var good = new ApiMethod("inv", "return 5;") { Id = 5, TimeoutSeconds = 10 };
|
||||
Assert.True(_executor.CompileAndRegister(good));
|
||||
var broken = new ApiMethod("invbad", "%%%") { Id = 6, TimeoutSeconds = 10 };
|
||||
Assert.False(_executor.CompileAndRegister(broken));
|
||||
|
||||
_executor.InvalidateMethod("inv");
|
||||
_executor.InvalidateMethod("invbad");
|
||||
|
||||
Assert.Equal(0, _executor.KnownBadMethodCount);
|
||||
// After invalidation the method lazily recompiles from the row — still serves.
|
||||
var r = await Run(good);
|
||||
Assert.True(r.Success);
|
||||
}
|
||||
```
|
||||
|
||||
2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests --filter InvalidateMethod_DropsHandlerAndKnownBadRecord` → **FAIL** (no such method).
|
||||
3. Implement:
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Drops the compiled handler AND any known-bad record for the method. The
|
||||
/// consumption seam for the script-artifact invalidation contract (bundle import,
|
||||
/// out-of-band updates, failover activation). Safe to call for unknown names.
|
||||
/// The per-request revision check in ExecuteAsync remains the correctness
|
||||
/// fallback when an invalidation is missed.
|
||||
/// </summary>
|
||||
public void InvalidateMethod(string methodName)
|
||||
{
|
||||
_scriptHandlers.TryRemove(methodName, out _);
|
||||
_knownBadMethods.TryRemove(methodName, out _);
|
||||
}
|
||||
```
|
||||
|
||||
Keep `RemoveHandler` as a one-line delegate to `InvalidateMethod` (public API preserved), and change `HandleDeleteApiMethod` to call `InvalidateMethod(method.Name)`.
|
||||
4. Run filter → **PASS**; run full `dotnet test tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests`.
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundScriptExecutor.cs src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/InboundScriptExecutorStalenessTests.cs && git commit -m "feat(inbound-api): InvalidateMethod seam purging handler + known-bad — consumer entry point for the script-artifact invalidation contract (plan 05)"`
|
||||
|
||||
### Task 4: Abandoned inbound executions — defer DI-scope disposal + orphan telemetry
|
||||
|
||||
**Classification:** high-risk (concurrency; per-request lifecycle)
|
||||
**Estimated implement time:** 5 min
|
||||
**Parallelizable with:** 8-16, 17-22
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundScriptExecutor.cs` (`ExecuteAsync` body + finally, lines 333, 388-394) (after Task 3)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Observability/ScadaBridgeTelemetry.cs` (add abandoned-execution counter, mirroring `RecordInboundApiRequest`)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/InboundScriptExecutorTests.cs` (append)
|
||||
|
||||
On timeout, `WaitAsync` returns but the handler task keeps running; the `finally` disposes the DI scope underneath it → `ObjectDisposedException` background faults, and orphans accumulate invisibly (no rate limiting on this API). Fix: capture the handler task; in `finally`, dispose the scope only if the task completed — otherwise increment an orphan gauge, log, and attach a continuation that observes the fault, disposes the scope, and decrements the gauge.
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Failing test:
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task TimedOutScript_ScopeDisposalDeferred_NoOrphanLeak()
|
||||
{
|
||||
// Handler ignores the token and keeps running past the method timeout.
|
||||
var release = new TaskCompletionSource();
|
||||
_executor.RegisterHandler("slow", async _ => { await release.Task; return 1; });
|
||||
var method = new ApiMethod("slow", "unused") { Id = 9, TimeoutSeconds = 1 };
|
||||
|
||||
var result = await _executor.ExecuteAsync(
|
||||
method, new Dictionary<string, object?>(), _route, TimeSpan.FromMilliseconds(50));
|
||||
|
||||
Assert.False(result.Success);
|
||||
Assert.Contains("timed out", result.ErrorMessage);
|
||||
Assert.Equal(1, _executor.AbandonedExecutionCount); // orphan is COUNTED
|
||||
|
||||
release.SetResult(); // orphan finishes; continuation cleans up
|
||||
await Task.Delay(200);
|
||||
Assert.Equal(0, _executor.AbandonedExecutionCount); // ...and DECREMENTED
|
||||
}
|
||||
```
|
||||
|
||||
2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests --filter TimedOutScript_ScopeDisposalDeferred_NoOrphanLeak` → **FAIL** (no `AbandonedExecutionCount`).
|
||||
3. Implement in `ExecuteAsync`:
|
||||
- Add `private long _abandonedExecutions;` and `internal long AbandonedExecutionCount => Interlocked.Read(ref _abandonedExecutions);` (public if the health surface needs it later).
|
||||
- Hoist the handler task: `Task<object?>? handlerTask = null;` declared next to `scope`; replace line 333 with `handlerTask = cached.Handler(context); var result = await handlerTask.WaitAsync(cts.Token);`.
|
||||
- Replace the `finally` block:
|
||||
|
||||
```csharp
|
||||
finally
|
||||
{
|
||||
if (handlerTask is null || handlerTask.IsCompleted)
|
||||
{
|
||||
scope?.Dispose();
|
||||
}
|
||||
else
|
||||
{
|
||||
// The handler outlived the request (non-cooperative script after a
|
||||
// timeout/client abort). Do NOT dispose the scope underneath it —
|
||||
// defer cleanup to the task's own completion, and count the orphan
|
||||
// so repeated timeouts of a slow method are visible.
|
||||
Interlocked.Increment(ref _abandonedExecutions);
|
||||
ScadaBridgeTelemetry.RecordInboundAbandonedExecution(method.Name);
|
||||
_logger.LogWarning(
|
||||
"Abandoned inbound execution for method {Method} is still running; DI scope disposal deferred", method.Name);
|
||||
var abandonedScope = scope;
|
||||
_ = handlerTask.ContinueWith(t =>
|
||||
{
|
||||
_ = t.Exception; // observe the fault so it never surfaces as unobserved
|
||||
abandonedScope?.Dispose();
|
||||
Interlocked.Decrement(ref _abandonedExecutions);
|
||||
}, TaskScheduler.Default);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- In `ScadaBridgeTelemetry`, add `RecordInboundAbandonedExecution(string method)` incrementing a counter (`scadabridge.inbound_api.abandoned_executions`) with a bounded `method` tag, copying the existing `RecordInboundApiRequest` pattern.
|
||||
4. Run filter → **PASS**; run full project.
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundScriptExecutor.cs src/ZB.MOM.WW.ScadaBridge.Commons/Observability/ScadaBridgeTelemetry.cs tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/InboundScriptExecutorTests.cs && git commit -m "fix(inbound-api): defer DI-scope disposal to handler completion and count abandoned executions — closes the timeout use-after-dispose"`
|
||||
|
||||
### Task 5: Inbound error contract — machine-readable `code` field + SITE_UNREACHABLE + 415 for non-JSON bodies
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** 5 min
|
||||
**Parallelizable with:** 8-16, 17-22
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.InboundAPI/EndpointExtensions.cs` (all `Results.Json(new { error = … })` sites: lines 111-113, 153-155, 184-194, 212-214, 253-255)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundScriptExecutor.cs` (`InboundScriptResult` + catch blocks) (after Task 4)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.InboundAPI/RouteHelper.cs` (~line 172: typed exception)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundApiEndpointFilter.cs` (503/413 bodies) (after Task 6)
|
||||
- Modify: `docs/requirements/Component-InboundAPI.md` (error-code table, ~lines 91-97)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/EndpointExtensionsTests.cs`, `tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/EndpointContentTypeTests.cs` (append)
|
||||
|
||||
Spec documents `{"error": …, "code": "SITE_UNREACHABLE"}`; code emits `{error}` only, and routed site-unreachable failures collapse into a generic 500. Implement the spec (the audience is external integrators). Codes: `UNAUTHORIZED` (401), `NOT_APPROVED` (403 — one shared code for both indistinguishable negatives, preserving enumeration-safety), `INVALID_JSON`, `VALIDATION_FAILED`, `UNSUPPORTED_MEDIA_TYPE` (new 415), `BODY_TOO_LARGE` (413), `STANDBY_NODE` (503), `TIMEOUT`, `SITE_UNREACHABLE`, `SCRIPT_ERROR` (500 catch-all).
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Failing tests (follow the existing patterns in `EndpointExtensionsTests`/`EndpointContentTypeTests`, which drive the endpoint via a test host):
|
||||
- `ErrorBodies_CarryMachineReadableCode`: unauthorized request → body has `code == "UNAUTHORIZED"`; in-scope-but-invalid-JSON body → `code == "INVALID_JSON"`; missing required parameter → `code == "VALIDATION_FAILED"`.
|
||||
- `NonJsonContentType_WithBody_Returns415`: `POST` with `Content-Type: text/plain` and body `"hello"` → HTTP 415, `code == "UNSUPPORTED_MEDIA_TYPE"`. A request with a body and **no** Content-Type header still parses as JSON (lenient, preserves existing callers).
|
||||
- `RoutedSiteUnreachable_Returns500WithSiteUnreachableCode` (unit-level on the executor): stub `IInstanceRouter.RouteToCallAsync` to return `Success=false, ErrorMessage="Site unreachable"`; script does `await Route.To("X").Call("s")`; assert `result.ErrorCode == "SITE_UNREACHABLE"`.
|
||||
2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests --filter "ErrorBodies_CarryMachineReadableCode|NonJsonContentType_WithBody_Returns415|RoutedSiteUnreachable"` → **FAIL**.
|
||||
3. Implement:
|
||||
- `RouteHelper.cs`: add `public sealed class SiteUnreachableException : InvalidOperationException` and throw it (instead of bare `InvalidOperationException`) when `RouteToCallAsync` reports failure whose message indicates unreachable/no-contact (and from the site-resolution failure path).
|
||||
- `InboundScriptResult` gains `string? ErrorCode = null` (additive positional-with-default). Catch blocks: timeout → `"TIMEOUT"`; add `catch (SiteUnreachableException ex)` before the generic catch → `new InboundScriptResult(false, null, "Site unreachable", "SITE_UNREACHABLE")`; generic → `"SCRIPT_ERROR"`; compile failure → `"SCRIPT_COMPILE_FAILED"`.
|
||||
- `EndpointExtensions.cs`: add a local helper `static IResult Error(string code, string message, int status) => Results.Json(new { error = message, code }, statusCode: status);` and route every error return through it. Content-type branch: if `ContentLength > 0` and `ContentType` is non-empty and does not contain `"json"` (ordinal-ignore-case) → `Error("UNSUPPORTED_MEDIA_TYPE", "Content-Type must be application/json", StatusCodes.Status415UnsupportedMediaType)`. Script-failure branch passes `scriptResult.ErrorCode ?? "SCRIPT_ERROR"` through.
|
||||
- `InboundApiEndpointFilter.cs`: add `code = "STANDBY_NODE"` / `"BODY_TOO_LARGE"` to its two bodies.
|
||||
- Update the doc's failure-body examples/table to the implemented code set.
|
||||
4. Run filters → **PASS**; run the full InboundAPI test project (existing error-shape assertions that check only `error` still pass — the field is additive).
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.InboundAPI tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests docs/requirements/Component-InboundAPI.md && git commit -m "feat(inbound-api): machine-readable error codes per spec (incl. SITE_UNREACHABLE) + 415 for non-JSON bodies"`
|
||||
|
||||
### Task 6: `InboundApiEndpointFilter` hot-reads options via `IOptionsMonitor`
|
||||
|
||||
**Classification:** trivial
|
||||
**Estimated implement time:** 2 min
|
||||
**Parallelizable with:** 1, 2, 7, 8, 17, 18, 23, 24
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundApiEndpointFilter.cs` (lines 26-38, 66)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/EndpointGatingTests.cs` (append)
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Failing test: `BodySizeCap_HotReloads_WithoutRestart` — construct the filter with a test `IOptionsMonitor<InboundApiOptions>` whose `CurrentValue` is swapped between two invocations (cap 10 bytes → reject; cap 1 MiB → accept); assert the second request passes. (A minimal `TestOptionsMonitor<T>` stub: implements `CurrentValue` from a mutable field, `Get`→CurrentValue, `OnChange`→null.)
|
||||
2. Run filter test → **FAIL** (ctor takes `IOptions`).
|
||||
3. Change the ctor parameter to `IOptionsMonitor<InboundApiOptions> options`, store the monitor, and read `_options.CurrentValue.MaxRequestBodyBytes` per request (matches the sibling `AuditWriteMiddleware` convention).
|
||||
4. Run → **PASS**; `dotnet test tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests`.
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundApiEndpointFilter.cs tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/EndpointGatingTests.cs && git commit -m "fix(inbound-api): endpoint filter hot-reads MaxRequestBodyBytes via IOptionsMonitor"`
|
||||
|
||||
### Task 7: Parallelize startup compile of inbound methods
|
||||
|
||||
**Classification:** trivial
|
||||
**Estimated implement time:** 2 min
|
||||
**Parallelizable with:** 1, 2, 6, 8, 17, 18, 23, 24
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` (lines 383-387)
|
||||
|
||||
`CompileAndRegister` is safe for concurrent callers (ConcurrentDictionary caches, no shared mutable compile state). Replace the foreach with:
|
||||
|
||||
```csharp
|
||||
Parallel.ForEach(methods, method => executor.CompileAndRegister(method));
|
||||
```
|
||||
|
||||
**Steps:** No practical unit test (Host composition root; behavior identical, latency only). 1. Make the edit with a comment citing the review finding (serial Roslyn compiles stretch the failover-recovery window at hundreds of methods). 2. `dotnet build ZB.MOM.WW.ScadaBridge.slnx` → green. 3. `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests` → green. 4. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Host/Program.cs && git commit -m "perf(host): compile inbound API methods in parallel at startup"`
|
||||
|
||||
### Task 8: `ExternalSystemDefinition.TimeoutSeconds` — entity + EF mapping + migration
|
||||
|
||||
**Classification:** high-risk (schema migration)
|
||||
**Estimated implement time:** 5 min
|
||||
**Parallelizable with:** 1, 2, 6, 7, 17, 23, 24
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Entities/ExternalSystems/ExternalSystemDefinition.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/ExternalSystemConfiguration.cs`
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/<timestamp>_AddExternalSystemTimeoutSeconds.cs` (+ Designer, + snapshot update — generated)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Commons.Tests` (entity default), `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests` (mapping round-trip, follow the existing external-system repository test pattern)
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Failing test (Commons.Tests): `ExternalSystemDefinition_TimeoutSeconds_DefaultsToZeroMeaningUseGatewayDefault` — construct the entity, assert `TimeoutSeconds == 0`. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter ExternalSystemDefinition_TimeoutSeconds` → **FAIL** (no property).
|
||||
2. Add to the entity:
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Per-system HTTP round-trip timeout in seconds for all method calls
|
||||
/// (spec: Component-ExternalSystemGateway "Timeout"). 0 = unset — the gateway's
|
||||
/// configured DefaultHttpTimeout applies.
|
||||
/// </summary>
|
||||
public int TimeoutSeconds { get; set; }
|
||||
```
|
||||
|
||||
3. EF mapping: in `ExternalSystemConfiguration`, add `builder.Property(e => e.TimeoutSeconds).HasDefaultValue(0);`.
|
||||
4. Generate the migration — **build first**: `dotnet build ZB.MOM.WW.ScadaBridge.slnx` then `dotnet ef migrations add AddExternalSystemTimeoutSeconds --project src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase --startup-project src/ZB.MOM.WW.ScadaBridge.Host`. Open the generated file and verify it is NOT empty (one `AddColumn<int>` on the external-systems table, default 0); if empty, delete the generated files, rebuild, regenerate.
|
||||
5. Run tests: the Commons filter → **PASS**; `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests` → green.
|
||||
6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Commons/Entities/ExternalSystems/ExternalSystemDefinition.cs src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase && git add tests/ZB.MOM.WW.ScadaBridge.Commons.Tests && git commit -m "feat(esg): per-system TimeoutSeconds column on ExternalSystemDefinition (spec Component-ExternalSystemGateway Timeout) — entity + EF + migration"`
|
||||
|
||||
### Task 9: Honor the per-system timeout in `ExternalSystemClient.InvokeHttpAsync`
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** 4 min
|
||||
**Parallelizable with:** 10, 11, 12 (Files disjoint)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemClient.cs` (lines 300-307, 322-323, 345-346)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs` (append)
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Failing test (use the project's existing stub-handler `HttpClient` pattern):
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task PerSystemTimeout_OverridesGatewayDefault()
|
||||
{
|
||||
// System timeout 1s; handler black-holes for 5s; gateway default is 30s.
|
||||
var system = new ExternalSystemDefinition("slow", "http://x", "none") { TimeoutSeconds = 1 };
|
||||
var method = new ExternalSystemMethod("m", "GET", "/p");
|
||||
SetupRepository(system, method); // existing helper pattern
|
||||
var handler = new DelegateHandler(async (_, ct) =>
|
||||
{
|
||||
await Task.Delay(TimeSpan.FromSeconds(5), ct);
|
||||
return new HttpResponseMessage(HttpStatusCode.OK);
|
||||
});
|
||||
_httpClientFactory.CreateClient(Arg.Any<string>()).Returns(new HttpClient(handler));
|
||||
var client = new ExternalSystemClient(_httpClientFactory, _repository, NullLogger<ExternalSystemClient>.Instance);
|
||||
|
||||
var sw = System.Diagnostics.Stopwatch.StartNew();
|
||||
var result = await client.CallAsync("slow", "m");
|
||||
sw.Stop();
|
||||
|
||||
Assert.False(result.Success);
|
||||
Assert.Contains("Timeout", result.ErrorMessage);
|
||||
Assert.True(sw.Elapsed < TimeSpan.FromSeconds(4), "per-system 1s timeout must apply, not the 30s default");
|
||||
}
|
||||
```
|
||||
|
||||
2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests --filter PerSystemTimeout_OverridesGatewayDefault` → **FAIL** (~30s default would apply; the test waits 5s then sees OK → success).
|
||||
3. Implement: before the CTS creation, `var effectiveTimeout = system.TimeoutSeconds > 0 ? TimeSpan.FromSeconds(system.TimeoutSeconds) : _options.DefaultHttpTimeout;` — use `effectiveTimeout` in `new CancellationTokenSource(effectiveTimeout)` and in both timeout error messages (replacing `_options.DefaultHttpTimeout.TotalSeconds`). Update the stale comment block at lines 300-305 ("has no per-system Timeout field yet") to describe the new resolution order.
|
||||
4. Run → **PASS**; full project run.
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemClient.cs tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs && git commit -m "feat(esg): honor per-system TimeoutSeconds in InvokeHttpAsync (falls back to DefaultHttpTimeout)"`
|
||||
|
||||
### Task 10: Carry TimeoutSeconds through the artifact pipeline to sites
|
||||
|
||||
**Classification:** high-risk (message contract + site SQLite schema)
|
||||
**Estimated implement time:** 5 min
|
||||
**Parallelizable with:** 9, 11, 12
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Artifacts/ExternalSystemArtifact.cs` (additive trailing param `int TimeoutSeconds = 0`)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/ArtifactDeploymentService.cs` (~line 194: pass `es.TimeoutSeconds`)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs` (`MigrateSchemaAsync`: `TryAddColumnAsync(connection, "external_systems", "timeout_seconds", "INTEGER NOT NULL DEFAULT 0")`; `StoreExternalSystemAsync`: new `int timeoutSeconds` param + column in the upsert)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs` (~line 1533: pass `es.TimeoutSeconds`)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/SiteExternalSystemRepository.cs` (`MapExternalSystem` + its SELECTs: read `timeout_seconds` into `TimeoutSeconds`)
|
||||
- Check/Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/SiteReplicationActor.cs` (it re-applies `ReplicateArtifacts` — verify its `StoreExternalSystemAsync` call site compiles with the new parameter)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests` (site storage round-trip test — follow the existing `StoreExternalSystemAsync` test if present, else add `SiteStorage_ExternalSystem_TimeoutSeconds_RoundTrips`)
|
||||
|
||||
The message contract change is **additive** (trailing positional with default), per the repo's additive-only evolution rule — an old central sending to a new site yields 0 (= use default), and vice versa deserializes cleanly.
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Failing test (SiteRuntime.Tests, in-memory/temp-file SQLite): store an external system with `timeoutSeconds: 7` via `SiteStorageService.StoreExternalSystemAsync`, read it back via `SiteExternalSystemRepository.GetExternalSystemByNameAsync`, assert `TimeoutSeconds == 7`. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter TimeoutSeconds_RoundTrips` → **FAIL** (compile error / missing column).
|
||||
2. Make the six edits above: add the artifact record's trailing `int TimeoutSeconds = 0`; thread it central-build → site-store → site-read. The SQLite upsert gains `timeout_seconds = excluded.timeout_seconds`.
|
||||
3. Run → **PASS**; then `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests` and `dotnet test tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests` (artifact-build assertions may enumerate record fields — update).
|
||||
4. `dotnet build ZB.MOM.WW.ScadaBridge.slnx` → green (catches any other `ExternalSystemArtifact` construction site).
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Artifacts/ExternalSystemArtifact.cs src/ZB.MOM.WW.ScadaBridge.DeploymentManager/ArtifactDeploymentService.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime tests/ && git commit -m "feat(esg): TimeoutSeconds travels the artifact pipeline (additive contract) into site SQLite so site-side calls honor it"`
|
||||
|
||||
### Task 11: TimeoutSeconds management commands + CLI
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** 4 min
|
||||
**Parallelizable with:** 9, 10, 12
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Management/ExternalSystemCommands.cs` (Create/Update records: trailing `int? TimeoutSeconds = null`)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (`HandleCreateExternalSystem` ~1599, `HandleUpdateExternalSystem` ~1612: apply `cmd.TimeoutSeconds ?? 0` / preserve-on-null semantics for update)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.CLI/Commands/ExternalSystemCommands.cs` (create/update: `--timeout-seconds` `Option<int?>`)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests` (append to the existing external-system handler tests); `tests/ZB.MOM.WW.ScadaBridge.CLI.Tests` (option parse test — **note: CLI.Tests is not in the slnx; test the csproj directly**)
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Failing test (ManagementService.Tests): `CreateExternalSystem_PersistsTimeoutSeconds` — send `CreateExternalSystemCommand(..., TimeoutSeconds: 15)`, assert the persisted definition has `TimeoutSeconds == 15`; and `UpdateExternalSystem_NullTimeout_PreservesExisting`. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests --filter TimeoutSeconds` → **FAIL**.
|
||||
2. Implement: additive record params; create handler sets `TimeoutSeconds = cmd.TimeoutSeconds ?? 0`; update handler `if (cmd.TimeoutSeconds is { } ts) def.TimeoutSeconds = ts;`. CLI: add the option to both `create` and `update`, pass through, add `--timeout-seconds` to the table projection if the list output enumerates fields.
|
||||
3. Run → **PASS**; run `dotnet test tests/ZB.MOM.WW.ScadaBridge.CLI.Tests` **directly** (not via slnx).
|
||||
4. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Management/ExternalSystemCommands.cs src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs src/ZB.MOM.WW.ScadaBridge.CLI tests/ && git commit -m "feat(esg): --timeout-seconds on external-system create/update (management command + CLI, additive contract)"`
|
||||
|
||||
### Task 12: TimeoutSeconds in Central UI form + spec doc sync
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** 3 min
|
||||
**Parallelizable with:** 9, 10, 11
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Design/ExternalSystemForm.razor` (numeric input next to the existing `_maxRetries`/`_retryDelaySeconds` fields; load in `OnInitializedAsync`, save in both create/update branches)
|
||||
- Modify: `docs/requirements/Component-ExternalSystemGateway.md` (line 38 area + §"Call Timeout & Error Handling" ~line 99: document per-system `TimeoutSeconds` with 0 = gateway `DefaultHttpTimeout` fallback and that it now ships to sites in the artifact)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests` (bUnit — append to the existing ExternalSystemForm test if one exists; otherwise assert the rendered form contains the `Timeout (seconds)` field)
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Failing bUnit test: render `ExternalSystemForm` (existing test scaffolding), assert a labeled `Timeout (seconds)` input exists and that saving with value 20 passes `TimeoutSeconds == 20` to the repository/command. Run → **FAIL**.
|
||||
2. Implement: `private int _timeoutSeconds;` bound field with helper text "0 = use gateway default"; wire load + both save paths (the form writes the entity directly per the existing pattern). Update the doc.
|
||||
3. Run → **PASS**; `dotnet test tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests`.
|
||||
4. Commit: `git add src/ZB.MOM.WW.ScadaBridge.CentralUI docs/requirements/Component-ExternalSystemGateway.md tests/ && git commit -m "feat(esg): TimeoutSeconds in ExternalSystem UI form + Component-ExternalSystemGateway doc sync — closes the spec-drift gap"`
|
||||
|
||||
### Task 13: Path-template substitution (`/recipes/{id}`) in `BuildUrl`
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** 5 min
|
||||
**Parallelizable with:** 10, 11, 12 (NOT 9/14/15/16 — same file; run after 9)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemClient.cs` (`BuildUrl` lines 431-463, `InvokeHttpAsync` body-serialization branch lines 286-298)
|
||||
- Modify: `docs/requirements/Component-ExternalSystemGateway.md` (path/method-definition section: document `{param}` substitution + consumed-parameter removal + missing-parameter error)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs` (append)
|
||||
|
||||
Design: `BuildUrl` becomes `BuildUrl(baseUrl, path, parameters, httpMethod, out ISet<string> consumedParams)` (or returns a tuple). Regex `\{([A-Za-z_][A-Za-z0-9_]*)\}` over the trimmed path: each placeholder is replaced with `Uri.EscapeDataString(value?.ToString())` from `parameters` and its name recorded as consumed; a placeholder with no matching parameter (or a null value) throws `ArgumentException("Path template parameter '{id}' has no value …")` — a clear authoring error, mirroring `ValidateHttpMethod`. Consumed names are excluded from the GET/DELETE query string AND from the POST/PUT/PATCH JSON body.
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Failing tests:
|
||||
- `PathTemplate_SubstitutesAndConsumesParameter`: method path `/recipes/{id}`, GET, parameters `{ id = "R 1", filter = "x" }` → captured request URI is `.../recipes/R%201?filter=x` (no `id` in the query).
|
||||
- `PathTemplate_Post_ConsumedParamExcludedFromBody`: POST `/orders/{orderId}/lines`, parameters `{ orderId = 5, qty = 2 }` → body JSON contains `qty` but not `orderId`.
|
||||
- `PathTemplate_MissingParameter_ThrowsClearError`: GET `/recipes/{id}` with no `id` → `ArgumentException` naming `id`.
|
||||
2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests --filter PathTemplate` → **FAIL** (URL contains literal `{id}`).
|
||||
3. Implement as designed (a `private static readonly Regex PathParamRegex` + substitution loop; thread `consumedParams` into the query-string LINQ filter and into a `parameters.Where(p => !consumed.Contains(p.Key))` projection before body serialization).
|
||||
4. Run → **PASS**; full project run.
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemClient.cs docs/requirements/Component-ExternalSystemGateway.md tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs && git commit -m "feat(esg): {param} path-template substitution with consumed-param removal — implements the spec'd /recipes/{id} form"`
|
||||
|
||||
### Task 14: Bound outbound HTTP response buffering
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** 5 min
|
||||
**Parallelizable with:** 10, 11, 12 (same-file chain: after 13)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemClient.cs` (line 312 `SendAsync`, lines 334-347 body read)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemGatewayOptions.cs` (add `MaxResponseBodyBytes`, default `10 * 1024 * 1024`)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs` (append)
|
||||
|
||||
Design: `SendAsync(request, HttpCompletionOption.ResponseHeadersRead, linkedCts.Token)`; pre-check `Content-Length` header when present (fail fast); then a bounded read (cap+1 pattern): stream into a `MemoryStream` in 16 KiB chunks, aborting once `MaxResponseBodyBytes` is exceeded. Oversize → `PermanentExternalSystemException` ("response exceeded N bytes" — retrying won't shrink it; buffering it into S&F would defeat the cap).
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Failing test: `OversizedResponseBody_FailsPermanently_WithoutBufferingWholeBody` — options with `MaxResponseBodyBytes = 1024`; stub handler returns 2 MiB of `'x'`; `CallAsync` → `result.Success == false`, error contains `"exceeded"`; and a same-shape `CachedCallAsync` does NOT enqueue to S&F (assert via the existing S&F test double if the suite has one, else assert `WasBuffered == false`).
|
||||
2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests --filter OversizedResponseBody` → **FAIL** (full body read succeeds).
|
||||
3. Implement: add the option; replace `ReadAsStringAsync` with a private `ReadBodyBoundedAsync(HttpContent content, long maxBytes, CancellationToken token)` returning `string` or throwing `PermanentExternalSystemException`; keep the existing ordered OCE catch filters around the read.
|
||||
4. Run → **PASS**; full project run.
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs && git commit -m "perf(esg): bound outbound response bodies (ResponseHeadersRead + MaxResponseBodyBytes cap; oversize = permanent failure)"`
|
||||
|
||||
### Task 15: Structured JSON `AuthConfiguration` with legacy colon fallback
|
||||
|
||||
**Classification:** high-risk (auth parsing; existing rows must keep working)
|
||||
**Estimated implement time:** 5 min
|
||||
**Parallelizable with:** 10, 11, 12 (same-file chain: after 14)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemClient.cs` (`ApplyAuth`, lines 465-533)
|
||||
- Modify: `docs/requirements/Component-ExternalSystemGateway.md` (auth-configuration format section)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs` (append)
|
||||
|
||||
The entity doc-comment ("JSON-serialized authentication configuration") and the UI placeholders (`{"key":"xyz"}` / `{"username":"u","password":"p"}`) already promise JSON — but `ApplyAuth` colon-splits, so a JSON-shaped config is silently misparsed **today** (a `{"key":"xyz"}` apikey config produces header name `{"key"`), and any API key containing `:` breaks. Parse order per branch: trimmed value starting with `{` → JSON (`apikey`: `{"header"?: string, "key": string}`, default header `X-API-Key`; `basic`: `{"username": string, "password": string}`); otherwise the legacy colon-split/bare-key behavior, unchanged. Malformed JSON → the existing "malformed config" warning path (send without header), never an exception.
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Failing tests:
|
||||
- `ApiKeyAuth_JsonConfig_SetsConfiguredHeaderAndKeepsColons`: `AuthConfiguration = "{\"header\":\"X-Auth\",\"key\":\"ab:cd:ef\"}"` → captured request has header `X-Auth: ab:cd:ef`.
|
||||
- `ApiKeyAuth_JsonConfig_DefaultHeader`: `{"key":"xyz"}` → `X-API-Key: xyz` (this is the UI-placeholder shape that is broken today).
|
||||
- `BasicAuth_JsonConfig_EncodesUsernamePassword`: `{"username":"u","password":"p:w"}` → `Authorization: Basic` of `u:p:w`.
|
||||
- `ApiKeyAuth_LegacyColonConfig_StillWorks`: `"X-Custom:secret"` → `X-Custom: secret` (backward compatibility pinned).
|
||||
2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests --filter "JsonConfig|LegacyColonConfig"` → **FAIL**.
|
||||
3. Implement a private `static bool TryParseJsonAuth(string config, out Dictionary<string, string?> fields)` (System.Text.Json, case-insensitive property names, catch `JsonException` → false) and branch `apikey`/`basic` on it before the legacy split. Log a warning (values never logged) when JSON parsing yields no usable `key`/`username`. Update the doc section with both accepted formats and the deprecation note on the colon form.
|
||||
4. Run → **PASS**; full project run.
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemClient.cs docs/requirements/Component-ExternalSystemGateway.md tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs && git commit -m "fix(esg): structured JSON AuthConfiguration (matches entity doc + UI placeholders) with legacy colon fallback — keys containing ':' no longer misparse"`
|
||||
|
||||
### Task 16: Cancellation-aware `ErrorClassifier.IsTransient` overload
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** 3 min
|
||||
**Parallelizable with:** 10, 11, 12, 17-24 (same-file chain: after 15)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ErrorClassifier.cs` (lines 27-38)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemClient.cs` (line 325 call site → pass the caller token)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ErrorClassifierTests.cs` (append)
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Failing test: `IsTransient_CallerCancelledOce_IsNotTransient` — `var cts = new CancellationTokenSource(); cts.Cancel(); Assert.False(ErrorClassifier.IsTransient(new OperationCanceledException(), cts.Token)); Assert.True(ErrorClassifier.IsTransient(new OperationCanceledException(), CancellationToken.None));` → **FAIL** (no overload).
|
||||
2. Implement (mirrors `SmtpErrorClassifier`/`SmsErrorClassifier`):
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Cancellation-aware overload: a caller-requested cancellation is NOT transient
|
||||
/// (buffering caller-abandoned work into S&F would be wrong). Prefer this overload
|
||||
/// at call sites that hold the caller's token.
|
||||
/// </summary>
|
||||
public static bool IsTransient(Exception exception, CancellationToken cancellationToken)
|
||||
{
|
||||
if (exception is OperationCanceledException && cancellationToken.IsCancellationRequested)
|
||||
return false;
|
||||
return IsTransient(exception);
|
||||
}
|
||||
```
|
||||
|
||||
Switch the `catch (Exception ex) when (ErrorClassifier.IsTransient(ex))` filter in `InvokeHttpAsync` to `ErrorClassifier.IsTransient(ex, cancellationToken)` (behavior identical — the ordered filters already peel caller-cancel — but the predicate is now misuse-resistant); add a doc-comment on the old overload pointing to the new one.
|
||||
3. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests --filter IsTransient_CallerCancelledOce` → **PASS**; full project run.
|
||||
4. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ErrorClassifierTests.cs && git commit -m "fix(esg): cancellation-aware ErrorClassifier.IsTransient overload — caller-cancelled work can never classify transient"`
|
||||
|
||||
### Task 17: SMS first-transient short-circuit (duplicate amplification + sweep bound)
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** 4 min
|
||||
**Parallelizable with:** 1-16, 19, 22, 23, 24
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/Delivery/SmsNotificationDeliveryAdapter.cs` (recipient loop, lines 148-172; `RollUp` doc-comment)
|
||||
- Modify: `docs/requirements/Component-NotificationService.md` (the re-send-to-all note ~line 102: add the short-circuit)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests/Delivery/` (append to the existing SMS adapter test file)
|
||||
|
||||
Any transient rolls the whole notification up to retry-all, so every recipient accepted *after* the first transient is a guaranteed duplicate text on retry. Breaking the loop on the first transient costs nothing (retry re-attempts everyone) and — because a black-holed endpoint's first request eats the full per-request timeout — also bounds the worst-case sweep occupancy to ~one timeout instead of recipients × timeout, which is the substance of the P2 performance finding at current scale. Bounded parallelism is deferred (see coverage table).
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Failing test: `MidListTransient_ShortCircuits_RemainingRecipientsNotAttempted` — list of 3 numbers; stub handler: first number → 500 (transient), and a request counter. Assert `DeliveryOutcome.Result == Transient` and **exactly 1** HTTP request was made (current code makes 3).
|
||||
2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests --filter MidListTransient_ShortCircuits` → **FAIL** (3 requests).
|
||||
3. Implement: in the `switch`, the `Transient` case sets `firstTransientError ??= attempt.Detail;` then `break`s out of the `foreach` (convert to `goto` -free form: set a flag and `break` the loop after the switch, or restructure to `if/else if`). Add the rationale comment: *"every recipient accepted after the first transient is a guaranteed duplicate on retry (the whole notification re-sends); stopping here also bounds a black-holed endpoint to one timeout per sweep."* Update `RollUp`'s doc-comment and the component doc line.
|
||||
4. Run → **PASS**; full `dotnet test tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests`.
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/Delivery/SmsNotificationDeliveryAdapter.cs docs/requirements/Component-NotificationService.md tests/ && git commit -m "fix(notifications): SMS adapter short-circuits on first transient — kills duplicate amplification and bounds the dispatch sweep"`
|
||||
|
||||
### Task 18: Honor `SmsConfiguration.MaxRetries`/`RetryDelay` — per-type retry policy
|
||||
|
||||
**Classification:** high-risk (behavioral change to dispatch/parking)
|
||||
**Estimated implement time:** 5 min
|
||||
**Parallelizable with:** 1-16, 23, 24 (NOT 17/21 — shared test dirs are fine, but 21 edits the same actor file: run 18 before 21)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/NotificationOutboxActor.cs` (`RunDispatchSweepAsync` ~line 319, `DeliverOneAsync` signature, `ResolveRetryPolicyAsync` lines 370-401)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Entities/Notifications/SmsConfiguration.cs` (lines 28-45: delete the RESERVED doc-comments, document the live behavior)
|
||||
- Modify: `docs/requirements/Component-NotificationService.md` (~line 65 deferred note), `CLAUDE.md` (the "Notification Outbox retry reuses central SMTP max-retry-count and fixed interval" decision line → "…for Email; Sms uses SmsConfiguration.MaxRetries/RetryDelay, falling back to the SMTP policy when unset")
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests/NotificationOutboxActorDispatchTests.cs` (append)
|
||||
|
||||
Design: `ResolveRetryPolicyAsync` becomes `ResolveRetryPoliciesAsync` returning a small `RetryPolicies(SmtpPolicy, SmsPolicy)` record, resolved once per sweep: the SMTP policy exactly as today; the SMS policy from `INotificationRepository.GetSmsConfigurationAsync()` with the same non-positive-clamp logic, falling back to the SMTP policy when no SMS config row exists. `DeliverOneAsync` selects by `notification.Type` (`Sms` → SMS policy, everything else → SMTP policy). This is plan 08's flagged dead-field finding, owned here.
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Failing test (follow the existing dispatch-test scaffolding, which drives the actor with substitute repositories): `SmsNotification_TransientFailure_UsesSmsRetryPolicy` — SMTP config `MaxRetries = 10`; SMS config `MaxRetries = 1, RetryDelay = 5 min`; an SMS notification with `RetryCount = 1` and a transient adapter outcome → row is **Parked** (SMS policy exhausted) not Retrying; a parallel `EmailNotification_StillUsesSmtpPolicy` asserts an email row with `RetryCount = 1` goes to Retrying with the SMTP delay.
|
||||
2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests --filter "UsesSmsRetryPolicy|StillUsesSmtpPolicy"` → **FAIL** (SMS row retries under SMTP policy).
|
||||
3. Implement as designed; keep the clamp warnings (parameterized by which config produced them). Update the entity doc-comments, component doc, and the CLAUDE.md decision line in the same commit (docs travel with code).
|
||||
4. Run → **PASS**; full NotificationOutbox.Tests run.
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/NotificationOutboxActor.cs src/ZB.MOM.WW.ScadaBridge.Commons/Entities/Notifications/SmsConfiguration.cs docs/requirements/Component-NotificationService.md CLAUDE.md tests/ && git commit -m "feat(notifications): SMS notifications retry under SmsConfiguration.MaxRetries/RetryDelay (SMTP policy remains the Email + fallback policy) — wires the dead fields"`
|
||||
|
||||
### Task 19: Configurable OAuth2 authority/scope — entity + EF + migration + token service
|
||||
|
||||
**Classification:** high-risk (schema migration + auth flow)
|
||||
**Estimated implement time:** 5 min
|
||||
**Parallelizable with:** 1-7, 17, 23, 24 (**migration serialization: run after Task 8** — both regenerate the EF model snapshot)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Entities/Notifications/SmtpConfiguration.cs` (add `string? OAuth2Authority`, `string? OAuth2Scope`)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/NotificationConfiguration.cs` (map both nullable columns)
|
||||
- Create: migration `AddSmtpOAuth2AuthorityScope` (same command pattern as Task 8; **build first**, verify non-empty)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.NotificationService/OAuth2TokenService.cs` (`GetTokenAsync` gains optional `string? authority = null, string? scope = null`; defaults preserve today's M365 URL/scope; cache key incorporates authority+scope so two configs with the same credentials but different authorities never share a token)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/Delivery/EmailNotificationDeliveryAdapter.cs` (line 188: pass `smtpConfig.OAuth2Authority`, `smtpConfig.OAuth2Scope`)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.NotificationService.Tests` (append to the OAuth2 token service tests)
|
||||
|
||||
Authority template: the stored value is the full token-endpoint URL with an optional `{tenant}` placeholder (e.g. `https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token`); null → today's hardcoded M365 endpoint. Scope: null → `https://outlook.office365.com/.default`.
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Failing tests: `GetTokenAsync_CustomAuthorityAndScope_PostsToConfiguredEndpoint` — stub HTTP handler captures the request; call with `authority: "https://idp.example/{tenant}/token", scope: "smtp.send"`; assert the POST went to `https://idp.example/T1/token` with form `scope=smtp.send`. And `GetTokenAsync_DefaultsPreserved_M365` (no authority/scope → today's URL + scope). Run filter → **FAIL** (no parameters).
|
||||
2. Implement entity + EF + migration (verify non-empty), then the token-service parameters and the adapter call site; cache key: `CredentialKey($"{credentials}|{authority}|{scope}")`.
|
||||
3. Run → **PASS**; `dotnet test tests/ZB.MOM.WW.ScadaBridge.NotificationService.Tests` and `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests`.
|
||||
4. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Commons/Entities/Notifications/SmtpConfiguration.cs src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase src/ZB.MOM.WW.ScadaBridge.NotificationService/OAuth2TokenService.cs src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/Delivery/EmailNotificationDeliveryAdapter.cs tests/ && git commit -m "feat(notifications): configurable OAuth2 authority + scope per SMTP configuration (defaults preserve M365) — entity + EF + migration + token service"`
|
||||
|
||||
### Task 20: OAuth2 authority/scope surfaces — management command + CLI + UI + doc
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** 4 min
|
||||
**Parallelizable with:** 9-16, 21, 22
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Management/NotificationCommands.cs` (`UpdateSmtpConfigCommand`: trailing `string? OAuth2Authority = null, string? OAuth2Scope = null`)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (the SMTP update handler: apply when non-null)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.CLI/Commands/NotificationCommands.cs` (`smtp update`: `--oauth2-authority`, `--oauth2-scope` options, ~line 127+ group)
|
||||
- Modify: the Central UI SMTP page (`src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Notifications/` — the SMTP form component; two text inputs shown only when AuthType is OAuth2)
|
||||
- Modify: `docs/requirements/Component-NotificationService.md` (~line 48: the "and other modern SMTP providers" sentence now true — document the two fields and their M365 defaults)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests` (handler applies fields), `tests/ZB.MOM.WW.ScadaBridge.CLI.Tests` (option parse — test csproj directly)
|
||||
|
||||
**Steps:** 1. Failing handler test `UpdateSmtpConfig_AppliesOAuth2AuthorityAndScope` → run filter → **FAIL**. 2. Implement the additive command params, handler application (preserve-on-null), CLI options, UI inputs, doc paragraph. 3. Run filters → **PASS**; run ManagementService.Tests + CLI.Tests (direct). 4. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.Commons/Messages src/ZB.MOM.WW.ScadaBridge.ManagementService src/ZB.MOM.WW.ScadaBridge.CLI src/ZB.MOM.WW.ScadaBridge.CentralUI docs/requirements/Component-NotificationService.md tests/ && git commit -m "feat(notifications): OAuth2 authority/scope management + CLI + UI surfaces"`
|
||||
|
||||
### Task 21: Notification low-severity cleanups — deterministic SMTP selection + OAuth2 cache visibility
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** 4 min
|
||||
**Parallelizable with:** 9-16, 20, 22 (after 18 and 19 — shared files)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/Delivery/EmailNotificationDeliveryAdapter.cs` (lines 75-76)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/NotificationOutboxActor.cs` (retry-policy resolution — post-Task-18 shape)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.NotificationService/OAuth2TokenService.cs` (`CacheEntry`)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests/Delivery/` (append)
|
||||
|
||||
Concrete edits (S7, S8):
|
||||
1. **Deterministic SMTP pick**: in both call sites replace `FirstOrDefault()` / `configurations[0]` with `configurations.OrderBy(c => c.Id).FirstOrDefault()` and, when `configurations.Count > 1`, log a warning: `"Multiple SMTP configurations exist; using the lowest-Id row ({Id}). Enforce a single row or delete extras."` Failing test first: `MultipleSmtpConfigs_LowestIdWins` — repository returns rows Id 7 then Id 3 (that order); assert the adapter connects with row 3's server. Run → **FAIL** (row 7 picked) → implement → **PASS**.
|
||||
2. **OAuth2 cache visibility**: replace `CacheEntry`'s mutable `Token`/`Expiry` fields with a single `volatile TokenValue? Current;` where `private sealed record TokenValue(string Token, DateTimeOffset Expiry);` — reads outside the lock see a consistent pair; writes inside the lock assign one reference. (Growth stays bounded by distinct credential triples — accepted per report; no eviction added.) Existing OAuth2 tests must stay green.
|
||||
|
||||
Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests` and `tests/ZB.MOM.WW.ScadaBridge.NotificationService.Tests`. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.NotificationOutbox src/ZB.MOM.WW.ScadaBridge.NotificationService tests/ && git commit -m "fix(notifications): deterministic lowest-Id SMTP config selection (+ multi-row warning); atomic OAuth2 token cache entry"`
|
||||
|
||||
### Task 22: Validate Twilio `AccountSid` format at config save
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** 3 min
|
||||
**Parallelizable with:** 9, 10, 12, 13-17, 20, 21, 23, 24 (after 11 — same `ManagementActor.cs`)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (the `UpdateSmsConfig` handler: reject a non-matching SID)
|
||||
- Modify: the Central UI SMS form (`/notifications/sms` page component: same client-side validation message)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests` (append)
|
||||
|
||||
`AccountSid` is interpolated un-escaped into the Twilio request URI (`SmsNotificationDeliveryAdapter.cs:131`); a validation regex at save time closes the door on URI-injection via a malformed SID.
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Failing test: `UpdateSmsConfig_MalformedAccountSid_Rejected` — send the SMS-config update command with `AccountSid = "../evil?x="`; assert `ManagementCommandException` containing `"Account SID"`; and `UpdateSmsConfig_ValidSid_Accepted` with `"AC" + new string('a', 32)`.
|
||||
2. Run filter → **FAIL**.
|
||||
3. Implement in the handler, before persisting: `if (!System.Text.RegularExpressions.Regex.IsMatch(cmd.AccountSid, "^AC[0-9a-fA-F]{32}$")) throw new ManagementCommandException("Account SID must match Twilio's format: 'AC' followed by 32 hex characters.");` Mirror the message in the UI form's validation.
|
||||
4. Run → **PASS**; run ManagementService.Tests.
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs src/ZB.MOM.WW.ScadaBridge.CentralUI tests/ && git commit -m "fix(notifications): validate Twilio AccountSid format at save — closes the un-escaped URI interpolation door"`
|
||||
|
||||
### Task 23: Document DelmiaNotifier at-least-once semantics + idempotency requirement
|
||||
|
||||
**Classification:** small (documentation)
|
||||
**Estimated implement time:** 4 min
|
||||
**Parallelizable with:** 1-22
|
||||
**Files:**
|
||||
- Modify: `docs/plans/2026-06-26-delmia-recipe-notifier-design.md` (after the failover table, ~line 100)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md` (operator-facing note)
|
||||
- Modify: `docs/requirements/Component-InboundAPI.md` (authoring note for the `DelmiaRecipeDownload` method)
|
||||
|
||||
The behavior is by design (timeout → failover), so no code change — but the consequence (at-least-once: a connection reset *after* the server processed the POST, or a slow response exceeding `TimeoutSeconds`, both roll over to the next URL and re-POST) is nowhere stated, and the deployed inbound method carries the burden.
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Add a "Delivery semantics: at-least-once" section to the design doc: enumerate the two duplicate-producing cases; state the requirement — *the `DelmiaRecipeDownload` inbound method MUST be idempotent on `(MachineCode, DownloadPath, WorkOrderNumber)`; a duplicate must return the same `Result`/`ResultText` as the original, not double-apply side effects.* Cross-link from the failover table row for timeout.
|
||||
2. Mirror one paragraph in the DelmiaNotifier README (`## Delivery semantics`) and add the authoring note to `Component-InboundAPI.md` where the method is referenced.
|
||||
3. Manual verification step (record the outcome in the commit message): fetch the deployed `DelmiaRecipeDownload` script (`scadabridge api-method get` via the CLI against the target environment, or read the row) and confirm it is idempotent on the natural key; if it is not, file/fix in the environment — this plan only mandates and documents the contract.
|
||||
4. `grep -n "at-least-once" docs/plans/2026-06-26-delmia-recipe-notifier-design.md src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md` → both hit.
|
||||
5. Commit: `git add docs/plans/2026-06-26-delmia-recipe-notifier-design.md src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md docs/requirements/Component-InboundAPI.md && git commit -m "docs(delmia): document at-least-once failover semantics + DelmiaRecipeDownload idempotency requirement"`
|
||||
|
||||
### Task 24: Document the dual-key API-key rotation procedure
|
||||
|
||||
**Classification:** trivial (documentation)
|
||||
**Estimated implement time:** 3 min
|
||||
**Parallelizable with:** 1, 3-22 (after 2 and 5 — same doc file)
|
||||
**Files:**
|
||||
- Modify: `docs/requirements/Component-InboundAPI.md` (new "### Key rotation" subsection under the API-key auth section, ~line 62+)
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Add the procedure the `sbk_<keyId>_<secret>` design already supports: (1) create a new key with the same scope set (`apikey create`), (2) deliver the new token to the caller and confirm cutover (the audit log's Actor field shows which key serves traffic), (3) disable — don't delete — the old key (`apikey update --enabled false`; reversible), (4) delete after a quarantine period. Note that scopes are copied at creation (scope drift between the pair is an operator error) and that both keys verifying simultaneously is the intended overlap window.
|
||||
2. Verify cross-references: the CLI command names used must match `src/ZB.MOM.WW.ScadaBridge.CLI/Commands/SecurityCommands.cs` (adjust wording to the real verbs).
|
||||
3. Commit: `git add docs/requirements/Component-InboundAPI.md && git commit -m "docs(inbound-api): dual-key API-key rotation procedure (create -> migrate -> disable -> delete)"`
|
||||
|
||||
---
|
||||
|
||||
## Dependencies on other plans
|
||||
|
||||
- **Plan 05 (Templates/Deployment/Transport)** designs the general "script artifact changed → invalidate everywhere" contract and owns the **bundle-import** side of cache invalidation. This plan's Task 3 provides the Inbound API consumer entry point (`InboundScriptExecutor.InvalidateMethod`) that plan 05's import path must call for every overwritten `ApiMethod`. **Deliberate resilience:** Task 1's per-request revision check is the correctness fallback — inbound methods heal even if an invalidation call is missed or plan 05 lands later, so there is no hard sequencing requirement in either direction.
|
||||
- **Plan 04 (Data/Audit backbone)** owns NotificationOutbox dispatcher/audit-row gaps. Tasks 17/18/21 touch `NotificationOutboxActor.cs` and the delivery adapters — if plan 04 is executed concurrently, coordinate on that file (this plan's edits are confined to `RunDispatchSweepAsync`'s policy resolution, `DeliverOneAsync`'s policy selection, and the two adapters).
|
||||
- **Plan 08** flagged `SmsConfiguration.MaxRetryCount` unused — ownership of the wiring is **here** (Task 18); plan 08 should not duplicate it.
|
||||
- Tasks 8 and 19 both add EF migrations against the shared model snapshot — they are sequenced (19 after 8) within this plan; if any other plan adds migrations in the same window, serialize migration-generation across plans.
|
||||
|
||||
## Execution order
|
||||
|
||||
**P0 (start immediately, independent):** Task 1 (the High-severity stale-handler fix) and Task 8 (the High-severity spec-gap migration).
|
||||
|
||||
- **Inbound chain:** 1 → 2 → 3 → 4 → 6 → 5 → 24 (2/3 also gate 11's ManagementActor edits ordering; 24 last because it shares `Component-InboundAPI.md` with 2 and 5).
|
||||
- **ESG chain:** 8 → {9, 10, 11, 12 in parallel} → 13 → 14 → 15 → 16 (9/13/14/15/16 serialize on `ExternalSystemClient.cs`; 10/11/12 are parallel with all of them).
|
||||
- **Notifications:** 17 (anytime) ; 18 → 21 ; 19 (after 8, migration serialization) → 20 → 21 ; 22 after 11 (shared `ManagementActor.cs`).
|
||||
- **Docs:** 23 anytime; 24 after 2/5.
|
||||
|
||||
Intra-plan critical path: **1 → 2 → 3 → 4 → 5** (inbound executor file chain) and **8 → 9 → 13 → 14 → 15 → 16** (ESG client file chain) — roughly equal length; run them in parallel lanes. Everything else hangs off those two spines or is free-floating.
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"planPath": "archreview/plans/PLAN-06-edge-integrations.md",
|
||||
"tasks": [
|
||||
{ "id": 1, "subject": "Task 1: Revision-checked inbound handler cache (self-healing across failover / direct SQL / known-bad)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 2, "subject": "Task 2: DB-authoritative compile semantics — management warning text + design doc", "status": "pending", "blockedBy": [1] },
|
||||
{ "id": 3, "subject": "Task 3: InvalidateMethod seam — the Inbound API consumer of plan 05's invalidation contract", "status": "pending", "blockedBy": [1, 2] },
|
||||
{ "id": 4, "subject": "Task 4: Abandoned inbound executions — defer DI-scope disposal + orphan telemetry", "status": "pending", "blockedBy": [3] },
|
||||
{ "id": 5, "subject": "Task 5: Inbound error contract — code field + SITE_UNREACHABLE + 415 for non-JSON bodies", "status": "pending", "blockedBy": [4, 6] },
|
||||
{ "id": 6, "subject": "Task 6: InboundApiEndpointFilter hot-reads options via IOptionsMonitor", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 7, "subject": "Task 7: Parallelize startup compile of inbound methods", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 8, "subject": "Task 8: ExternalSystemDefinition.TimeoutSeconds — entity + EF mapping + migration", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 9, "subject": "Task 9: Honor the per-system timeout in ExternalSystemClient.InvokeHttpAsync", "status": "pending", "blockedBy": [8] },
|
||||
{ "id": 10, "subject": "Task 10: Carry TimeoutSeconds through the artifact pipeline to sites", "status": "pending", "blockedBy": [8] },
|
||||
{ "id": 11, "subject": "Task 11: TimeoutSeconds management commands + CLI", "status": "pending", "blockedBy": [8, 3] },
|
||||
{ "id": 12, "subject": "Task 12: TimeoutSeconds in Central UI form + spec doc sync", "status": "pending", "blockedBy": [8] },
|
||||
{ "id": 13, "subject": "Task 13: Path-template substitution (/recipes/{id}) in BuildUrl", "status": "pending", "blockedBy": [9] },
|
||||
{ "id": 14, "subject": "Task 14: Bound outbound HTTP response buffering", "status": "pending", "blockedBy": [13] },
|
||||
{ "id": 15, "subject": "Task 15: Structured JSON AuthConfiguration with legacy colon fallback", "status": "pending", "blockedBy": [14] },
|
||||
{ "id": 16, "subject": "Task 16: Cancellation-aware ErrorClassifier.IsTransient overload", "status": "pending", "blockedBy": [15] },
|
||||
{ "id": 17, "subject": "Task 17: SMS first-transient short-circuit (duplicate amplification + sweep bound)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 18, "subject": "Task 18: Honor SmsConfiguration.MaxRetries/RetryDelay — per-type retry policy", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 19, "subject": "Task 19: Configurable OAuth2 authority/scope — entity + EF + migration + token service", "status": "pending", "blockedBy": [8] },
|
||||
{ "id": 20, "subject": "Task 20: OAuth2 authority/scope surfaces — management command + CLI + UI + doc", "status": "pending", "blockedBy": [19] },
|
||||
{ "id": 21, "subject": "Task 21: Notification low-severity cleanups — deterministic SMTP selection + OAuth2 cache visibility", "status": "pending", "blockedBy": [18, 19] },
|
||||
{ "id": 22, "subject": "Task 22: Validate Twilio AccountSid format at config save", "status": "pending", "blockedBy": [11] },
|
||||
{ "id": 23, "subject": "Task 23: Document DelmiaNotifier at-least-once semantics + idempotency requirement", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 24, "subject": "Task 24: Document the dual-key API-key rotation procedure", "status": "pending", "blockedBy": [2, 5] }
|
||||
],
|
||||
"lastUpdated": "2026-07-08"
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,39 @@
|
||||
{
|
||||
"planPath": "archreview/plans/PLAN-07-ui-management-security.md",
|
||||
"tasks": [
|
||||
{ "id": 1, "subject": "Task 1: DisableLoginGuard — refuse the flag outside Development", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 2, "subject": "Task 2: Wire the guard at the Host composition root + doc", "status": "pending", "blockedBy": [1] },
|
||||
{ "id": 3, "subject": "Task 3: Scrub deploy/wonder-app-vd03 — remove DisableLogin: true", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 4, "subject": "Task 4: ArtifactDeploymentService.DeployToSiteAsync", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 5, "subject": "Task 5: HandleDeployArtifacts honors SiteId + site-scope enforcement + docs", "status": "pending", "blockedBy": [4] },
|
||||
{ "id": 6, "subject": "Task 6: ConfigSecretScrubber — JSON secret elision + sentinel merge", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 7, "subject": "Task 7: DataConnection List/Get/audit secret elision + sentinel-preserving update", "status": "pending", "blockedBy": [6] },
|
||||
{ "id": 8, "subject": "Task 8: ExternalSystem AuthConfiguration elision + preserve-if-null update", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 9, "subject": "Task 9: DatabaseConnection ConnectionString elision + preserve-if-null update", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 10, "subject": "Task 10: GetRequiredRoles any-of refactor + gate ListSecuredWrites", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 11, "subject": "Task 11: Reconcile the area-management role gate (code + three docs)", "status": "pending", "blockedBy": [10] },
|
||||
{ "id": 12, "subject": "Task 12: Canonicalize role-mapping casing at the write path", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 13, "subject": "Task 13: GetRequiredRoles matrix test — freeze the authorization table", "status": "pending", "blockedBy": [10, 11, 22, 23] },
|
||||
{ "id": 14, "subject": "Task 14: SecuredWrite repository — TryMarkExpiredAsync + CountAsync", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 15, "subject": "Task 15: Server-side secured-write TTL — enforce at approve/reject", "status": "pending", "blockedBy": [14] },
|
||||
{ "id": 16, "subject": "Task 16: Opportunistic expiry sweep in HandleListSecuredWrites", "status": "pending", "blockedBy": [15] },
|
||||
{ "id": 17, "subject": "Task 17: Secured-write list paging — command, handler, CLI", "status": "pending", "blockedBy": [14] },
|
||||
{ "id": 18, "subject": "Task 18: SecuredWrites page — history pager + age in approve dialog + docs", "status": "pending", "blockedBy": [17] },
|
||||
{ "id": 19, "subject": "Task 19: Per-command long-running Ask timeout", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 20, "subject": "Task 20: Ship ManagementService timeout config + doc", "status": "pending", "blockedBy": [19] },
|
||||
{ "id": 21, "subject": "Task 21: Browse/Verify/Cert commands gain optional SiteIdentifier", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 22, "subject": "Task 22: Actor handlers — BrowseNode / SearchAddressSpace / VerifyEndpoint", "status": "pending", "blockedBy": [10, 21] },
|
||||
{ "id": 23, "subject": "Task 23: Actor handlers — TrustServerCert / RemoveServerCert / ListServerCerts (Admin)", "status": "pending", "blockedBy": [10, 21] },
|
||||
{ "id": 24, "subject": "Task 24: Dispatch-coverage test — every registered command reaches a handler", "status": "pending", "blockedBy": [22, 23] },
|
||||
{ "id": 25, "subject": "Task 25: CLI parity — browse / search / verify-endpoint / certs commands", "status": "pending", "blockedBy": [22, 23] },
|
||||
{ "id": 26, "subject": "Task 26: LoginThrottle — fixed-window LDAP-bind failure lockout", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 27, "subject": "Task 27: Wire the throttle — ManagementAuthenticator + auth endpoints + doc", "status": "pending", "blockedBy": [26] },
|
||||
{ "id": 28, "subject": "Task 28: Additive paging for ListTemplates / ListInstances", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 29, "subject": "Task 29: PollGate — non-reentrant poll timers on Health + Alarm Summary", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 30, "subject": "Task 30: Transport single-flight + base64 copy elimination", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 31, "subject": "Task 31: Low cleanup A — hub send fault logging, event-log filter comment, stale CLAUDE.md note", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 32, "subject": "Task 32: Low cleanup B — AlarmSummary memoization + sortable-header a11y", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 33, "subject": "Task 33: Add CLI.Tests to the slnx + retire the gotcha", "status": "pending", "blockedBy": [] }
|
||||
],
|
||||
"lastUpdated": "2026-07-08"
|
||||
}
|
||||
@@ -0,0 +1,739 @@
|
||||
# Conventions, Tests & Underdeveloped Areas Implementation Plan
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Close the cross-cutting gaps from architecture review 08 — the CLI.Tests slnx blind spot, the ~12 components binding options without startup validation, the vestigial site-side notification surface, the misnamed-thin PerformanceTests project, the docs-code drift owned by no other plan, and a triaged register for the 23-item deferred-work inventory.
|
||||
|
||||
**Architecture:** All changes are convention-hardening around existing seams: eager options validation copies the shipped `OptionsValidatorBase<T>` pattern (`ZB.MOM.WW.Configuration` package) already used by HealthMonitoring/ClusterInfrastructure/AuditLog/KpiHistory; the vestigial notification removal relies on the existing `PurgeCentralOnlyNotificationConfigAsync` guarantee (which stays); new tests slot into existing test projects (PerformanceTests, Commons.Tests). No new runtime components, no schema changes, no wire-contract changes.
|
||||
|
||||
**Tech Stack:** C# / .NET 10, xUnit, Akka.NET TestKit/Streams, `Microsoft.Extensions.Options` (`IValidateOptions<>` + `ValidateOnStart`), Newtonsoft.Json (contract-lock tests only), SQLite (site storage).
|
||||
|
||||
**Build:** `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Tests: `dotnet test ZB.MOM.WW.ScadaBridge.slnx` — after Task 1 this picks up CLI.Tests too.
|
||||
|
||||
## Findings Coverage
|
||||
|
||||
| Report finding (08-conventions-tests-underdeveloped.md) | Severity | Coverage |
|
||||
|---|---|---|
|
||||
| §1.1 Commons folders beyond the documented four | Low | Task 10 — **narrowed on re-verify**: `Serialization/` and `Validators/` are already in `Component-Commons.md`'s layout tree (lines 293–294); only `Observability/` is missing |
|
||||
| §1.2 POCO persistence-ignorance | Pass | No action |
|
||||
| §1.3 Site-side repository-implementation exception undocumented | Low | Task 10 |
|
||||
| §1.3 / §3 item 23 Vestigial `SiteNotificationRepository` + `StoreNotificationListAsync` write path | Low/Medium | Task 6 |
|
||||
| §1.4 `Communication` → `HealthMonitoring` layering inversion | Low | **Deferred** (Task 11 register entry): moving `ICentralHealthAggregator` + `SiteHealthState` to Commons ripples across HealthMonitoring, Communication, ManagementService, CentralUI, AuditLog + their tests — disproportionate to a cosmetic note; documented as accepted with revisit trigger |
|
||||
| §1.4 `ManagementService.csproj` `..\` path-separator inconsistency | Low | Task 10 |
|
||||
| §1.5 Options validation on only ~6 of ~20 option classes | Medium | Tasks 2, 3, 4, 5 |
|
||||
| §1.6 UTC / §1.7 correlation IDs / §1.9 naming | Pass | No action |
|
||||
| §1.8 No contract-lock tests for ClusterClient record contracts (also Rec 10) | Low | Task 9 |
|
||||
| §2.1 Thin test spots (SiteCallAudit.Tests, DeploymentManager.Tests) | Info | **Deferred** (Task 11 register entry): no defect identified; backfill tracked with revisit trigger |
|
||||
| §2.3 PerformanceTests is a correctness-at-small-scale suite | Medium | Tasks 7 (streaming throughput) + 8 (failover-timing harness placeholder, delegating the rig to Plan 01) |
|
||||
| §2.4 CLI.Tests excluded from slnx | High | Task 1 |
|
||||
| §3 item 1 / §4 Transport `AreaName: null` vs doc claim | High | **Owned by plan 05** |
|
||||
| §3 item 3 AuditLog reconciliation dials NodeA only | Medium | **Owned by plan 04** |
|
||||
| §3 item 5 `SmsConfiguration.MaxRetryCount` stored but unused | Medium | **Owned by plan 06** |
|
||||
| §3 item 6 Role-check case-sensitivity asymmetry | Medium | **Owned by plan 07** |
|
||||
| §3 items 7–22 (hash-chain, Parquet, live alarm stream, cert-trust central persistence, CSV parity, WaitForAttribute, BrowseNext, StubOpcUaClient, unified outbox page, folder drag-drop, bundle signing, EXPIRED purge, backlog-reporter options, KPI downsampling) | Low | Task 11 — each gets an explicit register row with defer rationale + revisit trigger (none needs code in this plan) |
|
||||
| §4 `docs/components/` lags 3 components vs README claim | Low | Task 10 (scope the claim; backfill tracked in Task 11 register) |
|
||||
| §4 DelmiaNotifier breaks `Component-<Name>.md` convention | Low | Task 10 (document the exception) |
|
||||
| §4 Component-KpiHistory / StoreAndForward / ClusterInfrastructure doc sync | Pass | No action |
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Add CLI.Tests to the slnx and retire the gotcha documentation
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~3 min
|
||||
**Parallelizable with:** 2, 3, 4, 5, 6, 7, 8, 9, 11
|
||||
**Files:**
|
||||
- Modify: `ZB.MOM.WW.ScadaBridge.slnx` (insert one line after line 51, the ManagementService.Tests entry, mirroring src ordering)
|
||||
- Modify: `CLAUDE.md` (line 69, the Editing Rules test-command bullet)
|
||||
- Modify: `/Users/dohertj2/.claude/projects/-Users-dohertj2-Desktop-ScadaBridge/memory/MEMORY.md` (index entry "CLI.Tests is not in the slnx") and `/Users/dohertj2/.claude/projects/-Users-dohertj2-Desktop-ScadaBridge/memory/cli-tests-not-in-slnx.md`
|
||||
|
||||
1. Verify current state (expect CLI.Tests absent):
|
||||
```bash
|
||||
grep -c "CLI.Tests" ZB.MOM.WW.ScadaBridge.slnx # expect: 0
|
||||
```
|
||||
2. Edit `ZB.MOM.WW.ScadaBridge.slnx`: after the line
|
||||
` <Project Path="tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests.csproj" />`
|
||||
insert:
|
||||
```xml
|
||||
<Project Path="tests/ZB.MOM.WW.ScadaBridge.CLI.Tests/ZB.MOM.WW.ScadaBridge.CLI.Tests.csproj" />
|
||||
```
|
||||
3. Verify the solution builds and the CLI tests now run at the solution level:
|
||||
```bash
|
||||
dotnet build ZB.MOM.WW.ScadaBridge.slnx
|
||||
dotnet test ZB.MOM.WW.ScadaBridge.slnx --no-build --filter "FullyQualifiedName~ScadaBridge.CLI.Tests"
|
||||
```
|
||||
Expected: build succeeds; the test run lists `ZB.MOM.WW.ScadaBridge.CLI.Tests` and reports ~279 passing tests (0 failed). If any CLI test fails to compile, fix forward — that is exactly the rot this task exists to expose.
|
||||
4. Update `CLAUDE.md:69`. Replace the bullet:
|
||||
> `- Run tests with \`dotnet test ZB.MOM.WW.ScadaBridge.slnx\`, **but \`ZB.MOM.WW.ScadaBridge.CLI.Tests\` is not in the slnx** and is silently skipped — when touching the CLI, test that project directly (\`dotnet test tests/ZB.MOM.WW.ScadaBridge.CLI.Tests\`), or a green solution run can hide CLI tests that never compiled.`
|
||||
|
||||
with:
|
||||
> `- Run tests with \`dotnet test ZB.MOM.WW.ScadaBridge.slnx\`. All 30 test projects, including \`ZB.MOM.WW.ScadaBridge.CLI.Tests\`, are members of the slnx (CLI.Tests was added 2026-07; the old "silently skipped" gotcha no longer applies).`
|
||||
5. Update the auto-memory: in `MEMORY.md`, change the index line for `cli-tests-not-in-slnx.md` to note the gotcha is fixed (e.g. "FIXED 2026-07: CLI.Tests is now in the slnx; entry kept as history"), and add a first-line status note to `cli-tests-not-in-slnx.md` itself.
|
||||
6. Commit:
|
||||
```bash
|
||||
git add ZB.MOM.WW.ScadaBridge.slnx CLAUDE.md
|
||||
git commit -m "build(slnx): add CLI.Tests to the solution — retire the silently-skipped-279-tests gotcha (arch-review 08 §2.4)"
|
||||
```
|
||||
(The memory files live outside the repo and are not committed.)
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Options validation — Communication + DataConnectionLayer
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 3, 4, 5, 6, 7, 8, 9, 10, 11
|
||||
**Files:**
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptionsValidator.cs`
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/DataConnectionOptionsValidator.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/ServiceCollectionExtensions.cs` (lines 13–14), `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/ServiceCollectionExtensions.cs` (lines 15–22)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/ZB.MOM.WW.ScadaBridge.Communication.csproj`, `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.csproj` (add `<PackageReference Include="ZB.MOM.WW.Configuration" />` — no Version attribute; central package management already has it, see `HealthMonitoring.csproj:15`)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/CommunicationOptionsValidatorTests.cs`, `tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/DataConnectionOptionsValidatorTests.cs`
|
||||
|
||||
The pattern to copy is `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/HealthMonitoringOptionsValidator.cs` (an `OptionsValidatorBase<T>` from the `ZB.MOM.WW.Configuration` package with `builder.RequireThat(...)` per field) registered via `TryAddEnumerable(ServiceDescriptor.Singleton<IValidateOptions<T>, TValidator>())` + `.ValidateOnStart()` on the `AddOptions` chain (`HealthMonitoring/ServiceCollectionExtensions.cs:157-162`). Tests copy `tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/HealthMonitoringOptionsValidatorTests.cs`.
|
||||
|
||||
1. Write failing tests first — `tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/CommunicationOptionsValidatorTests.cs`:
|
||||
```csharp
|
||||
using Microsoft.Extensions.Options;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;
|
||||
|
||||
public class CommunicationOptionsValidatorTests
|
||||
{
|
||||
private static ValidateOptionsResult Validate(CommunicationOptions options) =>
|
||||
new CommunicationOptionsValidator().Validate(Options.DefaultName, options);
|
||||
|
||||
[Fact]
|
||||
public void DefaultOptions_AreValid()
|
||||
{
|
||||
var result = Validate(new CommunicationOptions());
|
||||
Assert.True(result.Succeeded, result.FailureMessage);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ZeroDeploymentTimeout_IsRejected()
|
||||
{
|
||||
var result = Validate(new CommunicationOptions { DeploymentTimeout = TimeSpan.Zero });
|
||||
Assert.True(result.Failed);
|
||||
Assert.Contains("DeploymentTimeout", result.FailureMessage);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void NonPositiveGrpcMaxConcurrentStreams_IsRejected()
|
||||
{
|
||||
var result = Validate(new CommunicationOptions { GrpcMaxConcurrentStreams = 0 });
|
||||
Assert.True(result.Failed);
|
||||
Assert.Contains("GrpcMaxConcurrentStreams", result.FailureMessage);
|
||||
}
|
||||
}
|
||||
```
|
||||
Mirror the same three-test shape for `DataConnectionOptionsValidator` (`ReconnectInterval`, `SeedReadMaxAttempts`).
|
||||
2. Run, expect FAIL (validator types don't exist → compile error, which counts):
|
||||
```bash
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.Communication.Tests --filter "FullyQualifiedName~CommunicationOptionsValidator"
|
||||
```
|
||||
3. Implement `CommunicationOptionsValidator` (positive-duration checks for every `TimeSpan` on `CommunicationOptions` — `DeploymentTimeout`, `LifecycleTimeout`, `ArtifactDeploymentTimeout`, `QueryTimeout`, `IntegrationTimeout`, `DebugViewTimeout`, `HealthReportTimeout`, `NotificationForwardTimeout`, `GrpcKeepAlivePingDelay`, `GrpcKeepAlivePingTimeout`, `GrpcMaxStreamLifetime`, `TransportHeartbeatInterval`, `TransportFailureThreshold` — plus `GrpcMaxConcurrentStreams > 0`). Error messages must name the config key (`Communication:<Field>`), same style as `HealthMonitoringOptionsValidator`:
|
||||
```csharp
|
||||
using ZB.MOM.WW.Configuration;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Communication;
|
||||
|
||||
/// <summary>
|
||||
/// Validates <see cref="CommunicationOptions"/> at startup (ValidateOnStart) so a
|
||||
/// malformed "Communication" appsettings section fails fast at boot with a
|
||||
/// key-naming message instead of surfacing at first Ask/gRPC use.
|
||||
/// </summary>
|
||||
public sealed class CommunicationOptionsValidator : OptionsValidatorBase<CommunicationOptions>
|
||||
{
|
||||
protected override void Validate(ValidationBuilder builder, CommunicationOptions options)
|
||||
{
|
||||
builder.RequireThat(options.DeploymentTimeout > TimeSpan.Zero,
|
||||
$"Communication:DeploymentTimeout must be a positive duration (was {options.DeploymentTimeout}).");
|
||||
// ... one RequireThat per field listed above ...
|
||||
builder.RequireThat(options.GrpcMaxConcurrentStreams > 0,
|
||||
$"Communication:GrpcMaxConcurrentStreams must be positive (was {options.GrpcMaxConcurrentStreams}).");
|
||||
}
|
||||
}
|
||||
```
|
||||
`DataConnectionOptionsValidator` likewise: `ReconnectInterval`, `TagResolutionRetryInterval`, `WriteTimeout`, `SeedReadTimeout`, `StableConnectionThreshold`, `SeedReadRetryDelay` positive durations; `SeedReadMaxAttempts > 0`. Read `OpcUaGlobalOptions`/`MxGatewayGlobalOptions` (`src/.../DataConnectionLayer/`) and add `RequireThat` checks for any numeric/duration fields they carry (add validators for them only if they have such fields).
|
||||
4. Wire registration. `Communication/ServiceCollectionExtensions.cs:13-14` becomes:
|
||||
```csharp
|
||||
services.AddOptions<CommunicationOptions>()
|
||||
.BindConfiguration("Communication")
|
||||
.ValidateOnStart();
|
||||
services.TryAddEnumerable(
|
||||
ServiceDescriptor.Singleton<IValidateOptions<CommunicationOptions>, CommunicationOptionsValidator>());
|
||||
```
|
||||
Same shape for the three `AddOptions` calls in `DataConnectionLayer/ServiceCollectionExtensions.cs:15-22`. Add the `ZB.MOM.WW.Configuration` PackageReference to both csprojs.
|
||||
5. Run, expect PASS:
|
||||
```bash
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.Communication.Tests --filter "FullyQualifiedName~OptionsValidator"
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~OptionsValidator"
|
||||
dotnet build ZB.MOM.WW.ScadaBridge.slnx
|
||||
```
|
||||
6. Commit:
|
||||
```bash
|
||||
git add src/ZB.MOM.WW.ScadaBridge.Communication src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer tests/ZB.MOM.WW.ScadaBridge.Communication.Tests tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests
|
||||
git commit -m "feat(options): eager startup validation for Communication + DataConnectionLayer options (arch-review 08 §1.5)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Options validation — site pipeline (SiteRuntime, StoreAndForward, SiteEventLogging)
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 2, 4, 5, 7, 8, 9, 10, 11 (NOT 6 — both touch the SiteRuntime project; run sequentially to keep csproj/test churn separate)
|
||||
**Files:**
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs`, `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptionsValidator.cs`, `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogOptionsValidator.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs` (lines 124–127 — the `services.Configure<T>(config.GetSection(...))` calls for these three options)
|
||||
- Modify: the three component csprojs (add `<PackageReference Include="ZB.MOM.WW.Configuration" />`)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/SiteRuntimeOptionsValidatorTests.cs`, `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardOptionsValidatorTests.cs`, `tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLogOptionsValidatorTests.cs`
|
||||
|
||||
These three options classes are bound in the Host, not in the component SCEs (`Host/SiteServiceRegistration.cs:124-127`), so binding conversion happens there; validator classes live with their owners per the "options classes owned by component projects" convention.
|
||||
|
||||
1. Write failing tests (same shape as Task 2 step 1; one `DefaultOptions_AreValid` + 2–3 rejection cases per validator). Key rejection cases:
|
||||
- `SiteRuntimeOptions`: `StartupBatchSize = 0`, `ScriptExecutionThreadCount = 0`, `StreamBufferSize = 0` (all feed thread-pool/stream sizing; also validate `StartupBatchDelayMs >= 0`, `MaxScriptCallDepth > 0`, `ScriptExecutionTimeoutSeconds > 0`, `MirroredAlarmCapPerSource > 0`, `NativeAlarmRetryIntervalMs > 0`, `ConfigFetchTimeoutSeconds > 0`, `ConfigFetchRetryCount >= 0`).
|
||||
- `StoreAndForwardOptions`: `DefaultRetryInterval = TimeSpan.Zero`, `RetryTimerInterval = TimeSpan.Zero`, `SqliteDbPath = ""` (also `DefaultMaxRetries >= 0`).
|
||||
- `SiteEventLogOptions`: `RetentionDays = 0`, `PurgeInterval = TimeSpan.Zero`, `MaxQueryPageSize < QueryPageSize` cross-check (also `MaxStorageMb > 0`, `WriteQueueCapacity > 0`, `DatabasePath` non-empty).
|
||||
2. Run, expect FAIL (compile error on missing validator types):
|
||||
```bash
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests --filter "FullyQualifiedName~OptionsValidator"
|
||||
```
|
||||
3. Implement the three validators (`OptionsValidatorBase<T>`, key-naming messages using the `ScadaBridge:SiteRuntime` / `ScadaBridge:StoreAndForward` / `ScadaBridge:SiteEventLog` prefixes). Convert the Host bindings — `Host/SiteServiceRegistration.cs:124-127` pattern per option:
|
||||
```csharp
|
||||
services.AddOptions<SiteRuntimeOptions>()
|
||||
.Bind(config.GetSection("ScadaBridge:SiteRuntime"))
|
||||
.ValidateOnStart();
|
||||
services.TryAddEnumerable(
|
||||
ServiceDescriptor.Singleton<IValidateOptions<SiteRuntimeOptions>, SiteRuntimeOptionsValidator>());
|
||||
```
|
||||
(Keep any other `Configure<>` calls in that block untouched.)
|
||||
4. Run, expect PASS; then guard against startup regressions (Host composition-root tests boot both roles):
|
||||
```bash
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~OptionsValidator"
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests --filter "FullyQualifiedName~OptionsValidator"
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests --filter "FullyQualifiedName~OptionsValidator"
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~CompositionRoot"
|
||||
```
|
||||
5. Commit:
|
||||
```bash
|
||||
git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime src/ZB.MOM.WW.ScadaBridge.StoreAndForward src/ZB.MOM.WW.ScadaBridge.SiteEventLogging src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests
|
||||
git commit -m "feat(options): eager startup validation for site-pipeline options (SiteRuntime, S&F, SiteEventLog) (arch-review 08 §1.5)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Options validation — edge + management (InboundAPI, ExternalSystemGateway, NotificationService, ManagementService)
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 2, 3, 5, 6, 7, 8, 9, 10, 11
|
||||
**Files:**
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundApiOptionsValidator.cs`, `src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemGatewayOptionsValidator.cs`, `src/ZB.MOM.WW.ScadaBridge.NotificationService/NotificationOptionsValidator.cs`, `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementServiceOptionsValidator.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` (line 273 — `Configure<InboundApiOptions>` → `AddOptions().Bind().ValidateOnStart()` + validator registration), `src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ServiceCollectionExtensions.cs` (lines 23–24), `src/ZB.MOM.WW.ScadaBridge.NotificationService/ServiceCollectionExtensions.cs` (lines 20–21), `src/ZB.MOM.WW.ScadaBridge.ManagementService/ServiceCollectionExtensions.cs` (lines 15–16)
|
||||
- Modify: the four component csprojs (add `ZB.MOM.WW.Configuration` PackageReference)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/InboundApiOptionsValidatorTests.cs`, `tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemGatewayOptionsValidatorTests.cs`, `tests/ZB.MOM.WW.ScadaBridge.NotificationService.Tests/NotificationOptionsValidatorTests.cs`, `tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/ManagementServiceOptionsValidatorTests.cs`
|
||||
|
||||
1. Write failing tests (Task 2 shape). Rejection cases:
|
||||
- `InboundApiOptions`: `DefaultMethodTimeout = TimeSpan.Zero`, `MaxRequestBodyBytes = 0`. Do **not** require `ApiKeyPepper` non-empty (empty is a legitimate dev default — check `Host/Program.cs` usage before deciding; if a non-empty pepper is required in production the `StartupValidator` owns that, not this validator).
|
||||
- `ExternalSystemGatewayOptions`: `DefaultHttpTimeout = TimeSpan.Zero`, `MaxConcurrentConnectionsPerSystem = 0`.
|
||||
- `NotificationOptions`: `ConnectionTimeoutSeconds = 0`, `MaxConcurrentConnections = 0`.
|
||||
- `ManagementServiceOptions`: `CommandTimeout = TimeSpan.Zero`.
|
||||
2. Run one project, expect FAIL:
|
||||
```bash
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests --filter "FullyQualifiedName~OptionsValidator"
|
||||
```
|
||||
3. Implement the four validators (key prefixes `ScadaBridge:InboundApi`, `ScadaBridge:ExternalSystemGateway`, `ScadaBridge:Notification`, `ScadaBridge:ManagementService`) and convert each binding site to `AddOptions<T>().BindConfiguration(...)/.Bind(...).ValidateOnStart()` + `TryAddEnumerable` validator registration (exactly the Task 2 step 4 shape).
|
||||
4. Run, expect PASS:
|
||||
```bash
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests --filter "FullyQualifiedName~OptionsValidator"
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests --filter "FullyQualifiedName~OptionsValidator"
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.NotificationService.Tests --filter "FullyQualifiedName~OptionsValidator"
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests --filter "FullyQualifiedName~OptionsValidator"
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~CompositionRoot"
|
||||
```
|
||||
5. Commit:
|
||||
```bash
|
||||
git add src/ZB.MOM.WW.ScadaBridge.InboundAPI src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway src/ZB.MOM.WW.ScadaBridge.NotificationService src/ZB.MOM.WW.ScadaBridge.ManagementService src/ZB.MOM.WW.ScadaBridge.Host/Program.cs tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests tests/ZB.MOM.WW.ScadaBridge.NotificationService.Tests tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests
|
||||
git commit -m "feat(options): eager startup validation for edge/management options (InboundAPI, ESG, Notification, ManagementService) (arch-review 08 §1.5)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Options validation — central components (Transport, SiteCallAudit, DeploymentManager)
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 2, 3, 4, 6, 7, 8, 9, 10, 11
|
||||
**Files:**
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.Transport/TransportOptionsValidator.cs`, `src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/SiteCallAuditOptionsValidator.cs`, `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentManagerOptionsValidator.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Transport/ServiceCollectionExtensions.cs` (line 25), `src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/ServiceCollectionExtensions.cs` (lines 40–41), `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/ServiceCollectionExtensions.cs` (line 31)
|
||||
- Modify: the three component csprojs (add `ZB.MOM.WW.Configuration` PackageReference)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Transport.Tests/TransportOptionsValidatorTests.cs`, `tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests/SiteCallAuditOptionsValidatorTests.cs`, `tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests/DeploymentManagerOptionsValidatorTests.cs`
|
||||
|
||||
1. Write failing tests (Task 2 shape). Rejection cases:
|
||||
- `TransportOptions` (`ScadaBridge:Transport`): `BundleSessionTtlMinutes = 0`, `MaxBundleSizeMb = 0`, `Pbkdf2Iterations = 0` (security-sensitive: also require `Pbkdf2Iterations >= 100_000` so a config typo cannot silently weaken bundle key derivation), `MaxUnlockAttemptsPerSession = 0`, plus positivity for `MaxBundleEntryDecompressedMb`, `MaxBundleEntryCount`, `MaxBundleEntryCompressionRatio`, `MaxUnlockAttemptsPerIpPerHour`, `SchemaVersionMajor`, and `SourceEnvironment` non-empty.
|
||||
- `SiteCallAuditOptions` (`ScadaBridge:SiteCallAudit`): `StuckAgeThreshold = TimeSpan.Zero`, `KpiInterval = TimeSpan.Zero`, `ReconciliationBatchSize = 0`, `RetentionDays = 0`, plus positivity for `RelayTimeout`, `ReconciliationInterval`, `PurgeInterval`. Note: validate the base properties, not the `Resolved*` computed properties.
|
||||
- `DeploymentManagerOptions` (`ScadaBridge:DeploymentManager` — confirm the section name at the binding site; `ServiceCollectionExtensions.cs:31` uses bare `AddOptions<DeploymentManagerOptions>()`, i.e. code-configured defaults, so ValidateOnStart still applies): `LifecycleCommandTimeout = TimeSpan.Zero`, `ArtifactDeploymentTimeoutPerSite = TimeSpan.Zero`, `OperationLockTimeout = TimeSpan.Zero`.
|
||||
2. Run one project, expect FAIL:
|
||||
```bash
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.Tests --filter "FullyQualifiedName~OptionsValidator"
|
||||
```
|
||||
3. Implement the three validators + registration conversion (Task 2 step 4 shape). Transport becomes:
|
||||
```csharp
|
||||
services.AddOptions<TransportOptions>().BindConfiguration(OptionsSection).ValidateOnStart();
|
||||
services.TryAddEnumerable(
|
||||
ServiceDescriptor.Singleton<IValidateOptions<TransportOptions>, TransportOptionsValidator>());
|
||||
```
|
||||
4. Run, expect PASS:
|
||||
```bash
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.Transport.Tests --filter "FullyQualifiedName~OptionsValidator"
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests --filter "FullyQualifiedName~OptionsValidator"
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests --filter "FullyQualifiedName~OptionsValidator"
|
||||
dotnet build ZB.MOM.WW.ScadaBridge.slnx
|
||||
```
|
||||
5. Commit:
|
||||
```bash
|
||||
git add src/ZB.MOM.WW.ScadaBridge.Transport src/ZB.MOM.WW.ScadaBridge.SiteCallAudit src/ZB.MOM.WW.ScadaBridge.DeploymentManager tests/ZB.MOM.WW.ScadaBridge.Transport.Tests tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests
|
||||
git commit -m "feat(options): eager startup validation for central-component options (Transport, SiteCallAudit, DeploymentManager) (arch-review 08 §1.5)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Remove the vestigial site-side notification-list surface
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 2, 4, 5, 7, 8, 9, 11 (NOT 3 — same project; NOT 10 — both touch `docs/components/SiteRuntime.md`? No: Task 10 does not touch that file. Parallelizable with 10 too, but keep 3 sequential.)
|
||||
**Files:**
|
||||
- Delete: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/SiteNotificationRepository.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs` (line 65 — remove `services.AddScoped<INotificationRepository, SiteNotificationRepository>();`)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs` — remove `StoreNotificationListAsync` (lines 715–739) and `StoreSmtpConfigurationAsync` (lines ~741–785). **Keep** `PurgeCentralOnlyNotificationConfigAsync` (lines 691–713) and the table schemas — the purge is the security cleanup for DBs written by older builds.
|
||||
- Modify tests: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CompositionRootTests.cs` (lines 444–451), `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Repositories/SiteRepositoryTests.cs` (~lines 185–200), `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/ArtifactStorageTests.cs` (all `StoreNotificationListAsync`/`StoreSmtpConfigurationAsync`/`SiteNotificationRepository` usages: ~lines 125–195)
|
||||
- Modify: `docs/components/SiteRuntime.md` (line 15 — drop `SiteNotificationRepository` from the `Repositories/` listing and note why)
|
||||
|
||||
Context (verified): the central-only notification design means `PurgeCentralOnlyNotificationConfigAsync` empties `notification_lists` + `smtp_configurations` on every artifact apply (`DeploymentManagerActor.cs:1556`, `SiteReplicationActor.cs:267`), so the registered `SiteNotificationRepository` can only ever read empty tables, and the two `Store*` methods have **zero production callers** (grep: only tests). `SiteExternalSystemRepository` is NOT vestigial — external system definitions do ship to sites — leave it alone.
|
||||
|
||||
1. Write the failing test first — replace `Site_NotificationRepository_IsSiteImplementation` in `CompositionRootTests.cs:444-451` with a design-invariant lock:
|
||||
```csharp
|
||||
[Fact]
|
||||
public void Site_NotificationRepository_IsNotRegistered()
|
||||
{
|
||||
// Notification delivery is central-only (CLAUDE.md "Notification delivery is
|
||||
// central-only"); the site-local notification_lists/smtp_configurations tables
|
||||
// are purged on every artifact apply, so a site-side INotificationRepository
|
||||
// could only ever serve empty results. It must not be registered at all.
|
||||
using var scope = _host.Services.CreateScope();
|
||||
var repo = scope.ServiceProvider.GetService<INotificationRepository>();
|
||||
Assert.Null(repo);
|
||||
}
|
||||
```
|
||||
2. Run, expect FAIL (repo is still registered):
|
||||
```bash
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~Site_NotificationRepository"
|
||||
```
|
||||
3. Implement: delete `SiteNotificationRepository.cs`; remove the DI registration at `SiteRuntime/ServiceCollectionExtensions.cs:65`; delete `StoreNotificationListAsync` + `StoreSmtpConfigurationAsync` from `SiteStorageService.cs`; extend the doc comment on `PurgeCentralOnlyNotificationConfigAsync` with one sentence noting the write paths and site repository were removed (with date + review pointer) so a future maintainer doesn't reintroduce them. Check whether `SiteRuntime/Repositories/SyntheticId.cs` is still used by `SiteExternalSystemRepository` — if yes keep it, if the notification repo was its last consumer delete it too.
|
||||
4. Delete/rewrite the orphaned tests: the notification-list blocks in `SiteRepositoryTests.cs` and `ArtifactStorageTests.cs` (they exercise deleted API). Where a test asserted persistence-across-restart semantics generically, keep the external-system variant only.
|
||||
5. Run, expect PASS:
|
||||
```bash
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~CompositionRoot"
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests
|
||||
dotnet build ZB.MOM.WW.ScadaBridge.slnx
|
||||
```
|
||||
6. Update `docs/components/SiteRuntime.md:15` (`Repositories/` bullet → `SiteExternalSystemRepository` only, with a one-line note that the notification variant was removed because notification config never lives on sites).
|
||||
7. Commit:
|
||||
```bash
|
||||
git add -A src/ZB.MOM.WW.ScadaBridge.SiteRuntime tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests tests/ZB.MOM.WW.ScadaBridge.Host.Tests docs/components/SiteRuntime.md
|
||||
git commit -m "refactor(site-runtime): excise vestigial site-side notification-list surface — repo, DI registration, dead write paths (arch-review 08 §1.3/#23)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 7: PerformanceTests — site-stream throughput test
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 2, 3, 4, 5, 6, 9, 10, 11 (and 8 — different new files)
|
||||
**Files:**
|
||||
- Create: `tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Streaming/SiteStreamThroughputTests.cs`
|
||||
- (No csproj change needed: `PerformanceTests.csproj` already references SiteRuntime and Commons.)
|
||||
|
||||
This adds the first of the two highest-value real measurements the review called for (§2.3): sustained throughput of the site-wide Akka stream (`SiteStreamManager`, `Source.ActorRef` + `BroadcastHub`, per-subscriber buffering) that carries every attribute value and alarm state change on a site. Follows the existing `HotPathLatencyTests` style: `Stopwatch`-based, `[Trait("Category", "Performance")]`, conservative thresholds so CI stays green on slow machines.
|
||||
|
||||
1. Write the test (this is the implementation — a perf test is its own assertion):
|
||||
```csharp
|
||||
using System.Diagnostics;
|
||||
using Akka.Actor;
|
||||
using Akka.TestKit.Xunit2;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Streaming;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.PerformanceTests.Streaming;
|
||||
|
||||
/// <summary>
|
||||
/// Throughput measurement for the site-wide attribute/alarm stream
|
||||
/// (SiteStreamManager: Source.ActorRef → BroadcastHub with per-subscriber
|
||||
/// buffering, DropHead overflow). Design envelope: this stream carries every
|
||||
/// attribute value change on a site; it must comfortably sustain tens of
|
||||
/// thousands of events/sec. Thresholds are deliberately conservative
|
||||
/// (CI-machine safe) — this test exists to catch order-of-magnitude
|
||||
/// regressions, not to benchmark.
|
||||
/// </summary>
|
||||
public class SiteStreamThroughputTests : TestKit
|
||||
{
|
||||
private sealed class CountingActor : ReceiveActor
|
||||
{
|
||||
public static long Received;
|
||||
public CountingActor()
|
||||
{
|
||||
Receive<AttributeValueChanged>(_ => Interlocked.Increment(ref Received));
|
||||
ReceiveAny(_ => { });
|
||||
}
|
||||
}
|
||||
|
||||
[Trait("Category", "Performance")]
|
||||
[Fact]
|
||||
public async Task SiteStream_SustainsAtLeast10kEventsPerSecond_ToOneSubscriber()
|
||||
{
|
||||
const int eventCount = 100_000;
|
||||
CountingActor.Received = 0;
|
||||
|
||||
var manager = new SiteStreamManager(
|
||||
new SiteRuntimeOptions { StreamBufferSize = eventCount },
|
||||
NullLogger<SiteStreamManager>.Instance);
|
||||
manager.Initialize(Sys);
|
||||
|
||||
var counter = Sys.ActorOf(Props.Create<CountingActor>());
|
||||
manager.Subscribe("PerfInstance", counter);
|
||||
|
||||
var sw = Stopwatch.StartNew();
|
||||
for (var i = 0; i < eventCount; i++)
|
||||
{
|
||||
manager.PublishAttributeValueChanged(new AttributeValueChanged(
|
||||
"PerfInstance", "Attr", "Attr", i, "Good", DateTimeOffset.UtcNow));
|
||||
}
|
||||
|
||||
// Drain: wait until the counter goes quiet (no growth across a 200ms window).
|
||||
long last = -1;
|
||||
var deadline = DateTime.UtcNow.AddSeconds(30);
|
||||
while (DateTime.UtcNow < deadline)
|
||||
{
|
||||
var now = Interlocked.Read(ref CountingActor.Received);
|
||||
if (now == last && now > 0) break;
|
||||
last = now;
|
||||
await Task.Delay(200);
|
||||
}
|
||||
sw.Stop();
|
||||
|
||||
var received = Interlocked.Read(ref CountingActor.Received);
|
||||
var eventsPerSecond = received / sw.Elapsed.TotalSeconds;
|
||||
|
||||
// DropHead means under extreme pressure some events may drop; with the
|
||||
// buffer sized to the burst, deliver ratio should be ~1.0.
|
||||
Assert.True(received >= eventCount * 0.9,
|
||||
$"Expected >=90% delivery, got {received}/{eventCount}");
|
||||
Assert.True(eventsPerSecond >= 10_000,
|
||||
$"Expected >=10k events/s sustained, got {eventsPerSecond:F0}/s over {sw.Elapsed.TotalSeconds:F1}s");
|
||||
}
|
||||
}
|
||||
```
|
||||
Adjust to actual constructor/subscribe signatures on read (`SiteStreamManager.Subscribe(string instanceName, IActorRef subscriber)` — verified; subscription filters by `InstanceUniqueName`). If subscriber delivery turns out to route a wrapper message rather than the raw record, match on the wrapper.
|
||||
2. Run, expect PASS (and eyeball the reported rate in the failure-message format by temporarily inverting the assertion if you want the number):
|
||||
```bash
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.PerformanceTests --filter "FullyQualifiedName~SiteStreamThroughput"
|
||||
```
|
||||
3. Commit:
|
||||
```bash
|
||||
git add tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Streaming
|
||||
git commit -m "test(perf): site-wide stream throughput measurement — first real perf-envelope test (arch-review 08 §2.3)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 8: PerformanceTests — failover-timing benchmark harness placeholder
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~3 min
|
||||
**Parallelizable with:** 1, 2, 3, 4, 5, 6, 7, 9, 10, 11
|
||||
**Files:**
|
||||
- Create: `tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Failover/FailoverTimingTests.cs`
|
||||
|
||||
The design claims ~25s total failover (CLAUDE.md "Cluster & Failover"). Measuring it needs a real two-node cluster with a hard kill — **that rig is owned by Plan 01** (the two-node failover test rig from overall recommendation P2-10). This task ships the measurement protocol as an explicitly-skipped harness so the perf suite documents the envelope and Plan 01's rig has a ready seat to plug into; it must NOT duplicate the rig.
|
||||
|
||||
1. Create the placeholder:
|
||||
```csharp
|
||||
namespace ZB.MOM.WW.ScadaBridge.PerformanceTests.Failover;
|
||||
|
||||
/// <summary>
|
||||
/// Failover-timing benchmark harness (placeholder).
|
||||
///
|
||||
/// Design envelope under test (CLAUDE.md "Cluster & Failover"): failure
|
||||
/// detection 2s heartbeat / 10s threshold, SBR stable-after 15s, total
|
||||
/// failover ~25s for a hard node loss.
|
||||
///
|
||||
/// Measurement protocol (to be wired to the two-node cluster rig delivered
|
||||
/// by PLAN-01 — see archreview/plans/PLAN-01-*.md; do not duplicate that rig
|
||||
/// here):
|
||||
/// 1. Form a real 2-node cluster (docker/ topology or Akka.Remote in-proc
|
||||
/// multi-ActorSystem rig from PLAN-01).
|
||||
/// 2. Confirm a cluster singleton (e.g. site DeploymentManager) is hosted
|
||||
/// on node A; record T0.
|
||||
/// 3. Hard-kill node A (process kill / ActorSystem.Abort — not
|
||||
/// CoordinatedShutdown; graceful stop exercises a different path).
|
||||
/// 4. Poll node B for singleton re-host; record T1 at first successful
|
||||
/// response from the migrated singleton.
|
||||
/// 5. Assert T1 - T0 <= 40s (25s design + margin) and report the number.
|
||||
/// </summary>
|
||||
public class FailoverTimingTests
|
||||
{
|
||||
[Trait("Category", "Performance")]
|
||||
[Fact(Skip = "Requires the real two-node failover rig delivered by PLAN-01 " +
|
||||
"(overall review P2-10). This class documents the measurement " +
|
||||
"protocol; wire it to the rig when PLAN-01 lands.")]
|
||||
public void HardKillFailover_SingletonRehostedWithinDesignEnvelope()
|
||||
{
|
||||
// Intentionally empty until the PLAN-01 rig exists.
|
||||
}
|
||||
}
|
||||
```
|
||||
2. Verify it compiles and shows as skipped:
|
||||
```bash
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.PerformanceTests --filter "FullyQualifiedName~FailoverTiming"
|
||||
```
|
||||
Expected: 1 skipped, 0 failed.
|
||||
3. Commit:
|
||||
```bash
|
||||
git add tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Failover
|
||||
git commit -m "test(perf): failover-timing harness placeholder documenting the 25s envelope protocol, pending PLAN-01 two-node rig (arch-review 08 §2.3)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 9: Contract-lock tests for top ClusterClient message records
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 2, 3, 4, 5, 6, 7, 8, 10, 11
|
||||
**Files:**
|
||||
- Create: `tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Messages/ClusterClientContractLockTests.cs`
|
||||
- Modify: `Directory.Packages.props` (add `<PackageVersion Include="Newtonsoft.Json" Version="13.0.4" />` — first check `dotnet list tests/ZB.MOM.WW.ScadaBridge.Commons.Tests package --include-transitive | grep -i newtonsoft` and pin to the version Akka already pulls transitively, to avoid a version bump ride-along)
|
||||
- Modify: `tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/ZB.MOM.WW.ScadaBridge.Commons.Tests.csproj` (add `<PackageReference Include="Newtonsoft.Json" />`)
|
||||
|
||||
Rationale (§1.8, Rec 10): ClusterClient traffic rides Akka.NET's default Newtonsoft-JSON serialization; "additive-only evolution" is currently discipline, not enforcement — the gRPC side has proto contract-lock tests (`Communication.Tests/Protos/*ProtoTests.cs`, `Grpc/ProtoContractTests.cs`) but the record contracts have none. These tests lock the version-skew guarantees: (a) an old-shape payload (missing newer trailing optional fields) still deserializes with defaults, (b) unknown fields from a *newer* peer are ignored, (c) round-trip preserves values.
|
||||
|
||||
Records to lock (highest-traffic cross-cluster surface): `ManagementEnvelope` (`Commons/Messages/Management/ManagementEnvelope.cs`), `DeployInstanceCommand` (`Messages/Deployment/DeployInstanceCommand.cs` — has documented optional trailing fields `CentralFetchBaseUrl`/`FetchToken`), `GetAttributeRequest` + `SetStaticAttributeCommand` (`Messages/Instance/`), `AttributeValueChanged` (`Messages/Streaming/`), one notification-forward record from `Messages/Notification/NotificationMessages.cs`, and `CreateDataConnectionCommand` (`Messages/Management/DataConnectionCommands.cs` — the report's own example of defaulted trailing parameters).
|
||||
|
||||
1. Write the failing test file (fails to compile until the package reference lands — that is the FAIL step):
|
||||
```csharp
|
||||
using Newtonsoft.Json;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Commons.Tests.Messages;
|
||||
|
||||
/// <summary>
|
||||
/// Contract-lock tests for the highest-traffic ClusterClient record contracts.
|
||||
/// ClusterClient traffic uses Akka.NET's default Newtonsoft-JSON serializer and
|
||||
/// the design supports system-wide artifact version skew across sites, so these
|
||||
/// records must obey additive-only evolution: old-shape payloads deserialize
|
||||
/// (missing new fields → defaults) and unknown fields are ignored. If a change
|
||||
/// breaks one of these tests, it breaks rolling cross-cluster compatibility —
|
||||
/// do not "fix the test"; make the change additive.
|
||||
/// </summary>
|
||||
public class ClusterClientContractLockTests
|
||||
{
|
||||
private static readonly JsonSerializerSettings AkkaLikeSettings = new()
|
||||
{
|
||||
MissingMemberHandling = MissingMemberHandling.Ignore,
|
||||
};
|
||||
|
||||
[Fact]
|
||||
public void CreateDataConnectionCommand_OldShape_WithoutNewerTrailingFields_Deserializes()
|
||||
{
|
||||
// Shape as emitted before BackupConfiguration/FailoverRetryCount existed.
|
||||
// (Adjust property list to the record's actual required leading fields.)
|
||||
const string oldShape =
|
||||
"""{"Name":"conn1","SiteId":3,"Protocol":"OpcUa","Configuration":"{}"}""";
|
||||
|
||||
var cmd = JsonConvert.DeserializeObject<CreateDataConnectionCommand>(oldShape, AkkaLikeSettings);
|
||||
|
||||
Assert.NotNull(cmd);
|
||||
Assert.Null(cmd!.BackupConfiguration); // default applied
|
||||
Assert.Equal(3, cmd.FailoverRetryCount); // default applied
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void AttributeValueChanged_UnknownFieldFromNewerPeer_IsIgnored()
|
||||
{
|
||||
const string newerShape =
|
||||
"""{"InstanceUniqueName":"i","AttributePath":"a","AttributeName":"a","Value":1,"Quality":"Good","Timestamp":"2026-07-08T00:00:00+00:00","SomeFutureField":"x"}""";
|
||||
|
||||
var evt = JsonConvert.DeserializeObject<AttributeValueChanged>(newerShape, AkkaLikeSettings);
|
||||
|
||||
Assert.NotNull(evt);
|
||||
Assert.Equal("Good", evt!.Quality);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ManagementEnvelope_RoundTrip_PreservesCorrelationId()
|
||||
{
|
||||
// Construct with the real AuthenticatedUser shape (read Commons/…/AuthenticatedUser).
|
||||
var envelope = /* new ManagementEnvelope(user, command: new object(), "corr-1") */ default(ManagementEnvelope);
|
||||
// serialize → deserialize → assert CorrelationId round-trips.
|
||||
}
|
||||
}
|
||||
```
|
||||
Flesh out one old-shape + one unknown-field + one round-trip case per listed record (read each record first; the literals above must match real constructor signatures — that is part of the task, and any mismatch found IS the drift this suite exists to surface).
|
||||
2. Run, expect FAIL (missing Newtonsoft reference → compile error):
|
||||
```bash
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter "FullyQualifiedName~ClusterClientContractLock"
|
||||
```
|
||||
3. Add the `PackageVersion` to `Directory.Packages.props` and the `PackageReference` to the Commons.Tests csproj; finish the test bodies.
|
||||
4. Run, expect PASS:
|
||||
```bash
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter "FullyQualifiedName~ClusterClientContractLock"
|
||||
```
|
||||
5. Commit:
|
||||
```bash
|
||||
git add Directory.Packages.props tests/ZB.MOM.WW.ScadaBridge.Commons.Tests
|
||||
git commit -m "test(commons): contract-lock tests for top ClusterClient records — enforce additive-only evolution under version skew (arch-review 08 §1.8)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 10: Docs-code drift + conventions cleanup batch (Low findings, consolidated)
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 2, 3, 4, 5, 6, 7, 8, 9, 11 (NOT 1 — both edit CLAUDE.md)
|
||||
**Files:**
|
||||
- Modify: `docs/requirements/Component-Commons.md` (layout tree, lines ~293–294)
|
||||
- Modify: `docs/requirements/Component-ConfigurationDatabase.md` (repositories section)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ZB.MOM.WW.ScadaBridge.ManagementService.csproj` (line 22)
|
||||
- Modify: `README.md` (line ~111, the "One doc per component" claim)
|
||||
- Modify: `docs/components/README.md` (index)
|
||||
- Modify: `CLAUDE.md` (Document Conventions section)
|
||||
|
||||
Concrete edits (each verified against current file state):
|
||||
|
||||
1. **Component-Commons.md — add `Observability/`** (§1.1, narrowed: `Serialization/` and `Validators/` are already documented at lines 293–294; only `Observability/` — which contains `ScadaBridgeTelemetry.cs` — is missing). In the layout tree, before the `├── Serialization/` line add:
|
||||
```
|
||||
├── Observability/ # ScadaBridgeTelemetry (ActivitySource/metrics names)
|
||||
```
|
||||
(Adjust the comment after reading `src/ZB.MOM.WW.ScadaBridge.Commons/Observability/ScadaBridgeTelemetry.cs` — describe what it actually holds.)
|
||||
2. **Component-ConfigurationDatabase.md — document the site-side repository exception** (§1.3). In the section stating repository implementations live in Configuration Database, add:
|
||||
> Site-side exception: read-only SQLite-backed implementations of shared repository interfaces live in `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/` (currently `SiteExternalSystemRepository`) — same Commons interface, site-local store, writes throw `NotSupportedException` because site artifacts are managed exclusively by deployment from Central.
|
||||
|
||||
(If Task 6 has not run yet, mention `SiteNotificationRepository` too and let Task 6's doc step remove it; if running after Task 6, list only the external-system repository.)
|
||||
3. **ManagementService.csproj line 22** — change `..\ZB.MOM.WW.ScadaBridge.InboundAPI\ZB.MOM.WW.ScadaBridge.InboundAPI.csproj` to `../ZB.MOM.WW.ScadaBridge.InboundAPI/ZB.MOM.WW.ScadaBridge.InboundAPI.csproj` (§1.4 cosmetic; every other reference in the repo uses `/`).
|
||||
4. **README.md ~line 111** (§4): replace the claim "One doc per component (plus the shared TreeView)" with an accurate scope, e.g.:
|
||||
> Reference docs exist for 24 of the 27 components (plus the shared TreeView); ScriptAnalysis (#25), KpiHistory (#26), and DelmiaNotifier (#27) are pending — tracked in `docs/plans/2026-07-08-deferred-work-register.md`.
|
||||
5. **docs/components/README.md**: add the same three-pending note to the index so the two lists can't drift independently.
|
||||
6. **CLAUDE.md Document Conventions** (§4 DelmiaNotifier): add one bullet:
|
||||
> - Exception: DelmiaNotifier (#27) is an external client tool, not a cluster component; its spec is the project README (`src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md`) plus `docs/plans/2026-06-26-delmia-recipe-notifier-design.md` — there is intentionally no `Component-DelmiaNotifier.md`.
|
||||
7. Verify (config/docs task — verification replaces the failing test):
|
||||
```bash
|
||||
dotnet build src/ZB.MOM.WW.ScadaBridge.ManagementService # csproj separator change still builds
|
||||
grep -n "Observability" docs/requirements/Component-Commons.md # expect: 1 hit in the tree
|
||||
grep -rn '\\\\ZB.MOM' src --include="*.csproj" # expect: no hits
|
||||
grep -n "24 of the 27\|pending" README.md docs/components/README.md # expect: scoped claim present
|
||||
grep -n "Component-DelmiaNotifier" CLAUDE.md # expect: the exception bullet
|
||||
```
|
||||
8. Commit:
|
||||
```bash
|
||||
git add docs/requirements/Component-Commons.md docs/requirements/Component-ConfigurationDatabase.md src/ZB.MOM.WW.ScadaBridge.ManagementService/ZB.MOM.WW.ScadaBridge.ManagementService.csproj README.md docs/components/README.md CLAUDE.md
|
||||
git commit -m "docs: close arch-review 08 drift batch — Commons Observability folder, site-repo exception, csproj separator, docs/components scope, DelmiaNotifier convention exception"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 11: Deferred-work register — triage the 23-item inventory into a tracked list
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 2, 3, 4, 5, 6, 7, 8, 9 (write after 10 only if you want the README pointer target to exist first — create this file before or simultaneously; the README in Task 10 points here)
|
||||
**Files:**
|
||||
- Create: `docs/plans/2026-07-08-deferred-work-register.md`
|
||||
|
||||
Turn the review's §3 inventory (23 items) into the explicit tracked register the review asked for: every item gets a row with **status** (`fix-now → plan NN` / `deferred`), **owner/where-noted**, **rationale**, and a **revisit trigger** (the concrete condition that reopens it). This is the single place the "consciously logged" deferrals live from now on; new deferrals get appended here instead of scattering.
|
||||
|
||||
1. Create `docs/plans/2026-07-08-deferred-work-register.md` with this exact structure and content (source: `archreview/08-conventions-tests-underdeveloped.md` §3; carry each item's file/line citation over):
|
||||
|
||||
```markdown
|
||||
# Deferred-Work Register (established 2026-07-08, from architecture review 08)
|
||||
|
||||
Single tracked list of consciously-deferred work. Rules: every deferral gets a row
|
||||
(rationale + revisit trigger); fix-now items reference the archreview plan that owns
|
||||
them and are removed from this table when that plan's task lands.
|
||||
|
||||
## Fix-now (owned by archreview plans)
|
||||
| # | Item | Owner |
|
||||
|---|------|-------|
|
||||
| 1 | Transport: Area membership doesn't travel in bundles (EntitySerializer.cs:251,518) | PLAN-05 |
|
||||
| 2 | CLI.Tests not in slnx | PLAN-08 Task 1 |
|
||||
| 3 | AuditLog reconciliation dials site NodeA only (SiteEnumerator.cs:32,64) | PLAN-04 |
|
||||
| 4 | Options validation missing on ~12 components | PLAN-08 Tasks 2–5 |
|
||||
| 5 | SmsConfiguration.MaxRetryCount stored but not honored | PLAN-06 |
|
||||
| 6 | Role-check case-sensitivity asymmetry (ManagementActor.cs:1910) | PLAN-07 |
|
||||
| 23 | Vestigial site-side notification-list surface | PLAN-08 Task 6 |
|
||||
|
||||
## Deferred (with rationale + revisit trigger)
|
||||
| # | Item | Where noted | Rationale for deferral | Revisit trigger |
|
||||
|---|------|-------------|------------------------|-----------------|
|
||||
| 7 | SecuredWrite audit rows leave SourceNode NULL | CLAUDE.md Security | Logged follow-up; low-volume channel | First per-node forensic query against secured writes |
|
||||
| 8 | Hash-chain tamper evidence (T1); CLI verify-chain is a no-op stub | audit-log roadmap :12 | v1.x by locked decision; append-only DB roles are the control | Compliance requirement for cryptographic tamper evidence |
|
||||
| 9 | Parquet audit archival (T2); endpoint returns 501 | AuditEndpoints.cs:204 | v1.x; 501 + CLI messaging are honest | AuditLog partition volume nears retention ceiling |
|
||||
| 10 | Aggregated live alarm stream for Alarm Summary | m7 design :252 | Snapshot fan-out acceptable at current instance counts | Alarm Summary latency complaints or >~50 instances/site |
|
||||
| 11 | Central-persisted OPC UA cert-trust audit | m7 follow-ups | Broadcast-to-both-nodes covers HA | Governance/audit requirement for trust decisions |
|
||||
| 12 | Native-alarm-source-override CSV import | m7 follow-ups | Parity gap only; attribute CSV shipped | First bulk native-alarm rollout request |
|
||||
| 13 | WaitForAttribute quality-gated mode | ScriptRuntimeContext.cs:405 | Planned enhancement per spec §4.2 | First script needing Good-only waits |
|
||||
| 14 | WaitForAttribute in Test-Run sandbox | waitfor-deferred :222 | Test-Run parity gap; production unaffected | Author feedback on Test-Run fidelity |
|
||||
| 15 | BrowseNext final-page signal not surfaced | BrowseCommands.cs:34 | One wasted round-trip | UX complaint on browse paging |
|
||||
| 16 | StubOpcUaClient throws on browse | m7 design :245 | Limits offline UI coverage only | Next browse/search UI regression |
|
||||
| 17 | Unified notifications+site-calls outbox page | stillpending :118 | Explicit M9 decision to keep two pages | Operator confusion reports |
|
||||
| 18 | Folder drag-drop | same, [PERM] | Permanently closed; menu reorder shipped | — (closed) |
|
||||
| 19 | Bundle signing / cluster-to-cluster pull / differential bundles | transport-design :402 | v1 manifest hash + AES-GCM held sufficient | Non-repudiation requirement across orgs |
|
||||
| 20 | Deployment EXPIRED-row purge | DeploymentManagerRepository.cs:310 | Read path already compensates | EXPIRED rows visible in ops queries |
|
||||
| 21 | SiteAuditBacklogReporter threshold consolidation | SiteAuditBacklogReporter.cs:28 | Config-shape cleanup only | Next SqliteAuditWriterOptions change |
|
||||
| 22 | KPI history hourly rollups | Component-KpiHistory.md | YAGNI; 90-day retention bounds table | KpiSample query latency on dashboards |
|
||||
|
||||
## New deferrals from review 08 (this plan)
|
||||
| Item | Rationale | Revisit trigger |
|
||||
|------|-----------|-----------------|
|
||||
| Communication → HealthMonitoring layering (ICentralHealthAggregator consumed by CentralCommunicationActor.cs:351) | Moving the interface + SiteHealthState to Commons ripples across 5 projects for a cosmetic inversion | Next breaking change to ICentralHealthAggregator |
|
||||
| docs/components reference docs for ScriptAnalysis, KpiHistory, DelmiaNotifier | Reference docs are substantial (StyleGuide-conformant); README claim scoped instead (PLAN-08 Task 10) | Next doc-writing session touching those components |
|
||||
| Test-coverage backfill: SiteCallAudit.Tests (31 tests/1.6k LOC), DeploymentManager.Tests | No defect identified; coverage partly lives in ManagementService/Host/Integration suites | First regression escaping either component |
|
||||
| Failover-timing + broader perf envelope (S&F drain rate, per-subscriber backpressure) | Needs PLAN-01 two-node rig; placeholder harness shipped (PLAN-08 Task 8) | PLAN-01 rig landing |
|
||||
```
|
||||
2. Verify (docs task):
|
||||
```bash
|
||||
grep -c "^| " docs/plans/2026-07-08-deferred-work-register.md
|
||||
```
|
||||
Expected: ≥ 30 table rows (7 fix-now + 16 deferred + 4 new + headers).
|
||||
3. Commit:
|
||||
```bash
|
||||
git add docs/plans/2026-07-08-deferred-work-register.md
|
||||
git commit -m "docs(plans): deferred-work register — triage the 23-item arch-review inventory into fix-now (plan-owned) vs deferred-with-rationale"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dependencies on other plans
|
||||
|
||||
- **Plan 01 (Cluster/Host/Failover):** Task 8 is a placeholder that documents the failover-timing measurement protocol; the actual two-node kill rig is Plan 01's (overall review P2-10). When Plan 01's rig lands, un-skip `FailoverTimingTests` and wire it in. No file overlap expected, but Plan 01 may touch `Host/Program.cs` (Task 4 also edits it — coordinate merge order).
|
||||
- **Plan 05 (Templates/Deploy/Transport):** owns the Transport `AreaName: null` contradiction (§3 item 1 / §4). Plan 05 also touches `Transport/`; Task 5 here only touches `Transport/ServiceCollectionExtensions.cs` + a new validator file — small merge-conflict surface.
|
||||
- **Plan 04 (Data/Audit):** owns AuditLog NodeA-only reconciliation (§3 item 3).
|
||||
- **Plan 06 (Edge integrations):** owns `SmsConfiguration.MaxRetryCount` (§3 item 5).
|
||||
- **Plan 07 (UI/Management/Security):** owns role-casing asymmetry (§3 item 6). Plan 07 may also touch `ManagementActor.cs`; Task 4 here touches only `ManagementService/ServiceCollectionExtensions.cs` + csproj.
|
||||
- The register (Task 11) cites plan numbers 04/05/06/07 — content-only references, no execution dependency.
|
||||
|
||||
## Execution order
|
||||
|
||||
1. **Task 1 first** (slnx) — it changes what `dotnet test ZB.MOM.WW.ScadaBridge.slnx` covers, so every later task's solution-level verification includes CLI.Tests.
|
||||
2. **Tasks 2, 4, 5, 7, 8, 9, 11 in parallel** (disjoint files).
|
||||
3. **Task 3 then Task 6 sequentially** (both live in the SiteRuntime project; either order works, but running 6 after 3 keeps the CompositionRoot test churn in one direction).
|
||||
4. **Task 10 last among docs** (it edits CLAUDE.md — serialize with Task 1's CLAUDE.md edit — and its README pointer targets the Task 11 register, so run after 11).
|
||||
5. Finish with a full `dotnet build ZB.MOM.WW.ScadaBridge.slnx && dotnet test ZB.MOM.WW.ScadaBridge.slnx` — expect all 30 test projects (now including CLI.Tests' ~279) green.
|
||||
@@ -0,0 +1,72 @@
|
||||
{
|
||||
"planPath": "archreview/plans/PLAN-08-conventions-tests.md",
|
||||
"tasks": [
|
||||
{
|
||||
"id": 1,
|
||||
"subject": "Task 1: Add CLI.Tests to the slnx and retire the gotcha documentation",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"subject": "Task 2: Options validation — Communication + DataConnectionLayer",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"subject": "Task 3: Options validation — site pipeline (SiteRuntime, StoreAndForward, SiteEventLogging)",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"subject": "Task 4: Options validation — edge + management (InboundAPI, ExternalSystemGateway, NotificationService, ManagementService)",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"subject": "Task 5: Options validation — central components (Transport, SiteCallAudit, DeploymentManager)",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"subject": "Task 6: Remove the vestigial site-side notification-list surface",
|
||||
"status": "pending",
|
||||
"blockedBy": [3]
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"subject": "Task 7: PerformanceTests — site-stream throughput test",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"subject": "Task 8: PerformanceTests — failover-timing benchmark harness placeholder",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 9,
|
||||
"subject": "Task 9: Contract-lock tests for top ClusterClient message records",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": 10,
|
||||
"subject": "Task 10: Docs-code drift + conventions cleanup batch (Low findings, consolidated)",
|
||||
"status": "pending",
|
||||
"blockedBy": [1, 11]
|
||||
},
|
||||
{
|
||||
"id": 11,
|
||||
"subject": "Task 11: Deferred-work register — triage the 23-item inventory into a tracked list",
|
||||
"status": "pending",
|
||||
"blockedBy": []
|
||||
}
|
||||
],
|
||||
"lastUpdated": "2026-07-08"
|
||||
}
|
||||
Reference in New Issue
Block a user