docs(archreview): round-2 re-review (2026-07-12) + 8 fix plans (86 tasks)
Re-ran all 8 domain reviews at HEAD8c888f13against theb910f5ebbaseline: every round-1 finding source-verified (168 fixed, 0 regressions, 0 false claims); 56 new findings (1 Critical / 4 High / 15 Medium / 36 Low), concentrated in post-baseline code (anti-entropy resync, KPI rollup backfill, live alarm stream) and seams the fixes exposed. Headliners: S&F resync predicate inversion can wipe the delivering node's buffer (02-N1 Critical); resync snapshot exceeds the Akka remoting frame size (02-N2); failover drill kills the one node keep-oldest can't survive (01-N1); unbounded rollup backfill per failover (04-R1); live production API key in untracked test.txt (08-NF1). Adds PLAN-R2-01..08 + .tasks.json manifests and the Round-2 board, P0 list, cross-plan mutexes, and wave order in 00-MASTER-TRACKER.
This commit is contained in:
Joseph Doherty
@@ -1,180 +1,126 @@
|
||||
# Architecture Review 04 — Central Data Layer & Audit/KPI Backbone
|
||||
# Architecture Review 04 — Central Data Layer & Audit/KPI Backbone (Round 2)
|
||||
|
||||
**Date:** 2026-07-12 (round 2; round-1 baseline commit `b910f5eb`, current HEAD `8c888f13`)
|
||||
|
||||
**Scope reviewed:**
|
||||
|
||||
- `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase` — EF Core mappings, repositories, migrations, `IAuditService`, partition maintenance
|
||||
- `src/ZB.MOM.WW.ScadaBridge.Commons` — as the data-shape layer (entities, repository interfaces, audit enums/types, KPI types)
|
||||
- `src/ZB.MOM.WW.ScadaBridge.AuditLog` — central ingest/purge/reconciliation actors, `CentralAuditWriter`, site SQLite writer + telemetry
|
||||
- `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox` — `NotificationOutboxActor`, repository, delivery adapters, KPI source
|
||||
- `src/ZB.MOM.WW.ScadaBridge.SiteCallAudit` — `SiteCallAuditActor`, monotonic upsert repository, reconciliation, relay
|
||||
- `src/ZB.MOM.WW.ScadaBridge.KpiHistory` — `KpiHistoryRecorderActor`, `KpiHistoryRepository`, bucketer
|
||||
- Design docs: `docs/requirements/Component-ConfigurationDatabase.md`, `Component-AuditLog.md`, `Component-NotificationOutbox.md`, `Component-SiteCallAudit.md`, `Component-KpiHistory.md`, `Component-Commons.md`
|
||||
- Test projects: `AuditLog.Tests`, `NotificationOutbox.Tests`, `SiteCallAudit.Tests`, `KpiHistory.Tests`, `ConfigurationDatabase.Tests`
|
||||
- `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase` — EF Core mappings, repositories, migrations (5 new since baseline), `IAuditService`, partition maintenance
|
||||
- `src/ZB.MOM.WW.ScadaBridge.AuditLog` — central ingest/purge/reconciliation actors, pull clients, site SQLite writer + telemetry + new retention job
|
||||
- `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox` — dispatcher (now bounded-parallel), repository (single-query KPIs), delivery adapters
|
||||
- `src/ZB.MOM.WW.ScadaBridge.SiteCallAudit` — actor (composite keyset reconciliation cursor, pinned event, relay identity), repository (same-rank tiebreaker, RECOMPILE)
|
||||
- `src/ZB.MOM.WW.ScadaBridge.KpiHistory` — recorder actor **including the post-initiative hourly-rollup subsystem shipped 2026-07-10** (`KpiRollupHourly`, `kpi-rollup` tick, backfill, `KpiMetricAggregationCatalog`, raw-vs-rollup routing) — brand-new code no round-1 review saw
|
||||
- Design docs: `Component-ConfigurationDatabase.md`, `Component-AuditLog.md`, `Component-NotificationOutbox.md`, `Component-SiteCallAudit.md`, `Component-KpiHistory.md`, `Component-Commons.md`
|
||||
|
||||
## Maturity Verdict
|
||||
**Method:** read the round-1 report and `archreview/plans/PLAN-04-data-audit-backbone.md` in full; verified every round-1 finding's true disposition in current source (file:line, not trusting the plan's claims); fresh static sweep of `git diff b910f5eb..HEAD` scoped to the five projects + new migrations, with the rollup subsystem reviewed line-by-line. No builds/tests run (suites verified green 2026-07-10).
|
||||
|
||||
This is an unusually mature data layer for a system at this stage: idempotent ingest is implemented correctly and defensively everywhere (duplicate-key race handling in all three insert-if-not-exists paths), the combined cached-telemetry dual-write really is transactional (verified: `AuditLogIngestActor.OnCachedTelemetryAsync` shares one scoped `DbContext` across both repositories under `BeginTransactionAsync`), UTC discipline is enforced with converters and `SpecifyKind` re-tagging at every ADO.NET boundary, actor/DB interaction follows a consistent scope-per-message + PipeTo pattern with in-flight guards, and MSSQL-backed integration tests exist for the hard parts (partition purge, execution-tree walks, outage reconciliation). The weaknesses are concentrated in **scale-dependent operational behavior** rather than logic: the partition purge's full-table unique-index rebuild will start failing on default command timeouts as the table grows, several KPI predicates have no supporting index and full-scan year-sized tables every 60 seconds, the execution-tree query scans the entire AuditLog per invocation, the spec'd site SQLite 7-day retention purge simply does not exist, the append-only DB-role story is DDL-only (one connection principal runs both writer and maintenance paths), and the persisted `backlogTotal` KPI trend is hardwired to zero. None of these bites at demo scale; several will bite within months of production volume.
|
||||
**Round-1 vs round-2 verdict:** every round-1 time bomb was genuinely defused — all 12 Fixed dispositions verified in code, deferrals are honestly documented in the design docs rather than silently dropped — and the fix quality is uniformly high (the NodeB-failover pull clients and the keyset cursor are exemplary); the one new material risk is concentrated in the brand-new rollup **backfill** path, which re-introduces at startup exactly the "unbounded single-pass over a year-scale table" shape the plan spent five tasks eliminating elsewhere.
|
||||
|
||||
---
|
||||
|
||||
## Findings — Stability
|
||||
## Round-1 Finding Disposition
|
||||
|
||||
### [High] Site SQLite 7-day retention purge is specified but not implemented — unbounded site DB growth
|
||||
Every disposition below was verified against current source, not the plan's coverage table.
|
||||
|
||||
`Component-AuditLog.md:412-414` specifies: "Sites: daily site job; default 7-day retention (configurable, min 1, max 90). Respects the hard `ForwardState` invariant." No such job exists. There is no `DELETE` against `audit_event` / `audit_forward_state` anywhere in `src/` — the only destructive statement is the one-time legacy-table `DROP TABLE IF EXISTS AuditLog` at startup (`src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriter.cs:198`). The writer's own comments treat the purge as future work: `SqliteAuditWriter.cs:145` ("a future `PRAGMA incremental_vacuum` shrink the file after the 7-day retention purge") and `:33` claims the store is "recreated per deployment", which is not a retention mechanism.
|
||||
| # | Round-1 finding | R1 sev | Disposition | Evidence (file:line) |
|
||||
|---|-----------------|--------|-------------|----------------------|
|
||||
| S1/U2 | Site SQLite 7-day retention purge missing | High | **Fixed (verified)** | `SqliteAuditWriter.PurgeExpiredAsync` — temp-table purge honoring the ForwardState invariant (`AuditLog/Site/SqliteAuditWriter.cs:858-966`), post-purge `incremental_vacuum` (:949-961); `SiteAuditRetentionService` hosted loop (`Site/SiteAuditRetentionService.cs:62-117`); options bound (`AuditLog/ServiceCollectionExtensions.cs:107`) and service registered (:335); clamped options (`Site/SiteAuditRetentionOptions.cs`); tests `SqliteAuditWriterRetentionTests.cs`, `SiteAuditRetentionServiceTests.cs`, `SiteAuditRetentionOptionsTests.cs` |
|
||||
| S2 | `SwitchOutPartitionAsync` no CommandTimeout; purge failure log-only | High | **Fixed (verified)** | `commandTimeout` parameter + `SetCommandTimeout` with finally-restore (`ConfigurationDatabase/Repositories/AuditLogRepository.cs:230, 368, 389-402`), same on per-channel `DELETE TOP` (:472); actor passes `ResolvedMaintenanceCommandTimeout` (default 30 min, 1-min floor — `AuditLogPurgeOptions.cs:73-81`) at `AuditLogPurgeActor.cs:200, 295`; failure now publishes `AuditLogPurgeFailedEvent` + increments the `IAuditPurgeFailureCounter` health counter (`AuditLogPurgeActor.cs:235-247`, `Central/AuditLogPurgeFailedEvent.cs`, `Central/IAuditPurgeFailureCounter.cs`) |
|
||||
| S3 | Append-only role DDL-only; purger grants insufficient | High | **Fixed (verified)** | Migration `20260709105112_FixAuditPurgerRoleGrants.cs:25-33` grants `CREATE TABLE` + scoped `DELETE ON dbo.AuditLog` (idempotent, reversible Down); docs now state honestly that default deployments enforce append-only via the CI grep guard, with the roles as optional DBA hardening (`Component-AuditLog.md:489-495`; `Component-ConfigurationDatabase.md:342` retracts the stale "no row DELETE even for purge" claim) |
|
||||
| S4 | SiteCalls upsert freezes RetryCount/LastError at rank | Medium | **Fixed (verified)** | Same-rank freshness tiebreaker on `UpdatedAtUtc`, with terminal rows deliberately immutable so a later terminal never overwrites an earlier one (`SiteCallAuditRepository.cs:106-165`) |
|
||||
| S5 | No EF `EnableRetryOnFailure` | Medium | **Fixed (verified)** | `EnableRetryOnFailure(5, 30s)` fleet-wide (`ConfigurationDatabase/ServiceCollectionExtensions.cs:33-36`); the combined-telemetry dual-write transaction is wrapped in `CreateExecutionStrategy().ExecuteAsync` per entry (`AuditLogIngestActor.cs:263-298`) |
|
||||
| S6 | Operator Retry emits no audit row; relay lacks operator identity | Medium | **Fixed (verified)** | `RetryAsync` emits a `Submitted` NotifyDeliver row with `RequestedBy` as actor (`NotificationOutboxActor.cs:1078-1126`); Discard passes it too (:1183); `SiteCallAuditActor` takes an optional `ICentralAuditWriter` and emits relay-identity rows (`SiteCallAuditActor.cs:104, 199, 288-291`) |
|
||||
| S7 | Reconciliation dead-ends (cursor pin; permanent abandonment) | Medium | **Fixed (verified)** | Composite `(UpdatedAtUtc, TrackedOperationId)` keyset cursor per site (`SiteCallAuditActor.cs:135-159, 635-713`); legacy-site pin now publishes `SiteCallReconciliationPinnedChanged` on transitions (:737-746, `SiteCallReconciliationPinnedChanged.cs`); site store honors strict keyset with legacy `>=` fallback (`SiteRuntime/Tracking/OperationTrackingStore.cs:365-395`; proto `sitestream.proto:187-192`; server maps empty→null `SiteStreamGrpcServer.cs:607-613`); abandonment writes a durable synthetic `AuditKind.ReconciliationAbandoned` row (`SiteAuditReconciliationActor.cs:333-385`, `AuditKind.cs:43`) |
|
||||
| S8 | Outbox repository dual SQLite/T-SQL dialect | Medium | **Deferred (accepted, documented)** | Convention recorded: new repositories are MSSQL-only with MSSQL fixtures; `NotificationOutboxRepository` is the named legacy exception, "do not copy" (`Component-ConfigurationDatabase.md:69`) |
|
||||
| S9 | Purge timers' first tick waits a full interval | Low | **Fixed (verified)** | Shared `PurgeTimerSchedule.InitialDelay` (`Commons/Types/PurgeTimerSchedule.cs`) used by all three: `AuditLogPurgeActor.cs:100`, `SiteCallAuditActor.cs:422`, `KpiHistoryRecorderActor.cs:191` |
|
||||
| S10 | `SqliteAuditWriter.Dispose` sync-over-async | Low | **Won't-fix (accepted)** | Per plan; round-1 itself judged the thread-pool-hop mitigation correct and documented |
|
||||
| P1 | SiteCalls KPI predicates full-scan (no `TerminalAtUtc` index) | High | **Fixed (verified)** | Filtered `IX_SiteCalls_NonTerminal` on `CreatedAtUtc WHERE [TerminalAtUtc] IS NULL` with `SourceSite`/`SourceNode`/`Status` includes (`SiteCallEntityTypeConfiguration.cs:90-99`; migration `20260709110614_AddSiteCallsNonTerminalIndex.cs`) |
|
||||
| P2 | `GetExecutionTreeAsync` scans entire AuditLog | High | **Fixed (verified)** | Edges anchor bounded to `[rootFirst − 1h, rootFirst + 7d)` (`AuditLogRepository.cs:682-683, 756, 846-852`); row-less-root degenerate case deliberately keeps the unbounded form |
|
||||
| P3 | AuditLog clustered key leads with random GUID | Medium | **Deferred (accepted, tracked)** | Recorded as a tracked benchmark follow-up with the partition-scheme rationale (`Component-ConfigurationDatabase.md:70`) |
|
||||
| P4 | KpiSample volume + unbatched purge | Medium | **Fixed (purge) / Deferred (cadence, tracked)** | Time-sliced (≤1h/DELETE) batched purge (`KpiHistoryRepository.cs:65-84`), backed by the standalone `IX_KpiSample_Captured` index (`KpiSampleEntityTypeConfiguration.cs:54-55`); per-node sampling cadence recorded as a tracked follow-up (`Component-ConfigurationDatabase.md:71`) |
|
||||
| P5 | Catch-all `(@p IS NULL OR col=@p)` predicates | Medium | **Fixed (verified)** | `OPTION (RECOMPILE)` on the SiteCalls query page (`SiteCallAuditRepository.cs:236`) |
|
||||
| P6 | Outbox KPI 5-7 round trips; unbounded oldest materialization; offset paging | Medium | **Fixed (KPIs) / Deferred (paging, tracked)** | Global/per-site/per-node snapshots are each one grouped conditional-aggregation query with server-side `MIN`-over-CASE — no entity materialization (`NotificationOutboxRepository.cs:244-270, 286-315, 330-374`); offset→keyset page conversion deferred per plan, recorded in `Component-ConfigurationDatabase.md:71` |
|
||||
| P7 | Non-persisted `IngestedAtUtc` computed column | Low | **Fixed (guard)** | "NEVER use in a WHERE predicate" comment guard (`AuditLogEntityTypeConfiguration.cs:159-164`) |
|
||||
| P8 | `MarkForwarded/ReconciledAsync` near SQLite param limit | Low | **Fixed (verified)** | ≤500-id chunks inside one transaction, both methods (`SqliteAuditWriter.cs:61-63, 657-673, 759-775`) |
|
||||
| C1 | Design docs drifted (enums, Teams, per-channel, SiteCalls schema) | Medium | **Fixed (verified)** | `AuditChannel` 5 values / `AuditKind` 15 incl. `ReconciliationAbandoned` (`Component-Commons.md:46-47`); `NotificationType` Email+Sms with Teams-dropped rationale (:90); per-channel retention correctly described (`Component-ConfigurationDatabase.md:110, 408`); SiteCalls schema rewritten to shipped Channel/AuditStatus shape with explicit "no Kind/TargetSummary/provenance columns" (`Component-SiteCallAudit.md:41-72`); no load-bearing "Teams" references remain |
|
||||
| C2 | `AuditLogRow` lives in ConfigurationDatabase, not Commons | Medium | **Deferred (accepted, documented)** | Deviation documented as intentional persistence-only projection of the external `ZB.MOM.WW.Audit.AuditEvent` contract, "do not lift it into Commons" (`Component-Commons.md:51`) |
|
||||
| C3 | Mixed timestamp CLR types | Low | **Deferred (accepted, documented)** | Convention recorded: new tables use UTC `DateTime` + converter; `Notification`'s `DateTimeOffset` is the documented legacy exception (`Component-ConfigurationDatabase.md:68`) |
|
||||
| C4 | Uncharted KPI metric names as string literals | Low | **Fixed (verified)** | `KpiMetrics.NotificationOutbox.StuckCount`/`OldestPendingAgeSeconds` constants consumed by the source (`NotificationOutboxKpiSampleSource.cs:39-43`) — but see new finding R4: the pattern partially recurs inside `KpiMetricAggregationCatalog` |
|
||||
| U1 | `backlogTotal` KPI history hardwired to zero | High (underdev.) | **Fixed (verified)** | `IAuditBacklogProvider` seam consumed with snapshot fallback (`AuditLogKpiSampleSource.cs:47, 66, 95`); `CentralHealthAuditBacklogProvider` registered centrally (`HealthMonitoring/ServiceCollectionExtensions.cs:72`); test `AuditLogKpiSampleSourceTests.cs` |
|
||||
| U3 | Hash-chain / Parquet archival | — | **Deferred (accepted)** | Unchanged v1.x deferral; no drift |
|
||||
| U4 | SecuredWrite rows leave `SourceNode` NULL | — | **Fixed (elsewhere, verified seam)** | `CentralAuditWriter` stamps `SourceNode` via optional `INodeIdentityProvider` (`Central/CentralAuditWriter.cs:47, 80`) — landed under plan 07 as predicted; doc under-promise on per-channel keys fixed in C1 pass |
|
||||
| U5 | Reconciliation keyset upgrade untracked | — | **Fixed (SiteCalls) / Deferred (AuditLog pull, tracked)** | SiteCalls keyset shipped (see S7); `PullAuditEvents` remains inclusive-`>=` (idempotent on `EventId`) with the deferral recorded (`Component-ConfigurationDatabase.md:71`) |
|
||||
| U6 | Dispatcher throughput ceiling (sequential delivery) | — | **Fixed (verified)** | `SemaphoreSlim`-gated `Task.WhenAll` with per-notification DI scope, `MaxParallelDeliveries` default 4 clamped ≥1 (`NotificationOutboxActor.cs:288-408`; `NotificationOutboxOptions.cs:62-69`); fault isolation per delivery so `WhenAll` never throws |
|
||||
| U7 | Test gaps mirror findings | — | **Fixed (verified)** | Retention tests exist (S1 row); `PartitionPurgeTests` extended for the timeout path per plan; backlog-provider test exists |
|
||||
| X1 | Reconciliation dials site NodeA only (owned from report 08) | — | **Fixed (verified)** | `SiteEntry.FallbackGrpcEndpoint` populated from NodeB (`Central/SiteEnumerator.cs:34-41`); both pull clients retry once against the fallback on transport faults only, never on mapping faults, skipping when cancelled (`GrpcPullSiteCallsClient.cs:110-127, 172-211`; `GrpcPullAuditEventsClient.cs:194`) |
|
||||
|
||||
**Failure scenario:** a long-lived site node accumulates every Forwarded/Reconciled audit row forever. At even modest script activity (10 events/s ≈ 860k rows/day) the SQLite file grows by GBs per month; `GetBacklogStatsAsync`'s `COUNT(*)` stays cheap (it filters `Pending`), so nothing surfaces on the health dashboard — the disk just fills. The `auto_vacuum = INCREMENTAL` pragma (`SqliteAuditWriter.cs:149`) is set up for a purge that never runs.
|
||||
|
||||
### [High] `SwitchOutPartitionAsync` — full-table unique-index rebuild inside one transaction, no command timeout
|
||||
|
||||
`src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/AuditLogRepository.cs:257-332`: the monthly partition switch drops `UX_AuditLog_EventId`, switches, then `CREATE UNIQUE NONCLUSTERED INDEX UX_AuditLog_EventId ON dbo.AuditLog (EventId)` — a scan+sort over **every remaining row of the table** (up to 365 days of audit events), all inside a single transaction executed via `ExecuteSqlRawAsync`.
|
||||
|
||||
Three compounding problems:
|
||||
|
||||
1. **Default 30s command timeout.** No `CommandTimeout` is configured anywhere (`ServiceCollectionExtensions.cs:29-34` sets only `UseSqlServer(connectionString)`; repo-wide grep confirms no `CommandTimeout` on the data layer). Once the table is large enough that the index rebuild exceeds 30 seconds, the batch times out, the CATCH block rolls back and re-`THROW`s, `AuditLogPurgeActor.OnTickAsync` logs and moves on (`AuditLogPurgeActor.cs:198-206`) — and the same failure repeats every daily tick forever. Retention silently stops; the table grows past 365 days; the rebuild gets *slower* each day. This is a self-reinforcing failure with no alerting beyond an error log line.
|
||||
2. **Write blockage.** The transaction holds Sch-M from the DROP INDEX through the CREATE INDEX; all ingest (`InsertIfNotExistsAsync`, dual-writes, direct writes) blocks for the full rebuild duration. The actors tolerate this (rows stay Pending at sites), but central direct-write paths (Inbound API middleware, dispatcher audit emission) will eat the latency or the swallow-and-count path.
|
||||
3. **Uniqueness window.** While `UX_AuditLog_EventId` is absent, the `IF NOT EXISTS` probe becomes an unindexed scan and single-column EventId uniqueness is enforced only via the composite PK. In practice a retried event carries the same `OccurredAtUtc` so the PK still dedupes; the residual risk is small but the per-insert scan cost during the window is not.
|
||||
|
||||
**Recommendation:** set a long `CommandTimeout` for the maintenance path (or run the rebuild with `ONLINE = ON` where edition allows, outside the switch transaction), and monitor purge success as a health metric, not just a log line. Long term, consider making `UX_AuditLog_EventId` unnecessary (see the clustered-key finding under Performance).
|
||||
|
||||
### [High] Append-only DB-role enforcement is DDL-only — one principal runs writer and maintenance paths
|
||||
|
||||
The migrations create `scadabridge_audit_writer` (INSERT+SELECT, DENY UPDATE/DELETE) and `scadabridge_audit_purger` (SELECT + ALTER ON SCHEMA::dbo) — `20260520142214_AddAuditLogTable.cs:146-158`, re-granted in `20260602174346_CollapseAuditLogToCanonical.cs:212-223`. But the application registers **one** `ScadaBridgeDbContext` with **one** connection string (`ServiceCollectionExtensions.cs:29-45`); the same principal executes `InsertIfNotExistsAsync` (writer path), `PurgeChannelOlderThanAsync`'s `DELETE TOP` (`AuditLogRepository.cs:409-410`), `BackfillSourceNodeAsync`'s `UPDATE TOP` (`AuditLogRepository.cs:851-852`), and the switch-out DDL. Consequences:
|
||||
|
||||
- If the runtime login were actually placed in `scadabridge_audit_writer`, the DENY would break the purge/backfill on the same connection. So in any real deployment the login must hold INSERT+UPDATE+DELETE+DDL — meaning the append-only guarantee is enforced **only** by the CI grep guard (`tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/AuditLogAppendOnlyGuardTests.cs`), not by the database.
|
||||
- The purger role as granted cannot even perform the purge: `CREATE TABLE` (staging) requires the database-level `CREATE TABLE` permission, which `ALTER ON SCHEMA::dbo` alone does not confer.
|
||||
|
||||
`Component-AuditLog.md:417-424` presents the role separation as the enforcement mechanism ("The application accesses AuditLog via a dedicated DB role"). Spec and code have drifted: either wire a second connection string/principal for the maintenance path (and fix the purger grants), or re-document enforcement honestly as "CI guard + code review, roles provided for DBAs who choose to segregate principals."
|
||||
|
||||
### [Medium] SiteCalls monotonic upsert drops mid-lifecycle progress (RetryCount / LastError frozen at rank)
|
||||
|
||||
`SiteCallAuditRepository.cs:30-40` ranks `Attempted == Skipped == 2` and the UPDATE fires only when `incomingRank > storedRank` (`:128-138`). A cached call retrying 20 times emits 20 `Attempted` telemetry packets with increasing `RetryCount` and fresher `LastError`/`HttpStatus` — every one after the first is a **no-op** ("first-write-wins at each rank" per the comment). The Central UI Site Calls page therefore shows `RetryCount = 1` and the first error for the entire retry phase, until the terminal packet lands. The spec's intent (`Component-SiteCallAudit.md:76-78`, "upsert-on-newer-status... at-least-once and out-of-order telemetry are harmless") is about status regression, not about freezing progress fields. A safe refinement: within equal rank, apply the update when `incoming.UpdatedAtUtc > stored.UpdatedAtUtc` (still idempotent and regression-proof).
|
||||
|
||||
### [Medium] No EF Core connection resiliency (`EnableRetryOnFailure`)
|
||||
|
||||
`ServiceCollectionExtensions.cs:31` configures `UseSqlServer(connectionString)` with no execution strategy. The audit/dispatcher actors compensate with their own retry loops (rows stay Pending, next tick retries), so the backbone self-heals — but every *read-side* path (KPI requests, outbox queries, `Notify.Status` round-trips, execution-tree queries) surfaces transient faults (failover of the SQL instance, connection pool churn) directly to the caller as a failed response. One line of `sqlOptions.EnableRetryOnFailure(...)` would cover the fleet. (Note: retry strategies and user-initiated transactions interact — `OnCachedTelemetryAsync`'s `BeginTransactionAsync` would need `CreateExecutionStrategy` wrapping — so adopt deliberately.)
|
||||
|
||||
### [Medium] Operator Retry on a parked notification emits no audit row
|
||||
|
||||
`Component-NotificationOutbox.md:128`: "Operator Retry/Discard still mutates only the Notifications row, and each transition emits the corresponding `Notification.Attempt` / `Notification.Terminal` audit row." `DiscardAsync` does emit (`NotificationOutboxActor.cs:1021`), but `RetryAsync` (`NotificationOutboxActor.cs:951-977`) resets Parked→Pending with **no** audit emission. Forensically, a parked notification that an operator retried and that then delivered shows an unexplained `Parked → Attempted → Delivered` sequence with no record of who un-parked it or when. Same gap for `SiteCallAudit`'s retry/discard relay (site-side telemetry covers the state change but not the operator identity).
|
||||
|
||||
### [Medium] Reconciliation edge cases are detected but dead-ended
|
||||
|
||||
- **Single-timestamp cursor pin** (`SiteCallAuditActor.cs:636-654`): when > `ReconciliationBatchSize` rows share one `UpdatedAtUtc` the cursor cannot advance; the code correctly detects and logs it, but the backlog tail then **never** reconciles ("will not reconcile until those rows' timestamps differ") and nothing surfaces beyond a log Warning — no health metric, no stalled event (unlike the sibling `SiteAuditReconciliationActor`, which publishes `SiteAuditTelemetryStalledChanged`). The fix the log message itself names — a composite `(timestamp, id)` keyset in the pull contract — is deferred without a tracking artifact.
|
||||
- **Permanent row abandonment** (`SiteAuditReconciliationActor.cs:283-293`): after 5 failed inserts a row is dropped with `LogCritical` and the cursor advances. For a compliance-oriented audit log, "we permanently lost EventId X" deserves a durable record (e.g., a synthetic `Skipped` audit row or a health counter), not only a log line that rotates away.
|
||||
|
||||
### [Medium] `NotificationOutboxActor` ingest ack path can ack before the row is *committed* under a SQLite test provider mismatch — verify provider parity
|
||||
|
||||
`NotificationOutboxRepository.InsertIfNotExistsAsync` maintains two dialects (`NotificationOutboxRepository.cs:70-95` SQLite `INSERT OR IGNORE`, `:99-117` T-SQL `IF NOT EXISTS`). This is correct today but is production code carrying a test-only branch, and the two branches have subtly different concurrency semantics (the SQLite path can never observe the duplicate-key race the SQL Server path handles). The AuditLog and SiteCall repositories chose the opposite strategy (MSSQL-only SQL, MSSQL-backed test fixtures). Pick one convention; the divergence invites the next repository to guess.
|
||||
|
||||
### [Low] Purge actors' first tick waits a full interval
|
||||
|
||||
`AuditLogPurgeActor.PreStart` (`:95-100`), `SiteCallAuditActor.StartPurgeTimer` (`:380-386`), and `KpiHistoryRecorderActor` purge timer (`KpiHistoryRecorderActor.cs:124-127`) all use `initialDelay = interval` (24h). A central node that restarts daily (crash loop, scheduled recycling) would *never* purge. Low likelihood, cheap fix (short initial delay; purges are idempotent).
|
||||
|
||||
### [Low] `SqliteAuditWriter.Dispose` sync-over-async
|
||||
|
||||
`SqliteAuditWriter.cs:890-894` blocks on `DisposeAsync` via `Task.Run(...).GetAwaiter().GetResult()`. The mitigation (thread-pool hop) is correct and documented; flagged only because it remains a blocking dispose on a hot-path component.
|
||||
**Disposition counts:** 19 Fixed (verified), 2 Fixed-with-documented-deferral remainder (P4, P6, U5 counted once each as Fixed), 6 Deferred (accepted, all now documented/tracked: S8, P3, C2, C3, U3), 1 Won't-fix (accepted: S10), 0 Not fixed, 0 Regressed. No plan claim was found to be false.
|
||||
|
||||
---
|
||||
|
||||
## Findings — Performance
|
||||
## New Findings — KPI Rollup Subsystem (post-initiative, 2026-07-10)
|
||||
|
||||
### [High] SiteCalls KPI predicates full-scan the table — no index on `TerminalAtUtc`
|
||||
The rollup design is sound where it was thought through — the gauge-vs-rate catalog with its safe-Gauge default and the corrected `*LastHour` classification (`KpiMetricAggregation.cs:85-96`) is genuinely careful work, the `HasFilter(null)` suppression of EF's default filtered unique index (`KpiRollupHourlyEntityTypeConfiguration.cs:57-63`) dodges a real null-ScopeKey uniqueness trap, and the exclusive-upper-bound / in-progress-hour exclusion is correct. The problems are concentrated where the periodic-fold assumptions meet the backfill path.
|
||||
|
||||
The "schema-honest" non-terminal predicate is `TerminalAtUtc IS NULL` (`SiteCallAuditRepository.cs:224-232`), used by `ComputeKpisAsync` (buffered `:243-244`, stuck `:259-260`, oldest `:262-268`), the per-site and per-node variants (`:286-307`, `:336-362`), and `PurgeTerminalAsync` (`:213-218`). But the table's only indexes are `(SourceSite, CreatedAtUtc)` and `(Status, UpdatedAtUtc)` (`SiteCallEntityTypeConfiguration.cs:79-87`) — **nothing covers `TerminalAtUtc`**. Every such count is a clustered scan of a table retained for 365 days (`SiteCallAuditOptions.RetentionDays = 365`, potentially millions of rows for busy sites). These queries run:
|
||||
### [High] R1 — The startup backfill folds the entire raw-retention window in one unbounded pass, on every failover
|
||||
|
||||
- every 60 s from `SiteCallAuditKpiSampleSource` (global + per-site + per-node = ~9 scanning queries per recorder tick),
|
||||
- on every Health-dashboard poll (10 s cadence per the UI design) via `SiteCallKpiRequest`.
|
||||
`FoldHourlyRollupsAsync` loads its whole window with a single **tracked** `ToListAsync` and then issues one `FirstOrDefaultAsync` per (series, hour) group plus one giant `SaveChanges` (`ConfigurationDatabase/Repositories/KpiHistoryRepository.cs:104-106, 133-139, 167`). Its own comment says "The caller passes a small trailing lookback (e.g. 3 h), so the window is bounded" — but its second caller is the one-shot backfill, which passes `[TruncateToHour(now) − RetentionDays, TruncateToHour(now))` — **90 days by default** (`KpiHistory/KpiHistoryRecorderActor.cs:510-511`).
|
||||
|
||||
**Failure scenario:** at ~2M rows, each scan is hundreds of ms of CPU/IO; the KPI backbone alone drives a near-continuous scan load on the central DB, and the KPI Ask timeouts start failing intermittently. **Fix:** a filtered index `CREATE INDEX IX_SiteCalls_NonTerminal ON SiteCalls (CreatedAtUtc) WHERE TerminalAtUtc IS NULL` (plus `SourceSite`/`SourceNode` includes) makes every one of these predicates a small seek, since the non-terminal population is the live queue, not the archive.
|
||||
At round-1's own fleet estimate (~500 samples/min ≈ 700k rows/day for 10 sites), the backfill materializes tens of millions of tracked `KpiSample` entities into one `List<>` — a multi-GB allocation on the active central node that no `catch` can contain if it OOMs — then runs ~(series-count × 2,160 hours) single-row upsert probes and commits everything in one `SaveChanges`. Even a modest single-site deployment (≈26 samples/min) materializes ~3.4M tracked rows. And because `_backfillDone` is per actor lifetime (`KpiHistoryRecorderActor.cs:129-136`), **every singleton failover/restart re-runs the full-window fetch** — the pre-existing rollup rows shrink the writes but not the raw read. While it grinds, `_backfillInFlight` also blocks every periodic fold (`:435-441`), so fresh hours stall behind the historical pass. `Component-KpiHistory.md:93` calls the backfill "cheap and safe to re-run" — the "safe" is true (idempotent), the "cheap" is not.
|
||||
|
||||
### [High] `GetExecutionTreeAsync` scans the entire AuditLog per invocation
|
||||
**Fix:** slice the backfill in the actor into bounded windows (e.g. one day per `FoldHourlyRollupsAsync` call, oldest-first, yielding between slices) — the repository method then never sees more than 24 hours; this also releases the fold path between slices. Alternatively (or additionally) skip the backfill when the newest `KpiRollupHourly.HourStartUtc` is already near `now` (the failover case, which is the common one). This is the same "bound the maintenance pass" discipline Tasks 4/19 applied to every other year-scale sweep in this domain.
|
||||
|
||||
`AuditLogRepository.cs:704-756`: the recursive CTE's `Edges` anchor is `SELECT DISTINCT ExecutionId, ParentExecutionId FROM dbo.AuditLog WHERE ExecutionId IS NOT NULL`. Neither `IX_AuditLog_Execution` nor `IX_AuditLog_ParentExecution` covers the *other* column, so satisfying the DISTINCT requires scanning an entire index (or the clustered index) across **all partitions, all 365 days** — on every `audit tree` CLI call and every UI execution-tree drilldown. The upward walk (`:652-677`) is fine (seeks on `IX_AuditLog_Execution`), and the STRING_AGG subqueries seek per node; it is the edge materialization that is O(table).
|
||||
### [Medium] R2 — Fold fetch is change-tracked (`AsNoTracking` missing)
|
||||
|
||||
**Fix options:** (a) add `ParentExecutionId` as an INCLUDE on `IX_AuditLog_Execution` (index-only DISTINCT scan — better but still O(table)); (b) constrain the Edges CTE to a time window derived from the root's `FirstOccurredAtUtc` (execution trees span minutes, not years — a `OccurredAtUtc >= root - slack` predicate gets partition elimination); (c) seed the recursion by seeking children per level (`WHERE ParentExecutionId IN (…)`) instead of pre-materializing a global edge set.
|
||||
The sample fetch in `FoldHourlyRollupsAsync` (`KpiHistoryRepository.cs:104-106`) reads `KpiSample` rows it will never modify, without `AsNoTracking()`. Even on the healthy periodic path (3-hour lookback), that is ~5k-90k read-only entities registered in the change tracker every hour, all re-scanned by `DetectChanges` on the final `SaveChanges` alongside the rollup entities that actually changed. One-word fix; it also cuts the R1 backfill's memory overhead roughly in half (tracking snapshots) even before R1 is properly sliced.
|
||||
|
||||
### [Medium] AuditLog clustered key leads with a random GUID
|
||||
### [Medium] R3 — Rate metrics change units ~60× across the raw/rollup routing boundary, and the bucketer re-introduces the fold error the catalog warns about
|
||||
|
||||
`AuditLogEntityTypeConfiguration.cs:172` / migration `CollapseAuditLogToCanonical.cs:75`: clustered PK `(EventId, OccurredAtUtc)` with `EventId = Guid.NewGuid()`. Inserts land randomly across the current month's partition → page splits, fragmentation, and larger-than-necessary buffer pool footprint on the system's highest-volume table. The composite order also means the PK is useless for time-range queries (all served by secondary indexes). A clustered key of `(OccurredAtUtc, EventId)` would give append-locality and make `IX_AuditLog_OccurredAtUtc` redundant; `EventId` uniqueness would still need the non-aligned unique index (which exists anyway). This interacts with the switch-out dance either way; worth a deliberate benchmark before the table gets big — changing it later is a full rebuild.
|
||||
`KpiHistoryQueryService.FetchSeriesAsync` routes windows ≤168h to raw per-minute samples and wider windows to hourly rollups (`CentralUI/Services/KpiHistoryQueryService.cs:107-116`). For Gauge metrics both paths agree (last-value either way). For **Rate** metrics they do not: the raw path plots per-minute deltas, the rollup path plots **sum-per-hour** values — so the same `deliveredLastInterval` chart jumps by up to ~60× in magnitude when the operator widens the range from 7d to 30d, with no normalization in the bucketer or `KpiTrendChart`. Compounding it, `KpiSeriesBucketer.Bucket` takes **last-value-per-bucket** on top of the rollup series (`Component-KpiHistory.md:151`): a 90d/200-point chart keeps 1 of ~11 hourly sums per bucket and discards the rest — precisely the "keep one delta, discard the others" mistake `KpiMetricAggregationCatalog`'s doc warns silently corrupts rate trends (`KpiMetricAggregation.cs:26-28`), re-introduced one layer up. **Fix:** teach the bucketer (or the query service) a per-metric reduction — sum-per-bucket for Rate series — and normalize both paths to a common unit (e.g. per-hour rate) so the boundary is visually seamless; the catalog needed to drive this already exists in Commons.
|
||||
|
||||
### [Medium] KpiSample volume and unbatched purge
|
||||
### [Low] R4 — `KpiMetricAggregationCatalog` re-creates the private-metric-literal drift hazard C4 just fixed
|
||||
|
||||
Per recorder tick (60 s), sample count ≈ NotificationOutbox(5 × (1 global + N sites + M nodes)) + SiteCallAudit(6 × (1 + N + M)) + AuditLog(3) + SiteHealth(12 × N). For 10 sites / 20 site-nodes this is ~500 rows/min ≈ **700k rows/day ≈ 60M+ rows at the 90-day default retention** — with a 5-column composite index (`IX_KpiSample_Series`) maintained on every insert. Meanwhile `PurgeOlderThanAsync` is a single unbatched `ExecuteDeleteAsync` (`KpiHistoryRepository.cs:65-71`): in steady state it deletes ~700k rows in **one transaction** daily — a lock/log spike on the same DB serving the operational tables; after any purge outage the catch-up delete is worse (and subject to the same default 30s command timeout as the partition switch). Batch the delete (`TOP (n)` loop, mirroring `PurgeChannelOlderThanAsync`), and consider whether per-node scope sampling every 60 s is worth 40% of the row volume.
|
||||
The catalog hardcodes `"deliveredLastInterval"` (SiteCallAudit), `"alarmEvalErrors"`, and `"eventLogWriteFailures"` as its own private literals because the emitting sources keep those names private (`Commons/Types/Kpi/KpiMetricAggregation.cs:60-64`). If an emitter's literal ever drifts, the pair silently resolves to Gauge and the hourly sum quietly becomes a last-value — no error, just an under-counted long-range trend. The Gauge default makes this degradation safe-ish, but silent. A lock-in test asserting every `RatePairs` entry matches a metric actually emitted by the corresponding `IKpiSampleSource` (or promoting the three literals to `KpiMetrics`) closes it.
|
||||
|
||||
### [Medium] Catch-all `(@p IS NULL OR col = @p)` predicates in SiteCalls query
|
||||
### [Low] R5 — Overlapping-singleton fold race faults an entire fold batch
|
||||
|
||||
`SiteCallAuditRepository.QueryAsync` (`:186-202`) builds one static SQL shape where every filter is `({param} IS NULL OR col = {param})`. This is the classic optional-parameter anti-pattern: SQL Server compiles one plan for all filter combinations, generally a scan ordered by `CreatedAtUtc DESC`. Fine while the table is small; on a year of rows, a filtered query (e.g. `Status = 'Parked'`) will scan rather than use `IX_SiteCalls_Status_Updated`. `OPTION (RECOMPILE)` or dynamic predicate composition (as `AuditLogRepository.QueryAsync` does with LINQ, `:117-199`) fixes it.
|
||||
|
||||
### [Medium] Notification outbox query page uses offset paging + double query + unsargable LIKE
|
||||
|
||||
`NotificationOutboxRepository.QueryAsync` (`:165-228`): `COUNT` + `Skip((page-1)*size)` — deep pages re-scan; `Subject.Contains(keyword)` translates to `LIKE '%kw%'` (unsargable). The sibling AuditLog and SiteCalls pages both use keyset paging. For a 365-day table this page will degrade first in the UI. Also `ComputeKpisAsync`/`ComputePerSiteKpisAsync`/`ComputePerNodeKpisAsync` issue 5-7 sequential round trips each (`:239-283`, `:286-334`, `:337-391`), and the per-site/per-node "oldest" computations materialize every non-terminal row into memory (`:312-318`, `:368-375`) — bounded by live-queue size in normal operation, unbounded during a delivery outage (exactly when the KPI page gets watched). A single `GROUP BY` with conditional aggregation would serve each snapshot in one query.
|
||||
|
||||
### [Low] Non-persisted `IngestedAtUtc` computed column re-parses `DetailsJson` per row read
|
||||
|
||||
`AuditLogEntityTypeConfiguration.cs:158-164`: every materialization of `AuditLogRow` (e.g. `QueryAsync` pages) evaluates `SWITCHOFFSET(CAST(JSON_VALUE(DetailsJson …)))` over an `nvarchar(max)` per row. Page-sized reads make this tolerable; just avoid ever using `IngestedAtUtc` in a predicate (it would force a full-table JSON parse).
|
||||
|
||||
### [Low] `MarkForwardedAsync` / `MarkReconciledAsync` build per-id IN lists
|
||||
|
||||
`SqliteAuditWriter.cs:644-664, 735-753`: parameter-per-id is safe at the default telemetry batch of 256 (`SiteAuditTelemetryOptions.BatchSize = 256`) but silently approaches SQLite's 999-parameter limit if an operator raises the batch size ~4×. Worth a guard or chunking.
|
||||
All groups in a fold commit through one `SaveChanges` (`KpiHistoryRepository.cs:167`). During a failover overlap window (old incarnation's in-flight fold vs. the new node's first fold), both may `Add` the same `(series, hour)` row; the unique `IX_KpiRollupHourly_Series` then faults the loser's **whole** `SaveChanges`, discarding every group in that pass, not just the contested row. Self-heals on the next tick (idempotent re-fold), and the in-actor guards prevent same-node races, so this is noise rather than loss — but worth knowing the failure grain is the pass, not the row, when reading fold-failure logs after a failover.
|
||||
|
||||
---
|
||||
|
||||
## Findings — Conventions
|
||||
## New Findings — Other post-baseline code
|
||||
|
||||
### [Medium] Design docs have drifted from the code in several load-bearing places
|
||||
### [Medium] R6 — `PurgeTerminalAsync` on SiteCalls is the one remaining unbatched, unindexed year-scale maintenance DELETE
|
||||
|
||||
- `Component-Commons.md:47` lists `AuditKind` with 10 values; the code has 14 (`Commons/Types/Enums/AuditKind.cs:10-26`, adds the four `SecuredWrite*` kinds). `:46` lists `AuditChannel` with 4 values; the code has 5 (`AuditChannel.cs:8-15`, adds `SecuredWrite`).
|
||||
- `Component-Commons.md:89` still says `NotificationType`: "Email — currently the only value; Teams and other channels are planned but not yet present" — `Sms` shipped 2026-06-19 and the Teams plan was dropped (per CLAUDE.md).
|
||||
- `Component-ConfigurationDatabase.md:58` describes the NotificationList `Type` discriminator as "`Email` / `Teams` / …".
|
||||
- `Component-ConfigurationDatabase.md:399` states AuditLog retention is a "single global value in v1, **no per-channel overrides**", directly contradicting the shipped `PerChannelRetentionDays` feature (and `Component-AuditLog.md:396-411`, which documents it).
|
||||
- `Component-SiteCallAudit.md:44-51` describes the `SiteCalls` schema with `Kind`/`TargetSummary`/`Provenance` columns and a `Pending/Retrying/...` status set; the shipped schema (`Component-ConfigurationDatabase.md:64` and `SiteCallEntityTypeConfiguration.cs`) uses `Channel`/`Target` and `AuditStatus`-derived strings with no instance/script provenance columns. The ConfigDb doc is current; the SiteCallAudit doc's own table section is stale.
|
||||
`SiteCallAuditRepository.PurgeTerminalAsync` is still a single unbatched `DELETE FROM dbo.SiteCalls WHERE TerminalAtUtc IS NOT NULL AND TerminalAtUtc < cutoff` (`SiteCallAuditRepository.cs:247-252`). Nothing indexes non-null `TerminalAtUtc` — the new `IX_SiteCalls_NonTerminal` is filtered to `IS NULL` and unusable here — so the daily purge full-scans the 365-day table, and its steady-state day-of-terminal-rows (plus any catch-up after an outage) deletes in one transaction under the default command timeout. This is exactly the shape Task 19 batched for `KpiSample` and Task 4 time-boxed for `AuditLog`; SiteCalls was left out. Volume is lower than either sibling, so Medium not High — but it inherits both prior failure modes (lock/log spike; a 30s-timeout loop after an outage backlog). Fix: `TOP (n)` loop like `PurgeChannelOlderThanAsync` plus either an index on `TerminalAtUtc` or a `(Status, UpdatedAtUtc)`-driven predicate.
|
||||
|
||||
Per the repo's own rule ("design doc and code travel together"), these should be reconciled in one pass.
|
||||
### [Low] R7 — `SiteAuditRetentionService.StopAsync` surfaces the loop's cancellation to the host
|
||||
|
||||
### [Medium] `AuditLogRow` entity lives in ConfigurationDatabase, not Commons
|
||||
|
||||
`src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Entities/AuditLogRow.cs` breaks the "POCO entities in Commons / mappings in ConfigurationDatabase" convention that every sibling table follows (`SiteCall`, `Notification`, `KpiSample`, `AuditLogEntry` are all Commons POCOs). There is a defensible rationale — the *contract* type is the external `ZB.MOM.WW.Audit.AuditEvent` and `AuditLogRow` is a persistence-only projection — but the deviation is undocumented in `Component-Commons.md`/`Component-ConfigurationDatabase.md`, and `IAuditLogRepository` (in Commons) consequently traffics in `AuditEvent` while `ISiteCallAuditRepository` traffics in a Commons entity. Document the deviation or move the row shape.
|
||||
|
||||
### [Low] Mixed timestamp CLR types across sibling tables
|
||||
|
||||
`Notification` uses `DateTimeOffset` (`Commons/Entities/Notifications/Notification.cs:68-77`) while `SiteCall`, `KpiSample`, and `AuditLogRow` use UTC `DateTime`. Both satisfy the UTC convention, but the `DateTimeOffset` choice is what forced the awkward in-memory Min reductions and "converter makes SQL Min awkward" workarounds in `NotificationOutboxRepository` (`:262-274`, `:308-318`) that its `DateTime`-based siblings didn't need. New tables should standardize (the `DateTime`+`Utc`-converter pattern in `AuditLogEntityTypeConfiguration.cs:32-40` is the cleaner precedent).
|
||||
|
||||
### [Low] Uncharted KPI metric names are string literals while charted ones use the catalog
|
||||
|
||||
`NotificationOutboxKpiSampleSource.cs:40-43`: `"stuckCount"` / `"oldestPendingAgeSeconds"` are inline literals with a comment explaining the split, while charted metrics use `KpiMetrics.*` constants. Consistent cataloging (all persisted names in `KpiMetrics`) would remove a rename hazard on values that are, per the code's own comment, "persisted; do not rename."
|
||||
|
||||
### Positive observations (conventions done right)
|
||||
|
||||
- Options pattern with dedicated validators and clamped intervals everywhere (`SiteCallAuditOptions.ResolvedReconciliationInterval` guarding the Akka zero-interval spin footgun, `AuditLogOptionsValidator`, `KpiHistoryOptionsValidator`).
|
||||
- The `Sender`-capture-before-await and `Context.System.EventStream`-capture-before-await disciplines are applied consistently across all five actors — a subtle Akka correctness point handled uniformly.
|
||||
- Duplicate-key race hardening (2601/2627 swallow) is uniform across all three idempotent-insert repositories, with the check-then-act window explicitly documented at each site.
|
||||
- `AuditService` stages within the caller's unit of work exactly as the spec's transactional-audit guarantee requires, with cycle-tolerant serialization that can't roll back the business operation (`AuditService.cs:20-24, 71-85`).
|
||||
- The dual-write transaction (`AuditLogIngestActor.cs:262-311`) is genuinely atomic per entry with per-entry isolation, and `IngestedAtUtc` is stamped from one instant across both rows.
|
||||
`SafePurgeAsync` deliberately rethrows `OperationCanceledException` (`Site/SiteAuditRetentionService.cs:106-110`), but the two `await SafePurgeAsync(ct)` call sites in `RunLoopAsync` (:76, :89) are outside any try/catch, so a shutdown that lands mid-purge cancels the `_loop` task — and `StopAsync` returns that canceled task directly (:122-126), making the host's `StopAsync` await throw `TaskCanceledException`. The generic host logs and continues, so this is shutdown-log noise, not a hang — but the sibling `SiteAuditBacklogReporter` pattern it mirrors is worth aligning: catch OCE at the loop level (or `Task.WhenAny`-guard in `StopAsync`).
|
||||
|
||||
---
|
||||
|
||||
## Underdeveloped Areas
|
||||
## What's genuinely good
|
||||
|
||||
1. **`backlogTotal` KPI history series is permanently zero.** `AuditLogRepository.GetKpiSnapshotAsync` hardcodes `BacklogTotal: 0L` (`AuditLogRepository.cs:618`) — documented in the interface as "left at zero" — and the Central UI's live tile compensates by summing site backlog reports (`CentralUI/Services/AuditLogQueryService.cs:125-155`). But `AuditLogKpiSampleSource` persists the repo snapshot **directly** (`AuditLogKpiSampleSource.cs:69-85`), so every recorded `backlogTotal` sample is 0, and the Audit Log page's backlog trend chart (`AuditLogPage.razor.cs:297`) renders a flat zero line regardless of actual backlog. The one KPI whose *history* matters most during an outage post-mortem (how big did the backlog get, how fast did it drain) records nothing. Fix: inject the same health-aggregator read the query service uses into the sample source.
|
||||
2. **Site SQLite retention purge** — spec'd, commented as future, absent (see Stability [High] above).
|
||||
3. **Hash-chain tamper evidence (T1) and Parquet archival (T2)** — explicitly deferred to v1.x per `Component-AuditLog.md:433-437`; the `verify-chain` CLI is a documented no-op placeholder. Consistent with plan; no drift.
|
||||
4. **SecuredWrite audit rows leave `SourceNode` NULL** — documented as a logged follow-up (`Component-AuditLog.md:154-157`); also note `PerChannelRetentionDays` validation accepts `SecuredWrite` as a key (the enum has it) even though the design doc's config section (`:396-399`) enumerates only the four original channels — harmless, but the doc under-promises.
|
||||
5. **Reconciliation cursor keyset upgrade** — the `(timestamp, id)` composite cursor that would eliminate the single-timestamp pin is acknowledged in a log message (`SiteCallAuditActor.cs:640-654`) but not tracked anywhere; the pull contracts (`CachedCallReconcileRequest`, `PullAuditEvents`) would need an additive field.
|
||||
6. **Dispatcher throughput ceiling** — notification delivery is strictly sequential within a sweep (batch 100, 10 s interval, blocking SMTP/Twilio per row: `NotificationOutboxActor.RunDispatchPass:321-347`). A slow SMTP endpoint (5 s/attempt) caps throughput at ~1 notification/5 s regardless of batch size, and the in-flight guard means subsequent ticks drop. The spec's "dedicated blocking-I/O dispatcher" (`Component-NotificationOutbox.md:23`) is not literally implemented (delivery runs on the thread pool off the actor thread — functionally acceptable, doc drift), but there is no per-notification parallelism at all. Fine for alarm-notification volume; a burst (alarm storm fanning to hundreds of notifications) drains slowly.
|
||||
7. **Test coverage** is strong overall (MSSQL-backed integration tests for partition purge, execution trees, outage reconciliation, dual-write; per-actor TestKit suites; options validators tested). Gaps mirror the findings: no test asserts a site-SQLite retention purge (it can't — the feature is missing), no test exercises `SwitchOutPartitionAsync` at a size that would expose the command-timeout failure, and no test asserts `backlogTotal` history is non-zero under backlog.
|
||||
- **The fix execution matches the fix claims.** All 19 verified-Fixed items are real, tested, and in several cases exceed the plan: the purge-failure path got a dedicated no-op-defaulted counter seam (`IAuditPurgeFailureCounter` + `NoOpAuditPurgeFailureCounter`), and the S1 purge encodes the ForwardState invariant *in the SQL itself* with a param-limit-proof temp table (`SqliteAuditWriter.cs:877-897`).
|
||||
- **The NodeB-failover pull clients are exemplary defensive code**: transport-fault-only failover (a mapping fault correctly does not retry on the other node, "since a second node would hit the same non-transport fault"), per-row DTO fault isolation, race-safe channel cache with loser-dispose, and the `DateTime.MinValue`/`ToUniversalTime` underflow trap explicitly dodged and documented (`GrpcPullSiteCallsClient.cs:110-127, 236-242, 309-325`).
|
||||
- **The keyset cursor landed end-to-end with honest legacy behavior**: proto additive field, empty-string-as-null mapping at the server, strict-keyset vs. legacy-inclusive branches in the site store, and a latched pinned-state event that fires only on transitions (`OperationTrackingStore.cs:384-395`, `SiteCallAuditActor.cs:737-746`).
|
||||
- **Deferral hygiene**: every accepted deferral (S8, P3, P4-cadence, P6-paging, C2, C3, U5) now has a written home in a design doc — the round-1 complaint that dead-ends were "deferred without a tracking artifact" is fully answered.
|
||||
- **The rollup catalog's semantics section** (`KpiMetricAggregation.cs:85-96`) is a model of the kind of reasoning that prevents silent data corruption: it overrides the plan's tentative Rate classification for the `*LastHour` metrics after checking the actual source semantics, and documents why in place.
|
||||
- **Execution-strategy adoption was done deliberately**, exactly as round-1 cautioned: the dual-write's user-initiated transaction is wrapped per entry with the idempotency argument stated (`AuditLogIngestActor.cs:263-298`), rather than a blind `EnableRetryOnFailure` toggle.
|
||||
|
||||
---
|
||||
## Underdeveloped areas
|
||||
|
||||
## Prioritized Recommendations
|
||||
1. **Backfill scale testing** — no test exercises `FoldHourlyRollupsAsync` at a window larger than a few hours; the R1 failure mode (90-day materialization) is invisible to the current suites, mirroring how the round-1 partition-switch timeout was invisible until sized. A `SkippableFact` MSSQL test at representative volume, or a unit test asserting the actor slices the backfill, would lock in the R1 fix.
|
||||
2. **Rate-metric presentation across the routing boundary** (R3) — the storage layer has full fidelity (`MinValue`/`MaxValue`/`SampleCount` are stored but uncharted); the presentation semantics are the unfinished half of the rollup feature.
|
||||
3. **AuditLog pull keyset** (U5 residue) — tracked, additive, and lower-urgency; fine as-is, noting it here so the tracker's entry doesn't age out.
|
||||
|
||||
1. **Implement the site SQLite 7-day retention purge** (daily timer on `SiteAuditTelemetryActor` or a dedicated job; `DELETE` joined on `audit_forward_state.ForwardState IN ('Forwarded','Reconciled')` + age, followed by `PRAGMA incremental_vacuum`) — closes the unbounded-growth hole and honors the spec'd `ForwardState` invariant.
|
||||
2. **Fix the `backlogTotal` KPI sample source** to read the health aggregator like `AuditLogQueryService` does — one-file change, restores a shipped feature.
|
||||
3. **Harden `SwitchOutPartitionAsync` operationally**: explicit long `CommandTimeout` on the maintenance commands, a health metric/event on purge *failure* (the success event exists; the failure path is log-only), and an integration test at representative volume.
|
||||
4. **Add the filtered non-terminal index on `SiteCalls`** (`WHERE TerminalAtUtc IS NULL`) and turn each KPI snapshot into a single conditional-aggregation query — removes the dominant recurring scan load.
|
||||
5. **Bound the `GetExecutionTreeAsync` edge scan** with a time-window predicate (partition elimination) or per-level child seeks.
|
||||
6. **Decide the DB-role story honestly**: either a second maintenance connection string/principal (and grant `CREATE TABLE` to the purger), or update `Component-AuditLog.md`/`Component-ConfigurationDatabase.md` to state that append-only enforcement is CI-guard-based with roles offered as an optional DBA hardening.
|
||||
7. **Allow same-rank freshness updates in the SiteCalls upsert** (`UpdatedAtUtc` tiebreaker) so retrying calls show live RetryCount/LastError.
|
||||
8. **Batch the KpiSample purge** and reconsider per-node sampling cadence before fleet size multiplies row volume.
|
||||
9. **Emit an audit row for operator Retry** on parked notifications (and consider operator identity on the SiteCalls relay), matching the spec's transition-audit promise.
|
||||
10. **Reconcile the stale design-doc passages** (AuditKind/AuditChannel/NotificationType enum lists, `Teams` references, "no per-channel overrides", SiteCalls schema section) in one documentation pass, per the repo's docs-and-code-travel-together rule.
|
||||
## New-Findings Severity Tally (round 2 only)
|
||||
|
||||
| Severity | Count | Findings |
|
||||
|----------|-------|----------|
|
||||
| Critical | 0 | — |
|
||||
| High | 1 | R1 (unbounded rollup backfill, re-run per failover) |
|
||||
| Medium | 3 | R2 (tracked fold fetch), R3 (rate-unit discontinuity + bucketer re-fold error), R6 (SiteCalls terminal purge unbatched/unindexed) |
|
||||
| Low | 3 | R4 (catalog literal drift), R5 (fold-batch failover race grain), R7 (retention-service shutdown OCE) |
|
||||
|
||||
**Round-2 bottom line:** the round-1 debt is paid — verified, not just claimed. The new debt is small and localized: bound the rollup backfill (R1) and add `AsNoTracking` (R2) before any production deployment accumulates months of `KpiSample` rows, decide the rate-metric presentation (R3) before operators learn to distrust the 30d charts, and give `PurgeTerminalAsync` the same batching its two siblings received (R6).
|
||||
|
||||
Reference in New Issue
Block a user