# Data & Audit Backbone Fixes Implementation Plan > **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. **Goal:** Defuse the scale-dependent operational time bombs in the central data/audit layer (missing site SQLite retention purge, partition-switch timeout self-lock, unindexed KPI scans, table-wide execution-tree scans, hardcoded-zero backlog KPI) and close the audit-integrity/consistency gaps (append-only role drift, frozen SiteCalls progress fields, missing operator-Retry audit rows, reconciliation dead-ends, NodeA-only reconciliation) identified in `archreview/04-data-audit-backbone.md`. **Architecture:** All fixes stay inside the existing component boundaries: site-side purge lives in the AuditLog component's site half (SqliteAuditWriter + a new hosted retention job), central maintenance hardening lives in AuditLogRepository/AuditLogPurgeActor, index/DDL changes ride EF Core migrations in ConfigurationDatabase (with idempotent SQL scripts for production), and contract changes (keyset cursor, RequestedBy, SiteEntry fallback endpoint) are strictly additive per the repo's message-evolution rule. Design docs are updated in the same task as the code they describe (repo rule: doc + code + tests travel together). **Tech Stack:** C#/.NET 8, Akka.NET (actors/TestKit), EF Core + SQL Server (central), Microsoft.Data.Sqlite (site), xUnit. Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per-project: `dotnet test tests/ --filter `. MSSQL-backed integration tests need `cd infra && docker compose up -d`. **EF migrations gotcha (repo-documented):** always build first and run `dotnet ef migrations add ` WITHOUT `--no-build`, or it scaffolds an empty migration off the stale DLL. ## Parallelization (incomplete tasks) — 2026-07-09 **Open:** only **T26** (full `dotnet build`/`dotnet test` sweep) — T24 (design-doc reconciliation) completed 2026-07-09. T26 has no intra-plan parallelism and **no cross-plan shared-file conflict** (build/test only), so a PLAN-04 executor can run concurrently with any other plan's session. Whole-initiative view: [00-MASTER-TRACKER.md § Parallelization Map](00-MASTER-TRACKER.md#parallelization-map--incomplete-tasks-2026-07-09). ## Findings Coverage | # | Report finding (severity) | Task(s) | |---|---------------------------|---------| | S1 | Site SQLite 7-day retention purge missing (High) | 1, 2, 3 | | S2 | `SwitchOutPartitionAsync` no CommandTimeout / log-only failure (High) | 4, 5 | | S3 | Append-only DB-role enforcement DDL-only; purger grants insufficient (High) | 6, 7 | | S4 | SiteCalls upsert freezes RetryCount/LastError at rank (Medium) | 11 | | S5 | No EF `EnableRetryOnFailure` (Medium) | 12 | | S6 | Operator Retry on parked notification emits no audit row (+SiteCalls relay identity) (Medium) | 13, 14 | | S7 | Reconciliation edge cases dead-ended (cursor pin; permanent abandonment) (Medium) | 15, 16, 17 | | S8 | NotificationOutboxRepository dual SQLite/T-SQL dialect — provider parity (Medium) | 24 (documented convention decision: MSSQL-only SQL + MSSQL fixtures is the norm for new repositories; existing dual-dialect kept as a documented legacy exception — rewriting the shipped repository + its whole test suite is not worth the churn) | | S9 | Purge actors' first tick waits a full interval (Low) | 22 | | S10 | `SqliteAuditWriter.Dispose` sync-over-async (Low) | Won't-fix — report itself notes the thread-pool-hop mitigation is correct and documented; flagged-only | | P1 | SiteCalls KPI predicates full-scan — no `TerminalAtUtc` index (High) | 8 | | P2 | `GetExecutionTreeAsync` scans entire AuditLog (High) | 9 | | P3 | AuditLog clustered key leads with random GUID (Medium) | Deferred — report says "worth a deliberate benchmark before the table gets big"; changing it is a full-table rebuild. Task 24 records it as a tracked design decision in Component-ConfigurationDatabase.md so it is no longer untracked | | P4 | KpiSample volume + unbatched purge (Medium) | 19 (batched purge); per-node sampling cadence recorded as tracked follow-up in 24 | | P5 | Catch-all `(@p IS NULL OR col=@p)` predicates in SiteCalls query (Medium) | 21 | | P6 | Outbox offset paging + double query + LIKE; KPI 5-7 round trips; unbounded oldest materialization (Medium) | 20 (KPI single query + bounded oldest); offset→keyset conversion Deferred — it changes the page/TotalCount UI contract for a table whose live-queue portion stays small under Task 19/existing purge; recorded in 24 | | P7 | Non-persisted `IngestedAtUtc` computed column re-parse (Low) | 23 (predicate-ban comment guard) | | P8 | `MarkForwardedAsync`/`MarkReconciledAsync` per-id IN lists near SQLite param limit (Low) | 23 | | C1 | Design docs drifted (enum lists, Teams, per-channel overrides, SiteCalls schema) (Medium) | 24 | | C2 | `AuditLogRow` entity lives in ConfigurationDatabase not Commons (Medium) | 24 (document the deviation — the rationale in the report is accepted: contract type is external `ZB.MOM.WW.Audit.AuditEvent`, row is a persistence-only projection) | | C3 | Mixed timestamp CLR types across sibling tables (Low) | 24 (convention note: new tables use `DateTime`+Utc converter) | | C4 | Uncharted KPI metric names are string literals (Low) | 23 | | U1 | `backlogTotal` KPI history permanently zero | 10 | | U2 | Site SQLite retention purge | 1, 2, 3 | | U3 | Hash-chain tamper evidence / Parquet archival | No action — explicitly deferred to v1.x per Component-AuditLog.md; report confirms "consistent with plan; no drift" | | U4 | SecuredWrite audit rows leave `SourceNode` NULL; doc under-promises per-channel keys | SourceNode fix Deferred — already a logged follow-up in Component-AuditLog.md:154-157 and the SecuredWrite emission path belongs to plan 07's domain. Doc under-promise fixed in 24 | | U5 | Reconciliation cursor keyset upgrade untracked | 15, 16 (SiteCalls keyset implemented); AuditLog-pull keyset recorded as tracked follow-up in Component-AuditLog.md (24) | | U6 | Dispatcher throughput ceiling (sequential delivery) | 25 | | U7 | Test-coverage gaps mirror findings | Covered inside tasks 1, 4, 10 (the report's three named gaps each get a test) | | X1 | AuditLog reconciliation dials site NodeA only (flagged in report 08, owned here) | 18 | --- ### Task 1: Site SQLite retention purge — `PurgeExpiredAsync` on the writer **Classification:** standard **Estimated implement time:** ~5 min **Parallelizable with:** 4, 6, 8, 11, 12, 19, 20, 21 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Services/ISiteAuditQueue.cs` (add method) - Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriter.cs` (implement; the `auto_vacuum = INCREMENTAL` pragma at ~line 149 already anticipates this) - Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SqliteAuditWriterRetentionTests.cs` (create) 1. Write the failing test (pattern-match the fixture setup from `SqliteAuditWriterBacklogStatsTests.cs`): ```csharp public class SqliteAuditWriterRetentionTests { [Fact] public async Task PurgeExpired_DeletesForwardedAndReconciled_OlderThanCutoff() { await using var writer = CreateWriter(); // same in-memory/temp-file helper the sibling tests use var oldForwarded = await WriteEventAsync(writer, occurredAtUtc: DateTime.UtcNow.AddDays(-10)); var oldReconciled = await WriteEventAsync(writer, occurredAtUtc: DateTime.UtcNow.AddDays(-10)); var oldPending = await WriteEventAsync(writer, occurredAtUtc: DateTime.UtcNow.AddDays(-10)); var fresh = await WriteEventAsync(writer, occurredAtUtc: DateTime.UtcNow); await writer.MarkForwardedAsync(new[] { oldForwarded, oldReconciled, fresh }); await writer.MarkReconciledAsync(new[] { oldReconciled }); var purged = await writer.PurgeExpiredAsync(DateTime.UtcNow.AddDays(-7)); Assert.Equal(2, purged); // oldForwarded + oldReconciled // Hard ForwardState invariant: the old *Pending* row survives on age alone. var pending = await writer.ReadPendingAsync(10); Assert.Contains(pending, e => e.EventId == oldPending); // Fresh forwarded row inside the window survives. var stats = await writer.GetBacklogStatsAsync(); Assert.Equal(1, stats.PendingCount); } [Fact] public async Task PurgeExpired_IsIdempotent() { await using var writer = CreateWriter(); var id = await WriteEventAsync(writer, occurredAtUtc: DateTime.UtcNow.AddDays(-10)); await writer.MarkForwardedAsync(new[] { id }); Assert.Equal(1, await writer.PurgeExpiredAsync(DateTime.UtcNow.AddDays(-7))); Assert.Equal(0, await writer.PurgeExpiredAsync(DateTime.UtcNow.AddDays(-7))); } } ``` 2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter SqliteAuditWriterRetentionTests` — expect FAIL (method does not exist). 3. Add to `ISiteAuditQueue`: `Task PurgeExpiredAsync(DateTime olderThanUtc, CancellationToken ct = default);` with XML doc stating the hard invariant (`Pending` rows are never purged on age alone — only `Forwarded`/`Reconciled`). Implement in `SqliteAuditWriter` under `_writeLock` (mirror `MarkForwardedAsync`'s lock + `ObjectDisposedException.ThrowIf` shape), one transaction: ```sql CREATE TEMP TABLE IF NOT EXISTS purge_ids AS SELECT EventId FROM audit_forward_state WHERE ForwardState IN ('Forwarded','Reconciled') AND OccurredAtUtc < $cutoff; DELETE FROM audit_forward_state WHERE EventId IN (SELECT EventId FROM purge_ids); DELETE FROM audit_event WHERE EventId IN (SELECT EventId FROM purge_ids); DROP TABLE purge_ids; ``` Delete the sidecar first (it is the FK child of `audit_event`). `$cutoff` uses the same round-trip `"o"` timestamp format the writer already uses. Return `changes()` from the `audit_event` DELETE. After COMMIT, run `PRAGMA incremental_vacuum;` (the pragma the ctor set up — reference the existing comment at `SqliteAuditWriter.cs:145` and update it: the purge now exists). 4. Run the filter again — expect PASS. Also run the whole site suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter "FullyQualifiedName~Site"` — expect PASS. 5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Services/ISiteAuditQueue.cs src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriter.cs tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SqliteAuditWriterRetentionTests.cs && git commit -m "feat(audit-log): add site SQLite PurgeExpiredAsync honoring the ForwardState invariant"` ### Task 2: Site retention options **Classification:** small **Estimated implement time:** ~3 min **Parallelizable with:** 1, 4, 6, 8, 11, 12, 19, 20, 21 **Files:** - Create: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SiteAuditRetentionOptions.cs` - Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SiteAuditRetentionOptionsTests.cs` (create) 1. Failing test: ```csharp public class SiteAuditRetentionOptionsTests { [Theory] [InlineData(0, 1)] // clamp floor [InlineData(7, 7)] // default passthrough [InlineData(365, 90)] // clamp ceiling (spec: min 1, max 90) public void ResolvedRetentionDays_Clamps(int configured, int expected) => Assert.Equal(expected, new SiteAuditRetentionOptions { RetentionDays = configured }.ResolvedRetentionDays); [Fact] public void Defaults_Are_SevenDays_DailyPurge_FiveMinuteInitialDelay() { var o = new SiteAuditRetentionOptions(); Assert.Equal(7, o.ResolvedRetentionDays); Assert.Equal(TimeSpan.FromHours(24), o.ResolvedPurgeInterval); Assert.Equal(TimeSpan.FromMinutes(5), o.InitialDelay); } } ``` 2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter SiteAuditRetentionOptionsTests` — FAIL. 3. Implement the options class following the `SiteCallAuditOptions.ResolvedReconciliationInterval` clamp pattern: `RetentionDays` (default 7; `ResolvedRetentionDays` clamps to `[1, 90]`), `PurgeInterval` (default 24h; `ResolvedPurgeInterval` clamps to min 1 minute — Akka zero-interval spin footgun), `InitialDelay` (default 5 min — short so a daily-restarting node still purges). Config section: `ScadaBridge:AuditLog:SiteRetention`. 4. Run filter — PASS. 5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SiteAuditRetentionOptions.cs tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SiteAuditRetentionOptionsTests.cs && git commit -m "feat(audit-log): SiteAuditRetentionOptions (7-day default, clamped 1-90)"` ### Task 3: Site retention hosted job + registration + design doc **Classification:** standard **Estimated implement time:** ~5 min **Parallelizable with:** 4, 6, 8, 11, 12, 19, 20, 21 **Files:** - Create: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SiteAuditRetentionService.cs` - Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/ServiceCollectionExtensions.cs` (site registration path, near the `ISiteAuditQueue` registration at ~line 111) - Modify: `docs/requirements/Component-AuditLog.md` (~lines 412-414, the "Sites: daily site job" bullet) - Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SiteAuditRetentionServiceTests.cs` (create) 1. Failing test (fake `ISiteAuditQueue`, short intervals): ```csharp public class SiteAuditRetentionServiceTests { [Fact] public async Task Tick_Purges_With_RetentionCutoff() { var queue = new RecordingSiteAuditQueue(); // stub: records PurgeExpiredAsync calls var options = Options.Create(new SiteAuditRetentionOptions { RetentionDays = 7, PurgeInterval = TimeSpan.FromMilliseconds(50), InitialDelay = TimeSpan.Zero }); using var svc = new SiteAuditRetentionService(queue, options, NullLogger.Instance); await svc.StartAsync(CancellationToken.None); await WaitUntilAsync(() => queue.PurgeCalls.Count >= 1, TimeSpan.FromSeconds(5)); await svc.StopAsync(CancellationToken.None); var cutoff = queue.PurgeCalls[0]; Assert.InRange(cutoff, DateTime.UtcNow.AddDays(-7).AddMinutes(-1), DateTime.UtcNow.AddDays(-7).AddMinutes(1)); } [Fact] public async Task Tick_SwallowsExceptions_AndKeepsTicking() { var queue = new RecordingSiteAuditQueue { ThrowOnFirstCall = true }; // ... start, wait for >= 2 calls, assert no unobserved exception } } ``` 2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter SiteAuditRetentionServiceTests` — FAIL. 3. Implement `SiteAuditRetentionService : IHostedService, IDisposable` mirroring `SiteAuditBacklogReporter.cs` (same file layout: `System.Threading.Timer`, `InitialDelay` as due time, `ResolvedPurgeInterval` as period, per-tick try/catch that logs and continues, purge count logged at Information). Register in the site path of `ServiceCollectionExtensions`: bind `SiteAuditRetentionOptions` from `ScadaBridge:AuditLog:SiteRetention` and `services.AddHostedService();`. 4. Run filter — PASS. Then `dotnet build ZB.MOM.WW.ScadaBridge.slnx` — expect clean. 5. Update `Component-AuditLog.md` ~412-414: replace the aspirational bullet with the shipped mechanism — `SiteAuditRetentionService` hosted job, config keys (`ScadaBridge:AuditLog:SiteRetention:RetentionDays|PurgeInterval`), the `PurgeExpiredAsync` invariant, and the post-purge `incremental_vacuum`. 6. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.AuditLog tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site docs/requirements/Component-AuditLog.md && git commit -m "feat(audit-log): daily site SQLite retention purge job (closes unbounded site DB growth)"` ### Task 4: Maintenance command timeout on partition switch + per-channel purge **Classification:** high-risk **Estimated implement time:** ~5 min **Parallelizable with:** 1, 2, 3, 8 (different files), 11, 20 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeOptions.cs` - Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IAuditLogRepository.cs` (additive optional parameter) - Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/AuditLogRepository.cs` (`SwitchOutPartitionAsync` ~257-371, `PurgeChannelOlderThanAsync` ~374-445) - Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeActor.cs` (pass the timeout) - Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/AuditLogPurgeActorTests.cs` (extend), `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/PartitionPurgeTests.cs` (extend, MSSQL) 1. Failing unit test in `AuditLogPurgeActorTests` (these tests already drive the actor against a mocked `IAuditLogRepository`): assert the actor calls `SwitchOutPartitionAsync(boundary, commandTimeout)` with the options value: ```csharp [Fact] public async Task PurgeTick_Passes_MaintenanceCommandTimeout_To_Repository() { // options: MaintenanceCommandTimeoutMinutes = 45 // mock repo: capture the TimeSpan? argument // drive one tick; Assert.Equal(TimeSpan.FromMinutes(45), captured); } ``` Plus an options test: `MaintenanceCommandTimeoutMinutes` default 30, clamped to min 1 (`ResolvedMaintenanceCommandTimeout`). 2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter AuditLogPurgeActorTests` — FAIL (no such parameter). 3. Implementation: - `AuditLogPurgeOptions`: add `public int MaintenanceCommandTimeoutMinutes { get; set; } = 30;` + `ResolvedMaintenanceCommandTimeout` clamp (min 1 minute), XML-doc'd against the 30s-default-timeout failure mode from the review. - `IAuditLogRepository`: `Task SwitchOutPartitionAsync(DateTime monthBoundary, TimeSpan? commandTimeout = null, CancellationToken ct = default);` and `PurgeChannelOlderThanAsync(..., TimeSpan? commandTimeout = null, ...)` — additive defaults, existing callers compile unchanged. - `AuditLogRepository.SwitchOutPartitionAsync`: when `commandTimeout` is non-null, set `sampleCmd.CommandTimeout = (int)commandTimeout.Value.TotalSeconds;` and wrap the `ExecuteSqlRawAsync` with `_context.Database.SetCommandTimeout(commandTimeout)` (the context is scoped per purge tick — see `AuditLogPurgeActor.OnTickAsync`'s `CreateAsyncScope` — so no restore is needed, but restore the previous value in a `finally` anyway for hygiene). Same `cmd.CommandTimeout` treatment on the per-channel `DELETE TOP` command. - `AuditLogPurgeActor.OnTickAsync`: pass `_purgeOptions.ResolvedMaintenanceCommandTimeout` to both repository calls. 4. Run the actor filter — PASS. Extend `PartitionPurgeTests` (MSSQL; `cd infra && docker compose up -d` first) with one case that calls `SwitchOutPartitionAsync(boundary, TimeSpan.FromMinutes(30))` and asserts success — proving the timeout path executes against real SQL Server: `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter PartitionPurgeTests`. 5. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.AuditLog src/ZB.MOM.WW.ScadaBridge.Commons src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests && git commit -m "fix(audit-log): explicit long CommandTimeout on partition-switch and per-channel purge maintenance paths"` ### Task 5: Purge-failure health event (not just a log line) **Classification:** standard **Estimated implement time:** ~4 min **Parallelizable with:** 8, 11, 19, 20, 21 **Files:** - Create: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeFailedEvent.cs` - Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeActor.cs` (catch block ~198-206 and the boundary-enumeration catch) - Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditCentralHealthSnapshot.cs` + `IAuditCentralHealthSnapshot.cs` (add `PurgeFailures` counter, mirroring the existing counter members) - Modify: `docs/requirements/Component-AuditLog.md` (retention section: purge failure is now a health metric) - Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/AuditLogPurgeActorTests.cs` (extend) 1. Failing test: mock repository whose `SwitchOutPartitionAsync` throws; subscribe a TestProbe to `AuditLogPurgeFailedEvent` on the EventStream (same pattern the existing tests use for `AuditLogPurgedEvent`); drive a tick; expect one `AuditLogPurgeFailedEvent(boundary, errorMessage, elapsedMs)` and assert the health snapshot's `PurgeFailures` incremented. 2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter AuditLogPurgeActorTests` — FAIL. 3. Implement: `public sealed record AuditLogPurgeFailedEvent(DateTime MonthBoundary, string Error, long ElapsedMilliseconds);` — publish it in the per-boundary catch (alongside the existing `LogError`) using the already-captured `eventStream`; increment the new snapshot counter (inject `IAuditCentralHealthSnapshot` the same way sibling actors take their counters — follow the existing ctor DI shape). The success event (`AuditLogPurgedEvent`) already exists; this is its symmetric failure twin. 4. Run filter — PASS. 5. Doc: add to Component-AuditLog.md retention section: "Purge failure publishes `AuditLogPurgeFailedEvent` and increments the `PurgeFailures` central health counter — a silently failing retention job is a health-surface condition, not just a log line." 6. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.AuditLog tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central docs/requirements/Component-AuditLog.md && git commit -m "feat(audit-log): surface partition-purge failure as health event + counter"` ### Task 6: Fix purger role grants (migration) **Classification:** high-risk **Estimated implement time:** ~4 min **Parallelizable with:** 1, 2, 3, 11, 20 (NOT with Task 8 — both add migrations, serialize on the model snapshot) **Files:** - Create: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/_FixAuditPurgerRoleGrants.cs` (scaffolded) - Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Migrations/` (add `FixAuditPurgerRoleGrantsTests.cs`), plus re-run `AuditLogAppendOnlyGuardTests` 1. Failing test first — a migration-content guard in the style of the existing migration tests: assert the migration source contains `GRANT CREATE TABLE TO scadabridge_audit_purger` and `GRANT DELETE ON dbo.AuditLog TO scadabridge_audit_purger`. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter FixAuditPurgerRoleGrants` — FAIL (file absent). 2. Scaffold (repo gotcha — build first, NO `--no-build`): ```bash dotnet build ZB.MOM.WW.ScadaBridge.slnx cd src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase dotnet ef migrations add FixAuditPurgerRoleGrants ``` Verify the scaffold is model-empty (no schema change) and add the SQL by hand in `Up`: ```csharp migrationBuilder.Sql(@" IF DATABASE_PRINCIPAL_ID('scadabridge_audit_purger') IS NULL EXEC sp_executesql N'CREATE ROLE scadabridge_audit_purger'; -- The switch-out dance CREATEs a staging table; ALTER ON SCHEMA::dbo alone does not -- confer the database-level CREATE TABLE permission (review finding S3). GRANT CREATE TABLE TO scadabridge_audit_purger; -- The per-channel retention override (PurgeChannelOlderThanAsync) is a bounded row -- DELETE on the maintenance path; a segregated purger principal needs this grant. GRANT DELETE ON dbo.AuditLog TO scadabridge_audit_purger;"); // AUDIT-PURGE-ALLOWED: role grant for the maintenance principal ``` `Down`: `REVOKE` both grants (guarded on principal existence). 3. Run the new test — PASS. Then run `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter AuditLogAppendOnlyGuardTests` — if the CI grep guard trips on the `GRANT DELETE ON dbo.AuditLog` text, extend the guard's allowlist to permit `GRANT`/`DENY`/`REVOKE` statements (they are permission DDL, not data mutation) — keep the guard strict for `DELETE FROM`/`UPDATE `. 4. Production script step (repo convention — manual SQL for production): `dotnet ef migrations script AddPendingDeployment FixAuditPurgerRoleGrants --idempotent --output ../../docs/plans/sql/FixAuditPurgerRoleGrants.sql` (adjust the from-migration to the actual latest; review the output). 5. MSSQL sanity: with infra up, `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter Migrations` — expect PASS. 6. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests docs/plans/sql && git commit -m "fix(config-db): grant CREATE TABLE + scoped DELETE to scadabridge_audit_purger so a segregated maintenance principal can actually run the purge"` ### Task 7: Honest append-only enforcement story (docs) **Classification:** small **Estimated implement time:** ~4 min **Parallelizable with:** everything except 24 (same doc files) **Files:** - Modify: `docs/requirements/Component-AuditLog.md` (~417-424, "Security & Tamper-Evidence") - Modify: `docs/requirements/Component-ConfigurationDatabase.md` (~399-402, AuditLog Table Purge section) 1. Rewrite Component-AuditLog.md §Security "Append-only enforcement" to state reality (review finding S3): the application runs with **one** connection principal for both writer and maintenance paths, so in the default deployment append-only is enforced by (a) the CI grep guard (`AuditLogAppendOnlyGuardTests`) with its single marked DELETE exemption, and (b) code review. The `scadabridge_audit_writer` / `scadabridge_audit_purger` roles are **optional DBA hardening**: a deployment that wants database-level enforcement must provision two logins (runtime → writer role; a maintenance job/second connection → purger role) — and note the purger role now carries `CREATE TABLE` + scoped `DELETE` (Task 6) so it can genuinely execute the switch-out and per-channel purge. Remove the false claim "row-level DELETE is not granted even to purge" (stale since `PerChannelRetentionDays` shipped). 2. In Component-ConfigurationDatabase.md fix the same passage's stale "single global value in v1, no per-channel overrides" claim while in the file (coordinates with Task 24 — do this bullet here since it is in the same paragraph; Task 24 skips it if already fixed). 3. Verify no other doc contradicts: `grep -rn "audit_writer\|audit_purger" docs/`. 4. Commit: `git add docs/requirements/Component-AuditLog.md docs/requirements/Component-ConfigurationDatabase.md && git commit -m "docs(audit-log): document append-only enforcement honestly (CI guard default; DB roles are optional DBA hardening)"` ### Task 8: SiteCalls filtered non-terminal index (migration) **Classification:** high-risk **Estimated implement time:** ~4 min **Parallelizable with:** 1, 2, 3, 5, 11 (NOT with Task 6 — migration serialization) **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/SiteCallEntityTypeConfiguration.cs` (~66-88; also fix the now-stale "No index — … per-site, not per-node" SourceNode comment) - Create: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/_AddSiteCallsNonTerminalIndex.cs` (scaffolded) - Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/SchemaConfigurationTests.cs` (extend) 1. Failing model test (SchemaConfigurationTests already asserts index shapes): ```csharp [Fact] public void SiteCalls_Has_Filtered_NonTerminal_Index() { var entity = Model.FindEntityType(typeof(SiteCall))!; var index = entity.GetIndexes().Single(i => i.GetDatabaseName() == "IX_SiteCalls_NonTerminal"); Assert.Equal(new[] { nameof(SiteCall.CreatedAtUtc) }, index.Properties.Select(p => p.Name)); Assert.Equal("[TerminalAtUtc] IS NULL", index.GetFilter()); Assert.Equal(new[] { "SourceSite", "SourceNode", "Status" }, index.GetIncludeProperties()); } ``` 2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter SiteCalls_Has_Filtered_NonTerminal_Index` — FAIL. 3. Add to `SiteCallEntityTypeConfiguration.Configure`: ```csharp // NonTerminal backs every "live queue" KPI predicate (TerminalAtUtc IS NULL): // buffered/stuck/oldest counts run every 60 s from the KPI recorder plus every // 10 s Health-dashboard poll. Filtered: the non-terminal population is the live // queue, not the 365-day archive — each count becomes a small seek instead of a // clustered scan (arch-review 04, P1). builder.HasIndex(s => s.CreatedAtUtc) .HasDatabaseName("IX_SiteCalls_NonTerminal") .HasFilter("[TerminalAtUtc] IS NULL") .IncludeProperties(nameof(SiteCall.SourceSite), nameof(SiteCall.SourceNode), nameof(SiteCall.Status)); ``` 4. Scaffold the migration (build first, NO `--no-build`): `dotnet build ZB.MOM.WW.ScadaBridge.slnx && cd src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase && dotnet ef migrations add AddSiteCallsNonTerminalIndex` — verify the scaffold contains the `CreateIndex` with filter (if it is empty, the build/`--no-build` gotcha bit — delete and redo). 5. Run the model test — PASS. With infra up, run the MSSQL-backed repo suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter SiteCallAuditRepositoryTests` — PASS. 6. Production script: `dotnet ef migrations script FixAuditPurgerRoleGrants AddSiteCallsNonTerminalIndex --idempotent --output ../../docs/plans/sql/AddSiteCallsNonTerminalIndex.sql`. 7. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests docs/plans/sql && git commit -m "perf(site-call-audit): filtered IX_SiteCalls_NonTerminal index — KPI predicates seek the live queue instead of scanning 365 days"` ### Task 9: Bound the execution-tree edge scan with a root time window **Classification:** standard **Estimated implement time:** ~5 min **Parallelizable with:** 1, 2, 3, 11, 13, 18, 19 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/AuditLogRepository.cs` (`GetExecutionTreeAsync`, ~632-790) - Modify: `docs/requirements/Component-AuditLog.md` (audit-tree section: document the 7-day traversal bound) - Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/ExecutionIdCorrelationTests.cs` or the existing execution-tree test file (extend; MSSQL) 1. Failing test (add to the existing MSSQL execution-tree suite — locate with `grep -rln "GetExecutionTreeAsync" tests/`): ```csharp [Fact] public async Task ExecutionTree_Bounds_EdgeScan_To_Root_Window() { // Root execution at T0, child at T0+5min → both returned. // Unrelated execution pair at T0+30 days sharing no ancestry → NOT returned (was never returned; // this asserts correctness is preserved), AND a genuine descendant stamped at T0+30 days // (beyond the 7-day traversal bound) is documented-excluded: assert it is absent. } ``` 2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter ExecutionTree_Bounds_EdgeScan` — FAIL (descendant beyond bound is currently returned). 3. Implementation in `GetExecutionTreeAsync`, between Phase 1 and Phase 2: - New constants: `private static readonly TimeSpan ExecutionTreeWindowSlack = TimeSpan.FromHours(1);` and `private static readonly TimeSpan ExecutionTreeMaxSpan = TimeSpan.FromDays(7);` with a comment: execution trees span minutes, not years; the window enables partition elimination on the Edges anchor (arch-review 04, P2). - Query `SELECT MIN(OccurredAtUtc) FROM dbo.AuditLog WHERE ExecutionId = @root;` (a seek on `IX_AuditLog_Execution`). - If non-NULL, parameterize the Edges CTE anchor: `WHERE ExecutionId IS NOT NULL AND OccurredAtUtc >= @winStart AND OccurredAtUtc < @winEnd` with `@winStart = rootFirst - slack`, `@winEnd = rootFirst + maxSpan`. If NULL (row-less stub root), keep the unbounded form (current behavior) — correctness over speed for the degenerate case. 4. Run the execution-tree filter plus the full existing suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter "FullyQualifiedName~ExecutionTree|FullyQualifiedName~ExecutionId"` — PASS. 5. Doc: Component-AuditLog.md audit-tree section — "tree traversal is bounded to a 7-day window anchored at the root's first event (configurable constant); descendants beyond the window are excluded by design." 6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/AuditLogRepository.cs docs/requirements/Component-AuditLog.md tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests && git commit -m "perf(audit-log): bound GetExecutionTreeAsync edge scan to a root-anchored time window (partition elimination)"` ### Task 10: Fix the hardcoded-zero backlogTotal KPI history **Classification:** standard **Estimated implement time:** ~5 min **Parallelizable with:** 1, 2, 3, 6, 8, 11, 19, 21 **Files:** - Create: `src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Kpi/IAuditBacklogProvider.cs` - Create: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/Kpi/CentralHealthAuditBacklogProvider.cs` - Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ServiceCollectionExtensions.cs` (central registration; locate the file — `grep -n AddSingleton src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ServiceCollectionExtensions.cs`) - Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Kpi/AuditLogKpiSampleSource.cs` - Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Kpi/AuditLogKpiSampleSourceTests.cs` (extend/create), `tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests` (provider test) 1. Failing test: ```csharp [Fact] public async Task Collect_Uses_BacklogProvider_When_Present() { var repo = new FakeAuditLogRepository(new AuditLogKpiSnapshot(10, 1, BacklogTotal: 0, DateTime.UtcNow)); var source = new AuditLogKpiSampleSource(repo, new FakeBacklogProvider(42)); var samples = await source.CollectAsync(DateTime.UtcNow); Assert.Equal(42, samples.Single(s => s.Metric == KpiMetrics.AuditLog.BacklogTotal).Value); } [Fact] public async Task Collect_FallsBack_To_Snapshot_When_Provider_Absent() { // ctor with provider: null → BacklogTotal sample = snapshot value (0) — unchanged behavior } ``` 2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter AuditLogKpiSampleSource` — FAIL. 3. Implementation: - Commons: `public interface IAuditBacklogProvider { long GetPendingBacklogTotal(); }` (XML doc: sums latest per-site `SiteAuditBacklog.PendingCount` health reports; best-effort — sites not yet reporting contribute zero). - HealthMonitoring: `CentralHealthAuditBacklogProvider(ICentralHealthAggregator)` — port the exact summation from `CentralUI/Services/AuditLogQueryService.GetKpiSnapshotAsync` (~lines 145-155: `state.LatestReport?.SiteAuditBacklog?.PendingCount`, `> 0` guard). Register `services.AddSingleton();` in the central HealthMonitoring registration. - `AuditLogKpiSampleSource`: ctor gains `IAuditBacklogProvider? backlogProvider = null` (MS.DI resolves the default when unregistered — sites/tests unaffected); backlog sample value becomes `_backlogProvider?.GetPendingBacklogTotal() ?? snapshot.BacklogTotal`. - Optionally refactor `AuditLogQueryService` to consume the provider (skip if it drags UI DI churn — the live tile already works; the *history* was the broken surface). 4. Run filter — PASS. Build the solution. 5. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.Commons src/ZB.MOM.WW.ScadaBridge.HealthMonitoring src/ZB.MOM.WW.ScadaBridge.AuditLog tests && git commit -m "fix(kpi-history): backlogTotal trend records the real site-backlog aggregate instead of a hardwired zero"` ### Task 11: SiteCalls same-rank freshness upsert (unfreeze RetryCount/LastError) **Classification:** high-risk **Estimated implement time:** ~5 min **Parallelizable with:** 1, 2, 3, 4, 8, 9, 10 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs` (rank comment ~27-40; UPDATE predicate ~117-139) - Modify: `docs/requirements/Component-SiteCallAudit.md` (~76-78 upsert contract wording) - Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/SiteCallAuditRepositoryTests.cs` (extend; MSSQL — infra up) 1. Failing tests: ```csharp [Fact] public async Task Upsert_SameRank_NewerUpdatedAt_RefreshesProgressFields() { var id = NewId(); await Repo.UpsertAsync(Call(id, "Attempted", retryCount: 1, lastError: "timeout #1", updatedAt: T0)); await Repo.UpsertAsync(Call(id, "Attempted", retryCount: 5, lastError: "timeout #5", updatedAt: T0.AddMinutes(5))); var row = await Repo.GetAsync(id); Assert.Equal(5, row!.RetryCount); Assert.Equal("timeout #5", row.LastError); } [Fact] public async Task Upsert_SameRank_OlderUpdatedAt_IsNoOp() { /* retry 5 then out-of-order retry 2 with older stamp → stays 5 */ } [Fact] public async Task Upsert_LowerRank_IsStillNoOp() { /* Delivered then Attempted (newer stamp) → stays Delivered */ } ``` 2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter SiteCallAuditRepositoryTests` — FAIL (same-rank is a no-op today). 3. Change the UPDATE guard from `incomingRank > CASE...` to: ```sql AND ( {incomingRank} > (CASE Status ... END) OR ({incomingRank} = (CASE Status ... END) AND UpdatedAtUtc < {siteCall.UpdatedAtUtc}) ) ``` (repeat the CASE expression — both branches parameterize identically). Update the class comment: "monotonic on rank; within equal rank, newest `UpdatedAtUtc` wins (still idempotent — equal stamps are a no-op — and still regression-proof)". This unfreezes `RetryCount`/`LastError`/`HttpStatus` during the Attempted phase and lets a genuinely-newer terminal correction land. 4. Run filter — PASS (all pre-existing monotonic tests must stay green). 5. Doc: Component-SiteCallAudit.md upsert paragraph — replace "first-write-wins at each rank" semantics with "insert-if-not-exists, then upsert on newer status rank, with a newest-`UpdatedAtUtc` tiebreaker within equal rank so retrying calls show live RetryCount/LastError". 6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs docs/requirements/Component-SiteCallAudit.md tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests && git commit -m "fix(site-call-audit): same-rank freshness tiebreaker — retrying calls no longer freeze RetryCount/LastError at first write"` ### Task 12: EF Core connection resiliency (`EnableRetryOnFailure`) **Classification:** high-risk **Estimated implement time:** ~5 min **Parallelizable with:** 1, 2, 3, 8, 11 (do NOT run concurrently with any task executing MSSQL integration tests — behavior-wide change) **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ServiceCollectionExtensions.cs` (~29-34) - Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogIngestActor.cs` (`OnCachedTelemetryAsync` dual-write transaction ~262-311) - Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/ServiceCollectionExtensionsTests.cs` (extend), `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/CombinedTelemetryIdempotencyTests.cs` (must stay green) 1. First inventory every user-initiated transaction against `ScadaBridgeDbContext`: `grep -rn "BeginTransaction" src/ --include="*.cs"` — expect `AuditLogIngestActor.OnCachedTelemetryAsync`; treat any others found identically. 2. Failing test in `ServiceCollectionExtensionsTests`: ```csharp [Fact] public void AddConfigurationDatabase_Configures_RetryingExecutionStrategy() { var services = new ServiceCollection(); services.AddConfigurationDatabase("Server=unused;Database=unused;"); using var sp = services.BuildServiceProvider(); var options = sp.GetRequiredService>(); var sqlExt = options.Extensions.OfType().Single(); Assert.NotNull(sqlExt.ExecutionStrategyFactory); // retry strategy configured } ``` 3. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter Configures_RetryingExecutionStrategy` — FAIL. 4. Implementation: - `options.UseSqlServer(connectionString, sql => sql.EnableRetryOnFailure(maxRetryCount: 5, maxRetryDelay: TimeSpan.FromSeconds(30), errorNumbersToAdd: null))` — comment: covers every read-side path (KPI asks, outbox queries, `Notify.Status`, execution trees) that previously surfaced transient faults raw (arch-review 04, S5). - `OnCachedTelemetryAsync`: wrap the per-entry `BeginTransactionAsync` block in the execution strategy — `var strategy = context.Database.CreateExecutionStrategy(); await strategy.ExecuteAsync(async () => { await using var tx = await context.Database.BeginTransactionAsync(); ... await tx.CommitAsync(); });` — the block is already idempotent (insert-if-not-exists + monotonic upsert), so a retried unit is safe; say so in the comment. - Note in `SwitchOutPartitionAsync`'s comment: the raw batch carries its own server-side BEGIN TRAN/CATCH and is IF-EXISTS-guarded idempotent, so strategy-level replay of the whole batch is safe. 5. Run: the new filter → PASS; then with infra up, `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter "CombinedTelemetry"` → PASS (proves the dual-write still commits atomically under the strategy). 6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ServiceCollectionExtensions.cs src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogIngestActor.cs tests && git commit -m "fix(config-db): EnableRetryOnFailure fleet-wide + execution-strategy wrap of the combined-telemetry dual-write"` ### Task 13: Audit row for operator Retry on parked notifications **Classification:** standard **Estimated implement time:** ~5 min **Parallelizable with:** 8, 9, 11, 18, 19, 21 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Notification/NotificationOutboxQueries.cs` (`RetryNotificationRequest` ~54, `DiscardNotificationRequest` ~69 — additive `string? RequestedBy = null`) - Modify: `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/NotificationOutboxActor.cs` (`RetryAsync` ~951-977; `BuildNotifyDeliverEvent` ~675-714 gains `string? actorOverride = null`; `EmitTerminalAuditAsync` passes it for discard) - Modify: callers that send these requests — locate with `grep -rln "RetryNotificationRequest(" src/` (expect the Central UI outbox service and/or ManagementActor) and pass the authenticated username - Modify: `docs/requirements/Component-NotificationOutbox.md` (~128) - Test: `tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests/NotificationOutboxActorRetryEmissionTests.cs` (create; clone the audit-writer capture rig from `NotificationOutboxActorTerminalEmissionTests.cs`) 1. Failing test: ```csharp [Fact] public async Task Retry_OnParked_Emits_Submitted_NotifyDeliver_AuditRow_With_Operator() { // seed a Parked notification; send RetryNotificationRequest(corr, id, RequestedBy: "jdoe") // expect RetryNotificationResponse Success // assert captured audit writer received exactly one event: // Kind NotifyDeliver, Status Submitted, Actor "jdoe", CorrelationId == NotificationId } [Fact] public async Task Retry_AuditFailure_DoesNotFail_TheRetry() { /* throwing audit writer → response still Success (audit is best-effort) */ } ``` 2. `dotnet test tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests --filter RetryEmission` — FAIL. 3. Implementation: - Add `string? RequestedBy = null` to both request records (additive-only contract evolution — existing senders compile unchanged). - `BuildNotifyDeliverEvent(..., string? actorOverride = null)`: `actor: actorOverride ?? SystemActor`. - In `RetryAsync`, after `repository.UpdateAsync(notification)`: emit `BuildNotifyDeliverEvent(notification, now, AuditStatus.Submitted, errorMessage: null, actorOverride: request.RequestedBy)` via `_auditWriter.WriteAsync` inside the same try/catch-and-log shape as `EmitTerminalAuditAsync` (audit failure never aborts the retry). `Submitted` = "operator re-queued it" — the forensic gap the review names (`Parked → Attempted → Delivered` with no record of who un-parked). - `DiscardAsync`: pass `request.RequestedBy` through to the terminal row's actor. - Plumb the username at the senders (the UI/management layer knows the authenticated user; pass `null` where no identity exists). 4. Run filter + `dotnet test tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests` — PASS. 5. Doc: Component-NotificationOutbox.md ~128 — "…each transition emits the corresponding audit row: operator **Retry** emits a `Notification`-channel `NotifyDeliver` row with status `Submitted` and the operator as `Actor`; Discard emits the `Terminal` row (operator as `Actor`)." 6. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.Commons src/ZB.MOM.WW.ScadaBridge.NotificationOutbox src/ZB.MOM.WW.ScadaBridge.CentralUI src/ZB.MOM.WW.ScadaBridge.ManagementService docs/requirements/Component-NotificationOutbox.md tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests && git commit -m "feat(notification-outbox): operator Retry/Discard emit audit rows with operator identity"` ### Task 14: Operator identity on the SiteCalls Retry/Discard relay **Classification:** standard **Estimated implement time:** ~5 min **Parallelizable with:** 8, 9, 10, 19, 20 **Files:** - Modify: the relay request records — locate with `grep -rn "record RetryParkedOperation\|record DiscardParkedOperation" src/ZB.MOM.WW.ScadaBridge.Commons` (additive `string? RequestedBy = null`) - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/SiteCallAuditActor.cs` (relay handlers — see `SiteCallRelayTests.cs` for the handler names) - Modify: `docs/requirements/Component-SiteCallAudit.md` (relay section) - Test: `tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests/SiteCallRelayTests.cs` (extend) 1. Failing test: relay a Retry for a parked row with `RequestedBy: "jdoe"`; assert the injected central audit writer (same `ICentralAuditWriter` seam `NotificationOutboxActor` uses — confirm with `grep -rn "ICentralAuditWriter" src/ZB.MOM.WW.ScadaBridge.NotificationOutbox src/ZB.MOM.WW.ScadaBridge.Commons`) received one event: `Kind = CachedResolve`, `Status = Submitted`, `Actor = "jdoe"`, `CorrelationId = TrackedOperationId`, channel taken from the stored `SiteCalls` row's `Channel`. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests --filter SiteCallRelayTests` — FAIL. 2. Implementation: inject `ICentralAuditWriter` into `SiteCallAuditActor` (optional ctor param defaulting to null → no-op, so existing tests construct unchanged); in the Retry/Discard relay handlers, after the site relay succeeds, fetch the row (`ISiteCallAuditRepository.GetAsync`) for its `Channel`, build the event via `ScadaBridgeAuditEventFactory.Create` (mirror `BuildNotifyDeliverEvent`'s shape), write best-effort (try/catch + log — the relay outcome is authoritative, audit never aborts it). Site-side telemetry still records the state change itself; this row adds *who asked*. 3. Run filter — PASS. 4. Doc: Component-SiteCallAudit.md relay section — the relay now emits a central direct-write audit row carrying operator identity; sites remain the source of truth for the state change. 5. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.Commons src/ZB.MOM.WW.ScadaBridge.SiteCallAudit docs/requirements/Component-SiteCallAudit.md tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests && git commit -m "feat(site-call-audit): relay Retry/Discard emit operator-identity audit rows"` ### Task 15: Composite keyset in the SiteCalls reconciliation pull contract (site side) **Classification:** high-risk **Estimated implement time:** ~5 min **Parallelizable with:** 1, 2, 3, 8, 13 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto` (additive `string after_id` on the SiteCalls pull request message — locate exact message with `grep -n "PullSiteCalls" src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto`) - Modify: the Commons pull-request message (`grep -rn "record CachedCallReconcileRequest\|PullSiteCallsRequest" src/ZB.MOM.WW.ScadaBridge.Commons`) — additive `string? AfterId = null` - Modify: the site-side pull handler + tracking-store query (`grep -rn "ReadSince\|PullSiteCalls" src/ZB.MOM.WW.ScadaBridge.StoreAndForward src/ZB.MOM.WW.ScadaBridge.Communication --include="*.cs"`) — order by `(UpdatedAtUtc, TrackedOperationId)`, filter `UpdatedAtUtc > since OR (UpdatedAtUtc = since AND TrackedOperationId > afterId)` (when `afterId` present; absent → legacy `>= since` behavior) - Test: site-side handler/store tests (same project as the handler; locate via the greps above) 1. Failing test on the site store/handler: seed `batchSize + 2` rows sharing one `UpdatedAtUtc`; pull with `since = thatTimestamp, afterId = `; assert page 2 returns the remaining rows (today it returns the same page). Run the owning test project with `--filter ` — FAIL. 2. Implement the additive proto field (field number = next free; additive-only rule), regenerate/compile, map it through the DTO, and apply the keyset predicate + deterministic `(UpdatedAtUtc, TrackedOperationId)` ordering in the store query. Absent `AfterId` preserves the exact legacy contract (old central against new site: unchanged). 3. Run the owning project's full suite — PASS. 4. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.Communication src/ZB.MOM.WW.ScadaBridge.Commons src/ZB.MOM.WW.ScadaBridge.StoreAndForward tests && git commit -m "feat(site-call-audit): additive (UpdatedAtUtc, TrackedOperationId) keyset in the reconciliation pull contract"` ### Task 16: Central keyset cursor — un-pin the SiteCalls reconciliation loop **Classification:** high-risk **Estimated implement time:** ~5 min **Parallelizable with:** 5, 8, 9, 10 (requires Task 15) **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/SiteCallAuditActor.cs` (reconciliation loop ~600-670: cursor becomes `(DateTime, string?)`; pin branch ~636-654) - Create: `src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/SiteCallReconciliationPinnedChanged.cs` (EventStream record, mirroring `SiteAuditTelemetryStalledChanged`) - Modify: `docs/requirements/Component-SiteCallAudit.md` (reconciliation cursor paragraph) - Test: `tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests/SiteCallAuditReconciliationTests.cs` (extend) 1. Failing test: fake pull client honoring the keyset; seed `2 × batchSize` rows sharing one `UpdatedAtUtc`; drive one tick; assert **all** rows were upserted (today the tail never reconciles). Second test: a fake *legacy* client that ignores `AfterId` (returns the same page) → assert the actor publishes `SiteCallReconciliationPinnedChanged(siteId, pinned: true)` on the EventStream instead of silently logging. `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests --filter Reconciliation` — FAIL. 2. Implement: per-site cursor `(DateTime since, string? afterId)`; after each page set `afterId` to the max `(UpdatedAtUtc, TrackedOperationId)` row's id; pass it on the next pull. Keep the no-progress detection as the legacy-site fallback, but publish the new event (and clear it with `pinned: false` when progress resumes) — the dead-end becomes a health-observable condition. 3. Run filter — PASS; run the full SiteCallAudit.Tests suite — PASS. 4. Doc: Component-SiteCallAudit.md — cursor is now a composite `(UpdatedAtUtc, TrackedOperationId)` keyset (additive contract field; legacy sites fall back to the timestamp cursor and surface `SiteCallReconciliationPinnedChanged` when pinned). Also add one line to Component-AuditLog.md's reconciliation section: the analogous keyset for the `PullAuditEvents` cursor is a tracked follow-up (same shape; lower urgency because the audit cursor already re-pulls idempotently). 5. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.SiteCallAudit docs/requirements tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests && git commit -m "fix(site-call-audit): composite keyset cursor eliminates the single-timestamp reconciliation pin; pinned state now a published event"` ### Task 17: Durable record for permanently-abandoned reconciliation rows **Classification:** standard **Estimated implement time:** ~5 min **Parallelizable with:** 8, 11, 19, 20, 21 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Types/Enums/AuditKind.cs` (additive `ReconciliationAbandoned` value) - Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/SiteAuditReconciliationActor.cs` (abandonment branch ~283-293) - Modify: `docs/requirements/Component-AuditLog.md` (kind vocabulary + reconciliation section) - Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/SiteAuditReconciliationActorTests.cs` (extend) 1. Failing test: repository stub whose `InsertIfNotExistsAsync` throws for one specific EventId across `MaxPermanentInsertAttempts` ticks but accepts everything else; after the abandonment tick, assert a **synthetic** audit event was inserted (new EventId; `Kind = ReconciliationAbandoned`; `DetailsJson`/Extra carrying the abandoned `EventId`, the site id, and the final error) and the cursor advanced. `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter SiteAuditReconciliationActorTests` — FAIL. 2. Implement in the `attempts >= MaxPermanentInsertAttempts` branch: alongside the existing `LogCritical`, build a minimal synthetic event (fresh `Guid.NewGuid()` EventId, `OccurredAtUtc = now`, Actor = system, Action `"Audit.ReconciliationAbandoned"`, Category preserved from the lost event where parseable, Extra = `{ abandonedEventId, sourceSiteId, error }` — use the same factory/projection helpers the ingest path uses) and `InsertIfNotExistsAsync` it in its own try/catch (the synthetic row is small and well-formed, so the row-specific failure that killed the original won't recur; if it does, log and continue — never block the cursor twice). The permanent loss is now queryable in the audit log itself, not only in a rotating log file. 3. Run filter — PASS. 4. Doc: add `ReconciliationAbandoned` to the kind vocabulary in Component-AuditLog.md (Task 24 updates the Commons enum listing — coordinate: this task adds the value, 24 fixes the counts). 5. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.Commons src/ZB.MOM.WW.ScadaBridge.AuditLog docs/requirements/Component-AuditLog.md tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests && git commit -m "feat(audit-log): permanently-abandoned reconciliation rows leave a durable ReconciliationAbandoned audit record"` ### Task 18: Reconciliation NodeB failover (owned from report 08) **Classification:** standard **Estimated implement time:** ~5 min **Parallelizable with:** 9, 11, 13, 19, 20, 21 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/ISiteEnumerator.cs` (`SiteEntry` record ~37: additive `string? FallbackGrpcEndpoint = null`; fix the stale "NodeA-only first cut" doc comment) - Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/SiteEnumerator.cs` (~60-75: populate from `GrpcNodeBAddress`; include sites with blank NodeA but valid NodeB, primary = NodeA else NodeB) - Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/GrpcPullAuditEventsClient.cs` + `GrpcPullSiteCallsClient.cs` (retry-once-on-fallback) - Modify: `docs/requirements/Component-AuditLog.md` (reconciliation endpoint-resolution paragraph) - Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/SiteEnumeratorTests.cs`, `GrpcPullAuditEventsClientTests.cs`, `GrpcPullSiteCallsClientTests.cs` (extend) 1. Failing tests: (a) `SiteEnumeratorTests` — site with both addresses yields `SiteEntry(id, nodeA, FallbackGrpcEndpoint: nodeB)`; site with only NodeB is no longer skipped; (b) `GrpcPullAuditEventsClientTests` — fake invoker throws `Unavailable` for the primary endpoint and succeeds for the fallback → the pull returns the fallback's events (today: collapses to empty). `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter "SiteEnumeratorTests|GrpcPull"` — FAIL. 2. Implement: in both pull clients' transport-fault catch (the `Unavailable`/`DeadlineExceeded`/`Cancelled`/`HttpRequestException` set already enumerated there), if `FallbackGrpcEndpoint` is non-blank and differs from the primary, invoke once against it before collapsing to empty; log the failover at Information. The invoker's per-endpoint channel cache already handles the second endpoint. 3. Run filter + full Central test folder — PASS. 4. Doc: Component-AuditLog.md — reconciliation dials NodeA first, fails over to NodeB per call; during a NodeA outage the loss-recovery safety net stays available. 5. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.AuditLog docs/requirements/Component-AuditLog.md tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests && git commit -m "fix(audit-log): reconciliation pulls fail over to site NodeB when NodeA is unreachable"` ### Task 19: Batch the KpiSample purge **Classification:** standard **Estimated implement time:** ~4 min **Parallelizable with:** 1, 2, 3, 8, 9, 13, 18 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs` (`PurgeOlderThanAsync` ~65-71) - Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IKpiHistoryRepository.cs` (doc update; signature unchanged) - Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/KpiHistoryRepositoryTests.cs` (extend) 1. Failing test: seed samples spread across 5 days; purge with a 2-day-old cutoff; assert the correct rows are gone and the return value equals the total deleted — then assert via an EF command interceptor (or SQL capture, matching how sibling tests inspect SQL) that **more than one** DELETE statement was issued for a multi-window purge. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter KpiHistoryRepositoryTests` — FAIL (single `ExecuteDeleteAsync` today). 2. Implement provider-portable time-slice batching (avoids `DELETE TOP` dialect divergence — the convention Task 24 documents): ```csharp public async Task PurgeOlderThanAsync(DateTime before, CancellationToken cancellationToken = default) { // Time-sliced batches: each DELETE covers at most one hour of samples, capping the // lock/log footprint per statement (arch-review 04, P4 — steady state is ~1 day of // rows/day; after an outage the catch-up would otherwise be one giant transaction). var total = 0; var floor = await _context.KpiSamples.Where(s => s.CapturedAtUtc < before) .MinAsync(s => (DateTime?)s.CapturedAtUtc, cancellationToken); while (floor is not null && floor < before) { var ceiling = floor.Value.AddHours(1) < before ? floor.Value.AddHours(1) : before; total += await _context.KpiSamples .Where(s => s.CapturedAtUtc < ceiling) .ExecuteDeleteAsync(cancellationToken); floor = await _context.KpiSamples.Where(s => s.CapturedAtUtc < before) .MinAsync(s => (DateTime?)s.CapturedAtUtc, cancellationToken); } return total; } ``` 3. Run filter — PASS; run `dotnet test tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests` (recorder actor contract unchanged) — PASS. 4. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IKpiHistoryRepository.cs tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests && git commit -m "perf(kpi-history): time-sliced batched purge replaces the single daily mega-DELETE"` ### Task 20: NotificationOutbox KPI — single-query aggregation, bounded oldest **Classification:** standard **Estimated implement time:** ~5 min **Parallelizable with:** 1, 2, 3, 8, 9, 14, 18, 19 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/NotificationOutboxRepository.cs` (`ComputeKpisAsync` ~239-283, `ComputePerSiteKpisAsync` ~286-334, `ComputePerNodeKpisAsync` ~337-391) - Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/NotificationOutboxRepositoryPerSiteKpiTests.cs`, `...PerNodeKpiTests.cs`, and the global KPI tests (all must stay green — this is a pure refactor of query shape) 1. Add one new failing test capturing the *bounded-oldest* behavior: seed 3 non-terminal rows; assert the oldest-pending computation issues no query that materializes more than 1 row (assert via command interceptor / captured SQL: the oldest query is `ORDER BY … OFFSET 0 ROWS FETCH NEXT 1` or `TOP(1)`, not a full non-terminal SELECT). Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter NotificationOutboxRepository` — FAIL (per-site/per-node paths materialize every non-terminal row today, ~lines 312-318, 368-375). 2. Refactor each snapshot to: - **Counts:** one conditional-aggregation query per scope — `GroupBy(n => 1)` (global) / `GroupBy(n => n.SourceSiteId)` / `GroupBy(n => n.SourceNode)` with `g.Count(n => )` per metric, replacing the 5-7 sequential round trips. - **Oldest pending:** global — `Where(nonTerminal).OrderBy(n => n.CreatedAt).Select(n => (DateTimeOffset?)n.CreatedAt).FirstOrDefaultAsync()`; per-site/per-node — try server-side `GroupBy(...).Select(g => new { g.Key, Oldest = g.Min(n => n.CreatedAt) })` first; if the `DateTimeOffset` converter blocks translation (the review's noted awkwardness), fall back to projecting only `(Key, CreatedAt)` pairs of non-terminal rows (narrow projection, no entity materialization) and reduce in memory — either way the full-entity materialization is gone. 3. Run the full NotificationOutboxRepository test set + `dotnet test tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests --filter Kpi` — PASS (identical snapshot values). 4. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/NotificationOutboxRepository.cs tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests && git commit -m "perf(notification-outbox): single-query KPI aggregation; oldest-pending no longer materializes the live queue"` ### Task 21: `OPTION (RECOMPILE)` on the SiteCalls catch-all query **Classification:** small **Estimated implement time:** ~2 min **Parallelizable with:** 1, 2, 3, 8, 9, 13, 18, 19, 20 (NOT 11 — same file) **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs` (`QueryAsync` SQL ~186-202) - Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/SiteCallAuditRepositoryTests.cs` (existing query tests must pass) 1. Append `OPTION (RECOMPILE)` after the `ORDER BY` line of the `FormattableString` (with a comment: every filter is the `(@p IS NULL OR col=@p)` optional-parameter shape; RECOMPILE lets the optimizer prune dead predicates per invocation and pick `IX_SiteCalls_Status_Updated`/`IX_SiteCalls_NonTerminal` — arch-review 04, P5; per-invocation compile cost is fine at UI-page cadence). 2. With infra up: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter SiteCallAuditRepositoryTests` — PASS. 3. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs && git commit -m "perf(site-call-audit): OPTION (RECOMPILE) on the optional-parameter query page"` ### Task 22: Short first-tick delay on the three purge timers **Classification:** small **Estimated implement time:** ~3 min **Parallelizable with:** 1, 2, 6, 8, 19, 20, 21 (NOT 4/5 — AuditLogPurgeActor; NOT 16 — SiteCallAuditActor) **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeActor.cs` (`PreStart` ~95-100) - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/SiteCallAuditActor.cs` (`StartPurgeTimer` ~374-387) - Modify: `src/ZB.MOM.WW.ScadaBridge.KpiHistory/KpiHistoryRecorderActor.cs` (purge timer ~124-128) - Test: extend each actor's existing test file with one fast-first-tick case 1. Failing test (one per actor, using each suite's existing scheduler/interval-override rig): configure a long interval (24h) and assert a purge tick arrives within the short initial delay window rather than never. 2. Implement uniformly: `var initialDelay = interval < ShortFirstTick ? interval : ShortFirstTick;` with `private static readonly TimeSpan ShortFirstTick = TimeSpan.FromMinutes(5);` and a shared comment: purges are idempotent; `initialDelay = interval` meant a node recycled daily would never purge (arch-review 04, S9). Taking `min(interval, 5min)` keeps millisecond test cadences unchanged. 3. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter AuditLogPurgeActorTests && dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests --filter Purge && dotnet test tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests` — PASS. 4. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.AuditLog src/ZB.MOM.WW.ScadaBridge.SiteCallAudit src/ZB.MOM.WW.ScadaBridge.KpiHistory tests && git commit -m "fix(purge): short first tick on all three purge timers so daily-recycled nodes still purge"` ### Task 23: Low-severity cleanup batch (param chunking, KPI catalog, computed-column guard) **Classification:** small **Estimated implement time:** ~5 min **Parallelizable with:** 4, 5, 8, 9, 11, 20 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriter.cs` (`MarkForwardedAsync` ~644-664, `MarkReconciledAsync` ~735-753) - Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Types/Kpi/KpiMetrics.cs` (locate: `grep -rn "class KpiMetrics" src/ZB.MOM.WW.ScadaBridge.Commons`) - Modify: `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/Kpi/NotificationOutboxKpiSampleSource.cs` (~40-43) - Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/AuditLogEntityTypeConfiguration.cs` (~158-164) - Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SqliteAuditWriterWriteTests.cs` (extend), `tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests/Kpi` (extend) 1. **P8 — parameter chunking:** failing test — write 1200 events, `MarkForwardedAsync(all 1200 ids)`, assert all flip to Forwarded (today: one statement with 1200+ parameters risks/violates SQLite's 999 limit as batch sizes grow). Implement: chunk `eventIds` into slices of 500 inside the existing `_writeLock` (one transaction around all chunks so the batch stays atomic); same in `MarkReconciledAsync`. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter SqliteAuditWriter` — PASS. 2. **C4 — KPI metric catalog:** add `KpiMetrics.NotificationOutbox.StuckCount = "stuckCount"` and `OldestPendingAgeSeconds = "oldestPendingAgeSeconds"` (EXACT existing string values — persisted, renames forbidden); replace the inline literals in `NotificationOutboxKpiSampleSource` with the constants; add a test asserting the constant values equal the historical strings. 3. **P7 — computed-column guard:** extend the `IngestedAtUtc` comment in `AuditLogEntityTypeConfiguration`: "NEVER use IngestedAtUtc in a WHERE predicate — non-persisted, it forces a full-table `JSON_VALUE` parse; read-side projection only (arch-review 04, P7)." 4. Run both test filters + build — PASS. 5. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.AuditLog src/ZB.MOM.WW.ScadaBridge.Commons src/ZB.MOM.WW.ScadaBridge.NotificationOutbox src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase tests && git commit -m "chore(data-layer): low-severity cleanups — SQLite param chunking, KPI metric catalog, computed-column predicate guard"` ### Task 24: Design-doc reconciliation pass **Classification:** small **Estimated implement time:** ~5 min **Parallelizable with:** all code tasks except 7 (same files); run AFTER 17 (AuditKind gains a value) **Files:** - Modify: `docs/requirements/Component-Commons.md` (~46-47, ~89) - Modify: `docs/requirements/Component-ConfigurationDatabase.md` (~58; timestamp-convention + clustered-key + repository-dialect notes) - Modify: `docs/requirements/Component-SiteCallAudit.md` (~44-51 schema section) - Modify: `docs/requirements/Component-AuditLog.md` (~396-399 per-channel config channel list) - Modify: `docs/requirements/Component-NotificationOutbox.md` (~23 "dedicated blocking-I/O dispatcher" phrasing — align with Task 25 if it has landed, else describe the actual thread-pool off-actor delivery) Concrete edits (each verifiable against the code cited in the review): 1. Component-Commons.md ~47: `AuditKind` list → the current enum (`grep -A20 "enum AuditKind" src/ZB.MOM.WW.ScadaBridge.Commons/Types/Enums/AuditKind.cs`): the original 10 + `SecuredWriteSubmitted/Approved/Rejected/Executed` (4) + `ReconciliationAbandoned` (Task 17). ~46: `AuditChannel` → 5 values incl. `SecuredWrite`. 2. Component-Commons.md ~89: `NotificationType` → "`Email`, `Sms` (shipped 2026-06-19; the Teams plan was evaluated and dropped)". 3. Component-ConfigurationDatabase.md ~58: NotificationList Type discriminator → "`Email` / `Sms`". 4. Component-ConfigurationDatabase.md ~399: "single global value in v1, no per-channel overrides" → per-channel `AuditLog:PerChannelRetentionDays` overrides exist (skip if Task 7 already fixed it). 5. Component-SiteCallAudit.md ~44-51: rewrite the schema table to the shipped columns (`Channel`/`Target`/`AuditStatus`-derived `Status` strings, `SourceNode`, no provenance columns) — align with `SiteCallEntityTypeConfiguration.cs` and Component-ConfigurationDatabase.md:64 (the current source of truth per the review). 6. Component-AuditLog.md ~396-399: per-channel retention config enumerates all five channels including `SecuredWrite` (the validator already accepts it — doc under-promised). 7. Component-Commons.md / Component-ConfigurationDatabase.md: document the `AuditLogRow` deviation (persistence-only projection of the external `ZB.MOM.WW.Audit.AuditEvent` contract — deliberately not a Commons POCO) and two conventions: **new tables use UTC `DateTime` + the Utc converter** (the `DateTimeOffset` choice on `Notification` is the documented exception that forced in-memory Min workarounds), and **new repositories write MSSQL-only SQL with MSSQL-backed test fixtures** (the NotificationOutboxRepository dual SQLite/T-SQL dialect is a documented legacy exception — finding S8). Record two tracked follow-ups: the AuditLog clustered-key `(OccurredAtUtc, EventId)` benchmark question (P3) and the per-node KPI sampling cadence / outbox keyset-paging deferrals (P4/P6). 8. Verify: `grep -rn "Teams" docs/requirements/ | grep -iv "dropped\|evaluated"` returns nothing load-bearing; `grep -n "no per-channel" docs/requirements/Component-ConfigurationDatabase.md` returns nothing. 9. Commit: `git add docs/requirements && git commit -m "docs: reconcile design docs with shipped code (enums, Sms, per-channel retention, SiteCalls schema, conventions)"` ### Task 25: Bounded parallel notification delivery **Classification:** high-risk **Estimated implement time:** ~5 min **Parallelizable with:** 8, 9, 11, 18, 19 (NOT 13 — same actor file; do 13 first) **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/NotificationOutboxOptions.cs` (add `MaxParallelDeliveries`) - Modify: `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/NotificationOutboxActor.cs` (`RunDispatchPass` ~321-347) - Modify: `docs/requirements/Component-NotificationOutbox.md` (~23 dispatcher paragraph) - Test: `tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests/NotificationOutboxActorDispatchTests.cs` (extend) 1. Failing test: adapter stub whose `Deliver` blocks on a gate and records concurrent-invocation high-water mark; enqueue 8 pending notifications; with `MaxParallelDeliveries = 4` assert high-water ≥ 2 and all 8 reach terminal status; with `MaxParallelDeliveries = 1` assert high-water == 1 (existing sequential behavior preserved). `dotnet test tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests --filter Dispatch` — FAIL. 2. Implement: `MaxParallelDeliveries` (default **4**, clamp min 1 — the review's alarm-storm scenario: sequential 5 s SMTP attempts cap throughput at ~0.2/s regardless of batch size). In `RunDispatchPass`, replace the sequential foreach with `SemaphoreSlim(maxParallel)` + `Task.WhenAll`, giving **each notification its own DI scope/repository** (rows are claimed per-notification, so no shared-context concurrency; the existing in-flight pass guard is untouched). Audit emission stays per-notification inside its task. 3. Run the full outbox suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests` — PASS (attempt/terminal emission tests prove per-row semantics survived). 4. Doc: Component-NotificationOutbox.md dispatcher section — delivery runs off the actor thread on the thread pool with bounded per-sweep parallelism (`MaxParallelDeliveries`, default 4); drop the inaccurate "dedicated blocking-I/O dispatcher" phrasing (doc drift the review flagged). 5. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.NotificationOutbox docs/requirements/Component-NotificationOutbox.md tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests && git commit -m "perf(notification-outbox): bounded parallel delivery within a dispatch sweep (default 4)"` --- ## Dependencies on other plans - **Plan 02 (Communication/S&F)** owns the S&F retry-sweep batching — Task 15's site-side tracking-store query touches adjacent code in the StoreAndForward project; coordinate file ownership if both run concurrently. - **Plan 07 (UI/Management/Security)** owns SecuredWrite emission paths — the `SourceNode`-NULL follow-up (U4) stays with them; Task 24 only fixes the doc under-promise. - **Plan 08 (Conventions/Tests)** flagged the reconciliation NodeB gap; it is implemented here (Task 18) — plan 08 should not duplicate it. - Tasks 13/14 plumb `RequestedBy` through UI/ManagementActor senders — if plan 07 refactors those senders simultaneously, land this plan's additive message change first (defaults keep old senders compiling). - Migration serialization is **intra-plan only** (Tasks 6 → 8); no other plan currently adds ConfigurationDatabase migrations, but if one does, coordinate snapshot ordering. ## Execution order **P0 (the operational time bombs — do first, in this order):** 1 → 2 → 3 (site purge chain), 4 → 5 (partition-switch hardening), 6 → 8 (migrations, serialized), 9, 10. **P1:** 11, 12, 13, 15 → 16, 17, 18. **P2 (hardening/cleanup):** 14, 19, 20, 21, 22, 23, 25, then 7 and 24 (docs last so they describe what actually landed). Intra-plan critical path: **1 → 2 → 3** and **15 → 16**; everything else is broadly parallelizable per each task's "Parallelizable with" contract. Finish with a full `dotnet build ZB.MOM.WW.ScadaBridge.slnx && dotnet test ZB.MOM.WW.ScadaBridge.slnx` (infra compose up for the MSSQL suites) before declaring the plan done.