Joseph Doherty
5bbd7689fa
Re-ran all 8 domain reviews at HEAD8c888f13against theb910f5ebbaseline: every round-1 finding source-verified (168 fixed, 0 regressions, 0 false claims); 56 new findings (1 Critical / 4 High / 15 Medium / 36 Low), concentrated in post-baseline code (anti-entropy resync, KPI rollup backfill, live alarm stream) and seams the fixes exposed. Headliners: S&F resync predicate inversion can wipe the delivering node's buffer (02-N1 Critical); resync snapshot exceeds the Akka remoting frame size (02-N2); failover drill kills the one node keep-oldest can't survive (01-N1); unbounded rollup backfill per failover (04-R1); live production API key in untracked test.txt (08-NF1). Adds PLAN-R2-01..08 + .tasks.json manifests and the Round-2 board, P0 list, cross-plan mutexes, and wave order in 00-MASTER-TRACKER.
680 lines
49 KiB
Markdown
680 lines
49 KiB
Markdown
# PLAN-R2-04 — Data & Audit Backbone Round-2 Fixes Implementation Plan
|
||
|
||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
|
||
|
||
**Goal:** Close the seven NEW findings of the round-2 architecture review (`archreview/04-data-audit-backbone.md`, 2026-07-12): bound the KPI rollup backfill that currently re-folds the entire 90-day raw window in one tracked pass on every singleton failover (R1), make the fold fetch untracked and projected (R2), fix the ~60× rate-unit discontinuity at the raw/rollup routing boundary and the last-value-per-bucket re-fold error the aggregation catalog itself warns about (R3), close the catalog metric-literal drift hazard (R4), classify the failover fold-race failure grain (R5), give the SiteCalls terminal purge the same time-sliced batching + index support its two siblings received in round 1 (R6), and stop the site retention service from surfacing its own shutdown cancellation to the host (R7). Every accepted round-1 deferral stays deferred — coverage rows only, no code.
|
||
|
||
**Architecture:** All fixes stay inside the existing component boundaries. The rollup backfill/fold hardening lives in the KpiHistory recorder actor + `KpiHistoryRepository` (one additive `IKpiHistoryRepository` watermark method in Commons); the rate-presentation fix lives in the Commons `KpiSeriesBucketer` (additive optional parameter — existing call sites compile unchanged) driven by the already-shipped `KpiMetricAggregationCatalog`, wired at the `KpiHistoryQueryService` boundary; the SiteCalls purge fix rides one EF Core migration (a filtered `TerminalAtUtc` index) plus a repository-local time-sliced DELETE loop mirroring the round-1 `KpiSample` purge; the retention-service fix is a loop-boundary OCE catch. Design docs are updated in the same task as the code they describe (repo rule: doc + code + tests travel together). No new tables, no contract-breaking changes.
|
||
|
||
**Tech Stack:** C#/.NET 8, Akka.NET (actors/TestKit), EF Core + SQL Server (central; SQLite in-memory harness for the KpiHistory repository suite), xUnit. Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per-project with targeted filters only. MSSQL-backed `SkippableFact` suites (`SiteCallAuditRepositoryTests`) need `cd infra && docker compose up -d`. **EF migrations gotcha (repo-documented):** always build first and run `dotnet ef migrations add <Name>` WITHOUT `--no-build`, or it scaffolds an empty migration off the stale DLL — delete the empty files if it happens (`migrations remove` needs a live DB).
|
||
|
||
---
|
||
|
||
### Task 1: Untracked, projected fold fetch (R2)
|
||
|
||
**Classification:** standard
|
||
**Estimated implement time:** ~3 min
|
||
**Parallelizable with:** 3, 5, 6, 7, 8, 10, 11, 12 (NOT 2/9 — same repository file)
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs` (`FoldHourlyRollupsAsync` fetch at :104-106; group lambdas :112-128)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/KpiHistoryRepositoryTests.cs` (extend)
|
||
|
||
1. Write the failing test (SQLite in-memory harness — pattern-match `NewContext()`/`Sample()` already in the file; clear the tracker after seeding so the assertion sees only the fold's own work):
|
||
|
||
```csharp
|
||
[Fact]
|
||
public async Task FoldHourlyRollups_DoesNotTrack_KpiSampleEntities()
|
||
{
|
||
await using var ctx = NewContext();
|
||
var repo = new KpiHistoryRepository(ctx);
|
||
await repo.RecordSamplesAsync(new[]
|
||
{
|
||
Sample("NotificationOutbox", "queueDepth", "Global", null, 5, Base.AddMinutes(10)),
|
||
Sample("NotificationOutbox", "queueDepth", "Global", null, 7, Base.AddMinutes(50)),
|
||
});
|
||
ctx.ChangeTracker.Clear(); // isolate the fold's tracking behavior from the seed
|
||
|
||
await repo.FoldHourlyRollupsAsync(Base, Base.AddHours(1));
|
||
|
||
// The fold READS samples and WRITES rollups; the read must not register
|
||
// thousands of read-only KpiSample entries for DetectChanges to re-scan
|
||
// (arch-review 04 round 2, R2).
|
||
Assert.Empty(ctx.ChangeTracker.Entries<KpiSample>());
|
||
}
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter FoldHourlyRollups_DoesNotTrack_KpiSampleEntities` — expect FAIL (the fetch is a tracked `ToListAsync` today).
|
||
3. Replace the fetch at :104-106 with a projection (a `Select()` projection is never tracked, and it also drops the unneeded `Id` column, roughly halving per-row memory — which shrinks the R1 backfill's footprint even before Task 3 slices it):
|
||
|
||
```csharp
|
||
// Projected, untracked fetch: the fold only reads these six fields and never
|
||
// mutates a KpiSample. A tracked ToListAsync registered ~5k-90k read-only
|
||
// entities in the change tracker per healthy 3h fold — all re-scanned by
|
||
// DetectChanges on the final SaveChanges (arch-review 04 round 2, R2). A
|
||
// projection is inherently untracked and materializes no entity at all.
|
||
var samples = await _context.KpiSamples
|
||
.Where(s => s.CapturedAtUtc >= from && s.CapturedAtUtc < to)
|
||
.Select(s => new FoldSample(s.Source, s.Metric, s.Scope, s.ScopeKey, s.CapturedAtUtc, s.Value))
|
||
.ToListAsync(cancellationToken);
|
||
```
|
||
|
||
with, at the bottom of the class next to `SeriesHourKey`:
|
||
|
||
```csharp
|
||
/// <summary>Narrow, untracked projection of one KpiSample row for the fold (R2).</summary>
|
||
private readonly record struct FoldSample(
|
||
string Source, string Metric, string Scope, string? ScopeKey, DateTime CapturedAtUtc, double Value);
|
||
```
|
||
|
||
The grouping/aggregation lambdas (:112-128) compile unchanged — `FoldSample` carries the same member names.
|
||
4. Run the new filter — PASS. Then run the whole fold suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter KpiHistoryRepositoryTests` — PASS (all existing fold/purge/query tests must stay green; this is a pure fetch-shape change).
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/KpiHistoryRepositoryTests.cs && git commit -m "fix(kpi): untracked projected fold fetch — fold reads no longer flood the change tracker (plan R2-04 T1)"`
|
||
|
||
### Task 2: Rollup watermark seam — `GetLatestRollupHourAsync` (R1, part 1)
|
||
|
||
**Classification:** standard
|
||
**Estimated implement time:** ~4 min
|
||
**Parallelizable with:** 5, 8, 10, 11, 12 (NOT 1/9 — same repository file; NOT 3/4/6 — the interface addition ripples into their test fakes)
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IKpiHistoryRepository.cs` (additive method)
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs` (implement)
|
||
- Modify: every test fake implementing `IKpiHistoryRepository` — locate with `grep -rln ": IKpiHistoryRepository" tests/` (expect the recorder-actor fake in `tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests/KpiHistoryRecorderActorTests.cs` and the query-service stub in `tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Services/KpiHistoryQueryServiceTests.cs`) — add a trivial `Task.FromResult<DateTime?>(null)` member so they compile
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/KpiHistoryRepositoryTests.cs` (extend)
|
||
|
||
1. Failing tests:
|
||
|
||
```csharp
|
||
[Fact]
|
||
public async Task GetLatestRollupHour_ReturnsNull_WhenNoRollupsExist()
|
||
{
|
||
await using var ctx = NewContext();
|
||
var repo = new KpiHistoryRepository(ctx);
|
||
Assert.Null(await repo.GetLatestRollupHourAsync());
|
||
}
|
||
|
||
[Fact]
|
||
public async Task GetLatestRollupHour_ReturnsNewestHourStart_AcrossAllSeries()
|
||
{
|
||
await using var ctx = NewContext();
|
||
var repo = new KpiHistoryRepository(ctx);
|
||
await repo.RecordSamplesAsync(new[]
|
||
{
|
||
Sample("NotificationOutbox", "queueDepth", "Global", null, 5, Base.AddMinutes(10)),
|
||
Sample("SiteCallAudit", "buffered", "Global", null, 2, Base.AddHours(3).AddMinutes(10)),
|
||
});
|
||
await repo.FoldHourlyRollupsAsync(Base, Base.AddHours(4));
|
||
|
||
Assert.Equal(Base.AddHours(3), await repo.GetLatestRollupHourAsync());
|
||
}
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter GetLatestRollupHour` — expect FAIL (method does not exist / does not compile — comment the tests out to prove the baseline if needed, standard TDD-on-new-member flow).
|
||
3. Add to `IKpiHistoryRepository` (after `GetHourlySeriesAsync`, ~:108):
|
||
|
||
```csharp
|
||
/// <summary>
|
||
/// Returns the newest <c>KpiRollupHourly.HourStartUtc</c> across all series, or
|
||
/// <c>null</c> when no rollups exist. The recorder's one-shot backfill consults
|
||
/// this watermark to skip (or shrink to) the un-rolled tail instead of re-folding
|
||
/// the whole raw-retention window on every failover (arch-review 04 round 2, R1).
|
||
/// </summary>
|
||
Task<DateTime?> GetLatestRollupHourAsync(CancellationToken cancellationToken = default);
|
||
```
|
||
|
||
Implement in `KpiHistoryRepository`: `return await _context.KpiRollupHourly.MaxAsync(r => (DateTime?)r.HourStartUtc, cancellationToken);` — a single MAX over the unique `IX_KpiRollupHourly_Series` population (rollup table is small: one row per series-hour). Add the null-returning member to each located test fake.
|
||
4. Run the filter — PASS. Build the solution (`dotnet build ZB.MOM.WW.ScadaBridge.slnx`) to prove no other implementor was missed.
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IKpiHistoryRepository.cs src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs tests && git commit -m "feat(kpi): GetLatestRollupHourAsync watermark seam for the backfill fast-path (plan R2-04 T2)"`
|
||
|
||
### Task 3: Slice the rollup backfill into bounded day windows (R1, part 2)
|
||
|
||
**Classification:** high-risk
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** 5, 6, 7, 8, 10, 11, 12 (NOT 4 — same actor file; requires Task 2's fake update)
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.KpiHistory/KpiHistoryRecorderActor.cs` (`HandleBackfillTick` :494-526, `RunBackfillPass` :536-558, the `BackfillComplete` receive :163-167)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests/KpiHistoryRecorderActorTests.cs` (extend + update the two existing backfill tests)
|
||
|
||
1. Failing test (reuse the gated-fold fake-repository rig the existing `BackfillTick_FoldsFullRetentionWindow_Once` / `BackfillInFlight_SkipsPeriodicRollupTick` tests drive by hand):
|
||
|
||
```csharp
|
||
[Fact]
|
||
public void Backfill_SlicesRetentionWindow_IntoBoundedDayFolds_OldestFirst()
|
||
{
|
||
// RetentionDays = 3 → expect 3+ fold calls, each window ≤ 24 h, contiguous,
|
||
// oldest-first, whose union is exactly [TruncateToHour(now) − 3 d, TruncateToHour(now)).
|
||
// Today: FAIL — a single fold call spans the whole 3-day window.
|
||
}
|
||
|
||
[Fact]
|
||
public void PeriodicRollupTick_Interleaves_BetweenBackfillSlices()
|
||
{
|
||
// Gate slice 1 open; release it; before dequeuing slice 2 send a RollupTick —
|
||
// the periodic fold must claim the fold path (the backfill defers via its
|
||
// re-arm branch) and ALL backfill slices must still complete afterwards.
|
||
// Today: FAIL — _backfillInFlight blocks every periodic fold for the whole pass.
|
||
}
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests --filter "Backfill_SlicesRetentionWindow|PeriodicRollupTick_Interleaves"` — expect FAIL.
|
||
3. Implementation — convert the single full-window pass into a slice queue processed one bounded window per fold, releasing the fold path between slices:
|
||
- New constant + state:
|
||
|
||
```csharp
|
||
/// <summary>
|
||
/// Width of one backfill fold slice (24 h). The repository fold materializes its whole
|
||
/// window in memory, so the one-shot backfill must never hand it the full raw-retention
|
||
/// window (90 d ≈ tens of millions of rows at fleet volume — arch-review 04 round 2, R1).
|
||
/// One day per fold bounds each pass to roughly what a steady-state day writes, and the
|
||
/// guard is lowered between slices so periodic folds interleave instead of stalling
|
||
/// behind the historical pass.
|
||
/// </summary>
|
||
private static readonly TimeSpan BackfillSliceWidth = TimeSpan.FromHours(24);
|
||
|
||
private readonly Queue<(DateTime FromHourUtc, DateTime ToHourUtc)> _backfillSlices = new();
|
||
```
|
||
|
||
- `HandleBackfillTick`: keep the `_backfillDone || _backfillInFlight` and `_rollupInFlight`-defer branches verbatim. When `_backfillSlices` is empty, build the plan — enumerate `[toHourUtc − RetentionDays, toHourUtc)` into ≤24h windows oldest-first and enqueue them (log the one "backfill starting" line with the slice count); when it is non-empty, dequeue ONE slice, raise `_backfillInFlight`, and run `RunBackfillPass(slice.FromHourUtc, slice.ToHourUtc, ct)` piping `BackfillSliceComplete.Instance` on both success and (logged) failure.
|
||
- Replace the `BackfillComplete` receive with:
|
||
|
||
```csharp
|
||
Receive<BackfillSliceComplete>(_ =>
|
||
{
|
||
_backfillInFlight = false; // release the fold path BETWEEN slices (R1)
|
||
if (_backfillSlices.Count == 0)
|
||
{
|
||
_backfillDone = true; // latch: at most once per actor lifetime (unchanged semantics)
|
||
_logger.LogInformation("KPI rollup backfill completed — all slices folded.");
|
||
return;
|
||
}
|
||
|
||
// Next slice via the timer, not inline: a RollupTick already queued in the mailbox
|
||
// is processed first, so periodic folds interleave; the existing _rollupInFlight
|
||
// deferral branch in HandleBackfillTick then re-arms the backfill behind it.
|
||
Timers.StartSingleTimer(BackfillTimerKey, BackfillTick.Instance, TimeSpan.Zero);
|
||
});
|
||
```
|
||
|
||
Rename the `BackfillComplete` message class to `BackfillSliceComplete` (internal — used only by this actor and its tests). Move the "backfill completed" wall-time log from `RunBackfillPass` (which now logs per-slice at Debug) to the queue-drained branch. Ordering note in the class doc: periodic folds and backfill slices strictly alternate through the single `_rollupInFlight`/`_backfillInFlight` gate pair — the two idempotent upserts still never race on overlapping rows.
|
||
- Update `BackfillTick_FoldsFullRetentionWindow_Once` (assert the union of slice windows covers the retention window and a post-completion `BackfillTick` is a no-op) and `BackfillInFlight_SkipsPeriodicRollupTick` (unchanged semantics — a RollupTick arriving while a *slice* is in flight is still skipped).
|
||
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests --filter KpiHistoryRecorderActorTests` — PASS (all pre-existing sample/purge/rollup tests must stay green).
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.KpiHistory/KpiHistoryRecorderActor.cs tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests/KpiHistoryRecorderActorTests.cs && git commit -m "fix(kpi): slice the rollup backfill into bounded 24h folds that yield to periodic folds (plan R2-04 T3)"`
|
||
|
||
### Task 4: Backfill failover fast-path via the rollup watermark + doc correction (R1, part 3)
|
||
|
||
**Classification:** high-risk
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** 5, 6, 8, 10, 11, 12 (NOT 3 — same actor file, do 3 first; NOT 7/9 — same design-doc file)
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.KpiHistory/KpiHistoryRecorderActor.cs` (plan branch of `HandleBackfillTick` from Task 3)
|
||
- Modify: `docs/requirements/Component-KpiHistory.md` (:93, the "One-shot backfill" paragraph)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests/KpiHistoryRecorderActorTests.cs` (extend)
|
||
|
||
1. Failing tests (the fake repository's `GetLatestRollupHourAsync` — added in Task 2 — gains a settable return value):
|
||
|
||
```csharp
|
||
[Fact]
|
||
public void Backfill_Skips_WhenRollupsAlreadyCurrent()
|
||
{
|
||
// Fake watermark = TruncateToHour(now) − 1 h (i.e. within RollupLookbackHours of now —
|
||
// the failover case, which is the common one): expect ZERO fold calls, the done-latch
|
||
// set, and an Information log; the periodic fold alone covers the lookback tail.
|
||
}
|
||
|
||
[Fact]
|
||
public void Backfill_ShrinksToWatermark_InsteadOfRetentionFloor()
|
||
{
|
||
// RetentionDays = 90, fake watermark = 2 days ago: expect the first slice to start at
|
||
// the WATERMARK HOUR (inclusive — re-folding the newest already-folded hour is a cheap
|
||
// idempotent safety margin), not 90 days ago, and ≤ 3 slices total.
|
||
}
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests --filter "Backfill_Skips_WhenRollupsAlreadyCurrent|Backfill_ShrinksToWatermark"` — expect FAIL (the plan branch always enumerates from the retention floor).
|
||
3. Implementation — make the plan branch asynchronous: when `_backfillSlices` is empty, raise `_backfillInFlight` and run a plan pass that resolves `IKpiHistoryRepository` in a fresh scope and returns `GetLatestRollupHourAsync()` (never-faulting wrapper like the other passes), piping `BackfillPlan(DateTime? Watermark)` back to Self. In the `Receive<BackfillPlan>` handler (actor thread):
|
||
|
||
```csharp
|
||
var toHourUtc = TruncateToHour(DateTime.UtcNow);
|
||
var retentionFloor = toHourUtc - TimeSpan.FromDays(_options.RetentionDays);
|
||
// Shrink to the un-rolled tail; re-fold the watermark hour itself (idempotent) as a
|
||
// safety margin. A null watermark (fresh install) keeps the full retention window.
|
||
var floor = plan.Watermark is { } w && w > retentionFloor ? w : retentionFloor;
|
||
|
||
if (floor >= toHourUtc - TimeSpan.FromHours(_options.RollupLookbackHours))
|
||
{
|
||
// Failover fast-path: rollups are already current — the periodic fold's lookback
|
||
// window covers everything from the watermark forward, so the full historical
|
||
// pass is skipped entirely (arch-review 04 round 2, R1).
|
||
_backfillInFlight = false;
|
||
_backfillDone = true;
|
||
_logger.LogInformation(
|
||
"KPI rollup backfill skipped — rollups current through {Watermark:o}.", plan.Watermark);
|
||
return;
|
||
}
|
||
|
||
// enqueue ≤24h slices over [floor, toHourUtc) oldest-first (Task 3's builder),
|
||
// lower _backfillInFlight, then self-arm BackfillTick to dequeue the first slice.
|
||
```
|
||
|
||
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests --filter KpiHistoryRecorderActorTests` — PASS.
|
||
5. Doc: rewrite `Component-KpiHistory.md:93` — the backfill folds the window in **≤24h slices oldest-first**, yielding to periodic folds between slices; on a failover restart it consults the `GetLatestRollupHourAsync` watermark and **skips** (rollups current) or **shrinks to the un-rolled tail**. Drop the "cheap and safe to re-run" claim — replace with "safe (idempotent) to re-run; the watermark fast-path makes the failover re-run cheap as well" (the review's exact complaint: the "safe" was true, the "cheap" was not).
|
||
6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.KpiHistory/KpiHistoryRecorderActor.cs docs/requirements/Component-KpiHistory.md tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests/KpiHistoryRecorderActorTests.cs && git commit -m "fix(kpi): backfill skips or shrinks to the rollup watermark on failover restarts (plan R2-04 T4)"`
|
||
|
||
### Task 5: Per-metric reduction in the bucketer — sum-per-bucket for Rate series (R3, part 1)
|
||
|
||
**Classification:** standard
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** 1, 2, 3, 4, 8, 10, 11, 12
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Types/Kpi/KpiSeriesBucketer.cs`
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Kpi/KpiSeriesBucketerTests.cs` (extend)
|
||
|
||
1. Failing tests:
|
||
|
||
```csharp
|
||
[Fact]
|
||
public void Bucket_RateSeries_SumsPerBucket_InsteadOfLastValue()
|
||
{
|
||
// 6 points of value 10 across a window bucketed to 2 → each bucket = 30 (sum),
|
||
// not 10 (last value). Today: FAIL (last-value keeps 1 of 3 deltas per bucket —
|
||
// the exact "keep one delta, discard the others" fold error the catalog warns
|
||
// about, re-introduced one layer up: arch-review 04 round 2, R3).
|
||
}
|
||
|
||
[Fact]
|
||
public void Bucket_RateSeries_ShortSeries_StillSumsClusteredPoints()
|
||
{
|
||
// 3 points (< maxPoints) with two sharing a bucket → the shared bucket is their SUM.
|
||
// Rate series skip the raw.Count <= maxPoints early return so totals stay truthful.
|
||
}
|
||
|
||
[Fact]
|
||
public void Bucket_RateSeries_PreservesSeriesTotal()
|
||
{
|
||
// Strong invariant: sum(output values) == sum(in-window input values) for Rate.
|
||
}
|
||
|
||
[Fact]
|
||
public void Bucket_DefaultAggregation_IsGauge_AndBehaviorUnchanged()
|
||
{
|
||
// Calling the existing 4-arg shape produces byte-identical output to before.
|
||
}
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter KpiSeriesBucketerTests` — expect FAIL.
|
||
3. Implementation — additive optional parameter (every existing call site compiles unchanged):
|
||
|
||
```csharp
|
||
public static IReadOnlyList<KpiSeriesPoint> Bucket(
|
||
IReadOnlyList<KpiSeriesPoint> raw,
|
||
DateTime fromUtc,
|
||
DateTime toUtc,
|
||
int maxPoints,
|
||
KpiRollupAggregation aggregation = KpiRollupAggregation.Gauge)
|
||
```
|
||
|
||
- Keep the argument guards and the null/empty return unchanged. Restrict the `raw.Count <= maxPoints` early return to **Gauge only**: for Rate, two deltas landing in one bucket must sum even in a short series (for well-spaced short series the per-point buckets make sum == identity, so output ≈ input anyway).
|
||
- In the loop, branch per aggregation: Gauge keeps the existing last-value candidate logic verbatim; Rate accumulates `sums[bucketIndex] += point.Value` (plus the same `occupied` flag), emitting `new KpiSeriesPoint(bucketStart, sums[i])` in the collection pass.
|
||
- Update the class/method XML doc: "last-value-per-bucket / gauge semantics" becomes per-metric — Gauge = last value in the bucket (unchanged default), Rate = **sum-per-bucket** (each raw point of a Rate series is a per-interval delta, so the bucket's true total is the sum of its deltas; both the raw-minute and hourly-rollup paths then chart the same unit, "events per chart bucket", eliminating the ~60× jump at the routing boundary — R3).
|
||
4. Run the filter — PASS. Also `dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter "FullyQualifiedName~Kpi"` — PASS (catalog + point tests untouched).
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Commons/Types/Kpi/KpiSeriesBucketer.cs tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Kpi/KpiSeriesBucketerTests.cs && git commit -m "fix(kpi): per-metric bucket reduction — Rate series sum per bucket instead of keeping one delta (plan R2-04 T5)"`
|
||
|
||
### Task 6: Catalog-driven aggregation at the query-service boundary (R3, part 2)
|
||
|
||
**Classification:** standard
|
||
**Estimated implement time:** ~4 min
|
||
**Parallelizable with:** 1, 3, 4, 8, 10, 11, 12 (requires Task 5)
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/KpiHistoryQueryService.cs` (`GetSeriesAsync` :73-94, `FetchSeriesAsync` doc :96-106)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Services/KpiHistoryQueryServiceTests.cs` (extend)
|
||
|
||
1. Failing tests (stub-repository rig already in the file):
|
||
|
||
```csharp
|
||
[Fact]
|
||
public async Task GetSeriesAsync_RateMetric_SumsPerBucket_OnRollupPath()
|
||
{
|
||
// (KpiSources.SiteCallAudit, KpiMetrics.SiteCallAudit.FailedLastInterval) is a
|
||
// catalog Rate pair. Stub GetHourlySeriesAsync with 400 hourly points of value 1
|
||
// over a >168h window, maxPoints 200 → every output value is 2.0 (sum of the two
|
||
// hourly sums per bucket), not 1.0 (last value). Today: FAIL.
|
||
}
|
||
|
||
[Fact]
|
||
public async Task GetSeriesAsync_RateMetric_PreservesTotal_AcrossRoutingBoundary()
|
||
{
|
||
// Same total event count served as per-minute deltas (raw path, window == threshold)
|
||
// and as hourly sums (rollup path, window just over threshold): the summed chart
|
||
// total must be equal on both paths — the boundary is now visually seamless (R3).
|
||
}
|
||
|
||
[Fact]
|
||
public async Task GetSeriesAsync_GaugeMetric_KeepsLastValuePerBucket()
|
||
{
|
||
// queueDepth (uncatalogued → Gauge) keeps the existing last-value output unchanged.
|
||
}
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests --filter KpiHistoryQueryServiceTests` — expect FAIL.
|
||
3. Implementation in `GetSeriesAsync` (both ctor paths):
|
||
|
||
```csharp
|
||
// Per-metric reduction intent: the same catalog that drives the hourly fold drives
|
||
// the chart-time bucket reduction, so a Rate series is summed per bucket on BOTH the
|
||
// raw-minute and hourly-rollup routes — one unit ("events per chart bucket") on both
|
||
// sides of the RollupThresholdHours routing boundary (arch-review 04 round 2, R3).
|
||
var aggregation = KpiMetricAggregationCatalog.Resolve(source, metric);
|
||
...
|
||
return KpiSeriesBucketer.Bucket(injectedSeries, fromUtc, toUtc, effectiveMax, aggregation);
|
||
...
|
||
return KpiSeriesBucketer.Bucket(series, fromUtc, toUtc, effectiveMax, aggregation);
|
||
```
|
||
|
||
Extend the `FetchSeriesAsync` doc comment: the routing boundary no longer changes a Rate chart's magnitude — both paths reduce to per-bucket sums (a residual ≤~1.2× native-resolution difference exists only for an unbucketed just-over-threshold rollup series vs. an at-threshold bucketed raw series; documented in Task 7).
|
||
4. Run the filter — PASS (the pre-existing routing/forwarding tests must stay green — Gauge default preserves their expectations).
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/KpiHistoryQueryService.cs tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Services/KpiHistoryQueryServiceTests.cs && git commit -m "fix(kpi): query service applies catalog aggregation at the bucketer — rate charts keep one unit across the raw/rollup boundary (plan R2-04 T6)"`
|
||
|
||
### Task 7: Truthful trend presentation — chart doc + Component-KpiHistory.md (R3, part 3)
|
||
|
||
**Classification:** small
|
||
**Estimated implement time:** ~3 min
|
||
**Parallelizable with:** 1, 2, 3, 8, 10, 11, 12 (requires Task 6; NOT 4/9 — same design-doc file)
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Shared/KpiTrendChart.razor.cs` (XML docs on `Points` :~55 and `Unit` :~62 — doc-only, no markup/behavior change; the component stays custom-SVG, no third-party libs)
|
||
- Modify: `docs/requirements/Component-KpiHistory.md` (:151 "Bucketed query", the "Raw-vs-rollup range routing" section :~153-155, and one line under ":157 Aggregation intent")
|
||
- Modify: trend-panel `Unit` strings only where they now misstate the semantics — locate with `grep -rn "Unit=" src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages` and fix any Rate-metric panel whose unit text claims a per-minute/per-interval reading (e.g. the `deliveredLastInterval` / `failedLastInterval` panels) to say per-bucket totals (e.g. `Unit="delivered/bucket"`); leave Gauge panels untouched
|
||
|
||
1. `KpiTrendChart.razor.cs` doc: on `Points`, note that Rate-classified series (per `KpiMetricAggregationCatalog`) now carry **per-bucket totals** (sum reduction), while Gauge series carry last-value-per-bucket readings; on `Unit`, note the supplied text must state the per-bucket semantics for Rate metrics.
|
||
2. `Component-KpiHistory.md:151`: replace "returns the **last value per bucket**" with the per-metric reduction — `KpiSeriesBucketer.Bucket(series, fromUtc, toUtc, maxPoints, aggregation)` keeps the last value for Gauge series and **sums per bucket** for Rate series, driven by the same `KpiMetricAggregationCatalog` the fold consults (the previous last-value-on-rollups behavior kept 1 of ~11 hourly sums per 90d bucket — the fold error the catalog's own doc warns about, fixed per arch-review 04 round 2, R3). In the range-routing section add: "Rate charts keep one unit (events per chart bucket) on both sides of the boundary; widening 7d→30d no longer changes the magnitude ~60×. Residual: an unbucketed just-over-threshold rollup series plots native per-hour sums vs. the at-threshold raw path's ~50-minute bucket sums (≤~1.2×) — accepted and documented, not hidden." Under Aggregation intent, note the catalog now has two consumers: the hourly fold and the chart-time bucket reduction.
|
||
3. Verify no stale claim remains: `grep -n "last value per bucket" docs/requirements/Component-KpiHistory.md` returns nothing.
|
||
4. Run the chart component suite to prove the doc-only touch broke nothing: `dotnet test tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests --filter KpiTrendChartTests` — PASS.
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.CentralUI docs/requirements/Component-KpiHistory.md && git commit -m "docs(kpi): trend charts state per-bucket-total semantics for rate metrics (plan R2-04 T7)"`
|
||
|
||
### Task 8: Close the catalog metric-literal drift hazard (R4)
|
||
|
||
**Classification:** small
|
||
**Estimated implement time:** ~4 min
|
||
**Parallelizable with:** 1, 2, 3, 4, 5, 10, 11, 12
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Types/Kpi/KpiMetrics.cs` (add three constants + amend the class remark)
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Types/Kpi/KpiMetricAggregation.cs` (:60-64 private literals; :98-107 `RatePairs`)
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/Kpi/SiteCallAuditKpiSampleSource.cs` (:45)
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/Kpi/SiteHealthKpiSampleSource.cs` (:43, :51)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Kpi/KpiMetricAggregationTests.cs` (extend)
|
||
|
||
1. Failing test (does not compile until the constants exist — the TDD gate for symbol promotion):
|
||
|
||
```csharp
|
||
[Fact]
|
||
public void PromotedMetricConstants_LockHistoricalPersistedValues()
|
||
{
|
||
// Persisted data contract — these are symbol promotions, NOT renames.
|
||
Assert.Equal("deliveredLastInterval", KpiMetrics.SiteCallAudit.DeliveredLastInterval);
|
||
Assert.Equal("alarmEvalErrors", KpiMetrics.SiteHealth.AlarmEvalErrors);
|
||
Assert.Equal("eventLogWriteFailures", KpiMetrics.SiteHealth.EventLogWriteFailures);
|
||
}
|
||
|
||
[Theory]
|
||
[InlineData(KpiSources.SiteCallAudit, KpiMetrics.SiteCallAudit.DeliveredLastInterval)]
|
||
[InlineData(KpiSources.SiteHealth, KpiMetrics.SiteHealth.AlarmEvalErrors)]
|
||
[InlineData(KpiSources.SiteHealth, KpiMetrics.SiteHealth.EventLogWriteFailures)]
|
||
public void Resolve_PromotedRatePairs_StillClassifyAsRate(string source, string metric)
|
||
=> Assert.Equal(KpiRollupAggregation.Rate, KpiMetricAggregationCatalog.Resolve(source, metric));
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter KpiMetricAggregationTests` — expect FAIL (compile error: constants absent).
|
||
3. Implementation — compile-time coupling in the only dependency-legal direction (Commons cannot reference the source projects, so the shared constant lives in `KpiMetrics` and both the catalog and the emitting source consume it):
|
||
- `KpiMetrics.SiteCallAudit`: add `public const string DeliveredLastInterval = "deliveredLastInterval";`; `KpiMetrics.SiteHealth`: add `AlarmEvalErrors = "alarmEvalErrors"` and `EventLogWriteFailures = "eventLogWriteFailures"` (EXACT existing string values — persisted, renames forbidden).
|
||
- Amend the `KpiMetrics` class remark (:20-27): the catalog now also carries metrics the **rollup aggregation catalog** classifies, not only charted ones — the C4/R4 lesson is that any metric named in two places needs one shared symbol.
|
||
- `KpiMetricAggregationCatalog`: delete the three private literals (:62-64) and reference the constants in `RatePairs`; update the "otherwise the source's own private literal" remark (:46-48) — every catalogued name is now a shared `KpiMetrics` constant, so an emitter rename is a compile error, not a silent Gauge downgrade.
|
||
- The two sources keep their private-const style with the value redefined in terms of the shared symbol (tiny diff, zero call-site churn): `private const string MetricDeliveredLastInterval = KpiMetrics.SiteCallAudit.DeliveredLastInterval;` (SiteCallAudit :45) and likewise for the two SiteHealth consts (:43, :51).
|
||
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter KpiMetricAggregationTests && dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests --filter SiteCallAuditKpiSampleSource && dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests --filter SiteHealthKpiSampleSource` — PASS (emitted strings unchanged).
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Commons src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/Kpi/SiteCallAuditKpiSampleSource.cs src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/Kpi/SiteHealthKpiSampleSource.cs tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Kpi/KpiMetricAggregationTests.cs && git commit -m "fix(kpi): promote catalog metric literals to shared KpiMetrics constants — drift becomes a compile error (plan R2-04 T8)"`
|
||
|
||
### Task 9: Classify the failover fold-race failure grain (R5)
|
||
|
||
**Classification:** standard
|
||
**Estimated implement time:** ~4 min
|
||
**Parallelizable with:** 3, 5, 6, 8, 10, 11, 12 (NOT 1/2 — same repository file, do them first; NOT 4/7 — same design-doc file)
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs` (ctor + the fold's `SaveChangesAsync` at :167)
|
||
- Modify: `docs/requirements/Component-KpiHistory.md` (one sentence in the "Hourly rollup tick" paragraph, :~91)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/KpiHistoryRepositoryTests.cs` (extend)
|
||
|
||
1. Failing test — force a REAL unique violation deterministically with a `SaveChangesInterceptor` that, on the fold's first save, inserts the conflicting `(series, hour)` rollup row through a separate command on the same in-memory SQLite connection (simulating the old singleton incarnation's in-flight fold winning the race):
|
||
|
||
```csharp
|
||
[Fact]
|
||
public async Task Fold_LosingFailoverUpsertRace_DoesNotThrow_AndNextFoldSelfHeals()
|
||
{
|
||
// ctx built with .AddInterceptors(new InsertConflictingRollupOnFirstSave(connection))
|
||
// seed one sample in a complete hour; FoldHourlyRollupsAsync must NOT propagate
|
||
// DbUpdateException (today: FAIL — the whole pass faults);
|
||
// a second FoldHourlyRollupsAsync over the same window must land the row (self-heal).
|
||
}
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter Fold_LosingFailoverUpsertRace` — expect FAIL.
|
||
3. Implementation:
|
||
- Ctor gains `ILogger<KpiHistoryRepository>? logger = null` defaulting to `NullLogger` (mirrors `SiteCallAuditRepository` — MS.DI resolves `ILogger<>` automatically, no registration churn).
|
||
- Wrap the fold's final `SaveChangesAsync`:
|
||
|
||
```csharp
|
||
try
|
||
{
|
||
await _context.SaveChangesAsync(cancellationToken);
|
||
}
|
||
catch (DbUpdateException ex)
|
||
{
|
||
// Failover-overlap race: the old singleton incarnation's in-flight fold and the
|
||
// new node's first fold can both Add the same (series, hour) row; the unique
|
||
// IX_KpiRollupHourly_Series then faults the loser's ENTIRE SaveChanges — the
|
||
// failure grain is the PASS, not the row (arch-review 04 round 2, R5). The fold
|
||
// only ever writes KpiRollupHourly rows, so any save fault here is a lost upsert
|
||
// race or a transient the next idempotent re-fold repairs identically; classify
|
||
// at Information instead of surfacing a scary error for a self-healing no-op.
|
||
_logger.LogInformation(ex,
|
||
"KPI rollup fold lost a failover-overlap upsert race; pass discarded, next fold self-heals.");
|
||
}
|
||
```
|
||
|
||
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter KpiHistoryRepositoryTests` — PASS.
|
||
5. Doc: add to the rollup-tick paragraph of `Component-KpiHistory.md`: "During a failover overlap the loser of the `(series, hour)` upsert race discards its whole fold pass (single `SaveChanges`) — logged at Information, repaired by the next idempotent fold; when reading fold logs after a failover, the failure grain is the pass, not the row."
|
||
6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs docs/requirements/Component-KpiHistory.md tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/KpiHistoryRepositoryTests.cs && git commit -m "fix(kpi): classify the failover fold-race pass loss as a self-healing Information event (plan R2-04 T9)"`
|
||
|
||
### Task 10: SiteCalls filtered terminal index (migration) (R6, part 1)
|
||
|
||
**Classification:** high-risk
|
||
**Estimated implement time:** ~4 min
|
||
**Parallelizable with:** 1, 2, 3, 4, 5, 6, 7, 8, 9, 12 (the plan's only migration — but serialize against any OTHER plan adding ConfigurationDatabase migrations: model-snapshot ordering)
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/SiteCallEntityTypeConfiguration.cs` (after the `IX_SiteCalls_NonTerminal` block ending :~99)
|
||
- Create: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/<timestamp>_AddSiteCallsTerminalIndex.cs` (scaffolded)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/SchemaConfigurationTests.cs` (extend)
|
||
|
||
1. Failing model test (SchemaConfigurationTests already asserts index shapes):
|
||
|
||
```csharp
|
||
[Fact]
|
||
public void SiteCalls_Has_Filtered_Terminal_Index()
|
||
{
|
||
var entity = Model.FindEntityType(typeof(SiteCall))!;
|
||
var index = entity.GetIndexes().Single(i => i.GetDatabaseName() == "IX_SiteCalls_Terminal");
|
||
Assert.Equal(new[] { nameof(SiteCall.TerminalAtUtc) }, index.Properties.Select(p => p.Name).ToArray());
|
||
Assert.Equal("[TerminalAtUtc] IS NOT NULL", index.GetFilter());
|
||
}
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter SiteCalls_Has_Filtered_Terminal_Index` — expect FAIL.
|
||
3. Add to `SiteCallEntityTypeConfiguration.Configure`:
|
||
|
||
```csharp
|
||
// Terminal backs the daily retention purge's predicate (TerminalAtUtc IS NOT NULL
|
||
// AND TerminalAtUtc < cutoff) and Task 11's MIN() slice anchor: filtered to the
|
||
// terminal population so the purge seeks the expired tail instead of full-scanning
|
||
// the 365-day table (arch-review 04 round 2, R6). Complements IX_SiteCalls_NonTerminal,
|
||
// which is filtered to IS NULL and unusable for this predicate.
|
||
builder.HasIndex(s => s.TerminalAtUtc)
|
||
.HasDatabaseName("IX_SiteCalls_Terminal")
|
||
.HasFilter("[TerminalAtUtc] IS NOT NULL");
|
||
```
|
||
|
||
4. Scaffold the migration — **repo gotcha: BUILD FIRST, never `dotnet ef migrations add --no-build`** (it scaffolds an EMPTY migration off the stale previously-built DLL; if that happens, delete the empty `Migrations/` files — `migrations remove` needs a live DB):
|
||
|
||
```bash
|
||
dotnet build ZB.MOM.WW.ScadaBridge.slnx
|
||
cd src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase
|
||
dotnet ef migrations add AddSiteCallsTerminalIndex
|
||
```
|
||
|
||
Verify the scaffold contains the `CreateIndex` with the `IS NOT NULL` filter — if it is empty, the gotcha bit; delete and redo.
|
||
5. Run the model test — PASS. With infra up (`cd infra && docker compose up -d`), run the MSSQL-backed suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter SiteCallAuditRepositoryTests` — PASS. Production script (repo convention — manual SQL for production): `dotnet ef migrations script AddKpiRollupHourlyTable AddSiteCallsTerminalIndex --idempotent --output ../../docs/plans/sql/AddSiteCallsTerminalIndex.sql` (verify the from-migration is still the latest applied; review the output).
|
||
6. Commit: `git add -A src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/SchemaConfigurationTests.cs docs/plans/sql && git commit -m "perf(sitecallaudit): filtered IX_SiteCalls_Terminal index backs the retention purge predicate (plan R2-04 T10)"`
|
||
|
||
### Task 11: Time-sliced SiteCalls terminal purge (R6, part 2)
|
||
|
||
**Classification:** standard
|
||
**Estimated implement time:** ~4 min
|
||
**Parallelizable with:** 1, 2, 3, 4, 5, 6, 7, 8, 9, 12 (requires Task 10 — the MIN() anchor and DELETE predicate seek its index)
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs` (`PurgeTerminalAsync` :247-252)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/SiteCallAuditRepositoryTests.cs` (extend; MSSQL `SkippableFact` — infra up)
|
||
|
||
1. Failing test (pattern-match the fixture usage of the existing `SkippableFact` tests in the file; assert slicing via an EF `DbCommandInterceptor` capturing DELETE statements, or SQL capture matching how sibling tests inspect SQL):
|
||
|
||
```csharp
|
||
[SkippableFact]
|
||
public async Task PurgeTerminal_SlicesMultiDayBacklog_IntoBoundedDeletes()
|
||
{
|
||
// Seed terminal rows spread across 3 distinct days beyond the cutoff, one terminal
|
||
// row inside the window, and one OLD non-terminal row.
|
||
// Purge → returns 3-day total; the fresh terminal row and the old NON-terminal row
|
||
// survive (non-terminal rows are NEVER purged on age — sites remain source of truth);
|
||
// captured SQL shows MORE THAN ONE DELETE statement for the multi-day backlog.
|
||
// Today: FAIL — a single unbatched DELETE is issued.
|
||
}
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter PurgeTerminal_SlicesMultiDayBacklog` — expect FAIL.
|
||
3. Implement time-sliced batching mirroring `KpiHistoryRepository.PurgeOlderThanAsync` (the round-1 Task 19 shape), keeping the raw-T-SQL MSSQL-only style this repository already uses:
|
||
|
||
```csharp
|
||
public async Task<int> PurgeTerminalAsync(DateTime olderThanUtc, CancellationToken ct = default)
|
||
{
|
||
// Time-sliced batches (arch-review 04 round 2, R6 — the one maintenance DELETE
|
||
// that missed round 1's batching pass): each DELETE covers at most one DAY of
|
||
// terminal rows, capping the lock/log footprint per statement. Steady state
|
||
// (daily purge, 365-day retention) is a single slice; only catch-up after an
|
||
// outage runs several. One-day (not one-hour) slices are proportionate to
|
||
// SiteCalls volume, which is far below KpiSample's. The MIN() anchor and the
|
||
// DELETE predicate both seek IX_SiteCalls_Terminal (filtered IS NOT NULL).
|
||
var total = 0;
|
||
var floor = await _context.SiteCalls
|
||
.Where(s => s.TerminalAtUtc != null && s.TerminalAtUtc < olderThanUtc)
|
||
.MinAsync(s => s.TerminalAtUtc, ct);
|
||
while (floor is not null && floor < olderThanUtc)
|
||
{
|
||
var ceiling = floor.Value.AddDays(1) < olderThanUtc ? floor.Value.AddDays(1) : olderThanUtc;
|
||
total += await _context.Database.ExecuteSqlInterpolatedAsync(
|
||
$"DELETE FROM dbo.SiteCalls WHERE TerminalAtUtc IS NOT NULL AND TerminalAtUtc < {ceiling};",
|
||
ct);
|
||
floor = await _context.SiteCalls
|
||
.Where(s => s.TerminalAtUtc != null && s.TerminalAtUtc < olderThanUtc)
|
||
.MinAsync(s => s.TerminalAtUtc, ct);
|
||
}
|
||
|
||
return total;
|
||
}
|
||
```
|
||
|
||
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter SiteCallAuditRepositoryTests` — PASS (upsert/query/KPI tests untouched). Then the purge-actor contract: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests --filter SiteCallAuditPurgeTests` — PASS (return-total contract unchanged).
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/SiteCallAuditRepositoryTests.cs && git commit -m "fix(sitecallaudit): time-sliced terminal purge replaces the single year-scale DELETE (plan R2-04 T11)"`
|
||
|
||
### Task 12: Retention-service shutdown no longer surfaces cancellation (R7)
|
||
|
||
**Classification:** small
|
||
**Estimated implement time:** ~3 min
|
||
**Parallelizable with:** 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SiteAuditRetentionService.cs` (`RunLoopAsync` :62-91)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SiteAuditRetentionServiceTests.cs` (extend; the `RecordingSiteAuditQueue` stub gains a block-until-cancelled mode)
|
||
|
||
1. Failing test:
|
||
|
||
```csharp
|
||
[Fact]
|
||
public async Task StopAsync_MidPurge_CompletesCleanly_WithoutSurfacingCancellation()
|
||
{
|
||
var queue = new RecordingSiteAuditQueue { BlockUntilCancelled = true }; // PurgeExpiredAsync: await Task.Delay(Timeout.Infinite, ct)
|
||
var options = Options.Create(new SiteAuditRetentionOptions
|
||
{ RetentionDays = 7, PurgeInterval = TimeSpan.FromHours(24), InitialDelay = TimeSpan.Zero });
|
||
using var svc = new SiteAuditRetentionService(queue, options, NullLogger<SiteAuditRetentionService>.Instance);
|
||
await svc.StartAsync(CancellationToken.None);
|
||
await WaitUntilAsync(() => queue.PurgeCalls.Count >= 1, TimeSpan.FromSeconds(5));
|
||
|
||
// Today: FAIL — the OCE SafePurgeAsync deliberately rethrows escapes RunLoopAsync
|
||
// (the two awaits at :76/:89 sit outside any try/catch), cancels _loop, and
|
||
// StopAsync returns that canceled task straight to the host (:122-126), whose
|
||
// await throws TaskCanceledException into every shutdown log (R7).
|
||
await svc.StopAsync(CancellationToken.None); // must NOT throw
|
||
}
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter StopAsync_MidPurge` — expect FAIL (`TaskCanceledException`).
|
||
3. Implementation — catch OCE at the loop boundary (the report's preferred alignment with the sibling `SiteAuditBacklogReporter` pattern), leaving `SafePurgeAsync`'s deliberate rethrow (:106-110) untouched so a mid-purge shutdown still aborts the purge promptly:
|
||
|
||
```csharp
|
||
private async Task RunLoopAsync(CancellationToken ct)
|
||
{
|
||
try
|
||
{
|
||
// ... existing body verbatim (initial delay, first purge, delay/purge loop) ...
|
||
}
|
||
catch (OperationCanceledException)
|
||
{
|
||
// Shutdown landed mid-purge: SafePurgeAsync rethrows OCE by design so the purge
|
||
// aborts promptly, but the loop task must complete CLEANLY — StopAsync hands
|
||
// _loop straight to the host, and a canceled task there is shutdown-log noise
|
||
// (arch-review 04 round 2, R7). Cancellation here IS the clean exit.
|
||
}
|
||
}
|
||
```
|
||
|
||
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter SiteAuditRetentionServiceTests` — PASS (the two existing tick tests must stay green).
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SiteAuditRetentionService.cs tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SiteAuditRetentionServiceTests.cs && git commit -m "fix(audit-log): retention loop absorbs shutdown cancellation instead of faulting host StopAsync (plan R2-04 T12)"`
|
||
|
||
---
|
||
|
||
## Dependencies on other plans
|
||
|
||
- None of the round-1 plans have open tasks (the whole arch-review initiative closed 191/192 on 2026-07-10); no cross-plan file contention is expected.
|
||
- Task 10 is this plan's **only EF migration**. No other plan currently adds ConfigurationDatabase migrations, but if one appears, serialize on the model snapshot (same rule as round-1 Tasks 6→8).
|
||
- The `IKpiHistoryRepository` addition (Task 2) is additive; any concurrently developed fake/stub implementing the interface must add the member — build the solution after Task 2 to surface stragglers immediately.
|
||
|
||
## Execution order
|
||
|
||
**P0 (the High + its enablers, in order):** 1 → 2 → 3 → 4 (fold fetch, watermark seam, sliced backfill, failover fast-path).
|
||
**P1:** 5 → 6 → 7 (rate presentation chain), 10 → 11 (SiteCalls purge chain).
|
||
**P2 (low-severity cleanup):** 8, 9, 12 — broadly parallel per each task's "Parallelizable with" contract (watch the two serialization lanes: `KpiHistoryRepository.cs` = 1 → 2 → 9; `Component-KpiHistory.md` = 4 → 7 → 9 in whatever order they run, never concurrently).
|
||
Finish with `dotnet build ZB.MOM.WW.ScadaBridge.slnx` plus the targeted suites this plan touched — `dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter "KpiHistoryRepositoryTests|SiteCallAuditRepositoryTests|SchemaConfigurationTests"` (infra up for the MSSQL `SkippableFact`s), `dotnet test tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests`, `dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter "FullyQualifiedName~Kpi"`, `dotnet test tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests --filter "KpiHistoryQueryServiceTests|KpiTrendChartTests"`, `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests`, `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests --filter SiteHealthKpiSampleSource`, and `dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter "FullyQualifiedName~Site"` — before declaring the plan done.
|
||
|
||
## Findings Coverage
|
||
|
||
| # | Report finding (severity) | Task(s) |
|
||
|---|---------------------------|---------|
|
||
| R1 | Startup backfill folds the entire raw-retention window in one unbounded pass, re-run on every failover (High) | 2, 3, 4 |
|
||
| R2 | Fold fetch is change-tracked — `AsNoTracking`/projection missing (Medium) | 1 |
|
||
| R3 | Rate metrics change units ~60× across the raw/rollup routing boundary; bucketer re-introduces the last-value fold error on hourly Rate sums (Medium) | 5, 6, 7 |
|
||
| R4 | `KpiMetricAggregationCatalog` re-creates the private-metric-literal drift hazard C4 fixed (Low) | 8 |
|
||
| R5 | Overlapping-singleton fold race faults an entire fold batch — failure grain is the pass, not the row (Low) | 9 |
|
||
| R6 | `PurgeTerminalAsync` on SiteCalls is the one remaining unbatched, unindexed year-scale maintenance DELETE (Medium) | 10, 11 |
|
||
| R7 | `SiteAuditRetentionService.StopAsync` surfaces the loop's cancellation to the host (Low) | 12 |
|
||
| S8 | Outbox repository dual SQLite/T-SQL dialect | No action — accepted round-1 deferral; round-2 re-verified the documented convention (`Component-ConfigurationDatabase.md:69`, "do not copy") |
|
||
| S10 | `SqliteAuditWriter.Dispose` sync-over-async | No action — accepted round-1 won't-fix; round-2 re-verified |
|
||
| P3 | AuditLog clustered key leads with random GUID | No action — accepted round-1 deferral, tracked benchmark follow-up (`Component-ConfigurationDatabase.md:70`) |
|
||
| P4 (cadence) | Per-node KPI sampling cadence | No action — accepted round-1 deferral, tracked (`Component-ConfigurationDatabase.md:71`); the purge half shipped in round 1 |
|
||
| P6 (paging) | Outbox offset→keyset page conversion | No action — accepted round-1 deferral, tracked (`Component-ConfigurationDatabase.md:71`); the KPI half shipped in round 1 |
|
||
| C2 | `AuditLogRow` lives in ConfigurationDatabase, not Commons | No action — accepted round-1 deviation, documented (`Component-Commons.md:51`, "do not lift it into Commons") |
|
||
| C3 | Mixed timestamp CLR types | No action — accepted round-1 convention note (`Component-ConfigurationDatabase.md:68`) |
|
||
| U3 | Hash-chain tamper evidence / Parquet archival | No action — unchanged v1.x deferral; round-2 confirms no drift |
|
||
| U5 (residue) | AuditLog `PullAuditEvents` keyset upgrade | No action — accepted round-1 deferral, tracked (`Component-ConfigurationDatabase.md:71`); idempotent on `EventId`, lower urgency; round-2 lists it only so the tracker entry doesn't age out |
|