Re-ran all 8 domain reviews at HEAD8c888f13against theb910f5ebbaseline: every round-1 finding source-verified (168 fixed, 0 regressions, 0 false claims); 56 new findings (1 Critical / 4 High / 15 Medium / 36 Low), concentrated in post-baseline code (anti-entropy resync, KPI rollup backfill, live alarm stream) and seams the fixes exposed. Headliners: S&F resync predicate inversion can wipe the delivering node's buffer (02-N1 Critical); resync snapshot exceeds the Akka remoting frame size (02-N2); failover drill kills the one node keep-oldest can't survive (01-N1); unbounded rollup backfill per failover (04-R1); live production API key in untracked test.txt (08-NF1). Adds PLAN-R2-01..08 + .tasks.json manifests and the Round-2 board, P0 list, cross-plan mutexes, and wave order in 00-MASTER-TRACKER.
49 KiB
PLAN-R2-04 — Data & Audit Backbone Round-2 Fixes Implementation Plan
For Claude: REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
Goal: Close the seven NEW findings of the round-2 architecture review (archreview/04-data-audit-backbone.md, 2026-07-12): bound the KPI rollup backfill that currently re-folds the entire 90-day raw window in one tracked pass on every singleton failover (R1), make the fold fetch untracked and projected (R2), fix the ~60× rate-unit discontinuity at the raw/rollup routing boundary and the last-value-per-bucket re-fold error the aggregation catalog itself warns about (R3), close the catalog metric-literal drift hazard (R4), classify the failover fold-race failure grain (R5), give the SiteCalls terminal purge the same time-sliced batching + index support its two siblings received in round 1 (R6), and stop the site retention service from surfacing its own shutdown cancellation to the host (R7). Every accepted round-1 deferral stays deferred — coverage rows only, no code.
Architecture: All fixes stay inside the existing component boundaries. The rollup backfill/fold hardening lives in the KpiHistory recorder actor + KpiHistoryRepository (one additive IKpiHistoryRepository watermark method in Commons); the rate-presentation fix lives in the Commons KpiSeriesBucketer (additive optional parameter — existing call sites compile unchanged) driven by the already-shipped KpiMetricAggregationCatalog, wired at the KpiHistoryQueryService boundary; the SiteCalls purge fix rides one EF Core migration (a filtered TerminalAtUtc index) plus a repository-local time-sliced DELETE loop mirroring the round-1 KpiSample purge; the retention-service fix is a loop-boundary OCE catch. Design docs are updated in the same task as the code they describe (repo rule: doc + code + tests travel together). No new tables, no contract-breaking changes.
Tech Stack: C#/.NET 8, Akka.NET (actors/TestKit), EF Core + SQL Server (central; SQLite in-memory harness for the KpiHistory repository suite), xUnit. Build: dotnet build ZB.MOM.WW.ScadaBridge.slnx. Test per-project with targeted filters only. MSSQL-backed SkippableFact suites (SiteCallAuditRepositoryTests) need cd infra && docker compose up -d. EF migrations gotcha (repo-documented): always build first and run dotnet ef migrations add <Name> WITHOUT --no-build, or it scaffolds an empty migration off the stale DLL — delete the empty files if it happens (migrations remove needs a live DB).
Task 1: Untracked, projected fold fetch (R2)
Classification: standard Estimated implement time: ~3 min Parallelizable with: 3, 5, 6, 7, 8, 10, 11, 12 (NOT 2/9 — same repository file) Files:
- Modify:
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs(FoldHourlyRollupsAsyncfetch at :104-106; group lambdas :112-128) - Test:
tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/KpiHistoryRepositoryTests.cs(extend)
- Write the failing test (SQLite in-memory harness — pattern-match
NewContext()/Sample()already in the file; clear the tracker after seeding so the assertion sees only the fold's own work):
[Fact]
public async Task FoldHourlyRollups_DoesNotTrack_KpiSampleEntities()
{
await using var ctx = NewContext();
var repo = new KpiHistoryRepository(ctx);
await repo.RecordSamplesAsync(new[]
{
Sample("NotificationOutbox", "queueDepth", "Global", null, 5, Base.AddMinutes(10)),
Sample("NotificationOutbox", "queueDepth", "Global", null, 7, Base.AddMinutes(50)),
});
ctx.ChangeTracker.Clear(); // isolate the fold's tracking behavior from the seed
await repo.FoldHourlyRollupsAsync(Base, Base.AddHours(1));
// The fold READS samples and WRITES rollups; the read must not register
// thousands of read-only KpiSample entries for DetectChanges to re-scan
// (arch-review 04 round 2, R2).
Assert.Empty(ctx.ChangeTracker.Entries<KpiSample>());
}
- Run:
dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter FoldHourlyRollups_DoesNotTrack_KpiSampleEntities— expect FAIL (the fetch is a trackedToListAsynctoday). - Replace the fetch at :104-106 with a projection (a
Select()projection is never tracked, and it also drops the unneededIdcolumn, roughly halving per-row memory — which shrinks the R1 backfill's footprint even before Task 3 slices it):
// Projected, untracked fetch: the fold only reads these six fields and never
// mutates a KpiSample. A tracked ToListAsync registered ~5k-90k read-only
// entities in the change tracker per healthy 3h fold — all re-scanned by
// DetectChanges on the final SaveChanges (arch-review 04 round 2, R2). A
// projection is inherently untracked and materializes no entity at all.
var samples = await _context.KpiSamples
.Where(s => s.CapturedAtUtc >= from && s.CapturedAtUtc < to)
.Select(s => new FoldSample(s.Source, s.Metric, s.Scope, s.ScopeKey, s.CapturedAtUtc, s.Value))
.ToListAsync(cancellationToken);
with, at the bottom of the class next to SeriesHourKey:
/// <summary>Narrow, untracked projection of one KpiSample row for the fold (R2).</summary>
private readonly record struct FoldSample(
string Source, string Metric, string Scope, string? ScopeKey, DateTime CapturedAtUtc, double Value);
The grouping/aggregation lambdas (:112-128) compile unchanged — FoldSample carries the same member names.
4. Run the new filter — PASS. Then run the whole fold suite: dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter KpiHistoryRepositoryTests — PASS (all existing fold/purge/query tests must stay green; this is a pure fetch-shape change).
5. Commit: git add src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/KpiHistoryRepositoryTests.cs && git commit -m "fix(kpi): untracked projected fold fetch — fold reads no longer flood the change tracker (plan R2-04 T1)"
Task 2: Rollup watermark seam — GetLatestRollupHourAsync (R1, part 1)
Classification: standard Estimated implement time: ~4 min Parallelizable with: 5, 8, 10, 11, 12 (NOT 1/9 — same repository file; NOT 3/4/6 — the interface addition ripples into their test fakes) Files:
- Modify:
src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IKpiHistoryRepository.cs(additive method) - Modify:
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs(implement) - Modify: every test fake implementing
IKpiHistoryRepository— locate withgrep -rln ": IKpiHistoryRepository" tests/(expect the recorder-actor fake intests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests/KpiHistoryRecorderActorTests.csand the query-service stub intests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Services/KpiHistoryQueryServiceTests.cs) — add a trivialTask.FromResult<DateTime?>(null)member so they compile - Test:
tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/KpiHistoryRepositoryTests.cs(extend)
- Failing tests:
[Fact]
public async Task GetLatestRollupHour_ReturnsNull_WhenNoRollupsExist()
{
await using var ctx = NewContext();
var repo = new KpiHistoryRepository(ctx);
Assert.Null(await repo.GetLatestRollupHourAsync());
}
[Fact]
public async Task GetLatestRollupHour_ReturnsNewestHourStart_AcrossAllSeries()
{
await using var ctx = NewContext();
var repo = new KpiHistoryRepository(ctx);
await repo.RecordSamplesAsync(new[]
{
Sample("NotificationOutbox", "queueDepth", "Global", null, 5, Base.AddMinutes(10)),
Sample("SiteCallAudit", "buffered", "Global", null, 2, Base.AddHours(3).AddMinutes(10)),
});
await repo.FoldHourlyRollupsAsync(Base, Base.AddHours(4));
Assert.Equal(Base.AddHours(3), await repo.GetLatestRollupHourAsync());
}
- Run:
dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter GetLatestRollupHour— expect FAIL (method does not exist / does not compile — comment the tests out to prove the baseline if needed, standard TDD-on-new-member flow). - Add to
IKpiHistoryRepository(afterGetHourlySeriesAsync, ~:108):
/// <summary>
/// Returns the newest <c>KpiRollupHourly.HourStartUtc</c> across all series, or
/// <c>null</c> when no rollups exist. The recorder's one-shot backfill consults
/// this watermark to skip (or shrink to) the un-rolled tail instead of re-folding
/// the whole raw-retention window on every failover (arch-review 04 round 2, R1).
/// </summary>
Task<DateTime?> GetLatestRollupHourAsync(CancellationToken cancellationToken = default);
Implement in KpiHistoryRepository: return await _context.KpiRollupHourly.MaxAsync(r => (DateTime?)r.HourStartUtc, cancellationToken); — a single MAX over the unique IX_KpiRollupHourly_Series population (rollup table is small: one row per series-hour). Add the null-returning member to each located test fake.
4. Run the filter — PASS. Build the solution (dotnet build ZB.MOM.WW.ScadaBridge.slnx) to prove no other implementor was missed.
5. Commit: git add src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IKpiHistoryRepository.cs src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs tests && git commit -m "feat(kpi): GetLatestRollupHourAsync watermark seam for the backfill fast-path (plan R2-04 T2)"
Task 3: Slice the rollup backfill into bounded day windows (R1, part 2)
Classification: high-risk Estimated implement time: ~5 min Parallelizable with: 5, 6, 7, 8, 10, 11, 12 (NOT 4 — same actor file; requires Task 2's fake update) Files:
- Modify:
src/ZB.MOM.WW.ScadaBridge.KpiHistory/KpiHistoryRecorderActor.cs(HandleBackfillTick:494-526,RunBackfillPass:536-558, theBackfillCompletereceive :163-167) - Test:
tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests/KpiHistoryRecorderActorTests.cs(extend + update the two existing backfill tests)
- Failing test (reuse the gated-fold fake-repository rig the existing
BackfillTick_FoldsFullRetentionWindow_Once/BackfillInFlight_SkipsPeriodicRollupTicktests drive by hand):
[Fact]
public void Backfill_SlicesRetentionWindow_IntoBoundedDayFolds_OldestFirst()
{
// RetentionDays = 3 → expect 3+ fold calls, each window ≤ 24 h, contiguous,
// oldest-first, whose union is exactly [TruncateToHour(now) − 3 d, TruncateToHour(now)).
// Today: FAIL — a single fold call spans the whole 3-day window.
}
[Fact]
public void PeriodicRollupTick_Interleaves_BetweenBackfillSlices()
{
// Gate slice 1 open; release it; before dequeuing slice 2 send a RollupTick —
// the periodic fold must claim the fold path (the backfill defers via its
// re-arm branch) and ALL backfill slices must still complete afterwards.
// Today: FAIL — _backfillInFlight blocks every periodic fold for the whole pass.
}
- Run:
dotnet test tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests --filter "Backfill_SlicesRetentionWindow|PeriodicRollupTick_Interleaves"— expect FAIL. - Implementation — convert the single full-window pass into a slice queue processed one bounded window per fold, releasing the fold path between slices:
- New constant + state:
/// <summary>
/// Width of one backfill fold slice (24 h). The repository fold materializes its whole
/// window in memory, so the one-shot backfill must never hand it the full raw-retention
/// window (90 d ≈ tens of millions of rows at fleet volume — arch-review 04 round 2, R1).
/// One day per fold bounds each pass to roughly what a steady-state day writes, and the
/// guard is lowered between slices so periodic folds interleave instead of stalling
/// behind the historical pass.
/// </summary>
private static readonly TimeSpan BackfillSliceWidth = TimeSpan.FromHours(24);
private readonly Queue<(DateTime FromHourUtc, DateTime ToHourUtc)> _backfillSlices = new();
HandleBackfillTick: keep the_backfillDone || _backfillInFlightand_rollupInFlight-defer branches verbatim. When_backfillSlicesis empty, build the plan — enumerate[toHourUtc − RetentionDays, toHourUtc)into ≤24h windows oldest-first and enqueue them (log the one "backfill starting" line with the slice count); when it is non-empty, dequeue ONE slice, raise_backfillInFlight, and runRunBackfillPass(slice.FromHourUtc, slice.ToHourUtc, ct)pipingBackfillSliceComplete.Instanceon both success and (logged) failure.- Replace the
BackfillCompletereceive with:
Receive<BackfillSliceComplete>(_ =>
{
_backfillInFlight = false; // release the fold path BETWEEN slices (R1)
if (_backfillSlices.Count == 0)
{
_backfillDone = true; // latch: at most once per actor lifetime (unchanged semantics)
_logger.LogInformation("KPI rollup backfill completed — all slices folded.");
return;
}
// Next slice via the timer, not inline: a RollupTick already queued in the mailbox
// is processed first, so periodic folds interleave; the existing _rollupInFlight
// deferral branch in HandleBackfillTick then re-arms the backfill behind it.
Timers.StartSingleTimer(BackfillTimerKey, BackfillTick.Instance, TimeSpan.Zero);
});
Rename the BackfillComplete message class to BackfillSliceComplete (internal — used only by this actor and its tests). Move the "backfill completed" wall-time log from RunBackfillPass (which now logs per-slice at Debug) to the queue-drained branch. Ordering note in the class doc: periodic folds and backfill slices strictly alternate through the single _rollupInFlight/_backfillInFlight gate pair — the two idempotent upserts still never race on overlapping rows.
- Update
BackfillTick_FoldsFullRetentionWindow_Once(assert the union of slice windows covers the retention window and a post-completionBackfillTickis a no-op) andBackfillInFlight_SkipsPeriodicRollupTick(unchanged semantics — a RollupTick arriving while a slice is in flight is still skipped).
- Run:
dotnet test tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests --filter KpiHistoryRecorderActorTests— PASS (all pre-existing sample/purge/rollup tests must stay green). - Commit:
git add src/ZB.MOM.WW.ScadaBridge.KpiHistory/KpiHistoryRecorderActor.cs tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests/KpiHistoryRecorderActorTests.cs && git commit -m "fix(kpi): slice the rollup backfill into bounded 24h folds that yield to periodic folds (plan R2-04 T3)"
Task 4: Backfill failover fast-path via the rollup watermark + doc correction (R1, part 3)
Classification: high-risk Estimated implement time: ~5 min Parallelizable with: 5, 6, 8, 10, 11, 12 (NOT 3 — same actor file, do 3 first; NOT 7/9 — same design-doc file) Files:
- Modify:
src/ZB.MOM.WW.ScadaBridge.KpiHistory/KpiHistoryRecorderActor.cs(plan branch ofHandleBackfillTickfrom Task 3) - Modify:
docs/requirements/Component-KpiHistory.md(:93, the "One-shot backfill" paragraph) - Test:
tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests/KpiHistoryRecorderActorTests.cs(extend)
- Failing tests (the fake repository's
GetLatestRollupHourAsync— added in Task 2 — gains a settable return value):
[Fact]
public void Backfill_Skips_WhenRollupsAlreadyCurrent()
{
// Fake watermark = TruncateToHour(now) − 1 h (i.e. within RollupLookbackHours of now —
// the failover case, which is the common one): expect ZERO fold calls, the done-latch
// set, and an Information log; the periodic fold alone covers the lookback tail.
}
[Fact]
public void Backfill_ShrinksToWatermark_InsteadOfRetentionFloor()
{
// RetentionDays = 90, fake watermark = 2 days ago: expect the first slice to start at
// the WATERMARK HOUR (inclusive — re-folding the newest already-folded hour is a cheap
// idempotent safety margin), not 90 days ago, and ≤ 3 slices total.
}
- Run:
dotnet test tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests --filter "Backfill_Skips_WhenRollupsAlreadyCurrent|Backfill_ShrinksToWatermark"— expect FAIL (the plan branch always enumerates from the retention floor). - Implementation — make the plan branch asynchronous: when
_backfillSlicesis empty, raise_backfillInFlightand run a plan pass that resolvesIKpiHistoryRepositoryin a fresh scope and returnsGetLatestRollupHourAsync()(never-faulting wrapper like the other passes), pipingBackfillPlan(DateTime? Watermark)back to Self. In theReceive<BackfillPlan>handler (actor thread):
var toHourUtc = TruncateToHour(DateTime.UtcNow);
var retentionFloor = toHourUtc - TimeSpan.FromDays(_options.RetentionDays);
// Shrink to the un-rolled tail; re-fold the watermark hour itself (idempotent) as a
// safety margin. A null watermark (fresh install) keeps the full retention window.
var floor = plan.Watermark is { } w && w > retentionFloor ? w : retentionFloor;
if (floor >= toHourUtc - TimeSpan.FromHours(_options.RollupLookbackHours))
{
// Failover fast-path: rollups are already current — the periodic fold's lookback
// window covers everything from the watermark forward, so the full historical
// pass is skipped entirely (arch-review 04 round 2, R1).
_backfillInFlight = false;
_backfillDone = true;
_logger.LogInformation(
"KPI rollup backfill skipped — rollups current through {Watermark:o}.", plan.Watermark);
return;
}
// enqueue ≤24h slices over [floor, toHourUtc) oldest-first (Task 3's builder),
// lower _backfillInFlight, then self-arm BackfillTick to dequeue the first slice.
- Run:
dotnet test tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests --filter KpiHistoryRecorderActorTests— PASS. - Doc: rewrite
Component-KpiHistory.md:93— the backfill folds the window in ≤24h slices oldest-first, yielding to periodic folds between slices; on a failover restart it consults theGetLatestRollupHourAsyncwatermark and skips (rollups current) or shrinks to the un-rolled tail. Drop the "cheap and safe to re-run" claim — replace with "safe (idempotent) to re-run; the watermark fast-path makes the failover re-run cheap as well" (the review's exact complaint: the "safe" was true, the "cheap" was not). - Commit:
git add src/ZB.MOM.WW.ScadaBridge.KpiHistory/KpiHistoryRecorderActor.cs docs/requirements/Component-KpiHistory.md tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests/KpiHistoryRecorderActorTests.cs && git commit -m "fix(kpi): backfill skips or shrinks to the rollup watermark on failover restarts (plan R2-04 T4)"
Task 5: Per-metric reduction in the bucketer — sum-per-bucket for Rate series (R3, part 1)
Classification: standard Estimated implement time: ~5 min Parallelizable with: 1, 2, 3, 4, 8, 10, 11, 12 Files:
- Modify:
src/ZB.MOM.WW.ScadaBridge.Commons/Types/Kpi/KpiSeriesBucketer.cs - Test:
tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Kpi/KpiSeriesBucketerTests.cs(extend)
- Failing tests:
[Fact]
public void Bucket_RateSeries_SumsPerBucket_InsteadOfLastValue()
{
// 6 points of value 10 across a window bucketed to 2 → each bucket = 30 (sum),
// not 10 (last value). Today: FAIL (last-value keeps 1 of 3 deltas per bucket —
// the exact "keep one delta, discard the others" fold error the catalog warns
// about, re-introduced one layer up: arch-review 04 round 2, R3).
}
[Fact]
public void Bucket_RateSeries_ShortSeries_StillSumsClusteredPoints()
{
// 3 points (< maxPoints) with two sharing a bucket → the shared bucket is their SUM.
// Rate series skip the raw.Count <= maxPoints early return so totals stay truthful.
}
[Fact]
public void Bucket_RateSeries_PreservesSeriesTotal()
{
// Strong invariant: sum(output values) == sum(in-window input values) for Rate.
}
[Fact]
public void Bucket_DefaultAggregation_IsGauge_AndBehaviorUnchanged()
{
// Calling the existing 4-arg shape produces byte-identical output to before.
}
- Run:
dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter KpiSeriesBucketerTests— expect FAIL. - Implementation — additive optional parameter (every existing call site compiles unchanged):
public static IReadOnlyList<KpiSeriesPoint> Bucket(
IReadOnlyList<KpiSeriesPoint> raw,
DateTime fromUtc,
DateTime toUtc,
int maxPoints,
KpiRollupAggregation aggregation = KpiRollupAggregation.Gauge)
- Keep the argument guards and the null/empty return unchanged. Restrict the
raw.Count <= maxPointsearly return to Gauge only: for Rate, two deltas landing in one bucket must sum even in a short series (for well-spaced short series the per-point buckets make sum == identity, so output ≈ input anyway). - In the loop, branch per aggregation: Gauge keeps the existing last-value candidate logic verbatim; Rate accumulates
sums[bucketIndex] += point.Value(plus the sameoccupiedflag), emittingnew KpiSeriesPoint(bucketStart, sums[i])in the collection pass. - Update the class/method XML doc: "last-value-per-bucket / gauge semantics" becomes per-metric — Gauge = last value in the bucket (unchanged default), Rate = sum-per-bucket (each raw point of a Rate series is a per-interval delta, so the bucket's true total is the sum of its deltas; both the raw-minute and hourly-rollup paths then chart the same unit, "events per chart bucket", eliminating the ~60× jump at the routing boundary — R3).
- Run the filter — PASS. Also
dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter "FullyQualifiedName~Kpi"— PASS (catalog + point tests untouched). - Commit:
git add src/ZB.MOM.WW.ScadaBridge.Commons/Types/Kpi/KpiSeriesBucketer.cs tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Kpi/KpiSeriesBucketerTests.cs && git commit -m "fix(kpi): per-metric bucket reduction — Rate series sum per bucket instead of keeping one delta (plan R2-04 T5)"
Task 6: Catalog-driven aggregation at the query-service boundary (R3, part 2)
Classification: standard Estimated implement time: ~4 min Parallelizable with: 1, 3, 4, 8, 10, 11, 12 (requires Task 5) Files:
- Modify:
src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/KpiHistoryQueryService.cs(GetSeriesAsync:73-94,FetchSeriesAsyncdoc :96-106) - Test:
tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Services/KpiHistoryQueryServiceTests.cs(extend)
- Failing tests (stub-repository rig already in the file):
[Fact]
public async Task GetSeriesAsync_RateMetric_SumsPerBucket_OnRollupPath()
{
// (KpiSources.SiteCallAudit, KpiMetrics.SiteCallAudit.FailedLastInterval) is a
// catalog Rate pair. Stub GetHourlySeriesAsync with 400 hourly points of value 1
// over a >168h window, maxPoints 200 → every output value is 2.0 (sum of the two
// hourly sums per bucket), not 1.0 (last value). Today: FAIL.
}
[Fact]
public async Task GetSeriesAsync_RateMetric_PreservesTotal_AcrossRoutingBoundary()
{
// Same total event count served as per-minute deltas (raw path, window == threshold)
// and as hourly sums (rollup path, window just over threshold): the summed chart
// total must be equal on both paths — the boundary is now visually seamless (R3).
}
[Fact]
public async Task GetSeriesAsync_GaugeMetric_KeepsLastValuePerBucket()
{
// queueDepth (uncatalogued → Gauge) keeps the existing last-value output unchanged.
}
- Run:
dotnet test tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests --filter KpiHistoryQueryServiceTests— expect FAIL. - Implementation in
GetSeriesAsync(both ctor paths):
// Per-metric reduction intent: the same catalog that drives the hourly fold drives
// the chart-time bucket reduction, so a Rate series is summed per bucket on BOTH the
// raw-minute and hourly-rollup routes — one unit ("events per chart bucket") on both
// sides of the RollupThresholdHours routing boundary (arch-review 04 round 2, R3).
var aggregation = KpiMetricAggregationCatalog.Resolve(source, metric);
...
return KpiSeriesBucketer.Bucket(injectedSeries, fromUtc, toUtc, effectiveMax, aggregation);
...
return KpiSeriesBucketer.Bucket(series, fromUtc, toUtc, effectiveMax, aggregation);
Extend the FetchSeriesAsync doc comment: the routing boundary no longer changes a Rate chart's magnitude — both paths reduce to per-bucket sums (a residual ≤~1.2× native-resolution difference exists only for an unbucketed just-over-threshold rollup series vs. an at-threshold bucketed raw series; documented in Task 7).
4. Run the filter — PASS (the pre-existing routing/forwarding tests must stay green — Gauge default preserves their expectations).
5. Commit: git add src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/KpiHistoryQueryService.cs tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Services/KpiHistoryQueryServiceTests.cs && git commit -m "fix(kpi): query service applies catalog aggregation at the bucketer — rate charts keep one unit across the raw/rollup boundary (plan R2-04 T6)"
Task 7: Truthful trend presentation — chart doc + Component-KpiHistory.md (R3, part 3)
Classification: small Estimated implement time: ~3 min Parallelizable with: 1, 2, 3, 8, 10, 11, 12 (requires Task 6; NOT 4/9 — same design-doc file) Files:
- Modify:
src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Shared/KpiTrendChart.razor.cs(XML docs onPoints:~55 andUnit:~62 — doc-only, no markup/behavior change; the component stays custom-SVG, no third-party libs) - Modify:
docs/requirements/Component-KpiHistory.md(:151 "Bucketed query", the "Raw-vs-rollup range routing" section :~153-155, and one line under ":157 Aggregation intent") - Modify: trend-panel
Unitstrings only where they now misstate the semantics — locate withgrep -rn "Unit=" src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pagesand fix any Rate-metric panel whose unit text claims a per-minute/per-interval reading (e.g. thedeliveredLastInterval/failedLastIntervalpanels) to say per-bucket totals (e.g.Unit="delivered/bucket"); leave Gauge panels untouched
KpiTrendChart.razor.csdoc: onPoints, note that Rate-classified series (perKpiMetricAggregationCatalog) now carry per-bucket totals (sum reduction), while Gauge series carry last-value-per-bucket readings; onUnit, note the supplied text must state the per-bucket semantics for Rate metrics.Component-KpiHistory.md:151: replace "returns the last value per bucket" with the per-metric reduction —KpiSeriesBucketer.Bucket(series, fromUtc, toUtc, maxPoints, aggregation)keeps the last value for Gauge series and sums per bucket for Rate series, driven by the sameKpiMetricAggregationCatalogthe fold consults (the previous last-value-on-rollups behavior kept 1 of ~11 hourly sums per 90d bucket — the fold error the catalog's own doc warns about, fixed per arch-review 04 round 2, R3). In the range-routing section add: "Rate charts keep one unit (events per chart bucket) on both sides of the boundary; widening 7d→30d no longer changes the magnitude ~60×. Residual: an unbucketed just-over-threshold rollup series plots native per-hour sums vs. the at-threshold raw path's ~50-minute bucket sums (≤~1.2×) — accepted and documented, not hidden." Under Aggregation intent, note the catalog now has two consumers: the hourly fold and the chart-time bucket reduction.- Verify no stale claim remains:
grep -n "last value per bucket" docs/requirements/Component-KpiHistory.mdreturns nothing. - Run the chart component suite to prove the doc-only touch broke nothing:
dotnet test tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests --filter KpiTrendChartTests— PASS. - Commit:
git add src/ZB.MOM.WW.ScadaBridge.CentralUI docs/requirements/Component-KpiHistory.md && git commit -m "docs(kpi): trend charts state per-bucket-total semantics for rate metrics (plan R2-04 T7)"
Task 8: Close the catalog metric-literal drift hazard (R4)
Classification: small Estimated implement time: ~4 min Parallelizable with: 1, 2, 3, 4, 5, 10, 11, 12 Files:
- Modify:
src/ZB.MOM.WW.ScadaBridge.Commons/Types/Kpi/KpiMetrics.cs(add three constants + amend the class remark) - Modify:
src/ZB.MOM.WW.ScadaBridge.Commons/Types/Kpi/KpiMetricAggregation.cs(:60-64 private literals; :98-107RatePairs) - Modify:
src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/Kpi/SiteCallAuditKpiSampleSource.cs(:45) - Modify:
src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/Kpi/SiteHealthKpiSampleSource.cs(:43, :51) - Test:
tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Kpi/KpiMetricAggregationTests.cs(extend)
- Failing test (does not compile until the constants exist — the TDD gate for symbol promotion):
[Fact]
public void PromotedMetricConstants_LockHistoricalPersistedValues()
{
// Persisted data contract — these are symbol promotions, NOT renames.
Assert.Equal("deliveredLastInterval", KpiMetrics.SiteCallAudit.DeliveredLastInterval);
Assert.Equal("alarmEvalErrors", KpiMetrics.SiteHealth.AlarmEvalErrors);
Assert.Equal("eventLogWriteFailures", KpiMetrics.SiteHealth.EventLogWriteFailures);
}
[Theory]
[InlineData(KpiSources.SiteCallAudit, KpiMetrics.SiteCallAudit.DeliveredLastInterval)]
[InlineData(KpiSources.SiteHealth, KpiMetrics.SiteHealth.AlarmEvalErrors)]
[InlineData(KpiSources.SiteHealth, KpiMetrics.SiteHealth.EventLogWriteFailures)]
public void Resolve_PromotedRatePairs_StillClassifyAsRate(string source, string metric)
=> Assert.Equal(KpiRollupAggregation.Rate, KpiMetricAggregationCatalog.Resolve(source, metric));
- Run:
dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter KpiMetricAggregationTests— expect FAIL (compile error: constants absent). - Implementation — compile-time coupling in the only dependency-legal direction (Commons cannot reference the source projects, so the shared constant lives in
KpiMetricsand both the catalog and the emitting source consume it):KpiMetrics.SiteCallAudit: addpublic const string DeliveredLastInterval = "deliveredLastInterval";;KpiMetrics.SiteHealth: addAlarmEvalErrors = "alarmEvalErrors"andEventLogWriteFailures = "eventLogWriteFailures"(EXACT existing string values — persisted, renames forbidden).- Amend the
KpiMetricsclass remark (:20-27): the catalog now also carries metrics the rollup aggregation catalog classifies, not only charted ones — the C4/R4 lesson is that any metric named in two places needs one shared symbol. KpiMetricAggregationCatalog: delete the three private literals (:62-64) and reference the constants inRatePairs; update the "otherwise the source's own private literal" remark (:46-48) — every catalogued name is now a sharedKpiMetricsconstant, so an emitter rename is a compile error, not a silent Gauge downgrade.- The two sources keep their private-const style with the value redefined in terms of the shared symbol (tiny diff, zero call-site churn):
private const string MetricDeliveredLastInterval = KpiMetrics.SiteCallAudit.DeliveredLastInterval;(SiteCallAudit :45) and likewise for the two SiteHealth consts (:43, :51).
- Run:
dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter KpiMetricAggregationTests && dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests --filter SiteCallAuditKpiSampleSource && dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests --filter SiteHealthKpiSampleSource— PASS (emitted strings unchanged). - Commit:
git add src/ZB.MOM.WW.ScadaBridge.Commons src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/Kpi/SiteCallAuditKpiSampleSource.cs src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/Kpi/SiteHealthKpiSampleSource.cs tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Kpi/KpiMetricAggregationTests.cs && git commit -m "fix(kpi): promote catalog metric literals to shared KpiMetrics constants — drift becomes a compile error (plan R2-04 T8)"
Task 9: Classify the failover fold-race failure grain (R5)
Classification: standard Estimated implement time: ~4 min Parallelizable with: 3, 5, 6, 8, 10, 11, 12 (NOT 1/2 — same repository file, do them first; NOT 4/7 — same design-doc file) Files:
- Modify:
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs(ctor + the fold'sSaveChangesAsyncat :167) - Modify:
docs/requirements/Component-KpiHistory.md(one sentence in the "Hourly rollup tick" paragraph, :~91) - Test:
tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/KpiHistoryRepositoryTests.cs(extend)
- Failing test — force a REAL unique violation deterministically with a
SaveChangesInterceptorthat, on the fold's first save, inserts the conflicting(series, hour)rollup row through a separate command on the same in-memory SQLite connection (simulating the old singleton incarnation's in-flight fold winning the race):
[Fact]
public async Task Fold_LosingFailoverUpsertRace_DoesNotThrow_AndNextFoldSelfHeals()
{
// ctx built with .AddInterceptors(new InsertConflictingRollupOnFirstSave(connection))
// seed one sample in a complete hour; FoldHourlyRollupsAsync must NOT propagate
// DbUpdateException (today: FAIL — the whole pass faults);
// a second FoldHourlyRollupsAsync over the same window must land the row (self-heal).
}
- Run:
dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter Fold_LosingFailoverUpsertRace— expect FAIL. - Implementation:
- Ctor gains
ILogger<KpiHistoryRepository>? logger = nulldefaulting toNullLogger(mirrorsSiteCallAuditRepository— MS.DI resolvesILogger<>automatically, no registration churn). - Wrap the fold's final
SaveChangesAsync:
- Ctor gains
try
{
await _context.SaveChangesAsync(cancellationToken);
}
catch (DbUpdateException ex)
{
// Failover-overlap race: the old singleton incarnation's in-flight fold and the
// new node's first fold can both Add the same (series, hour) row; the unique
// IX_KpiRollupHourly_Series then faults the loser's ENTIRE SaveChanges — the
// failure grain is the PASS, not the row (arch-review 04 round 2, R5). The fold
// only ever writes KpiRollupHourly rows, so any save fault here is a lost upsert
// race or a transient the next idempotent re-fold repairs identically; classify
// at Information instead of surfacing a scary error for a self-healing no-op.
_logger.LogInformation(ex,
"KPI rollup fold lost a failover-overlap upsert race; pass discarded, next fold self-heals.");
}
- Run:
dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter KpiHistoryRepositoryTests— PASS. - Doc: add to the rollup-tick paragraph of
Component-KpiHistory.md: "During a failover overlap the loser of the(series, hour)upsert race discards its whole fold pass (singleSaveChanges) — logged at Information, repaired by the next idempotent fold; when reading fold logs after a failover, the failure grain is the pass, not the row." - Commit:
git add src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs docs/requirements/Component-KpiHistory.md tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/KpiHistoryRepositoryTests.cs && git commit -m "fix(kpi): classify the failover fold-race pass loss as a self-healing Information event (plan R2-04 T9)"
Task 10: SiteCalls filtered terminal index (migration) (R6, part 1)
Classification: high-risk Estimated implement time: ~4 min Parallelizable with: 1, 2, 3, 4, 5, 6, 7, 8, 9, 12 (the plan's only migration — but serialize against any OTHER plan adding ConfigurationDatabase migrations: model-snapshot ordering) Files:
- Modify:
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/SiteCallEntityTypeConfiguration.cs(after theIX_SiteCalls_NonTerminalblock ending :~99) - Create:
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/<timestamp>_AddSiteCallsTerminalIndex.cs(scaffolded) - Test:
tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/SchemaConfigurationTests.cs(extend)
- Failing model test (SchemaConfigurationTests already asserts index shapes):
[Fact]
public void SiteCalls_Has_Filtered_Terminal_Index()
{
var entity = Model.FindEntityType(typeof(SiteCall))!;
var index = entity.GetIndexes().Single(i => i.GetDatabaseName() == "IX_SiteCalls_Terminal");
Assert.Equal(new[] { nameof(SiteCall.TerminalAtUtc) }, index.Properties.Select(p => p.Name).ToArray());
Assert.Equal("[TerminalAtUtc] IS NOT NULL", index.GetFilter());
}
- Run:
dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter SiteCalls_Has_Filtered_Terminal_Index— expect FAIL. - Add to
SiteCallEntityTypeConfiguration.Configure:
// Terminal backs the daily retention purge's predicate (TerminalAtUtc IS NOT NULL
// AND TerminalAtUtc < cutoff) and Task 11's MIN() slice anchor: filtered to the
// terminal population so the purge seeks the expired tail instead of full-scanning
// the 365-day table (arch-review 04 round 2, R6). Complements IX_SiteCalls_NonTerminal,
// which is filtered to IS NULL and unusable for this predicate.
builder.HasIndex(s => s.TerminalAtUtc)
.HasDatabaseName("IX_SiteCalls_Terminal")
.HasFilter("[TerminalAtUtc] IS NOT NULL");
- Scaffold the migration — repo gotcha: BUILD FIRST, never
dotnet ef migrations add --no-build(it scaffolds an EMPTY migration off the stale previously-built DLL; if that happens, delete the emptyMigrations/files —migrations removeneeds a live DB):
dotnet build ZB.MOM.WW.ScadaBridge.slnx
cd src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase
dotnet ef migrations add AddSiteCallsTerminalIndex
Verify the scaffold contains the CreateIndex with the IS NOT NULL filter — if it is empty, the gotcha bit; delete and redo.
5. Run the model test — PASS. With infra up (cd infra && docker compose up -d), run the MSSQL-backed suite: dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter SiteCallAuditRepositoryTests — PASS. Production script (repo convention — manual SQL for production): dotnet ef migrations script AddKpiRollupHourlyTable AddSiteCallsTerminalIndex --idempotent --output ../../docs/plans/sql/AddSiteCallsTerminalIndex.sql (verify the from-migration is still the latest applied; review the output).
6. Commit: git add -A src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/SchemaConfigurationTests.cs docs/plans/sql && git commit -m "perf(sitecallaudit): filtered IX_SiteCalls_Terminal index backs the retention purge predicate (plan R2-04 T10)"
Task 11: Time-sliced SiteCalls terminal purge (R6, part 2)
Classification: standard Estimated implement time: ~4 min Parallelizable with: 1, 2, 3, 4, 5, 6, 7, 8, 9, 12 (requires Task 10 — the MIN() anchor and DELETE predicate seek its index) Files:
- Modify:
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs(PurgeTerminalAsync:247-252) - Test:
tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/SiteCallAuditRepositoryTests.cs(extend; MSSQLSkippableFact— infra up)
- Failing test (pattern-match the fixture usage of the existing
SkippableFacttests in the file; assert slicing via an EFDbCommandInterceptorcapturing DELETE statements, or SQL capture matching how sibling tests inspect SQL):
[SkippableFact]
public async Task PurgeTerminal_SlicesMultiDayBacklog_IntoBoundedDeletes()
{
// Seed terminal rows spread across 3 distinct days beyond the cutoff, one terminal
// row inside the window, and one OLD non-terminal row.
// Purge → returns 3-day total; the fresh terminal row and the old NON-terminal row
// survive (non-terminal rows are NEVER purged on age — sites remain source of truth);
// captured SQL shows MORE THAN ONE DELETE statement for the multi-day backlog.
// Today: FAIL — a single unbatched DELETE is issued.
}
- Run:
dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter PurgeTerminal_SlicesMultiDayBacklog— expect FAIL. - Implement time-sliced batching mirroring
KpiHistoryRepository.PurgeOlderThanAsync(the round-1 Task 19 shape), keeping the raw-T-SQL MSSQL-only style this repository already uses:
public async Task<int> PurgeTerminalAsync(DateTime olderThanUtc, CancellationToken ct = default)
{
// Time-sliced batches (arch-review 04 round 2, R6 — the one maintenance DELETE
// that missed round 1's batching pass): each DELETE covers at most one DAY of
// terminal rows, capping the lock/log footprint per statement. Steady state
// (daily purge, 365-day retention) is a single slice; only catch-up after an
// outage runs several. One-day (not one-hour) slices are proportionate to
// SiteCalls volume, which is far below KpiSample's. The MIN() anchor and the
// DELETE predicate both seek IX_SiteCalls_Terminal (filtered IS NOT NULL).
var total = 0;
var floor = await _context.SiteCalls
.Where(s => s.TerminalAtUtc != null && s.TerminalAtUtc < olderThanUtc)
.MinAsync(s => s.TerminalAtUtc, ct);
while (floor is not null && floor < olderThanUtc)
{
var ceiling = floor.Value.AddDays(1) < olderThanUtc ? floor.Value.AddDays(1) : olderThanUtc;
total += await _context.Database.ExecuteSqlInterpolatedAsync(
$"DELETE FROM dbo.SiteCalls WHERE TerminalAtUtc IS NOT NULL AND TerminalAtUtc < {ceiling};",
ct);
floor = await _context.SiteCalls
.Where(s => s.TerminalAtUtc != null && s.TerminalAtUtc < olderThanUtc)
.MinAsync(s => s.TerminalAtUtc, ct);
}
return total;
}
- Run:
dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter SiteCallAuditRepositoryTests— PASS (upsert/query/KPI tests untouched). Then the purge-actor contract:dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests --filter SiteCallAuditPurgeTests— PASS (return-total contract unchanged). - Commit:
git add src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/SiteCallAuditRepositoryTests.cs && git commit -m "fix(sitecallaudit): time-sliced terminal purge replaces the single year-scale DELETE (plan R2-04 T11)"
Task 12: Retention-service shutdown no longer surfaces cancellation (R7)
Classification: small Estimated implement time: ~3 min Parallelizable with: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11 Files:
- Modify:
src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SiteAuditRetentionService.cs(RunLoopAsync:62-91) - Test:
tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SiteAuditRetentionServiceTests.cs(extend; theRecordingSiteAuditQueuestub gains a block-until-cancelled mode)
- Failing test:
[Fact]
public async Task StopAsync_MidPurge_CompletesCleanly_WithoutSurfacingCancellation()
{
var queue = new RecordingSiteAuditQueue { BlockUntilCancelled = true }; // PurgeExpiredAsync: await Task.Delay(Timeout.Infinite, ct)
var options = Options.Create(new SiteAuditRetentionOptions
{ RetentionDays = 7, PurgeInterval = TimeSpan.FromHours(24), InitialDelay = TimeSpan.Zero });
using var svc = new SiteAuditRetentionService(queue, options, NullLogger<SiteAuditRetentionService>.Instance);
await svc.StartAsync(CancellationToken.None);
await WaitUntilAsync(() => queue.PurgeCalls.Count >= 1, TimeSpan.FromSeconds(5));
// Today: FAIL — the OCE SafePurgeAsync deliberately rethrows escapes RunLoopAsync
// (the two awaits at :76/:89 sit outside any try/catch), cancels _loop, and
// StopAsync returns that canceled task straight to the host (:122-126), whose
// await throws TaskCanceledException into every shutdown log (R7).
await svc.StopAsync(CancellationToken.None); // must NOT throw
}
- Run:
dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter StopAsync_MidPurge— expect FAIL (TaskCanceledException). - Implementation — catch OCE at the loop boundary (the report's preferred alignment with the sibling
SiteAuditBacklogReporterpattern), leavingSafePurgeAsync's deliberate rethrow (:106-110) untouched so a mid-purge shutdown still aborts the purge promptly:
private async Task RunLoopAsync(CancellationToken ct)
{
try
{
// ... existing body verbatim (initial delay, first purge, delay/purge loop) ...
}
catch (OperationCanceledException)
{
// Shutdown landed mid-purge: SafePurgeAsync rethrows OCE by design so the purge
// aborts promptly, but the loop task must complete CLEANLY — StopAsync hands
// _loop straight to the host, and a canceled task there is shutdown-log noise
// (arch-review 04 round 2, R7). Cancellation here IS the clean exit.
}
}
- Run:
dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter SiteAuditRetentionServiceTests— PASS (the two existing tick tests must stay green). - Commit:
git add src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SiteAuditRetentionService.cs tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SiteAuditRetentionServiceTests.cs && git commit -m "fix(audit-log): retention loop absorbs shutdown cancellation instead of faulting host StopAsync (plan R2-04 T12)"
Dependencies on other plans
- None of the round-1 plans have open tasks (the whole arch-review initiative closed 191/192 on 2026-07-10); no cross-plan file contention is expected.
- Task 10 is this plan's only EF migration. No other plan currently adds ConfigurationDatabase migrations, but if one appears, serialize on the model snapshot (same rule as round-1 Tasks 6→8).
- The
IKpiHistoryRepositoryaddition (Task 2) is additive; any concurrently developed fake/stub implementing the interface must add the member — build the solution after Task 2 to surface stragglers immediately.
Execution order
P0 (the High + its enablers, in order): 1 → 2 → 3 → 4 (fold fetch, watermark seam, sliced backfill, failover fast-path).
P1: 5 → 6 → 7 (rate presentation chain), 10 → 11 (SiteCalls purge chain).
P2 (low-severity cleanup): 8, 9, 12 — broadly parallel per each task's "Parallelizable with" contract (watch the two serialization lanes: KpiHistoryRepository.cs = 1 → 2 → 9; Component-KpiHistory.md = 4 → 7 → 9 in whatever order they run, never concurrently).
Finish with dotnet build ZB.MOM.WW.ScadaBridge.slnx plus the targeted suites this plan touched — dotnet test tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests --filter "KpiHistoryRepositoryTests|SiteCallAuditRepositoryTests|SchemaConfigurationTests" (infra up for the MSSQL SkippableFacts), dotnet test tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests, dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter "FullyQualifiedName~Kpi", dotnet test tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests --filter "KpiHistoryQueryServiceTests|KpiTrendChartTests", dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests, dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests --filter SiteHealthKpiSampleSource, and dotnet test tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests --filter "FullyQualifiedName~Site" — before declaring the plan done.
Findings Coverage
| # | Report finding (severity) | Task(s) |
|---|---|---|
| R1 | Startup backfill folds the entire raw-retention window in one unbounded pass, re-run on every failover (High) | 2, 3, 4 |
| R2 | Fold fetch is change-tracked — AsNoTracking/projection missing (Medium) |
1 |
| R3 | Rate metrics change units ~60× across the raw/rollup routing boundary; bucketer re-introduces the last-value fold error on hourly Rate sums (Medium) | 5, 6, 7 |
| R4 | KpiMetricAggregationCatalog re-creates the private-metric-literal drift hazard C4 fixed (Low) |
8 |
| R5 | Overlapping-singleton fold race faults an entire fold batch — failure grain is the pass, not the row (Low) | 9 |
| R6 | PurgeTerminalAsync on SiteCalls is the one remaining unbatched, unindexed year-scale maintenance DELETE (Medium) |
10, 11 |
| R7 | SiteAuditRetentionService.StopAsync surfaces the loop's cancellation to the host (Low) |
12 |
| S8 | Outbox repository dual SQLite/T-SQL dialect | No action — accepted round-1 deferral; round-2 re-verified the documented convention (Component-ConfigurationDatabase.md:69, "do not copy") |
| S10 | SqliteAuditWriter.Dispose sync-over-async |
No action — accepted round-1 won't-fix; round-2 re-verified |
| P3 | AuditLog clustered key leads with random GUID | No action — accepted round-1 deferral, tracked benchmark follow-up (Component-ConfigurationDatabase.md:70) |
| P4 (cadence) | Per-node KPI sampling cadence | No action — accepted round-1 deferral, tracked (Component-ConfigurationDatabase.md:71); the purge half shipped in round 1 |
| P6 (paging) | Outbox offset→keyset page conversion | No action — accepted round-1 deferral, tracked (Component-ConfigurationDatabase.md:71); the KPI half shipped in round 1 |
| C2 | AuditLogRow lives in ConfigurationDatabase, not Commons |
No action — accepted round-1 deviation, documented (Component-Commons.md:51, "do not lift it into Commons") |
| C3 | Mixed timestamp CLR types | No action — accepted round-1 convention note (Component-ConfigurationDatabase.md:68) |
| U3 | Hash-chain tamper evidence / Parquet archival | No action — unchanged v1.x deferral; round-2 confirms no drift |
| U5 (residue) | AuditLog PullAuditEvents keyset upgrade |
No action — accepted round-1 deferral, tracked (Component-ConfigurationDatabase.md:71); idempotent on EventId, lower urgency; round-2 lists it only so the tracker entry doesn't age out |