dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(kpihistory)+fix(notificationoutbox)+fix(sitecallaudit)+fix(host): P5-5 — rate-fold interval triple made self-consistent + cross-validated

Browse Source 
Defect: the hourly KPI rollup fold (KpiHistoryRepository.FoldHourlyRollupsAsync)
sums every Rate-classified sample recorded in an hour, correct only if each
sample is a non-overlapping delta covering exactly one recorder tick
(KpiHistoryOptions.SampleInterval). That invariant was an unenforced silent
assumption across three independently-owned config keys — SampleInterval,
NotificationOutboxOptions.DeliveredKpiWindow, SiteCallAuditOptions.KpiInterval —
so widening DeliveredKpiWindow to e.g. 15 min would silently over-count the
rollup ~15x.

Robust fix: NotificationOutboxKpiSampleSource and SiteCallAuditKpiSampleSource
now rescale their Rate-classified samples (deliveredLastInterval,
failedLastInterval) by SampleInterval / (DeliveredKpiWindow or KpiInterval)
before recording, so the rollup's per-hour Sum is correct regardless of how
the other two keys are tuned — the live dashboard tile's own cutoff/window is
untouched. Both sources gain a project reference to KpiHistory to read
SampleInterval; no cycle (KpiHistory references only Commons).

Eager cross-validation (defense-in-depth): KpiRateFoldIntervalCrossValidator
(Host, the one composition root referencing all three component projects)
fails boot naming all three config keys when they diverge, protecting the
separate (still-real) invariant that the recorded/charted KPI-history sample
equals the live tile at the same instant — which the rescale trades away
whenever the scale factor isn't 1. Rides KpiHistoryOptions' existing
ValidateOnStart trigger.

Also: NotificationOutboxOptions had no registered validator at all before this
(the gap this task's "whichever options class owns DeliveredKpiWindow" file
scope pointed at) — added NotificationOutboxOptionsValidator mirroring
SiteCallAuditOptionsValidator's positive-duration/count convention, wired with
ValidateOnStart.

Tests: new rescale tests in both sample-source suites prove the fold is
numerically correct at a non-default SampleInterval (30/40 counted over a
15/20-minute window correctly divides down to a 1-minute-tick delta); new
KpiRateFoldIntervalCrossValidatorTests prove divergent config fails with all
three keys named; new NotificationOutboxOptionsValidatorTests. Every new test
proven to fail against the unfixed computation first (temporarily
short-circuited the scale factor / cross-check to the pre-fix behavior,
confirmed failure, restored) — not via git stash (repo-wide, shared across
concurrent worktrees this session) but via file-copy swap.

Full suites green: Host.Tests (527), ConfigurationDatabase.Tests (401),
KpiHistory.Tests (54), NotificationOutbox.Tests (163),
SiteCallAudit.Tests (67).
This commit is contained in:
Joseph Doherty
2026-08-19 16:39:50 -04:00
parent 1cea01d924
commit abbbcd1a8d
13 changed files with 613 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/Kpi/SiteCallAuditKpiSampleSource.cs
+61 -8
View File
@@ -4,6 +4,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Kpi;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Kpi;
using ZB.MOM.WW.ScadaBridge.KpiHistory;
namespace ZB.MOM.WW.ScadaBridge.SiteCallAudit.Kpi;
@@ -21,8 +22,10 @@ namespace ZB.MOM.WW.ScadaBridge.SiteCallAudit.Kpi;
/// live <c>SiteCallAuditActor</c> KPI handlers derive them
/// (<c>stuckCutoff = capturedAtUtc - StuckAgeThreshold</c>,
/// <c>intervalSince = capturedAtUtc - KpiInterval</c>). The COUNT metrics
/// (buffered, parked, failedLastInterval, deliveredLastInterval, stuck) equal the
/// live tile at the same instant; the <c>oldestPendingAgeSeconds</c> metric is
/// (buffered, parked, stuck) equal the live tile at the same instant;
/// failedLastInterval and deliveredLastInterval are rescaled before recording (see
/// below) and equal the live tile only when that rescale factor is 1, which the
/// boot-time cross-validator guarantees; the <c>oldestPendingAgeSeconds</c> metric is
/// computed against the repository's internal clock and may differ from the live
/// tile by the query-execution latency. The recorder's <c>capturedAtUtc</c> is
/// the single anchor for both cutoffs.
@@ -32,6 +35,25 @@ namespace ZB.MOM.WW.ScadaBridge.SiteCallAudit.Kpi;
/// each sampling pass resolves a fresh repository scope, mirroring the actor's
/// scope-per-message repository access.
/// </para>
/// <para>
/// <b>failedLastInterval / deliveredLastInterval rescale (P5-5, arch-review Low).</b>
/// <see cref="KpiMetricAggregationCatalog"/> classifies both as
/// <see cref="KpiRollupAggregation.Rate"/>: the central hourly rollup fold sums every
/// sample recorded in the hour, which is only correct if each sample's value is a
/// non-overlapping delta covering exactly one recorder tick
/// (<c>KpiHistoryOptions.SampleInterval</c>). The live cutoff this class queries the
/// repository with is still anchored on the independently-configurable
/// <see cref="SiteCallAuditOptions.KpiInterval"/> — unchanged, so the live dashboard
/// tile keeps its own smoothing window — but the COUNTS that get written into
/// <see cref="KpiSample.Value"/> are rescaled by <c>SampleInterval / KpiInterval</c>
/// before they are recorded, so the rollup's sum is correct for the hour regardless
/// of how <c>KpiInterval</c> is tuned. At the documented defaults (both 1 minute) the
/// scale factor is exactly 1 and this is a no-op. The boot-time cross-validator
/// (Host's <c>KpiRateFoldIntervalCrossValidator</c>) enforces
/// <c>KpiInterval == SampleInterval</c> (alongside the outbox's
/// <c>DeliveredKpiWindow</c>) so a config where the recorded/charted samples no
/// longer equal the live tile can never actually boot.
/// </para>
/// </remarks>
public sealed class SiteCallAuditKpiSampleSource : IKpiSampleSource
{
@@ -48,6 +70,7 @@ public sealed class SiteCallAuditKpiSampleSource : IKpiSampleSource
private readonly ISiteCallAuditRepository _repository;
private readonly SiteCallAuditOptions _options;
private readonly KpiHistoryOptions _kpiHistoryOptions;
/// <summary>
/// Creates the sample source over the central <c>SiteCalls</c> repository and
@@ -56,17 +79,43 @@ public sealed class SiteCallAuditKpiSampleSource : IKpiSampleSource
/// </summary>
/// <param name="repository">The central <c>SiteCalls</c> operational-state repository.</param>
/// <param name="options">Site Call Audit windowing options (stuck-age + KPI interval).</param>
/// <param name="kpiHistoryOptions">
/// KPI History options carrying <see cref="KpiHistoryOptions.SampleInterval"/> — the
/// recorder's sampling cadence the failedLastInterval/deliveredLastInterval samples are
/// rescaled onto (P5-5), so the hourly rollup fold's per-sample sum stays correct
/// independent of <see cref="SiteCallAuditOptions.KpiInterval"/>.
/// </param>
public SiteCallAuditKpiSampleSource(
ISiteCallAuditRepository repository,
IOptions<SiteCallAuditOptions> options)
IOptions<SiteCallAuditOptions> options,
IOptions<KpiHistoryOptions> kpiHistoryOptions)
{
ArgumentNullException.ThrowIfNull(repository);
ArgumentNullException.ThrowIfNull(options);
ArgumentNullException.ThrowIfNull(kpiHistoryOptions);
_repository = repository;
_options = options.Value;
_kpiHistoryOptions = kpiHistoryOptions.Value;
}
/// <summary>
/// Rescale factor applied to the raw <c>failedLastInterval</c>/<c>deliveredLastInterval</c>
/// counts before they are recorded, so the values written into
/// <see cref="KpiSample.Value"/> represent non-overlapping
/// per-<see cref="KpiHistoryOptions.SampleInterval"/> deltas regardless of how wide
/// <see cref="SiteCallAuditOptions.KpiInterval"/> is configured. Exactly 1 at the
/// documented defaults (both 1 minute). Guards against a non-positive
/// <see cref="SiteCallAuditOptions.KpiInterval"/> (validated &gt; 0 at boot by
/// <see cref="SiteCallAuditOptionsValidator"/>, but this defends the arithmetic even if
/// that validation is ever bypassed) by falling back to the unscaled 1:1 factor rather
/// than dividing by zero.
/// </summary>
private double IntervalScaleFactor =>
_options.KpiInterval > TimeSpan.Zero
? _kpiHistoryOptions.SampleInterval.TotalSeconds / _options.KpiInterval.TotalSeconds
: 1.0;
/// <inheritdoc />
public string Source => KpiSources.SiteCallAudit;
@@ -90,12 +139,13 @@ public sealed class SiteCallAuditKpiSampleSource : IKpiSampleSource
.ConfigureAwait(false);
var samples = new List<KpiSample>();
var scaleFactor = IntervalScaleFactor;
// Global scope (null ScopeKey).
AddSnapshot(
samples, capturedAtUtc, KpiScopes.Global, scopeKey: null,
global.BufferedCount, global.ParkedCount,
global.FailedLastInterval, global.DeliveredLastInterval,
global.FailedLastInterval * scaleFactor, global.DeliveredLastInterval * scaleFactor,
global.OldestPendingAge, global.StuckCount);
// Per-site scope (ScopeKey = source site id).
@@ -104,7 +154,7 @@ public sealed class SiteCallAuditKpiSampleSource : IKpiSampleSource
AddSnapshot(
samples, capturedAtUtc, KpiScopes.Site, scopeKey: site.SourceSite,
site.BufferedCount, site.ParkedCount,
site.FailedLastInterval, site.DeliveredLastInterval,
site.FailedLastInterval * scaleFactor, site.DeliveredLastInterval * scaleFactor,
site.OldestPendingAge, site.StuckCount);
}
@@ -114,7 +164,7 @@ public sealed class SiteCallAuditKpiSampleSource : IKpiSampleSource
AddSnapshot(
samples, capturedAtUtc, KpiScopes.Node, scopeKey: node.SourceNode,
node.BufferedCount, node.ParkedCount,
node.FailedLastInterval, node.DeliveredLastInterval,
node.FailedLastInterval * scaleFactor, node.DeliveredLastInterval * scaleFactor,
node.OldestPendingAge, node.StuckCount);
}
@@ -125,6 +175,9 @@ public sealed class SiteCallAuditKpiSampleSource : IKpiSampleSource
/// Appends the six-metric catalog for one snapshot at the given scope. The
/// oldest-pending-age metric is omitted when the snapshot's age is
/// <c>null</c> (no non-terminal rows) rather than written as zero.
/// <paramref name="failedLastInterval"/>/<paramref name="deliveredLastInterval"/> are
/// already rescaled by <see cref="IntervalScaleFactor"/> (a <c>double</c>, not the raw
/// repository <c>int</c> counts) by the caller.
/// </summary>
private void AddSnapshot(
List<KpiSample> samples,
@@ -133,8 +186,8 @@ public sealed class SiteCallAuditKpiSampleSource : IKpiSampleSource
string? scopeKey,
int buffered,
int parked,
int failedLastInterval,
int deliveredLastInterval,
double failedLastInterval,
double deliveredLastInterval,
TimeSpan? oldestPendingAge,
int stuck)
{
src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/ZB.MOM.WW.ScadaBridge.SiteCallAudit.csproj
+6
View File
@@ -39,6 +39,12 @@
SiteCallAudit. Preferred over moving the interfaces into Commons (Commons has no Akka /
Communication dependency and would have to carry a Communication-adjacent message). -->
<ProjectReference Include="../ZB.MOM.WW.ScadaBridge.AuditLog/ZB.MOM.WW.ScadaBridge.AuditLog.csproj" />
<!-- The KPI sample source computes deliveredLastInterval/failedLastInterval
against KpiHistoryOptions.SampleInterval directly (P5-5), not the
independently configurable KpiInterval, so the central hourly rollup
fold's per-sample sum is correct regardless of how KpiInterval is tuned.
No cycle: KpiHistory references only Commons. -->
<ProjectReference Include="../ZB.MOM.WW.ScadaBridge.KpiHistory/ZB.MOM.WW.ScadaBridge.KpiHistory.csproj" />
</ItemGroup>
<ItemGroup>