fix(kpihistory)+fix(notificationoutbox)+fix(sitecallaudit)+fix(host): P5-5 — rate-fold interval triple made self-consistent + cross-validated
Defect: the hourly KPI rollup fold (KpiHistoryRepository.FoldHourlyRollupsAsync) sums every Rate-classified sample recorded in an hour, correct only if each sample is a non-overlapping delta covering exactly one recorder tick (KpiHistoryOptions.SampleInterval). That invariant was an unenforced silent assumption across three independently-owned config keys — SampleInterval, NotificationOutboxOptions.DeliveredKpiWindow, SiteCallAuditOptions.KpiInterval — so widening DeliveredKpiWindow to e.g. 15 min would silently over-count the rollup ~15x. Robust fix: NotificationOutboxKpiSampleSource and SiteCallAuditKpiSampleSource now rescale their Rate-classified samples (deliveredLastInterval, failedLastInterval) by SampleInterval / (DeliveredKpiWindow or KpiInterval) before recording, so the rollup's per-hour Sum is correct regardless of how the other two keys are tuned — the live dashboard tile's own cutoff/window is untouched. Both sources gain a project reference to KpiHistory to read SampleInterval; no cycle (KpiHistory references only Commons). Eager cross-validation (defense-in-depth): KpiRateFoldIntervalCrossValidator (Host, the one composition root referencing all three component projects) fails boot naming all three config keys when they diverge, protecting the separate (still-real) invariant that the recorded/charted KPI-history sample equals the live tile at the same instant — which the rescale trades away whenever the scale factor isn't 1. Rides KpiHistoryOptions' existing ValidateOnStart trigger. Also: NotificationOutboxOptions had no registered validator at all before this (the gap this task's "whichever options class owns DeliveredKpiWindow" file scope pointed at) — added NotificationOutboxOptionsValidator mirroring SiteCallAuditOptionsValidator's positive-duration/count convention, wired with ValidateOnStart. Tests: new rescale tests in both sample-source suites prove the fold is numerically correct at a non-default SampleInterval (30/40 counted over a 15/20-minute window correctly divides down to a 1-minute-tick delta); new KpiRateFoldIntervalCrossValidatorTests prove divergent config fails with all three keys named; new NotificationOutboxOptionsValidatorTests. Every new test proven to fail against the unfixed computation first (temporarily short-circuited the scale factor / cross-check to the pre-fix behavior, confirmed failure, restored) — not via git stash (repo-wide, shared across concurrent worktrees this session) but via file-copy swap. Full suites green: Host.Tests (527), ConfigurationDatabase.Tests (401), KpiHistory.Tests (54), NotificationOutbox.Tests (163), SiteCallAudit.Tests (67).
This commit is contained in:
Joseph Doherty
+66
-8
@@ -4,6 +4,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Kpi;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Kpi;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Notifications;
|
||||
using ZB.MOM.WW.ScadaBridge.KpiHistory;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.NotificationOutbox.Kpi;
|
||||
|
||||
@@ -20,9 +21,12 @@ namespace ZB.MOM.WW.ScadaBridge.NotificationOutbox.Kpi;
|
||||
/// <c>capturedAtUtc</c> rather than wall-clock <c>now</c>: the stuck cutoff is
|
||||
/// <c>capturedAtUtc - <see cref="NotificationOutboxOptions.StuckAgeThreshold"/></c> and the
|
||||
/// delivered window is <c>capturedAtUtc - <see cref="NotificationOutboxOptions.DeliveredKpiWindow"/></c>.
|
||||
/// The COUNT metrics (queueDepth, stuckCount, parkedCount, deliveredLastInterval) equal the
|
||||
/// live tile at the same instant; the <c>oldestPendingAgeSeconds</c> metric is computed against
|
||||
/// the repository's internal clock and may differ from the live tile by the query-execution latency.
|
||||
/// The COUNT metrics (queueDepth, stuckCount, parkedCount) equal the live tile at the same
|
||||
/// instant; deliveredLastInterval is rescaled onto <c>KpiHistoryOptions.SampleInterval</c>
|
||||
/// before being recorded (see the rescale note below) and equals the live tile only when
|
||||
/// that rescale factor is 1, which the boot-time cross-validator guarantees; the
|
||||
/// <c>oldestPendingAgeSeconds</c> metric is computed against the repository's internal clock
|
||||
/// and may differ from the live tile by the query-execution latency.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Emits Global (<c>ScopeKey == null</c>), per-Site (<c>ScopeKey == SourceSiteId</c>), and
|
||||
@@ -31,6 +35,28 @@ namespace ZB.MOM.WW.ScadaBridge.NotificationOutbox.Kpi;
|
||||
/// <c>oldestPendingAgeSeconds</c> via <see cref="TimeSpan.TotalSeconds"/>; that one metric is
|
||||
/// omitted when the age is <c>null</c>.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>deliveredLastInterval rescale (P5-5, arch-review Low).</b>
|
||||
/// <see cref="KpiMetricAggregationCatalog"/> classifies this metric as a
|
||||
/// <see cref="KpiRollupAggregation.Rate"/>: the central hourly rollup fold sums every
|
||||
/// sample recorded in the hour, which is only correct if each sample's value is a
|
||||
/// non-overlapping delta covering exactly one recorder tick
|
||||
/// (<c>KpiHistoryOptions.SampleInterval</c>). The live cutoff this class queries the
|
||||
/// repository with is still anchored on the independently-configurable
|
||||
/// <see cref="NotificationOutboxOptions.DeliveredKpiWindow"/> — unchanged, so the live
|
||||
/// dashboard tile keeps its own smoothing window — but the COUNT that gets written into
|
||||
/// <see cref="KpiSample.Value"/> is rescaled by
|
||||
/// <c>SampleInterval / DeliveredKpiWindow</c> before it is recorded, so the rollup's sum
|
||||
/// is correct for the hour regardless of how <c>DeliveredKpiWindow</c> is tuned (e.g.
|
||||
/// widened to smooth a noisy live tile). At the documented defaults (both 1 minute) the
|
||||
/// scale factor is exactly 1 and this is a no-op. A configuration where the scale factor
|
||||
/// is not 1 — i.e. where the live tile's window no longer equals the recorder's sampling
|
||||
/// cadence — still boots and rolls up correctly, but the recorded/charted
|
||||
/// deliveredLastInterval sample then no longer numerically equals the live tile at the
|
||||
/// same instant (only the <em>rollup total</em> is guaranteed correct); the boot-time
|
||||
/// cross-validator (Host's <c>KpiRateFoldIntervalCrossValidator</c>) enforces the equality
|
||||
/// so that divergence can never actually happen in a config that boots.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class NotificationOutboxKpiSampleSource : IKpiSampleSource
|
||||
{
|
||||
@@ -44,23 +70,51 @@ public sealed class NotificationOutboxKpiSampleSource : IKpiSampleSource
|
||||
|
||||
private readonly INotificationOutboxRepository _repository;
|
||||
private readonly NotificationOutboxOptions _options;
|
||||
private readonly KpiHistoryOptions _kpiHistoryOptions;
|
||||
|
||||
/// <summary>
|
||||
/// Creates the sample source.
|
||||
/// </summary>
|
||||
/// <param name="repository">Outbox repository providing the KPI computation methods.</param>
|
||||
/// <param name="options">Outbox options carrying the stuck-age and delivered-window cutoffs.</param>
|
||||
/// <param name="kpiHistoryOptions">
|
||||
/// KPI History options carrying <see cref="KpiHistoryOptions.SampleInterval"/> — the
|
||||
/// recorder's sampling cadence the deliveredLastInterval sample is rescaled onto (P5-5),
|
||||
/// so the hourly rollup fold's per-sample sum stays correct independent of
|
||||
/// <see cref="NotificationOutboxOptions.DeliveredKpiWindow"/>.
|
||||
/// </param>
|
||||
public NotificationOutboxKpiSampleSource(
|
||||
INotificationOutboxRepository repository,
|
||||
IOptions<NotificationOutboxOptions> options)
|
||||
IOptions<NotificationOutboxOptions> options,
|
||||
IOptions<KpiHistoryOptions> kpiHistoryOptions)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(repository);
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
ArgumentNullException.ThrowIfNull(kpiHistoryOptions);
|
||||
|
||||
_repository = repository;
|
||||
_options = options.Value;
|
||||
_kpiHistoryOptions = kpiHistoryOptions.Value;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Rescale factor applied to the raw <c>deliveredLastInterval</c> count before it is
|
||||
/// recorded, so the value written into <see cref="KpiSample.Value"/> represents a
|
||||
/// non-overlapping per-<see cref="KpiHistoryOptions.SampleInterval"/> delta regardless of
|
||||
/// how wide <see cref="NotificationOutboxOptions.DeliveredKpiWindow"/> is configured — the
|
||||
/// hourly rollup fold (<c>KpiHistoryRepository.FoldHourlyRollupsAsync</c>) sums this value
|
||||
/// across the hour's samples, which is only correct for non-overlapping deltas. Exactly 1
|
||||
/// at the documented defaults (both 1 minute). Guards against a non-positive
|
||||
/// <see cref="NotificationOutboxOptions.DeliveredKpiWindow"/> (validated > 0 at boot by
|
||||
/// <see cref="NotificationOutboxOptionsValidator"/>, but this defends the arithmetic even
|
||||
/// if that validation is ever bypassed) by falling back to the unscaled 1:1 factor rather
|
||||
/// than dividing by zero.
|
||||
/// </summary>
|
||||
private double DeliveredIntervalScaleFactor =>
|
||||
_options.DeliveredKpiWindow > TimeSpan.Zero
|
||||
? _kpiHistoryOptions.SampleInterval.TotalSeconds / _options.DeliveredKpiWindow.TotalSeconds
|
||||
: 1.0;
|
||||
|
||||
/// <inheritdoc />
|
||||
public string Source => KpiSources.NotificationOutbox;
|
||||
|
||||
@@ -75,13 +129,14 @@ public sealed class NotificationOutboxKpiSampleSource : IKpiSampleSource
|
||||
var deliveredSince = capturedAt - _options.DeliveredKpiWindow;
|
||||
|
||||
var samples = new List<KpiSample>();
|
||||
var scaleFactor = DeliveredIntervalScaleFactor;
|
||||
|
||||
var global = await _repository.ComputeKpisAsync(stuckCutoff, deliveredSince, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
AddSnapshot(
|
||||
samples, capturedAtUtc, KpiScopes.Global, scopeKey: null,
|
||||
global.QueueDepth, global.StuckCount, global.ParkedCount,
|
||||
global.DeliveredLastInterval, global.OldestPendingAge);
|
||||
global.DeliveredLastInterval * scaleFactor, global.OldestPendingAge);
|
||||
|
||||
var perSite = await _repository.ComputePerSiteKpisAsync(stuckCutoff, deliveredSince, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
@@ -90,7 +145,7 @@ public sealed class NotificationOutboxKpiSampleSource : IKpiSampleSource
|
||||
AddSnapshot(
|
||||
samples, capturedAtUtc, KpiScopes.Site, site.SourceSiteId,
|
||||
site.QueueDepth, site.StuckCount, site.ParkedCount,
|
||||
site.DeliveredLastInterval, site.OldestPendingAge);
|
||||
site.DeliveredLastInterval * scaleFactor, site.OldestPendingAge);
|
||||
}
|
||||
|
||||
var perNode = await _repository.ComputePerNodeKpisAsync(stuckCutoff, deliveredSince, cancellationToken)
|
||||
@@ -100,7 +155,7 @@ public sealed class NotificationOutboxKpiSampleSource : IKpiSampleSource
|
||||
AddSnapshot(
|
||||
samples, capturedAtUtc, KpiScopes.Node, node.SourceNode,
|
||||
node.QueueDepth, node.StuckCount, node.ParkedCount,
|
||||
node.DeliveredLastInterval, node.OldestPendingAge);
|
||||
node.DeliveredLastInterval * scaleFactor, node.OldestPendingAge);
|
||||
}
|
||||
|
||||
return samples;
|
||||
@@ -109,6 +164,9 @@ public sealed class NotificationOutboxKpiSampleSource : IKpiSampleSource
|
||||
/// <summary>
|
||||
/// Appends the five outbox metrics for one snapshot at the given scope, omitting
|
||||
/// <c>oldestPendingAgeSeconds</c> when <paramref name="oldestPendingAge"/> is <c>null</c>.
|
||||
/// <paramref name="deliveredLastInterval"/> is already rescaled by
|
||||
/// <see cref="DeliveredIntervalScaleFactor"/> (a <c>double</c>, not the raw repository
|
||||
/// <c>int</c> count) by the caller.
|
||||
/// </summary>
|
||||
private void AddSnapshot(
|
||||
List<KpiSample> samples,
|
||||
@@ -118,7 +176,7 @@ public sealed class NotificationOutboxKpiSampleSource : IKpiSampleSource
|
||||
int queueDepth,
|
||||
int stuckCount,
|
||||
int parkedCount,
|
||||
int deliveredLastInterval,
|
||||
double deliveredLastInterval,
|
||||
TimeSpan? oldestPendingAge)
|
||||
{
|
||||
samples.Add(Sample(capturedAtUtc, MetricQueueDepth, scope, scopeKey, queueDepth));
|
||||
|
||||
Reference in New Issue
Block a user