fix(kpihistory)+fix(notificationoutbox)+fix(sitecallaudit)+fix(host): P5-5 — rate-fold interval triple made self-consistent + cross-validated

Defect: the hourly KPI rollup fold (KpiHistoryRepository.FoldHourlyRollupsAsync) sums every Rate-classified sample recorded in an hour, correct only if each sample is a non-overlapping delta covering exactly one recorder tick (KpiHistoryOptions.SampleInterval). That invariant was an unenforced silent assumption across three independently-owned config keys — SampleInterval, NotificationOutboxOptions.DeliveredKpiWindow, SiteCallAuditOptions.KpiInterval — so widening DeliveredKpiWindow to e.g. 15 min would silently over-count the rollup ~15x. Robust fix: NotificationOutboxKpiSampleSource and SiteCallAuditKpiSampleSource now rescale their Rate-classified samples (deliveredLastInterval, failedLastInterval) by SampleInterval / (DeliveredKpiWindow or KpiInterval) before recording, so the rollup's per-hour Sum is correct regardless of how the other two keys are tuned — the live dashboard tile's own cutoff/window is untouched. Both sources gain a project reference to KpiHistory to read SampleInterval; no cycle (KpiHistory references only Commons). Eager cross-validation (defense-in-depth): KpiRateFoldIntervalCrossValidator (Host, the one composition root referencing all three component projects) fails boot naming all three config keys when they diverge, protecting the separate (still-real) invariant that the recorded/charted KPI-history sample equals the live tile at the same instant — which the rescale trades away whenever the scale factor isn't 1. Rides KpiHistoryOptions' existing ValidateOnStart trigger. Also: NotificationOutboxOptions had no registered validator at all before this (the gap this task's "whichever options class owns DeliveredKpiWindow" file scope pointed at) — added NotificationOutboxOptionsValidator mirroring SiteCallAuditOptionsValidator's positive-duration/count convention, wired with ValidateOnStart. Tests: new rescale tests in both sample-source suites prove the fold is numerically correct at a non-default SampleInterval (30/40 counted over a 15/20-minute window correctly divides down to a 1-minute-tick delta); new KpiRateFoldIntervalCrossValidatorTests prove divergent config fails with all three keys named; new NotificationOutboxOptionsValidatorTests. Every new test proven to fail against the unfixed computation first (temporarily short-circuited the scale factor / cross-check to the pre-fix behavior, confirmed failure, restored) — not via git stash (repo-wide, shared across concurrent worktrees this session) but via file-copy swap. Full suites green: Host.Tests (527), ConfigurationDatabase.Tests (401), KpiHistory.Tests (54), NotificationOutbox.Tests (163), SiteCallAudit.Tests (67).
2026-08-19 16:39:50 -04:00
parent 1cea01d924
commit abbbcd1a8d
13 changed files with 613 additions and 20 deletions
@@ -4,6 +4,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Kpi;
 using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
 using ZB.MOM.WW.ScadaBridge.Commons.Types.Kpi;
 using ZB.MOM.WW.ScadaBridge.Commons.Types.Notifications;
+using ZB.MOM.WW.ScadaBridge.KpiHistory;

 namespace ZB.MOM.WW.ScadaBridge.NotificationOutbox.Kpi;

@@ -20,9 +21,12 @@ namespace ZB.MOM.WW.ScadaBridge.NotificationOutbox.Kpi;
 /// <c>capturedAtUtc</c> rather than wall-clock <c>now</c>: the stuck cutoff is
 /// <c>capturedAtUtc - <see cref="NotificationOutboxOptions.StuckAgeThreshold"/></c> and the
 /// delivered window is <c>capturedAtUtc - <see cref="NotificationOutboxOptions.DeliveredKpiWindow"/></c>.
-/// The COUNT metrics (queueDepth, stuckCount, parkedCount, deliveredLastInterval) equal the
-/// live tile at the same instant; the <c>oldestPendingAgeSeconds</c> metric is computed against
-/// the repository's internal clock and may differ from the live tile by the query-execution latency.
+/// The COUNT metrics (queueDepth, stuckCount, parkedCount) equal the live tile at the same
+/// instant; deliveredLastInterval is rescaled onto <c>KpiHistoryOptions.SampleInterval</c>
+/// before being recorded (see the rescale note below) and equals the live tile only when
+/// that rescale factor is 1, which the boot-time cross-validator guarantees; the
+/// <c>oldestPendingAgeSeconds</c> metric is computed against the repository's internal clock
+/// and may differ from the live tile by the query-execution latency.
 /// </para>
 /// <para>
 /// Emits Global (<c>ScopeKey == null</c>), per-Site (<c>ScopeKey == SourceSiteId</c>), and
@@ -31,6 +35,28 @@ namespace ZB.MOM.WW.ScadaBridge.NotificationOutbox.Kpi;
 /// <c>oldestPendingAgeSeconds</c> via <see cref="TimeSpan.TotalSeconds"/>; that one metric is
 /// omitted when the age is <c>null</c>.
 /// </para>
+/// <para>
+/// <b>deliveredLastInterval rescale (P5-5, arch-review Low).</b>
+/// <see cref="KpiMetricAggregationCatalog"/> classifies this metric as a
+/// <see cref="KpiRollupAggregation.Rate"/>: the central hourly rollup fold sums every
+/// sample recorded in the hour, which is only correct if each sample's value is a
+/// non-overlapping delta covering exactly one recorder tick
+/// (<c>KpiHistoryOptions.SampleInterval</c>). The live cutoff this class queries the
+/// repository with is still anchored on the independently-configurable
+/// <see cref="NotificationOutboxOptions.DeliveredKpiWindow"/> — unchanged, so the live
+/// dashboard tile keeps its own smoothing window — but the COUNT that gets written into
+/// <see cref="KpiSample.Value"/> is rescaled by
+/// <c>SampleInterval / DeliveredKpiWindow</c> before it is recorded, so the rollup's sum
+/// is correct for the hour regardless of how <c>DeliveredKpiWindow</c> is tuned (e.g.
+/// widened to smooth a noisy live tile). At the documented defaults (both 1 minute) the
+/// scale factor is exactly 1 and this is a no-op. A configuration where the scale factor
+/// is not 1 — i.e. where the live tile's window no longer equals the recorder's sampling
+/// cadence — still boots and rolls up correctly, but the recorded/charted
+/// deliveredLastInterval sample then no longer numerically equals the live tile at the
+/// same instant (only the <em>rollup total</em> is guaranteed correct); the boot-time
+/// cross-validator (Host's <c>KpiRateFoldIntervalCrossValidator</c>) enforces the equality
+/// so that divergence can never actually happen in a config that boots.
+/// </para>
 /// </remarks>
 public sealed class NotificationOutboxKpiSampleSource : IKpiSampleSource
 {
@@ -44,23 +70,51 @@ public sealed class NotificationOutboxKpiSampleSource : IKpiSampleSource

    private readonly INotificationOutboxRepository _repository;
    private readonly NotificationOutboxOptions _options;
+    private readonly KpiHistoryOptions _kpiHistoryOptions;

    /// <summary>
    /// Creates the sample source.
    /// </summary>
    /// <param name="repository">Outbox repository providing the KPI computation methods.</param>
    /// <param name="options">Outbox options carrying the stuck-age and delivered-window cutoffs.</param>
+    /// <param name="kpiHistoryOptions">
+    /// KPI History options carrying <see cref="KpiHistoryOptions.SampleInterval"/> — the
+    /// recorder's sampling cadence the deliveredLastInterval sample is rescaled onto (P5-5),
+    /// so the hourly rollup fold's per-sample sum stays correct independent of
+    /// <see cref="NotificationOutboxOptions.DeliveredKpiWindow"/>.
+    /// </param>
    public NotificationOutboxKpiSampleSource(
        INotificationOutboxRepository repository,
-        IOptions<NotificationOutboxOptions> options)
+        IOptions<NotificationOutboxOptions> options,
+        IOptions<KpiHistoryOptions> kpiHistoryOptions)
    {
        ArgumentNullException.ThrowIfNull(repository);
        ArgumentNullException.ThrowIfNull(options);
+        ArgumentNullException.ThrowIfNull(kpiHistoryOptions);

        _repository = repository;
        _options = options.Value;
+        _kpiHistoryOptions = kpiHistoryOptions.Value;
    }

+    /// <summary>
+    /// Rescale factor applied to the raw <c>deliveredLastInterval</c> count before it is
+    /// recorded, so the value written into <see cref="KpiSample.Value"/> represents a
+    /// non-overlapping per-<see cref="KpiHistoryOptions.SampleInterval"/> delta regardless of
+    /// how wide <see cref="NotificationOutboxOptions.DeliveredKpiWindow"/> is configured — the
+    /// hourly rollup fold (<c>KpiHistoryRepository.FoldHourlyRollupsAsync</c>) sums this value
+    /// across the hour's samples, which is only correct for non-overlapping deltas. Exactly 1
+    /// at the documented defaults (both 1 minute). Guards against a non-positive
+    /// <see cref="NotificationOutboxOptions.DeliveredKpiWindow"/> (validated &gt; 0 at boot by
+    /// <see cref="NotificationOutboxOptionsValidator"/>, but this defends the arithmetic even
+    /// if that validation is ever bypassed) by falling back to the unscaled 1:1 factor rather
+    /// than dividing by zero.
+    /// </summary>
+    private double DeliveredIntervalScaleFactor =>
+        _options.DeliveredKpiWindow > TimeSpan.Zero
+            ? _kpiHistoryOptions.SampleInterval.TotalSeconds / _options.DeliveredKpiWindow.TotalSeconds
+            : 1.0;
+
    /// <inheritdoc />
    public string Source => KpiSources.NotificationOutbox;

@@ -75,13 +129,14 @@ public sealed class NotificationOutboxKpiSampleSource : IKpiSampleSource
        var deliveredSince = capturedAt - _options.DeliveredKpiWindow;

        var samples = new List<KpiSample>();
+        var scaleFactor = DeliveredIntervalScaleFactor;

        var global = await _repository.ComputeKpisAsync(stuckCutoff, deliveredSince, cancellationToken)
            .ConfigureAwait(false);
        AddSnapshot(
            samples, capturedAtUtc, KpiScopes.Global, scopeKey: null,
            global.QueueDepth, global.StuckCount, global.ParkedCount,
-            global.DeliveredLastInterval, global.OldestPendingAge);
+            global.DeliveredLastInterval * scaleFactor, global.OldestPendingAge);

        var perSite = await _repository.ComputePerSiteKpisAsync(stuckCutoff, deliveredSince, cancellationToken)
            .ConfigureAwait(false);
@@ -90,7 +145,7 @@ public sealed class NotificationOutboxKpiSampleSource : IKpiSampleSource
            AddSnapshot(
                samples, capturedAtUtc, KpiScopes.Site, site.SourceSiteId,
                site.QueueDepth, site.StuckCount, site.ParkedCount,
-                site.DeliveredLastInterval, site.OldestPendingAge);
+                site.DeliveredLastInterval * scaleFactor, site.OldestPendingAge);
        }

        var perNode = await _repository.ComputePerNodeKpisAsync(stuckCutoff, deliveredSince, cancellationToken)
@@ -100,7 +155,7 @@ public sealed class NotificationOutboxKpiSampleSource : IKpiSampleSource
            AddSnapshot(
                samples, capturedAtUtc, KpiScopes.Node, node.SourceNode,
                node.QueueDepth, node.StuckCount, node.ParkedCount,
-                node.DeliveredLastInterval, node.OldestPendingAge);
+                node.DeliveredLastInterval * scaleFactor, node.OldestPendingAge);
        }

        return samples;
@@ -109,6 +164,9 @@ public sealed class NotificationOutboxKpiSampleSource : IKpiSampleSource
    /// <summary>
    /// Appends the five outbox metrics for one snapshot at the given scope, omitting
    /// <c>oldestPendingAgeSeconds</c> when <paramref name="oldestPendingAge"/> is <c>null</c>.
+    /// <paramref name="deliveredLastInterval"/> is already rescaled by
+    /// <see cref="DeliveredIntervalScaleFactor"/> (a <c>double</c>, not the raw repository
+    /// <c>int</c> count) by the caller.
    /// </summary>
    private void AddSnapshot(
        List<KpiSample> samples,
@@ -118,7 +176,7 @@ public sealed class NotificationOutboxKpiSampleSource : IKpiSampleSource
        int queueDepth,
        int stuckCount,
        int parkedCount,
-        int deliveredLastInterval,
+        double deliveredLastInterval,
        TimeSpan? oldestPendingAge)
    {
        samples.Add(Sample(capturedAtUtc, MetricQueueDepth, scope, scopeKey, queueDepth));
@@ -0,0 +1,58 @@
+using ZB.MOM.WW.Configuration;
+
+namespace ZB.MOM.WW.ScadaBridge.NotificationOutbox;
+
+/// <summary>
+/// Validates <see cref="NotificationOutboxOptions"/> at host startup (arch-review
+/// §1.5 eager-options-validation pattern — this options class had no registered
+/// validator before P5-5). The durations feed the dispatch sweep, stuck-call
+/// detection, terminal-row purge, and the delivered-throughput KPI window; a
+/// zero/negative cadence would spin an Akka scheduler, and a non-positive
+/// <see cref="NotificationOutboxOptions.DeliveredKpiWindow"/> would collapse the
+/// KPI window and — because <see cref="Kpi.NotificationOutboxKpiSampleSource"/>
+/// divides by it to rescale the KPI-history sample onto
+/// <c>KpiHistoryOptions.SampleInterval</c> — divide by zero.
+/// <see cref="NotificationOutboxOptions.MaxParallelDeliveries"/> is deliberately
+/// validated on its BASE property, not the clamped
+/// <see cref="NotificationOutboxOptions.ResolvedMaxParallelDeliveries"/>, so a
+/// misconfigured non-positive value fails fast instead of silently clamping to 1
+/// (mirrors <c>SiteCallAuditOptionsValidator</c>'s cadence-field convention).
+/// </summary>
+public sealed class NotificationOutboxOptionsValidator : OptionsValidatorBase<NotificationOutboxOptions>
+{
+    /// <inheritdoc />
+    protected override void Validate(ValidationBuilder builder, NotificationOutboxOptions options)
+    {
+        builder.RequireThat(options.DispatchInterval > TimeSpan.Zero,
+            $"ScadaBridge:NotificationOutbox:{nameof(NotificationOutboxOptions.DispatchInterval)} " +
+            $"({options.DispatchInterval}) must be a positive duration; it is the dispatch-sweep cadence.");
+
+        builder.RequireThat(options.DispatchBatchSize > 0,
+            $"ScadaBridge:NotificationOutbox:{nameof(NotificationOutboxOptions.DispatchBatchSize)} " +
+            $"({options.DispatchBatchSize}) must be positive; it is the max rows claimed per sweep.");
+
+        builder.RequireThat(options.StuckAgeThreshold > TimeSpan.Zero,
+            $"ScadaBridge:NotificationOutbox:{nameof(NotificationOutboxOptions.StuckAgeThreshold)} " +
+            $"({options.StuckAgeThreshold}) must be a positive duration; it is the age past which a " +
+            "pending/retrying notification is flagged stuck.");
+
+        builder.RequireThat(options.TerminalRetention > TimeSpan.Zero,
+            $"ScadaBridge:NotificationOutbox:{nameof(NotificationOutboxOptions.TerminalRetention)} " +
+            $"({options.TerminalRetention}) must be a positive duration; it is the terminal-row retention window.");
+
+        builder.RequireThat(options.PurgeInterval > TimeSpan.Zero,
+            $"ScadaBridge:NotificationOutbox:{nameof(NotificationOutboxOptions.PurgeInterval)} " +
+            $"({options.PurgeInterval}) must be a positive duration; it is the terminal-row purge-sweep cadence.");
+
+        builder.RequireThat(options.DeliveredKpiWindow > TimeSpan.Zero,
+            $"ScadaBridge:NotificationOutbox:{nameof(NotificationOutboxOptions.DeliveredKpiWindow)} " +
+            $"({options.DeliveredKpiWindow}) must be a positive duration; it is the trailing window for the " +
+            "delivered-throughput KPI and the divisor the KPI-history sample source rescales onto " +
+            "KpiHistoryOptions.SampleInterval.");
+
+        builder.RequireThat(options.MaxParallelDeliveries > 0,
+            $"ScadaBridge:NotificationOutbox:{nameof(NotificationOutboxOptions.MaxParallelDeliveries)} " +
+            $"({options.MaxParallelDeliveries}) must be positive; a non-positive value silently clamps to 1 " +
+            $"via {nameof(NotificationOutboxOptions.ResolvedMaxParallelDeliveries)}, hiding the misconfiguration.");
+    }
+}
@@ -56,8 +56,16 @@ public static class ServiceCollectionExtensions
    {
        ArgumentNullException.ThrowIfNull(services);

+        // Eager startup validation (arch-review §1.5 / P5-5): this options class had no
+        // registered validator before — ValidateOnStart makes a bad
+        // ScadaBridge:NotificationOutbox section fail fast at boot with a clear,
+        // key-naming message instead of spinning a scheduler or, for DeliveredKpiWindow,
+        // dividing by zero in NotificationOutboxKpiSampleSource's KpiHistory rescale.
        services.AddOptions<NotificationOutboxOptions>()
-            .BindConfiguration(OptionsSection);
+            .BindConfiguration(OptionsSection)
+            .ValidateOnStart();
+        services.TryAddEnumerable(
+            ServiceDescriptor.Singleton<IValidateOptions<NotificationOutboxOptions>, NotificationOutboxOptionsValidator>());

        // Bind + validate SmsOptions (ScadaBridge:Sms): the SMS adapter's body cap and
        // per-request timeout fallback. ValidateOnStart surfaces a bad option at host
@@ -21,6 +21,11 @@
         (ISmtpClientWrapper, SmtpPermanentException, SmtpTlsModeParser,
         OAuth2TokenService). -->
    <ProjectReference Include="../ZB.MOM.WW.ScadaBridge.NotificationService/ZB.MOM.WW.ScadaBridge.NotificationService.csproj" />
+    <!-- The KPI sample source computes deliveredLastInterval against
+         KpiHistoryOptions.SampleInterval directly (P5-5), not the independently
+         configurable DeliveredKpiWindow, so the central hourly rollup fold's
+         per-sample sum is correct regardless of how DeliveredKpiWindow is tuned. -->
+    <ProjectReference Include="../ZB.MOM.WW.ScadaBridge.KpiHistory/ZB.MOM.WW.ScadaBridge.KpiHistory.csproj" />
  </ItemGroup>

  <ItemGroup>