fix(kpihistory)+fix(notificationoutbox)+fix(sitecallaudit)+fix(host): P5-5 — rate-fold interval triple made self-consistent + cross-validated
Defect: the hourly KPI rollup fold (KpiHistoryRepository.FoldHourlyRollupsAsync) sums every Rate-classified sample recorded in an hour, correct only if each sample is a non-overlapping delta covering exactly one recorder tick (KpiHistoryOptions.SampleInterval). That invariant was an unenforced silent assumption across three independently-owned config keys — SampleInterval, NotificationOutboxOptions.DeliveredKpiWindow, SiteCallAuditOptions.KpiInterval — so widening DeliveredKpiWindow to e.g. 15 min would silently over-count the rollup ~15x. Robust fix: NotificationOutboxKpiSampleSource and SiteCallAuditKpiSampleSource now rescale their Rate-classified samples (deliveredLastInterval, failedLastInterval) by SampleInterval / (DeliveredKpiWindow or KpiInterval) before recording, so the rollup's per-hour Sum is correct regardless of how the other two keys are tuned — the live dashboard tile's own cutoff/window is untouched. Both sources gain a project reference to KpiHistory to read SampleInterval; no cycle (KpiHistory references only Commons). Eager cross-validation (defense-in-depth): KpiRateFoldIntervalCrossValidator (Host, the one composition root referencing all three component projects) fails boot naming all three config keys when they diverge, protecting the separate (still-real) invariant that the recorded/charted KPI-history sample equals the live tile at the same instant — which the rescale trades away whenever the scale factor isn't 1. Rides KpiHistoryOptions' existing ValidateOnStart trigger. Also: NotificationOutboxOptions had no registered validator at all before this (the gap this task's "whichever options class owns DeliveredKpiWindow" file scope pointed at) — added NotificationOutboxOptionsValidator mirroring SiteCallAuditOptionsValidator's positive-duration/count convention, wired with ValidateOnStart. Tests: new rescale tests in both sample-source suites prove the fold is numerically correct at a non-default SampleInterval (30/40 counted over a 15/20-minute window correctly divides down to a 1-minute-tick delta); new KpiRateFoldIntervalCrossValidatorTests prove divergent config fails with all three keys named; new NotificationOutboxOptionsValidatorTests. Every new test proven to fail against the unfixed computation first (temporarily short-circuited the scale factor / cross-check to the pre-fix behavior, confirmed failure, restored) — not via git stash (repo-wide, shared across concurrent worktrees this session) but via file-copy swap. Full suites green: Host.Tests (527), ConfigurationDatabase.Tests (401), KpiHistory.Tests (54), NotificationOutbox.Tests (163), SiteCallAudit.Tests (67).
This commit is contained in:
@@ -166,6 +166,19 @@ public sealed class KpiHistoryRepository : IKpiHistoryRepository
|
||||
// Per-metric aggregation intent: Rate metrics sum the hour (last-value would
|
||||
// discard 59 of 60 intra-hour deltas); Gauge metrics keep the last (latest
|
||||
// timestamp) value in the hour. Min/Max/Count preserve the fold's fidelity.
|
||||
//
|
||||
// This Sum is only correct if every sample in the group is a non-overlapping
|
||||
// delta covering exactly one recorder tick (KpiHistoryOptions.SampleInterval).
|
||||
// That invariant used to be an unenforced assumption across three
|
||||
// independently-owned config keys (KpiHistoryOptions.SampleInterval,
|
||||
// NotificationOutboxOptions.DeliveredKpiWindow, SiteCallAuditOptions.KpiInterval)
|
||||
// — widening one silently over/under-counted the rollup (arch-review P5-5, Low).
|
||||
// As of P5-5, NotificationOutboxKpiSampleSource / SiteCallAuditKpiSampleSource
|
||||
// rescale their Rate-classified samples onto SampleInterval before recording, so
|
||||
// this Sum is correct here regardless of how the other two keys are tuned; a
|
||||
// boot-time cross-validator (Host's KpiRateFoldIntervalCrossValidator) still
|
||||
// enforces all three being equal so the recorded/charted samples never silently
|
||||
// diverge from what the live dashboard tile shows at the same instant.
|
||||
var aggregation = KpiMetricAggregationCatalog.Resolve(key.Source, key.Metric);
|
||||
var value = aggregation == KpiRollupAggregation.Rate
|
||||
? group.Sum(s => s.Value)
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.Configuration;
|
||||
using ZB.MOM.WW.ScadaBridge.KpiHistory;
|
||||
using ZB.MOM.WW.ScadaBridge.NotificationOutbox;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteCallAudit;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Host;
|
||||
|
||||
/// <summary>
|
||||
/// Boot-time cross-validation of the "rate-fold interval triple" (arch-review P5-5,
|
||||
/// Low). <see cref="KpiHistoryOptions.SampleInterval"/>,
|
||||
/// <see cref="NotificationOutboxOptions.DeliveredKpiWindow"/>, and
|
||||
/// <see cref="SiteCallAuditOptions.KpiInterval"/> are three independently-owned config
|
||||
/// keys, bound in three different component projects with no compile-time relationship
|
||||
/// to one another.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>What no longer depends on this equality.</b> <c>NotificationOutboxKpiSampleSource</c>
|
||||
/// and <c>SiteCallAuditKpiSampleSource</c> now rescale their Rate-classified KPI-history
|
||||
/// samples (deliveredLastInterval, failedLastInterval) by
|
||||
/// <c>SampleInterval / (DeliveredKpiWindow or KpiInterval)</c> before recording them, so
|
||||
/// <c>KpiHistoryRepository.FoldHourlyRollupsAsync</c>'s per-hour <c>Sum</c> over those
|
||||
/// samples is correct for <em>any</em> value of the other two keys — a widened
|
||||
/// <c>DeliveredKpiWindow</c> can no longer silently over-count the hourly rollup the way
|
||||
/// it did before P5-5.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>What still depends on it.</b> The rescale trades exact per-sample fidelity for
|
||||
/// rollup correctness: whenever the scale factor is not 1, the recorded/charted
|
||||
/// deliveredLastInterval/failedLastInterval sample no longer numerically equals what the
|
||||
/// live Health-dashboard tile shows at the same instant (the live tile computation is
|
||||
/// untouched and still uses the wider/narrower window directly) — a documented invariant
|
||||
/// both sample sources' XML docs otherwise promise. This validator keeps that promise true
|
||||
/// by construction: it fails the host at boot, naming all three configuration keys, unless
|
||||
/// they are all equal (scale factor exactly 1 on both sides).
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Registered against <c>IValidateOptions<KpiHistoryOptions></c> — that options type
|
||||
/// already calls <c>ValidateOnStart()</c> (see <c>KpiHistory.ServiceCollectionExtensions</c>),
|
||||
/// so this cross-validator rides the same eager trigger without either
|
||||
/// <see cref="NotificationOutboxOptions"/> or <see cref="SiteCallAuditOptions"/> needing
|
||||
/// their own <c>ValidateOnStart()</c> call for this specific check to fire. It lives in Host
|
||||
/// — the one composition root that references all three component projects — rather than in
|
||||
/// any of the three (which deliberately do not reference each other).
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class KpiRateFoldIntervalCrossValidator : OptionsValidatorBase<KpiHistoryOptions>
|
||||
{
|
||||
private readonly IOptions<NotificationOutboxOptions> _notificationOutboxOptions;
|
||||
private readonly IOptions<SiteCallAuditOptions> _siteCallAuditOptions;
|
||||
|
||||
/// <summary>
|
||||
/// Creates the validator with the two sibling options it cross-checks against
|
||||
/// <see cref="KpiHistoryOptions"/>.
|
||||
/// </summary>
|
||||
public KpiRateFoldIntervalCrossValidator(
|
||||
IOptions<NotificationOutboxOptions> notificationOutboxOptions,
|
||||
IOptions<SiteCallAuditOptions> siteCallAuditOptions)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(notificationOutboxOptions);
|
||||
ArgumentNullException.ThrowIfNull(siteCallAuditOptions);
|
||||
|
||||
_notificationOutboxOptions = notificationOutboxOptions;
|
||||
_siteCallAuditOptions = siteCallAuditOptions;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void Validate(ValidationBuilder builder, KpiHistoryOptions options)
|
||||
{
|
||||
var sampleInterval = options.SampleInterval;
|
||||
var deliveredKpiWindow = _notificationOutboxOptions.Value.DeliveredKpiWindow;
|
||||
var kpiInterval = _siteCallAuditOptions.Value.KpiInterval;
|
||||
|
||||
builder.RequireThat(
|
||||
deliveredKpiWindow == sampleInterval && kpiInterval == sampleInterval,
|
||||
"KPI rate-fold interval mismatch: " +
|
||||
$"ScadaBridge:KpiHistory:{nameof(KpiHistoryOptions.SampleInterval)} ({sampleInterval}), " +
|
||||
$"ScadaBridge:NotificationOutbox:{nameof(NotificationOutboxOptions.DeliveredKpiWindow)} " +
|
||||
$"({deliveredKpiWindow}), and " +
|
||||
$"ScadaBridge:SiteCallAudit:{nameof(SiteCallAuditOptions.KpiInterval)} ({kpiInterval}) " +
|
||||
"must all be equal. The KPI-history sample sources rescale their delivered/failed-last-interval " +
|
||||
"counts onto SampleInterval before recording so the hourly rollup fold's sum stays correct " +
|
||||
"regardless of this divergence, but a mismatch here still means the recorded/charted KPI-history " +
|
||||
"values would silently stop matching the live Health-dashboard tile at the same instant.");
|
||||
}
|
||||
}
|
||||
@@ -346,6 +346,12 @@ try
|
||||
// the KpiHistoryRecorderActor cluster singleton (started in
|
||||
// AkkaHostedService). Observability/best-effort: NOT readiness-gated.
|
||||
builder.Services.AddKpiHistory(builder.Configuration);
|
||||
// P5-5 (arch-review Low): cross-validates the rate-fold interval triple
|
||||
// (KpiHistoryOptions.SampleInterval, NotificationOutboxOptions.DeliveredKpiWindow,
|
||||
// SiteCallAuditOptions.KpiInterval) at boot. Rides KpiHistoryOptions' existing
|
||||
// ValidateOnStart trigger set by AddKpiHistory above.
|
||||
builder.Services.TryAddEnumerable(
|
||||
ServiceDescriptor.Singleton<IValidateOptions<KpiHistoryOptions>, KpiRateFoldIntervalCrossValidator>());
|
||||
// The Audit Log's IKpiSampleSource, enumerated by the recorder registered just
|
||||
// above. Central-only (its IAuditLogRepository is central-only); split out of
|
||||
// AddAuditLog by deferred register row 50 for the same reason as
|
||||
|
||||
+66
-8
@@ -4,6 +4,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Kpi;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Kpi;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Notifications;
|
||||
using ZB.MOM.WW.ScadaBridge.KpiHistory;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.NotificationOutbox.Kpi;
|
||||
|
||||
@@ -20,9 +21,12 @@ namespace ZB.MOM.WW.ScadaBridge.NotificationOutbox.Kpi;
|
||||
/// <c>capturedAtUtc</c> rather than wall-clock <c>now</c>: the stuck cutoff is
|
||||
/// <c>capturedAtUtc - <see cref="NotificationOutboxOptions.StuckAgeThreshold"/></c> and the
|
||||
/// delivered window is <c>capturedAtUtc - <see cref="NotificationOutboxOptions.DeliveredKpiWindow"/></c>.
|
||||
/// The COUNT metrics (queueDepth, stuckCount, parkedCount, deliveredLastInterval) equal the
|
||||
/// live tile at the same instant; the <c>oldestPendingAgeSeconds</c> metric is computed against
|
||||
/// the repository's internal clock and may differ from the live tile by the query-execution latency.
|
||||
/// The COUNT metrics (queueDepth, stuckCount, parkedCount) equal the live tile at the same
|
||||
/// instant; deliveredLastInterval is rescaled onto <c>KpiHistoryOptions.SampleInterval</c>
|
||||
/// before being recorded (see the rescale note below) and equals the live tile only when
|
||||
/// that rescale factor is 1, which the boot-time cross-validator guarantees; the
|
||||
/// <c>oldestPendingAgeSeconds</c> metric is computed against the repository's internal clock
|
||||
/// and may differ from the live tile by the query-execution latency.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Emits Global (<c>ScopeKey == null</c>), per-Site (<c>ScopeKey == SourceSiteId</c>), and
|
||||
@@ -31,6 +35,28 @@ namespace ZB.MOM.WW.ScadaBridge.NotificationOutbox.Kpi;
|
||||
/// <c>oldestPendingAgeSeconds</c> via <see cref="TimeSpan.TotalSeconds"/>; that one metric is
|
||||
/// omitted when the age is <c>null</c>.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>deliveredLastInterval rescale (P5-5, arch-review Low).</b>
|
||||
/// <see cref="KpiMetricAggregationCatalog"/> classifies this metric as a
|
||||
/// <see cref="KpiRollupAggregation.Rate"/>: the central hourly rollup fold sums every
|
||||
/// sample recorded in the hour, which is only correct if each sample's value is a
|
||||
/// non-overlapping delta covering exactly one recorder tick
|
||||
/// (<c>KpiHistoryOptions.SampleInterval</c>). The live cutoff this class queries the
|
||||
/// repository with is still anchored on the independently-configurable
|
||||
/// <see cref="NotificationOutboxOptions.DeliveredKpiWindow"/> — unchanged, so the live
|
||||
/// dashboard tile keeps its own smoothing window — but the COUNT that gets written into
|
||||
/// <see cref="KpiSample.Value"/> is rescaled by
|
||||
/// <c>SampleInterval / DeliveredKpiWindow</c> before it is recorded, so the rollup's sum
|
||||
/// is correct for the hour regardless of how <c>DeliveredKpiWindow</c> is tuned (e.g.
|
||||
/// widened to smooth a noisy live tile). At the documented defaults (both 1 minute) the
|
||||
/// scale factor is exactly 1 and this is a no-op. A configuration where the scale factor
|
||||
/// is not 1 — i.e. where the live tile's window no longer equals the recorder's sampling
|
||||
/// cadence — still boots and rolls up correctly, but the recorded/charted
|
||||
/// deliveredLastInterval sample then no longer numerically equals the live tile at the
|
||||
/// same instant (only the <em>rollup total</em> is guaranteed correct); the boot-time
|
||||
/// cross-validator (Host's <c>KpiRateFoldIntervalCrossValidator</c>) enforces the equality
|
||||
/// so that divergence can never actually happen in a config that boots.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class NotificationOutboxKpiSampleSource : IKpiSampleSource
|
||||
{
|
||||
@@ -44,23 +70,51 @@ public sealed class NotificationOutboxKpiSampleSource : IKpiSampleSource
|
||||
|
||||
private readonly INotificationOutboxRepository _repository;
|
||||
private readonly NotificationOutboxOptions _options;
|
||||
private readonly KpiHistoryOptions _kpiHistoryOptions;
|
||||
|
||||
/// <summary>
|
||||
/// Creates the sample source.
|
||||
/// </summary>
|
||||
/// <param name="repository">Outbox repository providing the KPI computation methods.</param>
|
||||
/// <param name="options">Outbox options carrying the stuck-age and delivered-window cutoffs.</param>
|
||||
/// <param name="kpiHistoryOptions">
|
||||
/// KPI History options carrying <see cref="KpiHistoryOptions.SampleInterval"/> — the
|
||||
/// recorder's sampling cadence the deliveredLastInterval sample is rescaled onto (P5-5),
|
||||
/// so the hourly rollup fold's per-sample sum stays correct independent of
|
||||
/// <see cref="NotificationOutboxOptions.DeliveredKpiWindow"/>.
|
||||
/// </param>
|
||||
public NotificationOutboxKpiSampleSource(
|
||||
INotificationOutboxRepository repository,
|
||||
IOptions<NotificationOutboxOptions> options)
|
||||
IOptions<NotificationOutboxOptions> options,
|
||||
IOptions<KpiHistoryOptions> kpiHistoryOptions)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(repository);
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
ArgumentNullException.ThrowIfNull(kpiHistoryOptions);
|
||||
|
||||
_repository = repository;
|
||||
_options = options.Value;
|
||||
_kpiHistoryOptions = kpiHistoryOptions.Value;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Rescale factor applied to the raw <c>deliveredLastInterval</c> count before it is
|
||||
/// recorded, so the value written into <see cref="KpiSample.Value"/> represents a
|
||||
/// non-overlapping per-<see cref="KpiHistoryOptions.SampleInterval"/> delta regardless of
|
||||
/// how wide <see cref="NotificationOutboxOptions.DeliveredKpiWindow"/> is configured — the
|
||||
/// hourly rollup fold (<c>KpiHistoryRepository.FoldHourlyRollupsAsync</c>) sums this value
|
||||
/// across the hour's samples, which is only correct for non-overlapping deltas. Exactly 1
|
||||
/// at the documented defaults (both 1 minute). Guards against a non-positive
|
||||
/// <see cref="NotificationOutboxOptions.DeliveredKpiWindow"/> (validated > 0 at boot by
|
||||
/// <see cref="NotificationOutboxOptionsValidator"/>, but this defends the arithmetic even
|
||||
/// if that validation is ever bypassed) by falling back to the unscaled 1:1 factor rather
|
||||
/// than dividing by zero.
|
||||
/// </summary>
|
||||
private double DeliveredIntervalScaleFactor =>
|
||||
_options.DeliveredKpiWindow > TimeSpan.Zero
|
||||
? _kpiHistoryOptions.SampleInterval.TotalSeconds / _options.DeliveredKpiWindow.TotalSeconds
|
||||
: 1.0;
|
||||
|
||||
/// <inheritdoc />
|
||||
public string Source => KpiSources.NotificationOutbox;
|
||||
|
||||
@@ -75,13 +129,14 @@ public sealed class NotificationOutboxKpiSampleSource : IKpiSampleSource
|
||||
var deliveredSince = capturedAt - _options.DeliveredKpiWindow;
|
||||
|
||||
var samples = new List<KpiSample>();
|
||||
var scaleFactor = DeliveredIntervalScaleFactor;
|
||||
|
||||
var global = await _repository.ComputeKpisAsync(stuckCutoff, deliveredSince, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
AddSnapshot(
|
||||
samples, capturedAtUtc, KpiScopes.Global, scopeKey: null,
|
||||
global.QueueDepth, global.StuckCount, global.ParkedCount,
|
||||
global.DeliveredLastInterval, global.OldestPendingAge);
|
||||
global.DeliveredLastInterval * scaleFactor, global.OldestPendingAge);
|
||||
|
||||
var perSite = await _repository.ComputePerSiteKpisAsync(stuckCutoff, deliveredSince, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
@@ -90,7 +145,7 @@ public sealed class NotificationOutboxKpiSampleSource : IKpiSampleSource
|
||||
AddSnapshot(
|
||||
samples, capturedAtUtc, KpiScopes.Site, site.SourceSiteId,
|
||||
site.QueueDepth, site.StuckCount, site.ParkedCount,
|
||||
site.DeliveredLastInterval, site.OldestPendingAge);
|
||||
site.DeliveredLastInterval * scaleFactor, site.OldestPendingAge);
|
||||
}
|
||||
|
||||
var perNode = await _repository.ComputePerNodeKpisAsync(stuckCutoff, deliveredSince, cancellationToken)
|
||||
@@ -100,7 +155,7 @@ public sealed class NotificationOutboxKpiSampleSource : IKpiSampleSource
|
||||
AddSnapshot(
|
||||
samples, capturedAtUtc, KpiScopes.Node, node.SourceNode,
|
||||
node.QueueDepth, node.StuckCount, node.ParkedCount,
|
||||
node.DeliveredLastInterval, node.OldestPendingAge);
|
||||
node.DeliveredLastInterval * scaleFactor, node.OldestPendingAge);
|
||||
}
|
||||
|
||||
return samples;
|
||||
@@ -109,6 +164,9 @@ public sealed class NotificationOutboxKpiSampleSource : IKpiSampleSource
|
||||
/// <summary>
|
||||
/// Appends the five outbox metrics for one snapshot at the given scope, omitting
|
||||
/// <c>oldestPendingAgeSeconds</c> when <paramref name="oldestPendingAge"/> is <c>null</c>.
|
||||
/// <paramref name="deliveredLastInterval"/> is already rescaled by
|
||||
/// <see cref="DeliveredIntervalScaleFactor"/> (a <c>double</c>, not the raw repository
|
||||
/// <c>int</c> count) by the caller.
|
||||
/// </summary>
|
||||
private void AddSnapshot(
|
||||
List<KpiSample> samples,
|
||||
@@ -118,7 +176,7 @@ public sealed class NotificationOutboxKpiSampleSource : IKpiSampleSource
|
||||
int queueDepth,
|
||||
int stuckCount,
|
||||
int parkedCount,
|
||||
int deliveredLastInterval,
|
||||
double deliveredLastInterval,
|
||||
TimeSpan? oldestPendingAge)
|
||||
{
|
||||
samples.Add(Sample(capturedAtUtc, MetricQueueDepth, scope, scopeKey, queueDepth));
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
using ZB.MOM.WW.Configuration;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.NotificationOutbox;
|
||||
|
||||
/// <summary>
|
||||
/// Validates <see cref="NotificationOutboxOptions"/> at host startup (arch-review
|
||||
/// §1.5 eager-options-validation pattern — this options class had no registered
|
||||
/// validator before P5-5). The durations feed the dispatch sweep, stuck-call
|
||||
/// detection, terminal-row purge, and the delivered-throughput KPI window; a
|
||||
/// zero/negative cadence would spin an Akka scheduler, and a non-positive
|
||||
/// <see cref="NotificationOutboxOptions.DeliveredKpiWindow"/> would collapse the
|
||||
/// KPI window and — because <see cref="Kpi.NotificationOutboxKpiSampleSource"/>
|
||||
/// divides by it to rescale the KPI-history sample onto
|
||||
/// <c>KpiHistoryOptions.SampleInterval</c> — divide by zero.
|
||||
/// <see cref="NotificationOutboxOptions.MaxParallelDeliveries"/> is deliberately
|
||||
/// validated on its BASE property, not the clamped
|
||||
/// <see cref="NotificationOutboxOptions.ResolvedMaxParallelDeliveries"/>, so a
|
||||
/// misconfigured non-positive value fails fast instead of silently clamping to 1
|
||||
/// (mirrors <c>SiteCallAuditOptionsValidator</c>'s cadence-field convention).
|
||||
/// </summary>
|
||||
public sealed class NotificationOutboxOptionsValidator : OptionsValidatorBase<NotificationOutboxOptions>
|
||||
{
|
||||
/// <inheritdoc />
|
||||
protected override void Validate(ValidationBuilder builder, NotificationOutboxOptions options)
|
||||
{
|
||||
builder.RequireThat(options.DispatchInterval > TimeSpan.Zero,
|
||||
$"ScadaBridge:NotificationOutbox:{nameof(NotificationOutboxOptions.DispatchInterval)} " +
|
||||
$"({options.DispatchInterval}) must be a positive duration; it is the dispatch-sweep cadence.");
|
||||
|
||||
builder.RequireThat(options.DispatchBatchSize > 0,
|
||||
$"ScadaBridge:NotificationOutbox:{nameof(NotificationOutboxOptions.DispatchBatchSize)} " +
|
||||
$"({options.DispatchBatchSize}) must be positive; it is the max rows claimed per sweep.");
|
||||
|
||||
builder.RequireThat(options.StuckAgeThreshold > TimeSpan.Zero,
|
||||
$"ScadaBridge:NotificationOutbox:{nameof(NotificationOutboxOptions.StuckAgeThreshold)} " +
|
||||
$"({options.StuckAgeThreshold}) must be a positive duration; it is the age past which a " +
|
||||
"pending/retrying notification is flagged stuck.");
|
||||
|
||||
builder.RequireThat(options.TerminalRetention > TimeSpan.Zero,
|
||||
$"ScadaBridge:NotificationOutbox:{nameof(NotificationOutboxOptions.TerminalRetention)} " +
|
||||
$"({options.TerminalRetention}) must be a positive duration; it is the terminal-row retention window.");
|
||||
|
||||
builder.RequireThat(options.PurgeInterval > TimeSpan.Zero,
|
||||
$"ScadaBridge:NotificationOutbox:{nameof(NotificationOutboxOptions.PurgeInterval)} " +
|
||||
$"({options.PurgeInterval}) must be a positive duration; it is the terminal-row purge-sweep cadence.");
|
||||
|
||||
builder.RequireThat(options.DeliveredKpiWindow > TimeSpan.Zero,
|
||||
$"ScadaBridge:NotificationOutbox:{nameof(NotificationOutboxOptions.DeliveredKpiWindow)} " +
|
||||
$"({options.DeliveredKpiWindow}) must be a positive duration; it is the trailing window for the " +
|
||||
"delivered-throughput KPI and the divisor the KPI-history sample source rescales onto " +
|
||||
"KpiHistoryOptions.SampleInterval.");
|
||||
|
||||
builder.RequireThat(options.MaxParallelDeliveries > 0,
|
||||
$"ScadaBridge:NotificationOutbox:{nameof(NotificationOutboxOptions.MaxParallelDeliveries)} " +
|
||||
$"({options.MaxParallelDeliveries}) must be positive; a non-positive value silently clamps to 1 " +
|
||||
$"via {nameof(NotificationOutboxOptions.ResolvedMaxParallelDeliveries)}, hiding the misconfiguration.");
|
||||
}
|
||||
}
|
||||
@@ -56,8 +56,16 @@ public static class ServiceCollectionExtensions
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
|
||||
// Eager startup validation (arch-review §1.5 / P5-5): this options class had no
|
||||
// registered validator before — ValidateOnStart makes a bad
|
||||
// ScadaBridge:NotificationOutbox section fail fast at boot with a clear,
|
||||
// key-naming message instead of spinning a scheduler or, for DeliveredKpiWindow,
|
||||
// dividing by zero in NotificationOutboxKpiSampleSource's KpiHistory rescale.
|
||||
services.AddOptions<NotificationOutboxOptions>()
|
||||
.BindConfiguration(OptionsSection);
|
||||
.BindConfiguration(OptionsSection)
|
||||
.ValidateOnStart();
|
||||
services.TryAddEnumerable(
|
||||
ServiceDescriptor.Singleton<IValidateOptions<NotificationOutboxOptions>, NotificationOutboxOptionsValidator>());
|
||||
|
||||
// Bind + validate SmsOptions (ScadaBridge:Sms): the SMS adapter's body cap and
|
||||
// per-request timeout fallback. ValidateOnStart surfaces a bad option at host
|
||||
|
||||
+5
@@ -21,6 +21,11 @@
|
||||
(ISmtpClientWrapper, SmtpPermanentException, SmtpTlsModeParser,
|
||||
OAuth2TokenService). -->
|
||||
<ProjectReference Include="../ZB.MOM.WW.ScadaBridge.NotificationService/ZB.MOM.WW.ScadaBridge.NotificationService.csproj" />
|
||||
<!-- The KPI sample source computes deliveredLastInterval against
|
||||
KpiHistoryOptions.SampleInterval directly (P5-5), not the independently
|
||||
configurable DeliveredKpiWindow, so the central hourly rollup fold's
|
||||
per-sample sum is correct regardless of how DeliveredKpiWindow is tuned. -->
|
||||
<ProjectReference Include="../ZB.MOM.WW.ScadaBridge.KpiHistory/ZB.MOM.WW.ScadaBridge.KpiHistory.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
|
||||
@@ -4,6 +4,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Kpi;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Kpi;
|
||||
using ZB.MOM.WW.ScadaBridge.KpiHistory;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.SiteCallAudit.Kpi;
|
||||
|
||||
@@ -21,8 +22,10 @@ namespace ZB.MOM.WW.ScadaBridge.SiteCallAudit.Kpi;
|
||||
/// live <c>SiteCallAuditActor</c> KPI handlers derive them
|
||||
/// (<c>stuckCutoff = capturedAtUtc - StuckAgeThreshold</c>,
|
||||
/// <c>intervalSince = capturedAtUtc - KpiInterval</c>). The COUNT metrics
|
||||
/// (buffered, parked, failedLastInterval, deliveredLastInterval, stuck) equal the
|
||||
/// live tile at the same instant; the <c>oldestPendingAgeSeconds</c> metric is
|
||||
/// (buffered, parked, stuck) equal the live tile at the same instant;
|
||||
/// failedLastInterval and deliveredLastInterval are rescaled before recording (see
|
||||
/// below) and equal the live tile only when that rescale factor is 1, which the
|
||||
/// boot-time cross-validator guarantees; the <c>oldestPendingAgeSeconds</c> metric is
|
||||
/// computed against the repository's internal clock and may differ from the live
|
||||
/// tile by the query-execution latency. The recorder's <c>capturedAtUtc</c> is
|
||||
/// the single anchor for both cutoffs.
|
||||
@@ -32,6 +35,25 @@ namespace ZB.MOM.WW.ScadaBridge.SiteCallAudit.Kpi;
|
||||
/// each sampling pass resolves a fresh repository scope, mirroring the actor's
|
||||
/// scope-per-message repository access.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>failedLastInterval / deliveredLastInterval rescale (P5-5, arch-review Low).</b>
|
||||
/// <see cref="KpiMetricAggregationCatalog"/> classifies both as
|
||||
/// <see cref="KpiRollupAggregation.Rate"/>: the central hourly rollup fold sums every
|
||||
/// sample recorded in the hour, which is only correct if each sample's value is a
|
||||
/// non-overlapping delta covering exactly one recorder tick
|
||||
/// (<c>KpiHistoryOptions.SampleInterval</c>). The live cutoff this class queries the
|
||||
/// repository with is still anchored on the independently-configurable
|
||||
/// <see cref="SiteCallAuditOptions.KpiInterval"/> — unchanged, so the live dashboard
|
||||
/// tile keeps its own smoothing window — but the COUNTS that get written into
|
||||
/// <see cref="KpiSample.Value"/> are rescaled by <c>SampleInterval / KpiInterval</c>
|
||||
/// before they are recorded, so the rollup's sum is correct for the hour regardless
|
||||
/// of how <c>KpiInterval</c> is tuned. At the documented defaults (both 1 minute) the
|
||||
/// scale factor is exactly 1 and this is a no-op. The boot-time cross-validator
|
||||
/// (Host's <c>KpiRateFoldIntervalCrossValidator</c>) enforces
|
||||
/// <c>KpiInterval == SampleInterval</c> (alongside the outbox's
|
||||
/// <c>DeliveredKpiWindow</c>) so a config where the recorded/charted samples no
|
||||
/// longer equal the live tile can never actually boot.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class SiteCallAuditKpiSampleSource : IKpiSampleSource
|
||||
{
|
||||
@@ -48,6 +70,7 @@ public sealed class SiteCallAuditKpiSampleSource : IKpiSampleSource
|
||||
|
||||
private readonly ISiteCallAuditRepository _repository;
|
||||
private readonly SiteCallAuditOptions _options;
|
||||
private readonly KpiHistoryOptions _kpiHistoryOptions;
|
||||
|
||||
/// <summary>
|
||||
/// Creates the sample source over the central <c>SiteCalls</c> repository and
|
||||
@@ -56,17 +79,43 @@ public sealed class SiteCallAuditKpiSampleSource : IKpiSampleSource
|
||||
/// </summary>
|
||||
/// <param name="repository">The central <c>SiteCalls</c> operational-state repository.</param>
|
||||
/// <param name="options">Site Call Audit windowing options (stuck-age + KPI interval).</param>
|
||||
/// <param name="kpiHistoryOptions">
|
||||
/// KPI History options carrying <see cref="KpiHistoryOptions.SampleInterval"/> — the
|
||||
/// recorder's sampling cadence the failedLastInterval/deliveredLastInterval samples are
|
||||
/// rescaled onto (P5-5), so the hourly rollup fold's per-sample sum stays correct
|
||||
/// independent of <see cref="SiteCallAuditOptions.KpiInterval"/>.
|
||||
/// </param>
|
||||
public SiteCallAuditKpiSampleSource(
|
||||
ISiteCallAuditRepository repository,
|
||||
IOptions<SiteCallAuditOptions> options)
|
||||
IOptions<SiteCallAuditOptions> options,
|
||||
IOptions<KpiHistoryOptions> kpiHistoryOptions)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(repository);
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
ArgumentNullException.ThrowIfNull(kpiHistoryOptions);
|
||||
|
||||
_repository = repository;
|
||||
_options = options.Value;
|
||||
_kpiHistoryOptions = kpiHistoryOptions.Value;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Rescale factor applied to the raw <c>failedLastInterval</c>/<c>deliveredLastInterval</c>
|
||||
/// counts before they are recorded, so the values written into
|
||||
/// <see cref="KpiSample.Value"/> represent non-overlapping
|
||||
/// per-<see cref="KpiHistoryOptions.SampleInterval"/> deltas regardless of how wide
|
||||
/// <see cref="SiteCallAuditOptions.KpiInterval"/> is configured. Exactly 1 at the
|
||||
/// documented defaults (both 1 minute). Guards against a non-positive
|
||||
/// <see cref="SiteCallAuditOptions.KpiInterval"/> (validated > 0 at boot by
|
||||
/// <see cref="SiteCallAuditOptionsValidator"/>, but this defends the arithmetic even if
|
||||
/// that validation is ever bypassed) by falling back to the unscaled 1:1 factor rather
|
||||
/// than dividing by zero.
|
||||
/// </summary>
|
||||
private double IntervalScaleFactor =>
|
||||
_options.KpiInterval > TimeSpan.Zero
|
||||
? _kpiHistoryOptions.SampleInterval.TotalSeconds / _options.KpiInterval.TotalSeconds
|
||||
: 1.0;
|
||||
|
||||
/// <inheritdoc />
|
||||
public string Source => KpiSources.SiteCallAudit;
|
||||
|
||||
@@ -90,12 +139,13 @@ public sealed class SiteCallAuditKpiSampleSource : IKpiSampleSource
|
||||
.ConfigureAwait(false);
|
||||
|
||||
var samples = new List<KpiSample>();
|
||||
var scaleFactor = IntervalScaleFactor;
|
||||
|
||||
// Global scope (null ScopeKey).
|
||||
AddSnapshot(
|
||||
samples, capturedAtUtc, KpiScopes.Global, scopeKey: null,
|
||||
global.BufferedCount, global.ParkedCount,
|
||||
global.FailedLastInterval, global.DeliveredLastInterval,
|
||||
global.FailedLastInterval * scaleFactor, global.DeliveredLastInterval * scaleFactor,
|
||||
global.OldestPendingAge, global.StuckCount);
|
||||
|
||||
// Per-site scope (ScopeKey = source site id).
|
||||
@@ -104,7 +154,7 @@ public sealed class SiteCallAuditKpiSampleSource : IKpiSampleSource
|
||||
AddSnapshot(
|
||||
samples, capturedAtUtc, KpiScopes.Site, scopeKey: site.SourceSite,
|
||||
site.BufferedCount, site.ParkedCount,
|
||||
site.FailedLastInterval, site.DeliveredLastInterval,
|
||||
site.FailedLastInterval * scaleFactor, site.DeliveredLastInterval * scaleFactor,
|
||||
site.OldestPendingAge, site.StuckCount);
|
||||
}
|
||||
|
||||
@@ -114,7 +164,7 @@ public sealed class SiteCallAuditKpiSampleSource : IKpiSampleSource
|
||||
AddSnapshot(
|
||||
samples, capturedAtUtc, KpiScopes.Node, scopeKey: node.SourceNode,
|
||||
node.BufferedCount, node.ParkedCount,
|
||||
node.FailedLastInterval, node.DeliveredLastInterval,
|
||||
node.FailedLastInterval * scaleFactor, node.DeliveredLastInterval * scaleFactor,
|
||||
node.OldestPendingAge, node.StuckCount);
|
||||
}
|
||||
|
||||
@@ -125,6 +175,9 @@ public sealed class SiteCallAuditKpiSampleSource : IKpiSampleSource
|
||||
/// Appends the six-metric catalog for one snapshot at the given scope. The
|
||||
/// oldest-pending-age metric is omitted when the snapshot's age is
|
||||
/// <c>null</c> (no non-terminal rows) rather than written as zero.
|
||||
/// <paramref name="failedLastInterval"/>/<paramref name="deliveredLastInterval"/> are
|
||||
/// already rescaled by <see cref="IntervalScaleFactor"/> (a <c>double</c>, not the raw
|
||||
/// repository <c>int</c> counts) by the caller.
|
||||
/// </summary>
|
||||
private void AddSnapshot(
|
||||
List<KpiSample> samples,
|
||||
@@ -133,8 +186,8 @@ public sealed class SiteCallAuditKpiSampleSource : IKpiSampleSource
|
||||
string? scopeKey,
|
||||
int buffered,
|
||||
int parked,
|
||||
int failedLastInterval,
|
||||
int deliveredLastInterval,
|
||||
double failedLastInterval,
|
||||
double deliveredLastInterval,
|
||||
TimeSpan? oldestPendingAge,
|
||||
int stuck)
|
||||
{
|
||||
|
||||
@@ -39,6 +39,12 @@
|
||||
SiteCallAudit. Preferred over moving the interfaces into Commons (Commons has no Akka /
|
||||
Communication dependency and would have to carry a Communication-adjacent message). -->
|
||||
<ProjectReference Include="../ZB.MOM.WW.ScadaBridge.AuditLog/ZB.MOM.WW.ScadaBridge.AuditLog.csproj" />
|
||||
<!-- The KPI sample source computes deliveredLastInterval/failedLastInterval
|
||||
against KpiHistoryOptions.SampleInterval directly (P5-5), not the
|
||||
independently configurable KpiInterval, so the central hourly rollup
|
||||
fold's per-sample sum is correct regardless of how KpiInterval is tuned.
|
||||
No cycle: KpiHistory references only Commons. -->
|
||||
<ProjectReference Include="../ZB.MOM.WW.ScadaBridge.KpiHistory/ZB.MOM.WW.ScadaBridge.KpiHistory.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
|
||||
Reference in New Issue
Block a user