Files
ScadaBridge/tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Kpi/KpiMetricAggregationTests.cs
T
Joseph Doherty 8da60f57ac feat(health): KPI history + sustained-backlog badge for LocalDb replication (row 33)
LocalDbOplogBacklog was reported and Prometheus-exported with nothing consuming
it: not sampled into KPI history, so a purge-burst spike could not be reviewed
after the fact, and no threshold treatment anywhere. Both halves shipped.

KPI SAMPLING. SiteHealthKpiSampleSource now emits localDbOplogBacklog and
localDbReplicationConnected (0/1), and the backlog is charted on the Health
dashboard's per-site trend panel. Charting is the point: a point-in-time tile
can never answer 'did that spike drain?', which is the only thing separating an
expected purge burst from a stalled peer.

Both are CONDITIONALLY emitted rather than coerced with ?? 0 like the
auditBacklogPending metric beside them. Both health-report fields are nullable
and null means 'no reading' — replication not configured (rig site-b/site-c) or
a failed poll — never zero. A fabricated 0 would render a pair that cannot read
its own oplog, or one deliberately unreplicated, as perfectly healthy once a
minute forever, burying the real series. So a site emits 12 metrics when
unreplicated and 14 when replicated, and a GAP in these two series truthfully
means 'not replicated / not reporting'. Note false and null differ for the
connected flag: a configured link that is DOWN samples as 0; no link samples as
nothing. Both fold as Gauge (summing an oplog depth, or a 0/1 flag, per hour is
meaningless).

THRESHOLD SEMANTICS — sustained, not instantaneous. This is the design, not a
refinement. The site_events retention purge is CDC-captured like any other
write, so the backlog jumps by the deleted batch and drains as the peer acks
it; a bare 'backlog > threshold' badge would fire on every replicated site
every day and be ignored within a week. Healthy is defined by the backlog
RETURNING to ~0, which only a dwell can express. CentralHealthAggregator holds
LocalDbBacklogAboveSince — the instant the backlog first rose above
LocalDbBacklogThreshold (default 1000) — and the badge appears only after
LocalDbBacklogSustainedFor (default 15 min). A report at or below the threshold
clears the anchor; one above leaves the ORIGINAL instant intact, so the window
is sustained rather than sliding.

The null case is deliberately a third branch: a null backlog leaves the anchor
UNCHANGED rather than clearing it. Null is evidence neither of a drain nor of a
backlog, so a node going blind mid-climb must not silently reset its own dwell,
and an unreplicated site (null forever) never accumulates an anchor and can
never be badged.

Validation rejects a non-positive threshold (0 would badge every replicated site
permanently, since an idle backlog already sits at it) and a dwell shorter than
one ReportInterval (satisfiable by a single sample, which collapses the check
back into the bare threshold it replaces).

DOC CORRECTION worth flagging: this row's premise was 'mirror the SiteAuditBacklog
configurable threshold badge'. That badge does not exist — Component-HealthMonitoring.md
described it in two places but no option, no comparison and no markup were ever
implemented. The two stale claims are corrected here rather than left to send the
next reader looking for it; the LocalDb badge is built fresh on the Metrics-stale
badge's pattern, which is a real configurable-threshold badge.

Tests: HealthMonitoring 115 (9 new aggregator tests incl. purge-burst-drains,
unreplicated-never-anchors, null-mid-climb, exact-threshold, stale-report; 5 new
validator tests; 4 new sampler tests incl. the null-semantics and false-vs-null
cases); CentralUI 1042 (5 new badge/chart tests incl. the purge-burst negative
control); Commons Kpi 93; KpiHistory 54.
2026-08-15 13:02:35 -04:00

178 lines
8.1 KiB
C#

using ZB.MOM.WW.ScadaBridge.Commons.Types.Kpi;
namespace ZB.MOM.WW.ScadaBridge.Commons.Tests.Kpi;
/// <summary>
/// Tests for <see cref="KpiMetricAggregationCatalog"/> — the gauge-vs-rate
/// classification the hourly rollup fold consults. Covers totality over every
/// metric the four <c>IKpiSampleSource</c> implementations actually emit, the
/// specific Rate set, a gauge sampling, and the unknown-metric default.
/// </summary>
public class KpiMetricAggregationTests
{
// Every (source, metric) pair emitted by the four sample sources, transcribed
// from the source code. The catalog must classify all of them (totality).
public static readonly (string Source, string Metric)[] AllEmittedMetrics =
{
// NotificationOutboxKpiSampleSource
(KpiSources.NotificationOutbox, "queueDepth"),
(KpiSources.NotificationOutbox, "stuckCount"),
(KpiSources.NotificationOutbox, "parkedCount"),
(KpiSources.NotificationOutbox, "deliveredLastInterval"),
(KpiSources.NotificationOutbox, "oldestPendingAgeSeconds"),
// SiteCallAuditKpiSampleSource
(KpiSources.SiteCallAudit, "buffered"),
(KpiSources.SiteCallAudit, "parked"),
(KpiSources.SiteCallAudit, "failedLastInterval"),
(KpiSources.SiteCallAudit, "deliveredLastInterval"),
(KpiSources.SiteCallAudit, "stuck"),
(KpiSources.SiteCallAudit, "oldestPendingAgeSeconds"),
// AuditLogKpiSampleSource
(KpiSources.AuditLog, "totalEventsLastHour"),
(KpiSources.AuditLog, "errorEventsLastHour"),
(KpiSources.AuditLog, "backlogTotal"),
// SiteHealthKpiSampleSource
(KpiSources.SiteHealth, "connectionsUp"),
(KpiSources.SiteHealth, "connectionsDown"),
(KpiSources.SiteHealth, "scriptErrors"),
(KpiSources.SiteHealth, "alarmEvalErrors"),
(KpiSources.SiteHealth, "sfBufferDepth"),
(KpiSources.SiteHealth, "deadLetters"),
(KpiSources.SiteHealth, "parkedMessages"),
(KpiSources.SiteHealth, "deployedInstances"),
(KpiSources.SiteHealth, "enabledInstances"),
(KpiSources.SiteHealth, "disabledInstances"),
(KpiSources.SiteHealth, "auditBacklogPending"),
(KpiSources.SiteHealth, "eventLogWriteFailures"),
// Register row 33. Both are instantaneous readings, so both fold as Gauge (the
// default) — summing an oplog depth per hour would be meaningless, and summing a
// 0/1 link flag doubly so. Both are CONDITIONALLY emitted: a site that reports no
// reading writes no sample rather than a fabricated 0.
(KpiSources.SiteHealth, "localDbOplogBacklog"),
(KpiSources.SiteHealth, "localDbReplicationConnected"),
};
// The (source, metric) pairs that must fold as Rate (sum-per-hour).
public static readonly (string Source, string Metric)[] RateMetrics =
{
(KpiSources.NotificationOutbox, "deliveredLastInterval"),
(KpiSources.SiteCallAudit, "failedLastInterval"),
(KpiSources.SiteCallAudit, "deliveredLastInterval"),
(KpiSources.SiteHealth, "scriptErrors"),
(KpiSources.SiteHealth, "alarmEvalErrors"),
(KpiSources.SiteHealth, "deadLetters"),
(KpiSources.SiteHealth, "eventLogWriteFailures"),
};
// A representative sampling of gauges — including the two *LastHour audit
// metrics that are rolling last-hour gauges, NOT rates (deliberate override).
public static readonly (string Source, string Metric)[] GaugeSampling =
{
(KpiSources.NotificationOutbox, "queueDepth"),
(KpiSources.NotificationOutbox, "oldestPendingAgeSeconds"),
(KpiSources.SiteCallAudit, "buffered"),
(KpiSources.SiteCallAudit, "stuck"),
(KpiSources.AuditLog, "backlogTotal"),
(KpiSources.AuditLog, "totalEventsLastHour"),
(KpiSources.AuditLog, "errorEventsLastHour"),
(KpiSources.SiteHealth, "connectionsUp"),
(KpiSources.SiteHealth, "connectionsDown"),
(KpiSources.SiteHealth, "sfBufferDepth"),
(KpiSources.SiteHealth, "deployedInstances"),
(KpiSources.SiteHealth, "localDbOplogBacklog"),
(KpiSources.SiteHealth, "localDbReplicationConnected"),
};
public static IEnumerable<object[]> AllEmittedCases() =>
AllEmittedMetrics.Select(m => new object[] { m.Source, m.Metric });
public static IEnumerable<object[]> RateCases() =>
RateMetrics.Select(m => new object[] { m.Source, m.Metric });
public static IEnumerable<object[]> GaugeCases() =>
GaugeSampling.Select(m => new object[] { m.Source, m.Metric });
[Theory]
[MemberData(nameof(AllEmittedCases))]
public void Resolve_EveryEmittedMetric_ReturnsDefinedAggregation(string source, string metric)
{
var result = KpiMetricAggregationCatalog.Resolve(source, metric);
Assert.True(
result is KpiRollupAggregation.Gauge or KpiRollupAggregation.Rate,
$"({source}, {metric}) resolved to an undefined aggregation: {result}");
}
[Theory]
[MemberData(nameof(RateCases))]
public void Resolve_RateMetrics_ClassifiedAsRate(string source, string metric)
{
Assert.Equal(KpiRollupAggregation.Rate, KpiMetricAggregationCatalog.Resolve(source, metric));
}
[Theory]
[MemberData(nameof(GaugeCases))]
public void Resolve_GaugeSampling_ClassifiedAsGauge(string source, string metric)
{
Assert.Equal(KpiRollupAggregation.Gauge, KpiMetricAggregationCatalog.Resolve(source, metric));
}
[Fact]
public void Resolve_LastHourAuditMetrics_AreGaugeNotRate()
{
// Rolling trailing-one-hour counts: last-value-per-hour is correct; summing
// 60 overlapping minute samples would ~60x over-count. Guard the decision.
Assert.Equal(
KpiRollupAggregation.Gauge,
KpiMetricAggregationCatalog.Resolve(KpiSources.AuditLog, "totalEventsLastHour"));
Assert.Equal(
KpiRollupAggregation.Gauge,
KpiMetricAggregationCatalog.Resolve(KpiSources.AuditLog, "errorEventsLastHour"));
}
[Theory]
[InlineData(KpiSources.NotificationOutbox, "someBrandNewMetricAddedLater")]
[InlineData("AnEntirelyNewSource", "anyMetric")]
[InlineData(KpiSources.SiteHealth, "")]
public void Resolve_UnknownMetric_DefaultsToGauge(string source, string metric)
{
// The fold must never crash on a metric added after this catalog was updated;
// Gauge (last-value) is the safe, non-fabricating default.
Assert.Equal(KpiRollupAggregation.Gauge, KpiMetricAggregationCatalog.Resolve(source, metric));
}
[Fact]
public void Resolve_SameMetricNameDifferentSource_KeysOnSourcePair()
{
// deliveredLastInterval is Rate under both sources here, but the point is the
// API keys on (source, metric) rather than metric alone — both resolve.
Assert.Equal(
KpiRollupAggregation.Rate,
KpiMetricAggregationCatalog.Resolve(KpiSources.NotificationOutbox, "deliveredLastInterval"));
Assert.Equal(
KpiRollupAggregation.Rate,
KpiMetricAggregationCatalog.Resolve(KpiSources.SiteCallAudit, "deliveredLastInterval"));
}
// ---- Task 8: catalog metric literals promoted to shared KpiMetrics constants (R4) ----
[Fact]
public void PromotedMetricConstants_LockHistoricalPersistedValues()
{
// Persisted data contract — these are symbol promotions, NOT renames.
Assert.Equal("deliveredLastInterval", KpiMetrics.SiteCallAudit.DeliveredLastInterval);
Assert.Equal("alarmEvalErrors", KpiMetrics.SiteHealth.AlarmEvalErrors);
Assert.Equal("eventLogWriteFailures", KpiMetrics.SiteHealth.EventLogWriteFailures);
}
[Theory]
[InlineData(KpiSources.SiteCallAudit, KpiMetrics.SiteCallAudit.DeliveredLastInterval)]
[InlineData(KpiSources.SiteHealth, KpiMetrics.SiteHealth.AlarmEvalErrors)]
[InlineData(KpiSources.SiteHealth, KpiMetrics.SiteHealth.EventLogWriteFailures)]
public void Resolve_PromotedRatePairs_StillClassifyAsRate(string source, string metric)
=> Assert.Equal(KpiRollupAggregation.Rate, KpiMetricAggregationCatalog.Resolve(source, metric));
}