feat(health): KPI history + sustained-backlog badge for LocalDb replication (row 33)
LocalDbOplogBacklog was reported and Prometheus-exported with nothing consuming it: not sampled into KPI history, so a purge-burst spike could not be reviewed after the fact, and no threshold treatment anywhere. Both halves shipped. KPI SAMPLING. SiteHealthKpiSampleSource now emits localDbOplogBacklog and localDbReplicationConnected (0/1), and the backlog is charted on the Health dashboard's per-site trend panel. Charting is the point: a point-in-time tile can never answer 'did that spike drain?', which is the only thing separating an expected purge burst from a stalled peer. Both are CONDITIONALLY emitted rather than coerced with ?? 0 like the auditBacklogPending metric beside them. Both health-report fields are nullable and null means 'no reading' — replication not configured (rig site-b/site-c) or a failed poll — never zero. A fabricated 0 would render a pair that cannot read its own oplog, or one deliberately unreplicated, as perfectly healthy once a minute forever, burying the real series. So a site emits 12 metrics when unreplicated and 14 when replicated, and a GAP in these two series truthfully means 'not replicated / not reporting'. Note false and null differ for the connected flag: a configured link that is DOWN samples as 0; no link samples as nothing. Both fold as Gauge (summing an oplog depth, or a 0/1 flag, per hour is meaningless). THRESHOLD SEMANTICS — sustained, not instantaneous. This is the design, not a refinement. The site_events retention purge is CDC-captured like any other write, so the backlog jumps by the deleted batch and drains as the peer acks it; a bare 'backlog > threshold' badge would fire on every replicated site every day and be ignored within a week. Healthy is defined by the backlog RETURNING to ~0, which only a dwell can express. CentralHealthAggregator holds LocalDbBacklogAboveSince — the instant the backlog first rose above LocalDbBacklogThreshold (default 1000) — and the badge appears only after LocalDbBacklogSustainedFor (default 15 min). A report at or below the threshold clears the anchor; one above leaves the ORIGINAL instant intact, so the window is sustained rather than sliding. The null case is deliberately a third branch: a null backlog leaves the anchor UNCHANGED rather than clearing it. Null is evidence neither of a drain nor of a backlog, so a node going blind mid-climb must not silently reset its own dwell, and an unreplicated site (null forever) never accumulates an anchor and can never be badged. Validation rejects a non-positive threshold (0 would badge every replicated site permanently, since an idle backlog already sits at it) and a dwell shorter than one ReportInterval (satisfiable by a single sample, which collapses the check back into the bare threshold it replaces). DOC CORRECTION worth flagging: this row's premise was 'mirror the SiteAuditBacklog configurable threshold badge'. That badge does not exist — Component-HealthMonitoring.md described it in two places but no option, no comparison and no markup were ever implemented. The two stale claims are corrected here rather than left to send the next reader looking for it; the LocalDb badge is built fresh on the Metrics-stale badge's pattern, which is a real configurable-threshold badge. Tests: HealthMonitoring 115 (9 new aggregator tests incl. purge-burst-drains, unreplicated-never-anchors, null-mid-climb, exact-threshold, stale-report; 5 new validator tests; 4 new sampler tests incl. the null-semantics and false-vs-null cases); CentralUI 1042 (5 new badge/chart tests incl. the purge-burst negative control); Commons Kpi 93; KpiHistory 54.
This commit is contained in:
Joseph Doherty
@@ -255,6 +255,32 @@ LocalDbReplicationConnected staying true with the backlog returning to ~0 is the
|
||||
An unreplicated node logs only the first line. If a backlog spike has no such line near it in the
|
||||
active node's log, the purge is *not* the explanation and the spike is worth investigating.
|
||||
|
||||
**Two dashboard surfaces make this reviewable rather than a log-archaeology exercise** (register
|
||||
row 33):
|
||||
|
||||
- **The "Replication backlog" badge** on the site tile of the Health dashboard. It is deliberately
|
||||
a *sustained* signal, not a bare threshold: the badge appears only once the backlog has stayed
|
||||
above `ScadaBridge:HealthMonitoring:LocalDbBacklogThreshold` (default **1000**) continuously for
|
||||
`ScadaBridge:HealthMonitoring:LocalDbBacklogSustainedFor` (default **15 minutes**). The daily
|
||||
purge burst described above drains in seconds to a couple of minutes, so it never reaches the
|
||||
dwell and never badges the site — which is the entire reason the dwell exists. A bare threshold
|
||||
would fire once a day on every replicated site and be ignored within a week. The dwell is
|
||||
validated to be at least one `ReportInterval`, since a shorter window could be satisfied by a
|
||||
single sample and would collapse back into a bare threshold.
|
||||
- **The "LocalDb Replication Backlog" trend chart** in the dashboard's per-site Site Health Trends
|
||||
panel, backed by the KPI history store (sampled every minute, retained 90 days raw / 365 days
|
||||
hourly). This is what answers the question a point-in-time reading cannot: *did that spike
|
||||
drain?* A past purge burst appears as a spike that returns to ~0; a real fault appears as a
|
||||
level that never comes back down.
|
||||
|
||||
**Both surfaces are silent on an unreplicated site, by design.** `LocalDbOplogBacklog` and
|
||||
`LocalDbReplicationConnected` are nullable, and null means "no reading", never zero. The KPI
|
||||
sampler therefore writes **no sample at all** rather than a fabricated 0, so site-b/site-c show an
|
||||
empty chart rather than a flat zero line, and they never accumulate the dwell anchor the badge
|
||||
keys off. A gap in those series means "not replicated / not reporting" — which is the truth. Note
|
||||
that `false` and null differ for the connected flag: a configured link that is DOWN samples as 0,
|
||||
while no link at all samples as nothing.
|
||||
|
||||
### Site Pair Upgrades — stop and start BOTH nodes together
|
||||
|
||||
**A rolling upgrade of a site pair, one node at a time, is no longer supported.** It worked while
|
||||
|
||||
@@ -34,7 +34,7 @@ Site clusters (metric collection and reporting). Central cluster (aggregation an
|
||||
| Notification Outbox queue depth | Notification Outbox (central) | Count of `Pending` + `Retrying` notifications — central-computed, not site-reported |
|
||||
| Notification Outbox stuck count | Notification Outbox (central) | Count of `Pending` / `Retrying` notifications older than the configurable stuck-age threshold — central-computed, not site-reported |
|
||||
| Notification Outbox parked count | Notification Outbox (central) | Count of `Parked` notifications — central-computed, not site-reported |
|
||||
| `SiteAuditBacklog` | Audit Log (site) | Count of `Pending` rows in the site-local `AuditLog` plus oldest-pending-age plus on-disk bytes. A configurable threshold drives a Health dashboard warning on the affected site tile. |
|
||||
| `SiteAuditBacklog` | Audit Log (site) | Count of `Pending` rows in the site-local `AuditLog` plus oldest-pending-age plus on-disk bytes. Surfaced on the Audit KPI tiles (which border-highlight any non-zero backlog) and sampled into KPI history as `auditBacklogPending`. **No configurable per-site threshold badge exists for this metric** — earlier revisions of this document described one, but none was ever implemented and no such option exists. The only configurable sustained-threshold site badge is the LocalDb replication backlog (register row 33, below). |
|
||||
| `SiteAuditWriteFailures` | Audit Log (site) | Count of failed hot-path audit appends at the site since the last health report. |
|
||||
| `AuditRedactionFailure` | Audit Log (central) | Count of payload redactor errors (over-redacted payloads, safety-net hit) since the last interval. |
|
||||
| `LocalDbReplicationConnected` | Consolidated site LocalDb | Whether a peer sync session is currently running on this node. **Nullable — null means "no reading", not "disconnected".** Reported only by a node with replication configured. |
|
||||
@@ -90,7 +90,7 @@ The Audit Log spans both sites (hot-path append + telemetry forward) and central
|
||||
|
||||
- **Audit volume** — events/min landing in the central `AuditLog` table, shown global plus per-site sparkline; sourced from the Audit Log component on the active central node.
|
||||
- **Audit error rate** — percent of central `AuditLog` rows with `Status` other than `Success` / `Delivered` / `Enqueued` over a rolling 5-minute window. This is the operational error rate of audited operations (HTTP 5xx, transient failures, parked deliveries, etc.) — NOT the audit writer's own health. Audit-writer issues surface separately via `AuditRedactionFailure`.
|
||||
- **Audit backlog** — global aggregate of `SiteAuditBacklog` across reporting sites (count of `Pending` site-local audit rows, oldest pending age, on-disk bytes); click drills into a per-site breakdown. The per-site tile surfaces a warning badge when its `SiteAuditBacklog` crosses the configurable threshold.
|
||||
- **Audit backlog** — global aggregate of `SiteAuditBacklog` across reporting sites (count of `Pending` site-local audit rows, oldest pending age, on-disk bytes); click drills into a per-site breakdown. The tile border-highlights a non-zero aggregate backlog; there is **no** configurable per-site `SiteAuditBacklog` threshold badge (see the note in Monitored Metrics — earlier revisions described one that was never built).
|
||||
|
||||
These tiles are **point-in-time** like the Notification Outbox and Site Call Audit KPI tiles — no time-series store; consistent with Health Monitoring's "current status only" philosophy. The site-scoped `SiteAuditBacklog` / `SiteAuditWriteFailures` metrics arrive in the existing site health report; the central-scoped `AuditRedactionFailure` metric is central-computed alongside the existing central KPIs.
|
||||
|
||||
@@ -101,6 +101,46 @@ These tiles are **point-in-time** like the Notification Outbox and Site Call Aud
|
||||
- Notification Outbox and Site Call Audit KPIs are not stored by Health Monitoring; they are computed point-in-time from the central `Notifications` and `SiteCalls` tables respectively each time the dashboard refreshes — consistent with the current-status-only philosophy.
|
||||
- Site connectivity history (online/offline transitions) may optionally be logged via the Audit Log or a separate mechanism if needed in the future.
|
||||
|
||||
## Sustained LocalDb Replication Backlog Badge
|
||||
|
||||
The per-site tile shows a **Replication backlog** warning badge when `LocalDbOplogBacklog` has
|
||||
stayed above `LocalDbBacklogThreshold` (default **1000**) continuously for
|
||||
`LocalDbBacklogSustainedFor` (default **15 minutes**). Both are configurable under
|
||||
`ScadaBridge:HealthMonitoring` and validated at startup.
|
||||
|
||||
- **Sustained, not instantaneous — and that is the design, not a refinement.** The daily
|
||||
`site_events` retention purge and the storage-cap trim are CDC-captured like any other write, so
|
||||
the backlog jumps by the deleted batch size and drains as the peer acks it. A bare
|
||||
`backlog > threshold` badge would therefore fire on every replicated site every day. Healthy is
|
||||
defined by the backlog *returning to ~0*, which only a dwell can express. `LocalDbBacklogSustainedFor`
|
||||
is validated to be at least one `ReportInterval`: a shorter dwell could be satisfied by a single
|
||||
sample and would collapse back into the bare threshold.
|
||||
- **Implementation.** `CentralHealthAggregator` holds `SiteHealthState.LocalDbBacklogAboveSince`,
|
||||
the instant the backlog first rose above the threshold. Reports at or below the threshold clear
|
||||
it; reports above it leave the original instant intact (a sustained window, not a sliding one).
|
||||
- **Null handling is load-bearing.** A report with a `null` backlog leaves the anchor *unchanged*
|
||||
rather than clearing it — null means "no reading" (unreplicated site, or a failed poll), which is
|
||||
evidence neither of a drain nor of a backlog. An unreplicated site (rig site-b/site-c) reports
|
||||
null forever, so it never accumulates an anchor and can never be badged.
|
||||
- Offline sites are not badged; the Offline badge is the signal that matters there.
|
||||
|
||||
## KPI History Sampling of Replication State
|
||||
|
||||
`SiteHealthKpiSampleSource` samples `localDbOplogBacklog` and `localDbReplicationConnected` (0/1)
|
||||
into the KPI history store, and the backlog is charted in the Health dashboard's per-site Site
|
||||
Health Trends panel as **LocalDb Replication Backlog**. This is the retrospective half of the
|
||||
signal: a point-in-time tile can never answer "did that spike drain?", which is the only thing
|
||||
distinguishing an expected purge burst from a stalled peer.
|
||||
|
||||
Both metrics are **conditionally emitted** — a site reporting `null` writes no sample rather than a
|
||||
coerced `0`, unlike the `auditBacklogPending` metric alongside them. A fabricated 0 would render a
|
||||
pair that cannot read its own oplog, or one deliberately not replicated, as perfectly healthy once
|
||||
a minute forever. Consequently a site's sample count varies: **12 metrics for an unreplicated site,
|
||||
14 for a replicated one**, and a gap in these two series truthfully means "not replicated / not
|
||||
reporting". Note `false` and `null` are different states for the connected flag — a configured link
|
||||
that is down samples as `0`; no link at all samples as nothing. Both fold as **Gauge** in the hourly
|
||||
rollup catalog (summing an oplog depth, or a 0/1 flag, per hour would be meaningless).
|
||||
|
||||
## Prometheus Metrics (site node)
|
||||
|
||||
Distinct from the health *report* above: these are scraped from the site node's `/metrics`
|
||||
@@ -151,4 +191,4 @@ endpoint (`MetricsPort`, default 8084) rather than sent to central.
|
||||
|
||||
- **Central UI**: Health Monitoring Dashboard displays aggregated metrics.
|
||||
- **Communication Layer**: Health reports flow as periodic messages.
|
||||
- **KPI History (#26)**: emits `IKpiSampleSource` (`SiteHealthKpiSampleSource`, per-Site) consumed by the KpiHistory recorder (#26). It reads the in-memory `ICentralHealthAggregator.GetAllSiteStates()` (no DB), turning the per-site snapshot — previously sequence-numbered every 30s but discarded — into 12 sampled `KpiSample` metrics per site, persisted every minute: `connectionsUp`/`connectionsDown`, `scriptErrors`, `alarmEvalErrors`, `sfBufferDepth`, `deadLetters`, `parkedMessages`, `deployedInstances`/`enabledInstances`/`disabledInstances`, `auditBacklogPending`, `eventLogWriteFailures`. Of these, **only four are charted** in the dashboard's per-site `KpiTrendChart` panel — `connectionsDown`, `scriptErrors`, `sfBufferDepth`, and `deadLetters` (the metrics carried in the shared `KpiMetrics.SiteHealth` Commons catalog and fetched by `Health.razor`'s trend selector). The remaining eight are **sampled and persisted but not (yet) charted** — they are retained in the `KpiSample` store and available for future trend surfaces or ad-hoc query. The synthetic central self-report (`CentralHealthReportLoop.CentralSiteId = "$central"`, fed by the leader-only report loop via `CollectReport($central)`) appears in `GetAllSiteStates()` and is **intentionally sampled into the per-Site KPI store as a real `KpiScopes.Site` series** keyed `$central`; the Central UI deliberately **pins `$central` first** in the per-site trend selector (rendered as "Central Cluster"), so the central cluster's own trends sit alongside the real sites'. (`$central`'s zero-valued connection/instance/S&F metrics reflect the central self-report carrying no site-runtime data, not a defect.) See [Component-KpiHistory.md](Component-KpiHistory.md).
|
||||
- **KPI History (#26)**: emits `IKpiSampleSource` (`SiteHealthKpiSampleSource`, per-Site) consumed by the KpiHistory recorder (#26). It reads the in-memory `ICentralHealthAggregator.GetAllSiteStates()` (no DB), turning the per-site snapshot — previously sequence-numbered every 30s but discarded — into 12–14 sampled `KpiSample` metrics per site, persisted every minute: `connectionsUp`/`connectionsDown`, `scriptErrors`, `alarmEvalErrors`, `sfBufferDepth`, `deadLetters`, `parkedMessages`, `deployedInstances`/`enabledInstances`/`disabledInstances`, `auditBacklogPending`, `eventLogWriteFailures`, plus the two conditionally-emitted LocalDb replication metrics `localDbOplogBacklog` and `localDbReplicationConnected` (register row 33 — emitted only when the site actually reports a reading, hence 12 for an unreplicated site and 14 for a replicated one; see *KPI History Sampling of Replication State* above). Of these, **only five are charted** in the dashboard's per-site `KpiTrendChart` panel — `connectionsDown`, `scriptErrors`, `sfBufferDepth`, `deadLetters`, and `localDbOplogBacklog` (the metrics carried in the shared `KpiMetrics.SiteHealth` Commons catalog and fetched by `Health.razor`'s trend selector). The rest are **sampled and persisted but not (yet) charted** — they are retained in the `KpiSample` store and available for future trend surfaces or ad-hoc query. The synthetic central self-report (`CentralHealthReportLoop.CentralSiteId = "$central"`, fed by the leader-only report loop via `CollectReport($central)`) appears in `GetAllSiteStates()` and is **intentionally sampled into the per-Site KPI store as a real `KpiScopes.Site` series** keyed `$central`; the Central UI deliberately **pins `$central` first** in the per-site trend selector (rendered as "Central Cluster"), so the central cluster's own trends sit alongside the real sites'. (`$central`'s zero-valued connection/instance/S&F metrics reflect the central self-report carrying no site-runtime data, not a defect.) See [Component-KpiHistory.md](Component-KpiHistory.md).
|
||||
|
||||
@@ -143,6 +143,17 @@ Reads `ICentralHealthAggregator.GetAllSiteStates()` (in-memory, no DB). Scope: *
|
||||
- `disabledInstances`
|
||||
- `auditBacklogPending`
|
||||
- `eventLogWriteFailures`
|
||||
- `localDbOplogBacklog` — **conditionally emitted** (register row 33)
|
||||
- `localDbReplicationConnected` — **conditionally emitted** (register row 33), 0/1
|
||||
|
||||
The last two are written **only when the site actually reports a reading**. Both underlying health-report
|
||||
fields are nullable and null means "no reading" (replication not configured, or a failed poll), never
|
||||
zero — so unlike `auditBacklogPending` above, they are not coerced with `?? 0`. A site therefore emits
|
||||
**12 metrics when unreplicated and 14 when replicated**, and a gap in these two series truthfully means
|
||||
"not replicated / not reporting" rather than a flat zero line that would render an unreplicated pair, or
|
||||
one that cannot read its own oplog, as perfectly healthy. Both fold as **Gauge** in
|
||||
`KpiMetricAggregationCatalog` (they are instantaneous readings; summing an oplog depth or a 0/1 flag per
|
||||
hour would be meaningless). `localDbOplogBacklog` is charted on the Health dashboard's per-site trend panel.
|
||||
|
||||
## Query + UI
|
||||
|
||||
|
||||
@@ -170,6 +170,17 @@
|
||||
IsAvailable="@_sfBufferDepthAvailable"
|
||||
ErrorMessage="@_sfBufferDepthError" />
|
||||
</div>
|
||||
@* Register row 33. The point of charting the backlog is retrospective:
|
||||
the daily purge burst is expected, so what an operator needs is to see
|
||||
whether a past spike DRAINED — which a point-in-time tile can never
|
||||
answer. An unreplicated site emits no samples, so this chart is
|
||||
legitimately empty there rather than a flat, misleading zero line. *@
|
||||
<div class="col-lg-3 col-md-6">
|
||||
<KpiTrendChart Title="LocalDb Replication Backlog"
|
||||
Points="@_localDbBacklogSeries"
|
||||
IsAvailable="@_localDbBacklogAvailable"
|
||||
ErrorMessage="@_localDbBacklogError" />
|
||||
</div>
|
||||
</div>
|
||||
}
|
||||
</div>
|
||||
@@ -248,6 +259,14 @@
|
||||
Metrics stale
|
||||
</span>
|
||||
}
|
||||
@if (IsLocalDbBacklogSustained(state))
|
||||
{
|
||||
<span class="badge bg-warning text-dark"
|
||||
data-test="localdb-backlog-badge"
|
||||
title="LocalDb replication oplog backlog has stayed above @HealthOptions.Value.LocalDbBacklogThreshold for over @BacklogSustainedDisplay. A short daily spike at the site_events purge is expected and is NOT this — the peer has stopped draining.">
|
||||
Replication backlog
|
||||
</span>
|
||||
}
|
||||
@if (!state.IsOnline && state.LastStatusChangeAt is { } changedAt)
|
||||
{
|
||||
<small class="text-muted">offline since @changedAt.ToString("u")</small>
|
||||
@@ -526,6 +545,27 @@
|
||||
private string StaleTimeoutDisplay =>
|
||||
FormatDuration(HealthOptions.Value.MetricsStaleTimeout);
|
||||
|
||||
// Human-friendly rendering of the configured sustained-backlog dwell for the
|
||||
// "Replication backlog" badge tooltip (e.g. "15 minutes").
|
||||
private string BacklogSustainedDisplay =>
|
||||
FormatDuration(HealthOptions.Value.LocalDbBacklogSustainedFor);
|
||||
|
||||
// Register row 33: the sustained LocalDb replication backlog badge.
|
||||
//
|
||||
// Reads the anchor the aggregator maintains (LocalDbBacklogAboveSince) rather than the
|
||||
// instantaneous backlog on the report, because a bare "backlog > threshold" badge would
|
||||
// fire on EVERY replicated site EVERY day: the site_events retention purge is CDC-captured
|
||||
// like any other write, so the backlog jumps by the deleted batch size and then drains.
|
||||
// Healthy is "the backlog comes back down"; only a dwell can express that.
|
||||
//
|
||||
// A site that reports no backlog at all (unreplicated — rig site-b/site-c — or a failed
|
||||
// poll) never gets an anchor, so it can never be badged. That is the whole null-semantics
|
||||
// requirement: null means "no reading", not zero.
|
||||
private bool IsLocalDbBacklogSustained(SiteHealthState state) =>
|
||||
state.IsOnline
|
||||
&& state.LocalDbBacklogAboveSince is { } since
|
||||
&& DateTimeOffset.UtcNow - since >= HealthOptions.Value.LocalDbBacklogSustainedFor;
|
||||
|
||||
// Online central nodes, from the same ClusterNodes list the Nodes column renders.
|
||||
// Fewer than two means no standby, which disables the manual-failover control. This is
|
||||
// a display-side guard only — AkkaManualFailoverService re-checks against live cluster
|
||||
@@ -616,6 +656,9 @@
|
||||
private IReadOnlyList<KpiSeriesPoint>? _sfBufferDepthSeries;
|
||||
private bool _sfBufferDepthAvailable = true;
|
||||
private string? _sfBufferDepthError;
|
||||
private IReadOnlyList<KpiSeriesPoint>? _localDbBacklogSeries;
|
||||
private bool _localDbBacklogAvailable = true;
|
||||
private string? _localDbBacklogError;
|
||||
|
||||
private static bool SiteHasActiveErrors(SiteHealthState state)
|
||||
{
|
||||
@@ -739,6 +782,8 @@
|
||||
await LoadTrendSeriesAsync(KpiMetrics.SiteHealth.ScriptErrors, siteId, fromUtc, toUtc);
|
||||
(_sfBufferDepthSeries, _sfBufferDepthAvailable, _sfBufferDepthError) =
|
||||
await LoadTrendSeriesAsync(KpiMetrics.SiteHealth.SfBufferDepth, siteId, fromUtc, toUtc);
|
||||
(_localDbBacklogSeries, _localDbBacklogAvailable, _localDbBacklogError) =
|
||||
await LoadTrendSeriesAsync(KpiMetrics.SiteHealth.LocalDbOplogBacklog, siteId, fromUtc, toUtc);
|
||||
}
|
||||
finally
|
||||
{
|
||||
|
||||
@@ -120,5 +120,22 @@ public static class KpiMetrics
|
||||
|
||||
/// <summary>Summed store-and-forward buffer depth across all buffers.</summary>
|
||||
public const string SfBufferDepth = "sfBufferDepth";
|
||||
|
||||
/// <summary>
|
||||
/// Unacked LocalDb replication oplog depth (register row 33). Emitted ONLY by sites
|
||||
/// that actually report a backlog — an unreplicated site reports null, and null means
|
||||
/// "no reading", never zero, so no sample is written at all rather than a fabricated 0.
|
||||
/// A gap in this series therefore means "not replicated / not reporting", which is
|
||||
/// exactly what it should mean.
|
||||
/// </summary>
|
||||
public const string LocalDbOplogBacklog = "localDbOplogBacklog";
|
||||
|
||||
/// <summary>
|
||||
/// LocalDb replication link state as 1/0 (register row 33). Emitted ONLY when the site
|
||||
/// reports a value; null (replication not configured) writes no sample. Note that
|
||||
/// <c>false</c> and null are different states here — false is a configured link that is
|
||||
/// down, null is no link configured — which is why this cannot be coerced to 0.
|
||||
/// </summary>
|
||||
public const string LocalDbReplicationConnected = "localDbReplicationConnected";
|
||||
}
|
||||
}
|
||||
|
||||
@@ -50,7 +50,8 @@ public class CentralHealthAggregator : BackgroundService, ICentralHealthAggregat
|
||||
FirstSeenAt = now,
|
||||
LastSequenceNumber = report.SequenceNumber,
|
||||
IsOnline = true,
|
||||
IsMetricsStale = false
|
||||
IsMetricsStale = false,
|
||||
LocalDbBacklogAboveSince = NextBacklogAboveSince(null, report, now)
|
||||
// LastStatusChangeAt stays null: first observation is not a flip.
|
||||
};
|
||||
|
||||
@@ -84,7 +85,9 @@ public class CentralHealthAggregator : BackgroundService, ICentralHealthAggregat
|
||||
IsMetricsStale = false,
|
||||
// Stamp a flip only when this report brings the site back online;
|
||||
// a report from an already-online site leaves the timestamp be.
|
||||
LastStatusChangeAt = existing.IsOnline ? existing.LastStatusChangeAt : now
|
||||
LastStatusChangeAt = existing.IsOnline ? existing.LastStatusChangeAt : now,
|
||||
LocalDbBacklogAboveSince =
|
||||
NextBacklogAboveSince(existing.LocalDbBacklogAboveSince, report, now)
|
||||
};
|
||||
|
||||
if (_siteStates.TryUpdate(report.SiteId, updated, existing))
|
||||
@@ -320,4 +323,43 @@ public class CentralHealthAggregator : BackgroundService, ICentralHealthAggregat
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Advances the sustained-backlog anchor for one incoming report (register row 33).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Three cases, and the null one is the subtle one:
|
||||
/// <list type="bullet">
|
||||
/// <item><description>
|
||||
/// Backlog <c>null</c> — no reading. An unreplicated site (rig site-b/site-c) reports this
|
||||
/// forever, and a replicated site reports it when a poll fails. Carry the previous anchor
|
||||
/// through UNCHANGED: null is not zero, so it is neither evidence of a drain (which would
|
||||
/// justify clearing) nor of a backlog (which would justify starting one). An unreplicated
|
||||
/// site therefore never accumulates an anchor and can never be badged.
|
||||
/// </description></item>
|
||||
/// <item><description>
|
||||
/// Backlog above the threshold — start the clock if it is not already running. Keeping the
|
||||
/// ORIGINAL start instant is what makes the window sustained rather than sliding.
|
||||
/// </description></item>
|
||||
/// <item><description>
|
||||
/// Backlog at or below the threshold — clear. This is the daily purge burst's exit path:
|
||||
/// the spike drains within a few reports and the anchor is discarded long before the dwell
|
||||
/// elapses, so the expected burst never badges the site.
|
||||
/// </description></item>
|
||||
/// </list>
|
||||
/// </remarks>
|
||||
/// <param name="current">The anchor currently held for this site, if any.</param>
|
||||
/// <param name="report">The incoming health report.</param>
|
||||
/// <param name="now">The current time.</param>
|
||||
/// <returns>The anchor to store.</returns>
|
||||
private DateTimeOffset? NextBacklogAboveSince(
|
||||
DateTimeOffset? current, SiteHealthReport report, DateTimeOffset now)
|
||||
{
|
||||
if (report.LocalDbOplogBacklog is not { } backlog)
|
||||
{
|
||||
return current;
|
||||
}
|
||||
|
||||
return backlog > _options.LocalDbBacklogThreshold ? current ?? now : null;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -32,4 +32,32 @@ public class HealthMonitoringOptions
|
||||
/// (~4 missed 30s reports).
|
||||
/// </summary>
|
||||
public TimeSpan MetricsStaleTimeout { get; set; } = TimeSpan.FromMinutes(2);
|
||||
|
||||
/// <summary>
|
||||
/// Unacked LocalDb replication oplog depth above which a site is considered backlogged
|
||||
/// (register row 33). Crossing this alone shows nothing — the backlog must also stay
|
||||
/// above it for <see cref="LocalDbBacklogSustainedFor"/> before the Health dashboard
|
||||
/// badges the site. Must be positive. Default: <c>1000</c> entries.
|
||||
/// </summary>
|
||||
public long LocalDbBacklogThreshold { get; set; } = 1000;
|
||||
|
||||
/// <summary>
|
||||
/// How long the oplog backlog must stay continuously above
|
||||
/// <see cref="LocalDbBacklogThreshold"/> before the Health dashboard badges the site
|
||||
/// (register row 33).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// This dwell time is the whole point of the signal, not a refinement of it. A bare
|
||||
/// threshold would fire every single day on a healthy replicated site: the `site_events`
|
||||
/// retention purge and the storage-cap trim are CDC-captured like any other write, so the
|
||||
/// backlog jumps by the deleted batch size and then drains as the peer acks it. That spike
|
||||
/// is expected and documented (see <c>docs/deployment/topology-guide.md</c> → *Reading the
|
||||
/// replication backlog*). Healthy means the backlog RETURNS TO ~0; unhealthy means it
|
||||
/// stays up. Only a sustained window can tell those apart, so the default is set well
|
||||
/// clear of a purge drain. Must be positive and no shorter than
|
||||
/// <see cref="ReportInterval"/> — a window shorter than one report interval could be
|
||||
/// satisfied by a single sample, which is exactly the bare threshold this exists to avoid.
|
||||
/// Default: 15 minutes.
|
||||
/// </remarks>
|
||||
public TimeSpan LocalDbBacklogSustainedFor { get; set; } = TimeSpan.FromMinutes(15);
|
||||
}
|
||||
|
||||
@@ -64,5 +64,29 @@ public sealed class HealthMonitoringOptionsValidator : OptionsValidatorBase<Heal
|
||||
$"must be >= ReportInterval ({options.ReportInterval}): a stale window shorter than one " +
|
||||
"report interval would flag every healthy site, since reports cannot arrive faster than " +
|
||||
"they are emitted.");
|
||||
|
||||
builder.RequireThat(options.LocalDbBacklogThreshold > 0,
|
||||
$"ScadaBridge:HealthMonitoring:LocalDbBacklogThreshold must be positive " +
|
||||
$"(was {options.LocalDbBacklogThreshold}); a zero/negative threshold would badge every " +
|
||||
"replicated site permanently, since an idle backlog of 0 already sits at or above it.");
|
||||
|
||||
builder.RequireThat(options.LocalDbBacklogSustainedFor > TimeSpan.Zero,
|
||||
$"ScadaBridge:HealthMonitoring:LocalDbBacklogSustainedFor must be a positive duration " +
|
||||
$"(was {options.LocalDbBacklogSustainedFor}); it is the dwell time a backlog must stay " +
|
||||
"above the threshold before the site is badged.");
|
||||
|
||||
// A dwell shorter than one report interval can be satisfied by a single sample,
|
||||
// degrading the sustained check back into the bare threshold it exists to replace —
|
||||
// which would fire on every daily purge burst. Same De Morgan guard shape as the
|
||||
// pairs above: silent unless BOTH fields are positive and the dwell is too short,
|
||||
// so the dedicated positive-value checks own that failure.
|
||||
builder.RequireThat(
|
||||
!(options.ReportInterval > TimeSpan.Zero
|
||||
&& options.LocalDbBacklogSustainedFor > TimeSpan.Zero
|
||||
&& options.LocalDbBacklogSustainedFor < options.ReportInterval),
|
||||
$"ScadaBridge:HealthMonitoring:LocalDbBacklogSustainedFor ({options.LocalDbBacklogSustainedFor}) " +
|
||||
$"must be >= ReportInterval ({options.ReportInterval}): a dwell shorter than one report " +
|
||||
"interval can be satisfied by a single sample, which turns the sustained check back into " +
|
||||
"a bare threshold and re-introduces the daily purge-burst false positive it exists to avoid.");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -49,6 +49,8 @@ public sealed class SiteHealthKpiSampleSource : IKpiSampleSource
|
||||
private const string MetricDisabledInstances = "disabledInstances";
|
||||
private const string MetricAuditBacklogPending = "auditBacklogPending";
|
||||
private const string MetricEventLogWriteFailures = KpiMetrics.SiteHealth.EventLogWriteFailures;
|
||||
private const string MetricLocalDbOplogBacklog = KpiMetrics.SiteHealth.LocalDbOplogBacklog;
|
||||
private const string MetricLocalDbReplicationConnected = KpiMetrics.SiteHealth.LocalDbReplicationConnected;
|
||||
|
||||
private readonly ICentralHealthAggregator _aggregator;
|
||||
|
||||
@@ -136,6 +138,32 @@ public sealed class SiteHealthKpiSampleSource : IKpiSampleSource
|
||||
report.SiteAuditBacklog?.PendingCount ?? 0));
|
||||
samples.Add(Sample(
|
||||
capturedAtUtc, MetricEventLogWriteFailures, siteId, report.SiteEventLogWriteFailures));
|
||||
|
||||
// Register row 33: LocalDb replication. Both fields are NULLABLE and null is
|
||||
// load-bearing — it means "no reading" (replication not configured, as on rig
|
||||
// site-b/site-c, or a failed poll), NOT zero. So unlike the audit-backlog line
|
||||
// above, these are CONDITIONALLY emitted rather than coerced with `?? 0`:
|
||||
//
|
||||
// - A fabricated 0 backlog would render a pair that cannot read its own oplog,
|
||||
// or one that simply isn't replicated, as perfectly healthy — and would do it
|
||||
// once a minute, forever, burying the real series for the sites that are.
|
||||
// - For the connected flag, `false` and null are genuinely different states: a
|
||||
// configured link that is DOWN versus no link at all. Collapsing null to 0
|
||||
// would alarm on every unreplicated site permanently.
|
||||
//
|
||||
// The cost is that a site's sample count varies (12 unreplicated, 14 replicated).
|
||||
// That is the intended shape: a GAP in these two series means "not replicated / not
|
||||
// reporting", which is a truthful reading, whereas a flat 0 line is a lie.
|
||||
if (report.LocalDbOplogBacklog is { } backlog)
|
||||
{
|
||||
samples.Add(Sample(capturedAtUtc, MetricLocalDbOplogBacklog, siteId, backlog));
|
||||
}
|
||||
|
||||
if (report.LocalDbReplicationConnected is { } connected)
|
||||
{
|
||||
samples.Add(Sample(
|
||||
capturedAtUtc, MetricLocalDbReplicationConnected, siteId, connected ? 1 : 0));
|
||||
}
|
||||
}
|
||||
|
||||
private static KpiSample Sample(
|
||||
|
||||
@@ -67,6 +67,28 @@ public sealed record SiteHealthState
|
||||
/// </summary>
|
||||
public bool IsMetricsStale { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Gets the instant this site's LocalDb replication oplog backlog first rose above
|
||||
/// <see cref="HealthMonitoringOptions.LocalDbBacklogThreshold"/> and stayed there, or
|
||||
/// <c>null</c> if the backlog is not currently above it (register row 33).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The anchor for the *sustained* backlog badge: the badge shows only once
|
||||
/// <see cref="HealthMonitoringOptions.LocalDbBacklogSustainedFor"/> has elapsed since this
|
||||
/// instant. Holding a start time rather than a boolean is what makes the signal survive the
|
||||
/// expected daily purge burst — that spike raises the backlog for a drain's worth of
|
||||
/// reports and then clears this back to null, never reaching the dwell.
|
||||
/// <para>
|
||||
/// Null is set two different ways and they mean different things. A report whose backlog is
|
||||
/// at or below the threshold CLEARS it (the backlog demonstrably drained). A report whose
|
||||
/// backlog is <c>null</c> — an unreplicated site, or a failed poll — LEAVES IT UNCHANGED:
|
||||
/// null means "no reading", never zero, so a node that stops reporting mid-climb must not
|
||||
/// silently reset the clock, and an unreplicated site (which reports null forever) never
|
||||
/// starts one.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public DateTimeOffset? LocalDbBacklogAboveSince { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Gets the instant of the last online↔offline flip, or <c>null</c> if the
|
||||
/// site has never changed status since it was first observed. Answers "when
|
||||
|
||||
@@ -346,6 +346,103 @@ public class HealthPageTests : BunitContext
|
||||
});
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
// Sustained LocalDb replication backlog badge (register row 33)
|
||||
// ------------------------------------------------------------------
|
||||
|
||||
[Fact]
|
||||
public void Renders_ReplicationBacklogBadge_WhenBacklogSustainedPastTheDwell()
|
||||
{
|
||||
SeedStates(new SiteHealthState
|
||||
{
|
||||
SiteId = "site-a",
|
||||
IsOnline = true,
|
||||
LastHeartbeatAt = DateTimeOffset.UtcNow,
|
||||
// Default dwell is 15 minutes; 30 minutes above threshold is well past it.
|
||||
LocalDbBacklogAboveSince = DateTimeOffset.UtcNow.AddMinutes(-30),
|
||||
});
|
||||
|
||||
var cut = RenderHealthPage();
|
||||
|
||||
cut.WaitForAssertion(() =>
|
||||
{
|
||||
Assert.Contains("data-test=\"localdb-backlog-badge\"", cut.Markup);
|
||||
Assert.Contains("Replication backlog", cut.Markup);
|
||||
});
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void DoesNotRender_ReplicationBacklogBadge_DuringAPurgeBurst()
|
||||
{
|
||||
// THE false-positive this badge is shaped to avoid. The daily site_events purge is
|
||||
// CDC-captured, so the backlog spikes and then drains. Two minutes above the
|
||||
// threshold is a purge burst, not a fault, and must stay silent — otherwise the
|
||||
// badge fires on every replicated site every day and gets ignored.
|
||||
SeedStates(new SiteHealthState
|
||||
{
|
||||
SiteId = "site-a",
|
||||
IsOnline = true,
|
||||
LastHeartbeatAt = DateTimeOffset.UtcNow,
|
||||
LocalDbBacklogAboveSince = DateTimeOffset.UtcNow.AddMinutes(-2),
|
||||
});
|
||||
|
||||
var cut = RenderHealthPage();
|
||||
|
||||
cut.WaitForAssertion(() => Assert.Contains("site-a", cut.Markup));
|
||||
Assert.DoesNotContain("data-test=\"localdb-backlog-badge\"", cut.Markup);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void DoesNotRender_ReplicationBacklogBadge_ForAnUnreplicatedSite()
|
||||
{
|
||||
// Rig site-b/site-c report a null backlog forever, so the aggregator never gives them
|
||||
// an anchor. A deliberately-unreplicated site must never be badged.
|
||||
SeedStates(new SiteHealthState
|
||||
{
|
||||
SiteId = "site-b",
|
||||
IsOnline = true,
|
||||
LastHeartbeatAt = DateTimeOffset.UtcNow,
|
||||
LocalDbBacklogAboveSince = null,
|
||||
});
|
||||
|
||||
var cut = RenderHealthPage();
|
||||
|
||||
cut.WaitForAssertion(() => Assert.Contains("site-b", cut.Markup));
|
||||
Assert.DoesNotContain("data-test=\"localdb-backlog-badge\"", cut.Markup);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void DoesNotRender_ReplicationBacklogBadge_ForAnOfflineSite()
|
||||
{
|
||||
// An offline site's last-known anchor is stale by definition; the Offline badge is the
|
||||
// signal that matters, and stacking a backlog warning on top is noise.
|
||||
SeedStates(new SiteHealthState
|
||||
{
|
||||
SiteId = "site-a",
|
||||
IsOnline = false,
|
||||
LastHeartbeatAt = DateTimeOffset.UtcNow.AddMinutes(-30),
|
||||
LocalDbBacklogAboveSince = DateTimeOffset.UtcNow.AddMinutes(-30),
|
||||
});
|
||||
|
||||
var cut = RenderHealthPage();
|
||||
|
||||
cut.WaitForAssertion(() => Assert.Contains("site-a", cut.Markup));
|
||||
Assert.DoesNotContain("data-test=\"localdb-backlog-badge\"", cut.Markup);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Renders_LocalDbBacklogTrendChart()
|
||||
{
|
||||
// The retrospective half of row 33: a point-in-time tile can never answer "did that
|
||||
// spike drain?", which is the only question that distinguishes healthy from faulty.
|
||||
SeedSites("site-a");
|
||||
|
||||
var cut = RenderHealthPage();
|
||||
|
||||
cut.WaitForAssertion(() =>
|
||||
Assert.Contains("LocalDb Replication Backlog", cut.Markup));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Renders_OfflineSince_ForOfflineSiteWithStatusChange()
|
||||
{
|
||||
|
||||
@@ -47,6 +47,12 @@ public class KpiMetricAggregationTests
|
||||
(KpiSources.SiteHealth, "disabledInstances"),
|
||||
(KpiSources.SiteHealth, "auditBacklogPending"),
|
||||
(KpiSources.SiteHealth, "eventLogWriteFailures"),
|
||||
// Register row 33. Both are instantaneous readings, so both fold as Gauge (the
|
||||
// default) — summing an oplog depth per hour would be meaningless, and summing a
|
||||
// 0/1 link flag doubly so. Both are CONDITIONALLY emitted: a site that reports no
|
||||
// reading writes no sample rather than a fabricated 0.
|
||||
(KpiSources.SiteHealth, "localDbOplogBacklog"),
|
||||
(KpiSources.SiteHealth, "localDbReplicationConnected"),
|
||||
};
|
||||
|
||||
// The (source, metric) pairs that must fold as Rate (sum-per-hour).
|
||||
@@ -76,6 +82,8 @@ public class KpiMetricAggregationTests
|
||||
(KpiSources.SiteHealth, "connectionsDown"),
|
||||
(KpiSources.SiteHealth, "sfBufferDepth"),
|
||||
(KpiSources.SiteHealth, "deployedInstances"),
|
||||
(KpiSources.SiteHealth, "localDbOplogBacklog"),
|
||||
(KpiSources.SiteHealth, "localDbReplicationConnected"),
|
||||
};
|
||||
|
||||
public static IEnumerable<object[]> AllEmittedCases() =>
|
||||
|
||||
+57
@@ -93,4 +93,61 @@ public class HealthMonitoringOptionsValidatorTests
|
||||
Assert.True(result.Failed);
|
||||
Assert.Contains("MetricsStaleTimeout", result.FailureMessage);
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
// LocalDb sustained-backlog badge (register row 33)
|
||||
// ------------------------------------------------------------------
|
||||
|
||||
[Theory]
|
||||
[InlineData(0)]
|
||||
[InlineData(-1)]
|
||||
public void NonPositiveLocalDbBacklogThreshold_IsRejected(long threshold)
|
||||
{
|
||||
// An idle backlog of 0 already sits at or above a zero/negative threshold, so this
|
||||
// would badge every replicated site permanently.
|
||||
var result = Validate(new HealthMonitoringOptions { LocalDbBacklogThreshold = threshold });
|
||||
|
||||
Assert.True(result.Failed);
|
||||
Assert.Contains("LocalDbBacklogThreshold", result.FailureMessage);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ZeroLocalDbBacklogSustainedFor_IsRejected()
|
||||
{
|
||||
var result = Validate(new HealthMonitoringOptions
|
||||
{
|
||||
LocalDbBacklogSustainedFor = TimeSpan.Zero,
|
||||
});
|
||||
|
||||
Assert.True(result.Failed);
|
||||
Assert.Contains("LocalDbBacklogSustainedFor", result.FailureMessage);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void LocalDbBacklogSustainedFor_ShorterThanReportInterval_IsRejected()
|
||||
{
|
||||
// A dwell shorter than one report interval can be satisfied by a single sample, which
|
||||
// collapses the sustained check back into the bare threshold it replaced — and that
|
||||
// fires on every daily purge burst.
|
||||
var result = Validate(new HealthMonitoringOptions
|
||||
{
|
||||
ReportInterval = TimeSpan.FromSeconds(30),
|
||||
LocalDbBacklogSustainedFor = TimeSpan.FromSeconds(10),
|
||||
});
|
||||
|
||||
Assert.True(result.Failed);
|
||||
Assert.Contains("LocalDbBacklogSustainedFor", result.FailureMessage);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void DefaultSustainedWindow_ComfortablyOutlastsAPurgeDrain()
|
||||
{
|
||||
// Guards the intent of the default rather than its digits: the dwell must be long
|
||||
// enough that an expected purge spike drains inside it, or the badge becomes daily
|
||||
// noise and gets ignored.
|
||||
var options = new HealthMonitoringOptions();
|
||||
|
||||
Assert.True(options.LocalDbBacklogSustainedFor >= TimeSpan.FromMinutes(5));
|
||||
Assert.True(options.LocalDbBacklogSustainedFor >= options.ReportInterval);
|
||||
}
|
||||
}
|
||||
|
||||
+85
-1
@@ -99,11 +99,95 @@ public class SiteHealthKpiSampleSourceTests
|
||||
Assert.Equal(9, byMetric["auditBacklogPending"]);
|
||||
Assert.Equal(11, byMetric["eventLogWriteFailures"]);
|
||||
|
||||
// All 12 metrics emitted, exactly once each, for the one populated site.
|
||||
// 12 metrics for an UNREPLICATED site — this fixture leaves both LocalDb replication
|
||||
// fields null, and register row 33 emits those two conditionally rather than coercing
|
||||
// null to 0. A replicated site emits 14; see the LocalDb tests below.
|
||||
Assert.Equal(12, byMetric.Count);
|
||||
Assert.Equal(12, samples.Count);
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
// LocalDb replication metrics (register row 33)
|
||||
// ------------------------------------------------------------------
|
||||
|
||||
[Fact]
|
||||
public async Task CollectAsync_NullLocalDbFields_EmitsNeitherLocalDbMetric()
|
||||
{
|
||||
// THE null-semantics case, and the reason these two are not `?? 0` like the audit
|
||||
// backlog above. Null means "no reading" — an unreplicated site (rig site-b/site-c) or
|
||||
// a failed poll. A fabricated 0 backlog would render a pair that cannot read its own
|
||||
// oplog, or one deliberately not replicated, as perfectly healthy once a minute forever.
|
||||
var report = MinimalReport("site-a") with
|
||||
{
|
||||
LocalDbOplogBacklog = null,
|
||||
LocalDbReplicationConnected = null,
|
||||
};
|
||||
var aggregator = new StubAggregator
|
||||
{
|
||||
States = { ["site-a"] = new SiteHealthState { SiteId = "site-a", LatestReport = report } },
|
||||
};
|
||||
|
||||
var samples = await new SiteHealthKpiSampleSource(aggregator).CollectAsync(CapturedAt);
|
||||
|
||||
Assert.DoesNotContain(samples, s => s.Metric == "localDbOplogBacklog");
|
||||
Assert.DoesNotContain(samples, s => s.Metric == "localDbReplicationConnected");
|
||||
Assert.Equal(12, samples.Count);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task CollectAsync_ReplicatedSite_EmitsBothLocalDbMetrics()
|
||||
{
|
||||
var report = MinimalReport("site-a") with
|
||||
{
|
||||
LocalDbOplogBacklog = 4242,
|
||||
LocalDbReplicationConnected = true,
|
||||
};
|
||||
var aggregator = new StubAggregator
|
||||
{
|
||||
States = { ["site-a"] = new SiteHealthState { SiteId = "site-a", LatestReport = report } },
|
||||
};
|
||||
|
||||
var samples = await new SiteHealthKpiSampleSource(aggregator).CollectAsync(CapturedAt);
|
||||
|
||||
Assert.Equal(4242, samples.Single(s => s.Metric == "localDbOplogBacklog").Value);
|
||||
Assert.Equal(1, samples.Single(s => s.Metric == "localDbReplicationConnected").Value);
|
||||
Assert.Equal(14, samples.Count);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task CollectAsync_ZeroBacklog_IsStillEmitted()
|
||||
{
|
||||
// A real, measured 0 is genuinely different from "no reading" and must be recorded —
|
||||
// otherwise a healthy replicated site would be indistinguishable from an unreplicated
|
||||
// one, and the drain half of a purge burst would never appear in the series.
|
||||
var report = MinimalReport("site-a") with { LocalDbOplogBacklog = 0 };
|
||||
var aggregator = new StubAggregator
|
||||
{
|
||||
States = { ["site-a"] = new SiteHealthState { SiteId = "site-a", LatestReport = report } },
|
||||
};
|
||||
|
||||
var samples = await new SiteHealthKpiSampleSource(aggregator).CollectAsync(CapturedAt);
|
||||
|
||||
Assert.Equal(0, samples.Single(s => s.Metric == "localDbOplogBacklog").Value);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task CollectAsync_DisconnectedReplication_EmitsZeroNotAbsent()
|
||||
{
|
||||
// false and null are different states: a configured link that is DOWN versus no link
|
||||
// at all. Only the former is a fault, and it must show up as a 0 sample rather than a
|
||||
// gap, or a broken link would look exactly like an unreplicated site.
|
||||
var report = MinimalReport("site-a") with { LocalDbReplicationConnected = false };
|
||||
var aggregator = new StubAggregator
|
||||
{
|
||||
States = { ["site-a"] = new SiteHealthState { SiteId = "site-a", LatestReport = report } },
|
||||
};
|
||||
|
||||
var samples = await new SiteHealthKpiSampleSource(aggregator).CollectAsync(CapturedAt);
|
||||
|
||||
Assert.Equal(0, samples.Single(s => s.Metric == "localDbReplicationConnected").Value);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task CollectAsync_NullAuditBacklog_EmitsZeroAuditBacklogPending()
|
||||
{
|
||||
|
||||
@@ -0,0 +1,176 @@
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Register row 33: the sustained-backlog anchor behind the Health dashboard's
|
||||
/// "Replication backlog" badge.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The design constraint these tests exist to pin: a bare "backlog > threshold" badge is
|
||||
/// unusable, because the daily `site_events` retention purge is CDC-captured like any other
|
||||
/// write and jumps the backlog by the deleted batch size before it drains. That spike is
|
||||
/// expected and documented. So the aggregator holds the instant the backlog FIRST rose above
|
||||
/// the threshold, and the badge only appears once the configured dwell has elapsed — healthy
|
||||
/// means the backlog comes back down, which no point-in-time check can express.
|
||||
/// </remarks>
|
||||
public sealed class LocalDbBacklogSustainedTests
|
||||
{
|
||||
private const long Threshold = 1000;
|
||||
|
||||
[Fact]
|
||||
public void BacklogAboveThreshold_StartsTheClock()
|
||||
{
|
||||
var (aggregator, time) = NewAggregator();
|
||||
var start = time.GetUtcNow();
|
||||
|
||||
aggregator.ProcessReport(Report("site-a", 1, backlog: Threshold + 1));
|
||||
|
||||
Assert.Equal(start, aggregator.GetSiteState("site-a")!.LocalDbBacklogAboveSince);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void BacklogStayingAbove_KeepsTheOriginalStartInstant()
|
||||
{
|
||||
// The window must be sustained, not sliding: if each above-threshold report reset the
|
||||
// anchor, the dwell would never elapse and the badge could never appear at all.
|
||||
var (aggregator, time) = NewAggregator();
|
||||
var start = time.GetUtcNow();
|
||||
|
||||
aggregator.ProcessReport(Report("site-a", 1, backlog: Threshold + 1));
|
||||
time.Advance(TimeSpan.FromMinutes(10));
|
||||
aggregator.ProcessReport(Report("site-a", 2, backlog: Threshold + 5000));
|
||||
|
||||
Assert.Equal(start, aggregator.GetSiteState("site-a")!.LocalDbBacklogAboveSince);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void BacklogAtOrBelowThreshold_ClearsTheClock()
|
||||
{
|
||||
// The purge-burst exit path. A spike that drains must discard the anchor well before
|
||||
// the dwell elapses, so the expected daily burst never badges the site.
|
||||
var (aggregator, time) = NewAggregator();
|
||||
|
||||
aggregator.ProcessReport(Report("site-a", 1, backlog: Threshold + 50_000));
|
||||
time.Advance(TimeSpan.FromMinutes(2));
|
||||
aggregator.ProcessReport(Report("site-a", 2, backlog: 0));
|
||||
|
||||
Assert.Null(aggregator.GetSiteState("site-a")!.LocalDbBacklogAboveSince);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void BacklogExactlyAtThreshold_DoesNotStartTheClock()
|
||||
{
|
||||
// The comparison is strictly greater-than, so the threshold is the highest ACCEPTABLE
|
||||
// depth rather than the lowest alarming one.
|
||||
var (aggregator, _) = NewAggregator();
|
||||
|
||||
aggregator.ProcessReport(Report("site-a", 1, backlog: Threshold));
|
||||
|
||||
Assert.Null(aggregator.GetSiteState("site-a")!.LocalDbBacklogAboveSince);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void UnreplicatedSite_NeverAccumulatesAnAnchor()
|
||||
{
|
||||
// Rig site-b/site-c: a null backlog forever. Null means "no reading", never zero, so
|
||||
// it must never start a clock — otherwise every unreplicated site would eventually be
|
||||
// badged for a backlog it does not have.
|
||||
var (aggregator, time) = NewAggregator();
|
||||
|
||||
aggregator.ProcessReport(Report("site-b", 1, backlog: null));
|
||||
time.Advance(TimeSpan.FromHours(6));
|
||||
aggregator.ProcessReport(Report("site-b", 2, backlog: null));
|
||||
|
||||
Assert.Null(aggregator.GetSiteState("site-b")!.LocalDbBacklogAboveSince);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void NullBacklogMidClimb_LeavesTheAnchorUntouched()
|
||||
{
|
||||
// A failed poll on a replicated node reports null. That is not evidence the backlog
|
||||
// drained, so it must not clear the clock — a node that goes blind mid-climb would
|
||||
// otherwise silently reset its own dwell and never be badged.
|
||||
var (aggregator, time) = NewAggregator();
|
||||
var start = time.GetUtcNow();
|
||||
|
||||
aggregator.ProcessReport(Report("site-a", 1, backlog: Threshold + 1));
|
||||
time.Advance(TimeSpan.FromMinutes(5));
|
||||
aggregator.ProcessReport(Report("site-a", 2, backlog: null));
|
||||
|
||||
Assert.Equal(start, aggregator.GetSiteState("site-a")!.LocalDbBacklogAboveSince);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void FirstReportForANewSite_CanStartTheClock()
|
||||
{
|
||||
// The registration path is a separate branch from the update path; a site whose very
|
||||
// first report is already backlogged must not be missed.
|
||||
var (aggregator, time) = NewAggregator();
|
||||
var start = time.GetUtcNow();
|
||||
|
||||
aggregator.ProcessReport(Report("brand-new", 1, backlog: Threshold + 1));
|
||||
|
||||
Assert.Equal(start, aggregator.GetSiteState("brand-new")!.LocalDbBacklogAboveSince);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ThresholdIsConfigurable()
|
||||
{
|
||||
var (aggregator, _) = NewAggregator(threshold: 10);
|
||||
|
||||
aggregator.ProcessReport(Report("site-a", 1, backlog: 11));
|
||||
|
||||
Assert.NotNull(aggregator.GetSiteState("site-a")!.LocalDbBacklogAboveSince);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void StaleReport_DoesNotDisturbTheAnchor()
|
||||
{
|
||||
// Out-of-order reports are rejected wholesale by the sequence-number guard; the anchor
|
||||
// must ride along with that rejection rather than being updated from stale data.
|
||||
var (aggregator, time) = NewAggregator();
|
||||
var start = time.GetUtcNow();
|
||||
|
||||
aggregator.ProcessReport(Report("site-a", 5, backlog: Threshold + 1));
|
||||
time.Advance(TimeSpan.FromMinutes(5));
|
||||
aggregator.ProcessReport(Report("site-a", 2, backlog: 0));
|
||||
|
||||
Assert.Equal(start, aggregator.GetSiteState("site-a")!.LocalDbBacklogAboveSince);
|
||||
}
|
||||
|
||||
private static (CentralHealthAggregator Aggregator, TestTimeProvider Time) NewAggregator(
|
||||
long threshold = Threshold)
|
||||
{
|
||||
var time = new TestTimeProvider(DateTimeOffset.UtcNow);
|
||||
var opts = new HealthMonitoringOptions
|
||||
{
|
||||
OfflineTimeout = TimeSpan.FromSeconds(60),
|
||||
LocalDbBacklogThreshold = threshold,
|
||||
};
|
||||
var aggregator = new CentralHealthAggregator(
|
||||
Options.Create(opts), NullLogger<CentralHealthAggregator>.Instance, time);
|
||||
return (aggregator, time);
|
||||
}
|
||||
|
||||
private static SiteHealthReport Report(string siteId, long seq, long? backlog) =>
|
||||
new(
|
||||
SiteId: siteId,
|
||||
SequenceNumber: seq,
|
||||
ReportTimestamp: DateTimeOffset.UtcNow,
|
||||
DataConnectionStatuses: new Dictionary<string, ConnectionHealth>(),
|
||||
TagResolutionCounts: new Dictionary<string, TagResolutionStatus>(),
|
||||
ScriptErrorCount: 0,
|
||||
AlarmEvaluationErrorCount: 0,
|
||||
StoreAndForwardBufferDepths: new Dictionary<string, int>(),
|
||||
DeadLetterCount: 0,
|
||||
DeployedInstanceCount: 0,
|
||||
EnabledInstanceCount: 0,
|
||||
DisabledInstanceCount: 0)
|
||||
{
|
||||
LocalDbOplogBacklog = backlog,
|
||||
};
|
||||
}
|
||||
Reference in New Issue
Block a user