feat(health): KPI history + sustained-backlog badge for LocalDb replication (row 33)
LocalDbOplogBacklog was reported and Prometheus-exported with nothing consuming it: not sampled into KPI history, so a purge-burst spike could not be reviewed after the fact, and no threshold treatment anywhere. Both halves shipped. KPI SAMPLING. SiteHealthKpiSampleSource now emits localDbOplogBacklog and localDbReplicationConnected (0/1), and the backlog is charted on the Health dashboard's per-site trend panel. Charting is the point: a point-in-time tile can never answer 'did that spike drain?', which is the only thing separating an expected purge burst from a stalled peer. Both are CONDITIONALLY emitted rather than coerced with ?? 0 like the auditBacklogPending metric beside them. Both health-report fields are nullable and null means 'no reading' — replication not configured (rig site-b/site-c) or a failed poll — never zero. A fabricated 0 would render a pair that cannot read its own oplog, or one deliberately unreplicated, as perfectly healthy once a minute forever, burying the real series. So a site emits 12 metrics when unreplicated and 14 when replicated, and a GAP in these two series truthfully means 'not replicated / not reporting'. Note false and null differ for the connected flag: a configured link that is DOWN samples as 0; no link samples as nothing. Both fold as Gauge (summing an oplog depth, or a 0/1 flag, per hour is meaningless). THRESHOLD SEMANTICS — sustained, not instantaneous. This is the design, not a refinement. The site_events retention purge is CDC-captured like any other write, so the backlog jumps by the deleted batch and drains as the peer acks it; a bare 'backlog > threshold' badge would fire on every replicated site every day and be ignored within a week. Healthy is defined by the backlog RETURNING to ~0, which only a dwell can express. CentralHealthAggregator holds LocalDbBacklogAboveSince — the instant the backlog first rose above LocalDbBacklogThreshold (default 1000) — and the badge appears only after LocalDbBacklogSustainedFor (default 15 min). A report at or below the threshold clears the anchor; one above leaves the ORIGINAL instant intact, so the window is sustained rather than sliding. The null case is deliberately a third branch: a null backlog leaves the anchor UNCHANGED rather than clearing it. Null is evidence neither of a drain nor of a backlog, so a node going blind mid-climb must not silently reset its own dwell, and an unreplicated site (null forever) never accumulates an anchor and can never be badged. Validation rejects a non-positive threshold (0 would badge every replicated site permanently, since an idle backlog already sits at it) and a dwell shorter than one ReportInterval (satisfiable by a single sample, which collapses the check back into the bare threshold it replaces). DOC CORRECTION worth flagging: this row's premise was 'mirror the SiteAuditBacklog configurable threshold badge'. That badge does not exist — Component-HealthMonitoring.md described it in two places but no option, no comparison and no markup were ever implemented. The two stale claims are corrected here rather than left to send the next reader looking for it; the LocalDb badge is built fresh on the Metrics-stale badge's pattern, which is a real configurable-threshold badge. Tests: HealthMonitoring 115 (9 new aggregator tests incl. purge-burst-drains, unreplicated-never-anchors, null-mid-climb, exact-threshold, stale-report; 5 new validator tests; 4 new sampler tests incl. the null-semantics and false-vs-null cases); CentralUI 1042 (5 new badge/chart tests incl. the purge-burst negative control); Commons Kpi 93; KpiHistory 54.
This commit is contained in:
Joseph Doherty
@@ -255,6 +255,32 @@ LocalDbReplicationConnected staying true with the backlog returning to ~0 is the
|
||||
An unreplicated node logs only the first line. If a backlog spike has no such line near it in the
|
||||
active node's log, the purge is *not* the explanation and the spike is worth investigating.
|
||||
|
||||
**Two dashboard surfaces make this reviewable rather than a log-archaeology exercise** (register
|
||||
row 33):
|
||||
|
||||
- **The "Replication backlog" badge** on the site tile of the Health dashboard. It is deliberately
|
||||
a *sustained* signal, not a bare threshold: the badge appears only once the backlog has stayed
|
||||
above `ScadaBridge:HealthMonitoring:LocalDbBacklogThreshold` (default **1000**) continuously for
|
||||
`ScadaBridge:HealthMonitoring:LocalDbBacklogSustainedFor` (default **15 minutes**). The daily
|
||||
purge burst described above drains in seconds to a couple of minutes, so it never reaches the
|
||||
dwell and never badges the site — which is the entire reason the dwell exists. A bare threshold
|
||||
would fire once a day on every replicated site and be ignored within a week. The dwell is
|
||||
validated to be at least one `ReportInterval`, since a shorter window could be satisfied by a
|
||||
single sample and would collapse back into a bare threshold.
|
||||
- **The "LocalDb Replication Backlog" trend chart** in the dashboard's per-site Site Health Trends
|
||||
panel, backed by the KPI history store (sampled every minute, retained 90 days raw / 365 days
|
||||
hourly). This is what answers the question a point-in-time reading cannot: *did that spike
|
||||
drain?* A past purge burst appears as a spike that returns to ~0; a real fault appears as a
|
||||
level that never comes back down.
|
||||
|
||||
**Both surfaces are silent on an unreplicated site, by design.** `LocalDbOplogBacklog` and
|
||||
`LocalDbReplicationConnected` are nullable, and null means "no reading", never zero. The KPI
|
||||
sampler therefore writes **no sample at all** rather than a fabricated 0, so site-b/site-c show an
|
||||
empty chart rather than a flat zero line, and they never accumulate the dwell anchor the badge
|
||||
keys off. A gap in those series means "not replicated / not reporting" — which is the truth. Note
|
||||
that `false` and null differ for the connected flag: a configured link that is DOWN samples as 0,
|
||||
while no link at all samples as nothing.
|
||||
|
||||
### Site Pair Upgrades — stop and start BOTH nodes together
|
||||
|
||||
**A rolling upgrade of a site pair, one node at a time, is no longer supported.** It worked while
|
||||
|
||||
@@ -34,7 +34,7 @@ Site clusters (metric collection and reporting). Central cluster (aggregation an
|
||||
| Notification Outbox queue depth | Notification Outbox (central) | Count of `Pending` + `Retrying` notifications — central-computed, not site-reported |
|
||||
| Notification Outbox stuck count | Notification Outbox (central) | Count of `Pending` / `Retrying` notifications older than the configurable stuck-age threshold — central-computed, not site-reported |
|
||||
| Notification Outbox parked count | Notification Outbox (central) | Count of `Parked` notifications — central-computed, not site-reported |
|
||||
| `SiteAuditBacklog` | Audit Log (site) | Count of `Pending` rows in the site-local `AuditLog` plus oldest-pending-age plus on-disk bytes. A configurable threshold drives a Health dashboard warning on the affected site tile. |
|
||||
| `SiteAuditBacklog` | Audit Log (site) | Count of `Pending` rows in the site-local `AuditLog` plus oldest-pending-age plus on-disk bytes. Surfaced on the Audit KPI tiles (which border-highlight any non-zero backlog) and sampled into KPI history as `auditBacklogPending`. **No configurable per-site threshold badge exists for this metric** — earlier revisions of this document described one, but none was ever implemented and no such option exists. The only configurable sustained-threshold site badge is the LocalDb replication backlog (register row 33, below). |
|
||||
| `SiteAuditWriteFailures` | Audit Log (site) | Count of failed hot-path audit appends at the site since the last health report. |
|
||||
| `AuditRedactionFailure` | Audit Log (central) | Count of payload redactor errors (over-redacted payloads, safety-net hit) since the last interval. |
|
||||
| `LocalDbReplicationConnected` | Consolidated site LocalDb | Whether a peer sync session is currently running on this node. **Nullable — null means "no reading", not "disconnected".** Reported only by a node with replication configured. |
|
||||
@@ -90,7 +90,7 @@ The Audit Log spans both sites (hot-path append + telemetry forward) and central
|
||||
|
||||
- **Audit volume** — events/min landing in the central `AuditLog` table, shown global plus per-site sparkline; sourced from the Audit Log component on the active central node.
|
||||
- **Audit error rate** — percent of central `AuditLog` rows with `Status` other than `Success` / `Delivered` / `Enqueued` over a rolling 5-minute window. This is the operational error rate of audited operations (HTTP 5xx, transient failures, parked deliveries, etc.) — NOT the audit writer's own health. Audit-writer issues surface separately via `AuditRedactionFailure`.
|
||||
- **Audit backlog** — global aggregate of `SiteAuditBacklog` across reporting sites (count of `Pending` site-local audit rows, oldest pending age, on-disk bytes); click drills into a per-site breakdown. The per-site tile surfaces a warning badge when its `SiteAuditBacklog` crosses the configurable threshold.
|
||||
- **Audit backlog** — global aggregate of `SiteAuditBacklog` across reporting sites (count of `Pending` site-local audit rows, oldest pending age, on-disk bytes); click drills into a per-site breakdown. The tile border-highlights a non-zero aggregate backlog; there is **no** configurable per-site `SiteAuditBacklog` threshold badge (see the note in Monitored Metrics — earlier revisions described one that was never built).
|
||||
|
||||
These tiles are **point-in-time** like the Notification Outbox and Site Call Audit KPI tiles — no time-series store; consistent with Health Monitoring's "current status only" philosophy. The site-scoped `SiteAuditBacklog` / `SiteAuditWriteFailures` metrics arrive in the existing site health report; the central-scoped `AuditRedactionFailure` metric is central-computed alongside the existing central KPIs.
|
||||
|
||||
@@ -101,6 +101,46 @@ These tiles are **point-in-time** like the Notification Outbox and Site Call Aud
|
||||
- Notification Outbox and Site Call Audit KPIs are not stored by Health Monitoring; they are computed point-in-time from the central `Notifications` and `SiteCalls` tables respectively each time the dashboard refreshes — consistent with the current-status-only philosophy.
|
||||
- Site connectivity history (online/offline transitions) may optionally be logged via the Audit Log or a separate mechanism if needed in the future.
|
||||
|
||||
## Sustained LocalDb Replication Backlog Badge
|
||||
|
||||
The per-site tile shows a **Replication backlog** warning badge when `LocalDbOplogBacklog` has
|
||||
stayed above `LocalDbBacklogThreshold` (default **1000**) continuously for
|
||||
`LocalDbBacklogSustainedFor` (default **15 minutes**). Both are configurable under
|
||||
`ScadaBridge:HealthMonitoring` and validated at startup.
|
||||
|
||||
- **Sustained, not instantaneous — and that is the design, not a refinement.** The daily
|
||||
`site_events` retention purge and the storage-cap trim are CDC-captured like any other write, so
|
||||
the backlog jumps by the deleted batch size and drains as the peer acks it. A bare
|
||||
`backlog > threshold` badge would therefore fire on every replicated site every day. Healthy is
|
||||
defined by the backlog *returning to ~0*, which only a dwell can express. `LocalDbBacklogSustainedFor`
|
||||
is validated to be at least one `ReportInterval`: a shorter dwell could be satisfied by a single
|
||||
sample and would collapse back into the bare threshold.
|
||||
- **Implementation.** `CentralHealthAggregator` holds `SiteHealthState.LocalDbBacklogAboveSince`,
|
||||
the instant the backlog first rose above the threshold. Reports at or below the threshold clear
|
||||
it; reports above it leave the original instant intact (a sustained window, not a sliding one).
|
||||
- **Null handling is load-bearing.** A report with a `null` backlog leaves the anchor *unchanged*
|
||||
rather than clearing it — null means "no reading" (unreplicated site, or a failed poll), which is
|
||||
evidence neither of a drain nor of a backlog. An unreplicated site (rig site-b/site-c) reports
|
||||
null forever, so it never accumulates an anchor and can never be badged.
|
||||
- Offline sites are not badged; the Offline badge is the signal that matters there.
|
||||
|
||||
## KPI History Sampling of Replication State
|
||||
|
||||
`SiteHealthKpiSampleSource` samples `localDbOplogBacklog` and `localDbReplicationConnected` (0/1)
|
||||
into the KPI history store, and the backlog is charted in the Health dashboard's per-site Site
|
||||
Health Trends panel as **LocalDb Replication Backlog**. This is the retrospective half of the
|
||||
signal: a point-in-time tile can never answer "did that spike drain?", which is the only thing
|
||||
distinguishing an expected purge burst from a stalled peer.
|
||||
|
||||
Both metrics are **conditionally emitted** — a site reporting `null` writes no sample rather than a
|
||||
coerced `0`, unlike the `auditBacklogPending` metric alongside them. A fabricated 0 would render a
|
||||
pair that cannot read its own oplog, or one deliberately not replicated, as perfectly healthy once
|
||||
a minute forever. Consequently a site's sample count varies: **12 metrics for an unreplicated site,
|
||||
14 for a replicated one**, and a gap in these two series truthfully means "not replicated / not
|
||||
reporting". Note `false` and `null` are different states for the connected flag — a configured link
|
||||
that is down samples as `0`; no link at all samples as nothing. Both fold as **Gauge** in the hourly
|
||||
rollup catalog (summing an oplog depth, or a 0/1 flag, per hour would be meaningless).
|
||||
|
||||
## Prometheus Metrics (site node)
|
||||
|
||||
Distinct from the health *report* above: these are scraped from the site node's `/metrics`
|
||||
@@ -151,4 +191,4 @@ endpoint (`MetricsPort`, default 8084) rather than sent to central.
|
||||
|
||||
- **Central UI**: Health Monitoring Dashboard displays aggregated metrics.
|
||||
- **Communication Layer**: Health reports flow as periodic messages.
|
||||
- **KPI History (#26)**: emits `IKpiSampleSource` (`SiteHealthKpiSampleSource`, per-Site) consumed by the KpiHistory recorder (#26). It reads the in-memory `ICentralHealthAggregator.GetAllSiteStates()` (no DB), turning the per-site snapshot — previously sequence-numbered every 30s but discarded — into 12 sampled `KpiSample` metrics per site, persisted every minute: `connectionsUp`/`connectionsDown`, `scriptErrors`, `alarmEvalErrors`, `sfBufferDepth`, `deadLetters`, `parkedMessages`, `deployedInstances`/`enabledInstances`/`disabledInstances`, `auditBacklogPending`, `eventLogWriteFailures`. Of these, **only four are charted** in the dashboard's per-site `KpiTrendChart` panel — `connectionsDown`, `scriptErrors`, `sfBufferDepth`, and `deadLetters` (the metrics carried in the shared `KpiMetrics.SiteHealth` Commons catalog and fetched by `Health.razor`'s trend selector). The remaining eight are **sampled and persisted but not (yet) charted** — they are retained in the `KpiSample` store and available for future trend surfaces or ad-hoc query. The synthetic central self-report (`CentralHealthReportLoop.CentralSiteId = "$central"`, fed by the leader-only report loop via `CollectReport($central)`) appears in `GetAllSiteStates()` and is **intentionally sampled into the per-Site KPI store as a real `KpiScopes.Site` series** keyed `$central`; the Central UI deliberately **pins `$central` first** in the per-site trend selector (rendered as "Central Cluster"), so the central cluster's own trends sit alongside the real sites'. (`$central`'s zero-valued connection/instance/S&F metrics reflect the central self-report carrying no site-runtime data, not a defect.) See [Component-KpiHistory.md](Component-KpiHistory.md).
|
||||
- **KPI History (#26)**: emits `IKpiSampleSource` (`SiteHealthKpiSampleSource`, per-Site) consumed by the KpiHistory recorder (#26). It reads the in-memory `ICentralHealthAggregator.GetAllSiteStates()` (no DB), turning the per-site snapshot — previously sequence-numbered every 30s but discarded — into 12–14 sampled `KpiSample` metrics per site, persisted every minute: `connectionsUp`/`connectionsDown`, `scriptErrors`, `alarmEvalErrors`, `sfBufferDepth`, `deadLetters`, `parkedMessages`, `deployedInstances`/`enabledInstances`/`disabledInstances`, `auditBacklogPending`, `eventLogWriteFailures`, plus the two conditionally-emitted LocalDb replication metrics `localDbOplogBacklog` and `localDbReplicationConnected` (register row 33 — emitted only when the site actually reports a reading, hence 12 for an unreplicated site and 14 for a replicated one; see *KPI History Sampling of Replication State* above). Of these, **only five are charted** in the dashboard's per-site `KpiTrendChart` panel — `connectionsDown`, `scriptErrors`, `sfBufferDepth`, `deadLetters`, and `localDbOplogBacklog` (the metrics carried in the shared `KpiMetrics.SiteHealth` Commons catalog and fetched by `Health.razor`'s trend selector). The rest are **sampled and persisted but not (yet) charted** — they are retained in the `KpiSample` store and available for future trend surfaces or ad-hoc query. The synthetic central self-report (`CentralHealthReportLoop.CentralSiteId = "$central"`, fed by the leader-only report loop via `CollectReport($central)`) appears in `GetAllSiteStates()` and is **intentionally sampled into the per-Site KPI store as a real `KpiScopes.Site` series** keyed `$central`; the Central UI deliberately **pins `$central` first** in the per-site trend selector (rendered as "Central Cluster"), so the central cluster's own trends sit alongside the real sites'. (`$central`'s zero-valued connection/instance/S&F metrics reflect the central self-report carrying no site-runtime data, not a defect.) See [Component-KpiHistory.md](Component-KpiHistory.md).
|
||||
|
||||
@@ -143,6 +143,17 @@ Reads `ICentralHealthAggregator.GetAllSiteStates()` (in-memory, no DB). Scope: *
|
||||
- `disabledInstances`
|
||||
- `auditBacklogPending`
|
||||
- `eventLogWriteFailures`
|
||||
- `localDbOplogBacklog` — **conditionally emitted** (register row 33)
|
||||
- `localDbReplicationConnected` — **conditionally emitted** (register row 33), 0/1
|
||||
|
||||
The last two are written **only when the site actually reports a reading**. Both underlying health-report
|
||||
fields are nullable and null means "no reading" (replication not configured, or a failed poll), never
|
||||
zero — so unlike `auditBacklogPending` above, they are not coerced with `?? 0`. A site therefore emits
|
||||
**12 metrics when unreplicated and 14 when replicated**, and a gap in these two series truthfully means
|
||||
"not replicated / not reporting" rather than a flat zero line that would render an unreplicated pair, or
|
||||
one that cannot read its own oplog, as perfectly healthy. Both fold as **Gauge** in
|
||||
`KpiMetricAggregationCatalog` (they are instantaneous readings; summing an oplog depth or a 0/1 flag per
|
||||
hour would be meaningless). `localDbOplogBacklog` is charted on the Health dashboard's per-site trend panel.
|
||||
|
||||
## Query + UI
|
||||
|
||||
|
||||
Reference in New Issue
Block a user