ScadaBridge/docs/requirements/Component-HealthMonitoring.md

# Component: Health Monitoring

## Purpose

The Health Monitoring component collects and reports operational health metrics from site clusters to the central cluster, providing engineers with visibility into the status of the distributed system.

## Location

Site clusters (metric collection and reporting). Central cluster (aggregation and display).

## Responsibilities

### Site Side
- Collect health metrics from local subsystems.
- Periodically report metrics to the central cluster via the Communication Layer.

### Central Side
- Receive and store health metrics from all sites.
- Detect site connectivity status (online/offline) based on heartbeat presence.
- Present health data in the Central UI dashboard.

## Monitored Metrics

| Metric | Source | Description |
|--------|--------|-------------|
| Site online/offline | Communication Layer | Whether the site is reachable (based on heartbeat) |
| Active/standby node status | Cluster Infrastructure | Which node is active, which is standby |
| Data connection health | Data Connection Layer | Connected/disconnected/reconnecting per data connection |
| Tag resolution counts | Data Connection Layer | Per connection: total subscribed tags vs. successfully resolved tags |
| Script error rates | Site Runtime (Script Actors) | Frequency of script failures |
| Alarm evaluation error rates | Site Runtime (Alarm Actors) | Frequency of alarm evaluation failures |
| Store-and-forward buffer depth | Store-and-Forward Engine | Pending messages by category — external, notification (notifications awaiting forward to central), DB write |
| Dead letter count | Akka.NET EventStream | Messages sent to actors that no longer exist — indicates stale references or timing issues |
| Notification Outbox queue depth | Notification Outbox (central) | Count of `Pending` + `Retrying` notifications — central-computed, not site-reported |
| Notification Outbox stuck count | Notification Outbox (central) | Count of `Pending` / `Retrying` notifications older than the configurable stuck-age threshold — central-computed, not site-reported |
| Notification Outbox parked count | Notification Outbox (central) | Count of `Parked` notifications — central-computed, not site-reported |
| `SiteAuditBacklog` | Audit Log (site) | Count of `Pending` rows in the site-local `AuditLog` plus oldest-pending-age plus on-disk bytes. Surfaced on the Audit KPI tiles (which border-highlight any non-zero backlog) and sampled into KPI history as `auditBacklogPending`. **No configurable per-site threshold badge exists for this metric** — earlier revisions of this document described one, but none was ever implemented and no such option exists. The only configurable sustained-threshold site badge is the LocalDb replication backlog (register row 33, below). |
| `SiteAuditWriteFailures` | Audit Log (site) | Count of failed hot-path audit appends at the site since the last health report. |
| `AuditRedactionFailure` | Audit Log (central) | Count of payload redactor errors (over-redacted payloads, safety-net hit) since the last interval. |
| `LocalDbReplicationConnected` | Consolidated site LocalDb | Whether a peer sync session is currently running on this node. **Nullable — null means "no reading", not "disconnected".** Reported only by a node with replication configured; an unconfigured node reports **null**, never `false` (register row 80 — see *Unconfigured replication reports null* below). |
| `StartupReconcilerActive` | Host (site) | Whether this node spawned its `SiteReconciliationActor` at boot. **Tri-state:** null = not reported (the central self-report, or a pre-row-73 site build), `true` = running, `false` = the node booted with no `IDeploymentConfigFetcher` and therefore has **no startup self-heal**. Latched once at boot by the Host — not polled. Drives the *No self-heal* dashboard badge (register row 73). |
| `SecretMirrorStatuses` | Site Runtime (`DeployedSecretsMirror` via the replicated `secret_mirror_status` table) | Every site node's deployed-secrets mirror status — node name, when the node last wrote the row (on a failure-set change, at start-up, or on the 10-min refresh), and its current failures (`{SecretName, Message}`; names and messages only, never values). Review finding M1: the **standby's** failures ride the **active** node's report, because each node writes its own row to a LocalDb-replicated table and the active node's `SecretMirrorStatusReporter` reads them all. **Nullable — null = no data** (central self-report, a pre-M1 site, reporter not yet run); an empty list = wired, no row yet; a node with an empty `Failures` list is healthy. Point-in-time (not drained on collect). Wire: `SiteHealthReportDto.secret_mirror_statuses` (field **33**, a presence wrapper so null ≠ empty). Drives the *Secret mirror failing* dashboard badge + per-node failure list. On an unreplicated pair only the active node's own row appears. A node that has not refreshed its row for `SiteRuntime:SecretMirrorStatusStaleAfter` (default 1 h — decommissioned, renamed or down) is left off and its row deleted. |
| `LocalDbOplogBacklog` | Consolidated site LocalDb | Unacked oplog depth (Prometheus `localdb_oplog_depth`). **Nullable — null means "no reading", NOT "connected with an empty backlog"** (a failed poll rendered as 0 would report a pair that cannot read its own oplog as perfectly healthy). **Expected transient spikes:** the daily `site_events` retention purge and the storage-cap trim are CDC-captured like any other write, so the backlog jumps by the batch size at purge time and drains as the peer acks it — see `Component-SiteEventLogging.md` → Storage and `docs/deployment/topology-guide.md` → *Reading the replication backlog* for the healthy-versus-faulty signature. |

## Reporting Protocol

- Sites send a **health report message** to central at a configurable interval (default: **30 seconds**).
- Each report is a **flat snapshot** containing the current values of all monitored metrics, a **monotonic sequence number**, and the **report timestamp** from the site. Central replaces the previous state for that site only if the incoming sequence number is higher than the last received — this prevents stale reports (e.g., delayed in transit or from a pre-failover node) from overwriting newer state.
- Central tracks two **distinct** signals per site — do not conflate them:
  - **Liveness (online/offline)**: driven by the **last signal of any kind** (a full report *or* a transport heartbeat, the latter arriving every ~5s from any reachable site node). If central receives no signal within `OfflineTimeout` (default: **60 seconds**), the site is marked **offline**. Because heartbeats are frequent, this only fires on genuine total loss of contact — not during a single-node failover, when the surviving node keeps heartbeating. The synthetic *central* self-report site has no heartbeat source and uses the longer `CentralOfflineTimeout` (default: **3 minutes**) so a single missed self-report does not flap it offline.
  - **Metrics staleness**: an **online** site (heartbeats still arriving) that has produced no full **report** within `MetricsStaleTimeout` (default: **2 minutes**) is flagged **metrics stale**. This surfaces the failure mode where a site's report pipeline (`HealthReportSender`) has died but the node is otherwise reachable — previously such a site showed "online with frozen metrics forever". A site that has **never** delivered a report is anchored on its first-contact time (`FirstSeenAt`), so a report pipeline dead from first boot is also flagged after `MetricsStaleTimeout` (previously such a site showed online-with-no-metrics indefinitely). `MetricsStaleTimeout` must be positive and no shorter than the report interval.
- **Online recovery**: when central receives a health report from a site that was marked offline, the site is automatically marked **online** and the metrics-stale flag cleared (a fresh report proves the pipeline is alive); a heartbeat alone also restores **online** but leaves the metrics-stale flag untouched. No manual acknowledgment required.
- **Status-transition timestamp**: each online↔offline flip stamps `LastStatusChangeAt`, so the UI can answer "since when has this site been offline/online" rather than only "is it offline now". Report/heartbeat traffic that leaves the status unchanged does not move it.

## Error Rate Metrics

Script error rates and alarm evaluation error rates are calculated as **raw counts per reporting interval**:

- The site maintains a counter for each metric that increments on every failure.
- Each health report includes the count since the last report. The counter resets after each report is sent.
- Central displays these as "X errors in the last 30 seconds" (or whatever the configured interval is).
- **Script errors** include all failures that prevent a script from completing successfully: unhandled exceptions, timeouts, recursion limit violations, and any other error condition.
- **Alarm evaluation errors** include all failures during alarm condition evaluation.
- For detailed diagnostics (error types, stack traces, affected instances), operators use the **Site Event Log Viewer** — the health dashboard is for quick triage, not forensics.

## Notification Outbox KPIs

The Notification Outbox is a **central** component, so its KPIs are **central-computed** rather than collected from sites and carried in the site health report:

- The dashboard surfaces three **headline** outbox KPIs: **queue depth** (`Pending` + `Retrying`), **stuck count** (`Pending` / `Retrying` rows older than the configurable stuck-age threshold), and **parked count** (`Parked`).
- The Notification Outbox component computes these on demand from the central `Notifications` table; the health dashboard polls it for the headline tiles.
- The fuller KPI set — which also includes **delivered (last interval)** and **oldest pending age** — lives on the Central UI **Notification Outbox** page, not the health dashboard.
- Outbox KPIs are **point-in-time**, computed on demand from the `Notifications` table. There is no time-series store — consistent with Health Monitoring's "current status only" philosophy. The outbox's own ~1-year row retention answers historical questions directly.

These are distinct from the site-reported **Store-and-forward buffer depth** notification metric, which now covers the **site→central leg** — notifications still buffered in a site's Store-and-Forward Engine awaiting forward to central — and remains part of the site health report.

## Site Call Audit KPIs

The Site Call Audit is a **central** component, so its KPIs — like the Notification Outbox's — are **central-computed** rather than collected from sites and carried in the site health report:

- The dashboard surfaces Site Call Audit **headline** KPI tiles alongside the existing Notification Outbox tiles.
- The Site Call Audit component computes these on demand from the central `SiteCalls` table, **global and per-source-site**; the health dashboard polls it for the headline tiles.
- The KPI set is **buffered count** (`Pending` + `Retrying`), **parked count** (`Parked`), **failed (last interval)**, **delivered (last interval)**, **oldest pending age**, and **stuck count** (`Pending` / `Retrying` rows older than the configurable stuck-age threshold).
- **Stuck** is `Pending` / `Retrying` rows older than a configurable threshold (default **10 minutes**) — **display-only** (KPI count plus a row badge), with no escalation or alerting, consistent with the Notification Outbox stuck metric.
- Site Call Audit KPIs are **point-in-time**, computed on demand from the `SiteCalls` table. There is no time-series store — consistent with Health Monitoring's "current status only" philosophy.

Unlike the Notification Outbox, the Site Call Audit is **not a dispatcher** — cached calls are delivered by each site's Store-and-Forward Engine, and the `SiteCalls` table is an eventually-consistent central mirror of site-owned status.

## Audit Log KPIs

The Audit Log spans both sites (hot-path append + telemetry forward) and central (direct-write + ingest + redaction). Its operational health surfaces as three new dashboard tiles grouped under **Audit**:

- **Audit volume** — events/min landing in the central `AuditLog` table, shown global plus per-site sparkline; sourced from the Audit Log component on the active central node.
- **Audit error rate** — percent of central `AuditLog` rows with `Status` other than `Success` / `Delivered` / `Enqueued` over a rolling 5-minute window. This is the operational error rate of audited operations (HTTP 5xx, transient failures, parked deliveries, etc.) — NOT the audit writer's own health. Audit-writer issues surface separately via `AuditRedactionFailure`.
- **Audit backlog** — global aggregate of `SiteAuditBacklog` across reporting sites (count of `Pending` site-local audit rows, oldest pending age, on-disk bytes); click drills into a per-site breakdown. The tile border-highlights a non-zero aggregate backlog; there is **no** configurable per-site `SiteAuditBacklog` threshold badge (see the note in Monitored Metrics — earlier revisions described one that was never built).

These tiles are **point-in-time** like the Notification Outbox and Site Call Audit KPI tiles — no time-series store; consistent with Health Monitoring's "current status only" philosophy. The site-scoped `SiteAuditBacklog` / `SiteAuditWriteFailures` metrics arrive in the existing site health report; the central-scoped `AuditRedactionFailure` metric is central-computed alongside the existing central KPIs.

## Central Storage

- Health metrics are held **in memory** at the central cluster for display in the UI.
- No historical health data is persisted — the dashboard shows current/latest status only.
- Notification Outbox and Site Call Audit KPIs are not stored by Health Monitoring; they are computed point-in-time from the central `Notifications` and `SiteCalls` tables respectively each time the dashboard refreshes — consistent with the current-status-only philosophy.
- Site connectivity history (online/offline transitions) may optionally be logged via the Audit Log or a separate mechanism if needed in the future.

## Sustained LocalDb Replication Backlog Badge

The per-site tile shows a **Replication backlog** warning badge when `LocalDbOplogBacklog` has
stayed above `LocalDbBacklogThreshold` (default **1000**) continuously for
`LocalDbBacklogSustainedFor` (default **15 minutes**). Both are configurable under
`ScadaBridge:HealthMonitoring` and validated at startup.

- **Sustained, not instantaneous — and that is the design, not a refinement.** The daily
  `site_events` retention purge and the storage-cap trim are CDC-captured like any other write, so
  the backlog jumps by the deleted batch size and drains as the peer acks it. A bare
  `backlog > threshold` badge would therefore fire on every replicated site every day. Healthy is
  defined by the backlog *returning to ~0*, which only a dwell can express. `LocalDbBacklogSustainedFor`
  is validated to be at least one `ReportInterval`: a shorter dwell could be satisfied by a single
  sample and would collapse back into the bare threshold.
- **Implementation.** `CentralHealthAggregator` holds `SiteHealthState.LocalDbBacklogAboveSince`,
  the instant the backlog first rose above the threshold. Reports at or below the threshold clear
  it; reports above it leave the original instant intact (a sustained window, not a sliding one).
- **Null handling is load-bearing.** A report with a `null` backlog leaves the anchor *unchanged*
  rather than clearing it — null means "no reading" (unreplicated site, or a failed poll), which is
  evidence neither of a drain nor of a backlog. An unreplicated site (rig site-b/site-c) reports
  null forever, so it never accumulates an anchor and can never be badged.
- Offline sites are not badged; the Offline badge is the signal that matters there.

## KPI History Sampling of Replication State

`SiteHealthKpiSampleSource` samples `localDbOplogBacklog` and `localDbReplicationConnected` (0/1)
into the KPI history store, and the backlog is charted in the Health dashboard's per-site Site
Health Trends panel as **LocalDb Replication Backlog**. This is the retrospective half of the
signal: a point-in-time tile can never answer "did that spike drain?", which is the only thing
distinguishing an expected purge burst from a stalled peer.

Both metrics are **conditionally emitted** — a site reporting `null` writes no sample rather than a
coerced `0`, unlike the `auditBacklogPending` metric alongside them. A fabricated 0 would render a
pair that cannot read its own oplog, or one deliberately not replicated, as perfectly healthy once
a minute forever. Consequently a site's sample count varies: **12 metrics for an unreplicated site,
14 for a replicated one**, and a gap in these two series truthfully means "not replicated / not
reporting". Note `false` and `null` are different states for the connected flag — a configured link
that is down samples as `0`; no link at all samples as nothing. Both fold as **Gauge** in the hourly
rollup catalog (summing an oplog depth, or a 0/1 flag, per hour would be meaningless).

## Unconfigured Replication Reports Null

The two LocalDb fields are produced by `LocalDbReplicationStatusReporter`, a 30 s site-side poll
over `ISyncStatus`. That poll is **gated on whether the node is configured to replicate at all**,
and an unconfigured node publishes `null` for *both* fields.

The gate is load-bearing because the replication engine is registered **unconditionally** on every
site node and sits inert without a peer — so `ISyncStatus` answers `Connected = false` with a real
`0` backlog on a site that was never meant to replicate. Passing that pair through (the behaviour
before register row 80) had two consequences:

- An operator reading a healthy, deliberately-unreplicated site saw `connected: false` and read a
  fault that did not exist.
- Every null branch downstream was dead code on the rig. The conditional KPI emission above and the
  sustained-badge anchor exclusion below both exist to handle "no reading", but rig site-b/site-c
  never sent one — they emitted a permanently misleading `localDbReplicationConnected = 0` series
  instead.

`false` therefore has exactly one meaning on this field: **replication is configured on this node
and the peer session is currently down** — a genuine fault. Beware the specific trap that produced
the bug: configured-ness is `PeerAddress` **OR** `ApiKey` (only the dialling half sets a peer
address; the passive half carries the key alone), so "no `PeerAddress`" does **not** mean
"unreplicated".

The gate value is supplied by the **Host** from `SiteLocalDbSetup.ReplicationIsConfigured`, exactly
as `SiteEventLogReplicationCheck` is supplied to Site Event Logging. Health Monitoring never reads
LocalDb configuration itself, and the PeerAddress-OR-ApiKey rule stays in one place.

## Startup Reconciler Presence

`SiteReconciliationActor` is the per-node startup self-heal: a standby that was DOWN during a
deploy reports its local inventory to central on its next boot, fetches the gap and writes it
(and, since register row 67, restarts Instance Actors the boot-time load missed). The Host skips
it when no `IDeploymentConfigFetcher` is resolvable.

That skip used to be a single boot Warning and nothing else, so a node that had silently lost its
entire self-heal path went on reporting itself perfectly healthy (register row 73). It is now
surfaced three ways:

- **`StartupReconcilerActive` on the health report**, latched at boot by the Host in both branches.
- **A boot-time `Error` log line** (raised from Warning — no configuration makes the fetcher
  legitimately absent on a site node) plus a **site event** (`deployment` / `Error`), so the
  evidence survives after the boot log has scrolled away.
- **A `No self-heal` warning badge** on the site's Health dashboard tile.

Central's treatment is deliberately **Warning, not failure**. Config still reaches the node through
LocalDb replication while it is up; what it has lost is recovery from a deploy that landed while it
was *down*. That is degraded, not offline. Only the explicit `false` is badged — `null` means "not
reported", and badging it would light up the whole fleet the moment central is upgraded ahead of
the sites. Offline sites are not badged, matching the replication-backlog badge.

## Prometheus Metrics (site node)

Distinct from the health *report* above: these are scraped from the site node's `/metrics`
endpoint (`MetricsPort`, default 8084) rather than sent to central.

| Series | Meaning |
| --- | --- |
| `scadabridge_site_data_connections_connected` | Data connections on this node in the `Connected` state |
| `scadabridge_site_data_connections_total` | Data connections known to this node, in any state |
| `scadabridge_site_stream_subscribers_active` | Open central→site gRPC subscriber streams |

- The two `data_connections` gauges (register row 54) are the **only exported data-connection
  signal**; before them the health report's `DataConnectionStatuses` was visible in the UI and
  the CLI but never as a metric. `connected < total` is the alert condition — neither number
  alone distinguishes "a device is down" from "nothing is configured". They are fed directly by
  the Data Connection Layer at each connection state transition, deliberately **not** derived
  from the health-report path, so the metric survives a dead report pipeline (the exact failure
  the *metrics stale* flag exists to catch).
- Both read **0 on the standby node** — data connections are opened by the deployment path on
  the active node only. A standby reporting 0 is healthy; alert on the active node, or on
  `max()` across the pair.
- `scadabridge_site_stream_subscribers_active` was renamed from `scadabridge_site_connection_up`
  by the same row. It has only ever counted open subscriber streams (Debug View + the site-wide
  alarm feed), so it correctly reads 0 on a healthy site nobody is watching — the old name
  invited reading it as data-connection health, and did cause exactly that misdiagnosis once.
- All three ride the `ZB.MOM.WW.ScadaBridge` meter, already in the `SiteServiceRegistration.ObservedMeters`
  allowlist. `ZbTelemetryOptions.Meters` is an **allowlist**: an unlisted meter exports nothing,
  with no error and no missing-metric signal.

## No Alerting

- Health monitoring is **display-only** for now — no automated notifications or alerts are triggered by health status changes.
- This can be extended in the future.

## Dependencies

- **Communication Layer**: Transports health reports from sites to central.
- **Data Connection Layer (site)**: Provides connection health metrics.
- **Site Runtime (site)**: Provides script error rate and alarm evaluation error rate metrics.
- **Site Runtime (site) — deployed secrets**: Provides `SecretMirrorStatuses` via its own `SecretMirrorStatusReporter` hosted service (SiteRuntime already references HealthMonitoring and owns the `secret_mirror_status` storage, following the `SiteStreamAlarmDropReporter` precedent).
- **Store-and-Forward Engine (site)**: Provides buffer depth metrics, including the notification backlog awaiting forward to central.
- **Cluster Infrastructure (site)**: Provides node role status.
- **Host (site)**: Supplies the two boot-time facts Health Monitoring must not derive for itself — the LocalDb replication-configured gate behind `LocalDbReplicationConnected`/`LocalDbOplogBacklog` (from `SiteLocalDbSetup.ReplicationIsConfigured`), and `StartupReconcilerActive`, latched where the Host decides whether to spawn `SiteReconciliationActor`. Both follow the `SiteEventLogReplicationCheck` precedent: the Host owns the configuration reading, the component owns the reporting.
- **Notification Outbox (central)**: Provides central-computed outbox KPIs — queue depth, stuck count, parked count — for the headline dashboard tiles.
- **Site Call Audit (central)**: Provides central-computed cached-call KPIs — buffered count, parked count, failed/delivered (last interval), oldest pending age, stuck count — for the headline dashboard tiles.
- **Site Event Logging (site)**: Provides `SiteEventLogWriteFailures`. Its daily retention purge and storage-cap trim are also the expected cause of transient `LocalDbOplogBacklog` spikes on a replicated node — see [Component-SiteEventLogging.md](Component-SiteEventLogging.md) → Storage.
- **Audit Log (#23)**: Provides the site-reported `SiteAuditBacklog` / `SiteAuditWriteFailures` metrics (via the site health report) and the central-computed `AuditRedactionFailure` metric, plus the central audit-row rate feeding the **Audit** dashboard tile group (Audit volume, Audit error rate, Audit backlog).

## Interactions

- **Central UI**: Health Monitoring Dashboard displays aggregated metrics.
- **Communication Layer**: Health reports flow as periodic messages.
- **KPI History (#26)**: emits `IKpiSampleSource` (`SiteHealthKpiSampleSource`, per-Site) consumed by the KpiHistory recorder (#26). It reads the in-memory `ICentralHealthAggregator.GetAllSiteStates()` (no DB), turning the per-site snapshot — previously sequence-numbered every 30s but discarded — into 12–14 sampled `KpiSample` metrics per site, persisted every minute: `connectionsUp`/`connectionsDown`, `scriptErrors`, `alarmEvalErrors`, `sfBufferDepth`, `deadLetters`, `parkedMessages`, `deployedInstances`/`enabledInstances`/`disabledInstances`, `auditBacklogPending`, `eventLogWriteFailures`, plus the two conditionally-emitted LocalDb replication metrics `localDbOplogBacklog` and `localDbReplicationConnected` (register row 33 — emitted only when the site actually reports a reading, hence 12 for an unreplicated site and 14 for a replicated one; see *KPI History Sampling of Replication State* above). Of these, **only five are charted** in the dashboard's per-site `KpiTrendChart` panel — `connectionsDown`, `scriptErrors`, `sfBufferDepth`, `deadLetters`, and `localDbOplogBacklog` (the metrics carried in the shared `KpiMetrics.SiteHealth` Commons catalog and fetched by `Health.razor`'s trend selector). The rest are **sampled and persisted but not (yet) charted** — they are retained in the `KpiSample` store and available for future trend surfaces or ad-hoc query. The synthetic central self-report (`CentralHealthReportLoop.CentralSiteId = "$central"`, fed by the leader-only report loop via `CollectReport($central)`) appears in `GetAllSiteStates()` and is **intentionally sampled into the per-Site KPI store as a real `KpiScopes.Site` series** keyed `$central`; the Central UI deliberately **pins `$central` first** in the per-site trend selector (rendered as "Central Cluster"), so the central cluster's own trends sit alongside the real sites'. (`$central`'s zero-valued connection/instance/S&F metrics reflect the central self-report carrying no site-runtime data, not a defect.) See [Component-KpiHistory.md](Component-KpiHistory.md).