# Component: Health Monitoring ## Purpose The Health Monitoring component collects and reports operational health metrics from site clusters to the central cluster, providing engineers with visibility into the status of the distributed system. ## Location Site clusters (metric collection and reporting). Central cluster (aggregation and display). ## Responsibilities ### Site Side - Collect health metrics from local subsystems. - Periodically report metrics to the central cluster via the Communication Layer. ### Central Side - Receive and store health metrics from all sites. - Detect site connectivity status (online/offline) based on heartbeat presence. - Present health data in the Central UI dashboard. ## Monitored Metrics | Metric | Source | Description | |--------|--------|-------------| | Site online/offline | Communication Layer | Whether the site is reachable (based on heartbeat) | | Active/standby node status | Cluster Infrastructure | Which node is active, which is standby | | Data connection health | Data Connection Layer | Connected/disconnected/reconnecting per data connection | | Tag resolution counts | Data Connection Layer | Per connection: total subscribed tags vs. successfully resolved tags | | Script error rates | Site Runtime (Script Actors) | Frequency of script failures | | Alarm evaluation error rates | Site Runtime (Alarm Actors) | Frequency of alarm evaluation failures | | Store-and-forward buffer depth | Store-and-Forward Engine | Pending messages by category — external, notification (notifications awaiting forward to central), DB write | | Dead letter count | Akka.NET EventStream | Messages sent to actors that no longer exist — indicates stale references or timing issues | | Notification Outbox queue depth | Notification Outbox (central) | Count of `Pending` + `Retrying` notifications — central-computed, not site-reported | | Notification Outbox stuck count | Notification Outbox (central) | Count of `Pending` / `Retrying` notifications older than the configurable stuck-age threshold — central-computed, not site-reported | | Notification Outbox parked count | Notification Outbox (central) | Count of `Parked` notifications — central-computed, not site-reported | | `SiteAuditBacklog` | Audit Log (site) | Count of `Pending` rows in the site-local `AuditLog` plus oldest-pending-age plus on-disk bytes. Surfaced on the Audit KPI tiles (which border-highlight any non-zero backlog) and sampled into KPI history as `auditBacklogPending`. **No configurable per-site threshold badge exists for this metric** — earlier revisions of this document described one, but none was ever implemented and no such option exists. The only configurable sustained-threshold site badge is the LocalDb replication backlog (register row 33, below). | | `SiteAuditWriteFailures` | Audit Log (site) | Count of failed hot-path audit appends at the site since the last health report. | | `AuditRedactionFailure` | Audit Log (central) | Count of payload redactor errors (over-redacted payloads, safety-net hit) since the last interval. | | `LocalDbReplicationConnected` | Consolidated site LocalDb | Whether a peer sync session is currently running on this node. **Nullable — null means "no reading", not "disconnected".** Reported only by a node with replication configured; an unconfigured node reports **null**, never `false` (register row 80 — see *Unconfigured replication reports null* below). | | `StartupReconcilerActive` | Host (site) | Whether this node spawned its `SiteReconciliationActor` at boot. **Tri-state:** null = not reported (the central self-report, or a pre-row-73 site build), `true` = running, `false` = the node booted with no `IDeploymentConfigFetcher` and therefore has **no startup self-heal**. Latched once at boot by the Host — not polled. Drives the *No self-heal* dashboard badge (register row 73). | | `SecretMirrorStatuses` | Site Runtime (`DeployedSecretsMirror` via the replicated `secret_mirror_status` table) | Every site node's deployed-secrets mirror status — node name, when the node last wrote the row (on a failure-set change, at start-up, or on the 10-min refresh), and its current failures (`{SecretName, Message}`; names and messages only, never values). Review finding M1: the **standby's** failures ride the **active** node's report, because each node writes its own row to a LocalDb-replicated table and the active node's `SecretMirrorStatusReporter` reads them all. **Nullable — null = no data** (central self-report, a pre-M1 site, reporter not yet run); an empty list = wired, no row yet; a node with an empty `Failures` list is healthy. Point-in-time (not drained on collect). Wire: `SiteHealthReportDto.secret_mirror_statuses` (field **33**, a presence wrapper so null ≠ empty). Drives the *Secret mirror failing* dashboard badge + per-node failure list. On an unreplicated pair only the active node's own row appears. A node that has not refreshed its row for `SiteRuntime:SecretMirrorStatusStaleAfter` (default 1 h — decommissioned, renamed or down) is left off and its row deleted. | | `LocalDbOplogBacklog` | Consolidated site LocalDb | Unacked oplog depth (Prometheus `localdb_oplog_depth`). **Nullable — null means "no reading", NOT "connected with an empty backlog"** (a failed poll rendered as 0 would report a pair that cannot read its own oplog as perfectly healthy). **Expected transient spikes:** the daily `site_events` retention purge and the storage-cap trim are CDC-captured like any other write, so the backlog jumps by the batch size at purge time and drains as the peer acks it — see `Component-SiteEventLogging.md` → Storage and `docs/deployment/topology-guide.md` → *Reading the replication backlog* for the healthy-versus-faulty signature. | ## Reporting Protocol - Sites send a **health report message** to central at a configurable interval (default: **30 seconds**). - Each report is a **flat snapshot** containing the current values of all monitored metrics, a **monotonic sequence number**, and the **report timestamp** from the site. Central replaces the previous state for that site only if the incoming sequence number is higher than the last received — this prevents stale reports (e.g., delayed in transit or from a pre-failover node) from overwriting newer state. - Central tracks two **distinct** signals per site — do not conflate them: - **Liveness (online/offline)**: driven by the **last signal of any kind** (a full report *or* a transport heartbeat, the latter arriving every ~5s from any reachable site node). If central receives no signal within `OfflineTimeout` (default: **60 seconds**), the site is marked **offline**. Because heartbeats are frequent, this only fires on genuine total loss of contact — not during a single-node failover, when the surviving node keeps heartbeating. The synthetic *central* self-report site has no heartbeat source and uses the longer `CentralOfflineTimeout` (default: **3 minutes**) so a single missed self-report does not flap it offline. - **Metrics staleness**: an **online** site (heartbeats still arriving) that has produced no full **report** within `MetricsStaleTimeout` (default: **2 minutes**) is flagged **metrics stale**. This surfaces the failure mode where a site's report pipeline (`HealthReportSender`) has died but the node is otherwise reachable — previously such a site showed "online with frozen metrics forever". A site that has **never** delivered a report is anchored on its first-contact time (`FirstSeenAt`), so a report pipeline dead from first boot is also flagged after `MetricsStaleTimeout` (previously such a site showed online-with-no-metrics indefinitely). `MetricsStaleTimeout` must be positive and no shorter than the report interval. - **Online recovery**: when central receives a health report from a site that was marked offline, the site is automatically marked **online** and the metrics-stale flag cleared (a fresh report proves the pipeline is alive); a heartbeat alone also restores **online** but leaves the metrics-stale flag untouched. No manual acknowledgment required. - **Status-transition timestamp**: each online↔offline flip stamps `LastStatusChangeAt`, so the UI can answer "since when has this site been offline/online" rather than only "is it offline now". Report/heartbeat traffic that leaves the status unchanged does not move it. ## Error Rate Metrics Script error rates and alarm evaluation error rates are calculated as **raw counts per reporting interval**: - The site maintains a counter for each metric that increments on every failure. - Each health report includes the count since the last report. The counter resets after each report is sent. - Central displays these as "X errors in the last 30 seconds" (or whatever the configured interval is). - **Script errors** include all failures that prevent a script from completing successfully: unhandled exceptions, timeouts, recursion limit violations, and any other error condition. - **Alarm evaluation errors** include all failures during alarm condition evaluation. - For detailed diagnostics (error types, stack traces, affected instances), operators use the **Site Event Log Viewer** — the health dashboard is for quick triage, not forensics. ## Notification Outbox KPIs The Notification Outbox is a **central** component, so its KPIs are **central-computed** rather than collected from sites and carried in the site health report: - The dashboard surfaces three **headline** outbox KPIs: **queue depth** (`Pending` + `Retrying`), **stuck count** (`Pending` / `Retrying` rows older than the configurable stuck-age threshold), and **parked count** (`Parked`). - The Notification Outbox component computes these on demand from the central `Notifications` table; the health dashboard polls it for the headline tiles. - The fuller KPI set — which also includes **delivered (last interval)** and **oldest pending age** — lives on the Central UI **Notification Outbox** page, not the health dashboard. - Outbox KPIs are **point-in-time**, computed on demand from the `Notifications` table. There is no time-series store — consistent with Health Monitoring's "current status only" philosophy. The outbox's own ~1-year row retention answers historical questions directly. These are distinct from the site-reported **Store-and-forward buffer depth** notification metric, which now covers the **site→central leg** — notifications still buffered in a site's Store-and-Forward Engine awaiting forward to central — and remains part of the site health report. ## Site Call Audit KPIs The Site Call Audit is a **central** component, so its KPIs — like the Notification Outbox's — are **central-computed** rather than collected from sites and carried in the site health report: - The dashboard surfaces Site Call Audit **headline** KPI tiles alongside the existing Notification Outbox tiles. - The Site Call Audit component computes these on demand from the central `SiteCalls` table, **global and per-source-site**; the health dashboard polls it for the headline tiles. - The KPI set is **buffered count** (`Pending` + `Retrying`), **parked count** (`Parked`), **failed (last interval)**, **delivered (last interval)**, **oldest pending age**, and **stuck count** (`Pending` / `Retrying` rows older than the configurable stuck-age threshold). - **Stuck** is `Pending` / `Retrying` rows older than a configurable threshold (default **10 minutes**) — **display-only** (KPI count plus a row badge), with no escalation or alerting, consistent with the Notification Outbox stuck metric. - Site Call Audit KPIs are **point-in-time**, computed on demand from the `SiteCalls` table. There is no time-series store — consistent with Health Monitoring's "current status only" philosophy. Unlike the Notification Outbox, the Site Call Audit is **not a dispatcher** — cached calls are delivered by each site's Store-and-Forward Engine, and the `SiteCalls` table is an eventually-consistent central mirror of site-owned status. ## Audit Log KPIs The Audit Log spans both sites (hot-path append + telemetry forward) and central (direct-write + ingest + redaction). Its operational health surfaces as three new dashboard tiles grouped under **Audit**: - **Audit volume** — events/min landing in the central `AuditLog` table, shown global plus per-site sparkline; sourced from the Audit Log component on the active central node. - **Audit error rate** — percent of central `AuditLog` rows with `Status` other than `Success` / `Delivered` / `Enqueued` over a rolling 5-minute window. This is the operational error rate of audited operations (HTTP 5xx, transient failures, parked deliveries, etc.) — NOT the audit writer's own health. Audit-writer issues surface separately via `AuditRedactionFailure`. - **Audit backlog** — global aggregate of `SiteAuditBacklog` across reporting sites (count of `Pending` site-local audit rows, oldest pending age, on-disk bytes); click drills into a per-site breakdown. The tile border-highlights a non-zero aggregate backlog; there is **no** configurable per-site `SiteAuditBacklog` threshold badge (see the note in Monitored Metrics — earlier revisions described one that was never built). These tiles are **point-in-time** like the Notification Outbox and Site Call Audit KPI tiles — no time-series store; consistent with Health Monitoring's "current status only" philosophy. The site-scoped `SiteAuditBacklog` / `SiteAuditWriteFailures` metrics arrive in the existing site health report; the central-scoped `AuditRedactionFailure` metric is central-computed alongside the existing central KPIs. ## Central Storage - Health metrics are held **in memory** at the central cluster for display in the UI. - No historical health data is persisted — the dashboard shows current/latest status only. - Notification Outbox and Site Call Audit KPIs are not stored by Health Monitoring; they are computed point-in-time from the central `Notifications` and `SiteCalls` tables respectively each time the dashboard refreshes — consistent with the current-status-only philosophy. - Site connectivity history (online/offline transitions) may optionally be logged via the Audit Log or a separate mechanism if needed in the future. ## Sustained LocalDb Replication Backlog Badge The per-site tile shows a **Replication backlog** warning badge when `LocalDbOplogBacklog` has stayed above `LocalDbBacklogThreshold` (default **1000**) continuously for `LocalDbBacklogSustainedFor` (default **15 minutes**). Both are configurable under `ScadaBridge:HealthMonitoring` and validated at startup. - **Sustained, not instantaneous — and that is the design, not a refinement.** The daily `site_events` retention purge and the storage-cap trim are CDC-captured like any other write, so the backlog jumps by the deleted batch size and drains as the peer acks it. A bare `backlog > threshold` badge would therefore fire on every replicated site every day. Healthy is defined by the backlog *returning to ~0*, which only a dwell can express. `LocalDbBacklogSustainedFor` is validated to be at least one `ReportInterval`: a shorter dwell could be satisfied by a single sample and would collapse back into the bare threshold. - **Implementation.** `CentralHealthAggregator` holds `SiteHealthState.LocalDbBacklogAboveSince`, the instant the backlog first rose above the threshold. Reports at or below the threshold clear it; reports above it leave the original instant intact (a sustained window, not a sliding one). - **Null handling is load-bearing.** A report with a `null` backlog leaves the anchor *unchanged* rather than clearing it — null means "no reading" (unreplicated site, or a failed poll), which is evidence neither of a drain nor of a backlog. An unreplicated site (rig site-b/site-c) reports null forever, so it never accumulates an anchor and can never be badged. - Offline sites are not badged; the Offline badge is the signal that matters there. ## KPI History Sampling of Replication State `SiteHealthKpiSampleSource` samples `localDbOplogBacklog` and `localDbReplicationConnected` (0/1) into the KPI history store, and the backlog is charted in the Health dashboard's per-site Site Health Trends panel as **LocalDb Replication Backlog**. This is the retrospective half of the signal: a point-in-time tile can never answer "did that spike drain?", which is the only thing distinguishing an expected purge burst from a stalled peer. Both metrics are **conditionally emitted** — a site reporting `null` writes no sample rather than a coerced `0`, unlike the `auditBacklogPending` metric alongside them. A fabricated 0 would render a pair that cannot read its own oplog, or one deliberately not replicated, as perfectly healthy once a minute forever. Consequently a site's sample count varies: **12 metrics for an unreplicated site, 14 for a replicated one**, and a gap in these two series truthfully means "not replicated / not reporting". Note `false` and `null` are different states for the connected flag — a configured link that is down samples as `0`; no link at all samples as nothing. Both fold as **Gauge** in the hourly rollup catalog (summing an oplog depth, or a 0/1 flag, per hour would be meaningless). ## Unconfigured Replication Reports Null The two LocalDb fields are produced by `LocalDbReplicationStatusReporter`, a 30 s site-side poll over `ISyncStatus`. That poll is **gated on whether the node is configured to replicate at all**, and an unconfigured node publishes `null` for *both* fields. The gate is load-bearing because the replication engine is registered **unconditionally** on every site node and sits inert without a peer — so `ISyncStatus` answers `Connected = false` with a real `0` backlog on a site that was never meant to replicate. Passing that pair through (the behaviour before register row 80) had two consequences: - An operator reading a healthy, deliberately-unreplicated site saw `connected: false` and read a fault that did not exist. - Every null branch downstream was dead code on the rig. The conditional KPI emission above and the sustained-badge anchor exclusion below both exist to handle "no reading", but rig site-b/site-c never sent one — they emitted a permanently misleading `localDbReplicationConnected = 0` series instead. `false` therefore has exactly one meaning on this field: **replication is configured on this node and the peer session is currently down** — a genuine fault. Beware the specific trap that produced the bug: configured-ness is `PeerAddress` **OR** `ApiKey` (only the dialling half sets a peer address; the passive half carries the key alone), so "no `PeerAddress`" does **not** mean "unreplicated". The gate value is supplied by the **Host** from `SiteLocalDbSetup.ReplicationIsConfigured`, exactly as `SiteEventLogReplicationCheck` is supplied to Site Event Logging. Health Monitoring never reads LocalDb configuration itself, and the PeerAddress-OR-ApiKey rule stays in one place. ## Startup Reconciler Presence `SiteReconciliationActor` is the per-node startup self-heal: a standby that was DOWN during a deploy reports its local inventory to central on its next boot, fetches the gap and writes it (and, since register row 67, restarts Instance Actors the boot-time load missed). The Host skips it when no `IDeploymentConfigFetcher` is resolvable. That skip used to be a single boot Warning and nothing else, so a node that had silently lost its entire self-heal path went on reporting itself perfectly healthy (register row 73). It is now surfaced three ways: - **`StartupReconcilerActive` on the health report**, latched at boot by the Host in both branches. - **A boot-time `Error` log line** (raised from Warning — no configuration makes the fetcher legitimately absent on a site node) plus a **site event** (`deployment` / `Error`), so the evidence survives after the boot log has scrolled away. - **A `No self-heal` warning badge** on the site's Health dashboard tile. Central's treatment is deliberately **Warning, not failure**. Config still reaches the node through LocalDb replication while it is up; what it has lost is recovery from a deploy that landed while it was *down*. That is degraded, not offline. Only the explicit `false` is badged — `null` means "not reported", and badging it would light up the whole fleet the moment central is upgraded ahead of the sites. Offline sites are not badged, matching the replication-backlog badge. ## Prometheus Metrics (site node) Distinct from the health *report* above: these are scraped from the site node's `/metrics` endpoint (`MetricsPort`, default 8084) rather than sent to central. | Series | Meaning | | --- | --- | | `scadabridge_site_data_connections_connected` | Data connections on this node in the `Connected` state | | `scadabridge_site_data_connections_total` | Data connections known to this node, in any state | | `scadabridge_site_stream_subscribers_active` | Open central→site gRPC subscriber streams | - The two `data_connections` gauges (register row 54) are the **only exported data-connection signal**; before them the health report's `DataConnectionStatuses` was visible in the UI and the CLI but never as a metric. `connected < total` is the alert condition — neither number alone distinguishes "a device is down" from "nothing is configured". They are fed directly by the Data Connection Layer at each connection state transition, deliberately **not** derived from the health-report path, so the metric survives a dead report pipeline (the exact failure the *metrics stale* flag exists to catch). - Both read **0 on the standby node** — data connections are opened by the deployment path on the active node only. A standby reporting 0 is healthy; alert on the active node, or on `max()` across the pair. - `scadabridge_site_stream_subscribers_active` was renamed from `scadabridge_site_connection_up` by the same row. It has only ever counted open subscriber streams (Debug View + the site-wide alarm feed), so it correctly reads 0 on a healthy site nobody is watching — the old name invited reading it as data-connection health, and did cause exactly that misdiagnosis once. - All three ride the `ZB.MOM.WW.ScadaBridge` meter, already in the `SiteServiceRegistration.ObservedMeters` allowlist. `ZbTelemetryOptions.Meters` is an **allowlist**: an unlisted meter exports nothing, with no error and no missing-metric signal. ## No Alerting - Health monitoring is **display-only** for now — no automated notifications or alerts are triggered by health status changes. - This can be extended in the future. ## Dependencies - **Communication Layer**: Transports health reports from sites to central. - **Data Connection Layer (site)**: Provides connection health metrics. - **Site Runtime (site)**: Provides script error rate and alarm evaluation error rate metrics. - **Site Runtime (site) — deployed secrets**: Provides `SecretMirrorStatuses` via its own `SecretMirrorStatusReporter` hosted service (SiteRuntime already references HealthMonitoring and owns the `secret_mirror_status` storage, following the `SiteStreamAlarmDropReporter` precedent). - **Store-and-Forward Engine (site)**: Provides buffer depth metrics, including the notification backlog awaiting forward to central. - **Cluster Infrastructure (site)**: Provides node role status. - **Host (site)**: Supplies the two boot-time facts Health Monitoring must not derive for itself — the LocalDb replication-configured gate behind `LocalDbReplicationConnected`/`LocalDbOplogBacklog` (from `SiteLocalDbSetup.ReplicationIsConfigured`), and `StartupReconcilerActive`, latched where the Host decides whether to spawn `SiteReconciliationActor`. Both follow the `SiteEventLogReplicationCheck` precedent: the Host owns the configuration reading, the component owns the reporting. - **Notification Outbox (central)**: Provides central-computed outbox KPIs — queue depth, stuck count, parked count — for the headline dashboard tiles. - **Site Call Audit (central)**: Provides central-computed cached-call KPIs — buffered count, parked count, failed/delivered (last interval), oldest pending age, stuck count — for the headline dashboard tiles. - **Site Event Logging (site)**: Provides `SiteEventLogWriteFailures`. Its daily retention purge and storage-cap trim are also the expected cause of transient `LocalDbOplogBacklog` spikes on a replicated node — see [Component-SiteEventLogging.md](Component-SiteEventLogging.md) → Storage. - **Audit Log (#23)**: Provides the site-reported `SiteAuditBacklog` / `SiteAuditWriteFailures` metrics (via the site health report) and the central-computed `AuditRedactionFailure` metric, plus the central audit-row rate feeding the **Audit** dashboard tile group (Audit volume, Audit error rate, Audit backlog). ## Interactions - **Central UI**: Health Monitoring Dashboard displays aggregated metrics. - **Communication Layer**: Health reports flow as periodic messages. - **KPI History (#26)**: emits `IKpiSampleSource` (`SiteHealthKpiSampleSource`, per-Site) consumed by the KpiHistory recorder (#26). It reads the in-memory `ICentralHealthAggregator.GetAllSiteStates()` (no DB), turning the per-site snapshot — previously sequence-numbered every 30s but discarded — into 12–14 sampled `KpiSample` metrics per site, persisted every minute: `connectionsUp`/`connectionsDown`, `scriptErrors`, `alarmEvalErrors`, `sfBufferDepth`, `deadLetters`, `parkedMessages`, `deployedInstances`/`enabledInstances`/`disabledInstances`, `auditBacklogPending`, `eventLogWriteFailures`, plus the two conditionally-emitted LocalDb replication metrics `localDbOplogBacklog` and `localDbReplicationConnected` (register row 33 — emitted only when the site actually reports a reading, hence 12 for an unreplicated site and 14 for a replicated one; see *KPI History Sampling of Replication State* above). Of these, **only five are charted** in the dashboard's per-site `KpiTrendChart` panel — `connectionsDown`, `scriptErrors`, `sfBufferDepth`, `deadLetters`, and `localDbOplogBacklog` (the metrics carried in the shared `KpiMetrics.SiteHealth` Commons catalog and fetched by `Health.razor`'s trend selector). The rest are **sampled and persisted but not (yet) charted** — they are retained in the `KpiSample` store and available for future trend surfaces or ad-hoc query. The synthetic central self-report (`CentralHealthReportLoop.CentralSiteId = "$central"`, fed by the leader-only report loop via `CollectReport($central)`) appears in `GetAllSiteStates()` and is **intentionally sampled into the per-Site KPI store as a real `KpiScopes.Site` series** keyed `$central`; the Central UI deliberately **pins `$central` first** in the per-site trend selector (rendered as "Central Cluster"), so the central cluster's own trends sit alongside the real sites'. (`$central`'s zero-valued connection/instance/S&F metrics reflect the central self-report carrying no site-runtime data, not a defect.) See [Component-KpiHistory.md](Component-KpiHistory.md).