ScadaBridge/docs/requirements/Component-HealthMonitoring.md

Component: Health Monitoring

Purpose

The Health Monitoring component collects and reports operational health metrics from site clusters to the central cluster, providing engineers with visibility into the status of the distributed system.

Location

Site clusters (metric collection and reporting). Central cluster (aggregation and display).

Responsibilities

Site Side

• Collect health metrics from local subsystems.
• Periodically report metrics to the central cluster via the Communication Layer.

Central Side

• Receive and store health metrics from all sites.
• Detect site connectivity status (online/offline) based on heartbeat presence.
• Present health data in the Central UI dashboard.

Monitored Metrics

Metric	Source	Description
Site online/offline	Communication Layer	Whether the site is reachable (based on heartbeat)
Active/standby node status	Cluster Infrastructure	Which node is active, which is standby
Data connection health	Data Connection Layer	Connected/disconnected/reconnecting per data connection
Tag resolution counts	Data Connection Layer	Per connection: total subscribed tags vs. successfully resolved tags
Script error rates	Site Runtime (Script Actors)	Frequency of script failures
Alarm evaluation error rates	Site Runtime (Alarm Actors)	Frequency of alarm evaluation failures
Store-and-forward buffer depth	Store-and-Forward Engine	Pending messages by category — external, notification (notifications awaiting forward to central), DB write
Dead letter count	Akka.NET EventStream	Messages sent to actors that no longer exist — indicates stale references or timing issues
Notification Outbox queue depth	Notification Outbox (central)	Count of Pending + Retrying notifications — central-computed, not site-reported
Notification Outbox stuck count	Notification Outbox (central)	Count of Pending / Retrying notifications older than the configurable stuck-age threshold — central-computed, not site-reported
Notification Outbox parked count	Notification Outbox (central)	Count of Parked notifications — central-computed, not site-reported
SiteAuditBacklog	Audit Log (site)	Count of Pending rows in the site-local AuditLog plus oldest-pending-age plus on-disk bytes. Surfaced on the Audit KPI tiles (which border-highlight any non-zero backlog) and sampled into KPI history as auditBacklogPending. No configurable per-site threshold badge exists for this metric — earlier revisions of this document described one, but none was ever implemented and no such option exists. The only configurable sustained-threshold site badge is the LocalDb replication backlog (register row 33, below).
SiteAuditWriteFailures	Audit Log (site)	Count of failed hot-path audit appends at the site since the last health report.
AuditRedactionFailure	Audit Log (central)	Count of payload redactor errors (over-redacted payloads, safety-net hit) since the last interval.
LocalDbReplicationConnected	Consolidated site LocalDb	Whether a peer sync session is currently running on this node. Nullable — null means "no reading", not "disconnected". Reported only by a node with replication configured; an unconfigured node reports null, never false (register row 80 — see Unconfigured replication reports null below).
StartupReconcilerActive	Host (site)	Whether this node spawned its SiteReconciliationActor at boot. Tri-state: null = not reported (the central self-report, or a pre-row-73 site build), true = running, false = the node booted with no IDeploymentConfigFetcher and therefore has no startup self-heal. Latched once at boot by the Host — not polled. Drives the No self-heal dashboard badge (register row 73).
SecretMirrorStatuses	Site Runtime (DeployedSecretsMirror via the replicated secret_mirror_status table)	Every site node's deployed-secrets mirror status — node name, when the node last wrote the row (on a failure-set change, at start-up, or on the 10-min refresh), and its current failures ({SecretName, Message}; names and messages only, never values). Review finding M1: the standby's failures ride the active node's report, because each node writes its own row to a LocalDb-replicated table and the active node's SecretMirrorStatusReporter reads them all. Nullable — null = no data (central self-report, a pre-M1 site, reporter not yet run); an empty list = wired, no row yet; a node with an empty Failures list is healthy. Point-in-time (not drained on collect). Wire: SiteHealthReportDto.secret_mirror_statuses (field 33, a presence wrapper so null ≠ empty). Drives the Secret mirror failing dashboard badge + per-node failure list. On an unreplicated pair only the active node's own row appears. A node that has not refreshed its row for SiteRuntime:SecretMirrorStatusStaleAfter (default 1 h — decommissioned, renamed or down) is left off and its row deleted.
LocalDbOplogBacklog	Consolidated site LocalDb	Unacked oplog depth (Prometheus localdb_oplog_depth). Nullable — null means "no reading", NOT "connected with an empty backlog" (a failed poll rendered as 0 would report a pair that cannot read its own oplog as perfectly healthy). Expected transient spikes: the daily site_events retention purge and the storage-cap trim are CDC-captured like any other write, so the backlog jumps by the batch size at purge time and drains as the peer acks it — see Component-SiteEventLogging.md → Storage and docs/deployment/topology-guide.md → Reading the replication backlog for the healthy-versus-faulty signature.

Reporting Protocol

• Sites send a health report message to central at a configurable interval (default: 30 seconds).
• Each report is a flat snapshot containing the current values of all monitored metrics, a monotonic sequence number, and the report timestamp from the site. Central replaces the previous state for that site only if the incoming sequence number is higher than the last received — this prevents stale reports (e.g., delayed in transit or from a pre-failover node) from overwriting newer state.
• Central tracks two distinct signals per site — do not conflate them:
  • Liveness (online/offline): driven by the last signal of any kind (a full report or a transport heartbeat, the latter arriving every ~5s from any reachable site node). If central receives no signal within OfflineTimeout (default: 60 seconds), the site is marked offline. Because heartbeats are frequent, this only fires on genuine total loss of contact — not during a single-node failover, when the surviving node keeps heartbeating. The synthetic central self-report site has no heartbeat source and uses the longer CentralOfflineTimeout (default: 3 minutes) so a single missed self-report does not flap it offline.
  • Metrics staleness: an online site (heartbeats still arriving) that has produced no full report within MetricsStaleTimeout (default: 2 minutes) is flagged metrics stale. This surfaces the failure mode where a site's report pipeline (HealthReportSender) has died but the node is otherwise reachable — previously such a site showed "online with frozen metrics forever". A site that has never delivered a report is anchored on its first-contact time (FirstSeenAt), so a report pipeline dead from first boot is also flagged after MetricsStaleTimeout (previously such a site showed online-with-no-metrics indefinitely). MetricsStaleTimeout must be positive and no shorter than the report interval.
• Online recovery: when central receives a health report from a site that was marked offline, the site is automatically marked online and the metrics-stale flag cleared (a fresh report proves the pipeline is alive); a heartbeat alone also restores online but leaves the metrics-stale flag untouched. No manual acknowledgment required.
• Status-transition timestamp: each online↔offline flip stamps LastStatusChangeAt, so the UI can answer "since when has this site been offline/online" rather than only "is it offline now". Report/heartbeat traffic that leaves the status unchanged does not move it.

Error Rate Metrics

Script error rates and alarm evaluation error rates are calculated as raw counts per reporting interval:

• The site maintains a counter for each metric that increments on every failure.
• Each health report includes the count since the last report. The counter resets after each report is sent.
• Central displays these as "X errors in the last 30 seconds" (or whatever the configured interval is).
• Script errors include all failures that prevent a script from completing successfully: unhandled exceptions, timeouts, recursion limit violations, and any other error condition.
• Alarm evaluation errors include all failures during alarm condition evaluation.
• For detailed diagnostics (error types, stack traces, affected instances), operators use the Site Event Log Viewer — the health dashboard is for quick triage, not forensics.

Notification Outbox KPIs

The Notification Outbox is a central component, so its KPIs are central-computed rather than collected from sites and carried in the site health report:

• The dashboard surfaces three headline outbox KPIs: queue depth (Pending + Retrying), stuck count (Pending / Retrying rows older than the configurable stuck-age threshold), and parked count (Parked).
• The Notification Outbox component computes these on demand from the central Notifications table; the health dashboard polls it for the headline tiles.
• The fuller KPI set — which also includes delivered (last interval) and oldest pending age — lives on the Central UI Notification Outbox page, not the health dashboard.
• Outbox KPIs are point-in-time, computed on demand from the Notifications table. There is no time-series store — consistent with Health Monitoring's "current status only" philosophy. The outbox's own ~1-year row retention answers historical questions directly.

These are distinct from the site-reported Store-and-forward buffer depth notification metric, which now covers the site→central leg — notifications still buffered in a site's Store-and-Forward Engine awaiting forward to central — and remains part of the site health report.

Site Call Audit KPIs

The Site Call Audit is a central component, so its KPIs — like the Notification Outbox's — are central-computed rather than collected from sites and carried in the site health report:

• The dashboard surfaces Site Call Audit headline KPI tiles alongside the existing Notification Outbox tiles.
• The Site Call Audit component computes these on demand from the central SiteCalls table, global and per-source-site; the health dashboard polls it for the headline tiles.
• The KPI set is buffered count (Pending + Retrying), parked count (Parked), failed (last interval), delivered (last interval), oldest pending age, and stuck count (Pending / Retrying rows older than the configurable stuck-age threshold).
• Stuck is Pending / Retrying rows older than a configurable threshold (default 10 minutes) — display-only (KPI count plus a row badge), with no escalation or alerting, consistent with the Notification Outbox stuck metric.
• Site Call Audit KPIs are point-in-time, computed on demand from the SiteCalls table. There is no time-series store — consistent with Health Monitoring's "current status only" philosophy.

Unlike the Notification Outbox, the Site Call Audit is not a dispatcher — cached calls are delivered by each site's Store-and-Forward Engine, and the SiteCalls table is an eventually-consistent central mirror of site-owned status.

Audit Log KPIs

The Audit Log spans both sites (hot-path append + telemetry forward) and central (direct-write + ingest + redaction). Its operational health surfaces as three new dashboard tiles grouped under Audit:

• Audit volume — events/min landing in the central AuditLog table, shown global plus per-site sparkline; sourced from the Audit Log component on the active central node.
• Audit error rate — percent of central AuditLog rows with Status other than Success / Delivered / Enqueued over a rolling 5-minute window. This is the operational error rate of audited operations (HTTP 5xx, transient failures, parked deliveries, etc.) — NOT the audit writer's own health. Audit-writer issues surface separately via AuditRedactionFailure.
• Audit backlog — global aggregate of SiteAuditBacklog across reporting sites (count of Pending site-local audit rows, oldest pending age, on-disk bytes); click drills into a per-site breakdown. The tile border-highlights a non-zero aggregate backlog; there is no configurable per-site SiteAuditBacklog threshold badge (see the note in Monitored Metrics — earlier revisions described one that was never built).

These tiles are point-in-time like the Notification Outbox and Site Call Audit KPI tiles — no time-series store; consistent with Health Monitoring's "current status only" philosophy. The site-scoped SiteAuditBacklog / SiteAuditWriteFailures metrics arrive in the existing site health report; the central-scoped AuditRedactionFailure metric is central-computed alongside the existing central KPIs.

Central Storage

• Health metrics are held in memory at the central cluster for display in the UI.
• No historical health data is persisted — the dashboard shows current/latest status only.
• Notification Outbox and Site Call Audit KPIs are not stored by Health Monitoring; they are computed point-in-time from the central Notifications and SiteCalls tables respectively each time the dashboard refreshes — consistent with the current-status-only philosophy.
• Site connectivity history (online/offline transitions) may optionally be logged via the Audit Log or a separate mechanism if needed in the future.

Sustained LocalDb Replication Backlog Badge

The per-site tile shows a Replication backlog warning badge when LocalDbOplogBacklog has stayed above LocalDbBacklogThreshold (default 1000) continuously for LocalDbBacklogSustainedFor (default 15 minutes). Both are configurable under ScadaBridge:HealthMonitoring and validated at startup.

• Sustained, not instantaneous — and that is the design, not a refinement. The daily site_events retention purge and the storage-cap trim are CDC-captured like any other write, so the backlog jumps by the deleted batch size and drains as the peer acks it. A bare backlog > threshold badge would therefore fire on every replicated site every day. Healthy is defined by the backlog returning to ~0, which only a dwell can express. LocalDbBacklogSustainedFor is validated to be at least one ReportInterval: a shorter dwell could be satisfied by a single sample and would collapse back into the bare threshold.
• Implementation. CentralHealthAggregator holds SiteHealthState.LocalDbBacklogAboveSince, the instant the backlog first rose above the threshold. Reports at or below the threshold clear it; reports above it leave the original instant intact (a sustained window, not a sliding one).
• Null handling is load-bearing. A report with a null backlog leaves the anchor unchanged rather than clearing it — null means "no reading" (unreplicated site, or a failed poll), which is evidence neither of a drain nor of a backlog. An unreplicated site (rig site-b/site-c) reports null forever, so it never accumulates an anchor and can never be badged.
• Offline sites are not badged; the Offline badge is the signal that matters there.

KPI History Sampling of Replication State

SiteHealthKpiSampleSource samples localDbOplogBacklog and localDbReplicationConnected (0/1) into the KPI history store, and the backlog is charted in the Health dashboard's per-site Site Health Trends panel as LocalDb Replication Backlog. This is the retrospective half of the signal: a point-in-time tile can never answer "did that spike drain?", which is the only thing distinguishing an expected purge burst from a stalled peer.

Both metrics are conditionally emitted — a site reporting null writes no sample rather than a coerced 0, unlike the auditBacklogPending metric alongside them. A fabricated 0 would render a pair that cannot read its own oplog, or one deliberately not replicated, as perfectly healthy once a minute forever. Consequently a site's sample count varies: 12 metrics for an unreplicated site, 14 for a replicated one, and a gap in these two series truthfully means "not replicated / not reporting". Note false and null are different states for the connected flag — a configured link that is down samples as 0; no link at all samples as nothing. Both fold as Gauge in the hourly rollup catalog (summing an oplog depth, or a 0/1 flag, per hour would be meaningless).

Unconfigured Replication Reports Null

The two LocalDb fields are produced by LocalDbReplicationStatusReporter, a 30 s site-side poll over ISyncStatus. That poll is gated on whether the node is configured to replicate at all, and an unconfigured node publishes null for both fields.

The gate is load-bearing because the replication engine is registered unconditionally on every site node and sits inert without a peer — so ISyncStatus answers Connected = false with a real 0 backlog on a site that was never meant to replicate. Passing that pair through (the behaviour before register row 80) had two consequences:

• An operator reading a healthy, deliberately-unreplicated site saw connected: false and read a fault that did not exist.
• Every null branch downstream was dead code on the rig. The conditional KPI emission above and the sustained-badge anchor exclusion below both exist to handle "no reading", but rig site-b/site-c never sent one — they emitted a permanently misleading localDbReplicationConnected = 0 series instead.

false therefore has exactly one meaning on this field: replication is configured on this node and the peer session is currently down — a genuine fault. Beware the specific trap that produced the bug: configured-ness is PeerAddress OR ApiKey (only the dialling half sets a peer address; the passive half carries the key alone), so "no PeerAddress" does not mean "unreplicated".

The gate value is supplied by the Host from SiteLocalDbSetup.ReplicationIsConfigured, exactly as SiteEventLogReplicationCheck is supplied to Site Event Logging. Health Monitoring never reads LocalDb configuration itself, and the PeerAddress-OR-ApiKey rule stays in one place.

Startup Reconciler Presence

SiteReconciliationActor is the per-node startup self-heal: a standby that was DOWN during a deploy reports its local inventory to central on its next boot, fetches the gap and writes it (and, since register row 67, restarts Instance Actors the boot-time load missed). The Host skips it when no IDeploymentConfigFetcher is resolvable.

That skip used to be a single boot Warning and nothing else, so a node that had silently lost its entire self-heal path went on reporting itself perfectly healthy (register row 73). It is now surfaced three ways:

• StartupReconcilerActive on the health report, latched at boot by the Host in both branches.
• A boot-time Error log line (raised from Warning — no configuration makes the fetcher legitimately absent on a site node) plus a site event (deployment / Error), so the evidence survives after the boot log has scrolled away.
• A No self-heal warning badge on the site's Health dashboard tile.

Central's treatment is deliberately Warning, not failure. Config still reaches the node through LocalDb replication while it is up; what it has lost is recovery from a deploy that landed while it was down. That is degraded, not offline. Only the explicit false is badged — null means "not reported", and badging it would light up the whole fleet the moment central is upgraded ahead of the sites. Offline sites are not badged, matching the replication-backlog badge.

Prometheus Metrics (site node)

Distinct from the health report above: these are scraped from the site node's /metrics endpoint (MetricsPort, default 8084) rather than sent to central.

Series	Meaning
scadabridge_site_data_connections_connected	Data connections on this node in the Connected state
scadabridge_site_data_connections_total	Data connections known to this node, in any state
scadabridge_site_stream_subscribers_active	Open central→site gRPC subscriber streams

• The two data_connections gauges (register row 54) are the only exported data-connection signal; before them the health report's DataConnectionStatuses was visible in the UI and the CLI but never as a metric. connected < total is the alert condition — neither number alone distinguishes "a device is down" from "nothing is configured". They are fed directly by the Data Connection Layer at each connection state transition, deliberately not derived from the health-report path, so the metric survives a dead report pipeline (the exact failure the metrics stale flag exists to catch).
• Both read 0 on the standby node — data connections are opened by the deployment path on the active node only. A standby reporting 0 is healthy; alert on the active node, or on max() across the pair.
• scadabridge_site_stream_subscribers_active was renamed from scadabridge_site_connection_up by the same row. It has only ever counted open subscriber streams (Debug View + the site-wide alarm feed), so it correctly reads 0 on a healthy site nobody is watching — the old name invited reading it as data-connection health, and did cause exactly that misdiagnosis once.
• All three ride the ZB.MOM.WW.ScadaBridge meter, already in the SiteServiceRegistration.ObservedMeters allowlist. ZbTelemetryOptions.Meters is an allowlist: an unlisted meter exports nothing, with no error and no missing-metric signal.

No Alerting

• Health monitoring is display-only for now — no automated notifications or alerts are triggered by health status changes.
• This can be extended in the future.

Dependencies

• Communication Layer: Transports health reports from sites to central.
• Data Connection Layer (site): Provides connection health metrics.
• Site Runtime (site): Provides script error rate and alarm evaluation error rate metrics.
• Site Runtime (site) — deployed secrets: Provides SecretMirrorStatuses via its own SecretMirrorStatusReporter hosted service (SiteRuntime already references HealthMonitoring and owns the secret_mirror_status storage, following the SiteStreamAlarmDropReporter precedent).
• Store-and-Forward Engine (site): Provides buffer depth metrics, including the notification backlog awaiting forward to central.
• Cluster Infrastructure (site): Provides node role status.
• Host (site): Supplies the two boot-time facts Health Monitoring must not derive for itself — the LocalDb replication-configured gate behind LocalDbReplicationConnected/LocalDbOplogBacklog (from SiteLocalDbSetup.ReplicationIsConfigured), and StartupReconcilerActive, latched where the Host decides whether to spawn SiteReconciliationActor. Both follow the SiteEventLogReplicationCheck precedent: the Host owns the configuration reading, the component owns the reporting.
• Notification Outbox (central): Provides central-computed outbox KPIs — queue depth, stuck count, parked count — for the headline dashboard tiles.
• Site Call Audit (central): Provides central-computed cached-call KPIs — buffered count, parked count, failed/delivered (last interval), oldest pending age, stuck count — for the headline dashboard tiles.
• Site Event Logging (site): Provides SiteEventLogWriteFailures. Its daily retention purge and storage-cap trim are also the expected cause of transient LocalDbOplogBacklog spikes on a replicated node — see Component-SiteEventLogging.md → Storage.
• Audit Log (#23): Provides the site-reported SiteAuditBacklog / SiteAuditWriteFailures metrics (via the site health report) and the central-computed AuditRedactionFailure metric, plus the central audit-row rate feeding the Audit dashboard tile group (Audit volume, Audit error rate, Audit backlog).

Interactions

• Central UI: Health Monitoring Dashboard displays aggregated metrics.
• Communication Layer: Health reports flow as periodic messages.
• KPI History (#26): emits IKpiSampleSource (SiteHealthKpiSampleSource, per-Site) consumed by the KpiHistory recorder (#26). It reads the in-memory ICentralHealthAggregator.GetAllSiteStates() (no DB), turning the per-site snapshot — previously sequence-numbered every 30s but discarded — into 12–14 sampled KpiSample metrics per site, persisted every minute: connectionsUp/connectionsDown, scriptErrors, alarmEvalErrors, sfBufferDepth, deadLetters, parkedMessages, deployedInstances/enabledInstances/disabledInstances, auditBacklogPending, eventLogWriteFailures, plus the two conditionally-emitted LocalDb replication metrics localDbOplogBacklog and localDbReplicationConnected (register row 33 — emitted only when the site actually reports a reading, hence 12 for an unreplicated site and 14 for a replicated one; see KPI History Sampling of Replication State above). Of these, only five are charted in the dashboard's per-site KpiTrendChart panel — connectionsDown, scriptErrors, sfBufferDepth, deadLetters, and localDbOplogBacklog (the metrics carried in the shared KpiMetrics.SiteHealth Commons catalog and fetched by Health.razor's trend selector). The rest are sampled and persisted but not (yet) charted — they are retained in the KpiSample store and available for future trend surfaces or ad-hoc query. The synthetic central self-report (CentralHealthReportLoop.CentralSiteId = "$central", fed by the leader-only report loop via CollectReport($central)) appears in GetAllSiteStates() and is intentionally sampled into the per-Site KPI store as a real KpiScopes.Site series keyed $central; the Central UI deliberately pins $central first in the per-site trend selector (rendered as "Central Cluster"), so the central cluster's own trends sit alongside the real sites'. ($central's zero-valued connection/instance/S&F metrics reflect the central self-report carrying no site-runtime data, not a defect.) See Component-KpiHistory.md.