feat(observability): honest stream-subscriber name + real data-connection gauges (row 54)

scadabridge_site_connection_up never counted data connections. It counts open central->site gRPC subscriber streams (Debug View + the site-wide alarm feed), so it correctly reads 0 on a healthy site nobody is watching — and the name invited reading it as device health. That misreading has already cost a live debugging session: register row 48's 'connection_up is 0 on all six site nodes' was healthy steady state, not the fault it was diagnosed as. Two halves, per the row: RENAME. scadabridge.site.connection.up -> scadabridge.site.stream.subscribers.active (helpers SiteConnectionOpened/Closed -> SiteStreamSubscriberOpened/Closed), and the instrument gains the unit:"1" the other gauges already carried. Every reference updated: the gRPC server, both test suites, Component-Communication.md, components/Commons.md, docker/README.md and docker/seed-live-probe.sh. A test now pins the new name AND asserts the old one is gone, so it cannot creep back. NEW. scadabridge.site.data_connections.connected / .total — the first exported data-connection signal (the state existed on the health report and in the CLI, but never as a metric). connected < total is the alert condition; neither number alone separates 'a device is down' from 'nothing is configured', which is why the denominator ships with the numerator. Sourcing is deliberate on two counts. It is fed by the DCL at its four state transitions rather than derived from the health report, so the metric survives a dead report pipeline — the exact failure the 'metrics stale' flag exists to catch, and a derived gauge would go blind at the same moment. And it uses a static registry reached through ScadaBridgeTelemetry (the SiteConnectionOpened precedent) rather than a DI-injected holder: the DCL actors already take ISiteHealthCollector through their constructors, and threading a second dependency through both would ripple into every test that builds them. It also keeps SiteHealthCollector untouched, which matters right now — another agent is concurrently editing that file for the tag-resolution path. Removal is wired too: an undeployed connection leaves the registry, so it stops counting toward the total instead of latching connected < total on forever. Both gauges read 0 on the standby (connections open on the active node only) — healthy, documented, and called out in the docs so it does not become the next site_connection_up. No ObservedMeters allowlist change: all three instruments ride the existing ZB.MOM.WW.ScadaBridge meter. (An unlisted meter exports nothing, silently.) Tests: Commons 711 passed (9 new, delta-based against a baseline with unique per-test connection names, so they do not inherit the register row 32 QueueDepthGaugeTests static-state flake); Communication 717; DCL 319.
2026-08-15 12:53:12 -04:00
parent 90ce099928
commit b5f120c370
12 changed files with 389 additions and 32 deletions
@@ -101,6 +101,35 @@ These tiles are **point-in-time** like the Notification Outbox and Site Call Aud
 - Notification Outbox and Site Call Audit KPIs are not stored by Health Monitoring; they are computed point-in-time from the central `Notifications` and `SiteCalls` tables respectively each time the dashboard refreshes — consistent with the current-status-only philosophy.
 - Site connectivity history (online/offline transitions) may optionally be logged via the Audit Log or a separate mechanism if needed in the future.

+## Prometheus Metrics (site node)
+
+Distinct from the health *report* above: these are scraped from the site node's `/metrics`
+endpoint (`MetricsPort`, default 8084) rather than sent to central.
+
+| Series | Meaning |
+| --- | --- |
+| `scadabridge_site_data_connections_connected` | Data connections on this node in the `Connected` state |
+| `scadabridge_site_data_connections_total` | Data connections known to this node, in any state |
+| `scadabridge_site_stream_subscribers_active` | Open central→site gRPC subscriber streams |
+
+- The two `data_connections` gauges (register row 54) are the **only exported data-connection
+  signal**; before them the health report's `DataConnectionStatuses` was visible in the UI and
+  the CLI but never as a metric. `connected < total` is the alert condition — neither number
+  alone distinguishes "a device is down" from "nothing is configured". They are fed directly by
+  the Data Connection Layer at each connection state transition, deliberately **not** derived
+  from the health-report path, so the metric survives a dead report pipeline (the exact failure
+  the *metrics stale* flag exists to catch).
+- Both read **0 on the standby node** — data connections are opened by the deployment path on
+  the active node only. A standby reporting 0 is healthy; alert on the active node, or on
+  `max()` across the pair.
+- `scadabridge_site_stream_subscribers_active` was renamed from `scadabridge_site_connection_up`
+  by the same row. It has only ever counted open subscriber streams (Debug View + the site-wide
+  alarm feed), so it correctly reads 0 on a healthy site nobody is watching — the old name
+  invited reading it as data-connection health, and did cause exactly that misdiagnosis once.
+- All three ride the `ZB.MOM.WW.ScadaBridge` meter, already in the `SiteServiceRegistration.ObservedMeters`
+  allowlist. `ZbTelemetryOptions.Meters` is an **allowlist**: an unlisted meter exports nothing,
+  with no error and no missing-metric signal.
+
 ## No Alerting

 - Health monitoring is **display-only** for now — no automated notifications or alerts are triggered by health status changes.