feat(observability): honest stream-subscriber name + real data-connection gauges (row 54)

scadabridge_site_connection_up never counted data connections. It counts open
central->site gRPC subscriber streams (Debug View + the site-wide alarm feed),
so it correctly reads 0 on a healthy site nobody is watching — and the name
invited reading it as device health. That misreading has already cost a live
debugging session: register row 48's 'connection_up is 0 on all six site
nodes' was healthy steady state, not the fault it was diagnosed as.

Two halves, per the row:

RENAME. scadabridge.site.connection.up -> scadabridge.site.stream.subscribers.active
(helpers SiteConnectionOpened/Closed -> SiteStreamSubscriberOpened/Closed), and
the instrument gains the unit:"1" the other gauges already carried. Every
reference updated: the gRPC server, both test suites, Component-Communication.md,
components/Commons.md, docker/README.md and docker/seed-live-probe.sh. A test
now pins the new name AND asserts the old one is gone, so it cannot creep back.

NEW. scadabridge.site.data_connections.connected / .total — the first exported
data-connection signal (the state existed on the health report and in the CLI,
but never as a metric). connected < total is the alert condition; neither
number alone separates 'a device is down' from 'nothing is configured', which
is why the denominator ships with the numerator.

Sourcing is deliberate on two counts. It is fed by the DCL at its four state
transitions rather than derived from the health report, so the metric survives
a dead report pipeline — the exact failure the 'metrics stale' flag exists to
catch, and a derived gauge would go blind at the same moment. And it uses a
static registry reached through ScadaBridgeTelemetry (the SiteConnectionOpened
precedent) rather than a DI-injected holder: the DCL actors already take
ISiteHealthCollector through their constructors, and threading a second
dependency through both would ripple into every test that builds them. It also
keeps SiteHealthCollector untouched, which matters right now — another agent is
concurrently editing that file for the tag-resolution path.

Removal is wired too: an undeployed connection leaves the registry, so it stops
counting toward the total instead of latching connected < total on forever.

Both gauges read 0 on the standby (connections open on the active node only) —
healthy, documented, and called out in the docs so it does not become the next
site_connection_up.

No ObservedMeters allowlist change: all three instruments ride the existing
ZB.MOM.WW.ScadaBridge meter. (An unlisted meter exports nothing, silently.)

Tests: Commons 711 passed (9 new, delta-based against a baseline with unique
per-test connection names, so they do not inherit the register row 32
QueueDepthGaugeTests static-state flake); Communication 717; DCL 319.

src/ZB.MOM.WW.ScadaBridge.Commons/Observability/ScadaBridgeTelemetry.cs

@@ -1,4 +1,5 @@
 using System.Diagnostics.Metrics;
+using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
 
 namespace ZB.MOM.WW.ScadaBridge.Commons.Observability;
 
@@ -126,8 +127,27 @@ public static class ScadaBridgeTelemetry
 
     // ---------------- Observable gauges ----------------
 
-    /// <summary>Current count of open site connections, mutated via <see cref="Interlocked"/>.</summary>
-    private static long _siteConnectionsUp;
+    /// <summary>
+    /// Current count of open central→site gRPC subscriber streams, mutated via
+    /// <see cref="Interlocked"/>. See <see cref="_siteStreamSubscribersActive"/>'s gauge for
+    /// why this is NOT a data-connection count.
+    /// </summary>
+    private static long _siteStreamSubscriberCount;
+
+    /// <summary>
+    /// Register row 54: last-known state of every data connection on this site node, keyed by
+    /// connection name. Written by the DCL at its four state-transition points and read by the
+    /// data-connection gauges below.
+    /// </summary>
+    /// <remarks>
+    /// A static registry, matching <see cref="_siteStreamSubscriberCount"/>'s precedent, rather
+    /// than a DI-injected holder: the DCL actors already receive <c>ISiteHealthCollector</c>
+    /// through their constructors, and threading a second dependency through both actor
+    /// constructors would ripple into every test that builds them. A site node hosts exactly
+    /// one DCL, so a process-global registry and a per-node one are the same thing here.
+    /// </remarks>
+    private static readonly System.Collections.Concurrent.ConcurrentDictionary<string, ConnectionHealth>
+        _dataConnectionStates = new(StringComparer.Ordinal);
 
     /// <summary>Current count of running per-site live-alarm aggregators, mutated via <see cref="Interlocked"/>.</summary>
     private static long _liveAlarmAggregatorsActive;
@@ -144,11 +164,52 @@ public static class ScadaBridgeTelemetry
     private static Func<long>? _importSessionCountProvider;
 
 #pragma warning disable IDE0052 // Held to keep the observable gauges alive for the meter's lifetime.
-    /// <summary>Gauge reporting the number of currently open site connections.</summary>
-    private static readonly ObservableGauge<long> _siteConnectionUp =
-        Meter.CreateObservableGauge<long>("scadabridge.site.connection.up",
-            () => Interlocked.Read(ref _siteConnectionsUp),
-            description: "Number of currently open site connections.");
+    /// <summary>
+    /// Gauge reporting the number of open central→site gRPC subscriber streams.
+    /// </summary>
+    /// <remarks>
+    /// Register row 54 RENAMED this from <c>scadabridge.site.connection.up</c>
+    /// (<c>scadabridge_site_connection_up</c> in Prometheus). The old name read as a
+    /// data-connection health signal, but it has only ever counted open subscriber streams
+    /// on <c>SiteStreamGrpcServer</c> — Debug View and the site-wide alarm feed. It therefore
+    /// sits at 0 on a perfectly healthy site that nobody is watching, and an alert written
+    /// against the old name fired on an idle operator console rather than a broken device.
+    /// For real data-connection health use the <c>scadabridge.site.data_connections.*</c>
+    /// gauges below. Counts both stream kinds together (no <c>stream</c> tag).
+    /// </remarks>
+    private static readonly ObservableGauge<long> _siteStreamSubscribersActive =
+        Meter.CreateObservableGauge<long>("scadabridge.site.stream.subscribers.active",
+            () => Interlocked.Read(ref _siteStreamSubscriberCount),
+            unit: "1",
+            description: "Open central→site gRPC subscriber streams (Debug View + site-wide alarm feed). NOT data connections.");
+
+    /// <summary>
+    /// Register row 54: gauge reporting how many of this site node's data connections are
+    /// currently in <see cref="ConnectionHealth.Connected"/> — the real data-connection
+    /// signal the misnamed stream-subscriber gauge was being mistaken for.
+    /// </summary>
+    /// <remarks>
+    /// Read together with <see cref="_dataConnectionsTotal"/>: <c>connected &lt; total</c> is the
+    /// alertable condition, and neither number alone distinguishes "one device down" from
+    /// "nothing configured". Expect **0 on the standby node** — connections are opened by the
+    /// deployment path on the active node only, so a standby reporting 0 is healthy, not a
+    /// fault. Alert on the active node, or on <c>max()</c> across the pair.
+    /// </remarks>
+    private static readonly ObservableGauge<long> _dataConnectionsConnected =
+        Meter.CreateObservableGauge<long>("scadabridge.site.data_connections.connected",
+            CountDataConnections,
+            unit: "1",
+            description: "Data connections on this site node currently in the Connected state.");
+
+    /// <summary>
+    /// Register row 54: gauge reporting how many data connections this site node knows about,
+    /// in any state. The denominator for <see cref="_dataConnectionsConnected"/>.
+    /// </summary>
+    private static readonly ObservableGauge<long> _dataConnectionsTotal =
+        Meter.CreateObservableGauge<long>("scadabridge.site.data_connections.total",
+            () => (long)_dataConnectionStates.Count,
+            unit: "1",
+            description: "Data connections known to this site node, in any state.");
 
     /// <summary>Gauge reporting the current StoreAndForward queue depth via the registered provider.</summary>
     private static readonly ObservableGauge<long> _storeAndForwardQueueDepth =
@@ -211,11 +272,68 @@ public static class ScadaBridgeTelemetry
     /// </summary>
     public static void RecordImportSessionRejected() => _transportImportSessionsRejected.Add(1);
 
-    /// <summary>Records that a site connection opened (increments the up-count gauge).</summary>
-    public static void SiteConnectionOpened() => Interlocked.Increment(ref _siteConnectionsUp);
+    /// <summary>
+    /// Records that a central→site gRPC subscriber stream opened (increments
+    /// <c>scadabridge.site.stream.subscribers.active</c>). Renamed from
+    /// <c>SiteConnectionOpened</c> by register row 54 — it never counted data connections.
+    /// </summary>
+    public static void SiteStreamSubscriberOpened() => Interlocked.Increment(ref _siteStreamSubscriberCount);
 
-    /// <summary>Records that a site connection closed (decrements the up-count gauge).</summary>
-    public static void SiteConnectionClosed() => Interlocked.Decrement(ref _siteConnectionsUp);
+    /// <summary>
+    /// Records that a central→site gRPC subscriber stream closed (decrements
+    /// <c>scadabridge.site.stream.subscribers.active</c>). Renamed from
+    /// <c>SiteConnectionClosed</c> by register row 54.
+    /// </summary>
+    public static void SiteStreamSubscriberClosed() => Interlocked.Decrement(ref _siteStreamSubscriberCount);
+
+    /// <summary>
+    /// Records the current state of one data connection, feeding the
+    /// <c>scadabridge.site.data_connections.*</c> gauges (register row 54). Called by the DCL
+    /// at each connection state transition; safe to call repeatedly with the same value.
+    /// </summary>
+    /// <param name="connectionName">The data connection's name (unique per site node).</param>
+    /// <param name="state">The connection's new state.</param>
+    public static void RecordDataConnectionState(string connectionName, ConnectionHealth state)
+    {
+        if (string.IsNullOrEmpty(connectionName))
+        {
+            return;
+        }
+
+        _dataConnectionStates[connectionName] = state;
+    }
+
+    /// <summary>
+    /// Drops a data connection from the gauges (register row 54). Called when the DCL removes
+    /// a connection, so an undeployed connection stops counting toward the total rather than
+    /// lingering as a permanently-not-connected row that would alert forever.
+    /// </summary>
+    /// <param name="connectionName">The data connection's name.</param>
+    public static void RemoveDataConnection(string connectionName)
+    {
+        if (string.IsNullOrEmpty(connectionName))
+        {
+            return;
+        }
+
+        _dataConnectionStates.TryRemove(connectionName, out _);
+    }
+
+    /// <summary>Counts data connections currently in <see cref="ConnectionHealth.Connected"/>.</summary>
+    /// <returns>The connected count.</returns>
+    private static long CountDataConnections()
+    {
+        long connected = 0;
+        foreach (var state in _dataConnectionStates.Values)
+        {
+            if (state == ConnectionHealth.Connected)
+            {
+                connected++;
+            }
+        }
+
+        return connected;
+    }
 
     /// <summary>Records that a per-site live-alarm aggregator started (increments the active-aggregator gauge).</summary>
     public static void LiveAlarmAggregatorStarted() => Interlocked.Increment(ref _liveAlarmAggregatorsActive);

src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcServer.cs

@@ -467,12 +467,12 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
 
         // Telemetry follow-on: the connection is now fully established (Subscribe
         // succeeded, so no leak via the catch above). Count it up here and balance
-        // it in the finally below so the scadabridge.site.connection.up gauge is
-        // decremented on EVERY exit path — normal completion, client-cancel /
+        // it in the finally below so the scadabridge.site.stream.subscribers.active
+        // gauge is decremented on EVERY exit path — normal completion, client-cancel /
         // duplicate-replacement (OperationCanceledException), server shutdown
         // (CancelAllStreams -> Cts.Cancel), and any other exception — guaranteeing
         // exactly one Closed per Opened and a gauge that never drifts up.
-        ScadaBridgeTelemetry.SiteConnectionOpened();
+        ScadaBridgeTelemetry.SiteStreamSubscriberOpened();
         try
         {
             // R2 — event batching. The pump replaces the old per-event
@@ -500,7 +500,7 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
         }
         finally
         {
-            ScadaBridgeTelemetry.SiteConnectionClosed();
+            ScadaBridgeTelemetry.SiteStreamSubscriberClosed();
             _streamSubscriber.RemoveSubscriber(relayActor);
             _actorSystem!.Stop(relayActor);
             channel.Writer.TryComplete();

src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs

@@ -3,6 +3,7 @@ using Akka.Event;
 using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol;
 using ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection;
 using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
+using ZB.MOM.WW.ScadaBridge.Commons.Observability;
 using ZB.MOM.WW.ScadaBridge.Commons.Types.Alarms;
 using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
 using ZB.MOM.WW.ScadaBridge.HealthMonitoring;
@@ -312,6 +313,10 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
         _log.Info("[{0}] Entering Connecting state", _connectionName);
         _healthCollector.UpdateConnectionHealth(_connectionName, ConnectionHealth.Connecting);
         _healthCollector.UpdateConnectionEndpoint(_connectionName, "Connecting");
+        // Register row 54 — mirror the state into the data-connection gauges. Kept alongside
+        // the health-collector write (rather than derived from it) so the metric never
+        // depends on the health-report path being alive.
+        ScadaBridgeTelemetry.RecordDataConnectionState(_connectionName, ConnectionHealth.Connecting);
         Become(Connecting);
         Self.Tell(new AttemptConnect());
     }
@@ -394,6 +399,8 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
         _healthCollector.UpdateTagResolution(_connectionName, TotalSubscribedTagCount, ResolvedTagCount);
         var endpointLabel = _backupConfig == null ? "Connected" : $"Connected to {_activeEndpoint.ToString().ToLower()}";
         _healthCollector.UpdateConnectionEndpoint(_connectionName, endpointLabel);
+        // Register row 54 — the only transition that raises the connected gauge.
+        ScadaBridgeTelemetry.RecordDataConnectionState(_connectionName, ConnectionHealth.Connected);
         Become(Connected);
         Stash.UnstashAll();
     }
@@ -475,6 +482,9 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
         _log.Warning("[{0}] Entering Reconnecting state", _connectionName);
         _healthCollector.UpdateConnectionHealth(_connectionName, ConnectionHealth.Disconnected);
         _healthCollector.UpdateConnectionEndpoint(_connectionName, "Disconnected");
+        // Register row 54 — drops this connection out of the connected gauge while the
+        // total holds, which is exactly the connected < total alert condition.
+        ScadaBridgeTelemetry.RecordDataConnectionState(_connectionName, ConnectionHealth.Disconnected);
 
         // Track unstable connections toward failover.
         // If we were connected for less than the stability threshold, this counts

src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionManagerActor.cs

@@ -4,6 +4,7 @@ using Microsoft.Extensions.Logging.Abstractions;
 using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol;
 using ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection;
 using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
+using ZB.MOM.WW.ScadaBridge.Commons.Observability;
 using ZB.MOM.WW.ScadaBridge.Commons.Serialization;
 using ZB.MOM.WW.ScadaBridge.Commons.Types.DataConnections;
 using ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Adapters;
@@ -307,6 +308,10 @@ public class DataConnectionManagerActor : ReceiveActor
             Context.Stop(actor);
             _connectionActors.Remove(command.ConnectionName);
             _healthCollector.RemoveConnection(command.ConnectionName);
+            // Register row 54 — drop it from the gauges too, so an undeployed connection
+            // stops counting toward the total instead of lingering as a permanently
+            // not-connected row that would hold the connected < total alert on forever.
+            ScadaBridgeTelemetry.RemoveDataConnection(command.ConnectionName);
             _log.Info("Removed DataConnectionActor for {0}", command.ConnectionName);
         }
     }