lmxopcua/docs/Telemetry.md

Live Telemetry Transport (v2)

Overview

The AdminUI's live observability panels (/alerts, /script-log, /hosts driver table, driver resilience status) are fed by four node→central event channels. Historically all four rode DistributedPubSub (DPS) on the shared Akka mesh — the same gossip ring that carries redundancy state and the command-plane topics. Per-cluster mesh Phase 5 adds a second transport for these four channels only: one gRPC server-streaming contract, selected per node/central pair by Telemetry:Mode / TelemetryDial:Mode (Dps default | Grpc).

This is the same motivation as Phase 2's MeshTransport and Phase 3's ConfigSource/ConfigServe: DPS only works when central and the node share a gossip ring, and Phase 6 splits the fleet into one Akka mesh per application Cluster. Once that split lands, a driver node in a site's mesh is no longer a cluster member of central's mesh, so DPS can no longer reach it — telemetry needs its own out-of-band transport, same as commands (Phase 2) and config bytes (Phase 3).

Direction: the node hosts, central dials

Load-bearing and easy to get backwards: each driver node hosts the telemetry gRPC server (a dedicated Kestrel h2c listener); central (admin role) is the client and dials in. This mirrors the inversion ScadaBridge already uses for the same problem (its SiteStreamService).

 driver node                          central (admin)
┌─────────────────────┐              ┌──────────────────────────┐
│ 4 publish seams      │              │ TelemetryDialSupervisor   │
│  -> node-local hub    │   Subscribe  │  - discovers ClusterNode   │
│  -> DPS (unchanged)   │◄─────────────│    rows (Host+GrpcPort)   │
│                       │   (gRPC      │  - one reconnecting       │
│ TelemetryStreamGrpc-  │    stream)   │    dialer per node        │
│  Service (h2c server) │─────────────►│  - feeds the SAME sinks   │
│  Telemetry:GrpcListen- │  TelemetryEvent │   the DPS bridges feed   │
│  Port                 │  (oneof, 4 kinds)│   today                  │
└─────────────────────┘              └──────────────────────────┘

A fused admin,driver node both hosts (as driver) and dials (as admin) — it dials itself plus its redundant pair peer, the same way central dials site nodes.

Central discovers driver-node telemetry endpoints from the ClusterNode table (Host + GrpcPort, the latter added in Phase 1 for exactly this purpose) — the same DB-sourced discovery Phase 1's ack set and Phase 2's ClusterClient contact set already use. No shared gossip membership is required to find or dial a node.

The dark switch

Telemetry:Mode (node/serve side) and TelemetryDial:Mode (central/dial side) each default to Dps and can be independently set to Grpc. Both code paths are compiled into every binary — flipping either flag is an appsettings/env change plus a restart, not a redeploy or rebuild — the same discipline as MeshTransport:Mode and ConfigSource:Mode.

What actually changes per mode, precisely:

• The node ALWAYS does both things, regardless of Telemetry:Mode. Every one of the four publish seams emits into the node-local ITelemetryLocalHub and publishes to DPS, unconditionally. The node additionally always hosts the gRPC server whenever Telemetry:GrpcListenPort > 0 — hosting is not mode-gated at all. Telemetry:Mode only affects central's own bookkeeping about which side it expects to be consulted (it is otherwise inert on the node).
• Only central's ingest source switches, driven by TelemetryDial:Mode:
  • Dps (default) — today's four DPS SignalR bridges (alert, script-log, driver-status, resilience) subscribe and feed the AdminUI sinks. Unchanged behavior.
  • Grpc — those four DPS bridges are not spawned. Instead the TelemetryDialSupervisor actor dials every driver node's gRPC stream and feeds the identical in-process sinks (IInProcessBroadcaster<AlarmTransitionEvent>, IInProcessBroadcaster<ScriptLogEntry>, IDriverStatusSnapshotStore, IDriverResilienceStatusStore).
• The AdminUI panels themselves are untouched. /alerts, /script-log, and the /hosts driver table all read from the same sinks in both modes — only the sinks' upstream feed swaps. The fleet-status bridge (out of Phase 5 scope — see below) stays on DPS in both modes.

Because hosting is unconditional and both ingest paths are always compiled in, a fleet can be flipped node-by-node and central-by-central with no coordination window where telemetry is lost — the node is always serving, so central can switch to Grpc whenever it likes.

The four migrated channels

DPS topic (today)	Domain record	Central sink fed
alerts	AlarmTransitionEvent	IInProcessBroadcaster<AlarmTransitionEvent> (+ AlertHub); /alerts page; live/disconnected pill
script-logs	ScriptLogEntry	IInProcessBroadcaster<ScriptLogEntry> (+ ScriptLogHub); /script-log page; live pill
driver-health	DriverHealthChanged	IDriverStatusSnapshotStore; /hosts driver table
driver-resilience-status	DriverResilienceStatusChanged	IDriverResilienceStatusStore

These four are carried as oneof event kinds on one TelemetryStreamService.Subscribe RPC — a single stream per (central, driver-node) pair carries all four, rather than one stream per topic. Proto field evolution is additive-only (never renumber/reuse a tag), locked by a contract test that reflects over the oneof cases.

The three deferred channels (NOT migrated in Phase 5 — do not read this as "seven done")

The program sketch originally named seven observability topics for Phase 5. Three were scoped out, each for a distinct, settled reason:

• redundancy-state — bidirectional, built directly from Cluster.State, and pair-local: it drives ServiceLevel and the Primary data-plane gate, consumed in-process by OpcUaPublishActor, ScriptedAlarmHostActor, DriverHostActor, and HistorianAdapterActor. It stays on DPS in both MeshTransport modes today, and it is genuinely mesh-bound by design — under Phase 6 each pair keeps sharing its own small mesh, so DPS keeps working for it in-mesh. Central's display of each pair's redundancy state is a Phase 6 cross-mesh concern (a future observability channel, once central no longer shares gossip with any site pair), not a Phase 5 telemetry-panel migration.
• fleet-status — central-internal, not a node→central stream at all. The admin singleton FleetStatusBroadcaster builds it from the admin node's own cluster membership/reachability/ leader events, and Fleet.razor polls the Config DB and ignores the feed entirely. There is nothing here for a per-node gRPC stream to carry. Revisit in Phase 6, once central genuinely loses gossip visibility of site nodes and needs another way to know a pair's membership.
• deployment-acks — already migrated, but onto a different transport: it rides the Phase 2 ClusterClient transport (MeshTransport:Mode=ClusterClient) as a command-plane reply, not an observability broadcast. It was never a Phase 5 candidate.

See docs/Redundancy.md for how redundancy-state and the command transports fit together.

Authentication — fail-closed from day one

A shared node bearer key gates the stream: Telemetry:ApiKey (node/serve side) must equal TelemetryDial:ApiKey (central/dial side). TelemetryStreamAuthInterceptor enforces it — path-scoped to /telemetry.v1.TelemetryStreamService/, comparing the Authorization: Bearer token with CryptographicOperations.FixedTimeEquals, and rejecting with PermissionDenied. An unset key rejects every call rather than allowing an open stream — the same fail-closed posture as ConfigServeAuthInterceptor and LocalDbSyncAuthInterceptor.

This supersedes the design doc's §6.3, which had provisionally decided to "match ScadaBridge's unauthenticated posture for now" for inter-cluster transports. ScadaBridge itself has since closed that gap with this identical interceptor pattern, so Phase 5 ships authenticated from the start rather than deferring auth to a later hardening pass. See the design doc's superseded note for the full history.

Upgrading an existing deployment: populate ClusterNode.GrpcPort first (#493)

GrpcPort is a nullable column added by Phase 5, and nothing backfills it onto rows that already existed. On an upgraded deployment every ClusterNode row therefore carries NULL, central finds no dial targets, and flipping TelemetryDial:Mode to Grpc connects to nothing. Fresh installs are unaffected. AkkaPort did not have this problem only because it is NOT NULL with a default of 4053.

Nothing fails loudly, which is what makes this worth stating up front — the code degrades gracefully, once per node, throttled:

ClusterNode <id> has no GrpcPort; it exposes no telemetry stream and is skipped in this dial refresh
(further skips of this node are silent)

Other symptoms: no ESTABLISHED sockets on the telemetry port (a LISTEN but nothing else in docker exec <node> cat /proc/net/tcp6), and the /alerts / /script-log pill never turning live in Grpc mode.

Before flipping any node to Grpc, set each driver node's GrpcPort to that node's own Telemetry:GrpcListenPort — via the AdminUI node editor, or directly:

UPDATE dbo.ClusterNode SET GrpcPort = <that node's Telemetry:GrpcListenPort> WHERE NodeId = '<node>';

There is deliberately no EF data migration backfilling this. The correct value is that node's own listener port, which is per-deployment configuration the database cannot derive; a migration could only invent one, and a wrong dial target is worse than the NULL the dialer already skips explicitly. The docker-dev seed does backfill (GrpcPort IS NULL AND CreatedBy = 'docker-dev-seed' → 4056) because there the port is fixed by docker-compose.yml and therefore actually knowable.

Reconnect and reliability

Central's TelemetryDialSupervisor (an admin-role actor) runs one reconnecting dialer per driver node:

• Discovery — enabled, non-maintenance ClusterNode rows are read on start and refreshed every TelemetryDial:ContactRefreshSeconds (default 60), plus on an admin-side topology change. A row with a null GrpcPort is skipped (that node exposes no telemetry endpoint — logged, not an error). Dial targets are added/removed as the row set changes.
• Reconnect backoff — immediate first retry, then a fixed ~5 second backoff, indefinitely. The dialer never gives up permanently: this is an observability channel, not the data plane, so a persistently-unreachable node just means a persistently-stale panel, not a fleet fault.
• Generation stamping — each dialer carries a monotonically-increasing generation counter, so a late error or late event from a superseded stream (e.g. one that raced a reconnect) is recognized and ignored rather than corrupting the current stream's state.
• Snapshot replay on reconnect — the node-local hub keeps a last-value cache for driver-health and driver-resilience-status (there is nothing meaningful to replay for alerts/script-logs, which are append logs). On every new Subscribe — including a reconnect — the hub drains the cached snapshots to the new stream before live deltas, so IDriverStatusSnapshotStore / IDriverResilienceStatusStore re-prime immediately rather than sitting stale until the next natural event. alerts/script-logs simply tolerate the gap — a reconnect loses whatever transitions happened while disconnected, same as a DPS resubscribe would.
• Connection indicator — the /alerts and /script-log live/disconnected pill is driven by aggregate stream health across all dialers: connected when at least one node stream is up, disconnected when all are down. This matches today's DPS SubscribeAck/PostStop pill semantics — the pill has never meant "every node is live," only "the panel has a source." The two store-backed panels (driver-health, driver-resilience-status) carry no fleet-wide connection flag today, on gRPC or on DPS; per-row staleness is unchanged.

Configuration reference

See docs/Configuration.md § Telemetry / TelemetryDial for the full keys table. In brief:

• Telemetry (node/serve side): Mode (Dps|Grpc, default Dps), GrpcListenPort (0 = disabled — the driver node's dedicated telemetry h2c port), ApiKey (the shared node key; supply via ${secret:}/env, never commit).
• TelemetryDial (central side): Mode (Dps|Grpc), ApiKey (must equal the nodes' Telemetry:ApiKey), ContactRefreshSeconds (default 60), CallTimeoutSeconds (default 30).

A driver-role node with Telemetry:Mode=Grpc must set GrpcListenPort > 0 (nothing to serve otherwise), and Grpc mode on either side requires a non-empty ApiKey — both are enforced by ValidateOnStart validators, fail-fast at boot.

Relationship to the rest of the mesh program

• Same dark-switch discipline as Phases 2 and 3. MeshTransport:Mode, ConfigSource:Mode, and Telemetry:Mode/TelemetryDial:Mode are all independently-flippable, restart-only config changes, and in every case the "new" side is always wired so the flip needs no coordinated redeploy.
• Why the inversion matters for Phase 6. Every other cross-boundary transport in this program (MeshTransport, ConfigSource/ConfigServe) already has central as the addressable, discoverable side and the node as the caller or the callee-by-address. Telemetry is the one channel that is naturally node-sourced and fleet-wide-fanned, so putting central in the client role — dialing out to each node by its ClusterNode-recorded address — is what lets telemetry survive the mesh split cleanly: central never needs cluster membership with a site's mesh to keep watching it.
• Status: code-complete on feat/mesh-phase5; the live gate (flip the docker-dev rig to Grpc, confirm all panels stay green with the DPS telemetry bridges unspawned, kill-and-reconnect a site node) has not yet run. See docs/plans/2026-07-22-per-cluster-mesh-program.md § Phase 5 and docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md for the implementation plan.