lmxopcua/docs/Telemetry.md

Live Telemetry Transport (v2)

Overview

The AdminUI's live observability panels (/alerts, /script-log, /hosts driver table, driver resilience status) are fed by four node→central event channels. Historically all four rode DistributedPubSub (DPS) on the shared Akka mesh — the same gossip ring that carries redundancy state and the command-plane topics. Per-cluster mesh Phase 5 adds a second transport for these four channels only: one gRPC server-streaming contract, selected per node/central pair by Telemetry:Mode / TelemetryDial:Mode (Dps default | Grpc).

This is the same motivation as Phase 2's MeshTransport and Phase 3's ConfigSource/ConfigServe: DPS only works when central and the node share a gossip ring, and Phase 6 splits the fleet into one Akka mesh per application Cluster. Once that split lands, a driver node in a site's mesh is no longer a cluster member of central's mesh, so DPS can no longer reach it — telemetry needs its own out-of-band transport, same as commands (Phase 2) and config bytes (Phase 3).

Direction: the node hosts, central dials

Load-bearing and easy to get backwards: each driver node hosts the telemetry gRPC server (a dedicated Kestrel h2c listener); central (admin role) is the client and dials in. This mirrors the inversion ScadaBridge already uses for the same problem (its SiteStreamService).

 driver node                          central (admin)
┌─────────────────────┐              ┌──────────────────────────┐
│ 4 publish seams      │              │ TelemetryDialSupervisor   │
│  -> node-local hub    │   Subscribe  │  - discovers ClusterNode   │
│  -> DPS (unchanged)   │◄─────────────│    rows (Host+GrpcPort)   │
│                       │   (gRPC      │  - one reconnecting       │
│ TelemetryStreamGrpc-  │    stream)   │    dialer per node        │
│  Service (h2c server) │─────────────►│  - feeds the SAME sinks   │
│  Telemetry:GrpcListen- │  TelemetryEvent │   the DPS bridges feed   │
│  Port                 │  (oneof, 4 kinds)│   today                  │
└─────────────────────┘              └──────────────────────────┘

A fused admin,driver node both hosts (as driver) and dials (as admin) — it dials itself plus its redundant pair peer, the same way central dials site nodes.

Central discovers driver-node telemetry endpoints from the ClusterNode table (Host + GrpcPort, the latter added in Phase 1 for exactly this purpose) — the same DB-sourced discovery Phase 1's ack set and Phase 2's ClusterClient contact set already use. No shared gossip membership is required to find or dial a node.

The dark switch

Telemetry:Mode (node/serve side) and TelemetryDial:Mode (central/dial side) each default to Dps and can be independently set to Grpc. Both code paths are compiled into every binary — flipping either flag is an appsettings/env change plus a restart, not a redeploy or rebuild — the same discipline as MeshTransport:Mode and ConfigSource:Mode.

What actually changes per mode, precisely:

• The node ALWAYS does both things, regardless of Telemetry:Mode. Every one of the four publish seams emits into the node-local ITelemetryLocalHub and publishes to DPS, unconditionally. The node additionally always hosts the gRPC server whenever Telemetry:GrpcListenPort > 0 — hosting is not mode-gated at all. Telemetry:Mode only affects central's own bookkeeping about which side it expects to be consulted (it is otherwise inert on the node).
• Only central's ingest source switches, driven by TelemetryDial:Mode:
  • Dps (default) — today's four DPS SignalR bridges (alert, script-log, driver-status, resilience) subscribe and feed the AdminUI sinks. Unchanged behavior.
  • Grpc — those four DPS bridges are not spawned. Instead the TelemetryDialSupervisor actor dials every driver node's gRPC stream and feeds the identical in-process sinks (IInProcessBroadcaster<AlarmTransitionEvent>, IInProcessBroadcaster<ScriptLogEntry>, IDriverStatusSnapshotStore, IDriverResilienceStatusStore).
• The AdminUI panels themselves are untouched. /alerts, /script-log, and the /hosts driver table all read from the same sinks in both modes — only the sinks' upstream feed swaps. The fleet-status bridge (out of Phase 5 scope — see below) stays on DPS in both modes.

Because hosting is unconditional and both ingest paths are always compiled in, a fleet can be flipped node-by-node and central-by-central with no coordination window where telemetry is lost — the node is always serving, so central can switch to Grpc whenever it likes.

The four migrated channels

DPS topic (today)	Domain record	Central sink fed
alerts	AlarmTransitionEvent	IInProcessBroadcaster<AlarmTransitionEvent> (+ AlertHub); /alerts page; live/disconnected pill
script-logs	ScriptLogEntry	IInProcessBroadcaster<ScriptLogEntry> (+ ScriptLogHub); /script-log page; live pill
driver-health	DriverHealthChanged	IDriverStatusSnapshotStore; /hosts driver table
driver-resilience-status	DriverResilienceStatusChanged	IDriverResilienceStatusStore

These four are carried as oneof event kinds on one TelemetryStreamService.Subscribe RPC — a single stream per (central, driver-node) pair carries all four, rather than one stream per topic. Proto field evolution is additive-only (never renumber/reuse a tag), locked by a contract test that reflects over the oneof cases.

The three deferred channels (NOT migrated in Phase 5 — do not read this as "seven done")

The program sketch originally named seven observability topics for Phase 5. Three were scoped out, each for a distinct, settled reason:

• redundancy-state — bidirectional, built directly from Cluster.State, and pair-local: it drives ServiceLevel and the Primary data-plane gate, consumed in-process by OpcUaPublishActor, ScriptedAlarmHostActor, DriverHostActor, and HistorianAdapterActor. It stays on DPS in both MeshTransport modes today, and it is genuinely mesh-bound by design — under Phase 6 each pair keeps sharing its own small mesh, so DPS keeps working for it in-mesh. Central's display of each pair's redundancy state is a Phase 6 cross-mesh concern (a future observability channel, once central no longer shares gossip with any site pair), not a Phase 5 telemetry-panel migration.
• fleet-status — central-internal, not a node→central stream at all. The admin singleton FleetStatusBroadcaster builds it from the admin node's own cluster membership/reachability/ leader events, and Fleet.razor polls the Config DB and ignores the feed entirely. There is nothing here for a per-node gRPC stream to carry. Revisit in Phase 6, once central genuinely loses gossip visibility of site nodes and needs another way to know a pair's membership.
• deployment-acks — already migrated, but onto a different transport: it rides the Phase 2 ClusterClient transport (MeshTransport:Mode=ClusterClient) as a command-plane reply, not an observability broadcast. It was never a Phase 5 candidate.

See docs/Redundancy.md for how redundancy-state and the command transports fit together.

Authentication — fail-closed from day one

A shared node bearer key gates the stream: Telemetry:ApiKey (node/serve side) must equal TelemetryDial:ApiKey (central/dial side). TelemetryStreamAuthInterceptor enforces it — path-scoped to /telemetry.v1.TelemetryStreamService/, comparing the Authorization: Bearer token with CryptographicOperations.FixedTimeEquals, and rejecting with PermissionDenied. An unset key rejects every call rather than allowing an open stream — the same fail-closed posture as ConfigServeAuthInterceptor and LocalDbSyncAuthInterceptor.

This supersedes the design doc's §6.3, which had provisionally decided to "match ScadaBridge's unauthenticated posture for now" for inter-cluster transports. ScadaBridge itself has since closed that gap with this identical interceptor pattern, so Phase 5 ships authenticated from the start rather than deferring auth to a later hardening pass. See the design doc's superseded note for the full history.

Reconnect and reliability

Central's TelemetryDialSupervisor (an admin-role actor) runs one reconnecting dialer per driver node:

• Discovery — enabled, non-maintenance ClusterNode rows are read on start and refreshed every TelemetryDial:ContactRefreshSeconds (default 60), plus on an admin-side topology change. A row with a null GrpcPort is skipped (that node exposes no telemetry endpoint — logged, not an error). Dial targets are added/removed as the row set changes.
• Reconnect backoff — immediate first retry, then a fixed ~5 second backoff, indefinitely. The dialer never gives up permanently: this is an observability channel, not the data plane, so a persistently-unreachable node just means a persistently-stale panel, not a fleet fault.
• Generation stamping — each dialer carries a monotonically-increasing generation counter, so a late error or late event from a superseded stream (e.g. one that raced a reconnect) is recognized and ignored rather than corrupting the current stream's state.
• Snapshot replay on reconnect — the node-local hub keeps a last-value cache for driver-health and driver-resilience-status (there is nothing meaningful to replay for alerts/script-logs, which are append logs). On every new Subscribe — including a reconnect — the hub drains the cached snapshots to the new stream before live deltas, so IDriverStatusSnapshotStore / IDriverResilienceStatusStore re-prime immediately rather than sitting stale until the next natural event. alerts/script-logs simply tolerate the gap — a reconnect loses whatever transitions happened while disconnected, same as a DPS resubscribe would.
• Connection indicator — the /alerts and /script-log live/disconnected pill is driven by aggregate stream health across all dialers: connected when at least one node stream is up, disconnected when all are down. This matches today's DPS SubscribeAck/PostStop pill semantics — the pill has never meant "every node is live," only "the panel has a source." The two store-backed panels (driver-health, driver-resilience-status) carry no fleet-wide connection flag today, on gRPC or on DPS; per-row staleness is unchanged.

Configuration reference

See docs/Configuration.md § Telemetry / TelemetryDial for the full keys table. In brief:

• Telemetry (node/serve side): Mode (Dps|Grpc, default Dps), GrpcListenPort (0 = disabled — the driver node's dedicated telemetry h2c port), ApiKey (the shared node key; supply via ${secret:}/env, never commit).
• TelemetryDial (central side): Mode (Dps|Grpc), ApiKey (must equal the nodes' Telemetry:ApiKey), ContactRefreshSeconds (default 60), CallTimeoutSeconds (default 30).

A driver-role node with Telemetry:Mode=Grpc must set GrpcListenPort > 0 (nothing to serve otherwise), and Grpc mode on either side requires a non-empty ApiKey — both are enforced by ValidateOnStart validators, fail-fast at boot.

Relationship to the rest of the mesh program

• Same dark-switch discipline as Phases 2 and 3. MeshTransport:Mode, ConfigSource:Mode, and Telemetry:Mode/TelemetryDial:Mode are all independently-flippable, restart-only config changes, and in every case the "new" side is always wired so the flip needs no coordinated redeploy.
• Why the inversion matters for Phase 6. Every other cross-boundary transport in this program (MeshTransport, ConfigSource/ConfigServe) already has central as the addressable, discoverable side and the node as the caller or the callee-by-address. Telemetry is the one channel that is naturally node-sourced and fleet-wide-fanned, so putting central in the client role — dialing out to each node by its ClusterNode-recorded address — is what lets telemetry survive the mesh split cleanly: central never needs cluster membership with a site's mesh to keep watching it.
• Status: code-complete on feat/mesh-phase5; the live gate (flip the docker-dev rig to Grpc, confirm all panels stay green with the DPS telemetry bridges unspawned, kill-and-reconnect a site node) has not yet run. See docs/plans/2026-07-22-per-cluster-mesh-program.md § Phase 5 and docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md for the implementation plan.