docs(mesh-phase5): telemetry stream — Telemetry.md, config keys, supersede §6.3, program status

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-23 16:33:29 -04:00
parent 1104785c89
commit e99ea40e2f
6 changed files with 380 additions and 11 deletions
+51
View File
@@ -315,6 +315,57 @@ Flip only the site nodes with `OTOPCUA_CONFIG_MODE=FetchAndCache` at `docker com
`docs/Configuration.md` §`ConfigSource` / `ConfigServe`, `docs/Redundancy.md` §"Command transport", and
`docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md`.
## Live telemetry transport (`Telemetry` / `TelemetryDial`)
Per-cluster mesh **Phase 5** (code-complete on `feat/mesh-phase5`, live gate pending) added a second
transport for the four live-observability channels the AdminUI's `/alerts`, `/script-log`, and
`/hosts` panels feed on, selected by `Telemetry:Mode` (node/serve side) and `TelemetryDial:Mode`
(central/dial side): `Dps` — the four existing DPS SignalR bridges, **the default**, today's
behaviour — or `Grpc`, one gRPC server-streaming contract carrying all four channels as `oneof`
event kinds. **The direction is inverted from `ConfigSource`/`MeshTransport`: the driver node hosts
the gRPC server (dedicated Kestrel h2c port), and central is the client that dials in** — mirroring
ScadaBridge's `SiteStreamService` shape, and required so telemetry survives once the meshes split
(Phase 6) and central no longer shares a gossip ring with a site node. Things worth knowing before
touching it:
- **Migrated (4 channels):** `alerts` (`AlarmTransitionEvent`), `script-logs` (`ScriptLogEntry`),
`driver-health` (`DriverHealthChanged`), `driver-resilience-status`
(`DriverResilienceStatusChanged`) — feeding the exact same central sinks
(`IInProcessBroadcaster<...>`, `IDriverStatusSnapshotStore`, `IDriverResilienceStatusStore`) in
either mode.
- **Deferred, with reasons (do not read "seven observability topics" from the program sketch as the
delivered scope):** `redundancy-state` stays on DPS in every mode (pair-local, built from
`Cluster.State`, drives ServiceLevel + the Primary gate — genuinely mesh-bound by design, not an
observability panel); `fleet-status` is central-internal (`FleetStatusBroadcaster` builds it from
the admin node's own membership events, `Fleet.razor` polls the Config DB and ignores the feed —
never a node→central stream); `deployment-acks` already rides the Phase 2 `ClusterClient`
transport as a command-plane reply, not telemetry.
- **The node ALWAYS hosts and ALWAYS double-publishes, in both modes.** Every publish seam emits
into a node-local `ITelemetryLocalHub` **and** still publishes DPS, unconditionally; the gRPC
server binds whenever `Telemetry:GrpcListenPort > 0`, independent of `Telemetry:Mode`. Only
`TelemetryDial:Mode` on central actually switches anything: which of the two always-available
sources (DPS bridges vs. the `TelemetryDialSupervisor` dialer) feeds the sinks. This is what
makes the flip a config change, not a coordinated redeploy — same dark-switch discipline as
[MeshTransport](#mesh-command-transport-meshtransport) and [ConfigSource](#config-source-configsource--configserve).
- **Authenticated from day one, fail-closed** — a shared node bearer key (`Telemetry:ApiKey` ==
`TelemetryDial:ApiKey`), gated by `TelemetryStreamAuthInterceptor`
(`CryptographicOperations.FixedTimeEquals`, `PermissionDenied` on mismatch, an unset key rejects
every call). This **supersedes** design doc §6.3's "match ScadaBridge's unauthenticated posture
for now" — ScadaBridge itself has since closed that gap with this identical interceptor pattern.
- **Reconnect story:** central runs one reconnecting dialer per driver node (discovered from
`ClusterNode.Host`/`GrpcPort`, refreshed every `TelemetryDial:ContactRefreshSeconds`), immediate
first retry then ~5s fixed backoff, generation-stamped so a superseded stream's late error/event
is ignored, and it never permanently gives up (observability, not data plane). The node-local hub
replays last-value snapshots for `driver-health`/`driver-resilience-status` on every (re)subscribe
so those stores re-prime immediately after a reconnect; `alerts`/`script-logs` are append logs and
tolerate the gap. The `/alerts`/`/script-log` pill reflects aggregate stream health (live when ≥1
node stream is up) — the same semantics the DPS pill has always had.
See `docs/Telemetry.md` for the full architecture, `docs/Configuration.md`
§`Telemetry`/`TelemetryDial` for the config keys, and
`docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md` for the implementation plan and live
gate procedure.
## LocalDb pair-local store (Phases 1 + 2)
Every **driver-role** node keeps a consolidated `ZB.MOM.WW.LocalDb` SQLite database (the retired