docs(mesh-phase5): telemetry stream — Telemetry.md, config keys, supersede §6.3, program status
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
@@ -315,6 +315,57 @@ Flip only the site nodes with `OTOPCUA_CONFIG_MODE=FetchAndCache` at `docker com
|
||||
`docs/Configuration.md` §`ConfigSource` / `ConfigServe`, `docs/Redundancy.md` §"Command transport", and
|
||||
`docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md`.
|
||||
|
||||
## Live telemetry transport (`Telemetry` / `TelemetryDial`)
|
||||
|
||||
Per-cluster mesh **Phase 5** (code-complete on `feat/mesh-phase5`, live gate pending) added a second
|
||||
transport for the four live-observability channels the AdminUI's `/alerts`, `/script-log`, and
|
||||
`/hosts` panels feed on, selected by `Telemetry:Mode` (node/serve side) and `TelemetryDial:Mode`
|
||||
(central/dial side): `Dps` — the four existing DPS SignalR bridges, **the default**, today's
|
||||
behaviour — or `Grpc`, one gRPC server-streaming contract carrying all four channels as `oneof`
|
||||
event kinds. **The direction is inverted from `ConfigSource`/`MeshTransport`: the driver node hosts
|
||||
the gRPC server (dedicated Kestrel h2c port), and central is the client that dials in** — mirroring
|
||||
ScadaBridge's `SiteStreamService` shape, and required so telemetry survives once the meshes split
|
||||
(Phase 6) and central no longer shares a gossip ring with a site node. Things worth knowing before
|
||||
touching it:
|
||||
|
||||
- **Migrated (4 channels):** `alerts` (`AlarmTransitionEvent`), `script-logs` (`ScriptLogEntry`),
|
||||
`driver-health` (`DriverHealthChanged`), `driver-resilience-status`
|
||||
(`DriverResilienceStatusChanged`) — feeding the exact same central sinks
|
||||
(`IInProcessBroadcaster<...>`, `IDriverStatusSnapshotStore`, `IDriverResilienceStatusStore`) in
|
||||
either mode.
|
||||
- **Deferred, with reasons (do not read "seven observability topics" from the program sketch as the
|
||||
delivered scope):** `redundancy-state` stays on DPS in every mode (pair-local, built from
|
||||
`Cluster.State`, drives ServiceLevel + the Primary gate — genuinely mesh-bound by design, not an
|
||||
observability panel); `fleet-status` is central-internal (`FleetStatusBroadcaster` builds it from
|
||||
the admin node's own membership events, `Fleet.razor` polls the Config DB and ignores the feed —
|
||||
never a node→central stream); `deployment-acks` already rides the Phase 2 `ClusterClient`
|
||||
transport as a command-plane reply, not telemetry.
|
||||
- **The node ALWAYS hosts and ALWAYS double-publishes, in both modes.** Every publish seam emits
|
||||
into a node-local `ITelemetryLocalHub` **and** still publishes DPS, unconditionally; the gRPC
|
||||
server binds whenever `Telemetry:GrpcListenPort > 0`, independent of `Telemetry:Mode`. Only
|
||||
`TelemetryDial:Mode` on central actually switches anything: which of the two always-available
|
||||
sources (DPS bridges vs. the `TelemetryDialSupervisor` dialer) feeds the sinks. This is what
|
||||
makes the flip a config change, not a coordinated redeploy — same dark-switch discipline as
|
||||
[MeshTransport](#mesh-command-transport-meshtransport) and [ConfigSource](#config-source-configsource--configserve).
|
||||
- **Authenticated from day one, fail-closed** — a shared node bearer key (`Telemetry:ApiKey` ==
|
||||
`TelemetryDial:ApiKey`), gated by `TelemetryStreamAuthInterceptor`
|
||||
(`CryptographicOperations.FixedTimeEquals`, `PermissionDenied` on mismatch, an unset key rejects
|
||||
every call). This **supersedes** design doc §6.3's "match ScadaBridge's unauthenticated posture
|
||||
for now" — ScadaBridge itself has since closed that gap with this identical interceptor pattern.
|
||||
- **Reconnect story:** central runs one reconnecting dialer per driver node (discovered from
|
||||
`ClusterNode.Host`/`GrpcPort`, refreshed every `TelemetryDial:ContactRefreshSeconds`), immediate
|
||||
first retry then ~5s fixed backoff, generation-stamped so a superseded stream's late error/event
|
||||
is ignored, and it never permanently gives up (observability, not data plane). The node-local hub
|
||||
replays last-value snapshots for `driver-health`/`driver-resilience-status` on every (re)subscribe
|
||||
so those stores re-prime immediately after a reconnect; `alerts`/`script-logs` are append logs and
|
||||
tolerate the gap. The `/alerts`/`/script-log` pill reflects aggregate stream health (live when ≥1
|
||||
node stream is up) — the same semantics the DPS pill has always had.
|
||||
|
||||
See `docs/Telemetry.md` for the full architecture, `docs/Configuration.md`
|
||||
§`Telemetry`/`TelemetryDial` for the config keys, and
|
||||
`docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md` for the implementation plan and live
|
||||
gate procedure.
|
||||
|
||||
## LocalDb pair-local store (Phases 1 + 2)
|
||||
|
||||
Every **driver-role** node keeps a consolidated `ZB.MOM.WW.LocalDb` SQLite database (the retired
|
||||
|
||||
Reference in New Issue
Block a user