dohertj2/lmxopcua
1
0
Fork 0
Code Issues 12 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs(mesh-phase5): telemetry stream — Telemetry.md, config keys, supersede §6.3, program status

Browse Source 
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-23 16:33:29 -04:00
parent 1104785c89
commit e99ea40e2f
6 changed files with 380 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plans/2026-07-22-per-cluster-mesh-program.md
+61 -11
View File
@@ -202,16 +202,66 @@ Phase 4 as code-complete, not verified, until this lands.**
**no ConfigDb connection string configured at all**; grep-level proof no driver-branch service can
resolve the ConfigDb context.
### Phase 5 — gRPC stream contract for live telemetry
**Scope:** one server-streaming contract carrying a `oneof` event, **cluster nodes host the gRPC
server, central dials in** (the inverted direction is the load-bearing ScadaBridge finding —
design §2); migrate the seven observability topics (`alerts`, `driver-health`,
`driver-resilience-status`, `fleet-status`, `script-logs`, plus redundancy-state distribution and
deployment-acks if Phase 2 left them on DPS); additive-only field evolution, contract locked by
test; per-panel reconnect story for the AdminUI (design §8 — losing gossip loses free fleet
observability).
**Exit gate:** all AdminUI live panels green against a pair with DPS telemetry topics deleted;
kill-and-reconnect of the central dialer recovers every stream.
### Phase 5 — gRPC stream contract for live telemetry — **code-complete on `feat/mesh-phase5`, live gate pending**
**Scope as shipped (narrowed from the program sketch's "seven observability topics," settled with the
user 2026-07-23 — mirrors how Phases 1 and 3 also honestly scoped down):** one server-streaming gRPC
contract carrying a `oneof` event with **four** kinds, **driver nodes host the gRPC server (dedicated
Kestrel h2c port), central dials in** — the inverted direction is the load-bearing ScadaBridge finding
(design §2), required so telemetry survives the Phase 6 mesh split without shared gossip membership.
**The four migrated channels:**
| DPS topic | Domain record | Central sink |
|---|---|---|
| `alerts` | `AlarmTransitionEvent` | `IInProcessBroadcaster<AlarmTransitionEvent>` (+ `AlertHub`) |
| `script-logs` | `ScriptLogEntry` | `IInProcessBroadcaster<ScriptLogEntry>` (+ `ScriptLogHub`) |
| `driver-health` | `DriverHealthChanged` | `IDriverStatusSnapshotStore` |
| `driver-resilience-status` | `DriverResilienceStatusChanged` | `IDriverResilienceStatusStore` |
**Three deferred, each with a settled reason (NOT migrated — do not read the program sketch's original
"seven" as the delivered scope):**
- `redundancy-state` — bidirectional, built from `Cluster.State`, pair-local control plane driving
ServiceLevel + the Primary gate. Stays on DPS in every mode; under Phase 6 it works in-mesh per pair.
Central's cross-pair *display* of redundancy is a Phase 6 concern, not a Phase 5 panel.
- `fleet-status` — central-internal; `FleetStatusBroadcaster` builds it from the admin node's own
cluster membership events and `Fleet.razor` polls the Config DB, ignoring the feed. Not a
node→central stream at all. Revisit in Phase 6 once central loses site-node gossip visibility.
- `deployment-acks` — already migrated, but onto the Phase 2 `ClusterClient` transport, as a
command-plane reply, not an observability broadcast.
**Dark switch:** `Telemetry:Mode` (node/serve, default `Dps`) and `TelemetryDial:Mode` (central/dial,
default `Dps`) flip independently to `Grpc`; both code paths are compiled into every binary, so
flipping is a config + restart, not a redeploy. The node **always** hosts the gRPC server whenever
`Telemetry:GrpcListenPort > 0` and **always** emits into its node-local hub + publishes DPS, in both
modes — only central's ingest source (`TelemetryDial:Mode`) actually switches which of the two
always-available sources feeds the AdminUI sinks. The AdminUI panels/components are unchanged; only
the sinks' upstream feed swaps.
**Auth from day one, fail-closed:** a shared node bearer key (`Telemetry:ApiKey` ==
`TelemetryDial:ApiKey`), gated by `TelemetryStreamAuthInterceptor`
(`CryptographicOperations.FixedTimeEquals`, `PermissionDenied` on mismatch, unset key rejects every
call). This supersedes design §6.3's "match ScadaBridge's unauthenticated posture for now" — ScadaBridge
itself has since closed that gap with this identical interceptor pattern (see the supersede note added
to design §6.3, and `docs/Telemetry.md` § Authentication).
**Reconnect story:** central's `TelemetryDialSupervisor` runs one reconnecting dialer per driver node,
discovered from `ClusterNode` rows (`Host`+`GrpcPort`, refreshed every
`TelemetryDial:ContactRefreshSeconds`); immediate first retry then ~5s fixed backoff, generation-stamped
so a superseded stream's late error/event is ignored; the dialer never permanently gives up
(observability, not data plane). The node-local hub replays last-value snapshots for
`driver-health`/`driver-resilience-status` on every (re)subscribe so those stores re-prime immediately;
`alerts`/`script-logs` are append logs and tolerate the gap. The `/alerts`/`/script-log` pill reflects
aggregate stream health (live when ≥1 node stream is up), matching today's DPS pill semantics.
**Docs shipped:** `docs/Telemetry.md` (new, canonical guide), `docs/Configuration.md` §
`Telemetry`/`TelemetryDial`, `docs/Redundancy.md` cross-reference, this section, and the design §6.3
supersede note.
**Exit gate (not yet run):** all AdminUI live panels green against the docker-dev rig with
`Telemetry:Mode=Grpc`/`TelemetryDial:Mode=Grpc` and the four DPS telemetry bridges NOT spawned;
kill-and-reconnect of a site node's dialer recovers every stream (driver-health/resilience re-prime
from the node hub's snapshot replay). Treat Phase 5 as **code-complete, not verified**, until this
lands — see `docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md` Task 12.
### Phase 6 — Mesh partition + co-location topology
**Scope:** per-cluster seed nodes — ~~adopt ScadaBridge's self-first ordering and RETIRE the
@@ -267,6 +317,6 @@ resource sizing on the site VMs should be checked once both products run the ful
| 2 ClusterClient transport | **DONE 2026-07-22** (live gate PASSED, shipped dark) — shipped dark (`MeshTransport:Mode=Dps` default), both comm actors registered in both modes. Corrections: `SendToAll` not `Send`, one fleet-wide client not one per cluster, `/user/node-communication` not `/user/cluster-communication`. The gate's "Ask timing out" item does not exist to test — no cross-boundary Ask in this phase; see the phase section. |
| 3 fetch-and-cache | **DONE 2026-07-23**, merged `d01b0695`, live gate PASSED, pushed to origin (scadaproj umbrella index updated + pushed `b5e7bc8`). gRPC fetch RPC + shared node key; dark switch `ConfigSource:Mode` (Direct default). Rig flip: `OTOPCUA_CONFIG_MODE=FetchAndCache` on the site nodes. See `2026-07-22-mesh-phase3-config-fetch-and-cache.md`. |
| 4 cut driver ConfigDb | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase4` (Tasks 0–8 + 1b + 10–11; Task 9 table-drop deferred). ConfigDb admin-only; driver-only ⇒ FetchAndCache (validator); `DbHealthProbeActor` not spawned driver-only (client-visible ServiceLevel 240/250 with central SQL down — proven live); alarm condition state in replicated LocalDb `alarm_condition_state`; central persists acks; `OpcUaPublish` guard split fixes the driver-only address-space wipe; driver-only LDAP maps from appsettings only (6th consumer found mid-phase). Gate: deploy sealed green w/ 4 DB-less site nodes acking, ServiceLevel held 240 w/ SQL down, restart booted last-known-good from the LocalDb pointer. See `2026-07-23-mesh-phase4-cut-driver-configdb.md` + `2026-07-23-mesh-phase4-live-gate.md`. |
| 5 gRPC telemetry | not started |
| 5 gRPC telemetry | **code-complete (live gate pending)** on `feat/mesh-phase5`. 4-channel scope (`alerts`/`script-logs`/`driver-health`/`driver-resilience-status`), 3 deferred with rationale (`redundancy-state`, `fleet-status`, `deployment-acks`); dark switch `Telemetry:Mode`/`TelemetryDial:Mode` (Dps default); auth-from-day-one fail-closed bearer key, superseding design §6.3. See `docs/Telemetry.md` + `2026-07-23-mesh-phase5-grpc-telemetry-stream.md`. |
| 6 mesh partition + co-location | not started |
| 7 drill + live gates | not started |