diff --git a/CLAUDE.md b/CLAUDE.md index 3a2f7ece..9073af3f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -315,6 +315,57 @@ Flip only the site nodes with `OTOPCUA_CONFIG_MODE=FetchAndCache` at `docker com `docs/Configuration.md` §`ConfigSource` / `ConfigServe`, `docs/Redundancy.md` §"Command transport", and `docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md`. +## Live telemetry transport (`Telemetry` / `TelemetryDial`) + +Per-cluster mesh **Phase 5** (code-complete on `feat/mesh-phase5`, live gate pending) added a second +transport for the four live-observability channels the AdminUI's `/alerts`, `/script-log`, and +`/hosts` panels feed on, selected by `Telemetry:Mode` (node/serve side) and `TelemetryDial:Mode` +(central/dial side): `Dps` — the four existing DPS SignalR bridges, **the default**, today's +behaviour — or `Grpc`, one gRPC server-streaming contract carrying all four channels as `oneof` +event kinds. **The direction is inverted from `ConfigSource`/`MeshTransport`: the driver node hosts +the gRPC server (dedicated Kestrel h2c port), and central is the client that dials in** — mirroring +ScadaBridge's `SiteStreamService` shape, and required so telemetry survives once the meshes split +(Phase 6) and central no longer shares a gossip ring with a site node. Things worth knowing before +touching it: + +- **Migrated (4 channels):** `alerts` (`AlarmTransitionEvent`), `script-logs` (`ScriptLogEntry`), + `driver-health` (`DriverHealthChanged`), `driver-resilience-status` + (`DriverResilienceStatusChanged`) — feeding the exact same central sinks + (`IInProcessBroadcaster<...>`, `IDriverStatusSnapshotStore`, `IDriverResilienceStatusStore`) in + either mode. +- **Deferred, with reasons (do not read "seven observability topics" from the program sketch as the + delivered scope):** `redundancy-state` stays on DPS in every mode (pair-local, built from + `Cluster.State`, drives ServiceLevel + the Primary gate — genuinely mesh-bound by design, not an + observability panel); `fleet-status` is central-internal (`FleetStatusBroadcaster` builds it from + the admin node's own membership events, `Fleet.razor` polls the Config DB and ignores the feed — + never a node→central stream); `deployment-acks` already rides the Phase 2 `ClusterClient` + transport as a command-plane reply, not telemetry. +- **The node ALWAYS hosts and ALWAYS double-publishes, in both modes.** Every publish seam emits + into a node-local `ITelemetryLocalHub` **and** still publishes DPS, unconditionally; the gRPC + server binds whenever `Telemetry:GrpcListenPort > 0`, independent of `Telemetry:Mode`. Only + `TelemetryDial:Mode` on central actually switches anything: which of the two always-available + sources (DPS bridges vs. the `TelemetryDialSupervisor` dialer) feeds the sinks. This is what + makes the flip a config change, not a coordinated redeploy — same dark-switch discipline as + [MeshTransport](#mesh-command-transport-meshtransport) and [ConfigSource](#config-source-configsource--configserve). +- **Authenticated from day one, fail-closed** — a shared node bearer key (`Telemetry:ApiKey` == + `TelemetryDial:ApiKey`), gated by `TelemetryStreamAuthInterceptor` + (`CryptographicOperations.FixedTimeEquals`, `PermissionDenied` on mismatch, an unset key rejects + every call). This **supersedes** design doc §6.3's "match ScadaBridge's unauthenticated posture + for now" — ScadaBridge itself has since closed that gap with this identical interceptor pattern. +- **Reconnect story:** central runs one reconnecting dialer per driver node (discovered from + `ClusterNode.Host`/`GrpcPort`, refreshed every `TelemetryDial:ContactRefreshSeconds`), immediate + first retry then ~5s fixed backoff, generation-stamped so a superseded stream's late error/event + is ignored, and it never permanently gives up (observability, not data plane). The node-local hub + replays last-value snapshots for `driver-health`/`driver-resilience-status` on every (re)subscribe + so those stores re-prime immediately after a reconnect; `alerts`/`script-logs` are append logs and + tolerate the gap. The `/alerts`/`/script-log` pill reflects aggregate stream health (live when ≥1 + node stream is up) — the same semantics the DPS pill has always had. + +See `docs/Telemetry.md` for the full architecture, `docs/Configuration.md` +§`Telemetry`/`TelemetryDial` for the config keys, and +`docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md` for the implementation plan and live +gate procedure. + ## LocalDb pair-local store (Phases 1 + 2) Every **driver-role** node keeps a consolidated `ZB.MOM.WW.LocalDb` SQLite database (the retired diff --git a/docs/Configuration.md b/docs/Configuration.md index 734867d8..0b3b76dd 100644 --- a/docs/Configuration.md +++ b/docs/Configuration.md @@ -177,6 +177,60 @@ PersistNodeAck`) and LDAP group→role DB grants (a driver-only node maps LDAP g the `Security:Ldap:GroupToRole` appsettings baseline only — see `docs/Redundancy.md` and `CLAUDE.md` §"Phase 4" for the full picture, including the client-visible ServiceLevel change). +### `Telemetry` / `TelemetryDial` (live telemetry transport) + +- **Purpose:** [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) **Phase 5**. Selects how + the four live-observability channels (`alerts`, `script-logs`, `driver-health`, + `driver-resilience-status`) reach central: `Dps` (DistributedPubSub — the four existing SignalR + bridges, today's behaviour) or `Grpc` (a driver node hosts one gRPC server-streaming contract over a + dedicated h2c port; central dials in and feeds the same in-process sinks). `Telemetry` is the + **node/serve** side; `TelemetryDial` is the **central/dial** side. +- **Options classes:** `TelemetryOptions` (`SectionName = "Telemetry"`) + `TelemetryDialOptions` + (`SectionName = "TelemetryDial"`), both in `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`. Bound by + `AddOtOpcUaCluster(config)`; their validators run at `ValidateOnStart`. + +**`Telemetry` (node-side, serve):** + +| Key | Type | Default | Meaning | +|---|---|---|---| +| `Mode` | string | `Dps` | `Dps` = no change, node just keeps publishing DPS as it always has. `Grpc` = central-ingest hint (see `TelemetryDial:Mode` below — this key does not gate anything on the node itself). Any other value **refuses to start**. | +| `GrpcListenPort` | int | `0` | Dedicated **h2c-only** Kestrel port for the telemetry stream service. `0` = disabled (nothing bound). The node hosts the server whenever this is `> 0`, **regardless of `Mode`**. | +| `ApiKey` | string | `""` | Shared bearer key the serve-side interceptor checks (constant-time, **fail-closed**: an unset key rejects every call). **Supply via env `Telemetry__ApiKey` — never commit.** | + +**`TelemetryDial` (central-side, dial):** + +| Key | Type | Default | Meaning | +|---|---|---|---| +| `Mode` | string | `Dps` | `Dps` = the four DPS SignalR bridges (alert, script-log, driver-status, resilience) subscribe as today. `Grpc` = those four bridges are **not spawned**; the `TelemetryDialSupervisor` actor dials every driver node instead and feeds the same sinks. Any other value **refuses to start**. | +| `ApiKey` | string | `""` | Shared bearer key; must equal the nodes' `Telemetry:ApiKey`. **Supply via env `TelemetryDial__ApiKey` — never commit.** Required under `Grpc`. | +| `ContactRefreshSeconds` | int | `60` | How often central rebuilds its dial-target set from the `ClusterNode` rows (`Host` + `GrpcPort`). | +| `CallTimeoutSeconds` | int | `30` | Per-call deadline for the streaming RPC's keepalive/health bookkeeping. | + +**It is a dark switch; the node always hosts.** Unlike `ConfigServe`/`MeshTransport`, where the serve +side is gated on a role, the telemetry server binds whenever `Telemetry:GrpcListenPort > 0` — hosting +is unconditional on a driver-role node, in both modes, because the node also always emits into its +local hub and always publishes DPS. Only `TelemetryDial:Mode` on central actually changes behaviour: +which of the two already-always-available sources (DPS bridges vs. the gRPC dial supervisor) feeds the +AdminUI sinks. Flipping either side is a config change plus restart, not a redeploy — the same +discipline as `MeshTransport:Mode` and `ConfigSource:Mode`. + +**A driver-role node with `Telemetry:Mode=Grpc` must set `GrpcListenPort > 0`** (nothing to serve +otherwise) — enforced by `TelemetryOptionsValidator`. **`Grpc` mode on either side requires a +non-empty `ApiKey`** — fail-closed, same as `ConfigServe`/`LocalDb:Replication`. + +**Three DPS observability topics stay out of scope for this transport, on purpose:** +`redundancy-state` (pair-local, built from `Cluster.State`, stays on DPS in both modes — see +[`Redundancy.md`](Redundancy.md#command-transport-centralnode)), `fleet-status` (central-internal; +`Fleet.razor` polls the Config DB, not any stream), and `deployment-acks` (already rides the Phase 2 +`ClusterClient` transport as a command-plane reply). See +[`docs/Telemetry.md`](Telemetry.md) for the full architecture, the reconnect/generation-stamped +dialer story, and the rationale for the node-hosts/central-dials inversion. + +On the docker-dev rig every driver-role node carries `Telemetry:GrpcListenPort` (`4056`) + a +committed dev key, and central carries the matching `TelemetryDial:ApiKey`; both `Mode` keys stay +unset (⇒ `Dps`) until the rig is flipped with `Telemetry__Mode=Grpc` / `TelemetryDial__Mode=Grpc` at +`docker compose up`. + ### `ConnectionStrings` → `ConfigDb` - **Purpose:** the central Config DB connection string. **Required on admin-role nodes only** — `Program.cs` calls `AddOtOpcUaConfigDb` when `hasAdmin` (per-cluster mesh Phase 4). A driver-only node holds no `ConfigDb` connection string; it fetches its configuration via `ConfigSource:Mode=FetchAndCache` instead (see above). A fused `admin,driver` node still requires it. @@ -283,6 +337,7 @@ These are consumed by the driver **integration-test fixtures** (under `tests/Dri - [`security.md`](security.md) — transport security, OPC UA authentication, LDAP (`Security:Ldap`), data-plane ACLs, control-plane roles. - [`Redundancy.md`](Redundancy.md) — the `Cluster` section in the context of warm/hot redundancy, ServiceLevel, peer discovery. +- [`Telemetry.md`](Telemetry.md) — the `Telemetry`/`TelemetryDial` gRPC live-telemetry stream: direction, dark switch, the four migrated channels + three deferrals, auth, reconnect story. - [`ServiceHosting.md`](ServiceHosting.md) — role-based host wiring and `OTOPCUA_ROLES`. - [`docs/drivers/Galaxy.md`](drivers/Galaxy.md) — Galaxy/MxAccess driver overview. - [`docs/v2/config-db-schema.md`](v2/config-db-schema.md) — the full Config DB schema. diff --git a/docs/Redundancy.md b/docs/Redundancy.md index 2733b966..d08756c8 100644 --- a/docs/Redundancy.md +++ b/docs/Redundancy.md @@ -465,6 +465,15 @@ are untouched, and `redundancy-state` itself stays on DPS (it is bidirectional a `Cluster.State`, which makes it genuinely mesh-bound — the hardest remaining dependency before the meshes can split). +**Observability telemetry took the opposite path (Phase 5).** The four live-panel channels +(`alerts`, `script-logs`, `driver-health`, `driver-resilience-status`) moved off DPS onto a +dedicated gRPC server-streaming contract — each driver node hosts, central dials in — selected by +`Telemetry:Mode`/`TelemetryDial:Mode` (default `Dps`). `redundancy-state` is deliberately **not** +part of that migration: it stays on DPS in every mode because it is pair-local and Cluster.State-built, +not a fleet-wide observability broadcast. See [`docs/Telemetry.md`](Telemetry.md) for the full +architecture and [`Configuration.md` § `Telemetry`/`TelemetryDial`](Configuration.md#telemetry--telemetrydial-live-telemetry-transport) +for the config keys. + **Config bytes travel out-of-band (Phase 3).** Commands stay on one of the two transports above, but the *configuration artifact itself* does not ride any Akka message — `DispatchDeployment` is payload-free (`DeploymentId` + `RevisionHash`). Under `ConfigSource:Mode = FetchAndCache` (see diff --git a/docs/Telemetry.md b/docs/Telemetry.md new file mode 100644 index 00000000..a24570f1 --- /dev/null +++ b/docs/Telemetry.md @@ -0,0 +1,190 @@ +# Live Telemetry Transport (v2) + +## Overview + +The AdminUI's live observability panels (`/alerts`, `/script-log`, `/hosts` driver table, driver +resilience status) are fed by four node→central event channels. Historically all four rode +DistributedPubSub (DPS) on the shared Akka mesh — the same gossip ring that carries redundancy +state and the command-plane topics. Per-cluster mesh **Phase 5** adds a second transport for these +four channels only: one gRPC server-streaming contract, selected per node/central pair by +`Telemetry:Mode` / `TelemetryDial:Mode` (`Dps` default | `Grpc`). + +This is the same motivation as [Phase 2's `MeshTransport`](Configuration.md#meshtransport-centralnode-command-transport) +and [Phase 3's `ConfigSource`/`ConfigServe`](Configuration.md#configsource--configserve-config-fetch-and-cache): +DPS only works when central and the node share a gossip ring, and [Phase 6](plans/2026-07-21-per-cluster-mesh-design.md) +splits the fleet into one Akka mesh per application `Cluster`. Once that split lands, a driver node +in a site's mesh is no longer a cluster member of central's mesh, so DPS can no longer reach it — +telemetry needs its own out-of-band transport, same as commands (Phase 2) and config bytes (Phase 3). + +## Direction: the node hosts, central dials + +**Load-bearing and easy to get backwards:** each **driver** node hosts the telemetry gRPC server +(a dedicated Kestrel h2c listener); **central (admin role) is the client** and dials in. This +mirrors the inversion ScadaBridge already uses for the same problem (its `SiteStreamService`). + +``` + driver node central (admin) +┌─────────────────────┐ ┌──────────────────────────┐ +│ 4 publish seams │ │ TelemetryDialSupervisor │ +│ -> node-local hub │ Subscribe │ - discovers ClusterNode │ +│ -> DPS (unchanged) │◄─────────────│ rows (Host+GrpcPort) │ +│ │ (gRPC │ - one reconnecting │ +│ TelemetryStreamGrpc- │ stream) │ dialer per node │ +│ Service (h2c server) │─────────────►│ - feeds the SAME sinks │ +│ Telemetry:GrpcListen- │ TelemetryEvent │ the DPS bridges feed │ +│ Port │ (oneof, 4 kinds)│ today │ +└─────────────────────┘ └──────────────────────────┘ +``` + +A fused `admin,driver` node both hosts (as driver) and dials (as admin) — it dials itself plus its +redundant pair peer, the same way central dials site nodes. + +Central discovers driver-node telemetry endpoints from the `ClusterNode` table (`Host` + +`GrpcPort`, the latter added in Phase 1 for exactly this purpose) — the same DB-sourced discovery +Phase 1's ack set and Phase 2's `ClusterClient` contact set already use. No shared gossip +membership is required to find or dial a node. + +## The dark switch + +`Telemetry:Mode` (node/serve side) and `TelemetryDial:Mode` (central/dial side) each default to +`Dps` and can be independently set to `Grpc`. Both code paths are compiled into every binary — +flipping either flag is an appsettings/env change plus a restart, **not a redeploy or rebuild** — +the same discipline as `MeshTransport:Mode` and `ConfigSource:Mode`. + +What actually changes per mode, precisely: + +- **The node ALWAYS does both things, regardless of `Telemetry:Mode`.** Every one of the four + publish seams emits into the node-local `ITelemetryLocalHub` **and** publishes to DPS, + unconditionally. The node additionally **always** hosts the gRPC server whenever + `Telemetry:GrpcListenPort > 0` — hosting is not mode-gated at all. `Telemetry:Mode` only affects + central's own bookkeeping about which side it expects to be consulted (it is otherwise inert on + the node). +- **Only central's ingest source switches**, driven by `TelemetryDial:Mode`: + - `Dps` (default) — today's four DPS SignalR bridges (alert, script-log, driver-status, + resilience) subscribe and feed the AdminUI sinks. Unchanged behavior. + - `Grpc` — those four DPS bridges are **not spawned**. Instead the `TelemetryDialSupervisor` + actor dials every driver node's gRPC stream and feeds the **identical** in-process sinks + (`IInProcessBroadcaster`, `IInProcessBroadcaster`, + `IDriverStatusSnapshotStore`, `IDriverResilienceStatusStore`). +- **The AdminUI panels themselves are untouched.** `/alerts`, `/script-log`, and the `/hosts` + driver table all read from the same sinks in both modes — only the sinks' upstream feed swaps. + The `fleet-status` bridge (out of Phase 5 scope — see below) stays on DPS in both modes. + +Because hosting is unconditional and both ingest paths are always compiled in, a fleet can be +flipped node-by-node and central-by-central with no coordination window where telemetry is lost — +the node is always serving, so central can switch to `Grpc` whenever it likes. + +## The four migrated channels + +| DPS topic (today) | Domain record | Central sink fed | +|---|---|---| +| `alerts` | `AlarmTransitionEvent` | `IInProcessBroadcaster` (+ `AlertHub`); `/alerts` page; live/disconnected pill | +| `script-logs` | `ScriptLogEntry` | `IInProcessBroadcaster` (+ `ScriptLogHub`); `/script-log` page; live pill | +| `driver-health` | `DriverHealthChanged` | `IDriverStatusSnapshotStore`; `/hosts` driver table | +| `driver-resilience-status` | `DriverResilienceStatusChanged` | `IDriverResilienceStatusStore` | + +These four are carried as `oneof` event kinds on one `TelemetryStreamService.Subscribe` RPC — a +single stream per (central, driver-node) pair carries all four, rather than one stream per topic. +Proto field evolution is additive-only (never renumber/reuse a tag), locked by a contract test that +reflects over the `oneof` cases. + +## The three deferred channels (NOT migrated in Phase 5 — do not read this as "seven done") + +The program sketch originally named seven observability topics for Phase 5. Three were scoped out, +each for a distinct, settled reason: + +- **`redundancy-state`** — bidirectional, built directly from `Cluster.State`, and **pair-local**: + it drives ServiceLevel and the Primary data-plane gate, consumed in-process by + `OpcUaPublishActor`, `ScriptedAlarmHostActor`, `DriverHostActor`, and `HistorianAdapterActor`. + It stays on DPS in **both** `MeshTransport` modes today, and it is genuinely mesh-bound by + design — under Phase 6 each pair keeps sharing its own small mesh, so DPS keeps working for it + in-mesh. Central's *display* of each pair's redundancy state is a Phase 6 cross-mesh concern + (a future observability channel, once central no longer shares gossip with any site pair), not a + Phase 5 telemetry-panel migration. +- **`fleet-status`** — **central-internal**, not a node→central stream at all. The admin singleton + `FleetStatusBroadcaster` builds it from the admin node's own cluster membership/reachability/ + leader events, and `Fleet.razor` **polls the Config DB** and ignores the feed entirely. There is + nothing here for a per-node gRPC stream to carry. Revisit in Phase 6, once central genuinely + loses gossip visibility of site nodes and needs another way to know a pair's membership. +- **`deployment-acks`** — already migrated, but onto a different transport: it rides the Phase 2 + `ClusterClient` transport (`MeshTransport:Mode=ClusterClient`) as a command-plane reply, not an + observability broadcast. It was never a Phase 5 candidate. + +See [`docs/Redundancy.md`](Redundancy.md#command-transport-centralnode) for how `redundancy-state` +and the command transports fit together. + +## Authentication — fail-closed from day one + +A shared node bearer key gates the stream: `Telemetry:ApiKey` (node/serve side) must equal +`TelemetryDial:ApiKey` (central/dial side). `TelemetryStreamAuthInterceptor` enforces it — path-scoped +to `/telemetry.v1.TelemetryStreamService/`, comparing the `Authorization: Bearer` token with +`CryptographicOperations.FixedTimeEquals`, and rejecting with `PermissionDenied`. An **unset** key +rejects every call rather than allowing an open stream — the same fail-closed posture as +`ConfigServeAuthInterceptor` and `LocalDbSyncAuthInterceptor`. + +This **supersedes** [the design doc's §6.3](plans/2026-07-21-per-cluster-mesh-design.md), which had +provisionally decided to "match ScadaBridge's unauthenticated posture for now" for inter-cluster +transports. ScadaBridge itself has since closed that gap with this identical +interceptor pattern, so Phase 5 ships authenticated from the start rather than deferring auth to a +later hardening pass. See the design doc's superseded note for the full history. + +## Reconnect and reliability + +Central's `TelemetryDialSupervisor` (an admin-role actor) runs **one reconnecting dialer per +driver node**: + +- **Discovery** — enabled, non-maintenance `ClusterNode` rows are read on start and refreshed every + `TelemetryDial:ContactRefreshSeconds` (default 60), plus on an admin-side topology change. A row + with a null `GrpcPort` is skipped (that node exposes no telemetry endpoint — logged, not an + error). Dial targets are added/removed as the row set changes. +- **Reconnect backoff** — immediate first retry, then a fixed ~5 second backoff, indefinitely. The + dialer never gives up permanently: this is an observability channel, not the data plane, so a + persistently-unreachable node just means a persistently-stale panel, not a fleet fault. +- **Generation stamping** — each dialer carries a monotonically-increasing generation counter, so + a late error or late event from a superseded stream (e.g. one that raced a reconnect) is + recognized and ignored rather than corrupting the current stream's state. +- **Snapshot replay on reconnect** — the node-local hub keeps a last-value cache for + `driver-health` and `driver-resilience-status` (there is nothing meaningful to replay for + `alerts`/`script-logs`, which are append logs). On every new `Subscribe` — including a + reconnect — the hub drains the cached snapshots to the new stream before live deltas, so + `IDriverStatusSnapshotStore` / `IDriverResilienceStatusStore` re-prime immediately rather than + sitting stale until the next natural event. `alerts`/`script-logs` simply tolerate the gap — a + reconnect loses whatever transitions happened while disconnected, same as a DPS resubscribe would. +- **Connection indicator** — the `/alerts` and `/script-log` live/disconnected pill is driven by + **aggregate** stream health across all dialers: connected when **at least one** node stream is + up, disconnected when all are down. This matches today's DPS `SubscribeAck`/`PostStop` pill + semantics — the pill has never meant "every node is live," only "the panel has a source." The + two store-backed panels (`driver-health`, `driver-resilience-status`) carry no fleet-wide + connection flag today, on gRPC or on DPS; per-row staleness is unchanged. + +## Configuration reference + +See [`docs/Configuration.md` § `Telemetry` / `TelemetryDial`](Configuration.md#telemetry--telemetrydial-live-telemetry-transport) +for the full keys table. In brief: + +- **`Telemetry`** (node/serve side): `Mode` (`Dps`|`Grpc`, default `Dps`), `GrpcListenPort` (`0` = + disabled — the driver node's dedicated telemetry h2c port), `ApiKey` (the shared node key; supply + via `${secret:}`/env, never commit). +- **`TelemetryDial`** (central side): `Mode` (`Dps`|`Grpc`), `ApiKey` (must equal the nodes' + `Telemetry:ApiKey`), `ContactRefreshSeconds` (default `60`), `CallTimeoutSeconds` (default `30`). + +A driver-role node with `Telemetry:Mode=Grpc` must set `GrpcListenPort > 0` (nothing to serve +otherwise), and `Grpc` mode on either side requires a non-empty `ApiKey` — both are enforced by +`ValidateOnStart` validators, fail-fast at boot. + +## Relationship to the rest of the mesh program + +- **Same dark-switch discipline as Phases 2 and 3.** `MeshTransport:Mode`, `ConfigSource:Mode`, and + `Telemetry:Mode`/`TelemetryDial:Mode` are all independently-flippable, restart-only config + changes, and in every case the "new" side is always wired so the flip needs no coordinated + redeploy. +- **Why the inversion matters for Phase 6.** Every other cross-boundary transport in this program + (`MeshTransport`, `ConfigSource`/`ConfigServe`) already has central as the addressable, discoverable + side and the node as the caller or the callee-by-address. Telemetry is the one channel that is + naturally node-sourced and fleet-wide-fanned, so putting central in the client role — dialing out + to each node by its `ClusterNode`-recorded address — is what lets telemetry survive the mesh split + cleanly: central never needs cluster membership with a site's mesh to keep watching it. +- **Status:** code-complete on `feat/mesh-phase5`; the live gate (flip the docker-dev rig to `Grpc`, + confirm all panels stay green with the DPS telemetry bridges unspawned, kill-and-reconnect a site + node) has not yet run. See `docs/plans/2026-07-22-per-cluster-mesh-program.md` § Phase 5 and + `docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md` for the implementation plan. diff --git a/docs/plans/2026-07-21-per-cluster-mesh-design.md b/docs/plans/2026-07-21-per-cluster-mesh-design.md index 8ccd3ed6..8a39d63b 100644 --- a/docs/plans/2026-07-21-per-cluster-mesh-design.md +++ b/docs/plans/2026-07-21-per-cluster-mesh-design.md @@ -285,6 +285,20 @@ trusted network. Recorded as an accepted risk rather than an oversight, with two it is revisited: `LocalDbSyncAuthInterceptor` is already in our tree as a fail-closed template, and the surface most worth gating first is any Pull-style RPC that returns historical rows. +**SUPERSEDED 2026-07-23.** The Phase 5 telemetry stream (`TelemetryStreamService` — the four +node→central observability channels) shipped **authenticated from day one**: a shared node bearer +key (`Telemetry:ApiKey` == `TelemetryDial:ApiKey`), enforced by a fail-closed +`TelemetryStreamAuthInterceptor` (`CryptographicOperations.FixedTimeEquals`, `PermissionDenied` on +mismatch, and — load-bearing — an **unset** key rejects every call rather than allowing an open +stream). This is exactly the `LocalDbSyncAuthInterceptor`/`ConfigServeAuthInterceptor` pattern this +section flagged as the fail-closed template to reach for, applied immediately rather than deferred. +ScadaBridge itself has since closed the same "unauthenticated inter-cluster transport" gap with this +identical interceptor pattern, so this decision no longer matches ScadaBridge's current posture +either — it matches where ScadaBridge moved to. See [`docs/Telemetry.md`](../Telemetry.md) § +Authentication for the full detail. The "return historical rows" surface this section called out as +worth gating first (Pull-style RPCs) remains open for `ConfigServe`/`ConfigSource` and any future +such surface — only the telemetry stream is closed by this note. + ## 7. Sequencing sketch Deliberately not a task plan — per-phase plans follow, one at a time. diff --git a/docs/plans/2026-07-22-per-cluster-mesh-program.md b/docs/plans/2026-07-22-per-cluster-mesh-program.md index 301c5f40..b1beff58 100644 --- a/docs/plans/2026-07-22-per-cluster-mesh-program.md +++ b/docs/plans/2026-07-22-per-cluster-mesh-program.md @@ -202,16 +202,66 @@ Phase 4 as code-complete, not verified, until this lands.** **no ConfigDb connection string configured at all**; grep-level proof no driver-branch service can resolve the ConfigDb context. -### Phase 5 — gRPC stream contract for live telemetry -**Scope:** one server-streaming contract carrying a `oneof` event, **cluster nodes host the gRPC -server, central dials in** (the inverted direction is the load-bearing ScadaBridge finding — -design §2); migrate the seven observability topics (`alerts`, `driver-health`, -`driver-resilience-status`, `fleet-status`, `script-logs`, plus redundancy-state distribution and -deployment-acks if Phase 2 left them on DPS); additive-only field evolution, contract locked by -test; per-panel reconnect story for the AdminUI (design §8 — losing gossip loses free fleet -observability). -**Exit gate:** all AdminUI live panels green against a pair with DPS telemetry topics deleted; -kill-and-reconnect of the central dialer recovers every stream. +### Phase 5 — gRPC stream contract for live telemetry — **code-complete on `feat/mesh-phase5`, live gate pending** +**Scope as shipped (narrowed from the program sketch's "seven observability topics," settled with the +user 2026-07-23 — mirrors how Phases 1 and 3 also honestly scoped down):** one server-streaming gRPC +contract carrying a `oneof` event with **four** kinds, **driver nodes host the gRPC server (dedicated +Kestrel h2c port), central dials in** — the inverted direction is the load-bearing ScadaBridge finding +(design §2), required so telemetry survives the Phase 6 mesh split without shared gossip membership. + +**The four migrated channels:** + +| DPS topic | Domain record | Central sink | +|---|---|---| +| `alerts` | `AlarmTransitionEvent` | `IInProcessBroadcaster` (+ `AlertHub`) | +| `script-logs` | `ScriptLogEntry` | `IInProcessBroadcaster` (+ `ScriptLogHub`) | +| `driver-health` | `DriverHealthChanged` | `IDriverStatusSnapshotStore` | +| `driver-resilience-status` | `DriverResilienceStatusChanged` | `IDriverResilienceStatusStore` | + +**Three deferred, each with a settled reason (NOT migrated — do not read the program sketch's original +"seven" as the delivered scope):** +- `redundancy-state` — bidirectional, built from `Cluster.State`, pair-local control plane driving + ServiceLevel + the Primary gate. Stays on DPS in every mode; under Phase 6 it works in-mesh per pair. + Central's cross-pair *display* of redundancy is a Phase 6 concern, not a Phase 5 panel. +- `fleet-status` — central-internal; `FleetStatusBroadcaster` builds it from the admin node's own + cluster membership events and `Fleet.razor` polls the Config DB, ignoring the feed. Not a + node→central stream at all. Revisit in Phase 6 once central loses site-node gossip visibility. +- `deployment-acks` — already migrated, but onto the Phase 2 `ClusterClient` transport, as a + command-plane reply, not an observability broadcast. + +**Dark switch:** `Telemetry:Mode` (node/serve, default `Dps`) and `TelemetryDial:Mode` (central/dial, +default `Dps`) flip independently to `Grpc`; both code paths are compiled into every binary, so +flipping is a config + restart, not a redeploy. The node **always** hosts the gRPC server whenever +`Telemetry:GrpcListenPort > 0` and **always** emits into its node-local hub + publishes DPS, in both +modes — only central's ingest source (`TelemetryDial:Mode`) actually switches which of the two +always-available sources feeds the AdminUI sinks. The AdminUI panels/components are unchanged; only +the sinks' upstream feed swaps. + +**Auth from day one, fail-closed:** a shared node bearer key (`Telemetry:ApiKey` == +`TelemetryDial:ApiKey`), gated by `TelemetryStreamAuthInterceptor` +(`CryptographicOperations.FixedTimeEquals`, `PermissionDenied` on mismatch, unset key rejects every +call). This supersedes design §6.3's "match ScadaBridge's unauthenticated posture for now" — ScadaBridge +itself has since closed that gap with this identical interceptor pattern (see the supersede note added +to design §6.3, and `docs/Telemetry.md` § Authentication). + +**Reconnect story:** central's `TelemetryDialSupervisor` runs one reconnecting dialer per driver node, +discovered from `ClusterNode` rows (`Host`+`GrpcPort`, refreshed every +`TelemetryDial:ContactRefreshSeconds`); immediate first retry then ~5s fixed backoff, generation-stamped +so a superseded stream's late error/event is ignored; the dialer never permanently gives up +(observability, not data plane). The node-local hub replays last-value snapshots for +`driver-health`/`driver-resilience-status` on every (re)subscribe so those stores re-prime immediately; +`alerts`/`script-logs` are append logs and tolerate the gap. The `/alerts`/`/script-log` pill reflects +aggregate stream health (live when ≥1 node stream is up), matching today's DPS pill semantics. + +**Docs shipped:** `docs/Telemetry.md` (new, canonical guide), `docs/Configuration.md` § +`Telemetry`/`TelemetryDial`, `docs/Redundancy.md` cross-reference, this section, and the design §6.3 +supersede note. + +**Exit gate (not yet run):** all AdminUI live panels green against the docker-dev rig with +`Telemetry:Mode=Grpc`/`TelemetryDial:Mode=Grpc` and the four DPS telemetry bridges NOT spawned; +kill-and-reconnect of a site node's dialer recovers every stream (driver-health/resilience re-prime +from the node hub's snapshot replay). Treat Phase 5 as **code-complete, not verified**, until this +lands — see `docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md` Task 12. ### Phase 6 — Mesh partition + co-location topology **Scope:** per-cluster seed nodes — ~~adopt ScadaBridge's self-first ordering and RETIRE the @@ -267,6 +317,6 @@ resource sizing on the site VMs should be checked once both products run the ful | 2 ClusterClient transport | **DONE 2026-07-22** (live gate PASSED, shipped dark) — shipped dark (`MeshTransport:Mode=Dps` default), both comm actors registered in both modes. Corrections: `SendToAll` not `Send`, one fleet-wide client not one per cluster, `/user/node-communication` not `/user/cluster-communication`. The gate's "Ask timing out" item does not exist to test — no cross-boundary Ask in this phase; see the phase section. | | 3 fetch-and-cache | **DONE 2026-07-23**, merged `d01b0695`, live gate PASSED, pushed to origin (scadaproj umbrella index updated + pushed `b5e7bc8`). gRPC fetch RPC + shared node key; dark switch `ConfigSource:Mode` (Direct default). Rig flip: `OTOPCUA_CONFIG_MODE=FetchAndCache` on the site nodes. See `2026-07-22-mesh-phase3-config-fetch-and-cache.md`. | | 4 cut driver ConfigDb | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase4` (Tasks 0–8 + 1b + 10–11; Task 9 table-drop deferred). ConfigDb admin-only; driver-only ⇒ FetchAndCache (validator); `DbHealthProbeActor` not spawned driver-only (client-visible ServiceLevel 240/250 with central SQL down — proven live); alarm condition state in replicated LocalDb `alarm_condition_state`; central persists acks; `OpcUaPublish` guard split fixes the driver-only address-space wipe; driver-only LDAP maps from appsettings only (6th consumer found mid-phase). Gate: deploy sealed green w/ 4 DB-less site nodes acking, ServiceLevel held 240 w/ SQL down, restart booted last-known-good from the LocalDb pointer. See `2026-07-23-mesh-phase4-cut-driver-configdb.md` + `2026-07-23-mesh-phase4-live-gate.md`. | -| 5 gRPC telemetry | not started | +| 5 gRPC telemetry | **code-complete (live gate pending)** on `feat/mesh-phase5`. 4-channel scope (`alerts`/`script-logs`/`driver-health`/`driver-resilience-status`), 3 deferred with rationale (`redundancy-state`, `fleet-status`, `deployment-acks`); dark switch `Telemetry:Mode`/`TelemetryDial:Mode` (Dps default); auth-from-day-one fail-closed bearer key, superseding design §6.3. See `docs/Telemetry.md` + `2026-07-23-mesh-phase5-grpc-telemetry-stream.md`. | | 6 mesh partition + co-location | not started | | 7 drill + live gates | not started |