Merge feat/mesh-phase5: per-cluster mesh Phase 5 — gRPC live-telemetry stream
Node-hosted telemetry gRPC server + central dial supervisor replace the DPS fan-out of alerts/script-logs/driver-health/driver-resilience-status, behind the Telemetry:Mode/TelemetryDial:Mode dark switch (Dps default), fail-closed bearer auth. 3 channels deferred with rationale (redundancy-state/fleet-status/deployment-acks). Live gate PASSED (12-stream mesh, pill live, kill-and-reconnect recovered). Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
@@ -315,6 +315,57 @@ Flip only the site nodes with `OTOPCUA_CONFIG_MODE=FetchAndCache` at `docker com
|
||||
`docs/Configuration.md` §`ConfigSource` / `ConfigServe`, `docs/Redundancy.md` §"Command transport", and
|
||||
`docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md`.
|
||||
|
||||
## Live telemetry transport (`Telemetry` / `TelemetryDial`)
|
||||
|
||||
Per-cluster mesh **Phase 5** (code-complete on `feat/mesh-phase5`, live gate pending) added a second
|
||||
transport for the four live-observability channels the AdminUI's `/alerts`, `/script-log`, and
|
||||
`/hosts` panels feed on, selected by `Telemetry:Mode` (node/serve side) and `TelemetryDial:Mode`
|
||||
(central/dial side): `Dps` — the four existing DPS SignalR bridges, **the default**, today's
|
||||
behaviour — or `Grpc`, one gRPC server-streaming contract carrying all four channels as `oneof`
|
||||
event kinds. **The direction is inverted from `ConfigSource`/`MeshTransport`: the driver node hosts
|
||||
the gRPC server (dedicated Kestrel h2c port), and central is the client that dials in** — mirroring
|
||||
ScadaBridge's `SiteStreamService` shape, and required so telemetry survives once the meshes split
|
||||
(Phase 6) and central no longer shares a gossip ring with a site node. Things worth knowing before
|
||||
touching it:
|
||||
|
||||
- **Migrated (4 channels):** `alerts` (`AlarmTransitionEvent`), `script-logs` (`ScriptLogEntry`),
|
||||
`driver-health` (`DriverHealthChanged`), `driver-resilience-status`
|
||||
(`DriverResilienceStatusChanged`) — feeding the exact same central sinks
|
||||
(`IInProcessBroadcaster<...>`, `IDriverStatusSnapshotStore`, `IDriverResilienceStatusStore`) in
|
||||
either mode.
|
||||
- **Deferred, with reasons (do not read "seven observability topics" from the program sketch as the
|
||||
delivered scope):** `redundancy-state` stays on DPS in every mode (pair-local, built from
|
||||
`Cluster.State`, drives ServiceLevel + the Primary gate — genuinely mesh-bound by design, not an
|
||||
observability panel); `fleet-status` is central-internal (`FleetStatusBroadcaster` builds it from
|
||||
the admin node's own membership events, `Fleet.razor` polls the Config DB and ignores the feed —
|
||||
never a node→central stream); `deployment-acks` already rides the Phase 2 `ClusterClient`
|
||||
transport as a command-plane reply, not telemetry.
|
||||
- **The node ALWAYS hosts and ALWAYS double-publishes, in both modes.** Every publish seam emits
|
||||
into a node-local `ITelemetryLocalHub` **and** still publishes DPS, unconditionally; the gRPC
|
||||
server binds whenever `Telemetry:GrpcListenPort > 0`, independent of `Telemetry:Mode`. Only
|
||||
`TelemetryDial:Mode` on central actually switches anything: which of the two always-available
|
||||
sources (DPS bridges vs. the `TelemetryDialSupervisor` dialer) feeds the sinks. This is what
|
||||
makes the flip a config change, not a coordinated redeploy — same dark-switch discipline as
|
||||
[MeshTransport](#mesh-command-transport-meshtransport) and [ConfigSource](#config-source-configsource--configserve).
|
||||
- **Authenticated from day one, fail-closed** — a shared node bearer key (`Telemetry:ApiKey` ==
|
||||
`TelemetryDial:ApiKey`), gated by `TelemetryStreamAuthInterceptor`
|
||||
(`CryptographicOperations.FixedTimeEquals`, `PermissionDenied` on mismatch, an unset key rejects
|
||||
every call). This **supersedes** design doc §6.3's "match ScadaBridge's unauthenticated posture
|
||||
for now" — ScadaBridge itself has since closed that gap with this identical interceptor pattern.
|
||||
- **Reconnect story:** central runs one reconnecting dialer per driver node (discovered from
|
||||
`ClusterNode.Host`/`GrpcPort`, refreshed every `TelemetryDial:ContactRefreshSeconds`), immediate
|
||||
first retry then ~5s fixed backoff, generation-stamped so a superseded stream's late error/event
|
||||
is ignored, and it never permanently gives up (observability, not data plane). The node-local hub
|
||||
replays last-value snapshots for `driver-health`/`driver-resilience-status` on every (re)subscribe
|
||||
so those stores re-prime immediately after a reconnect; `alerts`/`script-logs` are append logs and
|
||||
tolerate the gap. The `/alerts`/`/script-log` pill reflects aggregate stream health (live when ≥1
|
||||
node stream is up) — the same semantics the DPS pill has always had.
|
||||
|
||||
See `docs/Telemetry.md` for the full architecture, `docs/Configuration.md`
|
||||
§`Telemetry`/`TelemetryDial` for the config keys, and
|
||||
`docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md` for the implementation plan and live
|
||||
gate procedure.
|
||||
|
||||
## LocalDb pair-local store (Phases 1 + 2)
|
||||
|
||||
Every **driver-role** node keeps a consolidated `ZB.MOM.WW.LocalDb` SQLite database (the retired
|
||||
|
||||
@@ -76,6 +76,7 @@
|
||||
<Project Path="tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/ZB.MOM.WW.OtOpcUa.AdminUI.Tests.csproj" />
|
||||
<Project Path="tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.csproj" />
|
||||
<Project Path="tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests.csproj" />
|
||||
<Project Path="tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/ZB.MOM.WW.OtOpcUa.Host.Tests.csproj" />
|
||||
<Project Path="tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests.csproj" />
|
||||
<Project Path="tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests.csproj" />
|
||||
<Project Path="tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/ZB.MOM.WW.OtOpcUa.Runtime.Tests.csproj" />
|
||||
|
||||
@@ -40,6 +40,23 @@
|
||||
# + seeded clusters across `docker compose up` cycles; without it a recreate
|
||||
# silently drops the OtOpcUa database.
|
||||
#
|
||||
# Per-cluster mesh Phase 5 (gRPC live-telemetry stream): every driver-role node (all six,
|
||||
# including the fused central pair) carries Telemetry__GrpcListenPort=4056 (dedicated h2c
|
||||
# listener, verified free against every other per-node port: Akka 4053, ConfigServe 4055 on
|
||||
# central, LocalDb sync 9001 on site-a, OPC UA 4840, HTTP 9000) + Telemetry__ApiKey
|
||||
# (committed dev key, matches the SQL password / secrets KEK / ConfigServe key exception).
|
||||
# central-1/central-2 also carry the matching TelemetryDial__ApiKey. Both Telemetry__Mode and
|
||||
# TelemetryDial__Mode are left UNSET here, so every node defaults to "Dps" (dark — telemetry
|
||||
# keeps riding the existing DistributedPubSub topic). The ClusterNode.GrpcPort column (seeded
|
||||
# by docker-dev/seed/seed-clusters.sql) points central's dial supervisor at each node's :4056.
|
||||
# Unlike MeshTransport/ConfigSource, no `${VAR:-Dps}` interpolation is wired for either Mode key
|
||||
# (a bare shell export does nothing here — these keys aren't referenced anywhere in this file, so
|
||||
# there is nothing for it to substitute into). To run the Phase 5 live gate, manually add
|
||||
# `Telemetry__Mode: "Grpc"` to the `environment:` block of all six driver nodes (central-1,
|
||||
# central-2, site-a-1, site-a-2, site-b-1, site-b-2) and `TelemetryDial__Mode: "Grpc"` to
|
||||
# central-1's and central-2's blocks — or supply both via a docker-compose.override.yml carrying
|
||||
# the same keys — then `docker compose -f docker-dev/docker-compose.yml up -d` to recreate.
|
||||
#
|
||||
# Usage:
|
||||
# docker compose -f docker-dev/docker-compose.yml up -d --build
|
||||
# open http://localhost:9200 # central Blazor admin UI
|
||||
@@ -186,6 +203,16 @@ services:
|
||||
ConfigServe__GrpcListenPort: "4055"
|
||||
ConfigServe__ApiKey: "configserve-docker-dev-key"
|
||||
ConfigSource__Mode: "Direct"
|
||||
# Per-cluster mesh Phase 5 (gRPC live-telemetry stream). DARK SWITCH — Telemetry:Mode is left
|
||||
# unset here (defaults to "Dps"), so the node keeps publishing telemetry on the existing
|
||||
# DistributedPubSub topic; the dedicated gRPC listener below binds regardless (the node "always
|
||||
# hosts" per docs/Telemetry.md) so flipping the mode later is a config change, not a redeploy.
|
||||
# Port 4056 checked free against every other port this node binds: Akka 4053, ConfigServe 4055,
|
||||
# OPC UA 4840 (container-internal), HTTP 9000. TelemetryDial__ApiKey (below) is central's
|
||||
# dial-side key and must equal every node's Telemetry:ApiKey (fail-closed interceptor).
|
||||
Telemetry__GrpcListenPort: "4056"
|
||||
Telemetry__ApiKey: "telemetry-docker-dev-key"
|
||||
TelemetryDial__ApiKey: "telemetry-docker-dev-key"
|
||||
# Both redundancy peers are seeds (#459 / ScadaBridge parity) so a restarted node can
|
||||
# re-join the mesh via EITHER peer, not only central-1.
|
||||
#
|
||||
@@ -282,6 +309,11 @@ services:
|
||||
ConfigServe__GrpcListenPort: "4055"
|
||||
ConfigServe__ApiKey: "configserve-docker-dev-key"
|
||||
ConfigSource__Mode: "Direct"
|
||||
# Per-cluster mesh Phase 5 (gRPC live-telemetry stream) — see central-1. DARK SWITCH: Mode
|
||||
# stays unset (⇒ Dps); the dedicated :4056 listener binds regardless.
|
||||
Telemetry__GrpcListenPort: "4056"
|
||||
Telemetry__ApiKey: "telemetry-docker-dev-key"
|
||||
TelemetryDial__ApiKey: "telemetry-docker-dev-key"
|
||||
# Both redundancy peers are seeds (#459 / ScadaBridge parity) — see central-1. SELF FIRST:
|
||||
# central-2 lists itself as seed-nodes[0], which is what lets it cold-start while central-1
|
||||
# is down (it previously listed central-1 first and simply never came Up in that case).
|
||||
@@ -374,6 +406,13 @@ services:
|
||||
ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
|
||||
ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
|
||||
ConfigSource__ApiKey: "configserve-docker-dev-key"
|
||||
# Per-cluster mesh Phase 5 (gRPC live-telemetry stream) — see central-1/central-2. DARK
|
||||
# SWITCH: Telemetry:Mode stays unset (⇒ Dps); the dedicated :4056 listener binds regardless
|
||||
# (the node "always hosts" per docs/Telemetry.md). Port checked free against this node's
|
||||
# other ports: Akka 4053, OPC UA 4840, HTTP 9000, LocalDb sync 9001 (below). ApiKey must
|
||||
# equal central's TelemetryDial:ApiKey (fail-closed interceptor).
|
||||
Telemetry__GrpcListenPort: "4056"
|
||||
Telemetry__ApiKey: "telemetry-docker-dev-key"
|
||||
# Site nodes are deliberately NOT seeds — they join the central pair's mesh. The self-first
|
||||
# seed rule is conditional for exactly this reason (it binds only when a node's own address
|
||||
# is in its own seed list), so these configs are exempt rather than broken.
|
||||
@@ -462,6 +501,10 @@ services:
|
||||
ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
|
||||
ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
|
||||
ConfigSource__ApiKey: "configserve-docker-dev-key"
|
||||
# Per-cluster mesh Phase 5 (gRPC live-telemetry stream) — see central-1/site-a-1. DARK
|
||||
# SWITCH: Telemetry:Mode stays unset (⇒ Dps); the dedicated :4056 listener binds regardless.
|
||||
Telemetry__GrpcListenPort: "4056"
|
||||
Telemetry__ApiKey: "telemetry-docker-dev-key"
|
||||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
||||
Cluster__Roles__0: "driver"
|
||||
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
|
||||
@@ -539,6 +582,10 @@ services:
|
||||
ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
|
||||
ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
|
||||
ConfigSource__ApiKey: "configserve-docker-dev-key"
|
||||
# Per-cluster mesh Phase 5 (gRPC live-telemetry stream) — see central-1/site-a-1. DARK
|
||||
# SWITCH: Telemetry:Mode stays unset (⇒ Dps); the dedicated :4056 listener binds regardless.
|
||||
Telemetry__GrpcListenPort: "4056"
|
||||
Telemetry__ApiKey: "telemetry-docker-dev-key"
|
||||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
||||
Cluster__Roles__0: "driver"
|
||||
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
|
||||
@@ -599,6 +646,10 @@ services:
|
||||
ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
|
||||
ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
|
||||
ConfigSource__ApiKey: "configserve-docker-dev-key"
|
||||
# Per-cluster mesh Phase 5 (gRPC live-telemetry stream) — see central-1/site-a-1. DARK
|
||||
# SWITCH: Telemetry:Mode stays unset (⇒ Dps); the dedicated :4056 listener binds regardless.
|
||||
Telemetry__GrpcListenPort: "4056"
|
||||
Telemetry__ApiKey: "telemetry-docker-dev-key"
|
||||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
||||
Cluster__Roles__0: "driver"
|
||||
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
|
||||
|
||||
@@ -16,6 +16,12 @@
|
||||
-- Host = Compose service name (resolves inside the otopcua-dev network).
|
||||
-- OpcUaPort stays at the container-internal 4840; the host-side port mapping is in
|
||||
-- docker-compose.yml ports: blocks and is irrelevant to ClusterNode rows.
|
||||
--
|
||||
-- GrpcPort (per-cluster mesh Phase 5): every node's dedicated live-telemetry gRPC listener,
|
||||
-- matching Telemetry__GrpcListenPort=4056 set on all six host nodes in docker-compose.yml.
|
||||
-- TelemetryNodeSource (central) reads Host + GrpcPort to build its dial-target set
|
||||
-- (http://{Host}:{GrpcPort}); a NULL GrpcPort is skipped with a Warning. Mirrors how AkkaPort
|
||||
-- was seeded for Phase 1's ClusterNodeAddressReconcilerActor.
|
||||
|
||||
SET NOCOUNT ON;
|
||||
SET XACT_ABORT ON;
|
||||
@@ -64,13 +70,13 @@ IF NOT EXISTS (SELECT 1 FROM dbo.ServerCluster WHERE ClusterId = 'SITE-B')
|
||||
|
||||
IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'central-1:4053')
|
||||
INSERT INTO dbo.ClusterNode
|
||||
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
|
||||
VALUES ('central-1:4053', 'MAIN', 'central-1', 4840, 8081, 4053, 'urn:OtOpcUa:central-1', 200, 1, 'docker-dev-seed');
|
||||
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, GrpcPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
|
||||
VALUES ('central-1:4053', 'MAIN', 'central-1', 4840, 8081, 4053, 4056, 'urn:OtOpcUa:central-1', 200, 1, 'docker-dev-seed');
|
||||
|
||||
IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'central-2:4053')
|
||||
INSERT INTO dbo.ClusterNode
|
||||
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
|
||||
VALUES ('central-2:4053', 'MAIN', 'central-2', 4840, 8081, 4053, 'urn:OtOpcUa:central-2', 150, 1, 'docker-dev-seed');
|
||||
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, GrpcPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
|
||||
VALUES ('central-2:4053', 'MAIN', 'central-2', 4840, 8081, 4053, 4056, 'urn:OtOpcUa:central-2', 150, 1, 'docker-dev-seed');
|
||||
|
||||
------------------------------------------------------------------------------
|
||||
-- ClusterNode — site A
|
||||
@@ -78,13 +84,13 @@ IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'central-2:4053')
|
||||
|
||||
IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-a-1:4053')
|
||||
INSERT INTO dbo.ClusterNode
|
||||
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
|
||||
VALUES ('site-a-1:4053', 'SITE-A', 'site-a-1', 4840, 8081, 4053, 'urn:OtOpcUa:site-a-1', 200, 1, 'docker-dev-seed');
|
||||
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, GrpcPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
|
||||
VALUES ('site-a-1:4053', 'SITE-A', 'site-a-1', 4840, 8081, 4053, 4056, 'urn:OtOpcUa:site-a-1', 200, 1, 'docker-dev-seed');
|
||||
|
||||
IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-a-2:4053')
|
||||
INSERT INTO dbo.ClusterNode
|
||||
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
|
||||
VALUES ('site-a-2:4053', 'SITE-A', 'site-a-2', 4840, 8081, 4053, 'urn:OtOpcUa:site-a-2', 150, 1, 'docker-dev-seed');
|
||||
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, GrpcPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
|
||||
VALUES ('site-a-2:4053', 'SITE-A', 'site-a-2', 4840, 8081, 4053, 4056, 'urn:OtOpcUa:site-a-2', 150, 1, 'docker-dev-seed');
|
||||
|
||||
------------------------------------------------------------------------------
|
||||
-- ClusterNode — site B
|
||||
@@ -92,13 +98,13 @@ IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-a-2:4053')
|
||||
|
||||
IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-b-1:4053')
|
||||
INSERT INTO dbo.ClusterNode
|
||||
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
|
||||
VALUES ('site-b-1:4053', 'SITE-B', 'site-b-1', 4840, 8081, 4053, 'urn:OtOpcUa:site-b-1', 200, 1, 'docker-dev-seed');
|
||||
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, GrpcPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
|
||||
VALUES ('site-b-1:4053', 'SITE-B', 'site-b-1', 4840, 8081, 4053, 4056, 'urn:OtOpcUa:site-b-1', 200, 1, 'docker-dev-seed');
|
||||
|
||||
IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-b-2:4053')
|
||||
INSERT INTO dbo.ClusterNode
|
||||
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
|
||||
VALUES ('site-b-2:4053', 'SITE-B', 'site-b-2', 4840, 8081, 4053, 'urn:OtOpcUa:site-b-2', 150, 1, 'docker-dev-seed');
|
||||
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, GrpcPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
|
||||
VALUES ('site-b-2:4053', 'SITE-B', 'site-b-2', 4840, 8081, 4053, 4056, 'urn:OtOpcUa:site-b-2', 150, 1, 'docker-dev-seed');
|
||||
|
||||
------------------------------------------------------------------------------
|
||||
-- Galaxy MxAccess gateway — INTENTIONALLY NOT SEEDED (retired 2026-06-15)
|
||||
|
||||
@@ -177,6 +177,60 @@ PersistNodeAck`) and LDAP group→role DB grants (a driver-only node maps LDAP g
|
||||
the `Security:Ldap:GroupToRole` appsettings baseline only — see `docs/Redundancy.md` and
|
||||
`CLAUDE.md` §"Phase 4" for the full picture, including the client-visible ServiceLevel change).
|
||||
|
||||
### `Telemetry` / `TelemetryDial` (live telemetry transport)
|
||||
|
||||
- **Purpose:** [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) **Phase 5**. Selects how
|
||||
the four live-observability channels (`alerts`, `script-logs`, `driver-health`,
|
||||
`driver-resilience-status`) reach central: `Dps` (DistributedPubSub — the four existing SignalR
|
||||
bridges, today's behaviour) or `Grpc` (a driver node hosts one gRPC server-streaming contract over a
|
||||
dedicated h2c port; central dials in and feeds the same in-process sinks). `Telemetry` is the
|
||||
**node/serve** side; `TelemetryDial` is the **central/dial** side.
|
||||
- **Options classes:** `TelemetryOptions` (`SectionName = "Telemetry"`) + `TelemetryDialOptions`
|
||||
(`SectionName = "TelemetryDial"`), both in `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`. Bound by
|
||||
`AddOtOpcUaCluster(config)`; their validators run at `ValidateOnStart`.
|
||||
|
||||
**`Telemetry` (node-side, serve):**
|
||||
|
||||
| Key | Type | Default | Meaning |
|
||||
|---|---|---|---|
|
||||
| `Mode` | string | `Dps` | `Dps` = no change, node just keeps publishing DPS as it always has. `Grpc` = central-ingest hint (see `TelemetryDial:Mode` below — this key does not gate anything on the node itself). Any other value **refuses to start**. |
|
||||
| `GrpcListenPort` | int | `0` | Dedicated **h2c-only** Kestrel port for the telemetry stream service. `0` = disabled (nothing bound). The node hosts the server whenever this is `> 0`, **regardless of `Mode`**. |
|
||||
| `ApiKey` | string | `""` | Shared bearer key the serve-side interceptor checks (constant-time, **fail-closed**: an unset key rejects every call). **Supply via env `Telemetry__ApiKey` — never commit.** |
|
||||
|
||||
**`TelemetryDial` (central-side, dial):**
|
||||
|
||||
| Key | Type | Default | Meaning |
|
||||
|---|---|---|---|
|
||||
| `Mode` | string | `Dps` | `Dps` = the four DPS SignalR bridges (alert, script-log, driver-status, resilience) subscribe as today. `Grpc` = those four bridges are **not spawned**; the `TelemetryDialSupervisor` actor dials every driver node instead and feeds the same sinks. Any other value **refuses to start**. |
|
||||
| `ApiKey` | string | `""` | Shared bearer key; must equal the nodes' `Telemetry:ApiKey`. **Supply via env `TelemetryDial__ApiKey` — never commit.** Required under `Grpc`. |
|
||||
| `ContactRefreshSeconds` | int | `60` | How often central rebuilds its dial-target set from the `ClusterNode` rows (`Host` + `GrpcPort`). |
|
||||
| `CallTimeoutSeconds` | int | `30` | Per-call deadline for the streaming RPC's keepalive/health bookkeeping. |
|
||||
|
||||
**It is a dark switch; the node always hosts.** Unlike `ConfigServe`/`MeshTransport`, where the serve
|
||||
side is gated on a role, the telemetry server binds whenever `Telemetry:GrpcListenPort > 0` — hosting
|
||||
is unconditional on a driver-role node, in both modes, because the node also always emits into its
|
||||
local hub and always publishes DPS. Only `TelemetryDial:Mode` on central actually changes behaviour:
|
||||
which of the two already-always-available sources (DPS bridges vs. the gRPC dial supervisor) feeds the
|
||||
AdminUI sinks. Flipping either side is a config change plus restart, not a redeploy — the same
|
||||
discipline as `MeshTransport:Mode` and `ConfigSource:Mode`.
|
||||
|
||||
**A driver-role node with `Telemetry:Mode=Grpc` must set `GrpcListenPort > 0`** (nothing to serve
|
||||
otherwise) — enforced by `TelemetryOptionsValidator`. **`Grpc` mode on either side requires a
|
||||
non-empty `ApiKey`** — fail-closed, same as `ConfigServe`/`LocalDb:Replication`.
|
||||
|
||||
**Three DPS observability topics stay out of scope for this transport, on purpose:**
|
||||
`redundancy-state` (pair-local, built from `Cluster.State`, stays on DPS in both modes — see
|
||||
[`Redundancy.md`](Redundancy.md#command-transport-centralnode)), `fleet-status` (central-internal;
|
||||
`Fleet.razor` polls the Config DB, not any stream), and `deployment-acks` (already rides the Phase 2
|
||||
`ClusterClient` transport as a command-plane reply). See
|
||||
[`docs/Telemetry.md`](Telemetry.md) for the full architecture, the reconnect/generation-stamped
|
||||
dialer story, and the rationale for the node-hosts/central-dials inversion.
|
||||
|
||||
On the docker-dev rig every driver-role node carries `Telemetry:GrpcListenPort` (`4056`) + a
|
||||
committed dev key, and central carries the matching `TelemetryDial:ApiKey`; both `Mode` keys stay
|
||||
unset (⇒ `Dps`) until the rig is flipped with `Telemetry__Mode=Grpc` / `TelemetryDial__Mode=Grpc` at
|
||||
`docker compose up`.
|
||||
|
||||
### `ConnectionStrings` → `ConfigDb`
|
||||
|
||||
- **Purpose:** the central Config DB connection string. **Required on admin-role nodes only** — `Program.cs` calls `AddOtOpcUaConfigDb` when `hasAdmin` (per-cluster mesh Phase 4). A driver-only node holds no `ConfigDb` connection string; it fetches its configuration via `ConfigSource:Mode=FetchAndCache` instead (see above). A fused `admin,driver` node still requires it.
|
||||
@@ -283,6 +337,7 @@ These are consumed by the driver **integration-test fixtures** (under `tests/Dri
|
||||
|
||||
- [`security.md`](security.md) — transport security, OPC UA authentication, LDAP (`Security:Ldap`), data-plane ACLs, control-plane roles.
|
||||
- [`Redundancy.md`](Redundancy.md) — the `Cluster` section in the context of warm/hot redundancy, ServiceLevel, peer discovery.
|
||||
- [`Telemetry.md`](Telemetry.md) — the `Telemetry`/`TelemetryDial` gRPC live-telemetry stream: direction, dark switch, the four migrated channels + three deferrals, auth, reconnect story.
|
||||
- [`ServiceHosting.md`](ServiceHosting.md) — role-based host wiring and `OTOPCUA_ROLES`.
|
||||
- [`docs/drivers/Galaxy.md`](drivers/Galaxy.md) — Galaxy/MxAccess driver overview.
|
||||
- [`docs/v2/config-db-schema.md`](v2/config-db-schema.md) — the full Config DB schema.
|
||||
|
||||
@@ -465,6 +465,15 @@ are untouched, and `redundancy-state` itself stays on DPS (it is bidirectional a
|
||||
`Cluster.State`, which makes it genuinely mesh-bound — the hardest remaining dependency before the
|
||||
meshes can split).
|
||||
|
||||
**Observability telemetry took the opposite path (Phase 5).** The four live-panel channels
|
||||
(`alerts`, `script-logs`, `driver-health`, `driver-resilience-status`) moved off DPS onto a
|
||||
dedicated gRPC server-streaming contract — each driver node hosts, central dials in — selected by
|
||||
`Telemetry:Mode`/`TelemetryDial:Mode` (default `Dps`). `redundancy-state` is deliberately **not**
|
||||
part of that migration: it stays on DPS in every mode because it is pair-local and Cluster.State-built,
|
||||
not a fleet-wide observability broadcast. See [`docs/Telemetry.md`](Telemetry.md) for the full
|
||||
architecture and [`Configuration.md` § `Telemetry`/`TelemetryDial`](Configuration.md#telemetry--telemetrydial-live-telemetry-transport)
|
||||
for the config keys.
|
||||
|
||||
**Config bytes travel out-of-band (Phase 3).** Commands stay on one of the two transports above, but
|
||||
the *configuration artifact itself* does not ride any Akka message — `DispatchDeployment` is
|
||||
payload-free (`DeploymentId` + `RevisionHash`). Under `ConfigSource:Mode = FetchAndCache` (see
|
||||
|
||||
@@ -0,0 +1,190 @@
|
||||
# Live Telemetry Transport (v2)
|
||||
|
||||
## Overview
|
||||
|
||||
The AdminUI's live observability panels (`/alerts`, `/script-log`, `/hosts` driver table, driver
|
||||
resilience status) are fed by four node→central event channels. Historically all four rode
|
||||
DistributedPubSub (DPS) on the shared Akka mesh — the same gossip ring that carries redundancy
|
||||
state and the command-plane topics. Per-cluster mesh **Phase 5** adds a second transport for these
|
||||
four channels only: one gRPC server-streaming contract, selected per node/central pair by
|
||||
`Telemetry:Mode` / `TelemetryDial:Mode` (`Dps` default | `Grpc`).
|
||||
|
||||
This is the same motivation as [Phase 2's `MeshTransport`](Configuration.md#meshtransport-centralnode-command-transport)
|
||||
and [Phase 3's `ConfigSource`/`ConfigServe`](Configuration.md#configsource--configserve-config-fetch-and-cache):
|
||||
DPS only works when central and the node share a gossip ring, and [Phase 6](plans/2026-07-21-per-cluster-mesh-design.md)
|
||||
splits the fleet into one Akka mesh per application `Cluster`. Once that split lands, a driver node
|
||||
in a site's mesh is no longer a cluster member of central's mesh, so DPS can no longer reach it —
|
||||
telemetry needs its own out-of-band transport, same as commands (Phase 2) and config bytes (Phase 3).
|
||||
|
||||
## Direction: the node hosts, central dials
|
||||
|
||||
**Load-bearing and easy to get backwards:** each **driver** node hosts the telemetry gRPC server
|
||||
(a dedicated Kestrel h2c listener); **central (admin role) is the client** and dials in. This
|
||||
mirrors the inversion ScadaBridge already uses for the same problem (its `SiteStreamService`).
|
||||
|
||||
```
|
||||
driver node central (admin)
|
||||
┌─────────────────────┐ ┌──────────────────────────┐
|
||||
│ 4 publish seams │ │ TelemetryDialSupervisor │
|
||||
│ -> node-local hub │ Subscribe │ - discovers ClusterNode │
|
||||
│ -> DPS (unchanged) │◄─────────────│ rows (Host+GrpcPort) │
|
||||
│ │ (gRPC │ - one reconnecting │
|
||||
│ TelemetryStreamGrpc- │ stream) │ dialer per node │
|
||||
│ Service (h2c server) │─────────────►│ - feeds the SAME sinks │
|
||||
│ Telemetry:GrpcListen- │ TelemetryEvent │ the DPS bridges feed │
|
||||
│ Port │ (oneof, 4 kinds)│ today │
|
||||
└─────────────────────┘ └──────────────────────────┘
|
||||
```
|
||||
|
||||
A fused `admin,driver` node both hosts (as driver) and dials (as admin) — it dials itself plus its
|
||||
redundant pair peer, the same way central dials site nodes.
|
||||
|
||||
Central discovers driver-node telemetry endpoints from the `ClusterNode` table (`Host` +
|
||||
`GrpcPort`, the latter added in Phase 1 for exactly this purpose) — the same DB-sourced discovery
|
||||
Phase 1's ack set and Phase 2's `ClusterClient` contact set already use. No shared gossip
|
||||
membership is required to find or dial a node.
|
||||
|
||||
## The dark switch
|
||||
|
||||
`Telemetry:Mode` (node/serve side) and `TelemetryDial:Mode` (central/dial side) each default to
|
||||
`Dps` and can be independently set to `Grpc`. Both code paths are compiled into every binary —
|
||||
flipping either flag is an appsettings/env change plus a restart, **not a redeploy or rebuild** —
|
||||
the same discipline as `MeshTransport:Mode` and `ConfigSource:Mode`.
|
||||
|
||||
What actually changes per mode, precisely:
|
||||
|
||||
- **The node ALWAYS does both things, regardless of `Telemetry:Mode`.** Every one of the four
|
||||
publish seams emits into the node-local `ITelemetryLocalHub` **and** publishes to DPS,
|
||||
unconditionally. The node additionally **always** hosts the gRPC server whenever
|
||||
`Telemetry:GrpcListenPort > 0` — hosting is not mode-gated at all. `Telemetry:Mode` only affects
|
||||
central's own bookkeeping about which side it expects to be consulted (it is otherwise inert on
|
||||
the node).
|
||||
- **Only central's ingest source switches**, driven by `TelemetryDial:Mode`:
|
||||
- `Dps` (default) — today's four DPS SignalR bridges (alert, script-log, driver-status,
|
||||
resilience) subscribe and feed the AdminUI sinks. Unchanged behavior.
|
||||
- `Grpc` — those four DPS bridges are **not spawned**. Instead the `TelemetryDialSupervisor`
|
||||
actor dials every driver node's gRPC stream and feeds the **identical** in-process sinks
|
||||
(`IInProcessBroadcaster<AlarmTransitionEvent>`, `IInProcessBroadcaster<ScriptLogEntry>`,
|
||||
`IDriverStatusSnapshotStore`, `IDriverResilienceStatusStore`).
|
||||
- **The AdminUI panels themselves are untouched.** `/alerts`, `/script-log`, and the `/hosts`
|
||||
driver table all read from the same sinks in both modes — only the sinks' upstream feed swaps.
|
||||
The `fleet-status` bridge (out of Phase 5 scope — see below) stays on DPS in both modes.
|
||||
|
||||
Because hosting is unconditional and both ingest paths are always compiled in, a fleet can be
|
||||
flipped node-by-node and central-by-central with no coordination window where telemetry is lost —
|
||||
the node is always serving, so central can switch to `Grpc` whenever it likes.
|
||||
|
||||
## The four migrated channels
|
||||
|
||||
| DPS topic (today) | Domain record | Central sink fed |
|
||||
|---|---|---|
|
||||
| `alerts` | `AlarmTransitionEvent` | `IInProcessBroadcaster<AlarmTransitionEvent>` (+ `AlertHub`); `/alerts` page; live/disconnected pill |
|
||||
| `script-logs` | `ScriptLogEntry` | `IInProcessBroadcaster<ScriptLogEntry>` (+ `ScriptLogHub`); `/script-log` page; live pill |
|
||||
| `driver-health` | `DriverHealthChanged` | `IDriverStatusSnapshotStore`; `/hosts` driver table |
|
||||
| `driver-resilience-status` | `DriverResilienceStatusChanged` | `IDriverResilienceStatusStore` |
|
||||
|
||||
These four are carried as `oneof` event kinds on one `TelemetryStreamService.Subscribe` RPC — a
|
||||
single stream per (central, driver-node) pair carries all four, rather than one stream per topic.
|
||||
Proto field evolution is additive-only (never renumber/reuse a tag), locked by a contract test that
|
||||
reflects over the `oneof` cases.
|
||||
|
||||
## The three deferred channels (NOT migrated in Phase 5 — do not read this as "seven done")
|
||||
|
||||
The program sketch originally named seven observability topics for Phase 5. Three were scoped out,
|
||||
each for a distinct, settled reason:
|
||||
|
||||
- **`redundancy-state`** — bidirectional, built directly from `Cluster.State`, and **pair-local**:
|
||||
it drives ServiceLevel and the Primary data-plane gate, consumed in-process by
|
||||
`OpcUaPublishActor`, `ScriptedAlarmHostActor`, `DriverHostActor`, and `HistorianAdapterActor`.
|
||||
It stays on DPS in **both** `MeshTransport` modes today, and it is genuinely mesh-bound by
|
||||
design — under Phase 6 each pair keeps sharing its own small mesh, so DPS keeps working for it
|
||||
in-mesh. Central's *display* of each pair's redundancy state is a Phase 6 cross-mesh concern
|
||||
(a future observability channel, once central no longer shares gossip with any site pair), not a
|
||||
Phase 5 telemetry-panel migration.
|
||||
- **`fleet-status`** — **central-internal**, not a node→central stream at all. The admin singleton
|
||||
`FleetStatusBroadcaster` builds it from the admin node's own cluster membership/reachability/
|
||||
leader events, and `Fleet.razor` **polls the Config DB** and ignores the feed entirely. There is
|
||||
nothing here for a per-node gRPC stream to carry. Revisit in Phase 6, once central genuinely
|
||||
loses gossip visibility of site nodes and needs another way to know a pair's membership.
|
||||
- **`deployment-acks`** — already migrated, but onto a different transport: it rides the Phase 2
|
||||
`ClusterClient` transport (`MeshTransport:Mode=ClusterClient`) as a command-plane reply, not an
|
||||
observability broadcast. It was never a Phase 5 candidate.
|
||||
|
||||
See [`docs/Redundancy.md`](Redundancy.md#command-transport-centralnode) for how `redundancy-state`
|
||||
and the command transports fit together.
|
||||
|
||||
## Authentication — fail-closed from day one
|
||||
|
||||
A shared node bearer key gates the stream: `Telemetry:ApiKey` (node/serve side) must equal
|
||||
`TelemetryDial:ApiKey` (central/dial side). `TelemetryStreamAuthInterceptor` enforces it — path-scoped
|
||||
to `/telemetry.v1.TelemetryStreamService/`, comparing the `Authorization: Bearer` token with
|
||||
`CryptographicOperations.FixedTimeEquals`, and rejecting with `PermissionDenied`. An **unset** key
|
||||
rejects every call rather than allowing an open stream — the same fail-closed posture as
|
||||
`ConfigServeAuthInterceptor` and `LocalDbSyncAuthInterceptor`.
|
||||
|
||||
This **supersedes** [the design doc's §6.3](plans/2026-07-21-per-cluster-mesh-design.md), which had
|
||||
provisionally decided to "match ScadaBridge's unauthenticated posture for now" for inter-cluster
|
||||
transports. ScadaBridge itself has since closed that gap with this identical
|
||||
interceptor pattern, so Phase 5 ships authenticated from the start rather than deferring auth to a
|
||||
later hardening pass. See the design doc's superseded note for the full history.
|
||||
|
||||
## Reconnect and reliability
|
||||
|
||||
Central's `TelemetryDialSupervisor` (an admin-role actor) runs **one reconnecting dialer per
|
||||
driver node**:
|
||||
|
||||
- **Discovery** — enabled, non-maintenance `ClusterNode` rows are read on start and refreshed every
|
||||
`TelemetryDial:ContactRefreshSeconds` (default 60), plus on an admin-side topology change. A row
|
||||
with a null `GrpcPort` is skipped (that node exposes no telemetry endpoint — logged, not an
|
||||
error). Dial targets are added/removed as the row set changes.
|
||||
- **Reconnect backoff** — immediate first retry, then a fixed ~5 second backoff, indefinitely. The
|
||||
dialer never gives up permanently: this is an observability channel, not the data plane, so a
|
||||
persistently-unreachable node just means a persistently-stale panel, not a fleet fault.
|
||||
- **Generation stamping** — each dialer carries a monotonically-increasing generation counter, so
|
||||
a late error or late event from a superseded stream (e.g. one that raced a reconnect) is
|
||||
recognized and ignored rather than corrupting the current stream's state.
|
||||
- **Snapshot replay on reconnect** — the node-local hub keeps a last-value cache for
|
||||
`driver-health` and `driver-resilience-status` (there is nothing meaningful to replay for
|
||||
`alerts`/`script-logs`, which are append logs). On every new `Subscribe` — including a
|
||||
reconnect — the hub drains the cached snapshots to the new stream before live deltas, so
|
||||
`IDriverStatusSnapshotStore` / `IDriverResilienceStatusStore` re-prime immediately rather than
|
||||
sitting stale until the next natural event. `alerts`/`script-logs` simply tolerate the gap — a
|
||||
reconnect loses whatever transitions happened while disconnected, same as a DPS resubscribe would.
|
||||
- **Connection indicator** — the `/alerts` and `/script-log` live/disconnected pill is driven by
|
||||
**aggregate** stream health across all dialers: connected when **at least one** node stream is
|
||||
up, disconnected when all are down. This matches today's DPS `SubscribeAck`/`PostStop` pill
|
||||
semantics — the pill has never meant "every node is live," only "the panel has a source." The
|
||||
two store-backed panels (`driver-health`, `driver-resilience-status`) carry no fleet-wide
|
||||
connection flag today, on gRPC or on DPS; per-row staleness is unchanged.
|
||||
|
||||
## Configuration reference
|
||||
|
||||
See [`docs/Configuration.md` § `Telemetry` / `TelemetryDial`](Configuration.md#telemetry--telemetrydial-live-telemetry-transport)
|
||||
for the full keys table. In brief:
|
||||
|
||||
- **`Telemetry`** (node/serve side): `Mode` (`Dps`|`Grpc`, default `Dps`), `GrpcListenPort` (`0` =
|
||||
disabled — the driver node's dedicated telemetry h2c port), `ApiKey` (the shared node key; supply
|
||||
via `${secret:}`/env, never commit).
|
||||
- **`TelemetryDial`** (central side): `Mode` (`Dps`|`Grpc`), `ApiKey` (must equal the nodes'
|
||||
`Telemetry:ApiKey`), `ContactRefreshSeconds` (default `60`), `CallTimeoutSeconds` (default `30`).
|
||||
|
||||
A driver-role node with `Telemetry:Mode=Grpc` must set `GrpcListenPort > 0` (nothing to serve
|
||||
otherwise), and `Grpc` mode on either side requires a non-empty `ApiKey` — both are enforced by
|
||||
`ValidateOnStart` validators, fail-fast at boot.
|
||||
|
||||
## Relationship to the rest of the mesh program
|
||||
|
||||
- **Same dark-switch discipline as Phases 2 and 3.** `MeshTransport:Mode`, `ConfigSource:Mode`, and
|
||||
`Telemetry:Mode`/`TelemetryDial:Mode` are all independently-flippable, restart-only config
|
||||
changes, and in every case the "new" side is always wired so the flip needs no coordinated
|
||||
redeploy.
|
||||
- **Why the inversion matters for Phase 6.** Every other cross-boundary transport in this program
|
||||
(`MeshTransport`, `ConfigSource`/`ConfigServe`) already has central as the addressable, discoverable
|
||||
side and the node as the caller or the callee-by-address. Telemetry is the one channel that is
|
||||
naturally node-sourced and fleet-wide-fanned, so putting central in the client role — dialing out
|
||||
to each node by its `ClusterNode`-recorded address — is what lets telemetry survive the mesh split
|
||||
cleanly: central never needs cluster membership with a site's mesh to keep watching it.
|
||||
- **Status:** code-complete on `feat/mesh-phase5`; the live gate (flip the docker-dev rig to `Grpc`,
|
||||
confirm all panels stay green with the DPS telemetry bridges unspawned, kill-and-reconnect a site
|
||||
node) has not yet run. See `docs/plans/2026-07-22-per-cluster-mesh-program.md` § Phase 5 and
|
||||
`docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md` for the implementation plan.
|
||||
@@ -285,6 +285,20 @@ trusted network. Recorded as an accepted risk rather than an oversight, with two
|
||||
it is revisited: `LocalDbSyncAuthInterceptor` is already in our tree as a fail-closed template, and
|
||||
the surface most worth gating first is any Pull-style RPC that returns historical rows.
|
||||
|
||||
**SUPERSEDED 2026-07-23.** The Phase 5 telemetry stream (`TelemetryStreamService` — the four
|
||||
node→central observability channels) shipped **authenticated from day one**: a shared node bearer
|
||||
key (`Telemetry:ApiKey` == `TelemetryDial:ApiKey`), enforced by a fail-closed
|
||||
`TelemetryStreamAuthInterceptor` (`CryptographicOperations.FixedTimeEquals`, `PermissionDenied` on
|
||||
mismatch, and — load-bearing — an **unset** key rejects every call rather than allowing an open
|
||||
stream). This is exactly the `LocalDbSyncAuthInterceptor`/`ConfigServeAuthInterceptor` pattern this
|
||||
section flagged as the fail-closed template to reach for, applied immediately rather than deferred.
|
||||
ScadaBridge itself has since closed the same "unauthenticated inter-cluster transport" gap with this
|
||||
identical interceptor pattern, so this decision no longer matches ScadaBridge's current posture
|
||||
either — it matches where ScadaBridge moved to. See [`docs/Telemetry.md`](../Telemetry.md) §
|
||||
Authentication for the full detail. The "return historical rows" surface this section called out as
|
||||
worth gating first (Pull-style RPCs) remains open for `ConfigServe`/`ConfigSource` and any future
|
||||
such surface — only the telemetry stream is closed by this note.
|
||||
|
||||
## 7. Sequencing sketch
|
||||
|
||||
Deliberately not a task plan — per-phase plans follow, one at a time.
|
||||
|
||||
@@ -202,16 +202,66 @@ Phase 4 as code-complete, not verified, until this lands.**
|
||||
**no ConfigDb connection string configured at all**; grep-level proof no driver-branch service can
|
||||
resolve the ConfigDb context.
|
||||
|
||||
### Phase 5 — gRPC stream contract for live telemetry
|
||||
**Scope:** one server-streaming contract carrying a `oneof` event, **cluster nodes host the gRPC
|
||||
server, central dials in** (the inverted direction is the load-bearing ScadaBridge finding —
|
||||
design §2); migrate the seven observability topics (`alerts`, `driver-health`,
|
||||
`driver-resilience-status`, `fleet-status`, `script-logs`, plus redundancy-state distribution and
|
||||
deployment-acks if Phase 2 left them on DPS); additive-only field evolution, contract locked by
|
||||
test; per-panel reconnect story for the AdminUI (design §8 — losing gossip loses free fleet
|
||||
observability).
|
||||
**Exit gate:** all AdminUI live panels green against a pair with DPS telemetry topics deleted;
|
||||
kill-and-reconnect of the central dialer recovers every stream.
|
||||
### Phase 5 — gRPC stream contract for live telemetry — **DONE 2026-07-23 (live gate PASSED)**
|
||||
**Scope as shipped (narrowed from the program sketch's "seven observability topics," settled with the
|
||||
user 2026-07-23 — mirrors how Phases 1 and 3 also honestly scoped down):** one server-streaming gRPC
|
||||
contract carrying a `oneof` event with **four** kinds, **driver nodes host the gRPC server (dedicated
|
||||
Kestrel h2c port), central dials in** — the inverted direction is the load-bearing ScadaBridge finding
|
||||
(design §2), required so telemetry survives the Phase 6 mesh split without shared gossip membership.
|
||||
|
||||
**The four migrated channels:**
|
||||
|
||||
| DPS topic | Domain record | Central sink |
|
||||
|---|---|---|
|
||||
| `alerts` | `AlarmTransitionEvent` | `IInProcessBroadcaster<AlarmTransitionEvent>` (+ `AlertHub`) |
|
||||
| `script-logs` | `ScriptLogEntry` | `IInProcessBroadcaster<ScriptLogEntry>` (+ `ScriptLogHub`) |
|
||||
| `driver-health` | `DriverHealthChanged` | `IDriverStatusSnapshotStore` |
|
||||
| `driver-resilience-status` | `DriverResilienceStatusChanged` | `IDriverResilienceStatusStore` |
|
||||
|
||||
**Three deferred, each with a settled reason (NOT migrated — do not read the program sketch's original
|
||||
"seven" as the delivered scope):**
|
||||
- `redundancy-state` — bidirectional, built from `Cluster.State`, pair-local control plane driving
|
||||
ServiceLevel + the Primary gate. Stays on DPS in every mode; under Phase 6 it works in-mesh per pair.
|
||||
Central's cross-pair *display* of redundancy is a Phase 6 concern, not a Phase 5 panel.
|
||||
- `fleet-status` — central-internal; `FleetStatusBroadcaster` builds it from the admin node's own
|
||||
cluster membership events and `Fleet.razor` polls the Config DB, ignoring the feed. Not a
|
||||
node→central stream at all. Revisit in Phase 6 once central loses site-node gossip visibility.
|
||||
- `deployment-acks` — already migrated, but onto the Phase 2 `ClusterClient` transport, as a
|
||||
command-plane reply, not an observability broadcast.
|
||||
|
||||
**Dark switch:** `Telemetry:Mode` (node/serve, default `Dps`) and `TelemetryDial:Mode` (central/dial,
|
||||
default `Dps`) flip independently to `Grpc`; both code paths are compiled into every binary, so
|
||||
flipping is a config + restart, not a redeploy. The node **always** hosts the gRPC server whenever
|
||||
`Telemetry:GrpcListenPort > 0` and **always** emits into its node-local hub + publishes DPS, in both
|
||||
modes — only central's ingest source (`TelemetryDial:Mode`) actually switches which of the two
|
||||
always-available sources feeds the AdminUI sinks. The AdminUI panels/components are unchanged; only
|
||||
the sinks' upstream feed swaps.
|
||||
|
||||
**Auth from day one, fail-closed:** a shared node bearer key (`Telemetry:ApiKey` ==
|
||||
`TelemetryDial:ApiKey`), gated by `TelemetryStreamAuthInterceptor`
|
||||
(`CryptographicOperations.FixedTimeEquals`, `PermissionDenied` on mismatch, unset key rejects every
|
||||
call). This supersedes design §6.3's "match ScadaBridge's unauthenticated posture for now" — ScadaBridge
|
||||
itself has since closed that gap with this identical interceptor pattern (see the supersede note added
|
||||
to design §6.3, and `docs/Telemetry.md` § Authentication).
|
||||
|
||||
**Reconnect story:** central's `TelemetryDialSupervisor` runs one reconnecting dialer per driver node,
|
||||
discovered from `ClusterNode` rows (`Host`+`GrpcPort`, refreshed every
|
||||
`TelemetryDial:ContactRefreshSeconds`); immediate first retry then ~5s fixed backoff, generation-stamped
|
||||
so a superseded stream's late error/event is ignored; the dialer never permanently gives up
|
||||
(observability, not data plane). The node-local hub replays last-value snapshots for
|
||||
`driver-health`/`driver-resilience-status` on every (re)subscribe so those stores re-prime immediately;
|
||||
`alerts`/`script-logs` are append logs and tolerate the gap. The `/alerts`/`/script-log` pill reflects
|
||||
aggregate stream health (live when ≥1 node stream is up), matching today's DPS pill semantics.
|
||||
|
||||
**Docs shipped:** `docs/Telemetry.md` (new, canonical guide), `docs/Configuration.md` §
|
||||
`Telemetry`/`TelemetryDial`, `docs/Redundancy.md` cross-reference, this section, and the design §6.3
|
||||
supersede note.
|
||||
|
||||
**Exit gate (not yet run):** all AdminUI live panels green against the docker-dev rig with
|
||||
`Telemetry:Mode=Grpc`/`TelemetryDial:Mode=Grpc` and the four DPS telemetry bridges NOT spawned;
|
||||
kill-and-reconnect of a site node's dialer recovers every stream (driver-health/resilience re-prime
|
||||
from the node hub's snapshot replay). Treat Phase 5 as **code-complete, not verified**, until this
|
||||
lands — see `docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md` Task 12.
|
||||
|
||||
### Phase 6 — Mesh partition + co-location topology
|
||||
**Scope:** per-cluster seed nodes — ~~adopt ScadaBridge's self-first ordering and RETIRE the
|
||||
@@ -267,6 +317,6 @@ resource sizing on the site VMs should be checked once both products run the ful
|
||||
| 2 ClusterClient transport | **DONE 2026-07-22** (live gate PASSED, shipped dark) — shipped dark (`MeshTransport:Mode=Dps` default), both comm actors registered in both modes. Corrections: `SendToAll` not `Send`, one fleet-wide client not one per cluster, `/user/node-communication` not `/user/cluster-communication`. The gate's "Ask timing out" item does not exist to test — no cross-boundary Ask in this phase; see the phase section. |
|
||||
| 3 fetch-and-cache | **DONE 2026-07-23**, merged `d01b0695`, live gate PASSED, pushed to origin (scadaproj umbrella index updated + pushed `b5e7bc8`). gRPC fetch RPC + shared node key; dark switch `ConfigSource:Mode` (Direct default). Rig flip: `OTOPCUA_CONFIG_MODE=FetchAndCache` on the site nodes. See `2026-07-22-mesh-phase3-config-fetch-and-cache.md`. |
|
||||
| 4 cut driver ConfigDb | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase4` (Tasks 0–8 + 1b + 10–11; Task 9 table-drop deferred). ConfigDb admin-only; driver-only ⇒ FetchAndCache (validator); `DbHealthProbeActor` not spawned driver-only (client-visible ServiceLevel 240/250 with central SQL down — proven live); alarm condition state in replicated LocalDb `alarm_condition_state`; central persists acks; `OpcUaPublish` guard split fixes the driver-only address-space wipe; driver-only LDAP maps from appsettings only (6th consumer found mid-phase). Gate: deploy sealed green w/ 4 DB-less site nodes acking, ServiceLevel held 240 w/ SQL down, restart booted last-known-good from the LocalDb pointer. See `2026-07-23-mesh-phase4-cut-driver-configdb.md` + `2026-07-23-mesh-phase4-live-gate.md`. |
|
||||
| 5 gRPC telemetry | not started |
|
||||
| 5 gRPC telemetry | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase5`. 4-channel scope (`alerts`/`script-logs`/`driver-health`/`driver-resilience-status`), 3 deferred with rationale (`redundancy-state`, `fleet-status`, `deployment-acks`); dark switch `Telemetry:Mode`/`TelemetryDial:Mode` (Dps default); node hosts server / central dials; auth-from-day-one fail-closed bearer key, superseding design §6.3. Gate: full 12-stream mesh formed in Grpc mode (2 inbound per driver node, 6 outbound per central), AdminUI pill live (data path proven), kill-and-reconnect of a node recovered its stream in ~5s; Dps baseline dials nothing. Surfaced an upgrade gotcha — `ClusterNode.GrpcPort` must be populated on existing deployments (fresh installs seed it; nullable column doesn't backfill) — which live-validated the graceful null-`GrpcPort` skip. See `docs/Telemetry.md`, `2026-07-23-mesh-phase5-grpc-telemetry-stream.md` + `2026-07-23-mesh-phase5-live-gate.md`. |
|
||||
| 6 mesh partition + co-location | not started |
|
||||
| 7 drill + live gates | not started |
|
||||
|
||||
@@ -0,0 +1,650 @@
|
||||
# Per-Cluster Mesh Phase 5 — gRPC live-telemetry stream Implementation Plan
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: execute this with superpowers-extended-cc:subagent-driven-development
|
||||
> (chosen for the program's prior phases). Fresh subagent per task; classification-driven review chain.
|
||||
|
||||
**Goal:** Replace the DPS fan-out of the live-telemetry observability channels with one gRPC
|
||||
server-streaming contract — **each driver node hosts the telemetry gRPC server (Kestrel h2c),
|
||||
central dials in** — so AdminUI live panels keep working once the meshes split (Phase 6) and no
|
||||
longer depend on a shared Akka gossip ring for fleet observability. Ships as a per-node **dark
|
||||
switch** (`Telemetry:Mode` = `Dps` default | `Grpc`), authenticated fail-closed from day one.
|
||||
|
||||
**Architecture:** Mirror ScadaBridge's `SiteStreamService` shape (recon'd 2026-07-23), adapted to
|
||||
OtOpcUa's substrate. Node side: the four telemetry publish seams also emit into a **node-local
|
||||
in-process hub** (`ITelemetryLocalHub`) — DPS publishing is left untouched, so the switch is pure —
|
||||
and a streaming gRPC service fans the hub to connected clients through per-subscriber bounded
|
||||
`DropOldest` channels. Central side: a supervisor actor discovers driver-node gRPC endpoints from
|
||||
`ClusterNode` rows (`Host` + `GrpcPort`, added in Phase 1), keeps one reconnecting dialer per node,
|
||||
converts streamed events back to the domain records, and feeds **the exact same in-process sinks the
|
||||
DPS SignalR bridges feed today** (`IInProcessBroadcaster<AlarmTransitionEvent>`,
|
||||
`IInProcessBroadcaster<ScriptLogEntry>`, `IDriverStatusSnapshotStore`,
|
||||
`IDriverResilienceStatusStore`). The AdminUI components are untouched: only the bridge's *upstream*
|
||||
swaps.
|
||||
|
||||
**Tech Stack:** .NET 10, `Grpc.AspNetCore` / `Grpc.Net.Client` (already in-repo via Phase 3),
|
||||
`Grpc.Tools` codegen, Akka.NET, `System.Threading.Channels`, xUnit + Shouldly.
|
||||
|
||||
---
|
||||
|
||||
## Scope (settled with the user 2026-07-23)
|
||||
|
||||
**In scope — migrate these four node→central observability channels** to the stream as four `oneof`
|
||||
event kinds:
|
||||
|
||||
| DPS topic today | Message record | Central sink fed today |
|
||||
|---|---|---|
|
||||
| `alerts` | `AlarmTransitionEvent` | `IInProcessBroadcaster<AlarmTransitionEvent>` (+ `AlertHub`) |
|
||||
| `script-logs` | `ScriptLogEntry` | `IInProcessBroadcaster<ScriptLogEntry>` (+ `ScriptLogHub`) |
|
||||
| `driver-health` | `DriverHealthChanged` | `IDriverStatusSnapshotStore` (+ `DriverStatusHub`) |
|
||||
| `driver-resilience-status` | `DriverResilienceStatusChanged` | `IDriverResilienceStatusStore` (no hub) |
|
||||
|
||||
**Explicitly deferred, with rationale (do NOT migrate in Phase 5):**
|
||||
|
||||
- **`redundancy-state`** — bidirectional, built from `Cluster.State`, **pair-local control-plane**
|
||||
that drives ServiceLevel + the Primary gate (consumed by `OpcUaPublishActor`,
|
||||
`ScriptedAlarmHostActor`, `DriverHostActor`, `HistorianAdapterActor`). It stays on DPS in both
|
||||
MeshTransport modes today, and under Phase 6 it is pair-local and works in-mesh. Central's *display*
|
||||
of each pair's redundancy is a Phase 6 cross-mesh concern (possibly a later added event kind), not a
|
||||
Phase 5 observability panel.
|
||||
- **`fleet-status`** — **central-internal**: `FleetStatusBroadcaster` (admin singleton) builds it
|
||||
from the admin node's own cluster membership/reachability/leader events; `Fleet.razor` **polls the
|
||||
ConfigDB** and ignores the feed entirely. It is not a node→central stream, and its live UI path is
|
||||
already DB-polled. Revisit in Phase 6 when central loses gossip visibility of site nodes.
|
||||
- **`deployment-acks`** — already rides the Phase 2 ClusterClient transport when
|
||||
`MeshTransport:Mode=ClusterClient`; it is a command-plane reply, not telemetry.
|
||||
|
||||
This narrowing mirrors how Phases 1 and 3 honestly scoped down from the program sketch. The program
|
||||
doc's Phase 5 line lists all seven; this plan records the four that are genuinely live node→central
|
||||
observability and defers the rest with reasons above. **Update the program doc + design §6.3 in
|
||||
Task 10.**
|
||||
|
||||
## Direction & the dark switch (read before any task)
|
||||
|
||||
- **Node = server, central = client** (the load-bearing ScadaBridge inversion). Telemetry
|
||||
originates on driver nodes; central/admin consumes it. A fused `admin,driver` node both hosts (as
|
||||
driver) and dials (as admin) — it dials itself plus its pair peer, same as central dials site nodes.
|
||||
- **`Telemetry:Mode` is read at startup; both code paths are compiled into every binary.** Flipping
|
||||
the flag is an appsettings/env change + restart — NOT a rebuild — exactly the Phase 2/3 dark-switch
|
||||
discipline (`OTOPCUA_CONFIG_MODE` on the rig). The node **always** hosts the gRPC server when
|
||||
`Telemetry:GrpcListenPort > 0` and **always** emits into the local hub AND publishes DPS, in both
|
||||
modes — so central can ingest from either side without a node redeploy. Only **central's ingest
|
||||
source** switches: `Dps` → today's four DPS SignalR bridges subscribe and feed the sinks; `Grpc` →
|
||||
those four bridges are NOT spawned and the dial supervisor feeds the identical sinks instead.
|
||||
- **Auth from day one.** Reuse the fail-closed `FixedTimeEquals` bearer interceptor pattern
|
||||
(`ConfigServeAuthInterceptor` / `LocalDbSyncAuthInterceptor`). Shared node key
|
||||
`Telemetry:ApiKey` (serve side) == `TelemetryDial:ApiKey` (central side). This supersedes design
|
||||
§6.3's "unauthenticated for now" — ScadaBridge itself closed that gap with this same pattern.
|
||||
|
||||
## Reuse map (from recon 2026-07-23 — exact sites)
|
||||
|
||||
- **Proto + codegen:** add `telemetry.proto` beside `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos/deployment_artifact.proto`; register a `<Protobuf Include="Protos\telemetry.proto" GrpcServices="Both"/>` in `ZB.MOM.WW.OtOpcUa.Commons.csproj:29` (same block as the existing item). Generated types are shared by node (Runtime/Host) and central (AdminUI) from the one Commons reference.
|
||||
- **Server hosting + Kestrel h2c:** the dedicated-listener block `Program.cs:434-533` and the `MapGrpcService` gate `Program.cs:577-580`. Add `telemetryListenPort` alongside `syncListenPort`/`configServeGrpcPort`.
|
||||
- **Auth interceptor:** copy `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/ConfigServeAuthInterceptor.cs`; add to the shared `AddGrpc` pipeline at `Program.cs:425-431`.
|
||||
- **Client dialing:** `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/GrpcDeploymentArtifactFetcher.cs` (channel-cache, Bearer metadata, h2c `GrpcChannel.ForAddress`, linked-CTS deadline).
|
||||
- **Options + validator:** `ConfigSourceOptions`/`ConfigSourceOptionsValidator` in `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`; registered via `AddValidatedOptions` in `ServiceCollectionExtensions.AddOtOpcUaCluster` (`:32`).
|
||||
- **Node discovery:** `CentralCommunicationActor.cs:198-219` already reads `ClusterNodes` (enabled, non-maintenance) selecting `NodeId, Host, AkkaPort` — extend the same query shape to `GrpcPort` for telemetry dial targets. `ClusterNode.GrpcPort` (nullable) already exists (`Entities/ClusterNode.cs:46`) explicitly for "the Phase 5 telemetry stream."
|
||||
- **Central sinks (the untouched seam):** registered in `HubServiceCollectionExtensions.AddOtOpcUaDriverStatusServices` (`:29-35`); DPS bridges spawned in `WithOtOpcUaSignalRBridges` (`:53-83`).
|
||||
- **ScadaBridge reference to mirror:** `SiteStreamGrpcServer.cs` (relay-actor + bounded `DropOldest` channel + lifecycle/cleanup + concurrency cap + max-stream-lifetime), `SiteStreamGrpcClient.cs` / `SiteStreamGrpcClientFactory.cs` (channel cache, keepalive), `SiteAlarmAggregatorActor.cs` (generation-stamped, budget-limited, self-healing reconnect), `ProtoContractTests.cs` (reflection-over-oneof contract lock).
|
||||
|
||||
---
|
||||
|
||||
## Tasks
|
||||
|
||||
### Task 0: Telemetry proto contract + codegen + contract-lock test
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 1, Task 2
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos/telemetry.proto`
|
||||
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Commons/ZB.MOM.WW.OtOpcUa.Commons.csproj` (add the `<Protobuf>` item next to line 29)
|
||||
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests/Protos/TelemetryProtoContractTests.cs` (create; match the existing Commons.Tests project layout — if there is no Commons.Tests project, add the test to the nearest existing Core test project that already references generated Commons types and note it)
|
||||
|
||||
**Step 1: Write `telemetry.proto`.** Package `telemetry.v1`, `csharp_namespace = ZB.MOM.WW.OtOpcUa.Commons.Protos.Telemetry.V1`. One server-streaming RPC + a `oneof` envelope with the four event kinds. Every enum carries a `*_UNSPECIFIED = 0` zero value; every field comment marks additive intent.
|
||||
|
||||
```proto
|
||||
syntax = "proto3";
|
||||
package telemetry.v1;
|
||||
option csharp_namespace = "ZB.MOM.WW.OtOpcUa.Commons.Protos.Telemetry.V1";
|
||||
|
||||
import "google/protobuf/timestamp.proto";
|
||||
|
||||
// Central dials each driver node and opens Subscribe; the node streams its own live telemetry.
|
||||
// Additive-only field evolution: never renumber/reuse a tag; a pre-field peer must read a new
|
||||
// field's proto3 default correctly. Locked by TelemetryProtoContractTests.
|
||||
service TelemetryStreamService {
|
||||
rpc Subscribe(TelemetryStreamRequest) returns (stream TelemetryEvent);
|
||||
}
|
||||
|
||||
message TelemetryStreamRequest {
|
||||
string correlation_id = 1; // safe Akka path element; becomes the relay actor name on the node
|
||||
}
|
||||
|
||||
message TelemetryEvent {
|
||||
string correlation_id = 1;
|
||||
oneof event {
|
||||
AlarmTransition alarm_transition = 2; // <- DPS topic "alerts"
|
||||
ScriptLog script_log = 3; // <- DPS topic "script-logs"
|
||||
DriverHealth driver_health = 4; // <- DPS topic "driver-health"
|
||||
DriverResilienceStatus driver_resilience = 5; // <- DPS topic "driver-resilience-status"
|
||||
}
|
||||
}
|
||||
|
||||
// The four event bodies below mirror the C# records field-for-field. Encode the domain records'
|
||||
// enums as proto enums (UNSPECIFIED=0), and DateTimes as google.protobuf.Timestamp.
|
||||
message AlarmTransition { /* fields mirroring Commons/Messages/Alerts/AlarmTransitionEvent.cs */ }
|
||||
message ScriptLog { /* fields mirroring Commons/Messages/Logging/ScriptLogEntry.cs */ }
|
||||
message DriverHealth { /* fields mirroring Commons/Messages/Drivers/DriverHealthChanged.cs */ }
|
||||
message DriverResilienceStatus{ /* fields mirroring Commons/Messages/Drivers/DriverResilienceStatusChanged.cs */ }
|
||||
```
|
||||
|
||||
The implementer MUST open each of the four domain records (paths in the scope table above) and
|
||||
transcribe every field into the corresponding message with a stable tag number, choosing
|
||||
`Timestamp`/`Int32`/`string`/`bool`/enum as the type dictates. Where a record enum exists, define a
|
||||
matching proto enum with `_UNSPECIFIED = 0`. This is the contract — get it complete, because the
|
||||
mapping tasks (7) and the lock test below depend on it.
|
||||
|
||||
**Step 2: Register codegen.** In the `.csproj`, next to the existing `deployment_artifact.proto` item:
|
||||
```xml
|
||||
<Protobuf Include="Protos\telemetry.proto" GrpcServices="Both" />
|
||||
```
|
||||
|
||||
**Step 3: Write the contract-lock test (fails first).** Mirror ScadaBridge `ProtoContractTests.AllOneofVariants_HaveConversionHandlers`: reflect over `Enum.GetValues<TelemetryEvent.EventOneofCase>()` minus `None`, assert count- and membership-equality against a hand-maintained `HandledCases` array that Task 7's converter will also key off. This fails until the proto compiles and the array is filled.
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public void EveryOneofVariant_IsAccountedFor()
|
||||
{
|
||||
var variants = Enum.GetValues<TelemetryEvent.EventOneofCase>()
|
||||
.Where(c => c != TelemetryEvent.EventOneofCase.None).ToArray();
|
||||
variants.ShouldBe(TelemetryProtoContract.HandledCases, ignoreOrder: true);
|
||||
}
|
||||
```
|
||||
Add a tiny `TelemetryProtoContract.HandledCases` constant array in Commons (the single source both
|
||||
this test and the Task-7 converter reference) listing the four cases.
|
||||
|
||||
**Step 4: `dotnet build ZB.MOM.WW.OtOpcUa.slnx`** — expect the generated types to appear; test goes green.
|
||||
|
||||
**Step 5: Commit.** `feat(mesh-phase5): telemetry.proto contract + oneof + contract-lock test`
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Telemetry options (serve + dial) + validator + registration
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** Task 0, Task 2
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/TelemetryOptions.cs`
|
||||
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs` (register in `AddOtOpcUaCluster`, ~lines 40-49 block)
|
||||
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/TelemetryOptionsValidatorTests.cs`
|
||||
|
||||
**Step 1:** Define two options classes and one validator, mirroring `ConfigSourceOptions`/`ConfigServeOptions` + `ConfigSourceOptionsValidator` (which reads roles from `IConfiguration`, NOT `IClusterRoleInfo`).
|
||||
|
||||
```csharp
|
||||
public sealed class TelemetryOptions // section "Telemetry" (serve side, node)
|
||||
{
|
||||
public const string SectionName = "Telemetry";
|
||||
public string Mode { get; set; } = "Dps"; // Dps | Grpc (central-ingest selector; harmless on node)
|
||||
public int GrpcListenPort { get; set; } // 0 = disabled; driver node's telemetry h2c port
|
||||
public string ApiKey { get; set; } = ""; // shared node key; supply via ${secret:}/env
|
||||
}
|
||||
|
||||
public sealed class TelemetryDialOptions // section "TelemetryDial" (central)
|
||||
{
|
||||
public const string SectionName = "TelemetryDial";
|
||||
public string Mode { get; set; } = "Dps"; // Dps | Grpc
|
||||
public string ApiKey { get; set; } = ""; // must equal the nodes' Telemetry:ApiKey
|
||||
public int ContactRefreshSeconds { get; set; } = 60;
|
||||
public int CallTimeoutSeconds { get; set; } = 30;
|
||||
}
|
||||
```
|
||||
|
||||
**Step 2: Validator rules** (`TelemetryOptionsValidator : IValidateOptions<TelemetryOptions>`, ctor-inject `IConfiguration`, read `Cluster:Roles` like `ConfigSourceOptionsValidator:98`):
|
||||
- `Mode` must be `Dps` or `Grpc` (case-insensitive) — else Fail.
|
||||
- A **driver-role** node with `Mode=Grpc` must have `GrpcListenPort > 0` — else Fail ("nothing to serve").
|
||||
- `Mode=Grpc` with a non-empty role set must have a non-empty `ApiKey` (fail-closed: refuse to host an un-keyed telemetry surface) — else Fail.
|
||||
- Add a sibling `TelemetryDialOptionsValidator`: `Mode` in {Dps,Grpc}; `Mode=Grpc` ⇒ `ApiKey` non-empty.
|
||||
|
||||
**Step 3: Register** via `AddValidatedOptions<TelemetryOptions, TelemetryOptionsValidator>(configuration, TelemetryOptions.SectionName)` and the dial equivalent, in `AddOtOpcUaCluster`.
|
||||
|
||||
**Step 4:** Tests — Grpc+driver+port0 fails; Grpc+empty-key fails; Dps passes with defaults; unknown Mode fails. Run `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests`.
|
||||
|
||||
**Step 5: Commit.** `feat(mesh-phase5): Telemetry/TelemetryDial options + fail-closed validators`
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Node-local in-process telemetry hub
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 0, Task 1
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Telemetry/ITelemetryLocalHub.cs`
|
||||
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Telemetry/TelemetryLocalHub.cs`
|
||||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Telemetry/TelemetryLocalHubTests.cs`
|
||||
|
||||
**Design:** a process-wide singleton carrying **this node's own** telemetry (never cluster-wide — that
|
||||
is what keeps it correct on both the current single mesh and the Phase-6 pair mesh; DPS would leak
|
||||
peers' events). It holds a **domain-typed** union (not proto) so the publish seams stay decoupled from
|
||||
generated types. Two responsibilities:
|
||||
|
||||
1. `Emit(TelemetryItem)` — fan the item to every currently-subscribed writer through **bounded
|
||||
per-subscriber channels with `FullMode = DropOldest`** (live view is lossy under backpressure by
|
||||
design; a slow central never blocks the node).
|
||||
2. **Snapshot replay for last-value channels.** Keep a last-value cache for `DriverHealth` (keyed by
|
||||
driver instance) and `DriverResilienceStatus` (keyed by `(instance, host)`); `alerts`/`script-logs`
|
||||
are append logs and are NOT cached. On `Subscribe`, first drain the cached snapshots to the new
|
||||
writer, then attach it for live deltas — the simplified equivalent of ScadaBridge's seed-then-stream,
|
||||
so a central reconnect immediately re-primes the driver-health/resilience stores.
|
||||
|
||||
```csharp
|
||||
public abstract record TelemetryItem
|
||||
{
|
||||
public sealed record Alarm(AlarmTransitionEvent E) : TelemetryItem;
|
||||
public sealed record Script(ScriptLogEntry E) : TelemetryItem;
|
||||
public sealed record Health(DriverHealthChanged E) : TelemetryItem;
|
||||
public sealed record Resilience(DriverResilienceStatusChanged E) : TelemetryItem;
|
||||
}
|
||||
|
||||
public interface ITelemetryLocalHub
|
||||
{
|
||||
void Emit(TelemetryItem item);
|
||||
// Returns a reader that first yields cached snapshots, then live deltas; disposing the
|
||||
// subscription detaches and completes the channel.
|
||||
ITelemetrySubscription Subscribe(int boundedCapacity);
|
||||
}
|
||||
```
|
||||
|
||||
`TelemetryLocalHub` uses a `ConcurrentDictionary<Guid, Channel<TelemetryItem>>` of subscribers +
|
||||
two `ConcurrentDictionary` snapshot caches. `Emit` updates the snapshot cache (for Health/Resilience)
|
||||
then `TryWrite`s to each subscriber (drop-oldest handled by the bounded channel). `Subscribe` snapshots
|
||||
the caches into the new channel under a brief lock ordering guarantee (cache-then-attach) so no delta
|
||||
is lost across the attach boundary.
|
||||
|
||||
**Register** as a singleton in the driver-role DI branch (Task 3 wires the producers; do the
|
||||
`AddSingleton<ITelemetryLocalHub, TelemetryLocalHub>()` here in the Runtime `ServiceCollectionExtensions`
|
||||
`hasDriver` path, or wherever driver-role Runtime services register — locate and match).
|
||||
|
||||
**Tests:** Emit-before-Subscribe is not seen except via snapshot (Health/Resilience last-value IS seen;
|
||||
Alarm/Script are not); two subscribers each get their own copy; a full channel drops oldest not newest;
|
||||
dispose detaches. Run the Runtime.Tests subset.
|
||||
|
||||
**Step 5: Commit.** `feat(mesh-phase5): node-local telemetry hub (snapshot-replay + drop-oldest fan-out)`
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Tap the four publish seams into the hub
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** none (needs Task 2)
|
||||
|
||||
**Files (all Modify) — add a hub `Emit` beside the existing DPS `Publish`, leaving DPS intact:**
|
||||
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/AkkaDriverHealthPublisher.cs:36` (driver-health)
|
||||
- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Drivers/DriverResilienceStatusPublisherService.cs:80` (resilience)
|
||||
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/ScriptedAlarmHostActor.cs:379` + `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs:1405` (alerts — two producers)
|
||||
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/VirtualTagActor.cs:261` + `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Scripting/DpsScriptLogPublisher.cs:46` (script-logs — two producers)
|
||||
- Test: extend the nearest existing publisher tests, or add `tests/.../Telemetry/PublishSeamEmitsToHubTests.cs`
|
||||
|
||||
**Approach:** inject `ITelemetryLocalHub` into each producer (constructor for the services/publishers;
|
||||
for actors, pass via `Props` — match how each already receives dependencies). At each seam, immediately
|
||||
after the existing `Mediator.Tell(new Publish(...))`, add `_telemetryHub.Emit(new TelemetryItem.Xxx(msg))`.
|
||||
**Do NOT remove or gate the DPS publish** — it is the `Dps`-mode path and must remain unconditional so
|
||||
the switch stays pure. The hub is a no-op sink until a client subscribes, so this is safe on every node
|
||||
regardless of mode.
|
||||
|
||||
For actors constructed where a hub isn't readily resolvable, resolve the singleton once at spawn and
|
||||
thread it through `Props` (do not `DependencyResolver` inside the actor per-message). Where a producer
|
||||
is only present on driver-role nodes, the hub singleton is guaranteed registered (Task 2).
|
||||
|
||||
**Tests:** each producer, when driven, results in exactly one `hub.Emit` of the correct `TelemetryItem`
|
||||
subtype carrying the same payload it published to DPS. Use a fake `ITelemetryLocalHub` capturing emits.
|
||||
|
||||
**Step 5: Commit.** `feat(mesh-phase5): tap the 4 telemetry publish seams into the local hub (DPS intact)`
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Node-side gRPC streaming service
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min (mirror ScadaBridge closely)
|
||||
**Parallelizable with:** none (needs Task 0, Task 2)
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Grpc/TelemetryStreamGrpcService.cs` (place beside `DeploymentArtifactService.cs`; alias the generated base to avoid the name collision, as that file does at its lines 9-10)
|
||||
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Grpc/TelemetryProtoMap.Node.cs` (domain→proto mapping; the reverse of Task 7)
|
||||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Grpc/TelemetryStreamGrpcServiceTests.cs`
|
||||
|
||||
**Design — port `SiteStreamGrpcServer.RunSubscriptionStreamAsync` (`.cs:253-385`) shape:**
|
||||
|
||||
```csharp
|
||||
public sealed class TelemetryStreamGrpcService : GeneratedTelemetryBase
|
||||
{
|
||||
// ctor: ITelemetryLocalHub hub, ILogger, IOptions<TelemetryOptions> (for a GrpcMaxConcurrentStreams knob)
|
||||
public override async Task Subscribe(TelemetryStreamRequest request,
|
||||
IServerStreamWriter<TelemetryEvent> responseStream, ServerCallContext context)
|
||||
{
|
||||
// 1. Validate correlation_id is a safe id (it labels the subscription; reject empty/oversized).
|
||||
// 2. Concurrency cap (default 100) -> throw RpcException(ResourceExhausted) when exceeded.
|
||||
// 3. sub = _hub.Subscribe(boundedCapacity: 1000); // snapshot-then-live, DropOldest inside the hub
|
||||
// 4. MaxStreamLifetime linked-CTS CancelAfter (default 4h) ORed with context.CancellationToken,
|
||||
// so a zombie stream terminates even if h2c keepalive misses it.
|
||||
// 5. await foreach (item in sub.Reader.ReadAllAsync(ct))
|
||||
// await responseStream.WriteAsync(TelemetryProtoMap.ToProto(item, request.CorrelationId), ct);
|
||||
// 6. finally: sub.Dispose(); balance any opened/closed gauge; swallow OperationCanceled as normal.
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Key correctness points (from the ScadaBridge recon — do not skip):
|
||||
- The **bounded DropOldest channel lives in the hub** (Task 2), so the service just pumps its reader —
|
||||
a slow/blocked central cannot back-pressure the node's actor threads.
|
||||
- Wrap the whole body so a client disconnect (`RpcException`/`OperationCanceledException`) exits cleanly
|
||||
and disposes the subscription; never let it bubble as a fault.
|
||||
- No readiness race: the hub is a plain singleton available at host build, so unlike ScadaBridge's
|
||||
`SetReady(ActorSystem)` gate there is nothing to defer — but if the driver actor system isn't up yet
|
||||
the hub simply has no snapshots and no deltas, which is fine.
|
||||
|
||||
**`TelemetryProtoMap.ToProto`**: switch on `TelemetryItem` subtype → build the matching proto message,
|
||||
wrap in `TelemetryEvent { CorrelationId, <oneof> = ... }`. Transcribe every field (the reverse of the
|
||||
`.proto` transcription in Task 0). DateTimes → `Timestamp.FromDateTime(utc)`.
|
||||
|
||||
**Tests:** in-memory — subscribe, emit each of the four item kinds into the hub, assert the service
|
||||
writes the correct proto event with fields intact; a cancelled `ServerCallContext` ends the stream and
|
||||
disposes; the concurrency cap throws `ResourceExhausted` past the limit. (Full two-host wire test is
|
||||
covered by the Task-12 live gate + optionally a Task-8 boundary test.)
|
||||
|
||||
**Step 5: Commit.** `feat(mesh-phase5): node-side TelemetryStreamService (hub -> server-streaming)`
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Telemetry stream auth interceptor
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~3 min
|
||||
**Parallelizable with:** none (needs Task 1)
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/TelemetryStreamAuthInterceptor.cs` (copy `ConfigServeAuthInterceptor.cs` verbatim, change the gated prefix + options source)
|
||||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Configuration/TelemetryStreamAuthInterceptorTests.cs`
|
||||
|
||||
**Steps:**
|
||||
- `ServicePrefix = "/telemetry.v1.TelemetryStreamService/"`; read `IOptions<TelemetryOptions>.ApiKey`.
|
||||
- Override all four handler kinds (especially **ServerStreaming** — that is the RPC shape here) so the
|
||||
gate holds; non-matching paths pass through; empty key ⇒ throw (fail-closed); `FixedTimeEquals` on the
|
||||
Bearer token; reject with `PermissionDenied`.
|
||||
- **Keep exactly one public constructor** (the ScadaBridge recon flagged that `Grpc.AspNetCore` silently
|
||||
stops authorizing an interceptor with >1 ctor — pin it with a reflection test).
|
||||
|
||||
**Tests:** right key passes; wrong key `PermissionDenied`; empty configured key rejects all; a call to a
|
||||
different service path passes through.
|
||||
|
||||
**Step 5: Commit.** `feat(mesh-phase5): fail-closed bearer interceptor for the telemetry stream`
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Kestrel wiring + map the node telemetry server
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** none (needs Task 1, Task 4, Task 5)
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` (the `AddGrpc` block 425-431, the dedicated-listener block 434-533, and the map block 577-580)
|
||||
- Test: none new (covered by Task 12 live gate); build must stay green
|
||||
|
||||
**Steps — extend, do not rewrite, the existing block:**
|
||||
|
||||
1. **Interceptor registration** (`Program.cs:425-431`): inside the `AddGrpc(o => …)`, add
|
||||
`if (hasDriver) o.Interceptors.Add<TelemetryStreamAuthInterceptor>();` (alongside the LocalDbSync one —
|
||||
both are driver-side, both path-scoped, harmless when their service is unmapped).
|
||||
|
||||
2. **Port resolution** (near `:453-454`):
|
||||
```csharp
|
||||
var telemetryListenPort = hasDriver ? builder.Configuration.GetValue<int>("Telemetry:GrpcListenPort") : 0;
|
||||
```
|
||||
Add `|| telemetryListenPort > 0` to the block guard at `:455`.
|
||||
|
||||
3. **Bind it** inside the `ConfigureKestrel` closure (`:517-526`): capture `var telemetryPortToBind = telemetryListenPort;` beside the other two, and add
|
||||
`if (telemetryPortToBind > 0) kestrel.ListenAnyIP(telemetryPortToBind, o => o.Protocols = HttpProtocols.Http2);`.
|
||||
|
||||
4. **HTTPS-refuse branch** (`:500-511`): set `telemetryListenPort = 0;` too, and add it to the log
|
||||
message (all dedicated listeners disable together when the host serves HTTPS).
|
||||
|
||||
5. **Map the service** (after `:580`):
|
||||
```csharp
|
||||
if (hasDriver && telemetryListenPort > 0)
|
||||
app.MapGrpcService<TelemetryStreamGrpcService>();
|
||||
```
|
||||
|
||||
6. Update the block's header comment (434-452) to mention the third dedicated port.
|
||||
|
||||
**Verify:** `dotnet build`; a fused node with all three ports set binds all three exactly once (the
|
||||
"re-bind existing surface once" invariant already holds — the new port is just one more
|
||||
`ListenAnyIP`). A driver-only node with only `Telemetry:GrpcListenPort` set binds only that.
|
||||
|
||||
**Step 5: Commit.** `feat(mesh-phase5): host the telemetry gRPC server on driver nodes (dedicated h2c port)`
|
||||
|
||||
---
|
||||
|
||||
### Task 7: Central per-node dialer client + proto→domain mapping
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** none (needs Task 0)
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Telemetry/TelemetryStreamClient.cs`
|
||||
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Telemetry/TelemetryProtoMap.Central.cs` (proto→domain; reverse of Task 4; keyed off `TelemetryProtoContract.HandledCases`)
|
||||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Telemetry/TelemetryProtoMapTests.cs`
|
||||
|
||||
**Design — mirror `SiteStreamGrpcClient.cs` + `GrpcDeploymentArtifactFetcher` channel handling:**
|
||||
|
||||
```csharp
|
||||
public sealed class TelemetryStreamClient : IDisposable
|
||||
{
|
||||
// ctor: string endpoint (http://host:port), string apiKey, IClientFactory seam for tests
|
||||
// - GrpcChannel.ForAddress(endpoint) over http:// => prior-knowledge h2c; cache one channel per endpoint.
|
||||
// - HTTP/2 keepalive: PingDelay 15s / Timeout 10s / Always (SocketsHttpHandler on the channel).
|
||||
public async Task RunAsync(string correlationId, Action<TelemetryItem> onEvent,
|
||||
Action<Exception> onError, CancellationToken ct)
|
||||
{
|
||||
// headers: authorization: Bearer {apiKey}
|
||||
// using call = client.Subscribe(new TelemetryStreamRequest{CorrelationId=correlationId}, headers, ct)
|
||||
// await foreach (evt in call.ResponseStream.ReadAllAsync(ct)) onEvent(TelemetryProtoMap.ToDomain(evt));
|
||||
// RpcException(Cancelled) on shutdown => normal; anything else => onError (the reconnect trigger).
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`TelemetryProtoMap.ToDomain(TelemetryEvent)` switches on `evt.EventCase` and reconstructs the domain
|
||||
record (`AlarmTransitionEvent` / `ScriptLogEntry` / `DriverHealthChanged` / `DriverResilienceStatusChanged`).
|
||||
It MUST cover exactly `TelemetryProtoContract.HandledCases` — a `default: throw` on an unknown case makes
|
||||
"new event added but not mapped" a loud runtime failure, and the Task-0 lock test makes it a compile-time-ish
|
||||
guard.
|
||||
|
||||
**Tests:** round-trip each domain record → `TelemetryProtoMap.ToProto` (Task 4) → `ToDomain` → assert
|
||||
field-equality across all four kinds and all enum values (wire-fidelity, like ScadaBridge
|
||||
`ProtoRoundtripTests`); an unmapped `EventCase` throws.
|
||||
|
||||
**Step 5: Commit.** `feat(mesh-phase5): central telemetry dialer client + proto<->domain round-trip`
|
||||
|
||||
---
|
||||
|
||||
### Task 8: Central dial supervisor actor (discovery + reconnect + feed sinks)
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** none (needs Task 1, Task 7)
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Telemetry/TelemetryDialSupervisor.cs`
|
||||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Telemetry/TelemetryDialSupervisorTests.cs`
|
||||
|
||||
**Design — an admin-role actor holding one dialer per driver node, feeding the existing central sinks:**
|
||||
|
||||
- **Discovery:** on `PreStart` and every `TelemetryDialOptions.ContactRefreshSeconds` (+ on an
|
||||
admin-change signal if one is already wired for `CentralCommunicationActor`), read enabled,
|
||||
non-maintenance `ClusterNode` rows selecting `NodeId, Host, GrpcPort` (extend the
|
||||
`CentralCommunicationActor.cs:198-219` query). A row with `GrpcPort == null` is skipped with a Warning
|
||||
(the node exposes no telemetry port — honest, not an error). Build `http://{Host}:{GrpcPort}` dial
|
||||
targets; add dialers for new nodes, stop dialers for removed ones.
|
||||
- **Per-node dialer loop** (mirror `SiteAlarmAggregatorActor.OpenGrpcStream`/`HandleGrpcError`,
|
||||
simplified — one node per dialer, no NodeA/NodeB flip since each `ClusterNode` row is its own node):
|
||||
- Open `TelemetryStreamClient.RunAsync`; `onEvent` posts each `TelemetryItem` to the supervisor which
|
||||
routes to the sink (below); `onError` schedules a reconnect.
|
||||
- **Reconnect:** immediate first retry, then fixed `_reconnectDelay` (5s) backoff. A **monotonic
|
||||
generation stamp** per node makes late errors from a superseded stream ignorable. The dialer does
|
||||
**not** die on repeated failure — it keeps retrying (observability, not data plane); log the first
|
||||
failure + every Nth.
|
||||
- A `correlationId` per (node, generation) — a safe id string, e.g. `central-{NodeId}-{gen}`.
|
||||
- **Feed the sinks (the untouched seam):** inject the four singletons and route by item type:
|
||||
- `Alarm` → `IInProcessBroadcaster<AlarmTransitionEvent>.Publish(e)`
|
||||
- `Script` → `IInProcessBroadcaster<ScriptLogEntry>.Publish(e)`
|
||||
- `Health` → `IDriverStatusSnapshotStore.Upsert(e)`
|
||||
- `Resilience` → `IDriverResilienceStatusStore.Upsert(e)`
|
||||
- **Connection indicator:** drive the two broadcasters' `SetConnected(...)` off aggregate stream health
|
||||
— connected when **≥1** node stream is up, disconnected when all are down — matching today's
|
||||
DPS `SubscribeAck`/`PostStop` pill semantics on `/alerts` and `/script-log`. (The two store-backed
|
||||
panels have no connection flag today; leave their per-row staleness as-is — noted for Task 10 docs.)
|
||||
|
||||
**Registration** is done in Task 9 (mode-gated). This task only builds + unit-tests the actor with a
|
||||
fake `TelemetryStreamClient` factory and fake sinks.
|
||||
|
||||
**Tests:** discovery adds/removes dialers as the (fake) `ClusterNode` set changes; a null `GrpcPort`
|
||||
row is skipped; an emitted item of each kind lands on the correct sink; a stream error triggers a
|
||||
scheduled reconnect with an incremented generation; a late event from a superseded generation is
|
||||
dropped; broadcaster `IsConnected` flips true on first up / false on all-down.
|
||||
|
||||
**Step 5: Commit.** `feat(mesh-phase5): central telemetry dial supervisor (discover + reconnect + feed sinks)`
|
||||
|
||||
---
|
||||
|
||||
### Task 9: Wire the dark switch (central ingest source)
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** none (needs Task 8)
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Hubs/HubServiceCollectionExtensions.cs` (`WithOtOpcUaSignalRBridges`, lines 53-83)
|
||||
- Modify: the AdminUI Akka configurator call site that invokes `WithOtOpcUaSignalRBridges` (locate; it's inside the `hasAdmin` branch per the method's own doc example) — pass or resolve the mode
|
||||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/Hubs/TelemetryModeWiringTests.cs`
|
||||
|
||||
**Steps:**
|
||||
- Inside `WithActors`, resolve `IOptions<TelemetryDialOptions>` and read `Mode`.
|
||||
- **`Mode == Dps` (default):** spawn all five bridges exactly as today (unchanged behaviour).
|
||||
- **`Mode == Grpc`:** spawn the **fleet-status bridge only** (out of Phase 5 scope, stays on DPS), and
|
||||
do **NOT** spawn the four telemetry DPS bridges (alert, script-log, driver-status, resilience).
|
||||
Instead spawn `TelemetryDialSupervisor` (Task 8), resolving the four sink singletons + the
|
||||
`IDbContextFactory<OtOpcUaConfigDbContext>` (admin nodes have ConfigDb) + `IOptions<TelemetryDialOptions>`.
|
||||
- The sinks (`AddOtOpcUaDriverStatusServices`, `:29-35`) are registered identically in both modes — no
|
||||
change there; that is the whole point of the swap.
|
||||
|
||||
**Guard rails:**
|
||||
- Keep the fleet-status bridge on DPS in both modes (deferred).
|
||||
- A `Grpc`-mode admin node still needs the sinks registered (it does — same call), so the AdminUI
|
||||
components resolve them regardless.
|
||||
|
||||
**Tests:** with `Mode=Dps`, the registry has the four telemetry bridge keys and no supervisor; with
|
||||
`Mode=Grpc`, it has the supervisor and the fleet-status bridge but none of the four telemetry bridge
|
||||
keys. (Use the Akka.Hosting `ActorRegistry` + a TestKit or the existing bridge-registration test
|
||||
harness if one exists.)
|
||||
|
||||
**Step 5: Commit.** `feat(mesh-phase5): dark-switch central telemetry ingest (Dps bridges | Grpc dialer)`
|
||||
|
||||
---
|
||||
|
||||
### Task 10: Docs — Telemetry section, supersede §6.3, program status
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** Task 11 (disjoint files)
|
||||
|
||||
**Files (Modify/Create):**
|
||||
- Create: `docs/Telemetry.md` — the stream architecture (direction, dark switch, auth, the four channels, deferred three + why, reconnect story, per-panel connection indicator status).
|
||||
- Modify: `docs/Configuration.md` — add `Telemetry` + `TelemetryDial` sections (keys table, the `Mode` dark switch, port + shared-key `${secret:}` guidance).
|
||||
- Modify: `docs/Redundancy.md` — a short "Command/observability transport" cross-ref noting redundancy-state stays DPS (pair-local) and telemetry moved to gRPC.
|
||||
- Modify: `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.3 — supersede note: "telemetry stream authenticated from day one (fail-closed bearer), superseding the earlier 'unauthenticated for now'; ScadaBridge closed the same gap identically."
|
||||
- Modify: `docs/plans/2026-07-22-per-cluster-mesh-program.md` — Phase 5 section + Tracking row: record the four-channel scope, the three deferrals + rationale, the dark switch, auth-from-day-one.
|
||||
- Modify: `CLAUDE.md` — a "Live telemetry transport (`Telemetry`/`TelemetryDial`)" section mirroring the MeshTransport/ConfigSource sections (default `Dps`, node hosts server, central dials, four channels, deferred three, fail-closed key).
|
||||
|
||||
**Step 5: Commit.** `docs(mesh-phase5): telemetry stream — Telemetry.md, config, supersede §6.3, program status`
|
||||
|
||||
---
|
||||
|
||||
### Task 11: Rig config — telemetry ports + keys + flip env
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** Task 10 (disjoint files); needs Task 6 + Task 9 landed for the flip to mean anything
|
||||
|
||||
**Files:**
|
||||
- Modify: `docker-dev/docker-compose.yml`
|
||||
- Modify: any committed `appsettings.json` defaults that must carry the (default-OFF) keys — verify none are needed since `Mode` defaults to `Dps` and port defaults to 0.
|
||||
|
||||
**Steps:**
|
||||
- Give every **driver-role** node (central-1/2 fused + the four site nodes) `Telemetry__GrpcListenPort`
|
||||
(pick a non-colliding port — verify against 4053 Akka / 4055 ConfigServe / the LocalDb sync port; use
|
||||
e.g. **4056**) and `Telemetry__ApiKey: "telemetry-docker-dev-key"` (committed dev-secret exception, like
|
||||
`configserve-docker-dev-key`).
|
||||
- Give the **central (admin)** nodes `TelemetryDial__ApiKey: "telemetry-docker-dev-key"` (matching).
|
||||
- Add each driver node's `GrpcPort` to its `ClusterNode` seed row (the rig's SQL seed / migration seed) so
|
||||
central discovers the telemetry endpoint — mirror how `AkkaPort` was seeded in Phase 1.
|
||||
- Leave `Telemetry__Mode`/`TelemetryDial__Mode` unset (⇒ `Dps` default). Document the flip in the compose
|
||||
header: set `Telemetry__Mode=Grpc` on the driver nodes **and** `TelemetryDial__Mode=Grpc` on the central
|
||||
nodes at `docker compose up`, then recreate.
|
||||
- `lmxopcua-fix`/rig note: this is the local `docker-dev` rig, not the shared driver-fixture host.
|
||||
|
||||
**Step 5: Commit.** `chore(mesh-phase5): rig telemetry ports + shared key + ClusterNode.GrpcPort seed`
|
||||
|
||||
---
|
||||
|
||||
### Task 12: Live gate
|
||||
|
||||
**Classification:** high-risk (gate, not a code change — but the phase is not done until it passes)
|
||||
**Estimated implement time:** live rig run (not subagent wall-time)
|
||||
**Parallelizable with:** none (needs everything)
|
||||
|
||||
**Files:**
|
||||
- Create: `docs/plans/2026-07-23-mesh-phase5-live-gate.md` (record)
|
||||
|
||||
**Procedure (on the local `docker-dev` rig, matching prior phases' gates):**
|
||||
1. Build the Phase-5 image (`--platform linux/amd64` build stage; `--progress=plain`) and recreate the rig in **default `Dps` mode** — confirm all six nodes up, `/alerts`, `/script-log`, `/hosts` panels live exactly as before (regression check: DPS path unchanged).
|
||||
2. Flip: set `Telemetry__Mode=Grpc` on all driver nodes and `TelemetryDial__Mode=Grpc` on the central nodes; `docker compose up` (recreate).
|
||||
3. **Exit gate leg A — panels green over gRPC:** drive a scripted-alarm transition and a script-log emission and a driver health/resilience change; confirm `/alerts`, `/script-log`, and the `/hosts` driver table update live — **with the four DPS telemetry bridges NOT spawned** (grep the central logs to confirm the Grpc branch ran and the four bridge keys are absent). Confirm the `/alerts` + `/script-log` connection pill reads "live".
|
||||
4. **Exit gate leg B — kill-and-reconnect:** `docker kill` a site driver node (or drop its telemetry port), watch the central dialer log the error + retries and the pill flip to disconnected; restart the node and confirm the dialer reconnects, the driver-health/resilience stores re-prime from the node hub's snapshot replay, and the pill returns to live — **every stream recovers**.
|
||||
5. **Optionally** confirm the auth gate: a dial with a wrong `TelemetryDial:ApiKey` gets `PermissionDenied` and no panel data (a deliberate misconfig probe).
|
||||
6. Record all legs + evidence in the gate doc; note any deviations (as Phases 1-4 gates did).
|
||||
|
||||
**Exit gate (program):** all AdminUI live panels green against the rig with the telemetry DPS bridges
|
||||
off; kill-and-reconnect of the central dialer recovers every stream.
|
||||
|
||||
**Step: Commit** the gate record; then finish per superpowers-extended-cc:finishing-a-development-branch
|
||||
(merge `feat/mesh-phase5` to master locally, push, update the scadaproj umbrella index, update memory).
|
||||
|
||||
---
|
||||
|
||||
## Risks / watch-items carried into execution
|
||||
|
||||
- **Single-mesh correctness of the node-local hub.** The hub must carry only *this node's* events
|
||||
(Task 2) — never subscribe it to cluster DPS, or on the current single mesh every node would stream
|
||||
every other node's events and central would double-count. Snapshot replay + drop-oldest come from the
|
||||
hub, not the service.
|
||||
- **Do not gate/remove the DPS publish** at the four seams (Task 3) — that is the `Dps`-mode path and
|
||||
the pure-dark-switch guarantee. The hub is a no-op until a client connects.
|
||||
- **Interceptor single-ctor pin** (Task 5) — `Grpc.AspNetCore` silently disables an interceptor with
|
||||
>1 public ctor.
|
||||
- **Kestrel "re-bind once" invariant** (Task 6) — the third dedicated port is just one more
|
||||
`ListenAnyIP`; the existing-surface re-bind still happens exactly once. Don't duplicate it.
|
||||
- **Frame size is a non-issue** — server-streaming telemetry events are small and never cross Akka
|
||||
remoting; the 128 KB ClusterClient frame trap does not apply to this path.
|
||||
- **Additive-only proto evolution** — new event kinds/fields only; the Task-0 lock test + the
|
||||
Task-7 `default: throw` are the guard. Never renumber a tag.
|
||||
- **Deferred-channel honesty** — redundancy-state / fleet-status / deployment-acks are out by design,
|
||||
documented in Task 10; a future reviewer must not read "seven topics" in the program doc and think
|
||||
Phase 5 missed three.
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"planPath": "docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md",
|
||||
"program": "per-cluster-mesh",
|
||||
"phase": 5,
|
||||
"branch": "feat/mesh-phase5",
|
||||
"tasks": [
|
||||
{"id": 0, "subject": "Task 0: telemetry.proto contract + codegen + contract-lock test", "classification": "standard", "status": "completed", "commit": "a845a6d2", "parallelizableWith": [1, 2]},
|
||||
{"id": 1, "subject": "Task 1: Telemetry/TelemetryDial options + fail-closed validators", "classification": "small", "status": "completed", "commit": "53ae0100", "parallelizableWith": [0, 2]},
|
||||
{"id": 2, "subject": "Task 2: node-local telemetry hub (snapshot-replay + drop-oldest)", "classification": "standard", "status": "completed", "commit": "9882b1d2+7bdb6d00", "parallelizableWith": [0, 1]},
|
||||
{"id": 3, "subject": "Task 3: tap the 4 publish seams into the hub (DPS intact)", "classification": "standard", "status": "completed", "commit": "8e5090ed+909d7535", "blockedBy": [2]},
|
||||
{"id": 4, "subject": "Task 4: node-side TelemetryStreamService (hub -> server-streaming)","classification": "high-risk", "status": "completed", "commit": "e742fee4+2e70ef88", "blockedBy": [0, 2]},
|
||||
{"id": 5, "subject": "Task 5: fail-closed bearer interceptor for the telemetry stream", "classification": "small", "status": "completed", "commit": "50f1620d", "blockedBy": [1]},
|
||||
{"id": 6, "subject": "Task 6: host the telemetry gRPC server on driver nodes (Kestrel)", "classification": "standard", "status": "completed", "commit": "f131c1cc+af9c9d78", "blockedBy": [1, 4, 5]},
|
||||
{"id": 7, "subject": "Task 7: central telemetry dialer client + proto<->domain round-trip","classification": "high-risk","status": "completed", "commit": "c78034f0+a279a43e", "blockedBy": [0]},
|
||||
{"id": 8, "subject": "Task 8: central telemetry dial supervisor (discover+reconnect+sinks)","classification": "high-risk","status": "completed", "commit": "1104785c+84fa2e1e", "blockedBy": [1, 7]},
|
||||
{"id": 9, "subject": "Task 9: dark-switch central telemetry ingest (Dps bridges | Grpc)", "classification": "standard", "status": "completed", "commit": "d15c613e", "blockedBy": [8]},
|
||||
{"id": 10, "subject": "Task 10: docs — Telemetry.md, config, supersede §6.3, program status","classification": "small", "status": "completed", "commit": "e99ea40e", "parallelizableWith": [11], "blockedBy": [9]},
|
||||
{"id": 11, "subject": "Task 11: rig telemetry ports + shared key + ClusterNode.GrpcPort seed","classification": "small", "status": "completed", "commit": "ffb75b78", "note": "port 4056, docker compose config validated", "parallelizableWith": [10], "blockedBy": [6, 9]},
|
||||
{"id": 12, "subject": "Task 12: live gate (panels green over gRPC + kill-and-reconnect)", "classification": "high-risk","status": "completed", "note": "PASSED — 12-stream mesh, pill live, kill/reconnect recovered; found GrpcPort upgrade gotcha. Record: 2026-07-23-mesh-phase5-live-gate.md", "blockedBy": [10, 11]}
|
||||
],
|
||||
"lastUpdated": "2026-07-23T00:00:00Z"
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
# Per-Cluster Mesh Phase 5 — Live Gate Record
|
||||
|
||||
**Date:** 2026-07-23
|
||||
**Branch:** `feat/mesh-phase5` (Phase-5 image built + rig recreated with Phase-5 code)
|
||||
**Rig:** local `docker-dev` — central-1/2 (`admin,driver`, fused), site-a-1/2, site-b-1/2 (`driver`).
|
||||
Telemetry served on **:4056** (h2c), shared key `telemetry-docker-dev-key`.
|
||||
|
||||
**Result: PASSED.** Every load-bearing exit-gate leg proven live. In `Telemetry:Mode=Grpc`, central
|
||||
dials each driver node's telemetry gRPC server, the full stream mesh forms, real telemetry traverses
|
||||
node→hub→gRPC→central→sink (AdminUI pill goes live), and kill-and-reconnect of a node recovers its
|
||||
stream automatically. In the default `Dps` mode the node hosts the server but central does not dial.
|
||||
|
||||
Evidence was gathered primarily at the **TCP-socket level** (`/proc/net/tcp6`, port 4056 = hex
|
||||
`0FD8`) because it is decisive and log-independent: an established stream is an established stream.
|
||||
|
||||
## What was verified
|
||||
|
||||
### Leg 0 — Dps baseline (headline: node always hosts, central does NOT dial)
|
||||
Fresh rig in the committed default (`Telemetry:Mode` unset ⇒ `Dps`). site-a-1 log:
|
||||
`Now listening on: http://[::]:4056` — the dedicated h2c telemetry listener binds in Dps mode too
|
||||
(the node **always** hosts the server). Socket check: site-a-1 has a **LISTEN** on `[::]:4056` and
|
||||
**zero** inbound ESTABLISHED — central does not dial in Dps mode. Cluster formed clean (central-1 sees
|
||||
central-2 + all four site nodes at `Up`), OPC UA up, drivers subscribed. Baseline correct.
|
||||
|
||||
### Leg 1 — Grpc flip → full stream mesh forms ✅
|
||||
Flipped via a `docker-compose.override.yml` (gate artifact, since removed) setting
|
||||
`TelemetryDial__Mode=Grpc` on the central pair and `Telemetry__Mode=Grpc` on all six driver nodes,
|
||||
then recreated. After the central pair discovered the (populated) `ClusterNode.GrpcPort` rows and
|
||||
dialed:
|
||||
|
||||
| Node | inbound telemetry streams on :4056 |
|
||||
|---|---|
|
||||
| central-1 | 2 |
|
||||
| central-2 | 2 |
|
||||
| site-a-1 | 2 |
|
||||
| site-a-2 | 2 |
|
||||
| site-b-1 | 2 |
|
||||
| site-b-2 | 2 |
|
||||
|
||||
Both central nodes each held **6 outbound** streams (one per driver node incl. self). **12 streams
|
||||
total** — the exact designed topology (both admins dial every driver node). Zero telemetry failures in
|
||||
the steady window. Dial happens **only** in Grpc mode (Leg 0 had zero); discovery is DB-sourced from
|
||||
`ClusterNode` (Host + GrpcPort), matching Phase 1 + the rig seed.
|
||||
|
||||
### Leg 2 — data path end-to-end (AdminUI pill live) ✅
|
||||
`http://localhost:9200/alerts` connection pill read **"live"**. This is the load-bearing data-path
|
||||
proof: the dial supervisor marks a node **Connected — and flips both broadcasters' `SetConnected(true)`
|
||||
— only on its FIRST received telemetry event**. So "live" means a real telemetry event (the
|
||||
5 s-periodic `driver-resilience-status`) traversed the node's publish seam → node-local hub → gRPC
|
||||
stream → central dial supervisor → `TelemetryReceived` → the in-process sink the Blazor panel reads.
|
||||
Transport *and* payload proven, not just a connected socket. (The panel's static "…from the cluster's
|
||||
DPS topic" caption is now stale wording — cosmetic, the broadcaster seam is unchanged, only its
|
||||
upstream swapped; noted as a doc-copy follow-up, non-blocking.)
|
||||
|
||||
### Leg 3 — kill-and-reconnect recovers every stream ✅
|
||||
`docker kill otopcua-dev-site-b-1-1`:
|
||||
- central-1 logged `Telemetry stream to node site-b-1:4053 failed (failure streak 1); reconnecting`.
|
||||
- central-1 outbound streams dropped 6 → 5; the pill stayed **live** (the other 5 streams carried the
|
||||
panel — a single node loss does not blank the fleet view).
|
||||
|
||||
`docker start otopcua-dev-site-b-1-1`: within **5 s** of the node rejoining, site-b-1 was back to **2
|
||||
inbound** and central-1 back to **6 outbound**, with **zero** ongoing telemetry failures. The
|
||||
generation-stamped reconnect recovered the stream automatically. (The same recovery was independently
|
||||
observed as a transient at central startup, when central briefly dialed its own/peer server before it
|
||||
finished booting — failure-streak-1 → reconnect → 6 streams.)
|
||||
|
||||
### Auth — covered by the Task 6 integration test (not re-probed live)
|
||||
`TelemetryListenerTests` (Host.IntegrationTests) already proves the mapped, interceptor-gated endpoint
|
||||
with two real gRPC calls: **no bearer ⇒ `PermissionDenied`** (which can only originate in
|
||||
`TelemetryStreamAuthInterceptor` on a *mapped* service — an unmapped service returns `Unimplemented`),
|
||||
and correct bearer ⇒ a clean stream. The rig ran with matching keys throughout; a key mismatch would
|
||||
have surfaced as `PermissionDenied` reconnect-spam (it did not).
|
||||
|
||||
## Finding — `ClusterNode.GrpcPort` must be populated on an EXISTING deployment (upgrade gotcha)
|
||||
The first Grpc flip produced **no** dial connections. Root cause was **not** a code defect: all six
|
||||
`ClusterNode` rows had `GrpcPort = NULL`. The rig's SQL volume persisted from before this branch, and
|
||||
the seed is idempotent **INSERT-if-not-exists** — so it never backfilled the new nullable `GrpcPort`
|
||||
onto pre-existing rows (`AkkaPort` was non-null only because its Phase-1 column carries a NOT NULL
|
||||
default `4053`; `GrpcPort` is nullable with no default). The dial supervisor behaved **correctly**: it
|
||||
spawned in Grpc mode, ran discovery, and skipped all six nodes with the throttled
|
||||
`ClusterNode <id> has no GrpcPort … skipped … (further skips of this node are silent)` warning (the
|
||||
exact throttle added in Task 8's review fix). Populating `GrpcPort = 4056` and bouncing the central
|
||||
pair produced the full 12-stream mesh immediately.
|
||||
|
||||
**Operational takeaway (record in the upgrade notes):** a **fresh** install seeds `GrpcPort` correctly
|
||||
(the Phase-11 seed INSERT includes it), but an **existing** deployment upgrading to Phase 5 must
|
||||
populate `ClusterNode.GrpcPort` per node — via the AdminUI node edit (Phase 1 surfaced the field) or a
|
||||
data update — or the telemetry dial silently finds no targets (loud-but-throttled warning, last-known
|
||||
DPS panels unaffected since the flip is per-node). Optional rig follow-up: make the docker-dev seed
|
||||
UPSERT the port columns so a persisted volume backfills on re-run (not blocking; this volume is now
|
||||
populated).
|
||||
|
||||
## Cleanup
|
||||
`docker-compose.override.yml` removed; rig recreated to the `Dps` default. `ClusterNode.GrpcPort`
|
||||
left populated at 4056 (correct — matches the fresh-install seed intent).
|
||||
|
||||
## Verdict
|
||||
**Phase 5 live gate PASSED.** Driver nodes host the telemetry gRPC server; central dials in only in
|
||||
Grpc mode; the full stream mesh forms from DB-sourced discovery; real telemetry reaches the AdminUI
|
||||
panels; kill-and-reconnect recovers every stream. The one issue surfaced was a rig/upgrade data gap
|
||||
(`GrpcPort` backfill), not a Phase-5 code defect — and it validated the graceful null-`GrpcPort`
|
||||
handling live.
|
||||
@@ -48,6 +48,14 @@ public static class ServiceCollectionExtensions
|
||||
configuration, ConfigSourceOptions.SectionName);
|
||||
services.Configure<ConfigServeOptions>(configuration.GetSection(ConfigServeOptions.SectionName));
|
||||
|
||||
// Per-cluster mesh Phase 5: which transport carries a node's live-telemetry stream (node
|
||||
// serve side + central dial side). Validated at startup for the same reason ConfigSource is —
|
||||
// a Grpc-mode node with no listen port or no key produces silence, not an error.
|
||||
services.AddValidatedOptions<TelemetryOptions, TelemetryOptionsValidator>(
|
||||
configuration, TelemetryOptions.SectionName);
|
||||
services.AddValidatedOptions<TelemetryDialOptions, TelemetryDialOptionsValidator>(
|
||||
configuration, TelemetryDialOptions.SectionName);
|
||||
|
||||
services.AddSingleton<IClusterRoleInfo, ClusterRoleInfo>();
|
||||
|
||||
return services;
|
||||
|
||||
@@ -0,0 +1,191 @@
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using Microsoft.Extensions.Options;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
|
||||
/// <summary>
|
||||
/// Node-side selection of the transport a node's live-telemetry stream is carried over
|
||||
/// (per-cluster mesh Phase 5). The dark switch that lets a node stop publishing telemetry over
|
||||
/// the mesh-wide DistributedPubSub topic and instead serve it over a dedicated gRPC stream that
|
||||
/// does not require sharing a gossip ring with whoever is watching.
|
||||
/// </summary>
|
||||
public sealed class TelemetryOptions
|
||||
{
|
||||
/// <summary>Configuration section name.</summary>
|
||||
public const string SectionName = "Telemetry";
|
||||
|
||||
/// <summary>Publish telemetry over the mesh-wide DistributedPubSub topic — the default.</summary>
|
||||
public const string ModeDps = "Dps";
|
||||
|
||||
/// <summary>Serve telemetry over a dedicated gRPC stream.</summary>
|
||||
public const string ModeGrpc = "Grpc";
|
||||
|
||||
/// <summary>
|
||||
/// <see cref="ModeDps"/> (default) or <see cref="ModeGrpc"/>. Any other value fails host
|
||||
/// start.
|
||||
/// </summary>
|
||||
public string Mode { get; set; } = ModeDps;
|
||||
|
||||
/// <summary>
|
||||
/// Dedicated gRPC listen port for the telemetry stream. <c>0</c> (the default) disables it —
|
||||
/// nothing is bound. Required on a driver-role node under <see cref="ModeGrpc"/>.
|
||||
/// </summary>
|
||||
public int GrpcListenPort { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Shared bearer key the serve-side interceptor checks. Supply via the environment
|
||||
/// (<c>Telemetry__ApiKey</c>) — never commit it. Required under <see cref="ModeGrpc"/> on any
|
||||
/// roled node — fail-closed against hosting an un-keyed telemetry surface.
|
||||
/// </summary>
|
||||
public string ApiKey { get; set; } = string.Empty;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Central-side selection of how central dials a node for live telemetry (per-cluster mesh
|
||||
/// Phase 5). Mirrors <see cref="TelemetryOptions"/>'s mode but carries the dial-side knobs
|
||||
/// (contact refresh cadence, per-call timeout) instead of a listen port.
|
||||
/// </summary>
|
||||
public sealed class TelemetryDialOptions
|
||||
{
|
||||
/// <summary>Configuration section name.</summary>
|
||||
public const string SectionName = "TelemetryDial";
|
||||
|
||||
/// <summary>Read telemetry from the mesh-wide DistributedPubSub topic — the default.</summary>
|
||||
public const string ModeDps = "Dps";
|
||||
|
||||
/// <summary>Dial nodes' dedicated gRPC telemetry streams.</summary>
|
||||
public const string ModeGrpc = "Grpc";
|
||||
|
||||
/// <summary>
|
||||
/// <see cref="ModeDps"/> (default) or <see cref="ModeGrpc"/>. Any other value fails host
|
||||
/// start.
|
||||
/// </summary>
|
||||
public string Mode { get; set; } = ModeDps;
|
||||
|
||||
/// <summary>
|
||||
/// Shared bearer key; must equal a dialled node's <see cref="TelemetryOptions.ApiKey"/>.
|
||||
/// Supply via the environment (<c>TelemetryDial__ApiKey</c>) — never commit it. Required
|
||||
/// under <see cref="ModeGrpc"/>.
|
||||
/// </summary>
|
||||
public string ApiKey { get; set; } = string.Empty;
|
||||
|
||||
/// <summary>How often central refreshes its set of dialable node contacts, in seconds.</summary>
|
||||
public int ContactRefreshSeconds { get; set; } = 60;
|
||||
|
||||
/// <summary>Per-call deadline for a gRPC telemetry dial, in seconds.</summary>
|
||||
public int CallTimeoutSeconds { get; set; } = 30;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Fails the host at startup on a <see cref="TelemetryOptions"/> shape that would leave a node
|
||||
/// unable to serve its live-telemetry stream.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Like <see cref="ConfigSourceOptionsValidator"/>, every fault caught here otherwise surfaces as
|
||||
/// an <i>absence</i> — a telemetry consumer that simply never sees data from this node, with no
|
||||
/// stack trace and no failing node to point at; refusing to start is cheaper to diagnose.
|
||||
/// </remarks>
|
||||
public sealed class TelemetryOptionsValidator : IValidateOptions<TelemetryOptions>
|
||||
{
|
||||
private readonly IConfiguration _configuration;
|
||||
|
||||
/// <summary>
|
||||
/// DI-constructed by <c>AddValidatedOptions</c> (a plain <c>AddSingleton</c>), so a
|
||||
/// constructor dependency is safe here. <see cref="IConfiguration"/> — not
|
||||
/// <c>IClusterRoleInfo</c> — is the source of this node's roles: <c>IClusterRoleInfo</c>'s
|
||||
/// implementation needs the <c>ActorSystem</c>, which does not exist yet at
|
||||
/// <c>ValidateOnStart</c> time.
|
||||
/// </summary>
|
||||
public TelemetryOptionsValidator(IConfiguration configuration)
|
||||
{
|
||||
_configuration = configuration;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public ValidateOptionsResult Validate(string? name, TelemetryOptions options)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
|
||||
var errors = new List<string>();
|
||||
|
||||
var isDps = string.Equals(options.Mode, TelemetryOptions.ModeDps, StringComparison.OrdinalIgnoreCase);
|
||||
var isGrpc = string.Equals(options.Mode, TelemetryOptions.ModeGrpc, StringComparison.OrdinalIgnoreCase);
|
||||
|
||||
if (!isDps && !isGrpc)
|
||||
{
|
||||
errors.Add(
|
||||
$"{TelemetryOptions.SectionName}:{nameof(TelemetryOptions.Mode)} is '{options.Mode}'. "
|
||||
+ $"Expected '{TelemetryOptions.ModeDps}' or '{TelemetryOptions.ModeGrpc}'.");
|
||||
}
|
||||
|
||||
var roles = _configuration.GetSection("Cluster:Roles").Get<string[]>() ?? Array.Empty<string>();
|
||||
var isDriver = Array.IndexOf(roles, "driver") >= 0;
|
||||
|
||||
if (isGrpc)
|
||||
{
|
||||
if (isDriver && options.GrpcListenPort <= 0)
|
||||
{
|
||||
errors.Add(
|
||||
$"Cluster:Roles has 'driver' and {TelemetryOptions.SectionName}:"
|
||||
+ $"{nameof(TelemetryOptions.Mode)} is '{TelemetryOptions.ModeGrpc}', but "
|
||||
+ $"{TelemetryOptions.SectionName}:{nameof(TelemetryOptions.GrpcListenPort)} is "
|
||||
+ $"{options.GrpcListenPort}. A driver node in Grpc telemetry mode must set "
|
||||
+ $"{TelemetryOptions.SectionName}:{nameof(TelemetryOptions.GrpcListenPort)}.");
|
||||
}
|
||||
|
||||
if (roles.Length > 0 && string.IsNullOrEmpty(options.ApiKey))
|
||||
{
|
||||
errors.Add(
|
||||
$"{TelemetryOptions.SectionName}:{nameof(TelemetryOptions.ApiKey)} is empty. Under "
|
||||
+ $"{TelemetryOptions.ModeGrpc} the shared bearer key is the whole authentication "
|
||||
+ "boundary to this node's telemetry surface; without it every dial is rejected. "
|
||||
+ $"Supply it via the environment ({TelemetryOptions.SectionName}__ApiKey).");
|
||||
}
|
||||
}
|
||||
|
||||
return errors.Count == 0
|
||||
? ValidateOptionsResult.Success
|
||||
: ValidateOptionsResult.Fail(string.Join(" ", errors));
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Fails the host at startup on a <see cref="TelemetryDialOptions"/> shape that would leave
|
||||
/// central unable to dial a node's live-telemetry stream.
|
||||
/// </summary>
|
||||
public sealed class TelemetryDialOptionsValidator : IValidateOptions<TelemetryDialOptions>
|
||||
{
|
||||
/// <inheritdoc />
|
||||
public ValidateOptionsResult Validate(string? name, TelemetryDialOptions options)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
|
||||
var errors = new List<string>();
|
||||
|
||||
var isDps = string.Equals(
|
||||
options.Mode, TelemetryDialOptions.ModeDps, StringComparison.OrdinalIgnoreCase);
|
||||
var isGrpc = string.Equals(
|
||||
options.Mode, TelemetryDialOptions.ModeGrpc, StringComparison.OrdinalIgnoreCase);
|
||||
|
||||
if (!isDps && !isGrpc)
|
||||
{
|
||||
errors.Add(
|
||||
$"{TelemetryDialOptions.SectionName}:{nameof(TelemetryDialOptions.Mode)} is "
|
||||
+ $"'{options.Mode}'. Expected '{TelemetryDialOptions.ModeDps}' or "
|
||||
+ $"'{TelemetryDialOptions.ModeGrpc}'.");
|
||||
}
|
||||
|
||||
if (isGrpc && string.IsNullOrEmpty(options.ApiKey))
|
||||
{
|
||||
errors.Add(
|
||||
$"{TelemetryDialOptions.SectionName}:{nameof(TelemetryDialOptions.ApiKey)} is empty. "
|
||||
+ $"Under {TelemetryDialOptions.ModeGrpc} the shared bearer key is the whole "
|
||||
+ "authentication boundary to a node's telemetry surface; without it every dial is "
|
||||
+ $"rejected. Supply it via the environment ({TelemetryDialOptions.SectionName}__ApiKey).");
|
||||
}
|
||||
|
||||
return errors.Count == 0
|
||||
? ValidateOptionsResult.Success
|
||||
: ValidateOptionsResult.Fail(string.Join(" ", errors));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Protos.Telemetry.V1;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Commons.Protos;
|
||||
|
||||
/// <summary>
|
||||
/// The single source of truth for which <see cref="TelemetryEvent.EventOneofCase"/> variants the
|
||||
/// Phase 5 telemetry transport actually handles. Both the contract-lock test (which proves this
|
||||
/// list stays in lock-step with the generated oneof) and the later oneof↔domain converter
|
||||
/// reference this array, so adding a fifth channel to <c>telemetry.proto</c> forces a matching
|
||||
/// entry here or the build/test goes red.
|
||||
/// </summary>
|
||||
public static class TelemetryProtoContract
|
||||
{
|
||||
/// <summary>
|
||||
/// Exactly the four telemetry oneof cases mirrored from the domain records:
|
||||
/// alerts / script-logs / driver-health / driver-resilience-status.
|
||||
/// </summary>
|
||||
public static readonly TelemetryEvent.EventOneofCase[] HandledCases =
|
||||
[
|
||||
TelemetryEvent.EventOneofCase.AlarmTransition,
|
||||
TelemetryEvent.EventOneofCase.ScriptLog,
|
||||
TelemetryEvent.EventOneofCase.DriverHealth,
|
||||
TelemetryEvent.EventOneofCase.DriverResilience,
|
||||
];
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
syntax = "proto3";
|
||||
|
||||
package telemetry.v1;
|
||||
|
||||
option csharp_namespace = "ZB.MOM.WW.OtOpcUa.Commons.Protos.Telemetry.V1";
|
||||
|
||||
import "google/protobuf/timestamp.proto";
|
||||
|
||||
// Per-cluster mesh Phase 5: the four live-telemetry channels that used to fan out over
|
||||
// DistributedPubSub (alerts / script-logs / driver-health / driver-resilience-status) now travel over
|
||||
// ONE gRPC server-streaming contract. Central dials each driver node and opens Subscribe; the node
|
||||
// streams its own live telemetry as a sequence of TelemetryEvent envelopes.
|
||||
//
|
||||
// Additive-only field evolution: never renumber or reuse a tag; a pre-field peer must read a new
|
||||
// field's proto3 default correctly (that is why nullable domain fields are modelled with `optional`
|
||||
// for explicit presence). Locked by the contract test in the Commons test project.
|
||||
service TelemetryStreamService {
|
||||
rpc Subscribe(TelemetryStreamRequest) returns (stream TelemetryEvent);
|
||||
}
|
||||
|
||||
message TelemetryStreamRequest {
|
||||
string correlation_id = 1;
|
||||
}
|
||||
|
||||
message TelemetryEvent {
|
||||
string correlation_id = 1;
|
||||
oneof event {
|
||||
AlarmTransition alarm_transition = 2; // <- "alerts" / AlarmTransitionEvent
|
||||
ScriptLog script_log = 3; // <- "script-logs" / ScriptLogEntry
|
||||
DriverHealth driver_health = 4; // <- "driver-health" / DriverHealthChanged
|
||||
DriverResilienceStatus driver_resilience = 5; // <- "driver-resilience-status" / DriverResilienceStatusChanged
|
||||
}
|
||||
}
|
||||
|
||||
// Mirrors ZB.MOM.WW.OtOpcUa.Commons.Messages.Alerts.AlarmTransitionEvent.
|
||||
message AlarmTransition {
|
||||
string alarm_id = 1;
|
||||
string equipment_path = 2;
|
||||
string alarm_name = 3;
|
||||
string transition_kind = 4; // string in the record (Activated/Cleared/...); kept as string
|
||||
int32 severity = 5;
|
||||
string message = 6;
|
||||
string user = 7;
|
||||
google.protobuf.Timestamp timestamp_utc = 8;
|
||||
string alarm_type_name = 9; // record default "AlarmCondition"
|
||||
optional string comment = 10; // nullable in the record — presence distinguishes null from ""
|
||||
optional bool historize_to_aveva = 11; // bool? in the record — three states (null / true / false)
|
||||
repeated string referencing_equipment_paths = 12; // null in the record is treated as empty
|
||||
}
|
||||
|
||||
// Mirrors ZB.MOM.WW.OtOpcUa.Commons.Messages.Logging.ScriptLogEntry.
|
||||
message ScriptLog {
|
||||
string script_id = 1;
|
||||
string level = 2; // string in the record (Trace/Debug/Information/...); kept as string
|
||||
string message = 3;
|
||||
google.protobuf.Timestamp timestamp_utc = 4;
|
||||
optional string virtual_tag_id = 5; // nullable in the record
|
||||
optional string alarm_id = 6; // nullable in the record
|
||||
optional string equipment_id = 7; // nullable in the record
|
||||
}
|
||||
|
||||
// Mirrors ZB.MOM.WW.OtOpcUa.Commons.Messages.Drivers.DriverHealthChanged.
|
||||
message DriverHealth {
|
||||
string cluster_id = 1;
|
||||
string driver_instance_id = 2;
|
||||
string state = 3; // DriverState-as-string in the record; kept as string
|
||||
google.protobuf.Timestamp last_successful_read_utc = 4; // DateTime? — absent Timestamp encodes null
|
||||
optional string last_error = 5; // nullable in the record
|
||||
int32 error_count_5min = 6;
|
||||
google.protobuf.Timestamp published_utc = 7;
|
||||
}
|
||||
|
||||
// Mirrors ZB.MOM.WW.OtOpcUa.Commons.Messages.Drivers.DriverResilienceStatusChanged.
|
||||
message DriverResilienceStatus {
|
||||
string driver_instance_id = 1;
|
||||
string host_name = 2;
|
||||
bool breaker_open = 3;
|
||||
int32 consecutive_failures = 4;
|
||||
int32 current_in_flight = 5;
|
||||
google.protobuf.Timestamp last_breaker_open_utc = 6; // DateTime? — absent Timestamp encodes null
|
||||
google.protobuf.Timestamp last_sampled_utc = 7;
|
||||
google.protobuf.Timestamp published_utc = 8;
|
||||
}
|
||||
@@ -27,6 +27,7 @@
|
||||
|
||||
<ItemGroup>
|
||||
<Protobuf Include="Protos\deployment_artifact.proto" GrpcServices="Both"/>
|
||||
<Protobuf Include="Protos\telemetry.proto" GrpcServices="Both"/>
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
|
||||
@@ -1,7 +1,14 @@
|
||||
using Akka.Actor;
|
||||
using Akka.Hosting;
|
||||
using Microsoft.AspNetCore.SignalR;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.OtOpcUa.AdminUI.Telemetry;
|
||||
using ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration;
|
||||
using ZB.MOM.WW.OtOpcUa.ControlPlane.Telemetry;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.AdminUI.Hubs;
|
||||
|
||||
@@ -12,6 +19,7 @@ public static class HubServiceCollectionExtensions
|
||||
public const string ScriptLogSignalRBridgeName = "script-log-signalr-bridge";
|
||||
public const string DriverStatusSignalRBridgeName = "driver-status-signalr-bridge";
|
||||
public const string DriverResilienceStatusBridgeName = "driver-resilience-status-bridge";
|
||||
public const string TelemetryDialSupervisorName = "telemetry-dial-supervisor";
|
||||
|
||||
/// <summary>
|
||||
/// Registers the in-process live-push services the AdminUI's Blazor Server panels read
|
||||
@@ -54,30 +62,69 @@ public static class HubServiceCollectionExtensions
|
||||
{
|
||||
builder.WithActors((system, registry, resolver) =>
|
||||
{
|
||||
// Fleet-status always stays on DPS in both modes (deferred / out of Phase 5 scope) — it is
|
||||
// never gated by the telemetry dark switch.
|
||||
var fleetHub = resolver.GetService<IHubContext<FleetStatusHub>>();
|
||||
var fleetBridge = system.ActorOf(FleetStatusSignalRBridge.Props(fleetHub), FleetStatusSignalRBridgeName);
|
||||
registry.Register<FleetStatusSignalRBridgeKey>(fleetBridge);
|
||||
|
||||
var alertHub = resolver.GetService<IHubContext<AlertHub>>();
|
||||
// The four telemetry sinks are registered identically in both modes; only the upstream that
|
||||
// feeds them swaps (DPS bridges vs. the gRPC dial supervisor). Resolve them once, above the
|
||||
// branch, so both paths feed the SAME singletons the Blazor panels read.
|
||||
var alertBroadcaster = resolver.GetService<IInProcessBroadcaster<Commons.Messages.Alerts.AlarmTransitionEvent>>();
|
||||
var alertBridge = system.ActorOf(AlertSignalRBridge.Props(alertHub, alertBroadcaster), AlertSignalRBridgeName);
|
||||
registry.Register<AlertSignalRBridgeKey>(alertBridge);
|
||||
|
||||
var scriptLogHub = resolver.GetService<IHubContext<ScriptLogHub>>();
|
||||
var scriptLogBroadcaster = resolver.GetService<IInProcessBroadcaster<Commons.Messages.Logging.ScriptLogEntry>>();
|
||||
var scriptLogBridge = system.ActorOf(ScriptLogSignalRBridge.Props(scriptLogHub, scriptLogBroadcaster), ScriptLogSignalRBridgeName);
|
||||
registry.Register<ScriptLogSignalRBridgeKey>(scriptLogBridge);
|
||||
|
||||
var driverStatusHub = resolver.GetService<IHubContext<DriverStatusHub>>();
|
||||
var driverStatusStore = resolver.GetService<IDriverStatusSnapshotStore>();
|
||||
var driverStatusBridge = system.ActorOf(DriverStatusSignalRBridge.Props(driverStatusHub, driverStatusStore), DriverStatusSignalRBridgeName);
|
||||
registry.Register<DriverStatusSignalRBridgeKey>(driverStatusBridge);
|
||||
|
||||
// Resilience-status bridge: DPS topic -> in-process store (no SignalR hub — the panel reads
|
||||
// the store directly, and resilience has no browser-JS consumer).
|
||||
var resilienceStore = resolver.GetService<IDriverResilienceStatusStore>();
|
||||
var resilienceBridge = system.ActorOf(DriverResilienceStatusBridge.Props(resilienceStore), DriverResilienceStatusBridgeName);
|
||||
registry.Register<DriverResilienceStatusBridgeKey>(resilienceBridge);
|
||||
|
||||
// Phase 5 dark switch. Absent options ⇒ Dps (today's behaviour); case-insensitive.
|
||||
var telemetryMode = resolver.GetService<IOptions<TelemetryDialOptions>>()?.Value.Mode
|
||||
?? TelemetryDialOptions.ModeDps;
|
||||
|
||||
if (string.Equals(telemetryMode, TelemetryDialOptions.ModeGrpc, StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
// Grpc: central dials each enabled node's dedicated telemetry stream, feeding the SAME
|
||||
// four sinks the DPS bridges feed. No DPS telemetry bridges are spawned.
|
||||
var options = resolver.GetService<IOptions<TelemetryDialOptions>>()!.Value;
|
||||
var dbFactory = resolver.GetService<IDbContextFactory<OtOpcUaConfigDbContext>>();
|
||||
var loggerFactory = resolver.GetService<ILoggerFactory>();
|
||||
|
||||
var nodeSource = TelemetryNodeSource.Create(
|
||||
dbFactory!, loggerFactory!.CreateLogger(typeof(TelemetryNodeSource).FullName!));
|
||||
var dialLoop = TelemetryNodeSource.CreateDialLoop(
|
||||
options.ApiKey, loggerFactory.CreateLogger<TelemetryStreamClient>());
|
||||
|
||||
var supervisor = system.ActorOf(
|
||||
TelemetryDialSupervisor.Props(
|
||||
nodeSource,
|
||||
dialLoop,
|
||||
alertBroadcaster,
|
||||
scriptLogBroadcaster,
|
||||
driverStatusStore,
|
||||
resilienceStore,
|
||||
options),
|
||||
TelemetryDialSupervisorName);
|
||||
registry.Register<TelemetryDialSupervisorKey>(supervisor);
|
||||
}
|
||||
else
|
||||
{
|
||||
// Dps (default): the four DPS bridges subscribe their mesh-wide topics and feed the sinks.
|
||||
var alertHub = resolver.GetService<IHubContext<AlertHub>>();
|
||||
var alertBridge = system.ActorOf(AlertSignalRBridge.Props(alertHub, alertBroadcaster), AlertSignalRBridgeName);
|
||||
registry.Register<AlertSignalRBridgeKey>(alertBridge);
|
||||
|
||||
var scriptLogHub = resolver.GetService<IHubContext<ScriptLogHub>>();
|
||||
var scriptLogBridge = system.ActorOf(ScriptLogSignalRBridge.Props(scriptLogHub, scriptLogBroadcaster), ScriptLogSignalRBridgeName);
|
||||
registry.Register<ScriptLogSignalRBridgeKey>(scriptLogBridge);
|
||||
|
||||
var driverStatusHub = resolver.GetService<IHubContext<DriverStatusHub>>();
|
||||
var driverStatusBridge = system.ActorOf(DriverStatusSignalRBridge.Props(driverStatusHub, driverStatusStore), DriverStatusSignalRBridgeName);
|
||||
registry.Register<DriverStatusSignalRBridgeKey>(driverStatusBridge);
|
||||
|
||||
// Resilience-status bridge: DPS topic -> in-process store (no SignalR hub — the panel reads
|
||||
// the store directly, and resilience has no browser-JS consumer).
|
||||
var resilienceBridge = system.ActorOf(DriverResilienceStatusBridge.Props(resilienceStore), DriverResilienceStatusBridgeName);
|
||||
registry.Register<DriverResilienceStatusBridgeKey>(resilienceBridge);
|
||||
}
|
||||
});
|
||||
return builder;
|
||||
}
|
||||
@@ -89,3 +136,6 @@ public sealed class AlertSignalRBridgeKey { }
|
||||
public sealed class ScriptLogSignalRBridgeKey { }
|
||||
public sealed class DriverStatusSignalRBridgeKey { }
|
||||
public sealed class DriverResilienceStatusBridgeKey { }
|
||||
|
||||
/// <summary>Marker key for <see cref="ActorRegistry"/> lookup of the Grpc-mode telemetry dial supervisor.</summary>
|
||||
public sealed class TelemetryDialSupervisorKey { }
|
||||
|
||||
@@ -0,0 +1,499 @@
|
||||
using Akka.Actor;
|
||||
using Akka.Event;
|
||||
using ZB.MOM.WW.OtOpcUa.AdminUI.Hubs;
|
||||
using ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Alerts;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Drivers;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Logging;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.AdminUI.Telemetry;
|
||||
|
||||
/// <summary>
|
||||
/// A single dialable driver node: its logical id and the prior-knowledge
|
||||
/// <c>http://host:port</c> gRPC telemetry endpoint central connects to.
|
||||
/// </summary>
|
||||
/// <param name="NodeId">The node's stable logical id (the <c>ClusterNode.NodeId</c>).</param>
|
||||
/// <param name="Endpoint">The <c>http://{Host}:{GrpcPort}</c> h2c telemetry endpoint.</param>
|
||||
public sealed record TelemetryDialTarget(string NodeId, string Endpoint);
|
||||
|
||||
/// <summary>
|
||||
/// The dial-loop seam: opens one telemetry stream to <paramref name="target"/> and pumps mapped
|
||||
/// domain records to <paramref name="onMapped"/> until the stream ends or <paramref name="ct"/>
|
||||
/// fires, routing a transport failure to <paramref name="onError"/>. The production
|
||||
/// implementation wraps <c>TelemetryStreamClient</c> + <c>TelemetryProtoMapCentral</c>; tests
|
||||
/// substitute a fake that captures the callbacks and drives events/failures by hand.
|
||||
/// </summary>
|
||||
/// <param name="target">The node to dial.</param>
|
||||
/// <param name="correlationId">Per-generation stream correlation id.</param>
|
||||
/// <param name="onMapped">Sink for each mapped domain record (already projected from the wire).</param>
|
||||
/// <param name="onError">Invoked on a transport failure (the reconnect trigger).</param>
|
||||
/// <param name="ct">Cancels the stream (a normal, supervisor-initiated shutdown).</param>
|
||||
/// <returns>A task that completes when the stream ends or is cancelled.</returns>
|
||||
public delegate Task TelemetryDialLoop(
|
||||
TelemetryDialTarget target,
|
||||
string correlationId,
|
||||
Action<object> onMapped,
|
||||
Action<Exception> onError,
|
||||
CancellationToken ct);
|
||||
|
||||
/// <summary>
|
||||
/// Central-side supervisor of the Phase 5 telemetry dials. Maintains one reconnecting dialer per
|
||||
/// enabled, non-maintenance driver node and feeds every received record to the same four AdminUI
|
||||
/// in-process sinks the DPS bridges feed today — so flipping central from
|
||||
/// <c>TelemetryDial:Mode = Dps</c> to <c>Grpc</c> is transparent to the Blazor panels.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Discover → dial → reconnect, forever.</b> On a periodic refresh the supervisor resolves
|
||||
/// the current dialable node set (via the injected <see cref="Func{TResult}"/> node source),
|
||||
/// starts a dialer for each new node id, and stops+removes a dialer whose node id has left the
|
||||
/// set. A dialer whose stream drops reconnects indefinitely (first retry immediate, then a
|
||||
/// fixed backoff) — this is an observability plane, not a data plane, so it never gives up.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>All state mutation is on the actor thread.</b> The background dialer tasks only ever
|
||||
/// <c>Self.Tell</c> — they never touch the dialer table, the connection pill, or a generation
|
||||
/// counter directly. Each dialer carries a monotonic <em>generation</em>; a message tagged with
|
||||
/// a stale generation (a late event or a late failure from a superseded stream) is dropped, so a
|
||||
/// reconnect can neither double-fire nor route a stale record.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class TelemetryDialSupervisor : ReceiveActor, IWithTimers
|
||||
{
|
||||
private const string RefreshTimerKey = "telemetry-node-refresh";
|
||||
private static readonly TimeSpan ReconnectBackoff = TimeSpan.FromSeconds(5);
|
||||
|
||||
private readonly Func<Task<IReadOnlyList<TelemetryDialTarget>>> _nodeSource;
|
||||
private readonly TelemetryDialLoop _dialLoop;
|
||||
private readonly IInProcessBroadcaster<AlarmTransitionEvent> _alarmBroadcaster;
|
||||
private readonly IInProcessBroadcaster<ScriptLogEntry> _scriptBroadcaster;
|
||||
private readonly IDriverStatusSnapshotStore _healthStore;
|
||||
private readonly IDriverResilienceStatusStore _resilienceStore;
|
||||
private readonly TelemetryDialOptions _options;
|
||||
private readonly ILoggingAdapter _log = Context.GetLogger();
|
||||
|
||||
private readonly Dictionary<string, NodeDialer> _dialers = new(StringComparer.Ordinal);
|
||||
private readonly HashSet<string> _warnedUnroutedTypes = new(StringComparer.Ordinal);
|
||||
private bool _pillConnected;
|
||||
|
||||
/// <summary>Gets the timer scheduler driving the periodic refresh and per-node reconnect backoff.</summary>
|
||||
public ITimerScheduler Timers { get; set; } = null!;
|
||||
|
||||
/// <summary>Creates the props for the telemetry dial supervisor.</summary>
|
||||
/// <param name="nodeSource">Resolves the current dialable node set (off the actor thread).</param>
|
||||
/// <param name="dialLoop">Opens + pumps one node's telemetry stream (the reconnect unit).</param>
|
||||
/// <param name="alarmBroadcaster">Sink for alarm-transition records.</param>
|
||||
/// <param name="scriptBroadcaster">Sink for script-log records.</param>
|
||||
/// <param name="healthStore">Sink for driver-health snapshots.</param>
|
||||
/// <param name="resilienceStore">Sink for driver-resilience snapshots.</param>
|
||||
/// <param name="options">The bound central dial options (refresh cadence etc.).</param>
|
||||
/// <returns>The props.</returns>
|
||||
public static Props Props(
|
||||
Func<Task<IReadOnlyList<TelemetryDialTarget>>> nodeSource,
|
||||
TelemetryDialLoop dialLoop,
|
||||
IInProcessBroadcaster<AlarmTransitionEvent> alarmBroadcaster,
|
||||
IInProcessBroadcaster<ScriptLogEntry> scriptBroadcaster,
|
||||
IDriverStatusSnapshotStore healthStore,
|
||||
IDriverResilienceStatusStore resilienceStore,
|
||||
TelemetryDialOptions options) =>
|
||||
Akka.Actor.Props.Create(() => new TelemetryDialSupervisor(
|
||||
nodeSource, dialLoop, alarmBroadcaster, scriptBroadcaster, healthStore, resilienceStore, options));
|
||||
|
||||
/// <summary>Initializes a new instance of the <see cref="TelemetryDialSupervisor"/> class.</summary>
|
||||
/// <param name="nodeSource">Resolves the current dialable node set (off the actor thread).</param>
|
||||
/// <param name="dialLoop">Opens + pumps one node's telemetry stream (the reconnect unit).</param>
|
||||
/// <param name="alarmBroadcaster">Sink for alarm-transition records.</param>
|
||||
/// <param name="scriptBroadcaster">Sink for script-log records.</param>
|
||||
/// <param name="healthStore">Sink for driver-health snapshots.</param>
|
||||
/// <param name="resilienceStore">Sink for driver-resilience snapshots.</param>
|
||||
/// <param name="options">The bound central dial options (refresh cadence etc.).</param>
|
||||
public TelemetryDialSupervisor(
|
||||
Func<Task<IReadOnlyList<TelemetryDialTarget>>> nodeSource,
|
||||
TelemetryDialLoop dialLoop,
|
||||
IInProcessBroadcaster<AlarmTransitionEvent> alarmBroadcaster,
|
||||
IInProcessBroadcaster<ScriptLogEntry> scriptBroadcaster,
|
||||
IDriverStatusSnapshotStore healthStore,
|
||||
IDriverResilienceStatusStore resilienceStore,
|
||||
TelemetryDialOptions options)
|
||||
{
|
||||
_nodeSource = nodeSource;
|
||||
_dialLoop = dialLoop;
|
||||
_alarmBroadcaster = alarmBroadcaster;
|
||||
_scriptBroadcaster = scriptBroadcaster;
|
||||
_healthStore = healthStore;
|
||||
_resilienceStore = resilienceStore;
|
||||
_options = options;
|
||||
|
||||
Receive<RefreshNodes>(_ => HandleRefreshNodes());
|
||||
Receive<NodesResolved>(HandleNodesResolved);
|
||||
Receive<TelemetryReceived>(HandleTelemetryReceived);
|
||||
Receive<StreamStopped>(HandleStreamStopped);
|
||||
Receive<Reconnect>(HandleReconnect);
|
||||
|
||||
// A faulted node-source task pipes here. Log and wait for the next periodic refresh — a stale
|
||||
// node set is far better than crashing the supervisor and dropping every live dialer.
|
||||
Receive<Status.Failure>(f => _log.Warning(
|
||||
f.Cause,
|
||||
"Telemetry node source failed; the dialable node set was NOT refreshed and may be stale. "
|
||||
+ "Existing dialers keep running; the next periodic refresh retries"));
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void PreStart()
|
||||
{
|
||||
Timers.StartPeriodicTimer(
|
||||
RefreshTimerKey,
|
||||
new RefreshNodes(),
|
||||
TimeSpan.Zero,
|
||||
TimeSpan.FromSeconds(Math.Max(1, _options.ContactRefreshSeconds)));
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void PostStop()
|
||||
{
|
||||
foreach (var dialer in _dialers.Values)
|
||||
{
|
||||
dialer.Cancel();
|
||||
}
|
||||
|
||||
_dialers.Clear();
|
||||
}
|
||||
|
||||
private void HandleRefreshNodes()
|
||||
{
|
||||
Task<IReadOnlyList<TelemetryDialTarget>> task;
|
||||
try
|
||||
{
|
||||
task = _nodeSource();
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_log.Warning(ex, "Telemetry node source threw synchronously; skipping this refresh");
|
||||
return;
|
||||
}
|
||||
|
||||
task.PipeTo(
|
||||
Self,
|
||||
success: targets => new NodesResolved(targets),
|
||||
failure: ex => new Status.Failure(ex));
|
||||
}
|
||||
|
||||
private void HandleNodesResolved(NodesResolved msg)
|
||||
{
|
||||
var incoming = new Dictionary<string, TelemetryDialTarget>(StringComparer.Ordinal);
|
||||
foreach (var target in msg.Targets)
|
||||
{
|
||||
// Last write wins on a duplicate id; a malformed set is the node source's problem, not ours.
|
||||
incoming[target.NodeId] = target;
|
||||
}
|
||||
|
||||
// Drop dialers whose node is no longer present.
|
||||
foreach (var nodeId in _dialers.Keys.ToList())
|
||||
{
|
||||
if (!incoming.ContainsKey(nodeId))
|
||||
{
|
||||
StopDialer(nodeId);
|
||||
}
|
||||
}
|
||||
|
||||
// Start a dialer for each newly-present node; re-dial one whose endpoint moved. An existing
|
||||
// node whose endpoint is UNCHANGED keeps its dialer untouched (no churn — a restart would drop
|
||||
// its stream on every refresh).
|
||||
foreach (var (nodeId, target) in incoming)
|
||||
{
|
||||
if (_dialers.TryGetValue(nodeId, out var existing))
|
||||
{
|
||||
if (!string.Equals(existing.Target.Endpoint, target.Endpoint, StringComparison.Ordinal))
|
||||
{
|
||||
// A re-provisioned node moved host/port under the same id. Update the target in place
|
||||
// and re-dial — StartDialer bumps the SAME dialer's generation, so the old (dead)
|
||||
// endpoint's stream is cancelled AND its in-flight messages are dropped by the
|
||||
// generation guard. Replacing the dialer object would reset the generation and let a
|
||||
// late message from the old stream slip through.
|
||||
_log.Info(
|
||||
"Node {NodeId} telemetry endpoint changed {Old} -> {New}; re-dialing",
|
||||
nodeId, existing.Target.Endpoint, target.Endpoint);
|
||||
existing.Target = target;
|
||||
StartDialer(nodeId);
|
||||
}
|
||||
|
||||
continue;
|
||||
}
|
||||
|
||||
_dialers[nodeId] = new NodeDialer(target);
|
||||
StartDialer(nodeId);
|
||||
}
|
||||
}
|
||||
|
||||
private void HandleTelemetryReceived(TelemetryReceived msg)
|
||||
{
|
||||
if (!_dialers.TryGetValue(msg.NodeId, out var dialer) || msg.Generation != dialer.Generation)
|
||||
{
|
||||
// Superseded stream (a reconnect already advanced the generation) — drop, never route.
|
||||
return;
|
||||
}
|
||||
|
||||
if (!dialer.Connected)
|
||||
{
|
||||
dialer.Connected = true;
|
||||
dialer.FailureStreak = 0;
|
||||
RecomputePill();
|
||||
}
|
||||
|
||||
Route(msg.NodeId, msg.Record);
|
||||
}
|
||||
|
||||
private void HandleStreamStopped(StreamStopped msg)
|
||||
{
|
||||
if (!_dialers.TryGetValue(msg.NodeId, out var dialer) || msg.Generation != dialer.Generation)
|
||||
{
|
||||
// Late failure/end from a superseded stream — ignore, so a reconnect never double-fires.
|
||||
return;
|
||||
}
|
||||
|
||||
if (dialer.Connected)
|
||||
{
|
||||
dialer.Connected = false;
|
||||
RecomputePill();
|
||||
}
|
||||
|
||||
dialer.FailureStreak++;
|
||||
|
||||
if (msg.Error is not null && (dialer.FailureStreak == 1 || dialer.FailureStreak % 10 == 0))
|
||||
{
|
||||
_log.Warning(
|
||||
msg.Error,
|
||||
"Telemetry stream to node {NodeId} failed (failure streak {Streak}); reconnecting",
|
||||
msg.NodeId, dialer.FailureStreak);
|
||||
}
|
||||
else
|
||||
{
|
||||
_log.Debug(
|
||||
"Telemetry stream to node {NodeId} stopped (failure streak {Streak}); reconnecting",
|
||||
msg.NodeId, dialer.FailureStreak);
|
||||
}
|
||||
|
||||
if (dialer.FailureStreak == 1)
|
||||
{
|
||||
// First retry after a healthy stream is immediate.
|
||||
StartDialer(msg.NodeId);
|
||||
}
|
||||
else
|
||||
{
|
||||
// Subsequent retries back off. A single timer per node — a queued reconnect is replaced.
|
||||
Timers.StartSingleTimer(ReconnectTimerKey(msg.NodeId), new Reconnect(msg.NodeId), ReconnectBackoff);
|
||||
}
|
||||
}
|
||||
|
||||
private void HandleReconnect(Reconnect msg)
|
||||
{
|
||||
// The node may have been removed while the backoff timer was pending.
|
||||
if (_dialers.ContainsKey(msg.NodeId))
|
||||
{
|
||||
StartDialer(msg.NodeId);
|
||||
}
|
||||
}
|
||||
|
||||
private void StartDialer(string nodeId)
|
||||
{
|
||||
var dialer = _dialers[nodeId];
|
||||
|
||||
// Cancel any prior stream and advance the generation so its in-flight signals are ignored.
|
||||
dialer.Cancel();
|
||||
Timers.Cancel(ReconnectTimerKey(nodeId));
|
||||
|
||||
var generation = ++dialer.Generation;
|
||||
var cts = new CancellationTokenSource();
|
||||
dialer.Cts = cts;
|
||||
dialer.Connected = false;
|
||||
|
||||
var self = Self;
|
||||
var target = dialer.Target;
|
||||
var token = cts.Token;
|
||||
var correlationId = $"central-{nodeId}-{generation}";
|
||||
var dialLoop = _dialLoop;
|
||||
|
||||
// The background task ONLY ever Self.Tell()s — it never touches actor state.
|
||||
_ = Task.Run(async () =>
|
||||
{
|
||||
var faulted = false;
|
||||
try
|
||||
{
|
||||
await dialLoop(
|
||||
target,
|
||||
correlationId,
|
||||
onMapped: record => self.Tell(new TelemetryReceived(nodeId, generation, record)),
|
||||
onError: error =>
|
||||
{
|
||||
faulted = true;
|
||||
self.Tell(new StreamStopped(nodeId, generation, error));
|
||||
},
|
||||
token).ConfigureAwait(false);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
// Normal shutdown: our cancellation token fired. No reconnect.
|
||||
return;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
// The dial loop threw instead of routing to onError — treat as a transport failure.
|
||||
self.Tell(new StreamStopped(nodeId, generation, ex));
|
||||
return;
|
||||
}
|
||||
|
||||
// The loop returned. If it did not already fault and we did not cancel it, the server ended
|
||||
// the stream — reconnect. (A faulted loop already told StreamStopped with the error.)
|
||||
if (!faulted && !token.IsCancellationRequested)
|
||||
{
|
||||
self.Tell(new StreamStopped(nodeId, generation, Error: null));
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
private void StopDialer(string nodeId)
|
||||
{
|
||||
if (_dialers.Remove(nodeId, out var dialer))
|
||||
{
|
||||
dialer.Cancel();
|
||||
Timers.Cancel(ReconnectTimerKey(nodeId));
|
||||
RecomputePill();
|
||||
}
|
||||
}
|
||||
|
||||
private void Route(string nodeId, object record)
|
||||
{
|
||||
// Defensive: one poison record must never crash the supervisor and take every dialer with it.
|
||||
try
|
||||
{
|
||||
switch (record)
|
||||
{
|
||||
case AlarmTransitionEvent alarm:
|
||||
_alarmBroadcaster.Publish(alarm);
|
||||
break;
|
||||
case ScriptLogEntry script:
|
||||
_scriptBroadcaster.Publish(script);
|
||||
break;
|
||||
case DriverHealthChanged health:
|
||||
_healthStore.Upsert(health);
|
||||
break;
|
||||
case DriverResilienceStatusChanged resilience:
|
||||
_resilienceStore.Upsert(resilience);
|
||||
break;
|
||||
default:
|
||||
// Warn once per unexpected type — a version-skew event would otherwise Warn per
|
||||
// event, forever. Subsequent drops of the same type log at Debug.
|
||||
var typeName = record?.GetType().Name ?? "null";
|
||||
if (_warnedUnroutedTypes.Add(typeName))
|
||||
{
|
||||
_log.Warning(
|
||||
"Telemetry record of unrouted type {RecordType} from node {NodeId}; dropped "
|
||||
+ "(further drops of this type log at Debug)", typeName, nodeId);
|
||||
}
|
||||
else
|
||||
{
|
||||
_log.Debug(
|
||||
"Telemetry record of unrouted type {RecordType} from node {NodeId}; dropped",
|
||||
typeName, nodeId);
|
||||
}
|
||||
|
||||
break;
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_log.Warning(
|
||||
ex,
|
||||
"Failed to route a telemetry record of type {RecordType} from node {NodeId} to its sink",
|
||||
record?.GetType().Name ?? "null", nodeId);
|
||||
}
|
||||
}
|
||||
|
||||
private void RecomputePill()
|
||||
{
|
||||
var anyConnected = false;
|
||||
foreach (var dialer in _dialers.Values)
|
||||
{
|
||||
if (dialer.Connected)
|
||||
{
|
||||
anyConnected = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (anyConnected == _pillConnected)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
_pillConnected = anyConnected;
|
||||
_alarmBroadcaster.SetConnected(anyConnected);
|
||||
_scriptBroadcaster.SetConnected(anyConnected);
|
||||
}
|
||||
|
||||
private static string ReconnectTimerKey(string nodeId) => $"telemetry-reconnect-{nodeId}";
|
||||
|
||||
/// <summary>Per-node dialer bookkeeping. Mutated only on the actor thread.</summary>
|
||||
private sealed class NodeDialer(TelemetryDialTarget target)
|
||||
{
|
||||
/// <summary>The node's dial target. Reassigned in place when a known node's endpoint changes.</summary>
|
||||
public TelemetryDialTarget Target { get; set; } = target;
|
||||
|
||||
/// <summary>Monotonic stream generation; a message tagged with a stale value is dropped.</summary>
|
||||
public int Generation { get; set; }
|
||||
|
||||
/// <summary>Cancels the current stream's background task.</summary>
|
||||
public CancellationTokenSource? Cts { get; set; }
|
||||
|
||||
/// <summary>Whether the current stream has delivered at least one event.</summary>
|
||||
public bool Connected { get; set; }
|
||||
|
||||
/// <summary>Consecutive stream stops since the last successful event (drives immediate vs. backoff).</summary>
|
||||
public int FailureStreak { get; set; }
|
||||
|
||||
/// <summary>Cancels + disposes the current stream's token source, if any.</summary>
|
||||
public void Cancel()
|
||||
{
|
||||
var cts = Cts;
|
||||
Cts = null;
|
||||
if (cts is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
cts.Cancel();
|
||||
}
|
||||
catch (ObjectDisposedException)
|
||||
{
|
||||
// Already disposed — nothing to cancel.
|
||||
}
|
||||
|
||||
cts.Dispose();
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Timer tick: re-resolve the dialable node set.</summary>
|
||||
public sealed record RefreshNodes;
|
||||
|
||||
/// <summary>The node set resolved off the actor thread.</summary>
|
||||
/// <param name="Targets">The current dialable nodes.</param>
|
||||
public sealed record NodesResolved(IReadOnlyList<TelemetryDialTarget> Targets);
|
||||
|
||||
/// <summary>A mapped domain record received from a node's stream.</summary>
|
||||
/// <param name="NodeId">The source node id.</param>
|
||||
/// <param name="Generation">The dialer generation that produced it.</param>
|
||||
/// <param name="Record">The mapped domain record.</param>
|
||||
public sealed record TelemetryReceived(string NodeId, int Generation, object Record);
|
||||
|
||||
/// <summary>A node's stream stopped — transport failure (<paramref name="Error"/> set) or server-ended.</summary>
|
||||
/// <param name="NodeId">The node id whose stream stopped.</param>
|
||||
/// <param name="Generation">The dialer generation that stopped.</param>
|
||||
/// <param name="Error">The transport failure, or <see langword="null"/> when the server ended the stream.</param>
|
||||
public sealed record StreamStopped(string NodeId, int Generation, Exception? Error);
|
||||
|
||||
/// <summary>Backoff timer tick: start a fresh dialer for the node.</summary>
|
||||
/// <param name="NodeId">The node id to reconnect.</param>
|
||||
public sealed record Reconnect(string NodeId);
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration;
|
||||
using ZB.MOM.WW.OtOpcUa.ControlPlane.Telemetry;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.AdminUI.Telemetry;
|
||||
|
||||
/// <summary>
|
||||
/// Production seams for <see cref="TelemetryDialSupervisor"/>: the DB-backed node source and the
|
||||
/// gRPC dial loop. Kept out of the actor so the actor stays DB-free and gRPC-free (and therefore
|
||||
/// unit-testable with fakes).
|
||||
/// </summary>
|
||||
public static class TelemetryNodeSource
|
||||
{
|
||||
/// <summary>
|
||||
/// Builds the node source the supervisor polls: enabled, non-maintenance <c>ClusterNode</c>
|
||||
/// rows carrying a non-null <c>GrpcPort</c>, projected to <c>http://{Host}:{GrpcPort}</c>
|
||||
/// h2c endpoints. A row with a null <c>GrpcPort</c> exposes no telemetry surface yet and is
|
||||
/// skipped with a Warning — mirrors <c>CentralCommunicationActor.LoadContactsFromDb</c>'s
|
||||
/// enabled + non-maintenance read.
|
||||
/// </summary>
|
||||
/// <param name="dbFactory">Factory for the config database holding <c>ClusterNode</c> rows.</param>
|
||||
/// <param name="logger">Logger for skipped-row diagnostics.</param>
|
||||
/// <returns>An async delegate resolving the current dialable node set.</returns>
|
||||
public static Func<Task<IReadOnlyList<TelemetryDialTarget>>> Create(
|
||||
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
|
||||
ILogger logger)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(dbFactory);
|
||||
ArgumentNullException.ThrowIfNull(logger);
|
||||
|
||||
// A persistently null-GrpcPort node is skipped on every refresh (every ContactRefreshSeconds);
|
||||
// warn once per node id so it does not spam the log forever.
|
||||
var warnedNullPort = new HashSet<string>(StringComparer.Ordinal);
|
||||
|
||||
return async () =>
|
||||
{
|
||||
await using var db = await dbFactory.CreateDbContextAsync().ConfigureAwait(false);
|
||||
var rows = await db.ClusterNodes
|
||||
.AsNoTracking()
|
||||
.Where(n => n.Enabled && !n.MaintenanceMode)
|
||||
.Select(n => new { n.NodeId, n.Host, n.GrpcPort })
|
||||
.ToListAsync()
|
||||
.ConfigureAwait(false);
|
||||
|
||||
var targets = new List<TelemetryDialTarget>(rows.Count);
|
||||
foreach (var row in rows)
|
||||
{
|
||||
if (row.GrpcPort is null)
|
||||
{
|
||||
if (warnedNullPort.Add(row.NodeId))
|
||||
{
|
||||
logger.LogWarning(
|
||||
"ClusterNode {NodeId} has no GrpcPort; it exposes no telemetry stream and is "
|
||||
+ "skipped in this dial refresh (further skips of this node are silent)",
|
||||
row.NodeId);
|
||||
}
|
||||
|
||||
continue;
|
||||
}
|
||||
|
||||
targets.Add(new TelemetryDialTarget(row.NodeId, $"http://{row.Host}:{row.GrpcPort}"));
|
||||
}
|
||||
|
||||
return (IReadOnlyList<TelemetryDialTarget>)targets;
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Builds the production dial loop: one <see cref="TelemetryStreamClient"/> per stream, mapping
|
||||
/// every wire envelope via <see cref="TelemetryProtoMapCentral.MapEvent"/> before handing it to
|
||||
/// the supervisor. The client isolates a mapper throw (logs + continues) and routes only
|
||||
/// transport failures to <c>onError</c>, so the supervisor sees a poison event as "dropped",
|
||||
/// never as "node down".
|
||||
/// </summary>
|
||||
/// <param name="apiKey">Shared node bearer key sent on every dial.</param>
|
||||
/// <param name="clientLogger">Logger passed to each per-stream client (dropped-event diagnostics).</param>
|
||||
/// <returns>The dial-loop delegate the supervisor invokes per generation.</returns>
|
||||
public static TelemetryDialLoop CreateDialLoop(
|
||||
string apiKey,
|
||||
ILogger<TelemetryStreamClient>? clientLogger)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(apiKey);
|
||||
|
||||
return async (target, correlationId, onMapped, onError, ct) =>
|
||||
{
|
||||
using var client = new TelemetryStreamClient(target.Endpoint, apiKey, clientLogger);
|
||||
await client.RunAsync(
|
||||
correlationId,
|
||||
onEvent: evt => onMapped(TelemetryProtoMapCentral.MapEvent(evt)),
|
||||
onError: onError,
|
||||
ct).ConfigureAwait(false);
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,157 @@
|
||||
using Google.Protobuf.WellKnownTypes;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Alerts;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Drivers;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Logging;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Protos.Telemetry.V1;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Telemetry;
|
||||
|
||||
/// <summary>
|
||||
/// Central-side projection of a wire <see cref="TelemetryEvent"/> envelope back onto its domain
|
||||
/// record (per-cluster mesh Phase 5). The exact reverse of <c>TelemetryProtoMapNode</c>: it
|
||||
/// transcribes every proto field onto the matching domain field and selects the domain record from
|
||||
/// the envelope's <c>oneof</c> arm.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Timestamp presence is honoured.</b> A proto <c>google.protobuf.Timestamp</c> field is a
|
||||
/// message: when the node left it unset (a null nullable-<c>DateTime</c> on the domain side) the
|
||||
/// generated property is <see langword="null"/>, so the nullable domain fields map via
|
||||
/// <c>?.ToDateTime()</c> and an absent Timestamp becomes <see langword="null"/>. The
|
||||
/// non-nullable domain fields (published / sampled / transition timestamps) are always set by
|
||||
/// the node mapper, so they map via <see cref="Required"/> (which yields a
|
||||
/// <see cref="DateTimeKind.Utc"/> value).
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>A required Timestamp is defended, not assumed.</b> Across a mixed-version fleet an
|
||||
/// out-of-contract sender could leave a required Timestamp unset. <see cref="Required"/> throws
|
||||
/// a clear <see cref="InvalidOperationException"/> naming the kind + field instead of letting a
|
||||
/// bare <see cref="NullReferenceException"/> escape — the client's per-event isolation then logs
|
||||
/// it as a dropped malformed event and continues, because it is a code/version fault, never a
|
||||
/// transport fault.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>optional-string presence is honoured.</b> A proto3 <c>optional string</c> that the node
|
||||
/// left unset reads back as <see langword="null"/> (via the generated <c>Has…</c> presence),
|
||||
/// distinguishing null from the empty string; likewise the <c>optional bool</c>
|
||||
/// <c>historize_to_aveva</c> round-trips its three states (null / true / false).
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public static class TelemetryProtoMapCentral
|
||||
{
|
||||
/// <summary>
|
||||
/// Projects a wire <see cref="TelemetryEvent"/> onto its domain record, selecting the type from
|
||||
/// the populated <c>oneof</c> arm. This switch is the coverage guard: every
|
||||
/// <see cref="TelemetryProtoContract.HandledCases"/> value has an arm here, and an unset (or a
|
||||
/// future unmapped) case throws <see cref="NotSupportedException"/> so adding a fifth oneof case
|
||||
/// without a converter fails the coverage test.
|
||||
/// </summary>
|
||||
/// <param name="evt">The wire envelope to project.</param>
|
||||
/// <returns>The domain record for the populated arm.</returns>
|
||||
/// <exception cref="ArgumentNullException"><paramref name="evt"/> is null.</exception>
|
||||
/// <exception cref="NotSupportedException">The envelope carries no handled <c>oneof</c> arm.</exception>
|
||||
public static object MapEvent(TelemetryEvent evt)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(evt);
|
||||
|
||||
return evt.EventCase switch
|
||||
{
|
||||
TelemetryEvent.EventOneofCase.AlarmTransition => ToAlarm(evt.AlarmTransition),
|
||||
TelemetryEvent.EventOneofCase.ScriptLog => ToScript(evt.ScriptLog),
|
||||
TelemetryEvent.EventOneofCase.DriverHealth => ToHealth(evt.DriverHealth),
|
||||
TelemetryEvent.EventOneofCase.DriverResilience => ToResilience(evt.DriverResilience),
|
||||
_ => throw new NotSupportedException(
|
||||
$"TelemetryEvent carries no handled oneof arm (EventCase = {evt.EventCase})."),
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>Projects an <see cref="AlarmTransition"/> onto an <see cref="AlarmTransitionEvent"/>.</summary>
|
||||
/// <param name="msg">The proto sub-message.</param>
|
||||
/// <returns>The domain record.</returns>
|
||||
public static AlarmTransitionEvent ToAlarm(AlarmTransition msg)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(msg);
|
||||
|
||||
return new AlarmTransitionEvent(
|
||||
AlarmId: msg.AlarmId,
|
||||
EquipmentPath: msg.EquipmentPath,
|
||||
AlarmName: msg.AlarmName,
|
||||
TransitionKind: msg.TransitionKind,
|
||||
Severity: msg.Severity,
|
||||
Message: msg.Message,
|
||||
User: msg.User,
|
||||
TimestampUtc: Required(msg.TimestampUtc, "AlarmTransition", "timestamp_utc"),
|
||||
AlarmTypeName: msg.AlarmTypeName,
|
||||
Comment: msg.HasComment ? msg.Comment : null,
|
||||
HistorizeToAveva: msg.HasHistorizeToAveva ? msg.HistorizeToAveva : null,
|
||||
ReferencingEquipmentPaths: msg.ReferencingEquipmentPaths.ToList());
|
||||
}
|
||||
|
||||
/// <summary>Projects a <see cref="ScriptLog"/> onto a <see cref="ScriptLogEntry"/>.</summary>
|
||||
/// <param name="msg">The proto sub-message.</param>
|
||||
/// <returns>The domain record.</returns>
|
||||
public static ScriptLogEntry ToScript(ScriptLog msg)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(msg);
|
||||
|
||||
return new ScriptLogEntry(
|
||||
ScriptId: msg.ScriptId,
|
||||
Level: msg.Level,
|
||||
Message: msg.Message,
|
||||
TimestampUtc: Required(msg.TimestampUtc, "ScriptLog", "timestamp_utc"),
|
||||
VirtualTagId: msg.HasVirtualTagId ? msg.VirtualTagId : null,
|
||||
AlarmId: msg.HasAlarmId ? msg.AlarmId : null,
|
||||
EquipmentId: msg.HasEquipmentId ? msg.EquipmentId : null);
|
||||
}
|
||||
|
||||
/// <summary>Projects a <see cref="DriverHealth"/> onto a <see cref="DriverHealthChanged"/>.</summary>
|
||||
/// <param name="msg">The proto sub-message.</param>
|
||||
/// <returns>The domain record.</returns>
|
||||
public static DriverHealthChanged ToHealth(DriverHealth msg)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(msg);
|
||||
|
||||
return new DriverHealthChanged(
|
||||
ClusterId: msg.ClusterId,
|
||||
DriverInstanceId: msg.DriverInstanceId,
|
||||
State: msg.State,
|
||||
LastSuccessfulReadUtc: msg.LastSuccessfulReadUtc?.ToDateTime(),
|
||||
LastError: msg.HasLastError ? msg.LastError : null,
|
||||
ErrorCount5Min: msg.ErrorCount5Min,
|
||||
PublishedUtc: Required(msg.PublishedUtc, "DriverHealth", "published_utc"));
|
||||
}
|
||||
|
||||
/// <summary>Projects a <see cref="DriverResilienceStatus"/> onto a <see cref="DriverResilienceStatusChanged"/>.</summary>
|
||||
/// <param name="msg">The proto sub-message.</param>
|
||||
/// <returns>The domain record.</returns>
|
||||
public static DriverResilienceStatusChanged ToResilience(DriverResilienceStatus msg)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(msg);
|
||||
|
||||
return new DriverResilienceStatusChanged(
|
||||
DriverInstanceId: msg.DriverInstanceId,
|
||||
HostName: msg.HostName,
|
||||
BreakerOpen: msg.BreakerOpen,
|
||||
ConsecutiveFailures: msg.ConsecutiveFailures,
|
||||
CurrentInFlight: msg.CurrentInFlight,
|
||||
LastBreakerOpenUtc: msg.LastBreakerOpenUtc?.ToDateTime(),
|
||||
LastSampledUtc: Required(msg.LastSampledUtc, "DriverResilienceStatus", "last_sampled_utc"),
|
||||
PublishedUtc: Required(msg.PublishedUtc, "DriverResilienceStatus", "published_utc"));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Converts a <b>required</b> proto <see cref="Timestamp"/> to a UTC <see cref="DateTime"/>,
|
||||
/// throwing a clear <see cref="InvalidOperationException"/> naming the telemetry kind + proto
|
||||
/// field if an out-of-contract sender left it unset — rather than letting a bare
|
||||
/// <see cref="NullReferenceException"/> escape.
|
||||
/// </summary>
|
||||
/// <param name="ts">The required proto Timestamp (<see langword="null"/> when the sender omitted it).</param>
|
||||
/// <param name="kind">The telemetry sub-message name, for the diagnostic.</param>
|
||||
/// <param name="field">The proto field name, for the diagnostic.</param>
|
||||
/// <returns>The Timestamp as a <see cref="DateTimeKind.Utc"/> value.</returns>
|
||||
/// <exception cref="InvalidOperationException"><paramref name="ts"/> is unset.</exception>
|
||||
private static DateTime Required(Timestamp? ts, string kind, string field) =>
|
||||
ts is null
|
||||
? throw new InvalidOperationException($"telemetry {kind} missing required timestamp {field}")
|
||||
: ts.ToDateTime();
|
||||
}
|
||||
@@ -0,0 +1,180 @@
|
||||
using Grpc.Core;
|
||||
using Grpc.Net.Client;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Protos.Telemetry.V1;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Telemetry;
|
||||
|
||||
/// <summary>
|
||||
/// Central-side per-node dialer for the Phase 5 telemetry stream. Owns one h2c
|
||||
/// <see cref="GrpcChannel"/> to a single driver node's <c>TelemetryStreamService</c>, opens the
|
||||
/// server-streaming <c>Subscribe</c> call, and pumps every <see cref="TelemetryEvent"/> to a
|
||||
/// caller-supplied sink. The raw proto envelope is delivered as-is — the caller maps it via
|
||||
/// <see cref="TelemetryProtoMapCentral"/>.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Deliberately dumb.</b> Connect, pump, surface transport errors — no reconnect, no
|
||||
/// backoff, no buffering. A normal shutdown (the caller's <see cref="CancellationToken"/>
|
||||
/// firing, or the server ending the stream with <see cref="StatusCode.Cancelled"/>) completes
|
||||
/// <see cref="RunAsync"/> without calling <c>onError</c>.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>onError is strictly the transport-failure signal.</b> Only a failure from the wire itself
|
||||
/// — an exception thrown out of <c>call.ResponseStream</c> — reaches <c>onError</c>, and exactly
|
||||
/// once, so the supervisor's reconnect loop reacts only to a genuinely unreachable/faulted node.
|
||||
/// Everything that is NOT a transport fault is kept off that path:
|
||||
/// <list type="bullet">
|
||||
/// <item>
|
||||
/// <b>Programming faults</b> (null/empty <c>correlationId</c>, use-after-dispose) are
|
||||
/// validated <em>before</em> the pump and throw synchronously out of
|
||||
/// <see cref="RunAsync"/> — a caller bug must surface as a caller bug, never as "node
|
||||
/// down".
|
||||
/// </item>
|
||||
/// <item>
|
||||
/// <b>Consumer-callback faults</b> (a poison/unmappable event throwing inside
|
||||
/// <c>onEvent</c> — e.g. a future oneof case the mapper rejects, or a malformed required
|
||||
/// Timestamp) are caught <em>per event</em>, logged at Warning, and the pump
|
||||
/// <em>continues</em>. A code/version defect on one event must not tear down the stream
|
||||
/// or trigger a reconnect.
|
||||
/// </item>
|
||||
/// </list>
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>h2c + Bearer.</b> The endpoint is a prior-knowledge <c>http://host:port</c> address (no
|
||||
/// TLS); the shared node bearer key rides the <c>authorization</c> header on every call. The
|
||||
/// channel sets HTTP/2 keepalive pings so a long-idle telemetry stream is not silently dropped
|
||||
/// by an intermediary.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class TelemetryStreamClient : IDisposable
|
||||
{
|
||||
private readonly string _apiKey;
|
||||
private readonly GrpcChannel? _ownedChannel;
|
||||
private readonly TelemetryStreamService.TelemetryStreamServiceClient _client;
|
||||
private readonly ILogger<TelemetryStreamClient> _log;
|
||||
private bool _disposed;
|
||||
|
||||
/// <summary>
|
||||
/// Initializes a new instance of the <see cref="TelemetryStreamClient"/> class dialing a live
|
||||
/// driver node over h2c.
|
||||
/// </summary>
|
||||
/// <param name="endpoint">Prior-knowledge <c>http://host:port</c> address of the node's telemetry server.</param>
|
||||
/// <param name="apiKey">Shared node bearer key sent as <c>authorization: Bearer <key></c>.</param>
|
||||
/// <param name="log">Logger for dropped-malformed-event diagnostics; defaults to a no-op logger.</param>
|
||||
public TelemetryStreamClient(string endpoint, string apiKey, ILogger<TelemetryStreamClient>? log = null)
|
||||
{
|
||||
ArgumentException.ThrowIfNullOrEmpty(endpoint);
|
||||
ArgumentNullException.ThrowIfNull(apiKey);
|
||||
|
||||
_apiKey = apiKey;
|
||||
_log = log ?? NullLogger<TelemetryStreamClient>.Instance;
|
||||
_ownedChannel = GrpcChannel.ForAddress(endpoint, new GrpcChannelOptions
|
||||
{
|
||||
HttpHandler = new SocketsHttpHandler
|
||||
{
|
||||
KeepAlivePingDelay = TimeSpan.FromSeconds(15),
|
||||
KeepAlivePingTimeout = TimeSpan.FromSeconds(10),
|
||||
KeepAlivePingPolicy = HttpKeepAlivePingPolicy.Always,
|
||||
EnableMultipleHttp2Connections = true,
|
||||
},
|
||||
});
|
||||
_client = new TelemetryStreamService.TelemetryStreamServiceClient(_ownedChannel);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Initializes a new instance of the <see cref="TelemetryStreamClient"/> class over a
|
||||
/// caller-supplied client — the test seam. No channel is owned or disposed.
|
||||
/// </summary>
|
||||
/// <param name="client">The generated client to drive (typically a fake in tests).</param>
|
||||
/// <param name="apiKey">Shared node bearer key sent as <c>authorization: Bearer <key></c>.</param>
|
||||
/// <param name="log">Logger for dropped-malformed-event diagnostics; defaults to a no-op logger.</param>
|
||||
public TelemetryStreamClient(
|
||||
TelemetryStreamService.TelemetryStreamServiceClient client,
|
||||
string apiKey,
|
||||
ILogger<TelemetryStreamClient>? log = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(client);
|
||||
ArgumentNullException.ThrowIfNull(apiKey);
|
||||
|
||||
_client = client;
|
||||
_apiKey = apiKey;
|
||||
_log = log ?? NullLogger<TelemetryStreamClient>.Instance;
|
||||
_ownedChannel = null;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Opens the telemetry stream and pumps every envelope to <paramref name="onEvent"/> until the
|
||||
/// server ends the stream or <paramref name="ct"/> fires. A normal shutdown returns without
|
||||
/// invoking <paramref name="onError"/>; only a transport failure invokes
|
||||
/// <paramref name="onError"/> (exactly once). A consumer-callback exception on one event is
|
||||
/// logged and the pump continues; programming faults throw synchronously (see the type remarks).
|
||||
/// </summary>
|
||||
/// <param name="correlationId">Stream correlation id echoed on every envelope.</param>
|
||||
/// <param name="onEvent">Sink for each received raw <see cref="TelemetryEvent"/>.</param>
|
||||
/// <param name="onError">Invoked once on an abnormal transport failure (the reconnect trigger).</param>
|
||||
/// <param name="ct">Cancels the stream (normal shutdown).</param>
|
||||
/// <exception cref="ArgumentException"><paramref name="correlationId"/> is null or empty.</exception>
|
||||
/// <exception cref="ArgumentNullException"><paramref name="onEvent"/> or <paramref name="onError"/> is null.</exception>
|
||||
/// <exception cref="ObjectDisposedException">This client has been disposed.</exception>
|
||||
public async Task RunAsync(
|
||||
string correlationId,
|
||||
Action<TelemetryEvent> onEvent,
|
||||
Action<Exception> onError,
|
||||
CancellationToken ct)
|
||||
{
|
||||
// Programming faults are validated BEFORE the pump so they throw synchronously out of RunAsync
|
||||
// rather than funnelling to onError and masquerading as a transport fault (supervisor spin).
|
||||
ArgumentException.ThrowIfNullOrEmpty(correlationId);
|
||||
ArgumentNullException.ThrowIfNull(onEvent);
|
||||
ArgumentNullException.ThrowIfNull(onError);
|
||||
ObjectDisposedException.ThrowIf(_disposed, this);
|
||||
|
||||
var headers = new Metadata { { "authorization", $"Bearer {_apiKey}" } };
|
||||
|
||||
try
|
||||
{
|
||||
using var call = _client.Subscribe(
|
||||
new TelemetryStreamRequest { CorrelationId = correlationId }, headers, cancellationToken: ct);
|
||||
|
||||
await foreach (var evt in call.ResponseStream.ReadAllAsync(ct))
|
||||
{
|
||||
// Isolate the consumer callback: a poison/unmappable event is a code/version fault, not
|
||||
// "node down". Log and continue — never break the stream, never call onError.
|
||||
try
|
||||
{
|
||||
onEvent(evt);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_log.LogWarning(
|
||||
ex,
|
||||
"Dropping a malformed/unhandled telemetry event on stream {CorrelationId}; " +
|
||||
"the consumer callback threw. Continuing the stream (not a transport fault).",
|
||||
correlationId);
|
||||
}
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
// Normal shutdown: the caller's token fired.
|
||||
}
|
||||
catch (RpcException rex) when (rex.StatusCode == StatusCode.Cancelled)
|
||||
{
|
||||
// Normal shutdown: the server (or a token-driven cancel) ended the stream.
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
// The transport-failure signal — a later supervisor task decides retry/backoff.
|
||||
onError(ex);
|
||||
}
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public void Dispose()
|
||||
{
|
||||
_disposed = true;
|
||||
_ownedChannel?.Dispose();
|
||||
}
|
||||
}
|
||||
@@ -15,6 +15,9 @@
|
||||
<PackageReference Include="Akka.Cluster.Hosting"/>
|
||||
<PackageReference Include="Akka.Cluster.Tools"/>
|
||||
<PackageReference Include="Microsoft.EntityFrameworkCore"/>
|
||||
<!-- Phase 5: central dials each driver node's telemetry gRPC server (GrpcChannel h2c);
|
||||
Google.Protobuf + Grpc.Core.Api flow transitively from Commons (the generated client). -->
|
||||
<PackageReference Include="Grpc.Net.Client"/>
|
||||
<PackageReference Include="ZB.MOM.WW.Audit"/>
|
||||
</ItemGroup>
|
||||
|
||||
|
||||
@@ -0,0 +1,165 @@
|
||||
using System.Security.Cryptography;
|
||||
using System.Text;
|
||||
using Grpc.Core;
|
||||
using Grpc.Core.Interceptors;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
|
||||
|
||||
/// <summary>
|
||||
/// Gates the driver-node telemetry stream endpoint (per-cluster mesh Phase 5). The
|
||||
/// <c>TelemetryStreamService.Subscribe</c> RPC streams a node's live alerts / script-logs /
|
||||
/// driver-health / driver-resilience-status to whoever dials it; anything able to reach a driver
|
||||
/// node's telemetry port could otherwise pull that live feed out-of-band. This interceptor is the
|
||||
/// ONLY inbound auth on that endpoint.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Scoped by method path.</b> Only calls under
|
||||
/// <c>/telemetry.v1.TelemetryStreamService/</c> are gated; every other method on the shared
|
||||
/// gRPC pipeline (config-serve, LocalDb sync) passes through untouched, so this can share one
|
||||
/// <c>AddGrpc</c> registration with <see cref="ConfigServeAuthInterceptor"/> and
|
||||
/// <see cref="LocalDbSyncAuthInterceptor"/>.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Fail-closed.</b> With no <c>Telemetry:ApiKey</c> configured, NO dial is accepted,
|
||||
/// authenticated or not. Treating "no key" as "no auth required" would expose the live
|
||||
/// telemetry feed on exactly the default shape most nodes ship with. The caller dialing in
|
||||
/// must present the same <c>TelemetryDial:ApiKey</c> (the shared node key), so a key typo
|
||||
/// stops telemetry delivery outright rather than degrading to unauthenticated.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Comparison is <see cref="CryptographicOperations.FixedTimeEquals"/> over UTF-8 bytes, so a
|
||||
/// wrong key cannot be recovered byte-by-byte from response timing. Length differences are
|
||||
/// unavoidably observable and are not sensitive.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Exactly one public constructor.</b> <c>Grpc.AspNetCore</c> silently stops invoking an
|
||||
/// interceptor that has more than one public ctor — see
|
||||
/// <c>TelemetryStreamAuthInterceptorTests.HasExactlyOnePublicConstructor</c>.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class TelemetryStreamAuthInterceptor : Interceptor
|
||||
{
|
||||
private const string ServicePrefix = "/telemetry.v1.TelemetryStreamService/";
|
||||
private const string AuthorizationHeader = "authorization";
|
||||
private const string BearerPrefix = "Bearer ";
|
||||
|
||||
private readonly IOptions<TelemetryOptions> _options;
|
||||
private readonly ILogger<TelemetryStreamAuthInterceptor> _logger;
|
||||
|
||||
/// <summary>Creates the interceptor.</summary>
|
||||
/// <param name="options">Telemetry options; <c>ApiKey</c> is the expected bearer token.</param>
|
||||
/// <param name="logger">Logger for denial diagnostics.</param>
|
||||
public TelemetryStreamAuthInterceptor(
|
||||
IOptions<TelemetryOptions> options,
|
||||
ILogger<TelemetryStreamAuthInterceptor> logger)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
ArgumentNullException.ThrowIfNull(logger);
|
||||
|
||||
_options = options;
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public override Task<TResponse> UnaryServerHandler<TRequest, TResponse>(
|
||||
TRequest request,
|
||||
ServerCallContext context,
|
||||
UnaryServerMethod<TRequest, TResponse> continuation)
|
||||
{
|
||||
Authorize(context);
|
||||
return continuation(request, context);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public override Task DuplexStreamingServerHandler<TRequest, TResponse>(
|
||||
IAsyncStreamReader<TRequest> requestStream,
|
||||
IServerStreamWriter<TResponse> responseStream,
|
||||
ServerCallContext context,
|
||||
DuplexStreamingServerMethod<TRequest, TResponse> continuation)
|
||||
{
|
||||
Authorize(context);
|
||||
return continuation(requestStream, responseStream, context);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public override Task<TResponse> ClientStreamingServerHandler<TRequest, TResponse>(
|
||||
IAsyncStreamReader<TRequest> requestStream,
|
||||
ServerCallContext context,
|
||||
ClientStreamingServerMethod<TRequest, TResponse> continuation)
|
||||
{
|
||||
Authorize(context);
|
||||
return continuation(requestStream, context);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public override Task ServerStreamingServerHandler<TRequest, TResponse>(
|
||||
TRequest request,
|
||||
IServerStreamWriter<TResponse> responseStream,
|
||||
ServerCallContext context,
|
||||
ServerStreamingServerMethod<TRequest, TResponse> continuation)
|
||||
{
|
||||
Authorize(context);
|
||||
return continuation(request, responseStream, context);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Throws <see cref="RpcException"/> with <see cref="StatusCode.PermissionDenied"/> if this is
|
||||
/// a telemetry-stream call that does not carry the configured bearer token. Non-telemetry
|
||||
/// calls return immediately.
|
||||
/// </summary>
|
||||
private void Authorize(ServerCallContext context)
|
||||
{
|
||||
if (!context.Method.StartsWith(ServicePrefix, StringComparison.Ordinal))
|
||||
return;
|
||||
|
||||
var expected = _options.Value.ApiKey;
|
||||
if (string.IsNullOrEmpty(expected))
|
||||
{
|
||||
_logger.LogWarning(
|
||||
"Rejected a telemetry stream call to {Method}: no Telemetry:ApiKey is configured, so " +
|
||||
"the telemetry endpoint is closed. Configure the shared node key on this node and every " +
|
||||
"dialing caller.",
|
||||
context.Method);
|
||||
throw new RpcException(new Status(
|
||||
StatusCode.PermissionDenied,
|
||||
"Telemetry stream is not accepting connections: no API key is configured on this node."));
|
||||
}
|
||||
|
||||
var presented = ExtractBearerToken(context.RequestHeaders);
|
||||
if (presented is null || !FixedTimeEquals(presented, expected))
|
||||
{
|
||||
_logger.LogWarning(
|
||||
"Rejected a telemetry stream call to {Method}: {Reason}.",
|
||||
context.Method,
|
||||
presented is null ? "no bearer token presented" : "bearer token did not match");
|
||||
throw new RpcException(new Status(
|
||||
StatusCode.PermissionDenied,
|
||||
"Telemetry stream authentication failed."));
|
||||
}
|
||||
}
|
||||
|
||||
private static string? ExtractBearerToken(Metadata headers)
|
||||
{
|
||||
// gRPC lowercases header keys on the wire; compare case-insensitively anyway so a hand-built
|
||||
// Metadata in a test behaves the same as a real request.
|
||||
foreach (var entry in headers)
|
||||
{
|
||||
if (!string.Equals(entry.Key, AuthorizationHeader, StringComparison.OrdinalIgnoreCase))
|
||||
continue;
|
||||
|
||||
var value = entry.Value;
|
||||
if (value is not null && value.StartsWith(BearerPrefix, StringComparison.OrdinalIgnoreCase))
|
||||
return value[BearerPrefix.Length..];
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
private static bool FixedTimeEquals(string presented, string expected)
|
||||
=> CryptographicOperations.FixedTimeEquals(
|
||||
Encoding.UTF8.GetBytes(presented), Encoding.UTF8.GetBytes(expected));
|
||||
}
|
||||
@@ -4,6 +4,7 @@ using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Drivers;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Resilience;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Telemetry;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.Drivers;
|
||||
|
||||
@@ -23,22 +24,26 @@ public sealed class DriverResilienceStatusPublisherService : BackgroundService
|
||||
private readonly DriverResilienceStatusTracker _tracker;
|
||||
private readonly Func<ActorSystem> _actorSystemAccessor;
|
||||
private readonly ILogger<DriverResilienceStatusPublisherService> _logger;
|
||||
private readonly ITelemetryLocalHub _hub;
|
||||
private readonly TimeSpan _interval;
|
||||
|
||||
/// <summary>Initializes a new instance of <see cref="DriverResilienceStatusPublisherService"/>.</summary>
|
||||
/// <param name="tracker">The process-singleton resilience-status tracker to read each tick.</param>
|
||||
/// <param name="actorSystemAccessor">Lazy accessor for the Akka <see cref="ActorSystem"/> whose DPS mediator publishes the snapshots.</param>
|
||||
/// <param name="logger">Logger for publish diagnostics.</param>
|
||||
/// <param name="hub">The node-local live-telemetry hub each published snapshot is also emitted into (Phase 5).</param>
|
||||
/// <param name="interval">Publish cadence; defaults to 5 s when null.</param>
|
||||
public DriverResilienceStatusPublisherService(
|
||||
DriverResilienceStatusTracker tracker,
|
||||
Func<ActorSystem> actorSystemAccessor,
|
||||
ILogger<DriverResilienceStatusPublisherService> logger,
|
||||
ITelemetryLocalHub hub,
|
||||
TimeSpan? interval = null)
|
||||
{
|
||||
_tracker = tracker;
|
||||
_actorSystemAccessor = actorSystemAccessor;
|
||||
_logger = logger;
|
||||
_hub = hub;
|
||||
_interval = interval ?? DefaultInterval;
|
||||
}
|
||||
|
||||
@@ -77,7 +82,12 @@ public sealed class DriverResilienceStatusPublisherService : BackgroundService
|
||||
|
||||
var mediator = DistributedPubSub.Get(_actorSystemAccessor()).Mediator;
|
||||
foreach (var message in messages)
|
||||
{
|
||||
mediator.Tell(new Publish(DriverResilienceStatusChanged.TopicName, message));
|
||||
// Phase 5: fan the same snapshot into the node-local live-telemetry hub (no-op until a
|
||||
// gRPC client subscribes). The DPS publish above is unchanged — a strictly additive tap.
|
||||
_hub.Emit(new TelemetryItem.Resilience(message));
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
|
||||
@@ -0,0 +1,171 @@
|
||||
using Google.Protobuf.WellKnownTypes;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Alerts;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Drivers;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Logging;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Protos.Telemetry.V1;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Telemetry;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.Grpc;
|
||||
|
||||
/// <summary>
|
||||
/// Node-side projection of a domain <see cref="TelemetryItem"/> onto its wire
|
||||
/// <see cref="TelemetryEvent"/> envelope (per-cluster mesh Phase 5). The exact reverse of
|
||||
/// <c>telemetry.proto</c>: it transcribes every domain field onto the matching proto field and
|
||||
/// selects the envelope's <c>oneof</c> arm from the <see cref="TelemetryItem"/> subtype.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>DateTime → Timestamp is Kind-defensive.</b> <see cref="Timestamp.FromDateTime"/> throws
|
||||
/// unless the source <see cref="DateTime.Kind"/> is <see cref="DateTimeKind.Utc"/>. Producers
|
||||
/// use <see cref="DateTime.UtcNow"/> today, but a stray <see cref="DateTimeKind.Local"/> or
|
||||
/// <see cref="DateTimeKind.Unspecified"/> value must never crash the stream — so a Local value
|
||||
/// is converted and an Unspecified value is assumed already-UTC (matching the producer intent
|
||||
/// and preserving the instant on any machine).
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Nullable presence is preserved.</b> A null nullable-<c>DateTime</c> leaves the proto
|
||||
/// Timestamp unset (absent == null per the contract); a null <c>optional string</c> leaves the
|
||||
/// proto field unset so the generated <c>Has…</c> presence distinguishes null from "".
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public static class TelemetryProtoMapNode
|
||||
{
|
||||
/// <summary>Projects <paramref name="item"/> onto a wire <see cref="TelemetryEvent"/>.</summary>
|
||||
/// <param name="item">The domain telemetry item to project.</param>
|
||||
/// <param name="correlationId">Stream correlation id echoed on every envelope.</param>
|
||||
/// <returns>The wire envelope with the matching <c>oneof</c> arm populated.</returns>
|
||||
public static TelemetryEvent ToProto(TelemetryItem item, string correlationId)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(item);
|
||||
|
||||
return item switch
|
||||
{
|
||||
TelemetryItem.Alarm a => new TelemetryEvent
|
||||
{
|
||||
CorrelationId = correlationId,
|
||||
AlarmTransition = MapAlarm(a.E),
|
||||
},
|
||||
TelemetryItem.Script s => new TelemetryEvent
|
||||
{
|
||||
CorrelationId = correlationId,
|
||||
ScriptLog = MapScript(s.E),
|
||||
},
|
||||
TelemetryItem.Health h => new TelemetryEvent
|
||||
{
|
||||
CorrelationId = correlationId,
|
||||
DriverHealth = MapHealth(h.E),
|
||||
},
|
||||
TelemetryItem.Resilience r => new TelemetryEvent
|
||||
{
|
||||
CorrelationId = correlationId,
|
||||
DriverResilience = MapResilience(r.E),
|
||||
},
|
||||
_ => throw new ArgumentOutOfRangeException(
|
||||
nameof(item), item.GetType().Name, "Unknown TelemetryItem subtype"),
|
||||
};
|
||||
}
|
||||
|
||||
private static AlarmTransition MapAlarm(AlarmTransitionEvent e)
|
||||
{
|
||||
// Required (non-optional) proto string setters throw ArgumentNullException on null, and the
|
||||
// domain records carry no runtime null guard (nullable-ref annotations are compile-time only).
|
||||
// Coalesce every required string to "" so a stray null can never throw out of the pump and kill
|
||||
// the stream. The `optional` fields below keep their null-vs-"" presence guards.
|
||||
var msg = new AlarmTransition
|
||||
{
|
||||
AlarmId = e.AlarmId ?? "",
|
||||
EquipmentPath = e.EquipmentPath ?? "",
|
||||
AlarmName = e.AlarmName ?? "",
|
||||
TransitionKind = e.TransitionKind ?? "",
|
||||
Severity = e.Severity,
|
||||
Message = e.Message ?? "",
|
||||
User = e.User ?? "",
|
||||
TimestampUtc = ToUtcTimestamp(e.TimestampUtc),
|
||||
AlarmTypeName = e.AlarmTypeName ?? "",
|
||||
};
|
||||
|
||||
if (e.Comment is not null)
|
||||
msg.Comment = e.Comment;
|
||||
if (e.HistorizeToAveva is not null)
|
||||
msg.HistorizeToAveva = e.HistorizeToAveva.Value;
|
||||
msg.ReferencingEquipmentPaths.AddRange(e.ReferencingEquipmentPaths ?? Enumerable.Empty<string>());
|
||||
|
||||
return msg;
|
||||
}
|
||||
|
||||
private static ScriptLog MapScript(ScriptLogEntry e)
|
||||
{
|
||||
var msg = new ScriptLog
|
||||
{
|
||||
ScriptId = e.ScriptId ?? "",
|
||||
Level = e.Level ?? "",
|
||||
Message = e.Message ?? "",
|
||||
TimestampUtc = ToUtcTimestamp(e.TimestampUtc),
|
||||
};
|
||||
|
||||
if (e.VirtualTagId is not null)
|
||||
msg.VirtualTagId = e.VirtualTagId;
|
||||
if (e.AlarmId is not null)
|
||||
msg.AlarmId = e.AlarmId;
|
||||
if (e.EquipmentId is not null)
|
||||
msg.EquipmentId = e.EquipmentId;
|
||||
|
||||
return msg;
|
||||
}
|
||||
|
||||
private static DriverHealth MapHealth(DriverHealthChanged e)
|
||||
{
|
||||
var msg = new DriverHealth
|
||||
{
|
||||
ClusterId = e.ClusterId ?? "",
|
||||
DriverInstanceId = e.DriverInstanceId ?? "",
|
||||
State = e.State ?? "",
|
||||
ErrorCount5Min = e.ErrorCount5Min,
|
||||
PublishedUtc = ToUtcTimestamp(e.PublishedUtc),
|
||||
};
|
||||
|
||||
if (e.LastSuccessfulReadUtc is not null)
|
||||
msg.LastSuccessfulReadUtc = ToUtcTimestamp(e.LastSuccessfulReadUtc.Value);
|
||||
if (e.LastError is not null)
|
||||
msg.LastError = e.LastError;
|
||||
|
||||
return msg;
|
||||
}
|
||||
|
||||
private static DriverResilienceStatus MapResilience(DriverResilienceStatusChanged e)
|
||||
{
|
||||
var msg = new DriverResilienceStatus
|
||||
{
|
||||
DriverInstanceId = e.DriverInstanceId ?? "",
|
||||
HostName = e.HostName ?? "",
|
||||
BreakerOpen = e.BreakerOpen,
|
||||
ConsecutiveFailures = e.ConsecutiveFailures,
|
||||
CurrentInFlight = e.CurrentInFlight,
|
||||
LastSampledUtc = ToUtcTimestamp(e.LastSampledUtc),
|
||||
PublishedUtc = ToUtcTimestamp(e.PublishedUtc),
|
||||
};
|
||||
|
||||
if (e.LastBreakerOpenUtc is not null)
|
||||
msg.LastBreakerOpenUtc = ToUtcTimestamp(e.LastBreakerOpenUtc.Value);
|
||||
|
||||
return msg;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Converts a domain <see cref="DateTime"/> to a proto <see cref="Timestamp"/> without ever
|
||||
/// throwing on Kind: Utc is used as-is, Local is converted, and Unspecified is assumed
|
||||
/// already-UTC (the producer contract — <see cref="DateTime.UtcNow"/> — with the wall-clock
|
||||
/// ticks preserved so the instant round-trips).
|
||||
/// </summary>
|
||||
private static Timestamp ToUtcTimestamp(DateTime dt)
|
||||
{
|
||||
var utc = dt.Kind switch
|
||||
{
|
||||
DateTimeKind.Utc => dt,
|
||||
DateTimeKind.Local => dt.ToUniversalTime(),
|
||||
_ => DateTime.SpecifyKind(dt, DateTimeKind.Utc),
|
||||
};
|
||||
|
||||
return Timestamp.FromDateTime(utc);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,123 @@
|
||||
using Grpc.Core;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Protos.Telemetry.V1;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Telemetry;
|
||||
// The generated service container is itself named TelemetryStreamService; alias its nested server
|
||||
// base so this impl can keep the natural name without colliding with the generated type.
|
||||
using GeneratedServiceBase =
|
||||
ZB.MOM.WW.OtOpcUa.Commons.Protos.Telemetry.V1.TelemetryStreamService.TelemetryStreamServiceBase;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.Grpc;
|
||||
|
||||
/// <summary>
|
||||
/// Node-side live-telemetry streaming server (per-cluster mesh Phase 5). Central dials a driver
|
||||
/// node and opens <see cref="Subscribe"/>; the node attaches a fresh reader on its
|
||||
/// <see cref="ITelemetryLocalHub"/> and pumps this node's own telemetry
|
||||
/// (alarm-transitions / script-logs / driver-health / driver-resilience) to the caller as a
|
||||
/// sequence of <see cref="TelemetryEvent"/> envelopes until the client cancels or the
|
||||
/// max-lifetime cap fires.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The hub carries ONLY this node's telemetry, so a per-node stream never double-counts a peer's
|
||||
/// events. Fan-out is lossy under backpressure by design (the per-subscriber channel is
|
||||
/// bounded/DropOldest inside the hub). A simple process-wide concurrency cap sheds excess dials
|
||||
/// with <see cref="StatusCode.ResourceExhausted"/>, and every accepted stream is bounded by a
|
||||
/// linked max-lifetime CTS.
|
||||
/// </remarks>
|
||||
public sealed class TelemetryStreamGrpcService : GeneratedServiceBase
|
||||
{
|
||||
/// <summary>Process-wide cap on concurrently-open telemetry streams.</summary>
|
||||
private const int MaxConcurrentStreams = 100;
|
||||
|
||||
/// <summary>Upper bound on an accepted correlation id (defends against absurd inputs).</summary>
|
||||
private const int MaxCorrelationIdLength = 256;
|
||||
|
||||
/// <summary>Per-subscriber bounded-channel capacity requested from the hub.</summary>
|
||||
private const int SubscriptionCapacity = 1000;
|
||||
|
||||
/// <summary>Hard ceiling on a single stream's lifetime; the linked CTS cancels the pump when it fires.</summary>
|
||||
private static readonly TimeSpan MaxStreamLifetime = TimeSpan.FromHours(4);
|
||||
|
||||
/// <summary>Process-wide count of currently-open streams; mutated only via <see cref="Interlocked"/>.</summary>
|
||||
private static int _activeStreams;
|
||||
|
||||
private readonly ITelemetryLocalHub _hub;
|
||||
private readonly ILogger<TelemetryStreamGrpcService> _log;
|
||||
|
||||
/// <summary>Initializes a new instance of the <see cref="TelemetryStreamGrpcService"/> class.</summary>
|
||||
/// <param name="hub">The node-local telemetry fan-out hub this stream drains.</param>
|
||||
/// <param name="log">Logger.</param>
|
||||
public TelemetryStreamGrpcService(ITelemetryLocalHub hub, ILogger<TelemetryStreamGrpcService> log)
|
||||
{
|
||||
_hub = hub;
|
||||
_log = log;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public override async Task Subscribe(
|
||||
TelemetryStreamRequest request,
|
||||
IServerStreamWriter<TelemetryEvent> responseStream,
|
||||
ServerCallContext context)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(request);
|
||||
ArgumentNullException.ThrowIfNull(responseStream);
|
||||
ArgumentNullException.ThrowIfNull(context);
|
||||
|
||||
var correlationId = request.CorrelationId;
|
||||
if (string.IsNullOrWhiteSpace(correlationId) || correlationId.Length > MaxCorrelationIdLength)
|
||||
throw new RpcException(new Status(
|
||||
StatusCode.InvalidArgument,
|
||||
"correlation_id must be non-empty and at most 256 characters"));
|
||||
|
||||
// Concurrency cap. The increment is ALWAYS paired: on rejection we decrement here and throw
|
||||
// before entering the try; on acceptance the finally decrements exactly once. No path
|
||||
// decrements without a matching successful increment, and no path double-decrements.
|
||||
var active = Interlocked.Increment(ref _activeStreams);
|
||||
if (active > MaxConcurrentStreams)
|
||||
{
|
||||
Interlocked.Decrement(ref _activeStreams);
|
||||
_log.LogWarning(
|
||||
"Telemetry stream rejected (correlationId={CorrelationId}): {Active} open streams exceeds cap {Cap}",
|
||||
correlationId, active - 1, MaxConcurrentStreams);
|
||||
throw new RpcException(new Status(
|
||||
StatusCode.ResourceExhausted, "too many concurrent telemetry streams"));
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
using var lifetime = CancellationTokenSource.CreateLinkedTokenSource(context.CancellationToken);
|
||||
lifetime.CancelAfter(MaxStreamLifetime);
|
||||
using var sub = _hub.Subscribe(SubscriptionCapacity);
|
||||
|
||||
_log.LogDebug("Telemetry stream opened (correlationId={CorrelationId})", correlationId);
|
||||
|
||||
try
|
||||
{
|
||||
await foreach (var item in sub.Reader.ReadAllAsync(lifetime.Token).ConfigureAwait(false))
|
||||
await responseStream
|
||||
.WriteAsync(TelemetryProtoMapNode.ToProto(item, correlationId), lifetime.Token)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
// Normal termination: the client disconnected (context token) or the max-lifetime cap
|
||||
// fired (linked token). Neither is an error — swallow and let the stream close cleanly.
|
||||
}
|
||||
catch (Exception ex) when (ex is IOException or RpcException)
|
||||
{
|
||||
// Routine mid-stream disconnect: central's TelemetryStreamClient reconnects on ANY
|
||||
// non-Cancelled error, so a broken pipe / connection-reset here is expected churn, not a
|
||||
// fault. Swallow at Debug so grpc-dotnet doesn't log it at Error on every reconnect.
|
||||
// (A genuinely unexpected exception still escapes to surface as an error.)
|
||||
_log.LogDebug(
|
||||
ex, "Telemetry client disconnected mid-stream (correlationId={CorrelationId})", correlationId);
|
||||
}
|
||||
|
||||
_log.LogDebug("Telemetry stream closed (correlationId={CorrelationId})", correlationId);
|
||||
}
|
||||
finally
|
||||
{
|
||||
Interlocked.Decrement(ref _activeStreams);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -22,6 +22,7 @@ using ZB.MOM.WW.OtOpcUa.Core.Scripting;
|
||||
using ZB.MOM.WW.OtOpcUa.Host;
|
||||
using ZB.MOM.WW.OtOpcUa.Host.Configuration;
|
||||
using ZB.MOM.WW.OtOpcUa.Host.Drivers;
|
||||
using ZB.MOM.WW.OtOpcUa.Host.Grpc;
|
||||
using ZB.MOM.WW.OtOpcUa.Host.Engines;
|
||||
using ZB.MOM.WW.OtOpcUa.Host.Health;
|
||||
using ZB.MOM.WW.OtOpcUa.Host.Logging;
|
||||
@@ -34,6 +35,7 @@ using ZB.MOM.WW.OtOpcUa.OpcUaServer;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Historian;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Scripting;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Telemetry;
|
||||
using ZB.MOM.WW.OtOpcUa.OpcUaServer.Security;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime;
|
||||
using ZB.MOM.WW.Auth.Abstractions.Roles;
|
||||
@@ -310,7 +312,9 @@ if (hasDriver)
|
||||
? parsedLevel
|
||||
: LogEventLevel.Information;
|
||||
builder.Services.AddSingleton<IScriptLogPublisher>(sp =>
|
||||
new DpsScriptLogPublisher(() => sp.GetRequiredService<ActorSystem>()));
|
||||
new DpsScriptLogPublisher(
|
||||
() => sp.GetRequiredService<ActorSystem>(),
|
||||
sp.GetRequiredService<ITelemetryLocalHub>()));
|
||||
builder.Services.AddSingleton(sp => new ScriptRootLogger(
|
||||
ScriptRootLoggerFactory.Build(
|
||||
sp.GetRequiredService<IScriptLogPublisher>(), scriptLogFilePath, scriptLogTopicMinLevel, Serilog.Log.Logger)));
|
||||
@@ -413,46 +417,54 @@ builder.Services.AddOtOpcUaSecrets(builder.Configuration);
|
||||
builder.Services.AddOtOpcUaHealth(hasAdmin);
|
||||
builder.Services.AddOtOpcUaObservability(builder.Configuration);
|
||||
|
||||
// gRPC server plumbing shared by two endpoints: the LocalDb passive sync endpoint (driver-role,
|
||||
// mapped below on hasDriver && syncListenPort > 0) and the ConfigServe artifact endpoint
|
||||
// (admin-role, mapped on hasAdmin && configServeGrpcPort > 0). Each interceptor is scoped strictly
|
||||
// to its own service path, so both can share one pipeline and each is a harmless pass-through when
|
||||
// its service is unmapped. Both fail-closed with no ApiKey configured — the interceptor is the ONLY
|
||||
// inbound auth on either endpoint (the libraries verify nothing). Registered whenever either role is
|
||||
// present so an admin-only node still serves config and a driver-only node still gates sync.
|
||||
// gRPC server plumbing shared by three endpoints: the LocalDb passive sync endpoint (driver-role,
|
||||
// mapped below on hasDriver && syncListenPort > 0), the ConfigServe artifact endpoint (admin-role,
|
||||
// mapped on hasAdmin && configServeGrpcPort > 0), and the telemetry-stream endpoint (driver-role,
|
||||
// mapped on hasDriver && telemetryListenPort > 0, per-cluster mesh Phase 5). Each interceptor is
|
||||
// scoped strictly to its own service path, so all three can share one pipeline and each is a
|
||||
// harmless pass-through when its service is unmapped. All three fail-closed with no ApiKey
|
||||
// configured (LocalDbSyncAuthInterceptor, ConfigServeAuthInterceptor, TelemetryStreamAuthInterceptor)
|
||||
// — the interceptor is the ONLY inbound auth on any endpoint (the libraries verify nothing).
|
||||
// Registered whenever either role is present so an admin-only node still serves config and a
|
||||
// driver-only node still gates sync + telemetry.
|
||||
if (hasDriver || hasAdmin)
|
||||
{
|
||||
builder.Services.AddGrpc(o =>
|
||||
{
|
||||
if (hasDriver)
|
||||
{
|
||||
o.Interceptors.Add<LocalDbSyncAuthInterceptor>();
|
||||
o.Interceptors.Add<TelemetryStreamAuthInterceptor>();
|
||||
}
|
||||
if (hasAdmin)
|
||||
o.Interceptors.Add<ConfigServeAuthInterceptor>();
|
||||
});
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------------------------
|
||||
// Dedicated h2c listeners (both default-OFF): the LocalDb sync endpoint (driver) and the
|
||||
// ConfigServe artifact endpoint (admin, per-cluster mesh Phase 3).
|
||||
// Dedicated h2c listeners (all default-OFF): the LocalDb sync endpoint (driver), the ConfigServe
|
||||
// artifact endpoint (admin, per-cluster mesh Phase 3), and the telemetry-stream endpoint (driver,
|
||||
// per-cluster mesh Phase 5).
|
||||
//
|
||||
// DANGER: any explicit Kestrel Listen* call makes Kestrel IGNORE ASPNETCORE_URLS/urls ENTIRELY
|
||||
// (it logs "Overriding address(es)"). The host has no ConfigureKestrel today and binds solely via
|
||||
// that configuration, so adding a listener naively would silently unbind the AdminUI + deploy API
|
||||
// behind Traefik. Everything the host was already asked to serve is therefore re-bound explicitly
|
||||
// in the same block — and, critically, EXACTLY ONCE: a fused admin+driver node can have BOTH
|
||||
// in the same block — and, critically, EXACTLY ONCE: a fused admin+driver node can have ALL THREE
|
||||
// dedicated ports set, and re-applying the existing surface per-port would double-bind it and throw
|
||||
// "address already in use". So the existing surface is computed and applied once, then whichever of
|
||||
// the two dedicated ports are configured are added on top.
|
||||
// the three dedicated ports are configured are added on top.
|
||||
//
|
||||
// Each listener is HTTP/2-ONLY on purpose: both clients speak prior-knowledge h2c, which a cleartext
|
||||
// Each listener is HTTP/2-ONLY on purpose: every client speaks prior-knowledge h2c, which a cleartext
|
||||
// Http1AndHttp2 endpoint cannot negotiate (there is no ALPN without TLS). Hence dedicated ports
|
||||
// rather than multiplexing onto the main one.
|
||||
//
|
||||
// When both ports are 0 (the default) none of this runs and URL binding is untouched.
|
||||
// When all three ports are 0 (the default) none of this runs and URL binding is untouched.
|
||||
// ---------------------------------------------------------------------------------------------
|
||||
var syncListenPort = hasDriver ? LocalDbRegistration.SyncListenPort(builder.Configuration) : 0;
|
||||
var configServeGrpcPort = hasAdmin ? builder.Configuration.GetValue<int>("ConfigServe:GrpcListenPort") : 0;
|
||||
if (syncListenPort > 0 || configServeGrpcPort > 0)
|
||||
var telemetryListenPort = hasDriver ? builder.Configuration.GetValue<int>("Telemetry:GrpcListenPort") : 0;
|
||||
if (syncListenPort > 0 || configServeGrpcPort > 0 || telemetryListenPort > 0)
|
||||
{
|
||||
// Mirror Kestrel's own source precedence: URLS wins; else the HTTP_PORTS/HTTPS_PORTS bare-port
|
||||
// vars; else Kestrel's localhost:5000 default. Missing the HTTP_PORTS leg is not academic — the
|
||||
@@ -501,19 +513,22 @@ if (syncListenPort > 0 || configServeGrpcPort > 0)
|
||||
{
|
||||
Log.Error(
|
||||
"A dedicated h2c listener is requested (LocalDb:SyncListenPort={SyncPort}, " +
|
||||
"ConfigServe:GrpcListenPort={ConfigServePort}) but this host serves HTTPS endpoint(s) ({Urls}). " +
|
||||
"ConfigServe:GrpcListenPort={ConfigServePort}, Telemetry:GrpcListenPort={TelemetryPort}) but this " +
|
||||
"host serves HTTPS endpoint(s) ({Urls}). " +
|
||||
"Binding a dedicated listener requires re-binding every existing endpoint explicitly, and the " +
|
||||
"HTTPS certificate configuration cannot be replayed safely. BOTH dedicated listeners are DISABLED; " +
|
||||
"HTTPS certificate configuration cannot be replayed safely. ALL dedicated listeners are DISABLED; " +
|
||||
"terminate TLS upstream (as the docker-dev rig does) or leave these features off on this node.",
|
||||
syncListenPort, configServeGrpcPort, configuredUrls);
|
||||
syncListenPort, configServeGrpcPort, telemetryListenPort, configuredUrls);
|
||||
syncListenPort = 0;
|
||||
configServeGrpcPort = 0;
|
||||
telemetryListenPort = 0;
|
||||
}
|
||||
else
|
||||
{
|
||||
// Capture locals so the ConfigureKestrel closure does not observe later reassignment.
|
||||
var syncPortToBind = syncListenPort;
|
||||
var configServePortToBind = configServeGrpcPort;
|
||||
var telemetryPortToBind = telemetryListenPort;
|
||||
builder.WebHost.ConfigureKestrel(kestrel =>
|
||||
{
|
||||
foreach (var binding in existingBindings)
|
||||
@@ -523,12 +538,15 @@ if (syncListenPort > 0 || configServeGrpcPort > 0)
|
||||
kestrel.ListenAnyIP(syncPortToBind, o => o.Protocols = HttpProtocols.Http2);
|
||||
if (configServePortToBind > 0)
|
||||
kestrel.ListenAnyIP(configServePortToBind, o => o.Protocols = HttpProtocols.Http2);
|
||||
if (telemetryPortToBind > 0)
|
||||
kestrel.ListenAnyIP(telemetryPortToBind, o => o.Protocols = HttpProtocols.Http2);
|
||||
});
|
||||
|
||||
Log.Information(
|
||||
"Dedicated h2c listener(s) bound (sync=:{SyncPort}, config-serve=:{ConfigServePort}); " +
|
||||
"re-bound existing endpoint(s) {Urls}.",
|
||||
syncListenPort, configServeGrpcPort, configuredUrls ?? "http://localhost:5000 (Kestrel default)");
|
||||
"Dedicated h2c listener(s) bound (sync=:{SyncPort}, config-serve=:{ConfigServePort}, " +
|
||||
"telemetry=:{TelemetryPort}); re-bound existing endpoint(s) {Urls}.",
|
||||
syncListenPort, configServeGrpcPort, telemetryListenPort,
|
||||
configuredUrls ?? "http://localhost:5000 (Kestrel default)");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -579,6 +597,13 @@ if (hasAdmin && configServeGrpcPort > 0)
|
||||
app.MapGrpcService<DeploymentArtifactGrpcService>();
|
||||
}
|
||||
|
||||
// Driver-node live-telemetry stream (per-cluster mesh Phase 5). Central dials in; gated by
|
||||
// TelemetryStreamAuthInterceptor (fail-closed on empty Telemetry:ApiKey).
|
||||
if (hasDriver && telemetryListenPort > 0)
|
||||
{
|
||||
app.MapGrpcService<TelemetryStreamGrpcService>();
|
||||
}
|
||||
|
||||
app.MapOtOpcUaHealth();
|
||||
app.MapOtOpcUaMetrics();
|
||||
|
||||
|
||||
@@ -2,6 +2,7 @@ using Akka.Actor;
|
||||
using Akka.Cluster.Tools.PublishSubscribe;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Drivers;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Telemetry;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
|
||||
|
||||
@@ -17,10 +18,16 @@ public sealed class AkkaDriverHealthPublisher : IDriverHealthPublisher
|
||||
public const string TopicName = DriverHealthChanged.TopicName;
|
||||
|
||||
private readonly ActorSystem _system;
|
||||
private readonly ITelemetryLocalHub _hub;
|
||||
|
||||
/// <summary>Initializes a new instance of <see cref="AkkaDriverHealthPublisher"/>.</summary>
|
||||
/// <param name="system">The Akka actor system used to resolve the DPS mediator.</param>
|
||||
public AkkaDriverHealthPublisher(ActorSystem system) => _system = system;
|
||||
/// <param name="hub">The node-local live-telemetry hub each snapshot is also emitted into (Phase 5).</param>
|
||||
public AkkaDriverHealthPublisher(ActorSystem system, ITelemetryLocalHub hub)
|
||||
{
|
||||
_system = system;
|
||||
_hub = hub;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public void Publish(string clusterId, string driverInstanceId, DriverHealth health, int errorCount5Min)
|
||||
@@ -34,5 +41,8 @@ public sealed class AkkaDriverHealthPublisher : IDriverHealthPublisher
|
||||
errorCount5Min,
|
||||
DateTime.UtcNow);
|
||||
DistributedPubSub.Get(_system).Mediator.Tell(new Publish(TopicName, msg));
|
||||
// Phase 5: fan the same snapshot into the node-local live-telemetry hub (no-op until a gRPC
|
||||
// client subscribes). The DPS publish above is unchanged — the hub is a strictly additive tap.
|
||||
_hub.Emit(new TelemetryItem.Health(msg));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -26,6 +26,7 @@ using ZB.MOM.WW.OtOpcUa.OpcUaServer;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Redundancy;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.ScriptedAlarms;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Telemetry;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.VirtualTags;
|
||||
using CommonsNodeId = ZB.MOM.WW.OtOpcUa.Commons.Types.NodeId;
|
||||
|
||||
@@ -118,6 +119,11 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
|
||||
/// </summary>
|
||||
private readonly IAlarmStateStore? _alarmStateStore;
|
||||
|
||||
/// <summary>Optional node-local live-telemetry hub (Phase 5): each Primary-gated native-alarm
|
||||
/// <c>alerts</c> transition is also emitted here, and it is threaded into the spawned VirtualTag +
|
||||
/// ScriptedAlarm hosts. Null (admin-only nodes / tests) skips the emit and passes null down.</summary>
|
||||
private readonly ITelemetryLocalHub? _telemetryHub;
|
||||
|
||||
private readonly CommonsNodeId _localNode;
|
||||
private readonly IActorRef? _ackRouter;
|
||||
private readonly IDriverFactory _driverFactory;
|
||||
@@ -397,6 +403,9 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
|
||||
/// driver-role node this is the replicated LocalDb store, so condition state persists with no
|
||||
/// ConfigDb; null skips spawning the alarm host (the ConfigDb-backed EF fallback was retired in
|
||||
/// Phase 4 Task 9). Defaults to null.</param>
|
||||
/// <param name="telemetryHub">Per-cluster mesh Phase 5: optional node-local live-telemetry hub each
|
||||
/// Primary-gated native-alarm transition is also emitted into, and which is threaded into the spawned
|
||||
/// VirtualTag + ScriptedAlarm hosts. Defaults to null (admin-only nodes / tests skip the emit).</param>
|
||||
/// <returns>The Akka.NET <see cref="Akka.Actor.Props"/> used to spawn this actor.</returns>
|
||||
public static Props Props(
|
||||
IDbContextFactory<OtOpcUaConfigDbContext>? dbFactory,
|
||||
@@ -420,7 +429,8 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
|
||||
string? replicationPeerHost = null,
|
||||
bool fetchAndCacheMode = false,
|
||||
IDeploymentArtifactFetcher? artifactFetcher = null,
|
||||
IAlarmStateStore? alarmStateStore = null) =>
|
||||
IAlarmStateStore? alarmStateStore = null,
|
||||
ITelemetryLocalHub? telemetryHub = null) =>
|
||||
// WARNING: this forwarding list is POSITIONAL, and Props.Create compiles it into an
|
||||
// expression tree. Six IActorRef? parameters and several interface-typed ones mean a
|
||||
// mis-ordered argument is usually type-compatible and therefore compiles clean, then binds
|
||||
@@ -431,7 +441,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
|
||||
healthPublisher, virtualTagEvaluator, historyWriter, virtualTagHostOverride,
|
||||
loggerFactory, scriptRootLogger, scriptedAlarmHostOverride, invokerFactory, driverMemberCountProvider,
|
||||
deploymentArtifactCache, redundancyRoleView, replicationPeerHost, fetchAndCacheMode, artifactFetcher,
|
||||
alarmStateStore));
|
||||
alarmStateStore, telemetryHub));
|
||||
|
||||
/// <summary>Initializes a new DriverHostActor with the specified dependencies.</summary>
|
||||
/// <param name="dbFactory">Database context factory for configuration database access.</param>
|
||||
@@ -469,6 +479,9 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
|
||||
/// <param name="alarmStateStore">Per-cluster mesh Phase 4: scripted-alarm condition-state store. On a
|
||||
/// driver-role node this is the replicated LocalDb store; null skips spawning the alarm host (the
|
||||
/// ConfigDb-backed EF fallback was retired in Phase 4 Task 9).</param>
|
||||
/// <param name="telemetryHub">Per-cluster mesh Phase 5: optional node-local live-telemetry hub each
|
||||
/// Primary-gated native-alarm transition is also emitted into, and which is threaded into the spawned
|
||||
/// VirtualTag + ScriptedAlarm hosts. Null (admin-only nodes / tests) skips the emit and passes null down.</param>
|
||||
public DriverHostActor(
|
||||
IDbContextFactory<OtOpcUaConfigDbContext>? dbFactory,
|
||||
CommonsNodeId localNode,
|
||||
@@ -491,8 +504,10 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
|
||||
string? replicationPeerHost = null,
|
||||
bool fetchAndCacheMode = false,
|
||||
IDeploymentArtifactFetcher? artifactFetcher = null,
|
||||
IAlarmStateStore? alarmStateStore = null)
|
||||
IAlarmStateStore? alarmStateStore = null,
|
||||
ITelemetryLocalHub? telemetryHub = null)
|
||||
{
|
||||
_telemetryHub = telemetryHub;
|
||||
_deploymentArtifactCache = deploymentArtifactCache;
|
||||
_redundancyRoleView = redundancyRoleView;
|
||||
_replicationPeerHost = replicationPeerHost;
|
||||
@@ -577,7 +592,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
|
||||
}
|
||||
|
||||
_virtualTagHost = Context.ActorOf(
|
||||
VirtualTagHostActor.Props(_opcUaPublishActor, _dependencyMux, _virtualTagEvaluator, _historyWriter),
|
||||
VirtualTagHostActor.Props(_opcUaPublishActor, _dependencyMux, _virtualTagEvaluator, _historyWriter, _telemetryHub),
|
||||
"virtual-tag-host");
|
||||
}
|
||||
|
||||
@@ -630,7 +645,8 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
|
||||
var engine = new ScriptedAlarmEngine(
|
||||
upstream, store, new ScriptLoggerFactory(_scriptRootLogger.Logger), _scriptRootLogger.Logger);
|
||||
_scriptedAlarmHost = Context.ActorOf(
|
||||
ScriptedAlarmHostActor.Props(_opcUaPublishActor, _dependencyMux, upstream, engine, _localNode),
|
||||
ScriptedAlarmHostActor.Props(
|
||||
_opcUaPublishActor, _dependencyMux, upstream, engine, _localNode, telemetryHub: _telemetryHub),
|
||||
"scripted-alarm-host");
|
||||
}
|
||||
|
||||
@@ -1402,7 +1418,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
|
||||
var meta = _alarmMetaByNodeId.TryGetValue(nodeId, out var m)
|
||||
? m : (EquipmentId: nodeId, Name: nodeId, AlarmType: "AlarmCondition", HistorizeToAveva: (bool?)null,
|
||||
ReferencingEquipmentPaths: (IReadOnlyList<string>)Array.Empty<string>());
|
||||
_mediator.Tell(new Publish(ScriptedAlarmHostActor.AlertsTopic, new AlarmTransitionEvent(
|
||||
var alertEvent = new AlarmTransitionEvent(
|
||||
AlarmId: nodeId,
|
||||
EquipmentPath: meta.EquipmentId,
|
||||
AlarmName: meta.Name,
|
||||
@@ -1424,7 +1440,12 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
|
||||
HistorizeToAveva: meta.HistorizeToAveva,
|
||||
// WP4: the Area/Line/Equipment UNS paths that reference this raw condition — the /alerts row
|
||||
// renders them as the equipment list (empty for a raw condition no equipment references yet).
|
||||
ReferencingEquipmentPaths: meta.ReferencingEquipmentPaths)));
|
||||
ReferencingEquipmentPaths: meta.ReferencingEquipmentPaths);
|
||||
_mediator.Tell(new Publish(ScriptedAlarmHostActor.AlertsTopic, alertEvent));
|
||||
// Phase 5: fan the same transition into the node-local live-telemetry hub (no-op until a gRPC
|
||||
// client subscribes). The DPS publish above is unchanged — a strictly additive tap, riding the
|
||||
// same Primary gate (only reached when serviceAlertsAsPrimary).
|
||||
_telemetryHub?.Emit(new TelemetryItem.Alarm(alertEvent));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -11,6 +11,7 @@ using ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms;
|
||||
using ZB.MOM.WW.OtOpcUa.OpcUaServer;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.OpcUa;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Telemetry;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.VirtualTags;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Runtime.ScriptedAlarms;
|
||||
@@ -116,6 +117,10 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
|
||||
/// alerts-emit gate reads while the role is unknown. Null ⇒ read the live cluster state.</summary>
|
||||
private readonly Func<int>? _driverMemberCountProvider;
|
||||
|
||||
/// <summary>Optional node-local live-telemetry hub (Phase 5): each cluster-wide <c>alerts</c> transition
|
||||
/// is also emitted here for the gRPC streaming service. Null (tests / no hub wired) skips the emit.</summary>
|
||||
private readonly ITelemetryLocalHub? _telemetryHub;
|
||||
|
||||
/// <summary>Monotonic load generation, bumped on every <see cref="OnApply"/>. The continuation that
|
||||
/// pipes back an <see cref="AlarmsLoaded"/> captures the generation it was started under; a stale
|
||||
/// completion (an earlier generation arriving after a newer apply) is discarded in
|
||||
@@ -138,6 +143,10 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
|
||||
/// <param name="localNode">The local cluster node id, used to read this node's <see cref="RedundancyRole"/>
|
||||
/// from the <c>redundancy-state</c> topic so only the Primary publishes the cluster-wide <c>alerts</c>
|
||||
/// transition. Null (the default) leaves the role unknown ⇒ default-emit (single-node deploys + tests).</param>
|
||||
/// <param name="driverMemberCountProvider">Test seam (archreview 03/S4): overrides the count of Up
|
||||
/// <c>driver</c>-role cluster members the alerts-emit gate reads while the role is unknown.</param>
|
||||
/// <param name="telemetryHub">Optional node-local live-telemetry hub (Phase 5) each cluster-wide
|
||||
/// <c>alerts</c> transition is also emitted into; null (tests) skips the emit.</param>
|
||||
/// <returns>The <see cref="Akka.Actor.Props"/> used to instantiate the actor.</returns>
|
||||
public static Props Props(
|
||||
IActorRef publishActor,
|
||||
@@ -145,8 +154,9 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
|
||||
DependencyMuxTagUpstreamSource upstream,
|
||||
ScriptedAlarmEngine engine,
|
||||
NodeId? localNode = null,
|
||||
Func<int>? driverMemberCountProvider = null) =>
|
||||
Akka.Actor.Props.Create(() => new ScriptedAlarmHostActor(publishActor, mux, upstream, engine, localNode, driverMemberCountProvider));
|
||||
Func<int>? driverMemberCountProvider = null,
|
||||
ITelemetryLocalHub? telemetryHub = null) =>
|
||||
Akka.Actor.Props.Create(() => new ScriptedAlarmHostActor(publishActor, mux, upstream, engine, localNode, driverMemberCountProvider, telemetryHub));
|
||||
|
||||
/// <summary>Initializes a new instance of the <see cref="ScriptedAlarmHostActor"/> class.</summary>
|
||||
/// <param name="publishActor">The OPC UA publish actor emissions are bridged to.</param>
|
||||
@@ -159,13 +169,16 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
|
||||
/// <param name="driverMemberCountProvider">Test seam (archreview 03/S4): overrides the count of Up
|
||||
/// <c>driver</c>-role cluster members the alerts-emit gate reads while the role is unknown. Null reads
|
||||
/// the live cluster state (0 on a non-cluster ActorRefProvider ⇒ single-node default-emit).</param>
|
||||
/// <param name="telemetryHub">Optional node-local live-telemetry hub (Phase 5) each cluster-wide
|
||||
/// <c>alerts</c> transition is also emitted into; null (tests) skips the emit.</param>
|
||||
public ScriptedAlarmHostActor(
|
||||
IActorRef publishActor,
|
||||
IActorRef? mux,
|
||||
DependencyMuxTagUpstreamSource upstream,
|
||||
ScriptedAlarmEngine engine,
|
||||
NodeId? localNode = null,
|
||||
Func<int>? driverMemberCountProvider = null)
|
||||
Func<int>? driverMemberCountProvider = null,
|
||||
ITelemetryLocalHub? telemetryHub = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(publishActor);
|
||||
ArgumentNullException.ThrowIfNull(upstream);
|
||||
@@ -176,6 +189,7 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
|
||||
_engine = engine;
|
||||
_localNode = localNode;
|
||||
_driverMemberCountProvider = driverMemberCountProvider;
|
||||
_telemetryHub = telemetryHub;
|
||||
|
||||
// OnEvent fires on the engine's worker thread. NEVER touch Context / actor state here —
|
||||
// marshal onto the actor thread via the thread-safe Self.Tell. Keep the handler in a field
|
||||
@@ -377,6 +391,10 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
|
||||
}
|
||||
|
||||
_mediator.Tell(new Publish(AlertsTopic, evt));
|
||||
// Phase 5: fan the same transition into the node-local live-telemetry hub (no-op until a gRPC
|
||||
// client subscribes). The DPS publish above is unchanged — a strictly additive tap, and it rides
|
||||
// the same Primary gate as the publish (only the Primary reaches this line).
|
||||
_telemetryHub?.Emit(new TelemetryItem.Alarm(evt));
|
||||
}
|
||||
|
||||
/// <summary>Count of Up cluster members carrying the <c>driver</c> role, for the alerts-emit gate. Uses
|
||||
|
||||
@@ -2,6 +2,7 @@ using Akka.Actor;
|
||||
using Akka.Cluster.Tools.PublishSubscribe;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Logging;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Telemetry;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.VirtualTags;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Runtime.Scripting;
|
||||
@@ -27,15 +28,20 @@ namespace ZB.MOM.WW.OtOpcUa.Runtime.Scripting;
|
||||
public sealed class DpsScriptLogPublisher : IScriptLogPublisher
|
||||
{
|
||||
private readonly Func<ActorSystem> _system;
|
||||
private readonly ITelemetryLocalHub _hub;
|
||||
|
||||
/// <summary>Initializes a new instance of the <see cref="DpsScriptLogPublisher"/> class.</summary>
|
||||
/// <param name="system">
|
||||
/// Lazy accessor for the running <see cref="ActorSystem"/>. Invoked on each
|
||||
/// <see cref="Publish"/> so registration does not depend on Akka having started yet.
|
||||
/// </param>
|
||||
/// <exception cref="ArgumentNullException">Thrown when <paramref name="system"/> is <c>null</c>.</exception>
|
||||
public DpsScriptLogPublisher(Func<ActorSystem> system) =>
|
||||
/// <param name="hub">The node-local live-telemetry hub each script-log entry is also emitted into (Phase 5).</param>
|
||||
/// <exception cref="ArgumentNullException">Thrown when <paramref name="system"/> or <paramref name="hub"/> is <c>null</c>.</exception>
|
||||
public DpsScriptLogPublisher(Func<ActorSystem> system, ITelemetryLocalHub hub)
|
||||
{
|
||||
_system = system ?? throw new ArgumentNullException(nameof(system));
|
||||
_hub = hub ?? throw new ArgumentNullException(nameof(hub));
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public void Publish(ScriptLogEntry entry)
|
||||
@@ -44,6 +50,9 @@ public sealed class DpsScriptLogPublisher : IScriptLogPublisher
|
||||
{
|
||||
var mediator = DistributedPubSub.Get(_system()).Mediator;
|
||||
mediator.Tell(new Publish(VirtualTagActor.ScriptLogsTopic, entry));
|
||||
// Phase 5: fan the same entry into the node-local live-telemetry hub (no-op until a gRPC
|
||||
// client subscribes). The DPS publish above is unchanged — the hub is a strictly additive tap.
|
||||
_hub.Emit(new TelemetryItem.Script(entry));
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
|
||||
@@ -28,6 +28,7 @@ using ZB.MOM.WW.OtOpcUa.Runtime.Health;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Historian;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.OpcUa;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Redundancy;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Telemetry;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.VirtualTags;
|
||||
using ZB.MOM.WW.LocalDb;
|
||||
|
||||
@@ -70,6 +71,10 @@ public static class ServiceCollectionExtensions
|
||||
services.TryAddSingleton<IOpcUaAddressSpaceSink>(NullOpcUaAddressSpaceSink.Instance);
|
||||
services.TryAddSingleton<IServiceLevelPublisher>(NullServiceLevelPublisher.Instance);
|
||||
services.TryAddSingleton<IDriverHealthPublisher, AkkaDriverHealthPublisher>();
|
||||
// Per-cluster mesh Phase 5: the node-local live-telemetry fan-out hub. Feeds this node's OWN
|
||||
// telemetry (never DPS) to the Phase-5 gRPC streaming service central dials in on. AddOtOpcUaRuntime
|
||||
// runs only inside Program.cs's hasDriver block, so this lands on driver-role nodes only.
|
||||
services.TryAddSingleton<ITelemetryLocalHub, TelemetryLocalHub>();
|
||||
return services;
|
||||
}
|
||||
|
||||
@@ -490,7 +495,11 @@ public static class ServiceCollectionExtensions
|
||||
replicationPeerHost: replicationPeerHost,
|
||||
fetchAndCacheMode: fetchAndCacheMode,
|
||||
artifactFetcher: artifactFetcher,
|
||||
alarmStateStore: alarmStateStore),
|
||||
alarmStateStore: alarmStateStore,
|
||||
// Phase 5: the node-local live-telemetry hub (registered in AddOtOpcUaRuntime). The host
|
||||
// emits its Primary-gated native-alarm transitions into it and threads it into the
|
||||
// VirtualTag + ScriptedAlarm hosts it spawns.
|
||||
telemetryHub: resolver.GetService<ITelemetryLocalHub>()),
|
||||
DriverHostActorName);
|
||||
registry.Register<DriverHostActorKey>(driverHost);
|
||||
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
using System.Threading.Channels;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Alerts;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Drivers;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Logging;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Runtime.Telemetry;
|
||||
|
||||
/// <summary>
|
||||
/// A single live-telemetry item flowing through the node-local hub. A closed union over the four
|
||||
/// wire-facing telemetry records — carried as the domain records themselves, NOT as proto (the
|
||||
/// gRPC projection happens at the streaming-service seam, one layer out).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Two of the four are <b>snapshot-style</b> (a last-value-per-key view: <see cref="Health"/> keyed
|
||||
/// by <c>DriverInstanceId</c>, <see cref="Resilience"/> keyed by <c>(DriverInstanceId, HostName)</c>)
|
||||
/// and are cached so a newly-attaching subscriber is primed immediately. The other two
|
||||
/// (<see cref="Alarm"/>, <see cref="Script"/>) are <b>append-style</b> logs and are forwarded live-only.
|
||||
/// </remarks>
|
||||
public abstract record TelemetryItem
|
||||
{
|
||||
private TelemetryItem() { }
|
||||
|
||||
/// <summary>An append-style alarm transition. Live-forwarded only; never cached/replayed.</summary>
|
||||
public sealed record Alarm(AlarmTransitionEvent E) : TelemetryItem;
|
||||
|
||||
/// <summary>An append-style script-log line. Live-forwarded only; never cached/replayed.</summary>
|
||||
public sealed record Script(ScriptLogEntry E) : TelemetryItem;
|
||||
|
||||
/// <summary>A snapshot-style driver-health change, cached last-value per <c>DriverInstanceId</c>.</summary>
|
||||
public sealed record Health(DriverHealthChanged E) : TelemetryItem;
|
||||
|
||||
/// <summary>A snapshot-style resilience-status change, cached last-value per <c>(DriverInstanceId, HostName)</c>.</summary>
|
||||
public sealed record Resilience(DriverResilienceStatusChanged E) : TelemetryItem;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// One attached reader on the <see cref="ITelemetryLocalHub"/>. Disposing detaches it from the hub
|
||||
/// and completes its channel; the reader first yields the cached snapshots, then live deltas.
|
||||
/// </summary>
|
||||
public interface ITelemetrySubscription : IDisposable
|
||||
{
|
||||
/// <summary>The bounded reader this subscription drains. Completed on <see cref="IDisposable.Dispose"/>.</summary>
|
||||
ChannelReader<TelemetryItem> Reader { get; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Process-wide, node-local fan-out hub sitting between this node's own telemetry producers and the
|
||||
/// (Phase-5) gRPC streaming service central dials into.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>CRITICAL INVARIANT — this hub carries ONLY this node's own telemetry.</b> It is fed
|
||||
/// directly, in-process, by the node's producers and MUST NEVER subscribe to cluster
|
||||
/// DistributedPubSub. On the single mesh, DPS delivers every node's events to every node; if the
|
||||
/// hub read DPS it would stream peers' events and central — which dials each node individually —
|
||||
/// would double-count.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Fan-out is lossy under backpressure by design: each subscriber owns a bounded channel with
|
||||
/// <see cref="BoundedChannelFullMode.DropOldest"/>, so a slow consumer sheds its own oldest items
|
||||
/// and can never block the producer / node-actor threads.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public interface ITelemetryLocalHub
|
||||
{
|
||||
/// <summary>
|
||||
/// Fans <paramref name="item"/> to every currently-attached subscriber (non-blocking, drop-oldest
|
||||
/// on a full channel) and, for the two snapshot-style items, updates the last-value cache that
|
||||
/// primes future subscribers.
|
||||
/// </summary>
|
||||
void Emit(TelemetryItem item);
|
||||
|
||||
/// <summary>
|
||||
/// Attaches a new subscriber. Its reader first yields the cached snapshots (all cached
|
||||
/// <see cref="TelemetryItem.Health"/>, then all cached <see cref="TelemetryItem.Resilience"/>),
|
||||
/// then live deltas — with no delta lost across the attach boundary. Dispose to detach.
|
||||
/// </summary>
|
||||
/// <param name="boundedCapacity">
|
||||
/// Per-subscriber channel capacity (must be positive). The snapshot prelude is written into this
|
||||
/// same bounded/DropOldest channel before <see cref="Subscribe"/> returns, so
|
||||
/// <paramref name="boundedCapacity"/> should comfortably exceed this node's live driver-instance +
|
||||
/// (instance, host) count — otherwise the earliest cached snapshots are silently evicted during
|
||||
/// priming before the reader ever drains them. The node-local cache holds only THIS node's
|
||||
/// instances (a handful), so a normal large capacity leaves ample headroom.
|
||||
/// </param>
|
||||
ITelemetrySubscription Subscribe(int boundedCapacity);
|
||||
}
|
||||
@@ -0,0 +1,129 @@
|
||||
using System.Collections.Concurrent;
|
||||
using System.Threading.Channels;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Runtime.Telemetry;
|
||||
|
||||
/// <summary>
|
||||
/// Default <see cref="ITelemetryLocalHub"/>: a process-wide singleton on driver-role nodes.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>No DPS.</b> This hub is fed only by this node's own in-process producers (a later Phase-5
|
||||
/// task taps them). It never subscribes to cluster DistributedPubSub — see the invariant on
|
||||
/// <see cref="ITelemetryLocalHub"/>.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Snapshot-then-attach ordering (no lost delta).</b> <see cref="Emit"/> updates the snapshot
|
||||
/// cache and fans out under a single gate; <see cref="Subscribe"/> writes the cached snapshots
|
||||
/// into the new channel and only THEN adds it to the subscriber set — all under that same gate.
|
||||
/// Because both operations are serialized by the gate, an <see cref="Emit"/> racing a
|
||||
/// <see cref="Subscribe"/> is resolved one of two ways, both correct: it runs entirely before the
|
||||
/// subscribe (its value is in the snapshot, and it is not delivered live because the channel is
|
||||
/// not yet attached), or entirely after (it is delivered live, strictly after the snapshot). No
|
||||
/// item is dropped or duplicated across the boundary. Fan-out is a non-blocking
|
||||
/// <see cref="ChannelWriter{T}.TryWrite"/> into bounded/DropOldest channels, so holding the gate
|
||||
/// never blocks on a slow consumer.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class TelemetryLocalHub : ITelemetryLocalHub
|
||||
{
|
||||
private readonly object _gate = new();
|
||||
|
||||
// Plain Dictionary, NOT ConcurrentDictionary: every access is already under _gate, and
|
||||
// ConcurrentDictionary.Values would materialize a fresh List (and take its own internal lock) on
|
||||
// every Emit — the hot node-actor-thread path. Under the gate a plain Dictionary is safe and
|
||||
// allocation-free to iterate.
|
||||
private readonly Dictionary<Guid, Channel<TelemetryItem>> _subscribers = new();
|
||||
|
||||
// Snapshot last-value caches. Only ever mutated under _gate (kept ConcurrentDictionary so a
|
||||
// Subscribe reading them under the gate is trivially safe even against any future lock-free read).
|
||||
// TODO(mesh-phase5+): evict snapshot cache entries on driver-instance removal (needs a
|
||||
// driver-lifecycle hook); pre-production, accepted for now. Until then a decommissioned driver
|
||||
// instance's last-known Health/Resilience state replays to new subscribers forever.
|
||||
private readonly ConcurrentDictionary<string, TelemetryItem.Health> _healthCache = new();
|
||||
private readonly ConcurrentDictionary<(string InstanceId, string HostName), TelemetryItem.Resilience> _resilienceCache = new();
|
||||
|
||||
/// <inheritdoc />
|
||||
public void Emit(TelemetryItem item)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(item);
|
||||
|
||||
lock (_gate)
|
||||
{
|
||||
switch (item)
|
||||
{
|
||||
case TelemetryItem.Health h:
|
||||
_healthCache[h.E.DriverInstanceId] = h;
|
||||
break;
|
||||
case TelemetryItem.Resilience r:
|
||||
_resilienceCache[(r.E.DriverInstanceId, r.E.HostName)] = r;
|
||||
break;
|
||||
// Alarm / Script are append-style — never cached.
|
||||
}
|
||||
|
||||
foreach (var channel in _subscribers.Values)
|
||||
channel.Writer.TryWrite(item); // bounded + DropOldest ⇒ non-blocking, sheds oldest when full
|
||||
}
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public ITelemetrySubscription Subscribe(int boundedCapacity)
|
||||
{
|
||||
if (boundedCapacity < 1)
|
||||
throw new ArgumentOutOfRangeException(nameof(boundedCapacity), boundedCapacity, "Capacity must be positive.");
|
||||
|
||||
var channel = Channel.CreateBounded<TelemetryItem>(new BoundedChannelOptions(boundedCapacity)
|
||||
{
|
||||
FullMode = BoundedChannelFullMode.DropOldest,
|
||||
SingleReader = true,
|
||||
SingleWriter = false,
|
||||
});
|
||||
|
||||
var id = Guid.NewGuid();
|
||||
lock (_gate)
|
||||
{
|
||||
// Snapshot FIRST (Health then Resilience), attach SECOND — both under the gate so a
|
||||
// concurrent Emit cannot slip a delta between the snapshot read and the attach.
|
||||
foreach (var health in _healthCache.Values)
|
||||
channel.Writer.TryWrite(health);
|
||||
foreach (var resilience in _resilienceCache.Values)
|
||||
channel.Writer.TryWrite(resilience);
|
||||
|
||||
_subscribers[id] = channel;
|
||||
}
|
||||
|
||||
return new Subscription(this, id, channel);
|
||||
}
|
||||
|
||||
private void Detach(Guid id)
|
||||
{
|
||||
lock (_gate)
|
||||
{
|
||||
if (_subscribers.Remove(id, out var channel))
|
||||
channel.Writer.TryComplete();
|
||||
}
|
||||
}
|
||||
|
||||
private sealed class Subscription : ITelemetrySubscription
|
||||
{
|
||||
private readonly TelemetryLocalHub _hub;
|
||||
private readonly Guid _id;
|
||||
private int _disposed;
|
||||
|
||||
public Subscription(TelemetryLocalHub hub, Guid id, Channel<TelemetryItem> channel)
|
||||
{
|
||||
_hub = hub;
|
||||
_id = id;
|
||||
Reader = channel.Reader;
|
||||
}
|
||||
|
||||
public ChannelReader<TelemetryItem> Reader { get; }
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
if (Interlocked.Exchange(ref _disposed, 1) != 0)
|
||||
return; // idempotent
|
||||
_hub.Detach(_id);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -6,6 +6,7 @@ using ZB.MOM.WW.OtOpcUa.Commons.Messages.Logging;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Observability;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Types;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Telemetry;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Runtime.VirtualTags;
|
||||
|
||||
@@ -61,6 +62,7 @@ public sealed class VirtualTagActor : ReceiveActor
|
||||
private readonly Func<DPSPublisher>? _publisherFactory;
|
||||
private readonly IReadOnlyList<string> _dependencyRefs;
|
||||
private readonly IActorRef? _mux;
|
||||
private readonly ITelemetryLocalHub? _telemetryHub;
|
||||
private readonly ILoggingAdapter _log = Context.GetLogger();
|
||||
private readonly Dictionary<string, object?> _dependencies = new(StringComparer.Ordinal);
|
||||
|
||||
@@ -78,6 +80,8 @@ public sealed class VirtualTagActor : ReceiveActor
|
||||
/// <param name="publisherFactory">Optional factory for creating DPS publishers.</param>
|
||||
/// <param name="dependencyRefs">Optional list of dependency tag references; defaults to empty.</param>
|
||||
/// <param name="mux">Optional reference to a dependency multiplexer actor.</param>
|
||||
/// <param name="telemetryHub">Optional node-local live-telemetry hub each script-log entry is also
|
||||
/// emitted into (Phase 5); null (tests) simply skips the emit.</param>
|
||||
/// <returns>A configured <see cref="Akka.Actor.Props"/> for creating the actor.</returns>
|
||||
public static Props Props(
|
||||
string virtualTagId,
|
||||
@@ -86,14 +90,16 @@ public sealed class VirtualTagActor : ReceiveActor
|
||||
string? scriptId = null,
|
||||
Func<DPSPublisher>? publisherFactory = null,
|
||||
IReadOnlyList<string>? dependencyRefs = null,
|
||||
IActorRef? mux = null) =>
|
||||
IActorRef? mux = null,
|
||||
ITelemetryLocalHub? telemetryHub = null) =>
|
||||
Akka.Actor.Props.Create(() => new VirtualTagActor(
|
||||
virtualTagId, expression,
|
||||
evaluator ?? NullVirtualTagEvaluator.Instance,
|
||||
scriptId ?? virtualTagId,
|
||||
publisherFactory,
|
||||
dependencyRefs ?? Array.Empty<string>(),
|
||||
mux));
|
||||
mux,
|
||||
telemetryHub));
|
||||
|
||||
/// <summary>Initializes a virtual tag actor with the given configuration and dependencies.</summary>
|
||||
/// <param name="virtualTagId">Unique identifier for the virtual tag.</param>
|
||||
@@ -103,6 +109,8 @@ public sealed class VirtualTagActor : ReceiveActor
|
||||
/// <param name="publisherFactory">Optional factory for creating DPS publishers.</param>
|
||||
/// <param name="dependencyRefs">List of dependency tag references that this tag depends on.</param>
|
||||
/// <param name="mux">Optional reference to a dependency multiplexer actor.</param>
|
||||
/// <param name="telemetryHub">Optional node-local live-telemetry hub each script-log entry is also
|
||||
/// emitted into (Phase 5); null (tests) simply skips the emit.</param>
|
||||
public VirtualTagActor(
|
||||
string virtualTagId,
|
||||
string expression,
|
||||
@@ -110,7 +118,8 @@ public sealed class VirtualTagActor : ReceiveActor
|
||||
string scriptId,
|
||||
Func<DPSPublisher>? publisherFactory,
|
||||
IReadOnlyList<string> dependencyRefs,
|
||||
IActorRef? mux)
|
||||
IActorRef? mux,
|
||||
ITelemetryLocalHub? telemetryHub = null)
|
||||
{
|
||||
_virtualTagId = virtualTagId;
|
||||
_scriptId = scriptId;
|
||||
@@ -119,6 +128,7 @@ public sealed class VirtualTagActor : ReceiveActor
|
||||
_publisherFactory = publisherFactory;
|
||||
_dependencyRefs = dependencyRefs;
|
||||
_mux = mux;
|
||||
_telemetryHub = telemetryHub;
|
||||
|
||||
Receive<DependencyValueChanged>(OnDependencyChanged);
|
||||
Receive<ReassertValue>(_ => OnReassertValue());
|
||||
@@ -252,6 +262,12 @@ public sealed class VirtualTagActor : ReceiveActor
|
||||
AlarmId: null,
|
||||
EquipmentId: null);
|
||||
|
||||
// Phase 5: fan the same entry into the node-local live-telemetry hub (no-op until a gRPC client
|
||||
// subscribes). Emitted BEFORE the publish branch so BOTH the production DPS path and the test-seam
|
||||
// publisherFactory path (which returns early below) feed the hub. Both taps are fire-and-forget, so
|
||||
// emit-then-publish ordering is immaterial; the DPS/factory publish itself is unchanged.
|
||||
_telemetryHub?.Emit(new TelemetryItem.Script(entry));
|
||||
|
||||
if (_publisherFactory is not null)
|
||||
{
|
||||
_publisherFactory().Publish(ScriptLogsTopic, entry);
|
||||
|
||||
@@ -6,6 +6,7 @@ using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.VirtualTags;
|
||||
using ZB.MOM.WW.OtOpcUa.OpcUaServer;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.OpcUa;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Telemetry;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Runtime.VirtualTags;
|
||||
|
||||
@@ -39,6 +40,7 @@ public sealed class VirtualTagHostActor : ReceiveActor
|
||||
// Sink for historized VirtualTag results (plans with Historize=true). NullHistoryWriter when no
|
||||
// durable historian is wired, so OnResult always has a non-null target.
|
||||
private readonly IHistoryWriter _history;
|
||||
private readonly ITelemetryLocalHub? _telemetryHub;
|
||||
private readonly ILoggingAdapter _log = Context.GetLogger();
|
||||
|
||||
// vtagId -> spawned child VirtualTagActor.
|
||||
@@ -59,18 +61,22 @@ public sealed class VirtualTagHostActor : ReceiveActor
|
||||
/// <param name="historyWriter">Sink for results whose plan has <c>Historize=true</c>. Null ⇒
|
||||
/// <see cref="NullHistoryWriter.Instance"/> (no durable historian wired), so existing call sites
|
||||
/// compile unchanged and never historize.</param>
|
||||
/// <param name="telemetryHub">Optional node-local live-telemetry hub passed to each spawned child so
|
||||
/// its script-log entries also emit to the Phase-5 hub; null (tests) skips the emit.</param>
|
||||
/// <returns>The <see cref="Props"/> used to spawn a <see cref="VirtualTagHostActor"/>.</returns>
|
||||
public static Props Props(IActorRef publishActor, IActorRef? mux, IVirtualTagEvaluator evaluator,
|
||||
IHistoryWriter? historyWriter = null) =>
|
||||
Akka.Actor.Props.Create(() => new VirtualTagHostActor(publishActor, mux, evaluator, historyWriter));
|
||||
IHistoryWriter? historyWriter = null, ITelemetryLocalHub? telemetryHub = null) =>
|
||||
Akka.Actor.Props.Create(() => new VirtualTagHostActor(publishActor, mux, evaluator, historyWriter, telemetryHub));
|
||||
|
||||
/// <summary>Initializes a new instance of the <see cref="VirtualTagHostActor"/> class.</summary>
|
||||
/// <param name="publishActor">The OPC UA publish actor results are bridged to.</param>
|
||||
/// <param name="mux">Optional dependency multiplexer passed to each spawned child.</param>
|
||||
/// <param name="evaluator">The evaluator each child uses to compute its expression.</param>
|
||||
/// <param name="historyWriter">Sink for historized results; null ⇒ <see cref="NullHistoryWriter.Instance"/>.</param>
|
||||
/// <param name="telemetryHub">Optional node-local live-telemetry hub passed to each spawned child so
|
||||
/// its script-log entries also emit to the Phase-5 hub; null (tests) skips the emit.</param>
|
||||
public VirtualTagHostActor(IActorRef publishActor, IActorRef? mux, IVirtualTagEvaluator evaluator,
|
||||
IHistoryWriter? historyWriter = null)
|
||||
IHistoryWriter? historyWriter = null, ITelemetryLocalHub? telemetryHub = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(publishActor);
|
||||
ArgumentNullException.ThrowIfNull(evaluator);
|
||||
@@ -78,6 +84,7 @@ public sealed class VirtualTagHostActor : ReceiveActor
|
||||
_mux = mux;
|
||||
_evaluator = evaluator;
|
||||
_history = historyWriter ?? NullHistoryWriter.Instance;
|
||||
_telemetryHub = telemetryHub;
|
||||
|
||||
Receive<ApplyVirtualTags>(OnApply);
|
||||
Receive<VirtualTagActor.EvaluationResult>(OnResult);
|
||||
@@ -166,7 +173,8 @@ public sealed class VirtualTagHostActor : ReceiveActor
|
||||
scriptId: p.VirtualTagId,
|
||||
publisherFactory: null,
|
||||
dependencyRefs: p.DependencyRefs,
|
||||
mux: _mux));
|
||||
mux: _mux,
|
||||
telemetryHub: _telemetryHub));
|
||||
Context.Watch(child);
|
||||
_children[p.VirtualTagId] = child;
|
||||
newlySpawned.Add(p.VirtualTagId);
|
||||
|
||||
@@ -0,0 +1,157 @@
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using Microsoft.Extensions.Options;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// A Grpc-mode node with no listen port or no key does not error — it simply never serves
|
||||
/// telemetry, and whoever is watching sees nothing with no stack trace to point at. These tests
|
||||
/// make that misconfiguration a loud host-start failure instead.
|
||||
/// </summary>
|
||||
public class TelemetryOptionsValidatorTests
|
||||
{
|
||||
private static ValidateOptionsResult Validate(TelemetryOptions o, params string[] roles)
|
||||
{
|
||||
var pairs = new Dictionary<string, string?>();
|
||||
for (var i = 0; i < roles.Length; i++)
|
||||
{
|
||||
pairs[$"Cluster:Roles:{i}"] = roles[i];
|
||||
}
|
||||
|
||||
var configuration = new ConfigurationBuilder().AddInMemoryCollection(pairs).Build();
|
||||
|
||||
return new TelemetryOptionsValidator(configuration).Validate(TelemetryOptions.SectionName, o);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Default_options_are_valid()
|
||||
{
|
||||
Validate(new TelemetryOptions()).Succeeded.ShouldBeTrue();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Unknown_mode_fails()
|
||||
{
|
||||
var result = Validate(new TelemetryOptions { Mode = "stream" });
|
||||
|
||||
result.Failed.ShouldBeTrue();
|
||||
result.FailureMessage.ShouldContain("stream");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Mode_matching_is_case_insensitive()
|
||||
{
|
||||
Validate(new TelemetryOptions
|
||||
{
|
||||
Mode = "grpc",
|
||||
GrpcListenPort = 5100,
|
||||
ApiKey = "k",
|
||||
}, "driver").Succeeded.ShouldBeTrue();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Grpc_driver_node_with_no_listen_port_fails()
|
||||
{
|
||||
var result = Validate(
|
||||
new TelemetryOptions { Mode = TelemetryOptions.ModeGrpc, ApiKey = "k" },
|
||||
"driver");
|
||||
|
||||
result.Failed.ShouldBeTrue();
|
||||
result.FailureMessage.ShouldContain(nameof(TelemetryOptions.GrpcListenPort));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Grpc_with_empty_key_fails()
|
||||
{
|
||||
var result = Validate(
|
||||
new TelemetryOptions { Mode = TelemetryOptions.ModeGrpc, GrpcListenPort = 5100, ApiKey = "" },
|
||||
"driver");
|
||||
|
||||
result.Failed.ShouldBeTrue();
|
||||
result.FailureMessage.ShouldContain(nameof(TelemetryOptions.ApiKey));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Grpc_driver_with_port_and_key_is_valid()
|
||||
{
|
||||
Validate(
|
||||
new TelemetryOptions { Mode = TelemetryOptions.ModeGrpc, GrpcListenPort = 5100, ApiKey = "k" },
|
||||
"driver").Succeeded.ShouldBeTrue();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Grpc_admin_only_node_does_not_require_a_listen_port()
|
||||
{
|
||||
// The listen-port requirement is driver-specific — an admin-only node dialling out (via
|
||||
// TelemetryDialOptions) never serves, so it has nothing to bind.
|
||||
Validate(
|
||||
new TelemetryOptions { Mode = TelemetryOptions.ModeGrpc, ApiKey = "k" },
|
||||
"admin").Succeeded.ShouldBeTrue();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Grpc_with_no_roles_does_not_require_a_key()
|
||||
{
|
||||
// Fail-closed only applies to a node that actually carries a role — a roleless node hosts
|
||||
// nothing to protect.
|
||||
Validate(new TelemetryOptions { Mode = TelemetryOptions.ModeGrpc, ApiKey = "" })
|
||||
.Succeeded.ShouldBeTrue();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Dps_ignores_the_empty_grpc_surface()
|
||||
{
|
||||
Validate(
|
||||
new TelemetryOptions { Mode = TelemetryOptions.ModeDps, GrpcListenPort = 0, ApiKey = "" },
|
||||
"driver").Succeeded.ShouldBeTrue();
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Mirrors <see cref="TelemetryOptionsValidatorTests"/> for the central dial-side options.
|
||||
/// </summary>
|
||||
public class TelemetryDialOptionsValidatorTests
|
||||
{
|
||||
private static ValidateOptionsResult Validate(TelemetryDialOptions o) =>
|
||||
new TelemetryDialOptionsValidator().Validate(TelemetryDialOptions.SectionName, o);
|
||||
|
||||
[Fact]
|
||||
public void Default_options_are_valid()
|
||||
{
|
||||
Validate(new TelemetryDialOptions()).Succeeded.ShouldBeTrue();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Unknown_mode_fails()
|
||||
{
|
||||
var result = Validate(new TelemetryDialOptions { Mode = "poll" });
|
||||
|
||||
result.Failed.ShouldBeTrue();
|
||||
result.FailureMessage.ShouldContain("poll");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Grpc_with_empty_key_fails()
|
||||
{
|
||||
var result = Validate(new TelemetryDialOptions { Mode = TelemetryDialOptions.ModeGrpc, ApiKey = "" });
|
||||
|
||||
result.Failed.ShouldBeTrue();
|
||||
result.FailureMessage.ShouldContain(nameof(TelemetryDialOptions.ApiKey));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Grpc_with_a_key_is_valid()
|
||||
{
|
||||
Validate(new TelemetryDialOptions { Mode = TelemetryDialOptions.ModeGrpc, ApiKey = "k" })
|
||||
.Succeeded.ShouldBeTrue();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Dps_ignores_the_empty_key()
|
||||
{
|
||||
Validate(new TelemetryDialOptions { Mode = TelemetryDialOptions.ModeDps, ApiKey = "" })
|
||||
.Succeeded.ShouldBeTrue();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Protos;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Protos.Telemetry.V1;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Commons.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Contract-lock for the Phase 5 telemetry oneof. If a fifth variant is added to
|
||||
/// <c>telemetry.proto</c> without adding a matching entry to
|
||||
/// <see cref="TelemetryProtoContract.HandledCases"/>, this test goes red — the single guard that
|
||||
/// keeps the wire contract and the handled-case set from silently drifting apart.
|
||||
/// </summary>
|
||||
public class TelemetryProtoContractTests
|
||||
{
|
||||
[Fact]
|
||||
public void EveryOneofVariant_IsAccountedFor()
|
||||
{
|
||||
var variants = System.Enum.GetValues<TelemetryEvent.EventOneofCase>()
|
||||
.Where(c => c != TelemetryEvent.EventOneofCase.None).ToArray();
|
||||
|
||||
variants.ShouldBe(TelemetryProtoContract.HandledCases, ignoreOrder: true);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void The_service_base_and_client_types_generate()
|
||||
{
|
||||
// Referencing these types is the whole assertion — GrpcServices="Both" must emit both.
|
||||
typeof(TelemetryStreamService.TelemetryStreamServiceBase).ShouldNotBeNull();
|
||||
typeof(TelemetryStreamService.TelemetryStreamServiceClient).ShouldNotBeNull();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,134 @@
|
||||
using Akka.Actor;
|
||||
using Akka.Hosting;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Options;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.AdminUI.Hubs;
|
||||
using ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.AdminUI.Tests.Hubs;
|
||||
|
||||
/// <summary>
|
||||
/// Verifies the Phase 5 dark switch in <see cref="HubServiceCollectionExtensions.WithOtOpcUaSignalRBridges"/>:
|
||||
/// <c>TelemetryDial:Mode = Dps</c> (default) spawns the four telemetry DPS bridge actors and NOT the
|
||||
/// gRPC dial supervisor; <c>Grpc</c> spawns the dial supervisor and NONE of the four telemetry
|
||||
/// bridges. The fleet-status bridge (deferred / out of Phase 5 scope) is spawned in BOTH modes.
|
||||
/// The gate is proven on a real <see cref="ActorRegistry"/> from a started Akka host.
|
||||
/// </summary>
|
||||
public sealed class TelemetryModeWiringTests
|
||||
{
|
||||
/// <summary>Dps mode: the four telemetry bridges + fleet bridge are registered; the dial supervisor is not.</summary>
|
||||
[Fact]
|
||||
public async Task Dps_mode_spawns_the_four_telemetry_bridges_and_not_the_dial_supervisor()
|
||||
{
|
||||
using var host = BuildBridgeHost(TelemetryDialOptions.ModeDps);
|
||||
await host.StartAsync();
|
||||
try
|
||||
{
|
||||
var registry = host.Services.GetRequiredService<ActorRegistry>();
|
||||
|
||||
registry.TryGet<FleetStatusSignalRBridgeKey>(out _).ShouldBeTrue();
|
||||
registry.TryGet<AlertSignalRBridgeKey>(out _).ShouldBeTrue();
|
||||
registry.TryGet<ScriptLogSignalRBridgeKey>(out _).ShouldBeTrue();
|
||||
registry.TryGet<DriverStatusSignalRBridgeKey>(out _).ShouldBeTrue();
|
||||
registry.TryGet<DriverResilienceStatusBridgeKey>(out _).ShouldBeTrue();
|
||||
|
||||
registry.TryGet<TelemetryDialSupervisorKey>(out _).ShouldBeFalse();
|
||||
}
|
||||
finally
|
||||
{
|
||||
await host.StopAsync();
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Grpc mode: the dial supervisor + fleet bridge are registered; none of the four telemetry bridges are.</summary>
|
||||
[Fact]
|
||||
public async Task Grpc_mode_spawns_the_dial_supervisor_and_none_of_the_four_telemetry_bridges()
|
||||
{
|
||||
using var host = BuildBridgeHost(TelemetryDialOptions.ModeGrpc);
|
||||
await host.StartAsync();
|
||||
try
|
||||
{
|
||||
var registry = host.Services.GetRequiredService<ActorRegistry>();
|
||||
|
||||
registry.TryGet<FleetStatusSignalRBridgeKey>(out _).ShouldBeTrue();
|
||||
registry.TryGet<TelemetryDialSupervisorKey>(out _).ShouldBeTrue();
|
||||
|
||||
registry.TryGet<AlertSignalRBridgeKey>(out _).ShouldBeFalse();
|
||||
registry.TryGet<ScriptLogSignalRBridgeKey>(out _).ShouldBeFalse();
|
||||
registry.TryGet<DriverStatusSignalRBridgeKey>(out _).ShouldBeFalse();
|
||||
registry.TryGet<DriverResilienceStatusBridgeKey>(out _).ShouldBeFalse();
|
||||
}
|
||||
finally
|
||||
{
|
||||
await host.StopAsync();
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Unset (default) mode is treated as Dps — the four bridges spawn, the supervisor does not.</summary>
|
||||
[Fact]
|
||||
public async Task Default_mode_is_dps()
|
||||
{
|
||||
using var host = BuildBridgeHost(mode: null);
|
||||
await host.StartAsync();
|
||||
try
|
||||
{
|
||||
var registry = host.Services.GetRequiredService<ActorRegistry>();
|
||||
registry.TryGet<AlertSignalRBridgeKey>(out _).ShouldBeTrue();
|
||||
registry.TryGet<TelemetryDialSupervisorKey>(out _).ShouldBeFalse();
|
||||
}
|
||||
finally
|
||||
{
|
||||
await host.StopAsync();
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Builds an admin-role host that runs <c>WithOtOpcUaSignalRBridges</c> under the given telemetry mode.</summary>
|
||||
/// <param name="mode">The <c>TelemetryDial:Mode</c> value, or <see langword="null"/> to leave it at its default.</param>
|
||||
private static IHost BuildBridgeHost(string? mode)
|
||||
=> Host.CreateDefaultBuilder()
|
||||
.ConfigureServices((_, services) =>
|
||||
{
|
||||
services.AddSignalR();
|
||||
services.AddOtOpcUaDriverStatusServices();
|
||||
services.AddSingleton<IDbContextFactory<OtOpcUaConfigDbContext>>(
|
||||
new InMemoryConfigDbFactory(Guid.NewGuid().ToString("N")));
|
||||
|
||||
var options = new TelemetryDialOptions { ApiKey = "test-key" };
|
||||
if (mode is not null)
|
||||
{
|
||||
options.Mode = mode;
|
||||
}
|
||||
|
||||
services.AddSingleton<IOptions<TelemetryDialOptions>>(Options.Create(options));
|
||||
|
||||
services.AddAkka("otopcua-test", (ab, _) =>
|
||||
{
|
||||
ab.AddHocon(@"
|
||||
akka.actor.provider = ""Akka.Cluster.ClusterActorRefProvider, Akka.Cluster""
|
||||
akka.remote.dot-netty.tcp.hostname = ""127.0.0.1""
|
||||
akka.remote.dot-netty.tcp.port = 0
|
||||
akka.cluster.seed-nodes = []
|
||||
akka.cluster.roles = [""admin""]
|
||||
", HoconAddMode.Prepend);
|
||||
ab.WithOtOpcUaSignalRBridges();
|
||||
});
|
||||
})
|
||||
.Build();
|
||||
|
||||
/// <summary>An <see cref="IDbContextFactory{TContext}"/> whose contexts share one InMemory database.</summary>
|
||||
private sealed class InMemoryConfigDbFactory(string dbName) : IDbContextFactory<OtOpcUaConfigDbContext>
|
||||
{
|
||||
public OtOpcUaConfigDbContext CreateDbContext() =>
|
||||
new(new DbContextOptionsBuilder<OtOpcUaConfigDbContext>()
|
||||
.UseInMemoryDatabase(dbName)
|
||||
.Options);
|
||||
|
||||
public Task<OtOpcUaConfigDbContext> CreateDbContextAsync(CancellationToken cancellationToken = default) =>
|
||||
Task.FromResult(CreateDbContext());
|
||||
}
|
||||
}
|
||||
+471
@@ -0,0 +1,471 @@
|
||||
using System.Collections.Concurrent;
|
||||
using Akka.Actor;
|
||||
using Akka.TestKit.Xunit2;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.AdminUI.Hubs;
|
||||
using ZB.MOM.WW.OtOpcUa.AdminUI.Telemetry;
|
||||
using ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Alerts;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Drivers;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Logging;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.AdminUI.Tests.Telemetry;
|
||||
|
||||
/// <summary>
|
||||
/// Unit tests for <see cref="TelemetryDialSupervisor"/> — the central-side dial supervisor that
|
||||
/// discovers driver nodes, keeps one reconnecting dialer each, and feeds the four AdminUI
|
||||
/// in-process sinks. All seams are faked: no real DB, no real gRPC.
|
||||
/// </summary>
|
||||
public sealed class TelemetryDialSupervisorTests : TestKit
|
||||
{
|
||||
private static readonly TimeSpan Timeout = TimeSpan.FromSeconds(5);
|
||||
|
||||
[Fact]
|
||||
public void Discovery_starts_one_dialer_per_target_with_distinct_correlation_ids()
|
||||
{
|
||||
var dial = new FakeDialLoop();
|
||||
var targets = new[]
|
||||
{
|
||||
new TelemetryDialTarget("node-1", "http://h1:5300"),
|
||||
new TelemetryDialTarget("node-2", "http://h2:5300"),
|
||||
};
|
||||
|
||||
Spawn(dial, () => targets);
|
||||
|
||||
var invocations = dial.WaitForCount(2, Timeout);
|
||||
invocations.Select(i => i.Target.NodeId).ShouldBe(new[] { "node-1", "node-2" }, ignoreOrder: true);
|
||||
invocations.ShouldContain(i => i.CorrelationId == "central-node-1-1");
|
||||
invocations.ShouldContain(i => i.CorrelationId == "central-node-2-1");
|
||||
invocations.ShouldAllBe(i => i.Ct.CanBeCanceled);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Refresh_removes_departed_nodes_and_adds_new_ones()
|
||||
{
|
||||
var dial = new FakeDialLoop();
|
||||
var current = new List<TelemetryDialTarget>
|
||||
{
|
||||
new("node-1", "http://h1:5300"),
|
||||
new("node-2", "http://h2:5300"),
|
||||
};
|
||||
|
||||
var actor = Spawn(dial, () => current.ToArray());
|
||||
dial.WaitForCount(2, Timeout);
|
||||
|
||||
var node1 = dial.Invocations.Single(i => i.Target.NodeId == "node-1");
|
||||
|
||||
// node-1 leaves, node-3 joins.
|
||||
current.Clear();
|
||||
current.Add(new TelemetryDialTarget("node-2", "http://h2:5300"));
|
||||
current.Add(new TelemetryDialTarget("node-3", "http://h3:5300"));
|
||||
actor.Tell(new TelemetryDialSupervisor.RefreshNodes());
|
||||
|
||||
// node-1's dialer is cancelled and dropped.
|
||||
AwaitCondition(() => node1.Ct.IsCancellationRequested, Timeout);
|
||||
// node-3 gets a fresh dialer.
|
||||
var invocations = dial.WaitForCount(3, Timeout);
|
||||
invocations.ShouldContain(i => i.Target.NodeId == "node-3" && i.CorrelationId == "central-node-3-1");
|
||||
|
||||
// node-2 is present in BOTH refreshes with an unchanged endpoint: it must NOT be restarted, or
|
||||
// its live stream would be dropped on every refresh. Exactly one dial invocation for it.
|
||||
ExpectNoMsg(TimeSpan.FromMilliseconds(200));
|
||||
dial.Invocations.Count(i => i.Target.NodeId == "node-2").ShouldBe(1);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Refresh_redials_a_node_whose_endpoint_changed_but_not_an_unchanged_one()
|
||||
{
|
||||
var dial = new FakeDialLoop();
|
||||
var current = new List<TelemetryDialTarget>
|
||||
{
|
||||
new("node-1", "http://old-host:5300"),
|
||||
new("node-2", "http://h2:5300"),
|
||||
};
|
||||
|
||||
var actor = Spawn(dial, () => current.ToArray());
|
||||
dial.WaitForCount(2, Timeout);
|
||||
var node1Original = dial.Invocations.Single(i => i.Target.NodeId == "node-1");
|
||||
|
||||
// node-1 re-provisioned to a new host; node-2 unchanged.
|
||||
current.Clear();
|
||||
current.Add(new TelemetryDialTarget("node-1", "http://new-host:5300"));
|
||||
current.Add(new TelemetryDialTarget("node-2", "http://h2:5300"));
|
||||
actor.Tell(new TelemetryDialSupervisor.RefreshNodes());
|
||||
|
||||
// The old node-1 stream is torn down and a fresh dialer opens at the NEW endpoint (gen 2).
|
||||
AwaitCondition(() => node1Original.Ct.IsCancellationRequested, Timeout);
|
||||
var invocations = dial.WaitForCount(3, Timeout);
|
||||
invocations.ShouldContain(i =>
|
||||
i.Target.NodeId == "node-1"
|
||||
&& i.Target.Endpoint == "http://new-host:5300"
|
||||
&& i.CorrelationId == "central-node-1-2");
|
||||
|
||||
// node-2's endpoint did not change → still exactly one dial invocation (no churn).
|
||||
ExpectNoMsg(TimeSpan.FromMilliseconds(200));
|
||||
dial.Invocations.Count(i => i.Target.NodeId == "node-2").ShouldBe(1);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Each_record_type_lands_on_its_sink()
|
||||
{
|
||||
var dial = new FakeDialLoop();
|
||||
var sinks = new Sinks();
|
||||
Spawn(dial, () => new[] { new TelemetryDialTarget("node-1", "http://h1:5300") }, sinks);
|
||||
|
||||
var inv = dial.WaitForCount(1, Timeout).Single();
|
||||
|
||||
inv.OnMapped(Alarm("a1"));
|
||||
inv.OnMapped(Script("s1"));
|
||||
inv.OnMapped(Health("d1"));
|
||||
inv.OnMapped(Resilience("d1"));
|
||||
|
||||
AwaitAssert(
|
||||
() =>
|
||||
{
|
||||
sinks.Alarms.ShouldHaveSingleItem().AlarmId.ShouldBe("a1");
|
||||
sinks.Scripts.ShouldHaveSingleItem().ScriptId.ShouldBe("s1");
|
||||
sinks.Health.ShouldHaveSingleItem().DriverInstanceId.ShouldBe("d1");
|
||||
sinks.Resilience.ShouldHaveSingleItem().DriverInstanceId.ShouldBe("d1");
|
||||
},
|
||||
Timeout);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Stream_failure_reconnects_with_an_incremented_generation()
|
||||
{
|
||||
var dial = new FakeDialLoop();
|
||||
Spawn(dial, () => new[] { new TelemetryDialTarget("node-1", "http://h1:5300") });
|
||||
|
||||
var first = dial.WaitForCount(1, Timeout).Single();
|
||||
first.CorrelationId.ShouldBe("central-node-1-1");
|
||||
|
||||
// Transport failure → the client would end the stream after onError, so end the fake loop too.
|
||||
first.OnError(new InvalidOperationException("boom"));
|
||||
first.Complete();
|
||||
|
||||
// First retry is immediate; generation is bumped.
|
||||
var invocations = dial.WaitForCount(2, Timeout);
|
||||
invocations[1].CorrelationId.ShouldBe("central-node-1-2");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Late_event_from_a_superseded_generation_is_dropped()
|
||||
{
|
||||
var dial = new FakeDialLoop();
|
||||
var sinks = new Sinks();
|
||||
Spawn(dial, () => new[] { new TelemetryDialTarget("node-1", "http://h1:5300") }, sinks);
|
||||
|
||||
var first = dial.WaitForCount(1, Timeout).Single();
|
||||
first.OnError(new InvalidOperationException("boom"));
|
||||
first.Complete();
|
||||
dial.WaitForCount(2, Timeout); // gen-2 dialer is live.
|
||||
|
||||
// A late event arriving on the OLD (gen-1) stream must be dropped, not routed.
|
||||
first.OnMapped(Alarm("stale"));
|
||||
|
||||
// Give the message a chance to be (wrongly) routed, then assert it was not.
|
||||
ExpectNoMsg(TimeSpan.FromMilliseconds(300));
|
||||
sinks.Alarms.ShouldBeEmpty();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Late_failure_from_a_superseded_generation_does_not_trigger_a_second_reconnect()
|
||||
{
|
||||
var dial = new FakeDialLoop();
|
||||
Spawn(dial, () => new[] { new TelemetryDialTarget("node-1", "http://h1:5300") });
|
||||
|
||||
var first = dial.WaitForCount(1, Timeout).Single();
|
||||
first.OnError(new InvalidOperationException("boom"));
|
||||
first.Complete();
|
||||
dial.WaitForCount(2, Timeout); // gen-2 dialer.
|
||||
|
||||
// A duplicate/late failure from gen-1 must be ignored (no third dial).
|
||||
first.OnError(new InvalidOperationException("late duplicate"));
|
||||
|
||||
ExpectNoMsg(TimeSpan.FromMilliseconds(400));
|
||||
dial.Invocations.Count.ShouldBe(2);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Pill_flips_true_on_first_connection_and_false_only_when_all_down()
|
||||
{
|
||||
var dial = new FakeDialLoop();
|
||||
var sinks = new Sinks();
|
||||
Spawn(
|
||||
dial,
|
||||
() => new[]
|
||||
{
|
||||
new TelemetryDialTarget("node-1", "http://h1:5300"),
|
||||
new TelemetryDialTarget("node-2", "http://h2:5300"),
|
||||
},
|
||||
sinks);
|
||||
|
||||
var inv = dial.WaitForCount(2, Timeout);
|
||||
var node1 = inv.Single(i => i.Target.NodeId == "node-1");
|
||||
var node2 = inv.Single(i => i.Target.NodeId == "node-2");
|
||||
|
||||
// First node connects → both broadcasters flip to connected.
|
||||
node1.OnMapped(Alarm("a1"));
|
||||
AwaitAssert(() => sinks.PillStates.ToArray().ShouldBe(new[] { true }), Timeout);
|
||||
|
||||
// Second node connects → still connected, no extra transition.
|
||||
node2.OnMapped(Alarm("a2"));
|
||||
ExpectNoMsg(TimeSpan.FromMilliseconds(200));
|
||||
sinks.PillStates.ToArray().ShouldBe(new[] { true });
|
||||
|
||||
// node-1 drops but node-2 is still up → pill stays true.
|
||||
node1.OnError(new InvalidOperationException("down"));
|
||||
node1.Complete();
|
||||
ExpectNoMsg(TimeSpan.FromMilliseconds(200));
|
||||
sinks.PillStates.ToArray().ShouldBe(new[] { true });
|
||||
|
||||
// node-2 drops too → all down → pill flips false.
|
||||
node2.OnError(new InvalidOperationException("down"));
|
||||
node2.Complete();
|
||||
AwaitAssert(() => sinks.PillStates.ToArray().ShouldBe(new[] { true, false }), Timeout);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void PostStop_cancels_every_dialer()
|
||||
{
|
||||
var dial = new FakeDialLoop();
|
||||
var actor = Spawn(
|
||||
dial,
|
||||
() => new[]
|
||||
{
|
||||
new TelemetryDialTarget("node-1", "http://h1:5300"),
|
||||
new TelemetryDialTarget("node-2", "http://h2:5300"),
|
||||
});
|
||||
|
||||
var inv = dial.WaitForCount(2, Timeout);
|
||||
|
||||
Sys.Stop(actor);
|
||||
|
||||
AwaitCondition(() => inv.All(i => i.Ct.IsCancellationRequested), Timeout);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Production_node_source_skips_null_grpc_port_and_maintenance_rows()
|
||||
{
|
||||
var factory = new InMemoryFactory(Guid.NewGuid().ToString());
|
||||
await using (var db = factory.CreateDbContext())
|
||||
{
|
||||
db.ClusterNodes.AddRange(
|
||||
Node("has-port", "h1", grpcPort: 5300, enabled: true, maintenance: false),
|
||||
Node("null-port", "h2", grpcPort: null, enabled: true, maintenance: false),
|
||||
Node("maintenance", "h3", grpcPort: 5300, enabled: true, maintenance: true),
|
||||
Node("disabled", "h4", grpcPort: 5300, enabled: false, maintenance: false));
|
||||
await db.SaveChangesAsync();
|
||||
}
|
||||
|
||||
var source = TelemetryNodeSource.Create(factory, NullLogger.Instance);
|
||||
var targets = await source();
|
||||
|
||||
targets.ShouldHaveSingleItem();
|
||||
targets[0].NodeId.ShouldBe("has-port");
|
||||
targets[0].Endpoint.ShouldBe("http://h1:5300");
|
||||
}
|
||||
|
||||
private IActorRef Spawn(
|
||||
FakeDialLoop dial,
|
||||
Func<TelemetryDialTarget[]> nodeSource,
|
||||
Sinks? sinks = null)
|
||||
{
|
||||
sinks ??= new Sinks();
|
||||
var options = new TelemetryDialOptions { ContactRefreshSeconds = 3600 };
|
||||
return Sys.ActorOf(TelemetryDialSupervisor.Props(
|
||||
() => Task.FromResult<IReadOnlyList<TelemetryDialTarget>>(nodeSource()),
|
||||
dial.Loop,
|
||||
sinks.AlarmBroadcaster,
|
||||
sinks.ScriptBroadcaster,
|
||||
sinks.HealthStore,
|
||||
sinks.ResilienceStore,
|
||||
options));
|
||||
}
|
||||
|
||||
private static AlarmTransitionEvent Alarm(string id) =>
|
||||
new(id, "Area/Line/Eq", "hi", "Activated", 500, "msg", "system", DateTime.UtcNow);
|
||||
|
||||
private static ScriptLogEntry Script(string id) =>
|
||||
new(id, "Info", "msg", DateTime.UtcNow, null, null, null);
|
||||
|
||||
private static DriverHealthChanged Health(string id) =>
|
||||
new("C1", id, "Healthy", DateTime.UtcNow, null, 0, DateTime.UtcNow);
|
||||
|
||||
private static DriverResilienceStatusChanged Resilience(string id) =>
|
||||
new(id, "host", false, 0, 0, null, DateTime.UtcNow, DateTime.UtcNow);
|
||||
|
||||
private static ClusterNode Node(string id, string host, int? grpcPort, bool enabled, bool maintenance) =>
|
||||
new()
|
||||
{
|
||||
NodeId = id,
|
||||
ClusterId = "C1",
|
||||
Host = host,
|
||||
GrpcPort = grpcPort,
|
||||
Enabled = enabled,
|
||||
MaintenanceMode = maintenance,
|
||||
ApplicationUri = $"urn:{id}",
|
||||
CreatedBy = "test",
|
||||
};
|
||||
|
||||
/// <summary>A captured dial-loop invocation with the callbacks the test drives by hand.</summary>
|
||||
private sealed class DialInvocation
|
||||
{
|
||||
private readonly TaskCompletionSource _completion = new(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
|
||||
public required TelemetryDialTarget Target { get; init; }
|
||||
public required string CorrelationId { get; init; }
|
||||
public required Action<object> OnMapped { get; init; }
|
||||
public required Action<Exception> OnError { get; init; }
|
||||
public required CancellationToken Ct { get; init; }
|
||||
|
||||
public Task Completion => _completion.Task;
|
||||
|
||||
/// <summary>Ends the fake loop (as the client returns after a server-ended or post-onError stream).</summary>
|
||||
public void Complete() => _completion.TrySetResult();
|
||||
}
|
||||
|
||||
/// <summary>A fake <see cref="TelemetryDialLoop"/> that records each invocation and blocks until told to end.</summary>
|
||||
private sealed class FakeDialLoop
|
||||
{
|
||||
private readonly ConcurrentQueue<DialInvocation> _invocations = new();
|
||||
|
||||
public IReadOnlyList<DialInvocation> Invocations => _invocations.ToArray();
|
||||
|
||||
public TelemetryDialLoop Loop => async (target, correlationId, onMapped, onError, ct) =>
|
||||
{
|
||||
var invocation = new DialInvocation
|
||||
{
|
||||
Target = target,
|
||||
CorrelationId = correlationId,
|
||||
OnMapped = onMapped,
|
||||
OnError = onError,
|
||||
Ct = ct,
|
||||
};
|
||||
_invocations.Enqueue(invocation);
|
||||
|
||||
// Complete when the test ends the loop OR the supervisor cancels us.
|
||||
await using var reg = ct.Register(() => invocation.Complete());
|
||||
await invocation.Completion;
|
||||
};
|
||||
|
||||
public IReadOnlyList<DialInvocation> WaitForCount(int count, TimeSpan timeout)
|
||||
{
|
||||
var deadline = DateTime.UtcNow + timeout;
|
||||
while (DateTime.UtcNow < deadline)
|
||||
{
|
||||
var snapshot = _invocations.ToArray();
|
||||
if (snapshot.Length >= count)
|
||||
{
|
||||
return snapshot;
|
||||
}
|
||||
|
||||
Thread.Sleep(10);
|
||||
}
|
||||
|
||||
throw new Xunit.Sdk.XunitException(
|
||||
$"Expected at least {count} dial invocations within {timeout}, saw {_invocations.Count}.");
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>The four fake sinks plus the pill-transition record.</summary>
|
||||
private sealed class Sinks
|
||||
{
|
||||
public RecordingBroadcaster<AlarmTransitionEvent> AlarmBroadcaster { get; }
|
||||
public RecordingBroadcaster<ScriptLogEntry> ScriptBroadcaster { get; }
|
||||
public RecordingHealthStore HealthStore { get; } = new();
|
||||
public RecordingResilienceStore ResilienceStore { get; } = new();
|
||||
|
||||
public ConcurrentQueue<AlarmTransitionEvent> Alarms { get; } = new();
|
||||
public ConcurrentQueue<ScriptLogEntry> Scripts { get; } = new();
|
||||
public ConcurrentQueue<bool> PillStates { get; } = new();
|
||||
|
||||
public ConcurrentQueue<DriverHealthChanged> Health => HealthStore.Upserts;
|
||||
public ConcurrentQueue<DriverResilienceStatusChanged> Resilience => ResilienceStore.Upserts;
|
||||
|
||||
public Sinks()
|
||||
{
|
||||
// Both broadcasters share the ONE pill-transition record so a test can assert the alarm and
|
||||
// script pills move together (matching the supervisor calling SetConnected on both).
|
||||
AlarmBroadcaster = new RecordingBroadcaster<AlarmTransitionEvent>(Alarms, PillStates);
|
||||
ScriptBroadcaster = new RecordingBroadcaster<ScriptLogEntry>(Scripts, null);
|
||||
}
|
||||
}
|
||||
|
||||
private sealed class RecordingBroadcaster<T>(ConcurrentQueue<T> published, ConcurrentQueue<bool>? pill)
|
||||
: IInProcessBroadcaster<T>
|
||||
{
|
||||
public event Action<T>? Received;
|
||||
public event Action<bool>? ConnectionStateChanged;
|
||||
|
||||
public bool IsConnected { get; private set; }
|
||||
|
||||
public void Publish(T item)
|
||||
{
|
||||
published.Enqueue(item);
|
||||
Received?.Invoke(item);
|
||||
}
|
||||
|
||||
public void SetConnected(bool connected)
|
||||
{
|
||||
IsConnected = connected;
|
||||
pill?.Enqueue(connected);
|
||||
ConnectionStateChanged?.Invoke(connected);
|
||||
}
|
||||
}
|
||||
|
||||
private sealed class RecordingHealthStore : IDriverStatusSnapshotStore
|
||||
{
|
||||
public ConcurrentQueue<DriverHealthChanged> Upserts { get; } = new();
|
||||
|
||||
public event Action<DriverHealthChanged>? SnapshotChanged;
|
||||
|
||||
public void Upsert(DriverHealthChanged snapshot)
|
||||
{
|
||||
Upserts.Enqueue(snapshot);
|
||||
SnapshotChanged?.Invoke(snapshot);
|
||||
}
|
||||
|
||||
public bool TryGet(string driverInstanceId, out DriverHealthChanged snapshot)
|
||||
{
|
||||
snapshot = null!;
|
||||
return false;
|
||||
}
|
||||
|
||||
public IReadOnlyCollection<DriverHealthChanged> GetAll() => Upserts.ToArray();
|
||||
}
|
||||
|
||||
private sealed class RecordingResilienceStore : IDriverResilienceStatusStore
|
||||
{
|
||||
public ConcurrentQueue<DriverResilienceStatusChanged> Upserts { get; } = new();
|
||||
|
||||
public event Action<DriverResilienceStatusChanged>? SnapshotChanged;
|
||||
|
||||
public void Upsert(DriverResilienceStatusChanged snapshot)
|
||||
{
|
||||
Upserts.Enqueue(snapshot);
|
||||
SnapshotChanged?.Invoke(snapshot);
|
||||
}
|
||||
|
||||
public IReadOnlyList<DriverResilienceStatusChanged> GetForInstance(string driverInstanceId) =>
|
||||
Upserts.ToArray();
|
||||
|
||||
public IReadOnlyCollection<DriverResilienceStatusChanged> GetAll() => Upserts.ToArray();
|
||||
}
|
||||
|
||||
private sealed class InMemoryFactory(string name) : IDbContextFactory<OtOpcUaConfigDbContext>
|
||||
{
|
||||
public OtOpcUaConfigDbContext CreateDbContext() =>
|
||||
new(new DbContextOptionsBuilder<OtOpcUaConfigDbContext>()
|
||||
.UseInMemoryDatabase(name)
|
||||
.Options);
|
||||
|
||||
public Task<OtOpcUaConfigDbContext> CreateDbContextAsync(CancellationToken cancellationToken = default) =>
|
||||
Task.FromResult(CreateDbContext());
|
||||
}
|
||||
}
|
||||
+360
@@ -0,0 +1,360 @@
|
||||
using Google.Protobuf.WellKnownTypes;
|
||||
using Shouldly;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Protos;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Protos.Telemetry.V1;
|
||||
using ZB.MOM.WW.OtOpcUa.ControlPlane.Telemetry;
|
||||
using Xunit;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Telemetry;
|
||||
|
||||
/// <summary>
|
||||
/// Unit tests for <see cref="TelemetryProtoMapCentral"/> — the central-side proto→domain
|
||||
/// projection. Each kind asserts every field round-trips, including the null-vs-absent presence
|
||||
/// handling for nullable Timestamps, optional strings, and the tri-state optional bool. A coverage
|
||||
/// guard proves every <see cref="TelemetryProtoContract.HandledCases"/> value has a converter.
|
||||
/// </summary>
|
||||
public sealed class TelemetryProtoMapCentralTests
|
||||
{
|
||||
private static readonly DateTime SampleUtc =
|
||||
new(2026, 7, 23, 10, 30, 45, DateTimeKind.Utc);
|
||||
|
||||
private static readonly DateTime OtherUtc =
|
||||
new(2026, 7, 23, 9, 15, 0, DateTimeKind.Utc);
|
||||
|
||||
[Fact]
|
||||
public void ToAlarm_transcribes_every_field_with_nullables_present()
|
||||
{
|
||||
var proto = new AlarmTransition
|
||||
{
|
||||
AlarmId = "Plant/Modbus/dev1/Speed",
|
||||
EquipmentPath = "Area/Line/Equip",
|
||||
AlarmName = "HighSpeed",
|
||||
TransitionKind = "Activated",
|
||||
Severity = 700,
|
||||
Message = "Speed high",
|
||||
User = "operator1",
|
||||
TimestampUtc = Timestamp.FromDateTime(SampleUtc),
|
||||
AlarmTypeName = "LimitAlarm",
|
||||
Comment = "ack comment",
|
||||
HistorizeToAveva = false,
|
||||
};
|
||||
proto.ReferencingEquipmentPaths.AddRange(["Area/Line/E1", "Area/Line/E2"]);
|
||||
|
||||
var e = TelemetryProtoMapCentral.ToAlarm(proto);
|
||||
|
||||
e.AlarmId.ShouldBe("Plant/Modbus/dev1/Speed");
|
||||
e.EquipmentPath.ShouldBe("Area/Line/Equip");
|
||||
e.AlarmName.ShouldBe("HighSpeed");
|
||||
e.TransitionKind.ShouldBe("Activated");
|
||||
e.Severity.ShouldBe(700);
|
||||
e.Message.ShouldBe("Speed high");
|
||||
e.User.ShouldBe("operator1");
|
||||
e.TimestampUtc.ShouldBe(SampleUtc);
|
||||
e.TimestampUtc.Kind.ShouldBe(DateTimeKind.Utc);
|
||||
e.AlarmTypeName.ShouldBe("LimitAlarm");
|
||||
e.Comment.ShouldBe("ack comment");
|
||||
e.HistorizeToAveva.ShouldBe(false);
|
||||
e.ReferencingEquipmentPaths.ShouldBe(["Area/Line/E1", "Area/Line/E2"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ToAlarm_absent_optionals_map_to_null_and_empty_repeated_to_empty_list()
|
||||
{
|
||||
var proto = new AlarmTransition
|
||||
{
|
||||
AlarmId = "a",
|
||||
EquipmentPath = "p",
|
||||
AlarmName = "n",
|
||||
TransitionKind = "Cleared",
|
||||
Severity = 1,
|
||||
Message = "m",
|
||||
User = "system",
|
||||
TimestampUtc = Timestamp.FromDateTime(SampleUtc),
|
||||
AlarmTypeName = "AlarmCondition",
|
||||
// Comment, HistorizeToAveva unset; no referencing paths added.
|
||||
};
|
||||
|
||||
var e = TelemetryProtoMapCentral.ToAlarm(proto);
|
||||
|
||||
e.Comment.ShouldBeNull();
|
||||
e.HistorizeToAveva.ShouldBeNull();
|
||||
e.ReferencingEquipmentPaths.ShouldNotBeNull();
|
||||
e.ReferencingEquipmentPaths.ShouldBeEmpty();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ToAlarm_historize_present_true_round_trips()
|
||||
{
|
||||
var proto = new AlarmTransition
|
||||
{
|
||||
AlarmId = "a", EquipmentPath = "p", AlarmName = "n", TransitionKind = "Activated",
|
||||
Severity = 1, Message = "m", User = "system",
|
||||
TimestampUtc = Timestamp.FromDateTime(SampleUtc), AlarmTypeName = "AlarmCondition",
|
||||
HistorizeToAveva = true,
|
||||
};
|
||||
|
||||
TelemetryProtoMapCentral.ToAlarm(proto).HistorizeToAveva.ShouldBe(true);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ToScript_transcribes_every_field_with_optionals_present()
|
||||
{
|
||||
var proto = new ScriptLog
|
||||
{
|
||||
ScriptId = "script-1",
|
||||
Level = "Information",
|
||||
Message = "hello",
|
||||
TimestampUtc = Timestamp.FromDateTime(SampleUtc),
|
||||
VirtualTagId = "vt-1",
|
||||
AlarmId = "al-1",
|
||||
EquipmentId = "eq-1",
|
||||
};
|
||||
|
||||
var e = TelemetryProtoMapCentral.ToScript(proto);
|
||||
|
||||
e.ScriptId.ShouldBe("script-1");
|
||||
e.Level.ShouldBe("Information");
|
||||
e.Message.ShouldBe("hello");
|
||||
e.TimestampUtc.ShouldBe(SampleUtc);
|
||||
e.TimestampUtc.Kind.ShouldBe(DateTimeKind.Utc);
|
||||
e.VirtualTagId.ShouldBe("vt-1");
|
||||
e.AlarmId.ShouldBe("al-1");
|
||||
e.EquipmentId.ShouldBe("eq-1");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ToScript_absent_optionals_map_to_null()
|
||||
{
|
||||
var proto = new ScriptLog
|
||||
{
|
||||
ScriptId = "script-1",
|
||||
Level = "Error",
|
||||
Message = "boom",
|
||||
TimestampUtc = Timestamp.FromDateTime(SampleUtc),
|
||||
// VirtualTagId, AlarmId, EquipmentId unset.
|
||||
};
|
||||
|
||||
var e = TelemetryProtoMapCentral.ToScript(proto);
|
||||
|
||||
e.VirtualTagId.ShouldBeNull();
|
||||
e.AlarmId.ShouldBeNull();
|
||||
e.EquipmentId.ShouldBeNull();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ToHealth_transcribes_every_field_with_nullables_present()
|
||||
{
|
||||
var proto = new DriverHealth
|
||||
{
|
||||
ClusterId = "c1",
|
||||
DriverInstanceId = "d1",
|
||||
State = "Faulted",
|
||||
LastSuccessfulReadUtc = Timestamp.FromDateTime(OtherUtc),
|
||||
LastError = "timeout",
|
||||
ErrorCount5Min = 3,
|
||||
PublishedUtc = Timestamp.FromDateTime(SampleUtc),
|
||||
};
|
||||
|
||||
var e = TelemetryProtoMapCentral.ToHealth(proto);
|
||||
|
||||
e.ClusterId.ShouldBe("c1");
|
||||
e.DriverInstanceId.ShouldBe("d1");
|
||||
e.State.ShouldBe("Faulted");
|
||||
e.LastSuccessfulReadUtc.ShouldBe(OtherUtc);
|
||||
e.LastSuccessfulReadUtc!.Value.Kind.ShouldBe(DateTimeKind.Utc);
|
||||
e.LastError.ShouldBe("timeout");
|
||||
e.ErrorCount5Min.ShouldBe(3);
|
||||
e.PublishedUtc.ShouldBe(SampleUtc);
|
||||
e.PublishedUtc.Kind.ShouldBe(DateTimeKind.Utc);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ToHealth_absent_nullable_timestamp_and_optional_string_map_to_null()
|
||||
{
|
||||
var proto = new DriverHealth
|
||||
{
|
||||
ClusterId = "c1",
|
||||
DriverInstanceId = "d1",
|
||||
State = "Healthy",
|
||||
ErrorCount5Min = 0,
|
||||
PublishedUtc = Timestamp.FromDateTime(SampleUtc),
|
||||
// LastSuccessfulReadUtc, LastError unset.
|
||||
};
|
||||
|
||||
var e = TelemetryProtoMapCentral.ToHealth(proto);
|
||||
|
||||
e.LastSuccessfulReadUtc.ShouldBeNull();
|
||||
e.LastError.ShouldBeNull();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ToResilience_transcribes_every_field_with_nullable_present()
|
||||
{
|
||||
var proto = new DriverResilienceStatus
|
||||
{
|
||||
DriverInstanceId = "d1",
|
||||
HostName = "host-a",
|
||||
BreakerOpen = true,
|
||||
ConsecutiveFailures = 5,
|
||||
CurrentInFlight = 2,
|
||||
LastBreakerOpenUtc = Timestamp.FromDateTime(OtherUtc),
|
||||
LastSampledUtc = Timestamp.FromDateTime(SampleUtc),
|
||||
PublishedUtc = Timestamp.FromDateTime(SampleUtc),
|
||||
};
|
||||
|
||||
var e = TelemetryProtoMapCentral.ToResilience(proto);
|
||||
|
||||
e.DriverInstanceId.ShouldBe("d1");
|
||||
e.HostName.ShouldBe("host-a");
|
||||
e.BreakerOpen.ShouldBeTrue();
|
||||
e.ConsecutiveFailures.ShouldBe(5);
|
||||
e.CurrentInFlight.ShouldBe(2);
|
||||
e.LastBreakerOpenUtc.ShouldBe(OtherUtc);
|
||||
e.LastBreakerOpenUtc!.Value.Kind.ShouldBe(DateTimeKind.Utc);
|
||||
e.LastSampledUtc.ShouldBe(SampleUtc);
|
||||
e.PublishedUtc.ShouldBe(SampleUtc);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ToResilience_absent_nullable_timestamp_maps_to_null()
|
||||
{
|
||||
var proto = new DriverResilienceStatus
|
||||
{
|
||||
DriverInstanceId = "d1",
|
||||
HostName = "host-a",
|
||||
BreakerOpen = false,
|
||||
ConsecutiveFailures = 0,
|
||||
CurrentInFlight = 0,
|
||||
LastSampledUtc = Timestamp.FromDateTime(SampleUtc),
|
||||
PublishedUtc = Timestamp.FromDateTime(SampleUtc),
|
||||
// LastBreakerOpenUtc unset.
|
||||
};
|
||||
|
||||
TelemetryProtoMapCentral.ToResilience(proto).LastBreakerOpenUtc.ShouldBeNull();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ToAlarm_missing_required_timestamp_throws_clear_InvalidOperation()
|
||||
{
|
||||
var proto = new AlarmTransition
|
||||
{
|
||||
AlarmId = "a", EquipmentPath = "p", AlarmName = "n", TransitionKind = "Activated",
|
||||
Severity = 1, Message = "m", User = "system", AlarmTypeName = "AlarmCondition",
|
||||
// TimestampUtc deliberately unset — an out-of-contract sender.
|
||||
};
|
||||
|
||||
var ex = Should.Throw<InvalidOperationException>(() => TelemetryProtoMapCentral.ToAlarm(proto));
|
||||
ex.Message.ShouldContain("AlarmTransition");
|
||||
ex.Message.ShouldContain("timestamp_utc");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ToScript_missing_required_timestamp_throws_clear_InvalidOperation()
|
||||
{
|
||||
var proto = new ScriptLog { ScriptId = "s", Level = "Information", Message = "m" };
|
||||
|
||||
var ex = Should.Throw<InvalidOperationException>(() => TelemetryProtoMapCentral.ToScript(proto));
|
||||
ex.Message.ShouldContain("ScriptLog");
|
||||
ex.Message.ShouldContain("timestamp_utc");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ToHealth_missing_required_timestamp_throws_clear_InvalidOperation()
|
||||
{
|
||||
var proto = new DriverHealth
|
||||
{
|
||||
ClusterId = "c", DriverInstanceId = "d", State = "Healthy", ErrorCount5Min = 0,
|
||||
// PublishedUtc unset.
|
||||
};
|
||||
|
||||
var ex = Should.Throw<InvalidOperationException>(() => TelemetryProtoMapCentral.ToHealth(proto));
|
||||
ex.Message.ShouldContain("DriverHealth");
|
||||
ex.Message.ShouldContain("published_utc");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ToResilience_missing_required_timestamp_throws_clear_InvalidOperation()
|
||||
{
|
||||
var proto = new DriverResilienceStatus
|
||||
{
|
||||
DriverInstanceId = "d", HostName = "h", BreakerOpen = false,
|
||||
ConsecutiveFailures = 0, CurrentInFlight = 0,
|
||||
// LastSampledUtc + PublishedUtc unset.
|
||||
};
|
||||
|
||||
var ex = Should.Throw<InvalidOperationException>(() => TelemetryProtoMapCentral.ToResilience(proto));
|
||||
ex.Message.ShouldContain("DriverResilienceStatus");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void MapEvent_routes_each_case_to_its_record_type()
|
||||
{
|
||||
var alarm = new TelemetryEvent { AlarmTransition = MinimalAlarm() };
|
||||
var script = new TelemetryEvent { ScriptLog = MinimalScript() };
|
||||
var health = new TelemetryEvent { DriverHealth = MinimalHealth() };
|
||||
var resilience = new TelemetryEvent { DriverResilience = MinimalResilience() };
|
||||
|
||||
TelemetryProtoMapCentral.MapEvent(alarm).ShouldBeOfType<Commons.Messages.Alerts.AlarmTransitionEvent>();
|
||||
TelemetryProtoMapCentral.MapEvent(script).ShouldBeOfType<Commons.Messages.Logging.ScriptLogEntry>();
|
||||
TelemetryProtoMapCentral.MapEvent(health).ShouldBeOfType<Commons.Messages.Drivers.DriverHealthChanged>();
|
||||
TelemetryProtoMapCentral.MapEvent(resilience).ShouldBeOfType<Commons.Messages.Drivers.DriverResilienceStatusChanged>();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void MapEvent_unset_case_throws_NotSupported()
|
||||
{
|
||||
Should.Throw<NotSupportedException>(() => TelemetryProtoMapCentral.MapEvent(new TelemetryEvent()));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Coverage guard: every <see cref="TelemetryProtoContract.HandledCases"/> value must be routed
|
||||
/// by <see cref="TelemetryProtoMapCentral.MapEvent"/> to a non-null domain record. Adding a
|
||||
/// fifth oneof case (and thus a fifth HandledCases entry) without a converter arm fails here.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void MapEvent_covers_every_handled_case()
|
||||
{
|
||||
foreach (var handled in TelemetryProtoContract.HandledCases)
|
||||
{
|
||||
var evt = BuildFor(handled);
|
||||
evt.EventCase.ShouldBe(handled);
|
||||
TelemetryProtoMapCentral.MapEvent(evt).ShouldNotBeNull();
|
||||
}
|
||||
}
|
||||
|
||||
private static TelemetryEvent BuildFor(TelemetryEvent.EventOneofCase handled) => handled switch
|
||||
{
|
||||
TelemetryEvent.EventOneofCase.AlarmTransition => new TelemetryEvent { AlarmTransition = MinimalAlarm() },
|
||||
TelemetryEvent.EventOneofCase.ScriptLog => new TelemetryEvent { ScriptLog = MinimalScript() },
|
||||
TelemetryEvent.EventOneofCase.DriverHealth => new TelemetryEvent { DriverHealth = MinimalHealth() },
|
||||
TelemetryEvent.EventOneofCase.DriverResilience => new TelemetryEvent { DriverResilience = MinimalResilience() },
|
||||
_ => throw new InvalidOperationException($"Test does not know how to build case {handled}."),
|
||||
};
|
||||
|
||||
private static AlarmTransition MinimalAlarm() => new()
|
||||
{
|
||||
AlarmId = "a", EquipmentPath = "p", AlarmName = "n", TransitionKind = "Activated",
|
||||
Severity = 1, Message = "m", User = "system",
|
||||
TimestampUtc = Timestamp.FromDateTime(SampleUtc), AlarmTypeName = "AlarmCondition",
|
||||
};
|
||||
|
||||
private static ScriptLog MinimalScript() => new()
|
||||
{
|
||||
ScriptId = "s", Level = "Information", Message = "m",
|
||||
TimestampUtc = Timestamp.FromDateTime(SampleUtc),
|
||||
};
|
||||
|
||||
private static DriverHealth MinimalHealth() => new()
|
||||
{
|
||||
ClusterId = "c", DriverInstanceId = "d", State = "Healthy",
|
||||
ErrorCount5Min = 0, PublishedUtc = Timestamp.FromDateTime(SampleUtc),
|
||||
};
|
||||
|
||||
private static DriverResilienceStatus MinimalResilience() => new()
|
||||
{
|
||||
DriverInstanceId = "d", HostName = "h", BreakerOpen = false,
|
||||
ConsecutiveFailures = 0, CurrentInFlight = 0,
|
||||
LastSampledUtc = Timestamp.FromDateTime(SampleUtc),
|
||||
PublishedUtc = Timestamp.FromDateTime(SampleUtc),
|
||||
};
|
||||
}
|
||||
+225
@@ -0,0 +1,225 @@
|
||||
using Google.Protobuf.WellKnownTypes;
|
||||
using Grpc.Core;
|
||||
using Shouldly;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Protos.Telemetry.V1;
|
||||
using ZB.MOM.WW.OtOpcUa.ControlPlane.Telemetry;
|
||||
using Xunit;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Telemetry;
|
||||
|
||||
/// <summary>
|
||||
/// Unit tests for <see cref="TelemetryStreamClient"/> driving a fake generated client (the ctor
|
||||
/// seam), so the pump/normal-shutdown/error-routing behaviour is provable without a live gRPC
|
||||
/// server. The wire itself is covered by the Phase 5 live gate. The central assertion throughout is
|
||||
/// the onError contract: only a transport fault reaches onError; programming faults throw
|
||||
/// synchronously and consumer-callback faults are isolated.
|
||||
/// </summary>
|
||||
public sealed class TelemetryStreamClientTests
|
||||
{
|
||||
[Fact]
|
||||
public async Task RunAsync_pumps_every_event_to_onEvent()
|
||||
{
|
||||
var events = new[]
|
||||
{
|
||||
new TelemetryEvent { ScriptLog = Script("one") },
|
||||
new TelemetryEvent { ScriptLog = Script("two") },
|
||||
};
|
||||
var fake = new FakeClient(new FakeStreamReader(events));
|
||||
using var sut = new TelemetryStreamClient(fake, "key");
|
||||
|
||||
var received = new List<TelemetryEvent>();
|
||||
Exception? errored = null;
|
||||
|
||||
await sut.RunAsync("corr-1", received.Add, ex => errored = ex, CancellationToken.None);
|
||||
|
||||
received.Count.ShouldBe(2);
|
||||
received[0].ScriptLog.Message.ShouldBe("one");
|
||||
received[1].ScriptLog.Message.ShouldBe("two");
|
||||
errored.ShouldBeNull();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RunAsync_swallows_rpc_cancelled_without_calling_onError()
|
||||
{
|
||||
var fake = new FakeClient(new FakeStreamReader(
|
||||
[new TelemetryEvent { ScriptLog = Script("one") }],
|
||||
throwAtEnd: new RpcException(new Status(StatusCode.Cancelled, "cancelled"))));
|
||||
using var sut = new TelemetryStreamClient(fake, "key");
|
||||
|
||||
var received = new List<TelemetryEvent>();
|
||||
Exception? errored = null;
|
||||
|
||||
await sut.RunAsync("corr-1", received.Add, ex => errored = ex, CancellationToken.None);
|
||||
|
||||
received.Count.ShouldBe(1);
|
||||
errored.ShouldBeNull();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RunAsync_cancelled_token_ends_without_calling_onError()
|
||||
{
|
||||
// The primary shutdown branch the supervisor's routine reconnect-teardown drives: the token
|
||||
// fires mid-stream, the transport surfaces an OperationCanceledException, and RunAsync returns
|
||||
// cleanly — never onError.
|
||||
using var cts = new CancellationTokenSource();
|
||||
var fake = new FakeClient(new FakeStreamReader(
|
||||
[new TelemetryEvent { ScriptLog = Script("one") }],
|
||||
cancelAfterFirst: cts));
|
||||
using var sut = new TelemetryStreamClient(fake, "key");
|
||||
|
||||
var received = new List<TelemetryEvent>();
|
||||
Exception? errored = null;
|
||||
|
||||
await sut.RunAsync("corr-1", received.Add, ex => errored = ex, cts.Token);
|
||||
|
||||
received.Count.ShouldBe(1);
|
||||
errored.ShouldBeNull();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RunAsync_routes_real_rpc_failure_to_onError()
|
||||
{
|
||||
var boom = new RpcException(new Status(StatusCode.Unavailable, "node down"));
|
||||
var fake = new FakeClient(new FakeStreamReader([], throwAtEnd: boom));
|
||||
using var sut = new TelemetryStreamClient(fake, "key");
|
||||
|
||||
Exception? errored = null;
|
||||
|
||||
await sut.RunAsync("corr-1", _ => { }, ex => errored = ex, CancellationToken.None);
|
||||
|
||||
errored.ShouldBeSameAs(boom);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RunAsync_onEvent_exception_is_isolated_pump_continues_and_onError_not_called()
|
||||
{
|
||||
var events = new[]
|
||||
{
|
||||
new TelemetryEvent { ScriptLog = Script("poison") },
|
||||
new TelemetryEvent { ScriptLog = Script("healthy") },
|
||||
};
|
||||
var fake = new FakeClient(new FakeStreamReader(events));
|
||||
using var sut = new TelemetryStreamClient(fake, "key");
|
||||
|
||||
var received = new List<string>();
|
||||
Exception? errored = null;
|
||||
|
||||
await sut.RunAsync(
|
||||
"corr-1",
|
||||
evt =>
|
||||
{
|
||||
if (evt.ScriptLog.Message == "poison")
|
||||
throw new InvalidOperationException("unmappable event");
|
||||
received.Add(evt.ScriptLog.Message);
|
||||
},
|
||||
ex => errored = ex,
|
||||
CancellationToken.None);
|
||||
|
||||
// The poison event was dropped, the pump continued to the next event, and onError never fired.
|
||||
received.ShouldBe(["healthy"]);
|
||||
errored.ShouldBeNull();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RunAsync_empty_correlationId_throws_synchronously_and_never_calls_onError()
|
||||
{
|
||||
var fake = new FakeClient(new FakeStreamReader([]));
|
||||
using var sut = new TelemetryStreamClient(fake, "key");
|
||||
|
||||
Exception? errored = null;
|
||||
|
||||
await Should.ThrowAsync<ArgumentException>(() =>
|
||||
sut.RunAsync("", _ => { }, ex => errored = ex, CancellationToken.None));
|
||||
|
||||
errored.ShouldBeNull();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RunAsync_after_dispose_throws_ObjectDisposed_and_never_calls_onError()
|
||||
{
|
||||
var fake = new FakeClient(new FakeStreamReader([]));
|
||||
var sut = new TelemetryStreamClient(fake, "key");
|
||||
sut.Dispose();
|
||||
|
||||
Exception? errored = null;
|
||||
|
||||
await Should.ThrowAsync<ObjectDisposedException>(() =>
|
||||
sut.RunAsync("corr-1", _ => { }, ex => errored = ex, CancellationToken.None));
|
||||
|
||||
errored.ShouldBeNull();
|
||||
}
|
||||
|
||||
private static ScriptLog Script(string message) => new()
|
||||
{
|
||||
ScriptId = "s",
|
||||
Level = "Information",
|
||||
Message = message,
|
||||
TimestampUtc = Timestamp.FromDateTime(new DateTime(2026, 7, 23, 0, 0, 0, DateTimeKind.Utc)),
|
||||
};
|
||||
|
||||
/// <summary>Fake generated client whose Subscribe returns a canned server-streaming call.</summary>
|
||||
private sealed class FakeClient : TelemetryStreamService.TelemetryStreamServiceClient
|
||||
{
|
||||
private readonly IAsyncStreamReader<TelemetryEvent> _reader;
|
||||
|
||||
public FakeClient(IAsyncStreamReader<TelemetryEvent> reader) => _reader = reader;
|
||||
|
||||
public override AsyncServerStreamingCall<TelemetryEvent> Subscribe(
|
||||
TelemetryStreamRequest request,
|
||||
Metadata? headers = null,
|
||||
DateTime? deadline = null,
|
||||
CancellationToken cancellationToken = default) =>
|
||||
new(
|
||||
_reader,
|
||||
Task.FromResult(new Metadata()),
|
||||
() => Status.DefaultSuccess,
|
||||
() => [],
|
||||
() => { });
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Replays a fixed sequence of events. Honors the pump's <see cref="CancellationToken"/> (so a
|
||||
/// cancelled token surfaces as <see cref="OperationCanceledException"/> just as the real
|
||||
/// transport does), can throw a supplied exception at the end of the sequence, and can cancel a
|
||||
/// supplied source after delivering the first item (to drive the mid-stream cancel branch).
|
||||
/// </summary>
|
||||
private sealed class FakeStreamReader : IAsyncStreamReader<TelemetryEvent>
|
||||
{
|
||||
private readonly IReadOnlyList<TelemetryEvent> _items;
|
||||
private readonly Exception? _throwAtEnd;
|
||||
private readonly CancellationTokenSource? _cancelAfterFirst;
|
||||
private int _index = -1;
|
||||
|
||||
public FakeStreamReader(
|
||||
IReadOnlyList<TelemetryEvent> items,
|
||||
Exception? throwAtEnd = null,
|
||||
CancellationTokenSource? cancelAfterFirst = null)
|
||||
{
|
||||
_items = items;
|
||||
_throwAtEnd = throwAtEnd;
|
||||
_cancelAfterFirst = cancelAfterFirst;
|
||||
}
|
||||
|
||||
public TelemetryEvent Current => _items[_index];
|
||||
|
||||
public Task<bool> MoveNext(CancellationToken cancellationToken)
|
||||
{
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
|
||||
_index++;
|
||||
if (_index < _items.Count)
|
||||
{
|
||||
// After delivering the first item, request cancellation so the NEXT MoveNext throws
|
||||
// OperationCanceledException at the top — the transport-cancel shape.
|
||||
if (_index == 0)
|
||||
_cancelAfterFirst?.Cancel();
|
||||
return Task.FromResult(true);
|
||||
}
|
||||
|
||||
if (_throwAtEnd is not null)
|
||||
throw _throwAtEnd;
|
||||
|
||||
return Task.FromResult(false);
|
||||
}
|
||||
}
|
||||
}
|
||||
+82
@@ -1,7 +1,12 @@
|
||||
using Akka.Actor;
|
||||
using Akka.Cluster;
|
||||
using Akka.Configuration;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Resilience;
|
||||
using ZB.MOM.WW.OtOpcUa.Host.Drivers;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Telemetry;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
|
||||
|
||||
@@ -53,4 +58,81 @@ public sealed class DriverResilienceStatusPublisherServiceTests
|
||||
healthy.CurrentInFlight.ShouldBe(1);
|
||||
healthy.PublishedUtc.ShouldBe(Published);
|
||||
}
|
||||
|
||||
/// <summary>Per-cluster mesh Phase 5 — each snapshot the publish tick sends to the
|
||||
/// <c>driver-resilience-status</c> DPS topic is also emitted into the node-local
|
||||
/// <see cref="ITelemetryLocalHub"/> as a <see cref="TelemetryItem.Resilience"/>. Runs the background
|
||||
/// service against a single-node cluster (so DistributedPubSub resolves) for a few ticks and asserts the
|
||||
/// hub captured the tracked pair.</summary>
|
||||
[Fact]
|
||||
public async Task ExecuteAsync_emits_Resilience_item_per_published_snapshot()
|
||||
{
|
||||
// Plain self-joined single-node cluster (mirrors SecretsReplicationRegistrationTests — the
|
||||
// xunit.v3 project cannot use Akka.TestKit.Xunit2). public-hostname is load-bearing for the bind.
|
||||
var hocon = ConfigurationFactory.ParseString("""
|
||||
akka {
|
||||
loglevel = WARNING
|
||||
actor.provider = cluster
|
||||
remote.dot-netty.tcp {
|
||||
hostname = "127.0.0.1"
|
||||
public-hostname = "127.0.0.1"
|
||||
port = 0
|
||||
}
|
||||
cluster { roles = ["driver"] }
|
||||
}
|
||||
""");
|
||||
var sys = ActorSystem.Create("resil-hub-test", hocon);
|
||||
try
|
||||
{
|
||||
var cluster = Akka.Cluster.Cluster.Get(sys);
|
||||
cluster.Join(cluster.SelfAddress);
|
||||
var upDeadline = DateTime.UtcNow.AddSeconds(10);
|
||||
while (!cluster.State.Members.Any(m => m.Status == MemberStatus.Up) && DateTime.UtcNow < upDeadline)
|
||||
await Task.Delay(50, TestContext.Current.CancellationToken);
|
||||
cluster.State.Members.ShouldContain(m => m.Status == MemberStatus.Up);
|
||||
|
||||
var tracker = new DriverResilienceStatusTracker();
|
||||
tracker.RecordCallStart("drv-1", "host-a");
|
||||
|
||||
var hub = new CapturingHub();
|
||||
var service = new DriverResilienceStatusPublisherService(
|
||||
tracker, () => sys, NullLogger<DriverResilienceStatusPublisherService>.Instance, hub,
|
||||
interval: TimeSpan.FromMilliseconds(50));
|
||||
|
||||
await service.StartAsync(TestContext.Current.CancellationToken);
|
||||
try
|
||||
{
|
||||
var deadline = DateTime.UtcNow.AddSeconds(5);
|
||||
while (hub.Items.Count == 0 && DateTime.UtcNow < deadline)
|
||||
await Task.Delay(50, TestContext.Current.CancellationToken);
|
||||
}
|
||||
finally
|
||||
{
|
||||
await service.StopAsync(TestContext.Current.CancellationToken);
|
||||
}
|
||||
|
||||
hub.Items.ShouldNotBeEmpty();
|
||||
var item = hub.Items[0].ShouldBeOfType<TelemetryItem.Resilience>();
|
||||
item.E.DriverInstanceId.ShouldBe("drv-1");
|
||||
item.E.HostName.ShouldBe("host-a");
|
||||
item.E.CurrentInFlight.ShouldBe(1);
|
||||
}
|
||||
finally
|
||||
{
|
||||
await sys.Terminate();
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Fake hub recording every emit; <see cref="Subscribe"/> is unused here.</summary>
|
||||
private sealed class CapturingHub : ITelemetryLocalHub
|
||||
{
|
||||
public List<TelemetryItem> Items { get; } = new();
|
||||
|
||||
public void Emit(TelemetryItem item)
|
||||
{
|
||||
lock (Items) Items.Add(item);
|
||||
}
|
||||
|
||||
public ITelemetrySubscription Subscribe(int boundedCapacity) => throw new NotSupportedException();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,176 @@
|
||||
using System.Net;
|
||||
using System.Threading.Channels;
|
||||
using Grpc.Core;
|
||||
using Grpc.Net.Client;
|
||||
using Microsoft.AspNetCore.Builder;
|
||||
using Microsoft.AspNetCore.Server.Kestrel.Core;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Moq;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Protos.Telemetry.V1;
|
||||
using ZB.MOM.WW.OtOpcUa.Host.Configuration;
|
||||
using ZB.MOM.WW.OtOpcUa.Host.Grpc;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Telemetry;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
|
||||
|
||||
/// <summary>
|
||||
/// Per-cluster mesh Phase 5 (host-wiring task) — the telemetry-stream h2c listener coexisting with
|
||||
/// the two earlier dedicated ports (LocalDb sync + ConfigServe) and the main HTTP surface, and the
|
||||
/// <see cref="TelemetryStreamGrpcService"/> actually being <b>mapped and interceptor-gated</b> on
|
||||
/// that port.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Why this exists.</b> Phase 5 adds a THIRD dedicated h2c port to <c>Program.cs</c>'s merged
|
||||
/// listener block. That block must still re-bind the existing HTTP surface EXACTLY ONCE while
|
||||
/// adding all three dedicated ports (re-binding it per-port throws "address already in use";
|
||||
/// re-binding it zero times silently unbinds the AdminUI behind Traefik). The sibling
|
||||
/// <see cref="ConfigServeListenerTests"/> pins the two-port case; this pins the three-port case
|
||||
/// AND — the part a port-open check cannot prove — that the telemetry gRPC service is genuinely
|
||||
/// wired: an un-keyed dial gets <see cref="StatusCode.PermissionDenied"/> (mapped + interceptor
|
||||
/// reached), not <see cref="StatusCode.Unimplemented"/> (never mapped), and a correctly-keyed
|
||||
/// dial opens a real stream. This is the executable regression for the "ships clean but never
|
||||
/// actually wired" gap.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Like <see cref="ConfigServeListenerTests"/>, this drives a minimal <c>WebApplication</c>
|
||||
/// shaped exactly like <c>Program.cs</c>'s driver-role wiring (AddGrpc +
|
||||
/// <see cref="TelemetryStreamAuthInterceptor"/> + <c>Configure<TelemetryOptions></c> +
|
||||
/// <c>MapGrpcService<TelemetryStreamGrpcService></c>) rather than booting the real host
|
||||
/// (which needs SQL, an Akka mesh and LDAP). What is under test is the Kestrel binding contract
|
||||
/// and the gRPC map/intercept wiring, both fully reproduced here. The node's own telemetry hub
|
||||
/// is a stub — this test proves reachability, not fan-out content.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class TelemetryListenerTests
|
||||
{
|
||||
private const string ApiKey = "telemetry-test-key";
|
||||
|
||||
[Fact]
|
||||
public async Task AllThreeDedicatedH2cPorts_AndTheHttpSurface_AnswerSimultaneously_AndTelemetryServiceIsWired()
|
||||
{
|
||||
var httpPort = GetFreePort();
|
||||
var syncPort = GetFreePort();
|
||||
var configServePort = GetFreePort();
|
||||
var telemetryPort = GetFreePort();
|
||||
|
||||
var builder = WebApplication.CreateBuilder();
|
||||
builder.Environment.ApplicationName = typeof(TelemetryListenerTests).Assembly.GetName().Name!;
|
||||
|
||||
// Exactly the driver-role wiring Program.cs applies for the telemetry endpoint: the fail-closed
|
||||
// interceptor on the shared AddGrpc pipeline, the TelemetryOptions the interceptor reads its
|
||||
// expected key from, and a stub hub so the mapped service can be constructed on the keyed path.
|
||||
builder.Services.Configure<TelemetryOptions>(o => o.ApiKey = ApiKey);
|
||||
builder.Services.AddSingleton(StubHubYieldingAnEmptyStream());
|
||||
builder.Services.AddGrpc(o => o.Interceptors.Add<TelemetryStreamAuthInterceptor>());
|
||||
|
||||
// The shape Program.cs's merged block applies: compute the existing surface ONCE, apply it ONCE,
|
||||
// then add each configured dedicated h2c port. Here all THREE dedicated ports are set at once —
|
||||
// the fused-node case that would double-bind the HTTP surface if the block re-applied it per-port.
|
||||
var existingBindings = KestrelHttpBinding.Parse($"http://localhost:{httpPort}");
|
||||
builder.WebHost.ConfigureKestrel(kestrel =>
|
||||
{
|
||||
foreach (var binding in existingBindings)
|
||||
binding.Apply(kestrel);
|
||||
|
||||
kestrel.ListenAnyIP(syncPort, o => o.Protocols = HttpProtocols.Http2);
|
||||
kestrel.ListenAnyIP(configServePort, o => o.Protocols = HttpProtocols.Http2);
|
||||
kestrel.ListenAnyIP(telemetryPort, o => o.Protocols = HttpProtocols.Http2);
|
||||
});
|
||||
|
||||
await using var app = builder.Build();
|
||||
app.MapGet("/healthz", () => "ok");
|
||||
app.MapGrpcService<TelemetryStreamGrpcService>();
|
||||
await app.StartAsync(TestContext.Current.CancellationToken);
|
||||
|
||||
try
|
||||
{
|
||||
var ct = TestContext.Current.CancellationToken;
|
||||
|
||||
// 1) The re-bound HTTP/1.1 surface still answers (the AdminUI/Traefik guarantee) even with
|
||||
// all three dedicated ports added — i.e. the existing surface was re-bound exactly once.
|
||||
using var http1 = new HttpClient();
|
||||
(await http1.GetStringAsync($"http://localhost:{httpPort}/healthz", ct)).ShouldBe("ok");
|
||||
|
||||
// 2) The sync + config-serve dedicated ports negotiate prior-knowledge h2c. A 404 is fine —
|
||||
// it proves the HTTP/2 connection established and routed, which is all that is in question
|
||||
// for those two (their services are mapped elsewhere; here we only pin the port binding).
|
||||
using var http2 = new HttpClient
|
||||
{
|
||||
DefaultRequestVersion = HttpVersion.Version20,
|
||||
DefaultVersionPolicy = HttpVersionPolicy.RequestVersionExact,
|
||||
};
|
||||
(await http2.GetAsync($"http://localhost:{syncPort}/", ct)).Version.ShouldBe(HttpVersion.Version20);
|
||||
(await http2.GetAsync($"http://localhost:{configServePort}/", ct)).Version.ShouldBe(HttpVersion.Version20);
|
||||
|
||||
// 3) The telemetry port serves the MAPPED, interceptor-gated gRPC service. Dial it for real.
|
||||
using var channel = GrpcChannel.ForAddress(
|
||||
$"http://localhost:{telemetryPort}", new GrpcChannelOptions { HttpHandler = new SocketsHttpHandler() });
|
||||
var client = new TelemetryStreamService.TelemetryStreamServiceClient(channel);
|
||||
|
||||
// 3a) No bearer -> PermissionDenied. This is the load-bearing assertion: PermissionDenied can
|
||||
// only come from the interceptor, and the interceptor only runs because the method was
|
||||
// routed to the mapped service. A service that was never mapped would answer Unimplemented.
|
||||
var noKey = await Should.ThrowAsync<RpcException>(async () =>
|
||||
{
|
||||
using var call = client.Subscribe(new TelemetryStreamRequest { CorrelationId = "no-key" }, cancellationToken: ct);
|
||||
await foreach (var _ in call.ResponseStream.ReadAllAsync(ct)) { }
|
||||
});
|
||||
noKey.StatusCode.ShouldBe(StatusCode.PermissionDenied);
|
||||
noKey.StatusCode.ShouldNotBe(StatusCode.Unimplemented);
|
||||
|
||||
// 3b) Correct bearer -> the mapped handler actually executes end-to-end (interceptor passes,
|
||||
// the service resolves the stub hub and streams). The stub yields an empty, completed
|
||||
// stream, so the call completes cleanly with zero envelopes.
|
||||
var headers = new Metadata { { "authorization", $"Bearer {ApiKey}" } };
|
||||
using var keyed = client.Subscribe(
|
||||
new TelemetryStreamRequest { CorrelationId = "keyed" }, headers, cancellationToken: ct);
|
||||
var received = 0;
|
||||
await foreach (var _ in keyed.ResponseStream.ReadAllAsync(ct))
|
||||
received++;
|
||||
received.ShouldBe(0);
|
||||
}
|
||||
finally
|
||||
{
|
||||
await app.StopAsync(TestContext.Current.CancellationToken);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A hub whose subscription reader is already completed and empty, so the keyed
|
||||
/// <c>Subscribe</c> handler runs to completion and closes the stream with zero envelopes —
|
||||
/// enough to prove the mapped service executed without depending on any live telemetry producer.
|
||||
/// </summary>
|
||||
private static ITelemetryLocalHub StubHubYieldingAnEmptyStream()
|
||||
{
|
||||
var hub = new Mock<ITelemetryLocalHub>();
|
||||
hub.Setup(h => h.Subscribe(It.IsAny<int>())).Returns(() => new EmptySubscription());
|
||||
return hub.Object;
|
||||
}
|
||||
|
||||
private sealed class EmptySubscription : ITelemetrySubscription
|
||||
{
|
||||
private readonly Channel<TelemetryItem> _channel = Channel.CreateBounded<TelemetryItem>(1);
|
||||
|
||||
public EmptySubscription() => _channel.Writer.Complete();
|
||||
|
||||
public ChannelReader<TelemetryItem> Reader => _channel.Reader;
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
}
|
||||
}
|
||||
|
||||
private static int GetFreePort()
|
||||
{
|
||||
using var listener = new System.Net.Sockets.TcpListener(IPAddress.Loopback, 0);
|
||||
listener.Start();
|
||||
var port = ((IPEndPoint)listener.LocalEndpoint).Port;
|
||||
listener.Stop();
|
||||
return port;
|
||||
}
|
||||
}
|
||||
+5
@@ -9,6 +9,11 @@
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Moq" />
|
||||
<!-- Grpc.Net.Client: the initiator side of a gRPC call. The Host references only
|
||||
Grpc.AspNetCore (server side); TelemetryListenerTests dials the mapped telemetry service
|
||||
to prove it is reachable, which needs the client. Grpc.Core.Api (RpcException/StatusCode/
|
||||
Metadata + the generated client base) flows in transitively via the Commons project ref. -->
|
||||
<PackageReference Include="Grpc.Net.Client" />
|
||||
<PackageReference Include="xunit.v3" />
|
||||
<PackageReference Include="Shouldly" />
|
||||
<PackageReference Include="Microsoft.NET.Test.Sdk" />
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
using Xunit;
|
||||
|
||||
// The TelemetryStreamGrpcService concurrency counter is process-wide (static). Run this assembly's
|
||||
// tests serially so the cap test observes a deterministic active-stream count with no cross-test
|
||||
// interference. This project holds only telemetry tests, so serial execution costs nothing.
|
||||
[assembly: CollectionBehavior(DisableTestParallelization = true)]
|
||||
+199
@@ -0,0 +1,199 @@
|
||||
using Grpc.Core;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
using ZB.MOM.WW.OtOpcUa.Host.Configuration;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.Tests.Configuration;
|
||||
|
||||
/// <summary>
|
||||
/// Per-cluster mesh Phase 5 — the telemetry stream endpoint's inbound gate. Wiring this
|
||||
/// interceptor into the <c>AddGrpc</c> pipeline + Kestrel is a separate, later task; this test
|
||||
/// covers only the interceptor's own authorization behavior.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Same contract as <see cref="ConfigServeAuthInterceptor"/> / <see cref="LocalDbSyncAuthInterceptor"/>,
|
||||
/// scoped to <c>/telemetry.v1.TelemetryStreamService/</c> — fail-closed on an unconfigured key,
|
||||
/// constant-time comparison, gated on every handler shape so scoping holds regardless of RPC shape.
|
||||
/// </remarks>
|
||||
public sealed class TelemetryStreamAuthInterceptorTests
|
||||
{
|
||||
private const string SubscribeMethod = "/telemetry.v1.TelemetryStreamService/Subscribe";
|
||||
private const string OtherMethod = "/deployment_artifact.v1.DeploymentArtifactService/Fetch";
|
||||
|
||||
private static TelemetryStreamAuthInterceptor CreateInterceptor(string? apiKey)
|
||||
=> new(
|
||||
Options.Create(new TelemetryOptions { ApiKey = apiKey ?? string.Empty }),
|
||||
NullLogger<TelemetryStreamAuthInterceptor>.Instance);
|
||||
|
||||
private static ServerCallContext CreateContext(string method, string? authorizationHeader)
|
||||
{
|
||||
var headers = new Metadata();
|
||||
if (authorizationHeader is not null)
|
||||
headers.Add("authorization", authorizationHeader);
|
||||
|
||||
return new FakeServerCallContext(method, headers);
|
||||
}
|
||||
|
||||
/// <summary>Invokes the interceptor's server-streaming path (how <c>Subscribe</c> actually runs).</summary>
|
||||
private static Task Invoke(TelemetryStreamAuthInterceptor interceptor, ServerCallContext context)
|
||||
=> interceptor.ServerStreamingServerHandler<string, string>(
|
||||
"request", responseStream: null!, context, (_, _, _) => Task.CompletedTask);
|
||||
|
||||
[Fact]
|
||||
public async Task NonTelemetryMethod_PassesThrough_EvenWithNoKeyConfigured()
|
||||
{
|
||||
// The interceptor shares the AddGrpc pipeline with other gRPC services, so it sees every
|
||||
// call. It must be scoped strictly to the telemetry service.
|
||||
var interceptor = CreateInterceptor(apiKey: null);
|
||||
var context = CreateContext(OtherMethod, authorizationHeader: null);
|
||||
|
||||
await Invoke(interceptor, context); // no throw
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SubscribeMethod_WithNoKeyConfigured_IsDenied_EvenWithABearerToken()
|
||||
{
|
||||
// Fail-closed. "No key configured" is the DEFAULT — treating it as "no auth required" would
|
||||
// expose live telemetry on exactly the shape most nodes ship with. A token must not help.
|
||||
var interceptor = CreateInterceptor(apiKey: null);
|
||||
var context = CreateContext(SubscribeMethod, "Bearer anything-at-all");
|
||||
|
||||
var ex = await Should.ThrowAsync<RpcException>(() => Invoke(interceptor, context));
|
||||
ex.StatusCode.ShouldBe(StatusCode.PermissionDenied);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SubscribeMethod_WithNoBearerToken_IsDenied()
|
||||
{
|
||||
var interceptor = CreateInterceptor("the-shared-key");
|
||||
var context = CreateContext(SubscribeMethod, authorizationHeader: null);
|
||||
|
||||
var ex = await Should.ThrowAsync<RpcException>(() => Invoke(interceptor, context));
|
||||
ex.StatusCode.ShouldBe(StatusCode.PermissionDenied);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SubscribeMethod_WithMalformedAuthorizationHeader_IsDenied()
|
||||
{
|
||||
// No "Bearer " prefix at all — must be treated the same as a missing header.
|
||||
var interceptor = CreateInterceptor("the-shared-key");
|
||||
var context = CreateContext(SubscribeMethod, "the-shared-key");
|
||||
|
||||
var ex = await Should.ThrowAsync<RpcException>(() => Invoke(interceptor, context));
|
||||
ex.StatusCode.ShouldBe(StatusCode.PermissionDenied);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SubscribeMethod_WithWrongBearerToken_IsDenied()
|
||||
{
|
||||
var interceptor = CreateInterceptor("the-shared-key");
|
||||
var context = CreateContext(SubscribeMethod, "Bearer the-wrong-key");
|
||||
|
||||
var ex = await Should.ThrowAsync<RpcException>(() => Invoke(interceptor, context));
|
||||
ex.StatusCode.ShouldBe(StatusCode.PermissionDenied);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SubscribeMethod_WithCorrectBearerToken_PassesThrough()
|
||||
{
|
||||
var interceptor = CreateInterceptor("the-shared-key");
|
||||
var context = CreateContext(SubscribeMethod, "Bearer the-shared-key");
|
||||
|
||||
await Invoke(interceptor, context); // no throw
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SubscribeMethod_TokenComparison_IsNotAPrefixMatch()
|
||||
{
|
||||
// A prefix/StartsWith comparison would accept a truncated key and make the secret
|
||||
// recoverable one character at a time.
|
||||
var interceptor = CreateInterceptor("the-shared-key");
|
||||
var context = CreateContext(SubscribeMethod, "Bearer the-shared-ke");
|
||||
|
||||
var ex = await Should.ThrowAsync<RpcException>(() => Invoke(interceptor, context));
|
||||
ex.StatusCode.ShouldBe(StatusCode.PermissionDenied);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task UnaryPath_IsAlsoGated()
|
||||
{
|
||||
// Subscribe is server-streaming today, but gating only that path would leave any future
|
||||
// unary method on the same service open. Gate every handler shape.
|
||||
var interceptor = CreateInterceptor("the-shared-key");
|
||||
var context = CreateContext(SubscribeMethod, "Bearer the-wrong-key");
|
||||
|
||||
var ex = await Should.ThrowAsync<RpcException>(() =>
|
||||
interceptor.UnaryServerHandler<string, string>(
|
||||
"request", context, (_, _) => Task.FromResult("ok")));
|
||||
|
||||
ex.StatusCode.ShouldBe(StatusCode.PermissionDenied);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ClientStreamingPath_IsAlsoGated()
|
||||
{
|
||||
var interceptor = CreateInterceptor("the-shared-key");
|
||||
var context = CreateContext(SubscribeMethod, "Bearer the-wrong-key");
|
||||
|
||||
var ex = await Should.ThrowAsync<RpcException>(() =>
|
||||
interceptor.ClientStreamingServerHandler<string, string>(
|
||||
requestStream: null!, context, (_, _) => Task.FromResult("ok")));
|
||||
|
||||
ex.StatusCode.ShouldBe(StatusCode.PermissionDenied);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task DuplexStreamingPath_IsAlsoGated()
|
||||
{
|
||||
var interceptor = CreateInterceptor("the-shared-key");
|
||||
var context = CreateContext(SubscribeMethod, "Bearer the-wrong-key");
|
||||
|
||||
var ex = await Should.ThrowAsync<RpcException>(() =>
|
||||
interceptor.DuplexStreamingServerHandler<string, string>(
|
||||
requestStream: null!, responseStream: null!, context, (_, _, _) => Task.CompletedTask));
|
||||
|
||||
ex.StatusCode.ShouldBe(StatusCode.PermissionDenied);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void HasExactlyOnePublicConstructor()
|
||||
{
|
||||
// Grpc.AspNetCore silently stops invoking an interceptor with more than one public ctor —
|
||||
// this pins the shape so a well-meaning overload doesn't quietly disable auth in production.
|
||||
var publicCtors = typeof(TelemetryStreamAuthInterceptor).GetConstructors();
|
||||
|
||||
publicCtors.Length.ShouldBe(1);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Minimal <see cref="ServerCallContext"/> carrying just a method name and request headers —
|
||||
/// the only two things the interceptor reads. Hand-rolled because
|
||||
/// <c>Grpc.Core.Testing.TestServerCallContext</c> ships only in the retired native package.
|
||||
/// </summary>
|
||||
private sealed class FakeServerCallContext(string method, Metadata requestHeaders)
|
||||
: ServerCallContext
|
||||
{
|
||||
protected override string MethodCore => method;
|
||||
protected override string HostCore => "localhost";
|
||||
protected override string PeerCore => "ipv4:127.0.0.1:12345";
|
||||
protected override DateTime DeadlineCore => DateTime.UtcNow.AddMinutes(1);
|
||||
protected override Metadata RequestHeadersCore => requestHeaders;
|
||||
protected override CancellationToken CancellationTokenCore => CancellationToken.None;
|
||||
protected override Metadata ResponseTrailersCore { get; } = [];
|
||||
protected override Status StatusCore { get; set; }
|
||||
protected override WriteOptions? WriteOptionsCore { get; set; }
|
||||
|
||||
protected override AuthContext AuthContextCore { get; } =
|
||||
new(null, new Dictionary<string, List<AuthProperty>>());
|
||||
|
||||
protected override ContextPropagationToken CreatePropagationTokenCore(
|
||||
ContextPropagationOptions? options)
|
||||
=> throw new NotSupportedException();
|
||||
|
||||
protected override Task WriteResponseHeadersAsyncCore(Metadata responseHeaders)
|
||||
=> Task.CompletedTask;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,357 @@
|
||||
using Grpc.Core;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Alerts;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Drivers;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Logging;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Protos.Telemetry.V1;
|
||||
using ZB.MOM.WW.OtOpcUa.Host.Grpc;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Telemetry;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.Tests.Grpc;
|
||||
|
||||
/// <summary>
|
||||
/// Verifies the node-side telemetry streaming service (per-cluster mesh Phase 5): the hub is
|
||||
/// fanned to a connected caller as <see cref="TelemetryEvent"/> envelopes, every domain field is
|
||||
/// transcribed onto its proto field, DateTime→Timestamp mapping is Kind-defensive, a cancelled
|
||||
/// context ends the stream cleanly, and the process-wide concurrency cap sheds excess dials with
|
||||
/// <see cref="StatusCode.ResourceExhausted"/>.
|
||||
/// </summary>
|
||||
public sealed class TelemetryStreamGrpcServiceTests
|
||||
{
|
||||
private static TelemetryStreamGrpcService NewService(ITelemetryLocalHub hub) =>
|
||||
new(hub, NullLogger<TelemetryStreamGrpcService>.Instance);
|
||||
|
||||
private static ServerCallContext Ctx(CancellationToken token) => new FakeServerCallContext(token);
|
||||
|
||||
private static async Task WaitUntilAsync(Func<bool> condition, string because)
|
||||
{
|
||||
var deadline = DateTime.UtcNow.AddSeconds(5);
|
||||
while (DateTime.UtcNow < deadline)
|
||||
{
|
||||
if (condition())
|
||||
return;
|
||||
await Task.Delay(15);
|
||||
}
|
||||
|
||||
throw new TimeoutException($"Timed out waiting: {because}");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Streams_one_of_each_kind_with_fields_transcribed()
|
||||
{
|
||||
var hub = new TelemetryLocalHub();
|
||||
var service = NewService(hub);
|
||||
var writer = new CapturingStreamWriter();
|
||||
using var cts = new CancellationTokenSource();
|
||||
var request = new TelemetryStreamRequest { CorrelationId = "corr-1" };
|
||||
|
||||
var publishedHealth = new DateTime(2026, 7, 23, 10, 0, 0, DateTimeKind.Utc);
|
||||
var health = new DriverHealthChanged(
|
||||
"cluster-a", "drv-1", "Healthy", LastSuccessfulReadUtc: null, LastError: null,
|
||||
ErrorCount5Min: 3, PublishedUtc: publishedHealth);
|
||||
var resilience = new DriverResilienceStatusChanged(
|
||||
"drv-1", "host-x", BreakerOpen: true, ConsecutiveFailures: 4, CurrentInFlight: 2,
|
||||
LastBreakerOpenUtc: null, LastSampledUtc: DateTime.UtcNow, PublishedUtc: DateTime.UtcNow);
|
||||
|
||||
// Snapshot-style items are cached, so emitting them BEFORE the service attaches guarantees they
|
||||
// are replayed to the fresh subscription — a deterministic "the pump is draining" signal.
|
||||
hub.Emit(new TelemetryItem.Health(health));
|
||||
hub.Emit(new TelemetryItem.Resilience(resilience));
|
||||
|
||||
var pump = service.Subscribe(request, writer, Ctx(cts.Token));
|
||||
|
||||
await WaitUntilAsync(() => writer.Count >= 2, "cached snapshots to be replayed");
|
||||
|
||||
// Append-style items are live-only: emit them once the subscription is proven attached.
|
||||
var alarm = new AlarmTransitionEvent(
|
||||
"raw/dev/Speed", "Area/Line/Equip", "HighSpeed", "Activated", 700, "over limit", "operator1",
|
||||
new DateTime(2026, 7, 23, 11, 0, 0, DateTimeKind.Utc), "LimitAlarm",
|
||||
Comment: "manual ack", HistorizeToAveva: true,
|
||||
ReferencingEquipmentPaths: ["Area/Line/EquipA", "Area/Line/EquipB"]);
|
||||
var script = new ScriptLogEntry(
|
||||
"script-9", "Warning", "threshold near", new DateTime(2026, 7, 23, 12, 0, 0, DateTimeKind.Utc),
|
||||
VirtualTagId: "vt-1", AlarmId: null, EquipmentId: "equip-1");
|
||||
|
||||
hub.Emit(new TelemetryItem.Alarm(alarm));
|
||||
hub.Emit(new TelemetryItem.Script(script));
|
||||
|
||||
await WaitUntilAsync(() => writer.Count >= 4, "all four telemetry events to be streamed");
|
||||
|
||||
cts.Cancel();
|
||||
await pump; // cancellation must be swallowed — Subscribe returns cleanly.
|
||||
|
||||
var events = writer.Snapshot();
|
||||
events.ShouldAllBe(e => e.CorrelationId == "corr-1");
|
||||
|
||||
var healthEvt = events.Single(e => e.EventCase == TelemetryEvent.EventOneofCase.DriverHealth);
|
||||
healthEvt.DriverHealth.ClusterId.ShouldBe("cluster-a");
|
||||
healthEvt.DriverHealth.State.ShouldBe("Healthy");
|
||||
healthEvt.DriverHealth.ErrorCount5Min.ShouldBe(3);
|
||||
healthEvt.DriverHealth.PublishedUtc.ToDateTime().ShouldBe(publishedHealth);
|
||||
healthEvt.DriverHealth.LastSuccessfulReadUtc.ShouldBeNull(); // nullable DateTime absent
|
||||
healthEvt.DriverHealth.HasLastError.ShouldBeFalse(); // nullable string absent
|
||||
|
||||
var resilienceEvt = events.Single(e => e.EventCase == TelemetryEvent.EventOneofCase.DriverResilience);
|
||||
resilienceEvt.DriverResilience.DriverInstanceId.ShouldBe("drv-1");
|
||||
resilienceEvt.DriverResilience.BreakerOpen.ShouldBeTrue();
|
||||
resilienceEvt.DriverResilience.LastBreakerOpenUtc.ShouldBeNull();
|
||||
|
||||
var alarmEvt = events.Single(e => e.EventCase == TelemetryEvent.EventOneofCase.AlarmTransition);
|
||||
alarmEvt.AlarmTransition.AlarmId.ShouldBe("raw/dev/Speed");
|
||||
alarmEvt.AlarmTransition.TransitionKind.ShouldBe("Activated");
|
||||
alarmEvt.AlarmTransition.Severity.ShouldBe(700);
|
||||
alarmEvt.AlarmTransition.Comment.ShouldBe("manual ack");
|
||||
alarmEvt.AlarmTransition.HistorizeToAveva.ShouldBeTrue();
|
||||
alarmEvt.AlarmTransition.ReferencingEquipmentPaths.ShouldBe(["Area/Line/EquipA", "Area/Line/EquipB"]);
|
||||
|
||||
var scriptEvt = events.Single(e => e.EventCase == TelemetryEvent.EventOneofCase.ScriptLog);
|
||||
scriptEvt.ScriptLog.ScriptId.ShouldBe("script-9");
|
||||
scriptEvt.ScriptLog.Level.ShouldBe("Warning");
|
||||
scriptEvt.ScriptLog.VirtualTagId.ShouldBe("vt-1");
|
||||
scriptEvt.ScriptLog.HasAlarmId.ShouldBeFalse(); // nullable string absent
|
||||
scriptEvt.ScriptLog.EquipmentId.ShouldBe("equip-1");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Cancelled_context_ends_stream_cleanly()
|
||||
{
|
||||
var hub = new TelemetryLocalHub();
|
||||
var service = NewService(hub);
|
||||
var writer = new CapturingStreamWriter();
|
||||
using var cts = new CancellationTokenSource();
|
||||
var request = new TelemetryStreamRequest { CorrelationId = "corr-cancel" };
|
||||
|
||||
var pump = service.Subscribe(request, writer, Ctx(cts.Token));
|
||||
|
||||
cts.Cancel();
|
||||
await Should.NotThrowAsync(() => pump); // client disconnect is normal, not an error.
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("")]
|
||||
[InlineData(" ")]
|
||||
public async Task Empty_correlation_id_is_invalid_argument(string correlationId)
|
||||
{
|
||||
var service = NewService(new TelemetryLocalHub());
|
||||
using var cts = new CancellationTokenSource();
|
||||
|
||||
var ex = await Should.ThrowAsync<RpcException>(() => service.Subscribe(
|
||||
new TelemetryStreamRequest { CorrelationId = correlationId },
|
||||
new CapturingStreamWriter(), Ctx(cts.Token)));
|
||||
|
||||
ex.StatusCode.ShouldBe(StatusCode.InvalidArgument);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Over_long_correlation_id_is_invalid_argument()
|
||||
{
|
||||
var service = NewService(new TelemetryLocalHub());
|
||||
using var cts = new CancellationTokenSource();
|
||||
|
||||
var ex = await Should.ThrowAsync<RpcException>(() => service.Subscribe(
|
||||
new TelemetryStreamRequest { CorrelationId = new string('x', 257) },
|
||||
new CapturingStreamWriter(), Ctx(cts.Token)));
|
||||
|
||||
ex.StatusCode.ShouldBe(StatusCode.InvalidArgument);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Concurrency_cap_sheds_the_over_the_limit_dial()
|
||||
{
|
||||
// Serial assembly (see AssemblyInfo) => the static active-stream counter starts at 0 here.
|
||||
var hub = new TelemetryLocalHub();
|
||||
var service = NewService(hub);
|
||||
using var cts = new CancellationTokenSource();
|
||||
|
||||
// Open exactly the cap. Each stream blocks in the pump (hub emits nothing and is never completed),
|
||||
// so all 100 stay open, holding the counter at 100.
|
||||
var open = new List<Task>();
|
||||
for (var i = 0; i < 100; i++)
|
||||
open.Add(service.Subscribe(
|
||||
new TelemetryStreamRequest { CorrelationId = $"cap-{i}" },
|
||||
new CapturingStreamWriter(), Ctx(cts.Token)));
|
||||
|
||||
// The 101st dial must be rejected.
|
||||
var ex = await Should.ThrowAsync<RpcException>(() => service.Subscribe(
|
||||
new TelemetryStreamRequest { CorrelationId = "cap-overflow" },
|
||||
new CapturingStreamWriter(), Ctx(cts.Token)));
|
||||
ex.StatusCode.ShouldBe(StatusCode.ResourceExhausted);
|
||||
|
||||
// Drain the 100 open streams; the counter returns to 0 for the next test.
|
||||
cts.Cancel();
|
||||
await Task.WhenAll(open);
|
||||
|
||||
// With the counter back at 0, a fresh single dial is accepted (proves no leak / no double-count).
|
||||
using var cts2 = new CancellationTokenSource();
|
||||
var writer = new CapturingStreamWriter();
|
||||
var pump = service.Subscribe(
|
||||
new TelemetryStreamRequest { CorrelationId = "cap-after" }, writer, Ctx(cts2.Token));
|
||||
cts2.Cancel();
|
||||
await Should.NotThrowAsync(() => pump);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ToProto_health_with_unspecified_kind_datetime_does_not_throw_and_roundtrips()
|
||||
{
|
||||
// Producers use UtcNow, but a Kind=Unspecified value must never crash the stream. The mapper
|
||||
// assumes an Unspecified value is already-UTC, so the instant round-trips on any machine.
|
||||
var wall = new DateTime(2026, 7, 23, 13, 30, 0, DateTimeKind.Unspecified);
|
||||
var expectedUtc = DateTime.SpecifyKind(wall, DateTimeKind.Utc);
|
||||
var health = new DriverHealthChanged(
|
||||
"cluster-a", "drv-1", "Healthy", LastSuccessfulReadUtc: wall, LastError: null,
|
||||
ErrorCount5Min: 0, PublishedUtc: wall);
|
||||
|
||||
TelemetryEvent evt = null!;
|
||||
Should.NotThrow(() => evt = TelemetryProtoMapNode.ToProto(new TelemetryItem.Health(health), "c"));
|
||||
|
||||
evt.DriverHealth.PublishedUtc.ToDateTime().ShouldBe(expectedUtc);
|
||||
evt.DriverHealth.LastSuccessfulReadUtc.ToDateTime().ShouldBe(expectedUtc);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Client_disconnect_mid_stream_ends_cleanly_without_leaking_a_slot()
|
||||
{
|
||||
// A broken pipe surfaces as IOException out of WriteAsync. It must be swallowed as a routine
|
||||
// disconnect (not escape), and the finally must still decrement the counter. Looping well past
|
||||
// the cap proves no leak: a single leaked slot per pass would trip ResourceExhausted by pass 101.
|
||||
var hub = new TelemetryLocalHub();
|
||||
var service = NewService(hub);
|
||||
|
||||
// Cached snapshot ⇒ every fresh subscription is deterministically handed one item to write.
|
||||
hub.Emit(new TelemetryItem.Health(new DriverHealthChanged(
|
||||
"cluster-a", "drv-1", "Healthy", null, null, 0, DateTime.UtcNow)));
|
||||
|
||||
for (var i = 0; i < 150; i++)
|
||||
{
|
||||
using var cts = new CancellationTokenSource();
|
||||
var pump = service.Subscribe(
|
||||
new TelemetryStreamRequest { CorrelationId = $"disc-{i}" },
|
||||
new ThrowingStreamWriter(new IOException("connection reset")),
|
||||
Ctx(cts.Token));
|
||||
|
||||
// No exception escapes Subscribe, and no ResourceExhausted ever fires (proves the slot is freed).
|
||||
await Should.NotThrowAsync(() => pump);
|
||||
}
|
||||
|
||||
// Counter is back at its prior value: a normal stream still opens and pumps.
|
||||
using var okCts = new CancellationTokenSource();
|
||||
var writer = new CapturingStreamWriter();
|
||||
var okPump = service.Subscribe(
|
||||
new TelemetryStreamRequest { CorrelationId = "disc-after" }, writer, Ctx(okCts.Token));
|
||||
await WaitUntilAsync(() => writer.Count >= 1, "the cached snapshot to stream after the disconnect loop");
|
||||
okCts.Cancel();
|
||||
await okPump;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ToProto_null_required_string_does_not_throw_and_maps_to_empty()
|
||||
{
|
||||
// Required proto string setters throw on null; the domain records carry no runtime guard.
|
||||
var alarm = new AlarmTransitionEvent(
|
||||
AlarmId: null!, EquipmentPath: null!, AlarmName: null!, TransitionKind: null!,
|
||||
Severity: 100, Message: null!, User: null!, TimestampUtc: DateTime.UtcNow,
|
||||
AlarmTypeName: null!, Comment: null, HistorizeToAveva: null, ReferencingEquipmentPaths: null);
|
||||
|
||||
AlarmTransition msg = null!;
|
||||
Should.NotThrow(() => msg = TelemetryProtoMapNode.ToProto(new TelemetryItem.Alarm(alarm), "c").AlarmTransition);
|
||||
|
||||
msg.AlarmId.ShouldBe("");
|
||||
msg.EquipmentPath.ShouldBe("");
|
||||
msg.AlarmName.ShouldBe("");
|
||||
msg.TransitionKind.ShouldBe("");
|
||||
msg.Message.ShouldBe("");
|
||||
msg.User.ShouldBe("");
|
||||
msg.AlarmTypeName.ShouldBe("");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ToProto_alarm_omits_absent_nullables()
|
||||
{
|
||||
var alarm = new AlarmTransitionEvent(
|
||||
"id", "Area/Line/Equip", "Name", "Cleared", 100, "msg", "system",
|
||||
DateTime.UtcNow, Comment: null, HistorizeToAveva: null, ReferencingEquipmentPaths: null);
|
||||
|
||||
var msg = TelemetryProtoMapNode.ToProto(new TelemetryItem.Alarm(alarm), "c").AlarmTransition;
|
||||
|
||||
msg.HasComment.ShouldBeFalse();
|
||||
msg.HasHistorizeToAveva.ShouldBeFalse();
|
||||
msg.ReferencingEquipmentPaths.ShouldBeEmpty();
|
||||
}
|
||||
|
||||
/// <summary>Minimal <see cref="ServerCallContext"/> exposing a caller-controlled cancellation token.</summary>
|
||||
private sealed class FakeServerCallContext(CancellationToken token) : ServerCallContext
|
||||
{
|
||||
protected override string MethodCore => "/telemetry.v1.TelemetryStreamService/Subscribe";
|
||||
protected override string HostCore => "localhost";
|
||||
protected override string PeerCore => "ipv4:127.0.0.1:0";
|
||||
protected override DateTime DeadlineCore => DateTime.MaxValue;
|
||||
protected override Metadata RequestHeadersCore => [];
|
||||
protected override CancellationToken CancellationTokenCore => token;
|
||||
protected override Metadata ResponseTrailersCore { get; } = [];
|
||||
protected override Status StatusCore { get; set; }
|
||||
protected override WriteOptions? WriteOptionsCore { get; set; }
|
||||
protected override AuthContext AuthContextCore => new(null, new Dictionary<string, List<AuthProperty>>());
|
||||
|
||||
protected override ContextPropagationToken CreatePropagationTokenCore(ContextPropagationOptions? options) =>
|
||||
throw new NotSupportedException();
|
||||
|
||||
protected override Task WriteResponseHeadersAsyncCore(Metadata responseHeaders) => Task.CompletedTask;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Captures every streamed <see cref="TelemetryEvent"/>. Implements the cancellable
|
||||
/// <c>WriteAsync</c> overload explicitly — otherwise the interface's default method throws
|
||||
/// <see cref="NotSupportedException"/> for the pump's cancellable lifetime token.
|
||||
/// </summary>
|
||||
private sealed class CapturingStreamWriter : IServerStreamWriter<TelemetryEvent>
|
||||
{
|
||||
private readonly List<TelemetryEvent> _items = [];
|
||||
private readonly Lock _gate = new();
|
||||
|
||||
public WriteOptions? WriteOptions { get; set; }
|
||||
|
||||
public int Count
|
||||
{
|
||||
get
|
||||
{
|
||||
lock (_gate)
|
||||
return _items.Count;
|
||||
}
|
||||
}
|
||||
|
||||
public Task WriteAsync(TelemetryEvent message)
|
||||
{
|
||||
lock (_gate)
|
||||
_items.Add(message);
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
public Task WriteAsync(TelemetryEvent message, CancellationToken cancellationToken)
|
||||
{
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
return WriteAsync(message);
|
||||
}
|
||||
|
||||
public List<TelemetryEvent> Snapshot()
|
||||
{
|
||||
lock (_gate)
|
||||
return [.. _items];
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>A stream writer that always throws the given exception — models a broken client pipe.</summary>
|
||||
private sealed class ThrowingStreamWriter(Exception toThrow) : IServerStreamWriter<TelemetryEvent>
|
||||
{
|
||||
public WriteOptions? WriteOptions { get; set; }
|
||||
|
||||
public Task WriteAsync(TelemetryEvent message) => throw toThrow;
|
||||
|
||||
public Task WriteAsync(TelemetryEvent message, CancellationToken cancellationToken)
|
||||
{
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
return WriteAsync(message);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
|
||||
<PropertyGroup>
|
||||
<IsPackable>false</IsPackable>
|
||||
<IsTestProject>true</IsTestProject>
|
||||
<RootNamespace>ZB.MOM.WW.OtOpcUa.Host.Tests</RootNamespace>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="xunit"/>
|
||||
<PackageReference Include="Shouldly"/>
|
||||
<PackageReference Include="Microsoft.NET.Test.Sdk"/>
|
||||
<PackageReference Include="xunit.runner.visualstudio">
|
||||
<PrivateAssets>all</PrivateAssets>
|
||||
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
|
||||
</PackageReference>
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\..\..\src\Server\ZB.MOM.WW.OtOpcUa.Host\ZB.MOM.WW.OtOpcUa.Host.csproj"/>
|
||||
<ProjectReference Include="..\..\..\src\Server\ZB.MOM.WW.OtOpcUa.Runtime\ZB.MOM.WW.OtOpcUa.Runtime.csproj"/>
|
||||
<ProjectReference Include="..\..\..\src\Core\ZB.MOM.WW.OtOpcUa.Commons\ZB.MOM.WW.OtOpcUa.Commons.csproj"/>
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
+13
-3
@@ -4,6 +4,7 @@ using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Logging;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Scripting;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Telemetry;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.VirtualTags;
|
||||
|
||||
@@ -40,7 +41,7 @@ public sealed class DpsScriptLogPublisherTests : RuntimeActorTestBase
|
||||
probe.Send(mediator, new Subscribe(VirtualTagActor.ScriptLogsTopic, probe.Ref));
|
||||
probe.ExpectMsg<SubscribeAck>(TimeSpan.FromSeconds(10));
|
||||
|
||||
var publisher = new DpsScriptLogPublisher(() => Sys);
|
||||
var publisher = new DpsScriptLogPublisher(() => Sys, new NoopHub());
|
||||
var entry = SampleEntry();
|
||||
|
||||
// The SubscribeAck confirms the subscribe was registered, but DPS topic membership is
|
||||
@@ -61,7 +62,7 @@ public sealed class DpsScriptLogPublisherTests : RuntimeActorTestBase
|
||||
public void Publish_does_not_throw_when_the_system_accessor_throws()
|
||||
{
|
||||
var publisher = new DpsScriptLogPublisher(
|
||||
() => throw new InvalidOperationException("system not ready"));
|
||||
() => throw new InvalidOperationException("system not ready"), new NoopHub());
|
||||
|
||||
Should.NotThrow(() => publisher.Publish(SampleEntry()));
|
||||
}
|
||||
@@ -70,6 +71,15 @@ public sealed class DpsScriptLogPublisherTests : RuntimeActorTestBase
|
||||
[Fact]
|
||||
public void Null_system_accessor_throws_ArgumentNullException()
|
||||
{
|
||||
Should.Throw<ArgumentNullException>(() => new DpsScriptLogPublisher(null!));
|
||||
Should.Throw<ArgumentNullException>(() => new DpsScriptLogPublisher(null!, new NoopHub()));
|
||||
}
|
||||
|
||||
/// <summary>No-op telemetry hub for the publisher tests (the hub tap is asserted in
|
||||
/// <c>PublishSeamEmitsToHubTests</c>; here it must simply not interfere).</summary>
|
||||
private sealed class NoopHub : ITelemetryLocalHub
|
||||
{
|
||||
public void Emit(TelemetryItem item) { }
|
||||
|
||||
public ITelemetrySubscription Subscribe(int boundedCapacity) => throw new NotSupportedException();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,317 @@
|
||||
using System.Text.Json;
|
||||
using Akka.Actor;
|
||||
using Akka.TestKit;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using Serilog;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Engines;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Alerts;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Logging;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Types;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
|
||||
using ZB.MOM.WW.OtOpcUa.OpcUaServer;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.OpcUa;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.ScriptedAlarms;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Scripting;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Telemetry;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.VirtualTags;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Telemetry;
|
||||
|
||||
/// <summary>
|
||||
/// Per-cluster mesh Phase 5 — the four telemetry publish seams each additionally fan their DPS payload
|
||||
/// into the node-local <see cref="ITelemetryLocalHub"/>. Each test drives a producer with a capturing
|
||||
/// fake hub and asserts exactly one <see cref="TelemetryItem"/> of the correct subtype carrying the
|
||||
/// SAME payload that goes onto the DistributedPubSub topic. The DPS publish itself is unchanged and is
|
||||
/// covered by the existing per-producer tests — these only assert the additive tap.
|
||||
/// </summary>
|
||||
public sealed class PublishSeamEmitsToHubTests : RuntimeActorTestBase
|
||||
{
|
||||
private static readonly NodeId TestNode = NodeId.Parse("telemetry-seam-test");
|
||||
private static readonly RevisionHash RevA = RevisionHash.Parse(new string('a', 64));
|
||||
private static readonly DateTime Ts = new(2026, 7, 23, 10, 0, 0, DateTimeKind.Utc);
|
||||
private const string AlarmRawPath = "Plant/Modbus/dev1/temp_hi";
|
||||
|
||||
// --- driver-health seam (AkkaDriverHealthPublisher) → TelemetryItem.Health --------------------------
|
||||
|
||||
/// <summary>The driver-health publisher fans the SAME <see cref="DriverHealthChanged"/> it publishes to
|
||||
/// the <c>driver-health</c> DPS topic into the hub as a <see cref="TelemetryItem.Health"/>.</summary>
|
||||
[Fact]
|
||||
public void Driver_health_publish_emits_one_Health_item_with_same_payload()
|
||||
{
|
||||
var hub = new CapturingHub();
|
||||
var publisher = new AkkaDriverHealthPublisher(Sys, hub);
|
||||
|
||||
var health = new DriverHealth(DriverState.Degraded, Ts, "some error");
|
||||
publisher.Publish("c1", "drv-1", health, errorCount5Min: 4);
|
||||
|
||||
hub.Items.Count.ShouldBe(1);
|
||||
var item = hub.Items[0].ShouldBeOfType<TelemetryItem.Health>();
|
||||
item.E.ClusterId.ShouldBe("c1");
|
||||
item.E.DriverInstanceId.ShouldBe("drv-1");
|
||||
item.E.State.ShouldBe(DriverState.Degraded.ToString());
|
||||
item.E.LastError.ShouldBe("some error");
|
||||
item.E.ErrorCount5Min.ShouldBe(4);
|
||||
}
|
||||
|
||||
// --- script-logs seam (DpsScriptLogPublisher) → TelemetryItem.Script -------------------------------
|
||||
|
||||
/// <summary>The DPS script-log publisher fans the SAME <see cref="ScriptLogEntry"/> instance it publishes
|
||||
/// to the <c>script-logs</c> topic into the hub as a <see cref="TelemetryItem.Script"/>.</summary>
|
||||
[Fact]
|
||||
public void Script_log_publish_emits_one_Script_item_with_same_payload()
|
||||
{
|
||||
var hub = new CapturingHub();
|
||||
var publisher = new DpsScriptLogPublisher(() => Sys, hub);
|
||||
|
||||
var entry = new ScriptLogEntry(
|
||||
ScriptId: "S1",
|
||||
Level: "Information",
|
||||
Message: "hello",
|
||||
TimestampUtc: Ts,
|
||||
VirtualTagId: "V1",
|
||||
AlarmId: null,
|
||||
EquipmentId: "EQ1");
|
||||
|
||||
publisher.Publish(entry);
|
||||
|
||||
hub.Items.Count.ShouldBe(1);
|
||||
var item = hub.Items[0].ShouldBeOfType<TelemetryItem.Script>();
|
||||
item.E.ShouldBeSameAs(entry); // the exact instance published to DPS
|
||||
}
|
||||
|
||||
/// <summary>A null hub is rejected at construction (fail-fast, DI provides a real hub on driver nodes).</summary>
|
||||
[Fact]
|
||||
public void Script_log_publisher_rejects_null_hub()
|
||||
{
|
||||
Should.Throw<ArgumentNullException>(() => new DpsScriptLogPublisher(() => Sys, null!));
|
||||
}
|
||||
|
||||
// --- alerts seam (DriverHostActor native alarm) → TelemetryItem.Alarm ------------------------------
|
||||
|
||||
/// <summary>A Primary-gated native-alarm transition fans the SAME <see cref="AlarmTransitionEvent"/> it
|
||||
/// publishes to the <c>alerts</c> topic into the hub as a <see cref="TelemetryItem.Alarm"/>. Proves the
|
||||
/// hub was threaded through <see cref="DriverHostActor"/>'s Props into the emit seam.</summary>
|
||||
[Fact]
|
||||
public void Native_alarm_publish_emits_one_Alarm_item_with_matching_payload()
|
||||
{
|
||||
var db = NewInMemoryDbFactory();
|
||||
var deploymentId = SeedV3AlarmDeployment(db, RevA);
|
||||
|
||||
var hub = new CapturingHub();
|
||||
var (actor, publish) = SpawnHostAndApply(db, deploymentId, hub);
|
||||
|
||||
actor.Tell(new DriverInstanceActor.AttributeAlarmPublished("drv-1", new AlarmEventArgs(
|
||||
new StubAlarmHandle(),
|
||||
SourceNodeId: "Temp",
|
||||
ConditionId: AlarmRawPath,
|
||||
AlarmType: "OffNormalAlarm",
|
||||
Message: "temperature high",
|
||||
Severity: AlarmSeverity.High,
|
||||
SourceTimestampUtc: Ts,
|
||||
Kind: AlarmTransitionKind.Raise)));
|
||||
|
||||
// The OPC UA condition update confirms the transition was processed (ordered before the alerts fan-out).
|
||||
publish.ExpectMsg<OpcUaPublishActor.AlarmStateUpdate>(TimeSpan.FromSeconds(5));
|
||||
|
||||
// Exactly one Alarm item, carrying the same transition the alerts topic received.
|
||||
AwaitAssert(() => hub.Items.Count.ShouldBe(1), TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(100));
|
||||
var item = hub.Items[0].ShouldBeOfType<TelemetryItem.Alarm>();
|
||||
item.E.AlarmId.ShouldBe(AlarmRawPath);
|
||||
item.E.AlarmName.ShouldBe("temp_hi");
|
||||
item.E.TransitionKind.ShouldBe("Activated");
|
||||
item.E.Message.ShouldBe("temperature high");
|
||||
}
|
||||
|
||||
/// <summary>The scripted-alarm host's engine-emission seam fans the SAME <see cref="AlarmTransitionEvent"/>
|
||||
/// it publishes to the <c>alerts</c> topic into the hub as a <see cref="TelemetryItem.Alarm"/>. Drives a
|
||||
/// real engine Inactive→Active transition through the same harness the ScriptedAlarmHostActor tests use.</summary>
|
||||
[Fact]
|
||||
public void ScriptedAlarm_activation_emits_one_Alarm_item_with_matching_payload()
|
||||
{
|
||||
var publish = CreateTestProbe();
|
||||
var mux = CreateTestProbe();
|
||||
var hub = new CapturingHub();
|
||||
|
||||
var upstream = new DependencyMuxTagUpstreamSource();
|
||||
var logger = new LoggerConfiguration().CreateLogger();
|
||||
var engine = new ScriptedAlarmEngine(upstream, new InMemoryAlarmStateStore(), new ScriptLoggerFactory(logger), logger);
|
||||
var host = Sys.ActorOf(ScriptedAlarmHostActor.Props(
|
||||
publish.Ref, mux.Ref, upstream, engine, localNode: null, driverMemberCountProvider: null, telemetryHub: hub));
|
||||
|
||||
// One enabled alarm firing when M.T > 90; wait for load to complete (RegisterInterest lands).
|
||||
host.Tell(new ScriptedAlarmHostActor.ApplyScriptedAlarms(new[] { ScriptedPlan() }));
|
||||
mux.ExpectMsg<DependencyMuxActor.RegisterInterest>(TimeSpan.FromSeconds(8));
|
||||
|
||||
host.Tell(new VirtualTagActor.DependencyValueChanged("M.T", 99, DateTime.UtcNow));
|
||||
|
||||
// The OPC UA condition update confirms the transition was processed.
|
||||
publish.FishForMessage<OpcUaPublishActor.AlarmStateUpdate>(m => m.State.Active, TimeSpan.FromSeconds(8));
|
||||
|
||||
// Exactly one Alarm item, carrying the Activated transition the alerts topic received.
|
||||
AwaitAssert(() => hub.Items.Count.ShouldBe(1), TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(100));
|
||||
var item = hub.Items[0].ShouldBeOfType<TelemetryItem.Alarm>();
|
||||
item.E.AlarmId.ShouldBe("alm-1");
|
||||
item.E.TransitionKind.ShouldBe("Activated");
|
||||
item.E.Severity.ShouldBe(1000); // 800 → Critical bucket → 1000
|
||||
}
|
||||
|
||||
// --- script-logs seam (VirtualTagActor.PublishLog) → TelemetryItem.Script --------------------------
|
||||
|
||||
/// <summary>The virtual-tag actor's <c>PublishLog</c> seam emits a <see cref="TelemetryItem.Script"/> into
|
||||
/// the hub carrying the SAME <see cref="ScriptLogEntry"/> instance it hands to the publisher — proven via
|
||||
/// the test-only <c>publisherFactory</c> path (production uses the DPS Tell; the emit fires on both because
|
||||
/// it precedes the early-return branch).</summary>
|
||||
[Fact]
|
||||
public void VirtualTag_script_log_emits_one_Script_item_with_same_payload()
|
||||
{
|
||||
var hub = new CapturingHub();
|
||||
var published = new List<ScriptLogEntry>();
|
||||
var parent = CreateTestProbe();
|
||||
|
||||
var actor = parent.ChildActorOf(VirtualTagActor.Props(
|
||||
"vt-1", "broken",
|
||||
evaluator: new FailingEvaluator("syntax error"),
|
||||
scriptId: "script-7",
|
||||
publisherFactory: () => new DPSPublisher((_, payload) => published.Add((ScriptLogEntry)payload)),
|
||||
telemetryHub: hub));
|
||||
|
||||
actor.Tell(new VirtualTagActor.DependencyValueChanged("a", 1, DateTime.UtcNow));
|
||||
|
||||
AwaitAssert(() =>
|
||||
{
|
||||
published.Count.ShouldBe(1);
|
||||
hub.Items.Count.ShouldBe(1);
|
||||
}, TimeSpan.FromSeconds(2), TimeSpan.FromMilliseconds(100));
|
||||
|
||||
var item = hub.Items[0].ShouldBeOfType<TelemetryItem.Script>();
|
||||
item.E.ShouldBeSameAs(published[0]); // the exact instance handed to the publisher
|
||||
item.E.ScriptId.ShouldBe("script-7");
|
||||
item.E.VirtualTagId.ShouldBe("vt-1");
|
||||
item.E.Level.ShouldBe("Warning");
|
||||
}
|
||||
|
||||
/// <summary>The <c>M.T > 90</c> scripted-alarm plan shared by the activation test (mirrors the
|
||||
/// ScriptedAlarmHostActor test harness).</summary>
|
||||
private static EquipmentScriptedAlarmPlan ScriptedPlan() =>
|
||||
new(
|
||||
ScriptedAlarmId: "alm-1",
|
||||
EquipmentId: "Plant/Line1/M",
|
||||
Name: "HighTemp",
|
||||
AlarmType: "AlarmCondition",
|
||||
Severity: 800,
|
||||
MessageTemplate: "condition",
|
||||
PredicateScriptId: "alm-1-script",
|
||||
PredicateSource: "return (int)ctx.GetTag(\"M.T\").Value > 90;",
|
||||
DependencyRefs: new[] { "M.T" },
|
||||
HistorizeToAveva: true,
|
||||
Retain: true,
|
||||
Enabled: true);
|
||||
|
||||
/// <summary>Test evaluator that always fails, to drive <c>VirtualTagActor.PublishLog</c>.</summary>
|
||||
private sealed class FailingEvaluator : IVirtualTagEvaluator
|
||||
{
|
||||
private readonly string _reason;
|
||||
public FailingEvaluator(string reason) => _reason = reason;
|
||||
public VirtualTagEvalResult Evaluate(string id, string expr, IReadOnlyDictionary<string, object?> deps)
|
||||
=> VirtualTagEvalResult.Failure(_reason);
|
||||
}
|
||||
|
||||
/// <summary>Spawns the host wired with the capturing hub, dispatches the deployment, and waits for the
|
||||
/// Applied ack + the RebuildAddressSpace so the alarm map is built before the test raises an alarm.</summary>
|
||||
private (IActorRef Actor, Akka.TestKit.TestProbe Publish) SpawnHostAndApply(
|
||||
IDbContextFactory<OtOpcUaConfigDbContext> db, DeploymentId deploymentId, ITelemetryLocalHub hub)
|
||||
{
|
||||
var coordinator = CreateTestProbe();
|
||||
var publish = CreateTestProbe();
|
||||
var mux = CreateTestProbe();
|
||||
var vtHost = CreateTestProbe();
|
||||
|
||||
var actor = Sys.ActorOf(DriverHostActor.Props(
|
||||
db, TestNode, coordinator.Ref,
|
||||
localRoles: new HashSet<string> { "driver" },
|
||||
dependencyMux: mux.Ref,
|
||||
opcUaPublishActor: publish.Ref,
|
||||
virtualTagEvaluator: NullVirtualTagEvaluator.Instance,
|
||||
virtualTagHostOverride: vtHost.Ref,
|
||||
telemetryHub: hub));
|
||||
|
||||
actor.Tell(new DispatchDeployment(deploymentId, RevA, CorrelationId.NewId()));
|
||||
coordinator.ExpectMsg<ApplyAck>(TimeSpan.FromSeconds(5)).Outcome.ShouldBe(ApplyAckOutcome.Applied);
|
||||
publish.ExpectMsg<OpcUaPublishActor.RebuildAddressSpace>(TimeSpan.FromSeconds(5));
|
||||
|
||||
return (actor, publish);
|
||||
}
|
||||
|
||||
/// <summary>Seeds a Sealed v3 deployment with a single Boolean raw tag carrying an <c>alarm</c> object so
|
||||
/// the composer projects it as a Part 9 condition at its RawPath (mirrors DriverHostActorNativeAlarmTests).</summary>
|
||||
private static DeploymentId SeedV3AlarmDeployment(IDbContextFactory<OtOpcUaConfigDbContext> db, RevisionHash rev)
|
||||
{
|
||||
var artifact = JsonSerializer.SerializeToUtf8Bytes(new
|
||||
{
|
||||
RawFolders = new[] { new { RawFolderId = "rf-plant", ParentRawFolderId = (string?)null, Name = "Plant", ClusterId = "c1" } },
|
||||
DriverInstances = new[]
|
||||
{
|
||||
new { DriverInstanceId = "drv-1", RawFolderId = "rf-plant", Name = "Modbus", DriverType = "Modbus", DriverConfig = "{}", ClusterId = "c1", Enabled = false },
|
||||
},
|
||||
Devices = new[]
|
||||
{
|
||||
new { DeviceId = "drv-1:dev1", DriverInstanceId = "drv-1", Name = "dev1", DeviceConfig = "{}" },
|
||||
},
|
||||
TagGroups = Array.Empty<object>(),
|
||||
Tags = new[]
|
||||
{
|
||||
new
|
||||
{
|
||||
TagId = "tag-0",
|
||||
DeviceId = "drv-1:dev1",
|
||||
TagGroupId = (string?)null,
|
||||
Name = "temp_hi",
|
||||
DataType = "Boolean",
|
||||
AccessLevel = 0,
|
||||
TagConfig = JsonSerializer.Serialize(new { alarm = new { alarmType = "OffNormalAlarm", severity = 700 } }),
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
var id = DeploymentId.NewId();
|
||||
using var ctx = db.CreateDbContext();
|
||||
ctx.Deployments.Add(new Deployment
|
||||
{
|
||||
DeploymentId = id.Value,
|
||||
RevisionHash = rev.Value,
|
||||
Status = DeploymentStatus.Sealed,
|
||||
CreatedBy = "test",
|
||||
SealedAtUtc = DateTime.UtcNow,
|
||||
ArtifactBlob = artifact,
|
||||
});
|
||||
ctx.SaveChanges();
|
||||
return id;
|
||||
}
|
||||
|
||||
/// <summary>Minimal alarm-subscription handle for building <see cref="AlarmEventArgs"/>.</summary>
|
||||
private sealed class StubAlarmHandle : IAlarmSubscriptionHandle
|
||||
{
|
||||
public string DiagnosticId => "stub-alarm-sub";
|
||||
}
|
||||
|
||||
/// <summary>Fake hub that records every <see cref="ITelemetryLocalHub.Emit"/>. <see cref="Subscribe"/> is
|
||||
/// unused by the producer seams under test.</summary>
|
||||
private sealed class CapturingHub : ITelemetryLocalHub
|
||||
{
|
||||
public List<TelemetryItem> Items { get; } = new();
|
||||
|
||||
public void Emit(TelemetryItem item) => Items.Add(item);
|
||||
|
||||
public ITelemetrySubscription Subscribe(int boundedCapacity) => throw new NotSupportedException();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,249 @@
|
||||
using System.Collections.Concurrent;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Alerts;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Drivers;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Logging;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Telemetry;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Telemetry;
|
||||
|
||||
/// <summary>
|
||||
/// Verifies the node-local telemetry hub (per-cluster mesh Phase 5): snapshot-replay of the two
|
||||
/// snapshot-style channels (Health / Resilience), live-only forwarding of the append-style logs
|
||||
/// (Alarm / Script), independent bounded per-subscriber fan-out, DropOldest backpressure, and
|
||||
/// detach-on-dispose. The hub carries ONLY this node's own telemetry — it never touches DPS.
|
||||
/// </summary>
|
||||
public sealed class TelemetryLocalHubTests
|
||||
{
|
||||
private static DriverHealthChanged Health(string instanceId, string state = "Healthy") =>
|
||||
new("cluster-a", instanceId, state, null, null, 0, DateTime.UtcNow);
|
||||
|
||||
private static DriverResilienceStatusChanged Resilience(string instanceId, string host) =>
|
||||
new(instanceId, host, false, 0, 0, null, DateTime.UtcNow, DateTime.UtcNow);
|
||||
|
||||
private static AlarmTransitionEvent Alarm(string id) =>
|
||||
new(id, "Area/Line/Equip", "AlarmName", "Activated", 500, "msg", "system", DateTime.UtcNow);
|
||||
|
||||
private static ScriptLogEntry Script(string id) =>
|
||||
new(id, "Information", "log", DateTime.UtcNow, null, null, null);
|
||||
|
||||
private static List<TelemetryItem> Drain(ITelemetrySubscription sub)
|
||||
{
|
||||
var items = new List<TelemetryItem>();
|
||||
while (sub.Reader.TryRead(out var item))
|
||||
items.Add(item);
|
||||
return items;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void New_subscriber_is_primed_with_cached_Health_snapshot()
|
||||
{
|
||||
var hub = new TelemetryLocalHub();
|
||||
hub.Emit(new TelemetryItem.Health(Health("drv-1", "Faulted")));
|
||||
|
||||
using var sub = hub.Subscribe(boundedCapacity: 16);
|
||||
|
||||
var items = Drain(sub);
|
||||
items.ShouldHaveSingleItem();
|
||||
var health = items[0].ShouldBeOfType<TelemetryItem.Health>();
|
||||
health.E.DriverInstanceId.ShouldBe("drv-1");
|
||||
health.E.State.ShouldBe("Faulted");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void New_subscriber_is_primed_with_cached_Resilience_snapshot()
|
||||
{
|
||||
var hub = new TelemetryLocalHub();
|
||||
hub.Emit(new TelemetryItem.Resilience(Resilience("drv-1", "host-a")));
|
||||
|
||||
using var sub = hub.Subscribe(boundedCapacity: 16);
|
||||
|
||||
var items = Drain(sub);
|
||||
items.ShouldHaveSingleItem();
|
||||
var res = items[0].ShouldBeOfType<TelemetryItem.Resilience>();
|
||||
res.E.DriverInstanceId.ShouldBe("drv-1");
|
||||
res.E.HostName.ShouldBe("host-a");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Alarm_and_Script_are_not_replayed_to_a_new_subscriber()
|
||||
{
|
||||
var hub = new TelemetryLocalHub();
|
||||
hub.Emit(new TelemetryItem.Alarm(Alarm("a-1")));
|
||||
hub.Emit(new TelemetryItem.Script(Script("s-1")));
|
||||
|
||||
using var sub = hub.Subscribe(boundedCapacity: 16);
|
||||
|
||||
Drain(sub).ShouldBeEmpty();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Multiple_subscribers_each_receive_their_own_copy_of_a_post_subscribe_emit()
|
||||
{
|
||||
var hub = new TelemetryLocalHub();
|
||||
using var s1 = hub.Subscribe(boundedCapacity: 16);
|
||||
using var s2 = hub.Subscribe(boundedCapacity: 16);
|
||||
|
||||
hub.Emit(new TelemetryItem.Alarm(Alarm("a-1")));
|
||||
|
||||
Drain(s1).ShouldHaveSingleItem().ShouldBeOfType<TelemetryItem.Alarm>().E.AlarmId.ShouldBe("a-1");
|
||||
Drain(s2).ShouldHaveSingleItem().ShouldBeOfType<TelemetryItem.Alarm>().E.AlarmId.ShouldBe("a-1");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Health_snapshot_keeps_only_the_latest_per_instance()
|
||||
{
|
||||
var hub = new TelemetryLocalHub();
|
||||
hub.Emit(new TelemetryItem.Health(Health("drv-1", "Reconnecting")));
|
||||
hub.Emit(new TelemetryItem.Health(Health("drv-1", "Healthy")));
|
||||
|
||||
using var sub = hub.Subscribe(boundedCapacity: 16);
|
||||
|
||||
var items = Drain(sub);
|
||||
items.ShouldHaveSingleItem();
|
||||
items[0].ShouldBeOfType<TelemetryItem.Health>().E.State.ShouldBe("Healthy");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Health_snapshot_keeps_distinct_instances()
|
||||
{
|
||||
var hub = new TelemetryLocalHub();
|
||||
hub.Emit(new TelemetryItem.Health(Health("drv-1")));
|
||||
hub.Emit(new TelemetryItem.Health(Health("drv-2")));
|
||||
|
||||
using var sub = hub.Subscribe(boundedCapacity: 16);
|
||||
|
||||
var ids = Drain(sub)
|
||||
.OfType<TelemetryItem.Health>()
|
||||
.Select(h => h.E.DriverInstanceId)
|
||||
.OrderBy(x => x)
|
||||
.ToArray();
|
||||
ids.ShouldBe(new[] { "drv-1", "drv-2" });
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Resilience_snapshot_keeps_only_latest_per_instance_and_host()
|
||||
{
|
||||
var hub = new TelemetryLocalHub();
|
||||
// Same instance, two distinct hosts → two cache entries.
|
||||
hub.Emit(new TelemetryItem.Resilience(Resilience("drv-1", "host-a")));
|
||||
hub.Emit(new TelemetryItem.Resilience(Resilience("drv-1", "host-b")));
|
||||
// Overwrite (drv-1, host-a).
|
||||
hub.Emit(new TelemetryItem.Resilience(Resilience("drv-1", "host-a")));
|
||||
|
||||
using var sub = hub.Subscribe(boundedCapacity: 16);
|
||||
|
||||
var keys = Drain(sub)
|
||||
.OfType<TelemetryItem.Resilience>()
|
||||
.Select(r => (r.E.DriverInstanceId, r.E.HostName))
|
||||
.OrderBy(x => x.HostName)
|
||||
.ToArray();
|
||||
keys.ShouldBe(new[] { ("drv-1", "host-a"), ("drv-1", "host-b") });
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Bounded_channel_drops_oldest_not_newest_when_full()
|
||||
{
|
||||
var hub = new TelemetryLocalHub();
|
||||
using var sub = hub.Subscribe(boundedCapacity: 2);
|
||||
|
||||
// Append-style (uncached) items so the snapshot doesn't consume capacity.
|
||||
hub.Emit(new TelemetryItem.Alarm(Alarm("a-1")));
|
||||
hub.Emit(new TelemetryItem.Alarm(Alarm("a-2")));
|
||||
hub.Emit(new TelemetryItem.Alarm(Alarm("a-3")));
|
||||
|
||||
var ids = Drain(sub).OfType<TelemetryItem.Alarm>().Select(a => a.E.AlarmId).ToArray();
|
||||
// a-1 (oldest) evicted; the two newest survive in order.
|
||||
ids.ShouldBe(new[] { "a-2", "a-3" });
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Subscribe_with_non_positive_capacity_throws()
|
||||
{
|
||||
var hub = new TelemetryLocalHub();
|
||||
Should.Throw<ArgumentOutOfRangeException>(() => hub.Subscribe(0));
|
||||
Should.Throw<ArgumentOutOfRangeException>(() => hub.Subscribe(-1));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Concurrent_emit_subscribe_dispose_is_safe_and_never_double_delivers_a_snapshot()
|
||||
{
|
||||
var hub = new TelemetryLocalHub();
|
||||
var stop = new CancellationTokenSource(TimeSpan.FromSeconds(3));
|
||||
var exceptions = new ConcurrentQueue<Exception>();
|
||||
|
||||
// Writers: hammer Emit with a mix of all four kinds across a small key space so the snapshot
|
||||
// caches churn (same instance/host reused → last-value overwrites) while subscribers attach.
|
||||
var writers = Enumerable.Range(0, 4).Select(w => Task.Run(() =>
|
||||
{
|
||||
var rnd = new Random(w * 7919 + 1);
|
||||
try
|
||||
{
|
||||
var n = 0;
|
||||
while (!stop.IsCancellationRequested)
|
||||
{
|
||||
var instance = "drv-" + rnd.Next(0, 4);
|
||||
var host = "host-" + rnd.Next(0, 3);
|
||||
TelemetryItem item = (n++ % 4) switch
|
||||
{
|
||||
0 => new TelemetryItem.Health(Health(instance, "s" + n)),
|
||||
1 => new TelemetryItem.Resilience(Resilience(instance, host)),
|
||||
2 => new TelemetryItem.Alarm(Alarm("a-" + n)),
|
||||
_ => new TelemetryItem.Script(Script("s-" + n)),
|
||||
};
|
||||
hub.Emit(item);
|
||||
}
|
||||
}
|
||||
catch (Exception ex) { exceptions.Enqueue(ex); }
|
||||
})).ToArray();
|
||||
|
||||
// Subscriber/disposer churn: attach, drain, verify the exactly-once-across-attach invariant, dispose.
|
||||
var churners = Enumerable.Range(0, 6).Select(_ => Task.Run(() =>
|
||||
{
|
||||
try
|
||||
{
|
||||
while (!stop.IsCancellationRequested)
|
||||
{
|
||||
using var sub = hub.Subscribe(boundedCapacity: 4096);
|
||||
// The snapshot prelude is fully written before Subscribe returns, so any Health/
|
||||
// Resilience item read here that was NOT part of the prelude must be a live delta —
|
||||
// i.e. it arrived strictly after attach. The invariant we assert: for one snapshot
|
||||
// KEY, the hub never delivers the SAME snapshot instance both in the prelude and
|
||||
// again live. We approximate "same snapshot" by reference identity: a live re-emit
|
||||
// is always a freshly-allocated record, so a reference-equal duplicate would be a
|
||||
// genuine double-delivery of one cached object across the boundary.
|
||||
var seen = new HashSet<TelemetryItem>(ReferenceEqualityComparer.Instance);
|
||||
while (sub.Reader.TryRead(out var item))
|
||||
{
|
||||
// (c) every drained item is a valid, non-null TelemetryItem of a known kind.
|
||||
item.ShouldNotBeNull();
|
||||
item.ShouldBeAssignableTo<TelemetryItem>();
|
||||
// (b) no cached snapshot object delivered twice to THIS subscriber.
|
||||
seen.Add(item).ShouldBeTrue();
|
||||
}
|
||||
}
|
||||
}
|
||||
catch (Exception ex) { exceptions.Enqueue(ex); }
|
||||
})).ToArray();
|
||||
|
||||
await Task.WhenAll(writers.Concat(churners));
|
||||
|
||||
// (a) no thread threw.
|
||||
exceptions.ShouldBeEmpty();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Dispose_detaches_the_subscriber_and_completes_its_channel()
|
||||
{
|
||||
var hub = new TelemetryLocalHub();
|
||||
var sub = hub.Subscribe(boundedCapacity: 16);
|
||||
|
||||
sub.Dispose();
|
||||
|
||||
// A further Emit must neither throw nor deliver to the detached subscriber.
|
||||
Should.NotThrow(() => hub.Emit(new TelemetryItem.Alarm(Alarm("a-1"))));
|
||||
sub.Reader.Completion.IsCompleted.ShouldBeTrue();
|
||||
sub.Reader.TryRead(out _).ShouldBeFalse();
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user