From 53881c6220cf0bbf063320f58b6988a39f6b33cb Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sat, 15 Aug 2026 20:51:36 -0400 Subject: [PATCH] docs(dashboard): in-process page feeds, two-tier idle gating, hubs as the external surface --- docs/GatewayDashboardDesign.md | 202 ++++++++++++++++++++++++++------- 1 file changed, 161 insertions(+), 41 deletions(-) diff --git a/docs/GatewayDashboardDesign.md b/docs/GatewayDashboardDesign.md index 77be9d4..97500d3 100644 --- a/docs/GatewayDashboardDesign.md +++ b/docs/GatewayDashboardDesign.md @@ -98,6 +98,7 @@ ZB.MOM.WW.MxGateway.Server StatusBadge.razor FaultList.razor DashboardSnapshotService.cs + DashboardSnapshotFeed.cs DashboardAuthorizationHandler.cs DashboardAuthenticator.cs DashboardApiKeyAuthorization.cs @@ -110,9 +111,15 @@ ZB.MOM.WW.MxGateway.Server ``` The dashboard exposes three named SignalR hubs in addition to Blazor Server's -internal circuit; pages connect to those hubs from within the circuit via the -`DashboardHubConnectionFactory` helper. The hubs publish snapshot, alarm, and -per-session event updates that the pages render in place of polling. +internal circuit. The hubs are the **remote** surface: they publish snapshot, +alarm, and per-session event updates to clients outside the gateway process. +Server-rendered Blazor pages do not use them. A page runs inside this process, +so it consumes the producing services directly through in-process seams — +`IDashboardSnapshotFeed`, `IDashboardSessionEventSubscriber`, and +`IGatewayAlarmService` — instead of opening a loopback WebSocket back into its +own heap. `DashboardHubConnectionFactory`, the helper a circuit used to open +those connections, stays registered for out-of-tree consumers, but no in-repo +page resolves it. ## Dashboard Data Source @@ -159,7 +166,57 @@ gateway internals. ## Realtime Updates -Updates flow over three SignalR hubs, all guarded by the +Realtime data reaches two audiences over two seams: + +- **in-process**, for the server-rendered Blazor pages, which run inside the + gateway process and read the producing services directly; +- **SignalR hubs**, for clients outside the process. + +Pages originally took the hub path too, which put a loopback WebSocket, a +hub-token mint, and a serialize/deserialize round trip between a Blazor component +and an object already in its own heap. The in-process seams remove that hop. The +hubs stay for the audience that genuinely needs a wire. + +### In-process page feeds + +| Page | Seam | Producer | +|---|---|---| +| every page deriving from `DashboardPageBase` | `IDashboardSnapshotFeed.WatchAsync` | `DashboardSnapshotFeed` (singleton) multicasting one `IDashboardSnapshotService.WatchSnapshotsAsync` enumeration | +| `SessionDetailsPage` | `IDashboardSessionEventSubscriber.Subscribe(sessionId)` | `DashboardEventBroadcaster` — the same singleton the session mirror publishes to, registered behind both interfaces | +| `AlarmsPage` | `IGatewayAlarmService.StreamAsync` | the central alarm monitor, **provider status only**; the alarm rows still come from the 3 s `QueryAlarmsAsync` poll | + +The snapshot feed multicasts rather than handing each page its own enumeration: +`WatchSnapshotsAsync` is not multicast on its own — each enumeration owns a timer +and builds its own snapshot per tick — so a subscription per page would multiply +the snapshot cost by the number of open pages. Each subscriber reads through a +capacity-1 drop-oldest channel, so a circuit that renders slowly skips snapshots +instead of buffering without bound or stalling the pump. + +`DashboardPageBase` seeds `Snapshot` synchronously from +`IDashboardSnapshotService.GetSnapshot()` in `OnInitializedAsync` so the first +render is non-empty, then calls `InvokeAsync(StateHasChanged)` for every snapshot +the feed yields. On dispose it cancels the watch and waits at most **5 seconds** +for the loop to drain, logging a warning on timeout. The bound is deliberate: the +loop marshals renders through the renderer's dispatcher and disposal can run on +that same dispatcher, so an unconditional wait would hang on a wedged dispatcher. +The accepted cost is that an abandoned loop still holds its feed subscription — the +feed's idle gate stays open until it unwinds — and the warning is the operator's +only signal that a circuit teardown wedged. + +`SessionDetailsPage` subscribes for the current session id and renders the most +recent N events (default 50) in a "Recent events" table. Its pump drains everything +queued and renders once per batch rather than once per event, and it re-checks +**inside the renderer dispatch** — where the subscription field is written, making +the check an unsynchronized read of dispatcher-owned state — that the batch's +subscription is still the live one. A batch read before a session switch would +otherwise render the previous session's events under the new session's heading. +Detaching cancels the pump, disposes the subscription (which releases the viewer +registration and completes the channel, so the pump has an exit even if +cancellation is missed), then drains under its own timeout. + +### SignalR hubs (remote clients) + +Updates for out-of-process clients flow over three SignalR hubs, all guarded by the `MxGateway.Dashboard.HubClients` policy (cookie OR `MxGateway.Dashboard.HubToken` bearer). Each hub class is `[Authorize(Policy = HubClientsPolicy)]`. @@ -167,46 +224,68 @@ bearer). Each hub class is `[Authorize(Policy = HubClientsPolicy)]`. |---|---|---|---|---| | `DashboardSnapshotHub` | `/hubs/snapshot` | `DashboardSnapshotPublisher` (BackgroundService consuming `IDashboardSnapshotService.WatchSnapshotsAsync`) | `DashboardSnapshot` | Sent to all connected clients on every snapshot tick, but only while at least one client is connected (see "Idle gating" below); new connections receive the current snapshot synchronously in `OnConnectedAsync`. | | `AlarmsHub` | `/hubs/alarms` | `AlarmsHubPublisher` (BackgroundService consuming `IGatewayAlarmService.StreamAsync(filter: null)`) | `AlarmFeedMessage` (`active_alarm` / `snapshot_complete` / `transition`) | Connected clients auto-join `__alarms__`; all clients receive every message. Publisher auto-reconnects every 5s on stream faults. | -| `EventsHub` | `/hubs/events` | `DashboardEventBroadcaster` invoked by each session's internal dashboard-mirror subscriber on its `SessionEventDistributor` (registered when the session becomes Ready) | `MxEvent` | Clients call `SubscribeSession(sessionId)` to join `session:{id}`, which also registers them in `EventsHubViewerRegistry` — the mirror is gated on that registry (see "Mirror gating" below). The dashboard is a first-class distributor subscriber, so it receives the session's events whether or not a gRPC client is streaming. It sees RAW session events — not the per-gRPC-subscriber `AfterWorkerSequence` filtering that `EventStreamService` applies at its own boundary — because the dashboard is a separate LDAP-authenticated monitoring view meant to show the session's full event activity. Tag values are stripped from the mirrored `MxEvent` copy by `DashboardEventBroadcaster` when `Dashboard:ShowTagValues` is false (the default) — event metadata (tag reference, quality, status, timestamps) still renders, but the value fields are blanked, so no value leaks through this seam. The per-session hub ACL that would scope a Viewer to specific sessions is still outstanding (SEC-25 / remediation roadmap item 12); the value redaction is the near-term hardening that closes the value-leak seam independently of that ACL. | +| `EventsHub` | `/hubs/events` | `DashboardEventBroadcaster` invoked by each session's internal dashboard-mirror subscriber on its `SessionEventDistributor` (registered when the session becomes Ready) | `MxEvent` | Clients call `SubscribeSession(sessionId)` to join `session:{id}`, which also registers them in `EventsHubViewerRegistry` — the mirror is gated on that registry, which counts hub and in-process viewers alike (see "Mirror gating" below). The dashboard is a first-class distributor subscriber, so it receives the session's events whether or not a gRPC client is streaming. It sees RAW session events — not the per-gRPC-subscriber `AfterWorkerSequence` filtering that `EventStreamService` applies at its own boundary — because the dashboard is a separate LDAP-authenticated monitoring view meant to show the session's full event activity. Tag values are stripped from the mirrored `MxEvent` copy by `DashboardEventBroadcaster` when `Dashboard:ShowTagValues` is false (the default) — event metadata (tag reference, quality, status, timestamps) still renders, but the value fields are blanked, so no value leaks through this seam. The per-session ACL that would scope a Viewer to specific sessions is still outstanding for this seam and the in-process one alike (SEC-25 / remediation roadmap item 12); the value redaction is the near-term hardening that closes the value-leak seam independently of that ACL. | -`DashboardPageBase` opens a `DashboardSnapshotHub` connection via the connection -factory in `OnInitializedAsync`, seeds `Snapshot` synchronously from -`IDashboardSnapshotService.GetSnapshot()` so the first render is non-empty, and -calls `InvokeAsync(StateHasChanged)` on every `SnapshotUpdated` push. SignalR's -`WithAutomaticReconnect` handles transient disconnects. +### Default cadences -`SessionDetailsPage` additionally opens an `EventsHub` connection for the -current session id and renders the most recent N events (default 50) in a -"Recent events" table with a live/offline connection pill. - -Default cadences: +Both seams consume the same producing services, so they share these cadences: - snapshot service produces one snapshot per `MxGateway:Dashboard:SnapshotIntervalMilliseconds` (default 1s); - alarm publisher emits on each transition observed by the central monitor; - event publisher emits per event fanned by the session's `SessionEventDistributor` - to its internal dashboard-mirror subscriber (independent of any gRPC `StreamEvents`). + to its internal dashboard-mirror subscriber (independent of any gRPC `StreamEvents`); +- the alarms page's provider-status badge resubscribes one second after its + `IGatewayAlarmService.StreamAsync` enumeration ends — the monitor completes a + subscriber's stream when it falls behind and again when it restarts, both + recoverable by resubscribing — and holds its last value in between. The page's + alarm rows are independent of that stream and refresh on the 3 s poll. ### Idle gating and snapshot cost A snapshot is not free: each one takes a session-registry snapshot and sorts it, copies the metrics dictionaries under the global metrics lock, and projects sessions, workers, faults, and the Galaxy summary. Without gating that work ran -once a second for the life of the process even when no browser was connected. +once a second for the life of the process even when nothing was watching. -`DashboardSnapshotHub` counts live connections into the singleton +Gating is two-tier, because the two seams have independent audiences and each must +be able to reach zero on its own. + +**Hub tier.** `DashboardSnapshotHub` counts live connections into the singleton `DashboardSnapshotHubConnectionCounter` (`OnConnectedAsync` / `OnDisconnectedAsync`, clamped at zero). `DashboardSnapshotPublisher` reads that count before advancing the snapshot enumerator: while it is zero the publisher does not call `MoveNextAsync` at all, so the producing iterator stays suspended at its `yield` and builds nothing — the gate removes the snapshot *build*, not just the broadcast. The publisher -re-checks once a second while idle, so the first viewer to connect resumes the tick -within roughly one snapshot interval. That viewer does not wait for it either: -`DashboardPageBase` seeds its first render synchronously from -`IDashboardSnapshotService.GetSnapshot()`, and `OnConnectedAsync` pushes a snapshot -to the new connection immediately. +re-checks once a second while idle, so the first client to connect resumes the tick +within roughly one snapshot interval, and `OnConnectedAsync` pushes the current +snapshot to that connection immediately. Now that no in-repo page connects to the +hub, this tier stays idle unless a remote client connects. -Two per-tick costs inside the snapshot itself are bounded independently of the gate: +**In-process tier.** `DashboardSnapshotFeed` gates its own pump on its subscriber +list: the first subscriber starts the pump, the last one leaving cancels it and +awaits it, so a gateway with no page open runs no timer and builds no snapshots on +this seam either. Successive pumps are chained through the previous pump's task, so +an unsubscribe immediately followed by a resubscribe restarts a fresh pump without +ever running two enumerations at once. A page does not wait for the pump's first +tick — `DashboardPageBase` seeds its first render synchronously from +`IDashboardSnapshotService.GetSnapshot()`. + +That gate is generation-scoped rather than a plain subscriber count. Each pump owns +a generation, each subscriber is tagged with the generation it joined under, a pump +ends its generation the instant its source faults or completes — before the possibly +slow enumerator disposal — and a dying pump only ever detaches its own generation's +subscribers. Two races motivate the extra state: + +- a subscriber arriving mid-teardown must start a fresh generation rather than + attach to a pump that is about to detach everybody and leave nobody watching; +- an unsubscribe must compare its own generation against the live one before it + cancels anything. Subscribers of an ending generation linger in the list until + that pump's reset runs, so counting the whole list would let them hold the idle + gate open, and cancelling on their behalf would stop a live pump that other + viewers depend on. + +Two per-tick costs inside the snapshot itself are bounded independently of either gate: - the effective configuration (`EffectiveGatewayConfiguration`) is built once and cached. It is a projection of `IOptions`, which the gateway binds @@ -220,26 +299,43 @@ Two per-tick costs inside the snapshot itself are bounded independently of the g retried on the next tick and the previous summaries stay on screen. Avoid pushing every MXAccess data-change event into a wider broadcast group. -The current design routes events strictly through `session:{id}` groups; the -snapshot hub continues to carry aggregate event counters and rates. +Events are routed strictly per session (`session:{id}` groups on the hub, +per-session subscriber lists in process); the snapshot seams continue to carry +aggregate event counters and rates. ### Mirror gating Each session's dashboard-mirror subscriber calls `DashboardEventBroadcaster.Publish` for every event the session produces, -independently of whether any browser is watching that session. SignalR does not -expose group membership, so the broadcaster cannot ask whether `session:{id}` is -empty. `EventsHubViewerRegistry` (singleton) supplies that answer: `EventsHub` -mirrors its own `AddToGroup` / `RemoveFromGroup` calls into it, and -`OnDisconnectedAsync` releases every subscription a dropped connection held — the -only reliable signal for a browser tab that closes without unsubscribing. -`Publish` returns immediately when `HasViewers(sessionId)` is false, **before** -the redaction clone. That matters because redaction is on by default +independently of whether anything is watching that session. `Publish` returns +immediately when `EventsHubViewerRegistry.HasViewers(sessionId)` is false, +**before** the redaction clone. That matters because redaction is on by default (`Dashboard:ShowTagValues` false), so the unwatched steady state — nobody on any session-details page — previously paid a deep protobuf clone plus a send to an empty group for every event of every session. Behaviour for a watched session is unchanged. +The registry counts both audiences, which is what lets one gate serve both seams. +`EventsHub` mirrors its own `AddToGroup` / `RemoveFromGroup` calls into it — +SignalR does not expose group membership, so the broadcaster cannot ask whether +`session:{id}` is empty — and `OnDisconnectedAsync` releases every subscription a +dropped connection held, the only reliable signal for a browser tab that closes +without unsubscribing. An in-process subscription registers the same way under a +synthetic `inproc-`-prefixed connection id, which cannot collide with a SignalR +connection id and makes the origin obvious in a debugger; disposing it removes the +viewer and releases the synthetic connection, because that id never reconnects and +nothing else would ever release its per-connection entry. Both paths use the same +ordering — register before becoming a delivery target, deregister after ceasing to +be one — so the widest a race window opens is a redaction clone that reaches +nobody, never a dropped event that was owed to a live viewer. + +Redaction happens once per event, not once per audience: `Publish` produces a +single redacted clone and hands that same instance to the in-process subscribers +and to the hub group. In-process delivery runs first and synchronously — it cannot +throw, and it must not be skipped by the guard clause around the hub send — into +per-subscriber bounded drop-oldest channels, so a page that falls behind loses its +oldest queued events rather than blocking the session's event pipeline. + The mirror subscriber itself is still registered on the `SessionEventDistributor` for the session's whole lifetime; only the per-event work is gated. Starting and stopping the mirror lease lazily with the first and last viewer was considered @@ -369,8 +465,9 @@ panel. The panel shows each subscribed tag's live value, MXAccess data type, quality and source timestamp, refreshed every two seconds. The subscription panel is the explicit opt-in tag-value surface: it always shows values regardless of `Dashboard:ShowTagValues`, which governs the diagnostic -session/worker views and the per-session `EventsHub` mirror (values are -redacted from the mirrored events when the flag is false). +session/worker views and the per-session event mirror — both its hub and +in-process audiences (values are redacted from the mirrored events when the flag +is false). ### Alarms page @@ -380,7 +477,11 @@ defaults to showing unacknowledged `Active` alarms; filters add acknowledged alarms and narrow by area, severity range, and a reference/source/description text search. Cleared alarms are not retained — the gateway holds no alarm-history store, so the page reflects only the live active set. The page is -read-only; it does not acknowledge alarms. If `MxGateway:Alarms:Enabled` is +read-only; it does not acknowledge alarms. A provider-status badge tracks the +central monitor's health from `IGatewayAlarmService.StreamAsync` in process — the +alarm service is already a multi-subscriber fan-out, so the badge needs no SignalR +client, no loopback socket, and no hub token — while the alarm rows themselves +still come from the three-second poll. If `MxGateway:Alarms:Enabled` is false the central monitor never starts, and the page says so instead of showing an empty list with no explanation. @@ -536,6 +637,14 @@ Three authorization policies are registered: cookie OR a `MxGateway.Dashboard.HubToken` bearer (used by WebSocket upgrades where the cookie can't be forwarded). +The in-process page feeds carry no authentication of their own, and need none: +`MapRazorComponents()` applies `RequireAuthorization(ViewerPolicy)` to the +component endpoints, so a page can only run inside a circuit whose principal is +already an authorized Viewer. The hub-token flow below therefore covers only the +remote hub surface. Neither seam scopes a Viewer to particular sessions — SEC-25 +(the per-session ACL) is outstanding for both, and the mirror's value redaction +remains the near-term mitigation, unchanged by the move in-process. + Two environmental bypasses still apply, both scoped to **read-only** access: `MxGateway:Authentication:Mode = Disabled` and `MxGateway:Dashboard:AllowAnonymousLocalhost` (default `true`, loopback only) each satisfy a requirement that includes the Viewer @@ -580,12 +689,15 @@ surface is affected. Never enable in production. ### Hub bearer flow +This flow serves remote hub clients only; in-process pages are authorized by the +component endpoint's `ViewerPolicy` and never mint a token. + SignalR connections cannot reuse the `__Host-` cookie when the JS client upgrades to WebSocket — the cookie's `SameSite=Strict; Path=/` keeps it from being forwarded by the browser's WebSocket layer in some edge cases. The dashboard mints short-lived bearer tokens for the connection: -1. The cookie-authenticated Blazor page calls `GET /hubs/token` +1. The cookie-authenticated client calls `GET /hubs/token` (gated by `ViewerPolicy`, cookie-only). 2. `HubTokenService.Issue(user)` serializes the user's name, NameIdentifier, and role claims to JSON, encrypts with the ASP.NET Core data-protection @@ -605,7 +717,9 @@ dashboard mints short-lived bearer tokens for the connection: `DashboardHubConnectionFactory` (scoped to the Blazor circuit) wraps the HubConnectionBuilder and supplies a fresh token via `AccessTokenProvider` on -every (re)connect, so the short 5-minute lifetime is transparent to clients. +every (re)connect, so the short 5-minute lifetime is transparent to whoever uses +it. It remains registered, but no in-repo page opens a hub connection any more; +external clients implement the equivalent refresh themselves. Caveat — logout does not revoke outstanding tokens. Logout clears the dashboard cookie, but hub bearer tokens are self-contained, data-protection-encrypted, and @@ -700,8 +814,14 @@ Integration tests should verify: - a user in a Viewer-mapped LDAP group can render every page but cannot invoke the Admin-only management actions, - a user with no mapped LDAP group cannot sign in at all, -- live snapshot updates when a fake session changes state are delivered - via the `/hubs/snapshot` push, not by polling. +- live snapshot updates when a fake session changes state reach a page through + the in-process `IDashboardSnapshotFeed` and reach a remote client through the + `/hubs/snapshot` push — neither by polling; +- the snapshot feed's idle gate: no subscribers means no pump, the last + subscriber of the live generation stops it, and a subscriber arriving + mid-teardown gets a fresh generation rather than a dead one; +- the event mirror's viewer gate counts in-process subscriptions as well as hub + connections, and a disposed in-process subscription releases its viewer count. ## Initial Implementation Slice