docs(dashboard): in-process page feeds, two-tier idle gating, hubs as the external surface

docs/GatewayDashboardDesign.md

@@ -98,6 +98,7 @@ ZB.MOM.WW.MxGateway.Server
         StatusBadge.razor
         FaultList.razor
     DashboardSnapshotService.cs
+    DashboardSnapshotFeed.cs
     DashboardAuthorizationHandler.cs
     DashboardAuthenticator.cs
     DashboardApiKeyAuthorization.cs
@@ -110,9 +111,15 @@ ZB.MOM.WW.MxGateway.Server
 ```
 
 The dashboard exposes three named SignalR hubs in addition to Blazor Server's
-internal circuit; pages connect to those hubs from within the circuit via the
+internal circuit. The hubs are the **remote** surface: they publish snapshot,
-`DashboardHubConnectionFactory` helper. The hubs publish snapshot, alarm, and
+alarm, and per-session event updates to clients outside the gateway process.
-per-session event updates that the pages render in place of polling.
+Server-rendered Blazor pages do not use them. A page runs inside this process,
+so it consumes the producing services directly through in-process seams —
+`IDashboardSnapshotFeed`, `IDashboardSessionEventSubscriber`, and
+`IGatewayAlarmService` — instead of opening a loopback WebSocket back into its
+own heap. `DashboardHubConnectionFactory`, the helper a circuit used to open
+those connections, stays registered for out-of-tree consumers, but no in-repo
+page resolves it.
 
 ## Dashboard Data Source
 
@@ -159,7 +166,57 @@ gateway internals.
 
 ## Realtime Updates
 
-Updates flow over three SignalR hubs, all guarded by the
+Realtime data reaches two audiences over two seams:
+
+- **in-process**, for the server-rendered Blazor pages, which run inside the
+  gateway process and read the producing services directly;
+- **SignalR hubs**, for clients outside the process.
+
+Pages originally took the hub path too, which put a loopback WebSocket, a
+hub-token mint, and a serialize/deserialize round trip between a Blazor component
+and an object already in its own heap. The in-process seams remove that hop. The
+hubs stay for the audience that genuinely needs a wire.
+
+### In-process page feeds
+
+| Page | Seam | Producer |
+|---|---|---|
+| every page deriving from `DashboardPageBase` | `IDashboardSnapshotFeed.WatchAsync` | `DashboardSnapshotFeed` (singleton) multicasting one `IDashboardSnapshotService.WatchSnapshotsAsync` enumeration |
+| `SessionDetailsPage` | `IDashboardSessionEventSubscriber.Subscribe(sessionId)` | `DashboardEventBroadcaster` — the same singleton the session mirror publishes to, registered behind both interfaces |
+| `AlarmsPage` | `IGatewayAlarmService.StreamAsync` | the central alarm monitor, **provider status only**; the alarm rows still come from the 3 s `QueryAlarmsAsync` poll |
+
+The snapshot feed multicasts rather than handing each page its own enumeration:
+`WatchSnapshotsAsync` is not multicast on its own — each enumeration owns a timer
+and builds its own snapshot per tick — so a subscription per page would multiply
+the snapshot cost by the number of open pages. Each subscriber reads through a
+capacity-1 drop-oldest channel, so a circuit that renders slowly skips snapshots
+instead of buffering without bound or stalling the pump.
+
+`DashboardPageBase` seeds `Snapshot` synchronously from
+`IDashboardSnapshotService.GetSnapshot()` in `OnInitializedAsync` so the first
+render is non-empty, then calls `InvokeAsync(StateHasChanged)` for every snapshot
+the feed yields. On dispose it cancels the watch and waits at most **5 seconds**
+for the loop to drain, logging a warning on timeout. The bound is deliberate: the
+loop marshals renders through the renderer's dispatcher and disposal can run on
+that same dispatcher, so an unconditional wait would hang on a wedged dispatcher.
+The accepted cost is that an abandoned loop still holds its feed subscription — the
+feed's idle gate stays open until it unwinds — and the warning is the operator's
+only signal that a circuit teardown wedged.
+
+`SessionDetailsPage` subscribes for the current session id and renders the most
+recent N events (default 50) in a "Recent events" table. Its pump drains everything
+queued and renders once per batch rather than once per event, and it re-checks
+**inside the renderer dispatch** — where the subscription field is written, making
+the check an unsynchronized read of dispatcher-owned state — that the batch's
+subscription is still the live one. A batch read before a session switch would
+otherwise render the previous session's events under the new session's heading.
+Detaching cancels the pump, disposes the subscription (which releases the viewer
+registration and completes the channel, so the pump has an exit even if
+cancellation is missed), then drains under its own timeout.
+
+### SignalR hubs (remote clients)
+
+Updates for out-of-process clients flow over three SignalR hubs, all guarded by the
 `MxGateway.Dashboard.HubClients` policy (cookie OR `MxGateway.Dashboard.HubToken`
 bearer). Each hub class is `[Authorize(Policy = HubClientsPolicy)]`.
 
@@ -167,46 +224,68 @@ bearer). Each hub class is `[Authorize(Policy = HubClientsPolicy)]`.
 |---|---|---|---|---|
 | `DashboardSnapshotHub` | `/hubs/snapshot` | `DashboardSnapshotPublisher` (BackgroundService consuming `IDashboardSnapshotService.WatchSnapshotsAsync`) | `DashboardSnapshot` | Sent to all connected clients on every snapshot tick, but only while at least one client is connected (see "Idle gating" below); new connections receive the current snapshot synchronously in `OnConnectedAsync`. |
 | `AlarmsHub` | `/hubs/alarms` | `AlarmsHubPublisher` (BackgroundService consuming `IGatewayAlarmService.StreamAsync(filter: null)`) | `AlarmFeedMessage` (`active_alarm` / `snapshot_complete` / `transition`) | Connected clients auto-join `__alarms__`; all clients receive every message. Publisher auto-reconnects every 5s on stream faults. |
-| `EventsHub` | `/hubs/events` | `DashboardEventBroadcaster` invoked by each session's internal dashboard-mirror subscriber on its `SessionEventDistributor` (registered when the session becomes Ready) | `MxEvent` | Clients call `SubscribeSession(sessionId)` to join `session:{id}`, which also registers them in `EventsHubViewerRegistry` — the mirror is gated on that registry (see "Mirror gating" below). The dashboard is a first-class distributor subscriber, so it receives the session's events whether or not a gRPC client is streaming. It sees RAW session events — not the per-gRPC-subscriber `AfterWorkerSequence` filtering that `EventStreamService` applies at its own boundary — because the dashboard is a separate LDAP-authenticated monitoring view meant to show the session's full event activity. Tag values are stripped from the mirrored `MxEvent` copy by `DashboardEventBroadcaster` when `Dashboard:ShowTagValues` is false (the default) — event metadata (tag reference, quality, status, timestamps) still renders, but the value fields are blanked, so no value leaks through this seam. The per-session hub ACL that would scope a Viewer to specific sessions is still outstanding (SEC-25 / remediation roadmap item 12); the value redaction is the near-term hardening that closes the value-leak seam independently of that ACL. |
+| `EventsHub` | `/hubs/events` | `DashboardEventBroadcaster` invoked by each session's internal dashboard-mirror subscriber on its `SessionEventDistributor` (registered when the session becomes Ready) | `MxEvent` | Clients call `SubscribeSession(sessionId)` to join `session:{id}`, which also registers them in `EventsHubViewerRegistry` — the mirror is gated on that registry, which counts hub and in-process viewers alike (see "Mirror gating" below). The dashboard is a first-class distributor subscriber, so it receives the session's events whether or not a gRPC client is streaming. It sees RAW session events — not the per-gRPC-subscriber `AfterWorkerSequence` filtering that `EventStreamService` applies at its own boundary — because the dashboard is a separate LDAP-authenticated monitoring view meant to show the session's full event activity. Tag values are stripped from the mirrored `MxEvent` copy by `DashboardEventBroadcaster` when `Dashboard:ShowTagValues` is false (the default) — event metadata (tag reference, quality, status, timestamps) still renders, but the value fields are blanked, so no value leaks through this seam. The per-session ACL that would scope a Viewer to specific sessions is still outstanding for this seam and the in-process one alike (SEC-25 / remediation roadmap item 12); the value redaction is the near-term hardening that closes the value-leak seam independently of that ACL. |
 
-`DashboardPageBase` opens a `DashboardSnapshotHub` connection via the connection
+### Default cadences
-factory in `OnInitializedAsync`, seeds `Snapshot` synchronously from
-`IDashboardSnapshotService.GetSnapshot()` so the first render is non-empty, and
-calls `InvokeAsync(StateHasChanged)` on every `SnapshotUpdated` push. SignalR's
-`WithAutomaticReconnect` handles transient disconnects.
 
-`SessionDetailsPage` additionally opens an `EventsHub` connection for the
+Both seams consume the same producing services, so they share these cadences:
-current session id and renders the most recent N events (default 50) in a
-"Recent events" table with a live/offline connection pill.
-
-Default cadences:
 
 - snapshot service produces one snapshot per
   `MxGateway:Dashboard:SnapshotIntervalMilliseconds` (default 1s);
 - alarm publisher emits on each transition observed by the central monitor;
 - event publisher emits per event fanned by the session's `SessionEventDistributor`
-  to its internal dashboard-mirror subscriber (independent of any gRPC `StreamEvents`).
+  to its internal dashboard-mirror subscriber (independent of any gRPC `StreamEvents`);
+- the alarms page's provider-status badge resubscribes one second after its
+  `IGatewayAlarmService.StreamAsync` enumeration ends — the monitor completes a
+  subscriber's stream when it falls behind and again when it restarts, both
+  recoverable by resubscribing — and holds its last value in between. The page's
+  alarm rows are independent of that stream and refresh on the 3 s poll.
 
 ### Idle gating and snapshot cost
 
 A snapshot is not free: each one takes a session-registry snapshot and sorts it,
 copies the metrics dictionaries under the global metrics lock, and projects
 sessions, workers, faults, and the Galaxy summary. Without gating that work ran
-once a second for the life of the process even when no browser was connected.
+once a second for the life of the process even when nothing was watching.
 
-`DashboardSnapshotHub` counts live connections into the singleton
+Gating is two-tier, because the two seams have independent audiences and each must
+be able to reach zero on its own.
+
+**Hub tier.** `DashboardSnapshotHub` counts live connections into the singleton
 `DashboardSnapshotHubConnectionCounter` (`OnConnectedAsync` / `OnDisconnectedAsync`,
 clamped at zero). `DashboardSnapshotPublisher` reads that count before advancing the
 snapshot enumerator: while it is zero the publisher does not call `MoveNextAsync` at
 all, so the producing iterator stays suspended at its `yield` and builds nothing —
 the gate removes the snapshot *build*, not just the broadcast. The publisher
-re-checks once a second while idle, so the first viewer to connect resumes the tick
+re-checks once a second while idle, so the first client to connect resumes the tick
-within roughly one snapshot interval. That viewer does not wait for it either:
+within roughly one snapshot interval, and `OnConnectedAsync` pushes the current
-`DashboardPageBase` seeds its first render synchronously from
+snapshot to that connection immediately. Now that no in-repo page connects to the
-`IDashboardSnapshotService.GetSnapshot()`, and `OnConnectedAsync` pushes a snapshot
+hub, this tier stays idle unless a remote client connects.
-to the new connection immediately.
 
-Two per-tick costs inside the snapshot itself are bounded independently of the gate:
+**In-process tier.** `DashboardSnapshotFeed` gates its own pump on its subscriber
+list: the first subscriber starts the pump, the last one leaving cancels it and
+awaits it, so a gateway with no page open runs no timer and builds no snapshots on
+this seam either. Successive pumps are chained through the previous pump's task, so
+an unsubscribe immediately followed by a resubscribe restarts a fresh pump without
+ever running two enumerations at once. A page does not wait for the pump's first
+tick — `DashboardPageBase` seeds its first render synchronously from
+`IDashboardSnapshotService.GetSnapshot()`.
+
+That gate is generation-scoped rather than a plain subscriber count. Each pump owns
+a generation, each subscriber is tagged with the generation it joined under, a pump
+ends its generation the instant its source faults or completes — before the possibly
+slow enumerator disposal — and a dying pump only ever detaches its own generation's
+subscribers. Two races motivate the extra state:
+
+- a subscriber arriving mid-teardown must start a fresh generation rather than
+  attach to a pump that is about to detach everybody and leave nobody watching;
+- an unsubscribe must compare its own generation against the live one before it
+  cancels anything. Subscribers of an ending generation linger in the list until
+  that pump's reset runs, so counting the whole list would let them hold the idle
+  gate open, and cancelling on their behalf would stop a live pump that other
+  viewers depend on.
+
+Two per-tick costs inside the snapshot itself are bounded independently of either gate:
 
 - the effective configuration (`EffectiveGatewayConfiguration`) is built once and
   cached. It is a projection of `IOptions<GatewayOptions>`, which the gateway binds
@@ -220,26 +299,43 @@ Two per-tick costs inside the snapshot itself are bounded independently of the g
   retried on the next tick and the previous summaries stay on screen.
 
 Avoid pushing every MXAccess data-change event into a wider broadcast group.
-The current design routes events strictly through `session:{id}` groups; the
+Events are routed strictly per session (`session:{id}` groups on the hub,
-snapshot hub continues to carry aggregate event counters and rates.
+per-session subscriber lists in process); the snapshot seams continue to carry
+aggregate event counters and rates.
 
 ### Mirror gating
 
 Each session's dashboard-mirror subscriber calls
 `DashboardEventBroadcaster.Publish` for every event the session produces,
-independently of whether any browser is watching that session. SignalR does not
+independently of whether anything is watching that session. `Publish` returns
-expose group membership, so the broadcaster cannot ask whether `session:{id}` is
+immediately when `EventsHubViewerRegistry.HasViewers(sessionId)` is false,
-empty. `EventsHubViewerRegistry` (singleton) supplies that answer: `EventsHub`
+**before** the redaction clone. That matters because redaction is on by default
-mirrors its own `AddToGroup` / `RemoveFromGroup` calls into it, and
-`OnDisconnectedAsync` releases every subscription a dropped connection held — the
-only reliable signal for a browser tab that closes without unsubscribing.
-`Publish` returns immediately when `HasViewers(sessionId)` is false, **before**
-the redaction clone. That matters because redaction is on by default
 (`Dashboard:ShowTagValues` false), so the unwatched steady state — nobody on any
 session-details page — previously paid a deep protobuf clone plus a send to an
 empty group for every event of every session. Behaviour for a watched session is
 unchanged.
 
+The registry counts both audiences, which is what lets one gate serve both seams.
+`EventsHub` mirrors its own `AddToGroup` / `RemoveFromGroup` calls into it —
+SignalR does not expose group membership, so the broadcaster cannot ask whether
+`session:{id}` is empty — and `OnDisconnectedAsync` releases every subscription a
+dropped connection held, the only reliable signal for a browser tab that closes
+without unsubscribing. An in-process subscription registers the same way under a
+synthetic `inproc-`-prefixed connection id, which cannot collide with a SignalR
+connection id and makes the origin obvious in a debugger; disposing it removes the
+viewer and releases the synthetic connection, because that id never reconnects and
+nothing else would ever release its per-connection entry. Both paths use the same
+ordering — register before becoming a delivery target, deregister after ceasing to
+be one — so the widest a race window opens is a redaction clone that reaches
+nobody, never a dropped event that was owed to a live viewer.
+
+Redaction happens once per event, not once per audience: `Publish` produces a
+single redacted clone and hands that same instance to the in-process subscribers
+and to the hub group. In-process delivery runs first and synchronously — it cannot
+throw, and it must not be skipped by the guard clause around the hub send — into
+per-subscriber bounded drop-oldest channels, so a page that falls behind loses its
+oldest queued events rather than blocking the session's event pipeline.
+
 The mirror subscriber itself is still registered on the `SessionEventDistributor`
 for the session's whole lifetime; only the per-event work is gated. Starting and
 stopping the mirror lease lazily with the first and last viewer was considered
@@ -369,8 +465,9 @@ panel. The panel shows each subscribed tag's live value, MXAccess data type,
 quality and source timestamp, refreshed every two seconds. The subscription
 panel is the explicit opt-in tag-value surface: it always shows values
 regardless of `Dashboard:ShowTagValues`, which governs the diagnostic
-session/worker views and the per-session `EventsHub` mirror (values are
+session/worker views and the per-session event mirror — both its hub and
-redacted from the mirrored events when the flag is false).
+in-process audiences (values are redacted from the mirrored events when the flag
+is false).
 
 ### Alarms page
 
@@ -380,7 +477,11 @@ defaults to showing unacknowledged `Active` alarms; filters add acknowledged
 alarms and narrow by area, severity range, and a reference/source/description
 text search. Cleared alarms are not retained — the gateway holds no
 alarm-history store, so the page reflects only the live active set. The page is
-read-only; it does not acknowledge alarms. If `MxGateway:Alarms:Enabled` is
+read-only; it does not acknowledge alarms. A provider-status badge tracks the
+central monitor's health from `IGatewayAlarmService.StreamAsync` in process — the
+alarm service is already a multi-subscriber fan-out, so the badge needs no SignalR
+client, no loopback socket, and no hub token — while the alarm rows themselves
+still come from the three-second poll. If `MxGateway:Alarms:Enabled` is
 false the central monitor never starts, and the page says so instead of showing
 an empty list with no explanation.
 
@@ -536,6 +637,14 @@ Three authorization policies are registered:
   cookie OR a `MxGateway.Dashboard.HubToken` bearer (used by WebSocket upgrades
   where the cookie can't be forwarded).
 
+The in-process page feeds carry no authentication of their own, and need none:
+`MapRazorComponents<App>()` applies `RequireAuthorization(ViewerPolicy)` to the
+component endpoints, so a page can only run inside a circuit whose principal is
+already an authorized Viewer. The hub-token flow below therefore covers only the
+remote hub surface. Neither seam scopes a Viewer to particular sessions — SEC-25
+(the per-session ACL) is outstanding for both, and the mirror's value redaction
+remains the near-term mitigation, unchanged by the move in-process.
+
 Two environmental bypasses still apply, both scoped to **read-only** access:
 `MxGateway:Authentication:Mode = Disabled` and `MxGateway:Dashboard:AllowAnonymousLocalhost`
 (default `true`, loopback only) each satisfy a requirement that includes the Viewer
@@ -580,12 +689,15 @@ surface is affected. Never enable in production.
 
 ### Hub bearer flow
 
+This flow serves remote hub clients only; in-process pages are authorized by the
+component endpoint's `ViewerPolicy` and never mint a token.
+
 SignalR connections cannot reuse the `__Host-` cookie when the JS client
 upgrades to WebSocket — the cookie's `SameSite=Strict; Path=/` keeps it from
 being forwarded by the browser's WebSocket layer in some edge cases. The
 dashboard mints short-lived bearer tokens for the connection:
 
-1. The cookie-authenticated Blazor page calls `GET /hubs/token`
+1. The cookie-authenticated client calls `GET /hubs/token`
    (gated by `ViewerPolicy`, cookie-only).
 2. `HubTokenService.Issue(user)` serializes the user's name, NameIdentifier,
    and role claims to JSON, encrypts with the ASP.NET Core data-protection
@@ -605,7 +717,9 @@ dashboard mints short-lived bearer tokens for the connection:
 
 `DashboardHubConnectionFactory` (scoped to the Blazor circuit) wraps the
 HubConnectionBuilder and supplies a fresh token via `AccessTokenProvider` on
-every (re)connect, so the short 5-minute lifetime is transparent to clients.
+every (re)connect, so the short 5-minute lifetime is transparent to whoever uses
+it. It remains registered, but no in-repo page opens a hub connection any more;
+external clients implement the equivalent refresh themselves.
 
 Caveat — logout does not revoke outstanding tokens. Logout clears the dashboard
 cookie, but hub bearer tokens are self-contained, data-protection-encrypted, and
@@ -700,8 +814,14 @@ Integration tests should verify:
 - a user in a Viewer-mapped LDAP group can render every page but cannot
   invoke the Admin-only management actions,
 - a user with no mapped LDAP group cannot sign in at all,
-- live snapshot updates when a fake session changes state are delivered
+- live snapshot updates when a fake session changes state reach a page through
-  via the `/hubs/snapshot` push, not by polling.
+  the in-process `IDashboardSnapshotFeed` and reach a remote client through the
+  `/hubs/snapshot` push — neither by polling;
+- the snapshot feed's idle gate: no subscribers means no pump, the last
+  subscriber of the live generation stops it, and a subscriber arriving
+  mid-teardown gets a fresh generation rather than a dead one;
+- the event mirror's viewer gate counts in-process subscriptions as well as hub
+  connections, and a disposed in-process subscription releases its viewer count.
 
 ## Initial Implementation Slice