From 2c963b723013b7148cd9ad69df9f92e2eaea8676 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 22 Oct 2026 06:27:14 -0400 Subject: [PATCH] docs(dashboard): describe the pages as shipped and the bounded hub/feed behaviour Rewrite the Pages 'Show:' lists to what each page actually renders (no per-session subscriber/pending-command/queue counters, no worker memory/CPU, no per-family command counters, counters not rates). Document the per-session EventsHub pump, alarm-hub send bound and AlarmFeedReset marker, configuration sent once per snapshot-hub connection, health-derived gateway status, Browse search debounce/index, bounded disposal, 30s circuit retention with connection-gated polling, per-viewer advise-set share, and AlarmsPage memo. --- docs/GatewayDashboardDesign.md | 224 ++++++++++++++++++++++----------- 1 file changed, 151 insertions(+), 73 deletions(-) diff --git a/docs/GatewayDashboardDesign.md b/docs/GatewayDashboardDesign.md index be2544f..1c01074 100644 --- a/docs/GatewayDashboardDesign.md +++ b/docs/GatewayDashboardDesign.md @@ -273,9 +273,14 @@ bearer). Each hub class is `[Authorize(Policy = HubClientsPolicy)]`. | Hub | Path | Producer | Payload | Routing | |---|---|---|---|---| -| `DashboardSnapshotHub` | `/hubs/snapshot` | `DashboardSnapshotPublisher` (BackgroundService consuming `IDashboardSnapshotService.WatchSnapshotsAsync`) | `DashboardSnapshot` | Sent to all connected clients on every snapshot tick, but only while at least one client is connected (see "Idle gating" below); new connections receive the current snapshot synchronously in `OnConnectedAsync`. | -| `AlarmsHub` | `/hubs/alarms` | `AlarmsHubPublisher` (BackgroundService consuming `IGatewayAlarmService.StreamAsync(filter: null)`) | `AlarmFeedMessage` (`active_alarm` / `snapshot_complete` / `transition` / `provider_status` / `snapshot_status`) | Guarded by `MxGateway.Dashboard.Alarms`, not `HubClients`: a real authenticated dashboard role is required, so the `Authentication:Mode=Disabled` and anonymous-localhost bypasses are excluded (SEC-40). Connected clients auto-join `__alarms__`; all admitted clients receive every message — the feed is galaxy-wide and carries no session provenance, so no per-caller scoping applies (see "Alarm-feed visibility"). Publisher auto-reconnects every 5s on stream faults. The alarm value fields (`current_value` / `limit_value`) are stripped from a deep-cloned copy of the message when `Dashboard:ShowTagValues` is false (the default) — on both value-bearing payload arms, so neither a live transition nor a snapshot record leaks a process value; every other field still renders. The source message is never mutated: it fans out from the same feed to gRPC `StreamAlarms` subscribers, which this dashboard-display flag does not govern. Arms carrying no value are forwarded as-is, uncloned. | -| `EventsHub` | `/hubs/events` | `DashboardEventBroadcaster` invoked by each session's internal dashboard-mirror subscriber on its `SessionEventDistributor` (registered when the session becomes Ready) | `MxEvent` | Clients call `SubscribeSession(sessionId)` to join `session:{id}`, which also registers them in `EventsHubViewerRegistry` — the mirror is gated on that registry, which counts hub and in-process viewers alike (see "Mirror gating" below). The dashboard is a first-class distributor subscriber, so it receives the session's events whether or not a gRPC client is streaming. It sees RAW session events — not the per-gRPC-subscriber `AfterWorkerSequence` filtering that `EventStreamService` applies at its own boundary — because the dashboard is a separate LDAP-authenticated monitoring view meant to show the session's full event activity. Tag values are stripped from the mirrored `MxEvent` copy by `DashboardEventBroadcaster` when `Dashboard:ShowTagValues` is false (the default) — event metadata (tag reference, quality, status, timestamps) still renders, but the value fields are blanked, so no value leaks through this seam. `SubscribeSession` is gated by `IDashboardSessionAcl` (SEC-25 / TST-15): a denied caller gets a `HubException`, is not joined to the group, and is not registered as a viewer, so the mirror stays off for a session nobody is legitimately watching. The same ACL gates the in-process seam the session-details page uses, so neither path is the weaker one. Value redaction remains an independent layer — it bounds what a *permitted* subscriber sees. | +| `DashboardSnapshotHub` | `/hubs/snapshot` | `DashboardSnapshotPublisher` (BackgroundService consuming `IDashboardSnapshotService.WatchSnapshotsAsync`) | `SnapshotUpdated`: `DashboardSnapshot` **without** `configuration`; `ConfigurationUpdated`: `EffectiveGatewayConfiguration` | Ticks go to all connected clients on every snapshot interval, but only while at least one client is connected (see "Idle gating" below). The effective configuration is startup-static and the largest part of the record, so it is `[JsonIgnore]`d on `DashboardSnapshot` and sent once per connection instead: `OnConnectedAsync` pushes `ConfigurationUpdated` and then the current `SnapshotUpdated`, and the hub method `GetConfiguration()` returns it on demand. Each tick broadcast is bounded to 1 s; a send that misses the bound is abandoned, so one slow client cannot delay the next tick for everyone. | +| `AlarmsHub` | `/hubs/alarms` | `AlarmsHubPublisher` (BackgroundService consuming `IGatewayAlarmService.StreamAsync(filter: null)`) | `AlarmFeedMessage` (`active_alarm` / `snapshot_complete` / `transition` / `provider_status` / `snapshot_status`) | Guarded by `MxGateway.Dashboard.Alarms`, not `HubClients`: a real authenticated dashboard role is required, so the `Authentication:Mode=Disabled` and anonymous-localhost bypasses are excluded (SEC-40). Connected clients auto-join `__alarms__`; all admitted clients receive every message — the feed is galaxy-wide and carries no session provenance, so no per-caller scoping applies (see "Alarm-feed visibility"). Publisher auto-reconnects every 5s on stream faults. Every subscription to the monitor — the first and each resubscribe — is preceded by an `AlarmFeedReset` frame (`{ generation }`, monotonic per process): the monitor primes each new subscription with a full `active_alarm` snapshot ending in `snapshot_complete`, so a client must discard its accumulated alarm state on the marker and rebuild from that snapshot rather than double-counting it. Each broadcast is bounded to 2 s and abandoned on timeout: awaiting it unbounded let one slow client stall the drain until the monitor dropped the publisher as a slow subscriber, and the resubscribe then replayed the whole snapshot to every client. The alarm value fields (`current_value` / `limit_value`) are stripped from a deep-cloned copy of the message when `Dashboard:ShowTagValues` is false (the default) — on both value-bearing payload arms, so neither a live transition nor a snapshot record leaks a process value; every other field still renders. The source message is never mutated: it fans out from the same feed to gRPC `StreamAlarms` subscribers, which this dashboard-display flag does not govern. Arms carrying no value are forwarded as-is, uncloned. | +| `EventsHub` | `/hubs/events` | `DashboardEventBroadcaster` invoked by each session's internal dashboard-mirror subscriber on its `SessionEventDistributor` (registered when the session becomes Ready) | `MxEvent` | Clients call `SubscribeSession(sessionId)` to join `session:{id}`, which also registers them in `EventsHubViewerRegistry`. Delivery to the group goes through one bounded per-session pump (see "Mirror gating"), so a slow viewer loses its oldest events instead of accumulating pending sends — the mirror is gated on that registry, which counts hub and in-process viewers alike (see "Mirror gating" below). The dashboard is a first-class distributor subscriber, so it receives the session's events whether or not a gRPC client is streaming. It sees RAW session events — not the per-gRPC-subscriber `AfterWorkerSequence` filtering that `EventStreamService` applies at its own boundary — because the dashboard is a separate LDAP-authenticated monitoring view meant to show the session's full event activity. Tag values are stripped from the mirrored `MxEvent` copy by `DashboardEventBroadcaster` when `Dashboard:ShowTagValues` is false (the default) — event metadata (tag reference, quality, status, timestamps) still renders, but the value fields are blanked, so no value leaks through this seam. `SubscribeSession` is gated by `IDashboardSessionAcl` (SEC-25 / TST-15): a denied caller gets a `HubException`, is not joined to the group, and is not registered as a viewer, so the mirror stays off for a session nobody is legitimately watching. The same ACL gates the in-process seam the session-details page uses, so neither path is the weaker one. Value redaction remains an independent layer — it bounds what a *permitted* subscriber sees. | + +No hub client ships in this repository: the server-rendered pages read the +in-process seams above, and the hubs serve external consumers. Those consumers +must honour `AlarmFeedReset` and keep the configuration from +`ConfigurationUpdated`, since ticks no longer carry it. ### Default cadences @@ -343,12 +348,17 @@ subscribers. Two races motivate the extra state: gate open, and cancelling on their behalf would stop a live pump that other viewers depend on. -Two per-tick costs inside the snapshot itself are bounded independently of either gate: +Three per-tick costs inside the snapshot itself are bounded independently of either gate: - the effective configuration (`EffectiveGatewayConfiguration`) is built once and cached. It is a projection of `IOptions`, which the gateway binds at startup and never reloads, so rebuilding the whole option tree every tick - produced an identical object; + produced an identical object. For the same reason it is not serialized on the + `/hubs/snapshot` tick (see the hub table above); +- the gateway status is one `HealthCheckService.CheckHealthAsync` evaluation per + tick, bounded to 2 s and behind a zero-wait gate so concurrent enumerations (the + hub publisher and the page feed) do not both evaluate; `GetSnapshot()` reads the + cached result; - the API key summaries are refreshed at most once every 15 seconds (`ApiKeySummaryRefreshInterval`) instead of on every tick. The list is a SQLite read whose content changes only when an operator creates, rotates, or revokes a @@ -359,7 +369,7 @@ Two per-tick costs inside the snapshot itself are bounded independently of eithe Avoid pushing every MXAccess data-change event into a wider broadcast group. Events are routed strictly per session (`session:{id}` groups on the hub, per-session subscriber lists in process); the snapshot seams continue to carry -aggregate event counters and rates. +aggregate event counters. ### Mirror gating @@ -394,10 +404,21 @@ group. With `ShowTagValues` true there is no clone at all — the original `MxEv instance is handed to both audiences — so the "one clone per event" cost holds only in the redacting configuration, and in the value-showing one both audiences share a reference to the session pipeline's own event object. In-process delivery runs first -and synchronously — it cannot -throw, and it must not be skipped by the guard clause around the hub send — into -per-subscriber bounded drop-oldest channels, so a page that falls behind loses its -oldest queued events rather than blocking the session's event pipeline. +and synchronously — it cannot throw — into per-subscriber bounded drop-oldest +channels, so a page that falls behind loses its oldest queued events rather than +blocking the session's event pipeline. + +Hub delivery has the same shape. `EventsHubViewerRegistry` counts remote hub +connections separately from `inproc-` subscribers (`HasHubViewers`), and while a +session has at least one, `Publish` writes the event into that session's +`HubPump`: a 256-deep drop-oldest channel drained by a single task that sends one +event at a time to `session:{id}`, each send bounded to 5 s. A session watched only +in process starts no pump. The pump starts on the first event published while a +hub viewer is present and is torn down when the registry raises +`HubViewersReleased` for the session's last hub viewer (unsubscribe or disconnect). +Before this, `Publish` fired a group `SendAsync` per event and returned; a slow +remote viewer made each send stay pending in the SignalR lifetime manager, and the +pending sends — each holding a cloned event — grew without bound. The mirror subscriber itself is still registered on the `SessionEventDistributor` for the session's whole lifetime; only the per-event work is gated. Starting and @@ -411,55 +432,67 @@ clone and send this gate already removes. ### Dashboard home -Show top-level status: +The Overview page (`/`) shows, from the snapshot: -- gateway status, -- gateway version, -- uptime, -- open sessions, -- workers running, -- sessions faulted, -- command rate, -- command failure count, -- event rate, -- event queue depth, -- worker restart/kill count. +- gateway status badge — the aggregate status of the registered ASP.NET Core + health checks (`Healthy` / `Degraded` / `Unhealthy`), evaluated once per snapshot + tick with a 2 s bound and cached for the synchronous `GetSnapshot()` seed path. It + reads `Unknown` before the first evaluation, when an evaluation throws or times + out, and when no `HealthCheckService` is registered — never a hardcoded `Healthy` + (SEC-27); +- the snapshot's generation time; +- metric cards: uptime (with the gateway version as detail), open sessions, workers + running, worker event queue depth, failed commands, events received, faults, and + queue overflows — all cumulative counters or point-in-time gauges, not rates; +- a Galaxy Repository section: cache status, last deploy time, object count (with + area count), attribute, historized-attribute and alarm-attribute counts, and the + last refresh time; +- the recent faults list (`Dashboard:RecentFaultLimit`). -Use Bootstrap cards for individual metric summaries. Keep the layout compact -and operational. +There is no per-family command or event rate, no faulted-session counter, and no +worker restart/kill card on this page; `mxgateway.workers.killed` and +`mxgateway.workers.exited` are in the snapshot's metric list but not rendered. +Metric summaries use Bootstrap cards; the layout stays compact and operational. ### Sessions page -Show active and recent sessions in a table: +Shows the most recent sessions (`Dashboard:RecentSessionLimit`, newest first) in a +table with these columns: -- session id, -- client identity or API key display name, -- state, +- session id (links to the session details page), +- session state, +- client identity (redacted), - backend, -- worker process id, +- worker process id and worker state, +- events received for the session (the `mxgateway.events.received` per-session + counter), - open time, - last client activity, - last worker heartbeat, -- active event subscribers, -- pending commands, -- event queue depth, -- last fault summary. +- last fault summary (abbreviated, full text on hover), +- Close / Kill actions for Admin principals. -Rows should link to session details. +Event subscriber counts, pending command counts, and per-session queue depth are +not tracked per session and are not shown. ### Session details page -Show: +Shows: -- session metadata, -- worker metadata, -- command counters by method, -- event counters by family, -- active server handles and item counts if gateway shadow state has them, -- latest faults, -- last heartbeat payload, +- a session section: state badge, backend, client identity, client session name, + client correlation id (all redacted), open time, last activity, lease expiry, + events received, and last fault; +- a worker section: process id, worker state, and last heartbeat time; +- a "Recent events" table — the newest 50 events from the in-process mirror, with + worker sequence, family, server handle, item handle, and status (no values), plus + a live/offline pill; gated by the session ACL, so a caller who may not observe the + session sees "Not authorized" instead; - admin Close session / Kill worker controls (Admin role only). +Per-method command counters, per-family event counters for the session, server +handle / item counts, and the heartbeat payload are not shown — the gateway keeps +no per-session shadow state for them. + The Sessions list, the Workers list, and this details page all render the same admin controls when the signed-in principal carries the `Admin` role; viewers and the localhost-anonymous bypass see no action affordances and the server @@ -484,33 +517,31 @@ management audit path. ### Workers page -Show: +Shows one row per worker that is not closed, with: - worker process id, -- session id, -- executable path/version, -- state, -- startup duration, -- memory and CPU if available, -- last heartbeat, -- current command correlation id, -- pending command count, -- event queue depth, -- restart/kill reason if terminal. +- worker state, +- owning session id (links to the session details page), +- last heartbeat time, +- last fault summary (the owning session's final fault), +- a Kill worker action for Admin principals. + +Executable path and version, startup duration, memory/CPU, in-flight command +correlation, pending command count, and per-worker queue depth are not collected +and are not shown; the worker executable path is on the Settings page. ### Events page -Show aggregate event diagnostics: +Shows aggregate event diagnostics from the snapshot: -- event rate by session, -- event rate by event family, -- total events since start, +- events received since start, +- worker event queue depth and gRPC stream queue depth (gauges), - queue overflow count, -- stream disconnect count, -- recent terminal faults. +- gRPC stream disconnect count, +- an "Event Families" table: cumulative events received per event family. -Do not display full tag values by default. If value display is later added, make -it opt-in and redacted. +The figures are counters since start, not rates; per-session event counts are on +the Sessions page. No tag values are shown here. ### Browse page @@ -537,6 +568,33 @@ at the service, not in the page: one decision point, and the value never enters the render tree. Set the flag true to make the panel the explicit opt-in tag-value surface it was originally designed as. +The filter box is debounced: keystrokes within 150 ms coalesce into one search, +which runs on the thread pool with a cancellation token that the next keystroke +cancels, and only the latest query may publish its results. The search reads +`DashboardBrowseSearchIndex`, a singleton holding every attribute's full reference +and name lowercased once, keyed on the Galaxy cache `Sequence` and rebuilt only when +a deploy moves it. The previous filter rescanned the whole cached Galaxy with +case-insensitive comparisons, synchronously on the render dispatcher, once per +keystroke per circuit. + +The page's two background loops — the 2 s live-read poll and the Galaxy +deploy-notification subscription — handle faults inside the loop and retry; only +cancellation ends them. A poll fault surfaces in the "Live read failed" banner; a +notifier fault surfaces as a "redeploy notifications failed (retrying)" warning and +resubscribes after 5 s; the tree rebuild after a redeploy runs on the renderer +dispatcher. `DisposeAsync` cancels the loops and any in-flight search, drains them +concurrently with the same 5 s bound `DashboardPageBase` uses, and disposes its +cancellation sources in a `finally`. + +A closed or backgrounded tab leaves its circuit alive, disconnected, for +`DisconnectedCircuitRetentionPeriod`, which the dashboard sets to **30 seconds** +(`DashboardServiceCollectionExtensions.DisconnectedCircuitRetention`) instead of +the framework's three minutes. Within that window the poll does not read: a scoped +`DashboardCircuitConnectionTracker`, registered as the circuit's `CircuitHandler`, +reports the browser connection down and the poll skips its ticks until it comes +back. Every tick it does run is a `ReadBulk` against the shared dashboard worker, +so a closed tab previously kept the worker busy for three minutes. + ### Alarms page `/dashboard/alarms` lists the alarms the gateway's central alarm monitor @@ -552,7 +610,10 @@ client, no loopback socket, and no hub token — and the same subscription carri `snapshot_status` frame behind the truncated-snapshot banner ("Alarm snapshot may be incomplete"), so a capped provider fetch is caveated the moment the monitor decides it. The alarm rows themselves still come from the three-second poll, which also -re-asserts the truncation verdict as the reconcile baseline. If `MxGateway:Alarms:Enabled` is +re-asserts the truncation verdict as the reconcile baseline. The filtered rows and +the area list are memoized in `DashboardAlarmView` on the poll's alarm generation and +the filter inputs, so the "Showing" card and the table share one filter-and-sort per +change rather than recomputing it several times per render. If `MxGateway:Alarms:Enabled` is false the central monitor never starts, and the page says so instead of showing an empty list with no explanation. @@ -599,6 +660,19 @@ overflow against the oversized set and evicts the whole excess in one pass (a subscribes nothing new evicts nothing, but neither can it grow the set. A browse page requests far fewer tags than the cap, so in practice the set settles at 256. +The 256 is shared between viewers rather than one global LRU. Each Browse page +reads under its own viewer id, every advised tag records the viewer that read it +last, and each active viewer (one that read within the last 30 s) owns a share of +`256 / active viewers`, floored at 32. When a read needs room, eviction first takes +the least-recently-read tags of *other* viewers that are over their share — an +inactive viewer's share is zero, so a closed page's tags go first — and then the +reader's own tags that this read did not touch. A viewer within its share is never +evicted by someone else's read, so with several viewers the cap is soft: the set +can sit above 256 rather than strip a neighbour. Under the global LRU, two circuits +browsing different areas evicted each other's whole panels on every poll and +re-advised them two seconds later. With a single viewer the behaviour is exactly +the global LRU described above. + The Alarms page does **not** use the dashboard session: alarm data comes from the gateway's always-on central monitor. `QueryAlarmsAsync` reads `IGatewayAlarmService.CurrentAlarms` — the monitor's in-process cache — so the @@ -665,20 +739,24 @@ Secrets and pepper values are never logged. ### Settings page -Show read-only effective configuration: +Shows the read-only effective configuration: -- worker executable path, -- configured timeouts, -- queue capacities, -- auth mode, -- SQLite auth database path with sensitive parts redacted if needed, -- dashboard enabled state, +- authentication: mode, SQLite auth database path, pepper secret name, and whether + migrations run on startup; +- LDAP: enabled, server and port, fallback servers, transport, search base, + service account DN, service password (redacted), and the username and group + attributes; - the LDAP group mappings that decide what a signed-in user may see — `Dashboard:GroupToRole` (group → `Administrator` / `Viewer`) and - `Dashboard:GroupToTag` (group → the session tags it grants), -- `Dashboard:UntaggedSessionVisibility`, which decides who sees a session whose - owning API key carries no dashboard tags, -- protocol version. + `Dashboard:GroupToTag` (group → the session tags it grants); +- worker: executable path, required architecture, and the startup, shutdown, and + heartbeat-grace timeouts; +- sessions: default command timeout and max sessions; +- events: queue capacity and backpressure policy; +- dashboard: enabled, anonymous localhost, snapshot interval, recent fault and + session limits, `ShowTagValues`, and `Dashboard:UntaggedSessionVisibility` + (who sees a session whose owning API key carries no dashboard tags); +- the worker protocol version. Do not show API key secrets or pepper values.