docs(dashboard): describe the pages as shipped and the bounded hub/feed behaviour

Rewrite the Pages 'Show:' lists to what each page actually renders (no
per-session subscriber/pending-command/queue counters, no worker memory/CPU,
no per-family command counters, counters not rates). Document the per-session
EventsHub pump, alarm-hub send bound and AlarmFeedReset marker, configuration
sent once per snapshot-hub connection, health-derived gateway status, Browse
search debounce/index, bounded disposal, 30s circuit retention with
connection-gated polling, per-viewer advise-set share, and AlarmsPage memo.
This commit is contained in:
Joseph Doherty
2026-10-22 06:27:14 -04:00
parent db7833609d
commit 2c963b7230
+151 -73
View File
@@ -273,9 +273,14 @@ bearer). Each hub class is `[Authorize(Policy = HubClientsPolicy)]`.
| Hub | Path | Producer | Payload | Routing |
|---|---|---|---|---|
| `DashboardSnapshotHub` | `/hubs/snapshot` | `DashboardSnapshotPublisher` (BackgroundService consuming `IDashboardSnapshotService.WatchSnapshotsAsync`) | `DashboardSnapshot` | Sent to all connected clients on every snapshot tick, but only while at least one client is connected (see "Idle gating" below); new connections receive the current snapshot synchronously in `OnConnectedAsync`. |
| `AlarmsHub` | `/hubs/alarms` | `AlarmsHubPublisher` (BackgroundService consuming `IGatewayAlarmService.StreamAsync(filter: null)`) | `AlarmFeedMessage` (`active_alarm` / `snapshot_complete` / `transition` / `provider_status` / `snapshot_status`) | Guarded by `MxGateway.Dashboard.Alarms`, not `HubClients`: a real authenticated dashboard role is required, so the `Authentication:Mode=Disabled` and anonymous-localhost bypasses are excluded (SEC-40). Connected clients auto-join `__alarms__`; all admitted clients receive every message — the feed is galaxy-wide and carries no session provenance, so no per-caller scoping applies (see "Alarm-feed visibility"). Publisher auto-reconnects every 5s on stream faults. The alarm value fields (`current_value` / `limit_value`) are stripped from a deep-cloned copy of the message when `Dashboard:ShowTagValues` is false (the default) — on both value-bearing payload arms, so neither a live transition nor a snapshot record leaks a process value; every other field still renders. The source message is never mutated: it fans out from the same feed to gRPC `StreamAlarms` subscribers, which this dashboard-display flag does not govern. Arms carrying no value are forwarded as-is, uncloned. |
| `EventsHub` | `/hubs/events` | `DashboardEventBroadcaster` invoked by each session's internal dashboard-mirror subscriber on its `SessionEventDistributor` (registered when the session becomes Ready) | `MxEvent` | Clients call `SubscribeSession(sessionId)` to join `session:{id}`, which also registers them in `EventsHubViewerRegistry` — the mirror is gated on that registry, which counts hub and in-process viewers alike (see "Mirror gating" below). The dashboard is a first-class distributor subscriber, so it receives the session's events whether or not a gRPC client is streaming. It sees RAW session events — not the per-gRPC-subscriber `AfterWorkerSequence` filtering that `EventStreamService` applies at its own boundary — because the dashboard is a separate LDAP-authenticated monitoring view meant to show the session's full event activity. Tag values are stripped from the mirrored `MxEvent` copy by `DashboardEventBroadcaster` when `Dashboard:ShowTagValues` is false (the default) — event metadata (tag reference, quality, status, timestamps) still renders, but the value fields are blanked, so no value leaks through this seam. `SubscribeSession` is gated by `IDashboardSessionAcl` (SEC-25 / TST-15): a denied caller gets a `HubException`, is not joined to the group, and is not registered as a viewer, so the mirror stays off for a session nobody is legitimately watching. The same ACL gates the in-process seam the session-details page uses, so neither path is the weaker one. Value redaction remains an independent layer — it bounds what a *permitted* subscriber sees. |
| `DashboardSnapshotHub` | `/hubs/snapshot` | `DashboardSnapshotPublisher` (BackgroundService consuming `IDashboardSnapshotService.WatchSnapshotsAsync`) | `SnapshotUpdated`: `DashboardSnapshot` **without** `configuration`; `ConfigurationUpdated`: `EffectiveGatewayConfiguration` | Ticks go to all connected clients on every snapshot interval, but only while at least one client is connected (see "Idle gating" below). The effective configuration is startup-static and the largest part of the record, so it is `[JsonIgnore]`d on `DashboardSnapshot` and sent once per connection instead: `OnConnectedAsync` pushes `ConfigurationUpdated` and then the current `SnapshotUpdated`, and the hub method `GetConfiguration()` returns it on demand. Each tick broadcast is bounded to 1 s; a send that misses the bound is abandoned, so one slow client cannot delay the next tick for everyone. |
| `AlarmsHub` | `/hubs/alarms` | `AlarmsHubPublisher` (BackgroundService consuming `IGatewayAlarmService.StreamAsync(filter: null)`) | `AlarmFeedMessage` (`active_alarm` / `snapshot_complete` / `transition` / `provider_status` / `snapshot_status`) | Guarded by `MxGateway.Dashboard.Alarms`, not `HubClients`: a real authenticated dashboard role is required, so the `Authentication:Mode=Disabled` and anonymous-localhost bypasses are excluded (SEC-40). Connected clients auto-join `__alarms__`; all admitted clients receive every message — the feed is galaxy-wide and carries no session provenance, so no per-caller scoping applies (see "Alarm-feed visibility"). Publisher auto-reconnects every 5s on stream faults. Every subscription to the monitor — the first and each resubscribe — is preceded by an `AlarmFeedReset` frame (`{ generation }`, monotonic per process): the monitor primes each new subscription with a full `active_alarm` snapshot ending in `snapshot_complete`, so a client must discard its accumulated alarm state on the marker and rebuild from that snapshot rather than double-counting it. Each broadcast is bounded to 2 s and abandoned on timeout: awaiting it unbounded let one slow client stall the drain until the monitor dropped the publisher as a slow subscriber, and the resubscribe then replayed the whole snapshot to every client. The alarm value fields (`current_value` / `limit_value`) are stripped from a deep-cloned copy of the message when `Dashboard:ShowTagValues` is false (the default) — on both value-bearing payload arms, so neither a live transition nor a snapshot record leaks a process value; every other field still renders. The source message is never mutated: it fans out from the same feed to gRPC `StreamAlarms` subscribers, which this dashboard-display flag does not govern. Arms carrying no value are forwarded as-is, uncloned. |
| `EventsHub` | `/hubs/events` | `DashboardEventBroadcaster` invoked by each session's internal dashboard-mirror subscriber on its `SessionEventDistributor` (registered when the session becomes Ready) | `MxEvent` | Clients call `SubscribeSession(sessionId)` to join `session:{id}`, which also registers them in `EventsHubViewerRegistry`. Delivery to the group goes through one bounded per-session pump (see "Mirror gating"), so a slow viewer loses its oldest events instead of accumulating pending sends — the mirror is gated on that registry, which counts hub and in-process viewers alike (see "Mirror gating" below). The dashboard is a first-class distributor subscriber, so it receives the session's events whether or not a gRPC client is streaming. It sees RAW session events — not the per-gRPC-subscriber `AfterWorkerSequence` filtering that `EventStreamService` applies at its own boundary — because the dashboard is a separate LDAP-authenticated monitoring view meant to show the session's full event activity. Tag values are stripped from the mirrored `MxEvent` copy by `DashboardEventBroadcaster` when `Dashboard:ShowTagValues` is false (the default) — event metadata (tag reference, quality, status, timestamps) still renders, but the value fields are blanked, so no value leaks through this seam. `SubscribeSession` is gated by `IDashboardSessionAcl` (SEC-25 / TST-15): a denied caller gets a `HubException`, is not joined to the group, and is not registered as a viewer, so the mirror stays off for a session nobody is legitimately watching. The same ACL gates the in-process seam the session-details page uses, so neither path is the weaker one. Value redaction remains an independent layer — it bounds what a *permitted* subscriber sees. |
No hub client ships in this repository: the server-rendered pages read the
in-process seams above, and the hubs serve external consumers. Those consumers
must honour `AlarmFeedReset` and keep the configuration from
`ConfigurationUpdated`, since ticks no longer carry it.
### Default cadences
@@ -343,12 +348,17 @@ subscribers. Two races motivate the extra state:
gate open, and cancelling on their behalf would stop a live pump that other
viewers depend on.
Two per-tick costs inside the snapshot itself are bounded independently of either gate:
Three per-tick costs inside the snapshot itself are bounded independently of either gate:
- the effective configuration (`EffectiveGatewayConfiguration`) is built once and
cached. It is a projection of `IOptions<GatewayOptions>`, which the gateway binds
at startup and never reloads, so rebuilding the whole option tree every tick
produced an identical object;
produced an identical object. For the same reason it is not serialized on the
`/hubs/snapshot` tick (see the hub table above);
- the gateway status is one `HealthCheckService.CheckHealthAsync` evaluation per
tick, bounded to 2 s and behind a zero-wait gate so concurrent enumerations (the
hub publisher and the page feed) do not both evaluate; `GetSnapshot()` reads the
cached result;
- the API key summaries are refreshed at most once every 15 seconds
(`ApiKeySummaryRefreshInterval`) instead of on every tick. The list is a SQLite
read whose content changes only when an operator creates, rotates, or revokes a
@@ -359,7 +369,7 @@ Two per-tick costs inside the snapshot itself are bounded independently of eithe
Avoid pushing every MXAccess data-change event into a wider broadcast group.
Events are routed strictly per session (`session:{id}` groups on the hub,
per-session subscriber lists in process); the snapshot seams continue to carry
aggregate event counters and rates.
aggregate event counters.
### Mirror gating
@@ -394,10 +404,21 @@ group. With `ShowTagValues` true there is no clone at all — the original `MxEv
instance is handed to both audiences — so the "one clone per event" cost holds only
in the redacting configuration, and in the value-showing one both audiences share a
reference to the session pipeline's own event object. In-process delivery runs first
and synchronously — it cannot
throw, and it must not be skipped by the guard clause around the hub send — into
per-subscriber bounded drop-oldest channels, so a page that falls behind loses its
oldest queued events rather than blocking the session's event pipeline.
and synchronously — it cannot throw — into per-subscriber bounded drop-oldest
channels, so a page that falls behind loses its oldest queued events rather than
blocking the session's event pipeline.
Hub delivery has the same shape. `EventsHubViewerRegistry` counts remote hub
connections separately from `inproc-` subscribers (`HasHubViewers`), and while a
session has at least one, `Publish` writes the event into that session's
`HubPump`: a 256-deep drop-oldest channel drained by a single task that sends one
event at a time to `session:{id}`, each send bounded to 5 s. A session watched only
in process starts no pump. The pump starts on the first event published while a
hub viewer is present and is torn down when the registry raises
`HubViewersReleased` for the session's last hub viewer (unsubscribe or disconnect).
Before this, `Publish` fired a group `SendAsync` per event and returned; a slow
remote viewer made each send stay pending in the SignalR lifetime manager, and the
pending sends — each holding a cloned event — grew without bound.
The mirror subscriber itself is still registered on the `SessionEventDistributor`
for the session's whole lifetime; only the per-event work is gated. Starting and
@@ -411,55 +432,67 @@ clone and send this gate already removes.
### Dashboard home
Show top-level status:
The Overview page (`/`) shows, from the snapshot:
- gateway status,
- gateway version,
- uptime,
- open sessions,
- workers running,
- sessions faulted,
- command rate,
- command failure count,
- event rate,
- event queue depth,
- worker restart/kill count.
- gateway status badge — the aggregate status of the registered ASP.NET Core
health checks (`Healthy` / `Degraded` / `Unhealthy`), evaluated once per snapshot
tick with a 2 s bound and cached for the synchronous `GetSnapshot()` seed path. It
reads `Unknown` before the first evaluation, when an evaluation throws or times
out, and when no `HealthCheckService` is registered — never a hardcoded `Healthy`
(SEC-27);
- the snapshot's generation time;
- metric cards: uptime (with the gateway version as detail), open sessions, workers
running, worker event queue depth, failed commands, events received, faults, and
queue overflows — all cumulative counters or point-in-time gauges, not rates;
- a Galaxy Repository section: cache status, last deploy time, object count (with
area count), attribute, historized-attribute and alarm-attribute counts, and the
last refresh time;
- the recent faults list (`Dashboard:RecentFaultLimit`).
Use Bootstrap cards for individual metric summaries. Keep the layout compact
and operational.
There is no per-family command or event rate, no faulted-session counter, and no
worker restart/kill card on this page; `mxgateway.workers.killed` and
`mxgateway.workers.exited` are in the snapshot's metric list but not rendered.
Metric summaries use Bootstrap cards; the layout stays compact and operational.
### Sessions page
Show active and recent sessions in a table:
Shows the most recent sessions (`Dashboard:RecentSessionLimit`, newest first) in a
table with these columns:
- session id,
- client identity or API key display name,
- state,
- session id (links to the session details page),
- session state,
- client identity (redacted),
- backend,
- worker process id,
- worker process id and worker state,
- events received for the session (the `mxgateway.events.received` per-session
counter),
- open time,
- last client activity,
- last worker heartbeat,
- active event subscribers,
- pending commands,
- event queue depth,
- last fault summary.
- last fault summary (abbreviated, full text on hover),
- Close / Kill actions for Admin principals.
Rows should link to session details.
Event subscriber counts, pending command counts, and per-session queue depth are
not tracked per session and are not shown.
### Session details page
Show:
Shows:
- session metadata,
- worker metadata,
- command counters by method,
- event counters by family,
- active server handles and item counts if gateway shadow state has them,
- latest faults,
- last heartbeat payload,
- a session section: state badge, backend, client identity, client session name,
client correlation id (all redacted), open time, last activity, lease expiry,
events received, and last fault;
- a worker section: process id, worker state, and last heartbeat time;
- a "Recent events" table — the newest 50 events from the in-process mirror, with
worker sequence, family, server handle, item handle, and status (no values), plus
a live/offline pill; gated by the session ACL, so a caller who may not observe the
session sees "Not authorized" instead;
- admin Close session / Kill worker controls (Admin role only).
Per-method command counters, per-family event counters for the session, server
handle / item counts, and the heartbeat payload are not shown — the gateway keeps
no per-session shadow state for them.
The Sessions list, the Workers list, and this details page all render the same
admin controls when the signed-in principal carries the `Admin` role; viewers
and the localhost-anonymous bypass see no action affordances and the server
@@ -484,33 +517,31 @@ management audit path.
### Workers page
Show:
Shows one row per worker that is not closed, with:
- worker process id,
- session id,
- executable path/version,
- state,
- startup duration,
- memory and CPU if available,
- last heartbeat,
- current command correlation id,
- pending command count,
- event queue depth,
- restart/kill reason if terminal.
- worker state,
- owning session id (links to the session details page),
- last heartbeat time,
- last fault summary (the owning session's final fault),
- a Kill worker action for Admin principals.
Executable path and version, startup duration, memory/CPU, in-flight command
correlation, pending command count, and per-worker queue depth are not collected
and are not shown; the worker executable path is on the Settings page.
### Events page
Show aggregate event diagnostics:
Shows aggregate event diagnostics from the snapshot:
- event rate by session,
- event rate by event family,
- total events since start,
- events received since start,
- worker event queue depth and gRPC stream queue depth (gauges),
- queue overflow count,
- stream disconnect count,
- recent terminal faults.
- gRPC stream disconnect count,
- an "Event Families" table: cumulative events received per event family.
Do not display full tag values by default. If value display is later added, make
it opt-in and redacted.
The figures are counters since start, not rates; per-session event counts are on
the Sessions page. No tag values are shown here.
### Browse page
@@ -537,6 +568,33 @@ at the service, not in the page: one decision point, and the value never enters
the render tree. Set the flag true to make the panel the explicit opt-in
tag-value surface it was originally designed as.
The filter box is debounced: keystrokes within 150 ms coalesce into one search,
which runs on the thread pool with a cancellation token that the next keystroke
cancels, and only the latest query may publish its results. The search reads
`DashboardBrowseSearchIndex`, a singleton holding every attribute's full reference
and name lowercased once, keyed on the Galaxy cache `Sequence` and rebuilt only when
a deploy moves it. The previous filter rescanned the whole cached Galaxy with
case-insensitive comparisons, synchronously on the render dispatcher, once per
keystroke per circuit.
The page's two background loops — the 2 s live-read poll and the Galaxy
deploy-notification subscription — handle faults inside the loop and retry; only
cancellation ends them. A poll fault surfaces in the "Live read failed" banner; a
notifier fault surfaces as a "redeploy notifications failed (retrying)" warning and
resubscribes after 5 s; the tree rebuild after a redeploy runs on the renderer
dispatcher. `DisposeAsync` cancels the loops and any in-flight search, drains them
concurrently with the same 5 s bound `DashboardPageBase` uses, and disposes its
cancellation sources in a `finally`.
A closed or backgrounded tab leaves its circuit alive, disconnected, for
`DisconnectedCircuitRetentionPeriod`, which the dashboard sets to **30 seconds**
(`DashboardServiceCollectionExtensions.DisconnectedCircuitRetention`) instead of
the framework's three minutes. Within that window the poll does not read: a scoped
`DashboardCircuitConnectionTracker`, registered as the circuit's `CircuitHandler`,
reports the browser connection down and the poll skips its ticks until it comes
back. Every tick it does run is a `ReadBulk` against the shared dashboard worker,
so a closed tab previously kept the worker busy for three minutes.
### Alarms page
`/dashboard/alarms` lists the alarms the gateway's central alarm monitor
@@ -552,7 +610,10 @@ client, no loopback socket, and no hub token — and the same subscription carri
`snapshot_status` frame behind the truncated-snapshot banner ("Alarm snapshot may be
incomplete"), so a capped provider fetch is caveated the moment the monitor decides
it. The alarm rows themselves still come from the three-second poll, which also
re-asserts the truncation verdict as the reconcile baseline. If `MxGateway:Alarms:Enabled` is
re-asserts the truncation verdict as the reconcile baseline. The filtered rows and
the area list are memoized in `DashboardAlarmView` on the poll's alarm generation and
the filter inputs, so the "Showing" card and the table share one filter-and-sort per
change rather than recomputing it several times per render. If `MxGateway:Alarms:Enabled` is
false the central monitor never starts, and the page says so instead of showing
an empty list with no explanation.
@@ -599,6 +660,19 @@ overflow against the oversized set and evicts the whole excess in one pass (a
subscribes nothing new evicts nothing, but neither can it grow the set. A browse
page requests far fewer tags than the cap, so in practice the set settles at 256.
The 256 is shared between viewers rather than one global LRU. Each Browse page
reads under its own viewer id, every advised tag records the viewer that read it
last, and each active viewer (one that read within the last 30 s) owns a share of
`256 / active viewers`, floored at 32. When a read needs room, eviction first takes
the least-recently-read tags of *other* viewers that are over their share — an
inactive viewer's share is zero, so a closed page's tags go first — and then the
reader's own tags that this read did not touch. A viewer within its share is never
evicted by someone else's read, so with several viewers the cap is soft: the set
can sit above 256 rather than strip a neighbour. Under the global LRU, two circuits
browsing different areas evicted each other's whole panels on every poll and
re-advised them two seconds later. With a single viewer the behaviour is exactly
the global LRU described above.
The Alarms page does **not** use the dashboard session: alarm data comes from
the gateway's always-on central monitor. `QueryAlarmsAsync` reads
`IGatewayAlarmService.CurrentAlarms` — the monitor's in-process cache — so the
@@ -665,20 +739,24 @@ Secrets and pepper values are never logged.
### Settings page
Show read-only effective configuration:
Shows the read-only effective configuration:
- worker executable path,
- configured timeouts,
- queue capacities,
- auth mode,
- SQLite auth database path with sensitive parts redacted if needed,
- dashboard enabled state,
- authentication: mode, SQLite auth database path, pepper secret name, and whether
migrations run on startup;
- LDAP: enabled, server and port, fallback servers, transport, search base,
service account DN, service password (redacted), and the username and group
attributes;
- the LDAP group mappings that decide what a signed-in user may see —
`Dashboard:GroupToRole` (group → `Administrator` / `Viewer`) and
`Dashboard:GroupToTag` (group → the session tags it grants),
- `Dashboard:UntaggedSessionVisibility`, which decides who sees a session whose
owning API key carries no dashboard tags,
- protocol version.
`Dashboard:GroupToTag` (group → the session tags it grants);
- worker: executable path, required architecture, and the startup, shutdown, and
heartbeat-grace timeouts;
- sessions: default command timeout and max sessions;
- events: queue capacity and backpressure policy;
- dashboard: enabled, anonymous localhost, snapshot interval, recent fault and
session limits, `ShowTagValues`, and `Dashboard:UntaggedSessionVisibility`
(who sees a session whose owning API key carries no dashboard tags);
- the worker protocol version.
Do not show API key secrets or pepper values.