perf(dashboard): LRU cap on the shared live-read session's advised set

This commit is contained in:
Joseph Doherty
2026-08-15 12:21:13 -04:00
parent f1e26fed4f
commit 75e3dc2794
3 changed files with 540 additions and 10 deletions
+45 -1
View File
@@ -165,7 +165,7 @@ bearer). Each hub class is `[Authorize(Policy = HubClientsPolicy)]`.
| Hub | Path | Producer | Payload | Routing |
|---|---|---|---|---|
| `DashboardSnapshotHub` | `/hubs/snapshot` | `DashboardSnapshotPublisher` (BackgroundService consuming `IDashboardSnapshotService.WatchSnapshotsAsync`) | `DashboardSnapshot` | Sent to all connected clients on every snapshot tick; new connections receive the current snapshot synchronously in `OnConnectedAsync`. |
| `DashboardSnapshotHub` | `/hubs/snapshot` | `DashboardSnapshotPublisher` (BackgroundService consuming `IDashboardSnapshotService.WatchSnapshotsAsync`) | `DashboardSnapshot` | Sent to all connected clients on every snapshot tick, but only while at least one client is connected (see "Idle gating" below); new connections receive the current snapshot synchronously in `OnConnectedAsync`. |
| `AlarmsHub` | `/hubs/alarms` | `AlarmsHubPublisher` (BackgroundService consuming `IGatewayAlarmService.StreamAsync(filter: null)`) | `AlarmFeedMessage` (`active_alarm` / `snapshot_complete` / `transition`) | Connected clients auto-join `__alarms__`; all clients receive every message. Publisher auto-reconnects every 5s on stream faults. |
| `EventsHub` | `/hubs/events` | `DashboardEventBroadcaster` invoked by each session's internal dashboard-mirror subscriber on its `SessionEventDistributor` (registered when the session becomes Ready) | `MxEvent` | Clients call `SubscribeSession(sessionId)` to join `session:{id}`, which also registers them in `EventsHubViewerRegistry` — the mirror is gated on that registry (see "Mirror gating" below). The dashboard is a first-class distributor subscriber, so it receives the session's events whether or not a gRPC client is streaming. It sees RAW session events — not the per-gRPC-subscriber `AfterWorkerSequence` filtering that `EventStreamService` applies at its own boundary — because the dashboard is a separate LDAP-authenticated monitoring view meant to show the session's full event activity. Tag values are stripped from the mirrored `MxEvent` copy by `DashboardEventBroadcaster` when `Dashboard:ShowTagValues` is false (the default) — event metadata (tag reference, quality, status, timestamps) still renders, but the value fields are blanked, so no value leaks through this seam. The per-session hub ACL that would scope a Viewer to specific sessions is still outstanding (SEC-25 / remediation roadmap item 12); the value redaction is the near-term hardening that closes the value-leak seam independently of that ACL. |
@@ -187,6 +187,38 @@ Default cadences:
- event publisher emits per event fanned by the session's `SessionEventDistributor`
to its internal dashboard-mirror subscriber (independent of any gRPC `StreamEvents`).
### Idle gating and snapshot cost
A snapshot is not free: each one takes a session-registry snapshot and sorts it,
copies the metrics dictionaries under the global metrics lock, and projects
sessions, workers, faults, and the Galaxy summary. Without gating that work ran
once a second for the life of the process even when no browser was connected.
`DashboardSnapshotHub` counts live connections into the singleton
`DashboardSnapshotHubConnectionCounter` (`OnConnectedAsync` / `OnDisconnectedAsync`,
clamped at zero). `DashboardSnapshotPublisher` reads that count before advancing the
snapshot enumerator: while it is zero the publisher does not call `MoveNextAsync` at
all, so the producing iterator stays suspended at its `yield` and builds nothing —
the gate removes the snapshot *build*, not just the broadcast. The publisher
re-checks once a second while idle, so the first viewer to connect resumes the tick
within roughly one snapshot interval. That viewer does not wait for it either:
`DashboardPageBase` seeds its first render synchronously from
`IDashboardSnapshotService.GetSnapshot()`, and `OnConnectedAsync` pushes a snapshot
to the new connection immediately.
Two per-tick costs inside the snapshot itself are bounded independently of the gate:
- the effective configuration (`EffectiveGatewayConfiguration`) is built once and
cached. It is a projection of `IOptions<GatewayOptions>`, which the gateway binds
at startup and never reloads, so rebuilding the whole option tree every tick
produced an identical object;
- the API key summaries are refreshed at most once every 15 seconds
(`ApiKeySummaryRefreshInterval`) instead of on every tick. The list is a SQLite
read whose content changes only when an operator creates, rotates, or revokes a
key, so a key change reaches the dashboard within that interval. Only a
*successful* refresh restarts the interval, so a failed or timed-out read is
retried on the next tick and the previous summaries stay on screen.
Avoid pushing every MXAccess data-change event into a wider broadcast group.
The current design routes events strictly through `session:{id}` groups; the
snapshot hub continues to carry aggregate event counters and rates.
@@ -362,6 +394,18 @@ its lease expires. One session means one worker process backs every dashboard
circuit; all access is serialised so the worker sees one in-flight command at a
time. Tag reads go through `GatewaySession.SubscribeBulkAsync` / `ReadBulkAsync`.
The advise set that backs those reads is capped at 256 tags (one browse page plus
headroom) and evicted least-recently-read-first. Without the cap every tag any
viewer ever inspected stayed advised on the single dashboard worker until the
session faulted, so browsing a large galaxy accreted unbounded live MXAccess
subscriptions — and the event churn they feed — on one x86 process. Reading a tag
already in the set marks it most-recently-read; subscribing past the cap unadvises
the oldest entries with `GatewaySession.UnsubscribeBulkAsync` in one batch before
the new ones are advised. Tags read in the same call are never evicted to make
room for each other. A failed unadvise does not fail the read: the tags are
dropped from tracking anyway (they re-subscribe if read again), because the
session-invalidation path already handles gateway/worker drift.
The Alarms page does **not** use the dashboard session: alarm data comes from
the gateway's always-on central monitor. `QueryAlarmsAsync` reads
`IGatewayAlarmService.CurrentAlarms` — the monitor's in-process cache — so the