perf(dashboard): LRU cap on the shared live-read session's advised set
This commit is contained in:
@@ -165,7 +165,7 @@ bearer). Each hub class is `[Authorize(Policy = HubClientsPolicy)]`.
|
||||
|
||||
| Hub | Path | Producer | Payload | Routing |
|
||||
|---|---|---|---|---|
|
||||
| `DashboardSnapshotHub` | `/hubs/snapshot` | `DashboardSnapshotPublisher` (BackgroundService consuming `IDashboardSnapshotService.WatchSnapshotsAsync`) | `DashboardSnapshot` | Sent to all connected clients on every snapshot tick; new connections receive the current snapshot synchronously in `OnConnectedAsync`. |
|
||||
| `DashboardSnapshotHub` | `/hubs/snapshot` | `DashboardSnapshotPublisher` (BackgroundService consuming `IDashboardSnapshotService.WatchSnapshotsAsync`) | `DashboardSnapshot` | Sent to all connected clients on every snapshot tick, but only while at least one client is connected (see "Idle gating" below); new connections receive the current snapshot synchronously in `OnConnectedAsync`. |
|
||||
| `AlarmsHub` | `/hubs/alarms` | `AlarmsHubPublisher` (BackgroundService consuming `IGatewayAlarmService.StreamAsync(filter: null)`) | `AlarmFeedMessage` (`active_alarm` / `snapshot_complete` / `transition`) | Connected clients auto-join `__alarms__`; all clients receive every message. Publisher auto-reconnects every 5s on stream faults. |
|
||||
| `EventsHub` | `/hubs/events` | `DashboardEventBroadcaster` invoked by each session's internal dashboard-mirror subscriber on its `SessionEventDistributor` (registered when the session becomes Ready) | `MxEvent` | Clients call `SubscribeSession(sessionId)` to join `session:{id}`, which also registers them in `EventsHubViewerRegistry` — the mirror is gated on that registry (see "Mirror gating" below). The dashboard is a first-class distributor subscriber, so it receives the session's events whether or not a gRPC client is streaming. It sees RAW session events — not the per-gRPC-subscriber `AfterWorkerSequence` filtering that `EventStreamService` applies at its own boundary — because the dashboard is a separate LDAP-authenticated monitoring view meant to show the session's full event activity. Tag values are stripped from the mirrored `MxEvent` copy by `DashboardEventBroadcaster` when `Dashboard:ShowTagValues` is false (the default) — event metadata (tag reference, quality, status, timestamps) still renders, but the value fields are blanked, so no value leaks through this seam. The per-session hub ACL that would scope a Viewer to specific sessions is still outstanding (SEC-25 / remediation roadmap item 12); the value redaction is the near-term hardening that closes the value-leak seam independently of that ACL. |
|
||||
|
||||
@@ -187,6 +187,38 @@ Default cadences:
|
||||
- event publisher emits per event fanned by the session's `SessionEventDistributor`
|
||||
to its internal dashboard-mirror subscriber (independent of any gRPC `StreamEvents`).
|
||||
|
||||
### Idle gating and snapshot cost
|
||||
|
||||
A snapshot is not free: each one takes a session-registry snapshot and sorts it,
|
||||
copies the metrics dictionaries under the global metrics lock, and projects
|
||||
sessions, workers, faults, and the Galaxy summary. Without gating that work ran
|
||||
once a second for the life of the process even when no browser was connected.
|
||||
|
||||
`DashboardSnapshotHub` counts live connections into the singleton
|
||||
`DashboardSnapshotHubConnectionCounter` (`OnConnectedAsync` / `OnDisconnectedAsync`,
|
||||
clamped at zero). `DashboardSnapshotPublisher` reads that count before advancing the
|
||||
snapshot enumerator: while it is zero the publisher does not call `MoveNextAsync` at
|
||||
all, so the producing iterator stays suspended at its `yield` and builds nothing —
|
||||
the gate removes the snapshot *build*, not just the broadcast. The publisher
|
||||
re-checks once a second while idle, so the first viewer to connect resumes the tick
|
||||
within roughly one snapshot interval. That viewer does not wait for it either:
|
||||
`DashboardPageBase` seeds its first render synchronously from
|
||||
`IDashboardSnapshotService.GetSnapshot()`, and `OnConnectedAsync` pushes a snapshot
|
||||
to the new connection immediately.
|
||||
|
||||
Two per-tick costs inside the snapshot itself are bounded independently of the gate:
|
||||
|
||||
- the effective configuration (`EffectiveGatewayConfiguration`) is built once and
|
||||
cached. It is a projection of `IOptions<GatewayOptions>`, which the gateway binds
|
||||
at startup and never reloads, so rebuilding the whole option tree every tick
|
||||
produced an identical object;
|
||||
- the API key summaries are refreshed at most once every 15 seconds
|
||||
(`ApiKeySummaryRefreshInterval`) instead of on every tick. The list is a SQLite
|
||||
read whose content changes only when an operator creates, rotates, or revokes a
|
||||
key, so a key change reaches the dashboard within that interval. Only a
|
||||
*successful* refresh restarts the interval, so a failed or timed-out read is
|
||||
retried on the next tick and the previous summaries stay on screen.
|
||||
|
||||
Avoid pushing every MXAccess data-change event into a wider broadcast group.
|
||||
The current design routes events strictly through `session:{id}` groups; the
|
||||
snapshot hub continues to carry aggregate event counters and rates.
|
||||
@@ -362,6 +394,18 @@ its lease expires. One session means one worker process backs every dashboard
|
||||
circuit; all access is serialised so the worker sees one in-flight command at a
|
||||
time. Tag reads go through `GatewaySession.SubscribeBulkAsync` / `ReadBulkAsync`.
|
||||
|
||||
The advise set that backs those reads is capped at 256 tags (one browse page plus
|
||||
headroom) and evicted least-recently-read-first. Without the cap every tag any
|
||||
viewer ever inspected stayed advised on the single dashboard worker until the
|
||||
session faulted, so browsing a large galaxy accreted unbounded live MXAccess
|
||||
subscriptions — and the event churn they feed — on one x86 process. Reading a tag
|
||||
already in the set marks it most-recently-read; subscribing past the cap unadvises
|
||||
the oldest entries with `GatewaySession.UnsubscribeBulkAsync` in one batch before
|
||||
the new ones are advised. Tags read in the same call are never evicted to make
|
||||
room for each other. A failed unadvise does not fail the read: the tags are
|
||||
dropped from tracking anyway (they re-subscribe if read again), because the
|
||||
session-invalidation path already handles gateway/worker drift.
|
||||
|
||||
The Alarms page does **not** use the dashboard session: alarm data comes from
|
||||
the gateway's always-on central monitor. `QueryAlarmsAsync` reads
|
||||
`IGatewayAlarmService.CurrentAlarms` — the monitor's in-process cache — so the
|
||||
|
||||
Reference in New Issue
Block a user