fix(gateway): bound the session replay ring by bytes as well as count (mxaccessgw#172)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m26s
ci / java (push) Successful in 2m28s
ci / portable (push) Failing after 4m48s

The replay ring was bounded only by ReplayBufferCapacity (1024 events) and
ReplayRetentionSeconds. Event size is unbounded up to the worker frame size,
so a burst of large-array events could pin gigabytes per session while the
count looked small.

New MxGateway:Events:ReplayBufferMaxBytes (default 64 MiB, 0 disables) caps
the summed serialized size (MxEvent.CalculateSize, measured only when the cap
is on). Over the cap the oldest entries are evicted, exactly like the count
cap, so a resuming subscriber sees the existing ReplayGap path; an event bigger
than the whole cap is not retained. Every eviction (count, age or byte) now
clears its slot, so an evicted event is released immediately instead of
staying referenced until the slot is reused.

The 64 MiB default is a judgement call: it never binds for scalar traffic
(1024 x a few hundred bytes) and only trims large-array bursts.

Tests: SessionEventDistributorTests byte-cap eviction, single oversized event,
cap 0 = count only (the first two fail with the cap enforcement removed);
validator rejects a negative value. Docs: GatewayConfiguration.md, Sessions.md,
DesignDecisions.md; appsettings.json carries the key.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 12:38:22 -05:00
parent 7b21f304e9
commit 07badd139e
12 changed files with 186 additions and 17 deletions
+2 -2
View File
@@ -78,8 +78,8 @@ Setting it to `0` reverts to closing only on normal lease expiry.
A reconnecting client issues `StreamEvents` with `after_worker_sequence` set to
the last sequence it observed; the gateway replays retained events newer than
that watermark (capped by `MxGateway:Events:ReplayBufferCapacity` and
`MxGateway:Events:ReplayRetentionSeconds`) then transitions seamlessly to live
that watermark (capped by `MxGateway:Events:ReplayBufferCapacity`,
`MxGateway:Events:ReplayRetentionSeconds` and `MxGateway:Events:ReplayBufferMaxBytes`) then transitions seamlessly to live
delivery. If the requested position precedes the oldest retained event, a
`ReplayGap` sentinel signals the client to re-snapshot. The replay→live handoff
is atomic (no gap, no duplicate). See [Sessions](./Sessions.md) for the full
+4 -2
View File
@@ -48,6 +48,7 @@ paths, timeouts, queue sizes, enum values, or protocol values are invalid.
"BackpressurePolicy": "FailFast",
"ReplayBufferCapacity": 1024,
"ReplayRetentionSeconds": 300,
"ReplayBufferMaxBytes": 67108864,
"MaxSparseArrayLength": 1000000
},
"Dashboard": {
@@ -196,6 +197,7 @@ All numeric session options must be greater than zero.
| `MxGateway:Events:BackpressurePolicy` | `FailFast` | Per-subscriber event backpressure behavior when a subscriber's bounded event channel overflows. Overflow is isolated to the offending subscriber: it is always disconnected with an `EventQueueOverflow` fault while the session pump and other subscribers keep running. `FailFast` additionally faults the whole session only in the legacy single-subscriber case (the current default mode); with multiple subscribers it degrades to a per-subscriber disconnect so one slow consumer never faults a shared session. `DisconnectSubscriber` disconnects only the slow subscriber in all cases. |
| `MxGateway:Events:ReplayBufferCapacity` | `1024` | Maximum number of events retained per session in the replay ring buffer, used to re-deliver events a returning subscriber missed (reconnect/reattach). The oldest retained event is evicted once this count is exceeded. `0` disables replay retention. |
| `MxGateway:Events:ReplayRetentionSeconds` | `300` | Maximum age, in seconds, of an event retained in the replay ring buffer. Entries older than this are evicted regardless of capacity. `0` disables age-based eviction. |
| `MxGateway:Events:ReplayBufferMaxBytes` | `67108864` (64 MiB) | Maximum summed serialized size, in bytes, of the events retained per session in the replay ring buffer. The count cap alone does not bound memory: one large-array event can approach the worker frame size, so 1024 of them could pin gigabytes per session. When the retained total exceeds this cap the oldest events are evicted first, and an event larger than the whole cap is not retained at all; a resuming subscriber then sees the ordinary `ReplayGap`. The default never binds for scalar traffic and only trims bursts of large events. `0` disables the byte cap. |
| `MxGateway:Events:MaxSparseArrayLength` | `1000000` | Maximum `total_length` a sparse-array write (`MxSparseArray`) may declare. A write above this cap is rejected with `InvalidArgument` before the full array is materialized, guarding against a single write forcing a multi-GB allocation. Must be between `1` and `Array.MaxLength`. |
`QueueCapacity` must be greater than zero and no greater than `int.MaxValue / 2`
@@ -213,8 +215,8 @@ subscribers `FailFast` degrades to a per-subscriber disconnect, matching
healthy subscribers. With `DisconnectSubscriber`, overflow terminates only the
affected stream while the MXAccess session remains active.
`ReplayBufferCapacity` and `ReplayRetentionSeconds` must each be greater than or
equal to zero (either dimension can be disabled with `0`). A returning subscriber
`ReplayBufferCapacity`, `ReplayRetentionSeconds` and `ReplayBufferMaxBytes` must
each be greater than or equal to zero (any dimension can be disabled with `0`). A returning subscriber
that asks for events older than the oldest still-retained event is told it missed
events (a "gap") and must re-snapshot; whatever is still retained is replayed.
+1 -1
View File
@@ -290,7 +290,7 @@ When `after_worker_sequence` is inside the retained window (nothing was evicted)
**`oldest_available_sequence` when the ring is empty.** Age eviction (`ReplayRetentionSeconds`, default 300) and a disabled ring (`ReplayBufferCapacity = 0`) both leave nothing retained, so there is no oldest-retained sequence to report. In that case the sentinel carries the **next sequence that can possibly be delivered** — the highest sequence the distributor has observed plus one — rather than `0`. That keeps `after_worker_sequence = oldest_available_sequence - 1` the single universal resume formula: the follow-up resume lands exactly on the highest observed sequence, replays nothing, reports no gap, and receives every subsequent live event. Reporting `0` here would make an unsigned client compute `2^64 - 1` and then silently receive nothing, because the live filter drops every event at or below that watermark. Nothing is lost relative to reporting `0` — the evicted interval is unrecoverable either way, and the sentinel's job is to tell the client to re-snapshot. `0` remains the value when there is no gap, where the field is meaningless and never emitted.
The ReplayGap sentinel is emitted **only** on the `StreamEvents` server stream and only to the resuming subscriber — it is never fanned to other subscribers and never appears in `DrainEventsReply` (the diagnostic drain path is untouched). Replay retention itself is bounded by `MxGateway:Events:ReplayBufferCapacity` (count) and `ReplayRetentionSeconds` (age); see [Configuration](GatewayConfiguration.md).
The ReplayGap sentinel is emitted **only** on the `StreamEvents` server stream and only to the resuming subscriber — it is never fanned to other subscribers and never appears in `DrainEventsReply` (the diagnostic drain path is untouched). Replay retention itself is bounded by `MxGateway:Events:ReplayBufferCapacity` (count), `ReplayRetentionSeconds` (age) and `ReplayBufferMaxBytes` (summed serialized size, so a burst of large-array events cannot pin memory the count cap would allow); whichever binds first evicts oldest-first. Evicted slots are cleared at once, so an evicted event is released immediately rather than when its slot is next reused. See [Configuration](GatewayConfiguration.md).
### Close