fix(gateway): bound the session replay ring by bytes as well as count (mxaccessgw#172)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m26s
ci / java (push) Successful in 2m28s
ci / portable (push) Failing after 4m48s

The replay ring was bounded only by ReplayBufferCapacity (1024 events) and
ReplayRetentionSeconds. Event size is unbounded up to the worker frame size,
so a burst of large-array events could pin gigabytes per session while the
count looked small.

New MxGateway:Events:ReplayBufferMaxBytes (default 64 MiB, 0 disables) caps
the summed serialized size (MxEvent.CalculateSize, measured only when the cap
is on). Over the cap the oldest entries are evicted, exactly like the count
cap, so a resuming subscriber sees the existing ReplayGap path; an event bigger
than the whole cap is not retained. Every eviction (count, age or byte) now
clears its slot, so an evicted event is released immediately instead of
staying referenced until the slot is reused.

The 64 MiB default is a judgement call: it never binds for scalar traffic
(1024 x a few hundred bytes) and only trims large-array bursts.

Tests: SessionEventDistributorTests byte-cap eviction, single oversized event,
cap 0 = count only (the first two fail with the cap enforcement removed);
validator rejects a negative value. Docs: GatewayConfiguration.md, Sessions.md,
DesignDecisions.md; appsettings.json carries the key.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 12:38:22 -05:00
parent 7b21f304e9
commit 07badd139e
12 changed files with 186 additions and 17 deletions
+2 -2
View File
@@ -78,8 +78,8 @@ Setting it to `0` reverts to closing only on normal lease expiry.
A reconnecting client issues `StreamEvents` with `after_worker_sequence` set to A reconnecting client issues `StreamEvents` with `after_worker_sequence` set to
the last sequence it observed; the gateway replays retained events newer than the last sequence it observed; the gateway replays retained events newer than
that watermark (capped by `MxGateway:Events:ReplayBufferCapacity` and that watermark (capped by `MxGateway:Events:ReplayBufferCapacity`,
`MxGateway:Events:ReplayRetentionSeconds`) then transitions seamlessly to live `MxGateway:Events:ReplayRetentionSeconds` and `MxGateway:Events:ReplayBufferMaxBytes`) then transitions seamlessly to live
delivery. If the requested position precedes the oldest retained event, a delivery. If the requested position precedes the oldest retained event, a
`ReplayGap` sentinel signals the client to re-snapshot. The replay→live handoff `ReplayGap` sentinel signals the client to re-snapshot. The replay→live handoff
is atomic (no gap, no duplicate). See [Sessions](./Sessions.md) for the full is atomic (no gap, no duplicate). See [Sessions](./Sessions.md) for the full
+4 -2
View File
@@ -48,6 +48,7 @@ paths, timeouts, queue sizes, enum values, or protocol values are invalid.
"BackpressurePolicy": "FailFast", "BackpressurePolicy": "FailFast",
"ReplayBufferCapacity": 1024, "ReplayBufferCapacity": 1024,
"ReplayRetentionSeconds": 300, "ReplayRetentionSeconds": 300,
"ReplayBufferMaxBytes": 67108864,
"MaxSparseArrayLength": 1000000 "MaxSparseArrayLength": 1000000
}, },
"Dashboard": { "Dashboard": {
@@ -196,6 +197,7 @@ All numeric session options must be greater than zero.
| `MxGateway:Events:BackpressurePolicy` | `FailFast` | Per-subscriber event backpressure behavior when a subscriber's bounded event channel overflows. Overflow is isolated to the offending subscriber: it is always disconnected with an `EventQueueOverflow` fault while the session pump and other subscribers keep running. `FailFast` additionally faults the whole session only in the legacy single-subscriber case (the current default mode); with multiple subscribers it degrades to a per-subscriber disconnect so one slow consumer never faults a shared session. `DisconnectSubscriber` disconnects only the slow subscriber in all cases. | | `MxGateway:Events:BackpressurePolicy` | `FailFast` | Per-subscriber event backpressure behavior when a subscriber's bounded event channel overflows. Overflow is isolated to the offending subscriber: it is always disconnected with an `EventQueueOverflow` fault while the session pump and other subscribers keep running. `FailFast` additionally faults the whole session only in the legacy single-subscriber case (the current default mode); with multiple subscribers it degrades to a per-subscriber disconnect so one slow consumer never faults a shared session. `DisconnectSubscriber` disconnects only the slow subscriber in all cases. |
| `MxGateway:Events:ReplayBufferCapacity` | `1024` | Maximum number of events retained per session in the replay ring buffer, used to re-deliver events a returning subscriber missed (reconnect/reattach). The oldest retained event is evicted once this count is exceeded. `0` disables replay retention. | | `MxGateway:Events:ReplayBufferCapacity` | `1024` | Maximum number of events retained per session in the replay ring buffer, used to re-deliver events a returning subscriber missed (reconnect/reattach). The oldest retained event is evicted once this count is exceeded. `0` disables replay retention. |
| `MxGateway:Events:ReplayRetentionSeconds` | `300` | Maximum age, in seconds, of an event retained in the replay ring buffer. Entries older than this are evicted regardless of capacity. `0` disables age-based eviction. | | `MxGateway:Events:ReplayRetentionSeconds` | `300` | Maximum age, in seconds, of an event retained in the replay ring buffer. Entries older than this are evicted regardless of capacity. `0` disables age-based eviction. |
| `MxGateway:Events:ReplayBufferMaxBytes` | `67108864` (64 MiB) | Maximum summed serialized size, in bytes, of the events retained per session in the replay ring buffer. The count cap alone does not bound memory: one large-array event can approach the worker frame size, so 1024 of them could pin gigabytes per session. When the retained total exceeds this cap the oldest events are evicted first, and an event larger than the whole cap is not retained at all; a resuming subscriber then sees the ordinary `ReplayGap`. The default never binds for scalar traffic and only trims bursts of large events. `0` disables the byte cap. |
| `MxGateway:Events:MaxSparseArrayLength` | `1000000` | Maximum `total_length` a sparse-array write (`MxSparseArray`) may declare. A write above this cap is rejected with `InvalidArgument` before the full array is materialized, guarding against a single write forcing a multi-GB allocation. Must be between `1` and `Array.MaxLength`. | | `MxGateway:Events:MaxSparseArrayLength` | `1000000` | Maximum `total_length` a sparse-array write (`MxSparseArray`) may declare. A write above this cap is rejected with `InvalidArgument` before the full array is materialized, guarding against a single write forcing a multi-GB allocation. Must be between `1` and `Array.MaxLength`. |
`QueueCapacity` must be greater than zero and no greater than `int.MaxValue / 2` `QueueCapacity` must be greater than zero and no greater than `int.MaxValue / 2`
@@ -213,8 +215,8 @@ subscribers `FailFast` degrades to a per-subscriber disconnect, matching
healthy subscribers. With `DisconnectSubscriber`, overflow terminates only the healthy subscribers. With `DisconnectSubscriber`, overflow terminates only the
affected stream while the MXAccess session remains active. affected stream while the MXAccess session remains active.
`ReplayBufferCapacity` and `ReplayRetentionSeconds` must each be greater than or `ReplayBufferCapacity`, `ReplayRetentionSeconds` and `ReplayBufferMaxBytes` must
equal to zero (either dimension can be disabled with `0`). A returning subscriber each be greater than or equal to zero (any dimension can be disabled with `0`). A returning subscriber
that asks for events older than the oldest still-retained event is told it missed that asks for events older than the oldest still-retained event is told it missed
events (a "gap") and must re-snapshot; whatever is still retained is replayed. events (a "gap") and must re-snapshot; whatever is still retained is replayed.
+1 -1
View File
@@ -290,7 +290,7 @@ When `after_worker_sequence` is inside the retained window (nothing was evicted)
**`oldest_available_sequence` when the ring is empty.** Age eviction (`ReplayRetentionSeconds`, default 300) and a disabled ring (`ReplayBufferCapacity = 0`) both leave nothing retained, so there is no oldest-retained sequence to report. In that case the sentinel carries the **next sequence that can possibly be delivered** — the highest sequence the distributor has observed plus one — rather than `0`. That keeps `after_worker_sequence = oldest_available_sequence - 1` the single universal resume formula: the follow-up resume lands exactly on the highest observed sequence, replays nothing, reports no gap, and receives every subsequent live event. Reporting `0` here would make an unsigned client compute `2^64 - 1` and then silently receive nothing, because the live filter drops every event at or below that watermark. Nothing is lost relative to reporting `0` — the evicted interval is unrecoverable either way, and the sentinel's job is to tell the client to re-snapshot. `0` remains the value when there is no gap, where the field is meaningless and never emitted. **`oldest_available_sequence` when the ring is empty.** Age eviction (`ReplayRetentionSeconds`, default 300) and a disabled ring (`ReplayBufferCapacity = 0`) both leave nothing retained, so there is no oldest-retained sequence to report. In that case the sentinel carries the **next sequence that can possibly be delivered** — the highest sequence the distributor has observed plus one — rather than `0`. That keeps `after_worker_sequence = oldest_available_sequence - 1` the single universal resume formula: the follow-up resume lands exactly on the highest observed sequence, replays nothing, reports no gap, and receives every subsequent live event. Reporting `0` here would make an unsigned client compute `2^64 - 1` and then silently receive nothing, because the live filter drops every event at or below that watermark. Nothing is lost relative to reporting `0` — the evicted interval is unrecoverable either way, and the sentinel's job is to tell the client to re-snapshot. `0` remains the value when there is no gap, where the field is meaningless and never emitted.
The ReplayGap sentinel is emitted **only** on the `StreamEvents` server stream and only to the resuming subscriber — it is never fanned to other subscribers and never appears in `DrainEventsReply` (the diagnostic drain path is untouched). Replay retention itself is bounded by `MxGateway:Events:ReplayBufferCapacity` (count) and `ReplayRetentionSeconds` (age); see [Configuration](GatewayConfiguration.md). The ReplayGap sentinel is emitted **only** on the `StreamEvents` server stream and only to the resuming subscriber — it is never fanned to other subscribers and never appears in `DrainEventsReply` (the diagnostic drain path is untouched). Replay retention itself is bounded by `MxGateway:Events:ReplayBufferCapacity` (count), `ReplayRetentionSeconds` (age) and `ReplayBufferMaxBytes` (summed serialized size, so a burst of large-array events cannot pin memory the count cap would allow); whichever binds first evicts oldest-first. Evicted slots are cleared at once, so an evicted event is released immediately rather than when its slot is next reused. See [Configuration](GatewayConfiguration.md).
### Close ### Close
@@ -4,4 +4,5 @@ public sealed record EffectiveEventConfiguration(
int QueueCapacity, int QueueCapacity,
string BackpressurePolicy, string BackpressurePolicy,
int ReplayBufferCapacity, int ReplayBufferCapacity,
double ReplayRetentionSeconds); double ReplayRetentionSeconds,
long ReplayBufferMaxBytes);
@@ -28,6 +28,21 @@ public sealed class EventOptions
/// </summary> /// </summary>
public double ReplayRetentionSeconds { get; init; } = 300; public double ReplayRetentionSeconds { get; init; } = 300;
/// <summary>
/// Gets the maximum total serialized size, in bytes, of the events retained in the
/// per-session replay ring buffer. The count cap alone does not bound memory: a burst of
/// large-array events can make <see cref="ReplayBufferCapacity"/> events many times the
/// size typical traffic suggests (up to the worker frame size each). When the retained
/// total exceeds this cap the oldest events are evicted first, exactly as for the count
/// cap, so a resuming subscriber sees the usual <c>ReplayGap</c> sentinel. The default of
/// 64 MiB never binds for ordinary scalar traffic (1,024 events of a few hundred bytes)
/// and only trims bursts of large events. A value of <c>0</c> disables the byte cap.
/// </summary>
public long ReplayBufferMaxBytes { get; init; } = DefaultReplayBufferMaxBytes;
/// <summary>Default for <see cref="ReplayBufferMaxBytes"/> (64 MiB).</summary>
public const long DefaultReplayBufferMaxBytes = 64L * 1024 * 1024;
/// <summary> /// <summary>
/// Gets the maximum <c>total_length</c> a sparse-array write may declare before the /// Gets the maximum <c>total_length</c> a sparse-array write may declare before the
/// gateway rejects it with <c>InvalidArgument</c>, enforced in /// gateway rejects it with <c>InvalidArgument</c>, enforced in
@@ -56,7 +56,8 @@ public sealed class GatewayConfigurationProvider(IOptions<GatewayOptions> option
QueueCapacity: value.Events.QueueCapacity, QueueCapacity: value.Events.QueueCapacity,
BackpressurePolicy: value.Events.BackpressurePolicy.ToString(), BackpressurePolicy: value.Events.BackpressurePolicy.ToString(),
ReplayBufferCapacity: value.Events.ReplayBufferCapacity, ReplayBufferCapacity: value.Events.ReplayBufferCapacity,
ReplayRetentionSeconds: value.Events.ReplayRetentionSeconds), ReplayRetentionSeconds: value.Events.ReplayRetentionSeconds,
ReplayBufferMaxBytes: value.Events.ReplayBufferMaxBytes),
Dashboard: new EffectiveDashboardConfiguration( Dashboard: new EffectiveDashboardConfiguration(
Enabled: value.Dashboard.Enabled, Enabled: value.Dashboard.Enabled,
AllowAnonymousLocalhost: value.Dashboard.AllowAnonymousLocalhost, AllowAnonymousLocalhost: value.Dashboard.AllowAnonymousLocalhost,
@@ -385,6 +385,9 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase<GatewayOption
builder.RequireThat( builder.RequireThat(
options.ReplayRetentionSeconds >= 0, options.ReplayRetentionSeconds >= 0,
"MxGateway:Events:ReplayRetentionSeconds must be greater than or equal to zero."); "MxGateway:Events:ReplayRetentionSeconds must be greater than or equal to zero.");
builder.RequireThat(
options.ReplayBufferMaxBytes >= 0,
"MxGateway:Events:ReplayBufferMaxBytes must be greater than or equal to zero.");
builder.RequireThat( builder.RequireThat(
options.MaxSparseArrayLength >= 1 && options.MaxSparseArrayLength <= Array.MaxLength, options.MaxSparseArrayLength >= 1 && options.MaxSparseArrayLength <= Array.MaxLength,
@@ -741,7 +741,8 @@ public sealed class GatewaySession : IAsyncDisposable
_eventStreaming.DistributorLogger, _eventStreaming.DistributorLogger,
_eventStreaming.TimeProvider, _eventStreaming.TimeProvider,
CreateOverflowHandler(eventOptions.BackpressurePolicy), CreateOverflowHandler(eventOptions.BackpressurePolicy),
singleSubscriberMode: !_eventStreaming.AllowMultipleEventSubscribers); singleSubscriberMode: !_eventStreaming.AllowMultipleEventSubscribers,
replayBufferMaxBytes: eventOptions.ReplayBufferMaxBytes);
} }
startNow = false; startNow = false;
@@ -137,8 +137,14 @@ public sealed class SessionEventDistributor : IAsyncDisposable
// event) and _replayCount entries follow it, wrapping modulo the array length. The // event) and _replayCount entries follow it, wrapping modulo the array length. The
// logical entry at position i is _replayBuffer[(_replayHead + i) % Length]. Capacity // logical entry at position i is _replayBuffer[(_replayHead + i) % Length]. Capacity
// == 0 disables retention (the array is empty and never indexed); RetentionSeconds // == 0 disables retention (the array is empty and never indexed); RetentionSeconds
// <= 0 disables age-based eviction. // <= 0 disables age-based eviction. _replayMaxBytes > 0 additionally bounds the summed
// serialized size of the retained events (_replayBytes), evicting oldest-first
// (mxaccessgw#172); 0 disables the byte cap. Every eviction clears its slot so an evicted
// event — possibly a multi-MB array — is released at once rather than when the slot is
// next overwritten.
private readonly int _replayBufferCapacity; private readonly int _replayBufferCapacity;
private readonly long _replayMaxBytes;
private long _replayBytes;
private readonly TimeSpan _replayRetention; private readonly TimeSpan _replayRetention;
private readonly bool _ageEvictionEnabled; private readonly bool _ageEvictionEnabled;
private readonly ReplayEntry[] _replayBuffer; private readonly ReplayEntry[] _replayBuffer;
@@ -246,6 +252,11 @@ public sealed class SessionEventDistributor : IAsyncDisposable
/// handler. When <see langword="null"/> (unit/skeleton use) the offending subscriber is /// handler. When <see langword="null"/> (unit/skeleton use) the offending subscriber is
/// still disconnected but no metric/fault side effect runs. /// still disconnected but no metric/fault side effect runs.
/// </param> /// </param>
/// <param name="replayBufferMaxBytes">
/// Maximum summed serialized size, in bytes, of the retained replay events. When an append
/// takes the total above it the oldest entries are evicted first — including the new one
/// itself if it alone exceeds the cap. <c>0</c> (the default) disables the byte cap.
/// </param>
/// <param name="singleSubscriberMode"> /// <param name="singleSubscriberMode">
/// <see langword="true"/> when the owning session is in single-subscriber mode /// <see langword="true"/> when the owning session is in single-subscriber mode
/// (<c>AllowMultipleEventSubscribers == false</c>). This gates the FailFast /// (<c>AllowMultipleEventSubscribers == false</c>). This gates the FailFast
@@ -266,13 +277,15 @@ public sealed class SessionEventDistributor : IAsyncDisposable
ILogger<SessionEventDistributor> logger, ILogger<SessionEventDistributor> logger,
TimeProvider timeProvider, TimeProvider timeProvider,
SubscriberOverflowHandler? overflowHandler = null, SubscriberOverflowHandler? overflowHandler = null,
bool singleSubscriberMode = true) bool singleSubscriberMode = true,
long replayBufferMaxBytes = 0)
{ {
ArgumentException.ThrowIfNullOrWhiteSpace(sessionId); ArgumentException.ThrowIfNullOrWhiteSpace(sessionId);
ArgumentNullException.ThrowIfNull(eventSourceFactory); ArgumentNullException.ThrowIfNull(eventSourceFactory);
ArgumentOutOfRangeException.ThrowIfLessThan(subscriberQueueCapacity, 1); ArgumentOutOfRangeException.ThrowIfLessThan(subscriberQueueCapacity, 1);
ArgumentOutOfRangeException.ThrowIfNegative(replayBufferCapacity); ArgumentOutOfRangeException.ThrowIfNegative(replayBufferCapacity);
ArgumentOutOfRangeException.ThrowIfNegative(replayRetentionSeconds); ArgumentOutOfRangeException.ThrowIfNegative(replayRetentionSeconds);
ArgumentOutOfRangeException.ThrowIfNegative(replayBufferMaxBytes);
ArgumentNullException.ThrowIfNull(logger); ArgumentNullException.ThrowIfNull(logger);
ArgumentNullException.ThrowIfNull(timeProvider); ArgumentNullException.ThrowIfNull(timeProvider);
@@ -284,6 +297,7 @@ public sealed class SessionEventDistributor : IAsyncDisposable
_shutdownTimeout = DefaultShutdownTimeout; _shutdownTimeout = DefaultShutdownTimeout;
_replayBufferCapacity = replayBufferCapacity; _replayBufferCapacity = replayBufferCapacity;
_replayBuffer = new ReplayEntry[replayBufferCapacity]; _replayBuffer = new ReplayEntry[replayBufferCapacity];
_replayMaxBytes = replayBufferMaxBytes;
_ageEvictionEnabled = replayRetentionSeconds > 0; _ageEvictionEnabled = replayRetentionSeconds > 0;
_replayRetention = _ageEvictionEnabled _replayRetention = _ageEvictionEnabled
? TimeSpan.FromSeconds(replayRetentionSeconds) ? TimeSpan.FromSeconds(replayRetentionSeconds)
@@ -946,7 +960,10 @@ public sealed class SessionEventDistributor : IAsyncDisposable
// Append at the logical tail. When the ring is full the oldest entry is // Append at the logical tail. When the ring is full the oldest entry is
// overwritten in place (its slot becomes the new tail) and the head advances, // overwritten in place (its slot becomes the new tail) and the head advances,
// so the newest _replayBufferCapacity events are retained with no allocation. // so the newest _replayBufferCapacity events are retained with no allocation.
ReplayEntry entry = new(mxEvent, _timeProvider.GetUtcNow()); // The size is only measured when the byte cap is on, so the cap-off path
// adds no serialization work to the pump.
int sizeBytes = _replayMaxBytes > 0 ? mxEvent.CalculateSize() : 0;
ReplayEntry entry = new(mxEvent, _timeProvider.GetUtcNow(), sizeBytes);
if (_replayCount < _replayBufferCapacity) if (_replayCount < _replayBufferCapacity)
{ {
_replayBuffer[(_replayHead + _replayCount) % _replayBufferCapacity] = entry; _replayBuffer[(_replayHead + _replayCount) % _replayBufferCapacity] = entry;
@@ -954,11 +971,14 @@ public sealed class SessionEventDistributor : IAsyncDisposable
} }
else else
{ {
_replayBytes -= _replayBuffer[_replayHead].SizeBytes;
_replayBuffer[_replayHead] = entry; _replayBuffer[_replayHead] = entry;
_replayHead = (_replayHead + 1) % _replayBufferCapacity; _replayHead = (_replayHead + 1) % _replayBufferCapacity;
} }
_replayBytes += sizeBytes;
EvictAged(); EvictAged();
EvictOverByteCap();
} }
// Single capture point for both the retained and no-retention paths. // Single capture point for both the retained and no-retention paths.
@@ -982,12 +1002,38 @@ public sealed class SessionEventDistributor : IAsyncDisposable
DateTimeOffset cutoff = _timeProvider.GetUtcNow() - _replayRetention; DateTimeOffset cutoff = _timeProvider.GetUtcNow() - _replayRetention;
while (_replayCount > 0 && _replayBuffer[_replayHead].RetainedAt < cutoff) while (_replayCount > 0 && _replayBuffer[_replayHead].RetainedAt < cutoff)
{ {
_replayHead = (_replayHead + 1) % _replayBuffer.Length; EvictOldest();
_replayCount--;
} }
} }
private readonly record struct ReplayEntry(MxEvent Event, DateTimeOffset RetainedAt); // Must be called under _replayLock. Drops the oldest entries until the retained events'
// summed serialized size is within _replayMaxBytes (mxaccessgw#172). A single event larger
// than the cap is itself evicted, leaving the ring empty; the resume path then reports a gap
// anchored on the highest sequence seen, as for any other eviction.
private void EvictOverByteCap()
{
if (_replayMaxBytes <= 0)
{
return;
}
while (_replayCount > 0 && _replayBytes > _replayMaxBytes)
{
EvictOldest();
}
}
// Must be called under _replayLock with _replayCount > 0. Clears the slot so the evicted
// event is unreferenced immediately.
private void EvictOldest()
{
_replayBytes -= _replayBuffer[_replayHead].SizeBytes;
_replayBuffer[_replayHead] = default;
_replayHead = (_replayHead + 1) % _replayBuffer.Length;
_replayCount--;
}
private readonly record struct ReplayEntry(MxEvent Event, DateTimeOffset RetainedAt, int SizeBytes);
private sealed class Subscriber(long id, Channel<MxEvent> channel, bool isInternal) private sealed class Subscriber(long id, Channel<MxEvent> channel, bool isInternal)
{ {
@@ -70,7 +70,8 @@
"QueueCapacity": 10000, "QueueCapacity": 10000,
"BackpressurePolicy": "FailFast", "BackpressurePolicy": "FailFast",
"ReplayBufferCapacity": 1024, "ReplayBufferCapacity": 1024,
"ReplayRetentionSeconds": 300 "ReplayRetentionSeconds": 300,
"ReplayBufferMaxBytes": 67108864
}, },
"Dashboard": { "Dashboard": {
"Enabled": true, "Enabled": true,
@@ -121,6 +121,7 @@ public sealed class GatewayOptionsTests
[InlineData("MxGateway:Sessions:LeaseSweepIntervalSeconds", "0", "MxGateway:Sessions:LeaseSweepIntervalSeconds must be greater than zero.")] [InlineData("MxGateway:Sessions:LeaseSweepIntervalSeconds", "0", "MxGateway:Sessions:LeaseSweepIntervalSeconds must be greater than zero.")]
[InlineData("MxGateway:Sessions:DetachGraceSeconds", "-1", "MxGateway:Sessions:DetachGraceSeconds must be zero or greater (0 disables detach-grace retention).")] [InlineData("MxGateway:Sessions:DetachGraceSeconds", "-1", "MxGateway:Sessions:DetachGraceSeconds must be zero or greater (0 disables detach-grace retention).")]
[InlineData("MxGateway:Sessions:WorkerReadyWaitTimeoutMs", "-1", "MxGateway:Sessions:WorkerReadyWaitTimeoutMs must be greater than or equal to zero.")] [InlineData("MxGateway:Sessions:WorkerReadyWaitTimeoutMs", "-1", "MxGateway:Sessions:WorkerReadyWaitTimeoutMs must be greater than or equal to zero.")]
[InlineData("MxGateway:Events:ReplayBufferMaxBytes", "-1", "MxGateway:Events:ReplayBufferMaxBytes must be greater than or equal to zero.")]
[InlineData("MxGateway:Events:QueueCapacity", "0", "MxGateway:Events:QueueCapacity must be greater than zero.")] [InlineData("MxGateway:Events:QueueCapacity", "0", "MxGateway:Events:QueueCapacity must be greater than zero.")]
[InlineData("MxGateway:Protocol:MaxGrpcMessageBytes", "0", "MxGateway:Protocol:MaxGrpcMessageBytes must be between")] [InlineData("MxGateway:Protocol:MaxGrpcMessageBytes", "0", "MxGateway:Protocol:MaxGrpcMessageBytes must be between")]
[InlineData("MxGateway:Authentication:PepperSecretName", "", "MxGateway:Authentication:PepperSecretName is required")] [InlineData("MxGateway:Authentication:PepperSecretName", "", "MxGateway:Authentication:PepperSecretName is required")]
@@ -186,6 +186,97 @@ public sealed class SessionEventDistributorTests
Assert.Equal(new ulong[] { 3, 4, 5 }, replay.Select(e => e.WorkerSequence)); Assert.Equal(new ulong[] { 3, 4, 5 }, replay.Select(e => e.WorkerSequence));
} }
/// <summary>
/// mxaccessgw#172: the byte cap evicts oldest-first once the retained events' summed
/// serialized size exceeds it, even though the count cap is nowhere near reached, and a
/// resume from before the retained window reports a gap.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task ReplayBuffer_OverByteCap_EvictsOldestFirst_AndReportsGap()
{
int eventBytes = LargeEvent(1).CalculateSize();
Channel<MxEvent> source = Channel.CreateUnbounded<MxEvent>();
await using SessionEventDistributor distributor = CreateDistributor(
source.Reader,
replayBufferCapacity: 100,
replayRetentionSeconds: 0,
replayBufferMaxBytes: (eventBytes * 3) + (eventBytes / 2));
await distributor.StartAsync(CancellationToken.None);
using IEventSubscriberLease lease = distributor.Register();
for (ulong sequence = 1; sequence <= 5; sequence++)
{
source.Writer.TryWrite(LargeEvent(sequence));
}
for (ulong sequence = 1; sequence <= 5; sequence++)
{
Assert.Equal(sequence, (await ReadOneAsync(lease.Reader)).WorkerSequence);
}
Assert.True(distributor.TryGetReplayFrom(0, out IReadOnlyList<MxEvent> replay, out bool gap));
Assert.True(gap);
Assert.Equal(new ulong[] { 3, 4, 5 }, replay.Select(e => e.WorkerSequence));
}
/// <summary>
/// mxaccessgw#172: an event larger than the whole byte cap is not retained at all (the
/// cap is strict), and a resume from before it reports a gap with nothing to replay.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task ReplayBuffer_SingleEventOverByteCap_RetainsNothing_AndReportsGap()
{
Channel<MxEvent> source = Channel.CreateUnbounded<MxEvent>();
await using SessionEventDistributor distributor = CreateDistributor(
source.Reader,
replayBufferCapacity: 100,
replayRetentionSeconds: 0,
replayBufferMaxBytes: LargeEvent(1).CalculateSize() - 1);
await distributor.StartAsync(CancellationToken.None);
using IEventSubscriberLease lease = distributor.Register();
source.Writer.TryWrite(LargeEvent(1));
Assert.Equal(1UL, (await ReadOneAsync(lease.Reader)).WorkerSequence);
Assert.True(distributor.TryGetReplayFrom(0, out IReadOnlyList<MxEvent> replay, out bool gap));
Assert.True(gap);
Assert.Empty(replay);
}
/// <summary>
/// mxaccessgw#172: a byte cap of <c>0</c> disables the byte dimension, leaving only the
/// count cap — the pre-cap behaviour.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task ReplayBuffer_ByteCapZero_BoundsByCountOnly()
{
Channel<MxEvent> source = Channel.CreateUnbounded<MxEvent>();
await using SessionEventDistributor distributor = CreateDistributor(
source.Reader,
replayBufferCapacity: 100,
replayRetentionSeconds: 0,
replayBufferMaxBytes: 0);
await distributor.StartAsync(CancellationToken.None);
using IEventSubscriberLease lease = distributor.Register();
for (ulong sequence = 1; sequence <= 5; sequence++)
{
source.Writer.TryWrite(LargeEvent(sequence));
}
for (ulong sequence = 1; sequence <= 5; sequence++)
{
Assert.Equal(sequence, (await ReadOneAsync(lease.Reader)).WorkerSequence);
}
Assert.True(distributor.TryGetReplayFrom(0, out IReadOnlyList<MxEvent> replay, out bool gap));
Assert.False(gap);
Assert.Equal(new ulong[] { 1, 2, 3, 4, 5 }, replay.Select(e => e.WorkerSequence));
}
/// <summary> /// <summary>
/// Appending far more events than the capacity wraps the circular buffer's head /// Appending far more events than the capacity wraps the circular buffer's head
/// index past the array boundary multiple times; the retained window stays the /// index past the array boundary multiple times; the retained window stays the
@@ -1220,7 +1311,8 @@ public sealed class SessionEventDistributorTests
ChannelReader<MxEvent> source, ChannelReader<MxEvent> source,
int replayBufferCapacity, int replayBufferCapacity,
double replayRetentionSeconds, double replayRetentionSeconds,
TimeProvider? timeProvider = null) TimeProvider? timeProvider = null,
long replayBufferMaxBytes = 0)
=> new( => new(
"session-test", "session-test",
ct => source.ReadAllAsync(ct), ct => source.ReadAllAsync(ct),
@@ -1228,11 +1320,17 @@ public sealed class SessionEventDistributorTests
replayBufferCapacity: replayBufferCapacity, replayBufferCapacity: replayBufferCapacity,
replayRetentionSeconds: replayRetentionSeconds, replayRetentionSeconds: replayRetentionSeconds,
NullLogger<SessionEventDistributor>.Instance, NullLogger<SessionEventDistributor>.Instance,
timeProvider ?? TimeProvider.System); timeProvider ?? TimeProvider.System,
replayBufferMaxBytes: replayBufferMaxBytes);
private static MxEvent Event(ulong sequence) private static MxEvent Event(ulong sequence)
=> new() { SessionId = "session-test", WorkerSequence = sequence }; => new() { SessionId = "session-test", WorkerSequence = sequence };
// A ~4 KB event whose serialized size is the same for every sequence below 128 (the varint
// for WorkerSequence stays one byte), so byte-cap arithmetic in the tests is exact.
private static MxEvent LargeEvent(ulong sequence)
=> new() { SessionId = new string('x', 4096), WorkerSequence = sequence };
private static async Task<MxEvent> ReadOneAsync(ChannelReader<MxEvent> reader) private static async Task<MxEvent> ReadOneAsync(ChannelReader<MxEvent> reader)
{ {
await reader.WaitToReadAsync().AsTask().WaitAsync(ReadTimeout); await reader.WaitToReadAsync().AsTask().WaitAsync(ReadTimeout);