fix(GWC-25,CLI-35,CLI-36): make the empty-ring ReplayGap resumable end to end
An empty replay ring reported oldest_available_sequence = 0 even when gap was
true. Clients follow the documented after_worker_sequence = oldest - 1 formula,
so an unsigned client computed ulong.MaxValue: the follow-up resume replayed
nothing, reported no gap, and the live filter dropped every subsequent event —
a silently dead stream in the headline detach-and-resume scenario, reachable on
default config once ReplayRetentionSeconds (300) age-evicts the ring.
GWC-25: SessionEventDistributor.RegisterWithReplay's empty-ring branch now
reports _highestSequenceSeen + 1 — the next sequence that can possibly be
delivered — when gap is true, so oldest - 1 lands exactly on the highest
observed sequence and the resume delivers everything newer. Still 0 when there
is no gap, where the field is meaningless and never emitted. Nothing is lost:
the evicted interval was unrecoverable either way, and the sentinel's job is to
say "re-snapshot".
CLI-35: the Python CLI fed every stream item into MessageToDict, which raised on
the ReplayGap dataclass and aborted the command after consuming the stream. A
new _event_row helper renders a gap as {"replayGap": {...}} — the same camelCase
shape the Rust CLI emits — and leaves proto events on the existing path.
CLI-36: the Go CLI formatted result.Event on every row, but the library
deliberately clears Event on a gap, so text mode printed
"0 MX_EVENT_FAMILY_UNSPECIFIED" and JSON mode an empty object, discarding the
resume cursors. The loop now branches on result.IsReplayGap() and renders the
typed row in both modes, counting it toward -limit like any other row. The JSON
row's cursors are typed by hand rather than marshalled with protojson: the
proto3 JSON mapping renders 64-bit integers as strings ("7") while the Rust and
Python CLIs emit numbers (7), so going through protojson would have made Go the
only canonical CLI with a different value type.
Docs in the same change: docs/Sessions.md documents the empty-ring sentinel
value and that oldest - 1 is the universal resume formula in both the retained
and fully-evicted cases; docs/CrossLanguageSmokeMatrix.md gains a per-CLI
gap-rendering table covering both client findings, and records exactly what is
and is not comparable across CLIs (same keys and numeric cursors for Rust/Go/
Python; quoted cursors for .NET/Java; differing key order, whitespace, and
container), so a matrix runner compares parsed values rather than raw bytes.
Tests, all written red first and each reproducing its defect verbatim:
- SessionEventDistributorTests: RegisterWithReplayReportsNextDeliverableSequence
WhenRingEmptiedByAge, ...WithRetentionDisabled, and
ResumeUsingSentinelFormulaAfterEmptyRingGapDeliversLiveEvents.
- GatewayEndToEndReconnectReplayTests.ReconnectAfterFullAgeEvictionResumesWith
SentinelFormula — fake-worker e2e resume walk on a fake clock; the fixture now
takes a retention window and a TimeProvider.
- clients/python test_stream_events_renders_replay_gap.
- clients/go TestRunStreamEventsPrintsReplayGap.
GWC-25's ReplayGap.oldest_available_sequence proto-comment amendment is
deliberately deferred to the later codegen wave (see the tracker change log): it
is comment-only but triggers the full five-client regen fan-out.
This commit is contained in:
Joseph Doherty
@@ -841,6 +841,153 @@ public sealed class SessionEventDistributorTests
|
||||
Assert.Equal(4ul, live.WorkerSequence);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// GWC-25: when age eviction has emptied the replay ring, a resume behind the highest
|
||||
/// observed sequence still reports a gap, and the reported oldest-available sequence is
|
||||
/// the next sequence that can possibly be delivered (highest seen + 1) — never <c>0</c>,
|
||||
/// which would make the client's documented <c>oldest - 1</c> resume formula wrap to
|
||||
/// <see cref="ulong.MaxValue"/> and dead-stream the subscriber.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task RegisterWithReplayReportsNextDeliverableSequenceWhenRingEmptiedByAge()
|
||||
{
|
||||
FakeTimeProvider time = new(new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero));
|
||||
Channel<MxEvent> source = Channel.CreateUnbounded<MxEvent>();
|
||||
await using SessionEventDistributor distributor = CreateDistributor(
|
||||
source.Reader,
|
||||
replayBufferCapacity: 100,
|
||||
replayRetentionSeconds: 30,
|
||||
timeProvider: time);
|
||||
await distributor.StartAsync(CancellationToken.None);
|
||||
|
||||
// A primer subscriber forces the pump to retain events 1..3 deterministically.
|
||||
using IEventSubscriberLease primer = distributor.Register();
|
||||
for (ulong sequence = 1; sequence <= 3; sequence++)
|
||||
{
|
||||
source.Writer.TryWrite(Event(sequence));
|
||||
_ = await ReadOneAsync(primer.Reader);
|
||||
}
|
||||
|
||||
// Past the retention window: RegisterWithReplay's EvictAged() empties the ring entirely.
|
||||
time.Advance(TimeSpan.FromSeconds(60));
|
||||
|
||||
using IEventSubscriberLease resume = distributor.RegisterWithReplay(
|
||||
1,
|
||||
out IReadOnlyList<MxEvent> replay,
|
||||
out bool gap,
|
||||
out ulong oldestAvailable,
|
||||
out ulong liveResume);
|
||||
|
||||
Assert.True(gap);
|
||||
Assert.Empty(replay);
|
||||
Assert.Equal(1ul, liveResume);
|
||||
|
||||
// Highest seen is 3, so the next deliverable sequence is 4.
|
||||
Assert.Equal(4ul, oldestAvailable);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// GWC-25: the same next-deliverable-sequence rule applies when retention is disabled
|
||||
/// outright (replay capacity 0) — the ring is always empty, so a behind cursor must still
|
||||
/// get a usable resume anchor rather than <c>0</c>.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task RegisterWithReplayReportsNextDeliverableSequenceWithRetentionDisabled()
|
||||
{
|
||||
Channel<MxEvent> source = Channel.CreateUnbounded<MxEvent>();
|
||||
await using SessionEventDistributor distributor = CreateDistributor(
|
||||
source.Reader,
|
||||
replayBufferCapacity: 0,
|
||||
replayRetentionSeconds: 0);
|
||||
await distributor.StartAsync(CancellationToken.None);
|
||||
|
||||
using IEventSubscriberLease primer = distributor.Register();
|
||||
for (ulong sequence = 1; sequence <= 3; sequence++)
|
||||
{
|
||||
source.Writer.TryWrite(Event(sequence));
|
||||
_ = await ReadOneAsync(primer.Reader);
|
||||
}
|
||||
|
||||
using IEventSubscriberLease resume = distributor.RegisterWithReplay(
|
||||
1,
|
||||
out IReadOnlyList<MxEvent> replay,
|
||||
out bool gap,
|
||||
out ulong oldestAvailable,
|
||||
out _);
|
||||
|
||||
Assert.True(gap);
|
||||
Assert.Empty(replay);
|
||||
Assert.Equal(4ul, oldestAvailable);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// GWC-25 regression: a client that applies the documented
|
||||
/// <c>after_worker_sequence = oldest_available_sequence - 1</c> formula to an empty-ring
|
||||
/// ReplayGap resumes with a live watermark equal to the highest observed sequence, so the
|
||||
/// caller's live filter passes every subsequent event. Under the pre-fix <c>0</c> the
|
||||
/// formula wrapped to <see cref="ulong.MaxValue"/> and the stream went permanently silent.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task ResumeUsingSentinelFormulaAfterEmptyRingGapDeliversLiveEvents()
|
||||
{
|
||||
FakeTimeProvider time = new(new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero));
|
||||
Channel<MxEvent> source = Channel.CreateUnbounded<MxEvent>();
|
||||
await using SessionEventDistributor distributor = CreateDistributor(
|
||||
source.Reader,
|
||||
replayBufferCapacity: 100,
|
||||
replayRetentionSeconds: 30,
|
||||
timeProvider: time);
|
||||
await distributor.StartAsync(CancellationToken.None);
|
||||
|
||||
using IEventSubscriberLease primer = distributor.Register();
|
||||
for (ulong sequence = 1; sequence <= 3; sequence++)
|
||||
{
|
||||
source.Writer.TryWrite(Event(sequence));
|
||||
_ = await ReadOneAsync(primer.Reader);
|
||||
}
|
||||
|
||||
time.Advance(TimeSpan.FromSeconds(60));
|
||||
|
||||
// First resume: everything is evicted, so the sentinel carries the next deliverable
|
||||
// sequence. The subscriber is disposed immediately — the client reconnects with the
|
||||
// formula below.
|
||||
IEventSubscriberLease gapped = distributor.RegisterWithReplay(
|
||||
1,
|
||||
out _,
|
||||
out bool gap,
|
||||
out ulong oldestAvailable,
|
||||
out _);
|
||||
gapped.Dispose();
|
||||
|
||||
Assert.True(gap);
|
||||
|
||||
// The documented client formula. Unchecked ulong arithmetic: under the pre-fix value of
|
||||
// 0 this wraps to ulong.MaxValue, which is exactly the dead-stream defect.
|
||||
ulong resumeCursor = oldestAvailable - 1;
|
||||
|
||||
using IEventSubscriberLease resumed = distributor.RegisterWithReplay(
|
||||
resumeCursor,
|
||||
out IReadOnlyList<MxEvent> replay,
|
||||
out bool resumedGap,
|
||||
out _,
|
||||
out ulong liveResume);
|
||||
|
||||
Assert.False(resumedGap);
|
||||
Assert.Empty(replay);
|
||||
|
||||
// The live filter the caller applies is "sequence > liveResume": it must sit at the
|
||||
// highest observed sequence so the next event passes.
|
||||
Assert.Equal(3ul, liveResume);
|
||||
|
||||
source.Writer.TryWrite(Event(4));
|
||||
MxEvent live = await ReadOneAsync(resumed.Reader);
|
||||
Assert.Equal(4ul, live.WorkerSequence);
|
||||
Assert.True(live.WorkerSequence > liveResume);
|
||||
}
|
||||
|
||||
private static async Task DrainUntilFaultAsync(ChannelReader<MxEvent> reader)
|
||||
{
|
||||
// Drains any buffered events, then surfaces the channel's completion fault (if any)
|
||||
|
||||
Reference in New Issue
Block a user