dohertj2/mxaccessgw
1
0
Fork 0
Code Issues 3 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

fix(GWC-25,CLI-35,CLI-36): make the empty-ring ReplayGap resumable end to end

Browse Source 
An empty replay ring reported oldest_available_sequence = 0 even when gap was
true. Clients follow the documented after_worker_sequence = oldest - 1 formula,
so an unsigned client computed ulong.MaxValue: the follow-up resume replayed
nothing, reported no gap, and the live filter dropped every subsequent event —
a silently dead stream in the headline detach-and-resume scenario, reachable on
default config once ReplayRetentionSeconds (300) age-evicts the ring.

GWC-25: SessionEventDistributor.RegisterWithReplay's empty-ring branch now
reports _highestSequenceSeen + 1 — the next sequence that can possibly be
delivered — when gap is true, so oldest - 1 lands exactly on the highest
observed sequence and the resume delivers everything newer. Still 0 when there
is no gap, where the field is meaningless and never emitted. Nothing is lost:
the evicted interval was unrecoverable either way, and the sentinel's job is to
say "re-snapshot".

CLI-35: the Python CLI fed every stream item into MessageToDict, which raised on
the ReplayGap dataclass and aborted the command after consuming the stream. A
new _event_row helper renders a gap as {"replayGap": {...}} — the same camelCase
shape the Rust CLI emits — and leaves proto events on the existing path.

CLI-36: the Go CLI formatted result.Event on every row, but the library
deliberately clears Event on a gap, so text mode printed
"0 MX_EVENT_FAMILY_UNSPECIFIED" and JSON mode an empty object, discarding the
resume cursors. The loop now branches on result.IsReplayGap() and renders the
typed row in both modes, counting it toward -limit like any other row. The JSON
row's cursors are typed by hand rather than marshalled with protojson: the
proto3 JSON mapping renders 64-bit integers as strings ("7") while the Rust and
Python CLIs emit numbers (7), so going through protojson would have made Go the
only canonical CLI with a different value type.

Docs in the same change: docs/Sessions.md documents the empty-ring sentinel
value and that oldest - 1 is the universal resume formula in both the retained
and fully-evicted cases; docs/CrossLanguageSmokeMatrix.md gains a per-CLI
gap-rendering table covering both client findings, and records exactly what is
and is not comparable across CLIs (same keys and numeric cursors for Rust/Go/
Python; quoted cursors for .NET/Java; differing key order, whitespace, and
container), so a matrix runner compares parsed values rather than raw bytes.

Tests, all written red first and each reproducing its defect verbatim:
- SessionEventDistributorTests: RegisterWithReplayReportsNextDeliverableSequence
  WhenRingEmptiedByAge, ...WithRetentionDisabled, and
  ResumeUsingSentinelFormulaAfterEmptyRingGapDeliversLiveEvents.
- GatewayEndToEndReconnectReplayTests.ReconnectAfterFullAgeEvictionResumesWith
  SentinelFormula — fake-worker e2e resume walk on a fake clock; the fixture now
  takes a retention window and a TimeProvider.
- clients/python test_stream_events_renders_replay_gap.
- clients/go TestRunStreamEventsPrintsReplayGap.

GWC-25's ReplayGap.oldest_available_sequence proto-comment amendment is
deliberately deferred to the later codegen wave (see the tracker change log): it
is comment-only but triggers the full five-client regen fan-out.
This commit is contained in:
Joseph Doherty
2026-08-07 05:37:27 -04:00
parent ead921cace
commit 44b8e37900
12 changed files with 596 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ZB.MOM.WW.MxGateway.Tests/Gateway/GatewayEndToEndReconnectReplayTests.cs
+129 -6
View File
@@ -2,6 +2,7 @@ using Google.Protobuf.WellKnownTypes;
using Grpc.Core;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using Microsoft.Extensions.Time.Testing;
using ZB.MOM.WW.MxGateway.Contracts;
using ZB.MOM.WW.MxGateway.Contracts.Proto;
using ZB.MOM.WW.MxGateway.Server.Configuration;
@@ -231,6 +232,114 @@ public sealed class GatewayEndToEndReconnectReplayTests
Assert.Equal(expectedTail, tail.Select(e => e.WorkerSequence).ToArray());
}
/// <summary>
/// GWC-25 regression, end to end: when age eviction has emptied the replay ring, the
/// <c>ReplayGap</c> sentinel carries the next deliverable sequence (highest observed + 1),
/// so a client applying the documented
/// <c>after_worker_sequence = oldest_available_sequence - 1</c> formula on its follow-up
/// resume receives subsequent live events. Under the pre-fix sentinel value of <c>0</c>
/// the formula wrapped to <see cref="ulong.MaxValue"/> and the resumed stream never
/// delivered another event.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task ReconnectAfterFullAgeEvictionResumesWithSentinelFormula()
{
const int firstBatch = 4;
const double retentionSeconds = 30;
FakeTimeProvider clock = new(new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero));
GatedEventFakeWorkerProcessLauncher launcher = new();
// Capacity is ample; age is the eviction axis under test.
await using ReconnectReplayGatewayServiceFixture fixture = new(
launcher,
replayBufferCapacity: 16,
replayRetentionSeconds: retentionSeconds,
timeProvider: clock);
string sessionId = await OpenSessionAsync(fixture, "reconnect-aged-out");
using CancellationTokenSource writer1Cts = new();
RecordingServerStreamWriter<MxEvent> writer1 = new();
Task stream1Task = Task.Run(async () =>
await fixture.Service.StreamEvents(
new StreamEventsRequest { SessionId = sessionId },
writer1,
new TestServerCallContext(cancellationToken: writer1Cts.Token)));
await fixture.WaitForSubscriberCountAsync(sessionId, n: 1, TestTimeout);
await WireUpAdviseAsync(fixture, sessionId);
for (int i = 0; i < firstBatch; i++)
{
launcher.AllowNextEvent();
}
IReadOnlyList<MxEvent> batch1 = await writer1.WaitForMessageCountAsync(firstBatch, TestTimeout);
ulong[] batch1Sequences = batch1.Select(e => e.WorkerSequence).ToArray();
// The client's cursor is mid-batch: it detached before consuming the tail, so it is
// genuinely behind the highest sequence the distributor observed.
ulong staleCursor = batch1Sequences[1];
ulong highestSeen = batch1Sequences[firstBatch - 1];
await DetachAsync(writer1Cts, stream1Task);
// Age every retained event out of the ring: the next resume finds it empty.
clock.Advance(TimeSpan.FromSeconds(retentionSeconds * 2));
// ---- first reconnect: receives only the sentinel, nothing is replayable ----
using CancellationTokenSource writer2Cts = new();
RecordingServerStreamWriter<MxEvent> writer2 = new();
Task stream2Task = Task.Run(async () =>
await fixture.Service.StreamEvents(
new StreamEventsRequest { SessionId = sessionId, AfterWorkerSequence = staleCursor },
writer2,
new TestServerCallContext(cancellationToken: writer2Cts.Token)));
await fixture.WaitForSubscriberCountAsync(sessionId, n: 1, TestTimeout);
IReadOnlyList<MxEvent> gapOnly = await writer2.WaitForMessageCountAsync(1, TestTimeout);
MxEvent sentinel = gapOnly[0];
Assert.NotNull(sentinel.ReplayGap);
Assert.Equal(staleCursor, sentinel.ReplayGap.RequestedAfterSequence);
// The whole point of GWC-25: the sentinel anchors the resume at the next sequence that
// can still be delivered, not at 0.
Assert.Equal(highestSeen + 1, sentinel.ReplayGap.OldestAvailableSequence);
await DetachAsync(writer2Cts, stream2Task);
// ---- second reconnect: the client applies the documented oldest - 1 formula ----
// Unchecked ulong arithmetic: with the pre-fix sentinel value of 0 this wraps to
// ulong.MaxValue and the live filter drops every subsequent event.
ulong resumeCursor = sentinel.ReplayGap.OldestAvailableSequence - 1;
RecordingServerStreamWriter<MxEvent> writer3 = new();
Task stream3Task = Task.Run(async () =>
await fixture.Service.StreamEvents(
new StreamEventsRequest { SessionId = sessionId, AfterWorkerSequence = resumeCursor },
writer3,
new TestServerCallContext()));
await fixture.WaitForSubscriberCountAsync(sessionId, n: 1, TestTimeout);
launcher.AllowNextEvent();
IReadOnlyList<MxEvent> live = await writer3.WaitForMessageCountAsync(1, TestTimeout);
launcher.StopEmitting();
await CloseAndDrainAsync(fixture, sessionId, stream3Task, launcher);
// A live event arrives on the resumed stream, and it is a real event, not another gap.
MxEvent delivered = live[0];
Assert.Null(delivered.ReplayGap);
Assert.Equal(MxEventFamily.OnDataChange, delivered.Family);
Assert.True(
delivered.WorkerSequence > highestSeen,
$"Live event {delivered.WorkerSequence} must be newer than the highest previously seen {highestSeen}.");
}
// ---- shared flow helpers ----
private static async Task<string> OpenSessionAsync(
@@ -385,11 +494,23 @@ public sealed class GatewayEndToEndReconnectReplayTests
/// <summary>Initializes a new instance of the <see cref="ReconnectReplayGatewayServiceFixture"/> class.</summary>
/// <param name="launcher">Fake worker process launcher backing the session manager.</param>
/// <param name="replayBufferCapacity">Replay ring capacity for the session's event distributor.</param>
/// <param name="replayRetentionSeconds">
/// Replay retention window. The default keeps age-eviction effectively off for the
/// duration of a fast test; the age-eviction test shortens it and drives
/// <paramref name="timeProvider"/>.
/// </param>
/// <param name="timeProvider">
/// Clock handed to the session manager, which flows through to the session's event
/// distributor. Pass a fake to make age-eviction deterministic.
/// </param>
public ReconnectReplayGatewayServiceFixture(
IWorkerProcessLauncher launcher,
int replayBufferCapacity)
int replayBufferCapacity,
double replayRetentionSeconds = 300,
TimeProvider? timeProvider = null)
{
IOptions<GatewayOptions> options = Options.Create(CreateOptions(replayBufferCapacity));
IOptions<GatewayOptions> options = Options.Create(
CreateOptions(replayBufferCapacity, replayRetentionSeconds));
SessionWorkerClientFactory workerClientFactory = new(
launcher,
options,
@@ -401,6 +522,7 @@ public sealed class GatewayEndToEndReconnectReplayTests
options,
_metrics,
logger: NullLogger<SessionManager>.Instance,
timeProvider: timeProvider,
dashboardEventBroadcaster: NullDashboardEventBroadcaster.Instance);
MxAccessGrpcMapper mapper = new();
EventStreamService eventStreamService = new(
@@ -472,7 +594,7 @@ public sealed class GatewayEndToEndReconnectReplayTests
_metrics.Dispose();
}
private static GatewayOptions CreateOptions(int replayBufferCapacity) =>
private static GatewayOptions CreateOptions(int replayBufferCapacity, double replayRetentionSeconds) =>
new()
{
Worker = new WorkerOptions
@@ -501,9 +623,10 @@ public sealed class GatewayEndToEndReconnectReplayTests
QueueCapacity = 32,
ReplayBufferCapacity = replayBufferCapacity,
// Keep age-eviction effectively off for the duration of a fast test so
// capacity is the only eviction axis under test.
ReplayRetentionSeconds = 300,
// Defaults to a window long enough that age-eviction never fires during a
// fast test, so capacity is the only eviction axis; the age-eviction test
// shortens it and drives a fake clock.
ReplayRetentionSeconds = replayRetentionSeconds,
},
};
}