fix(GWC-25,CLI-35,CLI-36): make the empty-ring ReplayGap resumable end to end

An empty replay ring reported oldest_available_sequence = 0 even when gap was
true. Clients follow the documented after_worker_sequence = oldest - 1 formula,
so an unsigned client computed ulong.MaxValue: the follow-up resume replayed
nothing, reported no gap, and the live filter dropped every subsequent event —
a silently dead stream in the headline detach-and-resume scenario, reachable on
default config once ReplayRetentionSeconds (300) age-evicts the ring.

GWC-25: SessionEventDistributor.RegisterWithReplay's empty-ring branch now
reports _highestSequenceSeen + 1 — the next sequence that can possibly be
delivered — when gap is true, so oldest - 1 lands exactly on the highest
observed sequence and the resume delivers everything newer. Still 0 when there
is no gap, where the field is meaningless and never emitted. Nothing is lost:
the evicted interval was unrecoverable either way, and the sentinel's job is to
say "re-snapshot".

CLI-35: the Python CLI fed every stream item into MessageToDict, which raised on
the ReplayGap dataclass and aborted the command after consuming the stream. A
new _event_row helper renders a gap as {"replayGap": {...}} — the same camelCase
shape the Rust CLI emits — and leaves proto events on the existing path.

CLI-36: the Go CLI formatted result.Event on every row, but the library
deliberately clears Event on a gap, so text mode printed
"0 MX_EVENT_FAMILY_UNSPECIFIED" and JSON mode an empty object, discarding the
resume cursors. The loop now branches on result.IsReplayGap() and renders the
typed row in both modes, counting it toward -limit like any other row. The JSON
row's cursors are typed by hand rather than marshalled with protojson: the
proto3 JSON mapping renders 64-bit integers as strings ("7") while the Rust and
Python CLIs emit numbers (7), so going through protojson would have made Go the
only canonical CLI with a different value type.

Docs in the same change: docs/Sessions.md documents the empty-ring sentinel
value and that oldest - 1 is the universal resume formula in both the retained
and fully-evicted cases; docs/CrossLanguageSmokeMatrix.md gains a per-CLI
gap-rendering table covering both client findings, and records exactly what is
and is not comparable across CLIs (same keys and numeric cursors for Rust/Go/
Python; quoted cursors for .NET/Java; differing key order, whitespace, and
container), so a matrix runner compares parsed values rather than raw bytes.

Tests, all written red first and each reproducing its defect verbatim:
- SessionEventDistributorTests: RegisterWithReplayReportsNextDeliverableSequence
  WhenRingEmptiedByAge, ...WithRetentionDisabled, and
  ResumeUsingSentinelFormulaAfterEmptyRingGapDeliversLiveEvents.
- GatewayEndToEndReconnectReplayTests.ReconnectAfterFullAgeEvictionResumesWith
  SentinelFormula — fake-worker e2e resume walk on a fake clock; the fixture now
  takes a retention window and a TimeProvider.
- clients/python test_stream_events_renders_replay_gap.
- clients/go TestRunStreamEventsPrintsReplayGap.

GWC-25's ReplayGap.oldest_available_sequence proto-comment amendment is
deliberately deferred to the later codegen wave (see the tracker change log): it
is comment-only but triggers the full five-client regen fan-out.
This commit is contained in:
Joseph Doherty
2026-08-07 05:37:27 -04:00
parent ead921cace
commit 44b8e37900
12 changed files with 596 additions and 28 deletions
+3 -1
View File
@@ -225,12 +225,14 @@ The handoff is sealed by a watermark. `RegisterWithReplay` returns `LiveResumeSe
Emit order on a resumed stream:
1. **ReplayGap sentinel (only when events were evicted).** If the requested `after_worker_sequence` predates the oldest event still retained — i.e. events in the open interval were dropped by capacity or age eviction and are unrecoverable — the gateway first yields a single sentinel `MxEvent` with `replay_gap` populated (`requested_after_sequence` = the requested watermark, `oldest_available_sequence` = the oldest still-retained sequence). The sentinel carries the session id; its `family` is `UNSPECIFIED`, its `body` oneof is unset, and no per-item fields are populated. It is an explicit, documented control signal — *not* a synthesized MXAccess event — telling the client to discard local state and re-snapshot. A client that wants to resume without another gap should set `after_worker_sequence = oldest_available_sequence - 1` on its next request.
1. **ReplayGap sentinel (only when events were evicted).** If the requested `after_worker_sequence` predates the oldest event still retained — i.e. events in the open interval were dropped by capacity or age eviction and are unrecoverable — the gateway first yields a single sentinel `MxEvent` with `replay_gap` populated (`requested_after_sequence` = the requested watermark, `oldest_available_sequence` = the resume anchor described below). The sentinel carries the session id; its `family` is `UNSPECIFIED`, its `body` oneof is unset, and no per-item fields are populated. It is an explicit, documented control signal — *not* a synthesized MXAccess event — telling the client to discard local state and re-snapshot. A client that wants to resume without another gap should set `after_worker_sequence = oldest_available_sequence - 1` on its next request.
2. **Retained replay batch.** The still-retained events newer than the requested watermark, in ascending `worker_sequence` order.
3. **Live events**, resuming strictly after `LiveResumeSequence`.
When `after_worker_sequence` is inside the retained window (nothing was evicted), step 1 is skipped: the stream replays the retained tail then resumes live with no sentinel.
**`oldest_available_sequence` when the ring is empty.** Age eviction (`ReplayRetentionSeconds`, default 300) and a disabled ring (`ReplayBufferCapacity = 0`) both leave nothing retained, so there is no oldest-retained sequence to report. In that case the sentinel carries the **next sequence that can possibly be delivered** — the highest sequence the distributor has observed plus one — rather than `0`. That keeps `after_worker_sequence = oldest_available_sequence - 1` the single universal resume formula: the follow-up resume lands exactly on the highest observed sequence, replays nothing, reports no gap, and receives every subsequent live event. Reporting `0` here would make an unsigned client compute `2^64 - 1` and then silently receive nothing, because the live filter drops every event at or below that watermark. Nothing is lost relative to reporting `0` — the evicted interval is unrecoverable either way, and the sentinel's job is to tell the client to re-snapshot. `0` remains the value when there is no gap, where the field is meaningless and never emitted.
The ReplayGap sentinel is emitted **only** on the `StreamEvents` server stream and only to the resuming subscriber — it is never fanned to other subscribers and never appears in `DrainEventsReply` (the diagnostic drain path is untouched). Replay retention itself is bounded by `MxGateway:Events:ReplayBufferCapacity` (count) and `ReplayRetentionSeconds` (age); see [Configuration](GatewayConfiguration.md).
### Close