Merge branch 'fix/gwc-25-replaygap-trio'
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m10s
ci / windows-x86 (push) Successful in 1m30s
ci / portable (push) Successful in 7m39s

# Conflicts:
#	archreview/2026-07-12/remediation/00-tracking.md
#	archreview/2026-07-12/remediation/10-gateway-core.md
This commit is contained in:
Joseph Doherty
2026-08-07 05:49:13 -04:00
12 changed files with 596 additions and 28 deletions
+34 -3
View File
@@ -34,9 +34,40 @@ When `stream-events` is resumed with an `after_worker_sequence` cursor that
predates the oldest event still in the gateway's replay ring, the gateway emits a
single `ReplayGap` sentinel at the head of the stream. Every client surfaces this
as a distinct, typed, non-terminal signal (see each client README); the resume
contract is `after_worker_sequence = oldest_available_sequence - 1`. The default
smoke sequence opens a fresh stream (no cursor) and does not exercise the gap
path; a resume-with-gap fixture case is tracked separately (TST-24).
contract is `after_worker_sequence = oldest_available_sequence - 1`, and it holds
even when the ring has been emptied entirely by age eviction — the gateway then
reports the next deliverable sequence rather than `0` (see [Sessions](Sessions.md)).
The default smoke sequence opens a fresh stream (no cursor) and does not exercise
the gap path; a resume-with-gap fixture case is tracked separately (TST-24).
The CLIs differ in how they *print* that library-level signal. Three of them consume
the typed gap and emit a dedicated row rather than a degenerate event row; the other
two hand the raw sentinel `MxEvent` straight to the formatter, so they print the
sentinel itself, whose `replayGap` field carries the same cursors:
| CLI | Text mode | JSON mode |
|-----|-----------|-----------|
| `mxgw-rs` (Rust, canonical) | `REPLAY_GAP requested_after=<n> oldest_available=<n>` | `{"replayGap": {"requestedAfterSequence": <n>, "oldestAvailableSequence": <n>}}` as one entry of the `events` array |
| `mxgw-go` (Go) | `REPLAY_GAP requested_after=<n> oldest_available=<n>` | one `{"replayGap": {"requestedAfterSequence": <n>, "oldestAvailableSequence": <n>}}` line, counted toward `-limit` like any other row |
| `mxgw-py` (Python) | same JSON dump as `--json` | `{"replayGap": {"requestedAfterSequence": <n>, "oldestAvailableSequence": <n>}}` as one entry of the `events` array |
| `mxgw-dotnet` (.NET) | the raw sentinel `MxEvent` as protobuf JSON, including its `replayGap` field | same, as one entry of the `events` array |
| `mxgw-java` (Java) | the sentinel's `worker_sequence` and `family` (`0 MX_EVENT_FAMILY_UNSPECIFIED`) | the raw sentinel `MxEvent` as protobuf JSON, including its `replayGap` field |
Rust, Go, and Python emit the same two key names and, deliberately, the same JSON
value **types**: the cursors are JSON numbers (`7`), not strings. That is why the
Go CLI types the row by hand instead of marshalling `ReplayGap` with `protojson`
the proto3 JSON mapping renders 64-bit integers as strings (`"7"`), which is also
why the .NET and Java rows, which pass the sentinel through a protobuf JSON
formatter, carry **quoted** cursors. A matrix runner must therefore compare parsed
values, not raw bytes, and must not assume the same value type across all five
CLIs.
Two further formatting differences among the three canonical CLIs, none of them
semantic: Python sorts object keys and uses `", "` / `": "` separators
(`json.dumps(..., sort_keys=True)`), while Rust and Go emit compact,
declaration-ordered JSON; and the row sits alone on its own line for Go and for
Rust's `--jsonl`, but inside an `events` array for Python and for Rust's
aggregate `--json`.
## Integration Gate
+3 -1
View File
@@ -225,12 +225,14 @@ The handoff is sealed by a watermark. `RegisterWithReplay` returns `LiveResumeSe
Emit order on a resumed stream:
1. **ReplayGap sentinel (only when events were evicted).** If the requested `after_worker_sequence` predates the oldest event still retained — i.e. events in the open interval were dropped by capacity or age eviction and are unrecoverable — the gateway first yields a single sentinel `MxEvent` with `replay_gap` populated (`requested_after_sequence` = the requested watermark, `oldest_available_sequence` = the oldest still-retained sequence). The sentinel carries the session id; its `family` is `UNSPECIFIED`, its `body` oneof is unset, and no per-item fields are populated. It is an explicit, documented control signal — *not* a synthesized MXAccess event — telling the client to discard local state and re-snapshot. A client that wants to resume without another gap should set `after_worker_sequence = oldest_available_sequence - 1` on its next request.
1. **ReplayGap sentinel (only when events were evicted).** If the requested `after_worker_sequence` predates the oldest event still retained — i.e. events in the open interval were dropped by capacity or age eviction and are unrecoverable — the gateway first yields a single sentinel `MxEvent` with `replay_gap` populated (`requested_after_sequence` = the requested watermark, `oldest_available_sequence` = the resume anchor described below). The sentinel carries the session id; its `family` is `UNSPECIFIED`, its `body` oneof is unset, and no per-item fields are populated. It is an explicit, documented control signal — *not* a synthesized MXAccess event — telling the client to discard local state and re-snapshot. A client that wants to resume without another gap should set `after_worker_sequence = oldest_available_sequence - 1` on its next request.
2. **Retained replay batch.** The still-retained events newer than the requested watermark, in ascending `worker_sequence` order.
3. **Live events**, resuming strictly after `LiveResumeSequence`.
When `after_worker_sequence` is inside the retained window (nothing was evicted), step 1 is skipped: the stream replays the retained tail then resumes live with no sentinel.
**`oldest_available_sequence` when the ring is empty.** Age eviction (`ReplayRetentionSeconds`, default 300) and a disabled ring (`ReplayBufferCapacity = 0`) both leave nothing retained, so there is no oldest-retained sequence to report. In that case the sentinel carries the **next sequence that can possibly be delivered** — the highest sequence the distributor has observed plus one — rather than `0`. That keeps `after_worker_sequence = oldest_available_sequence - 1` the single universal resume formula: the follow-up resume lands exactly on the highest observed sequence, replays nothing, reports no gap, and receives every subsequent live event. Reporting `0` here would make an unsigned client compute `2^64 - 1` and then silently receive nothing, because the live filter drops every event at or below that watermark. Nothing is lost relative to reporting `0` — the evicted interval is unrecoverable either way, and the sentinel's job is to tell the client to re-snapshot. `0` remains the value when there is no gap, where the field is meaningless and never emitted.
The ReplayGap sentinel is emitted **only** on the `StreamEvents` server stream and only to the resuming subscriber — it is never fanned to other subscribers and never appears in `DrainEventsReply` (the diagnostic drain path is untouched). Replay retention itself is bounded by `MxGateway:Events:ReplayBufferCapacity` (count) and `ReplayRetentionSeconds` (age); see [Configuration](GatewayConfiguration.md).
### Close