Joseph Doherty
44b8e37900
An empty replay ring reported oldest_available_sequence = 0 even when gap was
true. Clients follow the documented after_worker_sequence = oldest - 1 formula,
so an unsigned client computed ulong.MaxValue: the follow-up resume replayed
nothing, reported no gap, and the live filter dropped every subsequent event —
a silently dead stream in the headline detach-and-resume scenario, reachable on
default config once ReplayRetentionSeconds (300) age-evicts the ring.
GWC-25: SessionEventDistributor.RegisterWithReplay's empty-ring branch now
reports _highestSequenceSeen + 1 — the next sequence that can possibly be
delivered — when gap is true, so oldest - 1 lands exactly on the highest
observed sequence and the resume delivers everything newer. Still 0 when there
is no gap, where the field is meaningless and never emitted. Nothing is lost:
the evicted interval was unrecoverable either way, and the sentinel's job is to
say "re-snapshot".
CLI-35: the Python CLI fed every stream item into MessageToDict, which raised on
the ReplayGap dataclass and aborted the command after consuming the stream. A
new _event_row helper renders a gap as {"replayGap": {...}} — the same camelCase
shape the Rust CLI emits — and leaves proto events on the existing path.
CLI-36: the Go CLI formatted result.Event on every row, but the library
deliberately clears Event on a gap, so text mode printed
"0 MX_EVENT_FAMILY_UNSPECIFIED" and JSON mode an empty object, discarding the
resume cursors. The loop now branches on result.IsReplayGap() and renders the
typed row in both modes, counting it toward -limit like any other row. The JSON
row's cursors are typed by hand rather than marshalled with protojson: the
proto3 JSON mapping renders 64-bit integers as strings ("7") while the Rust and
Python CLIs emit numbers (7), so going through protojson would have made Go the
only canonical CLI with a different value type.
Docs in the same change: docs/Sessions.md documents the empty-ring sentinel
value and that oldest - 1 is the universal resume formula in both the retained
and fully-evicted cases; docs/CrossLanguageSmokeMatrix.md gains a per-CLI
gap-rendering table covering both client findings, and records exactly what is
and is not comparable across CLIs (same keys and numeric cursors for Rust/Go/
Python; quoted cursors for .NET/Java; differing key order, whitespace, and
container), so a matrix runner compares parsed values rather than raw bytes.
Tests, all written red first and each reproducing its defect verbatim:
- SessionEventDistributorTests: RegisterWithReplayReportsNextDeliverableSequence
WhenRingEmptiedByAge, ...WithRetentionDisabled, and
ResumeUsingSentinelFormulaAfterEmptyRingGapDeliversLiveEvents.
- GatewayEndToEndReconnectReplayTests.ReconnectAfterFullAgeEvictionResumesWith
SentinelFormula — fake-worker e2e resume walk on a fake clock; the fixture now
takes a retention window and a TimeProvider.
- clients/python test_stream_events_renders_replay_gap.
- clients/go TestRunStreamEventsPrintsReplayGap.
GWC-25's ReplayGap.oldest_available_sequence proto-comment amendment is
deliberately deferred to the later codegen wave (see the tracker change log): it
is comment-only but triggers the full five-client regen fan-out.
6.6 KiB
Cross-Language Smoke Matrix
The cross-language smoke matrix defines the documented commands used to compare official clients against the same live gateway flow. It is a repository validation fixture and command reference; normal unit tests validate the matrix shape without connecting to a gateway.
The matrix lives in
clients/proto/fixtures/smoke/cross-language-smoke-matrix.json.
Scope
The matrix covers the supported client languages:
- .NET
- Go
- Rust
- Python
- Java
Each client entry defines commands for the same required operation sequence:
open-sessionregisteradd-itemadvisestream-eventsclose-session
The optional write command is documented separately because writing changes
provider state and should only run when the operator supplies a safe test value.
When stream-events is resumed with an after_worker_sequence cursor that
predates the oldest event still in the gateway's replay ring, the gateway emits a
single ReplayGap sentinel at the head of the stream. Every client surfaces this
as a distinct, typed, non-terminal signal (see each client README); the resume
contract is after_worker_sequence = oldest_available_sequence - 1, and it holds
even when the ring has been emptied entirely by age eviction — the gateway then
reports the next deliverable sequence rather than 0 (see Sessions).
The default smoke sequence opens a fresh stream (no cursor) and does not exercise
the gap path; a resume-with-gap fixture case is tracked separately (TST-24).
The CLIs differ in how they print that library-level signal. Three of them consume
the typed gap and emit a dedicated row rather than a degenerate event row; the other
two hand the raw sentinel MxEvent straight to the formatter, so they print the
sentinel itself, whose replayGap field carries the same cursors:
| CLI | Text mode | JSON mode |
|---|---|---|
mxgw-rs (Rust, canonical) |
REPLAY_GAP requested_after=<n> oldest_available=<n> |
{"replayGap": {"requestedAfterSequence": <n>, "oldestAvailableSequence": <n>}} as one entry of the events array |
mxgw-go (Go) |
REPLAY_GAP requested_after=<n> oldest_available=<n> |
one {"replayGap": {"requestedAfterSequence": <n>, "oldestAvailableSequence": <n>}} line, counted toward -limit like any other row |
mxgw-py (Python) |
same JSON dump as --json |
{"replayGap": {"requestedAfterSequence": <n>, "oldestAvailableSequence": <n>}} as one entry of the events array |
mxgw-dotnet (.NET) |
the raw sentinel MxEvent as protobuf JSON, including its replayGap field |
same, as one entry of the events array |
mxgw-java (Java) |
the sentinel's worker_sequence and family (0 MX_EVENT_FAMILY_UNSPECIFIED) |
the raw sentinel MxEvent as protobuf JSON, including its replayGap field |
Rust, Go, and Python emit the same two key names and, deliberately, the same JSON
value types: the cursors are JSON numbers (7), not strings. That is why the
Go CLI types the row by hand instead of marshalling ReplayGap with protojson —
the proto3 JSON mapping renders 64-bit integers as strings ("7"), which is also
why the .NET and Java rows, which pass the sentinel through a protobuf JSON
formatter, carry quoted cursors. A matrix runner must therefore compare parsed
values, not raw bytes, and must not assume the same value type across all five
CLIs.
Two further formatting differences among the three canonical CLIs, none of them
semantic: Python sorts object keys and uses ", " / ": " separators
(json.dumps(..., sort_keys=True)), while Rust and Go emit compact,
declaration-ordered JSON; and the row sits alone on its own line for Go and for
Rust's --jsonl, but inside an events array for Python and for Rust's
aggregate --json.
Integration Gate
Cross-language smoke execution is opt-in. Runners should skip the matrix unless this variable is set:
$env:MXGATEWAY_INTEGRATION = "1"
The shared inputs are:
| Variable | Default | Purpose |
|---|---|---|
MXGATEWAY_ENDPOINT |
localhost:5000 |
Gateway endpoint used by client CLIs. |
MXGATEWAY_API_KEY |
Empty | API key source for authenticated gateway deployments. |
MXGATEWAY_TEST_ITEM |
TestChildObject.TestInt |
MXAccess item used by add-item. |
MXGATEWAY_TEST_WRITE_VALUE |
Empty | Enables the optional write step when set by a runner. |
The commands in the matrix use MXGATEWAY_API_KEY through each CLI's
api-key-env flag. They must not embed bearer tokens or raw API keys.
TLS variant
The matrix runs over plaintext (h2c) by default. A TLS variant exists but stays
a manual/opt-in run, consistent with the gate above, because it needs the gateway
started with an HTTPS endpoint (an https:// MXGATEWAY_ENDPOINT) and each CLI
switched to its TLS flag (--tls / -tls / --plaintext=false /
plaintext=False). The clients are lenient by default and accept the gateway's
auto-generated self-signed certificate without extra trust setup, except the Rust
CLI, which is pin-only and needs --ca-file or --require-certificate-validation
(and Python uses trust-on-first-use). See
Gateway Configuration — Automatic self-signed certificate
and each client README for the per-client TLS flags.
JSON Comparison
Every command in the matrix requests JSON output. A runner can compare the normalized smoke record across languages with these fields:
- language,
- operation,
- session id,
- server handle,
- item handle,
- event count,
- event family,
- worker sequence,
- protocol status,
- HRESULT,
- status arrays,
- close status.
Failure output must include the client language, endpoint, and redacted auth
context. Auth context identifies the source, such as MXGATEWAY_API_KEY, but
does not include the secret value.
Bundled Smoke Commands
Each client also exposes a bundled smoke command. Those commands are useful
for quick local checks, but the full cross-language matrix uses explicit
operation commands because not every bundled smoke command streams events yet.
The explicit sequence remains the parity baseline for issue-level validation.
Validation
Run the matrix shape tests after changing the smoke matrix:
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter FullyQualifiedName~CrossLanguageSmokeMatrixTests
Live execution remains a separate opt-in step because it depends on a running gateway, the installed MXAccess worker path, and provider state.