test(tst-24): drive the .NET and Python clients against real in-process gRPC servers
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m19s
ci / portable (push) Successful in 23m59s
ci / windows-x86 (push) Failing after 58m42s

TST-24 asked for per-client wire tests against a fake gateway. An audit first
corrected the finding's premise: Go, Rust, and Java already had them — bufconn,
a loopback tonic server, and InProcessServerBuilder respectively — each already
asserting the round trip, the server-observed bearer header, and the ReplayGap
sentinel. The two genuine gaps were .NET (every test substituted the transport
interface; the test project had no server package at all) and Python (stub
monkeypatching everywhere but one opt-in TLS test).

Both now serve mxaccess_gateway.v1.MxAccessGateway over a real transport —
Kestrel h2c and grpc.aio, each on an ephemeral loopback port — and drive the
ordinary public client API against it. Only the gateway's behaviour is canned;
the framing, serialization, metadata, and status codes are genuine. Four shapes
each: full round trip with every reply field asserted, the authorization header
as received by the server (including on the streaming RPC), the ReplayGap
sentinel surfaced as the client's typed signal, and a real PERMISSION_DENIED
mapping to the typed authorization error.

The .NET client was only ever compiled in CI, never tested, so the portable job
gains a dotnet test step.

Fixes a bug the new tests caught on their first run: Python's connect() built the
grpc.aio channel inside asyncio.to_thread, and a grpc.aio channel binds to the
event loop current on the constructing thread, so every non-stub connection
raised 'There is no current event loop in thread'. No mock-based test could see
it, and the test guarding the off-loop behaviour patched create_channel and so
asserted the bug. Split resolve_channel_security (blocking TOFU probe, off-loop)
from create_channel (on-loop); the guard tests now assert both halves.
This commit is contained in:
Joseph Doherty
2026-08-10 08:22:25 -04:00
parent d4302c6ac4
commit a8f86b5336
16 changed files with 980 additions and 76 deletions
@@ -457,6 +457,24 @@ If TST-02's interim mitigation (flip retention off) is chosen instead of impleme
## TST-24 — Client wire behaviour has no automated verification `Low` · `—`
> **Resolution 2026-08-10 (branch `feat/tst-24-client-wire-tests`): `Done`.** All five clients now drive their public API against a fake gateway served over a real gRPC transport, in the client's own default suite, and all five run in CI.
>
> **Corrected premise.** The Finding's "no in-process gateway integration tests" was already stale when it was re-verified: **Go, Rust, and Java had real-server wire tests**, not mocks — Go's `newBufconnClient`/`fakeGatewayServer` (`clients/go/mxgateway/client_session_test.go`) over `grpc/test/bufconn`, Rust's `spawn_fake_gateway` (`clients/rust/tests/client_behavior.rs`) over a loopback `TcpListener` with tonic's `Server`, and Java's `InProcessGateway`/`TestGatewayService` (`MxGatewayClientSessionTests.java`) over `InProcessServerBuilder`. Each already asserted the round trip, the `authorization` bearer header as *observed by the server*, and the `ReplayGap` sentinel. The cycle-2 re-verification cited `clients/python/tests/test_replay_gap.py` as evidence for "the other four clients still unit-test against mocks"; that generalized from Python to Go/Rust/Java incorrectly. `InProcessGatewayHarness` (the "template" the Impact paragraph names) is in fact the *thinner* of the Java harnesses — it serves only `streamEvents`/`closeSession` for the CLI tests.
>
> **Real gap, and what was built.** Two clients genuinely had none. **.NET** substituted `FakeGatewayTransport` for `IMxGatewayClientTransport` in every test, so not even the generated stub ran, and its test project had no server package at all. **Python** monkeypatched `MxAccessGatewayStub` everywhere except one opt-in TLS test that served only `OpenSession`. Both now have the pattern:
> - `clients/dotnet/ZB.MOM.WW.MxGateway.Client.Tests/WireFakeGatewayServer.cs` + `MxGatewayClientWireTests.cs` — Kestrel h2c on `127.0.0.1:0` serving `MxAccessGateway.MxAccessGatewayBase`; needed new `Grpc.AspNetCore.Server` + `Microsoft.AspNetCore.App` references on the test project.
> - `clients/python/tests/test_wire_fake_gateway.py` — a `grpc.aio` server on `127.0.0.1:0` serving `MxAccessGatewayServicer`; no new dependencies (`grpcio` is a runtime dep).
>
> Each covers the four shapes the Design asked for: round trip (`OpenSession` → `Invoke`/`Register` → `StreamEvents` → `CloseSession` with every reply field asserted), the bearer header as received by the server on the streaming RPC as well as the unary ones, the `ReplayGap` sentinel surfaced as the client's typed signal (TST-01), and a real `PERMISSION_DENIED` mapping to the typed authorization error.
>
> **CI.** The `portable` job previously only *built* the .NET client; a `dotnet test` step was added, so its wire tests actually run. Go/Rust/Python already ran their suites there and Java in the `java` job.
>
> **Bug this immediately caught** — the justification for the whole finding. `GatewayClient.connect()` / `GalaxyRepositoryClient.connect()` in the Python client were **broken for every real (non-stub) connection**: they built the `grpc.aio` channel inside `asyncio.to_thread`, and a `grpc.aio` channel binds to the event loop current on the constructing thread, so the worker thread raised `RuntimeError: There is no current event loop in thread 'asyncio_0'`. No mock-based test could see it — the one test asserting the off-loop behaviour (`Client.Python-028`) monkeypatched `create_channel` and therefore asserted the bug. Fixed by splitting `resolve_channel_security` (blocking TOFU probe, runs off-loop) from `create_channel` (must run on the loop thread), keeping the Client.Python-028 guarantee; the two `-028` tests were retargeted to assert both halves.
>
> **Deliberately out of scope.** Only the four session RPCs are served — the alarm feed (`StreamAlarms`, `QueryActiveAlarms`, `AcknowledgeAlarm`) and Galaxy browse are not, matching the Design's "full parity is out of scope". Java's `InProcessGatewayHarness` still lacks `openSession`/`invoke`; the client-module tests cover those shapes, so it was left alone.
>
> **Docs.** `docs/GatewayTesting.md` § Client Wire Tests (the cross-client pattern + per-client harness table), `clients/dotnet/README.md`, `clients/python/README.md`.
**Finding.** All five clients have unit tests (13/8/3/13/7 files for dotnet/go/rust/python/java) but no in-process or containerized gateway integration tests; the only cross-language verification is the operator-run `scripts/run-client-e2e-tests.ps1`. `CrossLanguageSmokeMatrixTests` checks shapes only.
**Impact.** Low-to-moderate: a gateway contract change can pass every default suite and break all five clients (partly mitigated by shared-proto codegen). The Java CLI already proves the cheap pattern — `InProcessGatewayHarness` (`stillpending.md` §8).