test(tst-24): drive the .NET and Python clients against real in-process gRPC servers
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m19s
ci / portable (push) Successful in 23m59s
ci / windows-x86 (push) Failing after 58m42s

TST-24 asked for per-client wire tests against a fake gateway. An audit first
corrected the finding's premise: Go, Rust, and Java already had them — bufconn,
a loopback tonic server, and InProcessServerBuilder respectively — each already
asserting the round trip, the server-observed bearer header, and the ReplayGap
sentinel. The two genuine gaps were .NET (every test substituted the transport
interface; the test project had no server package at all) and Python (stub
monkeypatching everywhere but one opt-in TLS test).

Both now serve mxaccess_gateway.v1.MxAccessGateway over a real transport —
Kestrel h2c and grpc.aio, each on an ephemeral loopback port — and drive the
ordinary public client API against it. Only the gateway's behaviour is canned;
the framing, serialization, metadata, and status codes are genuine. Four shapes
each: full round trip with every reply field asserted, the authorization header
as received by the server (including on the streaming RPC), the ReplayGap
sentinel surfaced as the client's typed signal, and a real PERMISSION_DENIED
mapping to the typed authorization error.

The .NET client was only ever compiled in CI, never tested, so the portable job
gains a dotnet test step.

Fixes a bug the new tests caught on their first run: Python's connect() built the
grpc.aio channel inside asyncio.to_thread, and a grpc.aio channel binds to the
event loop current on the constructing thread, so every non-stub connection
raised 'There is no current event loop in thread'. No mock-based test could see
it, and the test guarding the off-loop behaviour patched create_channel and so
asserted the bug. Split resolve_channel_security (blocking TOFU probe, off-loop)
from create_channel (on-loop); the guard tests now assert both halves.
This commit is contained in:
Joseph Doherty
2026-08-10 08:22:25 -04:00
parent d4302c6ac4
commit a8f86b5336
16 changed files with 980 additions and 76 deletions
+14
View File
@@ -23,6 +23,20 @@ dotnet build clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx
dotnet test clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx --no-build
```
Most tests substitute `FakeGatewayTransport` for `IMxGatewayClientTransport`, so
they never touch the wire. `MxGatewayClientWireTests` is the exception: it drives
the ordinary public API against `WireFakeGatewayServer`, a real gRPC server
(Kestrel h2c on an ephemeral loopback port) serving
`MxAccessGateway.MxAccessGatewayBase`. Only the gateway's behaviour is canned —
the HTTP/2 framing, protobuf serialization, `authorization` metadata, and gRPC
status codes are genuine, so it catches decode and metadata breaks a transport
fake cannot see. No MXAccess or worker is involved; it runs in the default suite.
See `docs/GatewayTesting.md` (Client Wire Tests) for the cross-client pattern.
```powershell
dotnet test clients/dotnet/ZB.MOM.WW.MxGateway.Client.Tests/ZB.MOM.WW.MxGateway.Client.Tests.csproj --filter FullyQualifiedName~MxGatewayClientWireTests
```
## Packaging
Create local library and CLI artifacts from the repository root:
@@ -0,0 +1,162 @@
using ZB.MOM.WW.MxGateway.Contracts.Proto;
namespace ZB.MOM.WW.MxGateway.Client.Tests;
/// <summary>
/// Drives the public client API against <see cref="WireFakeGatewayServer"/> — a real
/// gRPC server on loopback — so the transport, protobuf serialization, call metadata,
/// and gRPC status mapping are all exercised. Every other test in this project
/// substitutes <see cref="FakeGatewayTransport"/> and therefore proves nothing about
/// what actually crosses the wire.
/// </summary>
public sealed class MxGatewayClientWireTests
{
private const string ApiKey = "mxgw_wiretest_secret";
/// <summary>
/// Verifies the full session happy path decodes real wire bytes end to end.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task SessionRoundTrip_OverRealTransport_DecodesEveryReplyField()
{
await using WireFakeGatewayServer server = await WireFakeGatewayServer.StartAsync();
await using MxGatewayClient client = server.CreateClient(ApiKey);
MxGatewaySession session = await client.OpenSessionAsync(
new OpenSessionRequest { ClientSessionName = "wire-test" });
Assert.Equal(WireFakeGatewayServer.FakeGatewayService.SessionId, session.SessionId);
Assert.Equal("fake-backend", session.OpenSessionReply.BackendName);
Assert.Equal(1234, session.OpenSessionReply.WorkerProcessId);
Assert.Equal(3u, session.OpenSessionReply.GatewayProtocolVersion);
Assert.Equal(["events", "invoke"], session.OpenSessionReply.Capabilities);
int serverHandle = await session.RegisterAsync("wire-test-client");
Assert.Equal(WireFakeGatewayServer.FakeGatewayService.ServerHandle, serverHandle);
MxCommandRequest? invoke = server.Service.InvokeRequest;
Assert.NotNull(invoke);
Assert.Equal(MxCommandKind.Register, invoke.Command.Kind);
Assert.Equal("wire-test-client", invoke.Command.Register.ClientName);
List<MxEvent> events = await CollectAsync(client.StreamEventsAsync(
new StreamEventsRequest { SessionId = session.SessionId }));
MxEvent single = Assert.Single(events);
Assert.Equal(MxEventFamily.OnDataChange, single.Family);
Assert.Equal(WireFakeGatewayServer.FakeGatewayService.ServerHandle, single.ServerHandle);
Assert.Equal(WireFakeGatewayServer.FakeGatewayService.ItemHandle, single.ItemHandle);
Assert.Equal(17, single.Value.Int32Value);
Assert.Equal(192, single.Quality);
Assert.Equal(9ul, single.WorkerSequence);
Assert.Equal(MxEvent.BodyOneofCase.OnDataChange, single.BodyCase);
CloseSessionReply closeReply = await session.CloseAsync();
Assert.Equal(SessionState.Closed, closeReply.FinalState);
Assert.Equal(
WireFakeGatewayServer.FakeGatewayService.SessionId,
server.Service.CloseSessionRequest?.SessionId);
}
/// <summary>
/// Verifies the API key reaches the server as a bearer header on unary and
/// streaming calls alike. A transport fake can only assert what the client passes;
/// this asserts what the server receives.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task ApiKey_ReachesTheServerAsBearerMetadata_OnEveryRpc()
{
await using WireFakeGatewayServer server = await WireFakeGatewayServer.StartAsync();
await using MxGatewayClient client = server.CreateClient(ApiKey);
MxGatewaySession session = await client.OpenSessionAsync(
new OpenSessionRequest { ClientSessionName = "wire-test" });
await session.RegisterAsync("wire-test-client");
await CollectAsync(client.StreamEventsAsync(
new StreamEventsRequest { SessionId = session.SessionId }));
await session.CloseAsync();
string expected = $"Bearer {ApiKey}";
Assert.Equal(
new Dictionary<string, string>
{
["OpenSession"] = expected,
["Invoke"] = expected,
["StreamEvents"] = expected,
["CloseSession"] = expected,
},
server.Service.AuthorizationByMethod);
}
/// <summary>
/// Verifies the gateway's replay-gap sentinel survives serialization and is
/// surfaced as a typed, non-terminal stream item.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task ReplayGapSentinel_SurvivesTheWire_AsTypedStreamItem()
{
await using WireFakeGatewayServer server = await WireFakeGatewayServer.StartAsync(service =>
service.ReplayGap = new ReplayGap
{
RequestedAfterSequence = 3,
OldestAvailableSequence = 8,
});
await using MxGatewayClient client = server.CreateClient(ApiKey);
MxGatewaySession session = await client.OpenSessionAsync(
new OpenSessionRequest { ClientSessionName = "wire-test" });
List<MxEventStreamItem> items = [];
IAsyncEnumerable<MxEvent> stream = client.StreamEventsAsync(new StreamEventsRequest
{
SessionId = session.SessionId,
AfterWorkerSequence = 3,
});
await foreach (MxEventStreamItem item in stream.AsStreamItemsAsync())
{
items.Add(item);
}
Assert.Equal(2, items.Count);
Assert.True(items[0].IsReplayGap);
Assert.Equal(3ul, items[0].ReplayGap!.RequestedAfterSequence);
Assert.Equal(8ul, items[0].ReplayGap!.OldestAvailableSequence);
Assert.False(items[1].IsReplayGap);
Assert.Equal(MxEventFamily.OnDataChange, items[1].Event.Family);
Assert.Equal(3ul, server.Service.StreamEventsRequest?.AfterWorkerSequence);
}
/// <summary>
/// Verifies a genuine <c>PERMISSION_DENIED</c> status maps to the typed client
/// exception rather than a bare RpcException.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task PermissionDeniedStatus_MapsToAuthorizationException()
{
await using WireFakeGatewayServer server = await WireFakeGatewayServer.StartAsync(
service => service.DenyInvoke = true);
await using MxGatewayClient client = server.CreateClient(ApiKey);
MxGatewaySession session = await client.OpenSessionAsync(
new OpenSessionRequest { ClientSessionName = "wire-test" });
await Assert.ThrowsAsync<MxGatewayAuthorizationException>(
() => session.RegisterAsync("wire-test-client"));
}
private static async Task<List<MxEvent>> CollectAsync(IAsyncEnumerable<MxEvent> stream)
{
List<MxEvent> events = [];
await foreach (MxEvent gatewayEvent in stream)
{
events.Add(gatewayEvent);
}
return events;
}
}
@@ -0,0 +1,264 @@
using System.Collections.Concurrent;
using System.Net;
using Grpc.Core;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Hosting.Server;
using Microsoft.AspNetCore.Hosting.Server.Features;
using Microsoft.AspNetCore.Server.Kestrel.Core;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.MxGateway.Contracts.Proto;
namespace ZB.MOM.WW.MxGateway.Client.Tests;
/// <summary>
/// Hosts the real <c>mxaccess_gateway.v1.MxAccessGateway</c> service on a loopback
/// Kestrel endpoint so client tests exercise genuine HTTP/2 framing, protobuf
/// serialization, call metadata, and gRPC status propagation.
/// </summary>
/// <remarks>
/// <para>
/// This is the counterpart of <see cref="FakeGatewayTransport"/>: that fake replaces
/// <c>IMxGatewayClientTransport</c>, so nothing below the client wrapper runs. This one
/// replaces only the gateway's <em>behaviour</em> — every byte between the client and
/// the service is the real wire format. Contract breaks that a transport fake cannot
/// see (a field the client never decodes, metadata it does not actually send, a status
/// code it maps differently once it arrives as a real <see cref="RpcException"/>) fail
/// here.
/// </para>
/// <para>
/// Plaintext h2c is used deliberately: TLS is covered by
/// <c>MxGatewayClientTlsHandlerTests</c>, and h2c keeps the harness certificate-free so
/// it runs identically on every CI host. See <c>docs/GatewayTesting.md</c>
/// (Client Wire Tests) for the shared pattern and its Python counterpart.
/// </para>
/// </remarks>
internal sealed class WireFakeGatewayServer : IAsyncDisposable
{
private readonly WebApplication _app;
private WireFakeGatewayServer(WebApplication app, FakeGatewayService service, int port)
{
_app = app;
Service = service;
Endpoint = new Uri($"http://127.0.0.1:{port}");
}
/// <summary>
/// Gets the canned service backing the endpoint; tests read its recorded requests.
/// </summary>
public FakeGatewayService Service { get; }
/// <summary>
/// Gets the h2c endpoint to point <see cref="MxGatewayClientOptions.Endpoint"/> at.
/// </summary>
public Uri Endpoint { get; }
/// <summary>
/// Starts a server on an ephemeral loopback port.
/// </summary>
/// <param name="configure">Optional configuration of the canned service.</param>
/// <returns>The started server.</returns>
public static async Task<WireFakeGatewayServer> StartAsync(Action<FakeGatewayService>? configure = null)
{
FakeGatewayService service = new();
configure?.Invoke(service);
WebApplicationBuilder builder = WebApplication.CreateBuilder();
builder.Logging.ClearProviders();
builder.WebHost.ConfigureKestrel(options =>
// Port 0 lets the OS pick; HTTP/2 without TLS (h2c) is what the client's
// plain http:// endpoint negotiates via RequestVersionExact.
options.Listen(IPAddress.Loopback, 0, listen => listen.Protocols = HttpProtocols.Http2));
builder.Services.AddGrpc();
builder.Services.AddSingleton(service);
WebApplication app = builder.Build();
app.MapGrpcService<FakeGatewayService>();
await app.StartAsync().ConfigureAwait(false);
return new WireFakeGatewayServer(app, service, ResolvePort(app));
}
/// <summary>
/// Creates a client bound to this server's endpoint.
/// </summary>
/// <param name="apiKey">API key the client should present.</param>
/// <returns>A client that talks to this server over h2c.</returns>
public MxGatewayClient CreateClient(string apiKey) =>
MxGatewayClient.Create(new MxGatewayClientOptions
{
Endpoint = Endpoint,
ApiKey = apiKey,
UseTls = false,
DefaultCallTimeout = TimeSpan.FromSeconds(30),
});
/// <inheritdoc />
public async ValueTask DisposeAsync()
{
await _app.StopAsync().ConfigureAwait(false);
await _app.DisposeAsync().ConfigureAwait(false);
}
private static int ResolvePort(WebApplication app)
{
IServerAddressesFeature? addresses = app.Services
.GetRequiredService<IServer>()
.Features
.Get<IServerAddressesFeature>();
string address = addresses?.Addresses.FirstOrDefault()
?? throw new InvalidOperationException("Kestrel did not report a bound address.");
return new Uri(address).Port;
}
/// <summary>
/// Canned gateway answering the four session RPCs with gateway-shaped replies.
/// </summary>
internal sealed class FakeGatewayService : MxAccessGateway.MxAccessGatewayBase
{
/// <summary>The session id every reply carries.</summary>
public const string SessionId = "wire-session-1";
/// <summary>The server handle the canned Register reply returns.</summary>
public const int ServerHandle = 4242;
/// <summary>The item handle the canned data-change event carries.</summary>
public const int ItemHandle = 77;
/// <summary>
/// Gets the <c>authorization</c> header value observed per RPC name.
/// </summary>
public ConcurrentDictionary<string, string> AuthorizationByMethod { get; } = new();
/// <summary>
/// Gets or sets a value indicating whether <c>Invoke</c> fails with
/// <see cref="StatusCode.PermissionDenied"/> instead of replying.
/// </summary>
public bool DenyInvoke { get; set; }
/// <summary>
/// Gets or sets the replay-gap sentinel emitted at the head of the event stream.
/// </summary>
public ReplayGap? ReplayGap { get; set; }
/// <summary>
/// Gets the last <c>Invoke</c> request the client sent, as decoded from the wire.
/// </summary>
public MxCommandRequest? InvokeRequest { get; private set; }
/// <summary>
/// Gets the last <c>StreamEvents</c> request the client sent.
/// </summary>
public StreamEventsRequest? StreamEventsRequest { get; private set; }
/// <summary>
/// Gets the last <c>CloseSession</c> request the client sent.
/// </summary>
public CloseSessionRequest? CloseSessionRequest { get; private set; }
/// <inheritdoc />
public override Task<OpenSessionReply> OpenSession(
OpenSessionRequest request,
ServerCallContext context)
{
Record(context);
return Task.FromResult(new OpenSessionReply
{
SessionId = SessionId,
BackendName = "fake-backend",
WorkerProcessId = 1234,
WorkerProtocolVersion = 1,
GatewayProtocolVersion = 3,
Capabilities = { "events", "invoke" },
ProtocolStatus = Ok(),
});
}
/// <inheritdoc />
public override Task<MxCommandReply> Invoke(MxCommandRequest request, ServerCallContext context)
{
Record(context);
InvokeRequest = request;
if (DenyInvoke)
{
throw new RpcException(new Status(StatusCode.PermissionDenied, "invoke scope required"));
}
return Task.FromResult(new MxCommandReply
{
SessionId = request.SessionId,
CorrelationId = request.ClientCorrelationId,
Kind = request.Command.Kind,
ProtocolStatus = Ok(),
Hresult = 0,
Register = new RegisterReply { ServerHandle = ServerHandle },
});
}
/// <inheritdoc />
public override async Task StreamEvents(
StreamEventsRequest request,
IServerStreamWriter<MxEvent> responseStream,
ServerCallContext context)
{
Record(context);
StreamEventsRequest = request;
if (ReplayGap is not null)
{
// The sentinel shape the gateway emits: family unspecified, body unset,
// only replay_gap populated.
await responseStream.WriteAsync(new MxEvent
{
SessionId = request.SessionId,
ReplayGap = ReplayGap,
}).ConfigureAwait(false);
}
await responseStream.WriteAsync(new MxEvent
{
SessionId = request.SessionId,
Family = MxEventFamily.OnDataChange,
ServerHandle = ServerHandle,
ItemHandle = ItemHandle,
Value = new MxValue { Int32Value = 17 },
Quality = 192,
WorkerSequence = 9,
OnDataChange = new OnDataChangeEvent(),
}).ConfigureAwait(false);
}
/// <inheritdoc />
public override Task<CloseSessionReply> CloseSession(
CloseSessionRequest request,
ServerCallContext context)
{
Record(context);
CloseSessionRequest = request;
return Task.FromResult(new CloseSessionReply
{
SessionId = request.SessionId,
FinalState = SessionState.Closed,
ProtocolStatus = Ok(),
});
}
private static ProtocolStatus Ok() => new() { Code = ProtocolStatusCode.Ok };
private void Record(ServerCallContext context)
{
string? authorization = context.RequestHeaders.GetValue("authorization");
if (authorization is not null)
{
// context.Method is the fully-qualified "/package.Service/Method";
// key on the bare method name so assertions stay readable.
AuthorizationByMethod[context.Method[(context.Method.LastIndexOf('/') + 1)..]] =
authorization;
}
}
}
}
@@ -12,6 +12,15 @@
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.14.1" />
<PackageReference Include="xunit" Version="2.9.3" />
<PackageReference Include="xunit.runner.visualstudio" Version="3.1.4" />
<!-- Wire tests only (WireFakeGatewayServer): hosts the real MxAccessGateway service
on loopback Kestrel so the client is driven over genuine HTTP/2 + protobuf rather
than a substituted transport. Version tracks the gateway server's Grpc.AspNetCore
(src/ZB.MOM.WW.MxGateway.Server) and the client's Grpc.Net.Client, both 2.76.0. -->
<PackageReference Include="Grpc.AspNetCore.Server" Version="2.76.0" />
</ItemGroup>
<ItemGroup>
<FrameworkReference Include="Microsoft.AspNetCore.App" />
</ItemGroup>
<ItemGroup>
+22
View File
@@ -47,6 +47,19 @@ The tests import the generated gateway and worker stubs, run fake async gateway
stubs, verify API key metadata, exercise stream cancellation, load shared value
and command fixtures, and check deterministic CLI output.
`tests/test_wire_fake_gateway.py` is the one suite that does **not** substitute a
stub: it serves a canned `MxAccessGatewayServicer` from a real `grpc.aio` server
on an ephemeral loopback port and drives the ordinary `GatewayClient` API against
it. Only the gateway's behaviour is canned — the HTTP/2 framing, protobuf
serialization, `authorization` metadata, and gRPC status codes are genuine, so it
catches decode and metadata breaks a stub fake cannot see. No MXAccess, no worker,
no TLS, so it runs in the default suite. See `docs/GatewayTesting.md`
(Client Wire Tests) for the cross-client pattern.
```powershell
python -m pytest tests/test_wire_fake_gateway.py
```
## Packaging
Install the package in editable mode for local development:
@@ -398,6 +411,15 @@ point: the `require_certificate_validation=True` keyword on
`--require-certificate-validation` CLI flag. See
[Gateway Configuration](../../docs/GatewayConfiguration.md#automatic-self-signed-certificate).
Channel construction is split in two: `resolve_channel_security(options)` performs
the blocking part (the trust-on-first-use certificate probe) and
`create_channel(options, security=...)` builds the channel. The async `connect`
classmethods run the first off the event loop and the second on it, because a
`grpc.aio` channel binds to the event loop current on the constructing thread —
building it inside `asyncio.to_thread` raises
`RuntimeError: There is no current event loop in thread 'asyncio_N'`. Callers that
build their own channel should keep `create_channel` on the loop thread.
## CLI
The CLI emits deterministic JSON for automation:
@@ -12,7 +12,7 @@ from .auth import merge_metadata
from .errors import ensure_protocol_success, map_rpc_error
from .generated import mxaccess_gateway_pb2 as pb
from .generated import mxaccess_gateway_pb2_grpc as pb_grpc
from .options import ClientOptions, create_channel
from .options import ClientOptions, create_channel, resolve_channel_security
class GatewayClient:
@@ -58,9 +58,13 @@ class GatewayClient:
if stub is not None:
return cls(options=resolved, stub=stub)
# create_channel may perform a blocking TLS certificate probe (TOFU
# default); run it off the event loop so connect never freezes the loop.
channel = await asyncio.to_thread(create_channel, resolved)
# Resolving security may perform a blocking TLS certificate probe (TOFU
# default); run that off the event loop so connect never freezes it. The
# channel itself must be built on the loop thread — a grpc.aio channel
# binds to the loop current on the constructing thread, and a worker
# thread has none.
security = await asyncio.to_thread(resolve_channel_security, resolved)
channel = create_channel(resolved, security=security)
return cls(
options=resolved,
stub=pb_grpc.MxAccessGatewayStub(channel),
@@ -21,7 +21,12 @@ from .auth import merge_metadata
from .errors import MxGatewayError, map_rpc_error
from .generated import galaxy_repository_pb2 as galaxy_pb
from .generated import galaxy_repository_pb2_grpc as galaxy_pb_grpc
from .options import BrowseChildrenOptions, ClientOptions, create_channel
from .options import (
BrowseChildrenOptions,
ClientOptions,
create_channel,
resolve_channel_security,
)
_DISCOVER_HIERARCHY_PAGE_SIZE = 5000
_BROWSE_CHILDREN_PAGE_SIZE = 500
@@ -70,9 +75,13 @@ class GalaxyRepositoryClient:
if stub is not None:
return cls(options=resolved, stub=stub)
# create_channel may perform a blocking TLS certificate probe (TOFU
# default); run it off the event loop so connect never freezes the loop.
channel = await asyncio.to_thread(create_channel, resolved)
# Resolving security may perform a blocking TLS certificate probe (TOFU
# default); run that off the event loop so connect never freezes it. The
# channel itself must be built on the loop thread — a grpc.aio channel
# binds to the loop current on the constructing thread, and a worker
# thread has none.
security = await asyncio.to_thread(resolve_channel_security, resolved)
channel = create_channel(resolved, security=security)
return cls(
options=resolved,
stub=galaxy_pb_grpc.GalaxyRepositoryStub(channel),
@@ -105,7 +105,72 @@ def _split_authority(endpoint: str) -> tuple[str, int]:
return (host or "localhost", int(port))
def create_channel(options: ClientOptions) -> grpc.aio.Channel:
@dataclass(frozen=True)
class ChannelSecurity:
"""Transport security resolved for one channel.
`credentials` is `None` for a plaintext channel. `target_name_override` is
the SNI/authority override the TOFU path needs, kept separate from the
caller's explicit `server_name_override` so the caller always wins.
"""
credentials: grpc.ChannelCredentials | None = None
target_name_override: str | None = None
def resolve_channel_security(options: ClientOptions) -> ChannelSecurity:
"""Resolve transport security for `options`, running any blocking probe.
This is the only blocking part of channel construction: the TOFU path opens
a real TCP+TLS socket to fetch the server's certificate. It is split out of
`create_channel` because a `grpc.aio` channel binds to the event loop
*current on the constructing thread*, so the channel itself must be built on
the loop thread — building it inside `asyncio.to_thread` raises
``RuntimeError: There is no current event loop in thread 'asyncio_N'``. The
async `connect` classmethods therefore run this function off the loop and
then call `create_channel` on it.
"""
if options.plaintext:
return ChannelSecurity()
if options.ca_file:
root_certificates = Path(options.ca_file).read_bytes()
return ChannelSecurity(
credentials=grpc.ssl_channel_credentials(root_certificates=root_certificates)
)
if options.require_certificate_validation:
return ChannelSecurity(credentials=grpc.ssl_channel_credentials())
# Lenient default: grpc-python has no per-channel skip-verify, so fetch the
# server's certificate (unverified) and pin it for this channel (TOFU).
# The probe opens a real blocking TCP+TLS socket, so it MUST be bounded —
# a black-holed / firewall-drop host would otherwise hang on the OS default
# connect timeout (minutes). Bound it by call_timeout (or a short fixed
# fallback) so the dial fails fast as a transport error.
host, port = _split_authority(options.endpoint)
probe_timeout = options.call_timeout if options.call_timeout else _TOFU_PROBE_TIMEOUT_SECONDS
try:
presented = ssl.get_server_certificate((host, port), timeout=probe_timeout)
except OSError as error:
raise MxGatewayTransportError(
f"failed to fetch TLS certificate from {options.endpoint}: {error}"
) from error
# The gateway self-signed cert always carries a "localhost" SAN, so default
# the SNI/target-name override to it when none was supplied, tolerating
# dial-by-IP or hostname mismatch.
return ChannelSecurity(
credentials=grpc.ssl_channel_credentials(root_certificates=presented.encode("ascii")),
target_name_override="localhost",
)
def create_channel(
options: ClientOptions,
*,
security: ChannelSecurity | None = None,
) -> grpc.aio.Channel:
"""Create a plaintext or TLS `grpc.aio` channel from client options.
The TLS default is lenient: grpc-python has no per-channel skip-verify, so
@@ -113,48 +178,29 @@ def create_channel(options: ClientOptions) -> grpc.aio.Channel:
as the channel's only trust root (trust-on-first-use). Set
`require_certificate_validation=True` to force system-trust verification, or
pass `ca_file` to verify against a specific CA — both bypass the TOFU path.
Pass *security* to reuse a `ChannelSecurity` already resolved off the event
loop by `resolve_channel_security`; omit it and this call resolves (and may
block) inline. Must run on the thread owning the event loop the channel will
be used from.
"""
security = security if security is not None else resolve_channel_security(options)
channel_options: list[tuple[str, str | int]] = [
("grpc.max_receive_message_length", options.max_grpc_message_bytes),
("grpc.max_send_message_length", options.max_grpc_message_bytes),
]
if options.server_name_override:
channel_options.append(("grpc.ssl_target_name_override", options.server_name_override))
elif security.target_name_override:
channel_options.append(("grpc.ssl_target_name_override", security.target_name_override))
if options.plaintext:
if security.credentials is None:
return grpc.aio.insecure_channel(options.endpoint, options=channel_options)
if options.ca_file:
root_certificates = Path(options.ca_file).read_bytes()
credentials = grpc.ssl_channel_credentials(root_certificates=root_certificates)
elif options.require_certificate_validation:
credentials = grpc.ssl_channel_credentials()
else:
# Lenient default: grpc-python has no per-channel skip-verify, so fetch the
# server's certificate (unverified) and pin it for this channel (TOFU).
# The probe opens a real blocking TCP+TLS socket, so it MUST be bounded —
# a black-holed / firewall-drop host would otherwise hang on the OS default
# connect timeout (minutes). Bound it by call_timeout (or a short fixed
# fallback) so the dial fails fast as a transport error. The async
# `connect` classmethods run this off the event loop (asyncio.to_thread).
host, port = _split_authority(options.endpoint)
probe_timeout = options.call_timeout if options.call_timeout else _TOFU_PROBE_TIMEOUT_SECONDS
try:
presented = ssl.get_server_certificate((host, port), timeout=probe_timeout)
except OSError as error:
raise MxGatewayTransportError(
f"failed to fetch TLS certificate from {options.endpoint}: {error}"
) from error
credentials = grpc.ssl_channel_credentials(root_certificates=presented.encode("ascii"))
# The gateway self-signed cert always carries a "localhost" SAN, so default
# the SNI/target-name override to it when none was supplied, tolerating
# dial-by-IP or hostname mismatch.
if not options.server_name_override:
channel_options.append(("grpc.ssl_target_name_override", "localhost"))
return grpc.aio.secure_channel(
options.endpoint,
credentials,
security.credentials,
options=channel_options,
)
+52 -34
View File
@@ -12,6 +12,7 @@ from zb_mom_ww_mxgateway import client as client_module
from zb_mom_ww_mxgateway import galaxy as galaxy_module
from zb_mom_ww_mxgateway.galaxy import GalaxyRepositoryClient
from zb_mom_ww_mxgateway.generated import mxaccess_gateway_pb2 as pb
from zb_mom_ww_mxgateway.options import ChannelSecurity
@pytest.mark.asyncio
@@ -21,11 +22,12 @@ async def test_gateway_connect_forwards_require_certificate_validation(
"""The connect convenience kwarg must reach ClientOptions (Client.Python-027)."""
captured: dict[str, Any] = {}
def fake_create_channel(options: ClientOptions) -> object:
def fake_resolve(options: ClientOptions) -> ChannelSecurity:
captured["options"] = options
return object()
return ChannelSecurity()
monkeypatch.setattr(client_module, "create_channel", fake_create_channel)
monkeypatch.setattr(client_module, "resolve_channel_security", fake_resolve)
monkeypatch.setattr(client_module, "create_channel", _stub_create_channel)
monkeypatch.setattr(client_module.pb_grpc, "MxAccessGatewayStub", lambda channel: object())
await GatewayClient.connect(
@@ -43,11 +45,12 @@ async def test_galaxy_connect_forwards_require_certificate_validation(
"""GalaxyRepositoryClient.connect must thread the flag too (Client.Python-027)."""
captured: dict[str, Any] = {}
def fake_create_channel(options: ClientOptions) -> object:
def fake_resolve(options: ClientOptions) -> ChannelSecurity:
captured["options"] = options
return object()
return ChannelSecurity()
monkeypatch.setattr(galaxy_module, "create_channel", fake_create_channel)
monkeypatch.setattr(galaxy_module, "resolve_channel_security", fake_resolve)
monkeypatch.setattr(galaxy_module, "create_channel", _stub_create_channel)
monkeypatch.setattr(
galaxy_module.galaxy_pb_grpc, "GalaxyRepositoryStub", lambda channel: object()
)
@@ -61,52 +64,67 @@ async def test_galaxy_connect_forwards_require_certificate_validation(
@pytest.mark.asyncio
async def test_gateway_connect_runs_create_channel_off_the_event_loop(
async def test_gateway_connect_splits_probe_off_loop_and_channel_on_loop(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""connect must run the blocking channel factory off the loop (Client.Python-028)."""
ran_in_thread: dict[str, bool] = {}
"""The blocking probe runs off the loop; the channel is built on it.
def fake_create_channel(options: ClientOptions) -> object:
# If this runs on the event loop thread, get_running_loop() succeeds.
try:
asyncio.get_running_loop()
ran_in_thread["off_loop"] = False
except RuntimeError:
ran_in_thread["off_loop"] = True
return object()
monkeypatch.setattr(client_module, "create_channel", fake_create_channel)
Client.Python-028 required the blocking TOFU probe off the event loop. The
channel itself must nonetheless be constructed *on* the loop thread: a
``grpc.aio`` channel binds to the loop current on the constructing thread,
and a ``to_thread`` worker has none, so building it off-loop raises
``RuntimeError: There is no current event loop``. Assert both halves.
"""
where = _record_connect_threads(monkeypatch, client_module)
monkeypatch.setattr(client_module.pb_grpc, "MxAccessGatewayStub", lambda channel: object())
await GatewayClient.connect(endpoint="gateway.example:5001")
assert ran_in_thread["off_loop"] is True
assert where == {"resolve_off_loop": True, "create_on_loop": True}
@pytest.mark.asyncio
async def test_galaxy_connect_runs_create_channel_off_the_event_loop(
async def test_galaxy_connect_splits_probe_off_loop_and_channel_on_loop(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""GalaxyRepositoryClient.connect must also run the probe off the loop (Client.Python-028)."""
ran_in_thread: dict[str, bool] = {}
def fake_create_channel(options: ClientOptions) -> object:
try:
asyncio.get_running_loop()
ran_in_thread["off_loop"] = False
except RuntimeError:
ran_in_thread["off_loop"] = True
return object()
monkeypatch.setattr(galaxy_module, "create_channel", fake_create_channel)
"""GalaxyRepositoryClient.connect splits the probe and the channel the same way."""
where = _record_connect_threads(monkeypatch, galaxy_module)
monkeypatch.setattr(
galaxy_module.galaxy_pb_grpc, "GalaxyRepositoryStub", lambda channel: object()
)
await GalaxyRepositoryClient.connect(endpoint="gateway.example:5001")
assert ran_in_thread["off_loop"] is True
assert where == {"resolve_off_loop": True, "create_on_loop": True}
def _stub_create_channel(options: ClientOptions, *, security: ChannelSecurity) -> object:
return object()
def _on_event_loop_thread() -> bool:
try:
asyncio.get_running_loop()
except RuntimeError:
return False
return True
def _record_connect_threads(monkeypatch: pytest.MonkeyPatch, module: Any) -> dict[str, bool]:
"""Patch *module*'s channel helpers to record which thread each ran on."""
where: dict[str, bool] = {}
def fake_resolve(options: ClientOptions) -> ChannelSecurity:
where["resolve_off_loop"] = not _on_event_loop_thread()
return ChannelSecurity()
def fake_create_channel(options: ClientOptions, *, security: ChannelSecurity) -> object:
where["create_on_loop"] = _on_event_loop_thread()
return object()
monkeypatch.setattr(module, "resolve_channel_security", fake_resolve)
monkeypatch.setattr(module, "create_channel", fake_create_channel)
return where
@pytest.mark.asyncio
@@ -0,0 +1,288 @@
"""Wire-level tests: the Python client against a real localhost gRPC server.
Every other test in this suite substitutes a fake *stub* object for
``pb_grpc.MxAccessGatewayStub``, so nothing between the client wrapper and the
generated stub is exercised: no HTTP/2 framing, no protobuf serialization, no
call metadata, no gRPC status translation. That leaves a class of contract break
— a field the gateway populates but the client never decodes, metadata the
client believes it sends but does not, a status code it maps differently once it
arrives as a real ``grpc.RpcError`` — invisible to the default suite.
These tests close that gap by serving the real ``mxaccess_gateway.v1.MxAccessGateway``
service from an in-process ``grpc.aio`` server bound to ``127.0.0.1:0`` and
driving the ordinary public client API against it. The bytes on the wire are the
real ones; only the gateway's *behavior* is canned. No MXAccess, no worker, no
network beyond loopback, so this runs everywhere the normal suite runs.
See ``docs/GatewayTesting.md`` (Client Wire Tests) for the shared pattern and its
counterpart in the .NET client.
"""
from __future__ import annotations
import socket
from collections.abc import AsyncIterator, Awaitable, Callable
import grpc
import pytest
import pytest_asyncio
from zb_mom_ww_mxgateway import ClientOptions, GatewayClient
from zb_mom_ww_mxgateway.errors import MxGatewayAuthorizationError
from zb_mom_ww_mxgateway.events import ReplayGap
from zb_mom_ww_mxgateway.generated import mxaccess_gateway_pb2 as pb
from zb_mom_ww_mxgateway.generated import mxaccess_gateway_pb2_grpc as pb_grpc
API_KEY = "mxgw_wiretest_secret"
SESSION_ID = "wire-session-1"
SERVER_HANDLE = 4242
ITEM_HANDLE = 77
def _free_port() -> int:
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.bind(("127.0.0.1", 0))
return int(sock.getsockname()[1])
def _ok() -> pb.ProtocolStatus:
return pb.ProtocolStatus(code=pb.PROTOCOL_STATUS_CODE_OK)
class FakeGateway(pb_grpc.MxAccessGatewayServicer):
"""Canned gateway serving the four session RPCs over a real transport.
Replies are shaped like the gateway's own: an OK ``ProtocolStatus``, the
echoed session id, and the typed payload the client wrapper reads (for
example ``RegisterReply.server_handle``). Set ``deny`` to make ``Invoke``
abort with ``PERMISSION_DENIED`` so the client's gRPC-status mapping is
exercised against a genuine ``grpc.RpcError`` rather than a hand-built one.
"""
def __init__(self, *, deny: bool = False, replay_gap: pb.ReplayGap | None = None) -> None:
self.deny = deny
self.replay_gap = replay_gap
self.endpoint = ""
self.metadata_by_method: dict[str, str] = {}
self.open_request: pb.OpenSessionRequest | None = None
self.invoke_request: pb.MxCommandRequest | None = None
self.stream_request: pb.StreamEventsRequest | None = None
self.close_request: pb.CloseSessionRequest | None = None
def _record(self, method: str, context: grpc.aio.ServicerContext) -> None:
for key, value in context.invocation_metadata() or ():
if key == "authorization":
self.metadata_by_method[method] = value
async def OpenSession( # noqa: N802 - generated gRPC method name
self, request: pb.OpenSessionRequest, context: grpc.aio.ServicerContext
) -> pb.OpenSessionReply:
"""Answer ``OpenSession`` with a fully populated reply."""
self._record("OpenSession", context)
self.open_request = request
return pb.OpenSessionReply(
session_id=SESSION_ID,
backend_name="fake-backend",
worker_process_id=1234,
worker_protocol_version=1,
capabilities=["events", "invoke"],
gateway_protocol_version=3,
protocol_status=_ok(),
)
async def Invoke( # noqa: N802 - generated gRPC method name
self, request: pb.MxCommandRequest, context: grpc.aio.ServicerContext
) -> pb.MxCommandReply:
"""Answer ``Invoke`` with a Register reply, or deny when configured."""
self._record("Invoke", context)
self.invoke_request = request
if self.deny:
await context.abort(grpc.StatusCode.PERMISSION_DENIED, "invoke scope required")
return pb.MxCommandReply(
session_id=request.session_id,
correlation_id=request.client_correlation_id,
kind=request.command.kind,
protocol_status=_ok(),
hresult=0,
register=pb.RegisterReply(server_handle=SERVER_HANDLE),
)
async def StreamEvents( # noqa: N802 - generated gRPC method name
self, request: pb.StreamEventsRequest, context: grpc.aio.ServicerContext
) -> AsyncIterator[pb.MxEvent]:
"""Stream an optional replay-gap sentinel followed by one data change."""
self._record("StreamEvents", context)
self.stream_request = request
if self.replay_gap is not None:
# The sentinel shape the gateway emits: family unspecified, body
# unset, only replay_gap populated.
yield pb.MxEvent(session_id=request.session_id, replay_gap=self.replay_gap)
yield pb.MxEvent(
session_id=request.session_id,
family=pb.MX_EVENT_FAMILY_ON_DATA_CHANGE,
server_handle=SERVER_HANDLE,
item_handle=ITEM_HANDLE,
value=pb.MxValue(int32_value=17),
quality=192,
worker_sequence=9,
on_data_change=pb.OnDataChangeEvent(),
)
async def CloseSession( # noqa: N802 - generated gRPC method name
self, request: pb.CloseSessionRequest, context: grpc.aio.ServicerContext
) -> pb.CloseSessionReply:
"""Answer ``CloseSession`` with a closed final state."""
self._record("CloseSession", context)
self.close_request = request
return pb.CloseSessionReply(
session_id=request.session_id,
final_state=pb.SESSION_STATE_CLOSED,
protocol_status=_ok(),
)
ServeGateway = Callable[..., Awaitable[FakeGateway]]
@pytest_asyncio.fixture
async def serve_gateway() -> AsyncIterator[ServeGateway]:
"""Yield a factory that serves a :class:`FakeGateway` on loopback.
Each call starts its own server on a free port and records it for teardown,
so a test can serve a differently-configured gateway without a fixture per
variant.
"""
servers: list[grpc.aio.Server] = []
async def _start(**kwargs: object) -> FakeGateway:
fake = FakeGateway(**kwargs) # type: ignore[arg-type]
server = grpc.aio.server()
pb_grpc.add_MxAccessGatewayServicer_to_server(fake, server)
port = _free_port()
server.add_insecure_port(f"127.0.0.1:{port}")
await server.start()
servers.append(server)
fake.endpoint = f"127.0.0.1:{port}"
return fake
try:
yield _start
finally:
for server in servers:
await server.stop(grace=None)
async def _connect(fake: FakeGateway) -> GatewayClient:
return await GatewayClient.connect(
ClientOptions(
endpoint=fake.endpoint,
api_key=API_KEY,
plaintext=True,
call_timeout=10.0,
)
)
@pytest.mark.asyncio
async def test_session_round_trip_decodes_real_wire_bytes(serve_gateway: ServeGateway) -> None:
"""Open, invoke, stream, and close against a real server over loopback."""
wire_gateway = await serve_gateway()
client = await _connect(wire_gateway)
try:
session = await client.open_session(client_session_name="wire-test")
assert session.session_id == SESSION_ID
assert session.open_reply.backend_name == "fake-backend"
assert list(session.open_reply.capabilities) == ["events", "invoke"]
server_handle = await session.register("wire-test-client")
assert server_handle == SERVER_HANDLE
assert wire_gateway.invoke_request is not None
assert wire_gateway.invoke_request.command.kind == pb.MX_COMMAND_KIND_REGISTER
assert wire_gateway.invoke_request.command.register.client_name == "wire-test-client"
events = [event async for event in session.stream_events()]
assert len(events) == 1
event = events[0]
assert not isinstance(event, ReplayGap)
assert event.family == pb.MX_EVENT_FAMILY_ON_DATA_CHANGE
assert event.server_handle == SERVER_HANDLE
assert event.item_handle == ITEM_HANDLE
assert event.value.int32_value == 17
assert event.quality == 192
assert event.worker_sequence == 9
assert event.HasField("on_data_change")
close_reply = await session.close()
assert close_reply.final_state == pb.SESSION_STATE_CLOSED
assert wire_gateway.close_request is not None
assert wire_gateway.close_request.session_id == SESSION_ID
finally:
await client.close()
@pytest.mark.asyncio
async def test_api_key_reaches_the_server_on_every_rpc(serve_gateway: ServeGateway) -> None:
"""The bearer header is on the wire for unary and streaming calls alike.
Stub-substituting tests can only assert what the client *passes*; this
asserts what the server *receives*, which is the property that matters.
"""
wire_gateway = await serve_gateway()
client = await _connect(wire_gateway)
try:
session = await client.open_session(client_session_name="wire-test")
await session.register("wire-test-client")
async for _ in session.stream_events():
break
await session.close()
finally:
await client.close()
expected = f"Bearer {API_KEY}"
assert wire_gateway.metadata_by_method == {
"OpenSession": expected,
"Invoke": expected,
"StreamEvents": expected,
"CloseSession": expected,
}
@pytest.mark.asyncio
async def test_replay_gap_sentinel_survives_the_wire(serve_gateway: ServeGateway) -> None:
"""A resumed stream surfaces the gateway's sentinel as a typed ``ReplayGap``."""
replay_gap_gateway = await serve_gateway(
replay_gap=pb.ReplayGap(requested_after_sequence=3, oldest_available_sequence=8)
)
client = await _connect(replay_gap_gateway)
try:
session = await client.open_session(client_session_name="wire-test")
items = [item async for item in session.stream_events(after_worker_sequence=3)]
finally:
await client.close()
assert len(items) == 2
gap = items[0]
assert isinstance(gap, ReplayGap)
assert gap.requested_after_sequence == 3
assert gap.oldest_available_sequence == 8
assert gap.resume_after_worker_sequence == 7
assert not isinstance(items[1], ReplayGap)
assert items[1].family == pb.MX_EVENT_FAMILY_ON_DATA_CHANGE
assert replay_gap_gateway.stream_request is not None
assert replay_gap_gateway.stream_request.after_worker_sequence == 3
@pytest.mark.asyncio
async def test_permission_denied_maps_to_authorization_error(
serve_gateway: ServeGateway,
) -> None:
"""A real ``PERMISSION_DENIED`` status becomes the typed client error."""
denying_gateway = await serve_gateway(deny=True)
client = await _connect(denying_gateway)
try:
session = await client.open_session(client_session_name="wire-test")
with pytest.raises(MxGatewayAuthorizationError):
await session.register("wire-test-client")
finally:
await client.close()