Merge fix/testhost-exit-hang: zero-buffer Windows pipes blocked one test's writes forever; windev suite baseline is 879, same as macOS
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m17s
ci / java (push) Successful in 2m45s
ci / portable (push) Successful in 7m57s

This commit is contained in:
Joseph Doherty
2026-08-10 10:04:40 -04:00
9 changed files with 217 additions and 70 deletions
+1 -1
View File
@@ -124,7 +124,7 @@ powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1
When source code changes, build and test the affected component before reporting work done. If the change crosses component boundaries, build each affected component — don't rely on a single top-level build: When source code changes, build and test the affected component before reporting work done. If the change crosses component boundaries, build each affected component — don't rely on a single top-level build:
**Run targeted tests per task, never the full suite each time.** When executing a plan task-by-task, run only the tests that exercise the code that task touched (`dotnet test --filter "FullyQualifiedName~<TestClass>"`, or the per-task test named in the plan). The full gateway suite is slow — run it at most once per phase (after a related batch of tasks lands), not after every task. This is a speed guideline on macOS, where the suite exits cleanly (0 surviving `testhost`/worker processes after a full run) filtered runs there are about turnaround, not about avoiding a process leak. **On windev a full-suite run wedges**: every test completes, then the x64 `testhost` never exits and `dotnet test` never returns (filtered runs exit normally). Run the full suite there with `--blame-hang --blame-hang-timeout 5m --blame-hang-dump-type none` and read the summary line, not the exit code — see "Running the Gateway Suite on windev" in `docs/GatewayTesting.md`. **Run targeted tests per task, never the full suite each time.** When executing a plan task-by-task, run only the tests that exercise the code that task touched (`dotnet test --filter "FullyQualifiedName~<TestClass>"`, or the per-task test named in the plan). The full gateway suite is slow — run it at most once per phase (after a related batch of tasks lands), not after every task. This is a speed guideline, not a correctness one: the suite exits cleanly on both macOS and windev (0 surviving `testhost`/worker processes after a full run), so filtered runs are about turnaround. The long-standing windev full-suite wedge every test reported, then `testhost` never exiting — was a zero-buffer named pipe blocking one test's write forever; it is fixed and the `--blame-hang` workaround is no longer needed. See "Running the Gateway Suite on windev" in `docs/GatewayTesting.md`, which also documents the load-sensitivity caveat that still applies there.
| Changed area | Required verification | | Changed area | Required verification |
|---|---| |---|---|
+53 -16
View File
@@ -497,10 +497,14 @@ Run it from an isolated clone under `C:\build` checked out to the SHA under test
the dirty Desktop checkout, and never the CI clone `C:\build\mxaccessgw-ci`, whose worktree the dirty Desktop checkout, and never the CI clone `C:\build\mxaccessgw-ci`, whose worktree
lock belongs to the Worker tier. lock belongs to the Worker tier.
Baseline on an otherwise idle windev (2026-08-10): **855 passed, 0 failed, 29 s**. The Baseline on an otherwise idle windev (2026-08-10): **879 passed, 0 failed, 31 s** — the same
suite is smaller there than the 879 the macOS box runs because some cases are gated to 879 the macOS box runs, with nothing gated away. Any failure is therefore a real signal, but
Unix. Any failure is therefore a real signal — but read the load caveat below before read the load caveat below before acting on one.
acting on one.
Runs before the pipe-buffer fix below reported 855, which was long read as "windev runs a
smaller suite because some cases are gated to Unix". It was not: 855 is simply what had been
flushed when the wedged host was torn down. Do not treat a short count on this suite as
platform gating.
### Two long-standing "windev-environmental" failures were test bugs, not the environment ### Two long-standing "windev-environmental" failures were test bugs, not the environment
@@ -529,7 +533,7 @@ These suites drive real named pipes against a five-second worker startup timeout
failing when windev is busy — most often when the x86 Worker tier is building or testing at failing when windev is busy — most often when the x86 Worker tier is building or testing at
the same time. All five passed in the idle baseline above and all five failed in a run taken the same time. All five passed in the idle baseline above and all five failed in a run taken
while an x86 build and `Worker.Tests` were in flight (that run also took 2 m 21 s against the while an x86 build and `Worker.Tests` were in flight (that run also took 2 m 21 s against the
idle 29 s): idle half-minute):
- `GatewayEndToEndFakeWorkerSmokeTests`, `GatewayEndToEndMultiSubscriberTests`, - `GatewayEndToEndFakeWorkerSmokeTests`, `GatewayEndToEndMultiSubscriberTests`,
`GatewayEndToEndReconnectReplayTests` — fail as `GatewayEndToEndReconnectReplayTests` — fail as
@@ -550,22 +554,55 @@ windev has 36 logical CPUs and `xunit.runner.json` sets `maxParallelThreads: -1`
suite runs far wider there than on the macOS dev box — that width is what turns these suite runs far wider there than on the macOS dev box — that width is what turns these
real-clock deadlines into failures. real-clock deadlines into failures.
### The full-suite testhost does not exit on windev ### The full-suite testhost hang was a zero-buffer named pipe (fixed)
After the last test completes, the x64 `testhost` process stops doing work but never exits, For months a full-suite run on windev reported `855 passed, 0 failed` and then never
so `dotnet test` never returns and the run has to be killed. This does **not** happen on returned: the x64 `testhost` stopped consuming CPU but stayed alive indefinitely, and the run
filtered runs (`--filter …`), which exit normally, and does not happen on the macOS dev box — had to be killed with `--blame-hang`. That guard is no longer needed — run the suite plainly:
it is specific to a full-suite run on windev. Run the full suite with a hang guard so the
wedged host is torn down and the pass/fail summary is still printed:
```powershell ```powershell
dotnet test src\ZB.MOM.WW.MxGateway.Tests\ZB.MOM.WW.MxGateway.Tests.csproj ` dotnet test src\ZB.MOM.WW.MxGateway.Tests\ZB.MOM.WW.MxGateway.Tests.csproj
--blame-hang --blame-hang-timeout 5m --blame-hang-dump-type none
``` ```
The summary line ahead of the abort is the real result; the process exit code is nonzero The cause is worth recording because the shape of it is easy to hit again.
because of the abort even when every test passed, so read the summary rather than the exit
code. Prefer filtered runs on windev whenever the change under test allows it. `dotnet-stack report` on the wedged host showed no thread running test code; xUnit's
`RunTestsInAssembly` was simply parked on `WaitHandle.WaitOne()` waiting for the
assembly-finished event. The wait was therefore in a suspended async state machine, which only
`dotnet-dump analyze <dump> -c dumpasync` can see. It named the exact frame:
`WorkerClientTests.StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout`
awaiting `WorkerFrameWriter.WriteAsync` — a 63-byte pipe write that never completed. The `855`
was never the whole suite: the same clone now reports 879, so the wedge was also costing 24
results, and the summary still looked clean because the hung test is not counted as a failure.
That test pushes events past the worker client's staging bound to prove the client faults, and
after the fault the client's read loop stops reading by design. The test-side pipe was created
through `NamedPipeServerStream(string, PipeDirection, int, PipeTransmissionMode, PipeOptions)`,
whose omitted buffer arguments become `inBufferSize: 0` / `outBufferSize: 0`. On Windows that
reserves *no* buffer: a write completes only when the peer reads it. Measured directly on
windev, that pipe absorbed **0 bytes** before blocking against a non-reading peer, while the
same pipe declared with 64 KiB buffers absorbed **65 520**. On macOS and Linux .NET backs named
pipes with Unix domain sockets, whose socket buffer swallows a few kilobytes regardless — which
is why the identical test never hung there, and why the bug read as "a windev thing".
Two changes make it structural rather than incidental:
- Test-owned server pipes are created through `TestSupport/TestNamedPipe.CreateServer`, which
declares explicit 64 KiB buffers, in both the gateway and worker test projects. This scopes
those tests to the backpressure they are actually asserting — the gateway's staging and event
queues — instead of the OS pipe's flow control.
- Every fake-worker write in `WorkerClientTests` goes through `PipePair.WriteAsync`, which
bounds the write by the class's five-second `TestTimeout` and fails with a message naming the
stopped reader. A blocked write is now a named test failure rather than a silent wedge.
The severity came from the second point being missing, not the first. A test method that never
returns keeps xUnit from raising `ITestAssemblyFinished`, so the runner waits forever and
`testhost` never exits — one unbounded `await` in one test costs the entire suite its result.
Any new test that writes to a pipe whose reader may stop must bound the write.
The gateway's production pipe in `SessionWorkerClientFactory.CreatePipe` deliberately keeps the
unbuffered declaration: both ends run continuous read loops and every write there is bounded by
the worker client's `_stopCts`, so a stalled peer cancels the write instead of blocking on it.
## Continuous Integration ## Continuous Integration
+10
View File
@@ -37,6 +37,16 @@ $env:Path = [Environment]::GetEnvironmentVariable('Path','Machine') + ';' + [Env
| C compiler x86 | 14.44.35207 | `C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\bin\Hostx64\x86\cl.exe` | | C compiler x86 | 14.44.35207 | `C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\bin\Hostx64\x86\cl.exe` |
| Linker x86 | 14.44.35207 | `C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\bin\Hostx64\x86\link.exe` | | Linker x86 | 14.44.35207 | `C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\bin\Hostx64\x86\link.exe` |
| LibMan CLI | 3.0.71 | `C:\Users\dohertj2\.dotnet\tools\libman.exe` | | LibMan CLI | 3.0.71 | `C:\Users\dohertj2\.dotnet\tools\libman.exe` |
| dotnet-stack | 9.0.661903 | `C:\Users\dohertj2\.dotnet\tools\dotnet-stack.exe` |
| dotnet-dump | 9.0.661903 | `C:\Users\dohertj2\.dotnet\tools\dotnet-dump.exe` |
`dotnet-stack` and `dotnet-dump` are the diagnostics pair for a process that stops
making progress but does not exit. `dotnet-stack report -p <pid>` prints every
managed thread's stack, which is enough when a *thread* is blocked; when nothing is
on a thread the wait lives in a suspended async state machine, and only
`dotnet-dump collect -p <pid>` followed by `dotnet-dump analyze <dump> -c dumpasync`
reveals it. Both were installed user-local with `dotnet tool install -g` while
root-causing the windev test-host hang described in `docs/GatewayTesting.md`.
Reference assemblies: Reference assemblies:
@@ -5,6 +5,7 @@ using ZB.MOM.WW.MxGateway.Contracts;
using ZB.MOM.WW.MxGateway.Contracts.Proto; using ZB.MOM.WW.MxGateway.Contracts.Proto;
using ZB.MOM.WW.MxGateway.Server.Metrics; using ZB.MOM.WW.MxGateway.Server.Metrics;
using ZB.MOM.WW.MxGateway.Server.Workers; using ZB.MOM.WW.MxGateway.Server.Workers;
using ZB.MOM.WW.MxGateway.Tests.TestSupport;
namespace ZB.MOM.WW.MxGateway.Tests.Gateway.Workers.Fakes; namespace ZB.MOM.WW.MxGateway.Tests.Gateway.Workers.Fakes;
@@ -61,12 +62,7 @@ public sealed class FakeWorkerHarness : IAsyncDisposable
CancellationToken cancellationToken = default) CancellationToken cancellationToken = default)
{ {
string pipeName = $"mxgw-fw-{Guid.NewGuid():N}"; string pipeName = $"mxgw-fw-{Guid.NewGuid():N}";
NamedPipeServerStream gatewayStream = new( NamedPipeServerStream gatewayStream = TestNamedPipe.CreateServer(pipeName);
pipeName,
PipeDirection.InOut,
maxNumberOfServerInstances: 1,
PipeTransmissionMode.Byte,
PipeOptions.Asynchronous);
NamedPipeClientStream workerStream = CreateWorkerStream(pipeName); NamedPipeClientStream workerStream = CreateWorkerStream(pipeName);
Task waitForConnectionTask = gatewayStream.WaitForConnectionAsync(cancellationToken); Task waitForConnectionTask = gatewayStream.WaitForConnectionAsync(cancellationToken);
@@ -75,7 +75,7 @@ public sealed class WorkerClientTests
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerCommand, commandEnvelope.BodyCase); Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerCommand, commandEnvelope.BodyCase);
Assert.False(string.IsNullOrWhiteSpace(commandEnvelope.CorrelationId)); Assert.False(string.IsNullOrWhiteSpace(commandEnvelope.CorrelationId));
await pipePair.WorkerWriter.WriteAsync( await pipePair.WriteAsync(
CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping)); CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping));
WorkerCommandReply reply = await invokeTask.WaitAsync(TestTimeout); WorkerCommandReply reply = await invokeTask.WaitAsync(TestTimeout);
@@ -123,7 +123,7 @@ public sealed class WorkerClientTests
TestTimeout, TestTimeout,
CancellationToken.None); CancellationToken.None);
WorkerEnvelope commandEnvelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout); WorkerEnvelope commandEnvelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
await pipePair.WorkerWriter.WriteAsync( await pipePair.WriteAsync(
CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping)); CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping));
WorkerCommandReply reply = await nextInvoke.WaitAsync(TestTimeout); WorkerCommandReply reply = await nextInvoke.WaitAsync(TestTimeout);
Assert.Equal(MxCommandKind.Ping, reply.Reply.Kind); Assert.Equal(MxCommandKind.Ping, reply.Reply.Kind);
@@ -152,7 +152,7 @@ public sealed class WorkerClientTests
// Send the stale reply for the already-timed-out command, then the second // Send the stale reply for the already-timed-out command, then the second
// command's reply. The pipe is FIFO, so the read loop processes (and discards) // command's reply. The pipe is FIFO, so the read loop processes (and discards)
// the stale reply before the second reply — no fixed Task.Delay needed. // the stale reply before the second reply — no fixed Task.Delay needed.
await pipePair.WorkerWriter.WriteAsync( await pipePair.WriteAsync(
CreateCommandReplyEnvelope(timedOutCommand.CorrelationId, MxCommandKind.Ping)); CreateCommandReplyEnvelope(timedOutCommand.CorrelationId, MxCommandKind.Ping));
Task<WorkerCommandReply> secondInvokeTask = client.InvokeAsync( Task<WorkerCommandReply> secondInvokeTask = client.InvokeAsync(
@@ -160,7 +160,7 @@ public sealed class WorkerClientTests
TestTimeout, TestTimeout,
CancellationToken.None); CancellationToken.None);
WorkerEnvelope secondCommand = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout); WorkerEnvelope secondCommand = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
await pipePair.WorkerWriter.WriteAsync( await pipePair.WriteAsync(
CreateCommandReplyEnvelope(secondCommand.CorrelationId, MxCommandKind.GetWorkerInfo)); CreateCommandReplyEnvelope(secondCommand.CorrelationId, MxCommandKind.GetWorkerInfo));
WorkerCommandReply reply = await secondInvokeTask.WaitAsync(TestTimeout); WorkerCommandReply reply = await secondInvokeTask.WaitAsync(TestTimeout);
@@ -203,7 +203,7 @@ public sealed class WorkerClientTests
+ "envelope sequences must be strictly increasing in wire order."); + "envelope sequences must be strictly increasing in wire order.");
previousSequence = commandEnvelope.Sequence; previousSequence = commandEnvelope.Sequence;
await pipePair.WorkerWriter.WriteAsync( await pipePair.WriteAsync(
CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping)); CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping));
} }
@@ -224,9 +224,9 @@ public sealed class WorkerClientTests
await using IAsyncEnumerator<WorkerEvent> events = await using IAsyncEnumerator<WorkerEvent> events =
client.ReadEventsAsync(cancellationTokenSource.Token).GetAsyncEnumerator(cancellationTokenSource.Token); client.ReadEventsAsync(cancellationTokenSource.Token).GetAsyncEnumerator(cancellationTokenSource.Token);
await pipePair.WorkerWriter.WriteAsync( await pipePair.WriteAsync(
CreateEventEnvelope(sequence: 11, MxEventFamily.OnDataChange)); CreateEventEnvelope(sequence: 11, MxEventFamily.OnDataChange));
await pipePair.WorkerWriter.WriteAsync( await pipePair.WriteAsync(
CreateEventEnvelope(sequence: 12, MxEventFamily.OperationComplete)); CreateEventEnvelope(sequence: 12, MxEventFamily.OperationComplete));
Assert.True(await events.MoveNextAsync()); Assert.True(await events.MoveNextAsync());
@@ -276,9 +276,9 @@ public sealed class WorkerClientTests
}); });
await CompleteHandshakeAsync(client, pipePair); await CompleteHandshakeAsync(client, pipePair);
await pipePair.WorkerWriter.WriteAsync( await pipePair.WriteAsync(
CreateEventEnvelope(sequence: 11, MxEventFamily.OnDataChange)); CreateEventEnvelope(sequence: 11, MxEventFamily.OnDataChange));
await pipePair.WorkerWriter.WriteAsync( await pipePair.WriteAsync(
CreateEventEnvelope(sequence: 12, MxEventFamily.OnDataChange)); CreateEventEnvelope(sequence: 12, MxEventFamily.OnDataChange));
await WaitUntilAsync( await WaitUntilAsync(
@@ -315,9 +315,9 @@ public sealed class WorkerClientTests
// No StreamEvents consumer is attached, so the capacity-1 event channel fills and the event // No StreamEvents consumer is attached, so the capacity-1 event channel fills and the event
// writer blocks on the second event's timed WriteAsync. // writer blocks on the second event's timed WriteAsync.
await pipePair.WorkerWriter.WriteAsync( await pipePair.WriteAsync(
CreateEventEnvelope(sequence: 11, MxEventFamily.OnDataChange)); CreateEventEnvelope(sequence: 11, MxEventFamily.OnDataChange));
await pipePair.WorkerWriter.WriteAsync( await pipePair.WriteAsync(
CreateEventEnvelope(sequence: 12, MxEventFamily.OnDataChange)); CreateEventEnvelope(sequence: 12, MxEventFamily.OnDataChange));
Task<WorkerCommandReply> invokeTask = client.InvokeAsync( Task<WorkerCommandReply> invokeTask = client.InvokeAsync(
@@ -325,7 +325,7 @@ public sealed class WorkerClientTests
TestTimeout, TestTimeout,
CancellationToken.None); CancellationToken.None);
WorkerEnvelope commandEnvelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout); WorkerEnvelope commandEnvelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
await pipePair.WorkerWriter.WriteAsync( await pipePair.WriteAsync(
CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping)); CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping));
WorkerCommandReply reply = await invokeTask.WaitAsync(TestTimeout); WorkerCommandReply reply = await invokeTask.WaitAsync(TestTimeout);
@@ -359,9 +359,9 @@ public sealed class WorkerClientTests
processHandle: CreateProcessHandle(process)); processHandle: CreateProcessHandle(process));
await CompleteHandshakeAsync(client, pipePair); await CompleteHandshakeAsync(client, pipePair);
await pipePair.WorkerWriter.WriteAsync( await pipePair.WriteAsync(
CreateEventEnvelope(sequence: 11, MxEventFamily.OnDataChange)); CreateEventEnvelope(sequence: 11, MxEventFamily.OnDataChange));
await pipePair.WorkerWriter.WriteAsync( await pipePair.WriteAsync(
CreateEventEnvelope(sequence: 12, MxEventFamily.OnDataChange)); CreateEventEnvelope(sequence: 12, MxEventFamily.OnDataChange));
// Deterministic: this completes the instant Kill() runs, with no timing window. // Deterministic: this completes the instant Kill() runs, with no timing window.
@@ -427,7 +427,7 @@ public sealed class WorkerClientTests
WorkerEnvelope commandEnvelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout); WorkerEnvelope commandEnvelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerCommand, commandEnvelope.BodyCase); Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerCommand, commandEnvelope.BodyCase);
await pipePair.WorkerWriter.WriteAsync(CreateWorkerFaultEnvelope("scripted mid-command fault")); await pipePair.WriteAsync(CreateWorkerFaultEnvelope("scripted mid-command fault"));
WorkerClientException exception = await Assert.ThrowsAsync<WorkerClientException>( WorkerClientException exception = await Assert.ThrowsAsync<WorkerClientException>(
async () => await invokeTask.WaitAsync(TestTimeout)); async () => await invokeTask.WaitAsync(TestTimeout));
@@ -530,7 +530,7 @@ public sealed class WorkerClientTests
DateTimeOffset previousHeartbeat = client.LastHeartbeatAt; DateTimeOffset previousHeartbeat = client.LastHeartbeatAt;
clock.Advance(TimeSpan.FromSeconds(1)); clock.Advance(TimeSpan.FromSeconds(1));
await pipePair.WorkerWriter.WriteAsync(CreateHeartbeatEnvelope(workerProcessId: 9876)); await pipePair.WriteAsync(CreateHeartbeatEnvelope(workerProcessId: 9876));
await WaitUntilAsync( await WaitUntilAsync(
() => client.ProcessId == 9876 && client.LastHeartbeatAt > previousHeartbeat, () => client.ProcessId == 9876 && client.LastHeartbeatAt > previousHeartbeat,
@@ -693,7 +693,7 @@ public sealed class WorkerClientTests
// write would block forever on a full OS pipe buffer. // write would block forever on a full OS pipe buffer.
for (ulong sequence = 1; sequence <= 5; sequence++) for (ulong sequence = 1; sequence <= 5; sequence++)
{ {
await pipePair.WorkerWriter.WriteAsync( await pipePair.WriteAsync(
CreateEventEnvelope(sequence: sequence, MxEventFamily.OnDataChange)); CreateEventEnvelope(sequence: sequence, MxEventFamily.OnDataChange));
} }
@@ -756,7 +756,7 @@ public sealed class WorkerClientTests
ulong sequence = 1; ulong sequence = 1;
for (; sequence <= (ulong)stagingBound; sequence++) for (; sequence <= (ulong)stagingBound; sequence++)
{ {
await pipePair.WorkerWriter.WriteAsync( await pipePair.WriteAsync(
CreateEventEnvelope(sequence, MxEventFamily.OnDataChange)); CreateEventEnvelope(sequence, MxEventFamily.OnDataChange));
} }
@@ -771,7 +771,7 @@ public sealed class WorkerClientTests
TestTimeout, TestTimeout,
CancellationToken.None); CancellationToken.None);
WorkerEnvelope commandEnvelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout); WorkerEnvelope commandEnvelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
await pipePair.WorkerWriter.WriteAsync( await pipePair.WriteAsync(
CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping)); CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping));
WorkerCommandReply reply = await invokeTask.WaitAsync(TestTimeout); WorkerCommandReply reply = await invokeTask.WaitAsync(TestTimeout);
Assert.Equal(MxCommandKind.Ping, reply.Reply.Kind); Assert.Equal(MxCommandKind.Ping, reply.Reply.Kind);
@@ -782,7 +782,7 @@ public sealed class WorkerClientTests
// ones the stopped read loop never drains stay in the OS pipe buffer instead of blocking. // ones the stopped read loop never drains stay in the OS pipe buffer instead of blocking.
for (int extra = 0; extra < 3 * capacity; extra++, sequence++) for (int extra = 0; extra < 3 * capacity; extra++, sequence++)
{ {
await pipePair.WorkerWriter.WriteAsync( await pipePair.WriteAsync(
CreateEventEnvelope(sequence, MxEventFamily.OnDataChange)); CreateEventEnvelope(sequence, MxEventFamily.OnDataChange));
} }
@@ -836,7 +836,7 @@ public sealed class WorkerClientTests
// Above EventChannelCapacity but below the 2× staging bound: no consumer, no fault. // Above EventChannelCapacity but below the 2× staging bound: no consumer, no fault.
for (ulong sequence = 1; sequence <= eventCount; sequence++) for (ulong sequence = 1; sequence <= eventCount; sequence++)
{ {
await pipePair.WorkerWriter.WriteAsync( await pipePair.WriteAsync(
CreateEventEnvelope(sequence, MxEventFamily.OnDataChange)); CreateEventEnvelope(sequence, MxEventFamily.OnDataChange));
} }
@@ -902,8 +902,8 @@ public sealed class WorkerClientTests
Assert.Equal(Nonce, gatewayHello.GatewayHello.Nonce); Assert.Equal(Nonce, gatewayHello.GatewayHello.Nonce);
Assert.Equal(GatewayContractInfo.WorkerProtocolVersion, gatewayHello.GatewayHello.SupportedProtocolVersion); Assert.Equal(GatewayContractInfo.WorkerProtocolVersion, gatewayHello.GatewayHello.SupportedProtocolVersion);
await pipePair.WorkerWriter.WriteAsync(CreateWorkerHelloEnvelope()); await pipePair.WriteAsync(CreateWorkerHelloEnvelope());
await pipePair.WorkerWriter.WriteAsync(CreateWorkerReadyEnvelope()); await pipePair.WriteAsync(CreateWorkerReadyEnvelope());
await startTask.WaitAsync(TestTimeout); await startTask.WaitAsync(TestTimeout);
} }
@@ -1065,17 +1065,39 @@ public sealed class WorkerClientTests
/// <summary>Frame writer for worker messages.</summary> /// <summary>Frame writer for worker messages.</summary>
public WorkerFrameWriter WorkerWriter { get; } public WorkerFrameWriter WorkerWriter { get; }
/// <summary>
/// Writes one envelope from the fake worker side, bounded by <see cref="TestTimeout"/>.
/// </summary>
/// <remarks>
/// Every test-side write goes through here rather than <see cref="WorkerWriter"/> directly so
/// that a write which cannot complete fails this test instead of hanging it. An unbounded
/// write is not a local problem: a test method that never returns leaves xUnit's assembly
/// runner awaiting it forever, so <c>ITestAssemblyFinished</c> is never raised and the whole
/// <c>testhost</c> process never exits — the run reports no failure at all, just a wedge.
/// </remarks>
/// <param name="envelope">The envelope to write.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
public async Task WriteAsync(WorkerEnvelope envelope)
{
using CancellationTokenSource writeTimeout = new(TestTimeout);
try
{
await WorkerWriter.WriteAsync(envelope, writeTimeout.Token);
}
catch (OperationCanceledException) when (writeTimeout.IsCancellationRequested)
{
Assert.Fail(
$"The fake worker's pipe write did not complete within {TestTimeout}. The gateway " +
"side has stopped reading and the pipe buffer is full.");
}
}
/// <summary>Creates a connected pipe pair for testing.</summary> /// <summary>Creates a connected pipe pair for testing.</summary>
/// <returns>The connected <see cref="PipePair"/>.</returns> /// <returns>The connected <see cref="PipePair"/>.</returns>
public static async Task<PipePair> CreateAsync() public static async Task<PipePair> CreateAsync()
{ {
string pipeName = $"mxgw-wc-{Guid.NewGuid():N}"; string pipeName = $"mxgw-wc-{Guid.NewGuid():N}";
NamedPipeServerStream gatewayStream = new( NamedPipeServerStream gatewayStream = TestNamedPipe.CreateServer(pipeName);
pipeName,
PipeDirection.InOut,
maxNumberOfServerInstances: 1,
PipeTransmissionMode.Byte,
PipeOptions.Asynchronous);
NamedPipeClientStream workerStream = new( NamedPipeClientStream workerStream = new(
".", ".",
pipeName, pipeName,
@@ -0,0 +1,55 @@
using System.IO.Pipes;
namespace ZB.MOM.WW.MxGateway.Tests.TestSupport;
/// <summary>
/// Creates the gateway-side <see cref="NamedPipeServerStream"/> that in-process worker fakes connect
/// to, with an explicit OS buffer so a test that deliberately stops draining the pipe cannot block
/// its own writer forever.
/// </summary>
/// <remarks>
/// The <c>NamedPipeServerStream(string, PipeDirection, int, PipeTransmissionMode, PipeOptions)</c>
/// overload passes <c>inBufferSize: 0</c> and <c>outBufferSize: 0</c> to <c>CreateNamedPipe</c>. On
/// Windows that reserves <em>no</em> buffer at all: a write completes only once the peer reads it —
/// measured on a Windows 10 host, the very first 63-byte write to a non-reading peer never completed,
/// against 65 520 bytes absorbed by the same pipe declared with 64 KiB buffers. On macOS and Linux
/// .NET backs named pipes with Unix domain sockets, whose socket buffer absorbs several kilobytes
/// regardless, so the zero-buffer declaration is invisible there.
/// <para>
/// That asymmetry wedged the whole Windows test host: <c>WorkerClientTests</c> pushes events past the
/// worker client's staging bound to prove the client faults, and after the fault the client's read
/// loop stops by design. On Windows the next test-side write then blocked forever with no timeout, so
/// the test never returned, xUnit never raised <c>ITestAssemblyFinished</c>, and <c>testhost</c> never
/// exited even though every other test had already passed.
/// </para>
/// <para>
/// Buffering the test pipe scopes those tests to the behavior they are actually asserting — the
/// gateway's own staging/queue backpressure — instead of the OS pipe's flow control. The gateway's
/// production pipe in <c>SessionWorkerClientFactory</c> keeps the unbuffered declaration on purpose:
/// there both ends run continuous read loops and every write is bounded by the worker client's stop
/// token, so a stalled peer cancels rather than blocks.
/// </para>
/// </remarks>
internal static class TestNamedPipe
{
/// <summary>
/// The buffer reserved for each direction. Comfortably larger than any frame volume a test
/// pushes at a stopped reader, and the size Windows itself uses for a typical named pipe.
/// </summary>
internal const int BufferBytes = 64 * 1024;
/// <summary>Creates the gateway-side server pipe for <paramref name="pipeName"/>.</summary>
/// <param name="pipeName">The pipe name the fake worker connects to.</param>
/// <returns>An asynchronous, byte-mode server pipe with explicit buffers.</returns>
internal static NamedPipeServerStream CreateServer(string pipeName)
{
return new NamedPipeServerStream(
pipeName,
PipeDirection.InOut,
maxNumberOfServerInstances: 1,
PipeTransmissionMode.Byte,
PipeOptions.Asynchronous,
inBufferSize: BufferBytes,
outBufferSize: BufferBytes);
}
}
@@ -27,12 +27,7 @@ public sealed class WorkerPipeClientTests
"nonce-secret"); "nonce-secret");
WorkerFrameProtocolOptions frameOptions = new(workerOptions); WorkerFrameProtocolOptions frameOptions = new(workerOptions);
using NamedPipeServerStream server = new( using NamedPipeServerStream server = TestNamedPipe.CreateServer(pipeName);
pipeName,
PipeDirection.InOut,
1,
PipeTransmissionMode.Byte,
PipeOptions.Asynchronous);
WorkerPipeClient client = new( WorkerPipeClient client = new(
connectTimeoutMilliseconds: 5000, connectTimeoutMilliseconds: 5000,
@@ -105,12 +100,7 @@ public sealed class WorkerPipeClientTests
await Task.Delay(150); await Task.Delay(150);
using NamedPipeServerStream server = new( using NamedPipeServerStream server = TestNamedPipe.CreateServer(pipeName);
pipeName,
PipeDirection.InOut,
1,
PipeTransmissionMode.Byte,
PipeOptions.Asynchronous);
await Task.Factory.FromAsync(server.BeginWaitForConnection, server.EndWaitForConnection, null); await Task.Factory.FromAsync(server.BeginWaitForConnection, server.EndWaitForConnection, null);
@@ -2405,12 +2405,7 @@ public sealed class WorkerPipeSessionTests
public static async Task<PipePair> CreateAsync(CancellationToken cancellationToken) public static async Task<PipePair> CreateAsync(CancellationToken cancellationToken)
{ {
string pipeName = $"mxaccessgw-worker-session-tests-{Guid.NewGuid():N}"; string pipeName = $"mxaccessgw-worker-session-tests-{Guid.NewGuid():N}";
NamedPipeServerStream gatewayStream = new( NamedPipeServerStream gatewayStream = TestNamedPipe.CreateServer(pipeName);
pipeName,
PipeDirection.InOut,
maxNumberOfServerInstances: 1,
PipeTransmissionMode.Byte,
PipeOptions.Asynchronous);
NamedPipeClientStream workerStream = new( NamedPipeClientStream workerStream = new(
".", ".",
pipeName, pipeName,
@@ -0,0 +1,42 @@
using System.IO.Pipes;
namespace ZB.MOM.WW.MxGateway.Worker.Tests.TestSupport;
/// <summary>
/// Creates the gateway-side <see cref="NamedPipeServerStream"/> the worker tests connect to, with an
/// explicit OS buffer so a test that stops draining the pipe cannot block its own writer forever.
/// </summary>
/// <remarks>
/// The <c>NamedPipeServerStream(string, PipeDirection, int, PipeTransmissionMode, PipeOptions)</c>
/// overload passes <c>inBufferSize: 0</c> and <c>outBufferSize: 0</c> to <c>CreateNamedPipe</c>, which
/// on Windows reserves no buffer at all: a write completes only once the peer reads it. Measured on a
/// Windows 10 host, the first 63-byte write to a non-reading peer never completed, against 65 520
/// bytes absorbed by the same pipe declared with 64 KiB buffers. A test-side write with no timeout
/// against a stopped reader therefore hangs the test method, and a test method that never returns
/// keeps xUnit from raising <c>ITestAssemblyFinished</c> — so <c>testhost</c> never exits even after
/// every other test has passed. This suite runs only on Windows, so it has no macOS Unix-domain-socket
/// buffer to hide behind. See the matching helper in the gateway test project.
/// </remarks>
internal static class TestNamedPipe
{
/// <summary>
/// The buffer reserved for each direction. Comfortably larger than any frame volume a test
/// pushes at a stopped reader, and the size Windows itself uses for a typical named pipe.
/// </summary>
internal const int BufferBytes = 64 * 1024;
/// <summary>Creates the gateway-side server pipe for <paramref name="pipeName"/>.</summary>
/// <param name="pipeName">The pipe name the worker under test connects to.</param>
/// <returns>An asynchronous, byte-mode server pipe with explicit buffers.</returns>
internal static NamedPipeServerStream CreateServer(string pipeName)
{
return new NamedPipeServerStream(
pipeName,
PipeDirection.InOut,
1,
PipeTransmissionMode.Byte,
PipeOptions.Asynchronous,
BufferBytes,
BufferBytes);
}
}