dohertj2/mxaccessgw
1
0
Fork 0
Code Issues 14 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

test(windev): buffer the test named pipes so the full-suite testhost exits
ci / nightly-windev (push) Has been skipped
Details
ci / windows-x86 (push) Successful in 1m14s
Details
ci / java (push) Successful in 2m19s
Details
ci / portable (push) Successful in 8m48s
Details

Browse Source 
The windev full-suite wedge — every test reported, then the x64 testhost sitting
at ~0 CPU forever while `dotnet test` never returns — was one test blocked on a
pipe write, not a leaked thread or an undisposed fixture.

`dotnet-stack report` on the wedged host showed no thread running test code:
xUnit's RunTestsInAssembly was parked on WaitHandle.WaitOne() waiting for the
assembly-finished event, so the wait lived in a suspended async state machine.
`dotnet-dump analyze -c dumpasync` named the frame — WorkerClientTests
.StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout awaiting
WorkerFrameWriter.WriteAsync on a 63-byte frame, with <extra>5__15 = 6, i.e. the
seventh of the twelve events the test pushes past the client's staging bound.

That test faults the worker client on purpose, and a faulted client stops its
read loop by design. The test-side pipe came from the NamedPipeServerStream
overload without buffer arguments, which passes inBufferSize: 0 / outBufferSize: 0
to CreateNamedPipe; on Windows that reserves no buffer at all, so a write
completes only once the peer reads it. Measured on windev, that pipe absorbed
0 bytes against a non-reading peer where the same pipe declared with 64 KiB
buffers absorbed 65 520. On macOS and Linux .NET backs named pipes with Unix
domain sockets whose socket buffer swallows the writes regardless, which is why
the identical test never hung there and the bug read as environmental.

Test-owned server pipes now go through TestSupport/TestNamedPipe.CreateServer in
both test projects, declaring explicit 64 KiB buffers so those tests exercise the
gateway's own staging/queue backpressure rather than the OS pipe's flow control.

Separately, every fake-worker write in WorkerClientTests now goes through
PipePair.WriteAsync, bounded by the class's five-second TestTimeout. That is
where the severity came from: a test method that never returns keeps xUnit from
raising ITestAssemblyFinished, so one unbounded await cost the whole suite its
result. A blocked write is now a named test failure instead of a silent wedge.

The fix also retires a wrong belief the wedge had created. windev reported 855
where macOS reported 879, and that gap was recorded in GatewayTesting.md as
Unix-gated test cases; it was really the results lost when the wedged host was
torn down. The same clone now reports 879 passed, matching macOS exactly.

Product code is unaffected. SessionWorkerClientFactory.CreatePipe keeps the
unbuffered declaration deliberately: both ends run continuous read loops and every
gateway write is bounded by the worker client's _stopCts, so a stalled peer
cancels the write rather than blocking on it.

Verified on windev at this SHA: gateway suite x64 three times (879 passed,
exit 0, no surviving testhost each time) and Worker.Tests x86 twice (400 passed,
11 skipped, exit 0, clean), plus the macOS gateway suite once (879 passed).

Docs: GatewayTesting.md replaces the --blame-hang workaround section with the
root cause and corrects the baseline to 879, CLAUDE.md's Source Update Workflow
no longer tells readers the windev suite wedges, and ToolchainLinks.md records
dotnet-stack and dotnet-dump as installed on windev.
This commit is contained in:
Joseph Doherty
2026-08-10 09:55:58 -04:00
parent f78781d9ef
commit bfcf82975c
9 changed files with 217 additions and 70 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/WorkerClientTests.cs
+51 -29
View File
@@ -75,7 +75,7 @@ public sealed class WorkerClientTests
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerCommand, commandEnvelope.BodyCase);
Assert.False(string.IsNullOrWhiteSpace(commandEnvelope.CorrelationId));
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping));
WorkerCommandReply reply = await invokeTask.WaitAsync(TestTimeout);
@@ -123,7 +123,7 @@ public sealed class WorkerClientTests
TestTimeout,
CancellationToken.None);
WorkerEnvelope commandEnvelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping));
WorkerCommandReply reply = await nextInvoke.WaitAsync(TestTimeout);
Assert.Equal(MxCommandKind.Ping, reply.Reply.Kind);
@@ -152,7 +152,7 @@ public sealed class WorkerClientTests
// Send the stale reply for the already-timed-out command, then the second
// command's reply. The pipe is FIFO, so the read loop processes (and discards)
// the stale reply before the second reply — no fixed Task.Delay needed.
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateCommandReplyEnvelope(timedOutCommand.CorrelationId, MxCommandKind.Ping));
Task<WorkerCommandReply> secondInvokeTask = client.InvokeAsync(
@@ -160,7 +160,7 @@ public sealed class WorkerClientTests
TestTimeout,
CancellationToken.None);
WorkerEnvelope secondCommand = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateCommandReplyEnvelope(secondCommand.CorrelationId, MxCommandKind.GetWorkerInfo));
WorkerCommandReply reply = await secondInvokeTask.WaitAsync(TestTimeout);
@@ -203,7 +203,7 @@ public sealed class WorkerClientTests
+ "envelope sequences must be strictly increasing in wire order.");
previousSequence = commandEnvelope.Sequence;
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping));
}
@@ -224,9 +224,9 @@ public sealed class WorkerClientTests
await using IAsyncEnumerator<WorkerEvent> events =
client.ReadEventsAsync(cancellationTokenSource.Token).GetAsyncEnumerator(cancellationTokenSource.Token);
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateEventEnvelope(sequence: 11, MxEventFamily.OnDataChange));
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateEventEnvelope(sequence: 12, MxEventFamily.OperationComplete));
Assert.True(await events.MoveNextAsync());
@@ -276,9 +276,9 @@ public sealed class WorkerClientTests
});
await CompleteHandshakeAsync(client, pipePair);
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateEventEnvelope(sequence: 11, MxEventFamily.OnDataChange));
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateEventEnvelope(sequence: 12, MxEventFamily.OnDataChange));
await WaitUntilAsync(
@@ -315,9 +315,9 @@ public sealed class WorkerClientTests
// No StreamEvents consumer is attached, so the capacity-1 event channel fills and the event
// writer blocks on the second event's timed WriteAsync.
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateEventEnvelope(sequence: 11, MxEventFamily.OnDataChange));
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateEventEnvelope(sequence: 12, MxEventFamily.OnDataChange));
Task<WorkerCommandReply> invokeTask = client.InvokeAsync(
@@ -325,7 +325,7 @@ public sealed class WorkerClientTests
TestTimeout,
CancellationToken.None);
WorkerEnvelope commandEnvelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping));
WorkerCommandReply reply = await invokeTask.WaitAsync(TestTimeout);
@@ -359,9 +359,9 @@ public sealed class WorkerClientTests
processHandle: CreateProcessHandle(process));
await CompleteHandshakeAsync(client, pipePair);
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateEventEnvelope(sequence: 11, MxEventFamily.OnDataChange));
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateEventEnvelope(sequence: 12, MxEventFamily.OnDataChange));
// Deterministic: this completes the instant Kill() runs, with no timing window.
@@ -427,7 +427,7 @@ public sealed class WorkerClientTests
WorkerEnvelope commandEnvelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerCommand, commandEnvelope.BodyCase);
await pipePair.WorkerWriter.WriteAsync(CreateWorkerFaultEnvelope("scripted mid-command fault"));
await pipePair.WriteAsync(CreateWorkerFaultEnvelope("scripted mid-command fault"));
WorkerClientException exception = await Assert.ThrowsAsync<WorkerClientException>(
async () => await invokeTask.WaitAsync(TestTimeout));
@@ -530,7 +530,7 @@ public sealed class WorkerClientTests
DateTimeOffset previousHeartbeat = client.LastHeartbeatAt;
clock.Advance(TimeSpan.FromSeconds(1));
await pipePair.WorkerWriter.WriteAsync(CreateHeartbeatEnvelope(workerProcessId: 9876));
await pipePair.WriteAsync(CreateHeartbeatEnvelope(workerProcessId: 9876));
await WaitUntilAsync(
() => client.ProcessId == 9876 && client.LastHeartbeatAt > previousHeartbeat,
@@ -693,7 +693,7 @@ public sealed class WorkerClientTests
// write would block forever on a full OS pipe buffer.
for (ulong sequence = 1; sequence <= 5; sequence++)
{
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateEventEnvelope(sequence: sequence, MxEventFamily.OnDataChange));
}
@@ -756,7 +756,7 @@ public sealed class WorkerClientTests
ulong sequence = 1;
for (; sequence <= (ulong)stagingBound; sequence++)
{
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateEventEnvelope(sequence, MxEventFamily.OnDataChange));
}
@@ -771,7 +771,7 @@ public sealed class WorkerClientTests
TestTimeout,
CancellationToken.None);
WorkerEnvelope commandEnvelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping));
WorkerCommandReply reply = await invokeTask.WaitAsync(TestTimeout);
Assert.Equal(MxCommandKind.Ping, reply.Reply.Kind);
@@ -782,7 +782,7 @@ public sealed class WorkerClientTests
// ones the stopped read loop never drains stay in the OS pipe buffer instead of blocking.
for (int extra = 0; extra < 3 * capacity; extra++, sequence++)
{
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateEventEnvelope(sequence, MxEventFamily.OnDataChange));
}
@@ -836,7 +836,7 @@ public sealed class WorkerClientTests
// Above EventChannelCapacity but below the 2× staging bound: no consumer, no fault.
for (ulong sequence = 1; sequence <= eventCount; sequence++)
{
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateEventEnvelope(sequence, MxEventFamily.OnDataChange));
}
@@ -902,8 +902,8 @@ public sealed class WorkerClientTests
Assert.Equal(Nonce, gatewayHello.GatewayHello.Nonce);
Assert.Equal(GatewayContractInfo.WorkerProtocolVersion, gatewayHello.GatewayHello.SupportedProtocolVersion);
await pipePair.WorkerWriter.WriteAsync(CreateWorkerHelloEnvelope());
await pipePair.WorkerWriter.WriteAsync(CreateWorkerReadyEnvelope());
await pipePair.WriteAsync(CreateWorkerHelloEnvelope());
await pipePair.WriteAsync(CreateWorkerReadyEnvelope());
await startTask.WaitAsync(TestTimeout);
}
@@ -1065,17 +1065,39 @@ public sealed class WorkerClientTests
/// <summary>Frame writer for worker messages.</summary>
public WorkerFrameWriter WorkerWriter { get; }
/// <summary>
/// Writes one envelope from the fake worker side, bounded by <see cref="TestTimeout"/>.
/// </summary>
/// <remarks>
/// Every test-side write goes through here rather than <see cref="WorkerWriter"/> directly so
/// that a write which cannot complete fails this test instead of hanging it. An unbounded
/// write is not a local problem: a test method that never returns leaves xUnit's assembly
/// runner awaiting it forever, so <c>ITestAssemblyFinished</c> is never raised and the whole
/// <c>testhost</c> process never exits — the run reports no failure at all, just a wedge.
/// </remarks>
/// <param name="envelope">The envelope to write.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
public async Task WriteAsync(WorkerEnvelope envelope)
{
using CancellationTokenSource writeTimeout = new(TestTimeout);
try
{
await WorkerWriter.WriteAsync(envelope, writeTimeout.Token);
}
catch (OperationCanceledException) when (writeTimeout.IsCancellationRequested)
{
Assert.Fail(
$"The fake worker's pipe write did not complete within {TestTimeout}. The gateway " +
"side has stopped reading and the pipe buffer is full.");
}
}
/// <summary>Creates a connected pipe pair for testing.</summary>
/// <returns>The connected <see cref="PipePair"/>.</returns>
public static async Task<PipePair> CreateAsync()
{
string pipeName = $"mxgw-wc-{Guid.NewGuid():N}";
NamedPipeServerStream gatewayStream = new(
pipeName,
PipeDirection.InOut,
maxNumberOfServerInstances: 1,
PipeTransmissionMode.Byte,
PipeOptions.Asynchronous);
NamedPipeServerStream gatewayStream = TestNamedPipe.CreateServer(pipeName);
NamedPipeClientStream workerStream = new(
".",
pipeName,