Merge fix/testhost-exit-hang: zero-buffer Windows pipes blocked one test's writes forever; windev suite baseline is 879, same as macOS
This commit is contained in:
@@ -124,7 +124,7 @@ powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1
|
|||||||
|
|
||||||
When source code changes, build and test the affected component before reporting work done. If the change crosses component boundaries, build each affected component — don't rely on a single top-level build:
|
When source code changes, build and test the affected component before reporting work done. If the change crosses component boundaries, build each affected component — don't rely on a single top-level build:
|
||||||
|
|
||||||
**Run targeted tests per task, never the full suite each time.** When executing a plan task-by-task, run only the tests that exercise the code that task touched (`dotnet test --filter "FullyQualifiedName~<TestClass>"`, or the per-task test named in the plan). The full gateway suite is slow — run it at most once per phase (after a related batch of tasks lands), not after every task. This is a speed guideline on macOS, where the suite exits cleanly (0 surviving `testhost`/worker processes after a full run) — filtered runs there are about turnaround, not about avoiding a process leak. **On windev a full-suite run wedges**: every test completes, then the x64 `testhost` never exits and `dotnet test` never returns (filtered runs exit normally). Run the full suite there with `--blame-hang --blame-hang-timeout 5m --blame-hang-dump-type none` and read the summary line, not the exit code — see "Running the Gateway Suite on windev" in `docs/GatewayTesting.md`.
|
**Run targeted tests per task, never the full suite each time.** When executing a plan task-by-task, run only the tests that exercise the code that task touched (`dotnet test --filter "FullyQualifiedName~<TestClass>"`, or the per-task test named in the plan). The full gateway suite is slow — run it at most once per phase (after a related batch of tasks lands), not after every task. This is a speed guideline, not a correctness one: the suite exits cleanly on both macOS and windev (0 surviving `testhost`/worker processes after a full run), so filtered runs are about turnaround. The long-standing windev full-suite wedge — every test reported, then `testhost` never exiting — was a zero-buffer named pipe blocking one test's write forever; it is fixed and the `--blame-hang` workaround is no longer needed. See "Running the Gateway Suite on windev" in `docs/GatewayTesting.md`, which also documents the load-sensitivity caveat that still applies there.
|
||||||
|
|
||||||
| Changed area | Required verification |
|
| Changed area | Required verification |
|
||||||
|---|---|
|
|---|---|
|
||||||
|
|||||||
+53
-16
@@ -497,10 +497,14 @@ Run it from an isolated clone under `C:\build` checked out to the SHA under test
|
|||||||
the dirty Desktop checkout, and never the CI clone `C:\build\mxaccessgw-ci`, whose worktree
|
the dirty Desktop checkout, and never the CI clone `C:\build\mxaccessgw-ci`, whose worktree
|
||||||
lock belongs to the Worker tier.
|
lock belongs to the Worker tier.
|
||||||
|
|
||||||
Baseline on an otherwise idle windev (2026-08-10): **855 passed, 0 failed, 29 s**. The
|
Baseline on an otherwise idle windev (2026-08-10): **879 passed, 0 failed, 31 s** — the same
|
||||||
suite is smaller there than the 879 the macOS box runs because some cases are gated to
|
879 the macOS box runs, with nothing gated away. Any failure is therefore a real signal, but
|
||||||
Unix. Any failure is therefore a real signal — but read the load caveat below before
|
read the load caveat below before acting on one.
|
||||||
acting on one.
|
|
||||||
|
Runs before the pipe-buffer fix below reported 855, which was long read as "windev runs a
|
||||||
|
smaller suite because some cases are gated to Unix". It was not: 855 is simply what had been
|
||||||
|
flushed when the wedged host was torn down. Do not treat a short count on this suite as
|
||||||
|
platform gating.
|
||||||
|
|
||||||
### Two long-standing "windev-environmental" failures were test bugs, not the environment
|
### Two long-standing "windev-environmental" failures were test bugs, not the environment
|
||||||
|
|
||||||
@@ -529,7 +533,7 @@ These suites drive real named pipes against a five-second worker startup timeout
|
|||||||
failing when windev is busy — most often when the x86 Worker tier is building or testing at
|
failing when windev is busy — most often when the x86 Worker tier is building or testing at
|
||||||
the same time. All five passed in the idle baseline above and all five failed in a run taken
|
the same time. All five passed in the idle baseline above and all five failed in a run taken
|
||||||
while an x86 build and `Worker.Tests` were in flight (that run also took 2 m 21 s against the
|
while an x86 build and `Worker.Tests` were in flight (that run also took 2 m 21 s against the
|
||||||
idle 29 s):
|
idle half-minute):
|
||||||
|
|
||||||
- `GatewayEndToEndFakeWorkerSmokeTests`, `GatewayEndToEndMultiSubscriberTests`,
|
- `GatewayEndToEndFakeWorkerSmokeTests`, `GatewayEndToEndMultiSubscriberTests`,
|
||||||
`GatewayEndToEndReconnectReplayTests` — fail as
|
`GatewayEndToEndReconnectReplayTests` — fail as
|
||||||
@@ -550,22 +554,55 @@ windev has 36 logical CPUs and `xunit.runner.json` sets `maxParallelThreads: -1`
|
|||||||
suite runs far wider there than on the macOS dev box — that width is what turns these
|
suite runs far wider there than on the macOS dev box — that width is what turns these
|
||||||
real-clock deadlines into failures.
|
real-clock deadlines into failures.
|
||||||
|
|
||||||
### The full-suite testhost does not exit on windev
|
### The full-suite testhost hang was a zero-buffer named pipe (fixed)
|
||||||
|
|
||||||
After the last test completes, the x64 `testhost` process stops doing work but never exits,
|
For months a full-suite run on windev reported `855 passed, 0 failed` and then never
|
||||||
so `dotnet test` never returns and the run has to be killed. This does **not** happen on
|
returned: the x64 `testhost` stopped consuming CPU but stayed alive indefinitely, and the run
|
||||||
filtered runs (`--filter …`), which exit normally, and does not happen on the macOS dev box —
|
had to be killed with `--blame-hang`. That guard is no longer needed — run the suite plainly:
|
||||||
it is specific to a full-suite run on windev. Run the full suite with a hang guard so the
|
|
||||||
wedged host is torn down and the pass/fail summary is still printed:
|
|
||||||
|
|
||||||
```powershell
|
```powershell
|
||||||
dotnet test src\ZB.MOM.WW.MxGateway.Tests\ZB.MOM.WW.MxGateway.Tests.csproj `
|
dotnet test src\ZB.MOM.WW.MxGateway.Tests\ZB.MOM.WW.MxGateway.Tests.csproj
|
||||||
--blame-hang --blame-hang-timeout 5m --blame-hang-dump-type none
|
|
||||||
```
|
```
|
||||||
|
|
||||||
The summary line ahead of the abort is the real result; the process exit code is nonzero
|
The cause is worth recording because the shape of it is easy to hit again.
|
||||||
because of the abort even when every test passed, so read the summary rather than the exit
|
|
||||||
code. Prefer filtered runs on windev whenever the change under test allows it.
|
`dotnet-stack report` on the wedged host showed no thread running test code; xUnit's
|
||||||
|
`RunTestsInAssembly` was simply parked on `WaitHandle.WaitOne()` waiting for the
|
||||||
|
assembly-finished event. The wait was therefore in a suspended async state machine, which only
|
||||||
|
`dotnet-dump analyze <dump> -c dumpasync` can see. It named the exact frame:
|
||||||
|
`WorkerClientTests.StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout`
|
||||||
|
awaiting `WorkerFrameWriter.WriteAsync` — a 63-byte pipe write that never completed. The `855`
|
||||||
|
was never the whole suite: the same clone now reports 879, so the wedge was also costing 24
|
||||||
|
results, and the summary still looked clean because the hung test is not counted as a failure.
|
||||||
|
|
||||||
|
That test pushes events past the worker client's staging bound to prove the client faults, and
|
||||||
|
after the fault the client's read loop stops reading by design. The test-side pipe was created
|
||||||
|
through `NamedPipeServerStream(string, PipeDirection, int, PipeTransmissionMode, PipeOptions)`,
|
||||||
|
whose omitted buffer arguments become `inBufferSize: 0` / `outBufferSize: 0`. On Windows that
|
||||||
|
reserves *no* buffer: a write completes only when the peer reads it. Measured directly on
|
||||||
|
windev, that pipe absorbed **0 bytes** before blocking against a non-reading peer, while the
|
||||||
|
same pipe declared with 64 KiB buffers absorbed **65 520**. On macOS and Linux .NET backs named
|
||||||
|
pipes with Unix domain sockets, whose socket buffer swallows a few kilobytes regardless — which
|
||||||
|
is why the identical test never hung there, and why the bug read as "a windev thing".
|
||||||
|
|
||||||
|
Two changes make it structural rather than incidental:
|
||||||
|
|
||||||
|
- Test-owned server pipes are created through `TestSupport/TestNamedPipe.CreateServer`, which
|
||||||
|
declares explicit 64 KiB buffers, in both the gateway and worker test projects. This scopes
|
||||||
|
those tests to the backpressure they are actually asserting — the gateway's staging and event
|
||||||
|
queues — instead of the OS pipe's flow control.
|
||||||
|
- Every fake-worker write in `WorkerClientTests` goes through `PipePair.WriteAsync`, which
|
||||||
|
bounds the write by the class's five-second `TestTimeout` and fails with a message naming the
|
||||||
|
stopped reader. A blocked write is now a named test failure rather than a silent wedge.
|
||||||
|
|
||||||
|
The severity came from the second point being missing, not the first. A test method that never
|
||||||
|
returns keeps xUnit from raising `ITestAssemblyFinished`, so the runner waits forever and
|
||||||
|
`testhost` never exits — one unbounded `await` in one test costs the entire suite its result.
|
||||||
|
Any new test that writes to a pipe whose reader may stop must bound the write.
|
||||||
|
|
||||||
|
The gateway's production pipe in `SessionWorkerClientFactory.CreatePipe` deliberately keeps the
|
||||||
|
unbuffered declaration: both ends run continuous read loops and every write there is bounded by
|
||||||
|
the worker client's `_stopCts`, so a stalled peer cancels the write instead of blocking on it.
|
||||||
|
|
||||||
## Continuous Integration
|
## Continuous Integration
|
||||||
|
|
||||||
|
|||||||
@@ -37,6 +37,16 @@ $env:Path = [Environment]::GetEnvironmentVariable('Path','Machine') + ';' + [Env
|
|||||||
| C compiler x86 | 14.44.35207 | `C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\bin\Hostx64\x86\cl.exe` |
|
| C compiler x86 | 14.44.35207 | `C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\bin\Hostx64\x86\cl.exe` |
|
||||||
| Linker x86 | 14.44.35207 | `C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\bin\Hostx64\x86\link.exe` |
|
| Linker x86 | 14.44.35207 | `C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\bin\Hostx64\x86\link.exe` |
|
||||||
| LibMan CLI | 3.0.71 | `C:\Users\dohertj2\.dotnet\tools\libman.exe` |
|
| LibMan CLI | 3.0.71 | `C:\Users\dohertj2\.dotnet\tools\libman.exe` |
|
||||||
|
| dotnet-stack | 9.0.661903 | `C:\Users\dohertj2\.dotnet\tools\dotnet-stack.exe` |
|
||||||
|
| dotnet-dump | 9.0.661903 | `C:\Users\dohertj2\.dotnet\tools\dotnet-dump.exe` |
|
||||||
|
|
||||||
|
`dotnet-stack` and `dotnet-dump` are the diagnostics pair for a process that stops
|
||||||
|
making progress but does not exit. `dotnet-stack report -p <pid>` prints every
|
||||||
|
managed thread's stack, which is enough when a *thread* is blocked; when nothing is
|
||||||
|
on a thread the wait lives in a suspended async state machine, and only
|
||||||
|
`dotnet-dump collect -p <pid>` followed by `dotnet-dump analyze <dump> -c dumpasync`
|
||||||
|
reveals it. Both were installed user-local with `dotnet tool install -g` while
|
||||||
|
root-causing the windev test-host hang described in `docs/GatewayTesting.md`.
|
||||||
|
|
||||||
Reference assemblies:
|
Reference assemblies:
|
||||||
|
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ using ZB.MOM.WW.MxGateway.Contracts;
|
|||||||
using ZB.MOM.WW.MxGateway.Contracts.Proto;
|
using ZB.MOM.WW.MxGateway.Contracts.Proto;
|
||||||
using ZB.MOM.WW.MxGateway.Server.Metrics;
|
using ZB.MOM.WW.MxGateway.Server.Metrics;
|
||||||
using ZB.MOM.WW.MxGateway.Server.Workers;
|
using ZB.MOM.WW.MxGateway.Server.Workers;
|
||||||
|
using ZB.MOM.WW.MxGateway.Tests.TestSupport;
|
||||||
|
|
||||||
namespace ZB.MOM.WW.MxGateway.Tests.Gateway.Workers.Fakes;
|
namespace ZB.MOM.WW.MxGateway.Tests.Gateway.Workers.Fakes;
|
||||||
|
|
||||||
@@ -61,12 +62,7 @@ public sealed class FakeWorkerHarness : IAsyncDisposable
|
|||||||
CancellationToken cancellationToken = default)
|
CancellationToken cancellationToken = default)
|
||||||
{
|
{
|
||||||
string pipeName = $"mxgw-fw-{Guid.NewGuid():N}";
|
string pipeName = $"mxgw-fw-{Guid.NewGuid():N}";
|
||||||
NamedPipeServerStream gatewayStream = new(
|
NamedPipeServerStream gatewayStream = TestNamedPipe.CreateServer(pipeName);
|
||||||
pipeName,
|
|
||||||
PipeDirection.InOut,
|
|
||||||
maxNumberOfServerInstances: 1,
|
|
||||||
PipeTransmissionMode.Byte,
|
|
||||||
PipeOptions.Asynchronous);
|
|
||||||
NamedPipeClientStream workerStream = CreateWorkerStream(pipeName);
|
NamedPipeClientStream workerStream = CreateWorkerStream(pipeName);
|
||||||
|
|
||||||
Task waitForConnectionTask = gatewayStream.WaitForConnectionAsync(cancellationToken);
|
Task waitForConnectionTask = gatewayStream.WaitForConnectionAsync(cancellationToken);
|
||||||
|
|||||||
@@ -75,7 +75,7 @@ public sealed class WorkerClientTests
|
|||||||
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerCommand, commandEnvelope.BodyCase);
|
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerCommand, commandEnvelope.BodyCase);
|
||||||
Assert.False(string.IsNullOrWhiteSpace(commandEnvelope.CorrelationId));
|
Assert.False(string.IsNullOrWhiteSpace(commandEnvelope.CorrelationId));
|
||||||
|
|
||||||
await pipePair.WorkerWriter.WriteAsync(
|
await pipePair.WriteAsync(
|
||||||
CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping));
|
CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping));
|
||||||
|
|
||||||
WorkerCommandReply reply = await invokeTask.WaitAsync(TestTimeout);
|
WorkerCommandReply reply = await invokeTask.WaitAsync(TestTimeout);
|
||||||
@@ -123,7 +123,7 @@ public sealed class WorkerClientTests
|
|||||||
TestTimeout,
|
TestTimeout,
|
||||||
CancellationToken.None);
|
CancellationToken.None);
|
||||||
WorkerEnvelope commandEnvelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
|
WorkerEnvelope commandEnvelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
|
||||||
await pipePair.WorkerWriter.WriteAsync(
|
await pipePair.WriteAsync(
|
||||||
CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping));
|
CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping));
|
||||||
WorkerCommandReply reply = await nextInvoke.WaitAsync(TestTimeout);
|
WorkerCommandReply reply = await nextInvoke.WaitAsync(TestTimeout);
|
||||||
Assert.Equal(MxCommandKind.Ping, reply.Reply.Kind);
|
Assert.Equal(MxCommandKind.Ping, reply.Reply.Kind);
|
||||||
@@ -152,7 +152,7 @@ public sealed class WorkerClientTests
|
|||||||
// Send the stale reply for the already-timed-out command, then the second
|
// Send the stale reply for the already-timed-out command, then the second
|
||||||
// command's reply. The pipe is FIFO, so the read loop processes (and discards)
|
// command's reply. The pipe is FIFO, so the read loop processes (and discards)
|
||||||
// the stale reply before the second reply — no fixed Task.Delay needed.
|
// the stale reply before the second reply — no fixed Task.Delay needed.
|
||||||
await pipePair.WorkerWriter.WriteAsync(
|
await pipePair.WriteAsync(
|
||||||
CreateCommandReplyEnvelope(timedOutCommand.CorrelationId, MxCommandKind.Ping));
|
CreateCommandReplyEnvelope(timedOutCommand.CorrelationId, MxCommandKind.Ping));
|
||||||
|
|
||||||
Task<WorkerCommandReply> secondInvokeTask = client.InvokeAsync(
|
Task<WorkerCommandReply> secondInvokeTask = client.InvokeAsync(
|
||||||
@@ -160,7 +160,7 @@ public sealed class WorkerClientTests
|
|||||||
TestTimeout,
|
TestTimeout,
|
||||||
CancellationToken.None);
|
CancellationToken.None);
|
||||||
WorkerEnvelope secondCommand = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
|
WorkerEnvelope secondCommand = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
|
||||||
await pipePair.WorkerWriter.WriteAsync(
|
await pipePair.WriteAsync(
|
||||||
CreateCommandReplyEnvelope(secondCommand.CorrelationId, MxCommandKind.GetWorkerInfo));
|
CreateCommandReplyEnvelope(secondCommand.CorrelationId, MxCommandKind.GetWorkerInfo));
|
||||||
|
|
||||||
WorkerCommandReply reply = await secondInvokeTask.WaitAsync(TestTimeout);
|
WorkerCommandReply reply = await secondInvokeTask.WaitAsync(TestTimeout);
|
||||||
@@ -203,7 +203,7 @@ public sealed class WorkerClientTests
|
|||||||
+ "envelope sequences must be strictly increasing in wire order.");
|
+ "envelope sequences must be strictly increasing in wire order.");
|
||||||
previousSequence = commandEnvelope.Sequence;
|
previousSequence = commandEnvelope.Sequence;
|
||||||
|
|
||||||
await pipePair.WorkerWriter.WriteAsync(
|
await pipePair.WriteAsync(
|
||||||
CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping));
|
CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping));
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -224,9 +224,9 @@ public sealed class WorkerClientTests
|
|||||||
await using IAsyncEnumerator<WorkerEvent> events =
|
await using IAsyncEnumerator<WorkerEvent> events =
|
||||||
client.ReadEventsAsync(cancellationTokenSource.Token).GetAsyncEnumerator(cancellationTokenSource.Token);
|
client.ReadEventsAsync(cancellationTokenSource.Token).GetAsyncEnumerator(cancellationTokenSource.Token);
|
||||||
|
|
||||||
await pipePair.WorkerWriter.WriteAsync(
|
await pipePair.WriteAsync(
|
||||||
CreateEventEnvelope(sequence: 11, MxEventFamily.OnDataChange));
|
CreateEventEnvelope(sequence: 11, MxEventFamily.OnDataChange));
|
||||||
await pipePair.WorkerWriter.WriteAsync(
|
await pipePair.WriteAsync(
|
||||||
CreateEventEnvelope(sequence: 12, MxEventFamily.OperationComplete));
|
CreateEventEnvelope(sequence: 12, MxEventFamily.OperationComplete));
|
||||||
|
|
||||||
Assert.True(await events.MoveNextAsync());
|
Assert.True(await events.MoveNextAsync());
|
||||||
@@ -276,9 +276,9 @@ public sealed class WorkerClientTests
|
|||||||
});
|
});
|
||||||
await CompleteHandshakeAsync(client, pipePair);
|
await CompleteHandshakeAsync(client, pipePair);
|
||||||
|
|
||||||
await pipePair.WorkerWriter.WriteAsync(
|
await pipePair.WriteAsync(
|
||||||
CreateEventEnvelope(sequence: 11, MxEventFamily.OnDataChange));
|
CreateEventEnvelope(sequence: 11, MxEventFamily.OnDataChange));
|
||||||
await pipePair.WorkerWriter.WriteAsync(
|
await pipePair.WriteAsync(
|
||||||
CreateEventEnvelope(sequence: 12, MxEventFamily.OnDataChange));
|
CreateEventEnvelope(sequence: 12, MxEventFamily.OnDataChange));
|
||||||
|
|
||||||
await WaitUntilAsync(
|
await WaitUntilAsync(
|
||||||
@@ -315,9 +315,9 @@ public sealed class WorkerClientTests
|
|||||||
|
|
||||||
// No StreamEvents consumer is attached, so the capacity-1 event channel fills and the event
|
// No StreamEvents consumer is attached, so the capacity-1 event channel fills and the event
|
||||||
// writer blocks on the second event's timed WriteAsync.
|
// writer blocks on the second event's timed WriteAsync.
|
||||||
await pipePair.WorkerWriter.WriteAsync(
|
await pipePair.WriteAsync(
|
||||||
CreateEventEnvelope(sequence: 11, MxEventFamily.OnDataChange));
|
CreateEventEnvelope(sequence: 11, MxEventFamily.OnDataChange));
|
||||||
await pipePair.WorkerWriter.WriteAsync(
|
await pipePair.WriteAsync(
|
||||||
CreateEventEnvelope(sequence: 12, MxEventFamily.OnDataChange));
|
CreateEventEnvelope(sequence: 12, MxEventFamily.OnDataChange));
|
||||||
|
|
||||||
Task<WorkerCommandReply> invokeTask = client.InvokeAsync(
|
Task<WorkerCommandReply> invokeTask = client.InvokeAsync(
|
||||||
@@ -325,7 +325,7 @@ public sealed class WorkerClientTests
|
|||||||
TestTimeout,
|
TestTimeout,
|
||||||
CancellationToken.None);
|
CancellationToken.None);
|
||||||
WorkerEnvelope commandEnvelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
|
WorkerEnvelope commandEnvelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
|
||||||
await pipePair.WorkerWriter.WriteAsync(
|
await pipePair.WriteAsync(
|
||||||
CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping));
|
CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping));
|
||||||
|
|
||||||
WorkerCommandReply reply = await invokeTask.WaitAsync(TestTimeout);
|
WorkerCommandReply reply = await invokeTask.WaitAsync(TestTimeout);
|
||||||
@@ -359,9 +359,9 @@ public sealed class WorkerClientTests
|
|||||||
processHandle: CreateProcessHandle(process));
|
processHandle: CreateProcessHandle(process));
|
||||||
await CompleteHandshakeAsync(client, pipePair);
|
await CompleteHandshakeAsync(client, pipePair);
|
||||||
|
|
||||||
await pipePair.WorkerWriter.WriteAsync(
|
await pipePair.WriteAsync(
|
||||||
CreateEventEnvelope(sequence: 11, MxEventFamily.OnDataChange));
|
CreateEventEnvelope(sequence: 11, MxEventFamily.OnDataChange));
|
||||||
await pipePair.WorkerWriter.WriteAsync(
|
await pipePair.WriteAsync(
|
||||||
CreateEventEnvelope(sequence: 12, MxEventFamily.OnDataChange));
|
CreateEventEnvelope(sequence: 12, MxEventFamily.OnDataChange));
|
||||||
|
|
||||||
// Deterministic: this completes the instant Kill() runs, with no timing window.
|
// Deterministic: this completes the instant Kill() runs, with no timing window.
|
||||||
@@ -427,7 +427,7 @@ public sealed class WorkerClientTests
|
|||||||
|
|
||||||
WorkerEnvelope commandEnvelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
|
WorkerEnvelope commandEnvelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
|
||||||
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerCommand, commandEnvelope.BodyCase);
|
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerCommand, commandEnvelope.BodyCase);
|
||||||
await pipePair.WorkerWriter.WriteAsync(CreateWorkerFaultEnvelope("scripted mid-command fault"));
|
await pipePair.WriteAsync(CreateWorkerFaultEnvelope("scripted mid-command fault"));
|
||||||
|
|
||||||
WorkerClientException exception = await Assert.ThrowsAsync<WorkerClientException>(
|
WorkerClientException exception = await Assert.ThrowsAsync<WorkerClientException>(
|
||||||
async () => await invokeTask.WaitAsync(TestTimeout));
|
async () => await invokeTask.WaitAsync(TestTimeout));
|
||||||
@@ -530,7 +530,7 @@ public sealed class WorkerClientTests
|
|||||||
DateTimeOffset previousHeartbeat = client.LastHeartbeatAt;
|
DateTimeOffset previousHeartbeat = client.LastHeartbeatAt;
|
||||||
|
|
||||||
clock.Advance(TimeSpan.FromSeconds(1));
|
clock.Advance(TimeSpan.FromSeconds(1));
|
||||||
await pipePair.WorkerWriter.WriteAsync(CreateHeartbeatEnvelope(workerProcessId: 9876));
|
await pipePair.WriteAsync(CreateHeartbeatEnvelope(workerProcessId: 9876));
|
||||||
|
|
||||||
await WaitUntilAsync(
|
await WaitUntilAsync(
|
||||||
() => client.ProcessId == 9876 && client.LastHeartbeatAt > previousHeartbeat,
|
() => client.ProcessId == 9876 && client.LastHeartbeatAt > previousHeartbeat,
|
||||||
@@ -693,7 +693,7 @@ public sealed class WorkerClientTests
|
|||||||
// write would block forever on a full OS pipe buffer.
|
// write would block forever on a full OS pipe buffer.
|
||||||
for (ulong sequence = 1; sequence <= 5; sequence++)
|
for (ulong sequence = 1; sequence <= 5; sequence++)
|
||||||
{
|
{
|
||||||
await pipePair.WorkerWriter.WriteAsync(
|
await pipePair.WriteAsync(
|
||||||
CreateEventEnvelope(sequence: sequence, MxEventFamily.OnDataChange));
|
CreateEventEnvelope(sequence: sequence, MxEventFamily.OnDataChange));
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -756,7 +756,7 @@ public sealed class WorkerClientTests
|
|||||||
ulong sequence = 1;
|
ulong sequence = 1;
|
||||||
for (; sequence <= (ulong)stagingBound; sequence++)
|
for (; sequence <= (ulong)stagingBound; sequence++)
|
||||||
{
|
{
|
||||||
await pipePair.WorkerWriter.WriteAsync(
|
await pipePair.WriteAsync(
|
||||||
CreateEventEnvelope(sequence, MxEventFamily.OnDataChange));
|
CreateEventEnvelope(sequence, MxEventFamily.OnDataChange));
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -771,7 +771,7 @@ public sealed class WorkerClientTests
|
|||||||
TestTimeout,
|
TestTimeout,
|
||||||
CancellationToken.None);
|
CancellationToken.None);
|
||||||
WorkerEnvelope commandEnvelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
|
WorkerEnvelope commandEnvelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
|
||||||
await pipePair.WorkerWriter.WriteAsync(
|
await pipePair.WriteAsync(
|
||||||
CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping));
|
CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping));
|
||||||
WorkerCommandReply reply = await invokeTask.WaitAsync(TestTimeout);
|
WorkerCommandReply reply = await invokeTask.WaitAsync(TestTimeout);
|
||||||
Assert.Equal(MxCommandKind.Ping, reply.Reply.Kind);
|
Assert.Equal(MxCommandKind.Ping, reply.Reply.Kind);
|
||||||
@@ -782,7 +782,7 @@ public sealed class WorkerClientTests
|
|||||||
// ones the stopped read loop never drains stay in the OS pipe buffer instead of blocking.
|
// ones the stopped read loop never drains stay in the OS pipe buffer instead of blocking.
|
||||||
for (int extra = 0; extra < 3 * capacity; extra++, sequence++)
|
for (int extra = 0; extra < 3 * capacity; extra++, sequence++)
|
||||||
{
|
{
|
||||||
await pipePair.WorkerWriter.WriteAsync(
|
await pipePair.WriteAsync(
|
||||||
CreateEventEnvelope(sequence, MxEventFamily.OnDataChange));
|
CreateEventEnvelope(sequence, MxEventFamily.OnDataChange));
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -836,7 +836,7 @@ public sealed class WorkerClientTests
|
|||||||
// Above EventChannelCapacity but below the 2× staging bound: no consumer, no fault.
|
// Above EventChannelCapacity but below the 2× staging bound: no consumer, no fault.
|
||||||
for (ulong sequence = 1; sequence <= eventCount; sequence++)
|
for (ulong sequence = 1; sequence <= eventCount; sequence++)
|
||||||
{
|
{
|
||||||
await pipePair.WorkerWriter.WriteAsync(
|
await pipePair.WriteAsync(
|
||||||
CreateEventEnvelope(sequence, MxEventFamily.OnDataChange));
|
CreateEventEnvelope(sequence, MxEventFamily.OnDataChange));
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -902,8 +902,8 @@ public sealed class WorkerClientTests
|
|||||||
Assert.Equal(Nonce, gatewayHello.GatewayHello.Nonce);
|
Assert.Equal(Nonce, gatewayHello.GatewayHello.Nonce);
|
||||||
Assert.Equal(GatewayContractInfo.WorkerProtocolVersion, gatewayHello.GatewayHello.SupportedProtocolVersion);
|
Assert.Equal(GatewayContractInfo.WorkerProtocolVersion, gatewayHello.GatewayHello.SupportedProtocolVersion);
|
||||||
|
|
||||||
await pipePair.WorkerWriter.WriteAsync(CreateWorkerHelloEnvelope());
|
await pipePair.WriteAsync(CreateWorkerHelloEnvelope());
|
||||||
await pipePair.WorkerWriter.WriteAsync(CreateWorkerReadyEnvelope());
|
await pipePair.WriteAsync(CreateWorkerReadyEnvelope());
|
||||||
await startTask.WaitAsync(TestTimeout);
|
await startTask.WaitAsync(TestTimeout);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1065,17 +1065,39 @@ public sealed class WorkerClientTests
|
|||||||
/// <summary>Frame writer for worker messages.</summary>
|
/// <summary>Frame writer for worker messages.</summary>
|
||||||
public WorkerFrameWriter WorkerWriter { get; }
|
public WorkerFrameWriter WorkerWriter { get; }
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Writes one envelope from the fake worker side, bounded by <see cref="TestTimeout"/>.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// Every test-side write goes through here rather than <see cref="WorkerWriter"/> directly so
|
||||||
|
/// that a write which cannot complete fails this test instead of hanging it. An unbounded
|
||||||
|
/// write is not a local problem: a test method that never returns leaves xUnit's assembly
|
||||||
|
/// runner awaiting it forever, so <c>ITestAssemblyFinished</c> is never raised and the whole
|
||||||
|
/// <c>testhost</c> process never exits — the run reports no failure at all, just a wedge.
|
||||||
|
/// </remarks>
|
||||||
|
/// <param name="envelope">The envelope to write.</param>
|
||||||
|
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||||
|
public async Task WriteAsync(WorkerEnvelope envelope)
|
||||||
|
{
|
||||||
|
using CancellationTokenSource writeTimeout = new(TestTimeout);
|
||||||
|
try
|
||||||
|
{
|
||||||
|
await WorkerWriter.WriteAsync(envelope, writeTimeout.Token);
|
||||||
|
}
|
||||||
|
catch (OperationCanceledException) when (writeTimeout.IsCancellationRequested)
|
||||||
|
{
|
||||||
|
Assert.Fail(
|
||||||
|
$"The fake worker's pipe write did not complete within {TestTimeout}. The gateway " +
|
||||||
|
"side has stopped reading and the pipe buffer is full.");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>Creates a connected pipe pair for testing.</summary>
|
/// <summary>Creates a connected pipe pair for testing.</summary>
|
||||||
/// <returns>The connected <see cref="PipePair"/>.</returns>
|
/// <returns>The connected <see cref="PipePair"/>.</returns>
|
||||||
public static async Task<PipePair> CreateAsync()
|
public static async Task<PipePair> CreateAsync()
|
||||||
{
|
{
|
||||||
string pipeName = $"mxgw-wc-{Guid.NewGuid():N}";
|
string pipeName = $"mxgw-wc-{Guid.NewGuid():N}";
|
||||||
NamedPipeServerStream gatewayStream = new(
|
NamedPipeServerStream gatewayStream = TestNamedPipe.CreateServer(pipeName);
|
||||||
pipeName,
|
|
||||||
PipeDirection.InOut,
|
|
||||||
maxNumberOfServerInstances: 1,
|
|
||||||
PipeTransmissionMode.Byte,
|
|
||||||
PipeOptions.Asynchronous);
|
|
||||||
NamedPipeClientStream workerStream = new(
|
NamedPipeClientStream workerStream = new(
|
||||||
".",
|
".",
|
||||||
pipeName,
|
pipeName,
|
||||||
|
|||||||
@@ -0,0 +1,55 @@
|
|||||||
|
using System.IO.Pipes;
|
||||||
|
|
||||||
|
namespace ZB.MOM.WW.MxGateway.Tests.TestSupport;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Creates the gateway-side <see cref="NamedPipeServerStream"/> that in-process worker fakes connect
|
||||||
|
/// to, with an explicit OS buffer so a test that deliberately stops draining the pipe cannot block
|
||||||
|
/// its own writer forever.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// The <c>NamedPipeServerStream(string, PipeDirection, int, PipeTransmissionMode, PipeOptions)</c>
|
||||||
|
/// overload passes <c>inBufferSize: 0</c> and <c>outBufferSize: 0</c> to <c>CreateNamedPipe</c>. On
|
||||||
|
/// Windows that reserves <em>no</em> buffer at all: a write completes only once the peer reads it —
|
||||||
|
/// measured on a Windows 10 host, the very first 63-byte write to a non-reading peer never completed,
|
||||||
|
/// against 65 520 bytes absorbed by the same pipe declared with 64 KiB buffers. On macOS and Linux
|
||||||
|
/// .NET backs named pipes with Unix domain sockets, whose socket buffer absorbs several kilobytes
|
||||||
|
/// regardless, so the zero-buffer declaration is invisible there.
|
||||||
|
/// <para>
|
||||||
|
/// That asymmetry wedged the whole Windows test host: <c>WorkerClientTests</c> pushes events past the
|
||||||
|
/// worker client's staging bound to prove the client faults, and after the fault the client's read
|
||||||
|
/// loop stops by design. On Windows the next test-side write then blocked forever with no timeout, so
|
||||||
|
/// the test never returned, xUnit never raised <c>ITestAssemblyFinished</c>, and <c>testhost</c> never
|
||||||
|
/// exited even though every other test had already passed.
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// Buffering the test pipe scopes those tests to the behavior they are actually asserting — the
|
||||||
|
/// gateway's own staging/queue backpressure — instead of the OS pipe's flow control. The gateway's
|
||||||
|
/// production pipe in <c>SessionWorkerClientFactory</c> keeps the unbuffered declaration on purpose:
|
||||||
|
/// there both ends run continuous read loops and every write is bounded by the worker client's stop
|
||||||
|
/// token, so a stalled peer cancels rather than blocks.
|
||||||
|
/// </para>
|
||||||
|
/// </remarks>
|
||||||
|
internal static class TestNamedPipe
|
||||||
|
{
|
||||||
|
/// <summary>
|
||||||
|
/// The buffer reserved for each direction. Comfortably larger than any frame volume a test
|
||||||
|
/// pushes at a stopped reader, and the size Windows itself uses for a typical named pipe.
|
||||||
|
/// </summary>
|
||||||
|
internal const int BufferBytes = 64 * 1024;
|
||||||
|
|
||||||
|
/// <summary>Creates the gateway-side server pipe for <paramref name="pipeName"/>.</summary>
|
||||||
|
/// <param name="pipeName">The pipe name the fake worker connects to.</param>
|
||||||
|
/// <returns>An asynchronous, byte-mode server pipe with explicit buffers.</returns>
|
||||||
|
internal static NamedPipeServerStream CreateServer(string pipeName)
|
||||||
|
{
|
||||||
|
return new NamedPipeServerStream(
|
||||||
|
pipeName,
|
||||||
|
PipeDirection.InOut,
|
||||||
|
maxNumberOfServerInstances: 1,
|
||||||
|
PipeTransmissionMode.Byte,
|
||||||
|
PipeOptions.Asynchronous,
|
||||||
|
inBufferSize: BufferBytes,
|
||||||
|
outBufferSize: BufferBytes);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -27,12 +27,7 @@ public sealed class WorkerPipeClientTests
|
|||||||
"nonce-secret");
|
"nonce-secret");
|
||||||
WorkerFrameProtocolOptions frameOptions = new(workerOptions);
|
WorkerFrameProtocolOptions frameOptions = new(workerOptions);
|
||||||
|
|
||||||
using NamedPipeServerStream server = new(
|
using NamedPipeServerStream server = TestNamedPipe.CreateServer(pipeName);
|
||||||
pipeName,
|
|
||||||
PipeDirection.InOut,
|
|
||||||
1,
|
|
||||||
PipeTransmissionMode.Byte,
|
|
||||||
PipeOptions.Asynchronous);
|
|
||||||
|
|
||||||
WorkerPipeClient client = new(
|
WorkerPipeClient client = new(
|
||||||
connectTimeoutMilliseconds: 5000,
|
connectTimeoutMilliseconds: 5000,
|
||||||
@@ -105,12 +100,7 @@ public sealed class WorkerPipeClientTests
|
|||||||
|
|
||||||
await Task.Delay(150);
|
await Task.Delay(150);
|
||||||
|
|
||||||
using NamedPipeServerStream server = new(
|
using NamedPipeServerStream server = TestNamedPipe.CreateServer(pipeName);
|
||||||
pipeName,
|
|
||||||
PipeDirection.InOut,
|
|
||||||
1,
|
|
||||||
PipeTransmissionMode.Byte,
|
|
||||||
PipeOptions.Asynchronous);
|
|
||||||
|
|
||||||
await Task.Factory.FromAsync(server.BeginWaitForConnection, server.EndWaitForConnection, null);
|
await Task.Factory.FromAsync(server.BeginWaitForConnection, server.EndWaitForConnection, null);
|
||||||
|
|
||||||
|
|||||||
@@ -2405,12 +2405,7 @@ public sealed class WorkerPipeSessionTests
|
|||||||
public static async Task<PipePair> CreateAsync(CancellationToken cancellationToken)
|
public static async Task<PipePair> CreateAsync(CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
string pipeName = $"mxaccessgw-worker-session-tests-{Guid.NewGuid():N}";
|
string pipeName = $"mxaccessgw-worker-session-tests-{Guid.NewGuid():N}";
|
||||||
NamedPipeServerStream gatewayStream = new(
|
NamedPipeServerStream gatewayStream = TestNamedPipe.CreateServer(pipeName);
|
||||||
pipeName,
|
|
||||||
PipeDirection.InOut,
|
|
||||||
maxNumberOfServerInstances: 1,
|
|
||||||
PipeTransmissionMode.Byte,
|
|
||||||
PipeOptions.Asynchronous);
|
|
||||||
NamedPipeClientStream workerStream = new(
|
NamedPipeClientStream workerStream = new(
|
||||||
".",
|
".",
|
||||||
pipeName,
|
pipeName,
|
||||||
|
|||||||
@@ -0,0 +1,42 @@
|
|||||||
|
using System.IO.Pipes;
|
||||||
|
|
||||||
|
namespace ZB.MOM.WW.MxGateway.Worker.Tests.TestSupport;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Creates the gateway-side <see cref="NamedPipeServerStream"/> the worker tests connect to, with an
|
||||||
|
/// explicit OS buffer so a test that stops draining the pipe cannot block its own writer forever.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// The <c>NamedPipeServerStream(string, PipeDirection, int, PipeTransmissionMode, PipeOptions)</c>
|
||||||
|
/// overload passes <c>inBufferSize: 0</c> and <c>outBufferSize: 0</c> to <c>CreateNamedPipe</c>, which
|
||||||
|
/// on Windows reserves no buffer at all: a write completes only once the peer reads it. Measured on a
|
||||||
|
/// Windows 10 host, the first 63-byte write to a non-reading peer never completed, against 65 520
|
||||||
|
/// bytes absorbed by the same pipe declared with 64 KiB buffers. A test-side write with no timeout
|
||||||
|
/// against a stopped reader therefore hangs the test method, and a test method that never returns
|
||||||
|
/// keeps xUnit from raising <c>ITestAssemblyFinished</c> — so <c>testhost</c> never exits even after
|
||||||
|
/// every other test has passed. This suite runs only on Windows, so it has no macOS Unix-domain-socket
|
||||||
|
/// buffer to hide behind. See the matching helper in the gateway test project.
|
||||||
|
/// </remarks>
|
||||||
|
internal static class TestNamedPipe
|
||||||
|
{
|
||||||
|
/// <summary>
|
||||||
|
/// The buffer reserved for each direction. Comfortably larger than any frame volume a test
|
||||||
|
/// pushes at a stopped reader, and the size Windows itself uses for a typical named pipe.
|
||||||
|
/// </summary>
|
||||||
|
internal const int BufferBytes = 64 * 1024;
|
||||||
|
|
||||||
|
/// <summary>Creates the gateway-side server pipe for <paramref name="pipeName"/>.</summary>
|
||||||
|
/// <param name="pipeName">The pipe name the worker under test connects to.</param>
|
||||||
|
/// <returns>An asynchronous, byte-mode server pipe with explicit buffers.</returns>
|
||||||
|
internal static NamedPipeServerStream CreateServer(string pipeName)
|
||||||
|
{
|
||||||
|
return new NamedPipeServerStream(
|
||||||
|
pipeName,
|
||||||
|
PipeDirection.InOut,
|
||||||
|
1,
|
||||||
|
PipeTransmissionMode.Byte,
|
||||||
|
PipeOptions.Asynchronous,
|
||||||
|
BufferBytes,
|
||||||
|
BufferBytes);
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user