test(windev): buffer the test named pipes so the full-suite testhost exits
The windev full-suite wedge — every test reported, then the x64 testhost sitting at ~0 CPU forever while `dotnet test` never returns — was one test blocked on a pipe write, not a leaked thread or an undisposed fixture. `dotnet-stack report` on the wedged host showed no thread running test code: xUnit's RunTestsInAssembly was parked on WaitHandle.WaitOne() waiting for the assembly-finished event, so the wait lived in a suspended async state machine. `dotnet-dump analyze -c dumpasync` named the frame — WorkerClientTests .StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout awaiting WorkerFrameWriter.WriteAsync on a 63-byte frame, with <extra>5__15 = 6, i.e. the seventh of the twelve events the test pushes past the client's staging bound. That test faults the worker client on purpose, and a faulted client stops its read loop by design. The test-side pipe came from the NamedPipeServerStream overload without buffer arguments, which passes inBufferSize: 0 / outBufferSize: 0 to CreateNamedPipe; on Windows that reserves no buffer at all, so a write completes only once the peer reads it. Measured on windev, that pipe absorbed 0 bytes against a non-reading peer where the same pipe declared with 64 KiB buffers absorbed 65 520. On macOS and Linux .NET backs named pipes with Unix domain sockets whose socket buffer swallows the writes regardless, which is why the identical test never hung there and the bug read as environmental. Test-owned server pipes now go through TestSupport/TestNamedPipe.CreateServer in both test projects, declaring explicit 64 KiB buffers so those tests exercise the gateway's own staging/queue backpressure rather than the OS pipe's flow control. Separately, every fake-worker write in WorkerClientTests now goes through PipePair.WriteAsync, bounded by the class's five-second TestTimeout. That is where the severity came from: a test method that never returns keeps xUnit from raising ITestAssemblyFinished, so one unbounded await cost the whole suite its result. A blocked write is now a named test failure instead of a silent wedge. The fix also retires a wrong belief the wedge had created. windev reported 855 where macOS reported 879, and that gap was recorded in GatewayTesting.md as Unix-gated test cases; it was really the results lost when the wedged host was torn down. The same clone now reports 879 passed, matching macOS exactly. Product code is unaffected. SessionWorkerClientFactory.CreatePipe keeps the unbuffered declaration deliberately: both ends run continuous read loops and every gateway write is bounded by the worker client's _stopCts, so a stalled peer cancels the write rather than blocking on it. Verified on windev at this SHA: gateway suite x64 three times (879 passed, exit 0, no surviving testhost each time) and Worker.Tests x86 twice (400 passed, 11 skipped, exit 0, clean), plus the macOS gateway suite once (879 passed). Docs: GatewayTesting.md replaces the --blame-hang workaround section with the root cause and corrects the baseline to 879, CLAUDE.md's Source Update Workflow no longer tells readers the windev suite wedges, and ToolchainLinks.md records dotnet-stack and dotnet-dump as installed on windev.
This commit is contained in:
+53
-16
@@ -497,10 +497,14 @@ Run it from an isolated clone under `C:\build` checked out to the SHA under test
|
||||
the dirty Desktop checkout, and never the CI clone `C:\build\mxaccessgw-ci`, whose worktree
|
||||
lock belongs to the Worker tier.
|
||||
|
||||
Baseline on an otherwise idle windev (2026-08-10): **855 passed, 0 failed, 29 s**. The
|
||||
suite is smaller there than the 879 the macOS box runs because some cases are gated to
|
||||
Unix. Any failure is therefore a real signal — but read the load caveat below before
|
||||
acting on one.
|
||||
Baseline on an otherwise idle windev (2026-08-10): **879 passed, 0 failed, 31 s** — the same
|
||||
879 the macOS box runs, with nothing gated away. Any failure is therefore a real signal, but
|
||||
read the load caveat below before acting on one.
|
||||
|
||||
Runs before the pipe-buffer fix below reported 855, which was long read as "windev runs a
|
||||
smaller suite because some cases are gated to Unix". It was not: 855 is simply what had been
|
||||
flushed when the wedged host was torn down. Do not treat a short count on this suite as
|
||||
platform gating.
|
||||
|
||||
### Two long-standing "windev-environmental" failures were test bugs, not the environment
|
||||
|
||||
@@ -529,7 +533,7 @@ These suites drive real named pipes against a five-second worker startup timeout
|
||||
failing when windev is busy — most often when the x86 Worker tier is building or testing at
|
||||
the same time. All five passed in the idle baseline above and all five failed in a run taken
|
||||
while an x86 build and `Worker.Tests` were in flight (that run also took 2 m 21 s against the
|
||||
idle 29 s):
|
||||
idle half-minute):
|
||||
|
||||
- `GatewayEndToEndFakeWorkerSmokeTests`, `GatewayEndToEndMultiSubscriberTests`,
|
||||
`GatewayEndToEndReconnectReplayTests` — fail as
|
||||
@@ -550,22 +554,55 @@ windev has 36 logical CPUs and `xunit.runner.json` sets `maxParallelThreads: -1`
|
||||
suite runs far wider there than on the macOS dev box — that width is what turns these
|
||||
real-clock deadlines into failures.
|
||||
|
||||
### The full-suite testhost does not exit on windev
|
||||
### The full-suite testhost hang was a zero-buffer named pipe (fixed)
|
||||
|
||||
After the last test completes, the x64 `testhost` process stops doing work but never exits,
|
||||
so `dotnet test` never returns and the run has to be killed. This does **not** happen on
|
||||
filtered runs (`--filter …`), which exit normally, and does not happen on the macOS dev box —
|
||||
it is specific to a full-suite run on windev. Run the full suite with a hang guard so the
|
||||
wedged host is torn down and the pass/fail summary is still printed:
|
||||
For months a full-suite run on windev reported `855 passed, 0 failed` and then never
|
||||
returned: the x64 `testhost` stopped consuming CPU but stayed alive indefinitely, and the run
|
||||
had to be killed with `--blame-hang`. That guard is no longer needed — run the suite plainly:
|
||||
|
||||
```powershell
|
||||
dotnet test src\ZB.MOM.WW.MxGateway.Tests\ZB.MOM.WW.MxGateway.Tests.csproj `
|
||||
--blame-hang --blame-hang-timeout 5m --blame-hang-dump-type none
|
||||
dotnet test src\ZB.MOM.WW.MxGateway.Tests\ZB.MOM.WW.MxGateway.Tests.csproj
|
||||
```
|
||||
|
||||
The summary line ahead of the abort is the real result; the process exit code is nonzero
|
||||
because of the abort even when every test passed, so read the summary rather than the exit
|
||||
code. Prefer filtered runs on windev whenever the change under test allows it.
|
||||
The cause is worth recording because the shape of it is easy to hit again.
|
||||
|
||||
`dotnet-stack report` on the wedged host showed no thread running test code; xUnit's
|
||||
`RunTestsInAssembly` was simply parked on `WaitHandle.WaitOne()` waiting for the
|
||||
assembly-finished event. The wait was therefore in a suspended async state machine, which only
|
||||
`dotnet-dump analyze <dump> -c dumpasync` can see. It named the exact frame:
|
||||
`WorkerClientTests.StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout`
|
||||
awaiting `WorkerFrameWriter.WriteAsync` — a 63-byte pipe write that never completed. The `855`
|
||||
was never the whole suite: the same clone now reports 879, so the wedge was also costing 24
|
||||
results, and the summary still looked clean because the hung test is not counted as a failure.
|
||||
|
||||
That test pushes events past the worker client's staging bound to prove the client faults, and
|
||||
after the fault the client's read loop stops reading by design. The test-side pipe was created
|
||||
through `NamedPipeServerStream(string, PipeDirection, int, PipeTransmissionMode, PipeOptions)`,
|
||||
whose omitted buffer arguments become `inBufferSize: 0` / `outBufferSize: 0`. On Windows that
|
||||
reserves *no* buffer: a write completes only when the peer reads it. Measured directly on
|
||||
windev, that pipe absorbed **0 bytes** before blocking against a non-reading peer, while the
|
||||
same pipe declared with 64 KiB buffers absorbed **65 520**. On macOS and Linux .NET backs named
|
||||
pipes with Unix domain sockets, whose socket buffer swallows a few kilobytes regardless — which
|
||||
is why the identical test never hung there, and why the bug read as "a windev thing".
|
||||
|
||||
Two changes make it structural rather than incidental:
|
||||
|
||||
- Test-owned server pipes are created through `TestSupport/TestNamedPipe.CreateServer`, which
|
||||
declares explicit 64 KiB buffers, in both the gateway and worker test projects. This scopes
|
||||
those tests to the backpressure they are actually asserting — the gateway's staging and event
|
||||
queues — instead of the OS pipe's flow control.
|
||||
- Every fake-worker write in `WorkerClientTests` goes through `PipePair.WriteAsync`, which
|
||||
bounds the write by the class's five-second `TestTimeout` and fails with a message naming the
|
||||
stopped reader. A blocked write is now a named test failure rather than a silent wedge.
|
||||
|
||||
The severity came from the second point being missing, not the first. A test method that never
|
||||
returns keeps xUnit from raising `ITestAssemblyFinished`, so the runner waits forever and
|
||||
`testhost` never exits — one unbounded `await` in one test costs the entire suite its result.
|
||||
Any new test that writes to a pipe whose reader may stop must bound the write.
|
||||
|
||||
The gateway's production pipe in `SessionWorkerClientFactory.CreatePipe` deliberately keeps the
|
||||
unbuffered declaration: both ends run continuous read loops and every write there is bounded by
|
||||
the worker client's `_stopCts`, so a stalled peer cancels the write instead of blocking on it.
|
||||
|
||||
## Continuous Integration
|
||||
|
||||
|
||||
@@ -37,6 +37,16 @@ $env:Path = [Environment]::GetEnvironmentVariable('Path','Machine') + ';' + [Env
|
||||
| C compiler x86 | 14.44.35207 | `C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\bin\Hostx64\x86\cl.exe` |
|
||||
| Linker x86 | 14.44.35207 | `C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\bin\Hostx64\x86\link.exe` |
|
||||
| LibMan CLI | 3.0.71 | `C:\Users\dohertj2\.dotnet\tools\libman.exe` |
|
||||
| dotnet-stack | 9.0.661903 | `C:\Users\dohertj2\.dotnet\tools\dotnet-stack.exe` |
|
||||
| dotnet-dump | 9.0.661903 | `C:\Users\dohertj2\.dotnet\tools\dotnet-dump.exe` |
|
||||
|
||||
`dotnet-stack` and `dotnet-dump` are the diagnostics pair for a process that stops
|
||||
making progress but does not exit. `dotnet-stack report -p <pid>` prints every
|
||||
managed thread's stack, which is enough when a *thread* is blocked; when nothing is
|
||||
on a thread the wait lives in a suspended async state machine, and only
|
||||
`dotnet-dump collect -p <pid>` followed by `dotnet-dump analyze <dump> -c dumpasync`
|
||||
reveals it. Both were installed user-local with `dotnet tool install -g` while
|
||||
root-causing the windev test-host hang described in `docs/GatewayTesting.md`.
|
||||
|
||||
Reference assemblies:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user