test(windev): buffer the test named pipes so the full-suite testhost exits
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m14s
ci / java (push) Successful in 2m19s
ci / portable (push) Successful in 8m48s

The windev full-suite wedge — every test reported, then the x64 testhost sitting
at ~0 CPU forever while `dotnet test` never returns — was one test blocked on a
pipe write, not a leaked thread or an undisposed fixture.

`dotnet-stack report` on the wedged host showed no thread running test code:
xUnit's RunTestsInAssembly was parked on WaitHandle.WaitOne() waiting for the
assembly-finished event, so the wait lived in a suspended async state machine.
`dotnet-dump analyze -c dumpasync` named the frame — WorkerClientTests
.StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout awaiting
WorkerFrameWriter.WriteAsync on a 63-byte frame, with <extra>5__15 = 6, i.e. the
seventh of the twelve events the test pushes past the client's staging bound.

That test faults the worker client on purpose, and a faulted client stops its
read loop by design. The test-side pipe came from the NamedPipeServerStream
overload without buffer arguments, which passes inBufferSize: 0 / outBufferSize: 0
to CreateNamedPipe; on Windows that reserves no buffer at all, so a write
completes only once the peer reads it. Measured on windev, that pipe absorbed
0 bytes against a non-reading peer where the same pipe declared with 64 KiB
buffers absorbed 65 520. On macOS and Linux .NET backs named pipes with Unix
domain sockets whose socket buffer swallows the writes regardless, which is why
the identical test never hung there and the bug read as environmental.

Test-owned server pipes now go through TestSupport/TestNamedPipe.CreateServer in
both test projects, declaring explicit 64 KiB buffers so those tests exercise the
gateway's own staging/queue backpressure rather than the OS pipe's flow control.

Separately, every fake-worker write in WorkerClientTests now goes through
PipePair.WriteAsync, bounded by the class's five-second TestTimeout. That is
where the severity came from: a test method that never returns keeps xUnit from
raising ITestAssemblyFinished, so one unbounded await cost the whole suite its
result. A blocked write is now a named test failure instead of a silent wedge.

The fix also retires a wrong belief the wedge had created. windev reported 855
where macOS reported 879, and that gap was recorded in GatewayTesting.md as
Unix-gated test cases; it was really the results lost when the wedged host was
torn down. The same clone now reports 879 passed, matching macOS exactly.

Product code is unaffected. SessionWorkerClientFactory.CreatePipe keeps the
unbuffered declaration deliberately: both ends run continuous read loops and every
gateway write is bounded by the worker client's _stopCts, so a stalled peer
cancels the write rather than blocking on it.

Verified on windev at this SHA: gateway suite x64 three times (879 passed,
exit 0, no surviving testhost each time) and Worker.Tests x86 twice (400 passed,
11 skipped, exit 0, clean), plus the macOS gateway suite once (879 passed).

Docs: GatewayTesting.md replaces the --blame-hang workaround section with the
root cause and corrects the baseline to 879, CLAUDE.md's Source Update Workflow
no longer tells readers the windev suite wedges, and ToolchainLinks.md records
dotnet-stack and dotnet-dump as installed on windev.
This commit is contained in:
Joseph Doherty
2026-08-10 09:55:58 -04:00
parent f78781d9ef
commit bfcf82975c
9 changed files with 217 additions and 70 deletions
+53 -16
View File
@@ -497,10 +497,14 @@ Run it from an isolated clone under `C:\build` checked out to the SHA under test
the dirty Desktop checkout, and never the CI clone `C:\build\mxaccessgw-ci`, whose worktree
lock belongs to the Worker tier.
Baseline on an otherwise idle windev (2026-08-10): **855 passed, 0 failed, 29 s**. The
suite is smaller there than the 879 the macOS box runs because some cases are gated to
Unix. Any failure is therefore a real signal — but read the load caveat below before
acting on one.
Baseline on an otherwise idle windev (2026-08-10): **879 passed, 0 failed, 31 s** — the same
879 the macOS box runs, with nothing gated away. Any failure is therefore a real signal, but
read the load caveat below before acting on one.
Runs before the pipe-buffer fix below reported 855, which was long read as "windev runs a
smaller suite because some cases are gated to Unix". It was not: 855 is simply what had been
flushed when the wedged host was torn down. Do not treat a short count on this suite as
platform gating.
### Two long-standing "windev-environmental" failures were test bugs, not the environment
@@ -529,7 +533,7 @@ These suites drive real named pipes against a five-second worker startup timeout
failing when windev is busy — most often when the x86 Worker tier is building or testing at
the same time. All five passed in the idle baseline above and all five failed in a run taken
while an x86 build and `Worker.Tests` were in flight (that run also took 2 m 21 s against the
idle 29 s):
idle half-minute):
- `GatewayEndToEndFakeWorkerSmokeTests`, `GatewayEndToEndMultiSubscriberTests`,
`GatewayEndToEndReconnectReplayTests` — fail as
@@ -550,22 +554,55 @@ windev has 36 logical CPUs and `xunit.runner.json` sets `maxParallelThreads: -1`
suite runs far wider there than on the macOS dev box — that width is what turns these
real-clock deadlines into failures.
### The full-suite testhost does not exit on windev
### The full-suite testhost hang was a zero-buffer named pipe (fixed)
After the last test completes, the x64 `testhost` process stops doing work but never exits,
so `dotnet test` never returns and the run has to be killed. This does **not** happen on
filtered runs (`--filter …`), which exit normally, and does not happen on the macOS dev box —
it is specific to a full-suite run on windev. Run the full suite with a hang guard so the
wedged host is torn down and the pass/fail summary is still printed:
For months a full-suite run on windev reported `855 passed, 0 failed` and then never
returned: the x64 `testhost` stopped consuming CPU but stayed alive indefinitely, and the run
had to be killed with `--blame-hang`. That guard is no longer needed — run the suite plainly:
```powershell
dotnet test src\ZB.MOM.WW.MxGateway.Tests\ZB.MOM.WW.MxGateway.Tests.csproj `
--blame-hang --blame-hang-timeout 5m --blame-hang-dump-type none
dotnet test src\ZB.MOM.WW.MxGateway.Tests\ZB.MOM.WW.MxGateway.Tests.csproj
```
The summary line ahead of the abort is the real result; the process exit code is nonzero
because of the abort even when every test passed, so read the summary rather than the exit
code. Prefer filtered runs on windev whenever the change under test allows it.
The cause is worth recording because the shape of it is easy to hit again.
`dotnet-stack report` on the wedged host showed no thread running test code; xUnit's
`RunTestsInAssembly` was simply parked on `WaitHandle.WaitOne()` waiting for the
assembly-finished event. The wait was therefore in a suspended async state machine, which only
`dotnet-dump analyze <dump> -c dumpasync` can see. It named the exact frame:
`WorkerClientTests.StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout`
awaiting `WorkerFrameWriter.WriteAsync` — a 63-byte pipe write that never completed. The `855`
was never the whole suite: the same clone now reports 879, so the wedge was also costing 24
results, and the summary still looked clean because the hung test is not counted as a failure.
That test pushes events past the worker client's staging bound to prove the client faults, and
after the fault the client's read loop stops reading by design. The test-side pipe was created
through `NamedPipeServerStream(string, PipeDirection, int, PipeTransmissionMode, PipeOptions)`,
whose omitted buffer arguments become `inBufferSize: 0` / `outBufferSize: 0`. On Windows that
reserves *no* buffer: a write completes only when the peer reads it. Measured directly on
windev, that pipe absorbed **0 bytes** before blocking against a non-reading peer, while the
same pipe declared with 64 KiB buffers absorbed **65 520**. On macOS and Linux .NET backs named
pipes with Unix domain sockets, whose socket buffer swallows a few kilobytes regardless — which
is why the identical test never hung there, and why the bug read as "a windev thing".
Two changes make it structural rather than incidental:
- Test-owned server pipes are created through `TestSupport/TestNamedPipe.CreateServer`, which
declares explicit 64 KiB buffers, in both the gateway and worker test projects. This scopes
those tests to the backpressure they are actually asserting — the gateway's staging and event
queues — instead of the OS pipe's flow control.
- Every fake-worker write in `WorkerClientTests` goes through `PipePair.WriteAsync`, which
bounds the write by the class's five-second `TestTimeout` and fails with a message naming the
stopped reader. A blocked write is now a named test failure rather than a silent wedge.
The severity came from the second point being missing, not the first. A test method that never
returns keeps xUnit from raising `ITestAssemblyFinished`, so the runner waits forever and
`testhost` never exits — one unbounded `await` in one test costs the entire suite its result.
Any new test that writes to a pipe whose reader may stop must bound the write.
The gateway's production pipe in `SessionWorkerClientFactory.CreatePipe` deliberately keeps the
unbuffered declaration: both ends run continuous read loops and every write there is bounded by
the worker client's `_stopCts`, so a stalled peer cancels the write instead of blocking on it.
## Continuous Integration
+10
View File
@@ -37,6 +37,16 @@ $env:Path = [Environment]::GetEnvironmentVariable('Path','Machine') + ';' + [Env
| C compiler x86 | 14.44.35207 | `C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\bin\Hostx64\x86\cl.exe` |
| Linker x86 | 14.44.35207 | `C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\bin\Hostx64\x86\link.exe` |
| LibMan CLI | 3.0.71 | `C:\Users\dohertj2\.dotnet\tools\libman.exe` |
| dotnet-stack | 9.0.661903 | `C:\Users\dohertj2\.dotnet\tools\dotnet-stack.exe` |
| dotnet-dump | 9.0.661903 | `C:\Users\dohertj2\.dotnet\tools\dotnet-dump.exe` |
`dotnet-stack` and `dotnet-dump` are the diagnostics pair for a process that stops
making progress but does not exit. `dotnet-stack report -p <pid>` prints every
managed thread's stack, which is enough when a *thread* is blocked; when nothing is
on a thread the wait lives in a suspended async state machine, and only
`dotnet-dump collect -p <pid>` followed by `dotnet-dump analyze <dump> -c dumpasync`
reveals it. Both were installed user-local with `dotnet tool install -g` while
root-causing the windev test-host hang described in `docs/GatewayTesting.md`.
Reference assemblies: