Closes the 5 deterministically-race-hard test gaps that were previously
documented as known omissions (#5–9 in codereviews/2026-05-14/RemediationPlan.md).
Tests: 387 pass / 0 fail (baseline 382 + 5 new race tests). Three back-to-back
runs in isolation all green — no observable flakes.
Each test reaches the relevant code path deterministically by either:
- reaching into the multiplexer's private state via reflection (only used
for pre-saturating the TxIdAllocator — the test path that's externally
impossible to hit otherwise without spawning 65,536 real connections),
- constructing a backend stub that exercises the timing window directly, or
- asserting only the externally-observable contract that holds across all
valid interleavings (no-double-delivery, no-hang) rather than asserting
a specific ordering that flakes.
W3 #5 — TxIdAllocator_Saturated_NextRequest_GetsException04_WithOriginalTxId
Pre-saturates the multiplexer's _allocator via reflection (TryAllocate
×65536), then sends one FC06 write. The next request hits the
!_allocator.TryAllocate branch immediately and the test verifies exception
04 with the original TxId echoed.
W3 #6 — TxIdAllocator_Saturated_TwoConcurrentIdenticalReads_BothPipesGetException04
Pre-saturates the allocator, then fires two concurrent identical FC03 reads
from two pipes. Both pipes must receive exception 04 — regardless of whether
pipe B coalesces onto pipe A's stub (W1.2's deliver-to-late-attachers path)
OR opens its own factory failure path. The contract verified is "no late
attacher hangs" — the externally-observable invariant from the W1.2 fix.
W3 #7 — SlowUpstream_DoesNotStallPeerResponses_DropCounterIncrements
Wedges upstream A by leaving its socket-receive side undrained, pumps 500
FC03 requests through A so the bounded response channel + kernel buffer
fill, then sends one request from a healthy upstream B. B's response must
arrive within seconds (would block forever pre-W1.3) and A's
ResponseDropForFullUpstream counter must increment — proving the W1.3
TrySendResponse non-blocking fan-out works as designed.
W3 #8 — WatchdogVsResponse_Race_AlwaysExactlyOneOutcome_PerRequest
Custom SlowResponseBackend stub responds at a randomized 350–450 ms delay
while BackendRequestTimeoutMs=400. Across 30 iterations, the request races
the watchdog's per-tick scan. The contract asserts: every request gets
exactly ONE response (normal or exception 0x0B), the original TxId is
always echoed, and BOTH branches are exercised (proving the race window is
real). The W1 claim-then-dispatch design (CorrelationMap.TryRemove as the
single source of truth) makes this contract hold across all interleavings.
W3 #9 — CascadeVsNewAccept_StressChurn_NoCrash_NoHang
Stress-test: 3 cascade cycles, 8 concurrent connect+request tasks per
cycle. Backend is killed mid-cascade-storm to force teardown to race the
in-flight connect attempts. After all churn the multiplexer must still
serve a normal request. The originally-flagged race (a pipe added between
_pipes.Values.ToArray() and _pipes.Clear() in TearDownBackendAsync) is
microseconds wide and not deterministically reproducible without test
seams; this stress test instead proves the no-crash-under-churn property
that operators care about.
Helpers added:
DrainAllocator(PlcMultiplexer) — reflection-based saturation primitive,
only used by tests #5 and #6.
SlowResponseBackend — backend stub with caller-supplied per-request delay
via a Func<int>, only used by test #8.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Joseph Doherty
53f842a655