Add MxAccessWriteCompletionCache.Remove and RemoveServer and call them
from AlarmAckWriter when it drops an item (dirty re-add, LRU eviction,
failed AddItem/Advise/Write) and when it tears down its registration, so
the cache no longer keeps one entry per dropped handle for the life of
the worker.
Fixes#165
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
Resolve the operator user, password and comment before the interop try
blocks, and match ArgumentException as an MXAccess failure only when it
carries E_INVALIDARG and is not ArgumentNullException or
ArgumentOutOfRangeException. A programming error inside the try blocks
now propagates instead of being reported as an authentication or write
failure.
Fixes#164
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
- AlarmAckWriter: classify AuthenticateUser and AddItem/Advise/Write
failures by HRESULT for any interop exception (ArgumentException for
E_INVALIDARG, UnauthorizedAccessException, COMException); server-gone
HRESULTs at AuthenticateUser tear down the registration and report
WriteRejected.
- Bound the ack item cache with a 256-entry LRU (eviction best-effort
UnAdvise+RemoveItem).
- After an Unconfirmed ack, drop and re-add the item before the next
write so a late OnWriteComplete on the old handle is not credited.
- Pass StaRuntime.PumpPendingMessages to the ack writer through a
pump-aware alarm handler factory so long completion waits refresh the
STA watchdog heartbeat.
- Rust CLI: --operator is required for acknowledge-alarm.
- Docs: full ack status table, credential-oracle trade-off, upgrade note.
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
AlarmCommandHandler -> AlarmDispatcher -> IAlarmAckWriter: the dispatcher resolves the ack item through the consumer and the writer authenticates the operator and writes the comment. The executor builds AlarmAckOperator from operator_user/domain/password, maps AlarmAckOutcome to the reply (Applied/Unconfirmed Ok, AlarmNotFound/NotAckable InvalidRequest, AuthenticationFailed/WriteRejected MxaccessFailure) and reports escaping exceptions by type and HRESULT only, so operator_password cannot leak. Removes the inert managed/WNAL alarm-client ack path.
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
IMxAccessAlarmConsumer.ResolveAckTarget/ResolveAckTargetByName replace AcknowledgeByGuid/AcknowledgeByName. wnwrap maps Galaxy-provider alarms to <tag>.AckMsg and drops its inert parallel ack consumer; subtag resolves to the target's AckCommentSubtag and no longer advises or writes it; ISubtagAlarmSource.Write is removed. Does not build alone: AlarmDispatcher moves to the new API in the next commit.
Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT
Review fixes for the opt-in managed GUID ack:
- Subtag-mode GUID acks (forced subtag, or failover on the standby) stay on
the consumer; their GUIDs are synthetic and only the subtag consumer can
resolve them. Forced-subtag sessions no longer build a managed client.
- A failed managed Initialize/Register/Subscribe drops back to the wnwrap
ack path instead of failing the alarm subscription.
- Drop stale crefs to a nonexistent Unverified member in
WnalNativeAckConsumer.
- Unit tests for routing, fallback, and disposal; document
MXGATEWAY_USE_MANAGED_ALARM_ACK.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds a managed acknowledge path over AVEVA's modern alarm client so GUID
acks carry full operator identity (domain + full name) and work on builds
where the wnwrap COM AlarmAckByGUID is E_NOTIMPL. Default off; opt in with
MXGATEWAY_USE_MANAGED_ALARM_ACK=1 (wnwrap behavior unchanged otherwise).
- AaManagedAlarmAckConsumer: reflection wrapper over
aaAlarmManagedClient.AlarmClient (InitializeConsumer/RegisterConsumer/
Subscribe/AlarmAckByGUID); EnvGatedFactory degrades to null (wnwrap) if the
DLL can't load, so alarm reads never break.
- IManagedAlarmAckClient seam. AlarmDispatcher takes an optional client and
routes GUID acks to it (subscribing + disposing it alongside the reader).
- AlarmCommandHandler gains an optional managedAckFactory; WorkerPipeSession
wires AaManagedAlarmAckConsumer.EnvGatedFactory at the production sites.
- WnalNativeAckConsumer: confirmed native alarmlst.dll WNAL submit layer
(WNALalarmReqSetText / WNALUserRequest / WNALUserRequestEx + lpAlmAck
layout). List acquisition is intentionally not hand-rolled: the alarm-record
layout and selection semantics are provider-defined and not recoverable from
alarmlst.dll, so inferring them risks acking the wrong alarm.
- AlarmManagedGuidAckLiveSmokeTests: skip-gated live smoke that pulls a real
(non-degraded) alarm GUID and asserts the ack APPLIES (Acknowledge
transition / acked snapshot), not merely rc=0.
By-name acks stay on wnwrap (AlarmClient has no AlarmAckByName). Grounded in
static RE of alarmlst.dll, aaAlarmManagedClient.dll and alarmmgr.exe.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Tracking IDs are review bookkeeping that rots in shipped source; the rationale they tagged is kept. Comment-only change (CommentChecker 458 -> 16 false positives).
The worker budgeted a DrainEvents reply against the negotiated pipe frame
maximum less one 64 KiB reserve, but the gateway returns that same
MxCommandReply verbatim through Invoke, where MaxSendMessageSize applies. The
two limits were coupled only from below (pipe >= gRPC + reserve), so any
configuration the validator permitted with a wider gap let the worker pack a
reply the pipe carried and the gRPC layer then refused with ResourceExhausted --
after Drain had already removed the events from the worker queue, losing them to
both the caller and the event stream.
Option (a) from the finding: close the coupling at both ends.
- GatewayOptionsValidator adds the upper half of the headroom rule
(Worker:MaxMessageBytes <= Protocol:MaxGrpcMessageBytes + 2 x reserve), with a
message naming the DrainEvents budget as the reason and the two knobs to move.
- WorkerPipeSession.ResolveDrainReplyByteBudget subtracts the reserve twice --
once for the pipe frame, once so the MxCommandReply wrapper clears the gRPC
send cap -- keeping the MaxMessageBytes/2 floor that makes small frame maxima
usable.
The two changes are load-bearing for each other: the validator band is what
makes "frame max - 2 x reserve" provably at or below the gRPC cap, and the
doubled subtraction is what makes the band sufficient.
Docs updated in the same change (GatewayConfiguration.md both option rows plus
the headroom prose, WorkerFrameProtocol.md drain budget, gateway.md DrainEvents
byte axis). Tests: new GatewayOptionsValidatorTests cases for the upper bound
(reject) and both edges of the band (accept); GatewayOptionsTests' override case
now moves both message-size knobs together, since the band makes moving one
alone illegal.
The worker half cannot be built on macOS; verified on windev separately.
refreshStaActivityOnCapture was declared between DispatchAsync and the property that
wraps it, rather than with the other instance fields. Moved up to the field block per the
member ordering in docs/style-guides/CSharpStyleGuide.md. Declaration move only — no
behavior, no other edits.
The BlockDispatch branch waited 5 real seconds and then proceeded regardless — it does not
branch on the wait's result. The long-in-flight test's inspection loop is bounded by
elapsed time and a frame floor, so on a loaded box (the documented 4-5x slowdown class) it
can plausibly outrun that 5 s. When it does, the reply is emitted mid-window,
AssertNotWorkerFault waves it past, and the reply leg then waits for a reply already gone
by — failing at the 20 s cancellation with no message, on exactly the loaded-box run the
widened windows exist to survive.
The wait is a pure safety net: nothing asserts on it firing, and every test that blocks
dispatch releases it explicitly (ReleaseDispatch, or a WorkerShutdown envelope, both of
which Set the event) — none reaches the timeout on a healthy run. Named it
BlockedDispatchSafetyNet and raised it to 30 s, above any window a test opens and above the
20 s cancellation those tests arm, so a wedged test always fails on its own token with its
own message. Dispose still releases the wait, so teardown never waits on it either.
Inline rationale in the test now states the decoupling and what a close pairing would cost,
rather than asserting the window stays inside a 5 s ceiling. Nothing else changed.
Three review findings on 7da52b6:
1. The AssertNotFault doc block landed between the predicate-overload ReadUntilAsync's
doc block and the helper, so the compiler attached the merged block to the helper and
ReadUntilAsync lost its docs entirely. Helper and its docs moved above ReadUntilAsync,
whose docs are back where they belong.
2. The compressed watchdog windows (50 ms grace, 100 ms ceiling) reintroduced the
load-sensitivity the fix removed, one layer down.
ReportWatchdogFaultIfNeededAsync measures staleness AFTER the heartbeat frame is
written and flushed over the real named pipe, so a beat whose pipe I/O outlasts the
ceiling faults a healthy session no matter how fresh the captured activity was. At
100 ms that is a plausible stall on a loaded box, and this is the one test asserting
the watchdog NEVER fires. Widened to a 200 ms grace and a 1 s ceiling — still two
orders of magnitude under the 75 s production default.
The inspection loop is now bounded by a 2 s window (twice the ceiling, so a fake whose
activity stopped advancing still accumulates past it and faults) with a 30-frame floor,
rather than a fixed 30 frames that no longer outran the wider ceiling. The floor keeps
a window that saw almost no beats from passing as a clean one. Two seconds stays well
inside FakeRuntimeSession's 5 s dispatch-block ceiling, so the command is still in
flight for the whole window.
3. AssertNotFault renamed AssertNotWorkerFault, matching the WorkerFault body case it
tests.
Scenario intent unchanged: long in-flight command, pump refreshing, zero fault frames,
reply delivered. Test-only.
RunAsync_LongInFlightCommandThatKeepsPumping_DoesNotFaultAndDeliversReply failed
deterministically on the Windows box with a StaHung fault whose command_method was
empty and whose staleness was 233 ms — i.e. a fault raised with no command in flight,
before the scenario under test began. FakeRuntimeSession stamps LastStaActivityUtc once,
at construction, and the test only started refreshing it after the blocked dispatch
signalled. Everything between those two points — handshake, STA init, the first
heartbeat — captured a snapshot already stale past the compressed 50 ms grace, with no
correlation id for the watchdog to suppress on, so the watchdog correctly reported the
fake as hung.
The harness, not the product, was wrong: StaRuntime.ThreadMain calls MarkActivity() on
every WaitForWorkOrMessages iteration, so a live worker is never captured stale, idle or
busy. Model that where it belongs — FakeRuntimeSession.RefreshStaActivityOnCapture (opt
in, default off) stamps activity at each CaptureHeartbeat and leaves the rest of the
snapshot alone — and arm it before RunAsync so the first beat is covered. The test-owned
refresh loop goes away with it; a thread-pool loop racing a compressed grace could not
have held the invariant anyway.
Scenario intent is unchanged and slightly stronger: the command still blocks in dispatch
across 30 heartbeats (~600 ms, many multiples of the 100 ms stuck ceiling), no frame may
be a fault, and the reply must still arrive. The reply leg is now fault-checked too
(previously it skipped frames blindly), and the pump keeps running across the release, as
it does in production while the reply is marshalled off the STA. Fault assertions now
report the category and diagnostic message instead of a bare body-case mismatch.
Test-only change; no product code, frame protocol, or STA rule touched.
Review found AlarmDispatcher.SnapshotActiveAlarms reading the snapshot and the
truncation verdict through two independent lock acquisitions, defended by a
comment claiming read-order made a race "widen only, never narrow". That claim
was false: a not-truncated -> truncated poll landing between the two reads pairs
a stale false with a capped snapshot, which is exactly the false all-clear the
feature exists to prevent. It was safe only because AlarmCommandHandler
STA-serializes consumer calls — an accident of the call graph, not an invariant.
Made the invariant structural instead of documented. IMxAccessAlarmConsumer now
exposes ONE accessor, `IReadOnlyList<MxAlarmSnapshotRecord> SnapshotActiveAlarms(
out bool truncated)`, which implementations must satisfy from a single
acquisition of the lock guarding the retained snapshot — mirroring the write
side, where FoldFetch already updates snapshot and verdict together. The
separate LastSnapshotTruncated property is gone from every layer, so there is no
second read left to pair badly. `out` over a result struct follows the file's
established idiom (FoldFetch, ParseSnapshotXml).
The same threading applies one level up: IAlarmCommandHandler.QueryActive now
carries `out bool snapshotTruncated`, so MxAccessCommandExecutor stamps the reply
payload from the value the records were stamped with rather than reading the
state a second time.
Direct tests for the three hops that were only covered end-to-end:
- AlarmDispatcherTests: truncated consumer snapshot stamps FromTruncatedSnapshot
on every mapped record, with a complete-snapshot control, plus an assertion
that the independent per-record Degraded flag is not dragged along.
- AlarmCommandHandlerTests: the verdict delegates through the dispatcher
(Theory over both values), and survives a prefix filter that removes every
record — the case the per-record flag cannot cover.
- AlarmCommandExecutorTests: the reply payload's SnapshotTruncated comes from the
handler (Theory over both values), including the zero-record case.
The WnWrapAlarmConsumer truncation tests now assert through
SnapshotActiveAlarms(out ...) rather than an internal field, because the pairing
is the contract.
Also: GatewayAlarmMonitor's _snapshotTruncated comment now says "as of the last
full reconcile" rather than implying it tracks the current _alarms contents,
which live transitions keep moving via ApplyTransition between passes.
Detection heuristic still untouched (fetchedRecordCount >= maxAlarmsPerFetch);
no @COUNT parsing, per docs/AlarmProbeFindings.md. Still additive gateway
metadata about our fetch mechanics, not MXAccess behavior — not a parity
deviation, and no event is synthesized.
Gateway: NonWindows.slnx builds clean (0 warnings); ~Alarm filter 107/107 pass.
Worker + Worker.Tests are windev-gated; the signature change was reviewed by
inspection across all 7 IMxAccessAlarmConsumer implementers, all 3
IAlarmCommandHandler implementers, and every call site.
The truncation-cliff fix made alarm transitions truncation-safe but silent:
when GetXmlCurrentAlarms2 returns exactly maxAlmCnt records the worker
suppresses absence-implies-Clear inference and says so only in a rate-limited
stderr warning. No client and no operator could tell a complete active set
from a capped one.
Two additive proto3 booleans carry the verdict out:
- QueryActiveAlarmsReplyPayload.snapshot_truncated = 2 (worker IPC reply)
- ActiveAlarmSnapshot.from_truncated_snapshot = 16 (per record)
The per-record field is not an aesthetic choice. QueryActiveAlarms returns a
bare `stream ActiveAlarmSnapshot` with no envelope, header, or trailer, so a
per-record boolean is the only carrier that stays wire-compatible; an envelope
message would change every existing client's stream element type. The reply
payload states it too because a prefix filter can leave zero records and a
truncated fetch with nothing to report still has to say so. The flag means
"this set may be incomplete", never "this record is unreliable" — it is
independent of the subtag-fallback `degraded` field.
Detection is deliberately UNCHANGED: IsTruncatedFetch remains
`fetchedRecordCount >= maxAlarmsPerFetch`. The live probe (docs/AlarmProbeFindings.md,
ce5d8ae) could not verify whether ALARM_RECORDS/@COUNT reports the total active
count or only the records in the reply, so @COUNT is not parsed for detection;
switching to it stays blocked on probe evidence. The probe's comment
annotations in WnWrapAlarmConsumer.cs are preserved.
Reset semantics: not latched. WnWrapAlarmConsumer.FoldFetch replaces the
verdict on every poll under the same lock as the snapshot merge, so the first
sub-cap fetch clears it; GatewayAlarmMonitor.ClearCache drops it with the cache
generation it describes. A caveat that never turns off is one operators learn
to ignore.
Flow: WnWrapAlarmConsumer.LastSnapshotTruncated -> AlarmDispatcher (stamps every
record) / IAlarmCommandHandler (payload) -> MxAccessCommandExecutor reply ->
GatewayAlarmMonitor._snapshotTruncated -> IGatewayAlarmService.SnapshotTruncated
-> DashboardAlarmQueryResult -> AlarmsPage warning banner (render-side only; the
poll loop and DisposeAsync drain are untouched). The public QueryActiveAlarms
RPC forwards worker snapshots unmodified, so the per-record flag needed no
mapper change — a test pins that.
Parity: this describes OUR fetch mechanics — additive gateway metadata — not
MXAccess provider behavior. No event is synthesized and no MXAccess-observable
semantics change, so it is not a parity deviation.
Tests: worker LastSnapshotTruncated set/reset/consecutive-burst (windev-run);
gateway end-to-end truncated reply -> monitor -> public stream, with the
complete-reply control as the load-bearing assertion; AlarmsPage banner
present/absent. Docs: gateway.md alarm surface, docs/DesignDecisions.md entry.
WriteAsync enqueued its frame and then contended unconditionally for the write
lock, so a caller that lost the race stayed in WaitAsync until the winning
drainer released — even though that winner writes, flushes, and completes the
loser's control frame at the control-to-event class boundary, part-way through
its pass. The boundary flush made the delivery point honest; the awaited task
was still charged for the whole event backlog it had just been flushed ahead of.
WriteAsync now awaits its own frame's completion racing the lock acquisition.
Completion first: the caller returns at its frame's delivery point and the
outstanding acquisition is detached, not dropped — a continuation drains
whatever is queued and releases, so the lock is never acquired and silently
held and a frame enqueued between the previous drainer's last dequeue and its
release is still written. Lock first: drain as before. Cancellation keeps the
WRK-22 tombstone semantics exactly, and a wait cancelled after the caller has
already detached releases nothing (SemaphoreSlim hands no count to a wait it
cancels), so no count leaks and no queued frame is stranded. A token that fires
after the frame's completion won the race changes nothing — the frame was
delivered. WriteBatchAsync deliberately keeps the plain wait-then-drain shape:
its last completion resolves at the end-of-pass flush anyway.
Three tests: the latency win (a control caller returning while the winning
WriteBatchAsync event burst is demonstrably still blocked mid-pass), a
mixed-priority concurrency soak pinning exactly-once writes and a single
drainer, and the cancel-after-detach corner (a wrongly released count would
surface as the drainer's own Release throwing SemaphoreFullException).
edited on macOS, windev verification pending (plan Task 11). Verified here by
compiling and running WorkerFrameWriter plus the writer suite against net10.0
in a scratch harness: 31/31 pass, and the two behaviour-pinning tests fail
against the pre-change parked implementation.
The two-class writer already got control bytes out ahead of a queued event
backlog, but a frame counts as delivered only once flushed, and the drain
deferred its single FlushAsync — and every TrySetResult — to the end of the
pass. A heartbeat, command reply, fault, or shutdown ack was therefore written
first and completed last, behind up to a full 128-frame event batch.
The drain now records each frame's priority class on PendingFrame and flushes
at every control-to-event boundary, completing and clearing the written set
there. Cost stays bounded: a pure-event pass still pays exactly one flush, a
run of control frames still pays one for the run, and only a pass that mixes
both classes pays a second — never one flush per control frame, the
syscall-per-heartbeat cost WRK-12 removed.
A boundary flush that itself fails is a new failure window and is handled like
the end-of-pass flush failure, additionally failing the event frame the drain
had already claimed off its queue and every frame still queued. Frames a
boundary flush completed leave the written set, so a later failure in the same
pass can no longer reach back and fail an already-delivered control frame.
The awaited task of a caller that lost the write-lock race is still bounded by
the winning drainer's pass — that enqueue-then-contend parking is unchanged and
now documented on WriteAsync and in docs/WorkerFrameProtocol.md.
MxAccessValueCache.Set deep-copied the Value (recursive for an MxArray), the
SourceTimestamp, and the Statuses RepeatedField (container plus every
MxStatusProxy) on every OnDataChange. The aliasing audit found all three
removable: the sink enqueues the event first — which stamps
WorkerSequence/WorkerTimestamp inside the queue lock — and only then runs the
postPublish hook that reaches Set, so the event is write-once by then and the
queue's ownership invariant forbids later mutation. The producer never reuses
instances (fresh MxEvent per mapper call, fresh MxValue per convert), and the
alias already existed on the read side: MxAccessSession.SucceededRead puts the
cache's own Value/SourceTimestamp/status references on every BulkReadResult,
which the worker only serializes onto the IPC pipe.
Set and CachedValue now carry the ownership contract: the cache holds borrowed
references into an enqueued, write-once MxEvent; consumers may read and
serialize, never mutate. Mutation would corrupt the still-queued event AND
invalidate QueuedEvent.Size — the enqueue-time memoized serialized size the
byte-budgeted Drain charges — so a grown message could overshoot the negotiated
frame max and fault the session with MessageTooLarge. MxAccessEventQueue's
class remark, which claimed the cache keeps an independent snapshot, is
corrected to point at the borrow.
MxAccessWriteCompletionCache.Record keeps its parallel statuses.Clone()
deliberately, with a cross-reference explaining why: it takes a bare
RepeatedField whose provenance its signature cannot constrain, and it is on the
command-rate write path, not the streaming hot path.
Tests: Set_StoresIndependentSnapshot_UnaffectedByLaterEventMutation codified
the invariant being reversed, so it is replaced by
Set_BorrowsTheEventsOwnInstances_ByOwnershipContract (Assert.Same on Value,
SourceTimestamp, and the status row). Adds the missing cached-read-path test to
MxAccessCommandExecutorTests — nothing in the worker exercised was_cached ==
true end to end — asserting the cache hit, reference identity out to the
BulkReadResult, and that no COM call is made for the read.
Not built or tested here: these are net48/x86 worker files that cannot compile
on the macOS tree. Verification is deferred to the windev gate.
The worker-side half of the review tail. Tests and comments only — nothing here
changes worker behavior, and none of it compiles on the macOS tree (net48/x86),
so it was reviewed line by line against the already-windev-validated files.
- MxAccessHandleRegistryTests gains the multi-candidate case behind
MxAccessSession.TryGetCachedReadFor's fall-through: one tag under two item
handles, the lower registered-but-unadvised and the higher advised. Asserted
at the registry rather than the session because the session's read path needs
a live MXAccess COM instance; what the registry owes the scan is the stable
ascending candidate order and a per-item-handle (not per-tag) advice index,
and both are pinned here along with the fall-through contract in prose.
- A single adversarial lifecycle test — register, advise, re-register the same
item handle under a new tag, unadvise, unregister the server — asserting every
index agrees after each step. The individual transitions were already covered;
what was not was that they compose, and a stale entry in any one index
resurrects a handle MXAccess has already retired.
- StaWaitHelperTests.WaitForSignalOrMessages_PreSignalledHandle_ReturnsImmediately
drains pending messages first, like the other two wait tests. Without it a
stale message can end the wait instead of the handle, failing the
signal-consumed post-condition for an unrelated reason.
- GatewayTesting.md records the two findings from the Task 24 windev gate:
SecretsStorePathGuardTests.CreateBuilder_AcceptsSecretsStoreOutsideContentRoot_AndCreatesIt
fails deterministically on Windows on main too (SQLite pooling holds secrets.db
open across the cleanup's recursive delete; pre-existing, tracked separately),
and the StaWaitHelper timing tests' flake signature on a loaded box is a
message wake — the helper working as designed — not a broken wait.
The windev full-suite wedge — every test reported, then the x64 testhost sitting
at ~0 CPU forever while `dotnet test` never returns — was one test blocked on a
pipe write, not a leaked thread or an undisposed fixture.
`dotnet-stack report` on the wedged host showed no thread running test code:
xUnit's RunTestsInAssembly was parked on WaitHandle.WaitOne() waiting for the
assembly-finished event, so the wait lived in a suspended async state machine.
`dotnet-dump analyze -c dumpasync` named the frame — WorkerClientTests
.StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout awaiting
WorkerFrameWriter.WriteAsync on a 63-byte frame, with <extra>5__15 = 6, i.e. the
seventh of the twelve events the test pushes past the client's staging bound.
That test faults the worker client on purpose, and a faulted client stops its
read loop by design. The test-side pipe came from the NamedPipeServerStream
overload without buffer arguments, which passes inBufferSize: 0 / outBufferSize: 0
to CreateNamedPipe; on Windows that reserves no buffer at all, so a write
completes only once the peer reads it. Measured on windev, that pipe absorbed
0 bytes against a non-reading peer where the same pipe declared with 64 KiB
buffers absorbed 65 520. On macOS and Linux .NET backs named pipes with Unix
domain sockets whose socket buffer swallows the writes regardless, which is why
the identical test never hung there and the bug read as environmental.
Test-owned server pipes now go through TestSupport/TestNamedPipe.CreateServer in
both test projects, declaring explicit 64 KiB buffers so those tests exercise the
gateway's own staging/queue backpressure rather than the OS pipe's flow control.
Separately, every fake-worker write in WorkerClientTests now goes through
PipePair.WriteAsync, bounded by the class's five-second TestTimeout. That is
where the severity came from: a test method that never returns keeps xUnit from
raising ITestAssemblyFinished, so one unbounded await cost the whole suite its
result. A blocked write is now a named test failure instead of a silent wedge.
The fix also retires a wrong belief the wedge had created. windev reported 855
where macOS reported 879, and that gap was recorded in GatewayTesting.md as
Unix-gated test cases; it was really the results lost when the wedged host was
torn down. The same clone now reports 879 passed, matching macOS exactly.
Product code is unaffected. SessionWorkerClientFactory.CreatePipe keeps the
unbuffered declaration deliberately: both ends run continuous read loops and every
gateway write is bounded by the worker client's _stopCts, so a stalled peer
cancels the write rather than blocking on it.
Verified on windev at this SHA: gateway suite x64 three times (879 passed,
exit 0, no surviving testhost each time) and Worker.Tests x86 twice (400 passed,
11 skipped, exit 0, clean), plus the macOS gateway suite once (879 passed).
Docs: GatewayTesting.md replaces the --blame-hang workaround section with the
root cause and corrects the baseline to 879, CLAUDE.md's Source Update Workflow
no longer tells readers the windev suite wedges, and ToolchainLinks.md records
dotnet-stack and dotnet-dump as installed on windev.
Root cause. The test sampled its flush baseline after a bare `Task.Delay(100)`
and then charged every later flush to the 128-event burst. Two facts make that
window unsound:
1. `RunHeartbeatLoopAsync` sends its first beat *immediately* on entering the
message loop — the far-off `HeartbeatInterval` only spaces later beats — so
the pre-burst frames are WorkerHello, WorkerReady, and a heartbeat, not the
two the test's comment assumed.
2. `WorkerFrameWriter.DrainQueuedFramesAsync` flushes *after* writing a drained
batch, so the bytes reach the pipe before the flush runs. Reading a frame off
the gateway side is therefore no evidence that its flush has been counted,
and no sleep makes it evidence.
Under load on the shared windows-x86 runner the first heartbeat's flush was
scheduled after the 100 ms sample, so it landed inside the measured window and
the assertion saw two flushes for the burst — exactly the observed
`Expected: 1 / Actual: 2` (Gitea run 675, job 2558, and the same failure since
a346d51). Nothing about the coalescing behavior was wrong; only the test's
timing assumption.
Fix. Replace the sleep with explicit synchronization, no widened timeouts.
`FlushCountingPassthroughStream` now records the flush *shape* — the number of
stream writes coalesced into each flush — and exposes
`WaitForAllWritesFlushedAsync`, a TaskCompletionSource signal released when the
next flush drains the pending writes. The test reads the first heartbeat (the
last pre-burst frame), waits for its flush, and only then samples the baseline;
after the burst it takes the same edge before asserting. The assertion is also
sharpened from a bare count to the shape: exactly one flush beyond the baseline
*and* that flush carried all 128 event frames, so a split batch fails even if
the reader observes it mid-split.
docs/WorkerFrameProtocol.md notes the peer-visible ordering the fix turns on:
frames reach the pipe before the flush that follows them, so an observer of the
flush must wait for it rather than infer it from frames arriving.
A WriteAsync/WriteBatchAsync caller cancelled after the draining lock-holder
claimed its frame unwinds without awaiting that frame's completion; the same
holds for a frame already faulted by a concurrent FailAllQueued, where
TrySetCanceled loses. A later wire-write failure then lands TrySetException on
a task with no awaiter and surfaces as TaskScheduler.UnobservedTaskException.
The tombstone helpers now attach a fault-observing continuation to every frame
in the cancelled call (a cancelled task never fires OnlyOnFaulted, so
unconditional attach is safe), outside _gate because an already-faulted task
runs the continuation inline.
NEXT-05 is resolved as a documented decision, not a code change: tombstoned
entries keep their lazy DequeueNext purge — any subsequent write drains both
queues to empty and the heartbeat loop bounds residency to one interval, while
eager Queue<T> rebuilds under _gate would add ordering-invariant surface for
no gain. Rationale recorded in docs/WorkerFrameProtocol.md alongside the
WRK-22 residual-window contract.
New regression test drives the exact abandonment: gated stream holds writer A
mid-write, the queued event frame is claimed and blocked mid-write, its caller
is cancelled, the write then faults with a marker exception, and the test
asserts the marker never reaches UnobservedTaskException after a forced GC.
net48 x86 build/test runs on windev with the rest of this batch.
OtOpcUa's dominant FreeAccess write path goes out as MX_COMMAND_KIND_WRITE,
not WriteSecured — the original 06/S-1 brief mis-scoped the correlation, so
a refused plain write was invisible on the unary reply (verified live on
windev 2026-08-09). ExecuteWrite/ExecuteWrite2 now use the same pre-call
version baseline + bounded pump-wait as the secured kinds. Bulk writes stay
fire-and-forget.
WRK-22/IPC-26: tombstone a WriteAsync/WriteBatchAsync cancelled while
waiting for the write lock (PendingFrame.Claimed under _gate; DequeueNext
skips cancelled, claims the frame it returns) so a cancelled write never
reaches the wire unless already claimed mid-write (documented residual).
WRK-25: add WriteBatchAsync; RunEventDrainLoopAsync submits the drained
event batch through it, so a burst of N events costs one flush not N.
IPC-30 oversized-event structured fault preserved via FindOversizedEvent.
WRK-24: reject a below-1024 negotiated frame maximum at the handshake
(MinNegotiableFrameBytes, matching GatewayOptionsValidator floor).
WRK-27: alarm poll advertises StaCallInProgress on the heartbeat snapshot
so the watchdog suppresses to the ceiling, not the grace.
Docs (WorkerFrameProtocol.md, MxAccessWorkerInstanceDesign.md) and the
2026-07-12 remediation registers/change-log updated in the same commit.
Review follow-ups on the WRK-21 cluster.
1. ResolveDrainReplyByteBudget was a step function, not a floor: just above the
64 KiB reserve the budget collapsed to a few bytes (at the validator-permitted
floor MaxMessageBytes = 1024 + 64 KiB it was exactly 1024), too small to move a
byte-heavy event, so DrainEvents truncated on every call and the drain-until-
empty loop never terminated. It now takes the max of (frameMax - reserve) and
frameMax/2, so the budget is monotonic and never below half the frame max. New
test DrainEvents_AtValidatorFloorFrameMax_MakesProgressAndTerminates drives a
byte-heavy queue at the exact validator floor and asserts it drains to empty
with no head ever reported oversized.
2. The reply-too-large fallback write is now itself size-guarded
(WriteReplyTooLargeFallbackAsync, used by both the control and STA reply seams):
at a pathologically tiny negotiated max below the gateway's floor the fallback
could also throw MessageTooLarge and — uncaught — kill the session, defeating the
"no diagnostics command is session-fatal" invariant. It now log-and-swallows;
comment notes WRK-24 adds the negotiated-max lower bound that makes it unreachable.
3. Corrected the RepeatedFieldOverheadBytes doc comments: WorkerEvent.CalculateSize()
already includes the event's tag and length prefix (the same shape the reply's
repeated events field packs), so the 8 bytes is pure slack over an already-
conservative estimate, not compensation for a missing wrapper.
Change-log row for 2026-08-07: what landed for WRK-21/WRK-28/WRK-23/IPC-30, why
IPC-23 stays In progress (proto-comment/doc wave pending), and the verification
evidence — macOS NonWindows build + validator tests, and the documented windev
path (scripts/ci/windev-worker-ci.ps1 -Mode test) at a256560: x86 Worker build
clean, Worker.Tests 367 passed / 0 failed / 11 skipped.
PipePair runs both ends of a duplex pipe in one process with blocking
FlushFileBuffers under every frame write, so it wedges after roughly 85 large
round trips. Drain the full 10,000 byte-heavy events to empty at the queue layer,
where the no-loss property actually lives, and keep the pipe walk at 1,000 events
(29 replies) so it still proves the split end to end. Also give the truncation
test's budget slack: item handle 0 is a proto3 default and is not serialized, so
the probe measurement is a lower bound on the fixture's per-event cost.
Joseph Doherty