The worker logged its MXSTATUS_PROXY record-info pin outcome only to its own
stderr, which the gateway's launcher does not capture, so operators never saw
whether the pin took.
- Contract (additive, internal IPC): WorkerReady.record_info_pin_status = 5.
No protocol version bump; an older peer ignores it or sends it empty.
Regenerated Contracts/Generated, the client protoset, the Rust vendored
proto, and the Go/Python/Java worker bindings (real delta only).
- Worker: the handshake copies MxStatusProxyRecordInfoPin.Status into
WorkerReady unless the runtime already set it.
- Gateway: WorkerClient records it (IWorkerClient.RecordInfoPinStatus, default
null) and logs it at Information when pinned, Warning otherwise; empty is
not logged. Shown as "Record-info pin" in the dashboard session details
Worker table via DashboardSessionSummary.WorkerRecordInfoPinStatus.
- Tests: drain-thread cancellation (task ends Canceled; RunAsync cancels,
runtime disposed, thread exits, no EventDrain stop-failure log) and fault
(exception on the thread faults RunAsync with the same instance) paths;
WorkerReady carries the pin status; gateway logging/recording and the
dashboard projection.
- Docs: gateway.md STA thread model (drain thread, pin step in COM lifetime),
MxAccessWorkerInstanceDesign, GatewayProcessDesign, GatewayDashboardDesign,
WorkerFrameProtocol.
At ~240 events/s the worker burned 8-12% system CPU on wonder-app-vd03. A WPR
trace put 72% of it in lmxproxy's per-event GetRecordInfoFromGuids (type
library reload through the registry, amplified by WOW64 and the EDR registry
filter) and 26% in the .NET Framework thread pool spinning after each
per-event work item.
- Worker: MxStatusProxyRecordInfoPin holds the LMXProxy ITypeLib and the
MXSTATUS_PROXY IRecordInfo for the process lifetime, so oleaut32 serves the
per-event lookup from cache (~1 ms -> <1 us per call). Best-effort, never
fatal; status logged at handshake.
- Worker: the event drain loop runs on a dedicated background thread with a
synchronous wait and a synchronous pipe flush, removing the per-event
thread-pool hops.
- Gateway: runtimeconfig sets ThreadPool.UnfairSemaphoreSpinLimit=10 to cut
PortableThreadPool spin-wait CPU.
windev bench at ~242 ev/s: worker 67% -> 6.2% of one core; gateway 8.1% -> 6.9%.
Worker suite 785 passed / 14 skipped (x86, windev).
- New MxGateway:Worker:WorkerHeartbeatGraceSeconds (default 15), exported as
MXGATEWAY_WORKER_HEARTBEAT_GRACE_MS and read by WorkerPipeSessionOptions.FromEnvironment
(a grace the ceiling does not exceed falls back to the default so the worker
still starts). Validation now checks WorkerHeartbeatStuckCeilingSeconds against
the configured grace; the copied WorkerWatchdogGraceSeconds constant is gone.
- EffectiveWorkerConfiguration and the dashboard Settings page now show the worker
heartbeat send interval, watchdog grace and stuck ceiling (render test added).
- Docs updated.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The worker's heartbeat send interval (5 s) and its own STA-watchdog stuck
ceiling (75 s) were compile-time defaults the gateway never reached, and
nothing checked that the gateway's HeartbeatGraceSeconds exceeded the real send
interval, so a misconfigured grace could kill healthy workers.
- New MxGateway:Worker options WorkerHeartbeatSendIntervalSeconds (default 5)
and WorkerHeartbeatStuckCeilingSeconds (default 75), exported by
WorkerProcessLauncher as MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS and
MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS, the same env-var path the other
worker settings use.
- The worker's production WorkerPipeSession reads them through
WorkerPipeSessionOptions.FromEnvironment; a missing/invalid value, or a
ceiling not above the worker's 15 s watchdog grace, keeps the default rather
than failing the worker.
- Startup validation: HeartbeatGraceSeconds > WorkerHeartbeatSendIntervalSeconds;
send interval > 0; worker ceiling > 15 s; both at most 86400 s.
- Docs: GatewayConfiguration, WorkerProcessLauncher (env table now also lists
the previously missing MXGATEWAY_ALARM_ACK_CLIENT_NAME), worker design doc.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
R21-K12: WorkerFrameWriter.WriteBatchAsync allocated a TaskCompletionSource
(plus its Task) per frame on the event drain hot path. PendingFrame now holds
its terminal state itself (Interlocked pending/succeeded/faulted/canceled) and
only single-frame WriteAsync frames, which race their completion against the
write lock, get a TCS. A batch caller reads each frame's outcome after its own
drain pass: every frame a pass claims is terminal before that pass releases the
lock, so nothing is left to await. First-failure-in-batch-order, tombstoning,
claim/cancel interlock and flush coalescing are unchanged.
R21-K13: WnWrapAlarmConsumer.ParseSnapshotXml streams the GetXmlCurrentAlarms2
payload with an XmlTextReader (same defaults XmlDocument.LoadXml uses:
Normalization off, DTDs parsed, no resolver; Significant whitespace only)
instead of building a full XmlDocument every 500 ms poll.
Behaviour is pinned by new tests: batch first-failure ordering (own drain and
drained by another lock holder), batch cancellation tombstoning, batch flush
failure, and a differential test running the replaced XmlDocument parser as an
oracle over a 48-payload corpus (CRLF, CDATA, entities, DTD, namespaces,
nesting, malformed). Mutation checks on windev: 8 of 8 injected regressions
(whitespace, normalization, DTD, namespace, CDATA, batch last-failure, batch
ignored failures, ignored tombstone) fail the new tests.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
WorkerPipeSession had seven CreateEnvelope overloads that each forwarded to a
matching CreateBaseEnvelope overload; they are now one overload per body type
that stamps the header and sets the body. WorkerPipeClient's six chained
constructors collapse to one with optional arguments (null attempt timeout =
env/default, null session factory = production WorkerPipeSession). Adds
constructor-validation tests.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
A frame read that hits end of stream means the gateway closed its end of
the pipe (usually because it died). WorkerApplication mapped every
WorkerFrameProtocolException to ProtocolViolation (6); EndOfStream now
exits as PipeConnectionFailed (5), alongside the IOException a broken
pipe raises.
Event-queue overflow exited as the generic UnexpectedFailure (1). The
event drain loop now unwinds with WorkerEventQueueFaultException carrying
the fault category, and QueueOverflow exits with the new
WorkerExitCode.EventQueueOverflow (8). Other categories keep exit 1.
The gateway does not decode worker exit codes (it only logs the number),
so no gateway change is needed. WorkerBootstrap.md and
MxAccessWorkerInstanceDesign.md updated. (mxaccessgw#186)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
TryStartCommandTask returned silently when the session had stopped
accepting commands, so a command racing shutdown got no reply and no log
and the gateway's correlation wait ran to its timeout. The message loop
now answers such a command with a WorkerUnavailable reply (the status the
STA dispatcher already uses during shutdown) and logs
WorkerCommandRejectedShuttingDown. New test
CommandAfterShutdown_RepliesWorkerUnavailable (mxaccessgw#182)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Tracking IDs are review bookkeeping that rots in shipped source; the rationale they tagged is kept. Comment-only change (CommentChecker 458 -> 16 false positives).
The worker budgeted a DrainEvents reply against the negotiated pipe frame
maximum less one 64 KiB reserve, but the gateway returns that same
MxCommandReply verbatim through Invoke, where MaxSendMessageSize applies. The
two limits were coupled only from below (pipe >= gRPC + reserve), so any
configuration the validator permitted with a wider gap let the worker pack a
reply the pipe carried and the gRPC layer then refused with ResourceExhausted --
after Drain had already removed the events from the worker queue, losing them to
both the caller and the event stream.
Option (a) from the finding: close the coupling at both ends.
- GatewayOptionsValidator adds the upper half of the headroom rule
(Worker:MaxMessageBytes <= Protocol:MaxGrpcMessageBytes + 2 x reserve), with a
message naming the DrainEvents budget as the reason and the two knobs to move.
- WorkerPipeSession.ResolveDrainReplyByteBudget subtracts the reserve twice --
once for the pipe frame, once so the MxCommandReply wrapper clears the gRPC
send cap -- keeping the MaxMessageBytes/2 floor that makes small frame maxima
usable.
The two changes are load-bearing for each other: the validator band is what
makes "frame max - 2 x reserve" provably at or below the gRPC cap, and the
doubled subtraction is what makes the band sufficient.
Docs updated in the same change (GatewayConfiguration.md both option rows plus
the headroom prose, WorkerFrameProtocol.md drain budget, gateway.md DrainEvents
byte axis). Tests: new GatewayOptionsValidatorTests cases for the upper bound
(reject) and both edges of the band (accept); GatewayOptionsTests' override case
now moves both message-size knobs together, since the band makes moving one
alone illegal.
The worker half cannot be built on macOS; verified on windev separately.
The BlockDispatch branch waited 5 real seconds and then proceeded regardless — it does not
branch on the wait's result. The long-in-flight test's inspection loop is bounded by
elapsed time and a frame floor, so on a loaded box (the documented 4-5x slowdown class) it
can plausibly outrun that 5 s. When it does, the reply is emitted mid-window,
AssertNotWorkerFault waves it past, and the reply leg then waits for a reply already gone
by — failing at the 20 s cancellation with no message, on exactly the loaded-box run the
widened windows exist to survive.
The wait is a pure safety net: nothing asserts on it firing, and every test that blocks
dispatch releases it explicitly (ReleaseDispatch, or a WorkerShutdown envelope, both of
which Set the event) — none reaches the timeout on a healthy run. Named it
BlockedDispatchSafetyNet and raised it to 30 s, above any window a test opens and above the
20 s cancellation those tests arm, so a wedged test always fails on its own token with its
own message. Dispose still releases the wait, so teardown never waits on it either.
Inline rationale in the test now states the decoupling and what a close pairing would cost,
rather than asserting the window stays inside a 5 s ceiling. Nothing else changed.
Three review findings on 7da52b6:
1. The AssertNotFault doc block landed between the predicate-overload ReadUntilAsync's
doc block and the helper, so the compiler attached the merged block to the helper and
ReadUntilAsync lost its docs entirely. Helper and its docs moved above ReadUntilAsync,
whose docs are back where they belong.
2. The compressed watchdog windows (50 ms grace, 100 ms ceiling) reintroduced the
load-sensitivity the fix removed, one layer down.
ReportWatchdogFaultIfNeededAsync measures staleness AFTER the heartbeat frame is
written and flushed over the real named pipe, so a beat whose pipe I/O outlasts the
ceiling faults a healthy session no matter how fresh the captured activity was. At
100 ms that is a plausible stall on a loaded box, and this is the one test asserting
the watchdog NEVER fires. Widened to a 200 ms grace and a 1 s ceiling — still two
orders of magnitude under the 75 s production default.
The inspection loop is now bounded by a 2 s window (twice the ceiling, so a fake whose
activity stopped advancing still accumulates past it and faults) with a 30-frame floor,
rather than a fixed 30 frames that no longer outran the wider ceiling. The floor keeps
a window that saw almost no beats from passing as a clean one. Two seconds stays well
inside FakeRuntimeSession's 5 s dispatch-block ceiling, so the command is still in
flight for the whole window.
3. AssertNotFault renamed AssertNotWorkerFault, matching the WorkerFault body case it
tests.
Scenario intent unchanged: long in-flight command, pump refreshing, zero fault frames,
reply delivered. Test-only.
RunAsync_LongInFlightCommandThatKeepsPumping_DoesNotFaultAndDeliversReply failed
deterministically on the Windows box with a StaHung fault whose command_method was
empty and whose staleness was 233 ms — i.e. a fault raised with no command in flight,
before the scenario under test began. FakeRuntimeSession stamps LastStaActivityUtc once,
at construction, and the test only started refreshing it after the blocked dispatch
signalled. Everything between those two points — handshake, STA init, the first
heartbeat — captured a snapshot already stale past the compressed 50 ms grace, with no
correlation id for the watchdog to suppress on, so the watchdog correctly reported the
fake as hung.
The harness, not the product, was wrong: StaRuntime.ThreadMain calls MarkActivity() on
every WaitForWorkOrMessages iteration, so a live worker is never captured stale, idle or
busy. Model that where it belongs — FakeRuntimeSession.RefreshStaActivityOnCapture (opt
in, default off) stamps activity at each CaptureHeartbeat and leaves the rest of the
snapshot alone — and arm it before RunAsync so the first beat is covered. The test-owned
refresh loop goes away with it; a thread-pool loop racing a compressed grace could not
have held the invariant anyway.
Scenario intent is unchanged and slightly stronger: the command still blocks in dispatch
across 30 heartbeats (~600 ms, many multiples of the 100 ms stuck ceiling), no frame may
be a fault, and the reply must still arrive. The reply leg is now fault-checked too
(previously it skipped frames blindly), and the pump keeps running across the release, as
it does in production while the reply is marshalled off the STA. Fault assertions now
report the category and diagnostic message instead of a bare body-case mismatch.
Test-only change; no product code, frame protocol, or STA rule touched.
WriteAsync enqueued its frame and then contended unconditionally for the write
lock, so a caller that lost the race stayed in WaitAsync until the winning
drainer released — even though that winner writes, flushes, and completes the
loser's control frame at the control-to-event class boundary, part-way through
its pass. The boundary flush made the delivery point honest; the awaited task
was still charged for the whole event backlog it had just been flushed ahead of.
WriteAsync now awaits its own frame's completion racing the lock acquisition.
Completion first: the caller returns at its frame's delivery point and the
outstanding acquisition is detached, not dropped — a continuation drains
whatever is queued and releases, so the lock is never acquired and silently
held and a frame enqueued between the previous drainer's last dequeue and its
release is still written. Lock first: drain as before. Cancellation keeps the
WRK-22 tombstone semantics exactly, and a wait cancelled after the caller has
already detached releases nothing (SemaphoreSlim hands no count to a wait it
cancels), so no count leaks and no queued frame is stranded. A token that fires
after the frame's completion won the race changes nothing — the frame was
delivered. WriteBatchAsync deliberately keeps the plain wait-then-drain shape:
its last completion resolves at the end-of-pass flush anyway.
Three tests: the latency win (a control caller returning while the winning
WriteBatchAsync event burst is demonstrably still blocked mid-pass), a
mixed-priority concurrency soak pinning exactly-once writes and a single
drainer, and the cancel-after-detach corner (a wrongly released count would
surface as the drainer's own Release throwing SemaphoreFullException).
edited on macOS, windev verification pending (plan Task 11). Verified here by
compiling and running WorkerFrameWriter plus the writer suite against net10.0
in a scratch harness: 31/31 pass, and the two behaviour-pinning tests fail
against the pre-change parked implementation.
The two-class writer already got control bytes out ahead of a queued event
backlog, but a frame counts as delivered only once flushed, and the drain
deferred its single FlushAsync — and every TrySetResult — to the end of the
pass. A heartbeat, command reply, fault, or shutdown ack was therefore written
first and completed last, behind up to a full 128-frame event batch.
The drain now records each frame's priority class on PendingFrame and flushes
at every control-to-event boundary, completing and clearing the written set
there. Cost stays bounded: a pure-event pass still pays exactly one flush, a
run of control frames still pays one for the run, and only a pass that mixes
both classes pays a second — never one flush per control frame, the
syscall-per-heartbeat cost WRK-12 removed.
A boundary flush that itself fails is a new failure window and is handled like
the end-of-pass flush failure, additionally failing the event frame the drain
had already claimed off its queue and every frame still queued. Frames a
boundary flush completed leave the written set, so a later failure in the same
pass can no longer reach back and fail an already-delivered control frame.
The awaited task of a caller that lost the write-lock race is still bounded by
the winning drainer's pass — that enqueue-then-contend parking is unchanged and
now documented on WriteAsync and in docs/WorkerFrameProtocol.md.
The windev full-suite wedge — every test reported, then the x64 testhost sitting
at ~0 CPU forever while `dotnet test` never returns — was one test blocked on a
pipe write, not a leaked thread or an undisposed fixture.
`dotnet-stack report` on the wedged host showed no thread running test code:
xUnit's RunTestsInAssembly was parked on WaitHandle.WaitOne() waiting for the
assembly-finished event, so the wait lived in a suspended async state machine.
`dotnet-dump analyze -c dumpasync` named the frame — WorkerClientTests
.StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout awaiting
WorkerFrameWriter.WriteAsync on a 63-byte frame, with <extra>5__15 = 6, i.e. the
seventh of the twelve events the test pushes past the client's staging bound.
That test faults the worker client on purpose, and a faulted client stops its
read loop by design. The test-side pipe came from the NamedPipeServerStream
overload without buffer arguments, which passes inBufferSize: 0 / outBufferSize: 0
to CreateNamedPipe; on Windows that reserves no buffer at all, so a write
completes only once the peer reads it. Measured on windev, that pipe absorbed
0 bytes against a non-reading peer where the same pipe declared with 64 KiB
buffers absorbed 65 520. On macOS and Linux .NET backs named pipes with Unix
domain sockets whose socket buffer swallows the writes regardless, which is why
the identical test never hung there and the bug read as environmental.
Test-owned server pipes now go through TestSupport/TestNamedPipe.CreateServer in
both test projects, declaring explicit 64 KiB buffers so those tests exercise the
gateway's own staging/queue backpressure rather than the OS pipe's flow control.
Separately, every fake-worker write in WorkerClientTests now goes through
PipePair.WriteAsync, bounded by the class's five-second TestTimeout. That is
where the severity came from: a test method that never returns keeps xUnit from
raising ITestAssemblyFinished, so one unbounded await cost the whole suite its
result. A blocked write is now a named test failure instead of a silent wedge.
The fix also retires a wrong belief the wedge had created. windev reported 855
where macOS reported 879, and that gap was recorded in GatewayTesting.md as
Unix-gated test cases; it was really the results lost when the wedged host was
torn down. The same clone now reports 879 passed, matching macOS exactly.
Product code is unaffected. SessionWorkerClientFactory.CreatePipe keeps the
unbuffered declaration deliberately: both ends run continuous read loops and every
gateway write is bounded by the worker client's _stopCts, so a stalled peer
cancels the write rather than blocking on it.
Verified on windev at this SHA: gateway suite x64 three times (879 passed,
exit 0, no surviving testhost each time) and Worker.Tests x86 twice (400 passed,
11 skipped, exit 0, clean), plus the macOS gateway suite once (879 passed).
Docs: GatewayTesting.md replaces the --blame-hang workaround section with the
root cause and corrects the baseline to 879, CLAUDE.md's Source Update Workflow
no longer tells readers the windev suite wedges, and ToolchainLinks.md records
dotnet-stack and dotnet-dump as installed on windev.
Root cause. The test sampled its flush baseline after a bare `Task.Delay(100)`
and then charged every later flush to the 128-event burst. Two facts make that
window unsound:
1. `RunHeartbeatLoopAsync` sends its first beat *immediately* on entering the
message loop — the far-off `HeartbeatInterval` only spaces later beats — so
the pre-burst frames are WorkerHello, WorkerReady, and a heartbeat, not the
two the test's comment assumed.
2. `WorkerFrameWriter.DrainQueuedFramesAsync` flushes *after* writing a drained
batch, so the bytes reach the pipe before the flush runs. Reading a frame off
the gateway side is therefore no evidence that its flush has been counted,
and no sleep makes it evidence.
Under load on the shared windows-x86 runner the first heartbeat's flush was
scheduled after the 100 ms sample, so it landed inside the measured window and
the assertion saw two flushes for the burst — exactly the observed
`Expected: 1 / Actual: 2` (Gitea run 675, job 2558, and the same failure since
a346d51). Nothing about the coalescing behavior was wrong; only the test's
timing assumption.
Fix. Replace the sleep with explicit synchronization, no widened timeouts.
`FlushCountingPassthroughStream` now records the flush *shape* — the number of
stream writes coalesced into each flush — and exposes
`WaitForAllWritesFlushedAsync`, a TaskCompletionSource signal released when the
next flush drains the pending writes. The test reads the first heartbeat (the
last pre-burst frame), waits for its flush, and only then samples the baseline;
after the burst it takes the same edge before asserting. The assertion is also
sharpened from a bare count to the shape: exactly one flush beyond the baseline
*and* that flush carried all 128 event frames, so a split batch fails even if
the reader observes it mid-split.
docs/WorkerFrameProtocol.md notes the peer-visible ordering the fix turns on:
frames reach the pipe before the flush that follows them, so an observer of the
flush must wait for it rather than infer it from frames arriving.
A WriteAsync/WriteBatchAsync caller cancelled after the draining lock-holder
claimed its frame unwinds without awaiting that frame's completion; the same
holds for a frame already faulted by a concurrent FailAllQueued, where
TrySetCanceled loses. A later wire-write failure then lands TrySetException on
a task with no awaiter and surfaces as TaskScheduler.UnobservedTaskException.
The tombstone helpers now attach a fault-observing continuation to every frame
in the cancelled call (a cancelled task never fires OnlyOnFaulted, so
unconditional attach is safe), outside _gate because an already-faulted task
runs the continuation inline.
NEXT-05 is resolved as a documented decision, not a code change: tombstoned
entries keep their lazy DequeueNext purge — any subsequent write drains both
queues to empty and the heartbeat loop bounds residency to one interval, while
eager Queue<T> rebuilds under _gate would add ordering-invariant surface for
no gain. Rationale recorded in docs/WorkerFrameProtocol.md alongside the
WRK-22 residual-window contract.
New regression test drives the exact abandonment: gated stream holds writer A
mid-write, the queued event frame is claimed and blocked mid-write, its caller
is cancelled, the write then faults with a marker exception, and the test
asserts the marker never reaches UnobservedTaskException after a forced GC.
net48 x86 build/test runs on windev with the rest of this batch.
WRK-22/IPC-26: tombstone a WriteAsync/WriteBatchAsync cancelled while
waiting for the write lock (PendingFrame.Claimed under _gate; DequeueNext
skips cancelled, claims the frame it returns) so a cancelled write never
reaches the wire unless already claimed mid-write (documented residual).
WRK-25: add WriteBatchAsync; RunEventDrainLoopAsync submits the drained
event batch through it, so a burst of N events costs one flush not N.
IPC-30 oversized-event structured fault preserved via FindOversizedEvent.
WRK-24: reject a below-1024 negotiated frame maximum at the handshake
(MinNegotiableFrameBytes, matching GatewayOptionsValidator floor).
WRK-27: alarm poll advertises StaCallInProgress on the heartbeat snapshot
so the watchdog suppresses to the ceiling, not the grace.
Docs (WorkerFrameProtocol.md, MxAccessWorkerInstanceDesign.md) and the
2026-07-12 remediation registers/change-log updated in the same commit.
Review follow-ups on the WRK-21 cluster.
1. ResolveDrainReplyByteBudget was a step function, not a floor: just above the
64 KiB reserve the budget collapsed to a few bytes (at the validator-permitted
floor MaxMessageBytes = 1024 + 64 KiB it was exactly 1024), too small to move a
byte-heavy event, so DrainEvents truncated on every call and the drain-until-
empty loop never terminated. It now takes the max of (frameMax - reserve) and
frameMax/2, so the budget is monotonic and never below half the frame max. New
test DrainEvents_AtValidatorFloorFrameMax_MakesProgressAndTerminates drives a
byte-heavy queue at the exact validator floor and asserts it drains to empty
with no head ever reported oversized.
2. The reply-too-large fallback write is now itself size-guarded
(WriteReplyTooLargeFallbackAsync, used by both the control and STA reply seams):
at a pathologically tiny negotiated max below the gateway's floor the fallback
could also throw MessageTooLarge and — uncaught — kill the session, defeating the
"no diagnostics command is session-fatal" invariant. It now log-and-swallows;
comment notes WRK-24 adds the negotiated-max lower bound that makes it unreachable.
3. Corrected the RepeatedFieldOverheadBytes doc comments: WorkerEvent.CalculateSize()
already includes the event's tag and length prefix (the same shape the reply's
repeated events field packs), so the 8 bytes is pure slack over an already-
conservative estimate, not compensation for a missing wrapper.
PipePair runs both ends of a duplex pipe in one process with blocking
FlushFileBuffers under every frame write, so it wedges after roughly 85 large
round trips. Drain the full 10,000 byte-heavy events to empty at the queue layer,
where the no-loss property actually lives, and keep the pipe walk at 1,000 events
(29 replies) so it still proves the split end to end. Also give the truncation
test's budget slack: item handle 0 is a proto3 default and is not serialized, so
the probe measurement is a lower bound on the fixture's per-event cost.
PipePair has no continuous read pump — the test thread drains the pipe only
while it sits in ReadUntilAsync — so multi-megabyte DrainEvents frames
interleaved with the heartbeat loop wedge both ends inside FlushFileBuffers,
each waiting for the other to read. Negotiate a 128 KiB frame maximum instead:
the 10,000 byte-heavy events still overflow it many times over, so every
assertion (bounded reply, reported truncation, no event loss across repeated
drains, surviving session) is unchanged.
WRK-21 — DrainEvents was bounded by event count only, so a byte-heavy queue
(large string/array MxValues) built a reply above the negotiated frame maximum:
the writer rejected the frame, the exception unwound the session, and the events
already dequeued were destroyed. The drain is now byte-budgeted inside the queue
lock, so an event is dequeued only once it is known to fit and one that does not
stays at the head. Truncation is reported through the reply's existing
DiagnosticMessage (no contract change); callers drain until an empty reply. Both
reply-write seams — the control-command path and ProcessCommandAsync — now catch
MessageTooLarge and answer the correlation with an InvalidRequest reply instead
of unwinding or faulting the session. Satisfies IPC-23 R1-R3.
WRK-28 — the 10,000 drain ceiling moves to GatewayContractInfo
.MaxDrainEventsPerCommand, referenced by both the gateway request validator and
the worker clamp, replacing a comment-only sync contract. C# const only; no
.proto change.
WRK-23 — WorkerFrameWriter now peek-stamps, validates, then commits the sequence
counter immediately before the stream write, so a per-frame rejection leaves no
phantom gap on the wire.
IPC-30 — an oversized event frame stays session-fatal (it is undeliverable end to
end and neither dropping nor synthesizing a replacement is allowed), but the
death is structured: the event's identity and sizes are logged (never its value),
a WorkerFault with category PROTOCOL_VIOLATION and command method EventDrain is
written, then the session exits as before.
Docs updated in the same change: MxAccessWorkerInstanceDesign.md (drain byte cap,
truncation contract, oversized-head behavior, oversized-event policy, no control
reply is session-fatal on size), WorkerFrameProtocol.md (reply pre-sizing,
non-fatal reply-size rule, oversized-event policy, rejected frames do not consume
sequence numbers), gateway.md (DrainEvents two-axis bound).
Resolve all CommentChecker findings across the gateway server, worker, tests,
and .NET client (314 -> 0 real issues): add missing <returns>/<summary>/<param>
on public and test members, convert Stream/interface overrides to <inheritdoc/>,
and remove internal task/issue tracking IDs (SEC-*, IPC-*, WRK-*, GWC-*, TST-*,
Client.Dotnet-*) from shipped code documentation while preserving the design
rationale prose. Shipped comments should not carry internal bookkeeping, and
complete XML docs keep the analyzer/TreatWarningsAsErrors gate and generated API
docs clean. The 6 remaining flags are heuristic false positives (MD5, UTC-4,
capacity-1, near-1601) left intact so real documentation is not corrupted.
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
WRK-06: MxStatusProxyConverter caches the four resolved FieldInfo per status
type in a static ConcurrentDictionary (the GetField metadata scan ran 4x per
status per event on the STA path). GetValue+Convert.ToInt32 still run per event
(late-bound RCW). Exceptions byte-identical: missing-field message unchanged
(ResolveField, not cached on throw via GetOrAdd); null-value message unchanged.
WRK-11: MxAccessEventQueue.Enqueue takes ownership of the passed MxEvent -
stamps WorkerSequence/WorkerTimestamp on it in place and enqueues it, no
Clone(). Audited all 3 callers (base/alarm event sinks, provider-mode handler):
each builds a fresh event per Enqueue, none reuse it. MxAccessValueCache.Set now
deep-copies its retained Value/SourceTimestamp/Statuses so the cache snapshot
never aliases the queue-owned (later serialized) event. Net: alarm/other events
clone nothing (was full clone); data-change clones payload-only.
WRK-12: WorkerFrameWriter coalesces the flush across a drained batch - each
frame is written but not flushed individually; one FlushAsync after the batch,
then all written frames complete. Preserves the written+flushed completion
contract; a burst of N events costs 1 flush, not N. On write failure the whole
in-flight batch + queue fail so no caller hangs.
IPC-15 (doc): the multi-event WorkerEnvelope body remains unimplemented (wire
still carries one event per worker_event frame); gateway.md Performance section
now distinguishes the shipped flush-coalescing from that deferred proto change.
net48-safe (no init/records; readonly struct cache entry). Worker builds x86
only - verification on windev. Tests added: converter cache-reuse, queue
ownership-transfer, value-cache snapshot independence, writer batch-flush-once.
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
Worker half of the Wave 3 size/backpressure + write-ordering pass:
- IPC-02: the worker adopts GatewayHello.max_frame_bytes during the handshake
(WorkerFrameProtocolOptions.AdoptNegotiatedMaxMessageBytes) instead of a
hard-coded default; 0 keeps the default, a value above a 256 MiB ceiling is
rejected. Reader and writer share the options instance, applied before the
message loop.
- IPC-04: DrainEvents caps each reply at MaxDrainEventsPerReply (10_000) and
treats max_events = 0 as that cap rather than 'drain the entire queue', so one
diagnostic drain cannot pack a session-killing reply frame.
- WRK-04: WorkerFrameWriter stamps the envelope Sequence at the actual point of
writing (under the write lock) instead of at envelope creation, so the on-wire
order and the stamped sequence always agree under concurrent producers.
- WRK-07: the writer is now a cooperative priority scheduler — callers enqueue at
Control or Event priority and the draining lock-holder writes all control
frames before any event frame, so replies/faults/heartbeats jump ahead of an
event backlog. Per-frame validation/size rejections fail only that frame; a
stream write failure fails all queued frames.
Tests: monotonic gap-free sequence under concurrency, control-before-event
priority (gated stream), negotiated-max adoption, DrainEvents zero-bound.
Worker builds x86 only — verified on windev.
A long legitimate ReadBulk pumped Windows messages without refreshing
LastStaActivityUtc, so the watchdog false-positived StaHung past
HeartbeatStuckCeiling and then silently dropped every reply. PumpPendingMessages()
now calls MarkActivity() after pumping; a genuinely stuck STA (no pumping)
still accrues staleness and faults correctly. No MXAccess parity change.
archreview: WRK-01 (P0). Verified on the Windows host (x86): worker builds
clean, StaRuntimeTests + WorkerPipeSessionTests 33/33 pass.
Client.Java-040..048, Worker.Tests-034/035/036. Edits applied on the Mac,
which has no JRE and cannot build the x86+MXAccess worker tests; findings are
marked In Progress pending gradle + x86 build verification on windev. Do not
mark Resolved until verified there.
Ping is now intercepted as a worker control command and answered on the
message-loop thread, so the dispatch/heartbeat/shutdown-race tests must use a
genuine STA-dispatched command kind to keep exercising DispatchAsync.
Answer the five worker control/lifecycle commands at the WorkerPipeSession
message-loop layer instead of the STA-bound MxAccessCommandExecutor. These
replies are built from process-level state (worker pid, assembly version,
worker lifecycle, the runtime session's event queue) the executor cannot see,
and ShutdownWorker must emit its OK reply before the graceful shutdown joins
the STA thread - dispatching it onto the STA would deadlock.
- Ping: OK reply, echoes message into diagnostic_message.
- GetSessionState: maps WorkerState to proto SessionState.
- GetWorkerInfo: pid, worker version, MXAccess ProgID/CLSID.
- DrainEvents: drains the runtime event queue into DrainEventsReply.
- ShutdownWorker: OK reply, then graceful shutdown, then stops the loop.
Tests added in WorkerPipeSessionTests; FakeRuntimeSession gains a
batch-size drain suppressor so DrainEvents does not race the background
drain loop.
Adds missing <summary>/<param> XML docs across 99 server, worker, and test
files so CommentChecker reports zero issues (TreatWarningsAsErrors needs the
analyzer clean). Bundles in WIP dashboard work: NavSection extraction,
MainLayout/site.css/js styling alignment, and DashboardOptions/Auth tweaks.
Apply the ZB.MOM.WW. prefix to all gateway-side projects, folders,
.csproj/.sln contents, C# namespaces, using directives, generated proto
C# (csharp_namespace + checked-in generated files), InternalsVisibleTo
attributes, project-name string literals (LoadProject, .sln lookups,
worker exe paths, staticwebassets manifest), and the install/script/doc
references that point at any of the above. Migrate the solution from
.sln to .slnx via `dotnet sln migrate` and delete the old file.
External-runtime identifiers are intentionally NOT prefixed so external
configuration keeps working:
- GatewayMetrics.cs MeterName ("MxGateway.Server")
- DashboardAuthenticationDefaults Scheme/Policy ("MxGateway.Dashboard")
- GatewayRequestLoggingMiddleware logger category ("MxGateway.Request")
- StaRuntime thread name ("MxGateway.Worker.STA")
- appsettings.json root section "MxGateway" + env-var prefix
MxGateway__... and secret-name MxGateway:ApiKeyPepper
- C:\ProgramData\MxGateway\ data dir paths
Also fixes two tests that were not rename-related but became visible
while validating the rename:
- WorkerLiveMxAccessSmokeTests.ShutDownAsync: cancellation that the
gateway service correctly maps to RpcException(Cancelled) per gRPC
convention was being misclassified as a stream fault. Added a sibling
catch on RpcException with StatusCode.Cancelled.
- IntegrationTestEnvironment.ResolveRepositoryRoot: extracted IsRepositoryRoot
and made it accept either a .git marker OR a .sln/.slnx next to src/
so the worker-exe walker works in non-git working copies.
clients/proto/proto-inputs.json's protoRoot updated to point at
src/ZB.MOM.WW.MxGateway.Contracts/Protos.
Verified by `dotnet build` and a full `dotnet test` of the .slnx with
MXGATEWAY_RUN_LIVE_{MXACCESS,LDAP,GALAXY}_TESTS=1:
Tests: 472/472 pass
Worker.Tests: 280/280 pass (4 dev-rig [Fact(Skip=...)] skipped)
IntegrationTests: 18/18 pass
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>