Commit Graph

39 Commits

Author SHA1 Message Date
Joseph Doherty 30ac3f3225 feat(worker,gateway): report record-info pin outcome in WorkerReady; drain-thread completion tests
The worker logged its MXSTATUS_PROXY record-info pin outcome only to its own
stderr, which the gateway's launcher does not capture, so operators never saw
whether the pin took.

- Contract (additive, internal IPC): WorkerReady.record_info_pin_status = 5.
  No protocol version bump; an older peer ignores it or sends it empty.
  Regenerated Contracts/Generated, the client protoset, the Rust vendored
  proto, and the Go/Python/Java worker bindings (real delta only).
- Worker: the handshake copies MxStatusProxyRecordInfoPin.Status into
  WorkerReady unless the runtime already set it.
- Gateway: WorkerClient records it (IWorkerClient.RecordInfoPinStatus, default
  null) and logs it at Information when pinned, Warning otherwise; empty is
  not logged. Shown as "Record-info pin" in the dashboard session details
  Worker table via DashboardSessionSummary.WorkerRecordInfoPinStatus.
- Tests: drain-thread cancellation (task ends Canceled; RunAsync cancels,
  runtime disposed, thread exits, no EventDrain stop-failure log) and fault
  (exception on the thread faults RunAsync with the same instance) paths;
  WorkerReady carries the pin status; gateway logging/recording and the
  dashboard projection.
- Docs: gateway.md STA thread model (drain thread, pin step in COM lifetime),
  MxAccessWorkerInstanceDesign, GatewayProcessDesign, GatewayDashboardDesign,
  WorkerFrameProtocol.
2026-11-21 08:13:27 -05:00
Joseph Doherty 659bb831a2 perf(worker,gateway): pin MXSTATUS_PROXY record info, dedicated event-drain thread, cap thread-pool spin
At ~240 events/s the worker burned 8-12% system CPU on wonder-app-vd03. A WPR
trace put 72% of it in lmxproxy's per-event GetRecordInfoFromGuids (type
library reload through the registry, amplified by WOW64 and the EDR registry
filter) and 26% in the .NET Framework thread pool spinning after each
per-event work item.

- Worker: MxStatusProxyRecordInfoPin holds the LMXProxy ITypeLib and the
  MXSTATUS_PROXY IRecordInfo for the process lifetime, so oleaut32 serves the
  per-event lookup from cache (~1 ms -> <1 us per call). Best-effort, never
  fatal; status logged at handshake.
- Worker: the event drain loop runs on a dedicated background thread with a
  synchronous wait and a synchronous pipe flush, removing the per-event
  thread-pool hops.
- Gateway: runtimeconfig sets ThreadPool.UnfairSemaphoreSpinLimit=10 to cut
  PortableThreadPool spin-wait CPU.

windev bench at ~242 ev/s: worker 67% -> 6.2% of one core; gateway 8.1% -> 6.9%.
Worker suite 785 passed / 14 skipped (x86, windev).
2026-11-21 08:00:13 -05:00
Joseph Doherty 22964e192b feat(gateway,worker): configurable worker watchdog grace; heartbeat settings in effective config and Settings page (mxaccessgw#224)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m24s
ci / java (push) Successful in 2m15s
ci / portable (push) Failing after 4m17s
- New MxGateway:Worker:WorkerHeartbeatGraceSeconds (default 15), exported as
  MXGATEWAY_WORKER_HEARTBEAT_GRACE_MS and read by WorkerPipeSessionOptions.FromEnvironment
  (a grace the ceiling does not exceed falls back to the default so the worker
  still starts). Validation now checks WorkerHeartbeatStuckCeilingSeconds against
  the configured grace; the copied WorkerWatchdogGraceSeconds constant is gone.
- EffectiveWorkerConfiguration and the dashboard Settings page now show the worker
  heartbeat send interval, watchdog grace and stuck ceiling (render test added).
- Docs updated.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 13:29:10 -05:00
Joseph Doherty 2e52f9f5e8 feat(gateway,worker): configurable worker heartbeat send interval and stuck ceiling (mxaccessgw#218)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m32s
ci / java (push) Successful in 2m57s
ci / portable (push) Failing after 5m1s
The worker's heartbeat send interval (5 s) and its own STA-watchdog stuck
ceiling (75 s) were compile-time defaults the gateway never reached, and
nothing checked that the gateway's HeartbeatGraceSeconds exceeded the real send
interval, so a misconfigured grace could kill healthy workers.

- New MxGateway:Worker options WorkerHeartbeatSendIntervalSeconds (default 5)
  and WorkerHeartbeatStuckCeilingSeconds (default 75), exported by
  WorkerProcessLauncher as MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS and
  MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS, the same env-var path the other
  worker settings use.
- The worker's production WorkerPipeSession reads them through
  WorkerPipeSessionOptions.FromEnvironment; a missing/invalid value, or a
  ceiling not above the worker's 15 s watchdog grace, keeps the default rather
  than failing the worker.
- Startup validation: HeartbeatGraceSeconds > WorkerHeartbeatSendIntervalSeconds;
  send interval > 0; worker ceiling > 15 s; both at most 86400 s.
- Docs: GatewayConfiguration, WorkerProcessLauncher (env table now also lists
  the previously missing MXGATEWAY_ALARM_ACK_CLIENT_NAME), worker design doc.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 13:24:05 -05:00
Joseph Doherty 29d9510b4e perf(worker): batch frames skip per-frame TCS; stream the alarm-poll XML (mxaccessgw#177)
R21-K12: WorkerFrameWriter.WriteBatchAsync allocated a TaskCompletionSource
(plus its Task) per frame on the event drain hot path. PendingFrame now holds
its terminal state itself (Interlocked pending/succeeded/faulted/canceled) and
only single-frame WriteAsync frames, which race their completion against the
write lock, get a TCS. A batch caller reads each frame's outcome after its own
drain pass: every frame a pass claims is terminal before that pass releases the
lock, so nothing is left to await. First-failure-in-batch-order, tombstoning,
claim/cancel interlock and flush coalescing are unchanged.

R21-K13: WnWrapAlarmConsumer.ParseSnapshotXml streams the GetXmlCurrentAlarms2
payload with an XmlTextReader (same defaults XmlDocument.LoadXml uses:
Normalization off, DTDs parsed, no resolver; Significant whitespace only)
instead of building a full XmlDocument every 500 ms poll.

Behaviour is pinned by new tests: batch first-failure ordering (own drain and
drained by another lock holder), batch cancellation tombstoning, batch flush
failure, and a differential test running the replaced XmlDocument parser as an
oracle over a 48-payload corpus (CRLF, CDATA, entities, DTD, namespaces,
nesting, malformed). Mutation checks on windev: 8 of 8 injected regressions
(whitespace, normalization, DTD, namespace, CDATA, batch last-failure, batch
ignored failures, ignored tombstone) fail the new tests.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 13:24:05 -05:00
Joseph Doherty 7d44ecd62d refactor(worker): dedupe CreateEnvelope overloads and WorkerPipeClient constructors (mxaccessgw#185)
WorkerPipeSession had seven CreateEnvelope overloads that each forwarded to a
matching CreateBaseEnvelope overload; they are now one overload per body type
that stamps the header and sets the body. WorkerPipeClient's six chained
constructors collapse to one with optional arguments (null attempt timeout =
env/default, null session factory = production WorkerPipeSession). Adds
constructor-validation tests.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 13:24:05 -05:00
Joseph Doherty f49b7a0cc1 fix(worker): exit 5 on gateway death, dedicated exit 8 on event-queue overflow (mxaccessgw#186)
A frame read that hits end of stream means the gateway closed its end of
the pipe (usually because it died). WorkerApplication mapped every
WorkerFrameProtocolException to ProtocolViolation (6); EndOfStream now
exits as PipeConnectionFailed (5), alongside the IOException a broken
pipe raises.

Event-queue overflow exited as the generic UnexpectedFailure (1). The
event drain loop now unwinds with WorkerEventQueueFaultException carrying
the fault category, and QueueOverflow exits with the new
WorkerExitCode.EventQueueOverflow (8). Other categories keep exit 1.

The gateway does not decode worker exit codes (it only logs the number),
so no gateway change is needed. WorkerBootstrap.md and
MxAccessWorkerInstanceDesign.md updated. (mxaccessgw#186)

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 13:01:58 -05:00
Joseph Doherty cf575bfca8 fix(worker): reply WorkerUnavailable to commands that race shutdown (mxaccessgw#182)
TryStartCommandTask returned silently when the session had stopped
accepting commands, so a command racing shutdown got no reply and no log
and the gateway's correlation wait ran to its timeout. The message loop
now answers such a command with a WorkerUnavailable reply (the status the
STA dispatcher already uses during shutdown) and logs
WorkerCommandRejectedShuttingDown. New test
CommandAfterShutdown_RepliesWorkerUnavailable (mxaccessgw#182)

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 13:01:58 -05:00
Joseph Doherty f40409a688 docs: strip task-tracking IDs from code comments and fill missing XML docs
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m16s
ci / java (push) Successful in 2m5s
ci / portable (push) Failing after 5m9s
Tracking IDs are review bookkeeping that rots in shipped source; the rationale they tagged is kept. Comment-only change (CommentChecker 458 -> 16 false positives).
2026-10-07 15:34:32 -04:00
Joseph Doherty 577a0db862 fix(IPC-33): bound the DrainEvents byte budget against the gRPC send cap, not just the pipe frame max
The worker budgeted a DrainEvents reply against the negotiated pipe frame
maximum less one 64 KiB reserve, but the gateway returns that same
MxCommandReply verbatim through Invoke, where MaxSendMessageSize applies. The
two limits were coupled only from below (pipe >= gRPC + reserve), so any
configuration the validator permitted with a wider gap let the worker pack a
reply the pipe carried and the gRPC layer then refused with ResourceExhausted --
after Drain had already removed the events from the worker queue, losing them to
both the caller and the event stream.

Option (a) from the finding: close the coupling at both ends.

- GatewayOptionsValidator adds the upper half of the headroom rule
  (Worker:MaxMessageBytes <= Protocol:MaxGrpcMessageBytes + 2 x reserve), with a
  message naming the DrainEvents budget as the reason and the two knobs to move.
- WorkerPipeSession.ResolveDrainReplyByteBudget subtracts the reserve twice --
  once for the pipe frame, once so the MxCommandReply wrapper clears the gRPC
  send cap -- keeping the MaxMessageBytes/2 floor that makes small frame maxima
  usable.

The two changes are load-bearing for each other: the validator band is what
makes "frame max - 2 x reserve" provably at or below the gRPC cap, and the
doubled subtraction is what makes the band sufficient.

Docs updated in the same change (GatewayConfiguration.md both option rows plus
the headroom prose, WorkerFrameProtocol.md drain budget, gateway.md DrainEvents
byte axis). Tests: new GatewayOptionsValidatorTests cases for the upper bound
(reject) and both edges of the band (accept); GatewayOptionsTests' override case
now moves both message-size knobs together, since the band makes moving one
alone illegal.

The worker half cannot be built on macOS; verified on windev separately.
2026-08-20 09:30:39 -04:00
Joseph Doherty aaeb86b132 fix(worker-tests): decouple FakeRuntimeSession dispatch safety-net from the inspection window
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m13s
ci / java (push) Successful in 2m26s
ci / portable (push) Failing after 4m56s
The BlockDispatch branch waited 5 real seconds and then proceeded regardless — it does not
branch on the wait's result. The long-in-flight test's inspection loop is bounded by
elapsed time and a frame floor, so on a loaded box (the documented 4-5x slowdown class) it
can plausibly outrun that 5 s. When it does, the reply is emitted mid-window,
AssertNotWorkerFault waves it past, and the reply leg then waits for a reply already gone
by — failing at the 20 s cancellation with no message, on exactly the loaded-box run the
widened windows exist to survive.

The wait is a pure safety net: nothing asserts on it firing, and every test that blocks
dispatch releases it explicitly (ReleaseDispatch, or a WorkerShutdown envelope, both of
which Set the event) — none reaches the timeout on a healthy run. Named it
BlockedDispatchSafetyNet and raised it to 30 s, above any window a test opens and above the
20 s cancellation those tests arm, so a wedged test always fails on its own token with its
own message. Dispose still releases the wait, so teardown never waits on it either.

Inline rationale in the test now states the decoupling and what a close pairing would cost,
rather than asserting the window stays inside a 5 s ceiling. Nothing else changed.
2026-08-18 07:08:47 -04:00
Joseph Doherty 462850afb7 fix(worker-tests): doc-comment placement + watchdog-window headroom in long-in-flight test
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Failing after 17s
ci / java (push) Successful in 2m38s
ci / portable (push) Successful in 8m34s
Three review findings on 7da52b6:

1. The AssertNotFault doc block landed between the predicate-overload ReadUntilAsync's
   doc block and the helper, so the compiler attached the merged block to the helper and
   ReadUntilAsync lost its docs entirely. Helper and its docs moved above ReadUntilAsync,
   whose docs are back where they belong.

2. The compressed watchdog windows (50 ms grace, 100 ms ceiling) reintroduced the
   load-sensitivity the fix removed, one layer down.
   ReportWatchdogFaultIfNeededAsync measures staleness AFTER the heartbeat frame is
   written and flushed over the real named pipe, so a beat whose pipe I/O outlasts the
   ceiling faults a healthy session no matter how fresh the captured activity was. At
   100 ms that is a plausible stall on a loaded box, and this is the one test asserting
   the watchdog NEVER fires. Widened to a 200 ms grace and a 1 s ceiling — still two
   orders of magnitude under the 75 s production default.

   The inspection loop is now bounded by a 2 s window (twice the ceiling, so a fake whose
   activity stopped advancing still accumulates past it and faults) with a 30-frame floor,
   rather than a fixed 30 frames that no longer outran the wider ceiling. The floor keeps
   a window that saw almost no beats from passing as a clean one. Two seconds stays well
   inside FakeRuntimeSession's 5 s dispatch-block ceiling, so the command is still in
   flight for the whole window.

3. AssertNotFault renamed AssertNotWorkerFault, matching the WorkerFault body case it
   tests.

Scenario intent unchanged: long in-flight command, pump refreshing, zero fault frames,
reply delivered. Test-only.
2026-08-18 07:01:16 -04:00
Joseph Doherty 7da52b65b7 fix(worker-tests): model the STA pump at heartbeat capture — WorkerPipeSessionTests long-in-flight repro
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m11s
ci / java (push) Successful in 2m14s
ci / portable (push) Successful in 8m33s
RunAsync_LongInFlightCommandThatKeepsPumping_DoesNotFaultAndDeliversReply failed
deterministically on the Windows box with a StaHung fault whose command_method was
empty and whose staleness was 233 ms — i.e. a fault raised with no command in flight,
before the scenario under test began. FakeRuntimeSession stamps LastStaActivityUtc once,
at construction, and the test only started refreshing it after the blocked dispatch
signalled. Everything between those two points — handshake, STA init, the first
heartbeat — captured a snapshot already stale past the compressed 50 ms grace, with no
correlation id for the watchdog to suppress on, so the watchdog correctly reported the
fake as hung.

The harness, not the product, was wrong: StaRuntime.ThreadMain calls MarkActivity() on
every WaitForWorkOrMessages iteration, so a live worker is never captured stale, idle or
busy. Model that where it belongs — FakeRuntimeSession.RefreshStaActivityOnCapture (opt
in, default off) stamps activity at each CaptureHeartbeat and leaves the rest of the
snapshot alone — and arm it before RunAsync so the first beat is covered. The test-owned
refresh loop goes away with it; a thread-pool loop racing a compressed grace could not
have held the invariant anyway.

Scenario intent is unchanged and slightly stronger: the command still blocks in dispatch
across 30 heartbeats (~600 ms, many multiples of the 100 ms stuck ceiling), no frame may
be a fault, and the reply must still arrive. The reply leg is now fault-checked too
(previously it skipped frames blindly), and the pump keeps running across the release, as
it does in production while the reply is marshalled off the STA. Fault assertions now
report the category and diagnostic message instead of a bare body-case mismatch.

Test-only change; no product code, frame protocol, or STA rule touched.
2026-08-18 06:46:08 -04:00
Joseph Doherty b621d692d0 docs+test(closeout): final-review reservations — stale ACL prose, worker test gaps, config sample fix
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m14s
ci / java (push) Successful in 2m10s
ci / portable (push) Successful in 8m31s
2026-08-17 05:23:23 -04:00
Joseph Doherty 9130994736 perf(worker): unpark awaited control-frame writers from the winning drain pass
WriteAsync enqueued its frame and then contended unconditionally for the write
lock, so a caller that lost the race stayed in WaitAsync until the winning
drainer released — even though that winner writes, flushes, and completes the
loser's control frame at the control-to-event class boundary, part-way through
its pass. The boundary flush made the delivery point honest; the awaited task
was still charged for the whole event backlog it had just been flushed ahead of.

WriteAsync now awaits its own frame's completion racing the lock acquisition.
Completion first: the caller returns at its frame's delivery point and the
outstanding acquisition is detached, not dropped — a continuation drains
whatever is queued and releases, so the lock is never acquired and silently
held and a frame enqueued between the previous drainer's last dequeue and its
release is still written. Lock first: drain as before. Cancellation keeps the
WRK-22 tombstone semantics exactly, and a wait cancelled after the caller has
already detached releases nothing (SemaphoreSlim hands no count to a wait it
cancels), so no count leaks and no queued frame is stranded. A token that fires
after the frame's completion won the race changes nothing — the frame was
delivered. WriteBatchAsync deliberately keeps the plain wait-then-drain shape:
its last completion resolves at the end-of-pass flush anyway.

Three tests: the latency win (a control caller returning while the winning
WriteBatchAsync event burst is demonstrably still blocked mid-pass), a
mixed-priority concurrency soak pinning exactly-once writes and a single
drainer, and the cancel-after-detach corner (a wrongly released count would
surface as the drainer's own Release throwing SemaphoreFullException).

edited on macOS, windev verification pending (plan Task 11). Verified here by
compiling and running WorkerFrameWriter plus the writer suite against net10.0
in a scratch harness: 31/31 pass, and the two behaviour-pinning tests fail
against the pre-change parked implementation.
2026-08-17 03:55:05 -04:00
Joseph Doherty 3ef56be2dd fix(worker): unconditional fault observation for abandoned pipe I/O; exception-total transport dispose 2026-08-15 21:21:25 -04:00
Joseph Doherty e913dab5db fix(worker): session-owned stream disposal unblocks and observes the net48 pipe read at teardown 2026-08-15 21:06:03 -04:00
Joseph Doherty aac79579ab perf(worker): control-frame completions resolve at the class-transition flush, not after the event batch
The two-class writer already got control bytes out ahead of a queued event
backlog, but a frame counts as delivered only once flushed, and the drain
deferred its single FlushAsync — and every TrySetResult — to the end of the
pass. A heartbeat, command reply, fault, or shutdown ack was therefore written
first and completed last, behind up to a full 128-frame event batch.

The drain now records each frame's priority class on PendingFrame and flushes
at every control-to-event boundary, completing and clearing the written set
there. Cost stays bounded: a pure-event pass still pays exactly one flush, a
run of control frames still pays one for the run, and only a pass that mixes
both classes pays a second — never one flush per control frame, the
syscall-per-heartbeat cost WRK-12 removed.

A boundary flush that itself fails is a new failure window and is handled like
the end-of-pass flush failure, additionally failing the event frame the drain
had already claimed off its queue and every frame still queued. Frames a
boundary flush completed leave the written set, so a later failure in the same
pass can no longer reach back and fail an already-delivered control frame.

The awaited task of a caller that lost the write-lock race is still bounded by
the winning drainer's pass — that enqueue-then-contend parking is unchanged and
now documented on WriteAsync and in docs/WorkerFrameProtocol.md.
2026-08-15 21:05:57 -04:00
Joseph Doherty f4a6cb1db2 perf(worker): signal-driven event drain — removes the 25 ms latency floor and idle wakeups 2026-08-15 16:59:13 -04:00
Joseph Doherty bfcf82975c test(windev): buffer the test named pipes so the full-suite testhost exits
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m14s
ci / java (push) Successful in 2m19s
ci / portable (push) Successful in 8m48s
The windev full-suite wedge — every test reported, then the x64 testhost sitting
at ~0 CPU forever while `dotnet test` never returns — was one test blocked on a
pipe write, not a leaked thread or an undisposed fixture.

`dotnet-stack report` on the wedged host showed no thread running test code:
xUnit's RunTestsInAssembly was parked on WaitHandle.WaitOne() waiting for the
assembly-finished event, so the wait lived in a suspended async state machine.
`dotnet-dump analyze -c dumpasync` named the frame — WorkerClientTests
.StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout awaiting
WorkerFrameWriter.WriteAsync on a 63-byte frame, with <extra>5__15 = 6, i.e. the
seventh of the twelve events the test pushes past the client's staging bound.

That test faults the worker client on purpose, and a faulted client stops its
read loop by design. The test-side pipe came from the NamedPipeServerStream
overload without buffer arguments, which passes inBufferSize: 0 / outBufferSize: 0
to CreateNamedPipe; on Windows that reserves no buffer at all, so a write
completes only once the peer reads it. Measured on windev, that pipe absorbed
0 bytes against a non-reading peer where the same pipe declared with 64 KiB
buffers absorbed 65 520. On macOS and Linux .NET backs named pipes with Unix
domain sockets whose socket buffer swallows the writes regardless, which is why
the identical test never hung there and the bug read as environmental.

Test-owned server pipes now go through TestSupport/TestNamedPipe.CreateServer in
both test projects, declaring explicit 64 KiB buffers so those tests exercise the
gateway's own staging/queue backpressure rather than the OS pipe's flow control.

Separately, every fake-worker write in WorkerClientTests now goes through
PipePair.WriteAsync, bounded by the class's five-second TestTimeout. That is
where the severity came from: a test method that never returns keeps xUnit from
raising ITestAssemblyFinished, so one unbounded await cost the whole suite its
result. A blocked write is now a named test failure instead of a silent wedge.

The fix also retires a wrong belief the wedge had created. windev reported 855
where macOS reported 879, and that gap was recorded in GatewayTesting.md as
Unix-gated test cases; it was really the results lost when the wedged host was
torn down. The same clone now reports 879 passed, matching macOS exactly.

Product code is unaffected. SessionWorkerClientFactory.CreatePipe keeps the
unbuffered declaration deliberately: both ends run continuous read loops and every
gateway write is bounded by the worker client's _stopCts, so a stalled peer
cancels the write rather than blocking on it.

Verified on windev at this SHA: gateway suite x64 three times (879 passed,
exit 0, no surviving testhost each time) and Worker.Tests x86 twice (400 passed,
11 skipped, exit 0, clean), plus the macOS gateway suite once (879 passed).

Docs: GatewayTesting.md replaces the --blame-hang workaround section with the
root cause and corrects the baseline to 879, CLAUDE.md's Source Update Workflow
no longer tells readers the windev suite wedges, and ToolchainLinks.md records
dotnet-stack and dotnet-dump as installed on windev.
2026-08-10 10:01:57 -04:00
Joseph Doherty 50322bacb3 test(worker): deflake EventBurst_DrainLoopCoalescesFlushes
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m13s
ci / java (push) Successful in 2m38s
ci / portable (push) Successful in 8m17s
Root cause. The test sampled its flush baseline after a bare `Task.Delay(100)`
and then charged every later flush to the 128-event burst. Two facts make that
window unsound:

1. `RunHeartbeatLoopAsync` sends its first beat *immediately* on entering the
   message loop — the far-off `HeartbeatInterval` only spaces later beats — so
   the pre-burst frames are WorkerHello, WorkerReady, and a heartbeat, not the
   two the test's comment assumed.
2. `WorkerFrameWriter.DrainQueuedFramesAsync` flushes *after* writing a drained
   batch, so the bytes reach the pipe before the flush runs. Reading a frame off
   the gateway side is therefore no evidence that its flush has been counted,
   and no sleep makes it evidence.

Under load on the shared windows-x86 runner the first heartbeat's flush was
scheduled after the 100 ms sample, so it landed inside the measured window and
the assertion saw two flushes for the burst — exactly the observed
`Expected: 1 / Actual: 2` (Gitea run 675, job 2558, and the same failure since
a346d51). Nothing about the coalescing behavior was wrong; only the test's
timing assumption.

Fix. Replace the sleep with explicit synchronization, no widened timeouts.
`FlushCountingPassthroughStream` now records the flush *shape* — the number of
stream writes coalesced into each flush — and exposes
`WaitForAllWritesFlushedAsync`, a TaskCompletionSource signal released when the
next flush drains the pending writes. The test reads the first heartbeat (the
last pre-burst frame), waits for its flush, and only then samples the baseline;
after the burst it takes the same edge before asserting. The assertion is also
sharpened from a bare count to the shape: exactly one flush beyond the baseline
*and* that flush carried all 128 event frames, so a split batch fails even if
the reader observes it mid-split.

docs/WorkerFrameProtocol.md notes the peer-visible ordering the fix turns on:
frames reach the pipe before the flush that follows them, so an observer of the
flush must wait for it rather than infer it from frames arriving.
2026-08-10 08:13:30 -04:00
Joseph Doherty 84dbf20a43 fix(worker): observe faults on frames abandoned by cancellation (NEXT-04, NEXT-05 decision)
A WriteAsync/WriteBatchAsync caller cancelled after the draining lock-holder
claimed its frame unwinds without awaiting that frame's completion; the same
holds for a frame already faulted by a concurrent FailAllQueued, where
TrySetCanceled loses. A later wire-write failure then lands TrySetException on
a task with no awaiter and surfaces as TaskScheduler.UnobservedTaskException.
The tombstone helpers now attach a fault-observing continuation to every frame
in the cancelled call (a cancelled task never fires OnlyOnFaulted, so
unconditional attach is safe), outside _gate because an already-faulted task
runs the continuation inline.

NEXT-05 is resolved as a documented decision, not a code change: tombstoned
entries keep their lazy DequeueNext purge — any subsequent write drains both
queues to empty and the heartbeat loop bounds residency to one interval, while
eager Queue<T> rebuilds under _gate would add ordering-invariant surface for
no gain. Rationale recorded in docs/WorkerFrameProtocol.md alongside the
WRK-22 residual-window contract.

New regression test drives the exact abandonment: gated stream holds writer A
mid-write, the queued event frame is claimed and blocked mid-write, its caller
is cancelled, the write then faults with a marker exception, and the test
asserts the marker never reaches UnobservedTaskException after a forced GC.
net48 x86 build/test runs on windev with the rest of this batch.
2026-08-10 05:57:53 -04:00
Joseph Doherty 8df35cd63a fix(WRK-22,WRK-24,WRK-25,WRK-27,IPC-26): worker write-seam hardening
ci / java (push) Successful in 2m7s
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Failing after 1m13s
ci / portable (push) Failing after 4m6s
WRK-22/IPC-26: tombstone a WriteAsync/WriteBatchAsync cancelled while
waiting for the write lock (PendingFrame.Claimed under _gate; DequeueNext
skips cancelled, claims the frame it returns) so a cancelled write never
reaches the wire unless already claimed mid-write (documented residual).

WRK-25: add WriteBatchAsync; RunEventDrainLoopAsync submits the drained
event batch through it, so a burst of N events costs one flush not N.
IPC-30 oversized-event structured fault preserved via FindOversizedEvent.

WRK-24: reject a below-1024 negotiated frame maximum at the handshake
(MinNegotiableFrameBytes, matching GatewayOptionsValidator floor).

WRK-27: alarm poll advertises StaCallInProgress on the heartbeat snapshot
so the watchdog suppresses to the ceiling, not the grace.

Docs (WorkerFrameProtocol.md, MxAccessWorkerInstanceDesign.md) and the
2026-07-12 remediation registers/change-log updated in the same commit.
2026-08-07 07:50:38 -04:00
Joseph Doherty 6bc3f9b991 fix(WRK-21): make drain budget monotonic at the reserve boundary; guard the reply-too-large fallback
ci / windows-x86 (push) Successful in 1m16s
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m17s
ci / portable (push) Successful in 9m31s
Review follow-ups on the WRK-21 cluster.

1. ResolveDrainReplyByteBudget was a step function, not a floor: just above the
   64 KiB reserve the budget collapsed to a few bytes (at the validator-permitted
   floor MaxMessageBytes = 1024 + 64 KiB it was exactly 1024), too small to move a
   byte-heavy event, so DrainEvents truncated on every call and the drain-until-
   empty loop never terminated. It now takes the max of (frameMax - reserve) and
   frameMax/2, so the budget is monotonic and never below half the frame max. New
   test DrainEvents_AtValidatorFloorFrameMax_MakesProgressAndTerminates drives a
   byte-heavy queue at the exact validator floor and asserts it drains to empty
   with no head ever reported oversized.

2. The reply-too-large fallback write is now itself size-guarded
   (WriteReplyTooLargeFallbackAsync, used by both the control and STA reply seams):
   at a pathologically tiny negotiated max below the gateway's floor the fallback
   could also throw MessageTooLarge and — uncaught — kill the session, defeating the
   "no diagnostics command is session-fatal" invariant. It now log-and-swallows;
   comment notes WRK-24 adds the negotiated-max lower bound that makes it unreachable.

3. Corrected the RepeatedFieldOverheadBytes doc comments: WorkerEvent.CalculateSize()
   already includes the event's tag and length prefix (the same shape the reply's
   repeated events field packs), so the 8 bytes is pure slack over an already-
   conservative estimate, not compensation for a missing wrapper.
2026-08-07 07:09:13 -04:00
Joseph Doherty a2565604df test(WRK-21): keep the drain-to-empty walk inside the pipe harness envelope
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m14s
ci / java (push) Successful in 2m12s
ci / portable (push) Successful in 8m2s
PipePair runs both ends of a duplex pipe in one process with blocking
FlushFileBuffers under every frame write, so it wedges after roughly 85 large
round trips. Drain the full 10,000 byte-heavy events to empty at the queue layer,
where the no-loss property actually lives, and keep the pipe walk at 1,000 events
(29 replies) so it still proves the split end to end. Also give the truncation
test's budget slack: item handle 0 is a proto3 default and is not serialized, so
the probe measurement is a lower bound on the fixture's per-event cost.
2026-08-07 06:50:17 -04:00
Joseph Doherty 7c2eaf09e2 test(WRK-21): size the byte-heavy drain fixture for the pipe harness
ci / java (push) Successful in 2m12s
ci / portable (push) Successful in 8m23s
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Failing after 11m46s
PipePair has no continuous read pump — the test thread drains the pipe only
while it sits in ReadUntilAsync — so multi-megabyte DrainEvents frames
interleaved with the heartbeat loop wedge both ends inside FlushFileBuffers,
each waiting for the other to read. Negotiate a 128 KiB frame maximum instead:
the 10,000 byte-heavy events still overflow it many times over, so every
assertion (bounded reply, reported truncation, no event loss across repeated
drains, surviving session) is unchanged.
2026-08-07 06:25:15 -04:00
Joseph Doherty 33ba612ddd fix(WRK-21,WRK-28,WRK-23,IPC-30): byte-budget the DrainEvents reply, stop size errors from killing sessions
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m8s
ci / portable (push) Successful in 7m41s
ci / windows-x86 (push) Failing after 12m32s
WRK-21 — DrainEvents was bounded by event count only, so a byte-heavy queue
(large string/array MxValues) built a reply above the negotiated frame maximum:
the writer rejected the frame, the exception unwound the session, and the events
already dequeued were destroyed. The drain is now byte-budgeted inside the queue
lock, so an event is dequeued only once it is known to fit and one that does not
stays at the head. Truncation is reported through the reply's existing
DiagnosticMessage (no contract change); callers drain until an empty reply. Both
reply-write seams — the control-command path and ProcessCommandAsync — now catch
MessageTooLarge and answer the correlation with an InvalidRequest reply instead
of unwinding or faulting the session. Satisfies IPC-23 R1-R3.

WRK-28 — the 10,000 drain ceiling moves to GatewayContractInfo
.MaxDrainEventsPerCommand, referenced by both the gateway request validator and
the worker clamp, replacing a comment-only sync contract. C# const only; no
.proto change.

WRK-23 — WorkerFrameWriter now peek-stamps, validates, then commits the sequence
counter immediately before the stream write, so a per-frame rejection leaves no
phantom gap on the wire.

IPC-30 — an oversized event frame stays session-fatal (it is undeliverable end to
end and neither dropping nor synthesizing a replacement is allowed), but the
death is structured: the event's identity and sizes are logged (never its value),
a WorkerFault with category PROTOCOL_VIOLATION and command method EventDrain is
written, then the session exits as before.

Docs updated in the same change: MxAccessWorkerInstanceDesign.md (drain byte cap,
truncation contract, oversized-head behavior, oversized-event policy, no control
reply is session-fatal on size), WorkerFrameProtocol.md (reply pre-sizing,
non-fatal reply-size rule, oversized-event policy, rejected frames do not consume
sequence numbers), gateway.md (DrainEvents two-axis bound).
2026-08-07 05:38:23 -04:00
Joseph Doherty 4f5371fdec Merge branch 'fix/archreview-p2' into main (P2 tier: completeness & polish)
# Conflicts:
#	archreview/remediation/00-tracking.md
#	clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli/MxGatewayCliSecretRedactor.cs
#	clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli/MxGatewayClientCli.cs
#	src/ZB.MOM.WW.MxGateway.Worker.Tests/Ipc/WorkerFrameProtocolTests.cs
2026-07-12 22:12:41 -04:00
Joseph Doherty b86c6bb47f docs: complete XML-doc coverage and strip internal tracking IDs from code comments
ci / java (push) Successful in 5m51s
ci / portable (push) Successful in 6m43s
Resolve all CommentChecker findings across the gateway server, worker, tests,
and .NET client (314 -> 0 real issues): add missing <returns>/<summary>/<param>
on public and test members, convert Stream/interface overrides to <inheritdoc/>,
and remove internal task/issue tracking IDs (SEC-*, IPC-*, WRK-*, GWC-*, TST-*,
Client.Dotnet-*) from shipped code documentation while preserving the design
rationale prose. Shipped comments should not carry internal bookkeeping, and
complete XML docs keep the analyzer/TreatWarningsAsErrors gate and generated API
docs clean. The 6 remaining flags are heuristic false positives (MD5, UTC-4,
capacity-1, near-1601) left intact so real documentation is not corrupted.

Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
2026-07-10 06:15:47 -04:00
Joseph Doherty f61c816acf perf(worker): event hot-path allocation + flush cuts (WRK-06/11/12, IPC-15)
WRK-06: MxStatusProxyConverter caches the four resolved FieldInfo per status
type in a static ConcurrentDictionary (the GetField metadata scan ran 4x per
status per event on the STA path). GetValue+Convert.ToInt32 still run per event
(late-bound RCW). Exceptions byte-identical: missing-field message unchanged
(ResolveField, not cached on throw via GetOrAdd); null-value message unchanged.

WRK-11: MxAccessEventQueue.Enqueue takes ownership of the passed MxEvent -
stamps WorkerSequence/WorkerTimestamp on it in place and enqueues it, no
Clone(). Audited all 3 callers (base/alarm event sinks, provider-mode handler):
each builds a fresh event per Enqueue, none reuse it. MxAccessValueCache.Set now
deep-copies its retained Value/SourceTimestamp/Statuses so the cache snapshot
never aliases the queue-owned (later serialized) event. Net: alarm/other events
clone nothing (was full clone); data-change clones payload-only.

WRK-12: WorkerFrameWriter coalesces the flush across a drained batch - each
frame is written but not flushed individually; one FlushAsync after the batch,
then all written frames complete. Preserves the written+flushed completion
contract; a burst of N events costs 1 flush, not N. On write failure the whole
in-flight batch + queue fail so no caller hangs.

IPC-15 (doc): the multi-event WorkerEnvelope body remains unimplemented (wire
still carries one event per worker_event frame); gateway.md Performance section
now distinguishes the shipped flush-coalescing from that deferred proto change.

net48-safe (no init/records; readonly struct cache entry). Worker builds x86
only - verification on windev. Tests added: converter cache-reuse, queue
ownership-transfer, value-cache snapshot independence, writer batch-flush-once.

Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
2026-07-09 15:59:45 -04:00
Joseph Doherty ebe6aeac98 feat(worker): adopt negotiated frame max, bound drain, priority write scheduler (IPC-02/04 + WRK-04/07 worker half)
Worker half of the Wave 3 size/backpressure + write-ordering pass:

- IPC-02: the worker adopts GatewayHello.max_frame_bytes during the handshake
  (WorkerFrameProtocolOptions.AdoptNegotiatedMaxMessageBytes) instead of a
  hard-coded default; 0 keeps the default, a value above a 256 MiB ceiling is
  rejected. Reader and writer share the options instance, applied before the
  message loop.
- IPC-04: DrainEvents caps each reply at MaxDrainEventsPerReply (10_000) and
  treats max_events = 0 as that cap rather than 'drain the entire queue', so one
  diagnostic drain cannot pack a session-killing reply frame.
- WRK-04: WorkerFrameWriter stamps the envelope Sequence at the actual point of
  writing (under the write lock) instead of at envelope creation, so the on-wire
  order and the stamped sequence always agree under concurrent producers.
- WRK-07: the writer is now a cooperative priority scheduler — callers enqueue at
  Control or Event priority and the draining lock-holder writes all control
  frames before any event frame, so replies/faults/heartbeats jump ahead of an
  event backlog. Per-frame validation/size rejections fail only that frame; a
  stream write failure fails all queued frames.

Tests: monotonic gap-free sequence under concurrency, control-before-event
priority (gated stream), negotiated-max adoption, DrainEvents zero-bound.
Worker builds x86 only — verified on windev.
2026-07-09 09:09:14 -04:00
Joseph Doherty 31eec41456 fix(WRK-01): STA pump step refreshes activity to prevent false StaHung
A long legitimate ReadBulk pumped Windows messages without refreshing
LastStaActivityUtc, so the watchdog false-positived StaHung past
HeartbeatStuckCeiling and then silently dropped every reply. PumpPendingMessages()
now calls MarkActivity() after pumping; a genuinely stuck STA (no pumping)
still accrues staleness and faults correctly. No MXAccess parity change.

archreview: WRK-01 (P0). Verified on the Windows host (x86): worker builds
clean, StaRuntimeTests + WorkerPipeSessionTests 33/33 pass.
2026-07-09 05:51:45 -04:00
Joseph Doherty fca978de07 docs(src): add missing XML docs and strip tracking-ID comments
Sweep of 203 source files resolving CommentChecker findings: add
<summary>/<param>/<returns>/<inheritdoc> where missing, and remove
resolved task/issue tracking markers (Tests-NNN, Worker-NNN, Server-NNN,
Task N) from code comments. Comment/doc-only — no logic changes.
Server+Tests build clean under TreatWarningsAsErrors.
2026-07-07 14:09:49 -04:00
Joseph Doherty 8df0479b99 fix: resolve Client.Java + Worker.Tests findings (pending windev verification)
Client.Java-040..048, Worker.Tests-034/035/036. Edits applied on the Mac,
which has no JRE and cannot build the x86+MXAccess worker tests; findings are
marked In Progress pending gradle + x86 build verification on windev. Do not
mark Resolved until verified there.
2026-06-17 05:23:14 -04:00
Joseph Doherty f94c206489 test(worker): use Register (a real STA command) for STA-dispatch race tests
Ping is now intercepted as a worker control command and answered on the
message-loop thread, so the dispatch/heartbeat/shutdown-race tests must use a
genuine STA-dispatched command kind to keep exercising DispatchAsync.
2026-06-15 10:25:34 -04:00
Joseph Doherty 72e1aca716 test(worker): fix control-command test helpers (CreatePipeSession overload, drop ConfigureAwait) 2026-06-15 10:23:45 -04:00
Joseph Doherty bf72cd8961 feat(worker): implement Ping/GetSessionState/GetWorkerInfo/DrainEvents/ShutdownWorker control commands
Answer the five worker control/lifecycle commands at the WorkerPipeSession
message-loop layer instead of the STA-bound MxAccessCommandExecutor. These
replies are built from process-level state (worker pid, assembly version,
worker lifecycle, the runtime session's event queue) the executor cannot see,
and ShutdownWorker must emit its OK reply before the graceful shutdown joins
the STA thread - dispatching it onto the STA would deadlock.

- Ping: OK reply, echoes message into diagnostic_message.
- GetSessionState: maps WorkerState to proto SessionState.
- GetWorkerInfo: pid, worker version, MXAccess ProgID/CLSID.
- DrainEvents: drains the runtime event queue into DrainEventsReply.
- ShutdownWorker: OK reply, then graceful shutdown, then stops the loop.

Tests added in WorkerPipeSessionTests; FakeRuntimeSession gains a
batch-size drain suppressor so DrainEvents does not race the background
drain loop.
2026-06-15 10:20:51 -04:00
Joseph Doherty 615b487a77 docs+ui: backfill XML doc comments and finish dashboard layout pass
Adds missing <summary>/<param> XML docs across 99 server, worker, and test
files so CommentChecker reports zero issues (TreatWarningsAsErrors needs the
analyzer clean). Bundles in WIP dashboard work: NavSection extraction,
MainLayout/site.css/js styling alignment, and DashboardOptions/Auth tweaks.
2026-05-27 14:20:10 -04:00
Joseph Doherty dc9c0c950c rename: prefix gateway projects/namespaces with ZB.MOM.WW + sln→slnx
Apply the ZB.MOM.WW. prefix to all gateway-side projects, folders,
.csproj/.sln contents, C# namespaces, using directives, generated proto
C# (csharp_namespace + checked-in generated files), InternalsVisibleTo
attributes, project-name string literals (LoadProject, .sln lookups,
worker exe paths, staticwebassets manifest), and the install/script/doc
references that point at any of the above. Migrate the solution from
.sln to .slnx via `dotnet sln migrate` and delete the old file.

External-runtime identifiers are intentionally NOT prefixed so external
configuration keeps working:
- GatewayMetrics.cs MeterName ("MxGateway.Server")
- DashboardAuthenticationDefaults Scheme/Policy ("MxGateway.Dashboard")
- GatewayRequestLoggingMiddleware logger category ("MxGateway.Request")
- StaRuntime thread name ("MxGateway.Worker.STA")
- appsettings.json root section "MxGateway" + env-var prefix
  MxGateway__... and secret-name MxGateway:ApiKeyPepper
- C:\ProgramData\MxGateway\ data dir paths

Also fixes two tests that were not rename-related but became visible
while validating the rename:

- WorkerLiveMxAccessSmokeTests.ShutDownAsync: cancellation that the
  gateway service correctly maps to RpcException(Cancelled) per gRPC
  convention was being misclassified as a stream fault. Added a sibling
  catch on RpcException with StatusCode.Cancelled.

- IntegrationTestEnvironment.ResolveRepositoryRoot: extracted IsRepositoryRoot
  and made it accept either a .git marker OR a .sln/.slnx next to src/
  so the worker-exe walker works in non-git working copies.

clients/proto/proto-inputs.json's protoRoot updated to point at
src/ZB.MOM.WW.MxGateway.Contracts/Protos.

Verified by `dotnet build` and a full `dotnet test` of the .slnx with
MXGATEWAY_RUN_LIVE_{MXACCESS,LDAP,GALAXY}_TESTS=1:
  Tests: 472/472 pass
  Worker.Tests: 280/280 pass (4 dev-rig [Fact(Skip=...)] skipped)
  IntegrationTests: 18/18 pass

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 16:22:23 -04:00