mxaccessgw/archreview/2026-07-12/20-worker.md

Worker Process — Architecture Review (re-run)

Scope & method

Date 2026-07-12 · commit 4f5371f (main) · macOS static review — the worker targets .NET Framework 4.8 x86 and was not built or tested here; every claim cites file:line evidence read directly from the working tree at HEAD. Scope: src/ZB.MOM.WW.MxGateway.Worker (STA runtime/pump, MxAccessSession COM lifetime, Ipc/ pipe session + frame reader/writer, event queue, alarm consumers, program lifecycle/exit codes) and src/ZB.MOM.WW.MxGateway.Worker.Tests. This is a verification pass over the 2026-07-08 review (archreview/20-worker.md, remediation IDs WRK-01..WRK-20 in archreview/remediation/20-worker.md / 00-tracking.md) plus a deep review of everything changed since the pre-remediation baseline 59856b8 and a lighter fresh sweep.

Changed-in-scope since 59856b8 (17 files, +890/−86): Ipc/WorkerFrameWriter.cs (priority scheduler, sequence-at-write, batch flush), Ipc/WorkerFrameWritePriority.cs (new), Ipc/WorkerFrameProtocolOptions.cs (negotiated max adoption), Ipc/WorkerPipeSession.cs (event priority tag, DrainEvents bound, sequence removal), Sta/StaRuntime.cs (pump refreshes activity), Conversion/MxStatusProxyConverter.cs (FieldInfo cache), MxAccess/MxAccessEventQueue.cs + MxAccessValueCache.cs (clone elimination + cache snapshot), comment strips in the two alarm consumers, and six test files.

Executive summary

• All seven Done-claimed fixes (WRK-01, 04, 06, 07, 11, 12, 15) are present at HEAD and fundamentally sound; none is a false Done. Two carry caveats: WRK-12's flush coalescing never engages on the pure event hot path it was aimed at (WRK-25), and WRK-07 landed without updating the component design doc that still specifies a five-level priority order (WRK-26).
• The new cooperative frame-write scheduler is well built — enqueue-then-contend, control-before-event drain under a single lock, sequence stamped at the moment of writing, per-frame rejections isolated from stream failures, no deadlock or lost-wakeup path found. Its weakest edges are cancellation (a cancelled WriteAsync leaves its frame queued and it is still written later, WRK-22) and sequence numbers consumed by rejected frames (WRK-23).
• The most significant new defect is that the DrainEvents bound is count-based only: 10,000 large events can still exceed the negotiated frame max, and because the drained events were already removed from the queue, the oversized-reply rejection both loses those events and unwinds the whole session as a protocol violation (WRK-21) — the exact "session-killing reply frame" the bound was added to prevent.
• All thirteen open findings (WRK-02/03/05/08/09/10/13/14/16/17/18/19 and the WRK-02/03 slice of WRK-20) remain present at HEAD, none incidentally fixed. The highest-value remaining items are still WRK-02 (silent STA thread death) and WRK-05 (one alarm-poll failure kills the session).
• Fresh sweep: the alarm poll loop runs outside the command dispatcher, so it never gets the watchdog's in-flight suppression — a single slow PollOnce COM call faults StaHung after the 15 s grace instead of the 75 s ceiling other commands get (WRK-27).

Prior-finding verification

ID	Claimed status	Verified?	Evidence (file:line at HEAD)	Notes
WRK-01	Done	Yes	Sta/StaRuntime.cs:95-100 (PumpPendingMessages → MarkActivity()); wiring MxAccess/MxAccessStaSession.cs:208 (pumpStep: () => staRuntime.PumpPendingMessages()), MxAccess/MxAccessSession.cs:885 (per-tag wait routes through pumpStep); docs docs/MxAccessWorkerInstanceDesign.md:690-703; tests StaRuntimeTests.PumpPendingMessages_RefreshesLastActivity, WorkerPipeSessionTests.RunAsync_LongInFlightCommandThatKeepsPumping_DoesNotFaultAndDeliversReply	Sound. A ReadBulk that keeps pumping stays fresh past the 75 s ceiling; a stuck (non-pumping) STA still faults. Residual by design: a single blocking COM call cannot pump and still faults at the ceiling. See WRK-27 for the alarm-poll asymmetry the fix does not cover.
WRK-02	Not started	Still open	Sta/StaRuntime.cs:265-269 (loop exception stored in write-only startupException, startedEvent.Set()); Sta/StaRuntime.cs:161-189 (InvokeAsync has no terminated check; enqueues into a consumer-less queue after thread death)	finally at :272 cancels only work queued at death; anything enqueued after death hangs forever, exception never logged/faulted. Unchanged since baseline.
WRK-03	Not started	Still open	Ipc/WorkerPipeSession.cs:717-720 (if (!_acceptingCommands) { return; } — no reply, no log)	Silent drop during shutdown race unchanged.
WRK-04	Done	Yes	Ipc/WorkerFrameWriter.cs:236-240 (sequence stamped inside WriteFrameAsync, which runs only under _writeLock per :116-125); Ipc/WorkerPipeSession.cs:1025-1035 (CreateBaseEnvelope no longer sets Sequence; NextSequence()/_nextSequence removed from the session); test WorkerFrameProtocolTests.WriteAsync_UnderConcurrentCalls_StampsGapFreeMonotonicSequence	Sound: wire order and stamped sequence agree under concurrency and priority reordering. Nuance: rejected frames consume sequence numbers (WRK-23).
WRK-06	Done	Yes	Conversion/MxStatusProxyConverter.cs:23 (static ConcurrentDictionary<Type, StatusFields>), :122-142 (GetFields/ResolveField, throw-through GetOrAdd so a bad type is not cached), :96-108 (per-field GetValue+Convert.ToInt32 retained), :186-207 (plain readonly struct, net48-safe)	Sound; exception messages preserved (missing-field from ResolveField, null-value from ReadInt32Field). Test Convert_RepeatedForSameType_ProducesIdenticalResults.
WRK-07	Done	Yes, with doc caveat	Ipc/WorkerFrameWritePriority.cs:10-17 (Control/Event); Ipc/WorkerFrameWriter.cs:88-98 (priority enqueue), :200-216 (DequeueNext control-first), :141-157 (per-frame rejection vs stream-failure semantics), :218-232 (FailAllQueued); Ipc/WorkerPipeSession.cs:374-382 (events tagged Event); test WriteAsync_WhenControlAndEventQueuedTogether_WritesControlFirst	Scheduler is correct and deadlock-free (analysis below). But it is a two-class scheduler while docs/MxAccessWorkerInstanceDesign.md:607-613 still specifies the five-level order (faults > replies > shutdown acks > heartbeats > events) — within Control it is FIFO. Doc not updated in the same change (WRK-26).
WRK-05	Not started	Still open	MxAccess/MxAccessStaSession.cs:278-291 (any single poll exception → RecordFault → loop stops); drain loop turns fault into session termination Ipc/WorkerPipeSession.cs:356-365	One transient alarm E_FAIL still kills the whole session (default standbyFactory: null, Ipc/WorkerPipeSession.cs:60).
WRK-08	Not started	Still open	Ipc/WorkerPipeSession.cs:298-304 (drain loop cancelled in finally after shutdown ack, residual queue unshipped); docs/MxAccessWorkerInstanceDesign.md:705-718 shutdown section still silent on the discard
WRK-09	Not started	Still open	No AppDomain/UnhandledException/UnobservedTaskException reference anywhere under src/ZB.MOM.WW.MxGateway.Worker/ (grep at HEAD)
WRK-10	Not started	Still open	WorkerApplication.cs:112-117, 123-127, 133-137 (all three catch blocks log exception_type only, never the redacted message)	The redactor (Bootstrap/WorkerLogRedactor.cs) still exists and works (nonce redaction verified via WorkerConsoleLogger.cs:36), so this stays a cheap win.
WRK-11	Done	Yes	MxAccess/MxAccessEventQueue.cs:146-160 (stamps sequence/timestamp on the caller's instance, no Clone(), ownership invariant documented :11-21); MxAccess/MxAccessValueCache.cs:43-70 (cache deep-copies Value/SourceTimestamp/Statuses.Clone() so the snapshot never aliases the queue-owned event); callers verified single-ownership (MxAccess/MxAccessBaseEventSink.cs:205-256 builds a fresh event per enqueue; postPublish runs after enqueue at :243-256 and only reads the event — safe against the concurrent drain-side serialization, which is also read-only)	Sound. Tests Enqueue_TakesOwnershipOfPassedEventInstance, Set_StoresIndependentSnapshot_UnaffectedByLaterEventMutation.
WRK-12	Done	Yes, but partially ineffective	Ipc/WorkerFrameWriter.cs:120-124, 160-182 (write each frame, one FlushAsync per drained batch, complete after flush; flush failure fails the whole written batch); test WriteAsync_WhenBatchDrainedTogether_FlushesOnce	The writer-side mechanism is correct, but the production event drain loop awaits each event write to completion before issuing the next (Ipc/WorkerPipeSession.cs:374-382), so the writer's queue never accumulates an event batch — see WRK-25. The tracking-log claim "a burst of N events costs 1 flush not N" does not hold on the event path.
WRK-13	Not started	Still open	Ipc/WorkerFrameWriter.cs:261-266 — still new byte[frameLength] per frame (now a single combined prefix+payload buffer and one stream write, a modest improvement over baseline, but still unpooled vs the reader's ArrayPool at Ipc/WorkerFrameReader.cs:55-77)
WRK-14	Not started	Still open	Ipc/WorkerPipeSession.cs:28-44 _camelCase vs MxAccess/MxAccessSession.cs:11-14 and Sta/StaRuntime.cs:10-24 bare camelCase
WRK-15	Done	Yes	docs/WorkerSta.md:23,29 and docs/MxAccessWorkerInstanceDesign.md:254 now state the actual thread name MxGateway.Worker.STA (matches Sta/StaRuntime.cs:61); heartbeat-counter note fixed at docs/MxAccessWorkerInstanceDesign.md:651-654 ("populated from the live event queue")
WRK-16	Not started	Still open	Ipc/WorkerPipeSession.cs:940-1023 (seven CreateEnvelope/CreateBaseEnvelope pairs); Ipc/WorkerPipeClient.cs (7 public constructors)
WRK-17	Not started	Still open	WorkerApplication.cs:110-119 maps every WorkerFrameProtocolException (including EndOfStream) to ProtocolViolation (6); PipeConnectionFailed (5) reserved for IOException/TimeoutException at :121-129
WRK-18	Not started	Still open	Ipc/WorkerPipeSession.cs:356-365 (drain-fault writes fault then throws InvalidOperationException) → generic catch WorkerApplication.cs:131-139 → exit 1	Overflow doc text at docs/MxAccessWorkerInstanceDesign.md:614-621 ("stop accepting new commands") also still stale.
WRK-19	Not started	Still open	Sta/StaCommandDispatcher.cs has no logger and no start/end log (grep: no Information/Error emit sites)
WRK-20	Not started	Partially closed as a side effect	Added since baseline: pump-refresh + long-in-flight-no-fault tests (WRK-01), gap-free concurrent-sequence test (WRK-04), control-before-event and flush-once tests (WRK-07/12), converter cache, queue ownership, cache snapshot, DrainEvents-bound tests	Still missing (because their fixes are open): STA-thread-death test (WRK-02) and command-refused-during-shutdown test (WRK-03).

Done claims that failed verification outright: none. Defective/incomplete aspects of Done claims: WRK-12 effectiveness (WRK-25) and WRK-07 documentation (WRK-26).

Deep review of the new code

Frame-write scheduler correctness. Enqueue happens before the lock wait (Ipc/WorkerFrameWriter.cs:87-98), every caller then contends for _writeLock and the winner drains all queued frames control-first (:100-113, :125-182), so there is no lost-wakeup: a frame enqueued while another holder drains is either drained by that holder or by its own caller's subsequent lock acquisition. Completions use RunContinuationsAsynchronously (:28), stream writes run with CancellationToken.None so frames are never half-written (:116-118, :266), _nextSequence is only touched under the lock (:44-48), and stream failure fails the current frame, the written-but-unflushed batch, and everything queued (:148-157, :165-176) so no caller hangs. Per-frame rejections (InvalidEnvelope/MessageTooLarge/version/session, :192-198) correctly fail only that frame. Event ordering is preserved because the single drain loop awaits each event write (Ipc/WorkerPipeSession.cs:374-382). No deadlock or STA-affinity issue found — the writer runs entirely on thread-pool/loop threads.

Negotiated max adoption. AdoptNegotiatedMaxMessageBytes (Ipc/WorkerFrameProtocolOptions.cs:124-140) is applied in ValidateGatewayHello before WorkerHello is written and before the message loop starts (Ipc/WorkerPipeSession.cs:235-239 then :192), so the single-threaded-handshake mutation claim holds; the reader/writer read the property per frame and pick the negotiated value up for all post-hello frames. 0-keeps-default and >256 MiB-rejected are implemented and tested. Gap: no lower bound (WRK-24).

DrainEvents bound. max_events = 0 and over-large requests are clamped to MaxDrainEventsPerReply = 10_000 (Ipc/WorkerPipeSession.cs:21-26, 545-557), matching the gateway validator's MaxDrainEventsPerRequest = 10_000 (src/ZB.MOM.WW.MxGateway.Server/Grpc/MxAccessGrpcRequestValidator.cs:13,81-85). Semantics change from "drain all" to "up to 10,000" is reasonable for a diagnostic command. The bound is count-only — see WRK-21.

Watchdog / long COM calls. The heartbeat loop and ReportWatchdogFaultIfNeededAsync (Ipc/WorkerPipeSession.cs:790-881) are unchanged in structure: suppression while CurrentCommandCorrelationId is non-empty up to HeartbeatStuckCeiling (75 s default, Ipc/WorkerPipeSessionOptions.cs:19), and WRK-01's pump refresh now keeps pumping commands fresh indefinitely. Exit-code/fault mapping unchanged (WRK-17/18 still apply).

New findings

WRK-21 — Medium (stability). The DrainEvents bound is count-based, so an oversized reply still kills the session — and now also loses the drained events. Evidence: Ipc/WorkerPipeSession.cs:545-557 clamps only the event count; the reply is written via WriteControlReplyAsync (:481, :485-496) with no catch; a reply whose serialized size exceeds MaxMessageBytes throws WorkerFrameProtocolException(MessageTooLarge) from the writer (Ipc/WorkerFrameWriter.cs:250-255) — a "per-frame" rejection at the writer, but HandleControlCommandAsync/DispatchGatewayEnvelopeAsync propagate it out of RunMessageLoopAsync, so the session dies and WorkerApplication.cs:110-119 exits ProtocolViolation (6). The 10,000 drained events were already removed from MxAccessEventQueue by runtimeSession.DrainEvents(maxEvents) (:551) and are gone. 10,000 events at ~1.7 KiB each (large string/array MxValues) exceeds the default 16 MiB + 64 KiB frame max, so the comment's stated goal ("cannot pack the whole queue into one session-killing reply frame", :21-26) is not met for large events. Recommendation: bound by bytes as well as count (stop adding events once the reply's CalculateSize() approaches MaxMessageBytes minus headroom), or catch MessageTooLarge on control-reply writes and reply with a ResourceExhausted-style error for that correlation instead of unwinding the session. The same catch would also stop an oversized STA command reply from faulting the whole session at Ipc/WorkerPipeSession.cs:640-655.

WRK-22 — Low (stability). A cancelled WriteAsync leaves its frame queued, and the frame is still written later by an unrelated drain. Evidence: the frame is enqueued before the cancellable lock wait (Ipc/WorkerFrameWriter.cs:87-103); if _writeLock.WaitAsync(cancellationToken) throws, nothing removes the frame, so the next lock-holder writes it. Concrete scenario: at shutdown the event-drain loop's token is cancelled (Ipc/WorkerPipeSession.cs:298-304) while its event frame waits for the lock; the shutdown-ack write then drains queues — ack first (Control), then the orphaned event — so an event frame is emitted after WorkerShutdownAck, and a write reported as cancelled to its caller still reaches the wire. Recommendation: on cancellation, remove the frame from its queue under _gate (mark the PendingFrame cancelled and have DequeueNext skip completed frames), or document that cancellation only abandons the wait, never the write.

WRK-23 — Low (conventions/diagnostics). Rejected frames consume sequence numbers, producing wire gaps. Evidence: Ipc/WorkerFrameWriter.cs:236-255 stamps ++_nextSequence (:240) before the empty-payload and MessageTooLarge checks (:243-255), so a per-frame rejection leaves a hole in the on-wire sequence. gateway.md:328 calls sequence a monotonic diagnostic counter, so nothing breaks, but the new test (WorkerFrameProtocolTests.WriteAsync_UnderConcurrentCalls_StampsGapFreeMonotonicSequence) asserts a gap-free stream, documenting a stronger expectation than the code guarantees once a rejection occurs. Recommendation: run the size/empty checks before stamping (move the stamp to just before WriteTo), keeping sequences contiguous.

WRK-24 — Low (stability/hardening). AdoptNegotiatedMaxMessageBytes has no lower-bound sanity check. Evidence: Ipc/WorkerFrameProtocolOptions.cs:124-140 — 0 keeps the default and >256 MiB throws, but any value from 1 byte up is adopted; a gateway bug or a foreign/old gateway sending e.g. 512 would leave the session alive but unable to write any frame (every write fails MessageTooLarge) and unable to read most inbound frames, failing confusingly mid-session rather than at the handshake. The gateway's own validator floors its config at 1024 (src/ZB.MOM.WW.MxGateway.Server/Configuration/GatewayOptionsValidator.cs:10,230-233), but the worker's declared intent (:14-18, "a nonsensical negotiated value is rejected at the handshake") is only enforced at the top end. Recommendation: reject negotiated values below a sane floor (e.g. 64 KiB, or at least the gateway's 1024 minimum) with the same InvalidConfiguration error.

WRK-25 — Low (performance). The WRK-12 flush coalescing never engages on the event hot path it targeted. Evidence: the event drain loop awaits each event's WriteAsync — which completes only after that frame is written and flushed — before writing the next (Ipc/WorkerPipeSession.cs:374-382), so the writer's event queue holds at most one frame from the drain loop and every event still costs one flush. The batch path (Ipc/WorkerFrameWriter.cs:120-124, 160-182) only coalesces when independent producers happen to queue behind a blocked in-progress write (control frames, or a slow pipe) — the tracking-log claim "a burst of N events now costs 1 flush not N" does not describe production behavior. Recommendation: issue the drained batch's writes without awaiting each (collect the tasks and Task.WhenAll after the loop — the writer already guarantees per-queue FIFO order for a single producer), or add an explicit WriteBatchAsync; either makes the existing coalescing effective.

WRK-26 — Low (conventions/doc drift). The write-priority and overflow sections of the component design doc were not updated with the WRK-07 change. Evidence: docs/MxAccessWorkerInstanceDesign.md:607-613 still specifies the five-level priority (faults > replies > shutdown acks > heartbeats > events) while the implementation is a two-class scheduler (Control FIFO, then Event — Ipc/WorkerFrameWritePriority.cs:10-17); within Control a fault does not jump queued heartbeats/replies. docs/WorkerFrameProtocol.md describes the negotiated max (:25-30) but says nothing about priority classes, flush coalescing, or the per-frame vs stream-failure semantics. CLAUDE.md requires affected docs to change in the same commit as the source. Recommendation: rewrite docs/MxAccessWorkerInstanceDesign.md:607-613 to describe the implemented two-class scheduler (and note the accepted FIFO-within-control decision), and add a short "write scheduling" paragraph to docs/WorkerFrameProtocol.md. Fold in the stale overflow-policy text (WRK-18's doc half) while there.

WRK-27 — Low (stability). The alarm poll runs outside the dispatcher, so it never gets the watchdog's in-flight-command suppression. Evidence: MxAccess/MxAccessStaSession.cs:245-253 invokes handler.PollOnce() via staRuntime.InvokeAsync directly, bypassing StaCommandDispatcher; CaptureHeartbeat takes CurrentCommandCorrelationId only from the dispatcher (MxAccess/MxAccessStaSession.cs:364-381), so during a long PollOnce the heartbeat shows no in-flight command and the watchdog fires StaHung as soon as staleness exceeds HeartbeatGrace (15 s default) rather than the 75 s ceiling granted to dispatched commands (Ipc/WorkerPipeSession.cs:843-861). A healthy-but-slow GetXmlCurrentAlarms2 (large alarm sets, busy provider) blocking the STA >15 s faults a healthy session. Partially defensible — a blocked STA can't deliver data events either — but the 15 s vs 75 s asymmetry between polls and commands is unintentional. Recommendation: surface poll-in-progress to the heartbeat snapshot (e.g. a synthetic correlation id or a boolean the suppression branch also honors), or run the poll through the dispatcher.

WRK-28 — Low (conventions). The 10,000 drain cap is a duplicated magic constant with a comment-only synchronization contract. Evidence: Ipc/WorkerPipeSession.cs:26 (MaxDrainEventsPerReply = 10_000, "Kept in step with that public ceiling") vs src/ZB.MOM.WW.MxGateway.Server/Grpc/MxAccessGrpcRequestValidator.cs:13 (MaxDrainEventsPerRequest = 10_000). Both projects already share the Contracts assembly (net10.0;net48), which could own the constant. Recommendation: move the ceiling into ZB.MOM.WW.MxGateway.Contracts (e.g. next to GatewayContractInfo) and reference it from both sides.

Severity roll-up (new findings)

Severity	Count	IDs
High	0	—
Medium	1	WRK-21
Low	7	WRK-22, WRK-23, WRK-24, WRK-25, WRK-26, WRK-27, WRK-28

Still good / preserved

The strengths recorded in the 2026-07-08 review are intact at HEAD: the STA core still runs the canonical MsgWaitForMultipleObjectsEx + PeekMessage/DispatchMessage loop with the wake event and 50 ms idle pump (Sta/StaRuntime.cs:255-261, Sta/StaMessagePump.cs); COM lifetime discipline (FinalReleaseComObject, sink-detach-before-release, UnAdvise → RemoveItem → Unregister order) is unchanged in MxAccess/MxAccessSession.cs and both alarm consumers (the only alarm-consumer diffs since baseline are XML-doc comment removals — commit b86c6bb); partial pipe reads, hello validation, dispatcher backpressure (128-entry bound), record-after-COM-success handle bookkeeping, and value-cache eviction on RemoveItem all match the prior positive observations; nothing synthesizes events (MxAccess/MxAccessBaseEventSink.cs:160-171); nonce/credential redaction still works and is applied by the console logger (Bootstrap/WorkerLogRedactor.cs:18, Bootstrap/WorkerConsoleLogger.cs:36). The new writer preserves the "written and flushed before the caller's task completes" contract, the ownership-transfer refactor is properly invariant-documented on the queue class, and the new tests (sequence monotonicity, control-before-event, flush-once, negotiated-max bounds, drain bound, pump refresh, long-in-flight no-fault, ownership/snapshot independence) close the WRK-01/04/06/07/11/12 slices of the WRK-20 gap. net48 constraints are respected throughout the new code (plain ctors, readonly struct, no init-only members).

Verification needed on Windows (windev)

• This review is static: the x86 worker was not built; Worker.Tests were not run. The tracking log records windev-green runs for the P0/P1/P2 batches (352–356 passed), but any fix for WRK-21..28 needs an x86 build + filtered WorkerFrameProtocolTests/WorkerPipeSessionTests run on windev.
• WRK-21 deserves a windev repro test: enqueue ~10,000 large events, issue DrainEvents max_events=0, assert the session survives (currently expected to die with exit code 6).
• WRK-22's post-ack event emission is best confirmed against the real gateway WorkerClient (does it tolerate/ignore frames after WorkerShutdownAck?) — gateway side is outside this domain's scope.