mxaccessgw/archreview/remediation/20-worker.md

Worker Process — Remediation Design & Implementation

Source review: 20-worker.md · Generated: 2026-07-09

The worker's STA pump and COM-teardown discipline are correct; the risk is concentrated at the IPC edges — a long ReadBulk that self-faults, silent thread/command deaths, and a wire-sequence race — plus a small event hot-path allocation cluster. Every fix below stays on the STA for COM work and respects net48 constraints (no init-only properties, no positional records). The worker builds and tests only on the Windows x86 host, so verification commands assume that host.

Finding index

ID	Sev	Title	Roadmap	Effort	Depends on	Files
WRK-01	High	Long ReadBulk self-faults as StaHung; all replies then dropped	P0	M	—	MxAccess/MxAccessSession.cs, Sta/StaRuntime.cs, Ipc/WorkerPipeSession.cs
WRK-02	Medium	STA thread death after startup is silent; future work hangs forever	—	M	—	Sta/StaRuntime.cs
WRK-03	Medium	Commands after shutdown starts are dropped with no reply	—	S	WRK-02	Ipc/WorkerPipeSession.cs
WRK-04	Medium	Envelope sequence can appear out of order on the wire	—	M	—	Ipc/WorkerFrameWriter.cs, Ipc/WorkerPipeSession.cs
WRK-05	Medium	One transient alarm-poll failure kills the whole session	—	M	—	MxAccess/MxAccessStaSession.cs, MxAccess/AlarmCommandHandler.cs
WRK-06	Medium	MXSTATUS_PROXY conversion reflects per field, per event	P2	S	—	Conversion/MxStatusProxyConverter.cs
WRK-07	Medium	Documented outbound write priority not implemented	P1	M	WRK-04	Ipc/WorkerFrameWriter.cs, Ipc/WorkerPipeSession.cs
WRK-08	Low	Residual event queue discarded at graceful shutdown, undocumented	—	S	—	Ipc/WorkerPipeSession.cs, docs/MxAccessWorkerInstanceDesign.md
WRK-09	Low	No AppDomain.UnhandledException hook	—	S	—	WorkerApplication.cs, Program.cs
WRK-10	Low	Top-level catch logs exception type but never message	—	S	—	WorkerApplication.cs, Conversion/HResultConverter.cs
WRK-11	Low	Every accepted event is defensively cloned on enqueue	P2	S	—	MxAccess/MxAccessEventQueue.cs
WRK-12	Low	No event batching per envelope; one flush per event; 25 ms poll	P2	M	WRK-04	Ipc/WorkerPipeSession.cs, Ipc/WorkerFrameWriter.cs
WRK-13	Low	Frame writer allocates a fresh buffer per frame; reader pools	P2	S	—	Ipc/WorkerFrameWriter.cs
WRK-14	Low	Private-field naming split _camelCase vs camelCase	—	M	—	Ipc/, Bootstrap/, Sta/, MxAccess/
WRK-15	Low	Doc drift: STA thread name and heartbeat-counter note	P2	S	—	docs/WorkerSta.md, docs/MxAccessWorkerInstanceDesign.md
WRK-16	Low	Boilerplate duplication in IPC envelope/ctor overloads	—	S	—	Ipc/WorkerPipeSession.cs, Ipc/WorkerPipeClient.cs
WRK-17	Low	Gateway death exits with wrong exit code (6 not 5)	—	S	—	WorkerApplication.cs
WRK-18	Low	Event-queue overflow exits as generic UnexpectedFailure	—	S	—	Ipc/WorkerPipeSession.cs, WorkerApplication.cs, docs/MxAccessWorkerInstanceDesign.md
WRK-19	Low	Command start/end logging with correlation id absent	—	S	—	Sta/StaCommandDispatcher.cs
WRK-20	Low	Test-coverage gaps for the failure modes above	—	M	WRK-01..04	Worker.Tests/Ipc, Worker.Tests/Sta

---

WRK-01 — Long ReadBulk self-faults as StaHung; all replies then dropped High · P0

Finding. ReadOneTag reads uncached tags one at a time, waiting up to timeout per tag via valueCache.TryWaitForUpdate(..., pumpStep, ...) (MxAccess/MxAccessSession.cs:876-889, 918-931). The pumpStep handed down is StaRuntime.PumpPendingMessages(), which pumps Windows messages but never refreshes the activity timestamp (Sta/StaRuntime.cs:83-90; MarkActivity is private, Sta/StaRuntime.cs:304-307). The watchdog suppresses StaHung only while staleFor <= HeartbeatStuckCeiling (default 75 s); past the ceiling it faults even with a command in flight (Ipc/WorkerPipeSession.cs:830-860, Ipc/WorkerPipeSessionOptions.cs). Once faulted, every completed reply is dropped at the _state != Ready gate (Ipc/WorkerPipeSession.cs:604-607).

Impact. A legitimate ReadBulk with timeout_ms=5000 over ~20 unreachable tags holds the STA ~100 s with LastActivityUtc frozen. At 75 s the worker emits StaHung, sets _state = Faulted, and thereafter silently drops every command reply, so the gateway kills a healthy session. This is the one path where a healthy worker declares itself hung. Also called out as P0 item 4 in the roadmap.

Design. The design doc's watchdog contract assumes "no legitimate STA command should run that long without periodically refreshing activity" (docs/MxAccessWorkerInstanceDesign.md:688-690) — but no refresh mechanism exists. The minimal, correct fix is to make the pump step used by long-running STA commands also refresh activity: have StaRuntime.PumpPendingMessages() call MarkActivity() after pumping. Because the pump step is invoked on every wait iteration inside TryWaitForUpdate while the STA legitimately holds the thread, activity stays fresh for the whole in-flight command; the moment the command stops pumping (a genuine hang) staleness accrues and the watchdog still fires correctly. This preserves the watchdog's purpose (detect a stuck STA) while removing the false positive (a busy STA).

Rejected alternatives: (a) clamping total ReadBulk duration below HeartbeatStuckCeiling — changes MXAccess parity (per-tag timeout is the contract) and still breaks for large tag counts; (b) exposing a public activity-refresh hook threaded through every executor — larger surface, easy to forget on new commands. Refreshing inside the shared pumpStep covers all current and future long-running STA commands in one place. Parity is unaffected: no MXAccess behavior changes, only the local liveness signal.

Implementation.

• Sta/StaRuntime.cs: in PumpPendingMessages(), capture the pump count, call MarkActivity(), return the count. (MarkActivity is already private on the same class — no visibility change.)
• Confirm every long-hold command routes its wait through this method: ReadBulk does (MxAccessSession.cs:930 passes pumpStep, wired from StaRuntime.PumpPendingMessages at the executor). No new config.
• Optional defense-in-depth: document in docs/GatewayConfiguration.md that operators running very long bulk ops may raise HeartbeatStuckCeiling (already the doc's stated escape hatch, docs/MxAccessWorkerInstanceDesign.md:689-690).
• Tests: add StaRuntimeTests.PumpPendingMessages_RefreshesLastActivity, and an integration-style test in Worker.Tests/Ipc/WorkerPipeSessionTests using the fake runtime to prove a >75 s simulated in-flight command that keeps pumping does not emit StaHung and its reply is delivered.
• Docs: adjust docs/MxAccessWorkerInstanceDesign.md:688-690 to state that the pump step refreshes activity, so the "no legitimate command runs that long" caveat is now enforced mechanically rather than assumed.

Verification. dotnet build src/ZB.MOM.WW.MxGateway.Worker/ZB.MOM.WW.MxGateway.Worker.csproj -p:Platform=x86 then dotnet test src/ZB.MOM.WW.MxGateway.Worker.Tests/ZB.MOM.WW.MxGateway.Worker.Tests.csproj -p:Platform=x86 --filter "FullyQualifiedName~StaRuntimeTests|FullyQualifiedName~WorkerPipeSessionTests".

---

WRK-02 — STA thread death after startup is silent; future work hangs forever Medium · —

Finding. ThreadMain catches any loop exception into the write-only startupException field and sets startedEvent (Sta/StaRuntime.cs:255-259). After Start() has returned, nothing observes that field: the exception is never logged, never turned into a WorkerFault, and shutdownRequested stays false, so InvokeAsync keeps enqueuing work into a queue with no consumer and returns tasks that never complete (Sta/StaRuntime.cs:165-179). The only in-loop throw site is a pump-wait failure (Sta/StaMessagePump.cs:38-42).

Impact. If MsgWaitForMultipleObjectsEx returns WAIT_FAILED once, the STA exits; the dispatcher's drain wedges on the first stuck InvokeAsync; heartbeats keep flowing with a frozen LastStaActivityUtc and a pinned CurrentCommandCorrelationId; the worker is only declared hung after the 75 s ceiling, and the true root cause is permanently lost. Matches cross-cutting theme 1 (silent failure modes) in 00-overall.md.

Design. Post-startup thread death must (1) fail all queued and future InvokeAsync calls deterministically and (2) surface the cause. Introduce a terminalException field set in the ThreadMain catch/finally, and a Faulted state distinct from graceful shutdown. In the finally, after CancelQueuedCommands(), transition to a terminal-faulted state so InvokeAsync returns Task.FromException (using the captured exception) for all subsequent calls, and CancelQueuedCommands completes already-queued items faulted rather than cancelled when the exit was abnormal. Add an optional Action<Exception>? onTerminalFault callback (constructor-injected, defaulted null) that MxAccessStaSession wires to record a WorkerFault on the event queue — reusing the existing fault→drain→IPC path (MxAccessStaSession.RecordFault, Ipc/WorkerPipeSession.cs:344-353) so the gateway sees a real fault frame instead of a slow watchdog timeout. Keep the callback optional to avoid disturbing the many unit tests that construct StaRuntime directly.

Rejected: polling IsRunning from the dispatcher — racy and still loses the exception. A push callback plus terminal task-completion is deterministic.

Implementation.

• Sta/StaRuntime.cs: add private Exception? terminalException; and a volatile bool terminated; (net48 — plain field, no init-only). In ThreadMain's catch, store the exception; in finally, set terminated = true before stoppedEvent.Set(). In InvokeAsync<T>, after the shutdownRequested check, if terminated return Task.FromException<T>(terminalException ?? new StaRuntimeShutdownException()). Make CancelQueuedCommands fault (not cancel) items when terminalException is not null.
• Add constructor param Action<Exception>? onTerminalFault = null; invoke it once from finally when terminalException is not null.
• MxAccess/MxAccessStaSession.cs: pass a callback that calls eventQueue.RecordFault(...) with category StaHung (or a new StaTerminated if the contract enum allows — otherwise reuse StaHung and set a descriptive DiagnosticMessage).
• Tests: Worker.Tests/Sta/StaRuntimeTests — inject a message pump stub that throws on WaitForWorkOrMessages; assert the terminal callback fires, InvokeAsync after death returns a faulted task, and queued items fault.
• Docs: note in docs/WorkerSta.md that abnormal STA exit faults pending/future work and emits a fault frame.

Verification. dotnet build ...Worker.csproj -p:Platform=x86 then dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~StaRuntimeTests".

---

WRK-03 — Commands after shutdown starts are dropped with no reply Medium · —

Finding. TryStartCommandTask returns silently when _acceptingCommands is false (Ipc/WorkerPipeSession.cs:690-707): no WorkerUnavailable reply, not even the LogCommandResultDropped diagnostic (which fires only for completed-then-dropped replies, Ipc/WorkerPipeSession.cs:604-607, 645-655). The dispatcher layer does reply WorkerUnavailable when it is the one shutting down (Sta/StaCommandDispatcher.cs:117-123), but this earlier gate short-circuits before reaching it.

Impact. A command racing WorkerShutdown leaves the gateway's correlation wait to expire on its own timeout with no trace, exactly the silent-drop pattern docs/MxAccessWorkerInstanceDesign.md:697-699 ("reject new commands") intends to avoid.

Design. Make the gate loud: when _acceptingCommands is false, write a WorkerCommandReply carrying ProtocolStatusCode.WorkerUnavailable for that correlation id (mirroring StaCommandDispatcher.CreateRejectedReply), then return. This matches the dispatcher-level rejection and gives the gateway an immediate, correlated failure. Guard the write with the same _state/TryWriteFault-style tolerance so a half-closed pipe during shutdown does not throw. Co-designed with WRK-02 (both close silent-death holes).

Implementation.

• Ipc/WorkerPipeSession.cs: in the !_acceptingCommands branch of TryStartCommandTask, build a rejection reply (reuse a small helper CreateRejectedCommandReply(correlationId, method, WorkerUnavailable, "Worker is shutting down.")) and enqueue a best-effort _writer.WriteAsync(CreateEnvelope(reply), ...) via a fire-and-forget observed task (same pattern as ObserveCommandTaskAsync). At minimum, log a WorkerCommandRefusedDuringShutdown diagnostic even if the write is skipped.
• Tests: Worker.Tests/Ipc/WorkerPipeSessionTests — after triggering shutdown, feed a command envelope and assert a WorkerUnavailable reply (or the diagnostic) is observed on the fake writer; covers the WRK-20 gap for S3.
• Docs: none beyond the existing shutdown sequence, which already promises rejection.

Verification. dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerPipeSessionTests".

---

WRK-04 — Envelope sequence can appear out of order on the wire Medium · —

Finding. NextSequence() is called while building the envelope in CreateBaseEnvelope() (Ipc/WorkerPipeSession.cs:1005-1018), but the write lock is acquired later, inside WorkerFrameWriter.WriteAsync (Ipc/WorkerFrameWriter.cs:68-77). Command replies are written from independent per-command tasks (Ipc/WorkerPipeSession.cs:690-706) concurrently with the heartbeat and event-drain loops, so task B can take sequence n+1 yet win the write lock before task A's sequence n. (Per-event ordering is safe — WorkerSequence is stamped inside the queue lock, MxAccess/MxAccessEventQueue.cs:135-143 — but the envelope-level guarantee is not.)

Impact. Violates the gateway.md envelope rule "sequence is monotonic per sender"; any gateway consumer trusting wire-order monotonicity mis-sorts frames.

Design. Assign the envelope sequence inside the writer's critical section so the number and the write are atomic. Change WorkerFrameWriter.WriteAsync to accept a sequence-stamping callback (Action<WorkerEnvelope> or a Func<ulong> that the writer invokes under _writeLock to set envelope.Sequence) rather than reading a pre-stamped value. CreateBaseEnvelope stops calling NextSequence(); the writer stamps Sequence immediately before serialization, under the lock. This keeps a single _nextSequence counter and removes the window entirely. Co-designed with WRK-07 (both restructure the write path) — sequence stamping and priority scheduling should land together to avoid two passes over the writer.

Rejected: making NextSequence Interlocked (already is) does not help — the reordering is between assignment and write, not in the increment. The atomic region must span both.

Implementation.

• Ipc/WorkerFrameWriter.cs: add an overload WriteAsync(WorkerEnvelope envelope, Func<ulong> sequenceProvider, CancellationToken) that, after acquiring _writeLock, sets envelope.Sequence = sequenceProvider() before Validate/CalculateSize/WriteTo. (Serialize inside the lock now, since the payload depends on the stamped sequence.)
• Ipc/WorkerPipeSession.cs: CreateBaseEnvelope() no longer sets Sequence; every _writer.WriteAsync(CreateEnvelope(...), ...) call site passes NextSequence as the provider. Keep NextSequence() on the session (single owner of _nextSequence).
• Tests: Worker.Tests/Ipc/WorkerPipeSessionTests — a concurrent-writer test that fires N reply writes and M event writes in parallel through a capturing stream and asserts observed Sequence values are strictly monotonic in wire order (covers WRK-20 gap for S4). A WorkerFrameWriterTests unit test that the provider runs under the lock.
• Docs: none — restores the documented invariant.

Verification. dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerFrameWriterTests|FullyQualifiedName~WorkerPipeSessionTests".

---

WRK-05 — One transient alarm-poll failure kills the whole session Medium · —

Finding. Any exception from the alarm poll loop is recorded as a fault on the shared event queue and permanently stops the loop (MxAccess/MxAccessStaSession.cs:278-291); the drain loop turns that fault into full session termination (Ipc/WorkerPipeSession.cs:344-353). Failover only absorbs primary failures in composite mode (MxAccess/FailoverAlarmConsumer.cs); the default WorkerPipeSession builds the alarm handler with standbyFactory: null (Ipc/WorkerPipeSession.cs:54), so AlarmCommandHandler.BuildConsumer returns a bare consumer and a single GetXmlCurrentAlarms2 COM error propagates unwrapped.

Impact. One transient E_FAIL from the AVEVA alarm subsystem terminates a client's healthy OnDataChange data stream even though the data path never failed — a whole-session death from an isolated alarm blip.

Design. Two viable scopes; recommend the smaller, parity-safe one:

1. Recommended — bounded retry before declaring the alarm subscription dead. Count consecutive poll failures against a threshold (mirror FailoverSettings.Threshold, default e.g. 3) with a short back-off; reset the counter on any successful poll. Only after the threshold record the fault. This tolerates transient COM errors without touching the data path and matches the existing failover semantics.
2. Scope the fault to the alarm feature (stop alarm delivery, keep data subscriptions and the session alive). Larger change — requires a per-feature fault channel the drain loop does not currently model, and risks masking a genuinely dead provider. Defer.

Parity note: this does not synthesize or suppress alarm events — it only changes how many consecutive poll infrastructure failures constitute "the subscription is dead." A persistent failure still faults the session, preserving fail-fast.

Implementation.

• MxAccess/MxAccessStaSession.cs: in the poll loop, maintain int consecutiveFailures; on catch, increment and only call eventQueue.RecordFault(...) + return when consecutiveFailures >= threshold; otherwise log a warning, await Task.Delay(backoff), and continue. Reset on success. Keep the STA-affinity InvalidOperationException (from EnsureOnAlarmConsumerThread) as an immediate fault (it is a programming-error regression, not transient) — distinguish it before the counting branch.
• Config: add AlarmPollFailureThreshold and AlarmPollBackoff to the alarm handler options (thread through AlarmCommandHandler); default threshold to match FailoverSettings.Threshold.
• Tests: Worker.Tests alarm units — a consumer stub that throws N-1 times then succeeds keeps the session alive; N consecutive throws faults it; an affinity InvalidOperationException faults immediately.
• Docs: document the threshold in docs/GatewayConfiguration.md and note the transient-tolerance behavior in docs/MxAccessWorkerInstanceDesign.md alarm section.

Verification. dotnet build ...Worker.csproj -p:Platform=x86 then dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~Alarm".

---

WRK-06 — MXSTATUS_PROXY conversion reflects per field, per event Medium · P2

Finding. MxStatusProxyConverter.Convert calls ReadInt32Field four times per status (Conversion/MxStatusProxyConverter.cs:22-26); each does Type.GetField + FieldInfo.GetValue + Convert.ToInt32 (:83-103). This runs on the STA event path for every status of every OnDataChange.

Impact. Eight reflection ops plus boxing per event under data-change/alarm bursts — the exact load the gateway exists to handle. The status type is always the interop MXSTATUS_PROXY struct, so the field lookups are fully cacheable. Roadmap P2 item 14 (event hot-path pass).

Design. Cache the four FieldInfo objects keyed by Type in a small static ConcurrentDictionary<Type, (FieldInfo success, category, detectedBy, detail)>, resolved once per type and reused. GetValue + Convert.ToInt32 still run per event (unavoidable via reflection over a late-bound COM RCW), but the GetField metadata scan — the expensive part — is eliminated. A direct cast to the interop struct type is faster still but couples the converter to the interop assembly and its exact struct shape; the review notes the type is stable, but the cached-FieldInfo approach keeps the converter interop-agnostic and testable with the existing plain-CLR test doubles. Recommend cached FieldInfo.

Implementation.

• Conversion/MxStatusProxyConverter.cs: add a static cache; a private GetFields(Type) that populates it once (throwing the same MxStatusConversionException if a field is missing) and Convert reads the four cached FieldInfos. Behavior and exceptions unchanged.
• Tests: existing MxStatusProxyConverter tests must still pass (same outputs); add one asserting two conversions of the same type reuse cached metadata (e.g. via a type whose GetField is instrumented, or simply a throughput/regression test).
• Docs: none (internal perf).

Verification. dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~MxStatusProxyConverter".

---

WRK-07 — Documented outbound write priority not implemented Medium · P1

Finding. docs/MxAccessWorkerInstanceDesign.md:600-613 specifies write priority faults > command replies > shutdown acks > heartbeats > events; the worker has no prioritized queue — all writers contend on a single FIFO SemaphoreSlim in WorkerFrameWriter (Ipc/WorkerFrameWriter.cs:14, 68-77), and events are written inline by the drain loop (Ipc/WorkerPipeSession.cs:362-367).

Impact. With a deep event backlog draining (up to 128 per batch), a WorkerFault or command reply queues behind those event writes; on a slow pipe this delays the gateway's fault reaction. Relates to roadmap P1 item 8 (backpressure/size topology).

Design. Two acceptable outcomes; the review explicitly allows either. Recommend the documented-decision path first, with a small scheduler as the follow-up:

1. Minimal now: amend the design doc to state that FIFO write ordering was accepted for v1 (the single _writeLock serializes but does not prioritize), and that event batching (WRK-12) plus the sequence fix (WRK-04) bound the worst-case delay. This removes the doc/code contradiction immediately.
2. Full fix (recommended for P1): introduce a priority write scheduler in WorkerFrameWriter — a small set of per-priority queues drained newest-priority-first under _writeLock, or a Channel-per-priority merged by a single writer task. Faults and replies jump ahead of queued events. Must preserve per-sender sequence monotonicity — co-designed with WRK-04, since sequence is now stamped inside the lock at actual write time, priority reordering before stamping keeps sequence consistent with wire order automatically.

The full scheduler is the correct end state; if effort is constrained, ship option 1 in the same commit that lands WRK-04/WRK-12 and file option 2.

Implementation.

• Ipc/WorkerFrameWriter.cs: add an internal priority-ordered pending set drained by the lock holder; or expose WriteAsync(envelope, priority, sequenceProvider, ct). Ipc/WorkerPipeSession.cs tags each write site with a priority (fault=0 … event=4).
• Tests: Worker.Tests/Ipc/WorkerFrameWriterTests — enqueue a fault behind many event writes on a blocked stream, unblock, assert the fault frame emerges first while sequences remain monotonic.
• Docs: update docs/MxAccessWorkerInstanceDesign.md:600-613 to describe the implemented scheduler (or the accepted-FIFO decision if option 1).

Verification. dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerFrameWriterTests|FullyQualifiedName~WorkerPipeSessionTests".

---

WRK-08 — Residual event queue discarded at graceful shutdown, undocumented Low · —

Finding. ShutdownAsync writes the ack and returns; RunMessageLoopAsync's finally cancels the event-drain loop (Ipc/WorkerPipeSession.cs:287-292) with whatever remains in MxAccessEventQueue unshipped; late replies on _state != Ready are dropped (Ipc/WorkerPipeSession.cs:604-607). An OnWriteComplete raised during cleanup never reaches the gateway. Acceptable for a closing session but not stated in docs/MxAccessWorkerInstanceDesign.md.

Design. Cheapest correct action: document the discard as intended v1 behavior — a session tearing down does not guarantee delivery of events queued after WorkerShutdown. Optionally (if a downstream wants last-gasp events) drain the queue once after ShutdownGracefullyAsync returns and before writing the ack, bounded by the grace period. Recommend documenting now; the final drain is a small enhancement to schedule only if a client needs it.

Implementation. Add a paragraph to docs/MxAccessWorkerInstanceDesign.md shutdown section. If implementing the drain: in ShutdownAsync, after ShutdownGracefullyAsync, call DrainEvents once and write each before the ack. Test: WorkerPipeSessionTests asserting either the documented discard or the final-drain delivery.

Verification. dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerPipeSessionTests".

---

WRK-09 — No AppDomain.UnhandledException hook Low · —

Finding. Program.cs:1-4 and WorkerApplication.Run (WorkerApplication.cs:46-141) install no unhandled-exception or unobserved-task handlers (confirmed: no AppDomain/UnhandledException reference exists anywhere in the worker). An exception on an unobserved thread crashes the process with no WorkerFault and no log.

Design. Register AppDomain.CurrentDomain.UnhandledException and TaskScheduler.UnobservedTaskException at process start, logging the (redacted) exception through IWorkerLogger before exit. The gateway still detects death via process exit + pipe closure, so this is diagnostics-only; keep it minimal.

Implementation. In WorkerApplication.Run (or a tiny bootstrap in Program.cs), before parsing args, subscribe both handlers and log WorkerUnhandledException / WorkerUnobservedTaskException with WorkerLogRedactor-scrubbed message + type. Test: unit test the handler delegate logs and does not throw. Docs: mention in docs/MxAccessWorkerInstanceDesign.md logging list.

Verification. dotnet build ...Worker.csproj -p:Platform=x86; dotnet test ...Worker.Tests... -p:Platform=x86.

---

WRK-10 — Top-level catch logs exception type but never message Low · —

Finding. WorkerApplication.cs:112-139 logs only exception_type for protocol, pipe, and unexpected failures; HResultConverter.CreateSafeDiagnosticMessage reduces every command exception to Type: HRESULT 0x… (Conversion/HResultConverter.cs:46-49).

Design. Log exception.Message (routed through Bootstrap/WorkerLogRedactor — which already scrubs nonces/credentials, WorkerLogRedactor.cs:16-25) alongside the type at the process boundary. Keep the credential-safe reply shape for IPC replies unchanged if the HRESULT-only stripping is intentional parity/secret policy — the fix is scoped to worker stderr/log, not the wire reply.

Implementation. In the three WorkerApplication.Run catch blocks, add ["exception_message"] = WorkerLogRedactor.Redact(exception.Message). Leave HResultConverter reply text as-is (document the intent in a code comment). Test: WorkerApplicationTests (or add) asserting the redacted message is present. Docs: none.

Verification. dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerApplication".

---

WRK-11 — Every accepted event is defensively cloned on enqueue Low · P2

Finding. MxAccessEventQueue.Enqueue calls mxEvent.Clone() (MxAccess/MxAccessEventQueue.cs:135) for an event the mapper built exclusively for this call (MxAccess/MxAccessBaseEventSink.cs:210-256); only the value-cache post-publish shares the original.

Design. Take ownership of the passed event in the queue (stamp WorkerSequence/WorkerTimestamp on it directly) and let the value cache store the copy — the cache already snapshots only value/quality/timestamp/statuses (MxAccess/MxAccessValueCache.cs:44-57), so it does not need the full MxEvent alias. This halves protobuf allocation on the hottest path. Requires confirming no caller reuses the passed MxEvent after enqueue (the mapper builds a fresh one per event — safe). Roadmap P2 item 14.

Implementation. Remove the Clone(); mutate the incoming mxEvent in place under the queue lock. Audit MxAccessBaseEventSink call sites to confirm single-ownership. Tests: existing event-queue tests must still pass; add one asserting the enqueued instance is the same reference passed in and that the value cache's stored copy is independent. Docs: none.

Verification. dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~MxAccessEventQueue|FullyQualifiedName~EventSink".

---

WRK-12 — No event batching per envelope; one flush per event; 25 ms poll Low · P2

Finding. The drain loop writes one WriteAsync per event (Ipc/WorkerPipeSession.cs:362-367), the writer flushes per frame (Ipc/WorkerFrameWriter.cs:71-72), and the poll interval is 25 ms (Ipc/WorkerPipeSession.cs:17). Each event costs a semaphore round-trip, a pipe write, and a flush; idle-to-active latency up to 25 ms. gateway.md lists event batching as the intended optimization.

Design. Acceptable for v1 parity (the review agrees). When throughput matters: either (a) add a repeated-event WorkerEnvelope body (a contracts/proto change — coordinate with domain 30 IPC, cross-references IPC findings; must preserve per-event order and sequence semantics), or (b) keep one event per envelope but coalesce flushes across a drained batch (flush once after writing the batch), which needs no proto change and captures most of the benefit. Recommend (b) now, (a) as a coordinated cross-domain change. Depends on WRK-04 (sequence stamping under the lock) so a coalesced batch keeps monotonic sequences.

Implementation. (Option b) WorkerFrameWriter: add a WriteBatchAsync(IEnumerable<WorkerEnvelope>, ...) that writes all frames then flushes once; drain loop calls it per drained batch. Tests: WorkerFrameWriterTests asserting one flush per batch and correct framing. Docs: note the flush-coalescing in docs/WorkerFrameProtocol.md; if option (a), update .proto and regenerate per CLAUDE.md contracts rule.

Verification. dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerFrameWriterTests".

---

WRK-13 — Frame writer allocates a fresh buffer per frame; reader pools Low · P2

Finding. WorkerFrameWriter does new byte[frameLength] per frame (Ipc/WorkerFrameWriter.cs:63-66) while WorkerFrameReader rents from ArrayPool<byte>.Shared (Ipc/WorkerFrameReader.cs:55-77).

Design. Rent the write buffer from ArrayPool<byte>.Shared for symmetry; return it in a finally after the write completes. Because ArrayPool may return an oversized buffer, pass explicit (0, frameLength) to WriteAsync (already does) and never leak the buffer's tail. Trivial, self-contained. Roadmap P2 item 14.

Implementation. WorkerFrameWriter.WriteAsync: byte[] frame = ArrayPool<byte>.Shared.Rent(frameLength); … try { … } finally { ArrayPool<byte>.Shared.Return(frame); }. Note the return must occur after the awaited write completes (it does — inside the same method). Tests: existing WorkerFrameWriterTests framing tests cover correctness; no behavior change. Docs: none.

Verification. dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerFrameWriterTests".

---

WRK-14 — Private-field naming split _camelCase vs camelCase Low · —

Finding. Ipc/ and Bootstrap/ use _camelCase (Ipc/WorkerPipeSession.cs:21-38, Ipc/WorkerFrameWriter.cs:13-15, Bootstrap/WorkerConsoleLogger.cs:10); Sta/ and MxAccess/ use bare camelCase (Sta/StaRuntime.cs:10-24, MxAccess/MxAccessStaSession.cs:16-27). docs/style-guides/CSharpStyleGuide.md:30-32 permits the underscore prefix "only when already established" — both are established, so the worker has no single convention.

Design. Pick one style project-wide and migrate opportunistically (not in one churn commit). The gateway server is the tie-breaker: adopt whatever the gateway uses so the whole solution converges. This is a mechanical rename with no behavior change; do it file-by-file as those files are touched for other findings to keep diffs reviewable.

Implementation. Decide the target (check src/ZB.MOM.WW.MxGateway.Server private-field style), record it in docs/style-guides/CSharpStyleGuide.md, and rename incrementally. Tests: build only. Docs: the style-guide note.

Verification. dotnet build ...Worker.csproj -p:Platform=x86 (analyzers/TreatWarningsAsErrors must stay green).

---

WRK-15 — Doc drift: STA thread name and heartbeat-counter note Low · P2

Finding. The STA thread is named "MxGateway.Worker.STA" (Sta/StaRuntime.cs:61) but docs/WorkerSta.md:23,30 and docs/MxAccessWorkerInstanceDesign.md:254 say ZB.MOM.WW.MxGateway.Worker.STA. And docs/MxAccessWorkerInstanceDesign.md:653-654 says event-queue depth and sequence "are reported as zero until the event queue implementation owns those counters," but CaptureHeartbeat now populates both from the live queue (MxAccess/MxAccessStaSession.cs:375-380).

Design. Docs must match source (CLAUDE.md rule). Choose: either rename the thread to the documented ZB.MOM.WW.MxGateway.Worker.STA (operators grep thread dumps for it) or update the docs to the actual name — recommend renaming the thread to the fully-qualified documented name for operability, in the same commit that fixes the heartbeat-counter note. Roadmap P2 item 15 (doc-drift sweep).

Implementation. Either edit Sta/StaRuntime.cs:61 thread Name to ZB.MOM.WW.MxGateway.Worker.STA, or edit the two docs. Fix docs/MxAccessWorkerInstanceDesign.md:653-654 to state the counters are live. Tests: if renaming, update any test asserting the thread name. Docs: as above.

Verification. dotnet build ...Worker.csproj -p:Platform=x86; dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~StaRuntime".

---

WRK-16 — Boilerplate duplication in IPC envelope/ctor overloads Low · —

Finding. Seven near-identical CreateEnvelope/CreateBaseEnvelope overload pairs (Ipc/WorkerPipeSession.cs:920-1003) and eight WorkerPipeClient constructor overloads (Ipc/WorkerPipeClient.cs:36-140). Maintenance noise — each new body means two more copy-paste methods.

Design. Collapse the envelope overloads to a single CreateEnvelope(Action<WorkerEnvelope> setBody) (or a switch on the body message type); keep the correlation-id special case for WorkerCommandReply inside the setter. Reduce the client constructors to one primary constructor with the rest chaining via defaulted parameters (net48 supports optional params — no init-only needed). Pure refactor, no behavior change. Fold into WRK-04/WRK-07 since those already rework CreateBaseEnvelope.

Implementation. Refactor WorkerPipeSession envelope factories and WorkerPipeClient ctors. Tests: existing IPC tests must pass unchanged; they are the regression guard. Docs: none.

Verification. dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerPipeSession|FullyQualifiedName~WorkerPipeClient".

---

WRK-17 — Gateway death exits with wrong exit code (6 not 5) Low · —

Finding. Pipe EOF surfaces as WorkerFrameProtocolException(EndOfStream) (Ipc/WorkerFrameReader.cs:104-109), which WorkerApplication.Run catches first and maps to ProtocolViolation (6) (WorkerApplication.cs:110-119) even though PipeConnectionFailed (5) exists (Bootstrap/WorkerExitCode.cs:10) and the in-session fault mapping already distinguishes pipe disconnect.

Design. Special-case WorkerFrameProtocolErrorCode.EndOfStream in the WorkerFrameProtocolException catch to return PipeConnectionFailed (5) — EOF means the gateway went away, not that the worker misbehaved. Improves orphan-worker post-mortem triage.

Implementation. In WorkerApplication.cs:110-119, branch on exception.ErrorCode == WorkerFrameProtocolErrorCode.EndOfStream → log + return PipeConnectionFailed; else ProtocolViolation. Tests: WorkerApplicationTests — inject a pipe client throwing EndOfStream and assert exit code 5. Docs: docs/WorkerFrameProtocol.md / exit-code table if one exists.

Verification. dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerApplication".

---

WRK-18 — Event-queue overflow exits as generic UnexpectedFailure Low · —

Finding. docs/MxAccessWorkerInstanceDesign.md:615-624 says overflow should "stop accepting new commands" and "let the gateway close or kill the worker"; the implementation instead terminates immediately — the drain loop writes the fault then throws (Ipc/WorkerPipeSession.cs:344-353), unwinding into the generic handler and exit code 1 (WorkerApplication.cs:131-139). The designed fault path is indistinguishable from a crash by exit code.

Design. The implemented fail-fast is arguably stronger than the doc and acceptable; the defect is observability. Give overflow a dedicated exit code (e.g. add EventQueueOverflow to WorkerExitCode) and catch the drain-fault termination in WorkerApplication to return it, then update the doc to describe the implemented immediate-terminate policy. Recommend keeping fail-fast (do not weaken to "stop accepting commands") and aligning the doc + exit code to it.

Implementation. Add WorkerExitCode.EventQueueOverflow; wrap the drain-fault throw in a typed exception (WorkerEventQueueOverflowTerminationException) so WorkerApplication.Run can map it. Tests: WorkerPipeSessionTests overflow path asserts the dedicated code. Docs: rewrite docs/MxAccessWorkerInstanceDesign.md:615-624 to match.

Verification. dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerPipeSessionTests|FullyQualifiedName~WorkerApplication".

---

WRK-19 — Command start/end logging with correlation id absent Low · —

Finding. docs/MxAccessWorkerInstanceDesign.md:790-791 lists "command start/end with correlation id" among required worker logs; the only per-command log is the dropped-reply diagnostic (Ipc/WorkerPipeSession.cs:645-655). StaCommand.EnqueueTimestamp is captured (Sta/StaCommand.cs) but never used for latency.

Design. Add optional, level-gated start/end logging in StaCommandDispatcher.ExecuteQueuedCommandAsync, which already brackets each command (Sta/StaCommandDispatcher.cs:265-281). Log correlation id + method at start; at end log outcome + latency (now - EnqueueTimestamp). Gate at a verbose/debug level so production noise is opt-in.

Implementation. Inject the optional IWorkerLogger into StaCommandDispatcher (or pass through the session); emit WorkerCommandStarted/WorkerCommandCompleted with correlation id, method, latency, outcome. Tests: StaCommandDispatcherTests asserting both logs fire with the correlation id. Docs: confirm the log names in docs/MxAccessWorkerInstanceDesign.md logging list.

Verification. dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~StaCommandDispatcher".

---

WRK-20 — Test-coverage gaps for the failure modes above Low · —

Finding. Worker.Tests covers the pump wake behavior, dispatcher ordering/cancellation/shutdown, handshake/heartbeat/watchdog (incl. the stuck ceiling), control commands, shutdown races, late-reply drops, frame protocol, conversion, event queue, and alarm units. Not covered: STA thread death mid-run (WRK-02), wire-level envelope sequence monotonicity under concurrent writers (WRK-04), the silent no-reply drop at the _acceptingCommands gate (WRK-03), and the ReadBulk-exceeds-ceiling false fault (WRK-01).

Design. Add the four tests alongside their fixes (named in each entry above) using the existing fake-runtime harness in WorkerPipeSessionTests, which already supports all four. This is not a separate work item so much as the acceptance criterion for WRK-01..04 — tracked here so it is not dropped.

Implementation. Tests to add: StaRuntimeTests.ThreadDeath_FaultsPendingAndFutureWork (WRK-02); WorkerPipeSessionTests.ConcurrentWriters_SequenceIsMonotonic (WRK-04); WorkerPipeSessionTests.CommandAfterShutdown_RepliesWorkerUnavailable (WRK-03); WorkerPipeSessionTests.LongReadBulk_DoesNotFaultWhilePumping + StaRuntimeTests.PumpPendingMessages_RefreshesLastActivity (WRK-01). Docs: none.

Verification. dotnet test src/ZB.MOM.WW.MxGateway.Worker.Tests/ZB.MOM.WW.MxGateway.Worker.Tests.csproj -p:Platform=x86 (full worker suite once, after the batch lands, per the targeted-tests-then-phase-suite rule).

---

Cross-references

• WRK-01 is roadmap P0 item 4; WRK-06/11/12/13 are the worker slice of P2 item 14 (event hot-path); WRK-15 is part of P2 item 15 (doc-drift sweep); WRK-07 relates to P1 item 8 (backpressure/size topology).
• WRK-04 and WRK-07 rework the write path together; WRK-16 folds into that refactor. WRK-12 depends on WRK-04's under-lock sequence stamping.
• WRK-02 and WRK-03 both close silent-failure edges (cross-cutting theme 1 in 00-overall.md); the same theme spans gateway and client findings — coordinate the "every death/drop is observable" pattern across domains.