mxaccessgw/archreview/remediation/20-worker.md

# Worker Process — Remediation Design & Implementation

Source review: [20-worker.md](../20-worker.md) · Generated: 2026-07-09

The worker's STA pump and COM-teardown discipline are correct; the risk is concentrated at the IPC edges — a long `ReadBulk` that self-faults, silent thread/command deaths, and a wire-sequence race — plus a small event hot-path allocation cluster. Every fix below stays on the STA for COM work and respects net48 constraints (no init-only properties, no positional records). The worker builds and tests only on the Windows x86 host, so verification commands assume that host.

## Finding index

| ID | Sev | Title | Roadmap | Effort | Depends on | Files |
|----|-----|-------|---------|--------|-----------|-------|
| WRK-01 | High | Long `ReadBulk` self-faults as `StaHung`; all replies then dropped | P0 | M | — | MxAccess/MxAccessSession.cs, Sta/StaRuntime.cs, Ipc/WorkerPipeSession.cs |
| WRK-02 | Medium | STA thread death after startup is silent; future work hangs forever | — | M | — | Sta/StaRuntime.cs |
| WRK-03 | Medium | Commands after shutdown starts are dropped with no reply | — | S | WRK-02 | Ipc/WorkerPipeSession.cs |
| WRK-04 | Medium | Envelope `sequence` can appear out of order on the wire | — | M | — | Ipc/WorkerFrameWriter.cs, Ipc/WorkerPipeSession.cs |
| WRK-05 | Medium | One transient alarm-poll failure kills the whole session | — | M | — | MxAccess/MxAccessStaSession.cs, MxAccess/AlarmCommandHandler.cs |
| WRK-06 | Medium | `MXSTATUS_PROXY` conversion reflects per field, per event | P2 | S | — | Conversion/MxStatusProxyConverter.cs |
| WRK-07 | Medium | Documented outbound write priority not implemented | P1 | M | WRK-04 | Ipc/WorkerFrameWriter.cs, Ipc/WorkerPipeSession.cs |
| WRK-08 | Low | Residual event queue discarded at graceful shutdown, undocumented | — | S | — | Ipc/WorkerPipeSession.cs, docs/MxAccessWorkerInstanceDesign.md |
| WRK-09 | Low | No `AppDomain.UnhandledException` hook | — | S | — | WorkerApplication.cs, Program.cs |
| WRK-10 | Low | Top-level catch logs exception type but never message | — | S | — | WorkerApplication.cs, Conversion/HResultConverter.cs |
| WRK-11 | Low | Every accepted event is defensively cloned on enqueue | P2 | S | — | MxAccess/MxAccessEventQueue.cs |
| WRK-12 | Low | No event batching per envelope; one flush per event; 25 ms poll | P2 | M | WRK-04 | Ipc/WorkerPipeSession.cs, Ipc/WorkerFrameWriter.cs |
| WRK-13 | Low | Frame writer allocates a fresh buffer per frame; reader pools | P2 | S | — | Ipc/WorkerFrameWriter.cs |
| WRK-14 | Low | Private-field naming split `_camelCase` vs `camelCase` | — | M | — | Ipc/*, Bootstrap/*, Sta/*, MxAccess/* |
| WRK-15 | Low | Doc drift: STA thread name and heartbeat-counter note | P2 | S | — | docs/WorkerSta.md, docs/MxAccessWorkerInstanceDesign.md |
| WRK-16 | Low | Boilerplate duplication in IPC envelope/ctor overloads | — | S | — | Ipc/WorkerPipeSession.cs, Ipc/WorkerPipeClient.cs |
| WRK-17 | Low | Gateway death exits with wrong exit code (6 not 5) | — | S | — | WorkerApplication.cs |
| WRK-18 | Low | Event-queue overflow exits as generic `UnexpectedFailure` | — | S | — | Ipc/WorkerPipeSession.cs, WorkerApplication.cs, docs/MxAccessWorkerInstanceDesign.md |
| WRK-19 | Low | Command start/end logging with correlation id absent | — | S | — | Sta/StaCommandDispatcher.cs |
| WRK-20 | Low | Test-coverage gaps for the failure modes above | — | M | WRK-01..04 | Worker.Tests/Ipc, Worker.Tests/Sta |

---

## WRK-01 — Long `ReadBulk` self-faults as `StaHung`; all replies then dropped   `High` · `P0`

**Finding.** `ReadOneTag` reads uncached tags one at a time, waiting up to `timeout` per tag via `valueCache.TryWaitForUpdate(..., pumpStep, ...)` (`MxAccess/MxAccessSession.cs:876-889`, `918-931`). The `pumpStep` handed down is `StaRuntime.PumpPendingMessages()`, which pumps Windows messages but never refreshes the activity timestamp (`Sta/StaRuntime.cs:83-90`; `MarkActivity` is private, `Sta/StaRuntime.cs:304-307`). The watchdog suppresses `StaHung` only while `staleFor <= HeartbeatStuckCeiling` (default 75 s); past the ceiling it faults even with a command in flight (`Ipc/WorkerPipeSession.cs:830-860`, `Ipc/WorkerPipeSessionOptions.cs`). Once faulted, every completed reply is dropped at the `_state != Ready` gate (`Ipc/WorkerPipeSession.cs:604-607`).

**Impact.** A legitimate `ReadBulk` with `timeout_ms=5000` over ~20 unreachable tags holds the STA ~100 s with `LastActivityUtc` frozen. At 75 s the worker emits `StaHung`, sets `_state = Faulted`, and thereafter silently drops every command reply, so the gateway kills a healthy session. This is the one path where a healthy worker declares itself hung. Also called out as P0 item 4 in the roadmap.

**Design.** The design doc's watchdog contract assumes "no legitimate STA command should run that long without periodically refreshing activity" (`docs/MxAccessWorkerInstanceDesign.md:688-690`) — but no refresh mechanism exists. The minimal, correct fix is to make the pump step used by long-running STA commands *also* refresh activity: have `StaRuntime.PumpPendingMessages()` call `MarkActivity()` after pumping. Because the pump step is invoked on every wait iteration inside `TryWaitForUpdate` while the STA legitimately holds the thread, activity stays fresh for the whole in-flight command; the moment the command stops pumping (a genuine hang) staleness accrues and the watchdog still fires correctly. This preserves the watchdog's purpose (detect a *stuck* STA) while removing the false positive (a *busy* STA).

Rejected alternatives: (a) clamping total `ReadBulk` duration below `HeartbeatStuckCeiling` — changes MXAccess parity (per-tag `timeout` is the contract) and still breaks for large tag counts; (b) exposing a public activity-refresh hook threaded through every executor — larger surface, easy to forget on new commands. Refreshing inside the shared `pumpStep` covers all current and future long-running STA commands in one place. Parity is unaffected: no MXAccess behavior changes, only the local liveness signal.

**Implementation.**
- `Sta/StaRuntime.cs`: in `PumpPendingMessages()`, capture the pump count, call `MarkActivity()`, return the count. (`MarkActivity` is already private on the same class — no visibility change.)
- Confirm every long-hold command routes its wait through this method: `ReadBulk` does (`MxAccessSession.cs:930` passes `pumpStep`, wired from `StaRuntime.PumpPendingMessages` at the executor). No new config.
- Optional defense-in-depth: document in `docs/GatewayConfiguration.md` that operators running very long bulk ops may raise `HeartbeatStuckCeiling` (already the doc's stated escape hatch, `docs/MxAccessWorkerInstanceDesign.md:689-690`).
- Tests: add `StaRuntimeTests.PumpPendingMessages_RefreshesLastActivity`, and an integration-style test in `Worker.Tests/Ipc/WorkerPipeSessionTests` using the fake runtime to prove a >75 s simulated in-flight command that keeps pumping does not emit `StaHung` and its reply is delivered.
- Docs: adjust `docs/MxAccessWorkerInstanceDesign.md:688-690` to state that the pump step refreshes activity, so the "no legitimate command runs that long" caveat is now enforced mechanically rather than assumed.

**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Worker/ZB.MOM.WW.MxGateway.Worker.csproj -p:Platform=x86` then `dotnet test src/ZB.MOM.WW.MxGateway.Worker.Tests/ZB.MOM.WW.MxGateway.Worker.Tests.csproj -p:Platform=x86 --filter "FullyQualifiedName~StaRuntimeTests|FullyQualifiedName~WorkerPipeSessionTests"`.

---

## WRK-02 — STA thread death after startup is silent; future work hangs forever   `Medium` · `—`

**Finding.** `ThreadMain` catches any loop exception into the write-only `startupException` field and sets `startedEvent` (`Sta/StaRuntime.cs:255-259`). After `Start()` has returned, nothing observes that field: the exception is never logged, never turned into a `WorkerFault`, and `shutdownRequested` stays false, so `InvokeAsync` keeps enqueuing work into a queue with no consumer and returns tasks that never complete (`Sta/StaRuntime.cs:165-179`). The only in-loop throw site is a pump-wait failure (`Sta/StaMessagePump.cs:38-42`).

**Impact.** If `MsgWaitForMultipleObjectsEx` returns `WAIT_FAILED` once, the STA exits; the dispatcher's drain wedges on the first stuck `InvokeAsync`; heartbeats keep flowing with a frozen `LastStaActivityUtc` and a pinned `CurrentCommandCorrelationId`; the worker is only declared hung after the 75 s ceiling, and the true root cause is permanently lost. Matches cross-cutting theme 1 (silent failure modes) in `00-overall.md`.

**Design.** Post-startup thread death must (1) fail all queued and future `InvokeAsync` calls deterministically and (2) surface the cause. Introduce a `terminalException` field set in the `ThreadMain` `catch`/`finally`, and a `Faulted` state distinct from graceful shutdown. In the `finally`, after `CancelQueuedCommands()`, transition to a terminal-faulted state so `InvokeAsync` returns `Task.FromException` (using the captured exception) for all subsequent calls, and `CancelQueuedCommands` completes already-queued items faulted rather than cancelled when the exit was abnormal. Add an optional `Action<Exception>? onTerminalFault` callback (constructor-injected, defaulted null) that `MxAccessStaSession` wires to record a `WorkerFault` on the event queue — reusing the existing fault→drain→IPC path (`MxAccessStaSession.RecordFault`, `Ipc/WorkerPipeSession.cs:344-353`) so the gateway sees a real fault frame instead of a slow watchdog timeout. Keep the callback optional to avoid disturbing the many unit tests that construct `StaRuntime` directly.

Rejected: polling `IsRunning` from the dispatcher — racy and still loses the exception. A push callback plus terminal task-completion is deterministic.

**Implementation.**
- `Sta/StaRuntime.cs`: add `private Exception? terminalException;` and a `volatile bool terminated;` (net48 — plain field, no init-only). In `ThreadMain`'s `catch`, store the exception; in `finally`, set `terminated = true` before `stoppedEvent.Set()`. In `InvokeAsync<T>`, after the `shutdownRequested` check, if `terminated` return `Task.FromException<T>(terminalException ?? new StaRuntimeShutdownException())`. Make `CancelQueuedCommands` fault (not cancel) items when `terminalException is not null`.
- Add constructor param `Action<Exception>? onTerminalFault = null`; invoke it once from `finally` when `terminalException is not null`.
- `MxAccess/MxAccessStaSession.cs`: pass a callback that calls `eventQueue.RecordFault(...)` with category `StaHung` (or a new `StaTerminated` if the contract enum allows — otherwise reuse `StaHung` and set a descriptive `DiagnosticMessage`).
- Tests: `Worker.Tests/Sta/StaRuntimeTests` — inject a message pump stub that throws on `WaitForWorkOrMessages`; assert the terminal callback fires, `InvokeAsync` after death returns a faulted task, and queued items fault.
- Docs: note in `docs/WorkerSta.md` that abnormal STA exit faults pending/future work and emits a fault frame.

**Verification.** `dotnet build ...Worker.csproj -p:Platform=x86` then `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~StaRuntimeTests"`.

---

## WRK-03 — Commands after shutdown starts are dropped with no reply   `Medium` · `—`

**Finding.** `TryStartCommandTask` returns silently when `_acceptingCommands` is false (`Ipc/WorkerPipeSession.cs:690-707`): no `WorkerUnavailable` reply, not even the `LogCommandResultDropped` diagnostic (which fires only for completed-then-dropped replies, `Ipc/WorkerPipeSession.cs:604-607`, `645-655`). The dispatcher layer *does* reply `WorkerUnavailable` when it is the one shutting down (`Sta/StaCommandDispatcher.cs:117-123`), but this earlier gate short-circuits before reaching it.

**Impact.** A command racing `WorkerShutdown` leaves the gateway's correlation wait to expire on its own timeout with no trace, exactly the silent-drop pattern `docs/MxAccessWorkerInstanceDesign.md:697-699` ("reject new commands") intends to avoid.

**Design.** Make the gate *loud*: when `_acceptingCommands` is false, write a `WorkerCommandReply` carrying `ProtocolStatusCode.WorkerUnavailable` for that correlation id (mirroring `StaCommandDispatcher.CreateRejectedReply`), then return. This matches the dispatcher-level rejection and gives the gateway an immediate, correlated failure. Guard the write with the same `_state`/`TryWriteFault`-style tolerance so a half-closed pipe during shutdown does not throw. Co-designed with WRK-02 (both close silent-death holes).

**Implementation.**
- `Ipc/WorkerPipeSession.cs`: in the `!_acceptingCommands` branch of `TryStartCommandTask`, build a rejection reply (reuse a small helper `CreateRejectedCommandReply(correlationId, method, WorkerUnavailable, "Worker is shutting down.")`) and enqueue a best-effort `_writer.WriteAsync(CreateEnvelope(reply), ...)` via a fire-and-forget observed task (same pattern as `ObserveCommandTaskAsync`). At minimum, log a `WorkerCommandRefusedDuringShutdown` diagnostic even if the write is skipped.
- Tests: `Worker.Tests/Ipc/WorkerPipeSessionTests` — after triggering shutdown, feed a command envelope and assert a `WorkerUnavailable` reply (or the diagnostic) is observed on the fake writer; covers the WRK-20 gap for S3.
- Docs: none beyond the existing shutdown sequence, which already promises rejection.

**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerPipeSessionTests"`.

---

## WRK-04 — Envelope `sequence` can appear out of order on the wire   `Medium` · `—`

**Finding.** `NextSequence()` is called while *building* the envelope in `CreateBaseEnvelope()` (`Ipc/WorkerPipeSession.cs:1005-1018`), but the write lock is acquired later, inside `WorkerFrameWriter.WriteAsync` (`Ipc/WorkerFrameWriter.cs:68-77`). Command replies are written from independent per-command tasks (`Ipc/WorkerPipeSession.cs:690-706`) concurrently with the heartbeat and event-drain loops, so task B can take sequence n+1 yet win the write lock before task A's sequence n. (Per-event ordering is safe — `WorkerSequence` is stamped inside the queue lock, `MxAccess/MxAccessEventQueue.cs:135-143` — but the envelope-level guarantee is not.)

**Impact.** Violates the `gateway.md` envelope rule "`sequence` is monotonic per sender"; any gateway consumer trusting wire-order monotonicity mis-sorts frames.

**Design.** Assign the envelope sequence *inside* the writer's critical section so the number and the write are atomic. Change `WorkerFrameWriter.WriteAsync` to accept a sequence-stamping callback (`Action<WorkerEnvelope>` or a `Func<ulong>` that the writer invokes under `_writeLock` to set `envelope.Sequence`) rather than reading a pre-stamped value. `CreateBaseEnvelope` stops calling `NextSequence()`; the writer stamps `Sequence` immediately before serialization, under the lock. This keeps a single `_nextSequence` counter and removes the window entirely. Co-designed with WRK-07 (both restructure the write path) — sequence stamping and priority scheduling should land together to avoid two passes over the writer.

Rejected: making `NextSequence` `Interlocked` (already is) does not help — the reordering is between *assignment* and *write*, not in the increment. The atomic region must span both.

**Implementation.**
- `Ipc/WorkerFrameWriter.cs`: add an overload `WriteAsync(WorkerEnvelope envelope, Func<ulong> sequenceProvider, CancellationToken)` that, after acquiring `_writeLock`, sets `envelope.Sequence = sequenceProvider()` *before* `Validate`/`CalculateSize`/`WriteTo`. (Serialize inside the lock now, since the payload depends on the stamped sequence.)
- `Ipc/WorkerPipeSession.cs`: `CreateBaseEnvelope()` no longer sets `Sequence`; every `_writer.WriteAsync(CreateEnvelope(...), ...)` call site passes `NextSequence` as the provider. Keep `NextSequence()` on the session (single owner of `_nextSequence`).
- Tests: `Worker.Tests/Ipc/WorkerPipeSessionTests` — a concurrent-writer test that fires N reply writes and M event writes in parallel through a capturing stream and asserts observed `Sequence` values are strictly monotonic in wire order (covers WRK-20 gap for S4). A `WorkerFrameWriterTests` unit test that the provider runs under the lock.
- Docs: none — restores the documented invariant.

**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerFrameWriterTests|FullyQualifiedName~WorkerPipeSessionTests"`.

---

## WRK-05 — One transient alarm-poll failure kills the whole session   `Medium` · `—`

**Finding.** Any exception from the alarm poll loop is recorded as a fault on the shared event queue and permanently stops the loop (`MxAccess/MxAccessStaSession.cs:278-291`); the drain loop turns that fault into full session termination (`Ipc/WorkerPipeSession.cs:344-353`). Failover only absorbs primary failures in composite mode (`MxAccess/FailoverAlarmConsumer.cs`); the default `WorkerPipeSession` builds the alarm handler with `standbyFactory: null` (`Ipc/WorkerPipeSession.cs:54`), so `AlarmCommandHandler.BuildConsumer` returns a bare consumer and a single `GetXmlCurrentAlarms2` COM error propagates unwrapped.

**Impact.** One transient `E_FAIL` from the AVEVA alarm subsystem terminates a client's healthy `OnDataChange` data stream even though the data path never failed — a whole-session death from an isolated alarm blip.

**Design.** Two viable scopes; recommend the smaller, parity-safe one:

1. **Recommended — bounded retry before declaring the alarm subscription dead.** Count *consecutive* poll failures against a threshold (mirror `FailoverSettings.Threshold`, default e.g. 3) with a short back-off; reset the counter on any successful poll. Only after the threshold record the fault. This tolerates transient COM errors without touching the data path and matches the existing failover semantics.
2. Scope the fault to the alarm feature (stop alarm delivery, keep data subscriptions and the session alive). Larger change — requires a per-feature fault channel the drain loop does not currently model, and risks masking a genuinely dead provider. Defer.

Parity note: this does not synthesize or suppress alarm *events* — it only changes how many consecutive *poll infrastructure* failures constitute "the subscription is dead." A persistent failure still faults the session, preserving fail-fast.

**Implementation.**
- `MxAccess/MxAccessStaSession.cs`: in the poll loop, maintain `int consecutiveFailures`; on catch, increment and only call `eventQueue.RecordFault(...)` + `return` when `consecutiveFailures >= threshold`; otherwise log a warning, `await Task.Delay(backoff)`, and continue. Reset on success. Keep the STA-affinity `InvalidOperationException` (from `EnsureOnAlarmConsumerThread`) as an *immediate* fault (it is a programming-error regression, not transient) — distinguish it before the counting branch.
- Config: add `AlarmPollFailureThreshold` and `AlarmPollBackoff` to the alarm handler options (thread through `AlarmCommandHandler`); default threshold to match `FailoverSettings.Threshold`.
- Tests: `Worker.Tests` alarm units — a consumer stub that throws N-1 times then succeeds keeps the session alive; N consecutive throws faults it; an affinity `InvalidOperationException` faults immediately.
- Docs: document the threshold in `docs/GatewayConfiguration.md` and note the transient-tolerance behavior in `docs/MxAccessWorkerInstanceDesign.md` alarm section.

**Verification.** `dotnet build ...Worker.csproj -p:Platform=x86` then `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~Alarm"`.

---

## WRK-06 — `MXSTATUS_PROXY` conversion reflects per field, per event   `Medium` · `P2`

**Finding.** `MxStatusProxyConverter.Convert` calls `ReadInt32Field` four times per status (`Conversion/MxStatusProxyConverter.cs:22-26`); each does `Type.GetField` + `FieldInfo.GetValue` + `Convert.ToInt32` (`:83-103`). This runs on the STA event path for every status of every `OnDataChange`.

**Impact.** Eight reflection ops plus boxing per event under data-change/alarm bursts — the exact load the gateway exists to handle. The status type is always the interop `MXSTATUS_PROXY` struct, so the field lookups are fully cacheable. Roadmap P2 item 14 (event hot-path pass).

**Design.** Cache the four `FieldInfo` objects keyed by `Type` in a small static `ConcurrentDictionary<Type, (FieldInfo success, category, detectedBy, detail)>`, resolved once per type and reused. `GetValue` + `Convert.ToInt32` still run per event (unavoidable via reflection over a late-bound COM RCW), but the `GetField` metadata scan — the expensive part — is eliminated. A direct cast to the interop struct type is faster still but couples the converter to the interop assembly and its exact struct shape; the review notes the type is stable, but the cached-`FieldInfo` approach keeps the converter interop-agnostic and testable with the existing plain-CLR test doubles. Recommend cached `FieldInfo`.

**Implementation.**
- `Conversion/MxStatusProxyConverter.cs`: add a static cache; a private `GetFields(Type)` that populates it once (throwing the same `MxStatusConversionException` if a field is missing) and `Convert` reads the four cached `FieldInfo`s. Behavior and exceptions unchanged.
- Tests: existing `MxStatusProxyConverter` tests must still pass (same outputs); add one asserting two conversions of the same type reuse cached metadata (e.g. via a type whose `GetField` is instrumented, or simply a throughput/regression test).
- Docs: none (internal perf).

**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~MxStatusProxyConverter"`.

---

## WRK-07 — Documented outbound write priority not implemented   `Medium` · `P1`

**Finding.** `docs/MxAccessWorkerInstanceDesign.md:600-613` specifies write priority faults > command replies > shutdown acks > heartbeats > events; the worker has no prioritized queue — all writers contend on a single FIFO `SemaphoreSlim` in `WorkerFrameWriter` (`Ipc/WorkerFrameWriter.cs:14`, `68-77`), and events are written inline by the drain loop (`Ipc/WorkerPipeSession.cs:362-367`).

**Impact.** With a deep event backlog draining (up to 128 per batch), a `WorkerFault` or command reply queues behind those event writes; on a slow pipe this delays the gateway's fault reaction. Relates to roadmap P1 item 8 (backpressure/size topology).

**Design.** Two acceptable outcomes; the review explicitly allows either. Recommend the *documented-decision* path first, with a small scheduler as the follow-up:

1. **Minimal now:** amend the design doc to state that FIFO write ordering was accepted for v1 (the single `_writeLock` serializes but does not prioritize), and that event batching (WRK-12) plus the sequence fix (WRK-04) bound the worst-case delay. This removes the doc/code contradiction immediately.
2. **Full fix (recommended for P1):** introduce a priority write scheduler in `WorkerFrameWriter` — a small set of per-priority queues drained newest-priority-first under `_writeLock`, or a `Channel`-per-priority merged by a single writer task. Faults and replies jump ahead of queued events. Must preserve per-sender sequence monotonicity — co-designed with WRK-04, since sequence is now stamped inside the lock at actual write time, priority reordering before stamping keeps sequence consistent with wire order automatically.

The full scheduler is the correct end state; if effort is constrained, ship option 1 in the same commit that lands WRK-04/WRK-12 and file option 2.

**Implementation.**
- `Ipc/WorkerFrameWriter.cs`: add an internal priority-ordered pending set drained by the lock holder; or expose `WriteAsync(envelope, priority, sequenceProvider, ct)`. `Ipc/WorkerPipeSession.cs` tags each write site with a priority (fault=0 … event=4).
- Tests: `Worker.Tests/Ipc/WorkerFrameWriterTests` — enqueue a fault behind many event writes on a blocked stream, unblock, assert the fault frame emerges first while sequences remain monotonic.
- Docs: update `docs/MxAccessWorkerInstanceDesign.md:600-613` to describe the implemented scheduler (or the accepted-FIFO decision if option 1).

**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerFrameWriterTests|FullyQualifiedName~WorkerPipeSessionTests"`.

---

## WRK-08 — Residual event queue discarded at graceful shutdown, undocumented   `Low` · `—`

**Finding.** `ShutdownAsync` writes the ack and returns; `RunMessageLoopAsync`'s `finally` cancels the event-drain loop (`Ipc/WorkerPipeSession.cs:287-292`) with whatever remains in `MxAccessEventQueue` unshipped; late replies on `_state != Ready` are dropped (`Ipc/WorkerPipeSession.cs:604-607`). An `OnWriteComplete` raised during cleanup never reaches the gateway. Acceptable for a closing session but not stated in `docs/MxAccessWorkerInstanceDesign.md`.

**Design.** Cheapest correct action: **document the discard** as intended v1 behavior — a session tearing down does not guarantee delivery of events queued after `WorkerShutdown`. Optionally (if a downstream wants last-gasp events) drain the queue once after `ShutdownGracefullyAsync` returns and before writing the ack, bounded by the grace period. Recommend documenting now; the final drain is a small enhancement to schedule only if a client needs it.

**Implementation.** Add a paragraph to `docs/MxAccessWorkerInstanceDesign.md` shutdown section. If implementing the drain: in `ShutdownAsync`, after `ShutdownGracefullyAsync`, call `DrainEvents` once and write each before the ack. Test: `WorkerPipeSessionTests` asserting either the documented discard or the final-drain delivery.

**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerPipeSessionTests"`.

---

## WRK-09 — No `AppDomain.UnhandledException` hook   `Low` · `—`

**Finding.** `Program.cs:1-4` and `WorkerApplication.Run` (`WorkerApplication.cs:46-141`) install no unhandled-exception or unobserved-task handlers (confirmed: no `AppDomain`/`UnhandledException` reference exists anywhere in the worker). An exception on an unobserved thread crashes the process with no `WorkerFault` and no log.

**Design.** Register `AppDomain.CurrentDomain.UnhandledException` and `TaskScheduler.UnobservedTaskException` at process start, logging the (redacted) exception through `IWorkerLogger` before exit. The gateway still detects death via process exit + pipe closure, so this is diagnostics-only; keep it minimal.

**Implementation.** In `WorkerApplication.Run` (or a tiny bootstrap in `Program.cs`), before parsing args, subscribe both handlers and log `WorkerUnhandledException` / `WorkerUnobservedTaskException` with `WorkerLogRedactor`-scrubbed message + type. Test: unit test the handler delegate logs and does not throw. Docs: mention in `docs/MxAccessWorkerInstanceDesign.md` logging list.

**Verification.** `dotnet build ...Worker.csproj -p:Platform=x86`; `dotnet test ...Worker.Tests... -p:Platform=x86`.

---

## WRK-10 — Top-level catch logs exception type but never message   `Low` · `—`

**Finding.** `WorkerApplication.cs:112-139` logs only `exception_type` for protocol, pipe, and unexpected failures; `HResultConverter.CreateSafeDiagnosticMessage` reduces every command exception to `Type: HRESULT 0x…` (`Conversion/HResultConverter.cs:46-49`).

**Design.** Log `exception.Message` (routed through `Bootstrap/WorkerLogRedactor` — which already scrubs nonces/credentials, `WorkerLogRedactor.cs:16-25`) alongside the type at the *process boundary*. Keep the credential-safe reply shape for IPC *replies* unchanged if the HRESULT-only stripping is intentional parity/secret policy — the fix is scoped to worker stderr/log, not the wire reply.

**Implementation.** In the three `WorkerApplication.Run` catch blocks, add `["exception_message"] = WorkerLogRedactor.Redact(exception.Message)`. Leave `HResultConverter` reply text as-is (document the intent in a code comment). Test: `WorkerApplicationTests` (or add) asserting the redacted message is present. Docs: none.

**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerApplication"`.

---

## WRK-11 — Every accepted event is defensively cloned on enqueue   `Low` · `P2`

**Finding.** `MxAccessEventQueue.Enqueue` calls `mxEvent.Clone()` (`MxAccess/MxAccessEventQueue.cs:135`) for an event the mapper built exclusively for this call (`MxAccess/MxAccessBaseEventSink.cs:210-256`); only the value-cache post-publish shares the original.

**Design.** Take ownership of the passed event in the queue (stamp `WorkerSequence`/`WorkerTimestamp` on it directly) and let the value cache store the copy — the cache already snapshots only value/quality/timestamp/statuses (`MxAccess/MxAccessValueCache.cs:44-57`), so it does not need the full `MxEvent` alias. This halves protobuf allocation on the hottest path. Requires confirming no caller reuses the passed `MxEvent` after enqueue (the mapper builds a fresh one per event — safe). Roadmap P2 item 14.

**Implementation.** Remove the `Clone()`; mutate the incoming `mxEvent` in place under the queue lock. Audit `MxAccessBaseEventSink` call sites to confirm single-ownership. Tests: existing event-queue tests must still pass; add one asserting the enqueued instance is the same reference passed in and that the value cache's stored copy is independent. Docs: none.

**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~MxAccessEventQueue|FullyQualifiedName~EventSink"`.

---

## WRK-12 — No event batching per envelope; one flush per event; 25 ms poll   `Low` · `P2`

**Finding.** The drain loop writes one `WriteAsync` per event (`Ipc/WorkerPipeSession.cs:362-367`), the writer flushes per frame (`Ipc/WorkerFrameWriter.cs:71-72`), and the poll interval is 25 ms (`Ipc/WorkerPipeSession.cs:17`). Each event costs a semaphore round-trip, a pipe write, and a flush; idle-to-active latency up to 25 ms. `gateway.md` lists event batching as the intended optimization.

**Design.** Acceptable for v1 parity (the review agrees). When throughput matters: either (a) add a repeated-event `WorkerEnvelope` body (a contracts/proto change — coordinate with domain 30 IPC, cross-references `IPC` findings; must preserve per-event order and sequence semantics), or (b) keep one event per envelope but coalesce *flushes* across a drained batch (flush once after writing the batch), which needs no proto change and captures most of the benefit. Recommend (b) now, (a) as a coordinated cross-domain change. Depends on WRK-04 (sequence stamping under the lock) so a coalesced batch keeps monotonic sequences.

**Implementation.** (Option b) `WorkerFrameWriter`: add a `WriteBatchAsync(IEnumerable<WorkerEnvelope>, ...)` that writes all frames then flushes once; drain loop calls it per drained batch. Tests: `WorkerFrameWriterTests` asserting one flush per batch and correct framing. Docs: note the flush-coalescing in `docs/WorkerFrameProtocol.md`; if option (a), update `.proto` and regenerate per CLAUDE.md contracts rule.

**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerFrameWriterTests"`.

---

## WRK-13 — Frame writer allocates a fresh buffer per frame; reader pools   `Low` · `P2`

**Finding.** `WorkerFrameWriter` does `new byte[frameLength]` per frame (`Ipc/WorkerFrameWriter.cs:63-66`) while `WorkerFrameReader` rents from `ArrayPool<byte>.Shared` (`Ipc/WorkerFrameReader.cs:55-77`).

**Design.** Rent the write buffer from `ArrayPool<byte>.Shared` for symmetry; return it in a `finally` after the write completes. Because `ArrayPool` may return an oversized buffer, pass explicit `(0, frameLength)` to `WriteAsync` (already does) and never leak the buffer's tail. Trivial, self-contained. Roadmap P2 item 14.

**Implementation.** `WorkerFrameWriter.WriteAsync`: `byte[] frame = ArrayPool<byte>.Shared.Rent(frameLength);` … `try { … } finally { ArrayPool<byte>.Shared.Return(frame); }`. Note the return must occur after the awaited write completes (it does — inside the same method). Tests: existing `WorkerFrameWriterTests` framing tests cover correctness; no behavior change. Docs: none.

**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerFrameWriterTests"`.

---

## WRK-14 — Private-field naming split `_camelCase` vs `camelCase`   `Low` · `—`

**Finding.** `Ipc/` and `Bootstrap/` use `_camelCase` (`Ipc/WorkerPipeSession.cs:21-38`, `Ipc/WorkerFrameWriter.cs:13-15`, `Bootstrap/WorkerConsoleLogger.cs:10`); `Sta/` and `MxAccess/` use bare `camelCase` (`Sta/StaRuntime.cs:10-24`, `MxAccess/MxAccessStaSession.cs:16-27`). `docs/style-guides/CSharpStyleGuide.md:30-32` permits the underscore prefix "only when already established" — both are established, so the worker has no single convention.

**Design.** Pick one style project-wide and migrate opportunistically (not in one churn commit). The gateway server is the tie-breaker: adopt whatever the gateway uses so the whole solution converges. This is a mechanical rename with no behavior change; do it file-by-file as those files are touched for other findings to keep diffs reviewable.

**Implementation.** Decide the target (check `src/ZB.MOM.WW.MxGateway.Server` private-field style), record it in `docs/style-guides/CSharpStyleGuide.md`, and rename incrementally. Tests: build only. Docs: the style-guide note.

**Verification.** `dotnet build ...Worker.csproj -p:Platform=x86` (analyzers/`TreatWarningsAsErrors` must stay green).

---

## WRK-15 — Doc drift: STA thread name and heartbeat-counter note   `Low` · `P2`

**Finding.** The STA thread is named `"MxGateway.Worker.STA"` (`Sta/StaRuntime.cs:61`) but `docs/WorkerSta.md:23,30` and `docs/MxAccessWorkerInstanceDesign.md:254` say `ZB.MOM.WW.MxGateway.Worker.STA`. And `docs/MxAccessWorkerInstanceDesign.md:653-654` says event-queue depth and sequence "are reported as zero until the event queue implementation owns those counters," but `CaptureHeartbeat` now populates both from the live queue (`MxAccess/MxAccessStaSession.cs:375-380`).

**Design.** Docs must match source (CLAUDE.md rule). Choose: either rename the thread to the documented `ZB.MOM.WW.MxGateway.Worker.STA` (operators grep thread dumps for it) or update the docs to the actual name — recommend renaming the thread to the fully-qualified documented name for operability, in the same commit that fixes the heartbeat-counter note. Roadmap P2 item 15 (doc-drift sweep).

**Implementation.** Either edit `Sta/StaRuntime.cs:61` thread `Name` to `ZB.MOM.WW.MxGateway.Worker.STA`, or edit the two docs. Fix `docs/MxAccessWorkerInstanceDesign.md:653-654` to state the counters are live. Tests: if renaming, update any test asserting the thread name. Docs: as above.

**Verification.** `dotnet build ...Worker.csproj -p:Platform=x86`; `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~StaRuntime"`.

---

## WRK-16 — Boilerplate duplication in IPC envelope/ctor overloads   `Low` · `—`

**Finding.** Seven near-identical `CreateEnvelope`/`CreateBaseEnvelope` overload pairs (`Ipc/WorkerPipeSession.cs:920-1003`) and eight `WorkerPipeClient` constructor overloads (`Ipc/WorkerPipeClient.cs:36-140`). Maintenance noise — each new body means two more copy-paste methods.

**Design.** Collapse the envelope overloads to a single `CreateEnvelope(Action<WorkerEnvelope> setBody)` (or a switch on the body message type); keep the correlation-id special case for `WorkerCommandReply` inside the setter. Reduce the client constructors to one primary constructor with the rest chaining via defaulted parameters (net48 supports optional params — no init-only needed). Pure refactor, no behavior change. Fold into WRK-04/WRK-07 since those already rework `CreateBaseEnvelope`.

**Implementation.** Refactor `WorkerPipeSession` envelope factories and `WorkerPipeClient` ctors. Tests: existing IPC tests must pass unchanged; they are the regression guard. Docs: none.

**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerPipeSession|FullyQualifiedName~WorkerPipeClient"`.

---

## WRK-17 — Gateway death exits with wrong exit code (6 not 5)   `Low` · `—`

**Finding.** Pipe EOF surfaces as `WorkerFrameProtocolException(EndOfStream)` (`Ipc/WorkerFrameReader.cs:104-109`), which `WorkerApplication.Run` catches first and maps to `ProtocolViolation` (6) (`WorkerApplication.cs:110-119`) even though `PipeConnectionFailed` (5) exists (`Bootstrap/WorkerExitCode.cs:10`) and the in-session fault mapping already distinguishes pipe disconnect.

**Design.** Special-case `WorkerFrameProtocolErrorCode.EndOfStream` in the `WorkerFrameProtocolException` catch to return `PipeConnectionFailed` (5) — EOF means the gateway went away, not that the worker misbehaved. Improves orphan-worker post-mortem triage.

**Implementation.** In `WorkerApplication.cs:110-119`, branch on `exception.ErrorCode == WorkerFrameProtocolErrorCode.EndOfStream` → log + return `PipeConnectionFailed`; else `ProtocolViolation`. Tests: `WorkerApplicationTests` — inject a pipe client throwing `EndOfStream` and assert exit code 5. Docs: `docs/WorkerFrameProtocol.md` / exit-code table if one exists.

**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerApplication"`.

---

## WRK-18 — Event-queue overflow exits as generic `UnexpectedFailure`   `Low` · `—`

**Finding.** `docs/MxAccessWorkerInstanceDesign.md:615-624` says overflow should "stop accepting new commands" and "let the gateway close or kill the worker"; the implementation instead terminates immediately — the drain loop writes the fault then throws (`Ipc/WorkerPipeSession.cs:344-353`), unwinding into the generic handler and exit code 1 (`WorkerApplication.cs:131-139`). The designed fault path is indistinguishable from a crash by exit code.

**Design.** The implemented fail-fast is arguably stronger than the doc and acceptable; the defect is *observability*. Give overflow a dedicated exit code (e.g. add `EventQueueOverflow` to `WorkerExitCode`) and catch the drain-fault termination in `WorkerApplication` to return it, then update the doc to describe the implemented immediate-terminate policy. Recommend keeping fail-fast (do not weaken to "stop accepting commands") and aligning the doc + exit code to it.

**Implementation.** Add `WorkerExitCode.EventQueueOverflow`; wrap the drain-fault throw in a typed exception (`WorkerEventQueueOverflowTerminationException`) so `WorkerApplication.Run` can map it. Tests: `WorkerPipeSessionTests` overflow path asserts the dedicated code. Docs: rewrite `docs/MxAccessWorkerInstanceDesign.md:615-624` to match.

**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerPipeSessionTests|FullyQualifiedName~WorkerApplication"`.

---

## WRK-19 — Command start/end logging with correlation id absent   `Low` · `—`

**Finding.** `docs/MxAccessWorkerInstanceDesign.md:790-791` lists "command start/end with correlation id" among required worker logs; the only per-command log is the dropped-reply diagnostic (`Ipc/WorkerPipeSession.cs:645-655`). `StaCommand.EnqueueTimestamp` is captured (`Sta/StaCommand.cs`) but never used for latency.

**Design.** Add optional, level-gated start/end logging in `StaCommandDispatcher.ExecuteQueuedCommandAsync`, which already brackets each command (`Sta/StaCommandDispatcher.cs:265-281`). Log correlation id + method at start; at end log outcome + latency (`now - EnqueueTimestamp`). Gate at a verbose/debug level so production noise is opt-in.

**Implementation.** Inject the optional `IWorkerLogger` into `StaCommandDispatcher` (or pass through the session); emit `WorkerCommandStarted`/`WorkerCommandCompleted` with correlation id, method, latency, outcome. Tests: `StaCommandDispatcherTests` asserting both logs fire with the correlation id. Docs: confirm the log names in `docs/MxAccessWorkerInstanceDesign.md` logging list.

**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~StaCommandDispatcher"`.

---

## WRK-20 — Test-coverage gaps for the failure modes above   `Low` · `—`

**Finding.** `Worker.Tests` covers the pump wake behavior, dispatcher ordering/cancellation/shutdown, handshake/heartbeat/watchdog (incl. the stuck ceiling), control commands, shutdown races, late-reply drops, frame protocol, conversion, event queue, and alarm units. Not covered: STA thread death mid-run (WRK-02), wire-level envelope sequence monotonicity under concurrent writers (WRK-04), the silent no-reply drop at the `_acceptingCommands` gate (WRK-03), and the `ReadBulk`-exceeds-ceiling false fault (WRK-01).

**Design.** Add the four tests *alongside* their fixes (named in each entry above) using the existing fake-runtime harness in `WorkerPipeSessionTests`, which already supports all four. This is not a separate work item so much as the acceptance criterion for WRK-01..04 — tracked here so it is not dropped.

**Implementation.** Tests to add: `StaRuntimeTests.ThreadDeath_FaultsPendingAndFutureWork` (WRK-02); `WorkerPipeSessionTests.ConcurrentWriters_SequenceIsMonotonic` (WRK-04); `WorkerPipeSessionTests.CommandAfterShutdown_RepliesWorkerUnavailable` (WRK-03); `WorkerPipeSessionTests.LongReadBulk_DoesNotFaultWhilePumping` + `StaRuntimeTests.PumpPendingMessages_RefreshesLastActivity` (WRK-01). Docs: none.

**Verification.** `dotnet test src/ZB.MOM.WW.MxGateway.Worker.Tests/ZB.MOM.WW.MxGateway.Worker.Tests.csproj -p:Platform=x86` (full worker suite once, after the batch lands, per the targeted-tests-then-phase-suite rule).

---

## Cross-references

- WRK-01 is roadmap P0 item 4; WRK-06/11/12/13 are the worker slice of P2 item 14 (event hot-path); WRK-15 is part of P2 item 15 (doc-drift sweep); WRK-07 relates to P1 item 8 (backpressure/size topology).
- WRK-04 and WRK-07 rework the write path together; WRK-16 folds into that refactor. WRK-12 depends on WRK-04's under-lock sequence stamping.
- WRK-02 and WRK-03 both close silent-failure edges (cross-cutting theme 1 in `00-overall.md`); the same theme spans gateway and client findings — coordinate the "every death/drop is observable" pattern across domains.