diff --git a/archreview/2026-08-19/remediation/00-tracking.md b/archreview/2026-08-19/remediation/00-tracking.md index 2822cd0..97def11 100644 --- a/archreview/2026-08-19/remediation/00-tracking.md +++ b/archreview/2026-08-19/remediation/00-tracking.md @@ -18,7 +18,7 @@ This plan is executed as **seven parallel batches**, each by one subagent workin | C | `fix/c3-tracker-hygiene` | GWC-31 + housekeeping (close GWC-16/IPC-08/IPC-31, refresh TST-06/TST-17, register WRK-31) | sonnet | Cross-file comment/ID sweep needs care about what each ID *means*, but no logic changes | Not started | | D | `fix/c3-dashboard-acl` | SEC-40, SEC-41, GWC-38 | opus | Authorization semantics, fail-closed decisions, claims re-resolution; each needs a regression test | Not started | | E | `fix/c3-ipc-boundaries` | IPC-33, IPC-39, IPC-34, IPC-35 | opus | Changes the WorkerClient command lifecycle, the options validator, and the worker drain budget; race-sensitive | Not started | -| F | `fix/c3-worker-residuals` | WRK-32, WRK-33 (+WRK-34 opportunistic) | sonnet | Small, precisely located worker edits + doc prose; cannot be built on macOS so the agent's job is a careful, minimal diff | Not started | +| F | `fix/c3-worker-residuals` | WRK-32, WRK-33 (+WRK-34 opportunistic) | sonnet | Small, precisely located worker edits + doc prose; cannot be built on macOS so the agent's job is a careful, minimal diff | Done-pending-windev | | G | `fix/c3-session-lifecycle` | GWC-32, GWC-34, GWC-37 | opus | Distributor/dispose lifecycle races and an alarm-monitor re-resolve path; needs lock-discipline reasoning | Not started | **Out of scope for this execution (recorded, not dropped):** @@ -66,6 +66,7 @@ This plan is executed as **seven parallel batches**, each by one subagent workin - WRK-32: one `pumpStep()` before capturing the `CurrentVersion` baseline in `MxAccessCommandExecutor.cs`; rewrite the executor doc comment (`:996-1003`) to state the recency-based contract; add the same to `docs/MxAccessWorkerInstanceDesign.md` write-completion section and the reply-field comment in `mxaccess_gateway.proto` (**comment-only proto edit → regenerate all generated artifacts in the same commit**, per the repo rule; if regen tooling is unavailable on macOS, skip the proto comment and say so). - WRK-34 if trivial: `CreateFault(StaHung, "Watchdog", …)` with the correlation id moved into the message. - Cannot build on macOS: keep diff minimal, net48-safe (no init-only/positional records). Verified on windev afterwards. +- **Done-pending-windev** (`93edb1b` WRK-33, `20a3eef` WRK-32, `3679e59` WRK-34) on `fix/c3-worker-residuals`. WRK-32 applied `CaptureCompletionBaseline` (pump-then-snapshot) uniformly to all four write call sites (Write/Write2/WriteSecured/WriteSecured2), not just the two the finding cited as evidence, since the correlation gap is identical at all four. **Deferred**: the `mxaccess_gateway.proto` reply-field comment was not edited — a comment-only proto change still requires regenerating every generated artifact (C#, Go, Python, Java, Rust, protoset) per the repo rule, and this macOS box cannot verify all five client generator pins for a Low-severity doc-only change; the contract is documented in `docs/MxAccessWorkerInstanceDesign.md` instead. `NonWindows.slnx` builds clean (0 warnings/errors); worker/Worker.Tests unverified until windev runs `MxAccessCommandExecutorTests`/`WorkerPipeSessionTests`. ### G — GWC-32, GWC-34, GWC-37 (opus) - GWC-32: `DisposeAsync` transitions `_state` to `Closed` under `_syncRoot` before completing channels (or `if (_disposed || IsTerminalState()) return;` in `StageWorkerEvent`); update the XML doc to name all three completion paths. Test: dispose a Ready client mid-dispatch → no `ProtocolViolation` fault/metric. diff --git a/docs/MxAccessWorkerInstanceDesign.md b/docs/MxAccessWorkerInstanceDesign.md index 28f49a1..fab6491 100644 --- a/docs/MxAccessWorkerInstanceDesign.md +++ b/docs/MxAccessWorkerInstanceDesign.md @@ -317,6 +317,30 @@ whose write workload is effectively fire-and-forget — no consumer reads the reply's `statuses` — can lower it, or set `0` to skip the wait entirely and reply on acceptance alone. +**Write-completion correlation is recency-based, not per-call.** MXAccess's +`OnWriteComplete` callback carries no correlation token — only the server/item +handle pair — so `MxAccessCommandExecutor` correlates a completion to a write +by *version*: it snapshots `MxAccessWriteCompletionCache.CurrentVersion` for +the handle as a baseline before the COM call, then reports the first +completion whose version exceeds that baseline as the write's outcome +(`AwaitWriteCompletion`). This is exact for the common case, but if an earlier +write to the same handle times out unconfirmed — its `OnWriteComplete` still +undelivered when the wait gives up — and the caller immediately issues another +write to the same tag, the stale callback can dispatch during the second +write's pump-wait and be reported as *its* outcome. The executor mitigates the +most common instance of this (`CaptureCompletionBaseline`) by running one +`pumpStep()` immediately before capturing the baseline, so a completion already +queued for the handle folds into the baseline instead of being available to +ride the next write's reply; this closes the gap for a callback that arrived +before the new write started, but a callback that MXAccess queues after that +pump step and before the COM call returns (or during the wait itself) is still +indistinguishable from the new write's own completion. Full closure is not +possible without an MXAccess-side correlation token, which does not exist — +this is a property of the COM surface, not a gap this worker introduced (a +native client correlating by handle has the identical ambiguity). The exposure +is bounded: `statuses` is an enrichment field, and `Hresult`/`ProtocolStatus` +stay acceptance-only regardless of which completion (if any) was attributed. + ## COM Creation The MXAccess analysis source at `C:\Users\dohertj2\Desktop\mxaccess` identifies diff --git a/src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerPipeSession.cs b/src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerPipeSession.cs index a92fa0b..d52f8c4 100644 --- a/src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerPipeSession.cs +++ b/src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerPipeSession.cs @@ -1232,7 +1232,14 @@ public sealed class WorkerPipeSession _activeCommandTasks.Add(commandTask); } - _ = ObserveCommandTaskAsync(commandTask); + // WRK-33: every other discarded task in this file is wrapped with ObserveFaultWhenever + // (NEXT-04 discipline) so a fault reaching the finalizer unobserved is impossible. This + // wrapper task was the one exception — ObserveCommandTaskAsync only swallows + // OperationCanceledException, so any other exception escaping ProcessCommandAsync's own + // catch path (e.g. a future edit, or a failure past TryWriteFaultAsync's narrow filter) + // would fault the discarded task with no observer. Close the gap the same way the rest of + // the file does. + ObserveFaultWhenever(ObserveCommandTaskAsync(commandTask)); } private async Task ObserveCommandTaskAsync(Task commandTask) @@ -1381,11 +1388,18 @@ public sealed class WorkerPipeSession // the StaHung fault just sent. Without this the heartbeat loop keeps // advertising a non-faulted state that contradicts the fault. _state = WorkerState.Faulted; + // WRK-34: CommandMethod is a method-name field, not a correlation-id field, so a + // consumer reading it (dashboard, gateway logs) shouldn't see a bare correlation id + // there. Use a stable label and carry the stuck command's correlation id (if any) in + // the diagnostic message instead. + string correlationSuffix = string.IsNullOrEmpty(snapshot.CurrentCommandCorrelationId) + ? string.Empty + : $" Stuck command correlation id: {snapshot.CurrentCommandCorrelationId}."; await TryWriteFaultAsync( CreateFault( WorkerFaultCategory.StaHung, - snapshot.CurrentCommandCorrelationId, - $"STA activity is stale by {staleFor}."), + "Watchdog", + $"STA activity is stale by {staleFor}.{correlationSuffix}"), cancellationToken).ConfigureAwait(false); } diff --git a/src/ZB.MOM.WW.MxGateway.Worker/MxAccess/MxAccessCommandExecutor.cs b/src/ZB.MOM.WW.MxGateway.Worker/MxAccess/MxAccessCommandExecutor.cs index 62b4e94..46cbc7e 100644 --- a/src/ZB.MOM.WW.MxGateway.Worker/MxAccess/MxAccessCommandExecutor.cs +++ b/src/ZB.MOM.WW.MxGateway.Worker/MxAccess/MxAccessCommandExecutor.cs @@ -445,7 +445,8 @@ public sealed class MxAccessCommandExecutor : IStaCommandExecutor // also fire-and-forget in MXAccess, and its commit outcome only exists // in the later OnWriteComplete callback. MxAccessWriteCompletionCache completionCache = session.WriteCompletionCache; - ulong completionBaseline = completionCache.CurrentVersion( + ulong completionBaseline = CaptureCompletionBaseline( + completionCache, writeCommand.ServerHandle, writeCommand.ItemHandle); @@ -485,7 +486,8 @@ public sealed class MxAccessCommandExecutor : IStaCommandExecutor // Same pre-call baseline rule as ExecuteWriteSecured. MxAccessWriteCompletionCache completionCache = session.WriteCompletionCache; - ulong completionBaseline = completionCache.CurrentVersion( + ulong completionBaseline = CaptureCompletionBaseline( + completionCache, write2Command.ServerHandle, write2Command.ItemHandle); @@ -523,7 +525,8 @@ public sealed class MxAccessCommandExecutor : IStaCommandExecutor // immediately after WriteSecured bumps the version past this snapshot, // so a fast commit still correlates (no missed-callback window). MxAccessWriteCompletionCache completionCache = session.WriteCompletionCache; - ulong completionBaseline = completionCache.CurrentVersion( + ulong completionBaseline = CaptureCompletionBaseline( + completionCache, writeSecuredCommand.ServerHandle, writeSecuredCommand.ItemHandle); @@ -564,7 +567,8 @@ public sealed class MxAccessCommandExecutor : IStaCommandExecutor // Same pre-call baseline rule as ExecuteWriteSecured. MxAccessWriteCompletionCache completionCache = session.WriteCompletionCache; - ulong completionBaseline = completionCache.CurrentVersion( + ulong completionBaseline = CaptureCompletionBaseline( + completionCache, writeSecured2Command.ServerHandle, writeSecured2Command.ItemHandle); @@ -994,7 +998,31 @@ public sealed class MxAccessCommandExecutor : IStaCommandExecutor } /// - /// Bounded pump-wait for the OnWriteComplete row matching a + /// Captures the write-completion baseline for / + /// immediately before a Write/Write2/WriteSecured/ + /// WriteSecured2 COM call. Pumps once first (WRK-32) so an OnWriteComplete + /// already queued for this handle — most commonly the still-undelivered callback of an + /// earlier write to the same tag that this worker gave up waiting on — folds into the + /// baseline instead of being available to ride this write's reply. + /// + /// + /// This narrows, but does not close, the correlation gap documented on + /// : a completion that MXAccess queues after this pump + /// step but before the COM call returns is still indistinguishable from this write's own + /// completion, because OnWriteComplete carries no per-call token — only the + /// server/item handle pair. + /// + private ulong CaptureCompletionBaseline( + MxAccessWriteCompletionCache completionCache, + int serverHandle, + int itemHandle) + { + pumpStep(); + return completionCache.CurrentVersion(serverHandle, itemHandle); + } + + /// + /// Bounded pump-wait for the OnWriteComplete row matching a Write/Write2/ /// WriteSecured/WriteSecured2 call, copied onto the reply when it /// arrives in time. The executor holds the STA thread but pumps /// Windows messages each poll (ReadBulk precedent) so the COM callback @@ -1002,6 +1030,21 @@ public sealed class MxAccessCommandExecutor : IStaCommandExecutor /// statuses — the consumer's unconfirmed path, never a synthesized /// failure. Protocol status/hresult stay acceptance-only either way. /// + /// + /// Correlation is recency-based per handle, not per call: MXAccess's + /// OnWriteComplete callback carries no correlation token, only the server/item + /// handle pair, so this method reports the first completion whose version exceeds + /// as this write's outcome. If an earlier write to + /// the same handle times out unconfirmed and its callback is still undelivered, that + /// stale completion can dispatch during this write's pump-wait and ride this reply + /// instead — 's pre-call pump only folds in a + /// completion already queued before this write started; it cannot fold in one queued + /// during the COM call or the wait itself. The Hresult/ProtocolStatus + /// fields stay acceptance-only regardless, so the exposure is limited to the enrichment + /// statuses field, and it is inherent to the COM surface — a native client + /// correlating by handle has the identical ambiguity. Full closure needs an + /// MXAccess-side correlation token, which does not exist. + /// private void AwaitWriteCompletion( MxCommandReply reply, MxAccessWriteCompletionCache completionCache,