# WriteSecured Completion Correlation — Design Date: 2026-08-09 Requested by: OtOpcUa (archreview finding 06/S-1, cross-repo residual — gateway half) Status: approved for implementation (peer contract confirmed over cross-session message) ## Problem The unary `Invoke` reply for a `WriteSecured` / `WriteSecured2` command proves worker-side command *acceptance* only. MXAccess writes are fire-and-forget at the toolkit level: the real per-item outcome arrives later in the `OnWriteComplete` COM callback. Today the worker returns `CreateOkReply` immediately after the COM call, so `MxCommandReply.statuses` is always empty and consumers (OtOpcUa's `GatewayGalaxyDataWriter.TranslateReply`) must treat every write as provisionally good (`galaxy.writes.unconfirmed` meter). Consumer contract (already shipped OtOpcUa-side): `statuses.Count > 0` → map `statuses[0]` through the MX status map to a real OPC UA StatusCode; empty → provisional Good + unconfirmed meter. The `statuses` field already exists on `MxCommandReply` (field 7); no proto shape change is needed. ## Approaches Considered 1. **Executor pump-wait + versioned completion cache (chosen).** After the `WriteSecured` COM call, the executor holds the STA thread but explicitly pumps Windows messages each poll iteration until the matching completion is recorded or a bounded deadline passes. This is exactly the shipped `ReadBulk` pattern (`MxAccessValueCache.TryWaitForUpdate` + `pumpStep` → `StaRuntime.PumpPendingMessages()`), so it adds no new threading model. 2. **Parked asynchronous reply.** Executor returns a "pending" marker; the dispatcher parks the correlation and the pipe reply is written later from the event dispatch. Rejected: changes the `IStaCommandExecutor`/dispatcher/pipe contracts for no real gain — commands serialize per session anyway, so freeing the STA during the wait buys nothing. 3. **Gateway-side correlation.** Gateway watches the session event stream for the `OnWriteComplete` after the worker reply. Rejected: races the event drain cadence, couples the gateway to event semantics, still holds the unary RPC, and spreads the feature across two processes. ## Design All changes are worker-side (`ZB.MOM.WW.MxGateway.Worker`, net48 x86). The gateway's `Invoke` already forwards the worker `MxCommandReply` (statuses included) verbatim. ### New: `MxAccessWriteCompletionCache` Mirror of `MxAccessValueCache`, keyed by `(serverHandle, itemHandle)` (packed long), one entry per key holding the most recent completion's `RepeatedField` (cloned) plus a monotonically increasing per-key `Version`. API: - `Record(int serverHandle, int itemHandle, RepeatedField statuses)` - `ulong CurrentVersion(int serverHandle, int itemHandle)` — 0 when absent - `bool TryWaitForCompletion(int serverHandle, int itemHandle, ulong sinceVersion, DateTime deadlineUtc, Action pumpStep, out RepeatedField statuses, int pollIntervalMs = 5)` — pump/poll loop identical in shape to `MxAccessValueCache.TryWaitForUpdate`. Same locking posture as the value cache: everything runs on the STA thread; a sync root keeps it nominally thread-safe for tests. ### Sink: record completions `MxAccessBaseEventSink` owns a `MxAccessWriteCompletionCache` (new optional ctor param, exposed as a property) and its `OnWriteComplete` handler records into it via the existing `EnqueueEvent` post-publish hook (same pattern as the value cache on `OnDataChange`): the streamed `MxEvent` is built exactly once by the mapper, enqueued unchanged for the event stream, and its `Statuses` are then recorded into the cache. The event stream is not altered — nothing is swallowed or synthesized. A new seam interface `IWriteCompletionCacheProvider { MxAccessWriteCompletionCache WriteCompletionCache { get; } }` is implemented by `MxAccessBaseEventSink` and by test sinks. `MxAccessSession.Create` pulls the cache from the sink through that interface (fallback: fresh instance), mirroring the existing `ValueCache` sharing, and exposes it on the session. ### Executor: bounded pump-wait In `ExecuteWriteSecured` / `ExecuteWriteSecured2`: 1. Capture `baseline = cache.CurrentVersion(serverHandle, itemHandle)` **before** the COM call — this closes the fast-completion ordering edge: a callback that dispatches during or immediately after the COM call bumps the version past the baseline and still correlates. 2. Call `session.WriteSecured(...)` as today. 3. `cache.TryWaitForCompletion(..., baseline, deadline, pumpStep, out statuses)`; on success, `reply.Statuses.Add(statuses)`; on timeout, return the reply exactly as today (protocol OK, empty statuses) — the consumer's honest-unconfirmed path. No invented failure rows. The reply's `ProtocolStatus`/`Hresult` stay untouched by the completion outcome: the command was accepted; the MX outcome (success *or* failure) is carried only in `statuses[0]`. That preserves MXAccess parity (the native API returns void; the outcome exists only in the callback) while enriching the reply with information that was already on the wire. Timeout default: **1.5 s** (`MxAccessCommandExecutor.DefaultWriteCompletionTimeout`). The consumer-side budget drives this: OtOpcUa wraps the driver write in a Tier A resilience policy with a 2 s timeout and a 5-failure breaker, so a worker wait longer than 2 s would convert slow-but-successful commits into consumer-side false failures (node revert + breaker pressure). 1.5 s covers the common fast-commit case and degrades a slow commit to the honest-unconfirmed path instead. It also sits well under the gateway's 30 s `DefaultCommandTimeoutSeconds` IPC wait and the STA watchdog's 75 s dispatched-command ceiling. The wait is deployment-configurable end to end, following the existing pipe-connect-timeout pattern: a new gateway option `MxGateway:Worker:WriteCompletionWaitMilliseconds` (default `1500`, validated `>= 0`; `0` disables the wait and restores pure fire-and-forget replies) is exported by `WorkerProcessLauncher` as the `MXGATEWAY_WORKER_WRITE_COMPLETION_WAIT_MS` environment variable, which the worker reads at session construction. A deployment that raises the gateway wait must raise the OtOpcUa driver's Write `ResilienceConfig` timeout in step (per-instance operator config on the OtOpcUa side) — documented in `docs/GatewayConfiguration.md`. `MxAccessStaSession` additionally gets an internal `WriteCompletionTimeout` seam (read when it constructs the executor in `StartAsync`) so tests can shorten it without env plumbing. Client cancellation mid-wait needs no new code path, only verification: when the caller cancels the unary RPC, the gateway abandons its IPC reply wait, but the worker command is already in flight — `CancelCommand` only dequeues *queued* commands. The executor simply finishes its bounded wait and replies; the gateway discards the reply. The session is never faulted and the `OnWriteComplete` event still flows on the stream. ### Scope - **In**: `WriteSecured`, `WriteSecured2` (single-item secured writes — inherently low-rate operator actions, and exactly the OtOpcUa single-write contract). Default-on. - **Out**: plain `Write`/`Write2` and all bulk write commands stay fire-and-forget — waiting would add a device round-trip of latency to high-rate supervisory write loops. - **Correlation fidelity is best-effort**: the MXAccess callback carries only `(hItem, statuses)` — no transaction id — so a concurrent write to the same item within the wait window can be attributed to the wrong writer (worst case two writes to the same item swap status rows — benign for the serialized single-write consumer contract). Documented on the proto field. ## Error handling - Completion never arrives (device down): bounded 1.5 s wait, then today's reply shape. - Event queue overflow during completion: the queue records a fault and the fail-fast design tears the session down; the post-publish hook not firing in that case is moot. - COM call throws: unchanged — the dispatcher's existing exception path replies with the native HResult; no wait is entered. ## Testing (Worker.Tests, x86 — verified on windev) - `MxAccessWriteCompletionCacheTests`: record/version monotonicity, wait success, deadline expiry, baseline-before-record fast-completion ordering. - `MxAccessBaseEventSinkTests`: `OnWriteComplete` both enqueues the event *and* records the completion; cache instance is the sink-bound one. - `MxAccessCommandExecutorTests` (via `MxAccessStaSession.DispatchAsync` + fake COM object + test sink implementing `IWriteCompletionCacheProvider`): - completion recorded synchronously inside the fake's `WriteSecured` (fast edge) → reply carries `statuses[0]`; - completion recorded from the test thread while the executor pump-waits → reply carries `statuses[0]`; - no completion + shortened timeout → protocol OK, empty statuses; - `WriteSecured2` mirrors; plain `Write` does not wait. - Gateway tests (macOS-runnable): `GatewayOptionsValidator` accepts `>= 0` and rejects negative `WriteCompletionWaitMilliseconds`; `WorkerProcessLauncher` exports `MXGATEWAY_WORKER_WRITE_COMPLETION_WAIT_MS` from the option (mirroring the existing pipe-connect-timeout env assertion). ## Docs in the same change - `mxaccess_gateway.proto`: comment on `MxCommandReply.statuses` and the `WriteSecuredCommand`/`WriteSecured2Command` messages describing the correlated completion contract (populated within the wait window; empty = unconfirmed; best-effort correlation). Comment-only → wire-identical; regenerate + commit `Contracts/Generated` (required for the C# build); other clients' generated code is functionally unchanged. - `docs/GatewayConfiguration.md`: the new `MxGateway:Worker:WriteCompletionWaitMilliseconds` option, including the pairing rule with the OtOpcUa driver's Write resilience timeout. - `gateway.md` command/event surface note; `docs/DesignDecisions.md` entry.