mxaccessgw/docs/plans/2026-08-09-write-completion-correlation-design.md

# WriteSecured Completion Correlation — Design

Date: 2026-08-09
Requested by: OtOpcUa (archreview finding 06/S-1, cross-repo residual — gateway half)
Status: approved for implementation (peer contract confirmed over cross-session message)

## Problem

The unary `Invoke` reply for a `WriteSecured` / `WriteSecured2` command proves worker-side
command *acceptance* only. MXAccess writes are fire-and-forget at the toolkit level: the
real per-item outcome arrives later in the `OnWriteComplete` COM callback. Today the worker
returns `CreateOkReply` immediately after the COM call, so `MxCommandReply.statuses` is
always empty and consumers (OtOpcUa's `GatewayGalaxyDataWriter.TranslateReply`) must treat
every write as provisionally good (`galaxy.writes.unconfirmed` meter).

Consumer contract (already shipped OtOpcUa-side): `statuses.Count > 0` → map `statuses[0]`
through the MX status map to a real OPC UA StatusCode; empty → provisional Good +
unconfirmed meter. The `statuses` field already exists on `MxCommandReply` (field 7); no
proto shape change is needed.

## Approaches Considered

1. **Executor pump-wait + versioned completion cache (chosen).** After the `WriteSecured`
   COM call, the executor holds the STA thread but explicitly pumps Windows messages each
   poll iteration until the matching completion is recorded or a bounded deadline passes.
   This is exactly the shipped `ReadBulk` pattern (`MxAccessValueCache.TryWaitForUpdate` +
   `pumpStep` → `StaRuntime.PumpPendingMessages()`), so it adds no new threading model.
2. **Parked asynchronous reply.** Executor returns a "pending" marker; the dispatcher parks
   the correlation and the pipe reply is written later from the event dispatch. Rejected:
   changes the `IStaCommandExecutor`/dispatcher/pipe contracts for no real gain — commands
   serialize per session anyway, so freeing the STA during the wait buys nothing.
3. **Gateway-side correlation.** Gateway watches the session event stream for the
   `OnWriteComplete` after the worker reply. Rejected: races the event drain cadence,
   couples the gateway to event semantics, still holds the unary RPC, and spreads the
   feature across two processes.

## Design

All changes are worker-side (`ZB.MOM.WW.MxGateway.Worker`, net48 x86). The gateway's
`Invoke` already forwards the worker `MxCommandReply` (statuses included) verbatim.

### New: `MxAccessWriteCompletionCache`

Mirror of `MxAccessValueCache`, keyed by `(serverHandle, itemHandle)` (packed long), one
entry per key holding the most recent completion's `RepeatedField<MxStatusProxy>` (cloned)
plus a monotonically increasing per-key `Version`. API:

- `Record(int serverHandle, int itemHandle, RepeatedField<MxStatusProxy> statuses)`
- `ulong CurrentVersion(int serverHandle, int itemHandle)` — 0 when absent
- `bool TryWaitForCompletion(int serverHandle, int itemHandle, ulong sinceVersion,
  DateTime deadlineUtc, Action pumpStep, out RepeatedField<MxStatusProxy> statuses,
  int pollIntervalMs = 5)` — pump/poll loop identical in shape to
  `MxAccessValueCache.TryWaitForUpdate`.

Same locking posture as the value cache: everything runs on the STA thread; a sync root
keeps it nominally thread-safe for tests.

### Sink: record completions

`MxAccessBaseEventSink` owns a `MxAccessWriteCompletionCache` (new optional ctor param,
exposed as a property) and its `OnWriteComplete` handler records into it via the existing
`EnqueueEvent` post-publish hook (same pattern as the value cache on `OnDataChange`):
the streamed `MxEvent` is built exactly once by the mapper, enqueued unchanged for the
event stream, and its `Statuses` are then recorded into the cache. The event stream is
not altered — nothing is swallowed or synthesized.

A new seam interface `IWriteCompletionCacheProvider { MxAccessWriteCompletionCache
WriteCompletionCache { get; } }` is implemented by `MxAccessBaseEventSink` and by test
sinks. `MxAccessSession.Create` pulls the cache from the sink through that interface
(fallback: fresh instance), mirroring the existing `ValueCache` sharing, and exposes it
on the session.

### Executor: bounded pump-wait

In `ExecuteWriteSecured` / `ExecuteWriteSecured2`:

1. Capture `baseline = cache.CurrentVersion(serverHandle, itemHandle)` **before** the COM
   call — this closes the fast-completion ordering edge: a callback that dispatches during
   or immediately after the COM call bumps the version past the baseline and still
   correlates.
2. Call `session.WriteSecured(...)` as today.
3. `cache.TryWaitForCompletion(..., baseline, deadline, pumpStep, out statuses)`; on
   success, `reply.Statuses.Add(statuses)`; on timeout, return the reply exactly as today
   (protocol OK, empty statuses) — the consumer's honest-unconfirmed path. No invented
   failure rows.

The reply's `ProtocolStatus`/`Hresult` stay untouched by the completion outcome: the
command was accepted; the MX outcome (success *or* failure) is carried only in
`statuses[0]`. That preserves MXAccess parity (the native API returns void; the outcome
exists only in the callback) while enriching the reply with information that was already
on the wire.

Timeout default: **1.5 s** (`MxAccessCommandExecutor.DefaultWriteCompletionTimeout`).
The consumer-side budget drives this: OtOpcUa wraps the driver write in a Tier A
resilience policy with a 2 s timeout and a 5-failure breaker, so a worker wait longer
than 2 s would convert slow-but-successful commits into consumer-side false failures
(node revert + breaker pressure). 1.5 s covers the common fast-commit case and degrades
a slow commit to the honest-unconfirmed path instead. It also sits well under the
gateway's 30 s `DefaultCommandTimeoutSeconds` IPC wait and the STA watchdog's 75 s
dispatched-command ceiling.

The wait is deployment-configurable end to end, following the existing
pipe-connect-timeout pattern: a new gateway option
`MxGateway:Worker:WriteCompletionWaitMilliseconds` (default `1500`, validated `>= 0`;
`0` disables the wait and restores pure fire-and-forget replies) is exported by
`WorkerProcessLauncher` as the `MXGATEWAY_WORKER_WRITE_COMPLETION_WAIT_MS` environment
variable, which the worker reads at session construction. A deployment that raises the
gateway wait must raise the OtOpcUa driver's Write `ResilienceConfig` timeout in step
(per-instance operator config on the OtOpcUa side) — documented in
`docs/GatewayConfiguration.md`. `MxAccessStaSession` additionally gets an internal
`WriteCompletionTimeout` seam (read when it constructs the executor in `StartAsync`) so
tests can shorten it without env plumbing.

Client cancellation mid-wait needs no new code path, only verification: when the caller
cancels the unary RPC, the gateway abandons its IPC reply wait, but the worker command
is already in flight — `CancelCommand` only dequeues *queued* commands. The executor
simply finishes its bounded wait and replies; the gateway discards the reply. The
session is never faulted and the `OnWriteComplete` event still flows on the stream.

### Scope

- **In**: `WriteSecured`, `WriteSecured2` (single-item secured writes — inherently
  low-rate operator actions, and exactly the OtOpcUa single-write contract). Default-on.
- **Out**: plain `Write`/`Write2` and all bulk write commands stay fire-and-forget —
  waiting would add a device round-trip of latency to high-rate supervisory write loops.
- **Correlation fidelity is best-effort**: the MXAccess callback carries only
  `(hItem, statuses)` — no transaction id — so a concurrent write to the same item within
  the wait window can be attributed to the wrong writer (worst case two writes to the
  same item swap status rows — benign for the serialized single-write consumer
  contract). Documented on the proto field.

## Error handling

- Completion never arrives (device down): bounded 1.5 s wait, then today's reply shape.
- Event queue overflow during completion: the queue records a fault and the fail-fast
  design tears the session down; the post-publish hook not firing in that case is moot.
- COM call throws: unchanged — the dispatcher's existing exception path replies with the
  native HResult; no wait is entered.

## Testing (Worker.Tests, x86 — verified on windev)

- `MxAccessWriteCompletionCacheTests`: record/version monotonicity, wait success,
  deadline expiry, baseline-before-record fast-completion ordering.
- `MxAccessBaseEventSinkTests`: `OnWriteComplete` both enqueues the event *and* records
  the completion; cache instance is the sink-bound one.
- `MxAccessCommandExecutorTests` (via `MxAccessStaSession.DispatchAsync` + fake COM
  object + test sink implementing `IWriteCompletionCacheProvider`):
  - completion recorded synchronously inside the fake's `WriteSecured` (fast edge) →
    reply carries `statuses[0]`;
  - completion recorded from the test thread while the executor pump-waits → reply
    carries `statuses[0]`;
  - no completion + shortened timeout → protocol OK, empty statuses;
  - `WriteSecured2` mirrors; plain `Write` does not wait.
- Gateway tests (macOS-runnable): `GatewayOptionsValidator` accepts `>= 0` and rejects
  negative `WriteCompletionWaitMilliseconds`; `WorkerProcessLauncher` exports
  `MXGATEWAY_WORKER_WRITE_COMPLETION_WAIT_MS` from the option (mirroring the existing
  pipe-connect-timeout env assertion).

## Docs in the same change

- `mxaccess_gateway.proto`: comment on `MxCommandReply.statuses` and the
  `WriteSecuredCommand`/`WriteSecured2Command` messages describing the correlated
  completion contract (populated within the wait window; empty = unconfirmed;
  best-effort correlation). Comment-only → wire-identical; regenerate + commit
  `Contracts/Generated` (required for the C# build); other clients' generated code is
  functionally unchanged.
- `docs/GatewayConfiguration.md`: the new `MxGateway:Worker:WriteCompletionWaitMilliseconds`
  option, including the pairing rule with the OtOpcUa driver's Write resilience timeout.
- `gateway.md` command/event surface note; `docs/DesignDecisions.md` entry.