mxaccessgw/docs/plans/2026-08-09-write-completion-correlation-design.md

WriteSecured Completion Correlation — Design

Date: 2026-08-09 Requested by: OtOpcUa (archreview finding 06/S-1, cross-repo residual — gateway half) Status: approved for implementation (peer contract confirmed over cross-session message)

Problem

The unary Invoke reply for a WriteSecured / WriteSecured2 command proves worker-side command acceptance only. MXAccess writes are fire-and-forget at the toolkit level: the real per-item outcome arrives later in the OnWriteComplete COM callback. Today the worker returns CreateOkReply immediately after the COM call, so MxCommandReply.statuses is always empty and consumers (OtOpcUa's GatewayGalaxyDataWriter.TranslateReply) must treat every write as provisionally good (galaxy.writes.unconfirmed meter).

Consumer contract (already shipped OtOpcUa-side): statuses.Count > 0 → map statuses[0] through the MX status map to a real OPC UA StatusCode; empty → provisional Good + unconfirmed meter. The statuses field already exists on MxCommandReply (field 7); no proto shape change is needed.

Approaches Considered

1. Executor pump-wait + versioned completion cache (chosen). After the WriteSecured COM call, the executor holds the STA thread but explicitly pumps Windows messages each poll iteration until the matching completion is recorded or a bounded deadline passes. This is exactly the shipped ReadBulk pattern (MxAccessValueCache.TryWaitForUpdate + pumpStep → StaRuntime.PumpPendingMessages()), so it adds no new threading model.
2. Parked asynchronous reply. Executor returns a "pending" marker; the dispatcher parks the correlation and the pipe reply is written later from the event dispatch. Rejected: changes the IStaCommandExecutor/dispatcher/pipe contracts for no real gain — commands serialize per session anyway, so freeing the STA during the wait buys nothing.
3. Gateway-side correlation. Gateway watches the session event stream for the OnWriteComplete after the worker reply. Rejected: races the event drain cadence, couples the gateway to event semantics, still holds the unary RPC, and spreads the feature across two processes.

Design

All changes are worker-side (ZB.MOM.WW.MxGateway.Worker, net48 x86). The gateway's Invoke already forwards the worker MxCommandReply (statuses included) verbatim.

New: MxAccessWriteCompletionCache

Mirror of MxAccessValueCache, keyed by (serverHandle, itemHandle) (packed long), one entry per key holding the most recent completion's RepeatedField<MxStatusProxy> (cloned) plus a monotonically increasing per-key Version. API:

• Record(int serverHandle, int itemHandle, RepeatedField<MxStatusProxy> statuses)
• ulong CurrentVersion(int serverHandle, int itemHandle) — 0 when absent
• bool TryWaitForCompletion(int serverHandle, int itemHandle, ulong sinceVersion, DateTime deadlineUtc, Action pumpStep, out RepeatedField<MxStatusProxy> statuses, int pollIntervalMs = 5) — pump/poll loop identical in shape to MxAccessValueCache.TryWaitForUpdate.

Same locking posture as the value cache: everything runs on the STA thread; a sync root keeps it nominally thread-safe for tests.

Sink: record completions

MxAccessBaseEventSink owns a MxAccessWriteCompletionCache (new optional ctor param, exposed as a property) and its OnWriteComplete handler records into it via the existing EnqueueEvent post-publish hook (same pattern as the value cache on OnDataChange): the streamed MxEvent is built exactly once by the mapper, enqueued unchanged for the event stream, and its Statuses are then recorded into the cache. The event stream is not altered — nothing is swallowed or synthesized.

A new seam interface IWriteCompletionCacheProvider { MxAccessWriteCompletionCache WriteCompletionCache { get; } } is implemented by MxAccessBaseEventSink and by test sinks. MxAccessSession.Create pulls the cache from the sink through that interface (fallback: fresh instance), mirroring the existing ValueCache sharing, and exposes it on the session.

Executor: bounded pump-wait

In ExecuteWriteSecured / ExecuteWriteSecured2:

1. Capture baseline = cache.CurrentVersion(serverHandle, itemHandle) before the COM call — this closes the fast-completion ordering edge: a callback that dispatches during or immediately after the COM call bumps the version past the baseline and still correlates.
2. Call session.WriteSecured(...) as today.
3. cache.TryWaitForCompletion(..., baseline, deadline, pumpStep, out statuses); on success, reply.Statuses.Add(statuses); on timeout, return the reply exactly as today (protocol OK, empty statuses) — the consumer's honest-unconfirmed path. No invented failure rows.

The reply's ProtocolStatus/Hresult stay untouched by the completion outcome: the command was accepted; the MX outcome (success or failure) is carried only in statuses[0]. That preserves MXAccess parity (the native API returns void; the outcome exists only in the callback) while enriching the reply with information that was already on the wire.

Timeout default: 1.5 s (MxAccessCommandExecutor.DefaultWriteCompletionTimeout). The consumer-side budget drives this: OtOpcUa wraps the driver write in a Tier A resilience policy with a 2 s timeout and a 5-failure breaker, so a worker wait longer than 2 s would convert slow-but-successful commits into consumer-side false failures (node revert + breaker pressure). 1.5 s covers the common fast-commit case and degrades a slow commit to the honest-unconfirmed path instead. It also sits well under the gateway's 30 s DefaultCommandTimeoutSeconds IPC wait and the STA watchdog's 75 s dispatched-command ceiling.

The wait is deployment-configurable end to end, following the existing pipe-connect-timeout pattern: a new gateway option MxGateway:Worker:WriteCompletionWaitMilliseconds (default 1500, validated >= 0; 0 disables the wait and restores pure fire-and-forget replies) is exported by WorkerProcessLauncher as the MXGATEWAY_WORKER_WRITE_COMPLETION_WAIT_MS environment variable, which the worker reads at session construction. A deployment that raises the gateway wait must raise the OtOpcUa driver's Write ResilienceConfig timeout in step (per-instance operator config on the OtOpcUa side) — documented in docs/GatewayConfiguration.md. MxAccessStaSession additionally gets an internal WriteCompletionTimeout seam (read when it constructs the executor in StartAsync) so tests can shorten it without env plumbing.

Client cancellation mid-wait needs no new code path, only verification: when the caller cancels the unary RPC, the gateway abandons its IPC reply wait, but the worker command is already in flight — CancelCommand only dequeues queued commands. The executor simply finishes its bounded wait and replies; the gateway discards the reply. The session is never faulted and the OnWriteComplete event still flows on the stream.

Scope

• In: WriteSecured, WriteSecured2 (single-item secured writes — inherently low-rate operator actions, and exactly the OtOpcUa single-write contract). Default-on.
• Out: plain Write/Write2 and all bulk write commands stay fire-and-forget — waiting would add a device round-trip of latency to high-rate supervisory write loops.
• Correlation fidelity is best-effort: the MXAccess callback carries only (hItem, statuses) — no transaction id — so a concurrent write to the same item within the wait window can be attributed to the wrong writer (worst case two writes to the same item swap status rows — benign for the serialized single-write consumer contract). Documented on the proto field.

Error handling

• Completion never arrives (device down): bounded 1.5 s wait, then today's reply shape.
• Event queue overflow during completion: the queue records a fault and the fail-fast design tears the session down; the post-publish hook not firing in that case is moot.
• COM call throws: unchanged — the dispatcher's existing exception path replies with the native HResult; no wait is entered.

Testing (Worker.Tests, x86 — verified on windev)

• MxAccessWriteCompletionCacheTests: record/version monotonicity, wait success, deadline expiry, baseline-before-record fast-completion ordering.
• MxAccessBaseEventSinkTests: OnWriteComplete both enqueues the event and records the completion; cache instance is the sink-bound one.
• MxAccessCommandExecutorTests (via MxAccessStaSession.DispatchAsync + fake COM object + test sink implementing IWriteCompletionCacheProvider):
  • completion recorded synchronously inside the fake's WriteSecured (fast edge) → reply carries statuses[0];
  • completion recorded from the test thread while the executor pump-waits → reply carries statuses[0];
  • no completion + shortened timeout → protocol OK, empty statuses;
  • WriteSecured2 mirrors; plain Write does not wait.
• Gateway tests (macOS-runnable): GatewayOptionsValidator accepts >= 0 and rejects negative WriteCompletionWaitMilliseconds; WorkerProcessLauncher exports MXGATEWAY_WORKER_WRITE_COMPLETION_WAIT_MS from the option (mirroring the existing pipe-connect-timeout env assertion).

Docs in the same change

• mxaccess_gateway.proto: comment on MxCommandReply.statuses and the WriteSecuredCommand/WriteSecured2Command messages describing the correlated completion contract (populated within the wait window; empty = unconfirmed; best-effort correlation). Comment-only → wire-identical; regenerate + commit Contracts/Generated (required for the C# build); other clients' generated code is functionally unchanged.
• docs/GatewayConfiguration.md: the new MxGateway:Worker:WriteCompletionWaitMilliseconds option, including the pairing rule with the OtOpcUa driver's Write resilience timeout.
• gateway.md command/event surface note; docs/DesignDecisions.md entry.