mxaccessgw/archreview/remediation/30-contracts-ipc.md

Contracts & IPC Protocol — Remediation Design & Implementation

Source review: 30-contracts-ipc.md · Generated: 2026-07-09

The frame protocol and proto evolution hygiene are sound; the remediable risk clusters in three seams: an unenforced codegen/descriptor freshness gate, an un-negotiated size/backpressure topology that converts legitimate large payloads into whole-session death, and contract-boundary documentation drift. All path:line citations below were re-verified against the working tree; corrections are noted inline where the review's lines had shifted.

Finding index

ID	Sev	Title	Roadmap	Effort	Depends on	Files
IPC-01	High	Published client descriptor set is 7 weeks stale, nothing enforces freshness	P1	M	—	clients/proto/descriptors/mxaccessgw-client-v1.protoset, scripts/publish-client-proto-inputs.ps1, src/ZB.MOM.WW.MxGateway.Tests/Contracts/ClientProtoInputTests.cs
IPC-02	Medium	Worker max frame size hard-coded, cannot follow gateway config	P1	M	IPC-03	src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerFrameProtocolOptions.cs, Contracts/Protos/mxaccess_worker.proto
IPC-03	Medium	gRPC max == pipe max with zero headroom; oversized write faults whole session	P1	M	IPC-02	src/ZB.MOM.WW.MxGateway.Server/Configuration/ProtocolOptions.cs, Workers/WorkerClient.cs, Workers/WorkerFrameWriter.cs
IPC-04	Medium	DrainEvents max_events=0 packs entire queue into one reply frame	P1	S	IPC-03	src/ZB.MOM.WW.MxGateway.Worker/MxAccess/MxAccessEventQueue.cs, Grpc/MxAccessGrpcRequestValidator.cs
IPC-05	Medium	Redundant deep copies on command and event hot paths	P2	S	—	src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerClient.cs, Grpc/MxAccessGrpcMapper.cs
IPC-06	Medium	gateway.md Worker Envelope sketch no longer matches the contract	P2	S	—	gateway.md
IPC-07	Medium	docs/Grpc.md says six RPCs; there are seven	P2	S	—	docs/Grpc.md
IPC-08	Medium	WorkerCancel defined and handled but never sent	—	M	—	src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerClient.cs, Worker/Ipc/WorkerPipeSession.cs, Contracts/Protos/mxaccess_worker.proto
IPC-09	Medium	Known codegen fragilities have no in-repo guards	P1	M	IPC-01	clients/python/pyproject.toml, clients/java/.../build.gradle, clients/*/generate-proto.ps1, scripts/publish-client-proto-inputs.ps1
IPC-10	Low	Envelope sequence is write-only; monotonicity never validated	—	S	—	src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerEnvelopeValidator.cs, Worker/Ipc/WorkerEnvelopeValidator.cs, gateway.md
IPC-11	Low	No protocol version negotiation despite supported_protocol_version name	—	S	—	Contracts/Protos/mxaccess_worker.proto, Worker/Ipc/WorkerPipeSession.cs
IPC-12	Low	Gateway frame writer has no write lock; integrity rests on an undocumented invariant	—	S	—	src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerFrameWriter.cs
IPC-13	Low	Gateway writer serializes twice and issues two stream writes	P2	S	IPC-05	src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerFrameWriter.cs
IPC-14	Low	Gateway reader allocates a fresh array per frame; worker rents from ArrayPool	P2	S	IPC-05	src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerFrameReader.cs
IPC-15	Low	Worker event delivery is a 25 ms poll with per-event write+flush, no batching	P2	M	—	src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerPipeSession.cs, Contracts/Protos/mxaccess_worker.proto
IPC-16	Low	Correlation id carried twice per reply (Info)	—	S	—	Contracts/Protos/mxaccess_worker.proto, mxaccess_gateway.proto
IPC-17	Low	Two docs name the wrong Python generated-output directory	P2	S	—	docs/ClientProtoGeneration.md, CLAUDE.md
IPC-18	Low	Generated-code hygiene verified good — preserve under change (positive)	—	S	IPC-01	src/ZB.MOM.WW.MxGateway.Contracts/Generated/
IPC-19	Low	Generated/-must-be-committed rule for net48 is undocumented and unguarded	P1	S	IPC-01	docs/Contracts.md, ZB.MOM.WW.MxGateway.Contracts.csproj
IPC-20	Low	Descriptor -Check byte-compares source-info-bearing bytes; protoc-version-sensitive	P1	S	IPC-01	scripts/publish-client-proto-inputs.ps1
IPC-21	Low	gateway.md presents unimplemented Session RPC inside the live service block	P2	S	IPC-06	gateway.md, Contracts/Protos/mxaccess_gateway.proto
IPC-22	Low	Public error-detail model stops at status codes plus prose (Info)	—	S	—	Contracts/Protos/mxaccess_gateway.proto, docs/Grpc.md

---

IPC-01 — Published client descriptor set is 7 weeks stale and nothing enforces freshness High · P1

Finding. clients/proto/descriptors/mxaccessgw-client-v1.protoset was last committed 2026-04-30 (git log verified: commit 0f88a95), while mxaccess_gateway.proto changed through 2026-06-18 (MxSparseArray, commit 8ac9a33), 2026-06-16 (ReplayGap), and 2026-06-15 (alarm provenance). strings over the protoset returns zero hits for MxSparseArray, replay_gap, or provider_status — confirmed stale. docs/Contracts.md:127 mandates regenerating the descriptor after any proto change, scripts/publish-client-proto-inputs.ps1 has a -Check mode (verified at line 3, 70, 88), but no CI workflow exists and ClientProtoInputTests.cs validates only manifest versions and path existence (verified lines 12-34: Manifest_DeclaresCurrentProtocolVersionsAndExistingInputs checks schemaVersion, protocol versions, File.Exists, Directory.Exists — never descriptor content).

Impact. The artifact documented as the "stable client input" (docs/ClientProtoGeneration.md:48) silently misrepresents the contract. Any consumer that prefers a descriptor input generates against a schema missing three shipped features; grpcurl users on the reflection-disabled deployments (per the deployment memory, gRPC reflection is off on both hosts) get a stale schema. This is the domain's single worst concrete defect and appears in the overall roadmap under P1 item 7.

Design. Two parts: (1) regenerate and commit the descriptor now; (2) add an automated freshness gate so the doc-mandated regeneration cannot be skipped silently again. The gate belongs in the gateway test project (ClientProtoInputTests) rather than only in the -Check script, because the test runs in the NonWindows build that the P1 CI will exercise, and it does not depend on protoc being on the runner. The most robust check compares the descriptor's own descriptor set against a set rebuilt from the current .proto sources at test time — but that reintroduces the protoc dependency and the byte-sensitivity of IPC-20. The lighter, sufficient check: reflect over the in-process MxaccessGatewayReflection.Descriptor / MxaccessWorkerReflection.Descriptor (already compiled into Contracts) and assert every message and field name present in the live descriptor also appears in the committed protoset's FileDescriptorSet (parsed with FileDescriptorSet.Parser). This catches "protoset missing a symbol the contract has" — exactly the stale-descriptor failure mode — without invoking protoc or being sensitive to source-info bytes. Co-designed with IPC-19 (same freshness-drift class for Generated/) and IPC-20 (protoc pinning for the script path).

Implementation.

• Regenerate: run scripts/publish-client-proto-inputs.ps1 (no -Check) on a box with the pinned protoc (IPC-20), commit the refreshed .protoset.
• Extend src/ZB.MOM.WW.MxGateway.Tests/Contracts/ClientProtoInputTests.cs: add Descriptor_ContainsEveryContractMessageAndField. Load the protoset bytes, FileDescriptorSet.Parser.ParseFrom, build the set of {file}.{message}.{field} names; enumerate MxaccessGatewayReflection.Descriptor.MessageTypes (recursively for nested types) and MxaccessWorkerReflection.Descriptor; assert each contract symbol is present in the protoset set. Fail with the missing symbol name.
• Wire the same assertion into the P1 CI job (see IPC-09) as a redundant guard.
• Config/proto surface: none changes.
• Docs: note in docs/ClientProtoGeneration.md that the freshness test guards the descriptor and that a red test means "regenerate and commit the protoset."

Verification. dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter FullyQualifiedName~ClientProtoInputTests (runs on macOS NonWindows tree). After regenerating, confirm strings clients/proto/descriptors/mxaccessgw-client-v1.protoset | grep MxSparseArray returns hits.

---

IPC-02 — Worker max frame size is hard-coded and cannot follow the gateway's configured limit Medium · P1

Finding. WorkerFrameProtocolOptions' WorkerOptions ctor always passes DefaultMaxMessageBytes = 16 MiB (verified src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerFrameProtocolOptions.cs:15-21). The gateway-side MxGateway:Worker:MaxMessageBytes is configurable (default 16 MiB, Server/Configuration/WorkerOptions.cs:37; validated 1 KiB–256 MiB per the review's GatewayOptionsValidator reference). Neither GatewayHello (mxaccess_worker.proto:41-45) nor the worker launch arguments carry the value, so the two limits are set independently and never reconciled.

Impact. An operator who raises the gateway limit above 16 MiB for large-array workloads gets a gateway that emits frames the worker's reader rejects with MessageTooLarge (Worker/Ipc/WorkerFrameReader.cs:44-48), faulting the session; the worker still cannot emit anything above 16 MiB in the return direction. The config appears to work until the first large frame. Part of the P1 backpressure-topology pass (roadmap item 8).

Design. Convey the negotiated max frame size in the handshake and fail fast on disagreement, rather than failing mid-traffic. Add uint32 max_frame_bytes = 4; to GatewayHello (additive, next free tag). The worker reads it during handshake and constructs its WorkerFrameProtocolOptions.MaxMessageBytes from the negotiated value instead of the hard-coded default; if the value is 0 (older gateway) it falls back to DefaultMaxMessageBytes. Because the worker and gateway are lockstep-deployed (one-worker-per-session invariant; gateway launches the worker), a launch-argument would also work, but the GatewayHello field keeps the value on the wire where an alternate-language worker (contemplated in gateway.md) can read it. Reject at handshake if the value exceeds a sane worker ceiling. This is a proto change, so it triggers the regen-and-commit rule (IPC-19) and the descriptor refresh (IPC-01). Co-designed with IPC-03; the negotiated value must sit above the public gRPC cap plus envelope headroom.

Implementation.

• Contracts/Protos/mxaccess_worker.proto: add uint32 max_frame_bytes = 4; to GatewayHello. Regenerate Contracts/Generated/ and commit (net48 rule).
• Gateway: populate the field when building GatewayHello (in the worker-client handshake path) from WorkerOptions.MaxMessageBytes.
• Worker: in WorkerPipeSession handshake (around the version check at :215), pass the received max_frame_bytes into the WorkerFrameProtocolOptions construction, replacing the DefaultMaxMessageBytes argument in the WorkerOptions ctor overload. Keep the > 0 guard already in the all-parameters ctor.
• Tests: WorkerFrameProtocolTests / handshake tests — assert the worker adopts the gateway's advertised limit and that a frame between the old (16 MiB) and new limit now succeeds; a fake-worker gateway test asserting handshake carries the value.
• Docs: docs/WorkerFrameProtocol.md and docs/GatewayConfiguration.md — document that MaxMessageBytes is negotiated to the worker via GatewayHello.

Verification. Contracts changed → regenerate, then dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx; dotnet build src/ZB.MOM.WW.MxGateway.Worker -p:Platform=x86 and dotnet test ...Worker.Tests -p:Platform=x86 (Windows host); dotnet test ...Tests --filter FullyQualifiedName~Handshake.

---

IPC-03 — Public gRPC max message size equals the pipe max with zero headroom; oversized outbound frame faults the whole session Medium · P1

Finding. MaxGrpcMessageBytes defaults to 16 MiB (Server/Configuration/ProtocolOptions.cs:16) and the pipe MaxMessageBytes defaults to the same 16 MiB (WorkerOptions.cs:37). A gRPC-accepted Invoke payload near the limit gains WorkerCommand + WorkerEnvelope overhead (session id, sequence, correlation id, enqueue timestamp, oneof tags) before hitting WorkerFrameWriter.cs:49-54, whose WorkerFrameProtocolException escapes the write loop at WorkerClient.cs:344-350 and calls SetFaulted, which kills the worker process (verified WorkerClient.cs:722-749: SetFaulted → KillOwnedProcess → CompletePendingCommands).

Impact. One legitimate, gateway-accepted oversized write tears down the session, all its subscriptions, and its event stream — instead of failing that single command. Roadmap item 8.

Design. Two coordinated changes. (1) Headroom: keep MaxGrpcMessageBytes strictly below the negotiated pipe max (IPC-02) by an explicit envelope-overhead margin. Rather than a second magic constant, derive the accepted public payload cap as pipeMax − EnvelopeOverheadReserve (a named constant, e.g. 64 KiB, sufficient for the fixed envelope fields plus timestamp/oneof framing), and validate at startup (GatewayOptionsValidator) that MaxGrpcMessageBytes ≤ pipeMax − reserve, failing config otherwise. (2) Per-command failure: at the enqueue/write boundary, catch WorkerFrameProtocolException with ErrorCode == MessageTooLarge and fail only the offending correlation id (complete its PendingCommand with a WorkerClientErrorCode → ResourceExhausted/InvalidArgument gRPC status) instead of letting it reach SetFaulted. Because all writes funnel through the single WriteLoopAsync (WorkerClient.cs:332-339), the cleanest seam is to pre-check envelope.CalculateSize() against MaxMessageBytes in CreateCommandEnvelope/enqueue and reject there synchronously in InvokeAsync, before the frame ever enters the outbound channel — this keeps the write loop's remaining exceptions genuinely fatal (a mid-frame MessageTooLarge there really is a desync). Co-designed with IPC-02 and IPC-04 (the three form the size/backpressure pass). Does not touch MXAccess parity — this is a transport cap, not a COM behavior.

Implementation.

• Server/Configuration/ProtocolOptions.cs / GatewayOptionsValidator: add the headroom invariant; document EnvelopeOverheadReserve.
• Server/Workers/WorkerClient.cs: in InvokeAsync before EnqueueAsync, size-check the built envelope; on overshoot, throw WorkerClientException(PayloadTooLarge, ...) mapped to gRPC ResourceExhausted in MxAccessGatewayService — do not enqueue, do not fault. Add WorkerClientErrorCode.PayloadTooLarge.
• Keep the write-loop MessageTooLarge → SetFaulted path for genuine desync (now unreachable for legitimate command payloads).
• Tests: WorkerClientTests — an over-cap Invoke fails that one command with the mapped status and leaves the session Ready and other pending commands intact (assert worker not killed). GatewayOptionsValidatorTests — headroom invariant rejects MaxGrpcMessageBytes ≥ pipeMax.
• Docs: docs/GatewayConfiguration.md (headroom rule), docs/Grpc.md (Invoke now returns ResourceExhausted for oversized payloads).

Verification. dotnet build src/ZB.MOM.WW.MxGateway.Server; dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~WorkerClientTests|FullyQualifiedName~GatewayOptionsValidatorTests".

---

IPC-04 — DrainEvents with max_events = 0 packs the entire event queue into one reply envelope Medium · P1

Finding. MxAccessEventQueue treats 0 as drain-all (review cite Worker/MxAccess/MxAccessEventQueue.cs:172-192); the reply is a single DrainEventsReply written as one frame (Worker/Ipc/WorkerPipeSession.cs control-reply path). MxAccessGrpcRequestValidator imposes no cap on DrainEventsCommand.max_events.

Impact. A deep queue of large array events produces a reply exceeding MaxMessageBytes; the control-reply write throws, the exception propagates out of the message loop, and the worker exits — a diagnostics command kills the session. Roadmap item 8.

Design. Cap the effective drain worker-side rather than chunking (chunking DrainEvents needs a streaming reply arm the control-command path doesn't have; a bounded single reply is simpler and sufficient for a diagnostics RPC). Introduce a MaxDrainEventsPerReply worker constant (e.g. WorkerPipeSession alongside the existing EventDrainInterval/batch-size constants at :17-19), and interpret max_events = 0 as "up to MaxDrainEventsPerReply", never unbounded. Optionally surface a has_more/remaining field on DrainEventsReply (additive) so a caller can drain iteratively. Also add an upper-bound validation in MxAccessGrpcRequestValidator for the public DrainEventsCommand.max_events. This keeps parity untouched (drain is a gateway diagnostics feature, not an MXAccess behavior). Co-designed with IPC-03 (both prevent one accepted request from producing a session-killing frame).

Implementation.

• Worker/Ipc/WorkerPipeSession.cs: clamp the requested count to MaxDrainEventsPerReply; treat 0 as the clamp value.
• (Optional, additive) Contracts/Protos/mxaccess_gateway.proto: add uint32 remaining = N; to DrainEventsReply — regen + commit if taken.
• Server/Grpc/MxAccessGrpcRequestValidator: reject max_events above the documented ceiling with InvalidArgument.
• Tests: MxAccessEventQueueTests / worker session tests — a queue larger than the cap drains at most MaxDrainEventsPerReply per reply and never builds an over-MaxMessageBytes frame; validator test for the public cap.
• Docs: docs/Grpc.md DrainEvents rules row; docs/WorkerFrameProtocol.md if the reply gains remaining.

Verification. dotnet build src/ZB.MOM.WW.MxGateway.Worker -p:Platform=x86 + dotnet test ...Worker.Tests -p:Platform=x86 --filter FullyQualifiedName~MxAccessEventQueue; gateway validator test on macOS.

---

IPC-05 — Redundant deep copies on the command and event hot paths Medium · P2

Finding. MxAccessGrpcMapper.MapCommand clones the inbound MxCommand, then WorkerClient.CreateCommandEnvelope clones the whole WorkerCommand a second time (verified WorkerClient.cs:902, envelope.WorkerCommand = command.Clone()); MapEvent deep-clones every worker event (verified MxAccessGrpcMapper.cs:68, workerEvent.Event?.Clone()). So each Invoke materializes the command graph three times (gRPC parse, mapper clone, worker-client clone); each event is parsed once and cloned once.

Impact. Measurable allocation pressure on array-heavy OnDataChange streams — the exact load the gateway exists to handle (overall theme 7). Not a correctness bug.

Design. Drop the second Clone() in CreateCommandEnvelope: the mapper's clone already isolates the graph from the caller-owned gRPC message, and the envelope is built and owned entirely inside WorkerClient, so no aliasing hazard remains once the envelope is enqueued. For MapEvent, evaluate ownership transfer: the WorkerEvent is parsed fresh from the pipe frame in the read loop and is not retained after mapping, so the mapper can move workerEvent.Event into the outbound MxEvent graph rather than cloning — but confirm no code path re-reads workerEvent after MapEvent (the fan-out distributor may share one MxEvent across subscribers, which is read-only and safe). Recommend dropping the command clone unconditionally (clear win, no aliasing) and gating the event ownership-transfer on a quick audit of the fan-out path (coordinate with the gateway-core distributor findings). Parity-neutral. Co-designed with IPC-13/IPC-14 as the frame-I/O + allocation pass (roadmap item 14).

Implementation.

• Server/Workers/WorkerClient.cs:902: envelope.WorkerCommand = command; (remove .Clone()), with a comment that the mapper already isolated the graph.
• Server/Grpc/MxAccessGrpcMapper.cs:68: keep the clone only if the audit shows the event is aliased downstream; otherwise transfer ownership.
• Tests: existing WorkerClientTests/mapper tests must still pass (no behavior change); add an assertion that the enqueued envelope's command is reference-equal to the mapper output where the clone was removed, to lock the intent.
• Docs: docs/Grpc.md:197-211 (the clone-count description) must be corrected to match the reduced copies.

Verification. dotnet build src/ZB.MOM.WW.MxGateway.Server; dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~WorkerClient|FullyQualifiedName~MxAccessGrpcMapper".

---

IPC-06 — gateway.md's Worker Envelope section no longer matches the shipped contract Medium · P2

Finding. Verified gateway.md:291-309 shows uint64 correlation_id = 4 and body cases command = 20; command_reply = 21; event = 22; heartbeat = 23; cancel = 24; shutdown = 25; fault = 26. The actual contract (mxaccess_worker.proto:20-39) is string correlation_id = 4 and worker_command = 13; worker_command_reply = 14; worker_cancel = 15; worker_shutdown = 16; worker_shutdown_ack = 17; worker_event = 18; worker_heartbeat = 19; worker_fault = 20. WorkerShutdownAck is absent from the doc sketch entirely.

Impact. The top-level architecture doc's field numbers and types are wrong; anyone implementing an alternate worker (the doc contemplates a C++ worker) from this section produces an incompatible peer. Violates the repo's docs-change-with-source rule. Roadmap item 15.

Design. Replace the hand-maintained protobuf sketch with either the verbatim current message or, preferably, a short prose description plus a pointer to mxaccess_worker.proto as the single source of truth — hand-copied proto in docs is exactly what drifted. Keep the "Rules" list but reconcile it (correlation_id is a string; note WorkerShutdownAck and WorkerReady). Do this in the same sweep as IPC-07, IPC-17, IPC-21, IPC-22.

Implementation.

• gateway.md:288-320: replace the code block with the real oneof (or a pointer), add WorkerReady/WorkerShutdownAck, fix correlation_id type.
• No source/proto/config change.

Verification. Doc-only; no build. Cross-check the doc block against mxaccess_worker.proto:20-39 after editing.

---

IPC-07 — docs/Grpc.md says the service has six RPCs; it has seven Medium · P2

Finding. Verified docs/Grpc.md:13 and :32 say "six in total" and omit QueryActiveAlarms. The proto declares seven RPCs (mxaccess_gateway.proto:18-37: OpenSession, CloseSession, Invoke, StreamEvents, AcknowledgeAlarm, StreamAlarms, QueryActiveAlarms); QueryActiveAlarms is implemented in MxAccessGatewayService.

Impact. The authoritative gRPC-layer doc undercounts the public surface and documents no validation/handler behavior for the missing RPC. Note the cross-domain link: the security review found QueryActiveAlarms also missing from the scope resolver (SEC domain) — the doc omission masked that gap. Roadmap item 15.

Design. Update the count (six → seven) at both cites, add QueryActiveAlarms to the collaborators/RPC table, and add a handler section describing its validation, streaming behavior, and required scope. Flag to the security-domain remediation that the same RPC needs a scope-resolver entry.

Implementation.

• docs/Grpc.md:13,32: fix count and RPC list; add QueryActiveAlarms row and handler subsection.
• No source change in this domain (scope resolver fix is SEC).

Verification. Doc-only.

---

IPC-08 — WorkerCancel is defined and handled but never sent Medium · —

Finding. WorkerCancel is defined (mxaccess_worker.proto:71-73) and dispatched by the worker to _runtimeSession.CancelCommand (verified Worker/Ipc/WorkerPipeSession.cs:399-401), but no gateway code sends it — WorkerClient handles timeout/cancel purely by abandoning the pending correlation (verified WorkerClient.cs:195-213: on timeout it calls RemovePendingCommandAsFailed and throws, never emitting a WorkerCancel).

Impact. The cancellation contract in gateway.md:713-719 ("the worker should finish the COM call and discard or log the late reply if the correlation was canceled") is half-implemented: the worker never learns a correlation was canceled, so it always writes the late reply (which the gateway then drops at WorkerClient.cs:565-571), and the worker-side discard logic behind CancelCommand is unreachable. Not in the roadmap.

Design. Decide between two coherent end states; recommend wiring it up because the worker handling already exists and the late-reply-drop path is real work being wasted. On timeout or caller-cancel in InvokeAsync, enqueue a WorkerCancel envelope carrying the correlation id (add string correlation_id-based routing — the envelope already has CorrelationId, and the worker's CancelCommand(envelope.CorrelationId) already reads it, so no proto change is needed; WorkerCancel.reason is the human note). The worker then discards the late reply instead of writing it, saving a frame and a pointless gateway-side drop. Keep the abandon-the-pending-correlation behavior as the gateway's own timeout resolution — WorkerCancel is best-effort cleanup, not a synchronous ack. The alternative (mark the arm reserved-for-future and delete the worker handling) loses working behavior and still needs a gateway.md edit; reject it. This does not synthesize events and does not alter MXAccess parity — the COM call still completes; only the late reply is suppressed. Because the worker must keep pumping the STA, CancelCommand must remain a non-blocking mark-and-discard (confirm it is).

Implementation.

• Server/Workers/WorkerClient.cs: add a CreateCancelEnvelope(correlationId, reason) (mirrors CreateShutdownEnvelope) and enqueue it in the timeout and cancellation-token branches of InvokeAsync (:195-213) after removing the pending command. Best-effort: swallow enqueue failures if the session is already terminal.
• Worker: confirm MxAccessSession.CancelCommand marks the correlation for late-reply discard and that WorkerPipeSession's reply-write path checks that mark (it already drops replies once state leaves Ready at :604-608; extend to "or correlation was canceled").
• Tests: WorkerClientTests — a timed-out Invoke enqueues a WorkerCancel with the right correlation id; worker session test — a canceled correlation suppresses the late reply.
• Docs: gateway.md:713-719 — mark the cancellation path as implemented; docs/WorkerFrameProtocol.md — document WorkerCancel semantics.

Verification. dotnet build src/ZB.MOM.WW.MxGateway.Server + dotnet test ...Tests --filter FullyQualifiedName~WorkerClient; worker side -p:Platform=x86.

---

IPC-09 — Known codegen fragilities have no in-repo guards Medium · P1

Finding. (a) Python — clients/python/pyproject.toml:15,41 allow grpcio>=1.80,<2 and grpcio-tools>=1.80,<2; clients/python/generate-proto.ps1 does no version check, so regenerating with any newer 1.x stamps a GRPC_GENERATED_VERSION above the pinned runtime and breaks pytest (a previously-hit failure per the memory). (b) Java — clients/java/zb-mom-ww-mxgateway-client/build.gradle:50 points generatedFilesBaseDir at the tracked src/main/generated, so every gradle build rewrites the tracked output with protobuf-version churn and no task detects spurious diffs. (c) Portability — clients/python/generate-proto.ps1:7 and clients/go/generate-proto.ps1:8-9 hard-code C:\Users\dohertj2\... tool paths (verified), making generation single-machine and Windows-only.

Impact. Contract evolution safety depends on operator memory. A regeneration on a different machine or tool version silently produces incompatible or noisy output. Roadmap item 7 (CI) and the "everything guarded by operator memory" theme.

Design. Three targeted guards, each fail-fast:

• Python: pin exact generator versions used for regeneration. Add a version assertion at the top of generate-proto.ps1 that reads grpcio-tools.__version__ and refuses to run if it differs from the pinned baseline (grpcio 1.80.0 / protobuf 6.31.1 per the memory). Keep the pyproject.toml runtime range but document that regeneration must use the exact pin.
• Java: add a checkGeneratedClean Gradle task (git-diff the src/main/generated tree after generation and fail on non-empty diff when no .proto changed) so the ~64k-line spurious churn is caught in review rather than committed. Alternatively, and preferred long-term, move generatedFilesBaseDir to $buildDir and stop tracking generated Java — but that is a larger change; the check task is the minimal guard.
• Portability: resolve protoc/python/plugins from PATH with a documented version assertion (the publish-client-proto-inputs.ps1 Resolve-Protoc at lines 15-25 already does PATH-first with a documented fallback — mirror that pattern in the per-client scripts) instead of absolute user paths.

Coordinate with IPC-01/IPC-20 (descriptor freshness + protoc pin) so the CI job that runs the freshness gate also runs the codegen-clean checks.

Implementation.

• clients/python/generate-proto.ps1: PATH-resolve python; assert grpcio-tools version equals the pin, else throw.
• clients/go/generate-proto.ps1: PATH-resolve protoc, protoc-gen-go, protoc-gen-go-grpc.
• clients/java/.../build.gradle: add checkGeneratedClean task; wire into CI (not into every local build).
• CI (new, per roadmap item 7): a job that regenerates and runs the clean/freshness checks.
• Docs: docs/ClientProtoGeneration.md — record the exact pinned generator versions and the check tasks; docs/Contracts.md — cross-link.

Verification. Per-client: python -m pytest from clients/python after a pinned regen; gradle checkGeneratedClean from clients/java on a Windows/JDK-17 host; go build ./... from clients/go. Note the Java build must run on windev (Mac has no JRE per the memory) via an isolated origin/<branch> worktree.

---

IPC-10 — Envelope sequence is write-only; monotonicity is never validated on receive Low · —

Finding. Sequences are assigned on send (verified WorkerClient.cs:934, Interlocked.Increment(ref _nextSequence); worker side at WorkerPipeSession.cs:1005-1018), but neither validator checks them (verified WorkerEnvelopeValidator.cs:15-39 checks only protocol version, session id, and non-empty body; the worker validator likewise). gateway.md:312 states "sequence is monotonic per sender" as a protocol rule.

Impact. Gap/duplication detection promised by the design is diagnostics-only; ordering integrity rests entirely on pipe FIFO semantics (fine for a local named pipe, but the rule is unenforced). Low.

Design. Choose the cheap enforcement or the honest annotation. Recommend annotate rather than enforce: on a local named pipe FIFO is guaranteed, and adding per-sender monotonicity tracking to the validators introduces state and a new fault mode for no real safety gain (a reordering would already be a kernel-level pipe bug). Mark sequence as diagnostic-only in the proto comment and soften gateway.md's "rule" to "diagnostic aid." If enforcement is later wanted, it belongs in the validators as a per-sender last-seen check, faulting on regression.

Implementation.

• Contracts/Protos/mxaccess_worker.proto:23: comment that sequence is a monotonic diagnostic counter, not validated on receive (regen + commit — comment-only proto changes still regenerate).
• gateway.md:312: reword the rule.

Verification. Regenerate contracts; dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx to confirm the comment-only regen is clean.

---

IPC-11 — No protocol version negotiation despite a field name that implies it Low · —

Finding. GatewayHello.supported_protocol_version (mxaccess_worker.proto:42) is a single value compared for strict equality (Worker/Ipc/WorkerPipeSession.cs:215-219); the worker also hard-pins its version to GatewayContractInfo.WorkerProtocolVersion (=1, verified GatewayContractInfo.cs:15) at options construction (WorkerFrameProtocolOptions.cs:65-70), and every envelope re-checks equality in both validators.

Impact. Acceptable for a lockstep-deployed pair (gateway.md:319 documents mismatch-fails-session), but the singular "supported" field cannot express a range; any future skewed upgrade needs new machinery. Low — no action required now.

Design. No change now. When the worker protocol first changes (bumps past 1), replace the single field with a min/max supported range in GatewayHello/WorkerHello and negotiate the highest common version rather than bumping the single constant. Record this as the intended evolution path in the proto comment so the next editor doesn't just increment the constant.

Implementation.

• Contracts/Protos/mxaccess_worker.proto: add a comment on supported_protocol_version describing the min/max-range migration when a second version appears (regen + commit).
• No code change.

Verification. Regenerate; NonWindows build clean.

---

IPC-12 — Gateway frame writer has no write lock; integrity rests on an undocumented single-writer invariant Low · —

Finding. Server/Workers/WorkerFrameWriter.cs has no synchronization (verified: no lock/semaphore; WriteAsync at :34-61 issues prefix and payload as two separate _stream.WriteAsync calls at :59-60). The worker's writer serializes with a SemaphoreSlim (verified Worker/Ipc/WorkerFrameWriter.cs:14,68-77) because its heartbeat, event-drain, and command tasks write concurrently. The gateway is safe only because all writes funnel through the single-reader outbound channel loop (WorkerClient.cs:332-339).

Impact. A future direct _writer.WriteAsync call outside the write loop interleaves the two stream writes and corrupts the frame stream unrecoverably. Latent, not live. Low.

Design. Defense in depth: either (a) collapse the two writes into one (adopt the worker's single-buffer approach — this is IPC-13, which also removes the interleaving surface entirely), or (b) add the same SemaphoreSlim write lock plus an XML-doc invariant note. Recommend doing IPC-13 (single write) and documenting the single-writer invariant on the class, which together make interleaving impossible even under a future stray caller. A bare assertion is weaker than a single atomic write.

Implementation.

• Fold into IPC-13's single-buffer rewrite; add a class-level <remarks> stating "all frames must be written through WorkerClient.WriteLoopAsync; the writer is not internally synchronized for concurrent callers" (or, if a lock is added, document that it is).
• Tests: covered by IPC-13's frame round-trip tests.

Verification. See IPC-13.

---

IPC-13 — Gateway writer serializes each envelope twice and issues two stream writes Low · P2

Finding. Verified Server/Workers/WorkerFrameWriter.cs:41,60: CalculateSize() then ToByteArray() (which re-runs size calculation internally) and separate prefix/payload writes at :59-60. The worker writer already builds one prefixed buffer with a single WriteTo(Span) and one write (verified Worker/Ipc/WorkerFrameWriter.cs:63-72, with a comment explaining exactly this).

Impact. Extra CPU pass and extra pipe write per frame on the higher-volume side (the gateway writes every command). Low; roadmap item 14.

Design. Back-port the worker's single-buffer implementation verbatim: allocate sizeof(uint) + payloadLength, write the little-endian length prefix, envelope.WriteTo(new Span<byte>(frame, 4, payloadLength)), one _stream.WriteAsync. This also closes IPC-12's interleaving surface. Keep the existing size/MessageTooLarge/empty-payload guards (already present at :41-54). Consider adding the FlushAsync the worker does — verify the gateway's underlying pipe stream doesn't already auto-flush before adding it.

Implementation.

• Server/Workers/WorkerFrameWriter.cs:56-60: replace the two-array/two-write block with the single-buffer/single-write block from Worker/Ipc/WorkerFrameWriter.cs:63-72.
• Tests: WorkerFrameWriterTests / frame round-trip tests must still pass (byte-identical wire output); add a test asserting a single write path if a mock stream is available.

Verification. dotnet build src/ZB.MOM.WW.MxGateway.Server; dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter FullyQualifiedName~WorkerFrame.

---

IPC-14 — Gateway reader allocates a fresh array per frame; the worker rents from ArrayPool Low · P2

Finding. Verified Server/Workers/WorkerFrameReader.cs:50: byte[] payload = new byte[payloadLength]; per frame. The worker rents/returns from ArrayPool<byte>.Shared with a comment that ParseFrom copies (verified Worker/Ipc/WorkerFrameReader.cs:55-77).

Impact. Large event frames (arrays near the cap) allocate LOH buffers per frame on the side that receives the entire event stream. Low; roadmap item 14.

Design. Mirror the pooled read: int length = checked((int)payloadLength); byte[] payload = ArrayPool<byte>.Shared.Rent(length); read exactly length, WorkerEnvelope.Parser.ParseFrom(payload, 0, length), Return in finally. The gateway reader already reads-exactly via ReadExactlyAsync; keep that. Note ParseFrom(payload, 0, length) is required (the rented buffer may be larger than length).

Implementation.

• Server/Workers/WorkerFrameReader.cs:50-56: adopt the rent/return pattern from the worker reader; use the length-bounded ParseFrom overload.
• Tests: WorkerFrameReaderTests round-trip must pass; add a large-frame test to exercise the pool path.

Verification. dotnet build src/ZB.MOM.WW.MxGateway.Server; dotnet test ...Tests --filter FullyQualifiedName~WorkerFrame.

---

IPC-15 — Worker event delivery is a 25 ms poll with per-event write+flush, no batching Low · P2

Finding. Worker/Ipc/WorkerPipeSession.cs:17-19 sets EventDrainInterval 25 ms and batch size 128; the drain loop writes each event with its own WriteAsync (each acquiring the writer lock and FlushAsync, verified Worker/Ipc/WorkerFrameWriter.cs:71-72). gateway.md:849-850 lists worker→gateway event batching as a planned optimization that doesn't exist — there is no multi-event envelope body.

Impact. Idle-to-active latency floor up to 25 ms per batch, plus per-event flush syscalls under burst. Acceptable for v1 parity; roadmap item 14.

Design. Do not tighten the poll (that trades latency for CPU wakeups). When event-rate targets firm up, add a batched event body as an additive oneof arm — a WorkerEventBatch { repeated MxEvent events = 1; } message and a worker_event_batch = N case in WorkerEnvelope — so the drain loop packs up to N events (bounded by MaxMessageBytes, coordinate with IPC-03/IPC-04) into one frame with one flush. The gateway read loop unpacks the batch and dispatches each event to the distributor. This preserves the "don't synthesize events" invariant (the worker still forwards only real events; batching is framing, not fabrication) and the one-worker-per-session invariant. Keep single-event worker_event for compatibility during rollout. This is deferred work, not a v1 fix — flagged so the batching claim in gateway.md is either implemented or re-marked as future.

Implementation.

• (When scheduled) Contracts/Protos/mxaccess_worker.proto: add WorkerEventBatch + worker_event_batch oneof arm (regen + commit + descriptor refresh).
• Worker/Ipc/WorkerPipeSession.cs: drain up to the batch size into one WorkerEventBatch frame under the frame-size cap.
• Server/Workers/WorkerClient.cs read dispatch: handle the batch arm.
• Now (doc-only, immediate): correct gateway.md:849-850 to mark event batching as not-yet-implemented.
• Tests: event round-trip under batch; ordering preserved.

Verification. Immediate: doc edit. When implemented: contracts regen + dotnet build NonWindows.slnx, worker -p:Platform=x86 tests, gateway fake-worker event tests.

---

IPC-16 — Correlation id is carried twice per reply Low (Info) · —

Finding. WorkerEnvelope.correlation_id (mxaccess_worker.proto:24) and MxCommandReply.correlation_id (review cite mxaccess_gateway.proto:520) both carry the value; CompleteCommand falls back from the envelope to the inner reply (verified WorkerClient.cs:559-562: uses envelope.CorrelationId, then envelope.WorkerCommandReply.Reply?.CorrelationId).

Impact. Two sources of truth for one value; harmless today, a divergence hazard for a future writer that sets one but not the other. Info.

Design. No wire change. Document that the envelope correlation_id is authoritative and the inner reply copy is the MXAccess-parity echo; the fallback in CompleteCommand exists only for defensive robustness. Add a proto comment on both fields naming the envelope as canonical.

Implementation.

• Contracts/Protos/mxaccess_worker.proto:24 and mxaccess_gateway.proto MxCommandReply.correlation_id: cross-referencing comments (regen + commit).

Verification. Regenerate; NonWindows build clean.

---

IPC-17 — Two docs name the wrong Python generated-output directory Low · P2

Finding. Verified docs/ClientProtoGeneration.md:80,145 say clients/python/src/mxgateway/generated; the manifest (clients/proto/proto-inputs.json:28) and the tree use clients/python/src/zb_mom_ww_mxgateway/generated, which is also what clients/python/generate-proto.ps1 writes. CLAUDE.md's generated-code bullet repeats the wrong path.

Impact. A follow-the-doc regeneration writes to a dead directory. Low; roadmap item 15.

Design. Fix both docs (and the CLAUDE.md bullet) to the manifest path clients/python/src/zb_mom_ww_mxgateway/generated. Trivial, no source change.

Implementation.

• docs/ClientProtoGeneration.md:80,145: correct the path.
• CLAUDE.md generated-code bullet: correct clients/python/src/mxgateway/generated → clients/python/src/zb_mom_ww_mxgateway/generated.

Verification. Doc-only; grep to confirm no src/mxgateway/generated remains.

---

IPC-18 — Generated-code hygiene verified good — preserve under change (positive) Low · —

Finding. No defect. Verified positive: Contracts/Generated/*.cs carry <auto-generated> headers and contain the newest symbols (MxSparseArray); all five client generated trees contain the sparse-array surface; galaxy_repository.proto is wire-identical to the GalaxyRepository package copy (only csharp_namespace differs), consistent with the retention comment in ZB.MOM.WW.MxGateway.Contracts.csproj:29-33 that keeps it as the language-client codegen source.

Impact. None — this is a property to preserve, not a fix. Recorded so it is protected under change.

Design. No remediation. The freshness gates in IPC-01 (descriptor) and IPC-19 (Generated/ net48 rule) are what keep this property true; the galaxy wire-identity should be asserted by a small test if the two copies ever diverge in practice (a byte-diff test comparing the two .proto files modulo csharp_namespace), but this is optional given the csproj comment already documents the invariant.

Implementation.

• Optional: a ContractsTests case diffing galaxy_repository.proto against the package source path modulo csharp_namespace. Not required.

Verification. N/A (no change).

---

IPC-19 — The Generated/-must-be-committed rule for net48 consumers is undocumented and unguarded Low · P1

Finding. Verified ZB.MOM.WW.MxGateway.Contracts.csproj:26-35 does Compile Remove="Generated\**\*.cs" + Protobuf ... OutputDir="Generated", regenerating tracked files on every build; the worker consumes contracts via ProjectReference. docs/Contracts.md:97-98 says only "do not hand-edit." The operational rule — a proto edit requires regenerating and committing Generated/, or the net48 worker build fails CS0246 on new types (confirmed by the repo memory project_proto_codegen_regen) — appears nowhere, and no test compares Generated/ to the protos.

Impact. Same silent-drift class as IPC-01: committed C# can lag the protos with nothing failing until a downstream net48 consumer breaks. Low; roadmap item 7.

Design. Document the rule and lean on the IPC-01 freshness test for enforcement. State in docs/Contracts.md (near the existing "do not hand-edit" line) that after any .proto edit you must regenerate Contracts/Generated/ and commit it, and why (net48 worker has no first-class regen-on-restore; a stale Generated/ fails CS0246). The IPC-01 reflection-based test (comparing MxaccessGatewayReflection.Descriptor against the committed protoset) already indirectly catches a proto edited without regenerating the descriptor; for Generated/ specifically, the .NET build itself compiles against the freshly regenerated output on the net10 side, so the practical guard is the net48 worker build in CI (IPC-09/roadmap item 7) plus the documented rule.

Implementation.

• docs/Contracts.md:97: add the regenerate-and-commit rule with the net48 rationale.
• Optionally reference the memory-known del Generated/*.cs force-regen trick in the doc.
• CI (roadmap item 7): the Windows net48 worker build job is the real enforcement.

Verification. Doc-only for the rule; the net48 build (dotnet build src/ZB.MOM.WW.MxGateway.Worker -p:Platform=x86 on Windows) is what proves a stale Generated/ would fail.

---

IPC-20 — Descriptor -Check byte-compares source-info-bearing bytes; protoc-version-sensitive Low · P1

Finding. Verified scripts/publish-client-proto-inputs.ps1: -Check (line 70) rebuilds the descriptor with --include_source_info (line 78) and Compare-FileBytes (line 36, 88) byte-compares. Source-info bytes differ across protoc releases even for identical schemas.

Impact. Once a check exists (IPC-01), a protoc upgrade produces a false "stale" failure or forces a churn commit unless protoc is pinned. Low; roadmap item 7.

Design. Two options, do both. (1) Pin protoc for descriptor generation: Resolve-Protoc (lines 15-25) already prefers PATH and documents a fallback; add a version assertion (protoc --version must equal the pinned toolchain in docs/ToolchainLinks.md, protoc 34.1 per CLAUDE.md) and fail fast on mismatch. (2) Make the in-process freshness test (IPC-01) the primary gate since it compares semantically (symbol presence) rather than byte-wise, sidestepping source-info sensitivity entirely; keep the script's byte--Check as a stricter secondary that runs only with the pinned protoc. Alternatively drop --include_source_info and compare descriptors without source info for a version-tolerant byte compare — but source info is useful for grpcurl UX, so prefer pin + semantic test.

Implementation.

• scripts/publish-client-proto-inputs.ps1: add a protoc-version assertion in Resolve-Protoc; document the pin.
• Rely on IPC-01's semantic test as the CI gate.
• Docs: docs/ClientProtoGeneration.md / docs/Contracts.md — record the pinned protoc version for descriptor generation.

Verification. Run scripts/publish-client-proto-inputs.ps1 -Check with the pinned protoc on Windows; the IPC-01 test on macOS.

---

IPC-21 — gateway.md presents the unimplemented Session RPC inside the live service block Low · P2

Finding. Verified gateway.md:328-345 sketches rpc Session(stream ClientMessage) returns (stream ServerMessage) inside the service MxAccessGateway block as "the best long-term shape," with a rollout plan whose step 3 is unimplemented; mxaccess_gateway.proto:18-37 has no such RPC.

Impact. Intentional phasing, not a defect — but presenting it inside the service definition (alongside the five real RPCs) compounds IPC-06's staleness and reads as shipped surface. Low; roadmap item 15.

Design. Move the Session RPC out of the live service block into a clearly-labeled "Future work / not yet implemented" subsection, keeping the rollout plan. Do this in the same doc sweep as IPC-06/IPC-07. No proto or code change.

Implementation.

• gateway.md:328-345: relabel; separate the bidi Session sketch from the shipped RPC list (which should also be reconciled to the actual seven RPCs — coordinate with IPC-07).

Verification. Doc-only.

---

IPC-22 — Public error-detail model stops at status codes plus prose Low (Info) · —

Finding. MxCommandReply preserves MXAccess parity detail well (hresult, statuses, diagnostic_message, review cite mxaccess_gateway.proto:518-529), but transport-level failures surface only as gRPC status codes with message strings; no google.rpc error details are attached, and AcknowledgeAlarmReply.status is a permanently-unset placeholder documented as such (verified mxaccess_gateway.proto:938-945: MxStatusProxy status = 5 with a comment that the by-name/by-GUID ack path produces only the int32 return code and the field is left UNSET; clients must read hresult/protocol_status).

Impact. Machine consumers must parse prose to distinguish sub-causes within a gRPC status code. Acceptable for the current in-repo client set. Info.

Design. No change now. If third-party clients appear, attach google.rpc.ErrorInfo (a stable reason enum + metadata) to transport failures so consumers branch on a machine-readable reason instead of message strings, and either populate or formally reserve AcknowledgeAlarmReply.status. The status placeholder is already honestly documented (parity-correct: the worker ack path genuinely produces only the int32), so leave it as-is unless the ack path grows a structured status. Recording the recommendation without acting keeps parity intact.

Implementation.

• None now. Future: add google.rpc.ErrorInfo details in MxAccessGatewayService exception mapping; document reason codes in docs/Grpc.md:217-241.

Verification. N/A (no change).