mxaccessgw/archreview/2026-07-12/30-contracts-ipc.md

Contracts & IPC Protocol — Architecture Review (re-run)

Date: 2026-07-12 · Commit: 4f5371f (main) · macOS static review (no build/test; sources read at HEAD) Prior review: ../30-contracts-ipc.md (2026-07-08, baseline 59856b8) · Remediation plan: ../remediation/30-contracts-ipc.md

Scope & method

Same domain as the prior review: src/ZB.MOM.WW.MxGateway.Contracts (three protos, Generated/ hygiene, net10.0;net48 multi-targeting), the frame protocol on both sides (src/ZB.MOM.WW.MxGateway.Worker/Ipc/*, src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerFrame* + WorkerClient), the codegen/descriptor pipeline (scripts/check-codegen.ps1, scripts/publish-client-proto-inputs.ps1, clients/proto/*, vendored client proto copies incl. clients/rust/protos), and proto-evolution hygiene. Method: static reading at HEAD, git diff 59856b8..HEAD for the delta, verification of each remediation claim in archreview/remediation/00-tracking.md against source.

Proto evolution since baseline — verified additive-only

git diff 59856b8..HEAD -- '*.proto' shows exactly two things: (1) one additive field, uint32 max_frame_bytes = 4; on GatewayHello (src/ZB.MOM.WW.MxGateway.Contracts/Protos/mxaccess_worker.proto:47, next free tag, documented zero-means-default semantics — safe for old peers); (2) new vendored Rust proto copies (clients/rust/protos/*.proto), byte-identical to the Contracts sources (cmp clean for all three files). No renumbering, no repurposing, reserved 1 / reserved "session_id" still present on the retired fields (mxaccess_gateway.proto:915-916,929-930), wire-compat policy headers intact. galaxy_repository.proto retention comment intact (ZB.MOM.WW.MxGateway.Contracts.csproj:37-42). Proto hygiene held.

Prior-finding verification

ID	Claimed	Verified?	Evidence (HEAD)	Notes
IPC-01	Done	Yes	Descriptor refreshed commit 309296f (contains MxSparseArray ×2, max_frame_bytes ×1 via strings); semantic freshness test src/ZB.MOM.WW.MxGateway.Tests/Contracts/ClientProtoInputTests.cs:21-59; CI runs scripts/check-codegen.ps1 + the test project (.gitea/workflows/ci.yml:60-65)	Test has blind spots — see IPC-27
IPC-02	Done	Yes	Gateway sends MaxFrameBytes in hello (Server/Workers/WorkerClient.cs:988 from FrameOptions.MaxMessageBytes, wired from Worker.MaxMessageBytes at Sessions/SessionWorkerClientFactory.cs:73-76); worker adopts after version+nonce checks, before the message loop (Worker/Ipc/WorkerPipeSession.cs:239; WorkerFrameProtocolOptions.cs:124-140): 0 → keep default, >256 MiB (MaxNegotiableFrameBytes, :19) → handshake fault. Docs: docs/WorkerFrameProtocol.md:25-30, docs/GatewayConfiguration.md:120	Adoption is single mutation pre-loop on the shared options instance; reader/writer read the property per frame — no TOCTOU. Handshake-fault path writes a fault then throws (WorkerPipeSession.cs:198-201) — fail-fast as designed
IPC-03	Done	Yes	EnvelopeOverheadReserveBytes = 64 KiB (Server/Workers/WorkerFrameProtocolOptions.cs:20); default pipe max = 16 MiB + reserve (Server/Configuration/WorkerOptions.cs:44-45); startup cross-validation with long arithmetic (no overflow) at GatewayOptionsValidator.cs:476-498; per-command pre-check at enqueue (WorkerClient.cs:207-221, WorkerClientErrorCode.CommandTooLarge at WorkerClientErrorCode.cs:19) → ResourceExhausted (Grpc/MxAccessGatewayService.cs:955); write-loop MessageTooLarge kept fatal as genuine desync (WorkerClient.cs:390-408)	Envelope not mutated between pre-check and write (gateway stamps Sequence at creation, WorkerClient.cs:1039) — no check/write TOCTOU. One doc gap: see IPC-28
IPC-04	Done	Partly	Public validator cap 10 000 (Grpc/MxAccessGrpcRequestValidator.cs:13,81-84); worker clamp MaxDrainEventsPerReply = 10 000, 0 → cap (Worker/Ipc/WorkerPipeSession.cs:26,547-550)	Cap is count-based only; a byte-oversized reply still kills the worker — residual finding IPC-23
IPC-05	Done	Yes	Second clone removed (WorkerClient.cs:1000-1007, transfers command into the envelope); MapEvent transfers ownership with an audit comment (Grpc/MxAccessGrpcMapper.cs:64-76); MapCommand's isolating clone kept (:34); docs/Grpc.md clone prose updated (~:218-220)
IPC-06	Done	Yes	gateway.md:303-337: hand-copied sketch replaced by proto-as-source-of-truth prose; string correlation_id correct (:311-313); all ten arms incl. worker_shutdown_ack listed (:316-324)
IPC-07	Done	Yes	docs/Grpc.md:13,32 say seven RPCs incl. QueryActiveAlarms; handler section at :97-99; rules-table row at :180
IPC-08	Open	Confirmed open	Zero WorkerCancel references in src/ZB.MOM.WW.MxGateway.Server outside Generated/ (grep); worker dispatch still present (WorkerPipeSession.cs:414-416); gateway.md:319 lists the arm ("best-effort cancellation") and :751-757 still describes the discard-late-reply contract	Still half-implemented: the arm is dead from the gateway side, the worker's CancelCommand path unreachable
IPC-09	Done	Mostly	Python: pinned grpcio-tools 1.80.0 + Assert-GrpcioToolsVersion (clients/python/generate-proto.ps1:10,36-41) + PATH-resolved python (:16-34); Go: Resolve-Tool PATH-first (clients/go/generate-proto.ps1:9-31); Java: checkGeneratedClean (clients/java/zb-mom-ww-mxgateway-client/build.gradle:61-70); CI java job verifies the generated tree (ci.yml:120-121)	Two guard holes found: the CI churn-revert masks real Java drift (IPC-24) and Go/Python committed bindings have no freshness guard at all — and are in fact stale at HEAD (IPC-25)
IPC-10	Open	Confirmed open (softened)	Validators still don't check sequence; but gateway.md:328-330 now honestly states sequence is a diagnostic aid, not validated (the "annotate" half of the original recommendation). Proto comment still absent (mxaccess_worker.proto:23 bare)	Acceptable state; see also IPC-31 (gateway stamp-vs-wire-order)
IPC-11	Open	Confirmed open	Strict equality check (WorkerPipeSession.cs:221-226); no min/max-range migration comment on supported_protocol_version (mxaccess_worker.proto:42)	No action was required; still true
IPC-12	Open	Confirmed open (surface reduced)	Server/Workers/WorkerFrameWriter.cs has no lock and no single-writer invariant note in its class doc (:8-11); all writes still funnel through the single write loop (WorkerClient.cs:390-397)	IPC-13's single WriteAsync per frame removed the two-write interleaving seam, but concurrent callers on the same Stream remain unsafe and undocumented
IPC-13	Done	Yes	Single serialization into one ArrayPool-rented prefixed buffer, one write (Server/Workers/WorkerFrameWriter.cs:57-76); wire format unchanged (LE uint32 + payload)
IPC-14	Done	Yes	Pooled payload read, length-bounded ParseFrom(payload, 0, length), return in finally (Server/Workers/WorkerFrameReader.cs:51-79)	Buffer never escapes; ParseFrom copies — lifetime sound
IPC-15	Done (as scoped)	Yes	Flush coalescing across the drained batch with written-and-flushed completion contract preserved (Worker/Ipc/WorkerFrameWriter.cs:116-182); multi-event envelope body intentionally not added (no WorkerEventBatch in the proto — grep 0); gateway.md:887-890 correctly distinguishes shipped flush-coalescing from the deferred additive batching
IPC-16	N/A	Improved	gateway.md:331-333 documents the envelope correlation_id as authoritative with the inner reply as parity echo; proto comments still absent
IPC-17	Done	Yes	docs/ClientProtoGeneration.md:110,175 and CLAUDE.md:83 use clients/python/src/zb_mom_ww_mxgateway/generated; grep finds no stale src/mxgateway/generated anywhere
IPC-18	Preserve	Preserved	Contracts/Generated/MxaccessWorker.cs carries MaxFrameBytes (×15) — committed C# in sync with the proto; Rust vendored protos byte-identical; galaxy proto wire-identity held	Exception: Go/Python worker bindings — IPC-25
IPC-19	Done	Yes	Regen-and-commit rule with net48/CS0246 rationale in docs/Contracts.md:100-115 and in the csproj comment (ZB.MOM.WW.MxGateway.Contracts.csproj:27-33); enforced by check-codegen.ps1 Check 2 (force-delete Generated/*.cs, rebuild, fail on git status --porcelain diff, lines 42-65) in CI
IPC-20	Done	Yes	protoc pinned to 34.1 (scripts/publish-client-proto-inputs.ps1:15); regeneration refuses a non-pinned protoc (:161-163); -Check is version-tolerant by normalizing both sides through the same protoc with source info stripped (New-SourceInfoFreeDescriptor, :59-79, compare at :145-147); CI installs the pinned protoc (ci.yml:43-47)	Sound design; the same-binary normalization cancels version-specific encoding
IPC-21	Done	Yes	gateway.md:341-362 shows the real seven-RPC service; the bidi Session sketch is now framed as "was considered as a possible long-term shape" (:368), outside the live service block
IPC-22	N/A	Unchanged	Status-code-plus-prose model unchanged; AcknowledgeAlarmReply.status placeholder comment intact	Info; no action planned

Done claims that do not fully hold: IPC-04 (residual byte-size failure mode, → IPC-23) and IPC-09 (guard coverage holes, → IPC-24/IPC-25). All other Done claims verified sound.

New findings

IPC-23 — Medium — DrainEvents bound is count-based only; a byte-heavy queue still builds a session-killing reply frame — Stability

Worker/Ipc/WorkerPipeSession.cs:537-562 caps the drain at MaxDrainEventsPerReply = 10 000 events (:26), but never sizes the reply. At the default negotiated frame max (16 MiB + 64 KiB), events averaging above ~1.7 KiB — exactly the array-heavy payloads this gateway exists for — overshoot MaxMessageBytes. The worker writer correctly rejects the oversized frame per-frame (Worker/Ipc/WorkerFrameWriter.cs:250-255, IsPerFrameRejection :192-198), so the stream survives, but the exception surfaces from WriteControlReplyAsync (WorkerPipeSession.cs:481,485-496) through DispatchGatewayEnvelopeAsync (:406) into RunMessageLoopAsync (:280) and out of RunAsync (:126) — the worker exits and the drained events are lost. This is the original S3 failure narrowed, not closed: a diagnostics command can still kill the session. Recommendation: size-aware packing in CreateDrainEventsReply (stop adding events when reply.CalculateSize() approaches the cap, report the truncation), or catch the per-frame rejection at the control-reply seam and answer with a protocol-status error reply instead of dying.

IPC-24 — Medium — CI's unconditional churn-revert step masks real Java generated-code drift for message-level proto changes — Underdeveloped

.gitea/workflows/ci.yml:114-121 runs git checkout -- .../MxaccessGateway.java and .../MxaccessWorker.java unconditionally before the "Verify generated tree is clean" step. The Java bindings are single-file aggregates (java_multiple_files unset), so any message/field change to a proto lands exactly and only in those two files. A .proto edit committed without regenerating the Java client therefore passes CI — the freshly regenerated (correct) files are reverted to the stale committed ones, and the diff check sees a clean tree. The step's own comment scopes it to "when no .proto changed", but nothing enforces that condition. This partially neuters the IPC-09 checkGeneratedClean intent; only service-level changes (which touch the *Grpc.java stubs) would still be caught. Recommendation: gate the revert on the push/PR diff containing no *.proto change (e.g., git diff --name-only $BASE..HEAD -- '*.proto'), or diff the two aggregates modulo the known protobuf-runtime-version lines instead of discarding them wholesale.

IPC-25 — Medium — Committed Go and Python worker bindings are stale at HEAD, and no guard covers them — Underdeveloped

clients/go/internal/generated/mxaccess_worker.pb.go (last regenerated 2026-05-23, commit 397d3c5) and clients/python/src/zb_mom_ww_mxgateway/generated/mxaccess_worker_pb2.py (2026-06-18, e328758) both predate the 2026-07-09 max_frame_bytes addition — grep finds zero occurrences in either, while the Java aggregate (regenerated 2026-07-10) has it. The freshness net closed by P1 covers Contracts/Generated (check-codegen Check 2), the descriptor (Check 1 + test), Rust vendored protos (Check 3), and Java (checkGeneratedClean) — but nothing compares the committed Go/Python bindings against the protos, and their unit tests don't exercise the worker types, so CI stays green. Functional impact today is nil (no language client consumes GatewayHello), but the published packages misrepresent the wire contract and the repo rule ("regenerate every generated client touched by the contract", CLAUDE.md verification matrix) was violated without any signal — the exact silent-drift class IPC-01/09 were meant to end. Recommendation: regenerate and commit both; add a Check 4 to scripts/check-codegen.ps1 (or per-client steps in CI) that regenerates Go/Python bindings and fails on diff, with the same churn-tolerance handling the Python pin already provides.

IPC-26 — Low — Worker frame writer: cancelling while waiting for the write lock leaves a ghost frame that is still written — Stability

Worker/Ipc/WorkerFrameWriter.cs:87-113: the frame is enqueued (:88-98) before _writeLock.WaitAsync(cancellationToken) (:103). If the caller's token fires while waiting, WriteAsync throws OperationCanceledException but the PendingFrame stays queued — the next lock-holder (heartbeat, reply, event) writes it anyway, and its Completion resolves with no observer. Consequences: a write the caller observed as cancelled still reaches the wire; during shutdown, leftover cancelled event frames drain in the same batch as (behind) the WorkerShutdownAck, i.e. events can be emitted after the ack that is nominally the last frame; and if no writer ever follows, the frame lingers unwritten with an unobserved completion. Benign today (the gateway drops post-shutdown/unknown frames), but the "cancelled means not written" contract is silently violated. Recommendation: on cancellation, remove or tombstone the pending frame (e.g., TrySetCanceled and skip tombstoned frames in DequeueNext).

IPC-27 — Low — The descriptor freshness test checks messages and fields only — enums, enum values, services/methods, and the Galaxy contract are blind spots — Underdeveloped

src/ZB.MOM.WW.MxGateway.Tests/Contracts/ClientProtoInputTests.cs:35-52 collects {message} and {message}/{field} symbols and enumerates only MxaccessGatewayReflection.Descriptor and MxaccessWorkerReflection.Descriptor. A new enum value (e.g., a new MxCommandKind case — the most likely future contract change), a new RPC on a service, or any change confined to galaxy_repository.proto would not redden the test. The script-side -Check in CI does catch all of these (full descriptor compare), so the gate as-a-whole holds — but docs/ClientProtoGeneration.md:152-157 presents the test as the protoc-free primary guard, and on any runner without protoc it is the only one. Recommendation: extend the test to enum values, service methods, and the Galaxy file descriptor (all available via the same reflection surface).

IPC-28 — Low — docs/Grpc.md exception-mapping prose omits the new CommandTooLarge → ResourceExhausted mapping — Conventions

Grpc/MxAccessGatewayService.cs:955 maps WorkerClientErrorCode.CommandTooLarge to ResourceExhausted, and the IPC-03 plan called for documenting that Invoke now returns ResourceExhausted for oversized payloads. docs/Grpc.md:250 still enumerates only CommandTimeout/GatewayShutdown/InvalidState/ProtocolViolation with "unmapped codes fall through to Unavailable" — a reader concludes an oversized command surfaces as Unavailable. (docs/GatewayConfiguration.md:120-129 does document the headroom rule and per-correlation failure.) Recommendation: add CommandTooLarge → ResourceExhausted to the mapping prose and the Invoke section.

IPC-29 — Low — Worker writer's priority scheduling and write-time sequence stamping are undocumented in the frame-protocol doc — Conventions

The worker writer is now a cooperative priority scheduler (control frames drain ahead of event frames) that stamps Sequence at the moment of writing under the lock (Worker/Ipc/WorkerFrameWriter.cs:11-17,116-124,238-240; WorkerFrameWritePriority.cs). These are protocol-relevant semantics: a per-frame rejection consumes a sequence number (stamped at :240 before the size check at :250 throws), so the wire sequence now has gaps after rejections, and event frames can be arbitrarily delayed behind a control backlog. docs/WorkerFrameProtocol.md:1-43 still describes the frame layer as only "message boundaries, size limits, protobuf parsing, and envelope validation"; gateway.md:328-330 covers the diagnostic-sequence half but not scheduling. Given the repo's docs-change-with-source rule, the component doc should carry this. Recommendation: add a short "Write scheduling and sequencing" section to docs/WorkerFrameProtocol.md.

IPC-30 — Low — An oversized worker→gateway event frame is still session-fatal despite the per-frame rejection machinery — Stability

Worker/Ipc/WorkerPipeSession.cs:374-382: the event drain loop awaits each event write; a MessageTooLarge per-frame rejection from the writer (an MXAccess array event serializing above the negotiated max) throws out of RunEventDrainLoopAsync, is rethrown by the message loop's await eventDrainTask (:294), and terminates RunAsync — the whole session dies for one unsendable event. Pre-existing behavior (not a regression), but the writer now survives the rejection and the session dies anyway one layer up, which wastes the new machinery. Handling it gracefully needs a policy decision (silently dropping an event conflicts with the delivery expectations behind "don't synthesize events"; a WorkerFault or a synthesized-loss marker are the honest options). Recommendation: decide and document the oversized-event policy; at minimum log the event identity before dying so the operator can find the offending tag.

IPC-31 — Info — Gateway envelope sequence is stamped at creation, not at write, so wire order can disagree with sequence order — Stability

Server/Workers/WorkerClient.cs:1039 assigns Sequence via Interlocked.Increment inside CreateEnvelope; the envelope is then enqueued to the outbound channel. Two concurrent InvokeAsync callers can increment and enqueue in opposite orders, producing non-monotonic sequences on the wire. The worker fixed exactly this on its side (WRK-04: stamp under the write lock at the point of writing, Worker/Ipc/WorkerFrameWriter.cs:238-240); the gateway was not given the same treatment. Harmless — gateway.md:328-330 declares sequence a diagnostic aid, never validated — but the two sides now implement different semantics for the same field. Recommendation: none required; if sequence ever becomes load-bearing, stamp in the gateway write loop.

IPC-32 — Info — check-codegen.ps1 check labels are wrong — Conventions

scripts/check-codegen.ps1:30,42,68 print "Check 1/2", "Check 2/2", then "Check 3/3" — the middle label predates the third check. Cosmetic only; CI log readers see a miscounted progression. Recommendation: relabel 1/3, 2/3, 3/3.

Severity roll-up (new findings)

Severity	Count	IDs
High	0	—
Medium	3	IPC-23, IPC-24, IPC-25
Low	5	IPC-26, IPC-27, IPC-28, IPC-29, IPC-30
Info	2	IPC-31, IPC-32

Still good / preserved

The frame protocol core remains sound and got materially better since baseline: length validation before allocation on both sides (Server/Workers/WorkerFrameReader.cs:36-49, Worker/Ipc/WorkerFrameReader.cs:36-49), exact-read loops with typed EndOfStream, pooled buffers with correct lifetimes on both readers and the gateway writer (rented buffers never escape; ParseFrom copies; returns in finally), single-serialization single-write framing on both writers, and the new negotiated max_frame_bytes handshake with a zero-fallback for old peers, a 256 MiB sanity ceiling, and adoption sequenced safely before the message loop. The 64 KiB headroom invariant is enforced at startup with overflow-safe arithmetic, and the oversized-command path now fails one correlation instead of the session. Proto hygiene held: the only contract change since baseline is additive, reserved ranges and UNSPECIFIED zero enums are intact, Contracts/Generated/ is in sync with the protos, the Rust vendored copies are byte-identical, and the descriptor set is fresh and doubly guarded (semantic test + normalized, protoc-pinned script check, both wired into CI). The doc layer at the contract boundary (gateway.md envelope section, seven-RPC surface, Session future-work framing, Python generated path, Generated/ regen rule) now matches the shipped contract.

Windows-only residue for this domain: the worker-side behaviors verified statically here (negotiated-max adoption, priority scheduler, drain clamp, batch flush) are exercised only by Worker.Tests -p:Platform=x86 on windev; the tracking log records them green there (356/0/11 on 2026-07-09), which this static review did not re-run.