mxaccessgw/archreview/2026-07-12/30-contracts-ipc.md

# Contracts & IPC Protocol — Architecture Review (re-run)

Date: 2026-07-12 · Commit: `4f5371f` (main) · macOS static review (no build/test; sources read at HEAD)
Prior review: [../30-contracts-ipc.md](../30-contracts-ipc.md) (2026-07-08, baseline `59856b8`) · Remediation plan: [../remediation/30-contracts-ipc.md](../remediation/30-contracts-ipc.md)

## Scope & method

Same domain as the prior review: `src/ZB.MOM.WW.MxGateway.Contracts` (three protos, `Generated/` hygiene, net10.0;net48 multi-targeting), the frame protocol on both sides (`src/ZB.MOM.WW.MxGateway.Worker/Ipc/*`, `src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerFrame*` + `WorkerClient`), the codegen/descriptor pipeline (`scripts/check-codegen.ps1`, `scripts/publish-client-proto-inputs.ps1`, `clients/proto/*`, vendored client proto copies incl. `clients/rust/protos`), and proto-evolution hygiene. Method: static reading at HEAD, `git diff 59856b8..HEAD` for the delta, verification of each remediation claim in `archreview/remediation/00-tracking.md` against source.

## Proto evolution since baseline — verified additive-only

`git diff 59856b8..HEAD -- '*.proto'` shows exactly two things: (1) one additive field, `uint32 max_frame_bytes = 4;` on `GatewayHello` (`src/ZB.MOM.WW.MxGateway.Contracts/Protos/mxaccess_worker.proto:47`, next free tag, documented zero-means-default semantics — safe for old peers); (2) new vendored Rust proto copies (`clients/rust/protos/*.proto`), byte-identical to the Contracts sources (`cmp` clean for all three files). No renumbering, no repurposing, `reserved 1` / `reserved "session_id"` still present on the retired fields (`mxaccess_gateway.proto:915-916,929-930`), wire-compat policy headers intact. `galaxy_repository.proto` retention comment intact (`ZB.MOM.WW.MxGateway.Contracts.csproj:37-42`). Proto hygiene held.

## Prior-finding verification

| ID | Claimed | Verified? | Evidence (HEAD) | Notes |
|----|---------|-----------|-----------------|-------|
| IPC-01 | Done | **Yes** | Descriptor refreshed commit `309296f` (contains `MxSparseArray` ×2, `max_frame_bytes` ×1 via `strings`); semantic freshness test `src/ZB.MOM.WW.MxGateway.Tests/Contracts/ClientProtoInputTests.cs:21-59`; CI runs `scripts/check-codegen.ps1` + the test project (`.gitea/workflows/ci.yml:60-65`) | Test has blind spots — see IPC-27 |
| IPC-02 | Done | **Yes** | Gateway sends `MaxFrameBytes` in hello (`Server/Workers/WorkerClient.cs:988` from `FrameOptions.MaxMessageBytes`, wired from `Worker.MaxMessageBytes` at `Sessions/SessionWorkerClientFactory.cs:73-76`); worker adopts after version+nonce checks, before the message loop (`Worker/Ipc/WorkerPipeSession.cs:239`; `WorkerFrameProtocolOptions.cs:124-140`): 0 → keep default, >256 MiB (`MaxNegotiableFrameBytes`, :19) → handshake fault. Docs: `docs/WorkerFrameProtocol.md:25-30`, `docs/GatewayConfiguration.md:120` | Adoption is single mutation pre-loop on the shared options instance; reader/writer read the property per frame — no TOCTOU. Handshake-fault path writes a fault then throws (`WorkerPipeSession.cs:198-201`) — fail-fast as designed |
| IPC-03 | Done | **Yes** | `EnvelopeOverheadReserveBytes` = 64 KiB (`Server/Workers/WorkerFrameProtocolOptions.cs:20`); default pipe max = 16 MiB + reserve (`Server/Configuration/WorkerOptions.cs:44-45`); startup cross-validation with `long` arithmetic (no overflow) at `GatewayOptionsValidator.cs:476-498`; per-command pre-check at enqueue (`WorkerClient.cs:207-221`, `WorkerClientErrorCode.CommandTooLarge` at `WorkerClientErrorCode.cs:19`) → `ResourceExhausted` (`Grpc/MxAccessGatewayService.cs:955`); write-loop `MessageTooLarge` kept fatal as genuine desync (`WorkerClient.cs:390-408`) | Envelope not mutated between pre-check and write (gateway stamps `Sequence` at creation, `WorkerClient.cs:1039`) — no check/write TOCTOU. One doc gap: see IPC-28 |
| IPC-04 | Done | **Partly** | Public validator cap 10 000 (`Grpc/MxAccessGrpcRequestValidator.cs:13,81-84`); worker clamp `MaxDrainEventsPerReply` = 10 000, 0 → cap (`Worker/Ipc/WorkerPipeSession.cs:26,547-550`) | Cap is **count-based only**; a byte-oversized reply still kills the worker — residual finding IPC-23 |
| IPC-05 | Done | **Yes** | Second clone removed (`WorkerClient.cs:1000-1007`, transfers `command` into the envelope); `MapEvent` transfers ownership with an audit comment (`Grpc/MxAccessGrpcMapper.cs:64-76`); `MapCommand`'s isolating clone kept (:34); `docs/Grpc.md` clone prose updated (~:218-220) | |
| IPC-06 | Done | **Yes** | `gateway.md:303-337`: hand-copied sketch replaced by proto-as-source-of-truth prose; `string correlation_id` correct (:311-313); all ten arms incl. `worker_shutdown_ack` listed (:316-324) | |
| IPC-07 | Done | **Yes** | `docs/Grpc.md:13,32` say seven RPCs incl. `QueryActiveAlarms`; handler section at :97-99; rules-table row at :180 | |
| IPC-08 | Open | **Confirmed open** | Zero `WorkerCancel` references in `src/ZB.MOM.WW.MxGateway.Server` outside `Generated/` (grep); worker dispatch still present (`WorkerPipeSession.cs:414-416`); `gateway.md:319` lists the arm ("best-effort cancellation") and :751-757 still describes the discard-late-reply contract | Still half-implemented: the arm is dead from the gateway side, the worker's `CancelCommand` path unreachable |
| IPC-09 | Done | **Mostly** | Python: pinned grpcio-tools 1.80.0 + `Assert-GrpcioToolsVersion` (`clients/python/generate-proto.ps1:10,36-41`) + PATH-resolved python (:16-34); Go: `Resolve-Tool` PATH-first (`clients/go/generate-proto.ps1:9-31`); Java: `checkGeneratedClean` (`clients/java/zb-mom-ww-mxgateway-client/build.gradle:61-70`); CI java job verifies the generated tree (`ci.yml:120-121`) | Two guard holes found: the CI churn-revert masks real Java drift (IPC-24) and Go/Python committed bindings have no freshness guard at all — and are in fact stale at HEAD (IPC-25) |
| IPC-10 | Open | **Confirmed open (softened)** | Validators still don't check sequence; but `gateway.md:328-330` now honestly states sequence is a diagnostic aid, not validated (the "annotate" half of the original recommendation). Proto comment still absent (`mxaccess_worker.proto:23` bare) | Acceptable state; see also IPC-31 (gateway stamp-vs-wire-order) |
| IPC-11 | Open | **Confirmed open** | Strict equality check (`WorkerPipeSession.cs:221-226`); no min/max-range migration comment on `supported_protocol_version` (`mxaccess_worker.proto:42`) | No action was required; still true |
| IPC-12 | Open | **Confirmed open (surface reduced)** | `Server/Workers/WorkerFrameWriter.cs` has no lock and no single-writer invariant note in its class doc (:8-11); all writes still funnel through the single write loop (`WorkerClient.cs:390-397`) | IPC-13's single `WriteAsync` per frame removed the two-write interleaving seam, but concurrent callers on the same `Stream` remain unsafe and undocumented |
| IPC-13 | Done | **Yes** | Single serialization into one ArrayPool-rented prefixed buffer, one write (`Server/Workers/WorkerFrameWriter.cs:57-76`); wire format unchanged (LE uint32 + payload) | |
| IPC-14 | Done | **Yes** | Pooled payload read, length-bounded `ParseFrom(payload, 0, length)`, return in `finally` (`Server/Workers/WorkerFrameReader.cs:51-79`) | Buffer never escapes; `ParseFrom` copies — lifetime sound |
| IPC-15 | Done (as scoped) | **Yes** | Flush coalescing across the drained batch with written-and-flushed completion contract preserved (`Worker/Ipc/WorkerFrameWriter.cs:116-182`); multi-event envelope body intentionally not added (no `WorkerEventBatch` in the proto — grep 0); `gateway.md:887-890` correctly distinguishes shipped flush-coalescing from the deferred additive batching | |
| IPC-16 | N/A | **Improved** | `gateway.md:331-333` documents the envelope `correlation_id` as authoritative with the inner reply as parity echo; proto comments still absent | |
| IPC-17 | Done | **Yes** | `docs/ClientProtoGeneration.md:110,175` and `CLAUDE.md:83` use `clients/python/src/zb_mom_ww_mxgateway/generated`; grep finds no stale `src/mxgateway/generated` anywhere | |
| IPC-18 | Preserve | **Preserved** | `Contracts/Generated/MxaccessWorker.cs` carries `MaxFrameBytes` (×15) — committed C# in sync with the proto; Rust vendored protos byte-identical; galaxy proto wire-identity held | Exception: Go/Python worker bindings — IPC-25 |
| IPC-19 | Done | **Yes** | Regen-and-commit rule with net48/CS0246 rationale in `docs/Contracts.md:100-115` and in the csproj comment (`ZB.MOM.WW.MxGateway.Contracts.csproj:27-33`); enforced by `check-codegen.ps1` Check 2 (force-delete `Generated/*.cs`, rebuild, fail on `git status --porcelain` diff, lines 42-65) in CI | |
| IPC-20 | Done | **Yes** | protoc pinned to 34.1 (`scripts/publish-client-proto-inputs.ps1:15`); regeneration refuses a non-pinned protoc (:161-163); `-Check` is version-tolerant by normalizing both sides through the *same* protoc with source info stripped (`New-SourceInfoFreeDescriptor`, :59-79, compare at :145-147); CI installs the pinned protoc (`ci.yml:43-47`) | Sound design; the same-binary normalization cancels version-specific encoding |
| IPC-21 | Done | **Yes** | `gateway.md:341-362` shows the real seven-RPC service; the bidi `Session` sketch is now framed as "was considered as a possible long-term shape" (:368), outside the live service block | |
| IPC-22 | N/A | **Unchanged** | Status-code-plus-prose model unchanged; `AcknowledgeAlarmReply.status` placeholder comment intact | Info; no action planned |

**Done claims that do not fully hold:** IPC-04 (residual byte-size failure mode, → IPC-23) and IPC-09 (guard coverage holes, → IPC-24/IPC-25). All other Done claims verified sound.

## New findings

### IPC-23 — Medium — `DrainEvents` bound is count-based only; a byte-heavy queue still builds a session-killing reply frame — Stability
`Worker/Ipc/WorkerPipeSession.cs:537-562` caps the drain at `MaxDrainEventsPerReply` = 10 000 events (:26), but never sizes the reply. At the default negotiated frame max (16 MiB + 64 KiB), events averaging above ~1.7 KiB — exactly the array-heavy payloads this gateway exists for — overshoot `MaxMessageBytes`. The worker writer correctly rejects the oversized frame per-frame (`Worker/Ipc/WorkerFrameWriter.cs:250-255`, `IsPerFrameRejection` :192-198), so the *stream* survives, but the exception surfaces from `WriteControlReplyAsync` (`WorkerPipeSession.cs:481,485-496`) through `DispatchGatewayEnvelopeAsync` (:406) into `RunMessageLoopAsync` (:280) and out of `RunAsync` (:126) — the worker exits and the drained events are lost. This is the original S3 failure narrowed, not closed: a diagnostics command can still kill the session.
**Recommendation:** size-aware packing in `CreateDrainEventsReply` (stop adding events when `reply.CalculateSize()` approaches the cap, report the truncation), or catch the per-frame rejection at the control-reply seam and answer with a protocol-status error reply instead of dying.

### IPC-24 — Medium — CI's unconditional churn-revert step masks real Java generated-code drift for message-level proto changes — Underdeveloped
`.gitea/workflows/ci.yml:114-121` runs `git checkout -- .../MxaccessGateway.java` and `.../MxaccessWorker.java` unconditionally before the "Verify generated tree is clean" step. The Java bindings are single-file aggregates (`java_multiple_files` unset), so *any* message/field change to a proto lands exactly and only in those two files. A `.proto` edit committed without regenerating the Java client therefore passes CI — the freshly regenerated (correct) files are reverted to the stale committed ones, and the diff check sees a clean tree. The step's own comment scopes it to "when no .proto changed", but nothing enforces that condition. This partially neuters the IPC-09 `checkGeneratedClean` intent; only service-level changes (which touch the `*Grpc.java` stubs) would still be caught.
**Recommendation:** gate the revert on the push/PR diff containing no `*.proto` change (e.g., `git diff --name-only $BASE..HEAD -- '*.proto'`), or diff the two aggregates modulo the known protobuf-runtime-version lines instead of discarding them wholesale.

### IPC-25 — Medium — Committed Go and Python worker bindings are stale at HEAD, and no guard covers them — Underdeveloped
`clients/go/internal/generated/mxaccess_worker.pb.go` (last regenerated 2026-05-23, commit `397d3c5`) and `clients/python/src/zb_mom_ww_mxgateway/generated/mxaccess_worker_pb2.py` (2026-06-18, `e328758`) both predate the 2026-07-09 `max_frame_bytes` addition — grep finds zero occurrences in either, while the Java aggregate (regenerated 2026-07-10) has it. The freshness net closed by P1 covers `Contracts/Generated` (check-codegen Check 2), the descriptor (Check 1 + test), Rust vendored protos (Check 3), and Java (checkGeneratedClean) — but **nothing** compares the committed Go/Python bindings against the protos, and their unit tests don't exercise the worker types, so CI stays green. Functional impact today is nil (no language client consumes `GatewayHello`), but the published packages misrepresent the wire contract and the repo rule ("regenerate every generated client touched by the contract", CLAUDE.md verification matrix) was violated without any signal — the exact silent-drift class IPC-01/09 were meant to end.
**Recommendation:** regenerate and commit both; add a Check 4 to `scripts/check-codegen.ps1` (or per-client steps in CI) that regenerates Go/Python bindings and fails on diff, with the same churn-tolerance handling the Python pin already provides.

### IPC-26 — Low — Worker frame writer: cancelling while waiting for the write lock leaves a ghost frame that is still written — Stability
`Worker/Ipc/WorkerFrameWriter.cs:87-113`: the frame is enqueued (:88-98) *before* `_writeLock.WaitAsync(cancellationToken)` (:103). If the caller's token fires while waiting, `WriteAsync` throws `OperationCanceledException` but the `PendingFrame` stays queued — the next lock-holder (heartbeat, reply, event) writes it anyway, and its `Completion` resolves with no observer. Consequences: a write the caller observed as cancelled still reaches the wire; during shutdown, leftover cancelled event frames drain in the same batch as (behind) the `WorkerShutdownAck`, i.e. events can be emitted after the ack that is nominally the last frame; and if no writer ever follows, the frame lingers unwritten with an unobserved completion. Benign today (the gateway drops post-shutdown/unknown frames), but the "cancelled means not written" contract is silently violated.
**Recommendation:** on cancellation, remove or tombstone the pending frame (e.g., `TrySetCanceled` and skip tombstoned frames in `DequeueNext`).

### IPC-27 — Low — The descriptor freshness test checks messages and fields only — enums, enum values, services/methods, and the Galaxy contract are blind spots — Underdeveloped
`src/ZB.MOM.WW.MxGateway.Tests/Contracts/ClientProtoInputTests.cs:35-52` collects `{message}` and `{message}/{field}` symbols and enumerates only `MxaccessGatewayReflection.Descriptor` and `MxaccessWorkerReflection.Descriptor`. A new enum value (e.g., a new `MxCommandKind` case — the most likely future contract change), a new RPC on a service, or any change confined to `galaxy_repository.proto` would not redden the test. The script-side `-Check` in CI *does* catch all of these (full descriptor compare), so the gate as-a-whole holds — but `docs/ClientProtoGeneration.md:152-157` presents the test as the protoc-free primary guard, and on any runner without protoc it is the only one.
**Recommendation:** extend the test to enum values, service methods, and the Galaxy file descriptor (all available via the same reflection surface).

### IPC-28 — Low — `docs/Grpc.md` exception-mapping prose omits the new `CommandTooLarge` → `ResourceExhausted` mapping — Conventions
`Grpc/MxAccessGatewayService.cs:955` maps `WorkerClientErrorCode.CommandTooLarge` to `ResourceExhausted`, and the IPC-03 plan called for documenting that Invoke now returns `ResourceExhausted` for oversized payloads. `docs/Grpc.md:250` still enumerates only `CommandTimeout`/`GatewayShutdown`/`InvalidState`/`ProtocolViolation` with "unmapped codes fall through to `Unavailable`" — a reader concludes an oversized command surfaces as `Unavailable`. (`docs/GatewayConfiguration.md:120-129` does document the headroom rule and per-correlation failure.)
**Recommendation:** add `CommandTooLarge → ResourceExhausted` to the mapping prose and the Invoke section.

### IPC-29 — Low — Worker writer's priority scheduling and write-time sequence stamping are undocumented in the frame-protocol doc — Conventions
The worker writer is now a cooperative priority scheduler (control frames drain ahead of event frames) that stamps `Sequence` at the moment of writing under the lock (`Worker/Ipc/WorkerFrameWriter.cs:11-17,116-124,238-240`; `WorkerFrameWritePriority.cs`). These are protocol-relevant semantics: a per-frame rejection consumes a sequence number (stamped at :240 before the size check at :250 throws), so the wire sequence now has gaps after rejections, and event frames can be arbitrarily delayed behind a control backlog. `docs/WorkerFrameProtocol.md:1-43` still describes the frame layer as only "message boundaries, size limits, protobuf parsing, and envelope validation"; `gateway.md:328-330` covers the diagnostic-sequence half but not scheduling. Given the repo's docs-change-with-source rule, the component doc should carry this.
**Recommendation:** add a short "Write scheduling and sequencing" section to `docs/WorkerFrameProtocol.md`.

### IPC-30 — Low — An oversized worker→gateway event frame is still session-fatal despite the per-frame rejection machinery — Stability
`Worker/Ipc/WorkerPipeSession.cs:374-382`: the event drain loop awaits each event write; a `MessageTooLarge` per-frame rejection from the writer (an MXAccess array event serializing above the negotiated max) throws out of `RunEventDrainLoopAsync`, is rethrown by the message loop's `await eventDrainTask` (:294), and terminates `RunAsync` — the whole session dies for one unsendable event. Pre-existing behavior (not a regression), but the writer now *survives* the rejection and the session dies anyway one layer up, which wastes the new machinery. Handling it gracefully needs a policy decision (silently dropping an event conflicts with the delivery expectations behind "don't synthesize events"; a `WorkerFault` or a synthesized-loss marker are the honest options).
**Recommendation:** decide and document the oversized-event policy; at minimum log the event identity before dying so the operator can find the offending tag.

### IPC-31 — Info — Gateway envelope sequence is stamped at creation, not at write, so wire order can disagree with sequence order — Stability
`Server/Workers/WorkerClient.cs:1039` assigns `Sequence` via `Interlocked.Increment` inside `CreateEnvelope`; the envelope is then enqueued to the outbound channel. Two concurrent `InvokeAsync` callers can increment and enqueue in opposite orders, producing non-monotonic sequences on the wire. The worker fixed exactly this on its side (WRK-04: stamp under the write lock at the point of writing, `Worker/Ipc/WorkerFrameWriter.cs:238-240`); the gateway was not given the same treatment. Harmless — `gateway.md:328-330` declares sequence a diagnostic aid, never validated — but the two sides now implement different semantics for the same field.
**Recommendation:** none required; if sequence ever becomes load-bearing, stamp in the gateway write loop.

### IPC-32 — Info — `check-codegen.ps1` check labels are wrong — Conventions
`scripts/check-codegen.ps1:30,42,68` print "Check 1/2", "Check 2/2", then "Check 3/3" — the middle label predates the third check. Cosmetic only; CI log readers see a miscounted progression.
**Recommendation:** relabel 1/3, 2/3, 3/3.

## Severity roll-up (new findings)

| Severity | Count | IDs |
|----------|-------|-----|
| High | 0 | — |
| Medium | 3 | IPC-23, IPC-24, IPC-25 |
| Low | 5 | IPC-26, IPC-27, IPC-28, IPC-29, IPC-30 |
| Info | 2 | IPC-31, IPC-32 |

## Still good / preserved

The frame protocol core remains sound and got materially better since baseline: length validation before allocation on both sides (`Server/Workers/WorkerFrameReader.cs:36-49`, `Worker/Ipc/WorkerFrameReader.cs:36-49`), exact-read loops with typed `EndOfStream`, pooled buffers with correct lifetimes on both readers and the gateway writer (rented buffers never escape; `ParseFrom` copies; returns in `finally`), single-serialization single-write framing on both writers, and the new negotiated `max_frame_bytes` handshake with a zero-fallback for old peers, a 256 MiB sanity ceiling, and adoption sequenced safely before the message loop. The 64 KiB headroom invariant is enforced at startup with overflow-safe arithmetic, and the oversized-command path now fails one correlation instead of the session. Proto hygiene held: the only contract change since baseline is additive, reserved ranges and `UNSPECIFIED` zero enums are intact, `Contracts/Generated/` is in sync with the protos, the Rust vendored copies are byte-identical, and the descriptor set is fresh and doubly guarded (semantic test + normalized, protoc-pinned script check, both wired into CI). The doc layer at the contract boundary (`gateway.md` envelope section, seven-RPC surface, `Session` future-work framing, Python generated path, `Generated/` regen rule) now matches the shipped contract.

Windows-only residue for this domain: the worker-side behaviors verified statically here (negotiated-max adoption, priority scheduler, drain clamp, batch flush) are exercised only by `Worker.Tests -p:Platform=x86` on windev; the tracking log records them green there (356/0/11 on 2026-07-09), which this static review did not re-run.