Joseph Doherty
eacdd2d453
Proto comments (comment-only, no wire change): - mxaccess_worker.proto GatewayHello.max_frame_bytes: every worker->gateway frame must serialize within the negotiated max; reply builders truncate (IPC-23). - mxaccess_gateway.proto DrainEventsReply: count-cap + byte-cap, drain-until-empty caller contract (IPC-23). - mxaccess_gateway.proto ReplayGap.oldest_available_sequence: empty-ring value is highest-observed+1, oldest-1 resume formula stays valid (GWC-25 deferred amendment). Regen wave: Contracts/Generated (C# XML doc), rust vendored protos (byte-copy), Go bindings (worker binding was genuinely stale - lacked MaxFrameBytes entirely), Python worker _pb2 (real descriptor delta), Java aggregates (javadoc, zero protobuf-version churn under the pinned toolchain), client descriptor set. IPC-24: pinned Java toolchain regenerates with no gencode-version churn, so the unconditional churn-revert step in ci.yml is a fossil - deleted it; git diff is now a true message-level drift gate for the single-file Java aggregates. IPC-25: pin protoc-gen-go v1.36.11 / protoc-gen-go-grpc 1.6.2 in the Go generate script (+ fix a latent pwsh-7 parse bug); add Check 4 to check-codegen.ps1 (regenerate Go+Python bindings, fail on diff, tool-missing fails not skips); add the pinned-generator installs to the portable CI job. IPC-32: relabel check-codegen banners 1/4..4/4 (folded into the Check 4 edit). Docs: ClientProtoGeneration.md, Contracts.md, GatewayTesting.md, build.gradle checkGeneratedClean caveat. Tracking: IPC-23/24/25/32 -> Done, GWC-25 proto note resolved, change-log 2026-08-07.
175 lines
9.2 KiB
Markdown
175 lines
9.2 KiB
Markdown
# Protobuf Contracts
|
|
|
|
The contracts project contains the public gRPC API and the gateway-to-worker
|
|
IPC messages. The `.proto` files are the source of truth; generated C# files are
|
|
recreated by the contracts project build.
|
|
|
|
## Files
|
|
|
|
`src/ZB.MOM.WW.MxGateway.Contracts/Protos/mxaccess_gateway.proto` defines the public
|
|
`MxAccessGateway` gRPC service, command payloads, command replies, event DTOs,
|
|
`MxValue`, `MxArray`, `MxSparseArray`, and `MxStatusProxy`.
|
|
|
|
`MxValue` carries a `kind` oneof with arms for all scalar and array types. One
|
|
arm is `sparse_array_value = 19` (field number 19), which carries an
|
|
`MxSparseArray`. `MxSparseArray` is a write-only value type: the gateway accepts
|
|
it on every write variant (`Write`, `Write2`, `WriteSecured`, `WriteSecured2`,
|
|
and the corresponding `*BulkEntry` shapes), expands it into a full,
|
|
default-filled `MxArray` before forwarding to the worker, and rejects it on
|
|
read or event paths. The worker never receives or produces it.
|
|
|
|
`MxSparseArray` has three fields: `element_data_type` (1, the `MxDataType` of
|
|
every element), `total_length` (2, the length of the expanded full array), and
|
|
`elements` (3, `repeated MxSparseElement`). Each `MxSparseElement` has `index`
|
|
(1, zero-based position in the expanded array) and `value` (2, a scalar
|
|
`MxValue`). Indices not mentioned in `elements` take the element type's default
|
|
value — they are reset, not preserved. See `gateway.md` section
|
|
"MxSparseArray — default-fill partial array writes" for the expansion rules,
|
|
validation constraints, and the scope requirements per write variant.
|
|
|
|
The public command model includes bulk subscription command kinds for
|
|
`AddItemBulk`, `AdviseItemBulk`, `RemoveItemBulk`, `UnAdviseItemBulk`,
|
|
`SubscribeBulk`, and `UnsubscribeBulk`. These commands are normal unary
|
|
`Invoke` payloads. They do not add separate gRPC methods, and they return a
|
|
`BulkSubscribeReply` containing per-item `SubscribeResult` records with
|
|
`ServerHandle`, `TagAddress`, `ItemHandle`, `WasSuccessful`, and
|
|
`ErrorMessage`.
|
|
|
|
The gateway forwards each bulk command as one worker command. The worker runs
|
|
the corresponding MXAccess `AddItem`, `Advise`, `UnAdvise`, and `RemoveItem`
|
|
calls sequentially on the session STA and preserves input order in the result
|
|
list.
|
|
|
|
The command model also includes bulk write/read command kinds:
|
|
`WriteBulk`, `Write2Bulk`, `WriteSecuredBulk`, `WriteSecured2Bulk`, and
|
|
`ReadBulk`. They are unary `Invoke` payloads on the same `MxAccessGateway`
|
|
surface (not separate gRPC methods) and exist so a caller can submit one list
|
|
of items per round trip while preserving MXAccess parity per entry.
|
|
|
|
- `WriteBulkCommand` / `Write2BulkCommand` / `WriteSecuredBulkCommand` /
|
|
`WriteSecured2BulkCommand` each carry a `server_handle` and a `repeated`
|
|
list of entries (`WriteBulkEntry`, `Write2BulkEntry`,
|
|
`WriteSecuredBulkEntry`, `WriteSecured2BulkEntry`). Each entry mirrors the
|
|
single-item command shape — `item_handle` + `value` (+ `timestamp_value` on
|
|
the `*2` variants, + `current_user_id` / `verifier_user_id` on the secured
|
|
variants). All four replies use `BulkWriteReply`, which carries
|
|
`repeated BulkWriteResult`. A `BulkWriteResult` has `server_handle`,
|
|
`item_handle`, `was_successful`, `optional int32 hresult`, `repeated
|
|
MxStatusProxy statuses`, and `error_message`. Per-entry failures populate
|
|
`error_message` + `hresult` and never raise — callers iterate and inspect
|
|
each entry. The credential-sensitive redaction rules for `WriteSecured` /
|
|
`WriteSecured2` apply to every `value` inside `WriteSecuredBulkEntry` and
|
|
`WriteSecured2BulkEntry`.
|
|
|
|
- `ReadBulkCommand` carries `server_handle`, `repeated string tag_addresses`,
|
|
and `uint32 timeout_ms` (0 means use the gateway-configured default). The
|
|
reply is `BulkReadReply` carrying `repeated BulkReadResult`. A
|
|
`BulkReadResult` has `server_handle`, `tag_address`, `item_handle`,
|
|
`was_successful`, `was_cached`, `value`, `quality`, `source_timestamp`,
|
|
`repeated MxStatusProxy statuses`, and `error_message`. MXAccess has no
|
|
synchronous `Read`, so `ReadBulk` is dual-mode per entry: when a tag is
|
|
already advised in the session the worker returns the cached
|
|
`OnDataChange` payload without touching the subscription
|
|
(`was_cached = true`); otherwise the worker takes a full
|
|
`AddItem` + `Advise` + wait-for-first-`OnDataChange` + `UnAdvise` +
|
|
`RemoveItem` snapshot lifecycle and returns the result
|
|
(`was_cached = false`). The asymmetry that `BulkReadResult` has no
|
|
`hresult` field is intentional — `ReadBulk` outcomes are timeout / cache
|
|
/ lifecycle states rather than MXAccess COM return codes.
|
|
|
|
See `gateway.md` for the full cached-vs-snapshot `ReadBulk` lifecycle and the
|
|
per-command scope requirements, and `docs/DesignDecisions.md` "Bulk Command
|
|
Family" for the rationale behind the per-entry result shape (independent
|
|
success tracking, input-order preservation, no partial-failure exceptions).
|
|
|
|
`src/ZB.MOM.WW.MxGateway.Contracts/Protos/mxaccess_worker.proto` defines the named-pipe
|
|
worker IPC envelope and control messages. It imports
|
|
`mxaccess_gateway.proto` so the worker and gateway use the same command, reply,
|
|
event, value, and status shapes.
|
|
|
|
`src/ZB.MOM.WW.MxGateway.Contracts/Protos/galaxy_repository.proto` defines the
|
|
`GalaxyRepository` service used by clients to browse the Galaxy Repository
|
|
(deployed object hierarchy and dynamic attributes). The service is metadata-
|
|
only and does not share types with `mxaccess_gateway.proto`. See
|
|
[Galaxy Repository Browse](./GalaxyRepository.md) for the RPC catalog and
|
|
behavior.
|
|
|
|
Generated C# output is written to `src/ZB.MOM.WW.MxGateway.Contracts/Generated/`. Do not
|
|
hand-edit generated files.
|
|
|
|
The `Generated/` C# is **tracked and must be committed after any `.proto` change.** The
|
|
contracts project `Compile Remove`s `Generated/**/*.cs` and has `Grpc.Tools` regenerate them,
|
|
but `Grpc.Tools` skips regeneration when the committed output looks up to date. On the net10
|
|
build the freshly regenerated code is compiled, so a stale `Generated/` is invisible there —
|
|
but the **net48 worker** consumes the committed `Generated/` and fails to build with `CS0246`
|
|
on any new type when the checked-in code lags the `.proto`. So after editing a `.proto` you must
|
|
regenerate and commit `Generated/`. If a build does not pick up your proto change, delete the
|
|
stale output to force a full regeneration:
|
|
|
|
```bash
|
|
rm src/ZB.MOM.WW.MxGateway.Contracts/Generated/*.cs
|
|
dotnet build src/ZB.MOM.WW.MxGateway.Contracts/ZB.MOM.WW.MxGateway.Contracts.csproj
|
|
```
|
|
|
|
`scripts/check-codegen.ps1` enforces this in CI (it force-regenerates and fails on any
|
|
`git diff` against the committed `Generated/`) — that regeneration diff in the `portable`
|
|
job is the primary guard. The SSH-driven `windows-x86` job's net48 worker build is the
|
|
secondary guard (a stale `Generated/` also breaks the x86 build with `CS0246`). The same
|
|
script runs four checks in total: the committed client descriptor set (Check 1), the C#
|
|
`Generated/` (Check 2), the Rust vendored protos (Check 3), and the Go/Python client bindings
|
|
(Check 4) each regenerate and fail on any diff. See
|
|
[Client Proto Generation](./ClientProtoGeneration.md) for the pinned generator versions.
|
|
|
|
Client generation inputs are published through
|
|
`clients/proto/proto-inputs.json` and the descriptor set under
|
|
`clients/proto/descriptors/`. See
|
|
[Client Proto Generation](./ClientProtoGeneration.md) for language-specific
|
|
generation inputs, output directories, and golden protobuf JSON fixtures.
|
|
|
|
## Generation
|
|
|
|
Run the contracts build to regenerate C# protobuf and gRPC code:
|
|
|
|
```bash
|
|
dotnet build src/ZB.MOM.WW.MxGateway.Contracts/ZB.MOM.WW.MxGateway.Contracts.csproj
|
|
```
|
|
|
|
Run the focused contract tests after changing either `.proto` file:
|
|
|
|
```bash
|
|
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter ProtobufContractRoundTripTests
|
|
```
|
|
|
|
The full solution build also regenerates the C# contracts before compiling
|
|
gateway and test projects:
|
|
|
|
```bash
|
|
dotnet build src/ZB.MOM.WW.MxGateway.slnx
|
|
```
|
|
|
|
Regenerate the client descriptor after changing either `.proto` file, using the pinned protoc
|
|
(34.1, see [Toolchain Links](./ToolchainLinks.md)):
|
|
|
|
```bash
|
|
pwsh -File scripts/publish-client-proto-inputs.ps1
|
|
```
|
|
|
|
Freshness is guarded two ways so a skipped regeneration cannot ship silently:
|
|
|
|
- `ClientProtoInputTests.Descriptor_ContainsEveryContractMessageAndField` (gateway test project)
|
|
reflects over the in-process contract descriptors and fails if any message or field is missing
|
|
from the committed protoset. It is semantic (symbol presence), needs no protoc, and runs in the
|
|
Linux CI. A red test means "regenerate and commit the protoset."
|
|
- `pwsh -File scripts/publish-client-proto-inputs.ps1 -Check` rebuilds the descriptor and compares
|
|
it to the committed one. The comparison normalizes both sides through the same protoc with
|
|
`source_code_info` stripped, so it does not false-fail across protoc releases.
|
|
- `pwsh scripts/check-codegen.ps1` runs both the descriptor check and the `Generated/`-clean check
|
|
together; it is the single guard the CI pipeline invokes.
|
|
|
|
## Related Documentation
|
|
|
|
- [Client Proto Generation](./ClientProtoGeneration.md)
|
|
- [Gateway Process Detailed Design](./GatewayProcessDesign.md)
|
|
- [MXAccess Worker Instance Detailed Design](./MxAccessWorkerInstanceDesign.md)
|
|
- [Protobuf Style Guide](./style-guides/ProtobufStyleGuide.md)
|