Joseph Doherty
d5248f61a2
- IPC-01: regenerate the stale client descriptor set and add ClientProtoInputTests (semantic, protoc-free) so a missing contract symbol fails the build. - IPC-20: make publish-client-proto-inputs.ps1 -Check source_code_info-normalized so it is protoc-version tolerant. - IPC-19: document + guard the "Generated/ must be committed for net48" rule (docs/Contracts.md, csproj comment, check-codegen.ps1). - IPC-09: harden the per-client generate-proto scripts (PATH resolution + pinned grpcio/protobuf/java version asserts) so regeneration is reproducible. - TST-03: add .gitea/workflows/ci.yml (portable/java/windows/live jobs) running the build, tests, client checks, and the codegen guards. - Also: check-codegen.ps1 Check 3 guards the CLI-02 vendored Rust protos against drift. archreview: IPC-01/09/19/20 Done, TST-03 In review (pipeline authored + validated, not yet run on a Gitea runner). Verified on macOS: NonWindows build clean, ClientProtoInputTests 5/5, -Check exit 0.
170 lines
8.7 KiB
Markdown
170 lines
8.7 KiB
Markdown
# Protobuf Contracts
|
|
|
|
The contracts project contains the public gRPC API and the gateway-to-worker
|
|
IPC messages. The `.proto` files are the source of truth; generated C# files are
|
|
recreated by the contracts project build.
|
|
|
|
## Files
|
|
|
|
`src/ZB.MOM.WW.MxGateway.Contracts/Protos/mxaccess_gateway.proto` defines the public
|
|
`MxAccessGateway` gRPC service, command payloads, command replies, event DTOs,
|
|
`MxValue`, `MxArray`, `MxSparseArray`, and `MxStatusProxy`.
|
|
|
|
`MxValue` carries a `kind` oneof with arms for all scalar and array types. One
|
|
arm is `sparse_array_value = 19` (field number 19), which carries an
|
|
`MxSparseArray`. `MxSparseArray` is a write-only value type: the gateway accepts
|
|
it on every write variant (`Write`, `Write2`, `WriteSecured`, `WriteSecured2`,
|
|
and the corresponding `*BulkEntry` shapes), expands it into a full,
|
|
default-filled `MxArray` before forwarding to the worker, and rejects it on
|
|
read or event paths. The worker never receives or produces it.
|
|
|
|
`MxSparseArray` has three fields: `element_data_type` (1, the `MxDataType` of
|
|
every element), `total_length` (2, the length of the expanded full array), and
|
|
`elements` (3, `repeated MxSparseElement`). Each `MxSparseElement` has `index`
|
|
(1, zero-based position in the expanded array) and `value` (2, a scalar
|
|
`MxValue`). Indices not mentioned in `elements` take the element type's default
|
|
value — they are reset, not preserved. See `gateway.md` section
|
|
"MxSparseArray — default-fill partial array writes" for the expansion rules,
|
|
validation constraints, and the scope requirements per write variant.
|
|
|
|
The public command model includes bulk subscription command kinds for
|
|
`AddItemBulk`, `AdviseItemBulk`, `RemoveItemBulk`, `UnAdviseItemBulk`,
|
|
`SubscribeBulk`, and `UnsubscribeBulk`. These commands are normal unary
|
|
`Invoke` payloads. They do not add separate gRPC methods, and they return a
|
|
`BulkSubscribeReply` containing per-item `SubscribeResult` records with
|
|
`ServerHandle`, `TagAddress`, `ItemHandle`, `WasSuccessful`, and
|
|
`ErrorMessage`.
|
|
|
|
The gateway forwards each bulk command as one worker command. The worker runs
|
|
the corresponding MXAccess `AddItem`, `Advise`, `UnAdvise`, and `RemoveItem`
|
|
calls sequentially on the session STA and preserves input order in the result
|
|
list.
|
|
|
|
The command model also includes bulk write/read command kinds:
|
|
`WriteBulk`, `Write2Bulk`, `WriteSecuredBulk`, `WriteSecured2Bulk`, and
|
|
`ReadBulk`. They are unary `Invoke` payloads on the same `MxAccessGateway`
|
|
surface (not separate gRPC methods) and exist so a caller can submit one list
|
|
of items per round trip while preserving MXAccess parity per entry.
|
|
|
|
- `WriteBulkCommand` / `Write2BulkCommand` / `WriteSecuredBulkCommand` /
|
|
`WriteSecured2BulkCommand` each carry a `server_handle` and a `repeated`
|
|
list of entries (`WriteBulkEntry`, `Write2BulkEntry`,
|
|
`WriteSecuredBulkEntry`, `WriteSecured2BulkEntry`). Each entry mirrors the
|
|
single-item command shape — `item_handle` + `value` (+ `timestamp_value` on
|
|
the `*2` variants, + `current_user_id` / `verifier_user_id` on the secured
|
|
variants). All four replies use `BulkWriteReply`, which carries
|
|
`repeated BulkWriteResult`. A `BulkWriteResult` has `server_handle`,
|
|
`item_handle`, `was_successful`, `optional int32 hresult`, `repeated
|
|
MxStatusProxy statuses`, and `error_message`. Per-entry failures populate
|
|
`error_message` + `hresult` and never raise — callers iterate and inspect
|
|
each entry. The credential-sensitive redaction rules for `WriteSecured` /
|
|
`WriteSecured2` apply to every `value` inside `WriteSecuredBulkEntry` and
|
|
`WriteSecured2BulkEntry`.
|
|
|
|
- `ReadBulkCommand` carries `server_handle`, `repeated string tag_addresses`,
|
|
and `uint32 timeout_ms` (0 means use the gateway-configured default). The
|
|
reply is `BulkReadReply` carrying `repeated BulkReadResult`. A
|
|
`BulkReadResult` has `server_handle`, `tag_address`, `item_handle`,
|
|
`was_successful`, `was_cached`, `value`, `quality`, `source_timestamp`,
|
|
`repeated MxStatusProxy statuses`, and `error_message`. MXAccess has no
|
|
synchronous `Read`, so `ReadBulk` is dual-mode per entry: when a tag is
|
|
already advised in the session the worker returns the cached
|
|
`OnDataChange` payload without touching the subscription
|
|
(`was_cached = true`); otherwise the worker takes a full
|
|
`AddItem` + `Advise` + wait-for-first-`OnDataChange` + `UnAdvise` +
|
|
`RemoveItem` snapshot lifecycle and returns the result
|
|
(`was_cached = false`). The asymmetry that `BulkReadResult` has no
|
|
`hresult` field is intentional — `ReadBulk` outcomes are timeout / cache
|
|
/ lifecycle states rather than MXAccess COM return codes.
|
|
|
|
See `gateway.md` for the full cached-vs-snapshot `ReadBulk` lifecycle and the
|
|
per-command scope requirements, and `docs/DesignDecisions.md` "Bulk Command
|
|
Family" for the rationale behind the per-entry result shape (independent
|
|
success tracking, input-order preservation, no partial-failure exceptions).
|
|
|
|
`src/ZB.MOM.WW.MxGateway.Contracts/Protos/mxaccess_worker.proto` defines the named-pipe
|
|
worker IPC envelope and control messages. It imports
|
|
`mxaccess_gateway.proto` so the worker and gateway use the same command, reply,
|
|
event, value, and status shapes.
|
|
|
|
`src/ZB.MOM.WW.MxGateway.Contracts/Protos/galaxy_repository.proto` defines the
|
|
`GalaxyRepository` service used by clients to browse the Galaxy Repository
|
|
(deployed object hierarchy and dynamic attributes). The service is metadata-
|
|
only and does not share types with `mxaccess_gateway.proto`. See
|
|
[Galaxy Repository Browse](./GalaxyRepository.md) for the RPC catalog and
|
|
behavior.
|
|
|
|
Generated C# output is written to `src/ZB.MOM.WW.MxGateway.Contracts/Generated/`. Do not
|
|
hand-edit generated files.
|
|
|
|
The `Generated/` C# is **tracked and must be committed after any `.proto` change.** The
|
|
contracts project `Compile Remove`s `Generated/**/*.cs` and has `Grpc.Tools` regenerate them,
|
|
but `Grpc.Tools` skips regeneration when the committed output looks up to date. On the net10
|
|
build the freshly regenerated code is compiled, so a stale `Generated/` is invisible there —
|
|
but the **net48 worker** consumes the committed `Generated/` and fails to build with `CS0246`
|
|
on any new type when the checked-in code lags the `.proto`. So after editing a `.proto` you must
|
|
regenerate and commit `Generated/`. If a build does not pick up your proto change, delete the
|
|
stale output to force a full regeneration:
|
|
|
|
```bash
|
|
rm src/ZB.MOM.WW.MxGateway.Contracts/Generated/*.cs
|
|
dotnet build src/ZB.MOM.WW.MxGateway.Contracts/ZB.MOM.WW.MxGateway.Contracts.csproj
|
|
```
|
|
|
|
`scripts/check-codegen.ps1` enforces this in CI (it force-regenerates and fails on any
|
|
`git diff` against the committed `Generated/`); the Windows CI job's net48 worker build is the
|
|
definitive guard.
|
|
|
|
Client generation inputs are published through
|
|
`clients/proto/proto-inputs.json` and the descriptor set under
|
|
`clients/proto/descriptors/`. See
|
|
[Client Proto Generation](./ClientProtoGeneration.md) for language-specific
|
|
generation inputs, output directories, and golden protobuf JSON fixtures.
|
|
|
|
## Generation
|
|
|
|
Run the contracts build to regenerate C# protobuf and gRPC code:
|
|
|
|
```bash
|
|
dotnet build src/ZB.MOM.WW.MxGateway.Contracts/ZB.MOM.WW.MxGateway.Contracts.csproj
|
|
```
|
|
|
|
Run the focused contract tests after changing either `.proto` file:
|
|
|
|
```bash
|
|
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter ProtobufContractRoundTripTests
|
|
```
|
|
|
|
The full solution build also regenerates the C# contracts before compiling
|
|
gateway and test projects:
|
|
|
|
```bash
|
|
dotnet build src/ZB.MOM.WW.MxGateway.slnx
|
|
```
|
|
|
|
Regenerate the client descriptor after changing either `.proto` file, using the pinned protoc
|
|
(34.1, see [Toolchain Links](./ToolchainLinks.md)):
|
|
|
|
```bash
|
|
pwsh -File scripts/publish-client-proto-inputs.ps1
|
|
```
|
|
|
|
Freshness is guarded two ways so a skipped regeneration cannot ship silently:
|
|
|
|
- `ClientProtoInputTests.Descriptor_ContainsEveryContractMessageAndField` (gateway test project)
|
|
reflects over the in-process contract descriptors and fails if any message or field is missing
|
|
from the committed protoset. It is semantic (symbol presence), needs no protoc, and runs in the
|
|
Linux CI. A red test means "regenerate and commit the protoset."
|
|
- `pwsh -File scripts/publish-client-proto-inputs.ps1 -Check` rebuilds the descriptor and compares
|
|
it to the committed one. The comparison normalizes both sides through the same protoc with
|
|
`source_code_info` stripped, so it does not false-fail across protoc releases.
|
|
- `pwsh scripts/check-codegen.ps1` runs both the descriptor check and the `Generated/`-clean check
|
|
together; it is the single guard the CI pipeline invokes.
|
|
|
|
## Related Documentation
|
|
|
|
- [Client Proto Generation](./ClientProtoGeneration.md)
|
|
- [Gateway Process Detailed Design](./GatewayProcessDesign.md)
|
|
- [MXAccess Worker Instance Detailed Design](./MxAccessWorkerInstanceDesign.md)
|
|
- [Protobuf Style Guide](./style-guides/ProtobufStyleGuide.md)
|