Joseph Doherty
d769244ac0
The `windows`/`live-mxaccess` CI jobs were removed in abb0930 because Gitea
act_runner host-mode on Windows is broken and a runs-on gate with no runner
wedges the queue. This left the entire x86/net48 Worker + Worker.Tests tier
(and the live-MXAccess smoke) unguarded by CI.
Restore it without a native Windows runner: Linux jobs on ubuntu-latest (which
always schedule) SSH to windev (10.100.0.48), fetch+checkout the SHA under test
in an isolated clone C:\build\mxaccessgw-ci, run the x86 build/tests there, and
propagate the remote exit code back — so a Worker regression turns the job red
and an unreachable host fails loud (never stuck).
- scripts/ci/run-windev-ci.sh: Linux driver (key/known-hosts, UTF-16LE base64
EncodedCommand bootstrap, ssh, exit-code passthrough).
- scripts/ci/windev-worker-ci.ps1: windev stage — worktree lock, re-fetch/
checkout under lock, build|test|live modes; PS 5.1-safe, checks $LASTEXITCODE.
- scripts/ci/windev.known_hosts: pinned host keys for StrictHostKeyChecking.
- scripts/ci/README.md: operator bring-up + acceptance checklist.
- ci.yml: windows-x86 (per-push `test`) + nightly-windev (scheduled `live` +
on-failure Gitea issue); drop the removal note; fix header/cron comments.
TST-26 (same commit, by rule): correct docs/scripts that still describe the
removed jobs — GatewayTesting.md, Contracts.md, check-codegen.ps1 — and
reattribute the Generated/-commit guard (primary = check-codegen diff in the
portable job; secondary = windows-x86 net48 compile).
Mechanism hand-verified on windev: build->0, bogus-SHA->nonzero (lock released),
test->356 passed/0 failed in ~50s, ssh exit-code propagation confirmed. Merge
requires operator bring-up first (dedicated ci@ key + Gitea secrets) or the
per-push job is red every push — see scripts/ci/README.md.
171 lines
8.9 KiB
Markdown
171 lines
8.9 KiB
Markdown
# Protobuf Contracts
|
|
|
|
The contracts project contains the public gRPC API and the gateway-to-worker
|
|
IPC messages. The `.proto` files are the source of truth; generated C# files are
|
|
recreated by the contracts project build.
|
|
|
|
## Files
|
|
|
|
`src/ZB.MOM.WW.MxGateway.Contracts/Protos/mxaccess_gateway.proto` defines the public
|
|
`MxAccessGateway` gRPC service, command payloads, command replies, event DTOs,
|
|
`MxValue`, `MxArray`, `MxSparseArray`, and `MxStatusProxy`.
|
|
|
|
`MxValue` carries a `kind` oneof with arms for all scalar and array types. One
|
|
arm is `sparse_array_value = 19` (field number 19), which carries an
|
|
`MxSparseArray`. `MxSparseArray` is a write-only value type: the gateway accepts
|
|
it on every write variant (`Write`, `Write2`, `WriteSecured`, `WriteSecured2`,
|
|
and the corresponding `*BulkEntry` shapes), expands it into a full,
|
|
default-filled `MxArray` before forwarding to the worker, and rejects it on
|
|
read or event paths. The worker never receives or produces it.
|
|
|
|
`MxSparseArray` has three fields: `element_data_type` (1, the `MxDataType` of
|
|
every element), `total_length` (2, the length of the expanded full array), and
|
|
`elements` (3, `repeated MxSparseElement`). Each `MxSparseElement` has `index`
|
|
(1, zero-based position in the expanded array) and `value` (2, a scalar
|
|
`MxValue`). Indices not mentioned in `elements` take the element type's default
|
|
value — they are reset, not preserved. See `gateway.md` section
|
|
"MxSparseArray — default-fill partial array writes" for the expansion rules,
|
|
validation constraints, and the scope requirements per write variant.
|
|
|
|
The public command model includes bulk subscription command kinds for
|
|
`AddItemBulk`, `AdviseItemBulk`, `RemoveItemBulk`, `UnAdviseItemBulk`,
|
|
`SubscribeBulk`, and `UnsubscribeBulk`. These commands are normal unary
|
|
`Invoke` payloads. They do not add separate gRPC methods, and they return a
|
|
`BulkSubscribeReply` containing per-item `SubscribeResult` records with
|
|
`ServerHandle`, `TagAddress`, `ItemHandle`, `WasSuccessful`, and
|
|
`ErrorMessage`.
|
|
|
|
The gateway forwards each bulk command as one worker command. The worker runs
|
|
the corresponding MXAccess `AddItem`, `Advise`, `UnAdvise`, and `RemoveItem`
|
|
calls sequentially on the session STA and preserves input order in the result
|
|
list.
|
|
|
|
The command model also includes bulk write/read command kinds:
|
|
`WriteBulk`, `Write2Bulk`, `WriteSecuredBulk`, `WriteSecured2Bulk`, and
|
|
`ReadBulk`. They are unary `Invoke` payloads on the same `MxAccessGateway`
|
|
surface (not separate gRPC methods) and exist so a caller can submit one list
|
|
of items per round trip while preserving MXAccess parity per entry.
|
|
|
|
- `WriteBulkCommand` / `Write2BulkCommand` / `WriteSecuredBulkCommand` /
|
|
`WriteSecured2BulkCommand` each carry a `server_handle` and a `repeated`
|
|
list of entries (`WriteBulkEntry`, `Write2BulkEntry`,
|
|
`WriteSecuredBulkEntry`, `WriteSecured2BulkEntry`). Each entry mirrors the
|
|
single-item command shape — `item_handle` + `value` (+ `timestamp_value` on
|
|
the `*2` variants, + `current_user_id` / `verifier_user_id` on the secured
|
|
variants). All four replies use `BulkWriteReply`, which carries
|
|
`repeated BulkWriteResult`. A `BulkWriteResult` has `server_handle`,
|
|
`item_handle`, `was_successful`, `optional int32 hresult`, `repeated
|
|
MxStatusProxy statuses`, and `error_message`. Per-entry failures populate
|
|
`error_message` + `hresult` and never raise — callers iterate and inspect
|
|
each entry. The credential-sensitive redaction rules for `WriteSecured` /
|
|
`WriteSecured2` apply to every `value` inside `WriteSecuredBulkEntry` and
|
|
`WriteSecured2BulkEntry`.
|
|
|
|
- `ReadBulkCommand` carries `server_handle`, `repeated string tag_addresses`,
|
|
and `uint32 timeout_ms` (0 means use the gateway-configured default). The
|
|
reply is `BulkReadReply` carrying `repeated BulkReadResult`. A
|
|
`BulkReadResult` has `server_handle`, `tag_address`, `item_handle`,
|
|
`was_successful`, `was_cached`, `value`, `quality`, `source_timestamp`,
|
|
`repeated MxStatusProxy statuses`, and `error_message`. MXAccess has no
|
|
synchronous `Read`, so `ReadBulk` is dual-mode per entry: when a tag is
|
|
already advised in the session the worker returns the cached
|
|
`OnDataChange` payload without touching the subscription
|
|
(`was_cached = true`); otherwise the worker takes a full
|
|
`AddItem` + `Advise` + wait-for-first-`OnDataChange` + `UnAdvise` +
|
|
`RemoveItem` snapshot lifecycle and returns the result
|
|
(`was_cached = false`). The asymmetry that `BulkReadResult` has no
|
|
`hresult` field is intentional — `ReadBulk` outcomes are timeout / cache
|
|
/ lifecycle states rather than MXAccess COM return codes.
|
|
|
|
See `gateway.md` for the full cached-vs-snapshot `ReadBulk` lifecycle and the
|
|
per-command scope requirements, and `docs/DesignDecisions.md` "Bulk Command
|
|
Family" for the rationale behind the per-entry result shape (independent
|
|
success tracking, input-order preservation, no partial-failure exceptions).
|
|
|
|
`src/ZB.MOM.WW.MxGateway.Contracts/Protos/mxaccess_worker.proto` defines the named-pipe
|
|
worker IPC envelope and control messages. It imports
|
|
`mxaccess_gateway.proto` so the worker and gateway use the same command, reply,
|
|
event, value, and status shapes.
|
|
|
|
`src/ZB.MOM.WW.MxGateway.Contracts/Protos/galaxy_repository.proto` defines the
|
|
`GalaxyRepository` service used by clients to browse the Galaxy Repository
|
|
(deployed object hierarchy and dynamic attributes). The service is metadata-
|
|
only and does not share types with `mxaccess_gateway.proto`. See
|
|
[Galaxy Repository Browse](./GalaxyRepository.md) for the RPC catalog and
|
|
behavior.
|
|
|
|
Generated C# output is written to `src/ZB.MOM.WW.MxGateway.Contracts/Generated/`. Do not
|
|
hand-edit generated files.
|
|
|
|
The `Generated/` C# is **tracked and must be committed after any `.proto` change.** The
|
|
contracts project `Compile Remove`s `Generated/**/*.cs` and has `Grpc.Tools` regenerate them,
|
|
but `Grpc.Tools` skips regeneration when the committed output looks up to date. On the net10
|
|
build the freshly regenerated code is compiled, so a stale `Generated/` is invisible there —
|
|
but the **net48 worker** consumes the committed `Generated/` and fails to build with `CS0246`
|
|
on any new type when the checked-in code lags the `.proto`. So after editing a `.proto` you must
|
|
regenerate and commit `Generated/`. If a build does not pick up your proto change, delete the
|
|
stale output to force a full regeneration:
|
|
|
|
```bash
|
|
rm src/ZB.MOM.WW.MxGateway.Contracts/Generated/*.cs
|
|
dotnet build src/ZB.MOM.WW.MxGateway.Contracts/ZB.MOM.WW.MxGateway.Contracts.csproj
|
|
```
|
|
|
|
`scripts/check-codegen.ps1` enforces this in CI (it force-regenerates and fails on any
|
|
`git diff` against the committed `Generated/`) — that regeneration diff in the `portable`
|
|
job is the primary guard. The SSH-driven `windows-x86` job's net48 worker build is the
|
|
secondary guard (a stale `Generated/` also breaks the x86 build with `CS0246`).
|
|
|
|
Client generation inputs are published through
|
|
`clients/proto/proto-inputs.json` and the descriptor set under
|
|
`clients/proto/descriptors/`. See
|
|
[Client Proto Generation](./ClientProtoGeneration.md) for language-specific
|
|
generation inputs, output directories, and golden protobuf JSON fixtures.
|
|
|
|
## Generation
|
|
|
|
Run the contracts build to regenerate C# protobuf and gRPC code:
|
|
|
|
```bash
|
|
dotnet build src/ZB.MOM.WW.MxGateway.Contracts/ZB.MOM.WW.MxGateway.Contracts.csproj
|
|
```
|
|
|
|
Run the focused contract tests after changing either `.proto` file:
|
|
|
|
```bash
|
|
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter ProtobufContractRoundTripTests
|
|
```
|
|
|
|
The full solution build also regenerates the C# contracts before compiling
|
|
gateway and test projects:
|
|
|
|
```bash
|
|
dotnet build src/ZB.MOM.WW.MxGateway.slnx
|
|
```
|
|
|
|
Regenerate the client descriptor after changing either `.proto` file, using the pinned protoc
|
|
(34.1, see [Toolchain Links](./ToolchainLinks.md)):
|
|
|
|
```bash
|
|
pwsh -File scripts/publish-client-proto-inputs.ps1
|
|
```
|
|
|
|
Freshness is guarded two ways so a skipped regeneration cannot ship silently:
|
|
|
|
- `ClientProtoInputTests.Descriptor_ContainsEveryContractMessageAndField` (gateway test project)
|
|
reflects over the in-process contract descriptors and fails if any message or field is missing
|
|
from the committed protoset. It is semantic (symbol presence), needs no protoc, and runs in the
|
|
Linux CI. A red test means "regenerate and commit the protoset."
|
|
- `pwsh -File scripts/publish-client-proto-inputs.ps1 -Check` rebuilds the descriptor and compares
|
|
it to the committed one. The comparison normalizes both sides through the same protoc with
|
|
`source_code_info` stripped, so it does not false-fail across protoc releases.
|
|
- `pwsh scripts/check-codegen.ps1` runs both the descriptor check and the `Generated/`-clean check
|
|
together; it is the single guard the CI pipeline invokes.
|
|
|
|
## Related Documentation
|
|
|
|
- [Client Proto Generation](./ClientProtoGeneration.md)
|
|
- [Gateway Process Detailed Design](./GatewayProcessDesign.md)
|
|
- [MXAccess Worker Instance Detailed Design](./MxAccessWorkerInstanceDesign.md)
|
|
- [Protobuf Style Guide](./style-guides/ProtobufStyleGuide.md)
|