fix(archreview): CI pipeline + codegen freshness guards (IPC-01/09/19/20, TST-03)
- IPC-01: regenerate the stale client descriptor set and add ClientProtoInputTests (semantic, protoc-free) so a missing contract symbol fails the build. - IPC-20: make publish-client-proto-inputs.ps1 -Check source_code_info-normalized so it is protoc-version tolerant. - IPC-19: document + guard the "Generated/ must be committed for net48" rule (docs/Contracts.md, csproj comment, check-codegen.ps1). - IPC-09: harden the per-client generate-proto scripts (PATH resolution + pinned grpcio/protobuf/java version asserts) so regeneration is reproducible. - TST-03: add .gitea/workflows/ci.yml (portable/java/windows/live jobs) running the build, tests, client checks, and the codegen guards. - Also: check-codegen.ps1 Check 3 guards the CLI-02 vendored Rust protos against drift. archreview: IPC-01/09/19/20 Done, TST-03 In review (pipeline authored + validated, not yet run on a Gitea runner). Verified on macOS: NonWindows build clean, ClientProtoInputTests 5/5, -Check exit 0.
This commit is contained in:
Joseph Doherty
@@ -48,25 +48,55 @@ session.
|
||||
## Descriptor Publishing
|
||||
|
||||
Run this command after changing either source `.proto` file or the client proto
|
||||
manifest:
|
||||
manifest, with the **pinned protoc 34.1** (see [Toolchain Links](./ToolchainLinks.md)):
|
||||
|
||||
```powershell
|
||||
scripts/publish-client-proto-inputs.ps1
|
||||
pwsh -File scripts/publish-client-proto-inputs.ps1
|
||||
```
|
||||
|
||||
The script writes
|
||||
`clients/proto/descriptors/mxaccessgw-client-v1.protoset` with imports and
|
||||
source information included. The descriptor is a generated artifact; do not edit
|
||||
it by hand.
|
||||
it by hand. Generating the committed artifact requires the pinned protoc version
|
||||
so the checked-in bytes are reproducible; the script fails fast on a version
|
||||
mismatch when generating.
|
||||
|
||||
Use the check mode in CI or before committing:
|
||||
|
||||
```powershell
|
||||
scripts/publish-client-proto-inputs.ps1 -Check
|
||||
pwsh -File scripts/publish-client-proto-inputs.ps1 -Check
|
||||
```
|
||||
|
||||
`-Check` rebuilds the descriptor in a temporary path and fails when the checked
|
||||
in descriptor is stale.
|
||||
`-Check` rebuilds the descriptor and fails when the checked-in descriptor is
|
||||
stale. The comparison normalizes both the committed and freshly built descriptor
|
||||
through the same protoc with `source_code_info` stripped, so it is tolerant of
|
||||
protoc-version encoding drift and does not false-fail across protoc releases
|
||||
(it warns, rather than fails, when protoc is off the pin).
|
||||
|
||||
The gateway test project carries an independent, protoc-free freshness guard:
|
||||
`ClientProtoInputTests.Descriptor_ContainsEveryContractMessageAndField` reflects
|
||||
over the in-process contract descriptors and fails if any contract message or
|
||||
field is missing from the committed protoset. This is the primary CI gate for
|
||||
descriptor staleness; a red test means "regenerate and commit the protoset."
|
||||
|
||||
### Pinned generator versions
|
||||
|
||||
Regeneration is reproducible only with the pinned toolchain. Regenerating with a
|
||||
different version silently produces incompatible or noisy output, so the per-client
|
||||
scripts assert the pin and resolve tools from `PATH`:
|
||||
|
||||
| Generator | Pinned version | Guard |
|
||||
|-----------|----------------|-------|
|
||||
| protoc (descriptor set) | 34.1 | version assertion in `scripts/publish-client-proto-inputs.ps1` |
|
||||
| `Grpc.Tools` (C# `Generated/`) | 2.80.0 (contracts csproj) | `scripts/check-codegen.ps1` git-diff of `Generated/` |
|
||||
| `grpcio-tools` (Python) | 1.80.0 (protobuf runtime 6.31.1) | version assertion in `clients/python/generate-proto.ps1` |
|
||||
| protobuf / grpc-java (Java) | `protobufVersion` / `grpcVersion` in `clients/java/build.gradle` | `checkGeneratedClean` gradle task |
|
||||
|
||||
A newer `grpcio-tools` stamps a `GRPC_GENERATED_VERSION` above the pinned grpcio
|
||||
runtime and breaks Python `pytest`; the Java protobuf plugin rewrites
|
||||
`MxaccessGateway.java` with spurious protobuf-runtime-version churn on every build
|
||||
(revert that one file when no `.proto` changed — see
|
||||
[Gateway Testing](./GatewayTesting.md) "Continuous Integration").
|
||||
|
||||
## Output Directories
|
||||
|
||||
|
||||
+33
-2
@@ -97,6 +97,24 @@ behavior.
|
||||
Generated C# output is written to `src/ZB.MOM.WW.MxGateway.Contracts/Generated/`. Do not
|
||||
hand-edit generated files.
|
||||
|
||||
The `Generated/` C# is **tracked and must be committed after any `.proto` change.** The
|
||||
contracts project `Compile Remove`s `Generated/**/*.cs` and has `Grpc.Tools` regenerate them,
|
||||
but `Grpc.Tools` skips regeneration when the committed output looks up to date. On the net10
|
||||
build the freshly regenerated code is compiled, so a stale `Generated/` is invisible there —
|
||||
but the **net48 worker** consumes the committed `Generated/` and fails to build with `CS0246`
|
||||
on any new type when the checked-in code lags the `.proto`. So after editing a `.proto` you must
|
||||
regenerate and commit `Generated/`. If a build does not pick up your proto change, delete the
|
||||
stale output to force a full regeneration:
|
||||
|
||||
```bash
|
||||
rm src/ZB.MOM.WW.MxGateway.Contracts/Generated/*.cs
|
||||
dotnet build src/ZB.MOM.WW.MxGateway.Contracts/ZB.MOM.WW.MxGateway.Contracts.csproj
|
||||
```
|
||||
|
||||
`scripts/check-codegen.ps1` enforces this in CI (it force-regenerates and fails on any
|
||||
`git diff` against the committed `Generated/`); the Windows CI job's net48 worker build is the
|
||||
definitive guard.
|
||||
|
||||
Client generation inputs are published through
|
||||
`clients/proto/proto-inputs.json` and the descriptor set under
|
||||
`clients/proto/descriptors/`. See
|
||||
@@ -124,12 +142,25 @@ gateway and test projects:
|
||||
dotnet build src/ZB.MOM.WW.MxGateway.slnx
|
||||
```
|
||||
|
||||
Regenerate the client descriptor after changing either `.proto` file:
|
||||
Regenerate the client descriptor after changing either `.proto` file, using the pinned protoc
|
||||
(34.1, see [Toolchain Links](./ToolchainLinks.md)):
|
||||
|
||||
```bash
|
||||
powershell -ExecutionPolicy Bypass -File scripts/publish-client-proto-inputs.ps1
|
||||
pwsh -File scripts/publish-client-proto-inputs.ps1
|
||||
```
|
||||
|
||||
Freshness is guarded two ways so a skipped regeneration cannot ship silently:
|
||||
|
||||
- `ClientProtoInputTests.Descriptor_ContainsEveryContractMessageAndField` (gateway test project)
|
||||
reflects over the in-process contract descriptors and fails if any message or field is missing
|
||||
from the committed protoset. It is semantic (symbol presence), needs no protoc, and runs in the
|
||||
Linux CI. A red test means "regenerate and commit the protoset."
|
||||
- `pwsh -File scripts/publish-client-proto-inputs.ps1 -Check` rebuilds the descriptor and compares
|
||||
it to the committed one. The comparison normalizes both sides through the same protoc with
|
||||
`source_code_info` stripped, so it does not false-fail across protoc releases.
|
||||
- `pwsh scripts/check-codegen.ps1` runs both the descriptor check and the `Generated/`-clean check
|
||||
together; it is the single guard the CI pipeline invokes.
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Client Proto Generation](./ClientProtoGeneration.md)
|
||||
|
||||
@@ -408,6 +408,34 @@ Run the gateway test project after shared gateway test infrastructure changes:
|
||||
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj
|
||||
```
|
||||
|
||||
## Continuous Integration
|
||||
|
||||
CI runs on Gitea Actions (`.gitea/workflows/ci.yml`; origin is Gitea at
|
||||
`gitea.dohertylan.com`) on every push and pull request. The pipeline is split by
|
||||
runtime because the x86 Worker cannot build on Linux:
|
||||
|
||||
- **`portable`** (Linux runner) — builds `src/ZB.MOM.WW.MxGateway.NonWindows.slnx`,
|
||||
runs the codegen/descriptor freshness guard (`scripts/check-codegen.ps1`), runs the
|
||||
gateway fake-worker tests, and builds/tests the clients that run on Linux: .NET client
|
||||
build, Go (`gofmt` + `go build` + `go test`), Rust (`cargo fmt --check` + `cargo test`
|
||||
+ `cargo clippy -D warnings`), and Python (`pytest`).
|
||||
- **`java`** (Linux, JDK 17) — `gradle test`. The protobuf gradle plugin rewrites
|
||||
`MxaccessGateway.java` with spurious protobuf-runtime-version churn on every build, so
|
||||
when no `.proto` changed the job reverts that one file (`git checkout`) before asserting
|
||||
the generated tree is clean. The dev Mac has no JRE, so Java verification is CI-only.
|
||||
- **`windows`** (self-hosted windev runner) — the only host that builds the **x86 /
|
||||
net48 Worker and Worker.Tests**; these are Windows-only and out of scope for the Linux
|
||||
jobs. This job also proves the net48 build against the committed `Generated/` (the
|
||||
definitive guard for the "regenerate and commit `Generated/`" rule).
|
||||
- **`live-mxaccess`** (self-hosted, MXAccess-installed) — scheduled only
|
||||
(`MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`); never gates a push.
|
||||
|
||||
The freshness guard `scripts/check-codegen.ps1` fails the build when the committed
|
||||
client descriptor set or the C# `Generated/` no longer matches the current `.proto`
|
||||
sources — the codegen drift class this repo has hit repeatedly (stale client
|
||||
descriptors, net48 `CS0246` on unregenerated protos). See
|
||||
[Client Proto Generation](./ClientProtoGeneration.md) and [Contracts](./Contracts.md).
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Cross-Language Smoke Matrix](./CrossLanguageSmokeMatrix.md)
|
||||
|
||||
Reference in New Issue
Block a user