mxaccessgw/docs/GatewayTesting.md

# Gateway Testing

Gateway tests run without installed MXAccess by using fake workers, fake
transports, and in-process gRPC service fakes. Live MXAccess verification belongs
in opt-in integration tests because it depends on installed COM components and
provider state.

## Fake Worker Harness

`FakeWorkerHarness` in `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/Fakes/` provides an
in-process worker side for named-pipe IPC tests. It uses the same
`WorkerFrameReader`, `WorkerFrameWriter`, and `WorkerEnvelope` contract as the
gateway so tests exercise real frame validation and worker-client state changes.

Use the harness when a gateway or session test needs worker behavior without
starting `ZB.MOM.WW.MxGateway.Worker.exe` or loading MXAccess COM. The harness scripts:

- `WorkerHello` and `WorkerReady` startup,
- command replies with matching correlation ids,
- ordered `WorkerEvent` frames,
- `WorkerHeartbeat` frames,
- `WorkerFault` frames,
- shutdown acknowledgements,
- malformed protobuf payloads and oversized frame headers,
- slow or hung workers by withholding a reply.

Session-level tests can connect the harness to the pipe created by
`SessionWorkerClientFactory` with `ConnectToGatewayPipeAsync`. Lower-level
`WorkerClient` tests can use `CreateConnectedPairAsync` to create both pipe ends
inside the test.

`GatewayEndToEndFakeWorkerSmokeTests` composes the real gRPC service,
`SessionManager`, `SessionWorkerClientFactory`, `WorkerClient`, and
`EventStreamService` with a scripted fake worker launcher. The smoke test covers
`OpenSession`, `Register`, `AddItem`, `Advise`, one streamed `OnDataChange`
event, and `CloseSession` without loading MXAccess COM.

## Live MXAccess Smoke

`WorkerLiveMxAccessSmokeTests` in `src/ZB.MOM.WW.MxGateway.IntegrationTests/` composes the
real gRPC service, `SessionManager`, `SessionWorkerClientFactory`,
`WorkerClient`, `WorkerProcessLauncher`, and `ZB.MOM.WW.MxGateway.Worker.exe`. It is
skipped unless `MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1` is set because it creates
the installed MXAccess COM object and depends on live provider state.

The live smoke opens a gateway session, launches the x86 worker, runs
`Register`, `AddItem`, and `Advise`, waits a bounded time for the first
`OnDataChange` event (skipping any earlier bootstrap/registration-state event),
and closes the session in a `finally` block so the worker gets a graceful
shutdown request even when a command or event assertion fails. Cleanup failures
in that `finally` block are logged rather than thrown, so a real assertion
failure is never masked by a shutdown timeout.

`WorkerLiveMxAccessSmokeTests` additionally covers seven MXAccess parity paths the
fake-worker tests cannot validate:

- a `Write` round-trip against an advised item, asserting both that the reply is
  `Ok` / `MxCommandKind.Write` *and* that the worker emits a matching
  `OnWriteComplete` event for the targeted (server, item) handle pair — the
  same round-trip proof used by `scripts/run-client-e2e-tests.ps1`,
- an `AddItem` against an invalid server handle, asserting the MXAccess failure
  surfaces in the command reply without faulting the gateway transport,
- the `UnAdvise` → `RemoveItem` → `Unregister` teardown chain, asserting each
  step replies `Ok` with the matching `MxCommandKind`, that no further
  `OnDataChange` events arrive for the un-advised pair, and that a second
  `RemoveItem` against the freed handle relays a non-`Ok` MXAccess failure,
- a `WriteSecured` round-trip after `AuthenticateUser`, asserting the reply
  carries `MxCommandKind.WriteSecured` and the credential password never
  appears in the diagnostic message (parity for both the secured-write
  ordering rule and the "do not log secrets" contract),
- an abnormal worker exit (the worker process is killed mid-session) where the
  gateway must transition the session to `SessionState.Faulted` with a
  non-empty fault description carrying a known worker-client classification
  (pipe disconnected / worker faulted / end-of-stream / heartbeat expired),
- the B8 new COM commands — `AuthenticateUser`, `ArchestrAUserToId`, `Suspend`,
  and `Activate` — each asserting a real MXAccess reply (not `InvalidRequest`)
  is returned against an added-but-not-advised item, and
- the buffered-data path — `AddBufferedItem` and `SetBufferedUpdateInterval` —
  asserting the commands round-trip and that the worker delivers at least one
  `OnBufferedDataChange` event (the empty NoData bootstrap) without crashing
  or dropping frames; live §3.2 multi-sample conversion is noted as a residual
  when the rig does not drive sample-bearing buffered batches on demand.

All eight tests are gated by the same `MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`
opt-in variable.

Build the worker before running the smoke:

```bash
dotnet build src/ZB.MOM.WW.MxGateway.Worker/ZB.MOM.WW.MxGateway.Worker.csproj -p:Platform=x86
```

Run the smoke explicitly:

```bash
$env:MXGATEWAY_RUN_LIVE_MXACCESS_TESTS = "1"
dotnet test src/ZB.MOM.WW.MxGateway.IntegrationTests/ZB.MOM.WW.MxGateway.IntegrationTests.csproj --filter FullyQualifiedName~WorkerLiveMxAccessSmokeTests
```

Optional live smoke variables:

| Variable | Default | Description |
|----------|---------|-------------|
| `MXGATEWAY_LIVE_MXACCESS_WORKER_EXE` | First existing `ZB.MOM.WW.MxGateway.Worker.exe` under `src/ZB.MOM.WW.MxGateway.Worker/bin/...` | Worker executable path. Set this when running against a packaged worker or a non-default build output. |
| `MXGATEWAY_LIVE_MXACCESS_ITEM` | `TestChildObject.TestInt` | MXAccess item reference used by `AddItem`. |
| `MXGATEWAY_LIVE_MXACCESS_CLIENT_NAME` | `ZB.MOM.WW.MxGateway.IntegrationTests` | Client name passed to `Register`. |
| `MXGATEWAY_LIVE_MXACCESS_EVENT_TIMEOUT_SECONDS` | `15` | Maximum wait for the first `OnDataChange` (also used for the `OnWriteComplete` round-trip and the abnormal-exit fault transition). |
| `MXGATEWAY_LIVE_MXACCESS_WRITE_SECURED_USER` | `admin` | ArchestrA user name passed to `AuthenticateUser` before the `WriteSecured` parity step. |
| `MXGATEWAY_LIVE_MXACCESS_WRITE_SECURED_PASSWORD` | `admin123` | Password paired with the user above. Never logged; the test asserts the value does not appear in the WriteSecured diagnostic message. |

The test output includes session id, worker process id, command status,
HRESULT/status diagnostics, event sequence and handles, close status, and worker
stdout/stderr lines emitted during the run.

## Dev-rig Probes

`src/ZB.MOM.WW.MxGateway.Worker.Tests/Probes/` partitions runtime probes from the regular
Worker.Tests regression suite. The folder is its own
`ZB.MOM.WW.MxGateway.Worker.Tests.Probes` namespace so a discovery filter (e.g. `dotnet
test --filter FullyQualifiedName~ZB.MOM.WW.MxGateway.Worker.Tests.Probes`) can target or
exclude them without enumerating individual class names. The probes are
`[Fact(Skip = "...")]` by default and exist to characterize live AVEVA
behavior on the dev rig, not to gate CI — flip `Skip = null` on the dev box
with installed MXAccess + a running Galaxy provider when running them:

- `AlarmsLiveSmokeTests` — end-to-end smoke for the alarms-over-gateway
  pipeline (`WnWrapAlarmConsumer` + `AlarmDispatcher` +
  `MxAccessAlarmEventSink`) against `\\<machine>\Galaxy!DEV` with the dev rig's
  10-second flip script writing `TestMachine_001.TestAlarm001`.
- `AlarmClientWmProbeTests` — registers as an `AlarmClient` consumer on a real
  hidden message-only window and logs every Win32 message that arrives during
  a fixed pump window. Used to identify the `WM_APP` /
  `RegisterWindowMessage` IDs alarm callbacks use.
- `WnWrapConsumerProbeTests` — instantiates AVEVA's standalone `wnwrapConsumer`
  COM class, subscribes to the dev rig's `\\<machine>\Galaxy!DEV` provider,
  and polls `GetXmlCurrentAlarms2`. The XML payload bypasses the
  `FILETIME→DateTime` auto-marshaling that crashes
  `aaAlarmManagedClient.AlarmClient.GetHighPriAlarm` on this rig.

The probes share the Worker.Tests project (so they can use its `net48`/`x86`
configuration and the installed `ArchestrA.MxAccess` / `aaAlarmManagedClient`
references), but they are not part of the regression contract — a Worker.Tests
run with `Skip` left in place passes them as skipped.

## Live Galaxy Repository

`GalaxyRepositoryLiveTests` in `src/ZB.MOM.WW.MxGateway.IntegrationTests/Galaxy/` exercises
`GalaxyRepository` directly against the `ZB` Galaxy Repository SQL database. It is
skipped unless `MXGATEWAY_RUN_LIVE_GALAXY_TESTS=1` is set because it depends on a
reachable SQL Server instance and deployed Galaxy state — fake-worker tests cannot
cover the SQL browse RPCs.

The suite covers `TestConnectionAsync`, `GetLastDeployTimeAsync`,
`GetHierarchyAsync`, and `GetAttributesAsync`. `GetHierarchyAsync` and
`GetAttributesAsync` assert a non-empty result, so the connected `ZB` database
must contain a deployed Galaxy, not just an empty schema.

Run the Galaxy live tests explicitly:

```bash
$env:MXGATEWAY_RUN_LIVE_GALAXY_TESTS = "1"
dotnet test src/ZB.MOM.WW.MxGateway.IntegrationTests/ZB.MOM.WW.MxGateway.IntegrationTests.csproj --filter FullyQualifiedName~GalaxyRepositoryLiveTests
```

Optional live Galaxy variables:

| Variable | Default | Description |
|----------|---------|-------------|
| `MXGATEWAY_LIVE_GALAXY_CONN` | `Server=localhost;Database=ZB;Integrated Security=True;TrustServerCertificate=True;Encrypt=False;` | Galaxy Repository connection string. Set this when the `ZB` database is on a non-default instance or needs SQL authentication. |

The default connection string targets `ZB` on `localhost` with Windows
authentication, which matches the Galaxy Repository conventions in CLAUDE.md.

## Galaxy Filter Safety

`GalaxyFilterInputSafetyTests` in `src/ZB.MOM.WW.MxGateway.Tests/Galaxy/` covers adversarial
input handling for the Galaxy Repository browse filter layer. It runs in the
unit-test project (no live SQL needed) and complements the live SQL coverage in
`GalaxyRepositoryLiveTests`.

The test class re-frames the original "Galaxy SQL injection" concern (Tests-002 in
`code-reviews/Tests/findings.md`). `GalaxyRepository` issues only four *constant*
SQL statements (`HierarchySql`, `AttributesSql`, `SELECT 1`,
`SELECT time_of_last_deploy FROM galaxy`) — no `DiscoverHierarchyRequest` field
is ever concatenated into a SQL string, so there is no dynamic SQL surface and no
`LIKE`-escaping helper to test. All filters (`TagNameGlob`, `RootTagName`,
template-chain, category, contained-path) are applied **in memory** by
`GalaxyHierarchyProjector` / `GalaxyGlobMatcher` against the cached snapshot.

The adversarial-input matrix (`'`, `' OR '1'='1`, `'; DROP TABLE gobject;--`,
`%`, `_`, `100%_off`, `[abc]`, `Pump'001`) pins the following invariants:

- SQL metacharacters (`'`, `;`) and `LIKE`-wildcards (`%`, `_`) are treated as
  opaque literals by `GalaxyGlobMatcher` — they never act as wildcards, never
  spuriously match unrelated text.
- Only `*` and `?` are glob wildcards.
- `GalaxyGlobMatcher` applies a 100 ms regex timeout so a pathological glob
  (e.g. 5 000 `a` characters plus a literal `!`) completes promptly rather than
  catastrophically backtracking.
- `GalaxyHierarchyProjector` returns zero matches (rather than the whole
  hierarchy) for an adversarial `TagNameGlob` or `TemplateChainContains`, and
  surfaces `NotFound` for an adversarial `RootTagName`.
- The `DiscoverHierarchy` RPC end-to-end returns zero matches for adversarial
  `TagNameGlob` rather than faulting.

These invariants are the real security surface of the Galaxy browse path; the
SQL-injection framing does not apply to a constant-query layer.

## Live LDAP

`DashboardLdapLiveTests` in `src/ZB.MOM.WW.MxGateway.IntegrationTests/` exercises
`DashboardAuthenticator` against the live GLAuth directory. It is skipped unless
`MXGATEWAY_RUN_LIVE_LDAP_TESTS=1` is set because it binds against the GLAuth
service described in `glauth.md`.

The suite builds the authenticator with `GatewayOptions.Dashboard.GroupToRole`
set to `{ GwAdmin: Admin }`. `GwAdmin` is the gateway-specific
dashboard-admin role and is **not** part of the five baseline GLAuth role
groups — it must be provisioned before the LDAP live tests pass.
`AuthenticateAsync_AdminInGwAdminGroup_Succeeds` fails (rather than skips)
when GLAuth has only the baseline groups, so this is a hard prerequisite
beyond "LDAP is up." See the "Adding a gw-specific group" section of
`glauth.md` for the provisioning step that adds `GwAdmin` and grants it to
`admin`.

`DashboardAuthenticator` delegates the LDAP bind and group search to the shared
`ZB.MOM.WW.Auth.Ldap` provider (`LdapAuthService`) and only maps the resulting
groups to dashboard roles via `DashboardGroupRoleMapper`; the bind/search
mechanics that decide each outcome live in that shared provider, not in
`DashboardAuthenticator`.

The suite covers both the success path and the failure outcomes: `admin` whose
LDAP groups resolve to the `Admin` role succeeds and emits the role claim;
`readonly` is denied because no group in their `memberOf` appears in
`GroupToRole`; `admin` with a wrong password fails authentication without leaking
the password into `FailureMessage`; an unknown username fails authentication; and
an unreachable LDAP server is absorbed into a failed result rather than throwing.

`appsettings.json` now ships the LDAP bind password as the unexpanded
`${secret:ldap/mxgateway/bind}` token (resolved at gateway startup by the
pre-host secrets expander, which this suite's bare `ConfigurationBuilder`
does not run). Before running the live LDAP suite, set
`MxGateway__Ldap__ServiceAccountPassword` to the real GLAuth service-account
password so the suite binds with the real password instead of the literal
token. Obtain the current value from the GLAuth source of truth
`scadaproj/infra/glauth/` (per `glauth.md`); it is not committed here.

Run the LDAP live tests explicitly:

```bash
$env:MXGATEWAY_RUN_LIVE_LDAP_TESTS = "1"
$env:MxGateway__Ldap__ServiceAccountPassword = "<service-account-password>"
dotnet test src/ZB.MOM.WW.MxGateway.IntegrationTests/ZB.MOM.WW.MxGateway.IntegrationTests.csproj --filter FullyQualifiedName~DashboardLdapLiveTests
```

## Client E2E Scripts

`scripts/discover-testmachine-tags.ps1` queries the ZB Galaxy Repository for the
deployed runtime references used by the live client e2e scripts. It reads
`TestMachine_001` through `TestMachine_020` and the expected attributes:

- `ProtectedValue`
- `TestChangingInt`
- `TestBoolArray`
- `TestIntArray`
- `TestDateTimeArray`
- `TestStringArray`

The discovery output includes the exact `fullTagReference`, data type, array
dimension, and security classification. The array attributes are expected to be
dimension 50. `ProtectedValue` has security classification 2 and requires
secured write semantics; the current client CLI e2e runner subscribes to it but
does not attempt a normal `Write`.

Run discovery directly when validating the Galaxy Repository inputs:

```powershell
powershell -ExecutionPolicy Bypass -File scripts/discover-testmachine-tags.ps1 -Json
```

`scripts/run-client-e2e-tests.ps1` drives the .NET, Go, Rust, Python, and Java
client CLIs through a live gateway session. The gateway and worker are assumed
to be already running at `-Endpoint`; the script does not start or stop them.
For each client it runs these phases, then closes the session in a `finally`
path and writes a JSON report under `artifacts/e2e/`:

1. **Session + register** — opens one session and registers.
2. **Bulk** — verifies `SubscribeBulk` / `UnsubscribeBulk` on a bounded tag
   subset (skip with `-SkipBulk`).
3. **Add-item / advise** — adds and advises every discovered test tag. The
   loop has no `StreamEvents` consumer attached, so advised tags accumulate
   MXAccess change events in the worker event channel
   (`MxGateway:Events:QueueCapacity`); left unbounded it overflows under
   `FailFast` backpressure and faults the worker. Every `-DrainEveryTags`
   advised tags (default 15) the loop connects a short-lived `StreamEvents`
   drain so the gateway pumps that channel empty. `-DrainEveryTags 0` disables
   the drain.
4. **Stream** — asserts a bounded event stream delivers at least one event
   (skip with `-SkipStream`).
5. **Parity** — asserts MXAccess error paths are rejected rather than silently
   succeeding: an invalid item handle and an unknown session id (skip with
   `-SkipParity`).
6. **Auth rejection** — asserts `open-session` is rejected when the API key is
   missing, and (when `-RejectScopeApiKeyEnv` names an insufficient-scope key)
   when the key lacks the required scope. Skip with `-SkipAuth`.
7. **Write round-trip** — *opt-in (`-VerifyWrite`).* Runs right after
   `register`: adds and advises a configurable writable attribute
   (`-WriteAttribute`, default `TestChangingInt`), writes a per-client
   sentinel value, then streams events and asserts an `OnWriteComplete` event
   for that item is observed — proof the write round-tripped through the
   gateway, worker, and MXAccess provider. The written value being echoed back
   in an `OnDataChange` is recorded best-effort (`echoObserved`): a
   provider-driven attribute such as `TestChangingInt` accepts the write but
   immediately overwrites it, so no data-change carries the value back. The
   Rust `stream-events` CLI emits full per-event JSON (`family`, `itemHandle`,
   `value`) so all five clients apply the same checks.

   It is opt-in because it mutates live tag state. The phase fails fast if the
   write command is rejected — e.g. against a gateway whose worker predates
   write support (`MxAccessCommandExecutor` returning `InvalidRequest` for
   `Write`/`Write2`/`WriteSecured`/`WriteSecured2`).
8. **Alarm feed + acknowledge** — *opt-in (`-VerifyAlarms`).* Runs after the
   stream phase. Exercises the two session-less alarm subcommands against the
   gateway's central alarm monitor: `stream-alarms` reads a bounded slice of
   the feed (`-AlarmStreamMax`, default 1 — the feed's first message always
   arrives immediately, whereas later ones depend on live transitions) and
   asserts at least one `AlarmFeedMessage`; `acknowledge-alarm` acknowledges
   `-AlarmReference` (default `Galaxy!TestArea.TestMachine_001.TestAlarm001`)
   and asserts the RPC round-trips. The native ack outcome is not asserted —
   it depends on whether that alarm is currently active.

   It is opt-in because it depends on the gateway's central alarm monitor
   being enabled (`MxGateway:Alarms:Enabled`) and a live alarm provider.

Each client CLI is driven through one long-lived `batch` process. Every CLI
exposes a `batch` subcommand: a process that reads one command line from stdin,
runs it through the normal subcommand dispatch, writes the JSON result, then a
line containing exactly `__MXGW_BATCH_EOR__`. The harness launches one such
process per client and pings the ~250 operations of the flow through it, so the
process — and, for the JVM, the runtime — cold-start is paid once per client
instead of once per operation. A command that fails inside the batch process
writes its `{"error":...}` envelope and the loop continues; the harness treats
that envelope as the operation failure (used by the parity and auth phases).

Before the per-client phases run, the script builds the .NET CLI
(`dotnet build`) and installs the Java CLI (`gradle :mxgateway-cli:installDist`)
once, so the `batch` process launches straight from the compiled exe / the
installed launcher. The Go, Rust, and Python batch processes are launched via
`go run` / `cargo run` / `python -m`, which compile-or-start once when that
single per-client process starts.

Build the gateway and worker, start the gateway, and provide a valid API key
before running the client e2e script:

```powershell
$env:MXGATEWAY_API_KEY = "<api-key>"
powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1
```

Useful runner options:

```powershell
powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1 -Clients dotnet,python -MachineStart 1 -MachineEnd 2
powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1 -BulkTagCount 10
powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1 -SkipStream
powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1 -SkipBulk
# Write round-trip (opt-in): point at a writable scalar attribute and its
# value type.
powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1 -VerifyWrite -WriteAttribute TestChangingInt -WriteType int32
# Alarm feed + acknowledge (opt-in): needs MxGateway:Alarms:Enabled on the gateway.
powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1 -VerifyAlarms -AlarmReference "Galaxy!TestArea.TestMachine_001.TestAlarm001"
# Auth rejection: also assert an insufficient-scope key is denied.
powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1 -RejectScopeApiKeyEnv MXGATEWAY_READONLY_API_KEY
# Run all five clients concurrently as isolated child processes.
powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1 -Parallel
# Validate the flow offline (prints commands, contacts no gateway).
powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1 -DryRun
powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1 -Endpoint localhost:5000 -ApiKeyEnv MXGATEWAY_API_KEY
```

When `-VerifyWrite` is enabled, the write round-trip fails loudly if the write
command is rejected, if `-WriteAttribute` does not name a writable scalar
attribute, or if no `OnWriteComplete` event is observed for the written item
within `-WriteEchoMaxEvents` (default 200) streamed events. Raise
`-WriteEchoMaxEvents` if the gateway's per-session event backlog is large
enough to push `OnWriteComplete` past that bound.

## Focused Commands

Run the cross-language smoke matrix tests after changing the documented client
smoke command list:

```bash
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter FullyQualifiedName~CrossLanguageSmokeMatrixTests
```

Run the parity fixture matrix tests after changing the integration parity
scenario list:

```bash
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter FullyQualifiedName~ParityFixtureMatrixTests
```

Run the fake worker tests after changing gateway worker IPC, session startup, or
event streaming behavior:

```bash
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter FullyQualifiedName~FakeWorkerHarnessTests
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter FullyQualifiedName~SessionWorkerClientFactoryFakeWorkerTests
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter FullyQualifiedName~GatewayEndToEndFakeWorkerSmokeTests
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter FullyQualifiedName~WorkerClientTests
dotnet test src/ZB.MOM.WW.MxGateway.Worker.Tests/ZB.MOM.WW.MxGateway.Worker.Tests.csproj -p:Platform=x86 --filter FullyQualifiedName~WorkerPipeSessionTests
```

Run the gateway test project after shared gateway test infrastructure changes:

```bash
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj
```

## Continuous Integration

CI runs on Gitea Actions (`.gitea/workflows/ci.yml`; origin is Gitea at
`gitea.dohertylan.com`) on every push and pull request. The pipeline is split by
runtime because the x86 Worker cannot build on Linux:

- **`portable`** (Linux runner) — builds `src/ZB.MOM.WW.MxGateway.NonWindows.slnx`,
  runs the codegen/descriptor freshness guard (`scripts/check-codegen.ps1`), runs the
  gateway fake-worker tests, and builds/tests the clients that run on Linux: .NET client
  build, Go (`gofmt` + `go build` + `go test`), Rust (`cargo fmt --check` + `cargo test`
  + `cargo clippy -D warnings`), and Python (`pytest`).
- **`java`** (Linux, JDK 17) — `gradle test`, then `git diff --exit-code` over the generated
  tree. The grpc/protobuf toolchain is pinned (`clients/java/build.gradle`), so a regeneration
  is byte-identical to the committed single-file aggregates modulo real `.proto` changes; the
  git-diff is therefore a true drift gate that now catches message-level proto drift in the Java
  client (IPC-24 deleted the old unconditional churn-revert step, which masked exactly that
  class). The dev Mac has a Homebrew JDK 17, so `gradle generateProto` can be run there to refresh
  the Java aggregates when a `.proto` changes.
- **`windows-x86`** (Linux runner, per push/PR) — builds the **x86 / net48 Worker and
  Worker.Tests**, which are Windows-only and out of scope for the Linux jobs. It runs on a
  Linux runner that always schedules and SSHes to windev (`10.100.0.48`), where
  `scripts/ci/run-windev-ci.sh` fast-forwards the isolated CI clone `C:\build\mxaccessgw-ci`
  to the SHA under test and runs `scripts/ci/windev-worker-ci.ps1 -Mode test` there; the
  remote exit code propagates back, so a Worker regression turns the job red. A native
  Windows runner is not used because act_runner host-mode on Windows is broken and a
  `runs-on` gate with no runner wedges the queue forever. **A red `windows-x86` can mean
  the tier is down, not the change**: an unreachable windev fails the job (never skips), so
  if it goes red without a related code change, check the tier — `ssh windev`, read
  `C:\build\mxaccessgw-ci\ci.log` — before assuming a real test failure.
- **`nightly-windev`** (Linux runner, scheduled `0 6 * * *`) — re-runs the full x86 build +
  Worker.Tests on windev, then the live-MXAccess smoke (`MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`)
  and a full `slnx` build (which surfaces Windows-only analyzer diagnostics such as
  `xUnit1030`). Gated `if: github.event_name == 'schedule'`, so it never gates a push. On
  failure it opens a Gitea issue via the Actions token, since nobody watches the Actions page.

### Runner capacity is shared and finite

All CI runs on a co-located `gitea-runner` container on docker host `10.100.0.35` with
`maxParallel=1`, and that runner instance is **shared across repos** — it interleaves
`dohertj2/mxaccessgw` and `dohertj2/lmxopcua` jobs on the single slot rather than being
scoped to this repo (`GET /repos/dohertj2/mxaccessgw/actions/runners` returns
`total_count: 0`; the runner is registered at the instance level). Every job in a run
(`portable`, `java`, `windows-x86`) executes serially, so queue latency is additive within
a run and an active `lmxopcua` run blocks `mxaccessgw` entirely — expect ~20–30 minutes of
queue depth under cross-repo contention, not a stuck pipeline. This Gitea version (1.26)
also exposes **no run cancel or delete via the API** (`POST .../actions/runs/{id}/cancel`
returns 404, `DELETE .../actions/runs/{id}` returns 400), so a superseded or hung run cannot
be cleared and holds the slot until it finishes or times out. See
`docs/runbooks/TST-30-second-ci-runner.md` for the operator runbook that registers a second
runner to relieve this; until that lands, treat single-runner contention as expected, not a
CI outage.

When queue depth (or the missing-cancel reality) makes waiting impractical, verify a
specific commit out of band instead of waiting behind the queue: run
`CI_SHA=<sha> scripts/ci/run-windev-ci.sh <build|test|live>` from a machine with SSH access
to windev (the same script the SSH-driven `windows-x86`/`nightly-windev` jobs use — see
`scripts/ci/README.md`), or fall back to the manual windev worktree procedure below. This is
the same escape hatch used when the windev tier itself is down — TST-30 generalizes it from
"tier down" to "runner contended": either way, a stuck or slow shared runner should not
block verifying a commit.

The freshness guard `scripts/check-codegen.ps1` runs four checks and fails the build when the
committed client descriptor set (Check 1), the C# `Generated/` (Check 2), the Rust vendored
protos (Check 3), or the Go/Python client bindings (Check 4, IPC-25) no longer match the current
`.proto` sources — the codegen drift class this repo has hit repeatedly (stale client
descriptors, net48 `CS0246` on unregenerated protos, silently stale Go/Python worker bindings).
Check 4 regenerates the Go and Python bindings with their pinned generators (`protoc-gen-go`
v1.36.11 / `protoc-gen-go-grpc` 1.6.2, `grpcio-tools` 1.80.0) and fails on any diff; a missing
generator fails the check rather than skipping it. The **primary** guard for the
"regenerate and commit `Generated/`" rule is that regeneration diff in the `portable` job;
the `windows-x86` net48 compile is the **secondary** guard (a stale `Generated/` also breaks
the x86 build with `CS0246`). See [Client Proto Generation](./ClientProtoGeneration.md) and
[Contracts](./Contracts.md).

If the SSH-driven Windows tier is unavailable for infrastructure reasons (windev down, CI
key/secret rotation in flight) **or** the shared Gitea runner is contended and the queue is
impractical to wait behind (see "Runner capacity is shared and finite" above), fall back to
the manual windev worktree procedure as a degraded mode: on windev, fast-forward an isolated
`origin/main` worktree under `C:\build` (never the dirty Desktop checkout), then run the x86
Worker build and `Worker.Tests` (`-p:Platform=x86`) there by hand. Do this per merge for
worker-touching changes until the `windows-x86` job is green again (tier-down case) or the
queue clears (contention case).

## Related Documentation

- [Cross-Language Smoke Matrix](./CrossLanguageSmokeMatrix.md)
- [Parity Fixture Matrix](./ParityFixtureMatrix.md)
- [Gateway Process Design](./GatewayProcessDesign.md)
- [Worker Frame Protocol](./WorkerFrameProtocol.md)
- [MXAccess Worker Instance Detailed Design](./MxAccessWorkerInstanceDesign.md)