mxaccessgw/docs/GatewayTesting.md

# Gateway Testing

Gateway tests run without installed MXAccess by using fake workers, fake
transports, and in-process gRPC service fakes. Live MXAccess verification belongs
in opt-in integration tests because it depends on installed COM components and
provider state.

## Fake Worker Harness

`FakeWorkerHarness` in `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/Fakes/` provides an
in-process worker side for named-pipe IPC tests. It uses the same
`WorkerFrameReader`, `WorkerFrameWriter`, and `WorkerEnvelope` contract as the
gateway so tests exercise real frame validation and worker-client state changes.

Use the harness when a gateway or session test needs worker behavior without
starting `ZB.MOM.WW.MxGateway.Worker.exe` or loading MXAccess COM. The harness scripts:

- `WorkerHello` and `WorkerReady` startup,
- command replies with matching correlation ids,
- ordered `WorkerEvent` frames,
- `WorkerHeartbeat` frames,
- `WorkerFault` frames,
- shutdown acknowledgements,
- malformed protobuf payloads and oversized frame headers,
- slow or hung workers by withholding a reply.

Session-level tests can connect the harness to the pipe created by
`SessionWorkerClientFactory` with `ConnectToGatewayPipeAsync`. Lower-level
`WorkerClient` tests can use `CreateConnectedPairAsync` to create both pipe ends
inside the test.

`GatewayEndToEndFakeWorkerSmokeTests` composes the real gRPC service,
`SessionManager`, `SessionWorkerClientFactory`, `WorkerClient`, and
`EventStreamService` with a scripted fake worker launcher. The smoke test covers
`OpenSession`, `Register`, `AddItem`, `Advise`, one streamed `OnDataChange`
event, and `CloseSession` without loading MXAccess COM.

## Live MXAccess Smoke

`WorkerLiveMxAccessSmokeTests` in `src/ZB.MOM.WW.MxGateway.IntegrationTests/` composes the
real gRPC service, `SessionManager`, `SessionWorkerClientFactory`,
`WorkerClient`, `WorkerProcessLauncher`, and `ZB.MOM.WW.MxGateway.Worker.exe`. It is
skipped unless `MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1` is set because it creates
the installed MXAccess COM object and depends on live provider state.

The live smoke opens a gateway session, launches the x86 worker, runs
`Register`, `AddItem`, and `Advise`, waits a bounded time for the first
`OnDataChange` event (skipping any earlier bootstrap/registration-state event),
and closes the session in a `finally` block so the worker gets a graceful
shutdown request even when a command or event assertion fails. Cleanup failures
in that `finally` block are logged rather than thrown, so a real assertion
failure is never masked by a shutdown timeout.

`WorkerLiveMxAccessSmokeTests` additionally covers seven MXAccess parity paths the
fake-worker tests cannot validate:

- a `Write` round-trip against an advised item, asserting both that the reply is
  `Ok` / `MxCommandKind.Write` *and* that the worker emits a matching
  `OnWriteComplete` event for the targeted (server, item) handle pair — the
  same round-trip proof used by `scripts/run-client-e2e-tests.ps1`,
- an `AddItem` against an invalid server handle, asserting the MXAccess failure
  surfaces in the command reply without faulting the gateway transport,
- the `UnAdvise` → `RemoveItem` → `Unregister` teardown chain, asserting each
  step replies `Ok` with the matching `MxCommandKind`, that no further
  `OnDataChange` events arrive for the un-advised pair, and that a second
  `RemoveItem` against the freed handle relays a non-`Ok` MXAccess failure,
- a `WriteSecured` round-trip after `AuthenticateUser`, asserting the reply
  carries `MxCommandKind.WriteSecured` and the credential password never
  appears in the diagnostic message (parity for both the secured-write
  ordering rule and the "do not log secrets" contract),
- an abnormal worker exit (the worker process is killed mid-session) where the
  gateway must transition the session to `SessionState.Faulted` with a
  non-empty fault description carrying a known worker-client classification
  (pipe disconnected / worker faulted / end-of-stream / heartbeat expired),
- the B8 new COM commands — `AuthenticateUser`, `ArchestrAUserToId`, `Suspend`,
  and `Activate` — each asserting a real MXAccess reply (not `InvalidRequest`)
  is returned against an added-but-not-advised item, and
- the buffered-data path — `AddBufferedItem` and `SetBufferedUpdateInterval` —
  asserting the commands round-trip and that the worker delivers at least one
  `OnBufferedDataChange` event (the empty NoData bootstrap) without crashing
  or dropping frames; live §3.2 multi-sample conversion is noted as a residual
  when the rig does not drive sample-bearing buffered batches on demand.

All eight tests are gated by the same `MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`
opt-in variable. Opt-in does not mean unscheduled: the `nightly-windev` job runs
this suite on windev every night at 06:00 UTC and files a Gitea issue when it goes
red (see [Continuous Integration](#continuous-integration)), so the smoke no longer
depends on someone remembering to set the variable.

Known coverage gap: the suite reaches all six late-added MXAccess **COM** commands
but none of the five **control** commands (`Ping`, `GetSessionState`,
`GetWorkerInfo`, `DrainEvents`, `ShutdownWorker`). Those are implemented off-STA in
`Worker/Ipc/WorkerPipeSession.cs` and are asserted only against
`FakeWorkerHarness`'s canned replies, so no test proves the *real* worker answers
them. Closing that is the residual half of archreview TST-05.

Build the worker before running the smoke:

```bash
dotnet build src/ZB.MOM.WW.MxGateway.Worker/ZB.MOM.WW.MxGateway.Worker.csproj -p:Platform=x86
```

Run the smoke explicitly:

```bash
$env:MXGATEWAY_RUN_LIVE_MXACCESS_TESTS = "1"
dotnet test src/ZB.MOM.WW.MxGateway.IntegrationTests/ZB.MOM.WW.MxGateway.IntegrationTests.csproj --filter FullyQualifiedName~WorkerLiveMxAccessSmokeTests
```

Optional live smoke variables:

| Variable | Default | Description |
|----------|---------|-------------|
| `MXGATEWAY_LIVE_MXACCESS_WORKER_EXE` | First existing `ZB.MOM.WW.MxGateway.Worker.exe` under `src/ZB.MOM.WW.MxGateway.Worker/bin/...` | Worker executable path. Set this when running against a packaged worker or a non-default build output. |
| `MXGATEWAY_LIVE_MXACCESS_ITEM` | `TestChildObject.TestInt` | MXAccess item reference used by `AddItem`. |
| `MXGATEWAY_LIVE_MXACCESS_CLIENT_NAME` | `ZB.MOM.WW.MxGateway.IntegrationTests` | Client name passed to `Register`. |
| `MXGATEWAY_LIVE_MXACCESS_EVENT_TIMEOUT_SECONDS` | `15` | Maximum wait for the first `OnDataChange` (also used for the `OnWriteComplete` round-trip and the abnormal-exit fault transition). |
| `MXGATEWAY_LIVE_MXACCESS_WRITE_SECURED_USER` | `admin` | ArchestrA user name passed to `AuthenticateUser` before the `WriteSecured` parity step. |
| `MXGATEWAY_LIVE_MXACCESS_WRITE_SECURED_PASSWORD` | `admin123` | Password paired with the user above. Never logged; the test asserts the value does not appear in the WriteSecured diagnostic message. |

The test output includes session id, worker process id, command status,
HRESULT/status diagnostics, event sequence and handles, close status, and worker
stdout/stderr lines emitted during the run.

## Dev-rig Probes

`src/ZB.MOM.WW.MxGateway.Worker.Tests/Probes/` partitions runtime probes from the regular
Worker.Tests regression suite. The folder is its own
`ZB.MOM.WW.MxGateway.Worker.Tests.Probes` namespace so a discovery filter (e.g. `dotnet
test --filter FullyQualifiedName~ZB.MOM.WW.MxGateway.Worker.Tests.Probes`) can target or
exclude them without enumerating individual class names. The probes are
`[Fact(Skip = "...")]` by default and exist to characterize live AVEVA
behavior on the dev rig, not to gate CI — flip `Skip = null` on the dev box
with installed MXAccess + a running Galaxy provider when running them:

- `AlarmsLiveSmokeTests` — end-to-end smoke for the alarms-over-gateway
  pipeline (`WnWrapAlarmConsumer` + `AlarmDispatcher` +
  `MxAccessAlarmEventSink`) against `\\<machine>\Galaxy!DEV` with the dev rig's
  10-second flip script writing `TestMachine_001.TestAlarm001`.
- `AlarmClientWmProbeTests` — registers as an `AlarmClient` consumer on a real
  hidden message-only window and logs every Win32 message that arrives during
  a fixed pump window. Used to identify the `WM_APP` /
  `RegisterWindowMessage` IDs alarm callbacks use.
- `WnWrapConsumerProbeTests` — instantiates AVEVA's standalone `wnwrapConsumer`
  COM class, subscribes to the dev rig's `\\<machine>\Galaxy!DEV` provider,
  and polls `GetXmlCurrentAlarms2`. The XML payload bypasses the
  `FILETIME→DateTime` auto-marshaling that crashes
  `aaAlarmManagedClient.AlarmClient.GetHighPriAlarm` on this rig.

The probes share the Worker.Tests project (so they can use its `net48`/`x86`
configuration and the installed `ArchestrA.MxAccess` / `aaAlarmManagedClient`
references), but they are not part of the regression contract — a Worker.Tests
run with `Skip` left in place passes them as skipped.

## Live Galaxy Repository

`GalaxyRepositoryLiveTests` in `src/ZB.MOM.WW.MxGateway.IntegrationTests/Galaxy/` exercises
`GalaxyRepository` directly against the `ZB` Galaxy Repository SQL database. It is
skipped unless `MXGATEWAY_RUN_LIVE_GALAXY_TESTS=1` is set because it depends on a
reachable SQL Server instance and deployed Galaxy state — fake-worker tests cannot
cover the SQL browse RPCs.

The suite covers `TestConnectionAsync`, `GetLastDeployTimeAsync`,
`GetHierarchyAsync`, and `GetAttributesAsync`. `GetHierarchyAsync` and
`GetAttributesAsync` assert a non-empty result, so the connected `ZB` database
must contain a deployed Galaxy, not just an empty schema.

Run the Galaxy live tests explicitly:

```bash
$env:MXGATEWAY_RUN_LIVE_GALAXY_TESTS = "1"
dotnet test src/ZB.MOM.WW.MxGateway.IntegrationTests/ZB.MOM.WW.MxGateway.IntegrationTests.csproj --filter FullyQualifiedName~GalaxyRepositoryLiveTests
```

Optional live Galaxy variables:

| Variable | Default | Description |
|----------|---------|-------------|
| `MXGATEWAY_LIVE_GALAXY_CONN` | `Server=localhost;Database=ZB;Integrated Security=True;TrustServerCertificate=True;Encrypt=False;` | Galaxy Repository connection string. Set this when the `ZB` database is on a non-default instance or needs SQL authentication. |

The default connection string targets `ZB` on `localhost` with Windows
authentication, which matches the Galaxy Repository conventions in CLAUDE.md.

## Galaxy Filter Safety

`GalaxyFilterInputSafetyTests` in `src/ZB.MOM.WW.MxGateway.Tests/Galaxy/` covers adversarial
input handling for the Galaxy Repository browse filter layer. It runs in the
unit-test project (no live SQL needed) and complements the live SQL coverage in
`GalaxyRepositoryLiveTests`.

The test class re-frames the original "Galaxy SQL injection" concern (Tests-002 in
`code-reviews/Tests/findings.md`). `GalaxyRepository` issues only four *constant*
SQL statements (`HierarchySql`, `AttributesSql`, `SELECT 1`,
`SELECT time_of_last_deploy FROM galaxy`) — no `DiscoverHierarchyRequest` field
is ever concatenated into a SQL string, so there is no dynamic SQL surface and no
`LIKE`-escaping helper to test. All filters (`TagNameGlob`, `RootTagName`,
template-chain, category, contained-path) are applied **in memory** by
`GalaxyHierarchyProjector` / `GalaxyGlobMatcher` against the cached snapshot.

The adversarial-input matrix (`'`, `' OR '1'='1`, `'; DROP TABLE gobject;--`,
`%`, `_`, `100%_off`, `[abc]`, `Pump'001`) pins the following invariants:

- SQL metacharacters (`'`, `;`) and `LIKE`-wildcards (`%`, `_`) are treated as
  opaque literals by `GalaxyGlobMatcher` — they never act as wildcards, never
  spuriously match unrelated text.
- Only `*` and `?` are glob wildcards.
- `GalaxyGlobMatcher` applies a 100 ms regex timeout so a pathological glob
  (e.g. 5 000 `a` characters plus a literal `!`) completes promptly rather than
  catastrophically backtracking.
- `GalaxyHierarchyProjector` returns zero matches (rather than the whole
  hierarchy) for an adversarial `TagNameGlob` or `TemplateChainContains`, and
  surfaces `NotFound` for an adversarial `RootTagName`.
- The `DiscoverHierarchy` RPC end-to-end returns zero matches for adversarial
  `TagNameGlob` rather than faulting.

These invariants are the real security surface of the Galaxy browse path; the
SQL-injection framing does not apply to a constant-query layer.

## Live LDAP

`DashboardLdapLiveTests` in `src/ZB.MOM.WW.MxGateway.IntegrationTests/` exercises
`DashboardAuthenticator` against the live GLAuth directory. It is skipped unless
`MXGATEWAY_RUN_LIVE_LDAP_TESTS=1` is set because it binds against the GLAuth
service described in `glauth.md`.

The suite builds the authenticator with `GatewayOptions.Dashboard.GroupToRole`
set to `{ GwAdmin: Admin }`. `GwAdmin` is the gateway-specific
dashboard-admin role and is **not** part of the baseline GLAuth role
groups — it must be provisioned before the LDAP live tests pass.
`AuthenticateAsync_AdminInGwAdminGroup_Succeeds` fails (rather than skips)
when GLAuth has only the baseline groups, so this is a hard prerequisite
beyond "LDAP is up." The shared directory
(`scadaproj/infra/glauth/config.toml`) already provisions `GwAdmin` (gid 5610)
and `GwReader` (gid 5611); see the "Adding a gw-specific group" section of
`glauth.md` for the per-box equivalent.

The fixtures name real users from that shared config, so a run only proves the
service-account bind when it targets the shared directory. `appsettings.json`
ships `Server=localhost` for the local-forward case, so point the suite at the
shared GLAuth with `MxGateway__Ldap__Server=10.100.0.35`; the suite's
`AddEnvironmentVariables()` layer applies the override to the same
`MxGateway:Ldap` section production binds.

`DashboardAuthenticator` delegates the LDAP bind and group search to the shared
`ZB.MOM.WW.Auth.Ldap` provider (`LdapAuthService`) and only maps the resulting
groups to dashboard roles via `DashboardGroupRoleMapper`; the bind/search
mechanics that decide each outcome live in that shared provider, not in
`DashboardAuthenticator`.

The suite covers both the success path and the failure outcomes: `admin`, whose
`othergroups` include `GwAdmin`, succeeds and emits the role claim — this is the
one test that proves the service-account bind, because every other outcome below
fails identically whether or not the bind credential is right; `gw-viewer` is
denied because its only group (`GwReader`) is absent from `GroupToRole`, and its
denial message must match the unknown-user denial so an authorization failure
cannot be used to enumerate valid accounts; `admin` with a wrong password fails
authentication without leaking the password into `FailureMessage`; an unknown
username fails authentication; and an unreachable LDAP server is absorbed into a
failed result rather than throwing. Both live users bind with the shared dev
password documented in `glauth.md`.

`appsettings.json` now ships the LDAP bind password as the unexpanded
`${secret:ldap/mxgateway/bind}` token (resolved at gateway startup by the
pre-host secrets expander, which this suite's bare `ConfigurationBuilder`
does not run). Before running the live LDAP suite, set
`MxGateway__Ldap__ServiceAccountPassword` to the real GLAuth service-account
password so the suite binds with the real password instead of the literal
token. Obtain the current value from the GLAuth source of truth
`scadaproj/infra/glauth/` (per `glauth.md`); it is not committed here.

Run the LDAP live tests explicitly:

```bash
$env:MXGATEWAY_RUN_LIVE_LDAP_TESTS = "1"
$env:MxGateway__Ldap__Server = "10.100.0.35"
$env:MxGateway__Ldap__ServiceAccountPassword = "<service-account-password>"
dotnet test src/ZB.MOM.WW.MxGateway.IntegrationTests/ZB.MOM.WW.MxGateway.IntegrationTests.csproj --filter FullyQualifiedName~DashboardLdapLiveTests
```

## Client Wire Tests

Each client's own suite drives the client's public API against a **fake gateway
served over a real gRPC transport** — an in-process or loopback server
implementing `mxaccess_gateway.v1.MxAccessGateway`. Only the gateway's *behaviour*
is canned; the HTTP/2 framing, protobuf serialization, call metadata, and gRPC
status codes are genuine. That is the difference from the per-client mocks: a mock
substituted for the generated stub (or, in .NET, for `IMxGatewayClientTransport`)
proves what the client *intends* to send, never what a server *receives*, so a
field the client fails to decode or a header it never actually attaches passes
every mock-based test. These tests need no MXAccess, no worker, and no network
beyond loopback, so they run in the default suite on every host.

The shared shape each client's wire test covers:

- **Round trip** — `OpenSession` → `Invoke` (a `Register`, asserting the decoded
  `RegisterReply.server_handle`) → `StreamEvents` (asserting the decoded
  `OnDataChange` fields) → `CloseSession`.
- **Auth on the wire** — the `authorization: Bearer <key>` header is asserted as
  *observed by the server*, on the streaming RPC as well as the unary ones.
- **Replay-gap sentinel** — a stream resumed with `after_worker_sequence` opens
  with the gateway's `replay_gap` sentinel, and the client surfaces it as its
  typed, non-terminal replay-gap signal rather than a normal event.
- **Status mapping** — a real `PERMISSION_DENIED` from the server becomes the
  client's typed authorization error, not a bare transport exception.

Per-client harness and command:

| Client | Harness | Command |
|---|---|---|
| .NET | `WireFakeGatewayServer` (Kestrel h2c on `127.0.0.1:0`, `MxAccessGatewayBase`) — `MxGatewayClientWireTests` | `dotnet test clients/dotnet/ZB.MOM.WW.MxGateway.Client.Tests/ZB.MOM.WW.MxGateway.Client.Tests.csproj` |
| Python | `FakeGateway` + `serve_gateway` fixture (`grpc.aio` server on `127.0.0.1:0`) — `tests/test_wire_fake_gateway.py` | `python -m pytest` from `clients/python` |
| Go | `fakeGatewayServer` + `newBufconnClient` (`grpc.NewServer` over `bufconn`) — `mxgateway/client_session_test.go` | `go test ./...` from `clients/go` |
| Rust | `spawn_fake_gateway` (tonic `Server` over a loopback `TcpListener`) — `tests/client_behavior.rs` | `cargo test --workspace` from `clients/rust` |
| Java | `TestGatewayService` + `InProcessGateway` (`InProcessServerBuilder`) — `MxGatewayClientSessionTests`; plus `InProcessGatewayHarness` for the CLI tests | `gradle test` from `clients/java` |

All five run in CI: Go, Rust, Python, and the .NET client tests in the `portable`
job, Java in the `java` job.

Adding an RPC to `mxaccess_gateway.proto` does not automatically extend these —
the fake gateways implement only the four session RPCs. Extend the fake in the
client whose behaviour changed rather than adding a parallel harness.

## Client E2E Scripts

`scripts/discover-testmachine-tags.ps1` queries the ZB Galaxy Repository for the
deployed runtime references used by the live client e2e scripts. It reads
`TestMachine_001` through `TestMachine_020` and the expected attributes:

- `ProtectedValue`
- `TestChangingInt`
- `TestBoolArray`
- `TestIntArray`
- `TestDateTimeArray`
- `TestStringArray`

The discovery output includes the exact `fullTagReference`, data type, array
dimension, and security classification. The array attributes are expected to be
dimension 50. `ProtectedValue` has security classification 2 and requires
secured write semantics; the current client CLI e2e runner subscribes to it but
does not attempt a normal `Write`.

Run discovery directly when validating the Galaxy Repository inputs:

```powershell
powershell -ExecutionPolicy Bypass -File scripts/discover-testmachine-tags.ps1 -Json
```

`scripts/run-client-e2e-tests.ps1` drives the .NET, Go, Rust, Python, and Java
client CLIs through a live gateway session. The gateway and worker are assumed
to be already running at `-Endpoint`; the script does not start or stop them.
For each client it runs these phases, then closes the session in a `finally`
path and writes a JSON report under `artifacts/e2e/`:

1. **Session + register** — opens one session and registers.
2. **Bulk** — verifies `SubscribeBulk` / `UnsubscribeBulk` on a bounded tag
   subset (skip with `-SkipBulk`).
3. **Add-item / advise** — adds and advises every discovered test tag. The
   loop has no `StreamEvents` consumer attached, so advised tags accumulate
   MXAccess change events in the worker event channel
   (`MxGateway:Events:QueueCapacity`); left unbounded it overflows under
   `FailFast` backpressure and faults the worker. Every `-DrainEveryTags`
   advised tags (default 15) the loop connects a short-lived `StreamEvents`
   drain so the gateway pumps that channel empty. `-DrainEveryTags 0` disables
   the drain.
4. **Stream** — asserts a bounded event stream delivers at least one event
   (skip with `-SkipStream`).
5. **Parity** — asserts MXAccess error paths are rejected rather than silently
   succeeding: an invalid item handle and an unknown session id (skip with
   `-SkipParity`).
6. **Auth rejection** — asserts `open-session` is rejected when the API key is
   missing, and (when `-RejectScopeApiKeyEnv` names an insufficient-scope key)
   when the key lacks the required scope. Skip with `-SkipAuth`.
7. **Write round-trip** — *opt-in (`-VerifyWrite`).* Runs right after
   `register`: adds and advises a configurable writable attribute
   (`-WriteAttribute`, default `TestChangingInt`), writes a per-client
   sentinel value, then streams events and asserts an `OnWriteComplete` event
   for that item is observed — proof the write round-tripped through the
   gateway, worker, and MXAccess provider. The written value being echoed back
   in an `OnDataChange` is recorded best-effort (`echoObserved`): a
   provider-driven attribute such as `TestChangingInt` accepts the write but
   immediately overwrites it, so no data-change carries the value back. The
   Rust `stream-events` CLI emits full per-event JSON (`family`, `itemHandle`,
   `value`) so all five clients apply the same checks.

   It is opt-in because it mutates live tag state. The phase fails fast if the
   write command is rejected — e.g. against a gateway whose worker predates
   write support (`MxAccessCommandExecutor` returning `InvalidRequest` for
   `Write`/`Write2`/`WriteSecured`/`WriteSecured2`).
8. **Alarm feed + acknowledge** — *opt-in (`-VerifyAlarms`).* Runs after the
   stream phase. Exercises the two session-less alarm subcommands against the
   gateway's central alarm monitor: `stream-alarms` reads a bounded slice of
   the feed (`-AlarmStreamMax`, default 1 — the feed's first message always
   arrives immediately, whereas later ones depend on live transitions) and
   asserts at least one `AlarmFeedMessage`; `acknowledge-alarm` acknowledges
   `-AlarmReference` (default `Galaxy!TestArea.TestMachine_001.TestAlarm001`)
   and asserts the RPC round-trips. The native ack outcome is not asserted —
   it depends on whether that alarm is currently active.

   It is opt-in because it depends on the gateway's central alarm monitor
   being enabled (`MxGateway:Alarms:Enabled`) and a live alarm provider.

Each client CLI is driven through one long-lived `batch` process. Every CLI
exposes a `batch` subcommand: a process that reads one command line from stdin,
runs it through the normal subcommand dispatch, writes the JSON result, then a
line containing exactly `__MXGW_BATCH_EOR__`. The harness launches one such
process per client and pings the ~250 operations of the flow through it, so the
process — and, for the JVM, the runtime — cold-start is paid once per client
instead of once per operation. A command that fails inside the batch process
writes its `{"error":...}` envelope and the loop continues; the harness treats
that envelope as the operation failure (used by the parity and auth phases).

Before the per-client phases run, the script builds the .NET CLI
(`dotnet build`) and installs the Java CLI (`gradle :mxgateway-cli:installDist`)
once, so the `batch` process launches straight from the compiled exe / the
installed launcher. The Go, Rust, and Python batch processes are launched via
`go run` / `cargo run` / `python -m`, which compile-or-start once when that
single per-client process starts.

Build the gateway and worker, start the gateway, and provide a valid API key
before running the client e2e script:

```powershell
$env:MXGATEWAY_API_KEY = "<api-key>"
powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1
```

Useful runner options:

```powershell
powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1 -Clients dotnet,python -MachineStart 1 -MachineEnd 2
powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1 -BulkTagCount 10
powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1 -SkipStream
powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1 -SkipBulk
# Write round-trip (opt-in): point at a writable scalar attribute and its
# value type.
powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1 -VerifyWrite -WriteAttribute TestChangingInt -WriteType int32
# Alarm feed + acknowledge (opt-in): needs MxGateway:Alarms:Enabled on the gateway.
powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1 -VerifyAlarms -AlarmReference "Galaxy!TestArea.TestMachine_001.TestAlarm001"
# Auth rejection: also assert an insufficient-scope key is denied.
powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1 -RejectScopeApiKeyEnv MXGATEWAY_READONLY_API_KEY
# Run all five clients concurrently as isolated child processes.
powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1 -Parallel
# Validate the flow offline (prints commands, contacts no gateway).
powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1 -DryRun
powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1 -Endpoint localhost:5000 -ApiKeyEnv MXGATEWAY_API_KEY
```

When `-VerifyWrite` is enabled, the write round-trip fails loudly if the write
command is rejected, if `-WriteAttribute` does not name a writable scalar
attribute, or if no `OnWriteComplete` event is observed for the written item
within `-WriteEchoMaxEvents` (default 200) streamed events. Raise
`-WriteEchoMaxEvents` if the gateway's per-session event backlog is large
enough to push `OnWriteComplete` past that bound.

## Focused Commands

Run the cross-language smoke matrix tests after changing the documented client
smoke command list:

```bash
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter FullyQualifiedName~CrossLanguageSmokeMatrixTests
```

Run the parity fixture matrix tests after changing the integration parity
scenario list:

```bash
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter FullyQualifiedName~ParityFixtureMatrixTests
```

Run the fake worker tests after changing gateway worker IPC, session startup, or
event streaming behavior:

```bash
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter FullyQualifiedName~FakeWorkerHarnessTests
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter FullyQualifiedName~SessionWorkerClientFactoryFakeWorkerTests
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter FullyQualifiedName~GatewayEndToEndFakeWorkerSmokeTests
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter FullyQualifiedName~WorkerClientTests
dotnet test src/ZB.MOM.WW.MxGateway.Worker.Tests/ZB.MOM.WW.MxGateway.Worker.Tests.csproj -p:Platform=x86 --filter FullyQualifiedName~WorkerPipeSessionTests
```

Run the gateway test project after shared gateway test infrastructure changes:

```bash
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj
```

## Running the Gateway Suite on windev

The gateway suite (`ZB.MOM.WW.MxGateway.Tests`, net10.0/x64) is not part of the CI
Windows tier — `windows-x86` and `nightly-windev` run only the x86 Worker build and
`Worker.Tests`. It is still run on windev by hand when a change needs Windows
confirmation, and that run has three Windows-specific characteristics worth knowing
before results are interpreted.

Run it from an isolated clone under `C:\build` checked out to the SHA under test — never
the dirty Desktop checkout, and never the CI clone `C:\build\mxaccessgw-ci`, whose worktree
lock belongs to the Worker tier.

Baseline on an otherwise idle windev (2026-08-10): **879 passed, 0 failed, 31 s** — the same
879 the macOS box runs, with nothing gated away. Any failure is therefore a real signal, but
read the load caveat below before acting on one.

Runs before the pipe-buffer fix below reported 855, which was long read as "windev runs a
smaller suite because some cases are gated to Unix". It was not: 855 is simply what had been
flushed when the wedged host was torn down. Do not treat a short count on this suite as
platform gating.

### Two long-standing "windev-environmental" failures were test bugs, not the environment

Both were dismissed as environmental for months and are now fixed. Neither depended on
anything installed on windev; both failed on **any** Windows host:

- `SelfSignedCertificateProviderTests.GenerateCertificate_HasExpectedSansEkuAndValidity`
  asserted SAN content by substring-matching `X509Extension.Format(false)`. That string is
  produced by the platform crypto library: Windows' `CryptFormatObject` renders the IPv6
  loopback fully expanded (`IP Address=0000:0000:0000:0000:0000:0000:0000:0001`) while the
  managed formatter used on macOS/Linux renders `::1`, so the loopback assertion failed on
  Windows only. The test now decodes the extension with `X509SubjectAlternativeNameExtension`
  and compares parsed `IPAddress` values and DNS names (case-insensitively, as DNS names
  are), which is platform-independent.
- `SessionManagerTests.OpenSessionAsync_PipeNameIsShortAndUniquePerPidAndSession` guards the
  104-byte macOS `sun_path` budget that NEXT-01 shortened the pipe name to fit. It padded the
  measured name up to a five-digit pid but never substituted that worst case *downward*, so a
  six-digit pid — routine on Windows, impossible on macOS, where pids stop at 99999 — made the
  name one character "too long" against a budget that does not apply to the host running the
  test. The check now replaces the running pid's digit count with the five-digit macOS worst
  case, so it measures the name *format* rather than the current process's pid.

### The real-pipe suites are load-sensitive

These suites drive real named pipes against a five-second worker startup timeout and start
failing when windev is busy — most often when the x86 Worker tier is building or testing at
the same time. All five passed in the idle baseline above and all five failed in a run taken
while an x86 build and `Worker.Tests` were in flight (that run also took 2 m 21 s against the
idle half-minute):

- `GatewayEndToEndFakeWorkerSmokeTests`, `GatewayEndToEndMultiSubscriberTests`,
  `GatewayEndToEndReconnectReplayTests` — fail as
  `RpcException Status(StatusCode="Unavailable", Detail="Failed to open session …")`.
- `SessionWorkerClientFactoryFakeWorkerTests.CreateAsync_WhenFakeWorkerStartupFails_ThrowsWorkerClientException`
  — the startup timeout beats the protocol violation the test is asserting, so the observed
  exception is `TimeoutException` instead of `WorkerClientException`.
- `WorkerClientTests.InvokeAsync_WhenCommandExceedsFrameMax_FailsOnlyThatCommandAndStaysReady`.
- `EventStreamServiceTests.StreamEventsAsync_WithConcurrentStreams_TracksAggregateQueueDepth`
  — polls a metric against a five-second deadline. Its helper now reports the unmet condition
  rather than letting a bare `TaskCanceledException` escape, so a load-induced timeout here
  names what it was waiting for instead of looking like an unexplained cancellation.

A failure in that list is evidence about machine load, not about the change under test. Check
for a concurrent x86 build/test (`Get-Process dotnet, testhost, testhost.net48.x86,
MSBuild, VBCSCompiler`) and re-run the affected class on its own before treating it as real.
windev has 36 logical CPUs and `xunit.runner.json` sets `maxParallelThreads: -1`, so the
suite runs far wider there than on the macOS dev box — that width is what turns these
real-clock deadlines into failures.

### Two more findings from the 2026-08-15 windev gate

- `SecretsStorePathGuardTests.CreateBuilder_AcceptsSecretsStoreOutsideContentRoot_AndCreatesIt`
  fails **deterministically on Windows, on `main` as well as on any branch**, so it is not a
  signal about the change under test. Creating the builder opens `secrets.db`, and
  `Microsoft.Data.Sqlite`'s connection pool keeps the file handle alive past the test body,
  so the recursive directory delete in the cleanup hits a still-open file — a sharing
  violation Windows enforces and Unix does not. Pre-existing and tracked separately; do not
  chase it as a regression. Subtract it from the expected pass count on Windows.
- The `StaWaitHelper` timing tests (`WaitForSignalOrMessages_*`) flake on a loaded box with a
  signature that reads like a broken wait but is not: the helper wakes on *input being
  present*, so a message posted to the test thread ends the wait early. That is the helper
  doing exactly what the STA pump needs. The tests drain the queue with
  `PumpPendingMessages()` first for that reason; a failure here means the box was busy enough
  to queue a message mid-test, not that the wait stopped honouring its handle or its timeout.
  Re-run the class on its own before treating it as real, per the load caveat above.

### The full-suite testhost hang was a zero-buffer named pipe (fixed)

For months a full-suite run on windev reported `855 passed, 0 failed` and then never
returned: the x64 `testhost` stopped consuming CPU but stayed alive indefinitely, and the run
had to be killed with `--blame-hang`. That guard is no longer needed — run the suite plainly:

```powershell
dotnet test src\ZB.MOM.WW.MxGateway.Tests\ZB.MOM.WW.MxGateway.Tests.csproj
```

The cause is worth recording because the shape of it is easy to hit again.

`dotnet-stack report` on the wedged host showed no thread running test code; xUnit's
`RunTestsInAssembly` was simply parked on `WaitHandle.WaitOne()` waiting for the
assembly-finished event. The wait was therefore in a suspended async state machine, which only
`dotnet-dump analyze <dump> -c dumpasync` can see. It named the exact frame:
`WorkerClientTests.StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout`
awaiting `WorkerFrameWriter.WriteAsync` — a 63-byte pipe write that never completed. The `855`
was never the whole suite: the same clone now reports 879, so the wedge was also costing 24
results, and the summary still looked clean because the hung test is not counted as a failure.

That test pushes events past the worker client's staging bound to prove the client faults, and
after the fault the client's read loop stops reading by design. The test-side pipe was created
through `NamedPipeServerStream(string, PipeDirection, int, PipeTransmissionMode, PipeOptions)`,
whose omitted buffer arguments become `inBufferSize: 0` / `outBufferSize: 0`. On Windows that
reserves *no* buffer: a write completes only when the peer reads it. Measured directly on
windev, that pipe absorbed **0 bytes** before blocking against a non-reading peer, while the
same pipe declared with 64 KiB buffers absorbed **65 520**. On macOS and Linux .NET backs named
pipes with Unix domain sockets, whose socket buffer swallows a few kilobytes regardless — which
is why the identical test never hung there, and why the bug read as "a windev thing".

Two changes make it structural rather than incidental:

- Test-owned server pipes are created through `TestSupport/TestNamedPipe.CreateServer`, which
  declares explicit 64 KiB buffers, in both the gateway and worker test projects. This scopes
  those tests to the backpressure they are actually asserting — the gateway's staging and event
  queues — instead of the OS pipe's flow control.
- Every fake-worker write in `WorkerClientTests` goes through `PipePair.WriteAsync`, which
  bounds the write by the class's five-second `TestTimeout` and fails with a message naming the
  stopped reader. A blocked write is now a named test failure rather than a silent wedge.

The severity came from the second point being missing, not the first. A test method that never
returns keeps xUnit from raising `ITestAssemblyFinished`, so the runner waits forever and
`testhost` never exits — one unbounded `await` in one test costs the entire suite its result.
Any new test that writes to a pipe whose reader may stop must bound the write.

The gateway's production pipe in `SessionWorkerClientFactory.CreatePipe` deliberately keeps the
unbuffered declaration: both ends run continuous read loops and every write there is bounded by
the worker client's `_stopCts`, so a stalled peer cancels the write instead of blocking on it.

## Continuous Integration

CI runs on Gitea Actions (`.gitea/workflows/ci.yml`; origin is Gitea at
`gitea.dohertylan.com`) on every push and pull request. The pipeline is split by
runtime because the x86 Worker cannot build on Linux:

- **`portable`** (Linux runner) — builds `src/ZB.MOM.WW.MxGateway.NonWindows.slnx`,
  runs the codegen/descriptor freshness guard (`scripts/check-codegen.ps1`), runs the
  gateway fake-worker tests, and builds/tests the clients that run on Linux: .NET client
  build, Go (`gofmt` + `go build` + `go test`), Rust (`cargo fmt --check` + `cargo test`
  + `cargo clippy -D warnings`), and Python (`pytest`).
- **`java`** (Linux, JDK 17) — `gradle test`, then `git diff --exit-code` over the generated
  tree. The grpc/protobuf toolchain is pinned (`clients/java/build.gradle`), so a regeneration
  is byte-identical to the committed single-file aggregates modulo real `.proto` changes; the
  git-diff is therefore a true drift gate that now catches message-level proto drift in the Java
  client (IPC-24 deleted the old unconditional churn-revert step, which masked exactly that
  class). The dev Mac has a Homebrew JDK 17, so `gradle generateProto` can be run there to refresh
  the Java aggregates when a `.proto` changes.
- **`windows-x86`** (Linux runner, per push/PR) — builds the **x86 / net48 Worker and
  Worker.Tests**, which are Windows-only and out of scope for the Linux jobs. It runs on a
  Linux runner that always schedules and SSHes to windev (`10.100.0.48`), where
  `scripts/ci/run-windev-ci.sh` fast-forwards the isolated CI clone `C:\build\mxaccessgw-ci`
  to the SHA under test and runs `scripts/ci/windev-worker-ci.ps1 -Mode test` there; the
  remote exit code propagates back, so a Worker regression turns the job red. A native
  Windows runner is not used because act_runner host-mode on Windows is broken and a
  `runs-on` gate with no runner wedges the queue forever. **A red `windows-x86` can mean
  the tier is down, not the change**: an unreachable windev fails the job (never skips), so
  if it goes red without a related code change, check the tier — `ssh windev`, read
  `C:\build\mxaccessgw-ci\ci.log` — before assuming a real test failure.
- **`nightly-windev`** (Linux runner, scheduled `0 6 * * *`) — re-runs the full x86 build +
  Worker.Tests on windev, then the live-MXAccess smoke (`MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`)
  and a full `slnx` build (which surfaces Windows-only analyzer diagnostics such as
  `xUnit1030`). Gated `if: github.event_name == 'schedule'`, so it never gates a push. On
  failure it opens a Gitea issue via the Actions token, since nobody watches the Actions page.

### Runner capacity is shared and finite

CI runs on two co-located runner containers on docker host `10.100.0.35` — `gitea-runner`
(capacity 4) and `gitea-runner-2` (capacity 2, registered 2026-08-07 per
`docs/runbooks/TST-30-second-ci-runner.md`) — and both runner instances are **shared across
repos**: they interleave `dohertj2/mxaccessgw` and `dohertj2/lmxopcua` jobs across the
combined slots rather than being scoped to this repo (`GET
/repos/dohertj2/mxaccessgw/actions/runners` returns `total_count: 0`; both runners are
registered at the instance level). Every job in a run (`portable`, `java`, `windows-x86`)
still executes serially within that run, so queue latency is additive within a run, but an
active `lmxopcua` run no longer blocks `mxaccessgw` entirely the way a single shared slot
did — the two runners relieve cross-repo contention. This Gitea version (1.26) also exposes
**no run cancel or delete via the API** (`POST .../actions/runs/{id}/cancel` returns 404,
`DELETE .../actions/runs/{id}` returns 400), so a superseded or hung run cannot be cleared
and holds the slot until it finishes or times out — with two runners this means a single
wedged run can still hold slots, because the no-cancel reality is unchanged. See
`docs/runbooks/TST-30-second-ci-runner.md` for the operator runbook that registered the
second runner.

When queue depth (or the missing-cancel reality) makes waiting impractical, verify a
specific commit out of band instead of waiting behind the queue: run
`CI_SHA=<sha> scripts/ci/run-windev-ci.sh <build|test|live>` from a machine with SSH access
to windev (the same script the SSH-driven `windows-x86`/`nightly-windev` jobs use — see
`scripts/ci/README.md`), or fall back to the manual windev worktree procedure below. This is
the same escape hatch used when the windev tier itself is down — TST-30 generalizes it from
"tier down" to "runner contended": either way, a stuck or slow shared runner should not
block verifying a commit.

The freshness guard `scripts/check-codegen.ps1` runs four checks and fails the build when the
committed client descriptor set (Check 1), the C# `Generated/` (Check 2), the Rust vendored
protos (Check 3), or the Go/Python client bindings (Check 4, IPC-25) no longer match the current
`.proto` sources — the codegen drift class this repo has hit repeatedly (stale client
descriptors, net48 `CS0246` on unregenerated protos, silently stale Go/Python worker bindings).
Check 4 regenerates the Go and Python bindings with their pinned generators (`protoc-gen-go`
v1.36.11 / `protoc-gen-go-grpc` 1.6.2, `grpcio-tools` 1.80.0) and fails on any diff; a missing
generator fails the check rather than skipping it. The **primary** guard for the
"regenerate and commit `Generated/`" rule is that regeneration diff in the `portable` job;
the `windows-x86` net48 compile is the **secondary** guard (a stale `Generated/` also breaks
the x86 build with `CS0246`). See [Client Proto Generation](./ClientProtoGeneration.md) and
[Contracts](./Contracts.md).

If the SSH-driven Windows tier is unavailable for infrastructure reasons (windev down, CI
key/secret rotation in flight) **or** the shared Gitea runner is contended and the queue is
impractical to wait behind (see "Runner capacity is shared and finite" above), fall back to
the manual windev worktree procedure as a degraded mode: on windev, fast-forward an isolated
`origin/main` worktree under `C:\build` (never the dirty Desktop checkout), then run the x86
Worker build and `Worker.Tests` (`-p:Platform=x86`) there by hand. Do this per merge for
worker-touching changes until the `windows-x86` job is green again (tier-down case) or the
queue clears (contention case).

## Related Documentation

- [Cross-Language Smoke Matrix](./CrossLanguageSmokeMatrix.md)
- [Parity Fixture Matrix](./ParityFixtureMatrix.md)
- [Gateway Process Design](./GatewayProcessDesign.md)
- [Worker Frame Protocol](./WorkerFrameProtocol.md)
- [MXAccess Worker Instance Detailed Design](./MxAccessWorkerInstanceDesign.md)