Compare commits

...

8 Commits

Author SHA1 Message Date
Joseph Doherty e2352d1666 test(windev): fix the two Windows-only gateway test failures dismissed as environmental
ci / java (push) Successful in 2m24s
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 3m33s
ci / portable (push) Successful in 8m13s
Both failures in the windev baseline were test bugs that reproduce on any Windows
host, not anything missing or misconfigured on windev.

SelfSignedCertificateProviderTests.GenerateCertificate_HasExpectedSansEkuAndValidity
asserted SAN content by substring-matching X509Extension.Format(false). That string
comes from the platform crypto library: Windows' CryptFormatObject renders the IPv6
loopback fully expanded (0000:0000:...:0001) where the managed formatter renders
"::1", so the loopback assertion could never hold on Windows. Decode the extension
with X509SubjectAlternativeNameExtension and compare parsed IPAddress values and DNS
names instead, which removes the platform-dependent formatting from the assertion.

SessionManagerTests.OpenSessionAsync_PipeNameIsShortAndUniquePerPidAndSession guards
the 104-byte macOS sun_path budget NEXT-01 shortened the pipe name to fit. It padded
the measured name up to a five-digit pid but never substituted that worst case
downward, so Windows' routine six-digit pids over-counted by a character against a
budget that does not constrain the host running the test. Substitute the five-digit
macOS worst case for the running pid's digit count so the check measures the name
format rather than the current pid.

EventStreamServiceTests.WaitUntilAsync now reports the unmet condition on timeout
instead of letting a bare TaskCanceledException escape. Its five-second real-clock
deadline is genuinely load-sensitive on windev (36 logical CPUs, maxParallelThreads
-1), and an opaque cancellation there is exactly what got the previous failures
filed as "environmental" and left unexplained.

Documents the windev run in docs/GatewayTesting.md: the two fixed bugs and their root
causes, the real-pipe suites whose failures are evidence of machine load rather than
of the change under test, and the full-suite testhost that completes every test and
then never exits (filtered runs exit normally; macOS exits cleanly). Corrects the
CLAUDE.md claim that the suite exits cleanly on the Windows dev box.
2026-08-10 09:05:38 -04:00
Joseph Doherty 347d59fc62 Merge fix/deflake-eventburst: flush-edge sampling replaces wall-clock window in EventBurst_DrainLoopCoalescesFlushes
ci / java (push) Successful in 2m13s
ci / nightly-windev (push) Has been skipped
ci / portable (push) Successful in 9m25s
ci / windows-x86 (push) Failing after 20m21s
2026-08-10 08:53:07 -04:00
Joseph Doherty 404c06b2fa Merge feat/tst-24-client-wire-tests: .NET/Python real-server wire tests, Python channel-loop fix, TST-05 revisit (partial)
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m45s
ci / portable (push) Successful in 11m39s
ci / windows-x86 (push) Failing after 21m2s
# Conflicts:
#	archreview/remediation/00-tracking.md
2026-08-10 08:23:39 -04:00
Joseph Doherty a8f86b5336 test(tst-24): drive the .NET and Python clients against real in-process gRPC servers
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m19s
ci / portable (push) Successful in 23m59s
ci / windows-x86 (push) Failing after 58m42s
TST-24 asked for per-client wire tests against a fake gateway. An audit first
corrected the finding's premise: Go, Rust, and Java already had them — bufconn,
a loopback tonic server, and InProcessServerBuilder respectively — each already
asserting the round trip, the server-observed bearer header, and the ReplayGap
sentinel. The two genuine gaps were .NET (every test substituted the transport
interface; the test project had no server package at all) and Python (stub
monkeypatching everywhere but one opt-in TLS test).

Both now serve mxaccess_gateway.v1.MxAccessGateway over a real transport —
Kestrel h2c and grpc.aio, each on an ephemeral loopback port — and drive the
ordinary public client API against it. Only the gateway's behaviour is canned;
the framing, serialization, metadata, and status codes are genuine. Four shapes
each: full round trip with every reply field asserted, the authorization header
as received by the server (including on the streaming RPC), the ReplayGap
sentinel surfaced as the client's typed signal, and a real PERMISSION_DENIED
mapping to the typed authorization error.

The .NET client was only ever compiled in CI, never tested, so the portable job
gains a dotnet test step.

Fixes a bug the new tests caught on their first run: Python's connect() built the
grpc.aio channel inside asyncio.to_thread, and a grpc.aio channel binds to the
event loop current on the constructing thread, so every non-stub connection
raised 'There is no current event loop in thread'. No mock-based test could see
it, and the test guarding the off-loop behaviour patched create_channel and so
asserted the bug. Split resolve_channel_security (blocking TOFU probe, off-loop)
from create_channel (on-loop); the guard tests now assert both halves.
2026-08-10 08:22:25 -04:00
Joseph Doherty 50322bacb3 test(worker): deflake EventBurst_DrainLoopCoalescesFlushes
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m13s
ci / java (push) Successful in 2m38s
ci / portable (push) Successful in 8m17s
Root cause. The test sampled its flush baseline after a bare `Task.Delay(100)`
and then charged every later flush to the 128-event burst. Two facts make that
window unsound:

1. `RunHeartbeatLoopAsync` sends its first beat *immediately* on entering the
   message loop — the far-off `HeartbeatInterval` only spaces later beats — so
   the pre-burst frames are WorkerHello, WorkerReady, and a heartbeat, not the
   two the test's comment assumed.
2. `WorkerFrameWriter.DrainQueuedFramesAsync` flushes *after* writing a drained
   batch, so the bytes reach the pipe before the flush runs. Reading a frame off
   the gateway side is therefore no evidence that its flush has been counted,
   and no sleep makes it evidence.

Under load on the shared windows-x86 runner the first heartbeat's flush was
scheduled after the 100 ms sample, so it landed inside the measured window and
the assertion saw two flushes for the burst — exactly the observed
`Expected: 1 / Actual: 2` (Gitea run 675, job 2558, and the same failure since
a346d51). Nothing about the coalescing behavior was wrong; only the test's
timing assumption.

Fix. Replace the sleep with explicit synchronization, no widened timeouts.
`FlushCountingPassthroughStream` now records the flush *shape* — the number of
stream writes coalesced into each flush — and exposes
`WaitForAllWritesFlushedAsync`, a TaskCompletionSource signal released when the
next flush drains the pending writes. The test reads the first heartbeat (the
last pre-burst frame), waits for its flush, and only then samples the baseline;
after the burst it takes the same edge before asserting. The assertion is also
sharpened from a bare count to the shape: exactly one flush beyond the baseline
*and* that flush carried all 128 event frames, so a split batch fails even if
the reader observes it mid-split.

docs/WorkerFrameProtocol.md notes the peer-visible ordering the fix turns on:
frames reach the pipe before the flush that follows them, so an observer of the
flush must wait for it rather than infer it from frames arriving.
2026-08-10 08:13:30 -04:00
Joseph Doherty 9e6f66dd8f Merge fix/tst25-nightly-issue-path: reachable run links in nightly failure issues (TST-25 Check 6 closed)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Failing after 1m16s
ci / java (push) Successful in 2m13s
ci / portable (push) Successful in 8m46s
2026-08-10 08:11:42 -04:00
Joseph Doherty 30c92e8e59 fix(ci): give the nightly-windev issue body a reachable run link (TST-25 Check 6)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Failing after 1m13s
ci / java (push) Successful in 2m7s
ci / portable (push) Successful in 8m30s
Exercising TST-25 acceptance Check 6 end-to-end showed the on-failure issue
step works — it has filed an issue on every red nightly since 2026-07-17
(#126-#139) — but that every one of those issues links `http://gitea:3000`,
the runner-internal origin Gitea hands the runner. That URL is correct for
the issue-creation API call (the job container resolves `gitea` only on the
docker network and has no LAN egress to the public origin) and unreachable
for anyone clicking out of the issue.

Keep `github.server_url` for the API call; add a `PUBLIC_SERVER_URL` job env
used only for the browser-facing link in the body. Verified with a
forced-failure probe run (677) whose issue (#140) carries a public link that
returns 200.

Also records Check 6 as done in scripts/ci/README.md and corrects the
2026-07-13 tracking entry that wrote the check off as abandoned.
2026-08-10 08:10:08 -04:00
Joseph Doherty d4302c6ac4 docs(tst-05): revisit under the restored windev tier — scheduled half closed, control-command coverage still open
The nightly-windev job (cycle-2 TST-25) discharges TST-05's scheduling design:
cron 0 6 * * * runs run-windev-ci.sh live, which sets
MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1, runs WorkerLiveMxAccessSmokeTests on windev,
and opens a Gitea issue on failure. That converts 'run by memory' into 'runs
nightly, reports failures'.

The finding's other half — audit the live suite for coverage of each of the eleven
late-added command kinds — is not discharged. The audit's answer is negative for
five: the suite covers all six COM commands and none of the five control commands
(Ping/GetSessionState/GetWorkerInfo/DrainEvents/ShutdownWorker), which are exactly
the kinds the finding calls masked by FakeWorkerHarness canned replies. Recorded as
Partially done with the residual test work specified, rather than claiming closure.
2026-08-10 08:08:01 -04:00
23 changed files with 1319 additions and 115 deletions
+15 -1
View File
@@ -83,6 +83,12 @@ jobs:
- name: .NET client
run: dotnet build clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx -c Release
# The .NET client was previously only compiled here, so its test project — including
# MxGatewayClientWireTests, which drives the client against a real loopback gRPC
# server (TST-24) — never ran in CI. Every other client job already runs its tests.
- name: .NET client tests
run: dotnet test clients/dotnet/ZB.MOM.WW.MxGateway.Client.Tests/ZB.MOM.WW.MxGateway.Client.Tests.csproj -c Release --no-build
- name: Go client
working-directory: clients/go
run: |
@@ -165,6 +171,14 @@ jobs:
# visible even though nobody watches the Actions page.
if: github.event_name == 'schedule'
runs-on: ubuntu-latest
env:
# `github.server_url` is the URL Gitea hands the runner — the docker-network-internal
# `http://gitea:3000`. That is the right base for the issue-creation API call below (the job
# container resolves `gitea` only on that network, and has no LAN egress to the public origin),
# but it is useless as a link a human clicks out of the issue. So browser-facing URLs in the
# issue body use the public origin instead. TST-25 acceptance Check 6 caught this: every
# nightly issue since #126 carried an unreachable `http://gitea:3000/...` run link.
PUBLIC_SERVER_URL: https://gitea.dohertylan.com
steps:
- uses: actions/checkout@v4
- name: Full Worker.Tests + live-MXAccess smoke on windev
@@ -184,7 +198,7 @@ jobs:
-H "Authorization: token ${{ github.token }}" \
-H "Content-Type: application/json" \
"${{ github.server_url }}/api/v1/repos/${{ github.repository }}/issues" \
-d "{\"title\":\"nightly-windev failed on ${{ github.sha }}\",\"body\":\"The scheduled windev Worker + live-MXAccess run failed: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} . A red nightly may mean the Windows tier is down rather than the change — see docs/GatewayTesting.md (Continuous Integration).\"}"
-d "{\"title\":\"nightly-windev failed on ${{ github.sha }}\",\"body\":\"The scheduled windev Worker + live-MXAccess run failed: ${PUBLIC_SERVER_URL}/${{ github.repository }}/actions/runs/${{ github.run_id }} . A red nightly may mean the Windows tier is down rather than the change — see docs/GatewayTesting.md (Continuous Integration).\"}"
# NOTE: there is intentionally no native `windows` runner job. act_runner v0.6.1 host-mode on
# Windows is broken and Windows containers are impractical for the net48/x86/MXAccess Worker, so the
+1 -1
View File
@@ -124,7 +124,7 @@ powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1
When source code changes, build and test the affected component before reporting work done. If the change crosses component boundaries, build each affected component — don't rely on a single top-level build:
**Run targeted tests per task, never the full suite each time.** When executing a plan task-by-task, run only the tests that exercise the code that task touched (`dotnet test --filter "FullyQualifiedName~<TestClass>"`, or the per-task test named in the plan). The full gateway suite is slow — run it at most once per phase (after a related batch of tasks lands), not after every task. This is a speed guideline, not a correctness one: the suite exits cleanly (verified on macOS and the Windows dev box — 0 surviving `testhost`/worker processes after a full run), so filtered runs are about turnaround, not about avoiding a process leak.
**Run targeted tests per task, never the full suite each time.** When executing a plan task-by-task, run only the tests that exercise the code that task touched (`dotnet test --filter "FullyQualifiedName~<TestClass>"`, or the per-task test named in the plan). The full gateway suite is slow — run it at most once per phase (after a related batch of tasks lands), not after every task. This is a speed guideline on macOS, where the suite exits cleanly (0 surviving `testhost`/worker processes after a full run) — filtered runs there are about turnaround, not about avoiding a process leak. **On windev a full-suite run wedges**: every test completes, then the x64 `testhost` never exits and `dotnet test` never returns (filtered runs exit normally). Run the full suite there with `--blame-hang --blame-hang-timeout 5m --blame-hang-dump-type none` and read the summary line, not the exit code — see "Running the Gateway Suite on windev" in `docs/GatewayTesting.md`.
| Changed area | Required verification |
|---|---|
@@ -150,7 +150,7 @@ Sequence these together rather than piecemeal — several are one change set spa
- Close **CLI-24** and **CLI-34** as `Done` (incidentally fixed; evidence in [../50-clients.md](../50-clients.md)).
- ~~When CLI-38 lands, close old **CLI-08** with a pointer here.~~ Done 2026-08-07: CLI-38 landed and old CLI-08 is now `Done` in the first-cycle tracker, pointing at [CLI-38](50-clients.md#cli-38--align-netgojava-on-hresult--0-lands-prior-cli-08-cures-the-doc-drift---medium--p1).
- ~~When WRK-26 lands, its doc section also discharges the WorkerFrameProtocol gap~~ Done 2026-08-07: WRK-26 landed and **IPC-29** is discharged, pointing at [WRK-26](20-worker.md#wrk-26--write-priority-and-overflow-doc-drift-from-the-wrk-07-change---low--p1). When TST-25 lands, revisit old **TST-05** (scheduled live smoke) and **TST-24** (client wire tests), which it unlocks.
- ~~When WRK-26 lands, its doc section also discharges the WorkerFrameProtocol gap~~ Done 2026-08-07: WRK-26 landed and **IPC-29** is discharged, pointing at [WRK-26](20-worker.md#wrk-26--write-priority-and-overflow-doc-drift-from-the-wrk-07-change---low--p1). ~~When TST-25 lands, revisit old **TST-05** (scheduled live smoke) and **TST-24** (client wire tests), which it unlocks.~~ TST-05 revisited 2026-08-10: `Partially done` in the first-cycle tracker — the `nightly-windev` job closes the scheduled-cadence half, but the finding's coverage-audit half stays open (the live suite reaches all six late-added COM commands and none of the five control commands). TST-24 revisited in the same change and closed `Done`: Go/Rust/Java already had real-server wire tests, and the two genuine gaps (.NET, Python) now have them plus a CI step.
## Change log
@@ -159,7 +159,7 @@ Sequence these together rather than piecemeal — several are one change set spa
| 2026-07-13 | Initial tracking doc generated from the six domain remediation designs. All 47 findings `Not started` (IPC-31, SEC-35 `N/A`). |
| 2026-07-13 | TST-25/TST-26 → `In progress` (branch `fix/tst-25-windev-ci`). Added `scripts/ci/{windev-worker-ci.ps1,run-windev-ci.sh,windev.known_hosts}`, `windows-x86` (per-push) + `nightly-windev` (scheduled) jobs in `ci.yml`, and the TST-26 doc/comment fixes (GatewayTesting.md, Contracts.md, check-codegen.ps1). Mechanism hand-verified on windev: `build`→0, bogus-SHA→nonzero (lock released), `test`→356 passed/0 failed in ~50s (per-push stays `test`, no demotion), and run-windev-ci.sh SSH+EncodedCommand exit-code propagation confirmed. |
| 2026-07-13 | Operator bring-up complete: dedicated CI ed25519 key installed in windev `administrators_authorized_keys` (authorized into `dohertj2`, which owns the working MXAccess/toolchain env — a fresh OS account would break the build; the key is independently revocable), Gitea secrets `WINDEV_SSH_KEY`/`WINDEV_SSH_KNOWN_HOSTS` + variable `WINDEV_SSH_USER=dohertj2` stored, runner→`10.100.0.48:22` egress verified on the `traefik` net, issue-write confirmed. **TST-25/TST-26 → `Done`:** credentialed `windows-x86` ran GREEN on `d769244` (Gitea run #37) — Linux runner SSHed windev, checked out the SHA in `C:\build\mxaccessgw-ci` under lock, ran the x86 Worker build + `Worker.Tests`, exit 0; `nightly-windev` correctly skipped on the push event. Branch merged to `main`. Follow-ups (old tracker): revisit **TST-05** (scheduled live smoke — now covered by `nightly-windev`) and **TST-24** (client wire tests) which this unlocks. |
| 2026-07-13 | Ran the TST-25 acceptance checks (scripts/ci/README.md) — they caught **two real CI defects, both fixed** on `fix/tst-25-ci-key-log-leak`: (1) **CI SSH key leaked in cleartext** in the `windows-x86` step env echo (Gitea's line-oriented masker missed the multiline PEM) — rotated the CI key on windev (old pubkey revoked), stored the key **base64-encoded** so the masker redacts it to `***` (confirmed on run #38), taught `run-windev-ci.sh` to decode, dropped the redundant public known-hosts secret from the job env; (2) **bootstrap lock race** — `run-windev-ci.sh`'s pre-hand-off `git fetch`/`checkout` ran outside the worktree lock, so concurrent runs collided on `.git/index.lock`; the bootstrap now holds the lock (ps1 re-uses it via `MXGW_CI_LOCK_HELD`), retest confirmed clean serialization. Also **deflaked** `SessionManagerTests` fail-fast timing assertions (absolute `<100ms` wall-clock bound flaked under CI load; now anchored to the configured timeout / dropped for the zero-timeout case). Checks passed: unreachable-host fast-fail (exit 255/15s), deliberate-red propagation (Worker.Tests failure → exit 1), lock concurrency (2nd run waits), no-key-in-logs (masked). Merge target `df7e20d` verified GREEN via the local windev path (Worker build + 356 tests); merged to `main` `19cbf7b`. Check 6 (forced-failure nightly issue): issue endpoint+token proven live at bring-up (#124); in-CI forced-failure probe abandoned to shared-runner congestion (residual `if: failure()` gating is standard Actions). |
| 2026-07-13 | Ran the TST-25 acceptance checks (scripts/ci/README.md) — they caught **two real CI defects, both fixed** on `fix/tst-25-ci-key-log-leak`: (1) **CI SSH key leaked in cleartext** in the `windows-x86` step env echo (Gitea's line-oriented masker missed the multiline PEM) — rotated the CI key on windev (old pubkey revoked), stored the key **base64-encoded** so the masker redacts it to `***` (confirmed on run #38), taught `run-windev-ci.sh` to decode, dropped the redundant public known-hosts secret from the job env; (2) **bootstrap lock race** — `run-windev-ci.sh`'s pre-hand-off `git fetch`/`checkout` ran outside the worktree lock, so concurrent runs collided on `.git/index.lock`; the bootstrap now holds the lock (ps1 re-uses it via `MXGW_CI_LOCK_HELD`), retest confirmed clean serialization. Also **deflaked** `SessionManagerTests` fail-fast timing assertions (absolute `<100ms` wall-clock bound flaked under CI load; now anchored to the configured timeout / dropped for the zero-timeout case). Checks passed: unreachable-host fast-fail (exit 255/15s), deliberate-red propagation (Worker.Tests failure → exit 1), lock concurrency (2nd run waits), no-key-in-logs (masked). Merge target `df7e20d` verified GREEN via the local windev path (Worker build + 356 tests); merged to `main` `19cbf7b`. Check 6 (forced-failure nightly issue): issue endpoint+token proven live at bring-up (#124); in-CI forced-failure probe abandoned to shared-runner congestion (residual `if: failure()` gating is standard Actions). **Superseded 2026-08-10 — Check 6 is now Done; see the 2026-08-10 change-log entry in `archreview/remediation/00-tracking.md`.** |
| 2026-07-13 | New finding **TST-30** (`Low`/`P2`) added — surfaced during TST-25 acceptance: CI runs on a single shared `gitea-runner` (`maxParallel=1`, co-located `10.100.0.35`) interleaved with `dohertj2/lmxopcua`, and Gitea 1.26 exposes no run cancel/delete, so queue latency is unbounded under cross-repo contention and the runner is a single point of failure. Design: add a second/labelled runner + document the no-cancel reality and the `run-windev-ci.sh` bypass. Roll-ups updated (Testing Low 2→3, total 47→48; P2 9→10). |
| 2026-08-07 | **TST-29 → `Done`:** migrated the Phase-5 (orphan-worker reattach) deferred-not-planned governance record and the settled Phase-4 Viewer-default decision from `oldtasks.md` into a new "Session-Resilience Epic Scope" entry in `docs/DesignDecisions.md`; repointed CLAUDE.md and `stillpending.md:7,165` from `oldtasks.md` to `docs/DesignDecisions.md` / `docs/plans/2026-06-15-session-resilience.md.tasks.json`; `git rm oldtasks.md`. The five untracked root docs-review artifacts (`MxAccessGateway-docs-{issues,fixed,final}.md`, `MxGatewayClient-docs-{issues,fixed}.md`) were absent from this worktree — delete from the main working tree separately. |
| 2026-08-07 | **GWC-24 → `Done`** (branch `fix/gwc-24-staging-bound`). `WorkerClient._eventStaging` is now `Channel.CreateBounded` at `2 × EventChannelCapacity` (`Wait`, single reader/writer, no sync continuations); a rejected staging `TryWrite` faults the client `ProtocolViolation` with `QueueOverflow("worker-event-staging")` unless `IsTerminalState()` (shutdown stays a silent drop), so a consumer draining slower than its worker produces dies at a fixed ceiling instead of growing gateway memory. Queue-depth accounting moved from `EnqueueWorkerEventAsync` to `StageWorkerEvent`, so the single gauge reports staged + queued; the timed-write fault (`EventChannelFullModeTimeout` / `QueueOverflow("worker-events")`) is unchanged and still catches the full-stall case first. No new config key — total gateway-side buffering is `3 × MxGateway:Events:QueueCapacity`, derived; coordination with still-open old **GWC-21** (`EventChannelFullModeTimeout` configurability) remains open and was not blocked on. Docs same commit: `GatewayProcessDesign.md` (two overflow faults), `MxAccessWorkerInstanceDesign.md`, `GatewayConfiguration.md`, `Metrics.md`. Tests: `WorkerClientTests.StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout` and `.WorkerEventQueueDepthGaugeCountsStagedEvents`; `WorkerClientTests` 22/22 green, `NonWindows.slnx` builds with 0 warnings. |
@@ -175,7 +175,7 @@ Independent of the runner count, document the **no-cancel** reality (Gitea 1.26
## Cross-domain dependencies
- **TST-25 → old TST-05 / old TST-24:** the SSH-driven nightly is where the scheduled live-MXAccess smoke lands (closes old TST-05), and a working CI Windows tier unblocks wiring client wire-behavior tests into CI (old TST-24).
- **TST-25 → old TST-05 / old TST-24:** the SSH-driven nightly is where the scheduled live-MXAccess smoke lands (closes old TST-05), and a working CI Windows tier unblocks wiring client wire-behavior tests into CI (old TST-24). *Followed up 2026-08-10:* old TST-05 is `Partially done` — the nightly closes the scheduling half, but the live suite still covers none of the five worker **control** commands; old TST-24 is `Done`, and its client wire tests turned out to need no Windows tier at all (they run in the `portable` job). See the first-cycle tracker.
- **TST-25 ↔ IPC-24/IPC-25:** the nightly windev job is also the natural home for any Windows-side codegen verification the contracts/IPC remediation adds; coordinate job naming so both plans extend the same `windows-x86`/nightly jobs rather than adding parallel ones.
- **TST-26 ⊂ TST-25:** same commit, by rule.
- **TST-27:** ships in the cycle's P1 doc-drift batch alongside WRK-26 and CLI-42 (roadmap item 8); its `/browse` residual stays with TST-16 (prior cycle).
+5 -2
View File
@@ -217,7 +217,7 @@ Full design + implementation for each row lives in the linked domain doc under i
| TST-02 | High | P0 | M | TST-04 | Done | Reconnect owner re-validation not implemented |
| TST-03 | High | P1 | M | — | Done | No CI exists |
| TST-04 | High | P2 | L | — | Done | Session-resilience epic 16/28 tasks unfinished |
| TST-05 | Medium | P1 | S | TST-03 | Not started | Real-worker control/COM paths verified opt-in only |
| TST-05 | Medium | P1 | S | TST-03 | Partially done | Real-worker control/COM paths verified opt-in only. **Scheduling half closed 2026-08-10** by the TST-25 `nightly-windev` job (`.gitea/workflows/ci.yml`, cron `0 6 * * *` → `scripts/ci/run-windev-ci.sh live`, which sets `MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`, runs `WorkerLiveMxAccessSmokeTests`, and opens a Gitea issue on failure) — "opt-in, run by memory" is now "runs nightly, reports failures". **Residual: the coverage-audit half.** The live suite's 8 facts cover all six late-added COM commands but none of the five control commands (`Ping`, `GetSessionState`, `GetWorkerInfo`, `DrainEvents`, `ShutdownWorker`), which real workers answer in `Worker/Ipc/WorkerPipeSession.cs` yet are still only exercised through `FakeWorkerHarness` canned replies — precisely the masking the finding named |
| TST-06 | Medium | — | M | — | Not started | Dashboard live-data path untested |
| TST-07 | Medium | — | S | — | Not started | Real-clock sleeps with negative assertions are latent flakes |
| TST-08 | Medium | P1 | M | — | Done | Full-suite orphaned testhost processes (does not reproduce; doc de-stale) |
@@ -236,7 +236,7 @@ Full design + implementation for each row lives in the linked domain doc under i
| TST-21 | Low | — | S | — | Not started | Log rotation configured but minimal |
| TST-22 | Low | — | S | — | Not started | Config-shape JSON block omits documented keys |
| TST-23 | Low | P2 | S | — | Done | Bidirectional `Session` RPC never built |
| TST-24 | Low | P2 | M | TST-03 | Not started | Client wire behaviour has no automated verification. **Gate cleared:** TST-03 CI is Done (live and green 2026-07-10; Windows/x86 tier green 2026-07-13 via the TST-25/TST-26 SSH-driven windev job), so TST-24 is unblocked — deferred by choice now, not CI-gated |
| TST-24 | Low | P2 | M | TST-03 | Done | Client wire behaviour has no automated verification — closed 2026-08-10. Go/Rust/Java already had real-server wire tests (the finding's premise was stale); the genuine gaps were .NET (transport-interface fake everywhere, no server package) and Python (stub monkeypatch everywhere but one opt-in TLS test). Added `MxGatewayClientWireTests` + `WireFakeGatewayServer` (Kestrel h2c) and `tests/test_wire_fake_gateway.py` (`grpc.aio` loopback), plus a `dotnet test` step for the .NET client in the `portable` CI job. Caught a real bug: Python `connect()` built the `grpc.aio` channel inside `asyncio.to_thread` and failed for every non-stub connection |
## Cross-cutting clusters
@@ -253,6 +253,9 @@ Findings the review flagged as one coordinated design pass — sequence them tog
| Date | Change |
|---|---|
| 2026-08-10 | **TST-25 acceptance Check 6 (forced-failure nightly issue) → Done.** The 2026-07-13 record wrote this check off as "abandoned to shared-runner congestion"; that was wrong on both counts. The 2026-07-13 probe *did* land (issue #125, `[CHECK6 PROBE]`, run 375), and since 2026-07-17 the `nightly-windev` `if: failure()` step has filed an issue on **every** red nightly — #126–#139, all authored by the `gitea-actions` bot. Traced run 672 (schedule, main, red) line by line: main step fails → `exitcode '1': failure` → the `if: failure()` step runs → `POST /api/v1/repos/dohertj2/mxaccessgw/issues` with the built-in token masked to `***` → issue #139 created at the matching timestamp. Re-confirmed by a fresh forced-failure probe on the throwaway branch `test/tst25-check6-nightly-issue` (temporary `tst25-check6-probe.yml` reproducing the job shape with `exit 1` for the live step; run 677 → issue #140). Branch deleted, issues #125 and #140 closed with explanatory comments. **One real defect found and fixed** (`fix/tst25-nightly-issue-path`, not merged): `${{ github.server_url }}` is the runner-internal `http://gitea:3000`, so every filed issue's run link was unreachable from a browser. The API call must keep using it (the job container resolves `gitea` only on the docker network and has no LAN egress to the public origin), so the fix adds a `PUBLIC_SERVER_URL: https://gitea.dohertylan.com` job env used **only** for the browser-facing link in the issue body; the probe validated the fixed template (#140 carries a `https://gitea.dohertylan.com/...` link that returns 200). **Separately observed, not fixed:** the nightly has been red continuously since at least 2026-07-17 (run 672: `x86 Worker.Tests failed with exit code 1`, 1 failed / 398 passed / 11 skipped — the known `EventBurst_DrainLoopCoalescesFlushes` class of flake), and the step de-duplicates nothing, so 14 issues are open, seven of them (#132–#138) for the identical SHA `47c0b64`. Worth a follow-up: fix the red nightly, and consider having the step reuse an open issue with the same title instead of filing a new one. |
| 2026-08-10 | **TST-24 → `Done`: per-client wire tests land for the two clients that lacked them** (branch `feat/tst-24-client-wire-tests`). Audit first corrected the finding's premise: **Go, Rust, and Java already had real-server wire tests** — `newBufconnClient`/`fakeGatewayServer` over `grpc/test/bufconn`, `spawn_fake_gateway` over a loopback `TcpListener` with tonic's `Server`, and `InProcessGateway`/`TestGatewayService` over `InProcessServerBuilder` — each already asserting the round trip, the server-observed `authorization` bearer header, and the `ReplayGap` sentinel. The real gaps were **.NET** (every test substituted `FakeGatewayTransport` for `IMxGatewayClientTransport`, and the test project had no server package) and **Python** (stub monkeypatching everywhere except one opt-in TLS test serving only `OpenSession`). Added `WireFakeGatewayServer` + `MxGatewayClientWireTests` (Kestrel h2c on `127.0.0.1:0` serving `MxAccessGatewayBase`; new `Grpc.AspNetCore.Server` 2.76.0 + `Microsoft.AspNetCore.App` refs on the test project) and `clients/python/tests/test_wire_fake_gateway.py` (`grpc.aio` server on `127.0.0.1:0`, no new deps). Four shapes each: full round trip with every reply field asserted, the bearer header **as received by the server** on the streaming RPC too, the `ReplayGap` sentinel surfaced as the client's typed signal, and a genuine `PERMISSION_DENIED` mapping to the typed authorization error. CI: the `portable` job only *built* the .NET client, so a `dotnet test` step was added. **The new tests immediately caught a shipped bug** — Python `GatewayClient.connect()`/`GalaxyRepositoryClient.connect()` constructed the `grpc.aio` channel inside `asyncio.to_thread`, which raises `RuntimeError: There is no current event loop in thread 'asyncio_0'` because a `grpc.aio` channel binds to the loop current on the constructing thread; every non-stub connection failed, and the one test guarding the off-loop behaviour (Client.Python-028) monkeypatched `create_channel` and so asserted the bug. Fixed by splitting `resolve_channel_security` (blocking TOFU probe, off-loop) from `create_channel` (on-loop), with the `-028` tests retargeted to assert both halves. Verified: .NET 133 passed/1 skipped (pre-existing live-gateway skip), Python 168 passed/1 skipped plus 6/6 opt-in TLS. Docs: `docs/GatewayTesting.md` § Client Wire Tests, `clients/dotnet/README.md`, `clients/python/README.md`. |
| 2026-08-10 | **TST-05 revisited under the restored Windows tier → `Partially done`** (branch `feat/tst-24-client-wire-tests`, doc/tracker-only). The finding's **scheduling** half is closed: cycle-2 TST-25's `nightly-windev` job (cron `0 6 * * *`) runs `scripts/ci/run-windev-ci.sh live` → `windev-worker-ci.ps1 -Mode live`, which sets `MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`, runs `WorkerLiveMxAccessSmokeTests` on windev after the x86 build/Worker.Tests/full-slnx steps, and files a Gitea issue when red. The **coverage-audit** half is *not* closed, and the audit the design asked for now has a negative answer: the suite's eight `[LiveMxAccessFact]`s cover all six late-added COM commands (`Suspend`, `Activate`, `AuthenticateUser`, `ArchestrAUserToId`, `AddBufferedItem`, `SetBufferedUpdateInterval`) but zero of the five control commands — `MxCommandKind.{Ping,GetSessionState,GetWorkerInfo,DrainEvents,ShutdownWorker}` appear nowhere in `WorkerLiveMxAccessSmokeTests.cs`, so the exact paths the Finding calls masked are still only proven against `FakeWorkerHarness` canned replies while the real implementations live in `Worker/Ipc/WorkerPipeSession.cs`. Residual work (two `[LiveMxAccessFact]`s, windev-only to author and verify) is specified in [60-testing-docs-gaps.md](60-testing-docs-gaps.md#tst-05--real-worker-controlcom-paths-verified-opt-in-only---medium--p1). |
| 2026-07-10 | **TST-15 design fleshed out** (still `Not started` — design only, not implementation): `docs/plans/2026-07-10-dashboard-session-acl-tst15.md`. Resolves the crux the deferral left open — the dashboard is LDAP-identity (Admin/Viewer) while sessions are API-key-owned (`OwnerKeyId`), two disjoint identity domains — via a **session tag** sourced from the owning API key (rides in the existing `ApiKeyConstraints` JSON blob, no SQLite migration). Admin-sees-all; Viewer may `SubscribeSession` iff `session.Tags ∩ viewer.GrantedTags ≠ ∅` (new `Dashboard:GroupToTag` map → hub-token tag claims); untagged sessions Admin-only by default (`Dashboard:UntaggedSessionVisibility`). Includes the enforcement path (`HubTokenPayload.Tags` + `IDashboardSessionAcl` gate at `SubscribeSession`), task breakdown (epic Tasks 16–19), test plan incl. live-LDAP, and rejected alternatives (client-supplied tag; group→key-id map). Tracker + `60-testing-docs-gaps.md` TST-15 section point at the doc. **TST-03 investigated:** the CI never ran because the repo had **zero registered Gitea Actions runners** (Actions is enabled; runs are created on push/PR/nightly but fail instantly with nothing to execute them). A Mac runner proved the pipeline executes but cannot clone — this Gitea hands runners the internal `http://gitea:3000` URL, reachable only by a runner co-located on the gitea Docker network. Fix = run a co-located runner on the Gitea host (recipe prepared, `scratchpad/gitea-runner/setup-gitea-host-runner.sh`); pending host access. TST-03 stays `In review`. |
| 2026-07-09 | **P2 Epic wrap — user decision: DEFER TST-15 + TST-24, close the epic.** Epic bucket result: 5 of 7 findings `Done` (CLI-15, CLI-04, CLI-30, TST-01, TST-04); **TST-15** and **TST-24** intentionally deferred to a follow-up (kept `Not started`, not `Won't fix` — they are gated, not rejected). **TST-15** (dashboard EventsHub per-session ACL) is epic Phase 4 — a real feature needing a new session-"tag" mechanism + dashboard group→tag config, not a mechanical fix; the `EventsHub` `TODO(per-session-acl)` stays, and the already-shipped **SEC-25** mitigation (tag *values* redacted from the dashboard mirror by default) means no sensitive payload leaks through the hub today regardless of the missing ACL — so deferring carries no value-leak risk. **TST-24** (per-client wire tests) depends on **TST-03** (CI), which is `In review` (YAML authored, never run on a Gitea runner) — no point wiring client tests into a pipeline that isn't live yet. Net P2: 35/38 `Done`; remaining = TST-15 (deferred feature), TST-24 (deferred, CI-gated), TST-14 (user deletes their own untracked gitignored `*-docs-*.md` files). |
| 2026-07-09 | P2 Epic — **Java client completes CLI-15 + CLI-04 locally** (commit `1cc0fa4`); **CLI-15, CLI-04, CLI-30, TST-01 all → `Done` (5/5 clients + server e2e)**. Java CLI-15: `MxEventStreamItem` record + `MxEventStream.nextItem()` (`isReplayGap()`/`replayGap()`/`event()`); existing `Iterator<MxEvent>` path unchanged, sentinel never swallowed. Java CLI-04: Phase 1 `adviseSupervisory`/`writeSecured`/`writeSecured2`/`authenticateUser`/`archestrAUserToId` + Phase 2 `addBufferedItem`/`setBufferedUpdateInterval`/`suspend`/`activate` (unregister already present) on `MxGatewaySession`, each through `invokeCommand` → `ensureProtocolSuccess`+`ensureMxAccessSuccess`; credentials scrubbed via `MxGatewaySecrets.redactCredentials` (tests assert absent from message/toString/CLI). `gradle test` 106/0 (58 client + 48 cli), no generated churn. Built locally with `JAVA_HOME=/opt/homebrew/opt/openjdk@17` — Java toolchain now works on the Mac (see prior note). Shared docs `ClientLibrariesDesign.md` + CLAUDE.md updated to "all five clients". **TST-01 → Done** (server e2e `fed0685` + all 5 client `ReplayGap` consumers). This closes session-resilience epic Phase 3 fully. |
@@ -159,6 +159,14 @@ This finding is the umbrella; TST-01/02/15 are its actionable slices. The remedi
## TST-05 — Real-worker control/COM paths verified opt-in only `Medium` · `P1`
> **Status revisit 2026-08-10 (unlocked by TST-25): `Partially done` — one half closed, one half open.**
>
> **Closed — the scheduled cadence.** The `nightly-windev` job in `.gitea/workflows/ci.yml` (cron `0 6 * * *`, gated `if: github.event_name == 'schedule'`) runs `scripts/ci/run-windev-ci.sh live`, which drives `scripts/ci/windev-worker-ci.ps1 -Mode live` on windev: x86 Worker build → full `Worker.Tests` → full-slnx build → `MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1 dotnet test … --filter FullyQualifiedName~WorkerLiveMxAccessSmokeTests`. A red nightly opens a Gitea issue, so nobody has to watch the Actions page. That is exactly this finding's **Design** paragraph, and it is the `live-mxaccess` job the design referred to (renamed; the removed job it originally pointed at is gone — see cycle-2 TST-25/TST-26).
>
> **Open — the coverage audit.** The design also required auditing `WorkerLiveMxAccessSmokeTests.cs` for coverage of *each* of the eleven late-added command kinds and adding missing `[LiveMxAccessFact]` cases. That audit now has an answer, and it is negative for five of the eleven. The suite's eight facts reach all six late-added **COM** commands (`Suspend`, `Activate`, `AuthenticateUser`, `ArchestrAUserToId`, `AddBufferedItem`, `SetBufferedUpdateInterval` — the `NewComCommands_RoundTripWithRealReplies` and `BufferedItem_*` facts). None of them sends any of the five **control** commands: `MxCommandKind.{Ping,GetSessionState,GetWorkerInfo,DrainEvents,ShutdownWorker}` do not appear anywhere in the file. Those are the very kinds the Finding below names as masked. The real worker answers them off-STA in `src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerPipeSession.cs` (dispatch switch at `:574`+), so the nightly exercises that code path only incidentally, never by assertion — a regression in `CreatePingReply`/`CreateSessionStateReply`/`CreateWorkerInfoReply`/the drain snapshot/the shutdown-after-reply ordering still ships green through both CI and the nightly.
>
> **Residual work to close TST-05 fully** (small, Windows-only): add one `[LiveMxAccessFact]` to `WorkerLiveMxAccessSmokeTests` that, against a live worker, invokes `Ping` → `GetSessionState` → `GetWorkerInfo` → `DrainEvents` and asserts each returns a non-`INVALID_REQUEST` reply carrying real worker state (e.g. `worker_process_id` matching the launched process), plus a separate fact for `ShutdownWorker` asserting the OK reply arrives *before* the worker exits and the session is then faulted/closed. `ShutdownWorker` needs `admin` scope and terminates the worker, so it must be the last fact in its own fixture. Not done here because it can only be authored and verified on windev with MXAccess installed; this revisit is doc/tracker-only.
**Finding.** All eleven late-added command kinds are unit-tested against fakes and live-verified once on the dev rig (`stillpending.md` §1.1), but the default suite exercises `Ping`/`GetWorkerInfo`/`DrainEvents`/`ShutdownWorker` only through `FakeWorkerHarness.RespondToControlCommandAsync` (verify current line range in `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/Fakes/FakeWorkerHarness.cs`), which returns canned replies.
**Impact.** A worker-side regression in these paths is invisible until someone sets `MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`.
@@ -449,6 +457,24 @@ If TST-02's interim mitigation (flip retention off) is chosen instead of impleme
## TST-24 — Client wire behaviour has no automated verification `Low` · `—`
> **Resolution 2026-08-10 (branch `feat/tst-24-client-wire-tests`): `Done`.** All five clients now drive their public API against a fake gateway served over a real gRPC transport, in the client's own default suite, and all five run in CI.
>
> **Corrected premise.** The Finding's "no in-process gateway integration tests" was already stale when it was re-verified: **Go, Rust, and Java had real-server wire tests**, not mocks — Go's `newBufconnClient`/`fakeGatewayServer` (`clients/go/mxgateway/client_session_test.go`) over `grpc/test/bufconn`, Rust's `spawn_fake_gateway` (`clients/rust/tests/client_behavior.rs`) over a loopback `TcpListener` with tonic's `Server`, and Java's `InProcessGateway`/`TestGatewayService` (`MxGatewayClientSessionTests.java`) over `InProcessServerBuilder`. Each already asserted the round trip, the `authorization` bearer header as *observed by the server*, and the `ReplayGap` sentinel. The cycle-2 re-verification cited `clients/python/tests/test_replay_gap.py` as evidence for "the other four clients still unit-test against mocks"; that generalized from Python to Go/Rust/Java incorrectly. `InProcessGatewayHarness` (the "template" the Impact paragraph names) is in fact the *thinner* of the Java harnesses — it serves only `streamEvents`/`closeSession` for the CLI tests.
>
> **Real gap, and what was built.** Two clients genuinely had none. **.NET** substituted `FakeGatewayTransport` for `IMxGatewayClientTransport` in every test, so not even the generated stub ran, and its test project had no server package at all. **Python** monkeypatched `MxAccessGatewayStub` everywhere except one opt-in TLS test that served only `OpenSession`. Both now have the pattern:
> - `clients/dotnet/ZB.MOM.WW.MxGateway.Client.Tests/WireFakeGatewayServer.cs` + `MxGatewayClientWireTests.cs` — Kestrel h2c on `127.0.0.1:0` serving `MxAccessGateway.MxAccessGatewayBase`; needed new `Grpc.AspNetCore.Server` + `Microsoft.AspNetCore.App` references on the test project.
> - `clients/python/tests/test_wire_fake_gateway.py` — a `grpc.aio` server on `127.0.0.1:0` serving `MxAccessGatewayServicer`; no new dependencies (`grpcio` is a runtime dep).
>
> Each covers the four shapes the Design asked for: round trip (`OpenSession` → `Invoke`/`Register` → `StreamEvents` → `CloseSession` with every reply field asserted), the bearer header as received by the server on the streaming RPC as well as the unary ones, the `ReplayGap` sentinel surfaced as the client's typed signal (TST-01), and a real `PERMISSION_DENIED` mapping to the typed authorization error.
>
> **CI.** The `portable` job previously only *built* the .NET client; a `dotnet test` step was added, so its wire tests actually run. Go/Rust/Python already ran their suites there and Java in the `java` job.
>
> **Bug this immediately caught** — the justification for the whole finding. `GatewayClient.connect()` / `GalaxyRepositoryClient.connect()` in the Python client were **broken for every real (non-stub) connection**: they built the `grpc.aio` channel inside `asyncio.to_thread`, and a `grpc.aio` channel binds to the event loop current on the constructing thread, so the worker thread raised `RuntimeError: There is no current event loop in thread 'asyncio_0'`. No mock-based test could see it — the one test asserting the off-loop behaviour (`Client.Python-028`) monkeypatched `create_channel` and therefore asserted the bug. Fixed by splitting `resolve_channel_security` (blocking TOFU probe, runs off-loop) from `create_channel` (must run on the loop thread), keeping the Client.Python-028 guarantee; the two `-028` tests were retargeted to assert both halves.
>
> **Deliberately out of scope.** Only the four session RPCs are served — the alarm feed (`StreamAlarms`, `QueryActiveAlarms`, `AcknowledgeAlarm`) and Galaxy browse are not, matching the Design's "full parity is out of scope". Java's `InProcessGatewayHarness` still lacks `openSession`/`invoke`; the client-module tests cover those shapes, so it was left alone.
>
> **Docs.** `docs/GatewayTesting.md` § Client Wire Tests (the cross-client pattern + per-client harness table), `clients/dotnet/README.md`, `clients/python/README.md`.
**Finding.** All five clients have unit tests (13/8/3/13/7 files for dotnet/go/rust/python/java) but no in-process or containerized gateway integration tests; the only cross-language verification is the operator-run `scripts/run-client-e2e-tests.ps1`. `CrossLanguageSmokeMatrixTests` checks shapes only.
**Impact.** Low-to-moderate: a gateway contract change can pass every default suite and break all five clients (partly mitigated by shared-proto codegen). The Java CLI already proves the cheap pattern — `InProcessGatewayHarness` (`stillpending.md` §8).
+14
View File
@@ -23,6 +23,20 @@ dotnet build clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx
dotnet test clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx --no-build
```
Most tests substitute `FakeGatewayTransport` for `IMxGatewayClientTransport`, so
they never touch the wire. `MxGatewayClientWireTests` is the exception: it drives
the ordinary public API against `WireFakeGatewayServer`, a real gRPC server
(Kestrel h2c on an ephemeral loopback port) serving
`MxAccessGateway.MxAccessGatewayBase`. Only the gateway's behaviour is canned —
the HTTP/2 framing, protobuf serialization, `authorization` metadata, and gRPC
status codes are genuine, so it catches decode and metadata breaks a transport
fake cannot see. No MXAccess or worker is involved; it runs in the default suite.
See `docs/GatewayTesting.md` (Client Wire Tests) for the cross-client pattern.
```powershell
dotnet test clients/dotnet/ZB.MOM.WW.MxGateway.Client.Tests/ZB.MOM.WW.MxGateway.Client.Tests.csproj --filter FullyQualifiedName~MxGatewayClientWireTests
```
## Packaging
Create local library and CLI artifacts from the repository root:
@@ -0,0 +1,162 @@
using ZB.MOM.WW.MxGateway.Contracts.Proto;
namespace ZB.MOM.WW.MxGateway.Client.Tests;
/// <summary>
/// Drives the public client API against <see cref="WireFakeGatewayServer"/> — a real
/// gRPC server on loopback — so the transport, protobuf serialization, call metadata,
/// and gRPC status mapping are all exercised. Every other test in this project
/// substitutes <see cref="FakeGatewayTransport"/> and therefore proves nothing about
/// what actually crosses the wire.
/// </summary>
public sealed class MxGatewayClientWireTests
{
private const string ApiKey = "mxgw_wiretest_secret";
/// <summary>
/// Verifies the full session happy path decodes real wire bytes end to end.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task SessionRoundTrip_OverRealTransport_DecodesEveryReplyField()
{
await using WireFakeGatewayServer server = await WireFakeGatewayServer.StartAsync();
await using MxGatewayClient client = server.CreateClient(ApiKey);
MxGatewaySession session = await client.OpenSessionAsync(
new OpenSessionRequest { ClientSessionName = "wire-test" });
Assert.Equal(WireFakeGatewayServer.FakeGatewayService.SessionId, session.SessionId);
Assert.Equal("fake-backend", session.OpenSessionReply.BackendName);
Assert.Equal(1234, session.OpenSessionReply.WorkerProcessId);
Assert.Equal(3u, session.OpenSessionReply.GatewayProtocolVersion);
Assert.Equal(["events", "invoke"], session.OpenSessionReply.Capabilities);
int serverHandle = await session.RegisterAsync("wire-test-client");
Assert.Equal(WireFakeGatewayServer.FakeGatewayService.ServerHandle, serverHandle);
MxCommandRequest? invoke = server.Service.InvokeRequest;
Assert.NotNull(invoke);
Assert.Equal(MxCommandKind.Register, invoke.Command.Kind);
Assert.Equal("wire-test-client", invoke.Command.Register.ClientName);
List<MxEvent> events = await CollectAsync(client.StreamEventsAsync(
new StreamEventsRequest { SessionId = session.SessionId }));
MxEvent single = Assert.Single(events);
Assert.Equal(MxEventFamily.OnDataChange, single.Family);
Assert.Equal(WireFakeGatewayServer.FakeGatewayService.ServerHandle, single.ServerHandle);
Assert.Equal(WireFakeGatewayServer.FakeGatewayService.ItemHandle, single.ItemHandle);
Assert.Equal(17, single.Value.Int32Value);
Assert.Equal(192, single.Quality);
Assert.Equal(9ul, single.WorkerSequence);
Assert.Equal(MxEvent.BodyOneofCase.OnDataChange, single.BodyCase);
CloseSessionReply closeReply = await session.CloseAsync();
Assert.Equal(SessionState.Closed, closeReply.FinalState);
Assert.Equal(
WireFakeGatewayServer.FakeGatewayService.SessionId,
server.Service.CloseSessionRequest?.SessionId);
}
/// <summary>
/// Verifies the API key reaches the server as a bearer header on unary and
/// streaming calls alike. A transport fake can only assert what the client passes;
/// this asserts what the server receives.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task ApiKey_ReachesTheServerAsBearerMetadata_OnEveryRpc()
{
await using WireFakeGatewayServer server = await WireFakeGatewayServer.StartAsync();
await using MxGatewayClient client = server.CreateClient(ApiKey);
MxGatewaySession session = await client.OpenSessionAsync(
new OpenSessionRequest { ClientSessionName = "wire-test" });
await session.RegisterAsync("wire-test-client");
await CollectAsync(client.StreamEventsAsync(
new StreamEventsRequest { SessionId = session.SessionId }));
await session.CloseAsync();
string expected = $"Bearer {ApiKey}";
Assert.Equal(
new Dictionary<string, string>
{
["OpenSession"] = expected,
["Invoke"] = expected,
["StreamEvents"] = expected,
["CloseSession"] = expected,
},
server.Service.AuthorizationByMethod);
}
/// <summary>
/// Verifies the gateway's replay-gap sentinel survives serialization and is
/// surfaced as a typed, non-terminal stream item.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task ReplayGapSentinel_SurvivesTheWire_AsTypedStreamItem()
{
await using WireFakeGatewayServer server = await WireFakeGatewayServer.StartAsync(service =>
service.ReplayGap = new ReplayGap
{
RequestedAfterSequence = 3,
OldestAvailableSequence = 8,
});
await using MxGatewayClient client = server.CreateClient(ApiKey);
MxGatewaySession session = await client.OpenSessionAsync(
new OpenSessionRequest { ClientSessionName = "wire-test" });
List<MxEventStreamItem> items = [];
IAsyncEnumerable<MxEvent> stream = client.StreamEventsAsync(new StreamEventsRequest
{
SessionId = session.SessionId,
AfterWorkerSequence = 3,
});
await foreach (MxEventStreamItem item in stream.AsStreamItemsAsync())
{
items.Add(item);
}
Assert.Equal(2, items.Count);
Assert.True(items[0].IsReplayGap);
Assert.Equal(3ul, items[0].ReplayGap!.RequestedAfterSequence);
Assert.Equal(8ul, items[0].ReplayGap!.OldestAvailableSequence);
Assert.False(items[1].IsReplayGap);
Assert.Equal(MxEventFamily.OnDataChange, items[1].Event.Family);
Assert.Equal(3ul, server.Service.StreamEventsRequest?.AfterWorkerSequence);
}
/// <summary>
/// Verifies a genuine <c>PERMISSION_DENIED</c> status maps to the typed client
/// exception rather than a bare RpcException.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task PermissionDeniedStatus_MapsToAuthorizationException()
{
await using WireFakeGatewayServer server = await WireFakeGatewayServer.StartAsync(
service => service.DenyInvoke = true);
await using MxGatewayClient client = server.CreateClient(ApiKey);
MxGatewaySession session = await client.OpenSessionAsync(
new OpenSessionRequest { ClientSessionName = "wire-test" });
await Assert.ThrowsAsync<MxGatewayAuthorizationException>(
() => session.RegisterAsync("wire-test-client"));
}
private static async Task<List<MxEvent>> CollectAsync(IAsyncEnumerable<MxEvent> stream)
{
List<MxEvent> events = [];
await foreach (MxEvent gatewayEvent in stream)
{
events.Add(gatewayEvent);
}
return events;
}
}
@@ -0,0 +1,264 @@
using System.Collections.Concurrent;
using System.Net;
using Grpc.Core;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Hosting.Server;
using Microsoft.AspNetCore.Hosting.Server.Features;
using Microsoft.AspNetCore.Server.Kestrel.Core;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.MxGateway.Contracts.Proto;
namespace ZB.MOM.WW.MxGateway.Client.Tests;
/// <summary>
/// Hosts the real <c>mxaccess_gateway.v1.MxAccessGateway</c> service on a loopback
/// Kestrel endpoint so client tests exercise genuine HTTP/2 framing, protobuf
/// serialization, call metadata, and gRPC status propagation.
/// </summary>
/// <remarks>
/// <para>
/// This is the counterpart of <see cref="FakeGatewayTransport"/>: that fake replaces
/// <c>IMxGatewayClientTransport</c>, so nothing below the client wrapper runs. This one
/// replaces only the gateway's <em>behaviour</em> — every byte between the client and
/// the service is the real wire format. Contract breaks that a transport fake cannot
/// see (a field the client never decodes, metadata it does not actually send, a status
/// code it maps differently once it arrives as a real <see cref="RpcException"/>) fail
/// here.
/// </para>
/// <para>
/// Plaintext h2c is used deliberately: TLS is covered by
/// <c>MxGatewayClientTlsHandlerTests</c>, and h2c keeps the harness certificate-free so
/// it runs identically on every CI host. See <c>docs/GatewayTesting.md</c>
/// (Client Wire Tests) for the shared pattern and its Python counterpart.
/// </para>
/// </remarks>
internal sealed class WireFakeGatewayServer : IAsyncDisposable
{
private readonly WebApplication _app;
private WireFakeGatewayServer(WebApplication app, FakeGatewayService service, int port)
{
_app = app;
Service = service;
Endpoint = new Uri($"http://127.0.0.1:{port}");
}
/// <summary>
/// Gets the canned service backing the endpoint; tests read its recorded requests.
/// </summary>
public FakeGatewayService Service { get; }
/// <summary>
/// Gets the h2c endpoint to point <see cref="MxGatewayClientOptions.Endpoint"/> at.
/// </summary>
public Uri Endpoint { get; }
/// <summary>
/// Starts a server on an ephemeral loopback port.
/// </summary>
/// <param name="configure">Optional configuration of the canned service.</param>
/// <returns>The started server.</returns>
public static async Task<WireFakeGatewayServer> StartAsync(Action<FakeGatewayService>? configure = null)
{
FakeGatewayService service = new();
configure?.Invoke(service);
WebApplicationBuilder builder = WebApplication.CreateBuilder();
builder.Logging.ClearProviders();
builder.WebHost.ConfigureKestrel(options =>
// Port 0 lets the OS pick; HTTP/2 without TLS (h2c) is what the client's
// plain http:// endpoint negotiates via RequestVersionExact.
options.Listen(IPAddress.Loopback, 0, listen => listen.Protocols = HttpProtocols.Http2));
builder.Services.AddGrpc();
builder.Services.AddSingleton(service);
WebApplication app = builder.Build();
app.MapGrpcService<FakeGatewayService>();
await app.StartAsync().ConfigureAwait(false);
return new WireFakeGatewayServer(app, service, ResolvePort(app));
}
/// <summary>
/// Creates a client bound to this server's endpoint.
/// </summary>
/// <param name="apiKey">API key the client should present.</param>
/// <returns>A client that talks to this server over h2c.</returns>
public MxGatewayClient CreateClient(string apiKey) =>
MxGatewayClient.Create(new MxGatewayClientOptions
{
Endpoint = Endpoint,
ApiKey = apiKey,
UseTls = false,
DefaultCallTimeout = TimeSpan.FromSeconds(30),
});
/// <inheritdoc />
public async ValueTask DisposeAsync()
{
await _app.StopAsync().ConfigureAwait(false);
await _app.DisposeAsync().ConfigureAwait(false);
}
private static int ResolvePort(WebApplication app)
{
IServerAddressesFeature? addresses = app.Services
.GetRequiredService<IServer>()
.Features
.Get<IServerAddressesFeature>();
string address = addresses?.Addresses.FirstOrDefault()
?? throw new InvalidOperationException("Kestrel did not report a bound address.");
return new Uri(address).Port;
}
/// <summary>
/// Canned gateway answering the four session RPCs with gateway-shaped replies.
/// </summary>
internal sealed class FakeGatewayService : MxAccessGateway.MxAccessGatewayBase
{
/// <summary>The session id every reply carries.</summary>
public const string SessionId = "wire-session-1";
/// <summary>The server handle the canned Register reply returns.</summary>
public const int ServerHandle = 4242;
/// <summary>The item handle the canned data-change event carries.</summary>
public const int ItemHandle = 77;
/// <summary>
/// Gets the <c>authorization</c> header value observed per RPC name.
/// </summary>
public ConcurrentDictionary<string, string> AuthorizationByMethod { get; } = new();
/// <summary>
/// Gets or sets a value indicating whether <c>Invoke</c> fails with
/// <see cref="StatusCode.PermissionDenied"/> instead of replying.
/// </summary>
public bool DenyInvoke { get; set; }
/// <summary>
/// Gets or sets the replay-gap sentinel emitted at the head of the event stream.
/// </summary>
public ReplayGap? ReplayGap { get; set; }
/// <summary>
/// Gets the last <c>Invoke</c> request the client sent, as decoded from the wire.
/// </summary>
public MxCommandRequest? InvokeRequest { get; private set; }
/// <summary>
/// Gets the last <c>StreamEvents</c> request the client sent.
/// </summary>
public StreamEventsRequest? StreamEventsRequest { get; private set; }
/// <summary>
/// Gets the last <c>CloseSession</c> request the client sent.
/// </summary>
public CloseSessionRequest? CloseSessionRequest { get; private set; }
/// <inheritdoc />
public override Task<OpenSessionReply> OpenSession(
OpenSessionRequest request,
ServerCallContext context)
{
Record(context);
return Task.FromResult(new OpenSessionReply
{
SessionId = SessionId,
BackendName = "fake-backend",
WorkerProcessId = 1234,
WorkerProtocolVersion = 1,
GatewayProtocolVersion = 3,
Capabilities = { "events", "invoke" },
ProtocolStatus = Ok(),
});
}
/// <inheritdoc />
public override Task<MxCommandReply> Invoke(MxCommandRequest request, ServerCallContext context)
{
Record(context);
InvokeRequest = request;
if (DenyInvoke)
{
throw new RpcException(new Status(StatusCode.PermissionDenied, "invoke scope required"));
}
return Task.FromResult(new MxCommandReply
{
SessionId = request.SessionId,
CorrelationId = request.ClientCorrelationId,
Kind = request.Command.Kind,
ProtocolStatus = Ok(),
Hresult = 0,
Register = new RegisterReply { ServerHandle = ServerHandle },
});
}
/// <inheritdoc />
public override async Task StreamEvents(
StreamEventsRequest request,
IServerStreamWriter<MxEvent> responseStream,
ServerCallContext context)
{
Record(context);
StreamEventsRequest = request;
if (ReplayGap is not null)
{
// The sentinel shape the gateway emits: family unspecified, body unset,
// only replay_gap populated.
await responseStream.WriteAsync(new MxEvent
{
SessionId = request.SessionId,
ReplayGap = ReplayGap,
}).ConfigureAwait(false);
}
await responseStream.WriteAsync(new MxEvent
{
SessionId = request.SessionId,
Family = MxEventFamily.OnDataChange,
ServerHandle = ServerHandle,
ItemHandle = ItemHandle,
Value = new MxValue { Int32Value = 17 },
Quality = 192,
WorkerSequence = 9,
OnDataChange = new OnDataChangeEvent(),
}).ConfigureAwait(false);
}
/// <inheritdoc />
public override Task<CloseSessionReply> CloseSession(
CloseSessionRequest request,
ServerCallContext context)
{
Record(context);
CloseSessionRequest = request;
return Task.FromResult(new CloseSessionReply
{
SessionId = request.SessionId,
FinalState = SessionState.Closed,
ProtocolStatus = Ok(),
});
}
private static ProtocolStatus Ok() => new() { Code = ProtocolStatusCode.Ok };
private void Record(ServerCallContext context)
{
string? authorization = context.RequestHeaders.GetValue("authorization");
if (authorization is not null)
{
// context.Method is the fully-qualified "/package.Service/Method";
// key on the bare method name so assertions stay readable.
AuthorizationByMethod[context.Method[(context.Method.LastIndexOf('/') + 1)..]] =
authorization;
}
}
}
}
@@ -12,6 +12,15 @@
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.14.1" />
<PackageReference Include="xunit" Version="2.9.3" />
<PackageReference Include="xunit.runner.visualstudio" Version="3.1.4" />
<!-- Wire tests only (WireFakeGatewayServer): hosts the real MxAccessGateway service
on loopback Kestrel so the client is driven over genuine HTTP/2 + protobuf rather
than a substituted transport. Version tracks the gateway server's Grpc.AspNetCore
(src/ZB.MOM.WW.MxGateway.Server) and the client's Grpc.Net.Client, both 2.76.0. -->
<PackageReference Include="Grpc.AspNetCore.Server" Version="2.76.0" />
</ItemGroup>
<ItemGroup>
<FrameworkReference Include="Microsoft.AspNetCore.App" />
</ItemGroup>
<ItemGroup>
+22
View File
@@ -47,6 +47,19 @@ The tests import the generated gateway and worker stubs, run fake async gateway
stubs, verify API key metadata, exercise stream cancellation, load shared value
and command fixtures, and check deterministic CLI output.
`tests/test_wire_fake_gateway.py` is the one suite that does **not** substitute a
stub: it serves a canned `MxAccessGatewayServicer` from a real `grpc.aio` server
on an ephemeral loopback port and drives the ordinary `GatewayClient` API against
it. Only the gateway's behaviour is canned — the HTTP/2 framing, protobuf
serialization, `authorization` metadata, and gRPC status codes are genuine, so it
catches decode and metadata breaks a stub fake cannot see. No MXAccess, no worker,
no TLS, so it runs in the default suite. See `docs/GatewayTesting.md`
(Client Wire Tests) for the cross-client pattern.
```powershell
python -m pytest tests/test_wire_fake_gateway.py
```
## Packaging
Install the package in editable mode for local development:
@@ -398,6 +411,15 @@ point: the `require_certificate_validation=True` keyword on
`--require-certificate-validation` CLI flag. See
[Gateway Configuration](../../docs/GatewayConfiguration.md#automatic-self-signed-certificate).
Channel construction is split in two: `resolve_channel_security(options)` performs
the blocking part (the trust-on-first-use certificate probe) and
`create_channel(options, security=...)` builds the channel. The async `connect`
classmethods run the first off the event loop and the second on it, because a
`grpc.aio` channel binds to the event loop current on the constructing thread —
building it inside `asyncio.to_thread` raises
`RuntimeError: There is no current event loop in thread 'asyncio_N'`. Callers that
build their own channel should keep `create_channel` on the loop thread.
## CLI
The CLI emits deterministic JSON for automation:
@@ -12,7 +12,7 @@ from .auth import merge_metadata
from .errors import ensure_protocol_success, map_rpc_error
from .generated import mxaccess_gateway_pb2 as pb
from .generated import mxaccess_gateway_pb2_grpc as pb_grpc
from .options import ClientOptions, create_channel
from .options import ClientOptions, create_channel, resolve_channel_security
class GatewayClient:
@@ -58,9 +58,13 @@ class GatewayClient:
if stub is not None:
return cls(options=resolved, stub=stub)
# create_channel may perform a blocking TLS certificate probe (TOFU
# default); run it off the event loop so connect never freezes the loop.
channel = await asyncio.to_thread(create_channel, resolved)
# Resolving security may perform a blocking TLS certificate probe (TOFU
# default); run that off the event loop so connect never freezes it. The
# channel itself must be built on the loop thread — a grpc.aio channel
# binds to the loop current on the constructing thread, and a worker
# thread has none.
security = await asyncio.to_thread(resolve_channel_security, resolved)
channel = create_channel(resolved, security=security)
return cls(
options=resolved,
stub=pb_grpc.MxAccessGatewayStub(channel),
@@ -21,7 +21,12 @@ from .auth import merge_metadata
from .errors import MxGatewayError, map_rpc_error
from .generated import galaxy_repository_pb2 as galaxy_pb
from .generated import galaxy_repository_pb2_grpc as galaxy_pb_grpc
from .options import BrowseChildrenOptions, ClientOptions, create_channel
from .options import (
BrowseChildrenOptions,
ClientOptions,
create_channel,
resolve_channel_security,
)
_DISCOVER_HIERARCHY_PAGE_SIZE = 5000
_BROWSE_CHILDREN_PAGE_SIZE = 500
@@ -70,9 +75,13 @@ class GalaxyRepositoryClient:
if stub is not None:
return cls(options=resolved, stub=stub)
# create_channel may perform a blocking TLS certificate probe (TOFU
# default); run it off the event loop so connect never freezes the loop.
channel = await asyncio.to_thread(create_channel, resolved)
# Resolving security may perform a blocking TLS certificate probe (TOFU
# default); run that off the event loop so connect never freezes it. The
# channel itself must be built on the loop thread — a grpc.aio channel
# binds to the loop current on the constructing thread, and a worker
# thread has none.
security = await asyncio.to_thread(resolve_channel_security, resolved)
channel = create_channel(resolved, security=security)
return cls(
options=resolved,
stub=galaxy_pb_grpc.GalaxyRepositoryStub(channel),
@@ -105,7 +105,72 @@ def _split_authority(endpoint: str) -> tuple[str, int]:
return (host or "localhost", int(port))
def create_channel(options: ClientOptions) -> grpc.aio.Channel:
@dataclass(frozen=True)
class ChannelSecurity:
"""Transport security resolved for one channel.
`credentials` is `None` for a plaintext channel. `target_name_override` is
the SNI/authority override the TOFU path needs, kept separate from the
caller's explicit `server_name_override` so the caller always wins.
"""
credentials: grpc.ChannelCredentials | None = None
target_name_override: str | None = None
def resolve_channel_security(options: ClientOptions) -> ChannelSecurity:
"""Resolve transport security for `options`, running any blocking probe.
This is the only blocking part of channel construction: the TOFU path opens
a real TCP+TLS socket to fetch the server's certificate. It is split out of
`create_channel` because a `grpc.aio` channel binds to the event loop
*current on the constructing thread*, so the channel itself must be built on
the loop thread — building it inside `asyncio.to_thread` raises
``RuntimeError: There is no current event loop in thread 'asyncio_N'``. The
async `connect` classmethods therefore run this function off the loop and
then call `create_channel` on it.
"""
if options.plaintext:
return ChannelSecurity()
if options.ca_file:
root_certificates = Path(options.ca_file).read_bytes()
return ChannelSecurity(
credentials=grpc.ssl_channel_credentials(root_certificates=root_certificates)
)
if options.require_certificate_validation:
return ChannelSecurity(credentials=grpc.ssl_channel_credentials())
# Lenient default: grpc-python has no per-channel skip-verify, so fetch the
# server's certificate (unverified) and pin it for this channel (TOFU).
# The probe opens a real blocking TCP+TLS socket, so it MUST be bounded —
# a black-holed / firewall-drop host would otherwise hang on the OS default
# connect timeout (minutes). Bound it by call_timeout (or a short fixed
# fallback) so the dial fails fast as a transport error.
host, port = _split_authority(options.endpoint)
probe_timeout = options.call_timeout if options.call_timeout else _TOFU_PROBE_TIMEOUT_SECONDS
try:
presented = ssl.get_server_certificate((host, port), timeout=probe_timeout)
except OSError as error:
raise MxGatewayTransportError(
f"failed to fetch TLS certificate from {options.endpoint}: {error}"
) from error
# The gateway self-signed cert always carries a "localhost" SAN, so default
# the SNI/target-name override to it when none was supplied, tolerating
# dial-by-IP or hostname mismatch.
return ChannelSecurity(
credentials=grpc.ssl_channel_credentials(root_certificates=presented.encode("ascii")),
target_name_override="localhost",
)
def create_channel(
options: ClientOptions,
*,
security: ChannelSecurity | None = None,
) -> grpc.aio.Channel:
"""Create a plaintext or TLS `grpc.aio` channel from client options.
The TLS default is lenient: grpc-python has no per-channel skip-verify, so
@@ -113,48 +178,29 @@ def create_channel(options: ClientOptions) -> grpc.aio.Channel:
as the channel's only trust root (trust-on-first-use). Set
`require_certificate_validation=True` to force system-trust verification, or
pass `ca_file` to verify against a specific CA — both bypass the TOFU path.
Pass *security* to reuse a `ChannelSecurity` already resolved off the event
loop by `resolve_channel_security`; omit it and this call resolves (and may
block) inline. Must run on the thread owning the event loop the channel will
be used from.
"""
security = security if security is not None else resolve_channel_security(options)
channel_options: list[tuple[str, str | int]] = [
("grpc.max_receive_message_length", options.max_grpc_message_bytes),
("grpc.max_send_message_length", options.max_grpc_message_bytes),
]
if options.server_name_override:
channel_options.append(("grpc.ssl_target_name_override", options.server_name_override))
elif security.target_name_override:
channel_options.append(("grpc.ssl_target_name_override", security.target_name_override))
if options.plaintext:
if security.credentials is None:
return grpc.aio.insecure_channel(options.endpoint, options=channel_options)
if options.ca_file:
root_certificates = Path(options.ca_file).read_bytes()
credentials = grpc.ssl_channel_credentials(root_certificates=root_certificates)
elif options.require_certificate_validation:
credentials = grpc.ssl_channel_credentials()
else:
# Lenient default: grpc-python has no per-channel skip-verify, so fetch the
# server's certificate (unverified) and pin it for this channel (TOFU).
# The probe opens a real blocking TCP+TLS socket, so it MUST be bounded —
# a black-holed / firewall-drop host would otherwise hang on the OS default
# connect timeout (minutes). Bound it by call_timeout (or a short fixed
# fallback) so the dial fails fast as a transport error. The async
# `connect` classmethods run this off the event loop (asyncio.to_thread).
host, port = _split_authority(options.endpoint)
probe_timeout = options.call_timeout if options.call_timeout else _TOFU_PROBE_TIMEOUT_SECONDS
try:
presented = ssl.get_server_certificate((host, port), timeout=probe_timeout)
except OSError as error:
raise MxGatewayTransportError(
f"failed to fetch TLS certificate from {options.endpoint}: {error}"
) from error
credentials = grpc.ssl_channel_credentials(root_certificates=presented.encode("ascii"))
# The gateway self-signed cert always carries a "localhost" SAN, so default
# the SNI/target-name override to it when none was supplied, tolerating
# dial-by-IP or hostname mismatch.
if not options.server_name_override:
channel_options.append(("grpc.ssl_target_name_override", "localhost"))
return grpc.aio.secure_channel(
options.endpoint,
credentials,
security.credentials,
options=channel_options,
)
+52 -34
View File
@@ -12,6 +12,7 @@ from zb_mom_ww_mxgateway import client as client_module
from zb_mom_ww_mxgateway import galaxy as galaxy_module
from zb_mom_ww_mxgateway.galaxy import GalaxyRepositoryClient
from zb_mom_ww_mxgateway.generated import mxaccess_gateway_pb2 as pb
from zb_mom_ww_mxgateway.options import ChannelSecurity
@pytest.mark.asyncio
@@ -21,11 +22,12 @@ async def test_gateway_connect_forwards_require_certificate_validation(
"""The connect convenience kwarg must reach ClientOptions (Client.Python-027)."""
captured: dict[str, Any] = {}
def fake_create_channel(options: ClientOptions) -> object:
def fake_resolve(options: ClientOptions) -> ChannelSecurity:
captured["options"] = options
return object()
return ChannelSecurity()
monkeypatch.setattr(client_module, "create_channel", fake_create_channel)
monkeypatch.setattr(client_module, "resolve_channel_security", fake_resolve)
monkeypatch.setattr(client_module, "create_channel", _stub_create_channel)
monkeypatch.setattr(client_module.pb_grpc, "MxAccessGatewayStub", lambda channel: object())
await GatewayClient.connect(
@@ -43,11 +45,12 @@ async def test_galaxy_connect_forwards_require_certificate_validation(
"""GalaxyRepositoryClient.connect must thread the flag too (Client.Python-027)."""
captured: dict[str, Any] = {}
def fake_create_channel(options: ClientOptions) -> object:
def fake_resolve(options: ClientOptions) -> ChannelSecurity:
captured["options"] = options
return object()
return ChannelSecurity()
monkeypatch.setattr(galaxy_module, "create_channel", fake_create_channel)
monkeypatch.setattr(galaxy_module, "resolve_channel_security", fake_resolve)
monkeypatch.setattr(galaxy_module, "create_channel", _stub_create_channel)
monkeypatch.setattr(
galaxy_module.galaxy_pb_grpc, "GalaxyRepositoryStub", lambda channel: object()
)
@@ -61,52 +64,67 @@ async def test_galaxy_connect_forwards_require_certificate_validation(
@pytest.mark.asyncio
async def test_gateway_connect_runs_create_channel_off_the_event_loop(
async def test_gateway_connect_splits_probe_off_loop_and_channel_on_loop(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""connect must run the blocking channel factory off the loop (Client.Python-028)."""
ran_in_thread: dict[str, bool] = {}
"""The blocking probe runs off the loop; the channel is built on it.
def fake_create_channel(options: ClientOptions) -> object:
# If this runs on the event loop thread, get_running_loop() succeeds.
try:
asyncio.get_running_loop()
ran_in_thread["off_loop"] = False
except RuntimeError:
ran_in_thread["off_loop"] = True
return object()
monkeypatch.setattr(client_module, "create_channel", fake_create_channel)
Client.Python-028 required the blocking TOFU probe off the event loop. The
channel itself must nonetheless be constructed *on* the loop thread: a
``grpc.aio`` channel binds to the loop current on the constructing thread,
and a ``to_thread`` worker has none, so building it off-loop raises
``RuntimeError: There is no current event loop``. Assert both halves.
"""
where = _record_connect_threads(monkeypatch, client_module)
monkeypatch.setattr(client_module.pb_grpc, "MxAccessGatewayStub", lambda channel: object())
await GatewayClient.connect(endpoint="gateway.example:5001")
assert ran_in_thread["off_loop"] is True
assert where == {"resolve_off_loop": True, "create_on_loop": True}
@pytest.mark.asyncio
async def test_galaxy_connect_runs_create_channel_off_the_event_loop(
async def test_galaxy_connect_splits_probe_off_loop_and_channel_on_loop(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""GalaxyRepositoryClient.connect must also run the probe off the loop (Client.Python-028)."""
ran_in_thread: dict[str, bool] = {}
def fake_create_channel(options: ClientOptions) -> object:
try:
asyncio.get_running_loop()
ran_in_thread["off_loop"] = False
except RuntimeError:
ran_in_thread["off_loop"] = True
return object()
monkeypatch.setattr(galaxy_module, "create_channel", fake_create_channel)
"""GalaxyRepositoryClient.connect splits the probe and the channel the same way."""
where = _record_connect_threads(monkeypatch, galaxy_module)
monkeypatch.setattr(
galaxy_module.galaxy_pb_grpc, "GalaxyRepositoryStub", lambda channel: object()
)
await GalaxyRepositoryClient.connect(endpoint="gateway.example:5001")
assert ran_in_thread["off_loop"] is True
assert where == {"resolve_off_loop": True, "create_on_loop": True}
def _stub_create_channel(options: ClientOptions, *, security: ChannelSecurity) -> object:
return object()
def _on_event_loop_thread() -> bool:
try:
asyncio.get_running_loop()
except RuntimeError:
return False
return True
def _record_connect_threads(monkeypatch: pytest.MonkeyPatch, module: Any) -> dict[str, bool]:
"""Patch *module*'s channel helpers to record which thread each ran on."""
where: dict[str, bool] = {}
def fake_resolve(options: ClientOptions) -> ChannelSecurity:
where["resolve_off_loop"] = not _on_event_loop_thread()
return ChannelSecurity()
def fake_create_channel(options: ClientOptions, *, security: ChannelSecurity) -> object:
where["create_on_loop"] = _on_event_loop_thread()
return object()
monkeypatch.setattr(module, "resolve_channel_security", fake_resolve)
monkeypatch.setattr(module, "create_channel", fake_create_channel)
return where
@pytest.mark.asyncio
@@ -0,0 +1,288 @@
"""Wire-level tests: the Python client against a real localhost gRPC server.
Every other test in this suite substitutes a fake *stub* object for
``pb_grpc.MxAccessGatewayStub``, so nothing between the client wrapper and the
generated stub is exercised: no HTTP/2 framing, no protobuf serialization, no
call metadata, no gRPC status translation. That leaves a class of contract break
— a field the gateway populates but the client never decodes, metadata the
client believes it sends but does not, a status code it maps differently once it
arrives as a real ``grpc.RpcError`` — invisible to the default suite.
These tests close that gap by serving the real ``mxaccess_gateway.v1.MxAccessGateway``
service from an in-process ``grpc.aio`` server bound to ``127.0.0.1:0`` and
driving the ordinary public client API against it. The bytes on the wire are the
real ones; only the gateway's *behavior* is canned. No MXAccess, no worker, no
network beyond loopback, so this runs everywhere the normal suite runs.
See ``docs/GatewayTesting.md`` (Client Wire Tests) for the shared pattern and its
counterpart in the .NET client.
"""
from __future__ import annotations
import socket
from collections.abc import AsyncIterator, Awaitable, Callable
import grpc
import pytest
import pytest_asyncio
from zb_mom_ww_mxgateway import ClientOptions, GatewayClient
from zb_mom_ww_mxgateway.errors import MxGatewayAuthorizationError
from zb_mom_ww_mxgateway.events import ReplayGap
from zb_mom_ww_mxgateway.generated import mxaccess_gateway_pb2 as pb
from zb_mom_ww_mxgateway.generated import mxaccess_gateway_pb2_grpc as pb_grpc
API_KEY = "mxgw_wiretest_secret"
SESSION_ID = "wire-session-1"
SERVER_HANDLE = 4242
ITEM_HANDLE = 77
def _free_port() -> int:
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.bind(("127.0.0.1", 0))
return int(sock.getsockname()[1])
def _ok() -> pb.ProtocolStatus:
return pb.ProtocolStatus(code=pb.PROTOCOL_STATUS_CODE_OK)
class FakeGateway(pb_grpc.MxAccessGatewayServicer):
"""Canned gateway serving the four session RPCs over a real transport.
Replies are shaped like the gateway's own: an OK ``ProtocolStatus``, the
echoed session id, and the typed payload the client wrapper reads (for
example ``RegisterReply.server_handle``). Set ``deny`` to make ``Invoke``
abort with ``PERMISSION_DENIED`` so the client's gRPC-status mapping is
exercised against a genuine ``grpc.RpcError`` rather than a hand-built one.
"""
def __init__(self, *, deny: bool = False, replay_gap: pb.ReplayGap | None = None) -> None:
self.deny = deny
self.replay_gap = replay_gap
self.endpoint = ""
self.metadata_by_method: dict[str, str] = {}
self.open_request: pb.OpenSessionRequest | None = None
self.invoke_request: pb.MxCommandRequest | None = None
self.stream_request: pb.StreamEventsRequest | None = None
self.close_request: pb.CloseSessionRequest | None = None
def _record(self, method: str, context: grpc.aio.ServicerContext) -> None:
for key, value in context.invocation_metadata() or ():
if key == "authorization":
self.metadata_by_method[method] = value
async def OpenSession( # noqa: N802 - generated gRPC method name
self, request: pb.OpenSessionRequest, context: grpc.aio.ServicerContext
) -> pb.OpenSessionReply:
"""Answer ``OpenSession`` with a fully populated reply."""
self._record("OpenSession", context)
self.open_request = request
return pb.OpenSessionReply(
session_id=SESSION_ID,
backend_name="fake-backend",
worker_process_id=1234,
worker_protocol_version=1,
capabilities=["events", "invoke"],
gateway_protocol_version=3,
protocol_status=_ok(),
)
async def Invoke( # noqa: N802 - generated gRPC method name
self, request: pb.MxCommandRequest, context: grpc.aio.ServicerContext
) -> pb.MxCommandReply:
"""Answer ``Invoke`` with a Register reply, or deny when configured."""
self._record("Invoke", context)
self.invoke_request = request
if self.deny:
await context.abort(grpc.StatusCode.PERMISSION_DENIED, "invoke scope required")
return pb.MxCommandReply(
session_id=request.session_id,
correlation_id=request.client_correlation_id,
kind=request.command.kind,
protocol_status=_ok(),
hresult=0,
register=pb.RegisterReply(server_handle=SERVER_HANDLE),
)
async def StreamEvents( # noqa: N802 - generated gRPC method name
self, request: pb.StreamEventsRequest, context: grpc.aio.ServicerContext
) -> AsyncIterator[pb.MxEvent]:
"""Stream an optional replay-gap sentinel followed by one data change."""
self._record("StreamEvents", context)
self.stream_request = request
if self.replay_gap is not None:
# The sentinel shape the gateway emits: family unspecified, body
# unset, only replay_gap populated.
yield pb.MxEvent(session_id=request.session_id, replay_gap=self.replay_gap)
yield pb.MxEvent(
session_id=request.session_id,
family=pb.MX_EVENT_FAMILY_ON_DATA_CHANGE,
server_handle=SERVER_HANDLE,
item_handle=ITEM_HANDLE,
value=pb.MxValue(int32_value=17),
quality=192,
worker_sequence=9,
on_data_change=pb.OnDataChangeEvent(),
)
async def CloseSession( # noqa: N802 - generated gRPC method name
self, request: pb.CloseSessionRequest, context: grpc.aio.ServicerContext
) -> pb.CloseSessionReply:
"""Answer ``CloseSession`` with a closed final state."""
self._record("CloseSession", context)
self.close_request = request
return pb.CloseSessionReply(
session_id=request.session_id,
final_state=pb.SESSION_STATE_CLOSED,
protocol_status=_ok(),
)
ServeGateway = Callable[..., Awaitable[FakeGateway]]
@pytest_asyncio.fixture
async def serve_gateway() -> AsyncIterator[ServeGateway]:
"""Yield a factory that serves a :class:`FakeGateway` on loopback.
Each call starts its own server on a free port and records it for teardown,
so a test can serve a differently-configured gateway without a fixture per
variant.
"""
servers: list[grpc.aio.Server] = []
async def _start(**kwargs: object) -> FakeGateway:
fake = FakeGateway(**kwargs) # type: ignore[arg-type]
server = grpc.aio.server()
pb_grpc.add_MxAccessGatewayServicer_to_server(fake, server)
port = _free_port()
server.add_insecure_port(f"127.0.0.1:{port}")
await server.start()
servers.append(server)
fake.endpoint = f"127.0.0.1:{port}"
return fake
try:
yield _start
finally:
for server in servers:
await server.stop(grace=None)
async def _connect(fake: FakeGateway) -> GatewayClient:
return await GatewayClient.connect(
ClientOptions(
endpoint=fake.endpoint,
api_key=API_KEY,
plaintext=True,
call_timeout=10.0,
)
)
@pytest.mark.asyncio
async def test_session_round_trip_decodes_real_wire_bytes(serve_gateway: ServeGateway) -> None:
"""Open, invoke, stream, and close against a real server over loopback."""
wire_gateway = await serve_gateway()
client = await _connect(wire_gateway)
try:
session = await client.open_session(client_session_name="wire-test")
assert session.session_id == SESSION_ID
assert session.open_reply.backend_name == "fake-backend"
assert list(session.open_reply.capabilities) == ["events", "invoke"]
server_handle = await session.register("wire-test-client")
assert server_handle == SERVER_HANDLE
assert wire_gateway.invoke_request is not None
assert wire_gateway.invoke_request.command.kind == pb.MX_COMMAND_KIND_REGISTER
assert wire_gateway.invoke_request.command.register.client_name == "wire-test-client"
events = [event async for event in session.stream_events()]
assert len(events) == 1
event = events[0]
assert not isinstance(event, ReplayGap)
assert event.family == pb.MX_EVENT_FAMILY_ON_DATA_CHANGE
assert event.server_handle == SERVER_HANDLE
assert event.item_handle == ITEM_HANDLE
assert event.value.int32_value == 17
assert event.quality == 192
assert event.worker_sequence == 9
assert event.HasField("on_data_change")
close_reply = await session.close()
assert close_reply.final_state == pb.SESSION_STATE_CLOSED
assert wire_gateway.close_request is not None
assert wire_gateway.close_request.session_id == SESSION_ID
finally:
await client.close()
@pytest.mark.asyncio
async def test_api_key_reaches_the_server_on_every_rpc(serve_gateway: ServeGateway) -> None:
"""The bearer header is on the wire for unary and streaming calls alike.
Stub-substituting tests can only assert what the client *passes*; this
asserts what the server *receives*, which is the property that matters.
"""
wire_gateway = await serve_gateway()
client = await _connect(wire_gateway)
try:
session = await client.open_session(client_session_name="wire-test")
await session.register("wire-test-client")
async for _ in session.stream_events():
break
await session.close()
finally:
await client.close()
expected = f"Bearer {API_KEY}"
assert wire_gateway.metadata_by_method == {
"OpenSession": expected,
"Invoke": expected,
"StreamEvents": expected,
"CloseSession": expected,
}
@pytest.mark.asyncio
async def test_replay_gap_sentinel_survives_the_wire(serve_gateway: ServeGateway) -> None:
"""A resumed stream surfaces the gateway's sentinel as a typed ``ReplayGap``."""
replay_gap_gateway = await serve_gateway(
replay_gap=pb.ReplayGap(requested_after_sequence=3, oldest_available_sequence=8)
)
client = await _connect(replay_gap_gateway)
try:
session = await client.open_session(client_session_name="wire-test")
items = [item async for item in session.stream_events(after_worker_sequence=3)]
finally:
await client.close()
assert len(items) == 2
gap = items[0]
assert isinstance(gap, ReplayGap)
assert gap.requested_after_sequence == 3
assert gap.oldest_available_sequence == 8
assert gap.resume_after_worker_sequence == 7
assert not isinstance(items[1], ReplayGap)
assert items[1].family == pb.MX_EVENT_FAMILY_ON_DATA_CHANGE
assert replay_gap_gateway.stream_request is not None
assert replay_gap_gateway.stream_request.after_worker_sequence == 3
@pytest.mark.asyncio
async def test_permission_denied_maps_to_authorization_error(
serve_gateway: ServeGateway,
) -> None:
"""A real ``PERMISSION_DENIED`` status becomes the typed client error."""
denying_gateway = await serve_gateway(deny=True)
client = await _connect(denying_gateway)
try:
session = await client.open_session(client_session_name="wire-test")
with pytest.raises(MxGatewayAuthorizationError):
await session.register("wire-test-client")
finally:
await client.close()
+136 -1
View File
@@ -82,7 +82,17 @@ fake-worker tests cannot validate:
when the rig does not drive sample-bearing buffered batches on demand.
All eight tests are gated by the same `MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`
opt-in variable.
opt-in variable. Opt-in does not mean unscheduled: the `nightly-windev` job runs
this suite on windev every night at 06:00 UTC and files a Gitea issue when it goes
red (see [Continuous Integration](#continuous-integration)), so the smoke no longer
depends on someone remembering to set the variable.
Known coverage gap: the suite reaches all six late-added MXAccess **COM** commands
but none of the five **control** commands (`Ping`, `GetSessionState`,
`GetWorkerInfo`, `DrainEvents`, `ShutdownWorker`). Those are implemented off-STA in
`Worker/Ipc/WorkerPipeSession.cs` and are asserted only against
`FakeWorkerHarness`'s canned replies, so no test proves the *real* worker answers
them. Closing that is the residual half of archreview TST-05.
Build the worker before running the smoke:
@@ -267,6 +277,49 @@ $env:MxGateway__Ldap__ServiceAccountPassword = "<service-account-password>"
dotnet test src/ZB.MOM.WW.MxGateway.IntegrationTests/ZB.MOM.WW.MxGateway.IntegrationTests.csproj --filter FullyQualifiedName~DashboardLdapLiveTests
```
## Client Wire Tests
Each client's own suite drives the client's public API against a **fake gateway
served over a real gRPC transport** — an in-process or loopback server
implementing `mxaccess_gateway.v1.MxAccessGateway`. Only the gateway's *behaviour*
is canned; the HTTP/2 framing, protobuf serialization, call metadata, and gRPC
status codes are genuine. That is the difference from the per-client mocks: a mock
substituted for the generated stub (or, in .NET, for `IMxGatewayClientTransport`)
proves what the client *intends* to send, never what a server *receives*, so a
field the client fails to decode or a header it never actually attaches passes
every mock-based test. These tests need no MXAccess, no worker, and no network
beyond loopback, so they run in the default suite on every host.
The shared shape each client's wire test covers:
- **Round trip** — `OpenSession` → `Invoke` (a `Register`, asserting the decoded
`RegisterReply.server_handle`) → `StreamEvents` (asserting the decoded
`OnDataChange` fields) → `CloseSession`.
- **Auth on the wire** — the `authorization: Bearer <key>` header is asserted as
*observed by the server*, on the streaming RPC as well as the unary ones.
- **Replay-gap sentinel** — a stream resumed with `after_worker_sequence` opens
with the gateway's `replay_gap` sentinel, and the client surfaces it as its
typed, non-terminal replay-gap signal rather than a normal event.
- **Status mapping** — a real `PERMISSION_DENIED` from the server becomes the
client's typed authorization error, not a bare transport exception.
Per-client harness and command:
| Client | Harness | Command |
|---|---|---|
| .NET | `WireFakeGatewayServer` (Kestrel h2c on `127.0.0.1:0`, `MxAccessGatewayBase`) — `MxGatewayClientWireTests` | `dotnet test clients/dotnet/ZB.MOM.WW.MxGateway.Client.Tests/ZB.MOM.WW.MxGateway.Client.Tests.csproj` |
| Python | `FakeGateway` + `serve_gateway` fixture (`grpc.aio` server on `127.0.0.1:0`) — `tests/test_wire_fake_gateway.py` | `python -m pytest` from `clients/python` |
| Go | `fakeGatewayServer` + `newBufconnClient` (`grpc.NewServer` over `bufconn`) — `mxgateway/client_session_test.go` | `go test ./...` from `clients/go` |
| Rust | `spawn_fake_gateway` (tonic `Server` over a loopback `TcpListener`) — `tests/client_behavior.rs` | `cargo test --workspace` from `clients/rust` |
| Java | `TestGatewayService` + `InProcessGateway` (`InProcessServerBuilder`) — `MxGatewayClientSessionTests`; plus `InProcessGatewayHarness` for the CLI tests | `gradle test` from `clients/java` |
All five run in CI: Go, Rust, Python, and the .NET client tests in the `portable`
job, Java in the `java` job.
Adding an RPC to `mxaccess_gateway.proto` does not automatically extend these —
the fake gateways implement only the four session RPCs. Extend the fake in the
client whose behaviour changed rather than adding a parallel harness.
## Client E2E Scripts
`scripts/discover-testmachine-tags.ps1` queries the ZB Galaxy Repository for the
@@ -432,6 +485,88 @@ Run the gateway test project after shared gateway test infrastructure changes:
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj
```
## Running the Gateway Suite on windev
The gateway suite (`ZB.MOM.WW.MxGateway.Tests`, net10.0/x64) is not part of the CI
Windows tier — `windows-x86` and `nightly-windev` run only the x86 Worker build and
`Worker.Tests`. It is still run on windev by hand when a change needs Windows
confirmation, and that run has three Windows-specific characteristics worth knowing
before results are interpreted.
Run it from an isolated clone under `C:\build` checked out to the SHA under test — never
the dirty Desktop checkout, and never the CI clone `C:\build\mxaccessgw-ci`, whose worktree
lock belongs to the Worker tier.
Baseline on an otherwise idle windev (2026-08-10): **855 passed, 0 failed, 29 s**. The
suite is smaller there than the 879 the macOS box runs because some cases are gated to
Unix. Any failure is therefore a real signal — but read the load caveat below before
acting on one.
### Two long-standing "windev-environmental" failures were test bugs, not the environment
Both were dismissed as environmental for months and are now fixed. Neither depended on
anything installed on windev; both failed on **any** Windows host:
- `SelfSignedCertificateProviderTests.GenerateCertificate_HasExpectedSansEkuAndValidity`
asserted SAN content by substring-matching `X509Extension.Format(false)`. That string is
produced by the platform crypto library: Windows' `CryptFormatObject` renders the IPv6
loopback fully expanded (`IP Address=0000:0000:0000:0000:0000:0000:0000:0001`) while the
managed formatter used on macOS/Linux renders `::1`, so the loopback assertion failed on
Windows only. The test now decodes the extension with `X509SubjectAlternativeNameExtension`
and compares parsed `IPAddress` values and DNS names (case-insensitively, as DNS names
are), which is platform-independent.
- `SessionManagerTests.OpenSessionAsync_PipeNameIsShortAndUniquePerPidAndSession` guards the
104-byte macOS `sun_path` budget that NEXT-01 shortened the pipe name to fit. It padded the
measured name up to a five-digit pid but never substituted that worst case *downward*, so a
six-digit pid — routine on Windows, impossible on macOS, where pids stop at 99999 — made the
name one character "too long" against a budget that does not apply to the host running the
test. The check now replaces the running pid's digit count with the five-digit macOS worst
case, so it measures the name *format* rather than the current process's pid.
### The real-pipe suites are load-sensitive
These suites drive real named pipes against a five-second worker startup timeout and start
failing when windev is busy — most often when the x86 Worker tier is building or testing at
the same time. All five passed in the idle baseline above and all five failed in a run taken
while an x86 build and `Worker.Tests` were in flight (that run also took 2 m 21 s against the
idle 29 s):
- `GatewayEndToEndFakeWorkerSmokeTests`, `GatewayEndToEndMultiSubscriberTests`,
`GatewayEndToEndReconnectReplayTests` — fail as
`RpcException Status(StatusCode="Unavailable", Detail="Failed to open session …")`.
- `SessionWorkerClientFactoryFakeWorkerTests.CreateAsync_WhenFakeWorkerStartupFails_ThrowsWorkerClientException`
— the startup timeout beats the protocol violation the test is asserting, so the observed
exception is `TimeoutException` instead of `WorkerClientException`.
- `WorkerClientTests.InvokeAsync_WhenCommandExceedsFrameMax_FailsOnlyThatCommandAndStaysReady`.
- `EventStreamServiceTests.StreamEventsAsync_WithConcurrentStreams_TracksAggregateQueueDepth`
— polls a metric against a five-second deadline. Its helper now reports the unmet condition
rather than letting a bare `TaskCanceledException` escape, so a load-induced timeout here
names what it was waiting for instead of looking like an unexplained cancellation.
A failure in that list is evidence about machine load, not about the change under test. Check
for a concurrent x86 build/test (`Get-Process dotnet, testhost, testhost.net48.x86,
MSBuild, VBCSCompiler`) and re-run the affected class on its own before treating it as real.
windev has 36 logical CPUs and `xunit.runner.json` sets `maxParallelThreads: -1`, so the
suite runs far wider there than on the macOS dev box — that width is what turns these
real-clock deadlines into failures.
### The full-suite testhost does not exit on windev
After the last test completes, the x64 `testhost` process stops doing work but never exits,
so `dotnet test` never returns and the run has to be killed. This does **not** happen on
filtered runs (`--filter …`), which exit normally, and does not happen on the macOS dev box —
it is specific to a full-suite run on windev. Run the full suite with a hang guard so the
wedged host is torn down and the pass/fail summary is still printed:
```powershell
dotnet test src\ZB.MOM.WW.MxGateway.Tests\ZB.MOM.WW.MxGateway.Tests.csproj `
--blame-hang --blame-hang-timeout 5m --blame-hang-dump-type none
```
The summary line ahead of the abort is the real result; the process exit code is nonzero
because of the abort even when every test passed, so read the summary rather than the exit
code. Prefer filtered runs on windev whenever the change under test allows it.
## Continuous Integration
CI runs on Gitea Actions (`.gitea/workflows/ci.yml`; origin is Gitea at
+5 -1
View File
@@ -123,7 +123,11 @@ runs after the whole batch, and only then does every successfully-written
frame's completion resolve — so a caller's `WriteAsync` still does not
complete until its bytes are both written *and* flushed, but a batch that
happened to contain several queued frames pays one flush instead of one per
frame. The event drain loop (`WorkerPipeSession.RunEventDrainLoopAsync`)
frame. Note the ordering this implies at the peer: the frames reach the pipe
before the flush that follows them, so the gateway can read a whole batch
while the writer has not yet flushed it. Anything observing the flush itself
(a test counting flushes, for instance) must wait for the flush, not infer it
from frames arriving. The event drain loop (`WorkerPipeSession.RunEventDrainLoopAsync`)
submits a whole drained event batch through `WriteBatchAsync`, which enqueues
every frame under one `_gate` acquisition, takes the write lock once, and
drains them together, so a burst of N events costs one flush rather than N —
+10
View File
@@ -53,6 +53,16 @@ first, then merge.
- Unreachable-host red: point at a bogus port / stop sshd, confirm the job fails fast, not hangs.
- Concurrency: push two branches back-to-back, confirm the second remote run waits on the lock.
- Nightly: trigger the schedule path, confirm `live` runs and a forced failure opens an issue.
**Done 2026-08-10** (Check 6). Verified two ways: (a) production — every red nightly since
2026-07-17 has auto-filed an issue (#126–#139) authored by the `gitea-actions` bot, e.g. run 672
→ issue #139, with the built-in token masked to `***` in the job log; (b) a forced-failure probe
on the throwaway branch `test/tst25-check6-nightly-issue` (run 677 → issue #140, since closed and
the branch deleted), which reproduced the job shape with `exit 1` in place of the live step and
confirmed the `if: failure()` step fires, the token carries issue-write, and the payload is
well-formed. The probe also caught the one defect: `${{ github.server_url }}` is the
runner-internal `http://gitea:3000`, so the run link in the issue body was unreachable from a
browser — the body now uses the `PUBLIC_SERVER_URL` job env instead (the API call still targets
`github.server_url`, which is what the job container can resolve).
- Confirm no key material appears in job logs.
## Degraded mode
@@ -699,12 +699,26 @@ public sealed class EventStreamServiceTests
};
}
private static async Task WaitUntilAsync(Func<bool> predicate)
// The real-clock deadline here is load-sensitive on a wide host (windev runs this suite
// 36-way parallel). Surfacing the unmet condition instead of letting the bare
// TaskCanceledException escape is what makes such a failure diagnosable rather than a
// mystery cancellation attributed to "the environment".
private static async Task WaitUntilAsync(
Func<bool> predicate,
[CallerArgumentExpression(nameof(predicate))] string? predicateExpression = null)
{
using CancellationTokenSource cancellationTokenSource = new(TestTimeout);
while (!predicate())
{
await Task.Delay(TimeSpan.FromMilliseconds(10), cancellationTokenSource.Token);
try
{
await Task.Delay(TimeSpan.FromMilliseconds(10), cancellationTokenSource.Token);
}
catch (OperationCanceledException)
{
Assert.Fail(
$"Timed out after {TestTimeout} waiting for condition: {predicateExpression}");
}
}
}
@@ -61,10 +61,20 @@ public sealed class SessionManagerTests
// 104-byte sun_path − NUL − ~49-char default macOS TMPDIR − "CoreFxPipe_".
const int MaxPipeNameLength = 104 - 1 - 49 - 11;
int worstCasePidDigits = 5 - Environment.ProcessId.ToString(System.Globalization.CultureInfo.InvariantCulture).Length;
// macOS pids top out at 99999, so 5 digits is the worst case the budget must survive.
// The check must substitute that worst case for the *running* pid's digit count rather
// than pad the measured length upward: Windows pids are routinely 6 digits, which would
// otherwise fail this assertion on a host whose own pipe-name limit (256) is irrelevant
// to the macOS budget being guarded here.
const int WorstCaseMacOsPidDigits = 5;
int runningPidDigits = Environment.ProcessId
.ToString(System.Globalization.CultureInfo.InvariantCulture).Length;
int worstCaseLength = session.PipeName.Length - runningPidDigits + WorstCaseMacOsPidDigits;
Assert.True(
session.PipeName.Length + Math.Max(0, worstCasePidDigits) <= MaxPipeNameLength,
$"Pipe name '{session.PipeName}' would overflow the macOS socket-path budget at a 5-digit pid.");
worstCaseLength <= MaxPipeNameLength,
$"Pipe name '{session.PipeName}' would overflow the macOS socket-path budget at a 5-digit pid " +
$"({worstCaseLength} > {MaxPipeNameLength}).");
}
/// <summary>Verifies that a session opened by an authenticated caller records that caller's API key id in OwnerKeyId.</summary>
@@ -1,3 +1,4 @@
using System.Net;
using System.Security.Cryptography.X509Certificates;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Time.Testing;
@@ -25,15 +26,19 @@ public sealed class SelfSignedCertificateProviderTests
Assert.True(cert.NotBefore.ToUniversalTime() < time.GetUtcNow().UtcDateTime);
Assert.True(cert.HasPrivateKey);
string sans = ReadSubjectAltNames(cert);
Assert.Contains("localhost", sans);
Assert.Contains("gw.internal", sans);
Assert.Contains(Environment.MachineName, sans);
// Format() renders IP SANs as "IP Address:<addr>"; the IPv6 loopback may appear
// as "::1" or its expanded form depending on the platform crypto library.
Assert.Contains("127.0.0.1", sans);
Assert.True(sans.Contains("::1") || sans.Contains("0:0:0:0:0:0:0:1"),
$"Expected IPv6 loopback in SANs but got: {sans}");
X509SubjectAlternativeNameExtension san = ReadSubjectAltNames(cert);
string[] dnsNames = [.. san.EnumerateDnsNames()];
IPAddress[] ipAddresses = [.. san.EnumerateIPAddresses()];
// DNS SANs are compared case-insensitively (DNS names are), and IP SANs are compared
// as parsed IPAddress values. Asserting against the extension's Format() string instead
// would be platform-dependent: Windows' CryptFormatObject renders the IPv6 loopback
// fully expanded ("0000:0000:...:0001") while the managed formatter renders "::1".
Assert.Contains(dnsNames, name => name.Equals("localhost", StringComparison.OrdinalIgnoreCase));
Assert.Contains(dnsNames, name => name.Equals("gw.internal", StringComparison.OrdinalIgnoreCase));
Assert.Contains(dnsNames, name => name.Equals(Environment.MachineName, StringComparison.OrdinalIgnoreCase));
Assert.Contains(IPAddress.Loopback, ipAddresses);
Assert.Contains(IPAddress.IPv6Loopback, ipAddresses);
X509EnhancedKeyUsageExtension eku = cert.Extensions.OfType<X509EnhancedKeyUsageExtension>().Single();
Assert.Contains(eku.EnhancedKeyUsages.Cast<System.Security.Cryptography.Oid>(),
@@ -155,8 +160,6 @@ public sealed class SelfSignedCertificateProviderTests
private const string SubjectAltNameOid = "2.5.29.17";
private static string ReadSubjectAltNames(X509Certificate2 cert)
=> cert.Extensions
.First(e => e.Oid?.Value == SubjectAltNameOid)
.Format(false);
private static X509SubjectAlternativeNameExtension ReadSubjectAltNames(X509Certificate2 cert)
=> new(cert.Extensions.First(e => e.Oid?.Value == SubjectAltNameOid).RawData);
}
@@ -1214,7 +1214,9 @@ public sealed class WorkerPipeSessionTests
using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(15));
using PipePair pipePair = await PipePair.CreateAsync(cancellation.Token);
FakeRuntimeSession runtime = new();
// A far-off heartbeat interval keeps heartbeat flushes out of the measurement window.
// A far-off heartbeat interval keeps every heartbeat after the first out of the measurement
// window; the first beat is sent immediately on entering the message loop (see
// RunHeartbeatLoopAsync) and is closed out explicitly below.
FlushCountingPassthroughStream countingStream = new(pipePair.WorkerStream);
WorkerPipeSession session = CreatePipeSession(
countingStream,
@@ -1227,8 +1229,18 @@ public sealed class WorkerPipeSessionTests
Task runTask = session.RunAsync(cancellation.Token);
await CompleteGatewayHandshakeAsync(pipePair, cancellation.Token);
// Let the idle drain loop settle (no events yet → no flushes) and record the baseline.
await Task.Delay(100, cancellation.Token);
// Close the pre-burst window on an explicit signal rather than a sleep. Three frames precede
// the burst — WorkerHello, WorkerReady, and the immediate first heartbeat — and each is
// flushed only after its bytes are already on the pipe, so having read a frame is no evidence
// that its flush has been counted. Read the first beat (the last pre-burst frame), then wait
// for the writer's deferred flush before sampling the baseline. A fixed delay left the first
// beat's flush free to land after the sample under CI scheduling pressure, where it was
// charged to the burst and the assertion below saw two flushes instead of one.
await ReadUntilAsync(
pipePair.GatewayReader,
WorkerEnvelope.BodyOneofCase.WorkerHeartbeat,
cancellation.Token);
await countingStream.WaitForAllWritesFlushedAsync(cancellation.Token);
int baselineFlushes = countingStream.FlushCount;
// Enqueue a full 128-event batch atomically so the drain loop sees it as one batch.
@@ -1250,8 +1262,15 @@ public sealed class WorkerPipeSessionTests
cancellation.Token);
}
// The whole burst cost exactly one additional flush.
Assert.Equal(1, countingStream.FlushCount - baselineFlushes);
// Take the same edge on the burst's own flush — the 128 frames are on the wire before the
// writer flushes them — then assert on the recorded flush shape: exactly one flush beyond the
// baseline, and that flush carried every event frame of the burst. Asserting the shape (not
// just the count) is what makes the coalescing claim faithful: a split batch would show its
// first flush carrying fewer than the whole burst.
await countingStream.WaitForAllWritesFlushedAsync(cancellation.Token);
IReadOnlyList<int> flushWriteCounts = countingStream.SnapshotFlushWriteCounts();
Assert.Equal(baselineFlushes + 1, flushWriteCounts.Count);
Assert.Equal(burst, flushWriteCounts[baselineFlushes]);
await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
}
@@ -2154,13 +2173,22 @@ public sealed class WorkerPipeSessionTests
}
}
// Wraps the worker side of the pipe and counts FlushAsync calls so a test can assert the event
// drain loop coalesces a burst into a single flush. Delegates every other operation to the inner
// stream; does not own the inner stream's lifetime (PipePair disposes it).
// Wraps the worker side of the pipe and records the flush shape of the frames the writer emits —
// how many stream writes each flush coalesced — so a test can assert the event drain loop turns a
// burst into a single flush. Delegates every other operation to the inner stream; does not own the
// inner stream's lifetime (PipePair disposes it).
//
// The writer flushes only after writing a whole drained batch, so the peer can read every frame of
// that batch before the flush runs. Neither "I read the frames" nor any fixed sleep is evidence
// that a flush has been counted; WaitForAllWritesFlushedAsync is the explicit edge a test must
// take before sampling the counters.
private sealed class FlushCountingPassthroughStream : Stream
{
private readonly Stream inner;
private int flushCount;
private readonly object gate = new();
private readonly List<int> flushWriteCounts = new();
private readonly List<FlushWaiter> waiters = new();
private int writesSinceLastFlush;
/// <summary>Initializes the passthrough over the given inner stream.</summary>
/// <param name="inner">The stream to delegate to.</param>
@@ -2169,8 +2197,53 @@ public sealed class WorkerPipeSessionTests
this.inner = inner;
}
/// <summary>Gets the number of <see cref="FlushAsync"/> calls observed so far.</summary>
public int FlushCount => Volatile.Read(ref flushCount);
/// <summary>Gets the number of flushes observed so far.</summary>
public int FlushCount
{
get
{
lock (gate)
{
return flushWriteCounts.Count;
}
}
}
/// <summary>
/// Returns the number of stream writes coalesced into each flush, in flush order.
/// </summary>
/// <returns>A snapshot of the per-flush write counts.</returns>
public IReadOnlyList<int> SnapshotFlushWriteCounts()
{
lock (gate)
{
return flushWriteCounts.ToArray();
}
}
/// <summary>
/// Completes once every write issued so far has been flushed, giving the caller a
/// happens-before edge on the writer's deferred flush instead of a timing guess.
/// </summary>
/// <param name="cancellationToken">Token to abandon the wait.</param>
/// <returns>A task that completes when no write is left unflushed.</returns>
public Task WaitForAllWritesFlushedAsync(CancellationToken cancellationToken)
{
FlushWaiter waiter;
lock (gate)
{
if (writesSinceLastFlush == 0)
{
return Task.CompletedTask;
}
// Any flush drains every pending write, so the next flush is exactly the edge wanted.
waiter = new FlushWaiter(flushWriteCounts.Count + 1);
waiters.Add(waiter);
}
return AwaitFlushWaiterAsync(waiter, cancellationToken);
}
/// <inheritdoc />
public override bool CanRead => inner.CanRead;
@@ -2194,14 +2267,14 @@ public sealed class WorkerPipeSessionTests
/// <inheritdoc />
public override void Flush()
{
Interlocked.Increment(ref flushCount);
RecordFlush();
inner.Flush();
}
/// <inheritdoc />
public override Task FlushAsync(CancellationToken cancellationToken)
{
Interlocked.Increment(ref flushCount);
RecordFlush();
return inner.FlushAsync(cancellationToken);
}
@@ -2219,11 +2292,87 @@ public sealed class WorkerPipeSessionTests
public override void SetLength(long value) => inner.SetLength(value);
/// <inheritdoc />
public override void Write(byte[] buffer, int offset, int count) => inner.Write(buffer, offset, count);
public override void Write(byte[] buffer, int offset, int count)
{
RecordWrite();
inner.Write(buffer, offset, count);
}
/// <inheritdoc />
public override Task WriteAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken)
=> inner.WriteAsync(buffer, offset, count, cancellationToken);
{
RecordWrite();
return inner.WriteAsync(buffer, offset, count, cancellationToken);
}
private static async Task AwaitFlushWaiterAsync(
FlushWaiter waiter,
CancellationToken cancellationToken)
{
using (cancellationToken.Register(() => waiter.Completion.TrySetCanceled()))
{
await waiter.Completion.Task.ConfigureAwait(false);
}
}
// Counted at write issue, not completion: the peer can observe the bytes as soon as the write
// is issued, so the pending count must already reflect the write by then.
private void RecordWrite()
{
lock (gate)
{
writesSinceLastFlush++;
}
}
private void RecordFlush()
{
List<FlushWaiter>? released = null;
lock (gate)
{
flushWriteCounts.Add(writesSinceLastFlush);
writesSinceLastFlush = 0;
for (int index = waiters.Count - 1; index >= 0; index--)
{
if (waiters[index].TargetFlushCount <= flushWriteCounts.Count)
{
released ??= new List<FlushWaiter>();
released.Add(waiters[index]);
waiters.RemoveAt(index);
}
}
}
if (released is null)
{
return;
}
foreach (FlushWaiter waiter in released)
{
waiter.Completion.TrySetResult(true);
}
}
// A pending WaitForAllWritesFlushedAsync call: completes once the recorded flush count
// reaches TargetFlushCount.
private sealed class FlushWaiter
{
/// <summary>Initializes a waiter released at the given flush ordinal.</summary>
/// <param name="targetFlushCount">Flush count that releases the waiter.</param>
public FlushWaiter(int targetFlushCount)
{
TargetFlushCount = targetFlushCount;
Completion = new TaskCompletionSource<bool>(
TaskCreationOptions.RunContinuationsAsynchronously);
}
/// <summary>Gets the flush count at which this waiter completes.</summary>
public int TargetFlushCount { get; }
/// <summary>Gets the completion signaled when the target flush count is reached.</summary>
public TaskCompletionSource<bool> Completion { get; }
}
}
private sealed class PipePair : IDisposable