Compare commits

..

20 Commits

Author SHA1 Message Date
Joseph Doherty 02baf0c27b Merge fix/testhost-exit-hang: zero-buffer Windows pipes blocked one test's writes forever; windev suite baseline is 879, same as macOS
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m17s
ci / java (push) Successful in 2m45s
ci / portable (push) Successful in 7m57s
2026-08-10 10:04:40 -04:00
Joseph Doherty bfcf82975c test(windev): buffer the test named pipes so the full-suite testhost exits
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m14s
ci / java (push) Successful in 2m19s
ci / portable (push) Successful in 8m48s
The windev full-suite wedge — every test reported, then the x64 testhost sitting
at ~0 CPU forever while `dotnet test` never returns — was one test blocked on a
pipe write, not a leaked thread or an undisposed fixture.

`dotnet-stack report` on the wedged host showed no thread running test code:
xUnit's RunTestsInAssembly was parked on WaitHandle.WaitOne() waiting for the
assembly-finished event, so the wait lived in a suspended async state machine.
`dotnet-dump analyze -c dumpasync` named the frame — WorkerClientTests
.StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout awaiting
WorkerFrameWriter.WriteAsync on a 63-byte frame, with <extra>5__15 = 6, i.e. the
seventh of the twelve events the test pushes past the client's staging bound.

That test faults the worker client on purpose, and a faulted client stops its
read loop by design. The test-side pipe came from the NamedPipeServerStream
overload without buffer arguments, which passes inBufferSize: 0 / outBufferSize: 0
to CreateNamedPipe; on Windows that reserves no buffer at all, so a write
completes only once the peer reads it. Measured on windev, that pipe absorbed
0 bytes against a non-reading peer where the same pipe declared with 64 KiB
buffers absorbed 65 520. On macOS and Linux .NET backs named pipes with Unix
domain sockets whose socket buffer swallows the writes regardless, which is why
the identical test never hung there and the bug read as environmental.

Test-owned server pipes now go through TestSupport/TestNamedPipe.CreateServer in
both test projects, declaring explicit 64 KiB buffers so those tests exercise the
gateway's own staging/queue backpressure rather than the OS pipe's flow control.

Separately, every fake-worker write in WorkerClientTests now goes through
PipePair.WriteAsync, bounded by the class's five-second TestTimeout. That is
where the severity came from: a test method that never returns keeps xUnit from
raising ITestAssemblyFinished, so one unbounded await cost the whole suite its
result. A blocked write is now a named test failure instead of a silent wedge.

The fix also retires a wrong belief the wedge had created. windev reported 855
where macOS reported 879, and that gap was recorded in GatewayTesting.md as
Unix-gated test cases; it was really the results lost when the wedged host was
torn down. The same clone now reports 879 passed, matching macOS exactly.

Product code is unaffected. SessionWorkerClientFactory.CreatePipe keeps the
unbuffered declaration deliberately: both ends run continuous read loops and every
gateway write is bounded by the worker client's _stopCts, so a stalled peer
cancels the write rather than blocking on it.

Verified on windev at this SHA: gateway suite x64 three times (879 passed,
exit 0, no surviving testhost each time) and Worker.Tests x86 twice (400 passed,
11 skipped, exit 0, clean), plus the macOS gateway suite once (879 passed).

Docs: GatewayTesting.md replaces the --blame-hang workaround section with the
root cause and corrects the baseline to 879, CLAUDE.md's Source Update Workflow
no longer tells readers the windev suite wedges, and ToolchainLinks.md records
dotnet-stack and dotnet-dump as installed on windev.
2026-08-10 10:01:57 -04:00
Joseph Doherty f78781d9ef Merge fix/windev-baseline-env-failures: the two 'environmental' windev failures were Windows-only test bugs
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m52s
ci / java (push) Successful in 2m10s
ci / portable (push) Successful in 7m51s
2026-08-10 09:13:55 -04:00
Joseph Doherty e2352d1666 test(windev): fix the two Windows-only gateway test failures dismissed as environmental
ci / java (push) Successful in 2m24s
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 3m33s
ci / portable (push) Successful in 8m13s
Both failures in the windev baseline were test bugs that reproduce on any Windows
host, not anything missing or misconfigured on windev.

SelfSignedCertificateProviderTests.GenerateCertificate_HasExpectedSansEkuAndValidity
asserted SAN content by substring-matching X509Extension.Format(false). That string
comes from the platform crypto library: Windows' CryptFormatObject renders the IPv6
loopback fully expanded (0000:0000:...:0001) where the managed formatter renders
"::1", so the loopback assertion could never hold on Windows. Decode the extension
with X509SubjectAlternativeNameExtension and compare parsed IPAddress values and DNS
names instead, which removes the platform-dependent formatting from the assertion.

SessionManagerTests.OpenSessionAsync_PipeNameIsShortAndUniquePerPidAndSession guards
the 104-byte macOS sun_path budget NEXT-01 shortened the pipe name to fit. It padded
the measured name up to a five-digit pid but never substituted that worst case
downward, so Windows' routine six-digit pids over-counted by a character against a
budget that does not constrain the host running the test. Substitute the five-digit
macOS worst case for the running pid's digit count so the check measures the name
format rather than the current pid.

EventStreamServiceTests.WaitUntilAsync now reports the unmet condition on timeout
instead of letting a bare TaskCanceledException escape. Its five-second real-clock
deadline is genuinely load-sensitive on windev (36 logical CPUs, maxParallelThreads
-1), and an opaque cancellation there is exactly what got the previous failures
filed as "environmental" and left unexplained.

Documents the windev run in docs/GatewayTesting.md: the two fixed bugs and their root
causes, the real-pipe suites whose failures are evidence of machine load rather than
of the change under test, and the full-suite testhost that completes every test and
then never exits (filtered runs exit normally; macOS exits cleanly). Corrects the
CLAUDE.md claim that the suite exits cleanly on the Windows dev box.
2026-08-10 09:05:38 -04:00
Joseph Doherty 347d59fc62 Merge fix/deflake-eventburst: flush-edge sampling replaces wall-clock window in EventBurst_DrainLoopCoalescesFlushes
ci / java (push) Successful in 2m13s
ci / nightly-windev (push) Has been skipped
ci / portable (push) Successful in 9m25s
ci / windows-x86 (push) Failing after 20m21s
2026-08-10 08:53:07 -04:00
Joseph Doherty 404c06b2fa Merge feat/tst-24-client-wire-tests: .NET/Python real-server wire tests, Python channel-loop fix, TST-05 revisit (partial)
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m45s
ci / portable (push) Successful in 11m39s
ci / windows-x86 (push) Failing after 21m2s
# Conflicts:
#	archreview/remediation/00-tracking.md
2026-08-10 08:23:39 -04:00
Joseph Doherty a8f86b5336 test(tst-24): drive the .NET and Python clients against real in-process gRPC servers
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m19s
ci / portable (push) Successful in 23m59s
ci / windows-x86 (push) Failing after 58m42s
TST-24 asked for per-client wire tests against a fake gateway. An audit first
corrected the finding's premise: Go, Rust, and Java already had them — bufconn,
a loopback tonic server, and InProcessServerBuilder respectively — each already
asserting the round trip, the server-observed bearer header, and the ReplayGap
sentinel. The two genuine gaps were .NET (every test substituted the transport
interface; the test project had no server package at all) and Python (stub
monkeypatching everywhere but one opt-in TLS test).

Both now serve mxaccess_gateway.v1.MxAccessGateway over a real transport —
Kestrel h2c and grpc.aio, each on an ephemeral loopback port — and drive the
ordinary public client API against it. Only the gateway's behaviour is canned;
the framing, serialization, metadata, and status codes are genuine. Four shapes
each: full round trip with every reply field asserted, the authorization header
as received by the server (including on the streaming RPC), the ReplayGap
sentinel surfaced as the client's typed signal, and a real PERMISSION_DENIED
mapping to the typed authorization error.

The .NET client was only ever compiled in CI, never tested, so the portable job
gains a dotnet test step.

Fixes a bug the new tests caught on their first run: Python's connect() built the
grpc.aio channel inside asyncio.to_thread, and a grpc.aio channel binds to the
event loop current on the constructing thread, so every non-stub connection
raised 'There is no current event loop in thread'. No mock-based test could see
it, and the test guarding the off-loop behaviour patched create_channel and so
asserted the bug. Split resolve_channel_security (blocking TOFU probe, off-loop)
from create_channel (on-loop); the guard tests now assert both halves.
2026-08-10 08:22:25 -04:00
Joseph Doherty 50322bacb3 test(worker): deflake EventBurst_DrainLoopCoalescesFlushes
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m13s
ci / java (push) Successful in 2m38s
ci / portable (push) Successful in 8m17s
Root cause. The test sampled its flush baseline after a bare `Task.Delay(100)`
and then charged every later flush to the 128-event burst. Two facts make that
window unsound:

1. `RunHeartbeatLoopAsync` sends its first beat *immediately* on entering the
   message loop — the far-off `HeartbeatInterval` only spaces later beats — so
   the pre-burst frames are WorkerHello, WorkerReady, and a heartbeat, not the
   two the test's comment assumed.
2. `WorkerFrameWriter.DrainQueuedFramesAsync` flushes *after* writing a drained
   batch, so the bytes reach the pipe before the flush runs. Reading a frame off
   the gateway side is therefore no evidence that its flush has been counted,
   and no sleep makes it evidence.

Under load on the shared windows-x86 runner the first heartbeat's flush was
scheduled after the 100 ms sample, so it landed inside the measured window and
the assertion saw two flushes for the burst — exactly the observed
`Expected: 1 / Actual: 2` (Gitea run 675, job 2558, and the same failure since
a346d51). Nothing about the coalescing behavior was wrong; only the test's
timing assumption.

Fix. Replace the sleep with explicit synchronization, no widened timeouts.
`FlushCountingPassthroughStream` now records the flush *shape* — the number of
stream writes coalesced into each flush — and exposes
`WaitForAllWritesFlushedAsync`, a TaskCompletionSource signal released when the
next flush drains the pending writes. The test reads the first heartbeat (the
last pre-burst frame), waits for its flush, and only then samples the baseline;
after the burst it takes the same edge before asserting. The assertion is also
sharpened from a bare count to the shape: exactly one flush beyond the baseline
*and* that flush carried all 128 event frames, so a split batch fails even if
the reader observes it mid-split.

docs/WorkerFrameProtocol.md notes the peer-visible ordering the fix turns on:
frames reach the pipe before the flush that follows them, so an observer of the
flush must wait for it rather than infer it from frames arriving.
2026-08-10 08:13:30 -04:00
Joseph Doherty 9e6f66dd8f Merge fix/tst25-nightly-issue-path: reachable run links in nightly failure issues (TST-25 Check 6 closed)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Failing after 1m16s
ci / java (push) Successful in 2m13s
ci / portable (push) Successful in 8m46s
2026-08-10 08:11:42 -04:00
Joseph Doherty 30c92e8e59 fix(ci): give the nightly-windev issue body a reachable run link (TST-25 Check 6)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Failing after 1m13s
ci / java (push) Successful in 2m7s
ci / portable (push) Successful in 8m30s
Exercising TST-25 acceptance Check 6 end-to-end showed the on-failure issue
step works — it has filed an issue on every red nightly since 2026-07-17
(#126-#139) — but that every one of those issues links `http://gitea:3000`,
the runner-internal origin Gitea hands the runner. That URL is correct for
the issue-creation API call (the job container resolves `gitea` only on the
docker network and has no LAN egress to the public origin) and unreachable
for anyone clicking out of the issue.

Keep `github.server_url` for the API call; add a `PUBLIC_SERVER_URL` job env
used only for the browser-facing link in the body. Verified with a
forced-failure probe run (677) whose issue (#140) carries a public link that
returns 200.

Also records Check 6 as done in scripts/ci/README.md and corrects the
2026-07-13 tracking entry that wrote the check off as abandoned.
2026-08-10 08:10:08 -04:00
Joseph Doherty d4302c6ac4 docs(tst-05): revisit under the restored windev tier — scheduled half closed, control-command coverage still open
The nightly-windev job (cycle-2 TST-25) discharges TST-05's scheduling design:
cron 0 6 * * * runs run-windev-ci.sh live, which sets
MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1, runs WorkerLiveMxAccessSmokeTests on windev,
and opens a Gitea issue on failure. That converts 'run by memory' into 'runs
nightly, reports failures'.

The finding's other half — audit the live suite for coverage of each of the eleven
late-added command kinds — is not discharged. The audit's answer is negative for
five: the suite covers all six COM commands and none of the five control commands
(Ping/GetSessionState/GetWorkerInfo/DrainEvents/ShutdownWorker), which are exactly
the kinds the finding calls masked by FakeWorkerHarness canned replies. Recorded as
Partially done with the residual test work specified, rather than claiming closure.
2026-08-10 08:08:01 -04:00
Joseph Doherty c46e5bbd15 Merge fix/next-cycle-batch: resolve NEXT-01/02/03/04/05/09 from the 2026-07-12 next-cycle candidates
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Failing after 1m18s
ci / java (push) Successful in 2m46s
ci / portable (push) Successful in 7m50s
- NEXT-01: pipe names shortened to mxgw-{pid}-{sessionUid}; macOS full gateway suite green (879/879) under default TMPDIR for the first time
- NEXT-02: .NET + Java CLIs render ReplayGap as the typed cross-CLI row; five-CLI contract converged
- NEXT-03: best-effort reconcile/live dedup on the alarm feed; at-least-once consumer contract unchanged
- NEXT-04: fault-observing continuation on frames abandoned by cancellation in the worker frame writer
- NEXT-05: lazy tombstone purge kept, decision documented in WorkerFrameProtocol.md
- NEXT-09: Windows builds stamp a real short SHA into InformationalVersion (verified 0.1.2+5fe74db on windev), with a SHA-shape guard

windev x86 Worker.Tests 399 passed / 11 skipped / 1 failed = the documented
EventBurst_DrainLoopCoalescesFlushes flake; .NET client 35/35, Java 52/52.
NEXT-08 and NEXT-10 remain open (posture/cross-repo decisions).
2026-08-10 06:11:22 -04:00
Joseph Doherty 5fe74db971 docs(tracking): strike NEXT-01/02/03/04/05/09 as resolved 2026-08-10
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Failing after 1m15s
ci / java (push) Successful in 2m9s
ci / portable (push) Successful in 8m12s
Six of the eight open next-cycle candidates are closed in this batch
(fix/next-cycle-batch): macOS pipe-path length, .NET/Java CLI ReplayGap
rendering, alarm reconcile/live dedup, frame-writer unobserved-fault
hygiene (plus the NEXT-05 lazy-purge decision), and the Windows
InformationalVersion stamp. NEXT-08 (GLAuth TLS posture) and NEXT-10
(glauth.md user-table reconciliation entangled with the OPC-UA group
taxonomy) remain open by design — both need cross-repo/posture decisions,
not gateway code.
2026-08-10 06:07:00 -04:00
Joseph Doherty 2c03e0a684 fix(alarms): dedup reconcile/live duplicate broadcasts on the alarm feed (NEXT-03)
A periodic reconcile can synthesize a repair transition whose matching live
transition is still buffered in the alarm lease; both then broadcast as
indistinguishable duplicates on StreamAlarms and the dashboard hub. Nothing
serializes the two paths, and a correct serialization needs a worker-side
high-water mark on QueryActiveAlarms (proto + worker change + a stall path),
so this closes the common case with a local best-effort dedup instead: both
paths already carry the same worker-derived identity — the worker stamps
record.TransitionTimestampUtc into both OnAlarmTransitionEvent's
transition_timestamp and ActiveAlarmSnapshot.last_transition_timestamp — so
ApplyTransition suppresses a live transition whose (timestamp, resulting
state) the cache already carries from a repair. The Clear leg has no cache
entry left to compare, so ApplyReconcile tombstones each synthesized Clear by
the instance's original_raise_timestamp for one reconcile generation; a
matching live Clear consumes the tombstone, while a new raise/clear cycle
carries a newer raise timestamp and passes. Suppression fires only on a
positive marker match — unset timestamps keep today's behavior — so the
documented at-least-once consumer contract stands (gateway.md, Sessions.md
updated in the same change).

Tests: two new regressions drive the exact race through the GWC-26 harness
(repair-then-buffered-live for Raise and for Clear, each with a genuine
follow-up transition proving no over-suppression); GatewayAlarmMonitor
suites 18/18.
2026-08-10 06:06:17 -04:00
Joseph Doherty 8624e21372 fix(clients): render ReplayGap as the typed cross-CLI row in the .NET and Java CLIs (NEXT-02)
The .NET and Java stream-events commands handed the raw ReplayGap sentinel
MxEvent to their protobuf JSON formatters (Java text mode printed
'0 MX_EVENT_FAMILY_UNSPECIFIED'), while the Go/Python/Rust CLIs already emit
the typed row (CLI-35/36). Both now branch on the sentinel: Java text mode
prints 'REPLAY_GAP requested_after=<n> oldest_available=<n>' and JSON mode a
hand-built {"replayGap":{...}} line via nextItem()/isReplayGap(); the .NET
CLI emits the same hand-built row in jsonl/text (its text mode is
JSON-per-line) and inside the --json events array. Rows are hand-built so
the cursors are JSON numbers like the other three CLIs, not the proto3 JSON
mapping's quoted uint64 strings — the CrossLanguageSmokeMatrix divergence
table collapses to a single converged contract.

Tests: .NET MxGatewayClientCliTests 35/35 (new RendersReplayGapAsTypedRow
covers jsonl + aggregate); Java gradle test 52/52 (new
streamEventsRendersReplayGapAsTypedRow covers --json + text over the
in-process harness). No generated-file churn.
2026-08-10 06:01:44 -04:00
Joseph Doherty 84dbf20a43 fix(worker): observe faults on frames abandoned by cancellation (NEXT-04, NEXT-05 decision)
A WriteAsync/WriteBatchAsync caller cancelled after the draining lock-holder
claimed its frame unwinds without awaiting that frame's completion; the same
holds for a frame already faulted by a concurrent FailAllQueued, where
TrySetCanceled loses. A later wire-write failure then lands TrySetException on
a task with no awaiter and surfaces as TaskScheduler.UnobservedTaskException.
The tombstone helpers now attach a fault-observing continuation to every frame
in the cancelled call (a cancelled task never fires OnlyOnFaulted, so
unconditional attach is safe), outside _gate because an already-faulted task
runs the continuation inline.

NEXT-05 is resolved as a documented decision, not a code change: tombstoned
entries keep their lazy DequeueNext purge — any subsequent write drains both
queues to empty and the heartbeat loop bounds residency to one interval, while
eager Queue<T> rebuilds under _gate would add ordering-invariant surface for
no gain. Rationale recorded in docs/WorkerFrameProtocol.md alongside the
WRK-22 residual-window contract.

New regression test drives the exact abandonment: gated stream holds writer A
mid-write, the queued event frame is claimed and blocked mid-write, its caller
is cancelled, the write then faults with a marker exception, and the test
asserts the marker never reaches UnobservedTaskException after a forced GC.
net48 x86 build/test runs on windev with the rest of this batch.
2026-08-10 05:57:53 -04:00
Joseph Doherty 8769ee9765 fix(sessions): keep named-pipe socket paths inside the macOS sun_path limit (NEXT-01)
The pipe name mxaccess-gateway-{pid}-session-{32hex} plus .NET's
CoreFxPipe_ prefix overflowed the 104-byte Unix-domain-socket path limit
under the default per-user macOS TMPDIR (~49 chars), so every test that
opened a real pipe threw ArgumentOutOfRangeException at pipe creation
unless TMPDIR=/tmp was exported. Rename to mxgw-{pid}-{sessionUid} (the
session guid hex without the session- prefix; worst-case 43 chars) and
shorten the three test-fixture names the same way. Uniqueness is
unchanged: gateway pid + full session guid. The worker receives the pipe
name via its launch command line, so mixed Server/Worker deploy SHAs are
unaffected. Docs updated in the same change (gateway.md,
GatewayProcessDesign, GatewayConfiguration, Sessions, CLAUDE.md); new
regression test pins the format and the length budget.

Verified: SessionManagerTests 39/39; SessionWorkerClientFactory,
GatewayEndToEndFakeWorkerSmoke, WorkerClient, and ReconnectReplay suites
33/33 under the default macOS TMPDIR — this also retires the
previously-misdiagnosed 'macOS pipe-timeout test failures': they were
this path-length throw, not a timeout-message defect.
2026-08-10 05:54:11 -04:00
Joseph Doherty 0152180929 build(tst-11): stop Windows builds stamping git stderr into InformationalVersion (NEXT-09)
MSBuildThisFileDirectory ends in a backslash, which escaped the closing quote
of the Exec command on Windows; git then failed and, because the target runs
with ContinueOnError + ConsoleToMSBuild (which mixes stderr into
ConsoleOutput), the failure text was stamped as the source revision — an
observed stamp read '0.1.2+fatal: cannot change to ...'. Append '.' to the
quoted path so the trailing separator can no longer escape the quote, and
gate SourceRevisionId on a short-SHA shape so no future git failure text can
become the revision either. Windows verification runs on windev with the
rest of this batch; macOS stamp confirmed unchanged (0.2.0+75c71ad).
2026-08-10 05:49:44 -04:00
Joseph Doherty 75c71adf45 docs(gateway): MxStatusDetail vocabulary table for statuses consumers
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Failing after 1m12s
ci / java (push) Successful in 2m2s
ci / portable (push) Successful in 8m14s
Requested by OtOpcUa after 06/S-1 closure: their OPC UA status mapping needs
the per-code detail vocabulary (a byte-truncation bug mapped refused writes
into the Good band). Source: installed toolkit interop enum via the MXAccess
analysis project.
2026-08-09 20:13:57 -04:00
Joseph Doherty 53f69cde37 Merge fix/write-completion-plain-writes: correlate OnWriteComplete onto plain Write/Write2 (06/S-1 follow-up)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Failing after 1m25s
ci / java (push) Successful in 2m23s
ci / portable (push) Successful in 8m5s
2026-08-09 19:48:26 -04:00
50 changed files with 2945 additions and 224 deletions
+15 -1
View File
@@ -83,6 +83,12 @@ jobs:
- name: .NET client
run: dotnet build clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx -c Release
# The .NET client was previously only compiled here, so its test project — including
# MxGatewayClientWireTests, which drives the client against a real loopback gRPC
# server (TST-24) — never ran in CI. Every other client job already runs its tests.
- name: .NET client tests
run: dotnet test clients/dotnet/ZB.MOM.WW.MxGateway.Client.Tests/ZB.MOM.WW.MxGateway.Client.Tests.csproj -c Release --no-build
- name: Go client
working-directory: clients/go
run: |
@@ -165,6 +171,14 @@ jobs:
# visible even though nobody watches the Actions page.
if: github.event_name == 'schedule'
runs-on: ubuntu-latest
env:
# `github.server_url` is the URL Gitea hands the runner — the docker-network-internal
# `http://gitea:3000`. That is the right base for the issue-creation API call below (the job
# container resolves `gitea` only on that network, and has no LAN egress to the public origin),
# but it is useless as a link a human clicks out of the issue. So browser-facing URLs in the
# issue body use the public origin instead. TST-25 acceptance Check 6 caught this: every
# nightly issue since #126 carried an unreachable `http://gitea:3000/...` run link.
PUBLIC_SERVER_URL: https://gitea.dohertylan.com
steps:
- uses: actions/checkout@v4
- name: Full Worker.Tests + live-MXAccess smoke on windev
@@ -184,7 +198,7 @@ jobs:
-H "Authorization: token ${{ github.token }}" \
-H "Content-Type: application/json" \
"${{ github.server_url }}/api/v1/repos/${{ github.repository }}/issues" \
-d "{\"title\":\"nightly-windev failed on ${{ github.sha }}\",\"body\":\"The scheduled windev Worker + live-MXAccess run failed: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} . A red nightly may mean the Windows tier is down rather than the change — see docs/GatewayTesting.md (Continuous Integration).\"}"
-d "{\"title\":\"nightly-windev failed on ${{ github.sha }}\",\"body\":\"The scheduled windev Worker + live-MXAccess run failed: ${PUBLIC_SERVER_URL}/${{ github.repository }}/actions/runs/${{ github.run_id }} . A red nightly may mean the Windows tier is down rather than the change — see docs/GatewayTesting.md (Continuous Integration).\"}"
# NOTE: there is intentionally no native `windows` runner job. act_runner v0.6.1 host-mode on
# Windows is broken and Windows containers are impractical for the net48/x86/MXAccess Worker, so the
+2 -2
View File
@@ -10,7 +10,7 @@ The architecture is a two-process design — read `gateway.md` before making str
- **Gateway** (`src/ZB.MOM.WW.MxGateway.Server`, .NET 10, x64): ASP.NET Core gRPC server. Owns the public API, sessions, auth, the Blazor dashboard, and the Galaxy Repository SQL browse RPCs. The Galaxy-browse implementation comes from the shared **`ZB.MOM.WW.GalaxyRepository`** package (`AddZbGalaxyRepository`/`MapZbGalaxyRepository`), not inline code; mxaccessgw adds `GatewayBrowseScopeProvider` (per-key browse-subtree scoping) and a host-side dashboard summary projector. See `A2-galaxyrepository-adoption-handoff.md`. **Never instantiates MXAccess COM directly.**
- **Worker** (`src/ZB.MOM.WW.MxGateway.Worker`, .NET Framework 4.8, **x86**): one process per session. Owns one MXAccess COM instance on a dedicated STA, pumps Windows messages, and converts COM events to protobuf.
- **IPC**: gateway↔worker uses one bidirectional named pipe per worker (`mxaccess-gateway-{gatewayPid}-{sessionId}`) with length-prefixed `WorkerEnvelope` protobuf frames. Gateway hosts the pipe server and launches the worker. **gRPC is not used inside the worker** — .NET Framework 4.8 doesn't have a first-class gRPC stack.
- **IPC**: gateway↔worker uses one bidirectional named pipe per worker (`mxgw-{gatewayPid}-{sessionUid}` — kept short so the macOS/Linux test matrix's Unix-domain-socket path fits the 104-byte macOS `sun_path` limit) with length-prefixed `WorkerEnvelope` protobuf frames. Gateway hosts the pipe server and launches the worker. **gRPC is not used inside the worker** — .NET Framework 4.8 doesn't have a first-class gRPC stack.
- **Contracts** (`src/ZB.MOM.WW.MxGateway.Contracts`): multi-targets `net10.0;net48` and owns the `.proto` files (`mxaccess_gateway.proto`, `mxaccess_worker.proto`, `galaxy_repository.proto`). All other projects consume the generated types from here. Do not hand-edit anything under `Generated/`. Note `galaxy_repository.proto` is intentionally kept here as the generation source for the language clients even though the gateway server consumes the wire-identical Galaxy types from the `ZB.MOM.WW.GalaxyRepository` package — it is not dead code; deleting it breaks all five clients.
The worker must do all MXAccess COM calls on its dedicated STA thread, and the STA loop must pump Windows messages (`MsgWaitForMultipleObjectsEx` + `PeekMessage`/`DispatchMessage`) so MXAccess events deliver. A plain blocking queue on an STA is not enough.
@@ -124,7 +124,7 @@ powershell -ExecutionPolicy Bypass -File scripts/run-client-e2e-tests.ps1
When source code changes, build and test the affected component before reporting work done. If the change crosses component boundaries, build each affected component — don't rely on a single top-level build:
**Run targeted tests per task, never the full suite each time.** When executing a plan task-by-task, run only the tests that exercise the code that task touched (`dotnet test --filter "FullyQualifiedName~<TestClass>"`, or the per-task test named in the plan). The full gateway suite is slow — run it at most once per phase (after a related batch of tasks lands), not after every task. This is a speed guideline, not a correctness one: the suite exits cleanly (verified on macOS and the Windows dev box — 0 surviving `testhost`/worker processes after a full run), so filtered runs are about turnaround, not about avoiding a process leak.
**Run targeted tests per task, never the full suite each time.** When executing a plan task-by-task, run only the tests that exercise the code that task touched (`dotnet test --filter "FullyQualifiedName~<TestClass>"`, or the per-task test named in the plan). The full gateway suite is slow — run it at most once per phase (after a related batch of tasks lands), not after every task. This is a speed guideline, not a correctness one: the suite exits cleanly on both macOS and windev (0 surviving `testhost`/worker processes after a full run), so filtered runs are about turnaround. The long-standing windev full-suite wedge — every test reported, then `testhost` never exiting — was a zero-buffer named pipe blocking one test's write forever; it is fixed and the `--blame-hang` workaround is no longer needed. See "Running the Gateway Suite on windev" in `docs/GatewayTesting.md`, which also documents the load-sensitivity caveat that still applies there.
| Changed area | Required verification |
|---|---|
@@ -150,7 +150,7 @@ Sequence these together rather than piecemeal — several are one change set spa
- Close **CLI-24** and **CLI-34** as `Done` (incidentally fixed; evidence in [../50-clients.md](../50-clients.md)).
- ~~When CLI-38 lands, close old **CLI-08** with a pointer here.~~ Done 2026-08-07: CLI-38 landed and old CLI-08 is now `Done` in the first-cycle tracker, pointing at [CLI-38](50-clients.md#cli-38--align-netgojava-on-hresult--0-lands-prior-cli-08-cures-the-doc-drift---medium--p1).
- ~~When WRK-26 lands, its doc section also discharges the WorkerFrameProtocol gap~~ Done 2026-08-07: WRK-26 landed and **IPC-29** is discharged, pointing at [WRK-26](20-worker.md#wrk-26--write-priority-and-overflow-doc-drift-from-the-wrk-07-change---low--p1). When TST-25 lands, revisit old **TST-05** (scheduled live smoke) and **TST-24** (client wire tests), which it unlocks.
- ~~When WRK-26 lands, its doc section also discharges the WorkerFrameProtocol gap~~ Done 2026-08-07: WRK-26 landed and **IPC-29** is discharged, pointing at [WRK-26](20-worker.md#wrk-26--write-priority-and-overflow-doc-drift-from-the-wrk-07-change---low--p1). ~~When TST-25 lands, revisit old **TST-05** (scheduled live smoke) and **TST-24** (client wire tests), which it unlocks.~~ TST-05 revisited 2026-08-10: `Partially done` in the first-cycle tracker — the `nightly-windev` job closes the scheduled-cadence half, but the finding's coverage-audit half stays open (the live suite reaches all six late-added COM commands and none of the five control commands). TST-24 revisited in the same change and closed `Done`: Go/Rust/Java already had real-server wire tests, and the two genuine gaps (.NET, Python) now have them plus a CI step.
## Change log
@@ -159,7 +159,7 @@ Sequence these together rather than piecemeal — several are one change set spa
| 2026-07-13 | Initial tracking doc generated from the six domain remediation designs. All 47 findings `Not started` (IPC-31, SEC-35 `N/A`). |
| 2026-07-13 | TST-25/TST-26 → `In progress` (branch `fix/tst-25-windev-ci`). Added `scripts/ci/{windev-worker-ci.ps1,run-windev-ci.sh,windev.known_hosts}`, `windows-x86` (per-push) + `nightly-windev` (scheduled) jobs in `ci.yml`, and the TST-26 doc/comment fixes (GatewayTesting.md, Contracts.md, check-codegen.ps1). Mechanism hand-verified on windev: `build`→0, bogus-SHA→nonzero (lock released), `test`→356 passed/0 failed in ~50s (per-push stays `test`, no demotion), and run-windev-ci.sh SSH+EncodedCommand exit-code propagation confirmed. |
| 2026-07-13 | Operator bring-up complete: dedicated CI ed25519 key installed in windev `administrators_authorized_keys` (authorized into `dohertj2`, which owns the working MXAccess/toolchain env — a fresh OS account would break the build; the key is independently revocable), Gitea secrets `WINDEV_SSH_KEY`/`WINDEV_SSH_KNOWN_HOSTS` + variable `WINDEV_SSH_USER=dohertj2` stored, runner→`10.100.0.48:22` egress verified on the `traefik` net, issue-write confirmed. **TST-25/TST-26 → `Done`:** credentialed `windows-x86` ran GREEN on `d769244` (Gitea run #37) — Linux runner SSHed windev, checked out the SHA in `C:\build\mxaccessgw-ci` under lock, ran the x86 Worker build + `Worker.Tests`, exit 0; `nightly-windev` correctly skipped on the push event. Branch merged to `main`. Follow-ups (old tracker): revisit **TST-05** (scheduled live smoke — now covered by `nightly-windev`) and **TST-24** (client wire tests) which this unlocks. |
| 2026-07-13 | Ran the TST-25 acceptance checks (scripts/ci/README.md) — they caught **two real CI defects, both fixed** on `fix/tst-25-ci-key-log-leak`: (1) **CI SSH key leaked in cleartext** in the `windows-x86` step env echo (Gitea's line-oriented masker missed the multiline PEM) — rotated the CI key on windev (old pubkey revoked), stored the key **base64-encoded** so the masker redacts it to `***` (confirmed on run #38), taught `run-windev-ci.sh` to decode, dropped the redundant public known-hosts secret from the job env; (2) **bootstrap lock race**`run-windev-ci.sh`'s pre-hand-off `git fetch`/`checkout` ran outside the worktree lock, so concurrent runs collided on `.git/index.lock`; the bootstrap now holds the lock (ps1 re-uses it via `MXGW_CI_LOCK_HELD`), retest confirmed clean serialization. Also **deflaked** `SessionManagerTests` fail-fast timing assertions (absolute `<100ms` wall-clock bound flaked under CI load; now anchored to the configured timeout / dropped for the zero-timeout case). Checks passed: unreachable-host fast-fail (exit 255/15s), deliberate-red propagation (Worker.Tests failure → exit 1), lock concurrency (2nd run waits), no-key-in-logs (masked). Merge target `df7e20d` verified GREEN via the local windev path (Worker build + 356 tests); merged to `main` `19cbf7b`. Check 6 (forced-failure nightly issue): issue endpoint+token proven live at bring-up (#124); in-CI forced-failure probe abandoned to shared-runner congestion (residual `if: failure()` gating is standard Actions). |
| 2026-07-13 | Ran the TST-25 acceptance checks (scripts/ci/README.md) — they caught **two real CI defects, both fixed** on `fix/tst-25-ci-key-log-leak`: (1) **CI SSH key leaked in cleartext** in the `windows-x86` step env echo (Gitea's line-oriented masker missed the multiline PEM) — rotated the CI key on windev (old pubkey revoked), stored the key **base64-encoded** so the masker redacts it to `***` (confirmed on run #38), taught `run-windev-ci.sh` to decode, dropped the redundant public known-hosts secret from the job env; (2) **bootstrap lock race**`run-windev-ci.sh`'s pre-hand-off `git fetch`/`checkout` ran outside the worktree lock, so concurrent runs collided on `.git/index.lock`; the bootstrap now holds the lock (ps1 re-uses it via `MXGW_CI_LOCK_HELD`), retest confirmed clean serialization. Also **deflaked** `SessionManagerTests` fail-fast timing assertions (absolute `<100ms` wall-clock bound flaked under CI load; now anchored to the configured timeout / dropped for the zero-timeout case). Checks passed: unreachable-host fast-fail (exit 255/15s), deliberate-red propagation (Worker.Tests failure → exit 1), lock concurrency (2nd run waits), no-key-in-logs (masked). Merge target `df7e20d` verified GREEN via the local windev path (Worker build + 356 tests); merged to `main` `19cbf7b`. Check 6 (forced-failure nightly issue): issue endpoint+token proven live at bring-up (#124); in-CI forced-failure probe abandoned to shared-runner congestion (residual `if: failure()` gating is standard Actions). **Superseded 2026-08-10 — Check 6 is now Done; see the 2026-08-10 change-log entry in `archreview/remediation/00-tracking.md`.** |
| 2026-07-13 | New finding **TST-30** (`Low`/`P2`) added — surfaced during TST-25 acceptance: CI runs on a single shared `gitea-runner` (`maxParallel=1`, co-located `10.100.0.35`) interleaved with `dohertj2/lmxopcua`, and Gitea 1.26 exposes no run cancel/delete, so queue latency is unbounded under cross-repo contention and the runner is a single point of failure. Design: add a second/labelled runner + document the no-cancel reality and the `run-windev-ci.sh` bypass. Roll-ups updated (Testing Low 2→3, total 47→48; P2 9→10). |
| 2026-08-07 | **TST-29 → `Done`:** migrated the Phase-5 (orphan-worker reattach) deferred-not-planned governance record and the settled Phase-4 Viewer-default decision from `oldtasks.md` into a new "Session-Resilience Epic Scope" entry in `docs/DesignDecisions.md`; repointed CLAUDE.md and `stillpending.md:7,165` from `oldtasks.md` to `docs/DesignDecisions.md` / `docs/plans/2026-06-15-session-resilience.md.tasks.json`; `git rm oldtasks.md`. The five untracked root docs-review artifacts (`MxAccessGateway-docs-{issues,fixed,final}.md`, `MxGatewayClient-docs-{issues,fixed}.md`) were absent from this worktree — delete from the main working tree separately. |
| 2026-08-07 | **GWC-24 → `Done`** (branch `fix/gwc-24-staging-bound`). `WorkerClient._eventStaging` is now `Channel.CreateBounded` at `2 × EventChannelCapacity` (`Wait`, single reader/writer, no sync continuations); a rejected staging `TryWrite` faults the client `ProtocolViolation` with `QueueOverflow("worker-event-staging")` unless `IsTerminalState()` (shutdown stays a silent drop), so a consumer draining slower than its worker produces dies at a fixed ceiling instead of growing gateway memory. Queue-depth accounting moved from `EnqueueWorkerEventAsync` to `StageWorkerEvent`, so the single gauge reports staged + queued; the timed-write fault (`EventChannelFullModeTimeout` / `QueueOverflow("worker-events")`) is unchanged and still catches the full-stall case first. No new config key — total gateway-side buffering is `3 × MxGateway:Events:QueueCapacity`, derived; coordination with still-open old **GWC-21** (`EventChannelFullModeTimeout` configurability) remains open and was not blocked on. Docs same commit: `GatewayProcessDesign.md` (two overflow faults), `MxAccessWorkerInstanceDesign.md`, `GatewayConfiguration.md`, `Metrics.md`. Tests: `WorkerClientTests.StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout` and `.WorkerEventQueueDepthGaugeCountsStagedEvents`; `WorkerClientTests` 22/22 green, `NonWindows.slnx` builds with 0 warnings. |
@@ -175,7 +175,7 @@ Independent of the runner count, document the **no-cancel** reality (Gitea 1.26
## Cross-domain dependencies
- **TST-25 → old TST-05 / old TST-24:** the SSH-driven nightly is where the scheduled live-MXAccess smoke lands (closes old TST-05), and a working CI Windows tier unblocks wiring client wire-behavior tests into CI (old TST-24).
- **TST-25 → old TST-05 / old TST-24:** the SSH-driven nightly is where the scheduled live-MXAccess smoke lands (closes old TST-05), and a working CI Windows tier unblocks wiring client wire-behavior tests into CI (old TST-24). *Followed up 2026-08-10:* old TST-05 is `Partially done` — the nightly closes the scheduling half, but the live suite still covers none of the five worker **control** commands; old TST-24 is `Done`, and its client wire tests turned out to need no Windows tier at all (they run in the `portable` job). See the first-cycle tracker.
- **TST-25 ↔ IPC-24/IPC-25:** the nightly windev job is also the natural home for any Windows-side codegen verification the contracts/IPC remediation adds; coordinate job naming so both plans extend the same `windows-x86`/nightly jobs rather than adding parallel ones.
- **TST-26 ⊂ TST-25:** same commit, by rule.
- **TST-27:** ships in the cycle's P1 doc-drift batch alongside WRK-26 and CLI-42 (roadmap item 8); its `/browse` residual stays with TST-16 (prior cycle).
@@ -4,15 +4,15 @@ These were discovered while remediating the 2026-07-12 backlog but were **out of
| ID (proposed) | Area | Severity (est.) | Summary |
|---|---|---|---|
| NEXT-01 | Testing / macOS | Low | Fake-worker/e2e gateway tests fail on macOS under the default `TMPDIR` because the `CoreFxPipe_mxaccess-gateway-{pid}-{sessionId}` path exceeds the 104-char Unix-domain-socket `sun_path` limit under `/var/folders/…/T/`. Workaround today is `TMPDIR=/tmp`. Fix options: shorten the pipe name, or document the `TMPDIR=/tmp` requirement in `docs/GatewayTesting.md`. Surfaced independently by multiple remediation agents. |
| NEXT-02 | Clients (.NET, Java) | Low | The .NET and Java CLIs render the raw `ReplayGap` sentinel `MxEvent` on `stream-events` instead of a typed gap row — Java text mode prints `0 MX_EVENT_FAMILY_UNSPECIFIED`. Same defect class as CLI-36 (Go) / CLI-35 (Python), which were fixed this cycle; the .NET/Java halves were out of scope. The cross-language smoke matrix now records this divergence honestly. |
| NEXT-03 | Gateway alarms | Low | `GatewayAlarmMonitor.ApplyReconcile` feed-repair broadcasts (the new acked-delta from GWC-26 **and** the pre-existing Raise/Clear repair) are **at-least-once, not exactly-once**: a periodic reconcile can synthesize a transition whose matching live transition is still buffered in the alarm lease, so both broadcast as indistinguishable duplicates on the alarm feed (StreamAlarms + dashboard hub). Pre-existing (the Raise/Clear repair always had it); GWC-26 documented the at-least-once contract rather than closing the race. Closing it needs reconcile/live serialization or a monotonic dedup marker. |
| NEXT-04 | Worker frame writer | Low | WRK-22/WRK-25 cancellation path: a frame `Claimed` by a concurrent lock-holder just before its caller's cancellation races in is never awaited by that caller; if the write then faults, `TrySetException` lands on a `Task` nobody observes (unobserved-task-exception). By-design residual, non-crash (no `UnobservedTaskException` handler registered), pre-existing to single-frame WRK-22 and amplified per-batch by WRK-25. Hygiene fix: attach a fault-observing continuation to abandoned/tombstoned frame completions. |
| NEXT-05 | Worker frame writer | Info | A batch whose remaining frames are tombstoned by cancellation leaves dead `PendingFrame` entries in `_eventFrames`/`_controlFrames` until a future `DequeueNext` pops and skips them. Same pre-existing behavior as single-frame WRK-22, amplified per-batch; in practice heartbeats purge them promptly, so not a real leak. |
| ~~NEXT-01~~ | Testing / macOS | Low | **Resolved 2026-08-10** — the pipe name is now `mxgw-{pid}-{sessionUid}` (session guid hex, worst-case 43 chars), which fits the 104-byte `sun_path` budget under the default macOS `TMPDIR`; the three test-fixture pipe names were shortened the same way, and a `SessionManagerTests` regression pins the format and length budget. All previously failing suites (SessionWorkerClientFactory, e2e fake-worker smoke, WorkerClient, reconnect-replay) pass 33/33 under the default `TMPDIR` — this also retired the separately-remembered "macOS pipe-timeout test failures", which were this throw misread. Docs updated (gateway.md, GatewayProcessDesign, GatewayConfiguration, Sessions, CLAUDE.md). Original finding: Fake-worker/e2e gateway tests fail on macOS under the default `TMPDIR` because the `CoreFxPipe_mxaccess-gateway-{pid}-{sessionId}` path exceeds the 104-char Unix-domain-socket `sun_path` limit under `/var/folders/…/T/`. Workaround today is `TMPDIR=/tmp`. Fix options: shorten the pipe name, or document the `TMPDIR=/tmp` requirement in `docs/GatewayTesting.md`. Surfaced independently by multiple remediation agents. |
| ~~NEXT-02~~ | Clients (.NET, Java) | Low | **Resolved 2026-08-10** — both CLIs now branch on the sentinel and emit the typed cross-CLI row with numeric cursors (Java text mode prints `REPLAY_GAP requested_after=<n> oldest_available=<n>`; .NET emits the `{"replayGap":{…}}` row in jsonl/text and inside the `--json` events array). CrossLanguageSmokeMatrix.md's divergence table collapsed to one converged contract. New CLI regressions in both languages (.NET 35/35, Java 52/52). Original finding: The .NET and Java CLIs render the raw `ReplayGap` sentinel `MxEvent` on `stream-events` instead of a typed gap row — Java text mode prints `0 MX_EVENT_FAMILY_UNSPECIFIED`. Same defect class as CLI-36 (Go) / CLI-35 (Python), which were fixed this cycle; the .NET/Java halves were out of scope. The cross-language smoke matrix now records this divergence honestly. |
| ~~NEXT-03~~ | Gateway alarms | Low | **Resolved 2026-08-10** — best-effort dedup in `GatewayAlarmMonitor`: a buffered live transition whose worker timestamp + resulting state the cache already carries from a repair is suppressed, and reconcile Clear repairs tombstone the instance by `original_raise_timestamp` for one reconcile generation so the buffered live Clear dedups too. Positive-match only (unset timestamps never suppress), so the documented at-least-once consumer contract stands; serialization was rejected as the larger change that still needs a worker-side high-water mark to be correct. Two new race-driving regressions; alarm suites 18/18. Original finding: `GatewayAlarmMonitor.ApplyReconcile` feed-repair broadcasts (the new acked-delta from GWC-26 **and** the pre-existing Raise/Clear repair) are **at-least-once, not exactly-once**: a periodic reconcile can synthesize a transition whose matching live transition is still buffered in the alarm lease, so both broadcast as indistinguishable duplicates on the alarm feed (StreamAlarms + dashboard hub). Pre-existing (the Raise/Clear repair always had it); GWC-26 documented the at-least-once contract rather than closing the race. Closing it needs reconcile/live serialization or a monotonic dedup marker. |
| ~~NEXT-04~~ | Worker frame writer | Low | **Resolved 2026-08-10** — the tombstone helpers now attach a fault-observing continuation to every frame of a cancelled call (a cancelled task never fires `OnlyOnFaulted`, so unconditional attach is safe; covers both the claimed-mid-write frame and the already-faulted-by-`FailAllQueued` frame where `TrySetCanceled` loses). New regression drives the exact abandonment and asserts a marker exception never reaches `TaskScheduler.UnobservedTaskException`. Original finding: WRK-22/WRK-25 cancellation path: a frame `Claimed` by a concurrent lock-holder just before its caller's cancellation races in is never awaited by that caller; if the write then faults, `TrySetException` lands on a `Task` nobody observes (unobserved-task-exception). By-design residual, non-crash (no `UnobservedTaskException` handler registered), pre-existing to single-frame WRK-22 and amplified per-batch by WRK-25. Hygiene fix: attach a fault-observing continuation to abandoned/tombstoned frame completions. |
| ~~NEXT-05~~ | Worker frame writer | Info | **Resolved 2026-08-10 as a documented decision** — the lazy `DequeueNext` purge stays: any subsequent write drains both queues to empty and the heartbeat loop bounds tombstone residency to one interval, while eager `Queue<T>` rebuilds under `_gate` would add ordering-invariant surface next to the WRK-22 interlock for no real gain. Rationale recorded in `docs/WorkerFrameProtocol.md`. Original finding: A batch whose remaining frames are tombstoned by cancellation leaves dead `PendingFrame` entries in `_eventFrames`/`_controlFrames` until a future `DequeueNext` pops and skips them. Same pre-existing behavior as single-frame WRK-22, amplified per-batch; in practice heartbeats purge them promptly, so not a real leak. |
| ~~NEXT-06~~ | Testing / live LDAP | Medium | **Resolved 2026-08-07** — fixtures realigned to the shared directory (`admin`/`password` for the GwAdmin success path, `gw-viewer`/`password` for the bind-succeeds-but-no-role path); verified `Failed: 0, Passed: 5` live against the shared GLAuth at `10.100.0.35:3893`, so the success-path assertion (GwAdmin group claim + Admin role claim) now fails if the service-account credential is wrong. Original finding: `DashboardLdapLiveTests` fixtures have drifted from the shared GLAuth directory, leaving the suite with **no positive-proof coverage of the service-account bind**. Its only success-path test, `AuthenticateAsync_AdminInGwAdminGroup_Succeeds`, binds `admin`/`admin123`, but the directory's `admin` user carries the standard dev password (`scadaproj/infra/glauth/config.toml`), so that assertion cannot pass. `AuthenticateAsync_ReadOnlyUserMissingGwAdminGroup_Fails` binds fixture user `readonly`, which **does not exist** in the GLAuth config at all — it passes for the wrong reason (user-not-found rather than the group-missing branch it names; the `readonly` name is in fact barred by the README's user/group case-collision rule). The three remaining tests are negative assertions that pass whether or not the service account can bind. Net effect: a green `DashboardLdapLiveTests` run proves nothing about the bind credential — surfaced during SEC-36, where the suite was considered as a substitute for the deferred dashboard-login check and rejected. Fix: realign the fixtures to real directory users (e.g. `multi-role`/`gw-viewer`) or add the missing users to the GLAuth config, and add one test that fails when the service-account credential is wrong. |
| ~~NEXT-07~~ | Deployment / windev | High | **Resolved 2026-08-07** — a fresh portable framework-dependent publish of `origin/main` (`a346d51`) was built in a clean clone at `C:\build\mxgw-redeploy`, deployed to `C:\publish\mxaccessgw\Server-20260807`, and the `MxAccessGw` NSSM service repointed at it; the service now holds a stable PID with both `5120`/`5130` listening, a worker spawned, the Galaxy snapshot restored (129 objects / 56,731 attributes) and a clean event log. Root cause confirmed as the version skew this row predicted: the deployed 2026-06-25 build carried `ZB.MOM.WW.Auth.ApiKeys` 0.1.2.0, which supports auth-DB schema 2, against a `gateway-auth.db` stamped at schema 3 on 2026-07-15 by an ephemeral run of newer code — schema 3 is the current shared-lib version (`SqliteAuthSchema.CurrentVersion=3` in Auth 0.1.5), so the redeploy is the forward fix and the DB was left alone. Rollback artifacts kept: `C:\ProgramData\MxGateway\gateway-auth.db.bak-next07` (with `-wal`/`-shm`) and the previous `C:\publish\mxaccessgw\Server` directory. Two side effects worth recording: the old deploy's `appsettings.json` held the LDAP bind password in **plaintext on disk**, while the new one keeps the repo's `${secret:ldap/mxgateway/bind}` token with the NSSM environment supplying the value, so no plaintext LDAP secret remains on that host; and the redeploy tripped the SEC-06 `Ldap:Transport=None` production hard-stop (`GatewayOptionsValidator.cs:178`), resolved by relabelling the host — windev runs `Dashboard:DisableLogin=true`, which this repo's own docs mark dev/test-only, so its `Production` label contradicted its configuration and `DOTNET_ENVIRONMENT` was changed to `Staging` (that one NSSM environment entry only; the other nine preserved byte-identical). SEC-06 is untouched for genuinely production hosts — see NEXT-08 for the posture problem that relabelling defers. Original finding: The `10.100.0.48` (windev) gateway deployment is **stale and crash-looping**, and has been since at least 2026-08-06 (~10k Hosting-failed events/day). The deployed Server binary dates to 2026-06-25 and predates the auth-DB migration of 2026-07-15: it opens a schema-version-3 `gateway-auth.db` that it supports only at version 2 and aborts at startup, so the `MxAccessGw` service never reaches a listening state. Not a code defect in the current tree — a deploy-drift/operations gap — but it means the repo's only deployed host has been dark for over a day and any host-level verification (including SEC-36's dashboard-login check) is blocked until it is repaired. Fix: deploy a current Server build to windev, or restore/downgrade the auth DB to schema 2 if the old binary must stand. Worth asking separately why a service in a permanent restart loop raised no alert. Discovered during SEC-36. |
| NEXT-08 | Security / LDAP posture | Medium | **The shared GLAuth offers no TLS, so SEC-06 makes it undeployable from a `Production`-labelled host.** `GatewayOptionsValidator` (`src/ZB.MOM.WW.MxGateway.Server/Configuration/GatewayOptionsValidator.cs:178`) refuses to start when `Ldap:Transport=None` in the `Production` environment, and `docs/GatewayConfiguration.md`'s `Transport` row states "Deployed hosts must set `Ldaps` or `StartTls`" — but the shared instance at `10.100.0.35:3893` has `[ldaps] enabled=false`, port `3894` closed, and answers StartTLS with `protocolError`, so neither value can work against it. That instruction is currently unsatisfiable for every host that authenticates there. windev sidestepped it on 2026-08-07 by moving to the `Staging` environment name (NEXT-07), which is honest for a dev/test rig but is not available to a real production host. Resolution needs either LDAPS/StartTLS on the shared GLAuth (certificate plus a trust story on each gateway host) or an explicit written posture decision that production gateways bind a different, TLS-capable directory. Surfaced during the NEXT-07 redeploy. |
| NEXT-09 | Build / versioning | Low | **Windows builds stamp git's error text into `InformationalVersion`.** `src/Directory.Build.props:29` runs `git -C "$(MSBuildThisFileDirectory)" …`; MSBuild's directory property ends in a backslash, which escapes the closing quote, so the command is malformed on Windows. The target carries `ContinueOnError`, so the failure is silent and git's stderr is captured as the revision — an observed stamp reads `0.1.2+fatal: cannot change to …`. Any Windows build without a preset `SourceRevisionId` therefore ships a binary that cannot be correlated back to a commit, defeating the point of TST-11. Not reproducible on macOS/Linux, where the separator is `/`. Fix sketch: append `.` to the path or trim the trailing separator before quoting. Surfaced while identifying the deployed binary during NEXT-07. |
| ~~NEXT-09~~ | Build / versioning | Low | **Resolved 2026-08-10** — the Exec path now appends `.` so the trailing backslash can no longer escape the closing quote, and `SourceRevisionId` is additionally gated on a short-SHA regex so no future git failure text can be stamped either. macOS stamp verified unchanged; Windows stamp verified on windev with this batch. Original finding: **Windows builds stamp git's error text into `InformationalVersion`.** `src/Directory.Build.props:29` runs `git -C "$(MSBuildThisFileDirectory)" …`; MSBuild's directory property ends in a backslash, which escapes the closing quote, so the command is malformed on Windows. The target carries `ContinueOnError`, so the failure is silent and git's stderr is captured as the revision — an observed stamp reads `0.1.2+fatal: cannot change to …`. Any Windows build without a preset `SourceRevisionId` therefore ships a binary that cannot be correlated back to a commit, defeating the point of TST-11. Not reproducible on macOS/Linux, where the separator is `/`. Fix sketch: append `.` to the path or trim the trailing separator before quoting. Surfaced while identifying the deployed binary during NEXT-07. |
| NEXT-10 | Docs / glauth | Medium | **`glauth.md`'s "Pre-provisioned users" table contradicts both the directory and the rest of its own file.** It documents `readonly`/`readonly123` and `admin`/`admin123`, neither of which matches `scadaproj/infra/glauth/config.toml` (`readonly` does not exist there; `admin` carries the standard dev password), and lists the `ReadOnly` gid as `5501` against an actual `5601`. Its dashboard section, by contrast, is correct — so the file is internally inconsistent and a reader cannot tell which half to trust. This table was the **root cause of the NEXT-06 fixture drift**, and it has propagated further: `docs/GatewayTesting.md`'s `MXGATEWAY_LIVE_MXACCESS_WRITE_SECURED_PASSWORD` default and the matching literal in `WorkerLiveMxAccessSmokeTests` both take `admin123` from it. Deliberately **not** fixed in the 2026-08-07 pass: the table is entangled with the OPC-UA group taxonomy (gids, role mapping, and the sister-repo consumers of the same directory), so reconciling it means sweeping that taxonomy as one unit rather than patching two rows. |
## Operator actions still pending (from this cycle's runbooks)
+5 -2
View File
@@ -217,7 +217,7 @@ Full design + implementation for each row lives in the linked domain doc under i
| TST-02 | High | P0 | M | TST-04 | Done | Reconnect owner re-validation not implemented |
| TST-03 | High | P1 | M | — | Done | No CI exists |
| TST-04 | High | P2 | L | — | Done | Session-resilience epic 16/28 tasks unfinished |
| TST-05 | Medium | P1 | S | TST-03 | Not started | Real-worker control/COM paths verified opt-in only |
| TST-05 | Medium | P1 | S | TST-03 | Partially done | Real-worker control/COM paths verified opt-in only. **Scheduling half closed 2026-08-10** by the TST-25 `nightly-windev` job (`.gitea/workflows/ci.yml`, cron `0 6 * * *``scripts/ci/run-windev-ci.sh live`, which sets `MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`, runs `WorkerLiveMxAccessSmokeTests`, and opens a Gitea issue on failure) — "opt-in, run by memory" is now "runs nightly, reports failures". **Residual: the coverage-audit half.** The live suite's 8 facts cover all six late-added COM commands but none of the five control commands (`Ping`, `GetSessionState`, `GetWorkerInfo`, `DrainEvents`, `ShutdownWorker`), which real workers answer in `Worker/Ipc/WorkerPipeSession.cs` yet are still only exercised through `FakeWorkerHarness` canned replies — precisely the masking the finding named |
| TST-06 | Medium | — | M | — | Not started | Dashboard live-data path untested |
| TST-07 | Medium | — | S | — | Not started | Real-clock sleeps with negative assertions are latent flakes |
| TST-08 | Medium | P1 | M | — | Done | Full-suite orphaned testhost processes (does not reproduce; doc de-stale) |
@@ -236,7 +236,7 @@ Full design + implementation for each row lives in the linked domain doc under i
| TST-21 | Low | — | S | — | Not started | Log rotation configured but minimal |
| TST-22 | Low | — | S | — | Not started | Config-shape JSON block omits documented keys |
| TST-23 | Low | P2 | S | — | Done | Bidirectional `Session` RPC never built |
| TST-24 | Low | P2 | M | TST-03 | Not started | Client wire behaviour has no automated verification. **Gate cleared:** TST-03 CI is Done (live and green 2026-07-10; Windows/x86 tier green 2026-07-13 via the TST-25/TST-26 SSH-driven windev job), so TST-24 is unblocked — deferred by choice now, not CI-gated |
| TST-24 | Low | P2 | M | TST-03 | Done | Client wire behaviour has no automated verification — closed 2026-08-10. Go/Rust/Java already had real-server wire tests (the finding's premise was stale); the genuine gaps were .NET (transport-interface fake everywhere, no server package) and Python (stub monkeypatch everywhere but one opt-in TLS test). Added `MxGatewayClientWireTests` + `WireFakeGatewayServer` (Kestrel h2c) and `tests/test_wire_fake_gateway.py` (`grpc.aio` loopback), plus a `dotnet test` step for the .NET client in the `portable` CI job. Caught a real bug: Python `connect()` built the `grpc.aio` channel inside `asyncio.to_thread` and failed for every non-stub connection |
## Cross-cutting clusters
@@ -253,6 +253,9 @@ Findings the review flagged as one coordinated design pass — sequence them tog
| Date | Change |
|---|---|
| 2026-08-10 | **TST-25 acceptance Check 6 (forced-failure nightly issue) → Done.** The 2026-07-13 record wrote this check off as "abandoned to shared-runner congestion"; that was wrong on both counts. The 2026-07-13 probe *did* land (issue #125, `[CHECK6 PROBE]`, run 375), and since 2026-07-17 the `nightly-windev` `if: failure()` step has filed an issue on **every** red nightly — #126#139, all authored by the `gitea-actions` bot. Traced run 672 (schedule, main, red) line by line: main step fails → `exitcode '1': failure` → the `if: failure()` step runs → `POST /api/v1/repos/dohertj2/mxaccessgw/issues` with the built-in token masked to `***` → issue #139 created at the matching timestamp. Re-confirmed by a fresh forced-failure probe on the throwaway branch `test/tst25-check6-nightly-issue` (temporary `tst25-check6-probe.yml` reproducing the job shape with `exit 1` for the live step; run 677 → issue #140). Branch deleted, issues #125 and #140 closed with explanatory comments. **One real defect found and fixed** (`fix/tst25-nightly-issue-path`, not merged): `${{ github.server_url }}` is the runner-internal `http://gitea:3000`, so every filed issue's run link was unreachable from a browser. The API call must keep using it (the job container resolves `gitea` only on the docker network and has no LAN egress to the public origin), so the fix adds a `PUBLIC_SERVER_URL: https://gitea.dohertylan.com` job env used **only** for the browser-facing link in the issue body; the probe validated the fixed template (#140 carries a `https://gitea.dohertylan.com/...` link that returns 200). **Separately observed, not fixed:** the nightly has been red continuously since at least 2026-07-17 (run 672: `x86 Worker.Tests failed with exit code 1`, 1 failed / 398 passed / 11 skipped — the known `EventBurst_DrainLoopCoalescesFlushes` class of flake), and the step de-duplicates nothing, so 14 issues are open, seven of them (#132#138) for the identical SHA `47c0b64`. Worth a follow-up: fix the red nightly, and consider having the step reuse an open issue with the same title instead of filing a new one. |
| 2026-08-10 | **TST-24 → `Done`: per-client wire tests land for the two clients that lacked them** (branch `feat/tst-24-client-wire-tests`). Audit first corrected the finding's premise: **Go, Rust, and Java already had real-server wire tests**`newBufconnClient`/`fakeGatewayServer` over `grpc/test/bufconn`, `spawn_fake_gateway` over a loopback `TcpListener` with tonic's `Server`, and `InProcessGateway`/`TestGatewayService` over `InProcessServerBuilder` — each already asserting the round trip, the server-observed `authorization` bearer header, and the `ReplayGap` sentinel. The real gaps were **.NET** (every test substituted `FakeGatewayTransport` for `IMxGatewayClientTransport`, and the test project had no server package) and **Python** (stub monkeypatching everywhere except one opt-in TLS test serving only `OpenSession`). Added `WireFakeGatewayServer` + `MxGatewayClientWireTests` (Kestrel h2c on `127.0.0.1:0` serving `MxAccessGatewayBase`; new `Grpc.AspNetCore.Server` 2.76.0 + `Microsoft.AspNetCore.App` refs on the test project) and `clients/python/tests/test_wire_fake_gateway.py` (`grpc.aio` server on `127.0.0.1:0`, no new deps). Four shapes each: full round trip with every reply field asserted, the bearer header **as received by the server** on the streaming RPC too, the `ReplayGap` sentinel surfaced as the client's typed signal, and a genuine `PERMISSION_DENIED` mapping to the typed authorization error. CI: the `portable` job only *built* the .NET client, so a `dotnet test` step was added. **The new tests immediately caught a shipped bug** — Python `GatewayClient.connect()`/`GalaxyRepositoryClient.connect()` constructed the `grpc.aio` channel inside `asyncio.to_thread`, which raises `RuntimeError: There is no current event loop in thread 'asyncio_0'` because a `grpc.aio` channel binds to the loop current on the constructing thread; every non-stub connection failed, and the one test guarding the off-loop behaviour (Client.Python-028) monkeypatched `create_channel` and so asserted the bug. Fixed by splitting `resolve_channel_security` (blocking TOFU probe, off-loop) from `create_channel` (on-loop), with the `-028` tests retargeted to assert both halves. Verified: .NET 133 passed/1 skipped (pre-existing live-gateway skip), Python 168 passed/1 skipped plus 6/6 opt-in TLS. Docs: `docs/GatewayTesting.md` § Client Wire Tests, `clients/dotnet/README.md`, `clients/python/README.md`. |
| 2026-08-10 | **TST-05 revisited under the restored Windows tier → `Partially done`** (branch `feat/tst-24-client-wire-tests`, doc/tracker-only). The finding's **scheduling** half is closed: cycle-2 TST-25's `nightly-windev` job (cron `0 6 * * *`) runs `scripts/ci/run-windev-ci.sh live``windev-worker-ci.ps1 -Mode live`, which sets `MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`, runs `WorkerLiveMxAccessSmokeTests` on windev after the x86 build/Worker.Tests/full-slnx steps, and files a Gitea issue when red. The **coverage-audit** half is *not* closed, and the audit the design asked for now has a negative answer: the suite's eight `[LiveMxAccessFact]`s cover all six late-added COM commands (`Suspend`, `Activate`, `AuthenticateUser`, `ArchestrAUserToId`, `AddBufferedItem`, `SetBufferedUpdateInterval`) but zero of the five control commands — `MxCommandKind.{Ping,GetSessionState,GetWorkerInfo,DrainEvents,ShutdownWorker}` appear nowhere in `WorkerLiveMxAccessSmokeTests.cs`, so the exact paths the Finding calls masked are still only proven against `FakeWorkerHarness` canned replies while the real implementations live in `Worker/Ipc/WorkerPipeSession.cs`. Residual work (two `[LiveMxAccessFact]`s, windev-only to author and verify) is specified in [60-testing-docs-gaps.md](60-testing-docs-gaps.md#tst-05--real-worker-controlcom-paths-verified-opt-in-only---medium--p1). |
| 2026-07-10 | **TST-15 design fleshed out** (still `Not started` — design only, not implementation): `docs/plans/2026-07-10-dashboard-session-acl-tst15.md`. Resolves the crux the deferral left open — the dashboard is LDAP-identity (Admin/Viewer) while sessions are API-key-owned (`OwnerKeyId`), two disjoint identity domains — via a **session tag** sourced from the owning API key (rides in the existing `ApiKeyConstraints` JSON blob, no SQLite migration). Admin-sees-all; Viewer may `SubscribeSession` iff `session.Tags ∩ viewer.GrantedTags ≠ ∅` (new `Dashboard:GroupToTag` map → hub-token tag claims); untagged sessions Admin-only by default (`Dashboard:UntaggedSessionVisibility`). Includes the enforcement path (`HubTokenPayload.Tags` + `IDashboardSessionAcl` gate at `SubscribeSession`), task breakdown (epic Tasks 1619), test plan incl. live-LDAP, and rejected alternatives (client-supplied tag; group→key-id map). Tracker + `60-testing-docs-gaps.md` TST-15 section point at the doc. **TST-03 investigated:** the CI never ran because the repo had **zero registered Gitea Actions runners** (Actions is enabled; runs are created on push/PR/nightly but fail instantly with nothing to execute them). A Mac runner proved the pipeline executes but cannot clone — this Gitea hands runners the internal `http://gitea:3000` URL, reachable only by a runner co-located on the gitea Docker network. Fix = run a co-located runner on the Gitea host (recipe prepared, `scratchpad/gitea-runner/setup-gitea-host-runner.sh`); pending host access. TST-03 stays `In review`. |
| 2026-07-09 | **P2 Epic wrap — user decision: DEFER TST-15 + TST-24, close the epic.** Epic bucket result: 5 of 7 findings `Done` (CLI-15, CLI-04, CLI-30, TST-01, TST-04); **TST-15** and **TST-24** intentionally deferred to a follow-up (kept `Not started`, not `Won't fix` — they are gated, not rejected). **TST-15** (dashboard EventsHub per-session ACL) is epic Phase 4 — a real feature needing a new session-"tag" mechanism + dashboard group→tag config, not a mechanical fix; the `EventsHub` `TODO(per-session-acl)` stays, and the already-shipped **SEC-25** mitigation (tag *values* redacted from the dashboard mirror by default) means no sensitive payload leaks through the hub today regardless of the missing ACL — so deferring carries no value-leak risk. **TST-24** (per-client wire tests) depends on **TST-03** (CI), which is `In review` (YAML authored, never run on a Gitea runner) — no point wiring client tests into a pipeline that isn't live yet. Net P2: 35/38 `Done`; remaining = TST-15 (deferred feature), TST-24 (deferred, CI-gated), TST-14 (user deletes their own untracked gitignored `*-docs-*.md` files). |
| 2026-07-09 | P2 Epic — **Java client completes CLI-15 + CLI-04 locally** (commit `1cc0fa4`); **CLI-15, CLI-04, CLI-30, TST-01 all → `Done` (5/5 clients + server e2e)**. Java CLI-15: `MxEventStreamItem` record + `MxEventStream.nextItem()` (`isReplayGap()`/`replayGap()`/`event()`); existing `Iterator<MxEvent>` path unchanged, sentinel never swallowed. Java CLI-04: Phase 1 `adviseSupervisory`/`writeSecured`/`writeSecured2`/`authenticateUser`/`archestrAUserToId` + Phase 2 `addBufferedItem`/`setBufferedUpdateInterval`/`suspend`/`activate` (unregister already present) on `MxGatewaySession`, each through `invokeCommand``ensureProtocolSuccess`+`ensureMxAccessSuccess`; credentials scrubbed via `MxGatewaySecrets.redactCredentials` (tests assert absent from message/toString/CLI). `gradle test` 106/0 (58 client + 48 cli), no generated churn. Built locally with `JAVA_HOME=/opt/homebrew/opt/openjdk@17` — Java toolchain now works on the Mac (see prior note). Shared docs `ClientLibrariesDesign.md` + CLAUDE.md updated to "all five clients". **TST-01 → Done** (server e2e `fed0685` + all 5 client `ReplayGap` consumers). This closes session-resilience epic Phase 3 fully. |
@@ -159,6 +159,14 @@ This finding is the umbrella; TST-01/02/15 are its actionable slices. The remedi
## TST-05 — Real-worker control/COM paths verified opt-in only `Medium` · `P1`
> **Status revisit 2026-08-10 (unlocked by TST-25): `Partially done` — one half closed, one half open.**
>
> **Closed — the scheduled cadence.** The `nightly-windev` job in `.gitea/workflows/ci.yml` (cron `0 6 * * *`, gated `if: github.event_name == 'schedule'`) runs `scripts/ci/run-windev-ci.sh live`, which drives `scripts/ci/windev-worker-ci.ps1 -Mode live` on windev: x86 Worker build → full `Worker.Tests` → full-slnx build → `MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1 dotnet test … --filter FullyQualifiedName~WorkerLiveMxAccessSmokeTests`. A red nightly opens a Gitea issue, so nobody has to watch the Actions page. That is exactly this finding's **Design** paragraph, and it is the `live-mxaccess` job the design referred to (renamed; the removed job it originally pointed at is gone — see cycle-2 TST-25/TST-26).
>
> **Open — the coverage audit.** The design also required auditing `WorkerLiveMxAccessSmokeTests.cs` for coverage of *each* of the eleven late-added command kinds and adding missing `[LiveMxAccessFact]` cases. That audit now has an answer, and it is negative for five of the eleven. The suite's eight facts reach all six late-added **COM** commands (`Suspend`, `Activate`, `AuthenticateUser`, `ArchestrAUserToId`, `AddBufferedItem`, `SetBufferedUpdateInterval` — the `NewComCommands_RoundTripWithRealReplies` and `BufferedItem_*` facts). None of them sends any of the five **control** commands: `MxCommandKind.{Ping,GetSessionState,GetWorkerInfo,DrainEvents,ShutdownWorker}` do not appear anywhere in the file. Those are the very kinds the Finding below names as masked. The real worker answers them off-STA in `src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerPipeSession.cs` (dispatch switch at `:574`+), so the nightly exercises that code path only incidentally, never by assertion — a regression in `CreatePingReply`/`CreateSessionStateReply`/`CreateWorkerInfoReply`/the drain snapshot/the shutdown-after-reply ordering still ships green through both CI and the nightly.
>
> **Residual work to close TST-05 fully** (small, Windows-only): add one `[LiveMxAccessFact]` to `WorkerLiveMxAccessSmokeTests` that, against a live worker, invokes `Ping``GetSessionState``GetWorkerInfo``DrainEvents` and asserts each returns a non-`INVALID_REQUEST` reply carrying real worker state (e.g. `worker_process_id` matching the launched process), plus a separate fact for `ShutdownWorker` asserting the OK reply arrives *before* the worker exits and the session is then faulted/closed. `ShutdownWorker` needs `admin` scope and terminates the worker, so it must be the last fact in its own fixture. Not done here because it can only be authored and verified on windev with MXAccess installed; this revisit is doc/tracker-only.
**Finding.** All eleven late-added command kinds are unit-tested against fakes and live-verified once on the dev rig (`stillpending.md` §1.1), but the default suite exercises `Ping`/`GetWorkerInfo`/`DrainEvents`/`ShutdownWorker` only through `FakeWorkerHarness.RespondToControlCommandAsync` (verify current line range in `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/Fakes/FakeWorkerHarness.cs`), which returns canned replies.
**Impact.** A worker-side regression in these paths is invisible until someone sets `MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`.
@@ -449,6 +457,24 @@ If TST-02's interim mitigation (flip retention off) is chosen instead of impleme
## TST-24 — Client wire behaviour has no automated verification `Low` · `—`
> **Resolution 2026-08-10 (branch `feat/tst-24-client-wire-tests`): `Done`.** All five clients now drive their public API against a fake gateway served over a real gRPC transport, in the client's own default suite, and all five run in CI.
>
> **Corrected premise.** The Finding's "no in-process gateway integration tests" was already stale when it was re-verified: **Go, Rust, and Java had real-server wire tests**, not mocks — Go's `newBufconnClient`/`fakeGatewayServer` (`clients/go/mxgateway/client_session_test.go`) over `grpc/test/bufconn`, Rust's `spawn_fake_gateway` (`clients/rust/tests/client_behavior.rs`) over a loopback `TcpListener` with tonic's `Server`, and Java's `InProcessGateway`/`TestGatewayService` (`MxGatewayClientSessionTests.java`) over `InProcessServerBuilder`. Each already asserted the round trip, the `authorization` bearer header as *observed by the server*, and the `ReplayGap` sentinel. The cycle-2 re-verification cited `clients/python/tests/test_replay_gap.py` as evidence for "the other four clients still unit-test against mocks"; that generalized from Python to Go/Rust/Java incorrectly. `InProcessGatewayHarness` (the "template" the Impact paragraph names) is in fact the *thinner* of the Java harnesses — it serves only `streamEvents`/`closeSession` for the CLI tests.
>
> **Real gap, and what was built.** Two clients genuinely had none. **.NET** substituted `FakeGatewayTransport` for `IMxGatewayClientTransport` in every test, so not even the generated stub ran, and its test project had no server package at all. **Python** monkeypatched `MxAccessGatewayStub` everywhere except one opt-in TLS test that served only `OpenSession`. Both now have the pattern:
> - `clients/dotnet/ZB.MOM.WW.MxGateway.Client.Tests/WireFakeGatewayServer.cs` + `MxGatewayClientWireTests.cs` — Kestrel h2c on `127.0.0.1:0` serving `MxAccessGateway.MxAccessGatewayBase`; needed new `Grpc.AspNetCore.Server` + `Microsoft.AspNetCore.App` references on the test project.
> - `clients/python/tests/test_wire_fake_gateway.py` — a `grpc.aio` server on `127.0.0.1:0` serving `MxAccessGatewayServicer`; no new dependencies (`grpcio` is a runtime dep).
>
> Each covers the four shapes the Design asked for: round trip (`OpenSession``Invoke`/`Register``StreamEvents``CloseSession` with every reply field asserted), the bearer header as received by the server on the streaming RPC as well as the unary ones, the `ReplayGap` sentinel surfaced as the client's typed signal (TST-01), and a real `PERMISSION_DENIED` mapping to the typed authorization error.
>
> **CI.** The `portable` job previously only *built* the .NET client; a `dotnet test` step was added, so its wire tests actually run. Go/Rust/Python already ran their suites there and Java in the `java` job.
>
> **Bug this immediately caught** — the justification for the whole finding. `GatewayClient.connect()` / `GalaxyRepositoryClient.connect()` in the Python client were **broken for every real (non-stub) connection**: they built the `grpc.aio` channel inside `asyncio.to_thread`, and a `grpc.aio` channel binds to the event loop current on the constructing thread, so the worker thread raised `RuntimeError: There is no current event loop in thread 'asyncio_0'`. No mock-based test could see it — the one test asserting the off-loop behaviour (`Client.Python-028`) monkeypatched `create_channel` and therefore asserted the bug. Fixed by splitting `resolve_channel_security` (blocking TOFU probe, runs off-loop) from `create_channel` (must run on the loop thread), keeping the Client.Python-028 guarantee; the two `-028` tests were retargeted to assert both halves.
>
> **Deliberately out of scope.** Only the four session RPCs are served — the alarm feed (`StreamAlarms`, `QueryActiveAlarms`, `AcknowledgeAlarm`) and Galaxy browse are not, matching the Design's "full parity is out of scope". Java's `InProcessGatewayHarness` still lacks `openSession`/`invoke`; the client-module tests cover those shapes, so it was left alone.
>
> **Docs.** `docs/GatewayTesting.md` § Client Wire Tests (the cross-client pattern + per-client harness table), `clients/dotnet/README.md`, `clients/python/README.md`.
**Finding.** All five clients have unit tests (13/8/3/13/7 files for dotnet/go/rust/python/java) but no in-process or containerized gateway integration tests; the only cross-language verification is the operator-run `scripts/run-client-e2e-tests.ps1`. `CrossLanguageSmokeMatrixTests` checks shapes only.
**Impact.** Low-to-moderate: a gateway contract change can pass every default suite and break all five clients (partly mitigated by shared-proto codegen). The Java CLI already proves the cheap pattern — `InProcessGatewayHarness` (`stillpending.md` §8).
+14
View File
@@ -23,6 +23,20 @@ dotnet build clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx
dotnet test clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx --no-build
```
Most tests substitute `FakeGatewayTransport` for `IMxGatewayClientTransport`, so
they never touch the wire. `MxGatewayClientWireTests` is the exception: it drives
the ordinary public API against `WireFakeGatewayServer`, a real gRPC server
(Kestrel h2c on an ephemeral loopback port) serving
`MxAccessGateway.MxAccessGatewayBase`. Only the gateway's behaviour is canned —
the HTTP/2 framing, protobuf serialization, `authorization` metadata, and gRPC
status codes are genuine, so it catches decode and metadata breaks a transport
fake cannot see. No MXAccess or worker is involved; it runs in the default suite.
See `docs/GatewayTesting.md` (Client Wire Tests) for the cross-client pattern.
```powershell
dotnet test clients/dotnet/ZB.MOM.WW.MxGateway.Client.Tests/ZB.MOM.WW.MxGateway.Client.Tests.csproj --filter FullyQualifiedName~MxGatewayClientWireTests
```
## Packaging
Create local library and CLI artifacts from the repository root:
@@ -1418,14 +1418,16 @@ public static class MxGatewayClientCli
.WithCancellation(cancellationToken)
.ConfigureAwait(false))
{
if (jsonLines)
{
output.WriteLine(ProtobufJsonFormatter.Format(gatewayEvent));
}
else if (json)
if (json && !jsonLines)
{
events.Add(gatewayEvent);
}
else if (gatewayEvent.ReplayGap is { } replayGap)
{
// Render the ReplayGap sentinel as the typed cross-CLI row instead of the raw
// sentinel MxEvent (NEXT-02, mirroring the Go/Python/Rust CLIs).
output.WriteLine(FormatReplayGapRow(replayGap));
}
else
{
output.WriteLine(ProtobufJsonFormatter.Format(gatewayEvent));
@@ -1835,7 +1837,31 @@ public static class MxGatewayClientCli
private static JsonElement EventToJsonElement(MxEvent gatewayEvent)
{
return JsonDocument.Parse(ProtobufJsonFormatter.Format(gatewayEvent)).RootElement.Clone();
string row = gatewayEvent.ReplayGap is { } replayGap
? FormatReplayGapRow(replayGap)
: ProtobufJsonFormatter.Format(gatewayEvent);
return JsonDocument.Parse(row).RootElement.Clone();
}
/// <summary>
/// Formats the typed ReplayGap row shared by the CLIs (NEXT-02). Hand-built so the
/// cursors are JSON numbers like the Go/Python/Rust rows, not the protobuf JSON
/// formatter's quoted uint64 strings.
/// </summary>
/// <param name="replayGap">Replay gap sentinel payload.</param>
/// <returns>A single-line JSON row describing the gap.</returns>
private static string FormatReplayGapRow(ReplayGap replayGap)
{
return JsonSerializer.Serialize(
new
{
replayGap = new
{
requestedAfterSequence = replayGap.RequestedAfterSequence,
oldestAvailableSequence = replayGap.OldestAvailableSequence,
},
},
JsonOptions);
}
private static MxValue ParseValue(CliArguments arguments)
@@ -1,3 +1,4 @@
using System.Text.Json;
using Google.Protobuf.WellKnownTypes;
using ZB.MOM.WW.MxGateway.Client.Cli;
using ZB.MOM.WW.MxGateway.Contracts.Proto;
@@ -585,6 +586,84 @@ public sealed class MxGatewayClientCliTests
Assert.DoesNotContain("ON_WRITE_COMPLETE", output.ToString());
}
/// <summary>
/// Verifies stream-events renders the ReplayGap sentinel as the typed cross-CLI row —
/// numeric cursors under a replayGap key — instead of the raw sentinel MxEvent (NEXT-02).
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task RunAsync_StreamEvents_RendersReplayGapAsTypedRow()
{
using var output = new StringWriter();
using var error = new StringWriter();
FakeCliClient fakeClient = new();
fakeClient.Events.Add(new MxEvent
{
ReplayGap = new ReplayGap
{
RequestedAfterSequence = 7,
OldestAvailableSequence = 42,
},
});
fakeClient.Events.Add(new MxEvent
{
SessionId = "session-fixture",
Family = MxEventFamily.OnDataChange,
WorkerSequence = 43,
});
int exitCode = await MxGatewayClientCli.RunAsync(
[
"stream-events",
"--endpoint",
"http://localhost:5000",
"--api-key",
"test-api-key",
"--session-id",
"session-fixture",
"--max-events",
"2",
],
output,
error,
_ => fakeClient);
Assert.Equal(0, exitCode);
string[] rows = output.ToString().Split(Environment.NewLine, StringSplitOptions.RemoveEmptyEntries);
Assert.Equal(2, rows.Length);
using JsonDocument gapRow = JsonDocument.Parse(rows[0]);
JsonElement gap = gapRow.RootElement.GetProperty("replayGap");
Assert.Equal(7UL, gap.GetProperty("requestedAfterSequence").GetUInt64());
Assert.Equal(42UL, gap.GetProperty("oldestAvailableSequence").GetUInt64());
Assert.Equal(JsonValueKind.Number, gap.GetProperty("requestedAfterSequence").ValueKind);
Assert.DoesNotContain("MX_EVENT_FAMILY_UNSPECIFIED", rows[0], StringComparison.Ordinal);
Assert.Contains("workerSequence", rows[1], StringComparison.Ordinal);
// The aggregate --json shape carries the same typed row inside the events array.
using var aggregateOutput = new StringWriter();
int aggregateExit = await MxGatewayClientCli.RunAsync(
[
"stream-events",
"--endpoint",
"http://localhost:5000",
"--api-key",
"test-api-key",
"--session-id",
"session-fixture",
"--max-events",
"2",
"--json",
],
aggregateOutput,
error,
_ => fakeClient);
Assert.Equal(0, aggregateExit);
using JsonDocument aggregate = JsonDocument.Parse(aggregateOutput.ToString());
JsonElement firstRow = aggregate.RootElement.GetProperty("events")[0];
Assert.Equal(42UL, firstRow.GetProperty("replayGap").GetProperty("oldestAvailableSequence").GetUInt64());
}
/// <summary>Verifies that stream-alarms with --max-events stops output and distinguishes payload cases.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
@@ -0,0 +1,162 @@
using ZB.MOM.WW.MxGateway.Contracts.Proto;
namespace ZB.MOM.WW.MxGateway.Client.Tests;
/// <summary>
/// Drives the public client API against <see cref="WireFakeGatewayServer"/> — a real
/// gRPC server on loopback — so the transport, protobuf serialization, call metadata,
/// and gRPC status mapping are all exercised. Every other test in this project
/// substitutes <see cref="FakeGatewayTransport"/> and therefore proves nothing about
/// what actually crosses the wire.
/// </summary>
public sealed class MxGatewayClientWireTests
{
private const string ApiKey = "mxgw_wiretest_secret";
/// <summary>
/// Verifies the full session happy path decodes real wire bytes end to end.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task SessionRoundTrip_OverRealTransport_DecodesEveryReplyField()
{
await using WireFakeGatewayServer server = await WireFakeGatewayServer.StartAsync();
await using MxGatewayClient client = server.CreateClient(ApiKey);
MxGatewaySession session = await client.OpenSessionAsync(
new OpenSessionRequest { ClientSessionName = "wire-test" });
Assert.Equal(WireFakeGatewayServer.FakeGatewayService.SessionId, session.SessionId);
Assert.Equal("fake-backend", session.OpenSessionReply.BackendName);
Assert.Equal(1234, session.OpenSessionReply.WorkerProcessId);
Assert.Equal(3u, session.OpenSessionReply.GatewayProtocolVersion);
Assert.Equal(["events", "invoke"], session.OpenSessionReply.Capabilities);
int serverHandle = await session.RegisterAsync("wire-test-client");
Assert.Equal(WireFakeGatewayServer.FakeGatewayService.ServerHandle, serverHandle);
MxCommandRequest? invoke = server.Service.InvokeRequest;
Assert.NotNull(invoke);
Assert.Equal(MxCommandKind.Register, invoke.Command.Kind);
Assert.Equal("wire-test-client", invoke.Command.Register.ClientName);
List<MxEvent> events = await CollectAsync(client.StreamEventsAsync(
new StreamEventsRequest { SessionId = session.SessionId }));
MxEvent single = Assert.Single(events);
Assert.Equal(MxEventFamily.OnDataChange, single.Family);
Assert.Equal(WireFakeGatewayServer.FakeGatewayService.ServerHandle, single.ServerHandle);
Assert.Equal(WireFakeGatewayServer.FakeGatewayService.ItemHandle, single.ItemHandle);
Assert.Equal(17, single.Value.Int32Value);
Assert.Equal(192, single.Quality);
Assert.Equal(9ul, single.WorkerSequence);
Assert.Equal(MxEvent.BodyOneofCase.OnDataChange, single.BodyCase);
CloseSessionReply closeReply = await session.CloseAsync();
Assert.Equal(SessionState.Closed, closeReply.FinalState);
Assert.Equal(
WireFakeGatewayServer.FakeGatewayService.SessionId,
server.Service.CloseSessionRequest?.SessionId);
}
/// <summary>
/// Verifies the API key reaches the server as a bearer header on unary and
/// streaming calls alike. A transport fake can only assert what the client passes;
/// this asserts what the server receives.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task ApiKey_ReachesTheServerAsBearerMetadata_OnEveryRpc()
{
await using WireFakeGatewayServer server = await WireFakeGatewayServer.StartAsync();
await using MxGatewayClient client = server.CreateClient(ApiKey);
MxGatewaySession session = await client.OpenSessionAsync(
new OpenSessionRequest { ClientSessionName = "wire-test" });
await session.RegisterAsync("wire-test-client");
await CollectAsync(client.StreamEventsAsync(
new StreamEventsRequest { SessionId = session.SessionId }));
await session.CloseAsync();
string expected = $"Bearer {ApiKey}";
Assert.Equal(
new Dictionary<string, string>
{
["OpenSession"] = expected,
["Invoke"] = expected,
["StreamEvents"] = expected,
["CloseSession"] = expected,
},
server.Service.AuthorizationByMethod);
}
/// <summary>
/// Verifies the gateway's replay-gap sentinel survives serialization and is
/// surfaced as a typed, non-terminal stream item.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task ReplayGapSentinel_SurvivesTheWire_AsTypedStreamItem()
{
await using WireFakeGatewayServer server = await WireFakeGatewayServer.StartAsync(service =>
service.ReplayGap = new ReplayGap
{
RequestedAfterSequence = 3,
OldestAvailableSequence = 8,
});
await using MxGatewayClient client = server.CreateClient(ApiKey);
MxGatewaySession session = await client.OpenSessionAsync(
new OpenSessionRequest { ClientSessionName = "wire-test" });
List<MxEventStreamItem> items = [];
IAsyncEnumerable<MxEvent> stream = client.StreamEventsAsync(new StreamEventsRequest
{
SessionId = session.SessionId,
AfterWorkerSequence = 3,
});
await foreach (MxEventStreamItem item in stream.AsStreamItemsAsync())
{
items.Add(item);
}
Assert.Equal(2, items.Count);
Assert.True(items[0].IsReplayGap);
Assert.Equal(3ul, items[0].ReplayGap!.RequestedAfterSequence);
Assert.Equal(8ul, items[0].ReplayGap!.OldestAvailableSequence);
Assert.False(items[1].IsReplayGap);
Assert.Equal(MxEventFamily.OnDataChange, items[1].Event.Family);
Assert.Equal(3ul, server.Service.StreamEventsRequest?.AfterWorkerSequence);
}
/// <summary>
/// Verifies a genuine <c>PERMISSION_DENIED</c> status maps to the typed client
/// exception rather than a bare RpcException.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task PermissionDeniedStatus_MapsToAuthorizationException()
{
await using WireFakeGatewayServer server = await WireFakeGatewayServer.StartAsync(
service => service.DenyInvoke = true);
await using MxGatewayClient client = server.CreateClient(ApiKey);
MxGatewaySession session = await client.OpenSessionAsync(
new OpenSessionRequest { ClientSessionName = "wire-test" });
await Assert.ThrowsAsync<MxGatewayAuthorizationException>(
() => session.RegisterAsync("wire-test-client"));
}
private static async Task<List<MxEvent>> CollectAsync(IAsyncEnumerable<MxEvent> stream)
{
List<MxEvent> events = [];
await foreach (MxEvent gatewayEvent in stream)
{
events.Add(gatewayEvent);
}
return events;
}
}
@@ -0,0 +1,264 @@
using System.Collections.Concurrent;
using System.Net;
using Grpc.Core;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Hosting.Server;
using Microsoft.AspNetCore.Hosting.Server.Features;
using Microsoft.AspNetCore.Server.Kestrel.Core;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.MxGateway.Contracts.Proto;
namespace ZB.MOM.WW.MxGateway.Client.Tests;
/// <summary>
/// Hosts the real <c>mxaccess_gateway.v1.MxAccessGateway</c> service on a loopback
/// Kestrel endpoint so client tests exercise genuine HTTP/2 framing, protobuf
/// serialization, call metadata, and gRPC status propagation.
/// </summary>
/// <remarks>
/// <para>
/// This is the counterpart of <see cref="FakeGatewayTransport"/>: that fake replaces
/// <c>IMxGatewayClientTransport</c>, so nothing below the client wrapper runs. This one
/// replaces only the gateway's <em>behaviour</em> — every byte between the client and
/// the service is the real wire format. Contract breaks that a transport fake cannot
/// see (a field the client never decodes, metadata it does not actually send, a status
/// code it maps differently once it arrives as a real <see cref="RpcException"/>) fail
/// here.
/// </para>
/// <para>
/// Plaintext h2c is used deliberately: TLS is covered by
/// <c>MxGatewayClientTlsHandlerTests</c>, and h2c keeps the harness certificate-free so
/// it runs identically on every CI host. See <c>docs/GatewayTesting.md</c>
/// (Client Wire Tests) for the shared pattern and its Python counterpart.
/// </para>
/// </remarks>
internal sealed class WireFakeGatewayServer : IAsyncDisposable
{
private readonly WebApplication _app;
private WireFakeGatewayServer(WebApplication app, FakeGatewayService service, int port)
{
_app = app;
Service = service;
Endpoint = new Uri($"http://127.0.0.1:{port}");
}
/// <summary>
/// Gets the canned service backing the endpoint; tests read its recorded requests.
/// </summary>
public FakeGatewayService Service { get; }
/// <summary>
/// Gets the h2c endpoint to point <see cref="MxGatewayClientOptions.Endpoint"/> at.
/// </summary>
public Uri Endpoint { get; }
/// <summary>
/// Starts a server on an ephemeral loopback port.
/// </summary>
/// <param name="configure">Optional configuration of the canned service.</param>
/// <returns>The started server.</returns>
public static async Task<WireFakeGatewayServer> StartAsync(Action<FakeGatewayService>? configure = null)
{
FakeGatewayService service = new();
configure?.Invoke(service);
WebApplicationBuilder builder = WebApplication.CreateBuilder();
builder.Logging.ClearProviders();
builder.WebHost.ConfigureKestrel(options =>
// Port 0 lets the OS pick; HTTP/2 without TLS (h2c) is what the client's
// plain http:// endpoint negotiates via RequestVersionExact.
options.Listen(IPAddress.Loopback, 0, listen => listen.Protocols = HttpProtocols.Http2));
builder.Services.AddGrpc();
builder.Services.AddSingleton(service);
WebApplication app = builder.Build();
app.MapGrpcService<FakeGatewayService>();
await app.StartAsync().ConfigureAwait(false);
return new WireFakeGatewayServer(app, service, ResolvePort(app));
}
/// <summary>
/// Creates a client bound to this server's endpoint.
/// </summary>
/// <param name="apiKey">API key the client should present.</param>
/// <returns>A client that talks to this server over h2c.</returns>
public MxGatewayClient CreateClient(string apiKey) =>
MxGatewayClient.Create(new MxGatewayClientOptions
{
Endpoint = Endpoint,
ApiKey = apiKey,
UseTls = false,
DefaultCallTimeout = TimeSpan.FromSeconds(30),
});
/// <inheritdoc />
public async ValueTask DisposeAsync()
{
await _app.StopAsync().ConfigureAwait(false);
await _app.DisposeAsync().ConfigureAwait(false);
}
private static int ResolvePort(WebApplication app)
{
IServerAddressesFeature? addresses = app.Services
.GetRequiredService<IServer>()
.Features
.Get<IServerAddressesFeature>();
string address = addresses?.Addresses.FirstOrDefault()
?? throw new InvalidOperationException("Kestrel did not report a bound address.");
return new Uri(address).Port;
}
/// <summary>
/// Canned gateway answering the four session RPCs with gateway-shaped replies.
/// </summary>
internal sealed class FakeGatewayService : MxAccessGateway.MxAccessGatewayBase
{
/// <summary>The session id every reply carries.</summary>
public const string SessionId = "wire-session-1";
/// <summary>The server handle the canned Register reply returns.</summary>
public const int ServerHandle = 4242;
/// <summary>The item handle the canned data-change event carries.</summary>
public const int ItemHandle = 77;
/// <summary>
/// Gets the <c>authorization</c> header value observed per RPC name.
/// </summary>
public ConcurrentDictionary<string, string> AuthorizationByMethod { get; } = new();
/// <summary>
/// Gets or sets a value indicating whether <c>Invoke</c> fails with
/// <see cref="StatusCode.PermissionDenied"/> instead of replying.
/// </summary>
public bool DenyInvoke { get; set; }
/// <summary>
/// Gets or sets the replay-gap sentinel emitted at the head of the event stream.
/// </summary>
public ReplayGap? ReplayGap { get; set; }
/// <summary>
/// Gets the last <c>Invoke</c> request the client sent, as decoded from the wire.
/// </summary>
public MxCommandRequest? InvokeRequest { get; private set; }
/// <summary>
/// Gets the last <c>StreamEvents</c> request the client sent.
/// </summary>
public StreamEventsRequest? StreamEventsRequest { get; private set; }
/// <summary>
/// Gets the last <c>CloseSession</c> request the client sent.
/// </summary>
public CloseSessionRequest? CloseSessionRequest { get; private set; }
/// <inheritdoc />
public override Task<OpenSessionReply> OpenSession(
OpenSessionRequest request,
ServerCallContext context)
{
Record(context);
return Task.FromResult(new OpenSessionReply
{
SessionId = SessionId,
BackendName = "fake-backend",
WorkerProcessId = 1234,
WorkerProtocolVersion = 1,
GatewayProtocolVersion = 3,
Capabilities = { "events", "invoke" },
ProtocolStatus = Ok(),
});
}
/// <inheritdoc />
public override Task<MxCommandReply> Invoke(MxCommandRequest request, ServerCallContext context)
{
Record(context);
InvokeRequest = request;
if (DenyInvoke)
{
throw new RpcException(new Status(StatusCode.PermissionDenied, "invoke scope required"));
}
return Task.FromResult(new MxCommandReply
{
SessionId = request.SessionId,
CorrelationId = request.ClientCorrelationId,
Kind = request.Command.Kind,
ProtocolStatus = Ok(),
Hresult = 0,
Register = new RegisterReply { ServerHandle = ServerHandle },
});
}
/// <inheritdoc />
public override async Task StreamEvents(
StreamEventsRequest request,
IServerStreamWriter<MxEvent> responseStream,
ServerCallContext context)
{
Record(context);
StreamEventsRequest = request;
if (ReplayGap is not null)
{
// The sentinel shape the gateway emits: family unspecified, body unset,
// only replay_gap populated.
await responseStream.WriteAsync(new MxEvent
{
SessionId = request.SessionId,
ReplayGap = ReplayGap,
}).ConfigureAwait(false);
}
await responseStream.WriteAsync(new MxEvent
{
SessionId = request.SessionId,
Family = MxEventFamily.OnDataChange,
ServerHandle = ServerHandle,
ItemHandle = ItemHandle,
Value = new MxValue { Int32Value = 17 },
Quality = 192,
WorkerSequence = 9,
OnDataChange = new OnDataChangeEvent(),
}).ConfigureAwait(false);
}
/// <inheritdoc />
public override Task<CloseSessionReply> CloseSession(
CloseSessionRequest request,
ServerCallContext context)
{
Record(context);
CloseSessionRequest = request;
return Task.FromResult(new CloseSessionReply
{
SessionId = request.SessionId,
FinalState = SessionState.Closed,
ProtocolStatus = Ok(),
});
}
private static ProtocolStatus Ok() => new() { Code = ProtocolStatusCode.Ok };
private void Record(ServerCallContext context)
{
string? authorization = context.RequestHeaders.GetValue("authorization");
if (authorization is not null)
{
// context.Method is the fully-qualified "/package.Service/Method";
// key on the bare method name so assertions stay readable.
AuthorizationByMethod[context.Method[(context.Method.LastIndexOf('/') + 1)..]] =
authorization;
}
}
}
}
@@ -12,6 +12,15 @@
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.14.1" />
<PackageReference Include="xunit" Version="2.9.3" />
<PackageReference Include="xunit.runner.visualstudio" Version="3.1.4" />
<!-- Wire tests only (WireFakeGatewayServer): hosts the real MxAccessGateway service
on loopback Kestrel so the client is driven over genuine HTTP/2 + protobuf rather
than a substituted transport. Version tracks the gateway server's Grpc.AspNetCore
(src/ZB.MOM.WW.MxGateway.Server) and the client's Grpc.Net.Client, both 2.76.0. -->
<PackageReference Include="Grpc.AspNetCore.Server" Version="2.76.0" />
</ItemGroup>
<ItemGroup>
<FrameworkReference Include="Microsoft.AspNetCore.App" />
</ItemGroup>
<ItemGroup>
@@ -5,6 +5,7 @@ import com.zb.mom.ww.mxgateway.client.DeployEventStream;
import com.zb.mom.ww.mxgateway.client.GalaxyRepositoryClient;
import com.zb.mom.ww.mxgateway.client.LazyBrowseNode;
import com.zb.mom.ww.mxgateway.client.MxEventStream;
import com.zb.mom.ww.mxgateway.client.MxEventStreamItem;
import com.zb.mom.ww.mxgateway.client.MxGatewayAlarmFeedSubscription;
import com.zb.mom.ww.mxgateway.client.MxGatewayClient;
import com.zb.mom.ww.mxgateway.client.MxGatewayClientOptions;
@@ -59,6 +60,7 @@ import mxaccess_gateway.v1.MxaccessGateway.MxValue;
import mxaccess_gateway.v1.MxaccessGateway.OnAlarmTransitionEvent;
import mxaccess_gateway.v1.MxaccessGateway.OpenSessionRequest;
import mxaccess_gateway.v1.MxaccessGateway.PingCommand;
import mxaccess_gateway.v1.MxaccessGateway.ReplayGap;
import mxaccess_gateway.v1.MxaccessGateway.StreamAlarmsRequest;
import mxaccess_gateway.v1.MxaccessGateway.SubscribeResult;
import mxaccess_gateway.v1.MxaccessGateway.Write2BulkEntry;
@@ -1654,12 +1656,31 @@ public final class MxGatewayCli implements Callable<Integer> {
MxEventStream events = client.session(sessionId).streamEventsAfter(afterWorkerSequence)) {
int count = 0;
while (events.hasNext()) {
MxEvent event = events.next();
MxEventStreamItem item = events.nextItem();
if (item.isReplayGap()) {
// Render the ReplayGap sentinel as the typed cross-CLI row (NEXT-02,
// mirroring the Go/Python/Rust/.NET CLIs) instead of the raw sentinel
// event, whose text form printed "0 MX_EVENT_FAMILY_UNSPECIFIED".
ReplayGap gap = item.replayGap();
if (json) {
client.out().printf(
"{\"replayGap\":{\"requestedAfterSequence\":%s,\"oldestAvailableSequence\":%s}}%n",
Long.toUnsignedString(gap.getRequestedAfterSequence()),
Long.toUnsignedString(gap.getOldestAvailableSequence()));
} else {
client.out().printf(
"REPLAY_GAP requested_after=%s oldest_available=%s%n",
Long.toUnsignedString(gap.getRequestedAfterSequence()),
Long.toUnsignedString(gap.getOldestAvailableSequence()));
}
} else {
MxEvent event = item.event();
if (json) {
client.out().println(protoJson(event));
} else {
client.out().printf("%d %s%n", event.getWorkerSequence(), event.getFamily());
}
}
count++;
if (limit > 0 && count >= limit) {
events.close();
@@ -43,6 +43,7 @@ import mxaccess_gateway.v1.MxaccessGateway.OpenSessionRequest;
import mxaccess_gateway.v1.MxaccessGateway.ProtocolStatus;
import mxaccess_gateway.v1.MxaccessGateway.ProtocolStatusCode;
import mxaccess_gateway.v1.MxaccessGateway.RegisterReply;
import mxaccess_gateway.v1.MxaccessGateway.ReplayGap;
import mxaccess_gateway.v1.MxaccessGateway.SessionState;
import mxaccess_gateway.v1.MxaccessGateway.StreamAlarmsRequest;
import mxaccess_gateway.v1.MxaccessGateway.SubscribeResult;
@@ -902,6 +903,59 @@ final class MxGatewayCliTests {
}
}
@Test
void streamEventsRendersReplayGapAsTypedRow() {
// NEXT-02: the ReplayGap sentinel must render as the typed cross-CLI
// row (numeric cursors under a replayGap key in --json, a REPLAY_GAP
// line in text mode), never as the raw sentinel event text mode
// used to print "0 MX_EVENT_FAMILY_UNSPECIFIED".
MxEvent gap = MxEvent.newBuilder()
.setReplayGap(ReplayGap.newBuilder()
.setRequestedAfterSequence(7L)
.setOldestAvailableSequence(42L)
.build())
.build();
MxEvent dataChange = MxEvent.newBuilder()
.setFamily(MxEventFamily.MX_EVENT_FAMILY_ON_DATA_CHANGE)
.setSessionId("session-cli")
.setWorkerSequence(43L)
.build();
try (InProcessGatewayHarness harness = new InProcessGatewayHarness()) {
harness.setScriptedEvents(List.of(gap, dataChange));
CliRun jsonRun = execute(
new HarnessClientFactory(harness),
"stream-events",
"--session-id",
"session-cli",
"--json");
assertEquals(0, jsonRun.exitCode(), "errors:\n" + jsonRun.errors());
String jsonOut = jsonRun.output();
assertTrue(
jsonOut.contains(
"{\"replayGap\":{\"requestedAfterSequence\":7,\"oldestAvailableSequence\":42}}"),
jsonOut);
assertTrue(jsonOut.contains("\"family\":\"MX_EVENT_FAMILY_ON_DATA_CHANGE\""), jsonOut);
assertFalse(jsonOut.contains("MX_EVENT_FAMILY_UNSPECIFIED"), jsonOut);
}
try (InProcessGatewayHarness harness = new InProcessGatewayHarness()) {
harness.setScriptedEvents(List.of(gap, dataChange));
CliRun textRun = execute(
new HarnessClientFactory(harness),
"stream-events",
"--session-id",
"session-cli");
assertEquals(0, textRun.exitCode(), "errors:\n" + textRun.errors());
String textOut = textRun.output();
assertTrue(textOut.contains("REPLAY_GAP requested_after=7 oldest_available=42"), textOut);
assertFalse(textOut.contains("MX_EVENT_FAMILY_UNSPECIFIED"), textOut);
assertTrue(textOut.contains("43 MX_EVENT_FAMILY_ON_DATA_CHANGE"), textOut);
}
}
// ---- galaxy-discover / galaxy-watch over the in-process harness (Task 6) ----
@Test
+22
View File
@@ -47,6 +47,19 @@ The tests import the generated gateway and worker stubs, run fake async gateway
stubs, verify API key metadata, exercise stream cancellation, load shared value
and command fixtures, and check deterministic CLI output.
`tests/test_wire_fake_gateway.py` is the one suite that does **not** substitute a
stub: it serves a canned `MxAccessGatewayServicer` from a real `grpc.aio` server
on an ephemeral loopback port and drives the ordinary `GatewayClient` API against
it. Only the gateway's behaviour is canned — the HTTP/2 framing, protobuf
serialization, `authorization` metadata, and gRPC status codes are genuine, so it
catches decode and metadata breaks a stub fake cannot see. No MXAccess, no worker,
no TLS, so it runs in the default suite. See `docs/GatewayTesting.md`
(Client Wire Tests) for the cross-client pattern.
```powershell
python -m pytest tests/test_wire_fake_gateway.py
```
## Packaging
Install the package in editable mode for local development:
@@ -398,6 +411,15 @@ point: the `require_certificate_validation=True` keyword on
`--require-certificate-validation` CLI flag. See
[Gateway Configuration](../../docs/GatewayConfiguration.md#automatic-self-signed-certificate).
Channel construction is split in two: `resolve_channel_security(options)` performs
the blocking part (the trust-on-first-use certificate probe) and
`create_channel(options, security=...)` builds the channel. The async `connect`
classmethods run the first off the event loop and the second on it, because a
`grpc.aio` channel binds to the event loop current on the constructing thread —
building it inside `asyncio.to_thread` raises
`RuntimeError: There is no current event loop in thread 'asyncio_N'`. Callers that
build their own channel should keep `create_channel` on the loop thread.
## CLI
The CLI emits deterministic JSON for automation:
@@ -12,7 +12,7 @@ from .auth import merge_metadata
from .errors import ensure_protocol_success, map_rpc_error
from .generated import mxaccess_gateway_pb2 as pb
from .generated import mxaccess_gateway_pb2_grpc as pb_grpc
from .options import ClientOptions, create_channel
from .options import ClientOptions, create_channel, resolve_channel_security
class GatewayClient:
@@ -58,9 +58,13 @@ class GatewayClient:
if stub is not None:
return cls(options=resolved, stub=stub)
# create_channel may perform a blocking TLS certificate probe (TOFU
# default); run it off the event loop so connect never freezes the loop.
channel = await asyncio.to_thread(create_channel, resolved)
# Resolving security may perform a blocking TLS certificate probe (TOFU
# default); run that off the event loop so connect never freezes it. The
# channel itself must be built on the loop thread — a grpc.aio channel
# binds to the loop current on the constructing thread, and a worker
# thread has none.
security = await asyncio.to_thread(resolve_channel_security, resolved)
channel = create_channel(resolved, security=security)
return cls(
options=resolved,
stub=pb_grpc.MxAccessGatewayStub(channel),
@@ -21,7 +21,12 @@ from .auth import merge_metadata
from .errors import MxGatewayError, map_rpc_error
from .generated import galaxy_repository_pb2 as galaxy_pb
from .generated import galaxy_repository_pb2_grpc as galaxy_pb_grpc
from .options import BrowseChildrenOptions, ClientOptions, create_channel
from .options import (
BrowseChildrenOptions,
ClientOptions,
create_channel,
resolve_channel_security,
)
_DISCOVER_HIERARCHY_PAGE_SIZE = 5000
_BROWSE_CHILDREN_PAGE_SIZE = 500
@@ -70,9 +75,13 @@ class GalaxyRepositoryClient:
if stub is not None:
return cls(options=resolved, stub=stub)
# create_channel may perform a blocking TLS certificate probe (TOFU
# default); run it off the event loop so connect never freezes the loop.
channel = await asyncio.to_thread(create_channel, resolved)
# Resolving security may perform a blocking TLS certificate probe (TOFU
# default); run that off the event loop so connect never freezes it. The
# channel itself must be built on the loop thread — a grpc.aio channel
# binds to the loop current on the constructing thread, and a worker
# thread has none.
security = await asyncio.to_thread(resolve_channel_security, resolved)
channel = create_channel(resolved, security=security)
return cls(
options=resolved,
stub=galaxy_pb_grpc.GalaxyRepositoryStub(channel),
@@ -105,39 +105,50 @@ def _split_authority(endpoint: str) -> tuple[str, int]:
return (host or "localhost", int(port))
def create_channel(options: ClientOptions) -> grpc.aio.Channel:
"""Create a plaintext or TLS `grpc.aio` channel from client options.
@dataclass(frozen=True)
class ChannelSecurity:
"""Transport security resolved for one channel.
The TLS default is lenient: grpc-python has no per-channel skip-verify, so
the server's presented certificate is fetched once (unverified) and pinned
as the channel's only trust root (trust-on-first-use). Set
`require_certificate_validation=True` to force system-trust verification, or
pass `ca_file` to verify against a specific CA both bypass the TOFU path.
`credentials` is `None` for a plaintext channel. `target_name_override` is
the SNI/authority override the TOFU path needs, kept separate from the
caller's explicit `server_name_override` so the caller always wins.
"""
channel_options: list[tuple[str, str | int]] = [
("grpc.max_receive_message_length", options.max_grpc_message_bytes),
("grpc.max_send_message_length", options.max_grpc_message_bytes),
]
if options.server_name_override:
channel_options.append(("grpc.ssl_target_name_override", options.server_name_override))
credentials: grpc.ChannelCredentials | None = None
target_name_override: str | None = None
def resolve_channel_security(options: ClientOptions) -> ChannelSecurity:
"""Resolve transport security for `options`, running any blocking probe.
This is the only blocking part of channel construction: the TOFU path opens
a real TCP+TLS socket to fetch the server's certificate. It is split out of
`create_channel` because a `grpc.aio` channel binds to the event loop
*current on the constructing thread*, so the channel itself must be built on
the loop thread building it inside `asyncio.to_thread` raises
``RuntimeError: There is no current event loop in thread 'asyncio_N'``. The
async `connect` classmethods therefore run this function off the loop and
then call `create_channel` on it.
"""
if options.plaintext:
return grpc.aio.insecure_channel(options.endpoint, options=channel_options)
return ChannelSecurity()
if options.ca_file:
root_certificates = Path(options.ca_file).read_bytes()
credentials = grpc.ssl_channel_credentials(root_certificates=root_certificates)
elif options.require_certificate_validation:
credentials = grpc.ssl_channel_credentials()
else:
return ChannelSecurity(
credentials=grpc.ssl_channel_credentials(root_certificates=root_certificates)
)
if options.require_certificate_validation:
return ChannelSecurity(credentials=grpc.ssl_channel_credentials())
# Lenient default: grpc-python has no per-channel skip-verify, so fetch the
# server's certificate (unverified) and pin it for this channel (TOFU).
# The probe opens a real blocking TCP+TLS socket, so it MUST be bounded —
# a black-holed / firewall-drop host would otherwise hang on the OS default
# connect timeout (minutes). Bound it by call_timeout (or a short fixed
# fallback) so the dial fails fast as a transport error. The async
# `connect` classmethods run this off the event loop (asyncio.to_thread).
# fallback) so the dial fails fast as a transport error.
host, port = _split_authority(options.endpoint)
probe_timeout = options.call_timeout if options.call_timeout else _TOFU_PROBE_TIMEOUT_SECONDS
try:
@@ -146,15 +157,50 @@ def create_channel(options: ClientOptions) -> grpc.aio.Channel:
raise MxGatewayTransportError(
f"failed to fetch TLS certificate from {options.endpoint}: {error}"
) from error
credentials = grpc.ssl_channel_credentials(root_certificates=presented.encode("ascii"))
# The gateway self-signed cert always carries a "localhost" SAN, so default
# the SNI/target-name override to it when none was supplied, tolerating
# dial-by-IP or hostname mismatch.
if not options.server_name_override:
channel_options.append(("grpc.ssl_target_name_override", "localhost"))
return ChannelSecurity(
credentials=grpc.ssl_channel_credentials(root_certificates=presented.encode("ascii")),
target_name_override="localhost",
)
def create_channel(
options: ClientOptions,
*,
security: ChannelSecurity | None = None,
) -> grpc.aio.Channel:
"""Create a plaintext or TLS `grpc.aio` channel from client options.
The TLS default is lenient: grpc-python has no per-channel skip-verify, so
the server's presented certificate is fetched once (unverified) and pinned
as the channel's only trust root (trust-on-first-use). Set
`require_certificate_validation=True` to force system-trust verification, or
pass `ca_file` to verify against a specific CA both bypass the TOFU path.
Pass *security* to reuse a `ChannelSecurity` already resolved off the event
loop by `resolve_channel_security`; omit it and this call resolves (and may
block) inline. Must run on the thread owning the event loop the channel will
be used from.
"""
security = security if security is not None else resolve_channel_security(options)
channel_options: list[tuple[str, str | int]] = [
("grpc.max_receive_message_length", options.max_grpc_message_bytes),
("grpc.max_send_message_length", options.max_grpc_message_bytes),
]
if options.server_name_override:
channel_options.append(("grpc.ssl_target_name_override", options.server_name_override))
elif security.target_name_override:
channel_options.append(("grpc.ssl_target_name_override", security.target_name_override))
if security.credentials is None:
return grpc.aio.insecure_channel(options.endpoint, options=channel_options)
return grpc.aio.secure_channel(
options.endpoint,
credentials,
security.credentials,
options=channel_options,
)
+52 -34
View File
@@ -12,6 +12,7 @@ from zb_mom_ww_mxgateway import client as client_module
from zb_mom_ww_mxgateway import galaxy as galaxy_module
from zb_mom_ww_mxgateway.galaxy import GalaxyRepositoryClient
from zb_mom_ww_mxgateway.generated import mxaccess_gateway_pb2 as pb
from zb_mom_ww_mxgateway.options import ChannelSecurity
@pytest.mark.asyncio
@@ -21,11 +22,12 @@ async def test_gateway_connect_forwards_require_certificate_validation(
"""The connect convenience kwarg must reach ClientOptions (Client.Python-027)."""
captured: dict[str, Any] = {}
def fake_create_channel(options: ClientOptions) -> object:
def fake_resolve(options: ClientOptions) -> ChannelSecurity:
captured["options"] = options
return object()
return ChannelSecurity()
monkeypatch.setattr(client_module, "create_channel", fake_create_channel)
monkeypatch.setattr(client_module, "resolve_channel_security", fake_resolve)
monkeypatch.setattr(client_module, "create_channel", _stub_create_channel)
monkeypatch.setattr(client_module.pb_grpc, "MxAccessGatewayStub", lambda channel: object())
await GatewayClient.connect(
@@ -43,11 +45,12 @@ async def test_galaxy_connect_forwards_require_certificate_validation(
"""GalaxyRepositoryClient.connect must thread the flag too (Client.Python-027)."""
captured: dict[str, Any] = {}
def fake_create_channel(options: ClientOptions) -> object:
def fake_resolve(options: ClientOptions) -> ChannelSecurity:
captured["options"] = options
return object()
return ChannelSecurity()
monkeypatch.setattr(galaxy_module, "create_channel", fake_create_channel)
monkeypatch.setattr(galaxy_module, "resolve_channel_security", fake_resolve)
monkeypatch.setattr(galaxy_module, "create_channel", _stub_create_channel)
monkeypatch.setattr(
galaxy_module.galaxy_pb_grpc, "GalaxyRepositoryStub", lambda channel: object()
)
@@ -61,52 +64,67 @@ async def test_galaxy_connect_forwards_require_certificate_validation(
@pytest.mark.asyncio
async def test_gateway_connect_runs_create_channel_off_the_event_loop(
async def test_gateway_connect_splits_probe_off_loop_and_channel_on_loop(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""connect must run the blocking channel factory off the loop (Client.Python-028)."""
ran_in_thread: dict[str, bool] = {}
"""The blocking probe runs off the loop; the channel is built on it.
def fake_create_channel(options: ClientOptions) -> object:
# If this runs on the event loop thread, get_running_loop() succeeds.
try:
asyncio.get_running_loop()
ran_in_thread["off_loop"] = False
except RuntimeError:
ran_in_thread["off_loop"] = True
return object()
monkeypatch.setattr(client_module, "create_channel", fake_create_channel)
Client.Python-028 required the blocking TOFU probe off the event loop. The
channel itself must nonetheless be constructed *on* the loop thread: a
``grpc.aio`` channel binds to the loop current on the constructing thread,
and a ``to_thread`` worker has none, so building it off-loop raises
``RuntimeError: There is no current event loop``. Assert both halves.
"""
where = _record_connect_threads(monkeypatch, client_module)
monkeypatch.setattr(client_module.pb_grpc, "MxAccessGatewayStub", lambda channel: object())
await GatewayClient.connect(endpoint="gateway.example:5001")
assert ran_in_thread["off_loop"] is True
assert where == {"resolve_off_loop": True, "create_on_loop": True}
@pytest.mark.asyncio
async def test_galaxy_connect_runs_create_channel_off_the_event_loop(
async def test_galaxy_connect_splits_probe_off_loop_and_channel_on_loop(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""GalaxyRepositoryClient.connect must also run the probe off the loop (Client.Python-028)."""
ran_in_thread: dict[str, bool] = {}
def fake_create_channel(options: ClientOptions) -> object:
try:
asyncio.get_running_loop()
ran_in_thread["off_loop"] = False
except RuntimeError:
ran_in_thread["off_loop"] = True
return object()
monkeypatch.setattr(galaxy_module, "create_channel", fake_create_channel)
"""GalaxyRepositoryClient.connect splits the probe and the channel the same way."""
where = _record_connect_threads(monkeypatch, galaxy_module)
monkeypatch.setattr(
galaxy_module.galaxy_pb_grpc, "GalaxyRepositoryStub", lambda channel: object()
)
await GalaxyRepositoryClient.connect(endpoint="gateway.example:5001")
assert ran_in_thread["off_loop"] is True
assert where == {"resolve_off_loop": True, "create_on_loop": True}
def _stub_create_channel(options: ClientOptions, *, security: ChannelSecurity) -> object:
return object()
def _on_event_loop_thread() -> bool:
try:
asyncio.get_running_loop()
except RuntimeError:
return False
return True
def _record_connect_threads(monkeypatch: pytest.MonkeyPatch, module: Any) -> dict[str, bool]:
"""Patch *module*'s channel helpers to record which thread each ran on."""
where: dict[str, bool] = {}
def fake_resolve(options: ClientOptions) -> ChannelSecurity:
where["resolve_off_loop"] = not _on_event_loop_thread()
return ChannelSecurity()
def fake_create_channel(options: ClientOptions, *, security: ChannelSecurity) -> object:
where["create_on_loop"] = _on_event_loop_thread()
return object()
monkeypatch.setattr(module, "resolve_channel_security", fake_resolve)
monkeypatch.setattr(module, "create_channel", fake_create_channel)
return where
@pytest.mark.asyncio
@@ -0,0 +1,288 @@
"""Wire-level tests: the Python client against a real localhost gRPC server.
Every other test in this suite substitutes a fake *stub* object for
``pb_grpc.MxAccessGatewayStub``, so nothing between the client wrapper and the
generated stub is exercised: no HTTP/2 framing, no protobuf serialization, no
call metadata, no gRPC status translation. That leaves a class of contract break
a field the gateway populates but the client never decodes, metadata the
client believes it sends but does not, a status code it maps differently once it
arrives as a real ``grpc.RpcError`` invisible to the default suite.
These tests close that gap by serving the real ``mxaccess_gateway.v1.MxAccessGateway``
service from an in-process ``grpc.aio`` server bound to ``127.0.0.1:0`` and
driving the ordinary public client API against it. The bytes on the wire are the
real ones; only the gateway's *behavior* is canned. No MXAccess, no worker, no
network beyond loopback, so this runs everywhere the normal suite runs.
See ``docs/GatewayTesting.md`` (Client Wire Tests) for the shared pattern and its
counterpart in the .NET client.
"""
from __future__ import annotations
import socket
from collections.abc import AsyncIterator, Awaitable, Callable
import grpc
import pytest
import pytest_asyncio
from zb_mom_ww_mxgateway import ClientOptions, GatewayClient
from zb_mom_ww_mxgateway.errors import MxGatewayAuthorizationError
from zb_mom_ww_mxgateway.events import ReplayGap
from zb_mom_ww_mxgateway.generated import mxaccess_gateway_pb2 as pb
from zb_mom_ww_mxgateway.generated import mxaccess_gateway_pb2_grpc as pb_grpc
API_KEY = "mxgw_wiretest_secret"
SESSION_ID = "wire-session-1"
SERVER_HANDLE = 4242
ITEM_HANDLE = 77
def _free_port() -> int:
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.bind(("127.0.0.1", 0))
return int(sock.getsockname()[1])
def _ok() -> pb.ProtocolStatus:
return pb.ProtocolStatus(code=pb.PROTOCOL_STATUS_CODE_OK)
class FakeGateway(pb_grpc.MxAccessGatewayServicer):
"""Canned gateway serving the four session RPCs over a real transport.
Replies are shaped like the gateway's own: an OK ``ProtocolStatus``, the
echoed session id, and the typed payload the client wrapper reads (for
example ``RegisterReply.server_handle``). Set ``deny`` to make ``Invoke``
abort with ``PERMISSION_DENIED`` so the client's gRPC-status mapping is
exercised against a genuine ``grpc.RpcError`` rather than a hand-built one.
"""
def __init__(self, *, deny: bool = False, replay_gap: pb.ReplayGap | None = None) -> None:
self.deny = deny
self.replay_gap = replay_gap
self.endpoint = ""
self.metadata_by_method: dict[str, str] = {}
self.open_request: pb.OpenSessionRequest | None = None
self.invoke_request: pb.MxCommandRequest | None = None
self.stream_request: pb.StreamEventsRequest | None = None
self.close_request: pb.CloseSessionRequest | None = None
def _record(self, method: str, context: grpc.aio.ServicerContext) -> None:
for key, value in context.invocation_metadata() or ():
if key == "authorization":
self.metadata_by_method[method] = value
async def OpenSession( # noqa: N802 - generated gRPC method name
self, request: pb.OpenSessionRequest, context: grpc.aio.ServicerContext
) -> pb.OpenSessionReply:
"""Answer ``OpenSession`` with a fully populated reply."""
self._record("OpenSession", context)
self.open_request = request
return pb.OpenSessionReply(
session_id=SESSION_ID,
backend_name="fake-backend",
worker_process_id=1234,
worker_protocol_version=1,
capabilities=["events", "invoke"],
gateway_protocol_version=3,
protocol_status=_ok(),
)
async def Invoke( # noqa: N802 - generated gRPC method name
self, request: pb.MxCommandRequest, context: grpc.aio.ServicerContext
) -> pb.MxCommandReply:
"""Answer ``Invoke`` with a Register reply, or deny when configured."""
self._record("Invoke", context)
self.invoke_request = request
if self.deny:
await context.abort(grpc.StatusCode.PERMISSION_DENIED, "invoke scope required")
return pb.MxCommandReply(
session_id=request.session_id,
correlation_id=request.client_correlation_id,
kind=request.command.kind,
protocol_status=_ok(),
hresult=0,
register=pb.RegisterReply(server_handle=SERVER_HANDLE),
)
async def StreamEvents( # noqa: N802 - generated gRPC method name
self, request: pb.StreamEventsRequest, context: grpc.aio.ServicerContext
) -> AsyncIterator[pb.MxEvent]:
"""Stream an optional replay-gap sentinel followed by one data change."""
self._record("StreamEvents", context)
self.stream_request = request
if self.replay_gap is not None:
# The sentinel shape the gateway emits: family unspecified, body
# unset, only replay_gap populated.
yield pb.MxEvent(session_id=request.session_id, replay_gap=self.replay_gap)
yield pb.MxEvent(
session_id=request.session_id,
family=pb.MX_EVENT_FAMILY_ON_DATA_CHANGE,
server_handle=SERVER_HANDLE,
item_handle=ITEM_HANDLE,
value=pb.MxValue(int32_value=17),
quality=192,
worker_sequence=9,
on_data_change=pb.OnDataChangeEvent(),
)
async def CloseSession( # noqa: N802 - generated gRPC method name
self, request: pb.CloseSessionRequest, context: grpc.aio.ServicerContext
) -> pb.CloseSessionReply:
"""Answer ``CloseSession`` with a closed final state."""
self._record("CloseSession", context)
self.close_request = request
return pb.CloseSessionReply(
session_id=request.session_id,
final_state=pb.SESSION_STATE_CLOSED,
protocol_status=_ok(),
)
ServeGateway = Callable[..., Awaitable[FakeGateway]]
@pytest_asyncio.fixture
async def serve_gateway() -> AsyncIterator[ServeGateway]:
"""Yield a factory that serves a :class:`FakeGateway` on loopback.
Each call starts its own server on a free port and records it for teardown,
so a test can serve a differently-configured gateway without a fixture per
variant.
"""
servers: list[grpc.aio.Server] = []
async def _start(**kwargs: object) -> FakeGateway:
fake = FakeGateway(**kwargs) # type: ignore[arg-type]
server = grpc.aio.server()
pb_grpc.add_MxAccessGatewayServicer_to_server(fake, server)
port = _free_port()
server.add_insecure_port(f"127.0.0.1:{port}")
await server.start()
servers.append(server)
fake.endpoint = f"127.0.0.1:{port}"
return fake
try:
yield _start
finally:
for server in servers:
await server.stop(grace=None)
async def _connect(fake: FakeGateway) -> GatewayClient:
return await GatewayClient.connect(
ClientOptions(
endpoint=fake.endpoint,
api_key=API_KEY,
plaintext=True,
call_timeout=10.0,
)
)
@pytest.mark.asyncio
async def test_session_round_trip_decodes_real_wire_bytes(serve_gateway: ServeGateway) -> None:
"""Open, invoke, stream, and close against a real server over loopback."""
wire_gateway = await serve_gateway()
client = await _connect(wire_gateway)
try:
session = await client.open_session(client_session_name="wire-test")
assert session.session_id == SESSION_ID
assert session.open_reply.backend_name == "fake-backend"
assert list(session.open_reply.capabilities) == ["events", "invoke"]
server_handle = await session.register("wire-test-client")
assert server_handle == SERVER_HANDLE
assert wire_gateway.invoke_request is not None
assert wire_gateway.invoke_request.command.kind == pb.MX_COMMAND_KIND_REGISTER
assert wire_gateway.invoke_request.command.register.client_name == "wire-test-client"
events = [event async for event in session.stream_events()]
assert len(events) == 1
event = events[0]
assert not isinstance(event, ReplayGap)
assert event.family == pb.MX_EVENT_FAMILY_ON_DATA_CHANGE
assert event.server_handle == SERVER_HANDLE
assert event.item_handle == ITEM_HANDLE
assert event.value.int32_value == 17
assert event.quality == 192
assert event.worker_sequence == 9
assert event.HasField("on_data_change")
close_reply = await session.close()
assert close_reply.final_state == pb.SESSION_STATE_CLOSED
assert wire_gateway.close_request is not None
assert wire_gateway.close_request.session_id == SESSION_ID
finally:
await client.close()
@pytest.mark.asyncio
async def test_api_key_reaches_the_server_on_every_rpc(serve_gateway: ServeGateway) -> None:
"""The bearer header is on the wire for unary and streaming calls alike.
Stub-substituting tests can only assert what the client *passes*; this
asserts what the server *receives*, which is the property that matters.
"""
wire_gateway = await serve_gateway()
client = await _connect(wire_gateway)
try:
session = await client.open_session(client_session_name="wire-test")
await session.register("wire-test-client")
async for _ in session.stream_events():
break
await session.close()
finally:
await client.close()
expected = f"Bearer {API_KEY}"
assert wire_gateway.metadata_by_method == {
"OpenSession": expected,
"Invoke": expected,
"StreamEvents": expected,
"CloseSession": expected,
}
@pytest.mark.asyncio
async def test_replay_gap_sentinel_survives_the_wire(serve_gateway: ServeGateway) -> None:
"""A resumed stream surfaces the gateway's sentinel as a typed ``ReplayGap``."""
replay_gap_gateway = await serve_gateway(
replay_gap=pb.ReplayGap(requested_after_sequence=3, oldest_available_sequence=8)
)
client = await _connect(replay_gap_gateway)
try:
session = await client.open_session(client_session_name="wire-test")
items = [item async for item in session.stream_events(after_worker_sequence=3)]
finally:
await client.close()
assert len(items) == 2
gap = items[0]
assert isinstance(gap, ReplayGap)
assert gap.requested_after_sequence == 3
assert gap.oldest_available_sequence == 8
assert gap.resume_after_worker_sequence == 7
assert not isinstance(items[1], ReplayGap)
assert items[1].family == pb.MX_EVENT_FAMILY_ON_DATA_CHANGE
assert replay_gap_gateway.stream_request is not None
assert replay_gap_gateway.stream_request.after_worker_sequence == 3
@pytest.mark.asyncio
async def test_permission_denied_maps_to_authorization_error(
serve_gateway: ServeGateway,
) -> None:
"""A real ``PERMISSION_DENIED`` status becomes the typed client error."""
denying_gateway = await serve_gateway(deny=True)
client = await _connect(denying_gateway)
try:
session = await client.open_session(client_session_name="wire-test")
with pytest.raises(MxGatewayAuthorizationError):
await session.register("wire-test-client")
finally:
await client.close()
+11 -14
View File
@@ -40,27 +40,24 @@ reports the next deliverable sequence rather than `0` (see [Sessions](Sessions.m
The default smoke sequence opens a fresh stream (no cursor) and does not exercise
the gap path; a resume-with-gap fixture case is tracked separately (TST-24).
The CLIs differ in how they *print* that library-level signal. Three of them consume
the typed gap and emit a dedicated row rather than a degenerate event row; the other
two hand the raw sentinel `MxEvent` straight to the formatter, so they print the
sentinel itself, whose `replayGap` field carries the same cursors:
All five CLIs consume the typed gap and emit a dedicated row rather than a
degenerate event row (the .NET and Java halves were the last to convert — NEXT-02):
| CLI | Text mode | JSON mode |
|-----|-----------|-----------|
| `mxgw-rs` (Rust, canonical) | `REPLAY_GAP requested_after=<n> oldest_available=<n>` | `{"replayGap": {"requestedAfterSequence": <n>, "oldestAvailableSequence": <n>}}` as one entry of the `events` array |
| `mxgw-go` (Go) | `REPLAY_GAP requested_after=<n> oldest_available=<n>` | one `{"replayGap": {"requestedAfterSequence": <n>, "oldestAvailableSequence": <n>}}` line, counted toward `-limit` like any other row |
| `mxgw-py` (Python) | same JSON dump as `--json` | `{"replayGap": {"requestedAfterSequence": <n>, "oldestAvailableSequence": <n>}}` as one entry of the `events` array |
| `mxgw-dotnet` (.NET) | the raw sentinel `MxEvent` as protobuf JSON, including its `replayGap` field | same, as one entry of the `events` array |
| `mxgw-java` (Java) | the sentinel's `worker_sequence` and `family` (`0 MX_EVENT_FAMILY_UNSPECIFIED`) | the raw sentinel `MxEvent` as protobuf JSON, including its `replayGap` field |
| `mxgw-dotnet` (.NET) | one `{"replayGap": {...}}` line (its "text" mode is JSON-per-line) | the same row — per line with `--jsonl`, as one entry of the `events` array with `--json` |
| `mxgw-java` (Java) | `REPLAY_GAP requested_after=<n> oldest_available=<n>` | one `{"replayGap": {"requestedAfterSequence": <n>, "oldestAvailableSequence": <n>}}` line |
Rust, Go, and Python emit the same two key names and, deliberately, the same JSON
value **types**: the cursors are JSON numbers (`7`), not strings. That is why the
Go CLI types the row by hand instead of marshalling `ReplayGap` with `protojson`
the proto3 JSON mapping renders 64-bit integers as strings (`"7"`), which is also
why the .NET and Java rows, which pass the sentinel through a protobuf JSON
formatter, carry **quoted** cursors. A matrix runner must therefore compare parsed
values, not raw bytes, and must not assume the same value type across all five
CLIs.
All five emit the same two key names and, deliberately, the same JSON value
**types**: the cursors are JSON numbers (`7`), not strings. That is why every
CLI types the row by hand instead of marshalling `ReplayGap` through its
protobuf JSON formatter — the proto3 JSON mapping renders 64-bit integers as
strings (`"7"`). Normal event rows still come from the protobuf formatters, so
a matrix runner must still compare parsed values, not raw bytes, when it mixes
gap rows with event rows.
Two further formatting differences among the three canonical CLIs, none of them
semantic: Python sorts object keys and uses `", "` / `": "` separators
+1 -1
View File
@@ -663,7 +663,7 @@ See each client README for the as-built behavior.
Transport security here applies only to the public gRPC channel. The
gateway↔worker link is a per-session **named pipe**
(`mxaccess-gateway-{gatewayPid}-{sessionId}`), not a network socket. It is not
(`mxgw-{gatewayPid}-{sessionUid}`), not a network socket. It is not
TLS-encrypted and does not need to be: it never leaves the local Windows host and
is secured by the OS pipe ACL. See [Worker Frame Protocol](./WorkerFrameProtocol.md).
+6 -1
View File
@@ -418,9 +418,14 @@ The gateway creates the pipe server before launching the worker.
Pipe name:
```text
mxaccess-gateway-{gatewayProcessId}-{sessionId}
mxgw-{gatewayProcessId}-{sessionUid}
```
`sessionUid` is the session id's guid hex without the `session-` prefix. The
short form keeps the Unix-domain-socket path .NET uses for named pipes on
macOS/Linux (`$TMPDIR/CoreFxPipe_{name}`) inside the 104-byte macOS `sun_path`
limit under the default per-user `TMPDIR`.
Message framing:
```text
+173 -1
View File
@@ -82,7 +82,17 @@ fake-worker tests cannot validate:
when the rig does not drive sample-bearing buffered batches on demand.
All eight tests are gated by the same `MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`
opt-in variable.
opt-in variable. Opt-in does not mean unscheduled: the `nightly-windev` job runs
this suite on windev every night at 06:00 UTC and files a Gitea issue when it goes
red (see [Continuous Integration](#continuous-integration)), so the smoke no longer
depends on someone remembering to set the variable.
Known coverage gap: the suite reaches all six late-added MXAccess **COM** commands
but none of the five **control** commands (`Ping`, `GetSessionState`,
`GetWorkerInfo`, `DrainEvents`, `ShutdownWorker`). Those are implemented off-STA in
`Worker/Ipc/WorkerPipeSession.cs` and are asserted only against
`FakeWorkerHarness`'s canned replies, so no test proves the *real* worker answers
them. Closing that is the residual half of archreview TST-05.
Build the worker before running the smoke:
@@ -267,6 +277,49 @@ $env:MxGateway__Ldap__ServiceAccountPassword = "<service-account-password>"
dotnet test src/ZB.MOM.WW.MxGateway.IntegrationTests/ZB.MOM.WW.MxGateway.IntegrationTests.csproj --filter FullyQualifiedName~DashboardLdapLiveTests
```
## Client Wire Tests
Each client's own suite drives the client's public API against a **fake gateway
served over a real gRPC transport** — an in-process or loopback server
implementing `mxaccess_gateway.v1.MxAccessGateway`. Only the gateway's *behaviour*
is canned; the HTTP/2 framing, protobuf serialization, call metadata, and gRPC
status codes are genuine. That is the difference from the per-client mocks: a mock
substituted for the generated stub (or, in .NET, for `IMxGatewayClientTransport`)
proves what the client *intends* to send, never what a server *receives*, so a
field the client fails to decode or a header it never actually attaches passes
every mock-based test. These tests need no MXAccess, no worker, and no network
beyond loopback, so they run in the default suite on every host.
The shared shape each client's wire test covers:
- **Round trip**`OpenSession``Invoke` (a `Register`, asserting the decoded
`RegisterReply.server_handle`) → `StreamEvents` (asserting the decoded
`OnDataChange` fields) → `CloseSession`.
- **Auth on the wire** — the `authorization: Bearer <key>` header is asserted as
*observed by the server*, on the streaming RPC as well as the unary ones.
- **Replay-gap sentinel** — a stream resumed with `after_worker_sequence` opens
with the gateway's `replay_gap` sentinel, and the client surfaces it as its
typed, non-terminal replay-gap signal rather than a normal event.
- **Status mapping** — a real `PERMISSION_DENIED` from the server becomes the
client's typed authorization error, not a bare transport exception.
Per-client harness and command:
| Client | Harness | Command |
|---|---|---|
| .NET | `WireFakeGatewayServer` (Kestrel h2c on `127.0.0.1:0`, `MxAccessGatewayBase`) — `MxGatewayClientWireTests` | `dotnet test clients/dotnet/ZB.MOM.WW.MxGateway.Client.Tests/ZB.MOM.WW.MxGateway.Client.Tests.csproj` |
| Python | `FakeGateway` + `serve_gateway` fixture (`grpc.aio` server on `127.0.0.1:0`) — `tests/test_wire_fake_gateway.py` | `python -m pytest` from `clients/python` |
| Go | `fakeGatewayServer` + `newBufconnClient` (`grpc.NewServer` over `bufconn`) — `mxgateway/client_session_test.go` | `go test ./...` from `clients/go` |
| Rust | `spawn_fake_gateway` (tonic `Server` over a loopback `TcpListener`) — `tests/client_behavior.rs` | `cargo test --workspace` from `clients/rust` |
| Java | `TestGatewayService` + `InProcessGateway` (`InProcessServerBuilder`) — `MxGatewayClientSessionTests`; plus `InProcessGatewayHarness` for the CLI tests | `gradle test` from `clients/java` |
All five run in CI: Go, Rust, Python, and the .NET client tests in the `portable`
job, Java in the `java` job.
Adding an RPC to `mxaccess_gateway.proto` does not automatically extend these —
the fake gateways implement only the four session RPCs. Extend the fake in the
client whose behaviour changed rather than adding a parallel harness.
## Client E2E Scripts
`scripts/discover-testmachine-tags.ps1` queries the ZB Galaxy Repository for the
@@ -432,6 +485,125 @@ Run the gateway test project after shared gateway test infrastructure changes:
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj
```
## Running the Gateway Suite on windev
The gateway suite (`ZB.MOM.WW.MxGateway.Tests`, net10.0/x64) is not part of the CI
Windows tier — `windows-x86` and `nightly-windev` run only the x86 Worker build and
`Worker.Tests`. It is still run on windev by hand when a change needs Windows
confirmation, and that run has three Windows-specific characteristics worth knowing
before results are interpreted.
Run it from an isolated clone under `C:\build` checked out to the SHA under test — never
the dirty Desktop checkout, and never the CI clone `C:\build\mxaccessgw-ci`, whose worktree
lock belongs to the Worker tier.
Baseline on an otherwise idle windev (2026-08-10): **879 passed, 0 failed, 31 s** — the same
879 the macOS box runs, with nothing gated away. Any failure is therefore a real signal, but
read the load caveat below before acting on one.
Runs before the pipe-buffer fix below reported 855, which was long read as "windev runs a
smaller suite because some cases are gated to Unix". It was not: 855 is simply what had been
flushed when the wedged host was torn down. Do not treat a short count on this suite as
platform gating.
### Two long-standing "windev-environmental" failures were test bugs, not the environment
Both were dismissed as environmental for months and are now fixed. Neither depended on
anything installed on windev; both failed on **any** Windows host:
- `SelfSignedCertificateProviderTests.GenerateCertificate_HasExpectedSansEkuAndValidity`
asserted SAN content by substring-matching `X509Extension.Format(false)`. That string is
produced by the platform crypto library: Windows' `CryptFormatObject` renders the IPv6
loopback fully expanded (`IP Address=0000:0000:0000:0000:0000:0000:0000:0001`) while the
managed formatter used on macOS/Linux renders `::1`, so the loopback assertion failed on
Windows only. The test now decodes the extension with `X509SubjectAlternativeNameExtension`
and compares parsed `IPAddress` values and DNS names (case-insensitively, as DNS names
are), which is platform-independent.
- `SessionManagerTests.OpenSessionAsync_PipeNameIsShortAndUniquePerPidAndSession` guards the
104-byte macOS `sun_path` budget that NEXT-01 shortened the pipe name to fit. It padded the
measured name up to a five-digit pid but never substituted that worst case *downward*, so a
six-digit pid — routine on Windows, impossible on macOS, where pids stop at 99999 — made the
name one character "too long" against a budget that does not apply to the host running the
test. The check now replaces the running pid's digit count with the five-digit macOS worst
case, so it measures the name *format* rather than the current process's pid.
### The real-pipe suites are load-sensitive
These suites drive real named pipes against a five-second worker startup timeout and start
failing when windev is busy — most often when the x86 Worker tier is building or testing at
the same time. All five passed in the idle baseline above and all five failed in a run taken
while an x86 build and `Worker.Tests` were in flight (that run also took 2 m 21 s against the
idle half-minute):
- `GatewayEndToEndFakeWorkerSmokeTests`, `GatewayEndToEndMultiSubscriberTests`,
`GatewayEndToEndReconnectReplayTests` — fail as
`RpcException Status(StatusCode="Unavailable", Detail="Failed to open session …")`.
- `SessionWorkerClientFactoryFakeWorkerTests.CreateAsync_WhenFakeWorkerStartupFails_ThrowsWorkerClientException`
— the startup timeout beats the protocol violation the test is asserting, so the observed
exception is `TimeoutException` instead of `WorkerClientException`.
- `WorkerClientTests.InvokeAsync_WhenCommandExceedsFrameMax_FailsOnlyThatCommandAndStaysReady`.
- `EventStreamServiceTests.StreamEventsAsync_WithConcurrentStreams_TracksAggregateQueueDepth`
— polls a metric against a five-second deadline. Its helper now reports the unmet condition
rather than letting a bare `TaskCanceledException` escape, so a load-induced timeout here
names what it was waiting for instead of looking like an unexplained cancellation.
A failure in that list is evidence about machine load, not about the change under test. Check
for a concurrent x86 build/test (`Get-Process dotnet, testhost, testhost.net48.x86,
MSBuild, VBCSCompiler`) and re-run the affected class on its own before treating it as real.
windev has 36 logical CPUs and `xunit.runner.json` sets `maxParallelThreads: -1`, so the
suite runs far wider there than on the macOS dev box — that width is what turns these
real-clock deadlines into failures.
### The full-suite testhost hang was a zero-buffer named pipe (fixed)
For months a full-suite run on windev reported `855 passed, 0 failed` and then never
returned: the x64 `testhost` stopped consuming CPU but stayed alive indefinitely, and the run
had to be killed with `--blame-hang`. That guard is no longer needed — run the suite plainly:
```powershell
dotnet test src\ZB.MOM.WW.MxGateway.Tests\ZB.MOM.WW.MxGateway.Tests.csproj
```
The cause is worth recording because the shape of it is easy to hit again.
`dotnet-stack report` on the wedged host showed no thread running test code; xUnit's
`RunTestsInAssembly` was simply parked on `WaitHandle.WaitOne()` waiting for the
assembly-finished event. The wait was therefore in a suspended async state machine, which only
`dotnet-dump analyze <dump> -c dumpasync` can see. It named the exact frame:
`WorkerClientTests.StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout`
awaiting `WorkerFrameWriter.WriteAsync` — a 63-byte pipe write that never completed. The `855`
was never the whole suite: the same clone now reports 879, so the wedge was also costing 24
results, and the summary still looked clean because the hung test is not counted as a failure.
That test pushes events past the worker client's staging bound to prove the client faults, and
after the fault the client's read loop stops reading by design. The test-side pipe was created
through `NamedPipeServerStream(string, PipeDirection, int, PipeTransmissionMode, PipeOptions)`,
whose omitted buffer arguments become `inBufferSize: 0` / `outBufferSize: 0`. On Windows that
reserves *no* buffer: a write completes only when the peer reads it. Measured directly on
windev, that pipe absorbed **0 bytes** before blocking against a non-reading peer, while the
same pipe declared with 64 KiB buffers absorbed **65 520**. On macOS and Linux .NET backs named
pipes with Unix domain sockets, whose socket buffer swallows a few kilobytes regardless — which
is why the identical test never hung there, and why the bug read as "a windev thing".
Two changes make it structural rather than incidental:
- Test-owned server pipes are created through `TestSupport/TestNamedPipe.CreateServer`, which
declares explicit 64 KiB buffers, in both the gateway and worker test projects. This scopes
those tests to the backpressure they are actually asserting — the gateway's staging and event
queues — instead of the OS pipe's flow control.
- Every fake-worker write in `WorkerClientTests` goes through `PipePair.WriteAsync`, which
bounds the write by the class's five-second `TestTimeout` and fails with a message naming the
stopped reader. A blocked write is now a named test failure rather than a silent wedge.
The severity came from the second point being missing, not the first. A test method that never
returns keeps xUnit from raising `ITestAssemblyFinished`, so the runner waits forever and
`testhost` never exits — one unbounded `await` in one test costs the entire suite its result.
Any new test that writes to a pipe whose reader may stop must bound the write.
The gateway's production pipe in `SessionWorkerClientFactory.CreatePipe` deliberately keeps the
unbuffered declaration: both ends run continuous read loops and every write there is bounded by
the worker client's `_stopCts`, so a stalled peer cancels the write instead of blocking on it.
## Continuous Integration
CI runs on Gitea Actions (`.gitea/workflows/ci.yml`; origin is Gitea at
+2 -2
View File
@@ -14,7 +14,7 @@ All four interfaces (`ISessionManager`, `ISessionRegistry`, `ISessionWorkerClien
`GatewaySession` is a sealed class that holds the identity, configured timeouts, worker client reference, and current `SessionState` for one session. State is protected by a private `_syncRoot` lock so that property reads and transitions are observed atomically by concurrent gRPC calls and the lease sweeper.
The session id is an opaque string in the form `session-{guid:N}` and the per-session pipe name is `mxaccess-gateway-{ProcessId}-{SessionId}`. Encoding the gateway PID into the pipe name avoids collisions when an old gateway process leaks pipes that the OS has not yet reclaimed.
The session id is an opaque string in the form `session-{guid:N}` and the per-session pipe name is `mxgw-{ProcessId}-{guid:N}` (the same guid hex, without the `session-` prefix). Encoding the gateway PID into the pipe name avoids collisions when an old gateway process leaks pipes that the OS has not yet reclaimed. The name is kept short because .NET named pipes on Unix-like hosts are Unix domain sockets at `$TMPDIR/CoreFxPipe_{name}`, and macOS caps that path at 104 bytes while its default per-user `TMPDIR` is already ~49 — the old `mxaccess-gateway-{pid}-{sessionId}` form overflowed it and broke the fake-worker/e2e tests on macOS.
`SessionState` itself is the protobuf-generated enum from `ZB.MOM.WW.MxGateway.Contracts.Proto`, so it is shared between the gateway and clients on the wire.
@@ -201,7 +201,7 @@ The single worker event channel has exactly one direct reader: the `SessionEvent
The monitor takes that lease **before** it issues `SubscribeAlarms`, so no transition window is missed: the pump has been running since `MarkReady` started the dashboard mirror, and the distributor only fans to subscribers registered at fan-out time, so a lease taken after the subscribe + first-reconcile round trips would lose every transition raised inside that window. Transitions arriving while the monitor subscribes and reconciles buffer in the lease's bounded channel and are applied after the reconcile, which is order-safe because a live transition can update an alarm the reconciled snapshot already holds.
The repair transitions the monitor's reconcile broadcasts on the alarm feed (Raise/Clear presence deltas and the acked-state delta) are **at-least-once**: a reconcile reads the worker's current state while the matching live transition may still be buffered in the lease, so both can be broadcast and the duplicates are indistinguishable — alarm-feed consumers must apply transitions idempotently. This is a property of the reconcile design, not of the buffering above.
The repair transitions the monitor's reconcile broadcasts on the alarm feed (Raise/Clear presence deltas and the acked-state delta) are **at-least-once**: a reconcile reads the worker's current state while the matching live transition may still be buffered in the lease, so both can be broadcast and the duplicates are indistinguishable — alarm-feed consumers must apply transitions idempotently. This is a property of the reconcile design, not of the buffering above. The monitor dedups the common case best-effort (NEXT-03): a buffered live transition that positively matches the cache's worker timestamp and resulting state — or, for Clear, a tombstone keyed on the cleared instance's original raise timestamp — was already broadcast as a repair and is suppressed; unset timestamps never suppress, so the consumer contract is unchanged.
`AttachInternalEventSubscriber` enforces the same readiness gate as `AttachEventSubscriber` — a session (or worker) that is not `Ready` throws `SessionNotReady` *before* the distributor is constructed. A premature attach would otherwise start the pump against a source that throws, completing every subscriber with that error and latching the distributor for the session's whole lifetime.
+10
View File
@@ -37,6 +37,16 @@ $env:Path = [Environment]::GetEnvironmentVariable('Path','Machine') + ';' + [Env
| C compiler x86 | 14.44.35207 | `C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\bin\Hostx64\x86\cl.exe` |
| Linker x86 | 14.44.35207 | `C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\bin\Hostx64\x86\link.exe` |
| LibMan CLI | 3.0.71 | `C:\Users\dohertj2\.dotnet\tools\libman.exe` |
| dotnet-stack | 9.0.661903 | `C:\Users\dohertj2\.dotnet\tools\dotnet-stack.exe` |
| dotnet-dump | 9.0.661903 | `C:\Users\dohertj2\.dotnet\tools\dotnet-dump.exe` |
`dotnet-stack` and `dotnet-dump` are the diagnostics pair for a process that stops
making progress but does not exit. `dotnet-stack report -p <pid>` prints every
managed thread's stack, which is enough when a *thread* is blocked; when nothing is
on a thread the wait lives in a suspended async state machine, and only
`dotnet-dump collect -p <pid>` followed by `dotnet-dump analyze <dump> -c dumpasync`
reveals it. Both were installed user-local with `dotnet tool install -g` while
root-causing the windev test-host hang described in `docs/GatewayTesting.md`.
Reference assemblies:
+18 -1
View File
@@ -123,7 +123,11 @@ runs after the whole batch, and only then does every successfully-written
frame's completion resolve — so a caller's `WriteAsync` still does not
complete until its bytes are both written *and* flushed, but a batch that
happened to contain several queued frames pays one flush instead of one per
frame. The event drain loop (`WorkerPipeSession.RunEventDrainLoopAsync`)
frame. Note the ordering this implies at the peer: the frames reach the pipe
before the flush that follows them, so the gateway can read a whole batch
while the writer has not yet flushed it. Anything observing the flush itself
(a test counting flushes, for instance) must wait for the flush, not infer it
from frames arriving. The event drain loop (`WorkerPipeSession.RunEventDrainLoopAsync`)
submits a whole drained event batch through `WriteBatchAsync`, which enqueues
every frame under one `_gate` acquisition, takes the write lock once, and
drains them together, so a burst of N events costs one flush rather than N —
@@ -147,6 +151,19 @@ so the caller observes `OperationCanceledException` while that one frame still
reaches the wire. That residual window is by design: blocking the canceller
behind the very write it is abandoning would defeat the point of cancellation.
Two hygiene notes on that residual (NEXT-04/NEXT-05). First, a frame the
cancelled caller abandons — claimed mid-write, or already faulted by a
concurrent queue-wide failure — completes on a task nobody awaits; the
tombstone path attaches a fault-observing continuation to it so a later write
failure never surfaces as a `TaskScheduler.UnobservedTaskException`. Second,
tombstoned entries stay in the class queues until a future `DequeueNext` pops
and skips them; that lazy purge is deliberate. Eagerly rebuilding a `Queue<T>`
under `_gate` on every cancellation would add ordering-invariant surface next
to the claim/cancel interlock for no real gain: any subsequent write of either
class drains both queues to empty, and the heartbeat loop guarantees one
arrives within a heartbeat interval, so worst-case residency is a few envelope
references for seconds — not a leak.
## Verification
The frame protocol lives in `ZB.MOM.WW.MxGateway.Worker.Ipc` (`WorkerFrameReader`,
@@ -0,0 +1,261 @@
# Follow-Ups: windev Redeploy, LDAP Test Fixtures, Runner Hygiene — Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:subagent-driven-development
> (opus implementers; controller verifies ops evidence; final review pass).
**Goal:** Close the five items surfaced by the 2026-08-07 live-actions cycle: repair windev's
crash-looping gateway and finish the deferred SEC-36 dashboard verification (NEXT-07), fix the
DashboardLdapLiveTests fixture drift (NEXT-06), resolve the unexpected macOS instance runner,
harden runner-1's plaintext registration token, and verify the cargo Bearer fix.
**Architecture:** Three independent live streams (windev serial: 1→2→3; repo test fix: 4;
Gitea/runner hygiene: 5, 6) run concurrently; task 7 closes out docs/trackers. No contract,
gateway-logic, or client changes — one test-file edit (Task 4) plus live ops plus docs.
**Tech Stack:** SSH + PowerShell `-EncodedCommand` (windev 10.100.0.48), SSH + docker compose
(10.100.0.35), Gitea admin API (`gitea.dohertylan.com`, token via `~/.zshenv` `GITEA_TOKEN`),
NSSM, xUnit live-LDAP suite, GLAuth at `10.100.0.35:3893`.
---
## Preflight facts (verified before planning)
- Unpushed local mxaccessgw commits: `0566716`, `9760497`, `5b153da`, `41e8648` (all docs-only).
**`origin/main` = `a346d51`** — contains all current code, so windev can build from
`origin/main` without any push.
- windev (`10.100.0.48`): NSSM service `MxAccessGw`; deployed Server build of 2026-06-25
(Auth 0.1.2.0, supports auth-DB schema 2) crash-loops on
`C:\ProgramData\MxGateway\gateway-auth.db` migrated to schema 3 on 2026-07-15
(`AuthStoreMigrationException`, ~3.8k10k Hosting-failed events/day). The NEW LDAP secret is
already staged as the 10th `AppEnvironmentExtra` entry (SEC-36 Task 3) — preserve it.
- `DashboardLdapLiveTests.cs` (`src/ZB.MOM.WW.MxGateway.IntegrationTests/`): uses
`admin`/`admin123` (3 tests) and `readonly`/`readonly123`. Directory reality
(`scadaproj/infra/glauth/config.toml`): `admin` exists, password is the standard dev test
password (`password`, hash `5e884898…42d8` — same as `multi-role`), and IS in GwAdmin
(othergroups `[5610, 5701]`); `readonly` does not exist; `gw-viewer` (primarygroup 5611 =
GwReader, NOT GwAdmin) is the natural not-an-admin fixture. Test binds
`MxGateway:Ldap` from `appsettings.json` (**`Server: localhost`**) + env overrides — so the
live run needs `MxGateway__Ldap__Server=10.100.0.35` as well as
`MxGateway__Ldap__ServiceAccountPassword` (from Mac user-secrets, never printed).
- Gitea instance runners (`GET /api/v1/admin/actions/runners`): id 1 `gitea-runner` (cap 4),
id 4 `macos-local-Josephs-MacBook-Pro` (**unexpected, online, labels overlap
ubuntu-latest**), id 5 `gitea-runner-2` (cap 2).
- `10.100.0.35:/opt/gitea/docker-compose.yml` (+ `docker-compose.yml.bak-tst30`): runner-1's
registration token inline in plaintext env, file world-readable. runner-2 uses
`GITEA_RUNNER_REGISTRATION_TOKEN_FILE: /run/secrets/runner_token` ← 0600
`/opt/gitea/runner_token`. runner-1 data volume `/opt/gitea/runner:/data` (its `.runner`
credential persists — the registration env is only needed for first registration).
- Cargo Bearer fix already applied (`~/.zshenv`, backup `~/.zshenv.bak-cli39`) and documented
(`docs/ClientPackaging.md`, commit `5b153da`). Task 7 verifies; no further action expected.
## Secret hygiene (binding, all tasks)
- Never print the LDAP service-account password, `GITEA_TOKEN`, cargo token, runner
registration tokens, or API keys — not in commands, logs, commits, or reports. Read the LDAP
password from `dotnet user-secrets list` into an env var without echoing
(e.g. `export MxGateway__Ldap__ServiceAccountPassword="$(dotnet user-secrets list --project src/ZB.MOM.WW.MxGateway.Server | awk -F' = ' '/ServiceAccountPassword/ {print $2}')"`).
- Documented dev **test users** (`multi-role`/`password`, `admin`/`password`,
`gw-viewer`/`password`) are NOT secrets — glauth.md publishes them; fine in code/commits.
- SSH→windev PowerShell: always `powershell -NoProfile -EncodedCommand <base64-UTF16LE>`;
never put secrets inside EncodedCommand blobs or argv.
---
### Task 1: NEXT-07 — Recon windev deployment layout + schema support
**Classification:** standard — read-only recon, but its output gates a service redeploy
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 4, Task 5, Task 6
**Files:** none edited. SSH recon on `10.100.0.48` + repo/scadaproj reads on the Mac.
Determine everything Task 2 needs, and confirm the fresh-deploy path is safe:
1. `nssm get MxAccessGw Application`, `AppDirectory`, `AppParameters`,
`AppEnvironmentExtra` (count entries; do NOT print values of secret-bearing entries —
names only).
2. Inventory the deployed dir: path, `ZB.MOM.WW.MxGateway.Server.exe` timestamp, whether
`appsettings.json`/`appsettings.Production.json` in the deploy dir differ from repo
`origin/main` (diff; windev-specific config must survive the redeploy).
3. Confirm build feasibility on windev: `dotnet --list-sdks` (need 10.x), locate an existing
mxaccessgw checkout/worktree (CI uses `scripts/ci/windev-worker-ci.ps1` — find its
worktree path) or pick a fresh clone location. Confirm `git fetch` reaches `origin/main`
= `a346d514dd24e775640e5667aa7cd8e561fec68a`.
4. Confirm current code supports auth-DB schema 3: find the auth-store supported-schema
constant (ZB.MOM.WW.Auth packages — check the package version the Server at `origin/main`
references, and/or the migration code in the shared scadaproj libs) and state the
evidence. **If current code does NOT support schema 3, STOP — report, do not deploy.**
5. Gateway endpoints for verification: bound URLs/ports (from deployed config/env), dashboard
scheme (http vs https → cookie will be `MxGatewayDashboard` vs `__Host-…`).
6. Check what migrated the DB to schema 3 on 2026-07-15 (event log / file timestamps) — only
to confirm schema 3 is the shared-lib current version, not an anomaly.
**Step: report** all findings as structured text (no secrets); no changes, no commits.
### Task 2: NEXT-07 — Build current Server on windev and redeploy the service
**Classification:** high-risk — replaces a running (crash-looping) service's binaries
**Estimated implement time:** ~10 min
**Parallelizable with:** none (needs Task 1)
**Files:** none in repo. windev filesystem + NSSM only.
Using Task 1's facts:
1. On windev, fetch/checkout `origin/main` (`a346d51…`) in the build worktree/clone.
2. `dotnet publish src/ZB.MOM.WW.MxGateway.Server -c Release` (match deployed layout/RID from
Task 1; framework-dependent vs self-contained must match what NSSM `Application` points at).
3. Stop the service (`nssm stop MxAccessGw`), confirm process exited.
4. Backup: deployed dir → sibling `*.bak-next07` copy; copy
`C:\ProgramData\MxGateway\gateway-auth.db` (+ `-wal`/`-shm` if present) to
`gateway-auth.db.bak-next07`. **Never delete the live DB.**
5. Deploy publish output over the deploy dir, then restore any windev-specific config files
identified in Task 1 (do not clobber live overrides; NSSM env entries are untouched by
file copies but verify count unchanged after start).
6. `nssm start MxAccessGw`; verify: service state RUNNING and stable ≥60 s (no restart
cycle), Application event log shows clean host start and **zero new
`AuthStoreMigrationException` / `Hosting failed to start`** after the start timestamp,
bound port answers (e.g. dashboard root or health endpoint returns HTTP).
7. Rollback if unhealthy: stop, restore `*.bak-next07` dir, start, report.
**Step: report** deployed SHA, verification evidence, backup paths. No repo commits.
### Task 3: SEC-36 deferred verification + NEXT-07/runbook closeout
**Classification:** standard
**Estimated implement time:** ~6 min
**Parallelizable with:** none (needs Task 2)
**Files:**
- Modify: `docs/runbooks/SEC-36-ldap-credential-rotation.md` (Correction 3 — mark the
deferred dashboard check done, dated)
- Modify: `archreview/2026-07-12/remediation/90-candidate-findings-next-cycle.md` (NEXT-07 →
resolved 2026-08-07, evidence one-liner)
1. Complete SEC-36 step 4 against the repaired windev gateway: log in to the dashboard as
`multi-role`/`password` end-to-end. Preferred: `curl` flow — GET `/login` (capture
antiforgery token + cookie), POST credentials, expect success redirect + auth cookie
(name per Task 1 scheme). If the login page resists scripting (Blazor circuit), report
exactly why and fall back to asserting a fresh `DashboardLdapLiveTests` green run
(Task 4) plus windev log evidence of successful LDAP bind on a manual attempt.
2. Update the two docs; commit locally (`docs(sec-36,next-07): …`), do NOT push.
### Task 4: NEXT-06 — Fix DashboardLdapLiveTests fixtures to match the shared directory
**Classification:** small — one test file, but must go green against live GLAuth
**Estimated implement time:** ~6 min
**Parallelizable with:** Task 1, Task 5, Task 6
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.IntegrationTests/DashboardLdapLiveTests.cs`
- Modify: `archreview/2026-07-12/remediation/90-candidate-findings-next-cycle.md` (NEXT-06 →
resolved)
- Possibly modify: `docs/GatewayTesting.md` (live-LDAP opt-in row: document the
`MxGateway__Ldap__Server` override needed when GLAuth is not localhost)
1. Read `DashboardAuthenticator` first: confirm a user who binds successfully but maps to no
role yields `Succeeded == false` (drives the gw-viewer fixture).
2. Fix fixtures: `admin`/`admin123``admin`/`password` (positive + wrong-password +
unreachable tests); `readonly`/`readonly123``gw-viewer`/`password` (exercises
user-binds-but-lacks-GwAdmin; keep the no-password-leak assertion, updating the asserted
literal). Update XML doc comments to match. Keep MXAccess-repo style rules
(TreatWarningsAsErrors).
3. Build: `dotnet build src/ZB.MOM.WW.MxGateway.IntegrationTests` (macOS OK — net10.0).
4. Live run (env only, never echo the password):
`MXGATEWAY_RUN_LIVE_LDAP_TESTS=1 MxGateway__Ldap__Server=10.100.0.35 MxGateway__Ldap__ServiceAccountPassword=<from user-secrets> dotnet test … --filter FullyQualifiedName~DashboardLdapLiveTests`
→ expect **5/5 passed** (this is also positive live proof of the SEC-36 service-account
bind).
5. Update tracker row (+ GatewayTesting.md if the Server-override note is missing); commit
locally (`test(ldap): …`), do NOT push.
### Task 5: Resolve the unexpected macOS instance runner (id 4)
**Classification:** standard — evidence-gated removal of a live runner registration
**Estimated implement time:** ~6 min
**Parallelizable with:** Task 1, Task 4, Task 6
**Files:**
- Modify: `docs/runbooks/TST-30-second-ci-runner.md` (the Correction paragraph mentions
"id 4 — an unrelated local macOS runner" — update to final state)
1. Evidence, local: `pgrep -fl act_runner`, `launchctl list | grep -i act`,
`brew services list | grep -i act`, look for `~/.runner`/act_runner config dirs. Evidence,
Gitea (token from `~/.zshenv`, never printed): runner detail for id 4 (labels, last
online), and whether any recent runs' jobs report `runner_id == 4`
(`GET /repos/{owner}/{repo}/actions/runs?…``…/runs/{id}/jobs` for both `mxaccessgw`
and `lmxopcua` recent runs).
2. Decision rule: the runner advertises ubuntu labels from a macOS host, so it can steal
Linux container jobs → **remove it** unless evidence shows it deliberately serves jobs
the docker runners cannot (none expected). Removal = stop the local act_runner process
AND disable its autostart (launchd/brew), then `DELETE /api/v1/admin/actions/runners/4`.
Keep the local config file (renamed `*.disabled-2026-08-07`) so re-registering with
mac-specific labels stays easy; note the re-registration recipe in the runbook edit.
3. Verify: admin runner list shows only ids 1 and 5, both online; no act_runner process
locally; a `pgrep` after 60 s still empty (nothing respawned).
4. Update the TST-30 runbook correction paragraph; commit locally, do NOT push.
### Task 6: Harden runner-1's registration token on 10.100.0.35
**Classification:** standard — touches the live CI stack's compose file
**Estimated implement time:** ~7 min
**Parallelizable with:** Task 1, Task 4, Task 5
**Files:** none in repo (host `/opt/gitea/` only; runbook note lands in Task 7 if needed).
1. Preconditions on the host: confirm runner-1's `/data/.runner` exists in its volume
(registration credential persists → the registration env var is no longer needed);
confirm both runners idle (no `act_runner`-spawned job containers, no in-progress runs
via API) before recreating.
2. Edit `/opt/gitea/docker-compose.yml` (backup first → `docker-compose.yml.bak-tst30b`):
replace runner-1's inline `GITEA_RUNNER_REGISTRATION_TOKEN: <plaintext>` with the same
`_FILE`/secrets pattern runner-2 uses (`/opt/gitea/runner_token`, 0600). Do NOT touch the
`gitea` service definition.
3. `docker compose up -d --no-deps` the runner-1 service only; verify it comes back online
in the admin runner list and its `.runner` identity is unchanged (still id 1).
4. Tighten perms: `chmod 600 /opt/gitea/docker-compose.yml docker-compose.yml.bak-tst30 docker-compose.yml.bak-tst30b`
(verify compose stack still operable by the deploy user).
5. Rotate the leaked registration token if the deployment allows:
`docker exec … gitea actions generate-runner-token` (or admin API) — if Gitea offers no
invalidation of the old value, say so explicitly in the report (residual risk: LAN actor
could register a rogue runner until rotation) rather than claiming it rotated.
6. Verify CI still works: trigger nothing; just confirm both runners online and the token
file perms; a real push lands naturally later. Report evidence.
### Task 7: Closeout — cargo Bearer verification, docs/tracker sync, commits
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** none (needs Tasks 3, 4, 5, 6)
**Files:**
- Modify: `archreview/2026-07-12/remediation/90-candidate-findings-next-cycle.md` (final
state of NEXT-06/NEXT-07 rows if Tasks 3/4 left anything)
- Possibly modify: `docs/GatewayTesting.md` / TST-30 runbook (runner topology now ids 1+5
only; token-hardening note)
1. Cargo Bearer verification (no printing): assert `~/.zshenv` line matches
`CARGO_REGISTRIES_DOHERTJ2_GITEA_TOKEN="Bearer …"` via `grep -c`, confirm
`docs/ClientPackaging.md` note present (commit `5b153da`); check no other credential
location (CI secrets, windev profiles) publishes to cargo — expected none.
2. Sweep: every doc touched this cycle consistent (runbooks, trackers, GatewayTesting.md);
`git grep` for stale phrases ("crash-loop… pending", "id 4", "admin123") and fix.
3. Commit remaining doc changes locally; do NOT push. List the full unpushed stack in the
report.
---
## Out of scope
- Pushing any mxaccessgw commits (user decides; stack listed at closeout).
- The five next-cycle candidate findings other than NEXT-06/NEXT-07.
- Auth-DB restore path for windev (fresh deploy chosen — preserves schema-3 data).
- `ci.yml` changes (labels, concurrency groups).
## Dependency graph
```
{1} → 2 → 3 ┐
{4} ├→ 7
{5} │
{6} ─────────┘
```
@@ -0,0 +1,13 @@
{
"planPath": "docs/plans/2026-08-07-followups-windev-ldapfixtures-runners.md",
"tasks": [
{"id": 1, "subject": "Task 1: NEXT-07 — Recon windev deployment layout + schema support", "status": "completed"},
{"id": 2, "subject": "Task 2: NEXT-07 — Build current Server on windev and redeploy the service", "status": "completed", "blockedBy": [1]},
{"id": 3, "subject": "Task 3: SEC-36 deferred verification + NEXT-07/runbook closeout", "status": "completed", "blockedBy": [2]},
{"id": 4, "subject": "Task 4: NEXT-06 — Fix DashboardLdapLiveTests fixtures to match the shared directory", "status": "completed"},
{"id": 5, "subject": "Task 5: Resolve the unexpected macOS instance runner (id 4)", "status": "completed"},
{"id": 6, "subject": "Task 6: Harden runner-1's registration token on 10.100.0.35", "status": "completed"},
{"id": 7, "subject": "Task 7: Closeout — cargo Bearer verification, docs/tracker sync, commits", "status": "completed", "blockedBy": [3, 4, 5, 6]}
],
"lastUpdated": "2026-08-07 (all tasks executed; SEC-36 verification done during Task 2's foreground smoke test; 8 commits local on main, not pushed; one pending operator action: Gitea registration-token UI reset)"
}
@@ -0,0 +1,420 @@
# Live Actions: SEC-36 Rotation, TST-30 Second Runner, Client Publish — Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task (or subagent-driven-development in-session).
**Goal:** Execute the three repo-complete-but-live-pending operator actions: rotate the dev GLAuth service-account credential (SEC-36), register a second Gitea Actions runner (TST-30), and publish the five client packages at 0.2.0 (Java 0.2.1).
**Architecture:** Three independent workstreams executed by subagents. SEC-36 is a strictly ordered cutover (pre-stage hosts → flip GLAuth → verify → finalize) with secret-hygiene rules. TST-30 is infra work on docker host 10.100.0.35 plus a concurrency verification. Publish runs the existing guarded `pack-clients.ps1 -Publish` + `tag-go-module.ps1` locally on macOS.
**Tech Stack:** ssh (BatchMode works to 10.100.0.35 and 10.100.0.48), PowerShell/nssm on windev, docker compose on 10.100.0.35, Gitea API (`~/.zshenv` has admin-scoped `GITEA_USERNAME`/`GITEA_TOKEN`), pwsh 7 on macOS.
---
## Preflight facts (verified 2026-08-07 from this macOS box)
- `ssh 10.100.0.35` OK. GLAuth container is **`zb-shared-glauth`**, compose working dir **`/home/dohertj2/zb-glauth`** (NOT the runbook's `~/Desktop/scadaproj/infra/glauth` — that path does not exist on the host; the runbook must be corrected in Task 5). Runner container **`gitea-runner`**, compose working dir **`/opt/gitea`**.
- `ssh 10.100.0.48` (windev) OK; `powershell -NoProfile` works; `nssm` at `C:\Users\dohertj2\AppData\Local\Microsoft\WinGet\Links\nssm.exe`.
- `wonder-app-vd03` does NOT resolve from macOS — check it from windev (Task 2).
- Gitea API: token valid (`/api/v1/user` → 200), admin (`/api/v1/admin/users` → 200, `POST /api/v1/admin/actions/runners/registration-token` → 200).
- Local `~/Desktop/scadaproj/infra/glauth/config.toml` exists (14 `passsha256` entries) — the git source of truth.
- `pwsh` at `/usr/local/bin/pwsh`.
## Secret hygiene (SEC-36, binding for every task)
- The new plaintext password lives ONLY in `$SECRET_FILE = /private/tmp/claude-501/-Users-dohertj2-Desktop-MxAccessGateway/67849767-a07c-4afa-94e2-3ce4a39d8d23/scratchpad/sec36-new-secret` (chmod 600), created in Task 1 and shredded in Task 5.
- **Never echo/cat the plaintext to stdout, never put it in a commit, a repo file, a log line, or a command whose text is captured verbatim.** Always load it into a shell variable from the file (`val=$(cat "$SECRET_FILE")`) and pass it via stdin or remote-side expansion, never inline in an `ssh "...literal..."` string where avoidable.
- The `passsha256` hash MAY appear in `config.toml` commits — that is the established pattern (14 existing entries).
- The OLD password must never be printed either. Its only uses are: GLAuth keeps honoring it until Task 4, and the single old-bind-must-fail probe in Task 4.
---
### Task 1: SEC-36 — Generate secret, stage GLAuth config change (repo + host copy diff)
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 2, Task 6, Task 9
**Files:**
- Modify: `~/Desktop/scadaproj/infra/glauth/config.toml` (the `serviceaccount` user's `passsha256`) — DO NOT commit yet (Task 5 commits)
- Create: `$SECRET_FILE` (scratchpad, chmod 600)
**Step 1: Generate the new secret and its hash**
```bash
SECRET_FILE="/private/tmp/claude-501/-Users-dohertj2-Desktop-MxAccessGateway/67849767-a07c-4afa-94e2-3ce4a39d8d23/scratchpad/sec36-new-secret"
umask 077
openssl rand -base64 24 | tr -d '\n' > "$SECRET_FILE"
chmod 600 "$SECRET_FILE"
NEW_SHA=$(cat "$SECRET_FILE" | tr -d '\n' | shasum -a 256 | awk '{print $1}')
echo "$NEW_SHA" # hash only — safe to display
```
Cross-check the hash recipe against `glauth.md` ("Generate `passsha256` from a plaintext password") in this repo and follow that recipe if it differs.
**Step 2: Diff host deployment config vs repo source of truth**
```bash
ssh 10.100.0.35 'cat /home/dohertj2/zb-glauth/config.toml' > /tmp/host-glauth-config.toml 2>/dev/null || true
diff ~/Desktop/scadaproj/infra/glauth/config.toml /tmp/host-glauth-config.toml
```
Small drift (comments, ports) is fine — note it. If the `serviceaccount` stanza differs structurally, STOP and surface before editing.
**Step 3: Edit the repo source of truth**
In `~/Desktop/scadaproj/infra/glauth/config.toml`, replace the `passsha256` value of the `[[users]]` entry whose `name`/`cn` is `serviceaccount` with `$NEW_SHA`. Edit ONLY that line. Do not `docker compose up` anything yet.
**Step 4: Record findings**
Report: hash staged (show hash, never plaintext), drift summary from step 2, and confirm `$SECRET_FILE` exists with mode 600.
---
### Task 2: SEC-36 — Determine wonder-app-vd03 LDAP status (via windev)
**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** Task 1, Task 6, Task 9
**Step 1: Try to reach vd03 from windev**
```bash
ssh 10.100.0.48 'powershell -NoProfile -Command "Test-Connection wonder-app-vd03 -Count 1 -Quiet"'
```
**Step 2: If reachable, read its gateway config for `MxGateway:Ldap:Enabled`**
Try (in order, stop at first success): `ssh` hop from windev; reading `\\wonder-app-vd03\c$\...` appsettings/environment via PowerShell remoting (`Invoke-Command -ComputerName wonder-app-vd03`); or `nssm get MxAccessGw AppEnvironmentExtra` remotely. Look for `MxGateway__Ldap__Enabled` / appsettings `Ldap:Enabled`.
**Step 3: Decide and record**
- `Enabled=false` or host unreachable/no gateway service → vd03 is OUT of scope; record why (runbook says its dashboard is disabled — `false` is the expected answer).
- `Enabled=true` → vd03 is IN scope for Task 3 pre-staging; record the connection method that worked.
---
### Task 3: SEC-36 — Pre-stage the NEW value on LDAP-enabled deployed hosts
**Classification:** high-risk
**Estimated implement time:** ~4 min
**Parallelizable with:** none (blocked by Tasks 1, 2)
**Step 1: Pre-stage windev (10.100.0.48)**
Load the secret locally, then set the env var remotely without leaking it into logged command text more than unavoidable (ssh arguments are not logged remotely by default; do NOT echo the value):
```bash
SECRET_FILE="/private/tmp/claude-501/-Users-dohertj2-Desktop-MxAccessGateway/67849767-a07c-4afa-94e2-3ce4a39d8d23/scratchpad/sec36-new-secret"
val=$(cat "$SECRET_FILE")
ssh 10.100.0.48 'powershell -NoProfile -Command "$v = [Console]::In.ReadLine(); $cur = (& nssm get MxAccessGw AppEnvironmentExtra) -join \"`n\"; Write-Output (\"CURRENT: \" + ($cur -replace \"Password=.*\", \"Password=<redacted>\")); & nssm set MxAccessGw AppEnvironmentExtra (\"MxGateway__Ldap__ServiceAccountPassword=\" + $v)"' <<< "$val"
```
**CAUTION:** `nssm set AppEnvironmentExtra` REPLACES the whole extra-environment block. First inspect `nssm get MxAccessGw AppEnvironmentExtra` (redacting any `Password=` values); if other variables exist, preserve them in the new value (newline-separated). Adapt quoting as needed — verify with a redacted `nssm get` afterwards.
**Step 2: Restart the service**
```bash
ssh 10.100.0.48 'nssm restart MxAccessGw'
```
Expected: service restarts. Binds against GLAuth now fail (old directory, new client value) — expected and brief; proceed immediately to Task 4.
**Step 3: vd03 (only if Task 2 said IN scope)** — same pre-stage + restart via the method Task 2 found.
---
### Task 4: SEC-36 — Rotate GLAuth and verify end-to-end
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** none (blocked by Task 3)
**Step 1: Back up current host config, sync the staged config, recreate**
```bash
ssh 10.100.0.35 'cp /home/dohertj2/zb-glauth/config.toml /home/dohertj2/zb-glauth/config.toml.bak-sec36'
scp ~/Desktop/scadaproj/infra/glauth/config.toml 10.100.0.35:/home/dohertj2/zb-glauth/config.toml
```
**If Task 1's diff showed host-vs-repo drift beyond the serviceaccount line:** do NOT wholesale-copy — instead edit only the serviceaccount `passsha256` line in the host copy (sed on the host), so unrelated host-local drift is preserved.
```bash
ssh 10.100.0.35 'cd /home/dohertj2/zb-glauth && docker compose up -d --force-recreate && sleep 3 && docker compose logs --tail 30'
```
Expected: clean startup, no TOML parse error. On parse error: restore `.bak-sec36`, recreate, STOP, surface.
**Step 2: Verify new credential binds (from the glauth host, ldapsearch or python)**
```bash
SECRET_FILE=".../sec36-new-secret" # full scratchpad path
val=$(cat "$SECRET_FILE")
ssh 10.100.0.35 'ldapsearch -x -H ldap://localhost:3893 -D "cn=serviceaccount,dc=zb,dc=local" -w "$(cat -)" -b "dc=zb,dc=local" "(cn=multi-role)" cn' <<< "$val"
```
Expected: search returns the `multi-role` entry. (If ldapsearch is missing on the host, run the equivalent from macOS against `10.100.0.35:3893`, or use `docker exec`.) Adjust the bind DN to match the actual `serviceaccount` DN in config.toml.
**Step 3: Verify the OLD value is dead — exactly ONE probe, from 10.100.0.35 itself**
One deliberately failing bind with the old password must return invalid credentials. **Only one attempt** (3-fail/10-min per-IP lockout; never probe from a shared-NAT box). The old value: recover it transiently from `config.toml.bak-sec36`'s hash? No — hash is not the plaintext. Instead: skip the plaintext probe if the old plaintext is not already known out-of-band; the hash replacement in config.toml is itself proof GLAuth no longer honors the old value (GLAuth compares against `passsha256` only). Record that reasoning instead of probing blind.
**Step 4: Verify dashboard login end-to-end on windev**
```bash
curl -sk -o /dev/null -w '%{http_code}' -c /tmp/mxgw-cookies.txt https://10.100.0.48:5001/login
```
Find the actual dashboard port from windev config first (`nssm get`/appsettings; likely https). Then POST the login form as `multi-role`/`password` (the GLAuth TEST USER password, not the service account) and expect a redirect + `__Host-MxGatewayDashboard` (or `MxGatewayDashboard`) cookie:
```bash
curl -sk -o /dev/null -w '%{http_code}\n' -b /tmp/mxgw-cookies.txt -c /tmp/mxgw-cookies.txt -d 'username=multi-role&password=password' <dashboard-base>/login
grep -i mxgatewaydashboard /tmp/mxgw-cookies.txt
```
Inspect the login page HTML first for real form field names / antiforgery token; adapt. A successful `multi-role` login proves the service-account search bind works with the new credential end-to-end. If HTTP verification proves impractical (antiforgery), fall back to grepping the gateway log on windev for a successful LDAP bind/login line after attempting — or run the live-LDAP integration test from macOS:
```bash
export MXGATEWAY_RUN_LIVE_LDAP_TESTS=1
export MxGateway__Ldap__ServiceAccountPassword="$(cat "$SECRET_FILE")"
dotnet test src/ZB.MOM.WW.MxGateway.IntegrationTests/ZB.MOM.WW.MxGateway.IntegrationTests.csproj --filter FullyQualifiedName~DashboardLdapLiveTests
```
Expected: green. (This binds from macOS to 10.100.0.35:3893 directly — it verifies the credential, and the curl/log check verifies windev.)
**Step 5: Rollback (only on failure)** — restore `.bak-sec36` on the host, `docker compose up -d --force-recreate`, re-point windev's env var back (old value from where it was before — if unknown, STOP and surface), `nssm restart MxAccessGw`.
---
### Task 5: SEC-36 — Finalize: commit source of truth, dev secrets, runbook fix, tracker, cleanup
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none (blocked by Task 4)
**Step 1: Commit and push the scadaproj glauth change (glauth paths ONLY)**
```bash
cd ~/Desktop/scadaproj
git add infra/glauth/config.toml
git commit -m "sec(glauth): rotate serviceaccount passsha256 (mxaccessgw SEC-36)"
git push
```
(`scadaproj` is a shared monorepo — stage only this path. If the worktree has unrelated staged changes, use `git commit -- infra/glauth/config.toml` style isolation.)
**Step 2: Set dev user-secrets on this macOS box**
```bash
cd ~/Desktop/MxAccessGateway
cat "$SECRET_FILE" | tr -d '\n' | dotnet user-secrets set "MxGateway:Ldap:ServiceAccountPassword" --project src/ZB.MOM.WW.MxGateway.Server/ZB.MOM.WW.MxGateway.Server.csproj
```
(Check `dotnet user-secrets set -h` for stdin support; if unsupported, pass via `"$(cat "$SECRET_FILE")"` — acceptable, it's a local process arg.)
**Step 3: Correct the runbook + flip tracker rows (mxaccessgw repo)**
- `docs/runbooks/SEC-36-ldap-credential-rotation.md`: fix the host deployment path (`/home/dohertj2/zb-glauth`, container `zb-shared-glauth`; repo source of truth remains `scadaproj/infra/glauth/`), and note vd03's actual status per Task 2.
- Grep `archreview/2026-07-12/remediation/` for SEC-36 pending-operator rows; flip to Done citing the runbook + today's date.
```bash
cd ~/Desktop/MxAccessGateway
grep -rn "SEC-36" archreview/2026-07-12/remediation/ docs/ | grep -iv binary
# edit the rows, then:
git add -A docs archreview && git commit -m "docs(sec-36): record live rotation done; correct runbook host paths"
```
**Step 4: Shred the secret file**
```bash
rm -P "$SECRET_FILE" 2>/dev/null || rm "$SECRET_FILE"
```
**Step 5: Done-criteria check** — walk the runbook's Done criteria list; report each as met/not-met.
---
### Task 6: TST-30 — Recon existing runner config on 10.100.0.35
**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 1, Task 2, Task 9
**Step 1: Inspect the existing runner**
```bash
ssh 10.100.0.35 'cat /opt/gitea/docker-compose.yml 2>/dev/null || sudo cat /opt/gitea/docker-compose.yml; ls /opt/gitea'
ssh 10.100.0.35 'docker inspect gitea-runner --format "{{json .Mounts}}"; docker exec gitea-runner cat /config.yaml 2>/dev/null || true'
```
Find: image/version, config file location (look for `container.network: traefik` and `capacity`/`maxParallel`), data volume, registration state file, docker socket mount, labels.
**Step 2: Check host capacity**
```bash
ssh 10.100.0.35 'nproc; free -h; df -h / | tail -1'
```
**Step 3: Decide (a)-variant** — second container vs raising `capacity` on the existing runner. Runbook prefers a second instance; if the existing runner's config shows a simple `capacity: 1` and resources are tight, raising capacity is the smaller change — but a second registered instance is the runbook default and survives one-runner wedge. Record the chosen variant, the exact compose/config snippets to reuse, and where the registration token goes.
---
### Task 7: TST-30 — Register and start the second runner
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** none (blocked by Task 6)
**Step 1: Mint an instance-level registration token**
```bash
source ~/.zshenv
curl -s -X POST -u "$GITEA_USERNAME:$GITEA_TOKEN" 'https://gitea.dohertylan.com/api/v1/admin/actions/runners/registration-token'
```
(Returns `{"token": "..."}` — a registration token, not a secret credential of lasting value; still avoid committing it.)
**Step 2: Create the second runner instance per Task 6's plan**
E.g. add a `gitea-runner-2` service to the compose (distinct name + data volume, same image, same `container.network: traefik`, same socket mount), inject the token via the runner's registration env (`GITEA_RUNNER_REGISTRATION_TOKEN`) or `act_runner register --no-interactive`, then `docker compose up -d gitea-runner-2` from `/opt/gitea`. Back up the compose file first (`cp docker-compose.yml docker-compose.yml.bak-tst30`). Do NOT touch the existing `gitea-runner` service definition.
**Step 3: Confirm both runners online**
```bash
curl -s -u "$GITEA_USERNAME:$GITEA_TOKEN" 'https://gitea.dohertylan.com/api/v1/admin/actions/runners' | python3 -m json.tool
```
Expected: ≥2 runners, both online. Also check `docker logs` of the new container for a clean registration + poll loop.
---
### Task 8: TST-30 — Verify concurrency, gitea:3000 resolution, tracker
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none (blocked by Task 7)
**Step 1: Trigger two concurrent runs**
Push two scratch branches to `mxaccessgw` back-to-back (empty commits off `main`, branch names `scratch/tst30-a`, `scratch/tst30-b`):
```bash
cd ~/Desktop/MxAccessGateway
git push origin main:refs/heads/scratch/tst30-a
git commit --allow-empty -m "tst30 concurrency probe" && git push origin HEAD:refs/heads/scratch/tst30-b && git reset --hard HEAD~1
```
(Adapt: any two pushes that fan out jobs. Clean up branches after: `git push origin :scratch/tst30-a :scratch/tst30-b`.)
**Step 2: Confirm parallel execution**
Poll the runs API/UI: the second run's jobs must START before the first run finishes.
```bash
curl -s -u "$GITEA_USERNAME:$GITEA_TOKEN" 'https://gitea.dohertylan.com/api/v1/repos/dohertj2/mxaccessgw/actions/tasks' | python3 -m json.tool | head -60
```
**Step 3: Confirm `gitea:3000` resolves on the new runner** — verify a job scheduled on runner-2 succeeds at checkout (checkout hits `gitea:3000` over the traefik network); identify which runner took each job from the runs UI/API or runner logs.
**Step 4: Flip TST-30 tracker rows** in `archreview/2026-07-12/remediation/` (grep `TST-30`) to Done with today's date; confirm `docs/GatewayTesting.md` prose is still accurate (it should be — it already describes the bypass as valid regardless of runner count). Commit.
---
### Task 9: Publish — Preflight audit (versions, registry collisions, toolchains)
**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 1, Task 2, Task 6
**Step 1: Audit source versions**
```bash
cd ~/Desktop/MxAccessGateway
grep -n 'version' clients/rust/Cargo.toml | head -5
grep -n 'version' clients/python/pyproject.toml clients/python/src/zb_mom_ww_mxgateway/version.py
grep -n 'ClientVersion' clients/go/mxgateway/version.go
grep -n '<Version>' clients/dotnet/ZB.MOM.WW.MxGateway.Client/ZB.MOM.WW.MxGateway.Client.csproj src/ZB.MOM.WW.MxGateway.Contracts/ZB.MOM.WW.MxGateway.Contracts.csproj
grep -n 'version' clients/java/build.gradle | head -5
grep -rn 'CLIENT_VERSION' clients/java --include=*.java | grep -i mxgatewayclientversion
```
Expected: Rust/Python/Go/.NET/Contracts = 0.2.0; Java build.gradle AND `MxGatewayClientVersion.CLIENT_VERSION` = 0.2.1. Any mismatch → STOP, surface (do not bump versions yourself; that's a scope change).
**Step 2: Query live registry for collisions**
```bash
source ~/.zshenv
for u in 'nuget/ZB.MOM.WW.MxGateway.Client/0.2.0' 'nuget/ZB.MOM.WW.MxGateway.Contracts/0.2.0' 'pypi/zb-mom-ww-mxaccess-gateway-client/0.2.0' 'cargo/zb-mom-ww-mxgateway-client/0.2.0' 'maven/com.zb.mom.ww.mxgateway-zb-mom-ww-mxgateway-client/0.2.1'; do
echo "$u => $(curl -s -o /dev/null -w '%{http_code}' -u "$GITEA_USERNAME:$GITEA_TOKEN" "https://gitea.dohertylan.com/api/v1/packages/dohertj2/$u")"
done
```
Expected: 404 for every target (unclaimed). Check the exact maven path convention against `pack-clients.ps1`'s own guard code and use its convention. 200 anywhere → STOP, surface.
**Step 3: Toolchain + workspace check**
```bash
git -C ~/Desktop/MxAccessGateway status --porcelain # must be clean (publish from a clean tree at origin/main)
for t in dotnet cargo go python3 gradle pwsh; do which $t; done
```
Also confirm `clients/go` module tag `clients/go/v0.2.0` does NOT already exist: `git ls-remote --tags origin 'clients/go/v*'`.
---
### Task 10: Publish — Run the guarded pack-and-publish
**Classification:** high-risk
**Estimated implement time:** ~5 min dispatch (script runtime longer)
**Parallelizable with:** none (blocked by Task 9)
**Step 1: Run pack-clients with publish**
```bash
cd ~/Desktop/MxAccessGateway
source ~/.zshenv
pwsh -NoProfile -File scripts/pack-clients.ps1 -Publish 2>&1 | tee /private/tmp/claude-501/-Users-dohertj2-Desktop-MxAccessGateway/67849767-a07c-4afa-94e2-3ce4a39d8d23/scratchpad/pack-clients-publish.log
```
Expected: per-language build+test+pack, collision guard prints "safe to publish" per artifact, uploads succeed. Timeout generously (Bash timeout 600000). If any language fails MID-loop, record exactly which artifacts pushed and which didn't — partial publish is the known failure mode; do not re-run blindly (re-run is safe only because the guard skips? NO — the guard ABORTS on existing versions. A re-run after partial publish will abort on the already-pushed artifact. If that happens, surface with the log; per-language `-Languages` selective re-run is the fix).
If macOS cannot build a language (e.g. gradle/java env), use `-Languages` to publish what builds and surface the remainder — do not fake success.
**Step 2: Verify each artifact now exists (200)** — re-run Task 9 step 2's loop; expected 200 everywhere published.
---
### Task 11: Publish — Go module tag
**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** Task 10 (blocked by Task 9)
**Step 1: Tag via the guarded script**
```bash
cd ~/Desktop/MxAccessGateway
pwsh -NoProfile -File scripts/tag-go-module.ps1 -Version 0.2.0
```
Read the script's param block first (`-Version` name may differ; it validates semver and that `version.go` matches, then creates+pushes `clients/go/v0.2.0`). Expected: tag created and pushed to origin.
**Step 2: Verify**
```bash
git ls-remote --tags origin 'clients/go/v0.2.0*'
```
Expected: exactly one tag. Optionally `GOPROXY=direct go list -m gitea.dohertylan.com/dohertj2/mxaccessgw/clients/go@v0.2.0` from a temp dir.
---
### Task 12: Publish — Docs/tracker closeout
**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** none (blocked by Tasks 10, 11)
**Step 1:** Update `docs/ClientPackaging.md`'s versioning narrative if it claims 0.2.0/0.2.1 are unpublished (it currently records the maven 0.2.1 exception; add a dated line that 0.2.0 (Java 0.2.1) published on 2026-08-07). Grep `archreview/2026-07-12/remediation/` for publish/CLI-39 pending-operator rows and flip to Done.
**Step 2:** Commit:
```bash
cd ~/Desktop/MxAccessGateway
git add docs archreview && git commit -m "docs(clients): record 0.2.0/0.2.1 publish + close operator actions"
```
**Step 3:** Report the full publish matrix (artifact → version → registry HTTP status).
---
## Dependency graph
```
{T1, T2} ──▶ T3 ──▶ T4 ──▶ T5 (SEC-36, strictly serial after recon)
T6 ──▶ T7 ──▶ T8 (TST-30)
T9 ──▶ {T10, T11} ──▶ T12 (Publish)
```
The three streams are mutually independent and run concurrently. All subagents run with model=opus per operator instruction.
## Out of scope (explicitly)
- Option (b)/(c) runner topologies and the `concurrency:` ci.yml experiment (TST-30 runbook marks them escalation/optional).
- The five next-cycle candidate findings in `archreview/2026-07-12/remediation/90-candidate-findings-next-cycle.md`.
- Any client version bumps (versions are already landed; a mismatch is a STOP-and-surface).
@@ -0,0 +1,18 @@
{
"planPath": "docs/plans/2026-08-07-live-actions-sec36-tst30-publish.md",
"tasks": [
{"id": 1, "subject": "Task 1: SEC-36 — Generate secret, stage GLAuth config change", "status": "completed"},
{"id": 2, "subject": "Task 2: SEC-36 — Determine wonder-app-vd03 LDAP status via windev", "status": "completed"},
{"id": 3, "subject": "Task 3: SEC-36 — Pre-stage NEW value on LDAP-enabled hosts (nssm + restart)", "status": "completed", "blockedBy": [1, 2]},
{"id": 4, "subject": "Task 4: SEC-36 — Rotate GLAuth and verify end-to-end", "status": "completed", "blockedBy": [3]},
{"id": 5, "subject": "Task 5: SEC-36 — Finalize: commit, dev secrets, runbook fix, tracker, cleanup", "status": "completed", "blockedBy": [4]},
{"id": 6, "subject": "Task 6: TST-30 — Recon existing runner config on 10.100.0.35", "status": "completed"},
{"id": 7, "subject": "Task 7: TST-30 — Register and start the second runner", "status": "completed", "blockedBy": [6]},
{"id": 8, "subject": "Task 8: TST-30 — Verify concurrency + gitea:3000 + tracker", "status": "completed", "blockedBy": [7]},
{"id": 9, "subject": "Task 9: Publish — Preflight audit (versions, collisions, toolchains)", "status": "completed"},
{"id": 10, "subject": "Task 10: Publish — Run pack-clients.ps1 -Publish", "status": "completed", "blockedBy": [9]},
{"id": 11, "subject": "Task 11: Publish — Go module tag clients/go/v0.2.0", "status": "completed", "blockedBy": [9]},
{"id": 12, "subject": "Task 12: Publish — Docs/tracker closeout", "status": "completed", "blockedBy": [10, 11]}
],
"lastUpdated": "2026-08-07 (all tasks executed; 4 closeout commits local on main, not pushed)"
}
+60 -4
View File
@@ -173,9 +173,15 @@ the worker's current state while the corresponding live transition may still be
buffered in the monitor's lease, so both can broadcast and the two are
indistinguishable on the feed. This applies to the acked-state delta and equally
to the older Raise/Clear presence repair: nothing serializes a reconcile pass
against the in-flight live stream. Alarm-feed consumers (`StreamAlarms` clients
and the dashboard alarm hub) must apply transitions idempotently — treat one as
"set this alarm to this state", never as an increment or a toggle.
against the in-flight live stream. The monitor narrows that window with a
best-effort dedup (NEXT-03): a buffered live transition whose worker timestamp
and resulting state the cache already carries from a repair — or whose Clear
matches a one-reconcile-generation tombstone keyed on the instance's original
raise timestamp — is suppressed instead of re-broadcast. The dedup fires only on
a positive marker match (unset timestamps never suppress), so the contract stays
at-least-once: alarm-feed consumers (`StreamAlarms` clients and the dashboard
alarm hub) must apply transitions idempotently — treat one as "set this alarm to
this state", never as an increment or a toggle.
### Alarm providers and failover
@@ -299,9 +305,16 @@ Default transport: one bidirectional named pipe per worker.
Pipe name:
```text
mxaccess-gateway-{gatewayProcessId}-{sessionId}
mxgw-{gatewayProcessId}-{sessionUid}
```
`sessionUid` is the session id without its `session-` prefix (the raw guid hex).
The name is deliberately short: on Unix-like hosts (the macOS/Linux test
matrix), .NET named pipes are Unix domain sockets at
`$TMPDIR/CoreFxPipe_{name}`, and macOS caps the socket path at 104 bytes while
its default per-user `TMPDIR` already spends ~49 of them. The gateway PID keeps
the name collision-free across gateway restarts.
Message framing:
```text
@@ -649,6 +662,49 @@ The exact field names should be adjusted to match the actual interop struct,
but the design principle is important: do not collapse status arrays into a
single success flag.
### MxStatus Detail Vocabulary
`MxStatusProxy.detail` carries MXAccess's `MxStatusDetail` code verbatim. The
vocabulary below is lifted from the installed toolkit's interop enum
(`Interop.aaMxDataConsumer``MxStatusDetail`, confirmed against the
`MxNativeCodec/MxStatus.cs` map in the MXAccess analysis project — see
`docs/DesignDecisions.md` external sources). Consumers mapping statuses onto
another protocol (e.g. OtOpcUa's OPC UA status mapping) should key on these
codes rather than truncating or banding the raw value:
| Detail | Name | Detail | Name |
|---|---|---|---|
| 0 | `MX_S_Success` | 1003 | `MX_E_IndexOutOfRange` |
| 1 | `MX_E_RequestTimedOut` | 1004 | `MX_E_DataOutOfRange` |
| 2 | `MX_E_PlatformCommunicationError` | 1005 | `MX_E_IncorrectDataType` |
| 3 | `MX_E_InvalidPlatformId` | 1006 | `MX_E_NotReadable` |
| 4 | `MX_E_InvalidEngineId` | 1007 | `MX_E_NotWriteable` |
| 5 | `MX_E_EngineCommunicationError` | 1008 | `MX_E_WriteAccessDenied` |
| 6 | `MX_E_InvalidReference` | 1009 | `MX_E_UnknownError` |
| 7 | `MX_E_NoGalaxyRepository` | 1010 | `MX_E_ObjectInitializing` |
| 8 | `MX_E_InvalidObjectId` | 1011 | `MX_E_EngineInitializing` |
| 9 | `MX_E_ObjectSignatureMismatch` | 1012 | `MX_E_SecuredWrite` |
| 10 | `MX_E_AttributeSignatureMismatch` | 1013 | `MX_E_VerifiedWrite` |
| 11 | `MX_E_ResolvingAttribute` | 1014 | `MX_E_NoAlarmAckPrivilege` |
| 12 | `MX_E_ResolvingObject` | 1015 | `MX_E_AlarmAckedAlready` |
| 13 | `MX_E_WrongDataType` | 1016 | `MX_E_UserNotHavingAccessRights` |
| 14 | `MX_E_WrongNumberOfDimensions` | 1017 | `MX_E_VerifierNotHavingVerifyRights` |
| 15 | `MX_E_InvalidIndex` | 8000 | `MX_E_AutomationObjectSpecificError` |
| 16 | `MX_E_IndexOutOfOrder` | 1000 | `MX_E_InvalidPrimitiveId` |
| 17 | `MX_E_DimensionDoesNotExist` | 1001 | `MX_E_InvalidAttributeId` |
| 18 | `MX_E_ConversionNotSupported` | 1002 | `MX_E_InvalidPropertyId` |
| 19 | `MX_E_UnableToConvertString` | 25 | `MX_E_GalaxyRepositoryBusy` |
| 20 | `MX_E_Overflow` | 26 | `MX_E_EngineOverloaded` |
| 21 | `MX_E_NmxVersionMismatch` | 23 | `MX_E_LmxVersionMismatch` |
| 22 | `MX_E_NmxInvalidCommand` | 24 | `MX_E_LmxInvalidCommand` |
Codes observed live in write-completion correlation: `1007`
(`MX_E_NotWriteable` — write to a read-only attribute) and `1008`
(`MX_E_WriteAccessDenied` — e.g. a write through a plain-advised handle that
lacks supervisory access). `1012`/`1013` mark attributes classified for
secured/verified writes; `1016`/`1017` are the secured-write credential
failures.
For command replies, return:
- protocol status,
+10
View File
@@ -53,6 +53,16 @@ first, then merge.
- Unreachable-host red: point at a bogus port / stop sshd, confirm the job fails fast, not hangs.
- Concurrency: push two branches back-to-back, confirm the second remote run waits on the lock.
- Nightly: trigger the schedule path, confirm `live` runs and a forced failure opens an issue.
**Done 2026-08-10** (Check 6). Verified two ways: (a) production — every red nightly since
2026-07-17 has auto-filed an issue (#126#139) authored by the `gitea-actions` bot, e.g. run 672
→ issue #139, with the built-in token masked to `***` in the job log; (b) a forced-failure probe
on the throwaway branch `test/tst25-check6-nightly-issue` (run 677 → issue #140, since closed and
the branch deleted), which reproduced the job shape with `exit 1` in place of the live step and
confirmed the `if: failure()` step fires, the token carries issue-write, and the payload is
well-formed. The probe also caught the one defect: `${{ github.server_url }}` is the
runner-internal `http://gitea:3000`, so the run link in the issue body was unreachable from a
browser — the body now uses the `PUBLIC_SERVER_URL` job env instead (the API call still targets
`github.server_url`, which is what the job container can resolve).
- Confirm no key material appears in job logs.
## Degraded mode
+7 -2
View File
@@ -26,7 +26,10 @@
<Target Name="StampSourceRevision"
BeforeTargets="GetAssemblyVersion;GenerateAssemblyInfo"
Condition="'$(SourceRevisionId)' == ''">
<Exec Command="git -C &quot;$(MSBuildThisFileDirectory)&quot; rev-parse --short HEAD"
<!-- The trailing "." is load-bearing: $(MSBuildThisFileDirectory) ends in a path
separator, and on Windows that trailing backslash escapes the closing quote,
mangling the command so git's stderr got stamped as the revision (NEXT-09). -->
<Exec Command="git -C &quot;$(MSBuildThisFileDirectory).&quot; rev-parse --short HEAD"
ConsoleToMSBuild="true"
StandardOutputImportance="Low"
ContinueOnError="true"
@@ -34,7 +37,9 @@
<Output TaskParameter="ConsoleOutput" PropertyName="_StampedGitSha" />
</Exec>
<PropertyGroup>
<SourceRevisionId Condition="'$(_StampedGitSha)' != ''">$(_StampedGitSha.Trim())</SourceRevisionId>
<!-- Accept only something that looks like a git short SHA; Exec's ConsoleOutput
mixes in stderr, so any git failure text must never become the revision. -->
<SourceRevisionId Condition="$([System.Text.RegularExpressions.Regex]::IsMatch('$(_StampedGitSha.Trim())', '^[0-9a-f]{7,40}$'))">$(_StampedGitSha.Trim())</SourceRevisionId>
</PropertyGroup>
</Target>
@@ -34,6 +34,14 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
private readonly Dictionary<string, ActiveAlarmSnapshot> _alarms = new(StringComparer.Ordinal);
private readonly List<Subscriber> _subscribers = [];
// NEXT-03 dedup tombstones, guarded by _sync: alarm instances whose Clear was synthesized by
// the most recent reconcile pass, keyed by reference with the instance's original raise
// timestamp as the identity marker. A buffered live Clear for the same instance is a duplicate
// of the repair and is suppressed. One generation deep: each reconcile pass replaces the map,
// so a tombstone lives at least one reconcile interval — far longer than the lease buffer the
// duplicate would be sitting in — and the map stays bounded by the feed's churn per interval.
private readonly Dictionary<string, Timestamp> _clearedByReconcile = new(StringComparer.Ordinal);
// Current provider status (mode + degraded + reason + since), guarded by _sync.
// Initialized to the alarm-manager, not-degraded baseline so a late joiner sees
// a sensible status even before any OnAlarmProviderModeChanged event arrives.
@@ -413,17 +421,60 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
{
if (transition.TransitionKind == AlarmTransitionKind.Clear)
{
_alarms.Remove(reference);
bool wasKnown = _alarms.Remove(reference);
if (!wasKnown && IsDuplicateOfReconcileClear(reference, transition))
{
return;
}
}
else
{
_alarms[reference] = SnapshotFromTransition(transition);
ActiveAlarmSnapshot snapshot = SnapshotFromTransition(transition);
bool duplicate = _alarms.TryGetValue(reference, out ActiveAlarmSnapshot? existing)
&& IsDuplicateOfCachedState(existing, snapshot);
_alarms[reference] = snapshot;
if (duplicate)
{
return;
}
}
Broadcast(new AlarmFeedMessage { Transition = transition }, reference);
}
}
// NEXT-03: best-effort dedup of the reconcile/live race. A reconcile that already synthesized
// this transition as a feed repair left the cache carrying the worker's transition timestamp
// and resulting state — both derived from the same worker-side value the live transition
// carries — so an exact (timestamp, state) match means this live transition's outcome has
// already been broadcast. Suppress only on a positive match: an unset timestamp on either
// side keeps today's at-least-once behavior.
private static bool IsDuplicateOfCachedState(ActiveAlarmSnapshot existing, ActiveAlarmSnapshot incoming)
{
return existing.LastTransitionTimestamp is not null
&& incoming.LastTransitionTimestamp is not null
&& existing.LastTransitionTimestamp.Equals(incoming.LastTransitionTimestamp)
&& existing.CurrentState == incoming.CurrentState;
}
// NEXT-03, the Clear leg. A reconcile Clear repair removes the cache entry before the buffered
// live Clear drains, so there is no cached state to compare against; the tombstone recorded by
// ApplyReconcile identifies the cleared instance by its original raise timestamp instead. The
// match consumes the tombstone, so a genuinely new raise/clear cycle (which carries a newer
// original raise timestamp) is never swallowed. Caller holds _sync.
private bool IsDuplicateOfReconcileClear(string reference, OnAlarmTransitionEvent transition)
{
if (transition.OriginalRaiseTimestamp is not null
&& _clearedByReconcile.TryGetValue(reference, out Timestamp? clearedInstance)
&& clearedInstance.Equals(transition.OriginalRaiseTimestamp))
{
_clearedByReconcile.Remove(reference);
return true;
}
return false;
}
// Handles the worker's provider-mode-change event: updates the stored provider
// status, broadcasts it to every subscriber (provider status is global, not
// alarm-scoped), records the switch metric, and forces a cache reconcile so the
@@ -533,11 +584,14 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
//
// Delivery semantics: feed repair transitions are AT-LEAST-ONCE, not exactly-once. A reconcile
// reads the worker's current state while the corresponding live transition may still be
// buffered in the alarm lease's channel; both then broadcast, and the two are indistinguishable
// on the feed. This is inherent to the reconcile design and pre-dates the acked-state delta
// (the Raise/Clear repair has always had it), since nothing serializes a reconcile against the
// in-flight live stream. Consumers must therefore treat alarm state idempotently — apply a
// transition as "set the alarm to this state", never as an increment or a toggle.
// buffered in the alarm lease's channel; both would then broadcast, and the two are
// indistinguishable on the feed, since nothing serializes a reconcile against the in-flight
// live stream. ApplyTransition narrows that window with a best-effort dedup (NEXT-03): a live
// transition whose worker timestamp and resulting state the cache already carries — or whose
// Clear matches a tombstone recorded below — was already broadcast as a repair and is
// suppressed. The dedup fires only on a positive marker match, so the contract stays
// at-least-once: consumers must still treat alarm state idempotently — apply a transition as
// "set the alarm to this state", never as an increment or a toggle.
private void ApplyReconcile(IEnumerable<ActiveAlarmSnapshot> snapshots)
{
Dictionary<string, ActiveAlarmSnapshot> next = new(StringComparer.Ordinal);
@@ -551,10 +605,19 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
lock (_sync)
{
// Previous-generation tombstones have outlived the buffered live transitions they
// guard against (one full reconcile interval); start this pass's generation fresh.
_clearedByReconcile.Clear();
foreach (KeyValuePair<string, ActiveAlarmSnapshot> existing in _alarms)
{
if (!next.ContainsKey(existing.Key))
{
if (existing.Value.OriginalRaiseTimestamp is not null)
{
_clearedByReconcile[existing.Key] = existing.Value.OriginalRaiseTimestamp;
}
Broadcast(
new AlarmFeedMessage { Transition = TransitionFromSnapshot(existing.Value, AlarmTransitionKind.Clear) },
existing.Key);
@@ -417,7 +417,8 @@ public sealed class SessionManager : ISessionManager
string? clientIdentity,
string? ownerKeyId)
{
string sessionId = CreateSessionId();
string sessionUid = Guid.NewGuid().ToString("N");
string sessionId = $"session-{sessionUid}";
string backendName = string.IsNullOrWhiteSpace(request.RequestedBackend)
? GatewayContractInfo.DefaultBackendName
: request.RequestedBackend!;
@@ -425,7 +426,11 @@ public sealed class SessionManager : ISessionManager
TimeSpan startupTimeout = TimeSpan.FromSeconds(_options.Worker.StartupTimeoutSeconds);
TimeSpan shutdownTimeout = TimeSpan.FromSeconds(_options.Worker.ShutdownTimeoutSeconds);
TimeSpan leaseDuration = TimeSpan.FromSeconds(_options.Sessions.DefaultLeaseSeconds);
string pipeName = $"mxaccess-gateway-{Environment.ProcessId}-{sessionId}";
// The short prefix and bare guid keep the pipe's Unix-domain-socket path
// (TMPDIR + "CoreFxPipe_" + name) inside the 104-byte sun_path limit on
// macOS, whose default per-user TMPDIR is ~49 chars; the gateway PID keeps
// the name collision-free across gateway restarts (NEXT-01).
string pipeName = $"mxgw-{Environment.ProcessId}-{sessionUid}";
string nonce = CreateNonce();
DateTimeOffset openedAt = _timeProvider.GetUtcNow();
string clientCorrelationId = CreateClientCorrelationId(request.ClientSessionName, sessionId);
@@ -484,11 +489,6 @@ public sealed class SessionManager : ISessionManager
: timeout;
}
private static string CreateSessionId()
{
return $"session-{Guid.NewGuid():N}";
}
private static string CreateNonce()
{
Span<byte> bytes = stackalloc byte[32];
@@ -155,6 +155,133 @@ public sealed class GatewayAlarmMonitorAttachOrderTests
await monitor.StopAsync(CancellationToken.None);
}
/// <summary>
/// NEXT-03. A reconcile Raise repair applied while the matching live Raise is still
/// buffered must not double-broadcast: the live transition carrying the same worker
/// timestamp and resulting state the cache already holds is a duplicate and is suppressed.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task LiveTransitionMatchingReconcileRepair_IsSuppressed()
{
using GatewayMetrics metrics = new();
await using FakeSessionManager sessions = new();
using GatewayAlarmMonitor monitor = CreateMonitor(sessions, metrics);
using CancellationTokenSource cts = new();
await monitor.StartAsync(cts.Token);
await sessions.WaitForSubscribeStartAsync(WaitTimeout);
// The reconcile sees the raised alarm first (its Raise repair broadcasts before this
// reader attaches) and stamps the cache with the worker's transition timestamp.
Timestamp raiseTime = Timestamp.FromDateTimeOffset(new DateTimeOffset(2026, 8, 10, 12, 0, 0, TimeSpan.Zero));
sessions.SetReconcileSnapshot(SnapshotAt(AlarmConditionState.Active, raiseTime, raiseTime));
sessions.EmitEvent(ProviderModeProbe(1));
await WaitUntilAsync(
() => monitor.CurrentAlarms.Any(alarm => alarm.AlarmFullReference == AlarmReference),
WaitTimeout);
List<AlarmFeedMessage> received = [];
TaskCompletionSource snapshotComplete = new(TaskCreationOptions.RunContinuationsAsynchronously);
using CancellationTokenSource streamCts = new();
Task reader = ReadFeedAsync(monitor, received, snapshotComplete, streamCts.Token, untilSnapshotComplete: true);
await snapshotComplete.Task.WaitAsync(WaitTimeout);
// The buffered live Raise drains with the same worker timestamp — a duplicate of the
// repair. The follow-up Acknowledge with a newer timestamp is genuine and must pass.
sessions.EmitEvent(TransitionAt(2, AlarmTransitionKind.Raise, raiseTime, raiseTime));
Timestamp ackTime = Timestamp.FromDateTimeOffset(new DateTimeOffset(2026, 8, 10, 12, 0, 5, TimeSpan.Zero));
sessions.EmitEvent(TransitionAt(3, AlarmTransitionKind.Acknowledge, ackTime, raiseTime));
await WaitForAsync(
received,
m => m.PayloadCase == AlarmFeedMessage.PayloadOneofCase.Transition
&& m.Transition.TransitionKind == AlarmTransitionKind.Acknowledge,
WaitTimeout);
lock (received)
{
AlarmFeedMessage[] transitions = received
.Where(m => m.PayloadCase == AlarmFeedMessage.PayloadOneofCase.Transition)
.ToArray();
AlarmFeedMessage single = Assert.Single(transitions);
Assert.Equal(AlarmTransitionKind.Acknowledge, single.Transition.TransitionKind);
}
await streamCts.CancelAsync();
await reader;
await cts.CancelAsync();
await monitor.StopAsync(CancellationToken.None);
}
/// <summary>
/// NEXT-03, the Clear leg. A reconcile Clear repair removes the cache entry, so the
/// buffered live Clear is deduped through the tombstone keyed on the instance's original
/// raise timestamp — and a genuinely new raise/clear cycle is never swallowed.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task LiveClearMatchingReconcileClearRepair_IsSuppressed()
{
using GatewayMetrics metrics = new();
await using FakeSessionManager sessions = new();
using GatewayAlarmMonitor monitor = CreateMonitor(sessions, metrics);
using CancellationTokenSource cts = new();
await monitor.StartAsync(cts.Token);
await sessions.WaitForSubscribeStartAsync(WaitTimeout);
Timestamp raiseTime = Timestamp.FromDateTimeOffset(new DateTimeOffset(2026, 8, 10, 13, 0, 0, TimeSpan.Zero));
sessions.SetReconcileSnapshot(SnapshotAt(AlarmConditionState.Active, raiseTime, raiseTime));
sessions.EmitEvent(ProviderModeProbe(1));
await WaitUntilAsync(
() => monitor.CurrentAlarms.Any(alarm => alarm.AlarmFullReference == AlarmReference),
WaitTimeout);
List<AlarmFeedMessage> received = [];
TaskCompletionSource snapshotComplete = new(TaskCreationOptions.RunContinuationsAsynchronously);
using CancellationTokenSource streamCts = new();
Task reader = ReadFeedAsync(monitor, received, snapshotComplete, streamCts.Token, untilSnapshotComplete: true);
await snapshotComplete.Task.WaitAsync(WaitTimeout);
// The worker no longer reports the alarm: the reconcile synthesizes the Clear repair and
// tombstones the instance by its original raise timestamp.
sessions.SetReconcileSnapshot();
sessions.EmitEvent(ProviderModeProbe(2));
await WaitForAsync(
received,
m => m.PayloadCase == AlarmFeedMessage.PayloadOneofCase.Transition
&& m.Transition.TransitionKind == AlarmTransitionKind.Clear,
WaitTimeout);
// The buffered live Clear for the SAME instance is a duplicate of the repair; the Raise
// that follows starts a new instance and must pass.
Timestamp clearTime = Timestamp.FromDateTimeOffset(new DateTimeOffset(2026, 8, 10, 13, 0, 10, TimeSpan.Zero));
sessions.EmitEvent(TransitionAt(3, AlarmTransitionKind.Clear, clearTime, raiseTime));
Timestamp newRaiseTime = Timestamp.FromDateTimeOffset(new DateTimeOffset(2026, 8, 10, 13, 0, 20, TimeSpan.Zero));
sessions.EmitEvent(TransitionAt(4, AlarmTransitionKind.Raise, newRaiseTime, newRaiseTime));
await WaitForAsync(
received,
m => m.PayloadCase == AlarmFeedMessage.PayloadOneofCase.Transition
&& m.Transition.TransitionKind == AlarmTransitionKind.Raise,
WaitTimeout);
lock (received)
{
AlarmTransitionKind[] kinds = received
.Where(m => m.PayloadCase == AlarmFeedMessage.PayloadOneofCase.Transition)
.Select(m => m.Transition.TransitionKind)
.ToArray();
Assert.Equal([AlarmTransitionKind.Clear, AlarmTransitionKind.Raise], kinds);
}
await streamCts.CancelAsync();
await reader;
await cts.CancelAsync();
await monitor.StopAsync(CancellationToken.None);
}
private static GatewayAlarmMonitor CreateMonitor(FakeSessionManager sessions, GatewayMetrics metrics)
{
AlarmsOptions options = new()
@@ -254,6 +381,31 @@ public sealed class GatewayAlarmMonitorAttachOrderTests
SourceProvider = AlarmProviderMode.Alarmmgr,
};
// Snapshot carrying the worker-side identity markers the NEXT-03 dedup compares on.
private static ActiveAlarmSnapshot SnapshotAt(
AlarmConditionState state,
Timestamp lastTransition,
Timestamp originalRaise)
{
ActiveAlarmSnapshot snapshot = Snapshot(state);
snapshot.LastTransitionTimestamp = lastTransition;
snapshot.OriginalRaiseTimestamp = originalRaise;
return snapshot;
}
// Live transition with explicit worker timestamps, for driving the NEXT-03 dedup.
private static MxEvent TransitionAt(
ulong sequence,
AlarmTransitionKind kind,
Timestamp transitionTimestamp,
Timestamp originalRaise)
{
MxEvent mxEvent = Transition(sequence, kind);
mxEvent.OnAlarmTransition.TransitionTimestamp = transitionTimestamp;
mxEvent.OnAlarmTransition.OriginalRaiseTimestamp = originalRaise;
return mxEvent;
}
private static async Task<AlarmFeedMessage> WaitForAsync(
List<AlarmFeedMessage> received,
Func<AlarmFeedMessage, bool> predicate,
@@ -699,13 +699,27 @@ public sealed class EventStreamServiceTests
};
}
private static async Task WaitUntilAsync(Func<bool> predicate)
// The real-clock deadline here is load-sensitive on a wide host (windev runs this suite
// 36-way parallel). Surfacing the unmet condition instead of letting the bare
// TaskCanceledException escape is what makes such a failure diagnosable rather than a
// mystery cancellation attributed to "the environment".
private static async Task WaitUntilAsync(
Func<bool> predicate,
[CallerArgumentExpression(nameof(predicate))] string? predicateExpression = null)
{
using CancellationTokenSource cancellationTokenSource = new(TestTimeout);
while (!predicate())
{
try
{
await Task.Delay(TimeSpan.FromMilliseconds(10), cancellationTokenSource.Token);
}
catch (OperationCanceledException)
{
Assert.Fail(
$"Timed out after {TestTimeout} waiting for condition: {predicateExpression}");
}
}
}
/// <summary>Fake session manager for testing event streams.</summary>
@@ -37,6 +37,46 @@ public sealed class SessionManagerTests
Assert.Equal(1, metrics.GetSnapshot().SessionsOpened);
}
/// <summary>
/// Verifies the pipe name stays short enough that its Unix-domain-socket path
/// (TMPDIR + "CoreFxPipe_" + name) fits the 104-byte macOS sun_path limit under the
/// default per-user TMPDIR (~49 chars), and keeps the pid + session-guid uniqueness
/// contract (NEXT-01).
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task OpenSessionAsync_PipeNameIsShortAndUniquePerPidAndSession()
{
FakeWorkerClient workerClient = new();
FakeSessionWorkerClientFactory factory = new(workerClient)
{
ApplyLifecycleTransitions = true,
};
SessionManager manager = CreateManager(factory);
GatewaySession session = await manager.OpenSessionAsync(CreateOpenRequest(), "client-1", ownerKeyId: null, CancellationToken.None);
Assert.Matches($"^mxgw-{Environment.ProcessId}-[0-9a-f]{{32}}$", session.PipeName);
Assert.EndsWith(session.SessionId["session-".Length..], session.PipeName, StringComparison.Ordinal);
// 104-byte sun_path NUL ~49-char default macOS TMPDIR "CoreFxPipe_".
const int MaxPipeNameLength = 104 - 1 - 49 - 11;
// macOS pids top out at 99999, so 5 digits is the worst case the budget must survive.
// The check must substitute that worst case for the *running* pid's digit count rather
// than pad the measured length upward: Windows pids are routinely 6 digits, which would
// otherwise fail this assertion on a host whose own pipe-name limit (256) is irrelevant
// to the macOS budget being guarded here.
const int WorstCaseMacOsPidDigits = 5;
int runningPidDigits = Environment.ProcessId
.ToString(System.Globalization.CultureInfo.InvariantCulture).Length;
int worstCaseLength = session.PipeName.Length - runningPidDigits + WorstCaseMacOsPidDigits;
Assert.True(
worstCaseLength <= MaxPipeNameLength,
$"Pipe name '{session.PipeName}' would overflow the macOS socket-path budget at a 5-digit pid " +
$"({worstCaseLength} > {MaxPipeNameLength}).");
}
/// <summary>Verifies that a session opened by an authenticated caller records that caller's API key id in OwnerKeyId.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
@@ -138,7 +138,7 @@ public sealed class SessionWorkerClientFactoryFakeWorkerTests : IAsyncDisposable
return new GatewaySession(
FakeWorkerHarness.DefaultSessionId,
GatewayContractInfo.DefaultBackendName,
$"mxaccessgw-session-fake-worker-{Guid.NewGuid():N}",
$"mxgw-sf-{Guid.NewGuid():N}",
FakeWorkerHarness.DefaultNonce,
"test-client",
"fake-worker-session-test",
@@ -5,6 +5,7 @@ using ZB.MOM.WW.MxGateway.Contracts;
using ZB.MOM.WW.MxGateway.Contracts.Proto;
using ZB.MOM.WW.MxGateway.Server.Metrics;
using ZB.MOM.WW.MxGateway.Server.Workers;
using ZB.MOM.WW.MxGateway.Tests.TestSupport;
namespace ZB.MOM.WW.MxGateway.Tests.Gateway.Workers.Fakes;
@@ -60,13 +61,8 @@ public sealed class FakeWorkerHarness : IAsyncDisposable
int maxMessageBytes = WorkerFrameProtocolOptions.DefaultMaxMessageBytes,
CancellationToken cancellationToken = default)
{
string pipeName = $"mxaccessgw-fake-worker-{Guid.NewGuid():N}";
NamedPipeServerStream gatewayStream = new(
pipeName,
PipeDirection.InOut,
maxNumberOfServerInstances: 1,
PipeTransmissionMode.Byte,
PipeOptions.Asynchronous);
string pipeName = $"mxgw-fw-{Guid.NewGuid():N}";
NamedPipeServerStream gatewayStream = TestNamedPipe.CreateServer(pipeName);
NamedPipeClientStream workerStream = CreateWorkerStream(pipeName);
Task waitForConnectionTask = gatewayStream.WaitForConnectionAsync(cancellationToken);
@@ -75,7 +75,7 @@ public sealed class WorkerClientTests
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerCommand, commandEnvelope.BodyCase);
Assert.False(string.IsNullOrWhiteSpace(commandEnvelope.CorrelationId));
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping));
WorkerCommandReply reply = await invokeTask.WaitAsync(TestTimeout);
@@ -123,7 +123,7 @@ public sealed class WorkerClientTests
TestTimeout,
CancellationToken.None);
WorkerEnvelope commandEnvelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping));
WorkerCommandReply reply = await nextInvoke.WaitAsync(TestTimeout);
Assert.Equal(MxCommandKind.Ping, reply.Reply.Kind);
@@ -152,7 +152,7 @@ public sealed class WorkerClientTests
// Send the stale reply for the already-timed-out command, then the second
// command's reply. The pipe is FIFO, so the read loop processes (and discards)
// the stale reply before the second reply — no fixed Task.Delay needed.
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateCommandReplyEnvelope(timedOutCommand.CorrelationId, MxCommandKind.Ping));
Task<WorkerCommandReply> secondInvokeTask = client.InvokeAsync(
@@ -160,7 +160,7 @@ public sealed class WorkerClientTests
TestTimeout,
CancellationToken.None);
WorkerEnvelope secondCommand = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateCommandReplyEnvelope(secondCommand.CorrelationId, MxCommandKind.GetWorkerInfo));
WorkerCommandReply reply = await secondInvokeTask.WaitAsync(TestTimeout);
@@ -203,7 +203,7 @@ public sealed class WorkerClientTests
+ "envelope sequences must be strictly increasing in wire order.");
previousSequence = commandEnvelope.Sequence;
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping));
}
@@ -224,9 +224,9 @@ public sealed class WorkerClientTests
await using IAsyncEnumerator<WorkerEvent> events =
client.ReadEventsAsync(cancellationTokenSource.Token).GetAsyncEnumerator(cancellationTokenSource.Token);
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateEventEnvelope(sequence: 11, MxEventFamily.OnDataChange));
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateEventEnvelope(sequence: 12, MxEventFamily.OperationComplete));
Assert.True(await events.MoveNextAsync());
@@ -276,9 +276,9 @@ public sealed class WorkerClientTests
});
await CompleteHandshakeAsync(client, pipePair);
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateEventEnvelope(sequence: 11, MxEventFamily.OnDataChange));
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateEventEnvelope(sequence: 12, MxEventFamily.OnDataChange));
await WaitUntilAsync(
@@ -315,9 +315,9 @@ public sealed class WorkerClientTests
// No StreamEvents consumer is attached, so the capacity-1 event channel fills and the event
// writer blocks on the second event's timed WriteAsync.
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateEventEnvelope(sequence: 11, MxEventFamily.OnDataChange));
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateEventEnvelope(sequence: 12, MxEventFamily.OnDataChange));
Task<WorkerCommandReply> invokeTask = client.InvokeAsync(
@@ -325,7 +325,7 @@ public sealed class WorkerClientTests
TestTimeout,
CancellationToken.None);
WorkerEnvelope commandEnvelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping));
WorkerCommandReply reply = await invokeTask.WaitAsync(TestTimeout);
@@ -359,9 +359,9 @@ public sealed class WorkerClientTests
processHandle: CreateProcessHandle(process));
await CompleteHandshakeAsync(client, pipePair);
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateEventEnvelope(sequence: 11, MxEventFamily.OnDataChange));
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateEventEnvelope(sequence: 12, MxEventFamily.OnDataChange));
// Deterministic: this completes the instant Kill() runs, with no timing window.
@@ -427,7 +427,7 @@ public sealed class WorkerClientTests
WorkerEnvelope commandEnvelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerCommand, commandEnvelope.BodyCase);
await pipePair.WorkerWriter.WriteAsync(CreateWorkerFaultEnvelope("scripted mid-command fault"));
await pipePair.WriteAsync(CreateWorkerFaultEnvelope("scripted mid-command fault"));
WorkerClientException exception = await Assert.ThrowsAsync<WorkerClientException>(
async () => await invokeTask.WaitAsync(TestTimeout));
@@ -530,7 +530,7 @@ public sealed class WorkerClientTests
DateTimeOffset previousHeartbeat = client.LastHeartbeatAt;
clock.Advance(TimeSpan.FromSeconds(1));
await pipePair.WorkerWriter.WriteAsync(CreateHeartbeatEnvelope(workerProcessId: 9876));
await pipePair.WriteAsync(CreateHeartbeatEnvelope(workerProcessId: 9876));
await WaitUntilAsync(
() => client.ProcessId == 9876 && client.LastHeartbeatAt > previousHeartbeat,
@@ -693,7 +693,7 @@ public sealed class WorkerClientTests
// write would block forever on a full OS pipe buffer.
for (ulong sequence = 1; sequence <= 5; sequence++)
{
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateEventEnvelope(sequence: sequence, MxEventFamily.OnDataChange));
}
@@ -756,7 +756,7 @@ public sealed class WorkerClientTests
ulong sequence = 1;
for (; sequence <= (ulong)stagingBound; sequence++)
{
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateEventEnvelope(sequence, MxEventFamily.OnDataChange));
}
@@ -771,7 +771,7 @@ public sealed class WorkerClientTests
TestTimeout,
CancellationToken.None);
WorkerEnvelope commandEnvelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping));
WorkerCommandReply reply = await invokeTask.WaitAsync(TestTimeout);
Assert.Equal(MxCommandKind.Ping, reply.Reply.Kind);
@@ -782,7 +782,7 @@ public sealed class WorkerClientTests
// ones the stopped read loop never drains stay in the OS pipe buffer instead of blocking.
for (int extra = 0; extra < 3 * capacity; extra++, sequence++)
{
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateEventEnvelope(sequence, MxEventFamily.OnDataChange));
}
@@ -836,7 +836,7 @@ public sealed class WorkerClientTests
// Above EventChannelCapacity but below the 2× staging bound: no consumer, no fault.
for (ulong sequence = 1; sequence <= eventCount; sequence++)
{
await pipePair.WorkerWriter.WriteAsync(
await pipePair.WriteAsync(
CreateEventEnvelope(sequence, MxEventFamily.OnDataChange));
}
@@ -902,8 +902,8 @@ public sealed class WorkerClientTests
Assert.Equal(Nonce, gatewayHello.GatewayHello.Nonce);
Assert.Equal(GatewayContractInfo.WorkerProtocolVersion, gatewayHello.GatewayHello.SupportedProtocolVersion);
await pipePair.WorkerWriter.WriteAsync(CreateWorkerHelloEnvelope());
await pipePair.WorkerWriter.WriteAsync(CreateWorkerReadyEnvelope());
await pipePair.WriteAsync(CreateWorkerHelloEnvelope());
await pipePair.WriteAsync(CreateWorkerReadyEnvelope());
await startTask.WaitAsync(TestTimeout);
}
@@ -1065,17 +1065,39 @@ public sealed class WorkerClientTests
/// <summary>Frame writer for worker messages.</summary>
public WorkerFrameWriter WorkerWriter { get; }
/// <summary>
/// Writes one envelope from the fake worker side, bounded by <see cref="TestTimeout"/>.
/// </summary>
/// <remarks>
/// Every test-side write goes through here rather than <see cref="WorkerWriter"/> directly so
/// that a write which cannot complete fails this test instead of hanging it. An unbounded
/// write is not a local problem: a test method that never returns leaves xUnit's assembly
/// runner awaiting it forever, so <c>ITestAssemblyFinished</c> is never raised and the whole
/// <c>testhost</c> process never exits — the run reports no failure at all, just a wedge.
/// </remarks>
/// <param name="envelope">The envelope to write.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
public async Task WriteAsync(WorkerEnvelope envelope)
{
using CancellationTokenSource writeTimeout = new(TestTimeout);
try
{
await WorkerWriter.WriteAsync(envelope, writeTimeout.Token);
}
catch (OperationCanceledException) when (writeTimeout.IsCancellationRequested)
{
Assert.Fail(
$"The fake worker's pipe write did not complete within {TestTimeout}. The gateway " +
"side has stopped reading and the pipe buffer is full.");
}
}
/// <summary>Creates a connected pipe pair for testing.</summary>
/// <returns>The connected <see cref="PipePair"/>.</returns>
public static async Task<PipePair> CreateAsync()
{
string pipeName = $"mxaccessgw-workerclient-tests-{Guid.NewGuid():N}";
NamedPipeServerStream gatewayStream = new(
pipeName,
PipeDirection.InOut,
maxNumberOfServerInstances: 1,
PipeTransmissionMode.Byte,
PipeOptions.Asynchronous);
string pipeName = $"mxgw-wc-{Guid.NewGuid():N}";
NamedPipeServerStream gatewayStream = TestNamedPipe.CreateServer(pipeName);
NamedPipeClientStream workerStream = new(
".",
pipeName,
@@ -1,3 +1,4 @@
using System.Net;
using System.Security.Cryptography.X509Certificates;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Time.Testing;
@@ -25,15 +26,19 @@ public sealed class SelfSignedCertificateProviderTests
Assert.True(cert.NotBefore.ToUniversalTime() < time.GetUtcNow().UtcDateTime);
Assert.True(cert.HasPrivateKey);
string sans = ReadSubjectAltNames(cert);
Assert.Contains("localhost", sans);
Assert.Contains("gw.internal", sans);
Assert.Contains(Environment.MachineName, sans);
// Format() renders IP SANs as "IP Address:<addr>"; the IPv6 loopback may appear
// as "::1" or its expanded form depending on the platform crypto library.
Assert.Contains("127.0.0.1", sans);
Assert.True(sans.Contains("::1") || sans.Contains("0:0:0:0:0:0:0:1"),
$"Expected IPv6 loopback in SANs but got: {sans}");
X509SubjectAlternativeNameExtension san = ReadSubjectAltNames(cert);
string[] dnsNames = [.. san.EnumerateDnsNames()];
IPAddress[] ipAddresses = [.. san.EnumerateIPAddresses()];
// DNS SANs are compared case-insensitively (DNS names are), and IP SANs are compared
// as parsed IPAddress values. Asserting against the extension's Format() string instead
// would be platform-dependent: Windows' CryptFormatObject renders the IPv6 loopback
// fully expanded ("0000:0000:...:0001") while the managed formatter renders "::1".
Assert.Contains(dnsNames, name => name.Equals("localhost", StringComparison.OrdinalIgnoreCase));
Assert.Contains(dnsNames, name => name.Equals("gw.internal", StringComparison.OrdinalIgnoreCase));
Assert.Contains(dnsNames, name => name.Equals(Environment.MachineName, StringComparison.OrdinalIgnoreCase));
Assert.Contains(IPAddress.Loopback, ipAddresses);
Assert.Contains(IPAddress.IPv6Loopback, ipAddresses);
X509EnhancedKeyUsageExtension eku = cert.Extensions.OfType<X509EnhancedKeyUsageExtension>().Single();
Assert.Contains(eku.EnhancedKeyUsages.Cast<System.Security.Cryptography.Oid>(),
@@ -155,8 +160,6 @@ public sealed class SelfSignedCertificateProviderTests
private const string SubjectAltNameOid = "2.5.29.17";
private static string ReadSubjectAltNames(X509Certificate2 cert)
=> cert.Extensions
.First(e => e.Oid?.Value == SubjectAltNameOid)
.Format(false);
private static X509SubjectAlternativeNameExtension ReadSubjectAltNames(X509Certificate2 cert)
=> new(cert.Extensions.First(e => e.Oid?.Value == SubjectAltNameOid).RawData);
}
@@ -0,0 +1,55 @@
using System.IO.Pipes;
namespace ZB.MOM.WW.MxGateway.Tests.TestSupport;
/// <summary>
/// Creates the gateway-side <see cref="NamedPipeServerStream"/> that in-process worker fakes connect
/// to, with an explicit OS buffer so a test that deliberately stops draining the pipe cannot block
/// its own writer forever.
/// </summary>
/// <remarks>
/// The <c>NamedPipeServerStream(string, PipeDirection, int, PipeTransmissionMode, PipeOptions)</c>
/// overload passes <c>inBufferSize: 0</c> and <c>outBufferSize: 0</c> to <c>CreateNamedPipe</c>. On
/// Windows that reserves <em>no</em> buffer at all: a write completes only once the peer reads it —
/// measured on a Windows 10 host, the very first 63-byte write to a non-reading peer never completed,
/// against 65 520 bytes absorbed by the same pipe declared with 64 KiB buffers. On macOS and Linux
/// .NET backs named pipes with Unix domain sockets, whose socket buffer absorbs several kilobytes
/// regardless, so the zero-buffer declaration is invisible there.
/// <para>
/// That asymmetry wedged the whole Windows test host: <c>WorkerClientTests</c> pushes events past the
/// worker client's staging bound to prove the client faults, and after the fault the client's read
/// loop stops by design. On Windows the next test-side write then blocked forever with no timeout, so
/// the test never returned, xUnit never raised <c>ITestAssemblyFinished</c>, and <c>testhost</c> never
/// exited even though every other test had already passed.
/// </para>
/// <para>
/// Buffering the test pipe scopes those tests to the behavior they are actually asserting — the
/// gateway's own staging/queue backpressure — instead of the OS pipe's flow control. The gateway's
/// production pipe in <c>SessionWorkerClientFactory</c> keeps the unbuffered declaration on purpose:
/// there both ends run continuous read loops and every write is bounded by the worker client's stop
/// token, so a stalled peer cancels rather than blocks.
/// </para>
/// </remarks>
internal static class TestNamedPipe
{
/// <summary>
/// The buffer reserved for each direction. Comfortably larger than any frame volume a test
/// pushes at a stopped reader, and the size Windows itself uses for a typical named pipe.
/// </summary>
internal const int BufferBytes = 64 * 1024;
/// <summary>Creates the gateway-side server pipe for <paramref name="pipeName"/>.</summary>
/// <param name="pipeName">The pipe name the fake worker connects to.</param>
/// <returns>An asynchronous, byte-mode server pipe with explicit buffers.</returns>
internal static NamedPipeServerStream CreateServer(string pipeName)
{
return new NamedPipeServerStream(
pipeName,
PipeDirection.InOut,
maxNumberOfServerInstances: 1,
PipeTransmissionMode.Byte,
PipeOptions.Asynchronous,
inBufferSize: BufferBytes,
outBufferSize: BufferBytes);
}
}
@@ -545,6 +545,70 @@ public sealed class WorkerFrameProtocolTests
Assert.Equal(2UL, frame2.Sequence);
}
/// <summary>
/// NEXT-04. A frame claimed by the draining lock-holder before its caller's cancellation lands
/// is abandoned — the cancelled caller never awaits its completion. If the wire write then
/// faults, the tombstone path's fault-observing continuation must still observe the exception
/// so it never surfaces as <see cref="TaskScheduler.UnobservedTaskException"/>.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task WriteAsync_ClaimedFrameAbandonedByCancellation_FaultIsObserved()
{
string marker = $"NEXT-04-{Guid.NewGuid():N}";
WorkerFrameProtocolOptions options = CreateOptions();
bool sawUnobservedMarkerFault = false;
EventHandler<UnobservedTaskExceptionEventArgs> handler = (sender, args) =>
{
if (args.Exception.ToString().Contains(marker))
{
sawUnobservedMarkerFault = true;
}
};
TaskScheduler.UnobservedTaskException += handler;
try
{
using (SecondWriteFaultingGatedStream stream = new SecondWriteFaultingGatedStream(marker))
{
WorkerFrameWriter writer = new WorkerFrameWriter(stream, options);
// Writer A holds the lock, blocked mid-write of its own frame.
Task firstWrite = writer.WriteAsync(CreateGatewayHelloEnvelope(), WorkerFrameWritePriority.Control);
await AwaitWithTimeoutAsync(stream.FirstWriteStarted);
using (CancellationTokenSource cts = new CancellationTokenSource())
{
// Queue the doomed event write behind A, release A so its drain claims the
// event frame and blocks mid-write of it, then cancel the queued caller —
// the frame is claimed, so the caller unwinds without an awaiter for it.
Task abandonedWrite = writer.WriteAsync(CreateEventEnvelope(), WorkerFrameWritePriority.Event, cts.Token);
stream.ReleaseFirstWrite();
await AwaitWithTimeoutAsync(stream.SecondWriteStarted);
cts.Cancel();
await Assert.ThrowsAnyAsync<OperationCanceledException>(async () => await abandonedWrite);
}
// Fault the abandoned frame's wire write; observe writer A's own outcome so only
// the abandoned frame's completion could ever raise the marker unobserved.
stream.ReleaseSecondWrite();
_ = await Record.ExceptionAsync(async () => await firstWrite);
}
GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect();
}
finally
{
TaskScheduler.UnobservedTaskException -= handler;
}
Assert.False(
sawUnobservedMarkerFault,
"The abandoned frame's write fault surfaced as an unobserved-task exception.");
}
/// <summary>
/// WRK-22 / IPC-26, the review's shutdown scenario. A cancelled event frame queued before a
/// shutdown-ack control frame must not trail the ack on the wire: the tombstone rule plus the
@@ -749,6 +813,69 @@ public sealed class WorkerFrameProtocolTests
}
}
// A MemoryStream whose first write blocks until released and whose second write blocks until
// released and then throws, so a test can abandon a claimed frame by cancellation and fault its
// wire write afterwards (NEXT-04).
private sealed class SecondWriteFaultingGatedStream : MemoryStream
{
private readonly SemaphoreSlim _firstRelease = new SemaphoreSlim(0);
private readonly SemaphoreSlim _secondRelease = new SemaphoreSlim(0);
private readonly TaskCompletionSource<bool> _firstWriteStarted =
new TaskCompletionSource<bool>(TaskCreationOptions.RunContinuationsAsynchronously);
private readonly TaskCompletionSource<bool> _secondWriteStarted =
new TaskCompletionSource<bool>(TaskCreationOptions.RunContinuationsAsynchronously);
private readonly string _faultMessage;
private int _writeCount;
public SecondWriteFaultingGatedStream(string faultMessage)
{
_faultMessage = faultMessage;
}
/// <summary>Gets a task that completes once the first <see cref="WriteAsync"/> call has started blocking.</summary>
public Task FirstWriteStarted => _firstWriteStarted.Task;
/// <summary>Gets a task that completes once the second <see cref="WriteAsync"/> call has started blocking.</summary>
public Task SecondWriteStarted => _secondWriteStarted.Task;
/// <summary>Releases the first blocked write so it can complete.</summary>
public void ReleaseFirstWrite() => _firstRelease.Release();
/// <summary>Releases the second blocked write so it can throw.</summary>
public void ReleaseSecondWrite() => _secondRelease.Release();
/// <inheritdoc />
public override async Task WriteAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken)
{
int writeIndex = Interlocked.Increment(ref _writeCount);
if (writeIndex == 1)
{
_firstWriteStarted.TrySetResult(true);
await _firstRelease.WaitAsync(cancellationToken);
}
else if (writeIndex == 2)
{
_secondWriteStarted.TrySetResult(true);
await _secondRelease.WaitAsync(cancellationToken);
throw new IOException(_faultMessage);
}
await base.WriteAsync(buffer, offset, count, cancellationToken);
}
/// <inheritdoc />
protected override void Dispose(bool disposing)
{
if (disposing)
{
_firstRelease.Dispose();
_secondRelease.Dispose();
}
base.Dispose(disposing);
}
}
// A MemoryStream whose first WriteAsync blocks until released, so a test can queue additional frames
// behind an in-progress write and observe the writer's priority ordering.
private sealed class GatedWriteStream : MemoryStream
@@ -27,12 +27,7 @@ public sealed class WorkerPipeClientTests
"nonce-secret");
WorkerFrameProtocolOptions frameOptions = new(workerOptions);
using NamedPipeServerStream server = new(
pipeName,
PipeDirection.InOut,
1,
PipeTransmissionMode.Byte,
PipeOptions.Asynchronous);
using NamedPipeServerStream server = TestNamedPipe.CreateServer(pipeName);
WorkerPipeClient client = new(
connectTimeoutMilliseconds: 5000,
@@ -105,12 +100,7 @@ public sealed class WorkerPipeClientTests
await Task.Delay(150);
using NamedPipeServerStream server = new(
pipeName,
PipeDirection.InOut,
1,
PipeTransmissionMode.Byte,
PipeOptions.Asynchronous);
using NamedPipeServerStream server = TestNamedPipe.CreateServer(pipeName);
await Task.Factory.FromAsync(server.BeginWaitForConnection, server.EndWaitForConnection, null);
@@ -1214,7 +1214,9 @@ public sealed class WorkerPipeSessionTests
using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(15));
using PipePair pipePair = await PipePair.CreateAsync(cancellation.Token);
FakeRuntimeSession runtime = new();
// A far-off heartbeat interval keeps heartbeat flushes out of the measurement window.
// A far-off heartbeat interval keeps every heartbeat after the first out of the measurement
// window; the first beat is sent immediately on entering the message loop (see
// RunHeartbeatLoopAsync) and is closed out explicitly below.
FlushCountingPassthroughStream countingStream = new(pipePair.WorkerStream);
WorkerPipeSession session = CreatePipeSession(
countingStream,
@@ -1227,8 +1229,18 @@ public sealed class WorkerPipeSessionTests
Task runTask = session.RunAsync(cancellation.Token);
await CompleteGatewayHandshakeAsync(pipePair, cancellation.Token);
// Let the idle drain loop settle (no events yet → no flushes) and record the baseline.
await Task.Delay(100, cancellation.Token);
// Close the pre-burst window on an explicit signal rather than a sleep. Three frames precede
// the burst — WorkerHello, WorkerReady, and the immediate first heartbeat — and each is
// flushed only after its bytes are already on the pipe, so having read a frame is no evidence
// that its flush has been counted. Read the first beat (the last pre-burst frame), then wait
// for the writer's deferred flush before sampling the baseline. A fixed delay left the first
// beat's flush free to land after the sample under CI scheduling pressure, where it was
// charged to the burst and the assertion below saw two flushes instead of one.
await ReadUntilAsync(
pipePair.GatewayReader,
WorkerEnvelope.BodyOneofCase.WorkerHeartbeat,
cancellation.Token);
await countingStream.WaitForAllWritesFlushedAsync(cancellation.Token);
int baselineFlushes = countingStream.FlushCount;
// Enqueue a full 128-event batch atomically so the drain loop sees it as one batch.
@@ -1250,8 +1262,15 @@ public sealed class WorkerPipeSessionTests
cancellation.Token);
}
// The whole burst cost exactly one additional flush.
Assert.Equal(1, countingStream.FlushCount - baselineFlushes);
// Take the same edge on the burst's own flush — the 128 frames are on the wire before the
// writer flushes them — then assert on the recorded flush shape: exactly one flush beyond the
// baseline, and that flush carried every event frame of the burst. Asserting the shape (not
// just the count) is what makes the coalescing claim faithful: a split batch would show its
// first flush carrying fewer than the whole burst.
await countingStream.WaitForAllWritesFlushedAsync(cancellation.Token);
IReadOnlyList<int> flushWriteCounts = countingStream.SnapshotFlushWriteCounts();
Assert.Equal(baselineFlushes + 1, flushWriteCounts.Count);
Assert.Equal(burst, flushWriteCounts[baselineFlushes]);
await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
}
@@ -2154,13 +2173,22 @@ public sealed class WorkerPipeSessionTests
}
}
// Wraps the worker side of the pipe and counts FlushAsync calls so a test can assert the event
// drain loop coalesces a burst into a single flush. Delegates every other operation to the inner
// stream; does not own the inner stream's lifetime (PipePair disposes it).
// Wraps the worker side of the pipe and records the flush shape of the frames the writer emits —
// how many stream writes each flush coalesced — so a test can assert the event drain loop turns a
// burst into a single flush. Delegates every other operation to the inner stream; does not own the
// inner stream's lifetime (PipePair disposes it).
//
// The writer flushes only after writing a whole drained batch, so the peer can read every frame of
// that batch before the flush runs. Neither "I read the frames" nor any fixed sleep is evidence
// that a flush has been counted; WaitForAllWritesFlushedAsync is the explicit edge a test must
// take before sampling the counters.
private sealed class FlushCountingPassthroughStream : Stream
{
private readonly Stream inner;
private int flushCount;
private readonly object gate = new();
private readonly List<int> flushWriteCounts = new();
private readonly List<FlushWaiter> waiters = new();
private int writesSinceLastFlush;
/// <summary>Initializes the passthrough over the given inner stream.</summary>
/// <param name="inner">The stream to delegate to.</param>
@@ -2169,8 +2197,53 @@ public sealed class WorkerPipeSessionTests
this.inner = inner;
}
/// <summary>Gets the number of <see cref="FlushAsync"/> calls observed so far.</summary>
public int FlushCount => Volatile.Read(ref flushCount);
/// <summary>Gets the number of flushes observed so far.</summary>
public int FlushCount
{
get
{
lock (gate)
{
return flushWriteCounts.Count;
}
}
}
/// <summary>
/// Returns the number of stream writes coalesced into each flush, in flush order.
/// </summary>
/// <returns>A snapshot of the per-flush write counts.</returns>
public IReadOnlyList<int> SnapshotFlushWriteCounts()
{
lock (gate)
{
return flushWriteCounts.ToArray();
}
}
/// <summary>
/// Completes once every write issued so far has been flushed, giving the caller a
/// happens-before edge on the writer's deferred flush instead of a timing guess.
/// </summary>
/// <param name="cancellationToken">Token to abandon the wait.</param>
/// <returns>A task that completes when no write is left unflushed.</returns>
public Task WaitForAllWritesFlushedAsync(CancellationToken cancellationToken)
{
FlushWaiter waiter;
lock (gate)
{
if (writesSinceLastFlush == 0)
{
return Task.CompletedTask;
}
// Any flush drains every pending write, so the next flush is exactly the edge wanted.
waiter = new FlushWaiter(flushWriteCounts.Count + 1);
waiters.Add(waiter);
}
return AwaitFlushWaiterAsync(waiter, cancellationToken);
}
/// <inheritdoc />
public override bool CanRead => inner.CanRead;
@@ -2194,14 +2267,14 @@ public sealed class WorkerPipeSessionTests
/// <inheritdoc />
public override void Flush()
{
Interlocked.Increment(ref flushCount);
RecordFlush();
inner.Flush();
}
/// <inheritdoc />
public override Task FlushAsync(CancellationToken cancellationToken)
{
Interlocked.Increment(ref flushCount);
RecordFlush();
return inner.FlushAsync(cancellationToken);
}
@@ -2219,11 +2292,87 @@ public sealed class WorkerPipeSessionTests
public override void SetLength(long value) => inner.SetLength(value);
/// <inheritdoc />
public override void Write(byte[] buffer, int offset, int count) => inner.Write(buffer, offset, count);
public override void Write(byte[] buffer, int offset, int count)
{
RecordWrite();
inner.Write(buffer, offset, count);
}
/// <inheritdoc />
public override Task WriteAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken)
=> inner.WriteAsync(buffer, offset, count, cancellationToken);
{
RecordWrite();
return inner.WriteAsync(buffer, offset, count, cancellationToken);
}
private static async Task AwaitFlushWaiterAsync(
FlushWaiter waiter,
CancellationToken cancellationToken)
{
using (cancellationToken.Register(() => waiter.Completion.TrySetCanceled()))
{
await waiter.Completion.Task.ConfigureAwait(false);
}
}
// Counted at write issue, not completion: the peer can observe the bytes as soon as the write
// is issued, so the pending count must already reflect the write by then.
private void RecordWrite()
{
lock (gate)
{
writesSinceLastFlush++;
}
}
private void RecordFlush()
{
List<FlushWaiter>? released = null;
lock (gate)
{
flushWriteCounts.Add(writesSinceLastFlush);
writesSinceLastFlush = 0;
for (int index = waiters.Count - 1; index >= 0; index--)
{
if (waiters[index].TargetFlushCount <= flushWriteCounts.Count)
{
released ??= new List<FlushWaiter>();
released.Add(waiters[index]);
waiters.RemoveAt(index);
}
}
}
if (released is null)
{
return;
}
foreach (FlushWaiter waiter in released)
{
waiter.Completion.TrySetResult(true);
}
}
// A pending WaitForAllWritesFlushedAsync call: completes once the recorded flush count
// reaches TargetFlushCount.
private sealed class FlushWaiter
{
/// <summary>Initializes a waiter released at the given flush ordinal.</summary>
/// <param name="targetFlushCount">Flush count that releases the waiter.</param>
public FlushWaiter(int targetFlushCount)
{
TargetFlushCount = targetFlushCount;
Completion = new TaskCompletionSource<bool>(
TaskCreationOptions.RunContinuationsAsynchronously);
}
/// <summary>Gets the flush count at which this waiter completes.</summary>
public int TargetFlushCount { get; }
/// <summary>Gets the completion signaled when the target flush count is reached.</summary>
public TaskCompletionSource<bool> Completion { get; }
}
}
private sealed class PipePair : IDisposable
@@ -2256,12 +2405,7 @@ public sealed class WorkerPipeSessionTests
public static async Task<PipePair> CreateAsync(CancellationToken cancellationToken)
{
string pipeName = $"mxaccessgw-worker-session-tests-{Guid.NewGuid():N}";
NamedPipeServerStream gatewayStream = new(
pipeName,
PipeDirection.InOut,
maxNumberOfServerInstances: 1,
PipeTransmissionMode.Byte,
PipeOptions.Asynchronous);
NamedPipeServerStream gatewayStream = TestNamedPipe.CreateServer(pipeName);
NamedPipeClientStream workerStream = new(
".",
pipeName,
@@ -0,0 +1,42 @@
using System.IO.Pipes;
namespace ZB.MOM.WW.MxGateway.Worker.Tests.TestSupport;
/// <summary>
/// Creates the gateway-side <see cref="NamedPipeServerStream"/> the worker tests connect to, with an
/// explicit OS buffer so a test that stops draining the pipe cannot block its own writer forever.
/// </summary>
/// <remarks>
/// The <c>NamedPipeServerStream(string, PipeDirection, int, PipeTransmissionMode, PipeOptions)</c>
/// overload passes <c>inBufferSize: 0</c> and <c>outBufferSize: 0</c> to <c>CreateNamedPipe</c>, which
/// on Windows reserves no buffer at all: a write completes only once the peer reads it. Measured on a
/// Windows 10 host, the first 63-byte write to a non-reading peer never completed, against 65 520
/// bytes absorbed by the same pipe declared with 64 KiB buffers. A test-side write with no timeout
/// against a stopped reader therefore hangs the test method, and a test method that never returns
/// keeps xUnit from raising <c>ITestAssemblyFinished</c> — so <c>testhost</c> never exits even after
/// every other test has passed. This suite runs only on Windows, so it has no macOS Unix-domain-socket
/// buffer to hide behind. See the matching helper in the gateway test project.
/// </remarks>
internal static class TestNamedPipe
{
/// <summary>
/// The buffer reserved for each direction. Comfortably larger than any frame volume a test
/// pushes at a stopped reader, and the size Windows itself uses for a typical named pipe.
/// </summary>
internal const int BufferBytes = 64 * 1024;
/// <summary>Creates the gateway-side server pipe for <paramref name="pipeName"/>.</summary>
/// <param name="pipeName">The pipe name the worker under test connects to.</param>
/// <returns>An asynchronous, byte-mode server pipe with explicit buffers.</returns>
internal static NamedPipeServerStream CreateServer(string pipeName)
{
return new NamedPipeServerStream(
pipeName,
PipeDirection.InOut,
1,
PipeTransmissionMode.Byte,
PipeOptions.Asynchronous,
BufferBytes,
BufferBytes);
}
}
@@ -92,6 +92,8 @@ public sealed class WorkerFrameWriter
/// already claimed it, in which case the frame may still reach the wire even though this call
/// observes <see cref="OperationCanceledException"/>. That residual window is by design: blocking
/// the canceller behind the very write it is abandoning would defeat the point of cancellation.
/// The abandoned frame's completion gets a fault-observing continuation so a write failure after
/// the caller unwinds never raises an unobserved-task exception (NEXT-04).
/// </remarks>
public async Task WriteAsync(
WorkerEnvelope envelope,
@@ -162,7 +164,9 @@ public sealed class WorkerFrameWriter
/// awaited completions as its <see cref="WorkerFrameProtocolException"/>; the remaining frames are
/// still observed so none faults unobserved. Cancellation while waiting for the lock tombstones
/// every still-unclaimed frame in the batch, per the WRK-22 contract on
/// <see cref="WriteAsync(WorkerEnvelope, WorkerFrameWritePriority, CancellationToken)"/>.
/// <see cref="WriteAsync(WorkerEnvelope, WorkerFrameWritePriority, CancellationToken)"/>; frames
/// the cancelled caller abandons (claimed mid-write, or already faulted) get a fault-observing
/// continuation so a later write failure never raises an unobserved-task exception (NEXT-04).
/// </remarks>
public async Task WriteBatchAsync(
IReadOnlyList<WorkerEnvelope> envelopes,
@@ -245,6 +249,8 @@ public sealed class WorkerFrameWriter
frame.Completion.TrySetCanceled(cancellationToken);
}
}
ObserveAbandonedFault(frame);
}
private void TombstoneUnclaimed(PendingFrame[] frames, CancellationToken cancellationToken)
@@ -259,6 +265,30 @@ public sealed class WorkerFrameWriter
}
}
}
foreach (PendingFrame frame in frames)
{
ObserveAbandonedFault(frame);
}
}
/// <summary>
/// Observes any fault on a frame the cancelled caller stops awaiting (NEXT-04). A frame
/// claimed by a draining lock-holder — or already faulted by a concurrent
/// <c>FailAllQueued</c> — completes on a task nobody awaits after cancellation unwinds the
/// caller; a later write failure would then surface as an unobserved-task exception. A
/// cancelled task never triggers the faulted continuation, so attaching unconditionally is
/// safe. Attached outside <c>_gate</c> because an already-faulted task runs the
/// continuation inline.
/// </summary>
/// <param name="frame">Frame whose completion may fault without an awaiter.</param>
private static void ObserveAbandonedFault(PendingFrame frame)
{
_ = frame.Completion.Task.ContinueWith(
task => _ = task.Exception,
CancellationToken.None,
TaskContinuationOptions.OnlyOnFaulted | TaskContinuationOptions.ExecuteSynchronously,
TaskScheduler.Default);
}
// Runs only under _writeLock. Drains control frames before event frames, stamping and writing each.