xUnit v2 (2.9.3) never invokes IAsyncDisposable.DisposeAsync on a test
class, so the unobserved-fault safety net this class documents — awaiting
every scripted worker task after each test — has been dead code since it
was written. Found via dumpasync on the wedged-testhost investigation: the
NeverReadyWorkerProcessLauncher's Task.Delay(Infinite, _stop.Token) was
still pending (DelayPromiseWithCancellation, detached) minutes after its
test finished, which is only possible if DisposeAsync never ran.
Proven both ways with a throw-probe in DisposeAsync: under IAsyncDisposable
all 3 tests pass (never called); under IAsyncLifetime all 3 fail from the
probe (called after every test). Sweep confirmed this is the only test
class on IAsyncDisposable — all other implementers are helpers disposed
via await using.
The windev full-suite wedge — every test reported, then the x64 testhost sitting
at ~0 CPU forever while `dotnet test` never returns — was one test blocked on a
pipe write, not a leaked thread or an undisposed fixture.
`dotnet-stack report` on the wedged host showed no thread running test code:
xUnit's RunTestsInAssembly was parked on WaitHandle.WaitOne() waiting for the
assembly-finished event, so the wait lived in a suspended async state machine.
`dotnet-dump analyze -c dumpasync` named the frame — WorkerClientTests
.StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout awaiting
WorkerFrameWriter.WriteAsync on a 63-byte frame, with <extra>5__15 = 6, i.e. the
seventh of the twelve events the test pushes past the client's staging bound.
That test faults the worker client on purpose, and a faulted client stops its
read loop by design. The test-side pipe came from the NamedPipeServerStream
overload without buffer arguments, which passes inBufferSize: 0 / outBufferSize: 0
to CreateNamedPipe; on Windows that reserves no buffer at all, so a write
completes only once the peer reads it. Measured on windev, that pipe absorbed
0 bytes against a non-reading peer where the same pipe declared with 64 KiB
buffers absorbed 65 520. On macOS and Linux .NET backs named pipes with Unix
domain sockets whose socket buffer swallows the writes regardless, which is why
the identical test never hung there and the bug read as environmental.
Test-owned server pipes now go through TestSupport/TestNamedPipe.CreateServer in
both test projects, declaring explicit 64 KiB buffers so those tests exercise the
gateway's own staging/queue backpressure rather than the OS pipe's flow control.
Separately, every fake-worker write in WorkerClientTests now goes through
PipePair.WriteAsync, bounded by the class's five-second TestTimeout. That is
where the severity came from: a test method that never returns keeps xUnit from
raising ITestAssemblyFinished, so one unbounded await cost the whole suite its
result. A blocked write is now a named test failure instead of a silent wedge.
The fix also retires a wrong belief the wedge had created. windev reported 855
where macOS reported 879, and that gap was recorded in GatewayTesting.md as
Unix-gated test cases; it was really the results lost when the wedged host was
torn down. The same clone now reports 879 passed, matching macOS exactly.
Product code is unaffected. SessionWorkerClientFactory.CreatePipe keeps the
unbuffered declaration deliberately: both ends run continuous read loops and every
gateway write is bounded by the worker client's _stopCts, so a stalled peer
cancels the write rather than blocking on it.
Verified on windev at this SHA: gateway suite x64 three times (879 passed,
exit 0, no surviving testhost each time) and Worker.Tests x86 twice (400 passed,
11 skipped, exit 0, clean), plus the macOS gateway suite once (879 passed).
Docs: GatewayTesting.md replaces the --blame-hang workaround section with the
root cause and corrects the baseline to 879, CLAUDE.md's Source Update Workflow
no longer tells readers the windev suite wedges, and ToolchainLinks.md records
dotnet-stack and dotnet-dump as installed on windev.
Both failures in the windev baseline were test bugs that reproduce on any Windows
host, not anything missing or misconfigured on windev.
SelfSignedCertificateProviderTests.GenerateCertificate_HasExpectedSansEkuAndValidity
asserted SAN content by substring-matching X509Extension.Format(false). That string
comes from the platform crypto library: Windows' CryptFormatObject renders the IPv6
loopback fully expanded (0000:0000:...:0001) where the managed formatter renders
"::1", so the loopback assertion could never hold on Windows. Decode the extension
with X509SubjectAlternativeNameExtension and compare parsed IPAddress values and DNS
names instead, which removes the platform-dependent formatting from the assertion.
SessionManagerTests.OpenSessionAsync_PipeNameIsShortAndUniquePerPidAndSession guards
the 104-byte macOS sun_path budget NEXT-01 shortened the pipe name to fit. It padded
the measured name up to a five-digit pid but never substituted that worst case
downward, so Windows' routine six-digit pids over-counted by a character against a
budget that does not constrain the host running the test. Substitute the five-digit
macOS worst case for the running pid's digit count so the check measures the name
format rather than the current pid.
EventStreamServiceTests.WaitUntilAsync now reports the unmet condition on timeout
instead of letting a bare TaskCanceledException escape. Its five-second real-clock
deadline is genuinely load-sensitive on windev (36 logical CPUs, maxParallelThreads
-1), and an opaque cancellation there is exactly what got the previous failures
filed as "environmental" and left unexplained.
Documents the windev run in docs/GatewayTesting.md: the two fixed bugs and their root
causes, the real-pipe suites whose failures are evidence of machine load rather than
of the change under test, and the full-suite testhost that completes every test and
then never exits (filtered runs exit normally; macOS exits cleanly). Corrects the
CLAUDE.md claim that the suite exits cleanly on the Windows dev box.
The pipe name mxaccess-gateway-{pid}-session-{32hex} plus .NET's
CoreFxPipe_ prefix overflowed the 104-byte Unix-domain-socket path limit
under the default per-user macOS TMPDIR (~49 chars), so every test that
opened a real pipe threw ArgumentOutOfRangeException at pipe creation
unless TMPDIR=/tmp was exported. Rename to mxgw-{pid}-{sessionUid} (the
session guid hex without the session- prefix; worst-case 43 chars) and
shorten the three test-fixture names the same way. Uniqueness is
unchanged: gateway pid + full session guid. The worker receives the pipe
name via its launch command line, so mixed Server/Worker deploy SHAs are
unaffected. Docs updated in the same change (gateway.md,
GatewayProcessDesign, GatewayConfiguration, Sessions, CLAUDE.md); new
regression test pins the format and the length budget.
Verified: SessionManagerTests 39/39; SessionWorkerClientFactory,
GatewayEndToEndFakeWorkerSmoke, WorkerClient, and ReconnectReplay suites
33/33 under the default macOS TMPDIR — this also retires the
previously-misdiagnosed 'macOS pipe-timeout test failures': they were
this path-length throw, not a timeout-message defect.
Same-commit docs rule (were missed in the prior commit):
- docs/GalaxyRepository.md: SnapshotCachePath now documents the per-OS derived
default and the GalaxyRepositoryOptionsValidator rooting/validity enforcement.
- A2-galaxyrepository-adoption-handoff.md: correct the now-inaccurate NSSM caveat
(SnapshotCachePath override is optional, not required; blank seeds a rooted host
default, no silent no-op) and repoint the option-validation item at the new
GalaxyRepositoryOptionsValidator.
SEC-34 guard confirmed and documented: TryParseKeyId's '_' split cannot truncate a
key id because both — and the only — gateway key-creation paths
(ApiKeyAdminCommandLineParser.IsValidKeyId, DashboardApiKeyManagementService.ValidateKeyId)
restrict key ids to IsAsciiLetterOrDigit || '.' || '-', and key ids are never
library-generated. Added a citing comment; no behavior change.
Test consolidation: moved the three host-start SqlitePath overrides into
TestHostEnvironmentInitializer (per-process temp store, mirroring Secrets__SqlitePath)
so future host-start tests auto-cover.
SEC-33: make rooting host-meaningful and stop shipping foreign-platform literals.
- Delete IsRootedForAnyPlatform; AddIfNotRooted now uses Path.IsPathRooted (current OS).
- Promote AddIfNotRooted/AddIfInvalidPath to shared GatewayConfigPathRules so the new
Galaxy validator reuses them and the two validators cannot drift.
- Remove Authentication:SqlitePath and Galaxy:SnapshotCachePath Windows literals from
appsettings.json; the CommonApplicationData-derived code defaults take over. The
Galaxy default is seeded as a configuration value before AddZbGalaxyRepository
(SnapshotCachePath is init-only, so a PostConfigure mutation cannot compile).
- New GalaxyRepositoryOptionsValidator (ValidateOnStart) enforces a valid, host-rooted
SnapshotCachePath when PersistSnapshot is true.
- Root-cause the stray junk-named auth DB: host start eagerly builds
AuthSqliteConnectionFactory; under the non-rooted Windows literal on macOS SQLite
wrote it relative to the test bin CWD. The three real-host-start tests now pin
SqlitePath to a temp path.
SEC-34: verification cache Invalidate-vs-in-flight-repopulation race closed with a
per-key generation counter (bump-before-evict, snapshot-then-recheck). The expiry
cap (window 2) takes the documented fallback: the library verification identity
carries no ExpiresUtc, so the cache cannot cap at the key's expiry (donor-library ask).
GWC-24 rider: cap MxGateway:Events:QueueCapacity at int.MaxValue/2 so the derived
checked(2 * EventChannelCapacity) in WorkerClient cannot overflow at session creation.
SEC-35 (doc-only): note IsProduction() env-name semantics in GatewayConfiguration.md.
Docs updated same commit (GatewayConfiguration.md, Authentication.md) and tracking
registers/change-log flipped (00-tracking.md, 40-security-dashboard.md).
ReadAsync allocated a fresh 4-byte array per inbound frame; the GWC-08
pass pooled the payload buffer but left the prefix. Replaced with a
per-instance scratch field — the reader is single-consumer by
construction (one read loop per WorkerClient, handshake reads complete
before the loop starts), so a per-instance buffer is safe and the
non-reentrancy that makes it safe is now stated on the class. Pooling
four bytes via ArrayPool would cost more than the allocation it saves.
Tests: WorkerFrameProtocolTests.ReadAsync_WithMultipleFramesOnOneReader_
ParsesEveryFrame reads five frames of differing payload length through
one reader, so a stale prefix carried between calls would misparse.
Invoke deep-cloned the whole MxCommandRequest — including its command
payload, potentially a large bulk-write graph — only to overwrite the
cloned command with commandToInvoke and discard it. MapCommand then did
the one clone actually needed. Net cost: a full wasted command deep-clone
per Invoke, worst for exactly the bulk writes that are largest.
Adds a MapCommand(MxCommand) overload (MapCommand reads nothing else off
the request) and has Invoke pass commandToInvoke directly; the request
overload delegates so other callers are untouched.
The remaining clone inside MapCommand stays and is now documented as
required rather than incidental: commandToInvoke may be the gRPC-owned
request.Command, and the caller reads it again after dispatch via
TrackCommandReply, so ownership transfer (à la GWC-07) is not safe here.
That clone is what keeps WorkerClient.CreateCommandEnvelope's no-aliasing
invariant true.
Tests: MxAccessGrpcMapperTests.MapCommandFromCommandClonesPayload
(mutating the input leaves the mapped command untouched; both overloads
produce equal results under a fixed TimeProvider).
CreateEnvelope stamped Sequence with an interlocked increment when the
envelope was built, so two concurrent InvokeAsync callers could take 1
and 2 and then enqueue in the order 2, 1 — non-monotonic on the wire,
breaking gateway.md's "monotonic per sender" contract. Benign today
(neither side validates inbound sequence, old GWC-10 still open) but it
would fault healthy sessions the moment worker-side validation lands.
WriteLoopAsync now stamps immediately before _writer.WriteAsync. It is
the outbound channel's single consumer (SingleReader = true), so wire
order and stamp order are the same thing by construction and
_nextSequence drops to a plain ulong with no interlocking. This mirrors
the worker's WRK-04 fix, which the gateway half never received.
Also adds TST-28: a [Theory] pinning that GatewayHello.MaxFrameBytes
carries the configured worker-frame maximum (default + 2 MiB override).
The adoption half is asserted only in the Windows-only worker suite, so
a regression to sending 0 — "older gateway, use default" to the worker —
would silently downgrade the negotiated IPC-02 limit with every CI test
still green. Mutation-checked (hard-coded 0 fails both cases).
Tests: WorkerClientTests.ConcurrentInvokesEmitStrictlyIncreasingSequences
OnTheWire (32 parallel invokes; failed 3/3 pre-fix) and
.StartAsync_SendsGatewayHelloWithConfiguredMaxFrameBytes.
An empty replay ring reported oldest_available_sequence = 0 even when gap was
true. Clients follow the documented after_worker_sequence = oldest - 1 formula,
so an unsigned client computed ulong.MaxValue: the follow-up resume replayed
nothing, reported no gap, and the live filter dropped every subsequent event —
a silently dead stream in the headline detach-and-resume scenario, reachable on
default config once ReplayRetentionSeconds (300) age-evicts the ring.
GWC-25: SessionEventDistributor.RegisterWithReplay's empty-ring branch now
reports _highestSequenceSeen + 1 — the next sequence that can possibly be
delivered — when gap is true, so oldest - 1 lands exactly on the highest
observed sequence and the resume delivers everything newer. Still 0 when there
is no gap, where the field is meaningless and never emitted. Nothing is lost:
the evicted interval was unrecoverable either way, and the sentinel's job is to
say "re-snapshot".
CLI-35: the Python CLI fed every stream item into MessageToDict, which raised on
the ReplayGap dataclass and aborted the command after consuming the stream. A
new _event_row helper renders a gap as {"replayGap": {...}} — the same camelCase
shape the Rust CLI emits — and leaves proto events on the existing path.
CLI-36: the Go CLI formatted result.Event on every row, but the library
deliberately clears Event on a gap, so text mode printed
"0 MX_EVENT_FAMILY_UNSPECIFIED" and JSON mode an empty object, discarding the
resume cursors. The loop now branches on result.IsReplayGap() and renders the
typed row in both modes, counting it toward -limit like any other row. The JSON
row's cursors are typed by hand rather than marshalled with protojson: the
proto3 JSON mapping renders 64-bit integers as strings ("7") while the Rust and
Python CLIs emit numbers (7), so going through protojson would have made Go the
only canonical CLI with a different value type.
Docs in the same change: docs/Sessions.md documents the empty-ring sentinel
value and that oldest - 1 is the universal resume formula in both the retained
and fully-evicted cases; docs/CrossLanguageSmokeMatrix.md gains a per-CLI
gap-rendering table covering both client findings, and records exactly what is
and is not comparable across CLIs (same keys and numeric cursors for Rust/Go/
Python; quoted cursors for .NET/Java; differing key order, whitespace, and
container), so a matrix runner compares parsed values rather than raw bytes.
Tests, all written red first and each reproducing its defect verbatim:
- SessionEventDistributorTests: RegisterWithReplayReportsNextDeliverableSequence
WhenRingEmptiedByAge, ...WithRetentionDisabled, and
ResumeUsingSentinelFormulaAfterEmptyRingGapDeliversLiveEvents.
- GatewayEndToEndReconnectReplayTests.ReconnectAfterFullAgeEvictionResumesWith
SentinelFormula — fake-worker e2e resume walk on a fake clock; the fixture now
takes a retention window and a TimeProvider.
- clients/python test_stream_events_renders_replay_gap.
- clients/go TestRunStreamEventsPrintsReplayGap.
GWC-25's ReplayGap.oldest_available_sequence proto-comment amendment is
deliberately deferred to the later codegen wave (see the tracker change log): it
is comment-only but triggers the full five-client regen fan-out.
RunMonitorAsync issued SubscribeAlarms and the first reconcile before the
internal distributor subscriber was attached (via ISessionManager
.ReadAlarmEventsAsync). The pump has been running since MarkReady started the
dashboard mirror and only fans to subscribers registered at fan-out time, so
every transition raised in that two-round-trip window bypassed the alarm feed —
and a missed Acknowledge was never repaired, because ApplyReconcile broadcast
presence deltas only.
- The monitor now takes the internal lease directly from its session BEFORE
SubscribeAlarms and drains it after the first reconcile; window transitions
buffer in the lease's bounded channel. Processing them after ApplyReconcile is
order-safe (ApplyTransition handles alarms the snapshot already placed).
- ISessionManager.ReadAlarmEventsAsync removed — zero remaining callers.
- ApplyReconcile broadcasts an Acknowledge feed transition when a both-present
alarm's state advanced to ActiveAcked. This is a feed-level repair on the
AlarmFeedMessage/StreamAlarms surface rebuilt from the worker's own snapshot,
not MxEvent emission, so the "never synthesize events" rule is untouched;
the reasoning is recorded on ApplyReconcile.
The alarm-monitor test fakes now hand the monitor a real Ready GatewaySession
with a dashboard mirror, which is what makes the window reproducible.
Docs: docs/Sessions.md and gateway.md alarm-monitor ordering notes.
Refs: archreview/2026-07-12/remediation/10-gateway-core.md GWC-26
The GWC-04 remediation decoupled the read loop from event backpressure by
staging events into an unbounded channel, so its TryWrite always succeeded and
the only overflow fault was a single timed WriteAsync exceeding
EventChannelFullModeTimeout. A consumer draining slower than the worker
produces — each individual write still completing inside the window — therefore
grew gateway memory without bound, without a fault, and without a metric: the
queue-depth gauge counted only the bounded consumer channel, so staged events
were invisible.
Bound _eventStaging at 2 x EventChannelCapacity (Wait, single reader/writer, no
synchronous continuations). A rejected staging TryWrite is the sustained
slow-drain signal and faults the client ProtocolViolation with
QueueOverflow("worker-event-staging"), guarded by IsTerminalState() so a
completed channel during shutdown stays a silent drop. SetFaulted is
non-blocking, so the read loop still never awaits behind events. The timed-write
fault is unchanged and still catches the full-stall case earlier.
Move the queue-depth increment from EnqueueWorkerEventAsync to StageWorkerEvent
so the single counter reports total undelivered events (staged + queued); the
decrement at consumer read was already correct. No new configuration key: the
bound is derived, and gateway-side buffering per session is now at most
3 x MxGateway:Events:QueueCapacity. Coordination with still-open GWC-21
(EventChannelFullModeTimeout configurability) remains open and was not blocked
on.
Tests: StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout (5-min
full-mode timeout so only the staging bound can fire; asserts an interleaved
command reply still completes) and WorkerEventQueueDepthGaugeCountsStagedEvents.
Docs updated in the same change: GatewayProcessDesign, MxAccessWorkerInstanceDesign,
GatewayConfiguration, Metrics. GWC-24 flipped to Done in both trackers.
AttachInternalEventSubscriber ran EnsureDistributorCreated / Register /
StartPumpIfRequested with no state check, unlike AttachEventSubscriber. A
premature attach would start the pump against a not-yet-Ready worker; the pump
source throws SessionNotReady, PumpAsync completes every subscriber with that
error and latches the distributor, and _eventDistributorStarted is never reset —
so the session would reach Ready with permanently dead event streaming.
Mirror AttachEventSubscriber's gate: check _state/_workerClient.State under
_syncRoot and throw SessionManagerException(SessionNotReady) before the
distributor is created, keeping the distributor calls outside the lock.
Refs: archreview/2026-07-12/remediation/10-gateway-core.md GWC-27
The nightly (run #47) portable job failed CreateAsync_WhenFakeWorkerNeverSendsReady
_TimesOutAndKillsWorker: it asserts the factory's TimeoutException carries 'did not
complete startup' (thrown at the session's 1s startup timeout), but the test's
.WaitAsync(TestTimeout=5s) anti-hang net tripped first under CI load and surfaced
.NET's generic 'The operation has timed out.' — same commit passed on the push run
(#46), so it's a load-sensitive timeout race, not a regression. Raising runner
capacity to 4 makes concurrent-load flakes like this more likely.
Fix: give the two CreateAsync failure/timeout tests a dedicated 30s HangGuardTimeout,
far above the 1s semantic timeout under test, so the factory's own exception is
always the one observed. The net still catches a genuine hang.
Note: this class cannot run on macOS (named pipes -> Unix domain sockets fail on
the pipe path); verification is via CI Linux.
CI (portable, run #38) intermittently failed SessionManagerTests
.InvokeAsync_WhenTimeoutZero_FailsFastUnchanged with 'expected immediate
fail-fast but took 155ms'. The fail-fast paths throw synchronously under the
lock (GatewaySession.GetReadyWorkerClientAsync) and never enter the poll loop,
so the absolute <100ms wall-clock bound measured only host load, not behavior,
and flaked under CI contention.
- WhenWorkerFaulted_FailsFastWithBothStates: anchor the bound to a large (5000ms)
ready-wait timeout and assert fail-fast returns in < timeout/3 — a regression
that burned the timeout is still caught, but scheduling jitter can't trip it.
- WhenTimeoutZero_FailsFastUnchanged: drop the wall-clock assertion entirely
(a zero timeout has no wait window to burn); the error code, both-states
message, and InvokeCount == 0 already pin the immediate fail-fast.
Verified: the 3 SessionManager timing tests pass (net10.0).
Resolve all CommentChecker findings across the gateway server, worker, tests,
and .NET client (314 -> 0 real issues): add missing <returns>/<summary>/<param>
on public and test members, convert Stream/interface overrides to <inheritdoc/>,
and remove internal task/issue tracking IDs (SEC-*, IPC-*, WRK-*, GWC-*, TST-*,
Client.Dotnet-*) from shipped code documentation while preserving the design
rationale prose. Shipped comments should not carry internal bookkeeping, and
complete XML docs keep the analyzer/TreatWarningsAsErrors gate and generated API
docs clean. The 6 remaining flags are heuristic false positives (MD5, UTC-4,
capacity-1, near-1601) left intact so real documentation is not corrupted.
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
Make the authored CI pipeline execute and go green on the co-located runner (portable + java), fix the five real latent defects the first real run surfaced (codegen-check null, stale rust proto, orphan-terminator Linux path, stale java worker codegen, py3.12 event loop), provision pwsh + Gradle for the self-hosted act image, and disable the Windows jobs (act host-mode broken). TST-03 -> Done.
https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
Proves the default-on reconnect protocol end to end through the real gRPC
StreamEvents path via the fake worker harness (no live COM). Two facts:
- ReconnectInsideRetainedWindow_ReplaysTailNoGap: capacity 16 retains all
events; reconnect from a mid-batch after_worker_sequence cursor replays
exactly the events with WorkerSequence > cursor (retained tail + events
emitted while detached), strictly ascending, distinct, no ReplayGap.
- ReconnectWithStaleCursor_EmitsReplayGapSentinelFirst: capacity 3 with 6
events forces eviction; reconnect with AfterWorkerSequence=1 yields the
ReplayGap sentinel first (Family=Unspecified, no body, RequestedAfterSequence=1,
OldestAvailableSequence=oldest retained), then the retained tail ascending.
Single-subscriber mode: the first stream is fully detached (cancel + await the
stream task runs EventStreamService's finally, dropping the subscriber count to
0) before reconnect; the distributor + replay ring are created once per session
and survive detach, so events emitted while detached are retained.
macOS note: fake-worker E2E tests need TMPDIR=/tmp (default macOS TMPDIR pushes
the CoreFxPipe Unix-socket path past the 104-byte sun_path limit). Windows CI
is unaffected. Server reconnect/replay behavior matched the contract exactly.
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
SEC-25 (near-term hardening; full per-session EventsHub ACL stays deferred to
roadmap item 12): DashboardEventBroadcaster now redacts tag values from a deep
CLONE of the event before mirroring to SignalR when Dashboard:ShowTagValues is
false (the default). Clears MxEvent.value (covers OnDataChange/OnWriteComplete/
OperationComplete/OnBufferedDataChange — their bodies are empty discriminators,
values ride in the top-level field incl. buffered arrays) and the alarm body's
current_value/limit_value. Source event never mutated (shared with the gRPC
path + replay ring) - verified by a source-not-mutated test. Makes the formerly
dead ShowTagValues flag live for the mirror. EventsHub TODO(per-session-acl)
kept and tied to roadmap item 12. Tests: DashboardEventBroadcasterTests (3).
SEC-30: trim docs/Diagnostics.md to mark opt-in command-value logging as NOT
YET IMPLEMENTED (no LogCommandValues knob, RedactCommandValue has no call
sites, no values logged); wiring deferred pending secured-bulk redaction
coverage (SEC-13). No option/call sites added.
Server build clean (0 warnings); Dashboard tests 152/152.
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
Replace the replay buffer's LinkedList<ReplayEntry> (a node allocation per
retained event on the fan-out hot path) with a preallocated ReplayEntry[] ring
sized to ReplayBufferCapacity, tracked by _replayHead (oldest) + _replayCount.
Appending a retained event now allocates nothing.
Behavior preserved exactly: ascending-WorkerSequence append order; capacity
eviction (overwrite head + advance when full); time trim via
_timeProvider.GetUtcNow() cutoff at the same three call sites; oldest-read for
ReplayGap math; both replay-from-sequence query paths + highest-seen tracking;
same _replayLock. Capacity-0 stays retain-nothing (guarded early return, no
modulo-by-zero).
Server build clean (0 warnings); Distributor/Replay tests 33/33 incl. two new
cases (multi-wrap ring keeps newest in order; capacity-1 overwrite + gap).
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
Behavior-preserving allocation cuts on the per-event/per-command path:
- GWC-06: StreamEvents send-timing uses Stopwatch.GetTimestamp() +
GetElapsedTime() instead of a per-event Stopwatch allocation (same measured
span).
- GWC-07/IPC-05 (event): MapEvent transfers ownership of the inner MxEvent
instead of .Clone()-ing it. Safe: WorkerEvent is parsed fresh per pipe frame
with the distributor pump as its single consumer (GWC-01), MapEvent runs once
before fan-out, and every downstream consumer (subscribers, replay ring)
only READS the event (WorkerSequence is stamped upstream; verified no
post-mapping mutation). Comment documents the invariant + restore-clone
caveat if a second consumer is added.
- IPC-05 (command): CreateCommandEnvelope no longer re-clones; MapCommand
already isolated the graph from the caller-owned gRPC message.
- GWC-15: grpc_stream_queue.depth converts from a per-event push counter to an
ObservableGauge summing registered channel sources at scrape time only
(name/semantics unchanged); removes all per-event .Count/lock work.
Kept every load-bearing isolation clone (MapCommand, Invoke, bulk filters,
MapCommandReply). Server build clean (0 warnings); EventStream/Metrics/
Distributor/Mapper tests 62/62 (incl. formerly-flaky queue-depth tests, now
green under the lazy gauge, + 2 new MapEvent ownership tests). Docs: Metrics.md,
Grpc.md.
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
Gateway server-side named-pipe frame path, wire format byte-identical:
- Writer (GWC-08/IPC-13): serialize once into a single ArrayPool-rented buffer
holding the 4-byte LE length prefix + payload, then one WriteAsync. Removes
the second serialization pass (ToByteArray re-ran CalculateSize), the
separate prefix array + second stream write, and per-frame heap allocation.
- Reader (IPC-14): rent the payload buffer from ArrayPool instead of new byte[]
per frame; read/parse bounded to the real length, return in finally. Matches
the worker side which already pools.
- Correctness: length tracked separately from (larger) pool capacity; every
rented buffer returned exactly once in finally incl. the malformed-protobuf
path; ParseFrom copies into the message so the envelope never aliases the
returned buffer.
Cross-checked LE-prefix framing agrees across gateway+worker writer/reader.
Server build clean (0 warnings); WorkerFrameProtocol tests 10/10 incl. a new
large-payload-near-cap round-trip (forces pool buffer > frame). The 2 failing
WorkerClient/FakeWorkerHarness tests are the known macOS UDS 104-char path
limit, not a regression.
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
Restore the __Host- browser guarantees for the default secure deployment
without breaking plaintext/dev:
- DashboardServiceCollectionExtensions PostConfigure now resolves the cookie
name as: explicit MxGateway:Dashboard:CookieName override wins; else
__Host-MxGatewayDashboard when SecurePolicy==Always (RequireHttpsCookie
true); else the plain MxGatewayDashboard default. Guard: never apply the
__Host- prefix without Secure (browsers silently drop it).
- DashboardAuthenticationDefaults: add SecureCookieName const; keep the plain
CookieName as the non-secure fallback.
- Docs corrected to the actual conditional contract (five stale claims):
gateway.md, GatewayProcessDesign.md, ImplementationPlanGateway.md,
GatewayDashboardDesign.md, CLAUDE.md; GatewayConfiguration.md phrasing
tightened.
- Test: DashboardCookieOptionsTests asserts the name flips with
RequireHttpsCookie and that an explicit override wins.
Server build clean (0 warnings); Dashboard tests 149/149.
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
The read loop awaited EnqueueWorkerEventAsync inline, which blocks in the bounded
event channel's timed WriteAsync (up to EventChannelFullModeTimeout, default 5s)
when the channel is full with no/slow consumer. A WorkerCommandReply or heartbeat
queued behind an event frame was then stalled, so an in-flight InvokeAsync could
hit CommandTimeout even though the worker replied in time.
Mirror the existing outbound WriteLoopAsync: add an unbounded event staging
channel and a dedicated EventWriteLoopAsync. DispatchEnvelope is now fully
synchronous — the WorkerEvent branch hands the event off with a non-blocking
TryWrite and the read loop never awaits. The event write loop owns the timed
WriteAsync into the bounded channel and the sustained-overflow ProtocolViolation
fault (unchanged contract). Registered in WaitForBackgroundTasks + completed on
close/fault/dispose.
Test: reply arriving after events with a full, consumer-less event channel is
dispatched promptly (no CommandTimeout) — pipe-harness, verified on windev.
Docs: GatewayProcessDesign read/write/event-loop section.
Proto foundation + gateway-side of the size/backpressure pass:
- IPC-02: add GatewayHello.max_frame_bytes (regen Generated/); gateway sends
its negotiated worker-frame max in the handshake so the worker can adopt it
instead of a hard-coded default. Worker read-half lands separately.
- IPC-03: give the pipe frame max envelope-overhead headroom above the public
gRPC cap (WorkerFrameProtocolOptions.EnvelopeOverheadReserveBytes = 64 KiB;
default Worker.MaxMessageBytes bumped to 16 MiB + reserve), cross-validate the
headroom at startup, and pre-check command envelope size in WorkerClient so an
oversized command fails only that correlation (ResourceExhausted) instead of
faulting the whole session.
- IPC-04: reject DrainEvents max_events above a public ceiling in the request
validator (worker per-reply cap lands with the worker half).
Docs: GatewayConfiguration, WorkerFrameProtocol, gateway.md.
Tests: headroom validation, DrainEvents bound, oversized-command per-command
failure (pipe-harness, verified on windev).
Full-host tests passed in isolation but failed 2-4/run under parallel execution on
Windows with 'the process cannot access the file ... because it is being used by
another process' (macOS never sees it — Unix deletes open files). Two shared-state
parallel collisions, discovered while verifying TST-08:
1. Self-signed cert generation. GatewayTlsBootstrapTests sets process-global
Kestrel/TLS env vars that GatewayApplication.Build reads; a parallel host-building
test inherits them mid-run and both generate a cert at the same path, racing on the
fixed-name '<path>.tmp'. Fixes:
- SelfSignedCertificateProvider: stage the PFX in a unique '<path>.<guid>.tmp' instead
of a fixed name, so concurrent/interrupted writers never collide on the temp file
(real robustness: two instances or a restart-during-write no longer clash). The
final atomic Move (last-writer-wins) still yields an equivalent cert.
- TestHostEnvironmentInitializer: default MxGateway__Tls__SelfSignedCertPath to a
per-process temp path so the suite never writes the shared ProgramData default or
fights the deployed service; first host-building test generates, the rest load it.
- GatewayTlsBootstrapTests: [Collection] with DisableParallelization so its global
env-var mutation cannot bleed into parallel tests (new GlobalEnvironmentCollection).
2. AuthStoreHealthCheckTests opened a real SQLite file; Microsoft.Data.Sqlite's
connection pool keeps the .db handle open after 'await using', so the finally
File.Delete threw. Reuse the existing TempDatabaseDirectory helper, which calls
SqliteConnection.ClearAllPools() before deleting.
Server build clean; affected classes 17/17 on macOS. Windows full-suite verified separately.
- SEC-02: DashboardAuthorizationHandler restricts the loopback and
Authentication:Mode=Disabled bypasses to read-only. They now satisfy only a
Viewer-bearing requirement (AnyDashboardRole), never AdminOnly, so anonymous
localhost can view the dashboard but cannot reach API-key CRUD or session
Close/Kill at the policy layer (previously guarded only by service re-checks).
- SEC-12: DashboardSessionAdminService emits canonical AuditEvents through
IAuditWriter (actions dashboard-close-session / dashboard-kill-worker, category
SessionAdmin) on Success/Failure/Denied, mirroring the API-key audit path so
destructive session actions leave durable, queryable rows.
- SEC-20: drop the unbounded session_id tag from the exported
mxgateway.heartbeats.failed counter (per-session detail stays in the snapshot/log).
Docs updated same-change: CLAUDE.md (read-only loopback + 5-min bearer),
GatewayDashboardDesign.md (bypass scoping + session-admin audit), Metrics.md.
Server build clean; 30/30 targeted + 295/295 Dashboard/Security/App/Metrics sweep.
SEC-07: add QueryActiveAlarmsRequest -> events:read scope arm; fix two tests that
constructed StreamAlarmsRequest instead of QueryActiveAlarmsRequest.
SEC-05: shorten hub-token lifetime 30m -> 5m; document that the ?access_token= query
carriage must never be request-logged.
SEC-08: gateway-side CachingApiKeyVerifier (short TTL, keyed on a hash of the presented
secret) skips the per-call store read+last_used write; CoalescingMarkApiKeyStore bounds
last_used writes to <=1/key/min; identity constraints are cached. Invalidation is wired
at the gateway admin sites (revoke/rotate/delete); short TTL backstops out-of-process CLI.
SEC-11: fixed-window rate limit on POST /auth/login + a per-peer (key-id) failure limiter
checked before VerifyAsync; new MxGateway:Security options bound + validated.
Also fixes regressions from the SEC-01/04/06 commit (c185f62) that a narrow test filter
missed (all now covered by a full-suite checkpoint):
- Rooting check is cross-platform: accepts Windows C:\/UNC forms on Unix so the shipped
appsettings path validates on the macOS dev box, still rejecting bare filenames.
- AddGatewayConfiguration TryAdds a non-production IHostEnvironment fallback so the validator
resolves in minimal test/tooling containers; the real host + apikey CLI register the actual
environment first (TryAdd no-op there).
- Test assembly defaults ASPNETCORE_ENVIRONMENT=Development (ModuleInitializer) so full-host
tests exercise wiring instead of tripping the SEC-04/06 production guards.
- GatewayOptionsTests asserts the SEC-01 CommonApplicationData-derived default (platform-correct).
archreview: SEC-07/05/08/11 (P1). Verified: NonWindows build clean; full gateway suite
747 passed / 42 failed, where all 42 are the pre-existing macOS named-pipe-harness failures
(Unix-socket path limit) and 0 are validation/regression failures.
Bump all four ZB.MOM.WW.Auth.* package refs 0.1.2 -> 0.1.4. The shared
ApiKeyVerifier now rejects any key whose ExpiresUtc is in the past;
existing keys have NULL expiry (never expire), so nothing breaks, and the
auth SQLite DB auto-migrates to schema v3 (nullable expires_utc column) on
first boot. Implement the two IApiKeyAdminStore members added in 0.1.3
(SetScopesAsync/SetEnabledAsync) on the test FakeApiKeyAdminStore.
Build green; no new test failures (the macOS worker-pipe IPC failures are
pre-existing/environmental, identical to the pre-bump baseline).
Interlocking changes across the gateway server (shared GatewaySession.cs /
SessionManager.cs), committed together:
- GWC-01 (Critical): alarm monitor now attaches as an internal
(non-counted) distributor subscriber instead of a second raw drain of the
single worker event channel; WorkerClient._events -> SingleReader with a
claimed-once guard so a future dual-consumer regression throws loudly.
- GWC-02 (High): faulted sessions are swept in CloseExpiredLeasesAsync
(IsFaultedReapable + FaultedReason); new FaultedGraceSeconds (default 0).
- GWC-03 (High): configurable MaxSparseArrayLength (default 1_000_000)
enforced before allocation.
- TST-02 (High, security): StreamEvents attach now enforces the opening key
id -> PermissionDenied on owner mismatch.
- TST-12 (Medium): CLAUDE.md retention-defaults sentence corrected.
Verified: NonWindows build clean; targeted tests 135/135 on macOS, plus
WorkerClientTests 18/18 on the Windows host.
Resolves Server-059, Tests-041, Server-060 (2026-06-25 re-review).
DashboardSnapshotService memoized the whole Galaxy summary keyed on the cache
Sequence, but the shared library bumps Sequence only on a heavy refresh: the
steady-state tick, the refresh-failure path, and the age-based status getter all
replace the entry via 'previous with { ... }' at the SAME Sequence. The dashboard
therefore froze LastQueriedAt and, during a Galaxy SQL outage, kept showing
Healthy with no error for the whole outage.
Split DashboardGalaxySummaryProjector into ComputeBreakdown (the O(N) template/
category work, the only sequence-bound part) and BuildSummary (cheap volatile
fields copied fresh). ResolveGalaxySummary now memoizes only the breakdown by
Sequence and rebuilds the summary from the current entry each tick. Removed the
redundant DashboardGalaxyProjector wrapper.
Tests: same-sequence status/error/timestamp now reflected (the regression);
memoization-hit and sequence-invalidation guards; GatewayApplicationTests asserts
the DI container resolves IGalaxyBrowseScopeProvider to GatewayBrowseScopeProvider
(pins the registration-order invariant over the library's no-op default).
Add GalaxyRepositoryHostWiringTests.BrowseChildren_BrowseSubtreesConstraintThroughHostWiring_FiltersChildren:
constructs the real GatewayRequestIdentityAccessor + GatewayBrowseScopeProvider, passes the
provider as IGalaxyBrowseScopeProvider to the lib GalaxyRepositoryGrpcService, and asserts
two children (unconstrained) then empty (BrowseSubtrees=["NonExistent"]) — proving the full
production authz-filtering chain is correctly wired.
Strengthen DashboardSnapshotServiceTests.GetSnapshot_ProjectsGalaxySummaryFromHierarchyCache:
add Assert.Equal(2, TopTemplates.Count) and Assert.Contains($Area, InstanceCount==1) so the
test guards the complete summary output, not just the $Pump entry.
- Server-057: extend []-suffix normalization to AddItemBulk/AddBufferedItem so bulk-added
array tags bind write-capable handles (authz check, worker bind, and registration kept
consistent); update gateway.md + client READMEs. Tests: AddItemBulk/AddBufferedItem wiring.
- Server-058: assert []-fallback-resolved bare array names are still denied when out of
read/write scope and that MaxWriteClassification is enforced on suffixed array registrations.
- Contracts-023/024/025: round-trip + field-19 descriptor pin for MxSparseArray; document
MxSparseArray in docs/Contracts.md; enumerate it in the protocol-version-3 test summary.
- Tests-040: add wiring tests for the six uncovered sparse-write arms (WriteSecured, Write2,
WriteSecured2, Write2Bulk, WriteSecuredBulk, WriteSecured2Bulk).
dotnet build + targeted tests green (184 passed).
Guard against proto uint32 total_length values that exceed Array.MaxLength
before casting; the previous checked cast threw OverflowException (gRPC
Internal) instead of the intended InvalidArgument. Adds tests for the new
guard, for the null-value ArgumentNullException path, and removes the
checked keyword (redundant after the guard).
Server-054/055/056, Contracts-020/021/022, Tests-036/038/039,
IntegrationTests-030/031/032 (+033 deferred to live rig),
Client.Dotnet-026/028/029 (+027 won't-fix), Client.Go-030..034,
Client.Python-032..036, Client.Rust-033..038.
Key fix: SessionEventDistributor orphaned a subscriber that registered after
the pump completed but before disposal (Server-056) -> register paths now
complete late registrants under _lifecycleLock; regression test added. The
racy dashboard-mirror gRPC test made deterministic (Tests-039).
Verified green locally: gateway Tests targeted classes (GatewaySession,
SessionEventDistributor, GatewayOptionsValidator, ProtobufContractRoundTrip,
GatewaySessionDashboardMirror) + dotnet/go/python/rust client suites.
Joseph Doherty
dohertj2