The nightly (run #47) portable job failed CreateAsync_WhenFakeWorkerNeverSendsReady
_TimesOutAndKillsWorker: it asserts the factory's TimeoutException carries 'did not
complete startup' (thrown at the session's 1s startup timeout), but the test's
.WaitAsync(TestTimeout=5s) anti-hang net tripped first under CI load and surfaced
.NET's generic 'The operation has timed out.' — same commit passed on the push run
(#46), so it's a load-sensitive timeout race, not a regression. Raising runner
capacity to 4 makes concurrent-load flakes like this more likely.
Fix: give the two CreateAsync failure/timeout tests a dedicated 30s HangGuardTimeout,
far above the 1s semantic timeout under test, so the factory's own exception is
always the one observed. The net still catches a genuine hang.
Note: this class cannot run on macOS (named pipes -> Unix domain sockets fail on
the pipe path); verification is via CI Linux.
CI (portable, run #38) intermittently failed SessionManagerTests
.InvokeAsync_WhenTimeoutZero_FailsFastUnchanged with 'expected immediate
fail-fast but took 155ms'. The fail-fast paths throw synchronously under the
lock (GatewaySession.GetReadyWorkerClientAsync) and never enter the poll loop,
so the absolute <100ms wall-clock bound measured only host load, not behavior,
and flaked under CI contention.
- WhenWorkerFaulted_FailsFastWithBothStates: anchor the bound to a large (5000ms)
ready-wait timeout and assert fail-fast returns in < timeout/3 — a regression
that burned the timeout is still caught, but scheduling jitter can't trip it.
- WhenTimeoutZero_FailsFastUnchanged: drop the wall-clock assertion entirely
(a zero timeout has no wait window to burn); the error code, both-states
message, and InvokeCount == 0 already pin the immediate fail-fast.
Verified: the 3 SessionManager timing tests pass (net10.0).
Resolve all CommentChecker findings across the gateway server, worker, tests,
and .NET client (314 -> 0 real issues): add missing <returns>/<summary>/<param>
on public and test members, convert Stream/interface overrides to <inheritdoc/>,
and remove internal task/issue tracking IDs (SEC-*, IPC-*, WRK-*, GWC-*, TST-*,
Client.Dotnet-*) from shipped code documentation while preserving the design
rationale prose. Shipped comments should not carry internal bookkeeping, and
complete XML docs keep the analyzer/TreatWarningsAsErrors gate and generated API
docs clean. The 6 remaining flags are heuristic false positives (MD5, UTC-4,
capacity-1, near-1601) left intact so real documentation is not corrupted.
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
Make the authored CI pipeline execute and go green on the co-located runner (portable + java), fix the five real latent defects the first real run surfaced (codegen-check null, stale rust proto, orphan-terminator Linux path, stale java worker codegen, py3.12 event loop), provision pwsh + Gradle for the self-hosted act image, and disable the Windows jobs (act host-mode broken). TST-03 -> Done.
https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
Proves the default-on reconnect protocol end to end through the real gRPC
StreamEvents path via the fake worker harness (no live COM). Two facts:
- ReconnectInsideRetainedWindow_ReplaysTailNoGap: capacity 16 retains all
events; reconnect from a mid-batch after_worker_sequence cursor replays
exactly the events with WorkerSequence > cursor (retained tail + events
emitted while detached), strictly ascending, distinct, no ReplayGap.
- ReconnectWithStaleCursor_EmitsReplayGapSentinelFirst: capacity 3 with 6
events forces eviction; reconnect with AfterWorkerSequence=1 yields the
ReplayGap sentinel first (Family=Unspecified, no body, RequestedAfterSequence=1,
OldestAvailableSequence=oldest retained), then the retained tail ascending.
Single-subscriber mode: the first stream is fully detached (cancel + await the
stream task runs EventStreamService's finally, dropping the subscriber count to
0) before reconnect; the distributor + replay ring are created once per session
and survive detach, so events emitted while detached are retained.
macOS note: fake-worker E2E tests need TMPDIR=/tmp (default macOS TMPDIR pushes
the CoreFxPipe Unix-socket path past the 104-byte sun_path limit). Windows CI
is unaffected. Server reconnect/replay behavior matched the contract exactly.
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
SEC-25 (near-term hardening; full per-session EventsHub ACL stays deferred to
roadmap item 12): DashboardEventBroadcaster now redacts tag values from a deep
CLONE of the event before mirroring to SignalR when Dashboard:ShowTagValues is
false (the default). Clears MxEvent.value (covers OnDataChange/OnWriteComplete/
OperationComplete/OnBufferedDataChange — their bodies are empty discriminators,
values ride in the top-level field incl. buffered arrays) and the alarm body's
current_value/limit_value. Source event never mutated (shared with the gRPC
path + replay ring) - verified by a source-not-mutated test. Makes the formerly
dead ShowTagValues flag live for the mirror. EventsHub TODO(per-session-acl)
kept and tied to roadmap item 12. Tests: DashboardEventBroadcasterTests (3).
SEC-30: trim docs/Diagnostics.md to mark opt-in command-value logging as NOT
YET IMPLEMENTED (no LogCommandValues knob, RedactCommandValue has no call
sites, no values logged); wiring deferred pending secured-bulk redaction
coverage (SEC-13). No option/call sites added.
Server build clean (0 warnings); Dashboard tests 152/152.
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
Replace the replay buffer's LinkedList<ReplayEntry> (a node allocation per
retained event on the fan-out hot path) with a preallocated ReplayEntry[] ring
sized to ReplayBufferCapacity, tracked by _replayHead (oldest) + _replayCount.
Appending a retained event now allocates nothing.
Behavior preserved exactly: ascending-WorkerSequence append order; capacity
eviction (overwrite head + advance when full); time trim via
_timeProvider.GetUtcNow() cutoff at the same three call sites; oldest-read for
ReplayGap math; both replay-from-sequence query paths + highest-seen tracking;
same _replayLock. Capacity-0 stays retain-nothing (guarded early return, no
modulo-by-zero).
Server build clean (0 warnings); Distributor/Replay tests 33/33 incl. two new
cases (multi-wrap ring keeps newest in order; capacity-1 overwrite + gap).
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
Behavior-preserving allocation cuts on the per-event/per-command path:
- GWC-06: StreamEvents send-timing uses Stopwatch.GetTimestamp() +
GetElapsedTime() instead of a per-event Stopwatch allocation (same measured
span).
- GWC-07/IPC-05 (event): MapEvent transfers ownership of the inner MxEvent
instead of .Clone()-ing it. Safe: WorkerEvent is parsed fresh per pipe frame
with the distributor pump as its single consumer (GWC-01), MapEvent runs once
before fan-out, and every downstream consumer (subscribers, replay ring)
only READS the event (WorkerSequence is stamped upstream; verified no
post-mapping mutation). Comment documents the invariant + restore-clone
caveat if a second consumer is added.
- IPC-05 (command): CreateCommandEnvelope no longer re-clones; MapCommand
already isolated the graph from the caller-owned gRPC message.
- GWC-15: grpc_stream_queue.depth converts from a per-event push counter to an
ObservableGauge summing registered channel sources at scrape time only
(name/semantics unchanged); removes all per-event .Count/lock work.
Kept every load-bearing isolation clone (MapCommand, Invoke, bulk filters,
MapCommandReply). Server build clean (0 warnings); EventStream/Metrics/
Distributor/Mapper tests 62/62 (incl. formerly-flaky queue-depth tests, now
green under the lazy gauge, + 2 new MapEvent ownership tests). Docs: Metrics.md,
Grpc.md.
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
Gateway server-side named-pipe frame path, wire format byte-identical:
- Writer (GWC-08/IPC-13): serialize once into a single ArrayPool-rented buffer
holding the 4-byte LE length prefix + payload, then one WriteAsync. Removes
the second serialization pass (ToByteArray re-ran CalculateSize), the
separate prefix array + second stream write, and per-frame heap allocation.
- Reader (IPC-14): rent the payload buffer from ArrayPool instead of new byte[]
per frame; read/parse bounded to the real length, return in finally. Matches
the worker side which already pools.
- Correctness: length tracked separately from (larger) pool capacity; every
rented buffer returned exactly once in finally incl. the malformed-protobuf
path; ParseFrom copies into the message so the envelope never aliases the
returned buffer.
Cross-checked LE-prefix framing agrees across gateway+worker writer/reader.
Server build clean (0 warnings); WorkerFrameProtocol tests 10/10 incl. a new
large-payload-near-cap round-trip (forces pool buffer > frame). The 2 failing
WorkerClient/FakeWorkerHarness tests are the known macOS UDS 104-char path
limit, not a regression.
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
Restore the __Host- browser guarantees for the default secure deployment
without breaking plaintext/dev:
- DashboardServiceCollectionExtensions PostConfigure now resolves the cookie
name as: explicit MxGateway:Dashboard:CookieName override wins; else
__Host-MxGatewayDashboard when SecurePolicy==Always (RequireHttpsCookie
true); else the plain MxGatewayDashboard default. Guard: never apply the
__Host- prefix without Secure (browsers silently drop it).
- DashboardAuthenticationDefaults: add SecureCookieName const; keep the plain
CookieName as the non-secure fallback.
- Docs corrected to the actual conditional contract (five stale claims):
gateway.md, GatewayProcessDesign.md, ImplementationPlanGateway.md,
GatewayDashboardDesign.md, CLAUDE.md; GatewayConfiguration.md phrasing
tightened.
- Test: DashboardCookieOptionsTests asserts the name flips with
RequireHttpsCookie and that an explicit override wins.
Server build clean (0 warnings); Dashboard tests 149/149.
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
The read loop awaited EnqueueWorkerEventAsync inline, which blocks in the bounded
event channel's timed WriteAsync (up to EventChannelFullModeTimeout, default 5s)
when the channel is full with no/slow consumer. A WorkerCommandReply or heartbeat
queued behind an event frame was then stalled, so an in-flight InvokeAsync could
hit CommandTimeout even though the worker replied in time.
Mirror the existing outbound WriteLoopAsync: add an unbounded event staging
channel and a dedicated EventWriteLoopAsync. DispatchEnvelope is now fully
synchronous — the WorkerEvent branch hands the event off with a non-blocking
TryWrite and the read loop never awaits. The event write loop owns the timed
WriteAsync into the bounded channel and the sustained-overflow ProtocolViolation
fault (unchanged contract). Registered in WaitForBackgroundTasks + completed on
close/fault/dispose.
Test: reply arriving after events with a full, consumer-less event channel is
dispatched promptly (no CommandTimeout) — pipe-harness, verified on windev.
Docs: GatewayProcessDesign read/write/event-loop section.
Proto foundation + gateway-side of the size/backpressure pass:
- IPC-02: add GatewayHello.max_frame_bytes (regen Generated/); gateway sends
its negotiated worker-frame max in the handshake so the worker can adopt it
instead of a hard-coded default. Worker read-half lands separately.
- IPC-03: give the pipe frame max envelope-overhead headroom above the public
gRPC cap (WorkerFrameProtocolOptions.EnvelopeOverheadReserveBytes = 64 KiB;
default Worker.MaxMessageBytes bumped to 16 MiB + reserve), cross-validate the
headroom at startup, and pre-check command envelope size in WorkerClient so an
oversized command fails only that correlation (ResourceExhausted) instead of
faulting the whole session.
- IPC-04: reject DrainEvents max_events above a public ceiling in the request
validator (worker per-reply cap lands with the worker half).
Docs: GatewayConfiguration, WorkerFrameProtocol, gateway.md.
Tests: headroom validation, DrainEvents bound, oversized-command per-command
failure (pipe-harness, verified on windev).
Full-host tests passed in isolation but failed 2-4/run under parallel execution on
Windows with 'the process cannot access the file ... because it is being used by
another process' (macOS never sees it — Unix deletes open files). Two shared-state
parallel collisions, discovered while verifying TST-08:
1. Self-signed cert generation. GatewayTlsBootstrapTests sets process-global
Kestrel/TLS env vars that GatewayApplication.Build reads; a parallel host-building
test inherits them mid-run and both generate a cert at the same path, racing on the
fixed-name '<path>.tmp'. Fixes:
- SelfSignedCertificateProvider: stage the PFX in a unique '<path>.<guid>.tmp' instead
of a fixed name, so concurrent/interrupted writers never collide on the temp file
(real robustness: two instances or a restart-during-write no longer clash). The
final atomic Move (last-writer-wins) still yields an equivalent cert.
- TestHostEnvironmentInitializer: default MxGateway__Tls__SelfSignedCertPath to a
per-process temp path so the suite never writes the shared ProgramData default or
fights the deployed service; first host-building test generates, the rest load it.
- GatewayTlsBootstrapTests: [Collection] with DisableParallelization so its global
env-var mutation cannot bleed into parallel tests (new GlobalEnvironmentCollection).
2. AuthStoreHealthCheckTests opened a real SQLite file; Microsoft.Data.Sqlite's
connection pool keeps the .db handle open after 'await using', so the finally
File.Delete threw. Reuse the existing TempDatabaseDirectory helper, which calls
SqliteConnection.ClearAllPools() before deleting.
Server build clean; affected classes 17/17 on macOS. Windows full-suite verified separately.
- SEC-02: DashboardAuthorizationHandler restricts the loopback and
Authentication:Mode=Disabled bypasses to read-only. They now satisfy only a
Viewer-bearing requirement (AnyDashboardRole), never AdminOnly, so anonymous
localhost can view the dashboard but cannot reach API-key CRUD or session
Close/Kill at the policy layer (previously guarded only by service re-checks).
- SEC-12: DashboardSessionAdminService emits canonical AuditEvents through
IAuditWriter (actions dashboard-close-session / dashboard-kill-worker, category
SessionAdmin) on Success/Failure/Denied, mirroring the API-key audit path so
destructive session actions leave durable, queryable rows.
- SEC-20: drop the unbounded session_id tag from the exported
mxgateway.heartbeats.failed counter (per-session detail stays in the snapshot/log).
Docs updated same-change: CLAUDE.md (read-only loopback + 5-min bearer),
GatewayDashboardDesign.md (bypass scoping + session-admin audit), Metrics.md.
Server build clean; 30/30 targeted + 295/295 Dashboard/Security/App/Metrics sweep.
SEC-07: add QueryActiveAlarmsRequest -> events:read scope arm; fix two tests that
constructed StreamAlarmsRequest instead of QueryActiveAlarmsRequest.
SEC-05: shorten hub-token lifetime 30m -> 5m; document that the ?access_token= query
carriage must never be request-logged.
SEC-08: gateway-side CachingApiKeyVerifier (short TTL, keyed on a hash of the presented
secret) skips the per-call store read+last_used write; CoalescingMarkApiKeyStore bounds
last_used writes to <=1/key/min; identity constraints are cached. Invalidation is wired
at the gateway admin sites (revoke/rotate/delete); short TTL backstops out-of-process CLI.
SEC-11: fixed-window rate limit on POST /auth/login + a per-peer (key-id) failure limiter
checked before VerifyAsync; new MxGateway:Security options bound + validated.
Also fixes regressions from the SEC-01/04/06 commit (c185f62) that a narrow test filter
missed (all now covered by a full-suite checkpoint):
- Rooting check is cross-platform: accepts Windows C:\/UNC forms on Unix so the shipped
appsettings path validates on the macOS dev box, still rejecting bare filenames.
- AddGatewayConfiguration TryAdds a non-production IHostEnvironment fallback so the validator
resolves in minimal test/tooling containers; the real host + apikey CLI register the actual
environment first (TryAdd no-op there).
- Test assembly defaults ASPNETCORE_ENVIRONMENT=Development (ModuleInitializer) so full-host
tests exercise wiring instead of tripping the SEC-04/06 production guards.
- GatewayOptionsTests asserts the SEC-01 CommonApplicationData-derived default (platform-correct).
archreview: SEC-07/05/08/11 (P1). Verified: NonWindows build clean; full gateway suite
747 passed / 42 failed, where all 42 are the pre-existing macOS named-pipe-harness failures
(Unix-socket path limit) and 0 are validation/regression failures.
Bump all four ZB.MOM.WW.Auth.* package refs 0.1.2 -> 0.1.4. The shared
ApiKeyVerifier now rejects any key whose ExpiresUtc is in the past;
existing keys have NULL expiry (never expire), so nothing breaks, and the
auth SQLite DB auto-migrates to schema v3 (nullable expires_utc column) on
first boot. Implement the two IApiKeyAdminStore members added in 0.1.3
(SetScopesAsync/SetEnabledAsync) on the test FakeApiKeyAdminStore.
Build green; no new test failures (the macOS worker-pipe IPC failures are
pre-existing/environmental, identical to the pre-bump baseline).
Interlocking changes across the gateway server (shared GatewaySession.cs /
SessionManager.cs), committed together:
- GWC-01 (Critical): alarm monitor now attaches as an internal
(non-counted) distributor subscriber instead of a second raw drain of the
single worker event channel; WorkerClient._events -> SingleReader with a
claimed-once guard so a future dual-consumer regression throws loudly.
- GWC-02 (High): faulted sessions are swept in CloseExpiredLeasesAsync
(IsFaultedReapable + FaultedReason); new FaultedGraceSeconds (default 0).
- GWC-03 (High): configurable MaxSparseArrayLength (default 1_000_000)
enforced before allocation.
- TST-02 (High, security): StreamEvents attach now enforces the opening key
id -> PermissionDenied on owner mismatch.
- TST-12 (Medium): CLAUDE.md retention-defaults sentence corrected.
Verified: NonWindows build clean; targeted tests 135/135 on macOS, plus
WorkerClientTests 18/18 on the Windows host.
Resolves Server-059, Tests-041, Server-060 (2026-06-25 re-review).
DashboardSnapshotService memoized the whole Galaxy summary keyed on the cache
Sequence, but the shared library bumps Sequence only on a heavy refresh: the
steady-state tick, the refresh-failure path, and the age-based status getter all
replace the entry via 'previous with { ... }' at the SAME Sequence. The dashboard
therefore froze LastQueriedAt and, during a Galaxy SQL outage, kept showing
Healthy with no error for the whole outage.
Split DashboardGalaxySummaryProjector into ComputeBreakdown (the O(N) template/
category work, the only sequence-bound part) and BuildSummary (cheap volatile
fields copied fresh). ResolveGalaxySummary now memoizes only the breakdown by
Sequence and rebuilds the summary from the current entry each tick. Removed the
redundant DashboardGalaxyProjector wrapper.
Tests: same-sequence status/error/timestamp now reflected (the regression);
memoization-hit and sequence-invalidation guards; GatewayApplicationTests asserts
the DI container resolves IGalaxyBrowseScopeProvider to GatewayBrowseScopeProvider
(pins the registration-order invariant over the library's no-op default).
Add GalaxyRepositoryHostWiringTests.BrowseChildren_BrowseSubtreesConstraintThroughHostWiring_FiltersChildren:
constructs the real GatewayRequestIdentityAccessor + GatewayBrowseScopeProvider, passes the
provider as IGalaxyBrowseScopeProvider to the lib GalaxyRepositoryGrpcService, and asserts
two children (unconstrained) then empty (BrowseSubtrees=["NonExistent"]) — proving the full
production authz-filtering chain is correctly wired.
Strengthen DashboardSnapshotServiceTests.GetSnapshot_ProjectsGalaxySummaryFromHierarchyCache:
add Assert.Equal(2, TopTemplates.Count) and Assert.Contains($Area, InstanceCount==1) so the
test guards the complete summary output, not just the $Pump entry.
- Server-057: extend []-suffix normalization to AddItemBulk/AddBufferedItem so bulk-added
array tags bind write-capable handles (authz check, worker bind, and registration kept
consistent); update gateway.md + client READMEs. Tests: AddItemBulk/AddBufferedItem wiring.
- Server-058: assert []-fallback-resolved bare array names are still denied when out of
read/write scope and that MaxWriteClassification is enforced on suffixed array registrations.
- Contracts-023/024/025: round-trip + field-19 descriptor pin for MxSparseArray; document
MxSparseArray in docs/Contracts.md; enumerate it in the protocol-version-3 test summary.
- Tests-040: add wiring tests for the six uncovered sparse-write arms (WriteSecured, Write2,
WriteSecured2, Write2Bulk, WriteSecuredBulk, WriteSecured2Bulk).
dotnet build + targeted tests green (184 passed).
Guard against proto uint32 total_length values that exceed Array.MaxLength
before casting; the previous checked cast threw OverflowException (gRPC
Internal) instead of the intended InvalidArgument. Adds tests for the new
guard, for the null-value ArgumentNullException path, and removes the
checked keyword (redundant after the guard).
Server-054/055/056, Contracts-020/021/022, Tests-036/038/039,
IntegrationTests-030/031/032 (+033 deferred to live rig),
Client.Dotnet-026/028/029 (+027 won't-fix), Client.Go-030..034,
Client.Python-032..036, Client.Rust-033..038.
Key fix: SessionEventDistributor orphaned a subscriber that registered after
the pump completed but before disposal (Server-056) -> register paths now
complete late registrants under _lifecycleLock; regression test added. The
racy dashboard-mirror gRPC test made deterministic (Tests-039).
Verified green locally: gateway Tests targeted classes (GatewaySession,
SessionEventDistributor, GatewayOptionsValidator, ProtobufContractRoundTrip,
GatewaySessionDashboardMirror) + dotnet/go/python/rust client suites.
- EventStreamService: remove dead per-item sequence guard in the replay
loop (RegisterWithReplay already returns only events > afterSequence)
and correct the comment that falsely claimed a "per-item constraint
filter" is applied; the event stream has no per-event constraint
filtering today.
- SessionEventDistributor.RegisterWithReplay: set oldestAvailableSequence=0
when gap==false so the implementation matches the documented contract
(OldestAvailableSequence is meaningful only when Gap is true).
Update the two RegisterWithReplay tests that asserted the old non-zero
value in the no-gap path.
- RegisterSubscriber: remove stray blank line at method entry.
- SessionEventDistributorTests: add RegisterWithReplay_AfterDispose_
ThrowsObjectDisposedException to pin nested-lock disposal behavior.
- EffectiveSessionConfiguration: add DetachGraceSeconds field; GatewayConfigurationProvider
forwards value.Sessions.DetachGraceSeconds (blocker fix).
- GatewaySession.InvokeAsync and ReadEventsAsync: switch TouchClientActivity calls from
DateTimeOffset.UtcNow to _eventStreaming.TimeProvider.GetUtcNow() so Task 12 fake-clock
control works end-to-end (split-clock fix).
- TOCTOU fix: add TryBeginCloseIfExpired(now, out alreadyClosing) to GatewaySession that
re-checks IsLeaseExpiredCore/IsDetachGraceExpiredCore AND _activeEventSubscriberCount==0
under _syncRoot before transitioning to Closing; CloseExpiredLeasesAsync calls it before
CloseSessionCoreAsync so a reattach that wins the race leaves the session Ready/usable.
- Minors: lease-expiry-takes-precedence comment in CloseExpiredLeasesAsync; TOCTOU comment
block; sweep-cycle latency note added to SessionOptions.DetachGraceSeconds XML doc and to
GatewayConfiguration.md DetachGraceSeconds row.
- New tests: TryBeginCloseIfExpired_ReattachedSubscriberWinsRace_DeclinesClose (GatewaySession),
CloseExpiredLeasesAsync_DoesNotCloseSessionThatReattachedBeforeSweepCloses (SessionManager),
plus IsLeaseExpiredCore/IsDetachGraceExpiredCore private helpers used by the guard.
Replace Task.Delay(100) subscriber-attachment races with WaitForSubscriberCountAsync,
a polling gate on GatewaySession.ActiveEventSubscriberCount so Advise and event fan-out
cannot proceed until all subscribers are confirmed registered.
Fix WaitForMessageCountAsync to honor a single CancellationTokenSource deadline across
the poll loop rather than resetting the timeout on each intermediate wakeup.
Add ordering comment in the cancellation test explaining why stream1Task must be awaited
before AllowNextEvent to guarantee sub1 is unregistered before the 2nd event is fanned.
Assert capException.Status.Detail contains "maximum" in the cap test to distinguish
EventSubscriberLimitReached (AllowMultiple=true cap) from EventSubscriberAlreadyActive
(single-subscriber rejection) — both map to ResourceExhausted.
Extract shared ConfigureCommandReply helper and move FakeWorkerProcess to TestSupport/
so both fake-worker test classes reference one definition.
Adds GatewayEndToEndMultiSubscriberTests covering three scenarios
through the real gRPC StreamEvents path with AllowMultipleEventSubscribers=true:
- Fan-out: two concurrent StreamEvents RPCs both receive every event the fake
worker emits, in the same order (WorkerSequence matches, values indexed).
- Independent cancellation: cancelling one subscriber's stream leaves the other
receiving subsequent events; the session stays usable.
- Cap enforcement: with MaxEventSubscribersPerSession=2 a third concurrent
StreamEvents is rejected with gRPC ResourceExhausted while the first two
keep streaming.
Extends RecordingServerStreamWriter<T> with WaitForMessageCountAsync to
allow deterministic bounded-timeout awaits for an N-message count without
fixed sleeps.
Remove the per-call allowMultipleSubscribers param from AttachEventSubscriber and
derive the mode internally from _eventStreaming.AllowMultipleEventSubscribers — the
same source SessionEventDistributor uses for singleSubscriberMode — so the two can
never structurally diverge. The maxSubscribers cap param is kept because
MaxEventSubscribersPerSession lives in SessionOptions, which the session does not hold
directly (only EventOptions flows through SessionEventStreaming).
Other nits:
- SubscriberCount XML doc clarifies it includes internal subscribers and differs from
GatewaySession.ActiveEventSubscriberCount (external/gRPC only).
- SingleSubscriberMode_LoneExternalOverflow test: add Assert.Equal(1, observedSet) guard
before the value assertion so the test cannot pass vacuously if the handler never fired.
- GatewayOptionsValidator.ValidateSessions: add explanatory code comment documenting why
!AllowMultipleEventSubscribers && MaxEventSubscribersPerSession > 1 is NOT rejected as
a hard error (the default config ships with this combination; the cap is simply unused
in single-subscriber mode, not a behavior bug).
- GatewaySession.DetachEventSubscriber: add Debug.Assert before the clamp so a genuine
double-decrement surfaces in debug builds.
(1) GatewaySession.StartDashboardMirror: publish _dashboardMirrorLease and _dashboardMirrorTask
atomically under one _syncRoot section; if the session is already Closing/Closed/Faulted,
dispose the just-created lease and return without starting the mirror task so nothing is orphaned.
(2) WaitUntilAsync test helper: catch OperationCanceledException and call Assert.Fail with the
timeout duration and predicate source text instead of letting the exception propagate raw.
(3) New SessionEventDistributorTests.InternalSubscriberOverflow_HandlerSeesIsOnlySubscriberFalse:
verifies CountExternalSubscribers excludes the internal subscriber, so isOnlySubscriber==false
even when the internal subscriber is the only registered subscriber.
(4) SubscriberOverflowHandler delegate gains isInternal parameter; overflow metric label is
"dashboard-mirror" for internal subscribers and "grpc-event-stream" for external ones.
(5) DashboardEventBroadcaster.Publish: wrap SendAsync Task acquisition in try/catch so a
synchronous throw cannot escape the never-throw Publish interface contract.
- Issue 1: document the isOnlySubscriber snapshot race-safety assumption in
OnSubscriberOverflow; flags the Task 7/8 revisit point explicitly.
- Issue 2: pin StreamDisconnects==1 in the FailFast overflow test so a
regression dropping the StreamDisconnected("Detached") finally call is caught.
- Issue 3: replace plain int/bool? reads in SlowSubscriberOverflow test with
Volatile.Read/Write + Interlocked.Increment stores to close the C# memory
model data race on overflowCalls and observedIsOnlySubscriber.
- Issue 4: add SlowSubscriberOverflow_WithMultipleSubscribers_... distributor
test pinning that isOnlySubscriber==false disables the session-fault path;
includes TODO(Task 8) note for the GatewaySession-level assertion.
- Issue 5: reword SubscriberOverflowHandler XML doc to make explicit that the
handler must NOT complete the subscriber's channel; the distributor owns that.
Joseph Doherty
dohertj2