Round 1 replaced a stale-session pump from any advise path and disposed the
old one in the background; that could start a second stream while the old
one still held the gateway's single-subscriber slot, and a kept
current-session pump could be dead with no fault re-reported.
- EnsureEventPumpStarted reuses again; only ReplayAsync restarts the pump.
RestartEventPumpForReplay keeps a current-session pump only while its loop
is running (EventPump.HasEnded), and flags the restart so no advise path
starts a competing pump in the dispose gap.
- AwaitCurrentEventStreamAsync (bounded by StreamRequestWait) starts a pump
only when none exists, WAITS for the replay while the pump is stale or
restarting, waits for StreamRequested, and returns the session it verified.
SubscribeAsync (per #861 attempt), read-via-subscribe and the probe hook
issue and fence the advise under that session, so a reconnect after the
check is caught as a straddle and retried.
- Read: a deadline (not caller) cancel during the wait falls through to
per-ref results.
- Tests: reopen-window probe test now asserts the advise waits for the
replay's single new stream; first Subscribe/Read wait for the stream; the
fencing tests that advise in the window use a short wait bound; the
production-path seed test uses an empty galaxy (a real probe advise now
starts a stream against the fake endpoint).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- EnsureEventPumpStarted replaces a pump created for an older registry
session (the reopen -> replay window still held the dead session's pump,
so an advise there lost its initial value and the replay skipped it);
RestartEventPumpForReplay keeps a pump already on the current session.
- SubscribeAsync, read-via-subscribe and ReplayAsync also await the
bounded StreamRequested wait before advising, not just the probes.
- TryEnsureEventPumpStarted tolerates a teardown flipping _runtimeStopping
mid-check; the probe hook honours an already-cancelled seed token.
- StreamRequested is documented as narrowing, not closing, the attach race.
- Tests: reopen-window probe test, EventPump.StreamRequested tests, and the
FixRound1 fresh-pump assertion now waits on a positive control.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
The gateway's StreamEvents with after_worker_sequence = 0 is a fresh stream
and never replays, and the event pump started only on the first
Read/Subscribe, so platform ScanState probes advised before then lost their
initial value (and a driver with no authored tags never got any host status).
PerPlatformProbeWatcher now awaits an ensure-event-stream hook before every
probe SubscribeBulk (seed, discover sync, reconnect ResubscribeStaleAsync).
The driver's hook starts the pump (refused while a teardown is in progress)
and waits, bounded, on the new EventPump.StreamRequested signal - completed
once the stream loop's first MoveNext has issued the StreamEvents RPC, or
when the loop ends - so the advise never precedes the stream request.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Under heavy parallel load (14 concurrent test processes + 14 CPU burners, with
DOTNET_ThreadPool_ForceMinWorkerThreads=1 as the reproducer) the Galaxy tests
with 1-3 s fixed waits failed 16/70 runs. Every failure was a background loop
started with Task.Run (alarm feed, DeployWatcher, EventPump, ReconnectSupervisor)
not being SCHEDULED within the ceiling - e.g. DeployWatcher's stream had not even
been opened after 2 s. No production race: the code under test was checked for
each (alarm feed dispatch is synchronous in-order, DeployWatcher/ReconnectSupervisor
state is lock- or loop-local, subscribe registers before returning).
- New shared TestWait (UntilAsync / ForAsync / HealthyAsync): 30 s presence
ceiling, failure reports elapsed ms + an observed-state snapshot (#856 shape;
GalaxyDriverFixRound1Tests.WaitUntilAsync now delegates to it).
- Converted the short waits in GatewayGalaxyAlarmFeedTests, DeployWatcherTests,
GalaxyDriverSubscribeTests, GalaxyDriverRawPathKeyingTests (its WaitForAsync
never threw on timeout), EventPumpStreamFaultTests, EventPumpReplayGapTests,
ReconnectSupervisorTests; FixRound2 InitBudget 2 s -> 10 s (the parked fetch
never completes, so any finite bound keeps the verdict).
- Absence by short sleep replaced with positive controls: DeployWatcher bootstrap
and same-deploy-time (follow with a real change, assert exactly that one raise -
verified red when suppression is removed), subscribe failed-tag / unsubscribed
handle (follow with a live handle's event), faulted pump (wait for its
onStreamFault instead of 100 ms).
- Fixed sleeps that raced scheduling: subscribe-once read tests wait for the
read's registration (TrackedSubscriptionCountForTests) instead of 50 ms;
supervisor Dispose test waits for reopen entry instead of 50 ms; the
already-Healthy WaitForHealthy test asserts synchronous completion instead of a
wall-clock bound. FakeSubscriber.Map is now a ConcurrentDictionary.
Loaded run (14 parallel x 5, 14 burners, ForceMinWorkerThreads=1):
before 16/70 failed runs; after 0/350 (70 + 210 + 70). Without the thread-pool
override: 0/70 before and after on this host. Serial: 472 passed, 8 skipped.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
PerPlatformProbeWatcher records a platform's handle and publishes the probe
binding into the driver registry as one step under _syncLock, but
WatchedPlatforms read the handle map lock-free - so it could report a
platform one step ahead of the registry Upsert. A ScanState event raised in
that window was dispatched by the EventPump, missed ResolveSubscribers and
was dropped as "stale after unsubscribe", leaving the host status empty.
That is the #856 flake: A_fetch_completing_after_init_populates_host_statuses
waited for WatchedPlatforms, emitted ScanState, and timed out at 10 s on the
final Running wait. Reproduced 11/700 (28 parallel runners x 25 + 14 CPU
burners), every failure at that step with the binding present by the time of
diagnosis and hostStatuses=[] - i.e. the event was dropped. 0/1400 after.
WatchedPlatforms now reads under _syncLock (test seam only; the dispatch path
already checks the map after the registry, the safe order). New
PerPlatformProbeWatcherVisibilityTests spins a reader on another thread and
asserts the registry entry exists the moment the platform is visible
(old code: torn in 1691/2000 iterations). WaitUntilAsync gains a describe
overload; the #856 test's waits now report elapsed time and the observed
seed/binding/pump/host-status chain on timeout.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Awaited deadline oracles get the runner token; sync-over-async tests made async;
nullable child models asserted via ShouldNotBeNull() return values; obsolete SDK
ctors replaced with the telemetry overloads; dead locals/fixture and the redundant
DI PackageReference removed. No suppressions.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Review fix round 2 for lane F:
- N1: the init platform seed no longer runs inline under the init token. Galaxy's Tier-A
Initialize budget is 5 s, and a cold gateway DiscoverHierarchy can spend all of it, so the
OperationCanceledException rethrow failed init, opened the Initialize breaker and cost a full
rebuild per retry. InitializeAsync now only resolves the platform source (no network) and
queues a background seed tied to a per-runtime CTS that the teardown cancels. A result that
lands after its runtime was torn down is discarded.
- N2: the seed worker serves the runtime that is live when each pass STARTS, so a request made
under a new runtime while a stale pass is in flight always gets a pass of its own.
- N3: SubscribeAsync and the subscribe-once read capture the runtime generation before
SubscribeBulk and register only if it is unchanged, atomically with the teardown's registry
Clear (_registryGate). On a mismatch they release the handles and fail the call. Re-subscribe
Rebind uses the same gate, and the production rebuild clears the registry unconditionally.
- PerPlatformProbeWatcher re-checks its stop after the SubscribeBulk await, atomically with
the record and publish. A pass that straddles StopAsync releases the handles it acquired
instead of writing the probe entry back into a cleared registry.
Refs #735
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
Review fix round 1 for lane F:
- The ScanState probe watcher is seeded at InitializeAsync from a platform-only
DiscoverHierarchy query (IGalaxyPlatformSource / GatewayGalaxyPlatformSource over the owned
repository client) and re-seeded on every deploy-watcher rediscovery. The runtime never
calls DiscoverAsync on a live driver, so per-platform host status was unreachable in
production. A failed fetch logs a Warning and never fails init; re-seeds coalesce so a
deploy during an in-flight pass still gets a pass that starts after it.
- EnsureEventPumpStarted refuses while the runtime is stopping, BEFORE reusing the live pump
(that pump belongs to the dying subscriber), and the production rebuild disposes any stale
pump before wiring the new subscriber.
- TeardownRuntimeAsync closes the MX session under the caller's token linked with a 5 s
SessionCloseDeadline, so a blackholed gateway cannot hold DriverInstanceActor.PostStop for 30 s.
- A runtime generation counter, bumped by every teardown, is re-checked before each Rebind
in ResubscribeAllAsync, so a replay-gap re-snapshot that straddles a teardown cannot
resurrect handles into the cleared registry.
Refs #735
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT