Commit Graph

5 Commits

Author SHA1 Message Date
Joseph Doherty 8653960ea3 fix(galaxy): #865 review round 4 - replay fails on a dead stream, backoff-aware wait
- ReplayAsync fails when its restarted pump's stream has already ended, so
  the supervisor backs off and stays Degraded instead of going Healthy over
  a dead stream (whose next advise re-report restarted at attempt 1 with no
  backoff - a tight reconnect loop).
- ReconnectSupervisor.AttemptInProgress (reopen through replay); advises
  wait for a stale pump's replacement only while an attempt is running, not
  for a whole backoff (the state still reads Reopening/Replaying there).
- ReconnectSupervisor: _disposed is set under the state lock and re-checked
  inside ReportTransportFailure, so a late report cannot start a recovery
  loop that StopAsync already missed.
- EventPump.HasEnded is a flag set before StreamRequested is released, and
  a stream that ends/faults on its first MoveNext is marked ended first.
- SubscriptionRegistry fast path: a miss now falls through to the locked
  path (lookup + session + window check atomically) instead of a lock-free
  re-lookup.
- Tests: replay over an immediately-ending stream fails; AttemptInProgress
  across attempt/backoff; ReportTransportFailure after stop; the
  reopen-window test uses a generous wait bound.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:07:26 -05:00
Joseph Doherty 1c1705fa77 fix(galaxy): #865 review round 3 - budget-safe stream wait, dead pump, fast-path drop
- StreamRequestWait 1 s (under Tier-A Read 2 s / Subscribe 5 s budgets) and
  a stale pump is only waited on while the supervisor is reopening or
  replaying (or a restart is underway); in a reconnect backoff the advise
  proceeds at once. The no-stream log is a rate-limited Warning while
  Healthy, Debug otherwise.
- A current-session pump whose loop has ended is not a stream: if the
  supervisor already reports Healthy (its fault was swallowed while it was
  recovering) the failure is re-reported to start a new cycle.
- Subscribe/Read no longer start the pump directly (only
  AwaitCurrentEventStreamAsync does, honouring the replay restart flag);
  the teardown refusal is an explicit ThrowIfRuntimeStopping.
- SubscriptionRegistry.ResolveOrPark fast path: the handle lookup and the
  open-window check were not atomic, so a subscribe that registered the
  handle and closed its window in between got its initial value dropped as
  stale (found by a stress run: the first event of a bulk advise lost). It
  re-looks-up before declaring stale. Deterministic test via a seam.
- Tests: first Read on a fresh driver, replay of a data subscription.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:00:19 -05:00
Joseph Doherty 9060ff5561 fix(galaxy): #865 review round 2 - wait for the replay's pump, never race a second stream
Round 1 replaced a stale-session pump from any advise path and disposed the
old one in the background; that could start a second stream while the old
one still held the gateway's single-subscriber slot, and a kept
current-session pump could be dead with no fault re-reported.

- EnsureEventPumpStarted reuses again; only ReplayAsync restarts the pump.
  RestartEventPumpForReplay keeps a current-session pump only while its loop
  is running (EventPump.HasEnded), and flags the restart so no advise path
  starts a competing pump in the dispose gap.
- AwaitCurrentEventStreamAsync (bounded by StreamRequestWait) starts a pump
  only when none exists, WAITS for the replay while the pump is stale or
  restarting, waits for StreamRequested, and returns the session it verified.
  SubscribeAsync (per #861 attempt), read-via-subscribe and the probe hook
  issue and fence the advise under that session, so a reconnect after the
  check is caught as a straddle and retried.
- Read: a deadline (not caller) cancel during the wait falls through to
  per-ref results.
- Tests: reopen-window probe test now asserts the advise waits for the
  replay's single new stream; first Subscribe/Read wait for the stream; the
  fencing tests that advise in the window use a short wait bound; the
  production-path seed test uses an empty galaxy (a real probe advise now
  starts a stream against the fake endpoint).

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:46:15 -05:00
Joseph Doherty cc1a7ec687 fix(galaxy): #865 review round 1 - stale-session pump, data-path stream wait
- EnsureEventPumpStarted replaces a pump created for an older registry
  session (the reopen -> replay window still held the dead session's pump,
  so an advise there lost its initial value and the replay skipped it);
  RestartEventPumpForReplay keeps a pump already on the current session.
- SubscribeAsync, read-via-subscribe and ReplayAsync also await the
  bounded StreamRequested wait before advising, not just the probes.
- TryEnsureEventPumpStarted tolerates a teardown flipping _runtimeStopping
  mid-check; the probe hook honours an already-cancelled seed token.
- StreamRequested is documented as narrowing, not closing, the attach race.
- Tests: reopen-window probe test, EventPump.StreamRequested tests, and the
  FixRound1 fresh-pump assertion now waits on a positive control.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:31:51 -05:00
Joseph Doherty e35f2a4e93 fix(galaxy): probe advises start the event stream first so initial ScanState arrives (#865)
The gateway's StreamEvents with after_worker_sequence = 0 is a fresh stream
and never replays, and the event pump started only on the first
Read/Subscribe, so platform ScanState probes advised before then lost their
initial value (and a driver with no authored tags never got any host status).

PerPlatformProbeWatcher now awaits an ensure-event-stream hook before every
probe SubscribeBulk (seed, discover sync, reconnect ResubscribeStaleAsync).
The driver's hook starts the pump (refused while a teardown is in progress)
and waits, bounded, on the new EventPump.StreamRequested signal - completed
once the stream loop's first MoveNext has issued the StreamEvents RPC, or
when the loop ends - so the advise never precedes the stream request.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 12:24:08 -05:00