chore(followups): reviewer-recommended tests, comments, and hardening from the remediation reviews

The remediation reviews approved every task but left a tail of small notes.
This lands the gateway-side half of them.

Hardening (behavior changes, all narrow):

- BuildFilteredWriteBulkCommand's unreachable default case failed OPEN: a
  fifth bulk-write kind added upstream without a filter case here would have
  shipped the DENIED entries to the worker while reporting them denied to the
  caller. It now throws UnreachableException.
- SqliteCanonicalAuditStore.ListRecentAsync no longer throws on a row it
  cannot date. The retention sweep deliberately preserves such rows (SQLite's
  datetime() yields NULL, so the DELETE never matches), which guaranteed the
  dashboard's recent-audit view would meet one eventually and lose the whole
  page to it. The row is now reported at DateTimeOffset.MinValue with every
  other column intact, behind an optional logger.
- The audit drain loop's finally now completes the channel writer alongside
  detaching the drain, so a producer that raced past the attached check takes
  the write-through branch instead of stranding its event in a buffer nobody
  reads until shutdown. TryComplete is idempotent, so StopAsync is unaffected.

Tests:

- MapCommandReply ownership (Assert.Same on the inner reply), mirroring the
  existing MapEvent ownership test.
- Redactor key-id length boundary at exactly 64 and 65 characters, pinning
  which way it fails. Nothing validates key-id length at creation, so
  docs/Diagnostics.md's "which no issued key id does" is now stated as the
  heuristic it is.
- ApiKeyFailureLimiter.Reset with a PartitionResolution whose partition was
  evicted between the Check and the Reset: inert, and clears nobody else's
  block.
- Constraint-cache concurrency stress: the cap is enforced by the inserting
  thread, so overshoot must be transient and proportional to the in-flight
  inserters, and the cache must settle at or under the cap.
- ListRecentAsync against a raw-SQL undateable row.

Comment/doc accuracy:

- EventsHubViewerRegistry.ReleaseConnection records that it relies on
  SignalR's default sequential per-connection dispatch
  (MaximumParallelInvocationsPerClient = 1).
- A PERF(followup) note on Invoke's double session resolve and why removing it
  needs a SessionManager overload.
- SessionEventDistributor: the volatile-field comment named the pump as the
  lock-free reader, but the pump's single capture point is inside _replayLock;
  the genuinely lock-free reader is SubscriberCount. OnSubscriberOverflow's
  "cannot be observed here" now excepts the DisposeAsync abandon path. The
  churn test names its ConcurrentDictionary bucket-order assumption and that a
  violation surfaces as a read timeout, not a silent pass.
- The two "restores the sequential drain's behavior" claims (SessionManager,
  docs/Sessions.md) were wrong: the sequential drain leaked too, because
  KillWorkerAsync's entry ThrowIfCancellationRequested aborted the whole loop
  on the first session for zero kills. Reworded to "fixes a leak the
  sequential drain also had", with the sweep-bound/shutdown-unbound
  ParallelOptions asymmetry explained.
- ISessionManager.ShutdownAsync's token doc: it degrades the drain to a kill
  sweep rather than cancelling it, with the bounded overrun stated.
  SessionShutdownHostedService.StopAsync records that its cancellation-logging
  branch is now unreachable.
This commit is contained in:
Joseph Doherty
2026-08-15 17:54:31 -04:00
parent 7755745f2f
commit dc2df628e3
19 changed files with 293 additions and 25 deletions
@@ -79,7 +79,14 @@ public interface ISessionManager
CancellationToken cancellationToken);
/// <summary>Shuts down all sessions and the session manager.</summary>
/// <param name="cancellationToken">Token to cancel the asynchronous operation.</param>
/// <param name="cancellationToken">
/// Token that <em>degrades</em> the drain rather than cancelling it. It is passed only to each
/// session's graceful close; the drain loop and the kill fallback are not bound to it, so
/// cancelling turns the drain into a kill sweep instead of abandoning the untried sessions as
/// leaked workers. The call therefore overruns a cancelled token by a bounded amount —
/// roughly <c>ceil(sessionCount / 4)</c> batches of the worker shutdown timeout in the worst
/// case, where 4 is <c>MaxParallelSessionCloses</c>.
/// </param>
/// <returns>A task that represents the asynchronous operation.</returns>
Task ShutdownAsync(CancellationToken cancellationToken);
}
@@ -112,8 +112,11 @@ public sealed class SessionEventDistributor : IAsyncDisposable
// inside the _lifecycleLock section of every register/unregister; never mutated in
// place, so the pump can walk the array it captured with no lock and no allocation.
// Volatile.Write / Volatile.Read ORDER the access — they keep the publishing store from
// sinking past the lock release and keep the pump's read from being hoisted out of the
// fan-out loop. They do NOT promise freshness, and nothing here needs them to: a reader
// sinking past the lock release, and they keep a lock-free reader's load from being hoisted
// or cached. The pump is NOT that reader: its single capture point sits inside the
// _replayLock section of AppendToReplayBufferAndCaptureSubscribers, so the lock edge already
// orders it. The genuinely lock-free reader is SubscriberCount, which loads the field with no
// lock at all. They do NOT promise freshness, and nothing here needs them to: a reader
// may legitimately observe the previous array, which IS the documented "late subscribers
// see events after they register" window. Where visibility must be guaranteed — the
// RegisterWithReplay handoff — it comes from the _replayLock edge, not from Volatile.
@@ -700,8 +703,11 @@ public sealed class SessionEventDistributor : IAsyncDisposable
// every path that completes a channel during fan-out (lease disposal via Unregister,
// and this method) removes the subscriber from the set BEFORE completing it, so a
// completed channel implies the subscriber is already gone and RemoveSubscriber
// returns false. (CompleteAllSubscribers completes without removing, but only after
// the pump has left its loop, so it cannot be observed here.)
// returns false. (CompleteAllSubscribers completes without removing, but only after the
// pump has left its loop, so it cannot be observed here — except on the DisposeAsync
// abandon path: a source factory that ignores cancellation past the 5 s shutdown timeout
// leaves the pump fanning while DisposeAsync completes subscribers, so a spurious overflow
// report is possible there. It is harmless, because the session is already being disposed.)
//
// Bailing out on false is what keeps a normal stream ending mid-traffic from emitting
// a bogus EventQueueOverflow metric and — under the default single-subscriber FailFast
@@ -365,11 +365,19 @@ public sealed class SessionManager : ISessionManager
// rather than adopting them).
//
// For the same reason the loop itself is NOT bound to cancellationToken: a cancelled
// ParallelOptions token stops dispatching the remaining sessions entirely, whereas the
// sequential drain this replaced let every remaining session fail its graceful close fast
// and still kill its worker. The token is passed to the graceful close instead, which
// preserves that behavior — a host stop deadline turns the drain into a kill sweep rather
// than into a leak.
// ParallelOptions token stops dispatching the remaining sessions entirely, so a stop
// deadline would leave the untried tail neither closed nor killed. Note this FIXES a leak
// the sequential drain also had rather than restoring its behavior: there the kill fallback
// ran on the caller's cancelled token, and KillWorkerAsync's entry
// ThrowIfCancellationRequested threw out of the loop on the first session — zero kills, not
// "fail fast and still kill". The token is passed to the graceful close only, and the kill
// runs on CancellationToken.None, so a host stop deadline turns the drain into a kill sweep
// rather than into a leak.
//
// The asymmetry with CloseExpiredLeasesAsync (whose ParallelOptions IS token-bound) is
// intentional: that sweep is periodic maintenance whose missed sessions are picked up by
// the next pass and, ultimately, by this drain. This drain is terminal — nothing runs after
// it — so it must not be abandoned partway.
await Parallel.ForEachAsync(
_registry.Snapshot(),
new ParallelOptions { MaxDegreeOfParallelism = MaxParallelSessionCloses },
@@ -16,7 +16,14 @@ public sealed class SessionShutdownHostedService(
return Task.CompletedTask;
}
/// <summary>Shuts down all gateway sessions as the host stops, logging (without throwing) if the host's shutdown timeout cancels the operation first.</summary>
/// <summary>Shuts down all gateway sessions as the host stops.</summary>
/// <remarks>
/// The catch below is now effectively unreachable: <see cref="ISessionManager.ShutdownAsync"/>
/// no longer aborts on the host's shutdown timeout, it degrades to a kill sweep and logs a
/// per-session warning for each session that failed its graceful close. The clause is kept as
/// a cheap guard against that contract regressing, not as an expected path — the operator
/// signal for a timed-out shutdown is now those per-session warnings.
/// </remarks>
/// <param name="cancellationToken">Token that signals the host's shutdown timeout has elapsed.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
public async Task StopAsync(CancellationToken cancellationToken)