fix(alarms): replacement drainer starts on a fresh stack; honest bounds in the latch comments

This commit is contained in:
Joseph Doherty
2026-08-16 05:57:05 -04:00
parent cd010c7165
commit c8a7014388
2 changed files with 150 additions and 7 deletions
@@ -623,12 +623,37 @@ public sealed class ScriptedAlarmEngine : IDisposable
// carrying dies with it — ReevaluateAsync's own catch is what normally makes
// that a logged non-event — but ids that queued BEHIND it are still in the
// set and would otherwise sit there until some later push happened to arrive,
// so hand the latch straight to a replacement drainer instead of releasing
// it. Handing over under the same lock keeps "exactly one drainer" true with
// no window a push could start a second one in. This cannot spin: every pass
// swaps the set out and clears it before evaluating, so a replacement finds
// work only if fresh pushes arrived, and a permanently-throwing evaluation
// costs one pass per push — the same rate as a healthy one.
// so hand the latch straight to a replacement drainer instead of releasing it.
//
// Exactly-one-drainer survives the handover because _drainScheduled never
// transiently CLEARS during one: deciding to hand over and clearing the flag
// are the same guarded read-modify-write, and the clear happens only in the
// branch that starts no replacement. The spawn itself is deliberately OUTSIDE
// the lock (nothing it does belongs under _dirtyLock) and is safe there
// precisely because the flag is still set — a push landing in that window only
// grows the set, which the replacement then drains.
//
// Two bounds, both honest:
// * CPU — every pass swaps the set out and clears it before evaluating, so a
// replacement finds work only if fresh pushes arrived. A permanently
// throwing evaluation therefore costs one pass per push, the same rate as
// a healthy one; it cannot spin against an idle upstream.
// * Stack — the replacement is QUEUED (Task.Run), not called. A replacement
// that is called runs INLINE for as long as it does not truly suspend, and
// its own escape then unwinds back into this same finally block, nesting a
// frame per throw with no ceiling — a StackOverflowException no catch can
// recover from. Whether it suspends first is not a property this method
// controls: _evalGate is already free when the escape is thrown (released
// in ReevaluateAsync's inner finally, before the outer catch's logging
// call), so a failure raised BEFORE the first awaited work — the clock
// read, anything synchronous ahead of the predicate — unwinds with no
// suspension at all. A failure raised at or after the predicate does not,
// because TimedScriptEvaluator hops to the pool on every evaluation;
// measured, that path holds depth flat even when called directly. Queuing
// makes the fresh stack unconditional instead of dependent on which throw
// shape happens to occur, at the price of one pool dispatch per faulted
// pass. Pinned by
// Sustained_synchronous_drain_failures_hand_over_without_deepening_the_stack.
if (!released)
{
bool restart;
@@ -638,7 +663,7 @@ public sealed class ScriptedAlarmEngine : IDisposable
if (!restart) _drainScheduled = false;
}
if (restart) TrackBackgroundTask(DrainDirtyAsync(ct));
if (restart) TrackBackgroundTask(Task.Run(() => DrainDirtyAsync(ct)));
}
}
}