fix(scripting): soft notice resumes the caller on the dispatcher; a late notice still cancels and counts (#827 review F1, F2)

F1: PendingEval.SoftExpired runs on the pipe reader and completed the
caller's outcome inline (production outcomes have no async continuations, by
design, because results resume on the ResultDispatcher). The caller's code
(the runner's Cancel write, the seam's publish) therefore ran on the reader,
which could hold up heartbeats (a false silence kill) and, with blocking I/O,
write to the pipe from the reader. The soft outcome is now posted to the
dispatcher like a result. Test: a 1.5 s continuation on a soft timeout with a
300 ms silence limit resumes on ScriptWorker[t].Results and the worker lives
(before: it resumed on the Reader).

F2: after a transport timeout ("worker unresponsive"), a late soft notice
armed G but nobody sent Cancel, so a polling script could not end itself and
the whole worker was killed at notice + G with a kill strike. PendingEval now
exposes LateSoftExpiry (asynchronous continuations, cancelled when the run
answers); the runner sends the Cancel and counts the timeout
(NoteTimeout) from it. A genuinely slow script whose notice is delayed past
T + G is thus still a timeout. Test: stall past T + G, thaw, a polling
script overruns T: the worker survives G and the key is quarantined for
timeouts, not a hard deadline (before: the worker was killed).
This commit is contained in:
Joseph Doherty
2026-11-11 02:21:45 -05:00
parent 30c6c3a0bf
commit c5dffbd8b8
3 changed files with 122 additions and 7 deletions
@@ -591,10 +591,22 @@ public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable,
{
// lmxopcua#827: T + G passed with neither a result nor a soft-deadline notice. The worker runs the soft
// clock from the run's start, so this is the pipe or a starved / stalled worker — the runner's fault,
// never the script's: no timeout strike, no Cancel (the run may not even have started), and the slot
// stays with the run until the worker answers. The seam re-evaluates when it does (or when the worker
// is replaced, if the heartbeat monitor kills it).
TransportTimeoutCounter.Add(1, new KeyValuePair<string, object?>("kind", KindTag(invocation.Kind)));
// not (yet) the script's: no timeout counted, no Cancel (the run may not even have started), and the
// slot stays with the run until the worker answers. The seam re-evaluates when it does (or when the
// worker is replaced, if the heartbeat monitor kills it).
TransportTimeoutCounter.Add(1, new KeyValuePair<string, object?>("kind", KindTag(invocation.Kind)),
new KeyValuePair<string, object?>("answer", "stalled"));
// …unless the worker then reports the script itself over T (slow AND late — a notice delayed past
// T + G by the watchdog poll and the pipe lands here too): it still gets its Cancel, so a polling
// script ends itself instead of costing the whole worker at notice + G, and it counts as a timeout
// (#827 review F2). Off the reader: the completion source runs this asynchronously.
var timeout = invocation.Timeout;
pending.LateSoftExpiry?.ContinueWith(_ =>
{
connection.Cancel(pending);
NoteTimeout(key, state, timeout);
}, CancellationToken.None, TaskContinuationOptions.OnlyOnRanToCompletion | TaskContinuationOptions.ExecuteSynchronously,
TaskScheduler.Default);
// Read by a refusal only while it is incomplete, so a write that loses a race with the slot release
// is harmless.
Volatile.Write(ref state.StalledRun, pending.Completion);