fix(scripting): out-of-process soft deadline measures the script's own run, not the pipe or a starved worker (#827)
The soft deadline T was armed on the host at submit, so the pipe round trip
and the time a starved worker waits to be scheduled counted against the
script: under load a trivial script reported 'script timed out', in Shadow
mode that was a script mismatch, and at T + G the host killed the worker and
struck the key.
- Protocol (additive, negotiated): Hello/Ready.worker_soft_clock and
EvalResults.soft_expired_eval_ids. protocol_version stays 1; a worker that
does not echo the flag keeps the pre-#827 host clock.
- Worker: the watchdog runs each run's soft clock from the moment an
evaluation thread starts it and reports the crossing (woken early for
budgets shorter than its 50 ms poll). It still never cancels by itself;
its fail-fast is anchored at the soft notice + G + margin.
- Host (PendingEval): the soft notice answers the caller and arms the kill G
later. The host's own timer becomes a transport deadline, T + G from submit
(restarted at a deferred run's start); passing it with no result and no
notice yields TransportTimedOut -> Unavailable 'script worker unresponsive'
(BadResourceUnavailable, recovery = the run's completion), no strike, no
kill, no Cancel. A re-trigger refused by that still-unanswered run is
Unavailable too, not 'previous run still running'. Counted as
otopcua.script.ipc.timeouts{kind}.
- Frame writer: coalescing now carries soft_expired_eval_ids (it silently
dropped them); the host processes soft notices before results in a frame.
- Shadow: unresponsive is shadow_unavailable (Unavailable), never a mismatch.
Tests: WorkerSoftClockTests suspends the real worker (SIGSTOP via libc kill /
NtSuspendProcess) - a stall past T still returns 7; a stall past T + G is a
runner fault with nothing struck; a late-starting runaway is still timed out
at T from its start and killed G after; Shadow stays match /
shadow_unavailable; both clocks enforce T and G; PendingEval state machine.
This commit is contained in:
@@ -41,12 +41,17 @@ message Hello {
|
||||
string pe_cache_dir = 8; // the node's compiled-PE cache (design §7); empty = no cache
|
||||
bytes pe_cache_key = 9; // its HMAC key; the worker loads only entries that verify under it
|
||||
uint32 compile_threads = 10; // dedicated compile threads; 0 = the worker's default (min(8, max(1, CPUs/2)))
|
||||
// lmxopcua#827: the host asks the worker to run each evaluation's SOFT clock itself, from when an evaluation thread
|
||||
// starts the run, and to report the crossing in EvalResults.soft_expired_eval_ids. The host then keeps only a
|
||||
// transport deadline of its own (see EvalResults). Honoured only when the worker echoes Ready.worker_soft_clock.
|
||||
bool worker_soft_clock = 11;
|
||||
}
|
||||
|
||||
message Ready {
|
||||
string nonce = 1;
|
||||
int32 pid = 2;
|
||||
string runtime_version = 3;
|
||||
bool worker_soft_clock = 4; // lmxopcua#827: this worker runs the soft clock (Hello.worker_soft_clock was set)
|
||||
}
|
||||
|
||||
enum ContextKind {
|
||||
@@ -141,10 +146,20 @@ message EvalResult {
|
||||
// when the evaluation appears in `started_eval_ids`, which an evaluation thread sends alongside the result of the run it
|
||||
// just finished. So a run waiting behind a sibling is never timed out, struck or killed for the sibling's time.
|
||||
// Either list may be the only content of a frame. A host that predates these fields ignores them.
|
||||
//
|
||||
// Worker soft clock (lmxopcua#827, negotiated by Hello/Ready.worker_soft_clock): the soft deadline T measures the
|
||||
// script's OWN execution — the worker starts it when an evaluation thread starts the run and reports the crossing in
|
||||
// `soft_expired_eval_ids` (it does not cancel the run itself: the host answers its caller first, then sends Cancel,
|
||||
// keeping the in-process ordering). The host arms the hard deadline G from that notice. What the host keeps on its own
|
||||
// clock is a TRANSPORT deadline, T + G from submit (restarted at a deferred run's start): no result and no soft notice
|
||||
// by then means the pipe or the worker process is late — the caller is answered as a runner fault ("script worker
|
||||
// unresponsive", BadResourceUnavailable), never as a script timeout, and nothing is struck. A starved or stalled worker
|
||||
// therefore cannot make a trivial script look over budget; a hung one is still caught by the heartbeat silence kill.
|
||||
message EvalResults {
|
||||
repeated EvalResult items = 1;
|
||||
repeated uint64 started_eval_ids = 2;
|
||||
repeated uint64 deferred_eval_ids = 3;
|
||||
repeated uint64 soft_expired_eval_ids = 4;
|
||||
}
|
||||
|
||||
message ScriptError {
|
||||
|
||||
@@ -237,6 +237,7 @@ public sealed class ScriptWorkerFrameWriter
|
||||
results.Items.AddRange(envelope.Results.Items);
|
||||
results.StartedEvalIds.AddRange(envelope.Results.StartedEvalIds);
|
||||
results.DeferredEvalIds.AddRange(envelope.Results.DeferredEvalIds);
|
||||
results.SoftExpiredEvalIds.AddRange(envelope.Results.SoftExpiredEvalIds);
|
||||
break;
|
||||
default:
|
||||
if (evals is not null) { Append(new Envelope { Eval = evals }); evals = null; }
|
||||
|
||||
Reference in New Issue
Block a user