fix(scripting): out-of-process soft deadline measures the script's own run, not the pipe or a starved worker (#827)
The soft deadline T was armed on the host at submit, so the pipe round trip
and the time a starved worker waits to be scheduled counted against the
script: under load a trivial script reported 'script timed out', in Shadow
mode that was a script mismatch, and at T + G the host killed the worker and
struck the key.
- Protocol (additive, negotiated): Hello/Ready.worker_soft_clock and
EvalResults.soft_expired_eval_ids. protocol_version stays 1; a worker that
does not echo the flag keeps the pre-#827 host clock.
- Worker: the watchdog runs each run's soft clock from the moment an
evaluation thread starts it and reports the crossing (woken early for
budgets shorter than its 50 ms poll). It still never cancels by itself;
its fail-fast is anchored at the soft notice + G + margin.
- Host (PendingEval): the soft notice answers the caller and arms the kill G
later. The host's own timer becomes a transport deadline, T + G from submit
(restarted at a deferred run's start); passing it with no result and no
notice yields TransportTimedOut -> Unavailable 'script worker unresponsive'
(BadResourceUnavailable, recovery = the run's completion), no strike, no
kill, no Cancel. A re-trigger refused by that still-unanswered run is
Unavailable too, not 'previous run still running'. Counted as
otopcua.script.ipc.timeouts{kind}.
- Frame writer: coalescing now carries soft_expired_eval_ids (it silently
dropped them); the host processes soft notices before results in a frame.
- Shadow: unresponsive is shadow_unavailable (Unavailable), never a mismatch.
Tests: WorkerSoftClockTests suspends the real worker (SIGSTOP via libc kill /
NtSuspendProcess) - a stall past T still returns 7; a stall past T + G is a
runner fault with nothing struck; a late-starting runaway is still timed out
at T from its start and killed G after; Shadow stays match /
shadow_unavailable; both clocks enforce T and G; PendingEval state machine.
This commit is contained in:
+15
-2
@@ -58,6 +58,18 @@ processes:
|
||||
`script timed out after …` failure and the run token is cancelled inside the worker; at `T + G`
|
||||
(`Scripting:Worker:HardGrace`, default `max(1 s, T)`) the worker is **killed** (`Process.Kill(entireProcessTree)`,
|
||||
plus a worker-side fail-fast watchdog) — the thread, its memory and its ALC go with it.
|
||||
- **`T` is the script's own run time (lmxopcua#827).** As in process (where the guard's clock starts when the
|
||||
thread pool picks the run up), `T` starts when a worker evaluation thread starts the run — not at submit — so the
|
||||
pipe round trip and the time a starved worker waits to be scheduled never count against the script. The worker runs
|
||||
that clock and reports the crossing (`EvalResults.soft_expired_eval_ids`); the host answers its caller, sends the
|
||||
cancel, and arms the kill `G` after the notice. What the host still times on its own is a **transport deadline**,
|
||||
`T + G` from submit (restarted at a queued run's start): no result and no soft notice by then is the runner's fault —
|
||||
the tag publishes **`BadResourceUnavailable`** (`"script worker unresponsive"`), nothing is struck or quarantined,
|
||||
the worker is not killed (a really hung one goes silent and the heartbeat monitor kills it), and the seam
|
||||
re-evaluates when the worker answers. A re-trigger while that run is still unanswered is the runner's fault too, not
|
||||
`previous run still running`. Counted as `otopcua.script.ipc.timeouts{kind}`. The clock is negotiated in the handshake
|
||||
(`Hello`/`Ready.worker_soft_clock`): a worker that does not echo it keeps the host clock from submit (the pre-#827
|
||||
contract), so a mismatched host/worker pair degrades to the old behaviour instead of losing its deadlines.
|
||||
- **Attribution and quarantine** live in the host and survive restarts: a deadline kill strikes exactly its key
|
||||
(other evaluations in flight are victims, not strikes); a crash with no deadline crossed makes every key in flight
|
||||
a suspect that must complete `Scripting:Worker:IsolationProbationRuns` (20) clean runs in the isolation worker,
|
||||
@@ -108,7 +120,7 @@ processes:
|
||||
"every process with this name", never a live host's worker (mxaccessgw's `OrphanWorkerTerminator` lesson).
|
||||
- **Observability.** On `OtOpcUaTelemetry.Meter`: `otopcua.script.runner.state` (0 ready / 1 degraded / 2 circuit
|
||||
open), `otopcua.script.worker.up` / `.inflight` / `.queue_depth` / `.heap_bytes` (tag `shard`; queue and heap from
|
||||
the worker heartbeat), `otopcua.script.worker.starts` / `.exits` (`reason` adds `idle`), `otopcua.script.ipc.roundtrip`
|
||||
the worker heartbeat), `otopcua.script.worker.starts` / `.exits` (`reason` adds `idle`), `otopcua.script.ipc.roundtrip`, `otopcua.script.ipc.timeouts` (transport deadline, lmxopcua#827)
|
||||
and `otopcua.script.eval.duration` (µs, tag `kind`; no per-key tag), `otopcua.script.compile` (`cache` = `hit` |
|
||||
`miss` | `rejected` | `uncached` | `negative`) and `otopcua.script.isolation.suspects`. The `script-runner` health check
|
||||
is on the **ready** tier and is at worst **Degraded** (still 200): a sick runner never fails readiness, so a pair
|
||||
@@ -168,7 +180,8 @@ what is published** — and each compiled evaluation is then mirrored to the `Sc
|
||||
|
||||
- **Compared:** value (structural for arrays), exact CLR type, status (Ok vs failed) and failure class (`compile`,
|
||||
`budget` = timed out / killed / refused / quarantined, `threw` with its message, `unsupported_type`). A shadow outcome
|
||||
that is the runner's own fault (worker restarted, none ready, worker busy) is **not compared**
|
||||
that is the runner's own fault (worker restarted, none ready, worker busy, worker unresponsive — a stalled or starved
|
||||
worker past the transport deadline, lmxopcua#827) is **not compared**
|
||||
(`result = shadow_unavailable`). An input or result the protocol cannot carry (design §14 #11) counts as a mismatch
|
||||
with reason `unsupported_type` — out of process that evaluation would publish Bad.
|
||||
- **Never reaches a published value or readiness.** The mirror copies the inputs and queues the shadow run on the pool;
|
||||
|
||||
Reference in New Issue
Block a user